Aviary

01Installation

# Set conda channels first (required)
conda config --add channels defaults
conda config --add channels bioconda
conda config --add channels conda-forge

# Option A — Bioconda
conda create -n aviary -c bioconda aviary
# OR into an existing environment:
conda install -c bioconda aviary

# Option B — Pip
pip install aviary-genome

# Option C — Source with pixi (recommended for development)
git clone https://github.com/rhysnewell/aviary.git
cd aviary
pixi run postinstall     # installs all dependencies

# Editable install for development
pip install -e .

02Quick Reference — Subcommands

aviary --version            # show installed version
aviary --help               # top-level help
aviary <subcommand> --help  # help for a specific subcommand

03Required Databases

Use aviary configure --download to auto-install all databases.

Environment variable paths

export GTDBTK_DATA_PATH=/path/to/gtdb/db/
export EGGNOG_DATA_DIR=/path/to/eggnog/
export SINGLEM_METAPACKAGE_PATH=/path/to/singlem.smpkg/
export CHECKM2DB=/path/to/checkm2db/
export CONDA_ENV_PATH=/path/to/conda/envs/

Tip: Point GTDBTK_DATA_PATH to the db/ folder inside the GTDB download directory, not the download root itself.

04Configure

aviary configure \
  -o logs/ \
  --eggnog-db-path /path/to/eggnog/ \
  --gtdb-path /path/to/gtdb/ \
  --checkm2-db-path /path/to/checkm2/ \
  --singlem-metapackage-path /path/to/singlem/ \
  --metabuli-db-path /path/to/metabuli/ \   # optional, for taxvamb
  --download                                 # auto-download missing databases

# Set a persistent tmpdir
aviary configure --tmpdir /scratch/tmp/

05Assemble

QC runs automatically before assembly unless --skip-qc is set.

# Short-read only (metaSPAdes by default)
aviary assemble -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz -o output_dir/ --n-cores 16

# Short-read with MEGAHIT (faster, lower memory)
aviary assemble -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz -o output_dir/ --n-cores 16 --use-megahit

# Long-read only (myloasm default)
aviary assemble -l sample.fastq.gz -z ont -o output_dir/ --n-cores 16

# Long-read with Flye
aviary assemble -l sample.fastq.gz -z ont --long-read-assembler flye -o output_dir/ --n-cores 16

# Hybrid (short + long reads)
aviary assemble -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz -l sample.fastq.gz -z ont -o output_dir/ --n-cores 16

# Host contamination removal
aviary assemble -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz -r host_genome.fasta -o output_dir/ --n-cores 16

# Co-assembly of multiple samples
aviary assemble -1 s1_R1.fq.gz s2_R1.fq.gz -2 s1_R2.fq.gz s2_R2.fq.gz --coassemble -o output_dir/ --n-cores 16

Read input flags

Assembly flags

Contig filtering flags (applied post-assembly)

Expected outputs

06QC / Read Filtering Flags

Applies to assemble, recover, and complete. QC runs automatically — use these flags to tune or skip it.

07Recover — Extract MAGs

Default binners

DAS_Tool is always run to produce a final consensus bin set from all binner outputs.

# Short reads only
aviary recover --assembly assembly.fasta -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz -o output_dir/ --n-cores 16 --max-threads 32

# With long reads for coverage (improves binning)
aviary recover --assembly assembly.fasta -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz -l sample.fastq.gz -z ont -o output_dir/ --n-cores 16

# GPU-accelerated binners (taxvamb, comebin, semibin)
aviary recover --assembly assembly.fasta -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz -o output_dir/ --n-cores 16 --request-gpu

# Add extra binners beyond defaults
aviary recover --assembly assembly.fasta -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz -o output_dir/ --n-cores 16 --extra-binners taxvamb comebin

# Skip specific binners
aviary recover --assembly assembly.fasta -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz -o output_dir/ --n-cores 16 --skip-binners concoct maxbin

# Binning only — skip annotation
aviary recover --assembly assembly.fasta -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz -o output_dir/ --n-cores 16 --binning-only

# Target a specific Snakemake rule
aviary recover --assembly assembly.fasta -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz -o output_dir/ -w rosella --max-threads 12

Recover flags

Expected outputs

08Annotate

aviary annotate \
  -d recover/bins/final_bins/ \
  -o output_dir/ \
  --n-cores 16

Note: The default extension is .fna, not .fasta. If your bins have a different extension (e.g. .fa, .fasta), always set -x accordingly.

Expected outputs

09Complete — Assemble + Recover + Annotate

Note: complete does NOT include cluster (dereplication). Run that separately after all samples.

# Short reads only
aviary complete -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz -o output_dir/ --n-cores 16

# Hybrid with myloasm (default)
aviary complete -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz -l sample.fastq.gz -z ont -o output_dir/ --n-cores 16

# Hybrid with Flye
aviary complete -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz -l sample.fastq.gz -z ont --long-read-assembler flye -o output_dir/ --n-cores 16

# With GPU binners
aviary complete -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz -l sample.fastq.gz -z ont -o output_dir/ --n-cores 16 --request-gpu

# Preview the workflow without running
aviary complete -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz -o output_dir/ --dry-run

10Cluster — Dereplicate MAGs

Run this once, after all samples are processed, to dereplicate the combined MAG set.

aviary cluster \
  -i run1/ run2/ run3/ \
  -o cluster_output/ \
  --n-cores 16

Expected outputs

11Batch — Multiple Samples

aviary batch \
  --batch-file samples.tsv \
  -o batch_output/ \
  --n-cores 16

The batch file is a TSV or CSV with one sample per row. Aviary will run each sample individually and then cluster the final output. See format example at https://rhysnewell.github.io/aviary/examples/example_batch.tsv.

12Diversity — Strain Analysis

aviary diversity \
  -d bins/final_bins/ \
  -1 sample_R1.fastq.gz \
  -2 sample_R2.fastq.gz \
  -o output_dir/ \
  --n-cores 16

Runs Lorikeet for strain-resolved variant calling and genotyping.

13Isolate — Single-Organism Assembly

Separate workflow from metagenomics — for isolated pure culture sequencing.

aviary isolate \
  -1 sample_R1.fastq.gz \
  -2 sample_R2.fastq.gz \
  -l sample.fastq.gz \
  -z ont \
  -o output_dir/ \
  --n-cores 16

14Common Flags

These flags apply across most subcommands.

Thread control explained

• -t / --threads — threads per program (how many each tool gets)
• -n / --n-cores — total cores Snakemake can use; if > --threads, multiple rules run in parallel
• -p / --pplacer-threads — pplacer gets its own limit (deadlocks on too many threads)

Unlock a directory after a crash

aviary complete -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz -o output_dir/ --snakemake-cmds '--unlock'

Resume an interrupted run

Snakemake automatically skips completed steps. Just re-run the exact same command.

15HPC / Cluster Job Submission

PBS / qsub (e.g. aqua cluster)

# Inline qsub submission — note the trailing space inside the quotes (argparse quirk)
aviary complete -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz -o output_dir/ --n-cores 16 \
  --snakemake-cmds '--cluster "qsub -V" '

# Using a named Snakemake profile
aviary complete -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz -o output_dir/ --n-cores 16 \
  --snakemake-profile aqua \
  --cluster-retries 2

Example PBS profile ~/.config/snakemake/aqua/config.yaml

cluster: qsub
cluster-status: qstat
jobs: 10000
cluster-cancel: qdel

PBS quirk: Add -V to your qsub command to pass environment variables to jobs. Without -V, pysam fails to activate properly in submitted jobs.

SLURM

aviary complete -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz -o output_dir/ --n-cores 16 \
  --snakemake-profile /path/to/slurm-profile/

Example SLURM wrapper script

#!/bin/bash
#SBATCH --job-name=aviary
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH --mem=64G
#SBATCH --time=48:00:00
#SBATCH --output=aviary_%j.log

conda activate aviary
aviary complete -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz -o output_dir/ --n-cores 16

sbatch submit_aviary.sh

Cluster retry behaviour

With --cluster-retries N, failed jobs are automatically resubmitted up to N times. Each retry increases the requested memory and wall time. Set --max-memory and --max-threads to cap these values.

Note on job resources: Job CPU and memory requirements were empirically derived from 1,000 Aviary runs. Binner rules are standardised at 24 CPUs. Memory is set to the nearest power of 2 above max RSS. These are capped by --max-memory / --max-threads.

16Full Workflow Walkthrough

Step-by-step (maximum control)

# Step 1 — Configure databases (once per system)
aviary configure -o logs/ \
  --eggnog-db-path /path/to/db/eggnog/ \
  --gtdb-path /path/to/db/gtdb/ \
  --checkm2-db-path /path/to/db/checkm2/ \
  --singlem-metapackage-path /path/to/db/singlem/ \
  --download

# Step 2 — QC + Assembly
aviary assemble -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz -l sample.fastq.gz -z ont -o 01_assemble/ --n-cores 16

# Step 3 — MAG Recovery
aviary recover \
  --assembly 01_assemble/assembly/final_contigs.fasta \
  -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz \
  -l sample.fastq.gz -z ont \
  -o 02_recover/ --n-cores 16 --max-threads 32 --request-gpu

# Step 4 — Annotate MAGs
aviary annotate -d 02_recover/bins/final_bins/ -o 03_annotate/ --n-cores 16

# Step 5 — Dereplicate (after all samples are done)
aviary cluster -i 02_recover/ run2/ run3/ -o 04_cluster/ --n-cores 16

Single-command alternative (steps 2–4)

aviary complete \
  -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz \
  -l sample.fastq.gz -z ont \
  -o output_dir/ --n-cores 16 --request-gpu

# Then run cluster (step 5) separately after all samples are done
aviary cluster -i output_dir/ run2/ run3/ -o cluster/ --n-cores 16

17Key Output Files

After assemble

After recover

After annotate

After cluster

18Debugging & Testing

Finding error messages

All log files are in output_dir/logs/. Each file is named after the rule that generated it.

# Example: SPAdes ran out of memory
cat output_dir/logs/run_spades.log

# List all log files
ls output_dir/logs/

run_tests_at_cmr — full expensive test suite

Must be run from a GPU interactive session. Start one with: mqinteractive gpu

# Recommended: submit via mqsub to control resources
mqsub -t 32 -m 250 --hours 12 pixi run run-tests-at-cmr

# OR: run directly (each test still submits its own job)
pixi run run-tests-at-cmr

• • Collects all @expensive and @qsub pytest tests
• • Submits each as a separate mqsub job (16 threads, 64 GB, 4 h)
• • GPU tests (taxvamb, comebin, semibin) are submitted with --A100
• • Waits for all jobs and reports failures

run_samples_at_cmr — benchmark across 9 real samples

pixi run -e dev postinstall                    # install dev environment
pixi run -e dev aviary build                   # build aviary in dev environment
pixi run -e dev python test/run_samples_at_cmr

• • Runs aviary recover --binning-only on 9 samples (Gut/Soil/Ocean) with both MEGAHIT and MetaSPAdes
• • Jobs dispatched via GNU parallel (-j 18) with aqua Snakemake profile
• • Output: timestamped directory per git commit under $PIXI_PROJECT_ROOT/aviary_test_samples/<COMMIT>/
• • Summary: {OUTPUT_DIR}/summary.tsv (columns: Env, Sample, Assembler, Score, Genomes)
• • Quality threshold: completeness − (5 × contamination) ≥ 50

Running tests locally

pixi run -e dev pytest test/                                       # standard tests only
pixi run -e dev pytest test/ --run-expensive                       # include expensive tests
pixi run -e dev pytest test/ --run-qsub                            # include qsub tests
pixi run -e dev pytest test/ --profile /path/to/snakemake-profile/ # use a cluster profile

19Tips, Gotchas & Common Errors

• 01/tmp filling up is the most common error (Error in prepare_binning_files). CoverM mapping during binning uses /tmp. Fix: --tmpdir /scratch/tmp/ or aviary configure --tmpdir /scratch/tmp/.
• 02SPAdes "Error code: -9" almost always means out of memory. Increase with -m (GB).
• 03qsub + pysam ModuleNotFoundError: Add -V to your qsub command to pass environment variables: --snakemake-cmds '--cluster "qsub -V" '
• 04Trailing space in --snakemake-cmds: The space after qsub in '--cluster qsub ' is required due to a Python argparse quirk.
• 05--request-gpu on a local machine: First install the cuda package into your conda environment. GPU-capable programs will detect it automatically.
• 06Resuming a run: Snakemake automatically skips completed rules. Just re-run the exact same command with the same -o output directory.
• 07Default fasta extension for annotate is .fna, not .fasta. If your bins have a different extension, always specify -x.
• 08aviary complete does NOT run cluster. You must always run aviary cluster separately after all samples have been processed.
• 09--binning-only in recover skips CheckM2 QC and GTDB-tk — useful when you want fast MAG binning and plan to run aviary annotate separately.
• 10--cluster-retries is highly recommended on HPC. Failed jobs are retried with progressively more memory and time, up to --max-memory / --max-threads caps.
• 11Setting GTDB path: Point GTDBTK_DATA_PATH to the db/ folder inside the GTDB download directory, not the download root itself.
• 12GPU binners require a GPU node. If using --request-gpu with a PBS/SLURM profile, ensure the profile is configured to submit GPU jobs to a GPU partition/queue.