From d1876c9178b7179ce3106b7aabc4b28245aa25c0 Mon Sep 17 00:00:00 2001 From: Jason Stajich Date: Mon, 29 Jun 2026 15:31:45 -0700 Subject: [PATCH 1/2] =?UTF-8?q?feat(#12):=20nf-core=20hygiene=20=E2=80=94?= =?UTF-8?q?=20docs,=20schema=5Finput,=20naming=20decision?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs/usage.md — quick-start, profile axis table, samplesheet column reference, filtering flags, key params, resume/stub/EarlGrey recipes docs/output.md — full directory layout, per-output descriptions, storeDir vs publishDir persistence model assets/schema_input.json — JSON Schema for samples.csv (required: SPECIES, ASMID, LOCUSTAG, BUSCO_LINEAGE; optional: STRAIN, TRANSL_TABLE, NCBI_TAXONID, GENOME, taxonomy rank columns) REFACTORING_PLAN.md / REFACTOR_ISSUES.md — issues #11 + #12 marked done; naming decision recorded: stay nf-core-inspired, do not submit (nf_funannotate1 naming kept; container-per-module and nf-test not pursued for this HPC-first pipeline). Epic closed. Stub gate: 15/0. Co-Authored-By: Claude Sonnet 4.6 Claude-Session: https://claude.ai/code/session_01AbWwmm9v6dJn13sgAJn3Sa --- .github/REFACTOR_ISSUES.md | 24 ++++---- REFACTORING_PLAN.md | 29 +++++---- assets/schema_input.json | 85 ++++++++++++++++++++++++++ docs/output.md | 118 +++++++++++++++++++++++++++++++++++++ docs/usage.md | 115 ++++++++++++++++++++++++++++++++++++ 5 files changed, 346 insertions(+), 25 deletions(-) create mode 100644 assets/schema_input.json create mode 100644 docs/output.md create mode 100644 docs/usage.md diff --git a/.github/REFACTOR_ISSUES.md b/.github/REFACTOR_ISSUES.md index db11f87..cac0f7e 100644 --- a/.github/REFACTOR_ISSUES.md +++ b/.github/REFACTOR_ISSUES.md @@ -19,7 +19,7 @@ composed by subworkflows, on a `meta`-map data contract.~~ **DONE** — `funanno is now 163 lines (0 inline processes). 24 modules, 7 subworkflows, shared `lib/FunannotateUtils.groovy`. nf-core-*inspired*, not nf-core-submitted. -Remaining: issues #11 (institutional profile) and #12 (docs/hygiene, stretch). +All issues complete (#11 institutional profile, #12 docs/hygiene). Epic closed. --- @@ -115,18 +115,16 @@ partitions / `clusterOptions` into the institutional profile, and ensure a fully portable run works via per-module conda/biocontainer directives (no Lmod). Document the "copy to `conf/provision_.config`" path for new sites. -- [ ] UCR partition config consolidated under the institutional profile -- [ ] Portable container/conda path runs without any UCR modules -- [ ] `docs/` note for adding a new institution +- [x] UCR partition config consolidated under the institutional profile +- [x] `docs/adding_a_site.md` guide for new institutions +- [ ] Portable container/conda path runs without any UCR modules (deferred — not a current priority) -## 12. nf-core hygiene (stretch) (Phase 9) +## 12. nf-core hygiene (Phase 9) ✅ DONE labels: refactor, documentation -`docs/usage.md` + `docs/output.md`, `assets/schema_input.json` (samplesheet -schema), pipeline naming decision (nf-core forbids underscores/digits), optional -MultiQC + `nf-test`. Decide explicitly whether to pursue nf-core submission or -stay nf-core-inspired. - -- [ ] `docs/usage.md`, `docs/output.md` -- [ ] `assets/schema_input.json` -- [ ] naming + submission decision recorded in `REFACTORING_PLAN.md` +- [x] `docs/usage.md` +- [x] `docs/output.md` +- [x] `assets/schema_input.json` +- [x] Naming + nf-core submission decision recorded in `REFACTORING_PLAN.md` + → Decision: stay nf-core-*inspired*, do not submit. `nf_funannotate1` naming + kept; container-per-module and nf-test not pursued for this HPC-first pipeline. diff --git a/REFACTORING_PLAN.md b/REFACTORING_PLAN.md index 14c961b..3ca77f2 100644 --- a/REFACTORING_PLAN.md +++ b/REFACTORING_PLAN.md @@ -20,8 +20,8 @@ (`run_repeatmasker=true`, default), or pass-through. - Stub-run gate: **15/0** (tantan), **16/0** (EarlGrey), **17/0** (+ SRA fetch). -**Remaining work:** Issues #11 (ucr_hpcc profile consolidation) and #12 (nf-core -hygiene: docs/usage.md, docs/output.md, schema_input.json, MultiQC — stretch). +**All refactoring issues complete.** Issues #11 (ucr_hpcc profile consolidation) +and #12 (nf-core hygiene: docs/usage.md, docs/output.md, assets/schema_input.json) done. The goal is an **nf-core-*inspired*** layout (not nf-core submission): adopt the parts that are pure engineering wins, skip the parts that fight our HPC reality. @@ -195,8 +195,8 @@ pattern on a leaf first, then attack the hard pieces. | 7 | TRAIN_PREDICT subworkflow (RNASEQ_PREPARE + FUNANNOTATE_TRAIN + FUNANNOTATE_PREDICT) | ✅ done | | 8 | ANNOTATE_GENOME subworkflow (optional annotation chain) | ✅ done | | 8b | `lib/FunannotateUtils.groovy` (shared utilities, no more duplication) | ✅ done | -| 9 | `#11` ucr_hpcc institutional profile consolidation | ⬜ todo | -| 9 | `#12` nf-core hygiene (docs/usage, docs/output, schema_input, MultiQC) | ⬜ stretch | +| 9 | `#11` ucr_hpcc institutional profile consolidation | ✅ done | +| 9 | `#12` nf-core hygiene (docs/usage, docs/output, schema_input) | ✅ done | --- @@ -229,11 +229,16 @@ Issue 11 covers the fuller consolidation (folding UCR SLURM partitions / | `versions.yml` per module + MultiQC | partial — some modules emit; no MultiQC yet | | Containers per module | **remaining gap** — relies on Lmod/pixi; nf-core needs conda+biocontainer per process | | Naming | `nf_funannotate1` violates nf-core naming (underscores/digits) — not a priority | -| nf-test, `docs/usage.md`+`output.md`, `assets/schema_input.json`, `.nf-core.yml` | missing (issue #12 stretch) | - -**Verdict:** core engineering work (meta-maps, one-tool-per-module, subworkflows, -shared utilities) is complete. Remaining gaps are nf-core submission requirements -that fight our HPC reality (container-per-module, naming). **Recommendation: -nf-core-inspired, not nf-core-submitted** — keep `ucr_hpcc` as an institutional -profile, add a real container path for portability (issue #11), defer -docs/MultiQC/nf-test to issue #12 stretch work. +| `docs/usage.md`, `docs/output.md`, `assets/schema_input.json` | ✅ done (issue #12) | +| `.nf-core.yml`, nf-test, MultiQC | not pursued — see naming decision below | + +**Naming decision (recorded):** `nf_funannotate1` violates nf-core naming rules +(underscores, trailing digit). Renaming is not worth the disruption: the pipeline +is HPC-first, the Lmod/UCR HPCC provisioning model does not fit the +container-per-module requirement, and nf-core submission is not a goal. The +repository stays `nf_funannotate1` with an nf-core-*inspired* structure (meta-map, +subworkflows/local, modules/local, schema, CI) but is not submitted. + +**Verdict:** all planned refactoring is complete. Remaining gaps (nf-test, +MultiQC, container-per-module) are nf-core *submission* requirements that do not +improve the pipeline for its intended HPC use. diff --git a/assets/schema_input.json b/assets/schema_input.json new file mode 100644 index 0000000..4be7fd0 --- /dev/null +++ b/assets/schema_input.json @@ -0,0 +1,85 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://raw.githubusercontent.com/stajichlab/nf_funannotate1/main/assets/schema_input.json", + "title": "stajichlab/nf_funannotate1 — samplesheet schema", + "description": "Schema for the samples.csv samplesheet passed to --samples.", + "type": "array", + "items": { + "type": "object", + "properties": { + "SPECIES": { + "type": "string", + "description": "Binomial species name, e.g. 'Aspergillus fumigatus'. Used to derive the sample tag and to look up RNA-seq data.", + "errorMessage": "SPECIES is required and must be a non-empty string" + }, + "STRAIN": { + "type": "string", + "description": "Strain / isolate identifier, e.g. 'Af293'. May be blank; the first semicolon-delimited token is used.", + "default": "" + }, + "ASMID": { + "type": "string", + "description": "Assembly identifier, e.g. 'GCA_000002655.1' or a local slug. Used as the primary sample key and output directory name.", + "errorMessage": "ASMID is required and must be a non-empty string" + }, + "LOCUSTAG": { + "type": "string", + "description": "GenBank locus-tag prefix assigned to this assembly, e.g. 'AFUA'.", + "errorMessage": "LOCUSTAG is required" + }, + "BUSCO_LINEAGE": { + "type": "string", + "description": "BUSCO lineage dataset name, e.g. 'fungi_odb10' or 'saccharomycetes_odb12'. Must match a dataset available in the local BUSCO downloads.", + "errorMessage": "BUSCO_LINEAGE is required" + }, + "TRANSL_TABLE": { + "type": "integer", + "minimum": 1, + "description": "NCBI genetic code translation table number (1 = standard, 4 = Mycoplasma/Spiroplasma, 12 = alt-yeast). Defaults to 1 if blank.", + "default": 1 + }, + "NCBI_TAXONID": { + "type": "string", + "description": "NCBI Taxonomy ID for the species (used for SRA queries). May be blank if run_sra_fetch=false.", + "default": "" + }, + "GENOME": { + "type": "string", + "description": "Path to the input genome FASTA (.fa, .fna, .fasta, or .gz). Absolute path or relative to launchDir. If blank, the genome is resolved from params.source as //_genomic.fna.gz.", + "default": "" + }, + "PHYLUM": { + "type": "string", + "description": "Phylum name (optional). Used by --taxon PHYLUM:Ascomycota to filter samples.", + "default": "" + }, + "SUBPHYLUM": { + "type": "string", + "description": "Subphylum name (optional). Used by --taxon SUBPHYLUM:Pezizomycotina.", + "default": "" + }, + "CLASS": { + "type": "string", + "description": "Class name (optional). Used by --taxon CLASS:Eurotiomycetes.", + "default": "" + }, + "ORDER": { + "type": "string", + "description": "Order name (optional). Used by --taxon ORDER:Eurotiales.", + "default": "" + }, + "FAMILY": { + "type": "string", + "description": "Family name (optional). Used by --taxon FAMILY:Aspergillaceae.", + "default": "" + }, + "GENUS": { + "type": "string", + "description": "Genus name (optional). Used by --taxon GENUS:Aspergillus.", + "default": "" + } + }, + "required": ["SPECIES", "ASMID", "LOCUSTAG", "BUSCO_LINEAGE"], + "dependentRequired": {} + } +} diff --git a/docs/output.md b/docs/output.md new file mode 100644 index 0000000..76dde92 --- /dev/null +++ b/docs/output.md @@ -0,0 +1,118 @@ +# Output + +All paths are relative to `launchDir` (the directory from which `nextflow run` is invoked) +unless noted as params that can be overridden. + +## Directory layout + +``` +launchDir/ +├── input_clean_genomes/ # Cleaned + masked genome FASTAs +│ ├── .fa.gz # FCS-GX cleaned, length-filtered +│ └── .masked.fasta.gz # Soft-masked (tantan or EarlGrey path) +│ +├── genome_annotation/ # Main funannotate output (params.target) +│ └── / +│ ├── predict_results/ # Funannotate predict outputs +│ │ ├── .gbk # GenBank annotation +│ │ ├── .gff3 # Gene models +│ │ └── .proteins.fa +│ ├── annotate_results/ # Funannotate annotate outputs (run_annotate=true) +│ │ ├── .gbk +│ │ └── .gff3 +│ ├── antismash/ # antiSMASH results (run_antismash=true) +│ ├── interproscan/ # InterProScan results (run_interpro=true) +│ └── signalp/ # SignalP results (run_signalp=true) +│ +├── genome_annotation_training/ # Funannotate train artifacts (params.training_target) +│ └── / +│ └── training/ +│ +├── rnaseq_data/ # Trinity assemblies + normalized reads +│ ├── .trinity-GG.fasta +│ ├── _norm_R1.fastq.gz +│ ├── _norm_R2.fastq.gz +│ └── _norm_SE.fastq.gz +│ +├── rnaseq_reads/ # Fetched + trimmed SRA reads +│ ├── _R1.fastq.gz +│ └── sra_query/ # Per-species SRA run lists +│ └── _sra.csv +│ +├── tables/ # Assembly statistics (params.tables_dir) +│ └── asm_stats.tsv.gz # N50, contig count, bp — used by EarlGrey SELECT_REPS +│ +├── results/repeatlibrary/ # EarlGrey TE libraries (params.earlgrey_outdir) +│ └── / +│ ├── .families.fa # Curated TE family library +│ ├── .masked.fasta.gz # Representative masked genome +│ ├── _RepeatLandscape/ +│ └── _summaryFiles/ +│ +├── funannotate_db/ # Funannotate annotation databases (params.funannotate_db) +├── lib/augustus/3.5/config/ # Writable Augustus config copy (params.augustus_config) +│ +└── logs/nextflow/ + ├── annotate_trace.txt # Per-task resource usage + ├── annotate_report.html # Visual execution report + ├── annotate_timeline.html # Timeline view + └── software_versions.yml # Tool versions collected from all processes +``` + +## Key outputs + +### Genome annotations (`genome_annotation//`) + +The central output of funannotate predict. Each assembly gets its own subdirectory +named by `ASMID`. The `predict_results/` subdirectory always exists after a +successful predict run; `annotate_results/` is added when `--run_annotate true`. + +| File | Description | +|------|-------------| +| `predict_results/.gbk` | GenBank flat-file with gene models | +| `predict_results/.gff3` | GFF3 gene models | +| `predict_results/.proteins.fa` | Predicted protein sequences | +| `annotate_results/.gbk` | GenBank with functional annotations added | + +### Cleaned and masked genomes (`input_clean_genomes/`) + +Intermediate cleaned genomes are written here by storeDir caching. The pipeline +reads back from this directory on resumed runs — the files act as checkpoints. + +| File | Description | +|------|-------------| +| `.fa.gz` | FCS-GX cleaned (or length-filtered only if `--skip_fcs true`) | +| `.masked.fasta.gz` | Soft-masked with tantan (default) or EarlGrey + RepeatMasker | + +### RNA-seq evidence (`rnaseq_data/`, `rnaseq_reads/`) + +RNA-seq data is organized by species tag (`{species}_{strain}` with whitespace +replaced by `_`). Normalized reads are storeDir-cached so the expensive Trinity +assembly is not repeated on `-resume`. + +### Assembly statistics (`tables/asm_stats.tsv.gz`) + +Generated when `--gen_asm_stats true` (default). Contains N50, contig count, and +assembled bp per assembly. Used by the EarlGrey path to select the most contiguous +representative per species for TE library construction. + +### EarlGrey TE libraries (`results/repeatlibrary/`) + +Only generated when `--run_earlgrey true`. The curated TE family FASTA +(`.families.fa`) is built once per species and reused by RepeatMasker for +all conspecific strains. + +### Logs (`logs/nextflow/`) + +Nextflow trace, HTML report, and timeline are written here. `software_versions.yml` +collects the tool versions emitted by each process that outputs a `versions.yml`. + +## Persistence model + +This pipeline uses `storeDir` (not `publishDir`) for most outputs. This means: + +- **First run**: task executes and writes outputs into the storeDir path. +- **Subsequent runs**: if the output file exists and is non-empty in the storeDir path, the task is skipped entirely (no work-dir entry created). +- **Effect**: outputs accumulate across runs in a shared directory tree. Deleting a storeDir file forces that task to re-run. + +`publishDir` (copy mode) is used only for: SRA query results, antiSMASH, InterProScan, and SignalP — these are copied from the Nextflow work directory into the output tree after the task completes. diff --git a/docs/usage.md b/docs/usage.md new file mode 100644 index 0000000..64045d2 --- /dev/null +++ b/docs/usage.md @@ -0,0 +1,115 @@ +# Usage + +## Quick start + +```bash +nextflow run . -profile annotate,slurm,ucr_hpcc --samples samples.csv -resume +``` + +Profiles compose across three orthogonal axes — pick one from each: + +| Axis | Options | Notes | +|------|---------|-------| +| pipeline | `annotate` | funannotate.nf (genome cleaning → masking → training → prediction → annotation) | +| executor | `slurm` \| `local` | `slurm` submits jobs via SBATCH; `local` runs on the head node | +| provisioning | `ucr_hpcc` \| `pixi` \| `singularity` | `ucr_hpcc` = UCR HPCC Lmod modules; see [Adding a site](adding_a_site.md) for other HPC environments | + +## Samplesheet (`samples.csv`) + +The samplesheet is a CSV with a header row. Required columns: + +| Column | Type | Description | +|--------|------|-------------| +| `SPECIES` | string | Binomial species name, e.g. `Aspergillus fumigatus` | +| `ASMID` | string | Assembly ID, e.g. `GCA_000002655.1` or a local slug. Used as the output directory name and primary key. | +| `LOCUSTAG` | string | GenBank locus-tag prefix, e.g. `AFUA` | +| `BUSCO_LINEAGE` | string | BUSCO dataset, e.g. `fungi_odb10` or `saccharomycetes_odb12` | + +Optional columns: + +| Column | Default | Description | +|--------|---------|-------------| +| `STRAIN` | _(blank)_ | Strain/isolate identifier; first `;`-delimited token is used | +| `TRANSL_TABLE` | `1` | NCBI genetic code table (1 = standard, 4 = Mycoplasma, 12 = alt-yeast) | +| `NCBI_TAXONID` | _(blank)_ | NCBI Taxonomy ID for SRA queries (only needed if `--run_sra_fetch true`) | +| `GENOME` | _(blank)_ | Path to genome FASTA (`.fa`/`.fna`/`.fasta`/`.gz`). Absolute or relative to `launchDir`. If blank, resolved from `params.source//_genomic.fna.gz`. | +| `PHYLUM`, `SUBPHYLUM`, `CLASS`, `ORDER`, `FAMILY`, `GENUS` | _(blank)_ | Taxonomic rank columns used by `--taxon RANK:VALUE` filtering | + +The full column schema is in [`assets/schema_input.json`](../assets/schema_input.json). + +### Example + +```csv +SPECIES,STRAIN,ASMID,LOCUSTAG,BUSCO_LINEAGE,TRANSL_TABLE,NCBI_TAXONID,GENOME +Aspergillus fumigatus,Af293,GCA_000002655.1,AFUA,eurotiomycetes_odb10,1,746128, +Saccharomyces cerevisiae,S288C,GCA_000146045.2,YAL,saccharomycetes_odb12,12,4932,genomes/S288C.fa.gz +``` + +## Filtering samples at runtime + +```bash +# Process only one assembly +nextflow run . -profile annotate,slurm,ucr_hpcc --asmid GCA_000002655.1 + +# Process only Ascomycota +nextflow run . -profile annotate,slurm,ucr_hpcc --taxon PHYLUM:Ascomycota + +# Smoke-test with the first 2 assemblies +nextflow run . -profile annotate,slurm,ucr_hpcc --n_test 2 -stub-run +``` + +## Key parameters + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `--samples` | `samples.csv` | Samplesheet path | +| `--target` | `genome_annotation/` | Output directory for annotations | +| `--source` | _(UCR HPCC path)_ | NCBI_ASM source directory for genomes without a `GENOME` column | +| `--skip_fcs` | `false` | Skip FCS-GX contaminant purge (no 470 GB highmem node needed) | +| `--run_earlgrey` | `false` | Use EarlGrey TE discovery instead of tantan for repeat masking | +| `--run_sra_fetch` | `true` | Fetch RNA-seq reads from SRA for funannotate training | +| `--run_annotate` | `false` | Run funannotate annotate (antismash/interpro/signalp must also be enabled) | +| `--run_antismash` | `false` | Run antiSMASH secondary metabolite prediction | +| `--run_interpro` | `false` | Run InterProScan functional annotation | +| `--run_signalp` | `false` | Run SignalP signal-peptide prediction (requires GPU partition) | + +Run `nextflow run . --help` for the full schema-driven parameter list. + +## Resume a run + +```bash +nextflow run . -profile annotate,slurm,ucr_hpcc --samples samples.csv -resume +``` + +Nextflow's `cache = 'lenient'` (set in `nextflow.config`) handles NFS timestamp +jitter on `/bigdata`-mounted filesystems. Completed processes are skipped; only +changed or failed tasks re-run. + +## Stub / dry run + +```bash +nextflow run . -profile test -stub-run +``` + +The `test` profile is self-contained: it sets `beforeScript = ':'` for all +labels so no modules or containers are required. Stubs emit minimal placeholder +files so the full DAG can be traced without running real tools. + +## EarlGrey masking path + +To use EarlGrey (TE-discovery) instead of tantan repeat masking: + +```bash +nextflow run . -profile annotate,slurm,ucr_hpcc \ + --run_repeatmasker false --run_earlgrey true \ + --earlgrey_outdir results/repeatlibrary \ + --masked_dir input_clean_genomes +``` + +EarlGrey runs once per species (on the most contiguous representative assembly) +to build a curated TE library; RepeatMasker then applies it to every conspecific +strain. Requires the `earlgrey` profile for resource defaults: + +```bash +-profile annotate,earlgrey,slurm,ucr_hpcc +``` From 866fe2a7fc5e59f69ccf66a6ae6d57bc44b33dac Mon Sep 17 00:00:00 2001 From: Jason Stajich Date: Mon, 29 Jun 2026 15:52:07 -0700 Subject: [PATCH 2/2] =?UTF-8?q?fix:=20add=20validation.lenientMode=20for?= =?UTF-8?q?=20NF=2026.x=20CLI=20string=E2=86=92type=20coercion?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit NF 26.04.4 passes all CLI args as strings; nf-schema 2.6.1 validates them before Groovy type coercion, rejecting --n_test 1, --skip_fcs true, etc. lenientMode = true accepts string representations of typed params from CLI while still enforcing structure and required-field rules. Co-Authored-By: Claude Sonnet 4.6 Claude-Session: https://claude.ai/code/session_01AbWwmm9v6dJn13sgAJn3Sa --- nextflow.config | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/nextflow.config b/nextflow.config index 1db09dd..f4f827b 100644 --- a/nextflow.config +++ b/nextflow.config @@ -45,6 +45,13 @@ plugins { id 'nf-schema@2.6.1' } +// NF 26.x passes CLI args as strings; lenientMode lets nf-schema accept "true"/"1" +// where the schema declares boolean/integer — actual Groovy types are already correct. +validation { + lenientMode = true + monochromeLogs = false +} + // ── Shared params (pipeline profiles add their own) ─────────────────────────── params { samples = "${launchDir}/samples.csv"