stajichlab · hyphaltip · Jun 29, 2026 · Jun 29, 2026 · Jun 29, 2026
diff --git a/.github/REFACTOR_ISSUES.md b/.github/REFACTOR_ISSUES.md
@@ -19,7 +19,7 @@ composed by subworkflows, on a `meta`-map data contract.~~ **DONE** — `funanno
 is now 163 lines (0 inline processes). 24 modules, 7 subworkflows, shared
 `lib/FunannotateUtils.groovy`. nf-core-*inspired*, not nf-core-submitted.
 
-Remaining: issues #11 (institutional profile) and #12 (docs/hygiene, stretch).
+All issues complete (#11 institutional profile, #12 docs/hygiene). Epic closed.
 
 ---
 
@@ -115,18 +115,16 @@ partitions / `clusterOptions` into the institutional profile, and ensure a fully
 portable run works via per-module conda/biocontainer directives (no Lmod).
 Document the "copy to `conf/provision_<site>.config`" path for new sites.
 
-- [ ] UCR partition config consolidated under the institutional profile
-- [ ] Portable container/conda path runs without any UCR modules
-- [ ] `docs/` note for adding a new institution
+- [x] UCR partition config consolidated under the institutional profile
+- [x] `docs/adding_a_site.md` guide for new institutions
+- [ ] Portable container/conda path runs without any UCR modules (deferred — not a current priority)
 
-## 12. nf-core hygiene (stretch)  (Phase 9)
+## 12. nf-core hygiene  (Phase 9) ✅ DONE
 labels: refactor, documentation
 
-`docs/usage.md` + `docs/output.md`, `assets/schema_input.json` (samplesheet
-schema), pipeline naming decision (nf-core forbids underscores/digits), optional
-MultiQC + `nf-test`. Decide explicitly whether to pursue nf-core submission or
-stay nf-core-inspired.
-
-- [ ] `docs/usage.md`, `docs/output.md`
-- [ ] `assets/schema_input.json`
-- [ ] naming + submission decision recorded in `REFACTORING_PLAN.md`
+- [x] `docs/usage.md`
+- [x] `docs/output.md`
+- [x] `assets/schema_input.json`
+- [x] Naming + nf-core submission decision recorded in `REFACTORING_PLAN.md`
+      → Decision: stay nf-core-*inspired*, do not submit. `nf_funannotate1` naming
+        kept; container-per-module and nf-test not pursued for this HPC-first pipeline.
diff --git a/REFACTORING_PLAN.md b/REFACTORING_PLAN.md
@@ -20,8 +20,8 @@
   (`run_repeatmasker=true`, default), or pass-through.
 - Stub-run gate: **15/0** (tantan), **16/0** (EarlGrey), **17/0** (+ SRA fetch).
 
-**Remaining work:** Issues #11 (ucr_hpcc profile consolidation) and #12 (nf-core
-hygiene: docs/usage.md, docs/output.md, schema_input.json, MultiQC — stretch).
+**All refactoring issues complete.** Issues #11 (ucr_hpcc profile consolidation)
+and #12 (nf-core hygiene: docs/usage.md, docs/output.md, assets/schema_input.json) done.
 
 The goal is an **nf-core-*inspired*** layout (not nf-core submission): adopt the
 parts that are pure engineering wins, skip the parts that fight our HPC reality.
@@ -195,8 +195,8 @@ pattern on a leaf first, then attack the hard pieces.
 | 7 | TRAIN_PREDICT subworkflow (RNASEQ_PREPARE + FUNANNOTATE_TRAIN + FUNANNOTATE_PREDICT) | ✅ done |
 | 8 | ANNOTATE_GENOME subworkflow (optional annotation chain) | ✅ done |
 | 8b | `lib/FunannotateUtils.groovy` (shared utilities, no more duplication) | ✅ done |
-| 9 | `#11` ucr_hpcc institutional profile consolidation | ⬜ todo |
-| 9 | `#12` nf-core hygiene (docs/usage, docs/output, schema_input, MultiQC) | ⬜ stretch |
+| 9 | `#11` ucr_hpcc institutional profile consolidation | ✅ done |
+| 9 | `#12` nf-core hygiene (docs/usage, docs/output, schema_input) | ✅ done |
 
 ---
 
@@ -229,11 +229,16 @@ Issue 11 covers the fuller consolidation (folding UCR SLURM partitions /
 | `versions.yml` per module + MultiQC | partial — some modules emit; no MultiQC yet |
 | Containers per module | **remaining gap** — relies on Lmod/pixi; nf-core needs conda+biocontainer per process |
 | Naming | `nf_funannotate1` violates nf-core naming (underscores/digits) — not a priority |
-| nf-test, `docs/usage.md`+`output.md`, `assets/schema_input.json`, `.nf-core.yml` | missing (issue #12 stretch) |
-
-**Verdict:** core engineering work (meta-maps, one-tool-per-module, subworkflows,
-shared utilities) is complete. Remaining gaps are nf-core submission requirements
-that fight our HPC reality (container-per-module, naming). **Recommendation:
-nf-core-inspired, not nf-core-submitted** — keep `ucr_hpcc` as an institutional
-profile, add a real container path for portability (issue #11), defer
-docs/MultiQC/nf-test to issue #12 stretch work.
+| `docs/usage.md`, `docs/output.md`, `assets/schema_input.json` | ✅ done (issue #12) |
+| `.nf-core.yml`, nf-test, MultiQC | not pursued — see naming decision below |
+
+**Naming decision (recorded):** `nf_funannotate1` violates nf-core naming rules
+(underscores, trailing digit). Renaming is not worth the disruption: the pipeline
+is HPC-first, the Lmod/UCR HPCC provisioning model does not fit the
+container-per-module requirement, and nf-core submission is not a goal. The
+repository stays `nf_funannotate1` with an nf-core-*inspired* structure (meta-map,
+subworkflows/local, modules/local, schema, CI) but is not submitted.
+
+**Verdict:** all planned refactoring is complete. Remaining gaps (nf-test,
+MultiQC, container-per-module) are nf-core *submission* requirements that do not
+improve the pipeline for its intended HPC use.
diff --git a/assets/schema_input.json b/assets/schema_input.json
@@ -0,0 +1,85 @@
+{
+    "$schema": "https://json-schema.org/draft/2020-12/schema",
+    "$id": "https://raw.githubusercontent.com/stajichlab/nf_funannotate1/main/assets/schema_input.json",
+    "title": "stajichlab/nf_funannotate1 — samplesheet schema",
+    "description": "Schema for the samples.csv samplesheet passed to --samples.",
+    "type": "array",
+    "items": {
+        "type": "object",
+        "properties": {
+            "SPECIES": {
+                "type": "string",
+                "description": "Binomial species name, e.g. 'Aspergillus fumigatus'. Used to derive the sample tag and to look up RNA-seq data.",
+                "errorMessage": "SPECIES is required and must be a non-empty string"
+            },
+            "STRAIN": {
+                "type": "string",
+                "description": "Strain / isolate identifier, e.g. 'Af293'. May be blank; the first semicolon-delimited token is used.",
+                "default": ""
+            },
+            "ASMID": {
+                "type": "string",
+                "description": "Assembly identifier, e.g. 'GCA_000002655.1' or a local slug. Used as the primary sample key and output directory name.",
+                "errorMessage": "ASMID is required and must be a non-empty string"
+            },
+            "LOCUSTAG": {
+                "type": "string",
+                "description": "GenBank locus-tag prefix assigned to this assembly, e.g. 'AFUA'.",
+                "errorMessage": "LOCUSTAG is required"
+            },
+            "BUSCO_LINEAGE": {
+                "type": "string",
+                "description": "BUSCO lineage dataset name, e.g. 'fungi_odb10' or 'saccharomycetes_odb12'. Must match a dataset available in the local BUSCO downloads.",
+                "errorMessage": "BUSCO_LINEAGE is required"
+            },
+            "TRANSL_TABLE": {
+                "type": "integer",
+                "minimum": 1,
+                "description": "NCBI genetic code translation table number (1 = standard, 4 = Mycoplasma/Spiroplasma, 12 = alt-yeast). Defaults to 1 if blank.",
+                "default": 1
+            },
+            "NCBI_TAXONID": {
+                "type": "string",
+                "description": "NCBI Taxonomy ID for the species (used for SRA queries). May be blank if run_sra_fetch=false.",
+                "default": ""
+            },
+            "GENOME": {
+                "type": "string",
+                "description": "Path to the input genome FASTA (.fa, .fna, .fasta, or .gz). Absolute path or relative to launchDir. If blank, the genome is resolved from params.source as <source>/<ASMID>/<ASMID>_genomic.fna.gz.",
+                "default": ""
+            },
+            "PHYLUM": {
+                "type": "string",
+                "description": "Phylum name (optional). Used by --taxon PHYLUM:Ascomycota to filter samples.",
+                "default": ""
+            },
+            "SUBPHYLUM": {
+                "type": "string",
+                "description": "Subphylum name (optional). Used by --taxon SUBPHYLUM:Pezizomycotina.",
+                "default": ""
+            },
+            "CLASS": {
+                "type": "string",
+                "description": "Class name (optional). Used by --taxon CLASS:Eurotiomycetes.",
+                "default": ""
+            },
+            "ORDER": {
+                "type": "string",
+                "description": "Order name (optional). Used by --taxon ORDER:Eurotiales.",
+                "default": ""
+            },
+            "FAMILY": {
+                "type": "string",
+                "description": "Family name (optional). Used by --taxon FAMILY:Aspergillaceae.",
+                "default": ""
+            },
+            "GENUS": {
+                "type": "string",
+                "description": "Genus name (optional). Used by --taxon GENUS:Aspergillus.",
+                "default": ""
+            }
+        },
+        "required": ["SPECIES", "ASMID", "LOCUSTAG", "BUSCO_LINEAGE"],
+        "dependentRequired": {}
+    }
+}
diff --git a/docs/output.md b/docs/output.md
@@ -0,0 +1,118 @@
+# Output
+
+All paths are relative to `launchDir` (the directory from which `nextflow run` is invoked)
+unless noted as params that can be overridden.
+
+## Directory layout
+
+```
+launchDir/
+├── input_clean_genomes/          # Cleaned + masked genome FASTAs
+│   ├── <ASMID>.fa.gz             # FCS-GX cleaned, length-filtered
+│   └── <ASMID>.masked.fasta.gz   # Soft-masked (tantan or EarlGrey path)
+│
+├── genome_annotation/            # Main funannotate output (params.target)
+│   └── <ASMID>/
+│       ├── predict_results/      # Funannotate predict outputs
+│       │   ├── <ASMID>.gbk       # GenBank annotation
+│       │   ├── <ASMID>.gff3      # Gene models
+│       │   └── <ASMID>.proteins.fa
+│       ├── annotate_results/     # Funannotate annotate outputs (run_annotate=true)
+│       │   ├── <ASMID>.gbk
+│       │   └── <ASMID>.gff3
+│       ├── antismash/            # antiSMASH results (run_antismash=true)
+│       ├── interproscan/         # InterProScan results (run_interpro=true)
+│       └── signalp/              # SignalP results (run_signalp=true)
+│
+├── genome_annotation_training/   # Funannotate train artifacts (params.training_target)
+│   └── <ASMID>/
+│       └── training/
+│
+├── rnaseq_data/                  # Trinity assemblies + normalized reads
+│   ├── <species_tag>.trinity-GG.fasta
+│   ├── <species_tag>_norm_R1.fastq.gz
+│   ├── <species_tag>_norm_R2.fastq.gz
+│   └── <species_tag>_norm_SE.fastq.gz
+│
+├── rnaseq_reads/                 # Fetched + trimmed SRA reads
+│   ├── <accession>_R1.fastq.gz
+│   └── sra_query/                # Per-species SRA run lists
+│       └── <species_tag>_sra.csv
+│
+├── tables/                       # Assembly statistics (params.tables_dir)
+│   └── asm_stats.tsv.gz          # N50, contig count, bp — used by EarlGrey SELECT_REPS
+│
+├── results/repeatlibrary/        # EarlGrey TE libraries (params.earlgrey_outdir)
+│   └── <species_safe>/
+│       ├── <rep_ASMID>.families.fa        # Curated TE family library
+│       ├── <rep_ASMID>.masked.fasta.gz    # Representative masked genome
+│       ├── <species_safe>_RepeatLandscape/
+│       └── <species_safe>_summaryFiles/
+│
+├── funannotate_db/               # Funannotate annotation databases (params.funannotate_db)
+├── lib/augustus/3.5/config/      # Writable Augustus config copy (params.augustus_config)
+│
+└── logs/nextflow/
+    ├── annotate_trace.txt        # Per-task resource usage
+    ├── annotate_report.html      # Visual execution report
+    ├── annotate_timeline.html    # Timeline view
+    └── software_versions.yml     # Tool versions collected from all processes
+```
+
+## Key outputs
+
+### Genome annotations (`genome_annotation/<ASMID>/`)
+
+The central output of funannotate predict. Each assembly gets its own subdirectory
+named by `ASMID`. The `predict_results/` subdirectory always exists after a
+successful predict run; `annotate_results/` is added when `--run_annotate true`.
+
+| File | Description |
+|------|-------------|
+| `predict_results/<ASMID>.gbk` | GenBank flat-file with gene models |
+| `predict_results/<ASMID>.gff3` | GFF3 gene models |
+| `predict_results/<ASMID>.proteins.fa` | Predicted protein sequences |
+| `annotate_results/<ASMID>.gbk` | GenBank with functional annotations added |
+
+### Cleaned and masked genomes (`input_clean_genomes/`)
+
+Intermediate cleaned genomes are written here by storeDir caching. The pipeline
+reads back from this directory on resumed runs — the files act as checkpoints.
+
+| File | Description |
+|------|-------------|
+| `<ASMID>.fa.gz` | FCS-GX cleaned (or length-filtered only if `--skip_fcs true`) |
+| `<ASMID>.masked.fasta.gz` | Soft-masked with tantan (default) or EarlGrey + RepeatMasker |
+
+### RNA-seq evidence (`rnaseq_data/`, `rnaseq_reads/`)
+
+RNA-seq data is organized by species tag (`{species}_{strain}` with whitespace
+replaced by `_`). Normalized reads are storeDir-cached so the expensive Trinity
+assembly is not repeated on `-resume`.
+
+### Assembly statistics (`tables/asm_stats.tsv.gz`)
+
+Generated when `--gen_asm_stats true` (default). Contains N50, contig count, and
+assembled bp per assembly. Used by the EarlGrey path to select the most contiguous
+representative per species for TE library construction.
+
+### EarlGrey TE libraries (`results/repeatlibrary/`)
+
+Only generated when `--run_earlgrey true`. The curated TE family FASTA
+(`<ASMID>.families.fa`) is built once per species and reused by RepeatMasker for
+all conspecific strains.
+
+### Logs (`logs/nextflow/`)
+
+Nextflow trace, HTML report, and timeline are written here. `software_versions.yml`
+collects the tool versions emitted by each process that outputs a `versions.yml`.
+
+## Persistence model
+
+This pipeline uses `storeDir` (not `publishDir`) for most outputs. This means:
+
+- **First run**: task executes and writes outputs into the storeDir path.
+- **Subsequent runs**: if the output file exists and is non-empty in the storeDir path, the task is skipped entirely (no work-dir entry created).
+- **Effect**: outputs accumulate across runs in a shared directory tree. Deleting a storeDir file forces that task to re-run.
+
+`publishDir` (copy mode) is used only for: SRA query results, antiSMASH, InterProScan, and SignalP — these are copied from the Nextflow work directory into the output tree after the task completes.