diff --git a/.github/REFACTOR_ISSUES.md b/.github/REFACTOR_ISSUES.md new file mode 100644 index 0000000..df0c8e7 --- /dev/null +++ b/.github/REFACTOR_ISSUES.md @@ -0,0 +1,167 @@ +# Refactor issue tracker + +Source-of-truth backlog for the DSL2 modularization (see `REFACTORING_PLAN.md`). +Create these on GitHub with `scripts/create_refactor_issues.sh` (uses `gh`). +Each non-epic issue closes one `## Phase N` of the plan and must pass the +stub-run gate before merge: + +``` +nextflow config . -profile test && nextflow run . -profile test -stub-run && nextflow run . --help +``` + +--- + +## EPIC: Modularize nf_funannotate1 into DSL2 modules + subworkflows +labels: refactor, epic + +Break the 2,470-line `funannotate.nf` monolith into one-process-per-file modules +composed by subworkflows, on a `meta`-map data contract. nf-core-*inspired*, not +nf-core-submitted (see `REFACTORING_PLAN.md` for the verdict). Child issues #1–#12 +below; they are ordered — #1 and #2 block the rest. + +Definition of done: monolith replaced by `main.nf` + `workflows/` + +`subworkflows/local/` + `modules/local/`; every process emits `versions.yml`; +earlgrey/funannotate share `INPUT_CHECK`; CI green at every step. + +--- + +## 1. Adopt `meta`-map data contract (Phase 0 — BLOCKER) +labels: refactor + +Replace the positional 10-tuple +`tuple(out, asmid, species, strain, locustag, busco, header_length, transl_table, gz, taxonid)` +with `tuple val(meta), path(genome)` where `meta` is the map defined in +`REFACTORING_PLAN.md` (Principle 0). `meta.id` is the only naming field; +`header_length` becomes `params.header_length`. No more extraction until this lands. + +- [ ] Build `meta` in the workflow channel construction +- [ ] Update every call site to consume `meta` +- [ ] `params.header_length` added to schema with default 24 +- [ ] Stub-run gate green + +## 2. Shared `INPUT_CHECK` subworkflow (dedupe earlgrey) (Phase 0) +labels: refactor + +Extract the samplesheet `splitCsv` + taxon/asmid/suppress filtering (duplicated +in `funannotate.nf` and `earlgrey_mask.nf`) into `subworkflows/local/input_check.nf` +emitting the `meta` channel. Both entrypoints call it. + +- [ ] `subworkflows/local/input_check.nf` emits `ch_genomes` (meta + genome) +- [ ] `funannotate.nf` and `earlgrey_mask.nf` both consume it; no duplicated parse +- [ ] Stub-run gate green for both entrypoints + +## 3. Repo skeleton + relocate existing modules (Phase 1) +labels: refactor + +Create `main.nf`, `workflows/funannotate.nf`, `subworkflows/local/`, +`modules/local/`. Move `modules/asm_stats.nf` and `modules/annotation_tools.nf` +to one-process-per-file under `modules/local/` (split `annotation_tools.nf` into +`antismash.nf` / `signalp.nf` / `interproscan.nf`). + +- [ ] Directory skeleton in place; `main.nf` is a thin entrypoint +- [ ] `annotation_tools.nf` split into 3 single-process modules +- [ ] Stub-run gate green + +## 4. `versions.yml` + `conf/base.config` + `conf/modules.config` (Phase 2) +labels: refactor + +Establish the conventions every later module copies: each process emits +`versions.yml`; resources move to label-based `conf/base.config` +(`process_low/medium/high`); per-process `publishDir`/`ext.args` move to +`conf/modules.config`. + +- [ ] `conf/base.config` with resource labels +- [ ] `conf/modules.config` with publishDir + ext.args +- [ ] At least one module emits and the workflow collects `versions.yml` +- [ ] Stub-run gate green + +## 5. Setup modules (Phase 3) +labels: refactor, good first issue + +Extract `SETUP_TAXONDB`, `SETUP_FUNANNOTATE_DB`, `SETUP_AUGUSTUS_CONFIG` into +`modules/local/` + `subworkflows/local/setup_dbs.nf`. Preserve `storeDir` +(run-at-most-once) caching. Good first real extraction — validates the gate. + +- [ ] 3 modules + `setup_dbs.nf` subworkflow; storeDir preserved +- [ ] Stub-run gate green + +## 6. Genome clean + `prepare_genome` subworkflow (Phase 4) +labels: refactor + +Extract `GENOME_CLEAN` / `GENOME_CLEAN_BATCH` into modules and a +`subworkflows/local/prepare_genome.nf` (clean → asm_stats → mask). +**Preserve the FCS-GX `/dev/shm` staging** and the "skip already-cleaned" batch +gating so a fully-cleaned batch never pays the ~30-min staging cost. + +- [ ] Modules + `prepare_genome.nf`; FCS-GX /dev/shm staging preserved +- [ ] Batch padding/skip behavior unchanged +- [ ] Stub-run gate green + +## 7. Masking subworkflow + per-tool modules (Phase 5) +labels: refactor + +Replace the planned single masking mega-process. One module per masker +(`mask_tantan.nf` now; `mask_repeatmodeler.nf`, `mask_repeatmasker.nf`, +`mask_earlgrey.nf` as stubs/follow-ups); selection logic in +`subworkflows/local/mask.nf` keyed on `params.mask_tool`. + +- [ ] `mask.nf` selects one masker by param; `NONE` path supported +- [ ] `mask_tantan.nf` extracted; others stubbed with clear TODO +- [ ] Stub-run gate green + +## 8. RNA-seq fetch subworkflow (Phase 6 — hardest) +labels: refactor + +Extract `SRA_QUERY` / `SRA_QUERY_BATCH` / `COLLECT_SRA_QUERY` / +`WRITE_EMPTY_READS` / `SRA_FETCH` / `SRA_FETCH_SE` / `RNASEQ_PREPARE` into modules ++ `subworkflows/local/rnaseq.nf`. Done **after** the pattern is proven on easier +processes. Preserve per-species shared Trinity-GG output and `maxForks` limits. + +- [ ] 7 modules + `rnaseq.nf`; shared Trinity-GG semantics preserved +- [ ] maxForks / rate limits preserved +- [ ] Stub-run gate green + +## 9. Funannotate predict subworkflow (Phase 7) +labels: refactor + +Extract `FUNANNOTATE_TRAIN` / `FUNANNOTATE_PREDICT` / `FUNANNOTATE_UPDATE` into +modules + `subworkflows/local/predict.nf`. Keep the pre-flight assembly +size/fragmentation validation and post-flight "not enough models" guard. + +- [ ] 3 modules + `predict.nf`; pre/post-flight checks preserved +- [ ] `update` is optional (param-gated) +- [ ] Stub-run gate green + +## 10. Annotation subworkflow (Phase 8) +labels: refactor + +`subworkflows/local/annotate.nf` composing optional `antismash` / `signalp` / +`interproscan` modules → `funannotate_annotate.nf`. Each optional tool +independently param-gated. + +- [ ] `annotate.nf` with per-tool gating; merges results into funannotate annotate +- [ ] Stub-run gate green + +## 11. Consolidate `ucr_hpcc` institutional profile + portable container path (Phase 9) +labels: refactor + +The `module`→`ucr_hpcc` rename is done. Finish the repivot: fold UCR SLURM +partitions / `clusterOptions` into the institutional profile, and ensure a fully +portable run works via per-module conda/biocontainer directives (no Lmod). +Document the "copy to `conf/provision_.config`" path for new sites. + +- [ ] UCR partition config consolidated under the institutional profile +- [ ] Portable container/conda path runs without any UCR modules +- [ ] `docs/` note for adding a new institution + +## 12. nf-core hygiene (stretch) (Phase 9) +labels: refactor, documentation + +`docs/usage.md` + `docs/output.md`, `assets/schema_input.json` (samplesheet +schema), pipeline naming decision (nf-core forbids underscores/digits), optional +MultiQC + `nf-test`. Decide explicitly whether to pursue nf-core submission or +stay nf-core-inspired. + +- [ ] `docs/usage.md`, `docs/output.md` +- [ ] `assets/schema_input.json` +- [ ] naming + submission decision recorded in `REFACTORING_PLAN.md` diff --git a/IMPLEMENTATION_SUMMARY.md b/IMPLEMENTATION_SUMMARY.md deleted file mode 100644 index b64f2ce..0000000 --- a/IMPLEMENTATION_SUMMARY.md +++ /dev/null @@ -1,331 +0,0 @@ -# nf_funannotate1 Modularization: Implementation Summary & Review Request - -**Date:** 2026-06-28 -**Status:** Plan Complete, Ready for Expert Review -**Branch:** main - ---- - -## Executive Summary - -A comprehensive modularization plan has been developed to refactor the monolithic `funannotate.nf` pipeline into reusable, independently-testable modules. The plan organizes processes into 4 phases reflecting the actual workflow structure: - -1. **Genome Preprocessing** (AAFTF + Repeat Masking) -2. **Gene Prediction** (RNA-seq + Funannotate) -3. **Annotation** (Post-prediction tools) -4. **Setup & Utilities** (Database initialization) - -Three working implementations are production-ready, with a detailed rollout plan for the remaining components. - ---- - -## What's Complete ✅ - -### 1. ASM_STATS Module -- **File:** `modules/asm_stats.nf` -- **Status:** ✅ Working, production-ready -- **Function:** Generate assembly statistics (ASMID, total_length_bp, N50_bp, contig_count) -- **Note:** Will be relocated to `modules/AAFTF/asm_stats.nf` in Phase 1 -- **Integration:** Used by both `funannotate.nf` and `earlgrey_mask.nf` - -### 2. Optional SELECT_REPS -- **File:** `earlgrey_mask.nf` (updated) -- **Status:** ✅ Working, production-ready -- **Feature:** `--skip_select_reps` flag -- **Function:** Process all genomes for EarlGrey without size filtering -- **Parameter:** `--gen_asm_stats` (default: true) auto-generates assembly stats - -### 3. Annotation Tools Module -- **File:** `modules/annotation_tools.nf` -- **Status:** ✅ Working, production-ready -- **Processes:** - - ANTISMASH_RUN (secondary metabolite detection) - - INTERPROSCAN_RUN (protein domain annotation) - - SIGNALP_RUN (signal peptide prediction) -- **Note:** Will be relocated to `modules/annotate/annotation_tools.nf` in Phase 3 - -### 4. Documentation -- **REFACTORING_PLAN.md** - Complete phase breakdown with workflow diagram -- **modules/README.md** - Development guidelines, usage examples, testing procedures -- **MODULE_STRUCTURE.txt** - ASCII diagram showing module hierarchy and data flow - ---- - -## Proposed Modularization Plan - -### Phase 1: Genome Preprocessing (AAFTF & Repeat Masking) - -**Directory Structure:** -``` -modules/ -├── AAFTF/ -│ ├── asm_stats.nf (move from root) -│ ├── FCS_GX.nf (extract from GENOME_CLEAN) -│ ├── sourpurge.nf (extract from GENOME_CLEAN) -│ └── vecscreen.nf (new) -└── repeatmasking/ - └── masking.nf (strategy selector: TANTAN, REPEATMODELER, REPEATMASKER, EARLGREY, NONE) -``` - -**Key Decisions:** -- Consolidate all genome cleaning tools in modules/AAFTF/ -- Create flexible masking strategy selector to support multiple methods -- Preserve FCS-GX /dev/shm caching pattern during extraction - ---- - -### Phase 2: Gene Prediction (RNA-seq & Funannotate) - -**Directory Structure:** -``` -modules/ -├── rnaseq_fetch/ -│ ├── sra_query.nf (SRA_QUERY, SRA_QUERY_BATCH, COLLECT_SRA_QUERY) -│ ├── sra_fetch.nf (SRA_FETCH, SRA_FETCH_SE, WRITE_EMPTY_READS) -│ └── prepare.nf (RNASEQ_PREPARE) -└── funannotate/ - ├── train.nf (FUNANNOTATE_TRAIN) - ├── predict.nf (FUNANNOTATE_PREDICT) - └── update.nf (FUNANNOTATE_UPDATE - optional) -``` - -**Key Decisions:** -- Separate RNA-seq tools into dedicated module for reusability -- Keep funannotate core (train/predict/update) together -- Preserve complex channel workflows and skip-caching patterns - ---- - -### Phase 3: Annotation Workflow - -**Directory Structure:** -``` -modules/annotate/ -├── annotation_tools.nf (move from root - ANTISMASH, SIGNALP, INTERPROSCAN) -└── funannotate.nf (FUNANNOTATE_ANNOTATE) -``` - -**Key Decisions:** -- Group annotation tools together for easy composition -- Enable independent tool selection (run any/all of them) -- Support conditional execution per tool - ---- - -### Phase 4: Setup & Utilities - -**Directory Structure:** -``` -modules/setup/ -└── databases.nf (SETUP_TAXONDB, SETUP_FUNANNOTATE_DB, SETUP_AUGUSTUS_CONFIG) -``` - -**Key Decisions:** -- Foundational utilities used across all phases -- Preserve storeDir caching behavior (run at most once) -- Maintain current parameter organization - ---- - -## Implementation Priority - -1. **Phase 2.1 (RNA-seq Fetch)** ← START HERE - - Least interdependent - - High reusability potential - - Enables flexible training workflows - -2. **Phase 2.2 (Funannotate Core)** - - Enables flexible prediction pipelines - - Supports optional update step - - Well-tested existing code - -3. **Phase 1 (Genome Preprocessing)** - - More complex conditional branching - - Foundation for rest of pipeline - - Larger refactoring effort - -4. **Phase 3 (Annotation)** - - Depends on Phase 2 - - Enables composition of optional tools - - Most flexible phase - -5. **Phase 4 (Setup)** - - Last; foundational only - - No blocking dependencies - ---- - -## Benefits & Use Cases - -### Immediate Benefits -| Benefit | Details | -|---------|---------| -| **Reusability** | Modules can be used in different pipelines (e.g., funannotate-only vs. funannotate+earlgrey) | -| **Flexibility** | Easy to swap masking strategies or annotation tools without modifying main workflow | -| **Testability** | Each module testable in isolation with `-stub-run` | -| **Maintainability** | Smaller, focused files easier to understand and modify | -| **Documentation** | Clear module contracts (inputs/outputs/parameters) | - -### Future Use Cases -- Simplified parameter composition for specific workflows -- Easier integration of new tools (e.g., new masking strategies) -- Support for different organism pipelines (fungi → eukaryotes → prokaryotes) -- CI/CD pipeline testing per module -- Community contributions of alternative modules - ---- - -## Documentation Files - -All files are committed to git and available in the repository: - -1. **REFACTORING_PLAN.md** (177 lines) - - Detailed phase breakdown - - Complete workflow diagram - - Module-by-module responsibility assignment - - Rollout plan with priority order - -2. **modules/README.md** (176 lines) - - Development guidelines for module contributors - - Module documentation template - - Testing procedures (stub-run, unit tests) - - Parameter handling best practices - - Implementation checklist - -3. **MODULE_STRUCTURE.txt** (280 lines) - - ASCII diagram of module hierarchy - - Data flow visualization for each phase - - Input/output contracts per process - - Current vs. planned status matrix - -4. **IMPLEMENTATION_SUMMARY.md** (this file) - - Executive summary - - Complete artifacts list - - Review focus areas - - Specific recommendations for reviewers - ---- - -## Items Requiring Expert Review - -### 1. Architectural Soundness -- [ ] Are phase dependencies correctly ordered? -- [ ] Are module boundaries well-defined? -- [ ] Will module reusability work as designed? -- [ ] Any missing interdependencies? - -### 2. Implementation Feasibility -- [ ] Are extraction patterns consistent? -- [ ] Will existing parameters need major reorganization? -- [ ] Hidden interdependencies violating modularity? -- [ ] Can each phase be developed independently? - -### 3. Performance Impact -- [ ] Will include statements add overhead? -- [ ] Are storeDir/publishDir patterns preserved? -- [ ] Resume/checkpoint functionality unaffected? -- [ ] Any workflow DAG complexity changes? - -### 4. Backward Compatibility -- [ ] Can existing wrapper scripts continue working? -- [ ] Will parameter changes be transparent? -- [ ] How to handle user-provided nextflow.config? -- [ ] Deprecation strategy for old parameter names? - -### 5. Testing Strategy -- [ ] Are stub-run tests sufficient for module validation? -- [ ] Should we add integration tests? -- [ ] How to validate module independence? -- [ ] CI/CD pipeline changes needed? - -### 6. Documentation Adequacy -- [ ] Are module templates clear and complete? -- [ ] Need more implementation examples? -- [ ] Data contracts documented sufficiently? -- [ ] Guidelines for when/where to create new modules? - ---- - -## Specific Items for Review - -### Critical Decision Points - -1. **RNA-seq Fetch Extraction (Phase 2.1)** - - Complex interdependencies: SRA_QUERY → SRA_FETCH → RNASEQ_PREPARE - - Are channel flows properly documented? - - Can parameters be cleanly separated per module? - - **Question:** Should this be extracted in smaller sub-phases? - -2. **Repeat Masking Strategy Selector (Phase 1)** - - Multiple conditional branches (NONE, TANTAN, REPEATMODELER, REPEATMASKER, EARLGREY) - - How should strategy selection work? - - Should each be a separate module or combined? - - **Question:** Is `modules/repeatmasking/masking.nf` the right structure? - -3. **AAFTF Contamination Tools (Phase 1)** - - FCS_GX currently in GENOME_CLEAN_BATCH with complex /dev/shm staging - - Can this be cleanly extracted without breaking performance? - - How to preserve one-time 30-min staging cost amortization? - - **Question:** Should FCS_GX remain batched or become per-genome? - -4. **Setup Database Ordering (Phase 4)** - - Current: main workflow calls SETUP_*, then branches - - Can setup be fully parallelized from main? - - Do modules need to call setup themselves or assume it's done? - - **Question:** Should setup be implicit or explicit in each module? - ---- - -## Current Git History - -``` -80d05fe Add comprehensive module structure documentation -9867d96 Update modularization plan with detailed workflow structure -ec925c5 Add annotation_tools module and refactoring plan -4c50fd8 Add modular ASM_STATS and make SELECT_REPS optional -b13cd84 Merge pull request #1 from stajichlab/copilot/fix-parse-config-stub-run -``` - ---- - -## Next Steps (Pending Approval) - -### Immediate (This Week) -- [ ] Schedule expert code review -- [ ] Collect feedback on critical decision points -- [ ] Refine Phase 2.1 extraction strategy based on review - -### Short-term (This Sprint) -- [ ] Start Phase 2.1 implementation (RNA-seq modules) -- [ ] Create feature branch for Phase 2.1 -- [ ] Extract and test individual modules - -### Medium-term (Next 2-3 Sprints) -- [ ] Complete remaining phases -- [ ] Update main funannotate.nf to include modules progressively -- [ ] Document module usage patterns - -### Long-term (Post-implementation) -- [ ] Update CI/CD to test modules independently -- [ ] Create community contribution guidelines for new modules -- [ ] Document use cases for different organism types - ---- - -## Appendix: Files Ready for Review - -All documentation is production-ready and committed to the main branch: - -1. `REFACTORING_PLAN.md` - Complete architecture plan -2. `modules/README.md` - Developer guidelines -3. `MODULE_STRUCTURE.txt` - Visual reference -4. `modules/asm_stats.nf` - Working implementation -5. `modules/annotation_tools.nf` - Working implementation -6. `earlgrey_mask.nf` - Updated with optional SELECT_REPS -7. `conf/profile_annotate.config` - Updated parameters - ---- - -**Prepared by:** Claude Code (AI) -**Date:** 2026-06-28 -**Review Status:** ⏳ Awaiting Expert Review diff --git a/MODULE_STRUCTURE.txt b/MODULE_STRUCTURE.txt deleted file mode 100644 index b00e51c..0000000 --- a/MODULE_STRUCTURE.txt +++ /dev/null @@ -1,268 +0,0 @@ -================================================================================ - nf_funannotate1 Module Structure -================================================================================ - -ROOT: funannotate.nf (orchestrator) -│ -├── PHASE 1: GENOME PREPROCESSING -│ -│ ┌──────────────────────────────────────────────────────────────┐ -│ │ modules/AAFTF/ │ -│ │ (Assembly Annotation & Function Transfer) │ -│ └──────────────────────────────────────────────────────────────┘ -│ ├── asm_stats.nf -│ │ └── ASM_STATS (assembly statistics generation) -│ │ Input: samples.csv, genome_dir -│ │ Output: asm_stats.tsv.gz -│ │ -│ ├── FCS_GX.nf -│ │ ├── GENOME_CLEAN_BATCH (FCS-GX path) -│ │ │ Input: raw genome, taxonid -│ │ │ Output: clean genome, taxonomy report -│ │ │ -│ │ └── FCS contamination removal -│ │ Uses: /dev/shm staging (30 min one-time cost) -│ │ -│ ├── sourpurge.nf -│ │ └── Source organism contamination screening -│ │ -│ └── vecscreen.nf -│ └── Vector contamination screening -│ -│ ┌──────────────────────────────────────────────────────────────┐ -│ │ modules/repeatmasking/ │ -│ │ (Repeat Masking Strategy Selection) │ -│ └──────────────────────────────────────────────────────────────┘ -│ └── masking.nf -│ ├── NONE -│ │ └── Skip masking (use clean genome directly) -│ │ -│ ├── TANTAN -│ │ ├── MASKREPEAT_TANTAN_RUN -│ │ │ Input: clean genome -│ │ │ Output: tantan-masked genome -│ │ │ -│ │ └── Fast soft-masking (current default) -│ │ -│ ├── REPEATMODELER -│ │ ├── De novo TE discovery -│ │ └── Build custom library per genome -│ │ -│ ├── REPEATMASKER -│ │ ├── Library-based masking -│ │ ├── Input options: -│ │ │ ├── Existing library path -│ │ │ └── Species name (Repbase) -│ │ │ -│ │ └── Conservative masking -│ │ -│ └── EARLGREY -│ ├── De novo TE discovery + masking -│ ├── Representative per species only -│ ├── Curated library applied to strains -│ │ Input: representative genome -│ │ Output: masked representative + member genomes -│ │ -│ └── Currently in separate earlgrey_mask.nf pipeline -│ -│ -├── PHASE 2: GENE PREDICTION -│ -│ ┌──────────────────────────────────────────────────────────────┐ -│ │ modules/rnaseq_fetch/ │ -│ │ (RNA-seq Discovery & Preparation) │ -│ └──────────────────────────────────────────────────────────────┘ -│ │ -│ ├── sra_query.nf -│ │ ├── SRA_QUERY -│ │ │ Input: species_tag, taxonid -│ │ │ Output: per-species SRA accessions (CSV) -│ │ │ -│ │ ├── SRA_QUERY_BATCH -│ │ │ Parallel batch queries (maxForks: 4) -│ │ │ -│ │ └── COLLECT_SRA_QUERY -│ │ Merge all per-species CSVs into manifest -│ │ -│ ├── sra_fetch.nf -│ │ ├── SRA_FETCH -│ │ │ Input: per-species SRA accessions (CSV) -│ │ │ Output: normalized R1/R2 FASTQ -│ │ │ Steps: parallel-fastq-dump → seqkit → fastp → bbnorm -│ │ │ -│ │ ├── SRA_FETCH_SE -│ │ │ Input: SE accessions or mislabeled PE data -│ │ │ Output: normalized SE FASTQ -│ │ │ -│ │ └── WRITE_EMPTY_READS -│ │ Create placeholders for species with no SRA -│ │ -│ └── prepare.nf -│ ├── RNASEQ_PREPARE -│ │ Input: representative genome, normalized R1/R2 -│ │ Output: shared trinity-GG.fasta per species -│ │ Steps: funannotate train --stop_after_trinity -│ │ -│ └── Single assembly per species runs full training; -│ output shared with all conspecific strains -│ -│ ┌──────────────────────────────────────────────────────────────┐ -│ │ modules/funannotate/ │ -│ │ (Gene Prediction & Refinement) │ -│ └──────────────────────────────────────────────────────────────┘ -│ │ -│ ├── train.nf -│ │ ├── FUNANNOTATE_TRAIN -│ │ │ Representatives: funannotate train --stop_after_trinity -│ │ │ Strains: funannotate train --trinity (shared) -│ │ │ Input: cleaned/masked genome, reads, trinity-GG.fasta -│ │ │ Output: training data for predict -│ │ │ -│ │ └── PASA integration for transcript alignment -│ │ -│ ├── predict.nf -│ │ ├── FUNANNOTATE_PREDICT -│ │ │ Input: masked genome, training data -│ │ │ Output: predict_results/gbk, proteins.fa, gff3 -│ │ │ Steps: pre-flight validation → funannotate predict -│ │ │ -│ │ ├── Pre-flight checks: -│ │ │ ├── Assembly size ≥ predict_min_asm_bp -│ │ │ ├── N50 ≥ predict_frag_max_n50 -│ │ │ └── Contigs < predict_frag_max_contigs -│ │ │ -│ │ └── Post-flight safety: -│ │ ├── Validate GBK produced -│ │ ├── Catch "not enough models" early -│ │ └── Flag too-small/fragmented assemblies -│ │ -│ └── update.nf (Optional) -│ ├── FUNANNOTATE_UPDATE -│ │ Input: predict_results/, normalized reads -│ │ Output: updated annotations -│ │ -│ └── Refinement step for models with transcriptomics -│ -│ -├── PHASE 3: ANNOTATION -│ -│ ┌──────────────────────────────────────────────────────────────┐ -│ │ modules/annotate/ │ -│ │ (Post-prediction Annotation) │ -│ └──────────────────────────────────────────────────────────────┘ -│ │ -│ ├── annotation_tools.nf -│ │ ├── ANTISMASH_RUN -│ │ │ Input: predict_results/gbk -│ │ │ Output: antismash_local/ (JSON reports) -│ │ │ -│ │ ├── SIGNALP_RUN -│ │ │ Input: predict_results/proteins.fa -│ │ │ Output: annotate_misc/signalp.results.txt -│ │ │ -│ │ └── INTERPROSCAN_RUN -│ │ Input: predict_results/proteins.fa -│ │ Output: annotate_misc/iprscan.xml -│ │ -│ └── funannotate.nf -│ ├── FUNANNOTATE_ANNOTATE -│ │ Input: predict_results/, all optional tool outputs -│ │ Output: annotate_results/gbk, proteins.fa, gff3 -│ │ Steps: merge antismash/interpro/signalp → funannotate annotate -│ │ -│ └── Final functional annotation step -│ -│ -└── PHASE 4: SETUP & UTILITIES - - ┌──────────────────────────────────────────────────────────────┐ - │ modules/setup/ │ - │ (Database Initialization) │ - └──────────────────────────────────────────────────────────────┘ - │ - └── databases.nf - ├── SETUP_TAXONDB - │ Purpose: Download NCBI taxonomy (for FCS-GX) - │ Cached: params.taxondb (storeDir) - │ - ├── SETUP_FUNANNOTATE_DB - │ Purpose: Build funannotate databases (BUSCO, etc.) - │ Cached: params.funannotate_db (storeDir) - │ - └── SETUP_AUGUSTUS_CONFIG - Purpose: Seed writable Augustus config - Cached: params.augustus_config (storeDir) - Note: Augustus writes species configs during training - - -================================================================================ - WORKFLOW DATA FLOW -================================================================================ - -Raw Genomes (samples.csv + GENOME column) - │ - ├─→ [PHASE 1: PREPROCESSING] ─────────┐ - │ ├─→ CLEAN (FCS-GX/sourpurge) │ - │ │ │ - │ ├─→ SUMMARY_STATS (ASM_STATS) │ - │ │ └─→ asm_stats.tsv.gz │ - │ │ (used by: earlgrey_mask) │ - │ │ │ - │ └─→ MASK (strategy selector) │ - │ ├─→ NONE: skip │ - │ ├─→ TANTAN: fast │ - │ ├─→ REPEATMODELER │ - │ ├─→ REPEATMASKER │ - │ └─→ EARLGREY: curated │ - │ │ - ├─→ [PHASE 2: PREDICTION] ───────────┐ - │ ├─→ RNA-seq Branch: │ - │ │ ├─→ SRA_QUERY (NCBI) │ - │ │ ├─→ SRA_FETCH (download) │ - │ │ └─→ RNASEQ_PREPARE (Trinity) │ - │ │ │ - │ ├─→ TRAIN (funannotate) │ - │ │ │ - │ └─→ PREDICT (funannotate) │ - │ └─→ predict_results/ │ - │ ├─→ gbk (gene calls) │ - │ ├─→ proteins.fa │ - │ └─→ gff3 (features) │ - │ │ - ├─→ [PHASE 3: ANNOTATION] ──────────┐ - │ ├─→ Optional Tools (parallel): │ - │ │ ├─→ ANTISMASH │ - │ │ ├─→ SIGNALP │ - │ │ └─→ INTERPROSCAN │ - │ │ │ - │ └─→ ANNOTATE (merge results) │ - │ └─→ annotate_results/ │ - │ ├─→ gbk (final) │ - │ ├─→ proteins.fa │ - │ └─→ gff3 (final) │ - │ │ - └─→ [Final Output] - └─→ genome_annotation/ - ├─→ Species1/Strain1/ - ├─→ Species1/Strain2/ - └─→ ... - - -================================================================================ - CURRENT STATUS -================================================================================ - -✅ READY FOR PRODUCTION: - - modules/asm_stats.nf (will move to modules/AAFTF/) - - modules/annotation_tools.nf (will move to modules/annotate/) - - --skip_select_reps in earlgrey_mask.nf - -📋 AWAITING IMPLEMENTATION (in priority order): - 1. Phase 2.1: modules/rnaseq_fetch/ (sra_query, sra_fetch, prepare) - 2. Phase 2.2: modules/funannotate/ (train, predict, update) - 3. Phase 1: modules/AAFTF/ (FCS_GX, sourpurge, vecscreen) - 4. Phase 1: modules/repeatmasking/ (masking strategy selector) - 5. Phase 3: Move modules/annotation_tools.nf to modules/annotate/ - 6. Phase 4: modules/setup/ (databases) - -================================================================================ diff --git a/README.md b/README.md index d376e06..36690fb 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ The pipeline lives at the repo root (`funannotate.nf` + `nextflow.config`), so i runs directly from GitHub — no clone required: ```bash -nextflow run stajichlab/nf_funannotate1 -profile annotate,slurm,module -resume +nextflow run stajichlab/nf_funannotate1 -profile annotate,slurm,ucr_hpcc -resume ``` Nextflow caches the repo under `~/.nextflow/assets/`; add `-r ` to pin @@ -26,7 +26,7 @@ a revision and `-latest` to pull updates. Outputs and the `samples.csv` / nextflow run stajichlab/nf_funannotate1 -profile test -stub-run # real run on SLURM with environment modules (from your launch dir, with samples.csv) -nextflow run stajichlab/nf_funannotate1 -profile annotate,slurm,module -resume --n_test 1 +nextflow run stajichlab/nf_funannotate1 -profile annotate,slurm,ucr_hpcc -resume --n_test 1 # or, from a local checkout, use the sbatch launcher sbatch /path/to/nf_funannotate1/run_annotate.sh --n_test 1 @@ -43,7 +43,7 @@ Compose one option from each of three axes: `-profile ,, This is the **single source of truth** for the DSL2 modularization effort. It +> supersedes the earlier `IMPLEMENTATION_SUMMARY.md` and `MODULE_STRUCTURE.txt` +> (removed). Progress is tracked in GitHub issues — see +> `.github/REFACTOR_ISSUES.md` and `scripts/create_refactor_issues.sh`. + +## Where we actually are + +- `funannotate.nf`: **2,470-line monolith, 20 inline processes.** +- `earlgrey_mask.nf`: separate 394-line pipeline that **duplicates** the + samplesheet parse + taxonomy/asmid/suppress filtering from `funannotate.nf`. +- Extracted so far: `modules/asm_stats.nf`, `modules/annotation_tools.nf` (2 of 20). +- Strong base already present: nf-schema + `nextflow_schema.json`, orthogonal + profiles, manifest, CITATIONS/COC/CHANGELOG/LICENSE, stub-run CI. + +The goal is an **nf-core-*inspired*** layout (not nf-core submission): adopt the +parts that are pure engineering wins, skip the parts that fight our HPC reality. + +--- + +## Principle 0 — the data contract (do this FIRST; blocks everything else) + +Every process is currently wired with a fragile positional 10-tuple: + +```groovy +tuple(out, asmid, species, strain, locustag, busco, header_length, transl_table, gz, taxonid) +``` + +Adding an 11th field touches every process. **Replace it with a `meta` map**, the +standard DSL2 idiom. Genome travels as a separate `path`: + +```groovy +// Canonical channel element: tuple val(meta), path(genome) +meta = [ + id : out, // unique sample tag — used for tag{} and file naming + asmid : asmid, + species : species, + strain : strain, + locustag : locustag, + busco : busco, // BUSCO_LINEAGE + transl_table: transl_table, // default '1' + taxonid : taxonid, +] +``` + +Rules: +- `meta.id` is the **only** field used for naming/`tag`; everything else is payload. +- `header_length` (constant 24) becomes `params.header_length`, **not** a meta field. +- Build `meta` once, in the `INPUT_CHECK` subworkflow (below). No process + re-parses the samplesheet. +- A module's `input:` declares `tuple val(meta), val(genome)` (genome is an + absolute-path **string**, kept as `val` so the networked FS isn't re-staged) + and never positionally unpacks fields it doesn't use. + +Until `meta` is adopted, **do not extract more modules** — every module written +against the old tuple is rework. + +### Groundwork landed (this is the only safe sub-step) + +- `SampleUtils.makeMeta(row)` (`lib/SampleUtils.groovy`) is the **single + authoritative definition** of `meta`, reproducing the current `jobs`-channel + cleaning field-for-field so wiring it in is behaviour-preserving. Not yet called. +- `params.header_length` (default 24) added to `nextflow.config` + schema. Still + threaded through the tuple for now; the conversion removes it from the tuple. + +### Conversion recipe (the atomic change, not yet done) + +The rest of #3 is **atomic** — source channel, ~40 workflow channel ops, and all +10 carrying processes move together. Recipe: + +1. **Source channels** (`jobs` and `postpredict` maps): replace the per-field + `def`s with `def meta = SampleUtils.makeMeta(row)` and emit `tuple(meta, gz)` + (jobs) / `meta` (postpredict). `header_length` comes from `params`. +2. **Processes** (shim to keep script bodies intact): change `input:`/`output:` + tuples to `tuple val(meta), val(genome)` (+ reads paths where present), add + `tag "${meta.id}"`, and at the top of `script:` add the alias block + `def out = meta.id; def asmid = meta.asmid; def species = meta.species; …` + so every existing `${out}`/`${asmid}` interpolation still resolves. +3. **Workflow ops** — translate the position-coupled patterns: + - `.map { out, asmid, … -> tuple(out, asmid, …) }` re-threads → `.map { meta, genome -> … }` + - index access: `it[8].exists()` (genome) → `genome.exists()`; reads + `it[10]/it[12]` → name them in the destructure. + - slice sentinels: `row[0..8]` / `row[0..-3]` (drop combine/gate tails) → + destructure `(meta, genome, _sentinel)` explicitly. + - species-keyed `groupTuple`/`combine`/`join` for RNA-seq → key on + `meta.species` (compute `species_tag` from `meta.species`), carry `meta`. + - the reduced 8-tuple in the annotate phase collapses to a single `meta`. +4. **Validate**: `‑profile test ‑stub-run` **and** `‑profile test ‑stub-run + --run_sra_fetch true` (the default stub skips the RNA-seq subgraph). Stub + proves graph wiring only — a real-data HPCC run confirms semantics (grouping + picks the right representative, reads join to the right strain). + +--- + +## Target architecture + +``` +main.nf # thin entrypoint: parse args, call workflow +workflows/ + funannotate.nf # wires the subworkflows (was the monolith) + earlgrey.nf # curated-mask entry, reuses shared subworkflows +subworkflows/local/ + input_check.nf # samplesheet -> meta channel + taxon/asmid/suppress filters (SHARED) + setup_dbs.nf # SETUP_TAXONDB / FUNANNOTATE_DB / AUGUSTUS_CONFIG gating + prepare_genome.nf # clean -> asm_stats -> mask + mask.nf # selects ONE masker module by params.mask_tool + rnaseq.nf # sra_query -> sra_fetch(_se) -> rnaseq_prepare + predict.nf # train -> predict -> (update) + annotate.nf # antismash|signalp|interpro -> funannotate annotate +modules/local/ # ONE process per file + setup_taxondb.nf setup_funannotate_db.nf setup_augustus_config.nf + genome_clean.nf genome_clean_batch.nf asm_stats.nf + mask_tantan.nf mask_repeatmodeler.nf mask_repeatmasker.nf mask_earlgrey.nf + sra_query.nf sra_query_batch.nf collect_sra_query.nf write_empty_reads.nf + sra_fetch.nf sra_fetch_se.nf rnaseq_prepare.nf + funannotate_train.nf funannotate_predict.nf funannotate_update.nf + antismash.nf interproscan.nf signalp.nf funannotate_annotate.nf + select_reps.nf # earlgrey representative selection +conf/ + base.config # resources by label (process_low/medium/high/...) + modules.config # per-process publishDir + ext.args (nf-core idiom) +``` + +### Why this and not the old plan + +- **One tool = one process = one module file.** The old plan bundled 3 processes + per file (`sra_query.nf`, `annotation_tools.nf`, `databases.nf`). That is the + *opposite* of the convention and kills reuse. Group sequences with + **subworkflows**, which the old plan never mentioned. +- **Masking is a subworkflow, not a mega-process.** A single process with an + `if/else` over NONE/TANTAN/REPEATMODELER/REPEATMASKER/EARLGREY is an + anti-pattern. Each masker is its own module; the *selection* lives in + `subworkflows/local/mask.nf`. +- **Kill the funannotate/earlgrey duplication.** Both entrypoints parse the same + samplesheet and apply the same filters. Extract that into `input_check.nf` once + and call it from both. EarlGrey is a whole pipeline (SELECT_REPS + asm_stats + + representative-per-species), not a masking flavor — it reuses shared + subworkflows rather than being folded into one module. + +--- + +## Per-process extraction checklist (the stub-run gate) + +Apply to **one process per commit/PR**. The monolith must stay runnable at every +commit. + +For process `P`: + +- [ ] Create `modules/local/

.nf` with `process P { ... }`. +- [ ] `input:` uses `tuple val(meta), path(...)` (no positional field unpacking). +- [ ] Add `tag "${meta.id}"` and a resource `label` (`process_low|medium|high`). +- [ ] Keep the existing `stub:` block; keep `storeDir`/`publishDir` behavior. +- [ ] Emit a version: `path "versions.yml", emit: versions` + a `cat <<-END_VERSIONS` + block capturing the tool version. (Foundational — see Issue 3.) +- [ ] Move the process's resource/`withName` block out of + `conf/profile_annotate.config` into `conf/modules.config` (name unchanged, + so existing `withName:` selectors keep matching). +- [ ] In the workflow, replace the inline `process P {}` with + `include { P } from '../modules/local/p'` and adapt the call site to pass `meta`. +- [ ] **Gate (must pass before commit):** + ``` + nextflow config . -profile test + nextflow run . -profile test -stub-run + nextflow run . --help + ``` +- [ ] Commit. One process. Repeat. + +--- + +## Migration order (corrected) + +The old plan started with RNA-seq fetch ("least interdependent") — but +`SRA_FETCH` is the **single most complex** process (~270 lines). Prove the +pattern on a leaf first, then attack the hard pieces. + +| Phase | Work | Why here | +|------|------|----------| +| 0 | `meta` map contract + `INPUT_CHECK` subworkflow | Blocks all extraction; dedupes earlgrey | +| 1 | Skeleton: `main.nf`, `workflows/`, `subworkflows/local/`, `modules/local/`; move existing `asm_stats` + `annotation_tools` to convention | Establishes layout cheaply | +| 2 | `versions.yml` + `conf/base.config` + `conf/modules.config` | Pattern every later module copies | +| 3 | Setup modules (3 leaf processes) | Easiest real extraction; validates gate | +| 4 | Genome clean + `prepare_genome` subworkflow | Preserve FCS-GX `/dev/shm` staging | +| 5 | `mask` subworkflow + per-tool masker modules | Replaces the mega-process design | +| 6 | `rnaseq` subworkflow (the hard one) | Done *after* pattern is proven | +| 7 | `predict` subworkflow (train/predict/update) | Core | +| 8 | `annotate` subworkflow | Composition of optional tools | +| 9 | nf-core hygiene (docs/usage, docs/output, schema_input, naming, MultiQC) | Stretch | + +--- + +## Provisioning repivot (done) + +The `module` provisioning profile is renamed to **`ucr_hpcc`** — an +*institutional* profile in the nf-core/configs sense. The Lmod module names and +`/bigdata` paths only exist at UCR, so the name now says so. Portable runs use +`singularity` (containers) or `pixi`. New sites copy +`conf/provision_ucr_hpcc.config` to `conf/provision_.config` and register a +matching profile. ``` -┌─────────────────────────────────────────────────────────────────┐ -│ GENOME PREPROCESSING │ -├─────────────────────────────────────────────────────────────────┤ -│ CLEAN │ MASK │ SUMMARY_STATS │ -│ ───── │ ──── │ ───────────── │ -│ • FCS_GX │ • NONE │ • ASM_STATS │ -│ • sourpurge │ • TANTAN │ │ -│ • vecscreen │ • REPEATMODELER │ │ -│ │ • REPEATMASKER │ │ -│ │ • EARLGREY │ │ -└─────────────────────────────────────────────────────────────────┘ - ↓ -┌─────────────────────────────────────────────────────────────────┐ -│ GENE PREDICTION │ -├──────────────────────────────────┬──────────────────────────────┤ -│ RNA-seq Preparation │ Funannotate Pipeline │ -│ ───────────────────── │ ────────────────── │ -│ • SRA_QUERY │ • TRAIN │ -│ • SRA_FETCH (PE & SE) │ • PREDICT │ -│ • RNASEQ_PREPARE (Trinity) │ • UPDATE (optional) │ -└──────────────────────────────────┴──────────────────────────────┘ - ↓ -┌─────────────────────────────────────────────────────────────────┐ -│ ANNOTATION │ -├─────────────────────────────────────────────────────────────────┤ -│ • ANTISMASH (secondary metabolites) │ -│ • SIGNALP (signal peptides) │ -│ • INTERPROSCAN (protein domains - when implemented) │ -│ • FUNANNOTATE_ANNOTATE (final annotation) │ -└─────────────────────────────────────────────────────────────────┘ +-profile annotate,slurm,ucr_hpcc # institutional (default on UCR HPCC) +-profile annotate,local,singularity # portable ``` -## Completed ✅ -✅ **ASM_STATS Module** (`modules/asm_stats.nf`) -- Extracted assembly statistics generation into a separate reusable module -- Used by both `funannotate.nf` and `earlgrey_mask.nf` -- Generates `asm_stats.tsv.gz` with ASMID, total_length_bp, N50_bp, contig_count - -✅ **Optional SELECT_REPS** (`earlgrey_mask.nf`) -- Added `--skip_select_reps` flag to skip representative selection -- When enabled, all genomes are processed for EarlGrey masking without size filtering -- Added `--gen_asm_stats` flag (default: true) to auto-generate assembly statistics - -✅ **Annotation Tools Module** (`modules/annotation_tools.nf`) -- ANTISMASH_RUN (secondary metabolite detection) -- INTERPROSCAN_RUN (protein domain annotation) -- SIGNALP_RUN (signal peptide prediction) - -## Proposed Modularization Structure (User-Approved) - -### Phase 1: Genome Preprocessing Modules - -#### `modules/AAFTF/asm_stats.nf` (Refactor existing) -**Summary Statistics Generation** -- ASM_STATS: Compute assembly stats (total_length_bp, N50_bp, contig_count) -- *Note: Move existing `modules/asm_stats.nf` here* - -#### `modules/AAFTF/FCS_GX.nf` (Extract from GENOME_CLEAN) -**Contamination Screening & Removal** -- FCS_GX contamination detection and removal -- Phylum-aware filtering using NCBI taxonomy - -#### `modules/AAFTF/sourpurge.nf` (Extract from GENOME_CLEAN) -**Sourpurge Contamination Detection** -- Source organism contamination screening - -#### `modules/AAFTF/vecscreen.nf` (New) -**Vector Contamination Screening** -- NCBI VecScreen vector contamination detection - -#### `modules/repeatmasking/masking.nf` -**Repeat Masking Strategy Selection** -- TANTAN: Soft masking (tantan algorithm) -- REPEATMODELER: De novo TE discovery -- REPEATMASKER: Library-based masking (existing library or species) -- EARLGREY: De novo TE discovery + masking (currently in separate earlgrey_mask.nf) -- NONE: Skip repeat masking entirely - -### Phase 2: Gene Prediction Modules - -#### `modules/rnaseq_fetch/sra_query.nf` -**SRA Discovery & Query** -- SRA_QUERY: Query NCBI SRA for RNA-seq accessions per species -- SRA_QUERY_BATCH: Batched SRA queries to NCBI -- COLLECT_SRA_QUERY: Merge per-species results into manifest - -#### `modules/rnaseq_fetch/sra_fetch.nf` -**RNA-seq Download & Normalization** -- SRA_FETCH: Download paired-end RNA-seq, normalize reads -- SRA_FETCH_SE: Download single-end RNA-seq, normalize -- WRITE_EMPTY_READS: Create placeholders for species with no SRA data - -#### `modules/rnaseq_fetch/prepare.nf` -**RNA-seq Assembly & Preparation** -- RNASEQ_PREPARE: Trinity assembly and normalization per species -- Output shared Trinity-GG for all strains of a species - -#### `modules/funannotate/train.nf` -**Gene Model Training** -- FUNANNOTATE_TRAIN: PASA-based training on representative assembly -- Full training (Trinity + HISAT2 + trimmomatic) for representatives -- PASA-only for non-representative strains - -#### `modules/funannotate/predict.nf` -**Gene Prediction** -- FUNANNOTATE_PREDICT: Ab initio and evidence-based gene prediction -- Pre-flight validation (assembly size/fragmentation checks) -- Post-prediction filtering and formatting - -#### `modules/funannotate/update.nf` (Optional) -**Prediction Update with RNA-seq** -- FUNANNOTATE_UPDATE: Update predictions with mapped RNA-seq reads -- Optional step for models with available transcriptomics data - -### Phase 3: Annotation Modules - -#### `modules/annotate/annotation_tools.nf` (Refactor existing) -**Post-prediction Annotation Tools** -- ANTISMASH_RUN: Secondary metabolite cluster detection -- SIGNALP_RUN: Signal peptide prediction -- INTERPROSCAN_RUN: Protein domain annotation (when implemented) - -#### `modules/annotate/funannotate.nf` -**Final Funannotate Annotation** -- FUNANNOTATE_ANNOTATE: Functional annotation merging - -### Phase 4: Setup & Utilities - -#### `modules/setup/databases.nf` -**Database Initialization** -- SETUP_TAXONDB: NCBI taxonomy database (for FCS-GX) -- SETUP_FUNANNOTATE_DB: Funannotate databases (BUSCO, etc.) -- SETUP_AUGUSTUS_CONFIG: Writable Augustus configuration - -## Benefits of Modularization - -1. **Reusability**: Modules can be composed into different pipelines -2. **Flexibility**: Easy to swap masking strategies or annotation tools -3. **Maintainability**: Smaller, focused files are easier to understand and modify -4. **Testing**: Individual modules can be tested in isolation -5. **Documentation**: Each module documents its inputs, outputs, and dependencies -6. **Scalability**: Easier to add new tools (e.g., new masking strategies) -7. **Git History**: Smaller commits with clear intent - -## Implementation Strategy - -### Priority Order -1. **Phase 2.1 (RNA-seq Fetch)**: Least interdependent, high reusability -2. **Phase 2.2 (Funannotate Modules)**: Core prediction pipeline -3. **Phase 1 (Genome Preprocessing)**: More complex due to conditional branching -4. **Phase 3 (Annotation)**: Depends on Phase 2 completion -5. **Phase 4 (Setup)**: Last, as foundational - -### Implementation Notes -- Each module should be standalone with clear input/output contracts -- Use `include` statements in main workflow files -- Maintain backward compatibility with existing wrapper scripts -- Update nextflow.config to support module-specific params -- Create detailed header comments in each module file -- Use consistent naming conventions: `modules/{category}/{function}.nf` - -### Testing & Validation -- Test each module in isolation with stub runs: `-stub-run` -- Verify module reuse works across different pipelines -- Document module interdependencies -- Create example usage in comments +Issue 11 covers the fuller consolidation (folding UCR SLURM partitions / +`clusterOptions` into the same institutional profile). + +--- + +## Distance from nf-core + +| Area | State | +|------|-------| +| Scaffolding (schema, CITATIONS, COC, CHANGELOG, LICENSE, CI) | ~30–40% there | +| `meta` map | none (Phase 0) | +| Structure (`main.nf`/`workflows`/`subworkflows`/`modules`) | monolith | +| `versions.yml` per module + MultiQC | none (mandatory for nf-core) | +| Containers per module | **biggest gap** — relies on Lmod/pixi; nf-core needs conda+biocontainer per process | +| Naming | `nf_funannotate1` violates nf-core naming (underscores/digits) | +| nf-test, `docs/usage.md`+`output.md`, `assets/schema_input.json`, `.nf-core.yml` | missing | + +**Verdict:** ~30–40% on peripheral scaffolding, ~0% on the two load-bearing items +(meta-maps + container-per-module). Full `nf-core lint` compliance is a multi-week +rewrite, much of which fights our HPC reality. **Recommendation: nf-core-inspired, +not nf-core-submitted** — adopt meta-maps, one-tool-per-module, subworkflows, +`versions.yml`, `conf/modules.config`; keep `ucr_hpcc` as an institutional profile +but add a real container path so the pipeline is portable. diff --git a/conf/profile_annotate.config b/conf/profile_annotate.config index daf708c..a8ed12b 100644 --- a/conf/profile_annotate.config +++ b/conf/profile_annotate.config @@ -2,8 +2,8 @@ * annotate pipeline profile: funannotate.nf * * Provides the pipeline params and per-process SLURM resources. Combine with an - * executor profile (slurm|local) and a provisioning profile (module|pixi| - * singularity), e.g. -profile annotate,slurm,module + * executor profile (slurm|local) and a provisioning profile (ucr_hpcc|pixi| + * singularity), e.g. -profile annotate,slurm,ucr_hpcc * * Defaults are fungal (busco fungi via BUSCO_LINEAGE column, antismash_taxon=fungi, * swissprot_fungi proteins) but every organism-specific value is a param, so other @@ -29,6 +29,10 @@ params { // ── assembly statistics (for earlgrey_mask.nf SELECT_REPS) ──────────────── tables_dir = "${launchDir}/tables" gen_asm_stats = true // generate asm_stats.tsv.gz from clean genomes + // Directory ASM_STATS scans for cleaned genomes — the storeDir target that + // GENOME_CLEAN writes .fa into. Must match earlgrey_mask.nf's + // params.genome_dir so both pipelines read the same cleaned assemblies. + genome_dir = "${launchDir}/input_clean_genomes" // Writable AUGUSTUS_CONFIG copy, seeded once by SETUP_AUGUSTUS_CONFIG (storeDir-cached). // augustus writes new species parameter sets here during training, so it must be a // private writable copy of the install's read-only config. diff --git a/conf/provision_module.config b/conf/provision_ucr_hpcc.config similarity index 86% rename from conf/provision_module.config rename to conf/provision_ucr_hpcc.config index c6759e2..afc622d 100644 --- a/conf/provision_module.config +++ b/conf/provision_ucr_hpcc.config @@ -1,5 +1,12 @@ /* - * Provisioning: Lmod modules (DEFAULT, proven on UCR HPCC). + * Provisioning: UCR HPCC Lmod modules — INSTITUTIONAL profile (-profile ucr_hpcc). + * + * This is a site-specific profile: the module names (funannotate/dev-1.8.18, + * AAFTF, taxonkit, ...) and the conda bootstrap below exist on the UCR HPCC and + * will NOT resolve elsewhere. For a portable run use a container/env provisioning + * profile instead (-profile <...>,singularity or ,pixi). New sites should copy + * this file to conf/provision_.config and register a matching profile in + * nextflow.config — the nf-core/configs institutional-profile model. * * Each process label gets a `beforeScript` that loads the tools it needs. The * beforeScript and the process `script:` are concatenated into one job script, diff --git a/funannotate.nf b/funannotate.nf index 6fdc3b2..f2a1884 100644 --- a/funannotate.nf +++ b/funannotate.nf @@ -10,33 +10,28 @@ * params.taxondb / params.funannotate_db are null and parsing fails): * sbatch nextflow/run_annotate.sh * nextflow run nextflow/funannotate.nf -c nextflow/nextflow.config \ - * -profile annotate,slurm,module -resume + * -profile annotate,slurm,ucr_hpcc -resume */ -// Metadata tuple order used throughout: -// val(out), val(asmid), val(species), val(strain), val(locustag), -// val(busco_lineage), val(header_length), val(transl_table) -// GENOME_CLEAN receives: ..., path(genome_gz), val(taxonid), val(taxondb) -// → emits: ..., path(genome_fa), val(taxonid) [storeDir moves .fa; workflow maps to abs string] -// → writes .fa to input_clean_genomes/ (storeDir; skip check targets this file) -// → purge/FCS intermediates written as side effects to input_clean_genomes/clean/ -// MASKREPEAT_TANTAN_RUN receives: ..., val(genome_fa), val(taxonid) -// → emits: ..., path(masked_fa), val(taxonid) [storeDir caches input_clean_genomes/.masked.fasta] -// [skipped unless --run_repeatmasker; masked_fa falls back to unmasked .fa if .masked.fasta absent] -// SRA_FETCH receives: val(species_tag), val(taxonid) [only when --run_sra_fetch; one per species] -// → emits: val(species_tag), path(norm_R1.fastq.gz), path(norm_R2.fastq.gz) -// → storeDir caches normalized reads at rnaseq_reads/_norm_{R1,R2}.fastq.gz -// → empty files (0 bytes) written when no RNA-seq found; downstream checks size to skip -// → SRA_FETCH handles: download → fastp trim → bbnorm normalization internally -// --stop_after_sra_fetch: when true, pipeline halts after SRA_FETCH (skips RNASEQ_PREPARE, -// FUNANNOTATE_TRAIN, FUNANNOTATE_PREDICT and all downstream steps). -// RNASEQ_PREPARE receives: ..., val(genome_fa), path(norm_r1), path(norm_r2) [representative only] +// Data contract: every channel element is `tuple val(meta), val/path(genome)`. +// meta is a Map built by SampleUtils.makeMeta(row) — see lib/SampleUtils.groovy. +// meta.id is the ONLY field used for tag{} and file naming. +// meta.asmid, meta.species, meta.strain, meta.locustag, meta.busco, +// meta.transl_table, meta.taxonid carry payload used inside process scripts. +// header_length is NOT in meta — it comes from params.header_length (default 24). +// +// GENOME_CLEAN receives: tuple val(meta), path(genome_gz), val(taxondb) +// → emits: tuple val(meta), path(genome_fa) [storeDir writes input_clean_genomes/.fa.gz] +// MASKREPEAT_TANTAN_RUN receives: tuple val(meta), val(genome_fa) +// → emits: tuple val(meta), path(masked_fa) [storeDir caches input_clean_genomes/.masked.fasta.gz] +// SRA_FETCH receives: val(species_tag), val(taxonid) [only when --run_sra_fetch] +// → emits: val(species_tag), path(norm_R1.fastq.gz), path(norm_R2.fastq.gz), path(se) +// RNASEQ_PREPARE receives: tuple val(species_tag), val(meta), val(genome_fa), path(r1), path(r2), path(se) // → emits: val(species_tag), path(trinity-GG.fasta) [storeDir caches in rnaseq_data/] -// → normalized reads stay in rnaseq_reads/ and are NOT re-emitted from RNASEQ_PREPARE -// FUNANNOTATE_TRAIN receives: ..., val(genome_fa), path(norm_r1), path(norm_r2), path(trinity_fa) -// → norm reads come directly from SRA_FETCH; trinity_fa from RNASEQ_PREPARE -// → emits: ..., val(genome_fa) -// FUNANNOTATE_PREDICT receives: ..., val(genome_fa) [from TRAIN or directly after masking/clean] +// FUNANNOTATE_TRAIN receives: tuple val(meta), val(genome_fa), path(r1), path(r2), path(se), path(trinity_fa) +// → emits: tuple val(meta), val(genome_fa) +// FUNANNOTATE_PREDICT receives: tuple val(meta), val(genome_fa) +// → emits metadata: tuple val(meta) // Download and extract NCBI taxdump once; storeDir caches it at params.taxondb so // subsequent runs skip this entirely. @@ -173,7 +168,7 @@ process SETUP_AUGUSTUS_CONFIG { process GENOME_CLEAN { label 'genome_clean' - tag "$asmid" + tag "${meta.id}" // container '/rhome/jstajich/projects/AAFTF/AAFTF_v0.6.1-signed.sif' @@ -185,16 +180,15 @@ process GENOME_CLEAN { time '6h' input: - tuple val(out), val(asmid), val(species), val(strain), val(locustag), - val(busco_lineage), val(header_length), val(transl_table), - path(genome_gz), val(taxonid), val(taxondb) + tuple val(meta), path(genome_gz), val(taxondb) output: - tuple val(out), val(asmid), val(species), val(strain), val(locustag), - val(busco_lineage), val(header_length), val(transl_table), - path("${asmid}.fa.gz"), val(taxonid), emit: genome + tuple val(meta), path("${meta.asmid}.fa.gz"), emit: genome script: + def out = meta.id + def asmid = meta.asmid + def taxonid = meta.taxonid """ if [ ! -f "${genome_gz}" ]; then echo "ERROR: genome_gz not found at path: ${genome_gz}" >&2 @@ -250,6 +244,7 @@ process GENOME_CLEAN { """ stub: + def asmid = meta.asmid """ echo ">stub_${asmid}" | pigz -c > ${asmid}.fa.gz mkdir -p ${launchDir}/input_clean_genomes/clean @@ -287,7 +282,7 @@ process GENOME_CLEAN_BATCH { path "clean_batch_*.manifest.tsv", emit: manifest script: - def batch_tsv = items.collect { row -> "${row[1]}\t${row[8]}\t${row[9]}" }.join('\n') + def batch_tsv = items.collect { row -> "${row[0].asmid}\t${row[1]}\t${row[0].taxonid}" }.join('\n') """ set -uo pipefail source /etc/profile.d/modules.sh 2>/dev/null || true @@ -374,7 +369,7 @@ BATCH_EOF """ stub: - def batch_tsv = items.collect { row -> "${row[1]}\t${row[8]}\t${row[9]}" }.join('\n') + def batch_tsv = items.collect { row -> "${row[0].asmid}\t${row[1]}\t${row[0].taxonid}" }.join('\n') """ DEST=${launchDir}/input_clean_genomes mkdir -p \$DEST/clean @@ -396,7 +391,7 @@ BATCH_EOF // storeDir caches the masked FASTA alongside the clean genome. process MASKREPEAT_TANTAN_RUN { label 'funannotate' - tag "$asmid" + tag "${meta.id}" storeDir "${launchDir}/input_clean_genomes" @@ -405,16 +400,13 @@ process MASKREPEAT_TANTAN_RUN { time '2h' input: - tuple val(out), val(asmid), val(species), val(strain), val(locustag), - val(busco_lineage), val(header_length), val(transl_table), - val(genome_fa), val(taxonid) + tuple val(meta), val(genome_fa) output: - tuple val(out), val(asmid), val(species), val(strain), val(locustag), - val(busco_lineage), val(header_length), val(transl_table), - path("${asmid}.masked.fasta.gz"), val(taxonid), emit: masked + tuple val(meta), path("${meta.asmid}.masked.fasta.gz"), emit: masked script: + def asmid = meta.asmid """ source /etc/profile.d/modules.sh 2>/dev/null || true # Inflate a gzipped clean genome to a local uncompressed copy; funannotate cannot @@ -430,6 +422,7 @@ process MASKREPEAT_TANTAN_RUN { """ stub: + def asmid = meta.asmid """ echo ">stub_${asmid}_masked" | pigz -c > ${asmid}.masked.fasta.gz """ @@ -1117,15 +1110,17 @@ process RNASEQ_PREPARE { time '120h' input: - tuple val(species_tag), val(out), val(asmid), val(species), val(strain), val(locustag), - val(busco_lineage), val(header_length), val(transl_table), - val(genome_fa), path(r1), path(r2), path(se) + tuple val(species_tag), val(meta), val(genome_fa), path(r1), path(r2), path(se) output: tuple val(species_tag), path("${species_tag}.trinity-GG.fasta"), emit: shared script: + def out = meta.id + def species = meta.species + def strain = meta.strain + def header_length = params.header_length """ # ── Empty-reads sentinel: no RNA-seq found by SRA_FETCH / SRA_FETCH_SE ── if [ ! -s "${r1}" ] && [ ! -s "${se}" ]; then @@ -1204,6 +1199,7 @@ process RNASEQ_PREPARE { """ stub: + def out = meta.id """ echo ">stub_trinity_${species_tag}" > ${species_tag}.trinity-GG.fasta mkdir -p ${params.training_target}/${out}/training @@ -1217,23 +1213,25 @@ process RNASEQ_PREPARE { // a single strain or when run_sra_fetch is false). process FUNANNOTATE_TRAIN { label 'funannotate' - tag "$out" + tag "${meta.id}" cpus 16 memory '96 GB' time '120h' input: - tuple val(out), val(asmid), val(species), val(strain), val(locustag), - val(busco_lineage), val(header_length), val(transl_table), - val(genome_fa), path(r1), path(r2), path(se), path(trinity_fa) + tuple val(meta), val(genome_fa), path(r1), path(r2), path(se), path(trinity_fa) output: - tuple val(out), val(asmid), val(species), val(strain), val(locustag), - val(busco_lineage), val(header_length), val(transl_table), - val(genome_fa) + tuple val(meta), val(genome_fa) script: + def out = meta.id + def asmid = meta.asmid + def species = meta.species + def strain = meta.strain + def locustag = meta.locustag + def header_length = params.header_length def pasa_db_arg = "--pasa_db sqlite" """ # ── Skip if no RNA-seq data at all ──────────────────────────────────────── @@ -1380,6 +1378,7 @@ process FUNANNOTATE_TRAIN { """ stub: + def out = meta.id """ echo "[STUB] FUNANNOTATE_TRAIN stub for ${out}" mkdir -p ${params.training_target}/${out}/training @@ -1398,23 +1397,28 @@ process FUNANNOTATE_TRAIN { // on-disk GBK), so emitting a marker keeps the DAG edge without copying the result tree. process FUNANNOTATE_PREDICT { label 'funannotate' - tag "$out" + tag "${meta.id}" cpus 16 memory '32 GB' time '32h' input: - tuple val(out), val(asmid), val(species), val(strain), val(locustag), - val(busco_lineage), val(header_length), val(transl_table), - val(genome_fa) + tuple val(meta), val(genome_fa) output: - tuple val(out), val(asmid), val(species), val(strain), val(locustag), - val(busco_lineage), val(header_length), val(transl_table), emit: metadata - path("${out}.predict.done"), emit: done + val meta, emit: metadata + path("${meta.id}.predict.done"), emit: done script: + def out = meta.id + def asmid = meta.asmid + def species = meta.species + def strain = meta.strain + def locustag = meta.locustag + def busco_lineage = meta.busco + def header_length = params.header_length + def transl_table = meta.transl_table """ source /etc/profile.d/modules.sh 2>/dev/null || true module load funannotate/dev-1.8.18 @@ -1554,6 +1558,7 @@ process FUNANNOTATE_PREDICT { """ stub: + def out = meta.id """ echo "[STUB] Would run funannotate predict for ${out} using ${genome_fa}" [ -f "${genome_fa}" ] || [ -f "${genome_fa}.gz" ] || { echo "ERROR: genome not found at ${genome_fa}[.gz]" >&2; exit 1; } @@ -1567,7 +1572,7 @@ process FUNANNOTATE_PREDICT { process ANTISMASH_RUN { label 'antismash' - tag "$out" + tag "${meta.id}" cpus 8 memory '16 GB' @@ -1576,13 +1581,13 @@ process ANTISMASH_RUN { publishDir "${params.target}", mode: 'copy', overwrite: true input: - tuple val(out), val(asmid), val(species), val(strain), val(locustag), - val(busco_lineage), val(header_length), val(transl_table) + val(meta) output: - tuple val(out), path("${out}/antismash_local/**") + tuple val(meta), path("${meta.id}/antismash_local/**") script: + def out = meta.id def gbk = "${params.target}/${out}/predict_results/${out}.gbk" """ # Accept a compressed prediction (.gbk.gz); antismash needs it uncompressed, so @@ -1608,6 +1613,7 @@ process ANTISMASH_RUN { """ stub: + def out = meta.id """ mkdir -p ${out}/antismash_local touch ${out}/antismash_local/${out}.json.gz @@ -1618,7 +1624,7 @@ process ANTISMASH_RUN { // IPRSCAN5 process INTERPROSCAN_RUN { label 'interproscan' - tag "$out" + tag "${meta.id}" cpus 8 memory '32 GB' @@ -1627,13 +1633,13 @@ process INTERPROSCAN_RUN { publishDir "${params.target}", mode: 'copy', overwrite: true input: - tuple val(out), val(asmid), val(species), val(strain), val(locustag), - val(busco_lineage), val(header_length), val(transl_table) + val(meta) output: - tuple val(out), path("${out}/annotate_misc/iprscan.xml") + tuple val(meta), path("${meta.id}/annotate_misc/iprscan.xml") script: + def out = meta.id def proteins = "${params.target}/${out}/predict_results/${out}.proteins.fa" """ if [ ! -f "${proteins}" ]; then @@ -1646,6 +1652,7 @@ process INTERPROSCAN_RUN { """ stub: + def out = meta.id """ mkdir -p ${out}/annotate_misc touch ${out}/annotate_misc/iprscan.xml @@ -1654,7 +1661,7 @@ process INTERPROSCAN_RUN { process SIGNALP_RUN { label 'signalp' - tag "$out" + tag "${meta.id}" cpus 8 memory '16 GB' @@ -1663,13 +1670,13 @@ process SIGNALP_RUN { publishDir "${params.target}", mode: 'copy', overwrite: true input: - tuple val(out), val(asmid), val(species), val(strain), val(locustag), - val(busco_lineage), val(header_length), val(transl_table) + val(meta) output: - tuple val(out), path("${out}/annotate_misc/signalp.results.txt") + tuple val(meta), path("${meta.id}/annotate_misc/signalp.results.txt") script: + def out = meta.id def proteins = "${params.target}/${out}/predict_results/${out}.proteins.fa" """ if [ ! -f "${proteins}" ]; then @@ -1687,6 +1694,7 @@ process SIGNALP_RUN { """ stub: + def out = meta.id """ mkdir -p ${out}/annotate_misc touch ${out}/annotate_misc/signalp.results.txt @@ -1695,21 +1703,26 @@ process SIGNALP_RUN { process FUNANNOTATE_ANNOTATE { label 'funannotate' - tag "$out" + tag "${meta.id}" cpus 16 memory '32 GB' time '48h' input: - tuple val(out), val(asmid), val(species), val(strain), val(locustag), - val(busco_lineage), val(header_length), val(transl_table) + val(meta) output: - tuple val(out), path("${out}.annotate.done"), emit: marker + tuple val(meta), path("${meta.id}.annotate.done"), emit: marker script: - def antiSm = file("${params.target}/${out}/antismash_local/${out}.gbk") + def out = meta.id + def species = meta.species + def strain = meta.strain + def locustag = meta.locustag + def busco_lineage = meta.busco + def header_length = params.header_length + def antiSm = file("${params.target}/${meta.id}/antismash_local/${meta.id}.gbk") def antiSmArg = antiSm.exists() ? "--antismash ${antiSm}" : "" """ source /etc/profile.d/modules.sh 2>/dev/null || true @@ -1735,6 +1748,7 @@ process FUNANNOTATE_ANNOTATE { """ stub: + def out = meta.id """ echo "[STUB] Would run funannotate annotate for ${out}" mkdir -p ${params.target}/${out}/annotate_results ${params.target}/${out}/annotate_misc @@ -1745,22 +1759,26 @@ process FUNANNOTATE_ANNOTATE { process FUNANNOTATE_UPDATE { label 'funannotate' - tag "$out" + tag "${meta.id}" cpus 16 memory '96 GB' time '48h' input: - tuple val(out), val(asmid), val(species), val(strain), val(locustag), - val(busco_lineage), val(header_length), val(transl_table), - path(r1), path(r2) + tuple val(meta), path(r1), path(r2) output: - tuple val(out), val(asmid), val(species), val(strain), val(locustag), - val(busco_lineage), val(header_length), val(transl_table) + val meta script: + def out = meta.id + def asmid = meta.asmid + def species = meta.species + def strain = meta.strain + def locustag = meta.locustag + def busco_lineage = meta.busco + def header_length = params.header_length def pasa_db_arg = "--pasa_db sqlite" """ # ── Skip if no reads (empty marker file from SRA_FETCH) ────────────────── @@ -1828,6 +1846,7 @@ process FUNANNOTATE_UPDATE { """ stub: + def out = meta.id """ echo "[STUB] FUNANNOTATE_UPDATE stub for ${out} (r1=${r1}, r2=${r2})" mkdir -p ${params.target}/${out}/update_results @@ -1938,46 +1957,35 @@ workflow { .filter(taxonFilter) .filter(asmidFilter) .map { row -> - def species = (row.SPECIES?.trim() ?: '').replaceAll(/['"]/, '') - def strain = (row.STRAIN?.trim() ?: '').replaceAll(/['"]/, '').replaceAll(/;.*$/, '').trim().replace(':', ' ') - def out = SampleUtils.makeSampleTag(row.SPECIES?.trim() ?: '', row.STRAIN?.trim() ?: '') - def asmid = row.ASMID?.trim() - def locustag = row.LOCUSTAG?.replaceAll(/[\r\n]/, '')?.trim() - def busco = row.BUSCO_LINEAGE?.trim() - def header_length = 24 - def transl_table = row.TRANSL_TABLE?.trim() ?: '1' - def taxonid = row.NCBI_TAXONID?.trim() - // Dual input model: a non-empty GENOME column points directly at a local - // assembly FASTA (.fa/.fna[.gz]); otherwise resolve from the NCBI_ASM - // source dir by ASMID. Relative GENOME paths resolve against launchDir. - def genome_col = row.GENOME?.trim() + def meta = SampleUtils.makeMeta(row) + def genome_col = row.GENOME?.trim() def gz = genome_col ? (genome_col.startsWith('/') ? file(genome_col) : file("${launchDir}/${genome_col}")) - : file("${params.source}/${asmid}/${asmid}_genomic.fna.gz") - tuple(out, asmid, species, strain, locustag, busco, header_length, transl_table, gz, taxonid) + : file("${params.source}/${meta.asmid}/${meta.asmid}_genomic.fna.gz") + tuple(meta, gz) } - .filter { out, asmid, _sp, _st, _lt, _bl, _hl, _tt, _gz, _tid -> out && asmid } + .filter { meta, gz -> meta.id && meta.asmid } .take((params.n_test as int) > 0 ? params.n_test as int : -1) - .filter { out, asmid, _sp, _st, _lt, _bl, _hl, _tt, _gz, _tid -> - if (suppressSet.contains(asmid)) { - log.info "Suppressing ${out} (asmid=${asmid})" + .filter { meta, gz -> + if (suppressSet.contains(meta.asmid)) { + log.info "Suppressing ${meta.id} (asmid=${meta.asmid})" return false } return true } - .filter { out, asmid, _sp, _st, _lt, _bl, _hl, _tt, gz, _tid -> + .filter { meta, gz -> if (!gz.exists()) { - log.warn "Missing genome for ${out} (asmid=${asmid}): ${gz}" + log.warn "Missing genome for ${meta.id} (asmid=${meta.asmid}): ${gz}" return false } if (params.debug.toBoolean()) { - log.info "Queuing ${out}: genome=${gz} (${gz.size()} bytes)" + log.info "Queuing ${meta.id}: genome=${gz} (${gz.size()} bytes)" } return true } if (params.debug.toBoolean()) { - jobs.view { t -> "[CHANNEL] Submitting: out=${t[0]}, asmid=${t[1]}, transl_table=${t[7]}, gz=${t[8]}" } + jobs.view { meta, gz -> "[CHANNEL] Submitting: out=${meta.id}, asmid=${meta.asmid}, transl_table=${meta.transl_table}, gz=${gz}" } } // Ensure taxondb is populated before any GENOME_CLEAN task starts. @@ -1989,8 +1997,8 @@ workflow { // from being padded with finished genomes — a batch that is entirely cleaned is never // scheduled, so it never pays the ~30-min /dev/shm staging cost. (GENOME_CLEAN_BATCH // also re-checks per genome at runtime, which handles partial completion on retry.) - def jobs_to_clean = jobs.filter { tup -> - !genomeFile("${launchDir}/input_clean_genomes/${tup[1]}.fa").exists() + def jobs_to_clean = jobs.filter { meta, gz -> + !genomeFile("${launchDir}/input_clean_genomes/${meta.asmid}.fa").exists() } // Genome cleaning. The FCS-GX DB staging into /dev/shm costs ~30 min per task, so by @@ -2012,7 +2020,7 @@ workflow { clean_done_ch = GENOME_CLEAN_BATCH.out.manifest.collect().ifEmpty([]) } else { GENOME_CLEAN(jobs_to_clean.combine(taxondb_ch)) - clean_done_ch = GENOME_CLEAN.out.genome.map { it[8] }.collect().ifEmpty([]) + clean_done_ch = GENOME_CLEAN.out.genome.map { meta, gz -> gz }.collect().ifEmpty([]) } if (!params.only_clean.toBoolean()) { @@ -2022,32 +2030,25 @@ workflow { // clean_done_ch (combine waits until all cleaning is done). genome_fa is emitted as // an absolute-path string so downstream val(genome_fa) processes reference the file // directly without Nextflow re-staging it. + // Note: combine on jobs (a [meta,gz] tuple) appends clean_done_ch as a third element; + // we drop gz and the sentinel together. Keeping the tuple avoids emitting a bare Map + // (which Nextflow's combine may flatten unexpectedly). def clean_genome_ch = jobs - .map { out, asmid, species, strain, locustag, busco, hlen, ttable, _gz, taxonid -> - tuple(out, asmid, species, strain, locustag, busco, hlen, ttable, taxonid) - } - .combine(clean_done_ch) // gate: blocks until all cleaning is done - .map { row -> row[0..8] } // drop the clean_done sentinel element - // Resolve the cleaned genome AFTER the gate so the just-written .fa.gz - // (or legacy .fa) is visible — genomeFile prefers the compressed form. Resolving - // before the combine would freeze the path at construction time (pre-clean), when - // neither file exists yet, and the .exists() filter below would drop every genome. - .map { out, asmid, species, strain, locustag, busco, hlen, ttable, taxonid -> - def g = genomeFile("${launchDir}/input_clean_genomes/${asmid}.fa") - tuple(out, asmid, species, strain, locustag, busco, hlen, ttable, g, taxonid) - } - .filter { tup -> - if (!tup[8].exists()) { - log.warn "No cleaned genome for ${tup[0]} (asmid=${tup[1]}) — skipping downstream" + .combine(clean_done_ch) // gate: blocks until cleaning done + // Use it[0] (meta) rather than destructuring: combine with ifEmpty([]) produces + // a variable-length tuple — 2 elements when the sentinel is empty, 3+ when it + // carries collected paths. A fixed-arity { meta, _gz, _sentinel -> } fails on + // the 2-element case, so we index into the element directly. + .map { tuple(it[0], genomeFile("${launchDir}/input_clean_genomes/${it[0].asmid}.fa")) } + .filter { meta, g -> + if (!g.exists()) { + log.warn "No cleaned genome for ${meta.id} (asmid=${meta.asmid}) — skipping downstream" return false } return true } - // genome_fa as an absolute-path string so downstream val(genome_fa) processes - // reference the file directly without Nextflow re-staging it. - .map { out, asmid, species, strain, locustag, busco, hlen, ttable, genome_fa, taxonid -> - tuple(out, asmid, species, strain, locustag, busco, hlen, ttable, - genome_fa.toAbsolutePath().toString(), taxonid) + .map { meta, genome_fa -> + tuple(meta, genome_fa.toAbsolutePath().toString()) } // ── Generate assembly statistics (for earlgrey_mask.nf SELECT_REPS) ──────── @@ -2075,20 +2076,19 @@ workflow { if (params.run_repeatmasker.toBoolean()) { MASKREPEAT_TANTAN_RUN(clean_genome_ch) predict_genome_ch = MASKREPEAT_TANTAN_RUN.out.masked - .map { out, asmid, species, strain, locustag, busco, hlen, ttable, masked_fa, taxonid -> - tuple(out, asmid, species, strain, locustag, busco, hlen, ttable, - masked_fa.toAbsolutePath().toString(), taxonid) + .map { meta, masked_fa -> + tuple(meta, masked_fa.toAbsolutePath().toString()) } } else { // --run_repeatmasker false: use masked genome if a prior run produced it, else unmasked. predict_genome_ch = clean_genome_ch - .map { out, asmid, species, strain, locustag, busco, hlen, ttable, genome_fa, taxonid -> - def masked = genomeFile("${launchDir}/input_clean_genomes/${asmid}.masked.fasta") + .map { meta, genome_fa -> + def masked = genomeFile("${launchDir}/input_clean_genomes/${meta.asmid}.masked.fasta") def use_fa = masked.exists() ? masked.toString() : genome_fa if (params.debug.toBoolean()) { - log.info "[DEBUG] ${asmid}: genome_fa=${use_fa} (masked=${masked.exists()})" + log.info "[DEBUG] ${meta.asmid}: genome_fa=${use_fa} (masked=${masked.exists()})" } - tuple(out, asmid, species, strain, locustag, busco, hlen, ttable, use_fa, taxonid) + tuple(meta, use_fa) } } @@ -2115,9 +2115,9 @@ workflow { if (params.run_sra_fetch.toBoolean()) { // Build per-species input: group assemblies, keep first taxonid per species. def sra_input = predict_genome_ch - .map { out, asmid, species, strain, locustag, busco, hlen, ttable, genome_fa, taxonid -> - def species_tag = species.replaceAll(/\s+/, '_') - tuple(species_tag, taxonid) + .map { meta, genome_fa -> + def species_tag = meta.species.replaceAll(/\s+/, '_') + tuple(species_tag, meta.taxonid) } .groupTuple(by: 0) .map { species_tag, taxonids -> tuple(species_tag, taxonids[0]) } @@ -2233,13 +2233,12 @@ workflow { // Build per-assembly channel keyed by species_tag with SRA reads joined. // reads_ch is now a 4-tuple: (species_tag, r1, r2, se) def assembly_with_reads = predict_genome_ch - .map { out, asmid, species, strain, locustag, busco, hlen, ttable, genome_fa, taxonid -> - def species_tag = species.replaceAll(/\s+/, '_') - tuple(species_tag, out, asmid, species, strain, locustag, busco, hlen, ttable, genome_fa) + .map { meta, genome_fa -> + def species_tag = meta.species.replaceAll(/\s+/, '_') + tuple(species_tag, meta, genome_fa) } .combine(reads_ch, by: 0) - // assembly_with_reads tuple: (species_tag, out, asmid, species, strain, locustag, - // busco, hlen, ttable, genome_fa, r1, r2, se) + // assembly_with_reads: (species_tag, meta, genome_fa, r1, r2, se) // RNASEQ_PREPARE: run funannotate train --stop_after_trinity once per species on // the representative (first) assembly, then cache the Trinity-GG FASTA in rnaseq_data/ @@ -2250,14 +2249,12 @@ workflow { // entirely; an empty trinity FASTA is written locally without submitting a SLURM job. def repr_ch = assembly_with_reads .groupTuple(by: 0) - .map { species_tag, outs, asmids, species_list, strains, locustags, - buscos, hlens, ttables, genomes, r1s, r2s, ses -> - tuple(species_tag, outs[0], asmids[0], species_list[0], strains[0], - locustags[0], buscos[0], hlens[0], ttables[0], genomes[0], r1s[0], r2s[0], ses[0]) + .map { species_tag, metas, genomes, r1s, r2s, ses -> + tuple(species_tag, metas[0], genomes[0], r1s[0], r2s[0], ses[0]) } def repr_branched = repr_ch.branch { - has_reads: it[10].size() > 0 || it[12].size() > 0 // r1=[10] or se=[12] + has_reads: it[3].size() > 0 || it[5].size() > 0 // r1=[3] or se=[5] no_reads: true } @@ -2266,7 +2263,7 @@ workflow { // For species with no RNA-seq reads, write an empty trinity FASTA to rnaseq_data/ // in the driver process (no SLURM job) and emit it directly as a shared channel item. def empty_shared_ch = repr_branched.no_reads - .map { species_tag, _out, _asmid, _sp, _st, _lt, _bl, _hl, _tt, _gfa, _r1, _r2, _se -> + .map { species_tag, _meta, _gfa, _r1, _r2, _se -> def empty_fa = file("${launchDir}/rnaseq_data/${species_tag}.trinity-GG.fasta") if (!empty_fa.exists()) { empty_fa.parent.mkdirs() @@ -2281,37 +2278,36 @@ workflow { // Normalized reads (r1/r2/se) come from SRA_FETCH/SRA_FETCH_SE via assembly_with_reads. def train_input = assembly_with_reads .combine(shared_ch, by: 0) - .map { species_tag, out, asmid, sp, st, lt, bl, hl, tt, genome_fa, r1, r2, se, trinity_fa -> - tuple(out, asmid, sp, st, lt, bl, hl, tt, genome_fa, r1, r2, se, trinity_fa) + .map { species_tag, meta, genome_fa, r1, r2, se, trinity_fa -> + tuple(meta, genome_fa, r1, r2, se, trinity_fa) } - // train_input tuple indices: out=0,asmid=1,sp=2,st=3,lt=4,bl=5,hl=6,tt=7, - // genome_fa=8, r1=9, r2=10, se=11, trinity_fa=12 + // train_input: meta=0, genome_fa=1, r1=2, r2=3, se=4, trinity_fa=5 - // Branch on r1 (idx 9), se (idx 11), or trinity_fa (idx 12) sizes. + // Branch on r1 (idx 2), se (idx 4), or trinity_fa (idx 5) sizes. // Assemblies with no RNA-seq bypass FUNANNOTATE_TRAIN entirely. def branched = train_input.branch { - has_rnaseq: it[9].size() > 0 || it[11].size() > 0 || it[12].size() > 0 + has_rnaseq: it[2].size() > 0 || it[4].size() > 0 || it[5].size() > 0 no_rnaseq: true } def predict_no_rnaseq = branched.no_rnaseq - .map { out, asmid, sp, st, lt, bl, hl, tt, genome_fa, _r1, _r2, _se, _tf -> - tuple(out, asmid, sp, st, lt, bl, hl, tt, genome_fa) + .map { meta, genome_fa, _r1, _r2, _se, _tf -> + tuple(meta, genome_fa) } // Skip TRAIN at the channel level when pasa.gff3 already exists and is non-empty, // UNLESS the rnaseq reads or trinity FASTA is newer than the existing prediction GBK // (staleRnaseq), in which case we re-run training so predict can be refreshed too. - def train_todo = branched.has_rnaseq.filter { out, _a, sp, _st, _lt, _bl, _hl, _tt, _gfa, _r1, _r2, _se, _tf -> - def gff3 = file("${params.training_target}/${out}/training/funannotate_train.pasa.gff3") - !gff3.exists() || gff3.size() == 0 || staleRnaseq(out as String, sp as String) + def train_todo = branched.has_rnaseq.filter { meta, _gfa, _r1, _r2, _se, _tf -> + def gff3 = file("${params.training_target}/${meta.id}/training/funannotate_train.pasa.gff3") + !gff3.exists() || gff3.size() == 0 || staleRnaseq(meta.id as String, meta.species as String) } def train_done = branched.has_rnaseq - .filter { out, _a, sp, _st, _lt, _bl, _hl, _tt, _gfa, _r1, _r2, _se, _tf -> - def gff3 = file("${params.training_target}/${out}/training/funannotate_train.pasa.gff3") - gff3.exists() && gff3.size() > 0 && !staleRnaseq(out as String, sp as String) + .filter { meta, _gfa, _r1, _r2, _se, _tf -> + def gff3 = file("${params.training_target}/${meta.id}/training/funannotate_train.pasa.gff3") + gff3.exists() && gff3.size() > 0 && !staleRnaseq(meta.id as String, meta.species as String) } - .map { out, asmid, sp, st, lt, bl, hl, tt, genome_fa, _r1, _r2, _se, _tf -> - tuple(out, asmid, sp, st, lt, bl, hl, tt, genome_fa) + .map { meta, genome_fa, _r1, _r2, _se, _tf -> + tuple(meta, genome_fa) } FUNANNOTATE_TRAIN(train_todo) predict_input_ch = FUNANNOTATE_TRAIN.out.mix(train_done).mix(predict_no_rnaseq) @@ -2319,15 +2315,12 @@ workflow { } // end if (!params.stop_after_sra_query) } else { predict_input_ch = predict_genome_ch - .map { out, asmid, species, strain, locustag, busco, hlen, ttable, genome_fa, _taxonid -> - tuple(out, asmid, species, strain, locustag, busco, hlen, ttable, genome_fa) - } } if ((!params.stop_after_sra_fetch.toBoolean() && !params.stop_after_sra_query.toBoolean()) || !params.run_sra_fetch.toBoolean()) { def predict_ch = predict_input_ch - .filter { out, _asmid, sp, _st, _lt, _bl, _hl, _tt, _gfa -> - gbkResult("${params.target}/${out}/predict_results", out as String) == null || staleRnaseq(out as String, sp as String) + .filter { meta, _gfa -> + gbkResult("${params.target}/${meta.id}/predict_results", meta.id as String) == null || staleRnaseq(meta.id as String, meta.species as String) } FUNANNOTATE_PREDICT(predict_ch) @@ -2339,27 +2332,17 @@ workflow { .splitCsv(header: true) .filter(taxonFilter) .filter(asmidFilter) - .map { row -> - def species = (row.SPECIES?.trim() ?: '').replaceAll(/['"]/, '') - def strain = (row.STRAIN?.trim() ?: '').replaceAll(/['"]/, '').replaceAll(/;.*$/, '').trim().replace(':', ' ') - def out = SampleUtils.makeSampleTag(row.SPECIES?.trim() ?: '', row.STRAIN?.trim() ?: '') - def asmid = row.ASMID?.trim() - def locustag = row.LOCUSTAG?.replaceAll(/[\r\n]/, '')?.trim() - def busco = row.BUSCO_LINEAGE?.trim() - def header_length = 24 - def transl_table = row.TRANSL_TABLE?.trim() ?: '1' - tuple(out, asmid, species, strain, locustag, busco, header_length, transl_table) - } - .filter { out, asmid, _sp, _st, _lt, _bl, _hl, _tt -> out && asmid } + .map { row -> SampleUtils.makeMeta(row) } + .filter { meta -> meta.id && meta.asmid } .take((params.n_test as int) > 0 ? params.n_test as int : -1) - .filter { out, asmid, _sp, _st, _lt, _bl, _hl, _tt -> !suppressSet.contains(asmid) } + .filter { meta -> !suppressSet.contains(meta.asmid) } // Only genomes whose prediction was already complete AND current in a PRIOR run. // This is the exact logical complement of the predict_ch filter, so this set is // disjoint from the genomes (re)predicted in THIS run (which arrive via // FUNANNOTATE_PREDICT.out.metadata below). Keeping them disjoint means no genome // is fed downstream twice and stale genomes correctly wait for the fresh predict. - .filter { out, _asmid, sp, _st, _lt, _bl, _hl, _tt -> - gbkResult("${params.target}/${out}/predict_results", out as String) != null && !staleRnaseq(out as String, sp as String) + .filter { meta -> + gbkResult("${params.target}/${meta.id}/predict_results", meta.id as String) != null && !staleRnaseq(meta.id as String, meta.species as String) } // annotate_ready_ch threads through optional pre-annotate steps. Each optional @@ -2379,49 +2362,43 @@ workflow { def annotate_ready_ch = predict_meta if (params.run_antismash.toBoolean()) { - def as_todo = annotate_ready_ch.filter { out, _a, _sp, _st, _lt, _bl, _hl, _tt -> - def asDir = file("${params.target}/${out}/antismash_local") + def as_todo = annotate_ready_ch.filter { meta -> + def asDir = file("${params.target}/${meta.id}/antismash_local") !(asDir.isDirectory() && asDir.list()?.any { it.endsWith('.json') || it.endsWith('.json.gz') }) } - def as_done = annotate_ready_ch.filter { out, _a, _sp, _st, _lt, _bl, _hl, _tt -> - def asDir = file("${params.target}/${out}/antismash_local") + def as_done = annotate_ready_ch.filter { meta -> + def asDir = file("${params.target}/${meta.id}/antismash_local") asDir.isDirectory() && asDir.list()?.any { it.endsWith('.json') || it.endsWith('.json.gz') } } ANTISMASH_RUN(as_todo) def as_completed = ANTISMASH_RUN.out - .map { out, _files -> tuple(out, 'done') } - .join(predict_meta) - .map { out, _flag, asmid, sp, st, lt, bl, hl, tt -> tuple(out, asmid, sp, st, lt, bl, hl, tt) } + .map { meta, _files -> meta } annotate_ready_ch = as_completed.mix(as_done) } if (params.run_interpro.toBoolean()) { - def ipr_todo = annotate_ready_ch.filter { out, _a, _sp, _st, _lt, _bl, _hl, _tt -> - !file("${params.target}/${out}/annotate_misc/iprscan.xml").exists() + def ipr_todo = annotate_ready_ch.filter { meta -> + !file("${params.target}/${meta.id}/annotate_misc/iprscan.xml").exists() } - def ipr_done = annotate_ready_ch.filter { out, _a, _sp, _st, _lt, _bl, _hl, _tt -> - file("${params.target}/${out}/annotate_misc/iprscan.xml").exists() + def ipr_done = annotate_ready_ch.filter { meta -> + file("${params.target}/${meta.id}/annotate_misc/iprscan.xml").exists() } INTERPROSCAN_RUN(ipr_todo) def ipr_completed = INTERPROSCAN_RUN.out - .map { out, _xml -> tuple(out, 'done') } - .join(predict_meta) - .map { out, _flag, asmid, sp, st, lt, bl, hl, tt -> tuple(out, asmid, sp, st, lt, bl, hl, tt) } + .map { meta, _xml -> meta } annotate_ready_ch = ipr_completed.mix(ipr_done) } if (params.run_signalp.toBoolean()) { - def sp_todo = annotate_ready_ch.filter { out, _a, _sp, _st, _lt, _bl, _hl, _tt -> - !file("${params.target}/${out}/annotate_misc/signalp.results.txt").exists() + def sp_todo = annotate_ready_ch.filter { meta -> + !file("${params.target}/${meta.id}/annotate_misc/signalp.results.txt").exists() } - def sp_done = annotate_ready_ch.filter { out, _a, _sp, _st, _lt, _bl, _hl, _tt -> - file("${params.target}/${out}/annotate_misc/signalp.results.txt").exists() + def sp_done = annotate_ready_ch.filter { meta -> + file("${params.target}/${meta.id}/annotate_misc/signalp.results.txt").exists() } SIGNALP_RUN(sp_todo) def sp_completed = SIGNALP_RUN.out - .map { out, _txt -> tuple(out, 'done') } - .join(predict_meta) - .map { out, _flag, asmid, sp, st, lt, bl, hl, tt -> tuple(out, asmid, sp, st, lt, bl, hl, tt) } + .map { meta, _txt -> meta } annotate_ready_ch = sp_completed.mix(sp_done) } @@ -2431,37 +2408,38 @@ workflow { // Reads are joined from SRA_FETCH (storeDir-cached, so prior-run reads are reused). // The join on upd_signal gates annotate_ready_ch so ANNOTATE waits for UPDATE. def upd_input = predict_meta - .map { out, asmid, species, strain, locustag, busco, hlen, ttable -> - def species_tag = species.replaceAll(/\s+/, '_') - tuple(species_tag, out, asmid, species, strain, locustag, busco, hlen, ttable) + .map { meta -> + def species_tag = meta.species.replaceAll(/\s+/, '_') + tuple(species_tag, meta) } .combine(reads_ch, by: 0) - .map { _st, out, asmid, species, strain, locustag, busco, hlen, ttable, r1, r2 -> - tuple(out, asmid, species, strain, locustag, busco, hlen, ttable, r1, r2) + .map { _st, meta, r1, r2 -> + tuple(meta, r1, r2) } - def upd_todo = upd_input.filter { out, _a, _sp, _st, _lt, _bl, _hl, _tt, _r1, _r2 -> - gbkResult("${params.target}/${out}/update_results", out as String) == null + def upd_todo = upd_input.filter { meta, _r1, _r2 -> + gbkResult("${params.target}/${meta.id}/update_results", meta.id as String) == null } def upd_done_signal = upd_input - .filter { out, _a, _sp, _st, _lt, _bl, _hl, _tt, _r1, _r2 -> - gbkResult("${params.target}/${out}/update_results", out as String) != null + .filter { meta, _r1, _r2 -> + gbkResult("${params.target}/${meta.id}/update_results", meta.id as String) != null } - .map { out, _a, _sp, _st, _lt, _bl, _hl, _tt, _r1, _r2 -> tuple(out, 'upd') } + .map { meta, _r1, _r2 -> tuple(meta.id, 'upd') } FUNANNOTATE_UPDATE(upd_todo) def upd_signal = FUNANNOTATE_UPDATE.out - .map { out, _a, _sp, _st, _lt, _bl, _hl, _tt -> tuple(out, 'upd') } + .map { meta -> tuple(meta.id, 'upd') } .mix(upd_done_signal) annotate_ready_ch = annotate_ready_ch + .map { meta -> tuple(meta.id, meta) } .join(upd_signal) - .map { out, asmid, sp, st, lt, bl, hl, tt, _flag -> tuple(out, asmid, sp, st, lt, bl, hl, tt) } + .map { _id, meta, _flag -> meta } } else { log.warn "run_update=true but run_sra_fetch=false; funannotate update skipped (no reads available)" } } if (params.run_annotate.toBoolean()) { - FUNANNOTATE_ANNOTATE(annotate_ready_ch.filter { out, _a, _sp, _st, _lt, _bl, _hl, _tt -> - gbkResult("${params.target}/${out}/annotate_results", out as String) == null + FUNANNOTATE_ANNOTATE(annotate_ready_ch.filter { meta -> + gbkResult("${params.target}/${meta.id}/annotate_results", meta.id as String) == null }) } } // end if (!params.stop_after_sra_fetch || !params.run_sra_fetch) diff --git a/lib/SampleUtils.groovy b/lib/SampleUtils.groovy index dc27917..11d8c80 100644 --- a/lib/SampleUtils.groovy +++ b/lib/SampleUtils.groovy @@ -36,4 +36,37 @@ class SampleUtils { .join('_') .replaceAll(/[\s\/\#\[\]\*\?\{\}]+/, '_') } + + /** + * Build the canonical per-sample `meta` map from a raw samples.csv row (the + * map produced by splitCsv(header: true)). + * + * This is the data contract for the DSL2 modularization (REFACTORING_PLAN.md + * Principle 0): channels carry `tuple val(meta), val(genome)` and `meta.id` + * is the ONLY field used for tag{}/naming. It reproduces, field-for-field, + * the cleaning the funannotate.nf `jobs` channel does today, so wiring it in + * is a behaviour-preserving swap. + * + * NOTE: `header_length` is intentionally NOT a meta field — it is a constant + * (params.header_length, default 24), not per-sample payload. The genome path + * also travels separately as the 2nd tuple element, not inside meta. + * + * Not yet wired into the workflow; introduced ahead of the atomic channel + * conversion so the contract has one authoritative definition. + */ + static Map makeMeta(Map row) { + def species = (row.SPECIES?.trim() ?: '').replaceAll(/['"]/, '') + def strain = (row.STRAIN?.trim() ?: '').replaceAll(/['"]/, '') + .replaceAll(/;.*$/, '').trim().replace(':', ' ') + return [ + id : makeSampleTag(row.SPECIES?.trim() ?: '', row.STRAIN?.trim() ?: ''), + asmid : row.ASMID?.trim(), + species : species, + strain : strain, + locustag : row.LOCUSTAG?.replaceAll(/[\r\n]/, '')?.trim(), + busco : row.BUSCO_LINEAGE?.trim(), + transl_table: row.TRANSL_TABLE?.trim() ?: '1', + taxonid : row.NCBI_TAXONID?.trim(), + ] + } } diff --git a/nextflow.config b/nextflow.config index ae7182a..e915a87 100644 --- a/nextflow.config +++ b/nextflow.config @@ -6,16 +6,23 @@ * * pipeline : annotate | earlgrey | test * executor : slurm | local - * provisioning : module | pixi | singularity + * provisioning : ucr_hpcc | pixi | singularity + * + * Provisioning is split into a PORTABLE path and an INSTITUTIONAL path: + * - singularity / pixi : portable — runnable at any site; tools come from + * containers or project-local pixi envs. + * - ucr_hpcc : institutional — UCR HPCC Lmod modules. Encodes + * site-specific module names/paths; analogue of an + * nf-core/configs institutional profile. * * Examples: - * -profile annotate,slurm,module (default / recommended on HPCC) - * -profile annotate,local,singularity (containers, run head locally) - * -profile annotate,local,module -stub-run (graph/dry test) + * -profile annotate,slurm,ucr_hpcc (default / recommended on UCR HPCC) + * -profile annotate,local,singularity (portable: containers, head local) + * -profile annotate,local,ucr_hpcc -stub-run (graph/dry test) * * The pipeline profile supplies params + per-label resources; the executor * profile selects SLURM vs local; the provisioning profile fills each process - * label's `beforeScript` (module/pixi) or `container` (singularity). Process + * label's `beforeScript` (ucr_hpcc/pixi) or `container` (singularity). Process * scripts themselves are tool-agnostic — they contain NO `module load` lines. */ @@ -50,6 +57,10 @@ params { // Optional newline/CSV list of ASMIDs to skip. suppress = "${launchDir}/suppress.txt" debug = false + // GenBank/funannotate locus-tag header length cap. Was a hardcoded constant (24) + // threaded through the per-sample tuple; promoted to a param ahead of the meta-map + // conversion (REFACTORING_PLAN.md Principle 0) since it is global, not per-sample. + header_length = 24 // --help prints the schema-driven parameter help and exits (see funannotate.nf). help = false } @@ -95,7 +106,8 @@ profiles { } // ---- provisioning axis --------------------------------------------------- - module { includeConfig 'conf/provision_module.config' } + // Institutional (UCR HPCC Lmod). Portable alternatives: pixi, singularity. + ucr_hpcc { includeConfig 'conf/provision_ucr_hpcc.config' } pixi { includeConfig 'conf/provision_pixi.config' } singularity { singularity.enabled = true diff --git a/nextflow_schema.json b/nextflow_schema.json index 0c50036..b076e1a 100644 --- a/nextflow_schema.json +++ b/nextflow_schema.json @@ -267,6 +267,12 @@ "minimum": 0, "description": "Contig count above this counts as 'fragmented' for the predict guard.", "default": 1000 + }, + "header_length": { + "type": "integer", + "minimum": 1, + "description": "GenBank/funannotate locus-tag header length cap (was a hardcoded per-sample constant).", + "default": 24 } } }, diff --git a/run_annotate.sh b/run_annotate.sh index b8d1b8c..127a302 100755 --- a/run_annotate.sh +++ b/run_annotate.sh @@ -16,8 +16,9 @@ # - or the current dir: PIPELINE=$PWD # REVISION git branch / tag / commit to run (default: pipeline default branch) # -# Default provisioning is Lmod modules on SLURM. Swap axes via env vars: -# PROVISION=singularity sbatch run_annotate.sh +# Default provisioning is the UCR HPCC institutional profile (Lmod modules) on +# SLURM. Swap axes via env vars: +# PROVISION=singularity sbatch run_annotate.sh # portable containers # EXECUTOR=local sbatch run_annotate.sh # head + tasks local # REVISION=v0.1.0 sbatch run_annotate.sh # pin a release # @@ -34,7 +35,7 @@ module load nextflow PIPELINE="${PIPELINE:-stajichlab/nf_funannotate1}" REVISION="${REVISION:-}" EXECUTOR="${EXECUTOR:-slurm}" -PROVISION="${PROVISION:-module}" +PROVISION="${PROVISION:-ucr_hpcc}" mkdir -p logs/nextflow diff --git a/scripts/create_refactor_issues.sh b/scripts/create_refactor_issues.sh new file mode 100755 index 0000000..2ff0e82 --- /dev/null +++ b/scripts/create_refactor_issues.sh @@ -0,0 +1,94 @@ +#!/usr/bin/env bash +# +# create_refactor_issues.sh — create the modularization backlog on GitHub. +# +# Idempotent-ish: skips any issue whose exact title already exists (open OR +# closed). Creates the `refactor`/`epic` labels and a milestone if missing. +# Requires: gh (authenticated), run from inside the repo. +# +# ./scripts/create_refactor_issues.sh # create everything +# DRY_RUN=1 ./scripts/create_refactor_issues.sh # print what would happen +# +# Mirrors .github/REFACTOR_ISSUES.md — keep them in sync. + +set -euo pipefail + +REPO="$(gh repo view --json nameWithOwner -q .nameWithOwner)" +MILESTONE="DSL2 modularization" +DRY_RUN="${DRY_RUN:-0}" + +say() { printf '%s\n' "$*"; } +run() { if [ "$DRY_RUN" = 1 ]; then say "DRY: $*"; else "$@"; fi; } + +ensure_label() { # name color description + if ! gh label list --repo "$REPO" --limit 200 | grep -qiE "^$1[[:space:]]"; then + run gh label create "$1" --repo "$REPO" --color "$2" --description "$3" || true + fi +} + +ensure_milestone() { + if ! gh api "repos/$REPO/milestones?state=all" -q '.[].title' | grep -qxF "$MILESTONE"; then + run gh api "repos/$REPO/milestones" -f title="$MILESTONE" \ + -f description="Break funannotate.nf monolith into modules + subworkflows" >/dev/null || true + fi +} + +issue_exists() { # title + gh issue list --repo "$REPO" --state all --limit 500 --json title -q '.[].title' \ + | grep -qxF "$1" +} + +make_issue() { # title labels body + local title="$1" labels="$2" body="$3" + if issue_exists "$title"; then + say "skip (exists): $title" + return + fi + run gh issue create --repo "$REPO" --title "$title" --label "$labels" \ + --milestone "$MILESTONE" --body "$body" +} + +ensure_label refactor 1d76db "DSL2 modularization work" +ensure_label epic 5319e7 "Tracking epic" +ensure_milestone + +make_issue "EPIC: Modularize nf_funannotate1 into DSL2 modules + subworkflows" "refactor,epic" \ +"Break the 2,470-line funannotate.nf monolith into one-process-per-file modules composed by subworkflows, on a meta-map data contract. See REFACTORING_PLAN.md and .github/REFACTOR_ISSUES.md. Child issues are ordered; the meta-map and INPUT_CHECK issues block the rest." + +make_issue "Adopt meta-map data contract (Phase 0, BLOCKER)" "refactor" \ +"Replace the positional 10-tuple with \`tuple val(meta), path(genome)\`. meta.id is the only naming field; header_length becomes params.header_length. No further extraction until this lands. See REFACTORING_PLAN.md Principle 0." + +make_issue "Shared INPUT_CHECK subworkflow (dedupe earlgrey)" "refactor" \ +"Extract samplesheet parse + taxon/asmid/suppress filtering (duplicated in funannotate.nf and earlgrey_mask.nf) into subworkflows/local/input_check.nf emitting the meta channel. Both entrypoints consume it." + +make_issue "Repo skeleton + relocate existing modules (Phase 1)" "refactor" \ +"Create main.nf, workflows/, subworkflows/local/, modules/local/. Move asm_stats.nf and split annotation_tools.nf into antismash/signalp/interproscan single-process modules." + +make_issue "versions.yml + conf/base.config + conf/modules.config (Phase 2)" "refactor" \ +"Each process emits versions.yml; resources move to label-based conf/base.config; per-process publishDir/ext.args move to conf/modules.config. The pattern every later module copies." + +make_issue "Extract setup modules (Phase 3)" "refactor,good first issue" \ +"Extract SETUP_TAXONDB / SETUP_FUNANNOTATE_DB / SETUP_AUGUSTUS_CONFIG into modules/local + subworkflows/local/setup_dbs.nf. Preserve storeDir run-at-most-once caching." + +make_issue "Genome clean + prepare_genome subworkflow (Phase 4)" "refactor" \ +"Extract GENOME_CLEAN / GENOME_CLEAN_BATCH + subworkflows/local/prepare_genome.nf (clean -> asm_stats -> mask). Preserve FCS-GX /dev/shm staging and the skip-already-cleaned batch gating." + +make_issue "Masking subworkflow + per-tool modules (Phase 5)" "refactor" \ +"Replace the single masking mega-process: one module per masker (tantan now; repeatmodeler/repeatmasker/earlgrey follow-ups), selection in subworkflows/local/mask.nf keyed on params.mask_tool. Support NONE." + +make_issue "RNA-seq fetch subworkflow (Phase 6, hardest)" "refactor" \ +"Extract SRA_QUERY/_BATCH, COLLECT_SRA_QUERY, WRITE_EMPTY_READS, SRA_FETCH/_SE, RNASEQ_PREPARE into modules + subworkflows/local/rnaseq.nf. Done after pattern proven. Preserve shared Trinity-GG and maxForks." + +make_issue "Funannotate predict subworkflow (Phase 7)" "refactor" \ +"Extract FUNANNOTATE_TRAIN/PREDICT/UPDATE into modules + subworkflows/local/predict.nf. Keep pre-flight size/fragmentation checks and post-flight not-enough-models guard. update is optional/param-gated." + +make_issue "Annotation subworkflow (Phase 8)" "refactor" \ +"subworkflows/local/annotate.nf composing optional antismash/signalp/interproscan modules -> funannotate_annotate.nf, each independently param-gated." + +make_issue "Consolidate ucr_hpcc institutional profile + portable container path (Phase 9)" "refactor" \ +"module->ucr_hpcc rename is done. Fold UCR SLURM partitions/clusterOptions into the institutional profile; ensure a fully portable container/conda run works without Lmod. Document the new-site path." + +make_issue "nf-core hygiene (Phase 9, stretch)" "refactor,documentation" \ +"docs/usage.md + docs/output.md, assets/schema_input.json, naming decision (nf-core forbids underscores/digits), optional MultiQC + nf-test. Record nf-core submission vs nf-core-inspired decision in REFACTORING_PLAN.md." + +say "Done."