Modularization plan, ucr_hpcc institutional profile, and CI-green fix

.github/REFACTOR_ISSUES.md

@@ -0,0 +1,167 @@
+# Refactor issue tracker
+
+Source-of-truth backlog for the DSL2 modularization (see `REFACTORING_PLAN.md`).
+Create these on GitHub with `scripts/create_refactor_issues.sh` (uses `gh`).
+Each non-epic issue closes one `## Phase N` of the plan and must pass the
+stub-run gate before merge:
+
+```
+nextflow config . -profile test && nextflow run . -profile test -stub-run && nextflow run . --help
+```
+
+---
+
+## EPIC: Modularize nf_funannotate1 into DSL2 modules + subworkflows
+labels: refactor, epic
+
+Break the 2,470-line `funannotate.nf` monolith into one-process-per-file modules
+composed by subworkflows, on a `meta`-map data contract. nf-core-*inspired*, not
+nf-core-submitted (see `REFACTORING_PLAN.md` for the verdict). Child issues #1–#12
+below; they are ordered — #1 and #2 block the rest.
+
+Definition of done: monolith replaced by `main.nf` + `workflows/` +
+`subworkflows/local/` + `modules/local/`; every process emits `versions.yml`;
+earlgrey/funannotate share `INPUT_CHECK`; CI green at every step.
+
+---
+
+## 1. Adopt `meta`-map data contract  (Phase 0 — BLOCKER)
+labels: refactor
+
+Replace the positional 10-tuple
+`tuple(out, asmid, species, strain, locustag, busco, header_length, transl_table, gz, taxonid)`
+with `tuple val(meta), path(genome)` where `meta` is the map defined in
+`REFACTORING_PLAN.md` (Principle 0). `meta.id` is the only naming field;
+`header_length` becomes `params.header_length`. No more extraction until this lands.
+
+- [ ] Build `meta` in the workflow channel construction
+- [ ] Update every call site to consume `meta`
+- [ ] `params.header_length` added to schema with default 24
+- [ ] Stub-run gate green
+
+## 2. Shared `INPUT_CHECK` subworkflow (dedupe earlgrey)  (Phase 0)
+labels: refactor
+
+Extract the samplesheet `splitCsv` + taxon/asmid/suppress filtering (duplicated
+in `funannotate.nf` and `earlgrey_mask.nf`) into `subworkflows/local/input_check.nf`
+emitting the `meta` channel. Both entrypoints call it.
+
+- [ ] `subworkflows/local/input_check.nf` emits `ch_genomes` (meta + genome)
+- [ ] `funannotate.nf` and `earlgrey_mask.nf` both consume it; no duplicated parse
+- [ ] Stub-run gate green for both entrypoints
+
+## 3. Repo skeleton + relocate existing modules  (Phase 1)
+labels: refactor
+
+Create `main.nf`, `workflows/funannotate.nf`, `subworkflows/local/`,
+`modules/local/`. Move `modules/asm_stats.nf` and `modules/annotation_tools.nf`
+to one-process-per-file under `modules/local/` (split `annotation_tools.nf` into
+`antismash.nf` / `signalp.nf` / `interproscan.nf`).
+
+- [ ] Directory skeleton in place; `main.nf` is a thin entrypoint
+- [ ] `annotation_tools.nf` split into 3 single-process modules
+- [ ] Stub-run gate green
+
+## 4. `versions.yml` + `conf/base.config` + `conf/modules.config`  (Phase 2)
+labels: refactor
+
+Establish the conventions every later module copies: each process emits
+`versions.yml`; resources move to label-based `conf/base.config`
+(`process_low/medium/high`); per-process `publishDir`/`ext.args` move to
+`conf/modules.config`.
+
+- [ ] `conf/base.config` with resource labels
+- [ ] `conf/modules.config` with publishDir + ext.args
+- [ ] At least one module emits and the workflow collects `versions.yml`
+- [ ] Stub-run gate green
+
+## 5. Setup modules  (Phase 3)
+labels: refactor, good first issue
+
+Extract `SETUP_TAXONDB`, `SETUP_FUNANNOTATE_DB`, `SETUP_AUGUSTUS_CONFIG` into
+`modules/local/` + `subworkflows/local/setup_dbs.nf`. Preserve `storeDir`
+(run-at-most-once) caching. Good first real extraction — validates the gate.
+
+- [ ] 3 modules + `setup_dbs.nf` subworkflow; storeDir preserved
+- [ ] Stub-run gate green
+
+## 6. Genome clean + `prepare_genome` subworkflow  (Phase 4)
+labels: refactor
+
+Extract `GENOME_CLEAN` / `GENOME_CLEAN_BATCH` into modules and a
+`subworkflows/local/prepare_genome.nf` (clean → asm_stats → mask).
+**Preserve the FCS-GX `/dev/shm` staging** and the "skip already-cleaned" batch
+gating so a fully-cleaned batch never pays the ~30-min staging cost.
+
+- [ ] Modules + `prepare_genome.nf`; FCS-GX /dev/shm staging preserved
+- [ ] Batch padding/skip behavior unchanged
+- [ ] Stub-run gate green
+
+## 7. Masking subworkflow + per-tool modules  (Phase 5)
+labels: refactor
+
+Replace the planned single masking mega-process. One module per masker
+(`mask_tantan.nf` now; `mask_repeatmodeler.nf`, `mask_repeatmasker.nf`,
+`mask_earlgrey.nf` as stubs/follow-ups); selection logic in
+`subworkflows/local/mask.nf` keyed on `params.mask_tool`.
+
+- [ ] `mask.nf` selects one masker by param; `NONE` path supported
+- [ ] `mask_tantan.nf` extracted; others stubbed with clear TODO
+- [ ] Stub-run gate green
+
+## 8. RNA-seq fetch subworkflow  (Phase 6 — hardest)
+labels: refactor
+
+Extract `SRA_QUERY` / `SRA_QUERY_BATCH` / `COLLECT_SRA_QUERY` /
+`WRITE_EMPTY_READS` / `SRA_FETCH` / `SRA_FETCH_SE` / `RNASEQ_PREPARE` into modules
++ `subworkflows/local/rnaseq.nf`. Done **after** the pattern is proven on easier
+processes. Preserve per-species shared Trinity-GG output and `maxForks` limits.
+
+- [ ] 7 modules + `rnaseq.nf`; shared Trinity-GG semantics preserved
+- [ ] maxForks / rate limits preserved
+- [ ] Stub-run gate green
+
+## 9. Funannotate predict subworkflow  (Phase 7)
+labels: refactor
+
+Extract `FUNANNOTATE_TRAIN` / `FUNANNOTATE_PREDICT` / `FUNANNOTATE_UPDATE` into
+modules + `subworkflows/local/predict.nf`. Keep the pre-flight assembly
+size/fragmentation validation and post-flight "not enough models" guard.
+
+- [ ] 3 modules + `predict.nf`; pre/post-flight checks preserved
+- [ ] `update` is optional (param-gated)
+- [ ] Stub-run gate green
+
+## 10. Annotation subworkflow  (Phase 8)
+labels: refactor
+
+`subworkflows/local/annotate.nf` composing optional `antismash` / `signalp` /
+`interproscan` modules → `funannotate_annotate.nf`. Each optional tool
+independently param-gated.
+
+- [ ] `annotate.nf` with per-tool gating; merges results into funannotate annotate
+- [ ] Stub-run gate green
+
+## 11. Consolidate `ucr_hpcc` institutional profile + portable container path  (Phase 9)
+labels: refactor
+
+The `module`→`ucr_hpcc` rename is done. Finish the repivot: fold UCR SLURM
+partitions / `clusterOptions` into the institutional profile, and ensure a fully
+portable run works via per-module conda/biocontainer directives (no Lmod).
+Document the "copy to `conf/provision_<site>.config`" path for new sites.
+
+- [ ] UCR partition config consolidated under the institutional profile
+- [ ] Portable container/conda path runs without any UCR modules
+- [ ] `docs/` note for adding a new institution
+
+## 12. nf-core hygiene (stretch)  (Phase 9)
+labels: refactor, documentation
+
+`docs/usage.md` + `docs/output.md`, `assets/schema_input.json` (samplesheet
+schema), pipeline naming decision (nf-core forbids underscores/digits), optional
+MultiQC + `nf-test`. Decide explicitly whether to pursue nf-core submission or
+stay nf-core-inspired.
+
+- [ ] `docs/usage.md`, `docs/output.md`
+- [ ] `assets/schema_input.json`
+- [ ] naming + submission decision recorded in `REFACTORING_PLAN.md`

README.md

@@ -12,7 +12,7 @@ The pipeline lives at the repo root (`funannotate.nf` + `nextflow.config`), so i
 runs directly from GitHub — no clone required:
 
 ```bash
-nextflow run stajichlab/nf_funannotate1 -profile annotate,slurm,module -resume
+nextflow run stajichlab/nf_funannotate1 -profile annotate,slurm,ucr_hpcc -resume
 ```
 
 Nextflow caches the repo under `~/.nextflow/assets/`; add `-r <branch|tag>` to pin
@@ -26,7 +26,7 @@ a revision and `-latest` to pull updates. Outputs and the `samples.csv` /
 nextflow run stajichlab/nf_funannotate1 -profile test -stub-run
 
 # real run on SLURM with environment modules (from your launch dir, with samples.csv)
-nextflow run stajichlab/nf_funannotate1 -profile annotate,slurm,module -resume --n_test 1
+nextflow run stajichlab/nf_funannotate1 -profile annotate,slurm,ucr_hpcc -resume --n_test 1
 
 # or, from a local checkout, use the sbatch launcher
 sbatch /path/to/nf_funannotate1/run_annotate.sh --n_test 1
@@ -43,7 +43,7 @@ Compose one option from each of three axes: `-profile <pipeline>,<executor>,<pro
 | **provisioning** | `module` (default) · `pixi` · `singularity` |
 
 ```bash
-nextflow run stajichlab/nf_funannotate1 -profile annotate,slurm,module -resume
+nextflow run stajichlab/nf_funannotate1 -profile annotate,slurm,ucr_hpcc -resume
 nextflow run stajichlab/nf_funannotate1 -profile annotate,local,singularity -resume
 ```
 

REFACTORING_PLAN.md

@@ -0,0 +1,192 @@
+# nf_funannotate1 — Modularization Plan (authoritative)
+
+> This is the **single source of truth** for the DSL2 modularization effort. It
+> supersedes the earlier `IMPLEMENTATION_SUMMARY.md` and `MODULE_STRUCTURE.txt`
+> (removed). Progress is tracked in GitHub issues — see
+> `.github/REFACTOR_ISSUES.md` and `scripts/create_refactor_issues.sh`.
+
+## Where we actually are
+
+- `funannotate.nf`: **2,470-line monolith, 20 inline processes.**
+- `earlgrey_mask.nf`: separate 394-line pipeline that **duplicates** the
+  samplesheet parse + taxonomy/asmid/suppress filtering from `funannotate.nf`.
+- Extracted so far: `modules/asm_stats.nf`, `modules/annotation_tools.nf` (2 of 20).
+- Strong base already present: nf-schema + `nextflow_schema.json`, orthogonal
+  profiles, manifest, CITATIONS/COC/CHANGELOG/LICENSE, stub-run CI.
+
+The goal is an **nf-core-*inspired*** layout (not nf-core submission): adopt the
+parts that are pure engineering wins, skip the parts that fight our HPC reality.
+
+---
+
+## Principle 0 — the data contract (do this FIRST; blocks everything else)
+
+Every process is currently wired with a fragile positional 10-tuple:
+
+```groovy
+tuple(out, asmid, species, strain, locustag, busco, header_length, transl_table, gz, taxonid)
+```
+
+Adding an 11th field touches every process. **Replace it with a `meta` map**, the
+standard DSL2 idiom. Genome travels as a separate `path`:
+
+```groovy
+// Canonical channel element: tuple val(meta), path(genome)
+meta = [
+    id          : out,          // unique sample tag — used for tag{} and file naming
+    asmid       : asmid,
+    species     : species,
+    strain      : strain,
+    locustag    : locustag,
+    busco       : busco,        // BUSCO_LINEAGE
+    transl_table: transl_table, // default '1'
+    taxonid     : taxonid,
+]
+```
+
+Rules:
+- `meta.id` is the **only** field used for naming/`tag`; everything else is payload.
+- `header_length` (constant 24) becomes `params.header_length`, **not** a meta field.
+- Build `meta` once, in the `INPUT_CHECK` subworkflow (below). No process
+  re-parses the samplesheet.
+- A module's `input:` declares `tuple val(meta), path(x)` and never positionally
+  unpacks fields it doesn't use.
+
+Until `meta` is adopted, **do not extract more modules** — every module written
+against the old tuple is rework.
+
+---
+
+## Target architecture
+
+```
+main.nf                              # thin entrypoint: parse args, call workflow
+workflows/
+    funannotate.nf                   # wires the subworkflows (was the monolith)
+    earlgrey.nf                      # curated-mask entry, reuses shared subworkflows
+subworkflows/local/
+    input_check.nf                   # samplesheet -> meta channel + taxon/asmid/suppress filters  (SHARED)
+    setup_dbs.nf                     # SETUP_TAXONDB / FUNANNOTATE_DB / AUGUSTUS_CONFIG gating
+    prepare_genome.nf                # clean -> asm_stats -> mask
+    mask.nf                          # selects ONE masker module by params.mask_tool
+    rnaseq.nf                        # sra_query -> sra_fetch(_se) -> rnaseq_prepare
+    predict.nf                       # train -> predict -> (update)
+    annotate.nf                      # antismash|signalp|interpro -> funannotate annotate
+modules/local/                       # ONE process per file
+    setup_taxondb.nf  setup_funannotate_db.nf  setup_augustus_config.nf
+    genome_clean.nf   genome_clean_batch.nf    asm_stats.nf
+    mask_tantan.nf    mask_repeatmodeler.nf    mask_repeatmasker.nf  mask_earlgrey.nf
+    sra_query.nf      sra_query_batch.nf       collect_sra_query.nf  write_empty_reads.nf
+    sra_fetch.nf      sra_fetch_se.nf          rnaseq_prepare.nf
+    funannotate_train.nf  funannotate_predict.nf  funannotate_update.nf
+    antismash.nf      interproscan.nf          signalp.nf            funannotate_annotate.nf
+    select_reps.nf                   # earlgrey representative selection
+conf/
+    base.config                      # resources by label (process_low/medium/high/...)
+    modules.config                   # per-process publishDir + ext.args (nf-core idiom)
+```
+
+### Why this and not the old plan
+
+- **One tool = one process = one module file.** The old plan bundled 3 processes
+  per file (`sra_query.nf`, `annotation_tools.nf`, `databases.nf`). That is the
+  *opposite* of the convention and kills reuse. Group sequences with
+  **subworkflows**, which the old plan never mentioned.
+- **Masking is a subworkflow, not a mega-process.** A single process with an
+  `if/else` over NONE/TANTAN/REPEATMODELER/REPEATMASKER/EARLGREY is an
+  anti-pattern. Each masker is its own module; the *selection* lives in
+  `subworkflows/local/mask.nf`.
+- **Kill the funannotate/earlgrey duplication.** Both entrypoints parse the same
+  samplesheet and apply the same filters. Extract that into `input_check.nf` once
+  and call it from both. EarlGrey is a whole pipeline (SELECT_REPS + asm_stats +
+  representative-per-species), not a masking flavor — it reuses shared
+  subworkflows rather than being folded into one module.
+
+---
+
+## Per-process extraction checklist (the stub-run gate)
+
+Apply to **one process per commit/PR**. The monolith must stay runnable at every
+commit.
+
+For process `P`:
+
+- [ ] Create `modules/local/<p>.nf` with `process P { ... }`.
+- [ ] `input:` uses `tuple val(meta), path(...)` (no positional field unpacking).
+- [ ] Add `tag "${meta.id}"` and a resource `label` (`process_low|medium|high`).
+- [ ] Keep the existing `stub:` block; keep `storeDir`/`publishDir` behavior.
+- [ ] Emit a version: `path "versions.yml", emit: versions` + a `cat <<-END_VERSIONS`
+      block capturing the tool version. (Foundational — see Issue 3.)
+- [ ] Move the process's resource/`withName` block out of
+      `conf/profile_annotate.config` into `conf/modules.config` (name unchanged,
+      so existing `withName:` selectors keep matching).
+- [ ] In the workflow, replace the inline `process P {}` with
+      `include { P } from '../modules/local/p'` and adapt the call site to pass `meta`.
+- [ ] **Gate (must pass before commit):**
+      ```
+      nextflow config .  -profile test
+      nextflow run    .  -profile test -stub-run
+      nextflow run    .  --help
+      ```
+- [ ] Commit. One process. Repeat.
+
+---
+
+## Migration order (corrected)
+
+The old plan started with RNA-seq fetch ("least interdependent") — but
+`SRA_FETCH` is the **single most complex** process (~270 lines). Prove the
+pattern on a leaf first, then attack the hard pieces.
+
+| Phase | Work | Why here |
+|------|------|----------|
+| 0 | `meta` map contract + `INPUT_CHECK` subworkflow | Blocks all extraction; dedupes earlgrey |
+| 1 | Skeleton: `main.nf`, `workflows/`, `subworkflows/local/`, `modules/local/`; move existing `asm_stats` + `annotation_tools` to convention | Establishes layout cheaply |
+| 2 | `versions.yml` + `conf/base.config` + `conf/modules.config` | Pattern every later module copies |
+| 3 | Setup modules (3 leaf processes) | Easiest real extraction; validates gate |
+| 4 | Genome clean + `prepare_genome` subworkflow | Preserve FCS-GX `/dev/shm` staging |
+| 5 | `mask` subworkflow + per-tool masker modules | Replaces the mega-process design |
+| 6 | `rnaseq` subworkflow (the hard one) | Done *after* pattern is proven |
+| 7 | `predict` subworkflow (train/predict/update) | Core |
+| 8 | `annotate` subworkflow | Composition of optional tools |
+| 9 | nf-core hygiene (docs/usage, docs/output, schema_input, naming, MultiQC) | Stretch |
+
+---
+
+## Provisioning repivot (done)
+
+The `module` provisioning profile is renamed to **`ucr_hpcc`** — an
+*institutional* profile in the nf-core/configs sense. The Lmod module names and
+`/bigdata` paths only exist at UCR, so the name now says so. Portable runs use
+`singularity` (containers) or `pixi`. New sites copy
+`conf/provision_ucr_hpcc.config` to `conf/provision_<site>.config` and register a
+matching profile.
+
+```
+-profile annotate,slurm,ucr_hpcc      # institutional (default on UCR HPCC)
+-profile annotate,local,singularity   # portable
+```
+
+Issue 11 covers the fuller consolidation (folding UCR SLURM partitions /
+`clusterOptions` into the same institutional profile).
+
+---
+
+## Distance from nf-core
+
+| Area | State |
+|------|-------|
+| Scaffolding (schema, CITATIONS, COC, CHANGELOG, LICENSE, CI) | ~30–40% there |
+| `meta` map | none (Phase 0) |
+| Structure (`main.nf`/`workflows`/`subworkflows`/`modules`) | monolith |
+| `versions.yml` per module + MultiQC | none (mandatory for nf-core) |
+| Containers per module | **biggest gap** — relies on Lmod/pixi; nf-core needs conda+biocontainer per process |
+| Naming | `nf_funannotate1` violates nf-core naming (underscores/digits) |
+| nf-test, `docs/usage.md`+`output.md`, `assets/schema_input.json`, `.nf-core.yml` | missing |
+
+**Verdict:** ~30–40% on peripheral scaffolding, ~0% on the two load-bearing items
+(meta-maps + container-per-module). Full `nf-core lint` compliance is a multi-week
+rewrite, much of which fights our HPC reality. **Recommendation: nf-core-inspired,
+not nf-core-submitted** — adopt meta-maps, one-tool-per-module, subworkflows,
+`versions.yml`, `conf/modules.config`; keep `ucr_hpcc` as an institutional profile
+but add a real container path so the pipeline is portable.