Run on Nextflow >=26.04 (strict syntax) - #128
Open
hoelzer wants to merge 13 commits into
Open
Conversation
Swap the short-read mapper alternative behind --bwa from bwa to bwa-mem2, mainly for the faster index build of large indices combining several eukaryotic genomes. The --bwa parameter and all result files/names are unchanged. - rename modules/bwa.nf -> modules/bwamem2.nf (bwa_index/bwa -> bwamem2_index/bwamem2, labels bwamem2_index/bwamem2) - new container nanozoo/bwa-mem2:2.3--4533398 - envs/bwamem2.yaml with bwa-mem2 2.3 and samtools/htslib 1.24 - separate label for the index step: it is single-threaded, so it no longer reserves 24 CPUs, but it gets more memory than the mapping step - a bwa-mem2 index is ~4x larger than a bwa index and is held in memory, so the memory requests in node.config went up - add the mapper to conda.config (it had no environment with the conda/mamba profiles) and to local.config (it had no cpus/memory with the local/standard profiles) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Update all containers and conda environments to the same samtools/htslib version, and pin htslib explicitly so bgzip/tabix are no longer pulled in only as a transitive samtools dependency. - containers: samtools 1.24, minimap2 2.31, bed_samtools 2.31.1, bwa-mem2 2.3 (now with samtools), seqkit 2.13.0 - envs follow the container versions: minimap2 2.26 -> 2.31, bedtools 2.30.0 -> 2.31.1, seqkit 2.6.1 -> 2.13.0, samtools -> 1.24 - envs/minimap2.yaml: pigz 2.3.4 -> 2.8, the old pin cannot be solved next to samtools 1.24 (zlib 1.2 vs libzlib 1.3.2) - envs/seqkit.yaml: tabix 1.11 -> htslib 1.24, which ships tabix and bgzip - samclipy gets an explicit container instead of implicitly taking the one of the smallTask label; the samtools image has the python and git it needs. Its env now pins python 3.14 (as in the container) and adds git, which the process calls but the env never provided. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The CI matrix uses NXF_VER "latest", which now resolves to Nextflow 26.04,
where the strict syntax is the default. That broke all "latest" jobs, on
this branch as well as on main.
Fixed here:
- nextflow.config: `def trace_timestamp` is a variable declaration, which
cannot be mixed with config statements -> moved into the params scope
- clean.nf: typed `for` loops are not part of the syntax -> use `each`
- clean.nf: `addParams()` on include statements was removed. `tool` was
never read anywhere, `lib_pairedness` is now set in the params scope,
which is where the modules read it from anyway
- modules: `if` blocks around publishDir/storeDir are not valid process
directives -> use the `enabled` option of publishDir, and a ternary for
the storeDir of download_host
- modules/utils.nf: `env(TOTALRECORDS)` refers to a shell variable, the
strict parser wants the string form `env('TOTALRECORDS')`
Note that `enabled` needs a real boolean: `enabled: params.keep` silently
publishes nothing, because Nextflow runs a String through
Boolean.parseBoolean().
This is not enough for Nextflow >=26.04 -- the ~180 top-level statements in
clean.nf have to move into the entry workflow first. Until then the CI
matrix is pinned to 25.10.0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The illumina conda job fails reproducibly while creating environments, with either `LockError: Failed to acquire lock` or a package archive that another process removed mid-extraction. Both come from several `conda env create` calls running at the same time against the shared package cache. - executor.queueSize = 1 for the test config, so environments are built one after another. The tasks are stubs, there is nothing to parallelize. - point conda.cacheDir at one shared directory. It defaults to a path relative to the launch dir, and nf-test gives every test its own launch dir, so each of the six illumina tests rebuilt every environment from scratch. Now they are built once and reused. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Closes #125. Nextflow >=26.04 parses scripts with the strict syntax, where a file may only contain declarations. clean.nf mixed ~180 lines of top-level statements with its include and workflow declarations, which `nextflow lint` reported as 140 errors and which made the pipeline fail to compile. clean.nf: - parameter validation moved into check_for_unknown_parameters() and validate_parameters(), the run information into print_run_info(), and the input channels into the entry workflow plus input_channel(), fasta_channel() and control_fasta(). The order of the checks and prints is unchanged. - `exit 1, "..."` -> `error "..."` and `exit 0, helpMSG()` -> log the help and return, `exit` is not part of the syntax anymore. helpMSG() returns the message instead of logging it, and its colour variables are declared. Directives cannot reference process inputs anymore: - modules/bbmap.nf: the publishDir path with ${map_target} is now a closure, which is still resolved lazily - modules/alignment_processing.nf: `pattern` takes no closure, so the two "${name}*.bam{,.bai}" patterns are static globs now. For the strict DCS filter that would newly publish dcs.bam, so saveAs skips it. params.lib_pairedness cannot be assigned from the entry workflow (the linter flags writes to the params scope), so modules/functions.nf provides it as lib_pairedness(), which the process scripts call. Verified on Nextflow 23.04.1, 25.10.0 and 26.04.6: 18 stub runs across all input types, and 4 real runs (--min_clip, --dcs_strict, --bbduk, --keep) whose published files are byte-identical to the previous implementation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
check_own always wrote checked.fa.gz, so more than one --own (or --keep) FASTA made concat_contamination fail with an input file name collision. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An --input glob that matches files but forms no read pairs left every process without a task, so the run reported success without cleaning anything. checkIfExists does not catch this. Also give bwamem2_index 8 cpus: the build is not as serial as assumed and uses ~10 cores, so a cgroup-pinned single core would slow it down badly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
minimap2 was handed the reference FASTA directly, so every sample rebuilt the index; for a multi-genome host that dominates the runtime. Add a minimap2_index process mirroring bwamem2_index. -k and -w are baked into the index, so index and mapping have to agree on the preset: minimap2_preset() is now the single source for both, which also collapses the four near-identical mapping commands into one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nothing in the config declared a time, so every task ran under the default limit of the queue. Mapping a large sample against a multi-genome host exceeded that and SLURM killed the job with exit 140; the retry ladder then only raised the memory, which does not buy more wall time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
index_bam, idxstats_from_bam and flagstats_from_bam ran under the minimap2 label, so each of them asked for 24 cpus, 24 GB and now also a 12 h wall time for a few minutes of work. On a backfill scheduler that delays them behind the mapping jobs they follow. They get the samtools container that the smallTask label already uses and a matching small Conda environment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #125.
Important
Stacked on #124 — it contains the smaller strict-syntax fixes this builds on. Merge #124 first, then this PR shows only its own commit.
Problem
Nextflow >=26.04 parses scripts with the strict syntax, where a script file may only contain declarations.
clean.nfmixed ~180 lines of top-level statements with itsincludeandworkflowdeclarations, so the pipeline did not compile at all:nextflow lintreported 140 errors, all inclean.nf.Changes
clean.nf
check_for_unknown_parameters()andvalidate_parameters()print_run_info()input_channel(),fasta_channel()andcontrol_fasta()for the branching partsexit 1, "..."→error "...", andexit 0, helpMSG()→ log the help andreturn;exitis no longer part of the syntax.helpMSG()returns the message instead of logging it, and its colour variables are declared withdef.The order of checks and prints is deliberately unchanged, so
--helpstill works without the otherwise required input parameters.Directives can no longer reference process inputs
This one is a runtime failure rather than a parse error, so it only showed up when actually running on 26.04 (
ERROR ~ No such variable: map_target):modules/bbmap.nf: thepublishDirpath containing${map_target}is now a closure, which is still resolved lazilymodules/alignment_processing.nf:patternaccepts no closure, so the two"${name}*.bam{,.bai}"patterns became static globs. Forfilter_true_dcs_alignmentsthat would newly publishdcs.bam, sosaveAsfilters it out — closures do get the process inputs.lib_pairedness
params.lib_pairednesswas assigned at the top level. Assigning it from the entry workflow is flagged (Params should be declared at the top-level), somodules/functions.nfprovideslib_pairedness()and the process scripts call it.Result
nextflow linton Nextflow 26.04.6: 140 errors → 0.The CI matrix goes back to
latest.Verification
--bwa,--bbduk— on Nextflow 23.04.1, 25.10.0 and 26.04.6--min_clip(soft-clipped publishing),--dcs_strict(strict-DCS publishing),--bbduk(themap_targetpath) and--keepdcs.bamis still not published.--helpexits 0 with the full message, and all four validation errors exit 1 with their original message, on both 23.04.1 and 26.04.6Not included
34 lint warnings remain (mostly
Channel.→channel.and implicit closure parameters). They are not fatal and touch every workflow file, so they would drown out this diff.