feat(l2g): add hyperparameter grid search with file-based CV output

[addramir]

Summary

Bug fix

• Fix silent bug: `set_params(**config)` was called after `fit()` in `cross_validate_single_fold`, meaning the hyperparameter grid values were never actually applied during training. All CV folds were silently trained with the base hyperparameters regardless of the sweep config.

New features

Hyperparameter grid search (`hyperparameter_grid`)
W&B-style grid (`{"param": {"values": [v1, v2, ...]}}`) now accepted by `LocusToGeneStep`, `LocusToGeneConfig`, and `LocusToGeneTrainer`. When provided without W&B, the grid is iterated explicitly using `itertools.product`. When `train_on_full_dataset=True` is also set, it is automatically ignored (with a warning) because no single winning config has been chosen yet.

File-based CV output (`cv_results_dir`)
When set (and `wandb_run_name` is not set), writes the following after all folds complete:

• `cv_results.json` — timestamp, hyperparameters, per-fold metrics, mean ± std per config
• `cv_folds.csv` — flat tabular view (one row per fold × config)
• `config_N/roc.png` — per-fold ROC curves with mean AUC in title
• `config_N/pr.png` — per-fold Precision-Recall curves with mean AP in title
• `config_N/confusion_matrix.png` — aggregated across all folds at threshold 0.5

Each config's raw arrays are released from driver memory as soon as its plots are written, so memory usage is O(n_samples × n_folds × 1 config) rather than O(n_samples × n_folds × n_configs).

GCS support: `cv_results_dir` accepts both local paths and `gs://` URIs; GCS output is written to a local temp dir then uploaded via `google-cloud-storage` (already a dependency).

Dark matter loci filter (`filter_dark_matter`)
Removes loci from the training set where every gold-standard positive has no molecular evidence linking it to the locus. A positive is considered "dark matter" when:

• All QTL colocalisation features and E2G are zero (or NULL), and VEP score is below the protein-altering threshold (0.6)
• The gene is not the nearest to the sentinel (all neighbourhood distance features < 1.0)

A locus is dropped only when ALL its positives qualify; loci with at least one signal-carrying or nearest-gene positive are kept. The entire locus (positives and negatives) is dropped to preserve class balance within retained loci. Filtering stats are written as `{model_path}_dark_matter_stats.json`.

Defaults to `False` — existing training runs are unaffected.

Refactor: `cross_validate_single_fold` extracted as `_run_cv_fold` private method; `_save_cv_results` split into `_summarise_and_plot_config` (per-config, releases raw arrays) and `_write_cv_files` (final JSON + CSV).

Behaviour matrix

`wandb_run_name`	`cv_results_dir`	`hyperparameter_grid`	Effect
set	—	any	W&B grid sweep (existing behaviour, unaffected)
unset	set	multi-value	Explicit grid iteration → files written
unset	set	None / single-value	Single config × 5 folds → files written
unset	unset	any	Terminal logging only (existing behaviour)

Test plan

• Run training locally with `cv_results_dir=/tmp/cv_test` and a small `hyperparameter_grid` (2–3 values per param), verify directory structure and file contents
• Verify `cv_results.json` contains correct mean/std across folds
• Verify plots are generated for each config subdirectory
• Run with `cv_results_dir=gs://...` on Dataproc, verify files appear in GCS
• Run with no `cv_results_dir` to confirm existing terminal-logging path is unaffected
• Run with `wandb_run_name` set to confirm existing W&B sweep path is unaffected
• Enable `filter_dark_matter=True` and verify stats JSON is written next to model path

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR extends the L2G training pipeline to support non-W&B hyperparameter grid search with persisted cross-validation artifacts, and adds an optional training-time “dark matter loci” filter for gold-standard feature matrices.

Changes:

• Fix CV hyperparameters not being applied during fold training by setting params before fit(), and refactor per-fold logic into a private helper.
• Add explicit (non-W&B) hyperparameter grid iteration with optional file/GCS output of CV metrics, CSV summaries, and diagnostic plots.
• Add filter_dark_matter option + implementation to drop loci where all positives lack functional signal and are not nearest-gene, plus stats output and unit tests.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/gentropy/dataset/test_l2g_feature_matrix.py	Adds unit tests covering dark matter locus filtering behavior and edge cases.
src/gentropy/method/l2g/trainer.py	Fixes CV param application timing; adds explicit grid expansion + file/GCS CV outputs and plotting helpers.
src/gentropy/l2g.py	Wires new training options (hyperparameter_grid, cv_results_dir, filter_dark_matter) into the step and writes dark-matter stats.
src/gentropy/dataset/l2g_feature_matrix.py	Implements filter_dark_matter_loci() with stats and logging.
src/gentropy/config.py	Exposes new configuration parameters for the step.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.