docs(governance): AGENTS.md steward network

[lbliii]

Summary

Introduces an AGENTS.md steward network for NeMo Curator — a root constitution plus ten scoped, domain-specific guides that activate passively when an agent works in their directory (per the agents.md convention). The system encodes the convictions, invariants, and review hooks each domain cares about so that an agent traversing the project has the local expert's voice in scope from the moment it enters a directory.

This is not an automated reviewer that replaces humans. It's a coordination layer on top of .github/CODEOWNERS: AI stewards advise; CODEOWNERS approve. The system already produced real findings during validation (see Validation below), including a determinism bug and an embedder-import regression.

How it works

graph TB
    USER([Human])
    USER --> AGENT

    subgraph ALWAYS[Always loaded]
        ROOT["Root AGENTS.md<br/>━━━━━━━<br/>• Product + architecture pillars<br/>• Stop &amp; Ask<br/>• Inference Acceleration concerns<br/>• Known Regression Patterns<br/>• Steward Signal Format<br/>• Doc Autopilot triggers"]
    end

    AGENT["Implementing agent<br/>━━━━━━━<br/>synthesis &amp; decisions"]
    ROOT --> AGENT

    subgraph SCOPED[Scoped — activate on directory entry]
        direction TB
        S1[nemo_curator/<br/>Pipeline &amp; Stage Contract]
        S2[backends/<br/>Executor Parity]
        S3[stages/deduplication/<br/>Determinism]
        S4[stages/text/<br/>Text modality]
        S5[stages/video/<br/>All-GPU video]
        S6[stages/synthetic/<br/>SDG]
        S7[tests/<br/>Parity + coverage]
        S8[fern/<br/>Canonical docs]
        S9[benchmarking/<br/>Defensible perf]
        S10[tutorials/<br/>First impression]
    end

    AGENT -->|enters scope| SCOPED
    AGENT -.->|delegates parallel work| SWARM["Subagent swarm<br/>(returns Steward Signal Format)"]
    SWARM -.-> AGENT

    SCOPED -.->|routes review to| CO["@NVIDIA-NeMo CODEOWNERS"]

Loading

Three activation modes:

1. Passive — when an agent enters a directory, the closest scoped AGENTS.md loads automatically via the agents.md convention. Root is always loaded, so repo-wide rules (Inference Acceleration, Known Regression Patterns, Doc Autopilot) are always in scope.
2. Synthesis — the implementing agent is the integration point. Stewards advise; the agent decides. Cross-references between stewards (e.g., "apply the Inference Acceleration concerns in root AGENTS.md") are followed by the agent, not by the file system.
3. Delegation — for parallel investigation or large reads, the agent spawns subagents. Each subagent reads root plus its closest scoped file and returns findings in the Steward Signal Format. The implementing agent synthesizes and decides. Triggers: `ask stewards`, `bugbash`, `review swarm`, `steward synthesis`, `audit docs`, `content audit`.

The stewards

Path	What it owns	CODEOWNERS
AGENTS.md (root)	Constitution: pillars, Stop & Ask, Inference Acceleration, Known Regression Patterns, Doc Autopilot, Steward Swarm protocol, Done Criteria	—
nemo_curator/AGENTS.md	Task / ProcessingStage / Pipeline / Resources ABI; per-stage resource ergonomics; task-centric / map-style / fault-tolerant pillars	curator_reviewers
nemo_curator/backends/AGENTS.md	Executor parity (Xenna / Ray Actor Pool / Ray Data); streaming + auto-balancing; backpressure; runtime_env honoring	@oyilmaz-nvidia @praateekmahajan @abhinavg4 @ayushdg
nemo_curator/stages/deduplication/AGENTS.md	exact / fuzzy / semantic determinism; ID-generator stability; RAPIDS dependency reality	@ayushdg @praateekmahajan
nemo_curator/stages/text/AGENTS.md	DocumentBatch contract; published Domain & Quality classifiers as public API; embedder surface	@sarahyurick @praateekmahajan @VibhuJawa (classifiers/embedders)
nemo_curator/stages/video/AGENTS.md	All-GPU end-to-end pipeline; speed-of-light per inference model; WebDataset output	@suiyoubi @abhinavg4
nemo_curator/stages/synthetic/AGENTS.md	Prompts as public API; in-process vs server-endpoint deployment patterns; OpenAI-compat	@huvunvidia
tests/AGENTS.md	@pytest.mark.gpu discipline; shared Ray cluster fixture; modality selection rules; 80% changed-line coverage	curator_reviewers + per-modality
fern/AGENTS.md	Canonical docs site; 2026-05-20 write-freeze on docs/; agentic features (Ask AI, llms.txt, MCP); Doc Autopilot ritual	@NVIDIA-NeMo/docs_team
benchmarking/AGENTS.md	Reproducibility; hardware + software-version + model + serving-stack capture; cost-per-token framing	@rlratzel + curator_reviewers split
tutorials/AGENTS.md	Imports resolve; extras-name correctness (<modality>_cpu / <modality>_cuda12); cluster-scheduled examples	curator_reviewers (advocacy: add per-modality routing)

Cross-cutting concerns (root-owned, always active)

• Inference Acceleration. Speed-of-light per model (TensorRT-LLM, memory optimization, quantization); explicit in-process vs server-endpoint deployment pattern; vLLM canonical / Ray Serve preferred / Dynamo supported; benchmarks must capture model + serving stack + hardware.
• Known Regression Patterns with verification recipes: fabricated CLI/config fields, stage-contract drift, executor parity drift, inference performance regression, deduplication CUDA gating, docs/ vs fern/ regression, doc-snippet rot, naming/counting drift, cross-page inconsistency, narrow-fix regression, unverified finding regression.
• Convergence rule. When two or more stewards independently flag the same finding, it auto-promotes to P0 regardless of individual severity.
• Global Sweep on accepted P0s. Wrong factual claims must be grep-corrected across the entire fern/ site plus tutorials/, .cursor/rules/, .github/copilot-instructions.md, README.md, api-design.md — narrow fixes are the dominant regression mode.
• Docs-First Agent Artifact Evaluation. Before creating or expanding a cursor rule, Claude skill, MCP workflow, or prompt template, fix fern/ first if that would solve the problem for both humans and agents.

Validation

The steward network produced real findings during three swarm passes before this PR landed:

• Bootstrap audit — 12 P0 corrections to the initial mandate (fabricated paths, wrong extras names, stale class names, CODEOWNERS gap for fern/).
• Post-tightening re-audit — 5 factual errors caught after a redundancy-cut rewrite, plus 1 false-positive that exemplified the "Unverified finding regression" Known Regression Pattern.
• State-of-domain pass — first audit of code (not just the mandate). Surfaced real issues including:
  • IdGenerator.hash_files is caller-order-dependent (silent ID drift hazard) — nemo_curator/stages/deduplication/id_generator.py:47-49
  • Embedders eagerly import torch / sentence_transformers / transformers at parse time, breaking CPU-only installs — nemo_curator/stages/text/embedders/{__init__.py:15,base.py:19-23}
  • Resources._get_gpu_memory_gb() returns a hardcoded 24 GB fallback that masks misconfiguration — nemo_curator/stages/resources.py:29
  • CompositeStage.with_(dict) is LSP-incompatible with ProcessingStage.with_(name=, resources=, ...) — nemo_curator/stages/base.py:262 vs :381
  • Lazy-import drift in three modalities (dedup top-level RAPIDS, text embedders, video ffmpeg subprocess paths) — convergent finding from three independent stewards

These are tracked as follow-up work; this PR ships the network, not the fixes.

Commits in this PR

1. Bootstrap of the network and first-audit P0 corrections
2. Tightening pass to remove redundancy (-34% lines)
3. Second-audit P0/P1 corrections
4. Motivation reshape + Inference Acceleration cross-cutting concern
5. Drop redundant bootnote (the agents.md convention already says "read this file")
6. Switch scoped stewards to second person; reframe Inference Acceleration as a concern, not a separately-scoped steward
7. One steward-discovered fix to a text/modifiers/ docstring (cross-page-inconsistency pattern)

Open follow-ups

• Wire one CI trigger per domain (Doc Autopilot merge gate; tutorial static-import smoke; backend parity test; benchmark regression detection).
• Close CODEOWNERS gaps on tutorials/ and several tests/stages/<modality>/ subtrees.
• Pin owned fern/ doc paths in each scoped steward's Own list (currently deferred to first Doc Autopilot sweep).
• Fix the validated bugs above as separate PRs against their CODEOWNERS.

Test plan

• markdownlint-cli2 clean across all 11 AGENTS.md files
• No internal customer / team / number / roadmap leaks (grep-verified)
• Spot-checked 6 of the highest-impact state-of-domain claims against source
• Steward activation tested via a small edit in nemo_curator/stages/text/modifiers/modifier.py — relevant stewards activated, irrelevant stewards stayed quiet
• First real PR review through the system (requires another PR to route through it)

🤖 Generated with Claude Code

[lbliii]

Pushed three commits responding to @sarahyurick's review and integrating Praateek's developer guide:

1. `docs(governance): apply Sarah's PR review — Ray Actor Pool is dedup-only` — reframed Ray Actor Pool as the dedup-batch executor across root, backends, benchmarking, video, tutorials. Dropped `ALM_BENCHMARK.md` / `AUDIO_PROFILING.md` references. Tutorials get ruff, not CI execution.

2. `docs(governance): seed stewards with Praateek's architectural discipline` — encoded the patterns from the NeMo Curator Developer Guide into the relevant stewards. The Pipeline & Stage Contract steward now carries the `init` / `setup_on_node` / `setup` lifecycle discipline, `process` vs `process_batch` rules, `_metadata` / `stage_perf` propagation (especially fan-out), task-size sweet spot, `with()` for variable resources, `CompositeStage`, and `Workflow`. Text / Video / SDG stewards cross-reference the setup discipline for model loading. Dedup steward gains the `Workflow` callout and explicit Ray Actor Pool scope. Tests steward gains the "no `TestThisClassFunction` over-classification" rule.

3. `docs(governance): integrate Praateek's What-NOT-to-Do + new KRPs` — root `AGENTS.md` Anti-Patterns is now a Don't / Do table (shape from docs: add AGENTS.md for AI agent and contributor guidance #1769), and six new Known Regression Patterns the swarm hunts by default: stage lifecycle drift, `process_batch` mis-use, metadata/_stage_perf drop on fan-out, Ray Data spec missing (`IS_ACTOR_STAGE`, `IS_FANOUT_STAGE`), Ray Actor Pool mis-attribution, test over-classification, and `EmptyTask` first-stage worker waste. Each pattern names a verification recipe so it's machine-checkable in autopilot mode.

The `What NOT to Do` shape and several concrete rules are borrowed from #1769 — the two PRs are complementary (#1769 = root-level developer onboarding; #2005 = scoped domain network). When both land they reinforce each other.

Verifying lint clean across all 11 `AGENTS.md` files.

[greptile-apps]

Fabricated version file — latest.yml does not exist

The Protect section lists fern/versions/{latest,main,v25.09,v26.02,v26.04}.yml as the canonical set of version files, but fern/versions/latest.yml does not exist in the repository (only main.yml, v25.09.yml, v26.02.yml, and v26.04.yml are present). This is the exact pattern the root AGENTS.md defines as "Fabricated CLI / config fields" — and this steward file is the first place an agent will look when managing fern/ versioning. An agent that trusts this inventory will attempt to read, validate, or update a file that doesn't exist, then treat the missing file as a regression rather than an authoring error.

[lbliii]

Disagreeing — fern/versions/latest.yml does exist on this branch:

$ ls fern/versions/
latest.yml
main
main.yml
v25.09
v25.09.yml
v26.02
v26.02.yml
v26.04
v26.04.yml

The file is referenced as the redirect-only manifest (no matching latest/ directory, which the steward already calls out two lines later: "latest.yml is redirect-only — no latest/ directory"). This is the second "Unverified finding regression" pattern on this PR from this tool — the Known Regression Pattern is in root AGENTS.md for a reason. No code change needed.

[sarahyurick]

This seems to be referencing an internal Slack channel.

[lbliii]

Good catch — dropped the internal Slack channel name. Now reads "team's results sink" in 02f6c75.

[sarahyurick]

I thought it was CC not red pajama?

[lbliii]

The current public README headline is RedPajama v2: "16× faster fuzzy deduplication on 8 TB RedPajama v2 (1.78 trillion tokens)". Nemotron-CC is cited separately as the end-to-end recipe. To avoid pinning a specific corpus that may drift, I softened the steward to point at README first as the source of truth (02f6c75). If we're swapping the headline benchmark to Nemotron-CC, the README + the public site update first, and the steward then auto-discovers via grep.

[sarahyurick]

I know this is true for text filters and modifiers but I don't think this auto detection is true across the board right? I think you normally have to specify if you want it to be an actor (appropriate for stages that persist some type of object in memory per actor, like a tokenizer or model).

[lbliii]

Good correction. Praateek's deck described auto-routing as a general behavior; you're saying it's opportunistic in practice. Reframed in 02f6c75: auto-detection works for simpler text filter/modifier stages but isn't reliable across the board, and any stage holding a tokenizer / model / lookup table in memory should explicitly set RayStageSpecKeys.IS_ACTOR_STAGE: True in ray_stage_spec.

[sarahyurick]

Lingering comment should not group Ray Actor Pool with Xenna and Ray Data.

[lbliii]

Fixed in 02f6c75 — root now reads: "The same pipeline definition must run unchanged across the streaming executors (Xenna and Ray Data). Ray Actor Pool is the dedup-batch executor — appropriate for shuffle-based dedup and full-data-state passes, not for general streaming workloads."

[sarahyurick]

This isn't necessarily text-specific but since it is common for text, I would like us to specify that tokenization and model inference should always be split into 2 stages and defined together in a composite stage. Like in general, if we are loading some type of model object in memory, it is usually better for each model to be its own stage. This is especially true for the model forward pass which has a huge advantage when it has more GPU resources available in the pipeline, as opposed to having to share the time and resources with the tokenization step.

[lbliii]

Strong guidance — added in 02f6c75 as a new Protect bullet on the text steward:\n\n> Tokenization and model inference are always separate stages. When a pipeline needs both (classifier inference, embedder inference), split them into two stages and compose them with CompositeStage. Tokenization is CPU-bound; inference is GPU-bound. Keeping them in one stage forces the GPU stage to share its replicas with tokenization, hurting throughput. Each model loaded in memory deserves its own stage so the GPU stage can scale independently.\n\nI kept it on the text steward since that's where it activates most often, but the convention is general — happy to move it to Pipeline steward if you'd prefer it broader.

[sarahyurick]

Maybe worth mentioning, if you are adding a model test that requires a HF_TOKEN, then you will need to coordinate with the automation team to get the model configured with the token in CI/CD.

[lbliii]

Added in 02f6c75:\n\n> HF_TOKEN-gated tests require automation-team coordination. If a test loads a HuggingFace model that needs an access token, the test will fail in CI without the token configured. Coordinate with @NVIDIA-NeMo/automation to add the token (and any model-specific access grants) to the CI secrets before merging. Use pytest.skip if the token is absent locally — never pytest.fail, which hard-blocks contributors without the secret.\n\nThe steward swarm flagged three test files using pytest.fail on missing HF_TOKEN — those are tracked as follow-up fixes.

[sarahyurick]

I think small Hugging Face downloads are okay. We use them for classifier and embedder tests.

[sarahyurick]

Maybe let's remove this point. I think text/experimental/ will not always be there.