feat(build): aggregate docs from source repos (pilot: spectrum-ts)

[Tom Tang (qwerzl)]

Closes ENG-1742 · part of ENG-1741.

What & why

Boss wants docs authored in each SDK repo, next to the code, instead of all in this repo. This makes the docs repo an aggregator: it pulls each repo's docs, renders with vellum, merges nav, and ships to Mintlify via the existing dist flow. Pilot scope: spectrum-ts.

How

• scripts/sync-docs assembles .vellum-src/ (gitignored) from local docs-src/ (areas not yet migrated) + each source in scripts/sources.json. Dual-mode per source: git sparse-checkout of docs-site/ at the tag matching the installed package version (so prose stays aligned with the types vellum extracts), with a local fallback so the build stays offline/working until a repo actually has its docs.
• scripts/build-nav generates docs.json from docs.base.json (committed site skeleton) + each source's nav.json fragment, replacing {$source, group} markers in place (order-preserving).
• vellum.config.ts → templates: '.vellum-src'; docs:generate = sync → build-nav → vellum → llms; docs.json is now generated (untracked); deploy-dist.yml force-adds docs.json and derives the mdx list from .vellum-src.

Verification

• All 44 spectrum-ts/**/*.mdx render byte-identical to before; all 7 llms*.txt identical; docs.json semantically identical (only diff: generator normalized a pre-existing indentation glitch in contextual.options).
• pnpm lint + pnpm typecheck:docs pass (also via the pre-commit hook).
• Git transport validated against the real remote: resolved v5.2.0, cloned, sparse-checked-out, fell back to local.

Note / decision

docs/ is already used in spectrum-ts for internal dev notes, so the source-repo templates dir is docs-site/ (configurable per-source via docsDir).

Follow-up (ENG-1743, after this merges)

Add docs-site/ + nav.json to the spectrum-ts repo, then flip the manifest to git-only (drop local, delete docs-src/spectrum-ts/) and widen the deploy-dist app token to read spectrum-ts. The token is already wired through as GITHUB_TOKEN.

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

Summary by CodeRabbit

• New Features

  • Documentation builds now support pulling content from multiple documentation sources into one site.
  • Added clearer sectioned navigation for Spectrum docs, including grouped provider pages.

• Bug Fixes

  • Improved documentation generation so source-specific content is authenticated and synced more reliably.
  • Fixed generated docs handling so build artifacts are consistently produced and updated.

• Documentation

  • Expanded the README with clearer guidance on where docs live and how to edit source files.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: e7a7a2bd-279d-4748-b227-1f67ed6141a9

📥 Commits

Reviewing files that changed from the base of the PR and between d7b20ef and bfe173e.

📒 Files selected for processing (10)

• .github/workflows/deploy-dist.yml
• .gitignore
• README.md
• docs-src/spectrum-ts/nav.json
• docs.base.json
• package.json
• scripts/build-nav/index.ts
• scripts/sources.json
• scripts/sync-docs/index.ts
• vellum.config.ts

Disabled knowledge base sources:

• Linear integration is disabled

You can enable these sources in your CodeRabbit configuration.

---

📝 Walkthrough

Walkthrough

Adds a docs synchronization script that stages local and source-repo content into .vellum-src, generates docs.json from source nav fragments and docs.base.json, and updates the generation, deploy, ignore, and documentation files around that pipeline.

Changes

Docs build pipeline

Layer / File(s)	Summary
Source registry and Spectrum nav scripts/sources.json, docs-src/spectrum-ts/nav.json	Adds the source list for docs aggregation and a grouped nav manifest for spectrum-ts.
Source resolution and fetch scripts/sync-docs/index.ts	Adds ref resolution, authenticated clone URL construction, sparse fetch, and local-versus-git source selection.
Staging tree assembly scripts/sync-docs/index.ts, vellum.config.ts	Builds .vellum-src from local templates and per-source docs, copies nav fragments into staging, and points Vellum at the staged templates directory.
Nav merge and build wiring scripts/build-nav/index.ts, docs.base.json, package.json	Replaces Spectrum page lists with $source markers, merges staged nav fragments into docs.json, and updates docs:generate to run sync and nav build steps first.
Deploy workflow and generated artifacts .github/workflows/deploy-dist.yml, .gitignore, README.md	Passes the app token into docs generation, force-adds .vellum-src outputs during dist commits, ignores generated docs artifacts, and updates the docs workflow notes.

Sequence Diagram(s)

sequenceDiagram
  participant DeployWorkflow as "deploy-dist.yml"
  participant DocsGenerate as "docs:generate"
  participant SyncDocs as "scripts/sync-docs/index.ts"
  participant BuildNav as "scripts/build-nav/index.ts"
  participant VellumBuild as "vellum build"
  participant LLMsGenerator as "scripts/llms-generator/index.ts"

  DeployWorkflow->>DocsGenerate: sets GITHUB_TOKEN and runs docs generation
  DocsGenerate->>SyncDocs: pnpm docs:sync
  SyncDocs-->>DocsGenerate: writes .vellum-src and .nav fragments
  DocsGenerate->>BuildNav: tsx scripts/build-nav/index.ts
  BuildNav-->>DocsGenerate: writes docs.json
  DocsGenerate->>VellumBuild: vellum build from .vellum-src
  DocsGenerate->>LLMsGenerator: tsx scripts/llms-generator/index.ts

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Poem

Hop hop, I stitched the docs in place,
From .vellum-src the pages race.
New navs bloom bright with a twitch of nose,
And happy paws applaud where the build flows. 🐇

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch tom/eng-1742-docs-repo-build-the-aggregator-sync-docs-build-nav-generated

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

ESLint install failed: private package registry requires authentication. Disable ESLint in CodeRabbit settings or use public packages.

---

_{Comment @coderabbitai help to get the list of available commands.}

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
Photon	🟡 Building	–	Jun 26, 2026, 3:08 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.