Make Govbot a DAG - #73
Draft
sartaj wants to merge 1 commit into
Draft
Conversation
Reshape govbot around a data-driven DAG: a govbot.yml manifest declares
datasets, transforms, publishers, and named pipelines, and `govbot run`
walks a pipeline by spawning each stage as a subprocess over the stream
protocol (newline-JSON, stable `id`, typed `kind`).
The built-in tagger is decoupled into an ordinary transform node:
`govbot source --select docs | govbot classify | govbot apply`. Swapping
the classifier (e.g. to `fastclass classify -`) is a one-line manifest
`command` change — no code change. classify scores the `docs` projection's
text via new `TagMatcher::match_text` / `match_tags_keywords_text` and is
offline-first (keyword fallback, never downloads mid-pipeline). apply is
the classification sink; both produce the same `.tag.json` format.
Details:
- config.rs: additive `Manifest` (uniform `Transform { command, reads,
writes }` — no classify-specific fields; `Publisher` map; `pipelines`).
Legacy `repos:`/`tags:` still parse; unknown keys ignored. 4 unit tests.
- schemas/govbot.schema.json: DAG schema (datasets/transforms/publish/
pipelines); `required` relaxed for back-compat.
- main.rs: `source --select docs` projection; new `classify`/`apply`/`run`
commands; subcommand rename Clone/Logs/Build -> Pull/Source/Publish with
old names kept as aliases; removed the deprecated `tag` command (split
into classify+apply). Bare `govbot` runs the DAG when govbot.yml exists,
wizard/init otherwise.
- pipeline.rs: data-driven runner (linear walk of a DAG-capable manifest).
- wizard.rs: `govbot init` scaffolds a DAG manifest (datasets + classify
transform + default pipeline).
- publish.rs: `get_repos_from_config` reads `datasets:` then `repos:`.
- action.yml/justfile: migrated to the DAG (source|classify|apply, publish).
Publisher stages currently emit via the existing `build` logic (minimal
wiring); a per-publisher module split is a follow-up. Includes a crate
`cargo fmt` normalization pass.
20 tests pass; verified end-to-end on the wy mock (`govbot run` produces
tag files + feed.xml).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014MAr5v6uw35YENW4HCCxyQ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reshapes govbot around a data-driven DAG. A
govbot.ymlmanifest declaresdatasets,transforms,publishers, and namedpipelines;govbot runwalks a pipeline by spawning each stage as a subprocess over the stream protocol (newline-JSON, stableid, typedkind).The built-in tagger is decoupled into an ordinary transform node:
Swapping the classifier is a one-line manifest
commandchange, no code change. This is the first slice of decoupling the "Govbot 2" mega-PR (#35); the fastclass switch itself is deferred indefinitely and becomes a future manifest edit.What's here
Manifest(uniformTransform { command, reads, writes }— no classify-specific fields),Publishermap,pipelines. Legacyrepos:/tags:still parse; unknown keys ignored. DAG JSON schema. 4 new unit tests.govbot classify— the built-in ONNX tagger as a stream transform; scores thedocsprojection text (TagMatcher::match_text); offline-first (keyword fallback, never downloads mid-run).govbot apply— the classification sink; same.tag.jsonformat.govbot run— data-driven runner:source | classify | apply, then publishers. Baregovbotruns the DAG whengovbot.ymlexists, wizard/init otherwise.source --select docs— emits{id, text, kind:"docs", sources}.Clone/Logs/Build → Pull/Source/Publish(old names kept as aliases); removed the deprecatedtagcommand (split into classify+apply).govbot initwrites a DAG manifest.action.ymlandjustfilemoved to the DAG.Notes for review
buildlogic (minimal wiring per plan); a per-publisher module split + Bluesky is a follow-up PR.cargo fmtnormalization pass (CI doesn't gate on fmt andmainwasn't fmt-clean). Happy to split it into its own commit if preferred.Verification
cd actions/govbot && cargo test→ 20 tests pass. Verified end-to-end on the wy mock:govbot runproduces.tag.jsonfiles +feed.xml, and swapping theclassifytransform'scommandrequires zero code change.🤖 Generated with Claude Code
https://claude.ai/code/session_014MAr5v6uw35YENW4HCCxyQ