test(evals): cover GitHub connector repo pagination and marketplace skill import end-to-end - #3529
Conversation
…kill import end-to-end Adds a deterministic GitHub provider witness (labs mock-github) and a stack spec that cold-boots Den, connects a 130-repository installation, and imports a private Claude marketplace repo through the connector: - all 130 repositories listed across cursor pages (regression: first 30 only before #3524), provider paged at per_page=100 pages 1-2, installation-token identity asserted per request - discovery + apply materializes the probe skill with frontmatter- derived title/description (regression: invalid_skill_frontmatter on every apply before #3523), companion markdown excluded, stored source byte-identical after canonical whitespace trim, re-apply idempotent with no content refetch den-api gains a lazy GITHUB_CONNECTOR_API_BASE override so specs can point the connector at the witness; production default is unchanged.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Photo roll — the-github-connector-lists-every-installation-repository-and-imports-marketplace — 0/0 frames passed · 8 facts✅ 0/0 frames passed · 8 facts · 8 expectations passed · 0 failed ℹ️ FACT — 1. The repository picker returns all 130 installation repositories across cursor pagesThe first page contained 100, the second contained 30, and the concatenated IDs were 1 through 130 in provider order (regression: only the first 30 before #3524).
ℹ️ FACT — 2. GitHub repository enumeration requests provider pages 1 and 2 at 100 items per pageProvider pagination requests were [{"page":"1","perPage":"100"},{"page":"2","perPage":"100"},{"page":"1","perPage":"100"},{"page":"2","perPage":"100"}]; there was no unpaged request or page 3 request.
ℹ️ FACT — 3. Repository discovery uses the GitHub installation access token522 repository and content requests carried Bearer mock-installation-token-777, not the app JWT.
ℹ️ FACT — 4. Discovery classifies the private marketplace repository with one supported plugin and no warningsClassification was claude_marketplace_repo; plugin support was true; discovery and plugin warnings were empty.
ℹ️ FACT — 5. Applying the marketplace import materializes the frontmatter-derived probe skillApply returned HTTP 200; the only materialized object was probe-plugin/skills/probe/SKILL.md titled probe (regression: invalid_skill_frontmatter before #3523).
ℹ️ FACT — 6. The companion markdown next to the skill is not importedThe repository tree included references/notes.md, while apply materialized 1 config object at probe-plugin/skills/probe/SKILL.md.
ℹ️ FACT — 7. The imported SKILL.md keeps frontmatter and body byte-for-byte after canonical whitespace trimmingThe stored version strictly equaled the provider source after surrounding-whitespace canonicalization; normalizeOptionalString trims every config-object version, including manual creates.
ℹ️ FACT — 8. Re-applying the same revision creates no duplicate version and refetches no file contentBoth applies returned config object cob_01kz83xkj5enetbf0xh732yxwr; versions remained 1; probe-plugin/skills/probe/SKILL.md was fetched 1 time since 2026-08-05T04:46:44.774Z.
Roll created 2026-08-05T04:46:25.773Z · Source: |
![diff-warden[bot]](https://avatars.githubusercontent.com/u/118544361?s=60&v=4){kind=small}
There was a problem hiding this comment.
Warden security clearance: clear. No new security issues found in this diff (d70ee25d4cd92c90f19bc03b10667106432919b5). Automated clearance satisfies the required-review gate only — a human still reviews and merges. Analysis run
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
Warden security clearance: clear. No new security issues found in this diff (133dc458fb57eca1f4a0595d483786e94242f601). Automated clearance satisfies the required-review gate only — a human still reviews and merges. Analysis run
…econciliation, renames (#3536) * test(evals): cover GitHub connector repo pagination and marketplace skill import end-to-end Adds a deterministic GitHub provider witness (labs mock-github) and a stack spec that cold-boots Den, connects a 130-repository installation, and imports a private Claude marketplace repo through the connector: - all 130 repositories listed across cursor pages (regression: first 30 only before #3524), provider paged at per_page=100 pages 1-2, installation-token identity asserted per request - discovery + apply materializes the probe skill with frontmatter- derived title/description (regression: invalid_skill_frontmatter on every apply before #3523), companion markdown excluded, stored source byte-identical after canonical whitespace trim, re-apply idempotent with no content refetch den-api gains a lazy GITHUB_CONNECTOR_API_BASE override so specs can point the connector at the witness; production default is unchanged. * feat(den-api): self-healing GitHub connector sync — queue, retries, reconciliation, renames Webhook deliveries now ack in-request (202) and enqueue connector sync events; a background worker (workers/github-sync.ts) claims them with per-target serialization and optimistic row guards, retries transient provider failures with jittered exponential backoff (attempt_count / next_attempt_at columns), and dead-letters after max attempts. The sync-event Retry action re-enqueues into the same worker instead of flipping a row nothing consumed. A reconciliation sweep compares each active target's branch head to the last completed sync and enqueues a manual_resync on drift, healing lost webhook deliveries; POST /v1/connector-instances/:id/sync-now (and a den-web Sync now button) enqueues the same check on demand. repository.renamed webhooks now follow the rename by repositoryId (previously sync silently died on rename); installation_repositories removals mark the instance disabled with a recorded event. Also fixes a pre-existing production bug this surfaced: connector automation contexts (session: null) tripped the interactive fresh- privileged-session reauth guard, so webhook-driven auto-import always failed with a reauth error. Automation contexts now carry an explicit automation flag that only server-side builders can set. Spec: evals/specs/github-sync-self-healing.slow.test.ts encodes the approved six-frame demo against the mock-github witness (now with fault injection, head advancement, and renames). Fixes #3530 Fixes #3531 Fixes #3532 * fix(evals): do not echo handler errors into mock-github HTTP responses CodeQL flagged js/stack-trace-exposure on the witness's catch-all (code scanning alert #202, blocking PR #3529). Log the detail server-side and return a generic 500 body instead.
2 participants
Follow-up coverage for #3523 (fixes #3521) and #3524 (fixes #3522), per the repo validation standard (
evals/specs/**+@openwork/testkit).What this adds
evals/packages/labs/src/mock-github.ts— a deterministic GitHub provider witness (node:http), following the mock-google pattern: installation token minting,/installation/repositorieswith real GitHub paging semantics (default per_page=30) so the pagination regression stays observable, repo/branch/commit/tree/contents endpoints with deterministic content-hash shas, and a request log capturing method/path/query/ISO-time/authorization for per-identity witness assertions.evals/specs/github-connector-import.slow.test.ts— a stack spec that cold-boots a real Den (server(), ephemeral MySQL DB), points the connector at the witness, and drives the org API end-to-end: create connector account (installation 777, 130 repositories) → list repositories → GitHub setup on a private Claude marketplace repo (branchdevelop, mirroring the original user report) → discovery → apply → config-object versions → re-apply.ee/apps/den-api/.../github-app.ts— a lazyGITHUB_CONNECTOR_API_BASEenv override (4 lines) so specs can aim the connector client at a witness. Read per-request, no env-module import (unit tests import this module bare), production default unchanged.Claims (all hold in the tape, every one with an observable assertion)
per_page=100— no unpaged call, no page 3.title: probe/ description (regression:invalid_skill_frontmatteron every apply before fix(den-api): keep SKILL.md frontmatter when deriving GitHub connector skill projections #3523).normalizeOptionalStringtrims every config-object version, including manual creates — product behavior, deliberately not changed here.Commands run
pnpm exec vitest run --config vitest.config.ts --project stack specs/github-connector-import.slow.test.ts(fromevals/, Docker MySQL on 3306) → 1 passed, ~20s, three consecutive fresh-DB cold-boot runs plus one at this exact HEAD; zero skips.bun test ee/apps/den-api/test/github-connector-app.test.ts→ 8 pass (seam did not disturb unit tests).tsc -p .inevals/→ the only error is pre-existing on dev in untouchedconnectors-quick-add.slow.test.ts(MockMcpHandle.handshakes); the new files are clean.Evidence tape publishes as the sticky PR comment (ambient testkit tape; screenshots don't apply — API-driven spec, verdict comes from the tape facts).