Skip to content

[codex] add deterministic mock E2E flow#179

Draft
AnkanMisra wants to merge 1 commit into
mainfrom
codex/deterministic-mock-e2e
Draft

[codex] add deterministic mock E2E flow#179
AnkanMisra wants to merge 1 commit into
mainfrom
codex/deterministic-mock-e2e

Conversation

@AnkanMisra
Copy link
Copy Markdown
Owner

Summary

  • Add deterministic AI_PROVIDER=mock support for local and CI payment-flow runs.
  • Default bun run stack and bun run test:e2e to the mock provider so the 402 challenge/sign/retry/receipt path runs without OpenRouter, Ollama, Redis, or secrets.
  • Update E2E, CI, and docs around the deterministic flow, plus add an SDK receipt verification example.

Why

The E2E workflow should exercise the payment flow consistently in CI instead of depending on live model-provider credentials. Docs now also avoid implying Redis-backed verifier nonce storage exists today; verifier replay protection remains single-process memory until a shared nonce store is implemented.

CI fix

The E2E workflow dependency prep now runs Go and Rust commands from their service directories:

  • cd gateway && go mod download
  • cd verifier && cargo fetch

Validation

  • git diff --check
  • cd gateway && go test -v ./...
  • cd gateway && go vet ./...
  • cd sdk/typescript && bun run typecheck
  • cd sdk/typescript && bun run test
  • bun run test:e2e
  • (cd gateway && go mod download) && (cd verifier && cargo fetch)

@vercel
Copy link
Copy Markdown

vercel Bot commented May 27, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
microai-paygate Ready Ready Preview, Comment May 27, 2026 12:21pm

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 27, 2026

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 5e101cf1-41ec-4fac-bfcd-dea86a0c6908

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/deterministic-mock-e2e

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Co-authored-by: codex <codex@users.noreply.github.com>
@AnkanMisra
Copy link
Copy Markdown
Owner Author

@codex please review this PR deeply.

Focus on the current diff against main at head 99c93452a957edf4bb4554c747ba5a951dea48aa.

Scope:

  • deterministic AI_PROVIDER=mock gateway provider and readiness behavior
  • E2E workflow/script changes, including no OpenRouter/Redis/secrets dependency
  • Docker Compose env wiring and local stack behavior
  • gateway config/readiness tests and test isolation under ambient AI_PROVIDER=mock
  • TypeScript SDK receipt verification example
  • docs/config parity across README, gateway README, tests README, .env.example, Compose, and CI

Please look for correctness, security, CI reliability, docs/config drift, test gaps, and cross-service contract mismatches. Prioritize payment-flow details: EIP-712 parity, nonce/timestamp replay behavior, receipt signing/verification, cache/receipt storage behavior, env-var defaults, and secret handling.

Return severity-ordered findings using:

  • P0 critical
  • P1 important bug/security/CI break
  • P2 maintainability/test/docs drift
  • P3 optional polish

Include file/line references and a final verdict: approve, approve with comments, or request changes.

@chatgpt-codex-connector
Copy link
Copy Markdown
Contributor

💡 Codex Review

case "mock":
return NewMockProvider(), nil

P2 Badge Include provider in cache keys for mock

When AI_PROVIDER=mock is enabled together with CACHE_ENABLED=true, the response cache still keys non-Ollama requests as the OpenRouter model default (gateway/cache.go currently only special-cases ollama). This means a Redis cache populated under OpenRouter can be served while running the mock provider, or mock summaries can later be served under OpenRouter for the same text/model default, even though the active provider changed. Please add a mock/provider discriminator to the cache key selection before enabling the new provider.


- name: Run E2E
run: |
chmod +x ./run_e2e.sh
timeout 150 ./run_e2e.sh

P2 Badge Give E2E enough time for cold Rust builds

Now that the OpenRouter-secret skip is removed, every PR path that hits this job runs timeout 150 ./run_e2e.sh, but the script performs a fresh cargo build --quiet for the verifier before starting services (run_e2e.sh:54-55). GNU timeout --help confirms it kills the command when it is still running after the duration; in a cold CI runner this can terminate during dependency compilation rather than testing the deterministic mock flow. Please either lengthen the timeout substantially or move/cache the build outside the timed section.


- AI_PROVIDER=${AI_PROVIDER:-mock}

P2 Badge Pass Ollama settings through Compose

Adding AI_PROVIDER to the Compose gateway environment makes AI_PROVIDER=ollama selectable in local stacks, but the service still does not pass OLLAMA_URL or OLLAMA_MODEL while NewOllamaProvider reads those variables and otherwise defaults to http://localhost:11434. Inside the gateway container, that localhost is the container itself, so a Compose user setting AI_PROVIDER=ollama in .env will get readiness/request failures even if they also set the Ollama URL. Please forward the Ollama env vars (and document the host/service URL to use) alongside the existing OpenRouter pass-throughs.


expect(res.status).toBe(200);
const data = await res.json() as any;
expect(data.result).toStartWith("Mock summary:");

P2 Badge Keep E2E assertions aligned with provider overrides

run_e2e.sh still honors an ambient AI_PROVIDER override (AI_PROVIDER=${AI_PROVIDER:-mock}), and tests/README.md documents overriding it, but the test now always requires the signed response to start with Mock summary:. A successful AI_PROVIDER=openrouter or AI_PROVIDER=ollama run will fail here after payment verification succeeds simply because the provider returned real model text. Either force the script to use mock for this deterministic suite or make the assertion conditional and update the docs.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant