* Fix Darkbloom analytics tracking
* Harden release workflow protections (#103)
* Harden release registration and binary hash policy (#99)
* Harden release registration and binary hash policy
* derive release download URL from allowlist
* Stabilize provider coordinator test
---------
Co-authored-by: Gajesh Naik <26431906+Gajesh2007@users.noreply.github.com>
* Remove stale Python integration test (#109)
* e2e: add local simulation environment skeleton
Introduces scripts/e2e-runner.py, a Python orchestrator that spins up the
real coordinator binary with test-friendly configuration (in-memory store,
mock billing, no trust requirements) alongside a simulated or real
provider, and runs HTTP/WebSocket-level assertions against the live stack.
Key components:
- Coordinator class: builds and spawns coordinator with EIGENINFERENCE_MIN_TRUST=none,
EIGENINFERENCE_BILLING_MOCK=true, and in-memory store
- SimulatedProvider: pure-Python WebSocket client speaking the full provider protocol
(register, attestation challenge/response, heartbeat, inference request/response)
- Test framework: decorator-based test registration, pass/fail summary, signal-safe
cleanup via atexit + signal handlers
- Test stubs: test_basic (registration + discovery), test_inference (consumer
request routing), test_multi_provider (two providers, same model)
TODO:
- RealProvider wrapper around darkbloom serve --coordinator
- Coordination between provider challenge cycle and consumer request timing
- API key handling for consumer vs admin routes
- Python dependency management (websockets, cryptography)
* Revert "e2e: add local simulation environment skeleton"
This reverts commit d02074e. The Python E2E runner adds noise on top of
the existing Go integration tests (internal/api/integration_test.go +
fullstack_integration_test.go) which already cover the full coordinator
protocol surface. The cross-language orchestration doesn't buy anything
over what httptest.Server + simulated providers already provide.
* Remove stale Python integration test
@ethenotethan
tests/integration_test.py is superseded by the Go-based coordinator
integration tests at coordinator/internal/api/:
- Test coverage for coordinator protocol (register, challenge, heartbeat,
inference) is covered by integration_test.go using httptest.Server +
Go simulated providers — same coverage, no binary build needed
- Full-stack GPU inference is covered by fullstack_integration_test.go
with real vllm-mlx backends (gated behind LIVE_FULLSTACK_TEST=1)
- The Python test uses stale binary names ('eigeninference-provider'),
old flags ('--backend mlx-lm'), and predates attestation challenges,
E2E encryption, and the vllm-mlx backend migration
- No external dependency coverage (Postgres, Stripe, etc.) is lost — the
coordinator main.go wiring for those is trivially tested elsewhere
- The Python SDK tests (4.5.x) belong in the SDK repo, not the infra repo
---------
Co-authored-by: Hank Bob <hankbob@researchoors.com>
* chore: remove unused dependencies (#112)
* chore: remove unused dependencies
* test: fix console ui test isolation
* chore: prune repo-wide dead code findings
* ci: run CI on any PR, not just master/main (#119)
* ci: remove racing deploy-dev-coordinator workflow (#137)
Cloud Build (deploy/gcp/cloudbuild.yaml) already deploys the coordinator
on the same trigger (push to master touching coordinator/** or deploy/gcp/**).
Having both paths active creates a race condition where two CI systems
simultaneously deploy to the same dev VM — see #115.
* feat: add Datadog observability stack for dev coordinator
Install Datadog Agent on the dev GCE VM (DogStatsD, APM, journald logs)
and wire the coordinator to emit structured metrics, split attestation
counters, model_type tags, reactive provider-count gauges, and a
completion-tokens counter. Rebuild the dev dashboard with 7 sections
covering metrics, logs, traces, and system health.
* fix: prevent double-decrement when untrusted provider disconnects
Disconnect now checks StatusUntrusted before decrementing the online
counter and model-provider gauges, since MarkUntrusted already
decremented them.
* feat: add fleet version and binary hash observability
New metrics:
- providers.per_version gauge (per provider binary version)
- providers.per_binary_hash gauge (per attested binary hash)
- coordinator.min_provider_version_set gauge (1 when configured)
- provider_version_below_minimum counter (tagged by gate and version)
Gates instrumented:
- registration (provider.go)
- challenge revalidation (provider.go)
- manifest sync (server.go)
Registry additions:
- ProviderCountByVersion()
- ProviderCountByBinaryHash()
Dashboard: Fleet Version & Binary Hash group with providers by version,
providers by binary hash, min provider version, below-minimum events,
and top binary hashes toplist.
* fix: update Dockerfile + cloudbuild for go.mod at repo root
go.mod moved from coordinator/ to repo root during the swift-provider
merge. Build context is now repo root, Dockerfile copies coordinator/
subdir explicitly.
* fix: chmod +x coordinator binary in Dockerfile
* fix: ensure coordinator binary is executable in builder stage
* fix: rename coordinator source dir in builder to avoid colliding with binary path
* fix: copy full repo in Dockerfile builder so go.mod resolves all packages
* fix: remove unused modelTypeTag and format Go files for CI
* fix: skip python/dangerous-modules check for swift runtime in private text gate
* billing telemetry + MarkUntrusted race fix + Swift routing tests
- Add Datadog histogram metrics for reservation amounts, settlement
refunds, provider credits, and platform fees
- Add store.debit/credit.latency_ms histograms for DB operation timing
- Add billing.cost_clamped and billing.reservation_refunds counters
- Fix race in MarkUntrusted: hold r.mu write lock through counter
decrement to prevent double-decrement with Disconnect
- Add unit tests for Swift provider privacy caps (with/without Python)
- Add E2E test for Swift provider routing via challenge-verified path
- Update dev-network-dashboard.json with Billing & Store group
* fix Heartbeat reviving untrusted providers causing onlineCount double-decrement
* revert orthogonal landing/console-ui/provider changes
* remove unbounded binary_hash cardinality, add input token metrics + store latency, fix dashboard group-by
* fix review feedback: ModelType() untrusted filter, routing.cost_ms by provider, billing in cents, dead comment
---------
Co-authored-by: Gajesh Naik <26431906+Gajesh2007@users.noreply.github.com>
Co-authored-by: anupsv <6407789+anupsv@users.noreply.github.com>
Co-authored-by: hankbob <hankbobtheresearchoor@gmail.com>
Co-authored-by: Hank Bob <hankbob@researchoors.com>
Summary
This PR hardens the pre-existing release/binary-hash trust path before the verifier work:
Validation