From 076a028c5166b81eb150ef09d47c3e7b2a402255 Mon Sep 17 00:00:00 2001 From: Paolo Dettori Date: Mon, 13 Jul 2026 22:35:44 -0400 Subject: [PATCH] docs: add AuthBridge two-hop egress-control demo (Kind + OCP) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit There was no demo doc for the RC1 AuthBridge/Cortex egress-control path: the runnable proof (leaf-smoke.sh on Kind, rc1-ocp-gate.sh on OCP) existed, but nothing walked a reader through the two-hop topology, how to enable it, or how to see credential injection + allow/deny control. README-kind.md/README-ocp.md don't mention SH_AUTHBRIDGE at all; serverless-harness-demo.md covers only scale-to-zero. Add deploy/knative/README-authbridge.md covering BOTH platforms: - the two hops (Profile A shared LLM gateway AB1; Profile B per-sandbox forward-proxy AB2) and the control-before-inject invariant; - Quick start for Kind (SH_AUTHBRIDGE=1 setup-kind.sh + leaf-smoke.sh) and OpenShift (SH_AUTHBRIDGE=1 setup-ocp.sh + rc1-ocp-gate.sh gate); - what it installs (AB1/AB2/ibac-stub/echo-target/secrets/tightened egress), the six capability claims with their PASS messages, and manual secret-free / allow / deny checks (spec §5, exact kubectl commands); - troubleshooting (incl. the OVN-K DNS 5353 fix #102 and the relay tsx fix) and cleanup/restore. Cross-linked from the Smoke test sections of README-kind.md and README-ocp.md. OCP flow live-verified on 4.20 (gate 6/6, 16/16); Kind commands are from setup-kind.sh/leaf-smoke.sh (the RC1-1/RC1-2 Kind gates: 13/13, 16/16). Assisted-By: Claude (Anthropic AI) Signed-off-by: Paolo Dettori --- deploy/knative/README-authbridge.md | 188 ++++++++++++++++++++++++++++ deploy/knative/README-kind.md | 4 + deploy/knative/README-ocp.md | 4 + 3 files changed, 196 insertions(+) create mode 100644 deploy/knative/README-authbridge.md diff --git a/deploy/knative/README-authbridge.md b/deploy/knative/README-authbridge.md new file mode 100644 index 0000000..e16c68f --- /dev/null +++ b/deploy/knative/README-authbridge.md @@ -0,0 +1,188 @@ +# AuthBridge Two-Hop Egress Control (RC1) + +This demo shows **Rosso Cortex / AuthBridge** acting as the zero-trust credential +plane on the serverless harness: it does both **credential injection** and +**action control** on the harness's two HTTP egress hops, and the real credential +is **never held by any model-influenced workload** — only a placeholder is, and the +real value is swapped in at the proxy, *after* an allow/deny gate. + +The path is gated behind the **`SH_AUTHBRIDGE`** feature flag (off by default). It +runs the same way on Kind and OpenShift; only the enable/run entrypoints differ. + +- Design: [`../specs/2026-07-10-authbridge-egress-control-plane-poc-design.md`](../specs/2026-07-10-authbridge-egress-control-plane-poc-design.md) · [ADR-0025](../adrs/0025-authbridge-deployment-topology.md) +- Base deploy guides: [`README-kind.md`](README-kind.md) · [`README-ocp.md`](README-ocp.md) + +## The two hops + +One AuthBridge role per egress hop; on each hop the **control plugin runs before +injection**, so a denied action never receives a real credential. + +``` +Hop 1 (Profile A) — harness → LLM. SHARED gateway. + harness pod ──Bearer PLACEHOLDER──▶ AB1 (Deployment+Service) + inference-parser → IBAC(allow/deny) → static-inject(real key) + ──x-api-key REAL──▶ api.anthropic.com + The harness holds only a placeholder + ANTHROPIC_BASE_URL=http://authbridge-ab1:8080. + +Hop 2 (Profile B) — sandbox → external API. PER-SANDBOX forward-proxy sidecar. + sandbox code ──http(s)_proxy=localhost:8081──▶ AB2 (sidecar in the sandbox pod) + mcp-parser → IBAC(allow/deny) → static-inject(real token) + ──Authorization REAL──▶ echo-target (external API stand-in) + The sandbox holds only ECHO_CRED=PLACEHOLDER-TOKEN. +``` + +The real credentials live only in the `ab1-llm-cred` / `ab2-egress-cred` secrets +(mounted into AB1 / AB2), never in the harness or sandbox pod env. + +## Prerequisites + +- A real Anthropic API key exported as `ANTHROPIC_API_KEY` — AB1 injects it on Hop 1. +- **Kind:** the [`README-kind.md`](README-kind.md) prerequisites (Docker, `kind`, `kubectl`). +- **OpenShift:** the [`README-ocp.md`](README-ocp.md) prerequisites (`oc` logged in, an + OCP 4.x cluster). The enforcing CNI (OVN-Kubernetes) is where the tightened egress + policy is actually enforced; on Kind (kindnet) NetworkPolicy egress is a no-op. + +## Quick start + +### Kind + +```bash +export ANTHROPIC_API_KEY=sk-ant-... + +# 1. Stand up the stack WITH the AuthBridge two-hop path (AB1 + AB2 + ibac-stub + echo-target) +SH_AUTHBRIDGE=1 ./deploy/knative/setup-kind.sh +# fast re-run against an existing cluster with images already loaded: +# SH_AUTHBRIDGE=1 ./deploy/knative/setup-kind.sh --skip-build --cluster-name sh-knative + +# 2. Run the demo/smoke — proves all six claims below (auto port-forwards Kourier) +LEAF_LIVE_SMOKE=1 SH_AUTHBRIDGE=1 NS=default SH_MODEL=claude-haiku-4-5 \ + bash deploy/knative/leaf-smoke.sh +# => ... === Results: 16 passed, 0 failed === / LEAF SMOKE PASS +``` + +On Kind, leave `KSVC_URL` unset: the smoke talks to `http://localhost:8080` with a +`Host:` header and starts the Kourier port-forward itself. `NS` **must** be `default` +(the AB1 manifests are pinned to that namespace). + +### OpenShift + +```bash +export ANTHROPIC_API_KEY=sk-ant-... + +# 1. Stand up the stack WITH the AuthBridge path (uses the published GHCR images via overlays/ocp-authbridge) +SH_AUTHBRIDGE=1 ./deploy/knative/setup-ocp.sh + +# 2. Run the live gate: applies the AB path, runs the smoke via the Route, then RESTORES to base +./deploy/knative/rc1-ocp-gate.sh gate +# => ... === Results: 16 passed, 0 failed === / SMOKE_EXIT:0 / RESTORE COMPLETE +``` + +`rc1-ocp-gate.sh` targets an **already-deployed** stack in `default` and has four +modes: `up` (apply the AB path), `smoke` (run leaf-smoke against the Route), `restore` +(return to the base direct-Anthropic state), and `gate` (= `up` → `smoke` → +`restore`, the default). To keep the AB path applied for manual inspection, run +`RC1_GATE_KEEP=1 ./deploy/knative/rc1-ocp-gate.sh gate` (then `restore` when done). +The gate resolves the Route automatically (`oc get ksvc serverless-harness -n default -o jsonpath='{.status.url}'`). + +## What it installs + +| Component | Manifest | Role | +|-----------|----------|------| +| **AB1** LLM gateway | [`authbridge/ab1-deployment.yaml`](authbridge/ab1-deployment.yaml) | Shared reverse-proxy `Deployment`+`Service`; Hop-1 gate + key injection | +| **ibac-stub** | [`ibac-stub.yaml`](ibac-stub.yaml) | Canned allow/deny policy decision (both hops) | +| tightened egress | [`authbridge/harness-egress-ab1.yaml`](authbridge/harness-egress-ab1.yaml) | `NetworkPolicy` — harness may reach only AB1 (overwrites the base policy) | +| **AB2** sidecar + config | [`authbridge/ab2-config.yaml`](authbridge/ab2-config.yaml), [`sandbox-pool-ab2.yaml`](sandbox-pool-ab2.yaml) | Per-sandbox forward proxy on loopback `:8081`; Hop-2 gate + token injection | +| **echo-target** | [`echo-target.yaml`](echo-target.yaml) | External-API stand-in that reflects the `Authorization` it received | +| secrets | (created by the setup script) | `ab1-llm-cred` (real key), `ab2-egress-cred` (real token); `llm-credentials` repointed to AB1 placeholders | + +On OpenShift these are wired via [`overlays/ocp-authbridge`](overlays/ocp-authbridge) +(image remaps + SCC/securityContext patches over the shared manifests); the base, +non-AuthBridge state is [`overlays/ocp`](overlays/ocp). + +## The six claims + +`leaf-smoke.sh` (with `SH_AUTHBRIDGE=1`) proves these before the base leaf claims — +Hop-2 first, then Hop-1: + +| Claim | What it proves | PASS message | +|-------|----------------|--------------| +| **H2-secret-free** | sandbox holds only the AB2 placeholder | `sandbox spec env holds only the AB2 placeholder + proxy vars, no real cred` | +| **H2-inject** | AB2 allow-path injects the real token at the proxy | `echo reflected the real cred (injected at AB2), sandbox never held it` | +| **H2-deny** | a `tools/call`-shaped request is blocked pre-egress | `denied pre-egress (ibac.no_session or ibac.no_intent ...), no injection` | +| **H1-secret-free** | harness holds only the AB1 placeholder | `harness env holds only the AB1 placeholder + base URL, no real key` | +| **H1-allow** | a leaf completes only because AB1 injected the real key | `allow-path leaf completed (real key injected at AB1)` | +| **H1-deny** | a denylisted call is 403'd at AB1 before injection | `deny-before-inject proven (AB1 returned 403 at ibac, pre-static-inject ...)` | + +## See it yourself (manual checks) + +These require the AuthBridge path to be **applied** — on Kind, after step 1 above; +on OpenShift, after `rc1-ocp-gate.sh up` (or `RC1_GATE_KEEP=1 ... gate`). Commands +use `kubectl`; on OpenShift `oc` works identically. + +**Hop 2 — the sandbox never holds the real egress credential:** + +```bash +# Sandbox env has only the placeholder: +kubectl -n default exec sandbox-0 -c sandbox -- printenv ECHO_CRED +# => PLACEHOLDER-TOKEN + +# The real value lives ONLY in the AB2 secret (not in any sandbox pod): +kubectl -n default get secret ab2-egress-cred -o jsonpath='{.data.echo-target}' | base64 -d; echo +# => REAL-ECHO-EGRESS-CRED-rc1demo (the default; whatever RC1_ECHO_REAL_CRED was set to) + +# Allow-path: a request through the sandbox's proxy reaches echo-target, which reflects +# the Authorization it received = the REAL token, injected at AB2 — the sandbox never had it. +kubectl -n default exec sandbox-0 -c sandbox -- \ + sh -c 'curl -s http://echo-target.default.svc.cluster.local/ | grep -i authorization' +# => authorization: Bearer REAL-ECHO-EGRESS-CRED-rc1demo +``` + +**Hop 1 — the harness never holds the real provider key:** + +```bash +# The harness reads its provider creds from the llm-credentials secret, which holds +# only placeholders + the in-cluster gateway URL: +kubectl -n default get secret llm-credentials \ + -o jsonpath='{.data.api-key}' | base64 -d; echo # => AB1-PLACEHOLDER +kubectl -n default get secret llm-credentials \ + -o jsonpath='{.data.base-url}' | base64 -d; echo # => http://authbridge-ab1:8080 + +# The real key lives ONLY in the AB1 secret: +kubectl -n default get secret ab1-llm-cred -o jsonpath='{.data.api\.anthropic\.com}' | base64 -d | head -c 8; echo '...' + +# On a running harness revision pod, the injected env is the placeholder too: +POD=$(kubectl -n default get pod -l serving.knative.dev/service=serverless-harness -o name | head -1) +kubectl -n default exec "$POD" -c user-container -- printenv ANTHROPIC_BASE_URL # => http://authbridge-ab1:8080 +``` + +(Knative scales the harness to zero when idle; send a request first, or run the +`leaf-smoke.sh` smoke, to have a pod to exec into. The smoke reads the pod **spec** +env via `jsonpath` instead of `exec` to sidestep exec flakiness on loaded single-node +clusters.) + +## Troubleshooting + +| Symptom | Cause / fix | +|---------|-------------| +| Harness can't resolve `authbridge-ab1` / DNS times out on OCP | OVN-Kubernetes enforces egress **post-DNAT**; the DNS rule must allow CoreDNS port **5353** (to `openshift-dns`), not 53. Fixed in [`authbridge/harness-egress-ab1.yaml`](authbridge/harness-egress-ab1.yaml) — see issue #102. | +| `leaf-smoke.sh` exits `SKIP` | Set `LEAF_LIVE_SMOKE=1`. | +| Smoke aborts with an `NS` error | The AuthBridge path requires `NS=default` (AB1 manifests are namespace-pinned). | +| H2 checks see the old sandbox image / no sidecar | The sandbox controller does not roll pods on CR change; the setup/gate force-`delete pod`s the pool and waits. If stale, `kubectl -n default delete pod -l sh.kagenti.io/sandbox-pool=default`. | +| `curl` in the sandbox ignores the proxy for `http://` URLs | It honors lowercase `http_proxy` — the pool sets both cases; use lowercase in ad-hoc curls. | +| sandbox-relay `CrashLoopBackOff` (`ERR_MODULE_NOT_FOUND: tsx`) | Unrelated to AuthBridge; the relay must run from its package dir. Fixed in [`relay-deployment.yaml`](relay-deployment.yaml). | + +## Cleanup / restore + +- **OpenShift:** `./deploy/knative/rc1-ocp-gate.sh restore` returns `default` to the + base direct-Anthropic state (removes AB resources, restores `llm-credentials`, + re-applies [`overlays/ocp`](overlays/ocp)). `gate` mode does this automatically on exit. +- **Kind:** re-run `./deploy/knative/setup-kind.sh` **without** `SH_AUTHBRIDGE=1` to + redeploy the base stack, or delete the cluster (`kind delete cluster --name sh-knative`). + +## Scope + +RC1 is a **single-tenant, static-credential** PoC: real injection + real control +pipeline, with a canned (`ibac-stub`) judge. Per-user identity, RFC 8693 +token-exchange, a real SPARC/IBAC judge, and baked-CA HTTPS interception are +deferred (see the design doc §10). The bring-your-own / remote-sandbox variant +(RC1-3) is a stretch gated on the SandboxTransport live gate (ST5, #88). diff --git a/deploy/knative/README-kind.md b/deploy/knative/README-kind.md index d2a5f92..1eb8528 100644 --- a/deploy/knative/README-kind.md +++ b/deploy/knative/README-kind.md @@ -112,6 +112,10 @@ Run the smoke suite (requires the Kourier port-forward to be active): See [`SMOKE.md`](SMOKE.md) for detailed results and what each claim proves. +For the **AuthBridge two-hop egress-control** demo (`SH_AUTHBRIDGE=1`) — credential +injection + allow/deny control on both harness egress hops — see +[`README-authbridge.md`](README-authbridge.md). + ## Troubleshooting | Symptom | Cause / fix | diff --git a/deploy/knative/README-ocp.md b/deploy/knative/README-ocp.md index 3bd8042..079c6c5 100644 --- a/deploy/knative/README-ocp.md +++ b/deploy/knative/README-ocp.md @@ -128,6 +128,10 @@ KSVC_URL=$(oc get ksvc serverless-harness -n default -o jsonpath='{.status.url}' ./deploy/knative/smoke.sh ``` +For the **AuthBridge two-hop egress-control** demo on OpenShift (`SH_AUTHBRIDGE=1` + +the [`rc1-ocp-gate.sh`](rc1-ocp-gate.sh) live gate) — credential injection + allow/deny +control on both harness egress hops — see [`README-authbridge.md`](README-authbridge.md). + See [`SMOKE.md`](SMOKE.md#smoke-test-on-openshift) for details. Claims that assert on the **LLM `/turn` response** require the harness to reach its configured Anthropic endpoint *from the cluster*; health, scale-to-zero/-up, Redis session