Fix harness egress NetworkPolicy blocking DNS on Kind (Connection timeout)#126
Merged
Merged
Conversation
…eout) The harness egress NetworkPolicy allowed DNS egress ONLY to the openshift-dns namespace on port 5353 (an OVN-Kubernetes post-DNAT specific rule, issue kagenti#102), on the documented assumption that kindnet does not enforce NetworkPolicy egress. That assumption is no longer true: modern kindnet enforces egress policy. On Kind, CoreDNS runs in kube-system on the standard port 53, so the harness pod's DNS was fully blocked — it could not resolve authbridge-ab1 or redis, and every LLM call (SH_AUTHBRIDGE=1 /runs and leaf-smoke H1-allow / Claims 2,3,4,6,7) failed with "Connection timeout". The request never reached AB1 (AB1 logs were empty); a direct TCP dial to the AB1 pod IP, bypassing DNS, succeeded. Add a second DNS egress entry allowing UDP+TCP 53 to the kube-system namespace alongside the existing openshift-dns:5353 entry, in both the base policy (harness-egress-policy.yaml) and the tightened AB1 variant (authbridge/harness-egress-ab1.yaml). NetworkPolicy egress rules are additive, so both platforms are covered and the non-matching entry is inert on each. Correct the stale "kindnet does not enforce" comments, and add manifest-shape tests asserting the Kind DNS rule in both policy variants. Verified on a live Kind cluster: after adding the kube-system:53 rule, the same /runs request that previously timed out returns a valid verdict. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Paolo Dettori <dettori@us.ibm.com>
pdettori
added a commit
that referenced
this pull request
Jul 15, 2026
The Hop-2 "see it yourself" allow-path command could never succeed:
- it sent no Authorization header, so AB2's static-inject had nothing to
swap and returned 401 static-inject.missing-auth; and
- it used the FQDN echo-target.default.svc.cluster.local, but static-inject
keys the credential off the Host (key_by: host) and the secret is keyed
`echo-target`, so the FQDN resolved to 401 static-inject.unresolved-key.
Use the short host `echo-target` and carry the placeholder token ($ECHO_CRED),
matching leaf-smoke.sh's H2-inject claim (verified live: HTTP 200, real token
reflected). Correct the expected-output comment to the actual JSON echo-target
returns.
Also correct two stale statements repeating the false "kindnet does not enforce
NetworkPolicy egress" premise (disproved by #126): the Prerequisites note and
the DNS troubleshooting row, which was framed OCP-only but applies to Kind too
(the harness DNS rule needs kube-system:53 on Kind alongside openshift-dns:5353
on OCP).
Verified: leaf-smoke.sh now reports 16 passed, 0 failed on Kind, and every
other manual "see it yourself" command was checked against a live cluster and
matches its documented output.
Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Paolo Dettori <dettori@us.ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Following
deploy/knative/README-authbridge.md,leaf-smoke.shfails every claim that requires an actual LLM call —H1-allowand Claims 2, 3, 4, 6, 7 — all with{"error":"Error: Connection timeout"}. The H2 sandbox-egress claims pass.Root cause
The harness (
serverless-harness) egressNetworkPolicyallowed DNS egress only to theopenshift-dnsnamespace on port5353— an OVN-Kubernetes post-DNAT-specific rule (issue #102) — on the documented assumption that "kindnet does not enforce NetworkPolicy egress."That assumption is no longer true: modern kindnet enforces egress policy. On Kind, CoreDNS lives in
kube-systemon the standard port53, so the harness pod's DNS was fully blocked. It could not resolveauthbridge-ab1(orredis), so every LLM call died in the connect phase with "Connection timeout."Evidence (live Kind cluster)
dns.lookupforauthbridge-ab1,redis→ allEAI_AGAIN(blocked).:8080(bypassing DNS) → succeeds.Fix
Add a second DNS egress entry allowing
UDP+TCP 53to thekube-systemnamespace, alongside the existingopenshift-dns:5353entry, in both:deploy/knative/harness-egress-policy.yaml(base / flag-off path)deploy/knative/authbridge/harness-egress-ab1.yaml(tightened AB1 variant)NetworkPolicy egress rules are additive, so both platforms are covered and the non-matching entry is inert on each. The stale "kindnet does not enforce" comments are corrected, and manifest-shape tests asserting the Kind DNS rule are added for both policy variants.
Verification
harness-egress-policy.test.ts+authbridge-manifests.test.ts→ 62 passed (incl. 2 new Kind-DNS assertions).kube-system:53rule, the same/runsrequest that previously timed out returns a valid verdict ({"status":"done","verdict":{...}}).Assisted-By: Claude Code