Buffer proxied request bodies; harden cloudsql-proxy readiness#365
Open
Cre-eD wants to merge 2 commits into
Open
Buffer proxied request bodies; harden cloudsql-proxy readiness#365Cre-eD wants to merge 2 commits into
Cre-eD wants to merge 2 commits into
Conversation
Cre-eD
requested review from
Laboratory,
smecsia and
universe-ops
as code owners
July 17, 2026 14:42
Semgrep Scan ResultsRepository:
Scanned at 2026-07-18 04:56 UTC |
Security Scan ResultsRepository:
Scanned at 2026-07-18 04:56 UTC |
📊 Statement coverageMeasured on the documented included set (see
Baseline: |
This comment has been minimized.
This comment has been minimized.
reverse_proxy render: add request_buffers (default 1MiB, per-stack lbConfig.requestBufferSize, "0" disables). Chunked request bodies reach WSGI upstreams without Content-Length; Django (ticket #28668) reads an empty body on such requests - webhook 500s, HMAC signature failures over the empty body, lost CSRF tokens. Bodies above the buffer keep streaming as before. cloudsql-proxy sidecar: readiness timeout 3s->10s, CPU request 50m->100m. At 50m the health server starves under node CPU pressure; 3s probe timeouts flap pods out of Service endpoints (sidecar readiness gates pod readiness per KEP-753 - the old comment claimed otherwise and is fixed too). Combined with node consolidation this can briefly leave a Service with zero ready pods. Signed-off-by: Dmitrii Creed <creeed22@gmail.com>
Cre-eD
force-pushed
the
fix/csql-probe-caddy-request-buffers
branch
from
July 18, 2026 04:36
1b9ffc1 to
0c72dd4
Compare
…egen - lbConfig.requestBufferSize is now parsed (humanize), capped at 16MiB, and re-rendered as a plain byte count - no user-controlled text reaches the shared Caddyfile (a malformed value previously produced a parse-fatal aggregated config detonating at the next Caddy restart, taking down every tenant). Invalid values fail only the offending stack's deploy. Field is now a plain string with omitempty (sibling idiom; keeps it out of the JSON schema's required list). - "0" now OMITS the request_buffers directive entirely (placeholder carries the whole line) - real escape hatch for edges pinned to Caddy < 2.6.0. A render-time warning fires when extraHelpers carries a duplicate request_buffers (Caddy last-wins would silently override the typed knob). - Liveness probe timeout 3s->10s to match readiness: /liveness is served by the same starvable health server; a 3s timeout would RESTART the sidecar (severing live DB connections) under the exact pressure the readiness change tolerates. Startup stays tight on purpose (documented). - Regenerated docs/schemas (requestBufferSize in stackconfigcompose). - Tests per panel findings: prefix-template coverage (mutation-verified to fail on regression), line-anchored+placement assertions instead of substrings, invalid/cap/zero/default-with-lbConfig cases, all probe numerics pinned. go-humanize promoted to direct dependency. Signed-off-by: Dmitrii Creed <creeed22@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Two production-observed failure modes on GKE Autopilot deployments. Both fixes are systemic: any SC consumer running WSGI apps behind the generated Caddy, or cloudsql-proxy native sidecars, is exposed.
1.
request_bufferson generated reverse_proxy blocksChunked request bodies (no Content-Length) pass through Caddy to upstreams as-is. WSGI frameworks read the body by CONTENT_LENGTH — Django (#28668) gets an empty request.body. Observed blast radius: webhook handlers returning 500 (sender retry storms), HMAC signature verification failing over the empty body, «CSRF token missing» 403s.
Fix: every generated reverse_proxy block gets
request_buffers(default 1MiB; bodies above the buffer keep streaming as before). Per-stack knoblbConfig.requestBufferSize:"0"omits the directive entirely — the escape hatch for edges pinned to Caddy < 2.6.0;extraHelpers: [request_buffers ...]workaround get a render-time warning: Caddy's last-wins would silently override the typed knob — drop the workaround on upgrade.2. cloudsql-proxy sidecar probes
timeoutSeconds3s → 10s (both served by the same starvable health server; a 3s liveness timeout would restart the sidecar — severing live DB connections — under the exact pressure the readiness change tolerates); startup stays tight deliberately (documented).Review
Multi-model panel (4 independent reviewer lenses + 2 external models) on the first revision; all confirmed findings addressed in the second commit: input validation/normalization, liveness parity, JSON schema regeneration, prefix-template test coverage (mutation-verified), line-anchored assertions, all probe numerics pinned.
Tests
go build,go vet,gofmtclean; kubernetes/gcp/api suites green (-count=1). New: RequestBuffers Default / DefaultWithLbConfigSet / OnPrefixTemplate / OverrideNormalized / ZeroOmitsDirective / InvalidValueFailsDeploy / CapEnforced; ProbesTolerantToStarvation pins every probe numeric.Follow-ups (out of scope here)
timeouts { read_body }default for the edge servers block (slow-body memory-pinning DoS surface; needs its own compat analysis).simplecontainer/caddy:latest/simplecontainer/kubectl:latestfloating tags.