feat(k8s): per-container VPA policies (exclude sidecars from the app's minAllowed floor)#340
Merged
Merged
Conversation
…s minAllowed floor) Signed-off-by: Dmitrii Creed <creeed22@gmail.com>
Semgrep Scan ResultsRepository:
Scanned at 2026-06-23 15:41 UTC |
Security Scan ResultsRepository:
Scanned at 2026-06-23 15:41 UTC |
📊 Statement coverageMeasured on the documented included set (see
Baseline: |
…ncy + backward-compat/edge tests - ValidateVPAConfiguration: reject empty/duplicate containerName, reserved '*', invalid mode; called at top of createVPA so a malformed config fails preview, not mid-pulumi-up (where a failed VPA replace could strand a stack). - Warn when containerPolicies are set without a top-level floor (the '*' catch-all is omitted and unlisted containers autoscale unbounded). - Drop omitempty on the new tags to match the file's non-omitempty convention. - Tests: backward-compat byte-identity (no policies -> single '*'), containerPolicies-only (no '*'), per-container minAllowed override, and validation negatives. Signed-off-by: Dmitrii Creed <creeed22@gmail.com>
Contributor
Author
|
Review pass applied (commit c38f1f5) — hardening on top of the original feature:
Backward compatibility is exact: with no |
Signed-off-by: Dmitrii Creed <creeed22@gmail.com>
smecsia
approved these changes
Jun 24, 2026
Laboratory
approved these changes
Jun 24, 2026
universe-ops
approved these changes
Jun 24, 2026
Cre-eD
added a commit
that referenced
this pull request
Jul 1, 2026
…ace (#347) ## Problem The runtime CloudSQL proxy is injected as a **plain sidecar container** ([`compute_proc.go`](../blob/main/pkg/clouds/pulumi/gcp/compute_proc.go) → `SidecarOutputs` → `PodSpec.Containers`). Sidecar containers start **in parallel** with the app container with no ordering guarantee, and the proxy container carries **no probe**. So on every pod (re)start — GKE Autopilot node scale-down, VPA `updateMode: Auto` eviction, or a rollout — the app process dials `localhost:5432` before the proxy is listening and logs: ``` error connecting in 'pool-1': connection to server at "127.0.0.1", port 5432 failed: Connection refused ``` It self-heals (the client retries), so there's no data loss — but it's recurring log noise across **every** GCP SC stack, and for app containers that have a startup probe it costs an **extra restart** to recover (observed in prod: a single-replica web pod took a `137`/startup-probe-fail restart on relocation, briefly dropping its uptime check). The existing probe machinery in `deployment.go` can't help: readiness/startup probes are only attached to the **ingress/app** container (or a container with a single port), never to the proxy — and **port-less worker containers get no gate at all**. ## Fix Run the runtime proxy as a **native sidecar** — an init container with `RestartPolicy: Always` — gated by a **startup probe** against the proxy's built-in `--health-check` HTTP server. Per the Kubernetes SidecarContainers contract, the app containers don't start until the proxy's startup probe passes, i.e. until the proxy is actually listening. The race is eliminated for **web and worker tiers alike**. Two files: - `cloudsql_proxy.go` — when `timeout == 0` (runtime proxy): enable `--health-check` (`--http-address=0.0.0.0 --http-port=9090`), declare the health port, add `RestartPolicy: Always` + `StartupProbe`/`ReadinessProbe` on `/startup` & `/readiness`. Refactored into pure `cloudsqlProxyCommandArgs` / `cloudsqlProxyContainerArgs` helpers so the behaviour is unit-testable. - `compute_proc.go` — append the runtime proxy to `InitContainerOutputs` instead of `SidecarOutputs`. ### Scope / safety - **Deployment proxy only.** The init-Job proxy (`timeout > 0`, the self-killing variant used by the db-user-init Job) stays an ordinary terminating container — `RestartPolicy: Always` on a `RestartPolicy: Never` Job's container would keep the Job from ever completing. This invariant is now covered by a test. - **`--health-check` binds `0.0.0.0`** (default is localhost) so the kubelet probe can reach it. - The `#340` per-container VPA policy matches the proxy **by name** (`cloudsql-proxy`), unaffected by the init/regular placement. - **Requires GKE ≥ 1.29** (`SidecarContainers`, beta-on-by-default in 1.29, GA in 1.33). Target clusters are on **1.34**. ### Follow-up (not in this PR) Native sidecars also make the init-Job proxy's `sleep`/`kill -9` timeout hack unnecessary (native sidecars in Jobs terminate when the main container exits) — can be simplified in a later change. ## Tests - `TestCloudsqlProxyCommandArgs_RuntimeEnablesHealthCheck` — runtime proxy runs the binary directly with `--health-check` + `--http-address=0.0.0.0`, not shell-wrapped. - `TestCloudsqlProxyCommandArgs_InitJobSelfKills` — init-Job proxy is `sh -c` wrapped, self-kills, no health server. - `TestCloudsqlProxyContainerArgs_RuntimeIsNativeSidecar` — `RestartPolicy: Always` + startup/readiness probes + health port present. - `TestCloudsqlProxyContainerArgs_InitJobIsNotSidecar` — no `RestartPolicy`, no probe, no port (regression guard against accidentally making the Job proxy a sidecar). `go build ./pkg/clouds/...`, `go vet`, and the full `gcp` + `kubernetes` package suites pass. ## Rollout **Staging-first.** This changes pod startup ordering; validate on a staging stack (confirm the proxy appears under `initContainers` with `restartPolicy: Always`, the app waits for it, and no connection-refused on a forced pod delete) before bumping the pinned SC version in the consuming repos. --------- Signed-off-by: Dmitrii Creed <creeed22@gmail.com> Co-authored-by: Ilya <smecsia@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
createVPArenders a singleresourcePolicy.containerPoliciesentry withcontainerName: "*", so every container in the pod is governed by the same policy. When a pod has an injected sidecar (e.g.cloudsql-proxy) alongside the app, the sidecar inherits the app'sminAllowedfloor — a small proxy gets over-provisioned to the app's CPU/memory minimum.Change
Add
cloudExtras.vpa.containerPolicies[]— per-container VPA overrides:minAllowed/maxAllowed/controlledResources/controlledValuesstill render the"*"catch-all exactly as before.containerPoliciesentry renders a per-container policy. The VPA admission controller matches an exactcontainerNamebefore the"*"wildcard, so a specific entry takes precedence regardless of array order.mode: "Off"disables VPA for that container so its requests stay at the deployment-template values.Compatibility
containerPolicies, the rendered VPA is byte-for-byte what it is today.VPAConfigis passed by pointer through the k8s and GKE-Autopilot converters (deployCfg.VPA = …), so the new field flows through both paths with no converter changes — no risk of the two drifting.Tests
TestCreateVPA_PerContainerPoliciesrenders a VPA with a top-level floor + amode: Offsidecar override and asserts the spec has exactly the"*"catch-all (carrying the floor +controlledValues) plus acloudsql-proxyentry withmode: Offand no inheritedminAllowed. Verified it fails without the per-container rendering and passes with it. Fullpkg/clouds/pulumi/kubernetes+pkg/clouds/k8ssuites green;go vet+gofmtclean.