Skip to content

feat(k8s): per-container VPA policies (exclude sidecars from the app's minAllowed floor)#340

Merged
Cre-eD merged 3 commits into
mainfrom
feat/vpa-per-container-policies
Jun 24, 2026
Merged

feat(k8s): per-container VPA policies (exclude sidecars from the app's minAllowed floor)#340
Cre-eD merged 3 commits into
mainfrom
feat/vpa-per-container-policies

Conversation

@Cre-eD

@Cre-eD Cre-eD commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Problem

createVPA renders a single resourcePolicy.containerPolicies entry with containerName: "*", so every container in the pod is governed by the same policy. When a pod has an injected sidecar (e.g. cloudsql-proxy) alongside the app, the sidecar inherits the app's minAllowed floor — a small proxy gets over-provisioned to the app's CPU/memory minimum.

Change

Add cloudExtras.vpa.containerPolicies[] — per-container VPA overrides:

vpa:
  enabled: true
  updateMode: Auto
  minAllowed: { cpu: 250m, memory: 64Mi }   # renders the "*" catch-all
  maxAllowed: { cpu: "2", memory: 2560Mi }
  controlledResources: [cpu, memory]
  controlledValues: RequestsOnly
  containerPolicies:
    - containerName: cloudsql-proxy
      mode: "Off"                             # excluded from VPA → keeps template request
  • The top-level minAllowed/maxAllowed/controlledResources/controlledValues still render the "*" catch-all exactly as before.
  • Each containerPolicies entry renders a per-container policy. The VPA admission controller matches an exact containerName before the "*" wildcard, so a specific entry takes precedence regardless of array order.
  • mode: "Off" disables VPA for that container so its requests stay at the deployment-template values.

Compatibility

  • Backward compatible: with no containerPolicies, the rendered VPA is byte-for-byte what it is today.
  • VPAConfig is passed by pointer through the k8s and GKE-Autopilot converters (deployCfg.VPA = …), so the new field flows through both paths with no converter changes — no risk of the two drifting.

Tests

TestCreateVPA_PerContainerPolicies renders a VPA with a top-level floor + a mode: Off sidecar override and asserts the spec has exactly the "*" catch-all (carrying the floor + controlledValues) plus a cloudsql-proxy entry with mode: Off and no inherited minAllowed. Verified it fails without the per-container rendering and passes with it. Full pkg/clouds/pulumi/kubernetes + pkg/clouds/k8s suites green; go vet + gofmt clean.

docs/schemas/* are generated artifacts (regenerated from scratch in CI's generate-schemas step, no diff gate) and are not committed here — consistent with prior field additions in this area.

…s minAllowed floor)

Signed-off-by: Dmitrii Creed <creeed22@gmail.com>
@github-actions

github-actions Bot commented Jun 22, 2026

Copy link
Copy Markdown

Semgrep Scan Results

Repository: api | Commit: cb9d5ea

Check Status Details
⚠️ Semgrep Warning 1 warning(s), 1 total

Scanned at 2026-06-23 15:41 UTC

@github-actions

github-actions Bot commented Jun 22, 2026

Copy link
Copy Markdown

Security Scan Results

Repository: api | Commit: cb9d5ea

Check Status Details
✅ Secret Scan Pass No secrets detected
✅ Dependencies (Trivy) Pass 0 total (no critical/high)
✅ Dependencies (Grype) Pass 0 total (no critical/high)
📦 SBOM Generated 522 components (CycloneDX)

Scanned at 2026-06-23 15:41 UTC

@github-actions

github-actions Bot commented Jun 22, 2026

Copy link
Copy Markdown

📊 Statement coverage

Measured on the documented included set (see docs/TESTING.md → Coverage scope). Observe-only — no regression gate is enforced yet.

Scope This PR main baseline Δ
Included set (Gold-tier denominator) 90.3% 90.3% +0.0 pp
Full set (whole repo, transparency) 27.7% 27.6% +0.1 pp

Baseline: main @ 9fd95bf

…ncy + backward-compat/edge tests

- ValidateVPAConfiguration: reject empty/duplicate containerName, reserved '*',
  invalid mode; called at top of createVPA so a malformed config fails preview,
  not mid-pulumi-up (where a failed VPA replace could strand a stack).
- Warn when containerPolicies are set without a top-level floor (the '*' catch-all
  is omitted and unlisted containers autoscale unbounded).
- Drop omitempty on the new tags to match the file's non-omitempty convention.
- Tests: backward-compat byte-identity (no policies -> single '*'),
  containerPolicies-only (no '*'), per-container minAllowed override, and
  validation negatives.

Signed-off-by: Dmitrii Creed <creeed22@gmail.com>
@Cre-eD

Cre-eD commented Jun 23, 2026

Copy link
Copy Markdown
Contributor Author

Review pass applied (commit c38f1f5) — hardening on top of the original feature:

  • ValidateVPAConfiguration rejects empty/duplicate containerName, a reserved "*" entry, and an invalid mode. Called at the top of createVPA, so a malformed config fails at preview/plan time instead of mid-pulumi up (where a failed VPA replace could briefly leave a stack with no VPA).
  • Warns when containerPolicies are set with no top-level floor (the "*" catch-all is omitted, so unlisted containers would autoscale unbounded).
  • Tag style aligned to the file's non-omitempty convention.
  • Tests added: backward-compat byte-identity (no policies → single "*"), containerPolicies-only branch (no "*"), a per-container minAllowed override, and validation negatives.

Backward compatibility is exact: with no containerPolicies, the rendered spec is the historical single "*" entry (locked by a test). All checks green.

Signed-off-by: Dmitrii Creed <creeed22@gmail.com>
@Cre-eD Cre-eD merged commit 4bdf2b3 into main Jun 24, 2026
35 checks passed
Cre-eD added a commit that referenced this pull request Jul 1, 2026
…ace (#347)

## Problem

The runtime CloudSQL proxy is injected as a **plain sidecar container**
([`compute_proc.go`](../blob/main/pkg/clouds/pulumi/gcp/compute_proc.go)
→ `SidecarOutputs` → `PodSpec.Containers`). Sidecar containers start
**in parallel** with the app container with no ordering guarantee, and
the proxy container carries **no probe**. So on every pod (re)start —
GKE Autopilot node scale-down, VPA `updateMode: Auto` eviction, or a
rollout — the app process dials `localhost:5432` before the proxy is
listening and logs:

```
error connecting in 'pool-1': connection to server at "127.0.0.1", port 5432 failed: Connection refused
```

It self-heals (the client retries), so there's no data loss — but it's
recurring log noise across **every** GCP SC stack, and for app
containers that have a startup probe it costs an **extra restart** to
recover (observed in prod: a single-replica web pod took a
`137`/startup-probe-fail restart on relocation, briefly dropping its
uptime check).

The existing probe machinery in `deployment.go` can't help:
readiness/startup probes are only attached to the **ingress/app**
container (or a container with a single port), never to the proxy — and
**port-less worker containers get no gate at all**.

## Fix

Run the runtime proxy as a **native sidecar** — an init container with
`RestartPolicy: Always` — gated by a **startup probe** against the
proxy's built-in `--health-check` HTTP server. Per the Kubernetes
SidecarContainers contract, the app containers don't start until the
proxy's startup probe passes, i.e. until the proxy is actually
listening. The race is eliminated for **web and worker tiers alike**.

Two files:
- `cloudsql_proxy.go` — when `timeout == 0` (runtime proxy): enable
`--health-check` (`--http-address=0.0.0.0 --http-port=9090`), declare
the health port, add `RestartPolicy: Always` +
`StartupProbe`/`ReadinessProbe` on `/startup` & `/readiness`. Refactored
into pure `cloudsqlProxyCommandArgs` / `cloudsqlProxyContainerArgs`
helpers so the behaviour is unit-testable.
- `compute_proc.go` — append the runtime proxy to `InitContainerOutputs`
instead of `SidecarOutputs`.

### Scope / safety
- **Deployment proxy only.** The init-Job proxy (`timeout > 0`, the
self-killing variant used by the db-user-init Job) stays an ordinary
terminating container — `RestartPolicy: Always` on a `RestartPolicy:
Never` Job's container would keep the Job from ever completing. This
invariant is now covered by a test.
- **`--health-check` binds `0.0.0.0`** (default is localhost) so the
kubelet probe can reach it.
- The `#340` per-container VPA policy matches the proxy **by name**
(`cloudsql-proxy`), unaffected by the init/regular placement.
- **Requires GKE ≥ 1.29** (`SidecarContainers`, beta-on-by-default in
1.29, GA in 1.33). Target clusters are on **1.34**.

### Follow-up (not in this PR)
Native sidecars also make the init-Job proxy's `sleep`/`kill -9` timeout
hack unnecessary (native sidecars in Jobs terminate when the main
container exits) — can be simplified in a later change.

## Tests
- `TestCloudsqlProxyCommandArgs_RuntimeEnablesHealthCheck` — runtime
proxy runs the binary directly with `--health-check` +
`--http-address=0.0.0.0`, not shell-wrapped.
- `TestCloudsqlProxyCommandArgs_InitJobSelfKills` — init-Job proxy is
`sh -c` wrapped, self-kills, no health server.
- `TestCloudsqlProxyContainerArgs_RuntimeIsNativeSidecar` —
`RestartPolicy: Always` + startup/readiness probes + health port
present.
- `TestCloudsqlProxyContainerArgs_InitJobIsNotSidecar` — no
`RestartPolicy`, no probe, no port (regression guard against
accidentally making the Job proxy a sidecar).

`go build ./pkg/clouds/...`, `go vet`, and the full `gcp` + `kubernetes`
package suites pass.

## Rollout
**Staging-first.** This changes pod startup ordering; validate on a
staging stack (confirm the proxy appears under `initContainers` with
`restartPolicy: Always`, the app waits for it, and no connection-refused
on a forced pod delete) before bumping the pinned SC version in the
consuming repos.

---------

Signed-off-by: Dmitrii Creed <creeed22@gmail.com>
Co-authored-by: Ilya <smecsia@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants