Skip to content

feat(k8s): expose UDP ports on cloud-compose via mixed-protocol LB#362

Open
RuslanMikailov wants to merge 3 commits into
simple-container-com:mainfrom
RuslanMikailov:feat/cloud-compose-udp-ports
Open

feat(k8s): expose UDP ports on cloud-compose via mixed-protocol LB#362
RuslanMikailov wants to merge 3 commits into
simple-container-com:mainfrom
RuslanMikailov:feat/cloud-compose-udp-ports

Conversation

@RuslanMikailov

Copy link
Copy Markdown
Contributor

sc parses the compose file and generates its own k8s Deployment + Service via Pulumi; during that translation the port protocol was discarded. toRunPorts mapped every compose port to a bare int, so "7882:7882/udp" silently became a TCP port and UDP never reached the pod. This blocks deploying a WebRTC media server (e.g. LiveKit SFU), where signaling is TCP (wss) but media is UDP.

Thread the port protocol end-to-end:

  • CloudRunContainer.Ports is now []k8s.ContainerPort{Port, Protocol} instead of []int; toRunPorts preserves the compose protocol (udp -> "UDP"; tcp/unspecified -> "").
  • New ports.go helpers set Protocol on ContainerPortArgs and ServicePortArgs, but only for non-TCP protocols, so TCP-only specs stay byte-for-byte identical (TCP keeps http- names; UDP gets udp-).
  • autoTCPReadinessProbe no longer attaches a TCP/HTTP probe to a UDP-only port (resolves the old "// TODO: support non-http ports").
  • serviceType is now expressible via cloudExtras and threaded CloudExtras -> DeploymentConfig -> kubeArgs, letting a service turn its own Service into a LoadBalancer (required for protocols Caddy cannot proxy).

A service exposing both TCP and UDP emits ONE mixed-protocol LoadBalancer Service (GA since k8s 1.26, supported on recent GKE) rather than splitting into two Services, so a single external IP fronts the whole service. Announcing that IP in ICE candidates (LiveKit rtc.use_external_ip / node_ip) is the app's job, not sc's.

Backward compatible: ports with no protocol behave exactly as before, and serviceType defaults to ClusterIP.

sc parses the compose file and generates its own k8s Deployment +
Service via Pulumi; during that translation the port protocol was
discarded. toRunPorts mapped every compose port to a bare int, so
"7882:7882/udp" silently became a TCP port and UDP never reached the
pod. This blocks deploying a WebRTC media server (e.g. LiveKit SFU),
where signaling is TCP (wss) but media is UDP.

Thread the port protocol end-to-end:

- CloudRunContainer.Ports is now []k8s.ContainerPort{Port, Protocol}
  instead of []int; toRunPorts preserves the compose protocol
  (udp -> "UDP"; tcp/unspecified -> "").
- New ports.go helpers set Protocol on ContainerPortArgs and
  ServicePortArgs, but only for non-TCP protocols, so TCP-only specs
  stay byte-for-byte identical (TCP keeps http-<port> names; UDP gets
  udp-<port>).
- autoTCPReadinessProbe no longer attaches a TCP/HTTP probe to a
  UDP-only port (resolves the old "// TODO: support non-http ports").
- serviceType is now expressible via cloudExtras and threaded
  CloudExtras -> DeploymentConfig -> kubeArgs, letting a service turn
  its own Service into a LoadBalancer (required for protocols Caddy
  cannot proxy).

A service exposing both TCP and UDP emits ONE mixed-protocol
LoadBalancer Service (GA since k8s 1.26, supported on recent GKE)
rather than splitting into two Services, so a single external IP
fronts the whole service. Announcing that IP in ICE candidates
(LiveKit rtc.use_external_ip / node_ip) is the app's job, not sc's.

Backward compatible: ports with no protocol behave exactly as before,
and serviceType defaults to ClusterIP.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Ruslan <ruslan.mikailov@fulldive.com>
@smecsia

smecsia commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Thanks for this @RuslanMikailov — reviewed it in depth and the approach is correct.

Why we re-established this as #363

CI here can't run the secret-gated jobs (build/test/security) on a PR from a fork — GitHub withholds Actions secrets from fork PRs, which is why the builds "failed" here. So I pushed your commit to an SC-owned branch and opened #363 to run the full pipeline. Your authorship is preserved (commit author stays Ruslan <ruslan.mikailov@fulldive.com>, your Signed-off-by intact); I only added a verified signature required by branch protection. Let's continue review + merge on #363 and close this one as superseded.

Review summary (the change is solid)

  • Root cause is right: toRunPorts collapsed every compose port to a bare int, dropping /udp, so "7882:7882/udp" silently became TCP. Now CloudRunContainer.Ports carries {Port, Protocol} end-to-end.
  • Backward-compatible by construction: TCP/unspecified → protocol omitted + http-<port> name (byte-for-byte identical to before); only UDP emits Protocol: UDP and a udp-<port> name. Tests assert Protocol == nil for TCP.
  • One mixed-protocol Service: all ingress ports land on a single Service whose Type comes from the new cloudExtras.serviceType, so LiveKit (TCP 7880 + UDP 7882) → one LB / one external IP.
  • Readiness probe correctly skips UDP-only ports (resolves the old // TODO: support non-http ports) and still probes the TCP main port on mixed containers.
  • Good coverage, incl. an end-to-end LiveKit-style TestSimpleContainer_UDPLoadBalancer.

Two non-blocking things to verify before merge

  1. GKE mixed-protocol LB is the real deployment risk. MixedProtocolLBService is GA in the k8s API since 1.26, but whether the GKE L4 load balancer actually provisions a single TCP+UDP setup depends on cluster version / LB type. Worth an actual deploy on the target Autopilot cluster to confirm the cloud LB behavior — static review can't.
  2. Multi-port without mainPort still hard-errors (unchanged). LiveKit users must set mainPort to the TCP signaling port; the docs page added in this PR should call that out explicitly.

Follow-up on #363.

@smecsia

smecsia commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

@RuslanMikailov I published a preview build of this branch so you can test the UDP change end-to-end on your LiveKit stack before merge — no need to build sc locally.

Preview version: 2026.7.4-pre.482661-preview.0482661

Install it (both linux-amd64 and darwin-arm64 are published):

SIMPLE_CONTAINER_VERSION=2026.7.4-pre.482661-preview.0482661 \
  bash -c "$(curl -fsSL https://dist.simple-container.com/sc.sh)"
sc --version   # should print 2026.7.4-pre.482661-preview.0482661

Then deploy your media server as usual. To exercise the fix, in your client.yaml:

  • keep the compose UDP port (e.g. "7882:7882/udp") — it now reaches the pod as UDP instead of being silently downgraded to TCP;
  • set cloudExtras.serviceType: LoadBalancer so the mixed TCP+UDP Service gets a real external IP (Caddy can't proxy UDP);
  • announce that LB IP to LiveKit via rtc.use_external_ip / node_ip.

Could you confirm on your GKE cluster that:

  1. the pod's UDP port is actually reachable (media flows), and
  2. GKE provisions a single mixed-protocol LoadBalancer for the TCP+UDP Service (the one item static review couldn't verify)?

Built from feat/cloud-compose-udp-ports @ 0482661 via the branch-preview workflow. Once you've confirmed it works, we'll finish review + merge on #363.

PR simple-container-com#362 threaded UDP protocol + a stack-level serviceType, but the Service was
still built solely from the ingress container's ports, and on GKE a mixed
TCP+UDP LoadBalancer failed to provision (LoadBalancerMixedProtocolNotSupported).
Both blocked self-hosting a LiveKit SFU alongside the HTTP backend in one stack.

- Service ports now include non-ingress run containers' ports (they share the
  pod's network namespace and the Service selects the pod), so a sidecar SFU's
  7880/7882 land on the same Service as backend:3000. Ingress ports win on a
  port-number clash; ports are deduped.
- When the emitted LoadBalancer Service mixes TCP and UDP, annotate it with
  cloud.google.com/l4-rbs: "enabled" so GKE uses the RBS/NEG-based NetLB, which
  supports mixed protocols (verified live: one external IP fronting
  7880/7881 TCP + 7882/UDP). No config passthrough exists for Service
  annotations, so this must be set by sc.

Tests: dedupePorts, hasMixedProtocols, and a non-ingress-UDP Service case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
universe-ops pushed a commit that referenced this pull request Jul 15, 2026
PR #362 threaded UDP protocol + a stack-level serviceType, but the Service was
still built solely from the ingress container's ports, and on GKE a mixed
TCP+UDP LoadBalancer failed to provision (LoadBalancerMixedProtocolNotSupported).
Both blocked self-hosting a LiveKit SFU alongside the HTTP backend in one stack.

- Service ports now include non-ingress run containers' ports (they share the
  pod's network namespace and the Service selects the pod), so a sidecar SFU's
  7880/7882 land on the same Service as backend:3000. Ingress ports win on a
  port-number clash; ports are deduped.
- When the emitted LoadBalancer Service mixes TCP and UDP, annotate it with
  cloud.google.com/l4-rbs: "enabled" so GKE uses the RBS/NEG-based NetLB, which
  supports mixed protocols (verified live: one external IP fronting
  7880/7881 TCP + 7882/UDP). No config passthrough exists for Service
  annotations, so this must be set by sc.

Tests: dedupePorts, hasMixedProtocols, and a non-ingress-UDP Service case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Ilya Sadykov <smecsia@gmail.com>
@smecsia

smecsia commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

@RuslanMikailov transferred your follow-up commit ab9ac935 into the SC-owned branch — it's now on #363 as a second commit (author preserved as you, signed for branch protection). It adds:

  • non-ingress container ports on the Service (deduped, ingress wins on clash) — so a sidecar SFU's 7880/7882 land on the same Service as the HTTP backend;
  • cloud.google.com/l4-rbs: "enabled" on the Service when it's a LoadBalancer with mixed TCP+UDP — switches GKE to the RBS/NEG NetLB that actually supports mixed protocols (fixes LoadBalancerMixedProtocolNotSupported).

New preview version (both commits, for final validation):
2026.7.5-pre.3a45d05-preview.3a45d05

SIMPLE_CONTAINER_VERSION=2026.7.5-pre.3a45d05-preview.3a45d05 \
  bash -c "$(curl -fsSL https://dist.simple-container.com/sc.sh)"
sc --version   # -> 2026.7.5-pre.3a45d05-preview.3a45d05

(linux-amd64 + darwin-arm64 published.)

Please re-deploy your LiveKit stack with this build and confirm end-to-end: the mixed TCP+UDP LoadBalancer provisions on GKE with one external IP, and media (UDP) actually flows. Once you give the 👍, we finish review + merge on #363.

The Service applied args.ExternalTrafficPolicy (simple_container.go) and the
DeploymentArgs carried it, but nothing populated it for a client stack — the
field only existed on CaddyConfig, and CloudExtras had no way to set it. So
`cloudExtras.externalTrafficPolicy` was silently dropped and the Service stayed
Cluster. Thread it like serviceType: CloudExtras -> DeploymentConfig ->
kubeArgs -> Service.

Needed for self-hosted WebRTC media behind the LB: externalTrafficPolicy: Local
preserves the LB IP (and client source IP) on the return path, so clients accept
the SFU downlink. Cluster SNATs to a node IP, which WebRTC rejects (dtls timeout,
call drops ~2 min in).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@smecsia

smecsia commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

@RuslanMikailov transferred your third commit (6f54c65e, externalTrafficPolicy from cloudExtras) onto the owned branch #363 — author preserved, signed.

⚠️ One correction: that commit only threaded externalTrafficPolicy through the custom-stack path (kube_run.go), but on GKE Autopilot (gke_autopilot_stack.go) only ServiceType was forwarded, so externalTrafficPolicy was silently dropped to the default Cluster → SNAT to a node IP → WebRTC return media breaks (exactly what Local fixes). I added the missing line as a separate commit (an exact copy of the ServiceType line) so the setting actually reaches the Service on Autopilot. If your LiveKit runs on a custom GKE stack rather than Autopilot, it's simply a no-op.

New preview version (all 4 commits):
2026.7.5-pre.509b914-preview.509b914

SIMPLE_CONTAINER_VERSION=2026.7.5-pre.509b914-preview.509b914 \
  bash -c "$(curl -fsSL https://dist.simple-container.com/sc.sh)"
sc --version   # -> 2026.7.5-pre.509b914-preview.509b914

(linux-amd64 + darwin-arm64 published.)

For the final check on GKE: set serviceType: LoadBalancer and externalTrafficPolicy: Local in cloudExtras, deploy LiveKit, and confirm media (UDP) actually flows and the call holds (no dtls-timeout drop). Once you give the 👍 we'll finalize review and merge #363.

smecsia added a commit that referenced this pull request Jul 21, 2026
)

Re-established on an SC-owned branch so CI runs with full secret access
(the original fork PR #362 could not run secret-gated jobs). Original
author preserved: **@RuslanMikailov** (`Ruslan
<ruslan.mikailov@fulldive.com>`). Supersedes #362.

## What
`sc` parses the compose file and generates its own k8s Deployment +
Service via Pulumi; during that translation the port **protocol was
discarded**. `toRunPorts` mapped every compose port to a bare `int`, so
`"7882:7882/udp"` silently became a TCP port and UDP never reached the
pod. This blocks a WebRTC media server (e.g. LiveKit SFU), where
signaling is TCP (wss) but media is UDP.

## Change
Thread the port protocol end-to-end:
- `CloudRunContainer.Ports` is now `[]k8s.ContainerPort{Port, Protocol}`
instead of `[]int`; `toRunPorts` preserves the compose protocol (`udp` →
`"UDP"`; tcp/unspecified → `""`).
- New `ports.go` helpers set `Protocol` on
`ContainerPortArgs`/`ServicePortArgs`, **only for non-TCP** protocols,
so TCP-only specs stay byte-for-byte identical (TCP keeps `http-<port>`
names; UDP gets `udp-<port>`).
- `autoTCPReadinessProbe` no longer attaches a TCP/HTTP probe to a
UDP-only port (resolves the old `// TODO: support non-http ports`).
- `serviceType` is now expressible via `cloudExtras` and threaded
`CloudExtras → DeploymentConfig → kubeArgs`, letting a service turn its
own Service into a `LoadBalancer` (required for protocols Caddy cannot
proxy).

A service exposing both TCP and UDP emits ONE mixed-protocol
LoadBalancer Service (GA since k8s 1.26) rather than two Services, so a
single external IP fronts the whole service. Announcing that IP in ICE
candidates (LiveKit `rtc.use_external_ip` / `node_ip`) is the app's job,
not `sc`'s.

**Backward compatible:** ports with no protocol behave exactly as
before; `serviceType` defaults to `ClusterIP`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Ruslan <ruslan.mikailov@fulldive.com>
Signed-off-by: Ilya Sadykov <smecsia@gmail.com>
Co-authored-by: Ruslan <ruslan.mikailov@fulldive.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: universe-ops <177390656+universe-ops@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants