feat(k8s): expose UDP ports on cloud-compose via mixed-protocol LB#362
feat(k8s): expose UDP ports on cloud-compose via mixed-protocol LB#362RuslanMikailov wants to merge 3 commits into
Conversation
sc parses the compose file and generates its own k8s Deployment +
Service via Pulumi; during that translation the port protocol was
discarded. toRunPorts mapped every compose port to a bare int, so
"7882:7882/udp" silently became a TCP port and UDP never reached the
pod. This blocks deploying a WebRTC media server (e.g. LiveKit SFU),
where signaling is TCP (wss) but media is UDP.
Thread the port protocol end-to-end:
- CloudRunContainer.Ports is now []k8s.ContainerPort{Port, Protocol}
instead of []int; toRunPorts preserves the compose protocol
(udp -> "UDP"; tcp/unspecified -> "").
- New ports.go helpers set Protocol on ContainerPortArgs and
ServicePortArgs, but only for non-TCP protocols, so TCP-only specs
stay byte-for-byte identical (TCP keeps http-<port> names; UDP gets
udp-<port>).
- autoTCPReadinessProbe no longer attaches a TCP/HTTP probe to a
UDP-only port (resolves the old "// TODO: support non-http ports").
- serviceType is now expressible via cloudExtras and threaded
CloudExtras -> DeploymentConfig -> kubeArgs, letting a service turn
its own Service into a LoadBalancer (required for protocols Caddy
cannot proxy).
A service exposing both TCP and UDP emits ONE mixed-protocol
LoadBalancer Service (GA since k8s 1.26, supported on recent GKE)
rather than splitting into two Services, so a single external IP
fronts the whole service. Announcing that IP in ICE candidates
(LiveKit rtc.use_external_ip / node_ip) is the app's job, not sc's.
Backward compatible: ports with no protocol behave exactly as before,
and serviceType defaults to ClusterIP.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Ruslan <ruslan.mikailov@fulldive.com>
|
Thanks for this @RuslanMikailov — reviewed it in depth and the approach is correct. Why we re-established this as #363CI here can't run the secret-gated jobs (build/test/security) on a PR from a fork — GitHub withholds Actions secrets from fork PRs, which is why the builds "failed" here. So I pushed your commit to an SC-owned branch and opened #363 to run the full pipeline. Your authorship is preserved (commit author stays Review summary (the change is solid)
Two non-blocking things to verify before merge
Follow-up on #363. |
|
@RuslanMikailov I published a preview build of this branch so you can test the UDP change end-to-end on your LiveKit stack before merge — no need to build Preview version: Install it (both linux-amd64 and darwin-arm64 are published): SIMPLE_CONTAINER_VERSION=2026.7.4-pre.482661-preview.0482661 \
bash -c "$(curl -fsSL https://dist.simple-container.com/sc.sh)"
sc --version # should print 2026.7.4-pre.482661-preview.0482661Then deploy your media server as usual. To exercise the fix, in your
Could you confirm on your GKE cluster that:
Built from |
PR simple-container-com#362 threaded UDP protocol + a stack-level serviceType, but the Service was still built solely from the ingress container's ports, and on GKE a mixed TCP+UDP LoadBalancer failed to provision (LoadBalancerMixedProtocolNotSupported). Both blocked self-hosting a LiveKit SFU alongside the HTTP backend in one stack. - Service ports now include non-ingress run containers' ports (they share the pod's network namespace and the Service selects the pod), so a sidecar SFU's 7880/7882 land on the same Service as backend:3000. Ingress ports win on a port-number clash; ports are deduped. - When the emitted LoadBalancer Service mixes TCP and UDP, annotate it with cloud.google.com/l4-rbs: "enabled" so GKE uses the RBS/NEG-based NetLB, which supports mixed protocols (verified live: one external IP fronting 7880/7881 TCP + 7882/UDP). No config passthrough exists for Service annotations, so this must be set by sc. Tests: dedupePorts, hasMixedProtocols, and a non-ingress-UDP Service case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PR #362 threaded UDP protocol + a stack-level serviceType, but the Service was still built solely from the ingress container's ports, and on GKE a mixed TCP+UDP LoadBalancer failed to provision (LoadBalancerMixedProtocolNotSupported). Both blocked self-hosting a LiveKit SFU alongside the HTTP backend in one stack. - Service ports now include non-ingress run containers' ports (they share the pod's network namespace and the Service selects the pod), so a sidecar SFU's 7880/7882 land on the same Service as backend:3000. Ingress ports win on a port-number clash; ports are deduped. - When the emitted LoadBalancer Service mixes TCP and UDP, annotate it with cloud.google.com/l4-rbs: "enabled" so GKE uses the RBS/NEG-based NetLB, which supports mixed protocols (verified live: one external IP fronting 7880/7881 TCP + 7882/UDP). No config passthrough exists for Service annotations, so this must be set by sc. Tests: dedupePorts, hasMixedProtocols, and a non-ingress-UDP Service case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Ilya Sadykov <smecsia@gmail.com>
|
@RuslanMikailov transferred your follow-up commit
New preview version (both commits, for final validation): SIMPLE_CONTAINER_VERSION=2026.7.5-pre.3a45d05-preview.3a45d05 \
bash -c "$(curl -fsSL https://dist.simple-container.com/sc.sh)"
sc --version # -> 2026.7.5-pre.3a45d05-preview.3a45d05(linux-amd64 + darwin-arm64 published.) Please re-deploy your LiveKit stack with this build and confirm end-to-end: the mixed TCP+UDP |
The Service applied args.ExternalTrafficPolicy (simple_container.go) and the DeploymentArgs carried it, but nothing populated it for a client stack — the field only existed on CaddyConfig, and CloudExtras had no way to set it. So `cloudExtras.externalTrafficPolicy` was silently dropped and the Service stayed Cluster. Thread it like serviceType: CloudExtras -> DeploymentConfig -> kubeArgs -> Service. Needed for self-hosted WebRTC media behind the LB: externalTrafficPolicy: Local preserves the LB IP (and client source IP) on the return path, so clients accept the SFU downlink. Cluster SNATs to a node IP, which WebRTC rejects (dtls timeout, call drops ~2 min in). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
@RuslanMikailov transferred your third commit (
New preview version (all 4 commits): SIMPLE_CONTAINER_VERSION=2026.7.5-pre.509b914-preview.509b914 \
bash -c "$(curl -fsSL https://dist.simple-container.com/sc.sh)"
sc --version # -> 2026.7.5-pre.509b914-preview.509b914(linux-amd64 + darwin-arm64 published.) For the final check on GKE: set |
) Re-established on an SC-owned branch so CI runs with full secret access (the original fork PR #362 could not run secret-gated jobs). Original author preserved: **@RuslanMikailov** (`Ruslan <ruslan.mikailov@fulldive.com>`). Supersedes #362. ## What `sc` parses the compose file and generates its own k8s Deployment + Service via Pulumi; during that translation the port **protocol was discarded**. `toRunPorts` mapped every compose port to a bare `int`, so `"7882:7882/udp"` silently became a TCP port and UDP never reached the pod. This blocks a WebRTC media server (e.g. LiveKit SFU), where signaling is TCP (wss) but media is UDP. ## Change Thread the port protocol end-to-end: - `CloudRunContainer.Ports` is now `[]k8s.ContainerPort{Port, Protocol}` instead of `[]int`; `toRunPorts` preserves the compose protocol (`udp` → `"UDP"`; tcp/unspecified → `""`). - New `ports.go` helpers set `Protocol` on `ContainerPortArgs`/`ServicePortArgs`, **only for non-TCP** protocols, so TCP-only specs stay byte-for-byte identical (TCP keeps `http-<port>` names; UDP gets `udp-<port>`). - `autoTCPReadinessProbe` no longer attaches a TCP/HTTP probe to a UDP-only port (resolves the old `// TODO: support non-http ports`). - `serviceType` is now expressible via `cloudExtras` and threaded `CloudExtras → DeploymentConfig → kubeArgs`, letting a service turn its own Service into a `LoadBalancer` (required for protocols Caddy cannot proxy). A service exposing both TCP and UDP emits ONE mixed-protocol LoadBalancer Service (GA since k8s 1.26) rather than two Services, so a single external IP fronts the whole service. Announcing that IP in ICE candidates (LiveKit `rtc.use_external_ip` / `node_ip`) is the app's job, not `sc`'s. **Backward compatible:** ports with no protocol behave exactly as before; `serviceType` defaults to `ClusterIP`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Ruslan <ruslan.mikailov@fulldive.com> Signed-off-by: Ilya Sadykov <smecsia@gmail.com> Co-authored-by: Ruslan <ruslan.mikailov@fulldive.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: universe-ops <177390656+universe-ops@users.noreply.github.com>
sc parses the compose file and generates its own k8s Deployment + Service via Pulumi; during that translation the port protocol was discarded. toRunPorts mapped every compose port to a bare int, so "7882:7882/udp" silently became a TCP port and UDP never reached the pod. This blocks deploying a WebRTC media server (e.g. LiveKit SFU), where signaling is TCP (wss) but media is UDP.
Thread the port protocol end-to-end:
A service exposing both TCP and UDP emits ONE mixed-protocol LoadBalancer Service (GA since k8s 1.26, supported on recent GKE) rather than splitting into two Services, so a single external IP fronts the whole service. Announcing that IP in ICE candidates (LiveKit rtc.use_external_ip / node_ip) is the app's job, not sc's.
Backward compatible: ports with no protocol behave exactly as before, and serviceType defaults to ClusterIP.