Skip to content

feat(operator): per-component mainContainerName override for DGD components#11421

Open
HanFa wants to merge 1 commit into
ai-dynamo:mainfrom
HanFa:fangh/dgd-main-container-name
Open

feat(operator): per-component mainContainerName override for DGD components#11421
HanFa wants to merge 1 commit into
ai-dynamo:mainfrom
HanFa:fangh/dgd-main-container-name

Conversation

@HanFa

@HanFa HanFa commented Jul 8, 2026

Copy link
Copy Markdown

Overview:

Adds an optional, backward-compatible spec.mainContainerName field (default "main") to DynamoComponentDeploymentSharedSpec, letting a DGD/DCD component rename its primary workload container while keeping every main-container semantic (default injection, first position in containers, runtime discovery identity).

Motivation: platform integrations often have fleet-wide conventions keyed on the primary container name — log routing/filtering, terminal attach, per-container CPU/memory metrics. Today main is a hard operator/runtime contract: submitting a differently-named container demotes it to a sidecar and auto-generates a separate main, so integrators must special-case Dynamo pods across their observability stack. An optional per-component override removes those integration workarounds without breaking any existing deployment: when the field is omitted, rendered objects are byte-identical to before (one scoped exception listed under Behavior changes below).

Details:

  • API (v1beta1): new mainContainerName on DynamoComponentDeploymentSharedSpec with DNS-1123 label validation, plus a GetMainContainerName() resolver; the field is immutable after creation (CEL transition rule). Field docs codify the ambiguity rules and discovery behavior.
  • Rendering: base-container naming, user-container merge, sidecar split, DGD-level env defaults, LWS checkMainContainer, and checkpoint target-container defaults all resolve the per-component name. The renamed main container keeps index 0, so positional Containers[0] consumers (failover, DRA, GMS sidecar injection, backends) are unaffected. Operator-owned identity envs are re-asserted after user env merges.
  • Discovery/runtime: in container-scoped kubernetes discovery mode the operator injects DYN_MAIN_CONTAINER_NAME alongside CONTAINER_NAME (only when the name differs from "main", keeping default renders byte-identical), and stamps nvidia.com/dynamo-main-container-name on rendered pods (including the Grove path). The Rust runtime resolves each remote pod's main container from that pod's own annotation, so mixed-name fleets and rolling renames stay mutually visible; a watcher's own env only determines its own identity. DYN_MAIN_CONTAINER_NAME participates in checkpoint/restore env refresh.
  • Planner: the SLA planner's get_main_container resolves the name from the component spec instead of the literal "main".
  • Validation webhook: rejects a custom mainContainerName combined with a podTemplate container literally named "main" (ambiguous — it would silently become a sidecar); rejects frontendSidecar equal to the main-container name when mainContainerName is explicitly set; rejects a mainContainerName that collides with generated intra-pod-failover engine names. Standalone DCDs are validated at v1beta1 via the dual-version webhook architecture that landed upstream in refactor(validation): make DCD validation structural #11479; this PR migrates the helm webhook entry to the v1beta1 path and adds admission-chain coverage.
  • Conversion (v1alpha1 storage version): the v1beta1-only field is preserved through the sparse spec payload per api/CONVERSION.md, and conversion keys the semantic main container on the effective name. The field-coverage tripwire and round-trip fuzz tests pass. This work also surfaced and fixes a pre-existing round-trip bug: the hub main container's volume-mount order was lost when a compilation-cache mount was flattened (extracted repro fails on HEAD without the fix — see TestBugDCD_HubMainContainerVolumeMountOrderRoundTrips). Happy to split that fix into its own PR if preferred.
  • Manifests: CRDs and helm chart CRD copies regenerated via make manifests.

Behavior changes (review hardening — full commit-per-finding map in the comments)

  1. Generated frontend sidecar env fix: the injected frontend sidecar previously rendered with CONTAINER_NAME=main; it now gets its own container name (and, for custom-named components, the parent's DYN_MAIN_CONTAINER_NAME). This changes rendered output for existing frontendSidecar users in container discovery mode — the old value was incorrect (it made the sidecar classify as the pod-identity holder).
  2. Sidecar-image CEL rule removed: the yq-injected self.name == "main" || has(self.image) rule on containers hard-coded the main name and cannot be expressed correctly at item scope; imageless sidecars now fail at reconcile time instead of admission. Rationale comment on the PR.
  3. frontendSidecar: main: still admitted for existing specs (no mainContainerName); rejected only when mainContainerName is explicitly set.
  4. mainContainerName is immutable after creation (CEL transition rule; add/change/remove all rejected).
  5. Standalone DCD v1beta1 validation: the dual-version DCD webhook landed upstream in refactor(validation): make DCD validation structural #11479 while this PR was in review; the rebase adopts it, and this PR now migrates the helm webhook entry to the v1beta1 path/apiVersions and adds admission-chain coverage for the mainContainerName checks on standalone DCDs. Upstream deliberately skips the EPP InferencePool availability check for standalone DCDs; that decision is kept.

Validation

  • go build ./..., go vet ./..., gofmt clean; golangci-lint (CI-pinned v1.64.8) clean
  • go test ./api/... ./internal/... all green, including the envtest controller suite and conversion round-trip fuzz (10k iterations, plus re-runs with fixed seeds after the conversion fixes)
  • Rust: cargo test -p dynamo-runtime --lib discovery::kube::utils → 9 passed (mixed-name remote-pod cases included)
  • Python: planner connector unit tests green; snapshot restore env replace/clear cases added
  • Local kind e2e (operator image built from this branch, helm chart install): custom-name components render with the renamed main at index 0 + discovery annotation; ambiguity, frontendSidecar collision, and both immutability directions rejected at admission; v1alpha1 legacy webhook path admits with deprecation warning; previously-admitted ambiguous standalone v1beta1 DCD now rejected (failover-name collision is covered by webhook table tests)

Where should the reviewer start?

  1. deploy/operator/api/v1beta1/dynamocomponentdeployment_types.go — the field contract, resolver, and immutability rule
  2. deploy/operator/internal/dynamo/graph.go + v1beta1_helpers.go — render-path threading and annotation stamping
  3. deploy/operator/internal/webhook/validation/ — shared v1beta1 checks and the DCD dual-version handler
  4. deploy/operator/api/v1alpha1/shared_spec_conversion.go — sparse-payload preservation + semantic-main threading
  5. lib/runtime/src/discovery/kube/utils.rs — per-pod remote identity resolution

Related Issues

🚫 This PR is NOT linked to an issue:

  • Confirmed — no related issue

Opening as a draft per the contribution guide's issue-first process: this is a concrete implementation of the proposal to make the main-container name configurable. If maintainers prefer, I'll file a design issue / enhancements RFC and link it here before marking ready for review.

🤖 Implemented with Claude Code

Summary by CodeRabbit

  • New Features

    • Added a configurable main-container name for component deployments, with validation and API/docs updates.
    • Pod generation and discovery now honor the configured main container instead of assuming "main".
  • Bug Fixes

    • Preserved custom main-container names and volume-mount ordering during round-trip conversions.
    • Fixed checkpoint and restore defaults so they target the resolved main container correctly.
  • Tests

    • Added coverage for custom main-container naming, validation rules, checkpoint behavior, and discovery identity routing.

@copy-pr-bot

copy-pr-bot Bot commented Jul 8, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@HanFa HanFa temporarily deployed to external_collaborator July 8, 2026 19:39 — with GitHub Actions Inactive
@HanFa HanFa had a problem deploying to external_collaborator July 8, 2026 19:39 — with GitHub Actions Failure
@github-actions github-actions Bot added feat deployment::k8s Relates to dynamo deployment in kubernetes planner labels Jul 8, 2026
@datadog-official

This comment has been minimized.

@HanFa HanFa temporarily deployed to external_collaborator July 8, 2026 20:13 — with GitHub Actions Inactive
@HanFa HanFa temporarily deployed to external_collaborator July 8, 2026 20:35 — with GitHub Actions Inactive
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Jul 8, 2026
@HanFa

HanFa commented Jul 8, 2026

Copy link
Copy Markdown
Author

Local e2e verification (kind, operator built from this branch)

Ran a full local e2e on a kind cluster with the operator image built from this branch (helm chart install, CRDs server-side applied). Evidence below; happy to add any of this as an automated e2e spec if maintainers want it.

case result
Custom name rendering — DCD with mainContainerName: engine, user engine container + one sidecar ✅ Deployment renders [engine, extra-sidecar]: renamed main at index 0, no main container anywhere, user image/command/env merged into engine
Runtime identity envs — container-scoped discovery annotation set kubectl exec -c engineCONTAINER_NAME=engine, DYN_MAIN_CONTAINER_NAME=engine; correctly absent in default pod-mode
Default regression — component without the field ✅ renders container main, pod Running — unchanged contract
Ambiguity rejectionmainContainerName: engine plus a podTemplate container literally named main (via DGD) ✅ admission rejected: spec.components[0].mainContainerName: Invalid value: "engine": is ambiguous: podTemplate.spec.containers also declares a container named "main"...
frontendSidecar collisionfrontendSidecar equal to the main-container name (via DGD) ✅ admission rejected: must designate a container other than the main container "engine"
Full graph path — valid DGD with a custom-named frontend component ✅ DGD → child DCD → Deployment → Running pod with container engine

Observation (pre-existing, not introduced here): the standalone-DCD validating webhook registers v1alpha1 only, so with matchPolicy: Equivalent a directly-created v1beta1 DCD is validated through the v1alpha1 path and the v1beta1-only checks (ambiguity/frontendSidecar) don't run — the object is admitted and the main-named container becomes a sidecar as documented. The DGD path (the primary user API) enforces both rejections. Worth deciding separately whether the DCD validation webhook should also register v1beta1.

Setup notes for anyone reproducing locally: CRDs need kubectl apply --server-side (too large for client-side apply), the chart's crds/ overflow the helm release-secret limit when installing the operator subchart standalone (apply them separately), and the manager needs the dynamo-deployment-env secret plus --set discoveryBackend=kubernetes.

🤖 Verified with Claude Code

@HanFa HanFa marked this pull request as ready for review July 8, 2026 21:18
@HanFa HanFa requested review from a team as code owners July 8, 2026 21:18
@HanFa HanFa requested a review from a team July 8, 2026 21:18

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

Open in Devin Review

Comment thread lib/runtime/src/discovery/kube/utils.rs Outdated
Comment thread deploy/operator/internal/dynamo/component_common.go Outdated
@coderabbitai

coderabbitai Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

This PR introduces a configurable mainContainerName field replacing the hardcoded "main" container assumption throughout the Dynamo operator, CRDs, v1beta1 API types, conversion logic, controllers, pod-spec generation, checkpoint/restore handling, webhook validation, planner monitoring, and Kubernetes discovery utilities, with accompanying tests and documentation updates.

Changes

Main container name override

Layer / File(s) Summary
API types, CRD schema, and docs
deploy/operator/api/v1beta1/dynamocomponentdeployment_types.go, deploy/helm/charts/.../*.yaml, deploy/operator/config/crd/bases/*.yaml, docs/kubernetes/api-reference.md
Adds MainContainerName field and GetMainContainerName() method; updates CRD schemas and docs with validation constraints and behavioral contract.
Pod spec generation and sidecar merging
deploy/operator/internal/dynamo/component_common.go, graph.go, v1beta1_helpers.go, graph_main_container_name_test.go
ComponentContext, container merging, sidecar extraction, and ensureMainContainer/GetMainContainer now use the resolved main container name; env var propagates identity; new tests validate override behavior.
Controller validation and checkpoint restore defaults
deploy/operator/internal/controller/dynamocomponentdeployment_controller.go, dynamographdeployment_controller.go, checkpoint/*.go, and tests
checkMainContainer and checkpoint restore-target/target-container defaulting now use the resolved main container name; comments clarify defaulting semantics; new tests added.
v1alpha1 conversion round-tripping
deploy/operator/api/v1alpha1/shared_spec_conversion.go, conversion tests
Preserves and restores MainContainerName and main container volume-mount order across v1alpha1 hub/spoke conversion; adds coverage and round-trip tests.
Webhook validation
deploy/operator/internal/webhook/validation/shared_v1beta1.go, shared_test.go
Rejects ambiguous main-container/sidecar name conflicts and frontendSidecar equal to mainContainerName.
Planner and Kubernetes discovery resolution
components/src/dynamo/planner/monitoring/dgd_services.py, lib/runtime/src/discovery/kube/utils.rs, tests
Resolves main container name from component config/env var for planner container selection and pod-identity discovery routing.

Estimated code review effort: 4 (Complex) | ~60 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title is concise and accurately describes the main change: adding a per-component mainContainerName override.
Description check ✅ Passed The description follows the template well and includes overview, details, reviewer start points, and the required related-issues section.

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (4)
lib/runtime/src/discovery/kube/utils.rs (1)

26-31: 🚀 Performance & Scalability | 🔵 Trivial | ⚖️ Poor tradeoff

Consider caching the env var lookup.

main_container_name() re-reads and re-parses the environment variable on every cr_name() call. Since DYN_MAIN_CONTAINER_NAME is set once at process start by the operator, caching the resolved value (e.g., with std::sync::OnceLock) would avoid repeated syscalls on what could be a hot discovery path.

♻️ Suggested refactor
+use std::sync::OnceLock;
+
+static MAIN_CONTAINER_NAME: OnceLock<String> = OnceLock::new();
+
 fn main_container_name() -> String {
-    match std::env::var(DYN_MAIN_CONTAINER_NAME) {
-        Ok(name) if !name.trim().is_empty() => name,
-        _ => DEFAULT_MAIN_CONTAINER_NAME.to_string(),
-    }
+    MAIN_CONTAINER_NAME
+        .get_or_init(|| match std::env::var(DYN_MAIN_CONTAINER_NAME) {
+            Ok(name) if !name.trim().is_empty() => name,
+            _ => DEFAULT_MAIN_CONTAINER_NAME.to_string(),
+        })
+        .clone()
 }

Note: this would need to be reconciled with tests that mutate the env var at runtime (temp_env::with_var), since a cached value would no longer reflect changes after the first call.

Also applies to: 68-77

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/runtime/src/discovery/kube/utils.rs` around lines 26 - 31,
`main_container_name()` currently re-reads `DYN_MAIN_CONTAINER_NAME` on every
`cr_name()` call, which is wasteful on a hot path. Cache the resolved container
name once using `std::sync::OnceLock` (or equivalent) inside
`main_container_name()` and have `cr_name()` reuse that cached value. Be sure to
account for the `temp_env::with_var` tests by either adjusting those tests to
avoid runtime env mutation or by isolating the cached lookup so tests can still
validate both default and overridden behavior.
components/src/dynamo/planner/monitoring/dgd_services.py (1)

51-58: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Whitespace handling differs from the Rust counterpart.

_resolve_main_container_name only guards against falsy values (None/""), but the Rust main_container_name() also trims whitespace before treating the value as empty. A whitespace-only mainContainerName (e.g. " ") would be treated as a valid custom name here, but would fall back to "main" in the Rust discovery layer — causing the planner to look for a different container than the Rust runtime treats as "main".

🔧 Suggested fix
 def _resolve_main_container_name(component: dict) -> str:
     """Return the component's main container name.

     v1beta1 components may rename the main container via
     ``mainContainerName``; fall back to the default ``main`` when the
     field is unset or empty.
     """
-    return component.get("mainContainerName") or MAIN_CONTAINER_NAME
+    name = component.get("mainContainerName")
+    return name.strip() if name and name.strip() else MAIN_CONTAINER_NAME
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/src/dynamo/planner/monitoring/dgd_services.py` around lines 51 -
58, Update _resolve_main_container_name in dgd_services.py so it matches the
Rust main_container_name() behavior by trimming whitespace before deciding
whether mainContainerName is usable. Right now the function only falls back on
None or empty strings, so whitespace-only values are treated as valid; normalize
the component["mainContainerName"] value with stripping and return
MAIN_CONTAINER_NAME when the trimmed result is empty. Keep the change localized
to _resolve_main_container_name and preserve the existing fallback behavior for
unset or blank names.
deploy/operator/api/v1alpha1/shared_spec_conversion.go (1)

1736-1762: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Duplicate reorder-with-bailout logic.

reorderVolumeMountsToPreserved (1788-1810) reimplements the same "remove-and-append, bail out on any mismatch" algorithm already used in restoreSharedPodTemplateContainerOrder (1736-1762), differing only in the match key (Name vs Name+MountPath). Consider extracting a small generic helper parameterized by a key-extraction function to avoid maintaining two copies of this matching logic.

♻️ Example generic helper
func reorderToPreserved[T any, K comparable](current, preserved []T, key func(T) K) []T {
	if len(current) != len(preserved) {
		return current
	}
	remaining := slices.Clone(current)
	out := make([]T, 0, len(current))
	for _, p := range preserved {
		matched := false
		for i := range remaining {
			if key(remaining[i]) != key(p) {
				continue
			}
			out = append(out, remaining[i])
			remaining = slices.Delete(remaining, i, i+1)
			matched = true
			break
		}
		if !matched {
			return current
		}
	}
	return out
}

Container-order and volume-mount-order call sites then supply their own key function (e.g., composite Name+MountPath key for mounts).

Also applies to: 1764-1811

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/operator/api/v1alpha1/shared_spec_conversion.go` around lines 1736 -
1762, The container-order restore logic in
restoreSharedPodTemplateContainerOrder is duplicated by
reorderVolumeMountsToPreserved with only the matching key changed, so extract
the shared remove-and-append, bail-out-on-mismatch algorithm into a small
generic helper. Make the helper parameterized by a key-extraction function, then
have restoreSharedPodTemplateContainerOrder and reorderVolumeMountsToPreserved
call it with their respective keys (Name for containers, composite
Name+MountPath for mounts) to keep the behavior identical while removing the
duplicate matching logic.
deploy/operator/internal/dynamo/component_common.go (1)

159-175: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Hoist repeated context.mainContainerName() call.

Called twice back-to-back (Lines 162, 169) for the same value; hoisting into a local avoids the redundant method call and slightly improves readability.

♻️ Proposed refactor
 	if context.Discovery.Mode == configv1alpha1.KubeDiscoveryModeContainer {
+		mainContainerName := context.mainContainerName()
 		container.Env = append(container.Env, corev1.EnvVar{
 			Name:  "CONTAINER_NAME",
-			Value: context.mainContainerName(),
+			Value: mainContainerName,
 		})
 		// Tell the runtime which container name carries main-container
 		// (pod-level) identity, so a renamed main container keeps the same
 		// discovery identity as the default "main".
 		container.Env = append(container.Env, corev1.EnvVar{
 			Name:  commonconsts.DynamoMainContainerEnvVar,
-			Value: context.mainContainerName(),
+			Value: mainContainerName,
 		})
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/operator/internal/dynamo/component_common.go` around lines 159 - 175,
Hoist the repeated context.mainContainerName() call in the
KubeDiscoveryModeContainer branch of the container env setup to a local
variable, then reuse it for both the CONTAINER_NAME and
commonconsts.DynamoMainContainerEnvVar env vars. Keep the logic in the same
block inside the function that builds the container env so the value is computed
once and the intent is clearer.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@components/src/dynamo/planner/monitoring/dgd_services.py`:
- Around line 51-58: Update _resolve_main_container_name in dgd_services.py so
it matches the Rust main_container_name() behavior by trimming whitespace before
deciding whether mainContainerName is usable. Right now the function only falls
back on None or empty strings, so whitespace-only values are treated as valid;
normalize the component["mainContainerName"] value with stripping and return
MAIN_CONTAINER_NAME when the trimmed result is empty. Keep the change localized
to _resolve_main_container_name and preserve the existing fallback behavior for
unset or blank names.

In `@deploy/operator/api/v1alpha1/shared_spec_conversion.go`:
- Around line 1736-1762: The container-order restore logic in
restoreSharedPodTemplateContainerOrder is duplicated by
reorderVolumeMountsToPreserved with only the matching key changed, so extract
the shared remove-and-append, bail-out-on-mismatch algorithm into a small
generic helper. Make the helper parameterized by a key-extraction function, then
have restoreSharedPodTemplateContainerOrder and reorderVolumeMountsToPreserved
call it with their respective keys (Name for containers, composite
Name+MountPath for mounts) to keep the behavior identical while removing the
duplicate matching logic.

In `@deploy/operator/internal/dynamo/component_common.go`:
- Around line 159-175: Hoist the repeated context.mainContainerName() call in
the KubeDiscoveryModeContainer branch of the container env setup to a local
variable, then reuse it for both the CONTAINER_NAME and
commonconsts.DynamoMainContainerEnvVar env vars. Keep the logic in the same
block inside the function that builds the container env so the value is computed
once and the intent is clearer.

In `@lib/runtime/src/discovery/kube/utils.rs`:
- Around line 26-31: `main_container_name()` currently re-reads
`DYN_MAIN_CONTAINER_NAME` on every `cr_name()` call, which is wasteful on a hot
path. Cache the resolved container name once using `std::sync::OnceLock` (or
equivalent) inside `main_container_name()` and have `cr_name()` reuse that
cached value. Be sure to account for the `temp_env::with_var` tests by either
adjusting those tests to avoid runtime env mutation or by isolating the cached
lookup so tests can still validate both default and overridden behavior.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2c53c531-9f99-4a48-acea-5f3ba31ed005

📥 Commits

Reviewing files that changed from the base of the PR and between c3c7e52 and 42ea484.

📒 Files selected for processing (28)
  • components/src/dynamo/planner/monitoring/dgd_services.py
  • components/src/dynamo/planner/tests/unit/test_kubernetes_connector.py
  • deploy/helm/charts/platform/components/operator/crds/nvidia.com_dynamocomponentdeployments.yaml
  • deploy/helm/charts/platform/components/operator/crds/nvidia.com_dynamographdeployments.yaml
  • deploy/operator/api/v1alpha1/conversion_field_coverage_test.go
  • deploy/operator/api/v1alpha1/dynamocomponentdeployment_conversion_test.go
  • deploy/operator/api/v1alpha1/shared_spec_conversion.go
  • deploy/operator/api/v1alpha1/shared_spec_conversion_bugs_test.go
  • deploy/operator/api/v1beta1/dynamocomponentdeployment_types.go
  • deploy/operator/config/crd/bases/nvidia.com_dynamocomponentdeployments.yaml
  • deploy/operator/config/crd/bases/nvidia.com_dynamographdeployments.yaml
  • deploy/operator/internal/checkpoint/gms_pod_validation.go
  • deploy/operator/internal/checkpoint/podspec.go
  • deploy/operator/internal/checkpoint/resource.go
  • deploy/operator/internal/consts/consts.go
  • deploy/operator/internal/controller/checkpoint_job.go
  • deploy/operator/internal/controller/dynamocomponentdeployment_controller.go
  • deploy/operator/internal/controller/dynamocomponentdeployment_controller_test.go
  • deploy/operator/internal/controller/dynamographdeployment_controller.go
  • deploy/operator/internal/controller/dynamographdeployment_controller_test.go
  • deploy/operator/internal/dynamo/component_common.go
  • deploy/operator/internal/dynamo/graph.go
  • deploy/operator/internal/dynamo/graph_main_container_name_test.go
  • deploy/operator/internal/dynamo/v1beta1_helpers.go
  • deploy/operator/internal/webhook/validation/shared_test.go
  • deploy/operator/internal/webhook/validation/shared_v1beta1.go
  • docs/kubernetes/api-reference.md
  • lib/runtime/src/discovery/kube/utils.rs

@HanFa HanFa temporarily deployed to external_collaborator July 8, 2026 21:46 — with GitHub Actions Inactive

@julienmancuso julienmancuso left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found two P1 correctness issues and three P2 completeness/backward-compatibility issues in the current mainContainerName design. The inline comments below focus on cases not covered by the current tests: mixed-name discovery, generated frontend sidecars, snapshot restore, intra-pod failover, and updates of existing resources.

Comment thread lib/runtime/src/discovery/kube/utils.rs Outdated
Comment thread deploy/operator/internal/dynamo/component_common.go Outdated
Comment thread deploy/operator/internal/dynamo/component_common.go
Comment thread deploy/operator/internal/dynamo/component_common.go
Comment thread deploy/operator/internal/webhook/validation/shared_v1beta1.go Outdated

@hhzhang16 hhzhang16 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mainContainerName is documented and exposed on standalone DCDs, but the standalone DCD validating webhook still appears to be v1alpha1-only, so the new v1beta1 shared-spec checks do not run for direct v1beta1 DCD creates/updates. That means a direct DCD can accept combinations the DGD path rejects, such as mainContainerName: engine plus a literal main container.

Can we either make the v1beta1 DCD path enforce the same validation, or narrow the PR contract/docs to say this is only validated on the DGD path and track DCD parity separately? My default would be to enforce it here if standalone DCD is intended to support this field.

@HanFa HanFa force-pushed the fangh/dgd-main-container-name branch from 8e4d033 to 3cc4f48 Compare July 9, 2026 20:28
@HanFa HanFa temporarily deployed to external_collaborator July 9, 2026 20:28 — with GitHub Actions Inactive
@HanFa HanFa temporarily deployed to external_collaborator July 9, 2026 20:48 — with GitHub Actions Inactive
@HanFa

HanFa commented Jul 9, 2026

Copy link
Copy Markdown
Author

Thanks for the thorough reviews — all findings are addressed. I force-pushed with the history reorganized into one feature-base commit (a1952e8e48) plus one commit per finding so each fix can be reviewed in isolation; the branch head is now 209b81563c.

Finding Fix
[P1] Remote main identity resolved from the observer's env (@julienmancuso) df915105a3 — the operator stamps nvidia.com/dynamo-main-container-name on rendered pods that use a custom name (Grove-path stamping completed in c615fa963a below); watchers resolve each remote pod's main container from that pod's own annotation instead of their own env. Mixed-name remote-pod test added.
[P1] Generated frontend sidecar inherits the main container's identity (@julienmancuso) 36e8fad855ComponentContext now models the current container's name separately from the component's main-container name; the generated sidecar gets CONTAINER_NAME=<sidecar name> and the correct DYN_MAIN_CONTAINER_NAME. Coverage for custom main + frontendSidecar added.
[P2] Snapshot restore misses the new identity env (@julienmancuso) a7980c584eDYN_MAIN_CONTAINER_NAME added to RESTORE_RUNTIME_ENV_NAMES; replacement and clearing both tested.
[P2] Failover engines retain the main identity env (@julienmancuso) 7019523805 strips it in the failover engine env removal set; e23114c4dd additionally rejects a mainContainerName that collides with the generated engine names when intra-pod failover is enabled.
[P2] frontendSidecar: main back-compat break (@julienmancuso) 7d96d9af29 — the collision check is scoped to an explicitly set mainContainerName; pre-existing frontendSidecar: main configs keep admitting. PR body corrected accordingly.
Immutability (@sttts) 09e68e8755 + 209b81563c — CEL transition rule; changing, adding, or removing the field after creation is rejected, on standalone DCDs and DGD component items. Details in the thread.
Standalone DCD v1beta1 validation gap (@hhzhang16) 7be6375f62 — the DCD webhook now mirrors the DGD dual-version pattern; details in a reply below.

Also in this push (hardening found while addressing the above):

  • c615fa963a — the discovery annotation is also stamped in the Grove pod-annotation path (it was previously stamped only on a DeepCopy'd component that feeds the PodSpec, so the Grove path — which composes pod annotations from the original component — lost it).
  • 555a4d320a — operator-owned identity envs (CONTAINER_NAME, DYN_MAIN_CONTAINER_NAME, DYN_KUBE_DISCOVERY_MODE) are re-asserted after user env merges, so user-supplied values can't corrupt discovery identity.
  • c05eafd287 — a stale nvidia.com/dynamo-main-container-name annotation is removed when a component reverts to the default name.
  • 6ea58a1cf5 — conversion keys the semantic main container on the effective main name (fixes a round-trip corruption when a custom-named main was edited through the v1alpha1 view); round-trip fuzz re-run.
  • af7009a550 — the yq-injected sidecar-image CEL rule is removed; rationale in a separate comment below.

Re-ran the local kind e2e on the pushed head: a custom-name DCD renders with the renamed main at index 0 and the discovery annotation; ambiguity, frontendSidecar collision, and both immutability directions reject with clear messages; a v1alpha1 create still admits through the legacy path with the deprecation warning; and the previously-admitted ambiguous standalone DCD is now rejected (see the DCD parity reply for the before/after). Earlier full e2e evidence for the render/discovery paths is in the PR comment above.

@HanFa

HanFa commented Jul 9, 2026

Copy link
Copy Markdown
Author

Why af7009a550 removes the injected sidecar-image CEL rule instead of adapting it

The operator's make manifests post-processing injected this item-scoped rule on podTemplate.spec.containers:

rule: self.name == "main" || (has(self.image) && self.image != "")
message: sidecar containers must specify a non-empty image

With spec.mainContainerName, the literal "main" is wrong: a custom-named main container that relies on the operator-injected default image is rejected at the API server and never reaches the operator (found in review; reproduced against a live CRD).

Adapting it isn't practical at the schema level:

  • an item-scoped rule evaluates one container at a time and cannot see the sibling spec.mainContainerName field, so it can't know which name is "the main";
  • a spec-level comprehension over the unbounded containers list blows the CRD CEL cost budget (this list intentionally has no maxItems — it's an external PodTemplateSpec type).

So the rule is removed rather than rewritten. Consequences:

  • imageless sidecars are no longer rejected at admission by the schema; they are still rejected by the workload API when the operator renders the pod (the failure moves from create-time to reconcile-time),
  • the initContainers image rule and the annotation rules are unchanged.

If earlier feedback is preferred, the same check can move into the validation webhook (which has full-spec context) — happy to add that here or as a follow-up.

@HanFa

HanFa commented Jul 9, 2026

Copy link
Copy Markdown
Author

@hhzhang16 Good catch — enforced in 7be6375f62, taking your default option: standalone DCDs are intended to support the field, so the v1beta1 path now runs the same validation instead of narrowing the docs.

The DCD webhook now mirrors the DGD dual-version pattern:

  • the primary handler validates the v1beta1 type at /validate/nvidia.com/v1beta1/dynamocomponentdeployments; it runs the shared v1beta1 spec validation (mainContainerName ambiguity, frontendSidecar collision, failover engine-name collision, minAvailable/GMS/EPP rules) and then replays the v1alpha1-era compatibility checks through conversion — the same two-recursion structure as the DGD validator;
  • the v1alpha1 handler stays registered at the legacy path as a convert-then-delegate wrapper (same TODO(1.5) removal note as DGD);
  • the helm webhook entry moves to apiVersions: [v1beta1] + the new path; the default matchPolicy: Equivalent keeps converting v1alpha1 requests.

Verified on a kind cluster with the operator built from this branch: your exact example — mainContainerName: engine plus a literal main container in a direct v1beta1 DCD create — was admitted before this commit and is now rejected with spec.mainContainerName: Invalid value: "engine": is ambiguous: ...; valid custom names still admit; a v1alpha1 create still admits through the legacy path with the deprecation warning.

Two behavior notes from the parity: shared-spec violations on standalone DCDs now report structural field.Error paths (previously ad-hoc strings), and the EPP InferencePool availability check now runs for standalone DCDs too (the old alpha validator had no manager handle and silently skipped it — stricter, but consistent with the DGD path).


Edit (post-rebase): while this PR was in review, #11479 (@sttts) landed this same dual-version DCD validation architecture on main, so after rebasing the enforcement now comes from upstream and the superseded implementation here was dropped. What remains in 7be6375f62: the helm webhook entry migration to the v1beta1 path, plus admission-chain tests covering the mainContainerName checks on standalone DCDs. One correction to the note above: upstream's refactor deliberately skips the EPP InferencePool availability check for standalone DCDs, and the rebase keeps that decision — so that check does not run for them, contrary to what this comment originally said.

@HanFa HanFa force-pushed the fangh/dgd-main-container-name branch from 6c81904 to 209b815 Compare July 9, 2026 21:24
@HanFa HanFa temporarily deployed to external_collaborator July 9, 2026 21:24 — with GitHub Actions Inactive
@HanFa

HanFa commented Jul 9, 2026

Copy link
Copy Markdown
Author

Rebased onto current main (was 38 commits behind; the PR previously showed conflicts). Head is now 209b81563c — same 15-commit structure, force-pushed.

Notable interaction with upstream: #11479 (@sttts) landed the dual-version structural DCD validation on main while this PR was in review — the same architecture the DCD-parity commit here carried. The rebase drops the superseded implementation in favor of upstream's; what remains in 7be6375f62 is the helm webhook entry migration to the v1beta1 path plus admission-chain tests proving standalone v1beta1 DCDs run the mainContainerName checks (ambiguity, frontendSidecar collision, failover engine-name collision). One consequence: upstream deliberately skips the EPP InferencePool availability check for standalone DCDs, and that decision is kept — the earlier note saying that check now runs for standalone DCDs no longer applies (corrected in place).

Commit links in the earlier comments have been updated to the rebased SHAs. Post-rebase verification: build + go test ./internal/... ./api/... all green (28 packages, envtest suite included), CI-pinned golangci-lint clean, make manifests/api-docs/helm-docs produce zero drift, make check passes.

@HanFa HanFa temporarily deployed to external_collaborator July 9, 2026 21:32 — with GitHub Actions Inactive
Comment thread deploy/operator/internal/dynamo/graph.go
Comment thread deploy/operator/internal/dynamo/graph.go

@julienmancuso julienmancuso left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the updated head. The original five findings are materially addressed; four new findings remain on this revision: two P1 correctness/backward-compatibility issues and two P2 validation/API-contract issues. Details are attached inline.

// reassertOperatorIdentityEnv restores operator-owned identity env vars,
// overwriting any user-supplied values of the same name.
func reassertOperatorIdentityEnv(container *corev1.Container, identity []corev1.EnvVar) {
for _, id := range identity {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Clear reserved identity variables even when the operator's expected value is absence. For a default-named component, the base intentionally omits DYN_MAIN_CONTAINER_NAME, so operatorIdentityEnv does not capture it and this loop never removes a user-supplied value added by the podTemplate merge. The local runtime can then treat another container as main while the unstamped pod tells remote watchers to fall back to main; a default-name frontend sidecar can similarly claim pod identity. Filter all reserved identity keys from the merged env first, then append only the operator's expected values. Please add default-name cases for both the main and frontend-sidecar paths.

// +kubebuilder:validation:XValidation:rule="!has(self.eppConfig) || (has(self.type) && self.type == 'epp')",message="eppConfig may only be set when type is epp"
// +kubebuilder:validation:XValidation:rule="!has(self.minAvailable) || (has(self.replicas) && self.replicas == 0) || self.minAvailable <= (has(self.replicas) ? self.replicas : 1)",message="minAvailable must be less than or equal to replicas unless replicas is 0"
// +kubebuilder:validation:XValidation:rule="!has(oldSelf.minAvailable) || (has(self.minAvailable) && self.minAvailable == oldSelf.minAvailable)",message="minAvailable is immutable after creation"
// +kubebuilder:validation:XValidation:rule="has(oldSelf.mainContainerName) == has(self.mainContainerName) && (!has(self.mainContainerName) || self.mainContainerName == oldSelf.mainContainerName)",message="mainContainerName is immutable after creation"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Enforce the effective-name immutability invariant in the shared webhook update validator as well as CEL. This transition rule only applies to the v1beta1 schema. A served v1alpha1 full replacement can drop the sparse hub-preservation annotation; old conversion still produces the custom name, new conversion falls back to main, and validateDynamoComponentDeploymentSharedSpecUpdate currently does not compare the names, so the live semantic rename is admitted. Compare old.GetMainContainerName() with new.GetMainContainerName() in the version-agnostic update path and cover v1alpha1 DCD and DGD updates that lose the preservation annotation.

Comment thread deploy/operator/Makefile
# intentionally NOT generated: with spec.mainContainerName the semantic main
# container is no longer identifiable at container-item scope, and a
# spec-level comprehension over the unbounded containers list would exceed
# the CEL cost budget. Imageless sidecars are rejected by the workload API

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Moving this failure from admission to reconciliation is a validation regression: the CR is accepted even though it can never produce a valid workload, leaving the controller to fail repeatedly. Removing the hard-coded item CEL is reasonable, but the check should move into the shared v1beta1 webhook, which can iterate podTemplate.spec.containers, skip spec.GetMainContainerName(), and require a non-empty image for every sidecar. The DCD/DGD admission-chain tests should continue asserting rejection.

// `podTemplate` must not also contain a container literally named
// `"main"` (which would be ambiguous).
//
// The field is immutable after creation (including adding or removing

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Immutability should compare the effective name, not field presence. Omitted and explicit main render identically, but the current CEL rejects both omitted → main and main → omitted. Please make the rule compare (has(field) ? field : 'main') on old and new values (and use the same semantic comparison in webhook validation). Otherwise this behaves like a presence-sensitive override while being named and documented as the effective main-container name.

// otherwise. Rendering, validation, and any consumer that needs to identify
// the main container must use this resolver instead of the MainContainerName
// constant so per-component overrides are honored.
func (s *DynamoComponentDeploymentSharedSpec) GetMainContainerName() string {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return ptr.Deref(s.MainContainerName, MainContainerName)

})
}

func TestValidateDynamoComponentDeploymentSharedSpecMainContainerName(t *testing.T) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no new tests please. Put it into the main test which is universal (likely you have to rebase as the refactor for that just landed).

// mainContainerName overrides the name of this component's main container
// (the container the operator injects its defaults into). When omitted,
// the well-known default `"main"` is used, preserving the existing
// contract. Setting it lets integrations align the primary container name

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is "existing contract"? Don't talk about implementation changes in API docs. These must use "absolute" terms to describe the semantics. There is not old, new, existing.

// +kubebuilder:validation:MinLength=1
// +kubebuilder:validation:MaxLength=63
// +kubebuilder:validation:Pattern=`^[a-z0-9]([-a-z0-9]*[a-z0-9])?$`
MainContainerName string `json:"mainContainerName,omitempty"`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still think it is misnamed. If it is not defaulted, than this is mainContainerNameOverride.

@HanFa HanFa force-pushed the fangh/dgd-main-container-name branch from 10854ce to d27fb9b Compare July 13, 2026 17:14
@HanFa HanFa requested review from a team as code owners July 13, 2026 17:14
@HanFa HanFa temporarily deployed to external_collaborator July 13, 2026 17:14 — with GitHub Actions Inactive
@HanFa HanFa force-pushed the fangh/dgd-main-container-name branch from d27fb9b to 066d5be Compare July 13, 2026 18:42
@HanFa HanFa temporarily deployed to external_collaborator July 13, 2026 18:42 — with GitHub Actions Inactive
…omponents

Add an optional, backward-compatible `spec.mainContainerNameOverride`
field (default "main") to DynamoComponentDeploymentSharedSpec, letting a
DGD/DCD component rename its primary workload container while keeping
every main-container semantic: default injection, first position in
`containers`, and runtime discovery identity. When the field is omitted,
rendered objects are byte-identical to before. The field is presence-
sensitive and immutable after creation (add, remove, or change all
rejected) -- an override, per its name.

Rendering, LWS checkMainContainer, checkpoint target defaulting, the
sidecar split, and DGD-level env defaults all resolve the per-component
name via GetMainContainerName(). In container-scoped kubernetes
discovery mode the operator injects DYN_MAIN_CONTAINER_NAME (only for
non-default names) and stamps the pod annotation
nvidia.com/dynamo-main-container-name (including the Grove path); the
Rust runtime resolves each remote pod's main container from that pod's
own annotation, so mixed-name fleets stay mutually discoverable. The SLA
planner resolves the name from the component spec. The override is
preserved across v1alpha1 conversion via the sparse hub payload.

Review hardening folded in:
- operator-owned identity envs (CONTAINER_NAME, DYN_MAIN_CONTAINER_NAME,
  DYN_KUBE_DISCOVERY_MODE) are the operator's sole authority: a user
  podTemplate value is replaced, and a reserved key the operator did not
  set (e.g. the main-name env on a default-named component) is stripped
- the frontend sidecar gets its own container identity; snapshot restore
  refreshes DYN_MAIN_CONTAINER_NAME; failover engines strip it and
  name collisions are rejected
- immutability is enforced both by a v1beta1 CEL transition rule and by
  the version-agnostic webhook update path, so a v1alpha1 replacement
  that drops the preservation annotation cannot silently rename the main
  container
- imageless sidecars are rejected at admission by the shared webhook
  (resolving the effective main and frontend-sidecar names), not deferred
  to reconcile
- standalone v1beta1 DCDs run the shared-spec validation via the
  dual-version webhook
- conversion keys the semantic main on the effective name; also fixes a
  pre-existing volume-mount-order round-trip conversion bug

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Fang Han <fangh@nvidia.com>
@HanFa HanFa force-pushed the fangh/dgd-main-container-name branch from 066d5be to 7d43d0f Compare July 14, 2026 00:20
@HanFa HanFa temporarily deployed to external_collaborator July 14, 2026 00:20 — with GitHub Actions Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deployment::k8s Relates to dynamo deployment in kubernetes documentation Improvements or additions to documentation feat planner size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants