BYOO collector scalability and centralized configuration

[sbaum1994]

Description

Track the work needed to make Bring Your Own Observability (BYOO) collectors scale with telemetry demand and centralize collector configuration in NVCA.

This epic covers:

• Deploying and managing the collector operator required for autoscaling.
• Configurable HPA enablement, replica bounds, scaling policy, and backward-compatible defaults.
• Independent log and trace sampling configuration.
• NVCA ownership of validation, defaults, version selection, compatibility translation, and rendered collector configuration.
• Support for the current and new collector versions during migration, including safe upgrade and rollback.
• Protection of customer secrets through the existing secure injection flow and hot reload when configuration or secret files change.
• Self-managed packaging through the observability stack and managed NVCF integration with release and SBOM gates.
• Failure isolation, actionable validation, operational status, and bounded-cardinality scaling and capacity telemetry.
• Capacity testing to establish supported load, sizing, HPA signals and defaults, and production acceptance targets.

Implementation tasks will be planned and linked separately when execution begins.

Definition of Done

• A managed NVCF test cluster deploys the collector operator and new collector through the supported NVCA flow.
• HPA can be enabled, disabled, and configured within documented bounds, and representative telemetry load produces observable replica changes.
• Log and trace sampling can be configured independently and verified at collector output.
• NVCA produces effective configuration for both supported collector versions while existing configurations continue to work without new required fields.
• A test cluster can upgrade to the new collector and roll back without losing unrelated configuration or interrupting inference workloads. Workloads are redeployed when required to adopt collector changes.
• The self-managed observability stack installs, upgrades, and removes required BYOO components when BYOO is enabled and leaves them disabled otherwise.
• Managed release artifacts include every deployed BYOO operator and runtime image in SBOM generation and standard release gates.
• Invalid HPA, sampling, version, or collector configuration is rejected before a partial rollout with an actionable error.
• Collector, operator, scaling, and export failures are visible to operators and do not block customer inference workloads.
• A representative workload baseline defines supported capacity per replica, scaling behavior, and production acceptance targets.
• Managed and self-managed documentation covers configuration, defaults, supported versions, operations, upgrade and rollback, and known limitations.

---

By submitting this issue, you acknowledge that you are an assigned member of the NVCF development team and agree to follow our code of conduct and our contributing guidelines.