[Spike] Migration from Current Architecture

[mikenairn]

Goal

Build a working migration POC that demonstrates the end-to-end migration path from current operator architecture (with OLMv0 dependencies) to consolidated kuadrant-operator, ensuring zero downtime and safe resource ownership transfer.

Critical: This is not a theoretical migration plan. We need a working migration process (script or controller) that can be executed and demonstrated to validate the approach works in practice without downtime.

Questions to Answer

• CRD Ownership Transfer: What happens when authorino-operator CSV is deleted? How long for OLM label removal? Can new CSV install immediately?
• CRD Bundling: Must we bundle Authorino/Limitador CRDs in v1.0? Can we remove them in v2.0 safely? What if standalone authorino-operator is already installed?
• Workload Ownership: Does stripping ownerReferences cause pod restarts? Can we adopt existing Deployments without recreation?
• Migration Automation: Manual script vs automated controller? What RBAC needed? User experience?
• Safety: How do we verify it's safe to delete old CRs? How do we rollback if migration fails mid-way?

Notes

This task depends on #183 establishing:

• The consolidated operator architecture
• Component packaging structure
• RBAC model
• How kuadrant-operator deploys workloads

Once #183 is complete, this spike will validate the migration path from current (OLMv0 with dependencies) to consolidated architecture.

Migration happens on OLMv0 - users must migrate before upgrading cluster to OLMv1.

Deliverables

Working POC:

• Migration script or controller code (whichever chosen)
• Demonstrated migration flow:
  • Existing installation with authorino-operator + limitador-operator
  • Execute migration
  • Verify zero pod restarts
  • Verify policies continue working
  • Rollback test (if migration fails)

Answers:

• Migration timing matrix (CRD label removal timing, safe installation window)
• CRD bundling decision (which CRDs in v1.0, deprecation timeline for v2.0)
• Workload ownership transfer procedure (tested, zero downtime)
• Chosen migration approach (manual script vs automated) with rationale
• User-facing migration guide (step-by-step runbook)
• Troubleshooting guide (common errors, rollback procedures)

[mikenairn]

Partially Answered by POC

The consolidation POC (#183) significantly reduced the migration complexity by preserving wrapper CRs and all existing Deployments. However, migration was not tested end-to-end in the POC.

What We Know

The scope of migration is much smaller than originally anticipated. Because wrapper CRs (Authorino CR, Limitador CR) are preserved and no CRDs or Deployments are deprecated, the migration is primarily about:

1. Transferring CRD ownership annotations from child operator CSVs to the kuadrant-operator CSV
2. Cleaning up orphaned OLM resources (child operator Subscriptions, CSVs)
3. The kuadrant-operator taking over deployment of child operator controllers (which were previously OLM-managed)

Data plane workloads (Authorino, Limitador Deployments) are unaffected because their ownerReferences point to wrapper CRs, which are preserved.

Resource baseline captured. We captured complete resource baselines from both the current (main) and consolidated (Phase 1) deployments, including all names, labels, selectors, ownerReferences, and service account mappings. These are the compatibility reference for migration testing:

• Resource Baseline (main)
• Resource Baseline (Phase 1)
• Resource Comparison

Answers to Questions

CRD Ownership Transfer: CRDs persist when CSVs are deleted (OLM never deletes CRDs). Annotation/label transfer timing not tested. All component CRDs are bundled in the kuadrant-operator bundle.

CRD Bundling: All component CRDs (Authorino, AuthConfig, Limitador, DNSRecord, DNSHealthCheckProbe, MCPGatewayExtension, MCPServerRegistration, MCPVirtualServer) are bundled in v1. Removal is deferred to a follow-on RFC.

Workload Ownership: Not tested, but significantly simplified. Wrapper CRs are preserved, so workload Deployments (Authorino, Limitador) keep their existing ownerReferences. No ownerReference stripping needed. Child operator Deployments (authorino-operator, limitador-operator, dns-operator) gain ownerReferences to the Kuadrant CR in the consolidated model.

Migration Automation: Not decided. The simplicity of the migration (no CRD removal, no wrapper CR removal, control plane resources can be safely deleted and recreated) may mean a documented manual process is sufficient.

Safety - Finalizer Ordering: Critical finding: the Kuadrant CR must have a finalizer that manages deletion order. Authorino CR has a finalizer (authorino.kuadrant.io/finalizer) and DNSRecords have finalizers for DNS provider cleanup. If the Kuadrant CR is deleted and child operator controllers are cascade-deleted simultaneously, these finalizers will be orphaned. Documented as a required implementation item.

Standalone Authorino Conflict: Listed as an unresolved question in the RFC. Under OLMv1's single-ownership model, two bundles cannot own the same CRDs.

What Still Needs Testing

• End-to-end migration from OLMv0 multi-operator install to consolidated operator
• CRD ownership annotation transfer timing
• SSA field manager conflicts when adopting existing Deployments
• OLMv0 behaviour when a CSV drops olm.package.required declarations (do orphaned child operator Subscriptions cause issues?)
• Rollback procedure if migration fails mid-way

Known Migration Issues (from Resource Comparison)

• limitador-operator selector collision: pre-existing bug where the Deployment selector matches kuadrant-operator pods. Needs upstream fix.
• Finalizer ordering: Kuadrant CR finalizer required before production use.
• Label changes: app.kubernetes.io/managed-by changes from kustomize to helm on some resources. SSA can handle this.

This spike should remain open for the actual migration testing work.

[mikenairn]

Migration Spike Findings

Tested on OpenShift CRC with OLMv0. Full details documented in RFC 0019.

Q1: CRD Ownership Transfer

No ownership transfer is needed. The migration is a clean removal of child operators followed by installation of the consolidated bundle which declares all CRDs as owned.

• Deleting child operator CSVs does not cascade-delete their CRDs. OLM uses labels/annotations for CRD tracking, not Kubernetes ownerReferences. CRDs are cluster-scoped and CSVs are namespace-scoped, so cross-scope GC does not apply.
• The consolidated bundle includes all child CRDs as owned. OLM installs/updates them as part of the bundle.

Q2: CRD Bundling

Child CRDs must be bundled in the consolidated operator and declared as owned in the CSV. Alternatives were tested and rejected:

• CRDs in bundle but not in owned list: opm render bakes olm.gvk properties from any CRDs in the bundle image regardless of what the CSV declares. The resolver still sees GVK conflicts.
• CRDs not in bundle at all: requires the operator to apply CRDs at runtime, needing broad apiextensions.k8s.io create permissions and leaving a window where CRDs are unmanaged by OLM.
• New API versions (v1alpha2): CRDs must still serve existing versions for backwards compatibility, so the GVK conflict remains.

Conflicts with standalone installations (e.g. standalone authorino-operator on the same cluster) follow the same pattern and require the standalone operator to be removed first.

Q3: Workload Ownership

Not relevant to this migration. Wrapper CRs (Authorino CR, Limitador CR) are preserved. The child operator controllers are removed when their CSVs are deleted, and the consolidated operator redeploys them via Helm rendering. Any pod restarts are expected behaviour from version changes, not ownership transfer.

Q4: Migration Automation

Documented manual steps (not an automated controller). The migration is a one-time procedure:

1. (Recommended) Set installPlanApproval: Manual on the kuadrant subscription
2. Delete child operator subscriptions
3. Delete child operator CSVs
4. Approve the install plan (if using manual approval)

No extra RBAC is needed beyond what the consolidated operator already requires. A migration script or runbook should be provided.

Automated migration (runtime controller) was rejected because it requires temporary apiextensions.k8s.io permissions and creates a transition release with unmanaged CRDs. The manual approach gives kuadrant-operator clean OLM ownership of all CRDs from day one.

Q5: Safety

OLM provides built-in safety:

• The upgrade stalls safely with ResolutionFailed when GVK conflicts exist. The existing deployment continues running with no data loss.
• Once child operators are removed, the resolver succeeds within seconds.
• CRDs and their CRs survive the entire migration.

Important caveat to document: when deleting CSVs/subscriptions via the OpenShift UI, there may be an option to delete operands (custom resources). Users must not select this option. The migration steps must only delete the subscription and CSV objects, not the CRs managed by those operators. This should be prominently called out in the migration guide.

Remaining work

• User-facing migration guide (standalone runbook, not just RFC)
• Verify data plane continuity (policies enforced, no traffic loss during the control plane gap)
• Document the control plane gap: between child CSV deletion (controllers stop) and consolidated operator starting (controllers redeployed), changes to wrapper CRs are not reconciled. Existing workloads continue running.
• Rollback procedure documentation