feat: add controller to identify and clean up cosmos records without resourceID#6194
Merged
Merged
Conversation
…resourceID Old records in Cosmos may lack a resourceID field. This adds a MissingResourceID controller that runs every hour, lists all such records via a new ListMissingResourceID method on ResourcesDBClient, dumps each to the log, and deletes items whose resourceType contains "controller" or "operation". Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds backend support for identifying and cleaning up legacy Cosmos DB records in the Resources container that are missing the resourceID field, by introducing a new periodic controller and the underlying database listing API/iterator needed to query those documents.
Changes:
- Added
ResourcesDBClient.ListMissingResourceID(...)plus a Cosmos query implementation to listTypedDocumentitems withoutresourceID. - Introduced a
queryTypedDocumentIteratorto support iterating rawTypedDocumentquery results (including single-page behavior forListAllpagination). - Added a new backend mismatch controller (
MissingResourceID) and wired it into backend controller startup.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| internal/databasetesting/mock_resources_db_client.go | Extends the in-memory mock to support listing documents missing resourceID. |
| internal/database/iterators.go | Adds a Cosmos query iterator for iterating TypedDocument results. |
| internal/database/database.go | Adds the ListMissingResourceID API and its Cosmos implementation/query. |
| backend/pkg/controllers/mismatchcontrollers/missing_resource_id.go | New periodic controller that logs and deletes selected missing-resourceID documents. |
| backend/pkg/app/backend.go | Wires the new controller into the backend controller run loop. |
Comment on lines
+181
to
+182
| controllerutils.ReconcileTotal.WithLabelValues(c.name).Inc() | ||
| err := c.SyncOnce(ctx, ref) |
Comment on lines
+133
to
+141
| logger := utils.LoggerFromContext(ctx) | ||
|
|
||
| syncErr := c.sweep(ctx) | ||
| if syncErr != nil { | ||
| logger.Error(syncErr, "sweep for documents missing resourceID had errors") | ||
| } | ||
|
|
||
| return nil // never requeue | ||
| } |
Comment on lines
+81
to
+86
| allDocs, err := database.ListAll[database.TypedDocument](ctx, 100, c.resourcesDBClient.ListMissingResourceID) | ||
| if err != nil { | ||
| return utils.TrackError(fmt.Errorf("listing documents missing resourceID: %w", err)) | ||
| } | ||
|
|
||
| logger.Info("found documents missing resourceID", "count", len(allDocs)) |
| logger.Info("found documents missing resourceID", "count", len(allDocs)) | ||
|
|
||
| errs := []error{} | ||
| for _, doc := range allDocs { |
Comment on lines
+112
to
+113
|
|
||
| return errors.Join(errs...) |
Comment on lines
+104
to
+110
| if shouldDelete(doc) { | ||
| docLogger.Info("deleting document missing resourceID") | ||
| if err := deleteByItemID(ctx, c.resourcesDBClient, doc.PartitionKey, doc.ID); err != nil { | ||
| docLogger.Error(err, "unable to delete document missing resourceID") | ||
| errs = append(errs, utils.TrackError(fmt.Errorf("unable to delete %v: %w", doc.ID, err))) | ||
| } | ||
| } |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: deads2k, stevekuznetsov The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Old records in Cosmos may lack a resourceID field. This adds a MissingResourceID controller that runs every hour, lists all such records via a new ListMissingResourceID method on ResourcesDBClient, dumps each to the log, and deletes items whose resourceType contains "controller" or "operation".