fix(script): parallelize manifest size backfill

[riderx]

Summary (AI generated)

• Parallelizes the manifest file size backfill by splitting the manifest id range across workers.
• Replaces one-row DB updates with per-page bulk updates keyed by manifest primary key.
• Adds bounded storage concurrency, DB/storage retries, resumable id bounds, and progress report writes.

Motivation (AI generated)

The existing backfill walked one ordered cursor and performed one DB update per file, which is too slow for millions of manifest entries and wastes work when the connection drops mid-run.

Business Impact (AI generated)

This makes the production repair for missing manifest file sizes finish much faster while keeping database load bounded to indexed manifest id range scans and primary-key updates. It reduces the time users see metadata not found in bundle manifests.

Test Plan (AI generated)

• bun scripts/backfill_manifest_file_sizes.mjs --help
• node --check scripts/backfill_manifest_file_sizes.mjs
• bunx eslint --no-ignore scripts/backfill_manifest_file_sizes.mjs
• git diff --check origin/main...HEAD
• bun lint

Summary by CodeRabbit

• New Features
  • Added CLI options for ID range bounds, concurrency tuning (workers, batch size, concurrency), and retry configuration (storage and database attempts).
  • Improved retry mechanisms for storage operations with bounded backoff strategy.
  • Enhanced reporting with worker configuration and timing metrics.
  • Optimized batch processing with bulk updates for improved performance.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 327547d3-5b8f-4221-ad78-dfdae2f9c04c

📥 Commits

Reviewing files that changed from the base of the PR and between 56f1484 and b8eb201.

📒 Files selected for processing (1)

• scripts/backfill_manifest_file_sizes.mjs

---

📝 Walkthrough

Walkthrough

The script is refactored to support configurable concurrency and retry logic for backfilling manifest file sizes from object storage. New CLI arguments control worker count, batch sizing, retry attempts, and ID bounds. Storage and database operations now include retry-with-backoff handling, queries are unified with centralized filtering, and updates are performed in bulk batches across parallel worker ranges.

Changes

Manifest backfill concurrency and retry architecture

Layer / File(s)	Summary
CLI arguments and documentation scripts/backfill_manifest_file_sizes.mjs	Usage examples and help text are updated to document new --workers, --concurrency, --batch-size, --storage-attempts, --db-attempts, --start-id, --end-id, and --report flags.
Retry utilities: sleep and storage backoff scripts/backfill_manifest_file_sizes.mjs	sleep(ms) helper and getObjectSizeWithRetry functions implement bounded retry logic with incremental backoff delays for storage object-size lookups, skipping retries when status is 404 or size already cached.
Database retry and query refactoring scripts/backfill_manifest_file_sizes.mjs	Database retry detection (isRetryableDatabaseError), queryWithRetry wrapper, and centralized appendCommonFilters unify WHERE-clause construction for missing/invalid file_size, non-null s3_path, and optional manifest ID bounds; buildBoundsQuery and buildCandidateQuery are refactored to use this shared logic.
Bulk updates and concurrent range processing scripts/backfill_manifest_file_sizes.mjs	buildBulkUpdateQuery generates single VALUES-based SQL update statements; createIdRanges partitions the manifest ID range; processCandidates and runRangeWorker implement per-worker concurrent processing with progress tracking and per-worker reporting.
Configuration parsing and execution orchestration scripts/backfill_manifest_file_sizes.mjs	New argument parsing computes workers, batchSize, concurrency, retry limits, and scope bounds; report object tracks worker configuration, timing, and derived settings. Main execution flow queries bounds, partitions ranges, runs workers concurrently via Promise.allSettled, records endedAt timestamp, and ensures pool shutdown.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• Cap-go/capgo#2292: Both PRs substantially change scripts/backfill_manifest_file_sizes.mjs to implement/extend manifest file-size backfill with S3/R2 lookup retry logic, new scoping/concurrency controls, and structured reporting/update behavior.
• Cap-go/capgo#2293: Both PRs modify scripts/backfill_manifest_file_sizes.mjs's manifest backfill logic (adding/altering CLI/DB target handling and execution/reporting behavior), so the main PR's script rework is directly related.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: parallelizing the manifest size backfill script, which is the primary objective of this PR.
Description check	✅ Passed	The description includes a comprehensive summary, motivation, business impact, and detailed test plan with all items checked off, covering the required template sections adequately.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

Warning

Review ran into problems

🔥 Problems

Stopped waiting for pipeline failures after 30000ms. One of your pipelines takes longer than our 30000ms fetch window to run, so review may not consider pipeline-failure results for inline comments if any failures occurred after the fetch window. Increase the timeout if you want to wait longer or run a @coderabbit review after the pipeline has finished.

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codspeed-hq]

Merging this PR will not alter performance

✅ 43 untouched benchmarks
⏩ 2 skipped benchmarks¹

---

_{Comparing codex/parallel-manifest-backfill (b8eb201) with main (56f1484)}

Footnotes

1. 2 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.