Defer run_check! to Sidekiq to fix web dyno memory pressure#216
Merged
Defer run_check! to Sidekiq to fix web dyno memory pressure#216
Conversation
POST /lockfiles was holding the web thread for 366+ sequential DB+Redis round-trips on a 182-gem lockfile (1 LockfileCheck INSERT + N GemCheck INSERTs + N perform_async calls). Under concurrent load, multiple Puma workers each retained hundreds of AR objects, causing RSS to grow and never release back to the OS. Changes: - Add Lockfiles::StartCheck Sidekiq job: does the LockfileCheck + GemCheck creation and enqueue off the web thread - Lockfile#run_check! now calls perform_async instead of doing the work inline; controllers return in milliseconds regardless of lockfile size - LockfileCheck.create_for! replaces the N-INSERT gem loop with a single insert_all (one batched INSERT for all GemCheck rows) - LockfileCheck#enqueue_gem_checks replaces N perform_async calls with a single perform_bulk (one Redis push for all pending checks) - Update specs throughout to match new async contract
- Rename 'no-op on deleted lockfile' to 'raises RecordNotFound' since raising is not a no-op; behavior is intentional (Sidekiq retries cover it) - Rewrite 'uses next_rails_release' test to actually assert a LockfileCheck is created targeting the correct release, not just that perform_bulk runs - Add comment on insert_all omitting returning: and the two-round-trip tradeoff
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #209
Problem
POST /lockfileswas holding the Puma web thread for the entire duration ofrun_check!. For a 182-gem lockfile that was 366 sequential DB + Redis round-trips before the response returned:Each Puma worker retained all those ActiveRecord objects in memory for the lifetime of the request. Ruby's heap does not release pages back to the OS after GC, so repeated large requests cause RSS to ratchet up permanently. This is consistent with the R14 pattern observed in production (average 114% quota, peak 209%).
Changes
Lockfiles::StartCheck(new Sidekiq job)Moves all post-save work off the web thread. The controller now does exactly:
The job does the LockfileCheck creation and gem check enqueue asynchronously on a worker dyno.
LockfileCheck.create_for!—insert_allReplaces the N-INSERT loop with a single batched
insert_all. All GemCheck rows are built in memory and written in one SQL statement.Before:
NsequentialINSERT INTO gem_checks(one per gem)After:
1INSERT INTO gem_checks ... VALUES (...), (...), (...)for all gemsLockfileCheck#enqueue_gem_checks—perform_bulkReplaces N individual
perform_asynccalls with oneperform_bulkcall.Before:
NsequentialLPUSHto Redis (one per gem_check)After:
1bulk Redis push for all pending gem_checksLockfile#run_check!Now a thin delegate to
Lockfiles::StartCheck.perform_async. Accepts the samerails_release:keyword for explicit override.Complexity reduction
For a 182-gem lockfile:
Tests
lockfile_spec.rb—#run_check!now assertsStartCheckis enqueued with correct idslockfile_check_spec.rb—#enqueue_gem_checksnow assertsperform_bulkis calledapi/lockfiles_controller_spec.rb— assertsStartCheckis enqueued on POSTstart_check_spec.rb— new, covers the happy path, no-next-release no-op, and deleted-lockfile error