Add Sentry Logs observability and memory tracker middleware#215
Merged
Conversation
- Swap capture_message to Sentry.logger.info on all legacy traffic endpoints so hits land in Sentry Logs instead of burning event quota and firing Slack alerts - Add extra debug fields per endpoint (format, delta_mb, check_name, success, gemmy_count) to answer who/what is hitting each path - Add MemoryTracker rack middleware: logs to Sentry.logger.warn when a single request grows RSS by 5+ MB, with full request context - Tune sentry.rb: drop enabled_patches logger (was routing all Rails logs through Sentry breadcrumb buffers), cap max_breadcrumbs at 20, disable include_local_variables to avoid holding large AR objects - Bump sentry-rails and sentry-sidekiq from ~> 5.5 to ~> 6.5 to match sentry-ruby ~> 6.5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Web dyno is hitting R14 continuously (average 114% quota, peak 209%). We are investigating the memory leak. This PR adds observability tooling to identify the source and reduces Sentry noise/memory pressure from the current
capture_messageapproach.Changes
capture_message->Sentry.logger.infoon all legacy traffic endpointsAll seven
Sentry.capture_messagecalls replaced withSentry.logger.info. Hits now land in Sentry Logs instead of firing Slack alerts and burning event quota. Extra debug fields added per endpoint to help answer who is hitting what:POST /gemsmethod,path,formatGET /gems/:idformat(crawler vs API consumer)GET /gems/compat_tablegemmy_count(spot bulk callers)GET /gems/:gemmy_id/compatibility/:idgemmy_id->gem_namePOST /api/github_notificationscheck_name,status(full check_run state)POST /api/resultssuccessfield (was checker reporting success or failure?)POST /api/releasesname->gem_namefor consistencyMemoryTrackerrack middleware (new)app/middleware/memory_tracker.rbreads/proc/$PID/statusVmRSS before and after each request. When a single request grows RSS by 5+ MB it logs toSentry.logger.warnwith:This will pinpoint exactly which endpoint and which request parameters are growing memory. The
/procread is Linux-only (Heroku dynos); therescuefallback returns0.0so it is silent locally on macOS.config/initializers/sentry.rbtuningThree changes to reduce Sentry's own memory footprint:
enabled_patches: [:logger]- was routing everyRails.loggercall through Sentry's breadcrumb buffers, holding string references for the lifetime of each requestmax_breadcrumbs: 20- caps per-request breadcrumb buffer (default was 100)include_local_variables: false- was capturing all local variable values on every exception, which can hold large ActiveRecord objects in Sentry's internal buffersSentry gem version alignment
sentry-railsandsentry-sidekiqbumped from~> 5.5to~> 6.5to matchsentry-ruby ~> 6.5. Having sentry-ruby 6.x with sentry-rails 5.x was a version mismatch.