feat: set evidence-based memory requests for backend and mgmt-agent - #6324
feat: set evidence-based memory requests for backend and mgmt-agent#6324weherdh wants to merge 1 commit into
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: weherdh The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Pull request overview
Adjusts Kubernetes memory requests for the backend and mgmt-agent services to better reflect observed production usage, and standardizes mgmt-agent resource configuration by sourcing requests from config.yaml rather than hardcoded chart defaults.
Changes:
- Increase backend memory request default from
500Mito1Gi. - Increase mgmt-agent memory request from
64Mito768Mi. - Wire mgmt-agent request values through
config.yamland regenerate rendered configs / Helm fixtures.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| mgmt-agent/zz_fixture_TestHelmTemplate_dev_westus3_mgmt_1_mgmt_agent.yaml | Updates Helm fixture to reflect new mgmt-agent memory request (768Mi). |
| mgmt-agent/values.yaml | Sources mgmt-agent resource requests from config.yaml instead of hardcoded values. |
| config/rendered/dev/pers/westus3.yaml | Rendered config updated for backend (1Gi) and mgmt-agent resources. |
| config/rendered/dev/perf/westus3.yaml | Rendered config updated for backend (1Gi) and mgmt-agent resources. |
| config/rendered/dev/dev/westus3.yaml | Rendered config updated for backend (1Gi) and mgmt-agent resources. |
| config/rendered/dev/cspr/westus3.yaml | Rendered config updated for backend (1Gi) and mgmt-agent resources. |
| config/rendered/dev/ci01/centralus.yaml | Rendered config updated for backend (1Gi) and mgmt-agent resources. |
| config/rendered/dev/ci00/centralus.yaml | Rendered config updated for backend (1Gi) and mgmt-agent resources. |
| config/config.yaml | Updates defaults: backend request to 1Gi; adds mgmt-agent resources defaults. |
| backend/zz_fixture_TestHelmTemplate_dev_westus3_svc_1_aro_hcp_backend_dev.yaml | Updates backend Helm fixture to reflect new memory request (1Gi). |
| backend/testdata/zz_fixture_TestHelmTemplate_backend_mi_mock_and_arm_perms_mgr_identities_unset.yaml | Updates backend test fixture to reflect new memory request (1Gi). |
| backend/testdata/zz_fixture_TestHelmTemplate_backend_clstr_scoped_identities_role_set_name_public.yaml | Updates backend test fixture to reflect new memory request (1Gi). |
Backend: 500Mi → 1Gi (actual steady-state: 750MB–1GB in uksouth) mgmt-agent: 64Mi → 768Mi (actual steady-state: 256–512MB, peak <768MB) Values determined by querying container_memory_working_set_bytes across 3 prod regions for 7 days using alert-tester. Formula: peak × 1.25 margin. Ref: AROSLSRE-1033, AROSLSRE-1654
4de15ab to
032a085
Compare
https://redhat.atlassian.net/browse/AROSLSRE-1033
https://redhat.atlassian.net/browse/AROSLSRE-1654
What
Set evidence-based memory requests for two services that are currently under-provisioned:
500Mi→1Gi(was running at 1.5–3.25x its request in uksouth)64Mi→768Mi(was running at 8–10x its request — 64Mi was never a real sizing)Also wires mgmt-agent's resource values through
config.yamlinstead of hardcoded defaults invalues.yaml.Why
Both services have memory requests that don't reflect actual production usage:
ServiceMemoryDriftalert (PR feat: add generic memory drift and trend alerts for ARO-HCP services #6011).Evidence gathered by running
alert-testeragainst prod Grafana (container_memory_working_set_bytes) across uksouth, eastus2, and australiaeast for July 21–28.Testing
make materializepasses — rendered configs and helm fixtures regeneratedServiceMemoryDriftalert (PR feat: add generic memory drift and trend alerts for ARO-HCP services #6011) will validate these values once deployed — right-sized requests should produce 0 firingsSpecial notes for your reviewer
values.yamlnow references{{ .mgmtAgent.k8s.resources.requests.memory }}instead of a hardcoded64Mi, consistent with how other services (backend, kube-applier) wire their resource values.PR Checklist