Skip to content

feat: set evidence-based memory requests for backend and mgmt-agent - #6324

Open
weherdh wants to merge 1 commit into
Azure:mainfrom
weherdh:aroslsre-1033-1654-memory-requests
Open

feat: set evidence-based memory requests for backend and mgmt-agent#6324
weherdh wants to merge 1 commit into
Azure:mainfrom
weherdh:aroslsre-1033-1654-memory-requests

Conversation

@weherdh

@weherdh weherdh commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

https://redhat.atlassian.net/browse/AROSLSRE-1033
https://redhat.atlassian.net/browse/AROSLSRE-1654

What

Set evidence-based memory requests for two services that are currently under-provisioned:

  • backend: 500Mi1Gi (was running at 1.5–3.25x its request in uksouth)
  • mgmt-agent: 64Mi768Mi (was running at 8–10x its request — 64Mi was never a real sizing)

Also wires mgmt-agent's resource values through config.yaml instead of hardcoded defaults in values.yaml.

Why

Both services have memory requests that don't reflect actual production usage:

  • backend in uksouth consistently uses 750MB–1GB, but its request was 500Mi. This caused 49 alert firings/week with the new generic ServiceMemoryDrift alert (PR feat: add generic memory drift and trend alerts for ARO-HCP services #6011).
  • mgmt-agent uses 256–512MB steady-state (peak <768MB), but its request was 64Mi — providing no meaningful scheduling signal or eviction protection.

Evidence gathered by running alert-tester against prod Grafana (container_memory_working_set_bytes) across uksouth, eastus2, and australiaeast for July 21–28.

Testing

Special notes for your reviewer

  • The mgmt-agent values.yaml now references {{ .mgmtAgent.k8s.resources.requests.memory }} instead of a hardcoded 64Mi, consistent with how other services (backend, kube-applier) wire their resource values.
  • Priority class changes are tracked separately and will follow.

PR Checklist

  • PR is scoped to a single task (no mixed concerns)
  • Title follows Conventional Commits format
  • Summary explains the "Why" behind the change
  • Linked to relevant ticket/issue
  • Screenshots included (if graph/UI/metrics changes)
  • Self-reviewed the diff
  • CI/CD checks are passing (ignore Tide)
  • Draft PR used for WIP (if applicable)
  • Commit history is clean (rebased/squashed)
  • Tricky code blocks are commented
  • Specific reviewers tagged
  • All comment threads resolved before merge

Copilot AI review requested due to automatic review settings July 30, 2026 04:27
@openshift-ci

openshift-ci Bot commented Jul 30, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: weherdh
Once this PR has been reviewed and has the lgtm label, please assign roivaz for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adjusts Kubernetes memory requests for the backend and mgmt-agent services to better reflect observed production usage, and standardizes mgmt-agent resource configuration by sourcing requests from config.yaml rather than hardcoded chart defaults.

Changes:

  • Increase backend memory request default from 500Mi to 1Gi.
  • Increase mgmt-agent memory request from 64Mi to 768Mi.
  • Wire mgmt-agent request values through config.yaml and regenerate rendered configs / Helm fixtures.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
mgmt-agent/zz_fixture_TestHelmTemplate_dev_westus3_mgmt_1_mgmt_agent.yaml Updates Helm fixture to reflect new mgmt-agent memory request (768Mi).
mgmt-agent/values.yaml Sources mgmt-agent resource requests from config.yaml instead of hardcoded values.
config/rendered/dev/pers/westus3.yaml Rendered config updated for backend (1Gi) and mgmt-agent resources.
config/rendered/dev/perf/westus3.yaml Rendered config updated for backend (1Gi) and mgmt-agent resources.
config/rendered/dev/dev/westus3.yaml Rendered config updated for backend (1Gi) and mgmt-agent resources.
config/rendered/dev/cspr/westus3.yaml Rendered config updated for backend (1Gi) and mgmt-agent resources.
config/rendered/dev/ci01/centralus.yaml Rendered config updated for backend (1Gi) and mgmt-agent resources.
config/rendered/dev/ci00/centralus.yaml Rendered config updated for backend (1Gi) and mgmt-agent resources.
config/config.yaml Updates defaults: backend request to 1Gi; adds mgmt-agent resources defaults.
backend/zz_fixture_TestHelmTemplate_dev_westus3_svc_1_aro_hcp_backend_dev.yaml Updates backend Helm fixture to reflect new memory request (1Gi).
backend/testdata/zz_fixture_TestHelmTemplate_backend_mi_mock_and_arm_perms_mgr_identities_unset.yaml Updates backend test fixture to reflect new memory request (1Gi).
backend/testdata/zz_fixture_TestHelmTemplate_backend_clstr_scoped_identities_role_set_name_public.yaml Updates backend test fixture to reflect new memory request (1Gi).

Comment thread mgmt-agent/values.yaml
Backend: 500Mi → 1Gi (actual steady-state: 750MB–1GB in uksouth)
mgmt-agent: 64Mi → 768Mi (actual steady-state: 256–512MB, peak <768MB)

Values determined by querying container_memory_working_set_bytes across
3 prod regions for 7 days using alert-tester. Formula: peak × 1.25 margin.

Ref: AROSLSRE-1033, AROSLSRE-1654
@weherdh
weherdh force-pushed the aroslsre-1033-1654-memory-requests branch from 4de15ab to 032a085 Compare July 30, 2026 04:59
Copilot AI review requested due to automatic review settings July 30, 2026 04:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants