Azure · Ghazi Alchammat (galchammat) · Jul 20, 2026 · Jul 20, 2026
diff --git a/docs/sops/ci-bot-secret-provisioning.md b/docs/sops/ci-bot-secret-provisioning.md
@@ -0,0 +1,55 @@
+# CI Bot Secret Provisioning
+
+This SOP describes how to manually provision or re-provision CI bot Entra application credentials in Azure Key Vault.
+
+## Background
+
+The [`ci-bot-ensure-secret.sh`](../../dev-infrastructure/scripts/ci-bot-ensure-secret.sh) script creates a client secret on an existing CI bot Entra application and stores it (along with the client ID and tenant ID) in Azure Key Vault. The pipeline runs this automatically for INT, STG, and PROD bots as part of the `Microsoft.Azure.ARO.HCP.DevCI.E2ESubscriptionRBAC` service group.
+
+Manual runs are needed when:
+
+- A Key Vault secret was accidentally deleted
+- A new environment's bot was created but the pipeline hasn't run yet
+- Troubleshooting credential issues in a specific environment
+
+The script is idempotent: it skips secrets that already exist in Key Vault.
+
+## Prerequisites
+
+- `az` CLI logged into the DEV tenant (`64dc69e4-d083-49fc-9569-ebece1dd1408`)
+- Key Vault write access on the target vault (e.g. `opstool-kv-usw3`)
+- The Entra application must already exist (created by `ci-bot-identity.bicep`)
+
+## Procedure
+
+### 1. Run the script
+
+```bash
+cd dev-infrastructure/scripts
+
+APP_NAME="OpenShift Release Bot - INT" \
+ENV_NAME="int" \
+KEY_VAULT_NAME="opstool-kv-usw3" \
+./ci-bot-ensure-secret.sh
+```
+
+Replace `APP_NAME` and `ENV_NAME` with the target environment. Valid combinations are defined in [`config/config-dev-ci.yaml`](../../config/config-dev-ci.yaml) under `ci.<env>.bot.applicationName`.
+
+### 2. Verify
+
+```bash
+az keyvault secret show --vault-name opstool-kv-usw3 --name ci-bot-int-client-secret --query '{name:name,created:attributes.created}' -o table
+```
+
+### 3. Admin consent (if warned)
+
+The script attempts `az ad app permission admin-consent` as a best-effort step. If it warns about insufficient privileges, ask a tenant admin to grant consent manually via the Azure Portal (Entra ID > App registrations > the bot app > API permissions > Grant admin consent).
+
+## Key Locations
+
+| What | Where |
+|------|-------|
+| Script | [`dev-infrastructure/scripts/ci-bot-ensure-secret.sh`](../../dev-infrastructure/scripts/ci-bot-ensure-secret.sh) |
+| Pipeline step | `ci-bot-secret-{env}` in [`dev-infrastructure/dev-ci/e2e-subscription-rbac/pipeline.yaml`](../../dev-infrastructure/dev-ci/e2e-subscription-rbac/pipeline.yaml) |
+| Key Vault secrets | `ci-bot-{env}-client-secret`, `ci-bot-{env}-client-id`, `ci-bot-{env}-tenant-id` |
+| Bot app names | [`config/config-dev-ci.yaml`](../../config/config-dev-ci.yaml) under `ci.<env>.bot.applicationName` |
diff --git a/docs/sops/release-bot-msft-test-tenant.md b/docs/sops/release-bot-msft-test-tenant.md
@@ -0,0 +1,79 @@
+# Release Bot MSFT Test Tenant
+
+This SOP covers creating and rotating credentials for the "OpenShift Release Bot MSFT Test" Entra application in the Microsoft E2E tenant.
+
+## Background
+
+The OpenShift Release Bot MSFT Test is a service principal in the `Test Test Azure Red Hat OpenShift` tenant (`93b21e64-4824-439a-b893-46c9b2a51082`). It runs E2E tests against Microsoft-hosted subscriptions. Its credentials are stored in HashiCorp Vault (`vault.ci.openshift.org`) and consumed by OpenShift CI Prow jobs.
+
+Two scripts manage it:
+
+- [`create-openshift-release-bot-msft-test.sh`](../../dev-infrastructure/openshift-ci/create-openshift-release-bot-msft-test.sh) -- one-time setup (creates SP, assigns roles, grants Graph permissions, generates initial credentials)
+- [`recycle-openshift-release-bot-creds.sh`](../../dev-infrastructure/openshift-ci/recycle-openshift-release-bot-creds.sh) -- rotates the client secret and updates Vault
+
+These cannot be migrated to Bicep because they operate cross-tenant and store credentials in external HashiCorp Vault.
+
+## Prerequisites
+
+- `az` CLI logged into the test tenant: `az login --tenant 93b21e64-4824-439a-b893-46c9b2a51082`
+- `vault` CLI installed; will prompt for OIDC login to `https://vault.ci.openshift.org`
+- `jq` installed
+- Permission to manage the application's credentials in the test tenant
+
+## Procedure: Initial Setup
+
+Only needed once per tenant or after a full teardown.
+
+```bash
+cd dev-infrastructure/openshift-ci
+./create-openshift-release-bot-msft-test.sh
+```
+
+This creates the SP (if missing), assigns Contributor and RBAC Administrator on both E2E subscriptions, grants Graph permissions, triggers admin consent, and calls the recycle script to generate credentials.
+
+## Procedure: Credential Rotation
+
+### 1. Rotate credentials
+
+```bash
+cd dev-infrastructure/openshift-ci
+./recycle-openshift-release-bot-creds.sh
+```
+
+By default this rotates both STG and PROD credentials. To rotate only one:
+
+```bash
+./recycle-openshift-release-bot-creds.sh --env stg
+```
+
+To also delete old credentials (instead of appending):
+
+```bash
+./recycle-openshift-release-bot-creds.sh --delete-old
+```
+
+### 2. Activate the new credentials
+
+After rotation, switch the active tenant configuration:
+
+```bash
+./switch-vault-tenant.sh --to test-tenant
+```
+
+### 3. Verify
+
+Confirm the new credentials work by checking a recent Prow job that uses the test tenant. Vault secrets are at:
+
+- `kv/selfservice/hcm-aro/aro-hcp-stg-test-tenant`
+- `kv/selfservice/hcm-aro/aro-hcp-prod-test-tenant`
+
+## Key Locations
+
+| What | Where |
+|------|-------|
+| Setup script | [`dev-infrastructure/openshift-ci/create-openshift-release-bot-msft-test.sh`](../../dev-infrastructure/openshift-ci/create-openshift-release-bot-msft-test.sh) |
+| Rotation script | [`dev-infrastructure/openshift-ci/recycle-openshift-release-bot-creds.sh`](../../dev-infrastructure/openshift-ci/recycle-openshift-release-bot-creds.sh) |
+| Tenant switch | [`dev-infrastructure/openshift-ci/switch-vault-tenant.sh`](../../dev-infrastructure/openshift-ci/switch-vault-tenant.sh) |
+| Vault secrets | `kv/selfservice/hcm-aro/aro-hcp-{stg,prod}-test-tenant` on `vault.ci.openshift.org` |
+| Test tenant ID | `93b21e64-4824-439a-b893-46c9b2a51082` |
+| Tenant access SOP | [`docs/sops/test-test-tenant-access.md`](test-test-tenant-access.md) |
diff --git a/docs/sops/tenant-quota-service-principals.md b/docs/sops/tenant-quota-service-principals.md
@@ -0,0 +1,94 @@
+# Tenant Quota Service Principals
+
+This SOP covers creating and rotating credentials for the tenant-quota collector service principals.
+
+## Background
+
+The tenant-quota collector monitors Azure directory and subscription quota across multiple tenants. Each tenant has its own Entra service principal with Reader access on the monitored subscriptions and (for the RedHat0 tenant) Organization.Read.All Graph permission for directory quota.
+
+Two scripts manage these SPs:
+
+- [`manage-service-principals.sh`](../../tooling/tenant-quota/scripts/manage-service-principals.sh) -- creates the SP, assigns roles, grants Graph permissions, stores the secret in Key Vault
+- [`renew-sp-secret.sh`](../../tooling/tenant-quota/scripts/renew-sp-secret.sh) -- rotates the client secret and updates Key Vault
+
+These cannot be migrated to Bicep because they operate cross-tenant and require tenant-specific admin consent.
+
+### Configured tenants
+
+| Tenant | Tenant ID | KV Secret Name |
+|--------|-----------|----------------|
+| RedHat0 | `64dc69e4-d083-49fc-9569-ebece1dd1408` | `custom-metrics-collector-redhat0-client-secret` |
+| TestTestARO | `93b21e64-4824-439a-b893-46c9b2a51082` | `custom-metrics-collector-test-test-azure-arohcp-client-secret` |
+
+## Prerequisites
+
+- `az` CLI logged into the **target tenant** (not the dev tenant): `az login --tenant <tenant-id>`
+- Key Vault access on `opstool-kv-usw3` (in the dev tenant) for storing secrets
+- Tenant admin role for admin consent (RedHat0 tenant only)
+
+## Procedure: Initial Setup
+
+For a new tenant (e.g. RedHat0):
+
+```bash
+cd tooling/tenant-quota/scripts
+./manage-service-principals.sh --tenant redhat
+```
+
+If the Key Vault is in a different tenant than the SP, use the split workflow:
+
+```bash
+# Step 1: Create SP in target tenant (saves secret to a local file)
+az login --tenant <target-tenant-id>
+./manage-service-principals.sh --tenant redhat --skip-keyvault
+
+# Step 2: Upload secret to Key Vault in dev tenant
+az login --tenant 64dc69e4-d083-49fc-9569-ebece1dd1408
+./manage-service-principals.sh --tenant redhat --keyvault-only --secret-name <kv-secret-name> --secret-file /tmp/sp-secret.txt
+```
+
+After setup, update both:
+- [`config/config-dev-ci.yaml`](../../config/config-dev-ci.yaml) under `opstool.tenantQuota.tenants` with the new SP's client ID and KV secret name
+- The `TENANTS` array in [`renew-sp-secret.sh`](../../tooling/tenant-quota/scripts/renew-sp-secret.sh) with the new tenant entry, so future rotations work
+
+## Procedure: Secret Rotation
+
+### 1. List current credential expiration
+
+```bash
+cd tooling/tenant-quota/scripts
+./renew-sp-secret.sh --list
+```
+
+### 2. Rotate
+
+```bash
+az login --tenant <tenant-id>
+./renew-sp-secret.sh --tenant RedHat0
+```
+
+To also restart the collector pod immediately:
+
+```bash
+./renew-sp-secret.sh --tenant RedHat0 --restart
+```
+
+Without `--restart`, the CSI driver picks up new KV secrets within ~2 minutes.
+
+### 3. Verify
+
+Check the collector logs for successful quota collection after rotation:
+
+```bash
+kubectl -n tenant-quota logs deploy/tenant-quota-collector --tail=20
+```
+
+## Key Locations
+
+| What | Where |
+|------|-------|
+| Setup script | [`tooling/tenant-quota/scripts/manage-service-principals.sh`](../../tooling/tenant-quota/scripts/manage-service-principals.sh) |
+| Rotation script | [`tooling/tenant-quota/scripts/renew-sp-secret.sh`](../../tooling/tenant-quota/scripts/renew-sp-secret.sh) |
+| Key Vault | `opstool-kv-usw3` |
+| Collector config | [`config/config-dev-ci.yaml`](../../config/config-dev-ci.yaml) under `opstool.tenantQuota.tenants` |
+| Collector deployment | `tenant-quota` namespace on the opstool AKS cluster |