From 94d0bae679761e46379eb555f2306b7e255c99bb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=85=A7=E5=BE=AE?= Date: Fri, 10 Jul 2026 11:26:27 +0800 Subject: [PATCH 1/2] doc: add doc best-practice/configuring-autoscaling --- .../en/08-configuring-autoscaling-guide.md | 875 ++++++++++++++++++ .../en/08-configuring-autoscaling.md | 500 ++++++++++ .../zh/08-configuring-autoscaling-guide.md | 875 ++++++++++++++++++ .../zh/08-configuring-autoscaling.md | 498 ++++++++++ 4 files changed, 2748 insertions(+) create mode 100644 doc/best-practice/en/08-configuring-autoscaling-guide.md create mode 100644 doc/best-practice/en/08-configuring-autoscaling.md create mode 100644 doc/best-practice/zh/08-configuring-autoscaling-guide.md create mode 100644 doc/best-practice/zh/08-configuring-autoscaling.md diff --git a/doc/best-practice/en/08-configuring-autoscaling-guide.md b/doc/best-practice/en/08-configuring-autoscaling-guide.md new file mode 100644 index 00000000..e572e1ca --- /dev/null +++ b/doc/best-practice/en/08-configuring-autoscaling-guide.md @@ -0,0 +1,875 @@ +# Operations Guide: Configuring Autoscaling Strategies for RBG Services + +> Corresponding concept document: [8. Configuring Autoscaling Strategies for RBG Services](08-configuring-autoscaling.md) + +## Objectives + +Validate RBG's autoscaling mechanism, including: +1. Enabling ScalingAdapter and manual scaling via Scale subresource +2. Configuring HPA for metric-driven scaling +3. Configuring KEDA for event-driven scaling +4. Configuring RBG Planner for SLA-driven scaling (PD-disaggregated scenario) + +## Prerequisites + +- Kubernetes cluster version >= 1.24 +- RBG Controller installed +- Image accessible: `lmsysorg/sglang:v0.5.9` +- Operation 2 requires Metrics Server installed +- Operation 3 requires KEDA and Prometheus installed +- Operation 4 requires RBG Planner installed (`helm install rbg-planner oci://ghcr.io/sgl-project/charts/rbg-planner -n rbg-system --create-namespace`) + +> **Note**: Operations 1-2 use `sleep 3600` as a placeholder command, focusing on validating ScalingAdapter and scaler configuration control plane behavior without requiring GPU. Operation 3 uses a simulated metrics endpoint to validate the full KEDA autoscaling flow (including scale-up and scale-down). Operation 4 requires a real inference service — replace with the full inference engine startup command. + +--- + +## Operation 1: Enable ScalingAdapter and Verify Scale Subresource + +### Step 1: Create an RBG with ScalingAdapter Enabled + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: scaling-demo +spec: + roles: + - name: backend + replicas: 2 + scalingAdapter: + enable: true + standalonePattern: + template: + spec: + containers: + - name: engine + image: lmsysorg/sglang:v0.5.9 + command: ["sleep", "3600"] + ports: + - containerPort: 8000 + resources: + requests: + cpu: "1" + memory: "1Gi" + limits: + cpu: "1" + memory: "1Gi" +EOF + +# Wait for Pods to be ready +kubectl wait --for=condition=ready pod -l rbg.workloads.x-k8s.io/group-name=scaling-demo --timeout=300s +``` + +### Step 2: Verify RBGSA Auto-Creation + +```bash +# Check auto-created RBGSA +kubectl get rbgsa + +> NAME PHASE REPLICAS READY_REPLICAS AGE +> scaling-demo-backend Bound 2 2 8m39s +``` + +```bash +# Check RBGSA details +kubectl get rbgsa scaling-demo-backend -o yaml + +> apiVersion: workloads.x-k8s.io/v1alpha2 +> kind: RoleBasedGroupScalingAdapter +> metadata: +> creationTimestamp: "2026-07-08T03:06:50Z" +> generation: 2 +> labels: +> rbg.workloads.x-k8s.io/group-name: scaling-demo +> rbg.workloads.x-k8s.io/role-name: backend +> name: scaling-demo-backend +> ownerReferences: +> - apiVersion: workloads.x-k8s.io/v1alpha2 +> blockOwnerDeletion: true +> kind: RoleBasedGroup +> name: scaling-demo +> uid: 91f2c7da-9a0f-4c6c-b032-7befc81ca1e6 +> resourceVersion: "27637990" +> uid: bbf80047-5e7d-4eac-9f49-6b87d045dceb +> spec: +> replicas: 2 +> scaleTargetRef: +> name: scaling-demo +> role: backend +> status: +> phase: Bound +> readyReplicas: 2 +> replicas: 2 +> selector: rbg.workloads.x-k8s.io/group-name=scaling-demo,rbg.workloads.x-k8s.io/group-uid=8481eaa0eb18d5d84918728022b8816dfb465d0c,rbg.workloads.x-k8s.io/role-name=backend +``` + +**Expected output:** +- RBGSA `scaling-demo-backend` auto-created (naming convention: `-`) +- RBGSA's `status.replicas` is 2 +- RBGSA's `status.selector` contains Pod label selector +- RBGSA's `status.phase` is `Ready` + +### Step 3: Manual Scaling via Scale Subresource + +```bash +# Scale up to 4 replicas +kubectl scale rbgsa scaling-demo-backend --replicas=4 + +> rolebasedgroupscalingadapter.workloads.x-k8s.io/scaling-demo-backend scaled +``` + +```bash +# Check Pod count changes +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=scaling-demo + +> NAME READY STATUS RESTARTS AGE +> scaling-demo-backend-0 1/1 Running 0 11m +> scaling-demo-backend-1 1/1 Running 0 11m +> scaling-demo-backend-2 0/1 ContainerCreating 0 12s +> scaling-demo-backend-3 0/1 ContainerCreating 0 12s +``` + +```bash +# Verify replica count has synced to RBG +kubectl get rbg scaling-demo -o jsonpath='{range .spec.roles[*]}{.name}{"="}{.replicas}{"\n"}{end}' + +> backend=4 +``` + +```bash +# Scale down to 1 replica +kubectl scale rbgsa scaling-demo-backend --replicas=1 + +> rolebasedgroupscalingadapter.workloads.x-k8s.io/scaling-demo-backend scaled +``` + +```bash +# Check Pod count changes +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=scaling-demo + +> NAME READY STATUS RESTARTS AGE +> scaling-demo-backend-0 1/1 Running 0 12m +``` + +**Expected output:** +- After scaling up, Pod count increases from 2 to 4 +- RBG's `spec.roles[0].replicas` syncs to 4 +- After scaling down, Pod count decreases to 1 + +### Cleanup + +```bash +kubectl delete rbg scaling-demo +``` + +--- + +## Operation 2: HPA Metric-Driven Scaling + +### Step 1: Create an RBG with ScalingAdapter Enabled + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: hpa-demo +spec: + roles: + - name: backend + replicas: 1 + scalingAdapter: + enable: true + standalonePattern: + template: + spec: + containers: + - name: engine + image: lmsysorg/sglang:v0.5.9 + command: ["sleep", "3600"] + ports: + - containerPort: 8000 + resources: + requests: + cpu: "500m" + memory: "512Mi" + limits: + cpu: "1" + memory: "1Gi" +EOF + +# Wait for Pod to be ready +kubectl wait --for=condition=ready pod -l rbg.workloads.x-k8s.io/group-name=hpa-demo --timeout=300s +``` + +### Step 2: Create HPA + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: hpa-demo-backend +spec: + scaleTargetRef: + apiVersion: workloads.x-k8s.io/v1alpha2 + kind: RoleBasedGroupScalingAdapter + name: hpa-demo-backend # - + minReplicas: 1 + maxReplicas: 10 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 50 +EOF +``` + +### Step 3: Verify HPA Configuration + +```bash +# Check HPA status +kubectl get hpa hpa-demo-backend + +> NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE +> hpa-demo-backend RoleBasedGroupScalingAdapter/hpa-demo-backend cpu: 0%/50% 1 10 1 30s +``` + +```bash +# Check HPA detailed configuration and status +kubectl get hpa hpa-demo-backend -o yaml + +> apiVersion: autoscaling/v2 +> kind: HorizontalPodAutoscaler +> metadata: +> creationTimestamp: "2026-07-08T03:36:00Z" +> name: hpa-demo-backend +> resourceVersion: "27658487" +> uid: 74789e96-7b21-444f-ae77-7384afb81d6d +> spec: +> maxReplicas: 10 +> metrics: +> - resource: +> name: cpu +> target: +> averageUtilization: 50 +> type: Utilization +> type: Resource +> minReplicas: 1 +> scaleTargetRef: +> apiVersion: workloads.x-k8s.io/v1alpha2 +> kind: RoleBasedGroupScalingAdapter +> name: hpa-demo-backend +> status: +> conditions: +> - lastTransitionTime: "2026-07-08T03:36:15Z" +> message: recent recommendations were higher than current one, applying the highest +> recent recommendation +> reason: ScaleDownStabilized +> status: "True" +> type: AbleToScale +> - lastTransitionTime: "2026-07-08T03:36:30Z" +> message: the HPA was able to successfully calculate a replica count from cpu resource +> utilization (percentage of request) +> reason: ValidMetricFound +> status: "True" +> type: ScalingActive +> - lastTransitionTime: "2026-07-08T03:36:30Z" +> message: the desired count is within the acceptable range +> reason: DesiredWithinRange +> status: "False" +> type: ScalingLimited +> currentMetrics: +> - resource: +> current: +> averageUtilization: 0 +> averageValue: "0" +> name: cpu +> type: Resource +> currentReplicas: 1 +> desiredReplicas: 1 +``` + +```bash +# Confirm HPA target points to RBGSA +kubectl get hpa hpa-demo-backend -o jsonpath='{.spec.scaleTargetRef}' + +> {"apiVersion":"workloads.x-k8s.io/v1alpha2","kind":"RoleBasedGroupScalingAdapter","name":"hpa-demo-backend"} +``` + +```bash +# Check RBGSA status (HPA controls replicas via RBGSA's Scale subresource) +kubectl get rbgsa hpa-demo-backend + +> NAME PHASE REPLICAS READY_REPLICAS AGE +> hpa-demo-backend Bound 1 1 2m43s +``` + +**Expected output:** +- HPA's `TARGETS` column shows current CPU utilization and target value (e.g., `0%/50%`) +- HPA's `scaleTargetRef.kind` is `RoleBasedGroupScalingAdapter` +- HPA's `scaleTargetRef.name` is `hpa-demo-backend` +- RBGSA's `status.replicas` matches the replica count managed by HPA + +### Step 4: Trigger Scaling and Observe + +```bash +# Generate CPU load in the Pod (trigger scale-up) +# Start 4 infinite loop processes to continuously consume CPU +POD_NAME=$(kubectl get pods -l rbg.workloads.x-k8s.io/group-name=hpa-demo -o jsonpath='{.items[0].metadata.name}') +kubectl exec ${POD_NAME} -- python3 -c " +import multiprocessing +def burn(): + while True: + pass +for _ in range(4): + multiprocessing.Process(target=burn).start() +" +``` + +```bash +# Continuously observe HPA status +kubectl get hpa hpa-demo-backend -w + +> NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE +> hpa-demo-backend RoleBasedGroupScalingAdapter/hpa-demo-backend cpu: 0%/50% 1 10 1 111s +> hpa-demo-backend RoleBasedGroupScalingAdapter/hpa-demo-backend cpu: 199%/50% 1 10 1 2m30s +``` + +```bash +# Observe Pod count changes +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=hpa-demo -w + +> NAME READY STATUS RESTARTS AGE +> hpa-demo-backend-0 1/1 Running 0 3m1s +> hpa-demo-backend-1 0/1 Pending 0 0s +> hpa-demo-backend-2 0/1 Pending 0 0s +> hpa-demo-backend-3 0/1 Pending 0 0s +> hpa-demo-backend-1 0/1 ContainerCreating 0 0s +> hpa-demo-backend-2 0/1 ContainerCreating 0 0s +> hpa-demo-backend-3 0/1 ContainerCreating 0 0s +``` + +> **Note**: `sleep 3600` itself does not consume CPU. To trigger HPA scale-up, replace the command with a CPU-intensive command (e.g., `python3 -c "while True: pass"`), or use custom metrics. This section primarily validates HPA configuration correctness. + +### Cleanup + +```bash +kubectl delete hpa hpa-demo-backend +kubectl delete rbg hpa-demo +``` + +--- + +## Operation 3: KEDA Event-Driven Scaling + +> **Note**: This operation validates the full KEDA autoscaling flow through a simulated metrics endpoint without requiring GPU. A Python HTTP server runs inside the Pod to simulate the `sglang_num_queue_requests` metric, and the metric value is controlled via an HTTP interface to trigger scale-up and scale-down. + +### Step 1: Deploy Prometheus + +```bash +# Deploy Prometheus, configured to auto-scrape Pods with prometheus.io/scrape: "true" annotation +cat <<'EOF' | kubectl apply -f - +apiVersion: v1 +kind: Namespace +metadata: + name: monitoring +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: prometheus + namespace: monitoring +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: prometheus +rules: + - apiGroups: [""] + resources: ["pods", "nodes", "endpoints", "services"] + verbs: ["get", "list", "watch"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: prometheus +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: prometheus +subjects: + - kind: ServiceAccount + name: prometheus + namespace: monitoring +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: prometheus-config + namespace: monitoring +data: + prometheus.yml: | + global: + scrape_interval: 5s + scrape_configs: + - job_name: 'kubernetes-pods' + kubernetes_sd_configs: + - role: pod + relabel_configs: + - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] + action: keep + regex: "true" + - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] + action: replace + regex: ([^:]+)(?::\d+)?;(\d+) + replacement: $1:$2 + target_label: __address__ +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: prometheus + namespace: monitoring +spec: + replicas: 1 + selector: + matchLabels: + app: prometheus + template: + metadata: + labels: + app: prometheus + spec: + serviceAccountName: prometheus + containers: + - name: prometheus + image: prom/prometheus:latest + args: + - "--config.file=/etc/prometheus/prometheus.yml" + - "--storage.tsdb.path=/prometheus" + ports: + - containerPort: 9090 + volumeMounts: + - name: config + mountPath: /etc/prometheus + - name: storage + mountPath: /prometheus + volumes: + - name: config + configMap: + name: prometheus-config + - name: storage + emptyDir: {} +--- +apiVersion: v1 +kind: Service +metadata: + name: prometheus + namespace: monitoring +spec: + selector: + app: prometheus + ports: + - port: 9090 + targetPort: 9090 +EOF + +# Wait for Prometheus to be ready +kubectl wait --for=condition=ready pod -l app=prometheus -n monitoring --timeout=60s +``` + +### Step 2: Create an RBG with ScalingAdapter Enabled (with Simulated Metrics Endpoint) + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: keda-demo +spec: + roles: + - name: backend + replicas: 1 + scalingAdapter: + enable: true + standalonePattern: + template: + metadata: + annotations: + prometheus.io/scrape: "true" + prometheus.io/port: "9091" + spec: + containers: + - name: engine + image: lmsysorg/sglang:v0.5.9 + command: + - python3 + - -c + - | + import http.server + from urllib.parse import urlparse, parse_qs + queue = [0] + class H(http.server.BaseHTTPRequestHandler): + def do_GET(self): + if self.path == '/metrics': + self.send_response(200) + self.send_header('Content-Type', 'text/plain') + self.end_headers() + self.wfile.write(f'sglang_num_queue_requests{{rbg="keda-demo",role="backend"}} {queue[0]}\n'.encode()) + elif self.path.startswith('/set'): + q = parse_qs(urlparse(self.path).query) + queue[0] = int(q.get('value', ['0'])[0]) + self.send_response(200) + self.end_headers() + self.wfile.write(b'OK') + else: + self.send_response(404) + self.end_headers() + def log_message(self, *a): pass + http.server.HTTPServer(('0.0.0.0', 9091), H).serve_forever() + ports: + - containerPort: 9091 + resources: + requests: + cpu: "1" + memory: "1Gi" + limits: + cpu: "1" + memory: "1Gi" +EOF + +# Wait for Pod to be ready +kubectl wait --for=condition=ready pod -l rbg.workloads.x-k8s.io/group-name=keda-demo --timeout=300s +``` + +> **Note**: A Python HTTP server runs inside the Pod, simulating SGLang's Prometheus metrics endpoint: +> - `GET /metrics`: Returns `sglang_num_queue_requests{rbg="keda-demo",role="backend"} 0` +> - `GET /set?value=200`: Sets the metric value to 200 (for triggering scale-up) +> - `GET /set?value=0`: Sets the metric value to 0 (for triggering scale-down) + +### Step 3: Create KEDA ScaledObject + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: keda-demo-backend +spec: + scaleTargetRef: + apiVersion: workloads.x-k8s.io/v1alpha2 + kind: RoleBasedGroupScalingAdapter + name: keda-demo-backend # - + minReplicaCount: 1 + maxReplicaCount: 10 + pollingInterval: 30 + triggers: + - type: prometheus + metadata: + serverAddress: http://prometheus.monitoring:9090 + metricName: inference_queue_length + query: | + avg(sglang_num_queue_requests{rbg="keda-demo", role="backend"}) + threshold: "100" +EOF +``` + +### Step 4: Verify KEDA Configuration and Metric Collection + +```bash +# Verify Prometheus has collected the metric (wait about 10s for Prometheus to complete first scrape) +kubectl run tmp-curl --rm -i --restart=Never --image=curlimages/curl:latest -- \ + curl -s 'http://prometheus.monitoring:9090/api/v1/query?query=sglang_num_queue_requests' + +> {"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"sglang_num_queue_requests","instance":"10.xx.xx.xx:9091","job":"kubernetes-pods","rbg":"keda-demo","role":"backend"},"value":[1783493293.481,"0"]}]}} +``` + +```bash +# Check ScaledObject status +kubectl get scaledobject keda-demo-backend + +> NAME SCALETARGETKIND SCALETARGETNAME MIN MAX READY ACTIVE FALLBACK PAUSED TRIGGERS AUTHENTICATIONS AGE +> keda-demo-backend workloads.x-k8s.io/v1alpha2.RoleBasedGroupScalingAdapter keda-demo-backend 1 10 True False False False prometheus 41s +``` + +```bash +# Check KEDA-managed HPA (KEDA creates an HPA in the background) +kubectl get hpa -l scaledobject.keda.sh/name=keda-demo-backend + +> NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE +> keda-hpa-keda-demo-backend RoleBasedGroupScalingAdapter/keda-demo-backend 0/100 (avg) 1 10 1 30s +``` + +**Expected output:** +- Prometheus query returns `sglang_num_queue_requests` value as `0` +- ScaledObject `READY=True`, `ACTIVE=False` (metric value 0 < threshold 100) +- KEDA auto-creates HPA (named `keda-hpa-keda-demo-backend`) pointing to RBGSA, `TARGETS` shows `0/100 (avg)` + +### Step 5: Trigger Scale-Up + +```bash +# Set queue depth to 200 (exceeds threshold 100), simulating inference request backlog +kubectl exec $(kubectl get pods -l rbg.workloads.x-k8s.io/group-name=keda-demo -o jsonpath='{.items[0].metadata.name}') -- \ + python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:9091/set?value=200')" + +# Wait for HPA polling (default 15-30s), observe HPA status changes +kubectl get hpa -l scaledobject.keda.sh/name=keda-demo-backend -w + +> NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE +> keda-hpa-keda-demo-backend RoleBasedGroupScalingAdapter/keda-demo-backend 0/100 (avg) 1 10 1 88s +> keda-hpa-keda-demo-backend RoleBasedGroupScalingAdapter/keda-demo-backend 200/100 (avg) 1 10 1 90s +``` + +```bash +# Observe Pod count changes +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=keda-demo -w + +> NAME READY STATUS RESTARTS AGE +> keda-demo-backend-0 1/1 Running 0 2m32s +> keda-demo-backend-1 0/1 ContainerCreating 0 26s +> keda-demo-backend-1 1/1 Running 0 29s +``` + +> **Note**: KEDA queries Prometheus for `avg(sglang_num_queue_requests) = 200 > threshold = 100`, calculates desired replicas = `ceil(1 × 200/100) = 2`, and scales up to 2 replicas via the RBGSA Scale subresource. After scaling up, HPA shows `100/100 (avg)`: KEDA's returned Prometheus query result is the average of 2 Pods `(200+0)/2 = 100`, equal to the threshold 100, so HPA does not continue scaling up. The system stabilizes at 2 replicas. + +### Step 6: Trigger Scale-Down + +```bash +# Set queue depth to 0, simulating request clearance +kubectl exec $(kubectl get pods -l rbg.workloads.x-k8s.io/group-name=keda-demo -o jsonpath='{.items[0].metadata.name}') -- \ + python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:9091/set?value=0')" + +# Wait for HPA scale-down (about 1-3 minutes), observe Pod changes +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=keda-demo -w + +> NAME READY STATUS RESTARTS AGE +> keda-demo-backend-0 1/1 Running 0 9m +> keda-demo-backend-1 1/1 Terminating 0 5m58s +``` + +```bash +# Verify HPA has scaled down to 1 replica +kubectl get hpa -l scaledobject.keda.sh/name=keda-demo-backend + +> NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE +> keda-hpa-keda-demo-backend RoleBased.../keda-demo-backend 0/100 (avg) 1 10 1 7m42s +``` + +> **Note**: After the metric drops to 0, HPA detects the metric below threshold within 1-3 minutes and completes the scale-down. KEDA does not set a custom scaleDown behavior by default — the scale-down speed depends on the HPA controller's default downscale stabilization window (usually within 5 minutes). + +### Cleanup + +```bash +kubectl delete scaledobject keda-demo-backend +kubectl delete rbg keda-demo +# (Optional) Delete Prometheus +# kubectl delete namespace monitoring +``` + +--- + +## Operation 4: RBG Planner SLA-Driven Scaling (PD Disaggregation) + +> **Note**: This operation requires GPU nodes and a real inference service, validating RBG Planner's PD-disaggregated intelligent scaling capability. If you only need to validate the AutoScaler CR configuration, set `dryRun: true`. + +### Step 1: Create PD-Disaggregated Inference Service (with ScalingAdapter Enabled) + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: pd-autoscaler-demo + namespace: inference +spec: + roles: + - name: prefill + replicas: 2 + scalingAdapter: + enable: true + standalonePattern: + template: + spec: + containers: + - name: engine + image: lmsysorg/sglang:v0.5.9 + command: + - python3 + - -m + - sglang.launch_server + - --model-path + - "Qwen/Qwen3-0.6B" + - --host + - "0.0.0.0" + - --port + - "8000" + - --disaggregation-mode + - "prefill" + - --tp-size + - "1" + ports: + - name: http + containerPort: 8000 + resources: + requests: + nvidia.com/gpu: "1" + limits: + nvidia.com/gpu: "1" + + - name: decode + replicas: 4 + scalingAdapter: + enable: true + standalonePattern: + template: + spec: + containers: + - name: engine + image: lmsysorg/sglang:v0.5.9 + command: + - python3 + - -m + - sglang.launch_server + - --model-path + - "Qwen/Qwen3-0.6B" + - --host + - "0.0.0.0" + - --port + - "8000" + - --disaggregation-mode + - "decode" + - --tp-size + - "1" + ports: + - name: http + containerPort: 8000 + resources: + requests: + nvidia.com/gpu: "1" + limits: + nvidia.com/gpu: "1" +EOF + +# Wait for all Pods to be ready +kubectl wait --for=condition=ready pod -l rbg.workloads.x-k8s.io/group-name=pd-autoscaler-demo -n inference --timeout=600s +``` + +### Step 2: Create AutoScaler CR + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: inference-extension.rolebasedgroup.io/v1alpha1 +kind: AutoScaler +metadata: + name: pd-autoscaler-demo # Must match the RBG name + namespace: inference +spec: + scalingInterval: 180 + + pattern: + PDDisaggregated: + prefill: + roleName: prefill + minReplicas: 1 + maxReplicas: 10 + decode: + roleName: decode + minReplicas: 1 + maxReplicas: 10 + + implementation: + DynamoPlanner: + modelName: "Qwen/Qwen3-0.6B" + + # SLA targets (milliseconds) + ttft: 200.0 + itl: 20.0 + + # Load prediction configuration + loadPredictor: arima + predictionWindow: 50 + + # Correction factor + noCorrection: false + + # Dry-run mode (observe only, no scaling executed) + dryRun: true + + profiling: + image: "ghcr.io/sgl-project/rbg-profiler:latest" + + metricsEndpoint: + metricSource: sglang + port: 9091 +EOF +``` + +### Step 3: Verify AutoScaler Running Status + +```bash +# Check AutoScaler status +kubectl get autoscaler -n inference + +# Check AutoScaler details +kubectl get autoscaler pd-autoscaler-demo -n inference -o yaml + +# Check Planner Pod +kubectl get pods -n inference -l app=rbg-planner + +# Check Planner logs (observe predictions and computed results) +kubectl logs -n inference -l app=rbg-planner --tail=50 + +# Check current replica count decision (dryRun mode does not actually scale) +kubectl get autoscaler pd-autoscaler-demo -n inference -o jsonpath='{.status}' + +# Check RBG role actual replica counts (should remain unchanged in dryRun mode) +kubectl get rbg pd-autoscaler-demo -n inference -o jsonpath='{range .spec.roles[*]}{.name}{"="}{.replicas}{"\n"}{end}' + +# Check RBGSA status (one for each role) +kubectl get rbgsa -n inference +``` + +**Expected output:** +- AutoScaler status is `Ready` or `Profiling` (first run requires Profiling) +- Planner Pod running normally +- Logs show observed TTFT, ITL metrics and predicted replica counts +- In dryRun mode, RBG replica counts remain unchanged (prefill=2, decode=4) +- Two RBGSAs created: `pd-autoscaler-demo-prefill` and `pd-autoscaler-demo-decode` + +### Step 4: Disable dryRun to Enable Actual Scaling + +```bash +# After confirming dryRun observations are reasonable, disable dryRun +kubectl patch autoscaler pd-autoscaler-demo -n inference --type='json' \ + -p='[{"op": "replace", "path": "/spec/implementation/DynamoPlanner/dryRun", "value": false}]' + +# Continuously observe replica count changes +watch -n 5 'kubectl get rbg pd-autoscaler-demo -n inference -o jsonpath="{range .spec.roles[*]}{.name}={.replicas}\n{end}"' + +# Observe Pod changes +kubectl get pods -n inference -l rbg.workloads.x-k8s.io/group-name=pd-autoscaler-demo -w +``` + +> **Note**: After disabling dryRun, the Planner automatically adjusts Prefill and Decode replica counts based on SLA targets and load predictions. Observe for a while — if the scaling behavior meets expectations, the configuration is complete. For tuning, refer to the "Recommended Tuning Process" in the concept document. + +### Cleanup + +```bash +kubectl delete autoscaler pd-autoscaler-demo -n inference +kubectl delete rbg pd-autoscaler-demo -n inference +``` + +--- + +## Summary + +| Operation | Scaling Method | Verification Point | Key Expectation | +| --- | --- | --- | --- | +| ScalingAdapter basics | Manual `kubectl scale` | RBGSA auto-creation, Scale subresource available | After `kubectl scale rbgsa`, replica count syncs to RBG | +| HPA metric-driven | Reactive (CPU utilization) | HPA target points to RBGSA | HPA controls replicas via RBGSA Scale subresource | +| KEDA event-driven | Reactive (external metrics) | ScaledObject target points to RBGSA | KEDA creates HPA in background pointing to RBGSA | +| RBG Planner SLA-driven | Predictive (TTFT/ITL) | AutoScaler CR creation, dryRun observation | Planner computes optimal replica counts for Prefill and Decode as a whole | diff --git a/doc/best-practice/en/08-configuring-autoscaling.md b/doc/best-practice/en/08-configuring-autoscaling.md new file mode 100644 index 00000000..ec400ac4 --- /dev/null +++ b/doc/best-practice/en/08-configuring-autoscaling.md @@ -0,0 +1,500 @@ +# Configuring Autoscaling for RBG Services +## Overview +RBG inference service autoscaling is implemented through `scalingAdapter` — it exposes a standard Kubernetes Scale subresource for each role, allowing any scaling strategy to deliver replica count decisions to RBG. `scalingAdapter` itself does not provide a scaling strategy; it is simply an execution channel. + +Different scaling strategies drive this channel through their respective decision logic, mainly falling into two categories: + ++ **Metric/Event-driven scaling**: Using community scalers like HPA, KEDA, based on CPU/memory utilization, custom metrics, or external events for reactive scaling. ++ **SLA-driven scaling**: Using [RBG Planner](https://github.com/sgl-project/rbg-planner), based on TTFT/ITL latency targets, load prediction, and offline performance profiling for predictive scaling, specifically designed for PD-disaggregated inference scenarios. + +```plain +Scaling Strategy (Decision Layer) +┌─────────────┐ ┌─────────────┐ ┌─────────────┐ +│ HPA │ │ KEDA │ │ RBG Planner │ +│ (Metric-driven)│ │ (Event-driven)│ │ (SLA-driven) │ +└──────┬──────┘ └──────┬──────┘ └──────┬──────┘ + │ │ │ + └────────────────┼────────────────┘ + │ Writes spec.replicas + ▼ + ┌─────────────────────┐ + │ ScalingAdapter │ ← Unified execution channel + │ (Scale subresource) │ + └──────────┬──────────┘ + │ Sync replicas + ▼ + ┌─────────────────────┐ + │ RoleBasedGroup │ ← Pod scaling + └─────────────────────┘ +``` + +## Prerequisites ++ Kubernetes cluster version >= 1.24 ++ RBG Controller installed (see [Installation Guide](https://github.com/sgl-project/rbg)) ++ Metrics Server installed (HPA resource metrics dependency) ++ When using custom metrics, Prometheus Adapter or KEDA must be installed + +--- + +## ScalingAdapter: Unified Scaling Execution Channel +### How It Works +Each role in RBG can enable autoscaling via `scalingAdapter.enable: true`. Once enabled, RBG Controller automatically creates a `RoleBasedGroupScalingAdapter` (RBGSA for short) resource that implements the standard Kubernetes `/scale` subresource interface. Whether HPA, KEDA, or RBG Planner, they all deliver replica count decisions to RBG roles through this Scale subresource. + +```plain +HPA / KEDA / RBG Planner + │ + │ Writes spec.replicas + ▼ +RoleBasedGroupScalingAdapter (Scale subresource) + │ + │ Sync replicas + ▼ +RoleBasedGroup → Role Pod scaling +``` + +### Enabling ScalingAdapter +Set `scalingAdapter.enable: true` in the role's spec: + +```yaml +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: inference-cluster +spec: + roles: + - name: prefill + replicas: 2 + scalingAdapter: + enable: true + standalonePattern: + template: + spec: + containers: + - name: engine + image: lmsysorg/sglang:v0.5.9 + command: ["sleep", "3600"] + ports: + - containerPort: 8000 + resources: + requests: + cpu: "1" + memory: "1Gi" + limits: + cpu: "1" + memory: "1Gi" +``` + +#### Parameter Description +| Parameter | Type | Required | Default | Description | +| --- | --- | --- | --- | --- | +| `scalingAdapter.enable` | bool | No | `false` | Whether to enable autoscaling | +| `scalingAdapter.labels` | map[string]string | No | - | Custom labels attached to the RBGSA resource | + + +### Auto-Created RBGSA Naming Convention +When `scalingAdapter.enable: true`, the Controller auto-created RBGSA resource follows this naming convention: + +```plain +- +``` + +For example, if the RBG name is `inference-cluster` and the role name is `prefill`, the auto-created RBGSA name is `inference-cluster-prefill`. + +> **Note**: The RBGSA's lifecycle is bound to the RBG via OwnerReference. When the RBG is deleted or the role is removed, the corresponding RBGSA is automatically cleaned up. No manual RBGSA creation is needed. +> + +--- + +## Scenario 1: Metric-Driven Scaling — HPA +HPA (Horizontal Pod Autoscaler) is Kubernetes' built-in autoscaler that supports reactive scaling based on CPU and memory utilization. This is the simplest scaling approach, suitable for scenarios with clear resource utilization thresholds. + +### Configuration Example +```yaml +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: prefill-hpa +spec: + scaleTargetRef: + apiVersion: workloads.x-k8s.io/v1alpha2 + kind: RoleBasedGroupScalingAdapter + name: inference-cluster-prefill # - + minReplicas: 1 + maxReplicas: 10 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 70 +``` + +#### HPA Key Fields +| Field | Description | +| --- | --- | +| `scaleTargetRef.apiVersion` | Fixed as `workloads.x-k8s.io/v1alpha2` | +| `scaleTargetRef.kind` | Fixed as `RoleBasedGroupScalingAdapter` | +| `scaleTargetRef.name` | RBGSA name, format `-` | +| `minReplicas` | Minimum replica count | +| `maxReplicas` | Maximum replica count | +| `metrics` | List of scaling metrics | + + +> **Note**: HPA obtains the Pod label selector via RBGSA's `status.selector` to collect metric data — no additional labelSelector configuration needed. +> + +### HPA Configuration Recommendations for GPU Inference Services +For GPU inference services, CPU utilization typically does not accurately reflect load. It is recommended to follow community practices (such as KServe, vLLM production deployment solutions) and use custom metrics for scaling: + +```yaml +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: hpa-demo-backend +spec: + scaleTargetRef: + apiVersion: workloads.x-k8s.io/v1alpha2 + kind: RoleBasedGroupScalingAdapter + name: hpa-demo-backend + minReplicas: 1 + maxReplicas: 10 + metrics: + # Based on custom metrics exposed by the inference engine + - type: Pods + pods: + metric: + name: inference_queue_length # Inference request queue length + target: + type: AverageValue + averageValue: "100" # Per-instance average queue length threshold +``` + +Common custom metrics for inference services (need to be exposed to HPA via Prometheus Adapter): + +| Metric | Description | Applicable Scenario | +| --- | --- | --- | +| `inference_queue_length` | Number of queued inference requests | Queue-driven scaling | +| `inference_tokens_per_second` | Tokens processed per second | Throughput-driven scaling | +| `inference_kv_cache_usage` | KV Cache usage rate | VRAM-driven scaling | + + +> **Note**: Custom metrics require the inference engine to expose Prometheus-format metrics at the `/metrics` endpoint, and use [Prometheus Adapter](https://github.com/kubernetes-sigs/prometheus-adapter) to convert them to Custom Metrics API consumable by HPA. The specific metric names and configuration methods vary by inference engine — please refer to the corresponding engine's documentation. +> + +--- + +## Scenario 2: Event-Driven Scaling — KEDA +KEDA (Kubernetes Event-Driven Autoscaling) supports richer external metric sources, suitable for scaling based on message queue depth, external API latency, and other metrics. + +### Configuration Example +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: keda-demo-backend +spec: + scaleTargetRef: + apiVersion: workloads.x-k8s.io/v1alpha2 + kind: RoleBasedGroupScalingAdapter + name: keda-demo-backend + minReplicaCount: 1 + maxReplicaCount: 10 + pollingInterval: 30 + cooldownPeriod: 300 + triggers: + - type: prometheus + metadata: + serverAddress: http://prometheus.monitoring:9090 # Replace with the actual Prometheus address in your cluster + metricName: inference_queue_length + query: | + avg(sglang_num_queue_requests{rbg="keda-demo", role="backend"}) + threshold: "100" +``` + +#### KEDA Key Fields +| Field | Description | +| --- | --- | +| `scaleTargetRef` | Same as HPA, points to RBGSA | +| `pollingInterval` | Metric collection interval (seconds) | +| `cooldownPeriod` | Scale-down cooldown period (seconds), prevents frequent scale-downs | +| `triggers` | Trigger list, supports prometheus, kafka, redis, and many other types | + + +> **Note**: KEDA's `cooldownPeriod` is particularly important for inference services. Inference engines have long startup times (model loading, KV Cache initialization), and overly frequent scale-downs cause service jitter. It is recommended to set `cooldownPeriod` to 5-10 minutes. +> + +--- + +## Scenario 3: SLA-Driven Scaling — RBG Planner +### Aggregated Deployment: HPA/KEDA Is Sufficient +For aggregated (non-PD-disaggregated) inference services, HPA or KEDA can meet most scaling needs. The inference engine serves as a single role, and scaling decisions are relatively simple — just focus on a single role's resource utilization or custom metrics. + +### PD Disaggregation: HPA/KEDA Falls Short +In PD-disaggregated architecture, Prefill and Decode are two independent roles, each with its own HPA. Although CoordinatedPolicy can coordinate scaling progress, the fundamental problem with this approach is: **HPA/KEDA does not understand the characteristics of inference workloads**. + +1. **Asymmetric resource requirements for Prefill and Decode**: Prefill is compute-intensive (processing long prompts), Decode is memory-intensive (generating tokens one by one). Under the same request pattern, their resource consumption is completely different — simple progress coordination cannot solve the resource ratio problem +2. **Reactive scaling lag**: GPU inference engines take minutes to start (model loading, KV Cache initialization). By the time HPA detects rising metrics and scales up, users have already experienced latency. For PD-disaggregated scenarios, lag in one role causes the entire pipeline to block +3. **Lack of SLA awareness**: HPA is based on CPU/memory utilization or queue depth, and cannot directly use inference quality metrics like TTFT and ITL as scaling targets. In PD-disaggregated scenarios, Prefill affects TTFT, Decode affects ITL — both need independent optimization +4. **Lack of performance profiling**: Different models have vastly different throughput on different hardware. HPA cannot know "how many Prefill instances are needed to handle 2048-token prompts" — it can only passively wait for metrics to rise + +These problems cannot be solved by CoordinatedPolicy — it only controls scaling progress synchronization, not scaling decision intelligence. PD-disaggregated inference needs a scaling strategy that **understands inference workloads and can simultaneously consider the relationship between Prefill and Decode**. + +### RBG Planner: Intelligent Scaling Designed for PD-Disaggregated Inference +[RBG Planner](https://github.com/sgl-project/rbg-planner) is an independent Kubernetes Operator that provides **SLA-driven predictive scaling** specifically for PD-disaggregated inference. Its core algorithm originates from the [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) project, adapted to native Kubernetes RBG API. Like HPA/KEDA, RBG Planner delivers replica count decisions to RBG through the ScalingAdapter's Scale subresource. + +Unlike HPA/KEDA which independently scale each role, RBG Planner makes scaling decisions for Prefill and Decode as a whole — based on the characteristics of the request load (input length, output length), it separately calculates the optimal replica count for both roles. + +RBG Planner's working loop: + +```plain +┌──────────────────────────────────────────────────────────────────┐ +│ RBG Planner Working Loop │ +│ │ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ +│ │ Observe │─→│ Predict │─→│ Compute │ │ +│ │ │ │ │ │ │ │ +│ │ Collect real- │ │ Predict next │ │ Based on │ │ +│ │ time metrics │ │ cycle request│ │ performance │ │ +│ │ TTFT, ITL, │ │ volume, ISL, │ │ profile │ │ +│ │ requests, ISL│ │ OSL (ARIMA) │ │ compute │ │ +│ │ │ │ │ │ required │ │ +│ │ │ │ │ │ Prefill/ │ │ +│ │ │ │ │ │ Decode repl. │ │ +│ └──────────────┘ └──────────────┘ └──────┬───────┘ │ +│ │ │ +│ ▼ │ +│ ┌──────────────┐ │ +│ │ Scale │ Via RBGSA or direct RBG Patch │ +│ │ │ Write replica count │ +│ └──────────────┘ │ +│ │ +│ Execute every N seconds (default 180 seconds) │ +└──────────────────────────────────────────────────────────────────┘ +``` + +### Key Capabilities +| Capability | Description | +| --- | --- | +| SLA target driven | Uses TTFT and ITL latency targets (milliseconds) as scaling constraints | +| Load prediction | Uses ARIMA/Prophet time series forecasting to anticipate load changes | +| Performance profiling | Obtains model throughput on specific hardware through offline Profiling, uses scipy cubic interpolation to calculate precise resource requirements | +| PD independent scaling | Separately calculates optimal replica counts for Prefill and Decode | +| GPU budget control | Sets total GPU limit, allocates resources proportionally within budget | +| Correction factor | Real-time comparison of observed vs. expected values, auto-corrects scaling decisions | + + +### Installing RBG Planner +```bash +helm install rbg-planner oci://ghcr.io/sgl-project/charts/rbg-planner \ + -n rbg-system --create-namespace +``` + +### Prerequisite: PD-Disaggregated Inference Service +RBG Planner requires the inference service to be deployed in PD-disaggregated architecture, with both roles having ScalingAdapter enabled: + +```yaml +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: pd-inference + namespace: inference +spec: + roles: + - name: prefill + replicas: 2 + scalingAdapter: + enable: true + standalonePattern: + template: + spec: + containers: + - name: engine + image: lmsysorg/sglang:v0.5.9 + ports: + - containerPort: 8000 + resources: + requests: + nvidia.com/gpu: "1" + limits: + nvidia.com/gpu: "1" + + - name: decode + replicas: 4 + scalingAdapter: + enable: true + standalonePattern: + template: + spec: + containers: + - name: engine + image: lmsysorg/sglang:v0.5.9 + ports: + - containerPort: 8000 + resources: + requests: + nvidia.com/gpu: "1" + limits: + nvidia.com/gpu: "1" +``` + +### Creating AutoScaler CR +```yaml +apiVersion: inference-extension.rolebasedgroup.io/v1alpha1 +kind: AutoScaler +metadata: + name: pd-inference # Must match the RBG name + namespace: inference # Must match the RBG namespace +spec: + scalingInterval: 180 # Scaling decision interval (seconds) + + pattern: + PDDisaggregated: + prefill: + roleName: prefill # Prefill role name in RBG + minReplicas: 1 + maxReplicas: 10 + decode: + roleName: decode # Decode role name in RBG + minReplicas: 1 + maxReplicas: 10 + + implementation: + DynamoPlanner: + modelName: "Qwen/Qwen3-0.6B" + + # SLA targets (milliseconds) + ttft: 200.0 # First token latency target + itl: 20.0 # Inter-token latency target + + # Load prediction configuration + loadPredictor: arima # Prediction algorithm: arima | constant | prophet + predictionWindow: 50 # Prediction window size (data points) + + # Correction factor + noCorrection: false # Whether to disable real-time correction + + # Dry-run mode (observe only, no scaling executed) + dryRun: false + + profiling: + image: "ghcr.io/sgl-project/rbg-profiler:latest" + + metricsEndpoint: + metricSource: sglang # Inference engine type: sglang | vllm | dynamo + port: 9091 # Metrics port +``` + +#### Parameter Description +| Parameter | Type | Description | +| --- | --- | --- | +| `spec.scalingInterval` | int | Scaling decision interval (seconds), default 180 | +| `spec.pattern.PDDisaggregated.prefill.roleName` | string | Prefill role name, must match the role name in RBG | +| `spec.pattern.PDDisAggregated.prefill.minReplicas` | int | Prefill minimum replicas | +| `spec.pattern.PDDisAggregated.prefill.maxReplicas` | int | Prefill maximum replicas | +| `spec.implementation.DynamoPlanner.modelName` | string | Model name, used for Profiling | +| `spec.implementation.DynamoPlanner.ttft` | float | TTFT SLA target (milliseconds) | +| `spec.implementation.DynamoPlanner.itl` | float | ITL SLA target (milliseconds) | +| `spec.implementation.DynamoPlanner.loadPredictor` | string | Load prediction algorithm: `arima` (recommended), `constant`, `prophet` | +| `spec.implementation.DynamoPlanner.predictionWindow` | int | Prediction window size (historical data points) | +| `spec.implementation.DynamoPlanner.noCorrection` | bool | Whether to disable real-time correction factor | +| `spec.implementation.DynamoPlanner.dryRun` | bool | Observe only without executing scaling (for tuning validation) | +| `spec.implementation.DynamoPlanner.profiling.image` | string | Profiler image address | +| `spec.implementation.DynamoPlanner.metricsEndpoint.metricSource` | string | Inference engine type | +| `spec.implementation.DynamoPlanner.metricsEndpoint.port` | int | Inference engine metrics port | + + +### Profiling Process +After creating the AutoScaler, the Operator automatically executes the following process: + +```plain +1. Create Profiling Job (one-time task) + └── Uses bench_serving to benchmark the model with multiple parameter sets + └── Generates performance profile data for Prefill and Decode + └── Saves as ConfigMap + +2. Deploy Planner Engine (long-running Deployment) + └── Loads Profiling data + └── Starts the observe-predict-compute-scale loop +``` + +The performance profile generated by the Profiler includes: + ++ **Prefill profile**: Input sequence length (ISL) → TTFT + per-GPU throughput (cubic spline interpolation) ++ **Decode profile**: KV Cache usage × context length → ITL + per-GPU throughput (2D scatter interpolation) + +> **Note**: Profiling only needs to be executed once. Performance profile data is stored in ConfigMap, and the AutoScaler automatically loads it on restart. If the model or hardware is changed, the AutoScaler needs to be deleted and recreated to trigger re-Profiling. +> + +### Verifying Planner Running Status +```bash +# Check AutoScaler status +kubectl get autoscaler -n inference + +# Check Planner Pod logs +kubectl logs -n inference -l app=rbg-planner + +# Check current replica count decision +kubectl get autoscaler pd-inference -n inference -o jsonpath='{.status.prefillReplicas}{""}{.status.decodeReplicas}' + +# Check RBG role replica counts (driven by Planner) +kubectl get rbg pd-inference -n inference -o jsonpath='{range .spec.roles[*]}{.name}{"="}{.replicas}{"\n"}{end}' +``` + +### Recommended Tuning Process +1. **Start with dryRun**: Set `dryRun: true`, Planner only observes and computes without executing scaling. Observe through logs whether predictions and computed results are reasonable. +2. **Adjust SLA targets**: Based on observations during dryRun, set reasonable TTFT/ITL targets. Targets that are too low cause over-scaling, too high degrades user experience. +3. **Disable dryRun**: After confirming the configuration is reasonable, set `dryRun: false` to enable actual scaling. +4. **Observe correction factor**: Planner calculates correction factor in real-time (observed / expected). If it persistently runs high, it means actual performance is below profile expectations — re-Profiling may be needed. + +--- + +## Solution Selection Recommendations +### By Deployment Architecture +| Deployment Architecture | Recommended Solution | Rationale | +| --- | --- | --- | +| **Aggregated** (non-PD-disaggregated) | HPA or KEDA | Inference engine as a single role, simple scaling decisions, resource utilization or custom metrics meet the needs | +| **PD-disaggregated** | RBG Planner | HPA/KEDA scales each role independently, cannot understand the resource ratio relationship between Prefill and Decode; RBG Planner makes SLA-driven scaling decisions for both as a whole | + + +### Capability Comparison +| Dimension | HPA | KEDA | RBG Planner | +| --- | --- | --- | --- | +| Decision type | Reactive (metric threshold) | Reactive (external events) | Predictive (SLA + load prediction) | +| Metric source | CPU/memory/custom metrics | External metric sources (Prometheus, Kafka, etc.) | Inference engine Prometheus metrics | +| SLA awareness | No | No | Yes (TTFT/ITL) | +| Load prediction | No | No | Yes (ARIMA/Prophet) | +| Performance profiling | No | No | Yes (offline Profiling) | +| PD coordination | Limited (CoordinatedPolicy only syncs progress) | Limited (CoordinatedPolicy only syncs progress) | Built-in (holistic computation of Prefill/Decode optimal ratio) | +| GPU budget control | No | No | Yes | +| Execution channel | ScalingAdapter | ScalingAdapter | ScalingAdapter | +| Applicable architecture | Aggregated | Aggregated | PD-disaggregated | + + +**Key difference**: HPA/KEDA creates independent scalers for each role. Even with CoordinatedPolicy, it can only control scaling progress synchronization — it cannot make intelligent resource ratio decisions based on inference workload characteristics (e.g., Prefill compute-intensive vs Decode memory-intensive). RBG Planner treats Prefill and Decode as a whole, calculating optimal replica counts for both roles based on request characteristics (input length, output length) and performance profiles. + +--- + +## Verify Scaling Status +```bash +# Check HPA status +kubectl get hpa + +# Check KEDA ScaledObject status +kubectl get scaledobject + +# Check AutoScaler status +kubectl get autoscaler -n inference + +# Check RBGSA status +kubectl get rbgsa + +# Check RBG role replica counts +kubectl get rbg -o wide +``` + +## Related Documents ++ [Deploying Inference Services with RBG](#) ++ [Configuring Rolling Update Strategies](#) ++ [In-Place Update and In-Place Scheduling](#) ++ [RBG Planner Project](https://github.com/sgl-project/rbg-planner) diff --git a/doc/best-practice/zh/08-configuring-autoscaling-guide.md b/doc/best-practice/zh/08-configuring-autoscaling-guide.md new file mode 100644 index 00000000..2441f5d6 --- /dev/null +++ b/doc/best-practice/zh/08-configuring-autoscaling-guide.md @@ -0,0 +1,875 @@ +# 操作文档：为 RBG 服务配置弹性伸缩策略 + +> 对应概念文档：[8. 为 RBG 服务配置弹性伸缩策略](08-configuring-autoscaling.md) + +## 目标 + +验证 RBG 的弹性伸缩机制，包括： +1. 启用 ScalingAdapter 并通过 Scale 子资源手动伸缩 +2. 配置 HPA 实现指标驱动伸缩 +3. 配置 KEDA 实现事件驱动伸缩 +4. 配置 RBG Planner 实现 SLA 驱动伸缩（PD 分离场景） + +## 前提条件 + +- Kubernetes 集群版本 >= 1.24 +- 已安装 RBG Controller +- 镜像可访问：`lmsysorg/sglang:v0.5.9` +- 操作二需要安装 Metrics Server +- 操作三需要安装 KEDA 和 Prometheus +- 操作四需要安装 RBG Planner（`helm install rbg-planner oci://ghcr.io/sgl-project/charts/rbg-planner -n rbg-system --create-namespace`） + +> **说明**：操作一至操作二使用 `sleep 3600` 作为占位命令，专注于验证 ScalingAdapter 和伸缩器配置的控制面行为，无需 GPU。操作三使用模拟指标端点验证 KEDA 自动伸缩全流程（含扩容和缩容）。操作四需要真实推理服务，请替换为完整的推理引擎启动命令。 + +--- + +## 操作一：启用 ScalingAdapter 并验证 Scale 子资源 + +### 步骤 1：创建启用 ScalingAdapter 的 RBG + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: scaling-demo +spec: + roles: + - name: backend + replicas: 2 + scalingAdapter: + enable: true + standalonePattern: + template: + spec: + containers: + - name: engine + image: lmsysorg/sglang:v0.5.9 + command: ["sleep", "3600"] + ports: + - containerPort: 8000 + resources: + requests: + cpu: "1" + memory: "1Gi" + limits: + cpu: "1" + memory: "1Gi" +EOF + +# 等待 Pod 就绪 +kubectl wait --for=condition=ready pod -l rbg.workloads.x-k8s.io/group-name=scaling-demo --timeout=300s +``` + +### 步骤 2：验证 RBGSA 自动创建 + +```bash +# 查看自动创建的 RBGSA +kubectl get rbgsa + +> NAME PHASE REPLICAS READY_REPLICAS AGE +> scaling-demo-backend Bound 2 2 8m39s +``` + +```bash +# 查看 RBGSA 详情 +kubectl get rbgsa scaling-demo-backend -o yaml + +> apiVersion: workloads.x-k8s.io/v1alpha2 +> kind: RoleBasedGroupScalingAdapter +> metadata: +> creationTimestamp: "2026-07-08T03:06:50Z" +> generation: 2 +> labels: +> rbg.workloads.x-k8s.io/group-name: scaling-demo +> rbg.workloads.x-k8s.io/role-name: backend +> name: scaling-demo-backend +> ownerReferences: +> - apiVersion: workloads.x-k8s.io/v1alpha2 +> blockOwnerDeletion: true +> kind: RoleBasedGroup +> name: scaling-demo +> uid: 91f2c7da-9a0f-4c6c-b032-7befc81ca1e6 +> resourceVersion: "27637990" +> uid: bbf80047-5e7d-4eac-9f49-6b87d045dceb +> spec: +> replicas: 2 +> scaleTargetRef: +> name: scaling-demo +> role: backend +> status: +> phase: Bound +> readyReplicas: 2 +> replicas: 2 +> selector: rbg.workloads.x-k8s.io/group-name=scaling-demo,rbg.workloads.x-k8s.io/group-uid=8481eaa0eb18d5d84918728022b8816dfb465d0c,rbg.workloads.x-k8s.io/role-name=backend +``` + +**预期输出：** +- RBGSA `scaling-demo-backend` 自动创建（命名规则：`-`） +- RBGSA 的 `status.replicas` 为 2 +- RBGSA 的 `status.selector` 包含 Pod 标签选择器 +- RBGSA 的 `status.phase` 为 `Ready` + +### 步骤 3：通过 Scale 子资源手动伸缩 + +```bash +# 扩容到 4 副本 +kubectl scale rbgsa scaling-demo-backend --replicas=4 + +> rolebasedgroupscalingadapter.workloads.x-k8s.io/scaling-demo-backend scaled +``` + +```bash +# 查看 Pod 数量变化 +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=scaling-demo + +> NAME READY STATUS RESTARTS AGE +> scaling-demo-backend-0 1/1 Running 0 11m +> scaling-demo-backend-1 1/1 Running 0 11m +> scaling-demo-backend-2 0/1 ContainerCreating 0 12s +> scaling-demo-backend-3 0/1 ContainerCreating 0 12s +``` + +```bash +# 验证副本数已同步到 RBG +kubectl get rbg scaling-demo -o jsonpath='{range .spec.roles[*]}{.name}{"="}{.replicas}{"\n"}{end}' + +> backend=4 +``` + +```bash +# 缩容到 1 副本 +kubectl scale rbgsa scaling-demo-backend --replicas=1 + +> rolebasedgroupscalingadapter.workloads.x-k8s.io/scaling-demo-backend scaled +``` + +```bash +# 查看 Pod 数量变化 +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=scaling-demo + +> NAME READY STATUS RESTARTS AGE +> scaling-demo-backend-0 1/1 Running 0 12m +``` + +**预期输出：** +- 扩容后 Pod 数量从 2 增加到 4 +- RBG 的 `spec.roles[0].replicas` 同步更新为 4 +- 缩容后 Pod 数量减少到 1 + +### 清理 + +```bash +kubectl delete rbg scaling-demo +``` + +--- + +## 操作二：HPA 指标驱动伸缩 + +### 步骤 1：创建启用 ScalingAdapter 的 RBG + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: hpa-demo +spec: + roles: + - name: backend + replicas: 1 + scalingAdapter: + enable: true + standalonePattern: + template: + spec: + containers: + - name: engine + image: lmsysorg/sglang:v0.5.9 + command: ["sleep", "3600"] + ports: + - containerPort: 8000 + resources: + requests: + cpu: "500m" + memory: "512Mi" + limits: + cpu: "1" + memory: "1Gi" +EOF + +# 等待 Pod 就绪 +kubectl wait --for=condition=ready pod -l rbg.workloads.x-k8s.io/group-name=hpa-demo --timeout=300s +``` + +### 步骤 2：创建 HPA + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: hpa-demo-backend +spec: + scaleTargetRef: + apiVersion: workloads.x-k8s.io/v1alpha2 + kind: RoleBasedGroupScalingAdapter + name: hpa-demo-backend # - + minReplicas: 1 + maxReplicas: 10 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 50 +EOF +``` + +### 步骤 3：验证 HPA 配置 + +```bash +# 查看 HPA 状态 +kubectl get hpa hpa-demo-backend + +> NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE +> hpa-demo-backend RoleBasedGroupScalingAdapter/hpa-demo-backend cpu: 0%/50% 1 10 1 30s +``` + +```bash +# 查看 HPA 详细配置和状态 +kubectl get hpa hpa-demo-backend -o yaml + +> apiVersion: autoscaling/v2 +> kind: HorizontalPodAutoscaler +> metadata: +> creationTimestamp: "2026-07-08T03:36:00Z" +> name: hpa-demo-backend +> resourceVersion: "27658487" +> uid: 74789e96-7b21-444f-ae77-7384afb81d6d +> spec: +> maxReplicas: 10 +> metrics: +> - resource: +> name: cpu +> target: +> averageUtilization: 50 +> type: Utilization +> type: Resource +> minReplicas: 1 +> scaleTargetRef: +> apiVersion: workloads.x-k8s.io/v1alpha2 +> kind: RoleBasedGroupScalingAdapter +> name: hpa-demo-backend +> status: +> conditions: +> - lastTransitionTime: "2026-07-08T03:36:15Z" +> message: recent recommendations were higher than current one, applying the highest +> recent recommendation +> reason: ScaleDownStabilized +> status: "True" +> type: AbleToScale +> - lastTransitionTime: "2026-07-08T03:36:30Z" +> message: the HPA was able to successfully calculate a replica count from cpu resource +> utilization (percentage of request) +> reason: ValidMetricFound +> status: "True" +> type: ScalingActive +> - lastTransitionTime: "2026-07-08T03:36:30Z" +> message: the desired count is within the acceptable range +> reason: DesiredWithinRange +> status: "False" +> type: ScalingLimited +> currentMetrics: +> - resource: +> current: +> averageUtilization: 0 +> averageValue: "0" +> name: cpu +> type: Resource +> currentReplicas: 1 +> desiredReplicas: 1 +``` + +```bash +# 确认 HPA 目标指向 RBGSA +kubectl get hpa hpa-demo-backend -o jsonpath='{.spec.scaleTargetRef}' + +> {"apiVersion":"workloads.x-k8s.io/v1alpha2","kind":"RoleBasedGroupScalingAdapter","name":"hpa-demo-backend"} +``` + +```bash +# 查看 RBGSA 状态（HPA 通过 RBGSA 的 Scale 子资源控制副本数） +kubectl get rbgsa hpa-demo-backend + +> NAME PHASE REPLICAS READY_REPLICAS AGE +> hpa-demo-backend Bound 1 1 2m43s +``` + +**预期输出：** +- HPA 的 `TARGETS` 列显示当前 CPU 利用率和目标值（如 `0%/50%`） +- HPA 的 `scaleTargetRef.kind` 为 `RoleBasedGroupScalingAdapter` +- HPA 的 `scaleTargetRef.name` 为 `hpa-demo-backend` +- RBGSA 的 `status.replicas` 与 HPA 管理的副本数一致 + +### 步骤 4：触发伸缩并观察 + +```bash +# 在 Pod 中制造 CPU 负载（触发扩容） +# 启动 4 个死循环进程，持续消耗 CPU +POD_NAME=$(kubectl get pods -l rbg.workloads.x-k8s.io/group-name=hpa-demo -o jsonpath='{.items[0].metadata.name}') +kubectl exec ${POD_NAME} -- python3 -c " +import multiprocessing +def burn(): + while True: + pass +for _ in range(4): + multiprocessing.Process(target=burn).start() +" +``` + +```bash +# 持续观察 HPA 状态 +kubectl get hpa hpa-demo-backend -w + +> NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE +> hpa-demo-backend RoleBasedGroupScalingAdapter/hpa-demo-backend cpu: 0%/50% 1 10 1 111s +> hpa-demo-backend RoleBasedGroupScalingAdapter/hpa-demo-backend cpu: 199%/50% 1 10 1 2m30s +``` + +```bash +# 观察 Pod 数量变化 +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=hpa-demo -w + +> NAME READY STATUS RESTARTS AGE +> hpa-demo-backend-0 1/1 Running 0 3m1s +> hpa-demo-backend-1 0/1 Pending 0 0s +> hpa-demo-backend-2 0/1 Pending 0 0s +> hpa-demo-backend-3 0/1 Pending 0 0s +> hpa-demo-backend-1 0/1 ContainerCreating 0 0s +> hpa-demo-backend-2 0/1 ContainerCreating 0 0s +> hpa-demo-backend-3 0/1 ContainerCreating 0 0s +``` + +> **说明**：`sleep 3600` 本身不消耗 CPU。如需触发 HPA 扩容，可替换 command 为 CPU 密集型命令（如 `python3 -c "while True: pass"`），或使用自定义指标。此处主要验证 HPA 配置正确性。 + +### 清理 + +```bash +kubectl delete hpa hpa-demo-backend +kubectl delete rbg hpa-demo +``` + +--- + +## 操作三：KEDA 事件驱动伸缩 + +> **说明**：本操作通过模拟指标端点验证 KEDA 自动伸缩全流程，无需 GPU。Pod 内运行 Python HTTP 服务器模拟 `sglang_num_queue_requests` 指标，通过 HTTP 接口控制指标值触发扩缩容。 + +### 步骤 1：部署 Prometheus + +```bash +# 部署 Prometheus，配置为自动采集带 prometheus.io/scrape: "true" annotation 的 Pod +cat <<'EOF' | kubectl apply -f - +apiVersion: v1 +kind: Namespace +metadata: + name: monitoring +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: prometheus + namespace: monitoring +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: prometheus +rules: + - apiGroups: [""] + resources: ["pods", "nodes", "endpoints", "services"] + verbs: ["get", "list", "watch"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: prometheus +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: prometheus +subjects: + - kind: ServiceAccount + name: prometheus + namespace: monitoring +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: prometheus-config + namespace: monitoring +data: + prometheus.yml: | + global: + scrape_interval: 5s + scrape_configs: + - job_name: 'kubernetes-pods' + kubernetes_sd_configs: + - role: pod + relabel_configs: + - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] + action: keep + regex: "true" + - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] + action: replace + regex: ([^:]+)(?::\d+)?;(\d+) + replacement: $1:$2 + target_label: __address__ +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: prometheus + namespace: monitoring +spec: + replicas: 1 + selector: + matchLabels: + app: prometheus + template: + metadata: + labels: + app: prometheus + spec: + serviceAccountName: prometheus + containers: + - name: prometheus + image: prom/prometheus:latest + args: + - "--config.file=/etc/prometheus/prometheus.yml" + - "--storage.tsdb.path=/prometheus" + ports: + - containerPort: 9090 + volumeMounts: + - name: config + mountPath: /etc/prometheus + - name: storage + mountPath: /prometheus + volumes: + - name: config + configMap: + name: prometheus-config + - name: storage + emptyDir: {} +--- +apiVersion: v1 +kind: Service +metadata: + name: prometheus + namespace: monitoring +spec: + selector: + app: prometheus + ports: + - port: 9090 + targetPort: 9090 +EOF + +# 等待 Prometheus 就绪 +kubectl wait --for=condition=ready pod -l app=prometheus -n monitoring --timeout=60s +``` + +### 步骤 2：创建启用 ScalingAdapter 的 RBG（含模拟指标端点） + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: keda-demo +spec: + roles: + - name: backend + replicas: 1 + scalingAdapter: + enable: true + standalonePattern: + template: + metadata: + annotations: + prometheus.io/scrape: "true" + prometheus.io/port: "9091" + spec: + containers: + - name: engine + image: lmsysorg/sglang:v0.5.9 + command: + - python3 + - -c + - | + import http.server + from urllib.parse import urlparse, parse_qs + queue = [0] + class H(http.server.BaseHTTPRequestHandler): + def do_GET(self): + if self.path == '/metrics': + self.send_response(200) + self.send_header('Content-Type', 'text/plain') + self.end_headers() + self.wfile.write(f'sglang_num_queue_requests{{rbg="keda-demo",role="backend"}} {queue[0]}\n'.encode()) + elif self.path.startswith('/set'): + q = parse_qs(urlparse(self.path).query) + queue[0] = int(q.get('value', ['0'])[0]) + self.send_response(200) + self.end_headers() + self.wfile.write(b'OK') + else: + self.send_response(404) + self.end_headers() + def log_message(self, *a): pass + http.server.HTTPServer(('0.0.0.0', 9091), H).serve_forever() + ports: + - containerPort: 9091 + resources: + requests: + cpu: "1" + memory: "1Gi" + limits: + cpu: "1" + memory: "1Gi" +EOF + +# 等待 Pod 就绪 +kubectl wait --for=condition=ready pod -l rbg.workloads.x-k8s.io/group-name=keda-demo --timeout=300s +``` + +> **说明**：Pod 内运行 Python HTTP 服务器，模拟 SGLang 的 Prometheus 指标端点： +> - `GET /metrics`：返回 `sglang_num_queue_requests{rbg="keda-demo",role="backend"} 0` +> - `GET /set?value=200`：将指标值设为 200（用于触发扩容） +> - `GET /set?value=0`：将指标值设为 0（用于触发缩容） + +### 步骤 3：创建 KEDA ScaledObject + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: keda-demo-backend +spec: + scaleTargetRef: + apiVersion: workloads.x-k8s.io/v1alpha2 + kind: RoleBasedGroupScalingAdapter + name: keda-demo-backend # - + minReplicaCount: 1 + maxReplicaCount: 10 + pollingInterval: 30 + triggers: + - type: prometheus + metadata: + serverAddress: http://prometheus.monitoring:9090 + metricName: inference_queue_length + query: | + avg(sglang_num_queue_requests{rbg="keda-demo", role="backend"}) + threshold: "100" +EOF +``` + +### 步骤 4：验证 KEDA 配置和指标采集 + +```bash +# 验证 Prometheus 已采集到指标（等待约 10s 让 Prometheus 完成首次采集） +kubectl run tmp-curl --rm -i --restart=Never --image=curlimages/curl:latest -- \ + curl -s 'http://prometheus.monitoring:9090/api/v1/query?query=sglang_num_queue_requests' + +> {"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"sglang_num_queue_requests","instance":"10.xx.xx.xx:9091","job":"kubernetes-pods","rbg":"keda-demo","role":"backend"},"value":[1783493293.481,"0"]}]}} +``` + +```bash +# 查看 ScaledObject 状态 +kubectl get scaledobject keda-demo-backend + +> NAME SCALETARGETKIND SCALETARGETNAME MIN MAX READY ACTIVE FALLBACK PAUSED TRIGGERS AUTHENTICATIONS AGE +> keda-demo-backend workloads.x-k8s.io/v1alpha2.RoleBasedGroupScalingAdapter keda-demo-backend 1 10 True False False False prometheus 41s +``` + +```bash +# 查看 KEDA 管理的 HPA（KEDA 会在后台创建 HPA） +kubectl get hpa -l scaledobject.keda.sh/name=keda-demo-backend + +> NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE +> keda-hpa-keda-demo-backend RoleBasedGroupScalingAdapter/keda-demo-backend 0/100 (avg) 1 10 1 30s +``` + +**预期输出：** +- Prometheus 查询返回 `sglang_num_queue_requests` 值为 `0` +- ScaledObject `READY=True`，`ACTIVE=False`（指标值 0 < 阈值 100） +- KEDA 自动创建 HPA（名为 `keda-hpa-keda-demo-backend`）指向 RBGSA，`TARGETS` 显示 `0/100 (avg)` + +### 步骤 5：触发扩容 + +```bash +# 将队列深度设为 200（超过 threshold 100），模拟推理请求积压 +kubectl exec $(kubectl get pods -l rbg.workloads.x-k8s.io/group-name=keda-demo -o jsonpath='{.items[0].metadata.name}') -- \ + python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:9091/set?value=200')" + +# 等待 HPA 轮询（默认 15-30s），观察 HPA 状态变化 +kubectl get hpa -l scaledobject.keda.sh/name=keda-demo-backend -w + +> NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE +> keda-hpa-keda-demo-backend RoleBasedGroupScalingAdapter/keda-demo-backend 0/100 (avg) 1 10 1 88s +> keda-hpa-keda-demo-backend RoleBasedGroupScalingAdapter/keda-demo-backend 200/100 (avg) 1 10 1 90s +``` + +```bash +# 观察 Pod 数量变化 +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=keda-demo -w + +> NAME READY STATUS RESTARTS AGE +> keda-demo-backend-0 1/1 Running 0 2m32s +> keda-demo-backend-1 0/1 ContainerCreating 0 26s +> keda-demo-backend-1 1/1 Running 0 29s +``` + +> **说明**：KEDA 通过 Prometheus 查询到 `avg(sglang_num_queue_requests) = 200 > threshold = 100`，计算 desired replicas = `ceil(1 × 200/100) = 2`，通过 RBGSA Scale 子资源扩容到 2 副本。扩容后 HPA 显示 `100/100 (avg)`：KEDA 返回的 Prometheus 查询结果为 2 个 Pod 的平均值 `(200+0)/2 = 100`，等于阈值 100，HPA 不再继续扩容，系统稳定在 2 副本。 + +### 步骤 6：触发缩容 + +```bash +# 将队列深度设为 0，模拟请求清空 +kubectl exec $(kubectl get pods -l rbg.workloads.x-k8s.io/group-name=keda-demo -o jsonpath='{.items[0].metadata.name}') -- \ + python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:9091/set?value=0')" + +# 等待 HPA 缩容（约 1-3 分钟），观察 Pod 变化 +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=keda-demo -w + +> NAME READY STATUS RESTARTS AGE +> keda-demo-backend-0 1/1 Running 0 9m +> keda-demo-backend-1 1/1 Terminating 0 5m58s +``` + +```bash +# 验证 HPA 已缩容到 1 副本 +kubectl get hpa -l scaledobject.keda.sh/name=keda-demo-backend + +> NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE +> keda-hpa-keda-demo-backend RoleBased.../keda-demo-backend 0/100 (avg) 1 10 1 7m42s +``` + +> **说明**：指标降为 0 后，HPA 在 1-3 分钟内检测到指标低于阈值并完成缩容。KEDA 默认不设置自定义 scaleDown behavior，缩容速度取决于 HPA 控制器的默认 downscale stabilization 窗口（通常 5 分钟内）。 + +### 清理 + +```bash +kubectl delete scaledobject keda-demo-backend +kubectl delete rbg keda-demo +# （可选）删除 Prometheus +# kubectl delete namespace monitoring +``` + +--- + +## 操作四：RBG Planner SLA 驱动伸缩（PD 分离） + +> **说明**：本操作需要 GPU 节点和真实推理服务，验证 RBG Planner 的 PD 分离智能伸缩能力。如仅需验证 AutoScaler CR 配置，可设置 `dryRun: true`。 + +### 步骤 1：创建 PD 分离推理服务（启用 ScalingAdapter） + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: pd-autoscaler-demo + namespace: inference +spec: + roles: + - name: prefill + replicas: 2 + scalingAdapter: + enable: true + standalonePattern: + template: + spec: + containers: + - name: engine + image: lmsysorg/sglang:v0.5.9 + command: + - python3 + - -m + - sglang.launch_server + - --model-path + - "Qwen/Qwen3-0.6B" + - --host + - "0.0.0.0" + - --port + - "8000" + - --disaggregation-mode + - "prefill" + - --tp-size + - "1" + ports: + - name: http + containerPort: 8000 + resources: + requests: + nvidia.com/gpu: "1" + limits: + nvidia.com/gpu: "1" + + - name: decode + replicas: 4 + scalingAdapter: + enable: true + standalonePattern: + template: + spec: + containers: + - name: engine + image: lmsysorg/sglang:v0.5.9 + command: + - python3 + - -m + - sglang.launch_server + - --model-path + - "Qwen/Qwen3-0.6B" + - --host + - "0.0.0.0" + - --port + - "8000" + - --disaggregation-mode + - "decode" + - --tp-size + - "1" + ports: + - name: http + containerPort: 8000 + resources: + requests: + nvidia.com/gpu: "1" + limits: + nvidia.com/gpu: "1" +EOF + +# 等待所有 Pod 就绪 +kubectl wait --for=condition=ready pod -l rbg.workloads.x-k8s.io/group-name=pd-autoscaler-demo -n inference --timeout=600s +``` + +### 步骤 2：创建 AutoScaler CR + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: inference-extension.rolebasedgroup.io/v1alpha1 +kind: AutoScaler +metadata: + name: pd-autoscaler-demo # 必须与 RBG 名称一致 + namespace: inference +spec: + scalingInterval: 180 + + pattern: + PDDisaggregated: + prefill: + roleName: prefill + minReplicas: 1 + maxReplicas: 10 + decode: + roleName: decode + minReplicas: 1 + maxReplicas: 10 + + implementation: + DynamoPlanner: + modelName: "Qwen/Qwen3-0.6B" + + # SLA 目标（毫秒） + ttft: 200.0 + itl: 20.0 + + # 负载预测配置 + loadPredictor: arima + predictionWindow: 50 + + # 修正因子 + noCorrection: false + + # Dry-run 模式（仅观测，不执行伸缩） + dryRun: true + + profiling: + image: "ghcr.io/sgl-project/rbg-profiler:latest" + + metricsEndpoint: + metricSource: sglang + port: 9091 +EOF +``` + +### 步骤 3：验证 AutoScaler 运行状态 + +```bash +# 查看 AutoScaler 状态 +kubectl get autoscaler -n inference + +# 查看 AutoScaler 详情 +kubectl get autoscaler pd-autoscaler-demo -n inference -o yaml + +# 查看 Planner Pod +kubectl get pods -n inference -l app=rbg-planner + +# 查看 Planner 日志（观察预测和计算结果） +kubectl logs -n inference -l app=rbg-planner --tail=50 + +# 查看当前副本数决策（dryRun 模式下不实际伸缩） +kubectl get autoscaler pd-autoscaler-demo -n inference -o jsonpath='{.status}' + +# 查看 RBG 各角色实际副本数（dryRun 模式下应保持不变） +kubectl get rbg pd-autoscaler-demo -n inference -o jsonpath='{range .spec.roles[*]}{.name}{"="}{.replicas}{"\n"}{end}' + +# 查看 RBGSA 状态（两个角色各一个） +kubectl get rbgsa -n inference +``` + +**预期输出：** +- AutoScaler 状态为 `Ready` 或 `Profiling`（首次运行需进行 Profiling） +- Planner Pod 正常运行 +- 日志中显示观测到的 TTFT、ITL 指标和预测的副本数 +- dryRun 模式下 RBG 副本数保持不变（prefill=2, decode=4） +- 两个 RBGSA 被创建：`pd-autoscaler-demo-prefill` 和 `pd-autoscaler-demo-decode` + +### 步骤 4：关闭 dryRun 启用实际伸缩 + +```bash +# 确认 dryRun 观测数据合理后，关闭 dryRun +kubectl patch autoscaler pd-autoscaler-demo -n inference --type='json' \ + -p='[{"op": "replace", "path": "/spec/implementation/DynamoPlanner/dryRun", "value": false}]' + +# 持续观察副本数变化 +watch -n 5 'kubectl get rbg pd-autoscaler-demo -n inference -o jsonpath="{range .spec.roles[*]}{.name}={.replicas}\n{end}"' + +# 观察 Pod 变化 +kubectl get pods -n inference -l rbg.workloads.x-k8s.io/group-name=pd-autoscaler-demo -w +``` + +> **说明**：关闭 dryRun 后，Planner 会根据 SLA 目标和负载预测自动调整 Prefill 和 Decode 的副本数。观察一段时间后，如果伸缩行为符合预期，则配置完成。如需调参，可参考概念文档中的「推荐的调参流程」。 + +### 清理 + +```bash +kubectl delete autoscaler pd-autoscaler-demo -n inference +kubectl delete rbg pd-autoscaler-demo -n inference +``` + +--- + +## 总结 + +| 操作 | 伸缩方式 | 验证点 | 关键预期 | +| --- | --- | --- | --- | +| ScalingAdapter 基础 | 手动 `kubectl scale` | RBGSA 自动创建、Scale 子资源可用 | `kubectl scale rbgsa` 后副本数同步到 RBG | +| HPA 指标驱动 | 响应式（CPU 利用率） | HPA 目标指向 RBGSA | HPA 通过 RBGSA Scale 子资源控制副本数 | +| KEDA 事件驱动 | 响应式（外部指标） | ScaledObject 目标指向 RBGSA | KEDA 后台创建 HPA 指向 RBGSA | +| RBG Planner SLA 驱动 | 预测式（TTFT/ITL） | AutoScaler CR 创建、dryRun 观测 | Planner 将 Prefill 和 Decode 作为整体计算最优副本数 | diff --git a/doc/best-practice/zh/08-configuring-autoscaling.md b/doc/best-practice/zh/08-configuring-autoscaling.md new file mode 100644 index 00000000..4fc03039 --- /dev/null +++ b/doc/best-practice/zh/08-configuring-autoscaling.md @@ -0,0 +1,498 @@ +# 为 RBG 服务配置弹性伸缩策略 +## 概述 +RBG 推理服务的弹性伸缩通过 `scalingAdapter` 实现——它为每个角色暴露一个标准的 Kubernetes Scale 子资源，使得任意伸缩策略都可以将副本数决策下发到 RBG。`scalingAdapter` 本身不提供伸缩策略，它只是一个执行通道。 + +不同的伸缩策略通过各自的决策逻辑驱动这个通道，主要分为两类： + ++ **指标/事件驱动伸缩**：使用 HPA、KEDA 等社区伸缩器，基于 CPU/内存利用率、自定义指标或外部事件进行响应式伸缩。 ++ **SLA 驱动伸缩**：使用 [RBG Planner](https://github.com/sgl-project/rbg-planner)，基于 TTFT/ITL 延迟目标、负载预测和离线性能画像进行预测式伸缩，专门针对 PD 分离推理场景设计。 + +```plain +伸缩策略（决策层） +┌─────────────┐ ┌─────────────┐ ┌─────────────┐ +│ HPA │ │ KEDA │ │ RBG Planner │ +│ (指标驱动) │ │ (事件驱动) │ │ (SLA 驱动) │ +└──────┬──────┘ └──────┬──────┘ └──────┬──────┘ + │ │ │ + └────────────────┼────────────────┘ + │ 写入 spec.replicas + ▼ + ┌─────────────────────┐ + │ ScalingAdapter │ ← 统一执行通道 + │ (Scale 子资源) │ + └──────────┬──────────┘ + │ 同步副本数 + ▼ + ┌─────────────────────┐ + │ RoleBasedGroup │ ← Pod 伸缩 + └─────────────────────┘ +``` + +## 前提条件 ++ Kubernetes 集群版本 >= 1.24 ++ 已安装 RBG Controller（参考 [安装指南](https://github.com/sgl-project/rbg)） ++ Metrics Server 已安装（HPA 资源指标依赖） ++ 使用自定义指标时需要安装 Prometheus Adapter 或 KEDA + +--- + +## ScalingAdapter：统一的伸缩执行通道 +### 工作原理 +RBG 的每个角色可以通过 `scalingAdapter.enable: true` 启用弹性伸缩。启用后，RBG Controller 自动创建一个 `RoleBasedGroupScalingAdapter`（简称 RBGSA）资源，该资源实现了 Kubernetes 标准的 `/scale` 子资源接口。无论是 HPA、KEDA 还是 RBG Planner，都通过这个 Scale 子资源将副本数决策下发到 RBG 角色。 + +```plain +HPA / KEDA / RBG Planner + │ + │ 写入 spec.replicas + ▼ +RoleBasedGroupScalingAdapter (Scale 子资源) + │ + │ 同步副本数 + ▼ +RoleBasedGroup → 角色 Pod 伸缩 +``` + +### 启用 ScalingAdapter +在角色的 spec 中设置 `scalingAdapter.enable: true`： + +```yaml +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: inference-cluster +spec: + roles: + - name: prefill + replicas: 2 + scalingAdapter: + enable: true + standalonePattern: + template: + spec: + containers: + - name: engine + image: lmsysorg/sglang:v0.5.9 + command: ["sleep", "3600"] + ports: + - containerPort: 8000 + resources: + requests: + cpu: "1" + memory: "1Gi" + limits: + cpu: "1" + memory: "1Gi" +``` + +#### 参数说明 +| 参数 | 类型 | 是否必填 | 默认值 | 说明 | +| --- | --- | --- | --- | --- | +| `scalingAdapter.enable` | bool | 否 | `false` | 是否启用弹性伸缩 | +| `scalingAdapter.labels` | map[string]string | 否 | - | 附加到 RBGSA 资源的自定义标签 | + + +### 自动创建的 RBGSA 命名规则 +当 `scalingAdapter.enable: true` 时，Controller 自动创建的 RBGSA 资源遵循以下命名规则： + +```plain +- +``` + +例如，RBG 名为 `inference-cluster`，角色名为 `prefill`，则自动创建的 RBGSA 名称为 `inference-cluster-prefill`。 + +> **说明**：RBGSA 的生命周期通过 OwnerReference 绑定到 RBG。当 RBG 被删除或角色被移除时，对应的 RBGSA 会自动清理。无需手动创建 RBGSA。 +> + +--- + +## 场景一：指标驱动伸缩 — HPA +HPA（Horizontal Pod Autoscaler）是 Kubernetes 内置的伸缩器，支持基于 CPU 和内存利用率进行响应式伸缩。这是最简单的伸缩方式，适合对资源利用率有明确阈值的场景。 + +### 配置示例 +```yaml +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: prefill-hpa +spec: + scaleTargetRef: + apiVersion: workloads.x-k8s.io/v1alpha2 + kind: RoleBasedGroupScalingAdapter + name: inference-cluster-prefill # - + minReplicas: 1 + maxReplicas: 10 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 70 +``` + +#### HPA 关键字段 +| 字段 | 说明 | +| --- | --- | +| `scaleTargetRef.apiVersion` | 固定为 `workloads.x-k8s.io/v1alpha2` | +| `scaleTargetRef.kind` | 固定为 `RoleBasedGroupScalingAdapter` | +| `scaleTargetRef.name` | RBGSA 名称，格式为 `-` | +| `minReplicas` | 最小副本数 | +| `maxReplicas` | 最大副本数 | +| `metrics` | 伸缩指标列表 | + + +> **说明**：HPA 通过 RBGSA 的 `status.selector` 获取 Pod 标签选择器来采集指标数据，无需额外配置 labelSelector。 +> + +### GPU 推理服务的 HPA 配置建议 +对于 GPU 推理服务，CPU 利用率通常不能准确反映负载情况。建议参考社区实践（如 KServe、vLLM 生产部署方案），使用自定义指标进行伸缩： + +```yaml +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: hpa-demo-backend +spec: + scaleTargetRef: + apiVersion: workloads.x-k8s.io/v1alpha2 + kind: RoleBasedGroupScalingAdapter + name: hpa-demo-backend # - + minReplicas: 1 + maxReplicas: 10 + metrics: + # 基于推理引擎暴露的自定义指标 + - type: Pods + pods: + metric: + name: inference_queue_length # 推理请求队列长度 + target: + type: AverageValue + averageValue: "100" # 每实例平均队列长度阈值 +``` + +常见的推理服务自定义指标（需通过 Prometheus Adapter 暴露给 HPA）： + +| 指标 | 说明 | 适用场景 | +| --- | --- | --- | +| `inference_queue_length` | 推理请求排队数量 | 队列驱动的伸缩 | +| `inference_tokens_per_second` | 每秒处理的 token 数 | 吞吐量驱动的伸缩 | +| `inference_kv_cache_usage` | KV Cache 使用率 | 显存驱动的伸缩 | + + +> **说明**：自定义指标需要推理引擎在 `/metrics` 端点暴露 Prometheus 格式的指标，并通过 [Prometheus Adapter](https://github.com/kubernetes-sigs/prometheus-adapter) 转换为 HPA 可消费的 Custom Metrics API。具体的指标名称和配置方式因推理引擎而异，请参考对应引擎的文档。 +> + +--- + +## 场景二：事件驱动伸缩 — KEDA +KEDA（Kubernetes Event-Driven Autoscaling）支持更丰富的外部指标源，适合基于消息队列深度、外部 API 延迟等指标进行伸缩。 + +### 配置示例 +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: keda-demo-backend +spec: + scaleTargetRef: + apiVersion: workloads.x-k8s.io/v1alpha2 + kind: RoleBasedGroupScalingAdapter + name: keda-demo-backend + minReplicaCount: 1 + maxReplicaCount: 10 + pollingInterval: 30 + cooldownPeriod: 300 + triggers: + - type: prometheus + metadata: + serverAddress: http://prometheus.monitoring:9090 # 替换为集群中实际的 Prometheus 地址 + metricName: inference_queue_length + query: | + avg(sglang_num_queue_requests{rbg="keda-demo", role="backend"}) + threshold: "100" +``` + +#### KEDA 关键字段 +| 字段 | 说明 | +| --- | --- | +| `scaleTargetRef` | 与 HPA 相同，指向 RBGSA | +| `pollingInterval` | 指标采集间隔（秒） | +| `cooldownPeriod` | 缩容冷却时间（秒），防止频繁缩容 | +| `triggers` | 触发器列表，支持 prometheus、kafka、redis 等多种类型 | + + +> **说明**：KEDA 的 `cooldownPeriod` 对于推理服务尤为重要。推理引擎的启动时间较长（模型加载、KV Cache 初始化），过于频繁的缩容会导致服务抖动。建议将 `cooldownPeriod` 设置为 5-10 分钟。 +> + +--- + +## 场景三：SLA 驱动伸缩 — RBG Planner +### 聚合部署：HPA/KEDA 足够 +对于聚合部署（非 PD 分离）的推理服务，HPA 或 KEDA 可以满足大多数伸缩需求。推理引擎作为一个整体角色，伸缩决策相对简单——只需关注单一角色的资源利用率或自定义指标即可。 + +### PD 分离：HPA/KEDA 力不从心 +在 PD 分离架构中，Prefill 和 Decode 是两个独立角色，各自有独立的 HPA。虽然可以通过 CoordinatedPolicy 协调伸缩进度，但这种方式的根本问题在于：**HPA/KEDA 不理解推理工作负载的特性**。 + +1. **Prefill 和 Decode 的资源需求不对称**：Prefill 是计算密集型（处理长 prompt），Decode 是内存密集型（逐 token 生成）。相同的请求模式下，两者的资源消耗完全不同，简单的进度协调无法解决资源配比问题 +2. **响应式伸缩的滞后**：GPU 推理引擎启动需要数分钟（模型加载、KV Cache 初始化），HPA 检测到指标升高后再扩容，用户已经感知到延迟。对于 PD 分离场景，一个角色的滞后会导致整条链路阻塞 +3. **缺乏 SLA 感知**：HPA 基于 CPU/内存利用率或队列深度，无法直接以 TTFT、ITL 等推理质量指标作为伸缩目标。而 PD 分离场景下，Prefill 影响 TTFT，Decode 影响 ITL，两者需要独立优化 +4. **缺乏性能画像**：不同模型在不同硬件上的吞吐能力差异巨大。HPA 无法知道"处理 2048 token 的 prompt 需要多少 Prefill 实例"，只能被动等待指标升高 + +这些问题不是 CoordinatedPolicy 能解决的——它只控制伸缩进度的同步，不提供伸缩决策的智能。PD 分离推理需要一个**理解推理工作负载、能同时考虑 Prefill 和 Decode 关系**的伸缩策略。 + +### RBG Planner：为 PD 分离推理设计的智能伸缩 +[RBG Planner](https://github.com/sgl-project/rbg-planner) 是一个独立的 Kubernetes Operator，专门为 PD 分离推理提供 **SLA 驱动的预测式伸缩**。它的核心算法源自 [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) 项目，已适配为原生 Kubernetes RBG API。与 HPA/KEDA 一样，RBG Planner 通过 ScalingAdapter 的 Scale 子资源将副本数决策下发到 RBG。 + +与 HPA/KEDA 独立伸缩各角色不同，RBG Planner 将 Prefill 和 Decode 作为一个整体进行伸缩决策——根据请求负载的特征（输入长度、输出长度），分别计算两个角色所需的最优副本数。 + +RBG Planner 的工作循环： + +```plain +┌──────────────────────────────────────────────────────────────────┐ +│ RBG Planner 工作循环 │ +│ │ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ +│ │ Observe │─→│ Predict │─→│ Compute │ │ +│ │ │ │ │ │ │ │ +│ │ 采集实时指标 │ │ 预测下一周期 │ │ 基于性能画像 │ │ +│ │ TTFT, ITL, │ │ 请求量, ISL, │ │ 计算所需 │ │ +│ │ 请求量, ISL │ │ OSL (ARIMA) │ │ Prefill/ │ │ +│ │ │ │ │ │ Decode 副本数 │ │ +│ └──────────────┘ └──────────────┘ └──────┬───────┘ │ +│ │ │ +│ ▼ │ +│ ┌──────────────┐ │ +│ │ Scale │ 通过 RBGSA 或直接 Patch RBG │ +│ │ │ 写入副本数 │ +│ └──────────────┘ │ +│ │ +│ 每 N 秒执行一次（默认 180 秒） │ +└──────────────────────────────────────────────────────────────────┘ +``` + +### 关键能力 +| 能力 | 说明 | +| --- | --- | +| SLA 目标驱动 | 以 TTFT 和 ITL 延迟目标（毫秒）作为伸缩约束 | +| 负载预测 | 使用 ARIMA/Prophet 时间序列预测，提前感知负载变化 | +| 性能画像 | 通过离线 Profiling 获取模型在特定硬件上的吞吐能力，使用 scipy 三次插值计算精确的资源需求 | +| PD 独立伸缩 | 分别计算 Prefill 和 Decode 的最优副本数 | +| GPU 预算控制 | 设定总 GPU 上限，在预算内按比例分配资源 | +| 修正因子 | 实时对比观测值与预期值的偏差，自动修正伸缩决策 | + + +### 安装 RBG Planner +```bash +helm install rbg-planner oci://ghcr.io/sgl-project/charts/rbg-planner \ + -n rbg-system --create-namespace +``` + +### 前置条件：PD 分离推理服务 +RBG Planner 要求推理服务已部署为 PD 分离架构，且两个角色都启用了 ScalingAdapter： + +```yaml +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: pd-inference + namespace: inference +spec: + roles: + - name: prefill + replicas: 2 + scalingAdapter: + enable: true + standalonePattern: + template: + spec: + containers: + - name: engine + image: lmsysorg/sglang:v0.5.9 + ports: + - containerPort: 8000 + resources: + requests: + nvidia.com/gpu: "1" + limits: + nvidia.com/gpu: "1" + + - name: decode + replicas: 4 + scalingAdapter: + enable: true + standalonePattern: + template: + spec: + containers: + - name: engine + image: lmsysorg/sglang:v0.5.9 + ports: + - containerPort: 8000 + resources: + requests: + nvidia.com/gpu: "1" + limits: + nvidia.com/gpu: "1" +``` + +### 创建 AutoScaler CR +```yaml +apiVersion: inference-extension.rolebasedgroup.io/v1alpha1 +kind: AutoScaler +metadata: + name: pd-inference # 必须与 RBG 名称一致 + namespace: inference # 必须与 RBG 命名空间一致 +spec: + scalingInterval: 180 # 伸缩决策间隔（秒） + + pattern: + PDDisaggregated: + prefill: + roleName: prefill # RBG 中的 Prefill 角色名 + minReplicas: 1 + maxReplicas: 10 + decode: + roleName: decode # RBG 中的 Decode 角色名 + minReplicas: 1 + maxReplicas: 10 + + implementation: + DynamoPlanner: + modelName: "Qwen/Qwen3-0.6B" + + # SLA 目标（毫秒） + ttft: 200.0 # 首 token 延迟目标 + itl: 20.0 # token 间延迟目标 + + # 负载预测配置 + loadPredictor: arima # 预测算法：arima | constant | prophet + predictionWindow: 50 # 预测窗口大小（数据点数） + + # 修正因子 + noCorrection: false # 是否禁用实时修正 + + # Dry-run 模式（仅观测，不执行伸缩） + dryRun: false + + profiling: + image: "ghcr.io/sgl-project/rbg-profiler:latest" + + metricsEndpoint: + metricSource: sglang # 推理引擎类型：sglang | vllm | dynamo + port: 9091 # 指标端口 +``` + +#### 参数说明 +| 参数 | 类型 | 说明 | +| --- | --- | --- | +| `spec.scalingInterval` | int | 伸缩决策间隔（秒），默认 180 | +| `spec.pattern.PDDisaggregated.prefill.roleName` | string | Prefill 角色名称，必须与 RBG 中的角色名一致 | +| `spec.pattern.PDDisAggregated.prefill.minReplicas` | int | Prefill 最小副本数 | +| `spec.pattern.PDDisAggregated.prefill.maxReplicas` | int | Prefill 最大副本数 | +| `spec.implementation.DynamoPlanner.modelName` | string | 模型名称，用于 Profiling | +| `spec.implementation.DynamoPlanner.ttft` | float | TTFT SLA 目标（毫秒） | +| `spec.implementation.DynamoPlanner.itl` | float | ITL SLA 目标（毫秒） | +| `spec.implementation.DynamoPlanner.loadPredictor` | string | 负载预测算法：`arima`（推荐）、`constant`、`prophet` | +| `spec.implementation.DynamoPlanner.predictionWindow` | int | 预测窗口大小（历史数据点数） | +| `spec.implementation.DynamoPlanner.noCorrection` | bool | 是否禁用实时修正因子 | +| `spec.implementation.DynamoPlanner.dryRun` | bool | 仅观测不执行伸缩（用于调参验证） | +| `spec.implementation.DynamoPlanner.profiling.image` | string | Profiler 镜像地址 | +| `spec.implementation.DynamoPlanner.metricsEndpoint.metricSource` | string | 推理引擎类型 | +| `spec.implementation.DynamoPlanner.metricsEndpoint.port` | int | 推理引擎指标端口 | + + +### Profiling 流程 +创建 AutoScaler 后，Operator 会自动执行以下流程： + +```plain +1. 创建 Profiling Job（一次性任务） + └── 使用 bench_serving 对模型进行多组参数基准测试 + └── 生成 Prefill 和 Decode 的性能画像数据 + └── 保存为 ConfigMap + +2. 部署 Planner Engine（常驻 Deployment） + └── 加载 Profiling 数据 + └── 开始观测-预测-计算-伸缩循环 +``` + +Profiler 生成的性能画像包含： + ++ **Prefill 画像**：输入序列长度（ISL）→ TTFT + 每 GPU 吞吐量（三次样条插值） ++ **Decode 画像**：KV Cache 使用率 × 上下文长度 → ITL + 每 GPU 吞吐量（二维散点插值） + +> **说明**：Profiling 只需执行一次。性能画像数据存储在 ConfigMap 中，AutoScaler 重启后自动加载。如果更换模型或硬件，需要删除 AutoScaler 重新创建以触发重新 Profiling。 +> + +### 验证 Planner 运行状态 +```bash +# 查看 AutoScaler 状态 +kubectl get autoscaler -n inference + +# 查看 Planner Pod 日志 +kubectl logs -n inference -l app=rbg-planner + +# 查看当前副本数决策 +kubectl get autoscaler pd-inference -n inference -o jsonpath='{.status.prefillReplicas}{""}{.status.decodeReplicas}' + +# 查看 RBG 角色副本数（由 Planner 驱动） +kubectl get rbg pd-inference -n inference -o jsonpath='{range .spec.roles[*]}{.name}{"="}{.replicas}{"\n"}{end}' +``` + +### 推荐的调参流程 +1. **先开启 dryRun**：设置 `dryRun: true`，Planner 仅观测和计算，不执行伸缩。通过日志观察预测值和计算结果是否合理。 +2. **调整 SLA 目标**：根据 dryRun 期间的观测数据，设置合理的 TTFT/ITL 目标。目标过低会导致过度扩容，过高则用户体验下降。 +3. **关闭 dryRun**：确认配置合理后，设置 `dryRun: false` 启用实际伸缩。 +4. **观察修正因子**：Planner 会实时计算修正因子（observed / expected），如果持续偏高说明实际性能低于画像预期，可能需要重新 Profiling。 + +--- + +## 方案选型建议 +### 按部署架构选择 +| 部署架构 | 推荐方案 | 理由 | +| --- | --- | --- | +| **聚合部署**（非 PD 分离） | HPA 或 KEDA | 推理引擎作为单一角色，伸缩决策简单，基于资源利用率或自定义指标即可满足需求 | +| **PD 分离** | RBG Planner | HPA/KEDA 独立伸缩各角色，无法理解 Prefill 和 Decode 之间的资源配比关系；RBG Planner 将两者作为整体进行 SLA 驱动的伸缩决策 | + + +### 各方案能力对比 +| 维度 | HPA | KEDA | RBG Planner | +| --- | --- | --- | --- | +| 决策类型 | 响应式（指标阈值） | 响应式（外部事件） | 预测式（SLA + 负载预测） | +| 指标来源 | CPU/内存/自定义指标 | 外部指标源（Prometheus、Kafka 等） | 推理引擎 Prometheus 指标 | +| SLA 感知 | 否 | 否 | 是（TTFT/ITL） | +| 负载预测 | 否 | 否 | 是（ARIMA/Prophet） | +| 性能画像 | 否 | 否 | 是（离线 Profiling） | +| PD 协调能力 | 有限（CoordinatedPolicy 仅同步进度） | 有限（CoordinatedPolicy 仅同步进度） | 内置（整体计算 Prefill/Decode 最优配比） | +| GPU 预算控制 | 否 | 否 | 是 | +| 执行通道 | ScalingAdapter | ScalingAdapter | ScalingAdapter | +| 适用部署架构 | 聚合部署 | 聚合部署 | PD 分离 | + + +**关键差异**：HPA/KEDA 为每个角色独立创建伸缩器，即使配合 CoordinatedPolicy，也只能控制伸缩进度的同步，无法根据推理工作负载的特性（如 Prefill 计算密集 vs Decode 内存密集）做出智能的资源配比决策。RBG Planner 将 Prefill 和 Decode 作为一个整体，根据请求特征（输入长度、输出长度）和性能画像，分别计算两个角色的最优副本数。 + +--- + +## 验证伸缩状态 +```bash +# 查看 HPA 状态 +kubectl get hpa + +# 查看 KEDA ScaledObject 状态 +kubectl get scaledobject + +# 查看 AutoScaler 状态 +kubectl get autoscaler -n inference + +# 查看 RBGSA 状态 +kubectl get rbgsa + +# 查看 RBG 各角色副本数 +kubectl get rbg -o wide +``` + +## 相关文档 ++ [使用 RBG 部署推理服务](#) ++ [配置滚动更新策略](#) ++ [原地升级与原地调度](#) ++ [RBG Planner 项目](https://github.com/sgl-project/rbg-planner) + From 3f5c29af69c01586bd1d432d0f6972d1917e4ad7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=85=A7=E5=BE=AE?= Date: Fri, 10 Jul 2026 15:33:19 +0800 Subject: [PATCH 2/2] fix markdown lint --- .../en/08-configuring-autoscaling-guide.md | 15 +++-- .../en/08-configuring-autoscaling.md | 55 +++++++++++++----- .../zh/08-configuring-autoscaling-guide.md | 15 +++-- .../zh/08-configuring-autoscaling.md | 56 +++++++++++++------ 4 files changed, 102 insertions(+), 39 deletions(-) diff --git a/doc/best-practice/en/08-configuring-autoscaling-guide.md b/doc/best-practice/en/08-configuring-autoscaling-guide.md index e572e1ca..39e8a6c8 100644 --- a/doc/best-practice/en/08-configuring-autoscaling-guide.md +++ b/doc/best-practice/en/08-configuring-autoscaling-guide.md @@ -5,6 +5,7 @@ ## Objectives Validate RBG's autoscaling mechanism, including: + 1. Enabling ScalingAdapter and manual scaling via Scale subresource 2. Configuring HPA for metric-driven scaling 3. Configuring KEDA for event-driven scaling @@ -105,6 +106,7 @@ kubectl get rbgsa scaling-demo-backend -o yaml ``` **Expected output:** + - RBGSA `scaling-demo-backend` auto-created (naming convention: `-`) - RBGSA's `status.replicas` is 2 - RBGSA's `status.selector` contains Pod label selector @@ -153,6 +155,7 @@ kubectl get pods -l rbg.workloads.x-k8s.io/group-name=scaling-demo ``` **Expected output:** + - After scaling up, Pod count increases from 2 to 4 - RBG's `spec.roles[0].replicas` syncs to 4 - After scaling down, Pod count decreases to 1 @@ -167,7 +170,7 @@ kubectl delete rbg scaling-demo ## Operation 2: HPA Metric-Driven Scaling -### Step 1: Create an RBG with ScalingAdapter Enabled +### Step 1: Create an RBG with ScalingAdapter Enabled (HPA) ```bash cat <<'EOF' | kubectl apply -f - @@ -309,6 +312,7 @@ kubectl get rbgsa hpa-demo-backend ``` **Expected output:** + - HPA's `TARGETS` column shows current CPU utilization and target value (e.g., `0%/50%`) - HPA's `scaleTargetRef.kind` is `RoleBasedGroupScalingAdapter` - HPA's `scaleTargetRef.name` is `hpa-demo-backend` @@ -355,7 +359,7 @@ kubectl get pods -l rbg.workloads.x-k8s.io/group-name=hpa-demo -w > **Note**: `sleep 3600` itself does not consume CPU. To trigger HPA scale-up, replace the command with a CPU-intensive command (e.g., `python3 -c "while True: pass"`), or use custom metrics. This section primarily validates HPA configuration correctness. -### Cleanup +### Cleanup (HPA) ```bash kubectl delete hpa hpa-demo-backend @@ -547,6 +551,7 @@ kubectl wait --for=condition=ready pod -l rbg.workloads.x-k8s.io/group-name=keda ``` > **Note**: A Python HTTP server runs inside the Pod, simulating SGLang's Prometheus metrics endpoint: +> > - `GET /metrics`: Returns `sglang_num_queue_requests{rbg="keda-demo",role="backend"} 0` > - `GET /set?value=200`: Sets the metric value to 200 (for triggering scale-up) > - `GET /set?value=0`: Sets the metric value to 0 (for triggering scale-down) @@ -605,6 +610,7 @@ kubectl get hpa -l scaledobject.keda.sh/name=keda-demo-backend ``` **Expected output:** + - Prometheus query returns `sglang_num_queue_requests` value as `0` - ScaledObject `READY=True`, `ACTIVE=False` (metric value 0 < threshold 100) - KEDA auto-creates HPA (named `keda-hpa-keda-demo-backend`) pointing to RBGSA, `TARGETS` shows `0/100 (avg)` @@ -661,7 +667,7 @@ kubectl get hpa -l scaledobject.keda.sh/name=keda-demo-backend > **Note**: After the metric drops to 0, HPA detects the metric below threshold within 1-3 minutes and completes the scale-down. KEDA does not set a custom scaleDown behavior by default — the scale-down speed depends on the HPA controller's default downscale stabilization window (usually within 5 minutes). -### Cleanup +### Cleanup (KEDA) ```bash kubectl delete scaledobject keda-demo-backend @@ -834,6 +840,7 @@ kubectl get rbgsa -n inference ``` **Expected output:** + - AutoScaler status is `Ready` or `Profiling` (first run requires Profiling) - Planner Pod running normally - Logs show observed TTFT, ITL metrics and predicted replica counts @@ -856,7 +863,7 @@ kubectl get pods -n inference -l rbg.workloads.x-k8s.io/group-name=pd-autoscaler > **Note**: After disabling dryRun, the Planner automatically adjusts Prefill and Decode replica counts based on SLA targets and load predictions. Observe for a while — if the scaling behavior meets expectations, the configuration is complete. For tuning, refer to the "Recommended Tuning Process" in the concept document. -### Cleanup +### Cleanup (RBG Planner) ```bash kubectl delete autoscaler pd-autoscaler-demo -n inference diff --git a/doc/best-practice/en/08-configuring-autoscaling.md b/doc/best-practice/en/08-configuring-autoscaling.md index ec400ac4..0436672b 100644 --- a/doc/best-practice/en/08-configuring-autoscaling.md +++ b/doc/best-practice/en/08-configuring-autoscaling.md @@ -1,5 +1,7 @@ # Configuring Autoscaling for RBG Services + ## Overview + RBG inference service autoscaling is implemented through `scalingAdapter` — it exposes a standard Kubernetes Scale subresource for each role, allowing any scaling strategy to deliver replica count decisions to RBG. `scalingAdapter` itself does not provide a scaling strategy; it is simply an execution channel. Different scaling strategies drive this channel through their respective decision logic, mainly falling into two categories: @@ -29,6 +31,7 @@ Scaling Strategy (Decision Layer) ``` ## Prerequisites + + Kubernetes cluster version >= 1.24 + RBG Controller installed (see [Installation Guide](https://github.com/sgl-project/rbg)) + Metrics Server installed (HPA resource metrics dependency) @@ -37,7 +40,9 @@ Scaling Strategy (Decision Layer) --- ## ScalingAdapter: Unified Scaling Execution Channel + ### How It Works + Each role in RBG can enable autoscaling via `scalingAdapter.enable: true`. Once enabled, RBG Controller automatically creates a `RoleBasedGroupScalingAdapter` (RBGSA for short) resource that implements the standard Kubernetes `/scale` subresource interface. Whether HPA, KEDA, or RBG Planner, they all deliver replica count decisions to RBG roles through this Scale subresource. ```plain @@ -53,6 +58,7 @@ RoleBasedGroup → Role Pod scaling ``` ### Enabling ScalingAdapter + Set `scalingAdapter.enable: true` in the role's spec: ```yaml @@ -84,14 +90,15 @@ spec: memory: "1Gi" ``` -#### Parameter Description +#### Parameter Description (ScalingAdapter) + | Parameter | Type | Required | Default | Description | | --- | --- | --- | --- | --- | | `scalingAdapter.enable` | bool | No | `false` | Whether to enable autoscaling | | `scalingAdapter.labels` | map[string]string | No | - | Custom labels attached to the RBGSA resource | - ### Auto-Created RBGSA Naming Convention + When `scalingAdapter.enable: true`, the Controller auto-created RBGSA resource follows this naming convention: ```plain @@ -106,9 +113,11 @@ For example, if the RBG name is `inference-cluster` and the role name is `prefil --- ## Scenario 1: Metric-Driven Scaling — HPA + HPA (Horizontal Pod Autoscaler) is Kubernetes' built-in autoscaler that supports reactive scaling based on CPU and memory utilization. This is the simplest scaling approach, suitable for scenarios with clear resource utilization thresholds. -### Configuration Example +### Configuration Example (HPA) + ```yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler @@ -131,6 +140,7 @@ spec: ``` #### HPA Key Fields + | Field | Description | | --- | --- | | `scaleTargetRef.apiVersion` | Fixed as `workloads.x-k8s.io/v1alpha2` | @@ -140,11 +150,11 @@ spec: | `maxReplicas` | Maximum replica count | | `metrics` | List of scaling metrics | - > **Note**: HPA obtains the Pod label selector via RBGSA's `status.selector` to collect metric data — no additional labelSelector configuration needed. > ### HPA Configuration Recommendations for GPU Inference Services + For GPU inference services, CPU utilization typically does not accurately reflect load. It is recommended to follow community practices (such as KServe, vLLM production deployment solutions) and use custom metrics for scaling: ```yaml @@ -178,16 +188,17 @@ Common custom metrics for inference services (need to be exposed to HPA via Prom | `inference_tokens_per_second` | Tokens processed per second | Throughput-driven scaling | | `inference_kv_cache_usage` | KV Cache usage rate | VRAM-driven scaling | - > **Note**: Custom metrics require the inference engine to expose Prometheus-format metrics at the `/metrics` endpoint, and use [Prometheus Adapter](https://github.com/kubernetes-sigs/prometheus-adapter) to convert them to Custom Metrics API consumable by HPA. The specific metric names and configuration methods vary by inference engine — please refer to the corresponding engine's documentation. > --- ## Scenario 2: Event-Driven Scaling — KEDA + KEDA (Kubernetes Event-Driven Autoscaling) supports richer external metric sources, suitable for scaling based on message queue depth, external API latency, and other metrics. -### Configuration Example +### Configuration Example (KEDA) + ```yaml apiVersion: keda.sh/v1alpha1 kind: ScaledObject @@ -213,6 +224,7 @@ spec: ``` #### KEDA Key Fields + | Field | Description | | --- | --- | | `scaleTargetRef` | Same as HPA, points to RBGSA | @@ -220,17 +232,19 @@ spec: | `cooldownPeriod` | Scale-down cooldown period (seconds), prevents frequent scale-downs | | `triggers` | Trigger list, supports prometheus, kafka, redis, and many other types | - > **Note**: KEDA's `cooldownPeriod` is particularly important for inference services. Inference engines have long startup times (model loading, KV Cache initialization), and overly frequent scale-downs cause service jitter. It is recommended to set `cooldownPeriod` to 5-10 minutes. > --- ## Scenario 3: SLA-Driven Scaling — RBG Planner + ### Aggregated Deployment: HPA/KEDA Is Sufficient + For aggregated (non-PD-disaggregated) inference services, HPA or KEDA can meet most scaling needs. The inference engine serves as a single role, and scaling decisions are relatively simple — just focus on a single role's resource utilization or custom metrics. ### PD Disaggregation: HPA/KEDA Falls Short + In PD-disaggregated architecture, Prefill and Decode are two independent roles, each with its own HPA. Although CoordinatedPolicy can coordinate scaling progress, the fundamental problem with this approach is: **HPA/KEDA does not understand the characteristics of inference workloads**. 1. **Asymmetric resource requirements for Prefill and Decode**: Prefill is compute-intensive (processing long prompts), Decode is memory-intensive (generating tokens one by one). Under the same request pattern, their resource consumption is completely different — simple progress coordination cannot solve the resource ratio problem @@ -241,6 +255,7 @@ In PD-disaggregated architecture, Prefill and Decode are two independent roles, These problems cannot be solved by CoordinatedPolicy — it only controls scaling progress synchronization, not scaling decision intelligence. PD-disaggregated inference needs a scaling strategy that **understands inference workloads and can simultaneously consider the relationship between Prefill and Decode**. ### RBG Planner: Intelligent Scaling Designed for PD-Disaggregated Inference + [RBG Planner](https://github.com/sgl-project/rbg-planner) is an independent Kubernetes Operator that provides **SLA-driven predictive scaling** specifically for PD-disaggregated inference. Its core algorithm originates from the [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) project, adapted to native Kubernetes RBG API. Like HPA/KEDA, RBG Planner delivers replica count decisions to RBG through the ScalingAdapter's Scale subresource. Unlike HPA/KEDA which independently scale each role, RBG Planner makes scaling decisions for Prefill and Decode as a whole — based on the characteristics of the request load (input length, output length), it separately calculates the optimal replica count for both roles. @@ -274,6 +289,7 @@ RBG Planner's working loop: ``` ### Key Capabilities + | Capability | Description | | --- | --- | | SLA target driven | Uses TTFT and ITL latency targets (milliseconds) as scaling constraints | @@ -283,14 +299,15 @@ RBG Planner's working loop: | GPU budget control | Sets total GPU limit, allocates resources proportionally within budget | | Correction factor | Real-time comparison of observed vs. expected values, auto-corrects scaling decisions | - ### Installing RBG Planner + ```bash helm install rbg-planner oci://ghcr.io/sgl-project/charts/rbg-planner \ -n rbg-system --create-namespace ``` ### Prerequisite: PD-Disaggregated Inference Service + RBG Planner requires the inference service to be deployed in PD-disaggregated architecture, with both roles having ScalingAdapter enabled: ```yaml @@ -339,6 +356,7 @@ spec: ``` ### Creating AutoScaler CR + ```yaml apiVersion: inference-extension.rolebasedgroup.io/v1alpha1 kind: AutoScaler @@ -385,7 +403,8 @@ spec: port: 9091 # Metrics port ``` -#### Parameter Description +#### Parameter Description (AutoScaler) + | Parameter | Type | Description | | --- | --- | --- | | `spec.scalingInterval` | int | Scaling decision interval (seconds), default 180 | @@ -403,8 +422,8 @@ spec: | `spec.implementation.DynamoPlanner.metricsEndpoint.metricSource` | string | Inference engine type | | `spec.implementation.DynamoPlanner.metricsEndpoint.port` | int | Inference engine metrics port | - ### Profiling Process + After creating the AutoScaler, the Operator automatically executes the following process: ```plain @@ -427,6 +446,7 @@ The performance profile generated by the Profiler includes: > ### Verifying Planner Running Status + ```bash # Check AutoScaler status kubectl get autoscaler -n inference @@ -442,6 +462,7 @@ kubectl get rbg pd-inference -n inference -o jsonpath='{range .spec.roles[*]}{.n ``` ### Recommended Tuning Process + 1. **Start with dryRun**: Set `dryRun: true`, Planner only observes and computes without executing scaling. Observe through logs whether predictions and computed results are reasonable. 2. **Adjust SLA targets**: Based on observations during dryRun, set reasonable TTFT/ITL targets. Targets that are too low cause over-scaling, too high degrades user experience. 3. **Disable dryRun**: After confirming the configuration is reasonable, set `dryRun: false` to enable actual scaling. @@ -450,14 +471,16 @@ kubectl get rbg pd-inference -n inference -o jsonpath='{range .spec.roles[*]}{.n --- ## Solution Selection Recommendations + ### By Deployment Architecture + | Deployment Architecture | Recommended Solution | Rationale | | --- | --- | --- | | **Aggregated** (non-PD-disaggregated) | HPA or KEDA | Inference engine as a single role, simple scaling decisions, resource utilization or custom metrics meet the needs | | **PD-disaggregated** | RBG Planner | HPA/KEDA scales each role independently, cannot understand the resource ratio relationship between Prefill and Decode; RBG Planner makes SLA-driven scaling decisions for both as a whole | - ### Capability Comparison + | Dimension | HPA | KEDA | RBG Planner | | --- | --- | --- | --- | | Decision type | Reactive (metric threshold) | Reactive (external events) | Predictive (SLA + load prediction) | @@ -470,12 +493,12 @@ kubectl get rbg pd-inference -n inference -o jsonpath='{range .spec.roles[*]}{.n | Execution channel | ScalingAdapter | ScalingAdapter | ScalingAdapter | | Applicable architecture | Aggregated | Aggregated | PD-disaggregated | - **Key difference**: HPA/KEDA creates independent scalers for each role. Even with CoordinatedPolicy, it can only control scaling progress synchronization — it cannot make intelligent resource ratio decisions based on inference workload characteristics (e.g., Prefill compute-intensive vs Decode memory-intensive). RBG Planner treats Prefill and Decode as a whole, calculating optimal replica counts for both roles based on request characteristics (input length, output length) and performance profiles. --- ## Verify Scaling Status + ```bash # Check HPA status kubectl get hpa @@ -494,7 +517,9 @@ kubectl get rbg -o wide ``` ## Related Documents -+ [Deploying Inference Services with RBG](#) -+ [Configuring Rolling Update Strategies](#) -+ [In-Place Update and In-Place Scheduling](#) + ++ [Deploying Inference Services with RBG](01-deploy-inference-service.md) ++ [Using RoleTemplates to Reduce Configuration Duplication](02-using-role-templates.md) ++ [Configuring Rolling Update Strategies](03-configuring-rolling-updates.md) ++ In-Place Update and In-Place Scheduling + [RBG Planner Project](https://github.com/sgl-project/rbg-planner) diff --git a/doc/best-practice/zh/08-configuring-autoscaling-guide.md b/doc/best-practice/zh/08-configuring-autoscaling-guide.md index 2441f5d6..143a9c81 100644 --- a/doc/best-practice/zh/08-configuring-autoscaling-guide.md +++ b/doc/best-practice/zh/08-configuring-autoscaling-guide.md @@ -5,6 +5,7 @@ ## 目标 验证 RBG 的弹性伸缩机制，包括： + 1. 启用 ScalingAdapter 并通过 Scale 子资源手动伸缩 2. 配置 HPA 实现指标驱动伸缩 3. 配置 KEDA 实现事件驱动伸缩 @@ -105,6 +106,7 @@ kubectl get rbgsa scaling-demo-backend -o yaml ``` **预期输出：** + - RBGSA `scaling-demo-backend` 自动创建（命名规则：`-`） - RBGSA 的 `status.replicas` 为 2 - RBGSA 的 `status.selector` 包含 Pod 标签选择器 @@ -153,6 +155,7 @@ kubectl get pods -l rbg.workloads.x-k8s.io/group-name=scaling-demo ``` **预期输出：** + - 扩容后 Pod 数量从 2 增加到 4 - RBG 的 `spec.roles[0].replicas` 同步更新为 4 - 缩容后 Pod 数量减少到 1 @@ -167,7 +170,7 @@ kubectl delete rbg scaling-demo ## 操作二：HPA 指标驱动伸缩 -### 步骤 1：创建启用 ScalingAdapter 的 RBG +### 步骤 1：创建启用 ScalingAdapter 的 RBG（HPA） ```bash cat <<'EOF' | kubectl apply -f - @@ -309,6 +312,7 @@ kubectl get rbgsa hpa-demo-backend ``` **预期输出：** + - HPA 的 `TARGETS` 列显示当前 CPU 利用率和目标值（如 `0%/50%`） - HPA 的 `scaleTargetRef.kind` 为 `RoleBasedGroupScalingAdapter` - HPA 的 `scaleTargetRef.name` 为 `hpa-demo-backend` @@ -355,7 +359,7 @@ kubectl get pods -l rbg.workloads.x-k8s.io/group-name=hpa-demo -w > **说明**：`sleep 3600` 本身不消耗 CPU。如需触发 HPA 扩容，可替换 command 为 CPU 密集型命令（如 `python3 -c "while True: pass"`），或使用自定义指标。此处主要验证 HPA 配置正确性。 -### 清理 +### 清理（HPA） ```bash kubectl delete hpa hpa-demo-backend @@ -547,6 +551,7 @@ kubectl wait --for=condition=ready pod -l rbg.workloads.x-k8s.io/group-name=keda ``` > **说明**：Pod 内运行 Python HTTP 服务器，模拟 SGLang 的 Prometheus 指标端点： +> > - `GET /metrics`：返回 `sglang_num_queue_requests{rbg="keda-demo",role="backend"} 0` > - `GET /set?value=200`：将指标值设为 200（用于触发扩容） > - `GET /set?value=0`：将指标值设为 0（用于触发缩容） @@ -605,6 +610,7 @@ kubectl get hpa -l scaledobject.keda.sh/name=keda-demo-backend ``` **预期输出：** + - Prometheus 查询返回 `sglang_num_queue_requests` 值为 `0` - ScaledObject `READY=True`，`ACTIVE=False`（指标值 0 < 阈值 100） - KEDA 自动创建 HPA（名为 `keda-hpa-keda-demo-backend`）指向 RBGSA，`TARGETS` 显示 `0/100 (avg)` @@ -661,7 +667,7 @@ kubectl get hpa -l scaledobject.keda.sh/name=keda-demo-backend > **说明**：指标降为 0 后，HPA 在 1-3 分钟内检测到指标低于阈值并完成缩容。KEDA 默认不设置自定义 scaleDown behavior，缩容速度取决于 HPA 控制器的默认 downscale stabilization 窗口（通常 5 分钟内）。 -### 清理 +### 清理（KEDA） ```bash kubectl delete scaledobject keda-demo-backend @@ -834,6 +840,7 @@ kubectl get rbgsa -n inference ``` **预期输出：** + - AutoScaler 状态为 `Ready` 或 `Profiling`（首次运行需进行 Profiling） - Planner Pod 正常运行 - 日志中显示观测到的 TTFT、ITL 指标和预测的副本数 @@ -856,7 +863,7 @@ kubectl get pods -n inference -l rbg.workloads.x-k8s.io/group-name=pd-autoscaler > **说明**：关闭 dryRun 后，Planner 会根据 SLA 目标和负载预测自动调整 Prefill 和 Decode 的副本数。观察一段时间后，如果伸缩行为符合预期，则配置完成。如需调参，可参考概念文档中的「推荐的调参流程」。 -### 清理 +### 清理（RBG Planner） ```bash kubectl delete autoscaler pd-autoscaler-demo -n inference diff --git a/doc/best-practice/zh/08-configuring-autoscaling.md b/doc/best-practice/zh/08-configuring-autoscaling.md index 4fc03039..6f165e66 100644 --- a/doc/best-practice/zh/08-configuring-autoscaling.md +++ b/doc/best-practice/zh/08-configuring-autoscaling.md @@ -1,5 +1,7 @@ # 为 RBG 服务配置弹性伸缩策略 + ## 概述 + RBG 推理服务的弹性伸缩通过 `scalingAdapter` 实现——它为每个角色暴露一个标准的 Kubernetes Scale 子资源，使得任意伸缩策略都可以将副本数决策下发到 RBG。`scalingAdapter` 本身不提供伸缩策略，它只是一个执行通道。 不同的伸缩策略通过各自的决策逻辑驱动这个通道，主要分为两类： @@ -29,6 +31,7 @@ RBG 推理服务的弹性伸缩通过 `scalingAdapter` 实现——它为每个 ``` ## 前提条件 + + Kubernetes 集群版本 >= 1.24 + 已安装 RBG Controller（参考 [安装指南](https://github.com/sgl-project/rbg)） + Metrics Server 已安装（HPA 资源指标依赖） @@ -37,7 +40,9 @@ RBG 推理服务的弹性伸缩通过 `scalingAdapter` 实现——它为每个 --- ## ScalingAdapter：统一的伸缩执行通道 + ### 工作原理 + RBG 的每个角色可以通过 `scalingAdapter.enable: true` 启用弹性伸缩。启用后，RBG Controller 自动创建一个 `RoleBasedGroupScalingAdapter`（简称 RBGSA）资源，该资源实现了 Kubernetes 标准的 `/scale` 子资源接口。无论是 HPA、KEDA 还是 RBG Planner，都通过这个 Scale 子资源将副本数决策下发到 RBG 角色。 ```plain @@ -53,6 +58,7 @@ RoleBasedGroup → 角色 Pod 伸缩 ``` ### 启用 ScalingAdapter + 在角色的 spec 中设置 `scalingAdapter.enable: true`： ```yaml @@ -84,14 +90,15 @@ spec: memory: "1Gi" ``` -#### 参数说明 +#### 参数说明（ScalingAdapter） + | 参数 | 类型 | 是否必填 | 默认值 | 说明 | | --- | --- | --- | --- | --- | | `scalingAdapter.enable` | bool | 否 | `false` | 是否启用弹性伸缩 | | `scalingAdapter.labels` | map[string]string | 否 | - | 附加到 RBGSA 资源的自定义标签 | - ### 自动创建的 RBGSA 命名规则 + 当 `scalingAdapter.enable: true` 时，Controller 自动创建的 RBGSA 资源遵循以下命名规则： ```plain @@ -106,9 +113,11 @@ spec: --- ## 场景一：指标驱动伸缩 — HPA + HPA（Horizontal Pod Autoscaler）是 Kubernetes 内置的伸缩器，支持基于 CPU 和内存利用率进行响应式伸缩。这是最简单的伸缩方式，适合对资源利用率有明确阈值的场景。 -### 配置示例 +### 配置示例（HPA） + ```yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler @@ -131,6 +140,7 @@ spec: ``` #### HPA 关键字段 + | 字段 | 说明 | | --- | --- | | `scaleTargetRef.apiVersion` | 固定为 `workloads.x-k8s.io/v1alpha2` | @@ -140,11 +150,11 @@ spec: | `maxReplicas` | 最大副本数 | | `metrics` | 伸缩指标列表 | - > **说明**：HPA 通过 RBGSA 的 `status.selector` 获取 Pod 标签选择器来采集指标数据，无需额外配置 labelSelector。 > ### GPU 推理服务的 HPA 配置建议 + 对于 GPU 推理服务，CPU 利用率通常不能准确反映负载情况。建议参考社区实践（如 KServe、vLLM 生产部署方案），使用自定义指标进行伸缩： ```yaml @@ -178,16 +188,17 @@ spec: | `inference_tokens_per_second` | 每秒处理的 token 数 | 吞吐量驱动的伸缩 | | `inference_kv_cache_usage` | KV Cache 使用率 | 显存驱动的伸缩 | - > **说明**：自定义指标需要推理引擎在 `/metrics` 端点暴露 Prometheus 格式的指标，并通过 [Prometheus Adapter](https://github.com/kubernetes-sigs/prometheus-adapter) 转换为 HPA 可消费的 Custom Metrics API。具体的指标名称和配置方式因推理引擎而异，请参考对应引擎的文档。 > --- ## 场景二：事件驱动伸缩 — KEDA + KEDA（Kubernetes Event-Driven Autoscaling）支持更丰富的外部指标源，适合基于消息队列深度、外部 API 延迟等指标进行伸缩。 -### 配置示例 +### 配置示例（KEDA） + ```yaml apiVersion: keda.sh/v1alpha1 kind: ScaledObject @@ -213,6 +224,7 @@ spec: ``` #### KEDA 关键字段 + | 字段 | 说明 | | --- | --- | | `scaleTargetRef` | 与 HPA 相同，指向 RBGSA | @@ -220,17 +232,19 @@ spec: | `cooldownPeriod` | 缩容冷却时间（秒），防止频繁缩容 | | `triggers` | 触发器列表，支持 prometheus、kafka、redis 等多种类型 | - > **说明**：KEDA 的 `cooldownPeriod` 对于推理服务尤为重要。推理引擎的启动时间较长（模型加载、KV Cache 初始化），过于频繁的缩容会导致服务抖动。建议将 `cooldownPeriod` 设置为 5-10 分钟。 > --- ## 场景三：SLA 驱动伸缩 — RBG Planner + ### 聚合部署：HPA/KEDA 足够 + 对于聚合部署（非 PD 分离）的推理服务，HPA 或 KEDA 可以满足大多数伸缩需求。推理引擎作为一个整体角色，伸缩决策相对简单——只需关注单一角色的资源利用率或自定义指标即可。 ### PD 分离：HPA/KEDA 力不从心 + 在 PD 分离架构中，Prefill 和 Decode 是两个独立角色，各自有独立的 HPA。虽然可以通过 CoordinatedPolicy 协调伸缩进度，但这种方式的根本问题在于：**HPA/KEDA 不理解推理工作负载的特性**。 1. **Prefill 和 Decode 的资源需求不对称**：Prefill 是计算密集型（处理长 prompt），Decode 是内存密集型（逐 token 生成）。相同的请求模式下，两者的资源消耗完全不同，简单的进度协调无法解决资源配比问题 @@ -241,6 +255,7 @@ spec: 这些问题不是 CoordinatedPolicy 能解决的——它只控制伸缩进度的同步，不提供伸缩决策的智能。PD 分离推理需要一个**理解推理工作负载、能同时考虑 Prefill 和 Decode 关系**的伸缩策略。 ### RBG Planner：为 PD 分离推理设计的智能伸缩 + [RBG Planner](https://github.com/sgl-project/rbg-planner) 是一个独立的 Kubernetes Operator，专门为 PD 分离推理提供 **SLA 驱动的预测式伸缩**。它的核心算法源自 [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) 项目，已适配为原生 Kubernetes RBG API。与 HPA/KEDA 一样，RBG Planner 通过 ScalingAdapter 的 Scale 子资源将副本数决策下发到 RBG。 与 HPA/KEDA 独立伸缩各角色不同，RBG Planner 将 Prefill 和 Decode 作为一个整体进行伸缩决策——根据请求负载的特征（输入长度、输出长度），分别计算两个角色所需的最优副本数。 @@ -271,6 +286,7 @@ RBG Planner 的工作循环： ``` ### 关键能力 + | 能力 | 说明 | | --- | --- | | SLA 目标驱动 | 以 TTFT 和 ITL 延迟目标（毫秒）作为伸缩约束 | @@ -280,14 +296,15 @@ RBG Planner 的工作循环： | GPU 预算控制 | 设定总 GPU 上限，在预算内按比例分配资源 | | 修正因子 | 实时对比观测值与预期值的偏差，自动修正伸缩决策 | - ### 安装 RBG Planner + ```bash helm install rbg-planner oci://ghcr.io/sgl-project/charts/rbg-planner \ -n rbg-system --create-namespace ``` ### 前置条件：PD 分离推理服务 + RBG Planner 要求推理服务已部署为 PD 分离架构，且两个角色都启用了 ScalingAdapter： ```yaml @@ -336,6 +353,7 @@ spec: ``` ### 创建 AutoScaler CR + ```yaml apiVersion: inference-extension.rolebasedgroup.io/v1alpha1 kind: AutoScaler @@ -382,7 +400,8 @@ spec: port: 9091 # 指标端口 ``` -#### 参数说明 +#### 参数说明（AutoScaler） + | 参数 | 类型 | 说明 | | --- | --- | --- | | `spec.scalingInterval` | int | 伸缩决策间隔（秒），默认 180 | @@ -400,8 +419,8 @@ spec: | `spec.implementation.DynamoPlanner.metricsEndpoint.metricSource` | string | 推理引擎类型 | | `spec.implementation.DynamoPlanner.metricsEndpoint.port` | int | 推理引擎指标端口 | - ### Profiling 流程 + 创建 AutoScaler 后，Operator 会自动执行以下流程： ```plain @@ -424,6 +443,7 @@ Profiler 生成的性能画像包含： > ### 验证 Planner 运行状态 + ```bash # 查看 AutoScaler 状态 kubectl get autoscaler -n inference @@ -439,6 +459,7 @@ kubectl get rbg pd-inference -n inference -o jsonpath='{range .spec.roles[*]}{.n ``` ### 推荐的调参流程 + 1. **先开启 dryRun**：设置 `dryRun: true`，Planner 仅观测和计算，不执行伸缩。通过日志观察预测值和计算结果是否合理。 2. **调整 SLA 目标**：根据 dryRun 期间的观测数据，设置合理的 TTFT/ITL 目标。目标过低会导致过度扩容，过高则用户体验下降。 3. **关闭 dryRun**：确认配置合理后，设置 `dryRun: false` 启用实际伸缩。 @@ -447,14 +468,16 @@ kubectl get rbg pd-inference -n inference -o jsonpath='{range .spec.roles[*]}{.n --- ## 方案选型建议 + ### 按部署架构选择 + | 部署架构 | 推荐方案 | 理由 | | --- | --- | --- | | **聚合部署**（非 PD 分离） | HPA 或 KEDA | 推理引擎作为单一角色，伸缩决策简单，基于资源利用率或自定义指标即可满足需求 | | **PD 分离** | RBG Planner | HPA/KEDA 独立伸缩各角色，无法理解 Prefill 和 Decode 之间的资源配比关系；RBG Planner 将两者作为整体进行 SLA 驱动的伸缩决策 | - ### 各方案能力对比 + | 维度 | HPA | KEDA | RBG Planner | | --- | --- | --- | --- | | 决策类型 | 响应式（指标阈值） | 响应式（外部事件） | 预测式（SLA + 负载预测） | @@ -467,12 +490,12 @@ kubectl get rbg pd-inference -n inference -o jsonpath='{range .spec.roles[*]}{.n | 执行通道 | ScalingAdapter | ScalingAdapter | ScalingAdapter | | 适用部署架构 | 聚合部署 | 聚合部署 | PD 分离 | - **关键差异**：HPA/KEDA 为每个角色独立创建伸缩器，即使配合 CoordinatedPolicy，也只能控制伸缩进度的同步，无法根据推理工作负载的特性（如 Prefill 计算密集 vs Decode 内存密集）做出智能的资源配比决策。RBG Planner 将 Prefill 和 Decode 作为一个整体，根据请求特征（输入长度、输出长度）和性能画像，分别计算两个角色的最优副本数。 --- ## 验证伸缩状态 + ```bash # 查看 HPA 状态 kubectl get hpa @@ -491,8 +514,9 @@ kubectl get rbg -o wide ``` ## 相关文档 -+ [使用 RBG 部署推理服务](#) -+ [配置滚动更新策略](#) -+ [原地升级与原地调度](#) -+ [RBG Planner 项目](https://github.com/sgl-project/rbg-planner) ++ [使用 RBG 部署推理服务](01-deploy-inference-service.md) ++ [使用 RoleTemplates 减少配置重复](02-using-role-templates.md) ++ [配置滚动更新策略](03-configuring-rolling-updates.md) ++ 原地升级与原地调度 ++ [RBG Planner 项目](https://github.com/sgl-project/rbg-planner)