diff --git a/doc/best-practice/en/03-configuring-rolling-updates-guide.md b/doc/best-practice/en/03-configuring-rolling-updates-guide.md new file mode 100644 index 000000000..ea153f860 --- /dev/null +++ b/doc/best-practice/en/03-configuring-rolling-updates-guide.md @@ -0,0 +1,512 @@ +# Operations Guide: Configuring Rolling Update Strategies + +> Corresponding concept document: [3. Configuring Rolling Update Strategies](03-configuring-rolling-updates.md) + +## Objectives + +Validate RBG's `rolloutStrategy` rolling update strategy, including: + +1. Basic rolling update configuration (maxUnavailable / maxSurge) +2. Partition canary release +3. Paused pause and resume updates +4. CoordinatedPolicy multi-role coordinated upgrade + +## Prerequisites + +- Kubernetes cluster version >= 1.24 +- RBG Controller installed +- Images accessible: `alpine:3.23.5` + +> **Note**: This document uses `sleep 3600` as a placeholder command, focusing on validating RBG rolling update control plane behavior without requiring GPU. To test real inference functionality, replace with the full inference engine startup command. + +--- + +## Operation 1: Basic Rolling Update + +### Step 1: Create a 4-Replica RBG + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: rolling-update-demo +spec: + roles: + - name: backend + replicas: 4 + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + maxSurge: 0 + standalonePattern: + template: + spec: + containers: + - name: engine + image: alpine:3.23.5 + command: ["sleep", "3600"] +EOF +``` + +### Expected Behavior + +- 4 Pods created (`rolling-update-demo-backend-0` through `rolling-update-demo-backend-3`) +- After all are ready, RBG status is Ready + +### Step 2: Trigger Rolling Update + +```bash +# Modify command (extend the sleep duration) to trigger a rolling update (Pod rebuild) +kubectl patch rbg rolling-update-demo --type='json' \ + -p='[{"op": "replace", "path": "/spec/roles/0/standalonePattern/template/spec/containers/0/command", "value": ["sleep", "7200"]}]' +``` + +### Expected Behavior (Rolling Update Triggered) + +- Instances are updated one by one from high to low ordinal (3 → 2 → 1 → 0) +- `maxUnavailable: 1`: At most 1 instance unavailable at a time +- `maxSurge: 0`: No extra instances created, delete old then create new +- 3 instances always available during the update process +- `command` is a non-image field, triggering Pod rebuild: the old Pod terminates first, then is recreated with the same ordinal (AGE resets, RESTARTS is 0) + +### Verification + +```bash +# Observe the update process (continuously execute to see changes) +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=rolling-update-demo -w + +> NAME READY STATUS RESTARTS AGE +> rolling-update-demo-backend-0 1/1 Running 0 2m5s +> rolling-update-demo-backend-1 1/1 Running 0 2m5s +> rolling-update-demo-backend-2 1/1 Running 0 2m5s +> rolling-update-demo-backend-3 1/1 Terminating 0 2m5s +> rolling-update-demo-backend-3 0/1 Pending 0 0s +> rolling-update-demo-backend-3 0/1 ContainerCreating 0 0s +> rolling-update-demo-backend-3 1/1 Running 0 28s +> rolling-update-demo-backend-2 1/1 Terminating 0 2m59s +> rolling-update-demo-backend-2 0/1 Pending 0 0s +> rolling-update-demo-backend-2 0/1 ContainerCreating 0 0s +> rolling-update-demo-backend-2 1/1 Running 0 26s +> rolling-update-demo-backend-1 1/1 Terminating 0 3m56s +> rolling-update-demo-backend-1 0/1 Pending 0 0s +> rolling-update-demo-backend-1 0/1 ContainerCreating 0 0s +> rolling-update-demo-backend-1 1/1 Running 0 30s +> rolling-update-demo-backend-0 1/1 Terminating 0 4m58s +> rolling-update-demo-backend-0 0/1 Pending 0 0s +> rolling-update-demo-backend-0 0/1 ContainerCreating 0 0s +> rolling-update-demo-backend-0 1/1 Running 0 28s +``` + +You can observe Pods being "terminated → recreated" one by one in ordinal order 3 → 2 → 1 → 0, with at most 1 instance unavailable at any moment. + +```bash +# Confirm all Pods have been updated to the new command +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=rolling-update-demo -o jsonpath='{range .items[*]}{.metadata.name}{"="}{.spec.containers[0].command}{"\n"}{end}' + +> rolling-update-demo-backend-0=["sleep","7200"] +> rolling-update-demo-backend-1=["sleep","7200"] +> rolling-update-demo-backend-2=["sleep","7200"] +> rolling-update-demo-backend-3=["sleep","7200"] +``` + +**Expected output:** + +- 3 Pods always Ready during the update process +- After update completes, all 4 Pods have command `["sleep","7200"]` + +### Cleanup + +```bash +kubectl delete rbg rolling-update-demo +``` + +--- + +## Operation 2: Canary Release (Partition) + +### Step 1: Create a 4-Replica RBG with partition + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: canary-deployment +spec: + roles: + - name: backend + replicas: 4 + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + partition: 4 # Initially set to 4, no instances updated + standalonePattern: + template: + spec: + containers: + - name: engine + image: alpine:3.23.5 + command: ["sleep", "3600"] +EOF +``` + +### Step 2: Update command but Keep partition=4 + +```bash +# Modify command (extend the sleep duration) +kubectl patch rbg canary-deployment --type='json' \ + -p='[{"op": "replace", "path": "/spec/roles/0/standalonePattern/template/spec/containers/0/command", "value": ["sleep", "7200"]}]' +``` + +### Expected Behavior (Partition=4) + +- Since `partition: 4`, only instances with ordinal >= 4 will be updated +- Ordinals 0-3 are all less than 4, so **no instances are updated** +- All Pods still use the old command `["sleep","3600"]` + +### Verification (Partition=4) + +```bash +# Confirm all Pods still use the old command +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=canary-deployment -o jsonpath='{range .items[*]}{.metadata.name}{"="}{.spec.containers[0].command}{"\n"}{end}' + +> canary-deployment-backend-0=["sleep","3600"] +> canary-deployment-backend-1=["sleep","3600"] +> canary-deployment-backend-2=["sleep","3600"] +> canary-deployment-backend-3=["sleep","3600"] +``` + +**Expected output:** All 4 Pods have command `["sleep","3600"]` + +### Step 3: Lower partition to 3, Update the Last Instance + +```bash +kubectl patch rbg canary-deployment --type='json' \ + -p='[{"op": "replace", "path": "/spec/roles/0/rolloutStrategy/rollingUpdate/partition", "value": 3}]' +``` + +### Expected Behavior (Partition=3) + +- Instances with ordinal >= 3 (i.e., instance 3) are updated to the new command +- Ordinals 0-2 remain on the old configuration + +### Verification (Partition=3) + +```bash +# Check each Pod's command +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=canary-deployment -o jsonpath='{range .items[*]}{.metadata.name}{"="}{.spec.containers[0].command}{"\n"}{end}' + +> canary-deployment-backend-0=["sleep","3600"] +> canary-deployment-backend-1=["sleep","3600"] +> canary-deployment-backend-2=["sleep","3600"] +> canary-deployment-backend-3=["sleep","7200"] +``` + +**Expected output:** + +- `canary-deployment-backend-3` uses `["sleep","7200"]` +- `canary-deployment-backend-0` through `2` use `["sleep","3600"]` + +### Step 4: Lower partition to 0, Complete Full Update + +```bash +kubectl patch rbg canary-deployment --type='json' \ + -p='[{"op": "replace", "path": "/spec/roles/0/rolloutStrategy/rollingUpdate/partition", "value": 0}]' +``` + +### Expected Behavior (Partition=0) + +- All instances with ordinal >= 0 are updated to the new command + +### Verification (Partition=0) + +```bash +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=canary-deployment -o jsonpath='{range .items[*]}{.metadata.name}{"="}{.spec.containers[0].command}{"\n"}{end}' + +> canary-deployment-backend-0=["sleep","7200"] +> canary-deployment-backend-1=["sleep","7200"] +> canary-deployment-backend-2=["sleep","7200"] +> canary-deployment-backend-3=["sleep","7200"] +``` + +**Expected output:** All 4 Pods have command `["sleep","7200"]` + +### Cleanup (Canary Release) + +```bash +kubectl delete rbg canary-deployment +``` + +--- + +## Operation 3: Pause and Resume Updates + +### Step 1: Create an RBG and Trigger Update Then Pause + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: paused-update +spec: + roles: + - name: backend + replicas: 4 + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 4 + paused: true # Pause updates + standalonePattern: + template: + spec: + containers: + - name: engine + image: alpine:3.23.5 + command: ["sleep", "3600"] +EOF +``` + +### Step 2: Update command + +```bash +kubectl patch rbg paused-update --type='json' \ + -p='[{"op": "replace", "path": "/spec/roles/0/standalonePattern/template/spec/containers/0/command", "value": ["sleep", "7200"]}]' +``` + +### Expected Behavior (Paused) + +- Since `paused: true`, the update is paused +- All Pods remain on the old configuration + +### Verification (Paused) + +```bash +# Confirm all Pods still use the old command +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=paused-update -o jsonpath='{range .items[*]}{.metadata.name}{"="}{.spec.containers[0].command}{"\n"}{end}' + +> paused-update-backend-0=["sleep","3600"] +> paused-update-backend-1=["sleep","3600"] +> paused-update-backend-2=["sleep","3600"] +> paused-update-backend-3=["sleep","3600"] +``` + +**Expected output:** All 4 Pods have command `["sleep","3600"]` (update is paused) + +### Step 3: Resume Updates + +```bash +kubectl patch rbg paused-update --type='json' \ + -p='[{"op": "replace", "path": "/spec/roles/0/rolloutStrategy/rollingUpdate/paused", "value": false}]' +``` + +### Expected Behavior (Resumed) + +- Updates resume +- Pods are updated to the new command one by one + +### Verification (Resumed) + +```bash +# Wait for update to complete, confirm all Pods use the new command +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=paused-update -o jsonpath='{range .items[*]}{.metadata.name}{"="}{.spec.containers[0].command}{"\n"}{end}' + +> paused-update-backend-0=["sleep","7200"] +> paused-update-backend-1=["sleep","7200"] +> paused-update-backend-2=["sleep","7200"] +> paused-update-backend-3=["sleep","7200"] +``` + +**Expected output:** All 4 Pods have command `["sleep","7200"]` + +### Cleanup (Pause and Resume) + +```bash +kubectl delete rbg paused-update +``` + +--- + +## Operation 4: Multi-Role Coordinated Upgrade (CoordinatedPolicy) + +### Step 1: Create a Multi-Role RBG + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: pd-rollout-demo +spec: + roles: + - name: prefill + replicas: 7 + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + standalonePattern: + template: + spec: + containers: + - name: engine + image: alpine:3.23.5 + command: ["sleep", "3600"] + + - name: decode + replicas: 3 + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + standalonePattern: + template: + spec: + containers: + - name: engine + image: alpine:3.23.5 + command: ["sleep", "3600"] +EOF +``` + +### Step 2: Create CoordinatedPolicy + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: CoordinatedPolicy +metadata: + name: pd-rollout-demo +spec: + policies: + - name: prefill-decode-rollout + roles: + - prefill + - decode + strategy: + rollingUpdate: + maxSkew: "10%" + maxUnavailable: "10%" +EOF +``` + +### Step 3: Update Both Roles' command Simultaneously + +```bash +kubectl patch rbg pd-rollout-demo --type='json' \ + -p='[{"op": "replace", "path": "/spec/roles/0/standalonePattern/template/spec/containers/0/command", "value": ["sleep", "7200"]}, + {"op": "replace", "path": "/spec/roles/1/standalonePattern/template/spec/containers/0/command", "value": ["sleep", "7200"]}]' +``` + +### Expected Behavior (Coordinated Upgrade) + +- Prefill and Decode start updating simultaneously +- `maxSkew: "10%"` constrains the update progress difference between the two roles to within 10% +- If Prefill updates too fast (progress difference > 10%), Controller pauses Prefill, waiting for Decode to catch up +- Both roles' update progress remains close at all times +- `command` is a non-image field, so the update proceeds via Pod rebuild (Pods terminate and are recreated with the same ordinals one by one) + +### Verification (Coordinated Upgrade) + +```bash +# Observe the Pod update process +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=pd-rollout-demo -w + +> NAME READY STATUS RESTARTS AGE +> pd-rollout-demo-decode-0 1/1 Running 0 43m +> pd-rollout-demo-decode-1 1/1 Running 0 43m +> pd-rollout-demo-decode-2 1/1 Terminating 0 43m +> pd-rollout-demo-prefill-0 1/1 Running 0 43m +> pd-rollout-demo-prefill-1 1/1 Running 0 43m +> pd-rollout-demo-prefill-2 1/1 Running 0 43m +> pd-rollout-demo-prefill-3 1/1 Running 0 43m +> pd-rollout-demo-prefill-4 1/1 Running 0 43m +> pd-rollout-demo-prefill-5 1/1 Running 0 43m +> pd-rollout-demo-prefill-6 1/1 Terminating 0 43m +> pd-rollout-demo-prefill-6 0/1 Pending 0 0s +> pd-rollout-demo-decode-2 0/1 Pending 0 0s +> pd-rollout-demo-prefill-6 0/1 ContainerCreating 0 0s +> pd-rollout-demo-decode-2 0/1 ContainerCreating 0 0s +> pd-rollout-demo-decode-2 1/1 Running 0 39s +> pd-rollout-demo-prefill-6 1/1 Running 0 48s +> pd-rollout-demo-prefill-5 1/1 Terminating 0 44m +> pd-rollout-demo-prefill-5 0/1 Pending 0 0s +> pd-rollout-demo-prefill-5 0/1 ContainerCreating 0 0s +> pd-rollout-demo-prefill-5 1/1 Running 0 28s +> pd-rollout-demo-prefill-4 1/1 Terminating 0 45m +> pd-rollout-demo-decode-1 1/1 Terminating 0 45m +> pd-rollout-demo-prefill-4 0/1 Pending 0 0s +> pd-rollout-demo-prefill-4 0/1 ContainerCreating 0 0s +> pd-rollout-demo-decode-1 0/1 Pending 0 0s +> pd-rollout-demo-decode-1 0/1 ContainerCreating 0 0s +> pd-rollout-demo-prefill-4 1/1 Running 0 37s +> pd-rollout-demo-prefill-3 1/1 Terminating 0 47m +> pd-rollout-demo-decode-1 0/1 ContainerCreating 0 38s +> pd-rollout-demo-decode-1 1/1 Running 0 38s +> pd-rollout-demo-prefill-3 0/1 Pending 0 0s +> pd-rollout-demo-prefill-3 0/1 ContainerCreating 0 0s +> pd-rollout-demo-prefill-3 1/1 Running 0 27s +> pd-rollout-demo-prefill-2 1/1 Terminating 0 48m +> pd-rollout-demo-decode-0 1/1 Terminating 0 48m +> pd-rollout-demo-prefill-2 0/1 Pending 0 0s +> pd-rollout-demo-decode-0 0/1 Pending 0 0s +> pd-rollout-demo-prefill-2 0/1 ContainerCreating 0 0s +> pd-rollout-demo-decode-0 0/1 ContainerCreating 0 0s +> pd-rollout-demo-decode-0 1/1 Running 0 38s +> pd-rollout-demo-prefill-2 1/1 Running 0 48s +> pd-rollout-demo-prefill-1 1/1 Terminating 0 49m +> pd-rollout-demo-prefill-1 0/1 Pending 0 0s +> pd-rollout-demo-prefill-1 0/1 ContainerCreating 0 0s +> pd-rollout-demo-prefill-1 1/1 Running 0 30s +> pd-rollout-demo-prefill-0 1/1 Terminating 0 50m +> pd-rollout-demo-prefill-0 0/1 Error 0 50m +> pd-rollout-demo-prefill-0 0/1 Pending 0 0s +> pd-rollout-demo-prefill-0 0/1 ContainerCreating 0 0s +> pd-rollout-demo-prefill-0 1/1 Running 0 29s +``` + +From the update cadence, you can observe the coordination steps: 1p1d (1 prefill + 1 decode) - 1p - 1p1d - 1p - 1p1d - 1p - 1p, with the progress difference between the two roles always constrained by `maxSkew`. + +```bash +# After update completes, confirm all Pods use the new command +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=pd-rollout-demo -o jsonpath='{range .items[*]}{.metadata.name}{"="}{.spec.containers[0].command}{"\n"}{end}' + +> pd-rollout-demo-decode-0=["sleep","7200"] +> pd-rollout-demo-decode-1=["sleep","7200"] +> pd-rollout-demo-decode-2=["sleep","7200"] +> pd-rollout-demo-prefill-0=["sleep","7200"] +> pd-rollout-demo-prefill-1=["sleep","7200"] +> pd-rollout-demo-prefill-2=["sleep","7200"] +> pd-rollout-demo-prefill-3=["sleep","7200"] +> pd-rollout-demo-prefill-4=["sleep","7200"] +> pd-rollout-demo-prefill-5=["sleep","7200"] +> pd-rollout-demo-prefill-6=["sleep","7200"] +``` + +**Expected output:** + +- During the update process, the progress difference between the two roles remains <= 10% +- After update completes, all Pods have command `["sleep","7200"]` +- CoordinatedPolicy's status shows the coordination state + +### Cleanup (Coordinated Upgrade) + +```bash +kubectl delete cpolicy pd-rollout-demo +kubectl delete rbg pd-rollout-demo +``` + +--- + +## Summary + +| Operation | Verification Point | Key Expectation | +| --- | --- | --- | +| Basic rolling update | maxUnavailable / maxSurge | One-by-one replacement, always 3/4 available | +| Canary release | partition partition control | Only instances with ordinal >= partition are updated | +| Pause and resume | paused pause mechanism | No updates when paused, continues after resume | +| Coordinated upgrade | CoordinatedPolicy maxSkew | Two roles' update progress difference <= 10% | diff --git a/doc/best-practice/en/03-configuring-rolling-updates.md b/doc/best-practice/en/03-configuring-rolling-updates.md new file mode 100644 index 000000000..7eee1535c --- /dev/null +++ b/doc/best-practice/en/03-configuring-rolling-updates.md @@ -0,0 +1,288 @@ +# Configuring Rolling Update Strategies + +## Overview + +RBG supports configuring rolling update strategies via `rolloutStrategy` to control how, at what rate, and to what extent role instances are updated. For multi-role inference services (such as PD-disaggregated architecture), you can also use `CoordinatedPolicy` to achieve cross-role coordinated upgrades, ensuring consistent update progress across roles. + +> **Note**: This document does not cover the detailed mechanisms of in-place update (In-Place Update). In-place update will be introduced in a separate document. +> + +## Prerequisites + ++ Kubernetes cluster version >= 1.24 ++ RBG Controller installed (see [Installation Guide](https://github.com/sgl-project/rbg)) + +--- + +## Rolling Update Basic Configuration + +Each role can configure an independent rolling update strategy via the `rolloutStrategy` field. When a role's `template` changes (e.g., updating the image version), RBG Controller gradually updates instances according to the configured strategy. + +```yaml +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: rolling-update-demo +spec: + roles: + - name: backend + replicas: 4 + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + maxSurge: 0 + standalonePattern: + template: + spec: + containers: + - name: engine + image: alpine:3.23.5 + command: ["sleep", "3600"] +``` + +### Parameter Description + +| Parameter | Type | Required | Default | Description | +| --- | --- | --- | --- | --- | +| `rolloutStrategy.type` | string | No | `RollingUpdate` | Update strategy type, currently only supports `RollingUpdate` | +| `rolloutStrategy.rollingUpdate.maxUnavailable` | intOrString | No | `1` | Maximum number of unavailable instances during update. Can be an absolute value (e.g., `1`) or a percentage (e.g., `"25%"`) | +| `rolloutStrategy.rollingUpdate.maxSurge` | intOrString | No | `0` | Maximum number of instances allowed beyond the desired replica count during update. Can be an absolute value or a percentage | + +#### How maxUnavailable and maxSurge Work + ++ **maxUnavailable**: Controls the upper limit of "unavailable instances" during the update process. Higher values mean faster updates but lower service availability. ++ **maxSurge**: Controls whether additional instances can be created to speed up updates. Set to `0` means delete old instances first then create new ones; set to a positive number means create new instances first then delete old ones. + +**Example combinations**: + +| maxUnavailable | maxSurge | Behavior | Applicable Scenario | +| --- | --- | --- | --- | +| `1` | `0` | One-by-one replacement, no extra instances created | Resource-constrained environments | +| `1` | `1` | Create 1 new instance first, then delete 1 old instance | Balance availability and update speed | +| `"25%"` | `"25%"` | Allow 25% instances unavailable, create 25% extra instances simultaneously | Fast updates, lower availability requirements | + +--- + +## Canary Release: Partition Control + +The `partition` parameter allows you to control update progress, enabling canary releases. Only instances with ordinal **greater than or equal to** `partition` are updated; instances with ordinal less than `partition` remain on the old version. + +```yaml +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: canary-deployment +spec: + roles: + - name: backend + replicas: 4 + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + partition: 2 # Only update instances with ordinal >= 2 (i.e., instances 2 and 3) + standalonePattern: + template: + spec: + containers: + - name: engine + image: alpine:3.23.5 + command: ["sleep", "3600"] +``` + +### Parameter Description (Canary Release) + +| Parameter | Type | Required | Default | Description | +| --- | --- | --- | --- | --- | +| `rolloutStrategy.rollingUpdate.partition` | intOrString | No | `0` | Partition value. Only instances with ordinal >= partition are updated. Can be an absolute value (e.g., `2`) or a percentage (e.g., `"25%"`) | + +### Canary Release Flow + +Assuming `replicas: 4`, the update flow is as follows: + +1. **Set** `partition: 4`: All instances remain on the old version, no update starts +2. **Set** `partition: 3`: Only update instance 3 (the last one), observe the new version's behavior +3. **Set** `partition: 2`: Update instances 2 and 3, continue observing +4. **Set** `partition: 0`: Update all instances, complete the release + +> **Note**: Instance ordinals start from 0. `partition: 0` (default value) means update all instances. +> + +--- + +## Pause and Resume Updates + +The `paused` parameter can pause rolling updates, allowing you to manually control the timing of updates. + +```yaml +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: paused-update +spec: + roles: + - name: backend + replicas: 4 + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + paused: true # Pause updates + standalonePattern: + template: + spec: + containers: + - name: engine + image: alpine:3.23.5 + command: ["sleep", "3600"] +``` + +### Parameter Description (Pause and Resume) + +| Parameter | Type | Required | Default | Description | +| --- | --- | --- | --- | --- | +| `rolloutStrategy.rollingUpdate.paused` | bool | No | `false` | Whether to pause rolling updates | + +### Use Cases + ++ **Manual approval**: Pause after updating the template, confirm configuration is correct before resuming updates ++ **Phased release**: First update some instances (with `partition`), observe for a while before continuing ++ **Emergency rollback**: Immediately pause updates when issues are discovered, preventing further impact + +To resume updates, set `paused` to `false` (or remove the field), and the update will continue. + +--- + +## Advanced Configuration: Multi-Role Coordinated Upgrade + +### Why Coordinated Upgrades Are Needed + +In PD-disaggregated architecture, Prefill and Decode are two independent roles, each with its own independent rolling update strategy. If updates are triggered simultaneously, the two roles may update at different speeds, leading to the following issues: + +1. **Version mismatch**: Prefill has been updated to the new version, but Decode is still using the old version — the protocol or data structures between them may be incompatible +2. **Service unavailability**: If Prefill updates too fast, many instances become unavailable while Decode hasn't prepared to receive traffic, potentially causing request failures +3. **Resource contention**: Both roles updating simultaneously may cause cluster resource (e.g., GPU, network bandwidth) contention, slowing down updates + +**Coordinated upgrades** ensure that Prefill and Decode's update progress remains consistent, avoiding the above issues. + +### Configuring CoordinatedPolicy + +`CoordinatedPolicy` is an independent CRD used to define cross-role coordination policies. It controls the update progress difference between roles via the `maxSkew` parameter. + +> **Important**: The binding between a `CoordinatedPolicy` and a `RoleBasedGroup` is based on **same-name matching within the same namespace** — that is, a `CoordinatedPolicy` only takes effect for the RBG that has the **same namespace and the same name**. Therefore, the `CoordinatedPolicy`'s `metadata.name` and `metadata.namespace` must exactly match those of the target RBG. Each RBG can be bound to at most one CoordinatedPolicy. + +```yaml +--- +# RBG definition +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: pd-rollout-demo +spec: + roles: + - name: prefill + replicas: 7 + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + standalonePattern: + template: + spec: + containers: + - name: engine + image: alpine:3.23.5 + command: ["sleep", "3600"] + + - name: decode + replicas: 3 + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + standalonePattern: + template: + spec: + containers: + - name: engine + image: alpine:3.23.5 + command: ["sleep", "3600"] + +--- +# CoordinatedPolicy defines coordinated upgrade +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: CoordinatedPolicy +metadata: + name: pd-rollout-demo +spec: + policies: + - name: prefill-decode-rollout + roles: + - prefill + - decode + strategy: + rollingUpdate: + maxSkew: "10%" + maxUnavailable: "10%" +``` + +### Parameter Description (Coordinated Upgrade) + +| Parameter | Type | Required | Default | Description | +| --- | --- | --- | --- | --- | +| `spec.policies[].name` | string | Yes | - | Policy name, unique within the CoordinatedPolicy | +| `spec.policies[].roles` | []string | Yes | - | List of role names that need coordination | +| `spec.policies[].strategy.rollingUpdate.maxSkew` | string | No | `"100%"` | Maximum allowed progress deviation (percentage). Lower values mean tighter coordination | +| `spec.policies[].strategy.rollingUpdate.maxUnavailable` | string | No | - | Maximum proportion of unavailable instances allowed during coordinated update | + +### How Coordinated Upgrades Work + +Assume Prefill has 7 instances, Decode has 3 instances, `maxSkew: "10%"`: + +1. **Calculate update progress**: Each role's update progress = updated instances / total instances +2. **Control progress difference**: Controller ensures the update progress difference between Prefill and Decode does not exceed 10% +3. **Coordinate update order**: If Prefill updates too fast, Controller pauses Prefill's update, waiting for Decode to catch up; and vice versa + +**Example**: + ++ Prefill updated 4/7 = 57% of instances ++ Decode updated 1/3 = 33% of instances ++ Progress difference = 57% - 33% = 24% > 10% (exceeds maxSkew) ++ Controller pauses Prefill's update, prioritizes updating Decode, until progress difference <= 10% + +### Choosing maxSkew + +| maxSkew Value | Behavior | Applicable Scenario | +| --- | --- | --- | +| `"1%"` | Near-synchronous update, minimal progress difference | Scenarios requiring strict version consistency | +| `"10%"` | Allows small progress difference | Most production environments | +| `"50%"` | Allows larger progress difference | Higher update speed requirements, lower consistency requirements | +| `"100%"` | No progress difference limit (default) | No coordination needed, each role updates independently | + +> **Note**: The lower the `maxSkew`, the slower the update speed, but the higher the version consistency across roles. It is recommended to choose an appropriate value based on business requirements and cluster scale. +> + +--- + +## Verify Update Status + +```bash +# Check RBG status +kubectl get rbg + +# Check role instance update status +kubectl get roleinstances -l rbg.workloads.x-k8s.io/group-name= + +# Check CoordinatedPolicy status +kubectl get coordinatedpolicy +``` + +## Related Documents + + + ++ Deploying Inference Services with RBG ++ Using RoleTemplates to Reduce Configuration Duplication ++ In-Place Update (In-Place Update) ++ Configuring HPA Autoscaling diff --git a/doc/best-practice/zh/03-configuring-rolling-updates-guide.md b/doc/best-practice/zh/03-configuring-rolling-updates-guide.md new file mode 100644 index 000000000..ee8ad38c3 --- /dev/null +++ b/doc/best-practice/zh/03-configuring-rolling-updates-guide.md @@ -0,0 +1,511 @@ +# 操作文档：配置滚动更新策略 + +> 对应概念文档：[3. 配置滚动更新策略](03-configuring-rolling-updates.md) + +## 目标 + +验证 RBG 的 `rolloutStrategy` 滚动更新策略，包括： + +1. 基础滚动更新配置（maxUnavailable / maxSurge） +2. Partition 金丝雀发布 +3. Paused 暂停和恢复更新 +4. CoordinatedPolicy 多角色协作升级 + +## 前提条件 + +- Kubernetes 集群版本 >= 1.24 +- 已安装 RBG Controller +- 镜像可访问：`alpine:3.23.5` + +> **说明**：本文档使用 `sleep 3600` 作为占位命令，专注于验证 RBG 滚动更新控制面行为，无需 GPU。如需测试真实推理功能，请替换为完整的推理引擎启动命令。 + +--- + +## 操作一：基础滚动更新 + +### 步骤 1：创建 4 副本 RBG + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: rolling-update-demo +spec: + roles: + - name: backend + replicas: 4 + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + maxSurge: 0 + standalonePattern: + template: + spec: + containers: + - name: engine + image: alpine:3.23.5 + command: ["sleep", "3600"] +EOF +``` + +### 预期行为 + +- 创建 4 个 Pod（`rolling-update-demo-backend-0` 到 `rolling-update-demo-backend-3`） +- 全部就绪后，RBG 状态为 Ready + +### 步骤 2：触发滚动更新 + +```bash +# 修改 command（延长 sleep 时长），触发滚动更新（Pod 重建） +kubectl patch rbg rolling-update-demo --type='json' \ + -p='[{"op": "replace", "path": "/spec/roles/0/standalonePattern/template/spec/containers/0/command", "value": ["sleep", "7200"]}]' +``` + +### 预期行为（滚动更新触发） + +- 按序号从高到低逐个更新实例（3 → 2 → 1 → 0） +- `maxUnavailable: 1`：同一时间最多 1 个实例不可用 +- `maxSurge: 0`：不创建额外实例，先删旧再建新 +- 更新过程中始终有 3 个实例可用 +- `command` 属于非镜像字段，触发 Pod 重建：旧 Pod 先终止，再按相同序号重建（AGE 归零，RESTARTS 为 0） + +### 验证 + +```bash +# 观察更新过程（持续执行查看变化） +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=rolling-update-demo -w + +> NAME READY STATUS RESTARTS AGE +> rolling-update-demo-backend-0 1/1 Running 0 2m5s +> rolling-update-demo-backend-1 1/1 Running 0 2m5s +> rolling-update-demo-backend-2 1/1 Running 0 2m5s +> rolling-update-demo-backend-3 1/1 Terminating 0 2m5s +> rolling-update-demo-backend-3 0/1 Pending 0 0s +> rolling-update-demo-backend-3 0/1 ContainerCreating 0 0s +> rolling-update-demo-backend-3 1/1 Running 0 28s +> rolling-update-demo-backend-2 1/1 Terminating 0 2m59s +> rolling-update-demo-backend-2 0/1 Pending 0 0s +> rolling-update-demo-backend-2 0/1 ContainerCreating 0 0s +> rolling-update-demo-backend-2 1/1 Running 0 26s +> rolling-update-demo-backend-1 1/1 Terminating 0 3m56s +> rolling-update-demo-backend-1 0/1 Pending 0 0s +> rolling-update-demo-backend-1 0/1 ContainerCreating 0 0s +> rolling-update-demo-backend-1 1/1 Running 0 30s +> rolling-update-demo-backend-0 1/1 Terminating 0 4m58s +> rolling-update-demo-backend-0 0/1 Pending 0 0s +> rolling-update-demo-backend-0 0/1 ContainerCreating 0 0s +> rolling-update-demo-backend-0 1/1 Running 0 28s +``` + +可以观测到 Pod 按序号 3 → 2 → 1 → 0 逐个「终止 → 重建」，且同一时刻最多 1 个实例不可用。 + +```bash +# 确认所有 Pod 已更新到新 command +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=rolling-update-demo -o jsonpath='{range .items[*]}{.metadata.name}{"="}{.spec.containers[0].command}{"\n"}{end}' + +> rolling-update-demo-backend-0=["sleep","7200"] +> rolling-update-demo-backend-1=["sleep","7200"] +> rolling-update-demo-backend-2=["sleep","7200"] +> rolling-update-demo-backend-3=["sleep","7200"] +``` + +**预期输出：** + +- 更新过程中始终有 3 个 Pod Ready +- 更新完成后 4 个 Pod 的 command 均为 `["sleep","7200"]` + +### 清理 + +```bash +kubectl delete rbg rolling-update-demo +``` + +--- + +## 操作二：金丝雀发布（Partition） + +### 步骤 1：创建 4 副本 RBG 并设置 partition + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: canary-deployment +spec: + roles: + - name: backend + replicas: 4 + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + partition: 4 # 初始设为 4，不更新任何实例 + standalonePattern: + template: + spec: + containers: + - name: engine + image: alpine:3.23.5 + command: ["sleep", "3600"] +EOF +``` + +### 步骤 2：更新 command 但保持 partition=4 + +```bash +# 修改 command（延长 sleep 时长） +kubectl patch rbg canary-deployment --type='json' \ + -p='[{"op": "replace", "path": "/spec/roles/0/standalonePattern/template/spec/containers/0/command", "value": ["sleep", "7200"]}]' +``` + +### 预期行为（Partition=4） + +- 由于 `partition: 4`，序号 >= 4 的实例才会更新 +- 序号 0~3 均小于 4，所以**不更新任何实例** +- 所有 Pod 仍使用旧 command `["sleep","3600"]` + +### 验证（Partition=4） + +```bash +# 确认所有 Pod 仍使用旧 command +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=canary-deployment -o jsonpath='{range .items[*]}{.metadata.name}{"="}{.spec.containers[0].command}{"\n"}{end}' + +> canary-deployment-backend-0=["sleep","3600"] +> canary-deployment-backend-1=["sleep","3600"] +> canary-deployment-backend-2=["sleep","3600"] +> canary-deployment-backend-3=["sleep","3600"] +``` + +**预期输出：** 4 个 Pod 的 command 均为 `["sleep","3600"]` + +### 步骤 3：降低 partition 到 3，更新最后一个实例 + +```bash +kubectl patch rbg canary-deployment --type='json' \ + -p='[{"op": "replace", "path": "/spec/roles/0/rolloutStrategy/rollingUpdate/partition", "value": 3}]' +``` + +### 预期行为（Partition=3） + +- 序号 >= 3 的实例（即实例 3）被更新到新 command +- 序号 0~2 保持旧配置 + +### 验证（Partition=3） + +```bash +# 查看各 Pod 的 command +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=canary-deployment -o jsonpath='{range .items[*]}{.metadata.name}{"="}{.spec.containers[0].command}{"\n"}{end}' + +> canary-deployment-backend-0=["sleep","3600"] +> canary-deployment-backend-1=["sleep","3600"] +> canary-deployment-backend-2=["sleep","3600"] +> canary-deployment-backend-3=["sleep","7200"] +``` + +**预期输出：** + +- `canary-deployment-backend-3` 使用 `["sleep","7200"]` +- `canary-deployment-backend-0` ~ `2` 使用 `["sleep","3600"]` + +### 步骤 4：降低 partition 到 0，完成全部更新 + +```bash +kubectl patch rbg canary-deployment --type='json' \ + -p='[{"op": "replace", "path": "/spec/roles/0/rolloutStrategy/rollingUpdate/partition", "value": 0}]' +``` + +### 预期行为（Partition=0） + +- 序号 >= 0 的全部实例被更新到新 command + +### 验证（Partition=0） + +```bash +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=canary-deployment -o jsonpath='{range .items[*]}{.metadata.name}{"="}{.spec.containers[0].command}{"\n"}{end}' + +> canary-deployment-backend-0=["sleep","7200"] +> canary-deployment-backend-1=["sleep","7200"] +> canary-deployment-backend-2=["sleep","7200"] +> canary-deployment-backend-3=["sleep","7200"] +``` + +**预期输出：** 4 个 Pod 的 command 均为 `["sleep","7200"]` + +### 清理（金丝雀发布） + +```bash +kubectl delete rbg canary-deployment +``` + +--- + +## 操作三：暂停和恢复更新 + +### 步骤 1：创建 RBG 并触发更新后暂停 + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: paused-update +spec: + roles: + - name: backend + replicas: 4 + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 4 + paused: true # 暂停更新 + standalonePattern: + template: + spec: + containers: + - name: engine + image: alpine:3.23.5 + command: ["sleep", "3600"] +EOF +``` + +### 步骤 2：更新 command + +```bash +kubectl patch rbg paused-update --type='json' \ + -p='[{"op": "replace", "path": "/spec/roles/0/standalonePattern/template/spec/containers/0/command", "value": ["sleep", "7200"]}]' +``` + +### 预期行为（已暂停） + +- 由于 `paused: true`，更新被暂停 +- 所有 Pod 保持旧配置 + +### 验证（已暂停） + +```bash +# 确认所有 Pod 仍使用旧 command +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=paused-update -o jsonpath='{range .items[*]}{.metadata.name}{"="}{.spec.containers[0].command}{"\n"}{end}' + +> paused-update-backend-0=["sleep","3600"] +> paused-update-backend-1=["sleep","3600"] +> paused-update-backend-2=["sleep","3600"] +> paused-update-backend-3=["sleep","3600"] +``` + +**预期输出：** 4 个 Pod 的 command 均为 `["sleep","3600"]`（更新被暂停） + +### 步骤 3：恢复更新 + +```bash +kubectl patch rbg paused-update --type='json' \ + -p='[{"op": "replace", "path": "/spec/roles/0/rolloutStrategy/rollingUpdate/paused", "value": false}]' +``` + +### 预期行为（已恢复） + +- 更新恢复执行 +- Pod 逐个更新到新 command + +### 验证（已恢复） + +```bash +# 等待更新完成，确认所有 Pod 使用新 command +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=paused-update -o jsonpath='{range .items[*]}{.metadata.name}{"="}{.spec.containers[0].command}{"\n"}{end}' + +> paused-update-backend-0=["sleep","7200"] +> paused-update-backend-1=["sleep","7200"] +> paused-update-backend-2=["sleep","7200"] +> paused-update-backend-3=["sleep","7200"] +``` + +**预期输出：** 4 个 Pod 的 command 均为 `["sleep","7200"]` + +### 清理（暂停和恢复） + +```bash +kubectl delete rbg paused-update +``` + +--- + +## 操作四：多角色协作升级（CoordinatedPolicy） + +### 步骤 1：创建多角色 RBG + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: pd-rollout-demo +spec: + roles: + - name: prefill + replicas: 7 + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + standalonePattern: + template: + spec: + containers: + - name: engine + image: alpine:3.23.5 + command: ["sleep", "3600"] + + - name: decode + replicas: 3 + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + standalonePattern: + template: + spec: + containers: + - name: engine + image: alpine:3.23.5 + command: ["sleep", "3600"] +EOF +``` + +### 步骤 2：创建 CoordinatedPolicy + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: CoordinatedPolicy +metadata: + name: pd-rollout-demo +spec: + policies: + - name: prefill-decode-rollout + roles: + - prefill + - decode + strategy: + rollingUpdate: + maxSkew: "10%" + maxUnavailable: "10%" +EOF +``` + +### 步骤 3：同时更新两个角色的 command + +```bash +kubectl patch rbg pd-rollout-demo --type='json' \ + -p='[{"op": "replace", "path": "/spec/roles/0/standalonePattern/template/spec/containers/0/command", "value": ["sleep", "7200"]}, + {"op": "replace", "path": "/spec/roles/1/standalonePattern/template/spec/containers/0/command", "value": ["sleep", "7200"]}]' +``` + +### 预期行为（协作升级） + +- Prefill 和 Decode 同时开始更新 +- `maxSkew: "10%"` 约束两个角色的更新进度差异不超过 10% +- 如果 Prefill 更新过快（进度差异 > 10%），Controller 暂停 Prefill，等待 Decode 跟上 +- 两个角色的更新进度始终保持接近 +- command 为非镜像字段，更新以 Pod 重建方式进行（Pod 逐个终止并按序号重建） + +### 验证（协作升级） + +```bash +# 观察 Pod 更新过程 +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=pd-rollout-demo -w + +> NAME READY STATUS RESTARTS AGE +> pd-rollout-demo-decode-0 1/1 Running 0 43m +> pd-rollout-demo-decode-1 1/1 Running 0 43m +> pd-rollout-demo-decode-2 1/1 Terminating 0 43m +> pd-rollout-demo-prefill-0 1/1 Running 0 43m +> pd-rollout-demo-prefill-1 1/1 Running 0 43m +> pd-rollout-demo-prefill-2 1/1 Running 0 43m +> pd-rollout-demo-prefill-3 1/1 Running 0 43m +> pd-rollout-demo-prefill-4 1/1 Running 0 43m +> pd-rollout-demo-prefill-5 1/1 Running 0 43m +> pd-rollout-demo-prefill-6 1/1 Terminating 0 43m +> pd-rollout-demo-prefill-6 0/1 Pending 0 0s +> pd-rollout-demo-decode-2 0/1 Pending 0 0s +> pd-rollout-demo-prefill-6 0/1 ContainerCreating 0 0s +> pd-rollout-demo-decode-2 0/1 ContainerCreating 0 0s +> pd-rollout-demo-decode-2 1/1 Running 0 39s +> pd-rollout-demo-prefill-6 1/1 Running 0 48s +> pd-rollout-demo-prefill-5 1/1 Terminating 0 44m +> pd-rollout-demo-prefill-5 0/1 Pending 0 0s +> pd-rollout-demo-prefill-5 0/1 ContainerCreating 0 0s +> pd-rollout-demo-prefill-5 1/1 Running 0 28s +> pd-rollout-demo-prefill-4 1/1 Terminating 0 45m +> pd-rollout-demo-decode-1 1/1 Terminating 0 45m +> pd-rollout-demo-prefill-4 0/1 Pending 0 0s +> pd-rollout-demo-prefill-4 0/1 ContainerCreating 0 0s +> pd-rollout-demo-decode-1 0/1 Pending 0 0s +> pd-rollout-demo-decode-1 0/1 ContainerCreating 0 0s +> pd-rollout-demo-prefill-4 1/1 Running 0 37s +> pd-rollout-demo-prefill-3 1/1 Terminating 0 47m +> pd-rollout-demo-decode-1 0/1 ContainerCreating 0 38s +> pd-rollout-demo-decode-1 1/1 Running 0 38s +> pd-rollout-demo-prefill-3 0/1 Pending 0 0s +> pd-rollout-demo-prefill-3 0/1 ContainerCreating 0 0s +> pd-rollout-demo-prefill-3 1/1 Running 0 27s +> pd-rollout-demo-prefill-2 1/1 Terminating 0 48m +> pd-rollout-demo-decode-0 1/1 Terminating 0 48m +> pd-rollout-demo-prefill-2 0/1 Pending 0 0s +> pd-rollout-demo-decode-0 0/1 Pending 0 0s +> pd-rollout-demo-prefill-2 0/1 ContainerCreating 0 0s +> pd-rollout-demo-decode-0 0/1 ContainerCreating 0 0s +> pd-rollout-demo-decode-0 1/1 Running 0 38s +> pd-rollout-demo-prefill-2 1/1 Running 0 48s +> pd-rollout-demo-prefill-1 1/1 Terminating 0 49m +> pd-rollout-demo-prefill-1 0/1 Pending 0 0s +> pd-rollout-demo-prefill-1 0/1 ContainerCreating 0 0s +> pd-rollout-demo-prefill-1 1/1 Running 0 30s +> pd-rollout-demo-prefill-0 1/1 Terminating 0 50m +> pd-rollout-demo-prefill-0 0/1 Pending 0 0s +> pd-rollout-demo-prefill-0 0/1 ContainerCreating 0 0s +> pd-rollout-demo-prefill-0 1/1 Running 0 29s +``` + +通过更新节奏，可以观测到协作步骤：1p1d（1 prefill + 1 decode）- 1p - 1p1d - 1p - 1p1d - 1p - 1p，两个角色的进度差异始终受 `maxSkew` 约束。 + +```bash +# 更新完成后确认所有 Pod 使用新 command +kubectl get pods -l rbg.workloads.x-k8s.io/group-name=pd-rollout-demo -o jsonpath='{range .items[*]}{.metadata.name}{"="}{.spec.containers[0].command}{"\n"}{end}' + +> pd-rollout-demo-decode-0=["sleep","7200"] +> pd-rollout-demo-decode-1=["sleep","7200"] +> pd-rollout-demo-decode-2=["sleep","7200"] +> pd-rollout-demo-prefill-0=["sleep","7200"] +> pd-rollout-demo-prefill-1=["sleep","7200"] +> pd-rollout-demo-prefill-2=["sleep","7200"] +> pd-rollout-demo-prefill-3=["sleep","7200"] +> pd-rollout-demo-prefill-4=["sleep","7200"] +> pd-rollout-demo-prefill-5=["sleep","7200"] +> pd-rollout-demo-prefill-6=["sleep","7200"] +``` + +**预期输出：** + +- 更新过程中两个角色的进度差异始终 <= 10% +- 更新完成后所有 Pod 的 command 均为 `["sleep","7200"]` +- CoordinatedPolicy 的 status 中显示协作状态 + +### 清理（协作升级） + +```bash +kubectl delete cpolicy pd-rollout-demo +kubectl delete rbg pd-rollout-demo +``` + +--- + +## 总结 + +| 操作 | 验证点 | 关键预期 | +| --- | --- | --- | +| 基础滚动更新 | maxUnavailable / maxSurge | 逐个替换，始终 3/4 可用 | +| 金丝雀发布 | partition 分区控制 | 仅序号 >= partition 的实例更新 | +| 暂停和恢复 | paused 暂停机制 | 暂停时不更新，恢复后继续 | +| 协作升级 | CoordinatedPolicy maxSkew | 两个角色更新进度差异 <= 10% | diff --git a/doc/best-practice/zh/03-configuring-rolling-updates.md b/doc/best-practice/zh/03-configuring-rolling-updates.md new file mode 100644 index 000000000..e032db8f9 --- /dev/null +++ b/doc/best-practice/zh/03-configuring-rolling-updates.md @@ -0,0 +1,288 @@ +# 配置滚动更新策略 + +## 概述 + +RBG 支持通过 `rolloutStrategy` 配置滚动更新策略，控制角色实例的更新方式、速率和进度。对于多角色推理服务（如 PD 分离架构），还可以通过 `CoordinatedPolicy` 实现跨角色的协作升级，确保各角色的更新进度保持一致。 + +> **说明**：本文档不涉及原地升级（In-Place Update）的详细机制，原地升级将在单独的文档中介绍。 +> + +## 前提条件 + ++ Kubernetes 集群版本 >= 1.24 ++ 已安装 RBG Controller（参考 [安装指南](https://github.com/sgl-project/rbg)） + +--- + +## 滚动更新基础配置 + +每个角色可以通过 `rolloutStrategy` 字段配置独立的滚动更新策略。当角色的 `template` 发生变化时（如更新镜像版本），RBG Controller 会按照配置的策略逐步更新实例。 + +```yaml +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: rolling-update-demo +spec: + roles: + - name: backend + replicas: 4 + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + maxSurge: 0 + standalonePattern: + template: + spec: + containers: + - name: engine + image: alpine:3.23.5 + command: ["sleep", "3600"] +``` + +### 参数说明 + +| 参数 | 类型 | 是否必填 | 默认值 | 说明 | +| --- | --- | --- | --- | --- | +| `rolloutStrategy.type` | string | 否 | `RollingUpdate` | 更新策略类型，目前仅支持 `RollingUpdate` | +| `rolloutStrategy.rollingUpdate.maxUnavailable` | intOrString | 否 | `1` | 更新过程中允许不可用的最大实例数。可以是绝对值（如 `1`）或百分比（如 `"25%"`） | +| `rolloutStrategy.rollingUpdate.maxSurge` | intOrString | 否 | `0` | 更新过程中允许超出期望副本数的最大实例数。可以是绝对值或百分比 | + +#### maxUnavailable 和 maxSurge 的作用 + ++ **maxUnavailable**: 控制更新过程中"不可用实例"的上限。值越大，更新速度越快，但服务可用性越低。 ++ **maxSurge**: 控制是否允许创建额外的实例来加速更新。设为 `0` 表示先删除旧实例再创建新实例；设为正数表示可以先创建新实例再删除旧实例。 + +**示例组合**： + +| maxUnavailable | maxSurge | 行为 | 适用场景 | +| --- | --- | --- | --- | +| `1` | `0` | 逐个替换，不创建额外实例 | 资源受限环境 | +| `1` | `1` | 先创建 1 个新实例，再删除 1 个旧实例 | 平衡可用性和更新速度 | +| `"25%"` | `"25%"` | 允许 25% 实例不可用，同时创建 25% 额外实例 | 快速更新，对可用性要求较低 | + +--- + +## 金丝雀发布：Partition 控制 + +`partition` 参数允许你控制更新的进度，实现金丝雀发布。只有序号（ordinal）**大于等于** `partition` 的实例会被更新，序号小于 `partition` 的实例保持旧版本。 + +```yaml +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: canary-deployment +spec: + roles: + - name: backend + replicas: 4 + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + partition: 2 # 只更新序号 >= 2 的实例（即实例 2 和 3） + standalonePattern: + template: + spec: + containers: + - name: engine + image: alpine:3.23.5 + command: ["sleep", "3600"] +``` + +### 参数说明（金丝雀发布） + +| 参数 | 类型 | 是否必填 | 默认值 | 说明 | +| --- | --- | --- | --- | --- | +| `rolloutStrategy.rollingUpdate.partition` | intOrString | 否 | `0` | 分区值。只有序号 >= partition 的实例会被更新。可以是绝对值（如 `2`）或百分比（如 `"25%"`） | + +### 金丝雀发布流程 + +假设 `replicas: 4`，更新流程如下： + +1. **设置** `partition: 4`：所有实例保持旧版本，不开始更新 +2. **设置** `partition: 3`：只更新实例 3（最后一个），观察新版本表现 +3. **设置** `partition: 2`：更新实例 2 和 3，继续观察 +4. **设置** `partition: 0`：更新所有实例，完成发布 + +> **说明**：实例序号从 0 开始。`partition: 0`（默认值）表示更新所有实例。 +> + +--- + +## 暂停和恢复更新 + +`paused` 参数可以暂停滚动更新，允许你手动控制更新的时机。 + +```yaml +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: paused-update +spec: + roles: + - name: backend + replicas: 4 + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + paused: true # 暂停更新 + standalonePattern: + template: + spec: + containers: + - name: engine + image: alpine:3.23.5 + command: ["sleep", "3600"] +``` + +### 参数说明（暂停和恢复） + +| 参数 | 类型 | 是否必填 | 默认值 | 说明 | +| --- | --- | --- | --- | --- | +| `rolloutStrategy.rollingUpdate.paused` | bool | 否 | `false` | 是否暂停滚动更新 | + +### 使用场景 + ++ **手动审批**：在更新模板后先暂停，确认配置无误后再恢复更新 ++ **分阶段发布**：先更新部分实例（配合 `partition`），观察一段时间后继续 ++ **紧急回滚**：发现问题时立即暂停更新，避免影响扩大 + +恢复更新时，将 `paused` 设为 `false`（或删除该字段），更新会继续执行。 + +--- + +## 高级配置：多角色协作升级 + +### 为什么需要协作升级？ + +在 PD 分离架构中，Prefill 和 Decode 是两个独立的角色，各自配置独立的滚动更新策略。如果同时触发更新，两个角色可能会以不同的速度进行更新，导致以下问题： + +1. **版本不匹配**：Prefill 已经更新到新版本，但 Decode 还在使用旧版本，两者之间的协议或数据结构可能不兼容 +2. **服务不可用**：如果 Prefill 更新过快，大量实例不可用，而 Decode 还没准备好接收流量，可能导致请求失败 +3. **资源争抢**：两个角色同时更新可能导致集群资源（如 GPU、网络带宽）争抢，影响更新速度 + +**协作升级**确保 Prefill 和 Decode 的更新进度保持一致，避免上述问题。 + +### 配置 CoordinatedPolicy + +`CoordinatedPolicy` 是一个独立的 CRD，用于定义跨角色的协作策略。它通过 `maxSkew` 参数控制各角色之间的更新进度差异。 + +> **重要**：`CoordinatedPolicy` 与 `RoleBasedGroup` 的绑定关系基于**同一命名空间下的同名匹配**——即 `CoordinatedPolicy` 只会对与其**命名空间相同、名称相同**的 RBG 生效。因此，`CoordinatedPolicy` 的 `metadata.name` 和 `metadata.namespace` 必须与目标 RBG 完全一致。一个 RBG 至多绑定一个 CoordinatedPolicy。 + +```yaml +--- +# RBG 定义 +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: RoleBasedGroup +metadata: + name: pd-rollout-demo +spec: + roles: + - name: prefill + replicas: 7 + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + standalonePattern: + template: + spec: + containers: + - name: engine + image: alpine:3.23.5 + command: ["sleep", "3600"] + + - name: decode + replicas: 3 + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + standalonePattern: + template: + spec: + containers: + - name: engine + image: alpine:3.23.5 + command: ["sleep", "3600"] + +--- +# CoordinatedPolicy 定义协作升级 +apiVersion: workloads.x-k8s.io/v1alpha2 +kind: CoordinatedPolicy +metadata: + name: pd-rollout-demo +spec: + policies: + - name: prefill-decode-rollout + roles: + - prefill + - decode + strategy: + rollingUpdate: + maxSkew: "10%" + maxUnavailable: "10%" +``` + +### 参数说明（协作升级） + +| 参数 | 类型 | 是否必填 | 默认值 | 说明 | +| --- | --- | --- | --- | --- | +| `spec.policies[].name` | string | 是 | - | 策略名称，在 CoordinatedPolicy 内唯一 | +| `spec.policies[].roles` | []string | 是 | - | 需要协作的角色名称列表 | +| `spec.policies[].strategy.rollingUpdate.maxSkew` | string | 否 | `"100%"` | 允许的最大进度偏差（百分比）。值越小，协作越紧密 | +| `spec.policies[].strategy.rollingUpdate.maxUnavailable` | string | 否 | - | 协作更新过程中允许不可用的最大实例比例 | + +### 协作升级的工作原理 + +假设 Prefill 有 7 个实例，Decode 有 3 个实例，`maxSkew: "10%"`： + +1. **计算更新进度**：每个角色的更新进度 = 已更新实例数 / 总实例数 +2. **控制进度差异**：Controller 会确保 Prefill 和 Decode 的更新进度差异不超过 10% +3. **协调更新顺序**：如果 Prefill 更新过快，Controller 会暂停 Prefill 的更新，等待 Decode 跟上；反之亦然 + +**示例**： + ++ Prefill 更新了 4/7 = 57% 的实例 ++ Decode 更新了 1/3 = 33% 的实例 ++ 进度差异 = 57% - 33% = 24% > 10%（超出 maxSkew） ++ Controller 会暂停 Prefill 的更新，优先更新 Decode，直到进度差异 <= 10% + +### maxSkew 的选择 + +| maxSkew 值 | 行为 | 适用场景 | +| --- | --- | --- | +| `"1%"` | 几乎同步更新，进度差异极小 | 对版本一致性要求极高的场景 | +| `"10%"` | 允许较小的进度差异 | 大多数生产环境 | +| `"50%"` | 允许较大的进度差异 | 对更新速度要求较高，对一致性要求较低 | +| `"100%"` | 不限制进度差异（默认值） | 不需要协作，各角色独立更新 | + +> **说明**：`maxSkew` 越小，更新速度越慢，但各角色的版本一致性越高。建议根据业务需求和集群规模选择合适的值。 +> + +--- + +## 验证更新状态 + +```bash +# 查看 RBG 状态 +kubectl get rbg + +# 查看角色实例的更新状态 +kubectl get roleinstances -l rbg.workloads.x-k8s.io/group-name= + +# 查看 CoordinatedPolicy 状态 +kubectl get coordinatedpolicy +``` + +## 相关文档 + + + ++ 使用 RBG 部署推理服务 ++ 使用 RoleTemplates 减少配置重复 ++ 原地升级（In-Place Update） ++ 配置 HPA 弹性伸缩