Skip to content

docs: clarify CPU burst behavior on cgroup v2 #557

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
miantalha45 wants to merge 1 commit into
base: master
Choose a base branch
from
Open

docs: clarify CPU burst behavior on cgroup v2 #557

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 15 additions & 21 deletions docs/KeyFeatures/cloudNativeColocation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,33 +198,27 @@ Use the `stress` tool to apply pressure to the nginx Pod.
wrk -H "Accept-Encoding: deflate, gzip" -t 2 -c 8 -d 120 --latency --timeout 2s http://$(kubectl get svc nginx -o jsonpath='{.spec.clusterIP}')
```

#### Checking CPU Throttling
#### Checking CPU Throttling and Burst Configuration

Check the CPU throttling status of the Pod's container. We can see that `nr_bursts` and `burst_time` are not zero, while `nr_throttled` and `throttled_time` are relatively small, indicating that the Pod has used burst CPU quotas.
The commands below must be run in the cgroup of the actual container process. Replace `/sys/fs/cgroup/<container-cgroup>` with that cgroup's path. These commands use the cgroup v2 interface:

```bash
cat /sys/fs/cgroup/cpu/kubepods/burstable/podd2988e14-83bc-4d3d-931a-59f8a3174396/cpu.stat # replace nginx pod uid in your Kubernetes cluster.
nr_periods 1210
nr_throttled 9
throttled_time 193613865
nr_bursts 448
burst_time 6543701690
cd /sys/fs/cgroup/<container-cgroup>
uname -r
cat cpu.max
test -e cpu.max.burst && cat cpu.max.burst || echo "cpu.max.burst is unsupported"
cat cpu.stat
```

If we set the Pod's annotation `volcano.sh/enable-quota-burst=false` (disabling CPU Burst) and perform another stress test, `nr_throttled` and `throttled_time` will be relatively large, indicating strict CPU throttling, while `nr_bursts` and `burst_time` will be zero, indicating no CPU Burst occurred.
A non-zero `cpu.max.burst` value means that CPU Burst is configured for the cgroup. The `cpu.stat` file always reports the regular CPU accounting fields, such as `nr_periods`, `nr_throttled`, and `throttled_usec`. The later `nr_bursts` and `burst_usec` fields are kernel-dependent accounting fields: their absence does not mean that CPU Burst is disabled or that no burst occurred. If `cpu.max.burst` is missing, the host kernel does not expose the cgroup v2 CPU Burst interface and Volcano cannot enable CPU Burst on that node.

```bash
cat /sys/fs/cgroup/cpu/kubepods/burstable/podeeb542c6-b667-4da4-9ac9-86ced4e93fbb/cpu.stat #replace nginx pod uid in your Kubernetes cluster.
nr_periods 1210
nr_throttled 488
throttled_time 10125826283
nr_bursts 0
burst_time 0
```
`nr_throttled` records periods in which the normal CPU quota was exceeded. A non-zero value therefore indicates quota throttling, not necessarily a failure to configure CPU Burst. With sustained load, throttling can still occur after the available burst budget is used.

The cgroup v1 interface uses different file and counter names. Do not use the v1 `burst_time` example to interpret cgroup v2 output. On cgroup v2, the expected burst accounting field is `burst_usec` when the running kernel provides it.

#### Notes

CPU Burst relies on Linux kernel functionality. This feature is only effective on hosts with Linux kernel versions >= 5.14 and certain Linux distributions (e.g., OpenEuler 22.03 SP2 or later).
CPU Burst relies on Linux kernel functionality and the cgroup CPU controller. On cgroup v2, verify that the target cgroup exposes `cpu.max.burst`; if it is missing, the host kernel does not support configuring CPU Burst through this interface. Volcano's application-level colocation features, such as unified scheduling, dynamic resource oversubscription, and node-pressure eviction, are supported on generic-purpose operating systems such as Ubuntu and CentOS. Kernel-level CPU, memory, and network isolation or suppression remains dependent on the capabilities of the host OS and kernel.

### Dynamic Resource Oversubscription Example

Expand Down Expand Up @@ -358,7 +352,7 @@ If the online job is stopped to release node resource pressure, the **Volcano Ag

#### Notes

The **Volcano Agent** defines a QoS resource model for online and offline jobs and provides application-level guarantees for online jobs (e.g., evicting offline jobs under node resource pressure). Meanwhile, CPU and memory isolation and suppression are guaranteed at the OS level by the **host kernel**. Note that the Volcano Agent currently only supports **openEuler 22.03 SP2** and later versions, so ensure the correct OS type and version when using this feature.
The **Volcano Agent** defines a QoS resource model for online and offline jobs and provides application-level guarantees for online jobs (e.g., evicting offline jobs under node resource pressure). Volcano supports these application-level colocation capabilities on generic-purpose operating systems such as Ubuntu and CentOS. CPU and memory isolation and suppression are guaranteed at the OS level by the **host kernel** and therefore depend on the host OS and kernel capabilities. When using kernel-level colocation features, ensure that the required kernel interfaces are available.

In the egress network bandwidth isolation mechanism, the bandwidth usage of offline jobs is limited, especially when online jobs require more bandwidth. To achieve finer bandwidth control, three watermark parameters are typically defined to dynamically adjust the bandwidth allocation for offline jobs.

Expand Down Expand Up @@ -524,7 +518,7 @@ For containers in Pods with the CPU Burst feature enabled, their CPU usage can b

You can specify a custom burst quota by setting the Pod's annotation `volcano.sh/quota-burst-time`. For example:

If a container's CPU limit is 4 cores, the Volcano Agent defaults the container's CGroup `cpu.cfs_quota_us` value to `400000` (the CFS base period is `100000`, so 4 cores correspond to `4 * 100000 = 400000`). This means the container can burst up to 4 additional CPU cores at a time. If you set `volcano.sh/quota-burst-time=200000`, the container can only burst up to 2 additional CPU cores at a time.
On cgroup v2, if a container's CPU limit is 4 cores, the Volcano Agent sets the quota in `cpu.max` using the CFS base period of `100000` microseconds (4 cores corresponds to `400000 100000`). The burst budget is exposed through `cpu.max.burst`; a value of `400000` allows up to 4 additional CPU cores during a scheduling period. If you set `volcano.sh/quota-burst-time=200000`, the cgroup's `cpu.max.burst` value is limited to `200000`, allowing up to 2 additional CPU cores at a time. On cgroup v1, the equivalent quota and accounting interfaces use different names.

#### Dynamic Resource Oversubscription

Expand Down Expand Up @@ -559,4 +553,4 @@ You can adjust the online and offline bandwidth watermarks by modifying the `vol

#### Custom Oversubscription Policy Development

The Volcano agent defaults to using the [extend](https://github.com/volcano-sh/volcano/tree/master/pkg/agent/oversubscription/policy) method to report and use oversubscribed resources, i.e., reporting oversubscribed resources as an extended resource type to the node. If you want to customize the reporting and usage of oversubscribed resources, such as reporting them as native CPU and memory resources, and customize suspension and resumption of scheduling behaviors, implement the [policy Interface](https://github.com/volcano-sh/volcano/blob/4dea29b334877058786615ac1ed79143601dc600/pkg/agent/oversubscription/policy/policy.go#L48) to develop a custom oversubscription policy, and set the Volcano agent's startup parameter `oversubscription-policy` to the corresponding policy.
The Volcano agent defaults to using the [extend](https://github.com/volcano-sh/volcano/tree/master/pkg/agent/oversubscription/policy) method to report and use oversubscribed resources, i.e., reporting oversubscribed resources as an extended resource type to the node. If you want to customize the reporting and usage of oversubscribed resources, such as reporting them as native CPU and memory resources, and customize suspension and resumption of scheduling behaviors, implement the [policy Interface](https://github.com/volcano-sh/volcano/blob/4dea29b334877058786615ac1ed79143601dc600/pkg/agent/oversubscription/policy/policy.go#L48) to develop a custom oversubscription policy, and set the Volcano agent's startup parameter `oversubscription-policy` to the corresponding policy.
36 changes: 15 additions & 21 deletions ...ans/docusaurus-plugin-content-docs/current/KeyFeatures/cloudNativeColocation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,33 +212,27 @@ spec:
wrk -H "Accept-Encoding: deflate, gzip" -t 2 -c 8 -d 120 --latency --timeout 2s http://$(kubectl get svc nginx -o jsonpath='{.spec.clusterIP}')
```

#### 检查CPU限流情况
#### 检查 CPU 限流和突发配置

检查 Pod 容器的 CPU 限流状态，我们可以看到 `nr_bursts` 和 `burst_time` 不为 0，而 `nr_throttled` 和 `throttled_time` 是一个较小的值，这表明 Pod 已经使用了突发的 CPU 配额。
以下命令必须在实际容器进程所在的 cgroup 中执行。请将 `/sys/fs/cgroup/<container-cgroup>` 替换为该 cgroup 的路径。以下命令使用 cgroup v2 接口：

```bash
cat /sys/fs/cgroup/cpu/kubepods/burstable/podd2988e14-83bc-4d3d-931a-59f8a3174396/cpu.stat # replace nginx pod uid in your kubernetes cluster.
nr_periods 1210
nr_throttled 9
throttled_time 193613865
nr_bursts 448
burst_time 6543701690
cd /sys/fs/cgroup/<container-cgroup>
uname -r
cat cpu.max
test -e cpu.max.burst && cat cpu.max.burst || echo "cpu.max.burst is unsupported"
cat cpu.stat
```

如果我们设置 Pod 的注解 `volcano.sh/enable-quota-burst=false`（禁用 Pod 的 CPU Burst）并进行另一次压力测试，`nr_throttled` 和 `throttled_time` 将会是一个相对较大的值，这表明 Pod 的 CPU 被严格限制；而 `nr_bursts` 和 `burst_time` 为 0，表明 Pod 的 CPU Burst 没有发生。
非零的 `cpu.max.burst` 值表示该 cgroup 已配置 CPU Burst。`cpu.stat` 始终会报告 `nr_periods`、`nr_throttled` 和 `throttled_usec` 等常规 CPU 统计字段。较新的 `nr_bursts` 和 `burst_usec` 字段取决于内核版本；缺少这些字段并不表示 CPU Burst 未启用或没有发生突发。如果缺少 `cpu.max.burst`，则说明主机内核没有公开 cgroup v2 的 CPU Burst 接口，Volcano 无法在该节点启用 CPU Burst。

```bash
cat /sys/fs/cgroup/cpu/kubepods/burstable/podeeb542c6-b667-4da4-9ac9-86ced4e93fbb/cpu.stat #replace nginx pod uid in your kubernetes cluster.
nr_periods 1210
nr_throttled 488
throttled_time 10125826283
nr_bursts 0
burst_time 0
```
`nr_throttled` 表示超过常规 CPU 配额的调度周期数。该值非零表示发生了配额限流，并不一定表示 CPU Burst 配置失败。持续高负载时，突发配额耗尽后仍可能发生限流。

cgroup v1 使用不同的文件名和统计字段。请勿使用 v1 的 `burst_time` 示例解释 cgroup v2 的输出；在 cgroup v2 中，如果运行中的内核提供突发统计字段，应使用 `burst_usec`。

#### 说明

CPU Burst 依赖于 Linux 内核提供的功能，该特性仅在主机使用的 Linux 内核版本 >= 5.14 以及某些 Linux 发行版（如 OpenEuler 22.03 SP2 或更高版本）上生效。
CPU Burst 依赖 Linux 内核功能和 cgroup CPU 控制器。在 cgroup v2 中，请确认目标 cgroup 暴露了 `cpu.max.burst`；如果缺少该文件，则主机内核不支持通过此接口配置 CPU Burst。Volcano 的应用级混部功能（例如统一调度、动态资源超卖和节点压力驱逐）支持 Ubuntu、CentOS 等通用操作系统。内核级 CPU、内存和网络隔离或抑制仍取决于主机操作系统和内核的能力。

### 动态资源超卖示例

Expand Down Expand Up @@ -372,7 +366,7 @@ Events:

#### 说明

**Volcano Agent** 为在线作业和离线作业定义了一个 QoS（服务质量）资源模型，并为在线作业提供了应用级别的保障机制（例如在节点资源压力下驱逐离线作业）。同时，CPU 和内存的隔离与抑制由**主机内核**（Host Kernel）在操作系统级别提供保障。需要注意的是，目前 Volcano Agent 仅适配 **openEuler 22.03 SP2** 及更高版本，因此使用该功能时请确保使用正确的操作系统类型和版本。
**Volcano Agent** 为在线作业和离线作业定义了一个 QoS（服务质量）资源模型，并为在线作业提供了应用级别的保障机制（例如在节点资源压力下驱逐离线作业）。Volcano 的这些应用级混部能力支持 Ubuntu、CentOS 等通用操作系统。CPU 和内存的隔离与抑制由**主机内核**（Host Kernel）在操作系统级别提供保障，因此取决于主机操作系统和内核的能力。使用内核级混部功能时，请确认所需的内核接口可用。

### 出口网络带宽保障示例

Expand Down Expand Up @@ -565,7 +559,7 @@ Connecting to host 192.168.2.30, port 5201

你可以通过设置 Pod 的注解 `volcano.sh/quota-burst-time` 来指定自定义的突增配额。例如：

如果一个容器的 CPU 限制为 4 核，Volcano Agent 默认会将容器的 CGroup `cpu.cfs_quota_us` 值设置为 `400000`（CFS 的基本周期为 `100000`，因此 4 核 CPU 对应 `4 * 100000 = 400000`）。这意味着容器在某一时刻最多可以额外使用 4 核 CPU。如果你设置 `volcano.sh/quota-burst-time=200000`，则表示容器在某一时刻最多只能额外使用 2 核 CPU。
在 cgroup v2 中，如果一个容器的 CPU 限制为 4 核，Volcano Agent 会使用 `100000` 微秒的 CFS 基本周期在 `cpu.max` 中设置配额（4 核对应 `400000 100000`）。突发配额通过 `cpu.max.burst` 暴露；值为 `400000` 表示每个调度周期最多可额外使用 4 核 CPU。如果设置 `volcano.sh/quota-burst-time=200000`，则 cgroup 的 `cpu.max.burst` 值会限制为 `200000`，表示每个调度周期最多可额外使用 2 核 CPU。cgroup v1 使用不同的配额和统计接口。

#### 动态资源超卖

Expand Down Expand Up @@ -600,4 +594,4 @@ Connecting to host 192.168.2.30, port 5201

#### 自定义开发超卖策略

Volcano agent默认使用[extend]([volcano/pkg/agent/oversubscription/policy at master · volcano-sh/volcano (github.com)](https://github.com/volcano-sh/volcano/tree/master/pkg/agent/oversubscription/policy))的方式上报和使用超卖资源，也就是将超卖资源作为一种扩展资源类型上报到节点，如果想自定义超卖资源的上报和使用，如上报为原生的cpu和memory资源，以及暂停和恢复调度的行为，请实现[policy Interface](https://github.com/volcano-sh/volcano/blob/4dea29b334877058786615ac1ed79143601dc600/pkg/agent/oversubscription/policy/policy.go#L48)实现自定义的超卖策略开发，并设置Volcano agent的启动参数`oversubscription-policy`为相应的策略。
Volcano agent默认使用[extend]([volcano/pkg/agent/oversubscription/policy at master · volcano-sh/volcano (github.com)](https://github.com/volcano-sh/volcano/tree/master/pkg/agent/oversubscription/policy))的方式上报和使用超卖资源，也就是将超卖资源作为一种扩展资源类型上报到节点，如果想自定义超卖资源的上报和使用，如上报为原生的cpu和memory资源，以及暂停和恢复调度的行为，请实现[policy Interface](https://github.com/volcano-sh/volcano/blob/4dea29b334877058786615ac1ed79143601dc600/pkg/agent/oversubscription/policy/policy.go#L48)实现自定义的超卖策略开发，并设置Volcano agent的启动参数`oversubscription-policy`为相应的策略。
Loading
Loading