v0.13.1

This is a new minor release of NRI Reference Plugins. It brings few bugfixes, minor documentation updates and increased e2e test coverage.

What's New

Balloons Policy

• Publishing high priority PCT cores as extended resources fixed with a robust reconciliation loop for managing extended resources. This includes deletion of stale plugin-owned extended resources.

What's Changed in Details

• docs: fix helm uninstalls in PCT how-to's by @askervin in #712
• balloons: fix publishing PCT HP CPU cores as extended resources by @askervin in #711
• e2e: fix reimporting images with multiple images loaded on vm by @askervin in #715
• balloons/agent: fix/rewrite/simplify extended resource handling by @askervin in #714

Full Changelog: v0.13.0...v0.13.1

v0.13.0

This is a new major release of NRI Reference Plugins. It comes with major new features, some bugfixes and dependency updates.

What's New

Balloons Policy

• cpuClasses with turbo priority and PCT The balloons policy has a new cpuClasses configuration section. You can use it to set CPU frequency limits, energy-performance preference, C-state overrides, uncore frequency, and the CPUFreq governor on a per-class basis. Frequency fields accept plain numbers (kHz), units (3.2GHz, 2900MHz), or the symbolic names min, base, and turbo — resolved from sysfs at startup.

  turboPriority handles the common case where you want only one balloon type to get actual turbo headroom at any given time. Among all active balloon types, the one with the highest turboPriority gets turbo; the rest are capped at base clock. The scope is configurable per socket or system-wide via turboDomain.

  pctPriority and pctClosID wire up Intel Speed Select Technology Priority Core Turbo (SST-CP/SST-TF). In managed mode the plugin programs the CLOS assignments itself. In assoc-only mode it associates CPUs to CLOSes that are already configured by the operator or BIOS. Setting publishExtendedResource: true on a PCT class makes the plugin advertise a Kubernetes extended resource (cpuclass.balloons.nri.io/<class>) whose value tracks available HP CPUs on the node, so the scheduler can avoid over-subscribing a single node's HP capacity. See Quick start: PCT in Kubernetes with Balloons for more information.

New nri-resctrl-mon Plugin

• nri-resctrl-mon is a new standalone plugin that creates per-pod RDT monitoring groups in /sys/fs/resctrl. Each pod gets its own mon_group for the lifetime of the pod sandbox, making per-pod L3 cache and memory bandwidth metrics available to Prometheus (for example via Kepler) without touching any policy configuration. Requires kernel RDT support. With CRI-O, version 1.36 or later is required. See Resctrl-Mon NRI Plugin documentation for more information.

Logging

• slog-based logging The internal logger now uses Go's standard slog package. The pkg/log API is unchanged, but the underlying implementation is smaller and cleaner.

• OpenTelemetry-based log exporting Produced logs can now be exported to an OpenTelemetry-compatible log collector. The following custom resource fragment enables gRPC based log exporting.

apiVersion: config.nri/v1alpha1
kind: TopologyAwarePolicy
metadata:
  name: default
spec:
...
  instrumentation:
    httpEndpoint: 8891
    logExportPeriod: 15s
    logExporter: otlp-grpc
  log:
    debug:
      - policy
...

Additionally you need to pass the collector endpoint to the resource policy plugin using the stock Opentelemetry environment variables. With the above configuration, you need to pass OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=http://otel-collector:4317 if your collector is otel-collector and configured to use the standard port.

You can set both the configuration and the necessary extra environment variable with a Helm config fragment like this:

config:
  reservedResources:
    cpu: 750m
  pinCPU: true
  pinMemory: true
  instrumentation:
    httpEndpoint: :8891
    logExporter: otlp-grpc
    logExportPeriod: 15s
  log:
    debug:
      - policy
extraEnv:
  OTEL_EXPORTER_OTLP_LOGS_ENDPOINT: http://otel-collector:4317

new goresctrl v0.13.0 and updated SST API

goresctrl is updated to v0.13.0, which restructures the SST model around a Platform handle and per-package Punit slices. The sysfs and cpuallocator packages are updated accordingly. sysfs.System has a new Sst() method that returns the platform handle directly for code that needs it.

Fixes

• Policies no longer fail to start on kernels built with CONFIG_NUMA=n.
• resctrl-mon: mon_groups now persist until pod sandbox removal rather than disappearing when the first container in the pod stops.
• resctrl-mon: pod UIDs are normalised to standard dashed-UUID format before directory lookups, fixing a lookup failure with certain runtimes.
• resctrl-mon: CRI-O versions before 1.36 are rejected at startup with a descriptive error (they don't populate Container.Pid in NRI events).
• memtierd: temporary file handling tightened to avoid stale files after a restart.
• cgroups: fixed an integer truncation in cgroup stats parsing that could silently corrupt values.
• instrumentation: fixed accidental Prometheus configuration activation on startup.
• Logging: fixed stderrthreshold being ignored when logtostderr is set.
• Security: the NRI socket directory is now mounted read-only in all Helm DaemonSets.
• sysfs: replaced unsafe integer-cast helpers with typed generics, eliminating a class of integer-truncation issues flagged by CodeQL.

Dependency updates

• goresctrl: v0.12.0 → v0.13.0
• OpenTelemetry SDK: v1.42.0 → v1.44.0 (plus new log exporter packages)
• prometheus/client_golang: v1.23.0 → v1.23.2
• grpc: v1.79.3 → v1.81.1

CI

Added Trivy vulnerability scanning and CodeQL security analysis workflows.

What's Changed In Details

• resource-annotator: Document command line options by @ozhuraki in #648
• build: Remove extra clean in topology's Dockerfile by @ozhuraki in #647
• Honor stderrthreshold when logtostderr is enabled by @pierluigilenoci in #646
• topology-aware: Add metrics unit tests by @ozhuraki in #654
• topology-aware: Skip size arbitration if both offers are nil by @ozhuraki in #649
• topology-aware: Add libmem unit tests by @ozhuraki in #653
• topology-aware: Add Node traversal stop by @ozhuraki in #652
• sysfs: Add tests for parsing entries into units by @ozhuraki in #655
• sysfs: run policies even if kernel CONFIG_NUMA=n by @askervin in #658
• topology-aware: Add Node traversal unit tests by @ozhuraki in #650
• build(deps): bump google.golang.org/grpc from 1.79.2 to 1.79.3 by @dependabot[bot] in #644
• operator: update the maintainer details by @fmuyassarov in #661
• build(deps): bump pygments from 2.17.0 to 2.20.0 in /docs by @dependabot[bot] in #651
• operator: replace deprecated gcr.io/kubebuilder/kube-rbac-proxy by @fmuyassarov in #662
• golang,golang-ci-lint: bump versions, fix linting errors. by @klihub in #664
• resctrl-mon: add NRI plugin for per-pod resctrl monitoring groups by @cmcantalupo in #666
• mount NRI socket directory as read-only by @fmuyassarov in #675
• Parallelize makefile running by @uniemimu in #678
• Move tar.go to testutils by @uniemimu in #677
• Fix unescaped dot in regexp by @uniemimu in #680
• Fix issues in instrumentation unit test by @askervin in #685
• e2e: install cri-tools/crictl on Fedora 43 and later. by @klihub in #684
• Fix "make test" issues caused by pkg/http by @askervin in #683
• buildx: load the images to local docker daemon. by @fmuyassarov in #673
• resctrl-mon: revert CRI-O pidfile fallback due to security concerns by @cmcantalupo in #671
• resctrl-mon: accept pod UIDs without dashes too by @fmuyassarov in #674
• Fix for incorrect conversion between integer types by @uniemimu in #679
• pkg/sysfs: replace interface{}-based sysfs I/O with typed generics by @kad in #682
• e2e: fix CIDRs added to vm's no_proxy to be effective by @askervin in #687
• build: dispel ghosts from topology-aware images by @askervin in #689
• instrumentation, log: add support for otel log collection. by @klihub in #663
• fixes: fix hint allow-/deny-list filtering tests. by @klihub in #692
• memtierd: harden temporary file handling by @askervin in #690
• cache: avoid a test cache population data race by @klihub in #696
• e2e: fix kubectl wait in otel log collection test. by @klihub in #695
• e2e: strict topology hint test fixes by @klihub in #694
• e2e: build custom guest OS kernels with AMD enabled by @askervin in #699
• e2e: fix otel log test wait error, possible instability by @askervin in #698
• Makefile: make sure test cleanup is not run in parallel...

v0.12.2

This is a new minor release of NRI Reference Plugins. It comes with a few new features, some bugfixes and dependency updates.

What's New

Topology Aware Policy

• Topology Pools for CPU Clusters by L3 Cache On hardware where clustering CPUs by shared L3 cache results in a distinctive topological entity when compared to clustering by sockets, nearest NUMA node, or dies, the policy now creates L3 cache pools in the topology tree. In such setups it is now also possible to put an L3 cache level cap on unlimited burstable containers, in which case the policy will try to fit such a container within a L3 cache clustered set of CPUs with enough free resources. This configuration fragment will set L3 cache burstability cap as the global default:

config:
  reservedResources:
    cpu: 1
  ...
  # Constrain unlimited burstable containers close to a single NUMA node.
  unlimitedBurstable: l3cache

You can also annotate an L3 cache burstability cap for containers. For instance with this pod spec

apiVersion: v1
kind: Pod
metadata:
  name: burstable
  annotations:
    # constrain unlimited burstable containers to a single socket by default
    unlimited-burstable.resource-policy.nri.io/pod: 
    # but tighten this for ctr0 to an L3 cache cluster
    unlimited-burstable.resource-policy.nri.io/container.ctr0: l3cache
spec:
  containers:
  - name: ctr0
    image: myimage-ctr0
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 2
        memory: 500M
      limits:
        memory: 500M
  - name: ctr1
    image: myimage-ctr1
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 1
        memory: 100M
      limits:
        memory: 100M
...

• Strict Topology Hint interpretation can be annotated on pods or containers. When annotated for strict hints, the policy will treat the fulfillment of topology hints a requirement instead of a preference and rather fail allocation of a container than assign resources to misaligned wrt. to a strict hint. For instance this pod spec annotates the highprio container within the pod for strict hints.

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  annotations:
    # fail creation of highprio if strict alignment by hints is not possible
    strict.topologyhints.resource-policy.nri.io/container.highprio: 'true'
spec:
  containers:
  - name: highprio
    image: myimage-highprio
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 1
        memory: 500M
      limits:
        cpu: 1
        memory: 500M
  - name: normal
    image: myimage-normal
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 500m
        memory: 100M
      limits:
        cpu: 500m
        memory: 100M

Note that you might need to ensure that any unexpected implicit topology hints are disabled so they don't cause misalignment failures in strict mode. For instance you might want to disable all implicit hints from mounts and devices, if you want to enable strict alignment for pod resource API hints.

• Strict CPU Isolation can now be annotated on pods or containers. Such an annotation turns the allocation of isolated CPUs into a requirement instead of a preference, failing container creation if isolated CPUs cannot be allocated. For instance this pod spec will require isolated CPUs for the highprio container of the pod.

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  annotations:
    # fail creation of highprio if we can't allocate an isolated CPU for it.
    require-isolated-cpus.resource-policy.nri.io/container.highprio: "true"
spec:
  containers:
  - name: highprio
    image: myimage-highprio
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 1
        memory: 500M
      limits:
        cpu: 1
        memory: 500M
  - name: normal
    image: myimage-normal
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 500m
        memory: 100M
      limits:
        cpu: 500m
        memory: 100M

What's Changed

• go.{mod,sum}: bump otel deps to latest 1.42.0. by @klihub in #641
• resmgr: block cache save during initial sync. by @klihub in #640
• go.{mod,sum}: bump goresctrl deps to v0.12.0. by @klihub in #642
• topology-aware: add l3Cache topology/pool nodes. by @wongchar in #635
• cpuallocator, topology-aware: handle weird die setups better by @klihub in #643
• topology-aware: implement strict hint and CPU isolation preferences. by @klihub in #638

New Contributors

• @wongchar made their first contribution in #635

Full Changelog: v0.12.1...v0.12.2

v0.12.1

This is a new minor release of NRI Reference Plugins. It brings a few minor new features, an overhauled documentation for the resource policy plugins, some bugfixes and dependency updates.

What's New

Balloons Policy

• Scheduling Class is a is a list of scheduling related parameters organized in classes. These parameters tune containers when they are created but do not affect already running containers. The class of a container is defined with the schedulingClass: <name> option in container's balloon type, and can be overridden by scheduling-class.resource-policy.nri.io pod annotation. Each class in the list has following properties.
  • name is the name of the scheduling class.
  • policy is the Linux scheduling policy. Supported policies are: none, other, fifo, rr, batch, idle, and deadline.

• priority is the scheduling priority. Refer to sched_setscheduler(2) documentation for valid values depending on the policy.
• flags is a list of scheduling flags. Supported flags are: reset-on-fork, reclaim, dl-overrun, keep-policy, keep-params, util-clamp-min, util-clamp-max.
• nice: nice value for the container process.
• runtime: runtime value for deadline scheduling policy (in microseconds).
• deadline: deadline value for deadline scheduling policy (in microseconds).
• period: period value for deadline scheduling policy (in microseconds).
• ioClass: IO class for the container process. Supported classes are: none, rt for realtime, be for best-effort, and idle.
• ioPriority: IO priority for the container process. Refer to ionice(1) documentation for valid values.
  For example the following configuration fragment defines two balloon types, high-priority and low-priority, assigning them to the critical and background scheduling classes, with the former using a realtime scheduling policy and elevated I/O priority, and the latter using an idle scheduling policy.

balloonTypes:
- name: high-priority
  schedulingClass: critical
  ...
- name: low-priority
  schedulingClass: background
  ...
schedulingClasses:
- name: critical
  policy: rr
  priority: 50
  ioClass: rt
  ioPriority: 0
- name: background
  policy: idle
  ioClass: idle

• Excluded CPU Set notation for allowed and reserved resources allows one to specify the allowed and reserved CPU set by specifying which CPUs should be excluded from the set instead of giving explicitly the included ones. This can be especially useful in a hybrid node cluster when the number of CPUs present varies between nodes. For the available resources exclusion is with respect to the CPUs present in the system. For reserved resources exclusion is with respect to the CPU present in the available resource set.

Topology Aware Policy

• Scheduling Class A container can be assigned to a known 'scheduling class' by name using the scheduling-class.resource-policy.nri.io effective annotation key. The value of the annotation is the name of the class for the container or the pod. The class itself needs to be defined in the active policy configuration using the schedulingClasses configuration option. For instance the following Helm configuration fragment defines two classes, realtime and idle with the corresponding scheduling and I/O priority attributes.

config:
  reservedResources:
    cpu: 2
  ...
  schedulingClasses:
  - name: realtime
    policy: fifo # SCHED_FIFO
    priority: 42

  - name: idle
    policy: idle # SCHED_IDLE
    nice: 17
    ioClass: be
    ioPriority: 6
...

The following pod annotation will then assign the container c0 to the realtime class:

metadata:
  annotations:
    scheduling-class.resource-policy.nri.io/container.c0: realtime

• Inherited Scheduling Classes The new NamespaceSchedulingClasses and PodQoSSchedulingClasses configuration CR options can be used to set up default scheduling classes per namespace and Pod QoS class. If a container is not assigned explicitly to any scheduling class by annotation, it inherits the default scheduling class for its namespace or Pod QoS class, in this order of precedence, if either or both is set.
• Excluded CPU Set notation for allowed and reserved resources allows one to specify the allowed and reserved CPU set by specifying which CPUs should be excluded from the set instead of giving explicitly the included ones. This can be especially useful in a hybrid node cluster when the number of CPUs present varies between nodes. For the available resources exclusion is with respect to the CPUs present in the system. For reserved resources exclusion is with respect to the CPU present in the available resource set.

Resource Annotator Webhook

• Cert-Manager support It is now possible to generate and inject HTTPS certificates for the webhook using cert-manager. Here is a sample session of setting up the webhook for cert-manager injection.

# Install cert-manager, if you don't have it yet.
$ helm install cert-manager oci://quay.io/jetstack/charts/cert-manager \
       --version v1.19.2 --namespace cert-manager --create-namespace \
       --set crds.enabled=true --set crds.keep=false

# Bootstrap a local issuer for cert-manager if you don't have one yet.
$ kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-cluster-issuer
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: internal-root-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: internal-root-ca
  secretName: internal-root-ca-secret
  issuerRef:
    name: selfsigned-cluster-issuer
    kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: console-ca-issuer
  namespace: cert-manager
spec:
  ca:
    secretName: internal-root-ca-secret
EOF
$ kubectl apply -f ca-bootstrap.yaml
$ kubectl wait --for=condition=Ready=True clusterissuer/console-ca-issuer

# Install the chart referring it to the certificate issuer.
$ helm install -n kube-system nri-webhook nri-plugins/nri-resource-annotator \
      --set image.tag=v0.12.0 --set image.pullPolicy=IfNotPresent \
      --set service.certificateIssuer=console-ca-issuer

This should set up the resource annotator with a cert-manager issued
certificate.

What's Changed

• helm: adjust resource-annotator helm chart for better consistency with the rest. by @klihub in #624
• scripts: add resource-annotator to artifact checker script. by @klihub in #623
• helm,resource-annotator: add chart support for cert-manager. by @klihub in #625
• balloons: support configuring Linux scheduling parameters by @askervin in #618
• topology-aware: initial scheduling class support. by @klihub in #626
• resmgr: update topology change on reconfigure by @askervin in #629
• balloons: fix insufficient memory on reconfigurations by @askervin in #630
• doc: balloons documentation rewrite + cookbook by @askervin in #627
• topology-aware: add 'excluded cpuset' config notation by @klihub in #631
• operator: bypass ansible-operator proxy for Helm deployments by @fmuyassarov in #633
• e2e: be more specific when looking for known container processes with pgrep. by @klihub in #636
• docs: restructure topology-aware documentation. by @klihub in #628

Full Changelog: v0.12.0...v0.12.1

v0.12.0

This new release of NRI Reference Plugins brings a few new features to resource policy plugins.

What's New

Balloons Policy

• Load Balancing for Composite Balloons
  Thjs feature enables creating equally many balloons from two or more balloon types for the same set of containers. For example, if balloon types B0, ..., B3 are local to similar but separate hardware resources that accelerate HPC applications, this feature enables balancing the number of balloons created from each of these types for HPC containers. Containers are assigned to a composite balloon type BC, consisting of components B0, ..., B3, and componentCreation strategy of the BC balloon type is set to balance-balloons.

Common Policy Improvements

• Resource Annotation
  Exact resource requirements for container can be annotated on containers' pod. This can be useful in clusters with high CPU count nodes when one wants to allocate more than 256 CPUs to a single container, which is not properly detected otherwise. The newly added nri-resource-annotator webhook can be used to automate this process. You can install the webhook using the provided Helm chart with the following commands:

$ NS=kube-system; SVC=resource-annotator; CERT=~/webhook-cert
$ mkdir -p $CERT
$ openssl req -x509 -newkey rsa:2048 -sha256 -days 365 -nodes \
      -keyout $CERT/server-key.pem -out $CERT/server-crt.pem \
      -subj "/CN=$SVC.$NS.svc" -addext "subjectAltName=DNS:$SVC,DNS:$SVC.$NS,DNS:$SVC.$NS.svc"
$ helm repo add nri-plugins https://containers.github.io/nri-plugins
$ helm repo update
$ helm -n $NS install webhook nri-plugins/nri-resource-annotator \
       --set image.name=ghcr.io/containers/nri-plugins/nri-resource-annotator \
       --set service.base64Crt=$(base64 -w0 < $CERT/server-crt.pem) \
       --set service.base64Key=$(base64 -w0 < $CERT/server-key.pem)

The policies should automatically take resource annotations into account when they are present.

Other Changes

• OpenTelemetry metrics collection
  Metrics collection has been updated to use OpenTelemetry for metrics instrumentation. In connection
  with this change, the names of some available metrics have changed when collected using Prometheus.
  In particular, metric names from the Balloons and Topology-Aware policies have changes.

What's Changed

• balloons: extend composite balloons with a kind of load balancing by @askervin in #617
• metrics: switch to OpenTelemetry based metrics collection. by @klihub in #600
• resource-annotator: add resource annotator mutating webhook. by @klihub in #619
• e2e: add global burstability limit test case. by @klihub in #607
• scripts: fix Helm release artifact checker. by @klihub in #609
• .github: fix unstable chart publishing with Helm v4.x. by @klihub in #610
• e2e: run_tests.sh accepts user overrides for vars read from files by @askervin in #613
• e2e: report skipped tests with SKIP instead of PASS in the summary by @askervin in #615
• e2e: bump default test distro to fedora/43. by @klihub in #616
• e2e: enable creating VMs up to 4096 CPUs and CPU hot-plugging/removing from test scripts by @askervin in #587
• e2e: add 8-socket 4k-CPU e2e vm and related test for balloons policy by @askervin in #591
• e2e: add 8-socket 4k-CPU test for topology-aware by @askervin in #599
• e2e: support "cxl" in hardware topology (CXL-1) by @askervin in #611
• e2e: enable custom kernel building, caching and installing (CXL-2) by @askervin in #612
• e2e: add a test that hotplugs and hotremoves CXL memory (CXL-3) by @askervin in #614
• build: ignore operator build failures for non releases by @klihub in #620
• e2e/balloons: update expected metrics pattern. by @klihub in #621

Full Changelog: v0.11.0...v0.12.0

v0.11.0

This new release of NRI Reference Plugins brings both new features and bug fixes to resource policy plugins.

What's New

Balloons Policy

• CPU C-state control allows selectively disabling some C-states of CPUs assigned to a balloon. A typical use case for this is to disable the deepest power saving C-states in balloons that host latency critical applications. This can improve hardware wakeup and consequently kernel scheduling latency for processes in the balloon.

config:
  reservedResources:
    cpu: 1
...
  idleCPUClass: default-class
...
  balloonTypes:
  - name: low-latency
    cpuClass: lowlatency-class
...
  control:
    cpu:
      classes:
        lowlatency-class:
          disabledCstates: [C4, C6, C8, C10]
        default-class:
          disabledCstates: []
...

Topology Aware Policy

• Improved burstable CPU allocation allows the policy to prefer topology pools which can provide enough CPU capacity for a burstable QoS class container to reach its burstable limit, whenever there are enough free or idle shared capacity in the pool.

• Configurable unlimited burstability puts a topological cap on the limit when picking CPUs for a burstable QoS class container without CPU limit, to prevent such containers from unconditionally getting assigned to the system pool with all CPUs and memory nodes. The supported caps are system (all sockets), package (a single socket), die (CPUs in a single die), and numa (CPUs close to a single NUMA node). The cap defaults to package. Additionally, containers can be annotated with a pod- or container-specific burstability cap.

config:
  reservedResources:
    cpu: 1
  ...
  # Constrain unlimited burstable containers close to a single NUMA node.
  unlimitedBurstable: numa

apiVersion: v1
kind: Pod
metadata:
  name: burstable
  annotations:
    # constrain unlimited burstable containers to a single die by default
    unlimited-burstable.resource-policy.nri.io/pod: die
    # but loosen this for ctr0 to a full socket
    unlimited-burstable.resource-policy.nri.io/container.ctr0: package
spec:
  containers:
  - name: ctr0
    image: myimage-ctr0
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 2
        memory: 500M
      limits:
        memory: 500M
  - name: ctr1
    image: myimage-ctr1
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 1
        memory: 100M
      limits:
        memory: 100M
...

• Exclusive allocation of all CPUs in a NUMA node, die and socket. The policy allows all CPUs in a shared pool to be allocated for exclusive use, as long as this does not cause any of the shared pools in active use to become completely devoid of CPU resources. This allows now tighter allocation and alignment of resources. The policy can now allocate a full NUMA node, die, or socket worth of CPUs exclusively to a single container from a single NUMA node, die or socket, while this was formerly refused forcing the container to at least one topology level above the tightest possible allocation.

What's Changed

• scripts/build/update-gh-pages: fix parsing of latest release tag by @marquiz in #556
• Bump golangci-lint and fix linter errors by @marquiz in #559
• e2e: fix VM disk sizing, provision rootfs with max possible size. by @klihub in #561
• build: bump k8s client and controller generator toolchains to latest. by @klihub in #562
• Bump golang to v1.25 by @marquiz in #558
• image-build: switch to bookworm for image building. by @klihub in #564
• build: speed up image building significantly. by @klihub in #565
• e2e: support custom image URLs, use fedora 42 by default. by @klihub in #568
• e2e: don't fail if partition resize fails. by @klihub in #566
• e2e: pre-download and cache custom vagrant images. by @klihub in #569
• e2e: use persistent connections for provisioning. by @klihub in #572
• e2e: adjust cloud image URLs, allow image override, fix ssh connectivity problems. by @klihub in #571
• docs: fix and improve reset (allow all) affinity by @askervin in #573
• memory-policy: fix crashes without configuration by @askervin in #574
• e2e: fix false failure in balloons test02-prometheus-metrics by @askervin in #575
• e2e: abort on vagrant bootstrap errors. by @klihub in #577
• e2e: always configure systemd cgroup driver for containerd. by @klihub in #576
• e2e: alternate test runtimes in tests. by @klihub in #578
• resmgr,helm: nuke obsolete metrics-interval command line flag. by @klihub in #582
• e2e: EFI BIOS support for e2e test VMs. by @klihub in #581
• e2e: cache and reuse release tarballs. by @klihub in #583
• Balloons: enabling disabling c-states in latency-critical balloons by @askervin in #579
• balloons: fix CpuLocations and its unit test by @askervin in #586
• topology-aware: consider burstable CPU limit when picking a pool by @klihub in #570
• policy: fix missing mem_node_capacity metrics data. by @klihub in #588
• e2e: fix helm install after helm v4.0 cli flag rename by @askervin in #589
• pkg/cgroups: remove obsoleted/unused cgroupblkio. by @klihub in #592
• config/blockio: fix typo, clarify block I/O setup. by @klihub in #593
• pkg/cgroupstats: nuke unused v1 cgroup collector. by @klihub in #594
• e2e: fix helm-launch when error is expected by @askervin in #595
• balloons: allow allocation of online CPUs only by @askervin in #590
• e2e: speed up test execution by @askervin in #596
• topology-aware: avoid slicing busy shared pools empty by @klihub in #601
• topology-aware: allow slicing idle shared pools empty. by @klihub in #602
• topology-aware: don't (ac)count 0 CPU req. containers in the reserved pool to shared pools by @klihub in #604
• e2e: add gentler and rougher vm-reboot methods by @askervin in #606

Full Changelog: v0.10.1...v0.11.0

v0.10.1

This is a new minor release of NRI Reference Plugins. It updates dependencies, enables RDT 'discovery mode', brings a few documentation improvements, and fixes a startup failure on machines with an asymmetric NUMA distance matrix.

What's Changed

• build(deps): bump golang.org/x/oauth2 from 0.21.0 to 0.27.0 by @dependabot[bot] in #548
• docs: update documentation for RDT monitoring/metrics. by @klihub in #549
• rdt,resmgr: allow running RDT in discovery mode. by @klihub in #551
• docs: update balloons debugging guidance and add examples by @askervin in #552
• go.mod,Makefile: bump golang to latest 1.24.x. by @klihub in #555
• sysfs: patch up asymmetric NUMA distances. by @klihub in #554

Full Changelog: v0.10.0...v0.10.1

v0.10.0

This new release of NRI Reference Plugins brings a new NRI plugin, new features in resource policy plugins, a number of bug fixes, end-to-end tests and few use cases in documentation.

What's New

Balloons Policy

• Composite balloons enables allocating a diverse set of CPUs for containers with complex CPU requirements. For example, "allocate an equal number of CPUs from both NUMA nodes on CPU socket 0". This allocation enables efficient parallelism inside an AI inference engine container that runs inference on CPU, and still isolate inference engines from each other.

  balloonTypes:
  - name: balance-pkg0-nodes
    components:
    - balloonType: node0
    - balloonType: node1
  - name: node0
    preferCloseToDevices:
    - /sys/devices/system/node/node0
  - name: node1
    preferCloseToDevices:
    - /sys/devices/system/node/node1

• Documentation includes recipes for preventing creation of certain containers on a worker node, and resetting CPU and memory pinning of all containers in a cluster.

Topology Aware Policy

• Pick CPU and Memory by Topology Hints Normally topology hints are only used to pick the assigned pool for a workload. Once a pool is selected the available resources within the pool are considered equally good for satisfying the topology hints. When the policy is allocating exclusive CPUs and picking pinned memory for the workload, only other potential criteria and attributes are considered for picking the individual resources.

  When multiple devices are allocated to a single container, it is possible that this default assumption of all resources within the pool being topologically equal is not true. If a container is allocated misaligned devices, IOW devices with different memory or CPU locality. To overcome this, containers can now be annotated to prefer hint based selection and pinning of CPU and memory resources using the pick-resources-by-hints.resource-policy.nri.io annotation. For example,

  apiVersion: v1
  kind: Pod
  metadata:
    name: data-pump
    annotations:
      k8s.v1.cni.cncf.io/networks: sriov-net1
      prefer-isolated-cpus.resource-policy.nri.io/container.ctr0: "true"
      pick-resources-by-hints.resource-policy.nri.io/container.ctr0: "true"
  spec:
    containers:
    - name: ctr0
      image: dpdk-pump
      imagePullPolicy: Always
      resources:
        requests:
          cpu: 2
          memory: 100M
          vendor.com/sriov_netdevice_A: '1'
          vendor.com/sriov_netdevice_B: '1'
        limits:
          vendor.com/sriov_netdevice_A: '1'
          vendor.com/sriov_netdevice_B: '1'
          cpu: 2
          memory: 100M

  When annotated like that, the policy will try to pick one exclusive isolated CPU with locality to one device and another with locality to the other. It will also try to pick and pin to memory aligned with these devices.

Common Policy Improvements

These are improvements to common infrastructure and as such are available for the balloons and topology-aware policy plugins, as well as for the wireframe template policy plugin.

• Cache Allocation

  Plugins can be configured to exercise class-based control over the L2 and L3 cache allocated to containers' processes. In practice, containers are assigned to classes. Classes have a corresponding cache allocation configuration. This configuration is applied to all containers and subsequently to all processes started in a container. To enable cache control use the control.rdt.enable option which defaults to false.

  Plugins can be configured to assign containers by default to a cache class named after the Pod QoS class of the container: one of BestEffort, Burstable, and Guaranteed. The configuration setting controlling this behavior is control.rdt.usagePodQoSAsDefaultClass and it defaults to false.

  Additionally, containers can be explicitly annotated to be assigned to a class. Use the rdtclass.resource-policy.nri.io annotation key for this. For instance

  apiVersion: v1
  kind: Pod
  metadata:
    name: test-pod
    annotations:
      rdtclass.resource-policy.nri.io/pod: poddefaultclass
      rdtclass.resource-policy.nri.io/container.special-container: specialclass
  ...

  This will assign the container named special-container within the pod to the specialclass RDT class and any other container within the pod to the poddefaultclass RDT class. Effectively these containers' processes will be assigned to the RDT CLOSes corresponding to those classes.

  Cache Class/Partitioning Configuration

  RDT configuration is supplied as part of thecontrol.rdt configuration block. Here is a sample snippet as a Helm chart value which assigns 33%, 66% and 100% of cache lines to BestEffort, Burstable and Guaranteed Pod QoS class containers correspondingly:

  config:
    control:
      rdt:
        enable: true
        usePodQoSAsDefaultClass: true
        options:
          l2:
            optional: true
          l3:
            optional: true
          mb:
            optional: true
        partitions:
          fullCache:
            l2Allocation:
              all:
                unified: 100%
            l3Allocation:
              all:
                unified: 100%
            classes:
              BestEffort:
                l2Allocation:
                  all:
                    unified: 33%
                l3Allocation:
                  all:
                    unified: 33%
              Burstable:
                l2Allocation:
                  all:
                    unified: 66%
                l3Allocation:
                  all:
                    unified: 66%
              Guaranteed:
                l2Allocation:
                  all:
                    unified: 100%
                l3Allocation:
                  all:
                    unified: 100%

  Cache Allocation Prerequisites

  Note that for cache allocation control to work, you must have

  • a hardware platform which supports cache allocation
  • resctrlfs pseudofilesystem enabled in your kernel, and loaded if it is a module
  • the resctrlfs filesystem mounted (possibly with extra options for your platform)

New plugin: nri-memory-policy

• The NRI memory policy plugin sets Linux memory policy for new containers.
• The memory policy plugin, for instance, advises kernel to interleave memory pages of a container on all NUMA nodes in the system, or on all NUMA nodes near the same socket where container's allowed CPUs are located.
• The plugin works as a stand-alone plugin, and it works together with NRI resource policy plugins and Kubernetes resource managers. It recognizes CPU and memory pinning set by resource management components. The memory policy plugin should be after the resource policy plugins in the NRI plugins chain.
• Memory policy for a container is defined in pod annotations.
• At the time of NRI plugins release, latest released containerd or CRI-O do not support NRI Linux memory policy adjustments, or NRI container command line adjustments for a workaround. Using this plugin requires a container runtime that is built with NRI version including command line adjustments. (NRI version > 0.9.0)

What's Changed

• resmgr,config: allow configuring cache allocation via goresctrl. by @klihub in #541
• resmgr: expose RDT metrics. by @klihub in #543
• Balloons with components by @askervin in #526
• topology-aware: try picking resources by hints first by @klihub in #545
• memory-policy: NRI plugin for setting memory policy by @askervin in #517
• mempolicy: go interface for set_mempolicy and get_mempolicy syscalls by @askervin in #514
• mpolset: get/set memory policy and exec a command by @askervin in #515
• topology-aware: fix format of container-exported memsets. by @klihub in #532
• resmgr: update container-exported resource data. by @klihub in #537
• sysfs: add a helper for gathering whatever IDs related to CPUs by @askervin in #513
• sysfs: fix CPU.GetCaches() to not return empty slice. by @klihub in #533
• sysfs: export CPUFreq.{Min,Max}. by @klihub in #534
• helm: add Chart for memory-policy deployment by @askervin in #519
• go.{mod,sum}: use new goresctrl tag v0.9.0. by @klihub in #544
• Drop tools.go in favor of native tool directive support in go 1.24 by @fmuyassarov in #535
• golang: bump go version to 1.24[.3]. by @klihub in #528

Full Changelog: v0.9.4...v0.10.0

v0.9.4

This is a new minor release of NRI Reference Plugins. It fixes incorrect caching of Pod Resource API query results which in some cases could result in incorrect generated topology hints.

What's Changed

• resmgr: purge cached pod resource list upon pod stop/removal. by @klihub in #507
• github: explicitly ensure contents-only copying by @fmuyassarov in #508

Full Changelog: v0.9.3...v0.9.4

v0.9.3

This is a new minor release of NRI Reference Plugins. It brings several new features, a number of bug fixes, end-to-end tests, and test coverage.

What's New

Balloons Policy

• Cluster level visibility to CPU affinity. Configuration option agent.nodeResourceTopology: true enables observing balloons as zones in NodeResourceTopology custom resources. Furthermore, if showContainersInNrt: true is defined, information on each container, including CPU affinity, will be shown as a subzone of its balloon.

  Example configuration:

  showContainersInNrt: true
  agent:
    nodeResourceTopology: true

  Enables listing balloons and their cpusets on K8SNODE with

  kubectl get noderesourcetopology K8SNODE -o json | jq '.zones[] | select(.type=="balloon") | {"balloon":.name, "cpuset":(.attributes[]|select(.name=="cpuset").value)}'

  and containers with their cpusets on the same node:

  kubectl get noderesourcetopology K8SNODE -o json | jq '.zones[] | select(.type=="allocation for container") | {"container":.name, "cpuset":(.attributes[]|select(.name=="cpuset").value)}'

• System load balancing. Even if two containers run on disjoint sets of logical CPUs, they may nevertheless affect each others performance. This happens, for instance, if two memory-intensive containers share the same level 2 cache, or if they are compute-intenstive, use the same compute resources of a physical CPU core, and run on two hyperthreads of the same core.

  New system load balancing in the balloons policy is based on classifying loads generated containers using new loadClasses configuration option. Based on the load classes associated with balloonTypes using loads, the policy allocates CPUs to new and existing balloons so that it avoids overloading level 2 caches or physical CPU cores.

  Example: policy prefers selecting CPUs for all "inference engine" and "computational-fluid-dynamics" balloons within separate level 2 cache blocks to prevent cache trashing by any two of containers in these balloons.

  balloonTypes:
  - name: inference-engine
    loads:
    - memory-intensive
    ...
  - name: computational-fluid-dynamics
    loads:
    - memory-intensive
  ...
  loadClasses:
  - name: memory-intensive
    level: l2cache

Topology Aware Policy

• Improved topology hint control: the topologyhints.resource-policy.nri.io annotation key can be used to enable or disable topology hint generation for one or more containers altogether, or selectively for mounts, devices, and pod resources types.
  For example:

metadata:
  annotations:
    # disable topology hint generation for all containers by default
    topologyhints.resource-policy.nri.io/pod: none
    # disable other than mount-based hints for the 'diskwriter' container
    topologyhints.resource-policy.nri.io/container.diskwriter: mounts
    # disable other than device-based hints for the 'videoencoder' container
    topologyhints.resource-policy.nri.io/container.videoencoder: devices
    # disable other than pod resource-based hints for the 'dpdk' container
    topologyhints.resource-policy.nri.io/container.dpdk: pod-resources
    # enable device and pod resource-based hints for 'networkpump' container
    topologyhints.resource-policy.nri.io/container.networkpump: devices,pod-resources

It is also possible to enable and disable topology hint generation based on mount or device path, using allow and deny lists. See the updated documentation for more details.

• relaxed system topology restrictions: the policy should not refuse to start up if a NUMA node is shared by more than one pool at the same topology hierarchy level. In particular, a single NUMA node shared by all sockets should not prevent startup any more.

• improved Burstable QoS class container handling: the policy now allocates memory to burstable QoS class containers based on memory request estimates. This should lower the probability for unexpected allocation failures when burstable containers are used to allocate a node close to full capacity.

• better global shared allocation preference: a preferSharedCPUs: true global configuration option now applies to all containers, unless they are annotated to opt out using the prefer-shared-cpus.resource-policy.nri.io annotation.

Common Policy Improvements

• Container cache and memory bandwidth allocation enables class-based management of system L2 and L3 cache and memory bandwidth. They are modeled as class-based uncountable and shareable resources. Containers can be assigned to predefined classes-of-service (CLOS), or RDT classes for short. Each class defines a specific configuration for cache and memory bandwidth allocation, which is applied to all containers within that class. The assigned container class is resolved and mapped to a CLOS in the runtime using goresctrl library. RDT control must be enabled in the runtime and the assigned classes must be defined in the runtime configuration. Otherwise the runtime might fail to create containers that are assigned to an RDT class. Refer to the containerd, cri-o, and goresctrl documentation for more details about configuration.

A container can be assigned either to an RDT class matching its pod's QoS class (BestEffort, Burstable or Guaranteed), or it can be assigned to an arbitrary class using the rdtclass.resource-policy.nri.io annotation. To enable QoS-class based default assignment you can use a configuration fragment similar to this:

apiVersion: config.nri/v1alpha1
kind: TopologyAwarePolicy # or 'BalloonsPolicy' for the 'balloons' policy
metadata:
  name: default
spec:
  ...
  control:
    rdt:
      enable: true
      usePodQoSAsDefaultClass: true

RDT class assignment is also possible using annotations. For instance, to assign the packetpump container to the highprio and the scheduler container to the midprio classes. Any other potential container in the pod will be assigned to the class matching their pd's QoS class:

metadata:
  annotations:
    rdtclass.resource-policy.nri.io/container.packetpump: highprio
    rdtclass.resource-policy.nri.io/container.scheduler: midprio

• Container block I/O prioritization allows class-based control block I/O prioritization and throttling. Containers can be assigned to predefined block I/O classes. Each class defines a specific configuration of prioritization and throttling parameters which are applied to all containers assigned to the class. The assigned container class is resolved and mapped to actual parameters in the runtime using goresctrl library. Block I/O control must be enabled in the runtime and the classes must be defined in the runtime configuration. Otherwise the runtime fails to create containers that are assigned to a block I/O class. Refer to the containerd, cri-o, and goresctrl documentation for more details about configuration.

A container can be assigned either to a Block I/O class matching its pod's QoS class (BestEffort, Burstable or Guaranteed), or it can be assigned to an arbitrary class using the blockioclass.resource-policy.nri.io annotation. To enable QoS-class based default assignment you can use a configuration fragment similar to this:

apiVersion: config.nri/v1alpha1
kind: TopologyAwarePolicy
metadata:
  name: default
spec:
  ...
  control:
    blockio:
      enable: true
      usePodQoSAsDefaultClass: true

Class assignment is also possible using annotations. For instance, to assign the database container to the highprio and
the logger container to the lowprio classes. Any other potential container in the pod will be assigned to the class matching their pd's QoS class:

metadata:
  annotations:
    blockioclass.resource-policy.nri.io/container.database: highprio
    blockioclass.resource-policy.nri.io/container.logger: lowprio

What's Changed

• balloons: do not require minFreq and maxFreq in CPU classes by @askervin in #455
• balloons: expose balloons and optionally containers with affinity in NRT by @askervin in #469
• balloons: introduce loadClasses for avoiding unwanted overloading in critical locations by @askervin in #493
• topology-aware: exclude isolated CPUs from policy-picked reserved cpusets. by @klihub in #474
• topology-aware: rework building the topology pool tree. by @klihub in #477
• topology-aware: allocate burstable container memory by requests. by @klihub in #491
• topology-aware: better semantics for globally configured shared CPU preference. by @klihub in #498
• topology-aware: more consistent setup error handling. by @klihub in #502
• memtierd: allow overriding go version for image build. by @klihub in #456
• resmgr: improve annotated topology hint control. by @klihub in #499
• resmgr: eliminate extra container state 'overlay'. by @klihub in https://github.com/containers/nr...