DRA Driver for CPU

Overview

Mon, 01 Jan 0001 00:00:00 +0000

dra-driver-cpu is a Kubernetes Dynamic Resource Allocation (DRA) driver that assigns exclusive CPUs to workloads through ResourceClaims. This driver provides an alternative to the CPUManager functionality implemented in the kubelet, offering additional benefits such as advanced topology selection through the rich DRA API and alignment with other DRA-managed resources (like GPUs and high-speed NICs).

Getting Started

The recommended way to install the driver is via the provided Helm chart:

helm install dra-driver-cpu oci://registry.k8s.io/dra-driver-cpu/charts/dra-driver-cpu -n kube-system

Important

Your cluster’s container runtime must support NRI and CDI — see Compatibility.
The kubelet’s CPUManager and this DRA driver are mutually incompatible, and only one can be enabled at a time on any given node — see Configuration for how to disable the CPUManager.

The Quickstart walks through installing the driver and running a pod on exclusive CPUs, with a verification step after each stage. See the Helm chart README for the full list of configuration options, and Installation for compatibility, upgrade, and uninstall details.

Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

This guide takes you from an empty cluster to a pod running on exclusive, pinned CPUs, with a verification step after each stage. It requires:

A Kubernetes 1.34+ cluster (with DRAConsumableCapacity enabled).
A container runtime with NRI and CDI enabled (containerd 2.0+ or CRI-O 1.30+).
The kubelet CPUManager disabled (cpuManagerPolicy: none) on driver nodes.

See Compatibility for the more details.

Create a test cluster

If you do not already have a compatible cluster, you can create a local kind cluster configured for DRA from a checkout of this repository:

Testing

Mon, 01 Jan 0001 00:00:00 +0000

Running the tests

Unit Tests

To run the node-local unit tests, run:

make test-unit

Chart render tests

To check what the Helm chart renders, without a cluster, run:

make helm-render-check

hack/ci/helm-render-check.sh renders each case and hands the DaemonSet to the validator in test/render. It needs Helm and the project’s Go toolchain, but no cluster. The cluster-side counterpart is the helm e2e workflow, which installs the chart against both the default and a relocated kubelet root.

Installation

Mon, 01 Jan 0001 00:00:00 +0000

The Quickstart walks through an install with a verification step after each stage, ending with a pod running on exclusive CPUs. This page is the reference: compatibility, runtime setup, security, uninstall, and migration.

Compatibility

Requirement	Minimum
Kubernetes	1.34 (DRA `resource.k8s.io/v1` is GA and enabled by default)
Container runtime	containerd 2.0 or CRI-O 1.30 (NRI and CDI enabled by default)
Kubelet	CPUManager disabled: `cpuManagerPolicy: none` — see Configuration

Optional features need additional cluster feature gates:

Linting

Mon, 01 Jan 0001 00:00:00 +0000

Run the linter against the codebase:

make lint

To automatically fix lint issues, pass --fix via the GOLANGCI_LINT_EXTRA_ARGS variable:

GOLANGCI_LINT_EXTRA_ARGS=--fix make lint

GOLANGCI_LINT_EXTRA_ARGS is forwarded verbatim to golangci-lint run, so any other supported flags can be passed the same way.

Logging Guidelines

Mon, 01 Jan 0001 00:00:00 +0000

picking a verbosity level

V(0): high-signal operational logs and errors always visible at default verbosity V(2): normal operation, important decisions, summary, container lifecycle V(4): allocation internals, fine details about decisions V(6): debug messages, very fine details, mostly for developers

V(7) or more is reserved for future usage.

when to use Error()

Summary: use Error() if we need to log an unrecoverable error which leads to a failed state, not to a degraded-but-operational state.

Upgrading

Mon, 01 Jan 0001 00:00:00 +0000

This document detail the upgrade actions that are specific to releases of the CPU DRA driver. It complements the general installation and configuration documentation.

Upgrades must preserve the singleton ownership invariant. An upgrade may temporarily leave a node without an active driver, but it must not create overlapping active owners.

Upgrade from 0.3.0 to 0.4.0 (EARLY WARNING)

Before the driver enters beta stage, a batch of breaking changes are planned. The idea is to coalesce together all the known backward incompatible changes in release 0.4.0. We advice users wishing to prepare early to track the linked issues.

Configuration

Mon, 01 Jan 0001 00:00:00 +0000

This page covers configuring the driver itself, the kubelet prerequisites required to run it, and the command-line flags currently being deprecated in favour of the configuration file.

Driver configuration

The driver supports two configuration mechanisms:

The config file - the main, preferred way to configure the driver.
Command-line flags — kept for a small set of important, mostly behavior altering settings and for backward compatibility. Some flags are being deprecated in favor of their config file equivalent (see Helm: deprecated args values vs driverConfig below).

If the same setting is provided both ways, the explicit command-line flag wins, so avoid mixing the two for the same field.

Configuration Guidelines

Mon, 01 Jan 0001 00:00:00 +0000

The configurable behavior of the DRA CPU driver can be changed using two approaches:

command line flags
configuration file

The preferred way to add or extend the configurable behavior is through the configuration file. The reasons why we prefer this approach are:

user friendliness: adding options quickly makes the command line long and awkward
auditable, reviewable artifact: the configuration file can be versioned, compared and verified more easily
extensibility: it’s usually easier to process configuration files with programs than command line options
configuration files are easier to version and evolve compatibly than command line flags.

Configuration values are applied incrementally:

Deep Dive: Cgroup v2, QoS Classes, and CPU Enforcement in Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

This document provides a technical deep-dive into how Kubernetes organizes Linux control groups under cgroup v2, calculates and applies Quality of Service (QoS) classes, manages cgroup hierarchies at node, QoS, pod, and container levels, and enforces CPU allocations. It also documents the architectural relationship between Kubelet’s cgroup management and out-of-tree Dynamic Resource Allocation (DRA) drivers such as dra-driver-cpu.

1. The Linux Cgroup v2 Hierarchy in Kubernetes

Under cgroup v2 with the systemd cgroup driver, Kubernetes organizes container processes into a strictly hierarchical tree under /sys/fs/cgroup/kubepods.slice:

How it Works

Mon, 01 Jan 0001 00:00:00 +0000

The driver is deployed as a DaemonSet. A single executable implements the DRA and NRI interfaces, and it owns the node-local CPU allocation state. Exactly one active driver instance is expected to manage each node. The driver manages all the available CPUs except those configured as reserved. The driver manages a single CPU pool, and partitioning that pool is currently unsupported. The singleton invariant is not currently enforced by the driver and must be guaranteed by the deployment.

Feature Support

Mon, 01 Jan 0001 00:00:00 +0000

Currently Supported

Exclusive CPU Allocation: Pods that request CPUs via a ResourceClaim are allocated exclusive CPUs based on the chosen mode and topology.
Shared CPU Pool Management: All other containers without a ResourceClaim are confined to a shared pool of CPUs that are not reserved.
Topology Awareness: The driver discovers detailed CPU topology including sockets, NUMA nodes, cores, SMT siblings, L3 cache (UncoreCache), and core types (Performance/Efficiency).
Advanced CPU Allocation Strategies: When in "grouped" mode, the driver utilizes allocation logic adapted from the Kubelet’s CPU Manager, including:
- NUMA aware best-fit allocation.
- Packing or spreading CPUs across cores.
- Preference for aligning allocations to UncoreCache boundaries.
Grouped Device Exposure: NUMA/socket/machine aggregates are exposed as consumable capacity, reducing API objects and scaling to large systems. The driver also support a individual device exposure mode which is deprecated; see Migrating from individual mode.
Device Health Reporting: The driver reports per-device health to the kubelet via the DRA WatchHealthStatus gRPC API, reflected in pod.status.containerStatuses[].allocatedResourcesStatus. Devices are reported Healthy and go Unknown if the driver stops sending updates. Unhealthy is reserved for future use. See Device Health Reporting.

Not Supported

Admin Access is not supported. A ResourceClaim allocated with adminAccess: true for a CPU device will fail claim preparation.
This driver currently only manages CPU resources. Memory allocation and management are not supported.
While the driver is topology-aware, the grouped mode currently abstracts some of the fine-grained details within the group. Future enhancements may explore combining consumable capacity with partitionable devices for more hierarchical control.

This driver strictly enforces a 1-to-1 mapping between Claims and Containers. It does not support sharing a single ResourceClaim among multiple containers or multiple pods, if that claims includes a resource (dra.cpu) managed by this driver. Attempting to share a claim among containers or pods will make all but the first pod consuming the claim to fail to start with the error CreateContainerError and remain in Pending state. When the driver runs with publishNodeAllocatableResourceMapping, sharing across pods is rejected earlier by kube-scheduler: the second pod stays unschedulable with the message node allocatable resource claim ... has a mapped device and cannot be shared across pods.

Deep dive: how linux reports topology attributes

Mon, 01 Jan 0001 00:00:00 +0000

disclosure: this document was created with AI assistance. The research was human-steered by LLM-operated; the document was then reviewed by humans for correctness and edited extensively.

This document assumes kernel code, architecture and hardware management knowledge.

This document tries to be architecture agnostic whenever possible, but the author can only claim sufficient knowledge about the x86_64 architecture. The reader must assume the “silicon” and hardware is assumed to be x86_64 unless clearly specified otherwise.

Deep dive: finding and exploring PCI/PCIe root buses

Mon, 01 Jan 0001 00:00:00 +0000

disclosure: this document was created with AI assistance. The research was human-steered by LLM-operated; the document was then reviewed by humans for correctness and edited extensively.

This document assumes kernel code, architecture and hardware management knowledge.

Workload Configuration Requirements

Mon, 01 Jan 0001 00:00:00 +0000

Currently, Kubernetes has two separate systems for requesting CPU resources: standard requests in pod/container fields (pod.spec.resources or pod.spec.containers[].resources) and DRA ResourceClaims.

The Kube-scheduler uses different plugins to account for these requests, and these plugins are mutually independent. This can lead to node CPU overcommitment because the scheduler might not have a complete picture of all allocated CPUs.
Kubelet only considers the standard CPU requests in the PodSpec for critical node-level enforcements like QoS class assignment and cgroup hierarchy setup, ignoring CPUs allocated via DRA claims.

Custom Opaque CPUSet Allocation Overrides

Mon, 01 Jan 0001 00:00:00 +0000

Migrate from individual mode

To keep scheduler-controlled selection of exact CPUs that the deprecated Individual mode enabled, use grouped mode with the external allocator. Note that also NUMA-node or Socket grouping supports external allocator like the Machine grouping. All grouping modes publish the external-allocator attributes, but the NUMA-node or Socket grouping also retain a useful topology boundary:

driverConfig:
 cpuDeviceMode: grouped
 groupBy: numanode # or socket
 allocator: external

The external allocator reads the selected grouped device’s external-allocator attributes, selects CPUs from that group, and writes the cpuset in the opaque configuration described below. The driver validates and enforces the selection. groupBy: machine is also supported, but is less preferred because it removes that topology boundary; use it only when the allocator must select from the entire node.

SMT layout encoding

Mon, 01 Jan 0001 00:00:00 +0000

dra.cpu/smtLayout is a compact representation of logical-CPU sibling pairs.

We reserve the option to version the encoding, adding new versions with the Vxxx suffix appended. The reason for this decision is because, albeit aiming to be correct and completed, the encoding format didn’t get sufficient testing and validation to be called confidently final. Therefore we version to enable smooth future upgrades. Users should consider then unversioned name dra.cpu/smtLayout effectively as synonym for V1. The V1 format represents only two-way SMT; see the user-facing attribute reference for more details.

Metrics

Mon, 01 Jan 0001 00:00:00 +0000

The driver exposes Prometheus metrics on the existing HTTP /metrics endpoint served by --bind-address (default :8080).

Note

Driver custom metrics are ALPHA. They are useful for early observability, but metric names, labels, buckets, and semantics may change in future releases.

Custom driver metrics can also be listed programmatically without starting the driver:

dracpu introspect metrics

The command prints JSON metadata for custom dra_cpu_* metrics only. It does not include default Go runtime, process, or Prometheus client metrics.

Device Attributes and Selectors

Mon, 01 Jan 0001 00:00:00 +0000

Every CPU device the driver publishes carries topology attributes that claims can select on with CEL expressions. This page is the attribute reference, with worked selector examples and sample ResourceSlice objects for each device mode.

Note

Attribute names and semantics are not yet a stable API: they may still change between driver minor releases while the project is pre-1.0.

Attribute reference

Which attributes a device carries depends on the driver’s device mode (cpuDeviceMode in Configuration): grouped exposes one device per CPU group. individual exposes one device per CPU, but is deprecated; see Migrating from individual mode.

Troubleshooting & Diagnostics

Mon, 01 Jan 0001 00:00:00 +0000

The repository includes a diagnostic tool available as dracpu gatherinfo to collect node-local CPU topology and driver configuration information. The legacy dracpu-gatherinfo alias remains supported in versions 0.2.z and 0.3.z. This is helpful for debugging CPU allocation issues or validating that the host topology is discovered correctly.

`dracpu gatherinfo` Overview

dracpu gatherinfo is a node-local diagnostic tool for collecting CPU information from the host where it runs. The command entrypoint is thin; the collection logic lives in internal/gatherinfo and writes a YAML report with a top-level layoutVersion.

DRA Driver for CPU

Overview

Getting Started

Quickstart

Create a test cluster

Testing

Running the tests

Unit Tests

Chart render tests

Installation

Compatibility

Linting

Logging Guidelines

picking a verbosity level

when to use Error()

Upgrading

Upgrade from 0.3.0 to 0.4.0 (EARLY WARNING)

Configuration

Driver configuration

Configuration Guidelines

Deep Dive: Cgroup v2, QoS Classes, and CPU Enforcement in Kubernetes

1. The Linux Cgroup v2 Hierarchy in Kubernetes

How it Works

Feature Support

Currently Supported

Not Supported

Sharing resource claims

Deep dive: how linux reports topology attributes

Deep dive: finding and exploring PCI/PCIe root buses

Workload Configuration Requirements

Custom Opaque CPUSet Allocation Overrides

Migrate from individual mode

SMT layout encoding

Metrics

Device Attributes and Selectors

Attribute reference

Troubleshooting & Diagnostics

dracpu gatherinfo Overview

`dracpu gatherinfo` Overview