Devices and holds

A GPU reservation reaches a machine as one entry of its NodeSpec's devices section: a DeviceAssignment per allocation. bcd's devices converger (daemon/internal/agent/devices.go) turns each into a hold: a CDI spec and an env file a workload manager on the node consumes, with a cgroup slice when the reservation booked host CPU or memory. It reports every bind and release as a node-signed event, refuses what it cannot honour with a reason, and never drops a hold on its own. Delivery summarises it among the other sections; this page is the full account. The control plane's side (placing, booking, binding states) is in reservations, atoms and bookings, fractional GPUs and Kubernetes workers.

The assignment

instance_id (for BINDING_INSTANCE) is not read: GPUs for VMs arrive in the instances section, not here.

Bindings

Atoms

Every device is an atom, resolved against the node's current inventory (resolveAtom, holds.go). The ids are the ones inventory gives:

A MIG instance the control plane booked but the mig-layout unit has not created yet is device-missing (GPU-… has no MIG instance gi1/ci0), as is a GPU the inventory does not list. A malformed MIG or slot id, or another class (nic.vf, disk), is unsupported-device. Slots are a bookkeeping unit: the driver sees the whole GPU, and MPS or time-slicing shares it (see gpu-sharing).

One pass

The converger plans the whole section on every pass (a new spec, the 30-second retry, the 10-minute drift pass). It reads the holds that exist from /etc/cdi/bcd-*.json: a hold is its CDI spec, which survives a reboot, and the real atom ids are read back from each device's bcd.dev/device-id annotation. Then, in this order:

1. Kubernetes releases pending from earlier passes are reported once no pod holds them, or dropped unreported when the spec wants the allocation again (below).
2. Holds no longer wanted are released: their CDI spec and env file removed, then AllocationReleased. A plain hold's slice is stopped once it is empty; one still running a workload is kept with its limits.
3. Leftover slices whose hold is neither wanted nor released in this pass are stopped again once empty (e01d7ff), so a slice whose workload outlived its release goes when that workload exits.
4. Holds still wanted are kept. Their slice is created again when it is missing (systemd's runtime properties vanish on a reboot; the hold does not), and a missing env file is rewritten (it lives on tmpfs).
5. New holds, in spec order, are checked: binding, privilege, atoms, conflicts, host admission. Each that passes is bound; each that fails is collected.
6. Failures, if any, are reported as one AllocationFailed per allocation, and the section fails terminally.

Releases come before binds, so a spec that moves a GPU from one allocation to another in one generation works. Within the pass, every atom a new hold claims counts against the holds after it.

Conflicts

A new hold is refused with overlap (holdConflict) when any of its atoms:

"Held" covers the live holds, the holds bound earlier in the same pass, and the devices of Kubernetes releases still pending. While the releases file is unreadable, every GPU of the node is held by (pending Kubernetes releases that could not be read). The VFIO side counts the GPUs in the current instance specs and those running domains had at the last pass, so a GPU dropped from a running VM's spec stays the VM's until its next start. The other direction holds too: a VM start, gpu.reset, diag.dcgm from level 2 and the mig-layout unit refuse a GPU a live hold or a pending release names (see gpu.reset and live holds).

Hold slices

A hold whose assignment carries limits runs in its own cgroup-v2 slice, created before the CDI spec names it (260e33c; daemon/internal/slice). A BINDING_KUBERNETES hold never gets one: its pods live in the kubelet's kubepods.slice, where the kubelet enforces their requests and limits (e01d7ff).

bcd does not move processes into the slice. It names it in the spec's bcd.dev/cgroup-parent annotation, in BCD_CGROUP_PARENT and in AllocationBound.cgroup_parent, and the workload manager places the workload there. Under bcd helper the slice is slice.apply and slice.remove, which accept only such a name and cpulists. A fakehw host keeps each slice as a JSON record under /run/bcd/slices instead.

Host admission

Since 31555bf a new hold whose limits carry a memory limit or huge pages is admitted against the host first (admission.go), the same way an instance start is, so VMs and holds never take what the other was promised. A hold with only cpus or cpu_millis is not admitted. The host is read once per pass, when the first such hold needs it:

A new hold must fit its memory_bytes in what normal memory is left, and its huge pages on its mems nodes (the whole host without mems), never more than the host-wide count. One that does not fit fails with resources-exhausted:

ResourcesExhausted: 4096 MiB of memory needed, 1024 MiB left on this host
ResourcesExhausted: 8 hugepages of 1048576 KiB needed, 2 free

A host whose memory cannot be read binds without the check and logs host memory unreadable; binding without host-side admission. A fakehw host is never admitted. The page sizes, pools and the instance side are on host configuration and delivery.

What a bind writes

A bind runs, in order: the slice (when sliced), the CDI spec, the env file, then the AllocationBound event. A failure of any write is retryable: the section stays PROGRESSING, the pass is tried again in 30 seconds, and no event is sent until every file is in place. Under bcd helper the node asks root for a file.write of kind cdi or allocation-env with the allocation id, and the helper builds the path itself; a refused operation is retried the same way.

The CDI spec

/etc/cdi/bcd-<allocation>.json, mode 0644, written atomically. A BINDING_HOLD on one whole GPU with limits, on a host whose toolkit wrote /etc/cdi/nvidia.json (vendor mounts and hooks shortened):

{
  "cdiVersion": "0.7.0",
  "kind": "bcd.dev/gpu",
  "annotations": {
    "bcd.dev/allocation-id": "alloc_01jz8k2v",
    "bcd.dev/reservation-id": "rsv_01jz8k1q",
    "bcd.dev/generation": "42",
    "bcd.dev/cgroup-parent": "bcd-alloc-alloc_01jz8k2v.slice"
  },
  "devices": [
    {
      "name": "GPU-5f1c2a90-0000-4000-8000-000000000001",
      "annotations": { "bcd.dev/device-id": "GPU-5f1c2a90-0000-4000-8000-000000000001" },
      "containerEdits": { "env": null, "deviceNodes": [{ "path": "/dev/nvidia0" }] }
    }
  ],
  "containerEdits": {
    "env": [
      "BCD_ALLOCATION_ID=alloc_01jz8k2v",
      "BCD_CGROUP_PARENT=bcd-alloc-alloc_01jz8k2v.slice",
      "NVIDIA_VISIBLE_DEVICES=GPU-5f1c2a90-0000-4000-8000-000000000001"
    ],
    "deviceNodes": [{ "path": "/dev/nvidiactl" }, { "path": "/dev/nvidia-uvm" }],
    "mounts": [{ "hostPath": "/usr/lib/x86_64-linux-gnu/libcuda.so.580.65.06", "containerPath": "…", "options": ["ro", "nosuid", "nodev", "bind"] }],
    "hooks": [{ "hookName": "createContainer", "path": "/usr/bin/nvidia-cdi-hook", "args": ["nvidia-cdi-hook", "update-ldcache", "…"] }],
    "additionalGids": [44]
  }
}

How bcd reads NVIDIA's spec, and what happens without one or with a stale one, is on GPUs on a node. Without a JSON vendor spec a hold carries only the env, which works with a toolkit in its legacy mode but grants nothing in CDI mode.

A container then asks for the devices by name, for example --device bcd.dev/gpu=GPU-5f1c2a90-… with Podman or Docker's CDI support, or the same name in containerd's CDI device list.

The env file

/run/bcd/allocations/<allocation>.env, mode 0644, one KEY=VALUE per line, for a workload manager that reads env files rather than CDI:

BCD_ALLOCATION_ID=alloc_01jz8k2v
BCD_CGROUP_PARENT=bcd-alloc-alloc_01jz8k2v.slice
BCD_RESERVATION_ID=rsv_01jz8k1q
NVIDIA_VISIBLE_DEVICES=GPU-5f1c2a90-0000-4000-8000-000000000001

The lines are BCD_ALLOCATION_ID, BCD_CGROUP_PARENT (sliced holds), CUDA_MPS_PIPE_DIRECTORY (slots with MPS running), BCD_RESERVATION_ID, NVIDIA_VISIBLE_DEVICES, then the assignment's env lines. Unlike the CDI spec, the env file carries NVIDIA_VISIBLE_DEVICES for a Kubernetes hold too. /run is tmpfs, so after a reboot the next pass writes it again from the hold's spec.

Events

Every bind and release is an AttestedReport: the body (with the node id and boot id) is signed with the node key under purpose attested-report, so a gateway cannot forge a binding. Failures are plain events. All three are journaled before they are sent and replayed until acknowledged (see events).

With attested bindings (on by default with command signing), a reservation turns active only on the node's own signed AllocationBound.

Failure reasons

The checks run in that order for each new hold, and the first that fails decides. Once the failures are reported, the section is FAILED terminally at that generation (its message lists <allocation>: <reason> for each), so observed_generation still advances and the other sections converge. The holds of the same pass that passed their checks are bound anyway. A later generation that still carries a failed assignment reports it again.

On the server, the gateway's allocationFailed handler marks the allocation failed (markAllocationFailed), which wakes the reservation at once instead of at the bind timeout: it gets the condition BindTimeout with reason NodeRejectedBinding, its bookings are released and it is placed again, up to two times (maxBindRetries), after which it fails. A node lease (binding: 'node') fails at once. See states and conditions.

Unprivileged nodes

Writing /etc/cdi and creating slices is root's work. A host build of bcd node running as an ordinary user without [helper] (Options.Unprivileged: not a fakehw build, mode node, effective uid not 0, no [helper] socket) can do neither. Before da60bbd such a node retried every hold every 30 seconds on systemctl's "Access denied" and stayed binding forever, logging nothing above debug. Now:

• Hello claims nothing. A GPU host normally advertises the capabilities cdi, devices.limits and binding.kubernetes (and nvml when nvidia-smi answered). An unprivileged node advertises none of the first three. The gateway writes capabilities as bc.dev/cap.<name> system labels, and capacity requires bc.dev/cap.devices.limits for a reservation that books host resources and bc.dev/cap.binding.kubernetes for a Kubernetes one (requiredCapabilities), so neither is ever placed there.
• A plain hold fails at once. A GPU reservation that books no host resources requires no capability label and can still be placed on the node. bcd fails it straight away with node-unprivileged, writes nothing, and the control plane re-places it as above.
• Doctor says so. bcd doctor warns gpu-binding on a host with GPUs whose bcd would run this way, and section failures are now logged (WARN while retrying, ERROR when failed).

Run bcd node as root, or as the unprivileged bcd account beside a root bcd helper (see running bcd node unprivileged); both bind holds. TestAnUnprivilegedNodeTakesNoHolds covers the capabilities and the refusal, and the real-host e2e checks on a real GPU host that the node carries no bc.dev/cap.devices.limits label and that a GPU reservation never becomes binding or active there.

Kubernetes releases

A plain hold is released in the pass that sees it gone. A Kubernetes hold cannot be: pods may still have its devices. So its release is split (kuberelease.go):

1. Recorded first. The release goes into <data_dir>/releasing.json (0600: device ids, generation, time) before anything is removed, so a crash in between never loses it.
2. Withdrawn. The CDI spec and env file are removed at once. The device plugins stop advertising the devices, and the kube reaper evicts pods still holding them.
3. Reported when free. Each pass asks whether any pod still holds a device of the allocation (the kubelet's pod-resources, and the host's pod processes for whole GPUs; unreachable counts as held). Only when none does is AllocationReleased sent and the record dropped. A pass is triggered when the reaper frees devices, with a 30-second recheck as a backstop.
4. Claimed meanwhile. Until reported, the devices stay claimed on the node: no new hold, VM, gpu.reset, diag.dcgm or mig-layout gets them, and the control plane keeps them booked.
5. Taken back. A spec that wants the allocation again drops the pending release unreported.

A release pending longer than 5 minutes raises KubernetesReleaseStuck (reason PodsHoldDevices). An unreadable releasing.json is renamed to releasing.json.unreadable-<unix time> (kept, never deleted), the marker <data_dir>/releasing.unreadable is written, every GPU of the node counts as claimed, and the node reports KubernetesReleasesUnreadable (reason ReleasesFileUnreadable). The marker goes once the kube side reports that no pod holds a bcd GPU of an allocation the node does not know, or when an operator removes it. What the control plane does meanwhile (carrying the booking, NoSchedule) is on Kubernetes workers and an unreadable releases file.

Seeing the holds

• bcd status shows, per GPU, the allocation holding it or one of its MIG instances or slots (see bcd status).
• ls /etc/cdi/bcd-*.json lists the holds, and /run/bcd/allocations/ their env files.
• systemctl status bcd-alloc-<allocation>.slice shows a sliced hold's cgroup and its processes.
• On the control plane, reservations.get returns the binding, with cgroupParent for a sliced hold (see bindings).

Tests