GPUs on a node

bcd manages NVIDIA GPUs only, and it talks to them through one tool: nvidia-smi, which ships with every NVIDIA driver. There is no NVML library binding in the binary. This page follows a GPU through bcd: how it is read, how it enters the inventory, how a GPU passed through to a VM stays there, how bcd builds the CDI specs that hand GPUs to containers, what it does with persistence mode, XIDs and resets, and what is different on WSL 2. Holding GPUs for a reservation is on devices and holds; the host units that install the driver, partition MIG and set up MPS are on host configuration; what the control plane does with GPU names, atoms and health is on inventory and health.

Where GPU facts come from

A host build of bcd collects the inventory in two layers (collectInventory in daemon/cmd/bcd/hardware_host.go):

1. procfs and sysfs. inventory.Collector lists /proc/driver/nvidia/gpus/*/information: each GPU's UUID, model, VBIOS and bus id, its NUMA node from /sys/bus/pci/devices/<bus>/numa_node, a driver version from /proc/driver/nvidia/version, and an architecture guessed from the model name. The GPUs are indexed in bus order, as nvidia-smi numbers them by default.
2. nvidia-smi. nvml.SMI.Snapshot reads everything else, within one minute for the whole read, and nvml.Enrich merges it into the first layer by UUID (below).

A host without nvidia-smi on the PATH has no NVIDIA driver to manage: the snapshot fails with nvml: nvidia-smi not found (no NVIDIA driver on this host) and the inventory keeps whatever procfs showed. Only the second layer fills the CUDA driver version, so bcd uses that field to tell the two apart: without it, Hello does not advertise the nvml capability and diag.nvml fails its nvml.init check.

The nvidia-smi queries

A snapshot runs these, each bounded to 30 seconds (SMI.Timeout; a wedged GPU can hang nvidia-smi):

Only the -q -x query is required. An optional query that fails leaves its fields empty, which is how a driver that answers "not supported here" with a non-zero exit is handled: topo -m exits 3 on WSL and mig -lgip exits 6 on a GPU without MIG, and the captured RTX 5090 fixtures replay exactly that.

What the parser tolerates

nvidia-smi's XML changes between driver branches and SKUs. The parser in daemon/internal/nvml/smi.go reads these cases, most of them found on the real RTX 5090 (0915ab7):

Live samples

The inventory holds facts that only change with the hardware: identity, memory size, ECC mode and counters, row remapping, MIG layout, NVLink, topology. Usage is a sample, and never enters the inventory hash, so a busy GPU does not resend its section every heartbeat.

A full snapshot runs nvidia-smi up to six times and -q -x walks every field of every GPU, which takes seconds of driver time on an 8-GPU board. Since 9a6fee9 the readers that sample often use SMI.Samples instead: one CSV query for exactly a sample's fields, plus one for the compute processes.

nvidia-smi --query-gpu=uuid,memory.total,memory.used,memory.reserved,memory.free,utilization.gpu,utilization.memory,temperature.gpu,power.draw,power.limit,clocks.sm,clocks.mem,pstate,mig.mode.current --format=csv,noheader,nounits
nvidia-smi --query-compute-apps=gpu_uuid --format=csv,noheader

Samples falls back to the full snapshot when a GPU is in MIG mode (a MIG device's memory is in -q -x only), when the driver refuses a field it predates (memory.reserved before R510), or when a line has the wrong number of columns. A host without nvidia-smi gets ErrNoDriver, not a fallback.

A sample's process count is compute processes only: in the -q -x path, entries whose type contains C (C, C+G, M+C for MPS clients); in the CSV path, the lines of --query-compute-apps. A desktop compositor or WSL's Xwayland is a graphics process and no workload. The live test checks that both paths agree on the real GPU.

The inventory merge

nvml.Enrich(base, snapshot) merges by UUID:

• The section's driver, CUDA driver and NVML versions are replaced by the snapshot's when it has them.
• A GPU both layers know takes every field from nvidia-smi, but keeps the collector's index, and the collector's NUMA node and architecture when nvidia-smi gave none (it reports no NUMA node at all).
• A GPU only nvidia-smi knows is appended with the next index. That is the WSL 2 case: the paravirtual GPU has no /proc/driver/nvidia entry and no PCI sysfs, so the whole section comes from nvidia-smi, its bus id names a virtual bus (00000000:F1:00.0 in the fixture) and it has no NUMA node.

The result is canonicalized and hashed like every other section. bcd collects the inventory again every 10 minutes (DefaultInventoryInterval), since ECC counters, row remapping and MIG state change without an event bcd sees, about 2 seconds after an XID burst, after a host unit changed something, and on inventory.refresh.

GPUs passed through to VMs

A GPU a VM holds is bound to vfio-pci. The NVIDIA driver lets go of it, so neither /proc/driver/nvidia nor nvidia-smi lists it any more, and an inventory read from them would drop the very GPU the VM uses: the control plane would retire its device mid-allocation, and a VM restarting would no longer find its GPU's PCI address. Since c5c52c4 bcd keeps them:

• Remember. After every enriched collection, inventory.GPUMemory records each GPU by its PCI address in <data_dir>/gpus-seen.json (0600, written atomically): UUID, name, architecture, VRAM, serial, VBIOS, board part number and compute capability. GPUs the collection did not list are kept, since they may be passed through right now.
• Find. Collector.PassedThrough walks /sys/bus/pci/devices for NVIDIA functions (vendor 0x10de) of class 0x0300xx (VGA) or 0x0302xx (3D controller) bound to vfio-pci that the driver did not list, and reports the GPU last seen at that address, with its NUMA node from sysfs.
• Order. They come after the driver's GPUs, so the driver-bound GPUs keep the indices nvidia-smi gives them.
• Limits. A function never seen on the NVIDIA driver is left out: without a UUID there is nothing to name it by. A remembered GPU carries only what stays true while a VM has it, not MIG, NVLink or topology state, which the guest's driver owns meanwhile.

Only bcd node keeps the memory (newHost and newRuntimes point it at the data directory). bcd hardware and bcd doctor collect without it, so they do not list a GPU that is passed through at that moment. The runtimes' GPUAddress, which resolves a GPU's UUID to its PCI address for VFIO, reads through the same memory.

CDI: NVIDIA's spec and bcd's

Containers get GPUs through CDI specs in /etc/cdi. Two kinds live there on a bcd GPU host:

With the toolkit in CDI mode, NVIDIA_VISIBLE_DEVICES grants nothing, and NVIDIA's own device names are not tied to any hold. So since 6ecb860 bcd copies NVIDIA's edits into each hold's spec (agent/cdivendor.go):

• bcd reads every *.json spec of kind nvidia.com/gpu in /etc/cdi, then /var/run/cdi, skipping its own bcd-* files. A YAML spec (the GPU operator's, or the nvidia.yaml older bcd versions wrote) is not read, and a hold then carries only NVIDIA_VISIBLE_DEVICES, as before 6ecb860.
• A hold's spec gets the vendor spec's common edits (device nodes such as /dev/nvidiactl and /dev/nvidia-uvm, library mounts, hooks, additional GIDs) once, without the vendor's NVIDIA_VISIBLE_DEVICES=void, and each held device gets exactly the edits of the device it stands for: a whole GPU or a slot its GPU's, a MIG instance its MIG-… device's. Another GPU's device node never leaks in.
• The hold spec's cdiVersion rises to the vendor's when that is later (additional GIDs need 0.7.0).
• A held device the vendor spec does not name is logged ("NVIDIA's CDI spec does not name held devices"): the spec is stale until the toolkit unit regenerates it, which it does whenever the driver version or nvidia-smi -L changes.

bcd.dev/gpu=<id> alone is therefore all a container needs. A Kubernetes binding is left alone: its spec carries no vendor edits, since the kubelet device plugin names NVIDIA's devices itself. The spec's full contents are on devices and holds.

Persistence mode around passthrough

The nvidia-driver unit keeps nvidia-persistenced running, which holds every GPU open in persistence mode, and bcd doctor warns when persistence is off. But the NVIDIA driver's unbind waits until a GPU is closed, so on a converged GPU host every passthrough start hung until the unbind timed out. Since f25abbe and 5bb9f22 bcd turns persistence off for exactly the GPUs a VM takes, and on again once they are back on nvidia, with nvidia-smi -i <pci address> -pm 0 or -pm 1:

Every call is best effort: a failure is ignored, and the unbind still has its own 30-second timeout (SysfsVFIO.UnbindTimeout), whose error names the likely holder: its driver has not let go after 30s: something still has the device open (nvidia-persistenced, a process on the GPU). Tests inject the persistence call; nothing runs the real nvidia-smi under a fake virsh or a fake sysfs.

No passthrough has run on real hardware, so this sequence is tested against fakes only.

XIDs and GPU health

XIDs from the kernel log

bcd watches XIDs for the life of the process (watchXids, daemon/internal/health). It opens /dev/kmsg, seeks to its end, and parses every record whose text matches the driver's line:

NVRM: Xid (PCI:0000:18:00): 79, pid=1234, name=python, GPU has fallen off the bus.

The bus id is widened to NVML's form (00000000:18:00.0) and matched against the inventory's GPUs. A match raises the device condition Xid<n> (status true, device_id the GPU's UUID, the driver's text as message), one per XID number and GPU; an XID on a bus the inventory does not know is only logged. Each XID also:

• asks for an inventory collection about 2 seconds later (a burst is read once), since the GPU's ECC and row-remap counters move with it;
• triggers the Kubernetes side, whose device plugins mark the GPU unhealthy and whose DRA slice drops it;
• re-sends the status report at once, with the condition.

The conditions are kept for the rest of the boot in <data_dir>/faults.json (0600) with the boot id, so a bcd restart in the same boot reports them again although it reads the kernel log from its end; the first start after a reboot finds another boot id and drops them. A gpu.reset that leaves the GPU healthy drops that GPU's (662fb55). What the control plane does with each XID (quarantine, NoSchedule, remediation) is on inventory and health.

PLAN.md's NVML event stream is not implemented: XIDs come only from the kernel log, and SXid lines from NVSwitches are not parsed.

Healthy

gpuHealthy is the one rule bcd uses for "may this GPU take work":

• no failed or pending row remap;
• no uncorrectable volatile ECC errors;
• no Xid* condition for the GPU this boot.

bcd status prints UNHEALTHY for a GPU that fails it, and the Kubernetes device plugins and DRA driver offer it as unhealthy.

diag.nvml

diag.nvml (control lane, roles node and ramdisk, attested result) collects the inventory again and checks the GPUs it lists, or only those in gpu_uuids:

The result's passed is true only when every check of every GPU passed.

gpu.reset

gpu.reset runs nvidia-smi --gpu-reset -i <uuid> per GPU, through bcd helper's gpu.reset operation when the node is unprivileged. It refuses the whole command with gpu-busy when a listed GPU, or one of its MIG instances or slots, has a live hold or a pending Kubernetes release, or is passed through to a VM; it answers unknown GPU for a UUID the inventory does not list; and after the resets it collects the inventory again and marks each reset GPU healthy unless it shows uncorrectable volatile ECC errors or a pending or failed row remap. The full rules are on delivery.

WSL 2 paravirtual GPUs

Under WSL 2 a GPU is a paravirtual device: the Windows host's driver serves it through /dev/dxg, CUDA works through the libraries in /usr/lib/wsl/lib, and the Linux side has no NVIDIA kernel module, no /dev/nvidia* and no PCI function for the GPU. bcd recognises it as /dev/dxg without /dev/nvidiactl (paravirtualGPU in host config, the gpu-host check in doctor). It is the machine round 6 ran on.

The container-toolkit behaviour on WSL is tested against a fake host only: the WSL machine round 6 used has no nvidia-ctk, so no WSL CDI spec was generated for real. TestRealGPUUnitsPlan planned the unit there and saw it plan the toolkit install, the runtime configuration and /etc/cdi/nvidia.json.

bcd doctor's GPU checks

bcd doctor reads every GPU once with the full snapshot and prints these lines (f9d7af9, f84f1f9, da60bbd). The real-host checks around them (systemd, cgroups, KVM, VFIO, IOMMU groups, the clock) are on running bcd on real hardware.

gpu-bus compares NVIDIA display functions (class 0x03xxxx) in /sys/bus/pci/devices with the GPUs nvidia-smi lists. A fakehw build skips every host check, since it has no real bus to compare.

When the snapshot fails, gpuErrorHint adds what to do for the failures seen on real hosts:

The NVML backend

daemon/internal/nvml is named for NVML, but no build links it today. Build tags pick what nvml.Probe (doctor's nvml check) does; the inventory and samples go through nvml.SMI in every host build:

Both host builds need no cgo and never load libnvidia-ml. The costs of going through nvidia-smi are that every read spawns a process (bounded to 30 seconds), and that there are no NVML events: XIDs come from the kernel log instead.

Tests and fixtures

# On a host with an NVIDIA GPU (scripts/daemon-test.mjs forwards BCD_* variables into WSL on Windows):
BCD_REAL_GPU=1 node scripts/daemon-test.mjs --run TestSnapshotLiveHost ./internal/nvml/
BCD_REAL_GPU=1 node scripts/daemon-test.mjs --run TestRealGPUUnitsPlan ./internal/hostconfig/