Host configuration

A node's NodeSpec has a host section: a list of config units, each a unit name and its settings as canonical JSON. bcd node applies them with its host converger (6c2343f): kernel modules and parameters, huge pages, the CPU governor and isolated CPUs, the IOMMU, the NVIDIA driver, Fabric Manager, GPU power and ECC settings, the MIG layout, GPU sharing, the container toolkit and a Kubernetes join. This page describes what each unit does to the machine, and what it refuses. The control plane's side of each unit, its settings manifest field by field, is the generated host config units reference.

Where the units come from

bcd does not choose units. The control plane builds the host section, and provision is the module that fills it (hostSpecFor in packages/provision/src/config-units.ts):

• A profile's hostConfig. When a workflow succeeds, the hostConfig of its profile becomes the node's units. A unit may carry when:, a label expression that turns it on per node. See profiles and images.
• The pool. policy.hugepages becomes hugepages, policy.kernelModules becomes kernel-modules, policy.cpu becomes cpu (09669d3), policy.iommu becomes iommu and policy.gpu becomes gpu-settings (e4b0c01), policy.kubernetes becomes k8s-join when it names the cluster (apiServer, caCertSha256 and joinTokenRef, all three or none; 7db110a), and a pool shared by mps or timeslice gets gpu-sharing (1f0f683), on every machine of the pool, adopted ones included. A pool's unit replaces the profile's unit of the same name.
• The MIG layout gate. mig-layout { from: 'pool' } takes the pool's policy.migLayout, or the layout the capacity module froze the node at while a layout change waits for its GPUs to be free (see the MIG layout gate).

The control plane validates each unit against its manifest and refuses an unknown unit with INVALID_INPUT. Since 37dcd02 the pools API runs the same checks on a pool's host policy when pools.create or pools.update writes it (provision registers them as a pool-policy extension), so a policy no node could take is refused there, not when a node's spec is built. The server writes the units to the wire sorted by name, each settings object as canonical JSON (wireNodeSpec). bcd decodes the settings again, strictly: an unknown field or trailing data is a terminal failure, so settings it does not understand are never half applied.

Order

bcd applies the units in a fixed order, whatever order the spec lists them in:

Modules load before the parameters they provide (net.bridge.* needs br_netfilter), memory and CPUs are set up before workloads need them, the driver comes before Fabric Manager, the GPU settings and the MIG layout, GPU sharing sits on the partitioned GPUs, the container toolkit comes after them so its CDI spec describes the GPUs as partitioned, and a cluster join is last.

Each unit that is not converged becomes one action, and that action plans the unit again when it runs, against the host as the units before it left it. So when kernel-modules loads br_netfilter, the sysctl unit in the same pass finds the net.bridge.* keys it could not find when the pass started.

Level-triggered

A unit's plan compares its settings with what the host shows now: the files bcd owns, /proc/sys, /sys, the loaded modules, the installed packages, nvidia-smi. It writes only what differs. A converged unit plans no steps, and a changed setting or a local edit is planned away the next time the unit runs: a sysctl someone changed by hand is set back, a deleted drop-in is written again, and every file bcd writes starts with a header saying local edits are replaced.

bcd runs a converge pass when it starts, when a new spec arrives, every 30 seconds (ConvergeRetry) while any section is PROGRESSING, after a few events of other sections (a failed firewall reload, a finished snapshot download) and, since 3c0fe02, every 10 minutes whatever happened (driftInterval). So a sysctl, module, service, CDI spec or mount changed behind bcd's back is repaired within 10 minutes, with no new spec.

After a unit applies, bcd refreshes its inventory, so the driver version, the MIG devices and the huge page pools it reports follow what the unit changed. Since 427787a the heartbeat right after such a change carries the inventory hashes too, so the gateway applies the new sections at once and capacity sees, say, reserved huge pages within a heartbeat instead of up to ten heartbeats later.

Status and failures

The host converger reports one section, host, in StatusReport (see convergence and status):

Since c21ef84 no unit holds back another. Every unit gets its turn in the pass, in order: a unit that fails, whether while it is planned or while it applies, is recorded and the pass goes on to the next unit. A last step then reports every failure together, each prefixed with its unit's name. The section is retryable while any failure is, and FAILED only when all of them are terminal. So a hugepages unit waiting for a reboot, or a mig-layout unit waiting for busy GPUs, no longer keeps nvidia-driver, gpu-sharing or container-toolkit from converging. A unit named twice fails the whole section, since bcd cannot tell which settings apply. The host section never holds back the other sections either: devices, instances and the rest converge on their own.

Three units report conditions besides the section state: FabricManagerDown and MPSDown, both with reason NotRunning, and k8s-join's KubernetesTaintsDrift (reason NodeRestriction, 6ef198b), as StatusReport.conditions with status true while the fault lasts, and the converger adds RebootRequired (below). The gateway's xidStatusHandler turns them into node conditions (c7613e1, 0aea5c0, 88219ea), written and published as node.condition.changed only when they change:

On a tenant-controlled node these are advisory: the condition is recorded, never a NoSchedule reason. See conditions.

Reboots

Some settings take effect only after a reboot, and bcd never reboots on its own. A unit in that case returns a RebootRequired error whose message says reboot the node; the unit stays retryable, so the section stays PROGRESSING, and the node reports the condition RebootRequired with status true, its reason the names of the waiting units (comma-separated) and its message their errors (at most 1024 bytes). A unit applied through bcd helper comes back as text, and the phrase is how the node still recognises it.

Nothing in the control plane reboots on this condition. An operator reboots the node with node.reboot, within the pool's disruption budget (under the v1.0 quorum, as part of an approved batch). After the reboot the next pass finds the setting in effect, and the condition goes.

What removing a unit does

Removing a unit from the spec stops bcd managing that concern. It does not undo it: a driver stays installed, a toolkit stays configured, huge page pools and their boot arguments stay, a MIG layout stays, Fabric Manager keeps running, GPU power limits and ECC modes stay, and a CPU governor stays. Five units clean up after themselves, because their state is only bcd's own files, and k8s-join hands the machine back:

Root's records

The units keep what they record about the host in /var/lib/bcd-hostconfig (hostconfig.StateDir), a directory only root creates and writes, never in bcd's data_dir (f8e13ef, 1bb78d6). Under bcd helper the unprivileged node owns data_dir, and a record there would let it plant a symlink and steer root's writes.

Whenever bcd runs as root (bcd node as root, or bcd helper applying a unit), every file a unit writes or removes is checked first (02485b3): each existing directory from / down to it must be a real directory, not a symlink, owned by root and writable by neither group nor others. /tmp, and anything the unprivileged node owns or could have created, is refused with <dir> is not a directory only root can write: bcd will not write as root under it. The helper's own file kinds are held to the same rule.

Under bcd helper

When bcd node runs unprivileged beside bcd helper, it still plans every unit itself, reading the host as its own account can. A unit that is not converged is sent to the helper as one host.unit request: the unit name, its settings (none when it left the spec and should be cleaned) and the GPUs that live holds use. The helper builds its own units, plans the unit again as root from the name and settings alone, with the unit's own validation, and applies it.

[host] units = false is honoured on both sides (3fa9021). bcd node then has no host converger and sends no host.unit, and bcd helper reads the key from its own config.toml and refuses every host.unit, applies and cleans alike, with host.unit: this host's configuration is managed outside bcd ([host] units = false in root's config.toml), writing nothing. Root's copy is the one that counts, so a compromised node that dropped the setting from its own view still gets no host change applied as root.

For the same reason, bcd helper running as root refuses to start unless its configuration file is a regular file (not a symlink) that root owns and no one else can write, in directories only root can write: <file> must be a regular file root owns that no one else can write. That file decides what root does for the node ([host] units, command_root_sha256, the join pin). The install script already writes it that way, root:bcd 0640 in /etc/bcd 0750 (see install); a hand-made config.toml owned by the bcd account, or writable by its group, stops the helper.

Busy GPUs are root's view (a894cd2). For mig-layout the helper ignores the busy GPUs the node sends and works them out itself: the GPUs with compute processes (nvidia-smi --query-compute-apps; a MIG device's UUID makes every GPU busy) and those the live holds' CDI specs name (/etc/cdi/bcd-*.json). A node that lied could otherwise have GPUs in use repartitioned under their tenants.

Signed specs

Since 03c023d the helper applies a unit only when a spec the control plane signed asks for exactly those settings. Every host.unit request carries the signed DesiredState body that last set the host section (a full spec, or a patch that sets host) and its worker signature; bcd node keeps that body in data_dir/host-intent.json (0600), where it is worth nothing without the signature. The helper takes the command-signing roots the signature must chain to from one of two places:

join_signing_key_pin = "sha256/..."       # always there: the install's out-of-band pin
command_root_sha256 = "<64 hex digits>"   # optional: pin the roots themselves

Both pins are given at install, never taken from the control plane or the node, and a malformed value stops bcd from starting. The helper refuses the request unless all of these hold:

• its roots check out: each is pinned in command_root_sha256, or they come from root's join record or a trust bundle signed by the pinned JoinService key whose version is not older than one the helper already accepted (an older one is a rollback). Once a presented bundle rotated the roots, leaving it out does not bring the record's older roots back: the node must present the current bundle (71ccf71);
• the signature verifies, from a certificate of the worker role, for a desired state;
• the body names this node. Since a966cd4 the node is the one root's own join record names: bcd join, run as root on a node with [helper] (the install script's join), writes the verified JoinResponse, as signed, to /var/lib/bcd-hostconfig/join-response.pb (0600, a root-only path) before the identity is written, and a failed record fails the join and leaves no identity. The helper takes the node id from that record, re-verified against root's join_signing_key_pin, so another node's signed spec is refused from the first request on. A record the pinned key did not sign refuses every host.unit; when root joins the machine again as another node, the binding moves and the generations start over. A node joined before a966cd4 has no record and keeps the older rule: the first signed spec the helper accepts fixes the node in /var/lib/bcd-hostconfig/helper-intent.json;
• its generation is not older than one the helper already accepted;
• the unit's settings are exactly the body's, and a clean is allowed only for a unit the body no longer names.

helper-intent.json also records the newest trust bundle version and that the node's specs are signed: from the first trust bundle with command-signing roots on, a host.unit without a signed spec, or with a bundle without roots, is refused, whatever the node sends. A body the helper already verified stays good after its short-lived worker certificate expires, so a spec unchanged for days keeps working on its first signature; a new body needs a valid one. A node and a helper of different versions fail closed, since the helper refuses the unknown request field.

The remaining caveat. Signed intent is root's decision since 71ccf71 (security review 2): before it, without command_root_sha256, it turned on only once the node itself sent a trust bundle with roots, so a node that never sent one got unchecked units. Now the pins or root's join record decide. What remains is an install where neither names command-signing roots: no command_root_sha256, and a join record without roots (a deployment that signs nothing, or a node joined before a966cd4 with no record). There the helper checks nothing until the node presents a bundle with roots, and applies what the unprivileged node asks for, as on an install without signed intent, bounded only by strict unit validation, the sysctl denylist, root writes only under root-only directories, and k8s-join only to root-pinned allowed_clusters. bcd doctor (as root, with [helper]) fails on such an install ("bcd helper applies host units unsigned unless the node itself presents a trust bundle"). Pinning command_root_sha256 closes it from the first request, and so does re-joining as root against a control plane that signs intent. daemon install-script writes the pins (59985ae): the digests given with --command-root-sha256, else the local PKI bundle's command-signing root and, during a rotation, the added one, and none when the bundle has no command signing; an upgrade by a script without them keeps the node's pins, and better-compute pki trust-version prints them. The unit checks stay in force either way: that is why the sysctl unit refuses keys that would run a program as root, and why a Kubernetes join under the helper needs root's own allowlist of clusters (see k8s-join). Programs a unit runs are bounded to 20 minutes each.

A second window, which node the specs must name, is closed since a966cd4 by root's join record. A node joined before a966cd4 has no join-response.pb and still binds to the first signed spec it presents, so a node compromised before that first host.unit could bind root to another node's specs. bcd doctor, run as root on a node with [helper], says which binding a node has (f77c6f9):

To pin a node that joined before the record, re-join it as root: since a1c1a46 bcd join --reset --yes moves the data directory's identity aside and joins, here with a new token (a tpm or provisioned re-join keeps the node id; see joining again with --reset). Re-running the install script is not enough: its join returns at once on a joined node and writes no record.

TestHelperHostUnit (e0fe5ec) covers the operation against a real helper socket and a fake host: a sysctl re-planned and applied as root with the busy GPUs passed through, kernel.core_pattern refused with nothing written, invalid names (../etc, Sysctl, empty, 65 characters) and an unknown unit refused, a clean that removes the drop-in, and a helper without host units refusing the operation. TestHostUnitNeedsTheSignedSpec covers the signed spec checks with pinned roots, and intent_bundle_test.go the trust-bundle path: a command-signing root rotation (add, switch, drop) on a node pinned only by the join key, a replayed old bundle, a bundle from another key and a rootless bundle. The agent's TestSignedHostSectionKeptForTheHelper covers the node keeping the body.

kernel-modules

Loads kernel modules now and at every boot, with their options. A sandbox or VPS host loads kvm_intel or kvm_amd, vhost_net and tun this way, and a host that passes GPUs through to VMs loads vfio-pci.

• Files. /etc/modules-load.d/bcd.conf lists the modules to load, sorted. /etc/modprobe.d/bcd.conf holds one options <module> <options> line per module with options, and is removed when there are none.
• Loading. A module not loaded (no /sys/module/<name>, dashes read as underscores) is loaded with modprobe -- <module>.
• Changed options. When a loaded module's parameter under /sys/module/<name>/parameters/ differs from the option (Y/N and 1/0 compare equal), bcd reloads it (modprobe -r, then modprobe) only when its refcnt is 0. A module in use, such as kvm_intel under running VMs, is never pulled from under them: its new options take effect the next time it loads, which in practice means the next reboot. bcd does not report that it is waiting.
• Refused options (a894cd2). vfio's enable_unsafe_noiommu_mode and vfio_iommu_type1's allow_unsafe_interrupts fail the unit terminally: they let a passed-through device reach host memory. Provision refuses them too (forbiddenModuleOptions, b6d5548; - and _ count alike).

sysctl

Kernel parameters, each a sysctl name with a scalar value: { "vm.swappiness": 10 }.

• Names match ^[a-z0-9_]+(?:\.[a-z0-9_-]+)+$ (the same check as provision's). Values are numbers, booleans (true is 1, false is 0) or single-line strings of at most 4096 characters; whitespace is collapsed, since the kernel shows multi-value tunables tab-separated.
• Files. /etc/sysctl.d/90-bcd.conf, one name = value line per key, sorted, for early boot. bcd also sets each live value under /proc/sys that differs. An empty object removes the file.
• Missing keys. A key the kernel does not have yet is retried, not refused: bcd applies the others, then reports this kernel has no <keys> (is the module that provides it loaded?) and the section stays PROGRESSING. Load the module with kernel-modules and the same pass sets it.
• Refused keys. Some keys are refused whoever sends them, because they run a program as root or lower the host's defenses, and bcd helper applies this unit as root on behalf of an unprivileged node. A refused key fails the unit terminally and nothing is written. Since 1f0f683 provision holds the same list (forbiddenSysctls), so a profile or pool naming one of these keys is INVALID_INPUT before it reaches a node.

Since a894cd2 some keys take only their hardened values: raising the defense is fine, lowering it fails the unit terminally (<key> = <value> is refused: it would lower the host's defenses). Provision holds the same rules (restrictedSysctls, b6d5548), so a profile or pool with another value is INVALID_INPUT before it reaches a node.

hugepages

The host's persistent huge page pools, the memory VMs back their guests with, and the transparent huge page policy.

A pool's policy.hugepages: { size: '2Mi' | '1Gi', count, perNuma? } becomes this unit with pages2m or pages1g.

• 2 MiB pages are allocated at runtime: bcd writes the count to nr_hugepages (the host's under /sys/kernel/mm/hugepages/hugepages-2048kB, or each NUMA node's under /sys/devices/system/node/node<N>/hugepages with perNuma) and reads back what the kernel allocated. A shortfall (memory fragmented or short) is retried.
• 1 GiB pages are tried at runtime too, which works on a freshly booted host. When the kernel cannot find that much contiguous memory, bcd reserves the pages on the kernel command line instead: with update-grub, a GRUB drop-in /etc/default/grub.d/90-bcd-hugepages.cfg adding hugepagesz=1G hugepages=<n> to GRUB_CMDLINE_LINUX, then update-grub; with grubby, grubby --update-kernel=ALL --args=... (removing the previous arguments first); with neither, the step fails. It records the reserved count in /var/lib/bcd-hostconfig/hugepages-1g-boot and reports ... the boot loader reserves them from the next boot: reboot the node, which is a RebootRequired. With perNuma the boot reservation is the host total (the count times the NUMA nodes), since the kernel spreads boot pages over the nodes. Once a boot reservation exists it is kept in step with the count, and pages1g: 0 removes it.
• A kernel without 1 GiB pages (no hugepages-1048576kB directory) fails the unit terminally unless pages1g is 0; so does a kernel without transparent huge pages when transparent is set, and perNuma on a kernel that shows no NUMA nodes.
• Mounts. With pages1g above 0 the VMMs need a hugetlbfs of 1 GiB pages at /dev/hugepages1G. Where systemd runs (3c0fe02), bcd writes the mount unit /etc/systemd/system/dev-hugepages1G.mount (What=hugetlbfs, Options=pagesize=1G, DefaultDependencies=no, Before=sysinit.target, WantedBy=sysinit.target), runs systemctl daemon-reload and systemctl enable dev-hugepages1G.mount, and systemctl starts it when /proc/mounts does not show it, so the mount is there from boot, before any VMM starts. Without systemd, bcd mounts it itself (mount -t hugetlbfs -o pagesize=1G), again after each boot. 2 MiB pages use systemd's /dev/hugepages, which bcd does not manage.

The configured pools are what inventory reports (memory.hugepages, and per NUMA node since badb8c2), what the capacity module books as hugepages2Mi and hugepages1Gi (and takes out of memoryGiB), and what bcd's own admission of instance starts and holds checks free pages against.

cpu

The CPU frequency governor, and CPUs isolated from the kernel scheduler for pinned vCPUs and holds (1cad924).

A pool's policy.cpu: { governor?, isolate? } becomes this unit (09669d3), and a profile's hostConfig may name it; the pool's wins. Provision checks the same rules (hostConfig.cpu.isolate must not include CPU 0 ...), and bcd checks them again, terminally.

• Governor. For every cpu<N> under /sys/devices/system/cpu with a cpufreq/scaling_governor, bcd writes the governor when it differs. A CPU whose driver does not list the governor in scaling_available_governors fails the unit terminally (intel_pstate offers only performance and powersave). A VM without cpufreq has nothing to set, and the unit converges.
• Isolation. isolate becomes kernel arguments, isolcpus=managed_irq,domain,<list> nohz_full=<list> rcu_nocbs=<list>, on the boot loader's command line: with update-grub, the drop-in /etc/default/grub.d/90-bcd-cpu.cfg (GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX ...") and update-grub; with grubby, --remove-args of the previous arguments, then --args; with neither, the step fails (retried). The arguments bcd last set are recorded in /var/lib/bcd-hostconfig/bootargs/cpu, so an unchanged list plans nothing.
• Reboot. Until /sys/devices/system/cpu/isolated shows exactly the list, the unit reports cpu: CPUs <list> are isolated from the next boot (now: "<what the kernel shows>"): reboot the node, a RebootRequired.
• Removal. Leaving the spec (or an empty isolate) removes the arguments; the governor stays as it is.

The inventory reports the isolated CPUs, and the node label cpu.isolated counts them (see labels). The control plane uses them (b6b804e): a pinned guest takes isolated CPUs first (see flavor placement), while hold cpusets and guest CPUs never include the host's reserved CPUs, the lowest non-isolated, policy.hostReserve.vcpu.

iommu

The IOMMU on, for GPUs passed through to VMs and sandboxes with VFIO: the runtimes bind each instance's GPUs to vfio-pci, which needs IOMMU groups (3fc3628). Load vfio-pci itself with kernel-modules.

A pool's policy.iommu: { passthrough? } becomes this unit (e4b0c01); set it on vps-host pools with GPU flavors. A profile's hostConfig may name it too, and the pool's wins.

• Arguments. From the CPU vendor in /proc/cpuinfo: GenuineIntel gets intel_iommu=on, AuthenticAMD and HygonGenuine get amd_iommu=on, a CPU with no vendor_id line (Arm, whose SMMU is on by default) gets only iommu=pt, and any other vendor fails the unit terminally. They go on the boot loader's command line the same way as cpu's, in /etc/default/grub.d/90-bcd-iommu.cfg or through grubby, recorded in /var/lib/bcd-hostconfig/bootargs/iommu.
• Reboot. Until /sys/class/iommu lists an IOMMU, the unit reports iommu: the IOMMU is on from the next boot (<arguments>): reboot the node; check the firmware enables VT-d or AMD-Vi, a RebootRequired. A machine whose firmware has VT-d or AMD-Vi off keeps reporting it after the reboot.
• Removal. Leaving the spec removes the arguments.

bcd doctor warns when a host with an active IOMMU has no vfio-pci driver loaded (see bcd doctor).

nvidia-driver

The NVIDIA driver branch, installed and loaded.

A loaded driver of the branch and flavor is converged; bcd then only keeps nvidia-persistenced running where it is installed (systemctl enable --now), so the GPUs stay initialized between jobs. Otherwise:

1. Install. When modinfo nvidia does not show the branch and flavor, bcd installs the branch's packages with the distribution's package manager (from /etc/os-release). Unset open installs the proprietary flavor. The repositories are the OS image's business: a package they do not carry fails the step, which is retried.

   Distribution	Packages
   Ubuntu	nvidia-headless-<branch>-server (-open for the open flavor) and nvidia-utils-<branch>-server
   Other Debian family	nvidia-open-<branch>, or cuda-drivers-<branch>
   RHEL family (RHEL, Rocky, Alma, Fedora, CentOS)	the DNF module stream @nvidia-driver:<branch>-open, or <branch>-dkms
   SUSE family	nvidia-open-driver-G06-signed-kmp-default or nvidia-driver-G06-kmp-default, with nvidia-compute-utils-G06 (these names do not carry the branch)
   Anything else	Refused (terminal)

   apt-get runs with DEBIAN_FRONTEND=noninteractive, --no-install-recommends and --force-confold after an apt-get update; dnf -y install and zypper --non-interactive install elsewhere.

2. Swap a loaded driver. When another driver is loaded, bcd stops nvidia-fabricmanager, nvidia-persistenced and nvidia-dcgm, then unloads nvidia_uvm, nvidia_drm, nvidia_modeset and nvidia. modprobe -r refuses while anything holds the GPUs, and the step fails with driver <version> is in use; reboot the node to load branch <branch>.

3. Load. modprobe nvidia and modprobe nvidia_uvm, then a check of the loaded driver. If the loaded one is still not the branch and flavor, the step fails with reboot the node.

Both reboot cases are RebootRequired: retried, with the section PROGRESSING and the node condition set, until someone reboots the machine.

fabric-manager

NVIDIA Fabric Manager, installed and running, on NVSwitch systems. It has no settings ({}).

• A host without NVSwitches (nothing under /proc/driver/nvidia-nvswitch/devices) has nothing to do, so a profile can name the unit for every node of a mixed pool.
• On an NVSwitch host it needs a loaded driver (until then it is retried) and installs Fabric Manager when nv-fabricmanager is missing: nvidia-fabricmanager-<branch of the loaded driver> on the Debian family, nvidia-fabric-manager elsewhere. It then runs systemctl enable --now nvidia-fabricmanager when it installed it or the service is not active.
• While an NVSwitch host runs without it, the unit reports FabricManagerDown: NVLink between its GPUs is down, and CUDA fails to initialize multi-GPU jobs. The node is NoSchedule while it lasts.

gpu-settings

Settings every GPU of the host shares, read and set with nvidia-smi (3fd1c41). It needs the driver: until nvidia-smi answers, the unit is retried. With neither setting it does nothing.

A pool's policy.gpu: { powerLimitWatts?, ecc? } becomes this unit (e4b0c01), and a profile's hostConfig may name it; the pool's wins.

• Power limit. bcd queries each GPU's limit and range. A limit outside a GPU's range fails the unit terminally (GPU <n> takes <min> to <max> W, not <limit>). A GPU whose limit differs by a watt or more is set with nvidia-smi -i <n> -pl <watts>. The driver forgets the limit when it reloads, so the next pass sets it again. A GPU that reports no power management ([N/A]) is skipped.
• ECC. On each GPU whose current ECC mode is Enabled or Disabled (a GPU without ECC is skipped), bcd sets the pending mode with nvidia-smi -i <n> -e 1 or -e 0 when it differs. The mode changes at the GPU's next reset: until the current mode matches, the unit reports gpu-settings: the ECC mode of GPU(s) <indexes> changes at their next reset: reboot the node, a RebootRequired.

mig-layout

Every MIG-capable GPU of the host partitioned to one layout.

The layout grammar:

A profile is <n>g.<n>gb. Profiles that carry a + of their own (such as 1g.10gb+me) cannot be expressed. A layout that does not parse, or a profile the GPU does not have, is terminal.

For each GPU that nvidia-smi --query-gpu shows as MIG-capable (current mode Enabled or Disabled):

• Mode. bcd turns MIG mode on (nvidia-smi -i <n> -mig 1) or off (-mig 0). When the change waits for a GPU reset, as on Ampere, it runs nvidia-smi --gpu-reset -i <n>; a failed reset fails the step with MIG mode ... waits for a GPU reset, which failed (reboot the node), a RebootRequired.
• Instances. When the GPU's instances (nvidia-smi mig -lgi) are not the layout's, bcd destroys the compute instances, then the GPU instances ("not found" counts as done), and creates the layout's GPU instances, largest first, each with a compute instance filling it (nvidia-smi mig -cgi <profiles> -C).
• Busy GPUs. A GPU that a live hold uses (a hold's device id names it, or a MIG instance or slot of it, <gpu>/gi<g>/ci<c> or <gpu>/s<k>; 0f68c9a) is left alone. The other GPUs are repartitioned, and the unit then fails, retryably, with mig-layout: <uuids> hold allocations; they are repartitioned once those end. The control plane changes a pool's layout only once the GPUs' atoms are free (the MIG layout gate); this is bcd's own guard.

A hold on a MIG instance the layout has not created yet fails with device-missing until this unit has made it (see device assignments).

gpu-sharing

How a timeslice or mps pool's GPUs are shared.

• mps needs nvidia-cuda-mps-control, which ships with the driver's utilities (until it is installed the unit is retried). bcd writes /etc/systemd/system/bcd-mps.service and, when it changed, runs systemctl daemon-reload, enable and restart; otherwise it starts the service when it is not active. The service runs the MPS control daemon with CUDA_MPS_PIPE_DIRECTORY=/run/nvidia-mps and CUDA_MPS_LOG_DIRECTORY=/var/log/nvidia-mps, sets every GPU's compute mode to EXCLUSIVE_PROCESS before it starts and back to DEFAULT after it stops, and limits each client by default to 100 / replicas percent of the SMs (at least 1), with set_default_active_thread_percentage. It restarts on failure after 5 seconds. While the service is not active the unit reports MPSDown, since its clients fail, and the node is NoSchedule. A slot hold gets CUDA_MPS_PIPE_DIRECTORY in its env when /run/nvidia-mps exists.
• timeslice uses the GPUs' default time slicing: the replicas are the slots the control plane hands out. bcd stops and removes bcd-mps.service if it installed one.

Provision renders this unit from the pool (1f0f683): a pool whose sharing is mps or timeslice gets gpu-sharing { mode, replicas }, with replicas from policy.sharing.replicas, by default 10 for MPS and 4 for time slicing, the same numbers of slots the capacity module derives. The limits agree end to end: pools.create and pools.update refuse more than 48 replicas (37dcd02), and since a708aa6 the inventory makes at most 48 slot atoms per GPU even for a pool that bypassed the check. A profile may also name the unit.

container-toolkit

The NVIDIA container toolkit, the container runtimes configured with it, and the vendor CDI spec.

• Install. nvidia-container-toolkit with the distribution's package manager when nvidia-ctk is missing.
• Runtimes. For each runtime installed on the host, containerd (containerd, /etc/containerd/config.toml) and Docker (dockerd, /etc/docker/daemon.json), whose configuration does not mention nvidia-container-runtime yet: nvidia-ctk runtime configure --runtime=<name> (with --cdi.enabled when cdi is on), then systemctl restart containerd or docker. A runtime already configured is not configured again, so turning cdi on later does not rewrite it.
• CDI only. With cdi, since b61b9c5 (security review 2) the unit sets accept-nvidia-visible-devices-envvar-when-unprivileged and accept-nvidia-visible-devices-as-volume-mounts to false in /etc/nvidia-container-runtime/config.toml (nvidia-ctk config --in-place --set <key>=false) unless the file already says so. nvidia-ctk runtime configure --cdi.enabled points containerd at the legacy nvidia-container-runtime, and the packaged configuration leaves the first key commented out, at its permissive default: any unprivileged container, any pod, could then choose GPUs, other tenants' included, with NVIDIA_VISIBLE_DEVICES, and on a k8s-join node bcd's own ForeignGpuExposure check withdrew every GPU. The runtime reads its configuration per container, so nothing restarts. cdi: false, the legacy mode that hands GPUs out through the variable, changes neither key.
• CDI. With cdi, bcd generates /etc/cdi/nvidia.json with nvidia-ctk cdi generate --format=json --device-name-strategy=uuid (replacing the /etc/cdi/nvidia.yaml older versions wrote) and records what it describes in /var/lib/bcd-hostconfig/cdi-nvidia.stamp: a SHA-256 of the loaded driver version and nvidia-smi -L, which lists every GPU and MIG device. The spec is generated again whenever that changes, so a driver update or a new MIG layout gets a new spec in the same pass. It needs a loaded driver; until then the unit is retried. cdi: false generates nothing and removes nothing.

This spec is NVIDIA's. bcd's own hold specs are /etc/cdi/bcd-<allocation>.json, written by the devices converger; the Kubernetes device plugin names both.

k8s-join

Joins the machine to a Kubernetes cluster as a worker, and keeps its kubelet and labels (88ecb07, wired into the agent, the helper and the configuration in 7579184). Compute never runs a control plane and never schedules pods; it hands a cluster machines and GPUs. The whole flow is on Kubernetes workers.

A pool's policy.kubernetes: { apiServer, caCertSha256, joinTokenRef, labels?, taints?, kubernetesVersion?, gpu?, gpuApi? } becomes this unit when it names the cluster: apiServer, caCertSha256 and joinTokenRef go together, and a partial set is INVALID_INPUT (7db110a). A pool without them is one whose nodes already run bcd as a DaemonSet inside the cluster, and needs no unit (see on Kubernetes). bcd checks the settings again on the node, since labels, taints and the fetched token end up in YAML and on command lines. Each pass converges, in order:

1. Packages. When kubeadm, kubelet or kubectl is missing, bcd adds the pkgs.k8s.io repository of kubernetesVersion's minor release and installs them: on the Debian family the source /etc/apt/sources.list.d/bcd-kubernetes.list with its key fetched over https (downloaded to /var/lib/bcd-hostconfig/k8s-release.key) into /etc/apt/keyrings/bcd-kubernetes.gpg, apt-get install, then apt-mark hold; on the RHEL family /etc/yum.repos.d/bcd-kubernetes.repo and dnf install --disableexcludes=bcd-kubernetes. It installs containerd too when it is missing. Missing packages without kubernetesVersion, or another distribution, are terminal. A pass that installed packages ends there, and the next one plans the rest.
2. Kernel. overlay and br_netfilter in /etc/modules-load.d/90-bcd-k8s.conf, loaded now; then net.bridge.bridge-nf-call-iptables, net.bridge.bridge-nf-call-ip6tables and net.ipv4.ip_forward at 1 in /etc/sysctl.d/90-bcd-k8s.conf and live; and swapoff -a whenever /proc/swaps lists swap, since the kubelet refuses to run with it. Do not set those net.bridge.* keys to 0 in the sysctl unit on the same machine: the two units would set them back and forth.
3. containerd. /etc/containerd/config.toml imports /etc/containerd/conf.d/*.toml (a new file when there is none; an imports list without it is terminal, for the operator to fix), and bcd's drop-in /etc/containerd/conf.d/90-bcd-k8s.toml turns CDI on (/etc/cdi, /var/run/cdi) and the systemd cgroup driver for runc, in config version 2 or 3 as the installed containerd reads. containerd is restarted when either changed, and started when it is not running.
4. Join. While /etc/kubernetes/kubelet.conf does not exist, bcd fetches the bootstrap token, writes a JoinConfiguration (API kubeadm.k8s.io/v1beta4 from kubeadm 1.31, v1beta3 before) with the endpoint, the token, the CA pin, the Node name, containerd's socket, the labels and the taints to /var/lib/bcd-hostconfig/k8s-join.yaml (0600), runs kubeadm join --config, and removes the file whatever happened. The Node name is the host name, lower-cased with _ and spaces as -; one that is not a DNS subdomain is terminal. It records what it manages in /var/lib/bcd-hostconfig/k8s-join.json.
5. Kept. Once joined: a kubelet whose kubeconfig names another API server is terminal (reset it by hand (kubeadm reset) to move it): bcd never resets a node joined elsewhere. It starts the kubelet when it is not active, and keeps the Node's labels with the kubelet's own credentials (kubectl --kubeconfig /etc/kubernetes/kubelet.conf label), removing only labels it set before or labels under bcd.dev/. Besides the spec's labels, bcd sets bcd.dev/node-id, bcd.dev/site, bcd.dev/gpu.count, bcd.dev/gpu.product and bcd.dev/gpu.driver from its identity and inventory.

Taints are set once (6ef198b). The NodeRestriction admission plugin forbids a kubelet to change its own Node's taints, so bcd sets them only at registration, in the JoinConfiguration, and records them in k8s-join.json as registered. When the unit's taints later differ from the Node's (the pool's taints changed), bcd does not retry: the unit reports the condition KubernetesTaintsDrift (reason NodeRestriction) while they differ, and its message carries the command a cluster admin runs, kubectl taint node <node> --overwrite <taints>, which sets the unit's taints and removes those bcd registered that the unit no longer names. The gateway records it as an informational node condition. Labels are not affected: the kubelet may label its own Node outside kubernetes.io and k8s.io.

The token. bcd node fetches it with a SecretRequest that names workflow host:k8s-join and step tokenRef. The secret broker releases it only while the node's current spec names that reference in that field (binding rule 3, e93bf9c; see sealed secrets). The token is removed once the join succeeded.

Which clusters. Joining a cluster gives it root on the machine (privileged pods), so root decides which clusters are allowed, in config.toml:

[kubernetes]
allowed_clusters = "https://10.0.0.1:6443#sha256:<64 hex digits>,https://k8s.example.com#sha256:<64 hex digits>"

Each entry is an API server and its CA pin; a malformed entry stops bcd from starting. Since 9f860ad the list is required whether bcd node runs as root or under bcd helper, where the helper builds the unit with root's list: an empty list refuses every join (joining a Kubernetes cluster needs [kubernetes] allowed_clusters in bcd's config), and bcd doctor fails a joined node without one. Since the helper cannot ask the control plane for secrets, the node fetches the token first and stages it at /var/lib/bcd-hostconfig/k8s/bootstrap-token (helper file kind k8s-token, mode 0600, in root's directory, so the node cannot point it at another file), where the helper's join reads it (1bb78d6).

Examples

A sandbox host

A pool of VM hosts for sandboxes: KVM with nested virtualization, vhost_net and tun for the guests' NICs, 1 GiB pages per NUMA node for pinned guests, and CPUs isolated for them. The modules, pages and CPUs belong to the pool, so they go in its policy, here in a FleetConfig document:

{
  "version": 1,
  "pools": [
    {
      "name": "sandbox-fra",
      "site": "fra",
      "class": "vps-host",
      "policy": {
        "kernelModules": {
          "load": ["kvm_intel", "vhost_net", "tun", "br_netfilter"],
          "options": { "kvm_intel": "nested=1" }
        },
        "hugepages": { "size": "1Gi", "count": 64, "perNuma": true },
        "cpu": { "governor": "performance", "isolate": "4-63,68-127" }
      }
    }
  ]
}

Kernel parameters come only from a profile's hostConfig, so the machines' deploy profile carries them:

hostConfig: {
  sysctl: {
    // Keep bridged guest traffic out of the host's IP firewall. bcd's instance rules live in the
    // bridge family and do not need br_netfilter, but the keys exist only once it is loaded.
    'net.bridge.bridge-nf-call-iptables': 0,
    'net.bridge.bridge-nf-call-ip6tables': 0,
  },
},

On such a host, kernel-modules writes both drop-ins and loads the modules; kvm_intel loaded before without nested=1 is reloaded only if no VM holds it. sysctl then finds the net.bridge.* keys in the same pass. The hugepages unit sets 64 pages on each NUMA node and mounts /dev/hugepages1G; on a host that has been up for a while it may instead set up the boot loader and report RebootRequired. The cpu unit sets the governor at once and adds the isolation arguments, which also wait for that reboot; one reboot serves both. An Intel host needs kvm_intel, an AMD one kvm_amd; use a when: expression on a profile unit, or separate pools, for mixed hardware.

A GPU host

A MIG pool of H100 hosts with the open driver of branch 580, Fabric Manager where there are NVSwitches, the pool's layout, and the container toolkit with CDI:

hostConfig: {
  'nvidia-driver': { branch: '580', open: true },
  'fabric-manager': {},
  'mig-layout': { from: 'pool' },
  'container-toolkit': { cdi: true },
},

with the pool's policy.migLayout, for example "all-1g.10gb". In one pass bcd installs and loads the driver (or reports that a reboot is needed, and still converges the units after it that do not need the new driver), installs and starts Fabric Manager on an NVSwitch board, turns MIG mode on and creates seven 1g.10gb instances on each idle GPU, configures containerd with the toolkit, and generates /etc/cdi/nvidia.json for the new MIG devices. The inventory refresh that follows reports them, and the control plane derives the pool's gpu.mig atoms from it.

What the tests cover

Apart from TestRealKernel, nothing here has run against a real machine.