# Host config units (/docs/reference/host-units)

> The 12 config units a node's spec may set, their settings and what bcd does with them.



Host config units are what a profile's `hostConfig` and a pool's policy turn into the NodeSpec `host` section; provision validates their settings against the manifests below, and bcd's host converger applies them, level-triggered: drift is repaired at the next pass (at least every 10 minutes). bcd converges them in this order, each seeing what the ones before it did: `kernel-modules`, `sysctl`, `hugepages`, `cpu`, `iommu`, `nvidia-driver`, `fabric-manager`, `gpu-settings`, `mig-layout`, `gpu-sharing`, `container-toolkit`, `k8s-join`. A unit that fails never holds back the others; the host section reports every failure, retried while any is. A unit whose state takes effect only after a reboot reports node condition `RebootRequired`: bcd never reboots on its own. Under `bcd helper`, the node plans each unit and the helper applies it as root, only as a spec signed under the pinned roots asks. With `[host] units = false` (bcd as a Kubernetes DaemonSet) no unit is applied. See [Host configuration](/docs/daemon/host-config).

## nvidia-driver [#nvidia-driver]

The NVIDIA driver branch (and the open kernel modules or not).

Installs the driver branch (Ubuntu's server packages, NVIDIA's CUDA repository, DNF module streams, SUSE's G06 packages) and loads it. A different driver already loaded is swapped only while no process holds the GPUs; otherwise the unit reports `RebootRequired`. It keeps `nvidia-persistenced` running where it is installed.

| Field    | Type    | Required | Constraints          |
| -------- | ------- | -------- | -------------------- |
| `branch` | string  | yes      | matches `^[0-9]{3}$` |
| `open`   | boolean |          |                      |

## container-toolkit [#container-toolkit]

The NVIDIA container toolkit, generating CDI specs.

Installs the NVIDIA container toolkit, configures containerd and Docker with its runtime (with `cdi`, CDI enabled), and regenerates `/etc/cdi/nvidia.json` (JSON, GPUs and MIG devices named by UUID; it replaces the `/etc/cdi/nvidia.yaml` older versions wrote) whenever the driver or the GPU and MIG device list changes. A hold's CDI spec copies its devices' nodes, mounts and hooks from it, so `bcd.dev/gpu=<id>` alone is enough. With `cdi` (the default), a container gets GPUs only by the CDI names bcd hands out. The unit sets `accept-nvidia-visible-devices-envvar-when-unprivileged` and `accept-nvidia-visible-devices-as-volume-mounts` to false in `/etc/nvidia-container-runtime/config.toml`. Otherwise an unprivileged container could choose GPUs other tenants hold with `NVIDIA_VISIBLE_DEVICES` (the packaged config leaves that key at its permissive default). On a Kubernetes node that is the `ForeignGpuExposure` condition.

| Field | Type    | Required | Constraints    |
| ----- | ------- | -------- | -------------- |
| `cdi` | boolean |          | default `true` |

## mig-layout [#mig-layout]

A MIG layout: a named profile per GPU, or the pool's layout.

Partitions every MIG-capable GPU to the layout: `all-disabled`, `all-<profile>` (as many as fit, e.g. `all-1g.10gb`) or `mixed:<n>x<profile>+<profile>…` (per GPU, e.g. `mixed:3g.40gb+2x2g.20gb`), with a compute instance in each GPU instance. A GPU holding allocations or running compute processes is left as it is until they end (under `bcd helper`, the helper sees that itself, never taking the node's word). A MIG mode change that needs a GPU reset, which fails, reports `RebootRequired`.

| Field    | Type                     | Required | Constraints           |
| -------- | ------------------------ | -------- | --------------------- |
| `from`   | one of `pool`, `profile` |          | default `"profile"`   |
| `layout` | string                   |          | at most 64 characters |

## fabric-manager [#fabric-manager]

NVIDIA Fabric Manager for NVSwitch systems.

On NVSwitch systems only (nothing to do elsewhere): installs and runs NVIDIA Fabric Manager; node condition `FabricManagerDown`, which makes the node NoSchedule, while it is not running.

Takes an empty object: `{}`.

## sysctl [#sysctl]

Kernel parameters.

Free-form: each key a sysctl name with a scalar value, written to `/etc/sysctl.d/90-bcd.conf` and set live. A key the kernel does not have yet (its module is not loaded) is retried. Keys that would run programs as root or lower the host's defenses are refused (`kernel.core_pattern`, `kernel.modprobe`, `kernel.usermodehelper.*`, `fs.binfmt_misc.*`, `kernel.randomize_va_space`, `kernel.yama.*`, `user.*` and the like), and a few take only their hardened values: `dev.tty.legacy_tiocsti` and `dev.tty.ldisc_autoload` only 0, `kernel.io_uring_disabled` only 1 or 2, `net.ipv4.ip_unprivileged_port_start` only 1024 and above.

Takes free-form keys: sysctl names (`^[a-z0-9_]+(?:\.[a-z0-9_-]+)+$`) with scalar values.

## hugepages [#hugepages]

Huge pages: 2 MiB pages allocated at runtime, 1 GiB pages through the kernel command line (effective after a reboot), and the transparent huge page policy.

2 MiB pages are allocated at runtime; 1 GiB pages at runtime when the kernel finds the memory, else through the boot loader (`hugepagesz=1G hugepages=N`) with `RebootRequired`. With `perNuma` the counts are per NUMA node. 1 GiB pages are mounted at `/dev/hugepages1G` (a systemd mount unit), 2 MiB pages at systemd's `/dev/hugepages`.

| Field         | Type                                | Required | Constraints |
| ------------- | ----------------------------------- | -------- | ----------- |
| `pages2m`     | integer                             |          | 0-4194304   |
| `pages1g`     | integer                             |          | 0-16384     |
| `transparent` | one of `always`, `madvise`, `never` |          |             |
| `perNuma`     | boolean                             |          |             |

## gpu-sharing [#gpu-sharing]

How a shared pool's GPUs are split between clients: MPS (bcd-mps.service, 100/replicas % of the SMs each) or time-slicing (MPS stopped).

`mps` runs the MPS control daemon (`bcd-mps.service`, each client limited to 100/`replicas` percent of the SMs, node condition `MPSDown` while it is down); `timeslice` stops it.

| Field      | Type                      | Required | Constraints |
| ---------- | ------------------------- | -------- | ----------- |
| `mode`     | one of `mps`, `timeslice` | yes      |             |
| `replicas` | integer                   |          | 1-48        |

## cpu [#cpu]

Host CPUs: the cpufreq governor (live), and CPUs isolated from the kernel scheduler for pinned VMs and holds (isolcpus/nohz\_full/rcu\_nocbs, effective after a reboot).

Sets the frequency governor on every CPU that has one, and isolates CPUs from the scheduler for pinned vCPUs (`isolcpus`, `nohz_full`, `rcu_nocbs` through the boot loader, `RebootRequired` until the kernel shows them isolated). CPU 0 is never isolated.

Takes an empty object: `{}`.

## iommu [#iommu]

The IOMMU for GPU passthrough to VMs and sandboxes: intel\_iommu=on or amd\_iommu=on (by CPU vendor) and iommu=pt on the kernel command line, effective after a reboot.

Turns the IOMMU on for GPUs passed through to VMs and sandboxes (`intel_iommu=on` or `amd_iommu=on`, and `iommu=pt` unless `passthrough` is false) through the boot loader, with `RebootRequired` until an IOMMU is active.

| Field         | Type    | Required | Constraints    |
| ------------- | ------- | -------- | -------------- |
| `passthrough` | boolean |          | default `true` |

## gpu-settings [#gpu-settings]

Per-GPU settings kept by bcd: the power cap (nvidia-smi -pl, re-applied on drift) and ECC (effective after a reboot).

Caps each GPU's board power (applied again whenever it drifts; outside a GPU's range the unit fails) and sets the ECC mode, which changes at the GPU's next reset (`RebootRequired` until then).

| Field             | Type    | Required | Constraints |
| ----------------- | ------- | -------- | ----------- |
| `powerLimitWatts` | integer |          | 1-10000     |
| `ecc`             | boolean |          |             |

## kernel-modules [#kernel-modules]

Kernel modules loaded at boot and now (modules-load.d), with their options (modprobe.d): kvm\_intel/kvm\_amd, vhost\_net and tun for VM hosts.

Modules loaded now and at every boot (`/etc/modules-load.d/bcd.conf`) with their options (`/etc/modprobe.d/bcd.conf`). A loaded module whose options differ is reloaded only when nothing uses it; otherwise they apply when it next loads. Sandbox hosts load `kvm_intel` or `kvm_amd`, `vhost_net`, `tun` and, for GPU passthrough, `vfio-pci`. Options that turn off DMA isolation for passed-through devices (`vfio` `enable_unsafe_noiommu_mode`, `vfio_iommu_type1` `allow_unsafe_interrupts`) are refused.

Takes an empty object: `{}`.

## k8s-join [#k8s-join]

Joins the node to a Kubernetes cluster as a worker (kubeadm join with a bootstrap token fetched by reference) and keeps its kubelet, labels and taints; GPUs through containerd's NVIDIA runtime and CDI.

Joins the machine to a Kubernetes cluster as a worker with a bootstrap token fetched through the secret broker, only to clusters root pins in `[kubernetes] allowed_clusters` under `bcd helper`. See [Kubernetes](/docs/daemon/kubernetes).

Takes an empty object: `{}`.
