# Inventory and health (/docs/guides/nodes/inventory-and-health) > NodeInventory v1 and its section hashes, snapshots and diffs, GPU naming, devices and atoms, conditions, XID handling, remediation and acceptance. Placement can only book what the control plane knows a node has, and should only book it on hardware that works. Inventory is what a node reports about its hardware; health is what the control plane concludes from it. Together they produce the node's devices and atoms, the labels selectors match, and the conditions that keep a broken node or GPU out of placement. Inventory is owned by `@better-compute/inventory`, a built-in module that every server registers. ## NodeInventory v1 [#nodeinventory-v1] The daemon reports its hardware as eight sections of the `NodeInventory v1` schema (`bettercompute.agent.v1`). Readings that change by the second (temperatures, power draw, clocks, utilization) are never inventory; they are telemetry, and they never go to Postgres. | Section | Source on the node | Holds | | -------- | ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `system` | DMI | vendor, product, serials (system, board, chassis), product UUID, BIOS, boot mode, Secure Boot, TPM | | `cpu` | `/proc/cpuinfo`, sysfs | vendor, model, architecture, sockets, cores, threads, flags, NUMA nodes | | `memory` | `/proc/meminfo`, EDAC, DMI | total bytes, huge page pools, DIMMs, EDAC counters | | `disks` | sysfs, udev, NVMe, SMART | name, model, serial, WWN, size, transport, SMART health | | `nics` | netlink, sysfs | physical NICs: permanent MAC, driver, firmware, speed, PCI address, NUMA node, SR-IOV VFs, RDMA, link, MTU | | `gpus` | NVML | driver, CUDA and NVML versions; per GPU: index, UUID, PCI bus id, raw NVML name, architecture, VRAM, serial, VBIOS, NUMA node, MIG state, NVLinks, topology row, fabric clique, ECC counters, row remapping | | `bmc` | in-band IPMI, Redfish host interface | IP, MAC, vendor, firmware, Redfish and IPMI support | | `os` | os-release, uname | distro and version, kernel, cgroup version, KVM, IOMMU, runtimes, hostname | The daemon collects inventory when its boot changes, on hardware events and XIDs, after provisioning steps, on a jittered 12-hour rescan, and when you run `inventory.refresh`. > **What bcd collects today.** bcd's `inventory.Collector` reads procfs and sysfs without cgo. It fills `system`, `cpu`, `memory`, `disks`, `nics` (from sysfs, not netlink) and `os`, and reads NVIDIA GPUs from the kernel driver's `/proc/driver/nvidia` files (UUID, model, VBIOS, bus id, driver version). Where `nvidia-smi` is installed, a host build completes the GPUs from it (cb987e5, `nvml.SMI`, no cgo): `nvidia-smi -q -x` for identity, VRAM, ECC counters, row remapping, MIG mode and instances, fabric clique, power limit, persistence and compute mode, plus `-L`, `--query-gpu compute_cap`, `topo -m`, `nvlink -s`, `nvlink -R` and `mig -lgip`, merged into the sysfs GPUs by UUID. `nvlink -R` (5026330) names each NVLink's peer from the PCI bus id at its far end: the UUID of this node's GPU on that bus, else `nvswitch:` for a link through an NVSwitch (`NvlinkLink.peer`); like every query besides `-q -x`, it is optional, and a failure leaves the peers empty. The control plane stores the peers with the section; the `nvlinkPeers` device attribute still counts the `NV` links of the `topo -m` row. Live samples (memory used, utilization, temperature, power, clocks, processes) travel with the section but stay out of its hash, so they never make a new inventory version. There is no `bmc` section. The `cpu` section carries the kernel's isolated CPUs (`/sys/devices/system/cpu/isolated`) and each NUMA node its own huge page pools (configured totals only: free pages change with every VM and would move the hash; badb8c2). It collects when the process starts (so after every boot), whenever `inventory.refresh` or `diag.nvml` asks for a rescan, after a config unit applied, every 10 minutes (GPU error counters and MIG state change without an event bcd sees) and about 2 seconds after an XID burst (662fb55), sending only the sections that changed; the 12-hour rescan is not built. The fakehw build reports the full fake HGX board instead. None of this has run on a real GPU host yet: the `nvidia-smi` parsers are tested against captured output. bcd does not use the NVML library or its events. It sees XIDs by following the kernel log (`/dev/kmsg`) and reports each one as a condition, and it samples host pressure every 30 seconds. See [XID events](#health-quarantines-and-remediation) and [host pressure](#host-pressure). ## How inventory travels [#how-inventory-travels] Inventory is large and rarely changes, so it is never re-sent whole. Each section travels, and is stored, by the hash of its bytes. ### Hash on the node [#hash-on-the-node] The daemon canonicalizes each section (repeated fields sorted by a stable key), encodes it, and takes the SHA-256 of those exact bytes. `Hello`, every tenth heartbeat, and the first heartbeat after changed sections were sent (427787a) carry the section hashes. ### Pull what changed [#pull-what-changed] The gateway records the hashes and asks, in `Welcome.request_sections`, only for sections whose hash differs from the stored one. Unknown section names from a newer daemon are ignored. ### Verify over the received bytes [#verify-over-the-received-bytes] Each `InventorySection{ section, content, sha256 }` is checked against its SHA-256 over the bytes as received and decoded; it is never re-encoded to compare, so only the daemon's encoding has to be stable. A section over 1 MiB, a hash mismatch or bytes that do not decode are `INVALID_INPUT`. ### Apply once complete [#apply-once-complete] The Inventory controller runs when every reported section is present. The snapshot digest is the SHA-256 over `name=hash` lines in section order; an unchanged digest does nothing. A changed one is applied in one transaction with the node row locked. Applying an inventory: * appends a snapshot (`inventory_snapshots`, id `inv_…`): snapshots are append-only; * plans the node's devices and atoms, and its inventory labels; * sets the hardware key; * writes the inventory conditions and health findings, and mirrors the NoSchedule ones into `noScheduleReasons`, keeping every entry it does not own; * publishes `node.inventory.changed` (`snapshotId`, `digest`, `sections`, `changes`) and one event per change: `node.gpu.missing`, `node.gpu.added`, `node.disk.replaced`, `node.driver.changed`, `node.mig.layout.changed`. ### Trust: tenant-controlled nodes are advisory [#trust-tenant-controlled-nodes-are-advisory] A node under a [dedicated-root lease](/docs/guides/bare-metal/dedicated-root) has `trust: 'tenant-controlled'`: the lessee has root, so it can report anything. Its reports are advisory (PLAN.md §10.1 Trust). For such a node: * `applyInventory` appends the snapshot and publishes `node.inventory.changed` with `advisory: true` (and no per-change events), and changes nothing else. Devices, device health, labels, conditions, `noScheduleReasons`, the image cache and the hardware key keep the platform's last view, and the applied snapshot stays the platform's last one. The result status is `advisory`. * The applied digest is stored as `advisory:`, so hash reports treat the set as applied and heartbeats do not queue it again. * `recordXid` ignores its XIDs (no condition, no quarantine, no NoSchedule), and `recordHostPressure` ignores its pressure, so a lessee cannot make a `vps-host` pool's node NoSchedule or raise conditions. * The pool's image cache query for the `image-cached` scorer skips tenant-controlled nodes. Without this, a GPU the lessee reports missing or unhealthy (or a faked XID) would turn the device `healthy: false` and leave the lease's bill. The `inventory.get`, `inventory.history` and `inventory.diff` methods still read the appended snapshots, the lessee's included. When metal ends the lease, `metal.exitTenantRoot` calls `endAdvisoryInventory`, which clears the advisory digest, so the next report applies the set for real under platform trust even if it did not change. ## Reading inventory [#reading-inventory] All three methods are decided as `compute:nodes:read` on the node, and answer `NOT_FOUND` for a node of another tenant or one with no inventory yet. | Method | Returns | | ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | [`inventory.get`](/docs/reference/api/inventory#get) | The latest snapshot: `nodeId`, `hash`, `sections` (decoded, 64-bit numbers as strings), `collectedAt`. | | [`inventory.history`](/docs/reference/api/inventory#history) (v0.2) | Snapshots newest first, `limit` 1 to 100 (20 by default), paged by snapshot id. | | [`inventory.diff`](/docs/reference/api/inventory#diff) (v0.2) | The changes between two snapshots, named by id (`inv_…`) or hash; by default the latest against the one before. | ```ts const latest = await client.inventory.get({ tenantId, nodeId }); console.log(latest.sections.gpus); const { from, to, changes } = await client.inventory.diff({ tenantId, nodeId }); for (const change of changes) console.log(change.type, change.deviceId, change.before, change.after); ``` ```bash better-compute api inventory.diff tenantId=$TENANT nodeId=node_… --format table ``` A diff lists, in this order, GPUs missing and added (by UUID), MIG layout changes (mode plus sorted instances), disks replaced (the same kernel name with another WWN or serial), and driver changes (NVIDIA driver or CUDA version). A section missing on either side yields no changes for it: a node that stopped reporting a section did not lose hardware. ## GPU naming [#gpu-naming] NVML names are inconsistent across drivers and SKUs ("NVIDIA H100 80GB HBM3", "NVIDIA H100 PCIe"). Every GPU gets two normalized names: * `gpuProduct`, the exact product in `-
-GB` form, such as `H100-SXM5-80GB`; * `gpuModel`, its family, such as `H100`. Pools, policies and meters use `gpuModel`: a pool holds exactly one. Selectors may use either. The mapping is the versioned table `gpuModels` (the plan's `gpu-models.json`, version 2), and `normalizeGpuProduct(name)` applies it: upper case, punctuation collapsed to `-`, a leading `NVIDIA` dropped. | Family | Products | | ---------------------------------- | ------------------------------------------------------------------------------------- | | `H100` | `H100-SXM5-80GB`, `H100-PCIE-80GB`, `H100-NVL-94GB` | | `H200` | `H200-SXM5-141GB`, `H200-NVL-141GB` | | `B200` | `B200-SXM-180GB` | | `GB200` | `GB200-SUPERCHIP-186GB` | | `A100` | `A100-SXM4-40GB`, `A100-SXM4-80GB`, `A100-PCIE-40GB`, `A100-PCIE-80GB` | | `L40S` | `L40S-PCIE-48GB` | | `L4` | `L4-PCIE-24GB` | | `RTX-PRO-6000` | `RTX-PRO-6000-SERVER-96GB`, `RTX-PRO-6000-WORKSTATION-96GB`, `RTX-PRO-6000-MAXQ-96GB` | | `GH200` | `GH200-SUPERCHIP-96GB`, `GH200-SUPERCHIP-144GB` | | `L40`, `A40`, `A30`, `A10` | `L40-PCIE-48GB`, `A40-PCIE-48GB`, `A30-PCIE-24GB`, `A10-PCIE-24GB` | | `RTX-6000-ADA`, `RTX-A6000` | `RTX-6000-ADA-PCIE-48GB`, `RTX-A6000-PCIE-48GB` | | `RTX-5090`, `RTX-5080`, `RTX-4090` | `RTX-5090-PCIE-32GB`, `RTX-5080-PCIE-16GB`, `RTX-4090-PCIE-24GB` | Version 2 added the parts real hosts reported that version 1 missed: data-center cards outside the flagship lines, and the workstation and GeForce cards that edge, lab and single-tenant hosts run. A name the table does not know keeps its canonical form as `gpuProduct`, maps to `gpuModel: 'other'`, and raises `GpuModelUnknown`, which keeps the node from scheduling until the table knows it. ## Devices and atoms [#devices-and-atoms] The `gpus` section becomes devices with stable ids, so a reboot or a re-inventory never renames them: | Device | Id | Class | Bookable | | ------------------------ | ----------------------- | ---------- | -------------------------- | | A whole GPU | its UUID, `GPU-…` | `gpu` | yes, unless partitioned | | A MIG instance | `/gi/ci` | `gpu.mig` | yes; its parent GPU is not | | A time-slice or MPS slot | `/s` | `gpu.slot` | yes; its parent GPU is not | Only atoms are booked. Slots exist in `timeslice` and `mps` pools: 4 per GPU for time-slicing and 10 for MPS by default (`policy.sharing.replicas`, 1 to 48, MPS's limit; a708aa6). A GPU whose MIG mode is about to change has every atom unbookable, with `partitionPending: true`, until the change completes. See [fractional GPUs](/docs/guides/capacity/fractional-gpus). Device attributes are what selectors and scorers read: `gpuProduct`, `gpuModel`, `vramGiB`, `index`, `pciBusId`, `numaNode`, `architecture`, `computeCapability`, `driverVersion`, `migCapable`, `nvlinkPeers`, `fabricCluster` and `fabricClique`. A device that vanished while it still has live bookings is kept with `healthy: false` and deleted once its bookings end. A device id another node already owns is left alone and raises `DuplicateIdentity`. ### Labels [#labels] Inventory also writes node labels with source `inventory`, the way NVIDIA's GPU Feature Discovery does: `gpu.present`, `gpu.count`, `gpu.product`, `gpu.model`, `gpu.vramGiB`, `cuda.driver.major`, `nvidia.driver.version`, `mig.capable`, `fabric.domain`, `cpu.arch`, `cpu.vendor`, `cpu.model`, `os.distro`, `os.kernel` and `kvm.available`. A label whose GPUs disagree reads `mixed`. `nodes.label` cannot overwrite them. Since badb8c2 the host's CPU and memory have labels too, for selectors and `when:` expressions such as `numa.nodes > 1`: | Label | Value | | -------------------------------- | -------------------------------------------------------------------------------------------- | | `cpu.count` | Logical CPUs | | `cpu.sockets` | Sockets | | `cpu.isolated` | How many CPUs the kernel isolates (0 when none) | | `numa.nodes` | NUMA nodes | | `memory.totalGiB` | Installed memory in whole GiB | | `hugepages.2Mi`, `hugepages.1Gi` | Configured pages in the host's 2 MiB and 1 GiB pools (0 when the size exists but holds none) | `cpu.model` is the reported model with trademark marks dropped and every other run of characters outside `A-Z a-z 0-9 . _` turned into `-` (`AMD EPYC 9654 96-Core Processor` reads `AMD-EPYC-9654-96-Core-Processor`); a sandbox snapshot that includes memory restores only on hosts with an equal value (see [snapshots](/docs/guides/vps/sandboxes#snapshots-and-restore)). The gateway writes one more kind at every `Hello`: each driver the daemon advertises in `Hello.drivers` becomes the label `bc.dev/driver.=` with source `system` (a missing version reads `unknown`; ids outside `[a-z0-9._-]` are skipped), and driver labels the daemon no longer advertises are dropped, in one statement that writes nothing when they did not change. Operators cannot overwrite them either. Placement can require them with `PlacementRequest.nodeLabels`, where `'*'` accepts any version (see [placement](/docs/guides/capacity/placement#eligibility-filters)). The vps module does this for every runtime: a hosted instance is placed only on hosts carrying `bc.dev/driver.`, bcd's built-in `libvirt-kvm` and `cloud-hypervisor` included since 0fca413 (before, only plugin runtimes needed it). Since 14ceb1e the same statement also writes one label per kind each advertised driver serves, `bc.dev/driver-kind..=1` (a site agent hosting `bcx-ipmi` carries `bc.dev/driver-kind.ipmi.power`, `.management` and `.inspect`), and drops the ones a later `Hello` no longer lists. Kinds outside `[a-z][a-z0-9-]` (1 to 32 characters) are skipped, and at most 32 are read per driver. A placement or selector can require one with `'*'`, as for driver labels. Metal reads them (a3ec8d0): before a verb starts a workflow, it keeps only the operations of the kinds the site's agents report for the node's BMC driver, and refuses the verb with `UNSUPPORTED` (reason `bmc-driver-unsupported`) when a step needs one they lack. The labels also make a driver metal's own table does not know checkable. See [what metal checks before a workflow](/docs/guides/bare-metal/bmc#what-metal-checks-before-a-workflow). ### The hardware key [#the-hardware-key] The hardware key identifies the machine rather than its operating system. It is the SHA-256 over the normalized product UUID, serial (system, else board, else chassis), sorted permanent MACs and sorted GPU UUIDs, after dropping firmware placeholders such as `To Be Filled By O.E.M.`. It is **trusted** when at least two independent identifiers survive, and trusted keys are unique across nodes. JoinService computes it from the join claims and the Inventory controller from inventory, with the same function, so both agree. A second node with the same trusted key (a cloned disk image, a moved board) gets `DuplicateIdentity` and keeps its key untrusted until an operator resolves it. A copied node certificate shows up at the gateway instead: two machines with the same certificate keep taking the node's session from each other, each second `Hello` publishing `security.node.duplicate-session`. More than 3 such alternations within 10 minutes revoke the certificate (`security.node.quarantined`, reason `duplicate-identity`), and the real machine must re-join with a new token. See [duplicate identity](/docs/daemon/sessions#duplicate-identity). ## Conditions [#conditions] A condition is `{ type, status, reason?, message?, since }`, where `status` is `true`, `false` or `unknown` and `since` moves only when the status changes. `nodes.get` lists them. A condition with a NoSchedule policy also puts its type into `noScheduleReasons`, in the same transaction. | Condition | Raised by | Effect | | ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | `GpuCountMismatch` (v0.1) | Inventory: a GPU count other than the pool's `policy.expect.gpuCount`, or a family other than the pool's `gpuModel` | NoSchedule. `unknown` with reason `NotReported` when the node sends no `gpus` section. | | `GpuModelUnknown` (v0.1) | Inventory: a GPU name missing from `gpuModels` | NoSchedule | | `DuplicateIdentity` | Inventory: the trusted hardware key or a device id belongs to another node | NoSchedule | | `AcceptanceFailed` | NodeLifecycle, see below | NoSchedule while `true` | | `NodeUnreachable` | Liveness, see [operating nodes](/docs/guides/nodes/operating-nodes) | the `unreachable` reason | | `Xid48` (v0.2) | an XID 48 (double-bit ECC) | the GPU is quarantined | | `Xid79`, `Xid95` (v0.2) | an XID 79 (fell off the bus) or 95 (uncontained ECC) | NoSchedule; a reboot once no allocation is live | | `RowRemapPending` (v0.2) | XID 63, or inventory's row-remap state | NoSchedule; a GPU reset once no allocation is live | | `RowRemapFailed` (v0.2) | XID 64, or inventory's row-remap state | the GPU is quarantined for replacement | | `EccUncorrectable`, `NvlinkDegraded` (v0.2) | inventory: volatile uncorrectable ECC errors, or NVLinks down on a GPU that has NVLink peers | the GPU is quarantined | | `DriverMismatch`, `CertExpiring`, `DaemonOutdated` | the hygiene sweep, see below | NoSchedule, plus `node.condition.changed` | | `CpuPressure`, `MemoryPressure`, `DiskPressure` (v0.3) | host pressure in status reports, see below | NoSchedule on `vps-host` pools; an alert elsewhere | | `FabricManagerDown` (v1.1) | bcd's `fabric-manager` unit: an NVSwitch system without Fabric Manager running, so CUDA cannot initialize multi-GPU jobs (c7613e1) | NoSchedule while `true` | | `MPSDown` (v1.1) | bcd's `gpu-sharing` unit: an `mps` pool's MPS control daemon is not running, so every client fails (c7613e1) | NoSchedule while `true` | | `RebootRequired` (v1.1) | bcd's host units: a setting takes effect only after a reboot; the reason names the units (0aea5c0), see below | Informational | | `HostManagedExternally` (v1.1) | the `host` section's state on a node with `[host] units = false` (c7613e1) | Informational | | `ForeignGpuExposure` (v1.1) | bcd on a Kubernetes worker: something besides bcd hands out GPUs (another `nvidia.com/*` device plugin, another GPU DRA driver in the node's ResourceSlices, or a container toolkit that accepts `NVIDIA_VISIBLE_DEVICES` from unprivileged pods, including one the container runtime runs whose configuration is missing or unreadable), or, on a DaemonSet node, no container runtime configuration is readable, so bcd offers none (9f860ad, f88bcdb; see [GPU isolation](/docs/daemon/kubernetes#gpu-isolation)) | NoSchedule while `true` (ebd9806), and the node's active reservations are `Degraded` with reason `ForeignGpuExposure` (4570dfe) | | `PodResourcesUnavailable` (v1.1) | bcd on a Kubernetes worker: it holds devices for Kubernetes and the kubelet's pod-resources socket is missing or failing, so released devices cannot be reclaimed from pods (9f860ad) | NoSchedule while `true` (ebd9806), for the whole node | | `KubernetesReleased` (v1.1) | bcd: the spec dropped `k8s-join`, and bcd cordoned and drained the Node (9f860ad) | Informational | | `KubernetesTaintsDrift` (v1.1) | bcd's `k8s-join` unit: the Node's taints differ from the pool's, which the kubelet may not change; the message is the `kubectl taint` command a cluster admin runs (6ef198b) | Informational | | `KubernetesReleaseStuck` (v1.1) | bcd: a released Kubernetes allocation still held after 5 minutes (by a pod, a prepared DRA claim or a pod process, or with the kubelet unreachable), so its release is not reported yet (cd5fc82, 8bcbe92) | NoSchedule while `true` (ebd9806) | | `KubernetesReleasesUnreadable` (v1.1) | bcd: its pending Kubernetes releases file (`releasing.json`) could not be read, so every GPU of the node stays claimed until no pod may hold a bcd GPU of an allocation the node does not know, or an operator removes the marker `/releasing.unreadable` (a18f423, b01130f; see [an unreadable releases file](/docs/guides/capacity/kubernetes#an-unreadable-releases-file)); reason `ReleasesFileUnreadable` | NoSchedule while `true` (4884a65) | The gateway's `daemonNodeConditions` (b91dcff, 88219ea) lists the Kubernetes conditions, so they are written as node conditions. Since ebd9806 (security review 2) three of them, and since 4884a65 a fourth, are NoSchedule while true on a node that is not `tenant-controlled` (on a tenant-controlled node they stay advisory, as below), because each means the node cannot hand out GPUs safely: * **`ForeignGpuExposure`**: another path (the GPU operator's device plugin, another GPU DRA driver, or `NVIDIA_VISIBLE_DEVICES` from unprivileged pods) reaches the node's GPUs, or bcd cannot rule one out, so a hold booked there would not be isolated. A change of it also wakes the node's started reservations, which then show `Degraded` / `ForeignGpuExposure` at once (4570dfe; see [reservation conditions](/docs/guides/capacity/reservations#states-and-conditions)). * **`PodResourcesUnavailable`**: bcd cannot tell which pods hold which devices. Placement has no NoSchedule for one binding only, so the whole node takes no work until the socket answers. * **`KubernetesReleaseStuck`**: until ebd9806 it rode bcd's `StatusReport.conditions` and the `node.status` event only. Now it is recorded, and while a released allocation's devices are still held by a pod the node takes no more work, since the GPUs may still be in use; an operator looks first. * **`KubernetesReleasesUnreadable`**: bcd could not read its pending releases, so every GPU of the node stays claimed on the node itself (no hold, VM, `gpu.reset` or `diag.dcgm` gets one). Until 4884a65 the gateway dropped it, so placement kept booking those GPUs and each new hold failed on the node with `overlap`. Now it is recorded and the node takes no work until the claim clears. `KubernetesReleased` and `KubernetesTaintsDrift` stay informational. See [Kubernetes workers](/docs/guides/capacity/kubernetes#gpu-isolation). `FabricManagerDown`, `MPSDown`, `RebootRequired`, `HostManagedExternally` and the Kubernetes conditions above come from bcd's [host configuration](/docs/daemon/host-config#status-and-failures) and its Kubernetes side. The gateway's `xidStatusHandler` makes each the node's condition exactly as the last status report says, `true` while reported and `false` once it is not, and writes and publishes `node.condition.changed` only on a change. On a `tenant-controlled` node they are advisory: recorded, never a NoSchedule reason. **RebootRequired.** Some host settings wait for a reboot: 1 GiB huge pages reserved on the kernel command line, CPUs isolated with the `cpu` unit, the `iommu` unit, an ECC mode set with `gpu-settings`, a driver swap while the GPUs are in use, and a MIG mode reset that failed. bcd never reboots on its own, and nothing in the control plane reboots on this condition: reboot the node yourself with [`node.reboot`](/docs/guides/nodes/commands), within the pool's disruption budget. The node keeps running and taking work meanwhile. The condition table in `@better-compute/inventory` (`nodeConditionTypes`) also defines `PcieReplayHigh` and `ClockSkew`. No controller raises those on nodes yet. ### Node hygiene [#node-hygiene] Some faults no single report shows. The inventory module's `inventory.hygiene` reconciler, a periodic singleton, sweeps every node that is not deleted or retired every 5 minutes, 500 nodes per transaction: | Condition | Raised when | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DriverMismatch` | The node converged (`observed_generation >= generation`) on a spec whose `nvidia-driver` host unit wants branch B, while its GPUs report a driver of another branch. | | `CertExpiring` | The newest unrevoked certificate of the node's identity generation (ramdisk certificates aside) has less than a tenth of its lifetime left, and at least an hour, so renewal is failing. | | `DaemonOutdated` | The session's `proto_max` is below the control plane's protocol version, or `bcd` runs below a configured `minDaemonVersion`. | Each raised condition adds a NoSchedule reason and publishes `node.condition.changed` (`condition`, `status`, `reason`, `noSchedule`); both go when the cause does. Only changes are written. The built-in reconciler uses the defaults; `hygieneReconciler(options)` and `runHygiene(sql, options)` take `every`, `minDaemonVersion`, `protocolMax`, `certExpiringFraction`, `certExpiringMinSeconds` and `batch`. ### Host pressure [#host-pressure] Since protocol v0.3, `StatusReport.pressure` carries the host's Linux pressure stall information ("some" avg10 from `/proc/pressure/*`, percent of wall time) and CPU steal (percent of the last report interval); it is absent when the kernel exposes no PSI. `recordHostPressure(tx, nodeId, report)` turns it into conditions: | Condition | Raised at | Cleared below | | ---------------- | ------------------------- | ------------- | | `CpuPressure` | CPU PSI 50%, or steal 20% | half of each | | `MemoryPressure` | memory PSI 20% | 10% | | `DiskPressure` | I/O PSI 40% | 20% | Between the two levels a condition keeps its status, so a host hovering at a threshold does not flap. On nodes of `vps-host` pools the conditions are NoSchedule; elsewhere they only alert through `node.condition.changed`. Nothing is written while nothing changes, and nothing at all for a `tenant-controlled` node (see [trust](#trust-tenant-controlled-nodes-are-advisory)). The gateway runs it only when the host installs the status handler: ```ts import { createGateway, xidStatusHandler } from 'better-compute/gateway'; import { hostPressureStatusHandler } from 'better-compute/inventory'; createGateway({ reports: { status: [xidStatusHandler, hostPressureStatusHandler({ events })] } }); ``` `hostPressureStatusHandler({ thresholds })` takes other raise thresholds (`cpu`, `steal`, `memory`, `io`). In tests, `FakeDaemon.setPressure()` puts a `HostPressure` into every status report. bcd reports real values (`daemon/internal/health`). Every 30 seconds it reads the `some avg10` line of `/proc/pressure/cpu`, `memory` and `io`, and computes CPU steal from the `cpu` line of `/proc/stat` as the share of CPU time stolen since the previous sample (0 on the first). It puts the sample into its latest `StatusReport` and sends that report again. A kernel without PSI yields no `pressure` at all, so no pressure condition is raised. ## Health, quarantines and remediation [#health-quarantines-and-remediation] Not every fault should take a whole node away. A GPU with a double-bit ECC error is one bad atom on an eight-GPU machine, while a GPU that fell off the bus usually needs a reboot of the node. * **Device quarantine.** A device-level fault writes a `device_health` row (source `xid` or `inventory`) and marks the device `healthy: false`. The atom leaves placement; the rest of the node keeps working. An XID quarantine stays until it is lifted with `clearDeviceHealth(tx, deviceId, reason?)` (a function of `@better-compute/inventory`) or by a successful remediation; inventory quarantines follow each new inventory. * **XID events.** The daemon reports XIDs as `Xid` conditions on a GPU in its status reports. With the gateway's `xidStatusHandler` installed (`createGateway({ reports: { ...gatewayReportHandlers, status: xidStatusHandler } })`), each one goes through `recordXid`. XIDs without a policy (13, 31, 43, 45 and other application faults) are only recorded as events, and XIDs from a `tenant-controlled` node change nothing. When an XID quarantines a GPU, the handler also wakes the node's started reservations at once (04f6bda), so a reservation holding that GPU turns `Degraded` with reason `AtomUnhealthy`, naming the GPU, within a pass, rather than at capacity's periodic recheck or provision's health reconciler. Nothing is released, and the node keeps its hold; on a Kubernetes worker bcd's device plugins or DRA driver stop offering the GPU as healthy (6f5a99f). `tests/e2e/kubernetes-bindings.test.ts` runs this with an XID 48 on a Kubernetes-bound GPU. * **XIDs in bcd.** bcd follows `/dev/kmsg` from its end and parses driver lines of the form `NVRM: Xid (PCI:0000:18:00): 79, ...`. It normalizes the bus id to NVML's form (`00000000:18:00.0`), finds the GPU with that bus id in its inventory, and raises condition `Xid79` (status `TRUE`, `device_id` the GPU's UUID, the driver's text as the message), one per XID and GPU. The status report goes out at once, and the condition stays in every later report. An XID for a bus id the inventory does not know is only logged. bcd keeps these conditions for the rest of the boot: they are stored with the boot id in `data_dir/faults.json` (mode 0600), so a restart of bcd in the same boot reports them again even though it reads the kernel log from its end, while the first start after a reboot finds another boot id and drops them. The reboot that remediation runs therefore clears them, and so does a `gpu.reset` that leaves the GPU healthy, for that GPU (662fb55). An XID also makes bcd collect its inventory again about 2 seconds later, since the GPU's ECC and row-remap counters move with it. * **Remediation.** Node-level faults queue the `health` reconciler of the provision module. It waits until the node holds no live allocation, then runs a `node.reboot` workflow (`Xid79`, `Xid95`) or a GPU reset (`RowRemapPending`; bcd's `gpu.reset` refuses a GPU that still has a live hold, see [delivery](/docs/daemon/delivery#gpureset-and-live-holds)). Runs share the pool's disruption budget (`policy.disruptionBudget`, 5% of the pool and at least one node, together with workflows and rollouts), and a circuit breaker pauses remediation for a pool when more than 10% of its nodes turned unhealthy within 10 minutes (`pool.remediation.paused`). A successful reboot clears the XID conditions and their quarantines; a GPU reset is confirmed by the next inventory. A failed or cancelled run, or two runs that did not clear the fault, put the node in `maintenance` for an operator (`node.remediation.escalated`). Progress is published as `node.remediation.started` and `node.remediation.succeeded`. ## Acceptance checks [#acceptance-checks] A node that just joined has not proven its hardware works. Acceptance runs the pool's checks before the node becomes schedulable, and again whenever you ask. ```json title="pool policy" { "acceptance": [{ "kind": "diag.nvml" }], "expect": { "gpuCount": 8 } } ``` The NodeLifecycle controller runs them for a node in `adopting` once it is connected: 1. It takes the pool's `policy.acceptance` list (each kind once). A site agent, a node without a pool, or an empty list passes at once (reason `NoChecks`). 2. It enqueues one command per check, with idempotency key `acceptance:::` and a 10-minute `not_after`, sets `AcceptanceFailed` to `unknown` (reason `Running`), and publishes `node.acceptance.started` with the attempt, command ids and checks. 3. `diag.nvml` checks the GPUs inventory found against `policy.expect.gpuCount` (else against the number found). Other kinds run with their default payload in this release; a kind that is not registered fails the attempt with `UnsupportedCheck`. On bcd (662fb55), `diag.nvml` fails `nvml.init` when the driver's management interface (`nvidia-smi`) did not answer, and per GPU fails `ecc.uncorrectable`, `row-remap.pending`, `row-remap.failure`, `pcie.link` (a link narrower than its maximum width) and `xid.recent` (an XID this boot), each with its reason; bcd advertises the `nvml` capability in `Hello` only when that data is there. 4. When every command settled: all succeeded (and `diag.nvml` reported `passed`) makes the node `adopted`, sets `AcceptanceFailed` to `false` (reason `Passed`), refreshes its IAM projection, asks the NodeSpec controller for generation 1 and publishes `node.adopted`. Otherwise `AcceptanceFailed` turns `true` with the reason (`Expired`, `diag.nvml:`, `:failed`, …), the node stays `adopting` with the `AcceptanceFailed` NoSchedule reason, and `node.acceptance.failed` carries the reason and command id. A failure is sticky: the controller does not retry on its own. Fix the cause, then call [`nodes.adopt`](/docs/reference/api/nodes#adopt) (`compute:nodes:provision`), which starts a new attempt. On an `adopted` node, `nodes.adopt` moves it back to `adopting`, so it leaves placement until the checks pass again. `diag.nvml` is the v0.1 check. The `diag.dcgm` kind comes with the metal module, whose commissioning profile runs it at level 2 (and the dedicated-root release at level 1); as a pool acceptance check it runs with its default payload until check arguments (`with`) are mapped per kind. From level 2, bcd refuses it `gpu-busy` for a GPU a hold or a VM is using (539c4ce). On the node itself, `bcd doctor` prints one `gpu` line per GPU with its load, and warns on a pending row remap or MIG mode change and fails on failed remapping or uncorrectable ECC errors (f9d7af9; see [bcd doctor](/docs/daemon#modes)).