Inventory and health

Placement can only book what the control plane knows a node has, and should only book it on hardware that works. Inventory is what a node reports about its hardware; health is what the control plane concludes from it. Together they produce the node's devices and atoms, the labels selectors match, and the conditions that keep a broken node or GPU out of placement.

Inventory is owned by @better-compute/inventory, a built-in module that every server registers.

NodeInventory v1

The daemon reports its hardware as eight sections of the NodeInventory v1 schema (bettercompute.agent.v1). Readings that change by the second (temperatures, power draw, clocks, utilization) are never inventory; they are telemetry, and they never go to Postgres.

The daemon collects inventory when its boot changes, on hardware events and XIDs, after provisioning steps, on a jittered 12-hour rescan, and when you run inventory.refresh.

How inventory travels

Inventory is large and rarely changes, so it is never re-sent whole. Each section travels, and is stored, by the hash of its bytes.

Applying an inventory:

• appends a snapshot (inventory_snapshots, id inv_…): snapshots are append-only;
• plans the node's devices and atoms, and its inventory labels;
• sets the hardware key;
• writes the inventory conditions and health findings, and mirrors the NoSchedule ones into noScheduleReasons, keeping every entry it does not own;
• publishes node.inventory.changed (snapshotId, digest, sections, changes) and one event per change: node.gpu.missing, node.gpu.added, node.disk.replaced, node.driver.changed, node.mig.layout.changed.

Trust: tenant-controlled nodes are advisory

A node under a dedicated-root lease has trust: 'tenant-controlled': the lessee has root, so it can report anything. Its reports are advisory (PLAN.md §10.1 Trust). For such a node:

• applyInventory appends the snapshot and publishes node.inventory.changed with advisory: true (and no per-change events), and changes nothing else. Devices, device health, labels, conditions, noScheduleReasons, the image cache and the hardware key keep the platform's last view, and the applied snapshot stays the platform's last one. The result status is advisory.
• The applied digest is stored as advisory:<digest>, so hash reports treat the set as applied and heartbeats do not queue it again.
• recordXid ignores its XIDs (no condition, no quarantine, no NoSchedule), and recordHostPressure ignores its pressure, so a lessee cannot make a vps-host pool's node NoSchedule or raise conditions.
• The pool's image cache query for the image-cached scorer skips tenant-controlled nodes.

Without this, a GPU the lessee reports missing or unhealthy (or a faked XID) would turn the device healthy: false and leave the lease's bill. The inventory.get, inventory.history and inventory.diff methods still read the appended snapshots, the lessee's included. When metal ends the lease, metal.exitTenantRoot calls endAdvisoryInventory, which clears the advisory digest, so the next report applies the set for real under platform trust even if it did not change.

Reading inventory

All three methods are decided as compute:nodes:read on the node, and answer NOT_FOUND for a node of another tenant or one with no inventory yet.

const latest = await client.inventory.get({ tenantId, nodeId });
console.log(latest.sections.gpus);

const { from, to, changes } = await client.inventory.diff({ tenantId, nodeId });
for (const change of changes) console.log(change.type, change.deviceId, change.before, change.after);

better-compute api inventory.diff tenantId=$TENANT nodeId=node_… --format table

A diff lists, in this order, GPUs missing and added (by UUID), MIG layout changes (mode plus sorted instances), disks replaced (the same kernel name with another WWN or serial), and driver changes (NVIDIA driver or CUDA version). A section missing on either side yields no changes for it: a node that stopped reporting a section did not lose hardware.

GPU naming

NVML names are inconsistent across drivers and SKUs ("NVIDIA H100 80GB HBM3", "NVIDIA H100 PCIe"). Every GPU gets two normalized names:

• gpuProduct, the exact product in <MODEL>-<FORM>-<MEMORY>GB form, such as H100-SXM5-80GB;
• gpuModel, its family, such as H100.

Pools, policies and meters use gpuModel: a pool holds exactly one. Selectors may use either. The mapping is the versioned table gpuModels (the plan's gpu-models.json, version 2), and normalizeGpuProduct(name) applies it: upper case, punctuation collapsed to -, a leading NVIDIA dropped.

Version 2 added the parts real hosts reported that version 1 missed: data-center cards outside the flagship lines, and the workstation and GeForce cards that edge, lab and single-tenant hosts run.

A name the table does not know keeps its canonical form as gpuProduct, maps to gpuModel: 'other', and raises GpuModelUnknown, which keeps the node from scheduling until the table knows it.

Devices and atoms

The gpus section becomes devices with stable ids, so a reboot or a re-inventory never renames them:

Only atoms are booked. Slots exist in timeslice and mps pools: 4 per GPU for time-slicing and 10 for MPS by default (policy.sharing.replicas, 1 to 48, MPS's limit; a708aa6). A GPU whose MIG mode is about to change has every atom unbookable, with partitionPending: true, until the change completes. See fractional GPUs.

Device attributes are what selectors and scorers read: gpuProduct, gpuModel, vramGiB, index, pciBusId, numaNode, architecture, computeCapability, driverVersion, migCapable, nvlinkPeers, fabricCluster and fabricClique. A device that vanished while it still has live bookings is kept with healthy: false and deleted once its bookings end. A device id another node already owns is left alone and raises DuplicateIdentity.

Labels

Inventory also writes node labels with source inventory, the way NVIDIA's GPU Feature Discovery does: gpu.present, gpu.count, gpu.product, gpu.model, gpu.vramGiB, cuda.driver.major, nvidia.driver.version, mig.capable, fabric.domain, cpu.arch, cpu.vendor, cpu.model, os.distro, os.kernel and kvm.available. A label whose GPUs disagree reads mixed. nodes.label cannot overwrite them. Since badb8c2 the host's CPU and memory have labels too, for selectors and when: expressions such as numa.nodes > 1:

cpu.model is the reported model with trademark marks dropped and every other run of characters outside A-Z a-z 0-9 . _ turned into - (AMD EPYC 9654 96-Core Processor reads AMD-EPYC-9654-96-Core-Processor); a sandbox snapshot that includes memory restores only on hosts with an equal value (see snapshots).

The gateway writes one more kind at every Hello: each driver the daemon advertises in Hello.drivers becomes the label bc.dev/driver.<id>=<version> with source system (a missing version reads unknown; ids outside [a-z0-9._-] are skipped), and driver labels the daemon no longer advertises are dropped, in one statement that writes nothing when they did not change. Operators cannot overwrite them either. Placement can require them with PlacementRequest.nodeLabels, where '*' accepts any version (see placement). The vps module does this for every runtime: a hosted instance is placed only on hosts carrying bc.dev/driver.<runtime>, bcd's built-in libvirt-kvm and cloud-hypervisor included since 0fca413 (before, only plugin runtimes needed it).

Since 14ceb1e the same statement also writes one label per kind each advertised driver serves, bc.dev/driver-kind.<id>.<kind>=1 (a site agent hosting bcx-ipmi carries bc.dev/driver-kind.ipmi.power, .management and .inspect), and drops the ones a later Hello no longer lists. Kinds outside [a-z][a-z0-9-] (1 to 32 characters) are skipped, and at most 32 are read per driver. A placement or selector can require one with '*', as for driver labels. Metal reads them (a3ec8d0): before a verb starts a workflow, it keeps only the operations of the kinds the site's agents report for the node's BMC driver, and refuses the verb with UNSUPPORTED (reason bmc-driver-unsupported) when a step needs one they lack. The labels also make a driver metal's own table does not know checkable. See what metal checks before a workflow.

The hardware key

The hardware key identifies the machine rather than its operating system. It is the SHA-256 over the normalized product UUID, serial (system, else board, else chassis), sorted permanent MACs and sorted GPU UUIDs, after dropping firmware placeholders such as To Be Filled By O.E.M.. It is trusted when at least two independent identifiers survive, and trusted keys are unique across nodes. JoinService computes it from the join claims and the Inventory controller from inventory, with the same function, so both agree. A second node with the same trusted key (a cloned disk image, a moved board) gets DuplicateIdentity and keeps its key untrusted until an operator resolves it.

A copied node certificate shows up at the gateway instead: two machines with the same certificate keep taking the node's session from each other, each second Hello publishing security.node.duplicate-session. More than 3 such alternations within 10 minutes revoke the certificate (security.node.quarantined, reason duplicate-identity), and the real machine must re-join with a new token. See duplicate identity.

Conditions

A condition is { type, status, reason?, message?, since }, where status is true, false or unknown and since moves only when the status changes. nodes.get lists them. A condition with a NoSchedule policy also puts its type into noScheduleReasons, in the same transaction.

The gateway's daemonNodeConditions (b91dcff, 88219ea) lists the Kubernetes conditions, so they are written as node conditions. Since ebd9806 (security review 2) three of them, and since 4884a65 a fourth, are NoSchedule while true on a node that is not tenant-controlled (on a tenant-controlled node they stay advisory, as below), because each means the node cannot hand out GPUs safely:

• ForeignGpuExposure: another path (the GPU operator's device plugin, another GPU DRA driver, or NVIDIA_VISIBLE_DEVICES from unprivileged pods) reaches the node's GPUs, or bcd cannot rule one out, so a hold booked there would not be isolated. A change of it also wakes the node's started reservations, which then show Degraded / ForeignGpuExposure at once (4570dfe; see reservation conditions).
• PodResourcesUnavailable: bcd cannot tell which pods hold which devices. Placement has no NoSchedule for one binding only, so the whole node takes no work until the socket answers.
• KubernetesReleaseStuck: until ebd9806 it rode bcd's StatusReport.conditions and the node.status event only. Now it is recorded, and while a released allocation's devices are still held by a pod the node takes no more work, since the GPUs may still be in use; an operator looks first.
• KubernetesReleasesUnreadable: bcd could not read its pending releases, so every GPU of the node stays claimed on the node itself (no hold, VM, gpu.reset or diag.dcgm gets one). Until 4884a65 the gateway dropped it, so placement kept booking those GPUs and each new hold failed on the node with overlap. Now it is recorded and the node takes no work until the claim clears.

KubernetesReleased and KubernetesTaintsDrift stay informational. See Kubernetes workers.

FabricManagerDown, MPSDown, RebootRequired, HostManagedExternally and the Kubernetes conditions above come from bcd's host configuration and its Kubernetes side. The gateway's xidStatusHandler makes each the node's condition exactly as the last status report says, true while reported and false once it is not, and writes and publishes node.condition.changed only on a change. On a tenant-controlled node they are advisory: recorded, never a NoSchedule reason.

RebootRequired. Some host settings wait for a reboot: 1 GiB huge pages reserved on the kernel command line, CPUs isolated with the cpu unit, the iommu unit, an ECC mode set with gpu-settings, a driver swap while the GPUs are in use, and a MIG mode reset that failed. bcd never reboots on its own, and nothing in the control plane reboots on this condition: reboot the node yourself with node.reboot, within the pool's disruption budget. The node keeps running and taking work meanwhile.

The condition table in @better-compute/inventory (nodeConditionTypes) also defines PcieReplayHigh and ClockSkew. No controller raises those on nodes yet.

Node hygiene

Some faults no single report shows. The inventory module's inventory.hygiene reconciler, a periodic singleton, sweeps every node that is not deleted or retired every 5 minutes, 500 nodes per transaction:

Each raised condition adds a NoSchedule reason and publishes node.condition.changed (condition, status, reason, noSchedule); both go when the cause does. Only changes are written. The built-in reconciler uses the defaults; hygieneReconciler(options) and runHygiene(sql, options) take every, minDaemonVersion, protocolMax, certExpiringFraction, certExpiringMinSeconds and batch.

Host pressure

Since protocol v0.3, StatusReport.pressure carries the host's Linux pressure stall information ("some" avg10 from /proc/pressure/*, percent of wall time) and CPU steal (percent of the last report interval); it is absent when the kernel exposes no PSI. recordHostPressure(tx, nodeId, report) turns it into conditions:

Between the two levels a condition keeps its status, so a host hovering at a threshold does not flap. On nodes of vps-host pools the conditions are NoSchedule; elsewhere they only alert through node.condition.changed. Nothing is written while nothing changes, and nothing at all for a tenant-controlled node (see trust). The gateway runs it only when the host installs the status handler:

import { createGateway, xidStatusHandler } from 'better-compute/gateway';
import { hostPressureStatusHandler } from 'better-compute/inventory';

createGateway({ reports: { status: [xidStatusHandler, hostPressureStatusHandler({ events })] } });

hostPressureStatusHandler({ thresholds }) takes other raise thresholds (cpu, steal, memory, io). In tests, FakeDaemon.setPressure() puts a HostPressure into every status report.

bcd reports real values (daemon/internal/health). Every 30 seconds it reads the some avg10 line of /proc/pressure/cpu, memory and io, and computes CPU steal from the cpu line of /proc/stat as the share of CPU time stolen since the previous sample (0 on the first). It puts the sample into its latest StatusReport and sends that report again. A kernel without PSI yields no pressure at all, so no pressure condition is raised.

Health, quarantines and remediation

Not every fault should take a whole node away. A GPU with a double-bit ECC error is one bad atom on an eight-GPU machine, while a GPU that fell off the bus usually needs a reboot of the node.

• Device quarantine. A device-level fault writes a device_health row (source xid or inventory) and marks the device healthy: false. The atom leaves placement; the rest of the node keeps working. An XID quarantine stays until it is lifted with clearDeviceHealth(tx, deviceId, reason?) (a function of @better-compute/inventory) or by a successful remediation; inventory quarantines follow each new inventory.
• XID events. The daemon reports XIDs as Xid<n> conditions on a GPU in its status reports. With the gateway's xidStatusHandler installed (createGateway({ reports: { ...gatewayReportHandlers, status: xidStatusHandler } })), each one goes through recordXid. XIDs without a policy (13, 31, 43, 45 and other application faults) are only recorded as events, and XIDs from a tenant-controlled node change nothing. When an XID quarantines a GPU, the handler also wakes the node's started reservations at once (04f6bda), so a reservation holding that GPU turns Degraded with reason AtomUnhealthy, naming the GPU, within a pass, rather than at capacity's periodic recheck or provision's health reconciler. Nothing is released, and the node keeps its hold; on a Kubernetes worker bcd's device plugins or DRA driver stop offering the GPU as healthy (6f5a99f). tests/e2e/kubernetes-bindings.test.ts runs this with an XID 48 on a Kubernetes-bound GPU.
• XIDs in bcd. bcd follows /dev/kmsg from its end and parses driver lines of the form NVRM: Xid (PCI:0000:18:00): 79, .... It normalizes the bus id to NVML's form (00000000:18:00.0), finds the GPU with that bus id in its inventory, and raises condition Xid79 (status TRUE, device_id the GPU's UUID, the driver's text as the message), one per XID and GPU. The status report goes out at once, and the condition stays in every later report. An XID for a bus id the inventory does not know is only logged. bcd keeps these conditions for the rest of the boot: they are stored with the boot id in data_dir/faults.json (mode 0600), so a restart of bcd in the same boot reports them again even though it reads the kernel log from its end, while the first start after a reboot finds another boot id and drops them. The reboot that remediation runs therefore clears them, and so does a gpu.reset that leaves the GPU healthy, for that GPU (662fb55). An XID also makes bcd collect its inventory again about 2 seconds later, since the GPU's ECC and row-remap counters move with it.
• Remediation. Node-level faults queue the health reconciler of the provision module. It waits until the node holds no live allocation, then runs a node.reboot workflow (Xid79, Xid95) or a GPU reset (RowRemapPending; bcd's gpu.reset refuses a GPU that still has a live hold, see delivery). Runs share the pool's disruption budget (policy.disruptionBudget, 5% of the pool and at least one node, together with workflows and rollouts), and a circuit breaker pauses remediation for a pool when more than 10% of its nodes turned unhealthy within 10 minutes (pool.remediation.paused). A successful reboot clears the XID conditions and their quarantines; a GPU reset is confirmed by the next inventory. A failed or cancelled run, or two runs that did not clear the fault, put the node in maintenance for an operator (node.remediation.escalated). Progress is published as node.remediation.started and node.remediation.succeeded.

Acceptance checks

A node that just joined has not proven its hardware works. Acceptance runs the pool's checks before the node becomes schedulable, and again whenever you ask.

{
  "acceptance": [{ "kind": "diag.nvml" }],
  "expect": { "gpuCount": 8 }
}

The NodeLifecycle controller runs them for a node in adopting once it is connected:

1. It takes the pool's policy.acceptance list (each kind once). A site agent, a node without a pool, or an empty list passes at once (reason NoChecks).
2. It enqueues one command per check, with idempotency key acceptance:<nodeId>:<attempt>:<kind> and a 10-minute not_after, sets AcceptanceFailed to unknown (reason Running), and publishes node.acceptance.started with the attempt, command ids and checks.
3. diag.nvml checks the GPUs inventory found against policy.expect.gpuCount (else against the number found). Other kinds run with their default payload in this release; a kind that is not registered fails the attempt with UnsupportedCheck. On bcd (662fb55), diag.nvml fails nvml.init when the driver's management interface (nvidia-smi) did not answer, and per GPU fails ecc.uncorrectable, row-remap.pending, row-remap.failure, pcie.link (a link narrower than its maximum width) and xid.recent (an XID this boot), each with its reason; bcd advertises the nvml capability in Hello only when that data is there.
4. When every command settled: all succeeded (and diag.nvml reported passed) makes the node adopted, sets AcceptanceFailed to false (reason Passed), refreshes its IAM projection, asks the NodeSpec controller for generation 1 and publishes node.adopted. Otherwise AcceptanceFailed turns true with the reason (Expired, diag.nvml:<check>, <kind>:failed, …), the node stays adopting with the AcceptanceFailed NoSchedule reason, and node.acceptance.failed carries the reason and command id.

A failure is sticky: the controller does not retry on its own. Fix the cause, then call nodes.adopt (compute:nodes:provision), which starts a new attempt. On an adopted node, nodes.adopt moves it back to adopting, so it leaves placement until the checks pass again.

diag.nvml is the v0.1 check. The diag.dcgm kind comes with the metal module, whose commissioning profile runs it at level 2 (and the dedicated-root release at level 1); as a pool acceptance check it runs with its default payload until check arguments (with) are mapped per kind. From level 2, bcd refuses it gpu-busy for a GPU a hold or a VM is using (539c4ce).

On the node itself, bcd doctor prints one gpu line per GPU with its load, and warns on a pending row remap or MIG mode change and fails on failed remapping or uncorrectable ECC errors (f9d7af9; see bcd doctor).