Kubernetes workers

PLAN.md §3.8 lets compute hand a cluster machines and GPUs and take them back, without scheduling a pod or running a control plane. A pool names a Kubernetes cluster; its machines join that cluster as workers; a reservation with binding: 'kubernetes' puts its GPUs on the node's kubelet, and bcd offers only those. The cluster's scheduler then places pods on them, and when the reservation ends the GPUs are withdrawn and pods still holding them are deleted. Pods get only the reserved GPUs when bcd is the only source of GPUs on the node; see GPU isolation.

Two ways into a cluster

await compute.api.pools.create(rootCredential, {
  tenantId: 'root',
  siteId: 'site_01k…',
  class: 'metal-host',
  gpuModel: 'H100',
  policy: {
    kubernetes: {
      apiServer: 'https://k8s.example.com:6443',
      caCertSha256: 'sha256:<64 hex digits>',
      joinTokenRef: 'secret://root/sec_01k…', // from secrets.put
      labels: { 'example.com/pool': 'h100' },
      taints: ['example.com/gpu=reserved:NoSchedule'],
      kubernetesVersion: '1.34.1',
      gpuApi: 'device-plugin',
      platformOperated: true, // required for a join on a platform site (e4d14cd)
    },
  },
  idempotencyKey: 'pool-k8s-h100',
});

[kubernetes]
allowed_clusters = "https://k8s.example.com:6443#sha256:<64 hex digits>"

await compute.api.reservations.create(credential, {
  tenantId: 'acme',
  poolId: 'pool_01k…',
  devices: [{ class: 'gpu', count: 2 }],
  duration: '8h',
  binding: 'kubernetes',
  resources: { vcpu: 32, memoryGiB: 256 },
  idempotencyKey: 'train-k8s-2026-09-26',
});

better-compute capacity --tenant acme --preview --pool pool_01k… --gpus 2 --for 8h --cpus 32 --memory 256 --kubernetes
better-compute reserve --tenant acme --pool pool_01k… --gpus 2 --for 8h --cpus 32 --memory 256 --kubernetes

The device plugin

bcd node runs kubelet device plugins (deviceplugin/v1beta1) while the kubelet's socket exists on a machine bcd joined with gpu on, or, in hosted mode, on any node of the DaemonSet. Every 5 seconds, and at once after a bind or a release, it reads the Kubernetes holds from /etc/cdi and advertises exactly their devices, one resource per device class, each on its own socket in /var/lib/kubelet/device-plugins:

• Devices. Since 9f860ad each device's kubelet id is <atom>@<allocation>, the atom's CDI device name (the atom id with / as _) tied to the allocation that holds it, so the same GPU re-bound to another allocation is another device, and a pod that got it under the old allocation holds an id nobody advertises and is reaped. Its NUMA node comes from the inventory, and it is advertised unhealthy while its GPU has a pending or failed row remap, uncorrectable ECC errors since the driver loaded, or an XID condition. Since 6f5a99f an XID, or a reset that clears it, triggers a pass at once, so a GPU that goes bad is never offered as healthy. Pods are not deleted for health, since the device is still bound: only a release reaps. On the control plane's side an XID that quarantines the GPU makes the reservation Degraded (AtomUnhealthy, naming the GPU) within a pass and releases nothing (04f6bda); tests/e2e/kubernetes-bindings.test.ts checks it with an XID 48.
• Allocate. For each container the plugin answers the CDI names bcd.dev/gpu=<device> and, where NVIDIA's CDI spec (in /etc/cdi or /var/run/cdi) names the GPU or MIG instance, nvidia.com/gpu=<uuid> too, so containerd injects the device nodes and driver libraries. It sets NVIDIA_VISIBLE_DEVICES to the GPU or MIG UUIDs and BCD_ALLOCATION_IDS to the allocations they belong to (also as the annotation bcd.dev/allocation-ids). A device that is no longer bound is refused, so the kubelet cannot start a container on a released GPU. With the NVIDIA GPU operator, let it generate its CDI specs with --device-name-strategy=uuid, so the nvidia.com/gpu=<uuid> names exist.
• Preferred allocation. Asked for a set of devices, the plugin prefers the ones it must include, then devices of the same allocation on the same NUMA node, then the allocation with the most free devices.
• Restarts. When the kubelet restarts (its socket changes) or the plugin's socket disappears, the plugin registers again. When the node is no longer a worker with gpu on, the plugins stop.

bcd status prints how many devices each resource offers the kubelet.

A pod asks for the resource the way it asks for any extended resource:

apiVersion: v1
kind: Pod
metadata:
  name: train
spec:
  tolerations:
    - key: example.com/gpu
      operator: Equal
      value: reserved
      effect: NoSchedule
  containers:
    - name: main
      image: registry.example.com/train:latest
      resources:
        limits:
          bcd.dev/gpu: 2

Compute decides which GPUs a node offers, not which pod takes them: any pod the cluster's scheduler places there may claim advertised devices. That is why only the pool's owner books binding: 'kubernetes' (3825ec7): the cluster and every GPU reserved into it belong to one tenant. Use the Node labels, taints and your cluster's own policies to keep a reservation's GPUs for the workloads it was made for; BCD_ALLOCATION_IDS tells a container which allocation it got.

GPU isolation

bcd's plugins and DRA driver offer exactly the bound GPUs, but pods are limited to them only when nothing else on the node hands out GPUs (d6f8549). Another device plugin, or a container runtime that gives a pod any GPU named in NVIDIA_VISIBLE_DEVICES, lets any pod take every GPU. With the NVIDIA GPU operator, set devicePlugin.enabled=false, set the container toolkit's accept-nvidia-visible-devices-envvar-when-unprivileged = false (CDI only), and generate CDI specs with --device-name-strategy=uuid. This applies to joined workers and to bcd on Kubernetes alike; the generated DaemonSet carries the requirements in its bcd.dev/requires annotation.

Since 9f860ad bcd enforces this, and since f88bcdb (security review 2) it follows every path it knows of. bcd offers no GPUs at all when the kubelet can also allocate an nvidia.com/* resource (the GPU operator's device plugin, read from pod-resources' allocatable resources); when the node's ResourceSlices (read with the kubelet's credentials at most once a minute) show another GPU DRA driver offering devices, such as gpu.nvidia.com; or when a container toolkit configuration accepts NVIDIA_VISIBLE_DEVICES from unprivileged pods (the key true, or absent, which is the toolkit's default). The toolkit configuration that counts is the one the container runtime runs: bcd reads containerd's BinaryName (plain, k3s, RKE2 and conf.d drop-ins) and CRI-O's runtime_path for nvidia-container-runtime (not the .cdi variant) and checks that binary's configuration, the GPU operator's <installDir>/toolkit/.config/nvidia-container-runtime/config.toml or /etc/nvidia-container-runtime/config.toml. A runtime that runs the toolkit with that file missing or unreadable is exposure, since the toolkit then takes its default; the two known files also count whenever they are present. As the cluster's DaemonSet, a node where no container runtime configuration is readable is exposure as well (see bcd on Kubernetes). In every case the device plugins stop, the DRA slice is deleted, and bcd reports the condition ForeignGpuExposure (reason OtherGpuPath) with a message naming what to close. On nodes bcd joins itself, the container-toolkit unit sets the toolkit's keys so this check passes. The gateway records the condition as a node condition (b91dcff), and since ebd9806 it is NoSchedule while true: a GPU booked on that node would not be isolated from pods that take it by the other path, so the node takes no new work until the path is closed. On a tenant-controlled node the condition stays advisory and changes no NoSchedule reason, as all of bcd's conditions do. A change of the condition also wakes the node's started reservations, and while it is true every active reservation with an allocation on the node is Degraded with reason ForeignGpuExposure ("GPUs exposed outside bcd on node_…", 4570dfe), whatever its binding, since its GPUs are no longer the tenant's alone. A lost binding still takes precedence, and the exposure takes precedence over AtomUnhealthy. Nothing is released: the reservation keeps its GPUs and the condition clears when bcd stops reporting the exposure.

bcd doctor runs the same checks (35534e5, f88bcdb): on a node bcd joined with gpu on, its gpu-isolation check fails, with the same message, while a toolkit configuration lets unprivileged pods pick GPUs, and, while the kubelet runs, when another GPU DRA driver offers devices on the node; on a DaemonSet node it fails on the toolkit and on an unreadable runtime configuration. So you see why no GPU is offered before you go looking.

Device plugin or DRA

Since 7d24836 bcd also has a Dynamic Resource Allocation driver, gpu.bcd.dev, for clusters that serve resource.k8s.io. A node runs the device plugins or the DRA driver, never both, since one GPU offered twice could go to two pods. The choice is gpuApi:

device-plugin is the default. With dra, the same rule holds as for the device plugins: only what compute bound with BINDING_KUBERNETES is offered, and GPU isolation applies the same way.

• Registration. bcd serves the kubelet's plugin registration at /var/lib/kubelet/plugins_registry/gpu.bcd.dev-reg.sock (type DRAPlugin, versions v1.DRAPlugin and v1beta1.DRAPlugin) and the DRA service at /var/lib/kubelet/plugins/gpu.bcd.dev/dra.sock.
• The ResourceSlice. One slice per node, <node>-gpu.bcd.dev (driver gpu.bcd.dev, pool the Node's name), in the newest resource.k8s.io version the API server serves (v1, v1beta2, or v1beta1 with its basic wrapper). It lists only the healthy devices of the node's Kubernetes holds, each named after its atom, gpu-<uuid>, or gpu-<uuid>-gi<g>-ci<c> and gpu-<uuid>-s<k> for MIG instances and slots, and since 9f860ad ending in 8 hex characters of its allocation, so a GPU re-bound to another allocation is another device; with the attributes uuid, class (gpu, gpu.mig, gpu.slot), allocation, visibleDevice (the GPU or MIG UUID) and numaNode. bcd applies it server-side (field manager bcd) with the kubelet's credentials when the devices change, and every 10 minutes otherwise, so a deleted slice comes back; with no device left it deletes the slice.
• Prepare. NodePrepareResources reads each claim with the kubelet's credentials, checks that its UID is the one the kubelet asked for and that it is allocated, then reads the holds again and answers each gpu.bcd.dev device's CDI ids (bcd.dev/gpu=<device> and, where NVIDIA's spec names it, nvidia.com/gpu=<uuid>). A device that is not bound to Kubernetes on this node at that moment is refused, whatever the slice said, and so is an unhealthy one, even for a claim allocated before it went bad (6f5a99f); the slice drops a GPU the moment it goes bad. Since 3f73449 two more results are refused: one with adminAccess, which would reach a device already allocated to another claim ("admin access to gpu.bcd.dev devices is refused"), and a device already prepared for another claim, until that claim is unprepared ("is prepared for claim …"). Since 8bcbe92 the device-to-claim record is kept on disk, in /var/lib/kubelet/plugins/gpu.bcd.dev/prepared.json (0600, written through a temporary file and a rename), and read back when bcd starts, so it survives a restart. The check and the record happen under one lock, so two concurrent prepares cannot both take a device, and the record is on disk before the kubelet hears of the prepare; a record that cannot be written fails the claim. NodeUnprepareResources changes nothing on the host (the CDI specs are the holds'); it only removes the claim's devices from the record. A failed write of that is returned to the kubelet, which retries, and until then the devices stay prepared.
• The reaper. A pod whose claim still holds one of this node's gpu.bcd.dev devices that compute released is deleted after the same 30 seconds as with the device plugins (fa8611e). Other drivers' claims are left alone.

A pod then asks through a ResourceClaim for driver gpu.bcd.dev, selecting on the attributes. The DRA driver needs kubectl on the node and the kubelet's kubeconfig, and on a DaemonSet the generated manifests' plugins_registry and plugins/gpu.bcd.dev mounts, which they add only with gpuApi: 'dra' (see host mounts).

Release

When the reservation ends, the node's spec loses the assignment. bcd removes the CDI spec, and the device plugins or the DRA slice withdraw the devices in the same moment. Since cd5fc82 it reports AllocationReleased only once no pod holds a device of the allocation any more:

• The pending release is written to <data_dir>/releasing.json before the CDI spec goes, so a crash never loses it. An unreadable file fails closed (a18f423, b01130f); see an unreadable releases file.
• Until the report, the devices stay claimed on the node: no hold, VM, gpu.reset, diag.dcgm or host config unit gets them. The control plane waits for the report too (e7d922c): a released, bound Kubernetes reservation keeps its GPUs and host share booked, and billed, until the node reports AllocationReleased or the node is lost, though never billed past the allocation's own end (metering stops at the earlier of its release and its end). The reservation shows as released at once, and the spec withdraws the devices, but no other allocation can take them while pods still hold them. Since 0127ed7 reservations.get and list show such a released reservation with the computed condition DevicesHeld (reason PodsHoldDevices, since the release, the message naming the nodes whose pods still hold the devices), so the tenant can see why it is still booked and billed; it disappears once the release ends. The secret broker gives such a releasing allocation nothing. tests/sim/kubernetes-host-resources-1k.test.ts checks this (see the control-plane simulator).
• A release stuck past the booking's end (462e28f). Pods can hold the GPUs longer than the reservation was booked for (bcd then reports KubernetesReleaseStuck, below). While the node is reachable and no other allocation booked those devices next, the hosts pass carries the release's device bookings stuckReleaseStepMs (5 minutes) further each time they come within half a step of their end, locking the devices as the placer does, so a GPU pods still hold is never offered again. finishStaleReleases closes a release without the node's report only when the node is lost, or once its bookings ended because another allocation had booked those devices next (that one may then fail on the node until the pods let go). Only the device bookings are carried, never the allocation's own end, so billing stays capped there: the carried bookings keep the devices, not the meter, running. Since e7a56d7 (security review 3) the carry stops maxStuckCarryMs (24 hours) past the allocation's own end: the node has been KubernetesReleaseStuck, and NoSchedule, for that long, and finishStaleReleases then closes the release without the node's report.
• A bound report does not end a release (e7a56d7). An AllocationBound that arrives while the release is pending (a node re-sending it, or a restarted bcd converging again before it received the withdrawing spec) keeps the release's pending mark, as does an attested binding that replaces the binding. Before, such a report erased the mark: the later AllocationReleased was then ignored, and the node's host share stayed claimed while the GPUs stayed booked to their original end. As a further guard, finishStaleReleases also releases any unreleased allocation whose reservation is terminal (denied, released, failed, cancelled or expired) and that is not waiting for a deferred release.
• "No pod holds it" comes from the kubelet's pod-resources API (a device plugin id <atom>@<allocation>, or a gpu.bcd.dev claim of this node's pool named for the allocation; 14111c1). Since 8bcbe92 (security review 2) two more sources can keep the devices in use, and every check fails closed:
  • A prepared DRA claim. A gpu.bcd.dev device named for the allocation that the DRA driver's prepared.json still lists is in use until the kubelet unprepares its claim, whatever pod-resources says (pod-resources reports DRA devices only with the KubeletPodResourcesDynamicResources feature gate). An unreadable record counts every device as prepared.
  • A pod process. A process in a kubepods cgroup with a released whole GPU's /dev/nvidia<minor> open (the minor from /proc/driver/nvidia/gpus/*/information) holds it, which covers what pod-resources does not account for: static pods, a lost kubelet checkpoint. It covers whole GPUs only, since MIG instances and slots share their GPU's device node, and it needs the host's /proc: on a DaemonSet node without hostPID it is off.
  • The kubelet. Any node that is or was a worker counts: a hosted node, one with a k8s-join record (even a released or unreadable one), or one with the kubelet's kubeconfig, /var/lib/kubelet/config.yaml or pod-resources socket on disk. There, a pod-resources socket that is missing or failing means the devices count as held. Only a node that never was a worker answers "not in use" without asking.
• bcd checks again when the reaper sees such a pod gone, on every converge pass, and at the latest 30 seconds later. A release pending for more than 5 minutes raises the condition KubernetesReleaseStuck (reason PodsHoldDevices), naming the allocation and its devices. Since ebd9806 the gateway records it as a node condition that is NoSchedule while true (advisory on a tenant-controlled node): the GPUs may still be in use, so an operator looks at the node before it takes more work.
• An allocation bound again before its release was reported is dropped from the pending list without a report.

Pods still holding a released bcd.dev/* device (the kubelet's PodResources API names them), or a released gpu.bcd.dev device through a claim, are deleted once they have held it for 30 seconds, with kubectl delete pod --wait=false --grace-period=10 and the kubelet's credentials; a pod that still holds the device a minute later is deleted with --grace-period=0 --force (9f860ad). DaemonSet pods are not spared. Since 8bcbe92 the reaper keeps running after the node stops offering GPUs, when compute released the machine, gpu is off, or the kubelet is restarting: on a node that is or was a worker it deletes any pod holding a bcd.dev device compute does not bind there, and with nothing offered, that is every such device. bcd never deletes a pod after a failed read of the holds. While bcd holds devices for Kubernetes and the kubelet's pod-resources socket is missing or failing, so released devices cannot be reclaimed, it reports the condition PodResourcesUnavailable. Since ebd9806 that condition is NoSchedule for the whole node while true (advisory on a tenant-controlled node): bcd cannot tell which pods hold which devices, and placement has no NoSchedule for one binding only, so the node takes no work of any kind until the socket answers again.

Taking the machine back. When a node's spec no longer names k8s-join (the pool dropped Kubernetes, or the machine left the pool), bcd cordons its Node, deletes its running and pending workload pods with each pod's own grace period capped at 30 seconds (--grace-period=30, 8bcbe92), records the release, which stops the device plugins or DRA driver, and reports KubernetesReleased (9f860ad, 028b81c). It leaves static mirror pods, finished pods and pods of other Nodes alone, and DaemonSet pods too unless they ask for a bcd.dev/ resource or a resource claim; the reaper, which keeps running, forces what still holds a device. A cluster that forbids a node to cordon itself still gets drained. bcd does not reset the kubelet: the machine stays a cordoned member until an operator removes it, and the unit's return uncordons it.

An unreadable releases file

If <data_dir>/releasing.json (by default /var/lib/bcd/releasing.json) exists but cannot be read or parsed, bcd no longer knows which releases were pending, and a pod of one of them may still hold its GPUs. Until a18f423 the file was read as empty, so those GPUs lost their claim. Since then (security review) bcd fails closed:

• It renames the file to releasing.json.unreadable-<unix seconds> (kept, never deleted) and writes the marker <data_dir>/releasing.unreadable with the time and the error.
• While the marker exists, every GPU of the node counts as claimed by "(pending Kubernetes releases that could not be read)": no hold, VM, gpu.reset or diag.dcgm gets one. The node's own live holds keep theirs as usual.
• The node reports the condition KubernetesReleasesUnreadable (reason ReleasesFileUnreadable), whose message names the marker's path.

The marker clears itself on a converge pass once the kube manager's HeldOutside (38f90fb, b01130f) answers that no pod may hold a bcd GPU of an allocation outside the node's live set: the allocations the spec wants, the node's holds and its pending releases. It checks the same sources as a release, each failing closed:

• device plugin ids <atom>@<allocation> in pod-resources, and gpu.bcd.dev devices, prepared in the DRA driver's record or claimed in pod-resources, whose name carries another allocation's tag;
• pod processes with a GPU open, asked only about the whole GPUs no live allocation holds (in whole or in part, as a MIG instance or slot), since a process cannot be told apart by allocation;
• a node that is or was a worker whose kubelet cannot be asked, or whose DRA record is unreadable, counts as held. A node that never was a worker answers "not held" at once.

When the node's holds cannot be read, nothing is decided that pass. An operator can also clear the claim by removing the marker (by default /var/lib/bcd/releasing.unreadable) once they know no pod holds a GPU of a lost release.

In the control plane the condition is a node condition, NoSchedule while true (4884a65; advisory on a tenant-controlled node, like the other Kubernetes conditions): the gateway writes it to node_conditions, publishes node.condition.changed on each change, and placement offers the node for no work until the claim clears. Before 4884a65 the gateway dropped it, so placement kept booking the node's GPUs and each new hold failed on the node with AllocationFailed (reason overlap). See node conditions.

Host resources and pods

A kubernetes binding's pods live in the kubelet's own cgroups (kubepods.slice), where the kubelet enforces their requests and limits. So since e01d7ff bcd creates no bcd-alloc-<allocation>.slice for such a hold and names no cgroup parent in its CDI spec or env. The booked resources still count: bcd's host admission counts a Kubernetes hold's memory and huge pages beside the VMs and other holds, and refuses a hold that does not fit with resources-exhausted, which the control plane re-places (see device assignments). Size the pods' requests to the reservation yourself; compute does not write them.

Older daemons

A bcd from before 409f783 does not know BINDING_KUBERNETES or DeviceAssignment.limits and drops both as unknown fields: the GPUs of a Kubernetes reservation would never reach the kubelet (they would fail as an unsupported binding only after placement), and a host share would run unbounded. A GPU host's bcd now advertises two capabilities in its Hello, which the gateway writes as system labels (see capabilities):

Placement enforces them on create, on re-placing a pending reservation, on repairing a booking and in capacity.preview, so in a pool whose nodes run mixed bcd versions a Kubernetes reservation lands only on upgraded nodes, and waits pending when none fits. An operator-set label of the same name does not count. A host without GPUs advertises neither.

What is missing

• No real cluster has run this. Neither the device plugins nor the DRA driver has served a real kubelet.
• Pod requests are not derived from the reservation. The kubelet enforces what the pods ask for, not what the reservation booked.