# Kubernetes workers (/docs/guides/capacity/kubernetes) > Join a pool's machines to a Kubernetes cluster and hand a reservation's GPUs to their kubelets with binding kubernetes, through bcd's device plugin or DRA driver. PLAN.md §3.8 lets compute hand a cluster machines and GPUs and take them back, without scheduling a pod or running a control plane. A pool names a Kubernetes cluster; its machines join that cluster as workers; a reservation with `binding: 'kubernetes'` puts its GPUs on the node's kubelet, and bcd offers only those. The cluster's scheduler then places pods on them, and when the reservation ends the GPUs are withdrawn and pods still holding them are deleted. Pods get only the reserved GPUs when bcd is the only source of GPUs on the node; see [GPU isolation](#gpu-isolation). > **Built end to end in code, run only against fakes.** Every piece below is committed: the pool policy and the rendered `k8s-join` unit, the reservation binding, the node's join, the devices converger's Kubernetes holds, the device plugins and the DRA driver. The capacity tests cover the reservation side on PGlite, and `tests/e2e/kubernetes-bindings.test.ts` runs the binding through the dev stack and its FakeDaemons (8a084ca). The join unit is tested against a fake host, and the device plugins and DRA driver against a fake kubelet over real unix sockets. No test has joined a real cluster or started a pod on a GPU this way. ## Two ways into a cluster [#two-ways-into-a-cluster] | | A Kubernetes pool (this page) | bcd on Kubernetes | | ----------------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | | Who owns the machines | The control plane: bcd is installed on the host and applies its host configuration | The cluster: bcd runs inside it as a [DaemonSet](/docs/daemon/kubernetes) | | How they get into the cluster | The `k8s-join` config unit runs `kubeadm join` | They are cluster nodes already | | GPUs for pods | bcd's device plugins or DRA driver offer a `binding: 'kubernetes'` reservation's GPUs, chosen by `gpuApi` | The same, in hosted mode (1e5f96b), chosen by `[kubernetes] gpu_api` in `config.toml` |
## The pool [#the-pool] Give the pool a `policy.kubernetes`. Provision reads it when it builds each node's spec. ```ts title="create-pool.ts" await compute.api.pools.create(rootCredential, { tenantId: 'root', siteId: 'site_01k…', class: 'metal-host', gpuModel: 'H100', policy: { kubernetes: { apiServer: 'https://k8s.example.com:6443', caCertSha256: 'sha256:<64 hex digits>', joinTokenRef: 'secret://root/sec_01k…', // from secrets.put labels: { 'example.com/pool': 'h100' }, taints: ['example.com/gpu=reserved:NoSchedule'], kubernetesVersion: '1.34.1', gpuApi: 'device-plugin', platformOperated: true, // required for a join on a platform site (e4d14cd) }, }, idempotencyKey: 'pool-k8s-h100', }); ``` `apiServer`, `caCertSha256` and `joinTokenRef` go together (7db110a): all three say the machines join that cluster through the `k8s-join` unit; none says bcd already runs inside the cluster as a [DaemonSet](/docs/daemon/kubernetes); a partial set is `INVALID_INPUT`. With the three, provision turns the policy into the `k8s-join` config unit on every machine of the pool, adopted ones included (see [host configuration](/docs/daemon/host-config#k8s-join)). **A platform site's pool joins only a cluster the platform operates** (e4d14cd). A cluster's admins get root on its workers through privileged pods, while machines on a platform site keep their platform identity and receive brokered secrets. So `pools.create` and `pools.update` refuse the three join settings on a pool whose site is not `custom` unless `platformOperated` is `true` (`INVALID_INPUT`, "a platform site's pool joins only a cluster the platform operates; confirm it with policy.kubernetes.platformOperated: true"). A tenant's `custom` site needs no confirmation, since its nodes are tenant-owned anyway, and a pool without the join settings (bcd as a DaemonSet) is not affected. bcd's half of the rule is the `[kubernetes] allowed_clusters` list below. Since 37dcd02 the pools API checks the policy when `pools.create` or `pools.update` writes it: provision registers its host-unit checks as a `pool-policy` extension, so a bad `policy.kubernetes` is `INVALID_INPUT` at the API (`policy: ...` for a failure that is not already a compute error) instead of breaking every node's spec later. **The bootstrap token.** Store it with [`secrets.put`](/docs/reference/api/secrets) in the tenant that owns the machines or the pool, and name it by its `secret:///` reference. The secret broker releases it to a machine only while that machine's current spec names exactly this reference in `k8s-join.tokenRef`, and only when the reference belongs to the node's owner or the pool's owner (binding rule 3; see [sealed secrets](/docs/operations/security#sealed-secrets)). Since e4b0c01 provision refuses a `vault://` or `env://` reference here, since the broker reads the tenant from a `secret://` reference and would never release it.
## The machines [#the-machines] Each machine runs `bcd node` the usual way. The first pass of its host converger installs kubeadm, the kubelet and containerd where missing, prepares the kernel and containerd (CDI on, the systemd cgroup driver), fetches the token, and runs `kubeadm join`, registering the Node with the pool's labels and taints. From then on it keeps the kubelet running and the Node's labels in step, adding `bcd.dev/node-id`, `bcd.dev/site`, `bcd.dev/gpu.count`, `bcd.dev/gpu.product` and `bcd.dev/gpu.driver`. It never moves a node that is joined to another cluster. Taints are set only at registration (6ef198b): Kubernetes' NodeRestriction forbids a kubelet to change its own taints. When you change the pool's `taints` later, joined nodes report the condition `KubernetesTaintsDrift`, whose message is the `kubectl taint node … --overwrite …` command a cluster admin runs to apply them (see [host configuration](/docs/daemon/host-config#k8s-join)). Joining a cluster gives it root on the machine, so root decides which clusters a machine may join, in `config.toml`: ```toml title="/etc/bcd/config.toml" [kubernetes] allowed_clusters = "https://k8s.example.com:6443#sha256:<64 hex digits>" ``` Since 9f860ad the list is required whether `bcd node` runs as root or under [`bcd helper`](/docs/daemon#privilege-separation): with none, every join is refused, and `bcd doctor` fails a joined node without one. The details, files and failure cases are on [host configuration](/docs/daemon/host-config#k8s-join).
## The reservation [#the-reservation] Reserve with `binding: 'kubernetes'`: ```ts title="reserve.ts" await compute.api.reservations.create(credential, { tenantId: 'acme', poolId: 'pool_01k…', devices: [{ class: 'gpu', count: 2 }], duration: '8h', binding: 'kubernetes', resources: { vcpu: 32, memoryGiB: 256 }, idempotencyKey: 'train-k8s-2026-09-26', }); ``` From the CLI (ea36a94), preview first, then reserve: ```bash better-compute capacity --tenant acme --preview --pool pool_01k… --gpus 2 --for 8h --cpus 32 --memory 256 --kubernetes better-compute reserve --tenant acme --pool pool_01k… --gpus 2 --for 8h --cpus 32 --memory 256 --kubernetes ``` `@better-compute/console-kit`'s reservation form takes the same host CPU, memory and Kubernetes fields (e833433), and the console app's reservation page shows them (4533265): its **Give the GPUs to the pool's Kubernetes cluster** checkbox appears only for a pool the tenant owns that has `policy.kubernetes`. * The pool must have `policy.kubernetes`, or the request is `INVALID_INPUT` (`Pool … does not join a Kubernetes cluster (policy.kubernetes)`). * **Only the pool's owner books it** (3825ec7). The device plugins and the DRA driver hand bound GPUs to any pod of the cluster that asks and cannot tell tenants apart, so a tenant the pool is merely offered to could pay for GPUs that another tenant's pods then take. A pool's cluster is therefore its owner's: `binding: 'kubernetes'` from any other tenant is `INVALID_INPUT` (`Pool …'s Kubernetes cluster is its owner's: only the owner books binding kubernetes`), checked when the draft is written, before the IAM decision. Tenants the pool is offered to book holds on it instead (or instances, where the pool runs them). * Everything else is a normal reservation: the same decision, quota, placement and lifecycle. `resources` books host CPU, memory and 1 GiB pages with the GPUs, or the pool's per-GPU share when left out (see [host resources](/docs/guides/capacity/reservations#host-resources)). `capacity.preview` books the same host resources, so a preview that fits is a create that fits (17f600b). * It places only on nodes whose bcd advertises `binding.kubernetes` (and `devices.limits` when it books host resources), since an older bcd would drop the binding; see [older daemons](#older-daemons). * An agent can ask for it too: the [MCP](/docs/guides/clients/mcp) tools `reserve_gpus` and `preview_reservation` take `kubernetes: true` (binding `kubernetes`), with optional `vcpu` and `memoryGiB`. * At the start, the node's spec gains a device assignment with `BINDING_KUBERNETES`. bcd binds it like a hold: it checks the devices, writes `/etc/cdi/bcd-.json` and reports a node-signed `AllocationBound`, which makes the reservation `active`. The CDI spec carries the annotation `bcd.dev/binding: kubernetes` and, unlike a hold's, no `NVIDIA_VISIBLE_DEVICES`, since a pod may get only part of the reservation.
## The device plugin [#the-device-plugin] `bcd node` runs kubelet device plugins (`deviceplugin/v1beta1`) while the kubelet's socket exists on a machine bcd joined with `gpu` on, or, in [hosted mode](/docs/daemon/kubernetes#gpus-for-pods), on any node of the DaemonSet. Every 5 seconds, and at once after a bind or a release, it reads the Kubernetes holds from `/etc/cdi` and advertises exactly their devices, one resource per device class, each on its own socket in `/var/lib/kubelet/device-plugins`: | Resource | Devices | Socket | | ------------------ | ------------------------ | ------------------- | | `bcd.dev/gpu` | Whole GPUs | `bcd-gpu.sock` | | `bcd.dev/gpu.mig` | MIG instances | `bcd-gpu.mig.sock` | | `bcd.dev/gpu.slot` | Time-slice and MPS slots | `bcd-gpu.slot.sock` | * **Devices.** Since 9f860ad each device's kubelet id is `@`, the atom's CDI device name (the atom id with `/` as `_`) tied to the allocation that holds it, so the same GPU re-bound to another allocation is another device, and a pod that got it under the old allocation holds an id nobody advertises and is reaped. Its NUMA node comes from the inventory, and it is advertised unhealthy while its GPU has a pending or failed row remap, uncorrectable ECC errors since the driver loaded, or an XID condition. Since 6f5a99f an XID, or a reset that clears it, triggers a pass at once, so a GPU that goes bad is never offered as healthy. Pods are not deleted for health, since the device is still bound: only a release reaps. On the control plane's side an XID that quarantines the GPU makes the reservation `Degraded` (`AtomUnhealthy`, naming the GPU) within a pass and releases nothing (04f6bda); `tests/e2e/kubernetes-bindings.test.ts` checks it with an XID 48. * **Allocate.** For each container the plugin answers the CDI names `bcd.dev/gpu=` and, where NVIDIA's CDI spec (in `/etc/cdi` or `/var/run/cdi`) names the GPU or MIG instance, `nvidia.com/gpu=` too, so containerd injects the device nodes and driver libraries. It sets `NVIDIA_VISIBLE_DEVICES` to the GPU or MIG UUIDs and `BCD_ALLOCATION_IDS` to the allocations they belong to (also as the annotation `bcd.dev/allocation-ids`). A device that is no longer bound is refused, so the kubelet cannot start a container on a released GPU. With the NVIDIA GPU operator, let it generate its CDI specs with `--device-name-strategy=uuid`, so the `nvidia.com/gpu=` names exist. * **Preferred allocation.** Asked for a set of devices, the plugin prefers the ones it must include, then devices of the same allocation on the same NUMA node, then the allocation with the most free devices. * **Restarts.** When the kubelet restarts (its socket changes) or the plugin's socket disappears, the plugin registers again. When the node is no longer a worker with `gpu` on, the plugins stop. `bcd status` prints how many devices each resource offers the kubelet. A pod asks for the resource the way it asks for any extended resource: ```yaml title="train.yaml" apiVersion: v1 kind: Pod metadata: name: train spec: tolerations: - key: example.com/gpu operator: Equal value: reserved effect: NoSchedule containers: - name: main image: registry.example.com/train:latest resources: limits: bcd.dev/gpu: 2 ``` Compute decides which GPUs a node offers, not which pod takes them: any pod the cluster's scheduler places there may claim advertised devices. That is why only the pool's owner books `binding: 'kubernetes'` (3825ec7): the cluster and every GPU reserved into it belong to one tenant. Use the Node labels, taints and your cluster's own policies to keep a reservation's GPUs for the workloads it was made for; `BCD_ALLOCATION_IDS` tells a container which allocation it got. ## GPU isolation [#gpu-isolation] bcd's plugins and DRA driver offer exactly the bound GPUs, but pods are limited to them only when nothing else on the node hands out GPUs (d6f8549). Another device plugin, or a container runtime that gives a pod any GPU named in `NVIDIA_VISIBLE_DEVICES`, lets any pod take every GPU. With the NVIDIA GPU operator, set `devicePlugin.enabled=false`, set the container toolkit's `accept-nvidia-visible-devices-envvar-when-unprivileged = false` (CDI only), and generate CDI specs with `--device-name-strategy=uuid`. This applies to joined workers and to [bcd on Kubernetes](/docs/daemon/kubernetes#gpu-isolation) alike; the generated DaemonSet carries the requirements in its `bcd.dev/requires` annotation. Since 9f860ad bcd enforces this, and since f88bcdb (security review 2) it follows every path it knows of. bcd offers no GPUs at all when the kubelet can also allocate an `nvidia.com/*` resource (the GPU operator's device plugin, read from pod-resources' allocatable resources); when the node's ResourceSlices (read with the kubelet's credentials at most once a minute) show another GPU DRA driver offering devices, such as `gpu.nvidia.com`; or when a container toolkit configuration accepts `NVIDIA_VISIBLE_DEVICES` from unprivileged pods (the key `true`, or absent, which is the toolkit's default). The toolkit configuration that counts is the one the container runtime runs: bcd reads containerd's `BinaryName` (plain, k3s, RKE2 and `conf.d` drop-ins) and CRI-O's `runtime_path` for `nvidia-container-runtime` (not the `.cdi` variant) and checks that binary's configuration, the GPU operator's `/toolkit/.config/nvidia-container-runtime/config.toml` or `/etc/nvidia-container-runtime/config.toml`. A runtime that runs the toolkit with that file missing or unreadable is exposure, since the toolkit then takes its default; the two known files also count whenever they are present. As the cluster's DaemonSet, a node where no container runtime configuration is readable is exposure as well (see [bcd on Kubernetes](/docs/daemon/kubernetes#gpu-isolation)). In every case the device plugins stop, the DRA slice is deleted, and bcd reports the condition `ForeignGpuExposure` (reason `OtherGpuPath`) with a message naming what to close. On nodes bcd joins itself, the [`container-toolkit` unit](/docs/daemon/host-config#container-toolkit) sets the toolkit's keys so this check passes. The gateway records the condition as a node condition (b91dcff), and since ebd9806 it is NoSchedule while true: a GPU booked on that node would not be isolated from pods that take it by the other path, so the node takes no new work until the path is closed. On a tenant-controlled node the condition stays advisory and changes no NoSchedule reason, as all of bcd's conditions do. A change of the condition also wakes the node's started reservations, and while it is true every active reservation with an allocation on the node is `Degraded` with reason `ForeignGpuExposure` ("GPUs exposed outside bcd on node\_…", 4570dfe), whatever its binding, since its GPUs are no longer the tenant's alone. A lost binding still takes precedence, and the exposure takes precedence over `AtomUnhealthy`. Nothing is released: the reservation keeps its GPUs and the condition clears when bcd stops reporting the exposure. `bcd doctor` runs the same checks (35534e5, f88bcdb): on a node bcd joined with `gpu` on, its `gpu-isolation` check fails, with the same message, while a toolkit configuration lets unprivileged pods pick GPUs, and, while the kubelet runs, when another GPU DRA driver offers devices on the node; on a DaemonSet node it fails on the toolkit and on an unreadable runtime configuration. So you see why no GPU is offered before you go looking. ## Device plugin or DRA [#device-plugin-or-dra] Since 7d24836 bcd also has a Dynamic Resource Allocation driver, `gpu.bcd.dev`, for clusters that serve `resource.k8s.io`. A node runs the device plugins or the DRA driver, never both, since one GPU offered twice could go to two pods. The choice is `gpuApi`: | Where | Setting | | ------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | A joined worker | The `k8s-join` unit's `gpuApi`, rendered from `policy.kubernetes.gpuApi` (e4b0c01) and recorded in the join state | | A hosted node (DaemonSet) | `[kubernetes] gpu_api` in `config.toml`, written by `bcdDaemonSetManifests({ gpuApi })` or `kubernetes manifest --gpu-api` | `device-plugin` is the default. With `dra`, the same rule holds as for the device plugins: only what compute bound with `BINDING_KUBERNETES` is offered, and [GPU isolation](#gpu-isolation) applies the same way. * **Registration.** bcd serves the kubelet's plugin registration at `/var/lib/kubelet/plugins_registry/gpu.bcd.dev-reg.sock` (type `DRAPlugin`, versions `v1.DRAPlugin` and `v1beta1.DRAPlugin`) and the DRA service at `/var/lib/kubelet/plugins/gpu.bcd.dev/dra.sock`. * **The ResourceSlice.** One slice per node, `-gpu.bcd.dev` (driver `gpu.bcd.dev`, pool the Node's name), in the newest `resource.k8s.io` version the API server serves (`v1`, `v1beta2`, or `v1beta1` with its `basic` wrapper). It lists only the healthy devices of the node's Kubernetes holds, each named after its atom, `gpu-`, or `gpu--gi-ci` and `gpu--s` for MIG instances and slots, and since 9f860ad ending in 8 hex characters of its allocation, so a GPU re-bound to another allocation is another device; with the attributes `uuid`, `class` (`gpu`, `gpu.mig`, `gpu.slot`), `allocation`, `visibleDevice` (the GPU or MIG UUID) and `numaNode`. bcd applies it server-side (field manager `bcd`) with the kubelet's credentials when the devices change, and every 10 minutes otherwise, so a deleted slice comes back; with no device left it deletes the slice. * **Prepare.** `NodePrepareResources` reads each claim with the kubelet's credentials, checks that its UID is the one the kubelet asked for and that it is allocated, then reads the holds again and answers each `gpu.bcd.dev` device's CDI ids (`bcd.dev/gpu=` and, where NVIDIA's spec names it, `nvidia.com/gpu=`). A device that is not bound to Kubernetes on this node at that moment is refused, whatever the slice said, and so is an unhealthy one, even for a claim allocated before it went bad (6f5a99f); the slice drops a GPU the moment it goes bad. Since 3f73449 two more results are refused: one with `adminAccess`, which would reach a device already allocated to another claim ("admin access to gpu.bcd.dev devices is refused"), and a device already prepared for another claim, until that claim is unprepared ("is prepared for claim …"). Since 8bcbe92 the device-to-claim record is kept on disk, in `/var/lib/kubelet/plugins/gpu.bcd.dev/prepared.json` (0600, written through a temporary file and a rename), and read back when bcd starts, so it survives a restart. The check and the record happen under one lock, so two concurrent prepares cannot both take a device, and the record is on disk before the kubelet hears of the prepare; a record that cannot be written fails the claim. `NodeUnprepareResources` changes nothing on the host (the CDI specs are the holds'); it only removes the claim's devices from the record. A failed write of that is returned to the kubelet, which retries, and until then the devices stay prepared. * **The reaper.** A pod whose claim still holds one of this node's `gpu.bcd.dev` devices that compute released is deleted after the same 30 seconds as with the device plugins (fa8611e). Other drivers' claims are left alone. A pod then asks through a ResourceClaim for driver `gpu.bcd.dev`, selecting on the attributes. The DRA driver needs `kubectl` on the node and the kubelet's kubeconfig, and on a DaemonSet the generated manifests' `plugins_registry` and `plugins/gpu.bcd.dev` mounts, which they add only with `gpuApi: 'dra'` (see [host mounts](/docs/daemon/kubernetes#host-mounts)). ## Release [#release] When the reservation ends, the node's spec loses the assignment. bcd removes the CDI spec, and the device plugins or the DRA slice withdraw the devices in the same moment. Since cd5fc82 it reports `AllocationReleased` only once no pod holds a device of the allocation any more: * The pending release is written to `/releasing.json` before the CDI spec goes, so a crash never loses it. An unreadable file fails closed (a18f423, b01130f); see [an unreadable releases file](#an-unreadable-releases-file). * Until the report, the devices stay claimed on the node: no hold, VM, `gpu.reset`, `diag.dcgm` or host config unit gets them. The control plane waits for the report too (e7d922c): a released, bound Kubernetes reservation keeps its GPUs and host share booked, and billed, until the node reports `AllocationReleased` or the node is lost, though never billed past the allocation's own end (metering stops at the earlier of its release and its end). The reservation shows as released at once, and the spec withdraws the devices, but no other allocation can take them while pods still hold them. Since 0127ed7 `reservations.get` and `list` show such a released reservation with the computed condition `DevicesHeld` (reason `PodsHoldDevices`, `since` the release, the message naming the nodes whose pods still hold the devices), so the tenant can see why it is still booked and billed; it disappears once the release ends. The secret broker gives such a releasing allocation nothing. `tests/sim/kubernetes-host-resources-1k.test.ts` checks this (see [the control-plane simulator](/docs/operations/testing#the-control-plane-simulator)). * **A release stuck past the booking's end** (462e28f). Pods can hold the GPUs longer than the reservation was booked for (bcd then reports `KubernetesReleaseStuck`, below). While the node is reachable and no other allocation booked those devices next, the hosts pass carries the release's device bookings `stuckReleaseStepMs` (5 minutes) further each time they come within half a step of their end, locking the devices as the placer does, so a GPU pods still hold is never offered again. `finishStaleReleases` closes a release without the node's report only when the node is lost, or once its bookings ended because another allocation had booked those devices next (that one may then fail on the node until the pods let go). Only the device bookings are carried, never the allocation's own end, so billing stays capped there: the carried bookings keep the devices, not the meter, running. Since e7a56d7 (security review 3) the carry stops `maxStuckCarryMs` (24 hours) past the allocation's own end: the node has been `KubernetesReleaseStuck`, and NoSchedule, for that long, and `finishStaleReleases` then closes the release without the node's report. * **A bound report does not end a release** (e7a56d7). An `AllocationBound` that arrives while the release is pending (a node re-sending it, or a restarted bcd converging again before it received the withdrawing spec) keeps the release's pending mark, as does an [attested binding](/docs/guides/capacity/reservations#attested-bindings) that replaces the binding. Before, such a report erased the mark: the later `AllocationReleased` was then ignored, and the node's host share stayed claimed while the GPUs stayed booked to their original end. As a further guard, `finishStaleReleases` also releases any unreleased allocation whose reservation is terminal (`denied`, `released`, `failed`, `cancelled` or `expired`) and that is not waiting for a deferred release. * "No pod holds it" comes from the kubelet's pod-resources API (a device plugin id `@`, or a `gpu.bcd.dev` claim of this node's pool named for the allocation; 14111c1). Since 8bcbe92 (security review 2) two more sources can keep the devices in use, and every check fails closed: * **A prepared DRA claim.** A `gpu.bcd.dev` device named for the allocation that the DRA driver's `prepared.json` still lists is in use until the kubelet unprepares its claim, whatever pod-resources says (pod-resources reports DRA devices only with the `KubeletPodResourcesDynamicResources` feature gate). An unreadable record counts every device as prepared. * **A pod process.** A process in a `kubepods` cgroup with a released whole GPU's `/dev/nvidia` open (the minor from `/proc/driver/nvidia/gpus/*/information`) holds it, which covers what pod-resources does not account for: static pods, a lost kubelet checkpoint. It covers whole GPUs only, since MIG instances and slots share their GPU's device node, and it needs the host's `/proc`: on a DaemonSet node without `hostPID` it is off. * **The kubelet.** Any node that is or was a worker counts: a hosted node, one with a `k8s-join` record (even a released or unreadable one), or one with the kubelet's kubeconfig, `/var/lib/kubelet/config.yaml` or pod-resources socket on disk. There, a pod-resources socket that is missing or failing means the devices count as held. Only a node that never was a worker answers "not in use" without asking. * bcd checks again when the reaper sees such a pod gone, on every converge pass, and at the latest 30 seconds later. A release pending for more than 5 minutes raises the condition `KubernetesReleaseStuck` (reason `PodsHoldDevices`), naming the allocation and its devices. Since ebd9806 the gateway records it as a node condition that is NoSchedule while true (advisory on a tenant-controlled node): the GPUs may still be in use, so an operator looks at the node before it takes more work. * An allocation bound again before its release was reported is dropped from the pending list without a report. Pods still holding a released `bcd.dev/*` device (the kubelet's PodResources API names them), or a released `gpu.bcd.dev` device through a claim, are deleted once they have held it for 30 seconds, with `kubectl delete pod --wait=false --grace-period=10` and the kubelet's credentials; a pod that still holds the device a minute later is deleted with `--grace-period=0 --force` (9f860ad). DaemonSet pods are not spared. Since 8bcbe92 the reaper keeps running after the node stops offering GPUs, when compute released the machine, `gpu` is off, or the kubelet is restarting: on a node that is or was a worker it deletes any pod holding a `bcd.dev` device compute does not bind there, and with nothing offered, that is every such device. bcd never deletes a pod after a failed read of the holds. While bcd holds devices for Kubernetes and the kubelet's pod-resources socket is missing or failing, so released devices cannot be reclaimed, it reports the condition `PodResourcesUnavailable`. Since ebd9806 that condition is NoSchedule for the whole node while true (advisory on a tenant-controlled node): bcd cannot tell which pods hold which devices, and placement has no NoSchedule for one binding only, so the node takes no work of any kind until the socket answers again. **Taking the machine back.** When a node's spec no longer names `k8s-join` (the pool dropped Kubernetes, or the machine left the pool), bcd cordons its Node, deletes its running and pending workload pods with each pod's own grace period capped at 30 seconds (`--grace-period=30`, 8bcbe92), records the release, which stops the device plugins or DRA driver, and reports `KubernetesReleased` (9f860ad, 028b81c). It leaves static mirror pods, finished pods and pods of other Nodes alone, and DaemonSet pods too unless they ask for a `bcd.dev/` resource or a resource claim; the reaper, which keeps running, forces what still holds a device. A cluster that forbids a node to cordon itself still gets drained. bcd does not reset the kubelet: the machine stays a cordoned member until an operator removes it, and the unit's return uncordons it. ### An unreadable releases file [#an-unreadable-releases-file] If `/releasing.json` (by default `/var/lib/bcd/releasing.json`) exists but cannot be read or parsed, bcd no longer knows which releases were pending, and a pod of one of them may still hold its GPUs. Until a18f423 the file was read as empty, so those GPUs lost their claim. Since then (security review) bcd fails closed: * It renames the file to `releasing.json.unreadable-` (kept, never deleted) and writes the marker `/releasing.unreadable` with the time and the error. * While the marker exists, every GPU of the node counts as claimed by "(pending Kubernetes releases that could not be read)": no hold, VM, `gpu.reset` or `diag.dcgm` gets one. The node's own live holds keep theirs as usual. * The node reports the condition `KubernetesReleasesUnreadable` (reason `ReleasesFileUnreadable`), whose message names the marker's path. The marker clears itself on a converge pass once the kube manager's `HeldOutside` (38f90fb, b01130f) answers that no pod may hold a bcd GPU of an allocation outside the node's live set: the allocations the spec wants, the node's holds and its pending releases. It checks the same sources as a release, each failing closed: * device plugin ids `@` in pod-resources, and `gpu.bcd.dev` devices, prepared in the DRA driver's record or claimed in pod-resources, whose name carries another allocation's tag; * pod processes with a GPU open, asked only about the whole GPUs no live allocation holds (in whole or in part, as a MIG instance or slot), since a process cannot be told apart by allocation; * a node that is or was a worker whose kubelet cannot be asked, or whose DRA record is unreadable, counts as held. A node that never was a worker answers "not held" at once. When the node's holds cannot be read, nothing is decided that pass. An operator can also clear the claim by removing the marker (by default `/var/lib/bcd/releasing.unreadable`) once they know no pod holds a GPU of a lost release. In the control plane the condition is a node condition, NoSchedule while true (4884a65; advisory on a `tenant-controlled` node, like the other Kubernetes conditions): the gateway writes it to `node_conditions`, publishes `node.condition.changed` on each change, and placement offers the node for no work until the claim clears. Before 4884a65 the gateway dropped it, so placement kept booking the node's GPUs and each new hold failed on the node with `AllocationFailed` (reason `overlap`). See [node conditions](/docs/guides/nodes/inventory-and-health). ## Host resources and pods [#host-resources-and-pods] A `kubernetes` binding's pods live in the kubelet's own cgroups (`kubepods.slice`), where the kubelet enforces their requests and limits. So since e01d7ff bcd creates no `bcd-alloc-.slice` for such a hold and names no cgroup parent in its CDI spec or env. The booked `resources` still count: bcd's host admission counts a Kubernetes hold's memory and huge pages beside the VMs and other holds, and refuses a hold that does not fit with `resources-exhausted`, which the control plane re-places (see [device assignments](/docs/daemon/delivery#device-assignments)). Size the pods' requests to the reservation yourself; compute does not write them. ## Older daemons [#older-daemons] A bcd from before 409f783 does not know `BINDING_KUBERNETES` or `DeviceAssignment.limits` and drops both as unknown fields: the GPUs of a Kubernetes reservation would never reach the kubelet (they would fail as an unsupported binding only after placement), and a host share would run unbounded. A GPU host's bcd now advertises two capabilities in its `Hello`, which the gateway writes as system labels (see [capabilities](/docs/daemon/sessions#hello)): | Capability | System label | Capacity requires it for | | -------------------- | ------------------------------- | -------------------------------------------------------------------------------------------------------------------- | | `binding.kubernetes` | `bc.dev/cap.binding.kubernetes` | Any reservation with `binding: 'kubernetes'` | | `devices.limits` | `bc.dev/cap.devices.limits` | Any reservation that books host resources, which a Kubernetes reservation does by default (the pool's per-GPU share) | Placement enforces them on create, on re-placing a pending reservation, on repairing a booking and in `capacity.preview`, so in a pool whose nodes run mixed bcd versions a Kubernetes reservation lands only on upgraded nodes, and waits `pending` when none fits. An operator-set label of the same name does not count. A host without GPUs advertises neither. ## What is missing [#what-is-missing] * **No real cluster has run this.** Neither the device plugins nor the DRA driver has served a real kubelet. * **Pod requests are not derived from the reservation.** The kubelet enforces what the pods ask for, not what the reservation booked. - [Host configuration](/docs/daemon/host-config#k8s-join): The k8s-join unit in detail. - [Reservations](/docs/guides/capacity/reservations): Requests, resources and bindings. - [On Kubernetes](/docs/daemon/kubernetes): bcd as a DaemonSet in an existing cluster.