# On Kubernetes (/docs/daemon/kubernetes) > Run bcd as a privileged DaemonSet on the GPU nodes of a cluster you already operate, so the control plane adopts them without owning their host configuration. Most machines run bcd the way [the install script](/docs/daemon/install) puts it there: a binary and a systemd unit on a host the control plane provisions and configures. A cluster you already operate is different. Its GPU nodes have a driver, a container toolkit and a MIG layout that something else manages, usually the NVIDIA GPU operator, and you want the control plane to adopt them without taking that over. For that, bcd runs inside the cluster as a DaemonSet: one privileged pod per GPU node, which joins the node once and then runs `bcd node`. This is the opposite direction from a [Kubernetes pool](/docs/guides/capacity/kubernetes). There, bcd is installed on machines the control plane owns, and the `k8s-join` config unit joins them to a cluster as workers. Here the cluster and its nodes exist first, and bcd runs inside the cluster to report their GPUs and hand the bound ones to pods. > **Built and dry-run in CI, never run in a cluster.** `tests/provider-kubernetes.test.ts`, `tests/provider-kubernetes-cli.test.ts` and bcd's `TestKubernetesDaemonSetContract` check the generated manifests and the `kubernetes manifest` command, and `TestHostedManager` runs the hosted device plugins against a fake kubelet. Since ac1ca4b the `container` job of `.github/workflows/daemon.yml` builds `daemon/Containerfile`, runs `bcd --version` and `kubectl version --client` from the image, renders the manifests (`scripts/k8s-render-manifests.mjs`, with placeholder pins and image digest) and has a [kind](https://kind.sigs.k8s.io) cluster's API server accept them: it creates the Namespace, then `kubectl apply --dry-run=server` the rest, which checks the schema and Pod Security admission. That is a dry run: no pod is scheduled, no image is pulled, and bcd has not run as a DaemonSet against a control plane anywhere yet. ## What you get [#what-you-get] * **One pod per selected node.** The DaemonSet runs on every node, or on those `--node-selector` names (for example `nvidia.com/gpu.present=true`, the label NVIDIA's GPU Feature Discovery sets), tolerates every taint by default, runs at `system-node-critical` priority and rolls one node at a time. * **The host's view, and only what bcd uses of it.** The `bcd` container is privileged, with host networking, and mounts exactly the host paths listed in [host mounts](#host-mounts): `/dev`, `/sys`, `/var/lib/bcd`, the CDI directories, the kubelet sockets bcd talks to, the kubelet's own credentials, and the container runtime's and NVIDIA toolkit's configuration, read-only. Never `/var/lib/kubelet` itself and never the host's `/etc/kubernetes`. The pod does not share the host's PID namespace (`hostPID: false`). It runs under the RuntimeClass `nvidia` (the GPU operator's; `nvidiaRuntimeClass`), and the `bcd` container asks for `NVIDIA_VISIBLE_DEVICES=all` with `NVIDIA_DRIVER_CAPABILITIES=utility` only, so bcd gets `nvidia-smi` and the driver's libraries to read and reset GPUs (b6d5548); workload pods still get CDI devices only. * **No Kubernetes identity of its own.** No service account token: bcd talks to its gateway, and what it does in the cluster (its Node's labels, deleting pods, DRA claims and slices) it does by running the image's `kubectl` with the kubelet's own credentials, which NodeRestriction confines to its Node (see [API access](#api-access)). * **One join per node.** The identity lives in the host's `/var/lib/bcd`, so it survives pod restarts and image rollouts; every later start finds it and needs no token. * **No host configuration.** `config.toml` sets `[host] units = false` by default. bcd applies no `NodeSpec.host` unit and reports the `host` section `UNSUPPORTED` with reason `HostManagedExternally` (see [host configuration](/docs/daemon/host-config)), which the gateway records as an informational node condition. The inventory still reports what the node has, so capacity books only the GPUs, MIG instances and huge pages that the cluster's own tooling set up. * **GPUs for pods.** bcd hands the GPUs of `binding: 'kubernetes'` reservations to the kubelet, with no `k8s-join` record: see [GPUs for pods](#gpus-for-pods). It gives pods only those GPUs only when bcd is the only source of GPUs on the node: see [GPU isolation](#gpu-isolation). * **No self-update and no reboots.** The configuration has no `[update] release_pins`, so bcd refuses every self-update and you upgrade by rolling out a new image; bcd never reboots a node. ## Host mounts [#host-mounts] The `bcd` container's host paths follow what `daemon/internal/kube` uses and nothing more (27f2c1b, a5eee84, d11810c). Mounting all of `/var/lib/kubelet` would give a compromised bcd pod every pod's volumes, secrets and service account tokens and the kubelet's checkpoints; the host's `/etc/kubernetes` holds `admin.conf` and the cluster CA's keys on a kubeadm control-plane node. `tests/provider-kubernetes.test.ts` asserts the exact set, with and without DRA. | Host path | Mode | hostPath type | Why | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | --------------------- | ---------------------------------------------------------------------------------------------------- | | `/dev`, `/sys` | read-write | `Directory` | GPUs, cgroups, device nodes | | `/var/lib/bcd` | read-write | `DirectoryOrCreate` | The node key, identity and journal, kept across pod restarts | | `/etc/cdi` | read-write | `DirectoryOrCreate` | bcd writes each hold's CDI spec | | `/var/run/cdi` | read-only | `DirectoryOrCreate` | bcd only reads NVIDIA's generated spec there | | `/var/lib/kubelet/device-plugins` | read-write | `Directory` | bcd's plugin sockets, and the kubelet's registration socket it dials | | `/var/lib/kubelet/pod-resources` | read-only | `Directory` | bcd only connects to the socket; a read-only bind mount still allows `connect()` | | `/var/lib/kubelet/plugins_registry`, `/var/lib/kubelet/plugins/gpu.bcd.dev` | read-write | `DirectoryOrCreate` | Only with `gpuApi: 'dra'`: the DRA registration and driver sockets, and the driver's `prepared.json` | | The kubelet's kubeconfig: `/etc/kubernetes/kubelet.conf` (kubeadm, the default), `/var/lib/rancher/k3s/agent/kubelet.kubeconfig` (k3s) or `/var/lib/rancher/rke2/agent/kubelet.kubeconfig` (rke2) | read-only | `File` | Mounted where bcd reads it, `/etc/kubernetes/kubelet.conf` (see [API access](#api-access)) | | The kubelet's certificate directories: `/var/lib/kubelet/pki` (kubeadm), `/var/lib/rancher/k3s/agent` (k3s) or `/var/lib/rancher/rke2/agent` (rke2) | read-only | `Directory` | Each at its own path, so the paths inside the kubeconfig resolve | | The container runtime's configuration, default `/etc/containerd/config.toml` | read-only | `File` or `Directory` | bcd reads which NVIDIA runtime the container runtime runs (see [GPU isolation](#gpu-isolation)) | | The NVIDIA toolkit's configuration, default `/usr/local/nvidia/toolkit/.config` | read-only | `Directory` | bcd reads whether the toolkit accepts `NVIDIA_VISIBLE_DEVICES` from unprivileged pods | Besides these, the pod has the ConfigMap's `config.toml`, a memory-backed `emptyDir` at `/run/bcd` and an `emptyDir` at `/tmp`. The container runtime's and toolkit's configuration are never created: a missing path stays missing, so the pod does not start on a node that lacks it rather than seeing an empty directory, and hosted bcd treats a runtime configuration it cannot read as `ForeignGpuExposure`. The kubelet credentials refuse a certificate directory that holds more than the kubelet's own: `/etc`, `/etc/kubernetes`, `/etc/kubernetes/pki`, `/var/lib/kubelet` or anything under it but `pki`, `/var/lib`, `/var` and `/`. Which paths these are is set with the options in [generate the manifests from TypeScript](#generate-the-manifests-from-typescript), or with `--kubelet-credentials`, `--container-runtime` and `--nvidia-toolkit`. ## Deploy it [#deploy-it] Since 3341e50 there is one path: the manifests are rendered for your deployment by the CLI, with `@better-compute/provider-kubernetes`'s `bcdDaemonSetManifests`, so they cannot drift from what bcd and the control plane expect. The hand-kept manifests that `deploy/kubernetes` used to carry are gone; its README walks the same steps. ### Build the image [#build-the-image] `daemon/Containerfile` builds the cgo-free host build (`-tags nonvml`; GPUs through `nvidia-smi`) on `debian:bookworm-slim` with `ca-certificates`, `nftables`, `kmod`, `pciutils` and `util-linux`. Build and push it, and note its digest: ```bash docker build -f daemon/Containerfile -t registry.example.com/better-compute/bcd:0.1.0 \ --build-arg VERSION=0.1.0 daemon docker push registry.example.com/better-compute/bcd:0.1.0 ``` `RELEASE_SEQUENCE` is a second build argument (default 0). No published image exists; build your own. Since 9bdd43a the image also carries `kubectl` v1.34.1 at `/usr/local/bin/kubectl`, which bcd runs for its Node's labels, for deleting pods of released devices and for DRA. The build downloads it from `dl.k8s.io` for `TARGETARCH` (default `amd64`; `arm64` too) and checks it against the pinned SHA-256 of that release; a mismatch, or another architecture, fails the build. bcd runs it as the kubelet (see [API access](#api-access)). CI builds this image for `linux/amd64` on every change to `daemon/**`, `proto/**` or `packages/provider-kubernetes/**` (the `container` job, ac1ca4b), but publishes nothing. ### Add the command [#add-the-command] The CLI never imports providers, so the application adds `kubernetes manifest` (a2eb28f) to its configuration: ```ts title="better-compute.config.ts" import { kubernetesManifestCommand } from '@better-compute/provider-kubernetes'; export const commands = [kubernetesManifestCommand()]; ``` An application may pass `kubernetesManifestCommand({ pins })` with the trust anchors from its own PKI bundle (`readPkiPins` of `@better-compute/pki`); otherwise the pins are flags. ### Render the manifests and the join token [#render-the-manifests-and-the-join-token] ```bash better-compute kubernetes manifest --tenant root --pool pool_01k… \ --image registry.example.com/better-compute/bcd@sha256:<64 hex digits> \ --serving-ca-pin sha256/… --join-signing-pin sha256/… --join-encryption-pin sha256/… \ --max-uses 8 --ttl 1h \ --secret-out bcd-join-secret.json --out bcd.json ``` The three pins are trust anchors, and they come only from you (d2cd813): the three flags together (all or none), from `better-compute pki show` on the control-plane host, or the application's `pins`. A run without them is refused before any call. The command never pins what the network says: from the control plane's `/.well-known/better-compute` (at `--control-plane`, which must be https, else the configured deployment in process) it takes only data that carries no trust, the gateway's URL and name and the join key, and it refuses a join key that does not match `--join-encryption-pin` before a token is minted. It renders the manifests first, so a bad image or pin fails before any token exists: a Namespace (`bcd-system`, labelled `pod-security.kubernetes.io/enforce: privileged`), a ConfigMap with `config.toml`, and the DaemonSet. Then it mints a multi-use join token with `hosted: 'kubernetes'` and writes it only as the join Secret's manifest, to `--secret-out`, a new file created 0600, never the output. The call is decided as `compute:join-tokens:create`. After writing the Secret, the command reminds you (ab9339b) to revoke the token (`join-tokens revoke `) and delete the Secret once every node has joined, with a fresh token for a later batch of nodes, and prints the [GPU isolation](#gpu-isolation) requirements. | Flag | Default | | | ----------------------------------------------------------------- | ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--tenant` | `BETTER_COMPUTE_TENANT` | The tenant that owns the pool (required) | | `--pool` | required | The pool the cluster's nodes join | | `--image` | required | The bcd image, pinned by digest (`registry/repo[:tag]@sha256:<64 hex>`) | | `--serving-ca-pin`, `--join-signing-pin`, `--join-encryption-pin` | `BCD_SERVING_CA_PIN`, `BCD_JOIN_SIGNING_PIN`, `BCD_JOIN_ENCRYPTION_PIN` | The gateway serving CA's and the JoinService keys' `sha256/…` pins, from the PKI bundle; all three, unless the application passes `pins` | | `--secret-out` | required | A new file (0600) for the join token Secret manifest | | `--out` | the output | A new file for the manifests | | `--control-plane` | the configuration | The API's public https URL, where the well-known document is served: the origin for `better-compute serve`, or the origin and the API's base path when compute is mounted inside an application | | `--namespace`, `--name` | `bcd-system`, `bcd` | | | `--node-selector` | every node | `key=value` pairs, such as `nvidia.com/gpu.present=true` | | `--host-units` | off | Let bcd apply `NodeSpec.host` units | | `--gpu-api` | `device-plugin` | `dra` for the DRA driver (see [GPUs for pods](#gpus-for-pods)) | | `--container-runtime` | `containerd` | `containerd` (`/etc/containerd/config.toml`), `k3s` (`/var/lib/rancher/k3s/agent/etc/containerd/config.toml`), `rke2` (`/var/lib/rancher/rke2/agent/etc/containerd/config.toml`) or `cri-o` (`/etc/crio/crio.conf` and `/etc/crio/crio.conf.d`): the nodes' container runtime, whose configuration is mounted read-only for the [GPU isolation](#gpu-isolation) check (`containerRuntime`; 3775f46) | | `--kubelet-credentials` | `kubeadm` | `kubeadm` (`/etc/kubernetes/kubelet.conf` and `/var/lib/kubelet/pki`), `k3s` (`/var/lib/rancher/k3s/agent/kubelet.kubeconfig` and `/var/lib/rancher/k3s/agent`) or `rke2` (the same under `/var/lib/rancher/rke2/agent`): the kubelet's kubeconfig and client certificates, mounted read-only, which bcd acts with (`kubeletCredentials`; 3d6f79a, a96b238) | | `--nvidia-toolkit` | `gpu-operator` | `gpu-operator` (`/usr/local/nvidia/toolkit/.config`) or `packages` (`/etc/nvidia-container-runtime`): where the NVIDIA container toolkit's configuration lives, mounted read-only for the same check (`nvidiaToolkit`; 3775f46) | | `--nvidia-runtime-class` | `nvidia` | The RuntimeClass bcd's own container runs under, which gives it `nvidia-smi` and the driver's libraries (`nvidiaRuntimeClass`; ab9339b); `none` for no RuntimeClass, when the image carries them | | `--ttl` | `24h` | How long the token admits nodes, 1 minute to 24 hours | | `--max-uses` | `100` | How many nodes it admits | | `--key` | a new UUID | The token's idempotency key, to retry the same request | **Distribution, runtime and toolkit.** Pick `--container-runtime` and `--kubelet-credentials` for the distribution the nodes run (`k3s` or `rke2` for those; `cri-o` for CRI-O, else `containerd`, with `kubeadm` credentials), and `--nvidia-toolkit packages` when the toolkit comes from NVIDIA's distribution packages rather than the GPU operator. Every path must exist on every selected node: the mounts are never created, so a wrong choice leaves the pod unable to start, and hosted bcd withdraws every GPU when it cannot read the runtime's configuration. Other paths (containerd's `conf.d`, another `toolkit.installDir`) need `bcdDaemonSetManifests` with `{ files, dirs }`, and so do kubelet credentials outside the three presets (`kubeletCredentials: { kubeconfig, certificateDirs }`). **Token hygiene** (d6f8549). A DaemonSet cannot use one single-use token per node, so the token is multi-use, and anyone who reads the Secret while it is live can join a machine to the pool. Give it `--max-uses` equal to the node count and a short `--ttl` (an hour covers a rollout). ### Apply [#apply] Configure the GPU stack as [GPU isolation](#gpu-isolation) requires, then apply the Secret first and the manifests after: ```bash kubectl apply -f bcd-join-secret.json kubectl apply -f bcd.json ``` On each node the `join` init container runs `bcd join` with the token Secret mounted read-only (mode 0400), as root with no capabilities, no privilege escalation and a read-only root filesystem; on a node that joined before it succeeds without spending a use. `bcd join --token-file` refuses a token file that group or others can read, one that is not a regular file (a Secret volume's symlink into `..data` is followed) and one over 4 KiB (c29bb16). The `bcd` container then runs `bcd node`, and never sees the token. The nodes appear in [`nodes.list`](/docs/reference/api/nodes#list) as `adopting`, as any joined machine does (see [joining nodes](/docs/guides/nodes/joining)). ### Revoke the token and delete the Secret [#revoke-the-token-and-delete-the-secret] Once every node has joined, revoke the token ([`joinTokens.revoke`](/docs/reference/api/join-tokens#revoke)) and delete the Secret: ```bash kubectl -n bcd-system delete secret ``` The pods keep running and restarting without it, since a joined node keeps its identity in `/var/lib/bcd`. Since 27f2c1b the join-token Secret volume is `optional: true`, so a pod of a joined node that restarts or is rescheduled after the Secret is gone starts instead of waiting in `ContainerCreating`: its `bcd join` finds the identity, exits 0 without a token file and spends nothing. For nodes added later, mint a new token and Secret, and remove both again. ## Trust [#trust] The cluster's admins have root on its machines: anyone who may run a privileged pod there can read what bcd holds. So a node that joins with a hosted token lands `tenant-controlled`, with a `node-tenant` certificate that never receives platform secrets, unless the pool's `policy.kubernetes.platformOperated` is `true`, which says the platform itself operates the cluster (a27156a). Since 259d82c that flag counts only on a pool the platform owns: the pool's owner must be the owner of its site, the site must not be `custom`, and, when the deployment names a root tenant, the owner must be it. A tenant's pool on a platform site cannot vouch for its own cluster by editing its policy. The token decides this, never what the node claims, and it only ever lowers trust: a node on a tenant's own `custom` site stays `tenant-owned`, and a reclaim through a hosted token lowers an existing `platform` node the same way. Replaying `joinTokens.create` with the same key and another `hosted` value is `CONFLICT`. A `tenant-controlled` node adds no capacity: placement excludes it, as for a [dedicated-root lease](/docs/guides/bare-metal/dedicated-root). For the platform's own clusters, give the pool `policy.kubernetes: { platformOperated: true }`, with none of `apiServer`, `caCertSha256` and `joinTokenRef`: those three together mean the machines should join a cluster through the `k8s-join` unit, and a partial set is `INVALID_INPUT` (7db110a). ### Host paths and Pod Security [#host-paths-and-pod-security] bcd keeps its node key and journal in the hostPath `/var/lib/bcd` (files 0600, owned by root). Any pod allowed to mount host paths as root can read them, and with the key it can act as the node. So restrict host paths for everything else (d6f8549): label workload namespaces with Pod Security Admission `baseline` or `restricted`, which both forbid hostPath and privileged pods. Only `bcd-system`, which the manifests label `pod-security.kubernetes.io/enforce: privileged`, and your platform's own system namespaces should run privileged. bcd's pod runs without `hostPID` and with no service account token. ### API access [#api-access] bcd reaches the Kubernetes API only by running the image's pinned `kubectl` as the kubelet: every call passes `--kubeconfig /etc/kubernetes/kubelet.conf`. Since 27f2c1b the DaemonSet mounts only that file there (read-only, hostPath type `File`, from the kubelet's own kubeconfig, which `--kubelet-credentials` or `kubeletCredentials` names), and the kubelet's client-certificate directories read-only at their own paths (default `/var/lib/kubelet/pki`), never the host's whole `/etc/kubernetes` (see [host mounts](#host-mounts)). The pod has `automountServiceAccountToken: false` and the manifests create no RBAC. That is deliberate (e7d0765, which reverted 9bdd43a's service-account credentials): RBAC cannot be scoped to one node, so a service account would have given every privileged bcd pod cluster-wide `patch` on Nodes and `delete` on pods, while the NodeRestriction admission plugin confines the kubelet's identity, and so bcd, to its own Node and that Node's pods. What bcd does with it: | Work | `kubectl` | | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | Its Node's `bcd.dev/*` labels | `get node`, `label node --overwrite` | | The reaper | `delete pod --wait=false --grace-period=10`, forced a minute later | | DRA (`gpu_api = "dra"`) | `api-versions`, `get resourceclaim`, `apply --server-side --field-manager=bcd` and `delete` of its `ResourceSlice` | | Other GPU DRA drivers ([GPU isolation](#gpu-isolation)) | `get resourceslices --field-selector spec.nodeName=`, at most once a minute (f88bcdb) | ## GPU isolation [#gpu-isolation] bcd hands a pod exactly the GPUs its reservation holds, and it can enforce that only when nothing else hands out GPUs on the node (d6f8549). With the NVIDIA GPU operator its defaults are unsafe here: its own device plugin advertises every GPU, and its container runtime gives an unprivileged pod any GPU it names in `NVIDIA_VISIBLE_DEVICES`, so any pod could take every GPU. Required: | Setting | Why | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `devicePlugin.enabled=false` in the GPU operator | bcd's device plugins (or its DRA driver) are the only source of GPUs for the kubelet | | `accept-nvidia-visible-devices-envvar-when-unprivileged = false` in the NVIDIA container toolkit's `config.toml` (the operator's `toolkit` env `ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED=false`) | Devices arrive by CDI only | | CDI specs generated with `--device-name-strategy=uuid` | bcd's device plugin then answers `nvidia.com/gpu=`; without that spec it answers `bcd.dev/gpu=` plus `NVIDIA_VISIBLE_DEVICES`, which is safe only with the setting above | The generated DaemonSet carries these requirements, and the Pod Security and token rules above, in its `bcd.dev/requires` annotation (`gpuIsolationRequirements`), since JSON manifests have no comments. bcd enforces the first two itself (9f860ad, widened in f88bcdb). It offers no GPUs while any of these holds: its device plugins stop, its DRA slice is deleted, and it reports the condition `ForeignGpuExposure` naming what to close. * **Another device plugin.** The kubelet can also allocate an `nvidia.com/*` resource (pod-resources' allocatable resources). * **Another GPU DRA driver.** The node's ResourceSlices, read with the kubelet's credentials at most once a minute, show a driver other than `gpu.bcd.dev` whose name marks it as a GPU driver (for example `gpu.nvidia.com`) offering devices. An API server without `resource.k8s.io`, or one that refuses the list, is no evidence either way. * **The toolkit the container runtime runs.** bcd reads the container runtime's configuration (containerd's `/etc/containerd/config.toml`, k3s's and RKE2's under `/var/lib/rancher//agent/etc/containerd/`, CRI-O's `/etc/crio/crio.conf`, and the `.toml` and `.conf` files of `/etc/containerd/conf.d`, `/etc/containerd/config.d` and `/etc/crio/crio.conf.d`) for a `BinaryName` or `runtime_path` that runs `nvidia-container-runtime` (the `.cdi` variant, which takes CDI names only, excepted). For each, it derives the configuration that binary reads: the GPU operator's `/toolkit/.config/nvidia-container-runtime/config.toml` when the binary sits in a `toolkit` directory, else `/etc/nvidia-container-runtime/config.toml`. That file must exist and set `accept-nvidia-visible-devices-envvar-when-unprivileged = false`: missing or unreadable, the toolkit takes its default and accepts `NVIDIA_VISIBLE_DEVICES` from any pod, so that is exposure too. * **Any known toolkit configuration.** `/etc/nvidia-container-runtime/config.toml` or the operator's `/usr/local/nvidia/toolkit/.config/nvidia-container-runtime/config.toml`, when present, accepts the variable (the key `true`, or absent, the toolkit's default). * **Hosted, no readable runtime configuration.** On a DaemonSet node bcd cannot rule the toolkit out when it can read none of the container runtime's configuration files, so it counts that as exposure. That is why the manifests mount them read-only as `File` or `Directory` and never `DirectoryOrCreate`, which would make a missing directory look empty (d11810c; see [host mounts](#host-mounts)). The gateway records the condition as a node condition (b91dcff). Since ebd9806 it is NoSchedule while true (advisory only on a tenant-controlled node), and the node's active reservations turn `Degraded` with reason `ForeignGpuExposure` (4570dfe), since their GPUs are no longer isolated. So a cluster with the GPU operator's defaults gets no GPUs through bcd, and its nodes take no new work, until you apply the settings above. `bcd doctor`'s `gpu-isolation` check (35534e5) runs the same checks. Since f88bcdb it also runs on a DaemonSet node (`[host] units = false`): it fails when no container runtime configuration is readable and when a toolkit configuration lets unprivileged pods pick GPUs. On a node bcd joined with `k8s-join` it also fails on another GPU DRA driver in the node's ResourceSlices while the kubelet is up. ## GPUs for pods [#gpus-for-pods] With `[host] units = false` the node is a worker because the cluster runs bcd there, not because `k8s-join` joined it, so bcd's Kubernetes side runs in hosted mode (1e5f96b). It needs no `k8s-join` record: the joined-mode files in `/var/lib/bcd-hostconfig` are not read here. * **Device plugins.** bcd runs its kubelet device plugins whenever `/var/lib/kubelet/device-plugins/kubelet.sock` exists, advertising the devices of the node's `binding: 'kubernetes'` holds as `bcd.dev/gpu`, `bcd.dev/gpu.mig` and `bcd.dev/gpu.slot`, as on a joined worker (see [the device plugin](/docs/guides/capacity/kubernetes#the-device-plugin)). * **Labels.** bcd keeps its Node's `bcd.dev/*` labels itself (`bcd.dev/node-id`, `bcd.dev/site`, `bcd.dev/gpu.count`, `bcd.dev/gpu.product`, `bcd.dev/gpu.driver`), naming the Node from `NODE_NAME` (the DaemonSet's `spec.nodeName` field reference) and using the kubelet's kubeconfig, `/etc/kubernetes/kubelet.conf` (see [API access](#api-access)). It updates them at most once a minute, touches only `bcd.dev/*`, sets no taints, and removes `bcd.dev/*` labels it no longer sets. * **The reaper.** Pods that keep a released device are deleted after 30 seconds with a 10-second grace period, and forced a minute later if they still hold it, as on a joined worker, DaemonSet pods included (8bcbe92), through the kubelet's pod-resources socket in the read-only `/var/lib/kubelet/pod-resources` mount; without that socket bcd reports `PodResourcesUnavailable`, which is NoSchedule for the whole node while true (ebd9806; advisory on a tenant-controlled node). A released allocation is reported to the control plane only once nothing holds its devices (cd5fc82, 8bcbe92), and one still held after 5 minutes raises `KubernetesReleaseStuck`, also NoSchedule while true (see [release](/docs/guides/capacity/kubernetes#release)). A hosted node always counts as a worker there, so a kubelet bcd cannot ask means the devices stay in use. The process backstop a joined worker has (a pod process with a released GPU's device node open) is off here: without `hostPID` bcd's `/proc` is its own pod's, where its own `nvidia-smi` would look like a holder (f88bcdb). A hosted node relies on pod-resources and, with DRA, the driver's `prepared.json` record in the `/var/lib/kubelet/plugins/gpu.bcd.dev` mount. The same holds for clearing an unreadable `releasing.json`, which keeps every GPU claimed and raises `KubernetesReleasesUnreadable` (NoSchedule while true, 4884a65) until no pod may hold a GPU of an allocation the node does not know (a18f423, b01130f; see [an unreadable releases file](/docs/guides/capacity/kubernetes#an-unreadable-releases-file)). * **Device plugin or DRA.** `[kubernetes] gpu_api` in `config.toml` chooses how the GPUs reach pods: `"device-plugin"` (the default) or `"dra"`, bcd's DRA driver `gpu.bcd.dev` (7d24836), never both. Any other value stops bcd from starting. `--gpu-api` and `bcdDaemonSetManifests({ gpuApi })` write it (e4b0c01). See [device plugin or DRA](/docs/guides/capacity/kubernetes#device-plugin-or-dra) for what the driver publishes. All of these run the `kubectl` the image carries, as the kubelet (see [API access](#api-access)). ## Generate the manifests from TypeScript [#generate-the-manifests-from-typescript] `kubernetes manifest` calls `bcdDaemonSetManifests` (2c9f953). An application can call it itself; it is not a VPS provider and creates no machines, it produces manifests you apply, and it mints no token. ```ts title="bcd-k8s.ts" import { writeFileSync } from 'node:fs'; import { bcdDaemonSetManifests, renderKubernetesManifests } from '@better-compute/provider-kubernetes'; const manifests = bcdDaemonSetManifests({ image: 'registry.example.com/better-compute/bcd@sha256:<64 hex digits>', gatewayUrl: 'https://gw.eu-central.example.com', siteId: 'site_01k…', pins: { servingCaPin, joinSigningPin, joinEncryptionPin }, // from your PKI bundle controlPlaneUrl: 'https://compute.example.com', // where /.well-known/better-compute is served, or joinEncryptionJwk joinTokenSecret: { name: 'bcd-join' }, nodeSelector: { 'nvidia.com/gpu.present': 'true' }, }); writeFileSync('bcd.json', renderKubernetesManifests(manifests)); // kubectl apply -f bcd.json ``` `bcdDaemonSetManifests` returns a Namespace, a ConfigMap (`config.toml`, and the pinned JoinService key when you pass `joinEncryptionJwk`) and a DaemonSet; `renderKubernetesManifests` turns them into one `v1` `List` for `kubectl apply -f`. It refuses (`INVALID_CONFIG`) an image not pinned by digest, a gateway or control-plane URL that is not https, malformed pins, names or resource quantities, a `gpuApi` other than `device-plugin` or `dra`, a JWK with a private part, and host paths that are not absolute or that [host mounts](#host-mounts) rules out. | Option | Default | | | ---------------------------------------- | ---------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `image` | required | `registry/repo[:tag]@sha256:<64 hex>` | | `gatewayUrl`, `siteId`, `pins` | required | As the pool's site and the PKI bundle give them | | `joinEncryptionJwk` or `controlPlaneUrl` | one required | The JoinService encryption key, or where `bcd join --server` fetches it; pinned either way | | `joinTokenSecret` | required | `{ name, key = 'token' }`, a Secret with a `hosted: 'kubernetes'` token | | `namespace`, `name`, `createNamespace` | `bcd-system`, `bcd`, `true` | | | `nodeSelector`, `tolerations` | every node, every taint | | | `priorityClassName` | `system-node-critical` | `null` for none | | `imagePullSecrets`, `resources` | none; requests 100m and 128Mi, limit 512Mi | | | `hostUnits` | `false` | `true` writes no `[host] units = false`, so the pool's host units apply | | `gpuApi` | none (bcd's default, `device-plugin`) | `'dra'` writes `[kubernetes] gpu_api = "dra"` (e4b0c01) | | `nvidiaRuntimeClass` | `nvidia` | The RuntimeClass that gives bcd's container the driver's user space; `null` when the image carries it (b6d5548) | | `kubeletCredentials` | `'kubeadm'`: `/etc/kubernetes/kubelet.conf` and `/var/lib/kubelet/pki` | A preset (3d6f79a; `kubeletCredentialPresets`): `'k3s'` (`/var/lib/rancher/k3s/agent/kubelet.kubeconfig`, `/var/lib/rancher/k3s/agent`) or `'rke2'` (the same under `/var/lib/rancher/rke2/agent`), or `{ kubeconfig, certificateDirs }` with 1 to 4 directories. The kubeconfig is mounted read-only at `/etc/kubernetes/kubelet.conf`, each directory at its own path; a directory holding more than the kubelet's certificates, or an unknown preset, is refused (27f2c1b) | | `containerRuntime` | `'containerd'` | `'k3s'`, `'rke2'`, `'cri-o'`, or `{ files, dirs }` (1 to 8 absolute paths in all): the runtime configuration mounted read-only (`containerRuntimeConfigs`; d11810c) | | `nvidiaToolkit` | `'gpu-operator'` | `'packages'`, or `{ files, dirs }` for another `toolkit.installDir`: the toolkit configuration mounted read-only (`nvidiaToolkitConfigs`; d11810c). A path both options name is refused | | `gatewayServerName`, `logLevel` | none, `info` | | The pod's `bcd` container gets `NODE_NAME` and has `bcd status` as its liveness probe, and the pod rolls when `config.toml` changes (annotation `bcd.dev/config-sha256`). Both sides check the contract: `scripts/daemon-k8s-contract.mjs` writes the generated pod to `daemon/cmd/bcd/testdata/k8s-daemonset.json`, and bcd's `TestKubernetesDaemonSetContract` parses its `config.toml` (the `gpu_api` key included) and command lines with bcd's own configuration parser and flag sets. ## Upgrades [#upgrades] Render the manifests again with the new image digest and apply them. The update strategy replaces one pod at a time. The identity and the journal are on the host, so a new pod resumes where the old one stopped. Rollouts and `daemon.upgrade` do not apply to these nodes, since bcd takes no self-update without release pins. ## Limits [#limits] * **No VPS instances.** The image carries neither `virsh` nor `cloud-hypervisor`, so these nodes run no instance runtime. * **No reboots.** `node.reboot` runs `systemctl reboot`, which the image does not have. * **The pod sees its own filesystem.** Only the mounted host paths are the host's; that is why the host converger is off by default. * **Host resources are the kubelet's.** A `binding: 'kubernetes'` hold gets no cgroup slice: the kubelet enforces its pods' requests and limits (e01d7ff). See [Kubernetes workers](/docs/guides/capacity/kubernetes#host-resources-and-pods). - [Kubernetes workers](/docs/guides/capacity/kubernetes): Reservations with binding kubernetes, the device plugin and DRA. - [Joining nodes](/docs/guides/nodes/joining): Join tokens, hosted tokens and trust. - [Installing bcd](/docs/daemon/install): config.toml and the files on a host.