On Kubernetes

Most machines run bcd the way the install script puts it there: a binary and a systemd unit on a host the control plane provisions and configures. A cluster you already operate is different. Its GPU nodes have a driver, a container toolkit and a MIG layout that something else manages, usually the NVIDIA GPU operator, and you want the control plane to adopt them without taking that over. For that, bcd runs inside the cluster as a DaemonSet: one privileged pod per GPU node, which joins the node once and then runs bcd node.

This is the opposite direction from a Kubernetes pool. There, bcd is installed on machines the control plane owns, and the k8s-join config unit joins them to a cluster as workers. Here the cluster and its nodes exist first, and bcd runs inside the cluster to report their GPUs and hand the bound ones to pods.

What you get

• One pod per selected node. The DaemonSet runs on every node, or on those --node-selector names (for example nvidia.com/gpu.present=true, the label NVIDIA's GPU Feature Discovery sets), tolerates every taint by default, runs at system-node-critical priority and rolls one node at a time.
• The host's view, and only what bcd uses of it. The bcd container is privileged, with host networking, and mounts exactly the host paths listed in host mounts: /dev, /sys, /var/lib/bcd, the CDI directories, the kubelet sockets bcd talks to, the kubelet's own credentials, and the container runtime's and NVIDIA toolkit's configuration, read-only. Never /var/lib/kubelet itself and never the host's /etc/kubernetes. The pod does not share the host's PID namespace (hostPID: false). It runs under the RuntimeClass nvidia (the GPU operator's; nvidiaRuntimeClass), and the bcd container asks for NVIDIA_VISIBLE_DEVICES=all with NVIDIA_DRIVER_CAPABILITIES=utility only, so bcd gets nvidia-smi and the driver's libraries to read and reset GPUs (b6d5548); workload pods still get CDI devices only.
• No Kubernetes identity of its own. No service account token: bcd talks to its gateway, and what it does in the cluster (its Node's labels, deleting pods, DRA claims and slices) it does by running the image's kubectl with the kubelet's own credentials, which NodeRestriction confines to its Node (see API access).
• One join per node. The identity lives in the host's /var/lib/bcd, so it survives pod restarts and image rollouts; every later start finds it and needs no token.
• No host configuration. config.toml sets [host] units = false by default. bcd applies no NodeSpec.host unit and reports the host section UNSUPPORTED with reason HostManagedExternally (see host configuration), which the gateway records as an informational node condition. The inventory still reports what the node has, so capacity books only the GPUs, MIG instances and huge pages that the cluster's own tooling set up.
• GPUs for pods. bcd hands the GPUs of binding: 'kubernetes' reservations to the kubelet, with no k8s-join record: see GPUs for pods. It gives pods only those GPUs only when bcd is the only source of GPUs on the node: see GPU isolation.
• No self-update and no reboots. The configuration has no [update] release_pins, so bcd refuses every self-update and you upgrade by rolling out a new image; bcd never reboots a node.

Host mounts

The bcd container's host paths follow what daemon/internal/kube uses and nothing more (27f2c1b, a5eee84, d11810c). Mounting all of /var/lib/kubelet would give a compromised bcd pod every pod's volumes, secrets and service account tokens and the kubelet's checkpoints; the host's /etc/kubernetes holds admin.conf and the cluster CA's keys on a kubeadm control-plane node. tests/provider-kubernetes.test.ts asserts the exact set, with and without DRA.

Besides these, the pod has the ConfigMap's config.toml, a memory-backed emptyDir at /run/bcd and an emptyDir at /tmp. The container runtime's and toolkit's configuration are never created: a missing path stays missing, so the pod does not start on a node that lacks it rather than seeing an empty directory, and hosted bcd treats a runtime configuration it cannot read as ForeignGpuExposure. The kubelet credentials refuse a certificate directory that holds more than the kubelet's own: /etc, /etc/kubernetes, /etc/kubernetes/pki, /var/lib/kubelet or anything under it but pki, /var/lib, /var and /. Which paths these are is set with the options in generate the manifests from TypeScript, or with --kubelet-credentials, --container-runtime and --nvidia-toolkit.

Deploy it

Since 3341e50 there is one path: the manifests are rendered for your deployment by the CLI, with @better-compute/provider-kubernetes's bcdDaemonSetManifests, so they cannot drift from what bcd and the control plane expect. The hand-kept manifests that deploy/kubernetes used to carry are gone; its README walks the same steps.

docker build -f daemon/Containerfile -t registry.example.com/better-compute/bcd:0.1.0 \
  --build-arg VERSION=0.1.0 daemon
docker push registry.example.com/better-compute/bcd:0.1.0

import { kubernetesManifestCommand } from '@better-compute/provider-kubernetes';

export const commands = [kubernetesManifestCommand()];

better-compute kubernetes manifest --tenant root --pool pool_01k… \
  --image registry.example.com/better-compute/bcd@sha256:<64 hex digits> \
  --serving-ca-pin sha256/… --join-signing-pin sha256/… --join-encryption-pin sha256/… \
  --max-uses 8 --ttl 1h \
  --secret-out bcd-join-secret.json --out bcd.json

kubectl apply -f bcd-join-secret.json
kubectl apply -f bcd.json

kubectl -n bcd-system delete secret <name>

Trust

The cluster's admins have root on its machines: anyone who may run a privileged pod there can read what bcd holds. So a node that joins with a hosted token lands tenant-controlled, with a node-tenant certificate that never receives platform secrets, unless the pool's policy.kubernetes.platformOperated is true, which says the platform itself operates the cluster (a27156a). Since 259d82c that flag counts only on a pool the platform owns: the pool's owner must be the owner of its site, the site must not be custom, and, when the deployment names a root tenant, the owner must be it. A tenant's pool on a platform site cannot vouch for its own cluster by editing its policy. The token decides this, never what the node claims, and it only ever lowers trust: a node on a tenant's own custom site stays tenant-owned, and a reclaim through a hosted token lowers an existing platform node the same way. Replaying joinTokens.create with the same key and another hosted value is CONFLICT.

A tenant-controlled node adds no capacity: placement excludes it, as for a dedicated-root lease. For the platform's own clusters, give the pool policy.kubernetes: { platformOperated: true }, with none of apiServer, caCertSha256 and joinTokenRef: those three together mean the machines should join a cluster through the k8s-join unit, and a partial set is INVALID_INPUT (7db110a).

Host paths and Pod Security

bcd keeps its node key and journal in the hostPath /var/lib/bcd (files 0600, owned by root). Any pod allowed to mount host paths as root can read them, and with the key it can act as the node. So restrict host paths for everything else (d6f8549): label workload namespaces with Pod Security Admission baseline or restricted, which both forbid hostPath and privileged pods. Only bcd-system, which the manifests label pod-security.kubernetes.io/enforce: privileged, and your platform's own system namespaces should run privileged. bcd's pod runs without hostPID and with no service account token.

API access

bcd reaches the Kubernetes API only by running the image's pinned kubectl as the kubelet: every call passes --kubeconfig /etc/kubernetes/kubelet.conf. Since 27f2c1b the DaemonSet mounts only that file there (read-only, hostPath type File, from the kubelet's own kubeconfig, which --kubelet-credentials or kubeletCredentials names), and the kubelet's client-certificate directories read-only at their own paths (default /var/lib/kubelet/pki), never the host's whole /etc/kubernetes (see host mounts). The pod has automountServiceAccountToken: false and the manifests create no RBAC. That is deliberate (e7d0765, which reverted 9bdd43a's service-account credentials): RBAC cannot be scoped to one node, so a service account would have given every privileged bcd pod cluster-wide patch on Nodes and delete on pods, while the NodeRestriction admission plugin confines the kubelet's identity, and so bcd, to its own Node and that Node's pods. What bcd does with it:

GPU isolation

bcd hands a pod exactly the GPUs its reservation holds, and it can enforce that only when nothing else hands out GPUs on the node (d6f8549). With the NVIDIA GPU operator its defaults are unsafe here: its own device plugin advertises every GPU, and its container runtime gives an unprivileged pod any GPU it names in NVIDIA_VISIBLE_DEVICES, so any pod could take every GPU. Required:

The generated DaemonSet carries these requirements, and the Pod Security and token rules above, in its bcd.dev/requires annotation (gpuIsolationRequirements), since JSON manifests have no comments.

bcd enforces the first two itself (9f860ad, widened in f88bcdb). It offers no GPUs while any of these holds: its device plugins stop, its DRA slice is deleted, and it reports the condition ForeignGpuExposure naming what to close.

• Another device plugin. The kubelet can also allocate an nvidia.com/* resource (pod-resources' allocatable resources).
• Another GPU DRA driver. The node's ResourceSlices, read with the kubelet's credentials at most once a minute, show a driver other than gpu.bcd.dev whose name marks it as a GPU driver (for example gpu.nvidia.com) offering devices. An API server without resource.k8s.io, or one that refuses the list, is no evidence either way.
• The toolkit the container runtime runs. bcd reads the container runtime's configuration (containerd's /etc/containerd/config.toml, k3s's and RKE2's under /var/lib/rancher/<k3s|rke2>/agent/etc/containerd/, CRI-O's /etc/crio/crio.conf, and the .toml and .conf files of /etc/containerd/conf.d, /etc/containerd/config.d and /etc/crio/crio.conf.d) for a BinaryName or runtime_path that runs nvidia-container-runtime (the .cdi variant, which takes CDI names only, excepted). For each, it derives the configuration that binary reads: the GPU operator's <installDir>/toolkit/.config/nvidia-container-runtime/config.toml when the binary sits in a toolkit directory, else /etc/nvidia-container-runtime/config.toml. That file must exist and set accept-nvidia-visible-devices-envvar-when-unprivileged = false: missing or unreadable, the toolkit takes its default and accepts NVIDIA_VISIBLE_DEVICES from any pod, so that is exposure too.
• Any known toolkit configuration. /etc/nvidia-container-runtime/config.toml or the operator's /usr/local/nvidia/toolkit/.config/nvidia-container-runtime/config.toml, when present, accepts the variable (the key true, or absent, the toolkit's default).
• Hosted, no readable runtime configuration. On a DaemonSet node bcd cannot rule the toolkit out when it can read none of the container runtime's configuration files, so it counts that as exposure. That is why the manifests mount them read-only as File or Directory and never DirectoryOrCreate, which would make a missing directory look empty (d11810c; see host mounts).

The gateway records the condition as a node condition (b91dcff). Since ebd9806 it is NoSchedule while true (advisory only on a tenant-controlled node), and the node's active reservations turn Degraded with reason ForeignGpuExposure (4570dfe), since their GPUs are no longer isolated. So a cluster with the GPU operator's defaults gets no GPUs through bcd, and its nodes take no new work, until you apply the settings above.

bcd doctor's gpu-isolation check (35534e5) runs the same checks. Since f88bcdb it also runs on a DaemonSet node ([host] units = false): it fails when no container runtime configuration is readable and when a toolkit configuration lets unprivileged pods pick GPUs. On a node bcd joined with k8s-join it also fails on another GPU DRA driver in the node's ResourceSlices while the kubelet is up.

GPUs for pods

With [host] units = false the node is a worker because the cluster runs bcd there, not because k8s-join joined it, so bcd's Kubernetes side runs in hosted mode (1e5f96b). It needs no k8s-join record: the joined-mode files in /var/lib/bcd-hostconfig are not read here.

• Device plugins. bcd runs its kubelet device plugins whenever /var/lib/kubelet/device-plugins/kubelet.sock exists, advertising the devices of the node's binding: 'kubernetes' holds as bcd.dev/gpu, bcd.dev/gpu.mig and bcd.dev/gpu.slot, as on a joined worker (see the device plugin).
• Labels. bcd keeps its Node's bcd.dev/* labels itself (bcd.dev/node-id, bcd.dev/site, bcd.dev/gpu.count, bcd.dev/gpu.product, bcd.dev/gpu.driver), naming the Node from NODE_NAME (the DaemonSet's spec.nodeName field reference) and using the kubelet's kubeconfig, /etc/kubernetes/kubelet.conf (see API access). It updates them at most once a minute, touches only bcd.dev/*, sets no taints, and removes bcd.dev/* labels it no longer sets.
• The reaper. Pods that keep a released device are deleted after 30 seconds with a 10-second grace period, and forced a minute later if they still hold it, as on a joined worker, DaemonSet pods included (8bcbe92), through the kubelet's pod-resources socket in the read-only /var/lib/kubelet/pod-resources mount; without that socket bcd reports PodResourcesUnavailable, which is NoSchedule for the whole node while true (ebd9806; advisory on a tenant-controlled node). A released allocation is reported to the control plane only once nothing holds its devices (cd5fc82, 8bcbe92), and one still held after 5 minutes raises KubernetesReleaseStuck, also NoSchedule while true (see release). A hosted node always counts as a worker there, so a kubelet bcd cannot ask means the devices stay in use. The process backstop a joined worker has (a pod process with a released GPU's device node open) is off here: without hostPID bcd's /proc is its own pod's, where its own nvidia-smi would look like a holder (f88bcdb). A hosted node relies on pod-resources and, with DRA, the driver's prepared.json record in the /var/lib/kubelet/plugins/gpu.bcd.dev mount. The same holds for clearing an unreadable releasing.json, which keeps every GPU claimed and raises KubernetesReleasesUnreadable (NoSchedule while true, 4884a65) until no pod may hold a GPU of an allocation the node does not know (a18f423, b01130f; see an unreadable releases file).
• Device plugin or DRA. [kubernetes] gpu_api in config.toml chooses how the GPUs reach pods: "device-plugin" (the default) or "dra", bcd's DRA driver gpu.bcd.dev (7d24836), never both. Any other value stops bcd from starting. --gpu-api and bcdDaemonSetManifests({ gpuApi }) write it (e4b0c01). See device plugin or DRA for what the driver publishes.

All of these run the kubectl the image carries, as the kubelet (see API access).

Generate the manifests from TypeScript

kubernetes manifest calls bcdDaemonSetManifests (2c9f953). An application can call it itself; it is not a VPS provider and creates no machines, it produces manifests you apply, and it mints no token.

import { writeFileSync } from 'node:fs';
import { bcdDaemonSetManifests, renderKubernetesManifests } from '@better-compute/provider-kubernetes';

const manifests = bcdDaemonSetManifests({
  image: 'registry.example.com/better-compute/bcd@sha256:<64 hex digits>',
  gatewayUrl: 'https://gw.eu-central.example.com',
  siteId: 'site_01k…',
  pins: { servingCaPin, joinSigningPin, joinEncryptionPin }, // from your PKI bundle
  controlPlaneUrl: 'https://compute.example.com', // where /.well-known/better-compute is served, or joinEncryptionJwk
  joinTokenSecret: { name: 'bcd-join' },
  nodeSelector: { 'nvidia.com/gpu.present': 'true' },
});
writeFileSync('bcd.json', renderKubernetesManifests(manifests)); // kubectl apply -f bcd.json

bcdDaemonSetManifests returns a Namespace, a ConfigMap (config.toml, and the pinned JoinService key when you pass joinEncryptionJwk) and a DaemonSet; renderKubernetesManifests turns them into one v1 List for kubectl apply -f. It refuses (INVALID_CONFIG) an image not pinned by digest, a gateway or control-plane URL that is not https, malformed pins, names or resource quantities, a gpuApi other than device-plugin or dra, a JWK with a private part, and host paths that are not absolute or that host mounts rules out.

The pod's bcd container gets NODE_NAME and has bcd status as its liveness probe, and the pod rolls when config.toml changes (annotation bcd.dev/config-sha256). Both sides check the contract: scripts/daemon-k8s-contract.mjs writes the generated pod to daemon/cmd/bcd/testdata/k8s-daemonset.json, and bcd's TestKubernetesDaemonSetContract parses its config.toml (the gpu_api key included) and command lines with bcd's own configuration parser and flag sets.

Upgrades

Render the manifests again with the new image digest and apply them. The update strategy replaces one pod at a time. The identity and the journal are on the host, so a new pod resumes where the old one stopped. Rollouts and daemon.upgrade do not apply to these nodes, since bcd takes no self-update without release pins.

Limits

• No VPS instances. The image carries neither virsh nor cloud-hypervisor, so these nodes run no instance runtime.
• No reboots. node.reboot runs systemctl reboot, which the image does not have.
• The pod sees its own filesystem. Only the mounted host paths are the host's; that is why the host converger is off by default.
• Host resources are the kubelet's. A binding: 'kubernetes' hold gets no cgroup slice: the kubelet enforces its pods' requests and limits (e01d7ff). See Kubernetes workers.