Security

Better Compute runs commands as root on your machines, so its security model starts from one question: who can make a machine do something? The answer is meant to be only the control plane, only after Better IAM allowed the call, and only in a form the machine can check itself. This page summarizes the model from the security section of PLAN.md, says which parts are built, and ends with a hardening checklist.

Trust boundaries

Threat model

Gateway processes

The gateway package holds no CA, signing or encryption key: it relays Join, Renew, SecretRequest and WorkloadTokenRequest to gatewayServices and returns the answers verbatim. The design puts those services in the api process behind internal mutual TLS.

In this release, serve --roles gateway builds compute.gatewayServices in the gateway's own process from the shared configuration, so that process loads the PKI bundle and the deployment secret like any other. Treat gateway hosts with the same care as api hosts until the services move out. createGateway({ services }) takes any implementation of GatewayServices, so a remote implementation can replace the in-process one without changing the gateway.

Controls

Mutual TLS, failing closed

• Daemons connect to gateways over mutual TLS. certificateAuthenticate checks the chain against the node CA, the SPIFFE trust domain, revocation, role and expiry. Join alone accepts a client without a certificate; Renew, Session and Tunnel require one.
• The gateway role refuses to start without node-CA roots and a trust domain (INVALID_CONFIG). serveGateway refuses without nodeRoots or an explicit authenticate: there is no weaker default.
• unsafeSanOnlyAuthenticate and insecure: true (cleartext h2c) exist for tests only. better-compute dev --sim runs its gateway on loopback h2c with header authentication, for local development only.
• The api reaches gateways' internal control APIs (pokes, tunnels, revocations, closing an agent's streams) over mutual TLS from the internal CA. The client side is configured in notifier: gatewayNotifier({ sql, tls }) and host.tunnels.control.tls.

Allowlisted command kinds

Each certificate role accepts only its own command kinds. A certificate can be narrowed further with node_certs.allowed_kinds, read at Hello: any other kind is cancelled kind-not-allowed, including in-flight ones after a restricted re-join. A ramdisk admitted through discovery gets a certificate of at most 2 h limited to the commissioning kinds. A node-tenant certificate gets no secrets, and destructive kinds sent to it are cancelled tenant-controlled.

Duplicate identities

A second live Hello for a node from another boot wins the session and publishes security.node.duplicate-session (with both boot ids and the running count). A certificate that keeps alternating between two machines, the sign of a cloned key, is quarantined once it does so more than 3 times within 10 minutes: its serial is revoked with reason duplicate-identity, so every gateway's revocation feed drops the other clone too, security.node.quarantined is published, and the Hello is refused UNAUTHENTICATED after the revocation commits. Re-join the real machine with a new token. A session that already went quiet is superseded without counting. See sessions.

Signed commands

With pki.commandSigning, every command is signed by a 24 h worker key that the command-signing root certifies, and so is every NodeSpec generation (v0.4): the NodeSpec controller signs the exact DesiredStateBody bytes of each new generation (purpose desired-state) and stores them in nodes.spec_body and nodes.spec_signature. The gateway relays those signatures verbatim; it cannot produce them. A daemon checks any signature that is present, and both bcd (0d42de8) and the FakeDaemon (12ccfa1) with command-signing roots require a valid one on every command, whatever its kind: a kind table that exempts "harmless" kinds let a compromised gateway inject an unsigned site.httpProxy that could reset a BMC or insert media around the quorum. bcd and metal also treat site.httpProxy as destructive (signed, and under the batch quorum like bmc.*) and never proxies to a BMC: its site agent remembers the BMC hosts it drives (data_dir/bmc-hosts.json) and refuses them by name and resolved address, and refuses any /redfish path and any address in its own [site] bmc_networks (local configuration the control plane cannot loosen; doctor warns about site agents without it, a019c85), so a BMC is reached only through the bmc.* kinds and their quorum. A request other than a GET runs only under a batch leaf bound to its exact payload. A request about one machine also cannot reach another (a2a853d, 8391d74): metal lists the site's agentless machines and their inspected addresses in the site agent's spec (SiteSpec.proxy_targets, never a BMC host), and bcd refuses a URL host that is not one of the target node's listed addresses or names (forbidden-target) once the list names any target. For desired state, the FakeDaemon with command-signing roots requires a signature on every spec by default (signedSpecs: 'always', v0.4), and so does bcd, since an unsigned spec with a high generation would freeze all later ones; signedSpecs: 'access' requires one only on a spec that changes the access section (v0.2). Because Welcome.server_time is unsigned, both daemons also refuse a Welcome more than 5 minutes before the highest server time they have seen, so a gateway cannot replay an expired worker's signed command by rewinding the daemon's clock. See PKI.

A worker key can still sign anything, so a compromised api or worker could wipe the fleet with valid signatures. PLAN.md §13's v1.0 answer is the destructive-batch quorum: pki.approvers pins operator approver keys, held off the control plane, in the join-key-signed trust bundle, and a destructive command then runs only as part of a batch that enough of them approved, proven by a Merkle inclusion proof the machine checks itself; unbatched destructive commands are refused by default. The chain is built end to end: bcd and the FakeDaemon enforce it, the server's quorum module stores batches and collects approvals (commandBatches.approve), the command queue holds batched commands until approval, the gateway sends each with its approval and inclusion proof, and the better-compute batch commands let approvers review a batch against its root and sign it. A change of the approver set itself needs the current approvers' signatures (batch sign-approvers), so the join key alone cannot lift or replace the quorum. The built-in workflows produce their batches: metal puts each workflow under a one-node batch of its own, or under the batch metal.bulk creates for one verb on many nodes (f66e236), and the health reconciler's reboots and GPU resets run under a node-scoped batch (e594a4f). Metal's leaves leave the payload unbound (a leaf admits any payload of its kind for its subject), which approvers see flagged; binding metal's payloads is open. See the quorum.

Writing into deployed images

deploy.image writes the NoCloud seed and bcd's join files into the image it just streamed, and that image may be a tenant's: a crafted filesystem attacks whatever kernel mounts it. For an ext2, ext3 or ext4 root filesystem, the ramdisk never mounts it (1ae1789): e2fsprogs' debugfs writes the files in userspace, unlinking whatever the image has at each path (a symlink, a device node) without following it, writing root-owned 0600 regular files, and reading each back. Ignition's config goes onto the boot partition the same way (debugfs for ext, mtools for FAT), and boot.kexec copies the installed kernel and initrd out of an ext root with debugfs, resolving symlinks inside the image, instead of mounting it (d9a900f). Only xfs and btrfs, or a ramdisk without the tools, still fall back to a nodev,nosuid mount (read-only for kexec) with every write confined by os.Root. See what the ramdisk does.

Privilege separation on the node

As installed, bcd node runs as root under a hardened systemd unit (ProtectSystem=strict, NoNewPrivileges, a bounded capability set; see the unit), so a memory-safety or logic bug in the code that parses the session, commands and specs would run as root. PLAN.md §6 (v0.4) answers that with a split, and bcd has it since 28dc2ae:

• The node agent is unprivileged. With [helper] socket and [helper] user, bcd node runs as that account. The session, the journal, and every frame, command and spec from the gateway are handled there.

• The root helper trusts nothing it is sent. bcd helper serves only a caller whose SO_PEERCRED uid and gid are [helper] user's (or root) and whose /proc/<pid>/exe is an installed bcd binary. Requests are versioned JSON, at most 4 MiB, decoded strictly (an unknown field or version is refused).

• Typed operations, not paths. Files are named by kind and id (cdi, allocation-env, ssh-dropin, ssh-ca, ssh-krl, ssh-principals); the helper builds the path from its own table, refuses a symlink or non-directory on the way down, caps a file at 1 MiB and writes it atomically without following a symlink at the target. A ruleset must stay inside table bridge bcd_instances and table inet bcd_taps (no other table or family, no include, define or ruleset); gpu.reset names only a GPU in the host's own inventory; a reboot reason is cut to 200 bytes.

• Everything is logged. Each operation and each refused caller is logged with the caller's pid.

• Self-update without trusting the node (2d67b25). The slots and the upgrade and release records live in root's [update] slots_dir (default /usr/local/lib/bcd, never inside data_dir), and the executable check trusts only slots there. The node only downloads, into <data_dir>/staging; the helper takes no path from it, verifies the signed manifest under its own pins, floor and clock, opens the staged file with O_NOFOLLOW and hashes it as it copies it into the slot, so what is checked is what runs. Its own watchdog swaps back an update no Welcome confirmed, whatever the node does.

• No symlink race on the socket (2d67b25, from a security review). Root chmod and chown follow symlinks, so with a node-writable socket directory the node could have made root set 0660 root:bcd on any file. The socket's directory must now be root's and closed to group and other writes, and the socket is bound, chmodded and chowned in a private 0700 directory, then renamed into place.

• Host units only from signed specs (03c023d, 2d18327). The helper's host.unit applies a config unit as root, and every request carries the signed DesiredState that last set the host section and its worker signature (bcd node keeps the body in data_dir/host-intent.json). The command-signing roots come from command_root_sha256 in root's config.toml (the SHA-256 of each root certificate, given at install, never taken from the control plane or the node), which decides alone when set, or else, since 71ccf71, from the trust bundle in root's own signed join record, which makes signed intent mandatory whatever the node sends; a newer SignedTrustBundle the node presents, checked against root's join_signing_key_pin, rotates the roots forward only, so a root rotation needs no new pin. The helper refuses unless the roots check out (pinned, or from a bundle no older than one it accepted), the signature verifies (worker role), the body names this node (since a966cd4 the node root's own signed join record names, /var/lib/bcd-hostconfig/join-response.pb, re-verified against join_signing_key_pin; a node joined before it keeps the first signed spec's binding in /var/lib/bcd-hostconfig/helper-intent.json), its generation is not older than one already applied, and the unit's settings are exactly the body's. Once a bundle with roots was accepted, a request without a signed spec is refused. A verified body stays good after its short-lived worker certificate expires, and a node and a helper of different versions fail closed. The caveat: nothing is checked on an install where neither command_root_sha256 nor root's join record names command-signing roots (a deployment that signs nothing, or a node joined before the record) until the node itself presents a bundle with roots; the helper is then bounded by strict unit validation, the sysctl denylist, root writes only under root-only directories, and k8s-join only to root-pinned allowed_clusters, and bcd doctor fails on it (71ccf71). daemon install-script pins command_root_sha256 from --command-root-sha256 or the local PKI bundle (59985ae), which closes that window. The trust-on-first-use binding of the node id is closed by root's join record (a966cd4), except on nodes joined before it, which have no record. bcd doctor (as root, with [helper]) warns on such a node (f77c6f9); re-join it as root with bcd join --reset --yes (a1c1a46), since re-running the install script on a joined node writes no record. See signed specs.

• Root writes only under root's directories (02485b3, f8e13ef). Whenever bcd writes as root for a config unit, every existing directory from / down to the file must be a real directory owned by root and writable by no one else, so /tmp and anything the node owns are refused, and the units' records live in /var/lib/bcd-hostconfig, never in the node's data_dir. The helper's file kinds follow the same rule.

• libvirt through the helper (c305a05, af09411, 8c0cf9e, fea4d2e, ed3c7d2, d418820). The libvirt group is root-equivalent, so an unprivileged node is not in it. The helper defines, starts, stops, resizes, saves and restores the node's libvirt domains and opens their consoles, only for domains bcd defined, after a default-deny check of each definition: disks only under bcd's directories, checked and re-checked as root, taps named for the instance on [helper] instance_bridges, hostdevs only of the host's GPUs, no QEMU command line, filesystems, graphics or foreign namespaces. libvirt runs with dynamic_ownership = 0, so it never chowns a path the node names. The node reads through libvirt's read-only socket. See libvirt through the helper.

• cloud-hypervisor (7fe74a1). Each VMM runs as its own uid in a confined unit, and the node shares the instance's files with that uid through ACL entries. Root creates only the instance's own tap names on [helper] instance_bridges, binds only whole IOMMU groups of this host's idle GPUs to vfio-pci, and builds each VMM's unit itself from the instance record it checks, with its own binary, never a command line of the node's. See cloud-hypervisor through the helper.

• The TPM (8870b8d, security review 3). The kernel's TPM resource manager filters no command: whoever opens /dev/tpmrm0 can change the hierarchies' authorization, clear the TPM through the default empty lockout authorization or trip its dictionary-attack lockout, breaking root's TPM-sealed secrets and attestation. So the node never opens it. Root's helper derives, signs with and certifies its node keys (tpm.*, keys named by their 32-byte unique), builds every TPM command itself, and answers only public areas, signatures and certifications, which the node checks against the key's public half. See node keys through the helper.

daemon install-script --unprivileged (169c459) sets the mode up: a [helper] section with the socket in the helper's own root-owned /run/bcd-helper, the bcd system account, bcd-helper.service, a bcd.service drop-in with User=bcd, and /etc/bcd readable by the bcd group only; on a libvirt host, dynamic_ownership = 0 and QEMU's user in the bcd group (d418820). It keeps the bcd user out of the tss group: scripts from 0ba2a79 until 8870b8d put it there, and an upgrade run takes it out and deletes their /etc/udev/rules.d/60-bcd-tpm.rules.

The limits are plain. A host installed with --unprivileged before d418820 keeps bcd node in the libvirt group until the install script runs again (see upgrading an unprivileged libvirt host); bcd doctor fails it until then. Likewise a host installed between 0ba2a79 and 8870b8d keeps bcd in tss until the script runs again; since e6c3f53 bcd doctor fails its tpm check there (gpasswd -d bcd tss, then restart bcd.service). The helper covers files, the firewall table, GPU resets, reboots, sshd reloads, host units, self-update, libvirt, cloud-hypervisor (tested against fakes only) and the TPM's node keys; PLAN.md's mig.apply is not a helper operation, and bcx plugins need a root node (--unprivileged refuses --plugins). The default install still runs bcd node as root. See privilege separation and running bcd node unprivileged.

Driver plugins on the host

bcx driver plugins run next to bcd, which runs as root as installed, so bcd confines them ([plugins] in config.toml):

• What runs is what was pinned. Each binary is hashed through an open file and, on Linux, executed from that file (/proc/self/fd/3), never by path again, so swapping the file between the check and the exec runs nothing else.
• Not as root, and not as each other. When bcd is root, each plugin runs under its own uid and gid, or all as [plugins] user; no supplementary groups. The uids come from a block bcd keeps for itself (0x7FFE0000 to 0x7FFEFFFF, clear of system users, systemd's DynamicUser range and container ranges), are derived from the plugin's name, skip any uid an account already has, and are assigned in name order, so they are stable across restarts and reordering the pins file changes nothing (6827df8). No plugin can read another's memory. user = "root" is refused: plugins that need root (a Kata runtime) are named in [plugins] privileged and run without the filter, and bcd logs each one.
• A narrow kernel surface. With [plugins] sandbox (default on), bcd's __bcx-exec shim sets no_new_privs and installs a seccomp filter on every thread (TSYNC) that refuses kexec, kernel modules, mounts, namespaces (unshare, setns, clone with CLONE_NEW*; clone3 answers ENOSYS), swap, reboot, bpf, io_uring, ptrace and process_vm_*, the keyring, perf_event_open, userfaultfd, open_by_handle_at, raw I/O ports, clock changes, acct, quotactl, lookup_dcookie and syslog with EPERM, and kills a call from a foreign ABI.
• Bounded and reaped. [plugins] cgroup puts each plugin in its own child cgroup with memory_max and pids_max; every plugin has its own process group, dies with bcd and is killed as a group on close, and with a cgroup also through cgroup.kill, so a child that left the group with setsid dies too (933c51b). The handshake line is at most 64 KiB, each plugin message at most 16 MiB, and Describe is bounded by the handshake timeout, so a plugin that never answers does not hold up bcd's start.

The install script's bcd.service stays as tight as before on nodes without plugins. On a node whose config.toml has a [plugins] section (daemon install-script --plugins), the script adds a drop-in, bcd.service.d/plugins.conf, with only what the user switch and the cgroup limits need: CAP_SETUID, CAP_SETGID and CAP_KILL in the bounding set, Delegate=yes with DelegateSubgroup=main, and ProtectControlGroups=no; RestrictNamespaces=no is added only when [plugins] privileged names a plugin, such as a Kata runtime. See the plugin drop-in and the plugin sandbox.

Attested results

A wipe or diagnostics result counts as attested only when the node signed it: the gateway stores CommandUpdate.node_sig, and metal verifies it against node_certs. A revoked certificate is refused; an expired one is accepted for 7 more days, for results journaled offline. Before this fix a kind's attested flag was trusted, so a gateway could have forged a wipe. A node-tenant certificate never attests.

GPU bindings get the same treatment. With pki.commandSigning, capacity's attestedBindings defaults to pkiAllocationBindingCheck(): a reservation goes active only once the node's own signed AllocationBound verifies, the signed CDI devices replace whatever the gateway said, and a rejected report publishes reservation.binding-rejected. A forged re-report of an active binding marks it lost instead of replacing it. The gateway's report handlers also ignore reports about allocations placed on another node. See attested bindings.

Sealed secrets

• PostgreSQL holds only secret:// references. The default store seals values in sealed_secrets with AES-256-GCM; the authenticated data binds each ciphertext to its reference and tenant.
• secrets.put is write-only: the API never reads a value back. Events and audit rows carry the reference and version, never the value. The CLI's secrets put takes the value only from --value-file or --stdin, never an argument, and metal enroll takes a BMC password only from --bmc-password-file; tokens the CLI creates (join-tokens create, site-tokens create, a webhook's signing secret) can go to a new 0600 file with --out or --secret-out instead of the output.
• The broker (gatewayServices.secret) releases a value only when the request is signed by the node's current, unrevoked, platform certificate, names the session's node, is bound to work that node is doing right now, and uses a fresh request id. The answer is a JWE to the request's one-time key, expiring within 5 minutes. A reference is bound when:
  1. a live command of the named workflow step, addressed to the node, carries it;
  2. a live allocation on the node references it (an instance's user data, or what a module's allocationRefs accepts);
  3. the node's own spec names it as a credential (e93bf9c, b7346a9): the request names workflow host:<unit> and step <field> (both signed), the pair is allowed, the node's current spec (nodes.spec, rewritten with every generation) has that unit or section naming exactly this reference in that field, and the reference is a secret:// reference of the node's owner or its pool's owner. The node's certificate is checked again in the same statement that reads the spec, so a spec from before a policy edit cannot release an old reference. Allowed by default: the k8s-join config unit's tokenRef (the kubeadm bootstrap token; see k8s-join) and the telemetry section's headersRef (the OTLP exporter's headers; see node metrics). createSecretBroker's hostUnitRefs and specRefs options set the lists, and [] turns either off; betterCompute() uses the defaults. The audit row's details carry hostUnit.
• A hosted VPS instance's user data is released only when its secret:// reference belongs to the instance's own tenant, so one tenant's VM cannot boot with another tenant's secret. Since the guest reads its user data, instances.create accepts only a secret the tenant wrote with secrets.put (a secret_refs row; never a platform-minted one such as a join token, a BMC credential or another instance's sealed user data) and also decides compute:secrets:manage on the tenant with the caller's own credential (and on the fleet in the root tenant), so compute:instances:create alone cannot read a write-only secret. A fleet secret (secret_refs.scope fleet, such as a shared site's BMC credentials in the root tenant) is never user data: instances.create refuses it with INVALID_INPUT even for a holder of fleet compute:secrets:manage (aac73a1). Sandbox templates follow the same rule.
• secrets.put records the scope it was decided on (secret_refs.scope, tenant or fleet), and a tenant-scoped put or delete never changes a fleet secret (INVALID_STATE).
• Nothing is released for a tenant that is suspended or deleted: after the signature and binding checks, a reference whose tenant has a non-active compute_tenant_states row is refused (reason tenant-suspended or tenant-deleted, recorded as security.secret.refused), so a deleted tenant's instance cannot fetch its user data while teardown is still reaching it. It is released again once the tenant is active.
• Commands carry references, never values.

Workload identity (v0.5)

With workloadIdentity configured, a VPS host's metadata service can get a hosted instance's OIDC token. The broker (gatewayServices.workloadToken) mints one only when the request is signed by the node's current platform certificate of role node (never node-tenant, ramdisk, site or a kind-restricted certificate) naming the session's node, for a live instance placed on that node with its live allocation there, in a pool with policy.workloadIdentity: true, for an allowed audience, with a fresh request id and within the instance's rate. The gateway refuses a tenant-controlled session without asking, since the lessee holds root there. The token is a JWE to the request's one-time key, lives at most 15 minutes, and its audit row carries only the jti. Refusals publish security.workload.refused. On the host, a session token binds each metadata request to one instance and forwarded requests are refused. Since the metadata service and instance tunnels name an instance by its address, bcd guards every instance tap with host nftables rules (only the NIC's own MAC, IP addresses, EUI-64 link-local address and ARP; each guarded address bound to its own tap so no other bridge port can send from it or claim it; on every tap, guarded or not, no DHCP server reply, router advertisement or redirect, and never the host's own bridge addresses; no foreign neighbour advert), trusts only addresses those loaded rules bind to an instance, drops traffic from or to a tap that is off its bridge (the inet bcd_taps backstop), counts the tables' rules every 5 seconds so a flushed table or chain is caught and loaded again at once, and never serves an address two instances claim or a loopback caller. See workload identity.

Tunnels, tickets and break-glass SSH

• tunnels.open decides the caller with Better IAM, then issues a 60 s, single-use exec ticket for one node. The gateway and the daemon each verify it again, including its jti.
• While a tunnel is open, the caller is re-authorized every 60 s by the ticket's own action; after 2 minutes of IAM failures it closes. A ticket with an unexpected action fails closed. Tunnels last at most 12 h. A revoked delegation or suspended agent closes its tunnels on every gateway when host.iam is configured.
• Break-glass shells use nodes.issueSshCert: compute:nodes:ssh-root on the node and, by default, an MFA session. Certificates last at most 1 h and never beyond the session, name <account>@<nodeId>, and grant only permit-pty. Each issuance is recorded; nodes.revokeSshCert puts the serial in the node's key revocation list. With host.iam, a role deactivation, an identity's offboarding, expiry or deletion, and its session revocations revoke its certificates too, and the tunnels.ssh-certificates-reauthorize reconciler re-decides every live certificate for its issuing session each minute. Only a false answer, NOT_FOUND or TENANT_SUSPENDED revokes. When IAM is unreachable, or refuses compute's own credential (a thrown ACCESS_DENIED or UNAUTHENTICATED), a certificate stays valid until it expires, on purpose: break-glass is for outages, and an expired service token must not revoke every certificate mid-outage.
• Per identity and per api replica: 10 tunnels a minute and 20 certificates an hour, then RATE_LIMITED.
• ssh-cert --connect HOST accepts only a host name or an IP address, never something ssh would read as an option. ssh-cert checks the server's answer before using it: the account must be a Unix account name (it reaches ssh's %r and %u), and the certificate an OpenSSH user certificate for the caller's own key.

See interactive access.

Sandboxes (v0.5)

A sandbox is a tenant VM for code you do not trust, often written by an agent. The controls:

• One way in. sandboxes.exec needs a signed-in session (not an API key), decides compute:sandboxes:exec on the sandbox, and signs a certificate of at most 15 minutes, never past the session or the sandbox's expiry, for a principal that names one instance. It refuses a sandbox that expires within a minute. tunnels.open refuses a sandbox's instance and a warm one (sandbox-managed), so compute:instances:exec cannot bypass that decision.

• Re-decided on the sandbox. The ticket carries the sandbox (sbx); the gateway keeps it and every 60-second re-check decides compute:sandboxes:exec on the sandbox, never on the instance it drives.

• Separate CAs. The sandbox SSH CA is never the break-glass CA: the tunnels module refuses the same key for both (INVALID_CONFIG), so a sandbox certificate never opens a node.

• Secrets sealed before storage. Template env values and a secret:// user data are resolved into user data that is sealed through the secret broker before the instance row is written; without a broker such a template is INVALID_CONFIG.

• Secrets only the tenant put. A template's env and secret:// user data must be secrets the tenant wrote with secrets.put, never fleet-scoped ones, and naming them decides compute:secrets:manage (on the fleet too in the root tenant, whose store also holds fleet secrets). An update that changes what the guest runs is decided on the template's secrets even when it names none, and each new instance re-checks them.

• A rotated secret leaves the warm pool. Warm guests hold resolved secret values, so each records the secret_refs.version of every secret it got; after a rotation or delete, claims skip and retire the stale ones (falling back to a cold create) and the reconciler replaces them (2c7072b, review 13 L4).

• Snapshots are the owner's. A snapshot holds memory and /etc/sandbox/env, so it belongs to the sandbox's owner, whoever took it, and only that owner can restore it (NOT_FOUND for anyone else in the tenant); someone allowed to snapshot another user's sandbox cannot boot its memory as their own. An agent restores only snapshots of sandboxes it created itself, a restore runs only under its snapshot's own template (so memory captured under a none network never resumes under egress), and an owner who leaves without a successor has every snapshot dropped (7b01053, security review 13). A deleted snapshot's object is purged only once its upload settled and no live instance restores from it.

• Each restore gets its own identity. A memory restore resumes with its source's sshd principal and host key, so a certificate for one such guest would open the others (review 13 H2). vps seals each memory restore its own principals and, with certifyHostKey, a fresh certified host key (5346523); bcd hot-adds them to the running guest on a 1 MiB bc-rebind disk that it zeroes and deletes afterwards (ca08d5c), and the guest half vps installs at first boot (3e25c78) installs them and acknowledges. Only when the host reports the acknowledged reference does exec sign for the restore alone. Until then sandboxes.exec refuses with INVALID_STATE (restore-shares-identity) while another live guest of the tenant shares the identity (87ff2c5): a guest that never acknowledges (booted before the guest half, on a runtime that cannot rebind) stays refused, which fails closed. Disk-only restores are not affected. See a restore's own SSH identity.

• SSH only from the host. Every sandbox instance's spec sets ssh_from_host_only, and bcd drops tcp/22 into its taps from anything but the host's own bridge addresses (474d51f, review 13 L5). An exec certificate therefore works only through a tunnel, which is re-authorized every 60 seconds and closed on revocation, and one guest cannot reach another's sshd. A bcd from before 474d51f would ignore the field, so since 82f4f51 sandbox and warm instances are placed only on hosts labelled bc.dev/cap.instance.ssh-host-only, which a bcd that enforces it advertises.

• Replays and races. A sandbox create's idempotency hash covers the caller's agent and delegation, so an agent cannot replay its person's key (or the reverse) to take over a sandbox, and a template update decided on one set of secrets is refused (CONFLICT, secrets-changed) if the template names others by the time it writes (7b01053).

• The CLI treats the guest as hostile. sandbox exec puts -- before the host, so a generated command is never an ssh option, and turns off agent and X11 forwarding, port forwardings, proxies, connection sharing and local commands for that connection. CLI output strips control characters from server text (table cells, string results, hints) and JSON escapes DEL and the C1 controls.

• Network policy on the host. A template's network (private, none) becomes host nftables rules on the instance's taps, whatever runs inside; bcd fails closed, stopping an instance whose rules cannot be expressed or loaded, and vps refuses private or none on a pool without an IP range. Such instances are placed only on hosts labelled bc.dev/cap.instance.network-policy, since an older bcd would drop the policy and give full egress. private spans the pool's whole IPAM subnet, other tenants' instances included on a shared pool.

• Capability labels are bcd's alone (c1fd430, 0ff3ded, 736d3e5; review L11). The bc.dev/ labels above, and the bc.dev/driver.* and bc.dev/driver-kind.* labels, are what bcd advertises, written by the gateway with source system. An operator could otherwise forge one and land sandboxes, placed flavors or volumes on a bcd that does not enforce them. Since 409f783 the same goes for GPU reservations: capacity books host resources only on nodes labelled bc.dev/cap.devices.limits and binds kubernetes reservations only on bc.dev/cap.binding.kubernetes ones, since an older bcd would run the hold without its CPU and memory limits (see capabilities). So nodes.label refuses to set or remove any bc.dev/ key, and every reader counts a bc.dev/ label only with source system: placement (the node labels it reads and the exact shortlist), metal's driver kinds for a site's BMC drivers, provision's when: expressions and the host spec it builds, and the vps self-check that trusts bc.dev/cap.instance.network-policy. A Hello whose bc.dev/ labels match an operator row left from before c1fd430 takes that row over as system.

• The tenant's image is hostile to the host. Both bcd runtimes read every qcow2 base image's header in Go, without qemu-img or any other tool probing it, and refuse one that names a backing file or an external data file, sets an unknown incompatible feature, is encrypted or is not a qcow2; raw and iso images are opened with their format stated and never probed (security review 12, 933c51b; the Go check since its verification, b12c217). An instance's own disk must name exactly its recorded base before every start, snapshot and resize, since a VMM can rewrite its header, and a restored snapshot's disk must name none. On cloud-hypervisor each VMM runs in a confined bcd-ch-<id>.service unit as its instance's own uid, with no capabilities and no network (bcd hands it its taps as file descriptors), /run and /var hidden, only KVM and VFIO devices, only its own instance directory and base image visible, and limits from its flavor; a memory snapshot restores only into the same vCPUs and memory, with its configuration rebuilt from an allowlist. See the runtimes.

• The guest is pinned too. With vps({ sandboxExec: { certifyHostKey } }), each guest presents a host key the sandbox CA certified (its private key only in sealed user data), and sandbox exec checks it with @cert-authority and StrictHostKeyChecking=yes. hostCertified is reported only while the guest's certificate (30 days, never renewed) still holds, and a sandbox's lifetime, extensions and memory restores never run past it; a guest without a valid one is refused unless --accept-host-key asks for trust on first use for that connection. sandbox exec also runs ssh with -F none, so no user or system ssh_config applies.

See sandboxes.

Tenant isolation

• Pre-check first. A caller outside the input's tenant and the root tenant gets ACCESS_DENIED before any database access or IAM decision, identical to an IAM denial, so there is no existence oracle and outsiders put no traffic on another tenant's audit chain.
• No ambient tenant. Tenant-scoped repository functions take tenantId as a required parameter.
• Attributes from the projection. IAM decides on attributes from compute's own authorization projection, never from the request.
• Idempotency keys are scoped to (tenant, identity), so one tenant cannot replay another's request.
• Tenant-controlled machines. Under a dedicated-root lease the node's identity is reissued as node-tenant, and its inventory becomes advisory.
• Suspended and deleted tenants. Once the IAM bridge's tenant reaction (tenantStateHandler) mirrors a tenant's state, the envelope refuses a suspended tenant's mutations and every call of a deleted one with TENANT_SUSPENDED, root callers exempt, within 5 seconds in every process. A deleted tenant is then torn down: its reservations end, offers to it are withdrawn, its pools archived, its workflows aborted, its private images soft deleted, its image signers revoked, its bare-metal machines retired (deployed ones wiped first) and its VPS instances deleted. The subtree is read from IAM by the tenancy reconciler, outside the inbox transaction; while IAM is down the read is retried, so a suspension is delayed, never dropped.
• Revocations before reactions. When an identity departs or an agent is suspended, serve's host revokes the identity's SSH certificates, closes its tunnels and drops cached authentications first, and only then runs module reactions (such as withdrawing an agent's sandboxes), each under its own savepoint (b3f7a81, review 7 M3). A reaction that throws is undone alone, so it can never roll the revocation back; failures of identity reactions go to host.iam.onError and the IamReconcile sweep re-runs them, and a failing agent reaction retries its event (a3be14c). See SSH.

See tenancy.

Nothing leaks

• Unexpected errors answer 500 with INTERNAL_ERROR and a generic message; the detail goes only to log.error.
• Join, renew and secret refusals are one generic ACCESS_DENIED to the agent; the reason goes to audit (security.join.refused, security.renew.refused since 0e42018, security.secret.refused), rate-limited per reason.
• A report a node sends about another node becomes security.node-mismatch and never reaches a handler, and a report about another node's allocation is ignored.
• /.well-known/better-compute publishes only public key members; a private JWK member never passes.
• The PKI bundle is sealed at rest, and pki show reads only its public half.
• bcd reads its join token from BCD_JOIN_TOKEN or standard input, never from arguments, and never echoes a positional argument.

Refused renewals

Until 0e42018 a refused Renew was only a renew refused line in the server log. JoinService now also publishes security.renew.refused for every ACCESS_DENIED renewal, in its own short transaction after the renewal's has rolled back, so the event survives the refusal. Who sees it depends on how far the caller got:

The split is deliberate: a node certificate is public, so anyone can present one with a bad proof, and such a refusal must not land in the owner's event stream or name its node. The payload carries reasons only:

The rate bound is the one join refusals use (refusalEventsPerMinute, default 60 a minute), keyed per renew reason. A renewal that bcd never sends (its TPM unavailable, or another AK than the certificate names) produces no event; bcd logs renewal failed. tests/server-join-tpm-renew.test.ts checks both cases: a missing key attestation is the owner's event naming the node, and a proof by someone else's key over the node's certificate is the root tenant's, subject renew, with no node.

The HTTP API

• POST requests need content-type: application/json and X-Better-Compute: 1, else CSRF_REJECTED. A plain HTML form cannot set that header.
• The IAM session cookie is ignored unless acceptIamCookie is set. Even then a cookie request needs an Origin exactly equal to baseURL's origin: a missing one is CSRF_REJECTED, another one UNTRUSTED_ORIGIN.
• Bodies are JSON objects of at most 64 KiB (PAYLOAD_TOO_LARGE). TRACE, CONNECT and TRACK are refused.
• Every answer carries cache-control: no-store, x-content-type-options: nosniff and referrer-policy: no-referrer.
• X-Request-Id is echoed only when it matches ^[A-Za-z0-9._:-]{1,128}$, and IAM sees the same id.
• IAM sees the caller's address only as the host verified it: nodeHandler passes the socket's peer, a fetch host passes compute.handler(request, { peer: { address } }), and remote IAM gets that as x-forwarded-for. The client's own x-forwarded-for, x-real-ip and forwarded are never forwarded, so a caller cannot choose the request.sourceIp its policies see (e7092bf, d269738, review 14 L3; the MCP server's remote IAM likewise, 7890821). Behind a load balancer, trustedProxies (0873d20) lists the proxies' addresses or CIDRs: only when the socket peer is one of them is the caller read from X-Forwarded-For, right to left, skipping trusted hops; a trusted proxy that sends no usable header, or a malformed hop, leaves sourceIp unset, never the proxy's own address. Since 0eba0ad the HTTP handler also strips X-Forwarded-For, Forwarded, X-Real-IP and the other client-address headers from what an embedded IAM sees and sets X-Forwarded-For to the verified address; since 004579e that happens for every credential on every path in (presentedCredential, the in-process API included), not only over HTTP, so a caller that passes headers to compute.api cannot choose its address either. The IAM caches key on it: the authentication cache (30 s) and allowed read decisions (10 s) are per caller address, so a stolen token used from a disallowed address no longer rides its owner's cached entries. See the caller address.

Delegated approvals

An agent's held-back request reaches a person as text, so the text must say what the request does, and the approval must admit only that (security review 14):

• Approvers see what they approve. reservations.create, reservations.extend, instances.create, sandboxes.create and sandboxes.extend describe themselves (MethodAdmission.describe, ec36c63): pool, devices, lifetime, image, SSH keys as fingerprints, user data by size and digest, the snapshot a sandbox restores. The text is deterministic, and caller strings are clipped to one line with control characters removed.
• An approval admits one request. The envelope records a digest of that text per confirmation, keyed by tenant, delegation, action and resource (confirmations/0001_bindings), and a delegated call whose text differs is refused with ACCESS_DENIED (approval-mismatch), even while Better IAM's approval for that action and resource is still open (06d7f04, M1).
• An agent's batch reads as the agent's. A destructive batch an agent proposed records the agent and its delegation (quorum/0008_batch_agent, f3c5c4e), and batch approve, batch show and the console say "proposed by agent X for Y" (bd80209, M3).
• The MCP gate stays out of confirmations. Build it with createMcpGate({ confirmations: false }) (daf58eb, M2); its rules accept only well-formed ids, create_instance takes no SSH keys (7890821, H1), doctor flags an ungated host.mcp handler (host.mcp.ungated, b491aed, L6), and one caller has at most 4 tool calls in flight (48d41a4, L5).

See delegated confirmation and MCP.

Rate limits and admission

Tamper-evident audit

Every allowed mutation writes an audit_events row with its admission context in the same transaction (composite methods write one per sub-call). So do node joins and reclaims, secret releases by the broker, and workload tokens issued to instances (workload.token.issued). Commands, workflow steps and exec tickets are not audit rows today; they are recorded in their own tables and events. With auditModule, an hourly job seals new rows into signed, hash-chained Merkle checkpoints per tenant (RFC 6962), in commit order so a long transaction cannot land behind a checkpoint. better-compute audit verify --tenant ID recomputes the chain and finds an edited, deleted or inserted row, a removed checkpoint or a foreign signature; audit prove writes an inclusion proof a third party checks offline with audit check-proof. The module's self-check warns when rows older than 3 hours are still unsealed. An audit sink (sinks.audit) copies every committed row, in commit order, to a store the database's operators cannot change; see sinks.

Hardening checklist

Reporting a vulnerability

The repository has no security policy file or dedicated disclosure address yet. Until it does, report a vulnerability privately to the maintainers rather than in a public issue, and include the version, the configuration involved, and steps to reproduce.