# Security (/docs/operations/security) > The trust boundaries and threat model of Better Compute, the controls that enforce them, and a checklist for hardening a deployment. Better Compute runs commands as root on your machines, so its security model starts from one question: who can make a machine do something? The answer is meant to be only the control plane, only after Better IAM allowed the call, and only in a form the machine can check itself. This page summarizes the model from the security section of PLAN.md, says which parts are built, and ends with a hardening checklist. ## Trust boundaries [#trust-boundaries] | Boundary | Protected by | | ---------------------------------- | --------------------------------------------------------------------------------------- | | Caller to api | TLS at your load balancer; Better IAM authenticates every caller and decides every call | | api to Better IAM | In-process (embedded) through the IAM bridge | | api and workers to gateway | Mutual TLS from the internal CA | | Gateway to daemon | Mutual TLS from the machine PKI: the node CA for clients, the serving CA for gateways | | Site agent to BMC | The management VLAN, reachable only from site agents | | Tenant to tenant | Tenant pre-checks, SQL tenant scoping, the authorization projection | | Tenant-controlled node to platform | A `node-tenant` certificate that gets no secrets and no destructive platform commands | ## Threat model [#threat-model] | Threat | Mitigation | State | | -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Cross-tenant access, forged attributes | Tenant pre-check before any database access; SQL tenant scoping; idempotency keys scoped to tenant and identity; attributes only from compute's own projection | Built | | Stolen join token | Short TTL, `maxUses` (default 1), `expect` binding; tokens stored hashed, revocable, and sealed to JoinService in transit, bound to the CSR | Built | | Compromised node | `nodeId` only from the certificate's SAN; 24 h certificates with pushed revocation; secrets bound to current work; reports only about its own allocations; its attested results speak only for itself | Built | | Cloned node identity | A certificate alternating between two machines more than 3 times in 10 minutes is revoked (duplicate-identity quarantine); a `tpm` join admits only a node key the machine's TPM certifies it made and never lets out (`nodeKey: 'certified'`, the default), and its certificate names that TPM's AK (`urn:bc:tpm-ak:`), so every renewal must be certified by the same AK (`RenewRequest.key_attestation`; refused otherwise, no downgrade) | Quarantine built in the gateway; the TPM node-key proof built at join (cf24876) and through renewal (d94f1f3), for certified `tpm` joins only; tested with software TPMs only | | Compromised gateway | Holds no key in its code; commands and NodeSpecs signed; Renew and `JoinResponse` verified end to end; secrets as JWE; results and `AllocationBound` reports signed by the node. It can delay, drop or read frames, not forge intent, identity, results, bindings or secrets | Built, with the [process caveat](#gateway-processes) | | Compromised api or worker | Admissions still need IAM decisions; worker keys last 24 h; tickets are audited; fleet-wide destructive batches need k-of-n operator approval checked on the machine (the [destructive-batch quorum](/docs/operations/pki#the-destructive-batch-quorum-v10)) | Built, the quorum included (v1.0): bcd and the FakeDaemon enforce it, the server stores, approves and holds batches, the gateway attaches approvals and proofs, the CLI reviews and signs, and the built-in workflows create their batches (f66e236, e594a4f). Open: metal's leaves do not bind payloads yet | | Exploited node agent process | With `bcd helper`, `bcd node` parses everything from the network as an unprivileged account; the root helper serves only that account (`SO_PEERCRED`) running an installed bcd binary, a few typed operations with fixed file paths, bcd's own nftables table and the host's own GPUs, and logs each one | Built in bcd (28dc2ae, Linux), self-update included (2d67b25); opt-in with `daemon install-script --unprivileged` (169c459), while the default install still runs `bcd node` as root | | Compromised driver plugin | Pinned by SHA-256 and executed from the file that was hashed; run under its own unprivileged uid with `no_new_privs`, a seccomp filter (no namespaces, mounts or `io_uring`) and optional cgroup limits; only plugins named `privileged` run as root | Built in bcd (Linux) | | Replay | Command-id journal, `not_after`, epoch fencing, a server time mark against rewound `Welcome`s, ticket `jti` sets, a Renew replay cache | Built in the control plane, the FakeDaemon and `bcd` (bcd keeps its ticket `jti` set in its journal, so a restart does not forget a spent ticket) | | Agent overspend | Delegation scope, confirmations, per-agent quotas, budgets, `agents.suspendAll` | Through Better IAM | | PXE and DHCP spoofing | One lease-holding boot server per site, isolated VLANs, signed images | Partly built (v0.2) | | Cross-tenant residue on bare metal | BMC-driven teardown: attested erase, DCGM, a firmware baseline check, a new identity generation | Built in `metal` (v0.2, v0.5) | > **What the real daemon checks today.** `bcd` is built and tested but not released. It implements the journal, the pinned join and TLS, `JoinResponse` verification, signed commands and specs (once its trust bundle holds command-signing roots), node-signed attested results and allocation reports, the server time mark (a `Welcome` never moves its working clock back), exec-ticket re-verification before it opens a tunnel (a ticket that names an instance is dialed to that instance's own address, never the host's), the SSH `access` section (node-bound principals and a KRL; a spec whose CA keys or principals could inject an sshd line is refused whole, signed or not), and node-signed `SecretRequest` and `WorkloadTokenRequest` answered to a one-time key (the site agent sends them for BMC credentials and discovery tokens, the ramdisk and VPS hosts for user data, the metadata service for workload tokens). The site agent's boot server answers PXE only for prepared machines while it holds the boot lease, serves its image cache only under per-boot or per-firmware grants (b7ed8a2), bounds TFTP to 64 transfers and 4 per source (4d8b493) and request headers to 8 KiB, and keeps only well-formed fields of unknown PXE clients (0442561). PXE discovery never binds a MAC from that unauthenticated DHCP data alone (8de6e2f): a discovered MAC is a candidate until an operator confirms it with `metal.confirmMac`, unless the pool sets `policy.trustProvisioningNetwork`, in which case the ramdisk's UUID and NICs must match the machine (see [discovering new machines](/docs/guides/bare-metal/sites#discovering-new-machines)). Driver plugins run sandboxed (below), and each `bmc.*` kind admits only its own BMC operations, so a `bmc.query`, which needs no batch, cannot carry a power or RAID change, and `site.httpProxy` never reaches a BMC. With approver keys in its trust bundle it enforces the destructive-batch quorum. Its node key is in the host's TPM when bcd can open one (non-exportable, but root on the node can use it), else a 0600 file. A `tpm` join proves it to the control plane: the TPM's AK certifies the key over the CSR's hash, the attestor refuses the join without that proof by default, and `node.joined` records `nodeKey: 'tpm'` (cf24876). The certificate then names the AK, and every renewal must carry that AK's certification of the next key, which bcd makes in the TPM or fails the renewal (d94f1f3). Other joins, and their renewals, prove nothing about where the key lives; see [the node key in the TPM](/docs/guides/nodes/attestation#the-node-key-in-the-tpm). `bcd node` can run unprivileged beside the root `bcd helper` ([below](#privilege-separation-on-the-node)), when installed with `--unprivileged`, self-update included; by default it runs as root. See [the daemon](/docs/daemon#what-bcd-does-today). ## Gateway processes [#gateway-processes] The gateway package holds no CA, signing or encryption key: it relays `Join`, `Renew`, `SecretRequest` and `WorkloadTokenRequest` to `gatewayServices` and returns the answers verbatim. The design puts those services in the api process behind internal mutual TLS. In this release, `serve --roles gateway` builds `compute.gatewayServices` in the gateway's own process from the shared configuration, so that process loads the PKI bundle and the deployment secret like any other. Treat gateway hosts with the same care as api hosts until the services move out. `createGateway({ services })` takes any implementation of `GatewayServices`, so a remote implementation can replace the in-process one without changing the gateway. ## Controls [#controls] ### Mutual TLS, failing closed [#mutual-tls-failing-closed] * Daemons connect to gateways over mutual TLS. `certificateAuthenticate` checks the chain against the node CA, the SPIFFE trust domain, revocation, role and expiry. `Join` alone accepts a client without a certificate; `Renew`, `Session` and `Tunnel` require one. * The gateway role refuses to start without node-CA roots and a trust domain (`INVALID_CONFIG`). `serveGateway` refuses without `nodeRoots` or an explicit `authenticate`: there is no weaker default. * `unsafeSanOnlyAuthenticate` and `insecure: true` (cleartext h2c) exist for tests only. `better-compute dev --sim` runs its gateway on loopback h2c with header authentication, for local development only. * The api reaches gateways' internal control APIs (pokes, tunnels, revocations, closing an agent's streams) over mutual TLS from the internal CA. The client side is configured in `notifier: gatewayNotifier({ sql, tls })` and `host.tunnels.control.tls`. ### Allowlisted command kinds [#allowlisted-command-kinds] Each certificate role accepts only its own command kinds. A certificate can be narrowed further with `node_certs.allowed_kinds`, read at `Hello`: any other kind is cancelled `kind-not-allowed`, including in-flight ones after a restricted re-join. A ramdisk admitted through discovery gets a certificate of at most 2 h limited to the commissioning kinds. A `node-tenant` certificate gets no secrets, and destructive kinds sent to it are cancelled `tenant-controlled`. ### Duplicate identities [#duplicate-identities] A second live `Hello` for a node from another boot wins the session and publishes `security.node.duplicate-session` (with both boot ids and the running count). A certificate that keeps alternating between two machines, the sign of a cloned key, is quarantined once it does so more than 3 times within 10 minutes: its serial is revoked with reason `duplicate-identity`, so every gateway's revocation feed drops the other clone too, `security.node.quarantined` is published, and the `Hello` is refused `UNAUTHENTICATED` after the revocation commits. Re-join the real machine with a new token. A session that already went quiet is superseded without counting. See [sessions](/docs/daemon/sessions#duplicate-identity). ### Signed commands [#signed-commands] With `pki.commandSigning`, every command is signed by a 24 h worker key that the command-signing root certifies, and so is every NodeSpec generation (v0.4): the NodeSpec controller signs the exact `DesiredStateBody` bytes of each new generation (purpose `desired-state`) and stores them in `nodes.spec_body` and `nodes.spec_signature`. The gateway relays those signatures verbatim; it cannot produce them. A daemon checks any signature that is present, and both bcd (0d42de8) and the FakeDaemon (12ccfa1) with command-signing roots require a valid one on every command, whatever its kind: a kind table that exempts "harmless" kinds let a compromised gateway inject an unsigned `site.httpProxy` that could reset a BMC or insert media around the quorum. bcd and metal also treat `site.httpProxy` as destructive (signed, and under the batch quorum like `bmc.*`) and never proxies to a BMC: its site agent remembers the BMC hosts it drives (`data_dir/bmc-hosts.json`) and refuses them by name and resolved address, and refuses any `/redfish` path and any address in its own `[site] bmc_networks` (local configuration the control plane cannot loosen; doctor warns about site agents without it, a019c85), so a BMC is reached only through the `bmc.*` kinds and their quorum. A request other than a `GET` runs only under a batch leaf bound to its exact payload. A request about one machine also cannot reach another (a2a853d, 8391d74): metal lists the site's agentless machines and their inspected addresses in the site agent's spec (`SiteSpec.proxy_targets`, never a BMC host), and bcd refuses a URL host that is not one of the target node's listed addresses or names (`forbidden-target`) once the list names any target. For desired state, the FakeDaemon with command-signing roots requires a signature on every spec by default (`signedSpecs: 'always'`, v0.4), and so does bcd, since an unsigned spec with a high generation would freeze all later ones; `signedSpecs: 'access'` requires one only on a spec that changes the `access` section (v0.2). Because `Welcome.server_time` is unsigned, both daemons also refuse a `Welcome` more than 5 minutes before the highest server time they have seen, so a gateway cannot replay an expired worker's signed command by rewinding the daemon's clock. See [PKI](/docs/operations/pki#signed-intent). A worker key can still sign anything, so a compromised api or worker could wipe the fleet with valid signatures. PLAN.md §13's v1.0 answer is the destructive-batch quorum: `pki.approvers` pins operator approver keys, held off the control plane, in the join-key-signed trust bundle, and a destructive command then runs only as part of a batch that enough of them approved, proven by a Merkle inclusion proof the machine checks itself; unbatched destructive commands are refused by default. The chain is built end to end: bcd and the FakeDaemon enforce it, the server's `quorum` module stores batches and collects approvals (`commandBatches.approve`), the command queue holds batched commands until approval, the gateway sends each with its approval and inclusion proof, and the `better-compute batch` commands let approvers review a batch against its root and sign it. A change of the approver set itself needs the current approvers' signatures (`batch sign-approvers`), so the join key alone cannot lift or replace the quorum. The built-in workflows produce their batches: metal puts each workflow under a one-node batch of its own, or under the batch `metal.bulk` creates for one verb on many nodes (f66e236), and the health reconciler's reboots and GPU resets run under a node-scoped batch (e594a4f). Metal's leaves leave the payload unbound (a leaf admits any payload of its kind for its subject), which approvers see flagged; binding metal's payloads is open. See [the quorum](/docs/operations/pki#the-destructive-batch-quorum-v10). ### Writing into deployed images [#writing-into-deployed-images] `deploy.image` writes the NoCloud seed and bcd's join files into the image it just streamed, and that image may be a tenant's: a crafted filesystem attacks whatever kernel mounts it. For an ext2, ext3 or ext4 root filesystem, the ramdisk never mounts it (1ae1789): e2fsprogs' `debugfs` writes the files in userspace, unlinking whatever the image has at each path (a symlink, a device node) without following it, writing root-owned 0600 regular files, and reading each back. Ignition's config goes onto the `boot` partition the same way (`debugfs` for ext, `mtools` for FAT), and `boot.kexec` copies the installed kernel and initrd out of an ext root with `debugfs`, resolving symlinks inside the image, instead of mounting it (d9a900f). Only xfs and btrfs, or a ramdisk without the tools, still fall back to a `nodev,nosuid` mount (read-only for kexec) with every write confined by `os.Root`. See [what the ramdisk does](/docs/guides/bare-metal/deploying#what-the-ramdisk-does). ### Privilege separation on the node [#privilege-separation-on-the-node] As installed, `bcd node` runs as root under a hardened systemd unit (`ProtectSystem=strict`, `NoNewPrivileges`, a bounded capability set; see [the unit](/docs/daemon/install#the-systemd-unit)), so a memory-safety or logic bug in the code that parses the session, commands and specs would run as root. PLAN.md §6 (v0.4) answers that with a split, and bcd has it since 28dc2ae: * **The node agent is unprivileged.** With `[helper] socket` and `[helper] user`, `bcd node` runs as that account. The session, the journal, and every frame, command and spec from the gateway are handled there. * **The root helper trusts nothing it is sent.** `bcd helper` serves only a caller whose `SO_PEERCRED` uid and gid are `[helper] user`'s (or root) and whose `/proc//exe` is an installed bcd binary. Requests are versioned JSON, at most 4 MiB, decoded strictly (an unknown field or version is refused). * **Typed operations, not paths.** Files are named by kind and id (`cdi`, `allocation-env`, `ssh-dropin`, `ssh-ca`, `ssh-krl`, `ssh-principals`); the helper builds the path from its own table, refuses a symlink or non-directory on the way down, caps a file at 1 MiB and writes it atomically without following a symlink at the target. A ruleset must stay inside `table bridge bcd_instances` and `table inet bcd_taps` (no other table or family, no `include`, `define` or `ruleset`); `gpu.reset` names only a GPU in the host's own inventory; a reboot reason is cut to 200 bytes. * **Everything is logged.** Each operation and each refused caller is logged with the caller's pid. * **Self-update without trusting the node** (2d67b25). The slots and the upgrade and release records live in root's `[update] slots_dir` (default `/usr/local/lib/bcd`, never inside `data_dir`), and the executable check trusts only slots there. The node only downloads, into `/staging`; the helper takes no path from it, verifies the signed manifest under its own pins, floor and clock, opens the staged file with `O_NOFOLLOW` and hashes it as it copies it into the slot, so what is checked is what runs. Its own watchdog swaps back an update no `Welcome` confirmed, whatever the node does. * **No symlink race on the socket** (2d67b25, from a security review). Root chmod and chown follow symlinks, so with a node-writable socket directory the node could have made root set `0660 root:bcd` on any file. The socket's directory must now be root's and closed to group and other writes, and the socket is bound, chmodded and chowned in a private 0700 directory, then renamed into place. * **Host units only from signed specs** (03c023d, 2d18327). The helper's `host.unit` applies a config unit as root, and every request carries the signed `DesiredState` that last set the host section and its worker signature (`bcd node` keeps the body in `data_dir/host-intent.json`). The command-signing roots come from `command_root_sha256` in root's `config.toml` (the SHA-256 of each root certificate, given at install, never taken from the control plane or the node), which decides alone when set, or else, since 71ccf71, from the trust bundle in root's own signed join record, which makes signed intent mandatory whatever the node sends; a newer `SignedTrustBundle` the node presents, checked against root's `join_signing_key_pin`, rotates the roots forward only, so a root rotation needs no new pin. The helper refuses unless the roots check out (pinned, or from a bundle no older than one it accepted), the signature verifies (worker role), the body names this node (since a966cd4 the node root's own signed join record names, `/var/lib/bcd-hostconfig/join-response.pb`, re-verified against `join_signing_key_pin`; a node joined before it keeps the first signed spec's binding in `/var/lib/bcd-hostconfig/helper-intent.json`), its generation is not older than one already applied, and the unit's settings are exactly the body's. Once a bundle with roots was accepted, a request without a signed spec is refused. A verified body stays good after its short-lived worker certificate expires, and a node and a helper of different versions fail closed. **The caveat:** nothing is checked on an install where neither `command_root_sha256` nor root's join record names command-signing roots (a deployment that signs nothing, or a node joined before the record) until the node itself presents a bundle with roots; the helper is then bounded by strict unit validation, the sysctl denylist, root writes only under root-only directories, and `k8s-join` only to root-pinned `allowed_clusters`, and `bcd doctor` fails on it (71ccf71). `daemon install-script` pins `command_root_sha256` from `--command-root-sha256` or the local PKI bundle (59985ae), which closes that window. The trust-on-first-use binding of the node id is closed by root's join record (a966cd4), except on nodes joined before it, which have no record. `bcd doctor` (as root, with `[helper]`) warns on such a node (f77c6f9); re-join it as root with `bcd join --reset --yes` (a1c1a46), since re-running the install script on a joined node writes no record. See [signed specs](/docs/daemon/host-config#signed-specs). * **Root writes only under root's directories** (02485b3, f8e13ef). Whenever bcd writes as root for a config unit, every existing directory from `/` down to the file must be a real directory owned by root and writable by no one else, so `/tmp` and anything the node owns are refused, and the units' records live in `/var/lib/bcd-hostconfig`, never in the node's `data_dir`. The helper's file kinds follow the same rule. * **libvirt through the helper** (c305a05, af09411, 8c0cf9e, fea4d2e, ed3c7d2, d418820). The `libvirt` group is root-equivalent, so an unprivileged node is not in it. The helper defines, starts, stops, resizes, saves and restores the node's libvirt domains and opens their consoles, only for domains bcd defined, after a default-deny check of each definition: disks only under bcd's directories, checked and re-checked as root, taps named for the instance on `[helper] instance_bridges`, hostdevs only of the host's GPUs, no QEMU command line, filesystems, graphics or foreign namespaces. libvirt runs with `dynamic_ownership = 0`, so it never chowns a path the node names. The node reads through libvirt's read-only socket. See [libvirt through the helper](/docs/daemon#libvirt-through-the-helper). * **cloud-hypervisor** (7fe74a1). Each VMM runs as its own uid in a confined unit, and the node shares the instance's files with that uid through ACL entries. Root creates only the instance's own tap names on `[helper] instance_bridges`, binds only whole IOMMU groups of this host's idle GPUs to `vfio-pci`, and builds each VMM's unit itself from the instance record it checks, with its own binary, never a command line of the node's. See [cloud-hypervisor through the helper](/docs/daemon#cloud-hypervisor-through-the-helper). * **The TPM** (8870b8d, security review 3). The kernel's TPM resource manager filters no command: whoever opens `/dev/tpmrm0` can change the hierarchies' authorization, clear the TPM through the default empty lockout authorization or trip its dictionary-attack lockout, breaking root's TPM-sealed secrets and attestation. So the node never opens it. Root's helper derives, signs with and certifies its node keys (`tpm.*`, keys named by their 32-byte unique), builds every TPM command itself, and answers only public areas, signatures and certifications, which the node checks against the key's public half. See [node keys through the helper](/docs/daemon#node-keys-through-the-helper). `daemon install-script --unprivileged` (169c459) sets the mode up: a `[helper]` section with the socket in the helper's own root-owned `/run/bcd-helper`, the `bcd` system account, `bcd-helper.service`, a `bcd.service` drop-in with `User=bcd`, and `/etc/bcd` readable by the `bcd` group only; on a libvirt host, `dynamic_ownership = 0` and QEMU's user in the `bcd` group (d418820). It keeps the `bcd` user out of the `tss` group: scripts from 0ba2a79 until 8870b8d put it there, and an upgrade run takes it out and deletes their `/etc/udev/rules.d/60-bcd-tpm.rules`. The limits are plain. A host installed with `--unprivileged` before d418820 keeps `bcd node` in the `libvirt` group until the install script runs again (see [upgrading an unprivileged libvirt host](/docs/daemon/install#upgrading-an-unprivileged-libvirt-host)); `bcd doctor` fails it until then. Likewise a host installed between 0ba2a79 and 8870b8d keeps `bcd` in `tss` until the script runs again; since e6c3f53 `bcd doctor` fails its `tpm` check there (`gpasswd -d bcd tss`, then restart `bcd.service`). The helper covers files, the firewall table, GPU resets, reboots, sshd reloads, host units, self-update, libvirt, cloud-hypervisor (tested against fakes only) and the TPM's node keys; PLAN.md's `mig.apply` is not a helper operation, and bcx plugins need a root node (`--unprivileged` refuses `--plugins`). The default install still runs `bcd node` as root. See [privilege separation](/docs/daemon#privilege-separation) and [running bcd node unprivileged](/docs/daemon/install#running-bcd-node-unprivileged). ### Driver plugins on the host [#driver-plugins-on-the-host] bcx driver plugins run next to bcd, which runs as root as installed, so bcd confines them (`[plugins]` in `config.toml`): * **What runs is what was pinned.** Each binary is hashed through an open file and, on Linux, executed from that file (`/proc/self/fd/3`), never by path again, so swapping the file between the check and the exec runs nothing else. * **Not as root, and not as each other.** When bcd is root, each plugin runs under its own uid and gid, or all as `[plugins] user`; no supplementary groups. The uids come from a block bcd keeps for itself (`0x7FFE0000` to `0x7FFEFFFF`, clear of system users, systemd's `DynamicUser` range and container ranges), are derived from the plugin's name, skip any uid an account already has, and are assigned in name order, so they are stable across restarts and reordering the `pins` file changes nothing (6827df8). No plugin can read another's memory. `user = "root"` is refused: plugins that need root (a Kata runtime) are named in `[plugins] privileged` and run without the filter, and bcd logs each one. * **A narrow kernel surface.** With `[plugins] sandbox` (default on), bcd's `__bcx-exec` shim sets `no_new_privs` and installs a seccomp filter on every thread (TSYNC) that refuses kexec, kernel modules, mounts, namespaces (`unshare`, `setns`, `clone` with `CLONE_NEW*`; `clone3` answers `ENOSYS`), swap, reboot, `bpf`, `io_uring`, `ptrace` and `process_vm_*`, the keyring, `perf_event_open`, `userfaultfd`, `open_by_handle_at`, raw I/O ports, clock changes, `acct`, `quotactl`, `lookup_dcookie` and `syslog` with `EPERM`, and kills a call from a foreign ABI. * **Bounded and reaped.** `[plugins] cgroup` puts each plugin in its own child cgroup with `memory_max` and `pids_max`; every plugin has its own process group, dies with bcd and is killed as a group on close, and with a cgroup also through `cgroup.kill`, so a child that left the group with `setsid` dies too (933c51b). The handshake line is at most 64 KiB, each plugin message at most 16 MiB, and `Describe` is bounded by the handshake timeout, so a plugin that never answers does not hold up bcd's start. The install script's `bcd.service` stays as tight as before on nodes without plugins. On a node whose `config.toml` has a `[plugins]` section (`daemon install-script --plugins`), the script adds a drop-in, `bcd.service.d/plugins.conf`, with only what the user switch and the cgroup limits need: `CAP_SETUID`, `CAP_SETGID` and `CAP_KILL` in the bounding set, `Delegate=yes` with `DelegateSubgroup=main`, and `ProtectControlGroups=no`; `RestrictNamespaces=no` is added only when `[plugins] privileged` names a plugin, such as a Kata runtime. See [the plugin drop-in](/docs/daemon/install#the-plugin-drop-in) and [the plugin sandbox](/docs/extending/drivers#the-plugin-sandbox). ### Attested results [#attested-results] A wipe or diagnostics result counts as attested only when the node signed it: the gateway stores `CommandUpdate.node_sig`, and metal verifies it against `node_certs`. A revoked certificate is refused; an expired one is accepted for 7 more days, for results journaled offline. Before this fix a kind's `attested` flag was trusted, so a gateway could have forged a wipe. A `node-tenant` certificate never attests. GPU bindings get the same treatment. With `pki.commandSigning`, capacity's `attestedBindings` defaults to `pkiAllocationBindingCheck()`: a reservation goes `active` only once the node's own signed `AllocationBound` verifies, the signed CDI devices replace whatever the gateway said, and a rejected report publishes `reservation.binding-rejected`. A forged re-report of an active binding marks it lost instead of replacing it. The gateway's report handlers also ignore reports about allocations placed on another node. See [attested bindings](/docs/guides/capacity/reservations#attested-bindings). ### Sealed secrets [#sealed-secrets] * PostgreSQL holds only `secret://` references. The default store seals values in `sealed_secrets` with AES-256-GCM; the authenticated data binds each ciphertext to its reference and tenant. * `secrets.put` is write-only: the API never reads a value back. Events and audit rows carry the reference and version, never the value. The CLI's `secrets put` takes the value only from `--value-file` or `--stdin`, never an argument, and `metal enroll` takes a BMC password only from `--bmc-password-file`; tokens the CLI creates (`join-tokens create`, `site-tokens create`, a webhook's signing secret) can go to a new 0600 file with `--out` or `--secret-out` instead of the output. * The broker (`gatewayServices.secret`) releases a value only when the request is signed by the node's current, unrevoked, platform certificate, names the session's node, is bound to work that node is doing right now, and uses a fresh request id. The answer is a JWE to the request's one-time key, expiring within 5 minutes. A reference is bound when: 1. a live command of the named workflow step, addressed to the node, carries it; 2. a live allocation on the node references it (an instance's user data, or what a module's `allocationRefs` accepts); 3. the node's own spec names it as a credential (e93bf9c, b7346a9): the request names workflow `host:` and step `` (both signed), the pair is allowed, the node's current spec (`nodes.spec`, rewritten with every generation) has that unit or section naming exactly this reference in that field, and the reference is a `secret://` reference of the node's owner or its pool's owner. The node's certificate is checked again in the same statement that reads the spec, so a spec from before a policy edit cannot release an old reference. Allowed by default: the `k8s-join` config unit's `tokenRef` (the kubeadm bootstrap token; see [k8s-join](/docs/daemon/host-config#k8s-join)) and the `telemetry` section's `headersRef` (the OTLP exporter's headers; see [node metrics](/docs/operations/observability#node-metrics-from-bcd)). `createSecretBroker`'s `hostUnitRefs` and `specRefs` options set the lists, and `[]` turns either off; `betterCompute()` uses the defaults. The audit row's details carry `hostUnit`. * A hosted VPS instance's user data is released only when its `secret://` reference belongs to the instance's own tenant, so one tenant's VM cannot boot with another tenant's secret. Since the guest reads its user data, `instances.create` accepts only a secret the tenant wrote with `secrets.put` (a `secret_refs` row; never a platform-minted one such as a join token, a BMC credential or another instance's sealed user data) and also decides `compute:secrets:manage` on the tenant with the caller's own credential (and on the fleet in the root tenant), so `compute:instances:create` alone cannot read a write-only secret. A fleet secret (`secret_refs.scope` `fleet`, such as a shared site's BMC credentials in the root tenant) is never user data: `instances.create` refuses it with `INVALID_INPUT` even for a holder of fleet `compute:secrets:manage` (aac73a1). Sandbox templates follow the same rule. * `secrets.put` records the scope it was decided on (`secret_refs.scope`, `tenant` or `fleet`), and a tenant-scoped put or delete never changes a fleet secret (`INVALID_STATE`). * Nothing is released for a tenant that is suspended or deleted: after the signature and binding checks, a reference whose tenant has a non-active `compute_tenant_states` row is refused (reason `tenant-suspended` or `tenant-deleted`, recorded as `security.secret.refused`), so a deleted tenant's instance cannot fetch its user data while teardown is still reaching it. It is released again once the tenant is active. * Commands carry references, never values. ### Workload identity (v0.5) [#workload-identity-v05] With `workloadIdentity` configured, a VPS host's metadata service can get a hosted instance's OIDC token. The broker (`gatewayServices.workloadToken`) mints one only when the request is signed by the node's current platform certificate of role `node` (never `node-tenant`, ramdisk, site or a kind-restricted certificate) naming the session's node, for a live instance placed on that node with its live allocation there, in a pool with `policy.workloadIdentity: true`, for an allowed audience, with a fresh request id and within the instance's rate. The gateway refuses a tenant-controlled session without asking, since the lessee holds root there. The token is a JWE to the request's one-time key, lives at most 15 minutes, and its audit row carries only the `jti`. Refusals publish `security.workload.refused`. On the host, a session token binds each metadata request to one instance and forwarded requests are refused. Since the metadata service and instance tunnels name an instance by its address, bcd guards every instance tap with host nftables rules (only the NIC's own MAC, IP addresses, EUI-64 link-local address and ARP; each guarded address bound to its own tap so no other bridge port can send from it or claim it; on every tap, guarded or not, no DHCP server reply, router advertisement or redirect, and never the host's own bridge addresses; no foreign neighbour advert), trusts only addresses those loaded rules bind to an instance, drops traffic from or to a tap that is off its bridge (the `inet bcd_taps` backstop), counts the tables' rules every 5 seconds so a flushed table or chain is caught and loaded again at once, and never serves an address two instances claim or a loopback caller. See [workload identity](/docs/guides/vps/instances#workload-identity). ### Tunnels, tickets and break-glass SSH [#tunnels-tickets-and-break-glass-ssh] * `tunnels.open` decides the caller with Better IAM, then issues a 60 s, single-use exec ticket for one node. The gateway and the daemon each verify it again, including its `jti`. * While a tunnel is open, the caller is re-authorized every 60 s by the ticket's own action; after 2 minutes of IAM failures it closes. A ticket with an unexpected action fails closed. Tunnels last at most 12 h. A revoked delegation or suspended agent closes its tunnels on every gateway when `host.iam` is configured. * Break-glass shells use `nodes.issueSshCert`: `compute:nodes:ssh-root` on the node and, by default, an MFA session. Certificates last at most 1 h and never beyond the session, name `@`, and grant only `permit-pty`. Each issuance is recorded; `nodes.revokeSshCert` puts the serial in the node's key revocation list. With `host.iam`, a role deactivation, an identity's offboarding, expiry or deletion, and its session revocations revoke its certificates too, and the `tunnels.ssh-certificates-reauthorize` reconciler re-decides every live certificate for its issuing session each minute. Only a `false` answer, `NOT_FOUND` or `TENANT_SUSPENDED` revokes. When IAM is unreachable, or refuses compute's own credential (a thrown `ACCESS_DENIED` or `UNAUTHENTICATED`), a certificate stays valid until it expires, on purpose: break-glass is for outages, and an expired service token must not revoke every certificate mid-outage. * Per identity and per api replica: 10 tunnels a minute and 20 certificates an hour, then `RATE_LIMITED`. * `ssh-cert --connect HOST` accepts only a host name or an IP address, never something ssh would read as an option. `ssh-cert` checks the server's answer before using it: the account must be a Unix account name (it reaches ssh's `%r` and `%u`), and the certificate an OpenSSH user certificate for the caller's own key. See [interactive access](/docs/guides/access). ### Sandboxes (v0.5) [#sandboxes-v05] A sandbox is a tenant VM for code you do not trust, often written by an agent. The controls: * **One way in.** `sandboxes.exec` needs a signed-in session (not an API key), decides `compute:sandboxes:exec` on the sandbox, and signs a certificate of at most 15 minutes, never past the session or the sandbox's expiry, for a principal that names one instance. It refuses a sandbox that expires within a minute. `tunnels.open` refuses a sandbox's instance and a warm one (`sandbox-managed`), so `compute:instances:exec` cannot bypass that decision. * **Re-decided on the sandbox.** The ticket carries the sandbox (`sbx`); the gateway keeps it and every 60-second re-check decides `compute:sandboxes:exec` on the sandbox, never on the instance it drives. * **Separate CAs.** The sandbox SSH CA is never the break-glass CA: the tunnels module refuses the same key for both (`INVALID_CONFIG`), so a sandbox certificate never opens a node. * **Secrets sealed before storage.** Template `env` values and a `secret://` user data are resolved into user data that is sealed through the secret broker before the instance row is written; without a broker such a template is `INVALID_CONFIG`. * **Secrets only the tenant put.** A template's `env` and `secret://` user data must be secrets the tenant wrote with `secrets.put`, never fleet-scoped ones, and naming them decides `compute:secrets:manage` (on the fleet too in the root tenant, whose store also holds fleet secrets). An update that changes what the guest runs is decided on the template's secrets even when it names none, and each new instance re-checks them. * **A rotated secret leaves the warm pool.** Warm guests hold resolved secret values, so each records the `secret_refs.version` of every secret it got; after a rotation or delete, claims skip and retire the stale ones (falling back to a cold create) and the reconciler replaces them (2c7072b, review 13 L4). * **Snapshots are the owner's.** A snapshot holds memory and `/etc/sandbox/env`, so it belongs to the sandbox's owner, whoever took it, and only that owner can restore it (`NOT_FOUND` for anyone else in the tenant); someone allowed to snapshot another user's sandbox cannot boot its memory as their own. An agent restores only snapshots of sandboxes it created itself, a restore runs only under its snapshot's own template (so memory captured under a `none` network never resumes under `egress`), and an owner who leaves without a successor has every snapshot dropped (7b01053, security review 13). A deleted snapshot's object is purged only once its upload settled and no live instance restores from it. * **Each restore gets its own identity.** A memory restore resumes with its source's sshd principal and host key, so a certificate for one such guest would open the others (review 13 H2). vps seals each memory restore its own principals and, with `certifyHostKey`, a fresh certified host key (5346523); bcd hot-adds them to the running guest on a 1 MiB `bc-rebind` disk that it zeroes and deletes afterwards (ca08d5c), and the guest half vps installs at first boot (3e25c78) installs them and acknowledges. Only when the host reports the acknowledged reference does exec sign for the restore alone. Until then `sandboxes.exec` refuses with `INVALID_STATE` (`restore-shares-identity`) while another live guest of the tenant shares the identity (87ff2c5): a guest that never acknowledges (booted before the guest half, on a runtime that cannot rebind) stays refused, which fails closed. Disk-only restores are not affected. See [a restore's own SSH identity](/docs/guides/vps/sandboxes#a-restores-own-ssh-identity). * **SSH only from the host.** Every sandbox instance's spec sets `ssh_from_host_only`, and bcd drops tcp/22 into its taps from anything but the host's own bridge addresses (474d51f, review 13 L5). An exec certificate therefore works only through a tunnel, which is re-authorized every 60 seconds and closed on revocation, and one guest cannot reach another's sshd. A bcd from before 474d51f would ignore the field, so since 82f4f51 sandbox and warm instances are placed only on hosts labelled `bc.dev/cap.instance.ssh-host-only`, which a bcd that enforces it advertises. * **Replays and races.** A sandbox create's idempotency hash covers the caller's agent and delegation, so an agent cannot replay its person's key (or the reverse) to take over a sandbox, and a template update decided on one set of secrets is refused (`CONFLICT`, `secrets-changed`) if the template names others by the time it writes (7b01053). * **The CLI treats the guest as hostile.** `sandbox exec` puts `--` before the host, so a generated command is never an ssh option, and turns off agent and X11 forwarding, port forwardings, proxies, connection sharing and local commands for that connection. CLI output strips control characters from server text (table cells, string results, hints) and JSON escapes DEL and the C1 controls. * **Network policy on the host.** A template's `network` (`private`, `none`) becomes host nftables rules on the instance's taps, whatever runs inside; bcd fails closed, stopping an instance whose rules cannot be expressed or loaded, and vps refuses `private` or `none` on a pool without an IP range. Such instances are placed only on hosts labelled `bc.dev/cap.instance.network-policy`, since an older bcd would drop the policy and give full egress. `private` spans the pool's whole IPAM subnet, other tenants' instances included on a shared pool. * **Capability labels are bcd's alone** (c1fd430, 0ff3ded, 736d3e5; review L11). The `bc.dev/` labels above, and the `bc.dev/driver.*` and `bc.dev/driver-kind.*` labels, are what bcd advertises, written by the gateway with source `system`. An operator could otherwise forge one and land sandboxes, placed flavors or volumes on a bcd that does not enforce them. Since 409f783 the same goes for GPU reservations: capacity books host resources only on nodes labelled `bc.dev/cap.devices.limits` and binds `kubernetes` reservations only on `bc.dev/cap.binding.kubernetes` ones, since an older bcd would run the hold without its CPU and memory limits (see [capabilities](/docs/daemon/sessions#hello)). So [`nodes.label`](/docs/reference/api/nodes#label) refuses to set or remove any `bc.dev/` key, and every reader counts a `bc.dev/` label only with source `system`: placement (the node labels it reads and the exact shortlist), metal's driver kinds for a site's BMC drivers, provision's `when:` expressions and the host spec it builds, and the vps self-check that trusts `bc.dev/cap.instance.network-policy`. A `Hello` whose `bc.dev/` labels match an operator row left from before c1fd430 takes that row over as `system`. * **The tenant's image is hostile to the host.** Both bcd runtimes read every `qcow2` base image's header in Go, without `qemu-img` or any other tool probing it, and refuse one that names a backing file or an external data file, sets an unknown incompatible feature, is encrypted or is not a qcow2; `raw` and `iso` images are opened with their format stated and never probed (security review 12, 933c51b; the Go check since its verification, b12c217). An instance's own disk must name exactly its recorded base before every start, snapshot and resize, since a VMM can rewrite its header, and a restored snapshot's disk must name none. On `cloud-hypervisor` each VMM runs in a confined `bcd-ch-.service` unit as its instance's own uid, with no capabilities and no network (bcd hands it its taps as file descriptors), `/run` and `/var` hidden, only KVM and VFIO devices, only its own instance directory and base image visible, and limits from its flavor; a memory snapshot restores only into the same vCPUs and memory, with its configuration rebuilt from an allowlist. See [the runtimes](/docs/daemon/delivery#the-runtimes). * **The guest is pinned too.** With `vps({ sandboxExec: { certifyHostKey } })`, each guest presents a host key the sandbox CA certified (its private key only in sealed user data), and `sandbox exec` checks it with `@cert-authority` and `StrictHostKeyChecking=yes`. `hostCertified` is reported only while the guest's certificate (30 days, never renewed) still holds, and a sandbox's lifetime, extensions and memory restores never run past it; a guest without a valid one is refused unless `--accept-host-key` asks for trust on first use for that connection. `sandbox exec` also runs ssh with `-F none`, so no user or system `ssh_config` applies. See [sandboxes](/docs/guides/vps/sandboxes#run-code-inside). ### Tenant isolation [#tenant-isolation] * **Pre-check first.** A caller outside the input's tenant and the root tenant gets `ACCESS_DENIED` before any database access or IAM decision, identical to an IAM denial, so there is no existence oracle and outsiders put no traffic on another tenant's audit chain. * **No ambient tenant.** Tenant-scoped repository functions take `tenantId` as a required parameter. * **Attributes from the projection.** IAM decides on attributes from compute's own authorization projection, never from the request. * **Idempotency keys** are scoped to `(tenant, identity)`, so one tenant cannot replay another's request. * **Tenant-controlled machines.** Under a dedicated-root lease the node's identity is reissued as `node-tenant`, and its inventory becomes advisory. * **Suspended and deleted tenants.** Once the IAM bridge's `tenant` reaction (`tenantStateHandler`) mirrors a tenant's state, the envelope refuses a suspended tenant's mutations and every call of a deleted one with `TENANT_SUSPENDED`, root callers exempt, within 5 seconds in every process. A deleted tenant is then torn down: its reservations end, offers to it are withdrawn, its pools archived, its workflows aborted, its private images soft deleted, its image signers revoked, its bare-metal machines retired (deployed ones wiped first) and its VPS instances deleted. The subtree is read from IAM by the `tenancy` reconciler, outside the inbox transaction; while IAM is down the read is retried, so a suspension is delayed, never dropped. * **Revocations before reactions.** When an identity departs or an agent is suspended, `serve`'s host revokes the identity's SSH certificates, closes its tunnels and drops cached authentications first, and only then runs module reactions (such as withdrawing an agent's sandboxes), each under its own savepoint (b3f7a81, review 7 M3). A reaction that throws is undone alone, so it can never roll the revocation back; failures of identity reactions go to `host.iam.onError` and the `IamReconcile` sweep re-runs them, and a failing agent reaction retries its event (a3be14c). See [SSH](/docs/guides/access/ssh). See [tenancy](/docs/guides/iam/tenancy). ### Nothing leaks [#nothing-leaks] * Unexpected errors answer `500` with `INTERNAL_ERROR` and a generic message; the detail goes only to `log.error`. * Join, renew and secret refusals are one generic `ACCESS_DENIED` to the agent; the reason goes to audit (`security.join.refused`, `security.renew.refused` since 0e42018, `security.secret.refused`), rate-limited per reason. * A report a node sends about another node becomes `security.node-mismatch` and never reaches a handler, and a report about another node's allocation is ignored. * `/.well-known/better-compute` publishes only public key members; a private JWK member never passes. * The PKI bundle is sealed at rest, and `pki show` reads only its public half. * `bcd` reads its join token from `BCD_JOIN_TOKEN` or standard input, never from arguments, and never echoes a positional argument. ### Refused renewals [#refused-renewals] Until 0e42018 a refused `Renew` was only a `renew refused` line in the server log. JoinService now also publishes `security.renew.refused` for every `ACCESS_DENIED` renewal, in its own short transaction after the renewal's has rolled back, so the event survives the refusal. Who sees it depends on how far the caller got: | The caller | Tenant | `subject` | Reasons | | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Proved possession of the current key (the leaf is compute's own, current certificate and `pop_sig` verified) | The node's owner (`nodes.owner_tenant_id`) | `node/` | `renew:replay`, and a refused key attestation: `renew:key-attestation-missing`, `renew:key-attestation-ak`, `renew:node-key-*`, `renew:tpm-ak-uri`, `renew:key-statement-unknown`, `renew:key-statement-unsupported` | | Anyone else | The IAM root tenant (`rootTenantId`); nothing is published without one | `renew` | `renew:chain`, `renew:client-time`, `renew:proof`, `renew:csr`, `renew:certificate`, `renew:revoked`, `renew:retired`, `renew:peer` | The split is deliberate: a node certificate is public, so anyone can present one with a bad proof, and such a refusal must not land in the owner's event stream or name its node. The payload carries reasons only: The rate bound is the one join refusals use (`refusalEventsPerMinute`, default 60 a minute), keyed per renew reason. A renewal that bcd never sends (its TPM unavailable, or another AK than the certificate names) produces no event; bcd logs `renewal failed`. `tests/server-join-tpm-renew.test.ts` checks both cases: a missing key attestation is the owner's event naming the node, and a proof by someone else's key over the node's certificate is the root tenant's, subject `renew`, with no node. ### The HTTP API [#the-http-api] * `POST` requests need `content-type: application/json` and `X-Better-Compute: 1`, else `CSRF_REJECTED`. A plain HTML form cannot set that header. * The IAM session cookie is ignored unless `acceptIamCookie` is set. Even then a cookie request needs an `Origin` exactly equal to `baseURL`'s origin: a missing one is `CSRF_REJECTED`, another one `UNTRUSTED_ORIGIN`. * Bodies are JSON objects of at most 64 KiB (`PAYLOAD_TOO_LARGE`). `TRACE`, `CONNECT` and `TRACK` are refused. * Every answer carries `cache-control: no-store`, `x-content-type-options: nosniff` and `referrer-policy: no-referrer`. * `X-Request-Id` is echoed only when it matches `^[A-Za-z0-9._:-]{1,128}$`, and IAM sees the same id. * IAM sees the caller's address only as the host verified it: `nodeHandler` passes the socket's peer, a fetch host passes `compute.handler(request, { peer: { address } })`, and remote IAM gets that as `x-forwarded-for`. The client's own `x-forwarded-for`, `x-real-ip` and `forwarded` are never forwarded, so a caller cannot choose the `request.sourceIp` its policies see (e7092bf, d269738, review 14 L3; the MCP server's remote IAM likewise, 7890821). Behind a load balancer, `trustedProxies` (0873d20) lists the proxies' addresses or CIDRs: only when the socket peer is one of them is the caller read from `X-Forwarded-For`, right to left, skipping trusted hops; a trusted proxy that sends no usable header, or a malformed hop, leaves `sourceIp` unset, never the proxy's own address. Since 0eba0ad the HTTP handler also strips `X-Forwarded-For`, `Forwarded`, `X-Real-IP` and the other client-address headers from what an embedded IAM sees and sets `X-Forwarded-For` to the verified address; since 004579e that happens for every credential on every path in (`presentedCredential`, the in-process API included), not only over HTTP, so a caller that passes `headers` to `compute.api` cannot choose its address either. The IAM caches key on it: the authentication cache (30 s) and allowed read decisions (10 s) are per caller address, so a stolen token used from a disallowed address no longer rides its owner's cached entries. See [the caller address](/docs/operations/deployment#the-caller-address). ### Delegated approvals [#delegated-approvals] An agent's held-back request reaches a person as text, so the text must say what the request does, and the approval must admit only that (security review 14): * **Approvers see what they approve.** `reservations.create`, `reservations.extend`, `instances.create`, `sandboxes.create` and `sandboxes.extend` describe themselves (`MethodAdmission.describe`, ec36c63): pool, devices, lifetime, image, SSH keys as fingerprints, user data by size and digest, the snapshot a sandbox restores. The text is deterministic, and caller strings are clipped to one line with control characters removed. * **An approval admits one request.** The envelope records a digest of that text per confirmation, keyed by tenant, delegation, action and resource (`confirmations/0001_bindings`), and a delegated call whose text differs is refused with `ACCESS_DENIED` (`approval-mismatch`), even while Better IAM's approval for that action and resource is still open (06d7f04, M1). * **An agent's batch reads as the agent's.** A destructive batch an agent proposed records the agent and its delegation (`quorum/0008_batch_agent`, f3c5c4e), and `batch approve`, `batch show` and the console say "proposed by agent X for Y" (bd80209, M3). * **The MCP gate stays out of confirmations.** Build it with `createMcpGate({ confirmations: false })` (daf58eb, M2); its rules accept only well-formed ids, `create_instance` takes no SSH keys (7890821, H1), `doctor` flags an ungated `host.mcp` handler (`host.mcp.ungated`, b491aed, L6), and one caller has at most 4 tool calls in flight (48d41a4, L5). See [delegated confirmation](/docs/guides/iam/delegated-confirmation) and [MCP](/docs/guides/clients/mcp). ### Rate limits and admission [#rate-limits-and-admission] | Where | Limit | | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | Gateway handshakes | A token bucket of 200 a second | | Gateway sessions | `maxStreams` (default 5,000) | | Pending `Hello`s | 64; more are refused `RESOURCE_EXHAUSTED` so a reconnect storm cannot starve the lease | | Duplicate `Hello`s | More than 3 from a second machine within 10 minutes quarantine the certificate (`createGateway({ duplicateHellos: { max, windowMs } })`) | | Tunnels, break-glass certificates | 10 a minute and 20 an hour per identity, per api replica | | Better IAM | A 429 from IAM becomes `RATE_LIMITED`, with `Retry-After` | | Security events | Join, secret and workload-token refusal events: 60 a minute per reason by default; the rest are counted as suppressed | | Workload tokens | 30 a minute per instance by default (`workloadIdentity.perInstancePerMinute`) | ### Tamper-evident audit [#tamper-evident-audit] Every allowed mutation writes an `audit_events` row with its admission context in the same transaction (composite methods write one per sub-call). So do node joins and reclaims, secret releases by the broker, and workload tokens issued to instances (`workload.token.issued`). Commands, workflow steps and exec tickets are not audit rows today; they are recorded in their own tables and events. With `auditModule`, an hourly job seals new rows into signed, hash-chained Merkle checkpoints per tenant (RFC 6962), in commit order so a long transaction cannot land behind a checkpoint. `better-compute audit verify --tenant ID` recomputes the chain and finds an edited, deleted or inserted row, a removed checkpoint or a foreign signature; `audit prove` writes an inclusion proof a third party checks offline with `audit check-proof`. The module's self-check warns when rows older than 3 hours are still unsealed. An audit sink (`sinks.audit`) copies every committed row, in commit order, to a store the database's operators cannot change; see [sinks](/docs/extending/sinks#audit-sinks). ## Hardening checklist [#hardening-checklist] ### Secrets and keys [#secrets-and-keys] * A deployment secret of at least 32 random characters, from a secret manager, identical in every process. * `pki init` with the secret (never `--plaintext` outside development) and `--offline-root`, with the root key kept off every server. * The bundle, the secret and the root key backed up, separately. * A bundle sealed in the former format resealed once: `pki reseal` prints its pins, you compare them with your records, then `pki reseal --accept-unauthenticated`. `BETTER_COMPUTE_PKI_ALLOW_UNAUTHENTICATED` unset afterwards; the commands that write the bundle refuse it anyway. See [PKI](/docs/operations/pki#pki-reseal). * Machines installed with their pins, taken from the sealed bundle (`bootstrapPins(bundle)` into provision's user data and `vps({ bootstrap: { pins } })`): a host build of `bcd join --server` without `config.toml` requires all three and refuses a `/.well-known/better-compute` whose pins differ, since only the control plane's TLS certificate vouches for that document. Only fakehw builds join on the document's word (trust on first use). See [installing bcd](/docs/daemon/install#joining-from-the-well-known-document). ### Network [#network] * api behind an L7 load balancer that terminates TLS; `baseURL` is exactly the public origin. * Gateways behind an L4 load balancer with TLS passthrough. Machines need only outbound 443. * Internal control APIs (`host.gateway.internal`) reachable only from the control plane, over internal-CA mTLS. * `GET /metrics` only with a `bearerToken`, and not exposed publicly. * BMC networks reachable only from site agents. * `policy.trustProvisioningNetwork` only on pools whose provisioning network no untrusted host can reach; elsewhere confirm each discovered PXE MAC with `metal.confirmMac`. ### Configuration [#configuration] * Better IAM wired through the bridge; `doctor` reports `iam.not-configured` otherwise. * `acceptIamCookie` only for a console on the same origin. * `pki.commandSigning` set, so commands and NodeSpecs are signed and GPU bindings are attested. * `tunnels` only if you need interactive access, with the SSH CA's MFA requirement left on. * `auditModule` registered with the bundle's audit key. * Nodes that run no bcx plugins installed with `daemon install-script --unprivileged`, so `bcd node` is not root, with `command_root_sha256` pinned so `bcd helper` checks host units against signed specs from the first request. * `trustedProxies` set to your load balancers when IAM policies condition on `request.sourceIp`. * On TPM 2.0 fleets, `tpmAttestor` left at `nodeKey: 'certified'`, with `policy.joinAttestors: ['tpm']` on the pools that must take no other kind of join, so every node there starts with a node key its TPM holds (see [tpm](/docs/guides/nodes/attestation#tpm)). ### Operations [#operations] * `better-compute doctor --strict` in CI and after every deploy. * Gateway serving certificates reissued before their 30 days end, and CA expiry watched (`doctor` warns 30 days ahead). * Revocations checked in practice: `nodes.revokeIdentity` should close the node's session within seconds. * `audit verify` run on a schedule for the tenants that matter. ## Reporting a vulnerability [#reporting-a-vulnerability] The repository has no security policy file or dedicated disclosure address yet. Until it does, report a vulnerability privately to the maintainers rather than in a public issue, and include the version, the configuration involved, and steps to reproduce.