# CLI (/docs/guides/clients/cli) > Use the better-compute command line as a client - authenticate with a token, run a command for every API group, reserve GPUs, run sandboxes, plan the fleet. `better-compute` is one binary for two jobs. Operators use it to run and maintain a deployment (`migrate`, `serve`, `doctor`, `pki`). Tenants, operators and CI jobs also use it as an API client: every token command sends the same request as the TypeScript client and is decided by Better IAM as the token's identity. This page covers the client side. The [CLI reference](/docs/reference/cli) lists every command and flag. ```bash npx better-compute help # the commands, grouped npx better-compute help reserve # one command's flags and examples npx better-compute help --json # the whole manifest, as the reference is generated ``` With some 180 commands, load shell completion (581d1fb): `better-compute completion bash`, `zsh` (through `bashcompinit`) or `fish` prints a script, generated from the program's commands (a configuration's project commands included), that completes command words, the flags of the command on the line and the values of flags with choices. Only plainly shaped names are emitted, and every word list and description is single-quoted without control characters, so nothing a command defines runs in your shell. ```bash source <(better-compute completion bash) # bash: in ~/.bashrc source <(better-compute completion zsh) # zsh: in ~/.zshrc better-compute completion fish | source # fish: in ~/.config/fish/config.fish ``` ## Two kinds of command [#two-kinds-of-command] Every command declares a target, and the target decides what it needs: | Target | Needs | Commands | | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `token` | a credential (`BETTER_COMPUTE_TOKEN` or a saved one, see [authenticate](#authenticate)), and either `--url` or a configuration | `api`, every command of the [command groups](#command-groups) below (`nodes`, `instances`, `rollouts`, `metal`, `webhooks`, ...), `reserve`, `release`, `extend`, `reservations list/get`, `pools list`, `capacity`, `sandbox …`, `sandbox-template …`, `sandbox-snapshot …`, `console`, `exec`, `forward`, `ssh-cert`, `fleet export/plan/status/apply`, `batch list/show/approve/create/members/cancel`, `auth login` | | `config` | the configuration (`--config`, `BETTER_COMPUTE_CONFIG`, the nearest `better-compute.config.{mjs,js,ts,mts}`, or `BETTER_COMPUTE_DATABASE_URL`) and no credential | `migrate`, `doctor`, `serve`, `audit checkpoint/verify/prove` | | `none` | neither | `init`, `version`, `completion`, `dev` (it finds a configuration itself, or runs PGlite), `sim`, `pki`, `daemon install-script`, `iam setup`, `release keygen/sign/cosign/verify`, `batch keygen/sign/sign-approvers`, `audit check-proof` | A `config` command talks to the database directly. It is an operator tool: whoever can run it can already read the database, so it is not decided by IAM. A `token` command never reads the database itself. ## Authenticate [#authenticate] Token commands act as a Better IAM session token, API key or session JWT. They take the first credential they find, in PLAN.md section 9.3's order: 1. `BETTER_COMPUTE_TOKEN`; 2. Better IAM's saved session (`better-iam login`), when the optional `@better-iam/cli` peer is installed: its profile store, profile `BETTER_IAM_PROFILE` or its current one; 3. compute's own credentials file, written by `better-compute auth login`: profile `BETTER_COMPUTE_PROFILE`, else the file's current profile, else `default`. Expired saved sessions are skipped, and a broken Better IAM file never blocks compute's own. Without any credential token commands stop with `MISSING_ENV`; public methods such as `system.version` need no token. ```bash export BETTER_COMPUTE_URL=https://compute.example.com export BETTER_COMPUTE_TOKEN=biam_… # from Better IAM: an API key or a session token export BETTER_COMPUTE_TENANT=acme # the default --tenant better-compute api system.whoami ``` **Saved credentials.** `auth login` reads the token from standard input or `BETTER_COMPUTE_TOKEN`, never from an argument, checks it with `system.whoami` against the deployment (`--url`, or the configuration in process; `--no-verify` skips the check), and saves it with its tenant under `--profile`, which becomes the current profile: ```bash better-compute auth login --url https://compute.example.com < token.txt BETTER_COMPUTE_TOKEN=biam_key_… better-compute auth login --profile ci --no-verify better-compute auth status # which credential commands use: source, profile, path; never a token better-compute auth logout --profile ci ``` The file is `BETTER_COMPUTE_CREDENTIALS`, else `%APPDATA%\better-compute\credentials.json` on Windows, else `$XDG_CONFIG_HOME/better-compute/credentials.json` or `~/.config/better-compute/credentials.json`. It is found only from the environment, and every write replaces it atomically with mode 0600; treat it like a password. `auth logout` forgets a profile but does not revoke the session in Better IAM (`better-iam logout` does). A saved profile records the `--url` it was saved for, but commands still take the URL from `--url` or `BETTER_COMPUTE_URL`. The profile's token is used only for that deployment: a command whose URL has another origin, or a profile saved in process (without `--url`) when a command calls a remote server, gets no token from it, so a token for one deployment is never sent to another. **Where the call goes.** * With `--url` (or `BETTER_COMPUTE_URL`) the command calls a running server over HTTP through the typed client, with a request id on every call and one retry after a short `RATE_LIMITED`. The URL is an origin (`https://compute.example.com`, base path `/api/compute`) or a full endpoint (`https://host/custom/compute`). Plain `http` is allowed only for `localhost`, `127.0.0.1`, `[::1]` and `*.localhost`, so a token never crosses a network in clear text. * Without `--url` the command loads your configuration and calls `compute.handler` in process. It still goes through the route table, the envelope, the CSRF rules and the 64 KiB body limit, and IAM still decides as the token, so a command behaves the same locally and remotely. `--tenant` names the tenant acting, and defaults to `BETTER_COMPUTE_TENANT`. It must be the token's own tenant, or the root tenant for platform operators; anything else is `ACCESS_DENIED` before compute reads its database. ## Output [#output] Commands that print a result share two flags: | Flag | Does | | ------------------ | ------------------------------------------------------------------------------------------------ | | `--format json` | indented JSON (the default) | | `--format compact` | JSON on one line, for piping | | `--format table` | a table, for lists | | `--query PATH` | prints part of the result, like `binding`, `items[].id` or `findings[].check`; strings print raw | A page of a list method (`{ items, nextCursor }`) printed as a table ends with `(more rows: --cursor …)` when there is a next page (5df9455). Output is terminal-safe, since results carry the server's words (names, reasons, messages): a string result, or a `--query` that selects one, prints with its control characters turned into spaces (line breaks and tabs stay); table cells are one line without control characters; JSON also escapes DEL and the C1 controls (U+007F to U+009F). A failure prints its code and message, plus a hint line saying what to run or set next. Usage mistakes (an unknown flag, a missing value) exit 2 with `INVALID_ARGUMENT`, and unknown commands and flags get a "did you mean" suggestion. `BETTER_COMPUTE_DEBUG=1` prints the stack trace of a failure. ## Call any method [#call-any-method] `better-compute api GROUP.METHOD` calls any method, including plugin methods (`plugins.ID.GROUP.METHOD`). Build the input from `key=value` items, `--data`, or both: | Item | Sets | | ---------------------------------------------- | ----------------------------------------------------------- | | `name=gpu-a` | a string | | `limit:=10`, `labels:='["a","b"]'` | a JSON value | | `spec:=@spec.json` | a JSON value read from a file | | `notes=@notes.md` | a string read from a file | | `resource.type=pool` | a nested field | | `ids[]=a` | an item appended to an array | | `--data JSON`, `--data @file.json`, `--data -` | the whole body, from the argument, a file or standard input | ```bash better-compute api --list # every route, whether it needs a token, and its IAM action better-compute api --list reservations # one group better-compute api events.list tenantId=acme type=reservation. limit:=20 --format table better-compute api webhooks.create tenantId=acme url=https://hooks.example.com/compute 'events[]=reservation.*' better-compute api reservations.get tenantId=acme reservationId=rsv_… --query binding ``` `api --list` reads the deployment's route table, so it shows plugin routes and marks the public ones. Anything `api` can reach, the [API reference](/docs/reference/api) documents. ## Command groups [#command-groups] PLAN.md section 9.3 has the commands mirror the API groups, and since 171d9be every API group has commands or an existing equivalent (44915e5, e2fe6f8, bb89083, d1e83ca, 573b0b0, 171d9be). Each one is the API call as the token's identity, with typed flags in place of `key=value` items; `api` stays the way to any method a group's commands leave out (a raw `commands.run`, for one) and to plugin methods. | Commands | What they do | API group | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`nodes`](/docs/reference/cli#nodes-list) `list`, `get`, `events`, `cordon`, `uncordon`, `drain`, `label`, `taint`, `delete` | Read nodes and their events; cordon, drain, label and taint them (`--if-version` for optimistic concurrency). [`delete`](/docs/reference/cli#nodes-delete) (4049621) removes a retired or cordoned adopted node for good and refuses without `--yes` before any call; see [removing an adopted host](/docs/guides/nodes/operating-nodes#removing-an-adopted-host) | [nodes](/docs/reference/api/nodes) | | [`join-tokens`](/docs/reference/cli#join-tokens-create) `create`, `list`, `revoke` | Mint the tokens `bcd join` presents | [joinTokens](/docs/reference/api/join-tokens) | | [`join-requests`](/docs/reference/cli#join-requests-list) `list`, `approve`, `reject` | Admit or turn down machines that asked to join | [joinRequests](/docs/reference/api/join-requests) | | [`inventory`](/docs/reference/cli#inventory-get) `get`, `history`, `diff` | A node's hardware inventory and what changed | [inventory](/docs/reference/api/inventory) | | [`site-tokens`](/docs/reference/cli#site-tokens-create) `create`, `revoke` | The tokens a site agent joins with | [siteTokens](/docs/reference/api/site-tokens) | | [`instances`](/docs/reference/cli#instances-list) `list`, `get`, `create`, `start`, `stop`, `delete`, `resize` | VPS instances; see [instances](/docs/guides/vps/instances#from-the-command-line) | [instances](/docs/reference/api/instances) | | [`events`](/docs/reference/cli#events-list) `list`, `watch` | Page through events, or follow them live (below) | [events](/docs/reference/api/events) | | [`releases`](/docs/reference/cli#releases-register) `register`, `list` | Signed bcd release manifests | [releases](/docs/reference/api/releases) | | [`rollouts`](/docs/reference/cli#rollouts-create) `create`, `list`, `get`, `pause`, `resume`, `abort` | Wave rollouts of a bcd release; see [rollouts](/docs/guides/rollouts) | [rollouts](/docs/reference/api/rollouts) | | [`workflows`](/docs/reference/cli#workflows-list) `list`, `get`, `abort`, `pause`, `resume`, `retry` | A node's workflows (`--node`) | [workflows](/docs/reference/api/workflows) | | [`commands`](/docs/reference/cli#commands-list) `list`, `get`, `cancel` | A node's queued commands (`--node`); there is no `commands run` | [commands](/docs/reference/api/commands) | | [`sites`](/docs/reference/cli#sites-list) `list`, `get`, `create`, `update`, `delete` | Sites | [sites](/docs/reference/api/sites) | | [`projects`](/docs/reference/cli#projects-list) `list`, `get`, `create`, `archive` | Projects | [projects](/docs/reference/api/projects) | | [`images`](/docs/reference/cli#images-list) `list`, `get`, `register`, `promote` | The image catalog (`register` takes `--disk-format`) | [images](/docs/reference/api/images) | | [`flavors`](/docs/reference/cli#flavors-list) `list`, `create`, `update`, `delete` | VPS flavors of a pool | [flavors](/docs/reference/api/flavors) | | [`quotas`](/docs/reference/cli#quotas-get) `get`, `set` | Quotas per pool and resource | [quotas](/docs/reference/api/quotas) | | [`usage`](/docs/reference/cli#usage-list) `list`, `export` | Usage windows, and a billing export (`--export-format csv` or `jsonl`) | [usage](/docs/reference/api/usage) | | [`webhooks`](/docs/reference/cli#webhooks-list) `list`, `get`, `create`, `update`, `delete`, `ping`, `rotate-secret`, `deliveries`, `redeliver`, `event-types` | Webhook subscriptions (`--endpoint` is the receiver's URL) | [webhooks](/docs/reference/api/webhooks) | | [`pools`](/docs/reference/cli#pools-get) `get`, `create`, `update`, `archive`, `offer`, `unoffer`, and [`pools list`](/docs/reference/cli#pools-list) | Pools and their offers | [pools](/docs/reference/api/pools) | | [`drivers list`](/docs/reference/cli#drivers-list) | The registered drivers | [drivers](/docs/reference/api/drivers) | | [`secrets`](/docs/reference/cli#secrets-put) `put`, `delete` | Write-only secrets, the value from a file or standard input only | [secrets](/docs/reference/api/secrets) | | [`metal`](/docs/reference/cli#metal-enroll) `enroll`, `manage`, `inspect`, `provide`, `rescue`, `unrescue`, `recover`, `retire`, `deploy`, `undeploy`, `abort`, `discovered`, `confirm-mac`, `power`, `bulk` | The bare-metal lifecycle; see [deploying](/docs/guides/bare-metal/deploying#from-the-command-line). `discovered` and `confirm-mac` (002dd95) list a site's PXE discovery and bind a machine's PXE MAC; see [discovering new machines](/docs/guides/bare-metal/sites#discovering-new-machines) | [metal](/docs/reference/api/metal) | | [`profiles`](/docs/reference/cli#profiles-list) `list`, `get`, `validate`, `render`, `create`, `update`, `delete` | Provision profiles, from JSON documents | [profiles](/docs/reference/api/profiles) | | [`provider-accounts`](/docs/reference/cli#provider-accounts-list) `list`, `get`, `create`, `update`, `delete`, `offers`, `bootstrap` | Provider accounts (`--config-file`, credentials in it as references) | [providerAccounts](/docs/reference/api/provider-accounts) | | [`reserve`](/docs/reference/cli#reserve), `release`, `extend`, `reservations list`, `reservations get` | Reservations ([below](#capacity)) | [reservations](/docs/reference/api/reservations) | | [`capacity`](/docs/reference/cli#capacity) | Free capacity per pool and GPU model, and a preview | [capacity](/docs/reference/api/capacity) | | [`sandbox`](/docs/reference/cli#sandbox-create), [`sandbox-template`](/docs/reference/cli#sandbox-template-create), [`sandbox-snapshot`](/docs/reference/cli#sandbox-snapshot-list), [`sandbox-volume`](/docs/reference/cli#sandbox-volume-create) | Sandboxes, their templates, their snapshots and their persistent volumes ([below](#sandboxes)) | [sandboxes](/docs/reference/api/sandboxes), [sandboxTemplates](/docs/reference/api/sandbox-templates), [sandboxSnapshots](/docs/reference/api/sandbox-snapshots), [sandboxVolumes](/docs/reference/api/sandbox-volumes) | | [`console`](/docs/reference/cli#console), `exec`, `forward`; [`ssh-cert`](/docs/reference/cli#ssh-cert) | Tunnels and break-glass SSH ([below](#interactive-access)) | [tunnels](/docs/reference/api/tunnels), `nodes.issueSshCert` | | [`fleet`](/docs/reference/cli#fleet-export) `export`, `plan`, `status`, `apply` | [Fleet configuration](#fleet-configuration) | [fleet](/docs/reference/api/fleet) | | [`batch`](/docs/reference/cli#batch-list) `list`, `show`, `approve`, `create`, `members`, `cancel` | [Destructive batches](#destructive-batches) | [commandBatches](/docs/reference/api/command-batches) | | [`auth login`](/docs/reference/cli#auth-login) and `api system.whoami` | Check a token (`auth login` calls `system.whoami`) | [system](/docs/reference/api/system) | What they share: * **Ids are checked before anything is sent.** An id that does not look like its kind (`node_…`, `inst_…`, `rollout_…`, `wf_…`, `cmd_…`, `pool_…`, `jreq_…`, `pacct_…`, `snap_…`, `vol_…`) is a usage error, exit 2. * **Lists print tables.** List commands default to `--format table`, and a page with more rows ends with `(more rows: --cursor …)` (5df9455); pass that cursor to get the next page. * **Creates are retry-safe.** A command that creates something sends a new idempotency key on every run unless you pass `--key`; repeat with the same `--key` to retry without creating twice. * **Secrets never pass through arguments or the output unless you ask.** `secrets put` reads `--value-file` or `--stdin`; `metal enroll` takes a BMC password only from `--bmc-password-file`; `join-tokens create` and `site-tokens create` with `--out`, and `webhooks create` and `rotate-secret` with `--secret-out`, write the one-time secret to a new 0600 file instead of printing it, never over an existing file (`ALREADY_EXISTS`), and so does `usage export --out` with its export. `join-tokens revoke` and `site-tokens revoke` refuse a pasted token (anything with a `.`), so a secret is never sent as an id. * **Bulk work goes through the quorum.** `metal bulk` runs one verb on the nodes of `--nodes` or `--nodes-file` (deduplicated) as one batch, and its required `--reason` is what the batch's approvers see. `tests/cli-framework.test.ts` ties the commands to the API (fb6304f): every `group.method` a command summary names, and every route a command calls, must exist in the method table and the live registry, so a renamed route fails the suite rather than a user's first call. The [CLI reference](/docs/reference/cli) has every command's flags. ### Follow events live [#follow-events-live] `events watch` (e2fe6f8) follows the host's server-sent events route, the same stream as the client's `$watch`, and prints one JSON line per event: ```bash better-compute events watch --url https://compute.example.com --tenant acme --types 'node.*,reservation.*' better-compute events watch --url https://compute.example.com --count 100 better-compute events watch --url https://compute.example.com --path /api/compute/watch # the console app's route ``` * It needs `--url` (`UNSUPPORTED` without it): it follows a running server's stream. `--path` defaults to `/watch`, the route the examples' hosts mount; the console app serves `/api/compute/watch`. * `--tenant` is sent as `?tenantId=`, which the host's watch authorizer decides as `events.list` for that tenant. * Each line carries the event's `sequence`, the stream's cursor, and `--since` resumes after one: pass the last `sequence` a previous run printed (86706b4). It reconnects after a broken connection from the last event, and stops on Ctrl-C, after `--count` events, or on a refusal (a revoked session, a withdrawn `compute:events:read`). * `--types` takes up to 20 type patterns; they and `--since` are checked before it connects. * Lines are terminal-safe: DEL, the C1 controls and U+2028/U+2029, which JSON leaves raw, are escaped as `\uXXXX`. Both event commands take an outbox position, written two ways. `events watch --since` accepts either form and sends the stream the dotted one (867c837), so a watch can resume from where an `events list` page ended; `events list --cursor` takes only the colon form: | Command | Flag | Form | Where it comes from | | -------------- | ---------- | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | | `events watch` | `--since` | `.`, with a dot (`184467.12`), or `:` | The `sequence` of a line the stream printed (`formatCursor` in `@better-compute/events`), or an `events.list` cursor | | `events list` | `--cursor` | `:`, with a colon (`184467:12`) | The `nextCursor` of the previous `events.list` page | ```bash # resume a watch after the last event a previous run printed better-compute events watch --url https://compute.example.com --since 184467.12 --count 100 # page through the tenant's events better-compute events list --limit 100 --cursor 184467:12 ``` See [watch](/docs/guides/events/watch). ## Capacity [#capacity] The capacity commands wrap the reservation methods with friendlier flags: ```bash better-compute pools list better-compute capacity # per pool and per GPU model better-compute capacity --preview --pool pool_01k… --gpus 8 --for 1d --selector "vramGiB >= 80" better-compute reserve --pool pool_01k… --gpus 4 --for 8h --nvlink better-compute reservations list --state active --format table better-compute extend rsv_01k… --by 4h better-compute release rsv_01k… ``` `reserve` sends a new idempotency key on every run unless you pass `--key`, and prints the key it used. To retry the same request, whether after a timeout or after a person approved a `CONFIRMATION_REQUIRED` call, repeat the command with that `--key`. It waits up to `--wait` seconds (default 20) while the reservation is queued. `--selector` takes a shorthand such as `gpuProduct in (H100-SXM5-80GB), vramGiB >= 80`, and `--model` and `--product` add a `gpuModel` or `gpuProduct` match. See [reservations](/docs/guides/capacity/reservations). ## Sandboxes [#sandboxes] The `sandbox` commands (help group `Sandboxes`) wrap the `sandboxes` and `sandboxTemplates` methods of [sandboxes](/docs/guides/vps/sandboxes): ```bash better-compute sandbox templates --format table # sandboxTemplates.list, with warm pools better-compute sandbox create --template sbxt_01k5w… --ttl 30m # sandboxes.create better-compute sandbox list --all --template sbxt_01k5w… better-compute sandbox get sbx_01k5w… # with its snapshots better-compute sandbox extend sbx_01k5w… --by 30m better-compute sandbox snapshot sbx_01k5w… --keep better-compute sandbox exec sbx_01k5w… -- pytest -q better-compute sandbox delete sbx_01k5w… ``` | Command | Flags | Calls | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | | `sandbox create` | `--template ID` (required), `--ttl DURATION` (`15m`, `2h`; whole seconds), `--from-snapshot ID`, `--wait SECONDS` (0 to 30, default 30), `--key KEY` | `sandboxes.create`, with a new idempotency key unless `--key` is given; waits until the sandbox leaves `starting` | | `sandbox list` | `--all` (the tenant's, not only yours), `--deleted`, `--template ID`, `--limit N` (1 to 200, as the API), `--cursor` | `sandboxes.list` | | `sandbox get SANDBOX` | | `sandboxes.get` | | `sandbox extend SANDBOX` | exactly one of `--by DURATION` or `--until TIME` (ISO 8601, epoch ms, or `+DURATION` from now) | `sandboxes.extend` | | `sandbox snapshot SANDBOX` | `--no-memory` (disk only), `--keep` (outlive the sandbox), `--key KEY` | `sandboxes.snapshot`, always with an idempotency key | | `sandbox delete SANDBOX` | | `sandboxes.delete` | | `sandbox exec SANDBOX [-- COMMAND…]` | `--key PATH` (default `~/.ssh/id_ed25519`), `--minutes N` (1 to 15, default 10), `--ssh PROGRAM` (default `ssh`), `--accept-host-key` (trust an uncertified guest's key on first use) | `sandboxes.exec`, then OpenSSH through the tunnel | | `sandbox templates` | | `sandboxTemplates.list` | A sandbox id that does not look like `sbx_…` is a usage error (exit 2) before anything is sent. `sandbox-snapshot list` (`--all`, `--sandbox ID`), `get` and `delete` (171d9be) wrap `sandboxSnapshots`, so a kept snapshot of a deleted sandbox can be found and removed. **`sandbox exec`** gets a short-lived certificate for your own key and runs `ssh` through a single-use tunnel; there is no agent in the guest. It sends the public key beside `--key` (`.pub`; a file that holds a private key is refused) and never the private key, writes the certificate into a temporary directory, listens on a random `127.0.0.1` port for one connection that it bridges to the tunnel, and runs `--ssh` with `-F none` (no user or system `ssh_config`), `-i `, `CertificateFile`, `IdentitiesOnly=yes` and a known-hosts file of its own (as both `UserKnownHostsFile` and `GlobalKnownHostsFile`), as the account the server names. A guest with a still-valid host certificate from the sandbox CA (the grant's `hostCertified`) is pinned: the file holds `@cert-authority `, with `HostKeyAlias=` and `StrictHostKeyChecking=yes`; a malformed `hostPrincipal` is `INVALID_RESPONSE`. Any other guest (from before host certificates, or whose certificate expired) is refused with `INVALID_STATE` and nothing is spawned, unless you pass `--accept-host-key`: then it gets an empty file, `StrictHostKeyChecking=accept-new` and `HostKeyAlias=`, with a note that its key is trusted on first use for this connection. The account must be a Unix account name (`INVALID_RESPONSE` otherwise). The command after `--` runs remotely; without one you get a login shell. The temporary directory is removed afterwards, whatever happened, and the CLI exits with ssh's own status (`REMOTE_EXIT`). Like the other tunnel commands it needs `--url`, and `sandboxes.exec` needs a signed-in session rather than an API key. It is refused (`INVALID_STATE`, `restore-shares-identity`) while a memory restore and its source, or two restores of one memory snapshot, are live together and still share one SSH identity, that is until the host gives the restore its own. The CLI then adds a hint (9958865): delete the sandbox it was restored from or its sibling restores, or wait until `sandbox get` shows `identityRebound: true`. See [a restore's own SSH identity](/docs/guides/vps/sandboxes#a-restores-own-ssh-identity). The guest is untrusted, so the connection is hardened against it and against the command you pass: * a `--` precedes the host, so the remote command is never parsed as an ssh option (`sandbox exec sbx_… -- -oProxyCommand=…` runs that text in the guest, not on your machine), which matters when an agent or a model writes the command; * none of your ssh configuration that would hand the guest anything applies: `-a` and `-x`, `ForwardAgent=no`, `ForwardX11=no`, `ClearAllForwardings=yes`, `ProxyCommand=none`, `ProxyJump=none`, `ControlMaster=no`, `ControlPath=none` and `PermitLocalCommand=no`; * the `CertificateFile` and `UserKnownHostsFile` paths are quoted, since ssh splits `-o` values on whitespace and a home or temporary directory may contain a space. ### Sandbox templates [#sandbox-templates] Tenant administrators define templates with `sandbox-template`, decided as `compute:sandboxes:manage`: ```bash better-compute sandbox-template create --name python --pool pool_01k5w… --flavor small --image img_01k5w… \ --warm-min 2 --max-ttl 4h --env OPENAI_API_KEY=secret://acme/sec_01k5w… --network private better-compute sandbox-template update sbxt_01k5w… --warm-min 4 --image img_01k5w… better-compute sandbox-template delete sbxt_01k5w… ``` | Command | Flags | Calls | | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- | | `sandbox-template create` | `--name NAME` and `--pool ID` (required), `--flavor NAME` and `--image ID` (needed), `--runtime ID` (checked against the pool's), and the template flags below | `sandboxTemplates.create` | | `sandbox-template update TEMPLATE` | the template flags, `--no-user-data` (remove it) | `sandboxTemplates.update` with only the fields given | | `sandbox-template delete TEMPLATE` | | `sandboxTemplates.delete` | The template flags are `--flavor`, `--image`, `--user-data FILE` (cloud-init read from a file), `--default-ttl` and `--max-ttl` (durations such as `30m` or `4h`, whole seconds), `--warm-min N` and `--warm-max N` (the API accepts 0 to 64; `--warm-max` needs `--warm-min`), `--env NAME=secret:///,…` (secret references, never values) and `--network egress|private|none`, and `--volumes NAME:/PATH[:ro],…` (f181f43): the persistent volumes every sandbox mounts, in order, each by name or by id (an entry that looks like `vol_…` is sent as `volumeId`), with `:ro` for a read-only mount; `--volumes ""` removes them all, and an entry without an absolute path or with a mode other than `ro` is a usage error. An `update` without any field flag, or with both `--user-data` and `--no-user-data`, is a usage error, and so is a template id that does not look like `sbxt_…`. See [templates](/docs/guides/vps/sandboxes#templates). ### Sandbox volumes [#sandbox-volumes] `sandbox-volume` (f181f43) manages [persistent volumes](/docs/guides/vps/sandboxes#volumes), an agent's memory, with the `sandboxVolumes` methods: ```bash better-compute sandbox-volume create --pool pool_01k5w… --name agent-notes --size 20 better-compute sandbox-volume list --pool pool_01k5w… --format table better-compute sandbox-volume get vol_01k5w… better-compute sandbox-volume resize vol_01k5w… --size 40 better-compute sandbox-volume delete vol_01k5w… better-compute sandbox-template update sbxt_01k5w… --warm-min 0 --warm-max 0 --volumes agent-notes:/workspace ``` | Command | Flags | Calls | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------- | | `sandbox-volume create` | `--pool ID`, `--name NAME` and `--size GIB` (1 to 16,384) required, `--key KEY` | `sandboxVolumes.create`, with a new idempotency key unless `--key` is given | | `sandbox-volume list` | `--all` (every volume of the tenant you may see, not only yours; e68aba7), `--pool ID` (only that pool's), `--deleted` (include deleted volumes) | `sandboxVolumes.list`, with `scope: 'all'` for `--all` and the API's default, your own volumes, otherwise | | `sandbox-volume get VOLUME` | | `sandboxVolumes.get` | | `sandbox-volume resize VOLUME` | `--size GIB` (1 to 16,384, required); volumes only grow | `sandboxVolumes.resize` | | `sandbox-volume delete VOLUME` | | `sandboxVolumes.delete`, which wipes the data | A volume id that does not look like `vol_…` is a usage error before anything is sent. A template that mounts volumes keeps no warm pool, so give it `--warm-min 0 --warm-max 0` (the API refuses one whose `warm.max` is not 0), and naming a volume in a template takes a decision on that volume (see [attach](/docs/guides/vps/sandboxes#volumes)). The console's **Volumes** page does the same list, create, grow and delete (see [the console](/docs/guides/clients/console#volumes)). ## Interactive access [#interactive-access] Four commands in the `Access` group reach a node or an instance from your terminal, decided like any other call as the token's identity: ```bash better-compute console --url https://compute.example.com --tenant acme --node node_01k5w… better-compute console --url https://compute.example.com --tenant acme --instance inst_01k5w… better-compute exec --url https://compute.example.com --tenant acme --instance inst_01k5w… better-compute forward --url https://compute.example.com --tenant acme --instance inst_01k5w… --port 5432 better-compute ssh-cert --tenant acme --node node_01k5w… && ssh -i ~/.ssh/id_ed25519 root@10.0.0.5 ``` * **`console`** opens a tunnel with op `console`: with `--node`, a root console on the node, decided as `compute:nodes:console` on the node; with `--instance`, the serial console of a running instance, decided as `compute:instances:exec` on the instance (798e710). Give exactly one of the two. An instance console needs a node daemon that advertises `tunnel.instance-dial`, like `forward` below, and a sandbox's instance is refused (`sandbox-managed`: use `sandbox exec`). **`exec`** (`--instance`) opens a shell in a running instance with op `exec`, decided as `compute:instances:exec`; the instance's node agent runs it. bcd refuses instance `exec` until guests have an agent, so on a real host `console --instance` is the terminal that works. * Both open the tunnel's WebSocket on the same server as the API call, so they need `--url` (or `BETTER_COMPUTE_URL`). Without it they fail with `UNSUPPORTED` before a ticket is spent: an in-process call has no WebSocket server. * The terminal is in raw mode until the remote program exits: every key goes to the remote, Ctrl-C included, and the window size follows resizes. **Ctrl-]** detaches. The command exits with the remote program's status, as ssh does, and with 255 when the tunnel itself breaks (`REMOTE_EXIT`). * **`forward`** (`--instance`, `--port` 1 to 65535) works like `ssh -L`: it listens on `--bind` (default `127.0.0.1`) at `--local-port` (default the same as `--port`; `0` picks a free one) and forwards each accepted connection to that port of the running instance through a `tcp` tunnel of its own, a fresh `tunnels.open` per connection decided as `compute:instances:exec`. It runs until Ctrl-C. The token is resolved before anything listens, and it needs `--url` like `console` and `exec`. A connection that cannot be opened is closed with a note; the tunnel has no half-close, so a client that ends its side ends the connection. Memory stays bounded: the local socket pauses while the tunnel holds more than 4 MiB unsent and resumes below 1 MiB, and a local reader more than 16 MiB behind is disconnected, since the WebSocket's receiving side cannot be paused. A `--bind` other than loopback lets anyone who reaches that address use your access. Instance `tcp` and `ssh` tunnels open only when the node's daemon advertises the `tunnel.instance-dial` capability in its `Hello`; otherwise `tunnels.open` answers `UNSUPPORTED` with `details.reason: 'daemon-capability'`. * **`ssh-cert`** sends your own public key (`--key`, default `~/.ssh/id_ed25519.pub`, else `id_ecdsa.pub`) to `nodes.issueSshCert` and writes the node-bound certificate beside it as `-cert.pub`, where `ssh -i ` finds it. `--account` (default `root`) is the Unix account and `--minutes` (default 15, 1 to 60) the validity, never past your session. It needs `compute:nodes:ssh-root` and, by default, an MFA session. The private key never leaves your machine: a `--key` that is not a `.pub` file, or that holds a private key, is refused before anything is sent. The answer is checked before use: an account that is not a Unix account name, or a certificate that is not an OpenSSH user certificate for your key, is `INVALID_RESPONSE` and nothing is written. With **`--connect HOST`** it then runs your OpenSSH client (`--ssh`, default `ssh`) to the node's address with the new certificate: `-i -o CertificateFile= -o IdentitiesOnly=yes -l -- HOST`, followed by the command after `--`, or a shell without one. `HOST` must be a host name or an IP address, never something ssh would read as an option (`-oProxyCommand=…`), and is checked before a certificate is issued; a command without `--connect` is a usage error. The command exits with ssh's status (`REMOTE_EXIT`). Unlike `sandbox exec`, it keeps your ssh configuration on purpose, so bastions and jump hosts still apply: ```bash better-compute ssh-cert --tenant acme --node node_01k5w… --connect 10.0.0.5 -- journalctl -u bcd -n 50 ``` See [tunnels](/docs/guides/access/tunnels) and [break-glass SSH](/docs/guides/access/ssh). ## Fleet configuration [#fleet-configuration] `fleet` manages a tenant's platform objects (sites, provider accounts, pools with quotas and offers, VPS flavors, sandbox templates) as one [FleetConfig](/docs/guides/fleet-config) document. The loop is export, edit, plan, review, apply: ### Export [#export] ```bash better-compute fleet export --out fleet.json ``` Objects the fleet did not create are keyed by id. Rename them in the document and the next apply keeps them. Secrets appear only as references. ### Plan [#plan] ```bash better-compute fleet plan fleet.json # + create, ~ update, - delete (only with --prune), BLOCKED better-compute fleet plan fleet.json --calls # also the API calls apply would make ``` A plan changes nothing and prints a `planHash`. A plan with a blocked change (a site's region, a pool's class or site, a dangling name) exits `PLAN_BLOCKED`. ### Apply the reviewed plan [#apply-the-reviewed-plan] ```bash better-compute fleet apply fleet.json --plan-hash 3f1c… ``` Every underlying call is decided before anything runs, then all of them run in one transaction, so a denial or a failure applies nothing. If the fleet changed since the plan, the hash no longer matches and the command exits `CONFIG_DRIFT`. Without `--plan-hash` or `--yes`, `apply` only plans. In CI, fail a job when the live fleet no longer matches the committed document, or when it drifted from the last apply: ```bash better-compute fleet plan fleet.json --fail-on-drift # exits FLEET_DRIFTED when the plan is not empty better-compute fleet status --fail-on-drift # exits FLEET_DRIFTED while the fleet.drift job sees drift ``` The fleet methods are decided on `compute:fleet:admin`, which only the root tenant's administrators hold. The MCP server's fleet tools stop at `fleet_plan` for the same reason: a person applies. See [fleet configuration](/docs/guides/fleet-config). ## Destructive batches [#destructive-batches] With the v1.0 destructive-batch quorum on (`pki.approvers`), fleet-wide destructive commands run only once enough operator approver keys have signed their batch. The `batch` commands are the approver's side (b3fd712, 6a7f1b7). They never take the server's word for what a batch holds: every review pages the batch's scope and recomputes its Merkle root before anything is signed. | Command | Needs | Does | | ----------------------------------------------------------------- | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `batch keygen --out PATH` | nothing | a P-256 approver key (mode 0600, never overwritten) and `PATH.pub`; prints the `kid`. Encrypted when `BETTER_COMPUTE_APPROVER_PASSPHRASE` is set | | `batch list [--state STATE]` | a credential, `compute:batches:read` | `commandBatches.list`: reason, state, size, approvals had and needed | | `batch show --batch ID [--scope-out FILE]` | a credential, `compute:batches:read` | the batch with its verified scope; `--scope-out` writes the input of `batch sign` | | `batch approve --batch ID --key PATH` | a credential, `compute:batches:approve` | reviews the batch, then signs and submits your approval | | `batch approve --batch ID --signature FILE` | a credential, `compute:batches:approve` | submits an approval made offline, after checking it is for this batch | | `batch sign --key PATH --scope FILE` | nothing | the same review and signature on an offline machine | | `batch create --reason TEXT --scope FILE [--hours N] [--key KEY]` | a credential, `compute:batches:create` | `commandBatches.create` from `{ subject, kind, payload }` leaves, each payload encoded as its kind's message so the leaf binds that operation (def72c8) | | `batch members --batch ID [--state S]` | a credential, `compute:batches:read` | the batch's commands, with `awaitingApproval` while held (45d217e) | | `batch cancel --batch ID` | a credential, `compute:batches:approve` | ends a pending batch; its held commands are cancelled | | `batch sign-approvers --key PATH --epoch N ...` | nothing | a current approver's signature over a new approver set at its epoch, `N` one above the published set's (c6214dc; see [changing the approver set](/docs/operations/pki#changing-the-approver-set)) | ```bash better-compute batch approve --tenant acme --batch batch_01k5w… --key ~/.better-compute/approver.pem # Batch batch_01k5w…: reimage rack 12 after the firmware recall # 12 command(s) on 12 subject(s): deploy.image ×12: # deploy.image: node_01…, node_02…, … # Approvals: 0 of 2; the window closes at 2026-09-25T18:00:00.000Z. # Not approved. Re-run with --yes, or with --root 3f1c…. better-compute batch approve --tenant acme --batch batch_01k5w… --key ~/.better-compute/approver.pem \ --root 3f1c… --expect rack12.txt ``` `batch approve` signs only what it showed you. Without `--yes` it stops after the summary; `--root` approves only if the batch still has the root you reviewed. `--expect FILE` (one subject per line, or `{ subject, kind }` commands) approves only a batch that holds exactly those. Leaves that admit any payload of their kind are refused unless `--allow-unbound`, and a batch that stays open longer than `--max-hours` (default 72) is refused. The signature covers the batch id, root, size, expiry and the deployment's trust domain. On an offline machine, write the scope with `batch show --scope-out batch.json`, carry it over, run `batch sign --key … --scope batch.json` there (a bare `--root` and `--size` needs `--unchecked`), and bring the output back to `batch approve --signature`. A batch an agent proposed for a person (MCP's `propose_command_batch`) carries the agent's own reason. `batch approve` then notes `Proposed by agent X for Y (delegation D): the reason is the agent's own text.` after the reason line, and `batch show` returns the same words as `proposedBy`; for a person's batch `proposedBy` is just their identity (bd80209). See [the destructive-batch quorum](/docs/operations/pki#the-destructive-batch-quorum-v10). Approver key files readable by other users are refused, as ssh does. Every refusal has its own exit code: | Code | Meaning | | ------------------ | ------------------------------------------------------------------------------------------------------------------------- | | `SCOPE_MISMATCH` | the listed leaves do not give the root and size the server asks you to sign, or a `--signature` file signs something else | | `SCOPE_CHANGED` | the batch no longer has the `--root` you reviewed | | `SCOPE_UNEXPECTED` | the batch is not what `--expect` lists; the message names the differences | | `UNBOUND_SCOPE` | leaves admit any payload and `--allow-unbound` was not given | | `INVALID_RESPONSE` | the server answered a scope page that is malformed or does not advance | | `INVALID_STATE` | the batch is not pending, its window is closed or longer than `--max-hours`, or the key file is readable by others | See [the quorum](/docs/operations/pki#the-destructive-batch-quorum-v10) for how batches are created, held and checked on each machine. ## Audit [#audit] The audit commands are operator commands: they read `audit_events` and the checkpoints straight from the database through your configuration, not through the API. ```bash better-compute audit verify --tenant acme # recompute every checkpoint; AUDIT_TAMPERED on any problem better-compute audit verify --tenant acme --query problems better-compute audit prove aud_01k… --tenant acme --out proof.json better-compute audit check-proof proof.json --pin sha256/… # offline, no database better-compute audit checkpoint # seal now, as the hourly job does ``` `verify` and `check-proof` check signatures under the audit key's pin: `--pin`, `BETTER_COMPUTE_AUDIT_PIN`, or the public half of the PKI bundle (`--pki`, default `.better-compute/pki.json`). `checkpoint` signs, so it needs the bundle's private audit key, unsealed with `BETTER_COMPUTE_SECRET`. Anyone you hand a proof file and the pin to can run `check-proof` without access to your deployment. See [audit](/docs/guides/events/audit). ## Operator commands [#operator-commands] The rest of the CLI runs and maintains a deployment. Briefly: | Command | Does | | -------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `init` | writes a starter `better-compute.config.ts` | | `migrate` | creates or upgrades the schema and every module's migrations; run it on every deploy | | `doctor [--strict]` | checks the database, schema, clock, method registry, IAM wiring, the PKI bundle and the host export | | `serve` | runs the API and the worker roles (`--roles api`, `controllers,sinks`, `gateway`, …) | | `dev [--sim N]` | a local server with PGlite and, with `--sim`, N simulated machines and an embedded Better IAM | | `sim` | the deterministic fleet simulator and its invariants | | `pki init / show / issue-gateway / renew-ca / rotate-command-root / add-sandbox-ca / reseal` | the machine PKI bundle, its pins, gateway certificates, CA renewal and rotation, the sandbox SSH CA, re-sealing in the current format; see [PKI](/docs/operations/pki) | | `daemon install-script` | the POSIX installer that joins a server to a pool; `--plugins` prepares a node for bcx driver plugins (see [installing bcd](/docs/daemon/install#the-plugin-drop-in)) | | `iam setup [--meters]` | prints the example policies; creates the billing meters in Better IAM | | `release keygen / sign / cosign / verify` | signed bcd release manifests | | `batch keygen / sign / sign-approvers` | an approver's offline tools: a P-256 approver key, an approval signed from a batch's scope file, and a signature over a new approver set; see [destructive batches](#destructive-batches) | See [installation](/docs/guides/installation), [local development](/docs/operations/local-development) and the [CLI reference](/docs/reference/cli). ## Exit codes and common errors [#exit-codes-and-common-errors] The binary exits 0 on success and 2 for a usage mistake (`INVALID_ARGUMENT`, `INVALID_COMMAND`). Other failures exit non-zero with the error's code: | Code | Meaning | | ------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `MISSING_ENV` | a token command with no credential: no `BETTER_COMPUTE_TOKEN` and no saved session or profile | | `CONFIG_NOT_FOUND` | no configuration found, and no `--url` or `BETTER_COMPUTE_DATABASE_URL` | | `UNAUTHENTICATED`, `ACCESS_DENIED`, `CONFIRMATION_REQUIRED`, … | the API's own codes, from the server; see [errors](/docs/reference/errors). `TENANT_SUSPENDED` also prints a hint: the tenant is suspended or deleted in IAM, and an administrator of a parent tenant can reactivate it | | `FLEET_DRIFTED`, `PLAN_BLOCKED`, `CONFIG_DRIFT` | fleet plan, status and apply outcomes | | `AUDIT_TAMPERED`, `INVALID_PROOF`, `AUDIT_KEY_MISSING`, `PKI_NOT_FOUND` | audit verification outcomes | | `DOCTOR_FINDINGS` | `doctor --strict` found an error or warning | | `ALREADY_EXISTS` | an `--out` or `--secret-out` file exists; a token, secret or export is never written over a file | | `SCOPE_MISMATCH`, `SCOPE_CHANGED`, `SCOPE_UNEXPECTED`, `UNBOUND_SCOPE`, `INVALID_RESPONSE` | `batch show`, `approve` and `sign` refused a batch; see [destructive batches](#destructive-batches) | | `REMOTE_EXIT` | `console`, `exec`, `sandbox exec` or `ssh-cert --connect`: the binary exits with the remote program's own status (1 to 255), or 255 when the tunnel broke (or ssh was ended by a signal) | ## Project commands [#project-commands] A configuration can export its own commands, which get the same flag parsing, help, configuration loading and output formatting as the built-in ones: ```ts title="better-compute.config.ts" import { defineCommand } from 'better-compute/cli'; export const commands = [ defineCommand({ name: 'seed', summary: 'Create the demo sites and pools', target: 'config', flags: { region: { type: 'string', description: 'Region label', default: 'eu' } }, async run({ compute, flags }) { const instance = await compute(); // … return { created: true, region: flags.region }; }, }), ]; ``` A `token` project command gets `context.api()`, the same transport the built-in commands use, so it is decided as the resolved credential like any other call. ## Inside another CLI [#inside-another-cli] PLAN.md section 9.3 asks for one binary for IAM and compute. `computeCommands(define, options?)` from `better-compute/cli` turns every built-in command into a spec of another framework's `defineCommand`; with Better IAM's: ```ts title="bin.ts" import { createCli, defineCommand } from '@better-iam/cli'; import { computeCommands } from 'better-compute/cli'; const cli = createCli({ commands: computeCommands(defineCommand, { exclude: ['dev', 'serve'] }) }); await cli.run(process.argv.slice(2)); // better-iam compute-reserve …, compute-api …, compute-support-bundle … ``` Names are `compute-` by default (`prefix`, and `separator: ' '` for a framework that matches several words), in a `Compute` help group. Each spec declares compute's own flags plus `--config` and `--url`, and the host adds its output flags; running it runs compute's program with the flags given, so compute reads its own `BETTER_COMPUTE_*` environment, loads its deployment and prints its own output, errors included. `exclude` leaves commands out, such as `dev` and `serve` for a client-only binary. ## Run commands from code [#run-commands-from-code] `runCli(argv, io?)` runs one command line and rejects with the failure's `ComputeError`, `CliError` or `ComputeClientError`; `createCli(options?)` returns a program whose `run(argv, io?)` does the same with your own commands added. `main(argv?, cli?, io?)` behaves like the binary: it prints `CODE: message` on failure and resolves with the exit status, 0, 2 for a usage mistake, 1 otherwise, and the remote program's own status when a `console`, `exec`, `sandbox exec` or `ssh-cert --connect` ends with one (a `RemoteExitError`, whose `exitStatus` is 1 to 255, or 255 when the tunnel broke). Everything a command touches comes through a `CliIO`, so programs and tests can run commands without a real terminal: | Field | Used for | Default (`processIO()`) | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | | `out(message)`, `err?(message)` | results on stdout; progress and failures on stderr | `console.log`, `console.error` | | `env` | the environment commands read; `process.env` is never read implicitly | `process.env` | | `stdin?()`, `cwd?`, `fetch?` | `--data -`, relative paths and configuration discovery, remote calls | the process's | | `signal?` | an `AbortSignal` that ends long-running commands (`serve`, `dev`, `forward`) and detaches `console` and `exec` | SIGINT or SIGTERM | | `terminal?` | the interactive terminal `console` and `exec` attach to: `onInput`, `write(data, channel)`, and optional `size`, `onResize` and `raw` | the process's stdin and stdout, raw mode on a TTY | | `spawn?(command, args)` | runs a program with inherited stdio and resolves with its exit status; `sandbox exec` and `ssh-cert --connect` run OpenSSH through it | `child_process.spawn`; a program ended by a signal reports 255 | When you pass your own `CliIO` without `terminal`, `console` and `exec` fail with `UNSUPPORTED` before a ticket is spent; without `spawn`, `sandbox exec` and `ssh-cert --connect` fail with `UNSUPPORTED` after the grant or the certificate, since they cannot run ssh. ```ts title="run.ts" import { RemoteExitError, runCli } from 'better-compute/cli'; try { await runCli(['sandbox', 'exec', sandboxId, '--', 'make', 'test'], { out: (line) => log.push(line), env: { ...process.env, BETTER_COMPUTE_URL: 'https://compute.example.com' }, spawn: (command, args) => runWithInheritedStdio(command, args), // resolves with the exit status }); } catch (error) { if (error instanceof RemoteExitError) process.exitCode = error.exitStatus; // the remote command failed else throw error; } ```