Commands

A command is a one-shot action on one node: run a diagnostic, re-send inventory, collect logs, reboot, wipe a disk. It is the other half of what a node receives, next to its desired state, and the rule for choosing between them is simple:

• if losing the message would leave the node in the wrong steady state, it is desired state (assignments, instances, host configuration, access, the daemon version), which the node converges to and re-reads after any gap;
• if running it late could do harm, it is a command, with a deadline (not_after) and, when it presupposes a spec, a minimum generation.

Commands are durable rows in Postgres. They are queued in the caller's transaction, delivered in order over the node's session, deduplicated by the node's journal, and settled by a result the node reports. See desired state and commands for the design.

Running a command

commands.run (compute:nodes:exec on compute-node/{nodeId}) queues a command from the API. It accepts an allowlist of kinds; any other kind is UNSUPPORTED.

bcd runs all four. Its diag.nvml works from the collected GPU inventory rather than the NVML library (cb987e5). Since 662fb55, nvml.init passes only when the driver's management interface (nvidia-smi) answered, not when the kernel driver alone names the GPUs; then it checks that the GPUs are visible (and the expected count), and per GPU ecc.uncorrectable, row-remap.pending, row-remap.failure, pcie.link (a link narrower than its maximum width; speed is left out, since idle GPUs lower it on purpose) and xid.recent (XID conditions this boot). Every failing check says why. Its node.reboot goes through systemctl reboot. It is how a node with the condition RebootRequired gets a host setting that waits for a boot (1 GiB huge pages, isolated CPUs, the IOMMU, an ECC mode, a new driver): bcd never reboots on its own, so reboot such nodes within the pool's disruption budget (see RebootRequired).

bcd also runs gpu.reset, which commands.run does not accept: the provision module's remediation sends it for RowRemapPending once no allocation is live. bcd refuses the whole command gpu-busy while any listed GPU is in use (539c4ce): a live hold names it or one of its MIG instances or slots (<uuid> has a live assignment), or a VM has it passed through (<uuid> is passed through to <instance>). It resets the others with nvidia-smi --gpu-reset, reports per GPU whether the reset worked and whether a fresh inventory shows it healthy, and clears the XID conditions of a GPU the reset left healthy (662fb55). diag.dcgm applies the same refusal from level 2, where DCGM loads the GPUs, to the listed GPUs or, with none listed, to every GPU; level 1 only checks the deployment. See delivery.

const command = await client.commands.run({
  tenantId,
  nodeId,
  kind: 'inventory.refresh',
  payload: { sections: ['gpus'], rescan: true },
  idempotencyKey: 'refresh-after-gpu-swap-0412',
  notAfterSeconds: 300,
});
// { id: 'cmd_…', seq: 42, state: 'pending', notAfter: …, retrySafe: true, … }

better-compute api commands.run tenantId=$TENANT nodeId=node_… kind=node.reboot idempotencyKey=reboot-0412

• idempotencyKey is scoped to your tenant and identity: a retry returns the command already queued, and the same key for another kind is CONFLICT.
• notAfterSeconds (10 to 3,600, default 600) is how long the command may wait for the node before it expires.
• The node must be visible to your tenant (NOT_FOUND otherwise), and a new command publishes node.command.queued.
• batchId (optional, batch_…) puts the command into a destructive batch (46afe75); see below.

Destructive commands under the quorum

With the v1.0 destructive-batch quorum on (approver keys in pki.approvers) and no break-glass allowance (unbatchedDestructivePerHour 0), every daemon refuses a destructive kind that is not part of an approved batch, so the queue refuses it up front: node.reboot without a batchId is INVALID_STATE with details: { reason: 'batch-required', kind } (508728e). Create the batch first, have enough approvers sign it, and run the command in it:

const batch = await client.commandBatches.create({
  tenantId,
  reason: 'Reboot node_0412 after the kernel update',
  scope: [{ subject: nodeId, kind: 'node.reboot' }], // payloadSha256 (hex) would bind one exact payload
  idempotencyKey: 'reboot-0412-batch',
});
await client.commands.run({ tenantId, nodeId, kind: 'node.reboot', idempotencyKey: 'reboot-0412', batchId: batch.batchId });
// pending until the batch is approved: better-compute batch approve --tenant … --batch batch_… --key approver.pem

commandBatches.create needs compute:batches:create on the tenant and is idempotent per caller and key. The command's (subject, kind) must be in the batch's scope (for a node's own kinds the subject is the node id), or commands.run is INVALID_INPUT. The command stays pending, held, until the batch has its approvals; then it is sent with the approval and its inclusion proof, and the node checks both itself. commands.run, get and list say so (a299e96): a batched command carries its batchId, and while it waits for approvals awaitingApproval: true and no notBefore; once the batch is approved, awaitingApproval is gone and notBefore is set. Without approver keys a batch is recorded for audit only and its commands go out at once.

The platform's own destructive work creates its batches itself: metal's workflows, metal.bulk and metal.power (f66e236, 478025a), and the health reconciler's reboots and GPU resets, each under a batch whose leaves name the node (e594a4f). An agent on the MCP server can propose a reboot batch with propose_command_batch and pass its batchId to exec_command, but has no tool to approve one: people approve with their approver keys (171ad2f). See who creates batches.

Modules queue any registered kind through the command queue, ctx.commands.enqueue(tx, command), in their own transaction. See modules.

Support bundles

logs.collect uploads only when the server is given supportBundles: { blob, maxBytes? }, a BlobStore that can presign (see configuration). commands.run then puts a presigned PUT URL for support-bundles/{nodeId}/{key}.bundle into the command (key is the queue's api:… idempotency key derived from yours, without the api: prefix; supportBundleKey(nodeId, key) computes it), valid until the command's not_after plus 15 minutes, with a byte cap (maxBytes, default 256 MiB; the daemon truncates oldest first) and supportBundle: true. No credential travels in the command, and a retry with the same idempotency key presigns nothing new.

commands.run, commands.get and commands.list show the key as the command's bundleKey. There is no API to download a bundle yet; operators read it from the blob store. Without supportBundles, logs.collect has no upload destination.

Kinds and lanes

Every command kind is registered with its wire flags, a lane, a timeout and the agent roles that may run it. The full list, with payload types, is in command kinds; the built-ins are diag.nvml, inventory.refresh, logs.collect, node.reboot and gpu.reset, and modules add the rest (the metal module the BMC, boot, disk, deploy, site.httpProxy and diag.dcgm kinds, the rollout module daemon.upgrade).

Lanes bound the work in flight per node. The gateway keeps at most 8 control, 4 provision and 2 bulk commands outstanding per session; the rest stay pending in Postgres until credits return. The daemon runs each lane on its own fixed worker pool.

States

Reports may skip steps: a lost acknowledgement does not stop a result from landing. succeeded, failed, rejected, expired and cancelled are terminal.

Before accepting a command, the daemon checks, in this order:

1. UNSUPPORTED: the body names another node, or the kind is unknown or not allowed for its role;
2. STALE_GENERATION: min_generation is ahead of the spec it applied; it asks for the spec, waits until not_after, then rejects;
3. EXPIRED: past not_after, in server time (the daemon keeps the offset from Welcome.server_time);
4. BAD_SIGNATURE: once the node's trust bundle holds command-signing roots, a command of any kind without a valid signature (0d42de8 in bcd, 12ccfa1 in the FakeDaemon; before, only destructive kinds needed one);
5. BATCH_UNAPPROVED or UNBATCHED_LIMIT: with approver keys, a destructive command outside an approved batch (see below);
6. BUSY: no credits; the gateway puts the command back to pending with a backoff from 1 to 30 seconds.

The refusals other than BUSY make the command rejected, and the daemon journals them, so a duplicate delivery gets the same answer. See acceptance checks.

Timing

The CommandSweeper, a singleton that runs every 5 seconds, enforces the deadlines on the control plane side:

• pending past not_after becomes expired (reason not_after);
• dispatched, acked or running past not_after plus the kind's timeout becomes failed (reason timeout). For self-terminating kinds the timeout covers the rejoin window. A kind that is no longer registered gets 15 minutes.

Each transition wakes the node's controller and the owner named by the idempotency key (wf:<workflowId>:… wakes that workflow), and publishes command.settled.

Delivery, idempotency and resume

• Identity and order. A command gets a cmd_… id and a per-node seq from nodes.next_cmd_seq in the enqueue transaction. The idempotency key is unique per node: workflow steps use wf:<id>:<stepId>, acceptance uses acceptance:<nodeId>:<attempt>:<kind>, and the API derives api:… from your key. A second enqueue with the same key returns the first command.
• Resume. When a node reconnects, Hello first settles what the reconnect proves, then the gateway replays every non-terminal command in seq order. The daemon's journal answers a duplicate with the stored result, or re-attaches to a running one.
• Crashes. A command the daemon accepted but never started runs again only if it is retry-safe; otherwise it fails with interrupted and its owner decides.
• Lost journal. A daemon whose journal is gone (a ramdisk reboot, a reinstall, corruption) comes back with a new journal id. In-flight commands that are not retry-safe fail with journal-lost instead of being replayed.
• Self-terminating kinds. A new boot or role on rejoin completes them as succeeded with rebooted: true.

The details, including epoch fencing, are in delivery.

Restricted identities

A node whose current certificate carries allowed kinds (a discovery ramdisk, for example) only accepts those. The command queue refuses other kinds at enqueue with INVALID_STATE ("runs a restricted identity that does not accept …"), and the gateway checks the session's certificate again at dispatch. See attestation.

Signatures and attested results

Two signatures protect commands from a compromised gateway, which can delay, drop or read frames but should never forge intent or results:

• Attested results. For attested kinds the node signs its result with its node key: CommandUpdate.node_sig, under purpose command-update, over the command id, the state and the SHA-256 of the result. The gateway stores it exactly as received, and consumers verify it against node_certs before any state transition (the metal module's nodeSignedResults() does this for wipes and diagnostics). A forged wipe result is refused.
• Signed commands (v0.2). Command.signature carries a signature by a worker key that chains to the command-signing root in the node's trust bundle. @better-compute/pki provides the pieces (createCommandSigningAuthority, 24-hour worker keys from issueWorkerKey, signCommand and verifyCommand). Once the daemon's trust bundle holds command-signing roots, it rejects every command without a valid signature with BAD_SIGNATURE, whatever its kind: exempting "harmless" kinds let a compromised gateway inject an unsigned site.httpProxy that could reach a BMC (0d42de8, 12ccfa1). When the server is given pki.commandSigning (the root from createCommandSigningAuthority, or the PKI bundle's commandSigning), every command is signed by a 24-hour worker key that root certifies, and JoinResponse and Welcome carry the trust bundle (the roots, plus pki.commandSigningRoots during a rotation, at a version that follows the bundle's content, never below pki.trustVersion). Without it, commands go unsigned as in v0.1. The same key signs each NodeSpec generation (purpose desired-state, v0.4); the gateway relays the stored signed bytes, and the daemon verifies them before applying a spec. See delivery.

With pki.commandSigning, GPU bindings are attested too: a reservation goes active only once the node's own signed AllocationBound verifies; see attested bindings.

Commands never carry secret values: they carry secretRefs, which the agent exchanges for a value sealed to a one-time key. See security.

Cancelling

commands.cancel (compute:nodes:exec on compute-node/{nodeId}, v0.2) takes { tenantId, nodeId, commandId } and returns the command. Modules cancel through the same command queue, ctx.commands.cancel(tx, commandId):

• a pending command becomes cancelled at once and never runs;
• an in-flight command keeps its state and is marked cancel_requested_at; the gateway sends Cancel until the daemon reports a terminal state. The daemon stops between actions and reports the real outcome, so a side effect that already happened is never hidden;
• a terminal command is returned unchanged; an unknown id (or a node your tenant cannot see) is NOT_FOUND.

commands.cancel publishes node.command.cancel-requested (nodeId, commandId, kind, state) for a command that was not yet terminal; state is cancelled for a pending one and the in-flight state otherwise.

const command = await client.commands.cancel({ tenantId, nodeId, commandId: 'cmd_…' });

Deleting a node cancels its pending commands with reason node-deleted.

Results

commands.get and commands.list (compute:nodes:read) show a node's commands; list is newest first by seq, filters by state, and returns up to 200 per page (100 by default).

Rather than polling, follow command.settled events (commandId, nodeId, kind, state, reason), which the gateway publishes when a node reports a terminal state and the sweeper publishes when it expires or fails a command. See watching events.

const { items } = await client.commands.list({ tenantId, nodeId, state: 'failed', limit: 20 });
for (const command of items) console.log(command.seq, command.kind, command.reason);