Operating nodes

Sooner or later every node needs attention: a firmware update, a flaky GPU, a rack move, a decommission. The nodes API gives you the controls, and each one moves exactly one of the node's independent axes, so you always know what a control did and what it left alone.

Every nodes method except list is decided on compute-node/{nodeId}, and a tenant sees the nodes the authorization projection holds for it; a node it cannot see is NOT_FOUND. Mutations accept an optional version: pass the version you read, and a node that changed since fails with CONFLICT instead of being overwritten. Most also take a free-text reason (up to 256 characters) that ends up in the event.

Cordon and uncordon

A cordon stops new work from landing on a node and changes nothing else: existing allocations keep running, commands still flow, the node stays connected.

await client.nodes.cordon({ tenantId, nodeId, reason: 'BIOS update window' });
// … later
await client.nodes.uncordon({ tenantId, nodeId });

better-compute nodes cordon node_01k5w… --tenant $TENANT --reason 'BIOS update window'
better-compute nodes uncordon node_01k5w… --tenant $TENANT

The nodes commands (44915e5) take the node id as their argument and --if-version N for the version check; a node id that does not look like node_… is a usage error before anything is sent.

nodes.cordon (compute:nodes:cordon) sets cordoned and publishes node.cordoned; cordoning a cordoned node returns it unchanged. nodes.uncordon (same action) clears cordoned, ends a drain in progress, and publishes node.uncordoned.

Drain

A drain is a cordon that also waits for the node to empty, so you know when it is safe to work on it. nodes.drain (compute:nodes:drain) cordons the node, adds the draining NoSchedule reason, publishes node.draining, and hands the node to the Drain controller.

The Drain controller does not evict anything in this release. It waits until the node holds no live allocation (every allocation released, or past the end of its interval), looking again at the earliest allocation end and at least every 5 minutes. Then it removes draining and publishes node.drained. The node stays cordoned until you call nodes.uncordon. Forced release within the pool's disruption budget is planned for v0.4.

await client.nodes.drain({ tenantId, nodeId, reason: 'replace PSU' });
// Wait for the node.drained event (see the events guide), or poll:
const node = await client.nodes.get({ tenantId, nodeId });
const drained = node.cordoned && !node.noScheduleReasons.includes('draining');

better-compute nodes drain node_01k5w… --tenant $TENANT --reason 'replace PSU'
better-compute nodes events node_01k5w… --tenant $TENANT --type node.drained

Draining a node that is already draining changes nothing. To shorten a drain, release the reservations on it; see reservations.

Taints

A taint is a named NoSchedule reason you own. Use it when you want a node out of placement for a reason that should stay visible, next to the conditions and liveness reasons the controllers write.

await client.nodes.taint({ tenantId, nodeId, add: ['rma-pending'], remove: ['burn-in'] });

better-compute nodes taint node_01k5w… --tenant $TENANT --add rma-pending --remove burn-in

nodes.taint (compute:nodes:update) adds and removes up to 32 keys each, stored in noScheduleReasons as taint:<key>. A key is 1 to 128 letters, digits and . _ / = : -, starting and ending with a letter or digit, and cannot be added and removed in one call. The method only ever moves operator taints: conditions, unreachable and draining belong to their controllers. It publishes node.tainted. Any entry in noScheduleReasons keeps a node out of placement, with one exception.

Dedicating a node to a tenant. The taint dedicated=<tenantId> (stored as taint:dedicated=acme) keeps every other tenant off the node but not that tenant: placement for acme tolerates a dedication to acme and nothing else, on every placement path (inline, shortlist, pool-lease index, bulk commits and BookingRepair) and for whole-node leases (binding: 'node' reservations in a tenantRoot pool, see dedicated-root metal), which ignore every other NoSchedule reason: a node dedicated to acme is never leased by another tenant. A dedication together with any other reason, or dedications to two tenants, blocks everyone. Capacity rollups still count a dedicated node as NoSchedule (bucket unhealthy). A node that becomes schedulable for its one dedicated tenant still fires the wake edge, so that tenant's pending reservations are placed at once instead of on their 30-second retry.

await client.nodes.taint({ tenantId, nodeId, add: ['dedicated=acme'] });

Labels

Labels describe a node for placement: selectors in reservations match device attributes and node labels (see placement). Inventory writes its own labels (gpu.model, cpu.arch, …); nodes.label (compute:nodes:update) manages yours.

const node = await client.nodes.label({
  tenantId,
  nodeId,
  set: { rack: 'b7', 'fabric/rail': '3' },
  remove: ['burn-in'],
});

set takes up to 64 labels and remove up to 64 keys. A key is up to 128 characters of letters, digits and . _ / -, starting and ending with a letter or digit; a value is up to 256 characters of letters, digits and . _ : / @ + = -. Setting a key that inventory or the system owns is INVALID_INPUT, and remove only removes operator labels. Since c1fd430 any bc.dev/ key is refused in set and remove alike: those labels are bcd's advertised drivers and capabilities, and placement, metal, provision and vps count one only with source system (see capability labels). The call returns the node with its labels and publishes node.labeled.

better-compute nodes label node_01k5w… --tenant $TENANT --set rack=b7,fabric/rail=3 --remove burn-in
better-compute nodes list --tenant $TENANT --cordoned

--set takes KEY=VALUE pairs and --remove keys, both comma-separated; one of them is required. The console's node page has the same controls: it lists the node's labels (key, value, source) and taints, sets and removes operator labels, and adds and removes taints, gated on the node's state and IAM's answer on compute:nodes:update (3ff9fb3; see the console).

Maintenance

maintenance is a separate axis from cordoned. It marks a node that needs an operator, and a node in maintenance is reported in the maintenance capacity bucket and never scheduled. In this release it is turned on by the system, not by an API call:

• a bare-metal workflow that fails or is aborted sets provisionState: 'failed' and turns maintenance on (see bare-metal workflows);
• health remediation that fails, is cancelled, or does not clear the fault in two attempts turns maintenance on and publishes node.remediation.escalated (see inventory and health).

Liveness and connectivity

A node's connectivity says whether it has a live session, not whether it is healthy or whether its workloads run. The daemon sends a heartbeat every 10 seconds, the gateway keeps the last-seen time in memory and flushes ages to Postgres every 15 seconds, and the node's liveness lease lasts 40 seconds.

Grace keeps an infrastructure problem from looking like a fleet of dead machines: 120 seconds after a gateway loss, so sessions can move to another gateway, and 5 minutes after the Liveness controller or the database was down for more than 20 seconds, so ages that could not be flushed meanwhile do not count against nodes.

"Last seen" is the later of the session's flushed last_seen and disconnected_at, else its connected_at. A node that has joined but whose first Hello is still queued (a whole rack powering on at once) has none of these yet, so both the Liveness sweep and the per-node check fall back to its newest node_certs.issued_at: it gets the same 40 seconds from its join instead of being marked unreachable the moment it joins.

Disconnected is not dead

None of these transitions releases anything. Allocations, bookings and queued commands stay exactly as they were: an unreachable node is only kept out of new placement. A partitioned server usually keeps running its workloads, and releasing its GPUs to someone else would double-book hardware that is still in use. lost is applied automatically only to nodes of provider and vps-host pools that set policy.lostAfter, never to bare metal (a vps-host pool opts in so that its hosted instances can be recreated elsewhere). When the node reconnects, it resumes where it left off: the gateway replays its unfinished commands and the desired state it missed (see delivery).

Revoking a node's identity

When you suspect a node's key is compromised, or you are handing the machine to someone else, revoke its identity. nodes.revokeIdentity (compute:nodes:revoke-identity):

• revokes every live certificate of the node (with your reason, or identity revoked);
• raises identityGeneration, so no certificate of the old generation can renew or re-join with x509pop;
• publishes security.node.identity-revoked and returns { node, revokedCertificates }.

A gateway that runs a revocation feed (createRevocationFeed from @better-compute/pki, which polls node_certs every second by default) learns about the revocation from Postgres and closes the node's live session at once; without a feed, a revoked certificate keeps its session until it expires. The machine can come back only by joining again with fresh evidence, such as a new join token.

The gateway revokes a certificate on its own when it looks cloned: after more than 3 alternations between two machines within 10 minutes it quarantines the certificate with reason duplicate-identity and publishes security.node.quarantined. Watch security.node.duplicate-session to see it coming, then find the copy and re-join the real machine. See duplicate identity.

Adopting again

nodes.adopt (compute:nodes:provision) re-runs acceptance on a node in adopting or adopted; any other state is INVALID_STATE. An adopted node goes back to adopting and leaves placement until the checks pass. See acceptance checks.

Deleting a node

nodes.delete (v0.2, compute:nodes:delete) removes a node that is done for good or never got going. It accepts only:

• a retired node;
• an adopted node that is cordoned (54cad9e);
• a node still in adopting or enrolling, or one that failed during enrollment;
• and only when the node has no open allocation (INVALID_STATE with the count otherwise).

Any other state is INVALID_STATE with { provisionState, cordoned } in its details. An adopted node that is not cordoned gets "Cordon the adopted node first (nodes.cordon), then delete it once it holds nothing".

In one transaction it makes the node's devices unbookable, removes its scalar resources and rebuilds the pool's capacity rollup, revokes its certificates (node-deleted), frees its attestor bindings and trusted hardware key, cancels its pending commands, and removes it from every tenant's projection. The row itself stays as a tombstone (deletedAt), because audit rows, certificates, inventory snapshots and past allocations still reference it. The machine can then join again as a new node. A retry of the same delete returns { nodeId, deleted: false }, and node.deleted is published once.

For bare-metal nodes, retire through the metal workflow first; see bare-metal workflows.

Removing an adopted host

An adopted host (bcd installed on a machine metal does not manage) has no retire workflow, so the retire path for it is a cordon and then the delete. Use it when the machine is decommissioned, or when it must join again under a new identity, for example after its TPM or disk was replaced (see recovering a node whose TPM changed):

1. Cordon the node, so nothing new lands on it. Drain it instead when you want to wait for its allocations to end.
2. Wait until it holds no open allocation: release its reservations, or let them end.
3. Delete it. Its certificates are revoked, and its attestor bindings (a tpm join's EK binding included) and its trusted hardware key are freed.
4. On the machine, as root, join again with --reset --yes and the join's credential (a new token, or --attestor tpm). The delete revoked the old certificate but left the files on the machine, and an identity that still loads would only answer already joined; --reset moves it aside first (see joining again with --reset). The machine joins as a new node.

better-compute nodes cordon node_01k5w… --tenant $TENANT --reason 'TPM replaced'
# ... once the node holds no open allocation:
better-compute nodes delete node_01k5w… --tenant $TENANT --reason 'TPM replaced' --yes
# then on the machine, as root:
bcd join --config /etc/bcd/config.toml --reset --yes --attestor tpm

nodes delete (4049621) takes --reason and --if-version like the other node changes, and without --yes it refuses with a usage error before making any call (nodes delete removes <node> for good; re-run with --yes). From a client:

await client.nodes.cordon({ tenantId, nodeId, reason: 'TPM replaced' });
// ... once the node holds no open allocation:
await client.nodes.delete({ tenantId, nodeId, reason: 'TPM replaced' });

The nodes API

To run diagnostics, collect logs or reboot a node, use commands.