# Operating nodes (/docs/guides/nodes/operating-nodes) > Cordon, drain, taint and label nodes, understand maintenance, liveness and connectivity, revoke a node's identity, and delete nodes, with the nodes API. Sooner or later every node needs attention: a firmware update, a flaky GPU, a rack move, a decommission. The `nodes` API gives you the controls, and each one moves exactly one of the node's [independent axes](/docs/guides/nodes), so you always know what a control did and what it left alone. Every `nodes` method except `list` is decided on `compute-node/{nodeId}`, and a tenant sees the nodes the authorization projection holds for it; a node it cannot see is `NOT_FOUND`. Mutations accept an optional `version`: pass the version you read, and a node that changed since fails with `CONFLICT` instead of being overwritten. Most also take a free-text `reason` (up to 256 characters) that ends up in the event. ## Cordon and uncordon [#cordon-and-uncordon] A cordon stops new work from landing on a node and changes nothing else: existing allocations keep running, commands still flow, the node stays connected. ```ts await client.nodes.cordon({ tenantId, nodeId, reason: 'BIOS update window' }); // … later await client.nodes.uncordon({ tenantId, nodeId }); ``` ```bash better-compute nodes cordon node_01k5w… --tenant $TENANT --reason 'BIOS update window' better-compute nodes uncordon node_01k5w… --tenant $TENANT ``` The `nodes` commands (44915e5) take the node id as their argument and `--if-version N` for the `version` check; a node id that does not look like `node_…` is a usage error before anything is sent. [`nodes.cordon`](/docs/reference/api/nodes#cordon) (`compute:nodes:cordon`) sets `cordoned` and publishes `node.cordoned`; cordoning a cordoned node returns it unchanged. [`nodes.uncordon`](/docs/reference/api/nodes#uncordon) (same action) clears `cordoned`, ends a drain in progress, and publishes `node.uncordoned`. ## Drain [#drain] A drain is a cordon that also waits for the node to empty, so you know when it is safe to work on it. [`nodes.drain`](/docs/reference/api/nodes#drain) (`compute:nodes:drain`) cordons the node, adds the `draining` NoSchedule reason, publishes `node.draining`, and hands the node to the Drain controller. The Drain controller does not evict anything in this release. It waits until the node holds no live allocation (every allocation released, or past the end of its interval), looking again at the earliest allocation end and at least every 5 minutes. Then it removes `draining` and publishes `node.drained`. The node stays cordoned until you call `nodes.uncordon`. Forced release within the pool's disruption budget is planned for v0.4. ```ts await client.nodes.drain({ tenantId, nodeId, reason: 'replace PSU' }); // Wait for the node.drained event (see the events guide), or poll: const node = await client.nodes.get({ tenantId, nodeId }); const drained = node.cordoned && !node.noScheduleReasons.includes('draining'); ``` ```bash better-compute nodes drain node_01k5w… --tenant $TENANT --reason 'replace PSU' better-compute nodes events node_01k5w… --tenant $TENANT --type node.drained ``` Draining a node that is already draining changes nothing. To shorten a drain, release the reservations on it; see [reservations](/docs/guides/capacity/reservations). ## Taints [#taints] A taint is a named NoSchedule reason you own. Use it when you want a node out of placement for a reason that should stay visible, next to the conditions and liveness reasons the controllers write. ```ts await client.nodes.taint({ tenantId, nodeId, add: ['rma-pending'], remove: ['burn-in'] }); ``` ```bash better-compute nodes taint node_01k5w… --tenant $TENANT --add rma-pending --remove burn-in ``` [`nodes.taint`](/docs/reference/api/nodes#taint) (`compute:nodes:update`) adds and removes up to 32 keys each, stored in `noScheduleReasons` as `taint:`. A key is 1 to 128 letters, digits and `. _ / = : -`, starting and ending with a letter or digit, and cannot be added and removed in one call. The method only ever moves operator taints: conditions, `unreachable` and `draining` belong to their controllers. It publishes `node.tainted`. Any entry in `noScheduleReasons` keeps a node out of placement, with one exception. **Dedicating a node to a tenant.** The taint `dedicated=` (stored as `taint:dedicated=acme`) keeps every other tenant off the node but not that tenant: placement for `acme` tolerates a dedication to `acme` and nothing else, on every placement path (inline, shortlist, pool-lease index, bulk commits and BookingRepair) and for whole-node leases (`binding: 'node'` reservations in a `tenantRoot` pool, see [dedicated-root metal](/docs/guides/bare-metal/dedicated-root)), which ignore every other NoSchedule reason: a node dedicated to `acme` is never leased by another tenant. A dedication together with any other reason, or dedications to two tenants, blocks everyone. Capacity rollups still count a dedicated node as NoSchedule (bucket `unhealthy`). A node that becomes schedulable for its one dedicated tenant still fires the wake edge, so that tenant's pending reservations are placed at once instead of on their 30-second retry. ```ts await client.nodes.taint({ tenantId, nodeId, add: ['dedicated=acme'] }); ``` ## Labels [#labels] Labels describe a node for placement: selectors in reservations match device attributes and node labels (see [placement](/docs/guides/capacity/placement)). Inventory writes its own labels (`gpu.model`, `cpu.arch`, …); [`nodes.label`](/docs/reference/api/nodes#label) (`compute:nodes:update`) manages yours. ```ts const node = await client.nodes.label({ tenantId, nodeId, set: { rack: 'b7', 'fabric/rail': '3' }, remove: ['burn-in'], }); ``` `set` takes up to 64 labels and `remove` up to 64 keys. A key is up to 128 characters of letters, digits and `. _ / -`, starting and ending with a letter or digit; a value is up to 256 characters of letters, digits and `. _ : / @ + = -`. Setting a key that inventory or the system owns is `INVALID_INPUT`, and `remove` only removes operator labels. Since c1fd430 any `bc.dev/` key is refused in `set` and `remove` alike: those labels are bcd's advertised drivers and capabilities, and placement, metal, provision and vps count one only with source `system` (see [capability labels](/docs/operations/security#sandboxes-v05)). The call returns the node with its labels and publishes `node.labeled`. ```bash better-compute nodes label node_01k5w… --tenant $TENANT --set rack=b7,fabric/rail=3 --remove burn-in better-compute nodes list --tenant $TENANT --cordoned ``` `--set` takes `KEY=VALUE` pairs and `--remove` keys, both comma-separated; one of them is required. The console's node page has the same controls: it lists the node's labels (key, value, source) and taints, sets and removes operator labels, and adds and removes taints, gated on the node's state and IAM's answer on `compute:nodes:update` (3ff9fb3; see [the console](/docs/guides/clients/console)). ## Maintenance [#maintenance] `maintenance` is a separate axis from `cordoned`. It marks a node that needs an operator, and a node in maintenance is reported in the `maintenance` capacity bucket and never scheduled. In this release it is turned on by the system, not by an API call: * a bare-metal workflow that fails or is aborted sets `provisionState: 'failed'` and turns maintenance on (see [bare-metal workflows](/docs/guides/bare-metal/workflows)); * health remediation that fails, is cancelled, or does not clear the fault in two attempts turns maintenance on and publishes `node.remediation.escalated` (see [inventory and health](/docs/guides/nodes/inventory-and-health)). ## Liveness and connectivity [#liveness-and-connectivity] A node's `connectivity` says whether it has a live session, not whether it is healthy or whether its workloads run. The daemon sends a heartbeat every 10 seconds, the gateway keeps the last-seen time in memory and flushes ages to Postgres every 15 seconds, and the node's liveness lease lasts 40 seconds. | Transition | When | Effect | | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | to `connected` | the node's `Hello` | `node.connected`; clears the `unreachable` reason and the `NodeUnreachable` condition | | to `disconnected` | the session closes, or its gateway's lease expires | `node.disconnected` with reason `gateway-lost` or `no-session` | | to `unreachable` | 40 seconds after the node was last seen, plus grace; a node that joined but never opened a session counts from its join (its newest certificate's `issued_at`) | adds the `unreachable` NoSchedule reason, sets `NodeUnreachable` (`HeartbeatTimeout` or `GatewayLost`), publishes `node.unreachable` | | to `lost` | the pool's `policy.lostAfter` after it became unreachable | `node.lost` | Grace keeps an infrastructure problem from looking like a fleet of dead machines: 120 seconds after a gateway loss, so sessions can move to another gateway, and 5 minutes after the Liveness controller or the database was down for more than 20 seconds, so ages that could not be flushed meanwhile do not count against nodes. "Last seen" is the later of the session's flushed `last_seen` and `disconnected_at`, else its `connected_at`. A node that has joined but whose first `Hello` is still queued (a whole rack powering on at once) has none of these yet, so both the Liveness sweep and the per-node check fall back to its newest `node_certs.issued_at`: it gets the same 40 seconds from its join instead of being marked unreachable the moment it joins. ### Disconnected is not dead [#disconnected-is-not-dead] None of these transitions releases anything. Allocations, bookings and queued commands stay exactly as they were: an unreachable node is only kept out of new placement. A partitioned server usually keeps running its workloads, and releasing its GPUs to someone else would double-book hardware that is still in use. `lost` is applied automatically only to nodes of `provider` and `vps-host` pools that set `policy.lostAfter`, never to bare metal (a `vps-host` pool opts in so that its hosted instances can be recreated elsewhere). When the node reconnects, it resumes where it left off: the gateway replays its unfinished commands and the desired state it missed (see [delivery](/docs/daemon/delivery)). ## Revoking a node's identity [#revoking-a-nodes-identity] When you suspect a node's key is compromised, or you are handing the machine to someone else, revoke its identity. [`nodes.revokeIdentity`](/docs/reference/api/nodes#revokeidentity) (`compute:nodes:revoke-identity`): * revokes every live certificate of the node (with your `reason`, or `identity revoked`); * raises `identityGeneration`, so no certificate of the old generation can renew or re-join with `x509pop`; * publishes `security.node.identity-revoked` and returns `{ node, revokedCertificates }`. A gateway that runs a revocation feed (`createRevocationFeed` from `@better-compute/pki`, which polls `node_certs` every second by default) learns about the revocation from Postgres and closes the node's live session at once; without a feed, a revoked certificate keeps its session until it expires. The machine can come back only by joining again with fresh evidence, such as a new join token. The gateway revokes a certificate on its own when it looks cloned: after more than 3 alternations between two machines within 10 minutes it quarantines the certificate with reason `duplicate-identity` and publishes `security.node.quarantined`. Watch `security.node.duplicate-session` to see it coming, then find the copy and re-join the real machine. See [duplicate identity](/docs/daemon/sessions#duplicate-identity). ## Adopting again [#adopting-again] [`nodes.adopt`](/docs/reference/api/nodes#adopt) (`compute:nodes:provision`) re-runs acceptance on a node in `adopting` or `adopted`; any other state is `INVALID_STATE`. An adopted node goes back to `adopting` and leaves placement until the checks pass. See [acceptance checks](/docs/guides/nodes/inventory-and-health). ## Deleting a node [#deleting-a-node] [`nodes.delete`](/docs/reference/api/nodes#delete) (v0.2, `compute:nodes:delete`) removes a node that is done for good or never got going. It accepts only: * a `retired` node; * an `adopted` node that is cordoned (54cad9e); * a node still in `adopting` or `enrolling`, or one that failed during enrollment; * and only when the node has no open allocation (`INVALID_STATE` with the count otherwise). Any other state is `INVALID_STATE` with `{ provisionState, cordoned }` in its details. An adopted node that is not cordoned gets "Cordon the adopted node first (nodes.cordon), then delete it once it holds nothing". In one transaction it makes the node's devices unbookable, removes its scalar resources and rebuilds the pool's capacity rollup, revokes its certificates (`node-deleted`), frees its attestor bindings and trusted hardware key, cancels its pending commands, and removes it from every tenant's projection. The row itself stays as a tombstone (`deletedAt`), because audit rows, certificates, inventory snapshots and past allocations still reference it. The machine can then join again as a new node. A retry of the same delete returns `{ nodeId, deleted: false }`, and `node.deleted` is published once. For bare-metal nodes, retire through the metal workflow first; see [bare-metal workflows](/docs/guides/bare-metal/workflows). ### Removing an adopted host [#removing-an-adopted-host] An adopted host (bcd installed on a machine metal does not manage) has no retire workflow, so the retire path for it is a cordon and then the delete. Use it when the machine is decommissioned, or when it must join again under a new identity, for example after its TPM or disk was replaced (see [recovering a node whose TPM changed](/docs/guides/nodes/attestation#recovering-a-node-whose-tpm-changed)): 1. Cordon the node, so nothing new lands on it. [Drain](#drain) it instead when you want to wait for its allocations to end. 2. Wait until it holds no open allocation: release its reservations, or let them end. 3. Delete it. Its certificates are revoked, and its attestor bindings (a `tpm` join's EK binding included) and its trusted hardware key are freed. 4. On the machine, as root, join again with `--reset --yes` and the join's credential (a new token, or `--attestor tpm`). The delete revoked the old certificate but left the files on the machine, and an identity that still loads would only answer `already joined`; `--reset` moves it aside first (see [joining again with --reset](/docs/daemon/install#joining-again-with---reset)). The machine joins as a new node. ```bash better-compute nodes cordon node_01k5w… --tenant $TENANT --reason 'TPM replaced' # ... once the node holds no open allocation: better-compute nodes delete node_01k5w… --tenant $TENANT --reason 'TPM replaced' --yes # then on the machine, as root: bcd join --config /etc/bcd/config.toml --reset --yes --attestor tpm ``` [`nodes delete`](/docs/reference/cli#nodes-delete) (4049621) takes `--reason` and `--if-version` like the other node changes, and without `--yes` it refuses with a usage error before making any call (`nodes delete removes for good; re-run with --yes`). From a client: ```ts await client.nodes.cordon({ tenantId, nodeId, reason: 'TPM replaced' }); // ... once the node holds no open allocation: await client.nodes.delete({ tenantId, nodeId, reason: 'TPM replaced' }); ``` > **A new node id and the old journal.** `bcd node` refuses a journal of another node ("the journal belongs to node ``, not ``"), and a plain `bcd join` moves an old journal aside only when the old identity still loads. After `--reset` it does not, so `--reset` moves `journal.db` aside with the identity (`journal.db.reset-`, b4fcc2e) and the new node starts a journal of its own. Under `[helper]` the root join also hands `identity.json` and `node.key` to the `[helper]` user when it finishes, so no `chown` is needed before restarting `bcd-helper.service` and `bcd.service`. ## The nodes API [#the-nodes-api] | Method | Action | Phase | What it does | | ---------------------------------------------------------------------------------------------- | ------------------------------------------------- | ----- | ---------------------------------------------------------------------------------------------------- | | [`list`](/docs/reference/api/nodes#list) | `compute:nodes:read` on the tenant, then per item | v0.1 | Filters: `poolId`, `siteId`, `cordoned`, `provisionState`; up to 50 per page | | [`get`](/docs/reference/api/nodes#get) | `compute:nodes:read` | v0.1 | The node with labels, conditions and its session | | [`events`](/docs/reference/api/nodes#events) | `compute:nodes:read` | v0.1 | The node's events, oldest first; `type` filters by prefix (`node.`, `security.`), up to 200 per page | | [`cordon`](/docs/reference/api/nodes#cordon), [`uncordon`](/docs/reference/api/nodes#uncordon) | `compute:nodes:cordon` | v0.1 | Stop or resume new placement | | [`drain`](/docs/reference/api/nodes#drain) | `compute:nodes:drain` | v0.1 | Cordon and wait for the node to empty | | [`label`](/docs/reference/api/nodes#label), [`taint`](/docs/reference/api/nodes#taint) | `compute:nodes:update` | v0.1 | Operator labels and NoSchedule taints | | [`adopt`](/docs/reference/api/nodes#adopt) | `compute:nodes:provision` | v0.1 | Re-run acceptance | | [`revokeIdentity`](/docs/reference/api/nodes#revokeidentity) | `compute:nodes:revoke-identity` | v0.1 | Revoke every certificate, new identity generation | | [`delete`](/docs/reference/api/nodes#delete) | `compute:nodes:delete` | v0.2 | Tombstone a retired, cordoned adopted, or never-adopted node | | [`issueSshCert`](/docs/reference/api/nodes#issuesshcert) | `compute:nodes:ssh-root` | v0.3 | Break-glass SSH certificate; see [SSH](/docs/guides/access/ssh) | To run diagnostics, collect logs or reboot a node, use [commands](/docs/guides/nodes/commands).