Rollouts

Upgrading the agent on every machine is one of the riskiest things a platform does. A bad build that crashes on one kernel, or loses its session under load, can take a fleet offline faster than anyone can react. A rollout moves one signed bcd version through the fleet in waves (1%, 10%, 50%, 100% by default), never disrupts more of a pool than its budget allows, pauses by itself when too many nodes of a wave fail, waits between waves, and can send nodes whose upgrade has not confirmed back. A release that nodes confirmed is undone by rolling forward to a newer one.

Nodes learn their target version through their NodeSpec (daemon), which is desired state rather than a command: a node that was offline during the rollout converges on the right version when it returns.

Nodes that run bcd node unprivileged beside bcd helper (installed with daemon install-script --unprivileged) take part in rollouts like any other: the node downloads the release, and the root helper verifies the manifest under its own release pins, swaps root's slots and confirms on the node's Welcome (2d67b25; see self-update through the helper). Their helper must be a build with the update.* operations; an older one refuses them.

Sign a release

A release is a manifest of one version's files, signed offline by keys that the control plane knows only by their sha256/... pins. Nothing on a server can sign one.

better-compute release keygen --out ~/keys/release-alice.pem   # prints the pin

better-compute release sign dist --product bcd --version 1.4.0 --sequence 14 \
  --key ~/keys/release-alice.pem --cosigners ~/keys/release-bob.pub.pem
better-compute release cosign dist/manifest.json --key ~/keys/release-bob.pem
better-compute release verify dist/manifest.json --pins sha256/...,sha256/... --threshold 2 --dir dist

The manifest body (type: bettercompute.release, spec: 1) lists product, version, sequence, issuedAt, expiresAt, the signing keys (kid, alg: ES256, spki) and the artifacts (name, platform, sha256, size), sorted by name. The published form is { manifest, signatures }: the canonical body in base64 and one { kid, sig } per signer.

Turn the module on

import { rolloutModule } from 'better-compute/rollout';

export const compute = betterCompute({
  // database, secret, iam, baseURL ...
  modules: [
    rolloutModule({
      release: { pins: ['sha256/...', 'sha256/...'], threshold: 2 },
      downloadsUrl: 'https://downloads.example.com',
    }),
  ],
});

The module only builds download URLs; it does not serve files. Put each release's bare binaries at {downloadsUrl}/bcd/{version}/linux-amd64 and linux-arm64. The api can serve them itself: with betterCompute({ downloads: { dir } }) (or { blob }), set downloadsUrl to {baseURL}{basePath}/downloads; see installing bcd.

The module adds the migration rollout/0001_rollouts (tables daemon_releases, rollouts, rollout_nodes), the rollout reconciler and the daemon.upgrade command kind. Every method decides compute:fleet:admin on compute-fleet/global, which exists only in the root tenant, so rollouts are for platform operators. See the releases reference and the rollouts reference.

Register a release

releases.register { tenantId, manifest } verifies the signed manifest and records it:

• a threshold of distinct pinned keys signed it, its body is canonical, its product is bcd, and it has not expired. A failure is INVALID_INPUT with details.reason: malformed, threshold, not-canonical, product or expired;
• its sequence is above every registered release's. An older one is INVALID_INPUT with reason rollback, so the release channel never moves backwards;
• it lists at most one bare binary per platform, and at least one for Linux. A bare binary is an artifact with a platform that is not an archive, checksum, signature or metadata file. Several candidates for one platform, or no Linux binary, is INVALID_INPUT.

Registering the same manifest again returns the stored release; the same version with another sequence is CONFLICT. releases.list returns every release, highest sequence first.

better-compute releases register dist/manifest.json --tenant root   # sends the file as it is
better-compute releases list --tenant root

Start a rollout

better-compute rollouts create --tenant root --version 1.4.0 --waves 1,10,50,100 --soak 10m --key bcd-1.4.0

The CLI takes --soak and --upgrade-timeout as durations (10m, 15m) and checks --waves and --max-failure-ratio before it calls; without --key it sends a new idempotency key. See the rollouts commands.

rollouts.create snapshots every node with desired state (generation above 0) in the pools in scope, shuffled by a hash of the rollout id so the canaries spread across pools, and assigns each to a wave. Every wave holds at least one more node than the one before while nodes remain: 20 nodes with [10, 50, 100] make waves of 2, 8 and 10. Nodes that already report the version start as succeeded. It publishes rollout.started.

How the controller advances

The rollout reconciler wakes every 30 seconds and queues each running or paused rollout, and each aborted one that is still rolling nodes back; resume and abort queue theirs at once. One pass locks the rollout and does this:

A rollout itself is running, paused, completed or aborted.

The disruption budget

A pool's policy.disruptionBudget (a fraction from 0 to 1, default 0.05) is shared with health remediation: a node counts as disrupted while it has a pending, running or paused workflow, or an upgrade or rollback in flight. At most max(1, floor(nodes × budget)) nodes of a pool are disrupted at once, so a pool of fewer than 40 nodes upgrades one node at a time under the default, and a pool of 10 with a budget of 0.2 upgrades two.

What a node is told

When a node starts, the controller marks it upgrading, records the version it reported as fromVersion, and invalidates its desired state. Its NodeSpec then carries:

{
  "version": "1.4.0",
  "url": "https://downloads.example.com/bcd/1.4.0/linux-amd64",
  "sha256": "9f2c...e41a",
  "signature": "eyJtYW5pZmVzdCI6..."
}

• The version is the node's row in the latest rollout that touched it: the rollout's version while upgrading or after success, its previous version after a failure or a rollback. A node no rollout touched gets the version of the latest completed rollout that covered its pool, so skipped offline nodes and nodes that join later converge on it. A node with neither gets no daemon section, and bcd is left alone.
• The platform comes from the node's cpu.arch label (x86_64 and amd64 are linux-amd64; aarch64 and arm64 are linux-arm64). Without the label, or without a binary for the platform, there is no section.
• The digest is the bare binary's SHA-256 from the verified manifest.
• The signature is the release's signed manifest as registered ({ manifest, signatures }, the manifest.json of release sign), its UTF-8 JSON as the section's bytes (releaseSignature, ecf7cd0). bcd verifies it against its own [update] release_pins, so a compromised control plane cannot push a binary of its own. daemon.upgrade carries the same field.

daemon.upgrade for restricted sessions

A daemon older than the server's minimum protocol gets Welcome.upgrade_required, and its session then accepts only daemon.upgrade, so it would never apply a NodeSpec. When the controller starts such a node (its node_sessions.proto_max is below the minimum), it also enqueues a daemon.upgrade command with the same target, idempotency key rollout:<rolloutId>:<nodeId>, expiring after the upgrade timeout. The kind runs on the control lane, is retry-safe and self-terminating, times out after 15 minutes, and takes the DaemonSection as its payload. The gateway dispatches only that kind to an upgrade-only session (and replays an in-flight one at Hello); the node's other commands stay pending until it rejoins at a supported version. See command kinds and sessions.

Pause, resume, abort

Reasons are at most 500 characters. Only a running or paused rollout changes; anything else is INVALID_STATE.

Without rollback, an abort marks nodes still upgrading as failed, and their NodeSpec points back at the version they reported before (or drops the daemon section when that version was unknown); nodes that already succeeded keep the new version. With rollback: true, every node still upgrading whose previous version is a registered release moves to rolling-back, gets that version as its target, and finishes as rolled-back when it reports it (8347a3d). Nodes that succeeded keep the release either way.

better-compute rollouts pause rollout_01jzq3v0m6y8x7w5t4s3r2p1n0 --tenant root --reason 'XID spike in wave 2'
better-compute rollouts abort rollout_01jzq3v0m6y8x7w5t4s3r2p1n0 --tenant root --rollback

Undo a bad release by rolling forward

A node that succeeded has confirmed the release: its release floor rose to that release's sequence, and bcd now refuses every older binary, including the one still in its other slot (c139b9c). Sending it back would leave it rolling-back until the timeout, so abort does not try. Only an upgrade that has not confirmed goes back, and bcd's own watchdog swaps such an update back anyway when the new binary gets no Welcome before its deadline.

To undo a release that nodes already confirmed, roll forward:

1. Build the fix, or rebuild the last good code, as a new version.
2. Sign it with a higher --sequence than the bad release (releases.register refuses a lower one with reason rollback).
3. Register it and start a new rollout of that version. Abort the bad rollout first if it is still running, since a node can be in only one running or paused rollout.

better-compute release sign dist --product bcd --version 1.4.1 --sequence 15 \
  --key ~/keys/release-alice.pem --cosigners ~/keys/release-bob.pub.pem
better-compute release cosign dist/manifest.json --key ~/keys/release-bob.pem
better-compute releases register dist/manifest.json --tenant root
better-compute rollouts abort rollout_01jzq3v0m6y8x7w5t4s3r2p1n0 --tenant root --reason 'bad release 1.4.0'
better-compute rollouts create --tenant root --version 1.4.1 --key bcd-1.4.1

bcd refuses an older release on its own too: the release floor is enforced on the node, so a control plane cannot push a node back to an older binary even by accident.

Watch progress

rollouts.list returns the tenant's latest 100 rollouts. rollouts.get adds nodeList: every node's pool, wave, state, fromVersion, start and finish times, and error.

better-compute rollouts list --tenant root
better-compute rollouts get rollout_01jzq3v0m6y8x7w5t4s3r2p1n0 --tenant root --query nodes

{
  "id": "rollout_01jzq3v0m6y8x7w5t4s3r2p1n0",
  "version": "1.4.0",
  "state": "running",
  "waves": [1, 10, 50, 100],
  "wave": 1,
  "maxFailureRatio": 0.05,
  "soakSeconds": 600,
  "upgradeTimeoutSeconds": 900,
  "nodes": {
    "pending": 180,
    "upgrading": 8,
    "succeeded": 12,
    "failed": 0,
    "skipped": 0,
    "rolling-back": 0,
    "rolled-back": 0
  }
}

wave counts from zero. The console's Rollouts page (platform administrators) shows each wave's progress, lists failures, and has pause, resume and abort buttons.

All have the subject rollout/<id> and reach webhooks and watch.

Defaults