Upgrades

An upgrade of Better Compute touches four things that run different code: the database schema, the control-plane processes (api and workers), the gateways, and bcd on every machine. They are upgraded one at a time, in an order that keeps every pair that talks to each other compatible at every moment. This page is the operator's runbook for that order. The detail of each piece lives on its own page and is linked rather than repeated.

The order

PLAN.md §7.6 fixes the order, and the code is built around it:

Two rules follow from it. A newer daemon is never put in front of an older control plane, and a database is never migrated backwards: a release whose migrations ran cannot be undone by deploying the previous code (see rolling back).

Before you start

• Read the changelog between the two versions for new migrations, changed options and new module requirements.
• Take a backup you know how to restore: backup and recovery. It is the only way back from a migration.
• Run better-compute doctor and clear its errors first. A half-done command-signing root rotation (pki.rotation), an unsigned quorum change (pki.quorum-unsigned) or overdue work (workers.queue) is easier to finish before the upgrade than to untangle in the middle of it.
• Check that no rollout is running (better-compute rollouts list --tenant root); a node can be in only one running or paused rollout.

Step 1: migrate

better-compute migrate

migrate runs compute.initialize(): it checks every method's IAM action against the merged catalog and the normative method table (INVALID_CONFIG before touching the store if one disagrees), applies the base schema and every module's migrations, anchors the process clock to compute_now(), settles the trust bundle version, and starts the modules. It is idempotent: run it as a one-off job on every deploy. The ledger, the naming rules and the full list of migrations are on database; what matters for an upgrade is how it decides what to run and what to refuse.

What runs. Migrations run in registration order: the base schema (owner core), the built-in modules, your modules, then your plugins. Each runs in its own transaction and is recorded in compute_migrations in the same transaction, so a failed one leaves nothing behind and is retried on the next run. A migration that fails is reported as INTERNAL_ERROR naming it (Migration <name> failed: …). Migrations run without the executor's statement timeout.

Concurrent migrators. Every migration transaction first takes pg_advisory_xact_lock(1650683248, 1), a transaction-level lock that is safe behind PgBouncer in transaction mode. Two processes migrating at once serialize, and the second skips what the first applied.

What it refuses. Before applying anything, migrate compares the ledger with what this build knows and stops with INVALID_CONFIG ("Refusing to migrate: … Run the better-compute release that migrated this database, or a newer one.") when:

The error's details carry newer, unknown, schemaVersion and knownSchemaVersion. Migrations of owners this build has no migrations for at all are ignored, so removing a module from the configuration does not block startup; its tables stay.

Verify. After the job, better-compute doctor reports migrations as an error when any migration of this release and these modules is not in the ledger ("Migrations not applied: …").

Expand and contract

The migrations in the tree today add tables, columns and indexes, and widen checks (core/0015_inventory_images and core/0028_volume_allocations drop a CHECK constraint and add a wider one). None drops a column or table that an earlier release reads. The one heavy migration is core/0018_partitioned_history: it renames audit_events, inventory_snapshots and usage_windows, creates partitioned tables under the same names, copies every row with its values intact and drops the old tables, all in one transaction, so those tables are locked for as long as the copy takes. Plan for that on a large database. PLAN.md puts destructive changes ("contract": dropping a column the old release still reads) in the release after the one that stopped using them, so that during step 2 the old and new processes can share the schema. None has been needed so far.

Step 2: workers and api

Roll api processes behind their load balancer as you would any stateless service, using GET /readyz for readiness (see health checks). Clients retry failed calls with their idempotency keys.

Workers can be replaced in any order, and old and new workers may run side by side for a while: shard and singleton leases make every job single-writer whatever release holds it. Stop old workers with SIGTERM so they release their leases at once; a worker that is killed loses them only when they expire, after 15 seconds (see high availability). When a reconciler is new in the release, it runs only once a new worker holds the lease for its shard or singleton.

The new processes publish the trust bundle again. Its version follows its content, so an upgrade that changes nothing in the bundle keeps the version, and one that does (new roots, a quorum) takes the next version once; see PKI.

Step 3: gateways

Upgrade gateways one at a time. serve handles SIGTERM by draining the gateway first: every daemon is told to reconnect elsewhere (Drain, with a random delay of up to 60 seconds), the listeners close with a 2-second grace and the gateway:<id> lease is released (see shutdown). Daemons redial through the L4 balancer and re-home onto the remaining gateways; the next Hello replays whatever was in flight, and the daemon's journal deduplicates it, so nothing runs twice. Wait for the drained gateway's sessions to settle on the others before stopping the next one: doctor's fleet.gateway-headroom warns when a gateway holds 80% of its maxStreams.

A gateway killed instead of drained costs the same reconnect, only all at once; see failure modes.

Step 4: bcd

Version skew

Each Hello carries the daemon's proto_min, proto_max, daemon_version and capabilities. The gateway picks the highest protocol version inside both ranges (negotiateProtocolVersion); when the ranges do not overlap it sets Welcome.upgrade_required, and that session accepts only the daemon.upgrade command kind: no desired state, no other command. The full rules, the capability labels (bc.dev/cap.<name>=1) and the per-feature gating are on version skew. In short:

The skew profiles are FakeDaemon behaviours, not old bcd binaries: no older bcd release exists to run. doctor reports fleet.daemon-skew when connected daemons speak a protocol below the server's minimum.

UPGRADE_REQUIRED is also a stable API error code (HTTP 426, gRPC FAILED_PRECONDITION) in @better-compute/core; nothing in the server raises it today. On the daemon path the signal is the Welcome.upgrade_required flag, not an error.

Rolling bcd out

From v0.4 the daemon is upgraded through desired state, not by hand. The full guide is rollouts; the sequence is:

A node too old to share a protocol version with the gateway is in an upgrade-only session; the rollout controller sees its proto_max below the minimum and enqueues daemon.upgrade with the same target, so it can still be upgraded (see daemon.upgrade for restricted sessions).

Nodes installed without --release-pins refuse every self-update and must be upgraded by re-running the install script; see upgrades before v0.4. Workloads are not touched by a bcd swap: in the simulator's upgrade test every VM keeps running and is booted exactly as often as before.

Kubernetes DaemonSets

bcd running as a DaemonSet is upgraded by Kubernetes, not by rollouts; see bcd on Kubernetes.

Step 5: contract migrations

A release that drops something the previous release used ships its contract migration one release later. Run better-compute migrate again only after no process of the old release is left, workers and gateways included (check the gateways directory rows' version and the workers heartbeats).

The API surface

Clients (the TypeScript client, the CLI, MCP, your own code) talk to the api through a method surface that PLAN.md §16 freezes for v1.0. scripts/api-surface.mjs runs in pnpm lint:repo and fails CI when a served method disappears, its route, kind, IAM action or resource changes, an IAM alternative goes, or its input schema rejects an input the frozen schema accepted; error codes may not disappear or change HTTP status. Additions pass. The rules and the update commands are on the API surface freeze.

For an operator, this means an upgrade within the frozen surface does not break a client built against the older release. A client built against a newer release can call methods an older server does not have: the older api answers those with NOT_FOUND ("Endpoint not found"), so upgrade servers before clients that need new methods.

Rolling back

Checklist

better-compute doctor                                    # clean before you start
# back up the database and the PKI bundle (backup-and-recovery)
better-compute migrate                                   # step 1, once
# step 2: replace api processes, then workers (SIGTERM releases their leases)
# step 3: one gateway at a time; SIGTERM drains it
better-compute doctor                                    # migrations, fleet.gateway-headroom
better-compute releases register dist/manifest.json --tenant root
better-compute rollouts create --tenant root --version 1.4.0 --key bcd-1.4.0   # step 4
better-compute rollouts get <rolloutId> --tenant root --query nodes