better/compute
Capacity

Reservations

Reserve GPUs from a pool with an idempotent request that Better IAM decides on, follow it from draft to released, and extend, renew or release it.

A is a tenant's claim on devices of one pool for a time window: four NVLink-connected H100s for eight hours from now, or two nodes for three days starting Monday. It is created as a request-as-resource: the request is first written as a , Better IAM decides on that draft with everything it asks for as attributes, and only an allowed draft is placed. So a policy such as "at most 8 H100s for at most 72 hours in these regions" is one decision on the reservation itself, not a check you write in code.

Reserve GPUs

reserve.ts
import { compute } from './compute';

const reservation = await compute.api.reservations.create(
  { headers: request.headers }, // the caller's credential
  {
    tenantId: 'acme',
    poolId: 'pool_01k…',
    devices: [{ class: 'gpu', count: 4, topology: 'nvlink-all' }],
    duration: '8h',
    idempotencyKey: 'train-2026-09-24',
  },
);
// reservation.state: 'placed' (or 'pending' with an Unschedulable condition)

Prop

Type

Unknown fields are INVALID_INPUT. binding: 'instance' is UNSUPPORTED: VPS instances are created with instances.create and book through the capacity port (see instances).

The reservation object

Prop

Type

An active reservation
{
  "id": "rsv_…",
  "tenantId": "acme",
  "poolId": "pool_01k…",
  "state": "active",
  "request": {
    "poolId": "pool_01k…",
    "devices": [{ "class": "gpu", "count": 4, "topology": "nvlink-all" }],
    "duration": "28800000ms",
    "binding": "hold"
  },
  "during": { "start": 1790000000000, "end": 1790028800000 },
  "ownerIdentityId": "…",
  "idempotencyKey": "train-2026-09-24",
  "conditions": [],
  "binding": {
    "nodeId": "node_01k…",
    "cdiDevices": ["bcd.dev/gpu=GPU-…", "bcd.dev/gpu=GPU-…", "bcd.dev/gpu=GPU-…", "bcd.dev/gpu=GPU-…"],
    "envFile": "/run/bcd/allocations/alloc_….env"
  },
  "operation": { "id": "rsv_…", "state": "succeeded" },
  "version": 4
}

Drafts and idempotency

The draft id is deterministic: rsv_ followed by the base32 SHA-256 of the tenant, the caller's identity and the idempotency key. Retrying with the same key reaches the same row, and the same key from another identity gets a row of its own, so nobody can squat on or replay someone else's request.

The stored request hash covers the validated input without wait and idempotencyKey. Durations are normalized to milliseconds and binding defaults to hold, so '2h' and 7200000 are the same request, with or without binding: 'hold'. A second call with the same key:

  • with the same request, on a reservation past the draft stage, returns the stored reservation with no second IAM decision;
  • with the same request, on a draft (for example after QUOTA_EXCEEDED), decides the same draft again;
  • with a different request, is CONFLICT with details.reservationId;
  • on a refused key, stays ACCESS_DENIED.

Drafts are internal: reservations.get answers NOT_FOUND for them and reservations.list skips them.

The admission sequence

Every create follows the normative sequence of PLAN.md section 3.4, and extend follows it against the existing reservation. IAM is never called inside a compute transaction.

Authenticate and pre-check

The credential is authenticated (cached for 30 seconds). If the caller's tenant is neither the input tenant nor the root tenant, the answer is ACCESS_DENIED before compute's database is read, the same answer whether or not anything exists.

Transaction 1: the draft

The idempotency row is looked up and locked. The pool must be owned by or offered to the tenant and not archived, or the answer is NOT_FOUND. Device classes must fit the pool's sharing mode, and binding: 'node' needs a tenantRoot pool, binding: 'kubernetes' a pool with policy.kubernetes that the tenant owns (3825ec7; a tenant the pool is only offered to is INVALID_INPUT). A request without resources gets the pool's per-GPU share of host CPU and memory here, stored in the request (see host resources). Then the draft is inserted with its row in the IAM , parented to the pool and owned by the caller:

AttributeValue
gpuModelThe pool's GPU model, or other
gpuCountWhole GPUs requested, times the gang size
gpuSlicesMIG instances and slots requested, times the gang size
gangSize1, or the gang size
durationHoursThe duration in hours, to three decimals
regionThe region of the pool's site
binding, spot, renewable, projectIdAs requested; renewable is true with leaseSeconds

Decide exactly once

iam.require decides compute:reservations:create on compute-reservation/<draft id>. The resolver hands IAM the attributes above and the pool's relations, so pool access and request limits are one decision; see policies. A refusal marks the draft denied, unless the caller is an agent in a delegated session and IAM opened a confirmation request (below).

Still outside any transaction, the budget check (budgetAdmissionCheck) runs when the pool's policy sets admission.budgetCheck: 'meter' or the tenant's offer has onBudgetExceeded: 'stop-new' or 'drain': it calls Better IAM's billing.check for the caller, and a spent budget answers SPEND_LIMIT_REACHED. The draft stays, so the same key is decided again once the budget's window resets. See budgets.

Transaction 2: quota and placement

Quota is charged all-or-nothing first. QUOTA_EXCEEDED ends the request and leaves the draft, so a retry with the same key is decided again. Then the placer runs. A regular request that does not fit may preempt spot work. The outcome is placed, pending with an Unschedulable condition (retried every 30 seconds), or, with the pool-lease placer, pending with a Queued condition. The same transaction schedules the lifecycle, writes the event to the outbox and records the audit row with the admission context.

Two details differ from the PLAN.md text. Quota is charged by the reservation service before placement, never by the placer, because a pending reservation keeps its charge (decision D9).

Waiting for placement

create is async-with-wait (PLAN.md section 9.1): every reservation carries an operation, and a request with wait: { timeoutMs } holds its answer until that operation stops running. The same client code therefore works with both placers.

operation.stateReservation
runningA draft, or pending with a Queued condition: waiting for the pool-lease batch. The only state a wait waits on
pendingpending without Queued (Unschedulable, retried every 30 seconds), or awaiting-confirmation
succeededplaced, binding, active, releasing or released
failedfailed (CAPACITY_UNAVAILABLE), denied (ACCESS_DENIED), cancelled or expired (INVALID_STATE), with error

After transaction 2 commits (or a repeated key returns the stored reservation), the server re-reads the reservation in read-only transactions, backing off from 50 ms to 1 s, until the operation is no longer running, timeoutMs passes (capped at 30 seconds), or the request is aborted. The answer is the last read either way: a timeout is not an error, it returns the reservation still Queued. With the inline placer a create is placed or Unschedulable at once, so wait returns immediately. wait is excluded from the request hash, and extend does not wait.

const reservation = await client.reservations.create({ ...input, wait: { timeoutMs: 20_000 } });
if (reservation.operation?.state === 'running') {
  // Still queued after 20 s: read it again later with reservations.get.
}

The better-compute reserve command sends wait (its --wait, 20 seconds by default) and, if the answer is still Queued, keeps polling reservations.get until the same deadline. The MCP reserve_gpus tool sends wait from its waitSeconds.

States and conditions

denied, released, failed, cancelled and expired are terminal. The capacity.reservations drives each reservation, woken by the work queue at the moments that matter: the start, 15 minutes before it, the bind deadline, the end and the lease deadline. While a reservation is active, the reconciler only sets and clears conditions from its nodes and atoms, and never releases it (PLAN.md section 10.5): node connectivity and health changes wake it, and it rechecks every 15 minutes without them.

ConditionReasonMeaning
UnschedulableNoFitNothing fits; the message is the placer's summary
UnschedulableMaxPendingExceededPending longer than the pool's maxPending (default 24 h); the reservation failed
UnschedulablePoolGoneThe pool is no longer offered to the tenant, or archived; the reservation failed
UnschedulableBookingRepairFailedThe booking went bad before the start and nothing else fits
UnschedulableSpotPreemptedA spot reservation that had not started was preempted and re-queued
QueuedPoolPlacerWaiting for the pool-lease placer's next batch
BindTimeoutNodeDidNotBind, NodeRejectedBindingThe node did not bind in time or refused; the message counts attempts
DegradedBindingLostA node reported it lost devices it had bound
DegradedForeignGpuExposureA node of an active reservation reports the node condition ForeignGpuExposure: something besides bcd can hand its GPUs to pods, so they are not the tenant's alone (4570dfe). The message lists the nodes. BindingLost wins over it, and it wins over AtomUnhealthy; nothing is released. See GPU isolation
DegradedAtomUnhealthyA booked atom of an active reservation is unhealthy; the message lists them. BindingLost wins when both apply
NodeUnreachableUnreachableA node of an active reservation is not connected; the message lists the nodes
PreemptedSpotPreemptedA running spot reservation was cut back to its notice deadline
ConfirmationRequiredthe confirmation idWaiting for a person; the message is the deadline in epoch milliseconds
UnhealthyAtomsNotBilledA node lease holds unhealthy GPUs that it is not billed for
DevicesHeldPodsHoldDevicesA released binding: 'kubernetes' reservation whose allocation is still releasing: pods on the named nodes still hold its GPUs, which stay booked until the node reports them free (0127ed7), billed never past the allocation's own end; a release stuck past that end is carried in 5-minute steps, unbilled, for at most 24 hours (e7a56d7). since is when the release began. Computed by reservations.get and list, not stored; it disappears once AllocationReleased or the hosts pass ends the release. See release

Lifecycle changes emit events with subject reservation/<id> and state and poolId in their data: reservation.placed (with nodeIds, start, end), reservation.pending, reservation.queued, reservation.binding, reservation.active, reservation.extended, reservation.released (with reason: release, expired, lease-lapsed or preempted), reservation.cancelled, reservation.failed, reservation.awaiting-confirmation, reservation.expired, reservation.repaired, reservation.preempting, reservation.preempted, reservation.reassigned (an offboarded owner's reservation handed to the successor, with reservationId, from, to and reason; it has its own row in the event catalog and webhooks.eventTypes; see IAM events) and reservation.binding-rejected (with allocationId, nodeId and reason; see attested bindings). Subscribe with webhooks or watch.

A node that becomes unreachable only raises conditions on the node; the reservation keeps its booking (principle P6).

Bindings

The binding says how the devices reach a workload.

hold (v0.1, the default) hands the devices to software already running on the node, such as Slurm, Kubernetes or containerd with CDI. At the start the reservation moves to binding and the node's desired state gains a devices assignment per allocation (allocationId, deviceIds, and until, the booking's end). The node writes a spec and an env file and reports AllocationBound with the CDI device names; once every allocation is bound the reservation is active, and reservations.get returns binding. Failure handling:

  • no report within 5 minutes, or an AllocationFailed report: the bookings are released and the reservation is placed again, at most twice; the third failure fails it;
  • an AllocationReleased report for a bound allocation: Degraded with BindingLost until the node reports it bound again.

The gateway accepts a report only about an allocation placed on the reporting node's own session; a report naming another node's allocation is ignored like an unknown one. A report that arrives while the reservation is still placed is judged as soon as the start is reached, not after the bind timeout.

kubernetes (v1.1, a21c1c1) is a hold whose devices the node hands to its kubelet: the assignment carries BINDING_KUBERNETES, bcd binds it like a hold and marks its CDI spec for its device plugin, which advertises exactly those devices as bcd.dev/gpu, bcd.dev/gpu.mig or bcd.dev/gpu.slot until the reservation ends. It needs a pool with policy.kubernetes (INVALID_INPUT otherwise), and only the pool's owner may book it (3825ec7): the kubelet gives bound GPUs to any pod of the cluster, so a tenant the pool is merely offered to gets INVALID_INPUT and books holds instead. It places only on nodes whose bcd advertises the binding.kubernetes capability (system label bc.dev/cap.binding.kubernetes, 409f783): an older bcd drops BINDING_KUBERNETES, and its GPUs would never reach the kubelet. See Kubernetes workers.

Host resources

A reservation can book a host's CPUs, memory and 1 GiB huge pages with its GPUs (PLAN.md §3.8, a21c1c1):

  • What is bookable. The capacity.hosts reconciler (every 30 seconds, hostSyncEvery) keeps each host's vcpu, memoryGiB, hugepages2Mi and hugepages1Gi rows in step with its applied inventory (8a7f8eb), and since b7e8bf5 inventory wakes it (and vps's vps.hosts) for a host as soon as it applies that host's snapshot, so a new or re-inventoried host is bookable at once rather than at the next pass; see pools. A host without rows books none.
  • resources. { vcpu, memoryGiB, hugepages1Gi } per node (per gang member for a gang), booked on the devices' node in the same guarded statement as the devices, so a node that cannot give them is not a fit.
  • Without resources, a request books the pool's per-GPU share of CPU and memory, times the GPUs' worth it asks for: policy.perGpu.vcpu and policy.perGpu.memoryGiB where set, otherwise the smallest share any GPU host of the pool can give (allocatable × ratio ÷ physical GPUs, counting every physical GPU whether bookable or not), so the share fits on every host; the product is rounded down. Since 004579e fractional GPUs count too (gpuEquivalent): a MIG instance is 1/7 of a GPU, and a time-slice or MPS slot 1/policy.sharing.replicas (default 4, or 10 for mps), so MIG and slot work also carries, and is limited to, its share of the host. It is resolved once, when the draft is written, and stored in the request. Pools whose hosts have no rows book none, so they behave as before.
  • The share is also the floor (41826a2). resources below the pool's per-GPU share, or resources: {}, are raised to it, so booked GPUs never run a workload without CPU and memory limits while the same CPU and memory are sold again.
  • Quota. The share rides on the GPUs and needs no quota beyond theirs; the reservation records it as hostShare when it is created, and since 004579e always, as {} when the share is empty (a pool without one), so anything booked above an empty share is charged too. Anything booked above it is charged to the tenant's vcpu and memoryGiB quota (1 GiB huge pages count as memory), and a missing quota row grants 0, so the excess is QUOTA_EXCEEDED unless the offer names these resources (41826a2). One GPU's quota can then not take a whole host's CPUs and memory. Reservations made before hostShare existed charge their devices only. IAM sees vcpu and memoryGiB (summed over a gang) as attributes of the reservation.
  • Preview. capacity.preview books the same host resources as reservations.create would, the request's or the pool's per-GPU share, so a preview that fits is a create that fits (17f600b); a node binding books none.
  • Only on a bcd that enforces them (409f783). An older bcd drops DeviceAssignment.limits as an unknown field and would run the workload unbounded, so a reservation that books any host resource (a non-zero vcpu, memoryGiB or hugepages1Gi, including the per-GPU share) places only on nodes with the system label bc.dev/cap.devices.limits, which the gateway writes from the devices.limits capability a GPU host's bcd advertises in Hello (see capabilities). An operator label of the same name does not count: capacity reads bc.dev/ labels only when their source is system. The requirement applies to create, to re-placing a pending reservation, to repairing a booking and to capacity.preview (a node without the label is rejected with filter labels, needs label bc.dev/cap.devices.limits=*); the placer also re-reads the labels under lock when it commits, so a daemon downgraded between plan and commit is caught. When no advertising node fits, the reservation waits pending like any other. A plain GPU hold that books no host resources (a pool whose hosts have no resource rows) needs no capability, so it still places on older daemons within the N-2 skew window, as does a node binding.
  • From the CLI and the console. better-compute reserve and better-compute capacity --preview take --cpus (0 to 4096) and --memory (GiB, 0 to 65,536), sent as resources, and --kubernetes for binding: 'kubernetes' (ea36a94). @better-compute/console-kit's reservation form takes the same three as vcpu, memoryGiB and kubernetes, and previews with them (e833433). The console app's reservation page shows them since 4533265: optional Host CPUs (0 to 4096) and Host memory, GiB (0 to 65,536), left empty for the pool's per-GPU share, and the Kubernetes checkbox only for a pool the tenant owns that has policy.kubernetes.
  • On the node (c5a419c, 260e33c), the booked resources become the assignment's limits: cpu_millis from vcpu, memory_bytes from memoryGiB, hugepages_1gi, and a cpuset on the NUMA nodes of the hold's devices when those nodes have enough CPUs (cpus) or memory (mems). Since b6b804e a cpuset never includes the host's reserved CPUs, the lowest non-isolated, policy.hostReserve.vcpu (default 2), and the reserve counts against the fit. The node's cpulists (its NUMA nodes' CPUs and isolated CPUs) come from its inventory, so capacity reads them within bounds (004579e): CPU ids up to 8191, at most 32 KiB and 4096 comma-separated parts, a longer list read as empty, so an oversized list cannot stall the control plane. bcd admits the hold against the host's memory beside its VMs and other holds, and one that does not fit fails with resources-exhausted, which re-places it (31555bf). It then creates the cgroup slice bcd-alloc-<allocation>.slice with those limits, names it in the CDI spec (bcd.dev/cgroup-parent) and the env (BCD_CGROUP_PARENT), and stops it once it is empty after the release, retrying every pass (e01d7ff).
  • Where the workload runs. The workload manager on the node (containerd, Slurm, a wrapper) places the workload in that slice; see device assignments. Once the reservation is active, reservations.get shows the slice as binding.cgroupParent (HoldBinding.cgroupParent, 6aa412b), which bcd reports in AllocationBound.cgroup_parent (c81f230, 19dbf60); the control plane keeps it only when it is exactly bcd-alloc-<allocation>.slice for that same allocation (004579e), so a node cannot point a reservation at another allocation's slice. It is empty for a hold without limits and for a kubernetes binding: that binding gets no slice, since its pods live in the kubelet's cgroups, where the kubelet enforces their requests and limits; its booked limits still count in bcd's host admission (see Kubernetes workers).

Attested bindings

A gateway relays AllocationBound, so on its own a compromised gateway could claim a binding the node never made. Attested bindings (v0.2, principle P14) close that gap: the node signs the report with its own key, the gateway stores the signed bytes as received (allocations.bound_report, bound_sig_kid, bound_sig, core/0017), and the reservation worker verifies them before the binding counts.

With pki.commandSigning configured, betterCompute() turns this on by default: capacity's attestedBindings is pkiAllocationBindingCheck() from @better-compute/server. You can also pass your own capacity: { attestedBindings } check. The pki check accepts a report only when:

  • it is signed by a certificate of that node (current, or recently expired within the offline grace, never revoked);
  • the certificate's role runs workloads (node or node-tenant by default; roles changes it);
  • the signed body reports exactly this allocation bound.

A good report replaces the binding with the signed CDI devices and env file, marked attested, whatever the gateway said. With the option on, only attested allocations count as bound. A report that fails the check never counts, and each rejection publishes reservation.binding-rejected with allocationId, nodeId and reason (unsigned report, a signature failure, the signed report is not this allocation bound, …):

State when rejectedWhat happens
bindingbound_at is cleared, so nothing is billed; the reservation waits for a genuine report or its bind timeout
activeThe binding is marked lost: Degraded with BindingLost, and billing keeps the real binding

Node leases (binding: 'node') are exempt: metal's worker binds them after its deploy workflow verified the machine. The end-to-end test tests/e2e/attested-bindings.test.ts runs this through a real gateway on the dev stack: an unsigned report, one signed by a foreign key, and the node's genuine signature over another allocation never activate the reservation, and a forged re-report of an active binding marks it lost. Both daemons sign these reports with their node key: the FakeDaemon, and the Go daemon bcd, whose devices converger sends every bind and release as a node-signed AttestedReport (purpose attested-report).

Both implement the node side the same way (CDI spec /etc/cdi/bcd-<alloc>.json, env file /run/bcd/allocations/<alloc>.env, device names bcd.dev/gpu=<id>); the CDI spec is the hold itself, and neither drops a hold on its own. The FakeDaemon drives the simulator and better-compute dev --sim N; bcd is built and tested end to end against the gateway but not released. See desired state and commands and the daemon.

node (v0.5) leases whole machines in a pool whose policy sets tenantRoot: true. It needs node: { profileId, profileVersion? } and cannot be spot. It books every bookable atom of ready, platform-trusted nodes; metal deploys the profile from the start and marks the allocation bound, with 2 hours to do so and no second placement. See dedicated root.

Start, lease, extend and release

Start. Without start the window begins at placement. A start in the future books the window now; see calendar reservations. A pending reservation keeps its requested start while it is still ahead; once it has passed, the window begins when a retry places it, with the full duration.

Renewable leases. With leaseSeconds, the lease runs out leaseSeconds after placement. reservations.renew (compute:reservations:extend) moves it to now plus leaseSeconds without touching during, in states placed, binding and active. A lapsed lease releases the reservation with reason lease-lapsed. Renewing a reservation created without leaseSeconds is INVALID_STATE. Leases suit agents that may die without cleaning up.

Extend. reservations.extend takes exactly one of until (epoch milliseconds) or by (a duration), plus an idempotencyKey. It is admitted like a create: transaction 1 writes the pending change into the authz row, with durationHours set to the total after the change, so a cap such as BoundedExtend also bounds extensions. On allow, every allocation's end moves later, all or nothing, and the EXCLUDE constraint decides:

const result = await compute.api.reservations.extend(credential, {
  tenantId: 'acme',
  reservationId,
  by: '4h',
  idempotencyKey: 'train-2026-09-24-ext-1',
});
// { reservation, extended: true }
// or { reservation, extended: false, availableUntil: 1790040000000 } when another booking starts first

An until that the reservation already reaches answers extended: false without a decision, so prefer until when you may retry: a repeated by extends again from the new end. A refusal clears the pending change; so does a handler that throws after an allow (the reservation changed between the transactions), through the admission's abandoned hook in a transaction of its own, so a refused lifetime is never merged into later decisions on the reservation (a2288ec). A retry with the same key decides again.

One extend at a time stages the pending change (f36248d, review 7), since the authorization projection keeps one pending value per resource: two concurrent extends could otherwise interleave, one decided on the other's shorter duration and then applying its own longer one past BoundedExtend, and one's refusal could clear the other's change. capacity_extend_holds (migration capacity/0005_extend_holds: the reservation, the hold and when it was taken; deleted with the reservation) gives the slot to one request. The hold is a nonce the server mints for each extend (newId('hold'), kept on the parsed input, which callers cannot set), not the caller's x-request-id, which two concurrent extends could share (d6f7ddc); its age is measured with compute_now(), since replicas' clocks may disagree. While another extend holds the slot, an extend is CONFLICT with details.reason: 'extend-in-flight' and retryAfterSeconds: 60; a hold older than 60 seconds is taken over. Transaction 2 applies the change only while its hold still owns the slot, and a refusal, an abandon or a budget refusal clears the pending change only when the hold is its own. The completing transaction takes the pool lock first (bb6da07), so an extend of a gang and a batch commit cannot deadlock on each other's bookings. Sandboxes do the same with their own hold (see sandboxes).

Release and cancel. reservations.release ends a reservation at compute_now(): the current booking is truncated, future ones are deleted, quota is refunded, and billing stops. Releasing a pending reservation cancels it, and releasing a terminal one changes nothing. reservations.cancel withdraws one that has not started (pending or placed); anything else is INVALID_STATE. Both are compute:reservations:release. Freed capacity wakes the pool's pending reservations at once instead of at their next 30-second retry.

better-compute extend rsv_… --tenant acme --until +12h
better-compute release rsv_… --tenant acme

Delegated confirmation

When an agent acting for a person in a delegated session is refused, and the person's delegation holds the action for confirmation, Better IAM opens a confirmation request and the call fails with CONFIRMATION_REQUIRED (details.confirmationId, and expiresAt when IAM gives one). The draft moves to awaiting-confirmation with a ConfirmationRequired condition and no quota charged. The deadline is IAM's, clamped to at most 30 minutes.

Once the person approves, the agent retries with the same idempotency key. The retry reaches the same draft, IAM decides it again, finds the approval, and the reservation is placed. After the deadline, the reservation becomes expired, either by the reconciler or on a late retry. See delegated confirmation and the MCP server, which runs this loop for agents.

The reservations API

MethodActionDecided onPhase
reservations.createcompute:reservations:createthe draftv0.1
reservations.getcompute:reservations:readreservationv0.1
reservations.listcompute:reservations:readtenantv0.1
reservations.releasecompute:reservations:releasereservationv0.1
reservations.cancelcompute:reservations:releasereservationv0.2
reservations.extendcompute:reservations:extendreservationv0.1
reservations.renewcompute:reservations:extendreservationv0.1

All seven are served today. reservations.list returns the caller's own reservations by default (scope: 'mine'); scope: 'all' lists the tenant's. It filters by state and poolId, returns newest first, and pages with limit (1 to 200, default 50) and cursor, the last reservation's id. Errors are listed in the error reference.

Was this page helpful?

Better Compute is created by Sean Filimon

Last updated

On this page