# Pools (/docs/guides/capacity/pools)

> Pools group the nodes of one site under one GPU model, sharing mode and policy, and offerings make a pool visible to other tenants with a quota.



A pool is the unit you schedule into and sell from. It lives in one
site, holds machines with **one GPU model**, shares its GPUs in one way, and
carries the policy its nodes and reservations follow. Every reservation names exactly one pool, and
every quota is per pool.

A pool has an owner tenant. For platform capacity that is the root tenant; a customer that brings its
own hardware owns the pools on its `custom` site. Other tenants see a pool only when its owner
**offers** it to them, with a quota.

## What a pool is made of [#what-a-pool-is-made-of]

<TypeTable
  type="{
  id: { type: 'string', description: 'pool_ followed by a ULID, or an id derived from an idempotency key.' },
  ownerTenantId: { type: 'string', description: 'The tenant that created the pool and may offer, update and archive it.' },
  siteId: { type: 'string', description: 'The site the pool lives in. It never changes.' },
  class: {
    type: &#x22;'metal-host' | 'metal-dedicated' | 'vps-host' | 'provider'&#x22;,
    description: 'What the machines are for. It sets the role joining machines get and never changes.',
  },
  gpuModel: { type: 'string', description: 'The GPU family, such as H100 (1 to 32 letters, digits or hyphens). Policies and meters use it.' },
  sharing: {
    type: &#x22;'exclusive' | 'mig' | 'timeslice' | 'mps'&#x22;,
    description: 'How GPUs are handed out. It never changes after creation.',
    default: &#x22;'exclusive'&#x22;,
  },
  agent: {
    type: &#x22;'bcd' | 'none'&#x22;,
    description: 'bcd when machines run the daemon; none when they are reached only through a provider API.',
    default: &#x22;'bcd' ('none' for provider pools)&#x22;,
  },
  providerAccountId: { type: 'string', description: 'provider pools only: the provider account (pacct_) that backs the pool.' },
  policy: { type: 'PoolPolicy', description: 'Acceptance checks, overcommit, timers and sharing parameters; see below.', default: '{}' },
  version: { type: 'number', description: 'Raised by every update.' },
}"
/>

### Classes [#classes]

The class decides the role a machine gets when it joins the pool (see
[joining](/docs/guides/nodes/joining)), and only `host` nodes add schedulable capacity:

| Class             | Node role   | Used for                                                                                                        |
| ----------------- | ----------- | --------------------------------------------------------------------------------------------------------------- |
| `metal-host`      | `host`      | Bare-metal GPU servers running `bcd`, reserved as holds                                                         |
| `vps-host`        | `host`      | Hypervisors that run [VPS instances](/docs/guides/vps/instances)                                                |
| `provider`        | `host`      | Machines of a [provider account](/docs/guides/vps/providers), with `agent: 'none'`                              |
| `metal-dedicated` | `dedicated` | Machines [metal](/docs/guides/bare-metal/deploying) deploys whole; `dedicated` nodes add no reservable capacity |

### One GPU model [#one-gpu-model]

A pool holds one `gpuModel`, the family name inventory derives from the NVML name (`H100` for
`H100-SXM5-80GB`). Policies, IAM attributes and meters (`gpu.h100.hours`) all key on it, so a
pool that mixed models would bill and authorize wrongly. Selectors can still narrow within the model by
`gpuProduct`, `vramGiB` or any other attribute ([placement](/docs/guides/capacity/placement)).

### Sharing modes [#sharing-modes]

* `exclusive` (the default): each GPU is one atom, booked whole.
* `mig`: GPUs in MIG mode expose their instances as `gpu.mig` atoms.
* `timeslice` and `mps`: each GPU is split into `gpu.slot` atoms. These modes give no isolation
  between workloads, so these pools are single-tenant: they can be offered only to their owner.

[Fractional GPUs](/docs/guides/capacity/fractional-gpus) covers the atoms and what a request must ask
for in each mode.

## Pool policy [#pool-policy]

`policy` is a JSON object. Unknown fields are stored as given; these are the fields the code reads:

<TypeTable
  type="{
  acceptance: {
    type: '{ kind: string; with?: object }[]',
    description: 'Checks a node must pass on adoption before it becomes schedulable. v0.1 runs diag.nvml.',
  },
  overcommit: {
    type: 'Record<string, number>',
    description: 'Overcommit ratios for scalar resources such as vcpu and memoryGiB. GPUs are never overcommitted.',
  },
  lostAfter: {
    type: 'Duration',
    description: 'How long an unreachable node waits before it is declared lost. Applied only to provider pools, never to metal.',
  },
  disruptionBudget: {
    type: 'number',
    description: 'The share of the pool that remediation and rollouts may disrupt at once, from 0 to 1.',
    default: '0.05',
  },
  maxPending: {
    type: 'Duration',
    description: 'How long a reservation may stay pending before it fails with MaxPendingExceeded.',
    default: &#x22;'24h'&#x22;,
  },
  hardwareProfile: { type: 'string', description: 'The metal hardware profile whose driver bindings nodes inherit.' },
  joinAttestors: {
    type: 'string[]',
    description: &#x22;(v0.5, 1516728) The attestors that may admit a machine into the pool, by id: for example ['tpm'] to require the TPM, or ['tpm', 'x509pop']. A token join is 'join-token' and must be listed to be allowed. JoinService refuses a join or reclaim admitted by any other attestor with ACCESS_DENIED (reason pool-attestor). pools.create and pools.update refuse a list that is not 1 to 16 ids of attestors the control plane runs. Unset: any. See /docs/guides/nodes/attestation#requiring-an-attestor.&#x22;,
  },
  trustProvisioningNetwork: {
    type: 'boolean',
    description: '(v1.0, 8de6e2f) Bare metal: bind a PXE MAC that discovery proposes at once, without metal.confirmMac. Only for a provisioning network no untrusted host can reach: any host on it can claim a machine\'s UUID. The ramdisk\'s UUID and NICs are then checked against the machine. See /docs/guides/bare-metal/sites#discovering-new-machines.',
    default: 'false',
  },
  sharing: {
    type: '{ replicas?: number }',
    description: 'timeslice and mps pools: slot atoms per GPU (1 to 48, the most MPS serves; pools.create and pools.update refuse more, and inventory makes at most 48, a708aa6). Defaults are 4 for timeslice and 10 for mps. The same number reaches each host in the gpu-sharing config unit (/docs/daemon/host-config#gpu-sharing).',
  },
  hostReserve: {
    type: '{ vcpu?: number; memoryGiB?: number }',
    description: &#x22;(v1.1) What each host keeps for itself out of its bookable CPUs and memory, for pools other than vps-host. Default 2 CPUs and 8 GiB; huge page pools come out of memory on top of it.&#x22;,
  },
  perGpu: {
    type: '{ vcpu?: number; memoryGiB?: number }',
    description: &#x22;(v1.1) The host CPUs and memory a reservation books per whole GPU when it asks for no resources. Default: the smallest share any GPU host of the pool can give. See /docs/guides/capacity/reservations#host-resources.&#x22;,
  },
  hugepages: {
    type: &#x22;{ size: '2Mi' | '1Gi'; count: number; perNuma?: boolean }&#x22;,
    description: &#x22;(v1.1) Huge pages every host reserves, through the hugepages config unit (per NUMA node with perNuma). See /docs/daemon/host-config#hugepages.&#x22;,
  },
  kernelModules: {
    type: '{ load?: string[]; options?: Record<string, string> }',
    description: '(v1.1) Kernel modules every host loads, through the kernel-modules config unit. See /docs/daemon/host-config#kernel-modules.',
  },
  cpu: {
    type: &#x22;{ governor?: 'performance' | 'schedutil' | 'ondemand' | 'powersave'; isolate?: string }&#x22;,
    description: '(v1.1) The cpu config unit on every host: the cpufreq governor, and a cpulist of CPUs isolated from the kernel scheduler (never CPU 0; effective after a reboot), which pinned guests take first. See /docs/daemon/host-config#cpu.',
  },
  iommu: {
    type: '{ passthrough?: boolean }',
    description: '(v1.1) The iommu config unit: the IOMMU on for GPU passthrough to VMs (effective after a reboot); set it on vps-host pools with GPU flavors. See /docs/daemon/host-config#iommu.',
  },
  gpu: {
    type: '{ powerLimitWatts?: number; ecc?: boolean }',
    description: &#x22;(v1.1) The gpu-settings config unit: each GPU's power cap (1 to 10,000 W) and ECC mode. See /docs/daemon/host-config#gpu-settings.&#x22;,
  },
  kubernetes: {
    type: &#x22;{ apiServer?, caCertSha256?, joinTokenRef?, labels?, taints?, kubernetesVersion?, gpu?, gpuApi?: 'device-plugin' | 'dra', platformOperated? }&#x22;,
    description: &#x22;(v1.1) The pool's hosts work for a Kubernetes cluster, and reservations may use binding 'kubernetes'. apiServer, caCertSha256 and joinTokenRef together make every host join through the k8s-join unit; none of them marks a pool whose bcd runs as a DaemonSet in the cluster. gpuApi picks bcd's device plugins or its DRA driver. On a site that is not custom, the join set needs platformOperated: true (e4d14cd). platformOperated counts only on a pool the platform owns. See /docs/guides/capacity/kubernetes.&#x22;,
  },
  telemetry: {
    type: '{ otlpEndpoint?, intervalSeconds?, headersRef?, resourceAttributes? }',
    description: &#x22;(v1.1) Where the pool's hosts export bcd's metrics, over the deployment's nodeTelemetryModule default. See /docs/operations/observability#node-metrics-from-bcd.&#x22;,
  },
  migLayout: { type: 'string', description: 'mig pools: the layout the mig-layout config unit applies, such as all-1g.10gb. A change reaches each node only once its GPUs are free (the MIG layout gate, /docs/guides/capacity/fractional-gpus#the-mig-layout-gate).' },
  tenantRoot: {
    type: 'boolean',
    description: &#x22;Dedicated-root pools (v0.5): reservations may use binding 'node' to lease whole machines.&#x22;,
  },
  singleTenant: {
    type: 'boolean',
    description: &#x22;Offered to at most one tenant besides the owner, never with includeDescendants (a second offer is INVALID_STATE). VPS container runtimes, which share the host kernel, require it. pools.update refuses to turn it on while more than one other tenant, or an includeDescendants offer, holds the pool.&#x22;,
  },
  workloadIdentity: {
    type: 'boolean',
    description: &#x22;VPS pools (v0.5): hosted instances may get workload identity tokens from their host's metadata service. Off by default. See /docs/guides/vps/instances#workload-identity.&#x22;,
  },
  admission: {
    type: &#x22;{ budgetCheck?: 'none' | 'meter' }&#x22;,
    description: &#x22;'meter': reservations.create, reservations.extend and instances.create call IAM billing.check after the decision and refuse a spent budget with SPEND_LIMIT_REACHED. An offer's onBudgetExceeded stop-new or drain turns the same check on.&#x22;,
  },
}"
/>

The placer reads the overcommit ratio stored on each `node_resources` row, not the policy. For `vps-host` pools the
vps module's `vps.hosts` work writes `vcpu`, `memoryGiB` (less the huge page pools) and `diskGiB` from each host's
inventory with the pool's ratio (default 1, clamped to 0.1 to 100). For every other pool, since 8a7f8eb, capacity's
`capacity.hosts` reconciler (every 30 seconds) writes `vcpu` (logical CPUs less `hostReserve.vcpu`, with the ratio)
and `memoryGiB` (installed memory less the huge page pools and `hostReserve.memoryGiB`, with the ratio), and for every
pool `hugepages2Mi` and `hugepages1Gi` (the configured pages, never overcommitted). `tenant-controlled` nodes get no
rows, a row never shrinks below what is allocated, and unchanged rows are not rewritten. GPU reservations book these
rows with their devices (see [host resources](/docs/guides/capacity/reservations#host-resources)).

The host policy fields (`hugepages`, `kernelModules`, `cpu`, `iommu`, `gpu`, `kubernetes`, and a `sharing` mode of
`mps` or `timeslice`) become config units in every host's NodeSpec, through provision (see
[host configuration](/docs/daemon/host-config#where-the-units-come-from)). Since 37dcd02 `pools.create` and
`pools.update` (when it changes the policy) run the modules' `pool-policy` validators before writing, and provision
registers one that checks these fields as nodes would get them, so a bad value is `INVALID_INPUT` at the API; a
validator's failure that is not a compute error reads `policy: ...`. The server adds its own validator for
`joinAttestors` (1516728), which knows the attestors JoinService runs: `join-token` and every registered one.
`admission.budgetCheck: 'meter'` adds a Better IAM
`billing.check` to every reservation and instance admission in the pool, on top of the IAM policy described in
[quotas and usage](/docs/guides/capacity/quotas-and-usage).

## Create a pool [#create-a-pool]

[`pools.create`](/docs/reference/api/pools#create) is authorized as `compute:pools:create` on the
tenant, and the site must belong to that tenant. Platform pools are therefore created by the root
tenant on its `shared` or `dedicated` sites, and a customer creates pools on its own `custom` site.

```ts title="create-pool.ts"
const pool = await compute.api.pools.create(rootCredential, {
  tenantId: 'root',
  siteId: 'site_01k…',
  class: 'metal-host',
  gpuModel: 'H100',
  sharing: 'exclusive',
  policy: {
    acceptance: [{ kind: 'diag.nvml' }],
    maxPending: '6h',
  },
  idempotencyKey: 'pool-fra1-h100',
});
```

* `ownerQuota` sets the owner's own quota rows. It defaults to `{ gpu: 2147483647 }`, effectively
  unlimited whole GPUs. Owners of `mig`, `timeslice` or `mps` pools must add `gpu.mig` or `gpu.slot`
  rows themselves, because a missing row grants 0.
* With `idempotencyKey`, the pool id is derived from the tenant, the caller and the key, as for
  reservation drafts. A retry returns the same pool; the same key with a different site, class,
  sharing mode, agent or provider account is `CONFLICT`. [Fleet config](/docs/guides/fleet-config)
  uses this to plan pools it has not created yet.
* `class: 'provider'` needs `providerAccountId` and `agent: 'none'`; other classes refuse
  `providerAccountId`. Both are `INVALID_INPUT`.
* A site the tenant does not own is `NOT_FOUND`.

The pool starts with an empty rollup and emits `pool.created`. Machines join it with a join token (see
[joining](/docs/guides/nodes/joining)).

## Offerings [#offerings]

An offering is a `pool_offers` row. It makes the pool exist in another
tenant's view and creates that tenant's quota rows in the same transaction.

```ts title="offer.ts"
await compute.api.pools.offer(rootCredential, {
  tenantId: 'root', // the pool's owner, acting
  poolId,
  toTenantId: 'acme',
  quota: { gpu: 32 },
  priority: 10,
});
```

* [`pools.offer`](/docs/reference/api/pools#offer) is `compute:pools:share` on the pool, and only the
  owner can call it (anyone else gets `NOT_FOUND`). Offering again updates the row and the quota.
* `quota` maps resources to limits: `gpu`, `gpu.mig`, `gpu.slot`, or a scalar such as `vcpu`. A
  resource without a quota row grants 0.
* `includeDescendants` and `priority` (-1000 to 1000, default 0) are stored with the offer and
  round-trip through fleet config. In this release the capacity module matches the offered tenant
  exactly and the placer does not read `priority`.
* The **offering validator** refuses, with `INVALID_STATE`, to offer a `timeslice` or `mps` pool, or a
  pool that holds `tenant-owned` nodes, to anyone but its owner. A `policy.singleTenant` pool can be
  offered to one tenant besides its owner, and never with `includeDescendants`. An archived pool cannot
  be offered either.

The offered tenant then sees the pool in [`pools.list`](/docs/reference/api/pools#list) and
[`pools.get`](/docs/reference/api/pools#get), each pool with `owned`, `archived` and the tenant's own
quota rows:

```json title="pools.list, as acme sees it"
{
  "items": [
    {
      "id": "pool_01k…",
      "ownerTenantId": "root",
      "siteId": "site_01k…",
      "class": "metal-host",
      "gpuModel": "H100",
      "sharing": "exclusive",
      "agent": "bcd",
      "policy": { "acceptance": [{ "kind": "diag.nvml" }], "maxPending": "6h" },
      "version": 1,
      "owned": false,
      "archived": false,
      "quotas": [{ "tenantId": "acme", "poolId": "pool_01k…", "resource": "gpu", "max": 32, "used": 4, "available": 28 }]
    }
  ]
}
```

[`pools.unoffer`](/docs/reference/api/pools#unoffer) removes the offer and answers
`{ removed: boolean }`. It lowers the tenant's quotas on the pool to what is in use, so live
reservations keep their bookings and charge while nothing new fits. Pending reservations of that
tenant fail with `PoolGone` at their next retry. A pool cannot be unoffered from its owner.

Offering also writes the pool's row in the IAM authz projection
for the tenant, with the attributes `gpuModel`, `class`, `sharing`, `region` and `owned`, so tenant
policies can decide on them (see [actions and resources](/docs/guides/iam/actions-and-resources)).

## Update and archive [#update-and-archive]

[`pools.update`](/docs/reference/api/pools#update) (`compute:pools:manage`, owner only) takes a new
`policy`, a new `gpuModel`, or both. The policy is replaced as a whole, not merged, so send every field
you want to keep. The update refreshes the pool's authz row for the owner and every offered tenant and
emits `pool.updated`. Class, site, sharing mode and agent cannot change.

[`pools.archive`](/docs/reference/api/pools#archive) (`compute:pools:manage`, owner only) retires a
pool without disturbing what runs on it:

* every offer is removed, each offered tenant's quotas drop to what is in use, and their authz rows
  are removed;
* the pool takes no new work: new reservations are `NOT_FOUND`, and the placer answers `pending` with
  `the pool is archived`;
* placed and active reservations keep their bookings until they end, while pending ones fail with
  `PoolGone` at their next retry;
* only the owner still sees the pool, with `archived: true`.

Archiving is idempotent and emits `pool.archived` with the list of unoffered tenants.

When Better IAM deletes a tenant, compute's tenancy teardown (`capacityTenantDeleted`) archives the pools
it owns the same way (`pool.archived` with `reason: 'tenant deleted'`, quotas of the offered tenants
zeroed), withdraws every offer made to it (`pool.unoffered` to the owner) and ends its reservations. Other
tenants' live bookings in an archived pool run to their end. See
[tenancy](/docs/guides/iam/tenancy#suspended-and-deleted-tenants).

## The pools API [#the-pools-api]

| Method                                                 | Action                  | Decided on | Phase |
| ------------------------------------------------------ | ----------------------- | ---------- | ----- |
| [`pools.list`](/docs/reference/api/pools#list)         | `compute:pools:read`    | tenant     | v0.1  |
| [`pools.get`](/docs/reference/api/pools#get)           | `compute:pools:read`    | pool       | v0.1  |
| [`pools.create`](/docs/reference/api/pools#create)     | `compute:pools:create`  | tenant     | v0.1  |
| [`pools.capacity`](/docs/reference/api/pools#capacity) | `compute:capacity:read` | pool       | v0.1  |
| [`pools.offer`](/docs/reference/api/pools#offer)       | `compute:pools:share`   | pool       | v0.1  |
| [`pools.unoffer`](/docs/reference/api/pools#unoffer)   | `compute:pools:share`   | pool       | v0.1  |
| [`pools.update`](/docs/reference/api/pools#update)     | `compute:pools:manage`  | pool       | v0.2  |
| [`pools.archive`](/docs/reference/api/pools#archive)   | `compute:pools:manage`  | pool       | v0.2  |

All eight are served today. Events: `pool.created`, `pool.offered`, `pool.unoffered`, `pool.updated`,
`pool.archived`, and `quota.updated` from `quotas.set`. `better-compute pools list --tenant acme`
lists pools from the [CLI](/docs/reference/cli).