better/compute
IAM integration

Tenancy

How Better Compute maps onto Better IAM tenants, with the root tenant as the platform, pools offered to organizations, and projects inside a tenant.

Better Compute has no tenant table of its own. A is a Better IAM tenant id, every compute record carries the id of the tenant that owns it, and every method takes a tenantId. Better IAM's tenant tree gives compute its structure:

  • The root tenant is the platform. It owns the shared fleet and decides fleet-wide operations.
  • Organization tenants are customers. They own their reservations, instances and projects, and any hardware they bring.
  • Child tenants below an organization exist only when you need billing isolation.

Inside a tenant, compute's group work without creating more tenants.

What lives where

Root tenant (platform)Organization tenant
Platform sites (shared and dedicated), their pools and nodesReservations and instances
of platform pools to organizationsProjects
Public images and profilesPrivate images and profiles
Platform provider accounts and secretscustom sites with their pools and tenant-owned nodes (bring your own hardware)
Billing meters and fleet rolesWebhooks, tenant secrets and tenant provider accounts

The method table encodes this split. sites.create decides on compute-fleet for shared and dedicated sites, which must name the root tenant (another tenantId is INVALID_INPUT), and on compute-tenant for an organization's custom site. pools.create decides on compute-fleet for a platform pool and on compute-tenant for a pool on the organization's own custom site. See Actions and resources.

The root tenant

Tell compute which Better IAM tenant is the root, either as betterCompute({ rootTenantId }) or through the port with bridge.embedded(iam, { rootTenantId }). When both are given they must agree, and a missing or malformed id is INVALID_CONFIG.

The root tenant is special in three ways:

  • compute-fleet/global resolves only in the root tenant, so fleet-wide methods (system.selfCheck, quotas.set, fleet.*, rollouts.*, platform sites.create and pools.create) are decided there, whatever tenant their input names.
  • Root-tenant callers pass compute's cross-tenant pre-check for any tenant. Better IAM then decides: it allows a root-tenant caller into another tenant only through its root override, which needs root authority and an MFA-verified user session, and it audits every override. Platform support on a customer's objects happens only this way.
  • The billing meters live in the root tenant, and Better IAM resolves them for every tenant below it.

Offering pools to organizations

A platform pool exists in an organization's view only once the platform offers it. pools.offer writes a pool_offers row with the organization's quota and priority, and a projection row that makes the pool resolve in that tenant:

offer.ts
await compute.api.pools.offer(rootOperator, {
  tenantId: rootTenantId, // the pool's owner
  poolId,
  toTenantId: acmeTenantId,
  quota: { gpu: 16 }, // per resource; a resource left out grants 0
  priority: 10, // -1000 to 1000, default 0
});
  • The call decides compute:pools:share on the pool. Offering again updates the quota and priority.
  • A pool holding tenant-owned nodes, or a single-tenant sharing mode (timeslice, mps), can be offered only to its owner. A pool with policy.singleTenant: true (required for VPS container runtimes) can be offered to one tenant besides its owner, without includeDescendants; another offer is INVALID_STATE.
  • pools.unoffer removes the pool from the organization's view and drops its quota to zero. Live reservations keep their charge; nothing new fits. A pool cannot be unoffered from its owner (INVALID_INPUT).
  • includeDescendants: true is stored on the offering, but this release writes the projection row only for toTenantId, so child tenants do not see the pool yet. PLAN.md expands offerings to descendants from a tenant tree cached from iam:tenants:* events; until then, offer the pool to each child tenant that needs it.

Organizations then share the pools they were offered with their own teams through Better IAM relationships. See Pool relations and Pools.

Visibility is the projection

What a tenant can see is exactly what compute's holds in its view: one row per resource type, id and tenant. List queries filter on the same rows, and the resolver answers only from them.

ResourceThe views it exists in
Platform poolThe root tenant, and each tenant it is offered to
Pool on a custom siteIts owner tenant
NodeIts owner tenant
SiteIts owner tenant
Reservation, instance, projectIts owner tenant
ImageIts owner tenant. images.promote writes a row under the wildcard tenant *, PLAN.md's public row; the resolver in this release matches a row's tenant exactly.

Inside your own tenant, an id that is not in your view resolves to NOT_FOUND. An id in another tenant never gets that far: compute refuses the caller with ACCESS_DENIED before reading its database.

Projects

A project is an app-owned grouping inside one tenant, stored in compute's projects table:

projects
CREATE TABLE projects (
  id text PRIMARY KEY,           -- prj_...
  tenant_id text NOT NULL,
  name text NOT NULL,            -- unique in the tenant
  state text NOT NULL,           -- active | archived
  version bigint NOT NULL,
  UNIQUE (tenant_id, name)
);
projects.ts
const project = await compute.api.projects.create(credential, {
  tenantId,
  name: 'ml-research', // a DNS label, at most 63 characters
  idempotencyKey: 'create-ml-research',
});
  • projects.create decides compute:projects:manage on compute-tenant/{t}. It is idempotent per tenant, caller and key: the id derives from them, the same request again returns the project without a second decision, and the same key with a different name is CONFLICT. A second project with the same name is CONFLICT too.
  • The creator becomes the project's ownerId in the projection. Membership is a Better IAM relationship on compute-project/{id}, with the relations member and admin.
  • projects.get decides compute:projects:read on the project; projects.list follows the list recipe and filters by state.
  • projects.archive (v0.2) decides compute:projects:manage on the project. It sets state to archived, bumps version and publishes project.archived (projectId, name); archiving an archived project returns it unchanged, with no event. Archiving only changes the state: compute does not check it when a reservation or instance names the project.

Reservations and instances take an optional projectId, which becomes the projectId attribute of their resource and the project tag of their usage windows. usage.list with a projectId decides compute:usage:read on the project instead of the tenant, so project members can read their project's usage:

Project members read their project and its usage
{
  "sid": "ProjectMembers",
  "effect": "allow",
  "actions": ["compute:projects:read", "compute:usage:read"],
  "resources": ["compute-project/*"],
  "conditions": { "ArrayContains": { "resource.relations": "member" } }
}

Compute checks only that a reservation's projectId is a prj_ id. A reservation's parent is its pool, not its project, so a policy cannot test project membership when it decides reservations.create. Restrict which projects a group may charge with resource.projectId conditions instead.

Child tenants for billing isolation

Create a Better IAM child tenant under an organization only when a unit needs its own bill. Better IAM grants nothing downward, so a child tenant needs its own role bindings and, in this release, its own pools.offer. Everything else about the organization's work fits in projects.

Quotas, flags and residency

  • Quotas live in compute, because Better IAM's tenant limits are a fixed set. They are per tenant, pool and resource (gpu, vcpu, and so on), set by offerings and by quotas.set (v0.2, decided on compute-fleet), and read with quotas.get. An admission that would exceed one fails with QUOTA_EXCEEDED. See Quotas and usage.
  • Flags such as compute.h100 are Better IAM feature flags. Policies test them through tenant.features, and code reads them through IamPort.featureEnabled, backed by a cache that holds each tenant and key for up to 5 minutes and drops entries on feature:* events.
  • Residency waits for Better IAM to expose a tenant region. The offerings table already has an allowed_regions column, but pools.offer does not accept it and nothing enforces it yet. Until then, restrict regions in policy with resource.region, as the gpu-user example does.

Suspended and deleted tenants

Compute mirrors the Better IAM tenants that are not active in its own compute_tenant_states table, owned by the built-in tenancy module (migration tenancy/0001_tenant_states):

compute_tenant_states
CREATE TABLE compute_tenant_states (
  tenant_id text PRIMARY KEY,
  state text NOT NULL,           -- active | suspended | deleted
  reason text,                   -- the IAM action that changed it
  updated_at timestamptz NOT NULL,
  teardown_done_at timestamptz   -- set once every module released a deleted tenant
);

The IAM reaction

The rows are written by the tenant reaction of the IAM inbox, on iam:tenants:update and iam:tenants:delete. The server ships the handler, tenantStateHandler, and better-compute serve installs it as the default tenant reaction when the configuration's host sets host.iam. Give it host.iam.readTenants so suspensions and deletions cascade (anything but a function is INVALID_CONFIG). iamTenantReader from @better-compute/iam builds one over a Better IAM instance or client:

better-compute.config.ts
import { iamTenantReader } from '@better-compute/iam';

export const host = defineHost(async () => ({
  iam: {
    bridge,
    // The tenant and every tenant below it, with effective statuses, read as root.
    readTenants: iamTenantReader(iam, { credential: rootCredential }),
  },
}));

iamTenantReader(iam, { credential, maxTenants? }) uses only tenants.get and tenants.listChildren, so iam can be either of:

  • Embedded: betterIam()'s result (TenantReaderHost); the credential is the first argument of each call.
  • Remote: a Better IAM HTTP client, createIamClient() from @better-iam/client (TenantReaderClient); the credential goes into each call's headers. A credential IAM refuses fails the read; it never reads as missing.
Remote mode
import { createIamClient } from '@better-iam/client';

const readTenants = iamTenantReader(createIamClient({ baseURL: 'https://iam.example.com' }), {
  credential: rootCredential,
});

Either way:

  • credential is a root credential (compute's service account in the IAM root tenant), or a function returning one: only root reads suspended and deleted tenants and every tenant's children.
  • Each status is the tenant's effective one, the worst of its own and its ancestors' (active < pending < suspended < deleted < missing). Better IAM writes a deletion into every descendant's own status, but a suspension only into its decisions, so a child of a suspended tenant reads as suspended here.
  • A tenant IAM no longer knows is missing, and only IAM's own "Tenant not found" answer means that. Any other error, a bare 404 from a proxy or a wrong base path included, throws and is retried, so it can never read as a deleted tenant. A subtree larger than maxTenants (default 10,000) is refused with INVALID_STATE.

better-compute dev --sim with the embedded Better IAM wires iamTenantReader in by itself, so suspending or deleting a tenant there reaches its whole subtree.

host.iam.handlers.tenant replaces the default. A server you build yourself adds the handler to what it gives bridge.setHandlers(), which replaces the whole set:

compute.server.ts
import { iamTenantReader } from '@better-compute/iam';
import { tenantStateHandler } from '@better-compute/server';

bridge.setHandlers({
  ...handlers, // agent, identity, sessions, ...
  tenant: tenantStateHandler(compute, {
    readTenants: iamTenantReader(iam, { credential: rootCredential }),
  }),
});
  • readTenants(tenantId) returns { tenantId, status } for the tenant acted on and its descendants, with status one of active, pending, suspended, deleted or missing (IAM no longer knows it). A suspension or deletion cascades to descendants, and the audit event names only the tenant acted on, so compute re-reads the subtree instead of trusting the event. suspended maps to suspended; deleted maps to deleted; anything else is active. missing only confirms a tenant already recorded as suspended or deleted: it never starts a teardown by itself (so an id IAM audits under iam:tenants:update without being a tenant, such as an invitation's, is never torn down). The root tenant is never suspended or deleted, whatever IAM answers.
  • Without a reader, only iam:tenants:delete is applied, and only to the named tenant; an update changes nothing. The handler queues subtree syncs only when it was given a reader, and a sync key that reaches a process without one fails with UNAVAILABLE and is retried with back-off rather than dropped.
  • Each change publishes tenant.state-changed (tenantId, state, previous, reason) on tenant/{id}. A replay changes nothing, and a tenant reactivated in IAM becomes active again.

No IAM call in the inbox transaction

The handler runs inside the IAM inbox's drain transaction, which holds its inbox rows FOR UPDATE, so it makes no IAM call there. It only writes SQL:

  1. On iam:tenants:delete, the named tenant is recorded as deleted at once, no IAM read needed, and refused from that commit on.
  2. On every update and delete, it queues the key sync:<tenantId> for the tenancy reconciler, and it hands readTenants to compute.tenants.setReader().

The reconciler then reads the subtree through the reader outside any transaction (syncTenantStates) and applies it in a transaction of its own (reason iam:tenants:sync). When IAM cannot answer, the read throws and the key is retried with back-off: a suspension or deletion is delayed while IAM is down, never dropped, and compute never fails open. As a backstop, the tenancy.resync reconciler, separate so it never delays syncs and teardowns, re-reads the whole tree from the root tenant every 15 minutes (reason iam:tenants:resync); past the reader's size limit it re-reads only the tenants already refused.

What the envelope refuses

Every call passes a tenant gate right after the cross-tenant pre-check, before compute's database or IAM decision:

Tenant stateReadsMutations
activeallowedallowed
suspendedallowedTENANT_SUSPENDED (403)
deletedTENANT_SUSPENDED (403)TENANT_SUSPENDED (403)

The gate checks the caller's tenant and the tenants the call acts in; the error's details carry tenantId and state. Root-tenant callers may still act in a suspended tenant and read a deleted one, but never write into a deleted tenant, since what they created after the teardown would never be torn down. The gate answers from memory: each process loads the non-active states at start, or on its first call when it never ran initialize() (an API-only replica started with --no-migrate; compute.tenants.ready() loads it up front), and reloads them every 5 seconds. The process that applied a change reloads before its next call (compute.tenants.invalidate(), a generation counter, so a load already in flight cannot swallow it). Another process may therefore admit a suspended tenant's mutation for up to 5 seconds. Operations built without a deployment (unit harnesses) get noTenantGate, which refuses nothing. A reload that fails is reported once per run of failures, not every 5 seconds, and the timer stops once the store is closed (a one-shot CLI command closes it without stopping compute).

Teardown of a deleted tenant

A deletion enqueues the tenancy reconciler for the tenant. It calls every hosted module's tenantDeleted hook, plugins included; a module that answers SYNC_AGAIN is asked again by the 30-second sweep, which also recovers a lost work item and retries the longest-waiting teardowns first (teardown_attempted_at, migration tenancy/0002_teardown_attempts). When every hook answers DONE in one pass, the reconciler sets teardown_done_at and publishes tenant.torn-down. The root tenant is never torn down, whatever a row says.

ModuleWhat it releases (idempotent, in batches of 100)
capacityUnstarted reservations cancelled, started ones released now (bookings and quota back); offers to the tenant withdrawn (quotas zeroed, pool.unoffered to the owner); pools it owns archived, with every offer withdrawn (pool.archived), while other tenants' live bookings there run to their end; its idle quota rows deleted. A dedicated-root lease is released this way too.
provisionActive workflows aborted (their cleanup still runs); private images and profiles soft deleted with their authz rows; image signers revoked. Images it promoted to public stay, for the tenants that deploy them.
metalEvery bare-metal node it owns retired, one legal transition per sweep: ready, adopted, enrolling and failed retire at once; deployed and rescued machines are released (wiped) first; a machine mid-workflow is aborted (it fails with TENANT_DELETED), then retired; releasing and retiring machines finish first.
vpsAdmission drafts dropped; every live instance set to desiredState: 'deleted' (100 per pass, instance.deleting with reason tenant deleted) and torn down through the ordinary lifecycle: runtime teardown, inline allocation released, IP freed, quota refunded. SYNC_AGAIN until every instance is deleted.

Not torn down yet

Only capacity, provision, metal and vps implement tenantDeleted in this release. A deleted tenant's non-metal tenant-owned nodes and its sites are not released by the teardown; the gate only stops the tenant from reaching them. While its instances wait for teardown, their VMs cannot mint workload identity tokens (the workload broker refuses a suspended or deleted tenant's instances with disabled), and the secret broker releases none of the tenant's secrets or user data (security.secret.refused, reason tenant-suspended or tenant-deleted). Plugins release their own state through the same hook; see modules.

Was this page helpful?

Better Compute is created by Sean Filimon

Last updated

On this page