# Modules (/docs/extending/modules) > A ComputeModule bundles API groups, migrations, reconcilers, command kinds and NodeSpec sections. How to write one, what its context offers, and how the host runs it. A module is the unit of extension. Everything Better Compute does beyond its envelope is a module: `capacity` and `inventory` are built in, and `events`, `fleet`, `rollout`, `audit`, `iam`, `vps`, `metal` and `provision` are modules you pass in. Yours use the same contract, `ComputeModule` from `@better-compute/core`, and reach the rest of the system only through the ports on their context. ## What a module declares [#what-a-module-declares] ```ts title="@better-compute/core" interface ComputeModule { name: Name; requires?: string[]; migrations?: Migration[]; extensions?: Extension[]; api?(context: ModuleContext): A; reconcilers?: Reconciler[]; commandKinds?: CommandKindSpec[]; nodeSpec?(context: ModuleContext, node: NodeRecord): Promise; statusHandlers?: readonly ModuleStatusHandler[]; catalog?: { actions?: string[]; resourceTypes?: Record }; commands?: CliCommandSpec[]; profiles?: ProfileDocument[]; selfCheck?(context: ModuleContext): Promise; start?(context: ModuleContext): Promise; stop?(): Promise; tenantDeleted?(context: ModuleContext, tenantId: string): Promise<'DONE' | 'SYNC_AGAIN'>; } ``` `defineModule(module)` returns its argument unchanged; it only keeps the name and API types literal, so the typed client and `compute.api` know your groups. ## A complete example [#a-complete-example] This plugin is the one the server's own tests register (`tests/support/server.ts`), trimmed and given a reconciler and a self-check: ```ts title="acme-widgets.ts" import { definePlugin, field, manifest } from 'better-compute'; import { ComputeError, defineMethod, type Finding } from 'better-compute/core'; const widgetInput = manifest({ tenantId: field.string({ pattern: '^[A-Za-z0-9._:-]{1,128}$' }), widgetId: field.string({ pattern: '^[a-z0-9-]{1,64}$' }), }); type WidgetInput = { tenantId: string; widgetId: string }; export const acmeWidgets = () => definePlugin({ id: 'acme', catalog: { actions: ['compute:acme:poke', 'compute:acme:read'] }, migrations: [ { name: 'acme/0001_widgets', sql: `CREATE TABLE plg_acme_widgets ( id text PRIMARY KEY, tenant_id text NOT NULL, created_at timestamptz NOT NULL DEFAULT compute_now())`, }, ], api: () => ({ widgets: { poke: defineMethod({ action: 'compute:acme:poke', kind: 'mutation', resource: (input) => ({ type: 'tenant', id: input.tenantId }), validate: (input) => widgetInput.parse(input), async handler(ctx, input) { await ctx.sql.query('INSERT INTO plg_acme_widgets (id, tenant_id) VALUES ($1, $2)', [ input.widgetId, input.tenantId, ]); await ctx.events.publish(ctx.sql, { type: 'acme.widget-poked', tenantId: input.tenantId, subject: `widget/${input.widgetId}`, data: { widgetId: input.widgetId }, }); await ctx.work.enqueue(ctx.sql, 'acme.widgets', input.tenantId); return { poked: input.widgetId }; }, }), }, }), reconcilers: [ { name: 'acme.widgets', shardBy: 'none', periodic: [{ key: 'sweep', every: '10m' }], async run(ctx, key) { // Level-triggered: look at the state for `key` and converge it. Safe to run twice. return 'DONE'; }, }, ], async selfCheck(ctx): Promise { const { rows } = await ctx.sql.query<{ n: string }>('SELECT count(*)::text AS n FROM plg_acme_widgets'); return Number(rows[0]?.n) > 1_000_000 ? [{ check: 'acme.widgets', severity: 'warning', message: 'Over a million widgets', fix: 'Prune old widgets' }] : []; }, }); ``` The method's handler runs inside the envelope's transaction, so the row, the event and the work item commit together or not at all. Its HTTP route is `POST /api/compute/plugins/acme/widgets/poke`; see [methods](/docs/extending/methods). ## Registering modules [#registering-modules] ```ts title="compute.ts" import { betterCompute } from 'better-compute'; import { eventsModule } from 'better-compute/events'; import { createComputeIamBridge } from 'better-compute/iam'; import { acmeWidgets } from './acme-widgets'; const plugins = [acmeWidgets()]; const bridge = createComputeIamBridge({ rootTenantId, plugins }); // merges the plugin's catalog export const compute = betterCompute({ // …database, secret, baseURL iam: bridge.embedded(iam, { rootTenantId }), modules: [...bridge.modules, eventsModule({ secret })], plugins, }); await compute.initialize(); ``` `betterCompute()` registers, in this order: the server's own groups, the built-in `node-controllers` and `site-boot` modules, `inventory`, `capacity`, the `tenancy` module (suspended and deleted tenants), the `quorum` module (the v1.0 destructive-batch quorum's batches and `commandBatches`, always hosted), a `pki` entry carrying the sealed-secrets migration when the default secret store is in use, the `tunnels` module when `tunnels` is configured, the `audit-sinks` module when any audit sink is listed (see [sinks](/docs/extending/sinks#audit-sinks)), then your `modules`, then your `plugins`. The order is the order of migrations, of `start()` calls and of NodeSpec merging. The `extensions` a module or plugin lists are read like the top-level options: drivers and attestors are found in either place, scheduler plugins join `scheduler.plugins` in the placer, and event and audit sinks join `sinks.events` and `sinks.audit`. A `pool-policy` extension (`PoolPolicyValidator`, 37dcd02) checks the parts of a pool policy its module reads: `validate(policy, { class, sharing })` runs in `pools.create` and in `pools.update` when the policy changes, before the write, and refuses by throwing a `ComputeError`; any other error becomes `INVALID_INPUT` `policy: ...`. Provision registers one for its host units. A usage sink, in `sinks.usage` or in `extensions`, is `INVALID_CONFIG`: usage goes to IAM billing through the IAM bridge. Registration checks everything it can without the database and throws `INVALID_CONFIG`: * module names are unique and well formed, and every `requires` entry is registered; * every migration of a module is named `/…`, and the whole list passes `assertMigrations`; * extensions are built with a `define*` helper and unique by `kind:id@version`; * command kinds are unique, including against the server's built-in kinds; * routes are unique, group and method names match `^[A-Za-z][A-Za-z0-9_-]{0,63}$` and are not `then`, `catch`, `finally`, `toJSON`, `constructor`, `prototype`, `__proto__` or `plugins`; * catalog actions match `compute::`; a plugin's actions must be `compute::`. `initialize()` then checks every method against the merged catalog and the method table of PLAN.md §9.2, migrates the store (its base schema first, then module and plugin migrations in order), anchors the clock, and awaits each module's `start()` in registration order. If a `start()` throws, the modules already started are stopped in reverse and the error is rethrown. `stop()` stops the worker loops first, then the modules in reverse order. > **Pass the catalog to IAM too.** A module's or plugin's `catalog` must reach every Better IAM host, not only the compute server. Pass the same modules and plugins to `createComputeIamBridge({ modules, plugins })`, which merges their fragments into the catalog IAM decides with. See [actions and resources](/docs/guides/iam/actions-and-resources). ## Plugins [#plugins] A plugin is a module with an `id`, built with `definePlugin`. Its name defaults to the id. Three things differ from a module: | | Module (`modules`) | Plugin (`plugins`) | | ---------- | ------------------------------------------- | ----------------------------------------------- | | Routes | `POST {basePath}/{group}/{method}` | `POST {basePath}/plugins/{id}/{group}/{method}` | | Server API | `compute.api..` | `compute.api.plugins...` | | Actions | Must match its row in the §9.2 method table | Must be `compute::` | A non-plugin method with no §9.2 row is an `undeclared` problem: `initialize()` still succeeds, `better-compute doctor` reports a `catalog.undeclared` warning, and `scripts/authz-matrix.mjs` fails. The method table is normative for Better Compute's own surface, so **third-party code ships as plugins**. By convention a plugin's tables are named `plg__*`; nothing enforces it, but it keeps plugin tables apart from compute's own. ## Migrations [#migrations] ```ts title="@better-compute/core" interface Migration { name: `${string}/${string}`; // /_ sql: string | ((tx: SqlTx) => Promise); } ``` A name matches `^([a-z][a-z0-9-]{0,63})/(\d{4})_[a-z0-9][a-z0-9_-]{0,95}$`, and the owner must be the module's own name: `rollout/0001_rollouts`, `events/0001_webhooks`, `acme/0001_widgets`. Within an owner the numbers must strictly increase. Names are checked before anything runs. The store applies them after its base schema (owner `core`). Each migration runs in its own transaction and is recorded in the append-only `compute_migrations` ledger; concurrent migrators serialize on an advisory lock and skip what another applied. So a failed migration leaves nothing behind, and an applied one is never run again. Never edit a migration that has shipped; add the next number instead. Use `compute_now()` for defaults, never `now()`. ## Reconcilers [#reconcilers] ```ts title="@better-compute/core" interface Reconciler { name: string; shardBy: 'node' | 'pool' | 'none'; run(context: ReconcileContext, key: string): Promise<'DONE' | 'SYNC_AGAIN'>; periodic?: readonly { key: string; every: Duration }[]; role?: 'controllers' | 'scheduler'; // default 'controllers' } ``` A reconciler's name is also its work kind: `ctx.work.enqueue(tx, 'acme.widgets', key)` wakes it for `key`. Names match `^[a-z][a-z0-9._-]{0,63}$`, are unique across modules, and may not be `command-sweeper`, `outbox-dispatch` or `event-subscribers`, which the runtime reserves. Keys are non-empty strings of at most 512 characters. Runs are level-triggered. The work queue coalesces: enqueuing a key that is already queued bumps its generation instead of adding a row, and a run that finishes deletes the key only at the generation it claimed. A key dirtied again while its reconciler runs survives, so no wake-up is lost. | Outcome | What happens to the key | | -------------- | ------------------------------------------------------------------------------------- | | `'DONE'` | Removed; for a periodic key, due again `every` after this run | | `'SYNC_AGAIN'` | Due again in 1 second | | a throw | Due again after a back-off of 1 s, doubling to at most 5 minutes; logged as a warning | `shardBy` decides which worker runs a key. Every key lands on shard `shardOf(key)`, an FNV-1a hash modulo 256. * **`node`** and **`pool`**: keys are node or pool ids. Each worker runs the keys of the shards it holds `shard:` leases for. Every 5 minutes plus up to 5 minutes of jitter (`worker.resyncMs`), it re-enqueues every node or pool of its shards, so a missed wake-up heals. * **`none`**: one worker at a time runs the reconciler, under a `singleton:` lease. `role` picks the worker role that runs it (v0.4). The default, `controllers`, suits almost everything. Placers declare `scheduler`, as the pool-lease placer `capacity.placer` does: they then run only in processes started with the `scheduler` role (`serve --roles scheduler`), under that role's own rendezvous membership (worker rows `scheduler/`, leases `scheduler/shard:`), so placement scales apart from the controllers. A role joins a shard membership only when it runs a sharded reconciler of its own, so a scheduler process with no placer registered holds no shard leases. `serve` with the default `--roles all` runs both roles. `periodic` keys are seeded once and then come back `every` after each run (at least `1s`), besides any edge enqueue. Every transaction a reconciler opens through `ctx.sql.transaction` begins by fencing its lease, so a worker that lost the lease cannot commit. `ctx.shard` and `ctx.lease` say what the run holds, and `ctx.signal` aborts on `stop()`. Reconcilers from the built-in modules, for reference: | Reconciler | Module | `shardBy` | Periodic | | --------------------------------------------------------------------- | ---------------------------------------- | --------- | ------------------------------------------------------------------------------------------------ | | `capacity.reservations` | capacity | `none` | edge-triggered, with `notBefore` wake-ups | | `capacity.placer` (role `scheduler`) | capacity (`placer: 'pool-lease'`) | `pool` | | | `capacity.rollup`, `capacity.usage` | capacity | `none` | `all` every `5m` | | `capacity.mig` | capacity (the MIG layout gate) | `node` | edge-triggered, with `notBefore` wake-ups | | `inventory.hygiene` | inventory | `none` | `all` every `5m` | | `tenancy` | tenancy (built in) | `none` | `sweep` every `30s`, plus one key per deleted tenant | | `quorum.batches` | quorum (built in) | `none` | `expire` every `1m`: pending batches past their end expire and their held commands are cancelled | | `tunnels.ssh-certificates-gc`, `tunnels.ssh-certificates-reauthorize` | tunnels (with `tunnels`) | `none` | `ssh-certificates` every `1h`, `1m` | | `events.fanout`, `events.deliver`, `events.prune` | events | `none` | every `1s`, `2s`, `1h` (`events.prune` only unless `retention: false`) | | `audit-sinks` | audit-sinks (built in, with audit sinks) | `none` | `tick` every `5s` | | `fleet.drift` | fleet | `none` | every `15m` by default | | `rollout` | rollout | `none` | `all` every `30s` | | `audit.checkpoint` | audit | `none` | `checkpoint` every `1h` | | `iam.inbox`, `iam.reconcile`, `iam.usage` | iam | `none` | `2s`, `5m`, `1h` | | `vps.instances`, `vps.bootstraps` | vps | `none` | edge-triggered, with `notBefore` wake-ups | | `vps.hosts` | vps | `none` | `all` every `30s` (`vps({ hostSyncEvery })`) | | `vps.sandboxes` | vps | `none` | `all` every `15s`: sandbox expiry, `sandbox.ready` and `sandbox.deleted`, warm pools | | `vps.usage` | vps | `none` | `all` every `5m` | All but `capacity.placer` run in the `controllers` role. Tests and the simulator step them without timers: `await compute.jobs['acme.widgets'].tick()` runs one batch. ## Command kinds [#command-kinds] A module that sends work to agents registers the kinds it sends. `ctx.commands.enqueue` refuses a kind nobody registered. For fleet-wide destructive work (v1.0), the queue also takes batches: `ctx.commands.createBatch?(tx, { tenantId, scope, reason, notAfterMs?, createdBy? })` stores a batch of `{ subject, kind }` leaves and answers `{ batchId, held, required, size }`, and `enqueue(tx, { ..., batchId })` puts a command into it; the command's `(subject, kind)` must be in the scope, and it is held until operators approve the batch. `createBatch` is optional on `CommandQueue`, so test fakes need not implement it. See [the destructive-batch quorum](/docs/operations/pki#the-destructive-batch-quorum-v10). ```ts title="packages/rollout/src/module.ts" export const daemonUpgradeKind: CommandKindSpec = { kind: 'daemon.upgrade', lane: 'control', retrySafe: true, selfTerminating: true, timeout: '15m', roles: ['node', 'site'], payload: DaemonSectionSchema, // a protobuf-es message descriptor }; ``` | Field | Meaning | | ----------------- | ---------------------------------------------------------------------------------- | | `lane` | `control`, `provision` or `bulk`; the daemon runs each lane on a fixed worker pool | | `retrySafe` | Safe to run again after a crash between accept and completion | | `selfTerminating` | Ends the agent's process or boot; completes when the agent rejoins | | `attested` | The result is node-signed and verified before any transition (v0.2) | | `destructive` | Must be signed from v0.2 | | `timeout` | A duration; the CommandSweeper fails the command after it | | `roles` | Agent roles allowed to run it: `node`, `site`, `ramdisk` | | `payload` | The protobuf message the payload encodes as | `enqueue(tx, { nodeId, kind, payload, idempotencyKey, notAfter, … })` writes a `pending` row in your transaction; the same idempotency key on the same node returns the existing command. When the CommandSweeper expires or times out a command, it wakes the node's controller and, if the key's first segment names a registered reconciler (`acme.widgets::…`, or `wf::…` for workflows), that reconciler for ``. See [command kinds](/docs/reference/command-kinds) and [commands](/docs/guides/nodes/commands). ## NodeSpec sections [#nodespec-sections] `nodeSpec(ctx, node)` returns this module's part of the node's NodeSpec, a `NodeSpecPart` from `@better-compute/core` (`Partial` whose `site` is a `Partial`). The NodeSpec controller calls every module's `nodeSpec` in registration order and merges the parts: * `host.config`, `telemetry` and `access` merge key by key, later modules winning; `access.sshCaPublicKeys` is a union; * `devices.assignments` merge by `allocationId` and `instances` by `id`; * `site` merges field by field, the last contribution per field winning, so one module's `boot` and another's `cacheImages` do not erase each other (the site-boot module contributes `boot`, provision contributes `cacheImages` from the pools' `policy.prewarmImages`). A `site` section with no `boot` goes on the wire as `boot { active: false }`; * `daemon` is taken whole from the last module that returns it. The controller bumps the node's generation only when the merged spec's hash changes. Your module must say when its contribution may have changed: call `ctx.desired.invalidate(tx, nodeId)` in the transaction that changes it. The capacity module does this on every binding change; rollout returns a `daemon` section from `daemonSpecFor`. See [desired state and commands](/docs/guides/concepts/desired-state-and-commands). ## The module context [#the-module-context] `api()`, `nodeSpec`, `selfCheck`, `start` and every reconciler run get a `ModuleContext`. Method handlers get a `MethodContext`, which adds the caller; see [methods](/docs/extending/methods#the-method-context). | Port | What it does | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `sql` | The store. In a method handler it is the envelope's transaction; in a reconciler every `transaction` is lease-fenced | | `clock` | Business time anchored to `compute_now()`; the sim clock in tests | | `work` | `enqueue(tx, kind, key, { notBefore? })` and `enqueueMany`: wake a reconciler | | `events` | `publish(tx, { type, tenantId, subject, data })` and `publishMany`: append to the outbox | | `commands` | `enqueue(tx, command)` and `cancel(tx, commandId)`: the durable command queue | | `desired` | `invalidate(tx, nodeId)`: recompute a node's NodeSpec | | `authz` | `upsert`, `remove`, `setPending`: maintain your rows of the authz projection | | `capacity` | `reserve(tx, request, { admission }?)` and `release(tx, allocationId)`: quota plus the inline placer; pass the caller's admission so scheduler plugins' `admit` hooks run | | `inventory` | `get(nodeId)`: the latest stored inventory snapshot | | `leases` | `acquire`, `release`, `fence`: named leases with a fencing epoch | | `iam` | The IamPort; never call it inside a transaction | | `registry` | `hasModule`, `commandKind`, `driver`: a read-only view of what is registered | | `secrets` | The secret broker: `put`, `resolve`, `delete` over the configured stores | | `pki` | `issueJoinToken(tx, …)`: single-use join tokens for agents a module brings up | | `workflows` | `start(tx, …)` and `abort(tx, id)`, present only when a hosted module runs workflows (`provisionModule()`) | | `afterCommit` | Runs a hook once the outermost transaction commits, never on rollback; for pokes and in-memory updates only | | `caches` | `dropIdentity`, `dropSession`, `dropDecisions`: invalidate the envelope's IAM caches | | `typePrefix` | The deployment's IAM resource type prefix (`ComputeOptions.typePrefix`, default `compute-`), always set by `betterCompute()`, for a module that decides on compute resources itself, such as vps deciding `compute:secrets:manage` on `${typePrefix}tenant/`; optional in the type, so fall back to your own default | | `log` | `debug`, `info`, `warn`, `error` | `usage` (a `UsageRecorder`) is declared on the contract but the v0.1 server leaves it unset. The built-in capacity and vps modules write usage windows in SQL themselves. ## start, stop and selfCheck [#start-stop-and-selfcheck] `start(ctx)` is for work that must happen before the process serves: the IAM bridge loads its authz projection and binds its inbox there, and `eventsModule` anchors its fan-out cursor at the outbox tail so no event committed after start is missed. It runs in every process mode, so keep it idempotent. `stop()` releases what `start()` took. `selfCheck(ctx)` only reads and returns findings: ```ts interface Finding { check: string; // 'events.webhook-dead-letters' severity: 'error' | 'warning' | 'info'; message: string; fix: string; count?: number; } ``` `compute.selfCheck()`, the `system.selfCheck` method and `better-compute doctor` collect them from every module. A module whose `selfCheck` throws becomes one `module.` error finding; any `error` finding makes the report not `ok`. Report what an operator can act on: a backlog, dead letters, rows left unsealed (the audit module warns about audit events older than 3 hours that no checkpoint covers). ## tenantDeleted [#tenantdeleted] When Better IAM deletes a tenant, the built-in `tenancy` module records it in `compute_tenant_states` and enqueues a teardown. Its `tenancy` reconciler then calls every hosted module's `tenantDeleted(ctx, tenantId)`, plugins included, and a 30-second `sweep` asks again while any of them answered `SYNC_AGAIN`. Once every hook answers `DONE` in one pass, it sets `teardown_done_at` and publishes `tenant.torn-down`. See [tenancy](/docs/guides/iam/tenancy#suspended-and-deleted-tenants). ```ts title="widgets.ts" const batch = 100; export const acmeWidgets = () => definePlugin({ id: 'acme', // ... async tenantDeleted(ctx, tenantId) { const { rowCount } = await ctx.sql.query( `DELETE FROM plg_acme_widgets WHERE id IN ( SELECT id FROM plg_acme_widgets WHERE tenant_id = $1 LIMIT $2)`, [tenantId, batch], ); return rowCount === batch ? 'SYNC_AGAIN' : 'DONE'; }, }); ``` * The hook runs on every sweep until the whole teardown finishes, so a second call must find nothing left and answer `DONE`. * Bound the work per call and answer `SYNC_AGAIN` while more remains, as the built-ins do (100 rows of each kind). * Write what other modules or watchers must see through `ctx.events` in the same transaction as the change. The built-in hooks: `capacity` ends the tenant's reservations, withdraws offers to it and archives the pools it owns; `provision` aborts its workflows, soft deletes its private images and profiles and revokes its image signers; `metal` retires every machine it owns; `vps` deletes its instances through the ordinary lifecycle.