Instances

An instance is a VM in a VPS pool. You ask for one with a flavor, an image and SSH keys; after that you only change what you want it to be (running, stopped or deleted). The vps.instances reconciler works out how to get there, from what the host's runtime reports in a hosted pool or from what the provider answers in a provider pool.

This page assumes you registered vps() and have a VPS pool with at least one flavor; see VPS and flavors.

Some instances belong to sandboxes: the instance a sandbox drives, and the warm instances a sandbox template keeps ready. Those can be read but not moved with the methods on this page; see sandbox and warm instances.

Create an instance

const instance = await compute.api.instances.create(
  { token },
  {
    tenantId: 'acme',
    poolId: 'pool_01jz…',
    flavor: 'g1.small',
    imageId: 'ubuntu-24.04',
    sshKeys: ['ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA… alice@laptop'],
    idempotencyKey: 'build-agent-7',
  },
);
// Hosted pool: { id: 'inst_…', state: 'provisioning', desiredState: 'running',
//                nodeId: 'node_…', addresses: ['10.0.0.2'], … }
// Provider pool: { id: 'inst_…', state: 'scheduling', … }

From a browser or another service, the typed client strips the credential, and every method is also a plain HTTP route:

import { createComputeClient } from 'better-compute/client';

const client = createComputeClient<typeof compute>({ baseURL, token: () => session.getToken() });
await client.instances.create({ tenantId: 'acme', poolId, flavor: 'g1.small', imageId: 'ubuntu-24.04', idempotencyKey: 'build-agent-7' });

Unknown fields are INVALID_INPUT. The API reference has the route and action: instances.create.

How a create is admitted

instances.create follows the same request-as-resource admission as reservations: the instance exists as a draft before anyone decides on it, so policies can read what is being asked for.

A policy can bound what tenants create, for example only small VMs from signed images:

{
  "sid": "SmallVerifiedVms",
  "effect": "allow",
  "actions": ["compute:instances:create"],
  "resources": ["compute-instance/*"],
  "conditions": {
    "Bool": { "resource.imageVerified": true },
    "NumericLessThanEquals": { "resource.vcpu": 8, "resource.gpus": 0 }
  }
}

imageVerified comes from the image resolver. In a provider pool without vps({ images }) it is false.

Errors

Every resource an instance uses needs a quota row in the pool: vcpu, memoryGiB, diskGiB, and gpu for GPU flavors. pools.offer sets them for a tenant. A pool's owner gets only a gpu quota by default, so give the owner ownerQuota for the scalar resources before it creates instances in its own pool. See quotas and usage.

Waiting for the instance

create is async-with-wait (PLAN.md section 9.1). Without wait it answers as soon as transaction 2 commits, with the instance in scheduling or provisioning. With wait: { timeoutMs } the server holds the answer while the instance is pending, scheduling, provisioning or starting, re-reading its row in read-only transactions (backing off from 50 ms to 1 s), and returns as soon as it reaches any other state (running, stopped, error, …), timeoutMs passes (at most 30 seconds), or the request is aborted. A timeout is not an error: the answer is the instance as last read.

const instance = await client.instances.create({ ...input, wait: { timeoutMs: 20_000 } });
if (instance.state !== 'running') {
  // Still starting, or failed: read it again later with instances.get, or watch instance.* events.
}

wait is kept out of the request hash, so a retry with the same key and a different wait replays the stored instance instead of failing with CONFLICT. The MCP create_instance tool sends wait from its waitSeconds (default 20).

Read and list

instances.get returns one instance of the tenant, deleted ones included, and sandbox and warm instances too. Drafts and other tenants' instances are NOT_FOUND.

instances.list returns { items, cursor? }, newest first. It lists the instances sandboxes drive, but never the warm instances of sandbox templates:

Start, stop and delete

start and stop (compute:instances:update) set desiredState to running or stopped, and delete (compute:instances:delete) sets it to deleted. Each returns the instance at once, before anything has happened, and wakes the reconciler; a hosted instance's node also gets a new NodeSpec. Watch state, or the instance.* events, to see the change land.

• start, stop and delete (and resize) on a sandbox's instance or a warm instance are INVALID_STATE; see sandbox and warm instances.
• start or stop on an instance that is being deleted is INVALID_STATE.
• start on an instance in error that has a host or a provider machine retries it: it moves to starting and the runtime or provider is asked again. An instance that failed before it was placed stays in error; delete it and create a new one.
• delete is idempotent. A deleted instance stays readable with state: 'deleted'; deleting it again returns it.
• Deletion releases everything the instance held: the capacity allocation, the IP lease and the quota, and its authorization row.
• When the IAM tenant is deleted, vps().tenantDeleted drops the tenant's drafts and sets every live instance, sandboxes' and warm ones included, to desiredState: 'deleted' (up to 100 per call), answering SYNC_AGAIN until all are deleted. Hosted instances hold inline allocations, which capacity's own teardown does not reach.

A hosted instance is deleted only once its runtime has dropped it: the next complete report from the host no longer lists it. If the host is lost, or the instance was never placed, deletion finishes without waiting. A provider instance is deleted once the provider's delete succeeds or finds nothing to delete.

From the command line

The instances commands (44915e5) wrap the same methods as the token's identity:

better-compute instances create --tenant acme --pool pool_01jz… --flavor g1.small --image ubuntu-24.04 \
  --ssh-key ./keys/alice.pub,./keys/ci.pub --user-data ./cloud-init.yaml
better-compute instances list --tenant acme --all --state running
better-compute instances get inst_01k5w… --tenant acme --query state
better-compute instances stop inst_01k5w… --tenant acme
better-compute instances resize inst_01k5w… --tenant acme --flavor g1.large
better-compute instances start inst_01k5w… --tenant acme
better-compute instances delete inst_01k5w… --tenant acme

• create reads --ssh-key from files (comma-separated paths; every non-empty line that is not a comment is a key) and refuses a file that holds a private key. --user-data is a cloud-init file, or a secret://<tenant>/<id> reference passed as it is. --stopped creates it with desiredState: 'stopped'.
• create waits up to --wait seconds (default 30, the API's maximum) for the instance to settle, and sends a new idempotency key on every run unless you pass --key: repeat the command with the same --key to retry safely.
• list shows your own instances; --all asks for the tenant's (scope: 'all').
• Instance ids that do not look like inst_… are usage errors (exit 2) before anything is sent.

See the instances commands for every flag.

Resize

instances.resize moves a stopped hosted instance to another flavor of its pool, on the same host. It exists so a tenant can grow a VM without rebuilding it.

await compute.api.instances.stop({ token }, { tenantId: 'acme', instanceId });
// … wait until state is 'stopped' …
await compute.api.instances.resize({ token }, { tenantId: 'acme', instanceId, flavor: 'g1.large' });

The call charges the growth to quota first, then resizes the allocation in place on the node, then refunds any shrinkage. It is all or nothing: if the host lacks room, the quota charge is undone.

The instance passes through resizing and is stopped again when the call returns, with its policy attributes updated. See flavors for how shapes become claims.

Sandbox and warm instances

Sandboxes (v0.5) are built on instances, and migration vps/0004_sandboxes marks the instances they own with two columns, never both set:

Such an instance is admitted, placed, charged to quota, addressed, metered and deleted exactly like one from instances.create, but only its sandbox or its template moves it. instances.start, stop, resize and delete refuse it with INVALID_STATE ("Instance … is managed by sandbox …" or "… is a warm instance of sandbox template …"); use sandboxes.delete or sandboxTemplates.update instead. instances.get returns both kinds, and instances.list returns a sandbox's instance but hides warm instances. The Instance view does not name the sandbox; the sandbox's instanceId points the other way. The instance events vps publishes for them carry data.sandboxId or data.warmTemplateId.

tunnels.open refuses both kinds too, for every op, with INVALID_STATE and details.reason: 'sandbox-managed', so compute:instances:exec on the instance cannot bypass the sandbox's own decision. The way into a sandbox is sandboxes.exec, decided as compute:sandboxes:exec on the sandbox; a warm instance takes no tunnels at all.

Paused instances

paused is an instance state and a desired state (v0.5): vCPUs stopped, memory resident, the host and allocation kept. No API method asks for it; only warm pools do, on runtimes whose RuntimeTraits.pause is true. The lifecycle converges it like the other settled states: from provisioning it walks through starting and running, then pauses (pause); from paused it resumes to running (resume) or stops. A host reports it as InstanceObservedState.PAUSED, and a lost instance can recover to paused. A runtime without pause treats a desired paused as stopped, and vps never asks one for it: warm pools on such runtimes stay stopped.

The reconciler

vps.instances runs one instance per work item, keyed by the instance id. Every API change, every host report that changes something and every provider answer enqueues it again, and it schedules its own next look:

Hosted pools

The API never calls a runtime. The reconciler changes the database, and the host acts on its NodeSpec:

Each hosted node's instances section lists every live instance placed on it (not pending, scheduling or deleted) with its runtime, flavor, image (url, sha256, format), its user data as a secret:// reference, the GPU atoms and id of its allocation (allocationId), and one NIC (nics): the host bridge (br0, or vps({ hostNetwork })), a stable locally administered MAC derived from the instance id, and its IPAM address in CIDR form. NIC order is the guest's interface order and is kept. An instance being deleted is sent with desiredState: 'deleted'.

User data never travels in the spec. At admission in a hosted pool, vps renders the cloud-config (the hostname from the instance id and sshKeys, unless you passed userData), seals it through the secret broker and keeps only its secret:// reference (instances.user_data_sealed, migration vps/0002_sealed_user_data); a tenant's own reference (one it wrote with secrets.put with the default tenant scope, checked in the create transaction; a fleet secret is refused) is kept as it is, and only a sealed one is deleted with the instance. The host fetches the value with a node-signed SecretRequest naming the instance's allocation, and the broker releases it only when the secret belongs to the instance's own tenant. The guest gets no join token, no bcd and no IAM key.

From the reports, the reconciler moves the instance:

• When the reported state equals desiredState, it walks state there: provisioning to starting to running, running to stopped and back, or running to paused and back for warm instances. Each step publishes instance.<state>.
• A reported error moves it to error with a RuntimeError condition and publishes instance.failed.
• A reported stopped whose message starts ResourcesExhausted: is bcd's host admission refusing the start: the instance gets a HostResourcesExhausted condition and, if nothing of its tenant's is on that host yet, is placed elsewhere after a minute (see hosts that cannot start an instance).
• While the report and desiredState disagree, it waits and looks again in 10 seconds.

Hosts that cannot start an instance

The control plane books a host's capacity, but the host can still lack room for a start: memory or huge pages taken by something the booking does not see, or a booking and a host that drifted apart. bcd's host admission then leaves the instance stopped with a message starting ResourcesExhausted: and retries on its next pass. Since 564f543 vps acts on that report instead of leaving a new instance provisioning on that host for good:

1. The condition. reportInstances records HostResourcesExhausted on the instance: reason is the reporting node, message bcd's words (up to 512 characters). It is cleared as soon as that node reports the instance in any other way.
2. Waiting a minute. Once the refusal has stood for exhaustedReplaceMs (60 seconds), the reconciler re-places the instance if nothing of its tenant's is on that host yet: an instance still in provisioning, or a warm instance in stopped that no sandbox has claimed or whose sandbox has not become ready. It never moves one whose desiredState is stopped, one that already ran there, or one whose volumes live on that host.
3. Placing it elsewhere. It releases the allocation and the IP lease, invalidates the old host's NodeSpec (which now drops the instance, so that host removes the domain it could not start), and moves the instance back to scheduling (recreate), publishing instance.recreating with reason: 'resources-exhausted' and fromNodeId. The ExhaustedHosts condition lists every host that refused it (comma-separated in message, oldest first), and for exhaustedAvoidMs (10 minutes) after the last re-placement, scheduling passes them to capacity as excludeNodes.
4. Giving up. After maxExhaustedReplacements (3) re-placements, the next refusal that stands a minute fails the instance: error, and instance.failed with reason: 'resources-exhausted' and the nodeId.

An instance that is not re-placed keeps waiting with HostResourcesExhausted, and bcd keeps retrying the start. Both conditions are removed once the host reports the instance in its desired state. A sandbox whose instance is re-placed keeps its TTL: the clock does not restart. Tested in tests/vps-instances.test.ts (re-placement away from the host, the growing host list and the cap, a refused restart that waits) and tests/vps-sandboxes.test.ts.

Provider pools

There is no node. Each round is three steps, so no network call ever runs inside a database transaction:

1. Plan in a short transaction: lock the instance, start deletion if desiredState is deleted, and read the provider account.
2. Call the account's ProviderImpl with the instance id as the idempotency key.
3. Apply the answer in a second transaction, only if the instance's version has not moved; otherwise try again at once.

The machine's id becomes externalId, its addresses replace addresses, and its state maps to running, stopped, error or, for deleted, absent. The first answer moves scheduling to provisioning and publishes instance.provisioning.

A failed call records a ProviderError condition with the error code as reason and the message (up to 500 characters). UNSUPPORTED, INVALID_STATE and INVALID_CONFIG move the instance to error; anything else, RATE_LIMITED and PROVIDER_ERROR included, is retried in 30 seconds. A machine that get no longer finds makes the instance lost, and the reconciler stops polling it; delete it to refund its quota.

Reports from hosts

The gateway hands each node's coalesced StatusReport to instanceStatusHandler, inside its ingest transaction. The report's instances lists every VM the host's runtimes hold, managed or not:

• States map from InstanceObservedState: RUNNING, STOPPED, PAUSED and ERROR. UNSPECIFIED entries are skipped.
• Only instances placed on the reporting node are touched, so one node can never move another node's instances.
• Reported addresses are added to the instance's addresses.
• With instancesComplete: true the list is the whole inventory: a placed instance missing from it is marked absent. That is how deletion completes. An empty, incomplete report changes nothing.

You can call the same seam yourself, which is what the tests and the churn acceptance do to emulate hosts:

import { reportInstances } from '@better-compute/vps';

// `work` is the runtime's work queue (ModuleContext['work']): changed instances are enqueued on it.
await db.transaction((tx) =>
  reportInstances(tx, work, nodeId, [{ id: instanceId, state: 'running', addresses: ['fe80::1'] }], {
    complete: true,
  }),
);
// → { updated: 1, orphans: [] }

Orphans and leak alerts

An instance the runtime holds that no live row places on that node is an orphan: a leaked VM, overlay or tap, or one reported by the wrong host. reportInstances returns orphan ids sorted, and never changes anything for them.

instanceStatusHandler({ events }) publishes instance.orphaned with subject node/<id> and data: { nodeId, instanceIds } (up to 100 ids). It announces a node's orphan set only when the set changes; hosts report about once a second and leaks persist. The last announced set is kept in memory, so a restarted process announces current leaks once more. Without events, orphans go to log (default console.warn). vpsInstanceStatusHandler is the handler with no event publisher.

The v0.3 acceptance checks the other direction too: after 200 instances churn on 3 emulated hosts, every runtime domain is a live placed instance and every allocation, IP lease, scalar claim and quota counter matches the live instances (tests/vps-churn.test.ts).

Lost hosts

A host becomes lost only when its vps-host pool sets policy.lostAfter: the liveness controller then marks it lost that long after it became unreachable (see operating nodes). Without lostAfter a host stays unreachable and its instances wait for it. When a hosted instance's node becomes lost, the instance moves to lost and instance.lost is published with the nodeId. The reconciler then looks every 60 seconds:

• If the node reconnects and reports the instance running, stopped or paused, it recovers to that state (instance.recovered).
• If the pool's policy has recreateOnHostLoss: true, the instance releases its allocation and IP lease, goes back to scheduling (instance.recreating) and is placed again, on a host that is still schedulable, where it boots from its image.

The same recreate verb also moves an instance in provisioning, or a warm one in stopped, off a host whose bcd cannot start it (564f543; instanceStateTable allows recreate from those two states); see hosts that cannot start an instance.

recreateOnHostLoss is read from the pool's stored policy document; it is not part of the PoolPolicy type yet, so a typed caller setting it through pools.create or pools.update needs a cast.

Runtimes

A runtime is the driver that runs VMs on a host, under the runtime driver kind that importing @better-compute/vps adds to DriverKinds. It runs on the node (runsOn: 'node'), and every operation is idempotent by instance id:

interface RuntimeDriver {
  readonly traits: RuntimeTraits; // isolation 'vm' | 'microvm' | 'container', multiTenant, gpuPassthrough, imageFormats
  list(): Promise<RuntimeInstanceStatus[]>; // everything it holds, so leaks are found by diffing
  apply(spec: InstanceSpec): Promise<RuntimeInstanceStatus>; // cache the image, build the overlay and NoCloud seed, start/stop/resize
  reboot(instanceId: string): Promise<void>;
  remove(instanceId: string): Promise<void>; // an absent instance counts as removed
  console?(instanceId: string): Promise<DuplexStream>; // the serial console, for tunnels
  // v0.5, for sandboxes: warm pools and snapshots
  pause?(instanceId: string): Promise<void>; // vCPUs stopped, memory resident
  resume?(instanceId: string): Promise<void>;
  snapshot?(instanceId: string, target: RuntimeSnapshotTarget): Promise<RuntimeSnapshotResult>;
}

pause and resume serve sandbox warm pools: a runtime that implements them sets pause: true in its RuntimeTraits, and builtinRuntimeTraits does so for libvirt-kvm, cloud-hypervisor and firecracker (not containerd). vps then keeps warm instances paused on that runtime; see paused instances. snapshot and InstanceSpec.restoreFrom serve sandbox snapshots: sandboxes.snapshot sends the host an instance.snapshot command, and sandboxes.create({ fromSnapshot }) boots a new instance from one. A runtime that implements snapshot sets snapshot: true in its RuntimeTraits (the three built-in VM runtimes do); on any other, sandboxes.snapshot is UNSUPPORTED up front. No instances.* method snapshots a plain instance.

libvirt-kvm and cloud-hypervisor are the runtimes built into bcd (v0.3), which its instances converger drives through its Runtime interface. Both are written, and a host build runs each one whose tools are on its PATH (there is no configuration key):

• libvirt-kvm (362317c), with virsh: virsh -c qemu:///system and qemu-img, a host-passthrough CPU, domains marked as bcd's in their metadata (others are reported as unmanaged and never touched), and iso images booted from a cdrom.
• cloud-hypervisor (2598f0a), with the cloud-hypervisor binary: one VMM per instance, started as a confined systemd transient unit bcd-ch-<id>.service in bcd-instances.slice so the VM outlives bcd, and driven over its REST API; bcd's own vm.json per instance says what it owns, since cloud-hypervisor keeps no registry. It boots disk images only (iso is refused), and needs systemd-run and the firmware at /usr/share/cloud-hypervisor/CLOUDHV.fd, which bcd doctor checks.

Both runtimes treat a tenant's image as hostile (security review 12, fixed in 933c51b and hardened in b12c217). bcd reads a qcow2 base image's header in Go when it is downloaded and again each time an instance uses it, with no tool probing the image: one that names a backing file or an external data file, sets an unknown incompatible feature, is encrypted or is not a qcow2 is refused, which fails that instance. raw and iso images are opened with their format stated and never probed. An instance's disk must name exactly its recorded base before every start, snapshot and resize, and a restored snapshot's disk must name none. On cloud-hypervisor bcd creates each instance's taps on their bridge before its VMM starts and hands them to the VMM as file descriptors, so a guest reboot never puts a tap outside the instance firewall; the VMM runs as the instance's own uid with no capabilities and no network, its unit sees only its own instance directory and base image with /run and /var hidden, opens only KVM and (with GPUs) VFIO devices, and is bounded by the flavor's memory, vCPUs and a task limit; and a memory snapshot restores only into an instance with the same vCPUs and memory. See the runtimes.

Both give a 60-second graceful shutdown, VFIO GPUs, a serial console for instance console tunnels, and snapshots and restores that refuse another runtime or CPU model; see the runtimes. Both have been tested only against fakes, not on real hardware. fakehw builds run a file-backed runtime named libvirt-kvm instead. Out-of-process runtimes are served too: on bcd node, a bcx plugin whose driver lists the kind runtime becomes a runtime (list, apply(spec), remove(id) over Invoke; no snapshots), selected by pools like any other. No such plugin (a Kata runtime is PLAN.md's walkthrough 4) is written yet; see walkthrough 4. vps({ runtimes }) only names the ids: the first one is recorded on every new hosted instance and sent in its spec.

A hosted instance is placed only on hosts carrying the node label bc.dev/driver.<runtime>, in any version (runtimeNodeLabels), for plugin runtimes and, since 0fca413, for bcd's built-ins (libvirt-kvm, cloud-hypervisor) too. The gateway writes that label from the drivers a daemon advertises in Hello; every bcd since v0.3 (fab94d4) advertises each runtime it runs, and runs one only where its tools are (cloud-hypervisor under bcd helper only since 7fe74a1). A pool with no host advertising the runtime leaves the instance Unschedulable, with a message naming the missing label; see placement.

Events

Instance events have subject instance/<id> and carry state, desiredState and poolId (plus sandboxId or warmTemplateId for sandbox and warm instances, and reason: 'tenant deleted' on instance.deleting during a tenant teardown):

Access to an instance

Exec, the serial console, SSH and TCP port forwards into a hosted instance go through a tunnel: tunnels.open({ tenantId, instanceId, op }) with op exec, console, ssh or tcp (plus a port) is decided as compute:instances:exec on the instance and returns a single-use exec ticket for the instance's node, which carries the instance (inst). The instance must be running and its node connected; provider instances are UNSUPPORTED, since no platform agent runs next to them. console, ssh and tcp also need the node's daemon to advertise tunnel.instance-dial, or they are UNSUPPORTED with details.reason: 'daemon-capability'. The method is registered only when the deployment configures tunnels. See tunnels and SSH.

console (798e710) attaches to the instance's serial console, which bcd opens from its runtime (libvirt-kvm hands it the pty of the domain's serial port, cloud-hypervisor its serial socket), never a host shell. bcd refuses exec into an instance until guests have an agent, so the serial console is the terminal that works on a real host: better-compute console --instance ID from a terminal, or Serial console (/instances/<id>/console) on the instance's page in the console. Sandbox and warm sandbox instances stay refused for every op (above).

Workload identity

A hosted instance can prove who it is to a third party (Better IAM's sts/assumeRoleWithWebIdentity, a cloud STS) with a short-lived OIDC token, sub = "t:<tenantId>:instance:<instanceId>", that it fetches from its host's metadata service (IMDS). This is v0.5 and optional: the control plane half is built and tested end to end against the FakeDaemon (tests/e2e/imds.test.ts). bcd node has both host halves: the session half (RequestWorkloadToken: a node-signed WorkloadTokenRequest with a one-time key, the sealed answer opened and checked against the instance and request id) and the metadata service, served on [imds] listen in its config.toml (for example 169.254.169.254:80 on the instances' bridge). It resolves the caller as the running instance whose address is the request's source. It serves the instances of whichever runtime bcd runs; since neither built-in runtime has run on real hardware yet, no real VM has reached it, and its Go tests and fakehw builds cover it.

• The metadata service. The host serves each instance on its own interface and knows the instance from where the request arrived, never from a request field. PUT /latest/api/token with x-bc-metadata-token-ttl-seconds (1 to 21,600) returns a session token bound to that instance; GET /latest/identity-token?audience=… with x-bc-metadata-token returns the token. A session token from one instance fails for any other, and requests with X-Forwarded-For or Forwarded are refused, which blocks the usual SSRF relays.
• The host holds no signing key. It sends a WorkloadTokenRequest over its session, signed by its node key, and gets the token sealed to a one-time key, so the gateway only relays ciphertext.
• The broker mints only when the request is signed by the node's current certificate of role node (never node-tenant, a ramdisk, a site agent or a kind-restricted certificate) naming the session's node; the instance is live, placed on that node, with its live allocation there; its pool has policy.workloadIdentity: true; its tenant is neither suspended nor deleted (otherwise disabled, recorded with the reason tenant-suspended or tenant-deleted); the audience is the issuer's default or a configured one; the request id is new; and the instance is within its rate (30 tokens a minute by default). Tokens live at most 15 minutes.
• Refusals are answers, not errors: not-bound, disabled, audience, rate-limited, access-denied or unavailable. Each one publishes security.workload.refused (rate-limited per reason); each token issued writes the audit row workload.token.issued with its jti, never the token.

Turn it on with a workload issuer from @better-compute/pki and the pools that should use it:

import { createWorkloadIssuer } from '@better-compute/pki';

export const compute = betterCompute({
  // …
  workloadIdentity: {
    issuer: createWorkloadIssuer({
      issuer: 'https://compute.example.com/workload', // https on 443
      defaultAudience: 'https://iam.example.com',
    }),
    audiences: ['sts.example.com'], // besides the default
    ttlSeconds: 900, // default and maximum
    perInstancePerMinute: 30,
  },
});

better-compute serve publishes the issuer's discovery document and JWKS under the issuer URL's path, so relying parties can verify the tokens. See workload identity for the keys and their rotation.