better/compute
VPS and sandboxes

VPS

Virtual machines on your own hosts or on a provider's cloud, with desired state, flavors, provider accounts and pool IPAM from the vps() module.

beta@better-compute/vps@better-compute/provider-hetzner@better-compute/provider-proxmoxREADME.mdmodule.tssandboxes.tstenancy.tsinstances.tslifecycle.tsruntime.tsstatus.tsstates.tsconverge.gocompose.tsPLAN.md

Some of your capacity is best sold as whole machines, and some as virtual machines: a tenant wants a 2 vCPU box with an SSH key for a build agent, or an 8 vCPU VM with one L40S passed through, without renting a whole server. Other times you want to burst onto a cloud you do not run, and still keep one API, one quota system and one audit trail.

The vps() module does both. It adds : VMs described by a , an and SSH keys, created in a . Where the VM runs depends on the pool:

  • A hosted pool runs VMs on your own servers. Capacity places each instance on a host, and the host's converges it.
  • A provider pool creates each instance on an external cloud, through a such as Hetzner Cloud or your own Proxmox VE cluster.

Either way, the API only records what you want. A does the work, and nothing in a request ever calls a hypervisor or a cloud API.

Register the module

vps() is an injected module: the server never imports it, so you add it to modules. It requires the built-in capacity module, which every server has.

compute.ts
import { betterCompute } from 'better-compute/server';
import { vps } from 'better-compute/vps';
import { hetznerCloud } from '@better-compute/provider-hetzner';
import { images } from './images';

export const compute = betterCompute({
  // database, secret, iam, …
  modules: [
    vps({
      runtimes: ['libvirt-kvm'], // hosted pools: the runtime driver bcd uses
      providers: [hetznerCloud], // provider pools: the providers accounts may use
      images, // hosted pools: resolves imageId for a tenant
    }),
  ],
});

The provider packages are on npm but sit outside the better-compute umbrella, so you install and import them by their own names (npm install @better-compute/provider-hetzner).

Prop

Type

vps ships no image resolver: Better Compute checks image visibility itself, and you decide where images come from. This one serves hosted VMs the images registered through the images group of provisionModule, when the tenant owns them or they are public:

images.ts
import type { ImageResolver } from '@better-compute/vps';

export const images: ImageResolver = {
  async resolve(tx, tenantId, imageId) {
    const { rows } = await tx.query<{
      source: string;
      sha256: string;
      format: 'raw' | 'qcow2' | 'iso';
      verified: boolean;
    }>(
      `SELECT source, sha256, format, verified FROM images
        WHERE id = $1 AND deleted_at IS NULL AND (tenant_id = $2 OR visibility = 'public')`,
      [imageId, tenantId],
    );
    const image = rows[0];
    // undefined: the tenant cannot see it, and instances.create answers NOT_FOUND.
    return image && { url: image.source, sha256: image.sha256, format: image.format, verified: image.verified };
  },
};

verified becomes the instance's imageVerified attribute, which policies can require.

Registering the module adds:

  • the instances, flavors and providerAccounts API groups (instances, flavors, providerAccounts), and the v0.5 sandboxes, sandboxTemplates and sandboxSnapshots groups (sandboxes, sandboxTemplates, sandboxSnapshots) for sandboxes on hosted pools;
  • the migrations vps/0001_instances, with the tables flavors, provider_accounts, instances, ip_pools and ip_leases, vps/0002_sealed_user_data (instances.user_data_sealed), vps/0003_provider_bootstraps (provider_bootstraps, for bootstrapping provider machines) and, for sandboxes, vps/0004_sandboxes (the tables sandbox_templates and sandboxes, the columns instances.sandbox_id and instances.warm_template_id, and paused as a desired state), vps/0005_sandbox_snapshots (sandbox_snapshots, instances.restore_from, sandboxes.from_snapshot), vps/0006_warm_claims (instances.claimed_at), vps/0007_template_env (sandbox_templates.env) and vps/0008_template_network (sandbox_templates.network and instance_version, instances.network_policy and warm_template_version), vps/0009_snapshot_fixes (sandbox_snapshots.principal_instance_id and purged), vps/0010_sandbox_metric_indexes (partial indexes for the sandbox metrics), vps/0011_host_certificates (instances.host_certified, sandbox_snapshots.host_certified) and vps/0012_host_cert_validity (host_cert_valid_before on both);
  • the vps.instances reconciler, woken through the by every change, the vps.sandboxes reconciler (every 15 seconds: sandbox expiry, sandbox events, warm pools and settling snapshots), and the vps.usage job, which meters hosted instances every 5 minutes;
  • the instance.snapshot command kind (lane bulk, timeout 1 h) for sandbox snapshots;
  • a tenantDeleted hook that deletes every instance of a deleted IAM tenant, sandboxes' and warm ones included;
  • an instances section in the of each hosted node;
  • the runtime driver kind, merged into DriverKinds of @better-compute/core when you import the package;
  • doctor findings for hosted pools: vps.pools.runtime (a runtime the deployment does not list), vps.pools.single-tenant (a container pool without policy.singleTenant), vps.pools.ipam (no IP range) and vps.hosts.capacity (adopted hosts without capacity rows); see doctor.

Hosted pools also need the to feed each host's runtime report to vps. Pass instanceStatusHandler as a status handler. It runs next to the built-in XID health handler, each under its own savepoint:

better-compute.config.ts
import { defineHost } from '@better-compute/cli';
import { instanceStatusHandler } from '@better-compute/vps';

export const host = defineHost(async ({ compute }) => ({
  gateway: {
    // region, listen, tls, …
    options: { reports: { status: instanceStatusHandler({ events: compute.context.events }) } },
  },
}));

Hosted pools and provider pools

A VPS pool is a pool of class vps-host or provider, created with pools.create like any other. Instances in pools of other classes are refused with INVALID_INPUT.

Hosted poolProvider pool
Poolclass: 'vps-host', agent: 'bcd'class: 'provider', agent: 'none', providerAccountId
Where the VM runsA node of the pool, running bcdThe provider's cloud; there is no nodes row
PlacementCapacity claims vCPU, memory, disk and whole GPUs on one host, in the create requestNone: the provider decides; the reconciler calls ProviderImpl.create
QuotaCharged by the placementCharged at admission from the flavor's shape
Who convergesThe host's runtime, from the instances section of its NodeSpecThe vps.instances reconciler, polling the provider
State comes fromThe host's StatusReport.instancesProviderImpl.get
AddressesA lease from the pool's IPv4 range, plus what the runtime reportsWhat the provider reports
ResizeIn place, while stopped, same GPU countUNSUPPORTED
Meteringvps.vcpu.hours, vps.memory.gib.hours, gpu.<model>.hoursNone: the provider bills you

No platform agent ever runs inside a tenant VM, and no ever goes into tenant-readable user data. A provider machine that should become a host of your own (a Hetzner Robot server that runs bcd) is a different thing: see bare metal.

The instance model

An instance has two states. desiredState is what you asked for: running, stopped or deleted. state is where the instance actually is. instances.create sets desiredState (default running); start, stop and delete change only desiredState and wake the reconciler. The reconciler moves state toward it, one guarded transition at a time, from what the host or the provider reports.

Every state except deleted and deleting can move to deleting, and most can move to error and lost. The full table is instanceStateTable in @better-compute/core; since 564f543 it also allows recreate from provisioning and stopped, for an instance whose host cannot start it (see hosts that cannot start an instance). rebooting is in the table, but no API method enters it in this release. paused (v0.5: running to paused by pause, back by resume) is used only by the warm pools of sandboxes; see paused instances.

The API returns the Instance view:

Prop

Type

Instances covers each operation and how the reconciler turns reports into transitions.

What ships in v0.3

The control plane is complete and tested: the three API groups, admission, the reconciler for both pool kinds, pool IPAM, the report seam and metering run against PGlite in tests/vps-instances.test.ts, and tests/vps-churn.test.ts runs the v0.3 acceptance shape (200 instances churned on 3 emulated hosts with no leaked allocation, IP lease, scalar claim or quota). What runs the VMs on a host is written but not yet proven: bcd's libvirt-kvm and cloud-hypervisor runtimes have been tested against fake tools only.

PartStatus in this release
instances, flavors, providerAccounts groupsServed (v0.3).
vps.instances reconciler, IPAM, vps.usage meteringShipped.
hetzner-cloud providerShipped in @better-compute/provider-hetzner, tested against an in-memory emulator of the Hetzner Cloud API.
proxmox providerShipped in @better-compute/provider-proxmox (PLAN.md scheduled it for v0.5), tested against an in-memory PVE emulator.
Host runtimes libvirt-kvm and cloud-hypervisorbcd (v0.3), built but not released. A host build with virsh runs libvirt-kvm (362317c) under the instances converger: qcow2 overlays on sha256-verified https base images, NoCloud seeds, VFIO GPUs, pause and resume, snapshots and restores. A host build with the cloud-hypervisor binary runs cloud-hypervisor (2598f0a): one VMM per instance as a confined systemd transient unit, over its REST API, with the same overlays and seeds, VFIO GPUs, pause and resume, memory snapshots and restores. Security review 12's two high-severity findings are fixed (933c51b): taps are bridged before the VMM starts and an inet bcd_taps backstop drops a tap off its bridge, and both runtimes refuse a tenant image or restored snapshot that names a backing file or an external data file. Both have been tested only against fake tools, never on a real KVM host. A host with neither and without a runtime plugin fails a non-empty instances section; fakehw builds run a file-backed runtime. See the runtimes.
Runtimes as out-of-process plugins (Kata, and others)v0.5: bcd serves a bcx plugin of kind runtime as an instances runtime (list, apply, remove; no snapshots), Go unit tests only. No Kata plugin is written. See walkthrough 4.
Single-tenant container runtimesPlanned for v0.3; the RuntimeTraits contract describes them, nothing enforces single tenancy yet.
Interactive access (exec, SSH, TCP through a tunnel)tunnels.open with an instanceId, decided as compute:instances:exec: hosted instances only, and only when the deployment configures tunnels. The API reference lists it as planned. See tunnels.
Sandboxes (sandboxes, sandboxTemplates, sandboxSnapshots)v0.5: served, all 15 methods. Templates with env and network, TTL expiry, warm pools of paused instances and their vps.sandbox.warm meter, snapshots, restores and snapshot management, sandboxes.exec (from the tunnels module), the instance guard, tenant teardown (tests/vps-sandboxes.test.ts, tests/e2e/sandboxes.test.ts). bcd enforces a template's network with host nftables rules. Like every hosted instance, a sandbox needs a host runtime. See sandboxes.
Per-instance metadata service and workload identityv0.5, optional. The control plane half (the workload broker, the session frames, the issuer and its published JWKS) is built and tested against the FakeDaemon's metadata service; bcd serves the same contract on [imds] listen, for the instances of whichever runtime it runs (end to end only with a fakehw build's file-backed runtime so far). See workload identity.

The TypeScript FakeDaemon in @better-compute/testing speaks the real protocol for the simulator and better-compute dev --sim N. As a VPS host it runs a FakeHypervisor (domains, overlays, NoCloud seeds, pause and resume), which is how tests/e2e/vps.test.ts runs hosted instances through the real server and gateway. The unit suites, tests/vps-sandboxes.test.ts among them, emulate hosts by calling reportInstances directly.

Was this page helpful?

Better Compute is created by Sean Filimon

Last updated

On this page