# Volumes and addresses (/docs/guides/vps/provider-volumes)

> Block volumes, shared (NFS) volumes, their snapshots and reserved public addresses on a provider account, as typed resources next to instances, and what happens to them when an instance is deleted.



A tenant on a provider pool can keep data and an address that outlive any one instance: a **provider volume** is a
block volume on the pool's provider account (or a [shared NFS volume](#shared-nfs-volumes) many instances mount),
a **provider snapshot** is a point-in-time copy of one, and a **provider
address** is a reserved public IP. All three are typed resources of the `vps` module, listed and authorized like
instances, and converged on the provider by one reconciler, `vps.provider-resources`.

They are different from the [sandbox and host volumes](/docs/guides/vps/sandboxes) that `bcd` keeps on a hosted
machine's own disks: those live on a node, these live in the cloud account.

| Provider     | Volumes                                                                 | Snapshots                  | Addresses                                   |
| ------------ | ----------------------------------------------------------------------- | -------------------------- | ------------------------------------------- |
| DigitalOcean | Block Storage (`kind: block`) and Network File Storage (`kind: shared`) | Volume and share snapshots | Reserved IPv4 and IPv6                      |
| Others       | When the provider implements `ProviderImpl.volumes`                     | With volumes               | When it implements `ProviderImpl.addresses` |

A pool whose provider implements neither answers `UNSUPPORTED` to every create.

## How a request converges [#how-a-request-converges]

Every mutation records what the tenant wants (`desiredState`, the wanted attachment or assignment) in its own
transaction and returns at once, usually with `state: 'pending'`. The reconciler then calls the provider outside any
transaction and writes back what it saw:

* States are `pending`, `ready`, `updating` (a resize running), `deleting`, `deleted` and `error`. An `error` carries
  the provider's `message` and is retried by the next change or the periodic pass (every 5 minutes). A settled
  resource is read once an hour for drift, those reads spread across the hour so they leave the token's request
  budget to instance polling.
* Provider actions (attach, detach, resize, snapshot) never block a run. A running action is looked at again 5 seconds
  later; a rate-limited or unavailable provider 30 seconds later.
* A create sends the provider a name derived from the resource id, so a create whose answer was lost finds what it
  made instead of making a second one. A reserved IP has no name on DigitalOcean: a reserve whose answer was lost puts
  the address in `error` and is never sent again (see [lost reserves](#lost-reserves)).

Each create takes an `idempotencyKey`. The same key with the same body returns the same resource; with a different
body it is `CONFLICT`.

## Volumes [#volumes]

```ts title="volume.ts"
const volume = await compute.api.providerVolumes.create(token, {
  tenantId: 'ten_01jz…',
  poolId: 'pool_01jz…',
  name: 'pg-data',
  region: 'nyc3',
  sizeGiB: 100,
  filesystem: 'ext4',
  instanceId: 'inst_01jz…', // attach once both are ready
  idempotencyKey: 'pg-data-1',
});
// → { id: 'pvol_…', state: 'pending', attachment: { instanceId, state: 'pending', desiredState: 'present' }, … }
```

<TypeTable
  type="{
  name: { type: 'string', description: '1-42 lower-case letters, digits or -, starting with a letter. Unique per tenant among volumes that are not deleted.', required: true },
  region: { type: 'string', description: &#x22;One of the provider account's regions, such as nyc3.&#x22;, required: true },
  sizeGiB: { type: 'number', description: 'From 1 to 16384 GiB.', required: true },
  filesystem: { type: &#x22;'ext4' | 'xfs' | 'none'&#x22;, description: 'Formats the volume when it is created. Unset or none: a raw block device.' },
  instanceId: { type: 'string', description: 'An instance to attach it to, of the same tenant, provider account and region.' },
  deleteWithInstance: { type: 'boolean', description: 'Delete the volume when the instance it is attached to is deleted; otherwise deleting the instance only detaches it.', default: 'false' },
  description: { type: 'string', description: 'Free text kept with the volume.' },
  projectId: { type: 'string', description: 'The project the volume belongs to, for listing.' },
}"
/>

Once it is `ready`, `device` is the path the guest sees it at. On DigitalOcean that is
`/dev/disk/by-id/scsi-0DO_Volume_<name>`, where `<name>` is the provider name derived from the volume id.

better-compute does not mount it yet: the volume arrives in the guest without a reboot, as a block device (formatted
when you asked for a `filesystem`), and the guest mounts it, for example with an `/etc/fstab` line on that path with
`nofail`. Mounting by bcd, and a cloud-init mount for volumes attached at create time, are planned.

The other methods of `providerVolumes`:

| Method        | What it does                                                                                                                                                                               |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `get`, `list` | One volume, or the tenant's (filter by `poolId` or `projectId`; `includeDeleted` adds deleted ones).                                                                                       |
| `attach`      | Attaches it to an instance. A block volume is attached to one instance at a time: a second instance is `CONFLICT`, never a silent move. A [shared volume](#shared-nfs-volumes) takes many. |
| `detach`      | Detaches it (from `instanceId`, required for a shared volume). The volume stays.                                                                                                           |
| `resize`      | Grows it to `sizeGiB`. A smaller size is `INVALID_INPUT`: volumes only grow.                                                                                                               |
| `snapshot`    | Takes a snapshot (`name`, `idempotencyKey`). At most 25 snapshots that are not deleted per volume.                                                                                         |
| `delete`      | Deletes it at the provider. `INVALID_STATE` while it is attached: detach it first.                                                                                                         |

Attach refuses before any provider call: an instance of another tenant is `ACCESS_DENIED`; one on another provider
account or in another region is `INVALID_INPUT`; an instance being deleted is `INVALID_STATE`.

## Shared (NFS) volumes [#shared-nfs-volumes]

A volume with `kind: 'shared'` is a network file share: one filesystem that many instances mount at once over NFS.
On DigitalOcean it is a Network File Storage share. A share has no access control of its own beyond the network it is
reachable from, so a shared volume always names a [tenant network](/docs/guides/vps/networking) and is reachable only
from that network's instances.

```ts title="shared.ts"
const shared = await compute.api.providerVolumes.create(token, {
  tenantId: 'ten_01jz…',
  poolId: 'pool_01jz…',
  name: 'assets',
  region: 'nyc3',
  sizeGiB: 50,
  kind: 'shared',
  networkId: 'net_01jz…', // required: the share lives in this network's subnet in nyc3
  idempotencyKey: 'assets-1',
});
await compute.api.providerVolumes.attach(token, { tenantId: shared.tenantId, volumeId: shared.id, instanceId: 'inst_a…' });
await compute.api.providerVolumes.attach(token, { tenantId: shared.tenantId, volumeId: shared.id, instanceId: 'inst_b…' });
// get → { kind: 'shared', networkId, attachments: [{ instanceId: 'inst_a…', state: 'ready', … }, …],
//         device: '10.110.0.5:/2633050/…', mount: { source: '10.110.0.5:/2633050/…', type: 'nfs4' }, … }
```

How it differs from a block volume:

* **The network.** `networkId` must be a network of the tenant, on the pool's provider account, provisioned, with a
  subnet in the volume's region (`INVALID_INPUT` otherwise; `INVALID_STATE` while the network or that subnet is not
  ready yet, or the network is being deleted). The share is made in that subnet, never in the region's default VPC.
  `networkId` on a block volume is `INVALID_INPUT`. A network a shared volume is on is not deleted
  (`networks.delete` answers `INVALID_STATE` until the volume is deleted), and neither is a subnet of it in the
  volume's region (`networks.removeSubnet` answers `INVALID_STATE`, with the volumes in `details.volumeIds`).
  A provider that makes no shared volumes is `UNSUPPORTED` (`reason: 'sharedVolumes'`).
* **Many instances.** `attach` adds the instance (attaching one it is on returns the volume); the instance must be on
  the volume's network (`INVALID_INPUT` otherwise). The view lists every attachment in `attachments` (a block volume
  keeps the single `attachment`). `detach` names the instance, `{ volumeId, instanceId }` (`INVALID_INPUT` without
  it). `delete` is `INVALID_STATE` while any instance still has it.
* **No filesystem choice.** A share is a filesystem already: leave `filesystem` out (`none` is accepted and
  ignored). `deleteWithInstance` is always `false`: deleting an instance detaches it from the share and keeps the
  share.
* **Size.** It grows like a block volume and books `diskGiB` quota the same way. DigitalOcean shares are 50 to 16384
  GiB: a smaller size is `INVALID_INPUT` (`reason: 'size'`) at create, before any quota is booked.
* **Billing.** A shared volume bills `vps.share.gib.hours` (its GiB), not `vps.volume.gib.hours`, so a rate card
  prices shares apart from block storage.
* **Snapshots.** `snapshot` works as for a block volume; `providerSnapshots.restore` of a shared volume's snapshot is
  `UNSUPPORTED` (`reason: 'sharedRestore'`) for now.

On DigitalOcean, attaching tags the Droplet `bc-nfs-<16 hex>` (the record of which Droplets have the share) and adds
the Droplet's VPC to the share when it is not there yet; that VPC must be one of the same network. Detaching removes
the tag and takes the VPC off the share once no tagged Droplet is left in it; the share's last VPC stays. When an
instance on the share is deleted, the detach that follows finds its Droplet gone and takes off every VPC no tagged
Droplet is left in; deleting the share removes its tag. The hourly drift pass asks every ready
attachment of a settled shared volume again (an attach is idempotent), which puts back a VPC a racing detach took off.

### Mounting a shared volume [#mounting-a-shared-volume]

`device` (and `mount.source`) is the NFS source, `<host>:<path>`. better-compute does not mount it in the guest
yet; `sharedVolumeMount` from `@better-compute/vps` makes the lines for cloud-init or `/etc/fstab`:

```ts title="mount.ts"
import { sharedVolumeMount } from '@better-compute/vps';

const mount = sharedVolumeMount(shared.mount!.source, '/mnt/assets'); // { readOnly: true } mounts it ro
mount.cloudConfig; // ['10.110.0.5:/2633050/…', '/mnt/assets', 'nfs4', 'defaults,_netdev,nofail', '0', '0']
mount.fstab; //       '10.110.0.5:/2633050/… /mnt/assets nfs4 defaults,_netdev,nofail 0 0'
```

Put `cloudConfig` in the `mounts:` list of an instance's cloud-config user data (the image needs an NFS client, such
as `nfs-common` on Ubuntu). `_netdev` waits for the network and `nofail` lets the guest boot while the share is
unreachable. The helper refuses a source that is not `<host>:<absolute path>`, and a mount path that is relative, has
`.` or `..` segments or would cover part of the system (`/`, `/etc`, `/usr`, …), as for a sandbox volume's mount.
Mounting by bcd on pool members is planned, as for block volumes.

## Snapshots [#snapshots]

A snapshot outlives its volume. `providerSnapshots` has `get`, `list` (filter by `volumeId`), `delete` and `restore`:

```ts title="restore.ts"
const copy = await compute.api.providerSnapshots.restore(token, {
  tenantId: 'ten_01jz…',
  snapshotId: 'psnap_01jz…',
  name: 'pg-data-copy',
  idempotencyKey: 'restore-1',
  // optional: sizeGiB (not below the snapshot's), description, deleteWithInstance
});
// → a new provider volume, created from the snapshot
```

A restore waits until the snapshot is `ready`, and fails if the snapshot is deleted first. The restored volume keeps the
filesystem of the volume the snapshot was taken of: a restore takes no `filesystem` (`INVALID_INPUT`), since asking the
provider to format it could wipe the restored data.

## Addresses [#addresses]

```ts title="address.ts"
const ip = await compute.api.providerAddresses.create(token, {
  tenantId: 'ten_01jz…',
  poolId: 'pool_01jz…',
  name: 'ingress',
  region: 'nyc3',
  family: 'ipv4', // or 'ipv6'
  idempotencyKey: 'ingress-1',
});
await compute.api.providerAddresses.assign(token, { tenantId: ip.tenantId, addressId: ip.id, instanceId: 'inst_01jz…' });
// get → { address: '203.0.113.7', assignmentState: 'ready', instanceId: 'inst_01jz…', … }
```

`assignmentState` is `none`, `pending`, `ready`, `deleting` (an unassign running) or `error` (with
`assignmentMessage`). An address is on one instance at a time: assigning it to a second instance is `CONFLICT`, and to
an instance of another tenant `ACCESS_DENIED`. `unassign` releases it and keeps it reserved. `delete` is `INVALID_STATE`
while it is assigned.

The reconciler never takes an address over from a machine it did not put it on: if the provider reports it on another
machine, the assignment goes to `error` and nothing is moved.

Since a reserved IP carries no tag, the row is the ownership record, and one address is recorded by at most one live
row. Where the provider has one, the row also keeps the `fingerprint` from the reserve (on DigitalOcean a reserved
IPv6's reservation time). A reserved IPv4 has none: its project is shared by every reservation in the project, so it
cannot tell a re-reservation of the same IP apart.

An address released at the provider outside better-compute is lost for good. The reconciler notices it on its next
read (the IP is gone, or another reservation's fingerprint or region shows there), or when a later reserve of the same
account is handed that IP. The row then goes to `error` saying so, forgets the provider address and its instance, and
never calls the provider again: it is never re-adopted when the IP is handed out again, never assigned, and never
released. `assign` is `INVALID_STATE`; `delete` finishes the row (refunding its quota) without a provider call. The
provider refuses a mismatched address too (`INVALID_STATE`, reason `notManaged`).

A volume deleted at the provider outside better-compute is not made again either: once a volume was created, the
reconciler only reads it, and one the provider no longer has goes to `error` (its data is gone). Delete it.

### Lost reserves [#lost-reserves]

DigitalOcean's reserved IPs carry no name or tag, so a reserve whose answer never arrived (a timeout or a dropped
connection) cannot be found again. Rather than risk a second address, the address goes to `error` with a message
saying so, and is never reserved again. Delete it, and check the account's reserved IPs for one that is not assigned. A
reserve that was refused or rate-limited reserved nothing, and is retried.

## Deleting an instance [#deleting-an-instance]

Deleting an instance takes care of its volumes and addresses:

1. While the machine is being deleted, a volume with `deleteWithInstance` makes no provider call (an ensure by name
   could otherwise make again a volume the delete already took).
2. The machine delete names the instance's `deleteWithInstance` volumes. On DigitalOcean that is one
   `DELETE /v2/droplets/{id}/destroy_with_associated_resources/selective` naming only volumes that carry this
   deployment's managed tags and are attached to that Droplet. A tenant's own volume attached by hand is never named
   and survives. With nothing to name it is a plain `DELETE /v2/droplets/{id}`. Tag-scoped bulk deletes are never used.
3. When the delete finishes, the instance's attachments are no longer wanted, its `deleteWithInstance` volumes move to
   `deleting`, and every volume and address it had is woken. The reconciler settles the detach, deletes what is to go
   (already gone after a selective destroy; a provider without one deletes it after the machine) and unassigns its
   addresses, which stay reserved.

Once the instance's delete is asked, `detach` of a `deleteWithInstance` volume is `INVALID_STATE`: it goes with the
machine. Detach it before deleting the instance to keep it; a volume detached before the delete is never deleted with
it.

An instance that finishes as a **detach** (an attached machine with `onDelete: 'detach'`, or one whose ownership label
is gone) leaves its machine running. Its volumes are only detached, `deleteWithInstance` or not.

## Quota, billing and events [#quota-billing-and-events]

* A volume books its `sizeGiB` as `diskGiB` quota on (tenant, pool), the resource a provider instance's disk books;
  growing it books the difference. An address books one `addresses`. A tenant without a quota row for the resource
  gets nothing (`QUOTA_EXCEEDED`, fail closed). The booking is refunded once the provider has deleted the resource.
  Snapshots book no quota.
* On a metered pool, each resource bills from the moment it was `ready` until it is deleted, in whole-hour usage
  windows written by `vps.ancillary-usage`: a snapshot `vps.snapshot.gib.hours` (its GiB), a block volume
  `vps.volume.gib.hours` (its GiB), a shared volume `vps.share.gib.hours` (its GiB) and an address `vps.address.hours` (one per hour). One that never became ready never
  bills; a BYO account's pool (platform fee) bills none of them. Volumes, snapshots and addresses are all metered by
  the `vps.ancillary-usage` pass (usage source kinds `volume`, `snapshot` and `address`).
* Events: `providerVolume.*` (created, updated, ready, attached, detached, resized, error, deleting, deleted),
  `providerVolume.snapshot.*` and `providerAddress.*` (created, updated, ready, assigned, unassigned, error, deleting,
  deleted). See the [event catalog](/docs/reference/events).

## Permissions [#permissions]

| Group               | Read             | Everything else    | Resource                 |
| ------------------- | ---------------- | ------------------ | ------------------------ |
| `providerVolumes`   | `volumes:read`   | `volumes:manage`   | `provider-volume/{id}`   |
| `providerSnapshots` | `volumes:read`   | `volumes:manage`   | `provider-snapshot/{id}` |
| `providerAddresses` | `addresses:read` | `addresses:manage` | `provider-address/{id}`  |

`list` is decided on the tenant. A create is decided on the new resource, a restore on the snapshot and a snapshot on
its volume.

`providerVolumes.attach`, `providerVolumes.create` with an `instanceId` and `providerAddresses.assign` put the volume
or address on an instance, so they are also decided on that instance: the caller needs `instances:update` on
`instance/{instanceId}` as well, with its own credential. A role that manages volumes in one project cannot attach one
to an instance it cannot change.