rendered from docs/deployment.md 1f957838f73a

Deployment

What running this safely requires. The shape of the system draws the same hosts; this page states the rules, and operations is the procedure an operator runs against them.

What runs

A container image, started by one systemd unit, on one instance. Not a binary and a unit file; not an orchestrator. scripts/ is a developer’s interface and no part of a deployment.

scripts/publish-image.sh builds the image from the repository’s Dockerfile with --features route53, tags it by git sha and, optionally, by a release name, and pushes both to terraform-aws-didbot-pds/ecr.tf’s own repository. GET /health reports the sha as revision.

infra/pds/ is did.bot’s deployment. It calls terraform-aws-didbot-pds, a module in its own repository that infra/pds/main.tf pins by tag; a terraform-aws-didbot-pds/ path below is a file in that repository. infra/pds/templates/didbot.toml.tftpl stays here, beside the source whose settings it names, and the module renders it.

What runs is the release parameter’s: terraform-aws-didbot-pds/release.tf writes the image, by digest, and the rendered didbot.toml into one SSM parameter. terraform-aws-didbot-pds/templates/user_data.sh.tftpl names that parameter and nothing it holds. It installs a converger, terraform-aws-didbot-pds/templates/converge.sh.tftpl, that a timer runs every thirty seconds. When the parameter changes, the converger pulls the image, writes /etc/didbot/didbot.toml and /etc/systemd/system/didbot-pds.service, and restarts the unit. The image reference and the hosted zone id go directly into the unit, so systemctl cat didbot-pds and cat /etc/didbot/didbot.toml between them show exactly what is running, with no environment file to also go read.

Three properties decide this, and each of them is a property of the process rather than a preference:

• The unit must be a single-writer service. The store is one write-ahead log behind one lock, and two live writers interleave frames neither can replay. terraform-aws-didbot-pds/ec2.tf is one instance, not an autoscaling group; the unit runs the container under a fixed --name with an ExecStartPre that removes a previous one, so a restart racing a slow shutdown cannot produce two. Systemd is not the only way to start a process on a host, so the exclusion does not rest on it: didbot_pds::lock takes an exclusive flock on pds.lock inside the data directory before the log is opened, and an operator running the binary by hand against the mounted volume is refused by name. The kernel releases the lock when the process dies, so a crash leaves nothing to clean up.
• The image is a name that can be said. The exit criterion for deploy is that a tagged release runs on a host nobody develops on. A container digest is that name, and a git pull on the instance is not.
• The process binds 443 and holds its own certificate. Nothing terminates TLS in front of it, so the run unit uses --network host rather than a published-port mapping — which is also what lets the container’s IMDSv2 request reach the metadata endpoint at http_put_response_hop_limit = 1. Raising that to 2 is required if the run unit ever leaves host networking. The image runs as an unprivileged user, and with the host’s network namespace there is no net.ipv4.ip_unprivileged_port_start to relax, so the binary carries cap_net_bind_service=+ep — one capability, bind a low port — rather than the container running as root.

Restarts are Restart=on-failure with RestartSec=5, capped at five failures in two minutes so a crash loop stops rather than burning the instance. A restart is not free: a sign-in in flight — a pushed authorization request, a consent reference, an authorization code — is in-process memory and does not survive one, though an OAuth grant already issued does; see “say what a restart does” in deploy.

Deploying

scripts/deploy-pds.sh --var-file <tfvars> <rev> deploys a commit in place. It publishes the image, plans with container_image_tag set to the commit’s sha, refuses a plan that replaces anything unless given --allow-replace, starts an on-demand backup of the data volume, applies, and writes the tag back into the var file. Then it runs the converger over SSM, prints what the converger said, and waits for /health to report the commit.

The instance and its volume stay. The container restarts on the new image:

1. The old process takes a checkpoint while it still serves, at most [server] stop_checkpoint_secs.
2. It stops accepting, waits at most [server] drain_secs for requests in flight, closes each subscribeRepos stream with close code 1001, syncs the log and exits.
3. The new process binds the port, replays the log past the checkpoint and starts serving. A connection made while it replays waits rather than being refused.

The new container has fifteen minutes to start listening, which covers replaying its log, and then two minutes to answer /health three times in a row. A container that exits fails at once. On a failure the converger writes the release to /var/lib/didbot/release.refused, so the timer does not try it again, and restores the previous image and config. The previous image is kept on the instance, so this needs no registry. A rollback that did not finish, because a pull failed or the instance restarted partway, runs again on the next tick. The deploy script exits saying so. An image that has rewritten the data directory’s layout cannot be rolled back this way, because the earlier image refuses the directory; the backup is the way back, see operations.

Writing the release parameter chooses what runs on the instance. The parameter is a SecureString under terraform-aws-didbot-pds/kms.tf’s release key, and only release_writer_arns can encrypt under that key; by default that is the role that applies. The instance decrypts under no other key, and the converger runs only an image in the stack’s own repository, by digest.

A change to the instance itself still replaces it: the AMI, the instance type, the module’s user_data.sh.tftpl or converge.sh.tftpl, or a nested zone’s hosted zone id in the unit. So can a new module version: plan it before moving the pin. That costs minutes and is what --allow-replace is for.

Bring-up

In order, against an empty account. Each step depends on an earlier one.

1. Supply the variables infra/pds/variables.tf has no default for: root_zone (the zone this deployment answers for), ami_id (a pinned, reviewed AMI id — Amazon Linux 2023 or Debian), instance_subnet_id (a public subnet, so the instance has egress on its first boot), and operator_did. Pick container_image_tag now too — the 12-character sha of the commit to deploy (git rev-parse --short=12 <rev>) or a release name — and carry the same value through every step below. A bring-up meant to reach step 8 also sets acme_environment = "production": steps 7 and 8 need a certificate the calling machine trusts. Set alarm_sns_topic_arns to an SNS topic in us-east-1 whose subscription is confirmed: the health alarm notifies the topics it lists, and its default lists none.
2. tofu -chdir=infra/pds init -backend-config=backend.hcl (see the state bucket), then tofu -chdir=infra/pds apply -target=module.pds.aws_ecr_repository.pds -target=module.pds.aws_route53_zone.root creates the image repository and the hosted zone, and nothing that runs.
3. scripts/publish-image.sh <rev> [<release-name>] builds <rev> and pushes it tagged by its 12-character sha, and by <release-name> when one is given. container_image_tag is whichever of the two step 1 picked.
4. Write the four name servers from tofu -chdir=infra/pds output name_servers into infra/site/pds.auto.tfvars as pds_zone_name_servers = [...], and commit it. Then tofu -chdir=infra/site apply -var release_sha=<sha> delegates root_zone from did.bot, where <sha> is the tree did.bot serves now (tofu -chdir=infra/site output release_sha). Tofu reads the file on every plan and apply, so each later site publish keeps the delegation.
5. Wait for the delegation to propagate: dig NS <root_zone> answering those four name servers, from a resolver outside this account, is enough.
6. tofu -chdir=infra/pds apply creates the rest: the instance, the IAM role, the release parameter and the zone’s address records. The converger starts the unit, which obtains its certificate over DNS-01 in the zone step 4 delegated.
7. didbot operate <root_zone> <operator_did> claims the deployment. A successful apply does not perform this step, and every write answers ServerNotReady until it does.
8. Confirm GET https://<root_zone>/health answers 200 and GET https://<root_zone>/xrpc/bot.did.listAccounts answers, both over a certificate the calling machine trusts.

A run with acme_environment = "staging" is checked with tools that skip certificate verification, such as curl -k. Its certificates chain to a root nothing trusts, and didbot operate verifies against a bundled root store. did.bot’s live Strict-Transport-Security header carries includeSubDomains, and a browser that has seen it refuses a staging certificate under every did.bot name, with no way through. Staging is for a zone outside did.bot.

The server sends its own Strict-Transport-Security header only under acme_environment = "production". [tls] hsts_max_age_secs sets its max-age, two years unless set. [tls] hsts_include_subdomains adds includeSubDomains, and is off unless set: turn it on only when every name below root_zone is one this server serves.

The zone

One zone, used for everything: the server, the agent handles, and the authority serving every did.json. HTTPS throughout.

The server sits at the zone’s apex and accounts one label below it, where *.<zone> answers. Accounts are off the apex because a wildcard certificate never covers the bare parent, so the server’s own name stays its own. A name deeper than one label would still resolve — a DNS wildcard synthesizes at any depth — but the certificate covers exactly one, so createAccount refuses it and names the zone that would carry it; the names an agent has puts the two rules side by side. Which server serves a given account is answered by resolving its DID document, not by comparing hostnames.

Several zones

[zone] zones may list more than one. The first is the primary: the server’s own hostname and account. Every zone mints accounts one label under it:

[zone]
zones = ["pds.example", "agents.pds.example", "pipelines.pds.example"]

A name lands in the zone it ends in. A zone’s own apex is no account’s name, and a label is unique across zones: kestrel.agents.pds.example keeps kestrel from every other zone.

Each zone needs its own A records at <zone> and *.<zone>, and its own --route53-zone-id <zone>=<id>. A nested zone kept in its parent’s hosted zone passes the parent’s id and still needs its own wildcard: once _acme-challenge.agents.pds.example exists, *.pds.example stops answering for names under agents.pds.example.

infra/pds does this for each entry in nested_zones. An entry with no hosted_zone_id keeps its records in root_zone’s hosted zone. The operator delegates an entry’s own hosted zone. With no [[pool]] in the rendered config, each nested zone is a pool named after itself.

Pools

A pool is a named source of names: some labels under one or more zones, and a scheme that makes them. With no [[pool]], each zone is a pool named after itself, and a create that names nothing lands in the first.

[[pool]]
name = "agents"
zones = ["agents.pds.example"]

[[pool]]
name = "ci"
zones = ["pds.example"]
labels = "ci-*"
names = "{repo}+n"

labels holds part of a zone and wraps each name the pool makes. A createAccount asking for pool: "ci" with the fragment repo: "didbot" takes ci-didbot-4813. A name asked for outright has to fall in a pool. --help lists each key’s range.

Two hostnames per account — one for the DID, one for the handle — and both must resolve before the account exists.

One wildcard record makes both of them resolve. The zone holds three names whatever the account count: the apex, *.<zone>, and _acme-challenge.<zone> while a certificate is being issued. terraform-aws-didbot-pds/route53.tf writes the wildcard beside the apex, both at the instance’s address, and provisioning writes no record of its own because the wildcard already answers. The names an agent has works one account through the whole path, from what exists in the zone to what a stranger reads.

The cost is that there is no per-name existence guarantee and no DNS-level garbage collection: a name no account holds still resolves, and answers 404 rather than NXDOMAIN. That is the correct trade here — DNS removal was never a revocation mechanism, since a withdrawn name stays cached downstream for its TTL, and an HTTP status is the only thing that can express a decommissioned account still serving its document.

A timer reaps what a killed process left. Provisioning unwinds its own row on every error it returns through; a process killed between storing an account and activating it runs none of that unwinding. didbot_serve::stale_sweep runs Provisioner::sweep_stale_bounded hourly. It removes a reserved or provisioning row once the row is a week old, whatever the account’s kind, and finishes an erasure a crash left half done. Every other account stays until the operator deletes it. One tick takes a hundred rows at most, and the next tick takes what the last left.

The DNS credential

Control of the zone is control of every name in it. It is as powerful as the signing keys, and it lives beside them: the server writes the _acme-challenge TXT records its certificate depends on and the CAA record below, so the credential has to be reachable from the process. Those are the only records it writes, and agent hostnames cost none. Holding it elsewhere would protect nothing: whoever holds this host holds the signing keys, and with those they can sign as any account it serves.

So the question is not where the credential sits. It is what it reaches, and what a holder of it can be caught doing.

What it can do

Precisely, for the record names and types the grant pins — the _acme-challenge.<zone> TXT and the apex CAA, per configured zone:

• Pass DNS-01 for a zone’s own names. _acme-challenge.<zone> is where the challenges for both <zone> and *.<zone> land, so a holder can obtain a publicly trusted certificate covering the zone and every agent hostname under it — names no account holds included.
• Rewrite the CAA record, which is the step that makes the one above reachable through an authority other than the one this zone authorises.

Serving a document at a name this server never minted is not on that list, and it is worth saying why: under a wildcard every name in the zone already resolves here without any DNS write at all. What answers at one is this server’s business, not DNS’s — which is why a name no account holds answers 404 rather than NXDOMAIN.

What it does not reach is any other record — the apex and wildcard A records every name resolves through stay as Terraform wrote them — and anything outside these zones: other zones in the account, and the NS records at the registrar that delegate into this one, which live in the parent zone. That boundary is the IAM policy below and nothing else, which is why the policy is the load-bearing part.

What is detectable

Detection is what bounds a compromise here, because isolation cannot. Two channels, and neither puts its evidence somewhere the holder of this host can reach.

Certificate transparency. Every publicly trusted certificate is logged by the authority that issued it, to logs this deployment does not run. A certificate naming any hostname in the zone that this server did not order is visible there, and only there: no record of it exists on this host, and nothing about the issuance touched it. Subscribe an external monitor to the zone. This is the one channel a compromised host cannot suppress, and it is after the fact — the certificate already exists when the log entry appears.

A refused issuance, reported by the authority. The CAA record the server publishes at the zone apex names one authority and, per RFC 8657’s accounturi, the one ACME account at it this deployment holds the key for. An attempt to issue for a name in this zone through any other account is refused by any authority honouring the record, and reported to the iodef contact in it — a report that arrives from the authority, not from this host. This does not stop the holder of the zone credential, who can rewrite the record. It makes doing so a second, separate act, and it closes issuance to everyone who does not hold the credential.

The zone’s shape is fixed — the apex, *.<zone>, the CAA at the apex, and _acme-challenge.<zone> while an order is open — so a further name is a name somebody else added. Reading the zone to compare it against that shape is an operator’s act at the console, with the operator’s own credential; the instance role’s grant is the write below.

What is not recoverable

Rotating the credential and rewriting the zone recovers the zone. It does not recover any of the following, and no procedure does:

• Confidence in anything signed before the compromise was noticed. A swapped did:web document leaves no trace of the key it replaced. A record signed by the agent and a record signed with the substituted key are indistinguishable afterwards, in both directions — the compromise does not only forge new records, it makes the real ones unprovable.
• Whatever resolved during the window. A repointed name was cached by every resolver that asked, and served to every client that followed it.

A certificate is the exception: an issued certificate can be revoked, and the authority that issued it is who to ask.

The policy that bounds it

crates/didbot-dns’s Route53Dns is the provider that scopes this: an IAM credential given only

route53:ChangeResourceRecordSets
  on arn:aws:route53:::hostedzone/<HOSTED_ZONE_ID>
  when every record in the change is a TXT or CAA, at
    `_acme-challenge.<zone>` or the zone apex
route53:GetChange
  on arn:aws:route53:::change/*

can write the record sets the server maintains, and nothing else in the zone or the account. GetChange has to be unscoped to a zone because Route53 change ids are not namespaced under one; it grants read of a change’s propagation status, not write of anything. A cloud DNS service is the right kind of provider for exactly this reason — the credential narrows to a single zone — and across several servers that narrowing is the blast-radius boundary between them. See the module docs on Route53Dns for the full policy JSON and for how it handles eventual consistency, rate limits and hosted-zone record limits, all things this posture note assumes are handled correctly one layer down.

The boundaries this design actually rests on are elsewhere, and neither is inside this host: the agent hosts are separate machines from this one, and the policy that governs it lives in a repository this server holds no credential for. A compromised server can break policy; it cannot change it.

Certificates

The server terminates TLS itself. Nothing in front of it holds a certificate, so it obtains and renews its own over ACME DNS-01, through the same zone credential above (crates/didbot-tls, on instant-acme).

One <zone> + *.<zone> pair per configured zone. A wildcard in a certificate matches exactly one label: *.pds.did.bot covers claudes.pds.did.bot and does not cover agent.claudes.pds.did.bot. This is not the DNS rule — a wildcard record of the same spelling answers at any depth — so what bounds how deep an agent hostname may sit is the certificate, not the zone, and a name is one label under its zone. See the names an agent has. A deployment configured with claudes.pds.did.bot and vllm.pds.did.bot therefore holds two certificates, each from one ACME order carrying both of that zone’s identifiers — the apex and the wildcard — whose two DNS-01 challenges share one _acme-challenge.<zone> record and must be published together. didbot_tls::fleet::CertificateFleet holds one manager per zone and ZoneResolver picks between their certificates on the SNI name each handshake carries. A name no configured zone covers is refused rather than handed a certificate that does not name it.

A renewal never rebinds the listener. The certificate lives behind an ArcSwap that rustls reads once per handshake, so a renewal is a pointer store: connections already established — the firehose relays and indexers hold open for hours — are untouched, and the next handshake sees the new certificate.

A failed renewal keeps serving what it has, and gets louder. Renewal starts 30 days before expiry and retries hourly; every failure leaves the current certificate in place and is logged with the zone, the failure count and the expiry it is measured against. Three consecutive failures raise that zone to Warning; inside three days of expiry it is Critical whether or not anything has failed, because at that point expiry itself is the risk. CertificateFleet::alert_level reports the worst level any zone is at and zone_alert_levels says which zone it came from. GET /health carries both, under certificates: worst is the summary an alarm expression matches on, zones names each zone and its own level. terraform-aws-didbot-pds/monitoring.tf is what reads it — a Route53 health check string-matching worst, and a CloudWatch alarm on that check — so a zone above ok and a server that stopped answering raise the same alarm. An expired certificate is a total outage for every hostname in that zone, agents included.

The ACME directory a run is configured for decides what it serves. acme_environment is production by default (infra/pds/variables.tf); a staging leaf is signed by an authority no client trusts. infra/site/’s pds_serves_a_production_certificate decides whether did.bot’s two-year HSTS header covers the names this stack serves, and a browser that has seen that header refuses a certificate error under every covered name with no way to click through. The certificate on disk is served only when meta.json records the directory this run is configured for, and the saved ACME account is restored only when its credentials record that same directory; otherwise the run registers an account there and orders a certificate from it, which is what makes the switch to production take effect on the first boot after it. A restart under an unchanged environment places no order and serves the certificate that is already on disk, unless that certificate cannot be served: a file missing or unparsable, or a key that is not the certificate’s, as a crash mid-save can leave. Boot then orders a new one. The CAA record below is republished from the account actually in hand before that order is placed, so a switch carries the accounturi with it.

The zone’s CAA record is written by the server, on every start. It is published before the first order and restated on each start, naming the authority the order is about to go to and the ACME account the order will be placed through — didbot_tls::dns::IssuancePolicy, reading both off the live objects rather than off configuration. That is deliberate: a CAA record naming an account this deployment cannot use does not fail safe, it stops the next renewal, and an expired certificate is a total outage for every hostname in the zone. Reading both halves from the objects the order itself uses makes that disagreement unreachable. A backend whose zone already says exactly this skips the write. There is no issuewild property — RFC 8659 falls back to issue for a wildcard request when none is present, and the wildcard is in the same order — and failing to publish the record is a warning rather than a startup failure, because a zone with no CAA record is the state every zone starts in.

The contact is what the authority warns. [tls] contact in the rendered config is the address Let’s Encrypt registers against this deployment’s ACME account and sends an expiry warning to; infra/pds/variables.tf’s acme_contact supplies it, and the rendered file carries the key only when that variable is non-empty. Empty is a supported deployment and the one this configuration defaults to: an ACME account with no contact registers, orders and renews identically, so the address buys a second warning beside terraform-aws-didbot-pds/monitoring.tf’s alarm rather than the ability to hold a certificate. Demanding one would make a deployment’s boot depend on a field the CA does not require, and an instance that does not come up is every agent on it offline. Terraform refuses anything but a mailto: URI or the empty string at plan time, because a contact Let’s Encrypt rejects fails at account registration, which is a startup this deployment does not come back from.

Where the address goes. Two places, both by design. IssuancePolicy publishes it as the RFC 8659 iodef value of the zone’s CAA record, which is what points a refused issuance at somebody the authority can reach without going through this host — so it is in public DNS, and the address to use is one suited to that. The DNS backend logs the record set it wrote at INFO when the set changes, so it is also in the instance’s journal. The two public surfaces publish their own complete lists: wire::StatsResponse is counts and byte totals over this deployment’s contents, and GET /health is the tick and per-zone certificate alert levels.

The server is handed the zones it may write. Each zone needs a hosted zone id configured for it (--route53-zone-id <zone>=<id>), and under --tls acme the server refuses to start while any zone lacks one.

Announcing to relays

A relay learns a personal data server exists by being told. This deployment names its relays in the relay_hostnames Terraform variable, and infra/pds/templates/didbot.toml.tftpl puts them in the rendered config’s [relay] hostnames. The default is bsky.network, the public relay, and relay.vibescrobble.com, which vibescrobble.com reads agents’ records from. [] names none. Those keys decide which relays com.atproto.sync.requestCrawl and the deprecated notifyOfUpdate are sent to; sending either is an operator action, and each announce asks every relay.

[relay] takes two keys. hostname names one relay, and hostnames a list of them; a server announces to every relay either names. Zero to a few relays is the useful range: each is asked at once, with its own ten-second limit. --relay-hostname, given once per relay, replaces both.

Boot does not dial the relay. Naming a relay records which one an operator can announce to, and nothing more. A relay that is down while this instance comes up therefore costs the instance nothing — there is no retry loop to wedge in and no startup step to fail. A relay that goes away later, or one that answers HostBanned, is felt by the operator who asks: the reply lists each relay with accepted, refused (with the relay’s status and body) or unreachable (with the transport error), and a call that gets no answer gives up after ten seconds.

Who may ask. POST /dashboard/api/relay/announce, behind the operator sign-in — from the dashboard, or from the operator’s own machine:

$ didbot login --server pds.example
$ didbot announce --server pds.example
{
  "relays": [
    {
      "relay": "https://bsky.network",
      "result": "accepted"
    },
    {
      "relay": "https://relay.vibescrobble.com",
      "result": "accepted"
    }
  ]
}

Under a .localhost zone the flag is read and left unwired, and the server says so at startup: no relay resolves a loopback name, so the only thing wiring it up there could produce is a failure an operator asked for.

The data directory

One directory, --data, on a volume separate from the root volume so an AMI upgrade or an instance resize can re-attach it rather than lose it. The instance mounts it at /data and the container sees /data/pds as /data.

Everything else a deployment has — its configuration — is elsewhere and is not state.

Permissions are set on every open, not only on creation. The signing keys are in the log, so the directory holding it is only as private as whoever made it. A directory the server creates is 0700 from DirBuilder::mode; a directory it was handed — an operator’s mkdir, a mount point, a restored snapshot’s filesystem root — carries whatever umask made it. didbot_pds::wal::create_dir therefore chmods an existing directory to 0700 every time it runs rather than trusting it, and didbot_tls::storage::create_dir does the same for tls/. That costs one syscall on the common path and closes the case that actually matters.

What it does not fix is ownership. A directory owned by another user is not made safe by its mode, and the server does not check: it will fail to write and say so, which is correct but is a boot failure rather than a diagnosis. Ownership is the boot script’s. The image runs as uid 10001, pinned in the Dockerfile rather than left to useradd, and user_data.sh.tftpl chowns /data/pds to that uid before the unit starts — which also re-homes a restored snapshot whose files were written by a different uid.

The log’s budget

[capacity] log_budget_bytes in the rendered config is the number of bytes the log may reach; past it an append is refused before a byte is written, the file stands exactly where it was, and the write comes back as 507 StorageFull. infra/pds/variables.tf‘s log_budget_bytes sets it, and takes a tenth of data_volume_size_gb unless a number is pinned — 2 GiB on the default 20 GiB volume. The blob store shares this volume and grows in megabytes at a time where the log grows in hundreds of bytes; “The blobs’ total” is its bound.

The number bounds the log written since the last checkpoint. A checkpoint is the state as one entry per live thing, in pds.checkpoint beside the log, and once it is written every whole segment before it is deleted. The running server takes one once the log reaches [capacity] checkpoint_log_bytes, half the budget unless set, and once [capacity] checkpoint_interval_secs passes with anything written, a day unless set. A boot takes one once the log since the last has grown past the state. A released name, a deleted account, an overwritten record and a reissued credential are history a checkpoint drops, so a log filling on churn is trimmed without a restart — didbot-pds/tests/durability.rs asserts it. While a checkpoint runs the process holds a second copy of the state in memory, and the volume holds the new pds.checkpoint beside the old one.

An operator sees this in four places. The startup line the write-ahead log will refuse writes past its budget carries the budget and the bytes held, on every boot. wrote a checkpoint while serving and trimmed the log against it carries the entries it kept, the trimmed_bytes, the log_bytes left and how long it took_ms; taking a checkpoint while serving before it names the reason. wrote the checkpoint the next boot reads the log from is the same from a boot. And a deployment at its budget answers 507 whose message names the bytes held, the budget, and the bytes the refused entry wanted. A log that reaches the budget is one written faster than checkpoints trim it, or one whose checkpoints fail, which logs a checkpoint while serving failed at ERROR: lower checkpoint_log_bytes, raise log_budget_bytes, or raise data_volume_size_gb and with it the tenth this derives.

Each account’s share of the log

The budget is shared, so [writes] bounds how fast one account can spend it between checkpoints. Each account has two token buckets, held in memory and full at every start. One counts write operations: a record written or deleted, a write inside applyWrites, a blob uploaded. The other counts request body bytes. A write is charged once write policy allows it, so a write policy refuses costs the account nothing. A write over either is refused with 429 RateLimitExceeded and a Retry-After, and logged at WARN as write refused: the account is over its write rate, naming the repo.

A write adds about 1 KiB to the log whatever the record’s size, so the operation bucket is the one that bounds the log. At the defaults one account writes at most 1,000 + 3,600 operations in an hour: about 4.5 MiB, 0.2% of the 2 GiB budget, a small fraction of what the 1 GiB watermark lets build up between two checkpoints. Its request bodies come to at most 8 + 64 MiB an hour, which bounds how fast it grows the record heap. didbot-pds --help gives each key’s range.

Each account’s record count

[capacity] max_records_per_account bounds how many records one repository holds: 50,000 by default, 1,000 to 100,000 accepted. A write to a repository the server holds built costs about the same at any size. A write to one it does not hold, after a restart or once more than 32 repositories have written since, builds it from every record, and that grows with the count:

crates/didbot-pds/tests/write_scale.rs measured these, on a 16-thread laptop under other builds.

A write that would add records past the ceiling is refused with 403 TooManyRecords, naming the count and the ceiling, and the refusal journal writes it at WARN naming the account. Updates and deletes are always accepted, as is a batch that deletes as many records as it creates. The server’s own records, such as the profile and bot.did.operator, are written past the ceiling and count towards it. Lowering the ceiling deletes nothing: an account already past it keeps its records and can update and delete them, and creates again once it is under. The dashboard’s account list shows each account’s record count.

The blobs’ total

[blobs] total_quota_bytes in the rendered config is what every account’s blobs may come to together. terraform-aws-didbot-pds/ec2.tf sets it to half of data_volume_size_gb — 10 GiB on the default 20 GiB volume — because each account’s own 64 MiB quota times the 1,000-account cap comes to more than the volume holds. The other half holds the log, two checkpoints while a new one is written, and the record heap, which grows with every record written.

Past the total an upload is refused with 507 StorageFull, and the message names the bytes held and the total. The startup banner’s blob total line states it. A blob the collector takes, and a deleted account’s blobs, give their bytes back. GET /health’s disk is the volume’s free and total bytes at the moment of the request, and it moves with everything on the volume, not only the blobs. Its alert names the free-space watermark the volume is below, and terraform-aws-didbot-pds/monitoring.tf alarms on it; see Disk filling.

Secrets, and where each lives

Three, and the process holds all three itself:

The DNS credential is the only one that is not a file, and deliberately: the instance profile in terraform-aws-didbot-pds/iam.tf grants ChangeResourceRecordSets pinned to the _acme-challenge TXT and apex CAA records, and GetChange unscoped. crates/didbot-dns/src/route53.rs reads that role’s short-lived credentials over IMDSv2 itself, and signs its own SigV4 requests with them. Nothing is passed on the unit’s command line and nothing is fetched into a file at boot.

The operator’s DID is in the rendered config file’s [zone] operator, not here. It is an identifier, not a credential — this server holds nothing that lets it write the operator’s repository, which is the whole reason policy can constrain it.

What one compromised host gets an attacker: all three. They are on one box by design, and there is no ordering that changes the answer — the keys sign as any account the server serves, and the zone credential extends that to names it has never minted. Separating the credential from the keys is not a mitigation here and is not offered as one: it has to be reachable from this process to write DNS-01 challenges, so whoever holds the process holds it. What bounds a compromise instead is scope and detection, and the DNS credential above states both.

Backups and restore

The commands are in operations; this section states what a restore is and is not.

terraform-aws-didbot-pds/backup.tf puts the data volume under an AWS Backup plan: daily at 09:00 UTC, snapshots kept thirty days. The volume is the whole of the state — the log and the blobs are both on it — so one snapshot is one consistent deployment, give or take writes in the seconds around it.

A restore recovers accounts, records, blobs, name holds and ledger history, because those are the log. It recovers the signing keys with them, which is what makes the restored accounts the same accounts rather than new ones with the same handles.

The policy evaluation log’s segments keep their real modified times across a restore, so a snapshot older than [capacity] evaluation_log_retention_days has its stale segments trimmed the moment the restored server opens the directory — trim runs at startup by wall-clock age, not by anything the snapshot itself records — so an operator restoring history specifically to investigate old denials must raise the retention before starting the restored server, not after. The write log’s segments are trimmed the same way, against [capacity] write_log_retention_days.

A restore does not recover five things, and each fails differently:

• Sign-ins in flight. The operator’s dashboard session and the pushed authorization requests behind an unfinished sign-in are in-process memory. A restore is a restart, so each has to be started again. An OAuth grant already issued is journaled and comes back with the scope it was granted.
• The DNS zone. Snapshots are of a volume, not of Route53. A zone that was also lost has to be recreated and re-delegated at the registrar before any hostname resolves; the apex and wildcard records are then Terraform’s to write again, and the CAA is the server’s once it is running. A zone that survived holds nothing per account, so a restore older than the zone has nothing in it to clean up: a hostname provisioned after the snapshot still resolves, and answers 404.
• A usable certificate. tls/ is on the volume, so a certificate restored within its validity is served immediately. One restored from a snapshot older than the certificate is expired, and the server will re-issue over ACME DNS-01 — which needs the zone above to be working first, and is subject to the issuer’s rate limits on a name that has been issued repeatedly.
• Anything written since the snapshot. Up to twenty-four hours of accounts and records. There is no point-in-time recovery here and no log shipping.
• The position of com.atproto.sync.subscribeRepos. The restored log carries the snapshot’s sequence floor and heads, and a relay holds the lost run’s. The restore step in operations leaves pds.restored, and the boot that finds it raises the floor past the lost run’s numbers and resets each live repository with #sync over a commit newer than any the lost run signed. A relay that reconnects gets #info OutdatedCursor and those frames, and a consumer that follows the sync spec fetches each repository whose tree moved back. See conformance.

The restore drill. Step 4 is the whole of it: a snapshot that attaches and mounts proves the filesystem, not the log.

1. Create a volume from a recovery point in the <name_prefix>-pds vault (didbot-pds by default), in the instance’s availability zone.
2. Attach it to a fresh instance — never to the running one; two mounts of one log is the corruption pds.lock exists to refuse.
3. Mount it at /data, where the fresh instance’s boot script chowns /data/pds to uid 10001, and run didbot-pds --data /data/pds against a development zone.
4. Confirm the accounts that should be there are there: bot.did.listAccounts returns the expected count, one agent’s did.json resolves out of the restored keys, and a blob that account referenced fetches back byte for byte.
5. Detach and delete the test volume.

Step 4 runs in this workspace against a copied data directory rather than a volume: crates/didbot-serve/tests/restore_drill.rs takes a snapshot of a live directory, serves the copy, and requires the two servers to answer the same listAccounts, getRecord, did.json and bot.did.stats — with the blob’s bytes fetched back through getBlob. What steps 1 through 3 add is the volume itself and the uid the instance’s boot script sets.

Where to run it

One server serves many agent hosts. Agents are grouped, and a compromise of one host does not reach the others.

The first row is the development posture. The step to the second is the one that matters and the cheapest: it moves the keys and the zone credential off the machine running agent code. The third also makes browser isolation moot — one sandbox per session is one browser profile per agent — and allows instance-identity attestation, which is stronger evidence than a credential file on a shared host.

Agents are capable and sandboxing between them is not guaranteed. Separating agent hosts from the server host is achievable; separating agents from each other on one host is not. Record what cannot be enforced.

The state bucket

infra/pds/ and infra/site/ keep their OpenTofu state in one S3 bucket, one key each, so neither root can lock or overwrite the other’s. Both roots are partial backends: the key stays in backend.tf, and the bucket is read at init time out of backend.hcl.

cd infra/pds && tofu init -backend-config=backend.hcl

The same line applies in infra/site/. Deploying to another AWS account means pointing init at your own file – -backend-config=mine.hcl, or -backend-config="bucket=..." for a CI job that already holds the name in its own configuration. Any versioned, encrypted, public-access-blocked bucket the deployer can write works.

Several copies of infra/pds/ can share one account. Each sets its own name_prefix, which starts every resource name but the zone’s, and its own root_zone, which names the zone. The state key follows name_prefix too: the default, didbot, keeps pds/terraform.tfstate, and any other prefix keeps its state under <name_prefix>-pds/terraform.tfstate.

tofu init reads name_prefix to find the key, as every later command does. Keep a copy’s variables in a *.auto.tfvars file, which every command loads, the tofu output in scripts/publish-image.sh among them. A command whose prefix differs from the one init used is refused until init runs again. To move a checkout from one copy to another, init with -reconfigure, which switches keys and copies no state:

cd infra/pds && tofu init -reconfigure -backend-config=backend.hcl

Init’s other offer, -migrate-state, copies the old key’s state into the new one, which would hand one copy the other’s resources.

Configuration tiers

Nothing below the operator’s repository may widen what an agent may do. The management surface is not a replacement for configuration: what the process needs in order to start is an argument, and what an agent may do is a record somewhere else.

Clocks

An attestation claim is valid inside a window. A host with a wrong clock cannot provision, and the failure does not look like a clock problem.