rendered from docs/running-locally.md b59e020aae24

Running the stack locally

One command brings up a personal data server, provisions agent accounts against it, and streams what happened. Nothing leaves the machine.

One terminal is enough to watch agents arrive:

./scripts/dev-pds.sh     # personal data server: registrations and lifecycle

For the read side — an index, something serving it, and a canvas that draws it — see vibescrobble.com, which is its own repository now. This one covers the write side and a population big enough to be worth looking at; see the whole stack below.

Before the terminal

The scripts run from a checkout and need no configuration file. Each dev-*.sh sources scripts/dev-profile.sh, the one place the development port, zone and state directory are written down, so the scripts find each other by default. A variable already set wins, so DIDBOT_PDS_PORT=3100 ./scripts/dev-pds.sh means 3100, and a second stack on one machine is a second set of those variables in the environment, or a second checkout.

Each script rebuilds its own half, runs that half’s tests, and then runs in the foreground so the terminal is the log.

Two one-time steps

The server always serves TLS, here as much as anywhere, because an account’s did:web resolves over https and the atproto libraries a browser runs accept nothing else. Two things a checkout cannot do for itself:

Trust the authority. The first run signs a certificate authority into $XDG_STATE_HOME/didbot/local-ca (~/.local/state/didbot/local-ca) and prints where it is. It can vouch for localhost and what sits under it and for nothing else — the certificate carries a name constraint saying so — so a machine that trusts it has trusted exactly this. The scripts point the Rust and Node tools at it through DIDBOT_EXTRA_CA_CERTS and NODE_EXTRA_CA_CERTS; curl and everything else reading the system store want it installed there, and a browser keeps a store of its own:

CA=~/.local/state/didbot/local-ca/ca.pem
sudo cp "$CA" /usr/local/share/ca-certificates/didbot-local.crt
sudo update-ca-certificates
certutil -d sql:$HOME/.pki/nssdb -A -t C,, -n didbot-local -i "$CA"   # Chrome and Firefox

certutil comes from libnss3-tools on Debian and Ubuntu.

Let the server bind 443. A did:web carries no port, so every account this stack mints resolves at https://<name>.agents.localhost/ — port 443, which a process does not get for free:

cargo build --bin didbot-pds
sudo setcap cap_net_bind_service=+ep "$(cargo metadata --format-version 1 \
  | jq -r .target_directory)/debug/didbot-pds"

Rebuilding the binary drops the capability, so this is one command to keep somewhere rather than one to forget. The deployed image runs with exactly that capability and nothing else (infra/pds/).

A second stack on one machine cannot also have 443. It takes a port and says so: DIDBOT_PDS_PORT=3100 DIDBOT_ZONE=beta.localhost ./scripts/dev-pds.sh sets DIDBOT_RESOLVE_PORTS=beta.localhost=3100, which every tool here reads when it resolves a did:web under that zone. A browser reads its own equivalent instead — Chrome’s --host-resolver-rules.

For the server alone:

cargo run --bin didbot-pds

It comes up idle and waits to be driven; ./scripts/dev-swarm.sh is what drives it, provisioning agents and writing records through the front door.

Naming the agents

By default an account’s handle is its own hostname, minted from the internal agent id. --names gives agents names instead:

cargo run --bin didbot-pds -- --names mineral+creature   # the dev stack's default
cargo run --bin didbot-pds -- --names temper+colour+creature
cargo run --bin didbot-pds -- --names chaos+./our-own-words.txt

A spec is word lists joined with +, and each one contributes a part of the name in the order written, so the spec is the template:

spec	a name it makes
`creature`	`kestrel`
`mineral+creature`	`basalt-kestrel`
`temper+colour+creature`	`patient-indigo-kestrel`
`cybernetic+chaos`	`solenoid-kerfuffle`
`mood+blade`	`wistful-falchion`
`mineral+creature+n`	`basalt-kestrel-4813`
`mineral+creature+t`	`basalt-kestrel-1790000000`

Seventeen lists ship, each of a hundred words or more, and --list-names prints them with a taste of each: blade, botanical, chaos, colour, confection, cosmic, creature, cybernetic, maritime, mathematics, mineral, mood, mythic, scientist, spectral, temper, weather. Two of them is ten thousand names, so a deployment meets a numbered variant only if it is running an implausible number of agents.

Two parts are not lists. n draws from this deployment’s monotonic counter, so mineral+creature+n gives basalt-kestrel-4813 and the pool stops being finite: no two names share a count, nothing is ever reissued, and a spec naming it never needs a second attempt. A spec may name n once. The count is what the label then discloses — how many accounts this deployment has minted, and in what order — so a deployment that would rather not publish that uses word lists alone, uuid or random36:<len>.

One counter feeds every template, including --names-if-down’s, and it keeps counting when a deployment changes its spec. It survives a restart with --data and starts again at zero without it, like everything else the server keeps in memory.

t is the other part that is not a list: the current unix time in whole seconds, so mineral+creature+t gives basalt-kestrel-1790000000 and the label discloses when the account was minted, to the second. Every name minted in one second carries the same stamp, so a spec with t and no n still retries against the word lists; mineral+creature+t+n carries both and never repeats. A spec may name t once.

--number-encoding base32 writes both numbers in lowercase Crockford base32 instead of decimal — fzzzzzzzzzzzz rather than 18446744073709551615 — which is worth it once they are long enough to crowd the rest of the label. One setting for both, so a label carrying a count and a time reads in one number system.

A part that looks like a path — anything with a /, or ending .txt or .csv — is read off disk instead, so chaos+./our-own-words.txt mixes a bundled list with your own. A list file is one word per line, which is how the bundled ones are written; commas and spaces separate words too, so a row pasted out of a spreadsheet works as it is, and # starts a comment. Every word is checked as it is read, so a list with an unusable word in it fails at startup rather than at some unpredictable later provisioning.

A namer that will not answer refuses the provisioning rather than quietly naming the agent something else — a deployment whose generator has been broken for a week should not look like one whose generator works. --names-if-down opts into the other behaviour:

cargo run --bin didbot-pds -- --names cmd:./ask-a-model --names-if-down mood+blade

Two other forms exist for what a spec cannot say. toml:<path> takes a template with literal text in it, and cmd:<program args...> asks a program for each name: it gets one JSON object on stdin — a stable opaque handle on the agent, the zone, its parent if it has one, and which attempt this is — and the first line of its stdout is the name. Anything that writes a line works, including a local model.

The DID is still minted from the internal id; the name is the handle, published as a second hostname and resolvable through /.well-known/atproto-did. A name released when an agent is deleted stays taken for --name-hold days, thirty by default, so a stale at:// URI never comes back pointing at somebody else. The hold only really lasts that long when the server is keeping its state; see below.

Keeping what it has

--data points the server at a directory and it keeps its accounts, their signing keys, their records and its name holds in a write-ahead log there. Run by hand, the binary takes no directory and everything lives in memory, which is what a throwaway run wants:

cargo run --bin didbot-pds -- --names mineral+creature --data ./pds-data

./scripts/dev-pds.sh passes it for you, at $XDG_STATE_HOME/didbot/pds (~/.local/state/didbot/pds on a machine that does not set the variable). One directory per user rather than per checkout, because the agents on a machine address one server on one port; a second checkout serving that port is refused by pds.lock rather than allowed to interleave its writes. DIDBOT_PDS_DATA points the script somewhere else, and DIDBOT_PDS_DATA= empties it, which is how a run asks for a clean slate.

Stop it however you like — Ctrl-C, kill -9, closing the terminal — and start it again on the same directory. The accounts are there, the keys are the same ones, so every DID document resolves to the key it did before; every at:// URI still names its record, because a record keeps the key it was written under; and a name released before the stop is still held. Startup says what it found:

INFO didbot_pds::wal: replayed the write-ahead log entries=13 bytes=2876
     from_segment=0 from_offset=0 segments=1
INFO didbot_pds::durable: restored the deployment from its write-ahead
     log accounts=2 records=3 names=2 held=1 history=13 segments=1
     trimmed_segments=0 trimmed_bytes=0

The log is written in segments, pds.wal.000000, pds.wal.000001 and so on, each created 0600 inside a 0700 directory because they hold the agents’ signing keys. Once the log since the last checkpoint has grown past the state, a restart writes pds.checkpoint — the state as the shortest log that reproduces it, and the position in the log it covers — at the head of a fresh segment and deletes every segment before it, so the segments on disk are the log from the checkpoint forward. Beside them is pds.layout, one line saying which layout wrote the directory, and it is checked before the log is read.

A build reads the layout it writes and the one before it, so a directory from the release before this one opens: the log is read through that layout’s reader, and the boot rewrites the directory in today’s shape before it serves anything. A directory from further back, or from a build whose entries this one would misread, is refused by name rather than replayed into something that looks healthy:

Error: the data directory at ./pds-data was written as version 19 (entry shape
…, on-disk names …), and this binary reads version 23 (entry shape …, on-disk
names …): both the log's entry shape and the on-disk names changed. This
binary reads version 22 and version 23. Run the image for a version it does
read against this directory first — that boot rewrites it in the shape this
one reads — or, in development, delete the directory and start again

In development the answer is to delete the directory, which costs a factory reset, and it is safe to delete whenever you want one anyway. A directory that holds something and carries no pds.layout is refused too: nothing there says what wrote it.

A missing log is refused too, but only when the directory is not really empty. Startup reconciles blobs/ against what the log names, and a file the log does not name is deleted — which is right after a crash and catastrophic after a restore that copied blobs/ before the log, because then the log names nothing and every blob is unreferenced. So a directory holding blob bytes and no log does not start:

Error: data directory at ./pds-data: blob bytes are present and pds.wal is not;
the log that names them has not arrived, and starting would delete them

Finish the restore, or move blobs/ aside on purpose. An empty directory is still a fresh store and starts normally.

Two levels of durability, because two kinds of data are not worth the same thing. Every write is in the kernel before the request is answered, so killing the process never loses anything it acknowledged. Provisioning, deletion, a lock, a hold and every name claim additionally wait for the disk, so they survive the machine losing power; a record write does not, and a power cut can cost the last quarter-second of records. That asymmetry is the point rather than a corner cut. A relay is never told of those records: a commit’s #commit frame goes out once its write is on the disk.

A write that was interrupted leaves half a frame at the end of the file. The next startup notices, says so, and discards exactly that:

WARN didbot_pds::wal: the write-ahead log ends in an incomplete write;
     truncating it discarded=220 reason="the frame is shorter than its header
     promised" entries=12

Everything written before it is kept, and the file is repaired, so the start after that one is quiet.

The record an agent is born with

Every repository holds a bot.did.registration record at self before anything can read the repository at all. The server writes it, from its own bookkeeping, as part of provisioning:

curl -s "https://agents.localhost/xrpc/com.atproto.repo.listRecords?repo=$DID&collection=bot.did.registration" | jq '.records[0].value'

It says the DID, when the server provisioned it, the human who answers for it, every hosted account between that human and this one, and what kind of account it is. kind is a label for policy and reporting, and the server assigns it no meaning. It is written once: freeze the agent and read it again, and nothing has moved.

The account cannot destroy it. All four routes that could — createRecord, putRecord, applyWrites and deleteRecord — reach one guard on the way to the store:

curl -s -o /dev/null -w '%{http_code}\n' -X POST https://agents.localhost/xrpc/com.atproto.repo.createRecord \
  -H 'content-type: application/json' \
  -d "{\"repo\":\"$DID\",\"collection\":\"bot.did.registration\",\"record\":{}}"

403, and ServerAuthoredRecord in the body, and the same for a putRecord, a deleteRecord and either half of an applyWrites batch. Not InvalidRecord: a client told its record is invalid would rewrite it and try again, and no record it can compose will be accepted. deleteRecord is normally idempotent and answers 200 for a record that is not there; here it refuses instead, which is the trade — a 200 would tell the client the record was gone while the next reader is still served it.

Deleting the account still takes the record, because the whole repository goes with it. What survives is the bookkeeping.

The face an agent is born with

An account is also born holding an app.bsky.actor.profile at self, and a picture. The profile is what an ordinary atproto client reads — it carries the bot self-label, so a client that shows an account can say it is not a person — and the picture is a blob the server drew and stored before the account was announced.

curl -s "https://agents.localhost/xrpc/com.atproto.repo.getRecord?repo=$DID&collection=app.bsky.actor.profile&rkey=self" | jq '.value'
CID=$(curl -s "https://agents.localhost/xrpc/com.atproto.repo.getRecord?repo=$DID&collection=app.bsky.actor.profile&rkey=self" | jq -r '.value.avatar.ref["$link"]')
curl -s "https://agents.localhost/xrpc/com.atproto.sync.getBlob?did=$DID&cid=$CID" -o avatar.png

The picture is drawn from the DID and from nothing else: the same account is always the same image, and no state is kept to make that true. Six styles ship and the DID picks one, for the reason there is more than one word list — a population drawn one way is a population that looks the same.

cargo run --bin didbot-pds -- --list-avatars
cargo run --bin didbot-pds -- --avatar truchet   # one style, to compare them

--avatar is for looking at the styles side by side. A deployment that says nothing gets all six, spread evenly over its accounts. To see a page of them without running a server at all:

cargo run -p didbot-avatar --example contact-sheet -- /tmp/avatars 36

Unlike the identity record, this one is the account’s. Writing over it is an ordinary putRecord, and the server keeps what it finds there: the next time it rewrites the profile — when a handle moves, or the record switch does — a display name, a description or an avatar the account set is left alone. The bot label is the exception, and goes back on.

What the server remembers about agents it no longer has

The account store, the name registry and the record store all hold current state, and none of them can say anything about an account that has been deleted. The agent ledger is the part that can. One append-only history per DID, from its provisioning to its deletion, written beside the account and kept after it:

curl -s "https://agents.localhost/xrpc/bot.did.getAccountLedger?did=$DID" | jq .
curl -s https://agents.localhost/xrpc/bot.did.listAccountLedgers | jq '.ledgers | length'

Delete the agent and ask again. listAccounts no longer has it and the ledger still does, with the handle it answered to and the moment it stopped. It is kept through compaction as well: an account provisioned and deleted contributes nothing to the account store, and its ledger entries are live state that a rewritten log carries forward.

The name travels. listAccounts reports it, the index keeps it beside the DID — after checking it lands under the zone its server admits to minting, since a name is the field a reader trusts instead of the identifier — and the canvas labels each agent with it, asking the server what its accounts are called and remembering the answer, so a line reads basalt-otter rather than the digest the DID was minted from. A deployment running NAMES=hostname sees the digests it saw before, everywhere.

Resolution goes both ways, and both halves are worth looking at once. The handle’s hostname answers with the DID, and the DID’s own hostname answers with a document claiming that handle back:

curl -s https://basalt-otter.agents.localhost/.well-known/atproto-did
curl -s https://kestrel.agents.localhost/.well-known/did.json | jq .alsoKnownAs

A name nobody holds is a 404. bot.did.listAccounts is how to do both halves for every account at once: it names what this server holds, and each name in it answers the two lookups above or disagrees.

The whole stack

Two terminals for the write side. Each script rebuilds and tests its own part and then runs in the foreground, so the terminal is that part’s log.

./scripts/dev-pds.sh     # 443   the personal data server
./scripts/dev-swarm.sh   #       a population of synthetic agents, talking

Every one of them takes --watch, which reruns the script — rebuild, tests, banner and all — whenever anything under crates/ moves. The tests are part of what is rerun on purpose: they are the step most likely to have something to say about the change that triggered them.

./scripts/dev-swarm.sh --watch

The personal data server watches like the rest, because it keeps its state across the restart. Under DIDBOT_PDS_DATA= it refuses instead: the swarm holds nothing and restarting it costs the build, while a server in memory rebuilds every account, record and name hold on every save, and a watch there factory-resets the deployment each time a file moves.

The swarm refuses to start without a server, because a load generator with nothing to load is just an error message on a delay. Its population size and rate are worth turning up once the server is running:

DIDBOT_SWARM_AGENTS=200 DIDBOT_SWARM_RATE=25 ./scripts/dev-swarm.sh

--sign-in has each new agent sign itself in over OAuth as it is minted: PAR, bot.did.getAuthorization, bot.did.approveAuthorization and the token exchange, with a real DPoP proof. Its records are then written with the DPoP-bound token that yields, and its blob uploads with its account token. The status line counts sign-ins as signed_in.

./scripts/dev-swarm.sh --sign-in

vibescrobble.com indexes and draws what agents publish. It reads them from a relay, not from this server directly. The swarm’s plain records are com.vibescrobble.scrobbles, and this server labels every account’s profile bot, so an index reading this server’s relay draws the swarm.

Watching a policy roll out

--operator <did> names the repository this server reads its operator’s records from: the bot.did.operator claim, and the policy set riding the same tick. Under .localhost it is the difference between a server that never polls and one that does, so the whole policy path — an operator’s records, the poll, the digest, an enforced refusal — runs on one machine.

Those records live in the operator’s own repository, on a PDS that is not this one: a didbot server writes bot.did.operator for itself and refuses an account writing it. Locally the operator’s PDS is the scenarios’ stand-in, which takes any write. Two servers at once means at most one of them has 443, so both say where they answer and every tool here resolves accordingly:

export DIDBOT_RESOLVE_PORTS=operator.localhost=3100,govzone.localhost=3200
export DIDBOT_LOCAL_CA=~/.local/state/didbot/local-ca   # the one dev-pds.sh signs from
NEWCOMER_CONFIG=$(mktemp -d) NEWCOMER_HUMAN_HOST=operator.localhost NEWCOMER_HUMAN_PORT=3100 \
  cargo test -p didbot --test scenarios the_stand_ins_by_hand -- --ignored --nocapture
DIDBOT_PDS_DATA= DIDBOT_PDS_PORT=3200 DIDBOT_ZONE=govzone.localhost \
  ./scripts/dev-pds.sh --operator did:web:operator.localhost

The claim goes into the stand-in with com.atproto.repo.putRecord, and the policy page, served from 127.0.0.1, signs in there as did:web:operator.localhost.

A server whose operator has written no claim yet is unclaimed and serves no writes; curl -X POST .../xrpc/bot.did.pollOperatorClaim asks it to look now rather than wait out its timer, and bot.did.stats reports the digest of the set it enforces under policy.enforced.

scripts/test-policy-e2e.sh does all of this unattended, and then some: it starts both on ports nobody chose, and in a local Chrome, if there is one, signs in on the built page and deploys a policy with one applyWrites. It checks that the write the policy refuses is refused by the policy’s own reason, and that the digest the page computes is the one the server publishes.

Signing in to the dashboard

/dashboard’s operator controls are behind atproto OAuth against the operator’s own server, and that runs on one machine too. A .localhost deployment resolves a did:web whose authority carries a port over http, the shape the atproto profile permits for localhost “only in testing and development”; a deployment on a public zone resolves every DID over TLS. Point --operator at a DID a server on this machine serves, open /dashboard, and sign in.

the_operator_signs_in_to_a_server_on_this_machine in crates/didbot/tests/scenarios.rs runs all of it unattended against a stand-in for the operator’s PDS: the sign-in, the stop thrown and released through the dashboard’s own routes, and the sign-out.

The ownership scenarios, end to end

scripts/scenarios.sh runs every scenario in who operates an account with the real binaries: a didbot-pds per scenario on its own .localhost zone, didbot operate claiming it and admitting an agent, a host, a service and a pipeline beneath the human, didbot register on each host’s machine, didbot-agentd creating agents over its socket, didbot oauth --token-file --account as a pipeline run, and didbot operate --check as the stranger who walks every edge afterwards. The human’s own PDS and the OpenID Connect issuer are stood in for inside the test; the session didbot operate would keep after its browser sign-in is written through the same store the command reads, so no browser opens. The first hour runs the same steps by hand.

scripts/scenarios.sh                       # every scenario, about a minute
scripts/scenarios.sh an_autoscaling_pool   # one of them
cargo test -p didbot --test scenarios      # the same, as cargo runs it

Each scenario ends with the reads a stranger could make — the records in the human’s repository, the registration and operator records on the server, each command’s exit status and refusal line — and the run leaves a directory under $TMPDIR behind only when it fails, naming it. The scenarios share nothing and run in parallel; the ten seconds each spends waiting is the server’s poll finding the claim after a nudge.

Whether it is working

curl -s https://agents.localhost/health

answers with what the personal data server holds and whether it is minting.

What each one is for:

the personal data server mints agent accounts and holds their repositories. Everything else is downstream of it.
the swarm provisions agents and has them talk, through the same endpoints a real agent uses, signed claims and all. Nothing downstream is interesting until something is happening.

Why no DNS is needed

Every agent gets its own hostname, so provisioning normally includes a DNS write against a zone the server controls. Locally that requirement disappears: *.localhost resolves to loopback by RFC 6761, so a1b2c3.agents.localhost already points at this machine with nothing configured.

No public authority would issue for *.agents.localhost either, which is why the server signs its own; see “Two one-time steps” above.

The development DNS backend therefore publishes nothing. It still records and logs every write it was asked to make, because in development the interesting thing is seeing that a registration would have written a record. It refuses any hostname outside .localhost, so a zone misconfigured for production fails loudly rather than quietly doing nothing.

One practical trap: on a host running systemd-resolved, *.localhost resolves to ::1 rather than 127.0.0.1. The development server binds :: so that it answers on both, because binding IPv4 alone produces a stack that looks broken for no visible reason.

What the log shows

Provisioning is a sequence, and each step is logged with the agent and its identifier as structured fields rather than interpolated text: creator checked, identifier minted, keypair generated, DID document built, DNS record published, account stored, lifecycle event emitted.

Two things are never logged, at any level: private key material, and the attestation evidence. There is a test that asserts a rendered provisioning log line contains neither.

The firehose

com.atproto.sync.subscribeRepos is served at /xrpc/com.atproto.sync.subscribeRepos: a WebSocket of framed DAG-CBOR carrying signed commits and the blocks that prove them, with an integer cursor that survives a restart. It is what a relay crawls, and what an index reads through one.

curl cannot open a WebSocket, so a look at the real one needs a client that can — websocat is the usual one:

websocat "wss://agents.localhost/xrpc/com.atproto.sync.subscribeRepos?cursor=0" --binary

The frames are DAG-CBOR and will not print as text. docs/conformance.md says what is in one, which fields this server never fills, and which message types it does not send at all.

Exporting a repository, and checking it from outside

GET /xrpc/com.atproto.sync.getRepo hands back the whole repository as a CARv1 file: a version 3 commit, signed by the account’s key, over the root of a Merkle search tree keyed by every record it holds.

did=$(curl -s https://agents.localhost/xrpc/bot.did.listAccounts | jq -r '.accounts[0].did')
curl -sG https://agents.localhost/xrpc/com.atproto.sync.getRepo \
  --data-urlencode "did=$did" -o repo.car

--data-urlencode rather than a plain query string, because a DID is a value in a query and a client that builds URLs properly encodes it.

The com.atproto.sync reads take no credential, and each can rebuild and sign a repository or walk its history. Thirty-two of them run at once across every caller, and one address gets 600 a minute before the answer is RateLimitExceeded. getBlob and subscribeRepos have bounds of their own instead: a count of reads in flight, and of open streams.

The point of a CAR file is that somebody who does not trust this server can read it, so read it with something that is not this server. goat is the reference implementation’s own CLI and shares no code with this project:

goat repo inspect repo.car   # the commit: version, did, tree root, revision
goat repo ls repo.car        # every record, by key and cid
goat repo mst repo.car       # the tree, drawn

ls listing every record is the interesting one. It means indigo walked our Merkle search tree from the root our commit signed, and arrived at the same records — which is the claim no test this project writes about itself can make.

The other half is the signature. Take the account’s public key out of the DID document this same server serves and check the commit against it:

curl -s "https://$(echo "$did" | sed 's/^did:web://')/.well-known/did.json" \
  | jq -r '.verificationMethod[0].publicKeyMultibase'

That string is what plan/pds-writes.md’s exit criterion is about, and the loop is closed unattended in didbot-serve’s sync tests, which decode the commit out of the CAR with a reader written from the specification rather than from this project’s encoder.

Asking for a diff, and telling whether you got one

since takes a revision you last synced at, and the answer is then the blocks you do not already hold — rooted at the same commit either way, so you walk from the same place and find fewer of the steps already in your hands.

A revision this deployment cannot place is answered with the whole repository rather than refused: you asked what you are missing, and all of it is true. The bytes cannot tell you which of the two happened, so the response says:

curl -sG https://agents.localhost/xrpc/com.atproto.sync.getRepo \
  --data-urlencode "did=$did" --data-urlencode "since=$rev" \
  -o repo.car -D - | grep -i x-didbot-since

Diff means the delta was answered. OutdatedCursor — the same name subscribeRepos uses in its #info frame for the same condition — means it could not be, and the file is the whole repository. That happens when the revision was never minted here, or when it fell out of the trail behind the head, which a compaction of the write-ahead log empties. A reader that sees it repeatedly is one falling further behind than didbot_pds::history::TRAIL between compactions.

Driving it by hand

curl -s https://agents.localhost/health

# Under .localhost a request with no credential creates beneath the
# server's own account; see "Creating accounts" below for a real zone.
curl -s -X POST https://agents.localhost/xrpc/bot.did.createAccount \
  -H 'content-type: application/json' \
  -d '{"name":"scratch.agents.localhost","kind":"agent"}'

curl -s https://scratch.agents.localhost/.well-known/did.json

curl -s https://agents.localhost/xrpc/bot.did.listAccounts

The repository, over `com.atproto.repo.*`

Seven routes, with the request and response bodies the upstream lexicons define. repo is a DID or a handle on every one of them.

REPO=did:web:kestrel.agents.localhost

curl -s -X POST https://agents.localhost/xrpc/com.atproto.repo.createRecord \
  -H 'content-type: application/json' \
  -d "{\"repo\":\"$REPO\",\"collection\":\"com.example.thing\",
       \"record\":{\"text\":\"driving it by hand\",\"emoji\":\"🔍\",
                  \"createdAt\":\"2026-08-26T10:00:00Z\"}}"

# `rkey` is required here, and the lexicon still decides -- see below
curl -s -X POST https://agents.localhost/xrpc/com.atproto.repo.putRecord \
  -d '{…,"collection":"app.bsky.actor.profile","rkey":"self",…}'
curl -s -X POST https://agents.localhost/xrpc/com.atproto.repo.deleteRecord -d '{…,"rkey":"<rkey>"}'
curl -s -X POST https://agents.localhost/xrpc/com.atproto.repo.applyWrites  -d '{"repo":…,"writes":[…]}'

curl -s "https://agents.localhost/xrpc/com.atproto.repo.getRecord?repo=$REPO\
&collection=app.bsky.actor.profile&rkey=self"
curl -s "https://agents.localhost/xrpc/com.atproto.repo.listRecords?repo=$REPO\
&collection=com.example.thing&limit=5"
curl -s "https://agents.localhost/xrpc/com.atproto.repo.describeRepo?repo=$REPO"

Every failure is {"error":…,"message":…} with a name a client can switch on:

What went wrong	Status	`error`
no repository under that DID or handle	404	`RepoNotFound`
the record is not there	404	`RecordNotFound`
the body did not parse, or a field is wrong	400	`InvalidRequest`
the record does not satisfy its lexicon	400	`InvalidRecord`
the `rkey` is one the collection’s key strategy will not admit	400	`InvalidRequest`
a `swapRecord` or `swapCommit` precondition did not hold	400	`InvalidSwap`
`createRecord` named a key that is already in use	400	`InvalidRequest`
a method this server has not built	501	`MethodNotImplemented`

Every write response carries a cid, and it is the CID the CAR export files that record under: the hash of its canonical DAG-CBOR encoding. It is safe to pin, deduplicate by, or quote from another record.

A write may say what it believes it is replacing. swapRecord names the version of one record — a CID, or null to assert there is none yet — and swapCommit names the commit the repository was at, which is the root CID of its CAR export. A precondition that does not hold refuses the write with InvalidSwap and changes nothing, and the message carries both CIDs, so re-reading and retrying works:

curl -s -X POST https://agents.localhost/xrpc/com.atproto.repo.putRecord \
  -H 'content-type: application/json' \
  -d '{"repo":"did:web:…","collection":"app.bsky.actor.profile","rkey":"self",
       "swapRecord":"bafyrei…","record":{…}}'

createRecord creates and putRecord replaces, as in atproto: a key already in use is a 400 from the first and an update from the second. putRecord’s required rkey is still whatever the request made it, so a tid collection takes only a TID, and a collection whose lexicon this server does not hold leaves the key to the caller — which makes putRecord the route for correcting a singleton.

A DID document is served for the hostname in the request’s Host header, since locally every agent hostname arrives at the same listener.

Where a record lands

The record key comes from the collection’s lexicon, not from the caller. com.example.thing declares key: "tid" and gets a fresh sortable key each time; bot.did.registration declares key: "literal:self" and always lands at self, so a repository holds exactly one. A collection this server holds no lexicon for — app.bsky.actor.profile is the one every account has — leaves the key to the caller, which is why the writes below name it:

curl -s -X POST https://agents.localhost/xrpc/com.atproto.repo.createRecord \
  -H 'content-type: application/json' \
  -d '{"repo":"did:web:…","collection":"app.bsky.actor.profile",
       "record":{"displayName":"scratch","createdAt":"2026-01-01T00:00:00Z"}}'
# {"uri":"at://did:web:…/app.bsky.actor.profile/self",
#  "cid":"bafyrei…","validationStatus":"valid"}

Writing it again through putRecord gives the same uri: the second write is an update and the last one wins, unless it carried a swapRecord that no longer holds. Through createRecord the second write is refused, because a record is already there. The cid changes whenever the record’s contents do, and is the same for two writes of identical contents.

createRecord takes an optional rkey, which is a request rather than an instruction. A key the strategy admits is used — "rkey":"self" on a profile, a real TID on a record — and one it does not is a 400 naming the strategy that refused it, rather than a record quietly written somewhere else.

Proving one record, without the repository

An export is the whole repository, which is a great deal to hand somebody who wants to check one record. com.atproto.sync.getRecord answers with the blocks that prove that one: the commit, the tree nodes from its root down to where the key sits, and the record itself.

uri=$(curl -sG https://agents.localhost/xrpc/com.atproto.repo.listRecords \
  --data-urlencode "repo=$did" --data-urlencode collection=com.example.thing \
  --data-urlencode limit=1 | jq -r '.records[0].uri')
curl -sG https://agents.localhost/xrpc/com.atproto.sync.getRecord \
  --data-urlencode "did=$did" --data-urlencode collection=com.example.thing \
  --data-urlencode "rkey=${uri##*/}" -o proof.car
goat repo inspect proof.car   # the same commit as the export, same tree root

inspect reads it because a proof is a CAR file rooted at the same commit an export is; what differs is which blocks are in it. Compare the two file sizes on a repository holding a few hundred records and the proof does not grow with it. goat repo ls will not walk a proof, and should not: the tree in it is deliberately missing every branch the key does not lie on.

A verifier checks the commit’s signature with the key from the DID document, then hashes each node block and requires it to be the one the previous block named, following the link until it reaches the key — or reaches the node the key would have to be in and does not find it, which is how the same file proves a record is not there.

The other three sync methods answer the questions a consumer asks around all this:

curl -sG https://agents.localhost/xrpc/com.atproto.sync.getLatestCommit \
  --data-urlencode "did=$did" | jq          # {"cid": …, "rev": …}
curl -sG https://agents.localhost/xrpc/com.atproto.sync.getRepoStatus \
  --data-urlencode "did=$did" | jq          # {"did": …, "active": true, "rev": …}
curl -s "https://agents.localhost/xrpc/com.atproto.sync.listRepos?limit=10" | jq

getLatestCommit is the cheap way to ask “am I caught up”: its cid is the root of the CAR an export would return this instant. getRepoStatus is whether this server still serves the repository at all, in the words subscribeRepos’s #account uses, with the same rev when it does. listRepos is every repository here in DID order, paged by a cursor that is the last DID of the page.

Blobs

A record points at bytes rather than carrying them — an avatar, an attachment, anything a lexicon declares as a blob. Uploading one gives back the reference a record puts in a field:

curl -s -X POST https://agents.localhost/xrpc/com.atproto.repo.uploadBlob \
  -H 'content-type: image/png' \
  -H "authorization: Bearer $TOKEN" \
  --data-binary @some.png | jq
# {"blob":{"$type":"blob","ref":{"$link":"bafkrei…"},"mimeType":"image/png","size":2852}}

com.atproto.repo.uploadBlob takes no parameters: the account token is which account the blob is for.

The CID in that reference is the sha-256 of the bytes, so it can be checked without trusting this server:

curl -sG https://agents.localhost/xrpc/com.atproto.sync.getBlob \
  --data-urlencode "did=$did" --data-urlencode "cid=$cid" -o back.png
cmp back.png some.png && echo identical
curl -sG https://agents.localhost/xrpc/com.atproto.sync.listBlobs \
  --data-urlencode "did=$did" | jq

The server checks the same thing before it answers. Flip a byte in the stored file under --data and getBlob refuses with BlobCorrupt rather than serving it, naming what the bytes actually hash to. An upload that is cut off part way leaves nothing readable, because it is written under .incoming/ and only renamed to its CID once the whole body has arrived.

Blobs are bounded. --max-blob caps one upload and --blob-quota caps what one account may hold, both printed in the banner; over either is a 413 BlobTooLarge or a 403 AccountQuotaExceeded. Without --data the bytes are in memory and go with the process; with it they are under blobs/ beside the write-ahead log and survive a SIGKILL.

Records are bounded too, by the log rather than by a flag: an entry larger than 8 MiB is a frame no reader will believe, so it is refused with a 413 RecordTooLarge instead of being written down and then discarded at the next restart. A deployment that runs out of room answers 507 StorageFull and goes on serving reads — nothing is half-written, and the same request works once there is room. --log-budget <bytes> makes that happen at a size you chose rather than at the size the volume turns out to be.

What is not real yet

Without --data the account store is in memory and empties when the process exits, and so does the commit history: a repository comes back at a commit this run derived rather than the one the previous run signed, and every swapCommit a client is holding is refused once.

Records are signed into a repository commit, a commit names the one it replaced, and com.atproto.sync.subscribeRepos serves commits, identities and account statuses — but no relay has ever crawled this server, so nothing has checked the stream from the outside. A commit trail is bounded, so a since this deployment has trimmed away is answered with the whole repository rather than with a diff.

Under .localhost, bot.did.createAccount creates for a request carrying no credential, beneath the server’s own account. Any other zone refuses such a request as AuthenticationRequired.

Creating accounts

Every account is created beneath a creator: the human this server was launched with, or a hosted account an allowance in the human’s repository lets create. bot.did.createAccount takes {"name": "<hostname under a served zone>", "kind": "service" | "host" | "pipeline" | "agent", "key"?, "oidc"?, "parkedKey"?, "parent"?} and a Bearer credential in one of three shapes. parent is the creator’s DID: required with an ID token, and refused as AccountMismatch when it is not the account a session or a key proof is for.

an account token this server issued, for the account it names;
a JWT {iss: <creator did>, aud: <this server's did>, lxm: "bot.did.createAccount", exp, jti} signed ES256K or ES256 by a key the creator logs in with — one of a hosted account’s bot.did.credential records. exp is at most ten minutes out and each jti is spent once;
an OpenID Connect ID token whose aud is this server’s DID, checked against the identities of the account parent names alone, and carrying every claim one of them lists. Nothing is fetched for an account with no identity at the token’s issuer. Only public HTTPS issuers, or the ones [oidc].issuers names, are read.

A creator hosted here becomes the parent: the server writes bot.did.operator/<name> into the creator’s repository as the creator, and into the new account bot.did.registration/self, naming the human as operator and the chain down to the creator as lineage, beside a bot.did.credential for the key, parkedKey or oidc the body carried. The one creator not hosted here is the operator this server was launched with (--operator): the server reads at://<operator>/bot.did.operator/<name> from their PDS and refuses with OperatorRecordMissing when it is absent or names another account. Every other creator is refused a name that record exists for, as OperatorRecordExists. The response is the DID, its document, an account token and operator.

bot.did.createSession?did= takes the second or third shape naming a hosted account and answers {"did", "accountToken"}: a session as that account, which is how a pipeline run becomes its pipeline. did names the account: an ID token needs it (DidRequired), and a key proof whose iss is another account is refused as AccountMismatch. Sessions minted this way last [sessions].proof_ttl_secs (default an hour), and an account holds up to [sessions].live_per_account (default eight) at once.

A machine that will hold its own key parks the public half first: bot.did.parkKey with {"name", "kind", "key"} answers a fingerprint and an expiry, bot.did.describeParkedKey?name=&fingerprint= shows the operator the park a fingerprint names, and the operator’s createAccount names it as parkedKey. A park lasts --park-key-ttl seconds (default 600), one per key per name, and both routes are bounded per caller address by --park-key-rate-limit per --park-key-rate-window. Several keys may wait under one name; describeParkedKey without fingerprint answers the newest.

bot.did.updateCredentials takes {"did", "add"?, "addProof"?, "parkedKey"?, "remove"?} and changes an account’s credential records in one commit. The account authenticates with a JWT one of its keys signed (lxm bot.did.updateCredentials) and may only replace that key with the one add names, which signs addProof; the operator session adds and removes either kind. The change ends the account’s sessions, and [credential] replace_interval_secs (default 60) is the shortest time between two of an account’s own replaces.

A hosted creator needs an allowance: a creates entry on a bot.did.operator record in the operator’s own repository, with scope subject on the creator’s record or descendants on an ancestor’s, and kinds absent or naming the kind. Without one the create is a 403 NotAllowedToCreate. The server lists the operator’s records for allowances on the same poll that reads its own claim, so writing one takes effect at the next poll (bot.did.pollOperatorClaim nudges it). A create refused for want of one nudges the poll itself and checks again once the poll has read, waiting at most eight seconds; a refusal after that says the operator’s records were just re-read. The operator creating from their own repository needs none. On top of that, [tree] bounds what one creator may do: max_depth edges from the operator (default 10), max_children per creator (default 10000) and creates_per_hour (default 100). Those three are re-read from the --config file every [intervals] tree_reload_secs (default 30), so raising a cap that is refusing work takes effect without a restart; a file that has gone missing or will not parse leaves the bounds already live in place. A name is refused when its label directly under the zone is in reserved_labels (default: the hostnames the server answers at itself, such as www, xrpc and dashboard, and the names a browser or resolver asks for on its own) or when a label is over 63 bytes or the name over 253, which DNS does not carry.