Notable changes

• Add server side support for sync v1.1 aka inductive firehose. xrpc_sync.subscribe_repos now includes covering proof blocks and new prev and prevData fields.
• Drastically redesign xrpc_sync.subscribeRepos to unify event stream generation across all subscribers. This significantly improves scalability and reduces CPU and I/O to near constant, with minimal additional overhead per subscriber (#52).
• MST:
  • Add new cids_for_path, add_covering_proofs methods.
• Repo:
  • apply_writes: skip no-op update operations where the new record value is the same as the existing stored record. (No-op updates are evidently illegal in ATProto.)
  • Emit new #sync event when a new repo is created.
• Storage:
  • read_events_by_seq: always include the MST root block in every commit event.
• DatastoreStorage:
  • AtpRemoteBlob.get_or_create: truncate URLs to 1500 characters.
  • Extract out new AtpRemoteBlob.generate_private_key method.
• did:
  • resolve_handle: support did:webs in the HTTPS /.well-known/atproto-did method.

Breaking changes:

• repo:
  • apply_commit, apply_writes: raise an exception if the repo is inactive.
• storage:
  • create_repo: remove signing_key and rotation_key kwargs, read them from input repo instead.
  • load_repo: don't raise an exception if the repo is tombstoned.
• datastore_storage:
  • Stop storing AtpBlock.decoded in the datastore, it's now just an in memory @property.
• util:
  • Rename TombstonedRepo to InactiveRepo.

Non-breaking changes:

• datastore_storage:
  • DatastoreStorage:
    • Add new ndb_context_kwargs constructor kwarg.
    • apply_commit: handle deactivated repos.
    • create_repo: propagate Repo.status into AtpRepo.
  • AtpRemoteBlob:
    • get_or_create: drop datastore transaction.
    • Add width and height properties, populated for images and videos, to be used in image/video embed aspectRatio (snarfed/bridgy-fed#1571).
    • Check video length, raise ValidationError on videos over 3 minutes.
• did:
  • Add new get_signing_key, get_handle functions.
  • create_plc: remove trailing slash from services.atproto_pds.endpoint.
• storage:
  • Storage: add new write_blocks method, implement in MemoryStorage and DatastoreStorage.
• xrpc_repo:
  • describe_server: include all app.bsky collections and others like chat.bsky.actor.declaration; fetch and include DID doc.
  • Implement com.atproto.repo.importRepo.
• xrpc_sync:
  • get_blob:
    • If we have more than one blob URL for the same CID, serve the latest one (bridgy-fed#1650.
    • Add HTTP Cache-Control to cache for 1h.
  • list_repos:
    • Bug fix: Use string TID for rev, not integer sequence number.
    • Bug fix: don't set status to null if the account is active.

Breaking changes:

• Add much more lexicon schema validation for records and XRPC method input, output, and parameters.
• storage:
  • Switch Storage.write to return Block instead of CID.

Non-breaking changes:

• did:
  • Add new update_plc method.
  • create_plc: add new also_known_as kwarg.
  • resolve_handle: drop Content-Type: text/plain requirement for HTTPS method.
• mst:
  • Add new optional start kwarg to load_all.
• repo:
  • Emit new #identity and #account events to subscribeRepos when creating new repos.
• storage:
  • Add new deactivate_repo, activate_repo, and write_event methods.
  • Add new optional repo kwarg to read_blocks_by_seq and read_events_by_seq to limit returned results to a single repo.
• datastore_storage:
  • Add new max_size and accept_types kwarg to AtpRemoteBlob.get_or_create for the blob's maxSize and accept parameters in its lexicon. If the fetched file doesn't satisfy those constraints, raises lexrpc.ValidationError.
  • DatastoreStorage.read_blocks_by_seq: use strong consistency for datastore query. May fix occasional AssertionError when serving subscribeRepos.
• xrpc_sync:
  • Switch getBlob from returning HTTP 302 to 301.
  • Implement since param in getRepo.
  • subscribeRepos: wait up to 60s on a skipped sequence number before giving up and emitting it as a gap.
• util:
  • service_jwt: add new **claims parameter for additional JWT claims, eg lxm.

Breaking changes:

• datastore_storage:
  • DatastoreStorage: add new required ndb_client kwarg to constructor, used to get new context in lexrpc websocket subscription handlers that run server methods like subscribeRepos in separate threads (snarfed/lexrpc#8).
  • DatastoreStorage.read_blocks_by_seq: if the ndb context gets closed while we're still running, log a warning and return. (This can happen in eg flask_server if the websocket client disconnects early.)
  • AtpRemoteBlob: if the blob URL doesn't return the Content-Type header, infer type from the URL, or fall back to application/octet-stream (bridgy-fed#1073).
• did:
  • Cache resolve_plc, resolve_web, and resolve_handle for 6h, up to 5000 total results per call.
• storage: rename Storage.read_commits_by_seq to read_events_by_seq for new account tombstone support.
• xrpc_sync: rename send_new_commits to send_events, ditto.
• xrpc_repo: stop requiring auth for read methods: getRecord, listRecords, describeRepo.

Non-breaking changes:

• did:
  • Add HANDLE_RE regexp for handle validation.
• storage:
  • Add new Storage.tombstone_repo method, implemented in MemoryStorage and DatastoreStorage. Used to delete accounts. (bridgy-fed#783)
  • Add new Storage.load_repos method, implemented in MemoryStorage and DatastoreStorage. Used for com.atproto.sync.listRepos.
• util:
  • service_jwt: add optional aud kwarg.
• xrpc_sync:
  • subscribeRepos:
    • Add support for non-commit events, starting with account tombstones.
    • Add ROLLBACK_WINDOW environment variable to limit size of rollback window. Defaults to no limit.
    • For commits with create or update operations, always include the record block, even if it already existed in the repo beforehand (snarfed/bridgy-fed#1016).
    • Bug fix, populate the time each commit was created in time instead of the current time (snarfed/bridgy-fed#1015).
  • Start serving getRepo queries with the since parameter. since still isn't actually implemented, but we now serve the entire repo instead of returning an error.
  • Implement getRepoStatus method.
  • Implement listRepos method.
  • getRepo bug fix: include the repo head commit block.
• xrpc_repo:
  • getRecord: encoded returned records correctly as ATProto-flavored DAG-JSON.
• xrpc_*: return RepoNotFound and RepoDeactivated errors when appropriate (snarfed/bridgy-fed#1083).

Notable changes

• Bug fix: base32-encode TIDs in record keys, at:// URIs, commit revs, etc. Before, we were using the integer UNIX timestamp directly, which happened to be the same 13 character length. Oops.
• Switch from BGS_HOST environment variable to RELAY_HOST. BGS_HOST is still supported for backward compatibility.
• datastore_storage:
  • Bug fix for DatastoreStorage.last_seq, handle new NSID.
  • Add new AtpRemoteBlob class for storing "remote" blobs, available at public HTTP URLs, that we don't store ourselves.
• did:
  • create_plc: strip padding from genesis operation signature (for did-method-plc#54, atproto#1839).
  • resolve_handle: return None on bad domain, eg .foo.com.
  • resolve_handle bug fix: handle charset specifier in HTTPS method response Content-Type.
• util:
  • new_key: add seed kwarg to allow deterministic key generation.
• xrpc_repo:
  • getRecord: try to load record locally first; if not available, forward to AppView.
• xrpc_sync:
  • Implement getBlob, right now only based on "remote" blobs stored in AtpRemoteBlobs in datastore storage.

Notable changes

• Migrate to ATProto repo v3. Specifically, the existing subscribeRepos sequence number is reused as the new rev field in commits. (Discussion.).
• Add new did module with utilities to create and resolve did:plcs and resolve did:webs.
• Add new util.service_jwt function that generates ATProto inter-service JWTs.
• Repo:
  • Add new signing_key/rotation_key attributes. Generate store, and load both in datastore_storage.
  • Remove format_init_commit, migrate existing calls to format_commit.
• Storage:
  • Rename read_from_seq => read_blocks_by_seq (and in MemoryStorage and DatastoreStorage), add new read_commits_by_seq method.
  • Merge load_repo did/handle kwargs into did_or_handle.
• XRPCs:
  • Make subscribeRepos check storage for all new commits every time it wakes up.
    • As part of this, replace xrpc_sync.enqueue_commit with new send_new_commits function that takes no parameters.
  • Drop bundled app.bsky/com.atproto lexicons, use lexrpc's instead.

Big milestone: arroba is successfully federating with the ATProto sandbox! See app.py for the minimal demo code needed to wrap arroba in a fully functional PDS.

• Add Google Cloud Datastore implementation of repo storage.
• Implement com.atproto XRPC methods needed to federate with sandbox, including most of repo and sync.
  • Notably, includes subscribeRepos server side over websocket.
• ...and much more.

Implement repo and commit chain in new Repo class, including pluggable storage. This completes the first pass at all PDS data structures. Next release will include initial implementations of the com.atproto.sync.* XRPC methods.

Initial release! Still very in progress. MST, Walker, and Diff classes are mostly complete and working. Repo, commits, and sync XRPC methods are coming soon.