68 lines · 9.5 KB

Moving state out of D1 into Durable Objects

gitfor's runtime cost should scale with the repositories and people that are active, not with one shared database. D1 shrinks to a directory: the lookups that turn a name into the object that owns the data. Everything else lives in a Durable Object keyed by the repository or account it belongs to.

Target shape

Layer Holds Keyed by
D1 directory accounts, users (login, email), user_emails, repos (owner/name → id, visibility, status), redirects, retired names, repos_fts for repo-name search global
RepoState (new) Issues, pull requests, reviews, comments, labels, milestones, reactions, events, pins, discussions, releases, webhooks, branch protection, collaborators, settings, secrets, variables, environments repo id
RepoSearch (done) Code index of the default branch repo id
CI objects (Runner…) Workflow runs, jobs, steps, statuses, artifacts, caches, schedules repo id / run id
UserInbox (exists) Notifications, digests, subscriptions, plus an involvement index (below) user id
AccountState (new) Sessions, passkeys, tokens, OAuth grants, notification settings, billing and usage account id
D1 operator tables Admin audit, abuse reports, instance settings, waitlist, Stripe events, cron locks global, low volume

Decisions

  • One RepoState per repository, not one object per table. An issue, its labels, comments and timeline change together; keeping them in one SQLite gives real transactions. CI stays in its own objects because its write rate is much higher and its lifetime is different.
  • The repo object is the authority for access to its own data. Collaborators and branch protection move in, so a permission check for a repo is answered by that repo's object. Org membership lives in the org's AccountState. D1 keeps only repos.visibility and the owner, enough to route and to list public repos.
  • Cross-repo views come from an index fed by events, not from joins. "Assigned to me", "review requested", "my open pull requests" and the dashboard read the viewer's UserInbox, which repo objects update when an item is created, assigned, requested, closed or deleted. The index stores (repo id, number, kind, state, title, updated_at), enough to render a list; the page then asks the repo object for anything more.
  • Global search covers repositories and accounts only. Issue, pull request and code search run inside a repository, or across one owner by fanning out to that owner's repo objects (as code search already does). This is in place.
  • Crons become alarms. Each object schedules its own cleanup (stale tokens, expired invites, digest flushes). The few instance-wide sweeps that remain walk the D1 directory and kick objects.

Migration mechanics (per domain)

The deployment is small, so each domain moves in one step rather than through a long dual-write period:

  1. Schema in the object. The object creates its tables lazily, mirroring the D1 columns.
  2. Code switches to the object. Every read and write for the domain goes through the repo object.
  3. One-time copy. A cron task guarded by a KV flag walks the repo directory once and has each object copy its rows from D1 (see backfillInvolvementOnce in src/server/repo-state/index.ts).
  4. Drop the D1 tables in the next migration.

Phases

Phase Domain D1 tables leaving Notes
0 Code search, global search reduced to repos and accounts code_files, code_fts, code_index_state Done
1 RepoState skeleton, involvement index in UserInbox; the dashboard reads it none yet Done
2 Issues and pull requests issues, issues_fts, comments, issue_events, issue_labels, issue_assignees, labels, milestones, issue_types, pinned_issues, reactions, pull_requests, reviews, review_comments, review_requests, pull_issue_links, pull_auto_merges, issue_redirects, thread_subscriptions Largest change: about 90 files read these tables
3 Repo config and access collaborators, branch_protections, webhooks, webhook_deliveries, secrets, repo_variables, environments (and their secrets and variables), deploy_tokens, repo_security_settings, repo_mergefix_settings Moves permission checks into the repo object
4 CI workflow_runs, workflow_jobs, job_steps, job_artifacts, commit_statuses, workflow_triggers, workflow_schedules, workflow_event_receipts, action_caches, job_tokens, job_token_pushes, ci_notify_state, merge_queue_entries, mergefix_runs Schedules become alarms
5 Releases, LFS, security, discussions, projects, activity releases, release_assets, lfs_objects, lfs_locks, code_scanning_alerts, sarif_uploads, secret_alerts, security_advisories, dependencies, discussions (and comments, votes, categories), projects (and items, fields), push_events, activity, stars, watches, commit_statuses Stars and watches need a per-user mirror for "your stars"
6 Accounts sessions, passkeys, recovery_codes, access_tokens, auth_tokens, OAuth tables, notification settings and deliveries, notifications, follows, usage_*, credit and billing tables Sessions need a fast path: KV cache in front of AccountState
7 Sites sites, site_deployments, site_env, site_usage, preview comments, PR aliases Host routing keeps a small directory (hostname → site)

What stays in D1

  • The directory: accounts, users, user_emails, repos, repos_fts, redirects, retired_account_names.
  • Operator tables: admin_audit_log, abuse_reports, site_abuse_reports, instance_settings, signup_waitlist, stripe_events, cron_locks, hourly_limits, Artifacts call stats.

These are small, global by nature, and rarely written.