295 lines · 21.8 KB

Usage metering and billing

All billable usage is recorded through src/server/metering/. Every event is billed to one account (a user, an org, or a scratch: account for anonymous usage) and never to two.

Design

  • Source of truth: D1. usage_events holds one row per event, keyed by its idempotency key. AFTER INSERT triggers fold each new row into usage_periods (per account, meter and month), usage_repo_periods and usage_days in the same write. A retried event hits INSERT OR IGNORE on the same key, so it inserts nothing and the triggers never fire: retries can never double count.
  • Analytics Engine (USAGE binding, dataset gitfor_usage) gets a data point for every newly inserted event (index = account id; blobs = meter, period, repo id, idempotency key, attributes; double = quantity). Use it for high-cardinality exploration and the site-admin dashboards. It is best-effort and is never used for billing.
  • Hard caps use D1, not a Durable Object. A reservation is one D1 batch, which is a single transaction. The batch inserts the reservation row (with a random nonce) only when quantity + reserved + q <= cap, then adds the hold only if that nonce landed. It does not rely on changes(). D1 serializes writes, so this conditional check is atomic. It cannot be overshot by concurrent reservations, and it needs no extra binding or per-account DO hop. A DO counter would only pay off at more than about 1k reservations per second per account, and hard-capped meters (AI calls, editor seconds, scratch sites) are nowhere near that rate.
  • Cron (runMeteringCron, every 5 minutes) releases expired reservations, snapshots storage gauges, sends budget alerts, and prunes events older than 400 days. The aggregates are kept forever.
  • Periods are UTC calendar months (YYYY-MM) and are taken from the event time. An event at 23:59:59 on the last day of a month counts toward that month, even if it is recorded later.
  • Precision. Compute meters are in seconds, and fractions are allowed. Never round up to the minute. Pass (endMs - startMs) / 1000.

Where the work runs

Nothing in metering walks every account or repository:

  • Storage gauges. Each repository or site scope's RepoState recomputes what it stores per billed account five minutes after its storage tables change (gauges task) and writes it to those accounts' scope_gauges; each account re-sums them into its gauges a minute later (scope-gauges). A transfer moves the gauges at once and deletion retracts them. LFS follows lfs_usage writes (lfs-gauge), and at the start of each month the account carries its levels into the new period (gauge-carryover).
  • Holds expire from the account's alarm (reservations), and budget alerts run five minutes after usage or gauges move (alerts). Usage events older than 400 days are collected hourly. An editor container session whose hold ran out (or vanished) without being settled is closed by the same alarm (editor-sessions), billed at most what it held.
  • The platform AI ceiling reads one running counter (platform-ai in Counters) instead of summing accounts.
  • Site usage is folded per scope: each repository or site scope's alarm wakes after a UTC day ends, folds that day's site counters into the billed accounts' meters once (site_rollups remembers folded days), and sets the serving stop for any account it billed that reached a hosting hard cap. The platform site ceiling reads one running counter (platform-sites) that every site counter forwards its flushed deltas to.

Meters (src/lib/meters.ts)

Meter Unit Kind Who records it
storage.artifacts bytes gauge repo alarm, from repo_storage_estimates (pack bytes on push)
storage.lfs bytes gauge account alarm, from refcounted lfs_usage
storage.ci_cache bytes gauge repo alarm, from action_caches
storage.sites bytes gauge repo alarm, from active site_deployments
storage.r2 bytes gauge setGauge by the owning subsystem
compute.runner seconds counter Runner DO / ci (metering)
compute.runner_large seconds counter same
compute.runner_xlarge seconds counter same
compute.runner_self_hosted seconds counter ci on job completion; free
compute.editor seconds counter editor containers (container-meter.ts, reserve/commit); hard cap
compute.site_requests count counter sites: daily rollup of site_usage (sites/metering.ts)
compute.site_cpu ms counter sites dispatch (orangered-llama)
compute.workflow_steps count counter ci; free, informational
network.lfs bytes counter LFS download (metering)
network.sites bytes counter sites dispatch (orangered-llama)
network.downloads bytes counter raw and archive (metering)
ai.calls count counter AI Gateway path (meteredAiCall); hard cap
ai.tokens tokens counter same; attribute model
ops.integration_deliveries count counter integrations (lightblue-moth)
ops.mcp_calls count counter MCP hub (slateblue-rabbit)
ops.git_ops count counter git proxy (metering); attribute op = clone, fetch or push
ops.git_bytes bytes counter git proxy (metering); attribute direction = in or out
ops.action_fetches count counter ci actions (ci/actions/archive.ts); free, cold GitHub downloads

API

import { recordUsage, recordUsageSoon, setGauge, claimScratchUsage } from "~/server/metering/record";
import {
  reserveUsage,
  commitUsage,
  releaseUsage,
  assertWithinQuota,
  QuotaExceededError,
} from "~/server/metering/quota";
import {
  repoOwnerAccount,
  ciRunAccount,
  tokenAccount,
  scratchSiteAccount,
  scratchIpAccount,
} from "~/server/metering/attribution";

recordUsage(input): Promise<boolean>

await recordUsage({
  accountId, // billed account (use the attribution helpers)
  meter: "compute.editor",
  quantity: (endMs - startMs) / 1000,
  unit: "seconds", // optional; checked against the catalog
  repoId, // optional; powers "usage by repo"
  attributes: { session: id }, // small JSON; stored and sent to Analytics Engine
  idempotencyKey: `editor:${sessionId}:${segment}`, // stable per logical event, max 200 chars
  at: endMs, // optional event time; decides the period
});

The call returns false when the key was already recorded, or when quantity is 0. It throws MeteringError on an unknown meter, a gauge meter, a wrong unit, a negative quantity or a bad key.

Idempotency keys. Prefix the key with your subsystem and derive it from durable ids, never from Date.now() or randomId(). Examples: ci:<jobId>, editor:<sessionId>:<heartbeatSeq>, integration:<deliveryId>, mcp:<requestId>, ai:<runId>:<callIndex>.

Long-running compute should record in segments: one event per heartbeat or per stop, each covering [lastRecordedAt, now] and keyed by the segment. This bills partial usage when a container crashes, and a replayed segment is a no-op.

recordUsageSoon(ctx, input)

Use this on hot request paths. It runs the same logic in ctx.waitUntil and logs failures instead of throwing, so metering never fails a user's request.

setGauge(accountId, meter, value)

For storage gauges your subsystem owns (storage.r2). The billed amount is the current level, and the period peak is kept.

Hard caps: reserveUsage, commitUsage and releaseUsage

const reservation = await reserveUsage({
  id: `editor:${sessionId}:${seq}`,
  accountId,
  meter: "compute.editor",
  quantity: 600,
  ttlMs: 15 * 60_000,
});
// throws QuotaExceededError (map it to 402 or 429) when the reservation won't fit
try {
  /* do the work */
  await commitUsage(reservation, { quantity: actualSeconds, repoId, attributes });
} catch (error) {
  await releaseUsage(reservation.id);
  throw error;
}
  • id is idempotent. Reserving the same id again returns the original reservation without holding more.
  • commitUsage releases the hold and records the real amount (key reservation:<id> unless you pass idempotencyKey) in one D1 batch. Calling it twice is safe.
  • If you never settle, the account's alarm releases the hold when ttlMs (default 1 hour) runs out, and nothing is billed.
  • assertWithinQuota(accountId, meter, quantity?) is the pre-check. Use it to fail fast before expensive setup. For counters it holds nothing. For gauge meters with quantity > 0 (storage writes) it takes a short hold instead, see "Storage gauges" below.
  • Both refuse a disabled billed account (accounts.disabled_at) with AccountDisabledError, a QuotaExceededError subclass that maps to 403. Work by members of a disabled org on the org's repos is billed to the org, so it is refused too.

Soft meters, meaning those without a hard cap on the account's plan, always pass. They raise budget alerts instead.

Plans, caps and spending limits (src/lib/plans.ts)

  • The plans are scratch (anonymous), free and pro. Override them with the METERING_PLANS var (JSON in the same shape).
  • included is the monthly allowance. Alerts fire at 50, 80 and 100%. hardCaps block at the limit. On free every allowance is a hard cap (it never bills), except storage.ci_cache, which evicts least-recently-used entries instead. Enforcement points:
    • AI calls and tokens, editor seconds: reserved before use.
    • Cloud runner seconds: checked when a cloud job is dispatched.
    • storage.artifacts: pushes that add objects are declined (ref deletes still work).
    • storage.lfs and network.lfs: LFS uploads and downloads.
    • storage.r2: release assets and attachments. storage.sites: site deploys.
    • Site requests, CPU and egress: each site scope's alarm rolls up site_usage daily and flags owners at the cap (sites:over-cap:<account>:<period> in KV); the dispatcher then answers 429 until the next period or a plan or spending-limit change, which clears the flag.
    • Storage checks read gauges that follow storage writes after a few minutes' debounce.
  • On scratch the caps are site storage, requests and egress, and AI and editor are 0.
  • On pro, overage is billed at microCentsPerUnit. The spending limit turns each priced meter into a hard cap at included + limit / price. 0 means "never bill overage" (included usage still works). Site egress (network.sites) is unpriced, so pro also has a fixed 2000 GB hard cap on it.
  • Default spending limit. An account that never chose a limit (spending_limit_cents NULL) gets the plan's defaultSpendingLimitCents: DEFAULT_PRO_SPENDING_LIMIT_CENTS = 5000 ($50 of overage a month, a little over 3x the $15 Pro price) on pro. There is no "unlimited" setting; owners who need more set a higher limit explicitly, up to MAX_SPENDING_LIMIT_CENTS ($1,000,000). Clearing the field on the billing page returns to the default.
  • past_due grace. A Pro subscription that has been past_due for more than PAST_DUE_GRACE_DAYS (default 14, from past_due_since) meters as Free until it is paid (docs/billing.md).
  • Budget alerts go to the user, or to every org owner, as notifications rows with reason billing and a link to the billing page. Each threshold fires once per period, deduped by usage_alerts.

Attribution

  • Usage is charged to the repo owner (repoOwnerAccount).
  • Fork PR CI is charged to the base repo owner. Fork runs execute in the base repository (workflow_runs.repo_id), and the base owner approves outside-contributor runs before cloud time is spent. ciRunAccount(runId) encodes this.
  • Agent usage (MCP or OAuth tokens) is charged to the account that authorized the agent (tokenAccount(tokenId) reads mcp_token_agents.authorized_by).
  • Anonymous scratch sites use scratchSiteAccount(siteId), and other anonymous usage uses scratchIpAccount(ipHash). On claim, claimScratchUsage(scratchId, accountId) merges the scratch aggregates into the claimer and redirects later events with that scratch id. Nothing is counted twice, because the rows move rather than being copied.
  • Third-party actions. Running an action is part of the job, so it is billed as runner seconds to the run's account (ciRunAccount, base owner for fork PRs). Action archives are public code, content-addressed by commit and shared by every account: they are stored once in R2 and not billed to anyone, pruned after 30 idle days. The run that triggers a cold download from GitHub records one free ops.action_fetches event (key action-fetch:<owner>/<repo>@<sha>, so only the first fetch counts) for abuse visibility, and cold fetches are rate limited per account. Calls an action makes to GitHub use the repository's own GITHUB_COM_TOKEN secret, if any.
  • No double billing. Every meter has one recorder, and storage follows the LFS refcount precedent: shared objects are billed once per account. Forks start with a storage.artifacts estimate of 0, because Artifacts forks share objects.

Enforcement and known limits

  • Spending limits. withinSpendingLimit(accountId, meter?) sums the period's overage across every priced meter. The boundary is the same as spendGuard's (the in-transaction check in reserveUsage): overage may reach the limit but never exceed it, compared in whole microcents. Once the total is past the limit, reserveUsage and assertWithinQuota refuse priced meters, and cloud CI dispatch holds new jobs as quota-exceeded; a hold that would take it past the limit is refused atomically. Other soft meters keep recording and alerting.
  • Storage gauges (BILL-P9). Gauges move a few minutes after storage changes (repository debounce, then the account's re-sum), so a storage write (LFS upload, release asset, attachment) takes a 10-minute hold in usage_reservations (gauge:<random> ids, holdGauge in metering/gauge-hold.ts) with one conditional INSERT that counts every unexpired hold. Two concurrent writes can no longer both fit into the same headroom. Holds expire rather than settle (10 minutes, longer than the gauge debounce), so for up to 10 minutes new bytes may count twice, and a write that fails or a retried LFS batch still holds its bytes until expiry. Both err on the strict side: an account right at its cap can be refused for up to 10 minutes.
  • AI. aiMeter.beforeCall (1) checks the platform-wide daily ceiling, (2) reserves AI_TOKEN_HOLD (8000) ai.tokens, or what is left under the cap if less, and (3) reserves and counts one ai.calls. afterCall settles the token hold at the real count. A call may still overrun by its own tokens, but concurrent calls can no longer all start on the same last tokens. Unsettled holds expire after 15 minutes.
  • Platform daily AI ceiling. AI_DAILY_SPEND_CEILING_CENTS (default 10000 = $100/day when Stripe is configured, off otherwise; off disables) caps today's (UTC) platform-billed AI cost across every account, at Cloudflare cost (METER_PRICES / MARKUP), from usage_days plus in-flight token holds. Past it every non-BYOK call fails with PlatformLimitError (maps to 503). It fails closed: a value that is not a number, or a failed read, refuses the call. BYOK calls are never affected.
  • Request rate limits (RL-1). enforceRateLimit and withinRateLimit (src/server/limits.ts) count each hit in the RateLimiter Durable Object (rate-limiter-do.ts): a fixed window per limiter and key, persisted in SQLite, sharded by sha256(limiter, key) over 256 named instances, one RPC per check. Budgets mirror wrangler.jsonc ratelimits in RATE_LIMITS: AUTH 10/60s, WRITE 30/60s, HEAVY 30/60s, GIT 120/60s, REPORT 5/60s; change both together. The Workers Rate Limiting binding still runs alongside and is honored when it refuses, but it was observed never to refuse in production. If the object is unreachable, AUTH and REPORT fail closed (503) and the rest fail open with an error log. Refusals are cached per isolate until the window ends, so floods stop costing RPCs. Hourly credential budgets (enforceHourlyLimit) stay in D1.
  • Streams. Git, raw, archive and asset bytes are counted as they stream. A client that disconnects is billed for the bytes it already received.
  • network.downloads is informational and unpriced. LFS content served through raw or archive downloads is charged once, on network.lfs. The same bytes also appear in network.downloads, so the money is billed once.
  • AI retries. aiMeter keys calls by <runId>:<index>. A replayed workflow step that calls the model again is not charged a second time. In exchange, retries can never double-bill.
  • CI. A job's runner seconds are recorded once, when it reaches its terminal state (key ci:<jobId>). If that D1 write fails, it is logged and not retried.
  • Claims. Once a scratch id is claimed it is bound to that account. A claim by any other account is rejected. In-flight holds move to the claimer together with the usage, and storage gauge rollups follow the claim.

HTTP

  • GET /api/v1/user/usage?period=YYYY-MM returns the viewer's summary.
  • GET /api/v1/orgs/:org/usage?period=YYYY-MM returns the org's summary (org members only).
  • The UI is at /settings/billing, /orgs/<org>/billing (owners can edit the limit) and /admin/billing (site admins).

The summary shape is UsageSummary in src/server/metering/summary.ts: { period, plan, spendingLimitCents, totalCents, meters[], repos[], days[], alerts[] }.

Integration status

Address: bisque-deer-9651 (branch roadmap/metering).

Area Status
Git proxy (ops, bytes, storage estimate) Wired: src/server/git/proxy.ts and metering/git.ts
LFS bandwidth Wired: chargeBandwidth in src/server/lfs/quota.ts
Raw, archive and asset downloads Wired: handleRawRoutes in src/server/raw.ts
CI runner seconds (cloud and self-hosted) and steps Wired: completeJob in src/server/ci/jobs.ts. The spending limit gates cloud dispatch in ci/dispatch.ts
AI (mergefix and Clef) Wired: aiMeter in metering/ai.ts, called from the planRun meter callback (mergefix owns that file from now on)
Storage gauges (LFS, CI cache, Artifacts estimate, sites, R2) Wired: scope gauges (metering/scope-gauges.ts)
Sites requests, CPU, egress and scratch claim Hand-off: src/server/sites/metering.ts adapter (orangered-llama). At merge, replace its bodies with recordUsageSoon, assertWithinQuota and claimScratchUsage
Editor containers Wired: src/server/editor/container-meter.ts reserves the session budget on start and commits the exact seconds on every stop
Integration deliveries Hand-off: recordOutboundDelivery in src/server/integrations/metering.ts (lightblue-moth) becomes recordUsageSoon(... "ops.integration_deliveries" ..., key integration:<deliveryId>)
MCP tool calls Hand-off: recordToolUsage in src/server/mcp/usage.ts (slateblue-rabbit) becomes recordUsageSoon(... "ops.mcp_calls" ...)
Budget alerts Interim direct notifications insert in deliverAlerts. Switch to notifyQuotaThreshold from roadmap/notify at merge