Skip to content

Canaries and gradual release

Every deploy to Live reaches Customers in steps: a canary first, then larger shares, each held while guards compare it with the Version before.

A rollout: its states

Filled: a state someone outside the run sees. Hollow: a private state, a step inside the run.

One deploy, hour by hour

A rollout moves Customer orgs, never single calls. Here is one, with a canary at 5%, steps held two hours, and the guards it carries.

HourOn the new VersionWhat happens
05% of orgs, picked at randomThe Verifier's suite has passed; the canary starts, and no org pinned to a Version is picked
0 to 25%Every minute, the guards compare the new Version with the old over the same minutes
225%The hold passed with every guard holding, so the next step takes its share
450%The same again
6Every orgThe last step: the rollout is promoted, and the Version before is superseded
Any minuteBack to noneOne guard fails: every org on the new Version moves back within a minute, and the rollout ends rolled_back

During a rollout, an org's API, MCP, SDK, site, portal and Watches all answer from one Version. Records written under the new Version stay after a rollback; the Version before reads them, because a rollout only carries changes it can read.

A rollout is canary, stepping or paused, then promoted or rolled_back. Each Customer org in it is old, then new, and either stays new for good or is moved_back. A guard is holding until one comparison leaves it failed.

The guards

Each guard reads one measure of the new Version against the old, and one failure ends the rollout.

GuardWhat it readsIt fails when
errorsThe share of calls that fail on the platform's side or the Function's, never a refusal a caller earnedThe new share is above the old by more than errors
slowerThe median time of each VerbThe new median is slower than the old by more than that share
evalsOutputs of each generative and agentic Verb, sampled and scored by its evalsThe new score falls below evals
ConformanceThe Verifier's suite, run on the new Version before the canaryAny check fails; nothing is declared

Every step and every failure writes an Event, so a Watch or an alert route hears it.

What rolls out, and what never does

Only Live rolls out. Find the change you are shipping.

The changeHow it reaches Customers
A new prompt, a price, a new Verb or fieldIn steps, guarded, as above
A removal, or a narrowed typeNever in steps: deploy refuses it with unsafe_rollout, and the change ships in two deploys
A breaking changeAs a new Version each Key moves to by sending Api-Version; the rollout governs which new Keys start on it
Anything on Test or a branchAt once
A rollout you want to hold, finish or undoBuild plane Verbs: Rollout.pause holds it at a step, Rollout.promote takes it to everyone with a person's approval, and Rollout.stop sends everyone back

Example

The agent product changes its booking prompt, and the change reaches 5% of Customers first.

definition.ts
27 lines
import { define, agentic } from 'apis.dev'

/** An agent that books three meetings a week with each Customer's leads, in their own inbox and CRM. */
export default define({
  name: 'meetings', domain: 'meetings.apis.dev',
  sources: {
    inbox: { system: 'email', scope: 'customer' },
    crm:   { system: 'crm',   scope: 'customer' },
  },
  Meeting: {
    with: 'string', company: 'string', at: 'date',
    book: agentic("Book a meeting with the next lead in {crm}, in the lead's own time zone",
      { tools: ['inbox', 'crm'], returns: { meeting: 'string' } }),
  },
  on: { 'every Mon, Wed, Fri 09:00': 'Meeting.book' },
  offer: {
    'Meeting.book': { per1000Tokens: 0.01, perMinute: 0.05 },
    'Meeting.get': 0, 'Meeting.list': 0,
  },
  operate: {
    promise: { 'Meeting.book': { success: 0.99, median: '10m' } },
    rollout: {
      canary: 0.05, steps: [0.25, 0.5, 1], hold: '2h',    // each step held two hours
      guard: { errors: 0.01, slower: 0.25, evals: 0.9 },   // past any, every Customer goes back
    },
  },
})

Your coding agent reads the rollout between steps.

Response
HTTP/1.1 200 OK
Content-Type: application/json
X-Cost: price=0, currency="USD"
X-Request-Id: req_4e81c0a2d9b7f3561e0c

{
  "id": "ro_7",
  "state": "stepping",
  "version": "v4",
  "from": "v3",
  "step": { "share": 0.25, "next": "2026-10-27T16:00:00Z" },
  "customers": { "v4": 31, "v3": 89, "pinned": 4 },
  "guards": {
    "errors": { "new": 0.004, "old": 0.003, "limit": 0.01, "ok": true },
    "slower": { "new": "6m", "old": "6m", "limit": 0.25, "ok": true },
    "evals": { "new": 0.93, "limit": 0.9, "ok": true }
  }
}

Your agent sent GET https://apis.dev/v1/products/meetings/rollouts/ro_7; npx apis.dev rollouts get and getRollout over MCP answer the same.