Skip to content

Price and offer variants

A variant is another version of the Offer, shown to a share of new Customers and measured against the Offer as written.

An experiment, from deploy to its conclusion

Filled: a state someone outside the run sees. Hollow: a private state, a step inside the run.

What to test, and how to write it

Start from the question you want answered. Each row is one variant in offer.variants, beside the control.

You want to knowThe variant changesThe goal to measure
Whether a higher price loses buyersA plan's pricerevenue, net revenue per exposure in the window
Whether a trial brings more paying CustomersA plan's trialpaid, a first paid charge
Whether more included calls keep CustomersA plan's includes or seatsretained, still paying after 30 days
Whether a lower price per call grows useA price per call, by consumption or by outcomerevenue
Whether a free plan brings sign-upsWhich plans the Offer holdssignup

A variant never changes the terms, the refund rule or the licence. The stop rule is experiment.exposures per variant or within, whichever comes first, and an experiment never concludes on less.

Who is in it, and what each sees

Visitors, new sign-ups and buyer accounts with no plan yet join; a current Customer and a signed agreement keep their prices.

WhoWhat they see
A visitorOne variant, assigned at its first exposure and kept for good across devices and every Interface; the same price after sign-up, and a line on the offer card that a test is running
A Customer already payingThe price it signed up at, until a change reaches it by the notice and grandfathering rules on agreements and price changes
A buyer's agentOne variant per buyer account on every card it fetches; the signed card, the quote and the receipt carry it, so it pays the price it compared
Anyone reading /pricing.jsonThe variant its caller sees, named
YouThe analysis per variant, the interval for each difference, and the one Verb that keeps a variant

For one Customer, /pricing, X-Cost, the offer card, the quote and the receipt print the same assigned price. An exposure is the first time a variant's price is shown there, or on a 402 offer or Checkout. Each is written as an Event.

Reading the result

The analysis is a projection of the Event stream, so it and the bill read the same records.

Example

The weekly brief tests a higher Member price and a free trial, each on a quarter of new readers.

definition.ts
28 lines
import { define, generative } from 'apis.dev'

/** A weekly brief on the software companies that changed headcount, for members and their agents. */
export default define({
  name: 'briefs', domain: 'briefs.apis.dev',
  sources: {
    firms: { api: 'companyfacts.apis.dev', cost: { 'Company.search': 0.005 } },
  },
  Issue: {
    week: 'string!#', title: 'string', body: 'markdown',
    status: {
      draft:   { run: generative('This week in {firms}: who hired, who shrank') },
      review:  { in: 'drafted', role: 'editor' },
      publish: { in: 'reviewed', role: 'editor' },
      send:    { in: 'published', to: 'Member' },
    },
  },
  on: { 'every Mon 06:00': 'Issue.draft', 'Issue.published': 'Issue.send' },
  offer: {
    'Issue.list': 0, 'Issue.read': 0.25,
    Member: { month: 8, includes: { 'Issue.read': 'all' } },
    variants: {
      higher: { share: 0.25, Member: { month: 10, includes: { 'Issue.read': 'all' } } },
      trial:  { share: 0.25, Member: { month: 8, trial: '14d', includes: { 'Issue.read': 'all' } } },
    },
    experiment: { goal: 'revenue', exposures: 1000, within: '30d' },  // half see the Offer as written
  },
})

After 30 days, the experiment is ready, and neither variant is clear of the control.

Response
HTTP/1.1 200 OK
Content-Type: application/json
X-Cost: price=0, currency="USD"
X-Request-Id: req_b2d07e45c9a1f3860d4e

{
  "id": "exp_3",
  "state": "ready",
  "goal": "revenue",
  "variants": [
    { "name": "control", "exposed": 2140, "converted": 96, "perExposure": 0.36 },
    { "name": "higher", "exposed": 1066, "converted": 41, "perExposure": 0.38,
      "difference": { "low": -0.03, "high": 0.08 } },
    { "name": "trial", "exposed": 1072, "converted": 52, "perExposure": 0.39,
      "difference": { "low": -0.02, "high": 0.08 } }
  ],
  "clear": false
}

Read it with GET https://apis.dev/v1/products/briefs/experiments/exp_3, npx apis.dev experiments get, or getExperiment over MCP.

Each variant prints its exposures, its conversions, the rate, revenue per exposure, and a 95% interval for its difference from the control. Here both intervals cross zero, so clear is false and neither variant has earned a decision.

Keeping a variant

Deciding what to test and which variant to keep stays yours.

Experiment.conclude keeps one, and the next deploy writes it as the Offer; Customers assigned another keep their price by changes.grandfather.

The experiment is running, ready, then concluded or stopped; a deploy without variants stops it. A variant runs until the conclusion makes it kept or retired. An assignment is assigned, exposed, then converted when the goal is met inside the window, or lapsed.