Do I need to know statistics to run an experiment readout?

No — Claude computes the rates, the difference, the confidence interval, and the guardrail and segment checks, and explains each in plain English. Your job is judgment, not math: commit to the primary metric and the worth-shipping bar before you look, read the effect against its uncertainty rather than as a single number, and decide whether a likely-real effect is big enough to justify the rollout. The playbook is built so a non-statistician can make a statistically honest call.

The dashboard says the result is statistically significant — can't I just ship?

Not on that alone. Significant means the effect is probably not zero; it says nothing about whether the effect is big enough to be worth shipping, whether a guardrail metric regressed, or whether the test itself was sound. A tiny significant effect that breaks refund rate is a loss, not a win. Read the effect size with its confidence interval against your minimum worth-shipping bar, and confirm the guardrails are clean, before the green number becomes a decision.

What if we didn't pre-register a hypothesis or primary metric?

Then the readout is exploratory, and you say so — which changes how much weight it can bear. Without a pre-committed metric and bar, it's easy to unconsciously pick the criteria that make the result 'win', so the honest move is to treat the findings as hypotheses for a properly pre-registered rerun rather than as a ship decision. Step one flags this explicitly; the fix is to pre-register the next test, which the playbook helps you template.

One segment shows a much bigger effect — can we ship just for them?

Only if that segment was pre-declared before you saw the data. If you found it by slicing after the fact, it's a hypothesis, not evidence — finely-sliced experiments throw up false winners by pure chance, and shipping on a post-hoc subgroup is one of the most common ways a readout misleads. Note it as something to test deliberately next time, and make this decision on the pre-committed primary metric across the pre-declared segments.

Read out an A/B test honestly — Claude Code Playbook

when to reach for this

An experiment ran, the dashboard shows green, and someone asks "did it win — can we ship it?" This is where analysis gets dangerous, because the wrong answer ships a change that does nothing (or quietly hurts) to every user, with a number that looked like proof. An honest readout is the discipline that stops that: state the hypothesis and decision rule before you peek, check the experiment itself was sound, read the primary metric with its uncertainty rather than as a single number, confirm the guardrail metrics didn't regress, and resist the urge to dredge a win out of one lucky segment. Claude runs the statistics; you own the call — and the whole point is to make that call survive a skeptic.

gather this first

The experiment data: who was in control vs treatment (the assignment) and their outcomes — experiment.csv with a variant column and the metric columns, ideally one row per unit (user/account).
What you pre-decided, if anything: the hypothesis, the one primary metric, the minimum effect worth shipping for, and how long it was meant to run. If none of that was written down, that's the first finding.
The guardrail metrics that must not regress (latency, refunds, unsubscribe, error rate) and who actually decides ship — so the readout lands as a decision, not a debate.

the workflow

State the hypothesis and decision rule before you look at the result

Open the folder with the experiment data in Claude Desktop and ask in the chat — no terminal needed. Lock the question and the bar before seeing the outcome, so the result can't bend the criteria. This is the single guard against unconsciously moving the goalposts to whatever the data did.

you ask

Before you compute any result, help me write down the test on the record: the hypothesis, the single primary metric that decides win/lose, the minimum effect size that would actually be worth shipping (not just 'positive'), the guardrail metrics that must not regress, and the segments we said up front we'd look at. If any of these weren't pre-decided, flag it — I want to commit to the criteria before I see the outcome.

what you get back A written decision rule: "primary = checkout conversion; ship if it's up at least 2% and no guardrail regresses; pre-declared segments: new vs returning." If nothing was pre-registered, an honest note that the readout is exploratory — which changes how much weight the result can bear.

Committing to the primary metric and the bar before the reveal is the whole game. A test where the success criteria are chosen after seeing the data can be made to 'win' almost every time — and everyone downstream pays for it.
Validate the experiment before you trust the result

A result is only as good as the test that produced it. Check the plumbing first — balanced groups, real randomization, enough data, ran long enough, no contamination — because a broken setup produces a confident number that means nothing.

you ask

Before reading the outcome, sanity-check the experiment itself. Are the control and treatment groups roughly the size we'd expect (flag a sample-ratio mismatch — a big imbalance suggests a broken assignment)? Do the groups look balanced on a couple of pre-treatment attributes, or did randomization fail? Did it run long enough and over full weeks to avoid a day-of-week skew? Any sign of contamination — users in both groups, or the same person counted twice?

what you get back A test-health verdict: "Groups 50.1/49.9 — fine. Balanced on signup cohort and plan. Ran 14 full days. No duplicate users across variants." Or a red flag — "treatment is 60% of users; assignment is broken, don't read the result" — caught before you draw a conclusion from a rigged base.

A sample-ratio mismatch or a failed randomization invalidates the result no matter how significant it looks. This is the step that separates 'the test says' from 'the dashboard says.'
Read the primary metric with its uncertainty — not as one number

Now read the result properly: the effect size and its confidence interval, so you see the range of plausible truth, not a point estimate that hides how unsure it is. "Significant" is a yes/no that throws away the most important information — how big, and how sure.

you ask

Now the primary metric only. Give me the control rate, the treatment rate, the absolute and relative difference, and the 95% confidence interval on that difference. State in plain English what the interval means here — the range of effects consistent with this data. Is the whole interval above my minimum worth-shipping effect, straddling zero, or below it? Don't dredge other metrics yet; just read the one we pre-committed to.

what you get back "Control 4.0%, treatment 4.3%, +0.3pp (+7.5% relative), 95% CI [+0.05pp, +0.55pp]. The interval is entirely positive but its low end (+0.05pp) is below your 2% worth-shipping bar — so it's likely a real but small effect, not clearly worth shipping." The uncertainty is on the table, not buried under a green checkmark.
Check the guardrails and the segments — honestly

Confirm the win didn't come at a cost, and resist the temptation to rescue a flat result with a cherry-picked segment. Subgroup effects are where false wins are manufactured; treat them as hypotheses, not conclusions.

you ask

Two checks. First, the guardrails: did any of [latency, refund rate, unsubscribes, errors] regress in treatment, even if the primary metric improved? A primary win that breaks a guardrail isn't a win. Second, the pre-declared segments only (new vs returning): does the effect hold, or is it concentrated in one? And be explicit: if you looked at any segment we did NOT pre-declare, label it as exploratory and not something we can ship on. Don't let a single lucky subgroup become the headline.

what you get back "Guardrails clean — no regression in refunds or latency. Effect is consistent across new and returning, not driven by one. (Note: the +14% you might want for 'enterprise' is an unplanned slice on 80 users — exploratory, not evidence.)" The honest read, with the cherry-picking pre-empted.

Slicing until something is significant — and then telling that story — is the most common way an honest-looking readout lies. Pre-declared segments confirm; post-hoc segments only generate hypotheses for the next test.
Write the ship / iterate / kill readout with the uncertainty stated

Turn it into a decision a leader can act on without re-deriving the stats — the verdict, the effect with its range, the guardrail status, and the honest caveats. State what you're unsure of; a readout that hides its uncertainty is how a marginal result gets oversold.

you ask

Write the experiment readout for the decision-maker: the verdict (ship / iterate / kill) against the rule we set, the primary effect with its confidence interval in plain English, the guardrail status, the segment read, and an honest 'what we can't conclude from this' line. Recommend a clear next step — ship, run longer for more power, or kill. Keep it to a page and lead with the decision. Save it as experiment-readout.md.

what you get back A one-page experiment-readout.md leading with the call: "ITERATE — effect is positive (+7.5%, CI +1% to +14%) but the interval includes effects too small to justify the rollout cost; recommend running one more week to tighten it before deciding. Guardrails clean. No segment surprises." A decision that survives the question "how sure are you?"

Leading with the decision and stating the uncertainty plainly is what makes the readout trustworthy. "It's significant, ship it" hides the size and the doubt; "likely a small real effect, here's the range, here's my recommendation" respects them.

make it your own

Pre-register the next one: the decision rule you wrote in step one is a template — save it as a /experiment-plan command (see the Features tab) and fill it in before the next test starts, so every experiment commits to its primary metric and bar up front and this readout becomes a formality, not a rescue.
Feed it into the report: when an experiment result belongs in the monthly numbers, hand the readout into Build a recurring analysis report — the reconciled effect and its caveats slot straight into the narrative without re-litigating the stats.
Bilingual decision: if the decision-maker reads Arabic, author the verdict and caveats in Arabic so the call and its uncertainty are unambiguous to them — the same effect and the same confidence interval, the words written for the reader.
Underpowered or messy test: if step two finds the test was too small or broken, the honest output is 'we can't conclude' plus a properly-sized plan for a rerun — Claude can estimate the sample size you'd actually need rather than over-reading a noisy result.

watch out for

Peeking and post-hoc criteria are how honest people fool themselves. Choosing the metric, the segment, or the stopping point after seeing the data turns almost any test into a 'win'. Commit to the primary metric and the bar in step one, before the reveal — that single discipline prevents most false ship decisions.
Statistically significant is not the same as worth shipping. A result can be real and far too small to justify the engineering and risk of a rollout — that's why the bar is a minimum effect size, not just 'positive and significant'. Read the effect against the cost, not against zero.
One lucky segment is a hypothesis, not a headline. Slice a flat experiment finely enough and something will look significant by chance. Only pre-declared segments confirm; any subgroup you found by hunting is fuel for the next test, and saying otherwise is the most common way a readout misleads.
Claude runs the statistics; you own the decision and its framing. It can compute the interval and flag a guardrail regression, but whether a likely-small effect is worth shipping is a judgment about cost and risk that's yours. Don't let 'the model said significant' stand in for a decision a human has to defend.

you'll end up with A one-page experiment readout that leads with a defensible ship / iterate / kill decision — the test validated, the effect read with its confidence interval, the guardrails confirmed, the segments read honestly, and the uncertainty stated plainly — so you ship on evidence that survives a skeptic instead of on a green number that doesn't.

Questions people ask

Do I need to know statistics to run an experiment readout?: No — Claude computes the rates, the difference, the confidence interval, and the guardrail and segment checks, and explains each in plain English. Your job is judgment, not math: commit to the primary metric and the worth-shipping bar before you look, read the effect against its uncertainty rather than as a single number, and decide whether a likely-real effect is big enough to justify the rollout. The playbook is built so a non-statistician can make a statistically honest call.
The dashboard says the result is statistically significant — can't I just ship?: Not on that alone. Significant means the effect is probably not zero; it says nothing about whether the effect is big enough to be worth shipping, whether a guardrail metric regressed, or whether the test itself was sound. A tiny significant effect that breaks refund rate is a loss, not a win. Read the effect size with its confidence interval against your minimum worth-shipping bar, and confirm the guardrails are clean, before the green number becomes a decision.
What if we didn't pre-register a hypothesis or primary metric?: Then the readout is exploratory, and you say so — which changes how much weight it can bear. Without a pre-committed metric and bar, it's easy to unconsciously pick the criteria that make the result 'win', so the honest move is to treat the findings as hypotheses for a properly pre-registered rerun rather than as a ship decision. Step one flags this explicitly; the fix is to pre-register the *next* test, which the playbook helps you template.
One segment shows a much bigger effect — can we ship just for them?: Only if that segment was pre-declared before you saw the data. If you found it by slicing after the fact, it's a hypothesis, not evidence — finely-sliced experiments throw up false winners by pure chance, and shipping on a post-hoc subgroup is one of the most common ways a readout misleads. Note it as something to test deliberately next time, and make this decision on the pre-committed primary metric across the pre-declared segments.