Your team already has the experiment-readout and csv-to-chart playbooks — the worked recipes for reading out an A/B test and rendering a CSV into a labeled PNG, each done once, well. This module is the layer above the recipe: the judgment that turns a statistics output and a chart-maker into two decisions a skeptic can’t take apart — a ship call that holds up to “how sure are you?” and a chart whose numbers were checked before anyone saw the picture.
It’s Module 4 of the certifiable Data & Analysts track, and it inherits everything you built in Module 1, Module 2, and Module 3. The dataset you profiled, the metrics you defined, and the queries and quality checks you built are what an experiment readout and a chart actually run on — a test read against an undefined “conversion,” or a chart built on an unreconciled column, is exactly the blind analysis Module 1 exists to prevent.
A green dashboard and a clean chart are the two most persuasive objects in a company — and the two easiest to get confidently wrong. An experiment readout that hides its uncertainty behind “statistically significant” ships a change that does nothing, dressed as proof. A chart that’s pretty before it’s verified launders a wrong number into something nobody re-checks once it’s in a slide. Both failures look identical to success right up until someone asks the second question. This module is where you learn to survive the second question, before anyone has to ask it.
Reading an experiment — the discipline that survives a skeptic
The playbook gives you the steps: compute the rates, get the interval, check the guardrails. Mastery is the judgment inside each step — the calls that separate a readout that holds up from one that just confirms what someone hoped the test would say.
- Pre-register before you peek, or say so out loud. The hypothesis, the one primary metric, and the minimum effect worth shipping for have to be written down before you look at the result — otherwise the criteria can quietly bend to whatever the data did. A primary metric chosen after the reveal isn’t analysis, it’s storytelling with a p-value attached. If a test wasn’t pre-registered, the honest move is to say the readout is exploratory, not to pretend it wasn’t.
- Read the interval, not the point estimate. “Conversion went up 0.3 points” is a number; “conversion went up 0.3 points, 95% CI [+0.05pp, +0.55pp]” is the truth. The interval is what tells you whether you’re looking at a real, sizeable effect or a real-but-too-small-to-matter one whose low end barely clears zero. A bare percentage with no interval is a guess wearing a lab coat.
- A win that breaks a guardrail isn’t a win. Before the primary metric gets to celebrate, check that nothing else moved the wrong way — revenue per account, support volume, refunds, latency. A treatment that lifts conversion by shipping more support tickets per account didn’t help the business; it moved the cost somewhere harder to see. Guardrails are the check that stops a local win from becoming a company-wide loss.
- One lucky segment is a hypothesis, not a headline. Slice a flat or modest result finely enough and something will clear significance by chance alone. The segments you’re allowed to read as evidence are the ones named before the test ran; anything found by hunting afterward — “but it’s up 14% just in the UAE” — goes in the readout labeled exploratory, never as the reason to ship.
- Lead with the decision, not the dashboard. A readout that buries “ship / iterate / kill” under six paragraphs of methodology makes the reader do the judgment work you were supposed to do. State the call first, then the evidence that earned it — effect size, interval, guardrail status, and what you genuinely can’t conclude yet.
- Validate the plumbing before you trust the result. A sample-ratio mismatch, a broken randomization, or a test that ran too short to clear day-of-week noise produces a confident-looking number that means nothing. Check group sizes against what you’d expect, check the groups are balanced on attributes that shouldn’t differ, and confirm the run covered full weeks — before the result earns any trust at all.
The chart — verified before it’s pretty
A chart is the most persuasive artifact an analyst produces, and the most dangerous one to get wrong, because once it’s a clean image nobody re-derives the math behind it. The discipline here is sequencing: verify, then render — never the other way around.
- Pick the view the question actually needs. A trend wants a line, not a single before/after bar that hides everything that happened in between. The wrong view doesn’t just look worse — it can actively misrepresent the story, flattening a steady decline into something that reads as a one-time dip.
- Reconcile before you render. The numbers behind the chart get checked against a second, independent source before they become a picture — never after. Once a chart looks finished, the visual polish gives the underlying math a credibility it hasn’t earned yet. Catching a discrepancy after the chart is in a deck means re-issuing a “the chart was wrong” correction in front of leadership, which costs far more credibility than the extra ten minutes would have.
- Label like the chart has to stand alone. A real title, labeled axes with units, and a caption naming the source file and the date range are what let the chart travel without you in the room to explain it. An unlabeled chart is a screenshot of a guess; once it’s pasted into a deck six months from now, the caption is the only thing left defending it.
- Own the headline, not just the picture. Claude can render an accurate chart and still let an overstated headline ride next to it. “CSAT is recovering” on a chart that shows three weeks of decline and one flat week is a claim that has your name on it the moment it ships — read the takeaway sentence as critically as you read the axis.
- Aggregate before anything identifying gets near a slide. If the chart is built from account-level or customer-level rows, group it to a pattern — by region, by week, by plan tier — before it’s rendered. A chart’s job is to show the shape of the data, not to expose the rows underneath it; that line matters as much for a CSAT trend as it does for revenue.
Your assignment
Do both deliverables for one real decision and one real chart — your own (recommended: a live test or a chart leadership is actually waiting on) or the sample brand Mizan, the GCC bookkeeping SaaS whose Business Analyst, Maya Haddad, runs throughout this track. Everything here inherits the foundation, query, and quality discipline from M1–M3; if you haven’t built those files yet, do those modules first. Open your data folder in Claude Desktop, approve each read in the “Ask permissions” prompt, and work in the chat — no terminal needed.
Module 4 deliverable — experiments & charts
Inherits from M1-M3: dataset-profile.md, metrics.md, analysis-brief.md,
your query/join discipline, your quality-audit habits
1. experiment-readout.md (one page, for one real A/B test)
- the pre-registered hypothesis, primary metric, and minimum effect
worth shipping for — stated as committed BEFORE the result, or
flagged honestly as exploratory if it wasn't
- a test-health check: sample ratio (group sizes as expected),
randomization/balance check, run length
- the primary metric read with its 95% confidence interval, not a
bare percentage — control rate, treatment rate, absolute and
relative difference, interval
- guardrail metrics checked and stated clean or flagged
- the pre-declared segment read, with any post-hoc segment
explicitly labeled exploratory, not evidence
- a clear verdict: ship / iterate / kill, with the one-line reason
and an honest "what we can't conclude from this" line
2. A verified, labeled chart (PNG + the numbers behind it)
- the view chosen for the question being asked, with the rejected
alternative named
- the underlying numbers table, reconciled against a second source
before rendering, with any discrepancy explained
- the rendered PNG: real title, labeled axes with units, a caption
naming the source file and date range
- a one-line, honest headline takeaway that matches what the chart
actually shows
Bilingual teams: if either deliverable goes to an Arabic-reading
leader, author the verdict/headline in Arabic from the same reconciled
numbers — never translate the English draft.The Foundation toolkit supplies the dataset-profile and metrics templates this module’s deliverables inherit from.
How it’s graded — the rubric
This is the part the free playbooks don’t have. Your two deliverables are scored against five criteria. Each is meets / nearly / not yet — and a “nearly” on any one is a revise, not a pass.
Experiments & charts rubric
1. The test was pre-registered, or the gap is named
Hypothesis, primary metric, and minimum worth-shipping effect are
stated as committed before the result — or the readout explicitly
flags itself as exploratory if they weren't. No criteria invented
after seeing the data.
2. The effect is read with its uncertainty
The primary result is reported as a confidence interval, not a
bare percentage. The verdict reasons about where the interval sits
relative to the worth-shipping bar, not just whether it's "positive."
3. Guardrails and segments are checked honestly
Guardrail metrics are explicitly confirmed clean or flagged. Any
post-hoc segment is labeled exploratory, never presented as the
reason to ship.
4. The chart's numbers were verified before rendering
The underlying table is reconciled against a second source, with
any discrepancy explained, before the PNG exists — not after.
5. The chart is labeled and the headline is honest
Real title, labeled axes with units, a caption with source and date
range, and a headline takeaway that matches what the data actually
shows — no overstated claim riding on an accurate picture.The discipline is deliberately what a skeptical stakeholder would demand on a call: a readout that hides its uncertainty, or a chart whose numbers nobody checked, fails quietly — right up until someone asks the second question. That’s why it’s caught here, not in the meeting.
The bar, shown — a worked model answer (Mizan)
You don’t have to guess what “meets” looks like. Here are passing excerpts for the sample brand — Maya Haddad built these the week Mizan’s growth lead asked “did the trial test win” and leadership separately asked for a CSAT chart for the board narrative. Yours doesn’t need to look identical; it needs to clear the same bar.
experiment-readout.md — 14-day vs. 30-day free trial (excerpt)
Pre-registered (committed before result, per the test plan in Slack
dated 2026-02-10):
Hypothesis: A 30-day trial gives small-business owners enough time
to import real bookkeeping data and see value, raising trial-to-
paid conversion versus the current 14-day trial.
Primary metric: trial-to-paid conversion rate within 45 days of
signup.
Minimum worth-shipping effect: +1.5 percentage points absolute
(the estimated lift needed to offset 16 extra days of free
hosting cost per trial, per Finance's break-even model).
Guardrails: average revenue per account (ARPA) at day 90, support
ticket volume per trial.
Pre-declared segments: UAE vs. rest-of-GCC, Starter-intent vs.
Pro-intent at signup.
Test health:
Group sizes: control (14-day) 2,210, treatment (30-day) 2,196 —
49.8% / 50.2%, no sample ratio mismatch.
Balance check: groups match on signup channel mix and plan-intent
at signup (chi-square not significant, p=0.61).
Run length: 6 full weeks (2026-01-19 to 2026-03-02), covering two
full billing cycles — no day-of-week or month-boundary skew.
Contamination: 0 accounts appear in both variant logs.
Primary metric:
Control (14-day): conversion 11.2% (n=2,210)
Treatment (30-day): conversion 13.6% (n=2,196)
Difference: +2.4pp absolute (+21.4% relative)
95% CI on the difference: [+0.6pp, +4.2pp]
Read: the interval is entirely positive and its low end (+0.6pp)
clears the +1.5pp worth-shipping bar at the midpoint and above, but
the bottom of the interval (+0.6pp) does NOT clear it — there's a
real chance the true effect is smaller than what's worth the cost.
Guardrails:
ARPA at day 90: control AED 187, treatment AED 183 — within noise,
no regression.
Support tickets per trial: control 0.41, treatment 0.44 — a small
increase, expected from a longer trial window generating more
setup questions; not flagged as a guardrail breach at this size.
Segments (pre-declared only):
UAE vs. rest-of-GCC: effect holds in both (+2.1pp UAE, +2.8pp
rest-of-GCC) — not concentrated in one.
Starter-intent vs. Pro-intent: effect concentrated in Starter-intent
signups (+3.6pp) vs. Pro-intent (+0.4pp, not significant) — both
pre-declared, so this is a real read, not a fishing result.
NOT used in this verdict: a UAE-only, Pro-intent-only slice was
flagged by the dashboard at +9.1% — this was not pre-declared and
sits on only 94 accounts. Logged as a hypothesis for a dedicated
follow-up test, not as evidence here.
VERDICT: SHIP. The interval clears the worth-shipping bar at the
midpoint and mostly above it, guardrails are clean, and the effect
is not an artifact of one lucky segment — it holds across both
pre-declared geography slices. Recommend shipping the 30-day trial
for Starter-intent signups first (where the effect is strongest and
most certain) and re-testing Pro-intent on a larger sample before
extending it there, rather than reading +0.4pp on Pro-intent as a
loss.
What we can't conclude from this: whether the effect holds beyond
day 90, or whether a 21-day trial would capture most of the lift at
lower hosting cost — both are reasonable follow-up tests, not this
one's job to answer.csat-trend.png — board chart (description, since this excerpt can't
embed the image)
View chosen: a line chart, weekly CSAT score on the y-axis, week on
the x-axis, 2026-01-05 through 2026-03-30. Rejected a single
before/after bar (Q4 avg vs. Q1 avg) — a two-bar chart would show
"CSAT dropped" but hide that the drop was a steady six-week slide,
not a one-time event, which matters for the board's "is this still
falling or has it stabilized" question.
Numbers reconciled before rendering: pulled the weekly average from
the support team's csat-responses.csv and cross-checked the
Q1 endpoints against Layla Al-Nasser's (CX Lead) own tracking
sheet — both show 4.7 the week of Jan 5 and 4.2 the week of
March 23. One week (Feb 16) was a 0.3-point outlier explained by a
known one-day survey-tool outage that under-sampled responses;
flagged in the caption rather than smoothed away.
Rendered chart:
Title: "Customer CSAT, Weekly — Q1 2026"
Y-axis: "CSAT (1-5 scale)", gridlines at 4.0 / 4.5 / 5.0
X-axis: week-ending dates, labeled by month
Annotation: the trend line is flagged at the point support ticket
volume rose sharply (week of Feb 9, coinciding with the VAT
export issue cluster from the support queue)
Caption: "Source: csat-responses.csv, reconciled against CX team
tracking sheet. Feb 16 reading affected by a known survey-tool
outage (see note). Generated 2026-03-31."
Headline takeaway (matches what the chart actually shows): "CSAT
declined steadily from 4.7 to 4.2 across Q1, tracking the rise in
VAT-export and login tickets in February — not a single event."
(Rejected the friendlier draft headline "CSAT dipped in Q1" as too
soft for a six-week, 0.5-point slide the board needs to act on.)What you’ve proven — and what’s next
Clear the rubric and you’ve proven something a green dashboard or a pretty chart alone never can: that your ship calls and your charts survive the second question — the confidence interval instead of a bare percentage, the guardrail nobody forgot to check, the segment temptation named and resisted, the chart number reconciled before anyone saw the picture. That’s the Experiments & Charts stage of “Certified Data & Analysts with Claude.”
From here the track turns honest one-off analysis into a standing operation, assessed the same way:
- Module 5 — the reporting system: building the recurring report that comes out the same way every month, and standing up the self-serve system so a non-analyst can get a trustworthy answer — then the capstone, Insight-in-a-Box, the full investigation taken from raw export to certificate.
First, make this module’s discipline reusable. The readout structure and the chart-verification sequence here are the templates worth keeping — the next experiment and the next chart should start from the same checklist, not a blank prompt. If you’re rolling this across a team, the operating guide is the data-safety and verification layer that belongs underneath every readout and every chart this module produces.