Marketing Engineering with AI · Coming soon
Stop calling A/B test winners that disappear after launch
For marketers and growth people who run A/B tests and are not sure the winners are real. You learn to size a test for your own traffic and conversion rate before it starts, write the stopping rule down, read the result with its uncertainty, and decide when your traffic is too small to test at all.
The course: A/B tests that are properly sized
$25 $49 early-bird price
This course is being built now. Join the early-access list to reserve the early-bird price and shape what it covers.
Joining the list is free. You only pay if you buy at launch, at the early-bird price.
If this sounds like your week
- The tool turned green after five days, you shipped the winner, and two months later conversion is back where it started.
- Someone checks the dashboard daily and wants to stop the test the moment one version pulls ahead.
- Your store gets a few hundred orders a month, and every test either runs for months or ends inconclusive.
- When leadership asks how sure you are, the only answer you have is the tool’s confidence percentage.
What you do now, and where it stops working
Most teams start a test, watch the dashboard, and stop when the tool reports significance or patience runs out. Testing tools show live results, and every early look raises the chance of calling noise a winner — so the habit produces wins that never appear in revenue.
What changes after the course
- Before a test starts, you know how many visitors it needs and how many weeks that takes at your traffic.
- A one-page pre-registration fixes the metric, the minimum lift and the stopping point before anyone sees data.
- Your readout reports the lift with a range, such as "between -1% and +6%", instead of a green badge.
- A small-traffic rule tells you whether to test a bigger change, use a more frequent metric, run longer or decide without a test.
How it works: Size, commit, read, decide
- 1Size: use the included calculator to find the sample and duration for your baseline rate and the smallest lift worth detecting.
- 2Commit: write the pre-registration — hypothesis, primary metric, sample size and stopping rule — before launch.
- 3Run: check that traffic splits evenly and the test is healthy, without reading the result early.
- 4Read: report the effect and its uncertainty, using worked examples of a clear win, a null and a misleading early lead.
- 5Decide: apply the small-traffic rule when the numbers say a test cannot answer the question in time.
What you get
- Sample size and power calculator spreadsheet
- Pre-registration template
- Test readout template with worked examples
- Video walkthrough of a test from plan to readout
This is for you if
- You run tests in VWO, Convert, Optimizely or a Shopify testing app and call winners from its dashboard.
- You are setting up a testing process for a team and want shared templates.
- You suspect your traffic is too small and want to know rather than guess.
Skip it if
- You have a statistics background and already run power analysis and sequential tests.
- You want a list of things to test. This course is about whether a test can answer your question.
- You run experiments on millions of users with an in-house platform.
Before you join
- I am not a statistics person. Will I be able to follow it?
- The course uses a spreadsheet calculator and plain definitions, and each idea appears in a worked example before it gets a name. You need to read a percentage and a chart, not derive a formula.
- Why not just use a free sample size calculator?
- You can, and the course shows you how. A calculator gives you a number; the course covers what to type into it, what to do when the answer is nine months, and how to report the result honestly once the test ends.
- My traffic is small. Is testing even possible for me?
- Sometimes it is not, and the course says so. You check feasibility first, then choose between a bolder change, a more frequent metric such as add-to-cart, a longer run, or a decision made without a test.
- How is this different from a $2,000 experimentation course?
- Live cohorts such as Ron Kohavi’s $1,999 course on Maven go deep on large-scale programmes. This course is self-paced, $49 at list price, and covers the planning, sizing and readout steps a small team runs every week.
- Is the course finished?
- No. It is in early access, built in public, with the calculator, templates and worked examples included at launch. Joining the list is free and gets you the launch date and the $25 early-bird price.
Rather have it built for you? Describe the project and the same work can be done for your team.