Practical A/B Testing
A/B testing is a powerful technique that allows businesses to understand the causal impact of changes made to their products on their users and overall business success. Running successful A/B tests requires a proper setup, execution, and the ability to thoughtfully analyze results. This course will equip learners to master the analysis and interpretation skills for practical A/B testing, even in the most complicated of scenarios.
Course taught by expert instructors
Stephanie Pancoast
Senior Data Science and Analytics Manager
Stephanie is a Senior Manager of Data Science and Analytics. She recently led the improvement of A/B testing at Strava through platform enhancements, education, and process implementation. Prior to Strava, Stephanie worked at Airbnb as a Senior Data Scientist, assisting with over 100 A/B tests and co-developing an internal course on common A/B test interpretation mistakes. Stephanie also received a Ph.D. in Electrical Engineering from Stanford University, specializing in Applied Machine Learning.
The course
Learn and apply skills with real-world projects.
Data Scientists working on experimentation platforms and strategies for their organization
Product or Data Analysts performing A/B tests on their products
ML Engineers involved in A/B testing their deployed models
Foundational knowledge of Python programming: Variables, Functions, Lists, Loops
Try these prep courses first
- Learn
- Why A/B test?
- What exactly is A/B testing?
- Building blocks to interpretation: P-values, Confidence Intervals
- Determining run-time (power analysis + assignment rate)
- Picking the right metrics
Project- Get your hypothetical product team up and running
- What metric should they focus on? What should the guardrail(s) be?
- Write the brief (hypothesis, run time) for 2 tests
- Analyze and summarize the findings of the 2 tests
- Learn
- Different types of A/B tests: decision, measurement, defensive
- Outliers: how to handle them
- Multiple hypothesis testing, including segmentation
Project- Analyze 2 test results that just came in and prepare the brief (experiment design) for 3 others
- No detectable effect - but segmenting the results shows there is
- P-value is close to 0.05. Also getting used to the idea that more of than not, there is no detectable impact
- More practice with hypothesis generation/ power analysis
- Learn
- The importance of setup: Imbalance, dilution
- Beware of early results - what can cause negative and early results to be different than the actual impact and how to handle it.
Project- The tests you designed last week just started and early results show one test very negative and another is positive - plan how you'd interact and communicate with product teams
- Dig into those two tests and figure out what’s going on and make a recommendation
- Analyze the results of that + the non-issue one and summarizing the findings.
- Learn
- Concurrent tests and how to approach them
Project- Bring your own test results to analyze OR
- Analyze two tests that interact, one has has an outlier.
A course you'll actually complete. AI-powered learning that drives results.
AI-powered learning
Transform your learning programs with personalized learning. Real-time feedback, hints at just the right moment, and the support for learners when they need it, driving 15x engagement.
Live courses by leading experts
Our instructors are renowned experts in AI, data, engineering, product, and business. Deep dive through always-current live sessions and round-the-clock support.
Practice on the cutting edge
Accelerate your learning with projects that mirror the work done at industry-leading tech companies. Put your skills to the test and start applying them today.
Flexible schedule for busy professionals
We know you’re busy, so we made it flexible. Attend live events or review the materials at your own pace. Our course team and global community will support you every step of the way.
Completion certificates
Each course comes with a certificate for learners to add to their resume.
Best-in-class outcomes
15-20x engagement compared to async courses
Support & accountability
You are never alone, we provide support throughout the course.
Get reimbursed by your company
More than half of learners get their Courses and Memberships reimbursed by their company.
Hundreds of companies have dedicated L&D and education budgets that have covered the costs.
