Sold out, but you can still join the waitlist!
Course logo

Practical A/B Testing

A/B testing is a powerful technique that allows businesses to understand the causal impact of changes made to their products on their users and overall business success. Running successful A/B tests requires a proper setup, execution, and the ability to thoughtfully analyze results. This course will equip learners to master the analysis and interpretation skills for practical A/B testing, even in the most complicated of scenarios.

Instructor profile photo
Stephanie Pancoast
Senior Data Science and Analytics Manager
Real-world projects that teach you industry skills.
Learn alongside a small group of your professional peers
Part-time program with 2 live events per week:
Next Cohort
May 22, 2023
4 weeks
US$ 400
or included with membership

Course taught by expert instructors

Instructor Photo
Affiliation logo

Stephanie Pancoast

Senior Data Science and Analytics Manager

Stephanie is a Senior Manager of Data Science and Analytics. She recently led the improvement of A/B testing at Strava through platform enhancements, education, and process implementation. Prior to Strava, Stephanie worked at Airbnb as a Senior Data Scientist, assisting with over 100 A/B tests and co-developing an internal course on common A/B test interpretation mistakes. Stephanie also received a Ph.D. in Electrical Engineering from Stanford University, specializing in Applied Machine Learning.

The course

Learn and apply skills with real-world projects.

Who is it for?
  • Data Scientists working on experimentation platforms and strategies for their organization

  • Product or Data Analysts performing A/B tests on their products

  • ML Engineers involved in A/B testing their deployed models

Prerequisites / Commitment
  • Foundational knowledge of Python programming: Variables, Functions, Lists, Loops

Not ready?

Try these prep courses first

  • Why A/B test?
  • What exactly is A/B testing?
  • Building blocks to interpretation: P-values, Confidence Intervals
  • Determining run-time (power analysis + assignment rate)
  • Picking the right metrics
  • Get your hypothetical product team up and running
  • What metric should they focus on? What should the guardrail(s) be?
  • Write the brief (hypothesis, run time) for 2 tests
  • Analyze and summarize the findings of the 2 tests
  • Different types of A/B tests: decision, measurement, defensive
  • Outliers: how to handle them
  • Multiple hypothesis testing, including segmentation
  • Analyze 2 test results that just came in and prepare the brief (experiment design) for 3 others
  • No detectable effect - but segmenting the results shows there is
  • P-value is close to 0.05. Also getting used to the idea that more of than not, there is no detectable impact
  • More practice with hypothesis generation/ power analysis
  • The importance of setup: Imbalance, dilution
  • Beware of early results - what can cause negative and early results to be different than the actual impact and how to handle it.
  • The tests you designed last week just started and early results show one test very negative and another is positive - plan how you'd interact and communicate with product teams
  • Dig into those two tests and figure out what’s going on and make a recommendation
  • Analyze the results of that + the non-issue one and summarizing the findings.
  • Concurrent tests and how to approach them
  • Bring your own test results to analyze OR
  • Analyze two tests that interact, one has has an outlier.

Real-world projects

Work on projects that bring your learning to life.
Made to be directly applicable in your work.

Live access to experts

Sessions and Q&As with our expert instructors, along with real-world projects.

Network & community

Core reviews a study groups. Share experiences and learn alongside a global network of professionals.

Support & accountability

We have a system in place to make sure you complete the course, and to help nudge you along the way.

Get reimbursed by your company

More than half of learners get their Courses and Memberships reimbursed by their company.

Hundreds of companies have dedicated L&D and education budgets that have covered the costs.


Frequently Asked Questions

Still not sure?

Get in touch and we'll help you decide.

Keep in touch for updates, discounts, and new courses.

Questions? Ask us anything at

© 2021-2022 CoRise Education