A/B Testing
A/B testing (also called split testing or online controlled experimentation) is a method of comparing two or more variants of a product, feature, or message by randomly assigning users to each variant and measuring a predefined outcome. One group (control) sees the existing experience while another (treatment) sees the proposed change. Statistical inference is then applied to determine whether observed differences are due to the change or random chance.
In 2026, AI-driven product teams at companies like Google, Meta, Netflix, and Spotify run tens of thousands of experiments per year, and hiring managers expect candidates to design, ship, and interpret controlled experiments as a core competency. A/B testing is also central to responsible AI deployment: models and ranking systems are evaluated via experiments before full rollout, making experimentation literacy essential for ML engineers and data scientists alike. Without rigorous experimentation skills, teams risk shipping harmful or ineffective changes based on intuition alone.
🎓 Courses
A/B Testing by Google
by Google Data Team
The most widely recommended free course on A/B testing. Covers experiment design, metric selection, sample size calculation, and result analysis using real Google product examples. Data science communities consistently rank it as the top starting resource.
A/B Testing for Business Analysts
by Udacity Instructors
Three-module course focused on foundational experiment design and decision-making for analysts who need to translate test results into business recommendations without deep statistics backgrounds.
Launch Effective A/B Tests
by Coursera Instructors
Short, practical course aimed at data analysts learning to design valid experiments. Free to enroll and focused on turning business hypotheses into testable experiments.
Run Smart A/B Tests
by Coursera Instructors
Focuses on the execution phase of experimentation — avoiding common pitfalls like peeking, multiple comparisons, and sample pollution. Practical for marketing and product professionals.
📖 Books
Practical A/B Testing: Creating Experimentation-Driven Products
Leemay Nassery · 2023
Published by Pragmatic Programmers in 2023, this book by a former Spotify and Comcast experimentation engineering leader provides step-by-step guidance on building an experimentation culture and platform. Covers the full lifecycle from hypothesis to decision, with special attention to organizational adoption. The most recent practitioner-focused book on the topic.
Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
Ron Kohavi, Diane Tang, Ya Xu · 2020
The definitive reference on online experimentation, written by the architects of experimentation programs at Microsoft, Google, and LinkedIn. Covers metric design, statistical pitfalls, platform engineering, and culture. Essential reading before senior-level interviews at big tech companies; widely referenced in academic papers on the topic.
🛠️ Tutorials & Guides
Sequential A/B Testing Keeps the World Streaming — Netflix Part 1
Netflix engineers explain how sequential testing (always-valid p-values) lets them stop experiments early without inflating false-positive rates — a concrete industry application of a statistically rigorous technique that is increasingly standard in 2026.
Experiment Like Spotify: A/B Tests and Rollouts
Spotify's internal experimentation team (300+ teams, tens of thousands of experiments annually) explains how they separate feature rollouts from experiment analysis, a key operational distinction that prevents common contamination bugs.
How to Successfully Run A/B Tests
Practical walkthrough of the end-to-end A/B testing process — from defining a hypothesis and choosing metrics to interpreting results and avoiding pitfalls like peeking, Simpson's Paradox, and the multiple-comparisons problem. Good bridge between theory and production.
Learning resources last updated: June 18, 2026