Experiment Design
Experiment design is the systematic process of planning studies or tests so that the data collected can answer specific causal questions reliably. It covers choosing what to measure, how to assign units to conditions, how to control for confounding variables, and how much data is needed to detect an effect. Core methods include randomized controlled trials (RCTs), factorial designs, A/B tests, blocked designs, and adaptive or sequential experiments.
In 2026, AI companies rely on controlled experiments to validate model changes, product features, and algorithmic decisions before full deployment — running thousands of experiments per year at scale as standard practice. Hiring for experiment design reflects a shift toward rigorous causal reasoning over purely correlational analysis, driven by the need to make defensible product decisions. Teams that design experiments well can detect meaningful signals with fewer users and shorter runtimes, compressing iteration cycles.
🎓 Courses
A/B Testing by Google
by Google / Udacity
Free course co-developed with Google that covers the full experiment design cycle: metric selection, statistical power, sample size calculation, and result analysis. Widely cited as the go-to practical introduction to online controlled experiments.
Experimental Design and Causal Inference in R
by Pluralsight
Covers randomized controlled trials, A/B testing, difference-in-differences, propensity score matching, and instrumental variables — a solid end-to-end course for practitioners who need both design and analysis skills.
Structuring Machine Learning Projects
by Andrew Ng
Teaches how to diagnose and iterate on ML experiments — error analysis, train/dev/test splits, orthogonalization of objectives — making it directly relevant to anyone running model-improvement experiments.
STAT 503: Design of Experiments (Penn State Online)
by Penn State Statistics Department
Free, rigorous university-level course covering factorial designs, blocking, response surface methods, and ANOVA — grounded in Montgomery's standard textbook. Ideal for building a solid theoretical foundation.
📖 Books
Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
Ron Kohavi, Diane Tang, Ya Xu · 2020
Written by experimentation leaders at Microsoft, Google, and LinkedIn, this is the definitive industry reference for running online experiments at scale. Covers metrics, pitfalls, statistical issues, platform architecture, and the cultural shift to data-driven decisions. Widely required reading at major tech companies.
Design and Analysis of Experiments (Springer Texts in Statistics, 2nd ed.)
Angela Dean, Daniel Voss, Danel Draguljic · 2023
Comprehensive graduate-level textbook covering factorial experiments, randomized blocks, response surface methods, and screening designs. The updated edition adds computer experiments and mixed models, making it relevant to modern ML workflows.
🛠️ Tutorials & Guides
Causal Inference in Product Experimentation
Industry-facing tutorial explaining how causal inference techniques extend beyond basic A/B testing for product decisions. Accessible and grounded in real platform examples — good for practitioners who want to connect statistical theory to day-to-day experimentation work.
CausalML – Analysing AB Test Results
Hands-on walkthrough using the CausalML Python package to analyze A/B test data with propensity matching and conditional treatment effects. Shows how to handle biased observational data when perfect randomization is unavailable.
Penn State STAT 503 – Lesson 1: Introduction to Design of Experiments
Free, well-structured lesson that defines the key vocabulary of DOE (treatment, factor, response, randomization, replication, blocking) and explains why each concept matters. Ideal starting point before tackling more complex designs.
Learning resources last updated: June 18, 2026