Spotify Engineers: LLM A/B Tests Recover Only 39% of Human Treatment Effects

Key Takeaways

• Spotify Engineering tested LLM-based A/B testing on the Upworthy dataset, finding raw predictions recover only 39% of human treatment effects.
• The bias is systematic, and calibration works only under unverifiable assumptions.

What Happened

Spotify Engineering has published a rigorous analysis of a question many product teams are asking: can LLMs replace human participants in A/B tests? The promise is seductive — run experiments on a model instead of users, get results in hours instead of weeks, and skip traffic allocation entirely. But Spotify's data scientists found that the reality is far more complicated.

The team formalized the statistical conditions under which LLM-based A/B testing could work, using surrogate endpoint theory from biostatistics. Their framework is elegant: if a surrogate outcome (here, an LLM prediction) captures everything about a treatment that matters for the outcome, then experimenting on the surrogate gives the correct result. That's how biomarkers work in clinical trials — fast, cheap stand-ins for clinical outcomes.

The Core Finding

The empirical results are sobering. Using the Upworthy Research Archive — the largest open-access dataset on A/B tests, containing click-through rates for news headline variants across thousands of experiments — the team prompted gpt-4o-mini to predict click-through rates for each headline variant.

The results:

• Raw predictions recovered only 39% of the observed human treatment effect.
• If you treated LLM predictions as human data, you'd conclude treatments are less than half as effective as they actually are.
• The bias is not random noise — it's systematic and directional.

LLM outcomes attenuate treatment effects toward zero. This means treatments consistently look less effective than they are. For organizations running many LLM-based experiments across features and product areas, this attenuation would cause systematic underestimation of what users value.

The Calibration Caveat

Spotify's analysis does show that calibrating LLM outputs against human data can recover the treatment effect — but only using a specific set of methods. This is the critical nuance. The conditions that make calibration work cannot be verified for new treatments, and they become less plausible the further a new treatment is from past experiments.

As the team puts it: "The promise is least justified precisely when it offers the most benefit." The scenarios where LLM-based testing would be most valuable — testing novel, unprecedented treatments without historical data — are exactly the scenarios where the method is least reliable.

Why This Matters for Retail & Luxury

This research has direct implications for any organization considering LLM-based evaluation of creative content. In retail, the most obvious applications are:

• Headline and copy testing for email campaigns, product pages, and marketing assets
• Product description evaluation — which framing drives engagement?
• Creative direction testing — which visual or messaging approach will resonate?

A/B testing is how retailers learn what customers value. If LLM-based experiments systematically underestimate treatment effects, decisions made on those results will be systematically wrong — just in a subtler way than a complete failure. The risk is that teams running many small LLM-based tests will make consistently conservative choices, missing genuinely impactful creative directions.

For luxury brands, where creative differentiation is the core of the value proposition, this is particularly concerning. A system that attenuates treatment effects toward zero will favor safe, incremental changes over bold creative moves — the exact opposite of what luxury marketing requires.

Implementation Approach

For teams considering LLM-based A/B testing, the research suggests a cautious path forward:

1. Use LLMs for screening, not decisions. LLM predictions can help prioritize which variants are worth testing with real users, but final decisions should be based on human data.
2. Calibrate against your own historical data. The Upworthy results show calibration can work — but the calibration must be specific to your domain, content type, and audience.
3. Validate on known treatments first. Before trusting LLM predictions for new experiments, verify the method recovers known treatment effects from your own past experiments.
4. Beware the attenuation bias. Even calibrated models may systematically underestimate effect sizes. Build this into decision-making.

Governance & Risk Assessment

Maturity level: Research stage. This is a single (though rigorous) analysis on a specific dataset with a specific model. The results may not generalize across domains, models, or outcome types.

Key risks:

• Systematic attenuation bias leading to conservative product decisions
• Unverifiable assumptions for novel treatments
• False confidence from apparently successful calibration

Recommendation: Treat LLM-based A/B testing as a complement to, not a replacement for, human experiments for the foreseeable future.

Source: engineering.atspotify.com