Tag archive: experimentation
May 15, 2024
Fixed-Power Designs: It’s Not IF You Peek, It’s WHAT You Peek at
TL;DR Sometimes we cannot estimate the required sample size needed to power an experiment before starting it....
Mar 5, 2024
Risk-Aware Product Decisions in A/B Tests with Multiple Metrics
TL;DR We summarize the findings in our recent paper, , where we explain how Spotify’s decision-making engine...
Sep 28, 2023
How to Accurately Test Significance with Difference in Difference Models
When we want to determine the causal effect of a product or business change at Spotify, A/B testing is the...
Aug 24, 2023
Encouragement Designs and Instrumental Variables for A/B Testing
At Spotify, . Most of these tests follow a standard design, where we assign users randomly to control and...
Aug 16, 2023
Experimentation at Spotify: Three Lessons for Maximizing Impact in Innovation
As companies mature, it’s easy to believe that the core experience and most user needs have been resolved,...
Aug 3, 2023
Coming Soon: Confidence — An Experimentation Platform from Spotify
TL;DR: Spotify is releasing a new commercial product for software development teams: a version of our...
Jul 25, 2023
Bringing Sequential Testing to Experiments with Longitudinal Data (Part 2): Sequential Testing
In of this series, we introduced the within-unit peeking problem that we call the “peeking problem 2.0”. We...
Jul 18, 2023
Bringing Sequential Testing to Experiments with Longitudinal Data (Part 1): The Peeking Problem 2.0
At Spotify, we’re constantly improving our data infrastructure, which means we can get feedback on...
Jun 28, 2023
Experimenting with Machine Learning to Target In-App Messaging
At Spotify, we use messaging to communicate with our listeners all over the world. Our Messaging team powers...
Jun 15, 2023
Experimenting at Scale, the Spotify Home Way
In the fast-paced world of streaming, personalization plays a vital role in enhancing user experiences. At...
Mar 21, 2023
Choosing a Sequential Testing Framework — Comparisons and Discussions
TL;DR Sequential tests are the bread and butter for any company conducting online experiments. The literature...
Mar 23, 2022
Comparing Quantiles at Scale in Online A/B Testing
TL;DR: Using the properties of the Poisson bootstrap algorithm and quantile estimators, we have been able to...