May 30, 2019
Scio 0.7: a deep dive
Introduction Large-scale data processing is a critical component of Spotify’s business model. It drives music recommendations, artist payouts based on [...]
Tag Archive: big data
-
-
November 20, 2017Autoscaling Pub/Sub Consumers
Spotify’s Event Delivery system is responsible for delivering hundreds of billions of events every day. Most of the events are generated as a response [...]
-
October 23, 2017Big Data Processing at Spotify: The Road to Scio (Part 2)
In this part we’ll take a closer look at Scio, including basic concepts, its unique features, and concrete use cases here at Spotify [...]
-
October 17, 2017TC4D: Data Quality By Engineers, For Engineers
Changing an engineering culture is one of the biggest challenges for any organization. It requires challenging an existing way of working, and introducing compelling improvements [...]
-
October 16, 2017Big Data Processing at Spotify: The Road to Scio (Part 1)
This is the first part of a 2 part blog series. In this series we will talk about Scio, a Scala API for Apache Beam and Google Cloud Dataflow, and [...]
-
August 7, 2016Commoditizing Music Machine Learning : Services
Five years ago, music personalization at Spotify was a tiny team. The team read papers, developed models, wrote data pipelines [...]
-
January 9, 2015Personalization at Spotify using Cassandra
At Spotify we have have over 60 million active users who have access to a vast music catalog of over 30 million [...]
-
January 5, 2015How Spotify Scales Apache Storm
Spotify has built several real-time pipelines using Apache Storm for use cases like ad targeting, music recommendation, and data visualization. Each of these [...]
-
November 27, 2014Data Processing with Apache Crunch at Spotify
All of our lovely Spotify users generate many terabytes of data every day. All the songs that are listened to, [...]
