When in Rome: How Spotify halved build times with just one script

How do you cut iOS app build times by 50%? Patrick Balestra, an engineer at Spotify, explains how his team helped improve developer productivity and the end user experience.

One of the great things about working at Spotify as an engineer, is the opportunity to improve the experience of our listeners. But it’s equally rewarding to create an environment where we can help our developers work faster and improve productivity. That’s where my team comes in. We build tools, systems, and libraries that enable them to move as quickly as possible.

A great example is our iOS app, Spotify for Artists, which provides musicians and bands with tools that help them understand their audience, promote music, and find new followers. As you can imagine, delivering an outstanding experience to this audience is critical to Spotify’s success.

Our engineers are constantly on the lookout for ways to enhance the app, and my team is equally vigilant when it comes to making their work simpler and faster. Here’s how we helped them cut build times by 50%.

Improving the productivity of Spotify engineers

Let’s rewind to the start of Summer 2018. If you’re familiar with the iOS ecosystem then you certainly know about Carthage, one of the most common dependency managers in this environment. We use it for several applications at Spotify including the aforementioned Spotify for Artists, and also Spotify Stations, our radio-like music curation app.

Using Spotify for Artists as an example, it has almost 20 internal and external dependencies. This is why we chose Carthage because it allows linking dependencies as binaries in order to reduce build times when working on the project.

This contrasts with other dependency managers such as CocoaPods and Swift Package Manager where you can only integrate dependencies from source (as they ship outside the box at least), which decreases developer productivity.

“60-70% of time is spent compiling the same dependencies over and over again”

Back to Carthage. It can download pre-built artifacts from GitHub or build the dependencies during the “bootstrap” phase of the project. This can take quite a long time, typically 15-20 minutes for the Spotify for Artists app, since every dependency has to be compiled from source into a binary.

Additionally, each developer on the project has to do this locally on their machine before contributing for the first time. Worst of all? You have to build all dependencies from source during continuous integration because build artifacts are cleaned before every new build.

Comparing the time spent building the Swift source code of the Spotify for Artists app with its dependencies in our continuous integration environment, we quickly realized that around 60-70% of the time was spent compiling the same dependencies over and over again before we could actually build our own code.

We saw something similar with the Spotify Stations app. When a new pull request is opened, the feedback loop was about 15 minutes. As you can see from the following build log, almost 12 minutes were spent bootstrapping the project dependencies compared to four minutes of build and test for the app’s source code.

Speeding up builds with Rome

The solution? One popular approach is to cache artifacts and reuse them across different machines and builds. A good example of this in action is Rome, an open-source tool written in Haskell that caches artifacts built by Carthage in a local folder and/or to S3.

It’s pretty simple. Rome takes every dependency in the Carthage/Build directory and uploads its .framework, .dSYM, .version and .bcsymbolmap to the specified local and/or remote cache. It supports multiple caching layers too. For example, we can specify both a local and remote cache which would prevent hitting the remote cache if the artifact had already been downloaded. The Carthage/Build directory is usually gitignored and its content needs to be rebuilt based on the submodules in Carthage/Checkouts.

Rome speeds up future builds by removing the need to rebuild the same dependencies. This is achieved by caching the contents of the Carthage/Build directory to an external local folder (e.g: ~/Library/Caches) or a remote server (e.g: Amazon S3).

Unfortunately Rome wasn’t an option as it only supports Amazon S3 and at Spotify we use Artifactory to store artifacts. So in August 2018, we raised an issue to the Rome project on GitHub asking for support that would enable us to use any remote cache. Here, the idea was to expand Rome to support executing an external script to upload and download artifacts. With such an architecture, Rome ignores the implementation details of the caching layer, and instead simply delegates the upload and download operations to an external script. The interaction between Rome and the script involves passing the required parameters such as local paths (where to download the artifacts) and the remote paths (where to upload them).

Finding the solution ourselves

Having raised the issue it was then a matter of waiting for the project author to implement our recommendation. A couple of months passed without further news, and in March 2019, following a conversation with our iOS colleague Daniel Fleming, we accepted the challenge and set out to implement the feature ourselves.

Since I had some previous experience with the Haskell programming language, I was happy to give it a try. The project author, Tommaso Piazza, was kind enough to follow our efforts as we developed the feature, and advise us during the project.

Starting in Rome 0.22.0.59, it’s now possible to provide a custom script in your preferred language be it Bash, Python, or Ruby that Rome executes to upload and download Carthage artifacts. This enabled us to start caching Carthage dependencies in Artifactory.

CI configuration time cut by 50%

Following the release of Rome that included our contribution, we immediately integrated it in Spotify for Artists. We wrote a 50-line Bash script that Rome invokes and which implements the three required operations:

• download: performs a GET request to Artifactory with the path received by Rome.

• upload: uses a command-line tool to upload the artifact at the given path to Artifactory (currently only works inside our CI agents).

• list: performs a HEAD request with the path received by Rome in order to check if an artifact exists.

At any time when we need to set up a project for building, we first check to see if cached artifacts exist. Both build agents and developers working locally can download the available artifacts to speed up the project bootstrap. If a dependency is not present, Carthage will build it. Once Carthage finishes running, we use the list command to check which newly built artifacts we need to upload to Artifactory. This is very powerful and saves a lot of time because we can build once and reuse artifacts everywhere, even across different projects that share the same dependency.

After merging support for Rome, it didn’t take long to notice an improvement. We saw an immediate reduction of around 50% in the time taken in our CI configurations that build the Spotify for Artists app: from around 20 minutes to under 10 minutes!

Job done. So what’s next?

This project was a lot of fun to work on and spanned many areas: from contributing to an unknown open-source codebase to figuring out its integration with our infrastructure.

One issue that we discovered is that developers are unable to debug locally with binaries downloaded from Artifactory. The compiler currently includes absolute paths to the symbols in the binary, which causes LLDB to throw errors when trying to debug it in a different machine. Hopefully, future versions of Xcode and the Swift compiler will support remapping of the symbols in the binary and improve the experience.

In the future, the engineers here at Spotify would like to add support for caching Carthage dependencies to more projects that currently suffer from long compilation times.

Before we finish, I’d also like to thank my Spotify colleagues Daniel Fleming and Aron Cedercrantz for their help during this experiment.

About the Writer

Patrick Balestra, iOS Infrastructure Engineer, Spotify

Hi Patrick, tell us a bit more about your role at Spotify
I work as an iOS infrastructure engineer in the Foundation squad. My team focuses on improving the developer experience of all our iOS engineers. We do this by building tools, systems and libraries that help our engineers move as fast as possible.

What do you enjoy most about the role?
One of the great things about working at Spotify is that I have the opportunity to learn a lot of new technologies that span across different domains including build systems, compilers, and even frontend and backend services.

How did you get to be working at Spotify?
I’m based in Stockholm, Sweden, but I was born in Locarno and grew up in Bellinzona, the capital of the Italian-speaking region Ticino in Switzerland. From there I studied Computer Science at the University of Lugano, Switzerland.

What do you enjoy outside of work?
I love sports and I’ve played floorball (a type of indoor floor hockey) for over 10 years. I’ve covered almost all roles as a player, coach and even a referee.

What’s on your playlist?
It depends! My favorite musician is the Dutch DJ Martin Garrix, I listen to a lot of EDM music and I try to attend as many music festivals as I can. But my playlist this week includes Summer Days by Martin Garrix and Macklemore, and Takeaway by The Chainsmokers.

Tags: backend