Reflections on the Last Two Years of Spotify’s Bug Bounty Program

We recently surpassed the two year anniversary of our bug bounty program on the HackerOne platform. This gave us pause to take a look back at our successes and learnings in engaging with the security community to help improve security at Spotify. 

What is a bug bounty program?

The Internet was conceived as an open and accessible place for the exchange of ideas and discussion with people from all over the world. It accomplished this by connecting computers and networks at an unprecedented scale. Along with this openness and accessibility came opportunity for those seeking to do harm. Malicious attackers spend countless hours scouring the Internet, looking for and exploiting weaknesses in web sites and applications. You may have heard of huge data breaches that resulted from malicious parties discovering vulnerabilities and exploiting them to their own gain.

Fortunately, there are ethical and responsible security researchers that use similar tactics and tools to discover weaknesses. Unlike the malicious attackers, they report these weaknesses to the owners of the sites so they can be fixed before others can use them for evil. Bug bounty programs exist to make it easier for security researchers to report these weaknesses to site owners. As a token of gratitude, the site owners can reward money or swag to the researchers for the efforts.

Spotify’s Bug Bounty Program

Spotify’s Security team launched its bug bounty program in 2015. We had a few motivations for starting a bug bounty program. In 2015 we were a very small team that occasionally received vulnerability reports from researchers responsibly disclosing bugs, many of which were not expecting anything in return. Although we didn’t receive a huge number of reports, it was clear that managing them by hand, primarily through email, wouldn’t scale. We also wanted to reward cash for bug submissions. We had been rewarding reporters with any swag we happened to have on hand, or giving them credit on our wall of fame at https://www.spotify.com/us/bounty/.

In May 2017 we moved our bug bounty program onto HackerOne to take advantage of their platform and managed services. Today, a typical bug bounty report works like this:

1. A security researchers submit a report to us on our page at https://hackerone.com/spotify. 
2. The HackerOne triage team reviews the reports for scope, validity, and severity.
3. If the report is valid, they forward them to the Spotify Security team.
4. We identify and work with the development team for a resolution.
5. Once a fix has been deployed, we reward the security researcher with a bug bounty reward commensurate with the severity of the report through the HackerOne platform.
6. This is also reflected on the security researcher HackerOne profile, such as https://hackerone.com/zulln.

HackerOne also serves as a security expert and if needed, a mediator between the reporter and Spotify for any complication regarding severity or validity. HackerOne utilizes the Common Vulnerability Scoring System (CVSS) — an industry standard calculator used to determine the severity of a bug. CVSS enables a common language around the severity of bugs. Researchers can suggest a severity rating with their report. Whether they choose to do that or not, the HackerOne triage team works with the researcher to determine an accurate severity rating for the Spotify program. You can read more about it here.

Security Wins

Since we started using the HackerOne platform and managed services, we’ve received over 365 valid and actionable reports and rewarded over $120,000 to security researchers for their efforts.

We receive the largest amount of reports on our most visible web assets, such as www.spotify.com and community.spotify.com, but receive reports on a broad range of assets, including our mobile applications, desktop applications, and API’s and SDKs.

A Few Bullets Dodged

One memorable event was a series of information disclosure vulnerabilities for one of our highly visible websites. Spotify employees credentials, including several high-profile employees were insufficiently protected. Fortunately these credentials were not used for other systems. Database credentials, API keys, source code, and other sensitive information could be trivially downloaded.

We kicked off a security investigation, involving our security, legal, and data privacy teams. After a thorough investigation, we were confident that this vulnerability had not been previously exploited, and we could let out a collective sigh of relief. Had these vulnerabilities been discovered by a malicious actor, the website could have been defaced, harming the brand and reputation of Spotify, or the credentials could have been used for lateral movement or in a phishing attack. We could rest assured knowing that $3,000 in bounties and our teams effort were well spent knowing what potential harm was avoided.

At Spotify our engineering team has a high level of interest in not only producing functional code, but also secure and reliable code. We also consider Security and Reliability “Priority 0” work. As such, it usually doesn’t take much effort or time to get security reports fixed — our average time to resolution is 24 days since joining HackerOne.

So What Have We Learned?

Golden Paths for the Win

Spotify has thousands of engineers, so we are very intentional about strategy to allow such a large organization to do great things. Let me provide some context about our strategy to set the stage for the learnings and insights we’ve gained from our bug bounty program.

At Spotify, one of our engineering strategies is the creation and promotion of the use of “Golden Paths.” Golden Paths are a blessed way to build products at Spotify. They consist of a set of APIs, application frameworks, best practices, and runtime environments that allow Spotify engineers to develop and deploy code safely, securely, and at scale. We complement these with opt-in programs that help increase quality. From our bug bounty program reports, we’ve found that the more that development adheres to a Golden Path, the less likely there is to be a vulnerability reported to us.

While this might not sound surprising, the data from our bug bounty program shows us that it’s possible to maintain autonomy and decentralization in development teams while ensuring high quality. Even if you’ve taken a different approach to engineering at your organization, you may want to think about what tools you have to support your program before opening your assets to submissions.

Global Preferred Production Partner Program

One area where we face challenges is with third party development. While we have inertia and battle-tested understanding of “known good” with our Golden Paths for Spotify developers developing code in the Spotify ecosystem, there’s much less inertia for development at large. There are references such as the OWASP Top 10 that give web developers things to watch out for when developing web sites, but creating a “one size fits all” methodology for secure development is something that has been proven to be difficult, if not impossible, for years.

So it’s not surprising that the majority of the reports we get are for sites that Spotify has contracted to have built, or companies that Spotify has acquired that didn’t have the benefit of being developed on Golden Paths.

To help our third party developers, building on the successes of the Spotify Golden Paths, we’re developing something we call the Global Preferred Production Partner Program. Similar to the Spotify Golden Paths, this is a security-focused set of standards and runtime environments. The difference is that this is designed for Partner Developers outside of Spotify. It also includes a set of expectations for vendors that help us ensure we can rapidly and effectively respond and correct vulnerabilities that are reported to us through the bug bounty program.

So weigh the pros and cons of including third-party developed assets as part of your bug bounty program. On one hand, addressing reports can be time-consuming, complex, and require help from your engineering teams, who might find it difficult to support sites they didn’t develop. On the other hand, do not underestimate the risk exposure from third-party developed assets, especially if you share sensitive or personal data with them, or they have privileged access to your data or infrastructure.

Closing

The Security team at Spotify has been pleased and astounded by how well the bug bounty program has been received. It has raised security awareness within our engineering organization, exposed weaknesses in our security posture, and helped us better understand our attack surface. We think there’s still opportunities to make it better and make Spotify even safer for its users and partners.

We want to build a stronger relationship with the research community, showing that we not only care deeply about security, but also want to foster an ongoing relationship and dialogue with those that spend significant amounts of time to make Spotify safer. To learn how we can help hackers find vulnerabilities before bad actors do, we’ve run surveys with our most frequent reporters to see what motivates and attracts them with our program, and a series of promotions to draw hackers to less-reported areas of our product.

We hope this has been an interesting read that brings some visibility into one way we’ve tackled security at Spotify. Even if you have no experience in bug hunting, check out our program page at https://hackerone.com/spotify. There’s less and less barrier of entry to finding security vulnerabilities. We don’t care about your qualifications, background, or experience. We accept reports from anyone in pursuit of making Spotify and the Internet at large safer. Thank you to the over 200 researchers who have contributed to our program over the last 2 years. We are more secure because of you.

Tags: engineering leadership