Managing Clouds from the Ground Up: Cost Engineering at Spotify

Like many of those in tech, we invest heavily in our cloud and data infrastructure. While seemingly routine, the ability to manage and scale our infrastructure to support our 299+ million listeners worldwide, 24/7, without missing a beat (or syllable) is crucial for the business and our brand. 

On top of that, our infrastructure teams are resolute when it comes to upholding a highly valued cultural goal: enabling our autonomous engineering teams (called squads) to work as freely and quickly as they possibly can. Finish that off with the fact that we’re a growing public company, and we’ve created a challenging problem for our cost engineering team.

Introducing Spotify’s Cost Insights tool

Managing costs in our unique situation is no easy feat, but that certainly doesn’t stop us from innovating on the process. We’re leaving behind the days of reducing costs via top-down requests and moving on to finding fun and rewarding ways engineers can strengthen technology while improving the company’s bottom line. Our new Cost Insights product in Backstage.io explains cloud costs in a way our engineers can relate to and identifies optimizations that have resulted in some big wins for Spotify.  

We’ve found that our engineers see these optimizations as an interesting challenge — they improve cost, performance, and reliability, turning our infrastructure into a lean, green execution machine. The Cost Insights tool brings those optimization opportunities to light, so engineers can move quickly in achieving those wins. 

Spotify’s unique take on cost management

Many great cloud cost management tools are on the market, but we were looking for a way to better understand the relationship between a business’s cloud cost and its overall growth. Without broader context, it’s difficult to determine whether $10,000, for example, might be an appropriate amount to spend. 

At Spotify, we believe showcasing cost and business data in a meaningful way will empower our engineers to understand where they are spending and optimize quickly. To encourage engineers to take action, their cloud cost tool should be located where other frequently used products and services are to increase productivity.

Growing the business while being cost conscious

Spotify continues to be heavily focused on growth, allowing teams that may be high spending on cloud costs to continue to do so if it results in growth opportunities for the business. We needed a tool that showcases costs and helps engineers, engineering managers, and product managers to be information driven when deciding between growth initiatives and worthwhile cost optimizations. Cost Insights is a solution that allows for engineering teams to:

• Become aware of their cloud spend and how it relates to their business unit’s growth.
• Understand how cost optimizations should be prioritized compared to the goals for business growth.
• Receive clear recommendations on how they can optimize or reduce their spend.

Cost Insights features

Our goal was to launch the Cost Insights plugin with a strong foundation and a great potential for growth. The open source version includes three key features:

• Cost vs. business graph: Users track how their team or a specific GCP project is trending compared to their company’s business growth.  
• Detailed product panels: Cost Insights currently includes detailed information on six cloud products. The product panels currently cater to GCP but can be configured to utilize other cloud providers. These panels help users understand the cost of their products down to the resource level and compare their growth over time.
• Project alerting: Teams are alerted when project costs exceed a chosen threshold, allowing them to deep dive into cost changes at the resource level.

Preventing over-optimizations

Developing a cost tool in a growth-focused, autonomous culture can have severe consequences if executed incorrectly. The Cost Insights product is accessible by any Spotifier, so it became essential to be explicit with teams when there is a cost increase to review. We’ve set several thresholds to capture how much a team is spending and to track their growth trends. Only teams that are growing faster than Spotify’s business will be nudged to investigate their spending. 

Upcoming on the roadmap

Cost Insights allows teams to determine for themselves if the time invested in an optimization is valuable compared to the costs saved. Now that it’s available in open source with Backstage, our focus will shift to open sourcing the backend, creating detailed cost breakdowns (SKU level), and delivering alert dismissals that incorporate user feedback.

For more information about Cost Insights, reach out to Janisa Anandamohan at janisa@spotify.com.

Tags: engineering leadership