What the Heck is Backstage Anyway?
Yesterday, we released the open source version of Backstage, our homegrown developer portal. And we learned a thing or two via the feedback we received. So, I wanted to take this opportunity to further explain what we’re trying to do with Backstage — and more importantly, what we want to give to the greater engineering community beyond Spotify.
What’s the big infrastructure problem?
As companies grow, their infrastructure systems get messier. Consider a team that wants to deploy something to the cloud. While Spotify has many awesome engineers, not every engineer is well-versed in our chosen cloud-provider tooling. Yet everyone is required to know and understand Terraform, GCP/AWS/Azure CLIs, Gitlab CI, Prometheus, Kubernetes, Docker, various monitoring and alerting tools, and much, much more. Once other resources come into play (databases, queueing, etc.), each engineer requires even more tools and domain-specific knowledge (or “disciplines”), from backend to machine learning, to mobile and data.
What’s the fix?
Backstage unifies all your infrastructure tooling, services, and documentation with a single, consistent UI. All of it! Imagine if all your tools — GCP, Bigtable, CI pipelines, TensorFlow Extended, and whatever else is hiding in your stack — all had the same, easy-to-use interface. That’s Backstage. One frontend for all your infrastructure.
Backstage gives developers a uniform overview of all their resources, regardless of how and where they are running, as well as an easy way to onboard and start using those tools. It also allows the creation of new resources, such as backend services running in Kubernetes, with a few clicks of a button — all without having to leave the same, familiar interface of Backstage.
Why did we build it?
To some observers, it may seem odd that a music company is launching a best-in-class developer portal. But if you dig deeper, you’ll find that since the very beginning, Spotify has been known for its agile, autonomous engineering culture. More than music, we’re a tech company that has always put engineers first, empowering our developers with the ability to innovate quickly and at scale. Backstage is the natural result of that focus.
What are examples of how Backstage is used at Spotify?
Our internal installation of Backstage has over 100 different integrations — we call them “plugins”. Since the open-source version currently does not have any end-to-end use cases, it can be challenging to understand what problems Backstage can solve for you. To make things more tangible, let’s have a look at four of the common use-cases:
- Creating a new microservice
- Following a pull request from review to production
- Centralized technical documentation
- Review performance of your team’s mobile features
1. Creating a new microservice
Creating any new software component at Spotify, such as a new microservice, is done with a few clicks in Backstage. Developers choose between a number of standard templates — all with best-practices built in.
After inputting some metadata about your service, a new repository is created with a “hello world” service that automatically builds and deploys in production on Kubernetes (GKE). Ownership information is automatically captured in our service/software catalog and users can see a list of all the services they own.
2. Following a pull request from review to production
As soon as you submit a pull request to Spotify’s GitHub Enterprise, our CI system automatically posts a link to the CI/CD view in Backstage. The view provides you with all the information you need: build progress, test coverage changes, a re-trigger button, etc., so that you don’t have to look for this information across different systems.
Our homegrown CI system uses Jenkins under the hood, but Spotify engineers don’t need to know that. They interact directly with GitHub Enterprise and Backstage.
3. Centralized technical documentation
Spotify uses a docs-like-code approach. Engineers write technical documentation in Markdown files that live together with the code. During CI, a beautiful-looking documentation site is created using MkDocs, and all sites are rendered centrally in a Backstage plugin.
On top of the static documentation we also incorporate additional metadata about the documentation site — such as owner, open issue and related Stack Overflow tags.
4. Review performance of your team’s mobile features
Our mobile apps are developed by many different teams. The codebase is divided up into different features, each owned and maintained by a separate team. If an app developer on one team wants to understand how their feature is affecting overall app performance, there’s a plugin for that:
Developers can also look at crashes, releases, test coverage over time and many more tools in the same location.
Why did we make Backstage open source?
When discussing infrastructure challenges with peer companies, it’s clear that we are not alone in struggling with fragmentation across our developer ecosystem. As companies adopt more open-source tooling, and build more infrastructure internally, the complexity grows. It gets harder for individual engineers to find and use all these distinct tools.
Similar to how Backstage ties together all of Spotify’s infrastructure, our ambition is to make the open-source version of Backstage the standard UX layer across the broader infrastructure landscape. We decided to release Backstage early so we could collaborate more closely with companies that have a similar problem — and that want to provide a better developer experience to their teams.
What’s next?
We created Backstage about four years ago. While our version of Backstage has had the benefit of time to mature and evolve, the first iteration of our open-source version is still nascent. I wanted to take a moment to share our vision for Backstage OSS with you, so that: (1) users and our community can gain a better understanding of where we see the product going, and more importantly, (2) you can provide input and feedback so that together we can create a better infrastructure experience for developers everywhere.
We are envisioning three phases of the project (so far), and we have already begun work on various aspects of these phases:
- Phase 1: Extensible frontend platform (now) — You will be able to easily create a single consistent UI layer for your internal infrastructure and tools. A set of reusable UX patterns and components help ensure a consistent experience between tools.
- Phase 2: Manage your software inventory (next 2–3 months) — A central software inventory with clear ownership and the ability to easily create and manage software at scale. Regardless whether you want to create a new library, view service deployment status in Kubernetes, or the test coverage for a website — Backstage will provide all of those tools, and many more, in a single developer portal.
- Phase 3: Ecosystem (later) — Everyone’s infrastructure stack is different. By fostering a vibrant community of contributors we hope to provide an ecosystem of open-source plugins/integrations that allows you to pick the tools that match your stack.
Our vision is for Backstage to become the trusted, standard toolbox (read: UX layer) for the open-source infrastructure landscape. We realize this is an ambitious goal. We can’t do it alone. If this sounds interesting or you’d like to help us shape our product vision, we’d love to talk. You can email us at: Backstage-interest@spotify.com.
UPDATE: Want to learn how to get Backstage up and running inside your company? We have a new website just for adopters: backstage.spotify.com.
Tags: backend, engineering leadership
