Solving documentation for monoliths and monorepos

At Spotify, we’re always trying to push the boundaries of the way the world experiences audio. In order to achieve this, our engineers need to learn and move quickly. It is the best way we believe we can achieve the Spotify company mission of unlocking the potential of human creativity.

One of the ways we can speed up is through documentation. It is how our engineers navigate effectively through Spotify’s vast and complex array of software every single day. We have seen that overlooking and underserving this area has an increasingly negative effect on our developer’s productivity and will continue to grow if it isn’t addressed.

We set out to create a clearer and more accountable process for writing documentation for code maintainers along with a consumption method that is more unified and digestible for Spotify engineers. The way we believe we can achieve this is by adopting the Docs Like Code approach where we optimize for bringing docs as close to the associated code as possible.

This two-sided problem was something others agreed existed. Every year, Spotify hosts a hack week where Spotifiers can tackle issues they are passionate about, and don’t normally get the chance to work on in their day-to-day work. In 2018’s Hack Week, as there were no clear solutions that addressed the issues we had at our scale, several engineers and tech writers gathered to prototype a “hack” to address this issue. After it was known that discovering technical information was the 3rd biggest blocker to Spotify’s engineering productivity, a cross-functional team (consisting of tech writers and engineers) was formed with two goals: to bring Spotify’s documentation culture on par with the industry and make it best-in-class.

Monoliths and monorepos

Although Spotify is heavily built on the micro-service architecture model, we continue to operate numerous large scale codebases. These often contain some of Spotify’s most important engineering properties, such as our Desktop app, Web Player, iOS, and Android apps. In the web community, there is also a growing use of monorepos for better dependency management with tools such as Yarn Workspaces.

The current documentation convention for codebases is to use README.md across the codebase, or have a single root docs/ folder. This works well for both open source as well as smaller internal codebases because these tend to have shared collective ownership. Larger internal codebases often have complex and distributed ownership, often with very different objectives and the current convention doesn’t serve them well. So we need to evolve the current convention in order to serve documentation needs for some of Spotify’s larger and more business-critical codebases.

Solving it with our customers

As mentioned earlier, only a few squads operate large codebases at Spotify and it was important we addressed their problems. Fortunately working on the Infrastructure team allows us to have easy access to our customers, as they are also our colleagues. One of our core beliefs in Infrastructure is to build infrastructure together. We built, iterated, validated, and shipped our solution in one week, focusing specifically on our users’ pain points, which are at the heart of the process.

We use Mkdocs to power all of our internal technical documentation, and given how much of a need it presented internally we were curious to see if other companies had already made documentation easier in large codebases. We were surprised to learn that this wasn’t the case. Despite this, we saw a consistently strong need to address this problem, and after going back and forth on how to tackle this problem we ended up introducing a new syntax for our engineering squads.

Building monorepo support for Mkdocs

One week later, we finalized a Mkdocs plugin which adds monorepo support. This introduces a new !include syntax for including subfolder mkdocs.yml configuration inside mkdocs.yml:

site_name: "Web Player Documentation"

nav:
  - Home: index.md
  - Features:
    - Home Screen: "!include src/features/homeScreen/mkdocs.yml"
    - Search: "!include src/features/search/mkdocs.yml"
    - Now Playing Screen: "!include src/features/nowPlayingView/mkdocs.yml"

plugins:
  - monorepo

This new syntax lets you include additional mkdocs.yml files. The plugin will handle merging each of their navigation trees into a single “master” navigation tree. You can then generate a single documentation site as you normally would (an example is above). We decided on this approach as it gives us a few clear advantages:

• Declarative approach to navigation. Each squad will have their own mkdocs.yml. This allows them to structure their documentation in a monorepo however they see fit and it will be centralized into a single documentation site.
• Leverages GitHub Codeowners for configuring multiple owners. Numerous squads at Spotify use GitHub Codeowners to assign “ownership” to different squads on a per-directory or per-file basis. We wanted to tap into this existing GitHub feature.
• Closer to the code. This is based on our belief that the respective documentation should be as close to the code as possible, but this now introduces support on a folder-level rather than previously on a repository-level.
• Running Mkdocs inside a folder. We wanted to allow squads to be able to invoke Mkdocs on their own part of the documentation (on a folder level), rather than the entire documentation (on a repository level) to ensure a better developer experience.

It was important that we built this infrastructure together with the people who will be using it in their day-to-day workflow, as it allows us to provide the highest level of productivity. It also helps create passionate advocates for the change or improvements we’re working towards.

Hello, open source!

We have been trialling our Mkdocs Monorepo plugin for several weeks with internal squads and we’re happy to share this in beta with the greater open source community. You can check out the repository on GitHub here, or install the plugin to get started:

$ pip install mkdocs-monorepo-plugin

And then simply add the monorepo plugin to your mkdocs.yml configuration:

plugins:
  - monorepo

nav:
  - v1 API: "!include src/versions/v1/mkdocs.yml"
  - v2 API: "!include src/versions/v2/mkdocs.yml"

Happy documenting!

Tags: backend