Stepping Up the Cloud Security Game

TL;DR: securing our Cloud infrastructure is incredibly important. We are now taking another step forward by leveraging open source tools we developed in partnership with Google.

Spotify engineering teams are fully embracing the devops culture: to increase development speed every dev team is responsible for their operational pipelines. From a security perspective we are continuously striving to bake security into those pipelines by leveraging security automation and self serve security tooling. The industry lingo for this is “SecDevOps”.

The friction to provisioning machines and experiment has been significantly reduced since we made our shift to a cloud infrastructure on the Google Cloud Platform. Dev teams are now able to freely add pieces to our infrastructure seamlessly, but it also introduced complexity from a security perspective since it also made it harder to scale our hardening efforts at the same pace.

For this reason we started building our own tools to audit GCP: the goal was to quickly detect and fix security misconfigurations as they arise.

Running these specialized tools has proven to be a huge step forward, but it also highlighted some non obvious problems on how to properly audit and address issues on a growing infrastructure. It was clear we still lacked precise visibility over our resources and therefore it was very hard to keep an audit trail of changes and additions. We couldn’t look back at what the state of the infrastructure was at a certain point in time, and thus we couldn’t properly investigate in case of incidents.

While detecting misconfigurations was a solved problem, we also discovered that identifying ownership and addressing misconfigured resources was adding too much to the workload for the security team.

Enter Forseti

To address these gaps we asked Google to join forces to step up in our cloud security game. The result of our efforts is Forseti, an open source security-swiss-army-knife for Google Cloud.

Forseti is comprised of different components with specialized use cases. As a first step, the tool takes a snapshot of the current infrastructure by building an inventory of every active resource. Secondly it runs different scanners on the snapshot to highlight security misconfigurations. Every violation discovered will raise an alert.

Finally we can also automatically enforce policies/configurations as soon as an issue is detected. For example in a project IAM policy or in a project firewall configuration any violation will be fixed.

While developing Forseti we deployed it from the start. This helped us refine the scanning rules and identify which resources we needed to inventorize the most.

We’ve now reached a first point of maturity and at the time of writing we are successfully auditing every day:

• ~ 1300 Projects
• ~ 5000 GCS Buckets
• ~ 14000 Compute instances
• ~ 200 CloudSQL instances
• ~ 6400 Google Groups
• ~ 1000 AppEngine instances

(not an exhaustive list of all resource types audited)

The inventory alone is giving us a grand overview of the state of the infrastructure and provides a detailed audit trail we can tap into in case of incidents and internal investigations.

With the current setup we are able to identify many violations on the spot, for example: public GCS buckets, exposed CloudSQL instances, unexpected entries in IAM policies and bogus firewall rules.

By plugging into Forseti’s extensible notification system we are able to automatically deliver alerts directly to the team owning the affected resource alongside with instruction on how to fix the problem, freeing the security team from the burden of manually getting involved.

This new level of automation allows the security team to be involved only when an alert hasn’t been acknowledged or the team affected needs direct support.

There is still lots of room for improvement and while we are already getting a lot of value out of Forseti and we will keep working on it. We would love to see a community grow around the tool: you can read more about Forseti on the Google Cloud Platform blog, on the Forseti web site and join the discussion in the forum and on GitHub!

Tags: security