SDN Internet Router – Part 1

Introduction

This is the first part of a series of posts about a project we have been working with for a while now that we call SIR (SDN Internet Router). To give some context to this we will first introduce how the Internet route packets, what peering is and how Spotify connects to the rest of the Internet. Feel free to skip this post if you feel you know these topics already.

What is the Internet?

Despite popular belief this is not Internet:

And despite all the powerpoints you might have seen in the recent years, this isn’t either:

In reality the Internet is a very complex and living entity. Nobody owns or builds the Internet. In fact, the Internet is not a single network. It’s a collection of networks built by various organizations, interconnected to each other. The following picture is a better representation of the Internet than the previous ones.

How does the Internet work?

This is a very broad question so we will focus on how your bits find their way across the world in the time it takes you to sneeze. Let me try an analogy. Your bits travel pretty much in the same way you would move around in your city when you want to go out for dinner; you leave home, you take the red line, then you change to the green line, you get off 3 stops later and walk 5 minutes to the restaurant. Check the following example where the traceroute tool is used to display the network nodes from my home to http://www.spotify.com:


$ traceroute -q 1 spotify.com
traceroute: Warning: spotify.com has multiple addresses; using 194.132.198.244
traceroute to weblb.gslb.spotify.com (194.132.198.244), 64 hops max, 52 byte packets
1 192.168.1.1 (192.168.1.1) 4.756 ms
2 192.168.144.1 (192.168.144.1) 3.761 ms
3 113.red-80-58-82.staticip.rima-tde.net (80.58.82.113) 6.916 ms
4 46.red-80-58-81.staticip.rima-tde.net (80.58.81.46) 6.499 ms
5 et7-0-0-400-grtbcntb1.net.telefonicaglobalsolutions.com (94.142.103.185) 4.299 ms
6 213.140.49.18 (213.140.49.18) 21.639 ms
7 prs-b8-link.telia.net (80.239.192.73) 37.260 ms
8 prs-bb2-link.telia.net (213.155.131.10) 35.814 ms
9 ldn-bb2-link.telia.net (80.91.247.241) 40.077 ms
10 ldn-b3-link.telia.net (62.115.140.241) 39.384 ms
11 spotify-ic-310193-ldn-b3.c.telia.net (213.248.100.102) 38.971 ms
12 weblb.gslb.spotify.com (194.132.198.244) 35.225 ms

First you leave home (hops 1 and 2), after a few stops at your Internet provider (hops 3 to 6) you change to another Internet Provider (hops 7 to 11) and finally you reach Spotify (hop 12). However, this analogy has an important flaw. When you leave home you know where you are going and how to get there. However, when you try to fetch some content from the Internet you know where you want to go, but you have no idea where the place actually is or how to get there. This is due to the nature of the Internet. The Internet grows organically; it is very dynamic and constantly changing.

BGP

Because the Internet is very flexible and dynamic and networks connect to each other in mysterious ways, there must be a way to find out where to find the networks we want to connect to. Every computer on the Internet has an IP, that IP belongs to an IP network and that IP network belongs to a an organisation. Organisations then announce their own IP networks to the other networks they are connected to. These other networks will then propagate this information to the rest of the networks they are connected to and so on. This is done by a protocol called BGP, which you might have heard of before.

As you can see, by propagating this information, networks that are not directly connected know they can reach each other by transiting other networks. This also means that you are at the mercy of the intermediary networks. For example, Fontaine Ind. might decide to reach Cyberdyne via Black Mesa instead of using their common direct link if they think that is more beneficial for them.

Connecting the World

Now that we know the basics of how traffic gets forwarded on the Internet and how you can reach your favorite content, let’s see why and how organisations decide to connect to each other:

1. Transit providers. These companies usually own very large networks; some span countries, continents or some even span the whole world. They will allow other organisations to connect to their network for a fee so they can reach the rest of the world.
2. Subscriber networks. Your broadband provider and your mobile provider (sometimes referred as eyeball networks). In these networks you can find the real reason why the Internet exists today; people using the Internet. Subscriber networks will connect to a transit provider network so the people can reach the rest of the Internet.
3. Content providers. These are companies providing some sort of service or content to users. Spotify, Netflix and YouTube are all content providers. Content providers will connect to at least one transit provider so they can reach the rest of the Internet. Content providers might also connect directly to eyeball networks to lower costs and improve user experience.
4. Content Delivery Network (CDN). CDNs are extremely well connected networks. They cover the whole world and help content providers to deliver their content to users. A content provider might need the help of a CDN to reach users located far away from their data centers or to help with the required bandwidth to serve the content. As mentioned before, these are very well connected networks so CDNs will want to connect to any network than can help them getting closer to their customers or to subscriber networks.
5. Internet eXchange Point (IXP). IXPs provide the infrastructure needed to conveniently connect other networks at one or more physical locations. These are hubs where a lot of networks can connect to each other easily.

Delivering music at warp speed

At Spotify we are committed to deliver music to you in the best way possible. To do that we do a few things:

1. We have two transit providers to make sure we can reach all of you. Having two transit providers ensures that we can still reach you even if one of them experiences technical issues.
2. We use CDNs to reach faraway users and help with the bandwidth required to send you the music. We have data centers in London, Stockholm, Ashburn (VA) and San Jose (CA). We love you all, no matter if you are in Stockholm, London, Barcelona, New York or Sydney. So by using CDNs we make sure our users don’t have to wait for their bits to travel all over the world.

In addition, recently we started connecting directly to several eyeball networks on several IXPs. By doing this we get closer to our users, lower latency, make internet providers happy and improve our user’s experience overall.

However, achieving this comes with some technical and economical challenges. Further up in this article we mentioned that every company announces their IP networks and that the information is propagated across the world. By the end of 2015 the Internet had more than 585,000 possible IP networks to connect to. This means that Internet routers have to be able to deal with that vast amount of information in a matter of nanoseconds. In order to forward traffic extremely fast routers require very specific and expensive components called ASICs. An Internet Router able to hold that amount of routes can cost around 500,000 USD per unit.

There must be a better way™

At Spotify we like being smart and innovative, especially if it means playing with cool technology and saving money with it. So we started thinking if we could lower costs by using cheaper equipment that didn’t have super expensive ASICs.

Switches are extremely fast and cheap equipment that you can find in any datacenter. Compared to routers they consume very little power, they are way smaller and the price per port is an order of magnitude cheaper. However, the hardware is limited and can’t hold all the possible routes in the world.

However:

1. At Spotify we already have transit providers and they know how to reach everybody.
2. We connect to eyeball networks.
3. We divide users per region. I.e. users in NY might go to the datacenter in Ashburn, users in Germany to the IXP in Frankfurt, Stockholm users to the IXP in Stockholm, etc.

So why do we need to hold the entire routing table in our devices? Wouldn’t it be smart to try to figure which routes we need and use our transit in case we don’t know how to reach a particular user?
Well, that’s actually a pretty good but long story and that is the subject of the next post in this series..

つづく