Facebook's New Data Center Architecture

Facebook's Alexey Andreyev outlined a new data center switching architecture deployed at the company's new facility in Altoona, Iowa.

In a blog posting, Andreyev writes that while its previous data centers have been built with a hierarchically oversubscribed system of clusters, Facebook set out to make its newest data center into a single, high-performance network for the whole building.

The new design is characterized by small, identical "server pods" that only require basic mid-size switches to aggregate the Top-of-rack (TOR) switches.  Each pod has 48 server racks, each with a 10G connection. This form factor is always the same for all pods.

Each pod is served by a set of four fabric switches.  Each TOR currently has 4 x 40G uplinks, providing 160G total bandwidth capacity for a rack of 10G-connected servers.  The smaller port density of the fabric switches makes their internal architecture very simple, modular, and robust.  The switches are available from multiple sources. Symmetrical bandwidth is provisioned to/from each pod. To implement building-wide connectivity, Facebook created four independent “planes” of spine switches, each scalable up to 48 independent devices within a plane.  The network is all Layer 3 – from TOR uplinks to the edge.  Standard BGP4 is the only routing protocol. Facebook is using its own centralized BGP controller that is able to override any routing paths on the fabric by pure software decisions.

The full posting is here:

https://code.facebook.com/posts/360346274145943/introducing-data-center-fabric-the-next-generation-facebook-data-center-network/