OSPF in the DataCenter

My first blog on CCDE will be OSPF in the DataCenter, as this is a subject that I will be working on at my regular job in the coming months. I’m reading up on google pages, and listening to Orhan lectures on it, and reading the Cisco Press CCDE Study Guide, and must confess that there are some terms I wasn’t familiar with. I helped to build a datacenter in 2013-2014, using the Nexus fabric, 6ks and 2ks, We used OSPF and BGP protocols. Here are some things I have learned about OSPF in the Data Center, but as always there are gaps in my knowledge in this area.

One thing I have to start with, I’m a little embarrassed to say, I’ve never heard of Clos before. So when the lectures started and Clos was referred to, I had to find out what Clos is. If you google Clos, you find, first of all, it’s not CLOS, it’s Clos fabric. So, datacenter network is called “fabric”. The term fabric refers to how all the switch, server and storage nodes all connect to each other like a tightly woven fabric. Clos is known as a two-tier data center fabric. A two-tier data center fabric is a collapsed core, so the core and the aggregate is in the same box. A picture can explain a lot easier than words:

Whereas the classical design is a separate layer for core and distribution/aggregate and access or Tier 3 topology as shown below:

OSPF can be used at the data center edge to advertise datacenter prefixes to the WAN and Campus Network. This means basically you have OSPF point-to-point links between the distribution and the core to advertise the datacenter subnets. But also, OSPF can be used as a data center fabric protocol. This means that access and aggregation can be layer 3. In this case the servers are not attached to layer 2 switches with first hop redundancy protocols but things like Anycast gateway with vxlan. So fabric means many paths non blocking and the traditional hsrp or first hop redundancy protocol with stp has blocking paths. But anycast with vxlan would be several machines sharing the same ip address. You can use things like fabricpath/trill/802.1ad. This provides scalability and redundancy.

This may present problems because data centers are very densely connected networks so OSPF flooding can cause scalability problems. The answer to this is the Clos topology. Large scale data centers mainly use Clos (leaf and spine). In order to scale, multi stage Clos topologies can be used. So the picture of a Clos topology looks like this:

In data centers the edge switches are normally called TOR or Top of Rack switches and the aggregation switches are called the Leaf and the core switches are called Spine. You can have 3 stage, 5 stage, 6 stage, depending on the requirements. The fabric provides basic connectivity with the possibility to carry one or more overlays. The fabric must support non equidistant end-points. The fabric must support spine and leaf (Clos) plus isomorphic topologies within it’s network.

According to the IETF Routing group they discuss what should be the requirements for routing in the data center. The fabric should support 250,000 routes at 5000 fabric nodes, convergence time less than 250ms. The fabric should support 500,000 routes at 750,000 fabric nodes with a convergence time of less than 500ms. The fabric should support 1 million routes at 10,000 fabric nodes with convergence time below 1 second. Ospf cannot support this. This is too much for any IGP protocol. These are large scale data centers where bgp should be used instead.

So OSPF is lacking many of the requirements for fabric in the datacenter. There are other requirements, like it must support equal cost multi-path, ECMP, also wECMP, which is weighted ECMP. It distributes traffic unequally over multiple paths for better load balancing. It also must support UCMP which is unequal cost multi-path for forwarding traffic of an ECMP route by using a ratio of weights. OSPF can only have 4 different ECMP, but with large data centers you should support up to 256 paths. They are working on the convergence with OSPF and BGP called BGP-SPF. They are also working on RIFT, routing in factories.

So in conclusion, OSPF can work in small data centers, where you are only doing 4 equal cost multi path, and small routing tables. If you have a Massively Scaled Data Center, (MSDC), you will need to look at these newer technologies, like BGP SPF or RIFT.