Next-generation 200G and 400G switch fabrics:

Meta’s data center fabrics have evolved from 100 Gbps to the next-generation 200 Gbps/400 Gbps. Meta has already deployed 200G-FR4 optics at scale in their data centers and contributed to specifications for 400G-FR4 optics that will be deployed in the future.

Meta has developed two next-generation 200G fabric switches, the Minipack2 [1.].   It is the latest version of Minipack, Meta’s own modular network switch) and the Arista 7388X5, in partnership with Arista Networks. Both of which are also backward compatible with previous 100G switches and will support upgrades to 400G.

Note 1.  Minipack2 is Facebook’s 200G fabric switch (leaf/spine switch) that provides 128 x 200G Ethernet ports by a single 25.6Tbps switch ASIC. It supports 128 QSFP56 ports or 64 QSFP-DD ports when deployed in Facebook’s F16 data center networks.  Similar to Minipack (128x 100G), Minipack2 has a modular architecture that supports multiple port interface types. The specification and the hardware design package of Minipack2 will be contributed to OCP. This workshop will go over hardware architecture of Minipack2 and details on key design decisions, including functional block diagrams, chassis architecture, external and internal interfaces, etc.

The Minipack2 is based on the Broadcom Tomahawk4 25.6T switch ASIC and Broadcom re-timer. The Arista 7388X5 is also based on the Broadcom Tomahawk4 25.6T switch ASIC, with versions of the 7388X5 also utilizing a Credo chipset. They’re high-performance switches that transmit up to 25.6 Tbps and 10.6 Bpps with modular line cards. They support 128x 200G-FR4 QSFP56 optics modules and can maintain a consistent SerDes speed at the switch ASIC, the optics host interface, and on the optics line/wavelength. They simplify connectivity without needing a gearbox to convert data streams. They also have significantly reduced power per bit compared with their previous models (the OCP-accepted Meta Minipack and OCP-Inspired Arista 7368X4, respectively).

Meta has deployed 200G optics (modules pictured above) in their data centers

………………………………………………………………………………………………………………………………………

The Minipack2, Meta’s own modular network switch, developed in partnership with Broadcom

………………………………………………………………………………………………………………………………………….

Meta’s network operating system now powered by an open API:

Meta’s network operating system for controlling the network, Facebook Open Switching System, traditionally used the specific API provided by the chip manufacturer. Now, FBOSS (Meta’s own network operating system for controlling network switches) has been adapted to use the Open Compute Project Switch Abstraction Interface, a standard and open API.

Additionally, Meta has worked with Cisco Systems to support FBOSS with SAI (Switch Abstraction Interface) with their ASICs.  Adapting and migrating FBOSS to SAI enables Meta to onboard multiple ASICS from different vendors more quickly and easily onboard new ones in the future. SAI’s API lets engineers configure new networking hardware without needing to delve into the specifics of the underlying chipset’s SDK. Furthermore, SAI has been extended to even the PHY layer, with Credo Semi supporting FBOSS with their own SAI implementation.

That means data centers can quickly and easily migrate FBOSS across multiple ASICs from different manufacturers with greater ease. It also allows engineers to rapidly configure new networking hardware without the need to tinker with chipset development kits.

Meta expects that with hardware being shared through OCP, supporting SAI will also mean closer collaboration with and feedback from the wider industry. Developers and engineers from across the world will have a chance to work with this open hardware and contribute their own software that can be shared with the industry.

The Metaverse and More:

The metaverse will rely on many technologies, including advanced AI at scale. To deliver a diversity of new workloads that will be created as a result, we continue down the path of disaggregated global networks and data centers that will underpin all of this. The technologies that Meta and the wider industry will create will, of course, need to be fast and flexible, but more than that, they will need to operate efficiently and sustainably — from the data center all the way to edge devices. The only way to achieve this will be through collaboration through communities like OCP and other partnerships.

Open hardware drives the innovation necessary to reach these goals. And our collaborations with both long-standing and new vendors to create open designs for racks, servers, storage boxes, motherboards, and more will help push Meta and the wider industry onto the next major computing platform. We’re only about one percent along on the journey, but the road to the metaverse will be paved with open advanced networking hardware.

References:

OCP Summit 2021: Open networking hardware lays the groundwork for the metaverse

https://siliconangle.com/2021/11/09/meta-announces-next-gen-networking-hardware-2021-ocp-summit/