HOTi Summary Part II. Optimal Network Topologies for HPC/DCs; SDN & NFV Impact on Interconnects

Posted on by

Introduction:

Two talks are summarized in this second HOTi summary article.  The first examines different topologies for High Performance Computing (HPC) networks with an assumption that Data Center (DC) networks will follow that topology. The second talk was a keynote on the impact of SDN, NFV and other Software Defined WAN/ open networking schemes.

1.  Network topologies for large-scale compute centers: It’s the diameter, stupid!   Presented by Torsten Hoefler, ETH Zurich 

Overview: Torsten discussed the history and design tradeoffs for large-scale topologies used in high-performance computing (HPC). He observed that Cloud Data Centers (DCs) are slowly following HPC attributes due to a variety of similarities, including: more East-West (vs North-South) traffic, the growing demand for low latency, high throughput, and lowest possible cost per node/attached endpoint.

  • Torsten stated that optical transceivers are the most expense equipment used in DCs (I think he meant per attachment or per network port).  While high speed copper interconnects are much cheaper, their range is limited to about 3 meters for high bandwidth (e.g. N x Gb/sec) interfaces.
  • HPC state of the art topology is known as Dragon Fly, which is based on a collection of hierarchical meshes. It effectively tries to make the HPC network look “fatter.”
  • A high-performance cost-effective network topology called Slim Fly approaches the theoretically optimal (minimized) network diameter.  For a smaller diameter topology, there are less cables needed to traverse nodes and fewer routers/switches are needed, resulting in lower system cost.
  • Slim Fly topology was analyzed and compared to both traditional and state-of-the-art networks in Torsten’s HOTi presentation slides.
  • Torsten’s analysis shows that Slim Fly has significant advantages over other topologies in: latency, bandwidth, resiliency, cost, and power consumption.
  • In particular, Slim Fly has the lowest diameter topology, is 25% “better cost at the same performance” than Dragon Fly and 50% better than a Fat Tree topology (which is very similar to a Clos network).
  • Furthermore, Slim Fly is resilient, has the lowest latency, and provides full global bandwidth throughout the network.
  • Here’s a Slim Fly topology illustration supplied post conference email by Torsten:

  • Torsten strongly believes that the Slim Fly topology will be used in large cloud resident Data Centers as their requirements and scale seem to be following those of HPC.
  • Conclusion: Slim Fly enables constructing cost effective and highly resilient DC and HPC networks that offer low latency and high bandwidth under different workloads.

 

Slim Fly and Dragon Fly References:

https://htor.inf.ethz.ch/publications/img/sf_sc_slides_website.pdf

http://htor.inf.ethz.ch/publications/index.php?pub=251

http://htor.inf.ethz.ch/publications/index.php?pub=187

https://www.computer.org/csdl/mags/mi/2009/01/mmi2009010033-abs.html (Dragon Fly)

2.  Software-Defined Everything (SDe) and the Implications for Interconnects, by Roy Chua, SDx Central

Overview: Software-defined networking (SDN), software-defined storage, and the all encompassing software-defined infrastructure have generated a tremendous amount of interest the past few years. Roy described the emerging trends, hot topics and the impact SDe will have on interconnect technology.

Roy addressed the critical question of why interconnects1 matter.  He said that improvements are needed in many areas, such as:

  • Embedded memory performance gap vs CPU (getting worse)
  • CPU-to-CPU communications
  • VM-to-VM and container-to-container

It’s hoped and expected that SDx will help improve cost-performance in each of the above areas.

Note 1.   In this context, “interconnects” are used to describe the connections between IT equipment (e.g. compute, storage, networking, management) within a Data Center (DC) or HPC center.  This is to be distinguished from DC Interconnect (DCI) which refers to the networking of two or more different DCs.  When the DC is operated by the same company, fiber is leased or purchased to interconnect the DCs in a private backbone network (e.g. Google).

When DCs operated by different companies (e.g. Cloud Service Provider DCs, Content Provider DCs or Enterprise DCs) need to be interconnected, a third party is used via ISP peering, colocation, cloud exchange (e.g. Equinix), or Content Delivery Networks (e.g. Akamai)

SDx Attributes and Hot Topics:

  • SDx provides: agility, mobility, scalability, flexibility (especially for configuration and turning up/down IT resources quickly).
  • Other SDx attributes include:  visibility, availability, readability, security and manageability.
  • SDx hot topics include:  SDN and NFV (Network Function Virtualization) in both DCs and WANs, policy driven networking, applications driven networking (well accepted API’s are urgently needed!)

Important NFV Projects:

  • Virtualization across a service providers network (via virtual appliances running on compute servers, rather than physical appliances/specialized network equipment)
  • CORD (Central Office Re-architected as a Data center)
  • 5G Infrastructure (what is 5G?)

Note:  Roy didn’t mention the Virtualized Packet Core for LTE networks, which network equipment vendors like Ericsson, Cisco and Affirmed Networks are building for wireless network operators.

SDx for Cloud Service Providers may address:

  • Containers
  • Software Defined Storage
  • Software Defined Security
  • Converged Infrastructure

Other potential areas for use of SDx:

  • Server to server/ToR switch communications- orchestration, buffer management, convergence, service chaining
  • Inter DC WAN communications (mostly proprietary to Google, Amazon, Facebook, Microsoft, Alibaba, Tencent, and other large cloud service providers
  • Software Defined WAN [SD-WAN is a specific application of SDN technology applied to WAN connections, which are used to connect enterprise networks – including branch offices and data centers – over large geographic distances.]
  • Bandwidth on Demand in a WAN or Cloud computing service (XaaS)
  • Bandwidth calendering (changing bandwidth based on time of day)
  • XaaS (Anything as a Service, including Infrastructure, Platform, and Applications/Software)
  • 5G (much talk, but no formal definition or specs)
  • Internet of Things (IoT)

NFV Deployment and Issues:

Roy said, “There are a lot of unsolved problems with NFV which have to be worked out.”  Contrary to a report by IHS Markit that claims most network operators would deploy NFV next year, Roy cautioned that there will be limited NFV deployments in 2017 with “different flavors of service chaining” and mostly via single vendor solutions.

–>This author strongly agrees with Roy!

Open Networking Specifications Noted:

  • Open Flow ® v1.4 from the Open Network Foundation (ONF) is the “southbound” API from the Control plane to the Data Forwarding plane in classical SDN.  Open Flow ® allows direct access to and manipulation of the Data Forwarding plane of network devices such as switches and routers, both physical and virtual (hypervisor-based).

[Note that the very popular network virtualization/overlay model of SDN does not use Open Flow.]

  • P4 is an open source programming language designed to allow programming of packet forwarding dataplanes. In particular, P4 programs specify how a switch processes packets. P4 is suitable for describing everything from high- performance forwarding ASICs to software switches.  In contrast to a general purpose language such as C or Python, P4 is a domain-specific language with a number of constructs optimized around network data forwarding.

 

Roy’s Conclusions:

  • SDx is an important approach/mindset to the building of today’s and tomorrows IT infrastructure and business applications.
  • Its requirement of agility, flexibility and scalability drives the need for higher-speed, higher-capacity, lower latency, lower cost, greater reliability interconnects at all scales (CPU-memory, CPU-CPU, CPU-network/storage, VM to VM, machine to machine, rack to rack, data center to data center).
  • Interconnect technology enables new architectures for SDx infrastructure and innovations at the interconnect level can impact or flip architectural designs (faster networks help drive cluster-based design, better multi-CPU approaches help vertical scalability on a server).
  • The next few years will see key SDx infrastructure themes play out in data centers as well as in the WAN, across enterprises and service providers (SDN and NFV are just the beginning).

Postscript:

I asked Roy to comment on an article titled: Lack of Automated Cloud Tools Hampers IT Teams.  Here’s an excerpt:

“While most organizations are likely to increase their investments in the cloud over the next five years, IT departments currently struggle with a lack of automated cloud apps and infrastructure tools, according to a recent survey from Logicworks. The resulting report, “Roadblocks to Cloud Success,” reveals that the vast majority of IT decision-makers are confident that their tech staffers are prepared to address the challenges of managing cloud resources. However, they also feel that their leadership underestimates the time and cost required to oversee cloud resources. Without more cloud automation, IT is devoting at least 17 hours a week on cloud maintenance. Currently, concerns about security and budget, along with a lack of expertise among staff members, is preventing more automation.”

One would think that SDx, with its promise and potential to deliver: agility, flexibility, zero touch provisioning, seamless (re-)configuration, scaleability, orchestration, etc. could be effectively used to automate cloud infrastructure and apps. When might that happen?

Roy replied via email:  “Interesting article—my short immediate reaction is that we’re going through a transitionary period and managing the number of cloud resources can bring its own nightmares, plus the management tools haven’t quite caught up yet to the rapid innovation across cloud infrastructures.”

Thanks Roy!

Archives