ComSocSCV Meeting Report: 40/100 Gigabit Ethernet – Market Needs, Applications, and Standards

Posted on by

ComSocSCV Meeting Report:  40/100 Gigabit Ethernet – Market Needs, Applications, and Standards

 

Introduction

 

At its October 13, 2010 meeting, IEEE ComSocSCV was most fortunate to have three subject matter experts present and discuss 40G/100G Ethernet- the first dual speed IEEE 802.3 Ethernet standard.  The market drivers, targeted applications, architectecture and overview of the the recently ratified IEEE 802.3ba standard, and the important PHY layer were all explored in detail.  A lively panel discussion followed the three presentations, In addtion to pre-planned questions from the moderator (ComSocSCV Emerging Applications Director Prasanta De), there were many relevent questions from the audience.  Of the 74 meeting attendees, 52 were IEEE members.

 

The presentation titles and speakers were as follows:

1. Ethernet’s Next Evolution – 40GbE and 100GbE by John D’Ambrosia of Force10 Networks

2. The IEEE Std 802.3ba-2010 40Gb/s and 100Gb/s Architecture by Ilango Ganga of Intel Corp

3. Physical Layer (PCS/PMA) Overview by Mark Gustlin of Cisco Systems

Note:  All three presentation pdf’s may be downloaded from the IEEE ComSocSCV web site – 2010 Meeting Archives section (http://www.ewh.ieee.org/r6/scv/comsoc/ComSoc_2010_Presentations.php)

 

Summary of Presentations 

 

1.  The IEEE 802.3ba standard was ratified on June 17, 2010 after several years of hard work.  What drove the market need for this standard?  According to John D’Ambrosia, the “bandwidth explosion” has created bottlenecks eveywhere.  In particular, Increased number of users, faster access rates and methods, new video based services have created the need for higher speeds in the core network.  Mr D’Ambrosia stated,  “IEEE 802.3ba standard for  40G/ 100G Ethernet will eliminate these bottlenecks by providing a robust, scalable architecture for meeting current bandwidth requirements and laying a solid foundation for future Ethernet speed increases.”   John sees 40G/ 100G Ethernet as an enabler of many new network architectures and high bandwidth/ low latencey applications.

 

Three such core networks were seen as likely candidates for higher speed Ethernet penetration: campus/ enterprise, data center, and service provider networks  John showed many illustrative graphs that corroborated the need for higher speeds in each of these application areas.  The “Many Roles and Options for Ethernet Interconnects (in the Data Center),”  “Ethernet 802.3 Umbrella,” and “Looking Ahead -Growing the 40GbE / 100GbE Family” charts were especially enlightening.  We were surprised to learn of the breadth and depth of the 40G/100G Ethernet standard, which can be used to reduce the number of links for: Chip-to-Chip / Modules, Backplane, Twin Ax, Twisted Pair (Data Center), MMF, SMF.  This also improves energy efficiency according to Mr. D’Ambrosia.

 

Looking Beyond 100GbE,  John noted that the industry is being challenged on two fronts: Low cost, high density 100GbE and the Next Rate of Ethernet (?).  To be sure, the IEEE 802.3ba Task Force co-operated with ITU-T Study Group 15 to ensure the new 40G/ 100G  Ethernet rates are transportable over optical transport networks (i.e. the OTN),  But what about higher fiber optic data rates?  Mr. Ambrosia identified the key higher speed market drivers as Data Centers, Internet Exchanges, Carrier’s Optical Backbone Networks.  He predicted that the economics of the application will dictate the solution.

 

2.  Ilango Ganga presented an Overview of the IEEE 802.3ba standard, which has the following characteristics:

 

  • Addresses the needs of computing, network aggregation and core networking applications
  • Uses a Common architecture for both 40 Gb/s and 100 Gb/s Ethernet
  • Uses IEEE 802.3 Ethernet MAC frame format
  • The architecture is flexible and scalable
  • Leverages existing 10 Gb/s technology where possible
  • Defines physical layer technologies for backplane, copper cable assembly and optical fiber medium

 

Mr. Ganga noted there were several sublayers that comprise the IEEE 802.3ba standard:

 

  • MAC  (Medium Access Control) –Data Encapsulation, Ethernet framing, addressing, error detection (e.g. CRC).  The term “Medium Access Control” is a carryover from the days when Ethernet used CSMA/CD to transmit on a shared medium.  Today, most all Ethernet MACs just use the Ethernet frame format and operate over non shared point to point physical media.
  • RS (Reconciliation sublayer) – converts the MAC serial data stream to the parallel data paths of XLGMII (40 Gb/s) or CGMII (100 Gb/s).  It also provides alignment at the beginning frame, while maintaining total MAC transmit IPG
  • 40GBASE-R and 100GBASE-R PCS (Physical Coding sublayer) – Encodes 64 bit data & 8 bit control of XLGMII or CGMII to 66 bit code groups for communication with 40GBASE-R and 100GBASE-R PMA (64B/66B encoding).  Distributes data to multiple physical lanes, provides lane alignment and deskew (due to different receiver arrival times of signals on each lane).  There’s also a Management interface to control and report status
  • Forward Error Correction (FEC) sublayer – Optional sublayer for 40GBASE-R and 100GBASE-R to improve the BER performance of copper and backplane PHYs.  FEC operates on a per PCS lane basis at a rate of 10.3125 GBd for 40G and 5.15625 GBd for 100G
  • 40GBASE-R and 100GBASE-R PMA (Physical Medium Attachment) –  Adapts PCS to a range of PMDs.  Provides: bit level multiplexing or mapping from n lane to m lanes;  clock and data recovery; optional loopback and test pattern geneneration/checking functions
  • 40GBASE-R and 100GBASE-R PMD (Physical Medium Dependent) –  Interfaces to various transmission medium (e.g., backplane, copper or optical fiber medium)/  Transmission/reception of data streams to/from the underlying wireline physical medium.  Provides signal detect and fault function to detect fault conditions.  There are different PMDs for each of the two speeds (40G and 100G bits/sec)

            -40G PMDs:  40GBASE-KR4, 40GBASE-CR4, 40GBASE-SR4, 40GBASE-LR4  

            -100G PMDs: 100GBASE-CR10, 100GBASE-SR10, 100GBASE-LR4, 100GBASE-ER4

  • Auto-Negotiation –  used for copper and backplane PHYs to detect the capabilities of the link partners and configure the link to the appropriate mode.  Allows FEC capability negotiation, and provides parallel detection capability to detect legacy PHYs
  • Management interface – Uses the optional MDIO/MDC management data interface specified for management of 40G and 100G Ethernet Physical layer devices

 

These were illustrated for both 40G and 100G Ethernet with several layer diagrams showing each functional block and inter- sublayer interfaces.  For the electrical interfaces, both Chip- to -Chip or Chip- to- Module electrical specifications might be implemented.  It was noted that PMD specification definesthe MDI electrical characteristics.  Next, 40G and 100 G Ethernet functional block diagram implementation examples were shown.  Finally, Ilango identified two future standards related to IEEE Std 802.3ba:

 

  • IEEE P802.3bg task force is developing a std for 40 Gb/s serial single mode fiber PMD
  • 100 Gb/s backplane and copper cable assemblies Call For Interest scheduled for Nov’10

 

3.  Mark Gustin explained the all important PHY layer, which is the heart of the 802.3ba standard.  The two key PHY sublayers are the  PCS = Physical Coding Sublayer and the PMA = Physical Medium Attachment.

 

  • The PCS performs the following functions:  Delineates Ethernet frames.  Supports the transport of fault information. Provides the data transitions which are needed for clock recovery on SerDes and optical interfaces. It bonds multiple lanes together through a striping/distribution mechanism. Supports data reassembly in the receive PCS – even in the face of significant parallel skew and with multiple multiplexing locations
  • The PMA performs the following functions: Bit level multiplexing from M lanes to N lanes. Clock recovery, clock generation and data drivers.  Loopbacks and test pattern generation and detection

 

Mark drilled down to detail important multi-lane PHY functions of transmit data striping and receiver data alignment.  These mechanisms are necessary because all 40G/ 100G Ethernet PMDs have multiple physical paths or “lanes.”  These are either multiple fibers, coax cables, wavelengths or backplane traces.  Individual bit rates of 10.3125 Gb/s or 25.78125 Gb/s (new PMD will have a rate of 41.25 Gb/s).  Module interfaces are also multiple lanes, which are not always the same number of lanes as the PMD interface.  Therefore the PCS must support a mechanism to distribute data to multiple lanes on the transmit side, and then reassemble the data in the face of skew on the receiver side before passing up to the MAC sublayer.

 

Like Ilango, Mark touched on the topic of higher speed (than 100G) Ethernet.   He speculated that the next higher speed might be 400 Gb/s, or even 1Tb/s?  Mr. Gustin opined that it was too early to tell.  He noted that the IEEE 802.3ba architecture is designed to be scaleable.  In the future, it can support higher data rates by increasing the bandwidth per PCS lane and the number of PCS lanes.  He suggested that for 400 Gb/s, the architecture could be 16 lanes @25 Gb/s for example, with the same block distribution and alignment marker methodology.   Mark summed up by reminding us that the 40G/100G Ethernet standard supports an evolution of optics and electrical interfaces (for example, a new Single-mode PMD will not need a change to the PCS), and that the same architecture (sublayers and interface between them) can support future faster Ethernet speeds.

 

Panel Discussion/ Audience Q and A Session 

 

The ensuing panel session covered 40G/ 100G Ethernet market segments, applications (data center, Internet exchanges, WAN aggregation on the backbone, campus/enterprise, etc),competing technologies (e.g. Infiniband for the data center), timing of implementations (e.g. on servers, switches, network controllers. There were also a few technical questions for clarification and research related to single lane high speed links.  It was noted by this author that almost 10 years after standardization, servers in the data center only recently have included 10G Ethernet port interfaces while 10G Ethernet switches only now can switch multiple ports at wire-line rates.  So how long will it take for 40G/ 100G Ethernet to be widely deployed in its targeted markets?  The panelists concurred that more and more traffic is being aggregated onto 10G Ethernet links and that will drive the need for 40G Ethernet in the data center.  Mark Gustin said, “100GE is needed today for uplinks in various layers of the network.”. But the timing is uncertain. Higher speed uplinks on Ethernet switches, high performance data centers (e.g. Google), Internet exchanges, wide area network aggregation, and box to box communications were seen as the first real markets for 40G/ 100G Ethernet.  Each market segment/ application area will evolve at its own pace, but for sure the 40G/ 100G Ethernet standard will be an enabler of all of them.

 

The final question was asked by former IEEE 802.3 Chair, Geoff Thompson.  Geoff first noted that 40G/ 100 G Ethernet standard and all the higher speed Ethernet studies being worked in IEEE 802.3 are for the core enterprise or carrier backbone network.  He then asked the panelists when would there be big enough technological advances in the access or edge network to enable higher speeds there, i,e, the on ramps/ off ranps to the core network.  The panelists could not answer this question as it was too far from their areas of expertise.  In particular, nothing was said about the very slow- to- improve telco wireline access network (DSL or fiber) and the need to build out fiber closer to the business and residential customers to achieve higher access rates.  Nonetheless, the audience was very pleased to learn the 802.3ba architecture was scalable and seems to be future proof for higher speed Ethernet.

Author Notes on 40G/ 100G Ethernet Market: 

 

  • The 802.3ba standard also complements efforts aimed at delivering greater broadband access.  An example is the Federal Communication Commission’s “Connecting America” National Broadband Plan, which calls for 100 M bit/sec access for a minimum of 100 million homes across the U.S.  If that were to happen, higher speed optical links would be needed between telco central offices and in the core and backbone networks.
  • We think that this standard will accelerate the adoption of 10G Ethernet now that higher-speed 40G/100G pipes are available to aggregate scores of 10G Ethernet links. By simplifying current link aggregation schemes, it will provide concrete benefits such as lowered operating expense costs and improved energy efficiencies.
  • Key stakeholders for IEEE 802.3ba will include users as well as makers of systems and components for servers, network storage, networking systems, high-performance computing, data centers.  Telecommunications carriers, and multiple system operators (MSOs) should also benefit as they can offer much better cost/ performance to their customers.

References:

 

1.  For further discussion and comments on 40G/ 100 G Ethernet, such as server virtualization and converged networks driving the need for higher network data rates, please refer to this article: When will 40G/100G Ethernet be a real market? https://techblog.comsoc.org/2010/09/09/when-will-40g100g-ethernet-be-a-r…

 

2.  IEEE ComSocSCV web site – 2010 Meeting Archives section (http://www.ewh.ieee.org/r6/scv/comsoc/ComSoc_2010_Presentations.php) for presentation slides.

Archives

Recent Posts