These days, any COMSOC member can learn various topics from online resources such as digital library, tutorials now, and industry now, besides You-Tube and IEEE TV. The question is why should IEEE members take time for commuting to a local chapter meeting if they can get the same content with a more professional delivery method online at comfort of their office or home. Can we offer services and values that will attract members to local chapter events? Which paramets are more important? Topic? Or networking opportunities? Or may be speaker? Can we combine online and local activities to provide even more benefits for our members? I will be eagerly waiting to see what you think. Post your comments and thoughts here in this blog.

ComSocSCV Meeting Report:  40/100 Gigabit Ethernet – Market Needs, Applications, and Standards

Introduction

At its October 13, 2010 meeting, IEEE ComSocSCV was most fortunate to have three subject matter experts present and discuss 40G/100G Ethernet- the first dual speed IEEE 802.3 Ethernet standard.  The market drivers, targeted applications, architectecture and overview of the the recently ratified IEEE 802.3ba standard, and the important PHY layer were all explored in detail.  A lively panel discussion followed the three presentations, In addtion to pre-planned questions from the moderator (ComSocSCV Emerging Applications Director Prasanta De), there were many relevent questions from the audience.  Of the 74 meeting attendees, 52 were IEEE members.

The presentation titles and speakers were as follows:

1. Ethernet’s Next Evolution – 40GbE and 100GbE by John D’Ambrosia of Force10 Networks

2. The IEEE Std 802.3ba-2010 40Gb/s and 100Gb/s Architecture by Ilango Ganga of Intel Corp

3. Physical Layer (PCS/PMA) Overview by Mark Gustlin of Cisco Systems

Note:  All three presentation pdf’s may be downloaded from the IEEE ComSocSCV web site – 2010 Meeting Archives section (http://www.ewh.ieee.org/r6/scv/comsoc/ComSoc_2010_Presentations.php)

Summary of Presentations 

1.  The IEEE 802.3ba standard was ratified on June 17, 2010 after several years of hard work.  What drove the market need for this standard?  According to John D’Ambrosia, the “bandwidth explosion” has created bottlenecks eveywhere.  In particular, Increased number of users, faster access rates and methods, new video based services have created the need for higher speeds in the core network.  Mr D’Ambrosia stated,  “IEEE 802.3ba standard for  40G/ 100G Ethernet will eliminate these bottlenecks by providing a robust, scalable architecture for meeting current bandwidth requirements and laying a solid foundation for future Ethernet speed increases.”   John sees 40G/ 100G Ethernet as an enabler of many new network architectures and high bandwidth/ low latencey applications.

Three such core networks were seen as likely candidates for higher speed Ethernet penetration: campus/ enterprise, data center, and service provider networks  John showed many illustrative graphs that corroborated the need for higher speeds in each of these application areas.  The “Many Roles and Options for Ethernet Interconnects (in the Data Center),”  “Ethernet 802.3 Umbrella,” and “Looking Ahead -Growing the 40GbE / 100GbE Family” charts were especially enlightening.  We were surprised to learn of the breadth and depth of the 40G/100G Ethernet standard, which can be used to reduce the number of links for: Chip-to-Chip / Modules, Backplane, Twin Ax, Twisted Pair (Data Center), MMF, SMF.  This also improves energy efficiency according to Mr. D’Ambrosia.

Looking Beyond 100GbE,  John noted that the industry is being challenged on two fronts: Low cost, high density 100GbE and the Next Rate of Ethernet (?).  To be sure, the IEEE 802.3ba Task Force co-operated with ITU-T Study Group 15 to ensure the new 40G/ 100G  Ethernet rates are transportable over optical transport networks (i.e. the OTN),  But what about higher fiber optic data rates?  Mr. Ambrosia identified the key higher speed market drivers as Data Centers, Internet Exchanges, Carrier’s Optical Backbone Networks.  He predicted that the economics of the application will dictate the solution.

2.  Ilango Ganga presented an Overview of the IEEE 802.3ba standard, which has the following characteristics:

• Addresses the needs of computing, network aggregation and core networking applications
• Uses a Common architecture for both 40 Gb/s and 100 Gb/s Ethernet
• Uses IEEE 802.3 Ethernet MAC frame format
• The architecture is flexible and scalable
• Leverages existing 10 Gb/s technology where possible
• Defines physical layer technologies for backplane, copper cable assembly and optical fiber medium

Mr. Ganga noted there were several sublayers that comprise the IEEE 802.3ba standard:

• MAC  (Medium Access Control) –Data Encapsulation, Ethernet framing, addressing, error detection (e.g. CRC).  The term “Medium Access Control” is a carryover from the days when Ethernet used CSMA/CD to transmit on a shared medium.  Today, most all Ethernet MACs just use the Ethernet frame format and operate over non shared point to point physical media.
• RS (Reconciliation sublayer) – converts the MAC serial data stream to the parallel data paths of XLGMII (40 Gb/s) or CGMII (100 Gb/s).  It also provides alignment at the beginning frame, while maintaining total MAC transmit IPG
• 40GBASE-R and 100GBASE-R PCS (Physical Coding sublayer) – Encodes 64 bit data & 8 bit control of XLGMII or CGMII to 66 bit code groups for communication with 40GBASE-R and 100GBASE-R PMA (64B/66B encoding).  Distributes data to multiple physical lanes, provides lane alignment and deskew (due to different receiver arrival times of signals on each lane).  There’s also a Management interface to control and report status
• Forward Error Correction (FEC) sublayer – Optional sublayer for 40GBASE-R and 100GBASE-R to improve the BER performance of copper and backplane PHYs.  FEC operates on a per PCS lane basis at a rate of 10.3125 GBd for 40G and 5.15625 GBd for 100G
• 40GBASE-R and 100GBASE-R PMA (Physical Medium Attachment) –  Adapts PCS to a range of PMDs.  Provides: bit level multiplexing or mapping from n lane to m lanes;  clock and data recovery; optional loopback and test pattern geneneration/checking functions
• 40GBASE-R and 100GBASE-R PMD (Physical Medium Dependent) –  Interfaces to various transmission medium (e.g., backplane, copper or optical fiber medium)/  Transmission/reception of data streams to/from the underlying wireline physical medium.  Provides signal detect and fault function to detect fault conditions.  There are different PMDs for each of the two speeds (40G and 100G bits/sec)

            -40G PMDs:  40GBASE-KR4, 40GBASE-CR4, 40GBASE-SR4, 40GBASE-LR4  

            -100G PMDs: 100GBASE-CR10, 100GBASE-SR10, 100GBASE-LR4, 100GBASE-ER4

• Auto-Negotiation –  used for copper and backplane PHYs to detect the capabilities of the link partners and configure the link to the appropriate mode.  Allows FEC capability negotiation, and provides parallel detection capability to detect legacy PHYs
• Management interface – Uses the optional MDIO/MDC management data interface specified for management of 40G and 100G Ethernet Physical layer devices

These were illustrated for both 40G and 100G Ethernet with several layer diagrams showing each functional block and inter- sublayer interfaces.  For the electrical interfaces, both Chip- to -Chip or Chip- to- Module electrical specifications might be implemented.  It was noted that PMD specification definesthe MDI electrical characteristics.  Next, 40G and 100 G Ethernet functional block diagram implementation examples were shown.  Finally, Ilango identified two future standards related to IEEE Std 802.3ba:

• IEEE P802.3bg task force is developing a std for 40 Gb/s serial single mode fiber PMD
• 100 Gb/s backplane and copper cable assemblies Call For Interest scheduled for Nov’10

3.  Mark Gustin explained the all important PHY layer, which is the heart of the 802.3ba standard.  The two key PHY sublayers are the  PCS = Physical Coding Sublayer and the PMA = Physical Medium Attachment.

• The PCS performs the following functions:  Delineates Ethernet frames.  Supports the transport of fault information. Provides the data transitions which are needed for clock recovery on SerDes and optical interfaces. It bonds multiple lanes together through a striping/distribution mechanism. Supports data reassembly in the receive PCS – even in the face of significant parallel skew and with multiple multiplexing locations
• The PMA performs the following functions: Bit level multiplexing from M lanes to N lanes. Clock recovery, clock generation and data drivers.  Loopbacks and test pattern generation and detection

Mark drilled down to detail important multi-lane PHY functions of transmit data striping and receiver data alignment.  These mechanisms are necessary because all 40G/ 100G Ethernet PMDs have multiple physical paths or “lanes.”  These are either multiple fibers, coax cables, wavelengths or backplane traces.  Individual bit rates of 10.3125 Gb/s or 25.78125 Gb/s (new PMD will have a rate of 41.25 Gb/s).  Module interfaces are also multiple lanes, which are not always the same number of lanes as the PMD interface.  Therefore the PCS must support a mechanism to distribute data to multiple lanes on the transmit side, and then reassemble the data in the face of skew on the receiver side before passing up to the MAC sublayer.

Like Ilango, Mark touched on the topic of higher speed (than 100G) Ethernet.   He speculated that the next higher speed might be 400 Gb/s, or even 1Tb/s?  Mr. Gustin opined that it was too early to tell.  He noted that the IEEE 802.3ba architecture is designed to be scaleable.  In the future, it can support higher data rates by increasing the bandwidth per PCS lane and the number of PCS lanes.  He suggested that for 400 Gb/s, the architecture could be 16 lanes @25 Gb/s for example, with the same block distribution and alignment marker methodology.   Mark summed up by reminding us that the 40G/100G Ethernet standard supports an evolution of optics and electrical interfaces (for example, a new Single-mode PMD will not need a change to the PCS), and that the same architecture (sublayers and interface between them) can support future faster Ethernet speeds.

Panel Discussion/ Audience Q and A Session 

The ensuing panel session covered 40G/ 100G Ethernet market segments, applications (data center, Internet exchanges, WAN aggregation on the backbone, campus/enterprise, etc),competing technologies (e.g. Infiniband for the data center), timing of implementations (e.g. on servers, switches, network controllers. There were also a few technical questions for clarification and research related to single lane high speed links.  It was noted by this author that almost 10 years after standardization, servers in the data center only recently have included 10G Ethernet port interfaces while 10G Ethernet switches only now can switch multiple ports at wire-line rates.  So how long will it take for 40G/ 100G Ethernet to be widely deployed in its targeted markets?  The panelists concurred that more and more traffic is being aggregated onto 10G Ethernet links and that will drive the need for 40G Ethernet in the data center.  Mark Gustin said, “100GE is needed today for uplinks in various layers of the network.”. But the timing is uncertain. Higher speed uplinks on Ethernet switches, high performance data centers (e.g. Google), Internet exchanges, wide area network aggregation, and box to box communications were seen as the first real markets for 40G/ 100G Ethernet.  Each market segment/ application area will evolve at its own pace, but for sure the 40G/ 100G Ethernet standard will be an enabler of all of them.

The final question was asked by former IEEE 802.3 Chair, Geoff Thompson.  Geoff first noted that 40G/ 100 G Ethernet standard and all the higher speed Ethernet studies being worked in IEEE 802.3 are for the core enterprise or carrier backbone network.  He then asked the panelists when would there be big enough technological advances in the access or edge network to enable higher speeds there, i,e, the on ramps/ off ranps to the core network.  The panelists could not answer this question as it was too far from their areas of expertise.  In particular, nothing was said about the very slow- to- improve telco wireline access network (DSL or fiber) and the need to build out fiber closer to the business and residential customers to achieve higher access rates.  Nonetheless, the audience was very pleased to learn the 802.3ba architecture was scalable and seems to be future proof for higher speed Ethernet.

Author Notes on 40G/ 100G Ethernet Market: 

• The 802.3ba standard also complements efforts aimed at delivering greater broadband access.  An example is the Federal Communication Commission’s “Connecting America” National Broadband Plan, which calls for 100 M bit/sec access for a minimum of 100 million homes across the U.S.  If that were to happen, higher speed optical links would be needed between telco central offices and in the core and backbone networks.
• We think that this standard will accelerate the adoption of 10G Ethernet now that higher-speed 40G/100G pipes are available to aggregate scores of 10G Ethernet links. By simplifying current link aggregation schemes, it will provide concrete benefits such as lowered operating expense costs and improved energy efficiencies.
• Key stakeholders for IEEE 802.3ba will include users as well as makers of systems and components for servers, network storage, networking systems, high-performance computing, data centers.  Telecommunications carriers, and multiple system operators (MSOs) should also benefit as they can offer much better cost/ performance to their customers.

References:

1.  For further discussion and comments on 40G/ 100 G Ethernet, such as server virtualization and converged networks driving the need for higher network data rates, please refer to this article: When will 40G/100G Ethernet be a real market? https://techblog.comsoc.org/2010/09/09/when-will-40g100g-ethernet-be-a-r…

2.  IEEE ComSocSCV web site – 2010 Meeting Archives section (http://www.ewh.ieee.org/r6/scv/comsoc/ComSoc_2010_Presentations.php) for presentation slides.

Note:  This article is co-authored by IEEE ComSocSCV officers Sameer Herlekar and Alan J Weissberger.  Some information used in this article was gathered during a July visit of ATT Labs in San Ramon, CA.

With the recent proliferation of triple-play (high-speed Internet, high-definition television, and phone) services being offered by telcos (such as Verizon and AT&T) and MSOs/ cable operators (including Comcast and Time Warner Cable), subscribers may be able to choose among an array of telecommunications services to meet their needs.  In some geographical areas, the MSO is only one choice for true triple play services, because the telco has not built out their advanced network to cover every U.S. city.  For example, if one lives in Santa Clara, CA- the heart of silicon valley- you can only get triple play services from Comcast.   In fact, if you are not a U-Verse customer, the ADSL based Internet service you can obtain is much lower speed than the VDSL2 based High Speed Internet AT&T offers as part of U-Verse.

Many questions arise as to the efficacy of these triple-play services delivered by the telcos and MSOs?  Are these services accessible to all potential subscribers and what do subscribers think about the services?

A recent thread on the IEEE ComSoc SCV email Discussion group (free registration for all IEEE members at www.comsocscv.org) yielded a wealth of first-hand information on precisely the aforementioned issues.

Given that telecommunications service provisioning, like any other business, is driven by customer demand, the latter, in turn, is determined by the subscribers’ perceived need for the service(s), quality of the offered service(s), and subscriber awareness of the availability of the services (determined by the marketing of the services by their respective providers).

The explosive growth of social networking sites including Facebook, Twitter and MySpace, video-sharing websites like YouTube and online gaming websites such as Final Fantasy and World of Warcraft indicates that subscriber demand for high-bandwidth internet services is at an all-time high. Combined with the growing demand for high-definition (HD) television programming, overall subscriber demand for bandwidth is growing exponentially. Consequently, both telcos and cable operators have been forced to upgrade their network hardware and architectures to accommodate the ever-burgeoning demand for bandwidth. At the same time, the key business objective to stay profitable has not been lost on the service providers who have responded by offering customers the so-called triple-play services of high-speed internet, HD television and digital phone service.

The two principal telcos in the telecommunications services sector are Verizon (VZ) and AT&T. According to a report released by Information Gatekeepers Inc. (IGI) on July 15, 2010 the two companies in a recent year combined for 76% of total capital expenditure by major phone companies and over 46% of the total capital spent that year by all telecommunication carriers.

According to a Wall Street Journal article in July titled “Verizon’s fiber optic hole” by Martin Peers, VZ has invested $23 billion on their triple-play service offering FiOS which is based on fiber-to-the-home (FTTH) technology. On the other hand, AT&T’s U-verse service features fiber-to-the-curb (FTTC)¹ with copper cables reaching individual subscriber premises over a digital subscriber line (DSL) access line.

Footnote 1.  FTTC is often referred to as Fiber to the Node (FTTN) or Fiber to the Cabinet.

On a recent visit to AT&T Labs in San Ramon, CA, several IEEE ComSoc SCV officers learned that AT&T is pouring money into U-verse as it foresees tremendous growth potential for the DSL-FTTC market.  The ComSocSCV officers went on a very impressive tour of AT&T’s U-Verse Lab, which appeared to be much bigger than most telco Central Offices!  AT&T is testing a FTTC/VDLS2 arrangement that will deliver three HD TV channels, High Speed Internet and either digital voice (VoIP) or POTs.

In terms of technology, VZ’s FiOS represents a significant telco plant upgrade compared to U-verse, since the high-bandwidth capable fibers are terminated at the subscriber premises rather than at the curb or cabinet.  For AT&T’s U-verse, it is the quality of the DSL link (from the network node to the subscriber premises) which determines the perceived quality of the overall service. 

Therefore, one would be led to believe that FiOS, built on Fiber to the Premises (FTTP) technology and backed by a major telco (VZ), would be holding a large, if not the largest, portion of the telecommunications services market. However, it is surprising to note that U-verse has, in fact, been outselling FiOS by a whopping 35-40% according to the report by IGI (http://www.igigroup.com/st/pages/FIOS_UVERSE.html).

Sameer Herlekar, IEEE ComSoc SCV Technical Activities Director (and a co-author of this article), believes that the reason for the discrepancy is the larger per-connection cost entailed in deploying FiOS compared to the per-connection cost of U-verse deployments. Moreover, according to WSJ’s Martin Peers, VZ has recently down-sized its promotions and added only 174,000 net connections to the FiOS network in Q02/2010 compared to 300,000 a year earlier. On the other hand, according to Todd Spangler of Multichannel News, AT&T’s revenues from U-verse TV, Internet and voice services nearly tripled over 2009 and is approaching an annual run rate of $3 billion as it “continues to pack on video and broadband subscribers.”

However, not all potential subscribers for U-Verse can get it, while other that have just had it installed “like it a lot, when it was working.”  A recent thread on the IEEE ComSoc SCV discussion group indicated that U-Verse is simply not available in parts of Santa Clara, CA despite U-verse cabinets being installed in the area.  The installation problems experienced by some Discussion Group members seem to have been resolved, but highlight the “growing pains” AT&T is experiencing to make it work reliability and correctly.”

Mr Herlekar states that “according to AT&T network planners, those subscribers served directly from the central office (CO) receive, at present, limited bandwidths sometimes in the order of just hundreds of bits per second. Furthermore, while some subscribers have high-speed connectivity via ADSL2 (newer installations) others have a slower connection with ADSL (older installations), both of which are slower than the state-of-the-art VDSL2 technology.”

Another key issue is technical support and customer service – troubleshooting problems and resolving them.  From the perspective of co-author Alan J. Weissberger, AT&T seems to do a much better job in this area.  Again, from the IEEE ComSocSCV Discussion list, we read of a U-Verse customer who received excellent tech support from AT&T – including customer care from an AT&T Labs Executive in San Ramon, to resolve his installation problems with TV service.  Perhaps, because AT&T is the new kid on the triple play service delivery block, it seems “they try harder.”

Yet, we’ve read that Comcast is gaining market share over the telcos in the broadband Internet market.  We suspect this is because non- triple play telco customers can’t get the higher speeds offered by the MSOs.  Those unlucky customers have to live with older and much slower wireline access technologies (ADSL or ADSL2) from the telcos, rather than the much higher speed Internet available with VDSL2 for U-Verse or FTTP/BPON/GPON for FiOS).  How fast will AT&T and VZ build out their triple play delivery systems?  We suspect that they are not now available in a majority of geographical areas in the U.S.

Dave Burstein of DSL Prime

U.S. Cable Clobbers DSL, U-Verse, FiOS

“Comcast added 399K new cable modem customers in 2010 Q1 to 16.329M. That’s more adds than the total of AT&T (255K to 16,044K) combined with Verizon (90K to 9.3M). Time Warner was also far ahead of Verizon with 212K to 9,206K.  John Hodulik of UBS estimates 67% of the Q! net adds will go to cable, a remarkable change from less than 50% a year ago. This is not because of DOCSIS 3.0, which at $99+ is not selling well,

Overall, cable added about 1M to over 40M. Telcos added about half a million to 33M. Add between 5% and 10% for the companies too small to appear in the chart below. While this could be the start of a precipitous decline, for now we might just be seeing the effect of price increases (Verizon, +12% in one key measure according to Bank of America) and the dramatic cut in U-Verse and now FiOS deployment.

My take is that the telcos would be damned fools not to hold more of the market so that femtocells/WiFi will provide them more robust and profitable wireless networks. Blair Levin came to a similar conclusion, that it’s too early to claim cable is the inevitable winner. But Verizon cutting FiOS by 2-4M homes is exactly the kind of damned fool move that will hurt them in the long run.  U.S. broadband is a two player game with many different possible strategies I can’t predict.”

For the complete article, including graphs and tables, please see:

http://www.dslprime.com/docsisreport/163-c/2957-us-cable-clobbers-dsl-u-verse-fios

In closing, Mr. Weissberger would like to make two key points:

1.  If U Verse or FiOS is not offered in your geographical area, you will have to go to Comcast, TW Cable (or other MSO in your area) by default to get high speed Internet and digital cable TV with On Demand.  Those non triple play reachable customers can NOT get high speed Internet access from ATT or VZ, because those telcos  haven’t upgraded their cable plant in many areas, e.g. from ADSL to FTTN with VDSL2 for U Verse; or from ADSL to FTTP for FiOS.   In my opinion, those non reachable triple play customers are being neglected or even discriminated against by the two big telcos.

Hence, Comcast (or TW Cable or whomever is the cable franchise holder in their geographical area) wins by default.  Perhaps that’s why Comcast is signing up many more high speed Internet (above 5M or 6M b/sec Downstream) customers than AT&T or VZ.   

2.  All triple play customers are in danger of losing all three services on an outage (cable break, power failure, CO/Head end server failure, etc).  The exception is U Verse with POTS where you’d still be able to make voice calls (but U- Verse- VoIP customers would be dead in the water!).  Hence, you need to have a working cell phone if your access or ISP fails.  And that’s not always possible if you are in a remote area, or the hills where cell phone coverage is bad.