Google’s largest internal network interconnects its Data Centers using Software Defined Network (SDN) in the WAN

Google’s use of SDN in its internal WAN backbone:

Urs Hölzle,Sr VP of Technical Infrastructure at Google presented the opening keynote speech at the 2012 Open Network Summit, April 17 in Santa Clara, CA.  The audience was surprised to learn that Google had built its own switches and SDN confrollers for use in its internal backbone network – the one which is used to interconnect its data centers.

Here are the key points made in Mr. Hölzle’s presentation:

Google currently operates two WAN backbones, according to Hölzle:

1] I-Scale is the public Internet-facing backbone that carries user traffic to and from Google’s data centers. It must have bulletproof performance.

2] G-Scale is the internal backbone that carries traffic between Google’s data centers worldwide. The G-Scale network has been used to experiment with SDN.

  • Google chose to pursue SDN in order to separate hardware from software. This enables the company to choose hardware based on necessary features and to choose software based on protocol requirements.
  • SDN provides logically, centralized network control. The goal is to be more deterministic, more efficient and more fault-tolerant.
  • SDN enables better centralized traffic engineering, such as an ability for the network to converge quickly to target optimum on a link failure.  Determinist behavior should simplify planning vs over provisioning for worst case variability.
  • The SDN controller uses modern server hardware, giving it more flexibility than conventional routers.
  • Switches are virtualized with real OpenFlow and the company can attach real monitoring and alerting servers. Testing is vastly simplified.
  • The move to SDN is really about picking the right tool for the right job.
  • Google’s OpenFlow WAN activity really started moving in 2010. Less than two years later, Google is now running the G-Scale network on OpenFlow-controlled switches. 100% of its production data center to data center traffic is now on this new SDN-powered network.
  • Google built their own OpenFlow switch because none were commercially available. The switch was built from merchant silicon. It has scaled to hundred of nonblocking 10GE ports.
  • Google’s practice is to simplify every software stack and hardware element as much as possible, removing anything that is not absolutely necessary.
  • Multiple switch chassis are used in each domain.
  • Google is using open source routing stacks for BGP and ISIS.
  • The OpenFlow-controlled switches (designed and built by Google) look like regular routers. BGP/ISIS/OSPF now interfaces with OpenFlow controller to program the switch state.
  • A preliminary version of the Open Flow protocol is being used now.  (The Open Flow standard is still maturing).
  • All data center backbone traffic is now carried by this new SDN based network. The old network has been shut down.
  • Google started rolling out centralized traffic engineering in January.
  • Google is already seeing higher network utilization and gaining the benefit of flexible management of end-to-end paths for maintenance.
  • Over the past six months, the new network has seen a high degree of stability with minimal outages.
  • The new SDN-powered network is meeting the company’s SLA objectives.
  • It is still too early to quantify the economics.
  • A key SDN benefit is the unified view of the network fabric — higher QoS awareness and predictability.

Mr. Hölzle said that Google’s software-defined networking system has been running for about six months and that it was therefore too early to accurately benchmark cost savings. “This will have a bigger impact in costs than any technical change like a larger router, or 10 gigabit optical switches instead of 2.5 gigabit.  I would expect the cost reduction to come from better system utilization, and substantially easier management,” he said.

“In utilization alone, we are hoping for a 20 percent to 30 percent reduction,” he continued.  Google’s very specific network applications, like search, made it hard to say what others could expect to save. Hölzle thought that the savings would be enough to compel large Internet service providers to change their systems to S.D.N. over the next five years.

Surprisingly, perhaps, Mr. Hölzle thought that the incumbent networking providers would lead the transition. Start-up networking companies likeNicira have created a stir with their SDN approaches, but Mr. Hölzle thought that the big service providers would have a level of trust with the incumbent network equipment companies (We don’t necessarily agree- there are no incumbent networking companies that are leaders in SDNs).

“The natural players are the ones already in the field – Cisco, Alcatel, Juniper,” he said, noting that NEC was an early leader in S.D.N. “They have the networking management software, just at the level of hardware ports, not data flows.” Google talks with all of these companies about their S.D.N. plans, Mr. Hölzle said. Within a year or two, he thought, Google would be purchasing S.D.N.-related products from one or more of these companies.

SDN use in global WANs:

There were also presentations from NEC, NTT, Verizon and other WAN players endorsing SDN and Open Flow at this conference.  NEC said it’s Open Flow controller, together with IBM switches, would be deployed in the WAN as early as this July.   NTT stated that a global cloud virtualization service that leverages SDN will also be launched this summer.

Complete program with selected presentations is at:


GigaOM wrote that the conference was like “a giant science fair for the networking industry. There are arcane demonstrations detailing how software-defined networks and the OpenFlow protocol will change the way networks are built, managed and operated. There are speakers from Google, Verizon and Yahoo detailing their projects and successes with OpenFlow as well as investors and bankers swarming the whole event.”

“The creation of the OpenFlow protocol, which separates the act of directing how packets move across a network from the physical act of moving those packets, has helped create excitement around networking, and is precipitating change. The change is actually the creation of software-defined networks that are programmable (for the record, a software defined network doesn’t need OpenFlow). There’s also a third change that’s been going on regarding the commoditization of networking hardware and the rise of merchant silicon.”

Opinion Piece:

Forbes magazine pointed out that SDN networks are “more secure, more dependable and much easier to manage” because the software that controls network traffic is separate from the physical routers and switches.

“By separating the software that controls network traffic from the physical routers and switches, SDN should make networks more secure, more dependable and much easier to manage. Because SDN runs on commodity hardware, it could translate into siginficant savings for network operators. Perhaps most important, it opens up the network to the possibility of vast innovation.

For Google, software defined networking represented a better way of moving traffic between its global data centers. According to Holzle, things that were hard to do on processors embedded in a networking box become much easier when they separately designed and merely communicated to the hardware using OpenFlow. “You can use all the [computer] tools for software development and that makes it faster to develop software that is higher in quality,” he said.

One of the big advantages for Google is better traffic management—this new approach basically ensures that every lane on its global network of data highways is smoothly moving packets toward their destinations. “Soon we will be able to get very close to 100 percent utilization of our network,” Holzle said. This is a big increase from the industry expectation of thirty to forty percent utilization.”

The market segments where SDN might be advantageously used include:

  • Cloud Services Providers / large website data centers
  • Universities and research campus networks
  • Metro Area CSP data center interconnect
  • Enterprise data centers
  • Internet service provider core routed networks
  • Campus LAN
  • Enterprise WAN

Author’s NOTE:Also see this blog post:

Check out IEEE ComSocSCV July 11 meeting: 6pm-*45pm @Texas Instruments Building E, 2900 Semiconductor Dr., Santa Clara, CA 95051. Software Defined Networking (SDN) Explained- New Epoch or Passing Fad?

Session Abstract: After several years of research, Software Define Networking (SDN) has finally become a reality. At this year’s Open Networking Summit, Google announced it had already deployed its own SDN design in the backbone network that interconnects all its Data Centers. NTT and Verizon hinted that they’d deploy SDN soon, while network equipment vendors indicated they were committed to the concept. IT executives and managers are also taking notice. One pundit predicted a ‘new epoch’ in networks based on SDN- for data centers, campus networks and WANs. But what exactly is SDN and the associated OpenFlow protocol that the Open Networking Foundation (ONF) is standardizing?


6:00 – 6:30pm Networking and Refreshments
6:30 – 6:40pm Opening remarks
6:40 – 7:25pm SDN Overview & Research Projects by Guru Parulkar
7:25 – 8:10pm ONF: Taking OpenFlow and SDN from lab to market by Dan Pitt
8:10 – 8:40pm Panel Session, Discussion & Q&A


 More info at:

Wireline Broadband network providers take largest slice of IT infrastructure pie in 2011

U.S.-based wireline-broadband network providers last year accounted for 41% of the $66 billion spent on information infrastructure in the U.S., while wireless carriers kicked in 40% of the total, according to a report by USTelecom. The other 19% came from cable operators (AKA MSOs).  This information is contained in the research brief: Updated Capital Spending Data Show Continued Significant Broadband Investment in Nation’s Information Infrastructure.”

2011 capital expenditures for the industry as a whole—including wireline, wireless, and cable operators—were approximately $66 billion, roughly the same as 2010.  Furthermore, the data show that broadband providers have made nearly $1.2 trillion in capital investments from 1996 through 2011.

Although the overall investment level has not changed much in the last year, the amount of data traffic that networks carry has increased dramatically– from the equivalent of 8.3 million DVDs per month in 2000 to the equivalent of more than 1.4 billion DVDs per month in 2010, USTelecom said. The association expects traffic levels to triple again over the next five years.

The largest driver of bandwidth demand in 2010 and 2011 was consumer video over fixed networks, representing more than two-thirds of U.S. data traffic in 2010 and approaching three-quarters in the next five years, said USTelecom.

According to the USTelecom research, 96% of Americans now have access to fixed broadband, up 1% since 2010. Researchers also noted that 80% of U.S. households have at least two broadband providers from which to choose.  The research brief does not indicate how researchers defined broadband.

It appears that some of USTelecom’s landline carriers may be feeling overshadowed by wireless carriers, which have garnered much of consumers’ attention in recent years. “Nearly all of U.S. wireless data traffic, the fastest growing data traffic segment, utilizes fixed network connections,” the research brief notes. “Fixed backhaul connections link cell towers to the network and, increasingly, mobile data traffic is offloaded onto Wi-Fi enabled fixed network connections via dual-mode Wi-Fi cellular devices in order to alleviate mobile network capacity limitations.”

Separately, AT&T reported that it netted 718,000 U-verse broadband and 200,000 TV customers in the first quarter. Broadband users signed up for faster service in greater numbers than a year ago with 45% opting for speeds of at least 6 Mbps. Overall, AT&T exceeded analysts’ estimates in the first quarter on a 5.2% increase in net profit due largely to higher wireless data service revenue. 

But the U.S. cellular market is nearing saturation after a growth spurt fueled by sales of the iPhone last year. Wireless penetration in the U.S. is 105 percent when including mobile devices like tablet computers, said Bob Roche, a statistician with CTIA, a wireless-industry trade group.

The slowing market is forcing AT&T into more intense competition with Verizon Wireless and Sprint Nextel Corp. (S), with the carriers fighting over a shrinking pool of people who don’t yet have mobile phones. They’re also trying to get customers to upgrade to smartphones such as the iPhone that let users browse the Web and stream video.

"Mobile Patent War" to be explored at May 9 IEEE ComSocSCV Meeting

The seemingly never ending “Mobile Patent War”  will be examined from many different perspectives at the May 9 IEEE ComSocSCV Meeting in Santa Clara, CA.   Samsung is co-sponsoring the event and will chair the panel session. Participating companies include: Google, HP, Qualcomm, Rambus, SCU Law Dept and AJIS LLC (our own IEEE member Jonathan Wells).  The panel will look at patent issues pertaining to smart phones, media tablets (which are now generating more Internet traffic than smart phones) and other mobile devices.  All of these devices are actually minitiarized, special purpose computers.

Here is the latest information on the program: MOBILE PATENT LITIGATION LANDSCAPE

Introduction:  Ken Korea of Samsung- Head of Silicon Valley Patent/IP Office

I.       Patent suits between competitors  (John Scott of Qualcomm) – Apple v. HTC/Motorola/Samsung, Microsoft v. Motorola, etc.

II.     NPE (Non Patent Entities) patent suits (Cynthia Bright of HP) are they increasing or decreasing?  Has the AIA had any impact?

III.    Acquiring patents for protection (Josh McGuire of Google)  – CPTN’s acquisition of Novell patents, Nortel patent auction, Google’s purchase of IBM patents etc.

IV.    Patent remedies (Colleen Chien of SCU Law School)- Patent remedies, case of ITC v district court

V.     Companies’ transfer of IP Assets to NPEs (Tom Lavelle of Rambus)Micron to RRR, Nokia to Mosaid, Apple to Digitude, etc.

VI.    What does this mean for engineers? (Jonathan Wells of AJIS LLC- IEEE ComSoc Discussion Group member)

For meeting details please visit:

Background Information

Below is an edited version of a related article, which appeared April 13, 2012, on page B1 in some U.S. editions of The Wall Street Journal, with the headline: Smartphone Patents: The Never-Ending War.

The WSJ notes that Apple is waging a patent crusade against mobile-phone giants it believes ripped off the iPhone.  Take the iPhone’s  “slide to unlock” feature, as an example.   Apple has secured two key U.S. patents on slide-to-unlock—a technology that lets users wake a dormant phone with a finger-swipe across the screen. And it is wielding those patents like swords against rivals around the world.  (Even though slide to unlock has nothing to do with mobile or wireless communications!)

In recent months, Apple has sued HTC Corp. in Delaware and Germany over one of those patents and others. It has used the patents to fight back against suits Motorola Mobility Holdings Inc. filed against it in Miami and Germany. And it has invoked them in lawsuits against Samsung Electronics Co. in Australia, the Netherlands, and San Jose, Calif.

As competition in the more than $200 billion global smartphone industry becomes more cutthroat, Apple and its competitors argue that even the most minor unique features are crucial to getting an edge. They are engaged in a lawsuit-filing frenzy, asserting their rights to dozens of patents to block rival products. Their goal: to find a patent that sticks, and to force competitors to work around it or strike a licensing deal.

About five years ago, the computer and mobile-phone industries collided. Technological advances turned phones into minicomputers, complete with email, Web access and other features.  Companies from different corners of industry saw opportunity—and moved into the smart phone market.  Among them: phone makers like Nokia Corp.and Motorola Mobility (soon to be owned by Google); hardware makers like Apple; software giants such as Google Inc. and Microsoft as well as South Korean electronics giant Samsung.

Behind the scenes, another battle was brewing over intellectual property. Almost overnight, every player had developed a gripe. The traditional phone makers claimed, for instance, that Apple was abusing their long-held rights to data-transmission designs. Apple complained others were ripping off its designs.

In the past two years, legal disputes have erupted over digital-image storage methods, camera designs, Wi-Fi technologies and well-known software applications like email and calendars, as well as secondary features most consumers barely notice.

The stakes are rising for Apple. Despite the iPhone’s popularity, its market share has been eclipsed by phones that run Android, Google’s mobile operating software.  Fueling the fire at Apple: a sense among executives there that rivals are blatantly stealing its designs. Apple co-founder Steve Jobs, who died in October, said in an authorized biography that he would “spend every penny” to fight copycats.  Apple CEO Tim Cook told investors in October that Apple spends “a lot of time and money and resource in coming up with incredible innovations. And we don’t like it when someone else takes those.”

A joint effort by the government and major wireless carriers to clamp down on stolen handsets, will likely raise the resale value of used smart phones, Through litigation—or the threat of it—Apple has prompted companies to install time-consuming and, in some instances, costly “workarounds” to avoid infringing its patents.

But Apple has yet to permanently knock any of its competitors’ products out of the market. Several rivals, meanwhile, have struck back against Apple, accusing the company of using their own patented designs.

At the center of the war is “slide-to-unlock.” feature.  It dates to late 2005, more than a year before Apple announced a product with a touchscreen. The first iPhone was in the works at the time, and Apple’s software engineers, including one of its current senior vice presidents, Scott Forstall, felt the need for a feature that would prevent the phone from accidentally making a call or sending a text message when pulled from a pocket or jostled in a purse.

Apple’s engineers regarded slide-to-unlock as important because it flavored a user’s first experience with the device, according to a person familiar with the matter. The team tried many iterations, this person said, from different finger-swiping speeds to different-shaped motions.

Two days before Christmas 2005, Apple filed a patent application with the U.S. Patent and Trademark Office containing a handful of rudimentary drawings with ovals and circles.  The diagrams showed an early version of the design that current iPhone models use: a white rectangle with rounded edges that, when touched and dragged to the right, slides alongside a horizontal channel until the device “unlocks” and opens to the home screen.

The patent office granted Apple the patent four years later, in early 2010. That March, Apple sued HTC in Delaware for allegedly infringing slide-to-unlock and other features.

According to a person familiar with the matter, Apple felt that it would be a good starter case because the company thought it was particularly easy to see that HTC had imitated the iPhone interface—by using similar rows of icons, for example.

Several months later, Apple asserted claims against Motorola in Florida, where Motorola was already suing Apple, alleging that over a dozen Motorola products violated Apple’s slide-to-unlock patent.

The slide-to-unlock used on many Motorola phones resembles Apple’s in many ways. Users open the phone by dragging a finger from left to right across the bottom of the phone’s screen. But the visual representations of the sliding motion are somewhat different.

Apple users see a white rectangle move across the screen while, with the Motorola phones, the slide of a finger extends a bar across the screen. Partly for this reason, Motorola claims its so-called “stretch to unlock” doesn’t infringe Apple’s patent.

Samsung, however, posed a unique challenge for Apple on slide-to-unlock. While Apple was waiting for its patent to be issued, Samsung unveiled phones that opened when a user touched the center of a circle on the screen, and dragged a finger to any point outside the circle.

Samsung’s design was different, but in the mind of Apple executives, not different enough. So, in 2009 Apple went back to the patent office, according to a person familiar with the matter, and asked for a patent that would cover a wider variety of slide-to-unlock designs.

Apple got such a patent last October, and in February the company filed suit in San Jose against Samsung, alleging Samsung violated an array of patents, including slide-to-unlock.

Earlier this year, the combatants all got a surprise: an obscure Swedish touchscreen maker called Neonode Inc. disclosed that it had received a patent for a version of slide-to-unlock. Its technology let a mobile-device user switch from one application to another by swiping a finger across a screen.  The company, which briefly made a line of phones prior to a 2008 bankruptcy, had used the mechanism in one of its models.

In a recent Apple-Samsung battle over slide-to-unlock in the Netherlands, Samsung held up Neonode designs as examples of “prior art,” or evidence that Apple’s patents on slide-to-unlock should never have been granted in the first place because someone else had actually beaten Apple to the idea.

A person close to Samsung said the company is likely to use Neonode’s patent to try to knock out Apple in the San Jose case. Samsung has other arguments as well: “Sliding locks have been around since the Middle Ages, and Apple didn’t invent touchscreens,” this person said. “And the combination of the two fits the definition of obvious.”

In Apple’s only win so far on slide-to-unlock, a judge in Munich ruled in February that two of Motorola’s designs violated a European version of Apple’s slide-to-unlock patent.

Motorola, however, quickly “designed around” Apple’s patent, and its phones remained on the German market.

Many intellectual-property experts think that the smartphone war will end in a flurry of licensing and cross-licensing agreements, but that it’s taking way too much time and money to get there.

“When you have companies spending hundreds of millions in litigation, something is seriously wrong with our patent system,” said Michael Carrier, a professor at Rutgers School of Law in Camden, N.J. “You’ve got to wonder whether it’s doing more harm than good,” he added.

Write to Ashby Jones at [email protected] and Jessica E. Vascellaro at [email protected]

Copyright 2012 Dow Jones & Company, Inc. All Rights Reserved

Also see this related NY Times article:  Smartphone Patent Wars: The Coming Sequel

Billions of dollars are being spent to amass patent arsenals, and lawsuits are flying worldwide. Apple, Samsung, Microsoft and Motorola Mobility (or Google, after that patent-inspired acquisition is complete) are the heavyweights in the fray, seeking an edge in the fast-growing smartphone market, using intellectual property as a weapon.


Acknowledgement: The author deeply thanks and is indebted to Samsung’s Ken Korea for doing such a splendid job of organizing this panel and accomodating many late panelist changes.  IEEE ComSocSCV also thanks Samsung for their generous co-sponsorship of this event.

Telecom Industry Fiber Optic Building Spree- Will It Lead to Another Fiber Glut?

According to the WSJ (Optical Delusion? Fiber Booms Again, Despite Bust), the telecom industry has embarked on another fiber optic building spree.  Some 19 million miles of optical fiber were installed in the U.S. last year, the most since the boom year of 2000, research firm CRU Group says.

This new fiber build-out bonanza comes 11 years after the last one fizzled out.  When the telecom bubble burst in 2001, it wiped out $2 trillion in stock market wealth, caused major dislocations at optical networking firms like Ciena and Nortel.  It also led to many start-ups going bust, especially those building Multi Service Provisioning Platforms (MSPPs or “God Boxes).  Will this time be any different?

The Journal article states, “It is early days in what some in the fiber-optic business are calling a new boom for their long-beaten-down industry. Demand is being driven by skyrocketing Internet video traffic, requests from the financial sector for ever-faster trading connections, and soaring mobile phone use—which has to be tied into landline networks. Even the 2009 economic stimulus plan, which set aside $7.2 billion for telecom projects, is pitching in.”

But already some skeptics caution whether enough demand exists to warrant more fiber build-outs. While stock market and commodity futures  trading firms are currently willing to pay a premium for faster connections, some worry that potential new regulations governing high-frequency trading could crimp the market. Skeptics also question how large a mobile traffic surge will materialize given the high cost of delivering wireless data.  And even if it does, the amount of fiber needed for mobile backhaul will not be that great.  That pales in comparison to the fiber already installed in telco’s long haul, backbone networks.

The WSJ concurs, “There is also plenty of excess capacity available on the nation’s core fiber-optic networks, according to TeleGeography, a telecom market research firm. And capacity is expected to increase as engineers find new ways of squeezing more data traffic into a single strand of glass.”  And that’s due to continuing advances in DWDM (Dense Wave Division Multiplexing) technology, which facilitates more optical channels, at higher speeds, to be transmitted and received over a Single Mode Fiber (SMF) cable.

“A lot of us look at the current construction boom and question if history may be repeating itself,” said Will Hughs, the top U.S. executive for Australian telecom giant Telstra Corp., which sells long-haul telecom services to U.S. customers. In the underwater-cable market, he said, “the possibility for system excess is even greater” than it was a decade ago because technological advances mean that new cables can be built more quickly.

Telecom build out construction firms say it is the location of the fiber-optic networks—rather than their capacity—that is driving new demand. Networks don’t exist where they are needed—at cellphone towers, suburban office parks, and remote data centers, for instance. Carriers that prize reliability want alternative routes. And new uses, such as high-frequency trading (HFT), are also emerging that call for new routes and “carrier hotels.”

In most cases, new fiber build projects aren’t started until customers have been lined up. But some speculators are installing unsold “dark fiber” that has yet to be lighted up for customers, raising concerns that builders may again be getting ahead of themselves.  Veterans of the 2001 collapse insist they are mindful of the lessons they learned and are being much more careful about where they deploy new cable.

“We are a lot smarter than we ever were 10 or 15 years ago,” says James Crowe, chief executive of Level 3 Communications Inc. His company helped define the telecom boom by building its own network from coast to coast, beginning in 1998. It lost more than 90% of its stock-market value a year after hitting its high in March 2000, but it was one of the few to survive the bust.

In recent months, for the first time since the 1990s, Mr. Crowe has started extending Level 3’s fiber-optic network into areas where he hasn’t yet signed up any customers, betting that the demand he needs to make money is there.

He is starting small, investing $50 million in such projects, less than 10% of Level 3’s capital investments this year. But he says the company built a database of three million office buildings, data centers and cellphone towers, pinpointing areas where Level 3 can risk expanding its network without first selling the new capacity.

“The demand has become so obvious,” Mr. Crowe said in an interview at his headquarters in Broomfield, Colo. referring to the surge in bandwidth intensive uses, like streaming video and smartphones. “You’ve got residential neighborhoods that consume more bandwidth than all of New York City did 15 years ago.”

But what’s so remarkable about this new fiber boom is that it doesn’t seem to be targeting commercial buildings in most metropolitan areas.  Recent market research reports indicate that only 15% of commercial buildings have direct fiber access.  The reason given is that “fiber to the business building” weren a much more difficult if not onerous buildout.  Especially when compared to the long-haul fiber connections that could easily be plowed along railroads or highways.

Opinions differ on whether another fiber glut will result from the new buildout.  Analyst Stephan Beckert says there is no lack of fiber in the country now—and with communications technology continuing to improve, it is hard to see a limit to how much data existing fiber can carry. 

However Allied Fiber’s CEO Hunter Newby says there is an opportunity to offer fiber-optic capacity to Internet companies, wireless carriers, hospitals and others who want an alternative to entrenched carriers like Verizon Communications Inc. and AT&T Inc. “The notion there’s a fiber glut is not true,” Mr. Newby says, arguing that much of the fiber-optic cable that is available is simply not in the right place—not at the suburban office parks and cellphone towers that need it.

We think a key driver of new fiber optic construction will be 40G/100G Ethernet in Internet exchanges and high capacity data centers.  We also see the need to interconnect many such “cloud resident” data centers to enterprise customers private networks and also to each other.  So cloud computing and cloud storage could be a big boon to the fiber optic connectivity industry.

For more information, please see: