AI/ML
Will AI clusters be interconnected via Infiniband or Ethernet: NVIDIA doesn’t care, but Broadcom sure does!
InfiniBand, which has been used extensively for HPC interconnect, currently dominates AI networking accounting for about 90% of deployments. That is largely due to its very low latency and architecture that reduces packet loss, which is beneficial for AI training workloads. Packet loss slows AI training workloads, and they’re already expensive and time-consuming. This is probably why Microsoft chose to run InfiniBand when building out its data centers to support machine learning workloads. However, InfiniBand tends to lag Ethernet in terms of top speeds. Nvidia’s very latest Quantum InfiniBand switch tops out at 51.2 Tb/s with 400 Gb/s ports. By comparison, Ethernet switching hit 51.2 Tb/s nearly two years ago and can support 800 Gb/s port speeds.
While InfiniBand currently has the edge, several factors point to increased Ethernet adoption for AI clusters in the future. Recent innovations are addressing Ethernet’s shortcomings compared to InfiniBand:
- Lossless Ethernet technologies
- RDMA over Converged Ethernet (RoCE)
- Ultra Ethernet Consortium’s AI-focused specifications
Some real-world tests have shown Ethernet offering up to 10% improvement in job completion performance across all packet sizes compared to InfiniBand in complex AI training tasks. By 2028, it’s estimated that: 1] 45% of generative AI workloads will run on Ethernet (up from <20% now) and 2] 30% will run on InfiniBand (up from <20% now).
In a lively session at VM Ware-Broadcom’s Explore event, panelists were asked how to best network together the GPUs, and other data center infrastructure, needed to deliver AI. Broadcom’s Ram Velaga, SVP and GM of the Core Switching Group, was unequivocal: “Ethernet will be the technology to make this happen.” Velaga opening remarks asked the audience, “Think about…what is machine learning and how is that different from cloud computing?” Cloud computing, he said, is about driving utilization of CPUs; with ML, it’s the opposite.
“No one…machine learning workload can run on a single GPU…No single GPU can run an entire machine learning workload. You have to connect many GPUs together…so machine learning is a distributed computing problem. It’s actually the opposite of a cloud computing problem,” Velaga added.
Nvidia (which acquired Israel interconnect fabless chip maker Mellanox [1.] in 2019) says, “Infiniband provides dramatic leaps in performance to achieve faster time to discovery with less cost and complexity.” Velaga disagrees saying “InfiniBand is expensive, fragile and predicated on the faulty assumption that the physical infrastructure is lossless.”
Note 1. Mellanox specialized in switched fabrics for enterprise data centers and high performance computing, when high data rates and low latency are required such as in a computer cluster.
…………………………………………………………………………………………………………………………………………..
Ethernet, on the other hand, has been the subject of ongoing innovation and advancement since, he cited the following selling points:
- Pervasive deployment
- Open and standards-based
- Highest Remote Direct Access Memory (RDMA) performance for AI fabrics
- Lowest cost compared to proprietary tech
- Consistent across front-end, back-end, storage and management networks
- High availability, reliability and ease of use
- Broad silicon, hardware, software, automation, monitoring and debugging solutions from a large ecosystem
To that last point, Velaga said, “We steadfastly have been innovating in this world of Ethernet. When there’s so much competition, you have no choice but to innovate.” InfiniBand, he said, is “a road to nowhere.” It should be noted that Broadcom (which now owns VMWare) is the largest supplier of Ethernet switching chips for every part of a service provider network (see diagram below). Broadcom’s Jericho3-AI silicon, which can connect up to 32,000 GPU chips together, competes head-on with InfiniBand!
Image Courtesy of Broadcom
………………………………………………………………………………………………………………………………………………………..
Conclusions:
While InfiniBand currently dominates AI networking, Ethernet is rapidly evolving to meet AI workload demands. The future will likely see a mix of both technologies, with Ethernet gaining significant ground due to its improvements, cost-effectiveness, and widespread compatibility. Organizations will need to evaluate their specific needs, considering factors like performance requirements, existing infrastructure, and long-term scalability when choosing between InfiniBand and Ethernet for AI clusters.
–>Well, it turns out that Nvidia’s Mellanox division in Israel makes BOTH Infiniband AND Ethernet chips so they win either way!
…………………………………………………………………………………………………………………………………………………………………………..
References:
https://www.perplexity.ai/search/will-ai-clusters-run-on-infini-uCYEbRjeR9iKAYH75gz8ZA
https://www.theregister.com/2024/01/24/ai_networks_infiniband_vs_ethernet/
Broadcom on AI infrastructure networking—’Ethernet will be the technology to make this happen’
https://www.nvidia.com/en-us/networking/products/infiniband/h
ttps://www.nvidia.com/en-us/networking/products/ethernet/
Part1: Unleashing Network Potentials: Current State and Future Possibilities with AI/ML
Using a distributed synchronized fabric for parallel computing workloads- Part II
Part-2: Unleashing Network Potentials: Current State and Future Possibilities with AI/ML
AI RAN Alliance selects Alex Choi as Chairman
Backgrounder:
The AI RAN Alliance, formed earlier this year, is a groundbreaking collaboration aimed at revolutionizing the RAN industry. Partnering with tech giants, the goal is to transform traditional Radio Access Networks (RANs) into intelligent, self-optimizing systems using advanced AI technologies. Their website states:
Bringing together the technology industry leaders and academic institutions, the AI-RAN Alliance is dedicated to driving the enhancement of RAN performance and capability with AI. Moreover, we aim to optimize RAN asset utilization, and unlock new revenue streams. By pioneering AI-based innovations in RAN, we aspire to profitably propel the telecom industry towards 6G.
The alliance’s founding members include Amazon Web Services, Inc. (AWS), Arm, DeepSig Inc. (DeepSig), Telefonaktiebolaget LM Ericsson (Ericsson), Microsoft Corporation (Microsoft), Nokia, Northeastern University, NVIDIA, Samsung Electronics, SoftBank Corp. (SoftBank) and T-Mobile USA, Inc. (T-Mobile).
The group’s mission is to enhance mobile network efficiency, reduce power consumption, and retrofit existing infrastructure, setting the stage for unlocking new economic opportunities for telecom companies with AI, facilitated by 5G and 6G.
Image Courtesy of the AI RAN Alliance.
Purpose:
The AI RAN Alliance is dedicated to eliminating the inefficiencies of traditional RAN systems by embedding AI directly into network infrastructures. This shift will enable, for example, dynamic resource allocation, predictive maintenance, and proactive network management.
Industry Benefits:
Enhanced Network Efficiency: Real-time optimized bandwidth allocation and improved user experiences.
Economic Advantages: Cost savings from AI-driven automation and reduced energy consumption.
Innovative Revenue Opportunities: New services such as real-time AI Assistants on your mobile devices.
Key Focus Areas:
- AI for RAN
- AI on RAN (RAN for AI)
- AI and RAN
………………………………………………………………………………………………………………………….
New AI RAN Alliance Chairman:
On August 15, 2024, the AI RAN Alliance appointed Dr. Alex Jinsung Choi, Principal Fellow of SoftBank Corp.’s Research Institute of Advanced Technology as Chairman.
“The AI-RAN Alliance is set to transform telecommunications through AI-RAN advancements, increased efficiency, and new economic opportunities,” said Choi. “As Chair, I’m excited to lead this AI-RAN initiative, working with industry leaders to enhance mobile networks, reduce power consumption, and modernize infrastructure with 5G and 6G with AI/ML. Our goal is to drive societal progress through AI-RAN, transitioning from traditional to next-generation communications infrastructure.”
Satadal Bhattacharjee, Sr. Director of Marketing, Infrastructure BU, ARM, said, “We’re excited to collaborate with Choi, the Chair of the AI-RAN Alliance. Like Choi, we believe that AI will fundamentally change the way wireless services are deployed, fostering broad innovation and enhancing operational efficiency. We look forward to working with key industry leaders from silicon to software to fulfill the promise of ubiquitous AI and 6G.”
Jim Shea, Co-founder and CEO of DeepSig, said, “As a pioneer in AI-native communications together with his prior experience growing the O-RAN ALLIANCE, Choi will lead this important initiative that is shaping the future of intelligent radio access networks. DeepSig’s extensive AI/ML wireless expertise will play a key role in this exciting collaboration to leverage advanced technologies to help the industry unlock unprecedented network efficiency and accelerate innovation.”
Mathias Riback, VP & Head of Advanced Technology U.S., Ericsson, said, “I’m thrilled to welcome Dr. Choi as Chair of the AI-RAN Alliance. As a non-standardization organization, the Alliance can uniquely complement the work of existing SDOs by focusing on shaping innovative use cases that integrate AI with RAN. In addition to realizing benefits from AI in RAN implementations, it will be important to advance ‘AI on RAN’ use cases, where mobile networks play a critical role in enabling AI applications. Ericsson is fully committed to fostering a collaborative environment that unites all players in the evolving AI ecosystem to shape the future of telecom together.”
Shawn Hakl, VP of 5G Strategy, Microsoft, said, “At Microsoft, we recognize artificial intelligence (AI) as a pivotal technology of our era. We are excited to be a part of the AI-RAN Alliance and are particularly pleased to see Choi step into the role of Chair. Choi’s leadership will be key as we collaborate to leverage AI in optimizing RAN infrastructure investments and expanding the capabilities of RAN to introduce new AI-driven services for modern mobile applications.”
Ari Kynäslahti, Head of Strategy and Technology, Mobile Networks at Nokia commented, “Nokia is proud to be part of the AI-RAN Alliance and contribute towards integrating AI into radio access networks. The potential of AI to optimize networks, predict and resolve issues, and enhance performance and service quality is significant. As we embark on this transformative journey, collaboration is essential to harness our collective expertise. We are pleased to see Dr. Alex Choi appointed to this role, and look forward to him guiding our efforts to achieve these goals.”
Tommaso Melodia, William L. Smith Professor, Northeastern University, said, “We are pleased to have Choi as the Chair of the AI-RAN Alliance, leading our efforts to transform the industry. Choi has been a strong advocate for the evolution towards a more open, software-driven, and AI-integrated future. Under Choi’s leadership, the AI-RAN Alliance is set to fast-track the development of new services and use cases by leveraging openness, softwarization, and AI integration to enhance network performance, energy efficiency, spectrum sharing, and security, ultimately redefining the landscape of global communications.”
Soma Velayutham, GM, AI and Telecoms, NVIDIA, said, “The AI-RAN Alliance is a critical initiative for advancing the convergence of AI and 5G/6G technologies to drive innovation in mobile networks. The consortium’s new leadership will bring a fresh perspective and focus on delivering the next generation of connectivity.”
Dr. Ardavan Tehrani, Samsung Research, AI-RAN Alliance Board of Directors Vice Chair, said, “We are excited to have Dr. Alex Choi leading the AI-RAN Alliance as the Chair of the Board. The Alliance will play a pivotal role in fostering collaboration, driving innovation, and transforming future 6G networks utilizing AI. Under Dr. Choi’s leadership, the Alliance will strive to deliver substantial value to end users and operators through pioneering AI-based use cases and innovations.”
Ryuji Wakikawa, VP and Head of Research Institute of Advanced Technology, SoftBank Corp., said, “SoftBank is committed to realizing an AI-powered network infrastructure, and we strongly believe that Choi’s extensive background and expertise will be a great force in advancing AI-RAN technology and driving significant progress for the mobile industry in this AI era with lightning speed.”
John Saw, EVP and CTO, T-Mobile, said, “We are thrilled to have Alex Choi as Chair of the AI-RAN Alliance. AI is advancing at an unprecedented rate and with our 5G network advantage we have a unique opportunity to harness this momentum. By developing solutions that make the most of both RAN and AI on GPUs — and working alongside Choi and the top industry leaders within the Alliance — we believe there is potential for change that will revolutionize the industry.”
Dr. Akihiro Nakao, Professor, The University of Tokyo, said, “Dr. Alex Jinsung Choi’s appointment as Chair of the AI-RAN Alliance represents a pivotal step in advancing AI within the telecommunications sector. His leadership is expected to unite academic and industry efforts, nurturing the next wave of innovators who will drive the future of AI and telecommunications. This initiative will not only fast-track the adoption of AI across diverse applications but also foster international collaboration and set new standards for efficiency, energy management, resilience, and the development of AI-driven services that will reshape the telecommunications industry and benefit society worldwide.”
……………………………………………………………………………………………………………………………………………….
References:
https://ai-ran.org/news/industry-leaders-in-ai-and-wireless-form-ai-ran-alliance/
AI sparks huge increase in U.S. energy consumption and is straining the power grid; transmission/distribution as a major problem
Dell’Oro & Omdia: Global RAN market declined in 2023 and again in 2024
AI Echo Chamber: “Upstream AI” companies huge spending fuels profit growth for “Downstream AI” firms
According to the Wall Street Journal, the AI industry has become an “Echo Chamber,” where huge capital spending by the AI infrastructure and application providers have fueled revenue and profit growth for everyone else. Market research firm Bespoke Investment Group has recently created baskets for “downstream” and “upstream” AI companies.
- The Downstream group involves “AI implementation,” which consist of firms that sell AI development tools, such as the large language models (LLMs) popularized by OpenAI’s ChatGPT since the end of 2022, or run products that can incorporate them. This includes Google/Alphabet, Microsoft, Amazon, Meta Platforms (FB), along with IBM, Adobe and Salesforce.
- Higher up the supply chain (Upstream group), are the “AI infrastructure” providers, which sell AI chips, applications, data centers and training software. The undisputed leader is Nvidia, which has seen its sales triple in a year, but it also includes other semiconductor companies, database developer Oracle and owners of data centers Equinix and Digital Realty.
The Upstream group of companies have posted profit margins that are far above what analysts expected a year ago. In the second quarter, and pending Nvidia’s results on Aug. 28th , Upstream AI members of the S&P 500 are set to have delivered a 50% annual increase in earnings. For the remainder of 2024, they will be increasingly responsible for the profit growth that Wall Street expects from the stock market—even accounting for Intel’s huge problems and restructuring.
It should be noted that the lines between the two groups can be blurry, particularly when it comes to giants such as Amazon, Microsoft and Alphabet, which provide both AI implementation (e.g. LLMs) and infrastructure: Their cloud-computing businesses are responsible for turning these companies into the early winners of the AI craze last year and reported breakneck growth during this latest earnings season. A crucial point is that it is their role as ultimate developers of AI applications that have led them to make super huge capital expenditures, which are responsible for the profit surge in the rest of the ecosystem. So there is a definite trickle down effect where the big tech players AI directed CAPEX is boosting revenue and profits for the companies down the supply chain.
As the path for monetizing this technology gets longer and harder, the benefits seem to be increasingly accruing to companies higher up in the supply chain. Meta Platforms Chief Executive Mark Zuckerberg recently said the company’s coming Llama 4 language model will require 10 times as much computing power to train as its predecessor. Were it not for AI, revenues for semiconductor firms would probably have fallen during the second quarter, rather than rise 18%, according to S&P Global.
………………………………………………………………………………………………………………………………………………………..
A paper written by researchers from the likes of Cambridge and Oxford uncovered that the large language models (LLMs) behind some of today’s most exciting AI apps may have been trained on “synthetic data” or data generated by other AI. This revelation raises ethical and quality concerns. If an AI model is trained primarily or even partially on synthetic data, it might produce outputs lacking human-generated content’s richness and reliability. It could be a case of the blind leading the blind, with AI models reinforcing the limitations or biases inherent in the synthetic data they were trained on.
In this paper, the team coined the phrase “model collapse,” claiming that training models this way will answer user prompts with low-quality outputs. The idea of “model collapse” suggests a sort of unraveling of the machine’s learning capabilities, where it fails to produce outputs with the informative or nuanced characteristics we expect. This poses a serious question for the future of AI development. If AI is increasingly trained on synthetic data, we risk creating echo chambers of misinformation or low-quality responses, leading to less helpful and potentially even misleading systems.
……………………………………………………………………………………………………………………………………………
In a recent working paper, Massachusetts Institute of Technology (MIT) economist Daron Acemoglu argued that AI’s knack for easy tasks has led to exaggerated predictions of its power to enhance productivity in hard jobs. Also, some of the new tasks created by AI may have negative social value (such as design of algorithms for online manipulation). Indeed, data from the Census Bureau show that only a small percentage of U.S. companies outside of the information and knowledge sectors are looking to make use of AI.
References:
https://deepgram.com/learn/the-ai-echo-chamber-model-collapse-synthetic-data-risks
https://economics.mit.edu/sites/default/files/2024-04/The%20Simple%20Macroeconomics%20of%20AI.pdf
AI wave stimulates big tech spending and strong profits, but for how long?
AI winner Nvidia faces competition with new super chip delayed
SK Telecom and Singtel partner to develop next-generation telco technologies using AI
Telecom and AI Status in the EU
Vodafone: GenAI overhyped, will spend $151M to enhance its chatbot with AI
Data infrastructure software: picks and shovels for AI; Hyperscaler CAPEX
SK Telecom (SKT) and Nokia to work on AI assisted “fiber sensing”
SK Telecom (SKT) and Nokia have agreed to work on artificial intelligence (AI) assisted “fiber sensing,” a wired network technology that employs AI to monitor the environment around optical cables. The two companies signed a memorandum of understanding (see photo below) last Wednesday, with a plan to “accumulate empirical data based on machine learning” from SKT’s commercial network. SKT, South Korea’s largest mobile network carrier, said on Monday that it will utilize Nokia’s product to detect earthquakes, climate changes and other unexpected situations that might arise from nearby construction areas in order to stabilize network conditions. The objective is nationwide deployment in South Korea by the end of this year.
In a joint statement, the companies explained when data runs through an optical cable, the phase of the light can change due to various factors like temperature fluctuations or physical strain on the cable. The changes can be detected and analyzed to provide precise measurements of the environmental conditions affecting the fiber. Using AI-based technology, SKT and Nokia aim to stabilize fiber optic networks in advance by tracking the impact of weather conditions and construction on optical cables. The statement added “fiber sensing” has no distance limitations, unlike some existing wired network monitoring technologies, making it possible to quickly apply the new technology to major backbone networks.
SKT-Nokia monitors wired network status with AI:
– Tracking the impact of weather, earthquakes, construction, etc. on optical cables with ‘fiber sensing’ technology
– Immediately applicable to existing networks and no distance restrictions, making it easy to apply to backbone networks
– Both companies’ capabilities will be combined to quickly internalize new AI-based wired network technology
A signing ceremony for the memorandum of understanding took place at SK Telecom’s headquarters Wednesday in central Seoul. SK Telecom’s Ryu Jung-hwan, head of infrastructure strategy and technology, and John Harrington, Nokia’s senior vice president and head of network infrastructure sales for the Asia-Pacific region, attended the event.
SK Telecom’s Ryu Jung-hwan, head of infrastructure strategy and technology, right, and John Harrington, Nokia’s senior vice president and head of network infrastructure sales for the Asia-Pacific region, pose for a photo after a signing ceremony at SK Telecom’s headquarters in central Seoul on Wednesday, August 7th. Photo Credit: SK TELECOM
…………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………….
In July, SKT and Singtel announced that they have signed a Memorandum of Understanding (MoU) to collaborate on building next-generation telecommunications networks that will drive innovation, improve network performance and security and deliver enhanced customer experiences over the next two years. The partners will explore the use of artificial intelligence (AI), orchestration tools, and deepen the domain knowledge of network virtualization and other technologies – central to laying the necessary building blocks for progressing to 6G.
References:
https://www.linkedin.com/feed/update/urn:li:activity:7228552138988134402/
AI winner Nvidia faces competition with new super chip delayed
The Clear AI Winner Is: Nvidia!
Strong AI spending should help Nvidia make its own ambitious numbers when it reports earnings at the end of the month (it’s 2Q-2024 ended July 31st). Analysts are expecting nearly $25 billion in data center revenue for the July quarter—about what that business was generating annually a year ago. But the latest results won’t quell the growing concern investors have with the pace of AI spending among the world’s largest tech giants—and how it will eventually pay off.
In March, Nvidia unveiled its Blackwell chip series, succeeding its earlier flagship AI chip, the GH200 Grace Hopper Superchip, which was designed to speed generative AI applications. The NVIDIA GH200 NVL2 fully connects two GH200 Superchips with NVLink, delivering up to 288GB of high-bandwidth memory, 10 terabytes per second (TB/s) of memory bandwidth, and 1.2TB of fast memory. The GH200 NVL2 offers up to 3.5X more GPU memory capacity and 3X more bandwidth than the NVIDIA H100 Tensor Core GPU in a single server for compute- and memory-intensive workloads. The GH200 meanwhile combines an H100 chip [1.] with an Arm CPU and more memory.
Photo Credit: Nvidia
Note 1. The Nvidia H100, sits in a 10.5 inch graphics card which is then bundled together into a server rack alongside dozens of other H100 cards to create one massive data center computer.
This week, Nvidia informed Microsoft and another major cloud service provider of a delay in the production of its most advanced AI chip in the Blackwell series, the Information website said, citing a Microsoft employee and another person with knowledge of the matter.
…………………………………………………………………………………………………………………………………………
Nvidia Competitors Emerge – but are their chips ONLY for internal use?
In addition to AMD, Nvidia has several big tech competitors that are currently not in the merchant market semiconductor business. These include:
- Huawei has developed the Ascend series of chips to rival Nvidia’s AI chips, with the Ascend 910B chip as its main competitor to Nvidia’s A100 GPU chip. Huawei is the second largest cloud services provider in China, just behind Alibaba and ahead of Tencent.
- Microsoft has unveiled an AI chip called the Azure Maia AI Accelerator, optimized for artificial intelligence (AI) tasks and generative AI as well as the Azure Cobalt CPU, an Arm-based processor tailored to run general purpose compute workloads on the Microsoft Cloud.
- Last year, Meta announced it was developing its own AI hardware. This past April, Meta announced its next generation of custom-made processor chips designed for their AI workloads. The latest version significantly improves performance compared to the last generation and helps power their ranking and recommendation ads models on Facebook and Instagram.
- Also in April, Google revealed the details of a new version of its data center AI chips and announced an Arm-based based central processor. Google’s 10 year old Tensor Processing Units (TPUs) are one of the few viable alternatives to the advanced AI chips made by Nvidia, though developers can only access them through Google’s Cloud Platform and not buy them directly.
As demand for generative AI services continues to grow, it’s evident that GPU chips will be the next big battleground for AI supremacy.
References:
AI Frenzy Backgrounder; Review of AI Products and Services from Nvidia, Microsoft, Amazon, Google and Meta; Conclusions
https://www.nvidia.com/en-us/data-center/grace-hopper-superchip/
https://www.theverge.com/2024/2/1/24058186/ai-chips-meta-microsoft-google-nvidia/archives/2
https://news.microsoft.com/source/features/ai/in-house-chips-silicon-to-service-to-meet-ai-demand/
AI wave stimulates big tech spending and strong profits, but for how long?
Big tech companies have made it clear over the last week that they have no intention of slowing down their stunning levels of spending on artificial intelligence (AI), even though investors are getting worried that a big payoff is further down the line than most believe.
In the last quarter, Apple, Amazon, Meta, Microsoft and Google’s parent company Alphabet spent a combined $59 billion on capital expenses, 63% more than a year earlier and 161 percent more than four years ago. A large part of that was funneled into building data centers and packing them with new computer systems to build artificial intelligence. Only Apple has not dramatically increased spending, because it does not build the most advanced AI systems and is not a cloud service provider like the others.
At the beginning of this year, Meta said it would spend more than $30 billion in 2024 on new tech infrastructure. In April, he raised that to $35 billion. On Wednesday, he increased it to at least $37 billion. CEO Mark Zuckerberg said Meta would spend even more next year. He said he’d rather build too fast “rather than too late,” and allow his competitors to get a big lead in the A.I. race. Meta gives away the advanced A.I. systems it develops, but Mr. Zuckerberg still said it was worth it. “Part of what’s important about A.I. is that it can be used to improve all of our products in almost every way,” he said.
………………………………………………………………………………………………………………………………………………………..
This new wave of Generative A.I. is incredibly expensive. The systems work with vast amounts of data and require sophisticated computer chips and new data centers to develop the technology and serve it to customers. The companies are seeing some sales from their A.I. work, but it is barely moving the needle financially.
In recent months, several high-profile tech industry watchers, including Goldman Sachs’s head of equity research and a partner at the venture firm Sequoia Capital, have questioned when or if A.I. will ever produce enough benefit to bring in the sales needed to cover its staggering costs. It is not clear that AI will come close to having the same impact as the internet or mobile phones, Goldman’s Jim Covello wrote in a June report.
“What $1 trillion problem will AI solve?” he wrote. “Replacing low wage jobs with tremendously costly technology is basically the polar opposite of the prior technology transitions I’ve witnessed in my 30 years of closely following the tech industry.” “The reality right now is that while we’re investing a significant amount in the AI.space and in infrastructure, we would like to have more capacity than we already have today,” said Andy Jassy, Amazon’s chief executive. “I mean, we have a lot of demand right now.”
That means buying land, building data centers and all the computers, chips and gear that go into them. Amazon executives put a positive spin on all that spending. “We use that to drive revenue and free cash flow for the next decade and beyond,” said Brian Olsavsky, the company’s finance chief.
There are plenty of signs the boom will persist. In mid-July, Taiwan Semiconductor Manufacturing Company, which makes most of the in-demand chips designed by Nvidia (the ONLY tech company that is now making money from AI – much more below) that are used in AI systems, said those chips would be in scarce supply until the end of 2025.
Mr. Zuckerberg said AI’s potential is super exciting. “It’s why there are all the jokes about how all the tech C.E.O.s get on these earnings calls and just talk about A.I. the whole time.”
……………………………………………………………………………………………………………………
Big tech profits and revenue continue to grow, but will massive spending produce a good ROI?
Last week’s Q2-2024 results:
- Google parent Alphabet reported $24 billion net profit on $85 billion revenue.
- Microsoft reported $22 billion net profit on $65 billion revenue.
- Meta reported $13.5 billion net profit on $39 billion revenue.
- Apple reported $21 billion net profit on $86 billion revenue.
- Amazon reported $13.5 billion net profit on $148 billion revenue.
This chart sums it all up:
………………………………………………………………………………………………………………………………………………………..
References:
https://www.nytimes.com/2024/08/02/technology/tech-companies-ai-spending.html
https://www.axios.com/2024/08/02/google-microsoft-meta-ai-earnings
https://www.nvidia.com/en-us/data-center/grace-hopper-superchip/
AI Frenzy Backgrounder; Review of AI Products and Services from Nvidia, Microsoft, Amazon, Google and Meta; Conclusions
Microsoft choses Lumen’s fiber based Private Connectivity Fabric℠ to expand Microsoft Cloud network capacity in the AI era
Lumen Technologies and Microsoft Corp. announced a new strategic partnership today. Microsoft has chosen Lumen to expand its network capacity and capability to meet the growing demand on its datacenters due to AI (i.e. huge processing required for Large Language Models, including data collection, preprocessing, training, and evaluation). Datacenters have become critical infrastructure that power the compute capabilities for the millions of people and organizations who rely on and trust the Microsoft Cloud.
Microsoft claims they are playing a leading role in ushering in the era of AI, offering tools and platforms like Azure OpenAI Service, Microsoft Copilot and others to help people be more creative, more productive and to help solve some of humanity’s biggest challenges. As Microsoft continues to evolve and scale its ecosystem, it is turning to Lumen as a strategic supplier for its network infrastructure needs and is investing with Lumen to support its next generation of applications for Microsoft platform customers worldwide.
Lumen’s Private Connectivity Fabric℠ is a custom network that includes dedicated access to existing fiber in the Lumen network, the installation of new fiber on existing and new routes, and the use of Lumen’s new digital services. This AI-ready infrastructure will strengthen the connectivity capabilities between Microsoft’s datacenters by providing the network capacity, performance, stability and speed that customers need as data demands increase.
Art by Midjourney for Fierce Network
…………………………………………………………………………………………………………………………………………………………………………………..
“AI is reshaping our daily lives and fundamentally changing how businesses operate,” said Erin Chapple, corporate vice president of Azure Core Product and Design, Microsoft. “We are focused both on the impact and opportunity for customers relative to AI today, and a generation ahead when it comes to our network infrastructure. Lumen has the network infrastructure and the digital capabilities needed to help support Azure’s mission in creating a reliable and scalable platform that supports the breadth of customer workloads—from general purpose and mission-critical, to cloud-native, high-performance computing, and AI, plus what’s on the horizon. Our work with Lumen is emblematic of our investments in our own cloud infrastructure, which delivers for today and for the long term to empower every person and every organization on the planet to achieve more.”
“We are preparing for a future where AI is the driving force of innovation and growth, and where a powerful network infrastructure is essential for companies to thrive,” said Kate Johnson, president and CEO, Lumen Technologies (a former Microsoft executive). “Microsoft has an ambitious vision for AI and this level of innovation requires a network that can make it reality. Lumen’s expansive network meets this challenge, with unique routes, unmatched coverage, and a digital platform built to give companies the flexibility, access and security they need to create an AI-enabled world.”
Lumen has launched an enterprise-wide transformation to simplify and optimize its operations. By embracing Microsoft’s cloud and AI technology, Lumen can reduce its overall technology costs, remove legacy systems and silos, improve its offerings, and create new solutions for its global customer base. Lumen will migrate and modernize its workloads to Microsoft Azure, use Microsoft Entra solutions to safeguard access and prevent identity attacks and partner with Microsoft to create and deliver new telecom industry-specific solutions. This element alone is expected to improve Lumen’s cash flow by more than $20 million over the next 12 months while also improving the company’s customer experience.
“Azure’s advanced global infrastructure helps customers and partners quickly adapt to changing economic conditions, accelerate technology innovation, and transform their business with AI,” said Chapple. “We are committed to partnering with Lumen to help deliver on their transformation goals, reimagine cloud connectivity and AI synergies, drive business growth, and help customers achieve more.”
This collaboration expands upon the longstanding relationship between Lumen Technologies and Microsoft. The companies have worked together for several years, with Lumen leveraging Copilot to automate routine tasks and reduce employee workloads and enhance Microsoft Teams.
……………………………………………………………………………………………………………………………………………………………………………………………………..
Lumen’s CMO Ryan Asdourian hinted the deal could be the first in a series of such partnerships, as network infrastructure becomes the next scarce resource in the era of AI. “When the world has talked about what’s needed for AI, you usually hear about power, space and cooling…[these] have been the scarce resources,” Asdourian told Fierce Telecom. Asdourian said Lumen will offer Microsoft access to a combination of new and existing routes in the U.S., and will overpull fiber where necessary. However, he declined to specify the speeds which will be made available or exactly how many of Microsoft’s data centers it will be connecting.
Microsoft will retain full control over network speeds, routes and redundancy options through Lumen’s freshly launched Private Connectivity Fabric digital interface. “That is not something traditional telecom has allowed,” Asdourian said.
Asdourian added that Lumen isn’t just looking to enable AI, but also incorporate it into its own operations. Indeed, part of its partnership deal with Microsoft involves Lumen’s adoption of Azure cloud and other Microsoft services to streamline its internal and network systems. Asdourian said AI could be used to make routing and switching on its network more intelligent and efficient.
…………………………………………………………………………………………………………………………………………………………………………………..
About Lumen Technologies:
Lumen connects the world. We are igniting business growth by connecting people, data, and applications – quickly, securely, and effortlessly. Everything we do at Lumen takes advantage of our network strength. From metro connectivity to long-haul data transport to our edge cloud, security, and managed service capabilities, we meet our customers’ needs today and as they build for tomorrow. For news and insights visit news.lumen.com, LinkedIn: /lumentechnologies, Twitter: @lumentechco, Facebook: /lumentechnologies, Instagram: @lumentechnologies and YouTube: /lumentechnologies.
About Microsoft:
Microsoft (Nasdaq “MSFT” @microsoft) creates platforms and tools powered by AI to deliver innovative solutions that meet the evolving needs of our customers. The technology company is committed to making AI available broadly and doing so responsibly, with a mission to empower every person and every organization on the planet to achieve more.
…………………………………………………………………………………………………………………………………………………………………………………..
References:
https://news.lumen.com/2024-07-24-Microsoft-and-Lumen-Technologies-partner-to-power-the-future-of-AI-and-enable-digital-transformation-to-benefit-hundreds-of-millions-of-customers
https://fierce-network.com/cloud/microsoft-taps-lumens-fiber-network-help-it-meet-ai-demand
AI Frenzy Backgrounder; Review of AI Products and Services from Nvidia, Microsoft, Amazon, Google and Meta; Conclusions
Lumen, Google and Microsoft create ExaSwitch™ – a new on-demand, optical networking ecosystem
ACSI report: AT&T, Lumen and Google Fiber top ranked in fiber network customer satisfaction
Lumen to provide mission-critical communications services to the U.S. Department of Defense
Dell’Oro: Optical Transport market to hit $17B by 2027; Lumen Technologies 400G wavelength market
SK Telecom and Singtel partner to develop next-generation telco technologies using AI
SK Telecom (South Korea) and Singtel (Singapore) have initiated a two-year project to develop advanced telecommunication networks. This collaboration aims to drive innovation, improve network performance and security, and enhance customer experiences through the use of artificial intelligence (AI), orchestration tools, and network virtualization.
The project will focus on creating innovative solutions like Edge-AI Infrastructure to enhance connectivity and provide unique AI service offerings. A white paper will describe advancements to assist other global telcos to harnessing the capabilities of 5G and preparing for 6G.
This MOU initiative is expected to not only enhance connectivity but also provide customers with unique AI service offerings and enable the operators to restore services faster, thus improving the customer experience.
Additionally, SKT and Singtel will be putting together a white paper on their advancements in areas such as virtualization, slicing and network evolution that can help other telcos globally to capitalize on the capabilities of 5G and to prepare for 6G in 2030.
SK Telecom (SKT) has signed a Memorandum of Understanding (MOU) with Singtel, Singapore’s leading telecommunications provider, to collaborate on the application of AI technology in communication networks, the development of use cases for 5G network slicing technology, and preparation for 6G technology, aimed at fostering advancements in 5G and next-generation communication technologies. Photo Courtesy of SKT
……………………………………………………………………………………………………………………
Kang Jong-ryeol, SKT’s Head of ICT Infra(CSPO) stated, “The collaboration between SKT and Singtel marks a significant first step in shaping the future of the global telecommunications industry.” He further emphasized, “By combining the strengths of both companies, we aim to achieve efficient high-performance network construction, enhance network stability, and discover new network-based services. Additionally, we will strive to make significant advancements in next-generation communication technologies, including AI-powered wired and wireless infrastructure.”
Tay Yeow Lian, Singtel’s Managing Director, Networks, said, “As a global leader in 5G technology, we’re keen to capitalize on the myriad of capabilities this technology has to offer, especially in the areas of network slicing and with the inclusion of AI. With SKT, we’re looking to not only enhance the experience of our customers but to also drive industry innovation and help us prepare for the evolution to 6G.”
ANNEX: Singtel’s 5G advancements
· Developed Paragon, the industry’s first all-in-one aggregation and business orchestration platform, which allows enterprises to interact with and manage networks, clouds and multi-access edge computing (MEC) infrastructure and applications
· Developed Singtel CUBΣ, a Network-as-a-Service (NaaS) that makes it easier for enterprises to subscribe and manage desired services and multiple vendors as well as gain insights on network utilisation, workload performance and sustainability metrics via a single sign-on digital portal. CUBΣ leverages and integrates AI into its network management systems to deliver enhanced services such as proactive user experience monitoring, incident automation and predictive analytics to anticipate, detect and address incidents faster. This results in improved network performance, optimised resource allocation, enhanced security protocols, elevated the overall user experience, and the development of a network that learns, evolves and self-improves over time – all of which enable faster digital transformation for greater economic growth and innovation.
Major 5G developments from Singtel:
2022
· Launched first public multi-access edge compute for enterprises in Asia with Microsoft
· Launched iSHIP to provide critical satellite-enabled connectivity and digital services for the maritime industry
2023
· Singapore’s first 5G-enabled smart retail showcase
· Achieved 5G upload speed of more than 1.6Gbps in an enterprise deployment
· Completed more than 30 5G trials at Sentosa
· Successfully trialed RedCap technology for better energy savings for IoT devices
2024
· Addition of Starlink satellites for maritime connectivity
· Offered the 5G Express Pass service to concertgoers for Coldplay and Taylor Swift
· Pioneered app-based network slicing, aka User Equipment Route Selection Policy
· Singtel Paragon integrated into Telkomsel’s enterprise product portfolio
· Launch of Paragon-S to spur digital transformation for satellite operators
About SK Telecom:
SK Telecom has been leading the growth of the mobile industry since 1984. Now, it is taking customer experience to new heights by extending beyond connectivity. By placing AI at the core of its business, SK Telecom is rapidly transforming into an AI company with a strong global presence. It is focusing on driving innovations in areas of AI Infrastructure, AI Transformation (AIX) and AI Service to deliver greater value for industry, society, and life.
References:
https://www.koreaittimes.com/news/articleView.html?idxno=132974
SK Telecom, DOCOMO, NTT and Nokia develop 6G AI-native air interface
SK Telecom, Intel develop low-latency technology for 6G core network
SK Telecom and Thales Trial Post-quantum Cryptography to Enhance Users’ Protection on 5G SA Network
Vodafone: GenAI overhyped, will spend $151M to enhance its chatbot with AI
GenAI is probably the most “overhyped” technology for many years in the telecom industry, said Vodafone Group’s chief technology officer (CTO) Scott Petty at a press briefing this week. “Hopefully, we are reaching the peak of those inflated expectations, because we are about to drop into a trough of disillusionment,” he said.
“This industry is moving too quickly,” Petty explained. “The evolution of particularly GPUs and the infrastructure means that by the time you’d actually bought them and got them installed you’d be N minus one or N minus two in terms of the technology, and you’d be spending a lot of effort and resource just trying to run the infrastructure and the LLMs that sit around that.”
Partnerships with hyper-scalers remain Vodafone’s preference, he said. Earlier this year, Vodafone and Microsoft signed a 10-year strategic agreement to use Microsoft GenAI in Vodafone’s network.
Vodafone is planning to invest some €140 million ($151 million) in artificial intelligence (AI) systems this year to improve the handling of customer inquiries, the company said on July 4th. Vodafone said it is investing in advanced AI from Microsoft and OpenAI to improve its chatbot, dubbed TOBi, so that it can respond faster and resolve customer issues more effectively.
The chatbot was introduced into Vodafone’s customer service five years ago and is equipped with the real voice of a Vodafone employee.
The new system, which is called SuperTOBi in many countries, has already been introduced in Italy and Portugal and will be rolled out in Germany and Turkey later this month with other markets to follow later in the year, Vodafone said in a press release.
According to the company, SuperTOBi “can understand and respond faster to complex customer enquiries better than traditional chatbots.” The new bot will assist customers with various tasks, such as troubleshooting hardware issues and setting up fixed-line routers, the company said.
Vodafone is not about to expose Vodafone’s data to publicly available models like ChatGPT. Nor will the UK based telco create large language models (LLMs) on its own. Instead, a team of 50 data scientists are working on fine-tuning LLMs like Anthropic and Vertex. Vodafone can expose information to those LLMs by dipping into its 24-petabyte data “ocean,” created with Google. Secure containers within public clouds ensure private information is securely cordoned off and unavailable to others.
According to Petty’s estimates, the performance speed of LLMs has improved by a factor of 12 in the last nine months alone, while operational costs have decreased by a factor of six. A telco that invested nine months ago would already have outdated and expensive technology. Petty, moreover, is not the only telco CTO wary of plunging into Nvidia’s GPU chips.
“This is a very weird moment in time where power is very expensive, natural resources are scarce and GPUs are extremely expensive,” said Bruno Zerbib, the CTO of France’s Orange, at the 2024 Mobile World Congress in Barcelona, Spain. “You have to be very careful with your investment because you might buy a GPU product from a famous company right now that has a monopolistic position.”
Petty thinks LLM processing may eventually need to be processed outside hyper-scalers’ facilities. “To really create the performance that we want, we are going to need to push those capabilities further toward the edge of the network,” he said. “It is not going to be the hype cycle of the back end of 2024. But in 2025 and 2026, you’ll start to see those applications and capabilities being deployed at speed.”
“The time it takes for that data to get up and back will dictate whether you’re happy as a consumer to use that interface as your primary interface, and the investment in latency is going to be critically important,” said Petty. “We’re fortunate that 5G standalone drives low latency capability, but it’s not deployed at scale. We don’t have ubiquitous coverage. We need to make sure that those things are available to enable those applications.”
Data from Ericsson supports that view, showing that 5G population coverage is just 70% across Europe, compared with 90% in North America and 95% in China. The figure for midband spectrum – considered a 5G sweet spot that combines decent coverage with high-speed service – is as low as 30% in Europe, against 85% in North America and 95% in China.
Non-standalone (NSA) 5G, which connects a 5G radio access network (RAN) to a 4G core (EPC), is “dominating the market,” said Ericsson.
Vodafone has pledged to spend £11 billion (US$14 billion) on the rollout of a nationwide standalone 5G network in the UK if authorities bless its proposed merger with Three. With more customers, additional spectrum and a bigger footprint, the combined company would be able to generate healthier returns and invest in network improvements, the company said. But a UK merger would not aid the operator in Europe’s four-player markets.
Petty believes a “pay for search” economic model may emerge using GenAI virtual assistants. “This will see an evolution of a two-sided economic model that probably didn’t get in the growth of the Internet in the last 20 years,” but it would not be unlike today’s market for content delivery networks (CDNs).
“Most CDNs are actually paid for by the content distribution companies – the Netflixes, the TV sports – because they want a great experience for their users for the paid content they’ve bought. When it’s free content, maybe the owner of that content is less willing to invest to build out the capabilities in the network.”
Like other industry executives, Petty must hope the debates about net neutrality and fair contribution do not plunge telcos into a long disillusionment trough.
References:
Vodafone CTO: AI will overhaul 5G networks and Internet economics (lightreading.com)
Vodafone UK report touts benefits of 5G SA for Small Biz; cover for proposed merger with Three UK?
Data infrastructure software: picks and shovels for AI; Hyperscaler CAPEX
For many years, data volumes have been accelerating. By 2025, global data volumes are expected to reach 180 zettabytes (1 zettabyte=1 sextillion bytes), up from 120 zettabytes in 2023.
In the age of AI, data is viewed as the currency for large language models (LLMs) and AI–enabled offerings. Therefore, demand for tools to integrate, store and process data is a growing priority amongst enterprises.
The median size of datasets required to train AI models increased from 5.9 million data points in 2010 to 750 billion in 2023, according to BofA Global Research. As demand rises for AI-enabled offerings, companies are prioritizing tools to integrate, store, and process data.
In BofA’s survey, data streaming/stream processing and data science/ML were selected as key use cases in regard to AI, with 44% and 37% of respondents citing usage, respectively. Further, AI enablement is accelerating data to the cloud. Gartner estimates that 74% of the data management market will be deployed in the cloud by 2027, up from 60% in 2023.
Data infrastructure software [1.] represents a top spending priority for the IT department. Survey respondents cite that data infrastructure represents 35% of total IT spending, with budgets expected to grow 9% for the next 12 months. No surprise that the public cloud hyper-scaler platforms were cited as top three vendors. Amazon AWS data warehouse/data lake offerings, Microsoft Azure database offerings, and Google BigQuery are chosen by 45%, 39% and 35% of respondents, respectively.
Note 1. Data infrastructure software refers to databases, data warehouses/lakes, data pipelines, data analytics and other software that facilitate data management, processing and analysis.
………………………………………………………………………………………………………………..
The top three factors for evaluating data infrastructure software vendors are security, enterprise capabilities (e.g., architecture scalability and reliability) and depth of technology.
BofA’s Software team estimates that the data infrastructure industry (e.g., data warehouses, data lakes, unstructured databases, etc.) is currently a $96bn market that could reach $153bn in 2028. The team’s proprietary survey revealed that data infrastructure is 35% of total IT spending with budgets expected to grow 9% over the next 12 months. Hyperscalers including Amazon and Google are among the top recipients of dollars and in-turn, those companies spend big on hardware.
Key takeaways:
- Data infrastructure is the largest and fastest growing segment of software ($96bn per our bottom-up analysis, 17% CAGR).
- AI/cloud represent enduring growth drivers. Data is the currency for LLMs, positioning data vendors well in this new cycle
- BofA survey (150 IT professionals) suggests best of breeds (MDB, SNOW and Databricks) seeing highest expected growth in spend
………………………………………………………………………………………………………….
BofA analyst Justin Post expects server and equipment capex for mega-cap internet companies (Amazon, Alphabet/Google, Meta/Facebook) to rise 43% y/y in 2024 to $145bn, which represents $27bn of the $37bn y/y total capex growth. Despite the spending surge, Mr. Post thinks these companies will keep free cash flow margins stable at 22% y/y before increasing in 2025. The technical infrastructure related capex spend at these three companies is expected to see steep rise in 2024, with the majority of the increase for servers and equipment.
Notes:
- Alphabet categorizes its technical infrastructure assets under the line item ‘Information Technology Assets‘
- Amazon take a much a broader categorization and includes Servers, networking equipment, retail related heavy equipment & fulfillment equipment under ‘Equipment‘.
- Meta gives more details and separately reports Server & Networking, and Equipment assets.
In 2024, BofA estimates CAPEX for the three hyperscalers as follows:
- Alphabet‘s capex for IT assets will increase by $12bn y/y to $28bn.
- Meta, following a big ramp in 2023, server, network and equipment asset spend is expected to increase $7bn y/y to $22bn.
- Amazon, equipment spend is expected to increase $8bn y/y to $41bn (driven by AWS, retail flattish). Amazon will see less relative growth due to retail equipment capex leverage in this line.
On a relative scale, Meta capex spend (% of revenue) remains highest in the group and the company has materially stepped up its AI related capex investments since 2022 (in–house supercomputer, LLM, leading computing power, etc.). We think it‘s interesting that Meta is spending almost as much as the hyperscalers on capex, which should likely lead to some interesting internal AI capabilities, and potential to build a “marketing cloud“ for its advertisers.
From 2016-22, the sector headcount grew 26% on average. In 2023, headcount decreased by 9%. BofA expects just 3% average. annual job growth from 2022-2026. Moreover, AI tools will likely drive higher employee efficiency, helping offset higher depreciation.
…………………………………………………………………………………………………………
Source for all of the above information: BofA Global Research