Month: September 2025
Hyperscaler design of networking equipment with ODM partners
Networking equipment for hyperscalers like Google, Amazon, Microsoft, Oracle, Meta, and others is a mix of in‑house engineering and partnerships with specialized vendors. These companies operate at such massive scale to design their own switches, routers, and interconnects — but rely on Original Design Manufacturers (ODMs) and network silicon providers to build them.
In‑House Networking Design:
Hyperscalers have dedicated hardware teams that create custom network gear to meet their unique performance, latency, and power‑efficiency needs.
- Google – Designs its own Jupiter and Andromeda data center network fabrics, plus custom top‑of‑rack (ToR) and spine switches. Uses merchant silicon from Broadcom, Intel (Barefoot Tofino), and others, but with Google‑built control planes and software.
- Amazon (AWS) – Builds custom switches and routers for its Scalable Reliable Datagram (SRD) and Elastic Fabric Adapter (EFA) HPC networks. Uses in‑house firmware and network operating systems, often on ODM‑built hardware.
- Microsoft (Azure) – Designs OCP‑compliant switches (e.g., SONiC network OS) and contributes to the Open Compute Project. Uses merchant silicon from Broadcom, Marvell, and Mellanox/NVIDIA.
- Oracle Cloud Infrastructure (OCI) – Designs its own high‑performance RDMA‑enabled network for HPC and AI workloads, with custom switches built by ODM partners.
- Meta – Designs Wedge, Backpack, and Minipack switches under OCP, manufactured by ODMs.
Manufacturing & ODM Partners:
While the hyperscaler’s network equipment designs are proprietary, the physical manufacturing is typically outsourced to ODMs who specialize in hyperscale networking gear:
| ODM / OEM | Builds for | Notes |
|---|---|---|
| Quanta Cloud Technology (QCT) | AWS, Azure, Oracle, Meta | Custom ToR/spine switches, OCP gear |
| WiWynn | Microsoft, Meta | OCP‑compliant switches and racks |
| Celestica | Multiple hyperscalers | High‑end switches, optical modules |
| Accton / Edgecore | Google, Meta, others | White‑box switches for OCP |
| Foxconn / Hon Hai | AWS, Google | Large‑scale manufacturing |
| Delta Networks | Multiple CSPs | Optical and Ethernet gear |
Network Silicon & Optics Suppliers:
Even though most hyperscalers design the chassis and racks, they often use merchant silicon and optics from:
- Broadcom – Tomahawk, Trident, Jericho switch ASICs
- Marvell – Prestera switch chips, OCTEON DPUs
- NVIDIA (Mellanox acquisition) – Spectrum Ethernet, InfiniBand for AI/HPC
- Intel (Barefoot acquisition) – Tofino programmable switch ASICs
- Cisco Silicon One – Used selectively in hyperscale builds
- Coherent optics & transceivers – From II‑VI (Coherent), Lumentum, InnoLight, etc.
Hyperscaler Networking Supply Chain Map:
| Layer | Key Players | Role | Example Hyperscaler Relationships |
|---|---|---|---|
| Network Silicon (ASICs / DPUs) | Broadcom (Tomahawk, Jericho), Marvell (Prestera, OCTEON), NVIDIA/Mellanox (Spectrum, InfiniBand), Intel (Barefoot Tofino), Cisco (Silicon One) | Core packet switching, programmability, congestion control | Google (Broadcom, Intel), AWS (Broadcom, Marvell), Microsoft (Broadcom, Marvell, NVIDIA), Oracle (Broadcom, NVIDIA) |
| Optics & Interconnects | Coherent (II‑VI), Lumentum, InnoLight, Source Photonics, Broadcom (optical PHYs) | 400G/800G transceivers, co‑packaged optics, DWDM modules | All hyperscalers source from multiple vendors for redundancy |
| ODM / Manufacturing | Quanta Cloud Technology (QCT), WiWynn, Celestica, Accton/Edgecore, Foxconn, Delta Networks | Build hyperscaler‑designed switches, routers, and chassis | AWS (QCT, Foxconn), Google (Accton, QCT), Microsoft (WiWynn, Celestica), Meta (Accton, WiWynn), Oracle (QCT, Celestica) |
| Network OS & Control Plane | In‑house NOS (Google proprietary, AWS custom OS, Microsoft SONiC, Oracle custom), OCP software | Routing, telemetry, automation, SDN control | Google (Jupiter fabric OS), AWS (custom SRD/EFA stack), Microsoft (SONiC), Oracle (OCI NOS) |
| Integration & Deployment | Hyperscaler internal engineering teams | Rack integration, cabling, fabric topology, automation pipelines | All hyperscalers do this in‑house for security and scale |
Design Flow:
- Chip Vendors → supply merchant silicon to ODMs or directly to hyperscaler design teams.
- Hyperscaler Hardware Teams → design chassis, PCB layouts, thermal systems, and specify optics.
- ODMs → manufacture to spec, often in Asia, with hyperscaler QA oversight.
- Optics Vendors → deliver transceivers and cables, often qualified by hyperscaler labs.
- In‑House NOS → loaded onto hardware, integrated into hyperscaler’s SDN fabric.
- Deployment → rolled out in data centers globally, often in multi‑tier Clos or AI‑optimized topologies.
Major Trends:
- Disaggregation – Hyperscalers separate hardware from software, running their own Network Operating System (NOS), (e.g., SONiC, Google’s proprietary OS) on ODM‑built “white‑box” or “bare metal” switches.
- AI‑Optimized Fabrics – New designs focus on ultra‑low latency, congestion control, and massive east‑west bandwidth for GPU clusters.
- Optical Integration – Co‑packaged optics and 800G+ transceivers are becoming standard for AI and HPC workloads.
- AI Cluster Networking – NVIDIA InfiniBand and 800G Ethernet fabrics are now common for GPU pods.
- Co‑Packaged Optics – Moving optics closer to the ASIC to reduce power and latency.
- Open Compute Project Influence – Many designs are OCP‑compliant but with proprietary tweaks.
- Multi‑Vendor Strategy – Hyperscalers dual‑source ASICs and optics to avoid supply chain risk.
References:
How it works: hyperscaler compute server in house design process with ODM partners
Cloud‑resident high performance compute servers used by hyperscale cloud providers like Amazon Web Services (AWS), Google Cloud Platform (GCP), Microsoft Azure, Oracle Cloud Infrastructure (OCI), Meta, and others use a mix of custom in‑house designs and ODM (Original Design Manufacturer) ‑ built hardware.
In‑House Design Teams:
- Amazon (AWS) – Designs its own Nitro System–based servers, including custom motherboards, networking cards, and security chips. AWS also develops Graviton (Arm‑based) and Trainium/Inferentia (AI) processors. HPC instances use Elastic Fabric Adapter (EFA) for low‑latency interconnects.
- Google (GCP) – Builds custom server boards and racks for its data centers, plus TPUs (Tensor Processing Units) for AI workloads. GCP builds custom HPC server boards and racks, plus TPUs for AI workloads. It uses high‑speed interconnects like Google’s Jupiter network for HPC clusters.
- Microsoft Azure – Designs Azure‑optimized servers and AI accelerators, often in collaboration with partners, and contributes designs to the Open Compute Project (OCP). It integrates InfiniBand and/or 400 Gbps Ethernet for HPC interconnects.
- Oracle – Designs bare‑metal HPC shapes with AMD EPYC, Intel Xeon, and NVIDIA GPUs, plus RDMA cluster networking for microsecond latency.
- Meta – Designs its compute servers, especially for AI workloads, by working closely with ODM partners like Quanta Computer, Wiwynn, and Foxconn.
Manufacturing Partners (ODMs/OEMs):
While the hyperscaler compute server designs are proprietary, the physical manufacturing is typically outsourced to Original Design Manufacturers (ODMs) who specialize in hyperscale data center gear as per these tables:
| ODM / OEM | Known for | Cloud Customers |
|---|---|---|
| Quanta Cloud Technology (QCT) | Custom rack servers, HPC nodes | AWS, Azure, Oracle |
| WiWynn | OCP‑compliant HPC servers | Microsoft, Meta |
| Inventec | HPC and AI‑optimized servers | AWS, GCP |
| Foxconn / Hon Hai | Large‑scale server manufacturing | Google, AWS |
| Celestica | HPC and networking gear | Multiple hyperscalers |
| Supermicro | GPU‑dense HPC systems | AWS, Oracle, Azure |
| ODM / OEM | Role in Hyperscale Cloud |
|---|---|
| Quanta Cloud Technology (QCT) | Major supplier for AWS, Azure, and others; builds custom rack servers and storage nodes. |
| WiWynn | Spun off from Wistron; manufactures OCP‑compliant servers for Microsoft and Facebook/Meta. |
| Inventec | Supplies compute and storage servers for AWS and other CSPs. |
| Foxconn / Hon Hai | Builds cloud server hardware for multiple providers, including Google. |
| Delta / Celestica | Provides specialized server and networking gear for hyperscale data centers. |
| Supermicro | Supplies both standard and custom AI‑optimized servers to cloud and enterprise customers. |
The global server market expected to reach $380 billion by 2028. Image credit: Alamy
……………………………………………………………………………………………………………………………………………………………..
Here’s a supply chain relationship map for cloud‑resident high‑performance compute (HPC) servers used by the major hyperscalers:
Hyperscale HPC Server Design & Manufacturing Landscape:
| Cloud Provider | In‑House Design Focus | Key Manufacturing / ODM Partners | Notable HPC Hardware Features |
|---|---|---|---|
| Amazon Web Services (AWS) | Custom Nitro boards, Graviton CPUs, Trainium/Inferentia AI chips, EFA networking | Quanta Cloud Technology (QCT), Inventec, Foxconn | Arm‑based HPC nodes, GPU clusters (NVIDIA H100/A100), ultra‑low‑latency RDMA |
| Google Cloud Platform (GCP) | Custom server boards, TPU accelerators, Jupiter network fabric | Quanta, Inventec, Foxconn | TPU pods, GPU supernodes, liquid‑cooled racks |
| Microsoft Azure | OCP‑compliant HPC designs, Maia AI chip, Cobalt CPU, InfiniBand networking | WiWynn, QCT, Celestica | Cray‑based HPC clusters, GPU/FPGA acceleration |
| Oracle Cloud Infrastructure (OCI) | Bare‑metal HPC shapes, RDMA cluster networking | QCT, Supermicro | AMD EPYC/Intel Xeon nodes, NVIDIA GPU dense racks |
| Meta (for AI/HPC research) | OCP‑based AI/HPC servers | WiWynn, QCT | AI Research SuperCluster, liquid cooling |
| Alibaba Cloud / Tencent Cloud | Custom AI/HPC boards, Arm CPUs | Inspur, Sugon, QCT | GPU/FPGA acceleration, high‑bandwidth fabrics |
Meta’s ODM Collaboration Model:
- Quanta Computer: Meta has partnered with Quanta for final assembly of its next-gen AI servers. Quanta is responsible for building up to 6,000 racks of the Santa Barbara servers, which feature advanced cooling and power delivery systems.
- Wiwynn & Foxconn: These ODMs also play key roles in Meta’s infrastructure. Wiwynn reportedly earns more than half its revenue from Meta, while Foxconn handles system assembly for NVIDIA’s NVL 72 servers, which Meta may also utilize.
- Broadcom Partnership: For chip supply, Meta collaborates with Broadcom to integrate custom ASICs into its server designs.
Hyperscaler/ODM Collaboration Process:
- Design Phase – Hyperscalers’ hardware teams define the architecture: CPU/GPU choice, interconnect, cooling, power density.
- ODM Manufacturing – Partners like Quanta, WiWynn, Inventec, Foxconn, Celestica, and Supermicro build the servers to spec.
- Integration & Deployment – Systems are tested, integrated into racks, and deployed in hyperscale data centers.
- Optimization – Providers fine‑tune firmware, drivers, and orchestration for HPC workloads (e.g., CFD, genomics, AI training).
Industry Trends:
- Open Compute Project (OCP) – Many designs are shared in the OCP community, allowing ODMs to build interoperable, cost‑optimized hardware at scale. Open Compute Project designs speed up deployment and interoperability.
- Vertical Integration – Hyperscalers increasingly design custom silicon (e.g., AWS Graviton, Google TPU, Microsoft Maia AI chip) to optimize performance and reduce dependency on third‑party CPUs/GPUs.
- AI‑Optimized Racks – New designs focus on high‑density GPU clusters, liquid cooling, and ultra‑low‑latency networking for AI workloads.
- Vertical integration: More custom silicon to optimize performance and cost. See Specialized HPC components below.
- Liquid cooling: Increasingly common for dense GPU/CPU HPC racks.
Specialized HPC Components:
- CPUs – AMD EPYC, Intel Xeon Scalable, AWS Graviton (Arm), custom Google CPUs.
- GPUs / Accelerators – NVIDIA H100/A100, AMD Instinct, Google TPU, AWS Trainium.
- Networking – Mellanox/NVIDIA InfiniBand, AWS EFA, Oracle RDMA cluster networking.
- Storage – Parallel file systems like Lustre, BeeGFS, IBM Spectrum Scale for HPC workloads
References:
ODM Sales Soar as Hyperscalers and Cloud Providers Go Direct
The future of US hyperscale data centers | McKinsey
100MW+ Wholesale Colocation Deals: Inside the Hyperscaler Surge
Hyperscaler design of networking equipment with ODM partners – IEEE ComSoc Technology Blog
Liquid Dreams: The Rise of Immersion Cooling and Underwater Data Centers
