Telecom network outages
Critical Network Outages: Why Isn’t Redundancy Working? Part II.
Introduction:
Part 1. of this two part series examined the November 13th FAA telecommunications outage at length & briefly described the November 12th Microsoft global cloud disruption. Although the incidents involved different systems and have not been shown to share a common cause, both raise a fundamental engineering question:
Why did existing infrastructure and recovery arrangements fail to maintain essential services?
The answer may involve more than insufficient backup capacity. Network resilience depends on the independence of redundant facilities, the ability to detect and isolate failures, automatic restoration, and realistic testing under adverse conditions.
Redundancy Is Not the Same as Resilience:
A network can have multiple connections and still contain single points of failure. Two fiber routes may share a conduit, bridge crossing, central office or power source. Two data centers may depend on the same control plane, identity service, software release or configuration system. These are examples of common-mode failures: one event or underlying dependency disables resources that were assumed to be independent.
True route diversity requires physically separate paths, diverse points of presence and, where appropriate, different supporting infrastructure. Cloud resilience similarly requires more than distributing servers across locations. Critical dependencies and management systems must also be considered.
The key question is not how many redundant components an operator owns, but whether those components can continue providing service when the primary system or a shared dependency fails.
A Telecommunications Case Study – Comcast’s Santa Clara, CA Outage:
A July 2024 IEEE Techblog article documented a prolonged Comcast outage in Santa Clara, CA, associated with a physical fiber cut and reported vandalism. Service was unavailable for approximately 17 hours. That incident, which negatively impacted this author, raised a basic question: why wasn’t traffic rerouted automatically around the damaged fiber?
Telecommunications networks have several established protection and restoration approaches. These include 1+1 protection, 1 protection, self-healing rings and mesh-network restoration. Depending on network design, these mechanisms can switch traffic to an alternate path when a link or facility fails. However, protection mechanisms are effective only when alternate paths are genuinely diverse, sufficient capacity is available, and switching and restoration procedures work as intended. A physically diverse route that shares a vulnerable conduit or aggregation point may provide little real protection.
–>That Comcast outage illustrates why network operators should document not only their redundancy architecture but also the actual restoration performance experienced during outages.
……………………………………………………………………………………………………………………………………………………………………………..
Image Generated by ChatGPT
………………………………………………………………………………………………………………………………………………………………….
Resiliency Recommendation for Network Operators:
Telecommunications providers, cloud operators and organizations supporting critical infrastructure should address several priorities:
- Eliminate common-mode failures. Identify shared physical routes, power systems, facilities, software dependencies and control-plane components that could disable supposedly redundant resources.
- Provide independent alternate paths. Ensure that backup connectivity is physically and logically diverse, adequately provisioned and available during peak demand.
- Test automatic failover. Verify that traffic switches successfully under realistic fault conditions, including fiber cuts, equipment failures, power loss, configuration errors and loss of control-plane functions.
- Test recovery under load. A backup system that works in isolation may fail when it suddenly receives the traffic of an entire failed region. Capacity planning must account for this scenario.
- Improve fault isolation and graceful degradation. A localized failure should not cascade across unrelated services. Architectures should limit failure domains and preserve essential functions when full service cannot be maintained.
- Exercise disaster-recovery procedures. Regularly test recovery plans, including manual intervention, alternate communications, restoration sequencing and coordination among network, cloud and customer operations teams.
Accountability and Operational Transparency:
For critical infrastructure, operators should provide meaningful post-incident reports explaining the failure, its scope, the role of backup systems, why automatic recovery was insufficient, and what corrective actions will prevent recurrence.
Regulators and public-sector agencies should also evaluate resilience based on demonstrated performance—not simply on the existence of documented backup plans or redundant equipment.
The FAA outage is particularly consequential because communications failures can affect the safe operation of the national airspace system. The Microsoft disruption demonstrates how dependence on centralized cloud services can affect organizations across industries and geographic regions.
Conclusions:
Redundancy is a design feature. It’s an operational result which must be proven to prevent the next network outage from being catastrophic. Multiple servers, fiber routes, data centers and backup systems provide little assurance unless they are sufficiently independent, have adequate capacity and can maintain essential services when primary resources fail.
The ultimate test of a resilient network is not whether it has backup infrastructure, but whether that infrastructure actually works when needed. Operators must demonstrate this through rigorous failure testing, realistic disaster-recovery exercises, transparent post-incident analysis and measurable improvements in service continuity.
The FAA and Microsoft incidents—and the earlier Comcast outage—should prompt a broader reassessment of whether critical network and cloud infrastructure is as resilient in operation as its architecture claims.
End Quote:
“When people hear ‘copper to fiber,’ upgrade as the backstory to the FAA ground stop, they picture that ‘copper’ as Ethernet cable. But Ethernet over copper doesn’t go past 100 meters, so it never leaves the building. A copper line connecting a remote site over miles is old telecom infrastructure, likely a T1 or something similar, and the equipment on either end is just as old. The risk was never the old copper sitting there working. It’s the moment you connect modern gear to a system that’s been frozen in time since the 1980s.”
— Thomas Doumas, VP of Product, NetAlly
…………………………………………………………………………………………………………………………………………………………………………………
References:
https://www.itu.int/rec/dologin_pub.asp?id=T-REC-L.Sup35-201706-I%21%21PDF-E&lang=e&type=items
https://www.itu.int/rec/dologin_pub.asp?id=T-REC-L.Sup25-201604-I%21%21PDF-E&lang=s&type=items
IETF RFC 9522 – Overview and Principles of Internet Traffic Engineering (2024)
https://www.rfc-editor.org/rfc/rfc9522.html
IETF RFC 9855 – Topology Independent Fast Reroute Using Segment Routing (October 2025)
https://www.rfc-editor.org/rfc/rfc9855.html
NIST SP 800-34 Rev. 1 – Contingency Planning Guide for Federal Information Systems
https://csrc.nist.gov/pubs/sp/800/34/r1/upd1/final
IETF – Problem Statement for Network Resilience (November 2, 2026)
https://datatracker.ietf.org/doc/draft-zhao-opsawg-network-resilience-ps/
Critical Network Outages: Why Isn’t Redundancy Working? Part I.
Telecom network outages: causes, effects, and remedies for telecom providers & IT enterprise
Comcast frequent, intermittent internet outages + long outage in Santa Clara, CA with no auto-recovery!
Ookla: AI platform reliability decreases as outages surge
Critical Network Outages: Why Isn’t Redundancy Working? Part I.
Introduction:
Today’s Federal Aviation Administration (FAA) nationwide U.S. flight departure halt, attributed to an unspecified “telecommunications problem,” Microsoft’s global cloud disruption yesterday, and other prolonged telecommunications outages highlight serious questions about the resilience of critical network infrastructure, the effectiveness of automatic failover and service restoration, and the adequacy of disaster-recovery capabilities.
These incidents involved different systems and should not be assumed to share a common technical cause. However, both illustrate the consequences when essential network services become unavailable and recovery is neither immediate nor seamless.
FAA – Telecommunications Failure Disrupts the U.S. National Airspace System:
On November 13, 2026, the FAA ordered a nationwide ground stop for all departing flights following a telecommunications outage affecting air traffic control operations. Transportation Secretary Sean Duffy said the communications issue was impacting all 22 U.S. Air Route Traffic Control Centers (ARTCCs), and FAA technicians were investigating. The FAA’s preliminary statement described the problem as a temporary telecommunications outage and said the ground stop was imposed for safety reasons.
The 22 ARTCCs are a critical part of the U.S. National Airspace System. Unlike airport control towers, which manage aircraft movements on runways and in the immediate vicinity of airports, ARTCCs manage aircraft during the en route portion of their flights, primarily at higher altitudes between airports. They coordinate aircraft separation, traffic flows, handoffs between control sectors and the safe movement of aircraft through the national airspace system. A communications failure affecting these centers can have consequences well beyond the geographic location where the original problem occurred. ARTCCs depend on reliable communications with aircraft, other control facilities and supporting systems to manage traffic safely. If controllers cannot reliably communicate with aircraft or coordinate traffic handoffs, the FAA may have to reduce traffic volumes or suspend departures to maintain safe operating conditions.
The disruption was substantial. According to FlightAware, more than 1,000 flights were canceled and nearly 10,000 delayed. Reuters reported that 4,042 flights were airborne and destined for U.S. airports when the FAA issued the ground stop. The disruption also affected international operations, forcing cancellations of some flights originating in Canada and Europe.
FAA Flight Control Center
…………………………………………………………………………………………………………………………………………..
The nationwide ground stop prevented additional aircraft from departing while the FAA investigated the problem and worked to restore communications. Thousands of aircraft already airborne were allowed to land, while flights still on the ground could not take off. Inbound international flights were to be handled on a case-by-case basis. The FAA was not ordering aircraft already in the air to land immediately; rather, it was restricting new departures to avoid adding traffic to an airspace system experiencing serious communications problems.
The FAA’s action was only the second nationwide ground stop of departing flights since the September 11, 2001 attacks. The previous occurrence was in January 2023, when the FAA halted domestic departures for less than two hours following a failure of its NOTAM (Notice to Air Missions) system. That incident involved a different system and failure scenario and should not be confused with the November 2026 telecommunications outage.
The Critical Question: Why Didn’t Redundancy Prevent a Nationwide Disruption?
The central engineering issue is not simply why the telecommunications system failed, but why the failure resulted in a nationwide ground stop. The incident raises several important questions:
- What telecommunications infrastructure and services were affected, and how did the problem propagate across the ARTCC network?
- Were independent primary and alternate communications routes available to the affected centers?
- Did the outage involve a common network, switching facility, software platform, power source or other shared dependency?
- What backup communications systems and failover procedures were in place, and why were they unable to maintain sufficient service to avoid a nationwide suspension of departures?
- Were alternate facilities or operating procedures available to isolate the failure and maintain at least limited air traffic operations?
These questions cannot be answered conclusively until the FAA provides a detailed technical explanation. As of the initial reports, the precise cause had not been established publicly. Homeland Security Secretary Markwayne Mullin said there was no indication at that time of a malicious attack, while emphasizing that the cause remained under investigation. The possibility of a problem involving an update to the private communications network supporting FAA operations was raised in subsequent reporting, but should be treated as preliminary until confirmed by the FAA and its contractors.
FAA Modernization: Acknowledged Problems, but Resilience Remains Essential
The November 13 incident occurred as the FAA is undertaking a major modernization of its aging air traffic control infrastructure. The Brand New Air Traffic Control System (BNATCS) initiative includes replacing legacy telecommunications infrastructure, modernizing voice switches and radio systems, and upgrading other critical air traffic control technologies.
A September 2026 Government Accountability Office (GAO) report, Air Traffic Control Systems: Ambitious New Modernization Effort Needs to Improve Cost and Schedule Planning (GAO-26-107992), describes both the scale of this effort and the challenges facing the FAA. By May 2026, the agency had replaced 2,560 of 5,170 aging copper communications wires with high-speed fiber-optic cables—approximately half of the planned replacements. The report also identifies significant cost-estimation and project-scheduling weaknesses that could delay modernization and prolong reliance on outdated systems.
Modernization is clearly necessary, but replacing aging copper lines with fiber and upgrading equipment are not sufficient by themselves to guarantee network resilience. The architecture must also provide physically and logically diverse communications paths, independent backup facilities, automatic failover, effective fault isolation and tested disaster-recovery procedures.
The FAA’s previously implemented backup arrangements for its NOTAM system should not be confused with backup communications for the ARTCC network. They address different systems and failure scenarios. The November 13 incident requires a separate examination of the communications architecture, route diversity, failover procedures and operational readiness supporting all 22 centers.
Ultimately, the question is whether the FAA’s communications infrastructure was engineered to withstand the failure of individual links, facilities or shared network components—and, if so, why those protections were insufficient to prevent a nationwide ground stop.
As of the initial reports, the precise technical cause had not been established publicly. Homeland Security Secretary Markwayne Mullin said there was no indication at that time of a malicious attack, while emphasizing that the cause was still under investigation. Subsequent reporting raised the possibility of a problem involving an update to the private communications network supporting FAA operations, but that explanation should be treated as preliminary until confirmed by the FAA and its contractors.
The FAA has been undertaking a major modernization of its aging air traffic control infrastructure, including replacing older copper communications lines with fiber and upgrading radio systems. The November 13 incident underscores why modernization must address not just equipment age and capacity, but also network architecture, physical and logical route diversity, automatic failover, fault isolation and disaster recovery.
The FAA has been working to modernize aging air traffic control infrastructure. A September 2026 Government Accountability Office report, Air Traffic Control Systems: Ambitious New Modernization Effort Needs to Improve Cost and Schedule Planning (GAO-26-107992), describes the scale of the modernization challenge and the risks of continued reliance on outdated systems.
The FAA’s previously implemented backup arrangements for its NOTAM system should not be confused with backup communications for the regional air traffic control centers. They address different systems and failure scenarios. The November 13th incident requires a separate examination of communications architecture, route diversity, failover procedures and operational readiness.
Microsoft: A Global Cloud-Service Disruption:
A day earlier, on November 12, Microsoft experienced a widespread disruption affecting Microsoft 365 services, including Exchange Online, Outlook and other applications dependent on its cloud infrastructure. Microsoft reported that a significant number of servers had become unexpectedly unavailable. The remaining infrastructure experienced increased load, complicating recovery as affected servers were returned to service. The disruption continued across regions and affected businesses, government organizations and other customers. This raises a different but equally important question: how could a cloud platform operating at Microsoft’s scale lose enough server capacity to cause prolonged disruption across multiple services and regions?
Cloud infrastructure is designed to distribute workloads across servers, availability zones and geographic regions. But distributing workloads does not automatically guarantee resilience. Shared software, configuration changes, dependencies, authentication services, traffic-management systems or common operational procedures can create failure domains that extend well beyond an individual server or data center. Microsoft’s incident also demonstrates that restoring infrastructure is not necessarily the same as restoring service. Recovery must account for traffic redistribution, overloaded surviving systems, dependency failures and the possibility that returning capacity to service may itself destabilize the environment.
The Common Question: Where Was the Effective Backup?
The FAA and Microsoft incidents are not evidence of the same underlying failure. They do, however, expose a common issue: the difference between having redundant infrastructure and maintaining uninterrupted service when something fails. A network may have multiple links, servers, data centers or geographic locations and still be vulnerable if those resources share a common failure point. Backup capacity is useful only if it is sufficiently independent, operationally available and capable of taking over under real-world failure conditions.
…………………………………………………………………………………………………………………………………………………………………………………………….
Part 2 will examine why redundancy can fail, the engineering principles required for genuine network resilience, and what telecommunications providers, cloud operators and critical-infrastructure organizations should do to improve service continuity.
…………………………………………………………………………………………………………………………………………………………………………………………….
References:
https://www.faa.gov/newsroom/statements/general-statements
https://www.gao.gov/products/gao-26-107992
Telecom network outages: causes, effects, and remedies for telecom providers & IT enterprise
Ookla: AI platform reliability decreases as outages surge
Telecom network outages: causes, effects, and remedies for telecom providers & IT enterprise
Network outages, historically caused by misconfigurations, software defects, or hardware failures, are increasingly disruptive for several reasons, such as hyper-connectivity, single points of failure and over-reliance on concentrated hyperscaler cloud infrastructures. This leads to an expanded “blast radius” from single points of failure. The latest Cloudflare outage reveals that enterprises heavily reliant on a dangerously few major IT providers face critical single points of failure, leading to authentication issues, lost revenue, and broken customer experiences.
Cloudflare is a global cloud services and cybersecurity firm. It provides data centers, website and email security, protection from data loss and defences against cyber threats, among other things. It describes itself as providing an “immune system for the internet”, with technology that sits between its clients and the wider world that blocks billions of cyber threats daily. It also uses its global infrastructure to speed up internet traffic. It makes more than $500m – a quarter from nearly 300,000 customers operating in 125 countries, including China. Users of several heavy-traffic websites reported that they went offline at the same time as the Cloudflare outage.
Akamai’s Reuben Koh advocates for a distributed compute and edge architecture, which acts as autonomous cells to mitigate systemic risk and improve resilience via graceful degradation. He also suggests adopting strategies like graceful degradation and diversifying cloud providers which help telecom operators and other organizations limit the spread of outage disruption.
Computer outage, error or failure causing by software update mistake, operating system crash or cyber attack, server down or technical issue concept, people victims looking at computer laptop outage.

- Systemic Risk Assessment: Regulations, such as the EU’s Digital Operational Resilience Act (DORA) which is effective from January 17, 2025, are moving from assessing a single firm’s risk to evaluating the broader market impact of a critical third-party provider failure. DORA specifically designates critical ICT third-party providers subject to direct oversight.
- Operational Resilience Mandates: Jurisdictions are pushing firms to demonstrate the ability to maintain operations or safely exit a non-performing CSP relationship. This includes requirements for robust contingency and exit plans.
- Geographic Examples:
–Singapore is framing cloud infrastructure as essential national computing, issuing specific resilience guidelines.–Australia has issued warnings to financial institutions regarding over-dependence on a narrow set of US-based hyperscalers.–Japan is tightening scrutiny and expectations around managing third-party cloud risks.
- Continuous Discovery & Inventory: Telecom operators must maintain an up-to-date, comprehensive inventory of all APIs (managed, unmanaged, “shadow,” and “zombie”) across the enterprise.
- Shift-Left Security: Integrate security testing and design principles early into the software development lifecycle to identify and remediate vulnerabilities before APIs reach production environments.
- Implement Zero Trust Architecture (ZTA): Adopt a “never trust, always verify” approach, assuming an attacker may already be internal. This means applying strict authentication and authorization controls at the API level, not just the network perimeter.
- Strong Authentication and Authorization: Use robust mechanisms like OAuth 2.0 and OpenID. Connect, employing the principle of least privilege to ensure entities only have the minimum necessary access.
- Runtime Protection and Monitoring: Implement API gateways for centralized traffic management, rate limiting to prevent Denial-of-Service (DoS) attacks, and use behavioral analytics to detect anomalous activity indicative of abuse.
- Input Validation and Data Handling: Strictly validate and sanitize all data inputs to prevent injection attacks, and ensure APIs only expose necessary information to minimize data leakage.
- Human Oversight in AI: As AI and automation increase, maintain robust human oversight in change management and incident response, as AI systems can behave unpredictably. Telecom staff should be closely involved in change management and incident response, even as network automation increases.
Note 1. The UK’s Telecommunications Security Act – 2021 is a landmark law establishing mandatory, tough security standards for public telecom networks, making cybersecurity a legal duty for providers to protect critical infrastructure. It empowers regulator Ofcom, introduces penalties for non-compliance (up to 10% of turnover), and mandates adherence to specific security measures in the Code of Practice (CoP) through phased deadlines, requiring strong governance, supply chain security, and proactive threat management.
……………………………………………………………………………………………………………………………………………………………….
Conclusions:
Koh advocates for the implementation of resilient network architectures and improved operational maturity to enhance system fault tolerance. Key steps include distributed design, optimized operational protocols, comprehensive network visibility, and pragmatic capacity planning. These measures are becoming increasingly important as telecommunications infrastructure underpins essential societal functions.
………………………………………………………………………………………………………………………………..
References:
What telecom operators can learn from recent network outages
Cloudflare outage highlights enterprise infrastructure dependence
https://www.techtarget.com/whatis/feature/8-largest-IT-outages-in-history
España hit with major telecom blackout after power outage April 28th
Comcast frequent, intermittent internet outages + long outage in Santa Clara, CA with no auto-recovery!
AT&T wireless outage effected more than 74,000 U.S. customers with service disruptions lasting up to 11 hours for some
Rogers Telecommunications restores service after 19 hour outage disrupting life in Canada
GSMA, ETSI, IEEE, ITU & TM Forum: AI Telco Troubleshooting Challenge + TelecomGPT: a dedicated LLM for telecom applications
Rogers Telecommunications restores service after 19 hour outage disrupting life in Canada



