Introduction:

Part 1. of this two part series examined the November 13th FAA telecommunications outage at length briefly described the November 12th Microsoft global cloud disruption. Although the incidents involved different systems and have not been shown to share a common cause, both raise a fundamental engineering question: why did existing infrastructure and recovery arrangements fail to maintain essential services?

The answer may involve more than insufficient backup capacity. Network resilience depends on the independence of redundant facilities, the ability to detect and isolate failures, automatic restoration, and realistic testing under adverse conditions.

Redundancy Is Not the Same as Resilience:

A network can have multiple connections and still contain single points of failure. Two fiber routes may share a conduit, bridge crossing, central office or power source. Two data centers may depend on the same control plane, identity service, software release or configuration system.  These are examples of common-mode failures: one event or underlying dependency disables resources that were assumed to be independent.

True route diversity requires physically separate paths, diverse points of presence and, where appropriate, different supporting infrastructure. Cloud resilience similarly requires more than distributing servers across locations. Critical dependencies and management systems must also be considered.

The key question is not how many redundant components an operator owns, but whether those components can continue providing service when the primary system or a shared dependency fails.

A Telecommunications Case Study – Comcast’s Santa Clara, CA Outage:

A July 2024 IEEE Techblog article documented a prolonged Comcast outage in Santa Clara, CA, associated with a physical fiber cut and reported vandalism. Service was unavailable for approximately 17 hours. That incident, which negatively impacted this author, raised a basic question: why wasn’t traffic rerouted automatically around the damaged fiber?

Telecommunications networks have several established protection and restoration approaches. These include 1+1 protection, 1 protection, self-healing rings and mesh-network restoration. Depending on network design, these mechanisms can switch traffic to an alternate path when a link or facility fails.  However, protection mechanisms are effective only when alternate paths are genuinely diverse, sufficient capacity is available, and switching and restoration procedures work as intended. A physically diverse route that shares a vulnerable conduit or aggregation point may provide little real protection.

–>That Comcast outage illustrates why network operators should document not only their redundancy architecture but also the actual restoration performance experienced during outages.

Image Generated by ChatGPT

Resiliency Recommendation for Network Operators:

Telecommunications providers, cloud operators and organizations supporting critical infrastructure should address several priorities:

1. Eliminate common-mode failures. Identify shared physical routes, power systems, facilities, software dependencies and control-plane components that could disable supposedly redundant resources.
2. Provide independent alternate paths. Ensure that backup connectivity is physically and logically diverse, adequately provisioned and available during peak demand.
3. Test automatic failover. Verify that traffic switches successfully under realistic fault conditions, including fiber cuts, equipment failures, power loss, configuration errors and loss of control-plane functions.
4. Test recovery under load. A backup system that works in isolation may fail when it suddenly receives the traffic of an entire failed region. Capacity planning must account for this scenario.
5. Improve fault isolation and graceful degradation. A localized failure should not cascade across unrelated services. Architectures should limit failure domains and preserve essential functions when full service cannot be maintained.
6. Exercise disaster-recovery procedures. Regularly test recovery plans, including manual intervention, alternate communications, restoration sequencing and coordination among network, cloud and customer operations teams.

Accountability and Operational Transparency:

For critical infrastructure, operators should provide meaningful post-incident reports explaining the failure, its scope, the role of backup systems, why automatic recovery was insufficient, and what corrective actions will prevent recurrence.

Regulators and public-sector agencies should also evaluate resilience based on demonstrated performance—not simply on the existence of documented backup plans or redundant equipment.

The FAA outage is particularly consequential because communications failures can affect the safe operation of the national airspace system. The Microsoft disruption demonstrates how dependence on centralized cloud services can affect organizations across industries and geographic regions.

Conclusions:

Redundancy is a design feature.  It’s an operational result which must be proven to prevent the next network outage from being catastrophic.  Multiple servers, fiber routes, data centers and backup systems provide little assurance unless they are sufficiently independent, have adequate capacity and can maintain essential services when primary resources fail.

The ultimate test of a resilient network is not whether it has backup infrastructure, but whether that infrastructure actually works when needed. Operators must demonstrate this through rigorous failure testing, realistic disaster-recovery exercises, transparent post-incident analysis and measurable improvements in service continuity.

The FAA and Microsoft incidents—and the earlier Comcast outage—should prompt a broader reassessment of whether critical network and cloud infrastructure is as resilient in operation as its architecture claims.

…………………………………………………………………………………………………………………………………………………………………………………………….

References:

https://www.cisa.gov/sites/default/files/publications/07202017_10_Keys_to_Public_Safety_Network_Resiliency_010418_FINAL508C.pdf

https://www.itu.int/rec/dologin_pub.asp?id=T-REC-L.Sup35-201706-I%21%21PDF-E&lang=e&type=items

https://www.itu.int/rec/dologin_pub.asp?id=T-REC-L.Sup25-201604-I%21%21PDF-E&lang=s&type=items

IETF RFC 9522 – Overview and Principles of Internet Traffic Engineering (2024)
https://www.rfc-editor.org/rfc/rfc9522.html

IETF RFC 9855 – Topology Independent Fast Reroute Using Segment Routing (October 2025)
https://www.rfc-editor.org/rfc/rfc9855.html

NIST SP 800-34 Rev. 1 – Contingency Planning Guide for Federal Information Systems
https://csrc.nist.gov/pubs/sp/800/34/r1/upd1/final

IETF – Problem Statement for Network Resilience (November 2, 2026)
https://datatracker.ietf.org/doc/draft-zhao-opsawg-network-resilience-ps/

Critical Network Outages: Why Isn’t Redundancy Working? Part I.

Telecom network outages: causes, effects, and remedies for telecom providers & IT enterprise

Comcast frequent, intermittent internet outages + long outage in Santa Clara, CA with no auto-recovery!

Ookla: AI platform reliability decreases as outages surge