AT&T wireless outage effected more than 74,000 U.S. customers with service disruptions lasting up to 11 hours for some
AT&T’s cellular network went down for many of its customers across the United States Thursday morning, leaving its wireless customers unable to place calls, text or access the internet. Thursday morning, more than 74,000 AT&T customers reported outages on digital-service tracking site DownDetector, with service disruptions beginning around 4 a.m. ET. Most of the complaints were focused on problems with mobile phones or wireless service.
At 3:10 p.m. EST, roughly 11 hours after reports of the outage first emerged, the company said that it had restored service to all impacted customers. “We have restored wireless service to all our affected customers. We sincerely apologize to them,” AT&T said in a statement. The company added that it is “taking steps to ensure our customers do not experience this again in the future.” AT&T hasn’t disclosed the cause of the outages, but the problem snarled 911 centers, with some law enforcement officials noting that some people were calling the emergency number to test whether their phones worked.
The Federal Communications Commission confirmed Thursday afternoon that it is investigating the outage. The White House says federal agencies are in touch with AT&T about network outages but that it doesn’t have all the answers yet on what exactly led to the interruptions.
Earlier Thursday, AT&T acknowledged that it had a widespread outage and suggested a ridiculous alternative. “Some of our customers are experiencing wireless service interruptions this morning. We are working urgently to restore service to them,” AT&T said in a statement at 11:15 a.m. ET. “We encourage the use of Wi-Fi calling until service is restored.” AT&T says on its website that there is no extra cost for WiFi calling. Once set up, Wi-Fi calling works automatically when you’re connected to a Wi-Fi network that you choose. The catch here is that if your away from home, there probably won’t be WiFi hot spot you can connect with or you end up paying for that WiFi Internet service. You are in luck if you’re an Xfinity home internet customer, in which case you can use Xfinity WiFi hotspots for free once you sign in using your Xfinity account log-in credentials.
Initially, AT&T provided no official reason for the outage, but provided an update at 7.46pm EST: “Based on our initial review, we believe that today’s outage was caused by the application and execution of an incorrect process used as we were expanding our network, not a cyber attack. We are continuing our assessment of today’s outage to ensure we keep delivering the service that our customers deserve.” That sounds like a flimsy excuse to this author. The software update went wrong, according to preliminary information from two anonymous sources familiar with the situation.
The U.S. Cybersecurity and Infrastructure Security Agency is “working closely with AT&T to understand the cause of the outage and its impacts, and stand[s] ready to offer any assistance needed,” Eric Goldstein, the agency’s executive assistant director for cybersecurity, said in a statement to CNN.
White House National Security spokesman John Kirby said Thursday afternoon that the Department of Homeland Security and the FBI were looking into the matter and working with partners in the tech industry to “see what we can do from a federal perspective to lend a hand to their investigative efforts to figure out what happened here. The bottom line is we don’t have all the answers,” he said. “We’re working very hard to see if we can get to the ground truth of exactly what happened.”
Several police departments and municipalities warned local residents of what they described as a nationwide outage. In turn, officials urged callers to contact emergency services by alternative means but did not specify what those means were?
……………………………………………………………………………………………………………………………………………………………………………..
References:
https://about.att.com/pages/network-update
https://www.wbaltv.com/article/report-nationwide-outage-atandt-users/46900396
https://www.cbsnews.com/news/outage-map-att-where-cell-phone-service-down/
https://abcnews.go.com/US/att-outage-impacting-us-customers-company/story?id=107440297
Rogers Telecommunications restores service after 19 hour outage disrupting life in Canada
Telecom layoffs continue unabated as AT&T leads the pack – a growth engine with only 1% YoY growth?
From Mike Dano of LightReading:
AT&T’s rivals took a victory lap:
“Verizon’s network remains fully operational. Some customers may have experienced issues this morning when calling or texting those served by another carrier. Our network continues to function normally,” the operator said in a release Thursday morning.
“Decentralized wireless (DeWi) helps mitigate these type of disruptions. @REALLYWIRELESS, built on our REALLY DeWi network has not been impacted by these outages,” Really Wireless CEO Adam Lyons posted to social media.
T-Mobile officials too have said the company’s operations haven’t been affected.
So what’s going on?
Informed speculation
AT&T officials have not yet responded to questions from Light Reading about the cause of the outage. That’s not a surprise considering the company is in the midst of triage. In an updated statement to Cnet later Thursday morning, the company said that service had been restored to 75% of its network.
UPDATE: On Friday, AT&T said the outage was over and that it was “caused by the application and execution of an incorrect process used as we were expanding our network, not a cyber attack. We are continuing our assessment of today’s outage to ensure we keep delivering the service that our customers deserve.”
Regardless, the situation could involve a software upgrade to AT&T’s core network. That upgrade could have affected AT&T’s wired transport operations – the fiber connections used by AT&T and other companies to shuttle Internet data from one part of the county to another – as well as AT&T’s wireless connections. It’s also possible that AT&T’s core network upgrade is somehow tied to the settings inside its customers’ phones, based on reports that some customers were affected and that others – including those using the same cell site – were not.
“Typically when something like that happens, it’s something in the core,” analyst Earl Lum, of EJL Wireless Research, told Light Reading on Thursday.
Lum explained that problems with core network upgrades can often cascade across the country.
“This is the downside to automation,” agreed analyst Roger Entner of Recon Analytics. He told Light Reading on Thursday that operators including AT&T are working to shift their operations from legacy hardware solutions to software-based products that are cheaper to operate. “That’s the beautiful thing about automation.”
However, that shift also allows glitches and bugs to quickly spread far beyond a particular radio or cell site. Entner pointed out that the time of the outage – in the early morning – is typically when software updates are deployed.
“At a high level, we believe this helps support our view that AT&T’s 5G network has fallen behind peers,” wrote the financial analysts at KeyBanc Capital Markets in a note to investors Thursday.
https://www.lightreading.com/mobile-core/at-t-s-outage-twists-up-its-mwc-story