Telstra’s mobile network across the whole of Australia suffered its biggest outage in recent memory yesterday after a key component to its network was accidentally switched off.
The telco’s chief operating officer Kate McKenzie spoke to the media late in the afternoon after the network had been struggling to function for some hours. She blamed the problem on “human error”, saying that a core node which provides network access for a large area was malfunctioning and restarted by an engineer who wasn’t following standard procedure.
As a result, customers were then pushed in spades onto the rest of Telstra’s nodes, triggering congestion and failure all through the network.
This meant both 3G and 4G networks were affected, with customers not able to make or receive calls and internet services were unavailable.
Many businesses got hit hard, with Eftpos terminals, regional internet connections and nearly all of Australian taxis with Cabcharge terminals out for half the day.
Worryingly, the West Australian Department of Fire and Emergency Services said that its SMS warning system for the Myalup fire zone was also “compromised” during the interruption.
Telstra said that its new product for emergency services called “Lanes” would not have been affected if it was online yesterday.
The outage itself began around midday until a little after 3pm when services started to slowly come back online.
“[The employee responsible] didn’t follow procedures and clearly that’s not a good thing but I wouldn’t want to pre-empt the proper investigation and we’ll figure out what the right response is when we’ve had a chance to dig into the detail,” Ms McKenzie said.
“I think he’s probably had the worst day of his career.”
Telstra operates with 10 nodes across the country, and is able to run the network off just seven if needed. Telstra’s investigation will continue to determine for sure why the network went down with nine nodes still operating properly.
It’s not sure what node went down, but the nodes in a mobile network is a connection point, and in this case a redistribution point. They act as the messenger between data at Telstra’s core network to the access network where customers connect to.
As a result of the interruption, Telstra will be offering free data to all its customers on Sunday.