This is what caused Telstra's nationwide outage

iStock

  • A software fault caused yesterday’s Telstra outage.
  • A built-in failsafe in the company system also failed.
  • Customers will only be compensated on a “case by case” basis.
  • There won’t be a free data day.

Telstra has revealed what caused its 3G and 4G services to go down yesterday, leaving customers across the country unable to make calls or use their data.

The telco says a software fault hit its 4G network, causing customers to be diverted to the 3G network which then overloaded.

“There is redundancy built into these systems but this did not operate as intended,” the telco said.

Which basically means, the failsafe failed.

Here’s the statement in full:

We are very sorry for any disruptions our customers experienced yesterday.

Starting from approximately 10:00am yesterday morning some key network equipment failed causing a disruption to 4G voice and data services nationally. The impact was widespread and with a large number of customers dropping back to 3G there was significant disruption to 3G voice and data services as demand exceeded the capacity of our 3G network.

We have identified that the initial cause of the disruption was a software fault which triggered multiple elements across the network to fail. The network is designed to switch onto standby hardware which it did. Following the failover however, a further fault caused an interruption which impacted 4G connections. There is redundancy built into these systems but this did not operate as intended.

By around 11:40am we had successfully isolated the impacted network infrastructure and services began to be progressively restored. It took about two hours for full service to be restored after that, although our team was working longer with some government and enterprise customers to help them with their systems recovery.

We are still investigating the root cause of the software fault. We are working closely with our technology vendors on the specific element of software which triggered the issue.

Our teams have worked around the clock to restore services and to investigate why the redundancy in our network did not prevent customer impact, for which we are deeply sorry.

We will continue to closely monitor network stability and performance.

Yesterday afternoon the company was unable to provide the public answers as to what was the root cause of the outage with Group Managing Director of Networks, Mike Wright, only able to confirm it was “not related to earlier impacts”.

“We know data traffic would have been reduced substantially,” he said, adding that “its hard to say whether any in particular state was impacted more than other states”.

He said the company would “work with customers on an individual basis” in fixing their service.

CEO Andrew Penn took to Twitter to apologise after services were back up and running as normal.

It’s the third time this month Telstra customers have been left without a service.

On May 1 widespread outages hit the telco’s NBN and 4G services just after midday and were not resolved until 4pm.

The issue was caused by “technical changes made ahead of upgrades to mobile traffic control equipment in Telstra’s Exhibition Street exchange in Melbourne.”

Three days later, a lightning strike took out an optic fibre pit in regional NSW. Triple zero calls in parts of NSW, Victoria and WA were affected from 3am to nearly midday.

Yesterday’s nationwide outage makes May 2018 Telstra’s worst patch since a disastrous first half of 2016 which saw five outages and eventually resulted in senior executives having millions in bonuses cut.

Following the previous outages, customers have been given “free data” days as compensation. It is still unknown if Telstra plans to do the same this time.

Telstra told Business Insider this time it will only be compensating customers on a case by case basis, as mentioned by Wright yesterday

We will treat this on a case by case basis. So if Business customers have been impacted, then we ask them to contact us and we will discuss their situation one on one.

Although it may come too little too late with many customer yesterday saying on Twitter the most recent outage was the final straw.

https://twitter.com/sokkleberryfinn/status/998373275804696579

Telstra’s share price is down 3% today, sitting a 7-year low.

NOW READ: Telstra shares are down 70% from their peak and heading for their lowest level since listing

NOW WATCH: Tech Insider videos

Business Insider Emails & Alerts

Site highlights each day to your inbox.

Follow Business Insider Australia on Facebook, Twitter, LinkedIn, and Instagram.