Commissioner of Taxation, Chris Jordan, has promised a full review into a series of embarrassing online failure by the ATO this week.
The review will look at why the back-up arrangements also failed when the system first failed on Monday this week.
Jordan called the crash, which took all of his organisation’s services offline for at least 36 hours as “the worst unplanned system outage in recent memory”.
The repeated failures, which have brought up memories of the ABS Census night outage, led Jordan to put out a rare statement on the issue amid a promise to review everything.
“This was an extremely unusual and unfortunate event with the outage caused by a significant and unprecedented failure of storage hardware,” he said.
Hewlett Packard Enterprise (HPE) put in a storage hardware upgrade in November 2015, which Jordan described “state-of-the-art’ at the time” and used by by many of HPE’s clients.
But when it failed, the back-up systems also failed, and as late as Friday, parts of the ATO network, including the MyTax individual website, were offline.
“The issues we have experienced this week do not relate to our overall IT capability or skills,” Jordan said.
“What compounded the problem beyond the initial failure was the subsequent failure of our back-up arrangements to work as planned. The failure of our back-up arrangements meant that restoration and resumption of data and services has been very complex and time consuming.”
Jordan said no data had been compromised or lost.
“ATO staff and people at HPE have worked around the clock over the last week to bring systems back online progressively. Our website, the Tax Agent Portal, BAS Portal and Business Portals are up and running and we are seeing continued improvements in their functionality. We are continuing to work on the stabilisation of ATO Online services,” the tax boss said.
“We are realistic that there may be some intermittent performance issues in the next couple of days as the full restoration process proceeds.”
He said all refunds that were due when the system went down on Monday have now been processed and will be paid today and tomorrow.
“Any other refunds in the system will be fast-tracked to ensure people get these payments as quickly as possible – we have devoted additional resources to ensure the vast majority of people get their refunds on time,” he said
“I repeat a message we issued earlier in the week – no taxpayers will be disadvantaged as a result of the outage. This means if you needed to make a payment or lodge a form but couldn’t, you won’t be penalised. If you have any other concerns about how the outage might have affected you, please get in touch with us on 13 28 61.
“We know the outages this week have caused inconvenience, especially so close to Christmas, and I apologise for that.”
Jordan said the review will begin next week once systems have return to normal to investigate what happened, how and why, and measures needed to avoid a repeat.
“The review will be conducted by an independent expert who will determine the nature of the failure(s) and their root cause(s), the adequacy of back-up and contingency arrangements, and the likelihood of recurrence,” Jordan said
“The review will also consider our immediate response to the failure, how we managed business resumption processes and the effectiveness of our communication with the community.
“I will be doing everything I can to learn from what has happened this week and to put in place any necessary changes to minimise the risk of any recurrence. Please be assured that I am committed to providing a quality, reliable and contemporary service to the Australian community.”
The commissioner has issued a draft scope of the review into the outage. Here’s how it looks:
Following the outage of the ATO business systems that occurred from Monday 12 December 2016, the ATO will engage an independent expert to undertake an end-to-end review into what happened and why, and what needs to happen to ensure the ATO and the community are not exposed to this type of incident in the future.
The independent reviewer will assist the ATO to fully understand the causes and impacts of the outage experienced and will provide the ATO with the following observations and advice:
a. a definitive description of the failure and its root cause
b. factors leading to the outage and contributing to the duration, scale, and scope of the outage
c. adequacy of back-up and contingency strategies and arrangements and explanation as to why failover to our secondary site did not work
d. adequacy of restoration and resumption procedures of technology, infrastructure and applications
e. whether there is anything unique or unusual in the physical and technical ATO technology infrastructure and/or architecture and/or environment that suggests there is a high risk of a repeat or like failure
f. adequacy, speed and robustness of the critical event response provided by various vendors and other non-ATO entities
g. any other observations or advice on improvements to IT systems to prevent a re-occurrence of the issues, along with the indicative costs and benefits for any improvement options to be considered.
The independent reviewer will also make observations about the ATO’s approaches to inform and update the community, the Government and staff on the situation.
The ATO is also after feedback about its business continuity management and its capability to:
a. detect the outage
b. ascertain the scale and scope of the impacts
c. mobilise sufficient response; and
d. manage the recovery and resumption of business.
NOW WATCH: Briefing videos
Business Insider Emails & Alerts
Site highlights each day to your inbox.