An investigation has for the first time measured the depth of the bungle which put the private health data of millions of Australians at risk of public exposure.
The Federal Health Department breached privacy laws when it made public data about claims for Medicare and Pharmaceutical benefits which researchers were later able to decode to identify patients, the official investigation found.
The 3 billion lines of data covered details on 2.5 million Australians.
The data included the services provided to patients and the providers of those services, unique patient identification numbers, sex, year of birth and which of four broad geographic regions the patient enrolled with Medicare.
The data was downloaded about 1,500 times in the one month it was publicly available.
The Australian Information Commissioner and Privacy Commissioner, Timothy Pilgrim, has just finished an investigation into the data leak in August 2016.
The data, a 10% sample of people who had made a claim for payment of Medicare Benefits since 1984, or for payment of Pharmaceutical Benefits since 2003, was published on data.gov.au, a central depository of public data.
However, a month after the dataset was published, researchers Chris Culnane, Benjamin Rubinstein and Vanessa Teague at the University of Melbourne identified a weakness in the technique used to encrypt Medicare service provider numbers in the dataset.
This enabled them to reverse the encryption, potentially allowing the identification of Medicare service providers.
Analysis from experts from the Australian Bureau of Statistics and Data61 then found that the detailed nature of the information created a risk that some individuals could be identified by linking the dataset with other information sources.
The Department of Health quickly removed the dataset from public access.
The Australian Information Commissioner and Privacy Commissioner says there were flaws in the process followed by the Department of Health in de-identifying the dataset, assessing the risk of re-identification and deciding to publish it.
The commissioner say decryption of Medicare service provider numbers does not mean that a provider is identified. However, this meant that there was potential to re-identify providers.
The Commissioner says the department, whose assessment of the risks was “inadequate”, had breached the Privacy Act.
The Department has since changed is data release policies.
“There are important lessons from this matter, both for the Department of Health and for other custodians of valuable repositories of personal information,” says the commissioner.
“The de-identification of large and rich datasets for publication to the world at large is extremely difficult.
“The decision making process the Department of Health followed before releasing this data did not involve a clear and documented approval process, rigorous risk management processes, or a significant degree of cross-government coordination.
“This incident offers an opportunity for the Australian Government to strengthen its approach to publishing data that is based on personal information.”
A new privacy code for government agencies is due in July this year.