A data-sharing deal between Google DeepMind and the Royal Free London NHS Foundation Trust was riddled with “inexcusable” mistakes, according to an academic paper published on Thursday.
The “Google DeepMind and healthcare in an age of algorithms” paper — coauthored by Cambridge University’s Julia Powles and The Economist’s Hal Hodson — questions why DeepMind was given permission to mine millions of NHS patient records so easily and without patient approval.
“There remain many ongoing issues and it was important to document how the deal was set up, how it played out in public, and to try and stop another deal from happening in this way in the future,” Powles told Business Insider in Berlin the day before the paper was published.
DeepMind says that the study “completely misrepresents the reality of how the NHS uses technology to process data” and that it contains “significant mistakes.”
DeepMind has used the data access it was given to create a mobile app called Streams, which was initially designed to help clinician’s monitor patients with acute kidney injury (AKI).
Powles, a research associate in law and computer science at St John’s College, Cambridge, and Hodson, a technology correspondent for The Economist, argue in the paper published in the Health and Technology journal that the terms of the initial deal (a subsequent one has been made) were “highly questionable” and that they lacked transparency.
DeepMind has tried to defend the deal by saying that it’s providing something known in the healthcare industry as “direct care,” which assumes that an identifiable individual has given implied consent for their information to be shared or uses that involve the prevention, investigation, or treatment of illness.
“The specific problems are they had access to every patient in the hospital on the legal basis that they were providing direct care to every patient,” said Powles. “We think that’s problematic given that they only ever asserted that they were interested in helping patients with acute kidney injury. They have since pivoted to look at a bunch of conditions but they haven’t addressed the gap in the initial deal between what the purpose was and why they had the access they did.”
She added: “If they’d had a small sample set that proved the results, that showed that this app was working, and then they said we’re going to roll it out on these terms, for this period and with this amount of money passing hands, then it would be a totally different game.”
Powles believes that the case should be considered a “cautionary tale” for the NHS and other public institutions that are look to work with innovative tech firms.
Regulators are investigating the deal
The deal has been under scrutiny ever since New Scientist revealed the scope of it some seven months after it was signed. A Freedom of Information request from the science and technology publication showed that DeepMind, a subsidiary of one of the world’s biggest companies, Alphabet, had quietly been given access to millions of identifiable patient records and that it was not possible to track how these were being used.
The patient records included information about people who were HIV-positive, as well as details about drug overdoses, abortions, and hospital visits.
In November 2016, DeepMind and the Trust replaced the old deal with a new one, possibly in a bid to patch things up.
The initial deal is currently being investigated by the Information Commissioner’s Office (ICO), which is yet to publish any of its findings publicly. The National Data Guardian (NDG) is also looking into the partnership.
DeepMind’s access to patient data has not been affected by the involvement of the ICO and the NDG, meaning that he company’s Streams app is still in the process of being rolled out to clinicians.
“I think it would be appropriate if the National Data Guardian was to find that the direct care basis was not satisfied here and the ICO could respond by some sort of admonishment of what happened,” Powles said.
“And I think it would be in DeepMind’s interest to move on from this because they are doing good work. The people there are motivated by the highest ordeals.”
The authors conclude their paper by saying that deals of this ilk should not be launched without full disclosure of the framework of the documents and the approvals that underpin it. “The failure of both sides to engage in any conversation with patients and citizens is inexcusable,” they write. “The reality is that the exact nature and extent of Google’s interests in NHS patient data remain ambiguous.”
Powles said that Google could look to use the data for advertising and other commercial uses in the future due to the lack of oversight from regulators.
Responding to the paper, a DeepMind spokesperson said:
“The Royal Free London and DeepMind are committed to working together to use technology to support world class care for patients.
“This paper completely misrepresents the reality of how the NHS uses technology to process data. It makes a series of significant factual and analytical errors, assuming that this kind of data agreement is unprecedented. In fact, every trust in the country uses IT systems to help clinicians access current and historic information about patients, under the same legal and regulatory regime.
“The Streams app is already being used on the wards at the Royal Free Hospital and the feedback from clinicians has been overwhelmingly positive. Nurses estimate that it is saving them up to two hours every day because patient information is now available via a secure mobile app. It also means that clinicians are able to respond to patients who are at risk of developing life-threatening conditions in minutes rather than hours or days.”
In a statement forwarded to Business Insider by DeepMind, Mark Watts, senior partner at Bristows law firm, wrote:
“This paper appears to not only misunderstand common and current practice as to how the NHS engages and works with data processors in many fields, but also makes several statements on the application of relevant law which, from an initial review, appear to be incorrect.”
Powles and Hodson are planning to publish a second paper that will seek to analyse the terms of the revised agreement between DeepMind and the Royal Free.
More from Business Insider UK:
- What we learned from app-only bank Monzo’s crowdfunding pitch
- The 21 best universities in the world for budding economists
- 10 things you need to know in markets today
- 10 things in tech you need to know today (GOOG, FB, MSFT)
- FOX NEWS POLL: Bernie Sanders remains the most popular politician in the US
Business Insider Emails & Alerts
Site highlights each day to your inbox.