An independent review panel has raised a number of concerns about how Google DeepMind is working with the National Health Service (NHS) in a report published on Wednesday.
DeepMind tasked the panel with scrutinising its NHS partnerships after New Scientist revealed that the company had quietly been given access to 1.6 million patient records to develop a kidney monitoring app called Streams without their prior knowledge.
The unpaid panel includes ex-Cambridge MP Julian Huppert, The Co-Operative Group’s current chief digital officer Mike Bracken, venture capital investor Eileen Burbidge, and six other members.
The panel’s 22-page report, which cost DeepMind £59,315, raised concerns about subjects including:
- The lack of clarity in the original information sharing agreement (ISA) with the Royal Free London NHS Foundation Trust
- 11 “minor” vulnerabilities that were found across the app, the API (application programming interface), and the servers in DeepMind’s data centre
- And the lack of work on public engagement, particularly in relation to the links between DeepMind Health and Google.
While there were criticisms, the report — published two days after the UK data watchdog ruled that DeepMind’s first deal with the NHS was illegal — also commends DeepMind for its overall high level of data security, and for openly publishing its new contracts with the NHS with minimal redactions. The report provides a number of recommendations for DeepMind to consider when working with the NHS in future.
Founded in London in 2010 by Demis Hassabis, Mustafa Suleyman, and Shane Legg, DeepMind is an artificial intelligence (AI) research lab that was acquired by Google in 2014 for £400 million. The company is best known for its AlphaGo algorithm that learned how to play and master the Chinese board game Go but it is now pursuing a number of commercial opportunities, including energy and healthcare.
DeepMind’s first deal with the NHS has proved to be highly controversial. It was signed with the Royal Free in 2015 and it has attracted a great deal of criticism due to its scope and the fact that DeepMind is owned by Google, one of the world’s most powerful technology companies.
Questions have been asked about why DeepMind was given access to so many patient records and so much patient data — including whether people have had an abortion or tested positive for HIV — to develop a kidney monitoring app, which doesn’t use any of the data-demanding AI DeepMind is known for.
“The initial information sharing agreement was September 2015 and was before DeepMind Health was launched as a separate organisation,” said Huppert, chair of the panel, at a press briefing on Tuesday.
“We do not think it was adequate. DeepMind Health have accepted that, which is why in November they replaced it with a better, more rigorous and much clearer contract, which they have also made publicly available.” Huppert was unable to say how the new contract with the Royal Free is different.
DeepMind initially intended to use considerably more AI in Streams when it first came up with the app but Huppert said that the company’s ambitions changed shortly after it started working with the NHS.
“DeepMind had a view that they could use AI to significantly improve healthcare within the NHS and frankly around the world,” said Huppert. “That was the starting conceptual point. I think it would then be fair to say that as they discussed this with people they found that the state of data and the state of information flow within the NHS was simply not as good as they had hoped. That’s why in some sense they had to step back from I guess the exciting AI that they intended to do right at the start to dealing with problems about how do you get information flow.”
The panel avoided some of the hard questions
The panel failed to focus on several key issues that are relevant to DeepMind’s work with the NHS including the relationship between DeepMind and Google parent company Alphabet, which DeepMind now sits under, and why DeepMind was able to rush through the data-sharing deal with the Royal Free in just four months. These are subjects that the panel could look at in its second report, Huppert said.
When asked specifically about why the panel didn’t look at the fact DeepMind signed a questionable ISA with the Royal Free, Huppert said: “We were very conscious of the NDG [National Data Guardian] and the ICO investigations and were very cautious not to do anything that could be seen as trying to pre-judge anything that they did. So we very carefully didn’t look at that aspect of it.”
DeepMind responded to the panel’s findings in a blog post co-authored by Suleyman and DeepMind Health clinical lead Dominic King.
“As a result of this process, DeepMind Health has committed to a series of changes to our work and practices to try to set higher standards in our second year,” the pair wrote.
“We know we need to work harder to be responsive and accountable to the needs of a far greater cross-section of medicine and society. This includes significantly improving our work with patients and the public, and continuing on the path of greater engagement with Royal Colleges, professional bodies and many other groups in the NHS community.”