Protecting Patient Privacy in the Age of AI: MIT Researchers Investigate Memorization Risks in Clinical Models

The foundation of patient privacy traces back to the Hippocratic Oath, an ancient text defining medical ethics. This Oath emphasizes the importance of physicians keeping their patients’ affairs private, illustrating how critical confidentiality has been to the healthcare profession for more than two millennia. In today’s digitally driven world where personal information is tracked and commodified, medicine remains a vital sphere where privacy is still intrinsic.

Indeed, patients need to feel secure when divulging delicate information to their doctors. This trust is a prerequisite for accurate diagnoses and effective treatments. Nonetheless, even this domain is not impervious to technologies that are changing many aspects of life. Specifically, the rise of artificial intelligence (AI) is posing new challenges to maintaining privacy in the health sector.

A New Breed of Threat to Confidentiality?

A recent study led by researchers at MIT is sounding the alarm about the potential for AI to undermine patient privacy. Focusing on AI models known as “foundation models,” which learn from large data sets to make predictions, the study revealed that these models sometimes “memorize” individual patient data instead of generalizing across many records. The implications are alarming – if an AI model reproduces sensitive patient information, it could violate their privacy.

Leading the study was postdoctoral researcher Sana Tonekaboni, who teamed up with MIT Associate Professor Marzyeh Ghassemi to develop tests for gauging how much information an attacker would need to extract sensitive data from a model, and how damaging such a leak could be.

The Growing Danger of Data Breaches

With the migration of health records to digital systems, the instances of data breaches have escalated significantly. Over the past two years alone, there were 747 breaches impacting more than 500 individuals each, primarily due to IT mishaps or hacking. This trend highlights the emerging dangers posed by AI-related privacy leaks, especially for patients with rare or unique conditions. Even when data is de-identified, these individuals are still at risk as the leaked information could potentially identify them.

In their research, the team found that leaks could vary greatly in their repercussions. It might not be a big issue if a patient’s age or general demographics were revealed. But if more sensitive information – such as a diagnosis of HIV or a history of substance misuse – were disclosed, it could have serious implications. The researchers’ tests are aimed at differentiating between benign and harmful leaks and assessing the likelihood of different types of attacks in real-world settings.

How to Safeguard Health AI?

Going forward, the research team hopes to incorporate insights from clinicians, privacy advocates, and legal experts to broaden their study. Their goal is to formulate a robust framework for evaluating and managing privacy risks before these foundation models are integrated into clinical settings. Additionally, their aim is to remind us of why maintaining privacy in healthcare is so essential. As Tonekaboni put it, “There’s a reason our health data is private…There’s no reason for others to know about it.”

The research was supported by several organizations including the Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard, the Wallenberg AI program, the Knut and Alice Wallenberg Foundation, the U.S. National Science Foundation, and others.