In May 2025, Magistrate Judge Ona Wang ordered OpenAI to preserve and segregate chat logs it would otherwise have deleted — including conversations users had explicitly asked to erase. In January 2026, District Judge Sidney Stein affirmed an order compelling OpenAI to hand over 20 million de-identified conversations to plaintiffs in the New York Times copyright lawsuit. The delete button, it turns out, was a request. A court can override it.
🔍 THE BOTTOM LINE
Your ChatGPT conversations have no legal privilege. They are ordinary business records — subject to subpoena, retained on a schedule you don’t control, and preservable by court order even after you hit delete. Sam Altman himself admitted this. The intimacy is new. The legal category for it does not exist.
What Changed
Two court orders, one quiet lesson. First, Judge Wang’s May 2025 preservation order: OpenAI had to keep chat logs it would normally purge, including ones users had deleted. Then Judge Stein’s January 2026 ruling: 20 million de-identified conversations handed to the NYT’s legal team. As Hidden Layer reports, “the chats you thought were yours to delete are the company’s to keep, and a court’s to call for.”
This isn’t a hypothetical. It has already happened. Twenty million conversations — people’s health questions, relationship worries, startup ideas, legal concerns — are now in the hands of a plaintiff’s legal team. Not because those users consented. Because a court said the data was relevant to a copyright dispute between OpenAI and the New York Times.
Context: Two Companies, Two Completely Different Deals
There is no unified standard for what AI companies do with your data. The two biggest products don’t even agree.
ChatGPT trains on your conversations by default. You can turn it off, but you have to know the setting exists and go find it. Your chats are stored until you delete them — except, as we now know, “delete” is a suggestion a court can override.
Anthropic, which makes Claude, changed its consumer terms in 2025. If you opt in to training, your conversations can stay in the pipeline in de-identified form for up to five years. That’s a sixty-fold jump from the thirty-day window that used to be the default. Two products that hundreds of millions of people treat as interchangeable, two completely different retention regimes, and almost none of it is legible to the person typing.
OpenAI did release a privacy filter model in April 2026 — a 1.5B parameter open-weight model for scrubbing personally identifiable information before it reaches the cloud. It’s a useful tool. It’s also an admission that the default pipeline wasn’t safe enough.
The Altman Admission
Sam Altman said it himself, on Theo Von’s podcast in July 2025, as reported by TechCrunch:
“People talk about the most personal stuff in their lives to ChatGPT. People use it, young people especially, as a therapist, a life coach. And right now, if you talk to a therapist or a lawyer or a doctor about those problems, there’s legal privilege for it. We haven’t figured that out yet for when you talk to ChatGPT.”
He’s right, and it’s a strange thing to hear from the CEO of the company. Your conversation with a therapist carries privilege — a legal wall that holds up even in court. Your conversation with ChatGPT carries none. The intimacy is new. The legal category for it does not exist. So it defaults to the weakest possible setting: ordinary business records, subject to subpoena, retained on a schedule you don’t control.
NZ Angle: Whose Rules Apply When Data Leaves Aotearoa?
New Zealand’s Privacy Act 2020 requires explicit consent for data collection. It’s a robust framework — on paper. But when a US court issues an order compelling a global company like OpenAI to produce chat logs, the Privacy Act doesn’t stop that. The data lives on US-based servers, governed by US legal process, and a Kiwi user has no seat at the table.
The extraterritorial reach of US court orders is a live, unresolved issue for every New Zealander who has ever typed something personal into ChatGPT. We rely on national privacy laws. The reality of cloud-based, globally managed AI services means our data can become subject to foreign legal mandates before it even touches our domestic jurisdiction.
This is the same structural problem we’ve seen with Florida’s lawsuit against Altman personally — US jurisdictions reaching through global platforms to affect users everywhere. The difference is that Florida is at least a domestic US matter. The NYT preservation order affects every ChatGPT user on the planet, including the ones in Auckland who thought their conversations were private.
The Bigger Picture
A KFF tracking poll found that about a third of US adults have turned to AI for health advice, and one in six for mental health. Many said it felt more private than asking a person. People disclose more to the machine, on the stigmatized subjects, for the same reason they trust it less: the format invites it.
The protection has not caught up to the disclosure. We adopted a technology that feels like a private confessional and is built like a corporate database. We did it faster than anyone wrote the rules for it. The result is a legal grey zone where 20 million conversations can be ordered into a courtroom and the people who typed them have no standing to object.
❓ FAQ
Q: If I delete a chat in ChatGPT, is it actually gone? From your view, yes. From OpenAI’s servers, not necessarily. And if a court issues a preservation order, the company is legally required to keep it even after you delete it. The May 2025 order proved this is not theoretical.
Q: Does the NZ Privacy Act 2020 protect my chats? It sets high standards for consent and collection within New Zealand. But enforcement against a foreign company responding to a US court order is complex and largely untested. Your data likely lives on US servers under US legal jurisdiction.
Q: Should I stop using AI for personal advice? Not necessarily — but treat anything you type as if it could be read aloud in a courtroom. If you wouldn’t put it in an email to a stranger, don’t put it in a chatbot. The oldest rule for anything that goes into someone else’s system still applies.
Q: What’s the difference between ChatGPT and Claude on data retention? ChatGPT trains on your conversations by default unless you opt out. Claude requires opt-in for training, but if you opt in, conversations can persist for up to five years. Neither has legal privilege. Both are subject to court orders.
🔍 THE BOTTOM LINE
The delete button is a request, not a guarantee. The legal privilege that protects your conversations with doctors and lawyers does not exist for your conversations with AI. Twenty million ChatGPT sessions are now in a courtroom because no one wrote the rule that says they shouldn’t be. Until that rule exists, the safest assumption is that everything you type into a chatbot could someday be read by someone you didn’t intend to share it with.
