Privacy and Data

Privacy and Data

AI systems are hungry for data. That creates friction with privacy — the right of individuals to control information about themselves. Understanding the privacy landscape around AI is essential both for using AI responsibly and for building systems people can trust.

What Data AI Uses

Training data: modern LLMs are trained on large fractions of the public web, books, and other text. This data includes personal information — names, emails, addresses, opinions, medical discussions — that people posted without intending it to train AI.

Some concerning examples:

• Models can sometimes reproduce memorised training data verbatim, including personal information
• Models can reproduce copyrighted material — leading to ongoing litigation
• Models trained on specific private datasets (medical records, financial data) must manage those datasets with appropriate care

Inference data: when you use an AI API, the prompts you send may be used to improve future models (depending on the provider's terms). Enterprise API plans typically offer data processing agreements that opt you out of training use.

As of Late 2025: Copyright and Training Data

Copyright litigation over training data moved from "ongoing" to producing real outcomes. In September 2025, Anthropic agreed to pay authors around $1.5 billion to settle a class action over pirated books used in training — one of the largest copyright settlements on record, and a signal that using unlicensed copyrighted works carries real financial risk, not just reputational risk. Other lawsuits against AI companies, brought by authors, publishers, news organisations, and artists, are still working through the courts, and rulings so far have been mixed — some courts have treated training on legitimately obtained text as fair use while rejecting the use of pirated sources.

This area is moving fast and varies by provider. Whether a consumer-tier chat product trains on your conversations by default — and whether you can opt out — differs between providers and changes over time as products and policies are updated. Don't assume last year's default still applies: check the current terms for whatever tool you're using before sending anything sensitive.

Key Privacy Risks

Data leakage in RAG systems: if different users have access to different documents, careful access control is required. A naive RAG implementation might let one user retrieve another's private documents.

Re-identification: supposedly anonymised data can often be re-identified when combined with other information. LLMs trained on "anonymised" datasets can sometimes reconstruct personally identifiable information.

Sensitive data in prompts: users often paste sensitive information (personal health details, financial data, company confidential information) into AI chat interfaces without considering where that data goes.

Model inversion: in some cases, adversarial techniques can extract training data from models. This is an active research area.

Privacy-Preserving Practices

For developers:

• Don't store sensitive user data longer than needed
• Use data processing agreements with AI providers for enterprise use cases
• Implement access controls in RAG systems — users should only retrieve documents they're authorised to see
• Consider running sensitive workloads on local models to keep data on-premises
• Be transparent about what data is collected and how it's used

For users:

• Check the data usage policy before sending sensitive information to an AI service
• Use enterprise tiers of AI tools for work with confidential information — these typically have stronger data protections
• Don't paste passwords, API keys, or personal identification into AI tools
• Understand that "the AI won't remember this" and "this data is deleted immediately" are different things