Data Privacy and Compliance in LLM Applications
Data privacy and compliance are non-negotiable in LLM deployments. Violations can result in $20M+ GDPR fines, loss of customer trust, and regulatory shutdown. This guide covers privacy-by-design patterns, compliance checklists, and code implementations for GDPR, HIPAA, CCPA, and SOC 2 requirements.
Key Takeaways
- Privacy-by-design means data minimization, encryption, and anonymization happen at the architecture level, not as an afterthought
- GDPR applies to any EU resident data; HIPAA applies to healthcare; CCPA applies to California residents; compliance is not optional
- Log every prompt and response for audit trails; implement data retention policies (auto-delete after N days) to meet "right to be forgotten" requirements
- Teams that document compliance controls in code (infrastructure-as-code) pass audits 5x faster than manual documentation
- Data classification (PII, PHI, public) must happen before data enters the LLM pipeline; use tokenizers to remove sensitive information before prompting
Why This Matters Now
Large language models ingest data, process it, and may log or retain it for model improvement. This creates three compliance risks:
- Data leakage: User data sent to third-party APIs (OpenAI, Anthropic) without contractual DPAs or data processing agreements
- Retention violations: Prompts retained indefinitely instead of being deleted per user request (GDPR Article 17, "right to be forgotten")
- Inadequate audit trails: No way to prove to regulators what data was processed, by whom, for how long
Proper privacy engineering eliminates these risks through data minimization, encryption, and documented controls.
Privacy-by-Design: Five Pillars
Privacy-by-design means privacy is engineered into the system, not bolted on later. The five pillars are:
1. Data Minimization
Only send data to the LLM that's necessary for the task. Don't send the entire customer record when you only need the purchase history.
def build_llm_prompt_minimal(user_id: str, query: str) -> str:
"""
Example: customer support query.
Only fetch and send the necessary data.
"""
# AVOID: fetch user.all_data()
# ONLY fetch what's needed for the task
recent_orders = fetch_recent_orders(user_id, limit=5)
account_status = fetch_account_status(user_id)
# Do NOT include: SSN, phone, email, billing address, payment method
prompt = f"""
User {user_id} has {account_status} account status.
Recent orders: {recent_orders}
Customer query: {query}
Respond helpfully without mentioning any personal details not in this context.
"""
return prompt
Why it matters: Smaller context = less data at risk. GDPR defines this as the "purpose limitation principle."
2. Encryption (In Transit and At Rest)
Encrypt data flowing to the LLM and data stored in logs or retrieval systems.
from cryptography.fernet import Fernet
import json
class EncryptedPromptLogger:
"""Log prompts and responses with encryption."""
def __init__(self, encryption_key: str):
self.cipher_suite = Fernet(encryption_key)
def log_prompt_encrypted(self, user_id: str, prompt: str, response: str) -> None:
"""Log to disk with encryption."""
log_entry = {
"user_id": user_id,
"prompt": prompt,
"response": response,
"timestamp": time.time()
}
# Encrypt before writing
plaintext = json.dumps(log_entry).encode()
encrypted = self.cipher_suite.encrypt(plaintext)
# Write encrypted data to log file
with open("llm_logs.enc", "ab") as f:
f.write(encrypted + b"\n")
def retrieve_encrypted_log(self, user_id: str) -> Dict:
"""Decrypt logs for audit (only authorized personnel)."""
# In production, restrict access to this method (authentication + authz)
with open("llm_logs.enc", "rb") as f:
for line in f:
decrypted = self.cipher_suite.decrypt(line.strip())
entry = json.loads(decrypted)
if entry["user_id"] == user_id:
return entry
return None
Why it matters: GDPR Article 32 requires "encryption of personal data." Encrypted logs satisfy audit requirements and reduce breach risk.
3. Anonymization and Tokenization
Remove or replace sensitive information before sending to LLMs. Use tokens (e.g., [USER_ID]) instead of real IDs.
import re
from datetime import datetime, timedelta
class DataTokenizer:
"""Replace sensitive data with tokens."""
PII_PATTERNS = {
"email": r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"phone": r"\b\d{3}[-.]?\d{3}[-.]?\d{4}\b",
"ssn": r"\b\d{3}-\d{2}-\d{4}\b",
"credit_card": r"\b\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}\b",
"ipv4": r"\b(?:\d{1,3}\.){3}\d{1,3}\b"
}
def tokenize(self, text: str) -> tuple[str, dict]:
"""Replace PII with tokens; return text and mapping."""
token_map = {}
tokenized = text
for pii_type, pattern in self.PII_PATTERNS.items():
for match in re.finditer(pattern, text):
original = match.group(0)
token = f"[{pii_type.upper()}_{len(token_map)}]"
tokenized = tokenized.replace(original, token, 1)
token_map[token] = original
return tokenized, token_map
def detokenize(self, text: str, token_map: dict) -> str:
"""Restore original PII (only for authorized purposes)."""
result = text
for token, original in token_map.items():
result = result.replace(token, original)
return result
# Usage
tokenizer = DataTokenizer()
sensitive_text = "User [email protected] with SSN 123-45-6789 called."
tokenized, mapping = tokenizer.tokenize(sensitive_text)
print(tokenized) # "User [EMAIL_0] with SSN [SSN_1] called."
# Send tokenized text to LLM
prompt = f"Support ticket: {tokenized}\nRespond helpfully."
# Only restore PII if returning data to authorized user
result_with_pii = tokenizer.detokenize(llm_response, mapping)
Why it matters: Tokenization reduces compliance risk—you can share logs and datasets with the model without exposing raw PII.
4. Data Retention and Right to Be Forgotten
GDPR Article 17 requires you to delete personal data when no longer needed. Implement automatic retention policies.
import time
from datetime import datetime, timedelta
class DataRetentionManager:
"""Manage data lifecycle and GDPR right-to-be-forgotten requests."""
def __init__(self, db_connection, retention_days: int = 30):
self.db = db_connection
self.retention_days = retention_days
def delete_user_data(self, user_id: str) -> bool:
"""
Delete all data associated with a user (right to be forgotten).
Required by GDPR Article 17.
"""
tables_to_clean = [
"prompts",
"responses",
"audit_logs",
"user_embeddings",
"retrieval_cache"
]
for table in tables_to_clean:
query = f"DELETE FROM {table} WHERE user_id = %s"
self.db.execute(query, (user_id,))
# Log deletion for audit purposes
self.db.execute(
"INSERT INTO deletion_log (user_id, deleted_at) VALUES (%s, %s)",
(user_id, datetime.now())
)
return True
def purge_expired_data(self) -> int:
"""
Automatically delete old data to meet retention policy.
Run daily via cron or Lambda.
"""
cutoff_date = datetime.now() - timedelta(days=self.retention_days)
tables_to_purge = ["prompts", "responses"]
total_deleted = 0
for table in tables_to_purge:
query = f"DELETE FROM {table} WHERE created_at < %s"
result = self.db.execute(query, (cutoff_date,))
total_deleted += result.rowcount
return total_deleted
# Example: Run daily
manager = DataRetentionManager(db, retention_days=30)
def daily_data_purge():
"""Scheduled task (cron job or Lambda)."""
rows_deleted = manager.purge_expired_data()
print(f"Purged {rows_deleted} rows older than 30 days")
# Example: Handle GDPR request
def handle_deletion_request(user_id: str) -> Dict:
"""Called when user submits deletion request."""
success = manager.delete_user_data(user_id)
return {"status": "deleted" if success else "error", "user_id": user_id}
Why it matters: Automatic purge policies ensure compliance without manual work. GDPR fines double if you don't respond to deletion requests within 30 days.
5. Access Control and Audit Logging
Restrict who can access logs and implement detailed audit trails.
from functools import wraps
import logging
class AuditLogger:
"""Log all data access for compliance audit."""
def __init__(self):
self.logger = logging.getLogger("audit")
handler = logging.FileHandler("audit.log")
formatter = logging.Formatter(
"%(asctime)s | user=%(user_id)s | action=%(action)s | resource=%(resource)s | result=%(result)s"
)
handler.setFormatter(formatter)
self.logger.addHandler(handler)
def log_access(self, user_id: str, action: str, resource: str, result: str = "success"):
"""Log every access to user data."""
self.logger.info(
"Data access",
extra={
"user_id": user_id,
"action": action,
"resource": resource,
"result": result
}
)
audit = AuditLogger()
def require_authorization(action: str):
"""Decorator to enforce access control."""
def decorator(func):
@wraps(func)
def wrapper(user_id: str, *args, **kwargs):
# Check if current user has permission for this action
current_user = get_current_user()
if not current_user.has_permission(action):
audit.log_access(user_id, action, func.__name__, "denied")
raise PermissionError(f"User {current_user.id} cannot {action}")
# Log successful access
audit.log_access(user_id, action, func.__name__, "success")
return func(user_id, *args, **kwargs)
return wrapper
return decorator
@require_authorization("view_prompt_logs")
def view_user_prompts(user_id: str) -> List[Dict]:
"""Retrieve prompts sent by or about a user."""
query = "SELECT prompt, response, created_at FROM prompts WHERE user_id = %s"
return db.fetch_all(query, (user_id,))
Why it matters: Audit logs are proof of compliance. GDPR Article 25 requires "accountability"—you must prove you followed the rules.
Compliance Checklist: Before Production
GDPR (EU Residents)
- Data Processing Agreement (DPA): Signed with all AI vendors (OpenAI, Anthropic, etc.) covering data sub-processing
- Privacy Policy: Published and translated; explains what data you collect, retain, and delete
- Right to Be Forgotten: Implement
DELETE FROM ... WHERE user_id = Xand test it quarterly - Data Subject Rights: Can users export their data in machine-readable format (JSON/CSV)?
- Consent: If collecting new data, document explicit consent in logs
- Encryption: All personal data at rest encrypted (AES-256); in transit over TLS 1.3
# Example: GDPR compliance checklist code
gdpr_compliance = {
"dpa_vendors": ["openai", "anthropic", "pinecone"],
"privacy_policy_url": "https://myapp.com/privacy",
"privacy_policy_version": "2.0",
"privacy_policy_last_updated": "2026-06-02",
"right_to_be_forgotten": {"method": "auto_delete", "tested_at": "2026-05-28"},
"data_export_format": "json",
"encryption_at_rest": "aes-256",
"encryption_in_transit": "tls-1.3",
"retention_days": 30
}
HIPAA (US Healthcare)
- Business Associate Agreement (BAA): Required if using third-party LLM APIs with patient data
- Minimum Necessary: Only send necessary clinical information (diagnosis, recent labs); never send full EMR
- Audit Controls: Log access to all PHI; generate audit reports quarterly
- Encryption: PHI encrypted with FIPS 140-2 validated algorithms
- Access Control: Multi-factor authentication required to view PHI logs
- Breach Notification Plan: Document process to notify patients within 60 days of breach
CCPA (California Residents)
- Data Collection Inventory: Document every data point collected and its purpose
- Opt-Out Link: User can request non-sale of personal information
- Data Deletion Request: Respond within 45 days
- Data Portability: User can export their data in machine-readable format
SOC 2 (All Vendors)
- Access Control: Documented user roles and approval workflows
- Change Management: All code changes logged and approved before deploy
- Incident Response Plan: Written plan for security incidents, tested quarterly
- Vendor Management: List of all subprocessors; update users within 30 days of changes
- Penetration Testing: Annual third-party pentest; fix critical findings within 30 days
Operational Pattern: Privacy-Aware Agent
Here's a complete agent that respects privacy constraints:
from enum import Enum
class DataClassification(Enum):
PUBLIC = "public"
INTERNAL = "internal"
CONFIDENTIAL = "confidential"
RESTRICTED = "restricted" # Do not send to LLM
class PrivacyAwareAgent:
"""Agent that enforces privacy constraints before calling LLM."""
def __init__(self, llm_client, tokenizer: DataTokenizer, logger: AuditLogger):
self.llm = llm_client
self.tokenizer = tokenizer
self.logger = logger
def classify_data(self, data: str) -> DataClassification:
"""Classify data sensitivity."""
if self._contains_pii(data):
return DataClassification.RESTRICTED
elif self._contains_phi(data):
return DataClassification.RESTRICTED
elif self._contains_internal_id(data):
return DataClassification.CONFIDENTIAL
else:
return DataClassification.PUBLIC
def _contains_pii(self, text: str) -> bool:
"""Check for PII patterns."""
for pattern in DataTokenizer.PII_PATTERNS.values():
if re.search(pattern, text):
return True
return False
def _contains_phi(self, text: str) -> bool:
"""Check for HIPAA-protected health information."""
phi_patterns = [
r"diagnosis\s*[:=]",
r"med.*: (aspirin|insulin|metformin)",
r"mrn|medical record"
]
for pattern in phi_patterns:
if re.search(pattern, text, re.IGNORECASE):
return True
return False
def _contains_internal_id(self, text: str) -> bool:
"""Check for internal identifiers."""
return re.search(r"(CUST_|EMP_|INT_)\d+", text) is not None
def run_with_privacy_check(self, user_id: str, task: str, data: str) -> str:
"""Run agent with privacy constraints enforced."""
# Step 1: Classify data
classification = self.classify_data(data)
# Step 2: Enforce restrictions
if classification == DataClassification.RESTRICTED:
self.logger.log_access(user_id, "llm_call_blocked", task, "data_too_sensitive")
return "Error: Task data contains sensitive information. Cannot proceed with LLM processing."
# Step 3: Tokenize if needed
if classification in [DataClassification.CONFIDENTIAL, DataClassification.INTERNAL]:
data, token_map = self.tokenizer.tokenize(data)
else:
token_map = {}
# Step 4: Call LLM
prompt = f"Task: {task}\nData: {data}"
response = self.llm.create(messages=[{"role": "user", "content": prompt}])
# Step 5: Detokenize response (if we tokenized input)
if token_map:
response = self.tokenizer.detokenize(response, token_map)
# Step 6: Log (encrypted)
self.logger.log_access(user_id, "llm_call", task, "success")
return response
Frequently Asked Questions
Do I need a DPA if I use OpenAI's API?
Yes. OpenAI is a "data processor"—you need a Data Processing Agreement (signed, available on OpenAI's website) that covers GDPR/CCPA. Without it, you're non-compliant if you send EU/CA resident data to their APIs.
What's the difference between anonymization and tokenization?
Anonymization: Data irreversibly transformed so you can't identify the person (true anonymization is rare). Tokenization: Data reversibly replaced with tokens; you can restore it. Tokenization is practical for LLM pipelines; anonymization is ideal for research.
If I host the LLM locally, do I still need compliance?
Yes. Local LLMs don't eliminate GDPR/HIPAA/CCPA requirements. You still need encryption, access controls, audit logs, and retention policies.
How often should I test my deletion procedures?
At minimum, quarterly. Run a test deletion request for a fake user and verify the data is actually gone from all systems (database, logs, caches, backups).
What if a user requests their data be deleted but we need it for legal reasons?
GDPR allows exceptions for legal obligations. Document the exception (e.g., "retained for fraud investigation, will delete on [date]") in your deletion request response.