Skip to main content

Secrets management in LLM pipelines

Secrets management in LLM pipelines protects sensitive data: API keys, database credentials, encryption keys, and authentication tokens required for deployment and inference. Secrets must be encrypted at rest, rotated regularly, and never committed to source control. A robust secrets strategy uses a centralized vault (HashiCorp Vault, AWS Secrets Manager, or similar), stores secrets as encrypted values, audits all access, and injects secrets into environments at runtime. Without proper secrets management, leaked API keys expose your LLM service to unauthorized use, data breaches, and financial penalties.

The Risk of Leaked Secrets in LLM Systems​

LLM applications depend on external API credentials (Anthropic API keys, OpenAI keys, database passwords, encryption keys). If a credential leaks, attackers can call your API endpoints, impersonate your service, or steal fine-tuned model weights. GitHub scanning tools report that hardcoded secrets are discovered and exploited within minutes of being pushed. A leaked Claude API key can cost thousands in unauthorized API calls. A leaked database password grants access to all training data. Proper secrets management is not optional—it is critical infrastructure.

Secret Types in LLM Pipelines​

Categorize secrets by sensitivity and rotation frequency:

  • API Keys (Claude, OpenAI, external services): rotate every 90 days, high-sensitivity.
  • Database Credentials (username, password, connection strings): rotate every 30 days, high-sensitivity.
  • Encryption Keys (for data at rest, in-transit): rotate every 12 months, critical-sensitivity.
  • Service Account Tokens (for CI/CD authentication, Kubernetes): rotate monthly, high-sensitivity.
  • Configuration Secrets (feature flags, API endpoints if sensitive): rotate as needed, medium-sensitivity.

Setting Up a Secrets Vault​

Use a centralized secrets management system. AWS Secrets Manager, HashiCorp Vault, Google Secret Manager, and Azure Key Vault all work; choose based on your infrastructure.

# Example using AWS Secrets Manager (boto3)
import boto3
import json

class SecretsManager:
    def __init__(self, region: str = "us-east-1"):
        self.client = boto3.client("secretsmanager", region_name=region)
    
    def get_secret(self, secret_name: str, version_id: str = None) -> dict:
        """Retrieve a secret by name."""
        try:
            if version_id:
                response = self.client.get_secret_value(
                    SecretId=secret_name,
                    VersionId=version_id
                )
            else:
                response = self.client.get_secret_value(SecretId=secret_name)
            
            # Secret is stored as JSON
            return json.loads(response["SecretString"])
        except Exception as e:
            raise RuntimeError(f"Failed to retrieve secret {secret_name}: {e}")
    
    def set_secret(self, secret_name: str, secret_value: dict):
        """Create or update a secret."""
        try:
            self.client.put_secret_value(
                SecretId=secret_name,
                SecretString=json.dumps(secret_value)
            )
        except Exception as e:
            raise RuntimeError(f"Failed to set secret {secret_name}: {e}")
    
    def rotate_secret(self, secret_name: str):
        """Trigger rotation for a secret."""
        try:
            self.client.rotate_secret(SecretId=secret_name)
        except Exception as e:
            raise RuntimeError(f"Failed to rotate secret {secret_name}: {e}")

# Usage
secrets = SecretsManager()
api_key = secrets.get_secret("anthropic-api-key")["key"]
db_creds = secrets.get_secret("postgres-credentials")

Never Commit Secrets to Git​

Use .gitignore to exclude secrets files and environment files:

# .gitignore
.env
.env.local
.env.*.local
secrets/
*.key
*.pem
credentials.json
config/secrets/

Use Git pre-commit hooks to scan for secrets before commit:

# .git/hooks/pre-commit
#!/bin/bash
# Detect common secret patterns and block commit if found

ERROR=0

# Check for common secret patterns
if git diff --cached | grep -iE "(password|api.?key|secret|token|credentials)" | grep -v "\.md"; then
    echo "ERROR: Suspicious secret patterns detected in staged changes"
    echo "Use 'git reset <file>' to unstage and remove secrets"
    exit 1
fi

# Use git-secrets tool for advanced detection
if command -v git-secrets &> /dev/null; then
    git secrets --pre_commit_hook || exit 1
fi

exit $ERROR

Injecting Secrets into Environments​

Avoid embedding secrets in deployment configurations. Instead, inject them at runtime:

In Docker/Kubernetes:

# k8s/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-api
spec:
  template:
    spec:
      containers:
      - name: api
        image: llm-api:latest
        env:
        # NEVER hardcode: - name: ANTHROPIC_API_KEY
        #                 value: "sk-..."
        
        # Instead, reference a secret
        - name: ANTHROPIC_API_KEY
          valueFrom:
            secretKeyRef:
              name: anthropic-api-key
              key: key
        
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: postgres-credentials
              key: connection-string

Create the Kubernetes secret separately (not in Git):

kubectl create secret generic anthropic-api-key \
    --from-literal=key="sk-..." \
    -n production

kubectl create secret generic postgres-credentials \
    --from-literal=connection-string="postgres://user:pass@host:5432/db" \
    -n production

In GitHub Actions CI/CD:

name: Deploy LLM Service
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      # Secrets are injected as environment variables; never print them
      - name: Deploy to staging
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          DATABASE_URL: ${{ secrets.STAGING_DATABASE_URL }}
        run: |
          # Secret environment variables are available; never echo them
          python deploy.py --environment staging
          
          # Wrong: do not print secrets
          # echo "API_KEY=$ANTHROPIC_API_KEY"

Audit Logging for Secret Access​

Log every access to secrets for compliance and forensics. Who accessed what secret, when, and from where?

import logging
import json
from datetime import datetime

class AuditedSecretsManager(SecretsManager):
    def __init__(self, region: str = "us-east-1", audit_logger=None):
        super().__init__(region)
        self.audit_logger = audit_logger or logging.getLogger("secrets_audit")
    
    def _log_access(self, action: str, secret_name: str, user: str, success: bool):
        """Log secret access for audit."""
        audit_event = {
            "timestamp": datetime.utcnow().isoformat(),
            "action": action,
            "secret_name": secret_name,
            "user": user,  # from environment variable or request context
            "success": success,
            "source_ip": "127.0.0.1"  # placeholder
        }
        self.audit_logger.info(json.dumps(audit_event))
    
    def get_secret(self, secret_name: str, user: str):
        """Retrieve secret with audit logging."""
        try:
            secret = super().get_secret(secret_name)
            self._log_access("get", secret_name, user, success=True)
            return secret
        except Exception as e:
            self._log_access("get", secret_name, user, success=False)
            raise

# Usage
import os
secrets = AuditedSecretsManager()
api_key = secrets.get_secret(
    "anthropic-api-key",
    user=os.getenv("CURRENT_USER", "ci-pipeline")
)

Secret Rotation​

Implement automated secret rotation to limit the blast radius of a leaked secret. Even if a key is compromised, rotating it every 30-90 days ensures the attacker's access window is limited.

def rotate_anthropic_api_key():
    """Rotate Anthropic API key: create new, update services, retire old."""
    import anthropic
    
    secrets = SecretsManager()
    
    # Step 1: Create a new API key via Anthropic console (manual or API if available)
    # For demo, assume new key is generated externally
    new_api_key = input("Enter new Anthropic API key: ")
    
    # Step 2: Store new key in secrets vault with a version label
    secrets.set_secret("anthropic-api-key", {"key": new_api_key, "version": 2})
    
    # Step 3: Update running services (rolling restart)
    restart_services()
    
    # Step 4: Verify new key works
    client = anthropic.Anthropic(api_key=new_api_key)
    msg = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=10,
        messages=[{"role": "user", "content": "test"}]
    )
    assert msg.content, "New API key does not work"
    
    # Step 5: Retire old key in Anthropic console
    print("Rotation complete. Remember to retire the old key in Anthropic console.")

Schedule rotation via cron or Lambda:

# AWS Lambda: rotate secrets monthly
# Trigger: CloudWatch Events / EventBridge (cron: 0 2 1 * ? *)
function:
  name: rotate-anthropic-key
  runtime: python3.11
  handler: rotation.rotate_anthropic_api_key
  environment:
    SECRETS_REGION: us-east-1

Least Privilege Access Control​

Grant services only the secrets they need. A web API needs the Anthropic API key but not the database password; a data pipeline needs the database password but not the API key.

# Example: IAM policy for web API service
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:us-east-1:*:secret:anthropic-api-key-*"
    },
    {
      "Effect": "Deny",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:us-east-1:*:secret:postgres-credentials-*"
    }
  ]
}

Key Takeaways​

  • Never commit secrets to Git; use .gitignore and pre-commit hooks to catch leaks.
  • Use a centralized secrets vault (AWS Secrets Manager, Vault, etc.) to store and encrypt secrets.
  • Inject secrets at runtime via environment variables or Kubernetes secrets; never hardcode them.
  • Audit all secret access for compliance and forensics.
  • Rotate secrets regularly (API keys every 90 days, database credentials every 30 days) to limit blast radius.
  • Apply least-privilege access control: services access only the secrets they need.

Frequently Asked Questions​

What should I do if a secret is accidentally committed to Git?​

Immediately rotate the secret. Remove it from Git history (use git filter-branch or BFG Repo-Cleaner), but note that force-pushing rewrites history and may break other branches. For critical secrets (API keys, passwords), rotation is faster than history cleanup. Log the incident for audit.

Can I use environment variables to pass secrets in development?​

Yes, for development. Use a .env.local file (in .gitignore) for local development. Load it via python-dotenv or similar. For production, always use a secrets vault. Never commit .env files or sample .env files with real credentials.

How do I handle secrets for third-party services (Anthropic, OpenAI, etc.)?​

Store them in your secrets vault under keys like anthropic-api-key, openai-api-key. Rotate them if the service deprecates API versions. If a service requires multiple keys (e.g., org ID plus API key), store them as a JSON object in one secret: {"org_id": "...", "api_key": "..."}.

Should I encrypt secrets in my database?​

Yes, if storing user data or configuration that includes secrets. Use AES-256 encryption with a key managed by your secrets vault. Never encrypt with a key stored in the same database. Encrypt at the application layer before inserting into the database.

How often should I rotate database passwords?​

Every 30 days for production databases. Coordinate rotation with application deployment to avoid authentication failures. Use connection pooling so existing connections remain open during rotation; new connections will use the new password.

Further Reading​