Schema Validation and Error Handling
A tool call fails validation when the LLM sends data that doesn't match the schema: a required field is missing, a type is wrong, or a constraint (min/max, enum, pattern) is violated. How you handle these errors determines system reliability. A poor approach logs the error and silently fails. A robust approach validates early, provides actionable feedback to the LLM, and falls back gracefully. This article teaches you validation strategies, error recovery patterns, and testing approaches to catch schema issues before production.
Validation Strategies
Strategy 1: Strict Validation (Fast-Fail)
Validate immediately upon receiving the tool call. If validation fails, return an error. The LLM sees the error and can adjust:
import jsonschema
schema = {
"type": "object",
"properties": {
"query": {"type": "string", "minLength": 1},
"limit": {"type": "integer", "minimum": 1, "maximum": 100}
},
"required": ["query"]
}
def validate_tool_call(data: dict) -> tuple[bool, str]:
"""Validate and return (is_valid, error_message)."""
try:
jsonschema.validate(instance=data, schema=schema)
return True, ""
except jsonschema.ValidationError as e:
# Format the error clearly
path = " -> ".join(str(p) for p in e.path) or "root"
return False, f"At {path}: {e.message}"
# Usage
tool_call = {"query": "", "limit": 25} # Empty query
is_valid, error = validate_tool_call(tool_call)
if not is_valid:
# Return error to LLM
return {"status": "error", "error": error}
Pros: Fast, clear errors, LLM can retry immediately. Cons: If validation fails, the tool call fails. No fallback.
Strategy 2: Lenient Validation with Coercion
Attempt to coerce invalid data to valid data before rejecting:
def validate_and_coerce(data: dict, schema: dict) -> tuple[dict, list[str]]:
"""Try to coerce data to match schema. Return (coerced_data, warnings)."""
warnings = []
coerced = {}
for prop_name, prop_schema in schema["properties"].items():
if prop_name not in data:
if prop_name in schema["required"]:
raise ValueError(f"Missing required field: {prop_name}")
continue
value = data[prop_name]
expected_type = prop_schema.get("type")
# Try to coerce
if expected_type == "integer" and isinstance(value, str):
try:
coerced[prop_name] = int(value)
warnings.append(f"Coerced {prop_name} from string to integer")
except ValueError:
raise ValueError(f"Cannot coerce {prop_name}='{value}' to integer")
elif expected_type == "number" and isinstance(value, str):
try:
coerced[prop_name] = float(value)
warnings.append(f"Coerced {prop_name} from string to number")
except ValueError:
raise ValueError(f"Cannot coerce {prop_name}='{value}' to number")
elif expected_type == "string" and not isinstance(value, str):
coerced[prop_name] = str(value)
warnings.append(f"Coerced {prop_name} to string")
else:
coerced[prop_name] = value
# Validate constraints
if "minimum" in prop_schema and coerced[prop_name] < prop_schema["minimum"]:
raise ValueError(f"{prop_name} is below minimum {prop_schema['minimum']}")
return coerced, warnings
# Usage
try:
tool_call = {"query": "python", "limit": "25"} # limit is string
coerced, warnings = validate_and_coerce(tool_call, schema)
result = search(**coerced)
return {"status": "success", "result": result, "warnings": warnings}
except ValueError as e:
return {"status": "error", "error": str(e)}
Pros: Handles minor type mismatches (string "42" → int 42). Cons: Coercion can hide bugs; may not catch invalid data.
Strategy 3: Partial Validation with Defaults
If a field is optional and missing, use a default. Validate only required fields strictly:
def validate_with_defaults(data: dict, schema: dict) -> dict:
"""Validate required fields; use defaults for optional fields."""
validated = {}
# Validate required fields
for field in schema.get("required", []):
if field not in data:
raise ValueError(f"Missing required field: {field}")
validated[field] = data[field]
# Fill in optional fields with defaults
for prop_name, prop_schema in schema["properties"].items():
if prop_name in data:
validated[prop_name] = data[prop_name]
elif "default" in prop_schema:
validated[prop_name] = prop_schema["default"]
return validated
# Usage
schema_with_defaults = {
"type": "object",
"properties": {
"query": {"type": "string"},
"limit": {"type": "integer", "default": 10},
"sort_by": {"type": "string", "enum": ["date", "relevance"], "default": "relevance"}
},
"required": ["query"]
}
tool_call = {"query": "python"} # Missing limit and sort_by
validated = validate_with_defaults(tool_call, schema_with_defaults)
# {"query": "python", "limit": 10, "sort_by": "relevance"}
Pros: Handles optional fields gracefully; clear required vs. optional distinction. Cons: Doesn't detect explicit wrong types.
Error Recovery Patterns
Pattern 1: Return Structured Error
When validation fails, return a structured error the LLM can parse:
def handle_tool_call(tool_call: dict) -> dict:
"""Handle a tool call with error recovery."""
try:
# Validate
jsonschema.validate(instance=tool_call, schema=schema)
# Call function
result = search(**tool_call)
return {
"status": "success",
"result": result
}
except jsonschema.ValidationError as e:
# Validation error
return {
"status": "error",
"error_type": "validation_error",
"error": str(e.message),
"path": list(e.path),
"suggestion": generate_suggestion(e)
}
except Exception as e:
# Runtime error
return {
"status": "error",
"error_type": "runtime_error",
"error": str(e)
}
def generate_suggestion(validation_error):
"""Provide a helpful suggestion based on the error."""
if "minLength" in str(validation_error):
return "String is too short. Provide at least 1 character."
elif "minimum" in str(validation_error):
return "Number is too small. Increase the value."
elif "enum" in str(validation_error):
return f"Invalid choice. Use one of: {get_enum_values(validation_error)}"
return "Check your input and try again."
Pattern 2: Fallback with Defaults
If validation fails, use sensible defaults and retry:
def call_tool_with_fallback(tool_call: dict, schema: dict,
defaults: dict) -> dict:
"""Call with fallback to defaults on error."""
try:
# Try strict validation first
jsonschema.validate(instance=tool_call, schema=schema)
return search(**tool_call)
except jsonschema.ValidationError:
# Fall back to filling in missing fields from defaults
fallback_call = {**defaults, **tool_call}
try:
jsonschema.validate(instance=fallback_call, schema=schema)
return search(**fallback_call)
except jsonschema.ValidationError as e:
# Still fails, give up
return {"status": "error", "error": str(e)}
# Usage
tool_call = {"query": ""} # Empty query (invalid)
defaults = {"query": "*", "limit": 10} # Fallback query
result = call_tool_with_fallback(tool_call, schema, defaults)
Pattern 3: Retry Loop with Feedback
Ask the LLM to retry if validation fails:
from anthropic import Anthropic
def tool_call_with_retry(client: Anthropic, tool_call: dict,
schema: dict, max_retries: int = 3) -> dict:
"""Allow the LLM to retry if validation fails."""
retries = 0
while retries < max_retries:
# Validate
try:
jsonschema.validate(instance=tool_call, schema=schema)
return {"status": "success", "result": search(**tool_call)}
except jsonschema.ValidationError as e:
retries += 1
if retries >= max_retries:
return {"status": "error", "error": str(e)}
# Ask the model to retry
feedback = f"Validation failed: {e.message}. Please retry with valid parameters."
# (In a real system, you'd send this feedback to the model
# and have it generate a new tool call)
tool_call = retry_with_feedback(client, feedback)
return {"status": "error", "error": "Max retries exceeded"}
Testing Schemas for Robustness
Fuzz Testing: Send Invalid Data
Test your schema with intentionally malformed data:
import pytest
from hypothesis import given, strategies as st
schema = {...}
class TestSchemaValidation:
def test_missing_required_field(self):
"""Test that missing required fields are caught."""
data = {"limit": 10} # Missing required 'query'
with pytest.raises(jsonschema.ValidationError):
jsonschema.validate(instance=data, schema=schema)
def test_wrong_type(self):
"""Test that wrong types are caught."""
data = {"query": 12345} # query should be string
with pytest.raises(jsonschema.ValidationError):
jsonschema.validate(instance=data, schema=schema)
def test_constraint_violation(self):
"""Test that constraint violations are caught."""
data = {"query": "", "limit": 200} # limit exceeds maximum
with pytest.raises(jsonschema.ValidationError):
jsonschema.validate(instance=data, schema=schema)
@given(query=st.text(min_size=1, max_size=100),
limit=st.integers(min_value=1, max_value=100))
def test_valid_inputs(self, query: str, limit: int):
"""Test that valid inputs pass."""
data = {"query": query, "limit": limit}
jsonschema.validate(instance=data, schema=schema) # Should not raise
# Run tests
pytest.main([__file__, "-v"])
LLM-Based Testing: Ask the Model to Break It
Prompt an LLM to generate invalid tool calls and test your validation:
def generate_invalid_tool_calls(schema: dict, count: int = 10) -> list[dict]:
"""Use an LLM to generate intentionally invalid tool calls."""
prompt = f"""
Given this JSON Schema:
{json.dumps(schema, indent=2)}
Generate {count} intentionally invalid tool calls that violate the schema.
Each should be missing required fields, have wrong types, or violate constraints.
Return a JSON array of tool calls.
"""
response = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1000,
messages=[{"role": "user", "content": prompt}]
)
# Parse the response and extract tool calls
# ...
return invalid_calls
# Test that all of them are rejected
invalid_calls = generate_invalid_tool_calls(schema)
for call in invalid_calls:
try:
jsonschema.validate(instance=call, schema=schema)
print(f"ERROR: Invalid call passed validation: {call}")
except jsonschema.ValidationError:
pass # Expected
Validation in Production
Metrics and Monitoring
Track validation failures to detect schema issues early:
from prometheus_client import Counter
validation_errors = Counter(
"tool_validation_errors_total",
"Total validation errors",
["tool_name", "error_type"]
)
def handle_tool_call(tool_call: dict, tool_name: str) -> dict:
try:
jsonschema.validate(instance=tool_call, schema=schema)
return search(**tool_call)
except jsonschema.ValidationError as e:
validation_errors.labels(
tool_name=tool_name,
error_type=type(e).__name__
).inc()
return {"status": "error", "error": str(e)}
Alerting on High Error Rates
Alert if validation error rate exceeds a threshold:
from collections import defaultdict
from datetime import datetime, timedelta
class SchemaHealthMonitor:
def __init__(self, error_threshold: float = 0.05):
self.error_threshold = error_threshold
self.errors = defaultdict(lambda: {"count": 0, "window_start": datetime.now()})
def record(self, tool_name: str, is_valid: bool):
now = datetime.now()
entry = self.errors[tool_name]
# Reset window if expired
if now - entry["window_start"] > timedelta(minutes=1):
entry = {"count": 0, "window_start": now}
self.errors[tool_name] = entry
if not is_valid:
entry["count"] += 1
# Check if error rate is too high
total_calls = self._get_total_calls(tool_name)
error_rate = entry["count"] / total_calls if total_calls > 0 else 0
if error_rate > self.error_threshold:
alert(f"High validation error rate for {tool_name}: {error_rate:.1%}")
monitor = SchemaHealthMonitor()
Key Takeaways
- Validate tool calls immediately after receipt using jsonschema or equivalent.
- Return structured errors with error type, location, and suggestion.
- Implement fallback strategies: coercion, defaults, or retry loops.
- Test schemas with fuzz testing and LLM-based testing.
- Monitor validation error rates in production and alert on anomalies.
Frequently Asked Questions
Should I log validation errors?
Yes, always log validation failures with context: the tool call, the schema, the error. This helps debug and detect patterns.
What's the best error message to return to the LLM?
Be specific: "query is missing" is better than "validation failed". Include the field name, constraint violated, and an example of valid input if possible.
Can I use a more lenient validator to increase acceptance?
Carefully. Lenient validation (coercion) can hide bugs. If the LLM sends string "42" for an integer, coercing it works, but it might indicate the model doesn't understand the schema. Log coercions and review them.
Should I block the tool call or return a warning?
Blocking is safer. If validation truly fails, the tool can't run correctly. Return a clear error. The LLM can then adjust and retry.
How do I test whether my validation is correct?
Use property-based testing (Hypothesis in Python, fast-check in JS): generate random valid and invalid data and verify your validator accepts/rejects appropriately.