Skip to main content

JSON Grammar Constraints: Structured Data Reliability

JSON schema constraints are the most practical application of constrained decoding for real-world LLM systems. Instead of hoping a model's JSON output is valid, you enforce schema compliance at the token level: required fields must be present, field types must match (string, number, boolean, object, array), enum values must come from a predefined list, and string lengths can be bounded. When combined with logit masking, the decoder guarantees that the output is syntactically valid JSON that also satisfies every constraint in your schema.

This eliminates three categories of failure common in unconstrained JSON generation: malformed JSON (unclosed braces, mismatched quotes), invalid structure (missing required fields, unexpected fields, wrong types), and semantic errors (enum values outside the allowed set). For applications that feed LLM output directly into downstream APIs, databases, or code execution, JSON schema constraints are essential.

JSON Schema Basics​

A JSON Schema is a declarative specification of what valid JSON looks like. Here's a simple example:

{
  "type": "object",
  "properties": {
    "name": {"type": "string"},
    "age": {"type": "integer", "minimum": 0, "maximum": 150},
    "email": {"type": "string", "format": "email"},
    "tags": {"type": "array", "items": {"type": "string"}}
  },
  "required": ["name", "age"],
  "additionalProperties": false
}

This schema says:

  • Top level is an object.
  • name (required): string.
  • age (required): integer, 0–150.
  • email (optional): string, email format.
  • tags (optional): array of strings.
  • No other properties allowed (additionalProperties: false).

Example valid JSON:

{
  "name": "Alice",
  "age": 30,
  "email": "[email protected]",
  "tags": ["engineer", "rust"]
}

Example invalid JSON (against the schema):

{
  "name": "Bob"
  // Missing required 'age' field
}
{
  "name": "Charlie",
  "age": "thirty",  // Type mismatch: should be integer
  "email": "[email protected]"
}

Using JSON Schema Constraints with Outlines​

Outlines makes JSON schema constraints nearly effortless:

from outlines import models, generate
from pydantic import BaseModel
import json

# Define schema using Pydantic (Outlines auto-converts to JSON Schema)
class PersonInfo(BaseModel):
    name: str
    age: int
    email: str
    tags: list[str] = []  # Optional, default empty

# Load model and create constrained generator
model = models.transformers("mistralai/Mistral-7B-v0.1")
generator = generate.json(model, PersonInfo)

# Generate JSON that matches schema exactly
prompt = "Extract person info from: John Doe, 28, [email protected], tags: engineer, python"
result = generator(prompt, max_tokens=200)

print(result)
# Output (guaranteed valid, schema-compliant JSON):
# {"name": "John Doe", "age": 28, "email": "[email protected]", "tags": ["engineer", "python"]}

Under the hood, Outlines:

  1. Converts the Pydantic model to JSON Schema.
  2. Compiles the schema into a finite-state machine (FSM).
  3. At each token, masks logits for tokens that would violate the schema.
  4. Guarantees the output is valid JSON + matches the schema.

Raw JSON Schema (Without Pydantic)​

If you prefer to write JSON Schema directly:

from outlines import models, generate
import json

schema_dict = {
    "type": "object",
    "properties": {
        "decision": {"enum": ["approve", "reject", "pending"]},
        "reason": {"type": "string"},
        "confidence": {"type": "number", "minimum": 0.0, "maximum": 1.0}
    },
    "required": ["decision", "reason"],
    "additionalProperties": False
}

model = models.transformers("mistralai/Mistral-7B-v0.1")
generator = generate.json(model, schema_dict)

prompt = "Loan decision for applicant: income $80k, credit score 650. Decide:"
result = generator(prompt, max_tokens=150)

print(result)
# Guaranteed to match:
# {"decision": "approve", "reason": "Sufficient income and acceptable credit", "confidence": 0.85}

Complex Nested Schemas​

JSON Schema shines for nested structures:

from pydantic import BaseModel
from typing import List
from outlines import models, generate

class Address(BaseModel):
    street: str
    city: str
    zip_code: str

class Person(BaseModel):
    name: str
    age: int
    addresses: List[Address]
    is_active: bool = True

model = models.transformers("mistralai/Mistral-7B-v0.1")
generator = generate.json(model, Person)

prompt = """
Extract person data from:
Name: Alice Smith
Age: 35
Addresses:
  - 123 Main St, Springfield, 12345
  - 456 Oak Ave, Shelbyville, 67890
Active: yes
"""

result = generator(prompt, max_tokens=400)
print(result)
# Output (valid JSON, schema-compliant):
# {
#   "name": "Alice Smith",
#   "age": 35,
#   "addresses": [
#     {"street": "123 Main St", "city": "Springfield", "zip_code": "12345"},
#     {"street": "456 Oak Ave", "city": "Shelbyville", "zip_code": "67890"}
#   ],
#   "is_active": true
# }

The decoder navigates nested objects and arrays, maintaining the FSM state as it generates each field and array element. If the model tries to output a field not in the schema, the token is masked out.

Enum Constraints in JSON Schemas​

Enums are especially powerful for constrained generation:

from pydantic import BaseModel
from enum import Enum
from outlines import models, generate

class DecisionType(str, Enum):
    APPROVE = "approve"
    REJECT = "reject"
    PENDING = "pending"

class DecisionRecord(BaseModel):
    decision: DecisionType
    reason: str
    severity: int  # 1-3 scale

model = models.transformers("mistralai/Mistral-7B-v0.1")
generator = generate.json(model, DecisionRecord)

prompt = "Make a decision: approve or reject? Loan app for $100k, credit 750."
result = generator(prompt, max_tokens=100)

print(result)
# Output: {"decision": "approve", "reason": "Strong credit score", "severity": 1}
# Guaranteed: decision is one of the three enum values, never "accept", "declined", or custom text

Handling Edge Cases and Constraints​

Large arrays: Schema can specify maxItems to limit array length:

class TaggedContent(BaseModel):
    title: str
    tags: list[str]  # Can add constraints:
    # tags: list[str] = Field(..., max_items=5)  # Max 5 tags

String length and pattern:

from pydantic import Field

class User(BaseModel):
    username: str = Field(..., min_length=3, max_length=20, pattern=r"^[a-zA-Z0-9_]+$")
    password: str = Field(..., min_length=8)

Numeric ranges:

class Product(BaseModel):
    price: float = Field(..., ge=0, le=100000)  # >= 0, <= 100000
    rating: int = Field(..., ge=1, le=5)  # 1-5 star rating

Performance: Schema Size and Generation Speed​

Larger schemas (many fields, deep nesting) require more complex FSMs, which can slow generation:

Schema complexityRelative speedTypical overhead
Tiny (5 fields, flat)1.0x5–10%
Medium (15 fields, 2 levels deep)0.95x10–20%
Large (50+ fields, 4+ levels)0.85x20–35%
Very large (100+ fields, complex nesting)0.7x–0.8x35–50%

For most applications (APIs with 5–20 fields), overhead is negligible. For very large schemas, consider splitting into multiple generation calls.

Debugging Failed Constraints​

If generation fails to produce valid JSON (shouldn't happen, but bugs exist):

import json
from pydantic import ValidationError

try:
    # Try to validate the output manually
    result_dict = json.loads(result)
    Person.model_validate(result_dict)
    print("Valid!")
except json.JSONDecodeError as e:
    print(f"Invalid JSON: {e}")
except ValidationError as e:
    print(f"Schema violation: {e}")
    print(f"Generated: {result}")

If the constraint fails:

  1. Check your schema for typos or overly restrictive constraints.
  2. Verify the model can theoretically generate the required format (test unconstrained first).
  3. Check the Outlines/inference engine version; update if outdated.
  4. For complex schemas, try a larger model (more capable).

Best Practices for JSON Schema Constraints​

1. Start simple: Begin with a basic schema (5–10 fields) and expand.

2. Use Pydantic: It's more readable than raw JSON Schema and easier to version-control.

from pydantic import BaseModel, Field

class OutputRecord(BaseModel):
    """Docstrings are useful for understanding intent."""
    action: str = Field(..., description="The action to take")
    params: dict = Field(default_factory=dict)

3. Make required fields explicit: Only mark fields required: true if the model must fill them. Optional fields with sensible defaults are more user-friendly.

4. Avoid overly restrictive regex patterns: If you use pattern in a string field, keep it simple (e.g., ^[A-Z][a-z]+$ for names). Complex patterns slow constraint compilation.

5. Test with unconstrained generation first: Run a few unconstrained examples to see what the model naturally produces, then design your schema to match.

6. Use enums for categorical choices: Instead of free-form strings, enumerate valid values. This is faster and more reliable.

Beyond JSON: Schema Evolution​

As your system evolves, you'll need to version schemas:

class UserV1(BaseModel):
    name: str
    age: int

class UserV2(BaseModel):
    name: str
    age: int
    email: str  # New field

# At runtime, specify which version to use
generator_v1 = generate.json(model, UserV1)
generator_v2 = generate.json(model, UserV2)

For backwards compatibility, older systems can still parse newer JSON (add additionalProperties: true to allow extra fields). Newer systems should gracefully handle old JSON (use Pydantic's extra="ignore").

Key Takeaways​

  • JSON schema constraints enforce both syntax validity and schema compliance at the token level.
  • Outlines and similar libraries convert schemas (Pydantic or raw JSON Schema) into FSMs that mask invalid tokens.
  • Constraints eliminate malformed JSON, missing/extra fields, type errors, and invalid enum values.
  • Complex nested schemas (objects, arrays) are supported; typical overhead is 10–20% slower generation.
  • Pydantic models are preferred for readability and type safety; raw JSON Schema for maximum flexibility.

Frequently Asked Questions​

What if my schema requires a field but the model doesn't provide information for it?​

The constraint forces the model to generate something for required fields, even if it's a placeholder. If you want truly optional fields, mark them optional in the schema (remove from required). Alternatively, use a default value: Field(..., default="unknown").

Can I mix constrained and unconstrained fields?​

Not directly. Either the entire output is constrained (matches the schema) or it's unconstrained. However, you can use a union type to allow multiple schemas: Union[TypeA, TypeB]. The model must match one of them.

How do I add custom validation beyond JSON Schema?​

Pydantic supports field_validator and model_validator for custom logic:

from pydantic import BaseModel, field_validator

class Age(BaseModel):
    value: int
    
    @field_validator('value')
    @classmethod
    def check_age(cls, v):
        if v < 0 or v > 150:
            raise ValueError('Age must be 0–150')
        return v

However, custom validators run after generation, so they won't constrain tokens. For runtime validation only.

Do all inference engines support JSON schema constraints?​

Most modern ones (Outlines, llama.cpp, vLLM, XGrammar) do. Proprietary APIs (OpenAI, Anthropic) have their own modes (e.g., response_format=json on OpenAI, JSON mode on Claude). Check your engine's documentation.

What's the difference between JSON schema constraints and instruction tuning?​

Instruction tuning trains a model to output JSON (behavioral); constraints make violations structurally impossible (hard guarantee). Constraints are stronger but less flexible. A hybrid approach (tuned model + constraints) gives both reliability and performance.

Further Reading​