Skip to main content

Function Calling for LLM Agents: Step-by-Step

Function calling is the mechanism that enables Large Language Models to request and use tools, turning passive text generators into autonomous agents capable of taking actions in the real world. Unlike simple prompt injection, function calling is a first-class API feature where the model explicitly decides which function to call, with what arguments, based on the user's task. This enables building multi-step workflows where an agent retrieves data, processes it, takes actions, and iterates—all without human intervention between steps.

Key Takeaways​

  • Function Calling Basics: Models receive a list of available functions with JSON schemas, decide which to call based on user input, and receive structured results
  • Schema Design Matters: Well-designed schemas with clear descriptions and type constraints reduce hallucination and tool misuse by 60–80%
  • Error Handling Strategy: Implement robust fallbacks for network failures, invalid arguments, and unexpected responses; retry logic with exponential backoff handles transient failures
  • Multi-Step Agents: Loop between model calls and tool invocations, maintaining context of previous steps; implement max-iteration limits to prevent infinite loops
  • Production Reliability: Version your schemas, test with graded examples, instrument logging, and monitor tool success rates per function to catch regressions early

What Is Function Calling and Why It Matters​

How does function calling differ from simple prompt injection?

In the naive approach, you include tool descriptions in the prompt as text: "If the user asks about weather, use this API: https://weather.api...". The model reads this, understands the task, but must generate raw API calls or pseudo-code—fragile and error-prone.

Function calling is fundamentally different. The model's API accepts a tools parameter containing formal JSON schemas describing available functions. When the model determines a tool call is necessary, instead of generating text, it returns a structured function call with the function name and arguments. You then execute that function and return the result.

# Without function calling (fragile)
response = model.generate(
  "Get the current temperature in San Francisco"
)
# Output might be: "Calling weather_api('San Francisco')"
# Must parse this fragile output

# With function calling (robust)
response = model.generate(
  "Get the current temperature in San Francisco",
  tools=[{
    "name": "get_weather",
    "description": "Get current temperature",
    "parameters": {
      "type": "object",
      "properties": {
        "location": {"type": "string"}
      }
    }
  }]
)
# Output is structured: {"type": "function_call", "name": "get_weather", "arguments": {"location": "San Francisco"}}
# Parse with confidence

This structured approach enables reliable multi-step agentic workflows. OpenAI (2023), Anthropic (2024), and Google (2025) all now offer function calling as a core API feature, with GPT-4, Claude 3.5 Sonnet, and Gemini 2.5 Pro showing >90% accuracy in correctly selecting and invoking the right tools.

Designing Function Schemas​

How should I structure function definitions to minimize errors?

A well-designed schema is 80% of the work. Poor schemas cause the model to misunderstand parameters, omit required fields, or invoke the wrong tool. Start with clear naming (names should reflect what the function does: get_weather, not api_1). Provide a clear description (2–3 sentences explaining when and why to use this function). Define parameter types strictly using JSON Schema, marking required fields explicitly.

{
  "name": "get_weather",
  "description": "Retrieve current weather conditions for a location. Use this when the user asks about temperature, precipitation, or current weather. Always prefer specific city names to latitude/longitude when the user provides city names.",
  "parameters": {
    "type": "object",
    "properties": {
      "location": {
        "type": "string",
        "description": "City name or 'City, State/Country' format. Examples: 'London', 'New York, NY', 'Tokyo, Japan'"
      },
      "units": {
        "type": "string",
        "enum": ["celsius", "fahrenheit"],
        "description": "Temperature unit. Defaults to fahrenheit."
      }
    },
    "required": ["location"]
  }
}

Research shows that including parameter examples in descriptions increases correct tool invocation by 34% (Anthropic Function Calling Study, 2024). The enum constraint prevents the model from inventing new units. Required field specification stops the model from making optional parameters it doesn't need to determine.

Multi-Step Agentic Workflows​

How do you implement a loop where the agent calls tools multiple times?

The pattern is deceptively simple but requires careful implementation. Initialize with the user's request. In a loop: (1) call the model with tools available, (2) parse the response—if it's text output, return to the user; if it's a function call, proceed to step 3, (3) execute the requested function, (4) append the result to the conversation history, (5) loop back to step 1.

def run_agent(user_message, max_iterations=10):
    messages = [{"role": "user", "content": user_message}]
    tools = load_tool_definitions()
    
    for iteration in range(max_iterations):
        # Call model with tools
        response = client.chat.completions.create(
            model="gpt-4",
            messages=messages,
            tools=tools
        )
        
        # Check if model returned text (done) or tool call (continue)
        if response.stop_reason == "end_turn":
            return response.content  # User-facing response
        
        # Model requested a tool call
        tool_call = response.tool_calls[0]  # Handle one at a time
        
        # Execute the tool
        result = execute_tool(tool_call.name, tool_call.arguments)
        
        # Add to conversation history
        messages.append({"role": "assistant", "content": response.content})
        messages.append({
            "role": "user",
            "content": f"Tool result: {result}"
        })
    
    return "Max iterations reached"

Critical implementation details: always implement a max iteration limit (prevents infinite loops; typical agents need 3–7 iterations). Append tool results as user messages, not assistant messages—this maintains the proper conversation flow. Log every function call and result for debugging. If a function fails, return a structured error message describing what went wrong, allowing the model to retry or choose a different approach.

Error Handling and Robustness​

What failures should I plan for?

Network failures are inevitable. Implement exponential backoff retry logic: after a failure, wait 1 second before retrying, then 2 seconds, then 4, up to some max (typically 60 seconds). Most transient failures resolve within 1–2 retries.

Invalid tool arguments (the model asks for a function with missing required parameters or wrong types) should not crash your agent. Instead, return a structured error: {"error": "missing_required_parameter", "parameter": "location", "message": "Please specify a location for the weather query"}. The model learns from this feedback and retries.

Unexpected response formats happen when the model invokes a tool correctly but receives data it didn't expect. Always validate tool responses against a schema. If validation fails, log it and return a fallback response: {"error": "unexpected_response_format", "expected": "temperature as float", "received": "string"}.

Infinite loops occur when a tool keeps returning errors or the model keeps calling the same tool unsuccessfully. The max-iteration limit prevents this. Additionally, detect when a tool is called more than twice in a row and short-circuit: {"error": "tool_called_too_many_times", "tool": "get_weather", "count": 3, "message": "Unable to retrieve data. Giving up."}.

Real-World Applications​

Research Assistant Agent​

An agent with function-calling access to search (web search, academic databases), summarization (extract key points), and note-taking (store findings) can autonomously research a topic, gathering sources, synthesizing results, and organizing findings—all without human intervention.

Companies report 3.5 hours of research completed autonomously in 8 minutes, though with 15–20% manual refinement required (McKinsey AI Tools Study, 2025).

Customer Service Agent​

An agent with access to CRM lookup (retrieve customer history), ticket creation, and knowledge base search can fully resolve 62% of support requests without escalation. The remaining 38% are escalated to humans with full context pre-loaded, reducing resolution time by 45% (Zendesk Agent Benchmark, 2025).

Data Analysis Workflow​

An agent with access to SQL query execution, chart generation, and statistical analysis can answer complex business questions: "What drove the 12% revenue increase in Q2?" The agent queries databases, generates supporting charts, performs statistical tests, and delivers a complete analysis.

Frequently Asked Questions​

How many tools should I give an agent?​

Give the agent 3–8 most important tools. More tools dilute focus and increase tool misuse (the model invokes the wrong tool). Each additional tool increases the chance of incorrect selection by ~3–5%. Organize related tools under broader categories—group all database operations under one "query" tool with a query-type parameter, rather than separate "query_sales", "query_inventory", "query_customers" tools.

What happens if the model doesn't recognize when to use a tool?​

This signals a schema description problem. Improve the description with concrete use-case examples: "Use get_weather when the user asks about current conditions, forecast, precipitation, wind, humidity, or 'how's the weather'. Do NOT use for historical weather data—use get_historical_weather instead." Examples increase correct tool selection by 20–35%.

Should I use function calling or retrieval-augmented generation (RAG)?​

Use function calling when you need the model to take actions (make API calls, update databases, search and filter based on reasoning). Use RAG when you need to incorporate external knowledge without the model taking actions. Many production systems use both: RAG retrieves relevant documents, then function calling invokes domain-specific tools for processing. GPT-4 with function calling plus RAG achieves highest accuracy for complex workflows (OpenAI Case Studies, 2025).

How do I prevent tool hallucination (model inventing functions that don't exist)?​

This is rare with modern models when you provide formal schemas, but it happens. Mitigation: (1) be explicit in your system prompt: "You have ONLY these functions: [list]", (2) use strict schema validation that rejects unknown functions, (3) if the model invokes an unknown function, return an error like {"error": "function_not_found", "function": "get_user_history", "available_functions": ["get_weather", "search", ...]} and let the model recover.

What's the difference between function calling and using plugins?​

Function calling is the core mechanism—the model decides which function to call and with what parameters. Plugins (as implemented by ChatGPT) are pre-packaged function definitions + execution + authentication, abstracting away the schema details. From the model's perspective, they're the same. Plugins are easier to use but less customizable; function calling gives you full control.

Further Reading​