Agentic AI and tools are becoming a big part of a developer’s daily life. But let’s be honest—many developers are still scratching their heads about how these systems work. How can something designed to generate text also interact with APIs and tools? Some developers want to know how models are trained to use tools, while others are curious about how they reach out to external systems.
In this post, I’ll break down the process for you and make it easier to understand what’s happening behind the scenes.
The system prompt
You probably know that AI relies heavily on prompt engineering. It’s all about giving clear and specific instructions to get the output you want. You are crafting messages that guide the model to respond appropriately.
What might surprise you is that the client application (such as the chat interface) also sends some initial instructions to the model. These instructions are called the system prompt, and they set the tone for how the model interacts with you. The system prompt is always included in the messages sent to the model, and it shapes how the model responds to your queries. It often includes details like the model’s role, allowed tasks, required tone, and how to handle questions outside its scope.
For this discussion, lets start with a simple system prompt like this:
You are a travel agent that provides short, accurate answers about visiting a city.This sets the stage for the model’s responses. With this setup, you can ask the model questions like:
What activities do you recommend for someone visiting Philadelphia?The model might respond with:
I recommend visiting the Independence National Historical Park,
exploring the Philadelphia Museum of Art, and enjoying a cheesesteak
at one of the local eateries. For outdoor activities, visit the
Schuylkill River Trail or the Fairmount Park.Now, if you ask a more specific question:
I'm visiting Philadelphia tonight. What activities do you recommend?The response might look like this:
For a night visit to Philadelphia, here are some activity suggestions based
on typical weather conditions:
- Warm or Mild Weather: Take a stroll along the Delaware River Waterfront, explore
the vibrant nightlife in Old City, or enjoy a rooftop bar like Morgan's Pier or
Barbuzzo.
- Cool or Rainy Weather: Visit the Philadelphia Museum of Art, catch a performance
at the Kimmel Center, or explore the Reading Terminal Market for late-night
dining options.Not bad, right? But it could be better. For instance, what if you could give the model a way to check the current weather conditions? Then it could provide a better answer instead of generalizations. This is where tools come in. Tools let the AI access real-time data, like weather info, to make its responses even better. The idea behind using tools is pretty simple: you teach the model how to use them.
Prompting with tools
To use tools, you tweak the system prompt to explain what tools are available, how to use them, and how to include their responses in the final answer. Essentially, you’re teaching the model how to handle missing information by asking for more details. Usually, the model tries to write a natural language response, but in this case a structured response, like JSON, is more useful. The client application can then use this JSON to call the tool and re-prompt the model for a final answer.
The flow looks like this:
As you can see, the model doesn’t directly call the tool. Instead, the client orchestrates the interaction, calling the tool and feeding the response back to the model. This orchestration is the essence of agentic AI and why it’s so powerful.
This approach also helps bypass token limits. Clients can break tasks into steps, call different tools or models, and combine the results for a final response.
Implementing tool prompts
To make this work, you need to update the system prompt with three key details:
- Information about the tools, what they do, and their parameters.
- Instructions for using the tools, including how to format the request and when to stop responding. This last part is crucial because it tells the model when to wait for the tool’s response before continuing.
- Guidelines for interpreting tool responses to answer the user’s question.
Here’s an example of a modified prompt:
You are a travel agent that provides brief and accurate answers to user queries.
You have access to the following tool to assist in obtaining accurate information:
- tool: get-weather
description: Retrieves details about the current weather conditions for a specific city.
parameters:
- name: city
required: true
description: The name of the city for which weather information is needed.
- name: time
required: false
description: The time for the weather information, formatted as "HH:mm" using 24-hour notation.
Instructions for interaction:
1. If a query requires information from the tool:
- Respond with a JSON object specifying the tool name and all required parameter values.
- Include optional parameters if they are relevant to the query.
- Example format:
```
{
"tool": "get-weather",
"parameters": {
"city": "cityName",
"time": "HH:mm"
}
}
```
- After generating the tool request, stop responding until you receive the tool's output.
2. Tool responses are provided as JSON in this format:
```
{
"tool": "get-weather",
"response": "Weather details for the requested input."
}
```
Use the response content to craft your final answer.
Ensure responses are concise, relevant, and directly address the query. With this prompt, the user can ask about activities in Philadelphia, and the model can request more information. In this example, the model might now respond with this request:
{
"tool": "get-weather",
"parameters": {
"city": "Philadelphia",
"time": "19:00"
}
}The client application detects this JSON, calls the tool, and sends the response back to the model:
{
"tool": "get-weather",
"response": "65°F. Cloudy. Slight chance of rain."
}The model then uses this info to generate a more accurate response:
Tonight in Philadelphia, it will be 65°F and cloudy with a slight chance of rain. Here are some indoor activities you might enjoy:
- Museum of Art: Explore the extensive collection of art.
- Pat's or Geno's: Famous for their cheesesteaks.
- Live Music: Check out local venues for live music performances.Building better tools
As you can see, the way you prompt the model can drastically change its responses, giving the client application the ability to do more. This is the core idea behind agentic AI. The client can ask the model for details, orchestrate tool calls, and even re-prompt the model for better results.
Modern models are also getting better at understanding tools and structured data. This means you can use tools with fewer tokens and less detail in the system prompt. Older models often required more detailed instructions to get similar results.
Taking it further, developers might wonder if they can make this process more seamless. For example, what if you could use an API to query available tools, dynamically build the system prompt, and invoke tools as needed? This is possible and is the premise behind the Model Context Protocol (MCPP). By standardizing how tools, resources, and prompts are provided, you can create flexible systems for building agentic applications.
As these technologies evolve, we’ll see more applications using tools and APIs to enhance AI responses. This will make interactions smarter and more dynamic, helping developers build applications that adapt to user needs in real time. Now that you know how it works, I hope you’re excited to build your own applications and leverage these techniques!