Tool call round trip from the eyes of a model

We know so little about how the current SoTA large language models process what they receive, and providers are only getting cagier about the internals. We're also losing the explicit knobs that used to control behavior. Parameters like temperature are being replaced by more semantic ones like reasoning effort, which I covered in my Understanding the deprecation of temperature parameter in LLMs post. Here's a non-exhaustive list of what we don't know or don't have access to:

• Access to raw reasoning tokens [1]
• How tools and tool results are parsed by the model [2]
• Internals of reasoning effort parameter [3]
• What does verbosity parameter do [4]

It's tempting to think this doesn't matter if you're just using the model as a product or building on top of it. But the lack of documentation costs you in two ways: directly in money, and in quality.

Prompt caching [5] is offered by every major provider and can cut time-to-first-token latency by up to 80% and input token costs by up to 90%. It works by caching the key and value tensors of a prompt across all layers and heads. The catch: the new request's prefix has to exactly match what's cached.

Several things silently break the cache even when the prefix and model stay the same that I found to be not very well documented:

• Reasoning effort parameter
• Changing any tool schema but it won't break the cache if you are adding a new tool via tool search
• Changing the verbosity parameter
• Caching thinking tokens is treated differently by different providers

It's hard to debug issues like the above without understanding how the prompt is processed by the model. Since we don't have access to closed-source chat templates, I'm using GPT-OSS and Qwen3 to render the same conversation through the chat templates of the models. GPT-OSS is trained on the Harmony response format [6], designed to mimic the OpenAI Responses API. Qwen3 is the most popular open-weight model, so it's a useful baseline.

Take one weather request, tool call, and the round trip back:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1")  # vLLM, Ollama, llama.cpp, ...

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {"type": "string"},
                "units": {"type": "string", "enum": ["celsius", "fahrenheit"]},
            },
            "required": ["city"],
        },
    },
}]

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What's the weather in Paris?"},
]

response = client.chat.completions.create(
    model="openai/gpt-oss-20b",  # or Qwen/Qwen3-8B, same payload
    messages=messages,
    tools=tools,
)

call = response.choices[0].message.tool_calls[0]

messages += [
    response.choices[0].message,  # the tool call, appended back verbatim
    {
        "role": "tool",
        "tool_call_id": call.id,
        "content": get_weather(**json.loads(call.function.arguments)),
    },
]

response = client.chat.completions.create(
    model="openai/gpt-oss-20b",
    messages=messages,
    tools=tools,
)

The tabs below render that same conversation through the unmodified chat templates of gpt-oss-20b and Qwen3-8B. Filled pills are special tokens. Dashed pills are vocabulary tokens not marked special.

For comparison, the other families I looked at while researching this:

Providers differ in more than tool-call syntax. Reasoning traces and multi-modal input get handled differently too.

Historically, Anthropic has used a variant of XML called ANTML for tool calls and tool results, with a different hierarchy of roles. OpenAI orders System -> Developer -> Tools -> Messages, while Anthropic orders Tools -> System -> Messages. That ordering changes how you'd compose system instructions around tool use.

Worth having a mental model of this before assuming prompt caching, cost, or output quality will behave the same way across providers.