Manual Instrumentation
Learn how to manually instrument your agents to capture spans, token usage, and tool execution.
If your AI library does not have automatic instrumentation, create spans manually. Spans need well-defined names and data attributes so agent data shows up correctly.
The @sentry_sdk.trace() decorator can also create these spans via templates.
For conversations and privacy (set_conversation_id, set_user, send_default_pii), see the Agent Tracing hub.
When instrumenting an agent loop, spans nest like this:
── invoke_agent My Agent (gen_ai.invoke_agent)
├── chat gpt-4o (gen_ai.chat) ← 1st LLM call
├── execute_tool get_weather (gen_ai.execute_tool) ← tool run
├── chat gpt-4o (gen_ai.chat) ← 2nd LLM call
└── ...
── invoke_agent My Agent (gen_ai.invoke_agent)
├── chat gpt-4o (gen_ai.chat) ← 1st LLM call
├── execute_tool get_weather (gen_ai.execute_tool) ← tool run
├── chat gpt-4o (gen_ai.chat) ← 2nd LLM call
└── ...
gen_ai.invoke_agent is the container. gen_ai.chat and gen_ai.execute_tool spans are its children (siblings of each other). A gen_ai.chat span can also appear without an agent parent for standalone LLM calls.
Set these when the span starts (before the model or tool call), so head-based sampling can see them:
gen_ai.operation.name— required; classifies the span (chat,invoke_agent,execute_tool, …)gen_ai.provider.name— e.g.openai,anthropicgen_ai.request.model— requested model (pass the raw provider string)gen_ai.agent.name/gen_ai.tool.name— when applicable
Complex values (messages, tool definitions, arrays) must be JSON strings — span attributes only accept primitives.
Prompt, response, and tool content are treated as PII. They are only captured when send_default_pii=True (or per-integration include_prompts). See Privacy Controls.
This span represents a request to an LLM model or service that generates a response based on the input prompt.
import json
import sentry_sdk
messages = [{"role": "user", "parts": [{"type": "text", "content": "Tell me a joke"}]}]
tools = [{"name": "get_weather", "description": "Get weather for a city"}]
with sentry_sdk.start_span(
op="gen_ai.chat",
name="chat o3-mini",
attributes={
"gen_ai.operation.name": "chat",
"gen_ai.request.model": "o3-mini",
"gen_ai.provider.name": "openai",
"gen_ai.agent.name": "Weather Agent", # when this call is under an agent
"gen_ai.system_instructions": "You are a helpful assistant.",
"gen_ai.tool.definitions": json.dumps(tools),
"gen_ai.input.messages": json.dumps(messages),
},
) as span:
result = client.chat.completions.create(model="o3-mini", messages=messages)
span.set_data("gen_ai.response.model", result.model)
span.set_data("gen_ai.response.id", result.id)
span.set_data("gen_ai.output.messages", json.dumps([
{"role": "assistant", "parts": [{"type": "text", "content": result.choices[0].message.content}]}
]))
span.set_data("gen_ai.response.finish_reasons", json.dumps([result.choices[0].finish_reason]))
span.set_data("gen_ai.usage.input_tokens", result.usage.prompt_tokens)
span.set_data("gen_ai.usage.output_tokens", result.usage.completion_tokens)
# If the provider reports cached tokens, record them as a subset of input tokens
if getattr(result.usage, "prompt_tokens_details", None):
span.set_data(
"gen_ai.usage.cache_read.input_tokens",
result.usage.prompt_tokens_details.cached_tokens,
)
import json
import sentry_sdk
messages = [{"role": "user", "parts": [{"type": "text", "content": "Tell me a joke"}]}]
tools = [{"name": "get_weather", "description": "Get weather for a city"}]
with sentry_sdk.start_span(
op="gen_ai.chat",
name="chat o3-mini",
attributes={
"gen_ai.operation.name": "chat",
"gen_ai.request.model": "o3-mini",
"gen_ai.provider.name": "openai",
"gen_ai.agent.name": "Weather Agent", # when this call is under an agent
"gen_ai.system_instructions": "You are a helpful assistant.",
"gen_ai.tool.definitions": json.dumps(tools),
"gen_ai.input.messages": json.dumps(messages),
},
) as span:
result = client.chat.completions.create(model="o3-mini", messages=messages)
span.set_data("gen_ai.response.model", result.model)
span.set_data("gen_ai.response.id", result.id)
span.set_data("gen_ai.output.messages", json.dumps([
{"role": "assistant", "parts": [{"type": "text", "content": result.choices[0].message.content}]}
]))
span.set_data("gen_ai.response.finish_reasons", json.dumps([result.choices[0].finish_reason]))
span.set_data("gen_ai.usage.input_tokens", result.usage.prompt_tokens)
span.set_data("gen_ai.usage.output_tokens", result.usage.completion_tokens)
# If the provider reports cached tokens, record them as a subset of input tokens
if getattr(result.usage, "prompt_tokens_details", None):
span.set_data(
"gen_ai.usage.cache_read.input_tokens",
result.usage.prompt_tokens_details.cached_tokens,
)
Keep system prompts in gen_ai.system_instructions, not inside gen_ai.input.messages. Conversation titles are derived from the first user message in the input messages.
Messages use {role, parts} where each part has a type. Common types:
text— user-visible contentreasoning— internal thinking (not shown in the user-facing Conversations view)tool_call/tool_call_response— tool invocations linked by a sharedid
Unknown part types are not shown prominently in the Conversations UI. They remain available only in the raw span attribute values.
Models with extended thinking (such as Anthropic's thinking blocks, Gemini's thought, or DeepSeek's reasoning_content) produce internal reasoning that isn't part of the user-visible reply. Represent this as a reasoning part alongside the user-facing text part — don't fold thinking into text.
span.set_data("gen_ai.output.messages", json.dumps([
{
"role": "assistant",
"parts": [
{"type": "reasoning", "content": "6 times 7 is 42."},
{"type": "text", "content": "The answer is 42."},
],
}
]))
span.set_data("gen_ai.usage.output_tokens", result.usage.completion_tokens)
# Reasoning tokens are a subset of output tokens
span.set_data(
"gen_ai.usage.reasoning.output_tokens",
result.usage.completion_tokens_details.reasoning_tokens,
)
span.set_data("gen_ai.output.messages", json.dumps([
{
"role": "assistant",
"parts": [
{"type": "reasoning", "content": "6 times 7 is 42."},
{"type": "text", "content": "The answer is 42."},
],
}
]))
span.set_data("gen_ai.usage.output_tokens", result.usage.completion_tokens)
# Reasoning tokens are a subset of output tokens
span.set_data(
"gen_ai.usage.reasoning.output_tokens",
result.usage.completion_tokens_details.reasoning_tokens,
)
When previous thinking is fed back into a multi-turn request, include the same reasoning parts in assistant messages within gen_ai.input.messages.
Link a tool request to its result with the same id:
# Model asked to call a tool
span.set_data("gen_ai.output.messages", json.dumps([
{
"role": "assistant",
"parts": [
{
"type": "tool_call",
"id": "call_abc",
"name": "get_weather",
"arguments": {"location": "Paris"},
}
],
}
]))
# Later turn: tool result fed back to the model (on input messages)
input_with_tool = [
{"role": "user", "parts": [{"type": "text", "content": "Weather in Paris?"}]},
{
"role": "assistant",
"parts": [
{
"type": "tool_call",
"id": "call_abc",
"name": "get_weather",
"arguments": {"location": "Paris"},
}
],
},
{
"role": "tool",
"parts": [
{
"type": "tool_call_response",
"id": "call_abc",
"name": "get_weather",
"content": "{\"temp_c\": 18}",
}
],
},
]
# Model asked to call a tool
span.set_data("gen_ai.output.messages", json.dumps([
{
"role": "assistant",
"parts": [
{
"type": "tool_call",
"id": "call_abc",
"name": "get_weather",
"arguments": {"location": "Paris"},
}
],
}
]))
# Later turn: tool result fed back to the model (on input messages)
input_with_tool = [
{"role": "user", "parts": [{"type": "text", "content": "Weather in Paris?"}]},
{
"role": "assistant",
"parts": [
{
"type": "tool_call",
"id": "call_abc",
"name": "get_weather",
"arguments": {"location": "Paris"},
}
],
},
{
"role": "tool",
"parts": [
{
"type": "tool_call_response",
"id": "call_abc",
"name": "get_weather",
"content": "{\"temp_c\": 18}",
}
],
},
]
This span represents the execution of an agent, capturing the full lifecycle from receiving a task to producing a final response.
For a complete guide on naming agents across all supported frameworks, see Naming Your Agents.
import json
import sentry_sdk
messages = [{"role": "user", "parts": [{"type": "text", "content": "What's the weather in Paris?"}]}]
tools = [{"name": "get_weather", "description": "Get weather for a city"}]
with sentry_sdk.start_span(
op="gen_ai.invoke_agent",
name="invoke_agent Weather Agent",
attributes={
"gen_ai.operation.name": "invoke_agent",
"gen_ai.agent.name": "Weather Agent",
"gen_ai.provider.name": "openai",
"gen_ai.request.model": "o3-mini",
"gen_ai.system_instructions": "You are a weather assistant.",
"gen_ai.tool.definitions": json.dumps(tools),
"gen_ai.input.messages": json.dumps(messages),
},
) as span:
final_output = my_agent.run()
span.set_data("gen_ai.output.messages", json.dumps([
{"role": "assistant", "parts": [{"type": "text", "content": str(final_output)}]}
]))
span.set_data("gen_ai.usage.input_tokens", final_output.usage.input_tokens)
span.set_data("gen_ai.usage.output_tokens", final_output.usage.output_tokens)
import json
import sentry_sdk
messages = [{"role": "user", "parts": [{"type": "text", "content": "What's the weather in Paris?"}]}]
tools = [{"name": "get_weather", "description": "Get weather for a city"}]
with sentry_sdk.start_span(
op="gen_ai.invoke_agent",
name="invoke_agent Weather Agent",
attributes={
"gen_ai.operation.name": "invoke_agent",
"gen_ai.agent.name": "Weather Agent",
"gen_ai.provider.name": "openai",
"gen_ai.request.model": "o3-mini",
"gen_ai.system_instructions": "You are a weather assistant.",
"gen_ai.tool.definitions": json.dumps(tools),
"gen_ai.input.messages": json.dumps(messages),
},
) as span:
final_output = my_agent.run()
span.set_data("gen_ai.output.messages", json.dumps([
{"role": "assistant", "parts": [{"type": "text", "content": str(final_output)}]}
]))
span.set_data("gen_ai.usage.input_tokens", final_output.usage.input_tokens)
span.set_data("gen_ai.usage.output_tokens", final_output.usage.output_tokens)
Child gen_ai.chat spans should also set gen_ai.agent.name so model usage can be attributed per agent.
This span represents the execution of a tool or function that was requested by an AI model, including the input arguments and resulting output.
import json
import sentry_sdk
with sentry_sdk.start_span(
op="gen_ai.execute_tool",
name="execute_tool get_weather",
attributes={
"gen_ai.operation.name": "execute_tool",
"gen_ai.tool.name": "get_weather",
"gen_ai.tool.description": "Get weather for a city",
"gen_ai.tool.call.arguments": json.dumps({"location": "Paris"}),
},
) as span:
try:
result = get_weather(location="Paris")
span.set_data("gen_ai.tool.call.result", json.dumps(result))
except Exception as exc:
span.set_status("internal_error")
span.set_data("error.type", type(exc).__name__)
raise
import json
import sentry_sdk
with sentry_sdk.start_span(
op="gen_ai.execute_tool",
name="execute_tool get_weather",
attributes={
"gen_ai.operation.name": "execute_tool",
"gen_ai.tool.name": "get_weather",
"gen_ai.tool.description": "Get weather for a city",
"gen_ai.tool.call.arguments": json.dumps({"location": "Paris"}),
},
) as span:
try:
result = get_weather(location="Paris")
span.set_data("gen_ai.tool.call.result", json.dumps(result))
except Exception as exc:
span.set_status("internal_error")
span.set_data("error.type", type(exc).__name__)
raise
Marking failed tools with an error status populates the Tool Errors widget.
When the model streams tokens, keep the span open until the stream finishes (including when you yield chunks to the client). Set response attributes when you have the final usage and text:
gen_ai.response.streaming—Truegen_ai.response.time_to_first_chunk— seconds until the first chunkgen_ai.response.tokens_per_second— output throughput, if you can measure itgen_ai.output.messages, token usage, andgen_ai.response.model— same as a non-streaming call, once the stream completes
How you hold the span open depends on your stack (generator, async iterator, callback). Instrument that path yourself so the span ends on completion or error.
When manually setting token attributes, be aware of how Sentry uses them to calculate model costs.
Cached and reasoning tokens are subsets, not separate counts. gen_ai.usage.input_tokens is the total input token count that already includes any cached tokens. Similarly, gen_ai.usage.output_tokens already includes reasoning tokens. Sentry subtracts the cached/reasoning counts from the totals to compute the "raw" portion, so reporting them incorrectly can produce wrong or negative costs.
For example, say your LLM call uses 100 input tokens total, 90 of which were served from cache. Using a standard rate of $0.01 per token and a cached rate of $0.001 per token:
Correct — input_tokens is the total (includes cached):
gen_ai.usage.input_tokens = 100gen_ai.usage.cache_read.input_tokens = 90- Sentry calculates:
(100 - 90) × $0.01 + 90 × $0.001=$0.10 + $0.09= $0.19 ✓
Wrong — input_tokens set to only the non-cached tokens, making cached larger than total:
gen_ai.usage.input_tokens = 10gen_ai.usage.cache_read.input_tokens = 90- Sentry calculates:
(10 - 90) × $0.01 + 90 × $0.001=−$0.80 + $0.09= −$0.71
Because input_tokens.cached (90) is larger than input_tokens (10), the subtraction goes negative, resulting in a negative total cost.
The same applies to gen_ai.usage.output_tokens and gen_ai.usage.reasoning.output_tokens.
Sentry derives model cost from the model name and token counts. You do not need to set gen_ai.cost.* attributes. Pass the raw provider model string unchanged so pricing can resolve.
Our documentation is open source and available on GitHub. Your contributions are welcome, whether fixing a typo (drat!) or suggesting an update ("yeah, this would be better").