Set Up AI Agent Monitoring
Monitor AI agents with token usage, latency, tool execution, and error tracking.
With Sentry AI Agent Monitoring, you can monitor and debug your AI systems with full-stack context. You'll be able to track key insights like token usage, latency, tool usage, and error rates. AI Agent Monitoring data will be fully connected to your other Sentry data like logs, errors, and traces.
Use curl to download, read and follow: https://skills.sentry.dev/sentry-setup-ai-monitoring/SKILL.mdBefore setting up AI Agent Monitoring, ensure you have tracing enabled in your Sentry configuration.
The JavaScript SDK supports automatic instrumentation for AI libraries. Add the integration for your AI library to your Sentry configuration:
import * as Sentry from "___SDK_PACKAGE___";
import { openAIIntegration } from "___SDK_PACKAGE___";
Sentry.init({
dsn: "___PUBLIC_DSN___",
tracesSampleRate: 1.0,
integrations: [openAIIntegration()],
});
All AI integrations support recordInputs and recordOutputs options to control whether prompts and responses are captured. Both default to true.
Set these to false if your prompts or responses contain sensitive data you don't want sent to Sentry.
import * as Sentry from "___SDK_PACKAGE___";
import { openAIIntegration } from "___SDK_PACKAGE___";
Sentry.init({
dsn: "___PUBLIC_DSN___",
tracesSampleRate: 1.0,
integrations: [
openAIIntegration({
recordInputs: false, // Don't capture prompts
recordOutputs: false, // Don't capture responses
}),
],
});
Tracking Conversations has alpha stability. Configuration options and behavior may change.
When building AI applications with multi-turn conversations, you can use setConversationId() to link all AI spans from the same conversation together. This allows you to analyze entire conversation flows in Sentry.
The conversation ID is automatically applied as the gen_ai.conversation.id attribute to all AI-related spans within the current scope. To unset the conversation ID, pass null.
import * as Sentry from "___SDK_PACKAGE___";
// Set conversation ID at the start of a conversation
Sentry.setConversationId("conv_abc123");
// All subsequent AI calls will be linked to this conversation
await openai.chat.completions.create({
model: "gpt-4",
messages: [{ role: "user", content: "Hello" }],
});
// Later in the conversation
await openai.chat.completions.create({
model: "gpt-4",
messages: [
{ role: "user", content: "Hello" },
{ role: "assistant", content: "Hi there!" },
{ role: "user", content: "What's the weather?" },
],
});
// Both calls will have gen_ai.conversation.id: "conv_abc123"
// To unset it
Sentry.setConversationId(null);
If you're using a library that Sentry does not automatically instrument, you can manually instrument your code to capture spans. For your AI agents data to show up in the AI Agents Dashboards, spans must have well-defined names and data attributes.
When instrumenting an agent loop, spans nest like this:
── invoke_agent My Agent (gen_ai.invoke_agent)
├── chat gpt-4o (gen_ai.chat) ← 1st LLM call
├── execute_tool get_weather (gen_ai.execute_tool) ← tool run
├── chat gpt-4o (gen_ai.chat) ← 2nd LLM call
└── ...
gen_ai.invoke_agent is the container. gen_ai.chat and gen_ai.execute_tool spans are its children (siblings of each other). A gen_ai.chat span can also appear without an agent parent for standalone LLM calls.
This span represents a request to an LLM model or service that generates a response based on the input prompt.
Key attributes:
gen_ai.operation.name— Required. Set to"chat"for chat completionsgen_ai.request.model— The model name (required)gen_ai.input.messages— The prompts sent to the LLMgen_ai.output.messages— The model's responsegen_ai.usage.input_tokens/output_tokens— Token counts
const messages = [
{ role: "user", parts: [{ type: "text", content: "Tell me a joke" }] },
];
await Sentry.startSpan(
{
op: "gen_ai.chat",
name: "chat o3-mini",
attributes: {
"gen_ai.operation.name": "chat",
"gen_ai.request.model": "o3-mini",
"gen_ai.provider.name": "openai",
"gen_ai.input.messages": JSON.stringify(messages),
},
},
async (span) => {
const result = await client.chat.completions.create({
model: "o3-mini",
messages,
});
span.setAttribute("gen_ai.response.model", result.model);
span.setAttribute(
"gen_ai.output.messages",
JSON.stringify([
{
role: "assistant",
parts: [
{ type: "text", content: result.choices[0].message.content },
],
},
]),
);
span.setAttribute(
"gen_ai.response.finish_reasons",
JSON.stringify([result.choices[0].finish_reason]),
);
span.setAttribute(
"gen_ai.usage.input_tokens",
result.usage.prompt_tokens,
);
span.setAttribute(
"gen_ai.usage.output_tokens",
result.usage.completion_tokens,
);
},
);
For a complete guide on naming agents across all supported frameworks, see Naming Your Agents.
This span represents the execution of an AI agent, capturing the full lifecycle from receiving a task to producing a final response.
Key attributes:
gen_ai.operation.name— Required. Set to"invoke_agent"gen_ai.agent.name— The agent's name (e.g., "Weather Agent")gen_ai.request.model— The underlying model usedgen_ai.output.messages— The agent's final outputgen_ai.usage.input_tokens/output_tokens— Total token counts
await Sentry.startSpan(
{
op: "gen_ai.invoke_agent",
name: "invoke_agent Weather Agent",
attributes: {
"gen_ai.operation.name": "invoke_agent",
"gen_ai.request.model": "o3-mini",
"gen_ai.agent.name": "Weather Agent",
},
},
async (span) => {
const result = await myAgent.run();
span.setAttribute(
"gen_ai.output.messages",
JSON.stringify([
{
role: "assistant",
parts: [{ type: "text", content: result.output }],
},
]),
);
span.setAttribute(
"gen_ai.usage.input_tokens",
result.usage.inputTokens,
);
span.setAttribute(
"gen_ai.usage.output_tokens",
result.usage.outputTokens,
);
},
);
This span represents the execution of a tool or function that was requested by an AI model, including the input arguments and resulting output.
Key attributes:
gen_ai.operation.name— Required. Set to"execute_tool"gen_ai.tool.name— The tool's name (e.g., "get_weather")gen_ai.tool.call.arguments— The arguments passed to the toolgen_ai.tool.call.result— The tool's return value
await Sentry.startSpan(
{
op: "gen_ai.execute_tool",
name: "execute_tool get_weather",
attributes: {
"gen_ai.operation.name": "execute_tool",
"gen_ai.tool.name": "get_weather",
"gen_ai.tool.call.arguments": JSON.stringify({ location: "Paris" }),
},
},
async (span) => {
const result = await getWeather({ location: "Paris" });
span.setAttribute("gen_ai.tool.call.result", JSON.stringify(result));
},
);
This span marks the transition of control from one agent to another, typically when the current agent determines another agent is better suited to handle the task.
Requirements:
opmust be"gen_ai.handoff"nameshould follow the pattern"handoff from {source} to {target}"- All Common Span Attributes should be set
The handoff span itself has no body — it just marks the transition point before the target agent starts.
await Sentry.startSpan(
{
op: "gen_ai.handoff",
name: "handoff from Weather Agent to Travel Agent",
},
() => {}, // Handoff span just marks the transition
);
await Sentry.startSpan(
{ op: "gen_ai.invoke_agent", name: "invoke_agent Travel Agent" },
async () => {
// Run the target agent here
},
);
When the LLM returns a stream, the span must outlive the initial callback. Use Sentry.startInactiveSpan to create the span, then end it when the stream finishes:
async function callLLMStreaming(model, messages) {
const span = Sentry.startInactiveSpan({
name: `chat ${model}`,
op: "gen_ai.chat",
attributes: {
"gen_ai.operation.name": "chat",
"gen_ai.request.model": model,
"gen_ai.input.messages": JSON.stringify(messages),
},
});
try {
const stream = await Sentry.withActiveSpan(span, () =>
yourLLMClient.stream({ model, messages }),
);
stream.on("end", (finalMessage) => {
span.setAttribute(
"gen_ai.output.messages",
JSON.stringify([
{
role: "assistant",
parts: [{ type: "text", content: finalMessage.text }],
},
]),
);
span.setAttribute(
"gen_ai.usage.input_tokens",
finalMessage.usage.input,
);
span.setAttribute(
"gen_ai.usage.output_tokens",
finalMessage.usage.output,
);
span.setAttribute("gen_ai.response.model", finalMessage.model);
span.setAttribute("gen_ai.response.streaming", true);
span.end();
});
stream.on("error", () => span.end());
return stream;
} catch (error) {
span.end();
throw error;
}
}
startInactiveSpan creates a span without automatically ending it. Sentry.withActiveSpan propagates context so any child spans nest correctly. Call span.end() when the stream completes or errors.
Some attributes are common to all AI Agents spans:
| Data Attribute | Type | Requirement Level | Description | Example |
|---|---|---|---|---|
gen_ai.operation.name | string | required | The name of the operation being performed. [4] | "chat" |
gen_ai.provider.name | string | optional | The Generative AI product as identified by the client or server instrumentation. | "openai" |
- [4]:
gen_ai.operation.nameis what Sentry uses to classify spans in AI dashboards. Well-defined values include:"chat","invoke_agent","execute_tool","embeddings","generate_content","text_completion","create_agent","handoff".
Well-defined values for gen_ai.provider.name: "anthropic", "aws.bedrock", "azure.ai.inference", "azure.ai.openai", "cohere", "deepseek", "gcp.gemini", "gcp.gen_ai", "gcp.vertex_ai", "groq", "ibm.watsonx.ai", "mistral_ai", "openai", "perplexity", "x_ai".
When manually setting token attributes, be aware of how Sentry uses them to calculate model costs.
Cached and reasoning tokens are subsets, not separate counts. gen_ai.usage.input_tokens is the total input token count that already includes any cached tokens. Similarly, gen_ai.usage.output_tokens already includes reasoning tokens. Sentry subtracts the cached/reasoning counts from the totals to compute the "raw" portion, so reporting them incorrectly can produce wrong or negative costs.
For example, say your LLM call uses 100 input tokens total, 90 of which were served from cache. Using a standard rate of $0.01 per token and a cached rate of $0.001 per token:
Correct — input_tokens is the total (includes cached):
gen_ai.usage.input_tokens = 100gen_ai.usage.input_tokens.cached = 90- Sentry calculates:
(100 - 90) × $0.01 + 90 × $0.001=$0.10 + $0.09= $0.19 ✓
Wrong — input_tokens set to only the non-cached tokens, making cached larger than total:
gen_ai.usage.input_tokens = 10gen_ai.usage.input_tokens.cached = 90- Sentry calculates:
(10 - 90) × $0.01 + 90 × $0.001=−$0.80 + $0.09= −$0.71
Because input_tokens.cached (90) is larger than input_tokens (10), the subtraction goes negative, resulting in a negative total cost.
The same applies to gen_ai.usage.output_tokens and gen_ai.usage.output_tokens.reasoning.
If you're using an AI framework with a Sentry exporter, you can send traces to Sentry:
If you're building MCP (Model Context Protocol) servers, Sentry can also track tool executions, prompt retrievals, and resource access. See Instrument MCP Servers for setup instructions.
Our documentation is open source and available on GitHub. Your contributions are welcome, whether fixing a typo (drat!) or suggesting an update ("yeah, this would be better").