Set Up AI Agent Monitoring
Monitor AI agents with token usage, latency, tool execution, and error tracking.
With Sentry AI Agent Monitoring, you can monitor and debug your AI systems with full-stack context. You'll be able to track key insights like token usage, latency, tool usage, and error rates. AI Agent Monitoring data will be fully connected to your other Sentry data like logs, errors, and traces.
Use curl to download, read and follow https://skills.sentry.dev/instrument to set up Sentry.Before setting up AI Agent Monitoring, ensure you have tracing enabled in your Sentry configuration.
The JavaScript SDK supports automatic instrumentation for AI libraries. Add the integration for your AI library to your Sentry configuration:
import * as Sentry from "___SDK_PACKAGE___";
import { openAIIntegration } from "___SDK_PACKAGE___";
Sentry.init({
dsn: "___PUBLIC_DSN___",
tracesSampleRate: 1.0,
integrations: [openAIIntegration()],
});
import * as Sentry from "___SDK_PACKAGE___";
import { openAIIntegration } from "___SDK_PACKAGE___";
Sentry.init({
dsn: "___PUBLIC_DSN___",
tracesSampleRate: 1.0,
integrations: [openAIIntegration()],
});
All AI integrations support recordInputs and recordOutputs options to control whether prompts and responses are captured. Both default to true.
Set these to false if your prompts or responses contain sensitive data you don't want sent to Sentry.
import * as Sentry from "___SDK_PACKAGE___";
import { openAIIntegration } from "___SDK_PACKAGE___";
Sentry.init({
dsn: "___PUBLIC_DSN___",
tracesSampleRate: 1.0,
integrations: [
openAIIntegration({
recordInputs: false, // Don't capture prompts
recordOutputs: false, // Don't capture responses
}),
],
});
import * as Sentry from "___SDK_PACKAGE___";
import { openAIIntegration } from "___SDK_PACKAGE___";
Sentry.init({
dsn: "___PUBLIC_DSN___",
tracesSampleRate: 1.0,
integrations: [
openAIIntegration({
recordInputs: false, // Don't capture prompts
recordOutputs: false, // Don't capture responses
}),
],
});
gen_ai spans are sent as standalone envelope items instead of being bundled in the transaction by default (since SDK version 10.61.0). This prevents AI spans with large inputs and outputs from hitting transaction payload size limits and being dropped, and is required for Conversations to work.
To send gen_ai spans as part of the transaction instead, set streamGenAiSpans to false. Self-hosted Sentry users should set this to false, as standalone gen_ai spans may not be ingested by their Sentry instance.
import * as Sentry from "___SDK_PACKAGE___";
Sentry.init({
dsn: "___PUBLIC_DSN___",
tracesSampleRate: 1.0,
streamGenAiSpans: false, // opt out of streaming gen_ai spans
});
import * as Sentry from "___SDK_PACKAGE___";
Sentry.init({
dsn: "___PUBLIC_DSN___",
tracesSampleRate: 1.0,
streamGenAiSpans: false, // opt out of streaming gen_ai spans
});
Tracking Conversations has beta stability. Configuration options and behavior may change.
When building AI applications with multi-turn conversations, you can use setConversationId() to link all AI spans from the same conversation together. This allows you to analyze entire conversation flows in Sentry.
The conversation ID is automatically applied as the gen_ai.conversation.id attribute to all AI-related spans within the current scope. To unset the conversation ID, pass null.
import * as Sentry from "___SDK_PACKAGE___";
// Set conversation ID at the start of a conversation
Sentry.setConversationId("conv_abc123");
// All subsequent AI calls will be linked to this conversation
await openai.chat.completions.create({
model: "gpt-4",
messages: [{ role: "user", content: "Hello" }],
});
// Later in the conversation
await openai.chat.completions.create({
model: "gpt-4",
messages: [
{ role: "user", content: "Hello" },
{ role: "assistant", content: "Hi there!" },
{ role: "user", content: "What's the weather?" },
],
});
// Both calls will have gen_ai.conversation.id: "conv_abc123"
// To unset it
Sentry.setConversationId(null);
import * as Sentry from "___SDK_PACKAGE___";
// Set conversation ID at the start of a conversation
Sentry.setConversationId("conv_abc123");
// All subsequent AI calls will be linked to this conversation
await openai.chat.completions.create({
model: "gpt-4",
messages: [{ role: "user", content: "Hello" }],
});
// Later in the conversation
await openai.chat.completions.create({
model: "gpt-4",
messages: [
{ role: "user", content: "Hello" },
{ role: "assistant", content: "Hi there!" },
{ role: "user", content: "What's the weather?" },
],
});
// Both calls will have gen_ai.conversation.id: "conv_abc123"
// To unset it
Sentry.setConversationId(null);
The Conversations view includes a User column. To populate it, call setUser once per request or session, before any AI calls:
import * as Sentry from "___SDK_PACKAGE___";
Sentry.setUser({
id: "user_123",
email: "jane@example.com",
username: "jane",
});
import * as Sentry from "___SDK_PACKAGE___";
Sentry.setUser({
id: "user_123",
email: "jane@example.com",
username: "jane",
});
Any of id, email, or username is sufficient — Conversations displays whichever fields are present. See the setUser API reference for the full list of supported fields.
If you're using a library that Sentry does not automatically instrument, you can manually instrument your code to capture spans. For your AI agents data to show up in the AI Agents Dashboards, spans must have well-defined names and data attributes.
When instrumenting an agent loop, spans nest like this:
── invoke_agent My Agent (gen_ai.invoke_agent)
├── chat gpt-4o (gen_ai.chat) ← 1st LLM call
├── execute_tool get_weather (gen_ai.execute_tool) ← tool run
├── chat gpt-4o (gen_ai.chat) ← 2nd LLM call
└── ...
── invoke_agent My Agent (gen_ai.invoke_agent)
├── chat gpt-4o (gen_ai.chat) ← 1st LLM call
├── execute_tool get_weather (gen_ai.execute_tool) ← tool run
├── chat gpt-4o (gen_ai.chat) ← 2nd LLM call
└── ...
gen_ai.invoke_agent is the container. gen_ai.chat and gen_ai.execute_tool spans are its children (siblings of each other). A gen_ai.chat span can also appear without an agent parent for standalone LLM calls.
This span represents a request to an LLM model or service that generates a response based on the input prompt.
Key attributes:
gen_ai.operation.name— Required. Set to"chat"for chat completionsgen_ai.request.model— The model name (required)gen_ai.input.messages— The prompts sent to the LLMgen_ai.output.messages— The model's responsegen_ai.usage.input_tokens/output_tokens— Token counts
const messages = [
{ role: "user", parts: [{ type: "text", content: "Tell me a joke" }] },
];
await Sentry.startSpan(
{
op: "gen_ai.chat",
name: "chat o3-mini",
attributes: {
"gen_ai.operation.name": "chat",
"gen_ai.request.model": "o3-mini",
"gen_ai.provider.name": "openai",
"gen_ai.input.messages": JSON.stringify(messages),
},
},
async (span) => {
const result = await client.chat.completions.create({
model: "o3-mini",
messages,
});
span.setAttribute("gen_ai.response.model", result.model);
span.setAttribute(
"gen_ai.output.messages",
JSON.stringify([
{
role: "assistant",
parts: [
{ type: "text", content: result.choices[0].message.content },
],
},
]),
);
span.setAttribute(
"gen_ai.response.finish_reasons",
JSON.stringify([result.choices[0].finish_reason]),
);
span.setAttribute(
"gen_ai.usage.input_tokens",
result.usage.prompt_tokens,
);
span.setAttribute(
"gen_ai.usage.output_tokens",
result.usage.completion_tokens,
);
},
);
const messages = [
{ role: "user", parts: [{ type: "text", content: "Tell me a joke" }] },
];
await Sentry.startSpan(
{
op: "gen_ai.chat",
name: "chat o3-mini",
attributes: {
"gen_ai.operation.name": "chat",
"gen_ai.request.model": "o3-mini",
"gen_ai.provider.name": "openai",
"gen_ai.input.messages": JSON.stringify(messages),
},
},
async (span) => {
const result = await client.chat.completions.create({
model: "o3-mini",
messages,
});
span.setAttribute("gen_ai.response.model", result.model);
span.setAttribute(
"gen_ai.output.messages",
JSON.stringify([
{
role: "assistant",
parts: [
{ type: "text", content: result.choices[0].message.content },
],
},
]),
);
span.setAttribute(
"gen_ai.response.finish_reasons",
JSON.stringify([result.choices[0].finish_reason]),
);
span.setAttribute(
"gen_ai.usage.input_tokens",
result.usage.prompt_tokens,
);
span.setAttribute(
"gen_ai.usage.output_tokens",
result.usage.completion_tokens,
);
},
);
For a complete guide on naming agents across all supported frameworks, see Naming Your Agents.
This span represents the execution of an AI agent, capturing the full lifecycle from receiving a task to producing a final response.
Key attributes:
gen_ai.operation.name— Required. Set to"invoke_agent"gen_ai.agent.name— The agent's name (e.g., "Weather Agent")gen_ai.request.model— The underlying model usedgen_ai.output.messages— The agent's final outputgen_ai.usage.input_tokens/output_tokens— Total token counts
await Sentry.startSpan(
{
op: "gen_ai.invoke_agent",
name: "invoke_agent Weather Agent",
attributes: {
"gen_ai.operation.name": "invoke_agent",
"gen_ai.request.model": "o3-mini",
"gen_ai.agent.name": "Weather Agent",
},
},
async (span) => {
const result = await myAgent.run();
span.setAttribute(
"gen_ai.output.messages",
JSON.stringify([
{
role: "assistant",
parts: [{ type: "text", content: result.output }],
},
]),
);
span.setAttribute(
"gen_ai.usage.input_tokens",
result.usage.inputTokens,
);
span.setAttribute(
"gen_ai.usage.output_tokens",
result.usage.outputTokens,
);
},
);
await Sentry.startSpan(
{
op: "gen_ai.invoke_agent",
name: "invoke_agent Weather Agent",
attributes: {
"gen_ai.operation.name": "invoke_agent",
"gen_ai.request.model": "o3-mini",
"gen_ai.agent.name": "Weather Agent",
},
},
async (span) => {
const result = await myAgent.run();
span.setAttribute(
"gen_ai.output.messages",
JSON.stringify([
{
role: "assistant",
parts: [{ type: "text", content: result.output }],
},
]),
);
span.setAttribute(
"gen_ai.usage.input_tokens",
result.usage.inputTokens,
);
span.setAttribute(
"gen_ai.usage.output_tokens",
result.usage.outputTokens,
);
},
);
This span represents the execution of a tool or function that was requested by an AI model, including the input arguments and resulting output.
Key attributes:
gen_ai.operation.name— Required. Set to"execute_tool"gen_ai.tool.name— The tool's name (e.g., "get_weather")gen_ai.tool.call.arguments— The arguments passed to the toolgen_ai.tool.call.result— The tool's return value
await Sentry.startSpan(
{
op: "gen_ai.execute_tool",
name: "execute_tool get_weather",
attributes: {
"gen_ai.operation.name": "execute_tool",
"gen_ai.tool.name": "get_weather",
"gen_ai.tool.call.arguments": JSON.stringify({ location: "Paris" }),
},
},
async (span) => {
const result = await getWeather({ location: "Paris" });
span.setAttribute("gen_ai.tool.call.result", JSON.stringify(result));
},
);
await Sentry.startSpan(
{
op: "gen_ai.execute_tool",
name: "execute_tool get_weather",
attributes: {
"gen_ai.operation.name": "execute_tool",
"gen_ai.tool.name": "get_weather",
"gen_ai.tool.call.arguments": JSON.stringify({ location: "Paris" }),
},
},
async (span) => {
const result = await getWeather({ location: "Paris" });
span.setAttribute("gen_ai.tool.call.result", JSON.stringify(result));
},
);
When the LLM returns a stream, the span must outlive the initial callback. Use Sentry.startInactiveSpan to create the span, then end it when the stream finishes:
async function callLLMStreaming(model, messages) {
const span = Sentry.startInactiveSpan({
name: `chat ${model}`,
op: "gen_ai.chat",
attributes: {
"gen_ai.operation.name": "chat",
"gen_ai.request.model": model,
"gen_ai.input.messages": JSON.stringify(messages),
},
});
try {
const stream = await Sentry.withActiveSpan(span, () =>
yourLLMClient.stream({ model, messages }),
);
stream.on("end", (finalMessage) => {
span.setAttribute(
"gen_ai.output.messages",
JSON.stringify([
{
role: "assistant",
parts: [{ type: "text", content: finalMessage.text }],
},
]),
);
span.setAttribute(
"gen_ai.usage.input_tokens",
finalMessage.usage.input,
);
span.setAttribute(
"gen_ai.usage.output_tokens",
finalMessage.usage.output,
);
span.setAttribute("gen_ai.response.model", finalMessage.model);
span.setAttribute("gen_ai.response.streaming", true);
span.end();
});
stream.on("error", () => span.end());
return stream;
} catch (error) {
span.end();
throw error;
}
}
async function callLLMStreaming(model, messages) {
const span = Sentry.startInactiveSpan({
name: `chat ${model}`,
op: "gen_ai.chat",
attributes: {
"gen_ai.operation.name": "chat",
"gen_ai.request.model": model,
"gen_ai.input.messages": JSON.stringify(messages),
},
});
try {
const stream = await Sentry.withActiveSpan(span, () =>
yourLLMClient.stream({ model, messages }),
);
stream.on("end", (finalMessage) => {
span.setAttribute(
"gen_ai.output.messages",
JSON.stringify([
{
role: "assistant",
parts: [{ type: "text", content: finalMessage.text }],
},
]),
);
span.setAttribute(
"gen_ai.usage.input_tokens",
finalMessage.usage.input,
);
span.setAttribute(
"gen_ai.usage.output_tokens",
finalMessage.usage.output,
);
span.setAttribute("gen_ai.response.model", finalMessage.model);
span.setAttribute("gen_ai.response.streaming", true);
span.end();
});
stream.on("error", () => span.end());
return stream;
} catch (error) {
span.end();
throw error;
}
}
startInactiveSpan creates a span without automatically ending it. Sentry.withActiveSpan propagates context so any child spans nest correctly. Call span.end() when the stream completes or errors.
When manually setting token attributes, be aware of how Sentry uses them to calculate model costs.
Cached and reasoning tokens are subsets, not separate counts. gen_ai.usage.input_tokens is the total input token count that already includes any cached tokens. Similarly, gen_ai.usage.output_tokens already includes reasoning tokens. Sentry subtracts the cached/reasoning counts from the totals to compute the "raw" portion, so reporting them incorrectly can produce wrong or negative costs.
For example, say your LLM call uses 100 input tokens total, 90 of which were served from cache. Using a standard rate of $0.01 per token and a cached rate of $0.001 per token:
Correct — input_tokens is the total (includes cached):
gen_ai.usage.input_tokens = 100gen_ai.usage.input_tokens.cached = 90- Sentry calculates:
(100 - 90) × $0.01 + 90 × $0.001=$0.10 + $0.09= $0.19 ✓
Wrong — input_tokens set to only the non-cached tokens, making cached larger than total:
gen_ai.usage.input_tokens = 10gen_ai.usage.input_tokens.cached = 90- Sentry calculates:
(10 - 90) × $0.01 + 90 × $0.001=−$0.80 + $0.09= −$0.71
Because input_tokens.cached (90) is larger than input_tokens (10), the subtraction goes negative, resulting in a negative total cost.
The same applies to gen_ai.usage.output_tokens and gen_ai.usage.output_tokens.reasoning.
If you're using an AI framework with a Sentry exporter, you can send traces to Sentry:
If you're building MCP (Model Context Protocol) servers, Sentry can also track tool executions, prompt retrievals, and resource access. See Instrument MCP Servers for setup instructions.
Our documentation is open source and available on GitHub. Your contributions are welcome, whether fixing a typo (drat!) or suggesting an update ("yeah, this would be better").