Set Up AI Agent Monitoring
Monitor AI agents with token usage, latency, tool execution, and error tracking.
With Sentry AI Agent Monitoring, you can monitor and debug your AI systems with full-stack context. You'll be able to track key insights like token usage, latency, tool usage, and error rates. AI Agent Monitoring data will be fully connected to your other Sentry data like logs, errors, and traces.
Before setting up AI Agent Monitoring, ensure you have tracing enabled in your Sentry configuration.
The JavaScript SDK supports automatic instrumentation for AI libraries. Add the integration for your AI library to your Sentry configuration:
import * as Sentry from "___SDK_PACKAGE___";
import { openAIIntegration } from "___SDK_PACKAGE___";
Sentry.init({
dsn: "___PUBLIC_DSN___",
tracesSampleRate: 1.0,
integrations: [openAIIntegration()],
});
All AI integrations support recordInputs and recordOutputs options to control whether prompts and responses are captured. Both default to true.
Set these to false if your prompts or responses contain sensitive data you don't want sent to Sentry.
import * as Sentry from "___SDK_PACKAGE___";
import { openAIIntegration } from "___SDK_PACKAGE___";
Sentry.init({
dsn: "___PUBLIC_DSN___",
tracesSampleRate: 1.0,
integrations: [
openAIIntegration({
recordInputs: false, // Don't capture prompts
recordOutputs: false, // Don't capture responses
}),
],
});
When building AI applications with multi-turn conversations, you can use setConversationId() to link all AI spans from the same conversation together. This allows you to analyze entire conversation flows in Sentry.
The conversation ID is automatically applied as the gen_ai.conversation.id attribute to all AI-related spans within the current scope. To unset the conversation ID, pass null.
import * as Sentry from "___SDK_PACKAGE___";
// Set conversation ID at the start of a conversation
Sentry.setConversationId("conv_abc123");
// All subsequent AI calls will be linked to this conversation
await openai.chat.completions.create({
model: "gpt-4",
messages: [{ role: "user", content: "Hello" }],
});
// Later in the conversation
await openai.chat.completions.create({
model: "gpt-4",
messages: [
{ role: "user", content: "Hello" },
{ role: "assistant", content: "Hi there!" },
{ role: "user", content: "What's the weather?" },
],
});
// Both calls will have gen_ai.conversation.id: "conv_abc123"
// To unset it
Sentry.setConversationId(null);
If you're using a library that Sentry does not automatically instrument, you can manually instrument your code to capture spans. For your AI agents data to show up in Sentry AI Agents Insights, spans must have well-defined names and data attributes.
This span represents a request to an LLM model or service that generates a response based on the input prompt.
Key attributes:
gen_ai.request.model— The model name (required)gen_ai.request.messages— The prompts sent to the LLMgen_ai.response.text— The model's responsegen_ai.usage.input_tokens/output_tokens— Token counts
const messages = [{ role: "user", content: "Tell me a joke" }];
await Sentry.startSpan(
{
op: "gen_ai.request",
name: "request o3-mini",
attributes: {
"gen_ai.request.model": "o3-mini",
"gen_ai.request.messages": JSON.stringify(messages),
},
},
async (span) => {
// Call your LLM here
const result = await client.chat.completions.create({
model: "o3-mini",
messages,
});
span.setAttribute(
"gen_ai.response.text",
JSON.stringify([result.choices[0].message.content]),
);
// Set token usage
span.setAttribute(
"gen_ai.usage.input_tokens",
result.usage.prompt_tokens,
);
span.setAttribute(
"gen_ai.usage.output_tokens",
result.usage.completion_tokens,
);
},
);
This span represents the execution of an AI agent, capturing the full lifecycle from receiving a task to producing a final response.
Key attributes:
gen_ai.agent.name— The agent's name (e.g., "Weather Agent")gen_ai.request.model— The underlying model usedgen_ai.response.text— The agent's final outputgen_ai.usage.input_tokens/output_tokens— Total token counts
await Sentry.startSpan(
{
op: "gen_ai.invoke_agent",
name: "invoke_agent Weather Agent",
attributes: {
"gen_ai.request.model": "o3-mini",
"gen_ai.agent.name": "Weather Agent",
},
},
async (span) => {
// Run the agent
const result = await myAgent.run();
span.setAttribute(
"gen_ai.response.text",
JSON.stringify([result.output]),
);
// Set token usage
span.setAttribute(
"gen_ai.usage.input_tokens",
result.usage.inputTokens,
);
span.setAttribute(
"gen_ai.usage.output_tokens",
result.usage.outputTokens,
);
},
);
This span represents the execution of a tool or function that was requested by an AI model, including the input arguments and resulting output.
Key attributes:
gen_ai.tool.name— The tool's name (e.g., "get_weather")gen_ai.tool.input— The arguments passed to the toolgen_ai.tool.output— The tool's return value
await Sentry.startSpan(
{
op: "gen_ai.execute_tool",
name: "execute_tool get_weather",
attributes: {
"gen_ai.tool.name": "get_weather",
"gen_ai.tool.input": JSON.stringify({ location: "Paris" }),
},
},
async (span) => {
// Call the tool
const result = await getWeather({ location: "Paris" });
span.setAttribute("gen_ai.tool.output", JSON.stringify(result));
},
);
This span marks the transition of control from one agent to another, typically when the current agent determines another agent is better suited to handle the task.
Requirements:
opmust be"gen_ai.handoff"nameshould follow the pattern"handoff from {source} to {target}"- All Common Span Attributes should be set
The handoff span itself has no body — it just marks the transition point before the target agent starts.
await Sentry.startSpan(
{
op: "gen_ai.handoff",
name: "handoff from Weather Agent to Travel Agent",
},
() => {}, // Handoff span just marks the transition
);
await Sentry.startSpan(
{ op: "gen_ai.invoke_agent", name: "invoke_agent Travel Agent" },
async () => {
// Run the target agent here
},
);
Some attributes are common to all AI Agents spans:
| Data Attribute | Type | Requirement Level | Description | Example |
|---|---|---|---|---|
gen_ai.request.model | string | required | The name of the AI model a request is being made to. | "o3-mini" |
gen_ai.operation.name | string | optional | The name of the operation being performed. | "summarize" |
gen_ai.agent.name | string | optional | The name of the agent this span belongs to. | "Weather Agent" |
When manually setting token attributes, be aware of how Sentry uses them to calculate model costs.
Cached and reasoning tokens are subsets, not separate counts. gen_ai.usage.input_tokens is the total input token count that already includes any cached tokens. Similarly, gen_ai.usage.output_tokens already includes reasoning tokens. Sentry subtracts the cached/reasoning counts from the totals to compute the "raw" portion, so reporting them incorrectly can produce wrong or negative costs.
For example, say your LLM call uses 100 input tokens total, 90 of which were served from cache. Using a standard rate of $0.01 per token and a cached rate of $0.001 per token:
Correct — input_tokens is the total (includes cached):
gen_ai.usage.input_tokens = 100gen_ai.usage.input_tokens.cached = 90- Sentry calculates:
(100 - 90) × $0.01 + 90 × $0.001=$0.10 + $0.09= $0.19 ✓
Wrong — input_tokens set to only the non-cached tokens, making cached larger than total:
gen_ai.usage.input_tokens = 10gen_ai.usage.input_tokens.cached = 90- Sentry calculates:
(10 - 90) × $0.01 + 90 × $0.001=−$0.80 + $0.09= −$0.71
Because input_tokens.cached (90) is larger than input_tokens (10), the subtraction goes negative, resulting in a negative total cost.
The same applies to gen_ai.usage.output_tokens and gen_ai.usage.output_tokens.reasoning.
If you're using an AI framework with a Sentry exporter, you can send traces to Sentry:
If you're building MCP (Model Context Protocol) servers, Sentry can also track tool executions, prompt retrievals, and resource access. See Instrument MCP Servers for setup instructions.
Our documentation is open source and available on GitHub. Your contributions are welcome, whether fixing a typo (drat!) or suggesting an update ("yeah, this would be better").