Mastra Classifier Evaluations

Learn how Sentry traces Mastra Classifier evaluations, such as calls to TypeSafe's Jev, as gen_ai.decide spans.

Available since: v11.7.0

Mastra's Classifier asks a decision model, such as TypeSafe's Jev, typed questions about your application state. These calls don't go through the Vercel AI SDK, so the Mastra integration traces them for you. This includes classifiers that your processors and scorers call. If you've followed the Mastra Quick Start, no additional integration or setup is needed.

src/mastra/classifiers/ticket-classifier.ts
Copied
import { Classifier } from "@mastra/core/classifier";
import { typeSafeAi } from "@ai-sdk/typesafe-ai";

const classifier = new Classifier({
  id: "ticket-classifier",
  model: typeSafeAi.evaluationModel("jev-latest"),
  questions: {
    department: {
      type: "choice",
      instructions: "Which team should handle this?",
      criteria: {
        billing: "Charges and refunds",
        technical: "Bugs and outages",
      },
    },
  },
});

const result = await classifier.evaluate({
  state: {
    message: "I was charged twice. Please refund the extra charge.",
  },
});

console.log(result.answers.department.choice);

Each evaluate() call creates a gen_ai.decide span named decide <model>, with the model, provider, latency, and token usage. If the evaluation fails, the span is marked as failed. When recording is enabled, Sentry captures state and questions as input messages and answers as output messages, including probabilities returned by the model.

Sentry runs this data through your Mastra span output processors before recording it, so Mastra's default SensitiveDataFilter redacts sensitive keys such as apiKey. Evaluation state and answers can still contain sensitive data. To stop recording them, set genAI: { inputs: false, outputs: false } in the dataCollection option.

  • @sentry/node: >=11.7.0
  • @mastra/core: >=1.69.0

If you call Jev through TypeSafeClient.systemOne() or Vercel AI SDK's experimental_decide() instead, see the TypeSafe or Vercel AI integration.

Was this helpful?
Help improve this content
Our documentation is open source and available on GitHub. Your contributions are welcome, whether fixing a typo (drat!) or suggesting an update ("yeah, this would be better").