a8fc9c7715
* fix(shellcheck): declare and assign separately in router.sh (SC2155) * feat(plugin): enhance context management and skill invocation - Update .context-manifest.json to include profile and detailed source information. - Revise context-scout.md to clarify context root discovery and navigation. - Modify commands to replace deprecated skills with updated ones (brainstorming → approach, systematic-debugging → debugger). - Streamline install-context.md for clarity and remove interactive profile selection. - Improve oac-status.md to provide a more comprehensive status report. - Add OAC system paths to session-start.sh for better context management. - Remove obsolete skills and improve documentation for existing commands.
1.4 KiB
1.4 KiB
Concept: Mastra Evaluations
Purpose: Quality assurance and scoring for LLM outputs.
Last Updated: 2026-01-09
Core Idea
Evaluations in Mastra use Scorers to assess the quality, accuracy, and safety of LLM-generated content. They provide a quantitative way to measure performance and detect issues like hallucinations or factual errors.
Key Points
- Scorers: Specialized functions that take LLM output (and optionally ground truth) and return a score (0-1).
- Integration: Registered in the Mastra instance and can be triggered automatically during workflow execution.
- Metrics: Common metrics include hallucination detection, fact validation, and relevance scoring.
- Audit Trail: Scorer results are stored in the
mastra_scorerstable for long-term analysis and reporting.
Quick Example
// Scorer definition
export const hallucinationDetector = new Scorer({
id: 'hallucination-detector',
description: 'Detects hallucinations in LLM output',
execute: async ({ output, context }) => {
// Logic to detect hallucinations
return { score: 0.95, rationale: 'No hallucinations found' };
},
});
// Registration
export const mastra = new Mastra({
scorers: { hallucinationDetector },
});
Reference: src/mastra/scorers/, src/mastra/evaluation/
Related:
- concepts/core.md
- concepts/workflows.md