Add evaluation framework structure and OpenCode logging documentation
- Add comprehensive evaluation framework in evals/ - Reusable framework for testing OpenCode agents - OpenAgent-specific test configuration - Scalable structure for multiple agents - Add OpenCode logging system documentation - Complete schema documentation for session storage - Model and provider tracking details - Performance analysis examples - Cost optimization strategies - Update .gitignore - Ignore evals/results/ directory - Ignore tasks/ directory (temporary) Structure: - evals/framework/ - Reusable evaluation framework - evals/opencode/openagent/ - OpenAgent tests - dev/ai-tools/opencode/logging-and-session-storage.md - Logging docs
This commit is contained in:
darrenhinde
parent
c6775bbc40
commit
f4b3d56aa2
11 changed files with 3038 additions and 1 deletions
7
.gitignore
vendored
7
.gitignore
vendored
|
|
@ -197,3 +197,10 @@ temp/
|
||||||
|
|
||||||
# Bun
|
# Bun
|
||||||
.bun
|
.bun
|
||||||
|
|
||||||
|
# Evaluation results
|
||||||
|
evals/results/*
|
||||||
|
!evals/results/.gitkeep
|
||||||
|
|
||||||
|
# Task tracking (temporary)
|
||||||
|
tasks/
|
||||||
1692
dev/ai-tools/opencode/logging-and-session-storage.md
Normal file
1692
dev/ai-tools/opencode/logging-and-session-storage.md
Normal file
File diff suppressed because it is too large
Load diff
279
evals/README.md
Normal file
279
evals/README.md
Normal file
|
|
@ -0,0 +1,279 @@
|
||||||
|
# OpenCode Agent Evaluation Framework
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Comprehensive evaluation framework for testing and validating OpenCode agent behavior against defined standards and rules.
|
||||||
|
|
||||||
|
## Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
evals/
|
||||||
|
├── framework/ # Reusable evaluation framework
|
||||||
|
│ ├── src/
|
||||||
|
│ │ ├── collector/ # Session data collection
|
||||||
|
│ │ ├── evaluators/ # Evaluation logic
|
||||||
|
│ │ ├── runner/ # Test execution
|
||||||
|
│ │ ├── reporters/ # Result reporting
|
||||||
|
│ │ └── types/ # TypeScript types
|
||||||
|
│ └── package.json
|
||||||
|
│
|
||||||
|
├── opencode/ # OpenCode agent evaluations
|
||||||
|
│ ├── openagent/ # OpenAgent-specific tests
|
||||||
|
│ ├── opencoder/ # OpenCoder-specific tests
|
||||||
|
│ └── shared/ # Shared test cases
|
||||||
|
│
|
||||||
|
└── results/ # Test results (gitignored)
|
||||||
|
└── YYYY-MM-DD/
|
||||||
|
```
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
### 1. Install Framework
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd evals/framework
|
||||||
|
npm install
|
||||||
|
npm run build
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Run Evaluations
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Evaluate a specific session
|
||||||
|
npm run eval -- --agent openagent --session ses_xxxxx
|
||||||
|
|
||||||
|
# Run all OpenAgent tests
|
||||||
|
npm run eval -- --agent openagent --all
|
||||||
|
|
||||||
|
# Run specific test case
|
||||||
|
npm run eval -- --agent openagent --test approval-gates
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. View Results
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# View latest results
|
||||||
|
cat evals/results/$(ls -t evals/results | head -1)/openagent/summary.json | jq
|
||||||
|
|
||||||
|
# Generate report
|
||||||
|
npm run report -- --agent openagent --date 2025-11-21
|
||||||
|
```
|
||||||
|
|
||||||
|
## Framework Components
|
||||||
|
|
||||||
|
### Collector
|
||||||
|
Reads and parses OpenCode session data from `~/.local/share/opencode/`
|
||||||
|
|
||||||
|
**Components:**
|
||||||
|
- `SessionReader` - Read session files
|
||||||
|
- `MessageParser` - Parse message structure
|
||||||
|
- `TimelineBuilder` - Build event timeline
|
||||||
|
|
||||||
|
### Evaluators
|
||||||
|
Validate agent behavior against rules
|
||||||
|
|
||||||
|
**Available Evaluators:**
|
||||||
|
- `ApprovalGateEvaluator` - Check approval before execution
|
||||||
|
- `ContextLoadingEvaluator` - Verify context loading
|
||||||
|
- `DelegationEvaluator` - Validate delegation decisions
|
||||||
|
- `ToolUsageEvaluator` - Check tool selection
|
||||||
|
- `ModelSelectionEvaluator` - Validate model choices
|
||||||
|
|
||||||
|
### Runner
|
||||||
|
Execute test cases and collect results
|
||||||
|
|
||||||
|
**Components:**
|
||||||
|
- `TestRunner` - Run test suites
|
||||||
|
- `SessionAnalyzer` - Analyze historical sessions
|
||||||
|
- `LiveRunner` - Run live tests (future)
|
||||||
|
|
||||||
|
### Reporters
|
||||||
|
Generate test reports
|
||||||
|
|
||||||
|
**Formats:**
|
||||||
|
- Console (pretty output)
|
||||||
|
- JSON (machine-readable)
|
||||||
|
- Markdown (documentation)
|
||||||
|
- HTML (dashboard, future)
|
||||||
|
|
||||||
|
## Agent-Specific Tests
|
||||||
|
|
||||||
|
### OpenAgent (`opencode/openagent/`)
|
||||||
|
|
||||||
|
Tests for the universal OpenAgent:
|
||||||
|
- Approval gate enforcement
|
||||||
|
- Context loading compliance
|
||||||
|
- Delegation appropriateness
|
||||||
|
- Tool usage patterns
|
||||||
|
- Model selection
|
||||||
|
|
||||||
|
**Test Categories:**
|
||||||
|
- Conversational (simple questions)
|
||||||
|
- Task (file operations)
|
||||||
|
- Complex (multi-file features)
|
||||||
|
- Edge cases (errors, permissions)
|
||||||
|
|
||||||
|
### OpenCoder (`opencode/opencoder/`)
|
||||||
|
|
||||||
|
Tests for the OpenCoder agent (future)
|
||||||
|
|
||||||
|
### Shared (`opencode/shared/`)
|
||||||
|
|
||||||
|
Common test cases applicable to all agents
|
||||||
|
|
||||||
|
## Test Case Format
|
||||||
|
|
||||||
|
Test cases are defined in YAML:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
test_cases:
|
||||||
|
- id: simple-question
|
||||||
|
name: "Simple Question (No Execution)"
|
||||||
|
description: "Ask a question that requires no execution tools"
|
||||||
|
category: conversational
|
||||||
|
input: "What does this code do?"
|
||||||
|
expected_behavior:
|
||||||
|
no_execution_tools: true
|
||||||
|
no_approval_required: true
|
||||||
|
response_provided: true
|
||||||
|
evaluators:
|
||||||
|
- approval-gate
|
||||||
|
- tool-usage
|
||||||
|
pass_threshold: 100
|
||||||
|
```
|
||||||
|
|
||||||
|
## Evaluation Results
|
||||||
|
|
||||||
|
Results include:
|
||||||
|
- **Pass/Fail** - Did the test pass?
|
||||||
|
- **Score** - 0-100 compliance score
|
||||||
|
- **Violations** - List of rule violations
|
||||||
|
- **Evidence** - Supporting data for violations
|
||||||
|
- **Metadata** - Agent, model, timing, costs
|
||||||
|
|
||||||
|
**Example Result:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"testCaseId": "simple-question",
|
||||||
|
"sessionId": "ses_xxxxx",
|
||||||
|
"passed": true,
|
||||||
|
"score": 100,
|
||||||
|
"evaluationResults": [
|
||||||
|
{
|
||||||
|
"evaluator": "approval-gate",
|
||||||
|
"passed": true,
|
||||||
|
"score": 100,
|
||||||
|
"violations": []
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"agent": "openagent",
|
||||||
|
"model": "claude-sonnet-4-20250514",
|
||||||
|
"duration": 7205,
|
||||||
|
"cost": 0.107
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Scoring System
|
||||||
|
|
||||||
|
- **100** - Perfect compliance
|
||||||
|
- **75-99** - Minor issues (warnings)
|
||||||
|
- **50-74** - Moderate issues (violations)
|
||||||
|
- **0-49** - Major issues (critical violations)
|
||||||
|
|
||||||
|
**Pass Threshold:** 75% (configurable per test)
|
||||||
|
|
||||||
|
## Development
|
||||||
|
|
||||||
|
### Adding New Evaluators
|
||||||
|
|
||||||
|
1. Create evaluator in `framework/src/evaluators/`
|
||||||
|
2. Extend `BaseEvaluator`
|
||||||
|
3. Implement `evaluate()` method
|
||||||
|
4. Add tests
|
||||||
|
5. Register in evaluator registry
|
||||||
|
|
||||||
|
### Adding New Test Cases
|
||||||
|
|
||||||
|
1. Create YAML file in `opencode/{agent}/test-cases/`
|
||||||
|
2. Define test case structure
|
||||||
|
3. Specify expected behavior
|
||||||
|
4. List evaluators to run
|
||||||
|
5. Set pass threshold
|
||||||
|
|
||||||
|
### Running Tests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Framework tests
|
||||||
|
cd evals/framework
|
||||||
|
npm test
|
||||||
|
|
||||||
|
# Integration tests
|
||||||
|
npm run test:integration
|
||||||
|
|
||||||
|
# Test specific evaluator
|
||||||
|
npm run test:evaluator -- approval-gate
|
||||||
|
```
|
||||||
|
|
||||||
|
## CI/CD Integration
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# .github/workflows/eval.yml
|
||||||
|
name: Agent Evaluation
|
||||||
|
|
||||||
|
on: [push, pull_request]
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
evaluate:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v3
|
||||||
|
- uses: actions/setup-node@v3
|
||||||
|
- run: cd evals/framework && npm install
|
||||||
|
- run: npm run eval -- --agent openagent --all
|
||||||
|
- run: npm run report -- --format json
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
### Framework Config (`framework/config.ts`)
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
export const config = {
|
||||||
|
projectPath: process.cwd(),
|
||||||
|
sessionStoragePath: '~/.local/share/opencode/',
|
||||||
|
resultsPath: 'evals/results/',
|
||||||
|
passThreshold: 75,
|
||||||
|
evaluators: ['approval-gate', 'context-loading', 'delegation', 'tool-usage']
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
### Agent Config (`opencode/openagent/config.yaml`)
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
agent: openagent
|
||||||
|
test_cases_path: ./test-cases
|
||||||
|
sessions_path: ./sessions
|
||||||
|
evaluators:
|
||||||
|
- approval-gate
|
||||||
|
- context-loading
|
||||||
|
- delegation
|
||||||
|
- tool-usage
|
||||||
|
pass_threshold: 75
|
||||||
|
```
|
||||||
|
|
||||||
|
## Related Documentation
|
||||||
|
|
||||||
|
- [Framework Architecture](./framework/README.md)
|
||||||
|
- [OpenAgent Tests](./opencode/openagent/README.md)
|
||||||
|
- [OpenCode Logging System](../dev/ai-tools/opencode/logging-and-session-storage.md)
|
||||||
|
- [Agent Validator Plugin](../.opencode/plugin/docs/VALIDATOR_GUIDE.md)
|
||||||
|
|
||||||
|
## Contributing
|
||||||
|
|
||||||
|
See [CONTRIBUTING.md](../docs/contributing/CONTRIBUTING.md)
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
MIT
|
||||||
348
evals/SETUP_COMPLETE.md
Normal file
348
evals/SETUP_COMPLETE.md
Normal file
|
|
@ -0,0 +1,348 @@
|
||||||
|
# Evaluation Framework Setup Complete ✅
|
||||||
|
|
||||||
|
## What Was Created
|
||||||
|
|
||||||
|
### Directory Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
evals/
|
||||||
|
├── README.md # Main overview
|
||||||
|
├── framework/ # Reusable evaluation framework
|
||||||
|
│ ├── README.md # Framework documentation
|
||||||
|
│ ├── package.json # Dependencies
|
||||||
|
│ ├── tsconfig.json # TypeScript config
|
||||||
|
│ ├── .gitignore # Git ignore rules
|
||||||
|
│ └── src/ # Source code (empty, ready for implementation)
|
||||||
|
│ ├── collector/ # Session data collection
|
||||||
|
│ ├── evaluators/ # Evaluation logic
|
||||||
|
│ ├── runner/ # Test execution
|
||||||
|
│ ├── reporters/ # Result reporting
|
||||||
|
│ └── types/ # TypeScript types
|
||||||
|
│
|
||||||
|
├── opencode/ # OpenCode agent evaluations
|
||||||
|
│ ├── openagent/ # OpenAgent-specific tests
|
||||||
|
│ │ ├── README.md # OpenAgent test documentation
|
||||||
|
│ │ ├── config/
|
||||||
|
│ │ │ └── config.yaml # OpenAgent configuration
|
||||||
|
│ │ ├── test-cases/ # Test case definitions (empty)
|
||||||
|
│ │ └── sessions/ # Recorded test sessions (empty)
|
||||||
|
│ └── shared/ # Shared test cases (empty)
|
||||||
|
│
|
||||||
|
└── results/ # Test results (gitignored)
|
||||||
|
└── .gitkeep
|
||||||
|
```
|
||||||
|
|
||||||
|
### Task Documentation
|
||||||
|
|
||||||
|
```
|
||||||
|
tasks/eval-framework/
|
||||||
|
├── README.md # Project overview
|
||||||
|
├── 01-session-reader.md # Task 1: Session Reader
|
||||||
|
├── 02-evaluators.md # Task 2: Core Evaluators
|
||||||
|
└── 03-test-runner.md # Task 3: Test Runner
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configuration Files
|
||||||
|
|
||||||
|
1. **Framework Package** (`evals/framework/package.json`)
|
||||||
|
- TypeScript setup
|
||||||
|
- Testing with Vitest
|
||||||
|
- Build scripts
|
||||||
|
- Linting
|
||||||
|
|
||||||
|
2. **OpenAgent Config** (`evals/opencode/openagent/config/config.yaml`)
|
||||||
|
- Evaluator settings
|
||||||
|
- Scoring weights
|
||||||
|
- Rule definitions
|
||||||
|
- Model preferences
|
||||||
|
|
||||||
|
3. **Git Ignore** (`.gitignore`)
|
||||||
|
- Results directory ignored
|
||||||
|
- Node modules ignored
|
||||||
|
- Build artifacts ignored
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
### Phase 1: Foundation (Start Here)
|
||||||
|
|
||||||
|
**Goal:** Build session reader and basic types
|
||||||
|
|
||||||
|
**Tasks:**
|
||||||
|
1. Implement TypeScript types (`framework/src/types/index.ts`)
|
||||||
|
2. Implement SessionReader (`framework/src/collector/session-reader.ts`)
|
||||||
|
3. Implement MessageParser (`framework/src/collector/message-parser.ts`)
|
||||||
|
4. Implement TimelineBuilder (`framework/src/collector/timeline-builder.ts`)
|
||||||
|
5. Write tests
|
||||||
|
|
||||||
|
**See:** [tasks/eval-framework/01-session-reader.md](../tasks/eval-framework/01-session-reader.md)
|
||||||
|
|
||||||
|
**Estimated Time:** 4-6 hours
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Phase 2: Core Evaluators
|
||||||
|
|
||||||
|
**Goal:** Implement evaluation logic
|
||||||
|
|
||||||
|
**Tasks:**
|
||||||
|
1. Implement BaseEvaluator
|
||||||
|
2. Implement ApprovalGateEvaluator
|
||||||
|
3. Implement ContextLoadingEvaluator
|
||||||
|
4. Implement DelegationEvaluator
|
||||||
|
5. Implement ToolUsageEvaluator
|
||||||
|
6. Write tests
|
||||||
|
|
||||||
|
**See:** [tasks/eval-framework/02-evaluators.md](../tasks/eval-framework/02-evaluators.md)
|
||||||
|
|
||||||
|
**Estimated Time:** 8-10 hours
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Phase 3: Test Runner
|
||||||
|
|
||||||
|
**Goal:** Execute test cases and generate results
|
||||||
|
|
||||||
|
**Tasks:**
|
||||||
|
1. Implement test case loader
|
||||||
|
2. Implement SessionAnalyzer
|
||||||
|
3. Implement TestRunner
|
||||||
|
4. Create test case library
|
||||||
|
5. Write tests
|
||||||
|
|
||||||
|
**See:** [tasks/eval-framework/03-test-runner.md](../tasks/eval-framework/03-test-runner.md)
|
||||||
|
|
||||||
|
**Estimated Time:** 6-8 hours
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Phase 4: Reporters (Future)
|
||||||
|
|
||||||
|
**Goal:** Generate useful reports
|
||||||
|
|
||||||
|
**Tasks:**
|
||||||
|
1. Implement ConsoleReporter
|
||||||
|
2. Implement JSONReporter
|
||||||
|
3. Implement MarkdownReporter
|
||||||
|
4. Add trend analysis
|
||||||
|
|
||||||
|
**Estimated Time:** 4-6 hours
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
### 1. Install Dependencies
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd evals/framework
|
||||||
|
npm install
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Start Building
|
||||||
|
|
||||||
|
Begin with Phase 1 - Session Reader:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Create types file
|
||||||
|
touch src/types/index.ts
|
||||||
|
|
||||||
|
# Create session reader
|
||||||
|
touch src/collector/session-reader.ts
|
||||||
|
|
||||||
|
# Create message parser
|
||||||
|
touch src/collector/message-parser.ts
|
||||||
|
|
||||||
|
# Create timeline builder
|
||||||
|
touch src/collector/timeline-builder.ts
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Run Tests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run tests
|
||||||
|
npm test
|
||||||
|
|
||||||
|
# Watch mode
|
||||||
|
npm run test:watch
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Build
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Build TypeScript
|
||||||
|
npm run build
|
||||||
|
|
||||||
|
# Watch mode
|
||||||
|
npm run build:watch
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture Overview
|
||||||
|
|
||||||
|
### Data Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
OpenCode Session Storage
|
||||||
|
↓
|
||||||
|
SessionReader (reads JSON files)
|
||||||
|
↓
|
||||||
|
MessageParser (parses structure)
|
||||||
|
↓
|
||||||
|
TimelineBuilder (builds timeline)
|
||||||
|
↓
|
||||||
|
Evaluators (validate behavior)
|
||||||
|
↓
|
||||||
|
TestRunner (executes tests)
|
||||||
|
↓
|
||||||
|
Reporters (generate reports)
|
||||||
|
↓
|
||||||
|
Results (JSON/Markdown/Console)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Key Components
|
||||||
|
|
||||||
|
**Collector:**
|
||||||
|
- Reads OpenCode session data
|
||||||
|
- Parses messages and parts
|
||||||
|
- Builds chronological timeline
|
||||||
|
|
||||||
|
**Evaluators:**
|
||||||
|
- Validate agent behavior
|
||||||
|
- Check compliance with rules
|
||||||
|
- Generate scores and violations
|
||||||
|
|
||||||
|
**Runner:**
|
||||||
|
- Load test case definitions
|
||||||
|
- Execute evaluators
|
||||||
|
- Collect results
|
||||||
|
|
||||||
|
**Reporters:**
|
||||||
|
- Format results
|
||||||
|
- Generate reports
|
||||||
|
- Export data
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
### Framework Config
|
||||||
|
|
||||||
|
Edit `framework/src/config.ts` (to be created):
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
export const config = {
|
||||||
|
projectPath: process.cwd(),
|
||||||
|
sessionStoragePath: '~/.local/share/opencode/',
|
||||||
|
resultsPath: '../results/',
|
||||||
|
passThreshold: 75,
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
### OpenAgent Config
|
||||||
|
|
||||||
|
Edit `opencode/openagent/config/config.yaml`:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
agent: openagent
|
||||||
|
pass_threshold: 75
|
||||||
|
scoring:
|
||||||
|
approval_gate: 40
|
||||||
|
context_loading: 40
|
||||||
|
delegation: 10
|
||||||
|
tool_usage: 10
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Testing Strategy
|
||||||
|
|
||||||
|
### Unit Tests
|
||||||
|
Test individual components in isolation:
|
||||||
|
- SessionReader
|
||||||
|
- MessageParser
|
||||||
|
- TimelineBuilder
|
||||||
|
- Each evaluator
|
||||||
|
|
||||||
|
### Integration Tests
|
||||||
|
Test components working together:
|
||||||
|
- Read real session data
|
||||||
|
- Build timeline
|
||||||
|
- Run evaluators
|
||||||
|
- Generate results
|
||||||
|
|
||||||
|
### Regression Tests
|
||||||
|
Test against recorded sessions:
|
||||||
|
- Store known-good sessions
|
||||||
|
- Run tests against them
|
||||||
|
- Detect behavior changes
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
### Main Docs
|
||||||
|
- [Evaluation Framework Overview](./README.md)
|
||||||
|
- [Framework Documentation](./framework/README.md)
|
||||||
|
- [OpenAgent Tests](./opencode/openagent/README.md)
|
||||||
|
|
||||||
|
### Task Breakdown
|
||||||
|
- [Project Overview](../tasks/eval-framework/README.md)
|
||||||
|
- [Task 1: Session Reader](../tasks/eval-framework/01-session-reader.md)
|
||||||
|
- [Task 2: Core Evaluators](../tasks/eval-framework/02-evaluators.md)
|
||||||
|
- [Task 3: Test Runner](../tasks/eval-framework/03-test-runner.md)
|
||||||
|
|
||||||
|
### Related Docs
|
||||||
|
- [OpenCode Logging System](../dev/ai-tools/opencode/logging-and-session-storage.md)
|
||||||
|
- [Agent Validator Plugin](../.opencode/plugin/docs/VALIDATOR_GUIDE.md)
|
||||||
|
- [OpenAgent Specification](../.opencode/agent/openagent.md)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Success Criteria
|
||||||
|
|
||||||
|
### Phase 1 Complete When:
|
||||||
|
- [ ] Can read any OpenCode session
|
||||||
|
- [ ] Can parse messages and parts
|
||||||
|
- [ ] Can build complete timeline
|
||||||
|
- [ ] All types defined
|
||||||
|
- [ ] Tests pass
|
||||||
|
|
||||||
|
### Phase 2 Complete When:
|
||||||
|
- [ ] All evaluators implemented
|
||||||
|
- [ ] Can detect violations
|
||||||
|
- [ ] Scoring system works
|
||||||
|
- [ ] Tests pass
|
||||||
|
|
||||||
|
### Phase 3 Complete When:
|
||||||
|
- [ ] Can load test cases from YAML
|
||||||
|
- [ ] Can run test suites
|
||||||
|
- [ ] Can generate results
|
||||||
|
- [ ] Tests pass
|
||||||
|
|
||||||
|
### Phase 4 Complete When:
|
||||||
|
- [ ] Can generate console reports
|
||||||
|
- [ ] Can export JSON
|
||||||
|
- [ ] Can generate Markdown
|
||||||
|
- [ ] Tests pass
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Getting Help
|
||||||
|
|
||||||
|
- Review task documentation in `tasks/eval-framework/`
|
||||||
|
- Check OpenCode logging docs in `dev/ai-tools/opencode/`
|
||||||
|
- Examine existing session data in `~/.local/share/opencode/`
|
||||||
|
- Look at validator plugin in `.opencode/plugin/agent-validator.ts`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Ready to Start!
|
||||||
|
|
||||||
|
Begin with **Phase 1: Session Reader**
|
||||||
|
|
||||||
|
See: [tasks/eval-framework/01-session-reader.md](../tasks/eval-framework/01-session-reader.md)
|
||||||
|
|
||||||
|
Good luck! 🚀
|
||||||
6
evals/framework/.gitignore
vendored
Normal file
6
evals/framework/.gitignore
vendored
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
node_modules/
|
||||||
|
dist/
|
||||||
|
*.log
|
||||||
|
.DS_Store
|
||||||
|
coverage/
|
||||||
|
.env
|
||||||
341
evals/framework/README.md
Normal file
341
evals/framework/README.md
Normal file
|
|
@ -0,0 +1,341 @@
|
||||||
|
# Evaluation Framework
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Reusable framework for evaluating OpenCode agent behavior. Can be used to test any agent against defined standards.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
framework/
|
||||||
|
├── src/
|
||||||
|
│ ├── collector/ # Session data collection
|
||||||
|
│ │ ├── session-reader.ts
|
||||||
|
│ │ ├── message-parser.ts
|
||||||
|
│ │ └── timeline-builder.ts
|
||||||
|
│ ├── evaluators/ # Evaluation logic
|
||||||
|
│ │ ├── base-evaluator.ts
|
||||||
|
│ │ ├── approval-gate.ts
|
||||||
|
│ │ ├── context-loading.ts
|
||||||
|
│ │ ├── delegation.ts
|
||||||
|
│ │ ├── tool-usage.ts
|
||||||
|
│ │ └── model-selection.ts
|
||||||
|
│ ├── runner/ # Test execution
|
||||||
|
│ │ ├── test-runner.ts
|
||||||
|
│ │ └── session-analyzer.ts
|
||||||
|
│ ├── reporters/ # Result reporting
|
||||||
|
│ │ ├── console-reporter.ts
|
||||||
|
│ │ ├── json-reporter.ts
|
||||||
|
│ │ └── markdown-reporter.ts
|
||||||
|
│ ├── types/ # TypeScript types
|
||||||
|
│ │ └── index.ts
|
||||||
|
│ ├── config.ts # Configuration
|
||||||
|
│ └── index.ts # Main exports
|
||||||
|
├── tests/ # Framework tests
|
||||||
|
├── package.json
|
||||||
|
├── tsconfig.json
|
||||||
|
└── README.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install
|
||||||
|
```
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
### Basic Example
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
import { SessionReader, ApprovalGateEvaluator, TimelineBuilder } from '@evals/framework';
|
||||||
|
|
||||||
|
// Read session
|
||||||
|
const reader = new SessionReader('/path/to/project');
|
||||||
|
const sessionInfo = reader.getSessionInfo('ses_xxxxx');
|
||||||
|
const messages = reader.getMessages('ses_xxxxx');
|
||||||
|
|
||||||
|
// Build timeline
|
||||||
|
const builder = new TimelineBuilder(reader);
|
||||||
|
const timeline = builder.buildTimeline('ses_xxxxx');
|
||||||
|
|
||||||
|
// Evaluate
|
||||||
|
const evaluator = new ApprovalGateEvaluator();
|
||||||
|
const result = evaluator.evaluate(timeline);
|
||||||
|
|
||||||
|
console.log(`Passed: ${result.passed}`);
|
||||||
|
console.log(`Score: ${result.score}/100`);
|
||||||
|
console.log(`Violations: ${result.violations.length}`);
|
||||||
|
```
|
||||||
|
|
||||||
|
### Running Test Suite
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
import { TestRunner } from '@evals/framework';
|
||||||
|
|
||||||
|
const runner = new TestRunner({
|
||||||
|
projectPath: process.cwd(),
|
||||||
|
evaluatorsPath: './src/evaluators',
|
||||||
|
resultsPath: '../results'
|
||||||
|
});
|
||||||
|
|
||||||
|
// Load test cases
|
||||||
|
const testCases = runner.loadTestCases('../opencode/openagent/test-cases/approval-gates.yaml');
|
||||||
|
|
||||||
|
// Run tests
|
||||||
|
const suite = runner.runAll(testCases);
|
||||||
|
|
||||||
|
console.log(`Pass Rate: ${suite.summary.passRate}%`);
|
||||||
|
```
|
||||||
|
|
||||||
|
## Components
|
||||||
|
|
||||||
|
### Collector
|
||||||
|
|
||||||
|
**SessionReader** - Read OpenCode session files
|
||||||
|
```typescript
|
||||||
|
const reader = new SessionReader(projectPath);
|
||||||
|
const info = reader.getSessionInfo(sessionId);
|
||||||
|
const messages = reader.getMessages(sessionId);
|
||||||
|
const parts = reader.getParts(sessionId, messageId);
|
||||||
|
```
|
||||||
|
|
||||||
|
**MessageParser** - Parse message structure
|
||||||
|
```typescript
|
||||||
|
const parser = new MessageParser();
|
||||||
|
const agent = parser.getAgent(message);
|
||||||
|
const model = parser.getModel(message);
|
||||||
|
const metrics = parser.getMetrics(message);
|
||||||
|
```
|
||||||
|
|
||||||
|
**TimelineBuilder** - Build event timeline
|
||||||
|
```typescript
|
||||||
|
const builder = new TimelineBuilder(reader);
|
||||||
|
const timeline = builder.buildTimeline(sessionId);
|
||||||
|
const toolCalls = builder.filterByType(timeline, 'tool_call');
|
||||||
|
```
|
||||||
|
|
||||||
|
### Evaluators
|
||||||
|
|
||||||
|
All evaluators extend `BaseEvaluator` and implement:
|
||||||
|
```typescript
|
||||||
|
evaluate(timeline: TimelineEvent[]): EvaluationResult
|
||||||
|
```
|
||||||
|
|
||||||
|
**ApprovalGateEvaluator** - Check approval before execution
|
||||||
|
```typescript
|
||||||
|
const evaluator = new ApprovalGateEvaluator();
|
||||||
|
const result = evaluator.evaluate(timeline);
|
||||||
|
```
|
||||||
|
|
||||||
|
**ContextLoadingEvaluator** - Verify context loading
|
||||||
|
```typescript
|
||||||
|
const evaluator = new ContextLoadingEvaluator();
|
||||||
|
const result = evaluator.evaluate(timeline);
|
||||||
|
```
|
||||||
|
|
||||||
|
**DelegationEvaluator** - Validate delegation decisions
|
||||||
|
```typescript
|
||||||
|
const evaluator = new DelegationEvaluator();
|
||||||
|
const result = evaluator.evaluate(timeline);
|
||||||
|
```
|
||||||
|
|
||||||
|
**ToolUsageEvaluator** - Check tool selection
|
||||||
|
```typescript
|
||||||
|
const evaluator = new ToolUsageEvaluator();
|
||||||
|
const result = evaluator.evaluate(timeline);
|
||||||
|
```
|
||||||
|
|
||||||
|
### Runner
|
||||||
|
|
||||||
|
**TestRunner** - Execute test suites
|
||||||
|
```typescript
|
||||||
|
const runner = new TestRunner(config);
|
||||||
|
const testCases = runner.loadTestCases(path);
|
||||||
|
const suite = runner.runAll(testCases);
|
||||||
|
```
|
||||||
|
|
||||||
|
**SessionAnalyzer** - Analyze historical sessions
|
||||||
|
```typescript
|
||||||
|
const analyzer = new SessionAnalyzer(reader);
|
||||||
|
const result = analyzer.analyze(sessionId, testCase);
|
||||||
|
```
|
||||||
|
|
||||||
|
### Reporters
|
||||||
|
|
||||||
|
**ConsoleReporter** - Pretty console output
|
||||||
|
```typescript
|
||||||
|
const reporter = new ConsoleReporter();
|
||||||
|
reporter.report(testSuite);
|
||||||
|
```
|
||||||
|
|
||||||
|
**JSONReporter** - Machine-readable JSON
|
||||||
|
```typescript
|
||||||
|
const reporter = new JSONReporter();
|
||||||
|
reporter.export(testSuite, 'results.json');
|
||||||
|
```
|
||||||
|
|
||||||
|
**MarkdownReporter** - Documentation format
|
||||||
|
```typescript
|
||||||
|
const reporter = new MarkdownReporter();
|
||||||
|
reporter.generate(testSuite, 'report.md');
|
||||||
|
```
|
||||||
|
|
||||||
|
## Types
|
||||||
|
|
||||||
|
### Core Types
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface SessionInfo {
|
||||||
|
id: string;
|
||||||
|
version: string;
|
||||||
|
title: string;
|
||||||
|
time: { created: number; updated: number };
|
||||||
|
}
|
||||||
|
|
||||||
|
interface Message {
|
||||||
|
id: string;
|
||||||
|
role: 'user' | 'assistant';
|
||||||
|
sessionID: string;
|
||||||
|
mode?: string;
|
||||||
|
modelID?: string;
|
||||||
|
providerID?: string;
|
||||||
|
tokens?: TokenUsage;
|
||||||
|
cost?: number;
|
||||||
|
time: { created: number; completed?: number };
|
||||||
|
}
|
||||||
|
|
||||||
|
interface TimelineEvent {
|
||||||
|
timestamp: number;
|
||||||
|
type: 'user_message' | 'assistant_message' | 'tool_call' | 'patch';
|
||||||
|
agent?: string;
|
||||||
|
model?: string;
|
||||||
|
data: any;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface EvaluationResult {
|
||||||
|
evaluator: string;
|
||||||
|
passed: boolean;
|
||||||
|
score: number;
|
||||||
|
violations: Violation[];
|
||||||
|
evidence: Evidence[];
|
||||||
|
}
|
||||||
|
|
||||||
|
interface TestResult {
|
||||||
|
testCaseId: string;
|
||||||
|
sessionId: string;
|
||||||
|
passed: boolean;
|
||||||
|
score: number;
|
||||||
|
evaluationResults: EvaluationResult[];
|
||||||
|
metadata: any;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// config.ts
|
||||||
|
export const config = {
|
||||||
|
projectPath: process.cwd(),
|
||||||
|
sessionStoragePath: '~/.local/share/opencode/',
|
||||||
|
resultsPath: '../results/',
|
||||||
|
passThreshold: 75,
|
||||||
|
evaluators: {
|
||||||
|
'approval-gate': ApprovalGateEvaluator,
|
||||||
|
'context-loading': ContextLoadingEvaluator,
|
||||||
|
'delegation': DelegationEvaluator,
|
||||||
|
'tool-usage': ToolUsageEvaluator,
|
||||||
|
}
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
## Development
|
||||||
|
|
||||||
|
### Setup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install
|
||||||
|
npm run build
|
||||||
|
```
|
||||||
|
|
||||||
|
### Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run all tests
|
||||||
|
npm test
|
||||||
|
|
||||||
|
# Run specific test
|
||||||
|
npm test -- session-reader
|
||||||
|
|
||||||
|
# Watch mode
|
||||||
|
npm run test:watch
|
||||||
|
```
|
||||||
|
|
||||||
|
### Building
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Build TypeScript
|
||||||
|
npm run build
|
||||||
|
|
||||||
|
# Watch mode
|
||||||
|
npm run build:watch
|
||||||
|
```
|
||||||
|
|
||||||
|
### Linting
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm run lint
|
||||||
|
npm run lint:fix
|
||||||
|
```
|
||||||
|
|
||||||
|
## Adding New Evaluators
|
||||||
|
|
||||||
|
1. Create file in `src/evaluators/`
|
||||||
|
2. Extend `BaseEvaluator`
|
||||||
|
3. Implement `evaluate()` method
|
||||||
|
4. Add tests
|
||||||
|
5. Register in config
|
||||||
|
6. Export from `index.ts`
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```typescript
|
||||||
|
// src/evaluators/my-evaluator.ts
|
||||||
|
import { BaseEvaluator } from './base-evaluator';
|
||||||
|
import { EvaluationResult, TimelineEvent } from '../types';
|
||||||
|
|
||||||
|
export class MyEvaluator extends BaseEvaluator {
|
||||||
|
evaluate(timeline: TimelineEvent[]): EvaluationResult {
|
||||||
|
// Your evaluation logic
|
||||||
|
const checks = [
|
||||||
|
{ name: 'check1', passed: true, weight: 50 },
|
||||||
|
{ name: 'check2', passed: false, weight: 50 }
|
||||||
|
];
|
||||||
|
|
||||||
|
const score = this.calculateScore(checks);
|
||||||
|
const violations = this.findViolations(timeline);
|
||||||
|
|
||||||
|
return {
|
||||||
|
evaluator: 'my-evaluator',
|
||||||
|
passed: score >= 75,
|
||||||
|
score,
|
||||||
|
violations,
|
||||||
|
evidence: []
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Reference
|
||||||
|
|
||||||
|
See [API.md](./API.md) for complete API documentation.
|
||||||
|
|
||||||
|
## Contributing
|
||||||
|
|
||||||
|
1. Fork the repository
|
||||||
|
2. Create feature branch
|
||||||
|
3. Add tests
|
||||||
|
4. Submit pull request
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
MIT
|
||||||
38
evals/framework/package.json
Normal file
38
evals/framework/package.json
Normal file
|
|
@ -0,0 +1,38 @@
|
||||||
|
{
|
||||||
|
"name": "@opencode-agents/eval-framework",
|
||||||
|
"version": "0.1.0",
|
||||||
|
"description": "Evaluation framework for OpenCode agents",
|
||||||
|
"main": "dist/index.js",
|
||||||
|
"types": "dist/index.d.ts",
|
||||||
|
"scripts": {
|
||||||
|
"build": "tsc",
|
||||||
|
"build:watch": "tsc --watch",
|
||||||
|
"test": "vitest",
|
||||||
|
"test:watch": "vitest --watch",
|
||||||
|
"test:integration": "vitest --config vitest.integration.config.ts",
|
||||||
|
"lint": "eslint src/**/*.ts",
|
||||||
|
"lint:fix": "eslint src/**/*.ts --fix",
|
||||||
|
"eval": "node dist/cli.js",
|
||||||
|
"report": "node dist/cli.js report"
|
||||||
|
},
|
||||||
|
"keywords": [
|
||||||
|
"opencode",
|
||||||
|
"evaluation",
|
||||||
|
"testing",
|
||||||
|
"agents",
|
||||||
|
"ai"
|
||||||
|
],
|
||||||
|
"author": "",
|
||||||
|
"license": "MIT",
|
||||||
|
"dependencies": {
|
||||||
|
"yaml": "^2.3.4"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"@types/node": "^20.10.0",
|
||||||
|
"@typescript-eslint/eslint-plugin": "^6.13.0",
|
||||||
|
"@typescript-eslint/parser": "^6.13.0",
|
||||||
|
"eslint": "^8.54.0",
|
||||||
|
"typescript": "^5.3.0",
|
||||||
|
"vitest": "^1.0.0"
|
||||||
|
}
|
||||||
|
}
|
||||||
20
evals/framework/tsconfig.json
Normal file
20
evals/framework/tsconfig.json
Normal file
|
|
@ -0,0 +1,20 @@
|
||||||
|
{
|
||||||
|
"compilerOptions": {
|
||||||
|
"target": "ES2020",
|
||||||
|
"module": "commonjs",
|
||||||
|
"lib": ["ES2020"],
|
||||||
|
"outDir": "./dist",
|
||||||
|
"rootDir": "./src",
|
||||||
|
"declaration": true,
|
||||||
|
"declarationMap": true,
|
||||||
|
"sourceMap": true,
|
||||||
|
"strict": true,
|
||||||
|
"esModuleInterop": true,
|
||||||
|
"skipLibCheck": true,
|
||||||
|
"forceConsistentCasingInFileNames": true,
|
||||||
|
"resolveJsonModule": true,
|
||||||
|
"moduleResolution": "node"
|
||||||
|
},
|
||||||
|
"include": ["src/**/*"],
|
||||||
|
"exclude": ["node_modules", "dist", "tests"]
|
||||||
|
}
|
||||||
218
evals/opencode/openagent/README.md
Normal file
218
evals/opencode/openagent/README.md
Normal file
|
|
@ -0,0 +1,218 @@
|
||||||
|
# OpenAgent Evaluation Tests
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Test suite for validating OpenAgent behavior against defined standards and critical rules.
|
||||||
|
|
||||||
|
## OpenAgent Rules
|
||||||
|
|
||||||
|
OpenAgent must follow these critical rules:
|
||||||
|
|
||||||
|
1. **Approval Gates** - Request approval before bash/write/edit/task operations
|
||||||
|
2. **Context Loading** - Load required context files before execution
|
||||||
|
3. **Delegation** - Delegate when 4+ files or complex tasks
|
||||||
|
4. **Stop on Failure** - Never auto-fix errors, always report first
|
||||||
|
5. **Tool Selection** - Use appropriate tools for tasks
|
||||||
|
|
||||||
|
See: [OpenAgent Specification](../../../.opencode/agent/openagent.md)
|
||||||
|
|
||||||
|
## Test Categories
|
||||||
|
|
||||||
|
### Conversational
|
||||||
|
Simple questions that require no execution:
|
||||||
|
- Code explanations
|
||||||
|
- Informational queries
|
||||||
|
- Analysis requests
|
||||||
|
|
||||||
|
**Expected Behavior:**
|
||||||
|
- No execution tools
|
||||||
|
- No approval required
|
||||||
|
- Response provided
|
||||||
|
|
||||||
|
### Task
|
||||||
|
Simple file operations:
|
||||||
|
- File creation
|
||||||
|
- File editing
|
||||||
|
- Simple changes
|
||||||
|
|
||||||
|
**Expected Behavior:**
|
||||||
|
- Approval requested
|
||||||
|
- Appropriate tool selection
|
||||||
|
- Context loading (if needed)
|
||||||
|
|
||||||
|
### Complex
|
||||||
|
Multi-file features and complex tasks:
|
||||||
|
- Architecture changes
|
||||||
|
- Feature implementation
|
||||||
|
- Complex refactoring
|
||||||
|
|
||||||
|
**Expected Behavior:**
|
||||||
|
- Approval requested
|
||||||
|
- Context loaded
|
||||||
|
- Delegation used (4+ files)
|
||||||
|
- Appropriate tool selection
|
||||||
|
|
||||||
|
### Edge Cases
|
||||||
|
Error handling and special scenarios:
|
||||||
|
- Permission denials
|
||||||
|
- Missing files
|
||||||
|
- Error recovery
|
||||||
|
- Invalid inputs
|
||||||
|
|
||||||
|
**Expected Behavior:**
|
||||||
|
- Stop on failure
|
||||||
|
- Report errors (no auto-fix)
|
||||||
|
- Graceful handling
|
||||||
|
|
||||||
|
## Test Cases
|
||||||
|
|
||||||
|
Test cases are defined in YAML files in `test-cases/`:
|
||||||
|
|
||||||
|
- `approval-gates.yaml` - Approval gate enforcement tests
|
||||||
|
- `context-loading.yaml` - Context loading compliance tests
|
||||||
|
- `delegation.yaml` - Delegation appropriateness tests
|
||||||
|
- `tool-usage.yaml` - Tool selection tests
|
||||||
|
- `edge-cases.yaml` - Error handling and special scenarios
|
||||||
|
|
||||||
|
## Running Tests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run all OpenAgent tests
|
||||||
|
cd evals/framework
|
||||||
|
npm run eval -- --agent openagent --all
|
||||||
|
|
||||||
|
# Run specific test category
|
||||||
|
npm run eval -- --agent openagent --test approval-gates
|
||||||
|
|
||||||
|
# Run single test case
|
||||||
|
npm run eval -- --agent openagent --test approval-gates --case file-creation-with-approval
|
||||||
|
|
||||||
|
# Analyze specific session
|
||||||
|
npm run eval -- --agent openagent --session ses_xxxxx
|
||||||
|
```
|
||||||
|
|
||||||
|
## Test Results
|
||||||
|
|
||||||
|
Results are stored in `evals/results/YYYY-MM-DD/openagent/`:
|
||||||
|
|
||||||
|
```
|
||||||
|
results/2025-11-21/openagent/
|
||||||
|
├── summary.json # Overall summary
|
||||||
|
├── approval-gates.json # Approval gate results
|
||||||
|
├── context-loading.json # Context loading results
|
||||||
|
├── delegation.json # Delegation results
|
||||||
|
└── report.md # Human-readable report
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
Configuration is in `config.yaml`:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
agent: openagent
|
||||||
|
agent_path: ../../../.opencode/agent/openagent.md
|
||||||
|
test_cases_path: ./test-cases
|
||||||
|
sessions_path: ./sessions
|
||||||
|
evaluators:
|
||||||
|
- approval-gate
|
||||||
|
- context-loading
|
||||||
|
- delegation
|
||||||
|
- tool-usage
|
||||||
|
pass_threshold: 75
|
||||||
|
scoring:
|
||||||
|
approval_gate: 40
|
||||||
|
context_loading: 40
|
||||||
|
delegation: 10
|
||||||
|
tool_usage: 10
|
||||||
|
```
|
||||||
|
|
||||||
|
## Recorded Sessions
|
||||||
|
|
||||||
|
The `sessions/` directory contains recorded test sessions for regression testing:
|
||||||
|
|
||||||
|
```
|
||||||
|
sessions/
|
||||||
|
├── simple-question/
|
||||||
|
│ ├── session.json
|
||||||
|
│ └── expected.yaml
|
||||||
|
├── file-creation/
|
||||||
|
│ ├── session.json
|
||||||
|
│ └── expected.yaml
|
||||||
|
└── complex-feature/
|
||||||
|
├── session.json
|
||||||
|
└── expected.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
## Success Criteria
|
||||||
|
|
||||||
|
### Overall
|
||||||
|
- **Pass Rate:** ≥ 90% of tests pass
|
||||||
|
- **Average Score:** ≥ 85/100
|
||||||
|
- **Critical Violations:** 0
|
||||||
|
|
||||||
|
### Per Evaluator
|
||||||
|
- **Approval Gates:** 100% compliance (critical)
|
||||||
|
- **Context Loading:** ≥ 90% compliance
|
||||||
|
- **Delegation:** ≥ 80% compliance
|
||||||
|
- **Tool Usage:** ≥ 85% compliance
|
||||||
|
|
||||||
|
## Adding New Tests
|
||||||
|
|
||||||
|
1. Create test case in appropriate YAML file:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
- id: my-new-test
|
||||||
|
name: "My New Test"
|
||||||
|
description: "Test description"
|
||||||
|
category: task
|
||||||
|
input: "User prompt"
|
||||||
|
expected_behavior:
|
||||||
|
approval_requested: true
|
||||||
|
tool_used: write
|
||||||
|
evaluators:
|
||||||
|
- approval-gate
|
||||||
|
- tool-usage
|
||||||
|
pass_threshold: 75
|
||||||
|
```
|
||||||
|
|
||||||
|
2. (Optional) Record a session for regression testing:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run OpenCode with the test prompt
|
||||||
|
opencode --agent openagent
|
||||||
|
> "User prompt"
|
||||||
|
|
||||||
|
# Copy session to sessions/
|
||||||
|
cp ~/.local/share/opencode/project/.../session/info/ses_xxxxx.json \
|
||||||
|
sessions/my-new-test/session.json
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Run the test:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm run eval -- --agent openagent --test my-new-test
|
||||||
|
```
|
||||||
|
|
||||||
|
## Continuous Integration
|
||||||
|
|
||||||
|
Tests run automatically on:
|
||||||
|
- Pull requests
|
||||||
|
- Commits to main
|
||||||
|
- Nightly builds
|
||||||
|
|
||||||
|
See: `.github/workflows/eval-openagent.yml`
|
||||||
|
|
||||||
|
## Related Documentation
|
||||||
|
|
||||||
|
- [Evaluation Framework](../../framework/README.md)
|
||||||
|
- [OpenAgent Specification](../../../.opencode/agent/openagent.md)
|
||||||
|
- [OpenCode Logging System](../../../dev/ai-tools/opencode/logging-and-session-storage.md)
|
||||||
|
|
||||||
|
## Metrics
|
||||||
|
|
||||||
|
Track these metrics over time:
|
||||||
|
- Pass rate trend
|
||||||
|
- Average score trend
|
||||||
|
- Violation frequency
|
||||||
|
- Model performance
|
||||||
|
- Cost per test
|
||||||
87
evals/opencode/openagent/config/config.yaml
Normal file
87
evals/opencode/openagent/config/config.yaml
Normal file
|
|
@ -0,0 +1,87 @@
|
||||||
|
# OpenAgent Evaluation Configuration
|
||||||
|
|
||||||
|
agent: openagent
|
||||||
|
agent_path: ../../../../.opencode/agent/openagent.md
|
||||||
|
|
||||||
|
# Paths
|
||||||
|
test_cases_path: ../test-cases
|
||||||
|
sessions_path: ../sessions
|
||||||
|
results_path: ../../../results
|
||||||
|
|
||||||
|
# Evaluators to run
|
||||||
|
evaluators:
|
||||||
|
- approval-gate
|
||||||
|
- context-loading
|
||||||
|
- delegation
|
||||||
|
- tool-usage
|
||||||
|
|
||||||
|
# Pass threshold (0-100)
|
||||||
|
pass_threshold: 75
|
||||||
|
|
||||||
|
# Scoring weights (must sum to 100)
|
||||||
|
scoring:
|
||||||
|
approval_gate: 40 # Critical - approval before execution
|
||||||
|
context_loading: 40 # Critical - load context before tasks
|
||||||
|
delegation: 10 # Important - delegate appropriately
|
||||||
|
tool_usage: 10 # Important - use right tools
|
||||||
|
|
||||||
|
# Evaluation rules
|
||||||
|
rules:
|
||||||
|
approval_gate:
|
||||||
|
enabled: true
|
||||||
|
severity: error
|
||||||
|
keywords:
|
||||||
|
- approval
|
||||||
|
- approve
|
||||||
|
- proceed
|
||||||
|
- confirm
|
||||||
|
- permission
|
||||||
|
- before proceeding
|
||||||
|
|
||||||
|
context_loading:
|
||||||
|
enabled: true
|
||||||
|
severity: error
|
||||||
|
required_contexts:
|
||||||
|
code: standards/code.md
|
||||||
|
docs: standards/docs.md
|
||||||
|
tests: standards/tests.md
|
||||||
|
review: workflows/review.md
|
||||||
|
delegation: workflows/delegation.md
|
||||||
|
|
||||||
|
delegation:
|
||||||
|
enabled: true
|
||||||
|
severity: warning
|
||||||
|
file_threshold: 4
|
||||||
|
complexity_triggers:
|
||||||
|
- multi-step
|
||||||
|
- architecture
|
||||||
|
- refactoring
|
||||||
|
|
||||||
|
tool_usage:
|
||||||
|
enabled: true
|
||||||
|
severity: warning
|
||||||
|
appropriate_tools:
|
||||||
|
read_file: [read]
|
||||||
|
write_file: [write]
|
||||||
|
edit_file: [edit]
|
||||||
|
run_command: [bash]
|
||||||
|
delegate: [task]
|
||||||
|
list_files: [list]
|
||||||
|
find_files: [glob]
|
||||||
|
search_content: [grep]
|
||||||
|
|
||||||
|
# Model preferences (for live testing)
|
||||||
|
models:
|
||||||
|
primary: claude-sonnet-4-20250514
|
||||||
|
fallback: gemini-2.5-flash
|
||||||
|
cost_limit: 1.00 # Max cost per test run
|
||||||
|
|
||||||
|
# Reporting
|
||||||
|
reporting:
|
||||||
|
formats:
|
||||||
|
- console
|
||||||
|
- json
|
||||||
|
- markdown
|
||||||
|
detail_level: detailed # minimal, summary, detailed
|
||||||
|
include_evidence: true
|
||||||
|
include_timeline: true
|
||||||
1
evals/results/.gitkeep
Normal file
1
evals/results/.gitkeep
Normal file
|
|
@ -0,0 +1 @@
|
||||||
|
# Evaluation Results (gitignored)
|
||||||
Loading…
Reference in a new issue