Add evaluation framework structure and OpenCode logging documentation
- Add comprehensive evaluation framework in evals/ - Reusable framework for testing OpenCode agents - OpenAgent-specific test configuration - Scalable structure for multiple agents - Add OpenCode logging system documentation - Complete schema documentation for session storage - Model and provider tracking details - Performance analysis examples - Cost optimization strategies - Update .gitignore - Ignore evals/results/ directory - Ignore tasks/ directory (temporary) Structure: - evals/framework/ - Reusable evaluation framework - evals/opencode/openagent/ - OpenAgent tests - dev/ai-tools/opencode/logging-and-session-storage.md - Logging docs
This commit is contained in:
darrenhinde
parent
c6775bbc40
commit
f4b3d56aa2
11 changed files with 3038 additions and 1 deletions
7
.gitignore
vendored
7
.gitignore
vendored
|
|
@ -197,3 +197,10 @@ temp/
|
|||
|
||||
# Bun
|
||||
.bun
|
||||
|
||||
# Evaluation results
|
||||
evals/results/*
|
||||
!evals/results/.gitkeep
|
||||
|
||||
# Task tracking (temporary)
|
||||
tasks/
|
||||
1692
dev/ai-tools/opencode/logging-and-session-storage.md
Normal file
1692
dev/ai-tools/opencode/logging-and-session-storage.md
Normal file
File diff suppressed because it is too large
Load diff
279
evals/README.md
Normal file
279
evals/README.md
Normal file
|
|
@ -0,0 +1,279 @@
|
|||
# OpenCode Agent Evaluation Framework
|
||||
|
||||
## Overview
|
||||
|
||||
Comprehensive evaluation framework for testing and validating OpenCode agent behavior against defined standards and rules.
|
||||
|
||||
## Structure
|
||||
|
||||
```
|
||||
evals/
|
||||
├── framework/ # Reusable evaluation framework
|
||||
│ ├── src/
|
||||
│ │ ├── collector/ # Session data collection
|
||||
│ │ ├── evaluators/ # Evaluation logic
|
||||
│ │ ├── runner/ # Test execution
|
||||
│ │ ├── reporters/ # Result reporting
|
||||
│ │ └── types/ # TypeScript types
|
||||
│ └── package.json
|
||||
│
|
||||
├── opencode/ # OpenCode agent evaluations
|
||||
│ ├── openagent/ # OpenAgent-specific tests
|
||||
│ ├── opencoder/ # OpenCoder-specific tests
|
||||
│ └── shared/ # Shared test cases
|
||||
│
|
||||
└── results/ # Test results (gitignored)
|
||||
└── YYYY-MM-DD/
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Install Framework
|
||||
|
||||
```bash
|
||||
cd evals/framework
|
||||
npm install
|
||||
npm run build
|
||||
```
|
||||
|
||||
### 2. Run Evaluations
|
||||
|
||||
```bash
|
||||
# Evaluate a specific session
|
||||
npm run eval -- --agent openagent --session ses_xxxxx
|
||||
|
||||
# Run all OpenAgent tests
|
||||
npm run eval -- --agent openagent --all
|
||||
|
||||
# Run specific test case
|
||||
npm run eval -- --agent openagent --test approval-gates
|
||||
```
|
||||
|
||||
### 3. View Results
|
||||
|
||||
```bash
|
||||
# View latest results
|
||||
cat evals/results/$(ls -t evals/results | head -1)/openagent/summary.json | jq
|
||||
|
||||
# Generate report
|
||||
npm run report -- --agent openagent --date 2025-11-21
|
||||
```
|
||||
|
||||
## Framework Components
|
||||
|
||||
### Collector
|
||||
Reads and parses OpenCode session data from `~/.local/share/opencode/`
|
||||
|
||||
**Components:**
|
||||
- `SessionReader` - Read session files
|
||||
- `MessageParser` - Parse message structure
|
||||
- `TimelineBuilder` - Build event timeline
|
||||
|
||||
### Evaluators
|
||||
Validate agent behavior against rules
|
||||
|
||||
**Available Evaluators:**
|
||||
- `ApprovalGateEvaluator` - Check approval before execution
|
||||
- `ContextLoadingEvaluator` - Verify context loading
|
||||
- `DelegationEvaluator` - Validate delegation decisions
|
||||
- `ToolUsageEvaluator` - Check tool selection
|
||||
- `ModelSelectionEvaluator` - Validate model choices
|
||||
|
||||
### Runner
|
||||
Execute test cases and collect results
|
||||
|
||||
**Components:**
|
||||
- `TestRunner` - Run test suites
|
||||
- `SessionAnalyzer` - Analyze historical sessions
|
||||
- `LiveRunner` - Run live tests (future)
|
||||
|
||||
### Reporters
|
||||
Generate test reports
|
||||
|
||||
**Formats:**
|
||||
- Console (pretty output)
|
||||
- JSON (machine-readable)
|
||||
- Markdown (documentation)
|
||||
- HTML (dashboard, future)
|
||||
|
||||
## Agent-Specific Tests
|
||||
|
||||
### OpenAgent (`opencode/openagent/`)
|
||||
|
||||
Tests for the universal OpenAgent:
|
||||
- Approval gate enforcement
|
||||
- Context loading compliance
|
||||
- Delegation appropriateness
|
||||
- Tool usage patterns
|
||||
- Model selection
|
||||
|
||||
**Test Categories:**
|
||||
- Conversational (simple questions)
|
||||
- Task (file operations)
|
||||
- Complex (multi-file features)
|
||||
- Edge cases (errors, permissions)
|
||||
|
||||
### OpenCoder (`opencode/opencoder/`)
|
||||
|
||||
Tests for the OpenCoder agent (future)
|
||||
|
||||
### Shared (`opencode/shared/`)
|
||||
|
||||
Common test cases applicable to all agents
|
||||
|
||||
## Test Case Format
|
||||
|
||||
Test cases are defined in YAML:
|
||||
|
||||
```yaml
|
||||
test_cases:
|
||||
- id: simple-question
|
||||
name: "Simple Question (No Execution)"
|
||||
description: "Ask a question that requires no execution tools"
|
||||
category: conversational
|
||||
input: "What does this code do?"
|
||||
expected_behavior:
|
||||
no_execution_tools: true
|
||||
no_approval_required: true
|
||||
response_provided: true
|
||||
evaluators:
|
||||
- approval-gate
|
||||
- tool-usage
|
||||
pass_threshold: 100
|
||||
```
|
||||
|
||||
## Evaluation Results
|
||||
|
||||
Results include:
|
||||
- **Pass/Fail** - Did the test pass?
|
||||
- **Score** - 0-100 compliance score
|
||||
- **Violations** - List of rule violations
|
||||
- **Evidence** - Supporting data for violations
|
||||
- **Metadata** - Agent, model, timing, costs
|
||||
|
||||
**Example Result:**
|
||||
```json
|
||||
{
|
||||
"testCaseId": "simple-question",
|
||||
"sessionId": "ses_xxxxx",
|
||||
"passed": true,
|
||||
"score": 100,
|
||||
"evaluationResults": [
|
||||
{
|
||||
"evaluator": "approval-gate",
|
||||
"passed": true,
|
||||
"score": 100,
|
||||
"violations": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"agent": "openagent",
|
||||
"model": "claude-sonnet-4-20250514",
|
||||
"duration": 7205,
|
||||
"cost": 0.107
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Scoring System
|
||||
|
||||
- **100** - Perfect compliance
|
||||
- **75-99** - Minor issues (warnings)
|
||||
- **50-74** - Moderate issues (violations)
|
||||
- **0-49** - Major issues (critical violations)
|
||||
|
||||
**Pass Threshold:** 75% (configurable per test)
|
||||
|
||||
## Development
|
||||
|
||||
### Adding New Evaluators
|
||||
|
||||
1. Create evaluator in `framework/src/evaluators/`
|
||||
2. Extend `BaseEvaluator`
|
||||
3. Implement `evaluate()` method
|
||||
4. Add tests
|
||||
5. Register in evaluator registry
|
||||
|
||||
### Adding New Test Cases
|
||||
|
||||
1. Create YAML file in `opencode/{agent}/test-cases/`
|
||||
2. Define test case structure
|
||||
3. Specify expected behavior
|
||||
4. List evaluators to run
|
||||
5. Set pass threshold
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Framework tests
|
||||
cd evals/framework
|
||||
npm test
|
||||
|
||||
# Integration tests
|
||||
npm run test:integration
|
||||
|
||||
# Test specific evaluator
|
||||
npm run test:evaluator -- approval-gate
|
||||
```
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
```yaml
|
||||
# .github/workflows/eval.yml
|
||||
name: Agent Evaluation
|
||||
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
evaluate:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- uses: actions/setup-node@v3
|
||||
- run: cd evals/framework && npm install
|
||||
- run: npm run eval -- --agent openagent --all
|
||||
- run: npm run report -- --format json
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Framework Config (`framework/config.ts`)
|
||||
|
||||
```typescript
|
||||
export const config = {
|
||||
projectPath: process.cwd(),
|
||||
sessionStoragePath: '~/.local/share/opencode/',
|
||||
resultsPath: 'evals/results/',
|
||||
passThreshold: 75,
|
||||
evaluators: ['approval-gate', 'context-loading', 'delegation', 'tool-usage']
|
||||
};
|
||||
```
|
||||
|
||||
### Agent Config (`opencode/openagent/config.yaml`)
|
||||
|
||||
```yaml
|
||||
agent: openagent
|
||||
test_cases_path: ./test-cases
|
||||
sessions_path: ./sessions
|
||||
evaluators:
|
||||
- approval-gate
|
||||
- context-loading
|
||||
- delegation
|
||||
- tool-usage
|
||||
pass_threshold: 75
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Framework Architecture](./framework/README.md)
|
||||
- [OpenAgent Tests](./opencode/openagent/README.md)
|
||||
- [OpenCode Logging System](../dev/ai-tools/opencode/logging-and-session-storage.md)
|
||||
- [Agent Validator Plugin](../.opencode/plugin/docs/VALIDATOR_GUIDE.md)
|
||||
|
||||
## Contributing
|
||||
|
||||
See [CONTRIBUTING.md](../docs/contributing/CONTRIBUTING.md)
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
348
evals/SETUP_COMPLETE.md
Normal file
348
evals/SETUP_COMPLETE.md
Normal file
|
|
@ -0,0 +1,348 @@
|
|||
# Evaluation Framework Setup Complete ✅
|
||||
|
||||
## What Was Created
|
||||
|
||||
### Directory Structure
|
||||
|
||||
```
|
||||
evals/
|
||||
├── README.md # Main overview
|
||||
├── framework/ # Reusable evaluation framework
|
||||
│ ├── README.md # Framework documentation
|
||||
│ ├── package.json # Dependencies
|
||||
│ ├── tsconfig.json # TypeScript config
|
||||
│ ├── .gitignore # Git ignore rules
|
||||
│ └── src/ # Source code (empty, ready for implementation)
|
||||
│ ├── collector/ # Session data collection
|
||||
│ ├── evaluators/ # Evaluation logic
|
||||
│ ├── runner/ # Test execution
|
||||
│ ├── reporters/ # Result reporting
|
||||
│ └── types/ # TypeScript types
|
||||
│
|
||||
├── opencode/ # OpenCode agent evaluations
|
||||
│ ├── openagent/ # OpenAgent-specific tests
|
||||
│ │ ├── README.md # OpenAgent test documentation
|
||||
│ │ ├── config/
|
||||
│ │ │ └── config.yaml # OpenAgent configuration
|
||||
│ │ ├── test-cases/ # Test case definitions (empty)
|
||||
│ │ └── sessions/ # Recorded test sessions (empty)
|
||||
│ └── shared/ # Shared test cases (empty)
|
||||
│
|
||||
└── results/ # Test results (gitignored)
|
||||
└── .gitkeep
|
||||
```
|
||||
|
||||
### Task Documentation
|
||||
|
||||
```
|
||||
tasks/eval-framework/
|
||||
├── README.md # Project overview
|
||||
├── 01-session-reader.md # Task 1: Session Reader
|
||||
├── 02-evaluators.md # Task 2: Core Evaluators
|
||||
└── 03-test-runner.md # Task 3: Test Runner
|
||||
```
|
||||
|
||||
### Configuration Files
|
||||
|
||||
1. **Framework Package** (`evals/framework/package.json`)
|
||||
- TypeScript setup
|
||||
- Testing with Vitest
|
||||
- Build scripts
|
||||
- Linting
|
||||
|
||||
2. **OpenAgent Config** (`evals/opencode/openagent/config/config.yaml`)
|
||||
- Evaluator settings
|
||||
- Scoring weights
|
||||
- Rule definitions
|
||||
- Model preferences
|
||||
|
||||
3. **Git Ignore** (`.gitignore`)
|
||||
- Results directory ignored
|
||||
- Node modules ignored
|
||||
- Build artifacts ignored
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
### Phase 1: Foundation (Start Here)
|
||||
|
||||
**Goal:** Build session reader and basic types
|
||||
|
||||
**Tasks:**
|
||||
1. Implement TypeScript types (`framework/src/types/index.ts`)
|
||||
2. Implement SessionReader (`framework/src/collector/session-reader.ts`)
|
||||
3. Implement MessageParser (`framework/src/collector/message-parser.ts`)
|
||||
4. Implement TimelineBuilder (`framework/src/collector/timeline-builder.ts`)
|
||||
5. Write tests
|
||||
|
||||
**See:** [tasks/eval-framework/01-session-reader.md](../tasks/eval-framework/01-session-reader.md)
|
||||
|
||||
**Estimated Time:** 4-6 hours
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: Core Evaluators
|
||||
|
||||
**Goal:** Implement evaluation logic
|
||||
|
||||
**Tasks:**
|
||||
1. Implement BaseEvaluator
|
||||
2. Implement ApprovalGateEvaluator
|
||||
3. Implement ContextLoadingEvaluator
|
||||
4. Implement DelegationEvaluator
|
||||
5. Implement ToolUsageEvaluator
|
||||
6. Write tests
|
||||
|
||||
**See:** [tasks/eval-framework/02-evaluators.md](../tasks/eval-framework/02-evaluators.md)
|
||||
|
||||
**Estimated Time:** 8-10 hours
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: Test Runner
|
||||
|
||||
**Goal:** Execute test cases and generate results
|
||||
|
||||
**Tasks:**
|
||||
1. Implement test case loader
|
||||
2. Implement SessionAnalyzer
|
||||
3. Implement TestRunner
|
||||
4. Create test case library
|
||||
5. Write tests
|
||||
|
||||
**See:** [tasks/eval-framework/03-test-runner.md](../tasks/eval-framework/03-test-runner.md)
|
||||
|
||||
**Estimated Time:** 6-8 hours
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: Reporters (Future)
|
||||
|
||||
**Goal:** Generate useful reports
|
||||
|
||||
**Tasks:**
|
||||
1. Implement ConsoleReporter
|
||||
2. Implement JSONReporter
|
||||
3. Implement MarkdownReporter
|
||||
4. Add trend analysis
|
||||
|
||||
**Estimated Time:** 4-6 hours
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Install Dependencies
|
||||
|
||||
```bash
|
||||
cd evals/framework
|
||||
npm install
|
||||
```
|
||||
|
||||
### 2. Start Building
|
||||
|
||||
Begin with Phase 1 - Session Reader:
|
||||
|
||||
```bash
|
||||
# Create types file
|
||||
touch src/types/index.ts
|
||||
|
||||
# Create session reader
|
||||
touch src/collector/session-reader.ts
|
||||
|
||||
# Create message parser
|
||||
touch src/collector/message-parser.ts
|
||||
|
||||
# Create timeline builder
|
||||
touch src/collector/timeline-builder.ts
|
||||
```
|
||||
|
||||
### 3. Run Tests
|
||||
|
||||
```bash
|
||||
# Run tests
|
||||
npm test
|
||||
|
||||
# Watch mode
|
||||
npm run test:watch
|
||||
```
|
||||
|
||||
### 4. Build
|
||||
|
||||
```bash
|
||||
# Build TypeScript
|
||||
npm run build
|
||||
|
||||
# Watch mode
|
||||
npm run build:watch
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
OpenCode Session Storage
|
||||
↓
|
||||
SessionReader (reads JSON files)
|
||||
↓
|
||||
MessageParser (parses structure)
|
||||
↓
|
||||
TimelineBuilder (builds timeline)
|
||||
↓
|
||||
Evaluators (validate behavior)
|
||||
↓
|
||||
TestRunner (executes tests)
|
||||
↓
|
||||
Reporters (generate reports)
|
||||
↓
|
||||
Results (JSON/Markdown/Console)
|
||||
```
|
||||
|
||||
### Key Components
|
||||
|
||||
**Collector:**
|
||||
- Reads OpenCode session data
|
||||
- Parses messages and parts
|
||||
- Builds chronological timeline
|
||||
|
||||
**Evaluators:**
|
||||
- Validate agent behavior
|
||||
- Check compliance with rules
|
||||
- Generate scores and violations
|
||||
|
||||
**Runner:**
|
||||
- Load test case definitions
|
||||
- Execute evaluators
|
||||
- Collect results
|
||||
|
||||
**Reporters:**
|
||||
- Format results
|
||||
- Generate reports
|
||||
- Export data
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### Framework Config
|
||||
|
||||
Edit `framework/src/config.ts` (to be created):
|
||||
|
||||
```typescript
|
||||
export const config = {
|
||||
projectPath: process.cwd(),
|
||||
sessionStoragePath: '~/.local/share/opencode/',
|
||||
resultsPath: '../results/',
|
||||
passThreshold: 75,
|
||||
};
|
||||
```
|
||||
|
||||
### OpenAgent Config
|
||||
|
||||
Edit `opencode/openagent/config/config.yaml`:
|
||||
|
||||
```yaml
|
||||
agent: openagent
|
||||
pass_threshold: 75
|
||||
scoring:
|
||||
approval_gate: 40
|
||||
context_loading: 40
|
||||
delegation: 10
|
||||
tool_usage: 10
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
Test individual components in isolation:
|
||||
- SessionReader
|
||||
- MessageParser
|
||||
- TimelineBuilder
|
||||
- Each evaluator
|
||||
|
||||
### Integration Tests
|
||||
Test components working together:
|
||||
- Read real session data
|
||||
- Build timeline
|
||||
- Run evaluators
|
||||
- Generate results
|
||||
|
||||
### Regression Tests
|
||||
Test against recorded sessions:
|
||||
- Store known-good sessions
|
||||
- Run tests against them
|
||||
- Detect behavior changes
|
||||
|
||||
---
|
||||
|
||||
## Documentation
|
||||
|
||||
### Main Docs
|
||||
- [Evaluation Framework Overview](./README.md)
|
||||
- [Framework Documentation](./framework/README.md)
|
||||
- [OpenAgent Tests](./opencode/openagent/README.md)
|
||||
|
||||
### Task Breakdown
|
||||
- [Project Overview](../tasks/eval-framework/README.md)
|
||||
- [Task 1: Session Reader](../tasks/eval-framework/01-session-reader.md)
|
||||
- [Task 2: Core Evaluators](../tasks/eval-framework/02-evaluators.md)
|
||||
- [Task 3: Test Runner](../tasks/eval-framework/03-test-runner.md)
|
||||
|
||||
### Related Docs
|
||||
- [OpenCode Logging System](../dev/ai-tools/opencode/logging-and-session-storage.md)
|
||||
- [Agent Validator Plugin](../.opencode/plugin/docs/VALIDATOR_GUIDE.md)
|
||||
- [OpenAgent Specification](../.opencode/agent/openagent.md)
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
### Phase 1 Complete When:
|
||||
- [ ] Can read any OpenCode session
|
||||
- [ ] Can parse messages and parts
|
||||
- [ ] Can build complete timeline
|
||||
- [ ] All types defined
|
||||
- [ ] Tests pass
|
||||
|
||||
### Phase 2 Complete When:
|
||||
- [ ] All evaluators implemented
|
||||
- [ ] Can detect violations
|
||||
- [ ] Scoring system works
|
||||
- [ ] Tests pass
|
||||
|
||||
### Phase 3 Complete When:
|
||||
- [ ] Can load test cases from YAML
|
||||
- [ ] Can run test suites
|
||||
- [ ] Can generate results
|
||||
- [ ] Tests pass
|
||||
|
||||
### Phase 4 Complete When:
|
||||
- [ ] Can generate console reports
|
||||
- [ ] Can export JSON
|
||||
- [ ] Can generate Markdown
|
||||
- [ ] Tests pass
|
||||
|
||||
---
|
||||
|
||||
## Getting Help
|
||||
|
||||
- Review task documentation in `tasks/eval-framework/`
|
||||
- Check OpenCode logging docs in `dev/ai-tools/opencode/`
|
||||
- Examine existing session data in `~/.local/share/opencode/`
|
||||
- Look at validator plugin in `.opencode/plugin/agent-validator.ts`
|
||||
|
||||
---
|
||||
|
||||
## Ready to Start!
|
||||
|
||||
Begin with **Phase 1: Session Reader**
|
||||
|
||||
See: [tasks/eval-framework/01-session-reader.md](../tasks/eval-framework/01-session-reader.md)
|
||||
|
||||
Good luck! 🚀
|
||||
6
evals/framework/.gitignore
vendored
Normal file
6
evals/framework/.gitignore
vendored
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
node_modules/
|
||||
dist/
|
||||
*.log
|
||||
.DS_Store
|
||||
coverage/
|
||||
.env
|
||||
341
evals/framework/README.md
Normal file
341
evals/framework/README.md
Normal file
|
|
@ -0,0 +1,341 @@
|
|||
# Evaluation Framework
|
||||
|
||||
## Overview
|
||||
|
||||
Reusable framework for evaluating OpenCode agent behavior. Can be used to test any agent against defined standards.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
framework/
|
||||
├── src/
|
||||
│ ├── collector/ # Session data collection
|
||||
│ │ ├── session-reader.ts
|
||||
│ │ ├── message-parser.ts
|
||||
│ │ └── timeline-builder.ts
|
||||
│ ├── evaluators/ # Evaluation logic
|
||||
│ │ ├── base-evaluator.ts
|
||||
│ │ ├── approval-gate.ts
|
||||
│ │ ├── context-loading.ts
|
||||
│ │ ├── delegation.ts
|
||||
│ │ ├── tool-usage.ts
|
||||
│ │ └── model-selection.ts
|
||||
│ ├── runner/ # Test execution
|
||||
│ │ ├── test-runner.ts
|
||||
│ │ └── session-analyzer.ts
|
||||
│ ├── reporters/ # Result reporting
|
||||
│ │ ├── console-reporter.ts
|
||||
│ │ ├── json-reporter.ts
|
||||
│ │ └── markdown-reporter.ts
|
||||
│ ├── types/ # TypeScript types
|
||||
│ │ └── index.ts
|
||||
│ ├── config.ts # Configuration
|
||||
│ └── index.ts # Main exports
|
||||
├── tests/ # Framework tests
|
||||
├── package.json
|
||||
├── tsconfig.json
|
||||
└── README.md
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
npm install
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Example
|
||||
|
||||
```typescript
|
||||
import { SessionReader, ApprovalGateEvaluator, TimelineBuilder } from '@evals/framework';
|
||||
|
||||
// Read session
|
||||
const reader = new SessionReader('/path/to/project');
|
||||
const sessionInfo = reader.getSessionInfo('ses_xxxxx');
|
||||
const messages = reader.getMessages('ses_xxxxx');
|
||||
|
||||
// Build timeline
|
||||
const builder = new TimelineBuilder(reader);
|
||||
const timeline = builder.buildTimeline('ses_xxxxx');
|
||||
|
||||
// Evaluate
|
||||
const evaluator = new ApprovalGateEvaluator();
|
||||
const result = evaluator.evaluate(timeline);
|
||||
|
||||
console.log(`Passed: ${result.passed}`);
|
||||
console.log(`Score: ${result.score}/100`);
|
||||
console.log(`Violations: ${result.violations.length}`);
|
||||
```
|
||||
|
||||
### Running Test Suite
|
||||
|
||||
```typescript
|
||||
import { TestRunner } from '@evals/framework';
|
||||
|
||||
const runner = new TestRunner({
|
||||
projectPath: process.cwd(),
|
||||
evaluatorsPath: './src/evaluators',
|
||||
resultsPath: '../results'
|
||||
});
|
||||
|
||||
// Load test cases
|
||||
const testCases = runner.loadTestCases('../opencode/openagent/test-cases/approval-gates.yaml');
|
||||
|
||||
// Run tests
|
||||
const suite = runner.runAll(testCases);
|
||||
|
||||
console.log(`Pass Rate: ${suite.summary.passRate}%`);
|
||||
```
|
||||
|
||||
## Components
|
||||
|
||||
### Collector
|
||||
|
||||
**SessionReader** - Read OpenCode session files
|
||||
```typescript
|
||||
const reader = new SessionReader(projectPath);
|
||||
const info = reader.getSessionInfo(sessionId);
|
||||
const messages = reader.getMessages(sessionId);
|
||||
const parts = reader.getParts(sessionId, messageId);
|
||||
```
|
||||
|
||||
**MessageParser** - Parse message structure
|
||||
```typescript
|
||||
const parser = new MessageParser();
|
||||
const agent = parser.getAgent(message);
|
||||
const model = parser.getModel(message);
|
||||
const metrics = parser.getMetrics(message);
|
||||
```
|
||||
|
||||
**TimelineBuilder** - Build event timeline
|
||||
```typescript
|
||||
const builder = new TimelineBuilder(reader);
|
||||
const timeline = builder.buildTimeline(sessionId);
|
||||
const toolCalls = builder.filterByType(timeline, 'tool_call');
|
||||
```
|
||||
|
||||
### Evaluators
|
||||
|
||||
All evaluators extend `BaseEvaluator` and implement:
|
||||
```typescript
|
||||
evaluate(timeline: TimelineEvent[]): EvaluationResult
|
||||
```
|
||||
|
||||
**ApprovalGateEvaluator** - Check approval before execution
|
||||
```typescript
|
||||
const evaluator = new ApprovalGateEvaluator();
|
||||
const result = evaluator.evaluate(timeline);
|
||||
```
|
||||
|
||||
**ContextLoadingEvaluator** - Verify context loading
|
||||
```typescript
|
||||
const evaluator = new ContextLoadingEvaluator();
|
||||
const result = evaluator.evaluate(timeline);
|
||||
```
|
||||
|
||||
**DelegationEvaluator** - Validate delegation decisions
|
||||
```typescript
|
||||
const evaluator = new DelegationEvaluator();
|
||||
const result = evaluator.evaluate(timeline);
|
||||
```
|
||||
|
||||
**ToolUsageEvaluator** - Check tool selection
|
||||
```typescript
|
||||
const evaluator = new ToolUsageEvaluator();
|
||||
const result = evaluator.evaluate(timeline);
|
||||
```
|
||||
|
||||
### Runner
|
||||
|
||||
**TestRunner** - Execute test suites
|
||||
```typescript
|
||||
const runner = new TestRunner(config);
|
||||
const testCases = runner.loadTestCases(path);
|
||||
const suite = runner.runAll(testCases);
|
||||
```
|
||||
|
||||
**SessionAnalyzer** - Analyze historical sessions
|
||||
```typescript
|
||||
const analyzer = new SessionAnalyzer(reader);
|
||||
const result = analyzer.analyze(sessionId, testCase);
|
||||
```
|
||||
|
||||
### Reporters
|
||||
|
||||
**ConsoleReporter** - Pretty console output
|
||||
```typescript
|
||||
const reporter = new ConsoleReporter();
|
||||
reporter.report(testSuite);
|
||||
```
|
||||
|
||||
**JSONReporter** - Machine-readable JSON
|
||||
```typescript
|
||||
const reporter = new JSONReporter();
|
||||
reporter.export(testSuite, 'results.json');
|
||||
```
|
||||
|
||||
**MarkdownReporter** - Documentation format
|
||||
```typescript
|
||||
const reporter = new MarkdownReporter();
|
||||
reporter.generate(testSuite, 'report.md');
|
||||
```
|
||||
|
||||
## Types
|
||||
|
||||
### Core Types
|
||||
|
||||
```typescript
|
||||
interface SessionInfo {
|
||||
id: string;
|
||||
version: string;
|
||||
title: string;
|
||||
time: { created: number; updated: number };
|
||||
}
|
||||
|
||||
interface Message {
|
||||
id: string;
|
||||
role: 'user' | 'assistant';
|
||||
sessionID: string;
|
||||
mode?: string;
|
||||
modelID?: string;
|
||||
providerID?: string;
|
||||
tokens?: TokenUsage;
|
||||
cost?: number;
|
||||
time: { created: number; completed?: number };
|
||||
}
|
||||
|
||||
interface TimelineEvent {
|
||||
timestamp: number;
|
||||
type: 'user_message' | 'assistant_message' | 'tool_call' | 'patch';
|
||||
agent?: string;
|
||||
model?: string;
|
||||
data: any;
|
||||
}
|
||||
|
||||
interface EvaluationResult {
|
||||
evaluator: string;
|
||||
passed: boolean;
|
||||
score: number;
|
||||
violations: Violation[];
|
||||
evidence: Evidence[];
|
||||
}
|
||||
|
||||
interface TestResult {
|
||||
testCaseId: string;
|
||||
sessionId: string;
|
||||
passed: boolean;
|
||||
score: number;
|
||||
evaluationResults: EvaluationResult[];
|
||||
metadata: any;
|
||||
}
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
```typescript
|
||||
// config.ts
|
||||
export const config = {
|
||||
projectPath: process.cwd(),
|
||||
sessionStoragePath: '~/.local/share/opencode/',
|
||||
resultsPath: '../results/',
|
||||
passThreshold: 75,
|
||||
evaluators: {
|
||||
'approval-gate': ApprovalGateEvaluator,
|
||||
'context-loading': ContextLoadingEvaluator,
|
||||
'delegation': DelegationEvaluator,
|
||||
'tool-usage': ToolUsageEvaluator,
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
### Setup
|
||||
|
||||
```bash
|
||||
npm install
|
||||
npm run build
|
||||
```
|
||||
|
||||
### Testing
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
npm test
|
||||
|
||||
# Run specific test
|
||||
npm test -- session-reader
|
||||
|
||||
# Watch mode
|
||||
npm run test:watch
|
||||
```
|
||||
|
||||
### Building
|
||||
|
||||
```bash
|
||||
# Build TypeScript
|
||||
npm run build
|
||||
|
||||
# Watch mode
|
||||
npm run build:watch
|
||||
```
|
||||
|
||||
### Linting
|
||||
|
||||
```bash
|
||||
npm run lint
|
||||
npm run lint:fix
|
||||
```
|
||||
|
||||
## Adding New Evaluators
|
||||
|
||||
1. Create file in `src/evaluators/`
|
||||
2. Extend `BaseEvaluator`
|
||||
3. Implement `evaluate()` method
|
||||
4. Add tests
|
||||
5. Register in config
|
||||
6. Export from `index.ts`
|
||||
|
||||
**Example:**
|
||||
```typescript
|
||||
// src/evaluators/my-evaluator.ts
|
||||
import { BaseEvaluator } from './base-evaluator';
|
||||
import { EvaluationResult, TimelineEvent } from '../types';
|
||||
|
||||
export class MyEvaluator extends BaseEvaluator {
|
||||
evaluate(timeline: TimelineEvent[]): EvaluationResult {
|
||||
// Your evaluation logic
|
||||
const checks = [
|
||||
{ name: 'check1', passed: true, weight: 50 },
|
||||
{ name: 'check2', passed: false, weight: 50 }
|
||||
];
|
||||
|
||||
const score = this.calculateScore(checks);
|
||||
const violations = this.findViolations(timeline);
|
||||
|
||||
return {
|
||||
evaluator: 'my-evaluator',
|
||||
passed: score >= 75,
|
||||
score,
|
||||
violations,
|
||||
evidence: []
|
||||
};
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
See [API.md](./API.md) for complete API documentation.
|
||||
|
||||
## Contributing
|
||||
|
||||
1. Fork the repository
|
||||
2. Create feature branch
|
||||
3. Add tests
|
||||
4. Submit pull request
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
38
evals/framework/package.json
Normal file
38
evals/framework/package.json
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
{
|
||||
"name": "@opencode-agents/eval-framework",
|
||||
"version": "0.1.0",
|
||||
"description": "Evaluation framework for OpenCode agents",
|
||||
"main": "dist/index.js",
|
||||
"types": "dist/index.d.ts",
|
||||
"scripts": {
|
||||
"build": "tsc",
|
||||
"build:watch": "tsc --watch",
|
||||
"test": "vitest",
|
||||
"test:watch": "vitest --watch",
|
||||
"test:integration": "vitest --config vitest.integration.config.ts",
|
||||
"lint": "eslint src/**/*.ts",
|
||||
"lint:fix": "eslint src/**/*.ts --fix",
|
||||
"eval": "node dist/cli.js",
|
||||
"report": "node dist/cli.js report"
|
||||
},
|
||||
"keywords": [
|
||||
"opencode",
|
||||
"evaluation",
|
||||
"testing",
|
||||
"agents",
|
||||
"ai"
|
||||
],
|
||||
"author": "",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"yaml": "^2.3.4"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^20.10.0",
|
||||
"@typescript-eslint/eslint-plugin": "^6.13.0",
|
||||
"@typescript-eslint/parser": "^6.13.0",
|
||||
"eslint": "^8.54.0",
|
||||
"typescript": "^5.3.0",
|
||||
"vitest": "^1.0.0"
|
||||
}
|
||||
}
|
||||
20
evals/framework/tsconfig.json
Normal file
20
evals/framework/tsconfig.json
Normal file
|
|
@ -0,0 +1,20 @@
|
|||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2020",
|
||||
"module": "commonjs",
|
||||
"lib": ["ES2020"],
|
||||
"outDir": "./dist",
|
||||
"rootDir": "./src",
|
||||
"declaration": true,
|
||||
"declarationMap": true,
|
||||
"sourceMap": true,
|
||||
"strict": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"resolveJsonModule": true,
|
||||
"moduleResolution": "node"
|
||||
},
|
||||
"include": ["src/**/*"],
|
||||
"exclude": ["node_modules", "dist", "tests"]
|
||||
}
|
||||
218
evals/opencode/openagent/README.md
Normal file
218
evals/opencode/openagent/README.md
Normal file
|
|
@ -0,0 +1,218 @@
|
|||
# OpenAgent Evaluation Tests
|
||||
|
||||
## Overview
|
||||
|
||||
Test suite for validating OpenAgent behavior against defined standards and critical rules.
|
||||
|
||||
## OpenAgent Rules
|
||||
|
||||
OpenAgent must follow these critical rules:
|
||||
|
||||
1. **Approval Gates** - Request approval before bash/write/edit/task operations
|
||||
2. **Context Loading** - Load required context files before execution
|
||||
3. **Delegation** - Delegate when 4+ files or complex tasks
|
||||
4. **Stop on Failure** - Never auto-fix errors, always report first
|
||||
5. **Tool Selection** - Use appropriate tools for tasks
|
||||
|
||||
See: [OpenAgent Specification](../../../.opencode/agent/openagent.md)
|
||||
|
||||
## Test Categories
|
||||
|
||||
### Conversational
|
||||
Simple questions that require no execution:
|
||||
- Code explanations
|
||||
- Informational queries
|
||||
- Analysis requests
|
||||
|
||||
**Expected Behavior:**
|
||||
- No execution tools
|
||||
- No approval required
|
||||
- Response provided
|
||||
|
||||
### Task
|
||||
Simple file operations:
|
||||
- File creation
|
||||
- File editing
|
||||
- Simple changes
|
||||
|
||||
**Expected Behavior:**
|
||||
- Approval requested
|
||||
- Appropriate tool selection
|
||||
- Context loading (if needed)
|
||||
|
||||
### Complex
|
||||
Multi-file features and complex tasks:
|
||||
- Architecture changes
|
||||
- Feature implementation
|
||||
- Complex refactoring
|
||||
|
||||
**Expected Behavior:**
|
||||
- Approval requested
|
||||
- Context loaded
|
||||
- Delegation used (4+ files)
|
||||
- Appropriate tool selection
|
||||
|
||||
### Edge Cases
|
||||
Error handling and special scenarios:
|
||||
- Permission denials
|
||||
- Missing files
|
||||
- Error recovery
|
||||
- Invalid inputs
|
||||
|
||||
**Expected Behavior:**
|
||||
- Stop on failure
|
||||
- Report errors (no auto-fix)
|
||||
- Graceful handling
|
||||
|
||||
## Test Cases
|
||||
|
||||
Test cases are defined in YAML files in `test-cases/`:
|
||||
|
||||
- `approval-gates.yaml` - Approval gate enforcement tests
|
||||
- `context-loading.yaml` - Context loading compliance tests
|
||||
- `delegation.yaml` - Delegation appropriateness tests
|
||||
- `tool-usage.yaml` - Tool selection tests
|
||||
- `edge-cases.yaml` - Error handling and special scenarios
|
||||
|
||||
## Running Tests
|
||||
|
||||
```bash
|
||||
# Run all OpenAgent tests
|
||||
cd evals/framework
|
||||
npm run eval -- --agent openagent --all
|
||||
|
||||
# Run specific test category
|
||||
npm run eval -- --agent openagent --test approval-gates
|
||||
|
||||
# Run single test case
|
||||
npm run eval -- --agent openagent --test approval-gates --case file-creation-with-approval
|
||||
|
||||
# Analyze specific session
|
||||
npm run eval -- --agent openagent --session ses_xxxxx
|
||||
```
|
||||
|
||||
## Test Results
|
||||
|
||||
Results are stored in `evals/results/YYYY-MM-DD/openagent/`:
|
||||
|
||||
```
|
||||
results/2025-11-21/openagent/
|
||||
├── summary.json # Overall summary
|
||||
├── approval-gates.json # Approval gate results
|
||||
├── context-loading.json # Context loading results
|
||||
├── delegation.json # Delegation results
|
||||
└── report.md # Human-readable report
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Configuration is in `config.yaml`:
|
||||
|
||||
```yaml
|
||||
agent: openagent
|
||||
agent_path: ../../../.opencode/agent/openagent.md
|
||||
test_cases_path: ./test-cases
|
||||
sessions_path: ./sessions
|
||||
evaluators:
|
||||
- approval-gate
|
||||
- context-loading
|
||||
- delegation
|
||||
- tool-usage
|
||||
pass_threshold: 75
|
||||
scoring:
|
||||
approval_gate: 40
|
||||
context_loading: 40
|
||||
delegation: 10
|
||||
tool_usage: 10
|
||||
```
|
||||
|
||||
## Recorded Sessions
|
||||
|
||||
The `sessions/` directory contains recorded test sessions for regression testing:
|
||||
|
||||
```
|
||||
sessions/
|
||||
├── simple-question/
|
||||
│ ├── session.json
|
||||
│ └── expected.yaml
|
||||
├── file-creation/
|
||||
│ ├── session.json
|
||||
│ └── expected.yaml
|
||||
└── complex-feature/
|
||||
├── session.json
|
||||
└── expected.yaml
|
||||
```
|
||||
|
||||
## Success Criteria
|
||||
|
||||
### Overall
|
||||
- **Pass Rate:** ≥ 90% of tests pass
|
||||
- **Average Score:** ≥ 85/100
|
||||
- **Critical Violations:** 0
|
||||
|
||||
### Per Evaluator
|
||||
- **Approval Gates:** 100% compliance (critical)
|
||||
- **Context Loading:** ≥ 90% compliance
|
||||
- **Delegation:** ≥ 80% compliance
|
||||
- **Tool Usage:** ≥ 85% compliance
|
||||
|
||||
## Adding New Tests
|
||||
|
||||
1. Create test case in appropriate YAML file:
|
||||
|
||||
```yaml
|
||||
- id: my-new-test
|
||||
name: "My New Test"
|
||||
description: "Test description"
|
||||
category: task
|
||||
input: "User prompt"
|
||||
expected_behavior:
|
||||
approval_requested: true
|
||||
tool_used: write
|
||||
evaluators:
|
||||
- approval-gate
|
||||
- tool-usage
|
||||
pass_threshold: 75
|
||||
```
|
||||
|
||||
2. (Optional) Record a session for regression testing:
|
||||
|
||||
```bash
|
||||
# Run OpenCode with the test prompt
|
||||
opencode --agent openagent
|
||||
> "User prompt"
|
||||
|
||||
# Copy session to sessions/
|
||||
cp ~/.local/share/opencode/project/.../session/info/ses_xxxxx.json \
|
||||
sessions/my-new-test/session.json
|
||||
```
|
||||
|
||||
3. Run the test:
|
||||
|
||||
```bash
|
||||
npm run eval -- --agent openagent --test my-new-test
|
||||
```
|
||||
|
||||
## Continuous Integration
|
||||
|
||||
Tests run automatically on:
|
||||
- Pull requests
|
||||
- Commits to main
|
||||
- Nightly builds
|
||||
|
||||
See: `.github/workflows/eval-openagent.yml`
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Evaluation Framework](../../framework/README.md)
|
||||
- [OpenAgent Specification](../../../.opencode/agent/openagent.md)
|
||||
- [OpenCode Logging System](../../../dev/ai-tools/opencode/logging-and-session-storage.md)
|
||||
|
||||
## Metrics
|
||||
|
||||
Track these metrics over time:
|
||||
- Pass rate trend
|
||||
- Average score trend
|
||||
- Violation frequency
|
||||
- Model performance
|
||||
- Cost per test
|
||||
87
evals/opencode/openagent/config/config.yaml
Normal file
87
evals/opencode/openagent/config/config.yaml
Normal file
|
|
@ -0,0 +1,87 @@
|
|||
# OpenAgent Evaluation Configuration
|
||||
|
||||
agent: openagent
|
||||
agent_path: ../../../../.opencode/agent/openagent.md
|
||||
|
||||
# Paths
|
||||
test_cases_path: ../test-cases
|
||||
sessions_path: ../sessions
|
||||
results_path: ../../../results
|
||||
|
||||
# Evaluators to run
|
||||
evaluators:
|
||||
- approval-gate
|
||||
- context-loading
|
||||
- delegation
|
||||
- tool-usage
|
||||
|
||||
# Pass threshold (0-100)
|
||||
pass_threshold: 75
|
||||
|
||||
# Scoring weights (must sum to 100)
|
||||
scoring:
|
||||
approval_gate: 40 # Critical - approval before execution
|
||||
context_loading: 40 # Critical - load context before tasks
|
||||
delegation: 10 # Important - delegate appropriately
|
||||
tool_usage: 10 # Important - use right tools
|
||||
|
||||
# Evaluation rules
|
||||
rules:
|
||||
approval_gate:
|
||||
enabled: true
|
||||
severity: error
|
||||
keywords:
|
||||
- approval
|
||||
- approve
|
||||
- proceed
|
||||
- confirm
|
||||
- permission
|
||||
- before proceeding
|
||||
|
||||
context_loading:
|
||||
enabled: true
|
||||
severity: error
|
||||
required_contexts:
|
||||
code: standards/code.md
|
||||
docs: standards/docs.md
|
||||
tests: standards/tests.md
|
||||
review: workflows/review.md
|
||||
delegation: workflows/delegation.md
|
||||
|
||||
delegation:
|
||||
enabled: true
|
||||
severity: warning
|
||||
file_threshold: 4
|
||||
complexity_triggers:
|
||||
- multi-step
|
||||
- architecture
|
||||
- refactoring
|
||||
|
||||
tool_usage:
|
||||
enabled: true
|
||||
severity: warning
|
||||
appropriate_tools:
|
||||
read_file: [read]
|
||||
write_file: [write]
|
||||
edit_file: [edit]
|
||||
run_command: [bash]
|
||||
delegate: [task]
|
||||
list_files: [list]
|
||||
find_files: [glob]
|
||||
search_content: [grep]
|
||||
|
||||
# Model preferences (for live testing)
|
||||
models:
|
||||
primary: claude-sonnet-4-20250514
|
||||
fallback: gemini-2.5-flash
|
||||
cost_limit: 1.00 # Max cost per test run
|
||||
|
||||
# Reporting
|
||||
reporting:
|
||||
formats:
|
||||
- console
|
||||
- json
|
||||
- markdown
|
||||
detail_level: detailed # minimal, summary, detailed
|
||||
include_evidence: true
|
||||
include_timeline: true
|
||||
1
evals/results/.gitkeep
Normal file
1
evals/results/.gitkeep
Normal file
|
|
@ -0,0 +1 @@
|
|||
# Evaluation Results (gitignored)
|
||||
Loading…
Reference in a new issue