OpenWriterControle/evals/README.md

# OpenCode Agent Evaluation Framework

## Overview

Comprehensive evaluation framework for testing and validating OpenCode agent behavior against defined standards and rules.

## Structure

```
evals/
├── framework/              # Reusable evaluation framework
│   ├── src/
│   │   ├── collector/     # Session data collection
│   │   ├── evaluators/    # Evaluation logic
│   │   ├── runner/        # Test execution
│   │   ├── reporters/     # Result reporting
│   │   └── types/         # TypeScript types
│   └── package.json
│
├── opencode/              # OpenCode agent evaluations
│   ├── openagent/        # OpenAgent-specific tests
│   ├── opencoder/        # OpenCoder-specific tests
│   └── shared/           # Shared test cases
│
└── results/              # Test results (gitignored)
    └── YYYY-MM-DD/
```

## Quick Start

### 1. Install Framework

```bash
cd evals/framework
npm install
npm run build
```

### 2. Run Evaluations

```bash
# Evaluate a specific session
npm run eval -- --agent openagent --session ses_xxxxx

# Run all OpenAgent tests
npm run eval -- --agent openagent --all

# Run specific test case
npm run eval -- --agent openagent --test approval-gates
```

### 3. View Results

```bash
# View latest results
cat evals/results/$(ls -t evals/results | head -1)/openagent/summary.json | jq

# Generate report
npm run report -- --agent openagent --date 2025-11-21
```

## Framework Components

### Collector
Reads and parses OpenCode session data from `~/.local/share/opencode/`

**Components:**
- `SessionReader` - Read session files
- `MessageParser` - Parse message structure
- `TimelineBuilder` - Build event timeline

### Evaluators
Validate agent behavior against rules

**Available Evaluators:**
- `ApprovalGateEvaluator` - Check approval before execution
- `ContextLoadingEvaluator` - Verify context loading
- `DelegationEvaluator` - Validate delegation decisions
- `ToolUsageEvaluator` - Check tool selection
- `ModelSelectionEvaluator` - Validate model choices

### Runner
Execute test cases and collect results

**Components:**
- `TestRunner` - Run test suites
- `SessionAnalyzer` - Analyze historical sessions
- `LiveRunner` - Run live tests (future)

### Reporters
Generate test reports

**Formats:**
- Console (pretty output)
- JSON (machine-readable)
- Markdown (documentation)
- HTML (dashboard, future)

## Agent-Specific Tests

### OpenAgent (`opencode/openagent/`)

Tests for the universal OpenAgent:
- Approval gate enforcement
- Context loading compliance
- Delegation appropriateness
- Tool usage patterns
- Model selection

**Test Categories:**
- Conversational (simple questions)
- Task (file operations)
- Complex (multi-file features)
- Edge cases (errors, permissions)

### OpenCoder (`opencode/opencoder/`)

Tests for the OpenCoder agent (future)

### Shared (`opencode/shared/`)

Common test cases applicable to all agents

## Test Case Format

Test cases are defined in YAML:

```yaml
test_cases:
  - id: simple-question
    name: "Simple Question (No Execution)"
    description: "Ask a question that requires no execution tools"
    category: conversational
    input: "What does this code do?"
    expected_behavior:
      no_execution_tools: true
      no_approval_required: true
      response_provided: true
    evaluators:
      - approval-gate
      - tool-usage
    pass_threshold: 100
```

## Evaluation Results

Results include:
- **Pass/Fail** - Did the test pass?
- **Score** - 0-100 compliance score
- **Violations** - List of rule violations
- **Evidence** - Supporting data for violations
- **Metadata** - Agent, model, timing, costs

**Example Result:**
```json
{
  "testCaseId": "simple-question",
  "sessionId": "ses_xxxxx",
  "passed": true,
  "score": 100,
  "evaluationResults": [
    {
      "evaluator": "approval-gate",
      "passed": true,
      "score": 100,
      "violations": []
    }
  ],
  "metadata": {
    "agent": "openagent",
    "model": "claude-sonnet-4-20250514",
    "duration": 7205,
    "cost": 0.107
  }
}
```

## Scoring System

- **100** - Perfect compliance
- **75-99** - Minor issues (warnings)
- **50-74** - Moderate issues (violations)
- **0-49** - Major issues (critical violations)

**Pass Threshold:** 75% (configurable per test)

## Development

### Adding New Evaluators

1. Create evaluator in `framework/src/evaluators/`
2. Extend `BaseEvaluator`
3. Implement `evaluate()` method
4. Add tests
5. Register in evaluator registry

### Adding New Test Cases

1. Create YAML file in `opencode/{agent}/test-cases/`
2. Define test case structure
3. Specify expected behavior
4. List evaluators to run
5. Set pass threshold

### Running Tests

```bash
# Framework tests
cd evals/framework
npm test

# Integration tests
npm run test:integration

# Test specific evaluator
npm run test:evaluator -- approval-gate
```

## CI/CD Integration

```yaml
# .github/workflows/eval.yml
name: Agent Evaluation

on: [push, pull_request]

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
      - run: cd evals/framework && npm install
      - run: npm run eval -- --agent openagent --all
      - run: npm run report -- --format json
```

## Configuration

### Framework Config (`framework/config.ts`)

```typescript
export const config = {
  projectPath: process.cwd(),
  sessionStoragePath: '~/.local/share/opencode/',
  resultsPath: 'evals/results/',
  passThreshold: 75,
  evaluators: ['approval-gate', 'context-loading', 'delegation', 'tool-usage']
};
```

### Agent Config (`opencode/openagent/config.yaml`)

```yaml
agent: openagent
test_cases_path: ./test-cases
sessions_path: ./sessions
evaluators:
  - approval-gate
  - context-loading
  - delegation
  - tool-usage
pass_threshold: 75
```

## Related Documentation

- [Framework Architecture](./framework/README.md)
- [OpenAgent Tests](./opencode/openagent/README.md)
- [OpenCode Logging System](../dev/ai-tools/opencode/logging-and-session-storage.md)
- [Agent Validator Plugin](../.opencode/plugin/docs/VALIDATOR_GUIDE.md)

## Contributing

See [CONTRIBUTING.md](../docs/contributing/CONTRIBUTING.md)

## License

MIT