8eb4b31ef4
- Create separate test folder for opencoder agent with 4 tests - Update openagent tests to use multi-turn prompts for approval workflow - Fix test runner to properly handle expectedViolations with shouldViolate: true - Add mustUseAnyOf behavior expectation for flexible tool matching - Add unit tests for timeline-builder and context-loading-evaluator - Update documentation to reflect agent-specific test structure - Remove outdated AGENT_DETECTION_STATUS.md Opencoder tests: 4/4 passing Key insight: openagent uses text-based approval, opencoder uses tool permissions only
433 lines
14 KiB
Markdown
433 lines
14 KiB
Markdown
# Eval System Architecture Overview
|
|
|
|
## Introduction
|
|
|
|
The OpenCode Evaluation Framework is a comprehensive system for testing and validating agent behavior. It captures real-time execution data, builds temporal timelines, and applies multiple evaluators to assess agent compliance with defined standards.
|
|
|
|
## System Architecture
|
|
|
|
The evaluation system consists of four main layers:
|
|
|
|
1. **Test Execution Layer** - Manages test case execution and event capture
|
|
2. **Data Collection Layer** - Captures and processes session events
|
|
3. **Timeline Building Layer** - Constructs temporal event sequences
|
|
4. **Evaluation Layer** - Applies behavioral checks and scoring
|
|
|
|
## Message Flow Diagram
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
│ TEST EXECUTION FLOW │
|
|
├─────────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ 1. TestRunner.runTest(testCase) │
|
|
│ │ │
|
|
│ ├─► EventStreamHandler.startListening() ──► Captures all ServerEvents │
|
|
│ │ │
|
|
│ ├─► ClientManager.createSession() │
|
|
│ │ │
|
|
│ ├─► ClientManager.sendPrompt() ──► Agent executes │
|
|
│ │ │
|
|
│ ├─► Events collected: session.*, message.*, part.*, permission.* │
|
|
│ │ │
|
|
│ └─► EvaluatorRunner.runAll(sessionId) │
|
|
│ │ │
|
|
│ ├─► SessionReader.getMessages() ──► Gets messages via SDK │
|
|
│ │ │
|
|
│ ├─► TimelineBuilder.buildTimeline() ──► Creates TimelineEvent[] │
|
|
│ │ │
|
|
│ └─► Each Evaluator.evaluate(timeline, sessionInfo) │
|
|
│ ├─► BehaviorEvaluator │
|
|
│ ├─► ApprovalGateEvaluator │
|
|
│ ├─► ContextLoadingEvaluator │
|
|
│ ├─► DelegationEvaluator │
|
|
│ └─► ToolUsageEvaluator │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Component Details
|
|
|
|
### 1. Test Execution Layer
|
|
|
|
#### TestRunner
|
|
- **Purpose**: Orchestrates test case execution
|
|
- **Key Methods**:
|
|
- `runTest(testCase)` - Executes a single test case
|
|
- `runAll(testCases)` - Runs multiple test cases in sequence
|
|
- `loadTestCases(path)` - Loads YAML test definitions
|
|
- **Responsibilities**:
|
|
- Initialize client session
|
|
- Send user prompts
|
|
- Coordinate event capture
|
|
- Invoke evaluators
|
|
- Generate results
|
|
|
|
#### EventStreamHandler
|
|
- **Purpose**: Captures real-time server events during execution
|
|
- **Event Types Captured**:
|
|
- `session.*` - Session lifecycle events
|
|
- `message.*` - Message creation and completion
|
|
- `part.*` - Message parts (text, tool use, etc.)
|
|
- `permission.*` - Approval requests and responses
|
|
- **Output**: Raw event stream for timeline construction
|
|
|
|
#### ClientManager
|
|
- **Purpose**: Manages OpenCode client lifecycle
|
|
- **Key Methods**:
|
|
- `createSession()` - Initialize new test session
|
|
- `sendPrompt(message)` - Send user message to agent
|
|
- `waitForCompletion()` - Wait for agent response
|
|
- **Integration**: Uses OpenCode SDK for client operations
|
|
|
|
### 2. Data Collection Layer
|
|
|
|
#### SessionReader
|
|
- **Purpose**: Reads session data from OpenCode storage
|
|
- **Storage Location**: `~/.local/share/opencode/`
|
|
- **Key Methods**:
|
|
- `getSessionInfo(sessionId)` - Retrieve session metadata
|
|
- `getMessages(sessionId)` - Get all messages in session
|
|
- `getParts(sessionId, messageId)` - Get message parts
|
|
- **Data Sources**:
|
|
- `session.json` - Session metadata
|
|
- `messages.jsonl` - Message stream
|
|
- `parts/` - Message part files
|
|
|
|
#### MessageParser
|
|
- **Purpose**: Extract structured data from messages
|
|
- **Parsing Operations**:
|
|
- Agent identification (openagent, subagent, etc.)
|
|
- Model selection tracking
|
|
- Token usage and cost metrics
|
|
- Timing information
|
|
- **Output**: Normalized message objects
|
|
|
|
### 3. Timeline Building Layer
|
|
|
|
#### TimelineBuilder
|
|
- **Purpose**: Construct temporal event sequences from session data
|
|
- **Algorithm**:
|
|
1. Read all messages via SessionReader
|
|
2. Parse each message for events (tool calls, approvals, etc.)
|
|
3. Sort events chronologically by timestamp
|
|
4. Enrich events with context (agent, model, metrics)
|
|
- **Event Types**:
|
|
- `user_message` - User prompts
|
|
- `assistant_message` - Agent responses
|
|
- `tool_call` - Tool invocations
|
|
- `patch` - Code edits
|
|
- `approval_request` - Permission requests
|
|
- `approval_response` - User approval/denial
|
|
- **Output**: `TimelineEvent[]` - Ordered sequence of events
|
|
|
|
### 4. Evaluation Layer
|
|
|
|
#### EvaluatorRunner
|
|
- **Purpose**: Coordinate execution of all evaluators
|
|
- **Process**:
|
|
1. Receive sessionId and timeline
|
|
2. Instantiate all registered evaluators
|
|
3. Execute each evaluator's `evaluate()` method
|
|
4. Aggregate results and calculate overall score
|
|
- **Output**: `TestResult` with all evaluation results
|
|
|
|
#### Individual Evaluators
|
|
|
|
##### BehaviorEvaluator
|
|
- **Checks**: General behavioral compliance
|
|
- **Rules**:
|
|
- Context file loading before execution
|
|
- Proper scratchpad usage
|
|
- Adherence to agent-specific rules
|
|
|
|
##### ApprovalGateEvaluator
|
|
- **Checks**: Approval gate compliance
|
|
- **Rules**:
|
|
- Request approval before bash, write, edit, task
|
|
- No execution without approval
|
|
- Proper approval handling
|
|
|
|
##### ContextLoadingEvaluator
|
|
- **Checks**: Context file loading
|
|
- **Rules**:
|
|
- Load docs.md before documentation tasks
|
|
- Load tests.md before testing tasks
|
|
- Load relevant context before specialized tasks
|
|
|
|
##### DelegationEvaluator
|
|
- **Checks**: Task delegation decisions
|
|
- **Rules**:
|
|
- Delegate when 4+ files involved
|
|
- Delegate complex multi-step tasks
|
|
- Use appropriate subagent types
|
|
|
|
##### ToolUsageEvaluator
|
|
- **Checks**: Tool selection appropriateness
|
|
- **Rules**:
|
|
- Use Read instead of bash cat
|
|
- Use Task for exploration
|
|
- Prefer specialized tools over bash
|
|
|
|
## Data Flow
|
|
|
|
### Phase 1: Test Execution
|
|
```
|
|
Test YAML → TestRunner → ClientManager → Agent Execution
|
|
↓
|
|
EventStreamHandler
|
|
↓
|
|
Event Collection
|
|
```
|
|
|
|
### Phase 2: Data Collection
|
|
```
|
|
SessionReader → ~/.local/share/opencode/
|
|
↓
|
|
Message Parsing → MessageParser
|
|
↓
|
|
Structured Data
|
|
```
|
|
|
|
### Phase 3: Timeline Construction
|
|
```
|
|
Messages + Events → TimelineBuilder
|
|
↓
|
|
Chronological Sorting
|
|
↓
|
|
Event Enrichment
|
|
↓
|
|
TimelineEvent[]
|
|
```
|
|
|
|
### Phase 4: Evaluation
|
|
```
|
|
Timeline → EvaluatorRunner
|
|
↓
|
|
BehaviorEvaluator ──┐
|
|
ApprovalGateEvaluator ──┤
|
|
ContextLoadingEvaluator ──┤→ Results Aggregation
|
|
DelegationEvaluator ──┤
|
|
ToolUsageEvaluator ──┘
|
|
↓
|
|
TestResult
|
|
```
|
|
|
|
## Key Design Principles
|
|
|
|
### 1. Event-Driven Architecture
|
|
- All agent actions captured as events
|
|
- Events stored in chronological order
|
|
- Evaluators work with event timeline, not raw data
|
|
|
|
### 2. Separation of Concerns
|
|
- **Collection** - Gather data without interpretation
|
|
- **Transformation** - Build timeline from raw events
|
|
- **Evaluation** - Apply business rules to timeline
|
|
|
|
### 3. Extensibility
|
|
- New evaluators implement `BaseEvaluator` interface
|
|
- Evaluators registered in config
|
|
- No changes to collection/timeline layers needed
|
|
|
|
### 4. Reproducibility
|
|
- All session data persisted
|
|
- Tests can be re-evaluated without re-execution
|
|
- Historical analysis of past sessions
|
|
|
|
### 5. Composability
|
|
- Evaluators run independently
|
|
- Results aggregated into overall score
|
|
- Individual evaluator results available
|
|
|
|
## Event Schema
|
|
|
|
### TimelineEvent
|
|
```typescript
|
|
interface TimelineEvent {
|
|
timestamp: number; // Unix timestamp in ms
|
|
type: EventType; // Event category
|
|
agent?: string; // Agent that generated event
|
|
model?: string; // Model used
|
|
data: EventData; // Event-specific payload
|
|
}
|
|
|
|
type EventType =
|
|
| 'user_message'
|
|
| 'assistant_message'
|
|
| 'tool_call'
|
|
| 'patch'
|
|
| 'approval_request'
|
|
| 'approval_response';
|
|
```
|
|
|
|
### Tool Call Event
|
|
```typescript
|
|
interface ToolCallEvent {
|
|
timestamp: number;
|
|
type: 'tool_call';
|
|
data: {
|
|
tool: string; // Tool name (e.g., 'read', 'bash')
|
|
parameters: any; // Tool parameters
|
|
result?: any; // Tool result (if available)
|
|
};
|
|
}
|
|
```
|
|
|
|
### Approval Event
|
|
```typescript
|
|
interface ApprovalRequestEvent {
|
|
timestamp: number;
|
|
type: 'approval_request';
|
|
data: {
|
|
tool: string; // Tool requiring approval
|
|
parameters: any; // Parameters for review
|
|
};
|
|
}
|
|
|
|
interface ApprovalResponseEvent {
|
|
timestamp: number;
|
|
type: 'approval_response';
|
|
data: {
|
|
approved: boolean; // User decision
|
|
requestTimestamp: number; // Link to request
|
|
};
|
|
}
|
|
```
|
|
|
|
## Evaluation Scoring
|
|
|
|
### Weighted Checks
|
|
Each evaluator defines weighted checks:
|
|
```typescript
|
|
const checks = [
|
|
{ name: 'approval_before_bash', passed: true, weight: 30 },
|
|
{ name: 'approval_before_write', passed: true, weight: 30 },
|
|
{ name: 'no_unapproved_execution', passed: false, weight: 40 }
|
|
];
|
|
```
|
|
|
|
### Score Calculation
|
|
```typescript
|
|
const totalWeight = sum(checks.map(c => c.weight));
|
|
const achievedWeight = sum(checks.filter(c => c.passed).map(c => c.weight));
|
|
const score = (achievedWeight / totalWeight) * 100;
|
|
```
|
|
|
|
### Overall Test Score
|
|
```typescript
|
|
const evaluatorScores = evaluationResults.map(r => r.score);
|
|
const overallScore = average(evaluatorScores);
|
|
const passed = overallScore >= passThreshold; // Default: 75
|
|
```
|
|
|
|
## Storage Structure
|
|
|
|
```
|
|
~/.local/share/opencode/
|
|
└── sessions/
|
|
└── {sessionId}/
|
|
├── session.json # Session metadata
|
|
├── messages.jsonl # Message stream
|
|
└── parts/ # Message parts
|
|
├── {partId}.txt
|
|
└── {partId}.json
|
|
```
|
|
|
|
## Configuration
|
|
|
|
### Evaluator Registration
|
|
```typescript
|
|
// config.ts
|
|
export const config = {
|
|
evaluators: {
|
|
'behavior': BehaviorEvaluator,
|
|
'approval-gate': ApprovalGateEvaluator,
|
|
'context-loading': ContextLoadingEvaluator,
|
|
'delegation': DelegationEvaluator,
|
|
'tool-usage': ToolUsageEvaluator,
|
|
},
|
|
passThreshold: 75,
|
|
};
|
|
```
|
|
|
|
### Test Configuration
|
|
```yaml
|
|
# test-case.yaml
|
|
id: test-001
|
|
description: Test approval gates
|
|
prompt: "Create a new file called test.js"
|
|
expected:
|
|
behavior:
|
|
- approval_requested
|
|
- no_unapproved_execution
|
|
evaluators:
|
|
- approval-gate
|
|
- tool-usage
|
|
```
|
|
|
|
## Error Handling
|
|
|
|
### Collection Errors
|
|
- **Session not found**: Return empty timeline, mark test as skipped
|
|
- **Malformed messages**: Log warning, skip message, continue
|
|
- **Missing parts**: Use partial data, note in metadata
|
|
|
|
### Evaluation Errors
|
|
- **Evaluator exception**: Mark evaluator as failed, continue with others
|
|
- **Missing required data**: Return 0 score with violation
|
|
- **Timeout**: Kill evaluator, mark as error
|
|
|
|
## Performance Considerations
|
|
|
|
### Timeline Building
|
|
- **Lazy loading**: Only load messages when needed
|
|
- **Caching**: Cache parsed messages within session
|
|
- **Streaming**: Process messages as stream, not all at once
|
|
|
|
### Evaluation
|
|
- **Parallel execution**: Run independent evaluators concurrently
|
|
- **Early termination**: Stop if critical failures detected
|
|
- **Incremental scoring**: Calculate scores progressively
|
|
|
|
## Future Enhancements
|
|
|
|
1. **Real-time Evaluation**
|
|
- Evaluate as events occur, not post-execution
|
|
- Provide live feedback during test execution
|
|
|
|
2. **Comparative Analysis**
|
|
- Compare results across test runs
|
|
- Track improvement over time
|
|
- Identify regression patterns
|
|
|
|
3. **Smart Approval**
|
|
- Auto-approve safe operations based on learned patterns
|
|
- Reduce test execution time
|
|
|
|
4. **Visual Timeline**
|
|
- Interactive timeline visualization
|
|
- Filter events by type/agent/tool
|
|
- Drill down into specific interactions
|
|
|
|
5. **Custom Evaluators**
|
|
- User-defined evaluation rules
|
|
- Domain-specific checks
|
|
- Plugin architecture
|
|
|
|
## Related Documentation
|
|
|
|
- [Test Design Guide](./test-design-guide.md) - How to write effective tests
|
|
- [SDK Evaluation README](../SDK_EVAL_README.md) - SDK-based evaluation approach
|
|
- [Agent Testing Guide](../../agents/AGENT_TESTING_GUIDE.md) - Testing specific agents
|
|
|
|
## Summary
|
|
|
|
The evaluation framework provides a robust, extensible system for validating agent behavior. By capturing real-time events, building temporal timelines, and applying multiple independent evaluators, it ensures comprehensive testing while maintaining clarity and debuggability.
|
|
|
|
Key strengths:
|
|
- **Separation of concerns** between collection, transformation, and evaluation
|
|
- **Event-driven** architecture for accurate temporal analysis
|
|
- **Extensible** evaluator system for custom checks
|
|
- **Reproducible** results through persisted session data
|
|
- **Composable** scoring from independent evaluators
|