0d1718e551
- Create evals/test_tmp/ directory for test-created files - Update all tests to write files to test_tmp/ instead of project root - Add cleanup logic to test runner (before and after tests) - Add comprehensive HOW_TESTS_WORK.md documentation - Remove src/ directory that was accidentally created by tests This prevents tests from polluting the project directory with test artifacts.
307 lines
8.5 KiB
Markdown
307 lines
8.5 KiB
Markdown
# How the Eval Tests Work
|
|
|
|
This document explains exactly how the evaluation tests work, what they verify, and how to be confident they're testing what we think they're testing.
|
|
|
|
## Test Execution Flow
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ TEST RUNNER │
|
|
├─────────────────────────────────────────────────────────────────┤
|
|
│ 1. Clean test_tmp/ directory │
|
|
│ 2. Start opencode server (from git root) │
|
|
│ 3. For each test: │
|
|
│ a. Create session │
|
|
│ b. Send prompt(s) with agent selection │
|
|
│ c. Capture events via event stream │
|
|
│ d. Run evaluators on session data │
|
|
│ e. Check behavior expectations │
|
|
│ f. Delete session (unless --debug) │
|
|
│ 4. Clean test_tmp/ directory │
|
|
│ 5. Print results │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## How We Verify Agent Behavior
|
|
|
|
### 1. Agent Selection Verification
|
|
|
|
When a test specifies `agent: opencoder`, we verify:
|
|
|
|
```typescript
|
|
// In test-runner.ts line 340-362
|
|
const sessionInfo = await this.client.getSession(sessionId);
|
|
const firstMessage = messages[0].info;
|
|
const actualAgent = firstMessage.agent;
|
|
|
|
if (actualAgent !== testCase.agent) {
|
|
errors.push(`Agent mismatch: expected '${testCase.agent}', got '${actualAgent}'`);
|
|
}
|
|
```
|
|
|
|
**Output you'll see:**
|
|
```
|
|
Agent: opencoder
|
|
Validating agent: opencoder...
|
|
✅ Agent verified: opencoder
|
|
```
|
|
|
|
### 2. Tool Usage Verification
|
|
|
|
The BehaviorEvaluator checks which tools were actually called:
|
|
|
|
```typescript
|
|
// In behavior-evaluator.ts
|
|
const toolCalls = this.getToolCalls(timeline);
|
|
const toolsUsed = toolCalls.map(tc => tc.data?.tool);
|
|
|
|
// Check mustUseTools
|
|
for (const requiredTool of this.behavior.mustUseTools) {
|
|
if (!toolsUsed.includes(requiredTool)) {
|
|
violations.push({
|
|
type: 'missing-required-tool',
|
|
message: `Required tool '${requiredTool}' was not used`
|
|
});
|
|
}
|
|
}
|
|
```
|
|
|
|
**Output you'll see:**
|
|
```
|
|
============================================================
|
|
BEHAVIOR VALIDATION
|
|
============================================================
|
|
Timeline Events: 10
|
|
Tool Calls: 2
|
|
Tools Used: glob, read
|
|
|
|
Tool Call Details:
|
|
1. glob: {"pattern":"**/*.ts","path":"/Users/.../src"}
|
|
2. read: {"filePath":"/Users/.../src/utils/math.ts"}
|
|
```
|
|
|
|
### 3. Event Stream Capture
|
|
|
|
We capture real events from the opencode server:
|
|
|
|
```typescript
|
|
// In event-stream-handler.ts
|
|
for await (const event of response.stream) {
|
|
const serverEvent = {
|
|
type: event.type, // 'tool.call', 'message.created', etc.
|
|
properties: event.properties,
|
|
timestamp: Date.now(),
|
|
};
|
|
// Trigger handlers
|
|
}
|
|
```
|
|
|
|
**Event types captured:**
|
|
- `session.created` - Session started
|
|
- `message.created` / `message.updated` - Agent messages
|
|
- `part.created` / `part.updated` - Tool calls, text output
|
|
- `permission.request` / `permission.response` - Approval flow
|
|
|
|
### 4. Approval Flow Verification
|
|
|
|
For agents that require approval (like openagent):
|
|
|
|
```typescript
|
|
// In test-runner.ts
|
|
this.eventHandler.onPermission(async (event) => {
|
|
const approved = await approvalStrategy.shouldApprove(event);
|
|
approvalsGiven++;
|
|
this.log(`Permission ${approved ? 'APPROVED' : 'DENIED'}: ${event.properties.tool}`);
|
|
return approved;
|
|
});
|
|
```
|
|
|
|
## Test File Structure
|
|
|
|
```yaml
|
|
# Example test file
|
|
id: bash-execution-001
|
|
name: Direct Tool Execution
|
|
agent: opencoder # Which agent to use
|
|
model: anthropic/claude-sonnet-4-5 # Which model
|
|
|
|
prompt: |
|
|
List the files in the current directory using ls.
|
|
|
|
behavior:
|
|
mustUseAnyOf: [[bash], [list]] # Either tool is acceptable
|
|
minToolCalls: 1 # At least 1 tool call
|
|
mustNotContain: # Text that should NOT appear
|
|
- "Approval needed"
|
|
|
|
expectedViolations:
|
|
- rule: approval-gate
|
|
shouldViolate: true # Opencoder WILL trigger this (expected)
|
|
severity: error
|
|
|
|
approvalStrategy:
|
|
type: auto-approve # Auto-approve tool permissions
|
|
|
|
timeout: 30000
|
|
```
|
|
|
|
## Key Differences Between Agents
|
|
|
|
### Opencoder (Direct Execution)
|
|
- Executes tools immediately
|
|
- Uses tool permission system only
|
|
- No text-based approval workflow
|
|
- Tests use single prompts
|
|
|
|
```yaml
|
|
agent: opencoder
|
|
prompt: "List files in current directory"
|
|
behavior:
|
|
mustUseAnyOf: [[bash], [list]]
|
|
expectedViolations:
|
|
- rule: approval-gate
|
|
shouldViolate: true # Expected - no text approval
|
|
```
|
|
|
|
### OpenAgent (Approval Workflow)
|
|
- Outputs "Proposed Plan" first
|
|
- Waits for user approval in text
|
|
- Then executes tools
|
|
- Tests use multi-turn prompts
|
|
|
|
```yaml
|
|
agent: openagent
|
|
prompts:
|
|
- text: "List files in current directory"
|
|
- text: "Yes, proceed with the plan"
|
|
delayMs: 2000
|
|
behavior:
|
|
mustUseTools: [bash]
|
|
expectedViolations:
|
|
- rule: approval-gate
|
|
shouldViolate: false # Should ask for approval
|
|
```
|
|
|
|
## File Cleanup
|
|
|
|
Tests that create files use `evals/test_tmp/`:
|
|
|
|
```yaml
|
|
prompt: |
|
|
Create a file at evals/test_tmp/test.txt with content "Hello"
|
|
```
|
|
|
|
The test runner cleans this directory:
|
|
- Before tests start
|
|
- After tests complete
|
|
|
|
```typescript
|
|
// In run-sdk-tests.ts
|
|
function cleanupTestTmp(testTmpDir: string): void {
|
|
const preserveFiles = ['README.md', '.gitignore'];
|
|
// Remove everything else
|
|
}
|
|
```
|
|
|
|
## How to Verify Tests Are Working
|
|
|
|
### 1. Run with --debug flag
|
|
```bash
|
|
npm run eval:sdk -- --agent=opencoder --debug
|
|
```
|
|
|
|
This shows:
|
|
- All events captured
|
|
- Tool call details
|
|
- Agent verification
|
|
- Keeps sessions for inspection
|
|
|
|
### 2. Check Tool Call Details
|
|
Look for the BEHAVIOR VALIDATION section:
|
|
```
|
|
Tool Call Details:
|
|
1. glob: {"pattern":"**/*.ts","path":"..."}
|
|
2. read: {"filePath":"..."}
|
|
```
|
|
|
|
### 3. Verify Agent Selection
|
|
Look for:
|
|
```
|
|
Agent: opencoder
|
|
Validating agent: opencoder...
|
|
✅ Agent verified: opencoder
|
|
```
|
|
|
|
### 4. Check Event Count
|
|
```
|
|
Events captured: 23
|
|
```
|
|
If this is 0 or very low, something is wrong.
|
|
|
|
### 5. Inspect Session (debug mode)
|
|
```bash
|
|
# Sessions are kept in debug mode
|
|
ls ~/.local/share/opencode/storage/session/
|
|
```
|
|
|
|
## Common Issues
|
|
|
|
### "Agent not set in message"
|
|
The SDK might not return the agent field. This is a warning, not an error.
|
|
|
|
### "0 events captured"
|
|
Event stream connection failed. Check server is running.
|
|
|
|
### "Tool X was not used"
|
|
Agent used a different tool. Consider using `mustUseAnyOf` for flexibility.
|
|
|
|
### Files created in wrong location
|
|
Update test prompts to use `evals/test_tmp/` path.
|
|
|
|
## Running Tests
|
|
|
|
```bash
|
|
cd evals/framework
|
|
|
|
# All tests for specific agent
|
|
npx tsx src/sdk/run-sdk-tests.ts --agent=opencoder
|
|
|
|
# Specific test pattern
|
|
npx tsx src/sdk/run-sdk-tests.ts --agent=opencoder --pattern="developer/*.yaml"
|
|
|
|
# Debug mode (keeps sessions, verbose output)
|
|
npx tsx src/sdk/run-sdk-tests.ts --agent=opencoder --debug
|
|
|
|
# Custom model
|
|
npx tsx src/sdk/run-sdk-tests.ts --agent=opencoder --model=anthropic/claude-sonnet-4-5
|
|
```
|
|
|
|
## Test Results Interpretation
|
|
|
|
```
|
|
======================================================================
|
|
TEST RESULTS
|
|
======================================================================
|
|
|
|
1. ✅ file-read-001 - File Read Operation
|
|
Duration: 18397ms # How long the test took
|
|
Events: 23 # Events captured from server
|
|
Approvals: 0 # Permission requests handled
|
|
Context Loading: ⊘ ... # Context file status
|
|
Violations: 0 (0 errors) # Rule violations found
|
|
|
|
======================================================================
|
|
SUMMARY: 4/4 tests passed (0 failed)
|
|
======================================================================
|
|
```
|
|
|
|
## Confidence Checklist
|
|
|
|
Before trusting test results, verify:
|
|
|
|
- [ ] Agent verified message shows correct agent
|
|
- [ ] Events captured > 0
|
|
- [ ] Tool Call Details show expected tools
|
|
- [ ] Duration is reasonable (not instant = timeout)
|
|
- [ ] No unexpected errors in output
|
|
- [ ] test_tmp/ is being cleaned up
|