f773b290ce
Documentation: - Add ARCHITECTURE.md with comprehensive system review - Add GETTING_STARTED.md for new users - Add DOCUMENTATION_CLEANUP.md tracking cleanup changes - Add SCRIPTS_ORGANIZATION.md documenting script reorganization - Remove outdated docs (TESTING_CONFIDENCE.md, TEST_REVIEW.md, SESSION_STORAGE_FIX.md) - Update README files with current information Script Organization: - Move 12 scripts from framework root to organized directories: - scripts/debug/ (4 files): Session debugging tools - scripts/test/ (6 files): Test execution scripts - scripts/utils/ (2 files): Utility scripts - Add comprehensive scripts/README.md with usage examples Test Infrastructure: - Add 5 new context loading tests for openagent - Add result-saver.ts for test result persistence - Add interactive results dashboard (index.html) - Add error-handling.test.ts for framework testing - Add result-saver.test.ts with comprehensive unit tests Evaluator Improvements: - Enhance context-loading-evaluator with better detection - Update run-sdk-tests.ts with improved result handling Agent Configuration: - Add .opencode/agent/AGENT.md for agent definition Test Utilities: - Add test-agent-manual.mjs for manual testing - Update test_tmp/README.md with usage guidelines This commit consolidates multiple improvements: - Better organization (scripts, docs) - Improved testing infrastructure - Comprehensive documentation - Cleaner codebase structure
435 lines
10 KiB
Markdown
435 lines
10 KiB
Markdown
# Getting Started with OpenCode Agent Evaluation
|
|
|
|
**Quick start guide for running and understanding agent tests**
|
|
|
|
---
|
|
|
|
## Prerequisites
|
|
|
|
```bash
|
|
# Install dependencies
|
|
cd evals/framework
|
|
npm install
|
|
npm run build
|
|
```
|
|
|
|
---
|
|
|
|
## Running Tests
|
|
|
|
### Quick Start
|
|
|
|
```bash
|
|
# Run all tests (uses free model by default)
|
|
npm run eval:sdk
|
|
|
|
# Run specific agent
|
|
npm run eval:sdk -- --agent=openagent
|
|
npm run eval:sdk -- --agent=opencoder
|
|
|
|
# Run specific test category
|
|
npm run eval:sdk -- --agent=openagent --pattern="context-loading/*.yaml"
|
|
|
|
# Debug mode (verbose output, keeps sessions)
|
|
npm run eval:sdk -- --debug
|
|
```
|
|
|
|
### Batch Execution (Avoid API Limits)
|
|
|
|
```bash
|
|
# Run tests in batches of 3 with 10s delays
|
|
./scripts/utils/run-tests-batch.sh openagent 3 10
|
|
```
|
|
|
|
---
|
|
|
|
## Understanding Test Results
|
|
|
|
### Test Output Example
|
|
|
|
```
|
|
======================================================================
|
|
TEST RESULTS
|
|
======================================================================
|
|
|
|
1. ✅ ctx-simple-coding-standards - Context Loading: Coding Standards
|
|
Duration: 22821ms
|
|
Events: 18
|
|
Approvals: 0
|
|
Context Loading: ⊘ Conversational session (not required)
|
|
Violations: 0 (0 errors, 0 warnings)
|
|
|
|
2. ✅ ctx-multi-standards-to-docs - Multi-Turn Standards to Documentation
|
|
Duration: 116455ms
|
|
Events: 164
|
|
Approvals: 0
|
|
Context Loading:
|
|
✓ Loaded: .opencode/context/core/standards/code.md
|
|
✓ Timing: Context loaded 44317ms before execution
|
|
Violations: 0 (0 errors, 0 warnings)
|
|
|
|
======================================================================
|
|
SUMMARY: 2/2 tests passed (0 failed)
|
|
======================================================================
|
|
```
|
|
|
|
### What Each Field Means
|
|
|
|
| Field | Meaning |
|
|
|-------|---------|
|
|
| **Duration** | Total test execution time (includes agent thinking + tool execution) |
|
|
| **Events** | Number of events captured from server (messages, tool calls, etc.) |
|
|
| **Approvals** | Tool permission requests handled (not text-based approvals) |
|
|
| **Context Loading** | Whether context files were loaded before execution |
|
|
| **Violations** | Rule violations detected by evaluators |
|
|
|
|
---
|
|
|
|
## Test Execution Flow
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ TEST RUNNER │
|
|
├─────────────────────────────────────────────────────────────────┤
|
|
│ 1. Clean test_tmp/ directory │
|
|
│ 2. Start opencode server (from git root) │
|
|
│ 3. For each test: │
|
|
│ a. Create session │
|
|
│ b. Send prompt(s) with agent selection │
|
|
│ c. Capture events via event stream │
|
|
│ d. Run evaluators on session data │
|
|
│ e. Check behavior expectations │
|
|
│ f. Delete session (unless --debug) │
|
|
│ 4. Clean test_tmp/ directory │
|
|
│ 5. Save results to JSON │
|
|
│ 6. Print results │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Agent Differences
|
|
|
|
### Opencoder (Direct Execution)
|
|
- Executes tools immediately
|
|
- Uses tool permission system only
|
|
- No text-based approval workflow
|
|
- Tests use single prompts
|
|
|
|
**Example Test:**
|
|
```yaml
|
|
agent: opencoder
|
|
prompt: "List files in current directory"
|
|
behavior:
|
|
mustUseAnyOf: [[bash], [list]]
|
|
```
|
|
|
|
### OpenAgent (Approval Workflow)
|
|
- Outputs "Proposed Plan" first
|
|
- Waits for user approval in text
|
|
- Then executes tools
|
|
- Tests use multi-turn prompts
|
|
|
|
**Example Test:**
|
|
```yaml
|
|
agent: openagent
|
|
prompts:
|
|
- text: "List files in current directory"
|
|
- text: "approve"
|
|
delayMs: 2000
|
|
behavior:
|
|
mustUseTools: [bash]
|
|
```
|
|
|
|
---
|
|
|
|
## Creating New Tests
|
|
|
|
### Simple Test (Single Prompt)
|
|
|
|
```yaml
|
|
# File: evals/agents/openagent/tests/context-loading/my-test.yaml
|
|
id: my-test-001
|
|
name: "My Test Name"
|
|
description: |
|
|
What this test validates
|
|
|
|
category: developer
|
|
agent: openagent
|
|
model: anthropic/claude-sonnet-4-5
|
|
|
|
prompt: "Your test prompt here"
|
|
|
|
behavior:
|
|
mustUseTools: [read]
|
|
requiresContext: true
|
|
minToolCalls: 1
|
|
|
|
expectedViolations:
|
|
- rule: context-loading
|
|
shouldViolate: false
|
|
severity: error
|
|
|
|
approvalStrategy:
|
|
type: auto-approve
|
|
|
|
timeout: 60000
|
|
|
|
tags:
|
|
- context-loading
|
|
- simple-test
|
|
```
|
|
|
|
### Complex Test (Multi-Turn)
|
|
|
|
```yaml
|
|
id: my-complex-test-001
|
|
name: "Multi-Turn Test"
|
|
description: |
|
|
Tests multi-turn conversation with context loading
|
|
|
|
category: developer
|
|
agent: openagent
|
|
model: anthropic/claude-sonnet-4-5
|
|
|
|
prompts:
|
|
- text: "What are our coding standards?"
|
|
expectContext: true
|
|
contextFile: "standards.md"
|
|
|
|
- text: "approve"
|
|
delayMs: 2000
|
|
|
|
- text: "Create documentation about these standards"
|
|
expectContext: true
|
|
contextFile: "docs.md"
|
|
|
|
- text: "approve"
|
|
delayMs: 2000
|
|
|
|
behavior:
|
|
mustUseTools: [read, write]
|
|
requiresApproval: true
|
|
requiresContext: true
|
|
minToolCalls: 3
|
|
|
|
expectedViolations:
|
|
- rule: approval-gate
|
|
shouldViolate: false
|
|
severity: error
|
|
|
|
- rule: context-loading
|
|
shouldViolate: false
|
|
severity: error
|
|
|
|
approvalStrategy:
|
|
type: auto-approve
|
|
|
|
timeout: 300000 # 5 minutes
|
|
|
|
tags:
|
|
- context-loading
|
|
- multi-turn
|
|
- complex-test
|
|
```
|
|
|
|
---
|
|
|
|
## Viewing Results
|
|
|
|
### Dashboard
|
|
|
|
```bash
|
|
cd evals/results
|
|
./serve.sh
|
|
```
|
|
|
|
This will:
|
|
1. Start HTTP server on port 8000
|
|
2. Open browser automatically
|
|
3. Load test results dashboard
|
|
4. Auto-shutdown after 15 seconds
|
|
|
|
The dashboard caches data in your browser, so it works even after the server shuts down.
|
|
|
|
### JSON Results
|
|
|
|
```bash
|
|
# Latest results
|
|
cat evals/results/latest.json
|
|
|
|
# Historical results
|
|
ls evals/results/history/2025-11/
|
|
```
|
|
|
|
---
|
|
|
|
## File Cleanup
|
|
|
|
Tests that create files use `evals/test_tmp/`:
|
|
|
|
```yaml
|
|
prompt: |
|
|
Create a file at evals/test_tmp/test.txt with content "Hello"
|
|
```
|
|
|
|
The test runner automatically cleans this directory:
|
|
- **Before tests start** - Removes all files except `.gitignore` and `README.md`
|
|
- **After tests complete** - Removes all test artifacts
|
|
|
|
---
|
|
|
|
## Debugging Tests
|
|
|
|
### Enable Debug Mode
|
|
|
|
```bash
|
|
npm run eval:sdk -- --agent=openagent --pattern="my-test.yaml" --debug
|
|
```
|
|
|
|
Debug mode shows:
|
|
- All events captured
|
|
- Tool call details with full inputs
|
|
- Agent verification steps
|
|
- Keeps sessions for inspection (not deleted)
|
|
|
|
### Inspect Sessions
|
|
|
|
```bash
|
|
# Sessions are stored here
|
|
ls ~/.local/share/opencode/storage/session/
|
|
|
|
# View session details (in debug mode)
|
|
cat ~/.local/share/opencode/storage/session/<session-id>.json
|
|
```
|
|
|
|
### Check Tool Calls
|
|
|
|
Look for the **BEHAVIOR VALIDATION** section in output:
|
|
|
|
```
|
|
============================================================
|
|
BEHAVIOR VALIDATION
|
|
============================================================
|
|
Timeline Events: 28
|
|
Tool Calls: 3
|
|
Tools Used: read, write
|
|
|
|
Tool Call Details:
|
|
1. read: {"filePath":".opencode/context/core/standards/code.md"}
|
|
2. read: {"filePath":".opencode/context/core/standards/docs.md"}
|
|
3. write: {"filePath":"evals/test_tmp/output.md"}
|
|
|
|
[behavior] Files Read (2):
|
|
1. .opencode/context/core/standards/code.md
|
|
2. .opencode/context/core/standards/docs.md
|
|
[behavior] Context Files Read: 2/2
|
|
|
|
Behavior Validation Summary:
|
|
Checks Passed: 4/4
|
|
Violations: 0
|
|
============================================================
|
|
```
|
|
|
|
---
|
|
|
|
## Common Issues
|
|
|
|
### "Agent not set in message"
|
|
**Cause**: SDK might not return the agent field
|
|
**Impact**: Warning only, not an error
|
|
**Action**: Ignore - test still validates correctly
|
|
|
|
### "0 events captured"
|
|
**Cause**: Event stream connection failed
|
|
**Action**: Check server is running, restart test
|
|
|
|
### "Tool X was not used"
|
|
**Cause**: Agent used a different tool
|
|
**Action**: Use `mustUseAnyOf` for flexibility:
|
|
```yaml
|
|
behavior:
|
|
mustUseAnyOf: [[bash], [list]] # Either tool is acceptable
|
|
```
|
|
|
|
### "Files created in wrong location"
|
|
**Cause**: Test prompt doesn't specify `evals/test_tmp/`
|
|
**Action**: Update test prompt to use correct path
|
|
|
|
### "Timeout"
|
|
**Cause**: Test took longer than timeout value
|
|
**Action**: Increase timeout in test YAML:
|
|
```yaml
|
|
timeout: 300000 # 5 minutes
|
|
```
|
|
|
|
---
|
|
|
|
## Test Categories
|
|
|
|
| Category | Purpose | Example Tests |
|
|
|----------|---------|---------------|
|
|
| **context-loading** | Verify context files loaded before execution | ctx-simple-coding-standards |
|
|
| **developer** | Developer workflow tests | create-component, install-dependencies |
|
|
| **business** | Business analysis tests | data-analysis |
|
|
| **edge-case** | Edge cases and error handling | just-do-it, missing-approval |
|
|
|
|
---
|
|
|
|
## Model Configuration
|
|
|
|
### Free Tier (Default)
|
|
```bash
|
|
# Uses opencode/grok-code-fast (free)
|
|
npm run eval:sdk
|
|
```
|
|
|
|
### Paid Models
|
|
```bash
|
|
# Claude 3.5 Sonnet
|
|
npm run eval:sdk -- --model=anthropic/claude-3-5-sonnet-20241022
|
|
|
|
# GPT-4 Turbo
|
|
npm run eval:sdk -- --model=openai/gpt-4-turbo
|
|
```
|
|
|
|
### Per-Test Override
|
|
```yaml
|
|
# In test YAML file
|
|
model: anthropic/claude-3-5-sonnet-20241022
|
|
```
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
1. **Read the docs**:
|
|
- [README.md](README.md) - System overview
|
|
- [ARCHITECTURE.md](ARCHITECTURE.md) - System architecture
|
|
- [framework/SDK_EVAL_README.md](framework/SDK_EVAL_README.md) - Complete SDK guide
|
|
|
|
2. **Explore tests**:
|
|
- `evals/agents/openagent/tests/context-loading/` - Context loading tests
|
|
- `evals/agents/opencoder/tests/developer/` - Opencoder tests
|
|
|
|
3. **Run tests**:
|
|
```bash
|
|
npm run eval:sdk -- --agent=openagent --pattern="context-loading/*.yaml"
|
|
```
|
|
|
|
4. **View results**:
|
|
```bash
|
|
cd ../results && ./serve.sh
|
|
```
|
|
|
|
---
|
|
|
|
## Support
|
|
|
|
- **Issues**: Check [HOW_TESTS_WORK.md](HOW_TESTS_WORK.md) for detailed explanations
|
|
- **Test Design**: See [framework/docs/test-design-guide.md](framework/docs/test-design-guide.md)
|
|
- **Agent Rules**: See [agents/openagent/docs/OPENAGENT_RULES.md](agents/openagent/docs/OPENAGENT_RULES.md)
|
|
|
|
---
|
|
|
|
**Happy Testing!** 🚀
|