OpenWriterControle/evals/GETTING_STARTED.md

Getting Started with OpenCode Agent Evaluation

Quick start guide for running and understanding agent tests

---

Prerequisites

# Install dependencies
cd evals/framework
npm install
npm run build

---

Running Tests

Quick Start

# Run all tests (uses free model by default)
npm run eval:sdk

# Run specific agent
npm run eval:sdk -- --agent=openagent
npm run eval:sdk -- --agent=opencoder

# Run specific test category
npm run eval:sdk -- --agent=openagent --pattern="context-loading/*.yaml"

# Debug mode (verbose output, keeps sessions)
npm run eval:sdk -- --debug

Batch Execution (Avoid API Limits)

# Run tests in batches of 3 with 10s delays
./scripts/utils/run-tests-batch.sh openagent 3 10

---

Understanding Test Results

Test Output Example

======================================================================
TEST RESULTS
======================================================================

1. ✅ ctx-simple-coding-standards - Context Loading: Coding Standards
   Duration: 22821ms
   Events: 18
   Approvals: 0
   Context Loading: ⊘ Conversational session (not required)
   Violations: 0 (0 errors, 0 warnings)

2. ✅ ctx-multi-standards-to-docs - Multi-Turn Standards to Documentation
   Duration: 116455ms
   Events: 164
   Approvals: 0
   Context Loading:
     ✓ Loaded: .opencode/context/core/standards/code.md
     ✓ Timing: Context loaded 44317ms before execution
   Violations: 0 (0 errors, 0 warnings)

======================================================================
SUMMARY: 2/2 tests passed (0 failed)
======================================================================

What Each Field Means

Field	Meaning
Duration	Total test execution time (includes agent thinking + tool execution)
Events	Number of events captured from server (messages, tool calls, etc.)
Approvals	Tool permission requests handled (not text-based approvals)
Context Loading	Whether context files were loaded before execution
Violations	Rule violations detected by evaluators

---

Test Execution Flow

┌─────────────────────────────────────────────────────────────────┐
│                        TEST RUNNER                               │
├─────────────────────────────────────────────────────────────────┤
│  1. Clean test_tmp/ directory                                    │
│  2. Start opencode server (from git root)                        │
│  3. For each test:                                               │
│     a. Create session                                            │
│     b. Send prompt(s) with agent selection                       │
│     c. Capture events via event stream                           │
│     d. Run evaluators on session data                            │
│     e. Check behavior expectations                               │
│     f. Delete session (unless --debug)                           │
│  4. Clean test_tmp/ directory                                    │
│  5. Save results to JSON                                         │
│  6. Print results                                                │
└─────────────────────────────────────────────────────────────────┘

---

Agent Differences

Opencoder (Direct Execution)

• Executes tools immediately
• Uses tool permission system only
• No text-based approval workflow
• Tests use single prompts

Example Test:

agent: opencoder
prompt: "List files in current directory"
behavior:
  mustUseAnyOf: [[bash], [list]]

OpenAgent (Approval Workflow)

• Outputs "Proposed Plan" first
• Waits for user approval in text
• Then executes tools
• Tests use multi-turn prompts

Example Test:

agent: openagent
prompts:
  - text: "List files in current directory"
  - text: "approve"
    delayMs: 2000
behavior:
  mustUseTools: [bash]

---

Creating New Tests

Simple Test (Single Prompt)

# File: evals/agents/openagent/tests/context-loading/my-test.yaml
id: my-test-001
name: "My Test Name"
description: |
  What this test validates  

category: developer
agent: openagent
model: anthropic/claude-sonnet-4-5

prompt: "Your test prompt here"

behavior:
  mustUseTools: [read]
  requiresContext: true
  minToolCalls: 1

expectedViolations:
  - rule: context-loading
    shouldViolate: false
    severity: error

approvalStrategy:
  type: auto-approve

timeout: 60000

tags:
  - context-loading
  - simple-test

Complex Test (Multi-Turn)

id: my-complex-test-001
name: "Multi-Turn Test"
description: |
  Tests multi-turn conversation with context loading  

category: developer
agent: openagent
model: anthropic/claude-sonnet-4-5

prompts:
  - text: "What are our coding standards?"
    expectContext: true
    contextFile: "standards.md"
  
  - text: "approve"
    delayMs: 2000
  
  - text: "Create documentation about these standards"
    expectContext: true
    contextFile: "docs.md"
  
  - text: "approve"
    delayMs: 2000

behavior:
  mustUseTools: [read, write]
  requiresApproval: true
  requiresContext: true
  minToolCalls: 3

expectedViolations:
  - rule: approval-gate
    shouldViolate: false
    severity: error
  
  - rule: context-loading
    shouldViolate: false
    severity: error

approvalStrategy:
  type: auto-approve

timeout: 300000  # 5 minutes

tags:
  - context-loading
  - multi-turn
  - complex-test

---

Viewing Results

Dashboard

cd evals/results
./serve.sh

This will:

1. Start HTTP server on port 8000
2. Open browser automatically
3. Load test results dashboard
4. Auto-shutdown after 15 seconds

The dashboard caches data in your browser, so it works even after the server shuts down.

JSON Results

# Latest results
cat evals/results/latest.json

# Historical results
ls evals/results/history/2025-11/

---

File Cleanup

Tests that create files use evals/test_tmp/:

prompt: |
  Create a file at evals/test_tmp/test.txt with content "Hello"  

The test runner automatically cleans this directory:

• Before tests start - Removes all files except .gitignore and README.md
• After tests complete - Removes all test artifacts

---

Debugging Tests

Enable Debug Mode

npm run eval:sdk -- --agent=openagent --pattern="my-test.yaml" --debug

Debug mode shows:

• All events captured
• Tool call details with full inputs
• Agent verification steps
• Keeps sessions for inspection (not deleted)

Inspect Sessions

# Sessions are stored here
ls ~/.local/share/opencode/storage/session/

# View session details (in debug mode)
cat ~/.local/share/opencode/storage/session/<session-id>.json

Check Tool Calls

Look for the BEHAVIOR VALIDATION section in output:

============================================================
BEHAVIOR VALIDATION
============================================================
Timeline Events: 28
Tool Calls: 3
Tools Used: read, write

Tool Call Details:
  1. read: {"filePath":".opencode/context/core/standards/code.md"}
  2. read: {"filePath":".opencode/context/core/standards/docs.md"}
  3. write: {"filePath":"evals/test_tmp/output.md"}

[behavior] Files Read (2):
  1. .opencode/context/core/standards/code.md
  2. .opencode/context/core/standards/docs.md
[behavior] Context Files Read: 2/2

Behavior Validation Summary:
  Checks Passed: 4/4
  Violations: 0
============================================================

---

Common Issues

"Agent not set in message"

Cause: SDK might not return the agent field
Impact: Warning only, not an error
Action: Ignore - test still validates correctly

"0 events captured"

Cause: Event stream connection failed
Action: Check server is running, restart test

"Tool X was not used"

Cause: Agent used a different tool
Action: Use mustUseAnyOf for flexibility:

behavior:
  mustUseAnyOf: [[bash], [list]]  # Either tool is acceptable

"Files created in wrong location"

Cause: Test prompt doesn't specify evals/test_tmp/
Action: Update test prompt to use correct path

"Timeout"

Cause: Test took longer than timeout value
Action: Increase timeout in test YAML:

timeout: 300000  # 5 minutes

---

Test Categories

Category	Purpose	Example Tests
context-loading	Verify context files loaded before execution	ctx-simple-coding-standards
developer	Developer workflow tests	create-component, install-dependencies
business	Business analysis tests	data-analysis
edge-case	Edge cases and error handling	just-do-it, missing-approval

---

Model Configuration

Free Tier (Default)

# Uses opencode/grok-code-fast (free)
npm run eval:sdk

Paid Models

# Claude 3.5 Sonnet
npm run eval:sdk -- --model=anthropic/claude-3-5-sonnet-20241022

# GPT-4 Turbo
npm run eval:sdk -- --model=openai/gpt-4-turbo

Per-Test Override

# In test YAML file
model: anthropic/claude-3-5-sonnet-20241022

---

Next Steps

1. Read the docs:

   • README.md - System overview
   • ARCHITECTURE.md - System architecture
   • framework/SDK_EVAL_README.md - Complete SDK guide

2. Explore tests:

   • evals/agents/openagent/tests/context-loading/ - Context loading tests
   • evals/agents/opencoder/tests/developer/ - Opencoder tests

3. Run tests:

   npm run eval:sdk -- --agent=openagent --pattern="context-loading/*.yaml"
   

4. View results:

   cd ../results && ./serve.sh
   

---

Support

• Issues: Check HOW_TESTS_WORK.md for detailed explanations
• Test Design: See framework/docs/test-design-guide.md
• Agent Rules: See agents/openagent/docs/OPENAGENT_RULES.md

---

Happy Testing! 🚀