041e3238bc
* docs: fix agent name casing in documentation - Update all instances of 'openagent' to 'OpenAgent' - Update all instances of 'opencoder' to 'OpenCoder' - Fixes #115: --agent flag is case-sensitive The --agent flag requires capitalized agent names (OpenAgent, OpenCoder) but documentation showed lowercase versions, causing confusion for users. Updated README.md and all docs/ files to use correct casing. * refactor: migrate from grok-code to big-pickle model - Remove model specification from contextscout.md subagent - Update eval YAML files to use opencode/big-pickle - Update documentation to reference big-pickle as default - Fixes compatibility as grok-code-fast is no longer available Files updated: - .opencode/agent/subagents/core/contextscout.md - evals/agents/shared/tests/golden/00-smoke-test.yaml - evals/agents/shared/tests/golden/02-delegation-test.yaml - evals/README.md - evals/EVAL_FRAMEWORK_GUIDE.md |
||
|---|---|---|
| .. | ||
| 00-smoke-test.yaml | ||
| 02-context-loading-explicit.yaml | ||
| 02-context-loading.yaml | ||
| 02-delegation-test.yaml | ||
| 03-read-before-write.yaml | ||
| 04-write-with-approval.yaml | ||
| 05-multi-turn-context.yaml | ||
| 06-delegation-decision.yaml | ||
| 07-tool-selection.yaml | ||
| 08-error-handling.yaml | ||
| README.md | ||
Golden Test Suite
A curated set of 8 tests that validate core agent behaviors. These tests are:
- Safe - All operations are read-only or write to
evals/test_tmp/(gitignored) - Agent-Agnostic - Work with any agent, not tied to specific implementations
- Fast - Complete in ~5-10 minutes total
- Reliable - Designed to pass consistently
Tests
| # | Test | Behavior Validated | Evaluator(s) |
|---|---|---|---|
| 01 | Smoke Test | Basic read operation | behavior |
| 02 | Context Loading | Agent reads context before answering | context-loading |
| 03 | Read Before Write | Agent inspects before modifying | execution-balance |
| 04 | Write With Approval | Agent asks before writing | approval-gate |
| 05 | Multi-Turn Context | Agent remembers conversation | behavior |
| 06 | Task Breakdown | Agent reads standards before implementing | context-loading |
| 07 | Tool Selection | Agent uses dedicated tools (not bash) | tool-usage |
| 08 | Error Handling | Agent handles errors gracefully | behavior |
Running the Golden Suite
# Run all golden tests for openagent
npm run eval:sdk -- --agent=openagent --pattern="**/golden/*.yaml"
# Run all golden tests for any agent
npm run eval:sdk -- --agent=<agent-name> --pattern="**/golden/*.yaml"
# Run with debug output
npm run eval:sdk -- --agent=openagent --pattern="**/golden/*.yaml" --debug
# Run a specific golden test
npm run eval:sdk -- --agent=openagent --pattern="**/golden/01-smoke-test.yaml"
Expected Results
All 8 tests should pass for a well-behaved agent. If tests fail:
- 01 fails: Basic infrastructure issue - check agent/server setup
- 02 fails: Agent doesn't load context - check context-loading behavior
- 03 fails: Agent writes without reading - check execution-balance
- 04 fails: Agent doesn't ask approval - check approval workflow
- 05 fails: Agent loses context - check conversation handling
- 06 fails: Agent doesn't read standards - check context-loading
- 07 fails: Agent uses bash antipatterns - check tool selection
- 08 fails: Agent crashes on errors - check error handling
Adding New Golden Tests
Golden tests should be:
- Essential - Test a core behavior that all agents need
- Safe - No dangerous operations, writes only to test_tmp/
- Reliable - Pass consistently across runs
- Fast - Complete in under 2 minutes each
- Clear - Easy to understand what's being tested