cclohmar/OpenWriterControle
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
OpenWriterControle/evals/agents/shared
History
darrenhinde cc96acc50e feat: add 5 essential workflow tests and reorganize with agents/ structure 
- Add 5 workflow tests (task, context loading, stop-on-failure, conversational)
- Add 1 shared test (agent-agnostic approval gate)
- Reorganize: opencode/ → agents/ for clearer structure
- Rename: sdk-tests/ → tests/ for simplicity
- Update test runner paths to new structure
- Add comprehensive documentation (test plan, guides, migration)
- Update all documentation with new paths

Total: 13 tests (was 6, added 7)
Coverage: +31% workflow compliance
All tests passing: 11/11 (100%)
2025-11-22 15:20:59 +00:00
..
tests/common feat: add 5 essential workflow tests and reorganize with agents/ structure 2025-11-22 15:20:59 +00:00
README.md feat: add 5 essential workflow tests and reorganize with agents/ structure 2025-11-22 15:20:59 +00:00

README.md

Shared Test Cases

Tests in this directory are agent-agnostic and can be used to test any agent that follows the same core rules.

Purpose

Shared tests validate universal behaviors that all agents should follow:

  • Approval gate enforcement
  • Tool usage patterns
  • Basic workflow compliance
  • Error handling

Usage

Run Shared Tests for OpenAgent

npm run eval:sdk -- --pattern="shared/**/*.yaml" --agent=openagent

Run Shared Tests for OpenCoder

npm run eval:sdk -- --pattern="shared/**/*.yaml" --agent=opencoder

Override Agent in Test File

# In the YAML file
agent: openagent  # Change to opencoder, or any other agent

Test Categories

common/ - Universal Rules

Tests that apply to all agents:

  • approval-gate-basic.yaml - Basic approval enforcement
  • tool-usage-basic.yaml - Basic tool selection (future)
  • error-handling-basic.yaml - Basic error handling (future)

Adding New Shared Tests

  1. Create test in shared/tests/common/
  2. Use generic prompts (not agent-specific)
  3. Test universal behaviors only
  4. Tag with shared-test and agent-agnostic
  5. Document which agents it applies to

Example

id: shared-example-001
name: Example Shared Test
category: edge-case
agent: openagent  # Default, can be overridden

prompt: "Generic prompt that works for any agent"

behavior:
  requiresApproval: true  # Universal rule

expectedViolations:
  - rule: approval-gate
    shouldViolate: false

tags:
  - shared-test
  - agent-agnostic

Benefits

  1. Reduce Duplication - Write once, test multiple agents
  2. Consistency - Same tests ensure consistent behavior
  3. Easy Comparison - Compare agent behaviors side-by-side
  4. Faster Onboarding - New agents inherit core test suite