darrenhinde 8eb4b31ef4 feat(evals): add opencoder test suite and fix expected violation handling

- Create separate test folder for opencoder agent with 4 tests
- Update openagent tests to use multi-turn prompts for approval workflow
- Fix test runner to properly handle expectedViolations with shouldViolate: true
- Add mustUseAnyOf behavior expectation for flexible tool matching
- Add unit tests for timeline-builder and context-loading-evaluator
- Update documentation to reflect agent-specific test structure
- Remove outdated AGENT_DETECTION_STATUS.md

Opencoder tests: 4/4 passing
Key insight: openagent uses text-based approval, opencoder uses tool permissions only

2025-11-26 01:52:18 +00:00

2.5 KiB

Raw Blame History

OpenAgent Evaluation Suite

Tests for the openagent agent - a universal agent with text-based approval workflow.

Agent Characteristics

Mode: Primary universal agent
Behavior: Text-based approval workflow (Analyze→Approve→Execute→Validate)
Best for: Complex workflows, context-aware tasks, delegation
Approval: Text-based approval + tool permission system

Key Difference from Opencoder

OpenAgent uses a text-based approval workflow:

Agent outputs "Proposed Plan" and asks for approval in text
User must respond with approval (e.g., "yes, proceed")
Then agent executes the tools

Testing OpenAgent requires multi-turn prompts:

prompts:
  - text: "List the files in the current directory"
  - text: "Yes, proceed with the plan"
    delayMs: 2000

Test Categories

Developer Tests (`tests/developer/`)

Context loading tests (ctx-*.yaml)
Approval workflow tests
Multi-turn conversation tests

Business Tests (`tests/business/`)

Data analysis tasks
Conversational queries

Edge Cases (`tests/edge-case/`)

Missing approval scenarios
Error handling

Running Tests

cd evals/framework

# Run all openagent tests
npx tsx src/sdk/run-sdk-tests.ts --agent=openagent

# Run specific test pattern
npx tsx src/sdk/run-sdk-tests.ts --agent=openagent --pattern="developer/ctx-*.yaml"

# Debug mode
npx tsx src/sdk/run-sdk-tests.ts --agent=openagent --debug

Context Loading Coverage

OpenAgent requires loading context files before execution:

Task Type	Required Context File	Test
Code	`standards/code.md`	`ctx-code-001.yaml`
Docs	`standards/docs.md`	`ctx-docs-001.yaml`
Tests	`standards/tests.md`	`ctx-tests-001.yaml`
Review	`workflows/review.md`	`ctx-review-001.yaml`
Delegation	`workflows/delegation.md`	`ctx-delegation-001.yaml`
Multi-turn	Per-task context	`ctx-multi-turn-001.yaml`

Critical Rules Tested

From .opencode/agent/openagent.md:

Approval Gate - Request approval before execution
Context Loading - Load context files before tasks
Stop on Failure - Never auto-fix, report first
Delegation - Delegate 4+ file tasks to task-manager

Documentation

OPENAGENT_RULES.md - Extracted testable rules
CONTEXT_LOADING_COVERAGE.md - Context test coverage
TEST_REVIEW.md - Test suite review and status

2.5 KiB Raw Blame History