cclohmar/OpenWriterControle
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
OpenWriterControle/evals/agents/openagent
History
darrenhinde 8eb4b31ef4 feat(evals): add opencoder test suite and fix expected violation handling 
- Create separate test folder for opencoder agent with 4 tests
- Update openagent tests to use multi-turn prompts for approval workflow
- Fix test runner to properly handle expectedViolations with shouldViolate: true
- Add mustUseAnyOf behavior expectation for flexible tool matching
- Add unit tests for timeline-builder and context-loading-evaluator
- Update documentation to reflect agent-specific test structure
- Remove outdated AGENT_DETECTION_STATUS.md

Opencoder tests: 4/4 passing
Key insight: openagent uses text-based approval, opencoder uses tool permissions only
2025-11-26 01:52:18 +00:00
..
config feat: add 5 essential workflow tests and reorganize with agents/ structure 2025-11-22 15:20:59 +00:00
docs chore: clean up redundant documentation and old test files 2025-11-22 15:23:09 +00:00
tests feat(evals): add opencoder test suite and fix expected violation handling 2025-11-26 01:52:18 +00:00
CONTEXT_LOADING_COVERAGE.md feat(evals): add opencoder test suite and fix expected violation handling 2025-11-26 01:52:18 +00:00
README.md feat(evals): add opencoder test suite and fix expected violation handling 2025-11-26 01:52:18 +00:00
TEST_REVIEW.md feat(evals): add opencoder test suite and fix expected violation handling 2025-11-26 01:52:18 +00:00

README.md

OpenAgent Evaluation Suite

Tests for the openagent agent - a universal agent with text-based approval workflow.

Agent Characteristics

  • Mode: Primary universal agent
  • Behavior: Text-based approval workflow (Analyze→Approve→Execute→Validate)
  • Best for: Complex workflows, context-aware tasks, delegation
  • Approval: Text-based approval + tool permission system

Key Difference from Opencoder

OpenAgent uses a text-based approval workflow:

  • Agent outputs "Proposed Plan" and asks for approval in text
  • User must respond with approval (e.g., "yes, proceed")
  • Then agent executes the tools

Testing OpenAgent requires multi-turn prompts:

prompts:
  - text: "List the files in the current directory"
  - text: "Yes, proceed with the plan"
    delayMs: 2000

Test Categories

Developer Tests (tests/developer/)

  • Context loading tests (ctx-*.yaml)
  • Approval workflow tests
  • Multi-turn conversation tests

Business Tests (tests/business/)

  • Data analysis tasks
  • Conversational queries

Edge Cases (tests/edge-case/)

  • Missing approval scenarios
  • Error handling

Running Tests

cd evals/framework

# Run all openagent tests
npx tsx src/sdk/run-sdk-tests.ts --agent=openagent

# Run specific test pattern
npx tsx src/sdk/run-sdk-tests.ts --agent=openagent --pattern="developer/ctx-*.yaml"

# Debug mode
npx tsx src/sdk/run-sdk-tests.ts --agent=openagent --debug

Context Loading Coverage

OpenAgent requires loading context files before execution:

Task Type Required Context File Test
Code standards/code.md ctx-code-001.yaml
Docs standards/docs.md ctx-docs-001.yaml
Tests standards/tests.md ctx-tests-001.yaml
Review workflows/review.md ctx-review-001.yaml
Delegation workflows/delegation.md ctx-delegation-001.yaml
Multi-turn Per-task context ctx-multi-turn-001.yaml

Critical Rules Tested

From .opencode/agent/openagent.md:

  1. Approval Gate - Request approval before execution
  2. Context Loading - Load context files before tasks
  3. Stop on Failure - Never auto-fix, report first
  4. Delegation - Delegate 4+ file tasks to task-manager

Documentation