cclohmar/OpenWriterControle
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
OpenWriterControle/evals/agents/opencoder/README.md
darrenhinde 8eb4b31ef4 feat(evals): add opencoder test suite and fix expected violation handling 
- Create separate test folder for opencoder agent with 4 tests
- Update openagent tests to use multi-turn prompts for approval workflow
- Fix test runner to properly handle expectedViolations with shouldViolate: true
- Add mustUseAnyOf behavior expectation for flexible tool matching
- Add unit tests for timeline-builder and context-loading-evaluator
- Update documentation to reflect agent-specific test structure
- Remove outdated AGENT_DETECTION_STATUS.md

Opencoder tests: 4/4 passing
Key insight: openagent uses text-based approval, opencoder uses tool permissions only
2025-11-26 01:52:18 +00:00

1.1 KiB
Raw Blame History

Opencoder Agent Tests

Tests for the opencoder agent - a development-focused agent that executes code tasks directly.

Agent Characteristics

  • Mode: Primary development agent
  • Behavior: Executes tools directly without text-based approval workflow
  • Best for: Code implementation, bash commands, file operations
  • Approval: Uses tool permission system (auto-approve in tests)

Test Categories

Developer Tests (tests/developer/)

  • Bash command execution
  • File operations
  • Code implementation tasks

Business Tests (tests/business/)

  • Data analysis tasks
  • Report generation

Edge Cases (tests/edge-case/)

  • Error handling
  • Permission boundaries

Running Tests

cd evals/framework
npx tsx src/sdk/run-sdk-tests.ts --agent opencoder

Key Differences from OpenAgent

Feature Opencoder OpenAgent
Approval Tool permission system Text-based + tool permission
Workflow Direct execution Analyze→Approve→Execute→Validate
Context Loading On-demand Mandatory before execution
Best for Simple tasks Complex workflows