OpenAgent requires text-based approval before executing write/edit tools.
Updated tests to use multi-turn prompts:
1. First prompt: Request the task
2. Second prompt: 'Yes, proceed' to approve
Also added TESTING_CONFIDENCE.md documenting current test reliability:
- Opencoder: HIGH confidence (4/4 tests passing)
- OpenAgent: MEDIUM confidence (multi-turn works, context loading needs verification)
- Create evals/test_tmp/ directory for test-created files
- Update all tests to write files to test_tmp/ instead of project root
- Add cleanup logic to test runner (before and after tests)
- Add comprehensive HOW_TESTS_WORK.md documentation
- Remove src/ directory that was accidentally created by tests
This prevents tests from polluting the project directory with test artifacts.
- Create separate test folder for opencoder agent with 4 tests
- Update openagent tests to use multi-turn prompts for approval workflow
- Fix test runner to properly handle expectedViolations with shouldViolate: true
- Add mustUseAnyOf behavior expectation for flexible tool matching
- Add unit tests for timeline-builder and context-loading-evaluator
- Update documentation to reflect agent-specific test structure
- Remove outdated AGENT_DETECTION_STATUS.md
Opencoder tests: 4/4 passing
Key insight: openagent uses text-based approval, opencoder uses tool permissions only
- Replace planned framework with working SDK implementation using @opencode-ai/sdk
- Add 6 YAML test cases across developer/business/edge-case categories (100% passing)
- Implement 4 evaluators for openagent.md rule compliance (approval-gate, context-loading, delegation, tool-usage)
- Add real-time event streaming and session recording
- Configure opencode/grok-code-fast as default model (zero API costs)
- Add comprehensive documentation (SDK_EVAL_README.md, test-design-guide.md, TEST_SCENARIOS.md)
- Support v2 schema with behavior-based expectations instead of message counts
- Enable model-agnostic test design that works across Claude, GPT-4, and Grok
- Add subagent invocation syntax documentation to openagent.md
