darrenhinde
8eb4b31ef4
feat(evals): add opencoder test suite and fix expected violation handling
...
- Create separate test folder for opencoder agent with 4 tests
- Update openagent tests to use multi-turn prompts for approval workflow
- Fix test runner to properly handle expectedViolations with shouldViolate: true
- Add mustUseAnyOf behavior expectation for flexible tool matching
- Add unit tests for timeline-builder and context-loading-evaluator
- Update documentation to reflect agent-specific test structure
- Remove outdated AGENT_DETECTION_STATUS.md
Opencoder tests: 4/4 passing
Key insight: openagent uses text-based approval, opencoder uses tool permissions only
2025-11-26 01:52:18 +00:00
darrenhinde
e43eaff317
feat(evals): add BehaviorEvaluator and improve test validation
...
- Add BehaviorEvaluator for validating agent behavior expectations:
- mustUseTools/mustNotUseTools validation
- minToolCalls/maxToolCalls validation
- requiresApproval/requiresContext/shouldDelegate checks
- Fix test schema to allow behavior alone (without expectedViolations)
- Add 41 unit tests for BehaviorEvaluator, TestCaseSchema, and YAML loading
- Improve test-runner event logging with meaningful tool/message details
- Fix glob import for ESM compatibility
- Add new test cases for bash, context loading, and approval scenarios
2025-11-26 01:10:51 +00:00
darrenhinde
c0a8395878
chore: clean up redundant documentation and old test files
...
- Delete 5 temporary root-level docs (alignment, migration, summaries)
- Delete 1 redundant agent guide (merged into main guide)
- Delete 4 old openagent files (test scenarios, results, old runner)
- Delete tests/simple/ directory (old synthetic test data)
- Delete 3 old framework test scripts
Result: 62% fewer documentation files (21 → 8)
Kept only essential docs with clear purposes
2025-11-22 15:23:09 +00:00
darrenhinde
cc96acc50e
feat: add 5 essential workflow tests and reorganize with agents/ structure
...
- Add 5 workflow tests (task, context loading, stop-on-failure, conversational)
- Add 1 shared test (agent-agnostic approval gate)
- Reorganize: opencode/ → agents/ for clearer structure
- Rename: sdk-tests/ → tests/ for simplicity
- Update test runner paths to new structure
- Add comprehensive documentation (test plan, guides, migration)
- Update all documentation with new paths
Total: 13 tests (was 6, added 7)
Coverage: +31% workflow compliance
All tests passing: 11/11 (100%)
2025-11-22 15:20:59 +00:00
