cclohmar/OpenWriterControle
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
115 commits 1 branch 10 tags 5.2 MiB
Commit graph

3 commits

Author SHA1 Message Date
darrenhinde
e43eaff317 feat(evals): add BehaviorEvaluator and improve test validation 
- Add BehaviorEvaluator for validating agent behavior expectations:
  - mustUseTools/mustNotUseTools validation
  - minToolCalls/maxToolCalls validation
  - requiresApproval/requiresContext/shouldDelegate checks

- Fix test schema to allow behavior alone (without expectedViolations)
- Add 41 unit tests for BehaviorEvaluator, TestCaseSchema, and YAML loading
- Improve test-runner event logging with meaningful tool/message details
- Fix glob import for ESM compatibility
- Add new test cases for bash, context loading, and approval scenarios
 2025-11-26 01:10:51 +00:00
darrenhinde
478c8e3e85 feat: implement SDK-based evaluation framework with real agent testing 
- Replace planned framework with working SDK implementation using @opencode-ai/sdk
- Add 6 YAML test cases across developer/business/edge-case categories (100% passing)
- Implement 4 evaluators for openagent.md rule compliance (approval-gate, context-loading, delegation, tool-usage)
- Add real-time event streaming and session recording
- Configure opencode/grok-code-fast as default model (zero API costs)
- Add comprehensive documentation (SDK_EVAL_README.md, test-design-guide.md, TEST_SCENARIOS.md)
- Support v2 schema with behavior-based expectations instead of message counts
- Enable model-agnostic test design that works across Claude, GPT-4, and Grok
- Add subagent invocation syntax documentation to openagent.md
 2025-11-22 14:30:11 +00:00
darrenhinde
f4b3d56aa2 Add evaluation framework structure and OpenCode logging documentation 
- Add comprehensive evaluation framework in evals/
  - Reusable framework for testing OpenCode agents
  - OpenAgent-specific test configuration
  - Scalable structure for multiple agents

- Add OpenCode logging system documentation
  - Complete schema documentation for session storage
  - Model and provider tracking details
  - Performance analysis examples
  - Cost optimization strategies

- Update .gitignore
  - Ignore evals/results/ directory
  - Ignore tasks/ directory (temporary)

Structure:
- evals/framework/ - Reusable evaluation framework
- evals/opencode/openagent/ - OpenAgent tests
- dev/ai-tools/opencode/logging-and-session-storage.md - Logging docs
 2025-11-21 14:50:44 +00:00