* fix(evals): repair six eval-framework behavioral regressions
Restore the deterministic eval baseline by fixing false-negatives and
identity/discovery bugs across the framework:
- error-handling: correct severity-boundary fixtures so mixed-severity
cases actually cross the >1000 warning threshold (deterministic
error/warning/info), plus explicit 1000-vs-1001 boundary tests.
- context-loading-evaluator: emit `no-context-loaded` and
`context-loaded-after-execution` as errors so missing/late context
fails (was a passing warning); update focused test accordingly.
- tool-usage-evaluator: enforce dedicated tools — bash `cat`/`ls`/`ls -la`
now fail as `bash-antipattern-*` errors (ls de-allowlisted); npm/git/
piped commands still pass.
- test-case-loader: canonicalize legacy agent ids via normalizeAgentId
(`openagent` -> `core/openagent`), preserving canonical/unknown ids.
- suite-validator: accept category-based agent ids, resolve test paths
under the canonical dir, add requested-vs-declared conflict detection;
new suite-validator.test.ts.
- validate-suites-cli: recursive suite discovery (nested <cat>/<agent>/
config), deterministic order, fail-closed on zero suites; new
validate-suites-cli.test.ts.
Deterministic Vitest allowlist green; validate:suites:all non-vacuous
and exits 0. No model/network/eval:sdk execution.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(evals): clear pre-existing TypeScript build errors (tsc 39 -> 0)
`npm run build` (tsc) failed on 39 pre-existing errors in files outside
the behavioral-repair scope, blocking the baseline gate. Root causes:
- glob: drop obsolete @types/glob ^8 so glob v13's own types (globSync)
resolve (fixes run-sdk-tests.ts).
- task-type-detector: introduce a local DetectedTaskType union for this
(dead/unused) detector; shared TaskType union left unchanged.
- behavior-evaluator: the expectedResponse check filtered a nonexistent
`message` event type and was silently dead — route through
base-evaluator.getAssistantMessages (assistant_message/text). Adds
three deterministic tests exercising the now-live check.
- integration tests: remove unused `expectedOutcome` fixture metadata
(not on TestCase, never asserted) and add the required `category`
field the excess-property error had masked.
tsc now exits 0. Vitest allowlist 112/112. Lint remains unconfigured in
evals/framework (no ESLint config has ever existed) and is deferred to a
separate follow-up; it is not part of this change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(evals): sync bun.lock after @types/glob removal
The previous commit removed @types/glob via npm, updating package-lock.json
but leaving the primary bun.lock stale, which broke CI's
`bun install --frozen-lockfile`. Regenerate bun.lock so both lockfiles
agree (drops @types/glob and its transitive @types/minimatch only).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* ci(evals): add evals/framework/package-lock.json for build-check
The pr-checks `build-check` job caches on and runs `npm ci` in
evals/framework, but that workspace had no local lockfile, so the job
failed at Setup Node.js ("Some specified paths were not resolved") and
would fail `npm ci`. Add a standalone, in-sync package-lock.json
(lockfileVersion 3, no @types/glob) so the job's cache path resolves and
`npm ci` succeeds. Validated locally with `npm ci --dry-run`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(evals): add ExecutionBalanceEvaluator, docs, tests, version bumps
* chore: remove obsolete package-lock in evals/framework pre-rebase
* chore: translate ExecutionBalanceEvaluator and test files from Spanish to English
- Translate all code comments and documentation in execution-balance-evaluator.ts
- Translate test descriptions and prompts in execution-balance-positive.yaml
- Translate test descriptions and prompts in execution-balance-negative.yaml
This ensures consistency with the rest of the English codebase as requested in PR review.
* fix(docs): translate Spanish content to English in documentation
---------
Co-authored-by: Alexander Daza <dev.alexander@example.com>
Co-authored-by: Darren Hinde <107584450+darrenhinde@users.noreply.github.com>
- Replace planned framework with working SDK implementation using @opencode-ai/sdk
- Add 6 YAML test cases across developer/business/edge-case categories (100% passing)
- Implement 4 evaluators for openagent.md rule compliance (approval-gate, context-loading, delegation, tool-usage)
- Add real-time event streaming and session recording
- Configure opencode/grok-code-fast as default model (zero API costs)
- Add comprehensive documentation (SDK_EVAL_README.md, test-design-guide.md, TEST_SCENARIOS.md)
- Support v2 schema with behavior-based expectations instead of message counts
- Enable model-agnostic test design that works across Claude, GPT-4, and Grok
- Add subagent invocation syntax documentation to openagent.md
