* fix(evals): repair six eval-framework behavioral regressions
Restore the deterministic eval baseline by fixing false-negatives and
identity/discovery bugs across the framework:
- error-handling: correct severity-boundary fixtures so mixed-severity
cases actually cross the >1000 warning threshold (deterministic
error/warning/info), plus explicit 1000-vs-1001 boundary tests.
- context-loading-evaluator: emit `no-context-loaded` and
`context-loaded-after-execution` as errors so missing/late context
fails (was a passing warning); update focused test accordingly.
- tool-usage-evaluator: enforce dedicated tools — bash `cat`/`ls`/`ls -la`
now fail as `bash-antipattern-*` errors (ls de-allowlisted); npm/git/
piped commands still pass.
- test-case-loader: canonicalize legacy agent ids via normalizeAgentId
(`openagent` -> `core/openagent`), preserving canonical/unknown ids.
- suite-validator: accept category-based agent ids, resolve test paths
under the canonical dir, add requested-vs-declared conflict detection;
new suite-validator.test.ts.
- validate-suites-cli: recursive suite discovery (nested <cat>/<agent>/
config), deterministic order, fail-closed on zero suites; new
validate-suites-cli.test.ts.
Deterministic Vitest allowlist green; validate:suites:all non-vacuous
and exits 0. No model/network/eval:sdk execution.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(evals): clear pre-existing TypeScript build errors (tsc 39 -> 0)
`npm run build` (tsc) failed on 39 pre-existing errors in files outside
the behavioral-repair scope, blocking the baseline gate. Root causes:
- glob: drop obsolete @types/glob ^8 so glob v13's own types (globSync)
resolve (fixes run-sdk-tests.ts).
- task-type-detector: introduce a local DetectedTaskType union for this
(dead/unused) detector; shared TaskType union left unchanged.
- behavior-evaluator: the expectedResponse check filtered a nonexistent
`message` event type and was silently dead — route through
base-evaluator.getAssistantMessages (assistant_message/text). Adds
three deterministic tests exercising the now-live check.
- integration tests: remove unused `expectedOutcome` fixture metadata
(not on TestCase, never asserted) and add the required `category`
field the excess-property error had masked.
tsc now exits 0. Vitest allowlist 112/112. Lint remains unconfigured in
evals/framework (no ESLint config has ever existed) and is deferred to a
separate follow-up; it is not part of this change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(evals): sync bun.lock after @types/glob removal
The previous commit removed @types/glob via npm, updating package-lock.json
but leaving the primary bun.lock stale, which broke CI's
`bun install --frozen-lockfile`. Regenerate bun.lock so both lockfiles
agree (drops @types/glob and its transitive @types/minimatch only).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* ci(evals): add evals/framework/package-lock.json for build-check
The pr-checks `build-check` job caches on and runs `npm ci` in
evals/framework, but that workspace had no local lockfile, so the job
failed at Setup Node.js ("Some specified paths were not resolved") and
would fail `npm ci`. Add a standalone, in-sync package-lock.json
(lockfileVersion 3, no @types/glob) so the job's cache path resolves and
`npm ci` succeeds. Validated locally with `npm ci --dry-run`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
926d68de93