926d68de93
* fix(evals): repair six eval-framework behavioral regressions
Restore the deterministic eval baseline by fixing false-negatives and
identity/discovery bugs across the framework:
- error-handling: correct severity-boundary fixtures so mixed-severity
cases actually cross the >1000 warning threshold (deterministic
error/warning/info), plus explicit 1000-vs-1001 boundary tests.
- context-loading-evaluator: emit `no-context-loaded` and
`context-loaded-after-execution` as errors so missing/late context
fails (was a passing warning); update focused test accordingly.
- tool-usage-evaluator: enforce dedicated tools — bash `cat`/`ls`/`ls -la`
now fail as `bash-antipattern-*` errors (ls de-allowlisted); npm/git/
piped commands still pass.
- test-case-loader: canonicalize legacy agent ids via normalizeAgentId
(`openagent` -> `core/openagent`), preserving canonical/unknown ids.
- suite-validator: accept category-based agent ids, resolve test paths
under the canonical dir, add requested-vs-declared conflict detection;
new suite-validator.test.ts.
- validate-suites-cli: recursive suite discovery (nested <cat>/<agent>/
config), deterministic order, fail-closed on zero suites; new
validate-suites-cli.test.ts.
Deterministic Vitest allowlist green; validate:suites:all non-vacuous
and exits 0. No model/network/eval:sdk execution.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(evals): clear pre-existing TypeScript build errors (tsc 39 -> 0)
`npm run build` (tsc) failed on 39 pre-existing errors in files outside
the behavioral-repair scope, blocking the baseline gate. Root causes:
- glob: drop obsolete @types/glob ^8 so glob v13's own types (globSync)
resolve (fixes run-sdk-tests.ts).
- task-type-detector: introduce a local DetectedTaskType union for this
(dead/unused) detector; shared TaskType union left unchanged.
- behavior-evaluator: the expectedResponse check filtered a nonexistent
`message` event type and was silently dead — route through
base-evaluator.getAssistantMessages (assistant_message/text). Adds
three deterministic tests exercising the now-live check.
- integration tests: remove unused `expectedOutcome` fixture metadata
(not on TestCase, never asserted) and add the required `category`
field the excess-property error had masked.
tsc now exits 0. Vitest allowlist 112/112. Lint remains unconfigured in
evals/framework (no ESLint config has ever existed) and is deferred to a
separate follow-up; it is not part of this change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(evals): sync bun.lock after @types/glob removal
The previous commit removed @types/glob via npm, updating package-lock.json
but leaving the primary bun.lock stale, which broke CI's
`bun install --frozen-lockfile`. Regenerate bun.lock so both lockfiles
agree (drops @types/glob and its transitive @types/minimatch only).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* ci(evals): add evals/framework/package-lock.json for build-check
The pr-checks `build-check` job caches on and runs `npm ci` in
evals/framework, but that workspace had no local lockfile, so the job
failed at Setup Node.js ("Some specified paths were not resolved") and
would fail `npm ci`. Add a standalone, in-sync package-lock.json
(lockfileVersion 3, no @types/glob) so the job's cache path resolves and
`npm ci` succeeds. Validated locally with `npm ci --dry-run`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
49 lines
1.4 KiB
JSON
49 lines
1.4 KiB
JSON
{
|
|
"name": "@opencode-agents/eval-framework",
|
|
"version": "0.1.1",
|
|
"type": "module",
|
|
"description": "Evaluation framework for OpenCode agents",
|
|
"main": "dist/index.js",
|
|
"types": "dist/index.d.ts",
|
|
"scripts": {
|
|
"build": "tsc",
|
|
"build:watch": "tsc --watch",
|
|
"test": "vitest",
|
|
"test:watch": "vitest --watch",
|
|
"test:integration": "vitest --config vitest.integration.config.ts",
|
|
"lint": "eslint src/**/*.ts",
|
|
"lint:fix": "eslint src/**/*.ts --fix",
|
|
"eval": "node dist/cli.js",
|
|
"report": "node dist/cli.js report",
|
|
"eval:sdk": "tsx src/sdk/run-sdk-tests.ts",
|
|
"eval:sdk:core": "tsx src/sdk/run-sdk-tests.ts --core",
|
|
"eval:sdk:debug": "tsx src/sdk/run-sdk-tests.ts --debug",
|
|
"eval:sdk:interactive": "tsx src/sdk/run-sdk-tests.ts --interactive",
|
|
"validate:suites": "tsx src/sdk/validate-suites-cli.ts",
|
|
"validate:suites:all": "tsx src/sdk/validate-suites-cli.ts --all"
|
|
},
|
|
"keywords": [
|
|
"opencode",
|
|
"evaluation",
|
|
"testing",
|
|
"agents",
|
|
"ai"
|
|
],
|
|
"author": "",
|
|
"license": "MIT",
|
|
"dependencies": {
|
|
"@opencode-ai/sdk": "^1.0.90",
|
|
"glob": "^13.0.0",
|
|
"yaml": "^2.3.4",
|
|
"zod": "^3.25.76"
|
|
},
|
|
"devDependencies": {
|
|
"@types/node": "^20.10.0",
|
|
"@typescript-eslint/eslint-plugin": "^6.13.0",
|
|
"@typescript-eslint/parser": "^6.13.0",
|
|
"eslint": "^8.54.0",
|
|
"tsx": "^4.20.6",
|
|
"typescript": "^5.3.0",
|
|
"vitest": "^1.6.1"
|
|
}
|
|
}
|