699b2d7769
* ci(security): run deterministic eval tests in affected-area CI (Task 04) The build-check job validated build + suite structure but never ran the tests, and its summary referenced a `test:ci` script that did not exist. Make affected-area CI meaningful: - Add `test:ci` to evals/framework running the offline, deterministic Vitest allowlist (7 files / 112 tests) — no model, network, or paid execution. - Add a required "Run deterministic tests" step to build-check; a failure fails the job and the existing PR summary gate (which keys on build-check.result) fails closed. - Correct the summary note to state test:ci runs the deterministic allowlist and that model/network agent suites are excluded on PRs. The broader deterministic suite has 3 pre-existing failing files (framework-confidence, logger, test-runner; confirmed failing before this work) which are intentionally out of the allowlist and tracked separately. Validated locally: build, validate:suites:all, and test:ci all exit 0. * ci(security): least privilege permissions + pin actions to SHAs (Task 05) Harden all 9 GitHub Actions workflows: Least privilege: - Add explicit top-level `permissions: contents: read` to installer-checks, validate-test-suites, and opencode (opencode's job keeps the writes it needs; the top level now defaults to read). - Every workflow now declares an explicit top-level permissions block. - Write scopes retained only where justified (releases, doc/registry commits, follow-up PRs, and sync-docs' issues.create). Immutable action pinning: - Pin every `uses:` to a full 40-char commit SHA with a version comment, including first-party actions/* (checkout, github-script, setup-node, upload-artifact) and third-party (oven-sh/setup-bun). - Replace the two mutable refs that could change under us at any push: ludeeus/action-shellcheck@master and sst/opencode/github@latest. Record all SHAs, versions, and permission justifications in docs/maintenance/repository-recovery/action-pin-inventory.md. Verified: all 9 workflows parse as valid YAML; no floating tag/branch refs remain; every workflow has a top-level permissions block. * docs(security): add SECURITY.md and private-reporting checklist (Task 06) - Add root SECURITY.md: supported versions (0.7.x), private reporting via GitHub's "Report a vulnerability", best-effort response expectations, coordinated disclosure, and no-secret guidance. - Add private-vulnerability-reporting checklist that separates repository file work (this PR) from GitHub settings changes that require their own approval (Task 07: enabling private vulnerability reporting). No external links to validate; the reporting path uses the repo Security tab. SECURITY.md documents the private reporting path ahead of Task 07 enabling it. * ci(security): add dependency governance and ownership controls (Task 08) - .github/dependabot.yml: weekly grouped github-actions updates (keeps the Task 05 SHA pins current). Scoped to actions only — the primary lockfile is bun.lock and Dependabot's npm ecosystem would desync it; npm dependency risk is covered by dependency-review below. - .github/workflows/dependency-review.yml: runs on PRs with contents: read, SHA-pinned actions, fail-on-severity: high to block newly introduced high-severity vulnerabilities (exceptions documented inline). - .github/CODEOWNERS: default owner @darrenhinde with explicit ownership of security-sensitive paths (workflows, dependabot, CODEOWNERS, SECURITY.md, scripts/validation). Verified: dependabot.yml and dependency-review.yml are valid YAML; dependency-review actions are SHA-pinned with contents: read; CODEOWNERS owner is the repository owner. * docs(security): record GitHub security settings evidence (Task 07) Enable private vulnerability reporting and record read-back evidence. Settings changed (each approved immediately before, verified by read-back): - Private vulnerability reporting: enabled=false -> enabled=true. The Security tab now offers "Report a vulnerability", making the path documented in SECURITY.md live. - Dependency Graph / Dependabot alerts: 404 -> 204. This was an undocumented prerequisite of Task 08's dependency-review.yml, which failed with "Dependency graph is not enabled"; the Dependency Review check on PR #336 went fail -> pass after enabling it. No other repository setting was modified; secret scanning and push protection are unchanged, and dependabot_security_updates remains disabled (out of scope, would need its own approval). Both changes are reversible via the corresponding DELETE calls. Marks Part B of the private-vulnerability-reporting checklist complete.
50 lines
1.8 KiB
JSON
50 lines
1.8 KiB
JSON
{
|
|
"name": "@opencode-agents/eval-framework",
|
|
"version": "0.1.1",
|
|
"type": "module",
|
|
"description": "Evaluation framework for OpenCode agents",
|
|
"main": "dist/index.js",
|
|
"types": "dist/index.d.ts",
|
|
"scripts": {
|
|
"build": "tsc",
|
|
"build:watch": "tsc --watch",
|
|
"test": "vitest",
|
|
"test:watch": "vitest --watch",
|
|
"test:ci": "vitest run src/__tests__/error-handling.test.ts src/evaluators/__tests__/evaluator-reliability.test.ts src/evaluators/__tests__/context-loading-evaluator.test.ts src/evaluators/__tests__/tool-usage-enhanced.test.ts src/sdk/__tests__/yaml-loader.test.ts src/sdk/__tests__/suite-validator.test.ts src/sdk/__tests__/validate-suites-cli.test.ts",
|
|
"test:integration": "vitest --config vitest.integration.config.ts",
|
|
"lint": "eslint src/**/*.ts",
|
|
"lint:fix": "eslint src/**/*.ts --fix",
|
|
"eval": "node dist/cli.js",
|
|
"report": "node dist/cli.js report",
|
|
"eval:sdk": "tsx src/sdk/run-sdk-tests.ts",
|
|
"eval:sdk:core": "tsx src/sdk/run-sdk-tests.ts --core",
|
|
"eval:sdk:debug": "tsx src/sdk/run-sdk-tests.ts --debug",
|
|
"eval:sdk:interactive": "tsx src/sdk/run-sdk-tests.ts --interactive",
|
|
"validate:suites": "tsx src/sdk/validate-suites-cli.ts",
|
|
"validate:suites:all": "tsx src/sdk/validate-suites-cli.ts --all"
|
|
},
|
|
"keywords": [
|
|
"opencode",
|
|
"evaluation",
|
|
"testing",
|
|
"agents",
|
|
"ai"
|
|
],
|
|
"author": "",
|
|
"license": "MIT",
|
|
"dependencies": {
|
|
"@opencode-ai/sdk": "^1.0.90",
|
|
"glob": "^13.0.0",
|
|
"yaml": "^2.3.4",
|
|
"zod": "^3.25.76"
|
|
},
|
|
"devDependencies": {
|
|
"@types/node": "^20.10.0",
|
|
"@typescript-eslint/eslint-plugin": "^6.13.0",
|
|
"@typescript-eslint/parser": "^6.13.0",
|
|
"eslint": "^8.54.0",
|
|
"tsx": "^4.20.6",
|
|
"typescript": "^5.3.0",
|
|
"vitest": "^1.6.1"
|
|
}
|
|
}
|