699b2d7769
* ci(security): run deterministic eval tests in affected-area CI (Task 04) The build-check job validated build + suite structure but never ran the tests, and its summary referenced a `test:ci` script that did not exist. Make affected-area CI meaningful: - Add `test:ci` to evals/framework running the offline, deterministic Vitest allowlist (7 files / 112 tests) — no model, network, or paid execution. - Add a required "Run deterministic tests" step to build-check; a failure fails the job and the existing PR summary gate (which keys on build-check.result) fails closed. - Correct the summary note to state test:ci runs the deterministic allowlist and that model/network agent suites are excluded on PRs. The broader deterministic suite has 3 pre-existing failing files (framework-confidence, logger, test-runner; confirmed failing before this work) which are intentionally out of the allowlist and tracked separately. Validated locally: build, validate:suites:all, and test:ci all exit 0. * ci(security): least privilege permissions + pin actions to SHAs (Task 05) Harden all 9 GitHub Actions workflows: Least privilege: - Add explicit top-level `permissions: contents: read` to installer-checks, validate-test-suites, and opencode (opencode's job keeps the writes it needs; the top level now defaults to read). - Every workflow now declares an explicit top-level permissions block. - Write scopes retained only where justified (releases, doc/registry commits, follow-up PRs, and sync-docs' issues.create). Immutable action pinning: - Pin every `uses:` to a full 40-char commit SHA with a version comment, including first-party actions/* (checkout, github-script, setup-node, upload-artifact) and third-party (oven-sh/setup-bun). - Replace the two mutable refs that could change under us at any push: ludeeus/action-shellcheck@master and sst/opencode/github@latest. Record all SHAs, versions, and permission justifications in docs/maintenance/repository-recovery/action-pin-inventory.md. Verified: all 9 workflows parse as valid YAML; no floating tag/branch refs remain; every workflow has a top-level permissions block. * docs(security): add SECURITY.md and private-reporting checklist (Task 06) - Add root SECURITY.md: supported versions (0.7.x), private reporting via GitHub's "Report a vulnerability", best-effort response expectations, coordinated disclosure, and no-secret guidance. - Add private-vulnerability-reporting checklist that separates repository file work (this PR) from GitHub settings changes that require their own approval (Task 07: enabling private vulnerability reporting). No external links to validate; the reporting path uses the repo Security tab. SECURITY.md documents the private reporting path ahead of Task 07 enabling it. * ci(security): add dependency governance and ownership controls (Task 08) - .github/dependabot.yml: weekly grouped github-actions updates (keeps the Task 05 SHA pins current). Scoped to actions only — the primary lockfile is bun.lock and Dependabot's npm ecosystem would desync it; npm dependency risk is covered by dependency-review below. - .github/workflows/dependency-review.yml: runs on PRs with contents: read, SHA-pinned actions, fail-on-severity: high to block newly introduced high-severity vulnerabilities (exceptions documented inline). - .github/CODEOWNERS: default owner @darrenhinde with explicit ownership of security-sensitive paths (workflows, dependabot, CODEOWNERS, SECURITY.md, scripts/validation). Verified: dependabot.yml and dependency-review.yml are valid YAML; dependency-review actions are SHA-pinned with contents: read; CODEOWNERS owner is the repository owner. * docs(security): record GitHub security settings evidence (Task 07) Enable private vulnerability reporting and record read-back evidence. Settings changed (each approved immediately before, verified by read-back): - Private vulnerability reporting: enabled=false -> enabled=true. The Security tab now offers "Report a vulnerability", making the path documented in SECURITY.md live. - Dependency Graph / Dependabot alerts: 404 -> 204. This was an undocumented prerequisite of Task 08's dependency-review.yml, which failed with "Dependency graph is not enabled"; the Dependency Review check on PR #336 went fail -> pass after enabling it. No other repository setting was modified; secret scanning and push protection are unchanged, and dependabot_security_updates remains disabled (out of scope, would need its own approval). Both changes are reversible via the corresponding DELETE calls. Marks Part B of the private-vulnerability-reporting checklist complete.
66 lines
2 KiB
YAML
66 lines
2 KiB
YAML
name: Validate Test Suites
|
|
|
|
on:
|
|
push:
|
|
paths:
|
|
- 'evals/agents/*/config/**/*.json'
|
|
- 'evals/agents/*/tests/**/*.yaml'
|
|
- 'scripts/validation/validate-test-suites.sh'
|
|
- '.github/workflows/validate-test-suites.yml'
|
|
pull_request:
|
|
paths:
|
|
- 'evals/agents/*/config/**/*.json'
|
|
- 'evals/agents/*/tests/**/*.yaml'
|
|
- 'scripts/validation/validate-test-suites.sh'
|
|
- '.github/workflows/validate-test-suites.yml'
|
|
workflow_dispatch:
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
validate:
|
|
name: Validate Test Suite Definitions
|
|
runs-on: ubuntu-latest
|
|
|
|
steps:
|
|
- name: Checkout code
|
|
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
|
|
with:
|
|
node-version: '20'
|
|
cache: 'npm'
|
|
cache-dependency-path: 'package-lock.json'
|
|
|
|
- name: Install dependencies
|
|
run: |
|
|
npm ci
|
|
|
|
- name: Validate all test suites
|
|
run: |
|
|
cd evals/framework
|
|
npm run validate:suites:all
|
|
|
|
- name: Comment on PR (if validation failed)
|
|
if: failure() && github.event_name == 'pull_request'
|
|
uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7
|
|
with:
|
|
script: |
|
|
github.rest.issues.createComment({
|
|
issue_number: context.issue.number,
|
|
owner: context.repo.owner,
|
|
repo: context.repo.repo,
|
|
body: '❌ **Test Suite Validation Failed**\n\nPlease check the test suite JSON files for errors. Run `npm run validate:suites` locally to see details.'
|
|
})
|
|
|
|
- name: Upload validation report
|
|
if: always()
|
|
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
|
|
with:
|
|
name: suite-validation-report
|
|
path: |
|
|
evals/agents/*/config/**/*.json
|
|
scripts/validation/validate-test-suites.sh
|
|
retention-days: 7
|