2deecb9fc4
Evidence-based validation of the hardened workflows (no GitHub test state created; no branch or worktree created or deleted). All six fork-safety claims were independently re-derived and CONFIRMED: - No live pull_request_target trigger (the only match is Task 02's warning comment; validate-registry.yml parses to pull_request + workflow_dispatch). - Fork-reachable surface is exactly 5 workflows, each with top-level contents: read and no job-level override. - Zero secret references in any fork-reachable workflow. - Every secret-bearing workflow is unreachable from a fork (push / workflow_dispatch, or issue_comment gated to OWNER/MEMBER). - Internal validation: PR #334's failed Build & Validate blocked the PR Checks Summary (exit 1); #336's 7 checks all executed and passed. Gaps found and closed: - Add persist-credentials: false to the 7 checkouts missing it (installer-checks 6, validate-test-suites 1), so all 11 checkouts across the fork-reachable surface now avoid persisting the token to .git/config. - Remove validate-test-suites.yml's failure-path PR comment step. It called issues.createComment, which needs issues: write. This was a regression introduced by Task 05: the workflow previously had no permissions block and inherited the repository's write default, so the step worked; adding the correct contents: read broke it. Deleted rather than granting write to a fork-reachable workflow — for fork PRs GitHub forces a read-only token, so the step could never work there anyway. Failures stay visible via the red check and the uploaded report. Repository setting changed (approved, read-back verified, recorded in the doc): default_workflow_permissions write -> read and can_approve_pull_request_reviews true -> false, so future workflows without an explicit permissions block default to read-only. No current workflow relied on the write default.
63 lines
2 KiB
YAML
63 lines
2 KiB
YAML
name: Validate Test Suites
|
|
|
|
on:
|
|
push:
|
|
paths:
|
|
- 'evals/agents/*/config/**/*.json'
|
|
- 'evals/agents/*/tests/**/*.yaml'
|
|
- 'scripts/validation/validate-test-suites.sh'
|
|
- '.github/workflows/validate-test-suites.yml'
|
|
pull_request:
|
|
paths:
|
|
- 'evals/agents/*/config/**/*.json'
|
|
- 'evals/agents/*/tests/**/*.yaml'
|
|
- 'scripts/validation/validate-test-suites.sh'
|
|
- '.github/workflows/validate-test-suites.yml'
|
|
workflow_dispatch:
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
validate:
|
|
name: Validate Test Suite Definitions
|
|
runs-on: ubuntu-latest
|
|
|
|
steps:
|
|
- name: Checkout code
|
|
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
|
|
with:
|
|
persist-credentials: false
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
|
|
with:
|
|
node-version: '20'
|
|
cache: 'npm'
|
|
cache-dependency-path: 'package-lock.json'
|
|
|
|
- name: Install dependencies
|
|
run: |
|
|
npm ci
|
|
|
|
- name: Validate all test suites
|
|
run: |
|
|
cd evals/framework
|
|
npm run validate:suites:all
|
|
|
|
# NOTE: A "Comment on PR (if validation failed)" step was removed here.
|
|
# It called github.rest.issues.createComment, which needs `issues: write`.
|
|
# This workflow is fork-reachable (pull_request) and intentionally runs with
|
|
# a read-only token, so the step would 403; and for fork PRs GitHub forces a
|
|
# read-only token regardless, so it could never work there. A validation
|
|
# failure is already surfaced as a failed check plus the uploaded report.
|
|
|
|
- name: Upload validation report
|
|
if: always()
|
|
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
|
|
with:
|
|
name: suite-validation-report
|
|
path: |
|
|
evals/agents/*/config/**/*.json
|
|
scripts/validation/validate-test-suites.sh
|
|
retention-days: 7
|