diff --git a/.github/EXTERNAL_PR_GUIDE.md b/.github/EXTERNAL_PR_GUIDE.md index fb17cff..cf20f12 100644 --- a/.github/EXTERNAL_PR_GUIDE.md +++ b/.github/EXTERNAL_PR_GUIDE.md @@ -8,27 +8,48 @@ When you submit a PR from a fork, workflows run but **cannot auto-commit fixes** ### What Runs Automatically +✅ **Fast Build Check (< 2 minutes)** +- TypeScript compilation check +- YAML test suite validation +- Quick feedback on code quality + ✅ **Registry Validation** - Checks if `registry.json` matches actual files - Validates all paths are correct - Checks prompts use defaults - **Cannot auto-commit** (you'll need to fix locally) -✅ **Agent Tests** -- OpenAgent and OpenCoder smoke tests run -- Results are reported in the PR +❌ **What Doesn't Run on PRs** +- Full AI agent tests (too expensive, run manually by maintainers) +- Auto-version bumping (happens after merge) ### What You Need to Do If the validation workflow reports issues, you'll need to fix them locally: -#### 1. Registry Issues +#### 1. Build Failures -If new components are detected: +If TypeScript compilation fails: + +```bash +# Build locally to see errors +cd evals/framework +npm install +npm run build + +# Fix the errors, then commit +git add . +git commit -m "fix: resolve build errors" +git push +``` + +#### 2. Registry Issues + +If new components are detected (you'll get a comment on your PR): ```bash # Run auto-detect locally -./scripts/auto-detect-components.sh --auto-add +./scripts/registry/auto-detect-components.sh --auto-add # Commit the updated registry git add registry.json @@ -36,7 +57,7 @@ git commit -m "chore: update registry with new components" git push ``` -#### 2. Prompt Issues +#### 3. Prompt Issues If prompts don't match defaults: @@ -50,7 +71,7 @@ git commit -m "chore: restore default prompts" git push ``` -#### 3. Path Issues +#### 4. Path Issues If registry paths are invalid: @@ -68,8 +89,10 @@ git push ### What Happens Next 1. **Fix Issues Locally**: If validation fails, fix issues and push -2. **Maintainer Review**: A maintainer will review your code -3. **Merge**: Once everything passes, your PR will be merged! +2. **Fast Feedback**: Build checks complete in < 2 minutes +3. **Maintainer Review**: A maintainer will review your code +4. **Merge**: Once everything passes, your PR will be merged! +5. **Auto-Release**: After merge, version is auto-bumped and release created (you don't need to do anything!) --- @@ -89,20 +112,21 @@ If you need to override checks: 3. Enable **"Skip validation checks"** 4. Run on the PR branch -**Skip Tests:** -1. Go to **Actions** → **Test Agents** +**Skip Version Bump:** +1. Go to **Actions** → **Post-Merge Automation** 2. Click **"Run workflow"** -3. Enable **"Skip tests"** -4. Run on the PR branch +3. Enable **"Skip version bump"** +4. Run manually ### Workflow Behavior -| PR Type | Registry Validation | Auto-Commit | Agent Tests | -|---------|-------------------|-------------|-------------| -| **Internal PR** (branch) | ✅ Yes | ✅ Yes | ✅ Yes | -| **External PR** (fork) | ✅ Yes | ❌ No* | ✅ Yes | +| PR Type | Build Check | Registry Validation | Auto-Commit | AI Tests | +|---------|------------|-------------------|-------------|----------| +| **Internal PR** (branch) | ✅ Yes | ✅ Yes | ✅ Yes | ❌ No* | +| **External PR** (fork) | ✅ Yes | ✅ Yes | ❌ No** | ❌ No* | -*Cannot push to fork branches - contributor must fix locally +*AI tests don't run on PRs (too expensive) - maintainers run manually if needed +**Cannot push to fork branches - contributor must fix locally ### Manual Testing External PRs @@ -112,14 +136,20 @@ If you want to test an external PR locally: # Fetch the PR gh pr checkout +# Run build check +cd evals/framework +npm install +npm run build +npm run validate:suites:all + # Run validation -./scripts/validate-registry.sh -v +./scripts/registry/validate-registry.sh -v ./scripts/prompts/validate-pr.sh # Auto-fix registry if needed -./scripts/auto-detect-components.sh --auto-add +./scripts/registry/auto-detect-components.sh --auto-add -# Run tests +# Run AI tests (optional - only if needed) npm run test:ci # If everything passes, approve and merge @@ -127,12 +157,15 @@ gh pr review --approve gh pr merge ``` +**Note:** AI tests are optional for PRs. The build check is usually sufficient. + --- ## Workflow Files +- **`.github/workflows/pr-checks.yml`**: Fast build validation (< 2 min) - **`.github/workflows/validate-registry.yml`**: Registry validation with override -- **`.github/workflows/test-agents.yml`**: Agent tests with override +- **`.github/workflows/post-merge.yml`**: Auto-versioning after merge (main only) ## Questions? diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 42ac3c6..3c2f37f 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -1 +1,207 @@ -# CI/CD Pipeline Test +# Workflow Guide + +This directory contains GitHub Actions workflows for automated testing, validation, and releases. + +## Workflow Overview + +### For Contributors (PR Workflows) + +#### `pr-checks.yml` - Fast Build Validation +**Triggers:** Pull requests to main/dev with evals changes +**Duration:** < 2 minutes +**What it does:** +- ✅ TypeScript compilation check +- ✅ YAML test suite validation +- ✅ Fast feedback on code quality + +**Fork-friendly:** ✅ Yes - read-only checks + +--- + +#### `validate-registry.yml` - Registry Validation +**Triggers:** Pull requests to main/dev +**Duration:** < 1 minute +**What it does:** +- ✅ Validates registry.json paths +- ✅ Auto-detects new components +- ✅ Validates prompts use defaults +- ✅ **Internal PRs:** Auto-commits registry updates +- ✅ **Fork PRs:** Posts comment with instructions + +**Fork-friendly:** ✅ Yes - detects forks and provides guidance + +--- + +### For Maintainers (Post-Merge Workflows) + +#### `post-merge.yml` - Auto-Versioning & Releases +**Triggers:** Push to main branch +**Duration:** < 1 minute +**What it does:** +- ✅ Auto-bumps version based on conventional commits +- ✅ Updates CHANGELOG.md +- ✅ Creates git tag +- ✅ Publishes GitHub release + +**Conventional Commit Examples:** +```bash +feat: add new feature # → Minor bump (0.1.0 → 0.2.0) +fix: bug fix # → Patch bump (0.1.0 → 0.1.1) +feat!: breaking change # → Major bump (0.1.0 → 1.0.0) +[alpha] experimental # → Alpha bump (0.1.0-alpha.1) +``` + +**Manual override:** Can skip version bump via workflow_dispatch + +--- + +#### `update-registry.yml` - Registry Auto-Update +**Triggers:** Push to main with .opencode changes +**Duration:** < 1 minute +**What it does:** +- ✅ Auto-detects new components +- ✅ Updates registry.json +- ✅ Commits changes to main + +--- + +#### `sync-docs.yml` - Documentation Sync +**Triggers:** Push to main with registry/component changes +**Duration:** Variable (depends on OpenCode) +**What it does:** +- ✅ Creates sync branch +- ✅ Opens issue for OpenCode to update docs +- ✅ Syncs component counts with registry + +--- + +### Integration Workflows + +#### `opencode.yml` - OpenCode Integration +**Triggers:** Issue comments with `/opencode` or `/oc` +**What it does:** +- ✅ Enables OpenCode AI assistance on issues +- ✅ Restricted to maintainers only + +--- + +## Workflow Philosophy + +### Design Principles + +1. **Fast Feedback** - PRs get results in < 2 minutes +2. **Fork-Friendly** - No push attempts to fork branches +3. **Cost-Effective** - No expensive AI tests on every PR +4. **Maintainer Control** - Manual overrides available +5. **Clear Communication** - Helpful error messages and guidance + +### What Changed (Dec 2025) + +**Before:** +- ❌ Expensive AI tests on every PR (15 min, costs money) +- ❌ Fork PRs failed with confusing errors +- ❌ Slow feedback loop + +**After:** +- ✅ Fast build checks (< 2 min, free) +- ✅ Fork-friendly workflows +- ✅ Manual AI testing when needed +- ✅ Auto-versioning preserved (moved to post-merge) + +See [_archive/README.md](_archive/README.md) for details on removed workflows. + +--- + +## For Contributors + +### What Runs on Your PR + +1. **Build Check** (`pr-checks.yml`) + - TypeScript compilation + - YAML validation + - Fast (< 2 min) + +2. **Registry Validation** (`validate-registry.yml`) + - Path validation + - Component detection + - Prompt validation + +### If Checks Fail + +See [EXTERNAL_PR_GUIDE.md](../EXTERNAL_PR_GUIDE.md) for detailed instructions. + +**Quick fixes:** +```bash +# Build errors +cd evals/framework && npm run build + +# Registry updates +./scripts/registry/auto-detect-components.sh --auto-add + +# Prompt issues +./scripts/prompts/use-prompt.sh default +``` + +--- + +## For Maintainers + +### Manual Testing + +```bash +# Test PR locally +gh pr checkout +cd evals/framework +npm install +npm run build + +# Run AI tests (optional) +npm run test:ci +``` + +### Manual Overrides + +**Skip validation:** +```bash +gh workflow run validate-registry.yml -f skip_validation=true +``` + +**Skip version bump:** +```bash +gh workflow run post-merge.yml -f skip_version_bump=true +``` + +### Version Management + +Version bumping is automatic based on commit messages. See [VERSION_BUMP_GUIDE.md](VERSION_BUMP_GUIDE.md). + +**Manual version bump:** +```bash +npm run version:bump:patch # 0.1.0 → 0.1.1 +npm run version:bump:minor # 0.1.0 → 0.2.0 +npm run version:bump:major # 0.1.0 → 1.0.0 +``` + +--- + +## Workflow Status + +| Workflow | Status | Purpose | Fork-Friendly | +|----------|--------|---------|---------------| +| pr-checks.yml | ✅ Active | Fast build validation | ✅ Yes | +| validate-registry.yml | ✅ Active | Registry validation | ✅ Yes | +| post-merge.yml | ✅ Active | Auto-versioning | N/A (main only) | +| update-registry.yml | ✅ Active | Registry auto-update | N/A (main only) | +| sync-docs.yml | ✅ Active | Doc sync | N/A (main only) | +| opencode.yml | ✅ Active | OpenCode integration | N/A (issues only) | +| test-agents.yml | 🗄️ Archived | Expensive AI tests | ❌ No | +| validate-test-suites.yml | 🗄️ Archived | Redundant validation | ✅ Yes | + +--- + +## Questions? + +- **Contributors:** See [EXTERNAL_PR_GUIDE.md](../EXTERNAL_PR_GUIDE.md) +- **Maintainers:** See [PROJECT_CLI_GUIDE.md](../PROJECT_CLI_GUIDE.md) +- **Version Bumping:** See [VERSION_BUMP_GUIDE.md](VERSION_BUMP_GUIDE.md) +- **Archived Workflows:** See [_archive/README.md](_archive/README.md) diff --git a/.github/workflows/_archive/README.md b/.github/workflows/_archive/README.md new file mode 100644 index 0000000..ffa27ab --- /dev/null +++ b/.github/workflows/_archive/README.md @@ -0,0 +1,160 @@ +# Archived Workflows + +These workflows were removed as part of the fork-friendly workflow simplification on 2025-12-08. + +## Why We Archived These Workflows + +### Problem Statement +The original workflows had several issues: +1. **Expensive AI tests on every PR** - Running full agent evaluations cost money and took 15+ minutes +2. **Fork PR failures** - External contributors experienced confusing workflow failures +3. **Over-engineering** - Too much automation for simple validation tasks +4. **Poor contributor experience** - Slow feedback and unclear error messages + +### Solution +We simplified the workflow architecture to be: +- **Fast** - Build checks complete in < 2 minutes +- **Fork-friendly** - No push attempts to fork branches +- **Cost-effective** - No AI tests on every PR +- **Clear** - Better error messages and contributor guidance + +--- + +## Archived Workflows + +### `test-agents.yml` (Archived: 2025-12-08) + +**What it did:** +- Ran full AI agent tests (OpenAgent and OpenCoder) on every PR +- Spawned OpenCode CLI and executed evaluation framework +- Auto-bumped versions after merge +- Created GitHub releases + +**Why removed:** +- ❌ Expensive (AI API calls on every PR) +- ❌ Slow (15 minutes per PR) +- ❌ Overkill for PR validation +- ❌ Failed on fork PRs due to permission issues + +**Replaced by:** +- ✅ `pr-checks.yml` - Fast build validation for PRs (< 2 min, free) +- ✅ `post-merge.yml` - Auto-versioning after merge (keeps the good automation!) +- ✅ Manual testing: `npm run test:ci` when needed + +**Key features preserved:** +- ✅ Auto-version bumping (moved to post-merge.yml) +- ✅ Automatic GitHub releases (moved to post-merge.yml) +- ✅ Conventional commit parsing (moved to post-merge.yml) + +--- + +### `validate-test-suites.yml` (Archived: 2025-12-08) + +**What it did:** +- Validated YAML test suite files +- Ran on every test file change +- Posted PR comments on validation failures + +**Why removed:** +- ❌ Redundant with build check +- ❌ Could be combined with other validation + +**Replaced by:** +- ✅ `pr-checks.yml` - Includes YAML validation as part of build check + +--- + +## New Workflow Architecture + +``` +.github/workflows/ +├── pr-checks.yml # Fast build validation (all PRs) +├── validate-registry.yml # Registry validation (fork-friendly) +├── post-merge.yml # Auto-version + releases (main only) +├── opencode.yml # OpenCode integration +├── sync-docs.yml # Doc sync +├── update-registry.yml # Registry updates +└── _archive/ + ├── README.md # This file + ├── test-agents.yml # Old expensive AI tests + └── validate-test-suites.yml # Old YAML validation +``` + +--- + +## Testing Philosophy + +### Before (Expensive) +- ❌ AI tests on every PR (15 min, costs money) +- ❌ Fork PRs failed with confusing errors +- ❌ Slow feedback loop +- ❌ Over-engineered for simple validation + +### After (Efficient) +- ✅ Fast build checks on PRs (< 2 min, free) +- ✅ Fork-friendly workflows +- ✅ Manual AI testing when needed +- ✅ Auto-versioning after merge (preserved!) +- ✅ Clear contributor guidance + +--- + +## For Maintainers + +### Running AI Tests Manually + +If you need to run the full agent tests: + +```bash +# Locally +npm run test:ci + +# Or specific agents +npm run test:openagent +npm run test:opencoder + +# With specific models +npm run test:openagent:claude +npm run test:openagent:grok +``` + +### Restoring Old Workflows (Not Recommended) + +If you need to restore these workflows for reference: + +```bash +# Copy back to workflows directory +cp .github/workflows/_archive/test-agents.yml .github/workflows/ + +# But consider: Do you really need expensive AI tests on every PR? +``` + +--- + +## Migration Notes + +**What changed for contributors:** +- ✅ Faster PR feedback (2 min vs 15 min) +- ✅ No confusing fork PR failures +- ✅ Clear instructions when fixes needed + +**What changed for maintainers:** +- ✅ Lower CI costs (no AI tests per PR) +- ✅ Same auto-versioning (moved to post-merge.yml) +- ✅ Same auto-releases (moved to post-merge.yml) +- ✅ Manual control over AI testing + +**What stayed the same:** +- ✅ Auto-version bumping based on conventional commits +- ✅ Automatic CHANGELOG updates +- ✅ Automatic GitHub releases +- ✅ Registry validation and auto-updates +- ✅ Documentation sync + +--- + +## Questions? + +See the main workflow documentation: +- [EXTERNAL_PR_GUIDE.md](../../EXTERNAL_PR_GUIDE.md) - For contributors +- [VERSION_BUMP_GUIDE.md](../VERSION_BUMP_GUIDE.md) - For version management diff --git a/.github/workflows/test-agents.yml b/.github/workflows/_archive/test-agents.yml similarity index 100% rename from .github/workflows/test-agents.yml rename to .github/workflows/_archive/test-agents.yml diff --git a/.github/workflows/_archive/validate-test-suites.yml b/.github/workflows/_archive/validate-test-suites.yml new file mode 100644 index 0000000..0639361 --- /dev/null +++ b/.github/workflows/_archive/validate-test-suites.yml @@ -0,0 +1,64 @@ +name: Validate Test Suites + +on: + push: + paths: + - 'evals/agents/*/config/**/*.json' + - 'evals/agents/*/tests/**/*.yaml' + - 'scripts/validation/validate-test-suites.sh' + - '.github/workflows/validate-test-suites.yml' + pull_request: + paths: + - 'evals/agents/*/config/**/*.json' + - 'evals/agents/*/tests/**/*.yaml' + - 'scripts/validation/validate-test-suites.sh' + - '.github/workflows/validate-test-suites.yml' + workflow_dispatch: + +jobs: + validate: + name: Validate Test Suite Definitions + runs-on: ubuntu-latest + + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: '20' + cache: 'npm' + cache-dependency-path: 'evals/framework/package-lock.json' + + - name: Install dependencies + run: | + cd evals/framework + npm ci + + - name: Validate all test suites + run: | + cd evals/framework + npm run validate:suites:all + + - name: Comment on PR (if validation failed) + if: failure() && github.event_name == 'pull_request' + uses: actions/github-script@v7 + with: + script: | + github.rest.issues.createComment({ + issue_number: context.issue.number, + owner: context.repo.owner, + repo: context.repo.repo, + body: '❌ **Test Suite Validation Failed**\n\nPlease check the test suite JSON files for errors. Run `npm run validate:suites` locally to see details.' + }) + + - name: Upload validation report + if: always() + uses: actions/upload-artifact@v4 + with: + name: suite-validation-report + path: | + evals/agents/*/config/**/*.json + scripts/validation/validate-test-suites.sh + retention-days: 7 diff --git a/.github/workflows/post-merge.yml b/.github/workflows/post-merge.yml new file mode 100644 index 0000000..82e1d41 --- /dev/null +++ b/.github/workflows/post-merge.yml @@ -0,0 +1,175 @@ +name: Post-Merge Automation + +on: + push: + branches: [main] + workflow_dispatch: + inputs: + skip_version_bump: + description: 'Skip version bump (maintainer override)' + required: false + type: boolean + default: false + +jobs: + check-trigger: + name: Check Trigger Type + runs-on: ubuntu-latest + outputs: + should_bump: ${{ steps.check.outputs.should_bump }} + steps: + - name: Checkout code + uses: actions/checkout@v4 + with: + fetch-depth: 2 + + - name: Determine if version bump needed + id: check + run: | + COMMIT_MSG="${{ github.event.head_commit.message }}" + + # Skip version bump commits to prevent loops + if echo "$COMMIT_MSG" | grep -qE "^\[skip ci\]|chore: bump version"; then + echo "should_bump=false" >> $GITHUB_OUTPUT + echo "Version bump commit detected - skipping to prevent loops" + else + echo "should_bump=true" >> $GITHUB_OUTPUT + echo "Will bump version based on commit message: $COMMIT_MSG" + fi + + auto-version-bump: + name: Auto Version Bump + runs-on: ubuntu-latest + needs: check-trigger + if: | + needs.check-trigger.outputs.should_bump == 'true' && + github.event.inputs.skip_version_bump != 'true' + permissions: + contents: write + + steps: + - name: Checkout code + uses: actions/checkout@v4 + with: + fetch-depth: 0 + token: ${{ secrets.GITHUB_TOKEN }} + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: '20' + + - name: Configure Git + run: | + git config user.name "github-actions[bot]" + git config user.email "github-actions[bot]@users.noreply.github.com" + + - name: Determine version bump type + id: bump-type + run: | + # Get the last commit message + COMMIT_MSG=$(git log -1 --pretty=%B) + + # Determine bump type from commit message + if echo "$COMMIT_MSG" | grep -qiE "^(feat|feature)\(.*\)!:|^BREAKING CHANGE:|^[a-z]+!:"; then + echo "type=major" >> $GITHUB_OUTPUT + echo "Detected BREAKING CHANGE - bumping major version" + elif echo "$COMMIT_MSG" | grep -qiE "^(feat|feature)(\(.*\))?:"; then + echo "type=minor" >> $GITHUB_OUTPUT + echo "Detected feature - bumping minor version" + elif echo "$COMMIT_MSG" | grep -qiE "^(fix|bugfix)(\(.*\))?:"; then + echo "type=patch" >> $GITHUB_OUTPUT + echo "Detected fix - bumping patch version" + elif echo "$COMMIT_MSG" | grep -qiE "^\[alpha\]"; then + echo "type=alpha" >> $GITHUB_OUTPUT + echo "Detected [alpha] tag - bumping alpha version" + elif echo "$COMMIT_MSG" | grep -qiE "^\[beta\]"; then + echo "type=beta" >> $GITHUB_OUTPUT + echo "Detected [beta] tag - bumping beta version" + elif echo "$COMMIT_MSG" | grep -qiE "^\[rc\]"; then + echo "type=rc" >> $GITHUB_OUTPUT + echo "Detected [rc] tag - bumping rc version" + else + echo "type=patch" >> $GITHUB_OUTPUT + echo "No specific type detected - defaulting to patch version bump" + fi + + - name: Bump version + run: | + BUMP_TYPE="${{ steps.bump-type.outputs.type }}" + + # Get current version + CURRENT_VERSION=$(cat VERSION) + echo "Current version: $CURRENT_VERSION" + + # Bump version in package.json + npm run version:bump:$BUMP_TYPE + + # Get new version + NEW_VERSION=$(cat VERSION) + echo "New version: $NEW_VERSION" + + # Update CHANGELOG.md + DATE=$(date +%Y-%m-%d) + COMMIT_MSG=$(git log -1 --pretty=%B) + + # Create changelog entry + cat > /tmp/changelog_entry.md << EOF + ## [$NEW_VERSION] - $DATE + + ### Changes + - $COMMIT_MSG + + EOF + + # Prepend to CHANGELOG.md (after the header) + if [ -f CHANGELOG.md ]; then + # Insert after the first occurrence of "## [" + awk '/^## \[/ && !found {print; system("cat /tmp/changelog_entry.md"); found=1; next} 1' CHANGELOG.md > /tmp/changelog_new.md + mv /tmp/changelog_new.md CHANGELOG.md + fi + + - name: Commit version bump + run: | + NEW_VERSION=$(cat VERSION) + + git add VERSION package.json CHANGELOG.md + git commit -m "chore: bump version to v$NEW_VERSION [skip ci]" + git tag "v$NEW_VERSION" + + - name: Push changes + run: | + git push origin main --tags + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + + - name: Create GitHub Release + run: | + NEW_VERSION=$(cat VERSION) + + # Extract changelog entry for this version + RELEASE_NOTES=$(awk '/^## \['"$NEW_VERSION"'\]/{flag=1; next} /^## \[/{flag=0} flag' CHANGELOG.md) + + # If no specific notes found, use commit message + if [ -z "$RELEASE_NOTES" ]; then + RELEASE_NOTES="Release v$NEW_VERSION" + fi + + # Create the release + gh release create "v$NEW_VERSION" \ + --title "v$NEW_VERSION" \ + --notes "$RELEASE_NOTES" \ + --latest + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + + - name: Summary + if: success() + run: | + NEW_VERSION=$(cat VERSION) + echo "## 🚀 Version Bumped to v$NEW_VERSION" >> $GITHUB_STEP_SUMMARY + echo "" >> $GITHUB_STEP_SUMMARY + echo "- ✅ Version updated in VERSION and package.json" >> $GITHUB_STEP_SUMMARY + echo "- ✅ CHANGELOG.md updated" >> $GITHUB_STEP_SUMMARY + echo "- ✅ Git tag created: v$NEW_VERSION" >> $GITHUB_STEP_SUMMARY + echo "- ✅ GitHub release published" >> $GITHUB_STEP_SUMMARY diff --git a/.github/workflows/pr-checks.yml b/.github/workflows/pr-checks.yml new file mode 100644 index 0000000..1424610 --- /dev/null +++ b/.github/workflows/pr-checks.yml @@ -0,0 +1,59 @@ +name: PR Checks + +on: + pull_request: + branches: [main, dev] + paths: + - 'evals/**' + - '.github/workflows/pr-checks.yml' + +jobs: + build-check: + name: Build & Validate + runs-on: ubuntu-latest + timeout-minutes: 5 + + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: '20' + cache: 'npm' + cache-dependency-path: 'evals/framework/package-lock.json' + + - name: Install dependencies + working-directory: evals/framework + run: npm ci + + - name: Build framework + working-directory: evals/framework + run: npm run build + + - name: Validate test suites + working-directory: evals/framework + run: npm run validate:suites:all + + - name: Summary + if: success() + run: | + echo "## ✅ Build Check Passed" >> $GITHUB_STEP_SUMMARY + echo "" >> $GITHUB_STEP_SUMMARY + echo "- ✅ TypeScript compilation successful" >> $GITHUB_STEP_SUMMARY + echo "- ✅ Test suite validation passed" >> $GITHUB_STEP_SUMMARY + echo "" >> $GITHUB_STEP_SUMMARY + echo "**Note:** Full agent tests are not run on PRs to save time and costs." >> $GITHUB_STEP_SUMMARY + echo "Maintainers can run \`npm run test:ci\` locally if needed." >> $GITHUB_STEP_SUMMARY + + - name: Failure summary + if: failure() + run: | + echo "## ❌ Build Check Failed" >> $GITHUB_STEP_SUMMARY + echo "" >> $GITHUB_STEP_SUMMARY + echo "Please check the logs above for details." >> $GITHUB_STEP_SUMMARY + echo "" >> $GITHUB_STEP_SUMMARY + echo "**Common fixes:**" >> $GITHUB_STEP_SUMMARY + echo "- TypeScript errors: Fix type issues in \`evals/framework/src/\`" >> $GITHUB_STEP_SUMMARY + echo "- YAML validation: Check test files in \`evals/agents/*/tests/\`" >> $GITHUB_STEP_SUMMARY diff --git a/.github/workflows/validate-registry.yml b/.github/workflows/validate-registry.yml index f5f031e..171f4d8 100644 --- a/.github/workflows/validate-registry.yml +++ b/.github/workflows/validate-registry.yml @@ -27,11 +27,23 @@ jobs: - name: Checkout PR branch uses: actions/checkout@v4 with: - # Use head SHA for forks (head_ref is empty), head_ref for internal PRs - ref: ${{ github.event.pull_request.head.sha || github.head_ref }} + # For forks: fetch from fork repo, for internal: use head_ref + repository: ${{ github.event.pull_request.head.repo.full_name }} + ref: ${{ github.event.pull_request.head.ref }} fetch-depth: 0 token: ${{ secrets.GITHUB_TOKEN }} + - name: Detect fork PR + id: fork_check + run: | + if [ "${{ github.event.pull_request.head.repo.full_name }}" != "${{ github.repository }}" ]; then + echo "is_fork=true" >> $GITHUB_OUTPUT + echo "🔀 Fork PR detected from: ${{ github.event.pull_request.head.repo.full_name }}" + else + echo "is_fork=false" >> $GITHUB_OUTPUT + echo "📝 Internal PR detected" + fi + - name: Install dependencies run: | sudo apt-get update @@ -127,10 +139,10 @@ jobs: exit 1 fi - - name: Commit registry updates + - name: Commit registry updates (Internal PRs only) if: | + steps.fork_check.outputs.is_fork == 'false' && steps.auto_detect.outputs.new_components == 'true' && - github.head_ref != '' && steps.validate_prompts.outputs.prompt_validation == 'passed' && steps.validate.outputs.validation == 'passed' run: | @@ -140,7 +152,7 @@ jobs: if ! git diff --quiet registry.json; then git add registry.json git commit -m "chore: auto-update registry with new components [skip ci]" - git push origin ${{ github.head_ref }} + git push origin ${{ github.event.pull_request.head.ref }} echo "## 🚀 Registry Updated" >> $GITHUB_STEP_SUMMARY echo "" >> $GITHUB_STEP_SUMMARY @@ -154,22 +166,49 @@ jobs: - name: Fork PR notice if: | + steps.fork_check.outputs.is_fork == 'true' && steps.auto_detect.outputs.new_components == 'true' && - github.head_ref == '' && steps.validate_prompts.outputs.prompt_validation == 'passed' && steps.validate.outputs.validation == 'passed' + uses: actions/github-script@v7 + with: + script: | + github.rest.issues.createComment({ + issue_number: context.issue.number, + owner: context.repo.owner, + repo: context.repo.repo, + body: `## 📝 Registry Update Needed + + Hi @${{ github.event.pull_request.user.login }}! 👋 + + New components were detected in your PR. Since this is a fork PR, I can't auto-commit the registry updates for security reasons. + + **Please run these commands locally:** + \`\`\`bash + ./scripts/registry/auto-detect-components.sh --auto-add + git add registry.json + git commit -m "chore: update registry" + git push + \`\`\` + + Once you push the updated registry, the checks will pass! ✅` + }); + + - name: Fork PR summary + if: steps.fork_check.outputs.is_fork == 'true' run: | - echo "## ⚠️ External PR - Manual Action Required" >> $GITHUB_STEP_SUMMARY + echo "## 🔀 Fork PR Detected" >> $GITHUB_STEP_SUMMARY echo "" >> $GITHUB_STEP_SUMMARY - echo "New components detected, but cannot auto-commit to fork." >> $GITHUB_STEP_SUMMARY + echo "This is an external contribution - thank you! 🎉" >> $GITHUB_STEP_SUMMARY echo "" >> $GITHUB_STEP_SUMMARY - echo "**Please run locally:**" >> $GITHUB_STEP_SUMMARY - echo "\`\`\`bash" >> $GITHUB_STEP_SUMMARY - echo "./scripts/registry/auto-detect-components.sh --auto-add" >> $GITHUB_STEP_SUMMARY - echo "git add registry.json" >> $GITHUB_STEP_SUMMARY - echo "git commit -m 'chore: update registry'" >> $GITHUB_STEP_SUMMARY - echo "git push" >> $GITHUB_STEP_SUMMARY - echo "\`\`\`" >> $GITHUB_STEP_SUMMARY + + if [ "${{ steps.auto_detect.outputs.new_components }}" == "true" ]; then + echo "⚠️ **Action Required:** New components detected" >> $GITHUB_STEP_SUMMARY + echo "" >> $GITHUB_STEP_SUMMARY + echo "A comment has been posted with instructions to update the registry." >> $GITHUB_STEP_SUMMARY + else + echo "✅ No registry updates needed" >> $GITHUB_STEP_SUMMARY + fi - name: Post validation summary if: always()