OpenWriterControle/.github/workflows/validate-registry.yml
Darren Hinde c8f7103cb6
refactor(evals): consolidate documentation and enhance test infrastructure (#56)
* feat(agents): implement category-based agent organization system

- Organize agents into domain categories (core, development, content, data, product, learning)
- Move 10 agents to category subdirectories with proper git rename tracking
- Update 13 subagents with category and type metadata in frontmatter
- Add category metadata files (0-category.json) documenting common patterns
- Implement local registry fallback in install script for offline development
- Add comprehensive validation suite with 15 automated tests (100% pass rate)
- Enhance registry validation with duplicate ID and consistency checks
- Update eval framework with intelligent path resolution (backward compatible)
- Archive legacy eval structure to _archive/ for reference
- Update all documentation to reflect category-based structure
- Bump version to 0.5.0 with accurate CHANGELOG

Technical Details:
- 23 agents organized (10 category agents, 13 subagents)
- 6 category directories created
- Path resolution supports both agent IDs and category paths
- Registry schema updated to v2.0.0
- 159 files changed, 872 insertions(+), 10738 deletions(-)

BREAKING CHANGE: Agent file paths now use category structure. Update references from .opencode/agent/openagent.md to .opencode/agent/core/openagent.md. Eval framework maintains backward compatibility via path resolution.

* refactor(evals): consolidate documentation and enhance test infrastructure

- Remove temporary project tracking files (PHASE_5_COMPLETE, PROJECT_COMPLETE, etc.)
- Consolidate evaluation framework docs into main README
- Enhance test execution with improved logging and multi-prompt support
- Move system-builder from core to meta category
- Add comprehensive test suites for openagent with organized structure
- Create evaluation test structure for all subagents
- Clean up archived workflows and redundant documentation
- Update registry to reflect new agent organization
- Add shared test templates and golden test patterns

* feat(ci): add manual trigger support for bot-created PRs in validate-registry workflow

- Add pr_number input to workflow_dispatch for manually triggering validation
- Fetch PR details dynamically when triggered manually
- Support both automatic (PR event) and manual (workflow_dispatch) triggers
- Enable validation of bot-created PRs like automated version bumps
- Update branch detection and push logic to handle both trigger types
- Add documentation explaining how to manually trigger for bot PRs

* feat(evals): add explicit context file validation to test framework

- Add expectedContextFiles field to test YAML schema for explicit context file specification
- Enhance context-loading-evaluator to support both auto-detect and explicit validation modes
- Update documentation with comprehensive guide and examples
- Clean up archived legacy test structure (90+ old test files)
- Add new example tests demonstrating explicit context validation
- Backward compatible with existing tests

* feat(evals): add multi-agent logging system and performance optimizations

Implement comprehensive multi-agent logging and performance improvements for the eval framework.

Task 01: Multi-Agent Logging System
- Add complete hierarchical logging module (evals/framework/src/logging/)
  - SessionTracker: tracks parent-child delegation hierarchies
  - MultiAgentLogger: pretty-prints logs with visual indentation
  - Formatters: box characters and emoji formatting
  - 37 passing unit tests (session-tracker, logger, integration)
- Integrate with SDK event stream handler
  - Hook into session.created, message.updated, message.part.updated events
  - Real-time child session detection via timestamp heuristics
  - Message deduplication for cleaner output
- Enable in debug mode only (<1% performance overhead)
- Add demo script and comprehensive documentation

Task 02: Performance Optimizations
- Reduce grace period from 5s to 2s (67% reduction, 10-20% faster tests)
- Add PerformanceMetricsEvaluator for bottleneck identification
  - Collects tool latencies, inference time, idle time
  - Provides full performance visibility
- Update smoke test to validate delegation (core feature)
  - Replace simple read test with multi-agent delegation test
  - Tests both parent and child agent functionality
  - Validates multi-agent logging system

Results:
- 37 unit tests passing
- Smoke test passing (95/100 score)
- Clean hierarchical logging output
- 10-20% faster test execution
- Full multi-agent visibility

Files changed: 16 files, +2678 lines
Time saved: ~6-8 days (completed in 6 hours vs 3-5 day estimate)

* feat(evals): show child agent execution in non-debug mode

- Enable MultiAgentLogger in both debug and non-debug modes
- Add verbose parameter to control output level
- Non-verbose mode shows concise child session lifecycle:
  - Child agent started message
  - Child agent completed with duration
- Verbose mode (--debug) shows full delegation hierarchy
- Update documentation to reflect new behavior

This provides visibility into delegation without overwhelming output,
giving confidence that child agents are actually running.

* feat(evals): improve delegation testing and behavior validation

Major improvements to eval framework:

Test Suite:
- Add 00-smoke-test.yaml (basic read operation)
- Rename 01-smoke-test.yaml → 02-delegation-test.yaml (delegation test)
- Add simple-responder test agent for delegation testing
- Add debug test: simple-subagent-call.yaml

Evaluators:
- Add agent-model-evaluator.ts for agent/model validation
- Enhance behavior-evaluator.ts with detailed task tool output
- Improve delegation-evaluator.ts with better evidence tracking

Schema & Execution:
- Add expectedAgent and expectedModel to test schema
- Improve test executor with better logging and error handling

Documentation:
- Update evals/README.md with new features and performance improvements
- Remove outdated MULTI_AGENT_LOGGING_COMPLETE.md

Registry:
- Add simple-responder test agent to registry

Fixes:
- Fix dashboard.sh path resolution

These changes provide better visibility into delegation, improved test
coverage, and clearer validation of agent behavior.

* chore: add GitHub Actions workflow and plugin documentation

- Add .github/workflows/evals/run-evaluations.yml for automated eval testing
- Add dev/ai-tools/opencode/plugins/Plugin-inspiration.md for plugin development reference
2025-12-18 09:52:11 +00:00

296 lines
13 KiB
YAML
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

name: Validate Registry on PR
# This workflow validates the registry.json and prompt library structure on all PRs.
#
# For bot-created PRs (like automated version bumps), the workflow won't trigger automatically
# due to GitHub's security restrictions. In those cases, you can manually trigger this workflow:
#
# 1. Go to Actions > Validate Registry on PR > Run workflow
# 2. Enter the PR number (e.g., 57)
# 3. Click "Run workflow"
#
# This will run the validation checks and report the status to the PR.
on:
pull_request:
branches:
- main
- dev
# Removed paths filter - this check is required by repository ruleset
# so it must run on ALL PRs to prevent blocking merges
workflow_dispatch:
inputs:
pr_number:
description: 'PR number to validate (for manual runs on bot-created PRs)'
required: false
type: number
skip_validation:
description: 'Skip validation checks (maintainer override)'
required: false
type: boolean
default: false
permissions:
contents: write
pull-requests: write
jobs:
validate-and-update:
runs-on: ubuntu-latest
steps:
- name: Get PR details (for manual runs)
if: github.event_name == 'workflow_dispatch' && github.event.inputs.pr_number != ''
id: get_pr
run: |
PR_DATA=$(gh pr view ${{ github.event.inputs.pr_number }} --json headRefName,headRepository,headRepositoryOwner)
echo "head_ref=$(echo $PR_DATA | jq -r '.headRefName')" >> $GITHUB_OUTPUT
echo "head_repo=$(echo $PR_DATA | jq -r '.headRepositoryOwner.login + "/" + .headRepository.name')" >> $GITHUB_OUTPUT
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Checkout PR branch
uses: actions/checkout@v4
with:
# For manual runs: use PR details from get_pr step
# For PR events: use event data
repository: ${{ github.event_name == 'workflow_dispatch' && steps.get_pr.outputs.head_repo || github.event.pull_request.head.repo.full_name }}
ref: ${{ github.event_name == 'workflow_dispatch' && steps.get_pr.outputs.head_ref || github.event.pull_request.head.ref }}
fetch-depth: 0
token: ${{ secrets.GITHUB_TOKEN }}
- name: Detect fork PR
id: fork_check
run: |
# For manual runs, use the fetched PR data
if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then
HEAD_REPO="${{ steps.get_pr.outputs.head_repo }}"
else
HEAD_REPO="${{ github.event.pull_request.head.repo.full_name }}"
fi
if [ "$HEAD_REPO" != "${{ github.repository }}" ]; then
echo "is_fork=true" >> $GITHUB_OUTPUT
echo "🔀 Fork PR detected from: $HEAD_REPO"
else
echo "is_fork=false" >> $GITHUB_OUTPUT
echo "📝 Internal PR detected"
fi
- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get install -y jq
- name: Make scripts executable
run: |
chmod +x scripts/registry/validate-registry.sh
chmod +x scripts/registry/auto-detect-components.sh
chmod +x scripts/registry/register-component.sh
chmod +x scripts/prompts/validate-pr.sh
- name: Auto-detect new components
id: auto_detect
run: |
echo "## 🔍 Auto-Detection Results" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
# Run auto-detect in dry-run mode first to see what would be added
if ./scripts/registry/auto-detect-components.sh --dry-run > /tmp/detect-output.txt 2>&1; then
cat /tmp/detect-output.txt >> $GITHUB_STEP_SUMMARY
# Check if new components were found
if grep -q "Found.*new component" /tmp/detect-output.txt; then
echo "new_components=true" >> $GITHUB_OUTPUT
echo "" >> $GITHUB_STEP_SUMMARY
echo "⚠️ New components detected - will auto-add to registry" >> $GITHUB_STEP_SUMMARY
else
echo "new_components=false" >> $GITHUB_OUTPUT
echo "✅ No new components found" >> $GITHUB_STEP_SUMMARY
fi
else
echo "new_components=false" >> $GITHUB_OUTPUT
echo "❌ Auto-detection failed" >> $GITHUB_STEP_SUMMARY
fi
- name: Add new components to registry
if: steps.auto_detect.outputs.new_components == 'true'
run: |
echo "## 📝 Adding New Components" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
./scripts/registry/auto-detect-components.sh --auto-add | tee -a $GITHUB_STEP_SUMMARY
- name: Validate prompt library structure
id: validate_prompts
run: |
echo "## 🔍 Prompt Library Validation" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
if ./scripts/prompts/validate-pr.sh > /tmp/prompt-validation.txt 2>&1; then
echo "prompt_validation=passed" >> $GITHUB_OUTPUT
echo "✅ Prompt library structure is valid!" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
cat /tmp/prompt-validation.txt >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
else
echo "prompt_validation=failed" >> $GITHUB_OUTPUT
echo "❌ Prompt library validation failed!" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
cat /tmp/prompt-validation.txt >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "**Architecture:**" >> $GITHUB_STEP_SUMMARY
echo "- Agent files (.opencode/agent/**/*.md) = Canonical defaults" >> $GITHUB_STEP_SUMMARY
echo "- Prompt variants (.opencode/prompts/<agent>/<model>.md) = Model-specific" >> $GITHUB_STEP_SUMMARY
echo "- default.md files should NOT exist" >> $GITHUB_STEP_SUMMARY
echo "- Agents organized in category subdirectories (core/, development/, content/, etc.)" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "See [CONTRIBUTING.md](docs/contributing/CONTRIBUTING.md) for details" >> $GITHUB_STEP_SUMMARY
exit 1
fi
- name: Validate registry
id: validate
run: |
echo "## ✅ Registry Validation" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
if ./scripts/registry/validate-registry.sh -v > /tmp/validation-output.txt 2>&1; then
echo "validation=passed" >> $GITHUB_OUTPUT
echo "✅ All registry paths are valid!" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
tail -20 /tmp/validation-output.txt >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
else
echo "validation=failed" >> $GITHUB_OUTPUT
echo "❌ Registry validation failed!" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
cat /tmp/validation-output.txt >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
exit 1
fi
- name: Commit registry updates (Internal PRs only)
if: |
steps.fork_check.outputs.is_fork == 'false' &&
steps.auto_detect.outputs.new_components == 'true' &&
steps.validate_prompts.outputs.prompt_validation == 'passed' &&
steps.validate.outputs.validation == 'passed'
run: |
git config --local user.email "github-actions[bot]@users.noreply.github.com"
git config --local user.name "github-actions[bot]"
if ! git diff --quiet registry.json; then
git add registry.json
git commit -m "chore: auto-update registry with new components [skip ci]"
# For manual runs, use the fetched branch name
BRANCH_NAME="${{ github.event_name == 'workflow_dispatch' && steps.get_pr.outputs.head_ref || github.event.pull_request.head.ref }}"
git push origin "$BRANCH_NAME"
echo "## 🚀 Registry Updated" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "Registry has been automatically updated with new components." >> $GITHUB_STEP_SUMMARY
echo "Changes have been pushed to this PR branch." >> $GITHUB_STEP_SUMMARY
else
echo "## ℹ️ No Changes to Commit" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "Registry is already up to date." >> $GITHUB_STEP_SUMMARY
fi
- name: Fork PR notice
if: |
steps.fork_check.outputs.is_fork == 'true' &&
steps.auto_detect.outputs.new_components == 'true' &&
steps.validate_prompts.outputs.prompt_validation == 'passed' &&
steps.validate.outputs.validation == 'passed'
uses: actions/github-script@v7
with:
script: |
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: `## 📝 Registry Update Needed
Hi @${{ github.event.pull_request.user.login }}! 👋
New components were detected in your PR. Since this is a fork PR, I can't auto-commit the registry updates for security reasons.
**Please run these commands locally:**
\`\`\`bash
./scripts/registry/auto-detect-components.sh --auto-add
git add registry.json
git commit -m "chore: update registry"
git push
\`\`\`
Once you push the updated registry, the checks will pass! ✅`
});
- name: Fork PR summary
if: steps.fork_check.outputs.is_fork == 'true'
run: |
echo "## 🔀 Fork PR Detected" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "This is an external contribution - thank you! 🎉" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
if [ "${{ steps.auto_detect.outputs.new_components }}" == "true" ]; then
echo "⚠️ **Action Required:** New components detected" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "A comment has been posted with instructions to update the registry." >> $GITHUB_STEP_SUMMARY
else
echo "✅ No registry updates needed" >> $GITHUB_STEP_SUMMARY
fi
- name: Post validation summary
if: always()
run: |
echo "" >> $GITHUB_STEP_SUMMARY
echo "---" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
PROMPT_VALID="${{ steps.validate_prompts.outputs.prompt_validation }}"
REGISTRY_VALID="${{ steps.validate.outputs.validation }}"
if [ "$PROMPT_VALID" = "passed" ] && [ "$REGISTRY_VALID" = "passed" ]; then
echo "### ✅ All Validations Passed" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "- ✅ Prompt library structure is valid" >> $GITHUB_STEP_SUMMARY
echo "- ✅ Registry paths are valid" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "This PR is ready for review!" >> $GITHUB_STEP_SUMMARY
else
echo "### ❌ Validation Failed" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
if [ "$PROMPT_VALID" != "passed" ]; then
echo "- ❌ Prompt library validation failed" >> $GITHUB_STEP_SUMMARY
else
echo "- ✅ Prompt library validation passed" >> $GITHUB_STEP_SUMMARY
fi
if [ "$REGISTRY_VALID" != "passed" ]; then
echo "- ❌ Registry validation failed" >> $GITHUB_STEP_SUMMARY
else
echo "- ✅ Registry validation passed" >> $GITHUB_STEP_SUMMARY
fi
echo "" >> $GITHUB_STEP_SUMMARY
echo "Please fix the issues above before merging." >> $GITHUB_STEP_SUMMARY
fi
- name: Fail if validation failed
if: |
(steps.validate_prompts.outputs.prompt_validation == 'failed' || steps.validate.outputs.validation == 'failed') &&
github.event.inputs.skip_validation != 'true'
run: |
echo "❌ Validation failed - blocking PR merge"
echo "Maintainer can override by running workflow manually with 'skip_validation' enabled"
exit 1