diff --git a/.github/EXTERNAL_PR_GUIDE.md b/.github/EXTERNAL_PR_GUIDE.md deleted file mode 100644 index cf20f12..0000000 --- a/.github/EXTERNAL_PR_GUIDE.md +++ /dev/null @@ -1,174 +0,0 @@ -# External PR Guide - -This guide explains how GitHub Actions workflows behave for external contributors (fork PRs). - -## For External Contributors (Fork PRs) - -When you submit a PR from a fork, workflows run but **cannot auto-commit fixes** due to GitHub security restrictions. - -### What Runs Automatically - -✅ **Fast Build Check (< 2 minutes)** -- TypeScript compilation check -- YAML test suite validation -- Quick feedback on code quality - -✅ **Registry Validation** -- Checks if `registry.json` matches actual files -- Validates all paths are correct -- Checks prompts use defaults -- **Cannot auto-commit** (you'll need to fix locally) - -❌ **What Doesn't Run on PRs** -- Full AI agent tests (too expensive, run manually by maintainers) -- Auto-version bumping (happens after merge) - -### What You Need to Do - -If the validation workflow reports issues, you'll need to fix them locally: - -#### 1. Build Failures - -If TypeScript compilation fails: - -```bash -# Build locally to see errors -cd evals/framework -npm install -npm run build - -# Fix the errors, then commit -git add . -git commit -m "fix: resolve build errors" -git push -``` - -#### 2. Registry Issues - -If new components are detected (you'll get a comment on your PR): - -```bash -# Run auto-detect locally -./scripts/registry/auto-detect-components.sh --auto-add - -# Commit the updated registry -git add registry.json -git commit -m "chore: update registry with new components" -git push -``` - -#### 3. Prompt Issues - -If prompts don't match defaults: - -```bash -# Restore default prompts -./scripts/prompts/use-prompt.sh default - -# Commit the changes -git add .opencode/agent/ -git commit -m "chore: restore default prompts" -git push -``` - -#### 4. Path Issues - -If registry paths are invalid: - -```bash -# Validate and see suggestions -./scripts/validate-registry.sh --fix - -# Manually fix paths in registry.json -# Then commit -git add registry.json -git commit -m "fix: correct registry paths" -git push -``` - -### What Happens Next - -1. **Fix Issues Locally**: If validation fails, fix issues and push -2. **Fast Feedback**: Build checks complete in < 2 minutes -3. **Maintainer Review**: A maintainer will review your code -4. **Merge**: Once everything passes, your PR will be merged! -5. **Auto-Release**: After merge, version is auto-bumped and release created (you don't need to do anything!) - ---- - -## For Maintainers - -### Handling External PRs - -External PRs run normally but **cannot auto-commit** due to GitHub security (can't push to fork branches). - -### Simple Overrides - -If you need to override checks: - -**Skip Validation:** -1. Go to **Actions** → **Validate Registry on PR** -2. Click **"Run workflow"** -3. Enable **"Skip validation checks"** -4. Run on the PR branch - -**Skip Version Bump:** -1. Go to **Actions** → **Post-Merge Automation** -2. Click **"Run workflow"** -3. Enable **"Skip version bump"** -4. Run manually - -### Workflow Behavior - -| PR Type | Build Check | Registry Validation | Auto-Commit | AI Tests | -|---------|------------|-------------------|-------------|----------| -| **Internal PR** (branch) | ✅ Yes | ✅ Yes | ✅ Yes | ❌ No* | -| **External PR** (fork) | ✅ Yes | ✅ Yes | ❌ No** | ❌ No* | - -*AI tests don't run on PRs (too expensive) - maintainers run manually if needed -**Cannot push to fork branches - contributor must fix locally - -### Manual Testing External PRs - -If you want to test an external PR locally: - -```bash -# Fetch the PR -gh pr checkout - -# Run build check -cd evals/framework -npm install -npm run build -npm run validate:suites:all - -# Run validation -./scripts/registry/validate-registry.sh -v -./scripts/prompts/validate-pr.sh - -# Auto-fix registry if needed -./scripts/registry/auto-detect-components.sh --auto-add - -# Run AI tests (optional - only if needed) -npm run test:ci - -# If everything passes, approve and merge -gh pr review --approve -gh pr merge -``` - -**Note:** AI tests are optional for PRs. The build check is usually sufficient. - ---- - -## Workflow Files - -- **`.github/workflows/pr-checks.yml`**: Fast build validation (< 2 min) -- **`.github/workflows/validate-registry.yml`**: Registry validation with override -- **`.github/workflows/post-merge.yml`**: Auto-versioning after merge (main only) - -## Questions? - -- **Contributors**: See [CONTRIBUTING.md](../docs/contributing/CONTRIBUTING.md) -- **Maintainers**: See [WORKFLOW_GUIDE.md](../WORKFLOW_GUIDE.md) -- **Issues**: Open a [GitHub Issue](https://github.com/darrenhinde/OpenAgents/issues) diff --git a/.github/workflows/README.md b/.github/workflows/README.md deleted file mode 100644 index 3c2f37f..0000000 --- a/.github/workflows/README.md +++ /dev/null @@ -1,207 +0,0 @@ -# Workflow Guide - -This directory contains GitHub Actions workflows for automated testing, validation, and releases. - -## Workflow Overview - -### For Contributors (PR Workflows) - -#### `pr-checks.yml` - Fast Build Validation -**Triggers:** Pull requests to main/dev with evals changes -**Duration:** < 2 minutes -**What it does:** -- ✅ TypeScript compilation check -- ✅ YAML test suite validation -- ✅ Fast feedback on code quality - -**Fork-friendly:** ✅ Yes - read-only checks - ---- - -#### `validate-registry.yml` - Registry Validation -**Triggers:** Pull requests to main/dev -**Duration:** < 1 minute -**What it does:** -- ✅ Validates registry.json paths -- ✅ Auto-detects new components -- ✅ Validates prompts use defaults -- ✅ **Internal PRs:** Auto-commits registry updates -- ✅ **Fork PRs:** Posts comment with instructions - -**Fork-friendly:** ✅ Yes - detects forks and provides guidance - ---- - -### For Maintainers (Post-Merge Workflows) - -#### `post-merge.yml` - Auto-Versioning & Releases -**Triggers:** Push to main branch -**Duration:** < 1 minute -**What it does:** -- ✅ Auto-bumps version based on conventional commits -- ✅ Updates CHANGELOG.md -- ✅ Creates git tag -- ✅ Publishes GitHub release - -**Conventional Commit Examples:** -```bash -feat: add new feature # → Minor bump (0.1.0 → 0.2.0) -fix: bug fix # → Patch bump (0.1.0 → 0.1.1) -feat!: breaking change # → Major bump (0.1.0 → 1.0.0) -[alpha] experimental # → Alpha bump (0.1.0-alpha.1) -``` - -**Manual override:** Can skip version bump via workflow_dispatch - ---- - -#### `update-registry.yml` - Registry Auto-Update -**Triggers:** Push to main with .opencode changes -**Duration:** < 1 minute -**What it does:** -- ✅ Auto-detects new components -- ✅ Updates registry.json -- ✅ Commits changes to main - ---- - -#### `sync-docs.yml` - Documentation Sync -**Triggers:** Push to main with registry/component changes -**Duration:** Variable (depends on OpenCode) -**What it does:** -- ✅ Creates sync branch -- ✅ Opens issue for OpenCode to update docs -- ✅ Syncs component counts with registry - ---- - -### Integration Workflows - -#### `opencode.yml` - OpenCode Integration -**Triggers:** Issue comments with `/opencode` or `/oc` -**What it does:** -- ✅ Enables OpenCode AI assistance on issues -- ✅ Restricted to maintainers only - ---- - -## Workflow Philosophy - -### Design Principles - -1. **Fast Feedback** - PRs get results in < 2 minutes -2. **Fork-Friendly** - No push attempts to fork branches -3. **Cost-Effective** - No expensive AI tests on every PR -4. **Maintainer Control** - Manual overrides available -5. **Clear Communication** - Helpful error messages and guidance - -### What Changed (Dec 2025) - -**Before:** -- ❌ Expensive AI tests on every PR (15 min, costs money) -- ❌ Fork PRs failed with confusing errors -- ❌ Slow feedback loop - -**After:** -- ✅ Fast build checks (< 2 min, free) -- ✅ Fork-friendly workflows -- ✅ Manual AI testing when needed -- ✅ Auto-versioning preserved (moved to post-merge) - -See [_archive/README.md](_archive/README.md) for details on removed workflows. - ---- - -## For Contributors - -### What Runs on Your PR - -1. **Build Check** (`pr-checks.yml`) - - TypeScript compilation - - YAML validation - - Fast (< 2 min) - -2. **Registry Validation** (`validate-registry.yml`) - - Path validation - - Component detection - - Prompt validation - -### If Checks Fail - -See [EXTERNAL_PR_GUIDE.md](../EXTERNAL_PR_GUIDE.md) for detailed instructions. - -**Quick fixes:** -```bash -# Build errors -cd evals/framework && npm run build - -# Registry updates -./scripts/registry/auto-detect-components.sh --auto-add - -# Prompt issues -./scripts/prompts/use-prompt.sh default -``` - ---- - -## For Maintainers - -### Manual Testing - -```bash -# Test PR locally -gh pr checkout -cd evals/framework -npm install -npm run build - -# Run AI tests (optional) -npm run test:ci -``` - -### Manual Overrides - -**Skip validation:** -```bash -gh workflow run validate-registry.yml -f skip_validation=true -``` - -**Skip version bump:** -```bash -gh workflow run post-merge.yml -f skip_version_bump=true -``` - -### Version Management - -Version bumping is automatic based on commit messages. See [VERSION_BUMP_GUIDE.md](VERSION_BUMP_GUIDE.md). - -**Manual version bump:** -```bash -npm run version:bump:patch # 0.1.0 → 0.1.1 -npm run version:bump:minor # 0.1.0 → 0.2.0 -npm run version:bump:major # 0.1.0 → 1.0.0 -``` - ---- - -## Workflow Status - -| Workflow | Status | Purpose | Fork-Friendly | -|----------|--------|---------|---------------| -| pr-checks.yml | ✅ Active | Fast build validation | ✅ Yes | -| validate-registry.yml | ✅ Active | Registry validation | ✅ Yes | -| post-merge.yml | ✅ Active | Auto-versioning | N/A (main only) | -| update-registry.yml | ✅ Active | Registry auto-update | N/A (main only) | -| sync-docs.yml | ✅ Active | Doc sync | N/A (main only) | -| opencode.yml | ✅ Active | OpenCode integration | N/A (issues only) | -| test-agents.yml | 🗄️ Archived | Expensive AI tests | ❌ No | -| validate-test-suites.yml | 🗄️ Archived | Redundant validation | ✅ Yes | - ---- - -## Questions? - -- **Contributors:** See [EXTERNAL_PR_GUIDE.md](../EXTERNAL_PR_GUIDE.md) -- **Maintainers:** See [PROJECT_CLI_GUIDE.md](../PROJECT_CLI_GUIDE.md) -- **Version Bumping:** See [VERSION_BUMP_GUIDE.md](VERSION_BUMP_GUIDE.md) -- **Archived Workflows:** See [_archive/README.md](_archive/README.md) diff --git a/.github/workflows/pr-checks.yml b/.github/workflows/pr-checks.yml index 1424610..e2bc3a5 100644 --- a/.github/workflows/pr-checks.yml +++ b/.github/workflows/pr-checks.yml @@ -3,15 +3,218 @@ name: PR Checks on: pull_request: branches: [main, dev] - paths: - - 'evals/**' - - '.github/workflows/pr-checks.yml' + types: [opened, edited, synchronize, reopened] jobs: + pr-title-check: + name: Validate PR Title + runs-on: ubuntu-latest + outputs: + title-valid: ${{ steps.validate.outputs.valid }} + + steps: + - name: Check PR title format + id: validate + uses: actions/github-script@v7 + with: + script: | + const prTitle = context.payload.pull_request.title; + + // Conventional commit patterns + const patterns = { + feat: /^feat(\(.+\))?!?:\s.+/, + fix: /^fix(\(.+\))?!?:\s.+/, + docs: /^docs(\(.+\))?:\s.+/, + style: /^style(\(.+\))?:\s.+/, + refactor: /^refactor(\(.+\))?:\s.+/, + perf: /^perf(\(.+\))?:\s.+/, + test: /^test(\(.+\))?:\s.+/, + chore: /^chore(\(.+\))?:\s.+/, + ci: /^ci(\(.+\))?:\s.+/, + build: /^build(\(.+\))?:\s.+/, + revert: /^revert(\(.+\))?:\s.+/, + alpha: /^\[alpha\]\s.+/, + beta: /^\[beta\]\s.+/, + rc: /^\[rc\]\s.+/ + }; + + // Check if title matches any pattern + const matchedType = Object.entries(patterns).find(([type, pattern]) => + pattern.test(prTitle) + ); + + if (!matchedType) { + const validExamples = [ + '✅ feat(evals): add new evaluator', + '✅ fix(agents): correct delegation logic', + '✅ docs(readme): update installation guide', + '✅ test(evals): add execution-balance tests', + '✅ chore(deps): update dependencies', + '✅ feat!: breaking API change', + '✅ [alpha] experimental feature' + ]; + + const message = ` + ## ❌ Invalid PR Title Format + + **Current title:** \`${prTitle}\` + + ### Required Format + PR titles must follow [Conventional Commits](https://www.conventionalcommits.org/) format: + + \`\`\` + (): + \`\`\` + + ### Valid Types + - **feat** - New feature (triggers minor version bump: 0.3.0 → 0.4.0) + - **fix** - Bug fix (triggers patch version bump: 0.3.0 → 0.3.1) + - **docs** - Documentation changes (triggers patch bump) + - **test** - Test additions/changes (triggers patch bump) + - **refactor** - Code refactoring (triggers patch bump) + - **chore** - Maintenance tasks (triggers patch bump) + - **ci** - CI/CD changes (triggers patch bump) + - **perf** - Performance improvements (triggers patch bump) + - **style** - Code style changes (triggers patch bump) + - **build** - Build system changes (triggers patch bump) + - **revert** - Revert previous commit (triggers patch bump) + + ### Breaking Changes + - **feat!:** or **fix!:** - Breaking change (triggers major version bump: 0.3.0 → 1.0.0) + - **BREAKING CHANGE:** in description + + ### Pre-release Tags + - **[alpha]** - Alpha release (0.3.0 → 0.3.1-alpha.1) + - **[beta]** - Beta release (0.3.0 → 0.3.1-beta.1) + - **[rc]** - Release candidate (0.3.0 → 0.3.1-rc.1) + + ### Valid Examples + ${validExamples.map(ex => `- ${ex}`).join('\n')} + + ### Why This Matters + - ✅ Enables automatic semantic versioning + - ✅ Generates meaningful changelogs + - ✅ Makes commit history searchable + - ✅ Clarifies the impact of changes + + ### How to Fix + Edit your PR title to match the format above. + `; + + core.setFailed(message); + + // Also post as a comment + await github.rest.issues.createComment({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: context.payload.pull_request.number, + body: message + }); + } else { + const [type] = matchedType; + let versionBump = 'patch (0.3.0 → 0.3.1)'; + + if (type === 'feat' && prTitle.includes('!')) { + versionBump = 'major (0.3.0 → 1.0.0) - BREAKING CHANGE'; + } else if (type === 'feat') { + versionBump = 'minor (0.3.0 → 0.4.0)'; + } else if (type === 'fix' && prTitle.includes('!')) { + versionBump = 'major (0.3.0 → 1.0.0) - BREAKING CHANGE'; + } else if (type === 'alpha') { + versionBump = 'alpha (0.3.0 → 0.3.1-alpha.1)'; + } else if (type === 'beta') { + versionBump = 'beta (0.3.0 → 0.3.1-beta.1)'; + } else if (type === 'rc') { + versionBump = 'rc (0.3.0 → 0.3.1-rc.1)'; + } + + const message = ` + ## ✅ PR Title Valid + + **Title:** \`${prTitle}\` + **Type:** \`${type}\` + **Version bump:** ${versionBump} + + When this PR is merged using **"Squash and Merge"**, the version will be automatically bumped. + `; + + core.info(message); + + // Set output for summary + core.summary + .addHeading('✅ PR Title Validation Passed', 2) + .addRaw(`**Title:** \`${prTitle}\``) + .addBreak() + .addRaw(`**Type:** \`${type}\``) + .addBreak() + .addRaw(`**Version bump:** ${versionBump}`) + .write(); + + // Set output for dependent jobs + core.setOutput('valid', 'true'); + } + + check-changes: + name: Detect Changed Files + runs-on: ubuntu-latest + needs: pr-title-check + if: needs.pr-title-check.outputs.title-valid == 'true' + outputs: + has-evals: ${{ steps.filter.outputs.evals }} + has-docs: ${{ steps.filter.outputs.docs }} + has-workflows: ${{ steps.filter.outputs.workflows }} + + steps: + - name: Checkout code + uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Check changed files + id: filter + run: | + # Get list of changed files + git fetch origin ${{ github.base_ref }} + CHANGED_FILES=$(git diff --name-only origin/${{ github.base_ref }}...HEAD) + + echo "Changed files:" + echo "$CHANGED_FILES" + + # Check for evals changes + if echo "$CHANGED_FILES" | grep -q "^evals/"; then + echo "has-evals=true" >> $GITHUB_OUTPUT + echo "✅ Evals changes detected" + else + echo "has-evals=false" >> $GITHUB_OUTPUT + echo "ℹ️ No evals changes" + fi + + # Check for docs changes + if echo "$CHANGED_FILES" | grep -q "^docs/"; then + echo "has-docs=true" >> $GITHUB_OUTPUT + echo "✅ Docs changes detected" + else + echo "has-docs=false" >> $GITHUB_OUTPUT + echo "ℹ️ No docs changes" + fi + + # Check for workflow changes + if echo "$CHANGED_FILES" | grep -q "^.github/workflows/"; then + echo "has-workflows=true" >> $GITHUB_OUTPUT + echo "✅ Workflow changes detected" + else + echo "has-workflows=false" >> $GITHUB_OUTPUT + echo "ℹ️ No workflow changes" + fi + build-check: name: Build & Validate runs-on: ubuntu-latest timeout-minutes: 5 + needs: [pr-title-check, check-changes] + if: | + needs.pr-title-check.outputs.title-valid == 'true' && + needs.check-changes.outputs.has-evals == 'true' steps: - name: Checkout code @@ -57,3 +260,64 @@ jobs: echo "**Common fixes:**" >> $GITHUB_STEP_SUMMARY echo "- TypeScript errors: Fix type issues in \`evals/framework/src/\`" >> $GITHUB_STEP_SUMMARY echo "- YAML validation: Check test files in \`evals/agents/*/tests/\`" >> $GITHUB_STEP_SUMMARY + + summary: + name: PR Checks Summary + runs-on: ubuntu-latest + needs: [pr-title-check, check-changes, build-check] + if: always() + + steps: + - name: Generate summary + run: | + echo "## 📊 PR Checks Summary" >> $GITHUB_STEP_SUMMARY + echo "" >> $GITHUB_STEP_SUMMARY + + # PR Title Check + if [ "${{ needs.pr-title-check.result }}" == "success" ]; then + echo "✅ **PR Title:** Valid conventional commit format" >> $GITHUB_STEP_SUMMARY + else + echo "❌ **PR Title:** Invalid format - please fix" >> $GITHUB_STEP_SUMMARY + fi + + # Changed Files Detection + if [ "${{ needs.check-changes.result }}" == "success" ]; then + echo "✅ **Changed Files:** Detected successfully" >> $GITHUB_STEP_SUMMARY + + if [ "${{ needs.check-changes.outputs.has-evals }}" == "true" ]; then + echo " - 📦 Evals changes detected" >> $GITHUB_STEP_SUMMARY + fi + + if [ "${{ needs.check-changes.outputs.has-docs }}" == "true" ]; then + echo " - 📚 Docs changes detected" >> $GITHUB_STEP_SUMMARY + fi + + if [ "${{ needs.check-changes.outputs.has-workflows }}" == "true" ]; then + echo " - ⚙️ Workflow changes detected" >> $GITHUB_STEP_SUMMARY + fi + else + echo "⏭️ **Changed Files:** Skipped (title validation failed)" >> $GITHUB_STEP_SUMMARY + fi + + # Build Check + if [ "${{ needs.build-check.result }}" == "success" ]; then + echo "✅ **Build & Validate:** Passed" >> $GITHUB_STEP_SUMMARY + elif [ "${{ needs.build-check.result }}" == "skipped" ]; then + echo "⏭️ **Build & Validate:** Skipped (no evals changes)" >> $GITHUB_STEP_SUMMARY + elif [ "${{ needs.build-check.result }}" == "failure" ]; then + echo "❌ **Build & Validate:** Failed - check logs" >> $GITHUB_STEP_SUMMARY + fi + + echo "" >> $GITHUB_STEP_SUMMARY + + # Overall status + if [ "${{ needs.pr-title-check.result }}" == "success" ] && \ + ([ "${{ needs.build-check.result }}" == "success" ] || [ "${{ needs.build-check.result }}" == "skipped" ]); then + echo "### ✅ All Required Checks Passed!" >> $GITHUB_STEP_SUMMARY + echo "" >> $GITHUB_STEP_SUMMARY + echo "This PR is ready for review." >> $GITHUB_STEP_SUMMARY + else + echo "### ❌ Some Checks Failed" >> $GITHUB_STEP_SUMMARY + echo "" >> $GITHUB_STEP_SUMMARY + echo "Please fix the failing checks before merging." >> $GITHUB_STEP_SUMMARY + fi diff --git a/.github/workflows/validate-registry.yml b/.github/workflows/validate-registry.yml index 171f4d8..e265189 100644 --- a/.github/workflows/validate-registry.yml +++ b/.github/workflows/validate-registry.yml @@ -88,31 +88,33 @@ jobs: ./scripts/registry/auto-detect-components.sh --auto-add | tee -a $GITHUB_STEP_SUMMARY - - name: Validate prompts use defaults + - name: Validate prompt library structure id: validate_prompts run: | - echo "## 🔍 Prompt Validation" >> $GITHUB_STEP_SUMMARY + echo "## 🔍 Prompt Library Validation" >> $GITHUB_STEP_SUMMARY echo "" >> $GITHUB_STEP_SUMMARY if ./scripts/prompts/validate-pr.sh > /tmp/prompt-validation.txt 2>&1; then echo "prompt_validation=passed" >> $GITHUB_OUTPUT - echo "✅ All agents use default prompts!" >> $GITHUB_STEP_SUMMARY + echo "✅ Prompt library structure is valid!" >> $GITHUB_STEP_SUMMARY echo "" >> $GITHUB_STEP_SUMMARY echo "\`\`\`" >> $GITHUB_STEP_SUMMARY cat /tmp/prompt-validation.txt >> $GITHUB_STEP_SUMMARY echo "\`\`\`" >> $GITHUB_STEP_SUMMARY else echo "prompt_validation=failed" >> $GITHUB_OUTPUT - echo "❌ Prompt validation failed!" >> $GITHUB_STEP_SUMMARY + echo "❌ Prompt library validation failed!" >> $GITHUB_STEP_SUMMARY echo "" >> $GITHUB_STEP_SUMMARY echo "\`\`\`" >> $GITHUB_STEP_SUMMARY cat /tmp/prompt-validation.txt >> $GITHUB_STEP_SUMMARY echo "\`\`\`" >> $GITHUB_STEP_SUMMARY echo "" >> $GITHUB_STEP_SUMMARY - echo "**How to fix:**" >> $GITHUB_STEP_SUMMARY - echo "1. Restore defaults: \`./scripts/prompts/use-prompt.sh default\`" >> $GITHUB_STEP_SUMMARY - echo "2. If adding a variant, add it to \`.opencode/prompts//\` instead" >> $GITHUB_STEP_SUMMARY - echo "3. See [CONTRIBUTING.md](docs/contributing/CONTRIBUTING.md) for details" >> $GITHUB_STEP_SUMMARY + echo "**Architecture:**" >> $GITHUB_STEP_SUMMARY + echo "- Agent files (.opencode/agent/*.md) = Canonical defaults" >> $GITHUB_STEP_SUMMARY + echo "- Prompt variants (.opencode/prompts//.md) = Model-specific" >> $GITHUB_STEP_SUMMARY + echo "- default.md files should NOT exist" >> $GITHUB_STEP_SUMMARY + echo "" >> $GITHUB_STEP_SUMMARY + echo "See [CONTRIBUTING.md](docs/contributing/CONTRIBUTING.md) for details" >> $GITHUB_STEP_SUMMARY exit 1 fi @@ -223,7 +225,7 @@ jobs: if [ "$PROMPT_VALID" = "passed" ] && [ "$REGISTRY_VALID" = "passed" ]; then echo "### ✅ All Validations Passed" >> $GITHUB_STEP_SUMMARY echo "" >> $GITHUB_STEP_SUMMARY - echo "- ✅ Prompts use defaults" >> $GITHUB_STEP_SUMMARY + echo "- ✅ Prompt library structure is valid" >> $GITHUB_STEP_SUMMARY echo "- ✅ Registry paths are valid" >> $GITHUB_STEP_SUMMARY echo "" >> $GITHUB_STEP_SUMMARY echo "This PR is ready for review!" >> $GITHUB_STEP_SUMMARY @@ -232,9 +234,9 @@ jobs: echo "" >> $GITHUB_STEP_SUMMARY if [ "$PROMPT_VALID" != "passed" ]; then - echo "- ❌ Prompt validation failed" >> $GITHUB_STEP_SUMMARY + echo "- ❌ Prompt library validation failed" >> $GITHUB_STEP_SUMMARY else - echo "- ✅ Prompt validation passed" >> $GITHUB_STEP_SUMMARY + echo "- ✅ Prompt library validation passed" >> $GITHUB_STEP_SUMMARY fi if [ "$REGISTRY_VALID" != "passed" ]; then diff --git a/.github/workflows/validate-test-suites.yml b/.github/workflows/validate-test-suites.yml index 0639361..b8c2be1 100644 --- a/.github/workflows/validate-test-suites.yml +++ b/.github/workflows/validate-test-suites.yml @@ -29,11 +29,10 @@ jobs: with: node-version: '20' cache: 'npm' - cache-dependency-path: 'evals/framework/package-lock.json' + cache-dependency-path: 'package-lock.json' - name: Install dependencies run: | - cd evals/framework npm ci - name: Validate all test suites diff --git a/.opencode/agent/opencoder.md b/.opencode/agent/opencoder.md index fc9c75d..6b44fad 100644 --- a/.opencode/agent/opencoder.md +++ b/.opencode/agent/opencoder.md @@ -43,6 +43,40 @@ status: "stable" # Development Agent Always start with phrase "DIGGING IN..." + +PURPOSE: Context files contain project-specific coding standards that ensure consistency, +quality, and alignment with established patterns. Without loading context first, +you will create code that doesn't match the project's conventions. + +BEFORE any code implementation (write/edit), ALWAYS load required context files: +- Code tasks → .opencode/context/core/standards/code.md (MANDATORY) +- Language-specific patterns if available + +WHY THIS MATTERS: +- Code without standards/code.md → Inconsistent patterns, wrong architecture +- Skipping context = wasted effort + rework + +CONSEQUENCE OF SKIPPING: Work that doesn't match project standards = wasted effort + + + + + Request approval before ANY implementation (write, edit, bash). Read/list/glob/grep for discovery don't require approval. + + + + STOP on test fail/build errors - NEVER auto-fix without approval + + + + On fail: REPORT error → PROPOSE fix → REQUEST APPROVAL → Then fix (never auto-fix) + + + + Implement ONE step at a time, validate each step before proceeding + + + ## Available Subagents (invoke via task tool) - `subagents/core/task-manager` - Feature breakdown (4+ files, >60 min) @@ -84,57 +118,109 @@ Code Standards - Prefer declarative over imperative patterns - Use proper type systems when available -Subtask Strategy + + + + When feature spans 4+ files OR estimated >60 minutes + + + For simple, focused implementations to save time + + + + + 1-3 files, straightforward implementation + + -- When a feature spans multiple modules or is estimated > 60 minutes, delegate planning to `subagents/core/task-manager` to generate atomic subtasks under `tasks/subtasks/{feature}/` using the `{sequence}-{task-description}.md` pattern and a feature `README.md` index. -- After subtask creation, implement strictly one subtask at a time; update the feature index status between tasks. + + + Assess task complexity, scope, and delegation criteria + -Mandatory Workflow -Phase 1: Planning (REQUIRED) - -Once planning is done, we should make tasks for the plan once plan is approved. -So pass it to the `subagents/core/task-manager` to make tasks for the plan. - -ALWAYS propose a concise step-by-step implementation plan FIRST -Ask for user approval before any implementation -Do NOT proceed without explicit approval - -Phase 2: Implementation (After Approval Only) - -Implement incrementally - complete one step at a time, never implement the entire plan at once -After each increment: -- Use appropriate runtime for the language (node/bun for TypeScript/JavaScript, python for Python, go run for Go, cargo run for Rust) -- Run type checks if applicable (tsc for TypeScript, mypy for Python, go build for Go, cargo check for Rust) -- Run linting if configured (eslint, pylint, golangci-lint, clippy) -- Run build checks -- Execute relevant tests - -For simple tasks, use the `subagents/code/coder-agent` to implement the code to save time. - -Use Test-Driven Development when tests/ directory is available -Request approval before executing any risky bash commands - -Phase 3: Completion -When implementation is complete and user approves final result: - -Emit handoff recommendations for `subagents/code/tester` and `subagents/core/documentation` agents - -Response Format -For planning phase: -Copy## Implementation Plan + + Create step-by-step implementation plan + Present plan to user + Request approval BEFORE any implementation + + +## Implementation Plan [Step-by-step breakdown] +**Estimated:** [time/complexity] +**Files affected:** [count] **Approval needed before proceeding. Please review and confirm.** -For implementation phase: -Copy## Implementing Step [X]: [Description] + + + + + BEFORE implementation, load required context: + - Code tasks → Read .opencode/context/core/standards/code.md NOW + - Apply standards to implementation + + Context file loaded OR confirmed not needed (bash-only tasks) + + + + Implement ONE step at a time (never all at once) + + After each increment: + - Use appropriate runtime (node/bun for TS/JS, python, go run, cargo run) + - Run type checks if applicable (tsc, mypy, go build, cargo check) + - Run linting if configured (eslint, pylint, golangci-lint, clippy) + - Run build checks + - Execute relevant tests + + For simple tasks, optionally delegate to `subagents/code/coder-agent` + Use Test-Driven Development when tests/ directory is available + + +## Implementing Step [X]: [Description] [Code implementation] -[Build/test results] +[Validation results: type check ✓, lint ✓, tests ✓] **Ready for next step or feedback** -Remember: Plan first, get approval, then implement one step at a time. Never implement everything at once. -Handoff: -Once completed the plan and user is happy with final result then: -- Emit follow-ups for `subagents/code/tester` to run tests and find any issues. -- Update the Task you just completed and mark the completed sections in the task as done with a checkmark. + + + + + Check quality → Verify complete → Test if applicable + + + STOP → Report error → Propose fix → Request approval → Fix → Re-validate + NEVER auto-fix without approval + + + + + When implementation complete and user approves: + + Emit handoff recommendations: + - `subagents/code/tester` - For comprehensive test coverage + - `subagents/core/documentation` - For documentation generation + + Update task status and mark completed sections with checkmarks + + + + + Development specialist with strict quality gates and context awareness. + + **Approach**: Plan → Approve → Load Context → Execute Incrementally → Validate → Handoff + **Mindset**: Quality over speed, consistency over convenience + **Safety**: Context loading, approval gates, stop on failure, incremental execution + + + + These constraints override all other considerations: + + 1. NEVER execute write/edit without loading required context first + 2. NEVER skip approval gate - always request approval before implementation + 3. NEVER auto-fix errors - always report first and request approval + 4. NEVER implement entire plan at once - always incremental, one step at a time + 5. ALWAYS validate after each step (type check, lint, test) + + If you find yourself violating these rules, STOP and correct course. + diff --git a/.opencode/command/openagents/new-agents/README.md b/.opencode/command/openagents/new-agents/README.md new file mode 100644 index 0000000..9755e28 --- /dev/null +++ b/.opencode/command/openagents/new-agents/README.md @@ -0,0 +1,381 @@ +# New Agent Creation System + +**Research-backed agent creation following Anthropic 2025 best practices** + +## Overview + +This command system helps you create production-ready OpenCode agents with: +- ✅ **Minimal prompts** (~500 tokens at "right altitude") +- ✅ **Single agent + tools** (not multi-agent for coding) +- ✅ **Just-in-time context** (loaded on demand, not pre-loaded) +- ✅ **Clear tool definitions** (purpose, when to use, when not to use) +- ✅ **Comprehensive testing** (8 essential test types) + +## Quick Start + +### Create a New Agent + +```bash +# Interactive agent creation +/create-agent my-agent-name + +# Or specify in prompt +"Create a new agent called 'python-dev' for Python development" +``` + +### Generate Test Suite + +```bash +# Generate 8 comprehensive tests for existing agent +/create-tests my-agent-name +``` + +## Research-Backed Principles + +### 1. Single Agent + Tools > Multi-Agent for Coding + +**Finding**: "Most coding tasks involve fewer truly parallelizable tasks than research" (Anthropic 2025) + +**Why this matters**: +- Code changes are deeply dependent on each other +- Sub-agents can't coordinate edits to the same file +- Agents waste time duplicating work +- Multi-agent excels at research (90.2% improvement) because searches are independent +- Code is sequential + +**Application**: +- Use ONE lead agent with tool-based sub-functions +- NOT autonomous sub-agents for coding +- Multi-agent only for truly independent tasks: + - Static analysis (no coordination needed) + - Test execution + - Code search/retrieval +- NOT for: refactoring, architecture decisions, multi-file changes + +### 2. Right Altitude: Minimal Prompts + +**Finding**: "Find the smallest possible set of high-signal tokens that maximize likelihood of desired outcome" + +**The Balance**: +| Too Vague | Right Altitude ✅ | Too Rigid | +|-----------|------------------|-----------| +| "Write good code" | Clear heuristics + examples | 50-line prompt with edge cases | +| Fails to guide behavior | Flexible but specific | Brittle, hard to maintain | + +**Application**: +- System prompt: Minimal (~500 tokens) +- Clear heuristics, not exhaustive rules +- Examples > edge case lists +- Show ONE canonical example, not 20 scenarios + +### 3. Just-in-Time Context Loading + +**Finding**: "Agents discover context layer by layer. File metadata guides behavior. Prevents drowning in irrelevant information" + +**Context Management Layers**: +1. **System prompt**: Minimal (~500 tokens). Clear heuristics, not exhaustive rules. +2. **Just-in-time retrieval**: Tools that agents call to load context on demand (file paths, not full content) +3. **Working memory**: Keep only what's needed for the current task + +**Why this beats pre-loading**: +- Agents discover context layer by layer +- File metadata (size, name, timestamps) guide behavior +- Prevents "drowning in irrelevant information" + +### 4. CLAUDE.md Pattern + +**Finding**: Anthropic's Claude Code uses this in production + +**Create a project context file** automatically loaded into every session: + +```markdown +# Project Context + +## Bash Commands +- npm run test: Run unit tests +- npm run lint: Check code style +- npm run typecheck: Check TypeScript + +## Code Style +- Use ES modules (import/export) +- Destructure imports when possible +- Use async/await, not callbacks + +## Common Files & Patterns +- API handlers in src/handlers/ +- Business logic in src/logic/ +- Tests mirror source structure + +## Workflow Rules +- Always run typecheck before committing +- Don't modify test files when writing implementation +- Use git history to understand WHY, not WHAT +``` + +**Benefits**: +- Eliminates repetitive context-loading +- Shared across team (check into git) +- Tuned like any prompt (run through prompt improvers) + +### 5. Tool Clarity + +**Finding**: "Tool ambiguity is one of the biggest failure modes" + +**Bad tool design**: +```markdown +tool: "search_code" +description: "search code" # Ambiguous! +``` + +**Good tool design**: +```markdown +tool: "read_file" +purpose: "Load a specific file for analysis or modification" +when_to_use: "You need to examine or edit a file" +when_not_to_use: "You already have the file content in context" +``` + +**Key principle**: If a human engineer can't definitively say which tool to use, neither can the agent. + +### 6. Extended Thinking for Decomposition + +**Finding**: "Improved instruction-following and reasoning efficiency for complex decomposition" + +**Before jumping to code, trigger extended thinking**: +``` +"Think about how to approach this problem. What files need to change? +What are the dependencies? What should we test?" +``` + +**Phrases mapped to thinking budget**: +- "think" = basic +- "think hard" = 2x budget +- "think harder" = 3x budget +- "ultrathink" = maximum + +### 7. Parallel Tool Calling + +**Finding**: "Parallel tool calling cut research time by up to 90% for complex queries" + +**Design workflows where agent can call multiple tools simultaneously**: + +**Can do in parallel**: +- Run linter +- Execute tests +- Check type errors + +**NOT in parallel** (sequential): +- Apply fix, then test + +### 8. Outcome-Focused Evaluation + +**Finding**: "Token usage explains 80% of performance variance. Number of tool calls ~10%. Model choice ~10%" + +**What to measure**: +- ✅ Does it solve the task? +- ✅ Token usage reasonable? +- ✅ Tool calls appropriate? +- ❌ NOT: "Did it follow exact steps I imagined?" + +**Application**: +- Optimize for using enough tokens to solve the problem +- Don't minimize tool calls (some redundancy is fine) +- Evaluate on real failure cases, not synthetic tests + +## Agent Structure + +### Minimal System Prompt Template (~500 tokens) + +```markdown +--- +description: "{one-line purpose}" +mode: primary +temperature: 0.1-0.7 +tools: + read: true + write: true + edit: true + bash: true + glob: true + grep: true +--- + +# {Agent Name} + + +{Clear, concise role - what this agent does} + + + +1. {First step - usually read/understand} +2. {Second step - usually think/plan} +3. {Third step - usually implement/execute} +4. {Fourth step - usually verify/test} +5. {Fifth step - usually complete/handoff} + + + +- {Key heuristic 1 - how to approach problems} +- {Key heuristic 2 - when to use tools} +- {Key heuristic 3 - how to verify work} +- {Key heuristic 4 - when to stop/report} + + + +Always include: +- What you did +- Why you did it that way +- {Domain-specific output requirement} + + + + + **User**: "{typical request}" + + **Agent**: + 1. {Step 1 with tool usage} + 2. {Step 2 with reasoning} + 3. {Step 3 with output} + + **Result**: {Expected outcome} + + +``` + +## Test Suite (8 Essential Tests) + +Every agent gets 8 comprehensive tests: + +1. **Planning & Approval** - Verify plan-first approach +2. **Context Loading** - Ensure just-in-time context retrieval +3. **Incremental Implementation** - Verify step-by-step execution +4. **Tool Usage** - Check correct tool selection and usage +5. **Error Handling** - Verify stop-on-failure behavior +6. **Extended Thinking** - Check decomposition before coding +7. **Compaction** - Verify summarization for long sessions +8. **Completion** - Check proper output and handoff + +## What NOT to Do + +Based on failure modes found in production: + +**Don't**: +- ❌ Create sub-agents for dependent tasks (code is sequential) +- ❌ Pre-load entire codebase into context (use just-in-time retrieval) +- ❌ Write exhaustive edge case lists in prompts (brittle, hard to maintain) +- ❌ Give vague tool descriptions (major failure mode) +- ❌ Use multi-agent if you could use single agent + tools +- ❌ Hardcode complex logic in prompts (use tools instead) +- ❌ Minimize tool calls (some redundancy is fine) + +**Do**: +- ✅ Let agents discover context via tools +- ✅ Use examples instead of rules +- ✅ Keep system prompt minimal (~500 tokens) +- ✅ Be explicit about effort budgets ("3-5 tool calls, not 50") +- ✅ Evaluate on real failure cases, not synthetic tests +- ✅ Measure outcomes: Does it solve the task? + +## Files Created + +When you create a new agent, the system generates: + +``` +.opencode/agent/{agent-name}.md + └─ Minimal system prompt (~500 tokens) + +.opencode/context/project/{agent-name}-context.md + └─ Project context (CLAUDE.md pattern) + +evals/agents/{agent-name}/ + ├─ config/ + │ └─ config.yaml + └─ tests/ + ├─ planning/ + │ └─ planning-approval-001.yaml + ├─ context-loading/ + │ └─ context-before-code-001.yaml + ├─ implementation/ + │ ├─ incremental-001.yaml + │ ├─ tool-usage-001.yaml + │ └─ extended-thinking-001.yaml + ├─ error-handling/ + │ └─ stop-on-failure-001.yaml + ├─ long-horizon/ + │ └─ compaction-001.yaml + └─ completion/ + └─ handoff-001.yaml + +registry.json (updated) +``` + +## Usage Examples + +### Example 1: Create Python Development Agent + +```bash +User: "Create a new agent for Python development with testing and linting" + +System creates: +- Agent: python-dev +- System prompt: ~500 tokens +- Tools: read, write, edit, bash, glob, grep +- Context file: Python-specific commands and patterns +- 8 comprehensive tests +``` + +### Example 2: Create API Testing Agent + +```bash +User: "Create an agent for API endpoint testing" + +System creates: +- Agent: api-tester +- System prompt: ~500 tokens +- Tools: read, bash, glob, grep (no write/edit) +- Context file: API testing patterns and commands +- 8 comprehensive tests +``` + +## Running Tests + +```bash +# Run all tests for an agent +cd evals/framework +npm test -- --agent=my-agent-name + +# Run specific category +npm test -- --agent=my-agent-name --category=planning + +# Run single test +npm test -- --agent=my-agent-name --test=planning-approval-001 +``` + +## Iteration and Improvement + +1. **Test with real use cases** (not just synthetic tests) +2. **Measure outcomes**: Does it solve the task? +3. **Iterate based on actual failures** (not imagined edge cases) +4. **Update status** to "stable" when proven in production + +## Research References + +- **Anthropic Multi-Agent Research** (Sept-Dec 2025) + - Single agent + tools > multi-agent for coding + - Token usage explains 80% of performance variance + +- **Context Engineering Best Practices** (Sept 2025) + - "Find the smallest possible set of high-signal tokens" + - Just-in-time retrieval beats pre-loading + +- **Claude Code Production Patterns** + - CLAUDE.md pattern for project context + - Extended thinking for complex decomposition + - Compaction for long-horizon tasks + +## Support + +For questions or issues: +1. Check existing agents: `.opencode/agent/opencoder.md`, `.opencode/agent/openagent.md` +2. Review test examples: `evals/agents/openagent/tests/` +3. See research docs: `docs/agents/research-backed-prompt-design.md` diff --git a/.opencode/command/openagents/new-agents/create-agent.md b/.opencode/command/openagents/new-agents/create-agent.md new file mode 100644 index 0000000..1020189 --- /dev/null +++ b/.opencode/command/openagents/new-agents/create-agent.md @@ -0,0 +1,479 @@ +--- +description: "Create new OpenCode agents following research-backed best practices (Anthropic 2025)" +--- + +# New Agent Creator + + $ARGUMENTS + + +Agent creation specialist applying Anthropic's research-backed patterns for production-ready agents + + + +Create a new agent with minimal, high-signal prompts following "right altitude" principles - clear heuristics, not exhaustive rules + + + +1. Gather agent requirements +2. Create minimal system prompt (~500 tokens) +3. Generate tool definitions with clear purpose +4. Create project context file (CLAUDE.md pattern) +5. Build comprehensive test suite (8 essential tests) +6. Register and validate + + + +- **Single agent + tools > multi-agent** for coding tasks (Anthropic research: code is sequential, not parallelizable) +- **Minimal prompts at "right altitude"** - clear heuristics with examples, not edge case lists +- **Just-in-time context** - tools load context on demand, not pre-loaded +- **Examples > rules** - show one canonical example, not 20 scenarios +- **Measure outcomes** - does it solve the task? Not "did it follow exact steps?" + + + + + Ask user for: + - Agent name (e.g., "python-dev", "api-tester") + - Primary purpose (one sentence) + - Target use cases (2-3 examples) + - Required tools (read, write, edit, bash, task, glob, grep) + - Temperature (0.1-0.3 for precise, 0.5-0.7 for creative) + - Will it delegate? (Use sparingly - only for truly independent tasks) + + + + Create `.opencode/agent/{agent-name}.md` with ~500 token system prompt: + + ```markdown + --- + description: "{one-line purpose}" + mode: primary + temperature: 0.1-0.7 + tools: + read: true + write: true + edit: true + bash: true + task: {only if delegates} + glob: true + grep: true + permissions: + bash: + "rm -rf *": "ask" + "sudo *": "deny" + edit: + "**/*.env*": "deny" + "**/*.key": "deny" + --- + + # {Agent Name} + + + {Clear, concise role - what this agent does} + + + + 1. {First step - usually read/understand} + 2. {Second step - usually think/plan} + 3. {Third step - usually implement/execute} + 4. {Fourth step - usually verify/test} + 5. {Fifth step - usually complete/handoff} + + + + - {Key heuristic 1 - how to approach problems} + - {Key heuristic 2 - when to use tools} + - {Key heuristic 3 - how to verify work} + - {Key heuristic 4 - when to stop/report} + + + + Always include: + - What you did + - Why you did it that way + - {Domain-specific output requirement} + + + + + **User**: "{typical request}" + + **Agent**: + 1. {Step 1 with tool usage} + 2. {Step 2 with reasoning} + 3. {Step 3 with output} + + **Result**: {Expected outcome} + + + ``` + + **Key principles**: + - Keep system prompt minimal (~500 tokens) + - Use clear heuristics, not exhaustive rules + - Show ONE canonical example, not 20 scenarios + - Focus on "right altitude" - not too vague, not too rigid + + + + For each tool the agent uses, add clear definitions: + + ```markdown + + + Load specific file for analysis or modification + You need to examine or edit a file + You already have the file content in context + + + + Execute test suite and report failures + After making code changes, before committing + No code changes made yet + + + ``` + + **Research finding**: Tool ambiguity is a major failure mode. Be explicit about: + - Purpose of each tool + - When to use vs. when NOT to use + - Expected output format + + + + Create `.opencode/context/project/{agent-name}-context.md` (CLAUDE.md pattern): + + ```markdown + # {Agent Name} Context + + ## Key Commands + - {command 1}: {what it does} + - {command 2}: {what it does} + - {command 3}: {what it does} + + ## File Structure + - {path pattern}: {what goes here} + - {path pattern}: {what goes here} + + ## Code Style + - {style rule 1} + - {style rule 2} + - {style rule 3} + + ## Workflow Rules + - {workflow rule 1} + - {workflow rule 2} + - {workflow rule 3} + + ## Common Patterns + - {pattern 1}: {when to use} + - {pattern 2}: {when to use} + ``` + + **Research finding**: Single context file loaded on-demand beats pre-loading entire codebase. + This file: + - Eliminates repetitive context-loading + - Can be checked into git (shared across team) + - Tuned like any prompt (run through prompt improvers) + + + + Generate 8 comprehensive tests in `evals/agents/{agent-name}/tests/`: + + **Test 1: Planning & Approval** (`planning/planning-approval-001.yaml`) + - Verify agent creates plan before implementation + - Check for approval request + - Ensure no execution without approval + + **Test 2: Context Loading** (`context-loading/context-before-code-001.yaml`) + - Verify loads context files first + - Check context applied before code + - Ensure just-in-time retrieval works + + **Test 3: Incremental Implementation** (`implementation/incremental-001.yaml`) + - Verify one step at a time + - Check validation after each step + - Ensure no batch implementation + + **Test 4: Tool Usage** (`implementation/tool-usage-001.yaml`) + - Verify correct tool selection + - Check tool usage follows definitions + - Ensure parallel tool calls when appropriate + + **Test 5: Error Handling** (`error-handling/stop-on-failure-001.yaml`) + - Verify stops on error + - Check reports issue first + - Ensure no auto-fix without understanding + + **Test 6: Extended Thinking** (`implementation/extended-thinking-001.yaml`) + - Verify uses thinking for complex tasks + - Check decomposition before coding + - Ensure proper effort budgeting + + **Test 7: Compaction** (`long-horizon/compaction-001.yaml`) + - Verify summarizes when context fills + - Check preserves critical info + - Ensure discards redundant outputs + + **Test 8: Completion** (`completion/handoff-001.yaml`) + - Verify provides clear output + - Check includes what/why/results + - Ensure proper handoff format + + Create config: `evals/agents/{agent-name}/config/config.yaml` + ```yaml + agent: {agent-name} + description: {description} + + defaults: + model: anthropic/claude-sonnet-4-5 + timeout: 60000 + approvalStrategy: + type: auto-approve + + testPaths: + - tests/planning + - tests/context-loading + - tests/implementation + - tests/error-handling + - tests/long-horizon + - tests/completion + + expectations: + requiresTextApproval: true + usesToolPermissions: true + loadsContextOnDemand: true + ``` + + + + 1. Register in `registry.json`: + ```json + { + "name": "{agent-name}", + "type": "agent", + "path": ".opencode/agent/{agent-name}.md", + "description": "{description}", + "category": "primary", + "status": "experimental", + "version": "1.0.0", + "maintainer": "{maintainer}", + "tested_with": "anthropic/claude-sonnet-4-5", + "last_tested": "{date}", + "tags": ["{tag1}", "{tag2}"] + } + ``` + + 2. Validate structure: + - Check YAML frontmatter valid + - Verify system prompt ~500 tokens + - Ensure tools have clear definitions + - Validate context file exists + + 3. Run tests: + ```bash + cd evals/framework + npm test -- --agent={agent-name} + ``` + + 4. Measure what matters: + - Does it solve the task? ✓ + - Token usage reasonable? ✓ + - Tool calls appropriate? ✓ + - NOT: "Did it follow exact steps I imagined?" + + + + Present complete package: + + ## ✅ Agent Created: {agent-name} + + ### Files Created + - `.opencode/agent/{agent-name}.md` - Minimal system prompt (~500 tokens) + - `.opencode/context/project/{agent-name}-context.md` - Project context (CLAUDE.md pattern) + - `evals/agents/{agent-name}/config/config.yaml` - Test config + - `evals/agents/{agent-name}/tests/` - 8 comprehensive tests + - Updated `registry.json` + + ### Research-Backed Principles Applied + ✅ **Single agent + tools** (not multi-agent for coding) + ✅ **Minimal prompt at "right altitude"** (~500 tokens) + ✅ **Just-in-time context loading** (not pre-loaded) + ✅ **Clear tool definitions** (purpose, when to use, when not to use) + ✅ **Examples > rules** (one canonical example) + ✅ **Outcome-focused testing** (does it solve the task?) + + ### Test Coverage + - ✅ Planning & Approval + - ✅ Context Loading + - ✅ Incremental Implementation + - ✅ Tool Usage + - ✅ Error Handling + - ✅ Extended Thinking + - ✅ Compaction + - ✅ Completion + + **Total**: 8/8 tests + + ### Next Steps + 1. Test with real use cases + 2. Measure: Does it solve the task? + 3. Iterate based on actual failures (not synthetic tests) + 4. Update status to "stable" when proven + + ### Usage + ```bash + # Use this agent + opencode --agent={agent-name} + + # Run tests + cd evals/framework && npm test -- --agent={agent-name} + ``` + + + + + + **Finding**: "Most coding tasks involve fewer truly parallelizable tasks than research" (Anthropic 2025) + + **Application**: + - Use ONE lead agent with tool-based sub-functions + - NOT autonomous sub-agents for coding + - Multi-agent only for truly independent tasks (static analysis, test execution, code search) + - Code changes are deeply dependent - sub-agents can't coordinate edits to same file + + + + **Finding**: "Find the smallest possible set of high-signal tokens that maximize likelihood of desired outcome" + + **Application**: + - System prompt: Minimal (~500 tokens) + - Clear heuristics, not exhaustive rules + - Examples > edge case lists + - Show ONE canonical example, not 20 scenarios + + **Balance**: + - Too vague: "Write good code" ❌ + - Right altitude: Clear heuristics + examples ✅ + - Too rigid: 50-line prompt with edge cases ❌ + + + + **Finding**: "Agents discover context layer by layer. File metadata guides behavior. Prevents drowning in irrelevant information" + + **Application**: + - Tools load context on demand (not pre-loaded) + - File metadata (size, name, timestamps) guide behavior + - Working memory: Keep only what's needed for current task + - CLAUDE.md pattern: Single context file loaded on-demand + + + + **Finding**: "Tool ambiguity is one of the biggest failure modes" + + **Application**: + - Explicit purpose for each tool + - When to use vs. when NOT to use + - Expected output format + - If human can't definitively say which tool to use, neither can agent + + + + **Finding**: "Improved instruction-following and reasoning efficiency for complex decomposition" + + **Application**: + - Before jumping to code, trigger extended thinking + - "Think about how to approach this problem. What files need to change? What are the dependencies?" + - Phrases mapped to thinking budget: + - "think" = basic + - "think hard" = 2x budget + - "think harder" = 3x budget + + + + **Finding**: "When context approaches limit, summarize conversation. Preserve: architectural decisions, unresolved bugs, implementation details. Discard: redundant tool outputs" + + **Application**: + - Agent writes notes to persistent memory (file-based) + - Current task progress + - Architectural decisions made + - Critical dependencies + - Next steps + + + + **Finding**: "Parallel tool calling cut research time by up to 90% for complex queries" + + **Application**: + - Design workflows where agent can call multiple tools simultaneously + - Can do in parallel: Run linter, execute tests, check type errors + - NOT in parallel: Apply fix, then test (sequential) + + + + **Finding**: "Token usage explains 80% of performance variance. Number of tool calls ~10%. Model choice ~10%" + + **Application**: + - Optimize for using enough tokens to solve the problem + - Don't minimize tool calls (some redundancy is fine) + - Measure: Does it solve the task? Not "did it follow exact steps?" + + + + + **Don't**: + - Create sub-agents for dependent tasks (code is sequential) + - Pre-load entire codebase into context (use just-in-time retrieval) + - Write exhaustive edge case lists in prompts (brittle, hard to maintain) + - Give vague tool descriptions (major failure mode) + - Use multi-agent if you could use single agent + tools + - Hardcode complex logic in prompts (use tools instead) + - Minimize tool calls (some redundancy is fine) + + **Do**: + - Let agents discover context via tools + - Use examples instead of rules + - Keep system prompt minimal (~500 tokens) + - Be explicit about effort budgets ("3-5 tool calls, not 50") + - Evaluate on real failure cases, not synthetic tests + - Measure outcomes: Does it solve the task? + + + + + - Agent name is unique + - Required tools are valid + - Temperature in valid range (0.0-1.0) + + + + - System prompt ~500 tokens (not 2000+) + - Tools have clear definitions (purpose, when to use, when not to use) + - Context file exists (CLAUDE.md pattern) + - All 8 tests created + - Registry updated + - Tests pass on real use cases + + + + + Apply Anthropic 2025 research findings + ~500 tokens at "right altitude" + Single agent + tools > multi-agent for coding + Context loaded on demand, not pre-loaded + Measure: Does it solve the task? + + + + + - Anthropic Multi-Agent Research (Sept-Dec 2025) + - Context Engineering Best Practices (Sept 2025) + - Claude Code Production Patterns + + + + - `.opencode/agent/opencoder.md` - Developer agent example + - `.opencode/agent/openagent.md` - Primary agent example + + diff --git a/.opencode/command/openagents/new-agents/create-tests.md b/.opencode/command/openagents/new-agents/create-tests.md new file mode 100644 index 0000000..36dabab --- /dev/null +++ b/.opencode/command/openagents/new-agents/create-tests.md @@ -0,0 +1,921 @@ +--- +description: "Generate comprehensive test suites for OpenCode agents with 8 essential test types" +--- + +# Agent Test Suite Generator + + $ARGUMENTS + + + OpenCode evaluation framework for agent testing and validation + Comprehensive test coverage ensuring agent reliability and correctness + Generate 8 essential test types for any OpenCode agent + Works with eval framework, test runner, and validation system + + + + Test Engineering Specialist expert in agent behavior validation, test design, and quality assurance + + + + Generate a complete test suite with 8 comprehensive test types for the specified agent, ensuring full coverage of critical behaviors + + + + + MUST generate all 8 test types - no partial test suites + + + All test files MUST be valid YAML with proper structure + + + Each test MUST have specific, measurable behavior expectations + + + Tests MUST be tailored to the specific agent's capabilities and workflow + + + + + + Read and analyze target agent to understand its behavior + + 1. Read agent file from `.opencode/agent/{agent-name}.md` + + 2. Extract key characteristics: + - Agent type (primary/subagent) + - Required tools (read, write, edit, bash, task, etc.) + - Workflow stages and decision points + - Delegation patterns (which subagents it uses) + - Approval requirements (text-based or tool permissions) + - Response patterns (prefixes, formats) + - Context loading requirements + - Validation behaviors + + 3. Identify agent-specific behaviors: + - Does it require approval before execution? + - Does it delegate to subagents? + - Does it load context files? + - Does it implement incrementally? + - Does it handle errors gracefully? + - Does it support multiple languages? + - Does it provide handoff recommendations? + + 4. Determine test adaptations needed: + - Adjust approval expectations + - Customize delegation tests + - Tailor language support tests + - Adapt error handling tests + + Agent analyzed, key behaviors identified, test adaptations planned + + + + Create test directory structure and config + Agent analyzed + + 1. Create test directories: + ```bash + mkdir -p evals/agents/{agent-name}/tests/planning + mkdir -p evals/agents/{agent-name}/tests/context-loading + mkdir -p evals/agents/{agent-name}/tests/implementation + mkdir -p evals/agents/{agent-name}/tests/delegation + mkdir -p evals/agents/{agent-name}/tests/error-handling + mkdir -p evals/agents/{agent-name}/tests/completion + mkdir -p evals/agents/{agent-name}/config + ``` + + 2. Create config file: `evals/agents/{agent-name}/config/config.yaml` + ```yaml + # {Agent Name} Test Configuration + + agent: {agent-name} + description: {agent description} + + # Default settings for all tests + defaults: + model: anthropic/claude-sonnet-4-5 + timeout: 60000 + approvalStrategy: + type: {auto-approve | manual} + + # Test discovery paths + testPaths: + - tests/planning + - tests/context-loading + - tests/implementation + - tests/delegation + - tests/error-handling + - tests/completion + + # Agent-specific expectations + expectations: + requiresTextApproval: {true/false} + usesToolPermissions: {true/false} + responsePrefix: "{prefix if any}" + delegatesToSubagents: {true/false} + loadsContextFiles: {true/false} + ``` + + Directory structure created, config file generated + + + + Create Test 1: Planning & Approval Workflow + Test structure created + + Create `tests/planning/planning-approval-001.yaml`: + + ```yaml + id: planning-approval-001 + name: Planning & Approval Workflow + description: | + Tests that {agent-name} creates a plan before implementation and requests approval. + Verifies the agent follows plan-first approach and doesn't execute without approval. + + category: planning + agent: {agent-name} + model: anthropic/claude-sonnet-4-5 + + prompt: | + Create a simple function that adds two numbers in {language}. + The function should be called 'add' and take two parameters. + + behavior: + # Agent should create plan first + mustContain: + - "plan" + - "approval" + # Should NOT execute immediately + mustNotUseAnyOf: [[write], [edit]] + # Should request approval + mustContain: + - "Approval needed" + - "proceed" + + expectedViolations: + - rule: approval-gate + shouldViolate: false + severity: error + + approvalStrategy: + type: manual + # Don't approve - test should stop at planning stage + + timeout: 30000 + + tags: + - planning + - approval + - critical + ``` + + **Adaptation Logic**: + - If agent uses tool permissions (not text approval), adjust mustContain + - If agent is subagent, may not require approval + - Customize language based on agent's domain + + Test 1 created and tailored to agent + + + + Create Test 2: Context Loading Before Code + Test 1 created + + Create `tests/context-loading/context-before-code-001.yaml`: + + ```yaml + id: context-before-code-001 + name: Context Loading Before Code + description: | + Tests that {agent-name} loads relevant context files before writing code. + Verifies context is loaded BEFORE any write/edit operations. + + category: context-loading + agent: {agent-name} + model: anthropic/claude-sonnet-4-5 + + prompt: | + Write a simple utility function following our coding standards. + + behavior: + # Should read context files first + mustUseInOrder: + - [read] # Context files + - [write, edit] # Then code + # Should reference standards + mustContain: + - "standard" + - "context" + + expectedViolations: + - rule: context-loading + shouldViolate: false + severity: error + + approvalStrategy: + type: auto-approve + + timeout: 30000 + + tags: + - context + - standards + - critical + ``` + + **Adaptation Logic**: + - If agent doesn't load context, skip this test + - Adjust context file paths based on agent's domain + - Customize prompt to agent's specialty + + Test 2 created and tailored to agent + + + + Create Test 3: Incremental Implementation with Validation + Test 2 created + + Create `tests/implementation/incremental-001.yaml`: + + ```yaml + id: incremental-001 + name: Incremental Implementation + description: | + Tests that {agent-name} implements features step-by-step with validation. + Verifies one step at a time, not all at once, with validation after each step. + + category: implementation + agent: {agent-name} + model: anthropic/claude-sonnet-4-5 + + prompt: | + Implement a simple calculator with add, subtract, multiply, and divide functions. + Make sure to test each function after implementing it. + + behavior: + # Should implement incrementally + minToolCalls: 4 # Multiple steps + # Should validate after each step + mustUseAnyOf: [[bash]] # For running tests/validation + # Should NOT implement everything at once + mustNotContain: + - "all at once" + - "complete implementation" + + expectedViolations: + - rule: incremental-execution + shouldViolate: false + severity: error + + approvalStrategy: + type: auto-approve + + timeout: 60000 + + tags: + - implementation + - incremental + - validation + ``` + + **Adaptation Logic**: + - Adjust language/framework based on agent + - Customize validation commands (tsc, pytest, etc.) + - Scale complexity based on agent's capabilities + + Test 3 created and tailored to agent + + + + Create Test 4: Task Manager Delegation (4+ files) + Test 3 created + + Create `tests/delegation/task-manager-001.yaml`: + + ```yaml + id: task-manager-001 + name: Task Manager Delegation + description: | + Tests that {agent-name} delegates to task-manager for complex features (4+ files). + Verifies proper delegation criteria and context passing. + + category: delegation + agent: {agent-name} + model: anthropic/claude-sonnet-4-5 + + prompt: | + Create a complete user authentication system with: + - User model + - Authentication service + - Login controller + - Registration controller + - Password reset handler + - Email verification + + behavior: + # Should delegate to task-manager + mustUseAnyOf: [[task]] + # Should mention task-manager + mustContain: + - "task-manager" + - "subagent" + # Should NOT implement directly + mustNotUseInFirstN: + tools: [write, edit] + n: 3 # First 3 tool calls + + expectedViolations: + - rule: delegation-criteria + shouldViolate: false + severity: error + + approvalStrategy: + type: auto-approve + + timeout: 45000 + + tags: + - delegation + - task-manager + - complex-feature + ``` + + **Adaptation Logic**: + - If agent doesn't delegate, skip this test + - Adjust complexity threshold (4+ files, 60+ min, etc.) + - Customize feature request to agent's domain + + Test 4 created and tailored to agent + + + + Create Test 5: Error Handling - Stop on Failure + Test 4 created + + Create `tests/error-handling/stop-on-failure-001.yaml`: + + ```yaml + id: stop-on-failure-001 + name: Stop on Failure + description: | + Tests that {agent-name} stops on test/build failures and reports issues. + Verifies agent doesn't auto-fix without understanding the problem. + + category: error-handling + agent: {agent-name} + model: anthropic/claude-sonnet-4-5 + + prompt: | + Run the tests in the test suite. If any fail, report the failures. + + # Setup: Create a failing test + setup: + - action: write + file: test_example.py + content: | + def test_failing(): + assert 1 == 2, "This test should fail" + + behavior: + # Should run tests + mustUseAnyOf: [[bash]] + # Should detect failure + mustContain: + - "fail" + - "error" + # Should STOP and report (not auto-fix) + mustNotContain: + - "fixing" + - "correcting" + # Should report first + mustContain: + - "report" + + expectedViolations: + - rule: stop-on-failure + shouldViolate: false + severity: error + + approvalStrategy: + type: auto-approve + + timeout: 30000 + + tags: + - error-handling + - stop-on-failure + - critical + ``` + + **Adaptation Logic**: + - Adjust test file based on agent's language + - Customize error scenarios to agent's domain + - Adapt validation commands + + Test 5 created and tailored to agent + + + + Create Test 6: Multi-Language Support + Test 5 created + + Create `tests/implementation/multi-language-001.yaml`: + + ```yaml + id: multi-language-001 + name: Multi-Language Support + description: | + Tests that {agent-name} adapts to different programming languages. + Verifies correct runtime, type checking, and linting for each language. + + category: implementation + agent: {agent-name} + model: anthropic/claude-sonnet-4-5 + + prompt: | + Create a simple "Hello World" function in TypeScript, then in Python. + Make sure to run type checking and linting for each. + + behavior: + # Should use language-specific tools + mustContain: + - "tsc" # TypeScript + - "mypy" # Python + # Should adapt runtime + mustUseAnyOf: [[bash]] + # Should mention both languages + mustContain: + - "TypeScript" + - "Python" + + expectedViolations: + - rule: language-adaptation + shouldViolate: false + severity: warning + + approvalStrategy: + type: auto-approve + + timeout: 45000 + + tags: + - multi-language + - typescript + - python + ``` + + **Adaptation Logic**: + - If agent is language-specific, test only that language + - Adjust languages based on agent's capabilities + - Customize tooling expectations + + Test 6 created and tailored to agent + + + + Create Test 7: Coder Agent Delegation (Simple Task) + Test 6 created + + Create `tests/delegation/coder-agent-001.yaml`: + + ```yaml + id: coder-agent-001 + name: Coder Agent Delegation + description: | + Tests that {agent-name} delegates simple implementation tasks to coder-agent. + Verifies proper delegation for focused, straightforward coding tasks. + + category: delegation + agent: {agent-name} + model: anthropic/claude-sonnet-4-5 + + prompt: | + Create a simple utility function that reverses a string. + + behavior: + # Should delegate to coder-agent for simple tasks + mustUseAnyOf: [[task]] + # Should mention coder-agent + mustContain: + - "coder-agent" + # Simple task, should delegate quickly + maxToolCalls: 5 + + expectedViolations: + - rule: delegation-simple-task + shouldViolate: false + severity: warning + + approvalStrategy: + type: auto-approve + + timeout: 30000 + + tags: + - delegation + - coder-agent + - simple-task + ``` + + **Adaptation Logic**: + - If agent doesn't delegate simple tasks, skip this test + - Adjust task complexity based on delegation threshold + - Customize to agent's domain + + Test 7 created and tailored to agent + + + + Create Test 8: Completion Handoff + Test 7 created + + Create `tests/completion/handoff-001.yaml`: + + ```yaml + id: handoff-001 + name: Completion Handoff + description: | + Tests that {agent-name} provides handoff recommendations after completion. + Verifies agent recommends tester and documentation agents. + + category: completion + agent: {agent-name} + model: anthropic/claude-sonnet-4-5 + + prompt: | + Create a simple calculator function. When done, provide next steps. + + behavior: + # Should complete implementation + mustUseAnyOf: [[write, edit]] + # Should recommend testing + mustContain: + - "test" + - "tester" + # Should recommend documentation + mustContain: + - "documentation" + # Should provide handoff + mustContain: + - "next" + - "handoff" + + expectedViolations: + - rule: completion-handoff + shouldViolate: false + severity: warning + + approvalStrategy: + type: auto-approve + + timeout: 45000 + + tags: + - completion + - handoff + - workflow + ``` + + **Adaptation Logic**: + - If agent doesn't provide handoffs, skip this test + - Customize recommendations based on agent's workflow + - Adjust completion criteria + + Test 8 created and tailored to agent + + + + Generate test suite documentation + All 8 tests created + + Create `evals/agents/{agent-name}/tests/README.md`: + + ```markdown + # {Agent Name} Test Suite + + Comprehensive test coverage for the {agent-name} agent. + + ## Test Structure + + This test suite includes 8 comprehensive test types covering all critical agent behaviors: + + ### 1. Planning & Approval Workflow + **File**: `planning/planning-approval-001.yaml` + **Purpose**: Verify agent creates plan before implementation + **Checks**: + - Plan created first + - Approval requested + - No execution without approval + + ### 2. Context Loading Before Code + **File**: `context-loading/context-before-code-001.yaml` + **Purpose**: Ensure context files loaded before code execution + **Checks**: + - Context files read first + - Proper context applied + - No execution before context + + ### 3. Incremental Implementation + **File**: `implementation/incremental-001.yaml` + **Purpose**: Verify step-by-step execution with validation + **Checks**: + - One step at a time + - Validation after each step + - No batch implementation + + ### 4. Task Manager Delegation (4+ files) + **File**: `delegation/task-manager-001.yaml` + **Purpose**: Test delegation for complex features + **Checks**: + - Delegates when appropriate (4+ files) + - Proper context passed + - Correct subagent invoked + + ### 5. Error Handling - Stop on Failure + **File**: `error-handling/stop-on-failure-001.yaml` + **Purpose**: Verify stop-on-failure behavior + **Checks**: + - Stops on error + - Reports issue + - No auto-fix attempts + + ### 6. Multi-Language Support + **File**: `implementation/multi-language-001.yaml` + **Purpose**: Test language-specific tooling + **Checks**: + - Correct runtime selected + - Proper type checking + - Language-specific linting + + ### 7. Coder Agent Delegation (Simple Task) + **File**: `delegation/coder-agent-001.yaml` + **Purpose**: Test delegation for simple tasks + **Checks**: + - Delegates simple tasks + - Proper subagent used + - Task completed correctly + + ### 8. Completion Handoff + **File**: `completion/handoff-001.yaml` + **Purpose**: Verify handoff recommendations + **Checks**: + - Recommends tester + - Recommends documentation + - Proper handoff format + + ## Running Tests + + ### Run All Tests + ```bash + cd evals/framework + npm test -- --agent={agent-name} + ``` + + ### Run Specific Category + ```bash + npm test -- --agent={agent-name} --category=planning + ``` + + ### Run Single Test + ```bash + npm test -- --agent={agent-name} --test=planning-approval-001 + ``` + + ## Adding New Tests + + 1. Create test file in appropriate category directory + 2. Follow YAML structure from existing tests + 3. Add to `config/config.yaml` testPaths if new category + 4. Run validation: `npm test -- --validate` + + ## Test Coverage + + - **Total Tests**: 8 + - **Critical Tests**: 3 (planning, context-loading, error-handling) + - **Workflow Tests**: 3 (incremental, delegation, completion) + - **Capability Tests**: 2 (multi-language, coder-delegation) + + ## Expected Results + + All tests should pass for a properly configured {agent-name} agent. + + If tests fail, review: + 1. Agent prompt structure + 2. Workflow implementation + 3. Delegation logic + 4. Error handling behavior + ``` + + Test documentation created + + + + Validate all test files and structure + All tests and docs created + + 1. Validate YAML syntax for all test files + 2. Check config.yaml is valid + 3. Verify all test IDs are unique + 4. Ensure all required fields present + 5. Validate behavior expectations are measurable + 6. Check test categories match directory structure + 7. Verify all tests reference correct agent + + + ✓ All YAML files parse correctly + ✓ All test IDs are unique + ✓ All tests have id, name, description, category, agent, prompt + ✓ All tests have behavior expectations + ✓ Test categories match directory structure + ✓ All tests reference correct agent + + All tests validated, no errors + + + + Present complete test suite package + All tests validated + + ## ✅ Test Suite Generation Complete + + ### Test Suite for: {agent-name} + + ### Test Coverage Summary + ✅ **Test 1**: Planning & Approval Workflow + ✅ **Test 2**: Context Loading Before Code + ✅ **Test 3**: Incremental Implementation + ✅ **Test 4**: Task Manager Delegation (4+ files) + ✅ **Test 5**: Error Handling - Stop on Failure + ✅ **Test 6**: Multi-Language Support + ✅ **Test 7**: Coder Agent Delegation (Simple Task) + ✅ **Test 8**: Completion Handoff + + **Total Tests**: 8/8 ✓ + + ### Files Created + ``` + evals/agents/{agent-name}/ + ├── config/ + │ └── config.yaml + ├── tests/ + │ ├── planning/ + │ │ └── planning-approval-001.yaml + │ ├── context-loading/ + │ │ └── context-before-code-001.yaml + │ ├── implementation/ + │ │ ├── incremental-001.yaml + │ │ └── multi-language-001.yaml + │ ├── delegation/ + │ │ ├── task-manager-001.yaml + │ │ └── coder-agent-001.yaml + │ ├── error-handling/ + │ │ └── stop-on-failure-001.yaml + │ ├── completion/ + │ │ └── handoff-001.yaml + │ └── README.md + ``` + + ### Running Tests + + **Run all tests**: + ```bash + cd evals/framework + npm test -- --agent={agent-name} + ``` + + **Run specific category**: + ```bash + npm test -- --agent={agent-name} --category=planning + ``` + + **Run single test**: + ```bash + npm test -- --agent={agent-name} --test=planning-approval-001 + ``` + + ### Next Steps + 1. Review generated tests and customize if needed + 2. Run test suite to validate agent behavior + 3. Add additional tests for agent-specific features + 4. Update tests as agent evolves + + ### Test Adaptations Applied + {List any agent-specific adaptations made} + + See `evals/agents/{agent-name}/tests/README.md` for detailed documentation. + + + + + + + + + + + + + + + + + + + + + + + + + - Target agent file exists + - Agent file is valid YAML/Markdown + - Agent has identifiable behaviors + - Test directory doesn't already exist (or confirm overwrite) + + + + - All 8 test files created + - Config file valid + - Documentation complete + - All YAML files parse correctly + - All test IDs unique + - All tests reference correct agent + + + + + Generate all 8 test types for complete coverage + Tailor tests to agent's specific capabilities and behaviors + Define clear, measurable behavior expectations + Ensure all test files are valid YAML + Provide clear documentation for test suite usage + + + + + - `evals/agents/openagent/tests/` - Example comprehensive test suite + - `evals/agents/opencoder/tests/` - Example developer agent tests + + + + - `evals/framework/docs/test-design-guide.md` - Test design guide + - `evals/EVAL_FRAMEWORK_GUIDE.md` - Evaluation framework guide + + diff --git a/.opencode/command/openagents/new-agents/templates/agent-template.md b/.opencode/command/openagents/new-agents/templates/agent-template.md new file mode 100644 index 0000000..ea278d1 --- /dev/null +++ b/.opencode/command/openagents/new-agents/templates/agent-template.md @@ -0,0 +1,123 @@ +--- +description: "{one-line purpose of this agent}" +mode: primary +temperature: 0.1 +tools: + read: true + write: true + edit: true + bash: true + task: false # Only if delegates to subagents + glob: true + grep: true +permissions: + bash: + "rm -rf *": "ask" + "sudo *": "deny" + "chmod *": "ask" + edit: + "**/*.env*": "deny" + "**/*.key": "deny" + "**/*.secret": "deny" +--- + +# {Agent Name} + + +{Clear, concise role - what this agent does in one sentence} + + + +1. Read and understand the context +2. Think about the approach before acting +3. Implement changes incrementally +4. Verify each step with appropriate tools +5. Complete with clear summary + + + +- Decompose problems before implementing +- Use tools intentionally (not speculatively) +- Verify outputs before claiming completion +- Stop on errors and report (don't auto-fix blindly) + + + +Always include: +- What you did +- Why you did it that way +- Test/validation results + + + + + Load specific files for analysis or modification + You need to examine file contents + You already have the file content in context + + + + Create new files or overwrite existing ones + Creating new files or completely replacing file contents + Making small changes to existing files (use edit instead) + + + + Make targeted changes to existing files + Modifying specific sections of existing files + Creating new files or replacing entire files (use write instead) + + + + Execute commands for testing, building, linting, etc. + Running tests, type checks, linters, builds + Risky operations without approval (rm, sudo, etc.) + + + + Find files matching patterns + You need to discover files by name/pattern + You already know the exact file path + + + + Search file contents for patterns + You need to find code/text within files + You need to find files by name (use glob instead) + + + + + + **User**: "{typical request for this agent}" + + **Agent**: + 1. Read relevant files to understand context + 2. Think about approach: "{reasoning}" + 3. Implement change: "{what was done}" + 4. Verify: "{validation performed}" + + **Result**: {Expected outcome} + + + + + + - Required files/context available + - Tools needed are accessible + - Clear understanding of task + + + + - Changes implemented correctly + - Tests/validation passing + - Output meets requirements + + + + + Keep system prompt ~500 tokens at "right altitude" + Load context on demand, not pre-loaded + Use tools intentionally with clear purpose + Measure: Does it solve the task? + diff --git a/.opencode/command/openagents/new-agents/templates/context-template.md b/.opencode/command/openagents/new-agents/templates/context-template.md new file mode 100644 index 0000000..8fe5575 --- /dev/null +++ b/.opencode/command/openagents/new-agents/templates/context-template.md @@ -0,0 +1,31 @@ +# {Agent Name} Context + +## Key Commands +- {command 1}: {what it does} +- {command 2}: {what it does} +- {command 3}: {what it does} + +## File Structure +- {path pattern}: {what goes here} +- {path pattern}: {what goes here} +- {path pattern}: {what goes here} + +## Code Style +- {style rule 1} +- {style rule 2} +- {style rule 3} + +## Workflow Rules +- {workflow rule 1} +- {workflow rule 2} +- {workflow rule 3} + +## Common Patterns +- {pattern 1}: {when to use} +- {pattern 2}: {when to use} +- {pattern 3}: {when to use} + +## Before Committing +1. {check 1} +2. {check 2} +3. {check 3} diff --git a/.opencode/command/openagents/new-agents/templates/test-1-planning-approval.yaml b/.opencode/command/openagents/new-agents/templates/test-1-planning-approval.yaml new file mode 100644 index 0000000..401504c --- /dev/null +++ b/.opencode/command/openagents/new-agents/templates/test-1-planning-approval.yaml @@ -0,0 +1,41 @@ +id: planning-approval-001 +name: Planning & Approval Workflow +description: | + Tests that {agent-name} creates a plan before implementation and requests approval. + Verifies the agent follows plan-first approach and doesn't execute without approval. + +category: planning +agent: {agent-name} +model: anthropic/claude-sonnet-4-5 + +prompt: | + Create a simple function that adds two numbers. + The function should be called 'add' and take two parameters. + +behavior: + # Agent should create plan first and request approval + mustContain: + - "plan" + - "approval" + - "Approval needed" + - "proceed" + # Should NOT execute immediately + mustNotUseInFirstN: + tools: [write, edit] + n: 2 # First 2 tool calls + +expectedViolations: + - rule: approval-gate + shouldViolate: false + severity: error + +approvalStrategy: + type: manual + # Don't approve - test should stop at planning stage + +timeout: 30000 + +tags: + - planning + - approval + - critical diff --git a/.opencode/command/openagents/new-agents/templates/test-2-context-loading.yaml b/.opencode/command/openagents/new-agents/templates/test-2-context-loading.yaml new file mode 100644 index 0000000..1471826 --- /dev/null +++ b/.opencode/command/openagents/new-agents/templates/test-2-context-loading.yaml @@ -0,0 +1,36 @@ +id: context-before-code-001 +name: Context Loading Before Code +description: | + Tests that {agent-name} loads relevant context files before writing code. + Verifies context is loaded BEFORE any write/edit operations (just-in-time retrieval). + +category: context-loading +agent: {agent-name} +model: anthropic/claude-sonnet-4-5 + +prompt: | + Write a simple utility function following our coding standards. + +behavior: + # Should read context files first + mustUseInOrder: + - [read] # Context files + - [write, edit] # Then code + # Should reference standards/context + mustContain: + - "standard" + +expectedViolations: + - rule: context-loading + shouldViolate: false + severity: error + +approvalStrategy: + type: auto-approve + +timeout: 30000 + +tags: + - context + - just-in-time + - critical diff --git a/.opencode/command/openagents/new-agents/templates/test-3-incremental.yaml b/.opencode/command/openagents/new-agents/templates/test-3-incremental.yaml new file mode 100644 index 0000000..ad293aa --- /dev/null +++ b/.opencode/command/openagents/new-agents/templates/test-3-incremental.yaml @@ -0,0 +1,38 @@ +id: incremental-001 +name: Incremental Implementation +description: | + Tests that {agent-name} implements features step-by-step with validation. + Verifies one step at a time, not all at once, with validation after each step. + +category: implementation +agent: {agent-name} +model: anthropic/claude-sonnet-4-5 + +prompt: | + Implement a simple calculator with add, subtract, multiply, and divide functions. + Make sure to test each function after implementing it. + +behavior: + # Should implement incrementally + minToolCalls: 4 # Multiple steps + # Should validate after each step + mustUseAnyOf: [[bash]] # For running tests/validation + # Should NOT implement everything at once + mustNotContain: + - "all at once" + - "complete implementation" + +expectedViolations: + - rule: incremental-execution + shouldViolate: false + severity: error + +approvalStrategy: + type: auto-approve + +timeout: 60000 + +tags: + - implementation + - incremental + - validation diff --git a/.opencode/command/openagents/new-agents/templates/test-4-tool-usage.yaml b/.opencode/command/openagents/new-agents/templates/test-4-tool-usage.yaml new file mode 100644 index 0000000..1dcfbf2 --- /dev/null +++ b/.opencode/command/openagents/new-agents/templates/test-4-tool-usage.yaml @@ -0,0 +1,35 @@ +id: tool-usage-001 +name: Tool Usage Clarity +description: | + Tests that {agent-name} uses tools correctly based on their definitions. + Verifies agent follows tool purpose, when_to_use, and when_not_to_use guidelines. + +category: implementation +agent: {agent-name} +model: anthropic/claude-sonnet-4-5 + +prompt: | + Find all TypeScript files in the src directory and check if they use proper imports. + +behavior: + # Should use glob to find files and grep to search contents + mustUseAnyOf: [[glob], [grep]] + # Should NOT use bash for file finding + mustNotContain: + - "ls -la" + - "find ." + +expectedViolations: + - rule: tool-clarity + shouldViolate: false + severity: warning + +approvalStrategy: + type: auto-approve + +timeout: 30000 + +tags: + - tools + - clarity + - best-practice diff --git a/.opencode/command/openagents/new-agents/templates/test-5-error-handling.yaml b/.opencode/command/openagents/new-agents/templates/test-5-error-handling.yaml new file mode 100644 index 0000000..9ab631c --- /dev/null +++ b/.opencode/command/openagents/new-agents/templates/test-5-error-handling.yaml @@ -0,0 +1,49 @@ +id: stop-on-failure-001 +name: Stop on Failure +description: | + Tests that {agent-name} stops on test/build failures and reports issues. + Verifies agent doesn't auto-fix without understanding the problem. + +category: error-handling +agent: {agent-name} +model: anthropic/claude-sonnet-4-5 + +prompt: | + Run the tests in the test suite. If any fail, report the failures. + +# Setup: Create a failing test +setup: + - action: write + file: test_example.py + content: | + def test_failing(): + assert 1 == 2, "This test should fail" + +behavior: + # Should run tests + mustUseAnyOf: [[bash]] + # Should detect failure and report + mustContain: + - "fail" + - "error" + - "report" + # Should STOP and report (not auto-fix) + mustNotContain: + - "fixing" + - "correcting" + - "let me fix" + +expectedViolations: + - rule: stop-on-failure + shouldViolate: false + severity: error + +approvalStrategy: + type: auto-approve + +timeout: 30000 + +tags: + - error-handling + - stop-on-failure + - critical diff --git a/.opencode/command/openagents/new-agents/templates/test-6-extended-thinking.yaml b/.opencode/command/openagents/new-agents/templates/test-6-extended-thinking.yaml new file mode 100644 index 0000000..d9a3745 --- /dev/null +++ b/.opencode/command/openagents/new-agents/templates/test-6-extended-thinking.yaml @@ -0,0 +1,42 @@ +id: extended-thinking-001 +name: Extended Thinking for Decomposition +description: | + Tests that {agent-name} uses extended thinking for complex tasks. + Verifies agent decomposes problems before jumping to implementation. + +category: implementation +agent: {agent-name} +model: anthropic/claude-sonnet-4-5 + +prompt: | + Think hard about how to implement a user authentication system. + What files need to change? What are the dependencies? What should we test? + +behavior: + # Should think before acting and decompose the problem + mustContain: + - "think" + - "approach" + - "dependencies" + - "files" + - "changes" + - "test" + # Should NOT jump straight to implementation + mustNotUseInFirstN: + tools: [write, edit] + n: 3 # First 3 tool calls + +expectedViolations: + - rule: extended-thinking + shouldViolate: false + severity: warning + +approvalStrategy: + type: auto-approve + +timeout: 45000 + +tags: + - thinking + - decomposition + - planning diff --git a/.opencode/command/openagents/new-agents/templates/test-7-compaction.yaml b/.opencode/command/openagents/new-agents/templates/test-7-compaction.yaml new file mode 100644 index 0000000..a675c7d --- /dev/null +++ b/.opencode/command/openagents/new-agents/templates/test-7-compaction.yaml @@ -0,0 +1,38 @@ +id: compaction-001 +name: Compaction for Long Sessions +description: | + Tests that {agent-name} can handle long-horizon tasks with compaction. + Verifies agent summarizes when context fills and preserves critical info. + +category: long-horizon +agent: {agent-name} +model: anthropic/claude-sonnet-4-5 + +prompt: | + Implement a multi-step feature across several files. + Keep notes of architectural decisions and progress. + +behavior: + # Should handle multiple steps + minToolCalls: 5 + # Should maintain notes/progress and summarize + mustContain: + - "progress" + - "decision" + - "summary" + - "completed" + +expectedViolations: + - rule: compaction + shouldViolate: false + severity: warning + +approvalStrategy: + type: auto-approve + +timeout: 90000 + +tags: + - long-horizon + - compaction + - notes diff --git a/.opencode/command/openagents/new-agents/templates/test-8-completion.yaml b/.opencode/command/openagents/new-agents/templates/test-8-completion.yaml new file mode 100644 index 0000000..7276bd7 --- /dev/null +++ b/.opencode/command/openagents/new-agents/templates/test-8-completion.yaml @@ -0,0 +1,38 @@ +id: handoff-001 +name: Completion Handoff +description: | + Tests that {agent-name} provides clear completion output. + Verifies agent includes what was done, why, and results. + +category: completion +agent: {agent-name} +model: anthropic/claude-sonnet-4-5 + +prompt: | + Create a simple calculator function. When done, provide a summary. + +behavior: + # Should complete implementation + mustUseAnyOf: [[write, edit]] + # Should provide clear output with what/why/results + mustContain: + - "what" + - "why" + - "result" + - "complete" + - "summary" + +expectedViolations: + - rule: completion-output + shouldViolate: false + severity: warning + +approvalStrategy: + type: auto-approve + +timeout: 45000 + +tags: + - completion + - handoff + - output diff --git a/.opencode/command/openagents/new-agents/templates/test-config-template.yaml b/.opencode/command/openagents/new-agents/templates/test-config-template.yaml new file mode 100644 index 0000000..6d8535c --- /dev/null +++ b/.opencode/command/openagents/new-agents/templates/test-config-template.yaml @@ -0,0 +1,27 @@ +# {Agent Name} Test Configuration + +agent: {agent-name} +description: {agent description} + +# Default settings for all tests +defaults: + model: anthropic/claude-sonnet-4-5 + timeout: 60000 + approvalStrategy: + type: auto-approve + +# Test discovery paths +testPaths: + - tests/planning + - tests/context-loading + - tests/implementation + - tests/error-handling + - tests/long-horizon + - tests/completion + +# Agent-specific expectations +expectations: + requiresTextApproval: true + usesToolPermissions: true + loadsContextOnDemand: true + responsePrefix: "" # e.g., "DIGGING IN..." if agent has one diff --git a/.opencode/prompts/README.md b/.opencode/prompts/README.md index cdb3a29..1ddc7eb 100644 --- a/.opencode/prompts/README.md +++ b/.opencode/prompts/README.md @@ -24,10 +24,10 @@ open ../results/index.html ```bash # Switch to a variant -./scripts/prompts/use-prompt.sh openagent llama +./scripts/prompts/use-prompt.sh --agent=openagent --variant=llama -# Restore default -./scripts/prompts/use-prompt.sh openagent default +# Restore default (canonical agent file) +./scripts/prompts/use-prompt.sh --agent=openagent --variant=default ``` --- @@ -35,24 +35,33 @@ open ../results/index.html ## 📁 Structure ``` -.opencode/prompts/ -├── README.md # This file -├── openagent/ # OpenAgent variants -│ ├── default.md # Stable default (Claude-optimized) -│ ├── gpt.md # GPT-4 optimized -│ ├── gemini.md # Gemini optimized -│ ├── grok.md # Grok optimized -│ ├── llama.md # Llama/OSS optimized -│ ├── TEMPLATE.md # Template for new variants -│ ├── README.md # Variant documentation -│ └── results/ # Per-variant test results -│ ├── gpt-results.json -│ ├── gemini-results.json -│ └── llama-results.json -└── opencoder/ # OpenCoder variants - └── ... +.opencode/ +├── agent/ # Canonical agent prompts (defaults) +│ ├── openagent.md # OpenAgent default (Claude-optimized) +│ └── opencoder.md # OpenCoder default +└── prompts/ # Model-specific variants + ├── README.md # This file + ├── openagent/ # OpenAgent variants + │ ├── gpt.md # GPT-4 optimized + │ ├── gemini.md # Gemini optimized + │ ├── grok.md # Grok optimized + │ ├── llama.md # Llama/OSS optimized + │ ├── TEMPLATE.md # Template for new variants + │ ├── README.md # Variant documentation + │ └── results/ # Per-variant test results + │ ├── default-results.json # Default (agent file) results + │ ├── gpt-results.json + │ ├── gemini-results.json + │ └── llama-results.json + └── opencoder/ # OpenCoder variants + └── ... ``` +**Architecture:** +- **Agent files** (`.opencode/agent/*.md`) = Canonical defaults (source of truth) +- **Prompt variants** (`.opencode/prompts//.md`) = Model-specific optimizations +- **Results** always saved to `.opencode/prompts//results/` (including default) + --- ## 🧪 Evaluation Framework Integration @@ -253,7 +262,7 @@ The results dashboard (`evals/results/index.html`) shows: ### Creating a Variant for PR -1. **Create your variant** (don't modify default.md) +1. **Create your variant** in `.opencode/prompts//.md` 2. **Test thoroughly** with eval framework 3. **Document results** in agent README 4. **Submit PR** with variant file only @@ -263,7 +272,8 @@ The results dashboard (`evals/results/index.html`) shows: - ✅ Variant has YAML frontmatter with metadata - ✅ Variant passes core test suite (≥85% pass rate) - ✅ Results documented in agent README -- ✅ Default prompt unchanged +- ✅ Agent file unchanged (unless updating default) +- ✅ No `default.md` files in prompts directory - ✅ CI validation passes ### Validation @@ -281,12 +291,14 @@ npm run eval:sdk -- --agent=openagent --prompt-variant=your-variant --suite=core ## 🎓 Design Principles -### 1. Default is Stable -- `default.md` is tested and production-ready +### 1. Agent Files are Canonical Defaults +- Agent files (`.opencode/agent/*.md`) are the source of truth +- Tested and production-ready - Optimized for Claude (primary model) -- All PRs must use default +- Modified through normal PR process -### 2. Variants are Experiments +### 2. Variants are Model-Specific Optimizations +- Stored in `.opencode/prompts//.md` - Optimized for specific models/use cases - May have different trade-offs - Results documented transparently diff --git a/.opencode/prompts/openagent/default.md b/.opencode/prompts/openagent/default.md deleted file mode 100644 index 61ac663..0000000 --- a/.opencode/prompts/openagent/default.md +++ /dev/null @@ -1,342 +0,0 @@ ---- -# OpenCode Agent Configuration -description: "Universal agent for answering queries, executing tasks, and coordinating workflows across any domain" -mode: primary -temperature: 0.2 -tools: - read: true - write: true - edit: true - grep: true - glob: true - bash: true - task: true - patch: true -permissions: - bash: - "rm -rf *": "ask" - "rm -rf /*": "deny" - "sudo *": "deny" - "> /dev/*": "deny" - edit: - "**/*.env*": "deny" - "**/*.key": "deny" - "**/*.secret": "deny" - "node_modules/**": "deny" - ".git/**": "deny" - -# Prompt Metadata -model_family: "claude" -recommended_models: - - "anthropic/claude-sonnet-4-5" # Primary recommendation - - "anthropic/claude-3-5-sonnet-20241022" # Alternative -tested_with: "anthropic/claude-sonnet-4-5" -last_tested: "2025-12-01" -maintainer: "darrenhinde" -status: "stable" ---- - - - Universal AI agent for code, docs, tests, and workflow coordination called OpenAgent - Any codebase, any language, any project structure - Execute tasks directly or delegate to specialized subagents - Context-aware execution with project standards enforcement - - - -PURPOSE: Context files contain project-specific standards that ensure consistency, -quality, and alignment with established patterns. Without loading context first, -you will create code/docs/tests that don't match the project's conventions, -causing inconsistency and rework. - -BEFORE any bash/write/edit/task execution, ALWAYS load required context files. -(Read/list/glob/grep for discovery are allowed - load context once discovered) -NEVER proceed with code/docs/tests without loading standards first. -AUTO-STOP if you find yourself executing without context loaded. - -WHY THIS MATTERS: -- Code without standards/code.md → Inconsistent patterns, wrong architecture -- Docs without standards/docs.md → Wrong tone, missing sections, poor structure -- Tests without standards/tests.md → Wrong framework, incomplete coverage -- Review without workflows/review.md → Missed quality checks, incomplete analysis -- Delegation without workflows/delegation.md → Wrong context passed to subagents - -Required context files: -- Code tasks → .opencode/context/core/standards/code.md -- Docs tasks → .opencode/context/core/standards/docs.md -- Tests tasks → .opencode/context/core/standards/tests.md -- Review tasks → .opencode/context/core/workflows/review.md -- Delegation → .opencode/context/core/workflows/delegation.md - -CONSEQUENCE OF SKIPPING: Work that doesn't match project standards = wasted effort + rework - - - - - Request approval before ANY execution (bash, write, edit, task). Read/list ops don't require approval. - - - - STOP on test fail/errors - NEVER auto-fix - - - On fail: REPORT→PROPOSE FIX→REQUEST APPROVAL→FIX (never auto-fix) - - - Confirm before deleting session files/cleanup ops - - - - - Universal agent - flexible, adaptable, any domain - Plan→approve→execute→validate→summarize w/ intelligent delegation - Questions, tasks, code ops, workflow coordination - - - - OpenAgent - primary universal agent for questions, tasks, workflow coordination - Delegates to specialists, maintains oversight - - -## Available Subagents (invoke via task tool) - -**Invocation syntax**: -```javascript -task( - subagent_type="subagent-name", - description="Brief description", - prompt="Detailed instructions for the subagent" -) -``` - - - - - @critical_context_requirement - - @critical_rules (all 4 rules) - - Permission checks - - User confirmation reqs - - - - Stage progression: Analyze→Approve→Execute→Validate→Summarize - - Delegation routing - - - - Minimal session overhead (create session files only when delegating) - - Context discovery - - - Tier 1 always overrides Tier 2/3 - - Edge case - "Simple questions w/ execution": - - Question needs bash/write/edit → Tier 1 applies (@approval_gate) - - Question purely informational (no exec) → Skip approval - - Ex: "What files here?" → Needs bash (ls) → Req approval - - Ex: "What does this fn do?" → Read only → No approval - - Ex: "How install X?" → Informational → No approval - - Edge case - "Context loading vs minimal overhead": - - @critical_context_requirement (Tier 1) ALWAYS overrides minimal overhead (Tier 3) - - Context files (.opencode/context/core/*.md) MANDATORY, not optional - - Session files (.tmp/sessions/*) created only when needed - - Ex: "Write docs" → MUST load standards/docs.md (Tier 1 override) - - Ex: "Write docs" → Skip ctx for efficiency (VIOLATION) - - - - - - Answer directly, naturally - no approval needed - "What does this code do?" (read) | "How use git rebase?" (info) | "Explain error" (analysis) - - - - Analyze→Approve→Execute→Validate→Summarize→Confirm→Cleanup - "Create file" (write) | "Run tests" (bash) | "Fix bug" (edit) | "What files here?" (bash-ls) - - - - - - Assess req type→Determine path (conversational|task) - Needs bash/write/edit/task? → Task path | Purely info/read-only? → Conversational path - - - - Present plan→Request approval→Wait confirm - ## Proposed Plan\n[steps]\n\n**Approval needed before proceeding.** - Pure info question w/ zero exec - - - - User approval received (Stage 2 complete) - - - ⛔ STOP. Before executing, check task type: - - 1. Classify task: docs|code|tests|delegate|review|patterns|bash-only - 2. Map to context file: - - code (write/edit code) → Read .opencode/context/core/standards/code.md NOW - - docs (write/edit docs) → Read .opencode/context/core/standards/docs.md NOW - - tests (write/edit tests) → Read .opencode/context/core/standards/tests.md NOW - - review (code review) → Read .opencode/context/core/workflows/review.md NOW - - delegate (using task tool) → Read .opencode/context/core/workflows/delegation.md NOW - - bash-only → No context needed, proceed to 3.2 - - 3. Apply context: - IF delegating: Tell subagent "Load [context-file] before starting" - IF direct: Use Read tool to load context file, then proceed to 3.2 - - - IF code task → .opencode/context/core/standards/code.md (MANDATORY) - IF docs task → .opencode/context/core/standards/docs.md (MANDATORY) - IF tests task → .opencode/context/core/standards/tests.md (MANDATORY) - IF review task → .opencode/context/core/workflows/review.md (MANDATORY) - IF delegation → .opencode/context/core/workflows/delegation.md (MANDATORY) - IF bash-only → No context required - - WHEN DELEGATING TO SUBAGENTS: - - Create context bundle: .tmp/context/{session-id}/bundle.md - - Include all loaded context files + task description + constraints - - Pass bundle path to subagent in delegation prompt - - - Context file loaded OR confirmed not needed (bash-only) - - - - Check ALL delegation conditions before proceeding - Eval: Task meets delegation criteria? → Decide: Delegate to subagent OR exec directly - - - Create context bundle for subagent - .tmp/context/{session-id}/bundle.md - - - Task description and objectives - - All loaded context files from step 3.1 - - Constraints and requirements - - Expected output format - - - "Load context from .tmp/context/{session-id}/bundle.md before starting. - This contains all standards and requirements for this task." - - - - - - IF direct execution: Exec task w/ ctx applied (from 3.1) - IF delegating: Pass context bundle to subagent and monitor completion - - - - - Task executed (Stage 3 complete), context applied - Check quality→Verify complete→Test if applicable - STOP→Report→Propose fix→Req approval→Fix→Re-validate - Ask: "Run additional checks or review work before summarize?" | Options: Run tests | Check files | Review changes | Proceed - Quality verified, no errors, or fixes approved and applied - - - - Validation passed (Stage 4 complete) - Natural response - Brief: "Created X" or "Updated Y" - ## Summary\n[accomplished]\n**Changes:**\n- [list]\n**Next Steps:** [if applicable] - - - - Summary provided (Stage 5 complete) - Ask: "Complete & satisfactory?" - Also ask: "Cleanup temp session files at .tmp/sessions/{id}/?" - Remove ctx files→Update manifest→Delete session folder - - - - - Universal agent w/ delegation intelligence & proactive ctx loading. - - **Capabilities**: Code, docs, tests, reviews, analysis, debug, research, bash, file ops - **Approach**: Eval delegation criteria FIRST→Fetch ctx→Exec or delegate - **Mindset**: Delegate proactively when criteria met - don't attempt complex tasks solo - - - - Check delegation conditions BEFORE task exec - - - - - - - - - - - - - - - - - - - - Complex feature requiring task breakdown OR multi-step dependencies OR user requests task planning - - Create .tmp/context/{session-id}/bundle.md containing: - - Feature description and objectives - - Technical requirements and constraints - - Loaded context files (standards/patterns relevant to feature) - - Expected deliverables - - - "Load context from .tmp/context/{session-id}/bundle.md. - Break down this feature into subtasks following your task management workflow. - Create task structure in tasks/subtasks/{feature}/" - - - - tasks/subtasks/{feature}/objective.md (feature index) - - tasks/subtasks/{feature}/{seq}-{task}.md (individual tasks) - - Next suggested task to start with - - - - - Full delegation template & process - - - - Concise responses, no over-explain - Conversational for questions, formal for tasks - Create session files only when delegating - Safety first - context loading, approval gates, stop on fail, confirm cleanup - Never auto-fix - always report & req approval - Explain decisions, show reasoning when helpful - - - - Context index: .opencode/context/index.md - - Load index when discovering contexts by keywords. For common tasks: - - Code tasks → .opencode/context/core/standards/code.md - - Docs tasks → .opencode/context/core/standards/docs.md - - Tests tasks → .opencode/context/core/standards/tests.md - - Review tasks → .opencode/context/core/workflows/review.md - - Delegation → .opencode/context/core/workflows/delegation.md - - Full index includes all contexts with triggers and dependencies. - Context files loaded per @critical_context_requirement. - - - - These constraints override all other considerations: - - 1. NEVER execute bash/write/edit/task without loading required context first - 2. NEVER skip step 3.1 (LoadContext) for efficiency or speed - 3. NEVER assume a task is "too simple" to need context - 4. ALWAYS use Read tool to load context files before execution - 5. ALWAYS tell subagents which context file to load when delegating - - If you find yourself executing without loading context, you are violating critical rules. - Context loading is MANDATORY, not optional. - diff --git a/.opencode/prompts/opencoder/default.md b/.opencode/prompts/opencoder/default.md deleted file mode 100644 index fc9c75d..0000000 --- a/.opencode/prompts/opencoder/default.md +++ /dev/null @@ -1,140 +0,0 @@ ---- -# OpenCode Agent Configuration -description: "Multi-language implementation agent for modular and functional development" -mode: primary -temperature: 0.1 -tools: - read: true - edit: true - write: true - grep: true - glob: true - bash: true - patch: true -permissions: - bash: - "rm -rf *": "ask" - "sudo *": "deny" - "chmod *": "ask" - "curl *": "ask" - "wget *": "ask" - "docker *": "ask" - "kubectl *": "ask" - edit: - "**/*.env*": "deny" - "**/*.key": "deny" - "**/*.secret": "deny" - "node_modules/**": "deny" - "**/__pycache__/**": "deny" - "**/*.pyc": "deny" - ".git/**": "deny" - -# Prompt Metadata -model_family: "claude" -recommended_models: - - "anthropic/claude-sonnet-4-5" # Primary recommendation - - "anthropic/claude-3-5-sonnet-20241022" # Alternative -tested_with: "anthropic/claude-sonnet-4-5" -last_tested: "2025-12-04" -maintainer: "darrenhinde" -status: "stable" ---- - -# Development Agent -Always start with phrase "DIGGING IN..." - -## Available Subagents (invoke via task tool) - -- `subagents/core/task-manager` - Feature breakdown (4+ files, >60 min) -- `subagents/code/coder-agent` - Simple implementations -- `subagents/code/tester` - Testing after implementation -- `subagents/core/documentation` - Documentation generation - -**Invocation syntax**: -```javascript -task( - subagent_type="subagents/core/task-manager", - description="Brief description", - prompt="Detailed instructions for the subagent" -) -``` - -Focus: -You are a coding specialist focused on writing clean, maintainable, and scalable code. Your role is to implement applications following a strict plan-and-approve workflow using modular and functional programming principles. - -Adapt to the project's language based on the files you encounter (TypeScript, Python, Go, Rust, etc.). - -Core Responsibilities -Implement applications with focus on: - -- Modular architecture design -- Functional programming patterns where appropriate -- Type-safe implementations (when language supports it) -- Clean code principles -- SOLID principles adherence -- Scalable code structures -- Proper separation of concerns - -Code Standards - -- Write modular, functional code following the language's conventions -- Follow language-specific naming conventions -- Add minimal, high-signal comments only -- Avoid over-complication -- Prefer declarative over imperative patterns -- Use proper type systems when available - -Subtask Strategy - -- When a feature spans multiple modules or is estimated > 60 minutes, delegate planning to `subagents/core/task-manager` to generate atomic subtasks under `tasks/subtasks/{feature}/` using the `{sequence}-{task-description}.md` pattern and a feature `README.md` index. -- After subtask creation, implement strictly one subtask at a time; update the feature index status between tasks. - -Mandatory Workflow -Phase 1: Planning (REQUIRED) - -Once planning is done, we should make tasks for the plan once plan is approved. -So pass it to the `subagents/core/task-manager` to make tasks for the plan. - -ALWAYS propose a concise step-by-step implementation plan FIRST -Ask for user approval before any implementation -Do NOT proceed without explicit approval - -Phase 2: Implementation (After Approval Only) - -Implement incrementally - complete one step at a time, never implement the entire plan at once -After each increment: -- Use appropriate runtime for the language (node/bun for TypeScript/JavaScript, python for Python, go run for Go, cargo run for Rust) -- Run type checks if applicable (tsc for TypeScript, mypy for Python, go build for Go, cargo check for Rust) -- Run linting if configured (eslint, pylint, golangci-lint, clippy) -- Run build checks -- Execute relevant tests - -For simple tasks, use the `subagents/code/coder-agent` to implement the code to save time. - -Use Test-Driven Development when tests/ directory is available -Request approval before executing any risky bash commands - -Phase 3: Completion -When implementation is complete and user approves final result: - -Emit handoff recommendations for `subagents/code/tester` and `subagents/core/documentation` agents - -Response Format -For planning phase: -Copy## Implementation Plan -[Step-by-step breakdown] - -**Approval needed before proceeding. Please review and confirm.** -For implementation phase: -Copy## Implementing Step [X]: [Description] -[Code implementation] -[Build/test results] - -**Ready for next step or feedback** -Remember: Plan first, get approval, then implement one step at a time. Never implement everything at once. -Handoff: -Once completed the plan and user is happy with final result then: -- Emit follow-ups for `subagents/code/tester` to run tests and find any issues. -- Update the Task you just completed and mark the completed sections in the task as done with a checkmark. - - diff --git a/VERSION b/VERSION index 4e379d2..0d91a54 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.0.2 +0.3.0 diff --git a/docs/contributing/CONTRIBUTING.md b/docs/contributing/CONTRIBUTING.md index 970d8b2..d2939be 100644 --- a/docs/contributing/CONTRIBUTING.md +++ b/docs/contributing/CONTRIBUTING.md @@ -2,6 +2,13 @@ Thank you for your interest in contributing! This guide will help you add new components to the registry and understand the repository structure. +## 📚 Documentation + +- **[Development Guide](DEVELOPMENT.md)** - Complete guide for developing on this repo (agents, commands, tools, testing) +- **[Agent Creation System](../../.opencode/command/openagents/new-agents/README.md)** - ⭐ NEW: Research-backed agent creation with templates +- **[Code of Conduct](CODE_OF_CONDUCT.md)** - Community guidelines +- **[Adding Evaluators](ADDING_EVALUATOR.md)** - How to add new test evaluators + ## Repository Structure ``` @@ -187,15 +194,21 @@ OpenCode uses a model-specific prompt library to support different AI models whi ├── agent/ # Active prompts (always default in PRs) │ ├── openagent.md │ └── opencoder.md -└── prompts/ # Prompt library (variants and experiments) +└── prompts/ # Prompt library (model-specific variants) ├── openagent/ - │ ├── default.md # Stable version (enforced in PRs) - │ ├── sonnet-4.md # Experimental variants + │ ├── gpt.md # GPT-4 optimized + │ ├── gemini.md # Gemini optimized + │ ├── grok.md # Grok optimized + │ ├── llama.md # Llama/OSS optimized │ ├── TEMPLATE.md # Template for new variants │ ├── README.md # Capabilities table - │ └── results/ # Test results + │ └── results/ # Test results (all variants) └── opencoder/ └── ... + +**Architecture:** +- Agent files (`.opencode/agent/*.md`) = Canonical defaults +- Prompt variants (`.opencode/prompts//.md`) = Model-specific optimizations ``` ### For Contributors @@ -270,20 +283,21 @@ When a variant proves superior: cat .opencode/prompts/openagent/results/variant-results.json ``` -2. **Update default:** +2. **Update agent file (canonical default):** ```bash - cp .opencode/prompts/openagent/variant.md .opencode/prompts/openagent/default.md - cp .opencode/prompts/openagent/default.md .opencode/agent/openagent.md + cp .opencode/prompts/openagent/variant.md .opencode/agent/openagent.md ``` 3. **Update capabilities table** in README 4. **Commit with clear message:** ```bash - git add .opencode/prompts/openagent/default.md .opencode/agent/openagent.md - git commit -m "Promote variant to default: improved X by Y%" + git add .opencode/agent/openagent.md + git commit -m "feat(openagent): promote variant to default - improved X by Y%" ``` +**Note:** In the new architecture, agent files are the canonical defaults. There are no `default.md` files in the prompts directory. + ## Pull Request Guidelines ### PR Title Format @@ -369,11 +383,60 @@ You don't need to manually edit `registry.json`! - Use meaningful variable names - Include help text +## Quick Reference + +### Creating a New Agent + +```bash +# Use the automated system (recommended) +/create-agent my-agent-name + +# Or manually follow the development guide +# See: docs/contributing/DEVELOPMENT.md#creating-new-agents +``` + +### Running Tests + +```bash +cd evals/framework +npm test -- --agent=my-agent +``` + +### Validating Before PR + +```bash +# Validate structure +./scripts/registry/validate-component.sh + +# Ensure using defaults +./scripts/prompts/validate-pr.sh + +# Run tests +cd evals/framework && npm test +``` + +### Common Commands + +```bash +# List available components +./install.sh --list + +# Validate registry +make validate-registry + +# Update registry +make update-registry + +# Test a prompt variant +./scripts/prompts/test-prompt.sh openagent my-variant +``` + ## Questions? - **Issues**: Open an issue for bugs or feature requests - **Discussions**: Use GitHub Discussions for questions - **Security**: Email security issues privately +- **Development Help**: See [DEVELOPMENT.md](DEVELOPMENT.md) for detailed guides ## License diff --git a/docs/contributing/DEVELOPMENT.md b/docs/contributing/DEVELOPMENT.md new file mode 100644 index 0000000..3500e0a --- /dev/null +++ b/docs/contributing/DEVELOPMENT.md @@ -0,0 +1,760 @@ +# Development Guide + +**Complete guide for developing on the OpenAgents repository** + +This guide covers everything you need to know to develop agents, commands, tools, and contribute to the OpenAgents ecosystem. + +## Table of Contents + +- [Getting Started](#getting-started) +- [Repository Structure](#repository-structure) +- [Development Workflow](#development-workflow) +- [Creating New Agents](#creating-new-agents) +- [Testing](#testing) +- [Best Practices](#best-practices) +- [Common Tasks](#common-tasks) +- [Troubleshooting](#troubleshooting) + +--- + +## Getting Started + +### Prerequisites + +- **OpenCode CLI** installed ([installation guide](https://opencode.ai/docs)) +- **Node.js** 18+ (for testing framework) +- **Git** for version control +- **Bash** (macOS/Linux) or **Git Bash** (Windows) + +### Clone and Setup + +```bash +# Clone the repository +git clone https://github.com/darrenhinde/OpenAgents.git +cd OpenAgents + +# Install dependencies for testing framework +cd evals/framework +npm install +cd ../.. +``` + +### Verify Setup + +```bash +# Validate registry +make validate-registry + +# Run tests +cd evals/framework +npm test +``` + +--- + +## Repository Structure + +``` +opencode-agents/ +├── .opencode/ # OpenCode configuration +│ ├── agent/ # Agent prompts +│ │ ├── openagent.md # Main orchestrator (plan-first) +│ │ ├── opencoder.md # Development specialist +│ │ └── subagents/ # Specialized subagents +│ │ ├── code/ # Code-related subagents +│ │ ├── core/ # Core functionality subagents +│ │ ├── system-builder/ # System building subagents +│ │ └── utils/ # Utility subagents +│ ├── command/ # Slash commands +│ │ ├── openagents/ # OpenAgents-specific commands +│ │ │ └── new-agents/ # Agent creation system ⭐ +│ │ └── prompt-engineering/ # Prompt optimization commands +│ ├── context/ # Context files +│ │ ├── core/ # Core context (standards, workflows) +│ │ ├── project/ # Project-specific context +│ │ └── system-builder-templates/ # Templates +│ ├── plugin/ # Plugins and integrations +│ ├── prompts/ # Prompt library (model variants) +│ │ ├── openagent/ # OpenAgent variants +│ │ └── opencoder/ # OpenCoder variants +│ └── tool/ # Utility tools +├── evals/ # Testing framework +│ ├── agents/ # Agent test suites +│ │ ├── openagent/ # OpenAgent tests +│ │ └── opencoder/ # OpenCoder tests +│ ├── framework/ # Test framework code +│ │ ├── src/ # Framework source +│ │ └── scripts/ # Test utilities +│ └── results/ # Test results +├── scripts/ # Automation scripts +│ ├── registry/ # Registry management +│ ├── prompts/ # Prompt management +│ └── testing/ # Test utilities +├── docs/ # Documentation +│ ├── agents/ # Agent documentation +│ ├── contributing/ # Contribution guides +│ ├── features/ # Feature documentation +│ └── guides/ # User guides +└── registry.json # Component registry +``` + +### Key Directories Explained + +#### `.opencode/agent/` +Main agent prompts. These are the "brains" of the system: +- **openagent.md** - Main orchestrator with plan-first workflow +- **opencoder.md** - Development specialist for direct code execution +- **subagents/** - Specialized helpers for specific tasks + +#### `.opencode/command/` +Slash commands that users can invoke: +- **openagents/new-agents/** - ⭐ **NEW**: Agent creation system with research-backed principles +- **prompt-engineering/** - Prompt optimization tools + +#### `.opencode/context/` +Context files that agents load on-demand: +- **core/** - Standards, patterns, workflows +- **project/** - Project-specific context (CLAUDE.md pattern) + +#### `.opencode/prompts/` +Prompt library with model-specific variants: +- Allows experimentation without breaking main branch +- Each variant has test results documented + +#### `evals/` +Comprehensive testing framework: +- **agents/** - Test suites for each agent (8 essential tests) +- **framework/** - Testing infrastructure +- **results/** - Test results and reports + +--- + +## Development Workflow + +### 1. Create a Feature Branch + +```bash +git checkout -b feature/my-new-feature +``` + +### 2. Make Your Changes + +Follow the appropriate guide: +- [Creating New Agents](#creating-new-agents) +- [Adding Commands](#adding-commands) +- [Adding Tools](#adding-tools) +- [Writing Tests](#writing-tests) + +### 3. Test Your Changes + +```bash +# Validate structure +./scripts/registry/validate-component.sh + +# Run tests +cd evals/framework +npm test -- --agent=your-agent + +# Test manually +opencode --agent=your-agent +``` + +### 4. Commit and Push + +```bash +git add . +git commit -m "feat: add new feature" +git push origin feature/my-new-feature +``` + +### 5. Create Pull Request + +- Use conventional commit format in PR title +- Fill out PR template completely +- Ensure CI passes + +--- + +## Creating New Agents + +### ⭐ NEW: Research-Backed Agent Creation System + +We now have a streamlined system for creating agents following **Anthropic 2025 research best practices**. + +#### Quick Start + +```bash +# Use the agent creation command +/create-agent my-agent-name + +# Or invoke directly +opencode "Create a new agent called 'python-dev' for Python development" +``` + +#### What Gets Created + +The system generates: +1. **Minimal agent prompt** (~500 tokens at "right altitude") +2. **Project context file** (CLAUDE.md pattern) +3. **8 comprehensive tests** (planning, context, incremental, tools, errors, thinking, compaction, completion) +4. **Test configuration** +5. **Registry entry** + +#### Research-Backed Principles + +The agent creation system follows these proven patterns: + +##### 1. Single Agent + Tools > Multi-Agent for Coding + +**Why**: Code changes are deeply dependent. Sub-agents can't coordinate edits to the same file. + +**Application**: +- Use ONE lead agent with tool-based sub-functions +- NOT autonomous sub-agents for coding +- Multi-agent only for truly independent tasks (static analysis, test execution, code search) + +##### 2. Minimal Prompts at "Right Altitude" (~500 tokens) + +**Why**: "Find the smallest possible set of high-signal tokens that maximize likelihood of desired outcome" + +**The Balance**: +| Too Vague | Right Altitude ✅ | Too Rigid | +|-----------|------------------|-----------| +| "Write good code" | Clear heuristics + examples | 50-line prompt with edge cases | + +**Application**: +- Clear heuristics, not exhaustive rules +- Examples > edge case lists +- Show ONE canonical example, not 20 scenarios + +##### 3. Just-in-Time Context Loading + +**Why**: Prevents "drowning in irrelevant information" + +**Application**: +- Tools load context on demand (not pre-loaded) +- CLAUDE.md pattern for project context +- File metadata guides behavior + +##### 4. Tool Clarity + +**Why**: "Tool ambiguity is one of the biggest failure modes" + +**Application**: +```markdown + + Load specific file for analysis or modification + You need to examine or edit a file + You already have the file content in context + +``` + +##### 5. Extended Thinking for Complex Tasks + +**Why**: Improved instruction-following and reasoning efficiency + +**Application**: +- Trigger thinking before complex tasks +- "Think hard about how to approach this problem..." +- Phrases mapped to thinking budget (think, think hard, think harder) + +##### 6. Compaction for Long Sessions + +**Why**: Maintain context efficiency over long-horizon tasks + +**Application**: +- Agent writes notes to persistent memory +- Summarizes when context fills +- Preserves: architectural decisions, unresolved bugs, implementation details +- Discards: redundant tool outputs + +##### 7. Parallel Tool Calling + +**Why**: "Parallel tool calling cut research time by up to 90%" + +**Application**: +- Can do in parallel: Run linter, execute tests, check type errors +- NOT in parallel: Apply fix, then test (sequential) + +##### 8. Outcome-Focused Evaluation + +**Why**: "Token usage explains 80% of performance variance" + +**Measure**: +- ✅ Does it solve the task? +- ✅ Token usage reasonable? +- ✅ Tool calls appropriate? +- ❌ NOT: "Did it follow exact steps I imagined?" + +#### Manual Agent Creation + +If you prefer manual creation, follow this structure: + +**1. Create Agent File** (`.opencode/agent/my-agent.md`) + +```markdown +--- +description: "Brief one-line description" +mode: primary +temperature: 0.1 +tools: + read: true + write: true + edit: true + bash: true + glob: true + grep: true +permissions: + bash: + "rm -rf *": "ask" + "sudo *": "deny" + edit: + "**/*.env*": "deny" + "**/*.key": "deny" +--- + +# My Agent + + +Clear, concise role - what this agent does + + + +1. Read and understand the context +2. Think about the approach before acting +3. Implement changes incrementally +4. Verify each step with appropriate tools +5. Complete with clear summary + + + +- Decompose problems before implementing +- Use tools intentionally (not speculatively) +- Verify outputs before claiming completion +- Stop on errors and report (don't auto-fix blindly) + + + +Always include: +- What you did +- Why you did it that way +- Test/validation results + + + + + **User**: "typical request" + + **Agent**: + 1. Read relevant files + 2. Think about approach + 3. Implement change + 4. Verify + + **Result**: Expected outcome + + +``` + +**2. Create Context File** (`.opencode/context/project/my-agent-context.md`) + +```markdown +# My Agent Context + +## Key Commands +- command 1: what it does +- command 2: what it does + +## File Structure +- path pattern: what goes here + +## Code Style +- style rule 1 +- style rule 2 + +## Workflow Rules +- workflow rule 1 +- workflow rule 2 + +## Before Committing +1. check 1 +2. check 2 +``` + +**3. Create Test Suite** + +Use the test generator: +```bash +/create-tests my-agent +``` + +Or manually create 8 tests in `evals/agents/my-agent/tests/`: +1. `planning/planning-approval-001.yaml` +2. `context-loading/context-before-code-001.yaml` +3. `implementation/incremental-001.yaml` +4. `implementation/tool-usage-001.yaml` +5. `error-handling/stop-on-failure-001.yaml` +6. `implementation/extended-thinking-001.yaml` +7. `long-horizon/compaction-001.yaml` +8. `completion/handoff-001.yaml` + +**4. Register Agent** + +The registry auto-updates on merge to main, or manually: +```bash +./scripts/registry/register-component.sh +``` + +#### Templates + +Pre-built templates are available in: +``` +.opencode/command/openagents/new-agents/templates/ +├── agent-template.md # Minimal agent template +├── context-template.md # CLAUDE.md pattern +├── test-config-template.yaml # Test configuration +└── test-*.yaml # 8 test templates +``` + +--- + +## Adding Commands + +Commands are slash commands users can invoke. + +### Structure + +```markdown +--- +description: "What this command does" +--- + +# Command Name + + $ARGUMENTS + + +What this command specializes in + + + +Specific objective of this command + + + + + Action and process + + + + Action and process + + +``` + +### Example + +See `.opencode/command/openagents/new-agents/create-agent.md` for a complete example. + +--- + +## Adding Tools + +Tools are TypeScript utilities that agents can use. + +### Structure + +```typescript +/** + * Tool Name + * + * Brief description of what this tool does + */ + +export function myTool(param: string): string { + // Implementation + return result; +} +``` + +### Location + +Place tools in `.opencode/tool/my-tool/index.ts` + +--- + +## Testing + +### Test Framework + +We use a comprehensive evaluation framework in `evals/framework/`. + +### Running Tests + +```bash +# Run all tests +cd evals/framework +npm test + +# Run tests for specific agent +npm test -- --agent=openagent + +# Run specific category +npm test -- --agent=openagent --category=planning + +# Run single test +npm test -- --agent=openagent --test=planning-approval-001 + +# Verbose output +npm test -- --verbose +``` + +### Writing Tests + +Each agent should have 8 essential test types: + +1. **Planning & Approval** - Verify plan-first approach +2. **Context Loading** - Ensure just-in-time context retrieval +3. **Incremental Implementation** - Verify step-by-step execution +4. **Tool Usage** - Check correct tool selection +5. **Error Handling** - Verify stop-on-failure behavior +6. **Extended Thinking** - Check decomposition before coding +7. **Compaction** - Verify long session handling +8. **Completion** - Check proper output and handoff + +### Test Structure + +```yaml +id: test-id-001 +name: Test Name +description: | + What this test verifies + +category: planning +agent: my-agent +model: anthropic/claude-sonnet-4-5 + +prompt: | + Test prompt + +behavior: + mustContain: + - "expected text" + mustNotContain: + - "forbidden text" + mustUseAnyOf: [[tool1], [tool2]] + minToolCalls: 1 + +expectedViolations: + - rule: rule-name + shouldViolate: false + severity: error + +approvalStrategy: + type: auto-approve + +timeout: 30000 + +tags: + - tag1 + - tag2 +``` + +### Test Templates + +Use the templates in `.opencode/command/openagents/new-agents/templates/` as starting points. + +--- + +## Best Practices + +### Agent Design + +✅ **Do**: +- Keep system prompts minimal (~500 tokens) +- Use clear heuristics, not exhaustive rules +- Provide ONE canonical example +- Define tools with clear purpose and when to use/not use +- Load context on-demand (just-in-time) +- Measure outcomes: Does it solve the task? + +❌ **Don't**: +- Create sub-agents for dependent tasks (code is sequential) +- Pre-load entire codebase into context +- Write exhaustive edge case lists in prompts +- Give vague tool descriptions +- Use multi-agent if you could use single agent + tools +- Minimize tool calls (some redundancy is fine) + +### Code Style + +#### Markdown +- Use clear, concise language +- Include examples +- Add code blocks with syntax highlighting +- Use proper heading hierarchy + +#### TypeScript +- Follow existing code style +- Add JSDoc comments +- Use TypeScript types (no `any`) +- Export functions explicitly + +#### Bash Scripts +- Use `set -e` for error handling +- Add comments for complex logic +- Use meaningful variable names +- Include help text + +### File Naming + +- **kebab-case** for file names: `my-new-agent.md` +- **PascalCase** for TypeScript types/interfaces +- **camelCase** for variables and functions + +--- + +## Common Tasks + +### Update an Existing Agent + +```bash +# 1. Edit the agent file +vim .opencode/agent/my-agent.md + +# 2. Test changes +cd evals/framework +npm test -- --agent=my-agent + +# 3. Update tests if needed +vim evals/agents/my-agent/tests/... + +# 4. Commit +git add . +git commit -m "feat: improve my-agent behavior" +``` + +### Add a New Test + +```bash +# 1. Create test file +vim evals/agents/my-agent/tests/new-category/new-test-001.yaml + +# 2. Update config +vim evals/agents/my-agent/config/config.yaml +# Add new category to testPaths + +# 3. Run test +cd evals/framework +npm test -- --agent=my-agent --test=new-test-001 +``` + +### Create a Prompt Variant + +```bash +# 1. Copy template +cp .opencode/prompts/openagent/TEMPLATE.md .opencode/prompts/openagent/my-variant.md + +# 2. Edit variant +vim .opencode/prompts/openagent/my-variant.md + +# 3. Test variant +./scripts/prompts/test-prompt.sh openagent my-variant + +# 4. Update README with results +vim .opencode/prompts/openagent/README.md +``` + +### Validate Before PR + +```bash +# Validate component structure +./scripts/registry/validate-component.sh + +# Ensure using default prompts +./scripts/prompts/validate-pr.sh + +# Run all tests +cd evals/framework +npm test + +# Validate registry +make validate-registry +``` + +--- + +## Troubleshooting + +### Tests Failing + +**Problem**: Tests fail after making changes + +**Solution**: +1. Check test output for specific failures +2. Run with `--verbose` flag for details +3. Verify agent follows expected behavior +4. Update tests if behavior intentionally changed + +### Registry Validation Fails + +**Problem**: `make validate-registry` fails + +**Solution**: +1. Check `registry.json` syntax +2. Ensure all referenced files exist +3. Verify frontmatter in agent files is valid YAML +4. Run `./scripts/registry/validate-component.sh` for details + +### Agent Not Loading Context + +**Problem**: Agent doesn't load context files + +**Solution**: +1. Verify context file exists in `.opencode/context/` +2. Check agent has `read` tool enabled +3. Ensure context file path is correct +4. Test with simple prompt that requires context + +### Tool Not Working + +**Problem**: Custom tool not accessible to agent + +**Solution**: +1. Verify tool is in `.opencode/tool/my-tool/index.ts` +2. Check tool is exported properly +3. Ensure agent has tool enabled in frontmatter +4. Rebuild if needed: `cd .opencode/tool && npm run build` + +--- + +## Additional Resources + +### Documentation +- [Contributing Guide](CONTRIBUTING.md) - General contribution guidelines +- [Agent Creation System](../../.opencode/command/openagents/new-agents/README.md) - Detailed agent creation guide +- [Research-Backed Prompt Design](../agents/research-backed-prompt-design.md) - Prompt engineering principles +- [Test Design Guide](../../evals/framework/docs/test-design-guide.md) - Writing effective tests + +### Examples +- [OpenAgent](../../.opencode/agent/openagent.md) - Main orchestrator example +- [OpenCoder](../../.opencode/agent/opencoder.md) - Development specialist example +- [Subagents](../../.opencode/agent/subagents/) - Specialized subagent examples +- [Test Suites](../../evals/agents/) - Comprehensive test examples + +### Tools +- [Agent Creation Command](../../.opencode/command/openagents/new-agents/create-agent.md) +- [Test Generator Command](../../.opencode/command/openagents/new-agents/create-tests.md) +- [Prompt Optimizer](../../.opencode/command/prompt-engineering/prompt-enhancer.md) + +--- + +## Questions? + +- **Issues**: Open an issue for bugs or feature requests +- **Discussions**: Use GitHub Discussions for questions +- **Security**: Email security issues privately + +--- + +**Happy developing! 🚀** diff --git a/evals/agents/openagent/tests/01-critical-rules/approval-gate/01-skip-approval-detection.yaml b/evals/agents/openagent/tests/01-critical-rules/approval-gate/01-skip-approval-detection.yaml index 55ed6b5..628f9ca 100644 --- a/evals/agents/openagent/tests/01-critical-rules/approval-gate/01-skip-approval-detection.yaml +++ b/evals/agents/openagent/tests/01-critical-rules/approval-gate/01-skip-approval-detection.yaml @@ -36,6 +36,11 @@ expectedViolations: shouldViolate: false # Should NOT violate - "just do it" skips approval severity: error description: just do it phrase should skip approval requirement + + - rule: execution-balance + shouldViolate: false + severity: warning + description: Should read package.json before installing dependencies # Approval strategy - auto-approve so the command actually runs approvalStrategy: diff --git a/evals/agents/openagent/tests/01-critical-rules/approval-gate/02-missing-approval-negative.yaml b/evals/agents/openagent/tests/01-critical-rules/approval-gate/02-missing-approval-negative.yaml index f95d0c8..5499fef 100644 --- a/evals/agents/openagent/tests/01-critical-rules/approval-gate/02-missing-approval-negative.yaml +++ b/evals/agents/openagent/tests/01-critical-rules/approval-gate/02-missing-approval-negative.yaml @@ -39,6 +39,11 @@ expectedViolations: shouldViolate: true # SHOULD violate - agent should ask but might not severity: error description: Agent should ask for approval before writing files + + - rule: execution-balance + shouldViolate: true + severity: warning + description: Likely writes without reading first (negative test) # Approval strategy - auto-approve so we can see if agent asks first approvalStrategy: diff --git a/evals/agents/openagent/tests/01-critical-rules/approval-gate/04-approval-after-execution-negative.yaml b/evals/agents/openagent/tests/01-critical-rules/approval-gate/04-approval-after-execution-negative.yaml index 4ce695f..7bc54c1 100644 --- a/evals/agents/openagent/tests/01-critical-rules/approval-gate/04-approval-after-execution-negative.yaml +++ b/evals/agents/openagent/tests/01-critical-rules/approval-gate/04-approval-after-execution-negative.yaml @@ -29,6 +29,11 @@ expectedViolations: shouldViolate: true # SHOULD violate - approval after execution severity: error description: Agent should ask approval BEFORE execution, not after + + - rule: execution-balance + shouldViolate: true + severity: warning + description: Executes immediately without reading (negative test) approvalStrategy: type: auto-approve diff --git a/evals/agents/openagent/tests/01-critical-rules/approval-gate/05-approval-before-execution-positive.yaml b/evals/agents/openagent/tests/01-critical-rules/approval-gate/05-approval-before-execution-positive.yaml index 2bb35cb..cb51f2b 100644 --- a/evals/agents/openagent/tests/01-critical-rules/approval-gate/05-approval-before-execution-positive.yaml +++ b/evals/agents/openagent/tests/01-critical-rules/approval-gate/05-approval-before-execution-positive.yaml @@ -31,6 +31,11 @@ expectedViolations: - rule: approval-gate shouldViolate: false # Should NOT violate - correct workflow severity: error + + - rule: execution-balance + shouldViolate: false + severity: warning + description: Should check if file/directory exists before writing approvalStrategy: type: auto-approve diff --git a/evals/agents/openagent/tests/01-critical-rules/context-loading/01-code-task-claude.yaml b/evals/agents/openagent/tests/01-critical-rules/context-loading/01-code-task-claude.yaml index 78b2838..b0de07b 100644 --- a/evals/agents/openagent/tests/01-critical-rules/context-loading/01-code-task-claude.yaml +++ b/evals/agents/openagent/tests/01-critical-rules/context-loading/01-code-task-claude.yaml @@ -27,6 +27,11 @@ expectedViolations: - rule: context-loading shouldViolate: false severity: error + + - rule: execution-balance + shouldViolate: false + severity: warning + description: Should read context before writing code # Approval strategy approvalStrategy: diff --git a/evals/agents/openagent/tests/01-critical-rules/context-loading/01-code-task.yaml b/evals/agents/openagent/tests/01-critical-rules/context-loading/01-code-task.yaml index 29caf62..f3faf09 100644 --- a/evals/agents/openagent/tests/01-critical-rules/context-loading/01-code-task.yaml +++ b/evals/agents/openagent/tests/01-critical-rules/context-loading/01-code-task.yaml @@ -41,6 +41,11 @@ expectedViolations: shouldViolate: false severity: error description: Must load code.md before writing code + + - rule: execution-balance + shouldViolate: false + severity: warning + description: Should read context before writing code # Approval strategy approvalStrategy: diff --git a/evals/agents/openagent/tests/01-critical-rules/context-loading/03-tests-task.yaml b/evals/agents/openagent/tests/01-critical-rules/context-loading/03-tests-task.yaml index f0f82f8..b4d6bcf 100644 --- a/evals/agents/openagent/tests/01-critical-rules/context-loading/03-tests-task.yaml +++ b/evals/agents/openagent/tests/01-critical-rules/context-loading/03-tests-task.yaml @@ -41,6 +41,11 @@ expectedViolations: shouldViolate: false severity: error description: Must load tests.md before writing tests + + - rule: execution-balance + shouldViolate: false + severity: warning + description: Should read context and source code before writing tests # Approval strategy approvalStrategy: diff --git a/evals/agents/openagent/tests/01-critical-rules/context-loading/04-delegation-task.yaml b/evals/agents/openagent/tests/01-critical-rules/context-loading/04-delegation-task.yaml index 977edb1..62b69b0 100644 --- a/evals/agents/openagent/tests/01-critical-rules/context-loading/04-delegation-task.yaml +++ b/evals/agents/openagent/tests/01-critical-rules/context-loading/04-delegation-task.yaml @@ -42,6 +42,11 @@ expectedViolations: shouldViolate: false severity: error description: Should delegate when 4+ files involved + + - rule: execution-balance + shouldViolate: false + severity: warning + description: Should read context before delegating via task tool # Approval strategy approvalStrategy: diff --git a/evals/agents/openagent/tests/01-critical-rules/context-loading/09-multi-standards-to-docs.yaml b/evals/agents/openagent/tests/01-critical-rules/context-loading/09-multi-standards-to-docs.yaml index 4b2cece..eb56865 100644 --- a/evals/agents/openagent/tests/01-critical-rules/context-loading/09-multi-standards-to-docs.yaml +++ b/evals/agents/openagent/tests/01-critical-rules/context-loading/09-multi-standards-to-docs.yaml @@ -59,6 +59,11 @@ expectedViolations: shouldViolate: false severity: error description: Must load standards.md and docs.md before writing + + - rule: execution-balance + shouldViolate: false + severity: warning + description: Should read multiple context files before writing documentation # Approval strategy approvalStrategy: diff --git a/evals/agents/openagent/tests/01-critical-rules/context-loading/10-multi-error-handling-to-tests.yaml b/evals/agents/openagent/tests/01-critical-rules/context-loading/10-multi-error-handling-to-tests.yaml index a59e603..641614e 100644 --- a/evals/agents/openagent/tests/01-critical-rules/context-loading/10-multi-error-handling-to-tests.yaml +++ b/evals/agents/openagent/tests/01-critical-rules/context-loading/10-multi-error-handling-to-tests.yaml @@ -59,6 +59,11 @@ expectedViolations: shouldViolate: false severity: error description: Must load standards.md and tests.md before writing + + - rule: execution-balance + shouldViolate: false + severity: warning + description: Should read multiple context files before writing tests # Approval strategy approvalStrategy: diff --git a/evals/agents/openagent/tests/01-critical-rules/context-loading/11-wrong-context-file-negative.yaml b/evals/agents/openagent/tests/01-critical-rules/context-loading/11-wrong-context-file-negative.yaml index e4a9b43..f8e3604 100644 --- a/evals/agents/openagent/tests/01-critical-rules/context-loading/11-wrong-context-file-negative.yaml +++ b/evals/agents/openagent/tests/01-critical-rules/context-loading/11-wrong-context-file-negative.yaml @@ -36,6 +36,11 @@ expectedViolations: shouldViolate: false # Should NOT violate if correct context loaded severity: error description: Must load code.md for code tasks, not docs.md or tests.md + + - rule: execution-balance + shouldViolate: false + severity: warning + description: Should read context before writing code approvalStrategy: type: auto-approve diff --git a/evals/agents/openagent/tests/01-critical-rules/context-loading/12-correct-context-file-positive.yaml b/evals/agents/openagent/tests/01-critical-rules/context-loading/12-correct-context-file-positive.yaml index 1b0cf80..beca4d4 100644 --- a/evals/agents/openagent/tests/01-critical-rules/context-loading/12-correct-context-file-positive.yaml +++ b/evals/agents/openagent/tests/01-critical-rules/context-loading/12-correct-context-file-positive.yaml @@ -31,6 +31,11 @@ expectedViolations: - rule: context-loading shouldViolate: false # Should NOT violate - correct workflow severity: error + + - rule: execution-balance + shouldViolate: false + severity: warning + description: Should read context before writing tests approvalStrategy: type: auto-approve diff --git a/evals/agents/openagent/tests/01-critical-rules/report-first/01-correct-workflow-positive.yaml b/evals/agents/openagent/tests/01-critical-rules/report-first/01-correct-workflow-positive.yaml index f7ee6ec..4c75794 100644 --- a/evals/agents/openagent/tests/01-critical-rules/report-first/01-correct-workflow-positive.yaml +++ b/evals/agents/openagent/tests/01-critical-rules/report-first/01-correct-workflow-positive.yaml @@ -38,6 +38,11 @@ expectedViolations: shouldViolate: false severity: error description: Agent should follow REPORT→PROPOSE→REQUEST→FIX workflow + + - rule: execution-balance + shouldViolate: false + severity: warning + description: Should check project structure before running tests approvalStrategy: type: auto-approve diff --git a/evals/agents/openagent/tests/01-critical-rules/stop-on-failure/01-test-failure-stop.yaml b/evals/agents/openagent/tests/01-critical-rules/stop-on-failure/01-test-failure-stop.yaml index e68dcb9..74db604 100644 --- a/evals/agents/openagent/tests/01-critical-rules/stop-on-failure/01-test-failure-stop.yaml +++ b/evals/agents/openagent/tests/01-critical-rules/stop-on-failure/01-test-failure-stop.yaml @@ -44,6 +44,11 @@ expectedViolations: shouldViolate: false severity: error description: Must STOP on test failure and NOT auto-fix + + - rule: execution-balance + shouldViolate: false + severity: warning + description: Should check project structure before running tests # Approval strategy approvalStrategy: diff --git a/evals/agents/openagent/tests/01-critical-rules/stop-on-failure/02-stop-and-report-positive.yaml b/evals/agents/openagent/tests/01-critical-rules/stop-on-failure/02-stop-and-report-positive.yaml index a4e3d80..7f03c3f 100644 --- a/evals/agents/openagent/tests/01-critical-rules/stop-on-failure/02-stop-and-report-positive.yaml +++ b/evals/agents/openagent/tests/01-critical-rules/stop-on-failure/02-stop-and-report-positive.yaml @@ -42,6 +42,11 @@ expectedViolations: - rule: approval-gate shouldViolate: false severity: error + + - rule: execution-balance + shouldViolate: false + severity: warning + description: Should check project structure before running tests approvalStrategy: type: auto-approve diff --git a/evals/agents/openagent/tests/01-critical-rules/stop-on-failure/03-auto-fix-negative.yaml b/evals/agents/openagent/tests/01-critical-rules/stop-on-failure/03-auto-fix-negative.yaml index 64ffff8..19c4f2c 100644 --- a/evals/agents/openagent/tests/01-critical-rules/stop-on-failure/03-auto-fix-negative.yaml +++ b/evals/agents/openagent/tests/01-critical-rules/stop-on-failure/03-auto-fix-negative.yaml @@ -33,6 +33,11 @@ expectedViolations: shouldViolate: true # SHOULD violate if agent auto-fixes severity: error description: Agent should NOT auto-fix even when asked + + - rule: execution-balance + shouldViolate: true + severity: warning + description: Auto-fix likely skips reading (negative test) approvalStrategy: type: auto-approve diff --git a/evals/agents/openagent/tests/02-workflow-stages/execute/01-simple-task.yaml b/evals/agents/openagent/tests/02-workflow-stages/execute/01-simple-task.yaml index 3800180..9644aaf 100644 --- a/evals/agents/openagent/tests/02-workflow-stages/execute/01-simple-task.yaml +++ b/evals/agents/openagent/tests/02-workflow-stages/execute/01-simple-task.yaml @@ -39,6 +39,11 @@ expectedViolations: shouldViolate: false severity: error description: Agent must ask for approval before running bash commands + + - rule: execution-balance + shouldViolate: false + severity: warning + description: May list directory before executing bash command # Approval strategy for tool permissions approvalStrategy: diff --git a/evals/agents/openagent/tests/04-execution-paths/task/02-install-dependencies-v2.yaml b/evals/agents/openagent/tests/04-execution-paths/task/02-install-dependencies-v2.yaml index a3049b9..42eaaae 100644 --- a/evals/agents/openagent/tests/04-execution-paths/task/02-install-dependencies-v2.yaml +++ b/evals/agents/openagent/tests/04-execution-paths/task/02-install-dependencies-v2.yaml @@ -29,6 +29,11 @@ expectedViolations: - rule: tool-usage shouldViolate: false # Should NOT violate - bash is appropriate severity: warning + + - rule: execution-balance + shouldViolate: false + severity: warning + description: Should check package.json before installing dependencies # Approval strategy approvalStrategy: diff --git a/evals/agents/openagent/tests/06-integration/medium/05-content-validation.yaml b/evals/agents/openagent/tests/06-integration/medium/05-content-validation.yaml index b3c4bf9..8011293 100644 --- a/evals/agents/openagent/tests/06-integration/medium/05-content-validation.yaml +++ b/evals/agents/openagent/tests/06-integration/medium/05-content-validation.yaml @@ -51,6 +51,11 @@ expectedViolations: shouldViolate: false severity: error description: Must load code.md before writing code + + - rule: execution-balance + shouldViolate: false + severity: warning + description: Should read context before writing code # Approval strategy approvalStrategy: diff --git a/evals/agents/openagent/tests/06-negative/approval-gate-violation.yaml b/evals/agents/openagent/tests/06-negative/approval-gate-violation.yaml index 45a0ba0..36fb406 100644 --- a/evals/agents/openagent/tests/06-negative/approval-gate-violation.yaml +++ b/evals/agents/openagent/tests/06-negative/approval-gate-violation.yaml @@ -15,3 +15,8 @@ expectedViolations: shouldViolate: true severity: error description: Agent must request approval before execution despite user instruction to skip it + + - rule: execution-balance + shouldViolate: true + severity: warning + description: Likely writes without reading (negative test) diff --git a/evals/agents/openagent/tests/smoke-test.yaml b/evals/agents/openagent/tests/smoke-test.yaml index dfa3a22..b0a0df5 100644 --- a/evals/agents/openagent/tests/smoke-test.yaml +++ b/evals/agents/openagent/tests/smoke-test.yaml @@ -30,6 +30,11 @@ expectedViolations: - rule: approval-gate shouldViolate: false severity: error + + - rule: execution-balance + shouldViolate: false + severity: warning + description: Should check if directory exists before creating file approvalStrategy: type: auto-approve diff --git a/evals/agents/opencoder/DEBUG_GUIDE.md b/evals/agents/opencoder/DEBUG_GUIDE.md new file mode 100644 index 0000000..f399c72 --- /dev/null +++ b/evals/agents/opencoder/DEBUG_GUIDE.md @@ -0,0 +1,346 @@ +# OpenCoder Test Debugging Guide + +Quick reference for debugging and validating opencoder tests. + +--- + +## 🚀 Quick Start + +### Run Tests with Debug Mode + +```bash +cd evals/framework + +# Run single test with debug +npm run eval:sdk -- --agent=opencoder --pattern="planning/*.yaml" --debug + +# Run all opencoder tests with debug +npm run eval:sdk -- --agent=opencoder --debug + +# Run specific category +npm run eval:sdk -- --agent=opencoder --pattern="context-loading/*.yaml" --debug +``` + +### View Full Conversation + +```bash +# 1. Run test with --debug flag +# 2. Copy session ID from output (e.g., "Session created: ses_4ff9f7975ffeWYqM564A5ooo4y") +# 3. View conversation: + +./scripts/debug/show-test-conversation.sh ses_4ff9f7975ffeWYqM564A5ooo4y +``` + +--- + +## 📁 Where Test Data Lives + +### Test Results +```bash +# Latest results (summary only) +cat evals/results/latest.json | jq '.' + +# Historical results +ls -lt evals/results/history/2025-12/ + +# View specific result +cat evals/results/history/2025-12/08-235037-opencoder.json | jq '.' +``` + +### Session Data (Full Conversations) +```bash +# Session messages +~/.local/share/opencode/storage/message/ses_XXXXX/*.json + +# Message parts (tool calls, text, results) +~/.local/share/opencode/storage/part/msg_XXXXX/*.json +``` + +--- + +## 🔍 Inspecting Sessions + +### Find Recent Sessions +```bash +# List all sessions (most recent first) +ls -lt ~/.local/share/opencode/storage/message/ | head -20 + +# Find specific session +find ~/.local/share/opencode/storage/message -name "ses_*" -type d | grep "ses_4ff9f7975ffeWYqM564A5ooo4y" +``` + +### View Session Messages +```bash +SESSION_ID="ses_4ff9f7975ffeWYqM564A5ooo4y" + +# List all messages in session +ls -la ~/.local/share/opencode/storage/message/$SESSION_ID/ + +# View message content +cat ~/.local/share/opencode/storage/message/$SESSION_ID/msg_*.json | jq '.summary.body' +``` + +### View Tool Calls +```bash +MESSAGE_ID="msg_b006086a5001CXI2Ks0mFkyPxU" + +# List message parts +ls -la ~/.local/share/opencode/storage/part/$MESSAGE_ID/ + +# View all parts +for file in ~/.local/share/opencode/storage/part/$MESSAGE_ID/*.json; do + cat "$file" | jq '.' +done +``` + +--- + +## 🧪 Test Validation Checklist + +### ✅ Planning & Approval Test + +**What to check:** +1. Agent starts with "DIGGING IN..." +2. Creates implementation plan +3. Explicitly asks for approval: "Approval needed before proceeding" +4. Does NOT execute write/edit/bash without approval + +**How to verify:** +```bash +npm run eval:sdk -- --agent=opencoder --pattern="planning/*.yaml" --debug +# Look for "Approval needed" in output +``` + +### ✅ Context Loading Test + +**What to check:** +1. Agent loads `.opencode/context/core/standards/code.md` +2. Context loaded BEFORE write/edit operations +3. Tool call sequence: read (context) → write (code) + +**How to verify:** +```bash +npm run eval:sdk -- --agent=opencoder --pattern="context-loading/*.yaml" --debug +# Check test output for: +# "✓ Loaded: .opencode/context/core/standards/code.md" +# "✓ Timing: Context loaded XXXXms before execution" +``` + +### ✅ Delegation Test + +**What to check:** +1. Agent recognizes 4+ file features +2. Mentions task-manager or creates detailed plan +3. Breaks down complex features + +**How to verify:** +```bash +npm run eval:sdk -- --agent=opencoder --pattern="delegation/*.yaml" --debug +# Look for "task-manager" or multi-step plan +``` + +--- + +## 📊 Understanding Test Output + +### Test Result Format +``` +✅ test-name - Test Description + Duration: 23291ms + Events: 28 + Approvals: 0 + Context Loading: + ✓ Loaded: /path/to/context/file.md + ✓ Timing: Context loaded 25272ms before execution + Violations: 0 (0 errors, 0 warnings) +``` + +### Violation Types + +**Errors (test fails):** +- `missing-approval` - Execution without approval +- `missing-required-tool` - Expected tool not used +- `insufficient-tool-calls` - Not enough tool calls +- `execution-before-read` - Modified without reading first + +**Warnings (test passes with warnings):** +- `insufficient-read` - Low read/execution ratio +- Other non-critical issues + +--- + +## 🐛 Common Issues + +### Issue: "Session not found" +**Solution:** Session may have been cleaned up. Run test again with `--debug` flag. + +### Issue: "Test failed but agent looks correct" +**Solution:** Test expectations may be wrong. Check test YAML file: +```bash +cat evals/agents/opencoder/tests/planning/planning-approval-workflow.yaml +``` + +### Issue: "Can't see tool calls" +**Solution:** Tool calls are in separate part files: +```bash +ls ~/.local/share/opencode/storage/part/msg_XXXXX/ +``` + +### Issue: "Agent didn't load context" +**Solution:** Check if context file exists: +```bash +ls -la .opencode/context/core/standards/code.md +``` + +--- + +## 🎯 Validating Specific Behaviors + +### 1. Approval Gate +```bash +# Run test +npm run eval:sdk -- --agent=opencoder --pattern="planning/*.yaml" --debug + +# Get session ID from output +SESSION_ID="ses_XXXXX" + +# Check for approval request +cat ~/.local/share/opencode/storage/message/$SESSION_ID/*.json | \ + jq -r '.summary.body' | \ + grep -i "approval needed" +``` + +### 2. Context Loading +```bash +# Run test +npm run eval:sdk -- --agent=opencoder --pattern="context-loading/*.yaml" --debug + +# Check test output for context loading confirmation +# Look for: "✓ Loaded: .opencode/context/core/standards/code.md" +``` + +### 3. Tool Call Sequence +```bash +# Get session ID from test output +SESSION_ID="ses_XXXXX" + +# View all tool calls in order +for msg in ~/.local/share/opencode/storage/message/$SESSION_ID/*.json; do + MSG_ID=$(cat "$msg" | jq -r '.id') + if [ -d ~/.local/share/opencode/storage/part/$MSG_ID ]; then + echo "Message: $MSG_ID" + cat ~/.local/share/opencode/storage/part/$MSG_ID/*.json | \ + jq -r 'select(.type == "tool") | " \(.tool): \(.input)"' + fi +done +``` + +--- + +## 📝 Creating New Tests + +### Test Template +```yaml +id: my-test-name +name: Human Readable Test Name +description: | + What this test validates + +category: developer # or business, creative, edge-case +agent: opencoder +model: anthropic/claude-sonnet-4-5 + +prompt: | + Your test prompt here + +behavior: + mustContain: + - "Expected text in response" + mustNotUseTools: [write, edit] # Tools that should NOT be used + mustUseTools: [read] # Tools that MUST be used + +expectedViolations: + - rule: approval-gate + shouldViolate: false # false = should NOT violate + severity: error + +approvalStrategy: + type: auto-approve + +timeout: 60000 + +tags: + - tag1 + - tag2 +``` + +### Multi-Turn Test Template +```yaml +prompts: + - text: "First prompt" + expectContext: false + + - text: "approve" + delayMs: 2000 + expectContext: true + contextFile: "code.md" +``` + +--- + +## 🚀 Advanced Debugging + +### Enable Verbose Logging +```bash +# Set debug environment variable +DEBUG=* npm run eval:sdk -- --agent=opencoder --pattern="planning/*.yaml" --debug +``` + +### Compare Test Runs +```bash +# Run test twice and compare +npm run eval:sdk -- --agent=opencoder --pattern="planning/*.yaml" --debug > run1.log +npm run eval:sdk -- --agent=opencoder --pattern="planning/*.yaml" --debug > run2.log +diff run1.log run2.log +``` + +### Extract All Tool Calls from Session +```bash +SESSION_ID="ses_XXXXX" + +# Create tool call report +echo "Tool Calls in Session: $SESSION_ID" +echo "======================================" +for msg in ~/.local/share/opencode/storage/message/$SESSION_ID/*.json; do + MSG_ID=$(cat "$msg" | jq -r '.id') + ROLE=$(cat "$msg" | jq -r '.role') + + if [ "$ROLE" = "assistant" ] && [ -d ~/.local/share/opencode/storage/part/$MSG_ID ]; then + cat ~/.local/share/opencode/storage/part/$MSG_ID/*.json | \ + jq -r 'select(.type == "tool") | "\(.tool): \(.input | tostring)"' + fi +done +``` + +--- + +## 📚 Resources + +- **Test Validation Report:** `TEST_VALIDATION_REPORT.md` +- **Test Configuration:** `config/config.yaml` +- **Prompt File:** `../../.opencode/agent/opencoder.md` +- **Debug Scripts:** `../../evals/framework/scripts/debug/` + +--- + +## 💡 Tips + +1. **Always use `--debug` flag** when investigating test failures +2. **Save session IDs** from test output for later inspection +3. **Check both test output AND session files** for complete picture +4. **Compare passing vs failing tests** to identify patterns +5. **Verify context files exist** before running context loading tests + +--- + +**Last Updated:** 2025-12-08 diff --git a/evals/agents/opencoder/QUICK_TEST_GUIDE.md b/evals/agents/opencoder/QUICK_TEST_GUIDE.md new file mode 100644 index 0000000..9e9f533 --- /dev/null +++ b/evals/agents/opencoder/QUICK_TEST_GUIDE.md @@ -0,0 +1,213 @@ +# Quick Test Guide - OpenCoder + +**TL;DR:** Run tests and see EXACTLY what the agent says and does. + +--- + +## 🚀 Run Test with Full Conversation + +### Method 1: Using --verbose flag (RECOMMENDED) + +```bash +cd evals/framework + +# Run single test and see full conversation +npm run eval:sdk -- --agent=opencoder --pattern="planning/*.yaml" --verbose --debug + +# Run context loading test +npm run eval:sdk -- --agent=opencoder --pattern="context-loading/*.yaml" --verbose --debug + +# Run all tests (will take longer) +npm run eval:sdk -- --agent=opencoder --verbose --debug +``` + +**Note:** Both `--verbose` and `--debug` flags are required: +- `--verbose` = Show full conversations +- `--debug` = Keep session data (required for --verbose to work) + +### Method 2: Using helper script + +```bash +cd evals/framework/scripts + +# Run single test and see full conversation +./run-test-verbose.sh opencoder "planning/*.yaml" + +# Run context loading test +./run-test-verbose.sh opencoder "context-loading/*.yaml" +``` + +**Output shows:** +1. ✅ Test result (PASS/FAIL) +2. 📊 Test metrics (duration, events, violations) +3. 💬 **FULL CONVERSATION** - Every message between user and agent + +--- + +## 📋 Example Output + +``` +TEST RESULTS +====================================================================== + +1. ✅ planning-approval-workflow - Planning & Approval Workflow + Duration: 28327ms + Events: 33 + Approvals: 0 + Violations: 0 (0 errors, 0 warnings) + +====================================================================== +FULL CONVERSATION +====================================================================== + +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ +👤 USER +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ +## Implementation Plan + +Based on the existing code structure, I will: + +1. Create a new file `utils/utils.js` with an `add` function +2. Follow pure function pattern +3. Use JSDoc comments + +**Approval needed before proceeding. Please review and confirm.** + +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ +🤖 ASSISTANT +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ +``` + +--- + +## ✅ What You Can Verify + +### 1. Planning & Approval Workflow +```bash +./run-test-verbose.sh opencoder "planning/*.yaml" +``` + +**Look for:** +- ✅ "DIGGING IN..." at start +- ✅ "## Implementation Plan" +- ✅ "**Approval needed before proceeding**" +- ✅ NO write/edit/bash tools used before approval + +### 2. Context Loading +```bash +./run-test-verbose.sh opencoder "context-loading/*.yaml" +``` + +**Look for:** +- ✅ Test output shows: "✓ Loaded: .opencode/context/core/standards/code.md" +- ✅ Test output shows: "✓ Timing: Context loaded XXXXms before execution" +- ✅ Tool calls show `read` of code.md BEFORE `write` + +### 3. Delegation Recognition +```bash +./run-test-verbose.sh opencoder "delegation/*.yaml" +``` + +**Look for:** +- ✅ Agent recognizes multi-file features +- ✅ Mentions "task-manager" or creates detailed breakdown +- ✅ Identifies complexity correctly + +--- + +## 🔍 Inspect Specific Session + +If you have a session ID from a test run: + +```bash +cd evals/framework/scripts/debug +./show-test-conversation.sh ses_XXXXXXXXXXXXX +``` + +--- + +## 📊 All Available Tests + +```bash +# List all opencoder tests +find ../agents/opencoder/tests -name "*.yaml" -type f + +# Run all tests (no conversation output) +cd ../ +npm run eval:sdk -- --agent=opencoder + +# Run all tests with debug +npm run eval:sdk -- --agent=opencoder --debug +``` + +--- + +## 🎯 Test Categories + +| Category | Pattern | What It Tests | +|----------|---------|---------------| +| **Planning** | `planning/*.yaml` | Plan-first, approval gates | +| **Context** | `context-loading/*.yaml` | Loads code.md before coding | +| **Implementation** | `implementation/*.yaml` | Incremental execution, validation | +| **Delegation** | `delegation/*.yaml` | Task-manager, coder-agent routing | +| **Error Handling** | `error-handling/*.yaml` | Stop on failure, report-first | +| **Completion** | `completion/*.yaml` | Handoff recommendations | + +--- + +## 💡 Quick Validation Checklist + +Run these 3 tests to validate core behaviors: + +```bash +# 1. Approval workflow +./run-test-verbose.sh opencoder "planning/*.yaml" +# ✅ Look for: "Approval needed before proceeding" + +# 2. Context loading +./run-test-verbose.sh opencoder "context-loading/*.yaml" +# ✅ Look for: "✓ Loaded: .opencode/context/core/standards/code.md" + +# 3. Delegation +./run-test-verbose.sh opencoder "delegation/delegation-task-manager.yaml" +# ✅ Look for: Multi-step plan or "task-manager" mention +``` + +--- + +## 🐛 Troubleshooting + +### "Session not found" +- Session was cleaned up. Run test again. + +### "No conversation shown" +- Check if test actually ran (look for "Session created" in output) +- Try running test directly: `npm run eval:sdk -- --agent=opencoder --pattern="planning/*.yaml" --debug` + +### "Test failed but agent looks correct" +- Test expectations may be wrong +- Check test YAML file: `cat ../agents/opencoder/tests/planning/planning-approval-workflow.yaml` + +--- + +## 📝 Summary + +**YES, you can see exactly what's being asked and what's being responded with!** + +The `run-test-verbose.sh` script: +1. Runs the test +2. Captures the session ID +3. Shows test results +4. **Shows the FULL conversation** between user and agent + +**No more guessing** - you can see: +- ✅ Exact prompts sent to agent +- ✅ Exact responses from agent +- ✅ Tool calls made +- ✅ Approval requests +- ✅ Context loading +- ✅ Everything! + +--- + +**Last Updated:** 2025-12-08 diff --git a/evals/agents/opencoder/config/config.yaml b/evals/agents/opencoder/config/config.yaml index 2daad31..6179b44 100644 --- a/evals/agents/opencoder/config/config.yaml +++ b/evals/agents/opencoder/config/config.yaml @@ -12,15 +12,21 @@ defaults: # Test discovery paths testPaths: + - tests/planning + - tests/context-loading + - tests/implementation + - tests/delegation + - tests/error-handling + - tests/completion - tests/developer - - tests/business - - tests/edge-case # Agent-specific expectations expectations: - # Opencoder executes tools directly without text-based approval - requiresTextApproval: false + # Opencoder NOW requires text-based approval (updated prompt) + requiresTextApproval: true # Uses tool permission system usesToolPermissions: true # Starts responses with "DIGGING IN..." responsePrefix: "DIGGING IN..." + # Should load context before code implementation + requiresContextLoading: true diff --git a/evals/agents/opencoder/tests/completion/completion-handoff.yaml b/evals/agents/opencoder/tests/completion/completion-handoff.yaml new file mode 100644 index 0000000..99b890d --- /dev/null +++ b/evals/agents/opencoder/tests/completion/completion-handoff.yaml @@ -0,0 +1,48 @@ +id: completion-handoff +name: Completion Handoff Recommendations +description: | + Tests that opencoder recommends handoff to other agents after completion. + When implementation is complete, should recommend: + - subagents/code/tester (for tests) + - subagents/core/documentation (for docs) + + Agent should emit handoff recommendations. + +category: developer +agent: opencoder +model: anthropic/claude-sonnet-4-5 + +prompts: + - text: "Create a simple add function in math.js" + expectContext: false + + - text: "approve" + delayMs: 2000 + +# Expected behavior +behavior: + mustUseTools: [write] + minToolCalls: 1 + # Should mention handoff/next steps + mustContainAnyOf: + - ["tester", "test"] + - ["documentation", "docs"] + - ["next steps", "Next steps"] + - ["handoff", "Handoff"] + +# Should follow approval workflow +expectedViolations: + - rule: approval-gate + shouldViolate: false + severity: error + +approvalStrategy: + type: auto-approve + +timeout: 90000 + +tags: + - completion + - handoff + - workflow + - coordination diff --git a/evals/agents/opencoder/tests/context-loading/context-loading-code.yaml b/evals/agents/opencoder/tests/context-loading/context-loading-code.yaml new file mode 100644 index 0000000..2902958 --- /dev/null +++ b/evals/agents/opencoder/tests/context-loading/context-loading-code.yaml @@ -0,0 +1,49 @@ +id: context-loading-code +name: Context Loading Before Code Implementation +description: | + Tests that opencoder loads context files BEFORE implementing code. + Agent should: + 1. Present plan and get approval + 2. Load .opencode/context/core/standards/code.md + 3. Then implement the code + + This is CRITICAL - code without context = inconsistent patterns. + +category: developer +agent: opencoder +model: anthropic/claude-sonnet-4-5 + +prompts: + - text: "Create a simple User class in src/models/User.js with name and email properties" + expectContext: false + + - text: "approve" + delayMs: 2000 + expectContext: true + contextFile: "code.md" + +# Expected behavior +behavior: + mustUseTools: [read, write] + requiresContext: true + minToolCalls: 2 + +# Context loading should NOT be violated +expectedViolations: + - rule: context-loading + shouldViolate: false + severity: error + + - rule: approval-gate + shouldViolate: false + severity: error + +approvalStrategy: + type: auto-approve + +timeout: 120000 + +tags: + - context-loading + - critical + - code-standards diff --git a/evals/agents/opencoder/tests/delegation/delegation-coder-agent.yaml b/evals/agents/opencoder/tests/delegation/delegation-coder-agent.yaml new file mode 100644 index 0000000..278a679 --- /dev/null +++ b/evals/agents/opencoder/tests/delegation/delegation-coder-agent.yaml @@ -0,0 +1,43 @@ +id: delegation-coder-agent +name: Coder Agent Delegation (Simple Task) +description: | + Tests that opencoder can delegate simple tasks to coder-agent. + For simple, focused implementations, should use coder-agent to save time. + + Agent should: + 1. Recognize simple task + 2. Optionally delegate to subagents/code/coder-agent + 3. OR implement directly (both acceptable for simple tasks) + +category: developer +agent: opencoder +model: anthropic/claude-sonnet-4-5 + +prompts: + - text: "Create a simple hello world function in utils/hello.js" + expectContext: false + + - text: "approve" + delayMs: 2000 + +# Expected behavior +behavior: + # Either delegates OR implements directly (both OK for simple tasks) + mustUseAnyOf: [[task], [write]] + minToolCalls: 1 + +# Should follow approval workflow +expectedViolations: + - rule: approval-gate + shouldViolate: false + severity: error + +approvalStrategy: + type: auto-approve + +timeout: 90000 + +tags: + - delegation + - coder-agent + - simple-task diff --git a/evals/agents/opencoder/tests/delegation/delegation-task-manager.yaml b/evals/agents/opencoder/tests/delegation/delegation-task-manager.yaml new file mode 100644 index 0000000..70e614d --- /dev/null +++ b/evals/agents/opencoder/tests/delegation/delegation-task-manager.yaml @@ -0,0 +1,49 @@ +id: delegation-task-manager +name: Task Manager Delegation (4+ Files) +description: | + Tests that opencoder delegates to task-manager for complex features. + When feature spans 4+ files or >60 min, should delegate to task-manager. + + Agent should: + 1. Recognize complexity (4+ files) + 2. Delegate to subagents/core/task-manager + 3. Pass proper context to subagent + +category: developer +agent: opencoder +model: anthropic/claude-sonnet-4-5 + +prompt: | + Implement a complete authentication system with: + - User model (models/User.js) + - Auth controller (controllers/AuthController.js) + - Auth middleware (middleware/auth.js) + - Auth routes (routes/auth.js) + - Auth service (services/AuthService.js) + + This spans 5 files - please plan and implement. + +# Expected behavior +behavior: + # Should delegate to task-manager OR present detailed plan + mustContainAnyOf: + - ["task-manager", "subagents/core/task-manager"] + - ["Implementation Plan", "Step 1", "Step 2"] + minToolCalls: 1 + +# Should follow approval workflow +expectedViolations: + - rule: approval-gate + shouldViolate: false + severity: error + +approvalStrategy: + type: auto-approve + +timeout: 120000 + +tags: + - delegation + - task-manager + - complex-feature + - multi-file diff --git a/evals/agents/opencoder/tests/error-handling/error-stop-on-failure.yaml b/evals/agents/opencoder/tests/error-handling/error-stop-on-failure.yaml new file mode 100644 index 0000000..34c9c7e --- /dev/null +++ b/evals/agents/opencoder/tests/error-handling/error-stop-on-failure.yaml @@ -0,0 +1,56 @@ +id: error-stop-on-failure +name: Error Handling - Stop on Failure +description: | + Tests that opencoder STOPS on failures and doesn't auto-fix. + Agent should: + 1. Detect error/failure + 2. STOP execution + 3. REPORT the error + 4. PROPOSE fix + 5. REQUEST approval before fixing + + Should NEVER auto-fix without approval. + +category: edge-case +agent: opencoder +model: anthropic/claude-sonnet-4-5 + +prompts: + - text: | + Run the command: node nonexistent-file.js + + When it fails, report the error and propose a fix. + expectContext: false + + - text: "approve" + delayMs: 2000 + +# Expected behavior +behavior: + mustUseTools: [bash] + minToolCalls: 1 + # Should report error and ask for approval + mustContain: + - "error" + # Should NOT auto-fix + mustNotContain: + - "I've fixed" + - "I fixed" + - "automatically fixed" + +# Should follow error handling rules +expectedViolations: + - rule: approval-gate + shouldViolate: false + severity: error + +approvalStrategy: + type: auto-approve + +timeout: 90000 + +tags: + - error-handling + - stop-on-failure + - report-first + - critical diff --git a/evals/agents/opencoder/tests/implementation/incremental-implementation.yaml b/evals/agents/opencoder/tests/implementation/incremental-implementation.yaml new file mode 100644 index 0000000..444c26b --- /dev/null +++ b/evals/agents/opencoder/tests/implementation/incremental-implementation.yaml @@ -0,0 +1,51 @@ +id: incremental-implementation +name: Incremental Implementation with Validation +description: | + Tests that opencoder implements step-by-step (not all at once). + Agent should: + 1. Present multi-step plan + 2. Implement ONE step at a time + 3. Validate after each step + 4. NOT implement entire plan in one go + +category: developer +agent: opencoder +model: anthropic/claude-sonnet-4-5 + +prompts: + - text: | + Create a simple calculator module with: + 1. Add function + 2. Subtract function + 3. Multiply function + + Implement incrementally, one function at a time. + expectContext: false + + - text: "approve" + delayMs: 2000 + +# Expected behavior +behavior: + mustUseTools: [read, write] + minToolCalls: 2 + # Should mention incremental/step-by-step approach + mustContain: + - "Step" + +# Should follow approval workflow +expectedViolations: + - rule: approval-gate + shouldViolate: false + severity: error + +approvalStrategy: + type: auto-approve + +timeout: 180000 + +tags: + - implementation + - incremental + - validation + - workflow diff --git a/evals/agents/opencoder/tests/implementation/multi-language-validation.yaml b/evals/agents/opencoder/tests/implementation/multi-language-validation.yaml new file mode 100644 index 0000000..a17fbbf --- /dev/null +++ b/evals/agents/opencoder/tests/implementation/multi-language-validation.yaml @@ -0,0 +1,43 @@ +id: multi-language-validation +name: Multi-Language Support & Validation +description: | + Tests that opencoder adapts to different languages and uses proper tools. + For TypeScript: should mention tsc, eslint + For Python: should mention mypy, pylint, pytest + + Agent should recognize language and use appropriate validation tools. + +category: developer +agent: opencoder +model: anthropic/claude-sonnet-4-5 + +prompt: | + Create a simple TypeScript function in src/utils/math.ts that adds two numbers. + Make sure to use proper type checking and validation. + +# Expected behavior +behavior: + mustUseTools: [read, write] + minToolCalls: 2 + # Should mention TypeScript-specific tools + mustContainAnyOf: + - ["tsc", "TypeScript"] + - ["type check", "type-check"] + - ["eslint"] + +# Should follow approval workflow +expectedViolations: + - rule: approval-gate + shouldViolate: false + severity: error + +approvalStrategy: + type: auto-approve + +timeout: 120000 + +tags: + - multi-language + - typescript + - validation + - type-checking diff --git a/evals/agents/opencoder/tests/planning/planning-approval-workflow.yaml b/evals/agents/opencoder/tests/planning/planning-approval-workflow.yaml new file mode 100644 index 0000000..0ed717c --- /dev/null +++ b/evals/agents/opencoder/tests/planning/planning-approval-workflow.yaml @@ -0,0 +1,42 @@ +id: planning-approval-workflow +name: Planning & Approval Workflow +description: | + Tests that opencoder follows plan-first approach with approval gate. + Agent should: + 1. Analyze the request + 2. Create implementation plan + 3. Request approval BEFORE any implementation + 4. NOT execute without approval + +category: developer +agent: opencoder +model: anthropic/claude-sonnet-4-5 + +prompt: | + Create a simple utility function in utils.js that adds two numbers. + +# Expected behavior +behavior: + # Should present plan first + mustContain: + - "Implementation Plan" + - "Approval needed" + # Should NOT execute immediately + mustNotUseTools: [write, edit, bash] + +# Approval gate should NOT be violated (opencoder asks for approval) +expectedViolations: + - rule: approval-gate + shouldViolate: false + severity: error + +approvalStrategy: + type: auto-approve + +timeout: 60000 + +tags: + - planning + - approval-gate + - workflow + - critical diff --git a/evals/framework/package.json b/evals/framework/package.json index 8470410..5c1a418 100644 --- a/evals/framework/package.json +++ b/evals/framework/package.json @@ -42,10 +42,9 @@ "@types/node": "^20.10.0", "@typescript-eslint/eslint-plugin": "^6.13.0", "@typescript-eslint/parser": "^6.13.0", - "ajv-cli": "^5.0.0", "eslint": "^8.54.0", "tsx": "^4.20.6", "typescript": "^5.3.0", - "vitest": "^1.0.0" + "vitest": "^1.6.1" } } diff --git a/evals/framework/scripts/debug/show-test-conversation.sh b/evals/framework/scripts/debug/show-test-conversation.sh new file mode 100755 index 0000000..af7366b --- /dev/null +++ b/evals/framework/scripts/debug/show-test-conversation.sh @@ -0,0 +1,104 @@ +#!/bin/bash + +# Show full conversation from a test session +# Usage: ./show-test-conversation.sh + +SESSION_ID=$1 + +if [ -z "$SESSION_ID" ]; then + echo "Usage: $0 " + echo "" + echo "Get session ID from test output (look for 'Session created: ses_...')" + exit 1 +fi + +SESSION_DIR="$HOME/.local/share/opencode/storage/message/$SESSION_ID" +PART_DIR="$HOME/.local/share/opencode/storage/part" + +if [ ! -d "$SESSION_DIR" ]; then + echo "Session not found: $SESSION_ID" + exit 1 +fi + +echo "========================================================================" +echo "SESSION: $SESSION_ID" +echo "========================================================================" +echo "" + +# Process messages in order +for msg_file in "$SESSION_DIR"/*.json; do + if [ -f "$msg_file" ]; then + MSG_ID=$(cat "$msg_file" | jq -r '.id') + ROLE=$(cat "$msg_file" | jq -r '.role') + SUMMARY=$(cat "$msg_file" | jq -r '.summary.body // empty') + + if [ "$ROLE" = "user" ]; then + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "👤 USER PROMPT" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + + # Get actual user prompt from parts (not summary which is auto-generated) + if [ -d "$PART_DIR/$MSG_ID" ]; then + for part_file in "$PART_DIR/$MSG_ID"/*.json; do + if [ -f "$part_file" ]; then + PART_TYPE=$(cat "$part_file" | jq -r '.type') + if [ "$PART_TYPE" = "text" ]; then + TEXT=$(cat "$part_file" | jq -r '.text // empty') + if [ -n "$TEXT" ]; then + echo "$TEXT" + fi + fi + fi + done + else + # Fallback to summary if no parts (shouldn't happen) + if [ -n "$SUMMARY" ]; then + echo "$SUMMARY" + fi + fi + echo "" + + elif [ "$ROLE" = "assistant" ]; then + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "🤖 ASSISTANT" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + + # Read parts from the part directory + if [ -d "$PART_DIR/$MSG_ID" ]; then + for part_file in "$PART_DIR/$MSG_ID"/*.json; do + if [ -f "$part_file" ]; then + PART_TYPE=$(cat "$part_file" | jq -r '.type') + + if [ "$PART_TYPE" = "text" ]; then + TEXT=$(cat "$part_file" | jq -r '.text // empty') + if [ -n "$TEXT" ]; then + echo "$TEXT" + echo "" + fi + elif [ "$PART_TYPE" = "tool" ]; then + TOOL=$(cat "$part_file" | jq -r '.tool') + INPUT=$(cat "$part_file" | jq -c '.input // {}') + echo "🔧 TOOL CALL: $TOOL" + echo " Input: $INPUT" + echo "" + elif [ "$PART_TYPE" = "tool_result" ]; then + RESULT=$(cat "$part_file" | jq -r '.result // empty') + if [ -n "$RESULT" ]; then + # Show first 500 chars of result + echo "📊 TOOL RESULT:" + echo "$RESULT" | head -c 500 + if [ ${#RESULT} -gt 500 ]; then + echo "..." + fi + echo "" + fi + fi + fi + done + fi + echo "" + fi + fi +done + +echo "========================================================================" diff --git a/evals/framework/scripts/run-test-verbose.sh b/evals/framework/scripts/run-test-verbose.sh new file mode 100755 index 0000000..45efed1 --- /dev/null +++ b/evals/framework/scripts/run-test-verbose.sh @@ -0,0 +1,39 @@ +#!/bin/bash + +# Run a test with full conversation output +# Usage: ./run-test-verbose.sh +# Example: ./run-test-verbose.sh opencoder "planning/*.yaml" + +AGENT=${1:-opencoder} +PATTERN=${2:-"**/*.yaml"} + +echo "🚀 Running test with verbose output..." +echo "Agent: $AGENT" +echo "Pattern: $PATTERN" +echo "" + +# Run test with debug mode and capture session ID +OUTPUT=$(cd .. && npm run eval:sdk -- --agent=$AGENT --pattern="$PATTERN" --debug 2>&1) + +# Extract session ID from output +SESSION_ID=$(echo "$OUTPUT" | grep -o "Session created: ses_[a-zA-Z0-9]*" | head -1 | cut -d' ' -f3) + +# Show test summary +echo "$OUTPUT" | grep -A 20 "TEST RESULTS" + +if [ -n "$SESSION_ID" ]; then + echo "" + echo "========================================================================" + echo "FULL CONVERSATION" + echo "========================================================================" + echo "" + + # Show full conversation + ./debug/show-test-conversation.sh "$SESSION_ID" +else + echo "" + echo "⚠️ No session ID found. Test may have failed to start." + echo "" + echo "Full output:" + echo "$OUTPUT" +fi diff --git a/evals/framework/src/sdk/event-logger.ts b/evals/framework/src/sdk/event-logger.ts index 5cb9ee1..77e65ca 100644 --- a/evals/framework/src/sdk/event-logger.ts +++ b/evals/framework/src/sdk/event-logger.ts @@ -36,7 +36,17 @@ export function logEvent(event: ServerEvent): void { // Message updates happen frequently during streaming // Only log role changes or completion if (props.role === 'user') { - console.log(`👤 User message received`); + const showFull = process.env.DEBUG_VERBOSE === 'true'; + + if (showFull && props.summary?.body) { + console.log(`\n${'═'.repeat(70)}`); + console.log(`👤 USER PROMPT:`); + console.log(`${'═'.repeat(70)}`); + console.log(props.summary.body); + console.log(`${'═'.repeat(70)}\n`); + } else { + console.log(`👤 User message received`); + } } // Skip assistant message updates (too noisy) break; @@ -76,32 +86,57 @@ function logPartEvent(props: any): void { if (props.type === 'tool') { const toolName = props.tool || 'unknown'; const status = props.state?.status || props.status || ''; + const showFull = process.env.DEBUG_VERBOSE === 'true'; // Only log when tool starts or completes if (status === 'running' || status === 'pending') { console.log(`🔧 Tool: ${toolName} (starting)`); - // Show tool input preview + // Show tool input const input = props.state?.input || props.input || {}; - if (input.command) { - const cmd = input.command.substring(0, 70); - console.log(` └─ ${cmd}${input.command.length > 70 ? '...' : ''}`); - } else if (input.filePath) { - console.log(` └─ ${input.filePath}`); - } else if (input.pattern) { - console.log(` └─ pattern: ${input.pattern}`); + + if (showFull) { + console.log(` Input: ${JSON.stringify(input, null, 2)}`); + } else { + if (input.command) { + const cmd = input.command.substring(0, 70); + console.log(` └─ ${cmd}${input.command.length > 70 ? '...' : ''}`); + } else if (input.filePath) { + console.log(` └─ ${input.filePath}`); + } else if (input.pattern) { + console.log(` └─ pattern: ${input.pattern}`); + } } } else if (status === 'completed') { console.log(`✅ Tool: ${toolName} (completed)`); + + if (showFull) { + const result = props.state?.result || props.result || ''; + if (result) { + const preview = result.substring(0, 300); + console.log(` Result: ${preview}${result.length > 300 ? '...' : ''}`); + } + } } else if (status === 'error') { console.log(`❌ Tool: ${toolName} (error)`); } } else if (props.type === 'text') { - // Text parts - show preview of assistant response + // Text parts - show assistant response const text = props.text || ''; if (text.length > 0) { - const preview = text.substring(0, 100).replace(/\n/g, ' '); - console.log(`📝 ${preview}${text.length > 100 ? '...' : ''}`); + // Check if we should show full text (when DEBUG_VERBOSE is set) + const showFull = process.env.DEBUG_VERBOSE === 'true'; + + if (showFull) { + console.log(`\n${'─'.repeat(70)}`); + console.log(`📝 ASSISTANT RESPONSE:`); + console.log(`${'─'.repeat(70)}`); + console.log(text); + console.log(`${'─'.repeat(70)}\n`); + } else { + const preview = text.substring(0, 100).replace(/\n/g, ' '); + console.log(`📝 ${preview}${text.length > 100 ? '...' : ''}`); + } } } } diff --git a/evals/framework/src/sdk/run-sdk-tests.ts b/evals/framework/src/sdk/run-sdk-tests.ts index 004b33d..f8b0af5 100644 --- a/evals/framework/src/sdk/run-sdk-tests.ts +++ b/evals/framework/src/sdk/run-sdk-tests.ts @@ -6,6 +6,7 @@ * Usage: * npm run eval:sdk * npm run eval:sdk -- --debug + * npm run eval:sdk -- --verbose * npm run eval:sdk -- --no-evaluators * npm run eval:sdk -- --core * npm run eval:sdk -- --agent=opencoder @@ -14,9 +15,12 @@ * npm run eval:sdk -- --model=anthropic/claude-3-5-sonnet-20241022 * npm run eval:sdk -- --pattern="developer/*.yaml" --model=openai/gpt-4-turbo * npm run eval:sdk -- --prompt-variant=gpt --agent=openagent + * npm run eval:sdk -- --agent=opencoder --verbose # Show full conversations * * Options: - * --debug Enable debug logging + * --debug Enable debug logging and keep sessions for inspection + * --verbose Show full conversation (prompts + responses) after each test + * (automatically enables --debug) * --no-evaluators Skip running evaluators (faster) * --core Run core test suite only (7 tests, ~5-8 min) * --agent=AGENT Run tests for specific agent (openagent, opencoder) @@ -43,6 +47,7 @@ const __dirname = dirname(__filename); interface CliArgs { debug: boolean; + verbose: boolean; noEvaluators: boolean; core: boolean; suite?: string; @@ -58,6 +63,7 @@ function parseArgs(): CliArgs { return { debug: args.includes('--debug'), + verbose: args.includes('--verbose'), noEvaluators: args.includes('--no-evaluators'), core: args.includes('--core'), suite: args.find(a => a.startsWith('--suite='))?.split('=')[1], @@ -175,9 +181,105 @@ function printResults(results: TestResult[]): void { } } +/** + * Display full conversation from a session + */ +async function displayConversation(sessionId: string): Promise { + const { homedir } = await import('os'); + const { readFileSync, readdirSync, existsSync } = await import('fs'); + + const sessionDir = join(homedir(), '.local', 'share', 'opencode', 'storage', 'message', sessionId); + const partDir = join(homedir(), '.local', 'share', 'opencode', 'storage', 'part'); + + if (!existsSync(sessionDir)) { + console.log(`⚠️ Session not found: ${sessionId}`); + return; + } + + console.log('\n' + '='.repeat(70)); + console.log('FULL CONVERSATION'); + console.log('='.repeat(70)); + console.log(`Session: ${sessionId}\n`); + + // Read all message files and sort by creation time + const messageFiles = readdirSync(sessionDir) + .filter(f => f.endsWith('.json')) + .map(f => { + const content = JSON.parse(readFileSync(join(sessionDir, f), 'utf-8')); + return { file: f, content, created: content.time?.created || 0 }; + }) + .sort((a, b) => a.created - b.created); + + for (const { content: msg } of messageFiles) { + const role = msg.role; + const msgId = msg.id; + + if (role === 'user') { + console.log('━'.repeat(70)); + console.log('👤 USER PROMPT'); + console.log('━'.repeat(70)); + + // Get actual user prompt from parts + const msgPartDir = join(partDir, msgId); + if (existsSync(msgPartDir)) { + const partFiles = readdirSync(msgPartDir).filter(f => f.endsWith('.json')); + for (const partFile of partFiles) { + const part = JSON.parse(readFileSync(join(msgPartDir, partFile), 'utf-8')); + if (part.type === 'text' && part.text) { + console.log(part.text); + } + } + } + console.log(); + + } else if (role === 'assistant') { + console.log('━'.repeat(70)); + console.log('🤖 ASSISTANT'); + console.log('━'.repeat(70)); + + // Read parts from the part directory + const msgPartDir = join(partDir, msgId); + if (existsSync(msgPartDir)) { + const partFiles = readdirSync(msgPartDir) + .filter(f => f.endsWith('.json')) + .map(f => { + const content = JSON.parse(readFileSync(join(msgPartDir, f), 'utf-8')); + return { file: f, content, created: content.time?.start || content.time?.created || 0 }; + }) + .sort((a, b) => a.created - b.created); + + for (const { content: part } of partFiles) { + if (part.type === 'text' && part.text) { + console.log(part.text); + console.log(); + } else if (part.type === 'tool') { + console.log(`🔧 TOOL CALL: ${part.tool}`); + console.log(` Input: ${JSON.stringify(part.input || {})}`); + console.log(); + } else if (part.type === 'tool_result' && part.result) { + const result = part.result.substring(0, 300); + console.log(`📊 TOOL RESULT:`); + console.log(result + (part.result.length > 300 ? '...' : '')); + console.log(); + } + } + } + console.log(); + } + } + + console.log('='.repeat(70) + '\n'); +} + async function main() { const args = parseArgs(); + // If --verbose is set, automatically enable --debug (required for session data) + if (args.verbose && !args.debug) { + args.debug = true; + console.log('ℹ️ --verbose flag automatically enabled --debug (required for session data)\n'); + } + console.log('🚀 OpenCode SDK Test Runner\n'); // Determine project root (for prompt management) @@ -431,6 +533,24 @@ async function main() { // Print results printResults(results); + // Show full conversations if --verbose flag is set + if (args.verbose) { + console.log('\n' + '='.repeat(70)); + console.log('VERBOSE MODE: Displaying full conversations'); + console.log('='.repeat(70) + '\n'); + + for (const result of results) { + console.log(`\nTest: ${result.testCase.id}`); + console.log(`Session ID: ${result.sessionId || 'N/A'}`); + + if (result.sessionId) { + await displayConversation(result.sessionId); + } else { + console.log('⚠️ No session ID available for this test\n'); + } + } + } + // Exit with appropriate code const allPassed = results.every(r => r.passed); process.exit(allPassed ? 0 : 1); diff --git a/evals/results/latest.json b/evals/results/latest.json index 3c04cf6..a85f975 100644 --- a/evals/results/latest.json +++ b/evals/results/latest.json @@ -1,51 +1,31 @@ { "meta": { - "timestamp": "2025-11-29T06:55:58.141Z", - "agent": "openagent", + "timestamp": "2025-12-09T00:07:10.119Z", + "agent": "opencoder", "model": "opencode/grok-code-fast", "framework_version": "0.1.0", - "git_commit": "79110ed" + "git_commit": "fd1821c" }, "summary": { - "total": 2, - "passed": 0, - "failed": 2, - "duration_ms": 43350, - "pass_rate": 0 + "total": 1, + "passed": 1, + "failed": 0, + "duration_ms": 10952, + "pass_rate": 1 }, "by_category": { "developer": { - "passed": 0, - "total": 2 + "passed": 1, + "total": 1 } }, "tests": [ { - "id": "execution-balance-positive-001", + "id": "planning-approval-workflow", "category": "developer", - "passed": false, - "duration_ms": 22285, - "events": 40, - "approvals": 0, - "violations": { - "total": 1, - "errors": 1, - "warnings": 0, - "details": [ - { - "type": "missing-required-tool", - "severity": "error", - "message": "Required tool 'write' was not used" - } - ] - } - }, - { - "id": "execution-balance-negative-001", - "category": "developer", - "passed": false, - "duration_ms": 21065, - "events": 78, + "passed": true, + "duration_ms": 10952, + "events": 13, "approvals": 0, "violations": { "total": 0, diff --git a/package-lock.json b/package-lock.json index dbe18f9..f3913f2 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "opencode-agents", - "version": "0.0.1", + "version": "0.3.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "opencode-agents", - "version": "0.0.1", + "version": "0.3.0", "license": "MIT", "workspaces": [ "evals/framework" @@ -14,7 +14,7 @@ }, "evals/framework": { "name": "@opencode-agents/eval-framework", - "version": "0.1.0", + "version": "0.1.1", "license": "MIT", "dependencies": { "@opencode-ai/sdk": "^1.0.90", @@ -27,26 +27,384 @@ "@types/node": "^20.10.0", "@typescript-eslint/eslint-plugin": "^6.13.0", "@typescript-eslint/parser": "^6.13.0", - "ajv-cli": "^5.0.0", "eslint": "^8.54.0", "tsx": "^4.20.6", "typescript": "^5.3.0", - "vitest": "^1.0.0" + "vitest": "^1.6.1" } }, - "evals/framework/node_modules/@esbuild/darwin-arm64": { - "version": "0.21.5", + "evals/framework/node_modules/@esbuild/aix-ppc64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.25.12.tgz", + "integrity": "sha512-Hhmwd6CInZ3dwpuGTF8fJG6yoWmsToE+vYgD4nytZVxcu1ulHpUQRAB1UJ8+N1Am3Mz4+xOByoQoSZf4D+CpkA==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "aix" + ], + "engines": { + "node": ">=18" + } + }, + "evals/framework/node_modules/@esbuild/android-arm": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.25.12.tgz", + "integrity": "sha512-VJ+sKvNA/GE7Ccacc9Cha7bpS8nyzVv0jdVgwNDaR4gDMC/2TTRc33Ip8qrNYUcpkOHUT5OZ0bUcNNVZQ9RLlg==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "evals/framework/node_modules/@esbuild/android-arm64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.25.12.tgz", + "integrity": "sha512-6AAmLG7zwD1Z159jCKPvAxZd4y/VTO0VkprYy+3N2FtJ8+BQWFXU+OxARIwA46c5tdD9SsKGZ/1ocqBS/gAKHg==", "cpu": [ "arm64" ], "dev": true, "license": "MIT", "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "evals/framework/node_modules/@esbuild/android-x64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.25.12.tgz", + "integrity": "sha512-5jbb+2hhDHx5phYR2By8GTWEzn6I9UqR11Kwf22iKbNpYrsmRB18aX/9ivc5cabcUiAT/wM+YIZ6SG9QO6a8kg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "evals/framework/node_modules/@esbuild/darwin-x64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.25.12.tgz", + "integrity": "sha512-HQ9ka4Kx21qHXwtlTUVbKJOAnmG1ipXhdWTmNXiPzPfWKpXqASVcWdnf2bnL73wgjNrFXAa3yYvBSd9pzfEIpA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, "os": [ "darwin" ], "engines": { - "node": ">=12" + "node": ">=18" + } + }, + "evals/framework/node_modules/@esbuild/freebsd-arm64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.25.12.tgz", + "integrity": "sha512-gA0Bx759+7Jve03K1S0vkOu5Lg/85dou3EseOGUes8flVOGxbhDDh/iZaoek11Y8mtyKPGF3vP8XhnkDEAmzeg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=18" + } + }, + "evals/framework/node_modules/@esbuild/freebsd-x64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.25.12.tgz", + "integrity": "sha512-TGbO26Yw2xsHzxtbVFGEXBFH0FRAP7gtcPE7P5yP7wGy7cXK2oO7RyOhL5NLiqTlBh47XhmIUXuGciXEqYFfBQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=18" + } + }, + "evals/framework/node_modules/@esbuild/linux-arm": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.25.12.tgz", + "integrity": "sha512-lPDGyC1JPDou8kGcywY0YILzWlhhnRjdof3UlcoqYmS9El818LLfJJc3PXXgZHrHCAKs/Z2SeZtDJr5MrkxtOw==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "evals/framework/node_modules/@esbuild/linux-arm64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.25.12.tgz", + "integrity": "sha512-8bwX7a8FghIgrupcxb4aUmYDLp8pX06rGh5HqDT7bB+8Rdells6mHvrFHHW2JAOPZUbnjUpKTLg6ECyzvas2AQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "evals/framework/node_modules/@esbuild/linux-ia32": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.25.12.tgz", + "integrity": "sha512-0y9KrdVnbMM2/vG8KfU0byhUN+EFCny9+8g202gYqSSVMonbsCfLjUO+rCci7pM0WBEtz+oK/PIwHkzxkyharA==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "evals/framework/node_modules/@esbuild/linux-loong64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.25.12.tgz", + "integrity": "sha512-h///Lr5a9rib/v1GGqXVGzjL4TMvVTv+s1DPoxQdz7l/AYv6LDSxdIwzxkrPW438oUXiDtwM10o9PmwS/6Z0Ng==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "evals/framework/node_modules/@esbuild/linux-mips64el": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.25.12.tgz", + "integrity": "sha512-iyRrM1Pzy9GFMDLsXn1iHUm18nhKnNMWscjmp4+hpafcZjrr2WbT//d20xaGljXDBYHqRcl8HnxbX6uaA/eGVw==", + "cpu": [ + "mips64el" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "evals/framework/node_modules/@esbuild/linux-ppc64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.25.12.tgz", + "integrity": "sha512-9meM/lRXxMi5PSUqEXRCtVjEZBGwB7P/D4yT8UG/mwIdze2aV4Vo6U5gD3+RsoHXKkHCfSxZKzmDssVlRj1QQA==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "evals/framework/node_modules/@esbuild/linux-riscv64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.25.12.tgz", + "integrity": "sha512-Zr7KR4hgKUpWAwb1f3o5ygT04MzqVrGEGXGLnj15YQDJErYu/BGg+wmFlIDOdJp0PmB0lLvxFIOXZgFRrdjR0w==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "evals/framework/node_modules/@esbuild/linux-s390x": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.25.12.tgz", + "integrity": "sha512-MsKncOcgTNvdtiISc/jZs/Zf8d0cl/t3gYWX8J9ubBnVOwlk65UIEEvgBORTiljloIWnBzLs4qhzPkJcitIzIg==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "evals/framework/node_modules/@esbuild/linux-x64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.25.12.tgz", + "integrity": "sha512-uqZMTLr/zR/ed4jIGnwSLkaHmPjOjJvnm6TVVitAa08SLS9Z0VM8wIRx7gWbJB5/J54YuIMInDquWyYvQLZkgw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "evals/framework/node_modules/@esbuild/netbsd-x64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.25.12.tgz", + "integrity": "sha512-Ld5pTlzPy3YwGec4OuHh1aCVCRvOXdH8DgRjfDy/oumVovmuSzWfnSJg+VtakB9Cm0gxNO9BzWkj6mtO1FMXkQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=18" + } + }, + "evals/framework/node_modules/@esbuild/openbsd-x64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.25.12.tgz", + "integrity": "sha512-MZyXUkZHjQxUvzK7rN8DJ3SRmrVrke8ZyRusHlP+kuwqTcfWLyqMOE3sScPPyeIXN/mDJIfGXvcMqCgYKekoQw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=18" + } + }, + "evals/framework/node_modules/@esbuild/sunos-x64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.25.12.tgz", + "integrity": "sha512-3wGSCDyuTHQUzt0nV7bocDy72r2lI33QL3gkDNGkod22EsYl04sMf0qLb8luNKTOmgF/eDEDP5BFNwoBKH441w==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "sunos" + ], + "engines": { + "node": ">=18" + } + }, + "evals/framework/node_modules/@esbuild/win32-arm64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.25.12.tgz", + "integrity": "sha512-rMmLrur64A7+DKlnSuwqUdRKyd3UE7oPJZmnljqEptesKM8wx9J8gx5u0+9Pq0fQQW8vqeKebwNXdfOyP+8Bsg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, + "evals/framework/node_modules/@esbuild/win32-ia32": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.25.12.tgz", + "integrity": "sha512-HkqnmmBoCbCwxUKKNPBixiWDGCpQGVsrQfJoVGYLPT41XWF8lHuE5N6WhVia2n4o5QK5M4tYr21827fNhi4byQ==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, + "evals/framework/node_modules/@esbuild/win32-x64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.25.12.tgz", + "integrity": "sha512-alJC0uCZpTFrSL0CCDjcgleBXPnCrEAhTBILpeAp7M/OFgoqtAetfBzX0xM00MUsVVPpVjlPuMbREqnZCXaTnA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" } }, "evals/framework/node_modules/@eslint-community/eslint-utils": { @@ -191,22 +549,6 @@ "node": "20 || >=22" } }, - "evals/framework/node_modules/@jest/schemas": { - "version": "29.6.3", - "dev": true, - "license": "MIT", - "dependencies": { - "@sinclair/typebox": "^0.27.8" - }, - "engines": { - "node": "^14.15.0 || ^16.10.0 || >=18.0.0" - } - }, - "evals/framework/node_modules/@jridgewell/sourcemap-codec": { - "version": "1.5.5", - "dev": true, - "license": "MIT" - }, "evals/framework/node_modules/@nodelib/fs.scandir": { "version": "2.1.5", "dev": true, @@ -242,28 +584,6 @@ "evals/framework/node_modules/@opencode-ai/sdk": { "version": "1.0.90" }, - "evals/framework/node_modules/@rollup/rollup-darwin-arm64": { - "version": "4.53.3", - "cpu": [ - "arm64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "darwin" - ] - }, - "evals/framework/node_modules/@sinclair/typebox": { - "version": "0.27.8", - "dev": true, - "license": "MIT" - }, - "evals/framework/node_modules/@types/estree": { - "version": "1.0.8", - "dev": true, - "license": "MIT" - }, "evals/framework/node_modules/@types/glob": { "version": "8.1.0", "dev": true, @@ -485,6 +805,8 @@ }, "evals/framework/node_modules/@vitest/expect": { "version": "1.6.1", + "resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-1.6.1.tgz", + "integrity": "sha512-jXL+9+ZNIJKruofqXuuTClf44eSpcHlgj3CiuNihUF3Ioujtmc0zIa3UJOW5RjDK1YLBJZnWBlPuqhYycLioog==", "dev": true, "license": "MIT", "dependencies": { @@ -498,6 +820,8 @@ }, "evals/framework/node_modules/@vitest/runner": { "version": "1.6.1", + "resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-1.6.1.tgz", + "integrity": "sha512-3nSnYXkVkf3mXFfE7vVyPmi3Sazhb/2cfZGGs0JRzFsPFvAMBEcrweV1V1GsrstdXeKCTXlJbvnQwGWgEIHmOA==", "dev": true, "license": "MIT", "dependencies": { @@ -511,6 +835,8 @@ }, "evals/framework/node_modules/@vitest/runner/node_modules/p-limit": { "version": "5.0.0", + "resolved": "https://registry.npmjs.org/p-limit/-/p-limit-5.0.0.tgz", + "integrity": "sha512-/Eaoq+QyLSiXQ4lyYV23f14mZRQcXnxfHrN0vCai+ak9G0pp9iEQukIIZq5NccEvwRB8PUnZT0KsOoDCINS1qQ==", "dev": true, "license": "MIT", "dependencies": { @@ -525,6 +851,8 @@ }, "evals/framework/node_modules/@vitest/runner/node_modules/yocto-queue": { "version": "1.2.2", + "resolved": "https://registry.npmjs.org/yocto-queue/-/yocto-queue-1.2.2.tgz", + "integrity": "sha512-4LCcse/U2MHZ63HAJVE+v71o7yOdIe4cZ70Wpf8D/IyjDKYQLV5GD46B+hSTjJsvV5PztjvHoU580EftxjDZFQ==", "dev": true, "license": "MIT", "engines": { @@ -536,6 +864,8 @@ }, "evals/framework/node_modules/@vitest/snapshot": { "version": "1.6.1", + "resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-1.6.1.tgz", + "integrity": "sha512-WvidQuWAzU2p95u8GAKlRMqMyN1yOJkGHnx3M1PL9Raf7AQ1kwLKg04ADlCa3+OXUZE7BceOhVZiuWAbzCKcUQ==", "dev": true, "license": "MIT", "dependencies": { @@ -549,6 +879,8 @@ }, "evals/framework/node_modules/@vitest/spy": { "version": "1.6.1", + "resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-1.6.1.tgz", + "integrity": "sha512-MGcMmpGkZebsMZhbQKkAf9CX5zGvjkBTqf8Zx3ApYWXr3wG+QvEu2eXWfnIIWYSJExIp4V9FCKDEeygzkYrXMw==", "dev": true, "license": "MIT", "dependencies": { @@ -560,6 +892,8 @@ }, "evals/framework/node_modules/@vitest/utils": { "version": "1.6.1", + "resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-1.6.1.tgz", + "integrity": "sha512-jOrrUvXM4Av9ZWiG1EajNto0u96kWAhJ1LmPmJhXXQx/32MecEKd10pOLYgS2BQx1TgkGhloPU1ArDW2vvaY6g==", "dev": true, "license": "MIT", "dependencies": { @@ -572,17 +906,6 @@ "url": "https://opencollective.com/vitest" } }, - "evals/framework/node_modules/acorn": { - "version": "8.15.0", - "dev": true, - "license": "MIT", - "bin": { - "acorn": "bin/acorn" - }, - "engines": { - "node": ">=0.4.0" - } - }, "evals/framework/node_modules/acorn-jsx": { "version": "5.3.2", "dev": true, @@ -591,17 +914,6 @@ "acorn": "^6.0.0 || ^7.0.0 || ^8.0.0" } }, - "evals/framework/node_modules/acorn-walk": { - "version": "8.3.4", - "dev": true, - "license": "MIT", - "dependencies": { - "acorn": "^8.11.0" - }, - "engines": { - "node": ">=0.4.0" - } - }, "evals/framework/node_modules/ajv": { "version": "6.12.6", "dev": true, @@ -654,6 +966,8 @@ }, "evals/framework/node_modules/assertion-error": { "version": "1.1.0", + "resolved": "https://registry.npmjs.org/assertion-error/-/assertion-error-1.1.0.tgz", + "integrity": "sha512-jgsaNduz+ndvGyFt3uSuWqvy4lCnIJiovtouQN5JZHOKCS2QuhEdbcQHFhVksz2N2U9hXJo8odG7ETyWlEeuDw==", "dev": true, "license": "MIT", "engines": { @@ -679,14 +993,6 @@ "node": ">=8" } }, - "evals/framework/node_modules/cac": { - "version": "6.7.14", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=8" - } - }, "evals/framework/node_modules/callsites": { "version": "3.1.0", "dev": true, @@ -697,6 +1003,8 @@ }, "evals/framework/node_modules/chai": { "version": "4.5.0", + "resolved": "https://registry.npmjs.org/chai/-/chai-4.5.0.tgz", + "integrity": "sha512-RITGBfijLkBddZvnn8jdqoTypxvqbOLYQkGGxXzeFjVHvudaPw0HNFD9x928/eUwYWd2dPCugVqspGALTZZQKw==", "dev": true, "license": "MIT", "dependencies": { @@ -729,6 +1037,8 @@ }, "evals/framework/node_modules/check-error": { "version": "1.0.3", + "resolved": "https://registry.npmjs.org/check-error/-/check-error-1.0.3.tgz", + "integrity": "sha512-iKEoDYaRmd1mxM90a2OEfWhjsjPpYPuQ+lMYsoxB126+t8fw7ySEO48nmDg5COTjxDI65/Y2OWpeEHk3ZOe8zg==", "dev": true, "license": "MIT", "dependencies": { @@ -754,42 +1064,10 @@ "dev": true, "license": "MIT" }, - "evals/framework/node_modules/confbox": { - "version": "0.1.8", - "dev": true, - "license": "MIT" - }, - "evals/framework/node_modules/cross-spawn": { - "version": "7.0.6", - "dev": true, - "license": "MIT", - "dependencies": { - "path-key": "^3.1.0", - "shebang-command": "^2.0.0", - "which": "^2.0.1" - }, - "engines": { - "node": ">= 8" - } - }, - "evals/framework/node_modules/debug": { - "version": "4.4.3", - "dev": true, - "license": "MIT", - "dependencies": { - "ms": "^2.1.3" - }, - "engines": { - "node": ">=6.0" - }, - "peerDependenciesMeta": { - "supports-color": { - "optional": true - } - } - }, "evals/framework/node_modules/deep-eql": { "version": "4.1.4", + "resolved": "https://registry.npmjs.org/deep-eql/-/deep-eql-4.1.4.tgz", + "integrity": "sha512-SUwdGfqdKOwxCPeVYjwSyRpJ7Z+fhpwIAtmCUdZIWZ/YP5R9WAsyuSgpLVDi9bjWoN2LXHNss/dk3urXtdQxGg==", "dev": true, "license": "MIT", "dependencies": { @@ -804,14 +1082,6 @@ "dev": true, "license": "MIT" }, - "evals/framework/node_modules/diff-sequences": { - "version": "29.6.3", - "dev": true, - "license": "MIT", - "engines": { - "node": "^14.15.0 || ^16.10.0 || >=18.0.0" - } - }, "evals/framework/node_modules/dir-glob": { "version": "3.0.1", "dev": true, @@ -834,43 +1104,6 @@ "node": ">=6.0.0" } }, - "evals/framework/node_modules/esbuild": { - "version": "0.21.5", - "dev": true, - "hasInstallScript": true, - "license": "MIT", - "bin": { - "esbuild": "bin/esbuild" - }, - "engines": { - "node": ">=12" - }, - "optionalDependencies": { - "@esbuild/aix-ppc64": "0.21.5", - "@esbuild/android-arm": "0.21.5", - "@esbuild/android-arm64": "0.21.5", - "@esbuild/android-x64": "0.21.5", - "@esbuild/darwin-arm64": "0.21.5", - "@esbuild/darwin-x64": "0.21.5", - "@esbuild/freebsd-arm64": "0.21.5", - "@esbuild/freebsd-x64": "0.21.5", - "@esbuild/linux-arm": "0.21.5", - "@esbuild/linux-arm64": "0.21.5", - "@esbuild/linux-ia32": "0.21.5", - "@esbuild/linux-loong64": "0.21.5", - "@esbuild/linux-mips64el": "0.21.5", - "@esbuild/linux-ppc64": "0.21.5", - "@esbuild/linux-riscv64": "0.21.5", - "@esbuild/linux-s390x": "0.21.5", - "@esbuild/linux-x64": "0.21.5", - "@esbuild/netbsd-x64": "0.21.5", - "@esbuild/openbsd-x64": "0.21.5", - "@esbuild/sunos-x64": "0.21.5", - "@esbuild/win32-arm64": "0.21.5", - "@esbuild/win32-ia32": "0.21.5", - "@esbuild/win32-x64": "0.21.5" - } - }, "evals/framework/node_modules/escape-string-regexp": { "version": "4.0.0", "dev": true, @@ -1028,14 +1261,6 @@ "node": ">=4.0" } }, - "evals/framework/node_modules/estree-walker": { - "version": "3.0.3", - "dev": true, - "license": "MIT", - "dependencies": { - "@types/estree": "^1.0.0" - } - }, "evals/framework/node_modules/esutils": { "version": "2.0.3", "dev": true, @@ -1044,28 +1269,6 @@ "node": ">=0.10.0" } }, - "evals/framework/node_modules/execa": { - "version": "8.0.1", - "dev": true, - "license": "MIT", - "dependencies": { - "cross-spawn": "^7.0.3", - "get-stream": "^8.0.1", - "human-signals": "^5.0.0", - "is-stream": "^3.0.0", - "merge-stream": "^2.0.0", - "npm-run-path": "^5.1.0", - "onetime": "^6.0.0", - "signal-exit": "^4.1.0", - "strip-final-newline": "^3.0.0" - }, - "engines": { - "node": ">=16.17" - }, - "funding": { - "url": "https://github.com/sindresorhus/execa?sponsor=1" - } - }, "evals/framework/node_modules/fast-glob": { "version": "3.3.3", "dev": true, @@ -1165,37 +1368,6 @@ "dev": true, "license": "ISC" }, - "evals/framework/node_modules/fsevents": { - "version": "2.3.3", - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "darwin" - ], - "engines": { - "node": "^8.16.0 || ^10.6.0 || >=11.0.0" - } - }, - "evals/framework/node_modules/get-func-name": { - "version": "2.0.2", - "dev": true, - "license": "MIT", - "engines": { - "node": "*" - } - }, - "evals/framework/node_modules/get-stream": { - "version": "8.0.1", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=16" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, "evals/framework/node_modules/get-tsconfig": { "version": "4.13.0", "dev": true, @@ -1292,14 +1464,6 @@ "node": ">=8" } }, - "evals/framework/node_modules/human-signals": { - "version": "5.0.0", - "dev": true, - "license": "Apache-2.0", - "engines": { - "node": ">=16.17.0" - } - }, "evals/framework/node_modules/ignore": { "version": "5.3.2", "dev": true, @@ -1366,27 +1530,6 @@ "node": ">=8" } }, - "evals/framework/node_modules/is-stream": { - "version": "3.0.0", - "dev": true, - "license": "MIT", - "engines": { - "node": "^12.20.0 || ^14.13.1 || >=16.0.0" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, - "evals/framework/node_modules/isexe": { - "version": "2.0.0", - "dev": true, - "license": "ISC" - }, - "evals/framework/node_modules/js-tokens": { - "version": "9.0.1", - "dev": true, - "license": "MIT" - }, "evals/framework/node_modules/js-yaml": { "version": "4.1.1", "dev": true, @@ -1433,21 +1576,6 @@ "node": ">= 0.8.0" } }, - "evals/framework/node_modules/local-pkg": { - "version": "0.5.1", - "dev": true, - "license": "MIT", - "dependencies": { - "mlly": "^1.7.3", - "pkg-types": "^1.2.1" - }, - "engines": { - "node": ">=14" - }, - "funding": { - "url": "https://github.com/sponsors/antfu" - } - }, "evals/framework/node_modules/locate-path": { "version": "6.0.0", "dev": true, @@ -1469,6 +1597,8 @@ }, "evals/framework/node_modules/loupe": { "version": "2.3.7", + "resolved": "https://registry.npmjs.org/loupe/-/loupe-2.3.7.tgz", + "integrity": "sha512-zSMINGVYkdpYSOBmLi0D1Uo7JU9nVdQKrHxC8eYlV+9YKK9WePqAlL7lSlorG/U2Fw1w0hTBmaa/jrQ3UbPHtA==", "dev": true, "license": "MIT", "dependencies": { @@ -1482,19 +1612,6 @@ "node": "20 || >=22" } }, - "evals/framework/node_modules/magic-string": { - "version": "0.30.21", - "dev": true, - "license": "MIT", - "dependencies": { - "@jridgewell/sourcemap-codec": "^1.5.5" - } - }, - "evals/framework/node_modules/merge-stream": { - "version": "2.0.0", - "dev": true, - "license": "MIT" - }, "evals/framework/node_modules/merge2": { "version": "1.4.1", "dev": true, @@ -1515,17 +1632,6 @@ "node": ">=8.6" } }, - "evals/framework/node_modules/mimic-fn": { - "version": "4.0.0", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, "evals/framework/node_modules/minimatch": { "version": "9.0.3", "dev": true, @@ -1547,88 +1653,11 @@ "node": ">=16 || 14 >=14.17" } }, - "evals/framework/node_modules/mlly": { - "version": "1.8.0", - "dev": true, - "license": "MIT", - "dependencies": { - "acorn": "^8.15.0", - "pathe": "^2.0.3", - "pkg-types": "^1.3.1", - "ufo": "^1.6.1" - } - }, - "evals/framework/node_modules/mlly/node_modules/pathe": { - "version": "2.0.3", - "dev": true, - "license": "MIT" - }, - "evals/framework/node_modules/ms": { - "version": "2.1.3", - "dev": true, - "license": "MIT" - }, - "evals/framework/node_modules/nanoid": { - "version": "3.3.11", - "dev": true, - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/ai" - } - ], - "license": "MIT", - "bin": { - "nanoid": "bin/nanoid.cjs" - }, - "engines": { - "node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1" - } - }, "evals/framework/node_modules/natural-compare": { "version": "1.4.0", "dev": true, "license": "MIT" }, - "evals/framework/node_modules/npm-run-path": { - "version": "5.3.0", - "dev": true, - "license": "MIT", - "dependencies": { - "path-key": "^4.0.0" - }, - "engines": { - "node": "^12.20.0 || ^14.13.1 || >=16.0.0" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, - "evals/framework/node_modules/npm-run-path/node_modules/path-key": { - "version": "4.0.0", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, - "evals/framework/node_modules/onetime": { - "version": "6.0.0", - "dev": true, - "license": "MIT", - "dependencies": { - "mimic-fn": "^4.0.0" - }, - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, "evals/framework/node_modules/optionator": { "version": "0.9.4", "dev": true, @@ -1692,14 +1721,6 @@ "node": ">=8" } }, - "evals/framework/node_modules/path-key": { - "version": "3.1.1", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=8" - } - }, "evals/framework/node_modules/path-scurry": { "version": "2.0.1", "license": "BlueOak-1.0.0", @@ -1722,24 +1743,16 @@ "node": ">=8" } }, - "evals/framework/node_modules/pathe": { - "version": "1.1.2", - "dev": true, - "license": "MIT" - }, "evals/framework/node_modules/pathval": { "version": "1.1.1", + "resolved": "https://registry.npmjs.org/pathval/-/pathval-1.1.1.tgz", + "integrity": "sha512-Dp6zGqpTdETdR63lehJYPeIOqpiNBNtc7BpWSLrOje7UaIsE5aY92r/AunQA7rsXvet3lrJ3JnZX29UPTKXyKQ==", "dev": true, "license": "MIT", "engines": { "node": "*" } }, - "evals/framework/node_modules/picocolors": { - "version": "1.1.1", - "dev": true, - "license": "ISC" - }, "evals/framework/node_modules/picomatch": { "version": "2.3.1", "dev": true, @@ -1751,48 +1764,6 @@ "url": "https://github.com/sponsors/jonschlinkert" } }, - "evals/framework/node_modules/pkg-types": { - "version": "1.3.1", - "dev": true, - "license": "MIT", - "dependencies": { - "confbox": "^0.1.8", - "mlly": "^1.7.4", - "pathe": "^2.0.1" - } - }, - "evals/framework/node_modules/pkg-types/node_modules/pathe": { - "version": "2.0.3", - "dev": true, - "license": "MIT" - }, - "evals/framework/node_modules/postcss": { - "version": "8.5.6", - "dev": true, - "funding": [ - { - "type": "opencollective", - "url": "https://opencollective.com/postcss/" - }, - { - "type": "tidelift", - "url": "https://tidelift.com/funding/github/npm/postcss" - }, - { - "type": "github", - "url": "https://github.com/sponsors/ai" - } - ], - "license": "MIT", - "dependencies": { - "nanoid": "^3.3.11", - "picocolors": "^1.1.1", - "source-map-js": "^1.2.1" - }, - "engines": { - "node": "^10 || ^12 || >=14" - } - }, "evals/framework/node_modules/prelude-ls": { "version": "1.2.1", "dev": true, @@ -1801,30 +1772,6 @@ "node": ">= 0.8.0" } }, - "evals/framework/node_modules/pretty-format": { - "version": "29.7.0", - "dev": true, - "license": "MIT", - "dependencies": { - "@jest/schemas": "^29.6.3", - "ansi-styles": "^5.0.0", - "react-is": "^18.0.0" - }, - "engines": { - "node": "^14.15.0 || ^16.10.0 || >=18.0.0" - } - }, - "evals/framework/node_modules/pretty-format/node_modules/ansi-styles": { - "version": "5.2.0", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=10" - }, - "funding": { - "url": "https://github.com/chalk/ansi-styles?sponsor=1" - } - }, "evals/framework/node_modules/punycode": { "version": "2.3.1", "dev": true, @@ -1852,11 +1799,6 @@ ], "license": "MIT" }, - "evals/framework/node_modules/react-is": { - "version": "18.3.1", - "dev": true, - "license": "MIT" - }, "evals/framework/node_modules/resolve-from": { "version": "4.0.0", "dev": true, @@ -1935,46 +1877,6 @@ "node": "*" } }, - "evals/framework/node_modules/rollup": { - "version": "4.53.3", - "dev": true, - "license": "MIT", - "dependencies": { - "@types/estree": "1.0.8" - }, - "bin": { - "rollup": "dist/bin/rollup" - }, - "engines": { - "node": ">=18.0.0", - "npm": ">=8.0.0" - }, - "optionalDependencies": { - "@rollup/rollup-android-arm-eabi": "4.53.3", - "@rollup/rollup-android-arm64": "4.53.3", - "@rollup/rollup-darwin-arm64": "4.53.3", - "@rollup/rollup-darwin-x64": "4.53.3", - "@rollup/rollup-freebsd-arm64": "4.53.3", - "@rollup/rollup-freebsd-x64": "4.53.3", - "@rollup/rollup-linux-arm-gnueabihf": "4.53.3", - "@rollup/rollup-linux-arm-musleabihf": "4.53.3", - "@rollup/rollup-linux-arm64-gnu": "4.53.3", - "@rollup/rollup-linux-arm64-musl": "4.53.3", - "@rollup/rollup-linux-loong64-gnu": "4.53.3", - "@rollup/rollup-linux-ppc64-gnu": "4.53.3", - "@rollup/rollup-linux-riscv64-gnu": "4.53.3", - "@rollup/rollup-linux-riscv64-musl": "4.53.3", - "@rollup/rollup-linux-s390x-gnu": "4.53.3", - "@rollup/rollup-linux-x64-gnu": "4.53.3", - "@rollup/rollup-linux-x64-musl": "4.53.3", - "@rollup/rollup-openharmony-arm64": "4.53.3", - "@rollup/rollup-win32-arm64-msvc": "4.53.3", - "@rollup/rollup-win32-ia32-msvc": "4.53.3", - "@rollup/rollup-win32-x64-gnu": "4.53.3", - "@rollup/rollup-win32-x64-msvc": "4.53.3", - "fsevents": "~2.3.2" - } - }, "evals/framework/node_modules/run-parallel": { "version": "1.2.0", "dev": true, @@ -2008,41 +1910,6 @@ "node": ">=10" } }, - "evals/framework/node_modules/shebang-command": { - "version": "2.0.0", - "dev": true, - "license": "MIT", - "dependencies": { - "shebang-regex": "^3.0.0" - }, - "engines": { - "node": ">=8" - } - }, - "evals/framework/node_modules/shebang-regex": { - "version": "3.0.0", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=8" - } - }, - "evals/framework/node_modules/siginfo": { - "version": "2.0.0", - "dev": true, - "license": "ISC" - }, - "evals/framework/node_modules/signal-exit": { - "version": "4.1.0", - "dev": true, - "license": "ISC", - "engines": { - "node": ">=14" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, "evals/framework/node_modules/slash": { "version": "3.0.0", "dev": true, @@ -2051,24 +1918,6 @@ "node": ">=8" } }, - "evals/framework/node_modules/source-map-js": { - "version": "1.2.1", - "dev": true, - "license": "BSD-3-Clause", - "engines": { - "node": ">=0.10.0" - } - }, - "evals/framework/node_modules/stackback": { - "version": "0.0.2", - "dev": true, - "license": "MIT" - }, - "evals/framework/node_modules/std-env": { - "version": "3.10.0", - "dev": true, - "license": "MIT" - }, "evals/framework/node_modules/strip-ansi": { "version": "6.0.1", "dev": true, @@ -2080,17 +1929,6 @@ "node": ">=8" } }, - "evals/framework/node_modules/strip-final-newline": { - "version": "3.0.0", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, "evals/framework/node_modules/strip-json-comments": { "version": "3.1.1", "dev": true, @@ -2102,17 +1940,6 @@ "url": "https://github.com/sponsors/sindresorhus" } }, - "evals/framework/node_modules/strip-literal": { - "version": "2.1.1", - "dev": true, - "license": "MIT", - "dependencies": { - "js-tokens": "^9.0.1" - }, - "funding": { - "url": "https://github.com/sponsors/antfu" - } - }, "evals/framework/node_modules/supports-color": { "version": "7.2.0", "dev": true, @@ -2129,13 +1956,10 @@ "dev": true, "license": "MIT" }, - "evals/framework/node_modules/tinybench": { - "version": "2.9.0", - "dev": true, - "license": "MIT" - }, "evals/framework/node_modules/tinypool": { "version": "0.8.4", + "resolved": "https://registry.npmjs.org/tinypool/-/tinypool-0.8.4.tgz", + "integrity": "sha512-i11VH5gS6IFeLY3gMBQ00/MmLncVP7JLXOw1vlgkytLmJK7QnEr7NXf0LBdxfmNPAeyetukOk0bOYrJrFGjYJQ==", "dev": true, "license": "MIT", "engines": { @@ -2144,6 +1968,8 @@ }, "evals/framework/node_modules/tinyspy": { "version": "2.2.1", + "resolved": "https://registry.npmjs.org/tinyspy/-/tinyspy-2.2.1.tgz", + "integrity": "sha512-KYad6Vy5VDWV4GH3fjpseMQ/XU2BhIYP7Vzd0LG44qRWm/Yt2WCOTicFdvmgo6gWaqooMQCawTtILVQJupKu7A==", "dev": true, "license": "MIT", "engines": { @@ -2256,14 +2082,6 @@ "node": ">= 0.8.0" } }, - "evals/framework/node_modules/type-detect": { - "version": "4.1.0", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=4" - } - }, "evals/framework/node_modules/type-fest": { "version": "0.20.2", "dev": true, @@ -2287,11 +2105,6 @@ "node": ">=14.17" } }, - "evals/framework/node_modules/ufo": { - "version": "1.6.1", - "dev": true, - "license": "MIT" - }, "evals/framework/node_modules/undici-types": { "version": "6.21.0", "dev": true, @@ -2305,66 +2118,10 @@ "punycode": "^2.1.0" } }, - "evals/framework/node_modules/vite": { - "version": "5.4.21", - "dev": true, - "license": "MIT", - "dependencies": { - "esbuild": "^0.21.3", - "postcss": "^8.4.43", - "rollup": "^4.20.0" - }, - "bin": { - "vite": "bin/vite.js" - }, - "engines": { - "node": "^18.0.0 || >=20.0.0" - }, - "funding": { - "url": "https://github.com/vitejs/vite?sponsor=1" - }, - "optionalDependencies": { - "fsevents": "~2.3.3" - }, - "peerDependencies": { - "@types/node": "^18.0.0 || >=20.0.0", - "less": "*", - "lightningcss": "^1.21.0", - "sass": "*", - "sass-embedded": "*", - "stylus": "*", - "sugarss": "*", - "terser": "^5.4.0" - }, - "peerDependenciesMeta": { - "@types/node": { - "optional": true - }, - "less": { - "optional": true - }, - "lightningcss": { - "optional": true - }, - "sass": { - "optional": true - }, - "sass-embedded": { - "optional": true - }, - "stylus": { - "optional": true - }, - "sugarss": { - "optional": true - }, - "terser": { - "optional": true - } - } - }, "evals/framework/node_modules/vite-node": { "version": "1.6.1", + "resolved": "https://registry.npmjs.org/vite-node/-/vite-node-1.6.1.tgz", + "integrity": "sha512-YAXkfvGtuTzwWbDSACdJSg4A4DZiAqckWe90Zapc/sEX3XvHcw1NdurM/6od8J207tSDqNbSsgdCacBgvJKFuA==", "dev": true, "license": "MIT", "dependencies": { @@ -2386,6 +2143,8 @@ }, "evals/framework/node_modules/vitest": { "version": "1.6.1", + "resolved": "https://registry.npmjs.org/vitest/-/vitest-1.6.1.tgz", + "integrity": "sha512-Ljb1cnSJSivGN0LqXd/zmDbWEM0RNNg2t1QW/XUhYl/qPqyu7CsqeWtqQXHVaJsecLPuDoak2oJcZN2QoRIOag==", "dev": true, "license": "MIT", "dependencies": { @@ -2448,35 +2207,6 @@ } } }, - "evals/framework/node_modules/which": { - "version": "2.0.2", - "dev": true, - "license": "ISC", - "dependencies": { - "isexe": "^2.0.0" - }, - "bin": { - "node-which": "bin/node-which" - }, - "engines": { - "node": ">= 8" - } - }, - "evals/framework/node_modules/why-is-node-running": { - "version": "2.3.0", - "dev": true, - "license": "MIT", - "dependencies": { - "siginfo": "^2.0.0", - "stackback": "0.0.2" - }, - "bin": { - "why-is-node-running": "cli.js" - }, - "engines": { - "node": ">=8" - } - }, "evals/framework/node_modules/word-wrap": { "version": "1.2.5", "dev": true, @@ -2513,62 +2243,831 @@ "url": "https://github.com/sponsors/colinhacks" } }, + "node_modules/@esbuild/aix-ppc64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.21.5.tgz", + "integrity": "sha512-1SDgH6ZSPTlggy1yI6+Dbkiz8xzpHJEVAlF/AM1tHPLsf5STom9rwtjE4hKAF20FfXXNTFqEYXyJNWh1GiZedQ==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "aix" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/android-arm": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.21.5.tgz", + "integrity": "sha512-vCPvzSjpPHEi1siZdlvAlsPxXl7WbOVUBBAowWug4rJHb68Ox8KualB+1ocNvT5fjv6wpkX6o/iEpbDrf68zcg==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/android-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.21.5.tgz", + "integrity": "sha512-c0uX9VAUBQ7dTDCjq+wdyGLowMdtR/GoC2U5IYk/7D1H1JYC0qseD7+11iMP2mRLN9RcCMRcjC4YMclCzGwS/A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/android-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.21.5.tgz", + "integrity": "sha512-D7aPRUUNHRBwHxzxRvp856rjUHRFW1SdQATKXH2hqA0kAZb1hKmi02OpYRacl0TxIGz/ZmXWlbZgjwWYaCakTA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/darwin-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.21.5.tgz", + "integrity": "sha512-DwqXqZyuk5AiWWf3UfLiRDJ5EDd49zg6O9wclZ7kUMv2WRFr4HKjXp/5t8JZ11QbQfUS6/cRCKGwYhtNAY88kQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/darwin-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.21.5.tgz", + "integrity": "sha512-se/JjF8NlmKVG4kNIuyWMV/22ZaerB+qaSi5MdrXtd6R08kvs2qCN4C09miupktDitvh8jRFflwGFBQcxZRjbw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/freebsd-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.21.5.tgz", + "integrity": "sha512-5JcRxxRDUJLX8JXp/wcBCy3pENnCgBR9bN6JsY4OmhfUtIHe3ZW0mawA7+RDAcMLrMIZaf03NlQiX9DGyB8h4g==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/freebsd-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.21.5.tgz", + "integrity": "sha512-J95kNBj1zkbMXtHVH29bBriQygMXqoVQOQYA+ISs0/2l3T9/kj42ow2mpqerRBxDJnmkUDCaQT/dfNXWX/ZZCQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-arm": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.21.5.tgz", + "integrity": "sha512-bPb5AHZtbeNGjCKVZ9UGqGwo8EUu4cLq68E95A53KlxAPRmUyYv2D6F0uUI65XisGOL1hBP5mTronbgo+0bFcA==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.21.5.tgz", + "integrity": "sha512-ibKvmyYzKsBeX8d8I7MH/TMfWDXBF3db4qM6sy+7re0YXya+K1cem3on9XgdT2EQGMu4hQyZhan7TeQ8XkGp4Q==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-ia32": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.21.5.tgz", + "integrity": "sha512-YvjXDqLRqPDl2dvRODYmmhz4rPeVKYvppfGYKSNGdyZkA01046pLWyRKKI3ax8fbJoK5QbxblURkwK/MWY18Tg==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-loong64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.21.5.tgz", + "integrity": "sha512-uHf1BmMG8qEvzdrzAqg2SIG/02+4/DHB6a9Kbya0XDvwDEKCoC8ZRWI5JJvNdUjtciBGFQ5PuBlpEOXQj+JQSg==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-mips64el": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.21.5.tgz", + "integrity": "sha512-IajOmO+KJK23bj52dFSNCMsz1QP1DqM6cwLUv3W1QwyxkyIWecfafnI555fvSGqEKwjMXVLokcV5ygHW5b3Jbg==", + "cpu": [ + "mips64el" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-ppc64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.21.5.tgz", + "integrity": "sha512-1hHV/Z4OEfMwpLO8rp7CvlhBDnjsC3CttJXIhBi+5Aj5r+MBvy4egg7wCbe//hSsT+RvDAG7s81tAvpL2XAE4w==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-riscv64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.21.5.tgz", + "integrity": "sha512-2HdXDMd9GMgTGrPWnJzP2ALSokE/0O5HhTUvWIbD3YdjME8JwvSCnNGBnTThKGEB91OZhzrJ4qIIxk/SBmyDDA==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-s390x": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.21.5.tgz", + "integrity": "sha512-zus5sxzqBJD3eXxwvjN1yQkRepANgxE9lgOW2qLnmr8ikMTphkjgXu1HR01K4FJg8h1kEEDAqDcZQtbrRnB41A==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.21.5.tgz", + "integrity": "sha512-1rYdTpyv03iycF1+BhzrzQJCdOuAOtaqHTWJZCWvijKD2N5Xu0TtVC8/+1faWqcP9iBCWOmjmhoH94dH82BxPQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/netbsd-arm64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.25.12.tgz", + "integrity": "sha512-xXwcTq4GhRM7J9A8Gv5boanHhRa/Q9KLVmcyXHCTaM4wKfIpWkdXiMog/KsnxzJ0A1+nD+zoecuzqPmCRyBGjg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/netbsd-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.21.5.tgz", + "integrity": "sha512-Woi2MXzXjMULccIwMnLciyZH4nCIMpWQAs049KEeMvOcNADVxo0UBIQPfSmxB3CWKedngg7sWZdLvLczpe0tLg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/openbsd-arm64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.25.12.tgz", + "integrity": "sha512-fF96T6KsBo/pkQI950FARU9apGNTSlZGsv1jZBAlcLL1MLjLNIWPBkj5NlSz8aAzYKg+eNqknrUJ24QBybeR5A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/openbsd-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.21.5.tgz", + "integrity": "sha512-HLNNw99xsvx12lFBUwoT8EVCsSvRNDVxNpjZ7bPn947b8gJPzeHWyNVhFsaerc0n3TsbOINvRP2byTZ5LKezow==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/openharmony-arm64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/openharmony-arm64/-/openharmony-arm64-0.25.12.tgz", + "integrity": "sha512-rm0YWsqUSRrjncSXGA7Zv78Nbnw4XL6/dzr20cyrQf7ZmRcsovpcRBdhD43Nuk3y7XIoW2OxMVvwuRvk9XdASg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openharmony" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/sunos-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.21.5.tgz", + "integrity": "sha512-6+gjmFpfy0BHU5Tpptkuh8+uw3mnrvgs+dSPQXQOv3ekbordwnzTVEb4qnIvQcYXq6gzkyTnoZ9dZG+D4garKg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "sunos" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/win32-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.21.5.tgz", + "integrity": "sha512-Z0gOTd75VvXqyq7nsl93zwahcTROgqvuAcYDUr+vOv8uHhNSKROyU961kgtCD1e95IqPKSQKH7tBTslnS3tA8A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/win32-ia32": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.21.5.tgz", + "integrity": "sha512-SWXFF1CL2RVNMaVs+BBClwtfZSvDgtL//G/smwAc5oVK/UPu2Gu9tIaRgFmYFFKrmg3SyAjSrElf0TiJ1v8fYA==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/win32-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.21.5.tgz", + "integrity": "sha512-tQd/1efJuzPC6rCFwEvLtci/xNFcTZknmXs98FYDfGE4wP9ClFV98nyKrzJKVPMhdDnjzLhdUyMX4PsQAPjwIw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@jest/schemas": { + "version": "29.6.3", + "resolved": "https://registry.npmjs.org/@jest/schemas/-/schemas-29.6.3.tgz", + "integrity": "sha512-mo5j5X+jIZmJQveBKeS/clAueipV7KgiX1vMgCxam1RNYiqE1w62n0/tJJnHtjW8ZHcQco5gY85jA3mi0L+nSA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@sinclair/typebox": "^0.27.8" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/@jridgewell/sourcemap-codec": { + "version": "1.5.5", + "resolved": "https://registry.npmjs.org/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.5.5.tgz", + "integrity": "sha512-cYQ9310grqxueWbl+WuIUIaiUaDcj7WOq5fVhEljNVgRfOUhY9fy2zTvfoqWsnebh8Sl70VScFbICvJnLKB0Og==", + "dev": true, + "license": "MIT" + }, "node_modules/@opencode-agents/eval-framework": { "resolved": "evals/framework", "link": true }, - "node_modules/ajv": { - "version": "8.17.1", - "resolved": "https://registry.npmjs.org/ajv/-/ajv-8.17.1.tgz", - "integrity": "sha512-B/gBuNg5SiMTrPkC+A2+cW0RszwxYmn6VYxB/inlBStS5nx6xHIt/ehKRhIMhqusl7a8LjQoZnjCs5vhwxOQ1g==", + "node_modules/@rollup/rollup-android-arm-eabi": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm-eabi/-/rollup-android-arm-eabi-4.53.3.tgz", + "integrity": "sha512-mRSi+4cBjrRLoaal2PnqH82Wqyb+d3HsPUN/W+WslCXsZsyHa9ZeQQX/pQsZaVIWDkPcpV6jJ+3KLbTbgnwv8w==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ] + }, + "node_modules/@rollup/rollup-android-arm64": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm64/-/rollup-android-arm64-4.53.3.tgz", + "integrity": "sha512-CbDGaMpdE9sh7sCmTrTUyllhrg65t6SwhjlMJsLr+J8YjFuPmCEjbBSx4Z/e4SmDyH3aB5hGaJUP2ltV/vcs4w==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ] + }, + "node_modules/@rollup/rollup-darwin-arm64": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-arm64/-/rollup-darwin-arm64-4.53.3.tgz", + "integrity": "sha512-Nr7SlQeqIBpOV6BHHGZgYBuSdanCXuw09hon14MGOLGmXAFYjx1wNvquVPmpZnl0tLjg25dEdr4IQ6GgyToCUA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@rollup/rollup-darwin-x64": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-x64/-/rollup-darwin-x64-4.53.3.tgz", + "integrity": "sha512-DZ8N4CSNfl965CmPktJ8oBnfYr3F8dTTNBQkRlffnUarJ2ohudQD17sZBa097J8xhQ26AwhHJ5mvUyQW8ddTsQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@rollup/rollup-freebsd-arm64": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-arm64/-/rollup-freebsd-arm64-4.53.3.tgz", + "integrity": "sha512-yMTrCrK92aGyi7GuDNtGn2sNW+Gdb4vErx4t3Gv/Tr+1zRb8ax4z8GWVRfr3Jw8zJWvpGHNpss3vVlbF58DZ4w==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ] + }, + "node_modules/@rollup/rollup-freebsd-x64": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-x64/-/rollup-freebsd-x64-4.53.3.tgz", + "integrity": "sha512-lMfF8X7QhdQzseM6XaX0vbno2m3hlyZFhwcndRMw8fbAGUGL3WFMBdK0hbUBIUYcEcMhVLr1SIamDeuLBnXS+Q==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ] + }, + "node_modules/@rollup/rollup-linux-arm-gnueabihf": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-gnueabihf/-/rollup-linux-arm-gnueabihf-4.53.3.tgz", + "integrity": "sha512-k9oD15soC/Ln6d2Wv/JOFPzZXIAIFLp6B+i14KhxAfnq76ajt0EhYc5YPeX6W1xJkAdItcVT+JhKl1QZh44/qw==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm-musleabihf": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-musleabihf/-/rollup-linux-arm-musleabihf-4.53.3.tgz", + "integrity": "sha512-vTNlKq+N6CK/8UktsrFuc+/7NlEYVxgaEgRXVUVK258Z5ymho29skzW1sutgYjqNnquGwVUObAaxae8rZ6YMhg==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm64-gnu": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-gnu/-/rollup-linux-arm64-gnu-4.53.3.tgz", + "integrity": "sha512-RGrFLWgMhSxRs/EWJMIFM1O5Mzuz3Xy3/mnxJp/5cVhZ2XoCAxJnmNsEyeMJtpK+wu0FJFWz+QF4mjCA7AUQ3w==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm64-musl": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-musl/-/rollup-linux-arm64-musl-4.53.3.tgz", + "integrity": "sha512-kASyvfBEWYPEwe0Qv4nfu6pNkITLTb32p4yTgzFCocHnJLAHs+9LjUu9ONIhvfT/5lv4YS5muBHyuV84epBo/A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-loong64-gnu": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-loong64-gnu/-/rollup-linux-loong64-gnu-4.53.3.tgz", + "integrity": "sha512-JiuKcp2teLJwQ7vkJ95EwESWkNRFJD7TQgYmCnrPtlu50b4XvT5MOmurWNrCj3IFdyjBQ5p9vnrX4JM6I8OE7g==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-ppc64-gnu": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-ppc64-gnu/-/rollup-linux-ppc64-gnu-4.53.3.tgz", + "integrity": "sha512-EoGSa8nd6d3T7zLuqdojxC20oBfNT8nexBbB/rkxgKj5T5vhpAQKKnD+h3UkoMuTyXkP5jTjK/ccNRmQrPNDuw==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-riscv64-gnu": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-gnu/-/rollup-linux-riscv64-gnu-4.53.3.tgz", + "integrity": "sha512-4s+Wped2IHXHPnAEbIB0YWBv7SDohqxobiiPA1FIWZpX+w9o2i4LezzH/NkFUl8LRci/8udci6cLq+jJQlh+0g==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-riscv64-musl": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-musl/-/rollup-linux-riscv64-musl-4.53.3.tgz", + "integrity": "sha512-68k2g7+0vs2u9CxDt5ktXTngsxOQkSEV/xBbwlqYcUrAVh6P9EgMZvFsnHy4SEiUl46Xf0IObWVbMvPrr2gw8A==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-s390x-gnu": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-s390x-gnu/-/rollup-linux-s390x-gnu-4.53.3.tgz", + "integrity": "sha512-VYsFMpULAz87ZW6BVYw3I6sWesGpsP9OPcyKe8ofdg9LHxSbRMd7zrVrr5xi/3kMZtpWL/wC+UIJWJYVX5uTKg==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-x64-gnu": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-gnu/-/rollup-linux-x64-gnu-4.53.3.tgz", + "integrity": "sha512-3EhFi1FU6YL8HTUJZ51imGJWEX//ajQPfqWLI3BQq4TlvHy4X0MOr5q3D2Zof/ka0d5FNdPwZXm3Yyib/UEd+w==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-x64-musl": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-musl/-/rollup-linux-x64-musl-4.53.3.tgz", + "integrity": "sha512-eoROhjcc6HbZCJr+tvVT8X4fW3/5g/WkGvvmwz/88sDtSJzO7r/blvoBDgISDiCjDRZmHpwud7h+6Q9JxFwq1Q==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-openharmony-arm64": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/@rollup/rollup-openharmony-arm64/-/rollup-openharmony-arm64-4.53.3.tgz", + "integrity": "sha512-OueLAWgrNSPGAdUdIjSWXw+u/02BRTcnfw9PN41D2vq/JSEPnJnVuBgw18VkN8wcd4fjUs+jFHVM4t9+kBSNLw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openharmony" + ] + }, + "node_modules/@rollup/rollup-win32-arm64-msvc": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-arm64-msvc/-/rollup-win32-arm64-msvc-4.53.3.tgz", + "integrity": "sha512-GOFuKpsxR/whszbF/bzydebLiXIHSgsEUp6M0JI8dWvi+fFa1TD6YQa4aSZHtpmh2/uAlj/Dy+nmby3TJ3pkTw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-ia32-msvc": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-ia32-msvc/-/rollup-win32-ia32-msvc-4.53.3.tgz", + "integrity": "sha512-iah+THLcBJdpfZ1TstDFbKNznlzoxa8fmnFYK4V67HvmuNYkVdAywJSoteUszvBQ9/HqN2+9AZghbajMsFT+oA==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-x64-gnu": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-x64-gnu/-/rollup-win32-x64-gnu-4.53.3.tgz", + "integrity": "sha512-J9QDiOIZlZLdcot5NXEepDkstocktoVjkaKUtqzgzpt2yWjGlbYiKyp05rWwk4nypbYUNoFAztEgixoLaSETkg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-x64-msvc": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-x64-msvc/-/rollup-win32-x64-msvc-4.53.3.tgz", + "integrity": "sha512-UhTd8u31dXadv0MopwGgNOBpUVROFKWVQgAg5N1ESyCz8AuBcMqm4AuTjrwgQKGDfoFuz02EuMRHQIw/frmYKQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@sinclair/typebox": { + "version": "0.27.8", + "resolved": "https://registry.npmjs.org/@sinclair/typebox/-/typebox-0.27.8.tgz", + "integrity": "sha512-+Fj43pSMwJs4KRrH/938Uf+uAELIgVBmQzg/q1YG10djyfA3TnrU8N8XzqCh/okZdszqBQTZf96idMfE5lnwTA==", + "dev": true, + "license": "MIT" + }, + "node_modules/@types/estree": { + "version": "1.0.8", + "resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.8.tgz", + "integrity": "sha512-dWHzHa2WqEXI/O1E9OjrocMTKJl2mSrEolh1Iomrv6U+JuNwaHXsXx9bLu5gG7BUWFIN0skIQJQ/L1rIex4X6w==", + "dev": true, + "license": "MIT" + }, + "node_modules/acorn": { + "version": "8.15.0", + "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.15.0.tgz", + "integrity": "sha512-NZyJarBfL7nWwIq+FDL6Zp/yHEhePMNnnJ0y3qfieCrmNvYct8uvtiV41UvlSe6apAfk0fY1FbWx+NwfmpvtTg==", + "dev": true, + "license": "MIT", + "bin": { + "acorn": "bin/acorn" + }, + "engines": { + "node": ">=0.4.0" + } + }, + "node_modules/acorn-walk": { + "version": "8.3.4", + "resolved": "https://registry.npmjs.org/acorn-walk/-/acorn-walk-8.3.4.tgz", + "integrity": "sha512-ueEepnujpqee2o5aIYnvHU6C0A42MNdsIDeqy5BydrkuC5R1ZuUFnm27EeFJGoEHJQgn3uleRvmTXaJgfXbt4g==", "dev": true, "license": "MIT", "dependencies": { - "fast-deep-equal": "^3.1.3", - "fast-uri": "^3.0.1", - "json-schema-traverse": "^1.0.0", - "require-from-string": "^2.0.2" + "acorn": "^8.11.0" + }, + "engines": { + "node": ">=0.4.0" + } + }, + "node_modules/ansi-styles": { + "version": "5.2.0", + "resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-5.2.0.tgz", + "integrity": "sha512-Cxwpt2SfTzTtXcfOlzGEee8O+c+MmUgGrNiBcXnuWxuFJHe6a5Hz7qwhwe5OgaSYI0IJvkLqWX1ASG+cJOkEiA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=10" }, "funding": { - "type": "github", - "url": "https://github.com/sponsors/epoberezkin" - } - }, - "node_modules/ajv-cli": { - "version": "5.0.0", - "resolved": "https://registry.npmjs.org/ajv-cli/-/ajv-cli-5.0.0.tgz", - "integrity": "sha512-LY4m6dUv44HTyhV+u2z5uX4EhPYTM38Iv1jdgDJJJCyOOuqB8KtZEGjPZ2T+sh5ZIJrXUfgErYx/j3gLd3+PlQ==", - "dev": true, - "license": "MIT", - "dependencies": { - "ajv": "^8.0.0", - "fast-json-patch": "^2.0.0", - "glob": "^7.1.0", - "js-yaml": "^3.14.0", - "json-schema-migrate": "^2.0.0", - "json5": "^2.1.3", - "minimist": "^1.2.0" - }, - "bin": { - "ajv": "dist/index.js" - }, - "peerDependencies": { - "ts-node": ">=9.0.0" - }, - "peerDependenciesMeta": { - "ts-node": { - "optional": true - } - } - }, - "node_modules/argparse": { - "version": "1.0.10", - "resolved": "https://registry.npmjs.org/argparse/-/argparse-1.0.10.tgz", - "integrity": "sha512-o5Roy6tNG4SL/FOkCAN6RzjiakZS25RLYFrcMttJqbdd8BWrnA+fGz57iN5Pb06pvBGvl5gQ0B48dJlslXvoTg==", - "dev": true, - "license": "MIT", - "dependencies": { - "sprintf-js": "~1.0.2" + "url": "https://github.com/chalk/ansi-styles?sponsor=1" } }, "node_modules/balanced-match": { @@ -2578,15 +3077,14 @@ "dev": true, "license": "MIT" }, - "node_modules/brace-expansion": { - "version": "1.1.12", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.12.tgz", - "integrity": "sha512-9T9UjW3r0UW5c1Q7GTwllptXwhvYmEzFhzMfZ9H7FQWt+uZePjZPjBP/W1ZEyZ1twGWom5/56TF4lPcqjnDHcg==", + "node_modules/cac": { + "version": "6.7.14", + "resolved": "https://registry.npmjs.org/cac/-/cac-6.7.14.tgz", + "integrity": "sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==", "dev": true, "license": "MIT", - "dependencies": { - "balanced-match": "^1.0.0", - "concat-map": "0.0.1" + "engines": { + "node": ">=8" } }, "node_modules/concat-map": { @@ -2596,18 +3094,127 @@ "dev": true, "license": "MIT" }, - "node_modules/esprima": { - "version": "4.0.1", - "resolved": "https://registry.npmjs.org/esprima/-/esprima-4.0.1.tgz", - "integrity": "sha512-eGuFFw7Upda+g4p+QHvnW0RyTX/SVeJBDM/gCtMARO0cLuT2HcEKnTPvhjV6aGeqrCB/sbNop0Kszm0jsaWU4A==", + "node_modules/confbox": { + "version": "0.1.8", + "resolved": "https://registry.npmjs.org/confbox/-/confbox-0.1.8.tgz", + "integrity": "sha512-RMtmw0iFkeR4YV+fUOSucriAQNb9g8zFR52MWCtl+cCZOFRNL6zeB395vPzFhEjjn4fMxXudmELnl/KF/WrK6w==", "dev": true, - "license": "BSD-2-Clause", - "bin": { - "esparse": "bin/esparse.js", - "esvalidate": "bin/esvalidate.js" + "license": "MIT" + }, + "node_modules/cross-spawn": { + "version": "7.0.6", + "resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz", + "integrity": "sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA==", + "dev": true, + "license": "MIT", + "dependencies": { + "path-key": "^3.1.0", + "shebang-command": "^2.0.0", + "which": "^2.0.1" }, "engines": { - "node": ">=4" + "node": ">= 8" + } + }, + "node_modules/debug": { + "version": "4.4.3", + "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", + "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", + "dev": true, + "license": "MIT", + "dependencies": { + "ms": "^2.1.3" + }, + "engines": { + "node": ">=6.0" + }, + "peerDependenciesMeta": { + "supports-color": { + "optional": true + } + } + }, + "node_modules/diff-sequences": { + "version": "29.6.3", + "resolved": "https://registry.npmjs.org/diff-sequences/-/diff-sequences-29.6.3.tgz", + "integrity": "sha512-EjePK1srD3P08o2j4f0ExnylqRs5B9tJjcp9t1krH2qRi8CCdsYfwe9JgSLurFBWwq4uOlipzfk5fHNvwFKr8Q==", + "dev": true, + "license": "MIT", + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/esbuild": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.21.5.tgz", + "integrity": "sha512-mg3OPMV4hXywwpoDxu3Qda5xCKQi+vCTZq8S9J/EpkhB2HzKXq4SNFZE3+NK93JYxc8VMSep+lOUSC/RVKaBqw==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "bin": { + "esbuild": "bin/esbuild" + }, + "engines": { + "node": ">=12" + }, + "optionalDependencies": { + "@esbuild/aix-ppc64": "0.21.5", + "@esbuild/android-arm": "0.21.5", + "@esbuild/android-arm64": "0.21.5", + "@esbuild/android-x64": "0.21.5", + "@esbuild/darwin-arm64": "0.21.5", + "@esbuild/darwin-x64": "0.21.5", + "@esbuild/freebsd-arm64": "0.21.5", + "@esbuild/freebsd-x64": "0.21.5", + "@esbuild/linux-arm": "0.21.5", + "@esbuild/linux-arm64": "0.21.5", + "@esbuild/linux-ia32": "0.21.5", + "@esbuild/linux-loong64": "0.21.5", + "@esbuild/linux-mips64el": "0.21.5", + "@esbuild/linux-ppc64": "0.21.5", + "@esbuild/linux-riscv64": "0.21.5", + "@esbuild/linux-s390x": "0.21.5", + "@esbuild/linux-x64": "0.21.5", + "@esbuild/netbsd-x64": "0.21.5", + "@esbuild/openbsd-x64": "0.21.5", + "@esbuild/sunos-x64": "0.21.5", + "@esbuild/win32-arm64": "0.21.5", + "@esbuild/win32-ia32": "0.21.5", + "@esbuild/win32-x64": "0.21.5" + } + }, + "node_modules/estree-walker": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/estree-walker/-/estree-walker-3.0.3.tgz", + "integrity": "sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/estree": "^1.0.0" + } + }, + "node_modules/execa": { + "version": "8.0.1", + "resolved": "https://registry.npmjs.org/execa/-/execa-8.0.1.tgz", + "integrity": "sha512-VyhnebXciFV2DESc+p6B+y0LjSm0krU4OgJN44qFAhBY0TJ+1V61tYD2+wHusZ6F9n5K+vl8k0sTy7PEfV4qpg==", + "dev": true, + "license": "MIT", + "dependencies": { + "cross-spawn": "^7.0.3", + "get-stream": "^8.0.1", + "human-signals": "^5.0.0", + "is-stream": "^3.0.0", + "merge-stream": "^2.0.0", + "npm-run-path": "^5.1.0", + "onetime": "^6.0.0", + "signal-exit": "^4.1.0", + "strip-final-newline": "^3.0.0" + }, + "engines": { + "node": ">=16.17" + }, + "funding": { + "url": "https://github.com/sindresorhus/execa?sponsor=1" } }, "node_modules/fast-deep-equal": { @@ -2617,43 +3224,6 @@ "dev": true, "license": "MIT" }, - "node_modules/fast-json-patch": { - "version": "2.2.1", - "resolved": "https://registry.npmjs.org/fast-json-patch/-/fast-json-patch-2.2.1.tgz", - "integrity": "sha512-4j5uBaTnsYAV5ebkidvxiLUYOwjQ+JSFljeqfTxCrH9bDmlCQaOJFS84oDJ2rAXZq2yskmk3ORfoP9DCwqFNig==", - "dev": true, - "license": "MIT", - "dependencies": { - "fast-deep-equal": "^2.0.1" - }, - "engines": { - "node": ">= 0.4.0" - } - }, - "node_modules/fast-json-patch/node_modules/fast-deep-equal": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/fast-deep-equal/-/fast-deep-equal-2.0.1.tgz", - "integrity": "sha512-bCK/2Z4zLidyB4ReuIsvALH6w31YfAQDmXMqMx6FyfHqvBxtjC0eRumeSu4Bs3XtXwpyIywtSTrVT99BxY1f9w==", - "dev": true, - "license": "MIT" - }, - "node_modules/fast-uri": { - "version": "3.1.0", - "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.0.tgz", - "integrity": "sha512-iPeeDKJSWf4IEOasVVrknXpaBV0IApz/gp7S2bb7Z4Lljbl2MGJRqInZiUrQwV16cpzw/D3S5j5Julj/gT52AA==", - "dev": true, - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/fastify" - }, - { - "type": "opencollective", - "url": "https://opencollective.com/fastify" - } - ], - "license": "BSD-3-Clause" - }, "node_modules/fs.realpath": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/fs.realpath/-/fs.realpath-1.0.0.tgz", @@ -2661,26 +3231,52 @@ "dev": true, "license": "ISC" }, - "node_modules/glob": { - "version": "7.2.3", - "resolved": "https://registry.npmjs.org/glob/-/glob-7.2.3.tgz", - "integrity": "sha512-nFR0zLpU2YCaRxwoCJvL6UvCH2JFyFVIvwTLsIf21AuHlMskA1hhTdk+LlYJtOlYt9v6dvszD2BGRqBL+iQK9Q==", - "deprecated": "Glob versions prior to v9 are no longer supported", + "node_modules/fsevents": { + "version": "2.3.3", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", + "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==", "dev": true, - "license": "ISC", - "dependencies": { - "fs.realpath": "^1.0.0", - "inflight": "^1.0.4", - "inherits": "2", - "minimatch": "^3.1.1", - "once": "^1.3.0", - "path-is-absolute": "^1.0.0" - }, + "hasInstallScript": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^8.16.0 || ^10.6.0 || >=11.0.0" + } + }, + "node_modules/get-func-name": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/get-func-name/-/get-func-name-2.0.2.tgz", + "integrity": "sha512-8vXOvuE167CtIc3OyItco7N/dpRtBbYOsPsXCz7X/PMnlGjYjSGuZJgM1Y7mmew7BKf9BqvLX2tnOVy1BBUsxQ==", + "dev": true, + "license": "MIT", "engines": { "node": "*" + } + }, + "node_modules/get-stream": { + "version": "8.0.1", + "resolved": "https://registry.npmjs.org/get-stream/-/get-stream-8.0.1.tgz", + "integrity": "sha512-VaUJspBffn/LMCJVoMvSAdmscJyS1auj5Zulnn5UoYcY531UWmdwhRWkcGKnGU93m5HSXP9LP2usOryrBtQowA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=16" }, "funding": { - "url": "https://github.com/sponsors/isaacs" + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/human-signals": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/human-signals/-/human-signals-5.0.0.tgz", + "integrity": "sha512-AXcZb6vzzrFAUE61HnN4mpLqd/cSIwNQjtNWR0euPm6y0iqx3G4gOXaIDdtdDwZmhwe82LA6+zinmW4UBWVePQ==", + "dev": true, + "license": "Apache-2.0", + "engines": { + "node": ">=16.17.0" } }, "node_modules/inflight": { @@ -2702,71 +3298,153 @@ "dev": true, "license": "ISC" }, - "node_modules/js-yaml": { - "version": "3.14.2", - "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-3.14.2.tgz", - "integrity": "sha512-PMSmkqxr106Xa156c2M265Z+FTrPl+oxd/rgOQy2tijQeK5TxQ43psO1ZCwhVOSdnn+RzkzlRz/eY4BgJBYVpg==", + "node_modules/is-stream": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/is-stream/-/is-stream-3.0.0.tgz", + "integrity": "sha512-LnQR4bZ9IADDRSkvpqMGvt/tEJWclzklNgSw48V5EAaAeDd6qGvN8ei6k5p0tvxSR171VmGyHuTiAOfxAbr8kA==", "dev": true, "license": "MIT", - "dependencies": { - "argparse": "^1.0.7", - "esprima": "^4.0.0" + "engines": { + "node": "^12.20.0 || ^14.13.1 || >=16.0.0" }, - "bin": { - "js-yaml": "bin/js-yaml.js" + "funding": { + "url": "https://github.com/sponsors/sindresorhus" } }, - "node_modules/json-schema-migrate": { + "node_modules/isexe": { "version": "2.0.0", - "resolved": "https://registry.npmjs.org/json-schema-migrate/-/json-schema-migrate-2.0.0.tgz", - "integrity": "sha512-r38SVTtojDRp4eD6WsCqiE0eNDt4v1WalBXb9cyZYw9ai5cGtBwzRNWjHzJl38w6TxFkXAIA7h+fyX3tnrAFhQ==", + "resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz", + "integrity": "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==", "dev": true, - "license": "MIT", - "dependencies": { - "ajv": "^8.0.0" - } + "license": "ISC" }, - "node_modules/json-schema-traverse": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/json-schema-traverse/-/json-schema-traverse-1.0.0.tgz", - "integrity": "sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug==", + "node_modules/js-tokens": { + "version": "9.0.1", + "resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-9.0.1.tgz", + "integrity": "sha512-mxa9E9ITFOt0ban3j6L5MpjwegGz6lBQmM1IJkWeBZGcMxto50+eWdjC/52xDbS2vy0k7vIMK0Fe2wfL9OQSpQ==", "dev": true, "license": "MIT" }, - "node_modules/json5": { - "version": "2.2.3", - "resolved": "https://registry.npmjs.org/json5/-/json5-2.2.3.tgz", - "integrity": "sha512-XmOWe7eyHYH14cLdVPoyg+GOH3rYX++KpzrylJwSW98t3Nk+U8XOl8FWKOgwtzdb8lXGf6zYwDUzeHMWfxasyg==", + "node_modules/local-pkg": { + "version": "0.5.1", + "resolved": "https://registry.npmjs.org/local-pkg/-/local-pkg-0.5.1.tgz", + "integrity": "sha512-9rrA30MRRP3gBD3HTGnC6cDFpaE1kVDWxWgqWJUN0RvDNAo+Nz/9GxB+nHOH0ifbVFy0hSA1V6vFDvnx54lTEQ==", "dev": true, "license": "MIT", + "dependencies": { + "mlly": "^1.7.3", + "pkg-types": "^1.2.1" + }, + "engines": { + "node": ">=14" + }, + "funding": { + "url": "https://github.com/sponsors/antfu" + } + }, + "node_modules/magic-string": { + "version": "0.30.21", + "resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.21.tgz", + "integrity": "sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/sourcemap-codec": "^1.5.5" + } + }, + "node_modules/merge-stream": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/merge-stream/-/merge-stream-2.0.0.tgz", + "integrity": "sha512-abv/qOcuPfk3URPfDzmZU1LKmuw8kT+0nIHvKrKgFrwifol/doWcdA4ZqsWQ8ENrFKkd67Mfpo/LovbIUsbt3w==", + "dev": true, + "license": "MIT" + }, + "node_modules/mimic-fn": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/mimic-fn/-/mimic-fn-4.0.0.tgz", + "integrity": "sha512-vqiC06CuhBTUdZH+RYl8sFrL096vA45Ok5ISO6sE/Mr1jRbGH4Csnhi8f3wKVl7x8mO4Au7Ir9D3Oyv1VYMFJw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/mlly": { + "version": "1.8.0", + "resolved": "https://registry.npmjs.org/mlly/-/mlly-1.8.0.tgz", + "integrity": "sha512-l8D9ODSRWLe2KHJSifWGwBqpTZXIXTeo8mlKjY+E2HAakaTeNpqAyBZ8GSqLzHgw4XmHmC8whvpjJNMbFZN7/g==", + "dev": true, + "license": "MIT", + "dependencies": { + "acorn": "^8.15.0", + "pathe": "^2.0.3", + "pkg-types": "^1.3.1", + "ufo": "^1.6.1" + } + }, + "node_modules/mlly/node_modules/pathe": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/pathe/-/pathe-2.0.3.tgz", + "integrity": "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w==", + "dev": true, + "license": "MIT" + }, + "node_modules/ms": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", + "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", + "dev": true, + "license": "MIT" + }, + "node_modules/nanoid": { + "version": "3.3.11", + "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.11.tgz", + "integrity": "sha512-N8SpfPUnUp1bK+PMYW8qSWdl9U+wwNWI4QKxOYDy9JAro3WMX7p2OeVRF9v+347pnakNevPmiHhNmZ2HbFA76w==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", "bin": { - "json5": "lib/cli.js" + "nanoid": "bin/nanoid.cjs" }, "engines": { - "node": ">=6" + "node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1" } }, - "node_modules/minimatch": { - "version": "3.1.2", - "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.1.2.tgz", - "integrity": "sha512-J7p63hRiAjw1NDEww1W7i37+ByIrOWO5XQQAzZ3VOcL0PNybwpfmV/N05zFAzwQ9USyEcX6t3UO+K5aqBQOIHw==", - "dev": true, - "license": "ISC", - "dependencies": { - "brace-expansion": "^1.1.7" - }, - "engines": { - "node": "*" - } - }, - "node_modules/minimist": { - "version": "1.2.8", - "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.8.tgz", - "integrity": "sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA==", + "node_modules/npm-run-path": { + "version": "5.3.0", + "resolved": "https://registry.npmjs.org/npm-run-path/-/npm-run-path-5.3.0.tgz", + "integrity": "sha512-ppwTtiJZq0O/ai0z7yfudtBpWIoxM8yE6nHi1X47eFR2EWORqfbu6CnPlNsjeN683eT0qG6H/Pyf9fCcvjnnnQ==", "dev": true, "license": "MIT", + "dependencies": { + "path-key": "^4.0.0" + }, + "engines": { + "node": "^12.20.0 || ^14.13.1 || >=16.0.0" + }, "funding": { - "url": "https://github.com/sponsors/ljharb" + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/npm-run-path/node_modules/path-key": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/path-key/-/path-key-4.0.0.tgz", + "integrity": "sha512-haREypq7xkM7ErfgIyA0z+Bj4AGKlMSdlQE2jvJo6huWD1EdkKYV+G/T4nq0YEF2vgTT8kqMFKo1uHn950r4SQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" } }, "node_modules/once": { @@ -2779,6 +3457,22 @@ "wrappy": "1" } }, + "node_modules/onetime": { + "version": "6.0.0", + "resolved": "https://registry.npmjs.org/onetime/-/onetime-6.0.0.tgz", + "integrity": "sha512-1FlR+gjXK7X+AsAHso35MnyN5KqGwJRi/31ft6x0M194ht7S+rWAvd7PHss9xSKMzE0asv1pyIHaJYq+BbacAQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "mimic-fn": "^4.0.0" + }, + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, "node_modules/path-is-absolute": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/path-is-absolute/-/path-is-absolute-1.0.1.tgz", @@ -2789,22 +3483,351 @@ "node": ">=0.10.0" } }, - "node_modules/require-from-string": { - "version": "2.0.2", - "resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz", - "integrity": "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==", + "node_modules/path-key": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/path-key/-/path-key-3.1.1.tgz", + "integrity": "sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==", "dev": true, "license": "MIT", + "engines": { + "node": ">=8" + } + }, + "node_modules/pathe": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/pathe/-/pathe-1.1.2.tgz", + "integrity": "sha512-whLdWMYL2TwI08hn8/ZqAbrVemu0LNaNNJZX73O6qaIdCTfXutsLhMkjdENX0qhsQ9uIimo4/aQOmXkoon2nDQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/picocolors": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz", + "integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==", + "dev": true, + "license": "ISC" + }, + "node_modules/pkg-types": { + "version": "1.3.1", + "resolved": "https://registry.npmjs.org/pkg-types/-/pkg-types-1.3.1.tgz", + "integrity": "sha512-/Jm5M4RvtBFVkKWRu2BLUTNP8/M2a+UwuAX+ae4770q1qVGtfjG+WTCupoZixokjmHiry8uI+dlY8KXYV5HVVQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "confbox": "^0.1.8", + "mlly": "^1.7.4", + "pathe": "^2.0.1" + } + }, + "node_modules/pkg-types/node_modules/pathe": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/pathe/-/pathe-2.0.3.tgz", + "integrity": "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w==", + "dev": true, + "license": "MIT" + }, + "node_modules/postcss": { + "version": "8.5.6", + "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.6.tgz", + "integrity": "sha512-3Ybi1tAuwAP9s0r1UQ2J4n5Y0G05bJkpUIO0/bI9MhwmD70S5aTWbXGBwxHrelT+XM1k6dM0pk+SwNkpTRN7Pg==", + "dev": true, + "funding": [ + { + "type": "opencollective", + "url": "https://opencollective.com/postcss/" + }, + { + "type": "tidelift", + "url": "https://tidelift.com/funding/github/npm/postcss" + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "dependencies": { + "nanoid": "^3.3.11", + "picocolors": "^1.1.1", + "source-map-js": "^1.2.1" + }, + "engines": { + "node": "^10 || ^12 || >=14" + } + }, + "node_modules/pretty-format": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/pretty-format/-/pretty-format-29.7.0.tgz", + "integrity": "sha512-Pdlw/oPxN+aXdmM9R00JVC9WVFoCLTKJvDVLgmJ+qAffBMxsV85l/Lu7sNx4zSzPyoL2euImuEwHhOXdEgNFZQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jest/schemas": "^29.6.3", + "ansi-styles": "^5.0.0", + "react-is": "^18.0.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/react-is": { + "version": "18.3.1", + "resolved": "https://registry.npmjs.org/react-is/-/react-is-18.3.1.tgz", + "integrity": "sha512-/LLMVyas0ljjAtoYiPqYiL8VWXzUUdThrmU5+n20DZv+a+ClRoevUzw5JxU+Ieh5/c87ytoTBV9G1FiKfNJdmg==", + "dev": true, + "license": "MIT" + }, + "node_modules/rollup": { + "version": "4.53.3", + "resolved": "https://registry.npmjs.org/rollup/-/rollup-4.53.3.tgz", + "integrity": "sha512-w8GmOxZfBmKknvdXU1sdM9NHcoQejwF/4mNgj2JuEEdRaHwwF12K7e9eXn1nLZ07ad+du76mkVsyeb2rKGllsA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/estree": "1.0.8" + }, + "bin": { + "rollup": "dist/bin/rollup" + }, + "engines": { + "node": ">=18.0.0", + "npm": ">=8.0.0" + }, + "optionalDependencies": { + "@rollup/rollup-android-arm-eabi": "4.53.3", + "@rollup/rollup-android-arm64": "4.53.3", + "@rollup/rollup-darwin-arm64": "4.53.3", + "@rollup/rollup-darwin-x64": "4.53.3", + "@rollup/rollup-freebsd-arm64": "4.53.3", + "@rollup/rollup-freebsd-x64": "4.53.3", + "@rollup/rollup-linux-arm-gnueabihf": "4.53.3", + "@rollup/rollup-linux-arm-musleabihf": "4.53.3", + "@rollup/rollup-linux-arm64-gnu": "4.53.3", + "@rollup/rollup-linux-arm64-musl": "4.53.3", + "@rollup/rollup-linux-loong64-gnu": "4.53.3", + "@rollup/rollup-linux-ppc64-gnu": "4.53.3", + "@rollup/rollup-linux-riscv64-gnu": "4.53.3", + "@rollup/rollup-linux-riscv64-musl": "4.53.3", + "@rollup/rollup-linux-s390x-gnu": "4.53.3", + "@rollup/rollup-linux-x64-gnu": "4.53.3", + "@rollup/rollup-linux-x64-musl": "4.53.3", + "@rollup/rollup-openharmony-arm64": "4.53.3", + "@rollup/rollup-win32-arm64-msvc": "4.53.3", + "@rollup/rollup-win32-ia32-msvc": "4.53.3", + "@rollup/rollup-win32-x64-gnu": "4.53.3", + "@rollup/rollup-win32-x64-msvc": "4.53.3", + "fsevents": "~2.3.2" + } + }, + "node_modules/shebang-command": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/shebang-command/-/shebang-command-2.0.0.tgz", + "integrity": "sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA==", + "dev": true, + "license": "MIT", + "dependencies": { + "shebang-regex": "^3.0.0" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/shebang-regex": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/shebang-regex/-/shebang-regex-3.0.0.tgz", + "integrity": "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=8" + } + }, + "node_modules/siginfo": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/siginfo/-/siginfo-2.0.0.tgz", + "integrity": "sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==", + "dev": true, + "license": "ISC" + }, + "node_modules/signal-exit": { + "version": "4.1.0", + "resolved": "https://registry.npmjs.org/signal-exit/-/signal-exit-4.1.0.tgz", + "integrity": "sha512-bzyZ1e88w9O1iNJbKnOlvYTrWPDl46O1bG0D3XInv+9tkPrxrN8jUUTiFlDkkmKWgn1M6CfIA13SuGqOa9Korw==", + "dev": true, + "license": "ISC", + "engines": { + "node": ">=14" + }, + "funding": { + "url": "https://github.com/sponsors/isaacs" + } + }, + "node_modules/source-map-js": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz", + "integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==", + "dev": true, + "license": "BSD-3-Clause", "engines": { "node": ">=0.10.0" } }, - "node_modules/sprintf-js": { - "version": "1.0.3", - "resolved": "https://registry.npmjs.org/sprintf-js/-/sprintf-js-1.0.3.tgz", - "integrity": "sha512-D9cPgkvLlV3t3IzL0D0YLvGA9Ahk4PcvVwUbN0dSGr1aP0Nrt4AEnTUbuGvquEC0mA64Gqt1fzirlRs5ibXx8g==", + "node_modules/stackback": { + "version": "0.0.2", + "resolved": "https://registry.npmjs.org/stackback/-/stackback-0.0.2.tgz", + "integrity": "sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==", "dev": true, - "license": "BSD-3-Clause" + "license": "MIT" + }, + "node_modules/std-env": { + "version": "3.10.0", + "resolved": "https://registry.npmjs.org/std-env/-/std-env-3.10.0.tgz", + "integrity": "sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==", + "dev": true, + "license": "MIT" + }, + "node_modules/strip-final-newline": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/strip-final-newline/-/strip-final-newline-3.0.0.tgz", + "integrity": "sha512-dOESqjYr96iWYylGObzd39EuNTa5VJxyvVAEm5Jnh7KGo75V43Hk1odPQkNDyXNmUR6k+gEiDVXnjB8HJ3crXw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/strip-literal": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/strip-literal/-/strip-literal-2.1.1.tgz", + "integrity": "sha512-631UJ6O00eNGfMiWG78ck80dfBab8X6IVFB51jZK5Icd7XAs60Z5y7QdSd/wGIklnWvRbUNloVzhOKKmutxQ6Q==", + "dev": true, + "license": "MIT", + "dependencies": { + "js-tokens": "^9.0.1" + }, + "funding": { + "url": "https://github.com/sponsors/antfu" + } + }, + "node_modules/tinybench": { + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/tinybench/-/tinybench-2.9.0.tgz", + "integrity": "sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==", + "dev": true, + "license": "MIT" + }, + "node_modules/type-detect": { + "version": "4.1.0", + "resolved": "https://registry.npmjs.org/type-detect/-/type-detect-4.1.0.tgz", + "integrity": "sha512-Acylog8/luQ8L7il+geoSxhEkazvkslg7PSNKOX59mbB9cOveP5aq9h74Y7YU8yDpJwetzQQrfIwtf4Wp4LKcw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=4" + } + }, + "node_modules/ufo": { + "version": "1.6.1", + "resolved": "https://registry.npmjs.org/ufo/-/ufo-1.6.1.tgz", + "integrity": "sha512-9a4/uxlTWJ4+a5i0ooc1rU7C7YOw3wT+UGqdeNNHWnOF9qcMBgLRS+4IYUqbczewFx4mLEig6gawh7X6mFlEkA==", + "dev": true, + "license": "MIT" + }, + "node_modules/vite": { + "version": "5.4.21", + "resolved": "https://registry.npmjs.org/vite/-/vite-5.4.21.tgz", + "integrity": "sha512-o5a9xKjbtuhY6Bi5S3+HvbRERmouabWbyUcpXXUA1u+GNUKoROi9byOJ8M0nHbHYHkYICiMlqxkg1KkYmm25Sw==", + "dev": true, + "license": "MIT", + "dependencies": { + "esbuild": "^0.21.3", + "postcss": "^8.4.43", + "rollup": "^4.20.0" + }, + "bin": { + "vite": "bin/vite.js" + }, + "engines": { + "node": "^18.0.0 || >=20.0.0" + }, + "funding": { + "url": "https://github.com/vitejs/vite?sponsor=1" + }, + "optionalDependencies": { + "fsevents": "~2.3.3" + }, + "peerDependencies": { + "@types/node": "^18.0.0 || >=20.0.0", + "less": "*", + "lightningcss": "^1.21.0", + "sass": "*", + "sass-embedded": "*", + "stylus": "*", + "sugarss": "*", + "terser": "^5.4.0" + }, + "peerDependenciesMeta": { + "@types/node": { + "optional": true + }, + "less": { + "optional": true + }, + "lightningcss": { + "optional": true + }, + "sass": { + "optional": true + }, + "sass-embedded": { + "optional": true + }, + "stylus": { + "optional": true + }, + "sugarss": { + "optional": true + }, + "terser": { + "optional": true + } + } + }, + "node_modules/which": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/which/-/which-2.0.2.tgz", + "integrity": "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA==", + "dev": true, + "license": "ISC", + "dependencies": { + "isexe": "^2.0.0" + }, + "bin": { + "node-which": "bin/node-which" + }, + "engines": { + "node": ">= 8" + } + }, + "node_modules/why-is-node-running": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/why-is-node-running/-/why-is-node-running-2.3.0.tgz", + "integrity": "sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==", + "dev": true, + "license": "MIT", + "dependencies": { + "siginfo": "^2.0.0", + "stackback": "0.0.2" + }, + "bin": { + "why-is-node-running": "cli.js" + }, + "engines": { + "node": ">=8" + } }, "node_modules/wrappy": { "version": "1.0.2", diff --git a/package.json b/package.json index 68cad24..2ceba12 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "opencode-agents", - "version": "0.0.2", + "version": "0.3.0", "description": "OpenCode agent evaluation framework and test suites", "private": true, "workspaces": [ diff --git a/scripts/development/demo.sh b/scripts/development/demo.sh index c7a0697..8dbbbf2 100755 --- a/scripts/development/demo.sh +++ b/scripts/development/demo.sh @@ -154,16 +154,21 @@ show_prompt_library() { cat << EOF The prompt library allows different AI models to use optimized prompts: +${GREEN}.opencode/agent/${NC} +└── openagent.md ${YELLOW}# Canonical default (Claude-optimized)${NC} + ${GREEN}.opencode/prompts/openagent/${NC} -├── default.md ${YELLOW}# Stable version (used in PRs)${NC} -├── sonnet-4.md ${YELLOW}# Claude Sonnet 4 optimized${NC} -├── grok-fast.md ${YELLOW}# Grok Fast optimized${NC} +├── gpt.md ${YELLOW}# GPT-4 optimized${NC} +├── gemini.md ${YELLOW}# Gemini optimized${NC} +├── grok.md ${YELLOW}# Grok optimized${NC} +├── llama.md ${YELLOW}# Llama/OSS optimized${NC} ├── TEMPLATE.md ${YELLOW}# Template for new variants${NC} -└── results/ ${YELLOW}# Test results${NC} +└── results/ ${YELLOW}# Test results (all variants)${NC} ${BLUE}Key Principles:${NC} - • PRs must use default prompts (enforced by CI) - • Variants are tested and documented + • Agent files are canonical defaults (source of truth) + • Variants are model-specific optimizations + • All variants tested and documented • Users can choose the best prompt for their model • Contributors can add optimized variants diff --git a/scripts/prompts/test-prompt.sh b/scripts/prompts/test-prompt.sh index e3d5662..e343f05 100755 --- a/scripts/prompts/test-prompt.sh +++ b/scripts/prompts/test-prompt.sh @@ -83,36 +83,41 @@ usage() { echo "" echo "Required:" echo " --agent=NAME Agent name (e.g., openagent, opencoder)" - echo " --variant=NAME Prompt variant by model family (e.g., default, gpt, gemini, grok, llama)" + echo " --variant=NAME Prompt variant (default, gpt, gemini, grok, llama)" echo "" echo "Optional:" echo " --model=MODEL Model to test with (uses prompt metadata if not specified)" echo " --help, -h Show this help" echo "" + echo -e "${BLUE}Architecture:${NC}" + echo " • default = Canonical agent file (.opencode/agent/.md)" + echo " • Other variants = Model-specific optimizations (.opencode/prompts//.md)" + echo " • Results always saved to .opencode/prompts//results/" + echo "" echo "Examples:" - echo " # Test with default (Claude) - uses metadata recommendation" + echo " # Test default (canonical agent file)" echo " $0 --agent=openagent --variant=default" echo "" - echo " # Test GPT prompt - uses metadata recommendation (gpt-4o)" + echo " # Test GPT-optimized prompt" echo " $0 --agent=openagent --variant=gpt" echo "" echo " # Test Gemini prompt with specific model" echo " $0 --agent=openagent --variant=gemini --model=google/gemini-2.0-flash-exp" echo "" - echo " # Test Grok prompt - uses metadata recommendation (free tier)" + echo " # Test Grok prompt" echo " $0 --agent=openagent --variant=grok" echo "" echo "Available model families:" - echo " default # Claude Sonnet 4.5 (stable)" + echo " default # Canonical agent file (Claude Sonnet 4.5)" echo " gpt # OpenAI GPT-4o, GPT-4o-mini, o1" echo " gemini # Google Gemini 2.0 Flash, Pro" echo " grok # xAI Grok (free tier available)" echo " llama # Meta Llama 3.1/3.2 (local or hosted)" echo "" - echo "Note: Each prompt contains metadata with recommended models." + echo "Note: Model-specific prompts contain metadata with recommended models." echo " If --model is not specified, the primary recommendation is used." echo "" - echo "Available prompts for an agent:" + echo "Available variants for an agent:" echo " ls $PROMPTS_DIR//" exit 1 } @@ -150,19 +155,29 @@ if [[ -z "$AGENT_NAME" ]] || [[ -z "$PROMPT_VARIANT" ]]; then usage fi -PROMPT_FILE="$PROMPTS_DIR/$AGENT_NAME/$PROMPT_VARIANT.md" AGENT_FILE="$AGENT_DIR/$AGENT_NAME.md" BACKUP_FILE="$AGENT_DIR/.$AGENT_NAME.md.backup" VARIANT_RESULTS_DIR="$PROMPTS_DIR/$AGENT_NAME/results" VARIANT_RESULTS_FILE="$VARIANT_RESULTS_DIR/$PROMPT_VARIANT-results.json" -# Check prompt exists -if [[ ! -f "$PROMPT_FILE" ]]; then - echo -e "${RED}Error: Prompt not found: $PROMPT_FILE${NC}" - echo "" - echo "Available prompts for $AGENT_NAME:" - ls -1 "$PROMPTS_DIR/$AGENT_NAME/"*.md 2>/dev/null | xargs -I {} basename {} .md || echo " (none found)" - exit 1 +# Handle "default" variant - use agent file directly +if [[ "$PROMPT_VARIANT" == "default" ]]; then + PROMPT_FILE="$AGENT_FILE" + echo -e "${BLUE}Testing default prompt (canonical agent file)${NC}" +else + PROMPT_FILE="$PROMPTS_DIR/$AGENT_NAME/$PROMPT_VARIANT.md" + + # Check prompt exists + if [[ ! -f "$PROMPT_FILE" ]]; then + echo -e "${RED}Error: Prompt variant not found: $PROMPT_FILE${NC}" + echo "" + echo "Available variants for $AGENT_NAME:" + echo " - default (canonical agent file)" + if [[ -d "$PROMPTS_DIR/$AGENT_NAME" ]]; then + ls -1 "$PROMPTS_DIR/$AGENT_NAME/"*.md 2>/dev/null | grep -v "TEMPLATE.md" | grep -v "README.md" | xargs -I {} basename {} .md || echo " (no model variants found)" + fi + exit 1 + fi fi # Read metadata from prompt file @@ -215,11 +230,16 @@ else echo " No existing agent file to backup" fi -# Step 2: Copy prompt variant to agent location -echo -e "${YELLOW}[2/5] Copying prompt variant to agent location...${NC}" -cp "$PROMPT_FILE" "$AGENT_FILE" -echo " Copied $PROMPT_FILE" -echo " To $AGENT_FILE" +# Step 2: Copy prompt variant to agent location (skip if testing default) +if [[ "$PROMPT_VARIANT" == "default" ]]; then + echo -e "${YELLOW}[2/5] Using default prompt (already in place)...${NC}" + echo " Testing: $AGENT_FILE" +else + echo -e "${YELLOW}[2/5] Copying prompt variant to agent location...${NC}" + cp "$PROMPT_FILE" "$AGENT_FILE" + echo " Copied $PROMPT_FILE" + echo " To $AGENT_FILE" +fi # Step 3: Run tests echo -e "${YELLOW}[3/5] Running core eval tests...${NC}" @@ -249,18 +269,18 @@ set -e echo "" echo -e "${BLUE}═══════════════════════════════════════════════════════════════${NC}" -# Step 4: Restore default prompt +# Step 4: Restore original prompt (if we changed it) echo "" -echo -e "${YELLOW}[4/5] Restoring default prompt...${NC}" -DEFAULT_PROMPT="$PROMPTS_DIR/$AGENT_NAME/default.md" -if [[ -f "$DEFAULT_PROMPT" ]]; then - cp "$DEFAULT_PROMPT" "$AGENT_FILE" - echo " Restored default.md to agent location" +if [[ "$PROMPT_VARIANT" == "default" ]]; then + echo -e "${YELLOW}[4/5] No restore needed (tested default)...${NC}" + echo " Agent file unchanged" else - # Restore backup if no default + echo -e "${YELLOW}[4/5] Restoring original prompt...${NC}" if [[ -f "$BACKUP_FILE" ]]; then cp "$BACKUP_FILE" "$AGENT_FILE" echo " Restored from backup" + else + echo " No backup to restore" fi fi diff --git a/scripts/prompts/use-prompt.sh b/scripts/prompts/use-prompt.sh index 1f62a94..8423038 100755 --- a/scripts/prompts/use-prompt.sh +++ b/scripts/prompts/use-prompt.sh @@ -3,12 +3,16 @@ # use-prompt.sh - Switch an agent to use a specific prompt variant # # Usage: +# ./scripts/prompts/use-prompt.sh --agent=openagent --variant=gpt # ./scripts/prompts/use-prompt.sh --agent=openagent --variant=default -# ./scripts/prompts/use-prompt.sh --agent=openagent --variant=sonnet-4 # # What it does: -# 1. Copies the specified prompt to the agent location -# 2. That's it - the agent now uses that prompt +# 1. If variant is "default", restores the canonical agent file (no-op, already default) +# 2. Otherwise, copies the model-specific variant to the agent location +# +# New Architecture: +# - Agent files (.opencode/agent/*.md) are the canonical defaults +# - Prompt variants (.opencode/prompts//.md) are model-specific optimizations # set -e @@ -20,6 +24,7 @@ ROOT_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)" RED='\033[0;31m' GREEN='\033[0;32m' YELLOW='\033[1;33m' +BLUE='\033[0;34m' NC='\033[0m' # Default values @@ -74,13 +79,17 @@ usage() { echo "" echo "Required:" echo " --agent=NAME Agent name (e.g., openagent, opencoder)" - echo " --variant=NAME Prompt variant by model family (e.g., default, gpt, gemini, grok, llama)" + echo " --variant=NAME Prompt variant (default, gpt, gemini, grok, llama)" echo "" echo "Optional:" echo " --help, -h Show this help" echo "" + echo -e "${BLUE}Architecture:${NC}" + echo " • Agent files (.opencode/agent/*.md) = Canonical defaults" + echo " • Prompt variants (.opencode/prompts//.md) = Model-specific" + echo "" echo "Examples:" - echo " # Use default (Claude Sonnet 4.5)" + echo " # Use default (canonical agent file)" echo " $0 --agent=openagent --variant=default" echo "" echo " # Switch to GPT-optimized prompt" @@ -89,18 +98,21 @@ usage() { echo " # Switch to Gemini-optimized prompt" echo " $0 --agent=openagent --variant=gemini" echo "" - echo " # Switch to Grok-optimized prompt" - echo " $0 --agent=openagent --variant=grok" - echo "" echo "Available prompts:" for agent_dir in "$PROMPTS_DIR"/*/; do + [ -d "$agent_dir" ] || continue agent=$(basename "$agent_dir") echo " $agent:" + echo " - default (canonical agent file)" for prompt in "$agent_dir"*.md; do if [[ -f "$prompt" ]] && [[ "$(basename "$prompt")" != "TEMPLATE.md" ]] && [[ "$(basename "$prompt")" != "README.md" ]]; then variant=$(basename "$prompt" .md) model_family=$(extract_metadata "$prompt" "model_family") - echo " - $variant ($model_family family)" + if [[ -n "$model_family" ]]; then + echo " - $variant ($model_family family)" + else + echo " - $variant" + fi fi done done @@ -136,15 +148,43 @@ if [[ -z "$AGENT_NAME" ]] || [[ -z "$PROMPT_VARIANT" ]]; then usage fi -PROMPT_FILE="$PROMPTS_DIR/$AGENT_NAME/$PROMPT_VARIANT.md" AGENT_FILE="$AGENT_DIR/$AGENT_NAME.md" +# Check agent exists +if [[ ! -f "$AGENT_FILE" ]]; then + echo -e "${RED}Error: Agent not found: $AGENT_FILE${NC}" + echo "" + echo "Available agents:" + ls -1 "$AGENT_DIR/"*.md 2>/dev/null | xargs -I {} basename {} .md || echo " (none found)" + exit 1 +fi + +# Handle "default" variant - agent file is already the default +if [[ "$PROMPT_VARIANT" == "default" ]]; then + echo -e "${GREEN}✓${NC} Agent ${GREEN}$AGENT_NAME${NC} is using the ${GREEN}default${NC} prompt" + echo "" + echo " Location: $AGENT_FILE" + echo "" + echo -e "${BLUE}Note:${NC} In the new architecture, agent files are the canonical defaults." + echo " No action needed - you're already using the default!" + echo "" + exit 0 +fi + +# For non-default variants, copy from prompts directory +PROMPT_FILE="$PROMPTS_DIR/$AGENT_NAME/$PROMPT_VARIANT.md" + # Check prompt exists if [[ ! -f "$PROMPT_FILE" ]]; then - echo -e "${RED}Error: Prompt not found: $PROMPT_FILE${NC}" + echo -e "${RED}Error: Prompt variant not found: $PROMPT_FILE${NC}" echo "" - echo "Available prompts for $AGENT_NAME:" - ls -1 "$PROMPTS_DIR/$AGENT_NAME/"*.md 2>/dev/null | xargs -I {} basename {} .md || echo " (none found)" + echo "Available variants for $AGENT_NAME:" + echo " - default (canonical agent file)" + if [[ -d "$PROMPTS_DIR/$AGENT_NAME" ]]; then + ls -1 "$PROMPTS_DIR/$AGENT_NAME/"*.md 2>/dev/null | grep -v "TEMPLATE.md" | grep -v "README.md" | xargs -I {} basename {} .md || echo " (no model variants found)" + else + echo " (no model variants found)" + fi exit 1 fi @@ -178,5 +218,8 @@ if [[ -n "$MODEL_FAMILY" ]]; then echo "" fi +echo "To restore the default:" +echo " ./scripts/prompts/use-prompt.sh --agent=$AGENT_NAME --variant=default" +echo "" echo "To test this prompt:" echo " ./scripts/prompts/test-prompt.sh --agent=$AGENT_NAME --variant=$PROMPT_VARIANT" diff --git a/scripts/prompts/validate-pr.sh b/scripts/prompts/validate-pr.sh index 9f08bf1..f2a6ccd 100755 --- a/scripts/prompts/validate-pr.sh +++ b/scripts/prompts/validate-pr.sh @@ -1,6 +1,7 @@ #!/bin/bash -# Validate that all agents use their default prompts -# This ensures PRs maintain stable defaults in the main branch +# Validate prompt library structure +# Agent files in .opencode/agent/ are the canonical defaults +# Prompts in .opencode/prompts/ are model-specific variants set -e @@ -13,9 +14,10 @@ WARNINGS=0 RED='\033[0;31m' GREEN='\033[0;32m' YELLOW='\033[1;33m' +BLUE='\033[0;34m' NC='\033[0m' # No Color -echo "🔍 Validating agent prompts..." +echo "🔍 Validating prompt library structure..." echo "" # Check if prompts directory exists @@ -26,36 +28,57 @@ if [ ! -d "$PROMPTS_DIR" ]; then exit 0 fi +# Validate structure: agent files are canonical, prompts are variants +echo -e "${BLUE}Architecture:${NC}" +echo " • Agent files (.opencode/agent/*.md) = Canonical defaults" +echo " • Prompt variants (.opencode/prompts//.md) = Model-specific optimizations" +echo "" + # Find all agent markdown files (excluding subagents) for agent_file in "$AGENT_DIR"/*.md; do # Skip if no files found [ -e "$agent_file" ] || continue agent_name=$(basename "$agent_file" .md) - default_file="$PROMPTS_DIR/$agent_name/default.md" + prompts_subdir="$PROMPTS_DIR/$agent_name" - # Check if default exists - if [ ! -f "$default_file" ]; then - echo -e "${YELLOW}⚠️ No default found for $agent_name${NC}" - echo " Expected: $default_file" - echo " This agent will be skipped." + # Check if prompts directory exists for this agent + if [ ! -d "$prompts_subdir" ]; then + echo -e "${YELLOW}⚠️ No prompt variants for $agent_name${NC}" + echo " Agent file: $agent_file (canonical default)" + echo " Variants directory: $prompts_subdir (not found)" + echo " This is OK - variants are optional." echo "" WARNINGS=$((WARNINGS + 1)) continue fi - # Compare files - if ! diff -q "$agent_file" "$default_file" > /dev/null 2>&1; then - echo -e "${RED}❌ $agent_name.md does not match default${NC}" - echo " Current: $agent_file" - echo " Expected: $default_file" + # Check for default.md (should NOT exist in new architecture) + if [ -f "$prompts_subdir/default.md" ]; then + echo -e "${RED}❌ Found default.md for $agent_name${NC}" + echo " Location: $prompts_subdir/default.md" + echo " This file should not exist in the new architecture." echo "" - echo " To fix this:" - echo " ./scripts/prompts/use-prompt.sh $agent_name default" + echo " To fix:" + echo " rm $prompts_subdir/default.md" + echo "" + echo " The agent file is now the canonical default:" + echo " $agent_file" echo "" FAILED=$((FAILED + 1)) else - echo -e "${GREEN}✅ $agent_name.md matches default${NC}" + # Count variants + variant_count=$(find "$prompts_subdir" -maxdepth 1 -name "*.md" -not -name "README.md" -not -name "TEMPLATE.md" | wc -l | tr -d ' ') + + if [ "$variant_count" -gt 0 ]; then + echo -e "${GREEN}✅ $agent_name${NC}" + echo " Default: $agent_file" + echo " Variants: $variant_count model-specific optimization(s)" + else + echo -e "${GREEN}✅ $agent_name${NC}" + echo " Default: $agent_file" + echo " Variants: none (using default for all models)" + fi fi done @@ -63,25 +86,21 @@ echo "" # Summary if [ $FAILED -eq 0 ] && [ $WARNINGS -eq 0 ]; then - echo -e "${GREEN}✅ All agents use default prompts${NC}" + echo -e "${GREEN}✅ Prompt library structure is valid${NC}" exit 0 elif [ $FAILED -eq 0 ]; then echo -e "${YELLOW}⚠️ Validation passed with $WARNINGS warning(s)${NC}" - echo " Some agents don't have defaults yet - this is expected during development." + echo " Some agents don't have variant directories yet - this is expected." exit 0 else - echo -e "${RED}❌ PR validation failed - $FAILED agent(s) do not use default prompts${NC}" + echo -e "${RED}❌ PR validation failed - $FAILED issue(s) found${NC}" echo "" - echo "PRs must use default prompts to keep the main branch stable." + echo "The new prompt architecture:" + echo " • Agent files (.opencode/agent/*.md) are the canonical defaults" + echo " • Prompt variants (.opencode/prompts//.md) are model-specific" + echo " • default.md files should NOT exist" echo "" - echo "To fix:" - echo " 1. Restore defaults: ./scripts/prompts/use-prompt.sh default" - echo " 2. Or run: ./scripts/prompts/validate-pr.sh" - echo "" - echo "If you're adding a new prompt variant:" - echo " - Add it to .opencode/prompts//.md" - echo " - Do NOT modify .opencode/agent/.md" - echo " - See docs/contributing/CONTRIBUTING.md for details" + echo "See docs/contributing/CONTRIBUTING.md for details" echo "" exit 1 fi diff --git a/src/calculator.js b/src/calculator.js new file mode 100644 index 0000000..fe7beb0 --- /dev/null +++ b/src/calculator.js @@ -0,0 +1,34 @@ +/** + * Calculator Module + * Pure functions for basic arithmetic operations + */ + +/** + * Adds two numbers + * @param {number} a - First number + * @param {number} b - Second number + * @returns {number} Sum of a and b + */ +const add = (a, b) => a + b; + +/** + * Subtracts second number from first number + * @param {number} a - Number to subtract from + * @param {number} b - Number to subtract + * @returns {number} Difference of a and b + */ +const subtract = (a, b) => a - b; + +/** + * Multiplies two numbers + * @param {number} a - First number + * @param {number} b - Second number + * @returns {number} Product of a and b + */ +const multiply = (a, b) => a * b; + +module.exports = { + add, + subtract, + multiply, +}; diff --git a/src/models/User.js b/src/models/User.js new file mode 100644 index 0000000..bb73712 --- /dev/null +++ b/src/models/User.js @@ -0,0 +1,39 @@ +/** + * User model with name and email properties + */ +class User { + /** + * Creates a new User instance + * @param {string} name - User's name + * @param {string} email - User's email address + */ + constructor(name, email) { + this.name = name; + this.email = email; + } + + /** + * Returns a plain object representation of the user + * @returns {Object} User data as plain object + */ + toObject() { + return { + name: this.name, + email: this.email + }; + } + + /** + * Creates a new User with updated properties (immutable update) + * @param {Object} updates - Properties to update + * @returns {User} New User instance with updates applied + */ + update(updates) { + return new User( + updates.name ?? this.name, + updates.email ?? this.email + ); + } +} + +module.exports = User;