784ffadf92
* chore: update package-lock.json after npm install * chore: remove unused ajv-cli, keep vitest at stable 1.6.1 - Removed ajv-cli (unused dev dependency, had high severity vulnerability) - Kept vitest at 1.6.1 (stable, upgrading to v4 requires extensive refactoring) - Reduced vulnerabilities from 6 to 4 (all moderate, dev-only) - All remaining vulnerabilities are in esbuild/vite (dev server only, no production impact) - Build and tests verified working * feat(evals): add execution-balance validation to 21 OpenAgent tests and enhance agent system Major improvements to evaluation framework and agent capabilities: ## Execution Balance Evaluator Integration - Added execution-balance rule to 21 OpenAgent tests (2.7% → 31.5% coverage) - Validates "read before execute" pattern across critical rules, workflows, and edge cases - Tests now check: read operations before execution tools, healthy read/exec ratio (≥1.0) - Coverage includes: approval-gate, context-loading, stop-on-failure, report-first tests ## Agent System Enhancements - Enhanced OpenCoder agent with critical rules enforcement (approval gates, context loading) - Added structured delegation rules and workflow phases - Improved error handling with REPORT→PROPOSE→REQUEST→FIX pattern ## New Agent Creation System - Added research-backed agent creation templates and guides - Includes 8 test templates covering planning, context, incremental execution, tools - Automated agent scaffolding with /create-agent command - Complete documentation in .opencode/command/openagents/new-agents/ ## OpenCoder Test Suite - Added 8 comprehensive test cases for OpenCoder agent - Tests cover: planning, context-loading, delegation, error-handling, implementation - Includes DEBUG_GUIDE.md and QUICK_TEST_GUIDE.md for developers ## Documentation & Tooling - Added DEVELOPMENT.md guide for contributors - Enhanced CONTRIBUTING.md with quick reference and common commands - Improved event-logger with DEBUG_VERBOSE mode for detailed output - Added debug scripts: show-test-conversation.sh, run-test-verbose.sh - Removed deprecated EXTERNAL_PR_GUIDE.md ## Version Bump - Bumped version to 0.0.3 (patch) Files changed: 60+ files (21 test updates, 16 new agent templates, 8 new tests, docs, tooling) * feat(ci): add PR title validation for semantic versioning Add automated PR title validation to ensure conventional commit format. This enables proper automatic version bumping when PRs are merged. Features: - Validates PR titles against conventional commit patterns - Shows expected version bump (major/minor/patch) - Posts helpful comment with examples if validation fails - Supports all conventional commit types (feat, fix, docs, etc.) - Supports breaking changes (feat!, fix!) - Supports pre-release tags ([alpha], [beta], [rc]) When PR titles follow the format, version bumping works correctly: - feat: → minor bump (0.3.0 → 0.4.0) - fix: → patch bump (0.3.0 → 0.3.1) - feat!: → major bump (0.3.0 → 1.0.0) * fix(ci): improve PR checks workflow with sequential execution Reorganize PR checks to run in logical sequence and skip unnecessary checks: Changes: - Run PR title validation first (fast, always required) - Detect changed files to determine which checks to run - Only run build checks if evals/ files changed - Add comprehensive summary job showing all check results - Prevent cache errors when evals files not changed Benefits: - Faster PR checks (skip unnecessary builds) - Clear sequential execution (title → changes → build) - Better error messages and summaries - Reduced CI/CD costs * fix(ci): resolve PR #42 issues and refactor prompt architecture ## Fixed PR #42 Issues 1. **Prompt Validation** - Refactored to new architecture where agent files are canonical defaults 2. **Package-lock.json** - Fixed workflow to use root package-lock (npm workspaces) 3. **Git Merge Base** - Added fetch-depth: 0 to PR checks workflow ## New Prompt Architecture **Before:** - Agent files had to match .opencode/prompts/<agent>/default.md - Caused validation failures when updating prompts - Redundant duplication **After:** - Agent files (.opencode/agent/*.md) = Canonical defaults (source of truth) - Prompt variants (.opencode/prompts/<agent>/<model>.md) = Model-specific optimizations - No more default.md files needed ## Changes ### Workflows - pr-checks.yml: Added fetch-depth: 0 for full git history - validate-test-suites.yml: Fixed package-lock path for npm workspaces - validate-registry.yml: Updated validation messaging ### Scripts - validate-pr.sh: Validates prompt library structure (rejects default.md files) - use-prompt.sh: Handles 'default' as agent file, variants as model-specific - test-prompt.sh: Tests default variant correctly, saves results to prompts/results/ ### Documentation - .opencode/prompts/README.md: Updated architecture explanation - docs/contributing/CONTRIBUTING.md: Updated prompt workflow - scripts/development/demo.sh: Updated structure display ## Results - 12 files changed, 231 insertions(+), 811 deletions(-) - Net reduction: 580 lines (cleaner codebase) - All validations passing - Results directory structure preserved - Backwards compatible with existing test results
213 lines
6 KiB
Markdown
213 lines
6 KiB
Markdown
# Quick Test Guide - OpenCoder
|
|
|
|
**TL;DR:** Run tests and see EXACTLY what the agent says and does.
|
|
|
|
---
|
|
|
|
## 🚀 Run Test with Full Conversation
|
|
|
|
### Method 1: Using --verbose flag (RECOMMENDED)
|
|
|
|
```bash
|
|
cd evals/framework
|
|
|
|
# Run single test and see full conversation
|
|
npm run eval:sdk -- --agent=opencoder --pattern="planning/*.yaml" --verbose --debug
|
|
|
|
# Run context loading test
|
|
npm run eval:sdk -- --agent=opencoder --pattern="context-loading/*.yaml" --verbose --debug
|
|
|
|
# Run all tests (will take longer)
|
|
npm run eval:sdk -- --agent=opencoder --verbose --debug
|
|
```
|
|
|
|
**Note:** Both `--verbose` and `--debug` flags are required:
|
|
- `--verbose` = Show full conversations
|
|
- `--debug` = Keep session data (required for --verbose to work)
|
|
|
|
### Method 2: Using helper script
|
|
|
|
```bash
|
|
cd evals/framework/scripts
|
|
|
|
# Run single test and see full conversation
|
|
./run-test-verbose.sh opencoder "planning/*.yaml"
|
|
|
|
# Run context loading test
|
|
./run-test-verbose.sh opencoder "context-loading/*.yaml"
|
|
```
|
|
|
|
**Output shows:**
|
|
1. ✅ Test result (PASS/FAIL)
|
|
2. 📊 Test metrics (duration, events, violations)
|
|
3. 💬 **FULL CONVERSATION** - Every message between user and agent
|
|
|
|
---
|
|
|
|
## 📋 Example Output
|
|
|
|
```
|
|
TEST RESULTS
|
|
======================================================================
|
|
|
|
1. ✅ planning-approval-workflow - Planning & Approval Workflow
|
|
Duration: 28327ms
|
|
Events: 33
|
|
Approvals: 0
|
|
Violations: 0 (0 errors, 0 warnings)
|
|
|
|
======================================================================
|
|
FULL CONVERSATION
|
|
======================================================================
|
|
|
|
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
👤 USER
|
|
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
## Implementation Plan
|
|
|
|
Based on the existing code structure, I will:
|
|
|
|
1. Create a new file `utils/utils.js` with an `add` function
|
|
2. Follow pure function pattern
|
|
3. Use JSDoc comments
|
|
|
|
**Approval needed before proceeding. Please review and confirm.**
|
|
|
|
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
🤖 ASSISTANT
|
|
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
```
|
|
|
|
---
|
|
|
|
## ✅ What You Can Verify
|
|
|
|
### 1. Planning & Approval Workflow
|
|
```bash
|
|
./run-test-verbose.sh opencoder "planning/*.yaml"
|
|
```
|
|
|
|
**Look for:**
|
|
- ✅ "DIGGING IN..." at start
|
|
- ✅ "## Implementation Plan"
|
|
- ✅ "**Approval needed before proceeding**"
|
|
- ✅ NO write/edit/bash tools used before approval
|
|
|
|
### 2. Context Loading
|
|
```bash
|
|
./run-test-verbose.sh opencoder "context-loading/*.yaml"
|
|
```
|
|
|
|
**Look for:**
|
|
- ✅ Test output shows: "✓ Loaded: .opencode/context/core/standards/code.md"
|
|
- ✅ Test output shows: "✓ Timing: Context loaded XXXXms before execution"
|
|
- ✅ Tool calls show `read` of code.md BEFORE `write`
|
|
|
|
### 3. Delegation Recognition
|
|
```bash
|
|
./run-test-verbose.sh opencoder "delegation/*.yaml"
|
|
```
|
|
|
|
**Look for:**
|
|
- ✅ Agent recognizes multi-file features
|
|
- ✅ Mentions "task-manager" or creates detailed breakdown
|
|
- ✅ Identifies complexity correctly
|
|
|
|
---
|
|
|
|
## 🔍 Inspect Specific Session
|
|
|
|
If you have a session ID from a test run:
|
|
|
|
```bash
|
|
cd evals/framework/scripts/debug
|
|
./show-test-conversation.sh ses_XXXXXXXXXXXXX
|
|
```
|
|
|
|
---
|
|
|
|
## 📊 All Available Tests
|
|
|
|
```bash
|
|
# List all opencoder tests
|
|
find ../agents/opencoder/tests -name "*.yaml" -type f
|
|
|
|
# Run all tests (no conversation output)
|
|
cd ../
|
|
npm run eval:sdk -- --agent=opencoder
|
|
|
|
# Run all tests with debug
|
|
npm run eval:sdk -- --agent=opencoder --debug
|
|
```
|
|
|
|
---
|
|
|
|
## 🎯 Test Categories
|
|
|
|
| Category | Pattern | What It Tests |
|
|
|----------|---------|---------------|
|
|
| **Planning** | `planning/*.yaml` | Plan-first, approval gates |
|
|
| **Context** | `context-loading/*.yaml` | Loads code.md before coding |
|
|
| **Implementation** | `implementation/*.yaml` | Incremental execution, validation |
|
|
| **Delegation** | `delegation/*.yaml` | Task-manager, coder-agent routing |
|
|
| **Error Handling** | `error-handling/*.yaml` | Stop on failure, report-first |
|
|
| **Completion** | `completion/*.yaml` | Handoff recommendations |
|
|
|
|
---
|
|
|
|
## 💡 Quick Validation Checklist
|
|
|
|
Run these 3 tests to validate core behaviors:
|
|
|
|
```bash
|
|
# 1. Approval workflow
|
|
./run-test-verbose.sh opencoder "planning/*.yaml"
|
|
# ✅ Look for: "Approval needed before proceeding"
|
|
|
|
# 2. Context loading
|
|
./run-test-verbose.sh opencoder "context-loading/*.yaml"
|
|
# ✅ Look for: "✓ Loaded: .opencode/context/core/standards/code.md"
|
|
|
|
# 3. Delegation
|
|
./run-test-verbose.sh opencoder "delegation/delegation-task-manager.yaml"
|
|
# ✅ Look for: Multi-step plan or "task-manager" mention
|
|
```
|
|
|
|
---
|
|
|
|
## 🐛 Troubleshooting
|
|
|
|
### "Session not found"
|
|
- Session was cleaned up. Run test again.
|
|
|
|
### "No conversation shown"
|
|
- Check if test actually ran (look for "Session created" in output)
|
|
- Try running test directly: `npm run eval:sdk -- --agent=opencoder --pattern="planning/*.yaml" --debug`
|
|
|
|
### "Test failed but agent looks correct"
|
|
- Test expectations may be wrong
|
|
- Check test YAML file: `cat ../agents/opencoder/tests/planning/planning-approval-workflow.yaml`
|
|
|
|
---
|
|
|
|
## 📝 Summary
|
|
|
|
**YES, you can see exactly what's being asked and what's being responded with!**
|
|
|
|
The `run-test-verbose.sh` script:
|
|
1. Runs the test
|
|
2. Captures the session ID
|
|
3. Shows test results
|
|
4. **Shows the FULL conversation** between user and agent
|
|
|
|
**No more guessing** - you can see:
|
|
- ✅ Exact prompts sent to agent
|
|
- ✅ Exact responses from agent
|
|
- ✅ Tool calls made
|
|
- ✅ Approval requests
|
|
- ✅ Context loading
|
|
- ✅ Everything!
|
|
|
|
---
|
|
|
|
**Last Updated:** 2025-12-08
|