refactor(evals): consolidate documentation and enhance test infrastructure (#56 )

* feat(agents): implement category-based agent organization system

- Organize agents into domain categories (core, development, content, data, product, learning)
- Move 10 agents to category subdirectories with proper git rename tracking
- Update 13 subagents with category and type metadata in frontmatter
- Add category metadata files (0-category.json) documenting common patterns
- Implement local registry fallback in install script for offline development
- Add comprehensive validation suite with 15 automated tests (100% pass rate)
- Enhance registry validation with duplicate ID and consistency checks
- Update eval framework with intelligent path resolution (backward compatible)
- Archive legacy eval structure to _archive/ for reference
- Update all documentation to reflect category-based structure
- Bump version to 0.5.0 with accurate CHANGELOG

Technical Details:
- 23 agents organized (10 category agents, 13 subagents)
- 6 category directories created
- Path resolution supports both agent IDs and category paths
- Registry schema updated to v2.0.0
- 159 files changed, 872 insertions(+), 10738 deletions(-)

BREAKING CHANGE: Agent file paths now use category structure. Update references from .opencode/agent/openagent.md to .opencode/agent/core/openagent.md. Eval framework maintains backward compatibility via path resolution.

* refactor(evals): consolidate documentation and enhance test infrastructure

- Remove temporary project tracking files (PHASE_5_COMPLETE, PROJECT_COMPLETE, etc.)
- Consolidate evaluation framework docs into main README
- Enhance test execution with improved logging and multi-prompt support
- Move system-builder from core to meta category
- Add comprehensive test suites for openagent with organized structure
- Create evaluation test structure for all subagents
- Clean up archived workflows and redundant documentation
- Update registry to reflect new agent organization
- Add shared test templates and golden test patterns

* feat(ci): add manual trigger support for bot-created PRs in validate-registry workflow

- Add pr_number input to workflow_dispatch for manually triggering validation
- Fetch PR details dynamically when triggered manually
- Support both automatic (PR event) and manual (workflow_dispatch) triggers
- Enable validation of bot-created PRs like automated version bumps
- Update branch detection and push logic to handle both trigger types
- Add documentation explaining how to manually trigger for bot PRs

* feat(evals): add explicit context file validation to test framework

- Add expectedContextFiles field to test YAML schema for explicit context file specification
- Enhance context-loading-evaluator to support both auto-detect and explicit validation modes
- Update documentation with comprehensive guide and examples
- Clean up archived legacy test structure (90+ old test files)
- Add new example tests demonstrating explicit context validation
- Backward compatible with existing tests

* feat(evals): add multi-agent logging system and performance optimizations

Implement comprehensive multi-agent logging and performance improvements for the eval framework.

Task 01: Multi-Agent Logging System
- Add complete hierarchical logging module (evals/framework/src/logging/)
  - SessionTracker: tracks parent-child delegation hierarchies
  - MultiAgentLogger: pretty-prints logs with visual indentation
  - Formatters: box characters and emoji formatting
  - 37 passing unit tests (session-tracker, logger, integration)
- Integrate with SDK event stream handler
  - Hook into session.created, message.updated, message.part.updated events
  - Real-time child session detection via timestamp heuristics
  - Message deduplication for cleaner output
- Enable in debug mode only (<1% performance overhead)
- Add demo script and comprehensive documentation

Task 02: Performance Optimizations
- Reduce grace period from 5s to 2s (67% reduction, 10-20% faster tests)
- Add PerformanceMetricsEvaluator for bottleneck identification
  - Collects tool latencies, inference time, idle time
  - Provides full performance visibility
- Update smoke test to validate delegation (core feature)
  - Replace simple read test with multi-agent delegation test
  - Tests both parent and child agent functionality
  - Validates multi-agent logging system

Results:
- 37 unit tests passing
- Smoke test passing (95/100 score)
- Clean hierarchical logging output
- 10-20% faster test execution
- Full multi-agent visibility

Files changed: 16 files, +2678 lines
Time saved: ~6-8 days (completed in 6 hours vs 3-5 day estimate)

* feat(evals): show child agent execution in non-debug mode

- Enable MultiAgentLogger in both debug and non-debug modes
- Add verbose parameter to control output level
- Non-verbose mode shows concise child session lifecycle:
  - Child agent started message
  - Child agent completed with duration
- Verbose mode (--debug) shows full delegation hierarchy
- Update documentation to reflect new behavior

This provides visibility into delegation without overwhelming output,
giving confidence that child agents are actually running.

* feat(evals): improve delegation testing and behavior validation

Major improvements to eval framework:

Test Suite:
- Add 00-smoke-test.yaml (basic read operation)
- Rename 01-smoke-test.yaml → 02-delegation-test.yaml (delegation test)
- Add simple-responder test agent for delegation testing
- Add debug test: simple-subagent-call.yaml

Evaluators:
- Add agent-model-evaluator.ts for agent/model validation
- Enhance behavior-evaluator.ts with detailed task tool output
- Improve delegation-evaluator.ts with better evidence tracking

Schema & Execution:
- Add expectedAgent and expectedModel to test schema
- Improve test executor with better logging and error handling

Documentation:
- Update evals/README.md with new features and performance improvements
- Remove outdated MULTI_AGENT_LOGGING_COMPLETE.md

Registry:
- Add simple-responder test agent to registry

Fixes:
- Fix dashboard.sh path resolution

These changes provide better visibility into delegation, improved test
coverage, and clearer validation of agent behavior.

* chore: add GitHub Actions workflow and plugin documentation

- Add .github/workflows/evals/run-evaluations.yml for automated eval testing
- Add dev/ai-tools/opencode/plugins/Plugin-inspiration.md for plugin development reference

2025-12-18 09:52:11 +00:00

10 KiB

Raw Blame History

Category-Based Agent System

Version: 2.0.0
Status: Active
Last Updated: 2025-12-09

🎯 Overview

The category-based agent system organizes agents into logical categories, making it easier to discover, contribute, and maintain domain-specific agents.

Key Concepts

Three-Tier Hierarchy:

Core Agents - Primary agents users interact with (orchestrator, coder)
Category Agents - Domain-specific specialists (frontend, copywriter, etc.)
Subagents - Delegated specialists called by core/category agents (tester, reviewer, etc.)

📁 Directory Structure

.opencode/agent/
├── core/                    # Core Agents (System-level)
│   ├── openagent.md         # Universal coordinator
│   └── opencoder.md         # Development specialist
│
├── meta/                    # Meta-level Agents
│   └── system-builder.md    # System architect
│
├── development/             # Development Specialists
│   ├── frontend-specialist.md
│   ├── backend-specialist.md
│   ├── devops-specialist.md
│   └── codebase-agent.md
│
├── content/                 # Content Creation
│   ├── copywriter.md
│   └── technical-writer.md
│
├── data/                    # Data & Analysis
│   └── data-analyst.md
│
├── product/                 # Product & Strategy
│   └── (empty - ready for contributions)
│
├── learning/                # Education & Coaching
│   └── (empty - ready for contributions)
│
└── subagents/              # Delegated Specialists
    ├── code/               # Code-related
    ├── core/               # Core workflow
    ├── system-builder/     # System generation
    └── utils/              # Utilities

🔧 Using Category Agents

Installation

# Install all development agents
./install.sh developer

# Install specific profile
./install.sh full

Running Tests

cd evals/framework

# Test core agent
npm run eval:sdk -- --agent=core/openagent

# Test category agent
npm run eval:sdk -- --agent=development/frontend-specialist

# Test subagent
npm run eval:sdk -- --agent=subagents/code/tester

Agent Invocation

Core Agents - Direct invocation:

opencode --agent core/openagent
opencode --agent core/opencoder

Category Agents - Direct invocation:

opencode --agent development/frontend-specialist
opencode --agent content/copywriter

Subagents - Via task delegation:

// From core or category agent
task(
  subagent_type="subagents/code/tester",
  description="Write tests",
  prompt="Create comprehensive tests for the authentication module"
)

📊 Categories

Core (`core/`)

Purpose: Essential system-level agents for daily development work

Agents:

core/openagent - Universal task coordinator
core/opencoder - Development specialist

When to use: Primary agents for general tasks and development

Meta (`meta/`)

Purpose: Meta-level agents for system generation and architecture design

Agents:

meta/system-builder - System architect and generator

When to use: Building custom AI systems and agent architectures

Development (`development/`)

Purpose: Software development specialists

Agents:

development/frontend-specialist - React, Vue, modern CSS
development/backend-specialist - APIs, databases, server-side
development/devops-specialist - CI/CD, infrastructure
development/codebase-agent - Multi-language implementation

When to use: Domain-specific development tasks

Content (`content/`)

Purpose: Writing and content creation

Agents:

content/copywriter - Marketing copy, brand messaging
content/technical-writer - Documentation, API docs

When to use: Content creation and documentation

Data (`data/`)

Purpose: Data analysis and research

Agents:

data/data-analyst - Data analysis, visualization, insights

When to use: Data-driven tasks

Product (`product/`)

Purpose: Product management and strategy

Status: Ready for contributions

Potential Agents:

Product manager
User researcher
Business analyst

Learning (`learning/`)

Purpose: Education and coaching

Status: Ready for contributions

Potential Agents:

Tutor
CBT coach
Curriculum designer

🆚 Core vs Category vs Subagents

Core Agents

Characteristics:

System-level, maintained by repository
General-purpose, work across domains
Primary entry points for users
Can delegate to category agents or subagents

Examples: core/openagent, core/opencoder

Category Agents

Characteristics:

Domain-specific specialists
User-facing (can be invoked directly)
Community-contributed
Can delegate to subagents

Examples: development/frontend-specialist, content/copywriter

Subagents

Characteristics:

Focused, single-purpose functionality
NOT directly invoked by users
Called via task delegation
Support core and category agents

Examples: subagents/code/tester, subagents/core/task-manager

🔄 Backward Compatibility

Old Format Still Works

The old flat structure is still supported:

# Old format (still works)
npm run eval:sdk -- --agent=openagent
npm run eval:sdk -- --agent=opencoder

# New format (recommended)
npm run eval:sdk -- --agent=core/openagent
npm run eval:sdk -- --agent=core/opencoder

Path Resolution

The eval framework automatically resolves old agent names:

openagent → core/openagent
opencoder → core/opencoder
system-builder → meta/system-builder

Deprecation Timeline

v2.0.0 (current): New structure, old paths work with warnings
v2.1.0: Old paths still work, louder warnings
v3.0.0: Remove old paths (breaking change)

🤝 Contributing Category Agents

See ADDING_CATEGORY_AGENT.md for detailed instructions.

Quick Start

Choose a category (development, content, data, product, learning)
Create agent file: .opencode/agent/{category}/{agent-name}.md
Add frontmatter with metadata
Create tests: evals/agents/{category}/{agent-name}/
Submit PR

Agent File Template

---
id: agent-name
name: Agent Name
description: Brief description
category: development
type: standard
version: 1.0.0
author: your-name

mode: primary
model: anthropic/claude-sonnet-4-5
temperature: 0.1

tools:
  read: true
  write: true
  edit: true
  bash: false
  glob: true
  grep: true
  task: true

dependencies:
  context:
    - development/patterns
  tools: []

tags:
  - development
  - specialist
---

# Agent Name

You are a specialist in...

## Your Role

[Define role and responsibilities]

## Workflow

1. **Analyze** - Understand the request
2. **Plan** - Create implementation plan
3. **Request Approval** - Present plan to user
4. **Implement** - Execute following patterns
5. **Validate** - Test and verify

Migration Guide - Migrating from v1.x to v2.0
Contributing Guide - General contribution guidelines
Adding Category Agents - Detailed agent creation guide
Eval Framework - Testing agents

🔍 Examples

Example 1: Using Frontend Specialist

# Install
./install.sh developer

# Test
cd evals/framework
npm run eval:sdk -- --agent=development/frontend-specialist

# Use
opencode --agent development/frontend-specialist

Example 2: Creating a New Category Agent

# Create agent file
vim .opencode/agent/product/product-manager.md

# Create test structure
mkdir -p evals/agents/product/product-manager/{config,tests}

# Create smoke test
vim evals/agents/product/product-manager/tests/smoke-test.yaml

# Validate
./scripts/registry/validate-agent-structure.sh

# Test
cd evals/framework
npm run eval:sdk -- --agent=product/product-manager

Example 3: Delegating to Subagent

From a category agent:

// Delegate test creation to tester subagent
task(
  subagent_type="subagents/code/tester",
  description="Create tests",
  prompt="Write comprehensive unit tests for the UserService class"
)

❓ FAQ

Q: Can I still use the old agent names?

A: Yes! Old names like openagent and opencoder still work and will continue to work until v3.0.0.

Q: How do I know which category to use?

A: Choose based on the agent's primary domain:

Code-related → development/
Writing-related → content/
Data-related → data/
Product-related → product/
Teaching-related → learning/

Q: Can an agent be in multiple categories?

A: No, each agent belongs to one category. If an agent spans multiple domains, choose the primary domain or create a more general agent in core/.

Q: What's the difference between a category agent and a subagent?

Category agents are user-facing and can be invoked directly
Subagents are internal and only called via task delegation

Q: How do I test my category agent?

A: Create a test suite in evals/agents/{category}/{agent-name}/ and run:

cd evals/framework
npm run eval:sdk -- --agent={category}/{agent-name}

🎓 Best Practices

Choose the right category - Place agents in their primary domain
Follow naming conventions - Use kebab-case for agent IDs
Write comprehensive tests - At least a smoke test
Document dependencies - List required context files
Keep agents focused - One domain, one responsibility
Use subagents - Delegate complex subtasks

📊 System Statistics

Categories: 6 (core, development, content, data, product, learning)
Core Agents: 3
Category Agents: 7
Subagents: 13
Total Agents: 23

Last Updated: 2025-12-09
Version: 2.0.0

10 KiB Raw Blame History

Category-Based Agent System

🎯 Overview

Key Concepts

📁 Directory Structure

🔧 Using Category Agents

Installation

Running Tests

Agent Invocation

📊 Categories

Core (core/)

Meta (meta/)

Development (development/)

Content (content/)

Data (data/)

Product (product/)

Learning (learning/)

🆚 Core vs Category vs Subagents

Core Agents

Category Agents

Subagents

🔄 Backward Compatibility

Old Format Still Works

Path Resolution

Deprecation Timeline

🤝 Contributing Category Agents

Quick Start

Agent File Template

📚 Related Documentation

🔍 Examples

Example 1: Using Frontend Specialist

Example 2: Creating a New Category Agent

Example 3: Delegating to Subagent

❓ FAQ

Q: Can I still use the old agent names?

Q: How do I know which category to use?

Q: Can an agent be in multiple categories?

Q: What's the difference between a category agent and a subagent?

Q: How do I test my category agent?

🎓 Best Practices

📊 System Statistics

10 KiB

Raw Blame History

Core (`core/`)

Meta (`meta/`)

Development (`development/`)

Content (`content/`)

Data (`data/`)

Product (`product/`)

Learning (`learning/`)