refactor(evals): consolidate documentation and enhance test infrastructure (#56 )

* feat(agents): implement category-based agent organization system

- Organize agents into domain categories (core, development, content, data, product, learning)
- Move 10 agents to category subdirectories with proper git rename tracking
- Update 13 subagents with category and type metadata in frontmatter
- Add category metadata files (0-category.json) documenting common patterns
- Implement local registry fallback in install script for offline development
- Add comprehensive validation suite with 15 automated tests (100% pass rate)
- Enhance registry validation with duplicate ID and consistency checks
- Update eval framework with intelligent path resolution (backward compatible)
- Archive legacy eval structure to _archive/ for reference
- Update all documentation to reflect category-based structure
- Bump version to 0.5.0 with accurate CHANGELOG

Technical Details:
- 23 agents organized (10 category agents, 13 subagents)
- 6 category directories created
- Path resolution supports both agent IDs and category paths
- Registry schema updated to v2.0.0
- 159 files changed, 872 insertions(+), 10738 deletions(-)

BREAKING CHANGE: Agent file paths now use category structure. Update references from .opencode/agent/openagent.md to .opencode/agent/core/openagent.md. Eval framework maintains backward compatibility via path resolution.

* refactor(evals): consolidate documentation and enhance test infrastructure

- Remove temporary project tracking files (PHASE_5_COMPLETE, PROJECT_COMPLETE, etc.)
- Consolidate evaluation framework docs into main README
- Enhance test execution with improved logging and multi-prompt support
- Move system-builder from core to meta category
- Add comprehensive test suites for openagent with organized structure
- Create evaluation test structure for all subagents
- Clean up archived workflows and redundant documentation
- Update registry to reflect new agent organization
- Add shared test templates and golden test patterns

* feat(ci): add manual trigger support for bot-created PRs in validate-registry workflow

- Add pr_number input to workflow_dispatch for manually triggering validation
- Fetch PR details dynamically when triggered manually
- Support both automatic (PR event) and manual (workflow_dispatch) triggers
- Enable validation of bot-created PRs like automated version bumps
- Update branch detection and push logic to handle both trigger types
- Add documentation explaining how to manually trigger for bot PRs

* feat(evals): add explicit context file validation to test framework

- Add expectedContextFiles field to test YAML schema for explicit context file specification
- Enhance context-loading-evaluator to support both auto-detect and explicit validation modes
- Update documentation with comprehensive guide and examples
- Clean up archived legacy test structure (90+ old test files)
- Add new example tests demonstrating explicit context validation
- Backward compatible with existing tests

* feat(evals): add multi-agent logging system and performance optimizations

Implement comprehensive multi-agent logging and performance improvements for the eval framework.

Task 01: Multi-Agent Logging System
- Add complete hierarchical logging module (evals/framework/src/logging/)
  - SessionTracker: tracks parent-child delegation hierarchies
  - MultiAgentLogger: pretty-prints logs with visual indentation
  - Formatters: box characters and emoji formatting
  - 37 passing unit tests (session-tracker, logger, integration)
- Integrate with SDK event stream handler
  - Hook into session.created, message.updated, message.part.updated events
  - Real-time child session detection via timestamp heuristics
  - Message deduplication for cleaner output
- Enable in debug mode only (<1% performance overhead)
- Add demo script and comprehensive documentation

Task 02: Performance Optimizations
- Reduce grace period from 5s to 2s (67% reduction, 10-20% faster tests)
- Add PerformanceMetricsEvaluator for bottleneck identification
  - Collects tool latencies, inference time, idle time
  - Provides full performance visibility
- Update smoke test to validate delegation (core feature)
  - Replace simple read test with multi-agent delegation test
  - Tests both parent and child agent functionality
  - Validates multi-agent logging system

Results:
- 37 unit tests passing
- Smoke test passing (95/100 score)
- Clean hierarchical logging output
- 10-20% faster test execution
- Full multi-agent visibility

Files changed: 16 files, +2678 lines
Time saved: ~6-8 days (completed in 6 hours vs 3-5 day estimate)

* feat(evals): show child agent execution in non-debug mode

- Enable MultiAgentLogger in both debug and non-debug modes
- Add verbose parameter to control output level
- Non-verbose mode shows concise child session lifecycle:
  - Child agent started message
  - Child agent completed with duration
- Verbose mode (--debug) shows full delegation hierarchy
- Update documentation to reflect new behavior

This provides visibility into delegation without overwhelming output,
giving confidence that child agents are actually running.

* feat(evals): improve delegation testing and behavior validation

Major improvements to eval framework:

Test Suite:
- Add 00-smoke-test.yaml (basic read operation)
- Rename 01-smoke-test.yaml → 02-delegation-test.yaml (delegation test)
- Add simple-responder test agent for delegation testing
- Add debug test: simple-subagent-call.yaml

Evaluators:
- Add agent-model-evaluator.ts for agent/model validation
- Enhance behavior-evaluator.ts with detailed task tool output
- Improve delegation-evaluator.ts with better evidence tracking

Schema & Execution:
- Add expectedAgent and expectedModel to test schema
- Improve test executor with better logging and error handling

Documentation:
- Update evals/README.md with new features and performance improvements
- Remove outdated MULTI_AGENT_LOGGING_COMPLETE.md

Registry:
- Add simple-responder test agent to registry

Fixes:
- Fix dashboard.sh path resolution

These changes provide better visibility into delegation, improved test
coverage, and clearer validation of agent behavior.

* chore: add GitHub Actions workflow and plugin documentation

- Add .github/workflows/evals/run-evaluations.yml for automated eval testing
- Add dev/ai-tools/opencode/plugins/Plugin-inspiration.md for plugin development reference

2025-12-18 09:52:11 +00:00

7.9 KiB

Raw Blame History

Core Concept: Agents

Purpose: Understanding how agents work in OpenAgents
Priority: CRITICAL - Load this before working with agents

What Are Agents?

Agents are AI prompt files that define specialized behaviors for different tasks. They are:

Markdown files with frontmatter metadata
Category-organized by domain (core, development, content, etc.)
Context-aware - load relevant context files
Testable - validated through eval framework

Agent Structure

File Format

---
description: "Brief description of what this agent does"
category: "category-name"
type: "agent"
tags: ["tag1", "tag2"]
dependencies: ["subagent:tester"]
---

# Agent Name

[Agent prompt content - instructions, workflows, constraints]

Key Components

Frontmatter (YAML metadata)
- description: Brief description
- category: Category name (core, development, content, etc.)
- type: Always "agent"
- tags: Optional tags for discovery
- dependencies: Optional dependencies (e.g., subagents)
Prompt Content
- Instructions and workflows
- Constraints and rules
- Context loading requirements
- Tool usage patterns

Category System

Agents are organized by domain expertise:

Core Category (`core/`)

Purpose: Essential system agents (always available)

Agents:

openagent.md - General-purpose orchestrator
opencoder.md - Development specialist
system-builder.md - System generation

When to use: System-level tasks, orchestration

Development Category (`development/`)

Purpose: Software development specialists

Agents:

frontend-specialist.md - React, Vue, modern CSS
backend-specialist.md - APIs, databases, servers
devops-specialist.md - CI/CD, deployment, infrastructure
codebase-agent.md - Codebase exploration and analysis

When to use: Building applications, dev tasks

Content Category (`content/`)

Purpose: Content creation specialists

Agents:

copywriter.md - Marketing copy, persuasive writing
technical-writer.md - Documentation, technical content

When to use: Writing, documentation, marketing

Data Category (`data/`)

Purpose: Data analysis specialists

Agents:

data-analyst.md - Data analysis, visualization

When to use: Data tasks, analysis, reporting

Product Category (`product/`)

Purpose: Product management specialists

Status: Ready for agents (no agents yet)

When to use: Product strategy, roadmaps, requirements

Learning Category (`learning/`)

Purpose: Education and coaching specialists

Status: Ready for agents (no agents yet)

When to use: Teaching, training, curriculum

Subagents

Location: .opencode/agent/subagents/

Purpose: Delegated specialists for specific subtasks

Subagent Categories

code/ - Code-related specialists
- tester.md - Test authoring and TDD
- reviewer.md - Code review and security
- coder-agent.md - Focused implementations
- build-agent.md - Type checking and builds
- codebase-pattern-analyst.md - Pattern analysis
core/ - Core workflow specialists
- task-manager.md - Task breakdown and management
- documentation.md - Documentation generation
system-builder/ - System generation specialists
- agent-generator.md - Generate agent files
- command-creator.md - Create slash commands
- domain-analyzer.md - Analyze domains
- context-organizer.md - Organize context
- workflow-designer.md - Design workflows
utils/ - Utility specialists
- image-specialist.md - Image editing and analysis

Subagents vs Category Agents

Aspect	Category Agents	Subagents
Purpose	User-facing specialists	Delegated subtasks
Invocation	Direct by user	Via task tool
Scope	Broad domain	Narrow focus
Example	`frontend-specialist`	`tester`

Path Resolution

The system supports multiple path formats for backward compatibility:

Supported Formats

# Short ID (backward compatible)
"openagent" → resolves to → ".opencode/agent/core/openagent.md"

# Category path
"core/openagent" → resolves to → ".opencode/agent/core/openagent.md"

# Full category path
"development/frontend-specialist" → resolves to → ".opencode/agent/development/frontend-specialist.md"

# Subagent path
"subagents/code/tester" → resolves to → ".opencode/agent/subagents/code/tester.md"

Resolution Rules

Check if path includes / → use as category path
If no / → check core/ first (backward compat)
If not in core/ → search all categories
If not found → error

Prompt Variants

Location: .opencode/prompts/{category}/{agent}/

Purpose: Model-specific prompt optimizations

Supported Models

gemini.md - Google Gemini optimizations
grok.md - xAI Grok optimizations
llama.md - Meta Llama optimizations
openrouter.md - OpenRouter optimizations

When to Create Variants

Model has specific formatting requirements
Model performs better with different structure
Model has unique capabilities to leverage

Fallback Behavior

If no variant exists for a model, the base agent file is used.

Context Loading

Agents should load relevant context files based on task type:

Core Context (Always Consider)

<!-- Context: standards/code | Priority: critical -->

Loads: .opencode/context/core/standards/code.md

Category Context

<!-- Context: development/react-patterns | Priority: high -->

Loads: .opencode/context/development/react-patterns.md

Multiple Contexts

<!-- Context: standards/code, standards/tests | Priority: critical -->

Agent Lifecycle

1. Creation

# Create agent file
touch .opencode/agent/{category}/{agent-name}.md

# Add frontmatter and content
# (See guides/adding-agent.md for details)

2. Testing

# Create test structure
mkdir -p evals/agents/{category}/{agent-name}/{config,tests}

# Run tests
cd evals/framework && npm run eval:sdk -- --agent={category}/{agent-name}

3. Registration

# Auto-detect and add to registry
./scripts/registry/auto-detect-components.sh --auto-add

# Validate
./scripts/registry/validate-registry.sh

4. Distribution

# Users install via install.sh
./install.sh {profile}

Best Practices

Agent Design

✅ Single responsibility - One domain, one agent
✅ Clear instructions - Explicit workflows and constraints
✅ Context-aware - Load relevant context files
✅ Testable - Include eval tests
✅ Well-documented - Clear description and usage

Naming Conventions

Category agents: {domain}-specialist.md (e.g., frontend-specialist.md)
Core agents: {name}.md (e.g., openagent.md)
Subagents: {purpose}.md (e.g., tester.md)

Frontmatter Requirements

---
description: "Required - brief description"
category: "Required - category name"
type: "Required - always 'agent'"
tags: ["Optional - for discovery"]
dependencies: ["Optional - e.g., 'subagent:tester'"]
---

Common Patterns

Delegation to Subagents

When task requires testing:
1. Implement feature
2. Delegate to subagents/code/tester for test creation

Context Loading

Before implementing:
1. Load core/standards/code.md
2. Load category-specific context if available
3. Apply standards to implementation

Approval Gates

Before execution:
1. Present plan to user
2. Request approval
3. Execute incrementally

Adding agents: guides/adding-agent.md
Testing agents: guides/testing-agent.md
Category system: core-concepts/categories.md
File locations: lookup/file-locations.md

Last Updated: 2025-12-10
Version: 0.5.0

7.9 KiB Raw Blame History

Core Concept: Agents

What Are Agents?

Agent Structure

File Format

Key Components

Category System

Core Category (core/)

Development Category (development/)

Content Category (content/)

Data Category (data/)

Product Category (product/)

Learning Category (learning/)

Subagents

Subagent Categories

Subagents vs Category Agents

Path Resolution

Supported Formats

Resolution Rules

Prompt Variants

Supported Models

When to Create Variants

Fallback Behavior

Context Loading

Core Context (Always Consider)

Category Context

Multiple Contexts

Agent Lifecycle

1. Creation

2. Testing

3. Registration

4. Distribution

Best Practices

Agent Design

Naming Conventions

Frontmatter Requirements

Common Patterns

Delegation to Subagents

Context Loading

Approval Gates

Related Files

7.9 KiB

Raw Blame History

Core Category (`core/`)

Development Category (`development/`)

Content Category (`content/`)

Data Category (`data/`)

Product Category (`product/`)

Learning Category (`learning/`)