OpenWriterControle/docs/agents/category-system.md
Darren Hinde c8f7103cb6
refactor(evals): consolidate documentation and enhance test infrastructure (#56)
* feat(agents): implement category-based agent organization system

- Organize agents into domain categories (core, development, content, data, product, learning)
- Move 10 agents to category subdirectories with proper git rename tracking
- Update 13 subagents with category and type metadata in frontmatter
- Add category metadata files (0-category.json) documenting common patterns
- Implement local registry fallback in install script for offline development
- Add comprehensive validation suite with 15 automated tests (100% pass rate)
- Enhance registry validation with duplicate ID and consistency checks
- Update eval framework with intelligent path resolution (backward compatible)
- Archive legacy eval structure to _archive/ for reference
- Update all documentation to reflect category-based structure
- Bump version to 0.5.0 with accurate CHANGELOG

Technical Details:
- 23 agents organized (10 category agents, 13 subagents)
- 6 category directories created
- Path resolution supports both agent IDs and category paths
- Registry schema updated to v2.0.0
- 159 files changed, 872 insertions(+), 10738 deletions(-)

BREAKING CHANGE: Agent file paths now use category structure. Update references from .opencode/agent/openagent.md to .opencode/agent/core/openagent.md. Eval framework maintains backward compatibility via path resolution.

* refactor(evals): consolidate documentation and enhance test infrastructure

- Remove temporary project tracking files (PHASE_5_COMPLETE, PROJECT_COMPLETE, etc.)
- Consolidate evaluation framework docs into main README
- Enhance test execution with improved logging and multi-prompt support
- Move system-builder from core to meta category
- Add comprehensive test suites for openagent with organized structure
- Create evaluation test structure for all subagents
- Clean up archived workflows and redundant documentation
- Update registry to reflect new agent organization
- Add shared test templates and golden test patterns

* feat(ci): add manual trigger support for bot-created PRs in validate-registry workflow

- Add pr_number input to workflow_dispatch for manually triggering validation
- Fetch PR details dynamically when triggered manually
- Support both automatic (PR event) and manual (workflow_dispatch) triggers
- Enable validation of bot-created PRs like automated version bumps
- Update branch detection and push logic to handle both trigger types
- Add documentation explaining how to manually trigger for bot PRs

* feat(evals): add explicit context file validation to test framework

- Add expectedContextFiles field to test YAML schema for explicit context file specification
- Enhance context-loading-evaluator to support both auto-detect and explicit validation modes
- Update documentation with comprehensive guide and examples
- Clean up archived legacy test structure (90+ old test files)
- Add new example tests demonstrating explicit context validation
- Backward compatible with existing tests

* feat(evals): add multi-agent logging system and performance optimizations

Implement comprehensive multi-agent logging and performance improvements for the eval framework.

Task 01: Multi-Agent Logging System
- Add complete hierarchical logging module (evals/framework/src/logging/)
  - SessionTracker: tracks parent-child delegation hierarchies
  - MultiAgentLogger: pretty-prints logs with visual indentation
  - Formatters: box characters and emoji formatting
  - 37 passing unit tests (session-tracker, logger, integration)
- Integrate with SDK event stream handler
  - Hook into session.created, message.updated, message.part.updated events
  - Real-time child session detection via timestamp heuristics
  - Message deduplication for cleaner output
- Enable in debug mode only (<1% performance overhead)
- Add demo script and comprehensive documentation

Task 02: Performance Optimizations
- Reduce grace period from 5s to 2s (67% reduction, 10-20% faster tests)
- Add PerformanceMetricsEvaluator for bottleneck identification
  - Collects tool latencies, inference time, idle time
  - Provides full performance visibility
- Update smoke test to validate delegation (core feature)
  - Replace simple read test with multi-agent delegation test
  - Tests both parent and child agent functionality
  - Validates multi-agent logging system

Results:
- 37 unit tests passing
- Smoke test passing (95/100 score)
- Clean hierarchical logging output
- 10-20% faster test execution
- Full multi-agent visibility

Files changed: 16 files, +2678 lines
Time saved: ~6-8 days (completed in 6 hours vs 3-5 day estimate)

* feat(evals): show child agent execution in non-debug mode

- Enable MultiAgentLogger in both debug and non-debug modes
- Add verbose parameter to control output level
- Non-verbose mode shows concise child session lifecycle:
  - Child agent started message
  - Child agent completed with duration
- Verbose mode (--debug) shows full delegation hierarchy
- Update documentation to reflect new behavior

This provides visibility into delegation without overwhelming output,
giving confidence that child agents are actually running.

* feat(evals): improve delegation testing and behavior validation

Major improvements to eval framework:

Test Suite:
- Add 00-smoke-test.yaml (basic read operation)
- Rename 01-smoke-test.yaml → 02-delegation-test.yaml (delegation test)
- Add simple-responder test agent for delegation testing
- Add debug test: simple-subagent-call.yaml

Evaluators:
- Add agent-model-evaluator.ts for agent/model validation
- Enhance behavior-evaluator.ts with detailed task tool output
- Improve delegation-evaluator.ts with better evidence tracking

Schema & Execution:
- Add expectedAgent and expectedModel to test schema
- Improve test executor with better logging and error handling

Documentation:
- Update evals/README.md with new features and performance improvements
- Remove outdated MULTI_AGENT_LOGGING_COMPLETE.md

Registry:
- Add simple-responder test agent to registry

Fixes:
- Fix dashboard.sh path resolution

These changes provide better visibility into delegation, improved test
coverage, and clearer validation of agent behavior.

* chore: add GitHub Actions workflow and plugin documentation

- Add .github/workflows/evals/run-evaluations.yml for automated eval testing
- Add dev/ai-tools/opencode/plugins/Plugin-inspiration.md for plugin development reference
2025-12-18 09:52:11 +00:00

454 lines
10 KiB
Markdown

# Category-Based Agent System
**Version**: 2.0.0
**Status**: Active
**Last Updated**: 2025-12-09
---
## 🎯 Overview
The category-based agent system organizes agents into logical categories, making it easier to discover, contribute, and maintain domain-specific agents.
### Key Concepts
**Three-Tier Hierarchy**:
1. **Core Agents** - Primary agents users interact with (orchestrator, coder)
2. **Category Agents** - Domain-specific specialists (frontend, copywriter, etc.)
3. **Subagents** - Delegated specialists called by core/category agents (tester, reviewer, etc.)
---
## 📁 Directory Structure
```
.opencode/agent/
├── core/ # Core Agents (System-level)
│ ├── openagent.md # Universal coordinator
│ └── opencoder.md # Development specialist
│
├── meta/ # Meta-level Agents
│ └── system-builder.md # System architect
│
├── development/ # Development Specialists
│ ├── frontend-specialist.md
│ ├── backend-specialist.md
│ ├── devops-specialist.md
│ └── codebase-agent.md
│
├── content/ # Content Creation
│ ├── copywriter.md
│ └── technical-writer.md
│
├── data/ # Data & Analysis
│ └── data-analyst.md
│
├── product/ # Product & Strategy
│ └── (empty - ready for contributions)
│
├── learning/ # Education & Coaching
│ └── (empty - ready for contributions)
│
└── subagents/ # Delegated Specialists
├── code/ # Code-related
├── core/ # Core workflow
├── system-builder/ # System generation
└── utils/ # Utilities
```
---
## 🔧 Using Category Agents
### Installation
```bash
# Install all development agents
./install.sh developer
# Install specific profile
./install.sh full
```
### Running Tests
```bash
cd evals/framework
# Test core agent
npm run eval:sdk -- --agent=core/openagent
# Test category agent
npm run eval:sdk -- --agent=development/frontend-specialist
# Test subagent
npm run eval:sdk -- --agent=subagents/code/tester
```
### Agent Invocation
**Core Agents** - Direct invocation:
```bash
opencode --agent core/openagent
opencode --agent core/opencoder
```
**Category Agents** - Direct invocation:
```bash
opencode --agent development/frontend-specialist
opencode --agent content/copywriter
```
**Subagents** - Via task delegation:
```javascript
// From core or category agent
task(
subagent_type="subagents/code/tester",
description="Write tests",
prompt="Create comprehensive tests for the authentication module"
)
```
---
## 📊 Categories
### Core (`core/`)
**Purpose**: Essential system-level agents for daily development work
**Agents**:
- `core/openagent` - Universal task coordinator
- `core/opencoder` - Development specialist
**When to use**: Primary agents for general tasks and development
---
### Meta (`meta/`)
**Purpose**: Meta-level agents for system generation and architecture design
**Agents**:
- `meta/system-builder` - System architect and generator
**When to use**: Building custom AI systems and agent architectures
---
### Development (`development/`)
**Purpose**: Software development specialists
**Agents**:
- `development/frontend-specialist` - React, Vue, modern CSS
- `development/backend-specialist` - APIs, databases, server-side
- `development/devops-specialist` - CI/CD, infrastructure
- `development/codebase-agent` - Multi-language implementation
**When to use**: Domain-specific development tasks
---
### Content (`content/`)
**Purpose**: Writing and content creation
**Agents**:
- `content/copywriter` - Marketing copy, brand messaging
- `content/technical-writer` - Documentation, API docs
**When to use**: Content creation and documentation
---
### Data (`data/`)
**Purpose**: Data analysis and research
**Agents**:
- `data/data-analyst` - Data analysis, visualization, insights
**When to use**: Data-driven tasks
---
### Product (`product/`)
**Purpose**: Product management and strategy
**Status**: Ready for contributions
**Potential Agents**:
- Product manager
- User researcher
- Business analyst
---
### Learning (`learning/`)
**Purpose**: Education and coaching
**Status**: Ready for contributions
**Potential Agents**:
- Tutor
- CBT coach
- Curriculum designer
---
## 🆚 Core vs Category vs Subagents
### Core Agents
**Characteristics**:
- System-level, maintained by repository
- General-purpose, work across domains
- Primary entry points for users
- Can delegate to category agents or subagents
**Examples**: `core/openagent`, `core/opencoder`
---
### Category Agents
**Characteristics**:
- Domain-specific specialists
- User-facing (can be invoked directly)
- Community-contributed
- Can delegate to subagents
**Examples**: `development/frontend-specialist`, `content/copywriter`
---
### Subagents
**Characteristics**:
- Focused, single-purpose functionality
- NOT directly invoked by users
- Called via task delegation
- Support core and category agents
**Examples**: `subagents/code/tester`, `subagents/core/task-manager`
---
## 🔄 Backward Compatibility
### Old Format Still Works
The old flat structure is still supported:
```bash
# Old format (still works)
npm run eval:sdk -- --agent=openagent
npm run eval:sdk -- --agent=opencoder
# New format (recommended)
npm run eval:sdk -- --agent=core/openagent
npm run eval:sdk -- --agent=core/opencoder
```
### Path Resolution
The eval framework automatically resolves old agent names:
- `openagent` → `core/openagent`
- `opencoder` → `core/opencoder`
- `system-builder` → `meta/system-builder`
### Deprecation Timeline
- **v2.0.0** (current): New structure, old paths work with warnings
- **v2.1.0**: Old paths still work, louder warnings
- **v3.0.0**: Remove old paths (breaking change)
---
## 🤝 Contributing Category Agents
See [ADDING_CATEGORY_AGENT.md](../contributing/ADDING_CATEGORY_AGENT.md) for detailed instructions.
### Quick Start
1. **Choose a category** (development, content, data, product, learning)
2. **Create agent file**: `.opencode/agent/{category}/{agent-name}.md`
3. **Add frontmatter** with metadata
4. **Create tests**: `evals/agents/{category}/{agent-name}/`
5. **Submit PR**
### Agent File Template
```markdown
---
id: agent-name
name: Agent Name
description: Brief description
category: development
type: standard
version: 1.0.0
author: your-name
mode: primary
model: anthropic/claude-sonnet-4-5
temperature: 0.1
tools:
read: true
write: true
edit: true
bash: false
glob: true
grep: true
task: true
dependencies:
context:
- development/patterns
tools: []
tags:
- development
- specialist
---
# Agent Name
You are a specialist in...
## Your Role
[Define role and responsibilities]
## Workflow
1. **Analyze** - Understand the request
2. **Plan** - Create implementation plan
3. **Request Approval** - Present plan to user
4. **Implement** - Execute following patterns
5. **Validate** - Test and verify
```
---
## 📚 Related Documentation
- [Migration Guide](../guides/MIGRATION_V2.md) - Migrating from v1.x to v2.0
- [Contributing Guide](../contributing/CONTRIBUTING.md) - General contribution guidelines
- [Adding Category Agents](../contributing/ADDING_CATEGORY_AGENT.md) - Detailed agent creation guide
- [Eval Framework](../../evals/framework/README.md) - Testing agents
---
## 🔍 Examples
### Example 1: Using Frontend Specialist
```bash
# Install
./install.sh developer
# Test
cd evals/framework
npm run eval:sdk -- --agent=development/frontend-specialist
# Use
opencode --agent development/frontend-specialist
```
### Example 2: Creating a New Category Agent
```bash
# Create agent file
vim .opencode/agent/product/product-manager.md
# Create test structure
mkdir -p evals/agents/product/product-manager/{config,tests}
# Create smoke test
vim evals/agents/product/product-manager/tests/smoke-test.yaml
# Validate
./scripts/registry/validate-agent-structure.sh
# Test
cd evals/framework
npm run eval:sdk -- --agent=product/product-manager
```
### Example 3: Delegating to Subagent
From a category agent:
```javascript
// Delegate test creation to tester subagent
task(
subagent_type="subagents/code/tester",
description="Create tests",
prompt="Write comprehensive unit tests for the UserService class"
)
```
---
## ❓ FAQ
### Q: Can I still use the old agent names?
**A**: Yes! Old names like `openagent` and `opencoder` still work and will continue to work until v3.0.0.
### Q: How do I know which category to use?
**A**: Choose based on the agent's primary domain:
- Code-related → `development/`
- Writing-related → `content/`
- Data-related → `data/`
- Product-related → `product/`
- Teaching-related → `learning/`
### Q: Can an agent be in multiple categories?
**A**: No, each agent belongs to one category. If an agent spans multiple domains, choose the primary domain or create a more general agent in `core/`.
### Q: What's the difference between a category agent and a subagent?
**A**:
- **Category agents** are user-facing and can be invoked directly
- **Subagents** are internal and only called via task delegation
### Q: How do I test my category agent?
**A**: Create a test suite in `evals/agents/{category}/{agent-name}/` and run:
```bash
cd evals/framework
npm run eval:sdk -- --agent={category}/{agent-name}
```
---
## 🎓 Best Practices
1. **Choose the right category** - Place agents in their primary domain
2. **Follow naming conventions** - Use kebab-case for agent IDs
3. **Write comprehensive tests** - At least a smoke test
4. **Document dependencies** - List required context files
5. **Keep agents focused** - One domain, one responsibility
6. **Use subagents** - Delegate complex subtasks
---
## 📊 System Statistics
- **Categories**: 6 (core, development, content, data, product, learning)
- **Core Agents**: 3
- **Category Agents**: 7
- **Subagents**: 13
- **Total Agents**: 23
---
**Last Updated**: 2025-12-09
**Version**: 2.0.0