OpenWriterControle/evals/SUBAGENT_TESTING.md

Subagent Testing Guide

Quick reference for testing subagents with the evaluation framework.

---

🎯 Two Testing Modes

1. Standalone Mode (Default)

Tests the subagent directly by forcing mode: primary.

# CLI
npm run eval:sdk -- --subagent=coder-agent

# Makefile
make test-subagent SUBAGENT=coder-agent

What happens:

• ✅ Subagent runs as primary agent (mode override)
• ✅ Tests subagent logic in isolation
• ⚠️ Not how it runs in production (production uses mode: subagent)

Use when:

• Debugging subagent behavior
• Testing subagent logic independently
• Developing new subagents

2. Delegation Mode

Tests the subagent via its parent agent (real-world usage).

# CLI
npm run eval:sdk -- --subagent=coder-agent --delegate

# Makefile
make test-subagent-delegate SUBAGENT=coder-agent

What happens:

• ✅ Parent agent delegates to subagent
• ✅ Tests real production workflow
• ✅ Validates delegation logic

Use when:

• Testing production workflows
• Validating delegation patterns
• Integration testing

---

📋 Available Subagents

Code Subagents (Parent: opencoder)

--subagent=coder-agent              # Simple task executor
--subagent=tester                   # Test authoring
--subagent=reviewer                 # Code review
--subagent=build-agent              # Type check & build
--subagent=codebase-pattern-analyst # Pattern analysis

Core Subagents (Parent: openagent)

--subagent=task-manager             # Feature breakdown
--subagent=documentation            # Doc generation
--subagent=context-retriever        # Context search

System-Builder Subagents (Parent: system-builder)

--subagent=agent-generator          # Agent creation
--subagent=command-creator          # Command creation
--subagent=context-organizer        # Context organization
--subagent=domain-analyzer          # Domain analysis
--subagent=workflow-designer        # Workflow design

Utils Subagents (Parent: openagent)

--subagent=image-specialist         # Image editing

---

🚀 Quick Examples

Test coder-agent standalone

# With npm
npm run eval:sdk -- --subagent=coder-agent --pattern="**/smoke-test.yaml"

# With make
make test-subagent SUBAGENT=coder-agent

Test coder-agent via opencoder delegation

# With npm
npm run eval:sdk -- --subagent=coder-agent --delegate --verbose

# With make
make test-subagent-delegate SUBAGENT=coder-agent

Test with specific model

npm run eval:sdk -- --subagent=tester --model=anthropic/claude-3-5-sonnet-20241022

Debug with verbose output

npm run eval:sdk -- --subagent=reviewer --verbose --debug

---

📊 Output Examples

Standalone Mode

⚡ Standalone Test Mode
   Subagent: coder-agent
   Mode: Forced to 'primary' for direct testing
   Note: In production, this subagent runs as 'mode: subagent'

Starting test runner...
[TestRunner] Forced mode: primary for standalone subagent testing
✅ Test runner started

Delegation Mode

🔗 Delegation Test Mode
   Subagent: coder-agent
   Parent: opencoder
   Tests will verify delegation from opencoder → coder-agent

Starting test runner...
✅ Test runner started

---

❌ Error Handling

Unknown Subagent

$ npm run eval:sdk -- --subagent=unknown-agent

❌ Error: Unknown subagent 'unknown-agent'

📋 Available subagents:

  Code subagents (parent: opencoder):
    - coder-agent, tester, reviewer, build-agent, codebase-pattern-analyst

  Core subagents (parent: openagent):
    - task-manager, documentation, context-retriever

  System-builder subagents (parent: system-builder):
    - agent-generator, command-creator, context-organizer
    - domain-analyzer, workflow-designer

  Utils subagents (parent: openagent):
    - image-specialist

Invalid Flag Combination

$ npm run eval:sdk -- --agent=openagent --subagent=coder-agent --delegate

❌ Error: Cannot use --delegate with --agent
   Use either:
     --subagent=NAME              (standalone mode)
     --subagent=NAME --delegate   (delegation mode)
     --agent=NAME                 (main agent)

Agent File Not Found

$ npm run eval:sdk -- --subagent=missing-agent

❌ Error: Agent file not found: .opencode/agent/subagents/code/missing-agent.md
Available agents: openagent, opencoder, coder-agent, tester, ...

---

🧪 Creating Subagent Tests

Directory Structure

evals/agents/subagents/
└── coder-agent/
    └── tests/
        ├── standalone/           # For --subagent mode
        │   ├── 01-simple-task.yaml
        │   └── 02-error-handling.yaml
        └── delegation/           # For --subagent --delegate mode
            ├── 01-opencoder-delegates.yaml
            └── 02-task-completion.yaml

Standalone Test Example

id: coder-standalone-01
name: Coder Agent - Simple Function Creation
description: Test coder-agent creates a simple function

prompts:
  - text: |
      Create a simple add function in src/utils/math.ts      

behavior:
  mustUseTools: [write]
  requiresApproval: true

expectedViolations:
  - rule: approval-gate
    shouldViolate: false

Delegation Test Example

id: coder-delegation-01
name: OpenCoder Delegates to Coder Agent
description: Test opencoder correctly delegates to coder-agent

prompts:
  - text: |
      Create a simple utility function.
      This is a simple task, delegate it to @coder-agent.      

behavior:
  mustUseTools: [task]  # Must delegate
  requiresApproval: true

expectedViolations:
  - rule: delegation
    shouldViolate: false

---

💡 Best Practices

1. Start with standalone mode for quick iteration
2. Use delegation mode for production validation
3. Test both modes for comprehensive coverage
4. Use --verbose when debugging
5. Check parent agent matches your use case

---

🔧 Troubleshooting

Subagent doesn't respond

• ✅ Check mode was forced to primary (standalone mode)
• ✅ Verify agent file exists
• ✅ Check OpenCode server logs with --debug

Delegation not working

• ✅ Verify parent agent is correct
• ✅ Check delegation syntax in test prompt
• ✅ Use --verbose to see full conversation

Tests failing unexpectedly

• ✅ Run with --verbose to see full output
• ✅ Check evaluator violations
• ✅ Verify test expectations match mode

---

Questions? See evals/README.md or open an issue.