cc96acc50e
- Add 5 workflow tests (task, context loading, stop-on-failure, conversational) - Add 1 shared test (agent-agnostic approval gate) - Reorganize: opencode/ → agents/ for clearer structure - Rename: sdk-tests/ → tests/ for simplicity - Update test runner paths to new structure - Add comprehensive documentation (test plan, guides, migration) - Update all documentation with new paths Total: 13 tests (was 6, added 7) Coverage: +31% workflow compliance All tests passing: 11/11 (100%)
394 lines
10 KiB
Markdown
394 lines
10 KiB
Markdown
# How Agent-Agnostic Testing Works (Simple Explanation)
|
|
|
|
## The Problem We Solved
|
|
|
|
**Question:** How do we test multiple agents (OpenAgent, OpenCoder, future agents) without duplicating code?
|
|
|
|
**Answer:** Separate the **framework** (shared) from the **tests** (per agent).
|
|
|
|
---
|
|
|
|
## Simple Analogy
|
|
|
|
Think of it like a **restaurant kitchen**:
|
|
|
|
- **Framework** = Kitchen equipment (oven, stove, knives) - works for any chef
|
|
- **Tests** = Recipes - each chef has their own recipes
|
|
- **Evaluators** = Quality inspectors - check if food is cooked properly (same standards for all chefs)
|
|
|
|
---
|
|
|
|
## How It Works (3 Simple Parts)
|
|
|
|
### **Part 1: Framework (The Kitchen Equipment)**
|
|
|
|
```
|
|
evals/framework/
|
|
├── src/sdk/test-runner.ts ← Runs tests for ANY agent
|
|
├── src/evaluators/ ← Checks behaviors for ANY agent
|
|
│ ├── approval-gate-evaluator.ts
|
|
│ ├── context-loading-evaluator.ts
|
|
│ └── tool-usage-evaluator.ts
|
|
```
|
|
|
|
**What it does:**
|
|
- Reads test files (YAML)
|
|
- Sends prompts to the agent specified in the test
|
|
- Captures events (tool calls, approvals, etc.)
|
|
- Runs evaluators to check if agent followed rules
|
|
|
|
**Key:** This code works with **any agent** - it doesn't care which agent it's testing.
|
|
|
|
---
|
|
|
|
### **Part 2: Tests (The Recipes)**
|
|
|
|
```
|
|
evals/agents/
|
|
├── openagent/ ← OpenAgent's recipes
|
|
│ └── tests/
|
|
│ ├── developer/
|
|
│ │ ├── task-simple-001.yaml agent: openagent
|
|
│ │ └── ctx-code-001.yaml agent: openagent
|
|
│ └── business/
|
|
│ └── conv-simple-001.yaml agent: openagent
|
|
│
|
|
├── opencoder/ ← OpenCoder's recipes (future)
|
|
│ └── tests/
|
|
│ └── developer/
|
|
│ └── refactor-001.yaml agent: opencoder
|
|
│
|
|
└── shared/ ← Recipes that work for ANY chef
|
|
└── tests/
|
|
└── common/
|
|
└── approval-gate-basic.yaml agent: openagent (default)
|
|
```
|
|
|
|
**What it does:**
|
|
- Each test file specifies which agent to test: `agent: openagent`
|
|
- Tests are organized by agent for easy management
|
|
- Shared tests can be used for multiple agents
|
|
|
|
---
|
|
|
|
### **Part 3: How They Connect**
|
|
|
|
```yaml
|
|
# Test file: openagent/tests/developer/task-simple-001.yaml
|
|
id: task-simple-001
|
|
name: Simple Bash Execution
|
|
agent: openagent ← This tells the framework which agent to test
|
|
prompt: "Run npm install"
|
|
|
|
behavior:
|
|
mustUseTools: [bash]
|
|
requiresApproval: true
|
|
```
|
|
|
|
**What happens:**
|
|
|
|
1. **Test Runner reads the file**
|
|
```typescript
|
|
const testCase = loadTestCase('task-simple-001.yaml');
|
|
// testCase.agent = 'openagent'
|
|
```
|
|
|
|
2. **Test Runner sends prompt to specified agent**
|
|
```typescript
|
|
const agent = testCase.agent; // 'openagent'
|
|
await sendPrompt(sessionId, testCase.prompt, { agent });
|
|
// SDK routes to OpenAgent
|
|
```
|
|
|
|
3. **Evaluators check behavior (works for any agent)**
|
|
```typescript
|
|
// Did the agent ask for approval?
|
|
const hasApproval = events.some(e => e.type === 'approval_request');
|
|
|
|
if (!hasApproval) {
|
|
violations.push({
|
|
type: 'approval-gate-missing',
|
|
message: 'Agent did not request approval'
|
|
});
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Example: Testing Two Different Agents
|
|
|
|
### **OpenAgent Test**
|
|
|
|
```yaml
|
|
# openagent/tests/developer/create-file.yaml
|
|
id: openagent-create-file-001
|
|
agent: openagent ← Routes to OpenAgent
|
|
prompt: "Create hello.ts"
|
|
|
|
behavior:
|
|
requiresContext: true ← OpenAgent must load code.md
|
|
requiresApproval: true
|
|
```
|
|
|
|
**What happens:**
|
|
1. Test runner sends "Create hello.ts" to **OpenAgent**
|
|
2. OpenAgent processes the request
|
|
3. Evaluators check:
|
|
- ✅ Did OpenAgent ask for approval?
|
|
- ✅ Did OpenAgent load code.md?
|
|
|
|
---
|
|
|
|
### **OpenCoder Test (Same Test, Different Agent)**
|
|
|
|
```yaml
|
|
# opencoder/tests/developer/create-file.yaml
|
|
id: opencoder-create-file-001
|
|
agent: opencoder ← Routes to OpenCoder
|
|
prompt: "Create hello.ts"
|
|
|
|
behavior:
|
|
requiresContext: false ← OpenCoder might not need context
|
|
requiresApproval: true
|
|
```
|
|
|
|
**What happens:**
|
|
1. Test runner sends "Create hello.ts" to **OpenCoder**
|
|
2. OpenCoder processes the request
|
|
3. Evaluators check:
|
|
- ✅ Did OpenCoder ask for approval?
|
|
- ⏭️ Context loading not required for OpenCoder
|
|
|
|
---
|
|
|
|
### **Shared Test (Works for Both)**
|
|
|
|
```yaml
|
|
# shared/tests/common/approval-gate-basic.yaml
|
|
id: shared-approval-001
|
|
agent: openagent ← Default (can be overridden)
|
|
prompt: "Create test.txt"
|
|
|
|
behavior:
|
|
requiresApproval: true ← Universal rule for ALL agents
|
|
```
|
|
|
|
**Run for OpenAgent:**
|
|
```bash
|
|
npm run eval:sdk -- --pattern="shared/**/*.yaml" --agent=openagent
|
|
```
|
|
|
|
**Run for OpenCoder:**
|
|
```bash
|
|
npm run eval:sdk -- --pattern="shared/**/*.yaml" --agent=opencoder
|
|
```
|
|
|
|
**What happens:**
|
|
- Same test file
|
|
- Different agent specified at runtime
|
|
- Same evaluators check both agents
|
|
|
|
---
|
|
|
|
## Why This Is Powerful
|
|
|
|
### **1. No Code Duplication**
|
|
|
|
**Without agent-agnostic design:**
|
|
```
|
|
evals/
|
|
├── openagent-framework/ ← Duplicate code
|
|
│ ├── test-runner.ts
|
|
│ └── evaluators/
|
|
├── opencoder-framework/ ← Duplicate code
|
|
│ ├── test-runner.ts
|
|
│ └── evaluators/
|
|
```
|
|
|
|
**With agent-agnostic design:**
|
|
```
|
|
evals/
|
|
├── framework/ ← Shared code (write once)
|
|
│ ├── test-runner.ts
|
|
│ └── evaluators/
|
|
├── agents/
|
|
│ ├── openagent/ ← Just tests
|
|
│ └── opencoder/ ← Just tests
|
|
```
|
|
|
|
---
|
|
|
|
### **2. Easy to Add New Agents**
|
|
|
|
**Step 1:** Create directory
|
|
```bash
|
|
mkdir -p evals/agents/my-new-agent/tests/developer
|
|
```
|
|
|
|
**Step 2:** Copy shared tests
|
|
```bash
|
|
cp evals/agents/shared/tests/common/*.yaml \
|
|
evals/agents/my-new-agent/tests/developer/
|
|
```
|
|
|
|
**Step 3:** Update agent field
|
|
```bash
|
|
sed -i 's/agent: openagent/agent: my-new-agent/g' \
|
|
evals/agents/my-new-agent/tests/developer/*.yaml
|
|
```
|
|
|
|
**Step 4:** Run tests
|
|
```bash
|
|
npm run eval:sdk -- --pattern="my-new-agent/**/*.yaml"
|
|
```
|
|
|
|
**Done!** No framework code changes needed.
|
|
|
|
---
|
|
|
|
### **3. Consistent Behavior Across Agents**
|
|
|
|
Same evaluators check all agents:
|
|
|
|
```typescript
|
|
// approval-gate-evaluator.ts
|
|
// This code runs for OpenAgent, OpenCoder, and any future agent
|
|
|
|
export class ApprovalGateEvaluator extends BaseEvaluator {
|
|
async evaluate(timeline: TimelineEvent[]) {
|
|
// Check if agent asked for approval
|
|
const hasApproval = timeline.some(e => e.type === 'approval_request');
|
|
|
|
if (!hasApproval) {
|
|
// This violation applies to ANY agent
|
|
violations.push({
|
|
type: 'approval-gate-missing',
|
|
message: 'Agent did not request approval'
|
|
});
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
**Result:** All agents are held to the same standards.
|
|
|
|
---
|
|
|
|
### **4. Easy to Compare Agents**
|
|
|
|
Run the same test on different agents:
|
|
|
|
```bash
|
|
# Test OpenAgent
|
|
npm run eval:sdk -- --pattern="shared/approval-gate-basic.yaml" --agent=openagent
|
|
|
|
# Test OpenCoder
|
|
npm run eval:sdk -- --pattern="shared/approval-gate-basic.yaml" --agent=opencoder
|
|
|
|
# Compare results
|
|
```
|
|
|
|
---
|
|
|
|
## Directory Organization (Simple View)
|
|
|
|
```
|
|
evals/
|
|
│
|
|
├── framework/ ← SHARED (works with any agent)
|
|
│ ├── src/sdk/ ← Test runner
|
|
│ │ ├── test-runner.ts ← Reads 'agent' field from YAML
|
|
│ │ └── client-manager.ts ← Routes to correct agent
|
|
│ └── src/evaluators/ ← Generic behavior checks
|
|
│ ├── approval-gate-evaluator.ts
|
|
│ └── context-loading-evaluator.ts
|
|
│
|
|
├── agents/
|
|
│ │
|
|
│ ├── openagent/ ← OpenAgent-specific
|
|
│ │ ├── tests/ ← Tests for OpenAgent
|
|
│ │ │ ├── developer/
|
|
│ │ │ │ ├── task-simple-001.yaml agent: openagent
|
|
│ │ │ │ └── ctx-code-001.yaml agent: openagent
|
|
│ │ │ └── business/
|
|
│ │ │ └── conv-simple-001.yaml agent: openagent
|
|
│ │ └── docs/
|
|
│ │ └── OPENAGENT_RULES.md ← Rules from openagent.md
|
|
│ │
|
|
│ ├── opencoder/ ← OpenCoder-specific (future)
|
|
│ │ ├── tests/ ← Tests for OpenCoder
|
|
│ │ │ └── developer/
|
|
│ │ │ └── refactor-001.yaml agent: opencoder
|
|
│ │ └── docs/
|
|
│ │ └── OPENCODER_RULES.md ← Rules from opencoder.md
|
|
│ │
|
|
│ └── shared/ ← Tests for ANY agent
|
|
│ └── tests/
|
|
│ └── common/
|
|
│ └── approval-gate-basic.yaml agent: ${AGENT}
|
|
```
|
|
|
|
---
|
|
|
|
## Running Tests (Simple Commands)
|
|
|
|
### **Run All Tests for One Agent**
|
|
|
|
```bash
|
|
# All OpenAgent tests
|
|
npm run eval:sdk -- --pattern="openagent/**/*.yaml"
|
|
|
|
# All OpenCoder tests
|
|
npm run eval:sdk -- --pattern="opencoder/**/*.yaml"
|
|
```
|
|
|
|
### **Run Specific Category**
|
|
|
|
```bash
|
|
# OpenAgent developer tests
|
|
npm run eval:sdk -- --pattern="openagent/developer/*.yaml"
|
|
|
|
# OpenCoder developer tests
|
|
npm run eval:sdk -- --pattern="opencoder/developer/*.yaml"
|
|
```
|
|
|
|
### **Run Shared Tests for Different Agents**
|
|
|
|
```bash
|
|
# Shared tests for OpenAgent
|
|
npm run eval:sdk -- --pattern="shared/**/*.yaml" --agent=openagent
|
|
|
|
# Shared tests for OpenCoder
|
|
npm run eval:sdk -- --pattern="shared/**/*.yaml" --agent=opencoder
|
|
```
|
|
|
|
---
|
|
|
|
## Key Takeaways
|
|
|
|
1. **Framework is agent-agnostic** - Works with any agent
|
|
2. **Tests specify which agent** - `agent: openagent` in YAML
|
|
3. **Evaluators are generic** - Check behaviors, not agent-specific logic
|
|
4. **Easy to add new agents** - Just create directory and tests
|
|
5. **No code duplication** - Framework code written once
|
|
6. **Consistent standards** - Same evaluators for all agents
|
|
7. **Easy to manage** - Clear directory structure
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
**The Magic:**
|
|
- Write framework code **once**
|
|
- Write evaluators **once**
|
|
- Write tests **per agent**
|
|
- Specify agent in test file: `agent: openagent`
|
|
- Test runner routes to correct agent
|
|
- Evaluators check generic behaviors
|
|
|
|
**The Result:**
|
|
- Easy to test multiple agents
|
|
- No code duplication
|
|
- Consistent behavior validation
|
|
- Simple to add new agents
|
|
- Clear organization
|