chore: cleanup documentation - remove temporary and redundant files
Cleanup Summary: - Deleted .tmp/ directory (20+ temporary development files) - Deleted tasks/ directory (old task tracking files) - Removed 10 redundant/historical markdown files from evals/ Files Deleted: - evals/DOCUMENTATION_CLEANUP.md (historical) - evals/SCRIPTS_ORGANIZATION.md (historical) - evals/agents/AGENT_TESTING_GUIDE.md (redundant) - evals/agents/openagent/CONTEXT_LOADING_COVERAGE.md (historical) - evals/agents/openagent/IMPLEMENTATION_SUMMARY.md (historical) - evals/agents/opencoder/README.md (redundant) - evals/agents/shared/README.md (redundant) - evals/framework/scripts/README.md (redundant) - evals/results/README.md (redundant) - evals/test_tmp/README.md (redundant) Result: - Reduced from 120+ markdown files to 92 (excluding node_modules) - Reduced evals/ from 20 to 10 markdown files - Kept essential documentation: - evals/README.md (main guide) - evals/GETTING_STARTED.md (quick start) - evals/HOW_TESTS_WORK.md (technical details) - evals/ARCHITECTURE.md (system architecture) - evals/framework/README.md (framework docs) - evals/framework/SDK_EVAL_README.md (SDK guide) - evals/agents/openagent/README.md (test suite) - Kept dev/ directory (important OpenCode internals documentation) Benefits: - Clearer documentation structure - Less confusion about where to find information - Removed outdated/historical files - Easier to navigate and maintain
This commit is contained in:
darrenhinde
parent
d7e1aa7f19
commit
f98a23c1d4
10 changed files with 0 additions and 2229 deletions
|
|
@ -1,273 +0,0 @@
|
||||||
# Documentation Cleanup Summary
|
|
||||||
|
|
||||||
**Date**: 2025-11-26
|
|
||||||
**Status**: ✅ Complete
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Changes Made
|
|
||||||
|
|
||||||
### Files Deleted (3)
|
|
||||||
|
|
||||||
1. **`evals/framework/SESSION_STORAGE_FIX.md`** (173 lines)
|
|
||||||
- **Reason**: Historical fix documentation, no longer relevant
|
|
||||||
- **Status**: ✅ Deleted
|
|
||||||
|
|
||||||
2. **`evals/TESTING_CONFIDENCE.md`** (121 lines)
|
|
||||||
- **Reason**: Outdated, superseded by IMPLEMENTATION_SUMMARY.md
|
|
||||||
- **Content**: Old test confidence assessment from before context loading fixes
|
|
||||||
- **Status**: ✅ Deleted
|
|
||||||
|
|
||||||
3. **`evals/agents/openagent/TEST_REVIEW.md`** (325 lines)
|
|
||||||
- **Reason**: Outdated test review from Nov 25 (before context loading fixes)
|
|
||||||
- **Content**: Old test results, superseded by CONTEXT_LOADING_COVERAGE.md and IMPLEMENTATION_SUMMARY.md
|
|
||||||
- **Status**: ✅ Deleted
|
|
||||||
|
|
||||||
### Files Renamed (1)
|
|
||||||
|
|
||||||
1. **`evals/SYSTEM_REVIEW.md` → `evals/ARCHITECTURE.md`**
|
|
||||||
- **Reason**: More descriptive name for system architecture review
|
|
||||||
- **Content**: Comprehensive architecture review (456 lines)
|
|
||||||
- **Status**: ✅ Renamed
|
|
||||||
|
|
||||||
### Files Created (2)
|
|
||||||
|
|
||||||
1. **`evals/GETTING_STARTED.md`** (NEW - 450 lines)
|
|
||||||
- **Purpose**: Consolidated quick start guide
|
|
||||||
- **Content**:
|
|
||||||
- Running tests
|
|
||||||
- Understanding results
|
|
||||||
- Creating new tests
|
|
||||||
- Debugging
|
|
||||||
- Common issues
|
|
||||||
- **Replaces**: Scattered information from README.md and HOW_TESTS_WORK.md
|
|
||||||
- **Status**: ✅ Created
|
|
||||||
|
|
||||||
2. **`evals/DOCUMENTATION_CLEANUP.md`** (THIS FILE)
|
|
||||||
- **Purpose**: Track documentation cleanup changes
|
|
||||||
- **Status**: ✅ Created
|
|
||||||
|
|
||||||
### Files Updated (3)
|
|
||||||
|
|
||||||
1. **`evals/README.md`** (322 → 280 lines)
|
|
||||||
- **Changes**:
|
|
||||||
- More concise overview
|
|
||||||
- Points to GETTING_STARTED.md for details
|
|
||||||
- Updated with recent achievements (Nov 26)
|
|
||||||
- Added context loading tests section
|
|
||||||
- Added smart timeout system section
|
|
||||||
- Updated test coverage numbers
|
|
||||||
- **Status**: ✅ Updated
|
|
||||||
|
|
||||||
2. **`evals/agents/openagent/README.md`** (85 → 350 lines)
|
|
||||||
- **Changes**:
|
|
||||||
- Comprehensive test coverage section
|
|
||||||
- Detailed context loading tests documentation
|
|
||||||
- Test structure overview
|
|
||||||
- Running instructions
|
|
||||||
- Test design examples
|
|
||||||
- Troubleshooting section
|
|
||||||
- **Status**: ✅ Updated
|
|
||||||
|
|
||||||
3. **`evals/HOW_TESTS_WORK.md`** (308 lines)
|
|
||||||
- **Changes**: None (kept as-is for detailed technical reference)
|
|
||||||
- **Status**: ✅ Kept
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Documentation Structure (After Cleanup)
|
|
||||||
|
|
||||||
### Top-Level Documentation
|
|
||||||
|
|
||||||
```
|
|
||||||
evals/
|
|
||||||
├── README.md # System overview (UPDATED)
|
|
||||||
├── GETTING_STARTED.md # Quick start guide (NEW)
|
|
||||||
├── HOW_TESTS_WORK.md # Detailed test execution guide
|
|
||||||
├── ARCHITECTURE.md # System architecture review (RENAMED)
|
|
||||||
└── DOCUMENTATION_CLEANUP.md # This file (NEW)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Framework Documentation
|
|
||||||
|
|
||||||
```
|
|
||||||
evals/framework/
|
|
||||||
├── README.md # Framework documentation
|
|
||||||
├── SDK_EVAL_README.md # Complete SDK guide
|
|
||||||
├── docs/
|
|
||||||
│ ├── architecture-overview.md # Framework architecture
|
|
||||||
│ └── test-design-guide.md # Test design philosophy
|
|
||||||
└── run-tests-batch.sh # Batch test runner
|
|
||||||
```
|
|
||||||
|
|
||||||
### Agent Documentation
|
|
||||||
|
|
||||||
```
|
|
||||||
evals/agents/openagent/
|
|
||||||
├── README.md # OpenAgent test suite (UPDATED)
|
|
||||||
├── CONTEXT_LOADING_COVERAGE.md # Context loading tests
|
|
||||||
├── IMPLEMENTATION_SUMMARY.md # Recent implementation
|
|
||||||
└── docs/
|
|
||||||
└── OPENAGENT_RULES.md # OpenAgent rules reference
|
|
||||||
```
|
|
||||||
|
|
||||||
### Results Documentation
|
|
||||||
|
|
||||||
```
|
|
||||||
evals/results/
|
|
||||||
├── README.md # Results dashboard guide
|
|
||||||
├── index.html # Interactive dashboard
|
|
||||||
└── serve.sh # One-command server
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Documentation Flow
|
|
||||||
|
|
||||||
### For New Users
|
|
||||||
|
|
||||||
1. **Start**: `README.md` - System overview
|
|
||||||
2. **Next**: `GETTING_STARTED.md` - Quick start guide
|
|
||||||
3. **Then**: Run tests and view results
|
|
||||||
4. **Deep Dive**: `HOW_TESTS_WORK.md` - Detailed explanations
|
|
||||||
|
|
||||||
### For Test Authors
|
|
||||||
|
|
||||||
1. **Start**: `GETTING_STARTED.md` - Creating tests section
|
|
||||||
2. **Reference**: `framework/docs/test-design-guide.md` - Design philosophy
|
|
||||||
3. **Examples**: `agents/openagent/README.md` - Test examples
|
|
||||||
4. **Rules**: `agents/openagent/docs/OPENAGENT_RULES.md` - Agent rules
|
|
||||||
|
|
||||||
### For Developers
|
|
||||||
|
|
||||||
1. **Start**: `ARCHITECTURE.md` - System architecture
|
|
||||||
2. **Framework**: `framework/SDK_EVAL_README.md` - Complete SDK guide
|
|
||||||
3. **Implementation**: `agents/openagent/IMPLEMENTATION_SUMMARY.md` - Recent changes
|
|
||||||
4. **Technical**: `HOW_TESTS_WORK.md` - Execution details
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Benefits of Cleanup
|
|
||||||
|
|
||||||
### Before Cleanup
|
|
||||||
|
|
||||||
- ❌ 19 markdown files (excluding node_modules)
|
|
||||||
- ❌ Outdated information (Nov 25 test reviews)
|
|
||||||
- ❌ Duplicate content (testing confidence in multiple places)
|
|
||||||
- ❌ Unclear entry point for new users
|
|
||||||
- ❌ Historical fix documentation cluttering framework/
|
|
||||||
|
|
||||||
### After Cleanup
|
|
||||||
|
|
||||||
- ✅ 16 markdown files (3 deleted, 2 new, net -1)
|
|
||||||
- ✅ All information current (Nov 26)
|
|
||||||
- ✅ No duplicate content
|
|
||||||
- ✅ Clear entry point (GETTING_STARTED.md)
|
|
||||||
- ✅ Clean framework directory
|
|
||||||
- ✅ Better organization
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Documentation Quality Metrics
|
|
||||||
|
|
||||||
### Coverage
|
|
||||||
|
|
||||||
| Audience | Documentation | Status |
|
|
||||||
|----------|---------------|--------|
|
|
||||||
| New Users | GETTING_STARTED.md | ✅ Complete |
|
|
||||||
| Test Authors | test-design-guide.md | ✅ Complete |
|
|
||||||
| Developers | ARCHITECTURE.md | ✅ Complete |
|
|
||||||
| OpenAgent Users | agents/openagent/README.md | ✅ Complete |
|
|
||||||
| Results Users | results/README.md | ✅ Complete |
|
|
||||||
|
|
||||||
### Accuracy
|
|
||||||
|
|
||||||
| Document | Last Updated | Accuracy |
|
|
||||||
|----------|--------------|----------|
|
|
||||||
| README.md | 2025-11-26 | ✅ Current |
|
|
||||||
| GETTING_STARTED.md | 2025-11-26 | ✅ Current |
|
|
||||||
| HOW_TESTS_WORK.md | 2025-11-26 | ✅ Current |
|
|
||||||
| ARCHITECTURE.md | 2025-11-26 | ✅ Current |
|
|
||||||
| agents/openagent/README.md | 2025-11-26 | ✅ Current |
|
|
||||||
| CONTEXT_LOADING_COVERAGE.md | 2025-11-26 | ✅ Current |
|
|
||||||
| IMPLEMENTATION_SUMMARY.md | 2025-11-26 | ✅ Current |
|
|
||||||
|
|
||||||
### Maintainability
|
|
||||||
|
|
||||||
- ✅ Clear naming conventions
|
|
||||||
- ✅ Logical organization
|
|
||||||
- ✅ No duplicate content
|
|
||||||
- ✅ Cross-references between docs
|
|
||||||
- ✅ Easy to find information
|
|
||||||
- ✅ Easy to update
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Maintenance Guidelines
|
|
||||||
|
|
||||||
### When to Update Documentation
|
|
||||||
|
|
||||||
1. **After Major Features**
|
|
||||||
- Update README.md with new features
|
|
||||||
- Update GETTING_STARTED.md with new usage examples
|
|
||||||
- Create/update implementation summaries
|
|
||||||
|
|
||||||
2. **After Bug Fixes**
|
|
||||||
- Update relevant documentation
|
|
||||||
- Add to troubleshooting sections if needed
|
|
||||||
|
|
||||||
3. **Monthly Review**
|
|
||||||
- Check for outdated information
|
|
||||||
- Update test coverage numbers
|
|
||||||
- Review and consolidate if needed
|
|
||||||
|
|
||||||
### What to Delete
|
|
||||||
|
|
||||||
- Historical fix documentation (after 3 months)
|
|
||||||
- Outdated test reviews (superseded by new ones)
|
|
||||||
- Duplicate content (consolidate instead)
|
|
||||||
- Temporary investigation notes
|
|
||||||
|
|
||||||
### What to Keep
|
|
||||||
|
|
||||||
- Architecture documentation
|
|
||||||
- Test design guides
|
|
||||||
- Getting started guides
|
|
||||||
- Current implementation summaries
|
|
||||||
- Troubleshooting guides
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Next Review
|
|
||||||
|
|
||||||
**Scheduled**: 2025-12-26 (1 month)
|
|
||||||
|
|
||||||
**Review Checklist**:
|
|
||||||
- [ ] Check for outdated information
|
|
||||||
- [ ] Update test coverage numbers
|
|
||||||
- [ ] Review new features added
|
|
||||||
- [ ] Check for duplicate content
|
|
||||||
- [ ] Verify all links work
|
|
||||||
- [ ] Update "Last Updated" dates
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
✅ **3 files deleted** (outdated/duplicate content)
|
|
||||||
✅ **1 file renamed** (better clarity)
|
|
||||||
✅ **2 files created** (better organization)
|
|
||||||
✅ **3 files updated** (current information)
|
|
||||||
✅ **Net result**: Cleaner, more organized, more maintainable documentation
|
|
||||||
|
|
||||||
**Documentation is now**:
|
|
||||||
- Current (all Nov 26, 2025)
|
|
||||||
- Well-organized (clear structure)
|
|
||||||
- Easy to navigate (clear entry points)
|
|
||||||
- Comprehensive (covers all audiences)
|
|
||||||
- Maintainable (no duplicates, clear guidelines)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Cleanup Completed**: 2025-11-26
|
|
||||||
**Next Review**: 2025-12-26
|
|
||||||
|
|
@ -1,367 +0,0 @@
|
||||||
# Scripts Organization Summary
|
|
||||||
|
|
||||||
**Date**: 2025-11-26
|
|
||||||
**Status**: ✅ Complete
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Changes Made
|
|
||||||
|
|
||||||
### Before Organization
|
|
||||||
|
|
||||||
```
|
|
||||||
evals/framework/
|
|
||||||
├── check-agent.mjs
|
|
||||||
├── debug-claude-session.mjs
|
|
||||||
├── debug-session.mjs
|
|
||||||
├── debug-session.ts
|
|
||||||
├── inspect-session.mjs
|
|
||||||
├── run-tests-batch.sh
|
|
||||||
├── test-agent-direct.ts
|
|
||||||
├── test-event-inspector.js
|
|
||||||
├── test-session-reader.mjs
|
|
||||||
├── test-simplified-approach.mjs
|
|
||||||
├── test-timeline.ts
|
|
||||||
├── verify-timeline.ts
|
|
||||||
└── ... (other framework files)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Issues**:
|
|
||||||
- ❌ 12 scripts cluttering framework root
|
|
||||||
- ❌ No clear organization
|
|
||||||
- ❌ Hard to find specific scripts
|
|
||||||
- ❌ Unclear which scripts are for what purpose
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### After Organization
|
|
||||||
|
|
||||||
```
|
|
||||||
evals/framework/
|
|
||||||
├── scripts/
|
|
||||||
│ ├── debug/ # Debugging scripts (4 files)
|
|
||||||
│ │ ├── debug-session.mjs
|
|
||||||
│ │ ├── debug-session.ts
|
|
||||||
│ │ ├── debug-claude-session.mjs
|
|
||||||
│ │ └── inspect-session.mjs
|
|
||||||
│ │
|
|
||||||
│ ├── test/ # Test scripts (6 files)
|
|
||||||
│ │ ├── test-agent-direct.ts
|
|
||||||
│ │ ├── test-event-inspector.js
|
|
||||||
│ │ ├── test-session-reader.mjs
|
|
||||||
│ │ ├── test-simplified-approach.mjs
|
|
||||||
│ │ ├── test-timeline.ts
|
|
||||||
│ │ └── verify-timeline.ts
|
|
||||||
│ │
|
|
||||||
│ ├── utils/ # Utility scripts (2 files)
|
|
||||||
│ │ ├── run-tests-batch.sh
|
|
||||||
│ │ └── check-agent.mjs
|
|
||||||
│ │
|
|
||||||
│ └── README.md # Script documentation
|
|
||||||
│
|
|
||||||
└── ... (other framework files)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Benefits**:
|
|
||||||
- ✅ Clean framework root
|
|
||||||
- ✅ Clear organization by purpose
|
|
||||||
- ✅ Easy to find scripts
|
|
||||||
- ✅ Comprehensive documentation
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Script Categories
|
|
||||||
|
|
||||||
### Debug Scripts (4 files)
|
|
||||||
|
|
||||||
Scripts for debugging sessions, events, and agent behavior.
|
|
||||||
|
|
||||||
| Script | Purpose | Lines |
|
|
||||||
|--------|---------|-------|
|
|
||||||
| `debug-session.mjs` | Debug session data and timeline | ~40 |
|
|
||||||
| `debug-session.ts` | TypeScript version of session debugger | ~100 |
|
|
||||||
| `debug-claude-session.mjs` | Debug Claude-specific sessions | ~50 |
|
|
||||||
| `inspect-session.mjs` | Inspect most recent session events | ~80 |
|
|
||||||
|
|
||||||
**Usage**:
|
|
||||||
```bash
|
|
||||||
node scripts/debug/inspect-session.mjs
|
|
||||||
node scripts/debug/debug-session.mjs <session-id>
|
|
||||||
npx tsx scripts/debug/debug-session.ts <session-id>
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Test Scripts (6 files)
|
|
||||||
|
|
||||||
Scripts for testing framework components during development.
|
|
||||||
|
|
||||||
| Script | Purpose | Lines |
|
|
||||||
|--------|---------|-------|
|
|
||||||
| `test-agent-direct.ts` | Direct agent execution test | ~150 |
|
|
||||||
| `test-event-inspector.js` | Test event capture system | ~40 |
|
|
||||||
| `test-session-reader.mjs` | Test session reader | ~60 |
|
|
||||||
| `test-simplified-approach.mjs` | Test simplified test approach | ~100 |
|
|
||||||
| `test-timeline.ts` | Test timeline builder | ~90 |
|
|
||||||
| `verify-timeline.ts` | Verify timeline accuracy | ~100 |
|
|
||||||
|
|
||||||
**Usage**:
|
|
||||||
```bash
|
|
||||||
npx tsx scripts/test/test-agent-direct.ts
|
|
||||||
node scripts/test/test-event-inspector.js
|
|
||||||
npx tsx scripts/test/verify-timeline.ts
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Utility Scripts (2 files)
|
|
||||||
|
|
||||||
General utility scripts for running tests and managing the framework.
|
|
||||||
|
|
||||||
| Script | Purpose | Lines |
|
|
||||||
|--------|---------|-------|
|
|
||||||
| `run-tests-batch.sh` | Run tests in batches | ~100 |
|
|
||||||
| `check-agent.mjs` | Check agent availability | ~30 |
|
|
||||||
|
|
||||||
**Usage**:
|
|
||||||
```bash
|
|
||||||
./scripts/utils/run-tests-batch.sh openagent 3 10
|
|
||||||
node scripts/utils/check-agent.mjs
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Documentation Updates
|
|
||||||
|
|
||||||
### Files Updated
|
|
||||||
|
|
||||||
1. **`evals/README.md`**
|
|
||||||
- Updated `run-tests-batch.sh` path references
|
|
||||||
- Updated directory structure
|
|
||||||
|
|
||||||
2. **`evals/GETTING_STARTED.md`**
|
|
||||||
- Updated batch execution examples
|
|
||||||
- Updated script paths
|
|
||||||
|
|
||||||
3. **`evals/agents/openagent/README.md`**
|
|
||||||
- Updated batch execution examples
|
|
||||||
- Updated script paths
|
|
||||||
|
|
||||||
4. **`evals/agents/openagent/IMPLEMENTATION_SUMMARY.md`**
|
|
||||||
- Updated script references
|
|
||||||
- Updated directory structure
|
|
||||||
|
|
||||||
5. **`evals/DOCUMENTATION_CLEANUP.md`**
|
|
||||||
- Updated directory structure
|
|
||||||
|
|
||||||
6. **`evals/framework/README.md`**
|
|
||||||
- Added scripts section
|
|
||||||
- Added quick examples
|
|
||||||
|
|
||||||
### New Documentation
|
|
||||||
|
|
||||||
1. **`evals/framework/scripts/README.md`** (NEW - 200 lines)
|
|
||||||
- Comprehensive script documentation
|
|
||||||
- Usage examples for all scripts
|
|
||||||
- Development workflow guide
|
|
||||||
- Script templates
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Path Changes
|
|
||||||
|
|
||||||
### Old Paths → New Paths
|
|
||||||
|
|
||||||
| Old Path | New Path |
|
|
||||||
|----------|----------|
|
|
||||||
| `run-tests-batch.sh` | `scripts/utils/run-tests-batch.sh` |
|
|
||||||
| `check-agent.mjs` | `scripts/utils/check-agent.mjs` |
|
|
||||||
| `debug-session.mjs` | `scripts/debug/debug-session.mjs` |
|
|
||||||
| `debug-session.ts` | `scripts/debug/debug-session.ts` |
|
|
||||||
| `debug-claude-session.mjs` | `scripts/debug/debug-claude-session.mjs` |
|
|
||||||
| `inspect-session.mjs` | `scripts/debug/inspect-session.mjs` |
|
|
||||||
| `test-agent-direct.ts` | `scripts/test/test-agent-direct.ts` |
|
|
||||||
| `test-event-inspector.js` | `scripts/test/test-event-inspector.js` |
|
|
||||||
| `test-session-reader.mjs` | `scripts/test/test-session-reader.mjs` |
|
|
||||||
| `test-simplified-approach.mjs` | `scripts/test/test-simplified-approach.mjs` |
|
|
||||||
| `test-timeline.ts` | `scripts/test/test-timeline.ts` |
|
|
||||||
| `verify-timeline.ts` | `scripts/test/verify-timeline.ts` |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Migration Guide
|
|
||||||
|
|
||||||
### For Users
|
|
||||||
|
|
||||||
If you have scripts or documentation referencing the old paths:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Old
|
|
||||||
./run-tests-batch.sh openagent 3 10
|
|
||||||
|
|
||||||
# New
|
|
||||||
./scripts/utils/run-tests-batch.sh openagent 3 10
|
|
||||||
```
|
|
||||||
|
|
||||||
### For Developers
|
|
||||||
|
|
||||||
If you have custom scripts importing from these files:
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
// Old
|
|
||||||
import { SessionReader } from './dist/collector/session-reader.js';
|
|
||||||
|
|
||||||
// New (from scripts directory)
|
|
||||||
import { SessionReader } from '../../dist/collector/session-reader.js';
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Benefits
|
|
||||||
|
|
||||||
### Organization
|
|
||||||
|
|
||||||
- ✅ **Clear structure** - Scripts grouped by purpose
|
|
||||||
- ✅ **Easy navigation** - Know where to find scripts
|
|
||||||
- ✅ **Clean root** - Framework root no longer cluttered
|
|
||||||
- ✅ **Scalable** - Easy to add new scripts
|
|
||||||
|
|
||||||
### Documentation
|
|
||||||
|
|
||||||
- ✅ **Comprehensive README** - All scripts documented
|
|
||||||
- ✅ **Usage examples** - Clear examples for each script
|
|
||||||
- ✅ **Development workflow** - Guide for using scripts
|
|
||||||
- ✅ **Templates** - Easy to create new scripts
|
|
||||||
|
|
||||||
### Maintainability
|
|
||||||
|
|
||||||
- ✅ **Easier to maintain** - Clear organization
|
|
||||||
- ✅ **Easier to find** - Logical grouping
|
|
||||||
- ✅ **Easier to update** - Centralized documentation
|
|
||||||
- ✅ **Easier to extend** - Clear patterns
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Statistics
|
|
||||||
|
|
||||||
### Before
|
|
||||||
|
|
||||||
- **Total scripts**: 12
|
|
||||||
- **In framework root**: 12
|
|
||||||
- **Organized**: 0
|
|
||||||
- **Documented**: Minimal
|
|
||||||
|
|
||||||
### After
|
|
||||||
|
|
||||||
- **Total scripts**: 12 (same)
|
|
||||||
- **In framework root**: 0
|
|
||||||
- **Organized**: 12 (100%)
|
|
||||||
- **Documented**: Comprehensive (200+ lines)
|
|
||||||
|
|
||||||
### File Count
|
|
||||||
|
|
||||||
- **Debug scripts**: 4
|
|
||||||
- **Test scripts**: 6
|
|
||||||
- **Utility scripts**: 2
|
|
||||||
- **Documentation**: 1 (README.md)
|
|
||||||
- **Total**: 13 files (12 scripts + 1 doc)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Maintenance Guidelines
|
|
||||||
|
|
||||||
### Adding New Scripts
|
|
||||||
|
|
||||||
1. **Determine category**:
|
|
||||||
- Debug? → `scripts/debug/`
|
|
||||||
- Test? → `scripts/test/`
|
|
||||||
- Utility? → `scripts/utils/`
|
|
||||||
|
|
||||||
2. **Create script** in appropriate directory
|
|
||||||
|
|
||||||
3. **Update `scripts/README.md`**:
|
|
||||||
- Add to table
|
|
||||||
- Add usage example
|
|
||||||
|
|
||||||
4. **Test the script**:
|
|
||||||
```bash
|
|
||||||
npm run build
|
|
||||||
node scripts/debug/my-script.mjs
|
|
||||||
```
|
|
||||||
|
|
||||||
### Removing Obsolete Scripts
|
|
||||||
|
|
||||||
1. **Delete the script file**
|
|
||||||
|
|
||||||
2. **Update `scripts/README.md`**:
|
|
||||||
- Remove from table
|
|
||||||
- Remove usage example
|
|
||||||
|
|
||||||
3. **Check for references**:
|
|
||||||
```bash
|
|
||||||
rg "my-script" --type md
|
|
||||||
```
|
|
||||||
|
|
||||||
### Updating Scripts
|
|
||||||
|
|
||||||
1. **Make changes to script**
|
|
||||||
|
|
||||||
2. **Test changes**:
|
|
||||||
```bash
|
|
||||||
npm run build
|
|
||||||
node scripts/debug/my-script.mjs
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Update documentation** if usage changed
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Next Steps
|
|
||||||
|
|
||||||
### Immediate
|
|
||||||
|
|
||||||
- ✅ Scripts organized
|
|
||||||
- ✅ Documentation updated
|
|
||||||
- ✅ References updated
|
|
||||||
- ✅ README created
|
|
||||||
|
|
||||||
### Future Enhancements
|
|
||||||
|
|
||||||
1. **Add more debug scripts**
|
|
||||||
- Session comparison tool
|
|
||||||
- Event diff tool
|
|
||||||
- Performance profiler
|
|
||||||
|
|
||||||
2. **Add more test scripts**
|
|
||||||
- Integration test runner
|
|
||||||
- Performance benchmarks
|
|
||||||
- Stress tests
|
|
||||||
|
|
||||||
3. **Add more utilities**
|
|
||||||
- Test result analyzer
|
|
||||||
- Coverage reporter
|
|
||||||
- Cleanup utilities
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
✅ **12 scripts organized** into 3 categories
|
|
||||||
✅ **Framework root cleaned** (0 scripts remaining)
|
|
||||||
✅ **Comprehensive documentation** (200+ lines)
|
|
||||||
✅ **All references updated** (6 files)
|
|
||||||
✅ **Clear structure** for future additions
|
|
||||||
|
|
||||||
**Organization is now**:
|
|
||||||
- Clean and organized
|
|
||||||
- Well-documented
|
|
||||||
- Easy to navigate
|
|
||||||
- Easy to maintain
|
|
||||||
- Easy to extend
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Organization Completed**: 2025-11-26
|
|
||||||
**Scripts Organized**: 12
|
|
||||||
**Documentation Created**: 1 README (200+ lines)
|
|
||||||
**Files Updated**: 6
|
|
||||||
|
|
@ -1,417 +0,0 @@
|
||||||
# Agent Testing Guide - Agent-Agnostic Architecture
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
Our evaluation framework is designed to be **agent-agnostic**, making it easy to test multiple agents with the same infrastructure.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Architecture Layers
|
|
||||||
|
|
||||||
### **Layer 1: Framework (Agent-Agnostic)**
|
|
||||||
```
|
|
||||||
evals/framework/
|
|
||||||
├── src/
|
|
||||||
│ ├── sdk/ # Test runner (works with any agent)
|
|
||||||
│ ├── evaluators/ # Generic behavior checks
|
|
||||||
│ └── types/ # Shared types
|
|
||||||
```
|
|
||||||
|
|
||||||
**Purpose:** Shared infrastructure that works with **any agent**
|
|
||||||
|
|
||||||
**Key Components:**
|
|
||||||
- `TestRunner` - Executes tests for any agent
|
|
||||||
- `Evaluators` - Check generic behaviors (approval, context, tools)
|
|
||||||
- `EventStreamHandler` - Captures events from any agent
|
|
||||||
- `TestCaseSchema` - Universal test format
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### **Layer 2: Agent-Specific Tests**
|
|
||||||
```
|
|
||||||
evals/agents/
|
|
||||||
├── openagent/ # OpenAgent-specific tests
|
|
||||||
│ ├── tests/
|
|
||||||
│ └── docs/
|
|
||||||
├── opencoder/ # OpenCoder-specific tests (future)
|
|
||||||
│ ├── tests/
|
|
||||||
│ └── docs/
|
|
||||||
└── shared/ # Tests for ANY agent
|
|
||||||
└── tests/
|
|
||||||
```
|
|
||||||
|
|
||||||
**Purpose:** Organize tests by agent for easy management
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Directory Structure
|
|
||||||
|
|
||||||
```
|
|
||||||
evals/
|
|
||||||
├── framework/ # SHARED FRAMEWORK
|
|
||||||
│ ├── src/
|
|
||||||
│ │ ├── sdk/
|
|
||||||
│ │ │ ├── test-runner.ts # Reads 'agent' field from YAML
|
|
||||||
│ │ │ ├── client-manager.ts # Routes to correct agent
|
|
||||||
│ │ │ └── test-case-schema.ts # Universal schema
|
|
||||||
│ │ └── evaluators/
|
|
||||||
│ │ ├── approval-gate-evaluator.ts # Works for any agent
|
|
||||||
│ │ ├── context-loading-evaluator.ts # Works for any agent
|
|
||||||
│ │ └── tool-usage-evaluator.ts # Works for any agent
|
|
||||||
│ └── package.json
|
|
||||||
│
|
|
||||||
├── agents/
|
|
||||||
│ ├── openagent/ # OPENAGENT TESTS
|
|
||||||
│ │ ├── tests/
|
|
||||||
│ │ │ ├── developer/
|
|
||||||
│ │ │ │ ├── task-simple-001.yaml # agent: openagent
|
|
||||||
│ │ │ │ ├── ctx-code-001.yaml # agent: openagent
|
|
||||||
│ │ │ │ └── ctx-docs-001.yaml # agent: openagent
|
|
||||||
│ │ │ ├── business/
|
|
||||||
│ │ │ │ └── conv-simple-001.yaml # agent: openagent
|
|
||||||
│ │ │ └── edge-case/
|
|
||||||
│ │ │ └── fail-stop-001.yaml # agent: openagent
|
|
||||||
│ │ └── docs/
|
|
||||||
│ │ └── OPENAGENT_RULES.md # OpenAgent-specific rules
|
|
||||||
│ │
|
|
||||||
│ ├── opencoder/ # OPENCODER TESTS (future)
|
|
||||||
│ │ ├── tests/
|
|
||||||
│ │ │ ├── developer/
|
|
||||||
│ │ │ │ ├── refactor-code-001.yaml # agent: opencoder
|
|
||||||
│ │ │ │ └── optimize-perf-001.yaml # agent: opencoder
|
|
||||||
│ │ └── docs/
|
|
||||||
│ │ └── OPENCODER_RULES.md # OpenCoder-specific rules
|
|
||||||
│ │
|
|
||||||
│ └── shared/ # SHARED TESTS (any agent)
|
|
||||||
│ ├── tests/
|
|
||||||
│ │ └── common/
|
|
||||||
│ │ ├── approval-gate-basic.yaml # agent: ${AGENT}
|
|
||||||
│ │ └── tool-usage-basic.yaml # agent: ${AGENT}
|
|
||||||
│ └── README.md
|
|
||||||
│
|
|
||||||
└── README.md
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## How Agent Selection Works
|
|
||||||
|
|
||||||
### **1. Test Specifies Agent**
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
# openagent/tests/developer/task-simple-001.yaml
|
|
||||||
id: task-simple-001
|
|
||||||
name: Simple Bash Execution
|
|
||||||
agent: openagent # ← Specifies which agent to test
|
|
||||||
prompt: "Run npm install"
|
|
||||||
```
|
|
||||||
|
|
||||||
### **2. Test Runner Routes to Agent**
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// framework/src/sdk/test-runner.ts
|
|
||||||
async runTest(testCase: TestCase) {
|
|
||||||
// Get agent from test case
|
|
||||||
const agent = testCase.agent || 'openagent';
|
|
||||||
|
|
||||||
// Route to specified agent
|
|
||||||
const result = await this.clientManager.sendPrompt(
|
|
||||||
sessionId,
|
|
||||||
testCase.prompt,
|
|
||||||
{ agent } // ← SDK routes to correct agent
|
|
||||||
);
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### **3. Evaluators Check Generic Behaviors**
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// framework/src/evaluators/approval-gate-evaluator.ts
|
|
||||||
export class ApprovalGateEvaluator extends BaseEvaluator {
|
|
||||||
async evaluate(timeline: TimelineEvent[]) {
|
|
||||||
// Check if ANY agent asked for approval
|
|
||||||
// Works for openagent, opencoder, or any future agent
|
|
||||||
|
|
||||||
const approvalRequested = timeline.some(event =>
|
|
||||||
event.type === 'approval_request'
|
|
||||||
);
|
|
||||||
|
|
||||||
if (!approvalRequested) {
|
|
||||||
violations.push({
|
|
||||||
type: 'approval-gate-missing',
|
|
||||||
severity: 'error',
|
|
||||||
message: 'Agent executed without requesting approval'
|
|
||||||
});
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Running Tests Per Agent
|
|
||||||
|
|
||||||
### **Run All Tests for Specific Agent**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Run ALL OpenAgent tests
|
|
||||||
npm run eval:sdk -- --pattern="openagent/**/*.yaml"
|
|
||||||
|
|
||||||
# Run ALL OpenCoder tests
|
|
||||||
npm run eval:sdk -- --pattern="opencoder/**/*.yaml"
|
|
||||||
```
|
|
||||||
|
|
||||||
### **Run Specific Category**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Run OpenAgent developer tests
|
|
||||||
npm run eval:sdk -- --pattern="openagent/developer/*.yaml"
|
|
||||||
|
|
||||||
# Run OpenCoder developer tests
|
|
||||||
npm run eval:sdk -- --pattern="opencoder/developer/*.yaml"
|
|
||||||
```
|
|
||||||
|
|
||||||
### **Run Shared Tests for Different Agents**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Run shared tests for OpenAgent
|
|
||||||
npm run eval:sdk -- --pattern="shared/**/*.yaml" --agent=openagent
|
|
||||||
|
|
||||||
# Run shared tests for OpenCoder
|
|
||||||
npm run eval:sdk -- --pattern="shared/**/*.yaml" --agent=opencoder
|
|
||||||
```
|
|
||||||
|
|
||||||
### **Run Single Test**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Run specific test
|
|
||||||
npx tsx src/sdk/show-test-details.ts openagent/developer/task-simple-001.yaml
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Adding a New Agent
|
|
||||||
|
|
||||||
### **Step 1: Create Agent Directory**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mkdir -p evals/agents/my-new-agent/tests/{developer,business,edge-case}
|
|
||||||
mkdir -p evals/agents/my-new-agent/docs
|
|
||||||
```
|
|
||||||
|
|
||||||
### **Step 2: Create Agent Rules Document**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Document agent-specific rules
|
|
||||||
touch evals/agents/my-new-agent/docs/MY_NEW_AGENT_RULES.md
|
|
||||||
```
|
|
||||||
|
|
||||||
### **Step 3: Copy Shared Tests**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Copy shared tests as starting point
|
|
||||||
cp evals/agents/shared/tests/common/*.yaml \
|
|
||||||
evals/agents/my-new-agent/tests/developer/
|
|
||||||
|
|
||||||
# Update agent field
|
|
||||||
sed -i 's/agent: openagent/agent: my-new-agent/g' \
|
|
||||||
evals/agents/my-new-agent/tests/developer/*.yaml
|
|
||||||
```
|
|
||||||
|
|
||||||
### **Step 4: Add Agent-Specific Tests**
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
# my-new-agent/tests/developer/custom-test-001.yaml
|
|
||||||
id: custom-test-001
|
|
||||||
name: My New Agent Custom Test
|
|
||||||
agent: my-new-agent # ← Your new agent
|
|
||||||
prompt: "Agent-specific prompt"
|
|
||||||
|
|
||||||
behavior:
|
|
||||||
mustUseTools: [bash]
|
|
||||||
requiresApproval: true
|
|
||||||
|
|
||||||
expectedViolations:
|
|
||||||
- rule: approval-gate
|
|
||||||
shouldViolate: false
|
|
||||||
```
|
|
||||||
|
|
||||||
### **Step 5: Run Tests**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
npm run eval:sdk -- --pattern="my-new-agent/**/*.yaml"
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Test Organization Best Practices
|
|
||||||
|
|
||||||
### **1. Agent-Specific Tests**
|
|
||||||
Put in `agents/{agent}/tests/`
|
|
||||||
|
|
||||||
**When to use:**
|
|
||||||
- Tests specific to agent's unique features
|
|
||||||
- Tests for agent-specific rules
|
|
||||||
- Tests that won't work for other agents
|
|
||||||
|
|
||||||
**Example:**
|
|
||||||
```yaml
|
|
||||||
# openagent/tests/developer/ctx-code-001.yaml
|
|
||||||
# OpenAgent-specific: Tests context loading from openagent.md
|
|
||||||
agent: openagent
|
|
||||||
behavior:
|
|
||||||
requiresContext: true # OpenAgent-specific rule
|
|
||||||
```
|
|
||||||
|
|
||||||
### **2. Shared Tests**
|
|
||||||
Put in `agents/shared/tests/common/`
|
|
||||||
|
|
||||||
**When to use:**
|
|
||||||
- Tests that work for ANY agent
|
|
||||||
- Tests for universal rules (approval, tool usage)
|
|
||||||
- Tests you want to run across multiple agents
|
|
||||||
|
|
||||||
**Example:**
|
|
||||||
```yaml
|
|
||||||
# shared/tests/common/approval-gate-basic.yaml
|
|
||||||
# Works for ANY agent
|
|
||||||
agent: openagent # Default, can be overridden
|
|
||||||
behavior:
|
|
||||||
requiresApproval: true # Universal rule
|
|
||||||
```
|
|
||||||
|
|
||||||
### **3. Category Organization**
|
|
||||||
|
|
||||||
```
|
|
||||||
tests/
|
|
||||||
├── developer/ # Developer workflow tests
|
|
||||||
├── business/ # Business/analysis tests
|
|
||||||
├── creative/ # Content creation tests
|
|
||||||
└── edge-case/ # Edge cases and error handling
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Evaluator Design (Agent-Agnostic)
|
|
||||||
|
|
||||||
### **Good: Generic Behavior Check**
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// ✅ Works for any agent
|
|
||||||
export class ApprovalGateEvaluator extends BaseEvaluator {
|
|
||||||
async evaluate(timeline: TimelineEvent[]) {
|
|
||||||
// Check generic behavior: did agent ask for approval?
|
|
||||||
const hasApproval = timeline.some(e => e.type === 'approval_request');
|
|
||||||
|
|
||||||
if (!hasApproval) {
|
|
||||||
violations.push({
|
|
||||||
type: 'approval-gate-missing',
|
|
||||||
message: 'Agent did not request approval'
|
|
||||||
});
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### **Bad: Agent-Specific Logic**
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// ❌ Hardcoded to specific agent
|
|
||||||
export class OpenAgentSpecificEvaluator extends BaseEvaluator {
|
|
||||||
async evaluate(timeline: TimelineEvent[]) {
|
|
||||||
// Don't do this - ties evaluator to specific agent
|
|
||||||
if (sessionInfo.agent === 'openagent') {
|
|
||||||
// OpenAgent-specific checks
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Benefits of Agent-Agnostic Design
|
|
||||||
|
|
||||||
### **1. Easy to Add New Agents**
|
|
||||||
- Copy shared tests
|
|
||||||
- Update `agent` field
|
|
||||||
- Add agent-specific tests
|
|
||||||
- Run tests
|
|
||||||
|
|
||||||
### **2. Consistent Behavior Across Agents**
|
|
||||||
- Same evaluators check all agents
|
|
||||||
- Same test format for all agents
|
|
||||||
- Easy to compare agent behaviors
|
|
||||||
|
|
||||||
### **3. Reduced Duplication**
|
|
||||||
- Shared tests written once
|
|
||||||
- Evaluators work for all agents
|
|
||||||
- Framework code reused
|
|
||||||
|
|
||||||
### **4. Easy Maintenance**
|
|
||||||
- Update evaluator once, affects all agents
|
|
||||||
- Update shared test once, affects all agents
|
|
||||||
- Clear separation of concerns
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Example: Testing Two Agents
|
|
||||||
|
|
||||||
### **OpenAgent Test**
|
|
||||||
```yaml
|
|
||||||
# openagent/tests/developer/create-file.yaml
|
|
||||||
id: openagent-create-file-001
|
|
||||||
agent: openagent
|
|
||||||
prompt: "Create hello.ts"
|
|
||||||
|
|
||||||
behavior:
|
|
||||||
requiresContext: true # OpenAgent loads code.md
|
|
||||||
```
|
|
||||||
|
|
||||||
### **OpenCoder Test**
|
|
||||||
```yaml
|
|
||||||
# opencoder/tests/developer/create-file.yaml
|
|
||||||
id: opencoder-create-file-001
|
|
||||||
agent: opencoder
|
|
||||||
prompt: "Create hello.ts"
|
|
||||||
|
|
||||||
behavior:
|
|
||||||
requiresContext: false # OpenCoder might not need context
|
|
||||||
```
|
|
||||||
|
|
||||||
### **Shared Test (Works for Both)**
|
|
||||||
```yaml
|
|
||||||
# shared/tests/common/create-file.yaml
|
|
||||||
id: shared-create-file-001
|
|
||||||
agent: openagent # Default
|
|
||||||
prompt: "Create hello.ts"
|
|
||||||
|
|
||||||
behavior:
|
|
||||||
requiresApproval: true # Both agents should ask
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
**Framework Layer:**
|
|
||||||
- ✅ Agent-agnostic test runner
|
|
||||||
- ✅ Generic evaluators
|
|
||||||
- ✅ Universal test schema
|
|
||||||
|
|
||||||
**Agent Layer:**
|
|
||||||
- ✅ Agent-specific tests in `agents/{agent}/`
|
|
||||||
- ✅ Shared tests in `agents/shared/`
|
|
||||||
- ✅ Agent-specific rules in `docs/`
|
|
||||||
|
|
||||||
**Benefits:**
|
|
||||||
- ✅ Easy to add new agents
|
|
||||||
- ✅ Consistent behavior validation
|
|
||||||
- ✅ Reduced duplication
|
|
||||||
- ✅ Clear organization
|
|
||||||
|
|
||||||
**To test a new agent:**
|
|
||||||
1. Create directory: `agents/my-agent/`
|
|
||||||
2. Copy shared tests
|
|
||||||
3. Update `agent` field
|
|
||||||
4. Add agent-specific tests
|
|
||||||
5. Run: `npm run eval:sdk -- --pattern="my-agent/**/*.yaml"`
|
|
||||||
|
|
@ -1,298 +0,0 @@
|
||||||
# Context Loading Test Coverage
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
This document describes the context loading tests created to verify OpenAgent correctly loads context files before responding to user queries and executing tasks.
|
|
||||||
|
|
||||||
**Test Location**: `evals/agents/openagent/tests/context-loading/`
|
|
||||||
|
|
||||||
**Total Tests**: 5 (3 simple, 2 complex multi-turn)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Test Results Summary
|
|
||||||
|
|
||||||
**Run Date**: 2025-11-26
|
|
||||||
**Pass Rate**: 3/5 (60%)
|
|
||||||
**Total Duration**: 430 seconds (~7 minutes)
|
|
||||||
|
|
||||||
| Test ID | Type | Status | Duration | Notes |
|
|
||||||
|---------|------|--------|----------|-------|
|
|
||||||
| ctx-simple-testing-approach | Simple | ✅ PASS | 35s | Loaded testing docs correctly |
|
|
||||||
| ctx-simple-documentation-format | Simple | ✅ PASS | 19s | Loaded docs.md correctly |
|
|
||||||
| ctx-simple-coding-standards | Simple | ✅ PASS | 20s | Loaded code.md correctly |
|
|
||||||
| ctx-multi-standards-to-docs | Complex | ❌ FAIL | 109s | No context loaded before execution |
|
|
||||||
| ctx-multi-error-handling-to-tests | Complex | ❌ FAIL | 246s | Timeout on prompt 4 |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Test Descriptions
|
|
||||||
|
|
||||||
### Simple Tests (Read-Only)
|
|
||||||
|
|
||||||
#### 1. `ctx-simple-coding-standards.yaml`
|
|
||||||
**Prompt**: "What are our coding standards for this project?"
|
|
||||||
|
|
||||||
**Expected Behavior**:
|
|
||||||
- Load `code.md` or `standards.md` before responding
|
|
||||||
- Reference project-specific standards
|
|
||||||
|
|
||||||
**Result**: ✅ **PASSED**
|
|
||||||
- Agent loaded `.opencode/context/core/standards/code.md`
|
|
||||||
- 1 read operation performed
|
|
||||||
- No violations detected
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
#### 2. `ctx-simple-documentation-format.yaml`
|
|
||||||
**Prompt**: "What format should I use for documentation in this project?"
|
|
||||||
|
|
||||||
**Expected Behavior**:
|
|
||||||
- Load `docs.md` or `documentation.md` before responding
|
|
||||||
- Reference project-specific documentation standards
|
|
||||||
|
|
||||||
**Result**: ✅ **PASSED**
|
|
||||||
- Agent loaded `.opencode/context/core/standards/docs.md`
|
|
||||||
- 1 read operation performed
|
|
||||||
- No violations detected
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
#### 3. `ctx-simple-testing-approach.yaml`
|
|
||||||
**Prompt**: "What's our testing strategy for this project?"
|
|
||||||
|
|
||||||
**Expected Behavior**:
|
|
||||||
- Load `tests.md` or `testing.md` before responding
|
|
||||||
- Reference project-specific testing standards
|
|
||||||
|
|
||||||
**Result**: ✅ **PASSED**
|
|
||||||
- Agent loaded multiple testing-related files:
|
|
||||||
- `evals/HOW_TESTS_WORK.md`
|
|
||||||
- `evals/README.md`
|
|
||||||
- `evals/TESTING_CONFIDENCE.md`
|
|
||||||
- `evals/agents/AGENT_TESTING_GUIDE.md`
|
|
||||||
- 4 read operations performed
|
|
||||||
- No violations detected
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Complex Tests (Multi-Turn with File Creation)
|
|
||||||
|
|
||||||
#### 4. `ctx-multi-standards-to-docs.yaml`
|
|
||||||
**Scenario**: Standards question → Documentation request → Format question
|
|
||||||
|
|
||||||
**Turn 1**: "What are our coding standards?"
|
|
||||||
- Expected: Load `standards.md` or `code.md`
|
|
||||||
|
|
||||||
**Turn 2**: "Can you create documentation about these standards in evals/test_tmp/coding-standards-doc.md?"
|
|
||||||
- Expected: Load `docs.md` (documentation format)
|
|
||||||
- Expected: Write file to `evals/test_tmp/`
|
|
||||||
|
|
||||||
**Turn 3**: "What will the documentation structure look like?"
|
|
||||||
- Expected: Reference both standards and docs context
|
|
||||||
|
|
||||||
**Result**: ❌ **FAILED**
|
|
||||||
- Agent loaded context files correctly:
|
|
||||||
- `.opencode/context/core/standards/code.md` (2x)
|
|
||||||
- `.opencode/context/core/standards/docs.md` (1x)
|
|
||||||
- Agent wrote file successfully
|
|
||||||
- **Violation**: "No context loaded before execution" (warning)
|
|
||||||
- **Issue**: Context loading evaluator flagged timing issue
|
|
||||||
|
|
||||||
**Files Created**: `evals/test_tmp/coding-standards-doc.md` (cleaned up after test)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
#### 5. `ctx-multi-error-handling-to-tests.yaml`
|
|
||||||
**Scenario**: Error handling question → Test request → Coverage policy
|
|
||||||
|
|
||||||
**Turn 1**: "How should we handle errors in this project?"
|
|
||||||
- Expected: Load `standards.md` or `processes.md`
|
|
||||||
|
|
||||||
**Turn 2**: "Can you write tests for error handling in evals/test_tmp/error-handling.test.ts?"
|
|
||||||
- Expected: Load `tests.md` (testing standards)
|
|
||||||
- Expected: Write test file to `evals/test_tmp/`
|
|
||||||
|
|
||||||
**Turn 3**: "What's our test coverage policy?"
|
|
||||||
- Expected: Reference test-related context
|
|
||||||
|
|
||||||
**Result**: ❌ **FAILED**
|
|
||||||
- **Error**: "Prompt 4 execution timed out"
|
|
||||||
- Test exceeded 180-second timeout
|
|
||||||
- Likely due to complex multi-turn conversation with file creation
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Cleanup Verification
|
|
||||||
|
|
||||||
✅ **Cleanup System Working Correctly**
|
|
||||||
|
|
||||||
**Before Tests**:
|
|
||||||
- Cleaned up 1 file from previous runs
|
|
||||||
|
|
||||||
**After Tests**:
|
|
||||||
- Cleaned up 2 files created during tests
|
|
||||||
- `test_tmp/` contains only:
|
|
||||||
- `.gitignore`
|
|
||||||
- `README.md`
|
|
||||||
|
|
||||||
**Cleanup Logic**: `evals/framework/src/sdk/run-sdk-tests.ts`
|
|
||||||
- Runs before test execution
|
|
||||||
- Runs after test execution
|
|
||||||
- Preserves only `.gitignore` and `README.md`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Key Findings
|
|
||||||
|
|
||||||
### ✅ Positive Results
|
|
||||||
|
|
||||||
1. **Simple Context Loading Works**: All 3 simple tests passed
|
|
||||||
- Agent correctly identifies and loads relevant context files
|
|
||||||
- Agent reads context BEFORE responding
|
|
||||||
- No violations in simple scenarios
|
|
||||||
|
|
||||||
2. **Cleanup System Reliable**:
|
|
||||||
- Files created during tests are properly cleaned up
|
|
||||||
- No test artifacts left in project root
|
|
||||||
- `test_tmp/` directory isolation working
|
|
||||||
|
|
||||||
3. **Context File Discovery**:
|
|
||||||
- Agent successfully finds context files in `.opencode/context/core/standards/`
|
|
||||||
- Agent loads multiple relevant files when appropriate
|
|
||||||
|
|
||||||
### ⚠️ Issues Identified
|
|
||||||
|
|
||||||
1. **Multi-Turn Context Loading**:
|
|
||||||
- Complex multi-turn tests show timing issues
|
|
||||||
- Context loading evaluator flagging warnings even when files are loaded
|
|
||||||
- May need to adjust evaluator logic for multi-turn scenarios
|
|
||||||
|
|
||||||
2. **Timeout on Complex Tests**:
|
|
||||||
- 180-second timeout insufficient for some multi-turn tests
|
|
||||||
- Test 5 timed out on prompt 4
|
|
||||||
- May need to increase timeout or simplify test scenarios
|
|
||||||
|
|
||||||
3. **False Positive Warning**:
|
|
||||||
- Test 4 loaded context correctly but still got "no-context-loaded" warning
|
|
||||||
- Evaluator may not be detecting context loads in multi-turn conversations
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Recommendations
|
|
||||||
|
|
||||||
### Immediate Actions
|
|
||||||
|
|
||||||
1. **Increase Timeout for Complex Tests**
|
|
||||||
- Change from 180s to 300s (5 minutes)
|
|
||||||
- Add timeout configuration per test
|
|
||||||
|
|
||||||
2. **Fix Context Loading Evaluator**
|
|
||||||
- Review timing detection logic for multi-turn tests
|
|
||||||
- Ensure evaluator tracks context loads across all prompts
|
|
||||||
|
|
||||||
3. **Simplify Complex Tests**
|
|
||||||
- Reduce number of turns in multi-turn tests
|
|
||||||
- Focus on specific context loading scenarios
|
|
||||||
|
|
||||||
### Future Enhancements
|
|
||||||
|
|
||||||
1. **Add More Edge Cases**
|
|
||||||
- Test context loading with missing files
|
|
||||||
- Test context loading with multiple context directories
|
|
||||||
- Test context loading with file attachments
|
|
||||||
|
|
||||||
2. **Add Performance Metrics**
|
|
||||||
- Track time between context load and execution
|
|
||||||
- Measure context file read performance
|
|
||||||
- Monitor API rate limits
|
|
||||||
|
|
||||||
3. **Batch Test Execution**
|
|
||||||
- Run tests in smaller batches to avoid API timeouts
|
|
||||||
- Add retry logic for transient failures
|
|
||||||
- Implement test result caching
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Running These Tests
|
|
||||||
|
|
||||||
### Run All Context Loading Tests
|
|
||||||
```bash
|
|
||||||
cd evals/framework
|
|
||||||
npm run eval:sdk -- --agent=openagent --pattern="context-loading/*.yaml"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Run Individual Test
|
|
||||||
```bash
|
|
||||||
npm run eval:sdk -- --agent=openagent --pattern="context-loading/ctx-simple-coding-standards.yaml"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Run with Debug Output
|
|
||||||
```bash
|
|
||||||
npm run eval:sdk -- --agent=openagent --pattern="context-loading/*.yaml" --debug
|
|
||||||
```
|
|
||||||
|
|
||||||
### View Results Dashboard
|
|
||||||
```bash
|
|
||||||
cd ../results
|
|
||||||
./serve.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Test File Structure
|
|
||||||
|
|
||||||
Each test follows this structure:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
id: test-id
|
|
||||||
name: "Test Name"
|
|
||||||
description: |
|
|
||||||
Detailed description of what the test validates
|
|
||||||
|
|
||||||
category: developer
|
|
||||||
agent: openagent
|
|
||||||
model: anthropic/claude-sonnet-4-5
|
|
||||||
|
|
||||||
# Single prompt OR multi-turn prompts
|
|
||||||
prompt: "Single prompt text"
|
|
||||||
# OR
|
|
||||||
prompts:
|
|
||||||
- text: "First prompt"
|
|
||||||
expectContext: true
|
|
||||||
contextFile: "standards.md"
|
|
||||||
- text: "approve"
|
|
||||||
delayMs: 2000
|
|
||||||
|
|
||||||
# Expected behavior
|
|
||||||
behavior:
|
|
||||||
mustUseTools: [read, write]
|
|
||||||
requiresContext: true
|
|
||||||
minToolCalls: 1
|
|
||||||
|
|
||||||
# Expected violations
|
|
||||||
expectedViolations:
|
|
||||||
- rule: context-loading
|
|
||||||
shouldViolate: false
|
|
||||||
severity: error
|
|
||||||
|
|
||||||
# Approval strategy
|
|
||||||
approvalStrategy:
|
|
||||||
type: auto-approve
|
|
||||||
|
|
||||||
timeout: 60000
|
|
||||||
|
|
||||||
tags:
|
|
||||||
- context-loading
|
|
||||||
- simple-test
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Maintenance
|
|
||||||
|
|
||||||
**Last Updated**: 2025-11-26
|
|
||||||
**Test Framework Version**: 0.1.0
|
|
||||||
**OpenAgent Version**: Latest
|
|
||||||
|
|
||||||
**Next Review**: After fixing context loading evaluator timing logic
|
|
||||||
|
|
@ -1,256 +0,0 @@
|
||||||
# Context Loading Tests - Implementation Summary
|
|
||||||
|
|
||||||
**Date**: 2025-11-26
|
|
||||||
**Status**: ✅ **COMPLETE - ALL TESTS PASSING (5/5)**
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## What We Built
|
|
||||||
|
|
||||||
### 1. **5 Context Loading Tests** ✅
|
|
||||||
Created comprehensive test suite to verify OpenAgent loads context files correctly:
|
|
||||||
|
|
||||||
**Simple Tests (3)** - Single prompt, read-only
|
|
||||||
- `ctx-simple-coding-standards.yaml` - Coding standards query
|
|
||||||
- `ctx-simple-documentation-format.yaml` - Documentation format query
|
|
||||||
- `ctx-simple-testing-approach.yaml` - Testing strategy query
|
|
||||||
|
|
||||||
**Complex Tests (2)** - Multi-turn with file creation
|
|
||||||
- `ctx-multi-standards-to-docs.yaml` - Standards → Documentation creation
|
|
||||||
- `ctx-multi-error-handling-to-tests.yaml` - Error handling → Test creation
|
|
||||||
|
|
||||||
### 2. **Smart Timeout System** ✅
|
|
||||||
Implemented intelligent timeout handling for multi-turn tests:
|
|
||||||
- **Activity monitoring**: Checks if events are still streaming
|
|
||||||
- **Base timeout**: 300s (5 minutes) of inactivity triggers timeout
|
|
||||||
- **Absolute max**: 600s (10 minutes) hard limit
|
|
||||||
- **Prevents false timeouts**: Extends timeout while agent is active
|
|
||||||
|
|
||||||
**Code**: `evals/framework/src/sdk/test-runner.ts` - `withSmartTimeout()` method
|
|
||||||
|
|
||||||
### 3. **Fixed Context Loading Evaluator** ✅
|
|
||||||
Corrected evaluator to properly detect context files in multi-turn sessions:
|
|
||||||
|
|
||||||
**Issues Fixed**:
|
|
||||||
- ❌ **Before**: File paths extracted from wrong location (`tool.data.input.filePath`)
|
|
||||||
- ✅ **After**: Correctly extracts from `tool.data.state.input.filePath`
|
|
||||||
- ❌ **Before**: Only checked context before FIRST execution
|
|
||||||
- ✅ **After**: Checks context for ALL executions requiring it
|
|
||||||
- ❌ **Before**: False positives on multi-turn tests
|
|
||||||
- ✅ **After**: Properly tracks context across multiple prompts
|
|
||||||
|
|
||||||
**Code**: `evals/framework/src/evaluators/context-loading-evaluator.ts`
|
|
||||||
|
|
||||||
### 4. **Batch Test Runner** ✅
|
|
||||||
Created helper script for running tests in controlled batches:
|
|
||||||
- Configurable batch size (default: 3 tests)
|
|
||||||
- Configurable delay between batches (default: 10s)
|
|
||||||
- Prevents API rate limits
|
|
||||||
- Better resource management
|
|
||||||
|
|
||||||
**Script**: `evals/framewor./scripts/utils/run-tests-batch.sh`
|
|
||||||
|
|
||||||
**Usage**:
|
|
||||||
```bash
|
|
||||||
cd evals/framework
|
|
||||||
./scripts/utils/run-tests-batch.sh openagent 3 10
|
|
||||||
```
|
|
||||||
|
|
||||||
### 5. **Cleanup System Verified** ✅
|
|
||||||
Confirmed automatic cleanup working correctly:
|
|
||||||
- Cleans `test_tmp/` before tests
|
|
||||||
- Cleans `test_tmp/` after tests
|
|
||||||
- Preserves only `.gitignore` and `README.md`
|
|
||||||
- No test artifacts left behind
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Test Results
|
|
||||||
|
|
||||||
### Final Run: 100% Pass Rate 🎉
|
|
||||||
|
|
||||||
| Test | Type | Duration | Status | Context Files Loaded |
|
|
||||||
|------|------|----------|--------|---------------------|
|
|
||||||
| ctx-simple-testing-approach | Simple | 38s | ✅ PASS | 4 files (README, HOW_TESTS_WORK, etc.) |
|
|
||||||
| ctx-simple-documentation-format | Simple | 26s | ✅ PASS | docs.md |
|
|
||||||
| ctx-simple-coding-standards | Simple | 21s | ✅ PASS | code.md |
|
|
||||||
| ctx-multi-standards-to-docs | Complex | 116s | ✅ PASS | code.md, docs.md (44s before execution) |
|
|
||||||
| ctx-multi-error-handling-to-tests | Complex | 148s | ✅ PASS | code.md, tests.md (58s before execution) |
|
|
||||||
|
|
||||||
**Total Duration**: 349 seconds (~6 minutes)
|
|
||||||
**Pass Rate**: 5/5 (100%)
|
|
||||||
**Violations**: 0
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Key Findings
|
|
||||||
|
|
||||||
### ✅ **OpenAgent Context Loading Works Correctly**
|
|
||||||
|
|
||||||
1. **Simple queries**: Agent loads appropriate context files before responding
|
|
||||||
2. **Multi-turn conversations**: Agent loads context for each execution phase
|
|
||||||
3. **File creation**: Agent loads both standards AND format context before writing
|
|
||||||
4. **Timing**: Context loaded 44-58 seconds before execution (plenty of time)
|
|
||||||
|
|
||||||
### ✅ **Test Infrastructure is Solid**
|
|
||||||
|
|
||||||
1. **Same session tracking**: Multi-turn tests use single session (verified)
|
|
||||||
2. **Smart timeout**: Prevents false timeouts while catching real hangs
|
|
||||||
3. **Cleanup**: No test artifacts left behind
|
|
||||||
4. **Evaluators**: Accurately detect context loading behavior
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Technical Details
|
|
||||||
|
|
||||||
### Session Tracking (Multi-Turn)
|
|
||||||
```typescript
|
|
||||||
// Single session created once
|
|
||||||
const session = await this.client.createSession({ title: testCase.name });
|
|
||||||
sessionId = session.id;
|
|
||||||
|
|
||||||
// All prompts use SAME session
|
|
||||||
for (let i = 0; i < testCase.prompts.length; i++) {
|
|
||||||
await this.client.sendPrompt(sessionId, { text: msg.text, ... });
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Smart Timeout Logic
|
|
||||||
```typescript
|
|
||||||
// Base timeout: 300s of inactivity
|
|
||||||
// Max timeout: 600s absolute
|
|
||||||
await this.withSmartTimeout(
|
|
||||||
promptPromise,
|
|
||||||
300000, // 5 min activity timeout
|
|
||||||
600000, // 10 min absolute max
|
|
||||||
`Prompt ${i + 1} execution timed out`
|
|
||||||
);
|
|
||||||
```
|
|
||||||
|
|
||||||
### Context File Detection
|
|
||||||
```typescript
|
|
||||||
// Fixed file path extraction
|
|
||||||
const filePath = tool.data?.state?.input?.filePath || // ✅ NEW
|
|
||||||
tool.data?.state?.input?.path ||
|
|
||||||
tool.data?.input?.filePath || // Old fallback
|
|
||||||
tool.data?.input?.path;
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Files Modified
|
|
||||||
|
|
||||||
### New Files Created
|
|
||||||
```
|
|
||||||
evals/agents/openagent/tests/context-loading/
|
|
||||||
├── ctx-simple-coding-standards.yaml
|
|
||||||
├── ctx-simple-documentation-format.yaml
|
|
||||||
├── ctx-simple-testing-approach.yaml
|
|
||||||
├── ctx-multi-standards-to-docs.yaml
|
|
||||||
└── ctx-multi-error-handling-to-tests.yaml
|
|
||||||
|
|
||||||
evals/agents/openagent/
|
|
||||||
├── CONTEXT_LOADING_COVERAGE.md
|
|
||||||
└── IMPLEMENTATION_SUMMARY.md (this file)
|
|
||||||
|
|
||||||
evals/framework/
|
|
||||||
└── scripts/
|
|
||||||
```
|
|
||||||
|
|
||||||
### Files Modified
|
|
||||||
```
|
|
||||||
evals/framework/src/sdk/test-runner.ts
|
|
||||||
- Added withSmartTimeout() method
|
|
||||||
- Updated multi-turn test execution to use smart timeout
|
|
||||||
|
|
||||||
evals/framework/src/evaluators/context-loading-evaluator.ts
|
|
||||||
- Fixed file path extraction (tool.data.state.input.filePath)
|
|
||||||
- Added multi-turn execution checking
|
|
||||||
- Improved violation detection
|
|
||||||
|
|
||||||
evals/agents/openagent/tests/context-loading/*.yaml
|
|
||||||
- Increased timeout from 180s to 300s for complex tests
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Recommendations Completed
|
|
||||||
|
|
||||||
### ✅ Recommendation 1: Fix Timeout Issue
|
|
||||||
- **Status**: COMPLETE
|
|
||||||
- **Solution**: Implemented smart timeout with activity monitoring
|
|
||||||
- **Result**: No more false timeouts, complex tests complete successfully
|
|
||||||
|
|
||||||
### ✅ Recommendation 2: Fix Context Loading Evaluator
|
|
||||||
- **Status**: COMPLETE
|
|
||||||
- **Solution**: Fixed file path extraction and multi-turn tracking
|
|
||||||
- **Result**: Evaluator correctly detects context loading in all scenarios
|
|
||||||
|
|
||||||
### ✅ Recommendation 3: Batch Test Execution
|
|
||||||
- **Status**: COMPLETE
|
|
||||||
- **Solution**: Created `run-tests-batch.sh` script
|
|
||||||
- **Result**: Can run tests in controlled batches with delays
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## How to Use
|
|
||||||
|
|
||||||
### Run All Context Loading Tests
|
|
||||||
```bash
|
|
||||||
cd evals/framework
|
|
||||||
npm run eval:sdk -- --agent=openagent --pattern="context-loading/*.yaml"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Run Single Test
|
|
||||||
```bash
|
|
||||||
npm run eval:sdk -- --agent=openagent --pattern="context-loading/ctx-simple-coding-standards.yaml"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Run in Batches (Avoid API Limits)
|
|
||||||
```bash
|
|
||||||
./scripts/utils/run-tests-batch.sh openagent 3 10
|
|
||||||
# Args: agent, batch_size, delay_seconds
|
|
||||||
```
|
|
||||||
|
|
||||||
### View Results Dashboard
|
|
||||||
```bash
|
|
||||||
cd ../results
|
|
||||||
./serve.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Next Steps (Optional Enhancements)
|
|
||||||
|
|
||||||
1. **Add More Edge Cases**
|
|
||||||
- Test with missing context files
|
|
||||||
- Test with multiple context directories
|
|
||||||
- Test with file attachments
|
|
||||||
|
|
||||||
2. **Performance Metrics**
|
|
||||||
- Track context load time vs execution time
|
|
||||||
- Measure API response times
|
|
||||||
- Monitor rate limit usage
|
|
||||||
|
|
||||||
3. **Test Coverage Expansion**
|
|
||||||
- Add tests for other agent behaviors
|
|
||||||
- Test delegation scenarios
|
|
||||||
- Test error handling paths
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Conclusion
|
|
||||||
|
|
||||||
✅ **All objectives achieved**
|
|
||||||
✅ **100% test pass rate**
|
|
||||||
✅ **OpenAgent context loading verified working correctly**
|
|
||||||
✅ **Test infrastructure improved and reliable**
|
|
||||||
✅ **Documentation complete**
|
|
||||||
|
|
||||||
The context loading test suite is production-ready and provides comprehensive coverage of OpenAgent's context file loading behavior across both simple and complex multi-turn scenarios.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Maintained by**: OpenCode Agents Team
|
|
||||||
**Last Updated**: 2025-11-26
|
|
||||||
**Test Framework Version**: 0.1.0
|
|
||||||
|
|
@ -1,41 +0,0 @@
|
||||||
# Opencoder Agent Tests
|
|
||||||
|
|
||||||
Tests for the `opencoder` agent - a development-focused agent that executes code tasks directly.
|
|
||||||
|
|
||||||
## Agent Characteristics
|
|
||||||
|
|
||||||
- **Mode**: Primary development agent
|
|
||||||
- **Behavior**: Executes tools directly without text-based approval workflow
|
|
||||||
- **Best for**: Code implementation, bash commands, file operations
|
|
||||||
- **Approval**: Uses tool permission system (auto-approve in tests)
|
|
||||||
|
|
||||||
## Test Categories
|
|
||||||
|
|
||||||
### Developer Tests (`tests/developer/`)
|
|
||||||
- Bash command execution
|
|
||||||
- File operations
|
|
||||||
- Code implementation tasks
|
|
||||||
|
|
||||||
### Business Tests (`tests/business/`)
|
|
||||||
- Data analysis tasks
|
|
||||||
- Report generation
|
|
||||||
|
|
||||||
### Edge Cases (`tests/edge-case/`)
|
|
||||||
- Error handling
|
|
||||||
- Permission boundaries
|
|
||||||
|
|
||||||
## Running Tests
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd evals/framework
|
|
||||||
npx tsx src/sdk/run-sdk-tests.ts --agent opencoder
|
|
||||||
```
|
|
||||||
|
|
||||||
## Key Differences from OpenAgent
|
|
||||||
|
|
||||||
| Feature | Opencoder | OpenAgent |
|
|
||||||
|---------|-----------|-----------|
|
|
||||||
| Approval | Tool permission system | Text-based + tool permission |
|
|
||||||
| Workflow | Direct execution | Analyze→Approve→Execute→Validate |
|
|
||||||
| Context Loading | On-demand | Mandatory before execution |
|
|
||||||
| Best for | Simple tasks | Complex workflows |
|
|
||||||
|
|
@ -1,74 +0,0 @@
|
||||||
# Shared Test Cases
|
|
||||||
|
|
||||||
Tests in this directory are **agent-agnostic** and can be used to test **any agent** that follows the same core rules.
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
|
|
||||||
Shared tests validate **universal behaviors** that all agents should follow:
|
|
||||||
- Approval gate enforcement
|
|
||||||
- Tool usage patterns
|
|
||||||
- Basic workflow compliance
|
|
||||||
- Error handling
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
|
|
||||||
### Run Shared Tests for OpenAgent
|
|
||||||
```bash
|
|
||||||
npm run eval:sdk -- --pattern="shared/**/*.yaml" --agent=openagent
|
|
||||||
```
|
|
||||||
|
|
||||||
### Run Shared Tests for OpenCoder
|
|
||||||
```bash
|
|
||||||
npm run eval:sdk -- --pattern="shared/**/*.yaml" --agent=opencoder
|
|
||||||
```
|
|
||||||
|
|
||||||
### Override Agent in Test File
|
|
||||||
```yaml
|
|
||||||
# In the YAML file
|
|
||||||
agent: openagent # Change to opencoder, or any other agent
|
|
||||||
```
|
|
||||||
|
|
||||||
## Test Categories
|
|
||||||
|
|
||||||
### `common/` - Universal Rules
|
|
||||||
Tests that apply to **all agents**:
|
|
||||||
- `approval-gate-basic.yaml` - Basic approval enforcement
|
|
||||||
- `tool-usage-basic.yaml` - Basic tool selection (future)
|
|
||||||
- `error-handling-basic.yaml` - Basic error handling (future)
|
|
||||||
|
|
||||||
## Adding New Shared Tests
|
|
||||||
|
|
||||||
1. Create test in `shared/tests/common/`
|
|
||||||
2. Use generic prompts (not agent-specific)
|
|
||||||
3. Test universal behaviors only
|
|
||||||
4. Tag with `shared-test` and `agent-agnostic`
|
|
||||||
5. Document which agents it applies to
|
|
||||||
|
|
||||||
## Example
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
id: shared-example-001
|
|
||||||
name: Example Shared Test
|
|
||||||
category: edge-case
|
|
||||||
agent: openagent # Default, can be overridden
|
|
||||||
|
|
||||||
prompt: "Generic prompt that works for any agent"
|
|
||||||
|
|
||||||
behavior:
|
|
||||||
requiresApproval: true # Universal rule
|
|
||||||
|
|
||||||
expectedViolations:
|
|
||||||
- rule: approval-gate
|
|
||||||
shouldViolate: false
|
|
||||||
|
|
||||||
tags:
|
|
||||||
- shared-test
|
|
||||||
- agent-agnostic
|
|
||||||
```
|
|
||||||
|
|
||||||
## Benefits
|
|
||||||
|
|
||||||
1. **Reduce Duplication** - Write once, test multiple agents
|
|
||||||
2. **Consistency** - Same tests ensure consistent behavior
|
|
||||||
3. **Easy Comparison** - Compare agent behaviors side-by-side
|
|
||||||
4. **Faster Onboarding** - New agents inherit core test suite
|
|
||||||
|
|
@ -1,195 +0,0 @@
|
||||||
# Framework Scripts
|
|
||||||
|
|
||||||
Utility scripts for debugging, testing, and development.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Directory Structure
|
|
||||||
|
|
||||||
```
|
|
||||||
scripts/
|
|
||||||
├── debug/ # Debugging scripts for sessions and events
|
|
||||||
├── test/ # Test scripts for framework development
|
|
||||||
├── utils/ # Utility scripts (batch runner, etc.)
|
|
||||||
└── README.md # This file
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Debug Scripts (`debug/`)
|
|
||||||
|
|
||||||
Scripts for debugging sessions, events, and agent behavior.
|
|
||||||
|
|
||||||
| Script | Purpose | Usage |
|
|
||||||
|--------|---------|-------|
|
|
||||||
| `debug-session.mjs` | Debug session data and timeline | `node scripts/debug/debug-session.mjs <session-id>` |
|
|
||||||
| `debug-session.ts` | TypeScript version of session debugger | `npx tsx scripts/debug/debug-session.ts <session-id>` |
|
|
||||||
| `debug-claude-session.mjs` | Debug Claude-specific sessions | `node scripts/debug/debug-claude-session.mjs <session-id>` |
|
|
||||||
| `inspect-session.mjs` | Inspect most recent session events | `node scripts/debug/inspect-session.mjs` |
|
|
||||||
|
|
||||||
### Examples
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Debug a specific session
|
|
||||||
node scripts/debug/debug-session.mjs ses_abc123
|
|
||||||
|
|
||||||
# Inspect latest session
|
|
||||||
node scripts/debug/inspect-session.mjs
|
|
||||||
|
|
||||||
# Debug with TypeScript
|
|
||||||
npx tsx scripts/debug/debug-session.ts ses_abc123
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Test Scripts (`test/`)
|
|
||||||
|
|
||||||
Scripts for testing framework components during development.
|
|
||||||
|
|
||||||
| Script | Purpose | Usage |
|
|
||||||
|--------|---------|-------|
|
|
||||||
| `test-agent-direct.ts` | Direct agent execution test | `npx tsx scripts/test/test-agent-direct.ts` |
|
|
||||||
| `test-event-inspector.js` | Test event capture system | `node scripts/test/test-event-inspector.js` |
|
|
||||||
| `test-session-reader.mjs` | Test session reader | `node scripts/test/test-session-reader.mjs` |
|
|
||||||
| `test-simplified-approach.mjs` | Test simplified test approach | `node scripts/test/test-simplified-approach.mjs` |
|
|
||||||
| `test-timeline.ts` | Test timeline builder | `npx tsx scripts/test/test-timeline.ts` |
|
|
||||||
| `verify-timeline.ts` | Verify timeline accuracy | `npx tsx scripts/test/verify-timeline.ts` |
|
|
||||||
|
|
||||||
### Examples
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Test agent execution
|
|
||||||
npx tsx scripts/test/test-agent-direct.ts
|
|
||||||
|
|
||||||
# Test event capture
|
|
||||||
node scripts/test/test-event-inspector.js
|
|
||||||
|
|
||||||
# Verify timeline
|
|
||||||
npx tsx scripts/test/verify-timeline.ts
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Utility Scripts (`utils/`)
|
|
||||||
|
|
||||||
General utility scripts for running tests and managing the framework.
|
|
||||||
|
|
||||||
| Script | Purpose | Usage |
|
|
||||||
|--------|---------|-------|
|
|
||||||
| `run-tests-batch.sh` | Run tests in batches | `./scripts/utils/run-tests-batch.sh <agent> <batch-size> <delay>` |
|
|
||||||
| `check-agent.mjs` | Check agent availability | `node scripts/utils/check-agent.mjs` |
|
|
||||||
|
|
||||||
### Examples
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Run tests in batches of 3 with 10s delay
|
|
||||||
./scripts/utils/run-tests-batch.sh openagent 3 10
|
|
||||||
|
|
||||||
# Check if agent is available
|
|
||||||
node scripts/utils/check-agent.mjs
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Development Workflow
|
|
||||||
|
|
||||||
### Debugging a Failed Test
|
|
||||||
|
|
||||||
1. Run test with debug flag:
|
|
||||||
```bash
|
|
||||||
npm run eval:sdk -- --pattern="my-test.yaml" --debug
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Note the session ID from output
|
|
||||||
|
|
||||||
3. Inspect the session:
|
|
||||||
```bash
|
|
||||||
node scripts/debug/inspect-session.mjs
|
|
||||||
# or
|
|
||||||
node scripts/debug/debug-session.mjs <session-id>
|
|
||||||
```
|
|
||||||
|
|
||||||
4. Check timeline events:
|
|
||||||
```bash
|
|
||||||
npx tsx scripts/debug/debug-session.ts <session-id>
|
|
||||||
```
|
|
||||||
|
|
||||||
### Testing Framework Changes
|
|
||||||
|
|
||||||
1. Make changes to framework code
|
|
||||||
|
|
||||||
2. Build:
|
|
||||||
```bash
|
|
||||||
npm run build
|
|
||||||
```
|
|
||||||
|
|
||||||
3. Test specific component:
|
|
||||||
```bash
|
|
||||||
npx tsx scripts/test/test-timeline.ts
|
|
||||||
```
|
|
||||||
|
|
||||||
4. Run full test suite:
|
|
||||||
```bash
|
|
||||||
npm run eval:sdk
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Script Dependencies
|
|
||||||
|
|
||||||
All scripts require the framework to be built first:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
npm run build
|
|
||||||
```
|
|
||||||
|
|
||||||
Some scripts use:
|
|
||||||
- `@opencode-ai/sdk` - For SDK client
|
|
||||||
- `tsx` - For TypeScript execution
|
|
||||||
- Framework dist files - Built TypeScript output
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Adding New Scripts
|
|
||||||
|
|
||||||
### Debug Script Template
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
// scripts/debug/my-debug-script.mjs
|
|
||||||
import { SessionReader } from '../../dist/collector/session-reader.js';
|
|
||||||
import { createOpencodeClient } from '@opencode-ai/sdk';
|
|
||||||
|
|
||||||
const client = createOpencodeClient({
|
|
||||||
baseUrl: 'http://localhost:3721'
|
|
||||||
});
|
|
||||||
|
|
||||||
// Your debug logic here
|
|
||||||
```
|
|
||||||
|
|
||||||
### Test Script Template
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// scripts/test/my-test-script.ts
|
|
||||||
#!/usr/bin/env npx tsx
|
|
||||||
|
|
||||||
import { TestRunner } from '../../dist/sdk/test-runner.js';
|
|
||||||
|
|
||||||
async function runTest() {
|
|
||||||
// Your test logic here
|
|
||||||
}
|
|
||||||
|
|
||||||
runTest().catch(console.error);
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Maintenance
|
|
||||||
|
|
||||||
- **Keep scripts organized** - Put debug scripts in `debug/`, test scripts in `test/`
|
|
||||||
- **Update this README** - When adding new scripts
|
|
||||||
- **Remove obsolete scripts** - Delete scripts that are no longer needed
|
|
||||||
- **Document usage** - Add clear usage examples
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Last Updated**: 2025-11-26
|
|
||||||
|
|
@ -1,279 +0,0 @@
|
||||||
# 📊 Test Results Dashboard
|
|
||||||
|
|
||||||
Interactive dashboard for visualizing OpenCode agent test results.
|
|
||||||
|
|
||||||
## ⚡ Quick Reference
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Run tests
|
|
||||||
cd evals/framework && npm run eval:sdk -- --agent=opencoder
|
|
||||||
|
|
||||||
# View dashboard (auto-opens browser, auto-shuts down)
|
|
||||||
cd evals/results && ./serve.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
That's it! 🎉
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Quick Start
|
|
||||||
|
|
||||||
1. **Run Tests:**
|
|
||||||
```bash
|
|
||||||
cd evals/framework
|
|
||||||
npm run eval:sdk -- --agent=opencoder
|
|
||||||
npm run eval:sdk -- --agent=openagent
|
|
||||||
```
|
|
||||||
|
|
||||||
2. **View Dashboard:**
|
|
||||||
|
|
||||||
**Option A: One-Command Solution (Easiest)** ⭐
|
|
||||||
```bash
|
|
||||||
cd evals/results
|
|
||||||
./serve.sh
|
|
||||||
```
|
|
||||||
- Auto-opens browser
|
|
||||||
- Loads dashboard
|
|
||||||
- Auto-shuts down after 15 seconds
|
|
||||||
- Dashboard stays cached in browser!
|
|
||||||
|
|
||||||
**Custom timeout:**
|
|
||||||
```bash
|
|
||||||
./serve.sh 8000 30 # Port 8000, 30 second timeout
|
|
||||||
```
|
|
||||||
|
|
||||||
**Option B: Keep Server Running**
|
|
||||||
```bash
|
|
||||||
cd evals/results
|
|
||||||
python3 -m http.server 8000
|
|
||||||
```
|
|
||||||
Press Ctrl+C to stop manually
|
|
||||||
|
|
||||||
**Option C: Direct File Access**
|
|
||||||
```bash
|
|
||||||
open evals/results/index.html
|
|
||||||
```
|
|
||||||
⚠️ Note: Some browsers block loading JSON from local files. If you see an error, use Option A or B.
|
|
||||||
|
|
||||||
## Features
|
|
||||||
|
|
||||||
### 📈 Overview Stats
|
|
||||||
- **Total Tests** - Count across all agents
|
|
||||||
- **Pass Rate** - Percentage of passing tests
|
|
||||||
- **Failed Tests** - Number of failures
|
|
||||||
- **Avg Duration** - Average test execution time
|
|
||||||
|
|
||||||
### 📊 Trend Chart
|
|
||||||
- Visual representation of pass rate over time
|
|
||||||
- Shows last 30 days of test runs
|
|
||||||
- Helps identify regressions
|
|
||||||
|
|
||||||
### 🔍 Filters
|
|
||||||
- **Agent** - Filter by openagent, opencoder, etc.
|
|
||||||
- **Category** - Developer, business, creative, edge-case
|
|
||||||
- **Status** - All, passed only, or failed only
|
|
||||||
- **Time Range** - Latest, today, last 7 days, last 30 days
|
|
||||||
|
|
||||||
### 🔎 Search
|
|
||||||
- Real-time search across test IDs
|
|
||||||
- Case-insensitive matching
|
|
||||||
|
|
||||||
### 📋 Test Table
|
|
||||||
- **Sortable Columns** - Click any header to sort
|
|
||||||
- **Expandable Rows** - Click a row to see details
|
|
||||||
- **Violation Details** - See error messages and severity
|
|
||||||
|
|
||||||
### 🌙 Dark Mode
|
|
||||||
- Toggle with moon/sun icon in header
|
|
||||||
- Preference saved to localStorage
|
|
||||||
- Easy on the eyes for long sessions
|
|
||||||
|
|
||||||
### 📥 Export
|
|
||||||
- Export filtered results to CSV
|
|
||||||
- Includes all test metadata
|
|
||||||
- Perfect for external analysis
|
|
||||||
|
|
||||||
## File Structure
|
|
||||||
|
|
||||||
```
|
|
||||||
results/
|
|
||||||
├── index.html # Dashboard (open this)
|
|
||||||
├── serve.sh # Helper script to start HTTP server
|
|
||||||
├── latest.json # Most recent test run
|
|
||||||
├── history/
|
|
||||||
│ └── 2025-11/
|
|
||||||
│ ├── 26-115759-opencoder.json
|
|
||||||
│ └── 26-115850-openagent.json
|
|
||||||
├── .gitignore # Retention policy
|
|
||||||
└── README.md # This file
|
|
||||||
```
|
|
||||||
|
|
||||||
## JSON Format
|
|
||||||
|
|
||||||
Each result file contains:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"meta": {
|
|
||||||
"timestamp": "2025-11-26T11:59:36.365Z",
|
|
||||||
"agent": "openagent",
|
|
||||||
"model": "opencode/grok-code-fast",
|
|
||||||
"framework_version": "0.1.0",
|
|
||||||
"git_commit": "f872007"
|
|
||||||
},
|
|
||||||
"summary": {
|
|
||||||
"total": 8,
|
|
||||||
"passed": 6,
|
|
||||||
"failed": 2,
|
|
||||||
"duration_ms": 32450,
|
|
||||||
"pass_rate": 0.75
|
|
||||||
},
|
|
||||||
"by_category": {
|
|
||||||
"developer": { "passed": 5, "total": 6 },
|
|
||||||
"business": { "passed": 1, "total": 1 },
|
|
||||||
"edge-case": { "passed": 0, "total": 1 }
|
|
||||||
},
|
|
||||||
"tests": [
|
|
||||||
{
|
|
||||||
"id": "task-simple-001",
|
|
||||||
"category": "developer",
|
|
||||||
"passed": true,
|
|
||||||
"duration_ms": 4200,
|
|
||||||
"events": 23,
|
|
||||||
"approvals": 2,
|
|
||||||
"violations": {
|
|
||||||
"total": 0,
|
|
||||||
"errors": 0,
|
|
||||||
"warnings": 0
|
|
||||||
}
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Retention Policy
|
|
||||||
|
|
||||||
Results are automatically managed:
|
|
||||||
|
|
||||||
- ✅ **Latest Run** - Always kept (`latest.json`)
|
|
||||||
- ✅ **Current Month** - All results committed to git
|
|
||||||
- ✅ **Previous Month** - All results committed to git
|
|
||||||
- ❌ **Older than 60 days** - Kept locally, not committed
|
|
||||||
|
|
||||||
This keeps the repo size manageable while preserving recent history.
|
|
||||||
|
|
||||||
## Tips
|
|
||||||
|
|
||||||
### Quick View Workflow
|
|
||||||
The fastest way to view results:
|
|
||||||
```bash
|
|
||||||
cd evals/results && ./serve.sh
|
|
||||||
```
|
|
||||||
- ✅ Opens browser automatically
|
|
||||||
- ✅ Loads all data
|
|
||||||
- ✅ Shuts down after 15 seconds
|
|
||||||
- ✅ Dashboard stays functional (data cached)
|
|
||||||
- ✅ No manual cleanup needed
|
|
||||||
|
|
||||||
**Want to keep exploring?** Press Ctrl+C during countdown to keep server running.
|
|
||||||
|
|
||||||
### Comparing Agents
|
|
||||||
1. Set **Time Range** to "Latest Run"
|
|
||||||
2. Set **Agent** to "All Agents"
|
|
||||||
3. Compare pass rates and durations
|
|
||||||
|
|
||||||
### Finding Flaky Tests
|
|
||||||
1. Set **Time Range** to "Last 30 Days"
|
|
||||||
2. Look for tests that alternate between pass/fail
|
|
||||||
3. Check violation details for patterns
|
|
||||||
|
|
||||||
### Tracking Improvements
|
|
||||||
1. Run tests regularly (daily/weekly)
|
|
||||||
2. Watch the trend chart for improvements
|
|
||||||
3. Export CSV for deeper analysis
|
|
||||||
|
|
||||||
### Debugging Failures
|
|
||||||
1. Filter **Status** to "Failed Only"
|
|
||||||
2. Click on a failed test row
|
|
||||||
3. Review violation details
|
|
||||||
4. Check error messages and severity
|
|
||||||
|
|
||||||
## Browser Compatibility
|
|
||||||
|
|
||||||
- ✅ Chrome/Edge (recommended)
|
|
||||||
- ✅ Firefox
|
|
||||||
- ✅ Safari
|
|
||||||
- ⚠️ IE11 (not supported)
|
|
||||||
|
|
||||||
## Performance
|
|
||||||
|
|
||||||
- **Dashboard Size:** ~31KB (no dependencies except Chart.js CDN)
|
|
||||||
- **Load Time:** < 1 second for 100 tests
|
|
||||||
- **Memory:** Minimal (pure JavaScript, no frameworks)
|
|
||||||
|
|
||||||
## How It Works
|
|
||||||
|
|
||||||
### Auto-Shutdown Feature
|
|
||||||
The `serve.sh` script:
|
|
||||||
1. Starts HTTP server on port 8000
|
|
||||||
2. Opens dashboard in your browser
|
|
||||||
3. Waits 15 seconds for data to load
|
|
||||||
4. Shuts down server automatically
|
|
||||||
5. Dashboard continues working (data cached in browser)
|
|
||||||
|
|
||||||
**Why does it still work after shutdown?**
|
|
||||||
- The browser caches the JSON data
|
|
||||||
- All filtering/sorting happens in JavaScript
|
|
||||||
- No server needed after initial load
|
|
||||||
- Refresh the page to load new data (server will need to restart)
|
|
||||||
|
|
||||||
### Stopping Manually
|
|
||||||
If you start the server manually:
|
|
||||||
```bash
|
|
||||||
# Find the process
|
|
||||||
lsof -ti:8000
|
|
||||||
|
|
||||||
# Kill it
|
|
||||||
kill $(lsof -ti:8000)
|
|
||||||
```
|
|
||||||
|
|
||||||
Or just press Ctrl+C in the terminal.
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
### Dashboard shows "No results found"
|
|
||||||
- Run tests first: `npm run eval:sdk`
|
|
||||||
- Check that `latest.json` exists
|
|
||||||
- Refresh the page
|
|
||||||
|
|
||||||
### Chart not displaying
|
|
||||||
- Check browser console for errors
|
|
||||||
- Ensure Chart.js CDN is accessible
|
|
||||||
- Try refreshing the page
|
|
||||||
|
|
||||||
### Dark mode not persisting
|
|
||||||
- Check browser localStorage is enabled
|
|
||||||
- Clear cache and try again
|
|
||||||
|
|
||||||
## Future Enhancements
|
|
||||||
|
|
||||||
Potential improvements:
|
|
||||||
- [ ] Historical comparison (compare two runs)
|
|
||||||
- [ ] Test duration trends per test
|
|
||||||
- [ ] Violation type breakdown chart
|
|
||||||
- [ ] Agent performance comparison chart
|
|
||||||
- [ ] Auto-refresh option
|
|
||||||
- [ ] Shareable URLs with filters
|
|
||||||
- [ ] CI/CD badge generation
|
|
||||||
|
|
||||||
## Contributing
|
|
||||||
|
|
||||||
To improve the dashboard:
|
|
||||||
|
|
||||||
1. Edit `index.html` (all code is in one file)
|
|
||||||
2. Test locally by opening in browser
|
|
||||||
3. Submit PR with description of changes
|
|
||||||
|
|
||||||
## License
|
|
||||||
|
|
||||||
MIT - Same as OpenCode Agents project
|
|
||||||
|
|
@ -1,29 +0,0 @@
|
||||||
# Test Artifacts
|
|
||||||
|
|
||||||
This directory contains temporary files created during test execution.
|
|
||||||
It should be cleaned up after tests complete.
|
|
||||||
|
|
||||||
**DO NOT COMMIT FILES IN THIS DIRECTORY**
|
|
||||||
|
|
||||||
## Installation
|
|
||||||
|
|
||||||
To install the project dependencies, navigate to the evaluation framework directory and run:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd evals/framework
|
|
||||||
npm install
|
|
||||||
```
|
|
||||||
|
|
||||||
This will install all required dependencies including:
|
|
||||||
- `@opencode-ai/sdk` - OpenCode AI SDK
|
|
||||||
- `yaml` - YAML parser for test cases
|
|
||||||
- `zod` - Schema validation
|
|
||||||
- `glob` - File pattern matching
|
|
||||||
|
|
||||||
### Development Dependencies
|
|
||||||
|
|
||||||
For development and testing, the following tools are also installed:
|
|
||||||
- TypeScript compiler
|
|
||||||
- Vitest testing framework
|
|
||||||
- ESLint for code linting
|
|
||||||
- tsx for TypeScript execution
|
|
||||||
Loading…
Reference in a new issue