4103805270
* feat(evals): restructure OpenAgent tests + fix SDK mode session creation
## Test Restructure
Reorganize OpenAgent tests into 6 priority-based categories for better
maintainability, scalability, and CI/CD integration.
New structure:
- 01-critical-rules/ (15 tests) - MUST PASS safety requirements
- 02-workflow-stages/ (2 tests) - Workflow validation
- 03-delegation/ (0 tests) - Delegation scenarios (ready for new tests)
- 04-execution-paths/ (2 tests) - Conversational vs task paths
- 05-edge-cases/ (1 test) - Edge cases and boundaries
- 06-integration/ (2 tests) - Complex multi-turn scenarios
Changes:
- Migrate 22 existing tests to new structure (verified identical)
- Add comprehensive documentation (5 markdown files)
- Add migration and verification scripts
- Preserve original test locations for backward compatibility
## Bug Fix: SDK Mode Session Creation
Fix session creation failure introduced in commit
|
||
|---|---|---|
| .. | ||
| tests | ||
| auto-detect-components.sh | ||
| bump-version.sh | ||
| cleanup-stale-sessions.sh | ||
| dashboard.sh | ||
| README.md | ||
| register-component.sh | ||
| test.sh | ||
| uninstall.sh | ||
| validate-component.sh | ||
| validate-context-refs.sh | ||
| validate-registry.sh | ||
Scripts
This directory contains utility scripts for the OpenAgents system.
Available Scripts
Testing
test.sh- Main test runner with multi-agent support- Run all tests:
./scripts/test.sh openagent - Run core tests:
./scripts/test.sh openagent --core(7 tests, ~5-8 min) - Run with specific model:
./scripts/test.sh openagent opencode/grok-code-fast - Debug mode:
./scripts/test.sh openagent --core --debug
- Run all tests:
See tests/ subdirectory for installer test scripts.
Component Management
register-component.sh- Register a new component in the registryvalidate-component.sh- Validate component structure and metadata
Session Management
cleanup-stale-sessions.sh- Remove stale agent sessions older than 24 hours
Session Cleanup
Agent instances create temporary context files in .tmp/sessions/{session-id}/ for subagent delegation. These sessions are automatically cleaned up, but you can manually remove stale sessions:
# Clean up sessions older than 24 hours
./scripts/cleanup-stale-sessions.sh
# Or manually delete all sessions
rm -rf .tmp/sessions/
Sessions are safe to delete anytime - they only contain temporary context files for agent coordination.
Usage Examples
Run Tests
# Run core test suite (fast, 7 tests, ~5-8 min)
./scripts/test.sh openagent --core
# Run all tests for OpenAgent
./scripts/test.sh openagent
# Run tests with specific model
./scripts/test.sh openagent anthropic/claude-sonnet-4-5
# Run core tests with debug mode
./scripts/test.sh openagent --core --debug
Register a Component
./scripts/register-component.sh path/to/component
Validate a Component
./scripts/validate-component.sh path/to/component
Clean Stale Sessions
./scripts/cleanup-stale-sessions.sh
Core Test Suite
The core test suite is a subset of 7 carefully selected tests that provide ~85% coverage of critical OpenAgent functionality in just 5-8 minutes.
Why Use Core Tests?
- ✅ Fast feedback - 5-8 minutes vs 40-80 minutes for full suite
- ✅ Prompt iteration - Quick validation when updating agent prompts
- ✅ Development - Fast validation during development cycles
- ✅ Pre-commit - Quick checks before committing changes
What's Covered?
- Approval Gate - Critical safety rule
- Context Loading (Simple) - Most common use case
- Context Loading (Multi-Turn) - Complex scenarios
- Stop on Failure - Error handling
- Simple Task - No unnecessary delegation
- Subagent Delegation - Proper delegation when needed
- Tool Usage - Best practices
When to Use Full Suite?
Use the full test suite (71 tests) for:
- 🔬 Release validation
- 🔬 Comprehensive testing
- 🔬 Edge case coverage
- 🔬 Regression testing
See evals/agents/openagent/CORE_TESTS.md for detailed documentation.