cclohmar/OpenWriterControle
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
OpenWriterControle/CHANGELOG.md
Darren Hinde c8f7103cb6
refactor(evals): consolidate documentation and enhance test infrastructure (#56) 
* feat(agents): implement category-based agent organization system

- Organize agents into domain categories (core, development, content, data, product, learning)
- Move 10 agents to category subdirectories with proper git rename tracking
- Update 13 subagents with category and type metadata in frontmatter
- Add category metadata files (0-category.json) documenting common patterns
- Implement local registry fallback in install script for offline development
- Add comprehensive validation suite with 15 automated tests (100% pass rate)
- Enhance registry validation with duplicate ID and consistency checks
- Update eval framework with intelligent path resolution (backward compatible)
- Archive legacy eval structure to _archive/ for reference
- Update all documentation to reflect category-based structure
- Bump version to 0.5.0 with accurate CHANGELOG

Technical Details:
- 23 agents organized (10 category agents, 13 subagents)
- 6 category directories created
- Path resolution supports both agent IDs and category paths
- Registry schema updated to v2.0.0
- 159 files changed, 872 insertions(+), 10738 deletions(-)

BREAKING CHANGE: Agent file paths now use category structure. Update references from .opencode/agent/openagent.md to .opencode/agent/core/openagent.md. Eval framework maintains backward compatibility via path resolution.

* refactor(evals): consolidate documentation and enhance test infrastructure

- Remove temporary project tracking files (PHASE_5_COMPLETE, PROJECT_COMPLETE, etc.)
- Consolidate evaluation framework docs into main README
- Enhance test execution with improved logging and multi-prompt support
- Move system-builder from core to meta category
- Add comprehensive test suites for openagent with organized structure
- Create evaluation test structure for all subagents
- Clean up archived workflows and redundant documentation
- Update registry to reflect new agent organization
- Add shared test templates and golden test patterns

* feat(ci): add manual trigger support for bot-created PRs in validate-registry workflow

- Add pr_number input to workflow_dispatch for manually triggering validation
- Fetch PR details dynamically when triggered manually
- Support both automatic (PR event) and manual (workflow_dispatch) triggers
- Enable validation of bot-created PRs like automated version bumps
- Update branch detection and push logic to handle both trigger types
- Add documentation explaining how to manually trigger for bot PRs

* feat(evals): add explicit context file validation to test framework

- Add expectedContextFiles field to test YAML schema for explicit context file specification
- Enhance context-loading-evaluator to support both auto-detect and explicit validation modes
- Update documentation with comprehensive guide and examples
- Clean up archived legacy test structure (90+ old test files)
- Add new example tests demonstrating explicit context validation
- Backward compatible with existing tests

* feat(evals): add multi-agent logging system and performance optimizations

Implement comprehensive multi-agent logging and performance improvements for the eval framework.

Task 01: Multi-Agent Logging System
- Add complete hierarchical logging module (evals/framework/src/logging/)
  - SessionTracker: tracks parent-child delegation hierarchies
  - MultiAgentLogger: pretty-prints logs with visual indentation
  - Formatters: box characters and emoji formatting
  - 37 passing unit tests (session-tracker, logger, integration)
- Integrate with SDK event stream handler
  - Hook into session.created, message.updated, message.part.updated events
  - Real-time child session detection via timestamp heuristics
  - Message deduplication for cleaner output
- Enable in debug mode only (<1% performance overhead)
- Add demo script and comprehensive documentation

Task 02: Performance Optimizations
- Reduce grace period from 5s to 2s (67% reduction, 10-20% faster tests)
- Add PerformanceMetricsEvaluator for bottleneck identification
  - Collects tool latencies, inference time, idle time
  - Provides full performance visibility
- Update smoke test to validate delegation (core feature)
  - Replace simple read test with multi-agent delegation test
  - Tests both parent and child agent functionality
  - Validates multi-agent logging system

Results:
- 37 unit tests passing
- Smoke test passing (95/100 score)
- Clean hierarchical logging output
- 10-20% faster test execution
- Full multi-agent visibility

Files changed: 16 files, +2678 lines
Time saved: ~6-8 days (completed in 6 hours vs 3-5 day estimate)

* feat(evals): show child agent execution in non-debug mode

- Enable MultiAgentLogger in both debug and non-debug modes
- Add verbose parameter to control output level
- Non-verbose mode shows concise child session lifecycle:
  - Child agent started message
  - Child agent completed with duration
- Verbose mode (--debug) shows full delegation hierarchy
- Update documentation to reflect new behavior

This provides visibility into delegation without overwhelming output,
giving confidence that child agents are actually running.

* feat(evals): improve delegation testing and behavior validation

Major improvements to eval framework:

Test Suite:
- Add 00-smoke-test.yaml (basic read operation)
- Rename 01-smoke-test.yaml → 02-delegation-test.yaml (delegation test)
- Add simple-responder test agent for delegation testing
- Add debug test: simple-subagent-call.yaml

Evaluators:
- Add agent-model-evaluator.ts for agent/model validation
- Enhance behavior-evaluator.ts with detailed task tool output
- Improve delegation-evaluator.ts with better evidence tracking

Schema & Execution:
- Add expectedAgent and expectedModel to test schema
- Improve test executor with better logging and error handling

Documentation:
- Update evals/README.md with new features and performance improvements
- Remove outdated MULTI_AGENT_LOGGING_COMPLETE.md

Registry:
- Add simple-responder test agent to registry

Fixes:
- Fix dashboard.sh path resolution

These changes provide better visibility into delegation, improved test
coverage, and clearer validation of agent behavior.

* chore: add GitHub Actions workflow and plugin documentation

- Add .github/workflows/evals/run-evaluations.yml for automated eval testing
- Add dev/ai-tools/opencode/plugins/Plugin-inspiration.md for plugin development reference
2025-12-18 09:52:11 +00:00

114 lines
5.5 KiB
Markdown
Raw Blame History

# Changelog
All notable changes to this project will be documented in this file.
## [Unreleased]
### Added
- **Explicit Context File Validation**: New `expectedContextFiles` field in test YAML files allows explicit specification of which context files the agent must read
- Overrides auto-detection when specified
- Uses flexible pattern matching (`includes()` or `endsWith()`)
- Supports partial paths (e.g., `standards/code.md`) or full paths
- See `evals/agents/shared/tests/EXPLICIT_CONTEXT_FILES.md` for detailed guide
- Example test: `evals/agents/shared/tests/golden/02-context-loading-explicit.yaml`
### Changed
- **Context Loading Evaluator**: Now accepts optional `BehaviorExpectation` config to support explicit file validation
- Shows detection mode in evidence: "Explicit (from YAML test)" or "Auto-detect (from user message)"
- Backward compatible - existing tests work unchanged
### Documentation
- Added `evals/agents/shared/tests/EXPLICIT_CONTEXT_FILES.md` - Complete feature guide
- Added `evals/PATTERN_MATCHING_GUIDE.md` - Pattern matching reference
- Updated `evals/CREATING_TESTS.md` - Added `expectedContextFiles` documentation
- Updated `evals/README.md` - Added new feature section
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [0.5.0] - 2025-12-10
### Added
- **Category-Based Agent Organization**: Agents now organized by domain in subdirectories
- `core/` - Core system agents (openagent, opencoder)
- `meta/` - Meta-level agents (system-builder)
- `development/` - Development specialists (frontend-specialist, backend-specialist, devops-specialist, codebase-agent)
- `content/` - Content creation agents (copywriter, technical-writer)
- `data/` - Data and analysis agents (data-analyst)
- `product/` - Product management agents (directory created, ready for agents)
- `learning/` - Education and coaching agents (directory created, ready for agents)
- **Category Metadata Files**: Each category has `0-category.json` documenting common subagents, tools, and context
- **Subagent Organization**: 13 subagents organized into 4 categories (code, core, system-builder, utils)
- **Local Registry Fallback**: Install script now supports local `registry.json` for development/testing
- **Enhanced Registry Validation**: Added checks for duplicate IDs and category consistency
- **Comprehensive Test Suite**: 15 automated validation tests for category system
- **Audit Tools**: Scripts to verify migration completeness and system integrity
### Changed
- **Agent File Structure**: Agents moved from flat structure to category-based organization
- Example: `.opencode/agent/openagent.md` → `.opencode/agent/core/openagent.md`
- Example: `.opencode/agent/opencoder.md` → `.opencode/agent/core/opencoder.md`
- **Registry Schema**: Updated to include category-based paths for all agents
- **Eval Framework**: Enhanced with path resolution supporting both agent IDs and category paths
- `--agent=openagent` resolves to `core/openagent` (backward compatible)
- `--agent=core/openagent` works directly (new format)
- **Documentation**: Updated all docs to reference category-based structure
- **Install Script**: Enhanced with local registry fallback for offline/development use
### Fixed
- **Frontend Specialist**: Renamed `frontend-design-agent.md` to `frontend-specialist.md` for consistency
- **Eval Directory Structure**: Archived legacy flat eval structure to `_archive/` for clarity
- **Registry Validation**: Fixed validation script to handle category-based paths correctly
### Developer Experience
- **Backward Compatibility**: Agent IDs like `openagent` still work via path resolution
- **Local Testing**: No need to push to GitHub to test registry changes
- **Clear Organization**: Agents grouped by domain make discovery easier
- **Validation Tools**: Automated tests ensure system integrity
### Migration Notes
- **Agent Paths**: Update references from flat to category-based paths
- Old: `.opencode/agent/openagent.md`
- New: `.opencode/agent/core/openagent.md`
- **Eval Framework**: Both formats work due to path resolution
- `--agent=openagent` (backward compatible)
- `--agent=core/openagent` (new format)
- **No Breaking Changes**: Path resolution maintains backward compatibility
### Technical Details
- **Files Modified**: 14+ files updated for category structure
- **Agents Organized**: 23 total agents (10 category agents, 13 subagents)
- **Test Coverage**: 15/15 validation tests passing (100%)
- **Audit Status**: 8/8 checks passing (100%)
## [0.3.1] - 2025-12-09
### Fixed
- CI: Check only commit title for skip patterns (#46)
## [0.0.2] - 2025-11-29
### Added
- New `ExecutionBalanceEvaluator` in `evals/framework` to assess read vs execution ordering and ratio
- Contributor guide: `docs/contributing/ADDING_EVALUATOR.md` describing evaluator design principles
- Test cases under `evals/agents/openagent/tests/10-execution-balance/` (positive & negative scenarios)
### Changed
- Framework README updated with section documenting `ExecutionBalanceEvaluator` and violation codes
---
## Version Format
```
v0.X.Y
│ │ │
│ │ └─ Patch version (bug fixes, minor changes)
│ └─── Minor version (new features, non-breaking changes)
└───── Major version (breaking changes, major milestones)
```
### Version History
- **0.5.0** - Category-based agent organization system
- **0.3.1** - CI improvements
- **0.0.2** - Execution balance evaluator
Reference in a new issue View git blame Copy permalink