cclohmar/OpenWriterControle
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
OpenWriterControle/CHANGELOG.md
Darren Hinde c8f7103cb6
refactor(evals): consolidate documentation and enhance test infrastructure (#56) 
* feat(agents): implement category-based agent organization system

- Organize agents into domain categories (core, development, content, data, product, learning)
- Move 10 agents to category subdirectories with proper git rename tracking
- Update 13 subagents with category and type metadata in frontmatter
- Add category metadata files (0-category.json) documenting common patterns
- Implement local registry fallback in install script for offline development
- Add comprehensive validation suite with 15 automated tests (100% pass rate)
- Enhance registry validation with duplicate ID and consistency checks
- Update eval framework with intelligent path resolution (backward compatible)
- Archive legacy eval structure to _archive/ for reference
- Update all documentation to reflect category-based structure
- Bump version to 0.5.0 with accurate CHANGELOG

Technical Details:
- 23 agents organized (10 category agents, 13 subagents)
- 6 category directories created
- Path resolution supports both agent IDs and category paths
- Registry schema updated to v2.0.0
- 159 files changed, 872 insertions(+), 10738 deletions(-)

BREAKING CHANGE: Agent file paths now use category structure. Update references from .opencode/agent/openagent.md to .opencode/agent/core/openagent.md. Eval framework maintains backward compatibility via path resolution.

* refactor(evals): consolidate documentation and enhance test infrastructure

- Remove temporary project tracking files (PHASE_5_COMPLETE, PROJECT_COMPLETE, etc.)
- Consolidate evaluation framework docs into main README
- Enhance test execution with improved logging and multi-prompt support
- Move system-builder from core to meta category
- Add comprehensive test suites for openagent with organized structure
- Create evaluation test structure for all subagents
- Clean up archived workflows and redundant documentation
- Update registry to reflect new agent organization
- Add shared test templates and golden test patterns

* feat(ci): add manual trigger support for bot-created PRs in validate-registry workflow

- Add pr_number input to workflow_dispatch for manually triggering validation
- Fetch PR details dynamically when triggered manually
- Support both automatic (PR event) and manual (workflow_dispatch) triggers
- Enable validation of bot-created PRs like automated version bumps
- Update branch detection and push logic to handle both trigger types
- Add documentation explaining how to manually trigger for bot PRs

* feat(evals): add explicit context file validation to test framework

- Add expectedContextFiles field to test YAML schema for explicit context file specification
- Enhance context-loading-evaluator to support both auto-detect and explicit validation modes
- Update documentation with comprehensive guide and examples
- Clean up archived legacy test structure (90+ old test files)
- Add new example tests demonstrating explicit context validation
- Backward compatible with existing tests

* feat(evals): add multi-agent logging system and performance optimizations

Implement comprehensive multi-agent logging and performance improvements for the eval framework.

Task 01: Multi-Agent Logging System
- Add complete hierarchical logging module (evals/framework/src/logging/)
  - SessionTracker: tracks parent-child delegation hierarchies
  - MultiAgentLogger: pretty-prints logs with visual indentation
  - Formatters: box characters and emoji formatting
  - 37 passing unit tests (session-tracker, logger, integration)
- Integrate with SDK event stream handler
  - Hook into session.created, message.updated, message.part.updated events
  - Real-time child session detection via timestamp heuristics
  - Message deduplication for cleaner output
- Enable in debug mode only (<1% performance overhead)
- Add demo script and comprehensive documentation

Task 02: Performance Optimizations
- Reduce grace period from 5s to 2s (67% reduction, 10-20% faster tests)
- Add PerformanceMetricsEvaluator for bottleneck identification
  - Collects tool latencies, inference time, idle time
  - Provides full performance visibility
- Update smoke test to validate delegation (core feature)
  - Replace simple read test with multi-agent delegation test
  - Tests both parent and child agent functionality
  - Validates multi-agent logging system

Results:
- 37 unit tests passing
- Smoke test passing (95/100 score)
- Clean hierarchical logging output
- 10-20% faster test execution
- Full multi-agent visibility

Files changed: 16 files, +2678 lines
Time saved: ~6-8 days (completed in 6 hours vs 3-5 day estimate)

* feat(evals): show child agent execution in non-debug mode

- Enable MultiAgentLogger in both debug and non-debug modes
- Add verbose parameter to control output level
- Non-verbose mode shows concise child session lifecycle:
  - Child agent started message
  - Child agent completed with duration
- Verbose mode (--debug) shows full delegation hierarchy
- Update documentation to reflect new behavior

This provides visibility into delegation without overwhelming output,
giving confidence that child agents are actually running.

* feat(evals): improve delegation testing and behavior validation

Major improvements to eval framework:

Test Suite:
- Add 00-smoke-test.yaml (basic read operation)
- Rename 01-smoke-test.yaml → 02-delegation-test.yaml (delegation test)
- Add simple-responder test agent for delegation testing
- Add debug test: simple-subagent-call.yaml

Evaluators:
- Add agent-model-evaluator.ts for agent/model validation
- Enhance behavior-evaluator.ts with detailed task tool output
- Improve delegation-evaluator.ts with better evidence tracking

Schema & Execution:
- Add expectedAgent and expectedModel to test schema
- Improve test executor with better logging and error handling

Documentation:
- Update evals/README.md with new features and performance improvements
- Remove outdated MULTI_AGENT_LOGGING_COMPLETE.md

Registry:
- Add simple-responder test agent to registry

Fixes:
- Fix dashboard.sh path resolution

These changes provide better visibility into delegation, improved test
coverage, and clearer validation of agent behavior.

* chore: add GitHub Actions workflow and plugin documentation

- Add .github/workflows/evals/run-evaluations.yml for automated eval testing
- Add dev/ai-tools/opencode/plugins/Plugin-inspiration.md for plugin development reference
2025-12-18 09:52:11 +00:00

5.5 KiB
Raw Blame History

Changelog

All notable changes to this project will be documented in this file.

[Unreleased]

Added

  • Explicit Context File Validation: New expectedContextFiles field in test YAML files allows explicit specification of which context files the agent must read
    • Overrides auto-detection when specified
    • Uses flexible pattern matching (includes() or endsWith())
    • Supports partial paths (e.g., standards/code.md) or full paths
    • See evals/agents/shared/tests/EXPLICIT_CONTEXT_FILES.md for detailed guide
    • Example test: evals/agents/shared/tests/golden/02-context-loading-explicit.yaml

Changed

  • Context Loading Evaluator: Now accepts optional BehaviorExpectation config to support explicit file validation
    • Shows detection mode in evidence: "Explicit (from YAML test)" or "Auto-detect (from user message)"
    • Backward compatible - existing tests work unchanged

Documentation

  • Added evals/agents/shared/tests/EXPLICIT_CONTEXT_FILES.md - Complete feature guide
  • Added evals/PATTERN_MATCHING_GUIDE.md - Pattern matching reference
  • Updated evals/CREATING_TESTS.md - Added expectedContextFiles documentation
  • Updated evals/README.md - Added new feature section

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[0.5.0] - 2025-12-10

Added

  • Category-Based Agent Organization: Agents now organized by domain in subdirectories
    • core/ - Core system agents (openagent, opencoder)
    • meta/ - Meta-level agents (system-builder)
    • development/ - Development specialists (frontend-specialist, backend-specialist, devops-specialist, codebase-agent)
    • content/ - Content creation agents (copywriter, technical-writer)
    • data/ - Data and analysis agents (data-analyst)
    • product/ - Product management agents (directory created, ready for agents)
    • learning/ - Education and coaching agents (directory created, ready for agents)
  • Category Metadata Files: Each category has 0-category.json documenting common subagents, tools, and context
  • Subagent Organization: 13 subagents organized into 4 categories (code, core, system-builder, utils)
  • Local Registry Fallback: Install script now supports local registry.json for development/testing
  • Enhanced Registry Validation: Added checks for duplicate IDs and category consistency
  • Comprehensive Test Suite: 15 automated validation tests for category system
  • Audit Tools: Scripts to verify migration completeness and system integrity

Changed

  • Agent File Structure: Agents moved from flat structure to category-based organization
    • Example: .opencode/agent/openagent.md → .opencode/agent/core/openagent.md
    • Example: .opencode/agent/opencoder.md → .opencode/agent/core/opencoder.md
  • Registry Schema: Updated to include category-based paths for all agents
  • Eval Framework: Enhanced with path resolution supporting both agent IDs and category paths
    • --agent=openagent resolves to core/openagent (backward compatible)
    • --agent=core/openagent works directly (new format)
  • Documentation: Updated all docs to reference category-based structure
  • Install Script: Enhanced with local registry fallback for offline/development use

Fixed

  • Frontend Specialist: Renamed frontend-design-agent.md to frontend-specialist.md for consistency
  • Eval Directory Structure: Archived legacy flat eval structure to _archive/ for clarity
  • Registry Validation: Fixed validation script to handle category-based paths correctly

Developer Experience

  • Backward Compatibility: Agent IDs like openagent still work via path resolution
  • Local Testing: No need to push to GitHub to test registry changes
  • Clear Organization: Agents grouped by domain make discovery easier
  • Validation Tools: Automated tests ensure system integrity

Migration Notes

  • Agent Paths: Update references from flat to category-based paths
    • Old: .opencode/agent/openagent.md
    • New: .opencode/agent/core/openagent.md
  • Eval Framework: Both formats work due to path resolution
    • --agent=openagent (backward compatible)
    • --agent=core/openagent (new format)
  • No Breaking Changes: Path resolution maintains backward compatibility

Technical Details

  • Files Modified: 14+ files updated for category structure
  • Agents Organized: 23 total agents (10 category agents, 13 subagents)
  • Test Coverage: 15/15 validation tests passing (100%)
  • Audit Status: 8/8 checks passing (100%)

[0.3.1] - 2025-12-09

Fixed

  • CI: Check only commit title for skip patterns (#46)

[0.0.2] - 2025-11-29

Added

  • New ExecutionBalanceEvaluator in evals/framework to assess read vs execution ordering and ratio
  • Contributor guide: docs/contributing/ADDING_EVALUATOR.md describing evaluator design principles
  • Test cases under evals/agents/openagent/tests/10-execution-balance/ (positive & negative scenarios)

Changed

  • Framework README updated with section documenting ExecutionBalanceEvaluator and violation codes

Version Format

v0.X.Y
│ │ │
│ │ └─ Patch version (bug fixes, minor changes)
│ └─── Minor version (new features, non-breaking changes)
└───── Major version (breaking changes, major milestones)

Version History

  • 0.5.0 - Category-based agent organization system
  • 0.3.1 - CI improvements
  • 0.0.2 - Execution balance evaluator