cclohmar/OpenWriterControle
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
OpenWriterControle/.opencode/context/openagents-repo/examples/context-bundle-example.md
Darren Hinde c8f7103cb6
refactor(evals): consolidate documentation and enhance test infrastructure (#56) 
* feat(agents): implement category-based agent organization system

- Organize agents into domain categories (core, development, content, data, product, learning)
- Move 10 agents to category subdirectories with proper git rename tracking
- Update 13 subagents with category and type metadata in frontmatter
- Add category metadata files (0-category.json) documenting common patterns
- Implement local registry fallback in install script for offline development
- Add comprehensive validation suite with 15 automated tests (100% pass rate)
- Enhance registry validation with duplicate ID and consistency checks
- Update eval framework with intelligent path resolution (backward compatible)
- Archive legacy eval structure to _archive/ for reference
- Update all documentation to reflect category-based structure
- Bump version to 0.5.0 with accurate CHANGELOG

Technical Details:
- 23 agents organized (10 category agents, 13 subagents)
- 6 category directories created
- Path resolution supports both agent IDs and category paths
- Registry schema updated to v2.0.0
- 159 files changed, 872 insertions(+), 10738 deletions(-)

BREAKING CHANGE: Agent file paths now use category structure. Update references from .opencode/agent/openagent.md to .opencode/agent/core/openagent.md. Eval framework maintains backward compatibility via path resolution.

* refactor(evals): consolidate documentation and enhance test infrastructure

- Remove temporary project tracking files (PHASE_5_COMPLETE, PROJECT_COMPLETE, etc.)
- Consolidate evaluation framework docs into main README
- Enhance test execution with improved logging and multi-prompt support
- Move system-builder from core to meta category
- Add comprehensive test suites for openagent with organized structure
- Create evaluation test structure for all subagents
- Clean up archived workflows and redundant documentation
- Update registry to reflect new agent organization
- Add shared test templates and golden test patterns

* feat(ci): add manual trigger support for bot-created PRs in validate-registry workflow

- Add pr_number input to workflow_dispatch for manually triggering validation
- Fetch PR details dynamically when triggered manually
- Support both automatic (PR event) and manual (workflow_dispatch) triggers
- Enable validation of bot-created PRs like automated version bumps
- Update branch detection and push logic to handle both trigger types
- Add documentation explaining how to manually trigger for bot PRs

* feat(evals): add explicit context file validation to test framework

- Add expectedContextFiles field to test YAML schema for explicit context file specification
- Enhance context-loading-evaluator to support both auto-detect and explicit validation modes
- Update documentation with comprehensive guide and examples
- Clean up archived legacy test structure (90+ old test files)
- Add new example tests demonstrating explicit context validation
- Backward compatible with existing tests

* feat(evals): add multi-agent logging system and performance optimizations

Implement comprehensive multi-agent logging and performance improvements for the eval framework.

Task 01: Multi-Agent Logging System
- Add complete hierarchical logging module (evals/framework/src/logging/)
  - SessionTracker: tracks parent-child delegation hierarchies
  - MultiAgentLogger: pretty-prints logs with visual indentation
  - Formatters: box characters and emoji formatting
  - 37 passing unit tests (session-tracker, logger, integration)
- Integrate with SDK event stream handler
  - Hook into session.created, message.updated, message.part.updated events
  - Real-time child session detection via timestamp heuristics
  - Message deduplication for cleaner output
- Enable in debug mode only (<1% performance overhead)
- Add demo script and comprehensive documentation

Task 02: Performance Optimizations
- Reduce grace period from 5s to 2s (67% reduction, 10-20% faster tests)
- Add PerformanceMetricsEvaluator for bottleneck identification
  - Collects tool latencies, inference time, idle time
  - Provides full performance visibility
- Update smoke test to validate delegation (core feature)
  - Replace simple read test with multi-agent delegation test
  - Tests both parent and child agent functionality
  - Validates multi-agent logging system

Results:
- 37 unit tests passing
- Smoke test passing (95/100 score)
- Clean hierarchical logging output
- 10-20% faster test execution
- Full multi-agent visibility

Files changed: 16 files, +2678 lines
Time saved: ~6-8 days (completed in 6 hours vs 3-5 day estimate)

* feat(evals): show child agent execution in non-debug mode

- Enable MultiAgentLogger in both debug and non-debug modes
- Add verbose parameter to control output level
- Non-verbose mode shows concise child session lifecycle:
  - Child agent started message
  - Child agent completed with duration
- Verbose mode (--debug) shows full delegation hierarchy
- Update documentation to reflect new behavior

This provides visibility into delegation without overwhelming output,
giving confidence that child agents are actually running.

* feat(evals): improve delegation testing and behavior validation

Major improvements to eval framework:

Test Suite:
- Add 00-smoke-test.yaml (basic read operation)
- Rename 01-smoke-test.yaml → 02-delegation-test.yaml (delegation test)
- Add simple-responder test agent for delegation testing
- Add debug test: simple-subagent-call.yaml

Evaluators:
- Add agent-model-evaluator.ts for agent/model validation
- Enhance behavior-evaluator.ts with detailed task tool output
- Improve delegation-evaluator.ts with better evidence tracking

Schema & Execution:
- Add expectedAgent and expectedModel to test schema
- Improve test executor with better logging and error handling

Documentation:
- Update evals/README.md with new features and performance improvements
- Remove outdated MULTI_AGENT_LOGGING_COMPLETE.md

Registry:
- Add simple-responder test agent to registry

Fixes:
- Fix dashboard.sh path resolution

These changes provide better visibility into delegation, improved test
coverage, and clearer validation of agent behavior.

* chore: add GitHub Actions workflow and plugin documentation

- Add .github/workflows/evals/run-evaluations.yml for automated eval testing
- Add dev/ai-tools/opencode/plugins/Plugin-inspiration.md for plugin development reference
2025-12-18 09:52:11 +00:00

214 lines
8 KiB
Markdown
Raw Blame History

# Context Bundle Example: Create Data Analyst Agent
Session: 20250121-143022-a4f2
Created: 2025-01-21T14:30:22Z
For: subagents/core/task-manager
Status: in_progress
## Task Overview
Create a new data analyst agent for the OpenAgents repository. This agent will specialize in data analysis tasks including data visualization, statistical analysis, and data transformation.
## User Request
"Create a new data analyst agent that can help with data analysis, visualization, and statistical tasks"
## Relevant Standards (Load These Before Starting)
**Core Standards**:
- `.opencode/context/core/standards/code.md` → Modular, functional code patterns
- `.opencode/context/core/standards/tests.md` → Testing requirements and TDD
- `.opencode/context/core/standards/docs.md` → Documentation standards
**Core Workflows**:
- `.opencode/context/core/workflows/task-breakdown.md` → Task breakdown methodology
## Repository-Specific Context (Load These Before Starting)
**Quick Start** (ALWAYS load first):
- `.opencode/context/openagents-repo/quick-start.md` → Repo orientation and common commands
**Core Concepts** (Load based on task type):
- `.opencode/context/openagents-repo/core-concepts/agents.md` → How agents work
- `.opencode/context/openagents-repo/core-concepts/evals.md` → How testing works
- `.opencode/context/openagents-repo/core-concepts/registry.md` → How registry works
- `.opencode/context/openagents-repo/core-concepts/categories.md` → How organization works
**Guides** (Load for specific workflows):
- `.opencode/context/openagents-repo/guides/adding-agent.md` → Step-by-step agent creation
- `.opencode/context/openagents-repo/guides/testing-agent.md` → Testing workflow
- `.opencode/context/openagents-repo/guides/updating-registry.md` → Registry workflow
## Key Requirements
**From Standards**:
- Agent must follow modular, functional programming patterns
- All code must be testable and maintainable
- Documentation must be concise and high-signal
- Include examples where helpful
**From Repository Context**:
- Agent file must be in `.opencode/agent/data/` directory (category-based organization)
- Must include proper frontmatter metadata (id, name, description, category, type, version, etc.)
- Must follow naming convention: `data-analyst.md` (kebab-case)
- Must include tags for discoverability
- Must specify tools and permissions
- Must be registered in `registry.json`
**Naming Conventions**:
- File name: `data-analyst.md` (kebab-case)
- Agent ID: `data-analyst`
- Category: `data`
- Type: `agent`
**File Structure**:
- Agent file: `.opencode/agent/data/data-analyst.md`
- Eval directory: `evals/agents/data/data-analyst/`
- Eval config: `evals/agents/data/data-analyst/config/eval-config.yaml`
- Eval tests: `evals/agents/data/data-analyst/tests/`
- README: `evals/agents/data/data-analyst/README.md`
## Technical Constraints
- Must use category-based organization (data category)
- Must include proper frontmatter metadata
- Must specify tools needed (read, write, bash, etc.)
- Must define permissions for sensitive operations
- Must include temperature setting (0.1-0.3 for analytical tasks)
- Must follow agent prompt structure (context, role, task, instructions)
- Eval tests must use YAML format
- Registry entry must follow schema
## Files to Create/Modify
**Create**:
- `.opencode/agent/data/data-analyst.md` - Main agent definition with frontmatter and prompt
- `evals/agents/data/data-analyst/config/eval-config.yaml` - Eval configuration
- `evals/agents/data/data-analyst/tests/smoke-test.yaml` - Basic smoke test
- `evals/agents/data/data-analyst/tests/data-analysis-test.yaml` - Data analysis capability test
- `evals/agents/data/data-analyst/README.md` - Agent documentation
**Modify**:
- `registry.json` - Add data-analyst agent entry
- `.opencode/context/index.md` - Add data category context if needed
## Success Criteria
- [x] Agent file created with proper frontmatter metadata
- [x] Agent prompt follows established patterns (context, role, task, instructions)
- [x] Eval test structure created with config and tests
- [x] Smoke test passes
- [x] Data analysis test passes
- [x] Registry entry added and validates
- [x] README documentation created
- [x] All validation scripts pass
## Validation Requirements
**Scripts to Run**:
- `./scripts/registry/validate-registry.sh` - Validates registry.json schema and entries
- `./scripts/validation/validate-test-suites.sh` - Validates eval test structure
**Tests to Run**:
- `cd evals/framework && npm run eval:sdk -- --agent=data/data-analyst --pattern="smoke-test.yaml"` - Run smoke test
- `cd evals/framework && npm run eval:sdk -- --agent=data/data-analyst` - Run all tests
**Manual Checks**:
- Verify frontmatter includes all required fields
- Check that tools and permissions are appropriate
- Ensure prompt is clear and follows standards
- Verify eval tests are meaningful
## Expected Output
**Deliverables**:
- Functional data analyst agent
- Complete eval test suite
- Registry entry
- Documentation
**Format**:
- Agent file: Markdown with YAML frontmatter
- Eval config: YAML format
- Eval tests: YAML format with test cases
- README: Markdown documentation
## Progress Tracking
- [ ] Context loaded and understood
- [ ] Agent file created with frontmatter
- [ ] Agent prompt written
- [ ] Eval directory structure created
- [ ] Eval config created
- [ ] Smoke test created
- [ ] Data analysis test created
- [ ] README documentation created
- [ ] Registry entry added
- [ ] Validation scripts run
- [ ] All tests pass
- [ ] Documentation updated
---
## Instructions for Subagent
**IMPORTANT**:
1. Load ALL context files listed in "Relevant Standards" and "Repository-Specific Context" sections BEFORE starting work
2. Follow ALL requirements from the loaded context
3. Apply naming conventions and file structure requirements
4. Validate your work using the validation requirements
5. Update progress tracking as you complete steps
**Your Task**:
Create a complete data analyst agent for the OpenAgents repository following all established conventions and standards.
**Approach**:
1. **Load Context**: Read all context files listed above to understand:
- How agents are structured (core-concepts/agents.md)
- How to add an agent (guides/adding-agent.md)
- Code standards (standards/code.md)
- Testing requirements (core-concepts/evals.md)
2. **Create Agent File**:
- Create `.opencode/agent/data/data-analyst.md`
- Add frontmatter with all required metadata
- Write agent prompt with:
- Context section (system, domain, task, execution context)
- Role definition
- Task description
- Instructions and workflow
- Tools and capabilities
- Examples if helpful
3. **Create Eval Structure**:
- Create directory: `evals/agents/data/data-analyst/`
- Create config: `config/eval-config.yaml`
- Create tests directory: `tests/`
- Create smoke test: `tests/smoke-test.yaml`
- Create capability test: `tests/data-analysis-test.yaml`
- Create README: `README.md`
4. **Update Registry**:
- Add entry to `registry.json` following schema
- Include: id, name, description, category, type, path, version, tags
5. **Validate**:
- Run validation scripts
- Run eval tests
- Fix any issues
**Constraints**:
- Agent must be in `data` category
- Must follow functional programming patterns
- Must include proper error handling
- Must specify appropriate tools (read, write, bash for data tasks)
- Temperature should be 0.1-0.3 for analytical precision
- Eval tests must be meaningful and test actual capabilities
**Questions/Clarifications**:
- What specific data analysis capabilities should be emphasized? (visualization, statistics, transformation)
- Should the agent support specific data formats? (CSV, JSON, Parquet)
- Should the agent integrate with specific tools? (pandas, matplotlib, etc.)
- What level of statistical analysis? (descriptive, inferential, predictive)
**Note**: This is an example context bundle. In practice, the subagent would receive this file and follow the instructions to complete the task.
Reference in a new issue View git blame Copy permalink