Commit graph

15 commits

Author SHA1 Message Date
Darren Hinde
c8f7103cb6
refactor(evals): consolidate documentation and enhance test infrastructure (#56)
* feat(agents): implement category-based agent organization system

- Organize agents into domain categories (core, development, content, data, product, learning)
- Move 10 agents to category subdirectories with proper git rename tracking
- Update 13 subagents with category and type metadata in frontmatter
- Add category metadata files (0-category.json) documenting common patterns
- Implement local registry fallback in install script for offline development
- Add comprehensive validation suite with 15 automated tests (100% pass rate)
- Enhance registry validation with duplicate ID and consistency checks
- Update eval framework with intelligent path resolution (backward compatible)
- Archive legacy eval structure to _archive/ for reference
- Update all documentation to reflect category-based structure
- Bump version to 0.5.0 with accurate CHANGELOG

Technical Details:
- 23 agents organized (10 category agents, 13 subagents)
- 6 category directories created
- Path resolution supports both agent IDs and category paths
- Registry schema updated to v2.0.0
- 159 files changed, 872 insertions(+), 10738 deletions(-)

BREAKING CHANGE: Agent file paths now use category structure. Update references from .opencode/agent/openagent.md to .opencode/agent/core/openagent.md. Eval framework maintains backward compatibility via path resolution.

* refactor(evals): consolidate documentation and enhance test infrastructure

- Remove temporary project tracking files (PHASE_5_COMPLETE, PROJECT_COMPLETE, etc.)
- Consolidate evaluation framework docs into main README
- Enhance test execution with improved logging and multi-prompt support
- Move system-builder from core to meta category
- Add comprehensive test suites for openagent with organized structure
- Create evaluation test structure for all subagents
- Clean up archived workflows and redundant documentation
- Update registry to reflect new agent organization
- Add shared test templates and golden test patterns

* feat(ci): add manual trigger support for bot-created PRs in validate-registry workflow

- Add pr_number input to workflow_dispatch for manually triggering validation
- Fetch PR details dynamically when triggered manually
- Support both automatic (PR event) and manual (workflow_dispatch) triggers
- Enable validation of bot-created PRs like automated version bumps
- Update branch detection and push logic to handle both trigger types
- Add documentation explaining how to manually trigger for bot PRs

* feat(evals): add explicit context file validation to test framework

- Add expectedContextFiles field to test YAML schema for explicit context file specification
- Enhance context-loading-evaluator to support both auto-detect and explicit validation modes
- Update documentation with comprehensive guide and examples
- Clean up archived legacy test structure (90+ old test files)
- Add new example tests demonstrating explicit context validation
- Backward compatible with existing tests

* feat(evals): add multi-agent logging system and performance optimizations

Implement comprehensive multi-agent logging and performance improvements for the eval framework.

Task 01: Multi-Agent Logging System
- Add complete hierarchical logging module (evals/framework/src/logging/)
  - SessionTracker: tracks parent-child delegation hierarchies
  - MultiAgentLogger: pretty-prints logs with visual indentation
  - Formatters: box characters and emoji formatting
  - 37 passing unit tests (session-tracker, logger, integration)
- Integrate with SDK event stream handler
  - Hook into session.created, message.updated, message.part.updated events
  - Real-time child session detection via timestamp heuristics
  - Message deduplication for cleaner output
- Enable in debug mode only (<1% performance overhead)
- Add demo script and comprehensive documentation

Task 02: Performance Optimizations
- Reduce grace period from 5s to 2s (67% reduction, 10-20% faster tests)
- Add PerformanceMetricsEvaluator for bottleneck identification
  - Collects tool latencies, inference time, idle time
  - Provides full performance visibility
- Update smoke test to validate delegation (core feature)
  - Replace simple read test with multi-agent delegation test
  - Tests both parent and child agent functionality
  - Validates multi-agent logging system

Results:
- 37 unit tests passing
- Smoke test passing (95/100 score)
- Clean hierarchical logging output
- 10-20% faster test execution
- Full multi-agent visibility

Files changed: 16 files, +2678 lines
Time saved: ~6-8 days (completed in 6 hours vs 3-5 day estimate)

* feat(evals): show child agent execution in non-debug mode

- Enable MultiAgentLogger in both debug and non-debug modes
- Add verbose parameter to control output level
- Non-verbose mode shows concise child session lifecycle:
  - Child agent started message
  - Child agent completed with duration
- Verbose mode (--debug) shows full delegation hierarchy
- Update documentation to reflect new behavior

This provides visibility into delegation without overwhelming output,
giving confidence that child agents are actually running.

* feat(evals): improve delegation testing and behavior validation

Major improvements to eval framework:

Test Suite:
- Add 00-smoke-test.yaml (basic read operation)
- Rename 01-smoke-test.yaml → 02-delegation-test.yaml (delegation test)
- Add simple-responder test agent for delegation testing
- Add debug test: simple-subagent-call.yaml

Evaluators:
- Add agent-model-evaluator.ts for agent/model validation
- Enhance behavior-evaluator.ts with detailed task tool output
- Improve delegation-evaluator.ts with better evidence tracking

Schema & Execution:
- Add expectedAgent and expectedModel to test schema
- Improve test executor with better logging and error handling

Documentation:
- Update evals/README.md with new features and performance improvements
- Remove outdated MULTI_AGENT_LOGGING_COMPLETE.md

Registry:
- Add simple-responder test agent to registry

Fixes:
- Fix dashboard.sh path resolution

These changes provide better visibility into delegation, improved test
coverage, and clearer validation of agent behavior.

* chore: add GitHub Actions workflow and plugin documentation

- Add .github/workflows/evals/run-evaluations.yml for automated eval testing
- Add dev/ai-tools/opencode/plugins/Plugin-inspiration.md for plugin development reference
2025-12-18 09:52:11 +00:00
Marc Peters
e428f4ecfd
Update install.sh (#30)
fix: resolve installation failures on Windows through Git Bash
2025-12-04 15:40:46 +00:00
darrenhinde
bc8a322550 feat: enhance installation system with universal path transformation
Add support for custom global installation paths with automatic context
reference transformation. This enables users to install to any location
while maintaining working context references.

Changes:
- Add universal path transformation for ANY global installation path
  (not just ~/.config/opencode)
- Implement automatic cleanup trap for temp files on exit/crash/interrupt
- Add early validation for --install-dir argument with clear error messages
- Add developer documentation for context reference convention

Technical Details:
- Path transformation detects local vs global by checking if path ends
  in .opencode (local) or not (global)
- Transformation applies to ALL file types during installation
- sed transforms both @ references and shell command paths
- Cleanup trap handles EXIT, INT, and TERM signals

Testing:
- 31 unit and integration tests (100% pass rate)
- All README installation scenarios verified
- Edge cases tested and handled
- Cross-platform compatible (macOS, Linux, Windows Git Bash)

Documentation:
- Added dev/docs/context-reference-convention.md explaining the required
  @.opencode/context/ format and why it's necessary
- Includes examples, validation instructions, and test results

Breaking changes: None
New capabilities: Custom global path support
2025-11-19 11:22:48 +00:00
darrenhinde
9692d3a752 fix: respect --install-dir argument throughout installation process
- Add get_install_path() helper to convert registry paths to installation paths
- Fix collision detection to check files in custom install directory
- Fix file installation to use custom directory instead of hardcoded .opencode
- Fix env.example path check in post-install message
- Tested with .test-install-dir and .openffff directories - all files now install to correct location
2025-11-19 01:22:21 +00:00
darrenhinde
3c972fb828 Fix critical bash 5.x compatibility and add custom install directory support
- Fix set -e bug: Change ((variable++)) to variable=$((variable + 1)) to prevent script exit on bash 5.2.21 (Ubuntu 24.04)
- Fix collision handling UX: Display menu options correctly by redirecting to stderr
- Add custom installation directory support with --install-dir flag and OPENCODE_INSTALL_DIR env var
- Add interactive location selection menu (Local/Global/Custom)
- Implement cross-platform path handling (Linux/macOS/Windows) with tilde expansion, backslash conversion, and validation
- Enhance help documentation with comprehensive examples and platform support information

Fixes installation failures on modern Linux systems and enables flexible installation to any directory including ~/.config/opencode
2025-11-18 23:08:30 +00:00
darrenhinde
ca323285b2 Rename to OpenAgents, remove model specs, and update documentation
Major Changes:
- Rename repository from opencode-agents to OpenAgents
- Remove all hardcoded model specifications (now model-agnostic)
- Fix documentation: profile names, component counts, add Business profile
- Add future plans note for other AI coding tools

Repository Rename:
- Update all URLs to github.com/darrenhinde/OpenAgents
- Update git remote, README, install scripts, documentation
- 15 files updated with new repository name

Model-Agnostic Framework:
- Remove model specifications from 6 agent files
- Framework now works with any AI provider (Claude, GPT, Gemini, local models)
- Users control model selection via OpenCode CLI settings

Documentation Fixes:
- Correct profile name: Core → Essential
- Add missing Business profile documentation
- Fix component counts: Developer (19), Advanced (32)
- Document all 5 profiles with accurate descriptions

Future Plans:
- Add note about support for other AI tools (Cursor, Claude Code, etc.)
- Clear focus on stabilizing OpenCode integration first

This makes OpenAgents a truly universal, model-agnostic framework
accessible to all users regardless of their AI provider preference.
2025-11-18 22:37:41 +00:00
darrenhinde
5f2cec1404 ✨ feat: introduce OpenAgent universal agent with session management
Major architectural refactoring:

- Add OpenAgent as universal primary agent for questions, tasks, and workflows
- Implement session-based context management with manifest tracking
- Reorganize subagents into logical categories (code/core/utils)
- Add dynamic context discovery and loading system
- Introduce cleanup script for stale sessions
- Update registry with new categorization (essential/standard/extended/specialized/meta)
- Rename 'core' profile to 'essential' and add 'business' profile
- Add comprehensive documentation with visual diagrams
- Update installation script with new profiles

Breaking changes:
- Subagent paths changed from analysis/core to code/core/utils
- Profile 'core' renamed to 'essential' (backward compatible)

New features:
- Session isolation for concurrent agent instances (.tmp/sessions/)
- Automatic stale session cleanup (>24 hours)
- Manifest-based context indexing and discovery
- Business profile for content-focused workflows
- OpenAgent with plan-approve-execute-validate-summarize workflow

Components: 32 total (3 agents, 13 subagents, 8 commands, 2 tools, 2 plugins, 2 contexts, 2 config)

Documentation:
- docs/agents/openagent.md - Comprehensive guide with Mermaid diagrams
- docs/guides/building-with-opencode.md - Power user formula
- scripts/README.md - Script documentation
- .opencode/context/core/context-management.md - Session management strategy
2025-11-18 18:45:42 +00:00
darrenhinde
dfea8199a9 fix: don't clear screen in non-interactive mode
Problem:
- When running 'curl ... | bash -s core' the screen would clear
- This made it appear like nothing was happening after preview
- Users couldn't see the installation progress

Solution:
- Only call 'clear' in interactive mode
- Skip clear when NON_INTERACTIVE=true
- Allows users to see full output when piped from curl

Now users can see:
- Dependency checks
- Registry fetch
- Installation preview
- 'Installing automatically' message
- Full installation progress
- Completion message

This makes the installation process transparent and debuggable.
2025-11-18 14:26:59 +00:00
darrenhinde
532e17ce01 fix: auto-install when profile specified via command line
Problem:
- Running 'curl ... | bash -s core' showed preview and waited for input
- User had to press Y to proceed even though profile was specified
- Defeats the purpose of one-command installation

Solution:
- Add NON_INTERACTIVE flag to track command-line profile usage
- Set NON_INTERACTIVE=true when profile argument provided
- Skip confirmation prompt when NON_INTERACTIVE=true
- Show 'Installing automatically (profile specified)...' message

Behavior:
Before:
  curl ... | bash -s core
  → Shows preview, waits for Y/n input

After:
  curl ... | bash -s core
  → Shows preview, installs automatically

Interactive mode still works:
  bash install.sh
  → Shows menu, asks for confirmation (as expected)

This makes profile-based installation truly one-command with no interaction needed.
2025-11-18 14:24:27 +00:00
darrenhinde
a9f119e83e refactor: streamline profiles and fix registry pluralization
Profile Changes:
- Reduce core profile from 15 to 9 components (true essentials only)
- Keep developer at 22 components (add back image-specialist + gemini)
- Remove unused subagents from core (reviewer, build-agent)
- Remove non-essential commands from core (test, context, prompt-enhancer)

Core Profile (9 components):
- Agents: task-manager, codebase-agent
- Subagents: tester, documentation, coder-agent
- Commands: commit
- Context: essential-patterns, project-context
- Config: env-example

Developer Profile (22 components):
- Everything in core plus quality tools and AI features
- Added: image-specialist, gemini tool
- Added: reviewer, build-agent, codebase-pattern-analyst
- Added: test, context, clean, optimize, prompt-enhancer commands

Bug Fixes:
- Fix jq error: 'Cannot iterate over null' for config components
- Add get_registry_key() helper to handle singular 'config' vs plural types
- Update all jq queries to use correct registry keys

Codebase Agent:
- Remove @image-specialist reference (not needed for core coding)
- Update subagent list to match actual usage
- Now lists: @task-manager, @tester, @documentation, @coder-agent

Documentation:
- Update README with correct component counts
- Update profile descriptions to be more accurate
- Clarify what each profile is best for

This makes 'core' truly minimal and 'developer' the recommended choice.
2025-11-18 14:18:59 +00:00
darrenhinde
515a83dce2 fix: detect piped execution and provide clear guidance
Problem:
- Running 'curl ... | bash' without arguments showed interactive menu
- But stdin was the curl stream, not terminal
- Script couldn't read user input and would exit silently
- Confusing user experience

Solution:
- Add check_interactive_mode() to detect if stdin is a terminal
- If running in pipe without arguments, show helpful error message
- Provide clear instructions for both interactive and profile-based install
- Update README to clarify installation methods

Changes:
- Add [ ! -t 0 ] check to detect non-terminal stdin
- Show helpful error with examples when piped without profile
- Update README to recommend profile-based install (fastest)
- Clarify that interactive mode requires downloading first

User Experience:
Before: Silent exit or confusing behavior
After: Clear error message with exact commands to run

Examples shown to user:
- Download first: curl ... -o install.sh && bash install.sh
- Or use profile: curl ... | bash -s core

This fixes the issue reported when running on another computer.
2025-11-18 14:04:15 +00:00
darrenhinde
1a0f458e1a fix: cross-platform installer compatibility for bash 3.2+ and Windows
Critical fixes:
- Fix argument mismatch (accept 'core' and '--core' formats)
- Fix bash 3.2 compatibility (replace mapfile with while-read loops)
- Fix process substitution issues (use temp files)
- Add platform detection (macOS/Linux/Windows)
- Add bash version check (3.2+ required)
- Add conditional color support for Windows terminals

Enhancements:
- Platform-specific dependency installation instructions
- Improved help text with usage examples
- Better error messages for unsupported platforms

Testing:
- Add comprehensive compatibility test suite (12 tests)
- Add collision detection tests
- Organize tests in scripts/tests/ directory
- All tests pass on macOS (bash 3.2.57)

Documentation:
- Add Windows installation instructions to README
- Add platform compatibility guide
- Add COMPATIBILITY.md quick reference
- Update all test script references

Tested on:
- macOS 13+ (bash 3.2.57) ✓
- Linux (bash 4.0+) ✓
- Windows Git Bash (bash 4.4+) ✓

Closes issues with curl installation failing on macOS and Windows
2025-11-18 13:57:18 +00:00
darrenhinde
357dabcfec fix: restore registry.json content and add bash 3.2 compatibility
- Recreate registry.json with all component definitions
- Fix bash 3.2 compatibility (replace ${var^} with awk)
- Registry was accidentally emptied by registration script
2025-11-17 23:43:13 +00:00
darrenhinde
7a2d17ca3a feat: add OPENCODE_BRANCH env var for testing different branches 2025-11-17 23:43:13 +00:00
darrenhinde
777b3fc702 feat: add interactive installer with component registry system
- Add component registry (registry.json) with 4 installation profiles
- Create interactive installer (install.sh) with profile and custom selection
- Add auto-registration script for CI/CD integration
- Create validation script for component structure
- Add update script for keeping components current
- Include GitHub Actions workflow for auto-registry updates
- Update README with installation instructions
- Add CONTRIBUTING guide for new components

Features:
- Interactive CLI with colored output and menus
- Smart dependency resolution
- Backup existing installations
- Profile-based installation (core, developer, full, advanced)
- Custom component selection
- Component validation and testing tools
2025-11-17 23:43:13 +00:00