OpenWriterControle/scripts/registry/validate-registry.sh
Darren Hinde 193271091a
feat: add ExternalScout and optimize ContextScout with research-backed patterns (#128)
* feat: add ExternalScout and optimize ContextScout with research-backed patterns

## New Features

### ExternalScout Subagent (v2.0.0)
- Fetches live, version-specific documentation for external libraries via Context7 API
- Supports 18+ libraries: Drizzle, Better Auth, Next.js, TanStack, Cloudflare Workers, AWS Lambda, Vercel, Shadcn/ui, Radix UI, Tailwind CSS, Zustand, Jotai, Zod, React Hook Form, Vitest, Playwright
- Lazy-loads query patterns for specific libraries only
- Filters and formats results by relevance (Critical → High → Medium)
- Fallback to official docs via webfetch
- 42% token reduction via research-backed optimization
- Added to registry with skill:context7 dependency

### Context7 Skill
- Registered in system for external documentation fetching
- Library registry with 18+ supported libraries
- Query optimization patterns per library
- Navigation guide and README

## Optimizations

### ContextScout v5.1.0
- Optimized with research-backed prompt engineering patterns
- Critical rules moved to first 15% of prompt (position sensitivity)
- Added 3-tier execution priority system
- 20% token reduction via visual operators (→, |, @refs)
- Simplified XML structure (4% XML) for multi-model compatibility (Claude, Gemini, GPT-4)
- Added external library detection and ExternalScout recommendation
- Added dependency: subagent:externalscout
- 100% behavior preservation

### ExternalScout v2.0.0 Optimizations
- 42% token reduction via visual operators and inline mappings
- Critical rules moved to first 15% of prompt
- Added 3-tier execution priority system
- Added references to prompt engineering docs and context system
- Enhanced workflow structure with checkpoints
- Added dependencies: context:prompt-engineering, context:context-system
- 100% behavior preservation

## Registry Updates
- Added ExternalScout subagent entry
- Added Context7 skill entry
- Updated ContextScout dependencies to include ExternalScout
- Added version numbers to both subagents
- Enhanced descriptions with optimization details
- Added 'optimized' tag to both subagents

## Files Added
- .opencode/agent/subagents/core/externalscout.md
- .opencode/skill/context7/README.md
- .opencode/skill/context7/library-registry.md
- .opencode/skill/context7/navigation.md
- .opencode/agent/subagents/core/contextscout-v4-backup.md (backup)
- .opencode/agent/subagents/core/externalscout-v1-backup.md (backup)

## Files Modified
- .opencode/agent/subagents/core/contextscout.md (v4.0.0 → v5.1.0)
- registry.json (added ExternalScout, Context7, updated ContextScout)
- CHANGELOG.md (documented all changes)
- VERSION (0.5.5 → 0.6.0)
- package.json (0.5.5 → 0.6.0)

## Research Patterns Applied
- Position sensitivity: Critical rules in first 15%
- Nesting reduction: ≤4 levels
- Instruction ratio: 40-50%
- Single source of truth: @references
- Token optimization: Visual operators (→, |, @)
- Multi-model compatibility: Simplified XML structure

## Breaking Changes
None - 100% backward compatible

## Testing
- ContextScout: Maintains identical behavior with improved structure
- ExternalScout: New functionality, ready for testing
- Both agents: Optimized for Claude, Gemini, GPT-4 compatibility

Closes #[issue-number-if-applicable]

* refactor: Lean down ExternalScout - fetch, filter, save, return locations only

- Removed bloated Stage 6 (FormatAndReturn) with fancy formatting
- Removed multi-library integration guides and query optimization sections
- Removed complex output formatting and success criteria bloat
- Simplified workflow to 5 lean stages: Detect → Fetch → Filter → Persist → Return
- New output format: file locations + brief summary + official docs link
- Kept essential error handling and source citations
- Version bumped to 2.0.0
- Focus: Fast, accurate documentation retrieval without extra bloat

* feat: Add ExternalScout and optimize context system

- Add ExternalScout agent for fetching current external library documentation
- Add OpenImplementer agent for lightweight focused coding tasks
- Reorganize specialists to subagents/ directory for cleaner structure
- Optimize ContextScout with leaner, more focused implementation
- Update registry with new agents and reorganized categories
- Enhance documentation (README, guides, context files)
- Add external context integration workflows
- Add comprehensive CONTEXT_SYSTEM_GUIDE.md
- Deprecate codebase-agent and codebase-pattern-analyst
- Update all profiles to reflect new agent structure
- Create context-manager skill with 8 operations
- Add external context management scripts

BREAKING CHANGES: None (all changes backward compatible)

Fixes: Outdated training data issues for external libraries

* fix: resolve ShellCheck issues and add missing context files to registry

- Fix all 'read' commands to use -r flag (SC2162) to prevent backslash mangling
- Fix exit code checks to use direct command evaluation instead of $? (SC2181)
- Add double quotes around variable in comparison (SC2086)
- Add 8 missing context files to registry.json:
  - external-libraries.md (new workflow for external library handling)
  - visual-development.md
  - development/README.md
  - learning/README.md
  - product/README.md
  - data/README.md
  - mastra-ai/README.md
  - context-bundle-template.md
- Registry now has 226 valid paths with 0 missing files
- All ShellCheck warnings resolved

* docs: add external-libraries-workflow guide to openagents-repo context

- Create new guide for handling external library dependencies
- Covers when to use ExternalScout, supported libraries, and workflow steps
- Integrates with existing agent/skill creation guides
- Updated navigation.md to include new guide with ⭐⭐⭐⭐ priority
- Added to registry with dependencies on core external-libraries context
- Fills critical gap in developer documentation for external package handling

* fix: resolve ShellCheck SC2145 errors in context router

- Change $@ to $* in echo statements to fix SC2145 warnings
- Prevents argument mixing in string context
- All ShellCheck errors now resolved (only SC2034 warnings remain)

* fix: resolve all ShellCheck warnings in manage-external-context.sh

- Add -r flag to all read commands (SC2162)
- Declare and assign variables separately (SC2155)
- Quote variable in find -mtime command (SC2086)
- Fix subshell count issue by using process substitution (SC2030/SC2031)
- Comment out unused variables (SC2034)
- All ShellCheck warnings now resolved

* fix: remove unused variables in context router to pass ShellCheck

- Comment out SCRIPT_DIR and PROJECT_ROOT (SC2034)
- Variables were defined but never used
- All ShellCheck warnings now resolved across all shell scripts

* fix: replace bash registry validation with Python script

- Create fast Python validation script (validate-registry.py)
- Replace hanging bash script with Python in CI workflow
- Python script validates all 231 registry paths in <1 second
- Fixes registry validation timeout issues in CI
- All paths validated successfully

* chore: trigger CI re-run for workflow updates

* Revert "fix: replace bash registry validation with Python script"

This reverts commit 03843935d075f1350afb721b5235e9406005810f.

* fix: remove verbose flag and add timeout to registry validation

- Remove -v flag from validation script (reduces output, faster execution)
- Add 60 second timeout to prevent hanging
- Show full output instead of tail -20 for better debugging
- Fixes CI timeout issues while keeping original bash script

* feat: add TypeScript registry validator with full skill support

- Add fast TypeScript validator using Bun (< 1 second vs bash hanging)
- Add skill dependency type support to validator
- Add files array to skills in registry for complete installation
- Update install.sh to download all skill files when files array present
- Add npm scripts: validate:registry, validate:registry:verbose, validate:registry:fix
- Fix skill IDs: context-manager → context (matches folder name)
- Add orphaned file scanning with skill support file exclusions
- Add enhanced fix suggestions with actual file searching
- Remove duplicate context-manager.md file
- Add comprehensive README for registry validation scripts

Skills now install completely with all supporting files:
- SKILL.md (main definition)
- router.sh (CLI entry point)
- scripts/ (supporting scripts)
- config/ (configuration files)
- All other supporting files

Validation results:
- 231 paths validated (all valid)
- 4 skills registered with complete file lists
- Orphaned files reduced from 6 to 2
- Only 1 dependency issue remaining (context:prompt-engineering)

* fix: use TypeScript validator in CI instead of hanging bash script

- Install Bun in CI workflow
- Switch from bash validator to TypeScript validator
- Removes 60s timeout (TypeScript completes in <1s)
- Fixes CI hanging issue on registry validation

* chore: retrigger CI checks

Empty commit to retrigger CI after workflow fix in PR #129

* fix: use official Bun setup action in CI

- Replace manual Bun installation with oven-sh/setup-bun@v2
- Fixes PATH issue where bun wasn't available in subsequent steps
- Ensures Bun is properly installed before running TypeScript validator

* chore: retrigger CI after Bun setup fix in PR #131

Empty commit to retrigger CI once PR #131 is merged to main

* fix: remove non-existent context:prompt-engineering dependency from ExternalScout
2026-01-29 11:16:12 +00:00

554 lines
18 KiB
Bash
Executable file
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

#!/usr/bin/env bash
#############################################################################
# Registry Validator Script
# Validates that all paths in registry.json point to actual files
# Exit codes:
# 0 = All paths valid
# 1 = Missing files found
# 2 = Registry parse error or missing dependencies
#############################################################################
set -e
# Colors
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
CYAN='\033[0;36m'
BOLD='\033[1m'
NC='\033[0m'
# Configuration
REGISTRY_FILE="registry.json"
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
VERBOSE=false
FIX_MODE=false
# Counters
TOTAL_PATHS=0
VALID_PATHS=0
MISSING_PATHS=0
ORPHANED_FILES=0
MISSING_DEPENDENCIES=0
# Arrays to store results
declare -a MISSING_FILES
declare -a ORPHANED_COMPONENTS
declare -a MISSING_DEPS
#############################################################################
# Utility Functions
#############################################################################
print_header() {
echo -e "${CYAN}${BOLD}"
echo "╔════════════════════════════════════════════════════════════════╗"
echo "║ ║"
echo "║ Registry Validator v1.0.0 ║"
echo "║ ║"
echo "╚════════════════════════════════════════════════════════════════╝"
echo -e "${NC}"
}
print_success() {
echo -e "${GREEN}✓${NC} $1"
}
print_error() {
echo -e "${RED}✗${NC} $1"
}
print_warning() {
echo -e "${YELLOW}⚠${NC} $1"
}
print_info() {
echo -e "${BLUE}ℹ${NC} $1"
}
usage() {
echo "Usage: $0 [OPTIONS]"
echo ""
echo "Options:"
echo " -v, --verbose Show detailed validation output"
echo " -f, --fix Suggest fixes for missing files"
echo " -h, --help Show this help message"
echo ""
echo "Exit codes:"
echo " 0 = All paths valid"
echo " 1 = Missing files found"
echo " 2 = Registry parse error or missing dependencies"
exit 0
}
#############################################################################
# Dependency Checks
#############################################################################
check_dependencies() {
local missing_deps=()
if ! command -v jq &> /dev/null; then
missing_deps+=("jq")
fi
if [ ${#missing_deps[@]} -ne 0 ]; then
print_error "Missing required dependencies: ${missing_deps[*]}"
echo ""
echo "Please install them:"
echo " macOS: brew install ${missing_deps[*]}"
echo " Ubuntu: sudo apt-get install ${missing_deps[*]}"
echo " Fedora: sudo dnf install ${missing_deps[*]}"
exit 2
fi
}
#############################################################################
# Registry Validation
#############################################################################
validate_registry_file() {
if [ ! -f "$REGISTRY_FILE" ]; then
print_error "Registry file not found: $REGISTRY_FILE"
exit 2
fi
if ! jq empty "$REGISTRY_FILE" 2>/dev/null; then
print_error "Registry file is not valid JSON"
exit 2
fi
print_success "Registry file is valid JSON"
}
validate_component_paths() {
local category=$1
local category_display=$2
echo "Checking ${category_display}..." >&2
# Get all components in this category
local components
components=$(jq -r ".components.${category}[]? | @json" "$REGISTRY_FILE" 2>/dev/null)
if [ -z "$components" ]; then
echo "No ${category_display} found" >&2
return
fi
while IFS= read -r component; do
local id
id=$(echo "$component" | jq -r '.id')
local path
path=$(echo "$component" | jq -r '.path')
local name
name=$(echo "$component" | jq -r '.name')
TOTAL_PATHS=$((TOTAL_PATHS + 1))
# Check if file exists
if [ -f "$REPO_ROOT/$path" ]; then
VALID_PATHS=$((VALID_PATHS + 1))
[ "$VERBOSE" = true ] && print_success "${category_display}: ${name} (${id})"
else
MISSING_PATHS=$((MISSING_PATHS + 1))
MISSING_FILES+=("${category}:${id}|${name}|${path}")
print_error "${category_display}: ${name} (${id}) - File not found: ${path}"
# Try to find similar files if in fix mode
if [ "$FIX_MODE" = true ]; then
suggest_fix "$path" "$id"
fi
fi
done <<< "$components"
}
suggest_fix() {
local missing_path=$1
local component_id=$2
# Extract directory and filename
local dir
dir=$(dirname "$missing_path")
# local filename=$(basename "$missing_path") # Unused
local base_dir
base_dir=$(echo "$dir" | cut -d'/' -f1-3) # e.g., .opencode/command
# Look for similar files in the expected directory and subdirectories
local similar_files
similar_files=$(find "$REPO_ROOT/$base_dir" -type f -name "*.md" 2>/dev/null | grep -i "$component_id" || true)
if [ -n "$similar_files" ]; then
echo -e " ${YELLOW}→ Possible matches:${NC}"
while IFS= read -r file; do
local rel_path="${file#$REPO_ROOT/}"
echo -e " ${CYAN}${rel_path}${NC}"
done <<< "$similar_files"
fi
}
scan_for_orphaned_files() {
[ "$VERBOSE" = true ] && echo -e "\n${BOLD}Scanning for orphaned files...${NC}"
# Get all paths from registry
local registry_paths
registry_paths=$(jq -r '.components | to_entries[] | .value[] | .path' "$REGISTRY_FILE" 2>/dev/null | sort -u)
# Scan .opencode directory for markdown files
local categories=("agent" "command" "tool" "plugin" "context")
for category in "${categories[@]}"; do
local category_dir="$REPO_ROOT/.opencode/$category"
if [ ! -d "$category_dir" ]; then
continue
fi
# Find all .md and .ts files (excluding node_modules)
while IFS= read -r file; do
local rel_path="${file#$REPO_ROOT/}"
# Skip node_modules
if [[ "$rel_path" == *"/node_modules/"* ]]; then
continue
fi
# Skip README files
if [[ "$rel_path" == *"README.md" ]]; then
continue
fi
# Skip template files
if [[ "$rel_path" == *"-template.md" ]]; then
continue
fi
# Skip tool/plugin TypeScript files
if [[ "$rel_path" == *"/tool/index.ts" ]] || [[ "$rel_path" == *"/tool/template/index.ts" ]]; then
continue
fi
if [[ "$rel_path" == *"/plugin/agent-validator.ts" ]]; then
continue
fi
# Skip plugin internal docs and tests
if [[ "$rel_path" == *"/plugin/docs/"* ]] || [[ "$rel_path" == *"/plugin/tests/"* ]]; then
continue
fi
# Skip scripts directories (internal CLI tools, not registry components)
if [[ "$rel_path" == *"/scripts/"* ]]; then
continue
fi
# Check if this path is in registry
if ! echo "$registry_paths" | grep -q "^${rel_path}$"; then
ORPHANED_FILES=$((ORPHANED_FILES + 1))
ORPHANED_COMPONENTS+=("$rel_path")
[ "$VERBOSE" = true ] && print_warning "Orphaned file (not in registry): ${rel_path}"
fi
done < <(find "$category_dir" -type f \( -name "*.md" -o -name "*.ts" \) 2>/dev/null)
done
}
#############################################################################
# Dependency Validation
#############################################################################
check_dependency_exists() {
local dep=$1
# Parse dependency format: type:id
if [[ ! "$dep" =~ ^([^:]+):(.+)$ ]]; then
echo "invalid_format"
return 1
fi
local dep_type="${BASH_REMATCH[1]}"
local dep_id="${BASH_REMATCH[2]}"
# Map dependency type to registry category
local registry_category=""
case "$dep_type" in
agent)
registry_category="agents"
;;
subagent)
registry_category="subagents"
;;
command)
registry_category="commands"
;;
tool)
registry_category="tools"
;;
plugin)
registry_category="plugins"
;;
context)
registry_category="contexts"
;;
config)
registry_category="config"
;;
*)
echo "unknown_type"
return 1
;;
esac
# Check if component exists in registry
# First try exact ID match
local exists
exists=$(jq -r ".components.${registry_category}[]? | select(.id == \"${dep_id}\") | .id" "$REGISTRY_FILE" 2>/dev/null)
if [ -n "$exists" ]; then
echo "found"
return 0
fi
# For context dependencies, also try path-based lookup
# Format: context:core/standards/code -> .opencode/context/core/standards/code.md
if [ "$dep_type" = "context" ]; then
# Check for wildcard pattern (e.g., context:core/context-system/*)
if [[ "$dep_id" == *"*" ]]; then
# Extract prefix before wildcard
local prefix="${dep_id%%\**}"
# Check if any context files match the prefix
local matches
matches=$(jq -r ".components.${registry_category}[]? | select(.path | startswith(\".opencode/context/${prefix}\")) | .id" "$REGISTRY_FILE" 2>/dev/null | head -1)
if [ -n "$matches" ]; then
echo "found"
return 0
fi
else
# Try exact path match
local context_path=".opencode/context/${dep_id}.md"
local exists_by_path
exists_by_path=$(jq -r ".components.${registry_category}[]? | select(.path == \"${context_path}\") | .id" "$REGISTRY_FILE" 2>/dev/null)
if [ -n "$exists_by_path" ]; then
echo "found"
return 0
fi
fi
fi
echo "not_found"
return 1
}
validate_component_dependencies() {
echo ""
print_info "Validating component dependencies..."
echo ""
# Get all component types
local component_types
component_types=$(jq -r '.components | keys[]' "$REGISTRY_FILE" 2>/dev/null)
while IFS= read -r comp_type; do
# Get all components of this type
local components
components=$(jq -r ".components.${comp_type}[]? | @json" "$REGISTRY_FILE" 2>/dev/null)
if [ -z "$components" ]; then
continue
fi
while IFS= read -r component; do
local id
id=$(echo "$component" | jq -r '.id')
local name
name=$(echo "$component" | jq -r '.name')
local dependencies
dependencies=$(echo "$component" | jq -r '.dependencies[]?' 2>/dev/null)
if [ -z "$dependencies" ]; then
continue
fi
# Check each dependency
while IFS= read -r dep; do
if [ -z "$dep" ]; then
continue
fi
local result
result=$(check_dependency_exists "$dep")
case "$result" in
found)
[ "$VERBOSE" = true ] && print_success "Dependency OK: ${name} → ${dep}"
;;
not_found)
MISSING_DEPENDENCIES=$((MISSING_DEPENDENCIES + 1))
MISSING_DEPS+=("${comp_type}|${id}|${name}|${dep}")
print_error "Missing dependency: ${name} (${comp_type%s}) depends on \"${dep}\" (not found in registry)"
;;
invalid_format)
MISSING_DEPENDENCIES=$((MISSING_DEPENDENCIES + 1))
MISSING_DEPS+=("${comp_type}|${id}|${name}|${dep}")
print_error "Invalid dependency format: ${name} (${comp_type%s}) has invalid dependency \"${dep}\" (expected format: type:id)"
;;
unknown_type)
MISSING_DEPENDENCIES=$((MISSING_DEPENDENCIES + 1))
MISSING_DEPS+=("${comp_type}|${id}|${name}|${dep}")
print_error "Unknown dependency type: ${name} (${comp_type%s}) has unknown dependency type in \"${dep}\""
;;
esac
done <<< "$dependencies"
done <<< "$components"
done <<< "$component_types"
}
#############################################################################
# Reporting
#############################################################################
print_summary() {
echo ""
echo -e "${BOLD}═══════════════════════════════════════════════════════════════${NC}"
echo -e "${BOLD}Validation Summary${NC}"
echo -e "${BOLD}═══════════════════════════════════════════════════════════════${NC}"
echo ""
echo -e "Total paths checked: ${CYAN}${TOTAL_PATHS}${NC}"
echo -e "Valid paths: ${GREEN}${VALID_PATHS}${NC}"
echo -e "Missing paths: ${RED}${MISSING_PATHS}${NC}"
echo -e "Missing dependencies: ${RED}${MISSING_DEPENDENCIES}${NC}"
if [ "$VERBOSE" = true ]; then
echo -e "Orphaned files: ${YELLOW}${ORPHANED_FILES}${NC}"
fi
echo ""
local has_errors=false
# Check for missing paths
if [ $MISSING_PATHS -gt 0 ]; then
has_errors=true
print_error "Found ${MISSING_PATHS} missing file(s)"
echo ""
echo "Missing files:"
for entry in "${MISSING_FILES[@]}"; do
IFS='|' read -r cat_id name path <<< "$entry"
echo " - ${path} (${cat_id})"
done
echo ""
if [ "$FIX_MODE" = false ]; then
print_info "Run with --fix flag to see suggested fixes"
echo ""
fi
fi
# Check for missing dependencies
if [ $MISSING_DEPENDENCIES -gt 0 ]; then
has_errors=true
print_error "Found ${MISSING_DEPENDENCIES} missing or invalid dependencies"
echo ""
echo "Missing dependencies:"
for entry in "${MISSING_DEPS[@]}"; do
IFS='|' read -r comp_type id name dep <<< "$entry"
echo " - ${name} (${comp_type%s}) → ${dep}"
done
echo ""
print_info "Fix by either:"
echo " 1. Adding the missing component to the registry"
echo " 2. Removing the dependency from the component's frontmatter"
echo ""
fi
# Success case
if [ "$has_errors" = false ]; then
print_success "All registry paths are valid!"
print_success "All component dependencies are valid!"
if [ $ORPHANED_FILES -gt 0 ] && [ "$VERBOSE" = true ]; then
echo ""
print_warning "Found ${ORPHANED_FILES} orphaned file(s) not in registry"
echo ""
echo "Orphaned files:"
for file in "${ORPHANED_COMPONENTS[@]}"; do
echo " - $file"
done
echo ""
echo "Consider adding these to registry.json or removing them."
fi
return 0
else
echo "Please fix these issues before proceeding."
return 1
fi
}
#############################################################################
# Main
#############################################################################
main() {
# Parse arguments
while [ $# -gt 0 ]; do
case "$1" in
-v|--verbose)
VERBOSE=true
shift
;;
-f|--fix)
FIX_MODE=true
VERBOSE=true
shift
;;
-h|--help)
usage
;;
*)
echo "Unknown option: $1"
usage
;;
esac
done
print_header
# Check dependencies
check_dependencies
# Validate registry file
validate_registry_file
echo ""
print_info "Validating component paths..."
echo ""
# Validate each category
validate_component_paths "agents" "Agents"
validate_component_paths "subagents" "Subagents"
validate_component_paths "commands" "Commands"
validate_component_paths "tools" "Tools"
validate_component_paths "plugins" "Plugins"
validate_component_paths "contexts" "Contexts"
validate_component_paths "config" "Config"
# Validate component dependencies
validate_component_dependencies
# Scan for orphaned files if verbose
if [ "$VERBOSE" = true ]; then
scan_for_orphaned_files
fi
# Print summary and exit with appropriate code
if print_summary; then
exit 0
else
exit 1
fi
}
main "$@"