cclohmar/OpenWriterControle
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
OpenWriterControle/evals/framework/scripts/README.md
darrenhinde f773b290ce chore(evals): comprehensive cleanup, documentation, and test infrastructure improvements 
Documentation:
- Add ARCHITECTURE.md with comprehensive system review
- Add GETTING_STARTED.md for new users
- Add DOCUMENTATION_CLEANUP.md tracking cleanup changes
- Add SCRIPTS_ORGANIZATION.md documenting script reorganization
- Remove outdated docs (TESTING_CONFIDENCE.md, TEST_REVIEW.md, SESSION_STORAGE_FIX.md)
- Update README files with current information

Script Organization:
- Move 12 scripts from framework root to organized directories:
  - scripts/debug/ (4 files): Session debugging tools
  - scripts/test/ (6 files): Test execution scripts
  - scripts/utils/ (2 files): Utility scripts
- Add comprehensive scripts/README.md with usage examples

Test Infrastructure:
- Add 5 new context loading tests for openagent
- Add result-saver.ts for test result persistence
- Add interactive results dashboard (index.html)
- Add error-handling.test.ts for framework testing
- Add result-saver.test.ts with comprehensive unit tests

Evaluator Improvements:
- Enhance context-loading-evaluator with better detection
- Update run-sdk-tests.ts with improved result handling

Agent Configuration:
- Add .opencode/agent/AGENT.md for agent definition

Test Utilities:
- Add test-agent-manual.mjs for manual testing
- Update test_tmp/README.md with usage guidelines

This commit consolidates multiple improvements:
- Better organization (scripts, docs)
- Improved testing infrastructure
- Comprehensive documentation
- Cleaner codebase structure
2025-11-26 21:34:27 +00:00

4.5 KiB
Raw Blame History

Framework Scripts

Utility scripts for debugging, testing, and development.

Directory Structure

scripts/
├── debug/          # Debugging scripts for sessions and events
├── test/           # Test scripts for framework development
├── utils/          # Utility scripts (batch runner, etc.)
└── README.md       # This file

Debug Scripts (debug/)

Scripts for debugging sessions, events, and agent behavior.

Script Purpose Usage
debug-session.mjs Debug session data and timeline node scripts/debug/debug-session.mjs <session-id>
debug-session.ts TypeScript version of session debugger npx tsx scripts/debug/debug-session.ts <session-id>
debug-claude-session.mjs Debug Claude-specific sessions node scripts/debug/debug-claude-session.mjs <session-id>
inspect-session.mjs Inspect most recent session events node scripts/debug/inspect-session.mjs

Examples

# Debug a specific session
node scripts/debug/debug-session.mjs ses_abc123

# Inspect latest session
node scripts/debug/inspect-session.mjs

# Debug with TypeScript
npx tsx scripts/debug/debug-session.ts ses_abc123

Test Scripts (test/)

Scripts for testing framework components during development.

Script Purpose Usage
test-agent-direct.ts Direct agent execution test npx tsx scripts/test/test-agent-direct.ts
test-event-inspector.js Test event capture system node scripts/test/test-event-inspector.js
test-session-reader.mjs Test session reader node scripts/test/test-session-reader.mjs
test-simplified-approach.mjs Test simplified test approach node scripts/test/test-simplified-approach.mjs
test-timeline.ts Test timeline builder npx tsx scripts/test/test-timeline.ts
verify-timeline.ts Verify timeline accuracy npx tsx scripts/test/verify-timeline.ts

Examples

# Test agent execution
npx tsx scripts/test/test-agent-direct.ts

# Test event capture
node scripts/test/test-event-inspector.js

# Verify timeline
npx tsx scripts/test/verify-timeline.ts

Utility Scripts (utils/)

General utility scripts for running tests and managing the framework.

Script Purpose Usage
run-tests-batch.sh Run tests in batches ./scripts/utils/run-tests-batch.sh <agent> <batch-size> <delay>
check-agent.mjs Check agent availability node scripts/utils/check-agent.mjs

Examples

# Run tests in batches of 3 with 10s delay
./scripts/utils/run-tests-batch.sh openagent 3 10

# Check if agent is available
node scripts/utils/check-agent.mjs

Development Workflow

Debugging a Failed Test

  1. Run test with debug flag:

    npm run eval:sdk -- --pattern="my-test.yaml" --debug

  2. Note the session ID from output

  3. Inspect the session:

    node scripts/debug/inspect-session.mjs
# or
node scripts/debug/debug-session.mjs <session-id>

  4. Check timeline events:

    npx tsx scripts/debug/debug-session.ts <session-id>

Testing Framework Changes

  1. Make changes to framework code

  2. Build:

    npm run build

  3. Test specific component:

    npx tsx scripts/test/test-timeline.ts

  4. Run full test suite:

    npm run eval:sdk

Script Dependencies

All scripts require the framework to be built first:

npm run build

Some scripts use:

  • @opencode-ai/sdk - For SDK client
  • tsx - For TypeScript execution
  • Framework dist files - Built TypeScript output

Adding New Scripts

Debug Script Template

// scripts/debug/my-debug-script.mjs
import { SessionReader } from '../../dist/collector/session-reader.js';
import { createOpencodeClient } from '@opencode-ai/sdk';

const client = createOpencodeClient({
  baseUrl: 'http://localhost:3721'
});

// Your debug logic here

Test Script Template

// scripts/test/my-test-script.ts
#!/usr/bin/env npx tsx

import { TestRunner } from '../../dist/sdk/test-runner.js';

async function runTest() {
  // Your test logic here
}

runTest().catch(console.error);

Maintenance

  • Keep scripts organized - Put debug scripts in debug/, test scripts in test/
  • Update this README - When adding new scripts
  • Remove obsolete scripts - Delete scripts that are no longer needed
  • Document usage - Add clear usage examples

Last Updated: 2025-11-26