OpenWriterControle/evals/framework/SESSION_STORAGE_FIX.md

Session Storage Fix - Simplified Approach

Problem Summary

The evaluation framework couldn't find sessions created by the SDK because:

1. Path Mismatch: SDK stores sessions in ~/.local/share/opencode/storage/session/{hash}/ but evaluators looked in ~/.local/share/opencode/project/{encoded-path}/storage/session/info/
2. Hash Calculation: We couldn't reliably calculate the project hash that OpenCode uses
3. Project Path Confusion: Tests run from /evals/framework but sessions created in /opencode-agents (git root)

Solution: SDK-First with Disk Fallback

Instead of reverse-engineering OpenCode's storage format, we now:

1. Use SDK Client Directly (Primary Method)

// SessionReader now accepts SDK client
const sessionReader = new SessionReader(sdkClient, sessionStoragePath);

// Get session via SDK (always up-to-date, no disk delays)
const session = await sessionReader.getSessionInfo(sessionId);

Benefits:

• ✅ No path calculations needed
• ✅ No hash discovery required
• ✅ No waiting for disk writes
• ✅ Always gets latest data
• ✅ Works for any agent, any project

2. Simple Disk Scan (Fallback)

// If SDK unavailable, scan all session directories for the session ID
private findSessionFile(sessionId: string): string | null {
  const sessionBasePath = '~/.local/share/opencode/storage/session';
  
  // Scan all hash directories
  for (const hashDir of fs.readdirSync(sessionBasePath)) {
    const sessionFile = path.join(sessionBasePath, hashDir, `${sessionId}.json`);
    if (fs.existsSync(sessionFile)) {
      return sessionFile;
    }
  }
  
  return null;
}

Benefits:

• ✅ Simple: Just find file by ID
• ✅ No project path matching
• ✅ Works for any agent
• ✅ Resilient fallback

What Was Removed

Complex Logic Eliminated ❌

• Hash calculation (unreliable)
• Git root detection (unnecessary)
• Project path encoding (fragile)
• Multiple fallback paths (confusing)
• Session data polling (slow)
• Project hash caching (complex)

Files Simplified ✅

1. config.ts: Removed complex path calculations, kept only simple helpers
2. session-reader.ts: Now SDK-first, simple disk scan fallback
3. test-runner.ts: Passes SDK client to evaluators, no waiting
4. evaluator-runner.ts: Made async to support SDK calls

Architecture

┌─────────────────┐
│  Test Runner    │
│                 │
│  1. Creates     │──────┐
│     session     │      │
│                 │      │
│  2. Gets        │      │
│     sessionId   │      │
│                 │      │
│  3. Passes SDK  │      │
│     client to   │      │
│     evaluators  │      │
└────────┬────────┘      │
         │               │
         ▼               │
┌─────────────────┐      │
│  Evaluators     │      │
│                 │      │
│  SessionReader  │◄─────┘ SDK Client
│  (SDK-based)    │
│                 │
│  1. Try SDK     │──────► session.get(id)
│     first       │        ✅ Fast, reliable
│                 │
│  2. Fallback    │──────► Scan disk by ID
│     to disk     │        ✅ Simple, works
└─────────────────┘

Testing Different Agents

This approach works for any agent because:

1. No project path dependency: We don't care where the agent runs
2. Session ID is universal: Every session has a unique ID
3. SDK knows everything: The SDK tracks all sessions regardless of project
4. Disk scan is comprehensive: Scans all hash directories

Example: Testing Multiple Agents

// Test OpenAgent
const openAgentTests = await loadTestCases('agents/openagent/tests/**/*.yaml');
await runner.runTests(openAgentTests);

// Test OpenCoder  
const openCoderTests = await loadTestCases('agents/opencoder/tests/**/*.yaml');
await runner.runTests(openCoderTests);

// Works for both! No configuration needed.

Results

Before Fix ❌

Test FAILED
Errors: Evaluator error: Session not found: ses_xxx
Events captured: 4
Violations: N/A (evaluators couldn't run)

After Fix ✅

Test PASSED
Duration: 5063ms
Events: 4
Violations: 0 (0 errors, 0 warnings)
Evaluators: ✅ All ran successfully

Key Takeaways

1. Use the SDK: Don't reverse-engineer storage formats
2. Keep it simple: Scan by ID when SDK unavailable
3. Async all the way: SDK calls are async, embrace it
4. Agent-agnostic: Design for testing any agent, not just one

Files Changed

• src/collector/session-reader.ts - Simplified to SDK-first approach
• src/collector/timeline-builder.ts - Made async for SDK calls
• src/evaluators/evaluator-runner.ts - Added SDK client support, made async
• src/sdk/test-runner.ts - Passes SDK client to evaluators
• src/config.ts - Removed complex path logic, added git root helper

Migration Notes

If you have existing code using SessionReader:

// Old (synchronous, disk-based)
const reader = new SessionReader(projectPath, sessionStoragePath);
const session = reader.getSessionInfo(sessionId);

// New (async, SDK-first)
const reader = new SessionReader(sdkClient, sessionStoragePath);
const session = await reader.getSessionInfo(sessionId);

All SessionReader methods are now async. Update your code accordingly.