f773b290ce
Documentation: - Add ARCHITECTURE.md with comprehensive system review - Add GETTING_STARTED.md for new users - Add DOCUMENTATION_CLEANUP.md tracking cleanup changes - Add SCRIPTS_ORGANIZATION.md documenting script reorganization - Remove outdated docs (TESTING_CONFIDENCE.md, TEST_REVIEW.md, SESSION_STORAGE_FIX.md) - Update README files with current information Script Organization: - Move 12 scripts from framework root to organized directories: - scripts/debug/ (4 files): Session debugging tools - scripts/test/ (6 files): Test execution scripts - scripts/utils/ (2 files): Utility scripts - Add comprehensive scripts/README.md with usage examples Test Infrastructure: - Add 5 new context loading tests for openagent - Add result-saver.ts for test result persistence - Add interactive results dashboard (index.html) - Add error-handling.test.ts for framework testing - Add result-saver.test.ts with comprehensive unit tests Evaluator Improvements: - Enhance context-loading-evaluator with better detection - Update run-sdk-tests.ts with improved result handling Agent Configuration: - Add .opencode/agent/AGENT.md for agent definition Test Utilities: - Add test-agent-manual.mjs for manual testing - Update test_tmp/README.md with usage guidelines This commit consolidates multiple improvements: - Better organization (scripts, docs) - Improved testing infrastructure - Comprehensive documentation - Cleaner codebase structure
74 lines
2.1 KiB
YAML
74 lines
2.1 KiB
YAML
id: ctx-multi-standards-to-docs
|
|
name: "Context Loading: Multi-Turn Standards to Documentation"
|
|
description: |
|
|
Complex multi-turn test: Standards question → Documentation request → Format question
|
|
|
|
Turn 1: Ask about coding standards
|
|
- Expected: Load standards.md or processes.md
|
|
- Validation: Read before response
|
|
|
|
Turn 2: Request documentation creation about standards
|
|
- Expected: Load docs.md (documentation format/template)
|
|
- Validation: Read docs.md before planning/writing
|
|
- Files created: evals/test_tmp/coding-standards-doc.md
|
|
|
|
Turn 3: Ask about documentation structure
|
|
- Expected: Reference both standards.md and docs.md
|
|
- Validation: Should have both files in context
|
|
|
|
Working directory: evals/test_tmp/
|
|
|
|
category: developer
|
|
agent: openagent
|
|
model: anthropic/claude-sonnet-4-5
|
|
|
|
# Multi-turn conversation
|
|
prompts:
|
|
- text: "What are our coding standards?"
|
|
expectContext: true
|
|
contextFile: "standards.md"
|
|
|
|
- text: "approve"
|
|
delayMs: 2000
|
|
|
|
- text: "Can you create documentation about these standards in evals/test_tmp/coding-standards-doc.md?"
|
|
expectContext: true
|
|
contextFile: "docs.md"
|
|
|
|
- text: "approve"
|
|
delayMs: 2000
|
|
|
|
- text: "What will the documentation structure look like?"
|
|
delayMs: 1000
|
|
|
|
# Expected behavior
|
|
behavior:
|
|
mustUseTools: [read, write] # Must read context files and write documentation
|
|
requiresApproval: true # OpenAgent requires approval before writing
|
|
requiresContext: true # Must load context files
|
|
minToolCalls: 3 # At least: read standards + read docs + write file
|
|
|
|
# Expected violations
|
|
expectedViolations:
|
|
- rule: approval-gate
|
|
shouldViolate: false
|
|
severity: error
|
|
description: Must ask approval before writing files
|
|
|
|
- rule: context-loading
|
|
shouldViolate: false
|
|
severity: error
|
|
description: Must load standards.md and docs.md before writing
|
|
|
|
# Approval strategy
|
|
approvalStrategy:
|
|
type: auto-approve
|
|
|
|
timeout: 300000 # 5 minutes for multi-turn (with smart timeout: 5min activity, 10min absolute max)
|
|
|
|
tags:
|
|
- context-loading
|
|
- multi-turn
|
|
- complex-test
|
|
- documentation
|
|
- standards
|