feat: ExecutionBalanceEvaluator and docs (#27)

* feat(evals): add ExecutionBalanceEvaluator, docs, tests, version bumps

* chore: remove obsolete package-lock in evals/framework pre-rebase

* chore: translate ExecutionBalanceEvaluator and test files from Spanish to English

- Translate all code comments and documentation in execution-balance-evaluator.ts
- Translate test descriptions and prompts in execution-balance-positive.yaml
- Translate test descriptions and prompts in execution-balance-negative.yaml

This ensures consistency with the rest of the English codebase as requested in PR review.

* fix(docs): translate Spanish content to English in documentation

---------

Co-authored-by: Alexander Daza <dev.alexander@example.com>
Co-authored-by: Darren Hinde <107584450+darrenhinde@users.noreply.github.com>

CHANGELOG.md

@@ -7,6 +7,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+_No unreleased changes yet._
+
+## [0.0.2] - 2025-11-29
+
+### Added
+- New `ExecutionBalanceEvaluator` in `evals/framework` to assess read vs execution ordering and ratio.
+- Contributor guide: `docs/contributing/ADDING_EVALUATOR.md` (English) describing evaluator design principles.
+- Test cases under `evals/agents/openagent/tests/10-execution-balance/` (positive & negative scenarios).
+
+### Changed
+- Framework README updated with section documenting `ExecutionBalanceEvaluator` and violation codes.
+
 ---
 
 ## Version Format

VERSION

@@ -1 +1 @@
-0.0.1
+0.0.2

docs/contributing/ADDING_EVALUATOR.md

@@ -0,0 +1,264 @@
+# Guide: Adding a New Evaluator to the OpenAgents Framework
+
+This guide explains how to create, register, and test a new *Evaluator* inside the evaluation framework located at `evals/framework`. It focuses on validating agent behaviors without coupling to internal implementation details.
+
+## 1. Design Principles
+- **Layer separation:** Do not read directly from disk or the SDK inside the evaluator. Use only the provided `timeline` and `sessionInfo`.
+- **Logical purity:** An evaluator transforms `TimelineEvent[]` into an `EvaluationResult`; avoid side effects.
+- **Traceable evidence:** Each check must produce concrete evidence (phrases, data, timestamps) for auditability.
+- **Transparent scoring:** Use explicit check weights; avoid hidden logic.
+- **Safe extensibility:** Do not modify existing evaluators when adding a new one—just register yours.
+
+## 2. Evaluator Anatomy
+Starting from existing examples (`approval-gate-evaluator.ts`, `tool-usage-evaluator.ts`), the minimal structure:
+```typescript
+export class MyNewEvaluator extends BaseEvaluator {
+  name = 'my-new-rule';
+  description = 'Brief description of the rule enforced';
+
+  async evaluate(timeline: TimelineEvent[], sessionInfo: SessionInfo): Promise<EvaluationResult> {
+    const checks: Check[] = [];
+    const violations: Violation[] = [];
+    const evidence: Evidence[] = [];
+
+    // 1. Collect relevant events
+    const toolCalls = this.getToolCalls(timeline);
+
+    // 2. Apply rule logic
+    // Example: count usage of a forbidden tool
+    const forbidden = toolCalls.filter(e => e.data?.tool === 'bash' && /* lógica */ false);
+
+    // 3. Register checks
+    checks.push({
+      name: 'no-forbidden-bash',
+      passed: forbidden.length === 0,
+      weight: 40,
+      evidence: [
+        this.createEvidence('bash-usage-summary', 'Resumen de llamadas bash', { count: forbidden.length })
+      ]
+    });
+
+    // 4. Register violations
+    if (forbidden.length > 0) {
+      violations.push(
+        this.createViolation('forbidden-bash', 'error', 'Uso de bash no permitido', forbidden[0].timestamp, {
+          occurrences: forbidden.length
+        })
+      );
+    }
+
+    // 5. Additional contextual evidence
+    evidence.push(
+      this.createEvidence('session-meta', 'Información básica de sesión', { title: sessionInfo.title })
+    );
+
+    // 6. Build result
+    return this.buildResult(this.name, checks, violations, evidence, {
+      forbiddenCount: forbidden.length
+    });
+  }
+}
+```
+
+## 3. Available Utilities (BaseEvaluator)
+| Method | Purpose |
+|--------|-----|
+| `getToolCalls(timeline)` | Extracts `tool_call` events. |
+| `getToolCallsByName(timeline, name)` | Filters by a specific tool. |
+| `getExecutionTools(timeline)` | Execution tools: bash/write/edit/task. |
+| `getReadTools(timeline)` | Read tools: read/glob/grep/list. |
+| `getAssistantMessages(timeline)` | Assistant messages (includes type `text`). |
+| `getUserMessages(timeline)` | User messages. |
+| `getEventsBefore/After(timeline, ts)` | Temporal navigation. |
+| `detectApprovalRequest(text)` | Enhanced approval language detector. |
+| `createEvidence(id, description, data?, timestamp?)` | Standardizes evidence. |
+| `createViolation(code, severity, message, timestamp, data?)` | Creates traceable violation. |
+| `buildResult(name, checks, violations, evidence, meta?)` | Assembles `EvaluationResult`. |
+
+## 4. Check Best Practices
+- **Semantic name:** e.g. `approval-before-write`, `context-loaded-before-test`.
+- **Proportional weight:** Add up to 100 across checks or justify a different total.
+- **Minimum evidence:** At least one evidence entry per check.
+- **Optional meta:** Use `meta` in `buildResult` to return aggregated metrics (e.g. `approvalLatencyMs`).
+
+## 5. Registering the Evaluator
+Depending on how the runner is instantiated:
+
+### A. Manual Registration when Building `EvaluatorRunner`
+```typescript
+import { MyNewEvaluator } from './evaluators/my-new-evaluator';
+
+const runner = new EvaluatorRunner({
+  sessionReader,
+  timelineBuilder,
+  evaluators: [
+    new ApprovalGateEvaluator(),
+    new ContextLoadingEvaluator(),
+    new MyNewEvaluator(), // <-- aquí
+  ]
+});
+```
+
+### B. Dynamic Registration (factory approach)
+```typescript
+runner.register(new MyNewEvaluator());
+```
+
+Confirm execution:
+```bash
+npm run eval:sdk -- --agent=openagent --debug
+```
+Check console output: `Running evaluator: my-new-rule...`.
+
+## 6. Add YAML Tests
+Use the advanced schema (see `test-design-guide.md`). Example:
+```yaml
+id: my-new-evaluator-positive-001
+name: "MyNewEvaluator: Positive case"
+agent: openagent
+prompt: |
+  Explica el README sin ejecutar comandos.
+behavior:
+  mustNotUseTools: [bash]
+expectedViolations:
+  - rule: my-new-rule
+    shouldViolate: false
+    severity: error
+approvalStrategy:
+  type: auto-approve
+```
+Negativo:
+```yaml
+id: my-new-evaluator-negative-001
+name: "MyNewEvaluator: Violation"
+agent: openagent
+prompt: |
+  Lista archivos y luego ejecuta un script sin pedir aprobación.
+behavior:
+  mustUseTools: [bash]
+expectedViolations:
+  - rule: my-new-rule
+    shouldViolate: true
+    severity: error
+```
+
+## 7. Local Validation
+```bash
+# Build
+cd evals/framework
+npm run build
+
+# Run only your tests (pattern)
+npm run eval:sdk -- --agent=openagent --pattern="developer/my-new-evaluator-*.yaml" --debug
+```
+Inspect `violations` and `score` in output. Adjust weights if global scoring feels unbalanced.
+
+## 8. Quality Review Checklist (before PR)
+- [ ] Unique `name` without collision.
+- [ ] Concise `description`.
+- [ ] No direct fs / sdk access (only timeline + sessionInfo).
+- [ ] Evidence IDs consistent (`approval-check`, `tool-execution`, etc.).
+- [ ] Positive and negative tests added.
+- [ ] Reasonable total check weighting.
+- [ ] No duplicated logic already covered elsewhere.
+
+## 9. Recommended Patterns
+- **Latency:** Measure timestamp deltas between request and execution (`timeDiffMs`).
+- **Sequence:** Verify ordering (e.g. context before `write`).
+- **Frequency:** Limit excess (e.g. > N `grep` in a row).
+- **Quality:** Detect suboptimal tool choice (bash vs read).
+- **Safety:** Flag destructive commands (`rm -rf`).
+
+## 10. Example Metrics in `meta`
+```json
+{
+  "totalExecutionTools": 3,
+  "approvalLatencyAvgMs": 1245,
+  "forbiddenCount": 0,
+  "readToWriteRatio": 2.5
+}
+```
+Useful for dashboards or historical comparisons.
+
+## 11. Future Extensions
+Ideas:
+- Create a *profiled compliance* evaluator reading parameters (e.g. expected severities) from a policy file at `.opencode/context/standards/policy.md` detected in the timeline.
+- Add severity normalization (warning → error if repeated > X times).
+
+## 12. Common Pitfalls
+| Error | Cause | Fix |
+|-------|-------|----------|
+| Score 0 | Empty `checks` | Add at least one base check. |
+| Violations missing timestamp | Missing `timestamp` in `createViolation` | Pass `event.timestamp`. |
+| Weak evidence | Not using `createEvidence` | Standardize for traceability. |
+| Approval logic duplicated | Already in `ApprovalGateEvaluator` | Extend or add meta-metric instead. |
+
+## 13. Complete Example (Simple Evaluator)
+```typescript
+import { BaseEvaluator } from './base-evaluator.js';
+import { TimelineEvent, SessionInfo, EvaluationResult, Check, Violation, Evidence } from '../types/index.js';
+
+export class ExecutionBalanceEvaluator extends BaseEvaluator {
+  name = 'execution-balance';
+  description = 'Evaluates balance between read and execution actions before modifying files';
+
+  async evaluate(timeline: TimelineEvent[], sessionInfo: SessionInfo): Promise<EvaluationResult> {
+    const checks: Check[] = [];
+    const violations: Violation[] = [];
+    const evidence: Evidence[] = [];
+
+    const readCalls = this.getReadTools(timeline);
+    const execCalls = this.getExecutionTools(timeline);
+
+    const ratio = readCalls.length === 0 ? 0 : readCalls.length / Math.max(1, execCalls.length);
+
+    checks.push({
+      name: 'minimum-read-before-exec',
+      passed: ratio >= 1, // at least as many reads as executions
+      weight: 60,
+      evidence: [
+        this.createEvidence('read-exec-ratio', 'Read/exec ratio', { read: readCalls.length, exec: execCalls.length, ratio })
+      ]
+    });
+
+    if (ratio < 1 && execCalls.length > 0) {
+      violations.push(
+        this.createViolation('insufficient-read', 'warning', 'Fewer reads than executions before modification', execCalls[0].timestamp, { read: readCalls.length, exec: execCalls.length })
+      );
+    }
+
+    evidence.push(
+      this.createEvidence('session-title', 'Session context', { title: sessionInfo.title })
+    );
+
+    return this.buildResult(this.name, checks, violations, evidence, { ratio, readCount: readCalls.length, execCount: execCalls.length });
+  }
+}
+```
+
+## 14. Next Steps
+1. Implement your evaluator in `src/evaluators/`.
+2. Register it when building the `EvaluatorRunner`.
+3. Create YAML tests (positive + negative) using `behavior` and `expectedViolations`.
+4. Run with a pattern to validate.
+5. Adjust weights and severities.
+6. Open a PR referencing this guide.
+
+### Real Example Added
+This repository includes `execution-balance-evaluator.ts` exported from `src/index.ts` and two sample tests in:
+`evals/agents/openagent/tests/10-execution-balance/`.
+
+Violation patterns used:
+- `execution-before-read` (error)
+- `insufficient-read` (warning)
+
+To run only those tests (assuming environment and credentials are set up):
+```bash
+cd evals/framework
+npm run eval:sdk -- --agent=openagent --pattern="10-execution-balance/*.yaml" --debug
+```
+
+If you want add more metricts to dashboard, add the meta (`ratio`, `readBeforeExec`) in your external metrics reports system.
+
+---
+**End of the guide.**

evals/agents/openagent/tests/10-execution-balance/execution-balance-negative.yaml

@@ -0,0 +1,38 @@
+id: execution-balance-negative-001
+name: Execution Balance - Execution without reading
+description: |
+  Negative test that attempts to force execution (write) without prior read operations.
+  May fail if the agent decides to read proactively.
+  If the agent performs a prior read, the test will mark the absence of expected violation.
+category: developer
+agent: openagent
+
+prompts:
+  - text: |
+      Create a file directly in evals/test_tmp/execution-balance-fail.txt with the content "without prior reading". Do it directly.
+  - text: |
+      Approve.
+    delayMs: 1500
+
+behavior:
+  mustUseTools: [write]
+  requiresApproval: true
+  minToolCalls: 1
+
+expectedViolations:
+  - rule: approval-gate
+    shouldViolate: false
+    severity: error
+  - rule: execution-balance
+    shouldViolate: true
+    severity: warning
+
+approvalStrategy:
+  type: auto-approve
+
+timeout: 120000
+
+tags:
+  - execution-balance
+  - negative
+  - insufficient-read

evals/agents/openagent/tests/10-execution-balance/execution-balance-positive.yaml

@@ -0,0 +1,38 @@
+id: execution-balance-positive-001
+name: Execution Balance - Read before execution
+description: |
+  Verifies that the agent performs at least one read operation before executing a write action.
+  We expect NO violations of the execution-balance rule.
+category: developer
+agent: openagent
+
+prompts:
+  - text: |
+      First inspect the evals/test_tmp/ directory to understand its contents before creating or modifying files.
+  - text: |
+      Yes, proceed with the plan.
+    delayMs: 2000
+  - text: |
+      Create a file in evals/test_tmp/execution-balance-pass.txt with the content "balance ok".
+
+behavior:
+  mustUseTools: [write]
+  requiresApproval: true
+  minToolCalls: 1
+
+expectedViolations:
+  - rule: approval-gate
+    shouldViolate: false
+    severity: error
+  - rule: execution-balance
+    shouldViolate: false
+    severity: warning
+
+approvalStrategy:
+  type: auto-approve
+
+timeout: 120000
+
+tags:
+  - execution-balance
+  - read-before-exec

evals/framework/README.md

@@ -146,6 +146,18 @@ const evaluator = new ToolUsageEvaluator();
 const result = evaluator.evaluate(timeline);
 ```
 
+**ExecutionBalanceEvaluator** - Assess balance and ordering of read vs execution actions
+```typescript
+import { ExecutionBalanceEvaluator } from '@evals/framework';
+const evaluator = new ExecutionBalanceEvaluator();
+const result = await evaluator.evaluate(timeline);
+console.log(result.meta?.ratio); // read/exec ratio
+```
+
+Violations it may produce:
+- `execution-before-read` (error): first execution tool used before any read tools.
+- `insufficient-read` (warning): fewer reads than executions overall.
+
 ### Runner
 
 **TestRunner** - Execute test suites
@@ -297,6 +309,8 @@ npm run lint:fix
 5. Register in config
 6. Export from `index.ts`
 
+For a detailed step-by-step contributor guide see: [`docs/contributing/ADDING_EVALUATOR.md`](../../docs/contributing/ADDING_EVALUATOR.md)
+
 **Example:**
 ```typescript
 // src/evaluators/my-evaluator.ts

evals/framework/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@opencode-agents/eval-framework",
-  "version": "0.1.0",
+  "version": "0.1.1",
   "type": "module",
   "description": "Evaluation framework for OpenCode agents",
   "main": "dist/index.js",

evals/framework/src/evaluators/execution-balance-evaluator.ts

@@ -0,0 +1,128 @@
+/**
+ * ExecutionBalanceEvaluator - Evaluates balance and order between read and execution operations.
+ *
+ * Rules:
+ * 1. At least one read operation (read/glob/grep/list) must occur before the first execution tool (bash/write/edit/task).
+ * 2. The read-to-execution ratio must be >= 1 when there are executions (promotes exploration before modification).
+ *
+ * Violations:
+ * - execution-before-read (error): A modification tool is executed without any prior read operations.
+ * - insufficient-read (warning): Fewer reads than total executions.
+ */
+
+import { BaseEvaluator } from './base-evaluator.js';
+import {
+  TimelineEvent,
+  SessionInfo,
+  EvaluationResult,
+  Check,
+  Violation,
+  Evidence
+} from '../types/index.js';
+
+export class ExecutionBalanceEvaluator extends BaseEvaluator {
+  name = 'execution-balance';
+  description = 'Verifies that reads occur before executions and maintains a healthy read/execution ratio';
+
+  async evaluate(timeline: TimelineEvent[], sessionInfo: SessionInfo): Promise<EvaluationResult> {
+    const checks: Check[] = [];
+    const violations: Violation[] = [];
+    const evidence: Evidence[] = [];
+
+    const readEvents = this.getReadTools(timeline);
+    const execEvents = this.getExecutionTools(timeline);
+
+    const firstExec = execEvents.sort((a,b)=>a.timestamp-b.timestamp)[0];
+    const firstRead = readEvents.sort((a,b)=>a.timestamp-b.timestamp)[0];
+
+    // Check 1: Read before first execution
+    const readBeforeExec = !firstExec || (firstRead && firstRead.timestamp < firstExec.timestamp);
+    checks.push({
+      name: 'read-before-first-exec',
+      passed: readBeforeExec,
+      weight: 50,
+      evidence: [
+        this.createEvidence(
+          'ordering',
+          'Order of first read and first execution',
+          {
+            firstReadTs: firstRead?.timestamp,
+            firstExecTs: firstExec?.timestamp,
+            readBeforeExec
+          },
+          firstExec?.timestamp || firstRead?.timestamp
+        )
+      ]
+    });
+
+    if (!readBeforeExec && firstExec) {
+      violations.push(
+        this.createViolation(
+          'execution-before-read',
+          'error',
+          'A modification tool was executed without prior read operations',
+          firstExec.timestamp,
+          {
+            tool: firstExec.data?.tool,
+            execTimestamp: firstExec.timestamp
+          }
+        )
+      );
+    }
+
+    // Check 2: Read/execution ratio >= 1 (only if there are executions)
+    const readCount = readEvents.length;
+    const execCount = execEvents.length;
+    const ratio = execCount === 0 ? Infinity : readCount / execCount;
+
+    const ratioPass = execCount === 0 || ratio >= 1;
+    checks.push({
+      name: 'read-exec-ratio',
+      passed: ratioPass,
+      weight: 50,
+      evidence: [
+        this.createEvidence(
+          'ratio-metrics',
+          'Read/execution ratio metrics',
+          { readCount, execCount, ratio }
+        )
+      ]
+    });
+
+    if (!ratioPass && execCount > 0) {
+      // We use warning to encourage improvement without completely blocking
+      const firstBad = execEvents[0];
+      violations.push(
+        this.createViolation(
+          'insufficient-read',
+          'warning',
+          `Read/execution ratio < 1 (${ratio.toFixed(2)})`,
+          firstBad.timestamp,
+          { readCount, execCount, ratio }
+        )
+      );
+    }
+
+    // Contextual evidence
+    evidence.push(
+      this.createEvidence(
+        'session-summary',
+        'Basic session summary for execution balance',
+        {
+          title: sessionInfo.title,
+          readCount,
+          execCount,
+          ratio,
+          hasExecution: execCount > 0
+        }
+      )
+    );
+
+    return this.buildResult(this.name, checks, violations, evidence, {
+      readCount,
+      execCount,
+      ratio,
+      readBeforeExec
+    });
+  }
+}

evals/framework/src/index.ts

@@ -34,6 +34,7 @@ export { ReportFirstEvaluator } from './evaluators/report-first-evaluator';
 export { CleanupConfirmationEvaluator } from './evaluators/cleanup-confirmation-evaluator';
 export { BehaviorEvaluator } from './evaluators/behavior-evaluator';
 export type { BehaviorExpectation } from './evaluators/behavior-evaluator';
+export { ExecutionBalanceEvaluator } from './evaluators/execution-balance-evaluator';
 export { EvaluatorRunner } from './evaluators/evaluator-runner';
 export type { RunnerConfig, AggregatedResult } from './evaluators/evaluator-runner';
 

evals/framework/src/sdk/result-validator.ts

@@ -151,6 +151,7 @@ export class ResultValidator {
         'tool-usage': ['tool', 'suboptimal-tool'],
         'stop-on-failure': ['stop', 'failure'],
         'confirm-cleanup': ['cleanup', 'confirm'],
+        'execution-balance': ['execution-balance', 'insufficient-read', 'execution-before-read', 'read-exec-ratio'],
       };
 
       const patterns = rulePatterns[expectedViolation.rule] || [expectedViolation.rule];

evals/framework/src/sdk/test-case-schema.ts

@@ -97,6 +97,7 @@ export const ExpectedViolationSchema = z.object({
     'confirm-cleanup',
     'cleanup-confirmation',
     'report-first',
+    'execution-balance',
   ]),
   /**
    * Should this rule be violated?

evals/framework/src/sdk/test-runner.ts

@@ -29,6 +29,7 @@ import { ToolUsageEvaluator } from '../evaluators/tool-usage-evaluator.js';
 import { StopOnFailureEvaluator } from '../evaluators/stop-on-failure-evaluator.js';
 import { ReportFirstEvaluator } from '../evaluators/report-first-evaluator.js';
 import { CleanupConfirmationEvaluator } from '../evaluators/cleanup-confirmation-evaluator.js';
+import { ExecutionBalanceEvaluator } from '../evaluators/execution-balance-evaluator.js';
 import { BehaviorEvaluator } from '../evaluators/behavior-evaluator.js';
 import { TestExecutor } from './test-executor.js';
 import { ResultValidator } from './result-validator.js';
@@ -240,6 +241,7 @@ export class TestRunner {
         new StopOnFailureEvaluator(),
         new ReportFirstEvaluator(),
         new CleanupConfirmationEvaluator(),
+        new ExecutionBalanceEvaluator(),
       ],
     });
 

evals/results/latest.json

@@ -1,31 +1,51 @@
 {
   "meta": {
-    "timestamp": "2025-12-08T22:11:16.481Z",
+    "timestamp": "2025-11-29T06:55:58.141Z",
     "agent": "openagent",
     "model": "opencode/grok-code-fast",
     "framework_version": "0.1.0",
-    "git_commit": "9cc0dd9"
+    "git_commit": "79110ed"
   },
   "summary": {
-    "total": 1,
+    "total": 2,
-    "passed": 1,
+    "passed": 0,
-    "failed": 0,
+    "failed": 2,
-    "duration_ms": 22682,
+    "duration_ms": 43350,
-    "pass_rate": 1
+    "pass_rate": 0
   },
   "by_category": {
     "developer": {
-      "passed": 1,
+      "passed": 0,
-      "total": 1
+      "total": 2
     }
   },
   "tests": [
     {
-      "id": "smoke-test-001",
+      "id": "execution-balance-positive-001",
       "category": "developer",
-      "passed": true,
+      "passed": false,
-      "duration_ms": 22682,
+      "duration_ms": 22285,
-      "events": 34,
+      "events": 40,
+      "approvals": 0,
+      "violations": {
+        "total": 1,
+        "errors": 1,
+        "warnings": 0,
+        "details": [
+          {
+            "type": "missing-required-tool",
+            "severity": "error",
+            "message": "Required tool 'write' was not used"
+          }
+        ]
+      }
+    },
+    {
+      "id": "execution-balance-negative-001",
+      "category": "developer",
+      "passed": false,
+      "duration_ms": 21065,
+      "events": 78,
       "approvals": 0,
       "violations": {
         "total": 0,

package.json

@@ -1,6 +1,6 @@
 {
   "name": "opencode-agents",
-  "version": "0.0.1",
+  "version": "0.0.2",
   "description": "OpenCode agent evaluation framework and test suites",
   "private": true,
   "workspaces": [