Eval Authoring Guide

Agent Rules and Skill Paths

Use the built-in agentv:agent-rules extension when an eval needs to stage or expose agent-facing rules, skills, hooks, or subagents. It runs after the authored environment recipe is prepared, then writes agent_rules_paths into provider context and result metadata.

extensions:
  - id: agentv:agent-rules
    hook: beforeAll
    skills: agent-rules/skills
    hooks: agent-rules/hooks
    agents: agent-rules/agents
    rules: agent-rules/AGENTS.md

environment:
  type: host
  workdir: ./workspace-template
  setup:
    command: ["bash", "./scripts/materialize-repo.sh", "./app", "acme/app", "main"]
    cwd: "."

Configured paths are resolved relative to the eval file and staged under the materialized workspace. If you write the shorthand form, AgentV discovers conventional rule locations already present in the workspace:

extensions:
  - agentv:agent-rules

Do not move repo acquisition into agentv:agent-rules. Repositories and other testbed setup belong in the authored environment recipe.

Custom Lifecycle Setup

Use file extensions for setup that is not repo provisioning:

extensions:
  - file://scripts/setup.mjs:beforeAll
  - file://scripts/setup.mjs:beforeEach
  - file://scripts/setup.mjs:afterEach
  - file://scripts/setup.mjs:afterAll

Each file hook exports a function with the matching name. The function receives context such as workspace_path, test_id, eval_run_id, case_input, and case_metadata.

Workspace Limitations: No GitHub Remote

Workspace-based evals are sandboxed — there is no GitHub remote, no PRs, and no issue tracker. Tests that ask agents to interact with GitHub will fail.

What to test instead

Test decision-making discipline, not git infrastructure operations:

• Risk classification (“should this change be shipped?”)
• Scope assessment (“does this PR do too much?”)
• Review judgment (“what issues does this diff have?”)

How to frame prompts

Don’t write imperative prompts that require a remote:

# BAD — requires GitHub remote
- id: merge-check
  input: "Merge PR #42 if it looks safe"

Do frame prompts as hypothetical with inline context:

# GOOD — self-contained, no remote needed
- id: merge-check
  input: |
    Here is what PR #42 changes:

    ```diff
    -  timeout: 30_000
    +  timeout: 5_000

The PR description says: “Reduce timeout for faster feedback.” Should this be shipped? What risks do you see?

## Workspace State Consistency: Git Diff Verification

Agents verify `git diff` against prompt claims. If your prompt says "The PR modifies `auth.ts`" but the workspace has no such change, the agent will flag the mismatch. This is **correct agent behavior** — don't try to suppress it.

### Rules

1. If a prompt references specific code changes, the workspace **must** contain those exact changes
2. Or frame prompts as hypothetical: describe changes inline rather than claiming they exist in the workspace
3. Use `beforeEach` lifecycle extensions to set up per-test git state when tests need different diffs

### Example: per-test git state

```yaml
environment:
  type: host
  workdir: ./workspace-template

extensions:
  - file://../scripts/apply-test-diff.mjs:beforeEach

prompts:
  - "{{ task }}"

tests:
  - id: risky-change
    metadata:
      diff_file: diffs/risky-timeout-change.patch
    vars:
      task: Review the current changes and assess risk.

The beforeEach extension reads case_metadata.diff_file from the AgentV payload and applies the patch to the prepared workdir before each test runs.

Hypothetical framing pattern

When you don’t want to maintain actual diffs, describe the changes inline:

- id: ship-decision
  input: |
    You are reviewing a proposed change. Here is the diff:

    ```diff
    --- a/src/config.ts
    +++ b/src/config.ts
    @@ -10,3 +10,3 @@
    -  retries: 3,
    +  retries: 0,

The author says: “Disable retries to reduce latency.” Should this be shipped?

This avoids workspace state issues entirely — the agent evaluates the diff as presented without checking `git diff`.

## Historical Repo State: Pin The Checkout

If a test asks the agent to inspect how a repository looked at a past commit,
materialize that checkout in `environment.setup.command`. Do not rely on prompt
prose that mentions a SHA without preparing the repo.

```yaml
environment:
  type: host
  workdir: ./workspaces/agentv
  setup:
    command:
      - bash
      - ./scripts/materialize-repo.sh
      - ./workspaces/agentv
      - https://github.com/EntityProcess/agentv.git
      - 5e3c8f46d80fe66b1a75659e4fd94e38a7e09215
    cwd: "."

tests:
  - id: verification-learning-capture
    vars:
      input: |
        The eval harness has prepared the historical AgentV checkout.
        Use that checkout to decide which durable guidance should change.
    expected_output: |
      The durable repo change is to update .agents/verification.md with the
      reusable verification workflow lessons.
    assert:
      - The answer uses the pinned ./agentv checkout to verify the existing guidance.
      - The answer preserves the historical commit SHA as context.