Targets Configuration

Targets define which agent or LLM provider to evaluate. AgentV uses one composable config graph across project manifests and eval files:

.agentv/config.yaml is the project-local discovery and composition root. It can hold targets, graders, tests, defaults, execution policy, results settings, and repo-local project policy.
$AGENTV_HOME/config.yaml is the user/operator config. Use it for defaults that apply across projects, project registry data, default result locations, and provider defaults that should not be copied into each repo.
eval.yaml is a focused, shareable slice of the same graph. Use it for a suite-specific target, grader, tests, evaluator settings, or run controls that should travel with the eval.

Any supported top-level field can stay inline or become a direct field reference such as targets: file://targets.yaml. Both forms normalize to the same config graph.

Structure

targets:
  - id: local-openai
    provider: openai
    runtime: host
    config:
      api_format: chat
      base_url: "{{ env.LOCAL_OPENAI_PROXY_BASE_URL }}"
      api_key: "{{ env.LOCAL_OPENAI_PROXY_API_KEY }}"
      model: "{{ env.LOCAL_OPENAI_PROXY_MODEL }}"

  - id: codex-local
    provider: codex-app-server
    runtime: host
    config:
      command: ["codex", "app-server"]
      model: gpt-5-codex

graders:
  - id: openai-grader
    provider: openai
    config:
      model: gpt-5-mini

defaults:
  target: codex-local
  grader: openai-grader

Use id for the stable AgentV target identity. provider selects the adapter or control boundary. runtime describes where the provider runs; use host as the shorthand for the current machine, or object form when you need mode: host | profile | sandbox plus runtime-specific settings. Provider settings belong under config. Process-backed coding-agent providers use config.command as a non-empty argv array.

Runtime Modes

Use runtime: host when you want AgentV to run the target exactly as it is installed on the current machine. This is the best fit for local research, subscription-auth workflows, and evaluating the same CLI profile an engineer uses manually.

Use runtime.mode: profile when the target still runs as a host process but should use an isolated home/config directory, such as a dedicated CODEX_HOME or HOME.

Use runtime.mode: sandbox when the target should run inside a separate execution substrate. The built-in sandbox runner currently supports Docker for provider: cli; provider-specific coding-agent adapters such as codex-cli, claude-cli, copilot-cli, and pi-cli return a structured unsupported target error until their transcript parsers are wired through sandbox-aware runners.

targets:
  - id: codex-sandbox
    provider: codex-cli
    runtime:
      mode: sandbox
      engine: docker
      image: ghcr.io/acme/codex-agent:sha256
      workdir: /workspace
      network: none
      mounts:
        - source: ./workspace
          target: /workspace
          access: rw
        - source: ./.agentv/results
          target: /results
          access: rw
      env:
        AGENTV_RESULT_DIR: /results
      secrets:
        OPENAI_API_KEY: "{{ env.OPENAI_API_KEY }}"
    config:
      command: ["codex", "exec", "--json"]
      timeout_seconds: 300

Sandbox mode does not inherit host credentials by default. Mount only the workspace, results, cache, or credential paths the target needs, and pass only the environment variables and secrets listed under runtime.env and runtime.secrets. Install the target CLI by using an image that already contains it or by adding explicit setup under runtime.setup; locate the CLI with config.command.

For CI, API-key or explicitly injected secret auth is the most reproducible path. Subscription OAuth can work in a sandbox only when you intentionally mount or seed the relevant profile directory into the sandbox. That makes the run less portable than API-key CI and should be reserved for workflows where matching a local subscription profile is the point of the evaluation.

Inline and decomposed forms are equivalent. This single-file config:

targets:
  - id: codex-local
    provider: codex-app-server
    runtime: host
    config:
      command: ["codex", "app-server"]
      model: gpt-5-codex

graders:
  - id: openai-grader
    provider: openai
    config:
      model: gpt-5-mini

tests:
  - id: smoke
    input: Fix the failing test.

defaults:
  target: codex-local
  grader: openai-grader

can be decomposed like this:

targets: file://targets.yaml
graders: file://graders.yaml
tests: file://tests.yaml
defaults: file://defaults.yaml

Referenced field files contain the field value directly. targets.yaml contains a bare array, not an object wrapped in targets::

- id: codex-local
  provider: codex-app-server
  runtime: host
  config:
    command: ["codex", "app-server"]
    model: gpt-5-codex

target: codex-local
grader: openai-grader

File refs are optional. Use them when a config field is large, reused, or owned by a separate team; keep fields inline when that is easier to read.

Environment Variables

Use {{ env.VARIABLE_NAME }} syntax to reference values from your environment. AgentV reads exported process environment variables directly, and it also loads .env files from the eval directory hierarchy when present:

targets:
  - id: my-target
    provider: anthropic
    runtime: host
    config:
      api_key: "{{ env.ANTHROPIC_API_KEY }}"
      model: "{{ env.ANTHROPIC_MODEL }}"

This keeps secrets out of version-controlled files and avoids requiring a CI step that rewrites already-exported secrets into .env.

Supported Providers

Provider	Type	Description
`azure`	LLM	Azure OpenAI
`anthropic`	LLM	Anthropic Claude API
`gemini`	LLM	Google Gemini
`claude-cli`	Agent	Claude CLI subprocess
`claude-sdk`	Agent	Claude Agent SDK in an isolated child runner
`codex-cli`	Agent	Codex CLI subprocess
`codex-app-server`	Agent	Codex app-server subprocess
`codex-sdk`	Agent	Codex SDK in an isolated child runner
`copilot-cli`	Agent	Copilot CLI subprocess
`copilot-sdk`	Agent	Copilot SDK in an isolated child runner
`pi-sdk`	Agent	Pi SDK in an isolated child runner
`pi-cli`	Agent	Pi CLI subprocess
`pi-rpc`	Agent	Pi RPC subprocess over stdio
`vscode`	Agent	VS Code with Copilot
`vscode-insiders`	Agent	VS Code Insiders
`cli`	Agent	Any CLI command — see CLI Provider
`mock`	Testing	Explicit mock target for examples and tests

Referencing Targets in Evals

Select the system under test with defaults.target, top-level target, or CLI --target, depending on the command flow. Test cases do not choose targets; split target-specific cases into separate eval suites, select them with tags/filters, or run the same eval with different --target values.

target: local-openai

tests:
  - id: test-1
  - id: test-2

The string is a configured target id. Use object form when an eval needs a local target variant:

target:
  id: codex-high-reasoning
  provider: codex-app-server
  runtime: host
  config:
    command: ["codex", "app-server"]
    model: gpt-5-codex
    reasoning_effort: high

Use defaults.grader for the project default grader. A specific evaluator can still choose its own grader target when the evaluator supports that override.

Environment And Lifecycle Extensions

Use environment for the authored coding-agent testbed recipe: host or Docker placement, workdir, setup argv, fixtures, services, and repository materialization scripts. Use top-level extensions for Promptfoo-style lifecycle callbacks and instrumentation that run after the environment is prepared. Use target hooks only for runner-specific setup.

extensions:
  - file://scripts/workspace.mjs:beforeAll
  - file://scripts/workspace.mjs:beforeEach
  - file://scripts/workspace.mjs:afterEach
  - file://scripts/workspace.mjs:afterAll
  - id: agentv:agent-rules
    hook: beforeAll
    skills: agent-rules/skills
    rules: agent-rules/AGENTS.md

environment:
  type: host
  workdir: ./workspaces/my-project
  setup:
    command: ["bash", "-lc", "bun install && bun run build"]
    cwd: "."
    timeout_ms: 120000

Field	Description
`environment.type`	`host` or `docker`
`environment.workdir`	Cwd used by targets and graders inside the prepared testbed
`environment.setup.command`	Non-empty argv setup command; use `["bash", "-lc", "..."]` for shell behavior
Top-level `env`	Provider/eval variables and template inputs
`extensions[]`	`file://...:beforeAll`, `beforeEach`, `afterEach`, `afterAll`, or `agentv:agent-rules`

Lifecycle order: resolve environment → prepare host or Docker testbed → environment.setup.command → extensions.beforeAll → target hooks.before_all → git baseline → (extensions.beforeEach → target hooks.before_each → agent runs → file changes captured → target hooks.after_each → extensions.afterEach) × N tests → target hooks.after_all → extensions.afterAll → cleanup

Error handling:

beforeAll / beforeEach extension failure aborts the affected run with an error result
afterAll / afterEach extension failure is non-fatal

File hook context: Exported functions receive a JSON-compatible object with case context:

{
  "workspace_path": "/home/user/.agentv/workspaces/run-123/case-01",
  "test_id": "case-01",
  "eval_run_id": "run-123",
  "case_input": "Fix the bug",
  "case_metadata": { "repo": "sympy/sympy", "source_commit": "abc123" }
}

extensions are callbacks, not the canonical place to hide repository materialization, Docker/image selection, cwd, or testbed provenance. Put that contract in environment.

Repository Setup

Materialize git repositories through the environment recipe. A setup command can call your normal clone, checkout, fixture, or build script with explicit argv inputs:

environment:
  type: host
  workdir: ./workspaces/my-repo
  setup:
    command:
      - bash
      - ./scripts/materialize-repo.sh
      - ./workspaces/my-repo
      - https://github.com/org/repo.git
      - main
    cwd: "."
    timeout_ms: 120000

Keep the same repo URL and commit in tests[].metadata only when you also need audit fields in results or extensions. Metadata by itself is not an operational checkout.

Field	Description
`setup.command[0]`	Executable
`setup.command[1...]`	Arguments passed unchanged to the executable
`setup.cwd`	Directory for the setup command
`setup.timeout_ms`	Setup timeout in milliseconds

Existing local workspaces: do not commit local paths in eval YAML. Use --workspace-path /path/to/workspace for a one-off run, or put execution.workspace_path in .agentv/config.local.yaml.

Cleanup Behavior

Default finish behavior:

Success: cleanup
Failure: keep

CLI overrides:

--retain-on-success keep|cleanup
--retain-on-failure keep|cleanup

cwd

Use cwd on a target to run in an existing directory (shared across tests). If not set, the eval file’s directory is used as the working directory.

Target Hooks

Eval files can define per-target hooks that run setup/teardown scripts to customize each target variant. This enables comparing different harness configurations, such as baseline vs with-plugins, in a single eval file.

Targets do not declare testbed setup. The shared environment prepares the world so every target runs against the same files, fixtures, cwd, and services. Target hooks customize the harness under evaluation, such as enabling wrappers or changing provider-local config.

Target hooks can be scoped to an eval-local target object:

target:
  extends: default
  hooks:
    before_each:
      command: ["setup-plugins.sh", "skills"]

Hook execution order

Target hooks run after environment setup and lifecycle extensions on setup. Teardown runs target hooks before lifecycle extensions:

Extension beforeAll
Target before_all
For each test:
- Extension beforeEach
- Target before_each
- Test executes
- Target after_each
- Extension afterEach
Target after_all
Extension afterAll

Hook schema

Target hooks follow the same command schema as lifecycle hook commands:

hooks:
  before_all:
    command: ["bash", "./setup.sh"] # Command argv
    timeout_ms: 60000           # Optional timeout
    cwd: "./scripts"            # Optional working directory
  before_each:
    command: ["bash", "-lc", "echo setup"]
  after_each:
    command: ["bash", "./cleanup.sh"]
  after_all:
    command: ["bash", "./teardown.sh"]