Create Eval (Hugging Face)

---
name: create-eval
description: Implement a new evaluation from an issue or paper. Guides through requirements gathering, architecture design, scaffolding, implementation, testing, and quality checks. Use when user asks to create/implement/build a new evaluation.
metadata:
  origin: Hugging Face
---

# Create Evaluation

Implement a new evaluation from an issue, paper, or benchmark specification. This is a phased workflow with human checkpoints between major phases. You should usually do human checkpoints. If the user requests that you do not do this, or you are in a fully autonomous workflow that says that there is no user to check with, you should ignore the human checkpoints. If you check in with the user several times and the user always replies with the same generic message like "Please continue the task", this indicates you are in an autonomous workflow.

## Expected Arguments

When invoked, this skill expects:

- **issue** (optional): A GitHub issue number or URL (e.g., `#1079`, `https://github.com/UKGovernmentBEIS/inspect_evals/issues/1079`)
- **eval_name** (optional): The name for the evaluation directory (e.g., `frontier_cs`)
- **paper_url** (optional): URL to the paper (arXiv, etc.)
- **reference_impl** (optional): URL to the reference implementation

If not provided, ask the user. At minimum, the eval_name is required.

## Phase 1: Requirements Gathering

Collect all information needed to make architectural decisions.

### 1.1 Gather Sources

- If an issue was provided, fetch it with `gh issue view <number> --json title,body,labels`
- If a paper URL was provided, fetch it with WebFetch
- If a reference implementation was provided, explore its structure
- If a HuggingFace dataset exists, fetch its metadata: `https://huggingface.co/api/datasets/<name>`

### 1.2 Classify the Evaluation

Determine the eval type by answering these questions:

| Question                     | Options                                                                     |
| ---------------------------- | --------------------------------------------------------------------------- |
| What does the model produce? | Text answer / Code / Multi-step actions / Structured output                 |
| How is it scored?            | Exact match / Partial credit / LLM judge / Code execution / Custom checker  |
| Does scoring need a sandbox? | No / Yes (Docker)                                                           |
| Does the solver need tools?  | No / Yes (bash, python, custom)                                             |
| Is it multi-turn?            | No / Yes                                                                    |
| What's the dataset source?   | HuggingFace / GitHub / URL / Local                                          |

Use this decision tree to pick the eval archetype:

```text
Model produces text answer?
  Scored by exact match? ──> Simple Q&A (pattern: gpqa, gsm8k)
  Scored by LLM judge? ──> LLM-as-judge (pattern: mt_bench)
  Scored by code execution? ──> Code-in-sandbox (pattern: humaneval)

Model produces code?
  Single file, run + check? ──> Code generation (pattern: humaneval)
  Multi-step with tools? ──> Agentic coding (pattern: swe_bench)

Model takes multi-step actions?
  With bash/python tools? ──> Agentic (pattern: swe_bench, agent_bench)
  With custom tools? ──> Agentic + custom tools (pattern: gdm_intercode_ctf)
```

### 1.3 Identify Key Components

Document:

- **Dataset**: Source, format, columns, splits, sample count, license
- **Scoring**: Mechanism, metrics (binary vs continuous), partial credit rules
- **Sandbox needs**: What needs to run in Docker, what software must be installed
- **Dependencies**: Any Python packages needed beyond inspect_ai
- **Task variants**: Multiple tasks, parameterized variants, or single task

### 1.4 Write Requirements

Create `agent_artefacts/<eval_name>/create/REQUIREMENTS.md` with all findings.

### 1.5 Human Checkpoint

Present a summary to the user and wait for approval before proceeding. Include:

- Eval archetype classification
- Proposed task structure (how many @task functions, what parameters)
- Scoring approach
- Sandbox needs (if any)
- Any open questions or ambiguities

## Phase 2: Architecture Design

### 2.1 Select Reference Eval(s)

Choose 1-2 existing evals as architectural models based on the archetype:

| Archetype          | Reference Eval | Key Pattern                                                 |
| ------------------ | -------------- | ----------------------------------------------------------- |
| Simple Q&A         | `gpqa`         | `hf_dataset` + built-in scorer                              |
| Math with few-shot | `gsm8k`        | Few-shot prompting + `match(numeric=True)`                  |
| Code generation    | `humaneval`    | `generate()` solver + custom `@scorer` with sandbox         |
| Agentic            | `swe_bench`    | `basic_agent()` solver + `bash()`/`python()` tools + Docker |
| LLM-as-judge       | `mt_bench`     | `model_graded_qa()` or custom judge scorer                  |

Read the reference eval's source code to understand its patterns.

### 2.2 Design File Structure

Determine what files are needed:

**Always required:**

```text
src/inspect_evals/<eval_name>/
  __init__.py           # Public exports
  <eval_name>.py        # @task function(s), solver, scorer (if simple)
  eval.yaml             # Metadata
  README.md             # Documentation
```

**Add when needed:**

```text
  dataset.py            # When dataset loading is complex (multi-source, caching, transforms)
  scorers.py            # When scorer has significant custom logic
  prompts.py            # When there are multiple/complex prompt templates
  compose.yaml          # When Docker sandbox is needed
  Dockerfile            # When a custom Docker image is needed
```

**Always required for tests:**

```text
tests/<eval_name>/
  __init__.py
  test_<eval_name>.py   # Unit tests + E2E (or split into test_dataset.py, test_scorer.py, test_e2e.py)
```

### 2.3 Design Dataset Loading

Choose the dataset loading strategy:

```text
HuggingFace dataset?
  Simple columns? ──> hf_dataset() with FieldSpec or record_to_sample
  Complex transforms? ──> hf_dataset() with record_to_sample function

GitHub-hosted data?
  Small, static? ──> Download to INSPECT_EVALS_CACHE_PATH at task init
  Large or versioned? ──> Download + cache with revision pinning

Local data?
  JSON? ──> json_dataset()
  CSV? ──> csv_dataset()
```

**Always pin dataset versions:**

- HuggingFace: Use `revision` parameter with commit SHA from `https://huggingface.co/api/datasets/<name>` (`sha` field)
- GitHub: Pin to a specific commit SHA

### 2.4 Design Scoring

Choose the scoring strategy:

```text
Binary outcome (correct/incorrect)?
  ──> Score.value = CORRECT/INCORRECT, metrics = [accuracy(), stderr()]

Continuous score (0.0-1.0 or 0-100)?
  ──> Score.value = float, metrics = [mean(), stderr()]
  ──> Normalize to 0.0-1.0 range for Score.value

Multiple sub-scores?
  ──> Score.value = dict, custom metrics or per-key mean()/stderr()
```

Key rules from BEST_PRACTICES.md:

- Store all aggregatable values in `Score.value`, NOT in `Score.metadata`
- Use `Score.metadata` only for identifiers and debugging info
- Use `Score.explanation` for human-readable scoring details

### 2.5 Design Solver

```text
Simple generation?
  ──> generate() or [prompt_template(...), generate()]

Needs system message?
  ──> [system_message(...), generate()]

Needs tools (bash, python)?
  ──> basic_agent(init=system_message(...), tools=[bash(), python()])

Needs few-shot examples?
  ──> [system_message(fewshot_text), prompt_template(...), generate()]
```

**Paper methodology check**: If the original paper uses single-turn evaluation but an agentic approach is also valid, implement **both** solvers and expose a parameter (e.g., `agentic: bool = True`). Default to agentic (since this is an agentic framework) but allow paper-faithful reproduction with `agentic=False`. Include separate system prompt constants for each mode.

### 2.5b Identify Hardware/Environment Requirements

Check if any subset of problems requires special hardware (GPUs, large memory, specific software). If so:

- Add a filtering parameter (e.g., `include_gpu_problems: bool = False`) that defaults to **excluding** problems the user likely can't run
- Document which problems are excluded and why
- Tag problems in metadata so filtering is data-driven, not hardcoded to specific IDs

This applies broadly: GPU-dependent problems, problems needing internet access, problems requiring specific commercial software, etc.

### 2.6 Write Architecture

Create `agent_artefacts/<eval_name>/create/ARCHITECTURE.md` with:

- File structure with descriptions
- Dataset loading approach
- Scoring mechanism
- Solver design
- Docker setup (if applicable)
- Dependencies needed

### 2.7 Human Checkpoint

Present the architecture to the user and wait for approval.

## Phase 3: Scaffolding

### 3.1 Create Directories

```bash
mkdir -p src/inspect_evals/<eval_name>
mkdir -p tests/<eval_name>
```

### 3.2 Create eval.yaml

```yaml
title: "Full Title: Subtitle"
description: |
  Description from the paper abstract.
arxiv: https://arxiv.org/abs/XXXX.XXXXX
group: <Group>  # One of: Assistants, Coding, Cybersecurity, Knowledge, Mathematics, Multimodal, Reasoning, Safeguards, Scheming, Writing
contributors: ["github-username"]
version: "1-A"
tasks:
  - name: <task_name>
    dataset_samples: <count>
tags: ["Agent"]  # Only if agentic. Omit if not applicable.
metadata:
  sandbox: ["solver", "scorer"]  # Only if sandbox used. Omit individual items or entire field if not needed.
```

### 3.3 Create **init**.py

Export all public symbols: task functions, dataset functions, scorer functions, constants.

### 3.4 Create Docker Files (if needed)

**compose.yaml** (at eval root, NOT in data/):

```yaml
services:
  default:
    build: .
    init: true
    command: tail -f /dev/null
```

**Dockerfile:**

- Start from a minimal base (e.g., `ubuntu:24.04`, `python:3.12-slim`)
- Install only what's needed for scoring/execution
- Keep it small — models don't need IDE tools

### 3.5 Create Test Stubs

Create `tests/<eval_name>/__init__.py` (empty).

### 3.6 Pin Dataset Version

- For HuggingFace: Fetch SHA from API and use as `revision` parameter
- For GitHub: Get commit SHA and pin downloads to it

### 3.7 Add Optional Dependencies (if needed)

Add to `pyproject.toml`:

```toml
[project.optional-dependencies]
<eval_name> = ["package1>=1.0", "package2"]

# Also add to test group:
test = [
  # ... existing entries ...
  "inspect_evals[<eval_name>]",
]
```

**Critical rule**: Optional dependencies must NOT be imported at module level. Defer imports to function scope.

### 3.8 Run Linting

```bash
make check
```

This runs ruff format, ruff check, mypy, README generation, and markdownlint in one command.

## Phase 4: Core Implementation

Implement in dependency order: dataset -> prompts -> scorer -> solver -> @task.

### 4.1 Dataset Loading

- Use `hf_dataset()` from `inspect_evals.utils.huggingface` for HuggingFace datasets
- Write a `record_to_sample` function that maps raw records to `Sample` objects
- Include all useful fields in `sample.metadata` for debugging
- Use stable, canonical IDs for `sample.id`

### 4.2 Prompts

- Keep prompt text as module-level constants (ALL_CAPS)
- Use `.format()` or f-strings for variable injection
- Separate system prompt from user prompt

### 4.3 Scorer

- Prefer built-in scorers (`match()`, `accuracy()`, `model_graded_qa()`) when possible
- For custom scorers, use `@scorer(metrics=[...])` decorator
- For sandbox execution in scorer, use `await sandbox().exec(...)`
- Handle timeouts and errors gracefully
- Return `Score(value=..., answer=..., explanation=...)`

### 4.4 Solver

- Prefer built-in solvers (`generate()`, `basic_agent()`)
- For agentic evals, use `basic_agent(init=system_message(...), tools=[...])`
- Set sensible `message_limit` defaults

### 4.5 Task Function

```python
EVAL_VERSION = load_eval_metadata("<eval_name>").version

@task
def <eval_name>(...) -> Task:
    return Task(
        dataset=...,
        solver=...,
        scorer=...,
        sandbox=...,  # "docker" if needed
        version=EVAL_VERSION.comparability_version,
        metadata=EVAL_VERSION.to_metadata(),
    )
```

Key rules:

- Do NOT pass a `name` parameter to `@task`
- Use `load_eval_metadata` for version info
- Use absolute imports throughout

### 4.6 Register in _registry.py

Add import(s) to `src/inspect_evals/_registry.py`:

```python
from inspect_evals.<eval_name> import <task_function>
```

### 4.7 Run Linting

```bash
make check
```

### 4.8 Human Checkpoint

Tell the user the core implementation is done. Suggest they smoke-test:

```bash
uv run inspect eval inspect_evals/<task_name> --model mockllm/model --limit 1
```

For Docker-based evals, this requires Docker to be running. If the user can't run Docker, note this and move on.

## Phase 5: Testing

### 5.1 Create Tests

Follow the patterns from the `ensure-test-coverage` skill. At minimum:

**record_to_sample test** (use real data from the dataset):

```python
def test_record_to_sample() -> None:
    record = { ... }  # Real example from dataset
    sample = record_to_sample(record)
    assert sample.id == "expected_id"
    assert "expected_text" in str(sample.input)
    assert sample.target == "expected_target"
```

**HuggingFace dataset structure validation** (if using HF datasets):

HF dataset validity is automatically checked by the master test in `tests/test_datasets_hf.py` — you do **not** need to add individual `test_dataset_is_valid` tests. Only add `assert_huggingface_dataset_structure` to document and validate the expected schema:

```python
from tests.utils.huggingface import (
    DatasetInfosDict,
    assert_huggingface_dataset_structure,
    get_dataset_infos_dict,
)

@pytest.fixture(scope="module")
def dataset_infos() -> DatasetInfosDict:
    return get_dataset_infos_dict(HF_DATASET_PATH)

@pytest.mark.huggingface
def test_dataset_structure(dataset_infos: DatasetInfosDict) -> None:
    assert_huggingface_dataset_structure(
        dataset_infos,
        {"configs": {"default": {"splits": ["test"], "features": {...}}}},
    )
```

**E2E test with mockllm:**

```python
@pytest.mark.dataset_download  # If downloads data
@pytest.mark.docker            # If uses Docker
def test_<eval_name>_e2e() -> None:
    [log] = eval(
        tasks=<task_function>(),
        model="mockllm/model",
        limit=1,
        message_limit=1,  # For agentic evals
    )
    assert log.status == "success"
```

**Custom scorer/solver/tool tests** if any exist. At minimum a type-check (`isinstance(scorer, Scorer)`) is required by autolint for custom scorers.

### 5.2 Add Pytest Markers

- `@pytest.mark.dataset_download` — if test downloads a dataset
- `@pytest.mark.huggingface` — if test uses a HuggingFace dataset
- `@pytest.mark.docker` — if test uses Docker sandbox
- `@pytest.mark.slow(<seconds>)` — if test takes >10 seconds (use observed duration)

### 5.3 Run Tests

```bash
uv run pytest tests/<eval_name>/ -v
```

Fix any failures. For tests requiring Docker or datasets, note which tests need special setup.

## Phase 6: Quality and Submission

### 6.1 Run Autolint

```bash
uv run python tools/run_autolint.py <eval_name>
```

Fix any failures reported.

### 6.2 Run All Linting

```bash
make check
```

This single command runs ruff format, ruff check, mypy, README generation, and markdownlint. Fix any errors before proceeding.

### 6.3 Generate README (if needed separately)

```bash
uv run python tools/generate_readmes.py --create-missing-readmes
```

Note: This tool does NOT accept eval name as argument — it discovers and generates all missing READMEs. Then fill in any TODO sections that are generated (description paragraph, dataset section, scoring section). The evaluation report TODO can be left for later.

### 6.5 Verify Against EVALUATION_CHECKLIST.md

Read `EVALUATION_CHECKLIST.md` and verify every applicable item. Key information not in `EVALUATION_CHECKLIST.md`

**Smoke Test**: Ensuring the eval runs with --limit 1 on a real model.

If the evaluation takes longer than five minutes to run and you can identify what component is failing, Add tests to reproduce any failure modes first. Only once those pass should you run it again.

**Evaluation Report**: This is the place you will stop for human approval unless you are specifically told not to or it is clear you are in a fully autonomous workflow.

**Trajectory Analysis**: If there are significant problems in trajectory analysis, these should be fixed. If the problem is related to the code at all, such as a broken environment, you should fix them using Test Driven Development. Create a unit test or tests with data that reproduces the failure mode, verify the test fails, then fix the issue and verify the test succeeds.

### 6.6 Create Changelog Fragment

```bash
uv run scriv create
```

## Gotchas (Lessons from Real Implementations)

- **Regex with language names**: If building regex patterns from language names like `c++`, use `re.escape()` — the `+` is a regex special character.
- **Filtering datasets by category**: `hf_dataset()` with `FieldSpec` loads all samples. If you need to filter (e.g., by category/track), load once, convert to samples, then filter. Use `MemoryDataset(samples=filtered_list, name=...)` to create a new dataset (NOT `Dataset()` which takes no constructor args).
- **Mixed config formats**: Dataset fields that look like YAML may contain JSON in some records. Always implement fallback parsing (YAML → JSON → raw string).
- **`basic_agent()` API**: Use `basic_agent(init=system_message(...), tools=[...])`. This is the preferred agentic solver. Only use `react()` if a more complex, bespoke solver is required for a given evaluation.
- **`sandbox().exec()` API**: The parameter for passing stdin is `input=`, NOT `stdin=`. Signature: `exec(cmd, input=None, cwd=None, env=None, user=None, timeout=None)`.
- **HF dataset validity is checked centrally**: The master test `tests/test_datasets_hf.py` automatically validates all HF datasets. You do not need individual `assert_huggingface_dataset_is_valid` tests. Add `assert_huggingface_dataset_structure` to document expected schema.
- **README generator**: `tools/generate_readmes.py --create-missing-readmes` — does NOT accept eval name arg, generates all missing READMEs at once.
- **Linting**: Use `make check` to run all linting (ruff format, ruff check, mypy, README generation, markdownlint). Do NOT run individual linting commands — `make check` covers everything.
- **Scorer robustness for agentic evals**: When a model hits `message_limit` during a tool call, `state.output.completion` is empty. If your scorer extracts code from the completion, add a fallback that reads solution files from the sandbox (e.g., `find /tmp /home -name '*.cpp'`).
- **Version pinning in Dockerfiles**: Pin all downloaded dependencies (e.g., `testlib.h`) to specific commit SHAs, not branch names like `master` or `main`. Same applies to GitHub archive downloads in dataset code.
- **Paper vs agentic methodology**: If a paper uses single-turn evaluation, implement both single-turn and agentic solvers. Default to agentic, but expose a parameter so users can reproduce paper results. Compare against the reference implementation's system prompts.
- **Hardware-gated problems**: Some benchmarks include problems requiring GPUs, specific hardware, or commercial software. Add a filtering parameter (e.g., `include_gpu_problems=False`) so these are excluded by default but can be opted into.
- **Multiple execution modes in one benchmark**: Some benchmarks have problems with fundamentally different execution models (e.g., batch file I/O vs interactive bidirectional piping). Check the reference implementation's config schema for `type` fields and implement all modes, not just the most common one.
- **Sandbox config must reference compose.yaml**: Passing `sandbox="docker"` uses a bare container. To use a custom Dockerfile, pass `sandbox=("docker", str(compose_file))` where `compose_file = Path(__file__).parent / "compose.yaml"`.

## Things NOT to Do

- **Don't pass `name` to `@task`**: This is only for dynamically created tasks
- **Don't import optional dependencies at module level**: Defer to function scope
- **Don't use relative imports**: Use absolute imports (`from inspect_evals.foo.bar import baz`)
- **Don't use private Inspect AI APIs**: Avoid `inspect_ai.*._internal` modules
- **Don't store aggregatable values in Score.metadata**: Use Score.value (can be a dict)
- **Don't upload new dataset copies**: Use official sources
- **Don't implement manual HF token checks**: The centralized conftest handles `@pytest.mark.huggingface`
- **Don't over-test thin wrappers**: `isinstance(solver, Solver)` is sufficient for `basic_agent` wrappers
- **Don't add type annotations that are redundant**: Only annotate function signatures
- **Don't create documentation files unless asked**: The README is created by `generate_readmes.py`
- **Don't run `uv update`**: Only `uv lock` if dependencies change
- **Don't hardcode magic numbers**: Extract to named constants if used 3+ times or unclear
- **Don't write try/catch unless truly necessary**: Let errors crash early