Prepare Submission Workflow (Hugging Face)

---
name: prepare-submission-workflow
description: Prepare an evaluation for PR submission — add dependencies, tests, linting, eval.yaml, README, and changelog. Use when user asks to prepare an eval for submission or finalize a PR. Trigger when the user asks you to run the "Prepare Evaluation For Submission" workflow.
metadata:
  origin: Hugging Face
---

# Prepare Eval For Submission

## Workflow Steps

To prepare an evaluation for submission as a pull request:

1. Add custom dependencies in the [pyproject.toml][pyproject] file, if required. Don't forget to add your evaluation to the `[test]` dependencies if you've added any custom dependencies.

   ```toml
   [project.optional-dependencies]
   worldsense = ["pandas", "matplotlib"]  # existing groups
   your_evaluation = ["example-python-package"]  # add this line for your evaluation

   test = [
      "inspect_evals[dev]",  # existing groups
      "inspect_evals[your_evaluation]",  # add this line for your evaluation
   ]
   ```

   Pin the package version if your evaluation might be sensitive to version updates.

   Make sure your module does not import your custom dependencies when the module is imported. This breaks the way `inspect_ai` imports tasks from the registry.

   To check for this, `pip uninstall` your optional dependences and run another eval with `--limit=0`. If it fails due to one of your imports, you need to change how you are importing that dependency

2. Add pytest tests covering the custom functions included in your evaluation. See the [Testing Standards][testing-standards] section for more details.

3. Run `pytest` to ensure all new and existing tests pass.

   To install test dependencies:

   ```bash
   uv sync --extra test
   ```

   Then run the tests

   ```bash
   uv run pytest
   ```

4. Add an `eval.yaml` file to your evaluation directory (e.g., `src/inspect_evals/your_eval/eval.yaml`) to include it in the documentation (which is automatically generated and deployed by Github Actions after merge). See the [eval.yaml Reference][evalyaml-reference] section in CONTRIBUTING.md for a complete example and field documentation.

   Don't forget to add any custom dependencies to [pyproject.toml][pyproject] if applicable.

5. Run linting, formatting, and type checking and address any issues:

   ```bash
   make check EVAL=<eval_name>
   ```

   Note that we automatically run these commands on all incoming PRs, so be sure your contribution passes those before opening a PR. run_autolint.py handles [automated checks][automated-checks]

6. Fill in the missing sections of the generated README.md which are marked with 'TODO:'.

7. Create a changelog fragment to document your changes by running `uv run scriv create`.

<!-- Reference-style links -->
[automated-checks]: https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/EVALUATION_CHECKLIST.md#automated-checks
[evalyaml-reference]: https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/CONTRIBUTING.md#evalyaml-reference
[pyproject]: https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/pyproject.toml
[testing-standards]: https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/CONTRIBUTING.md#testing-standards