Prepare Release (Hugging Face)

---
name: prepare-release
description: Prepare a new release of inspect_evals by creating a release branch, collecting changelog fragments, and opening a PR. Use when user asks to cut/prepare/create a new release or version bump.
metadata:
  origin: Hugging Face
---

# Prepare a Release

This workflow prepares a new release of `inspect_evals`. It creates a release branch, collects changelog fragments, bumps the version tag, and opens a PR. After merge it tags the merge commit and creates a GitHub release.

Reference: `PACKAGE_VERSIONING.md`

## Prerequisites

- The `gh` CLI must be authenticated
- You must have push access to the repository
- There must be changelog fragments in `changelog.d/` (beyond `.gitkeep` and `TEMPLATE.md`)

## Phase 1 — Prepare the release branch

1. **Ensure `main` is up to date**:

   ```bash
   git fetch origin
   git checkout main
   git merge --ff-only origin/main
   ```

   If the merge fails, stop and inform the user that their local `main` has diverged from `origin/main`.

2. **Check for changelog fragments**:

   List files in `changelog.d/` excluding `.gitkeep`, `TEMPLATE.md`, and `README.*`. If there are no fragments, stop and tell the user there is nothing to release.

3. **Determine the new version**:

   a. Find the current version from the latest `v*` git tag:

   ```bash
   git tag --sort=-v:refname --list 'v*' | head -1
   ```

   b. Ask the user what kind of bump this is, presenting the semver table from `PACKAGE_VERSIONING.md`:

   | Component | When to bump | Examples |
   | --------- | ------------ | -------- |
   | **Major** | Breaking changes | Removing an eval, API changes, scorer output format changes |
   | **Minor** | New features | Adding new evals, new task parameters, new utilities |
   | **Patch** | Bug fixes | Eval fixes, scorer fixes, dataset loading fixes |

   c. Compute the new version string (e.g. `0.4.0`, `0.3.107`). Confirm with the user.

4. **Create the release branch**:

   The branch name is `release-YYYY-MM-DD` using today's date.

   ```bash
   git checkout -b release-YYYY-MM-DD
   ```

## Phase 2 — Collect the changelog

1. **Run scriv collect**:

   ```bash
   uv run scriv collect --version <NEW_VERSION>
   ```

   This removes the individual fragment files from `changelog.d/` and prepends a new section to `CHANGELOG.md`.

2. **Present the changelog diff to the user for review**:

   ```bash
   git diff CHANGELOG.md
   ```

   Tell the user: *"Please review the collected changelog above. You can edit `CHANGELOG.md` directly — let me know when you are satisfied, or tell me what changes to make."*

   **Do not proceed until the user confirms the changelog is ready.**

3. **Stage and commit**:

   ```bash
   git add CHANGELOG.md changelog.d/
   git commit -m "Prepare release v<NEW_VERSION>"
   ```

## Phase 3 — Push and open a PR

1. **Push the branch**:

   ```bash
   git push -u origin release-YYYY-MM-DD
   ```

2. **Open a draft PR**:

   <!-- markdownlint-disable-next-line no-space-in-code -->
   Extract the new version's section from `CHANGELOG.md` (everything from the version heading up to but not including the next `## ` heading). Use this as the PR body:

   ```bash
   gh pr create --draft \
     --title "Release v<NEW_VERSION>" \
     --body "<EXTRACTED_CHANGELOG_SECTION>"
   ```

   **Important**: The PR title **must** start with `Release v` (e.g. `Release v0.4.0`) — this is how the `release-on-merge.yml` workflow identifies release PRs.

   Tell the user the PR URL and that it is in draft. They should mark it ready for review when appropriate.

## Phase 4 — After the PR is merged (automated)

Once the release PR is merged into `main`, the `.github/workflows/release-on-merge.yml` GitHub Actions workflow automatically:

1. Tags the merge commit with `v<NEW_VERSION>`
2. Creates a GitHub release with the changelog section as the body
3. Builds and publishes the package to PyPI
4. Notifies Slack

No manual action is required. The package version is derived from the new tag by `setuptools_scm`.

## Notes

- The package version is derived entirely from git tags via `setuptools_scm` — there is no version file to edit.
- If something goes wrong mid-workflow, the release branch can be deleted and the process restarted.
- The `v` prefix on tags is required (e.g. `v0.3.107`, not `0.3.107`).
- The `weekly-release.yml` workflow can also automate Phases 1–3 on a schedule or via manual dispatch, creating the release branch and draft PR for you.