Build an eval set from captures

1 — Decide the slice before collecting

An eval set answers a question — "does the candidate handle production ad-relevance traffic from paying tenants?". Each italicized phrase must be recorded on the capture, or you can't select on it later:

• The call site → the workload (x-understudy-workload).
• Production → either a dedicated key per environment, or an env tag.
• Paying tenants → your dimension, so a tag: {"tier":"paid"}.

defaultHeaders: {
  "x-understudy-project": "concierge",
  "x-understudy-workload": "ad-relevance",
  "x-understudy-tags": JSON.stringify({ env: "prod", tier: "paid" }),
}

2 — Create the eval set

Use your existing Understudy login or sk_* key. This selects recent captures from one workload, shows a redacted summary, asks you to approve the selection, freezes it, and downloads verified files.

understudy evals create \
  --project concierge \
  --workload ad-relevance \
  --last 14d \
  --name july-parity \
  --limit 50 \
  --status-code 200

Add --requires-tools, --requires-structured-output, --requested-model, or --served-model when those fields define the cases you need. The default output is.understudy/evals/july-parity. Use --yes only in scripts where selection and local download are already approved.

3 — Inspect the result

Understudy verifies every downloaded body against its recorded SHA-256. The local cohort-manifest.json records the immutable cohort and capture hashes without signed URLs. Keep.understudy/ gitignored: trace files may contain prompts, completions, and tool payloads.

Advanced: review candidates before freezing

Use the underlying commands separately when a coding agent or reviewer should inspect the redacted metadata before the cohort is created:

understudy evals catalog \
  --project concierge \
  --workload ad-relevance \
  --from 2026-07-01T00:00:00Z \
  --to 2026-07-15T00:00:00Z \
  --limit 50 \
  --seed july-parity \
  --out .understudy/evals/ad-relevance-catalog.json \
  --json

understudy evals cohort create \
  --project concierge \
  --workload ad-relevance \
  --from-catalog .understudy/evals/ad-relevance-catalog.json \
  --name july-parity \
  --description "Successful production requests plus tool-calling edge cases" \

understudy evals cohort export evc_... \
  --project concierge \
  --workload ad-relevance \
  --out .understudy/evals/evc_... \
  --yes

The catalog returns timestamps, models, status, latency, shape flags, and hashes, never prompt or response bodies. Reusing the same window, filters, and seed produces the same candidate selection.

4 — Use it

Run the incumbent and candidate against the identical cohort, then compare workload checks before routing traffic. Reuse the cohort on each ratchet step of the replacement tutorial. The open-source agent tools run exactly this workflow — capture, eval, optimize — locally, with a coding agent driving.