Phase 1 accepts the harness only when it orders known-good above known-bad
exemplars and does not reward a trivial shortcut; a judge primary also needs
a human-labeled calibration sample (or an explicit waiver) before the run
leaves Phase 1. A held-out set (measurement.holdout.command or
metric.judge.confirmation_seed) is scored only before a keep and at final
confirmation: decide.mjs answers a would-be keep with next_measurement:
holdout until the pair is supplied, and a holdout that disagrees withholds
the keep. Judge items carry a required feedback string that the digest
groups into failure themes for hypothesis generation; identical items are
judged once per run via a content-hash cache; per-experiment cost and
per-case regressions are logged, report only.
New references/text-targets.md and example-text-target-spec.yaml cover
instruction-text targets: the eval-set coverage rule, model roles, hypothesis
moves, and saturation. The judge model enum becomes capability tiers
(cheap|strong) with legacy haiku/sonnet read as aliases.
Co-authored-by: Kieran Klaassen <kieranklaassen@users.noreply.github.com>
