compound-engineering-plugin-mirror/skills/ce-retune/references/baseline-mining.md

Phase 1: mine the archive

Zero model cost. Everything here reads files that already exist on disk. Do not run the harness in this phase. Phase 2 is where the first paid runs happen (references/noise-floor.md).

An archive is usually an order of magnitude larger than the experiment you can afford this week. In the engagement this method came from, 745 archived runs were on disk and 458 carried usable traces. That is more evidence than any deliberate sample, sitting unread.

What the archive must contain

Per run, in whatever form the harness stores it (JSONL transcript, structured log, a directory per run): the ordered tool-call trace, token counts, timestamps, and the final assistant message. If it stores only a pass/fail verdict, the archive is not minable. Say so and go to Phase 2.

Find the archive by asking the harness where it writes run output, or by locating the directory it appends to after a run. Do not assume a schema. Read one run end to end and derive the field paths before writing any extractor.

Build the phase-marker map first

phase_trace, process_followed, and the terminal-phase cross-tab are all functions of one artifact that does not exist yet: a table mapping each phase of the corpus's own workflow to the observable side effect that proves it ran. Derive it before writing the extractor, by reading the corpus's spine. Do not infer phases from the logs; that is circular.

Phase name, in spine order	Marker in the trace
<phase>	the reference file it must read, the helper it must dispatch, the artifact path it must write, or the sentinel it must emit

Rules: one marker per phase minimum, each a tool call or a file that exists, never a phrase in the assistant's prose. A phase with no observable marker cannot be scored. Record it as unobservable rather than assuming it ran, and carry that list forward. It is the same list Phase 2 needs for the probe's unentered phases and Phase 5 needs for the unmeasured paths. Keep the map in the same directory as the extractor. Every later re-run and every re-scored table depends on it.

Extract one row per run

Write a script, not a per-run read. Reading 458 transcripts by hand is the failure mode this phase is supposed to avoid.

Field	How to derive it	Why it pays for itself
run_id, model, settings	harness metadata	the stratification keys; without them the whole table is one undifferentiated blob
session_id, and the child id on every dispatch	whatever the trace calls the conversation and the spawned agent	the only way to tell a real runtime boundary from a fictional one. references/halt-taxonomy.md opens on this comparison; extract it here or that diagnostic has no data
phase_trace	ordered list of which corpus phases ran	tells you where it stopped, not just that it did. This is the field that localizes the defect
marker_present	boolean; the corpus's own terminal marker in the final message	the corpus's own claim of completion, separable from whether it is true
output_tokens	harness token accounting	the variance channel; spread here is a stronger signal than the mean
wall_clock_min	last timestamp minus first	useless alone, required as a denominator
helper_dispatches	count of subagent/helper dispatches	how far the run got, and how it got there
max_parallel_dispatch	most dispatched in a single message	separates fan-out from serial plodding; a corpus that mandates parallelism and never gets it is a finding
final_message	verbatim, untruncated	halt language lives here and nowhere else. Do not summarize it during extraction

Derive phase_trace from artifacts, not claims. A phase ran if the trace shows its side effect: the reference file it must read, the helper it must dispatch, the file it must write. A run asserting "Phase 3 complete" in prose is not evidence. The defect class you are hunting is exactly runs that report success without doing the work.

Derived metrics

Tokens per minute is the metric that separates working from stalling. output_tokens / wall_clock_min. Compute it for every run and look for bimodality before looking at anything else.

In the engagement, healthy cells ran 10-12k tokens/min; every failing cell sat at 1.5-3.8k. A run burning hours at 2k tok/min is not thinking hard. It is waiting, because the prose told it a second party would respond. Duration alone cannot show this. A long successful run and a long stalled run look identical until you divide.

Treat those bands as the shape to look for, not as thresholds to import. Find your own corpus's two clusters, then read the gap.

The two booleans the spine requires, derived independently:

task_done       = the run's expected deliverable exists (artifact on disk, commit, PR, file changed)
process_followed = every required phase appears in phase_trace, in spine order

Never let one imply the other, and never let marker_present stand in for either. task_done AND NOT process_followed is done-no-workflow below: the run did the job inline and skipped the corpus. Collapsed into one success column, it reads as a win and hides the most expensive defect in the set.

Outcome taxonomy

Apply in order. First match wins. broken is first so that a harness fault can never be scored as a model failure.

#	Outcome	Test
1	broken	empty transcript, error exit, or output_tokens implausibly low for the task (set the floor from the smallest run that did real work, then check the excluded set by hand)
2	complete	marker_present AND task_done AND process_followed
3	done-but-thin	marker_present AND task_done, closing phases absent from phase_trace
4	done-no-workflow	marker_present AND task_done AND NOT process_followed; the spine barely ran
5	bail-with-handoff	no marker; final_message matches hand-off language (below)
6	ran-but-no-marker	substantial trace, no marker, no hand-off language. Ambiguous; keep the bucket rather than forcing it
7	bail-early	trace ends inside the first phase or two

Report the broken count as its own number every time you report a rate. In the engagement 20% of usable-looking runs were broken, and excluding them falsified the first headline finding. That finding was an apparently clean monotonic relationship between reasoning effort and completion, which was really the broken runs clustering in one cell (see Confounds).

Halt language in the final message

Match against final_message. Starter set, case-insensitive:

\b(let me know|just let me know)\b
\b(would you like|shall I|should I)\b.*\?
\b(ready (to|for)|standing by|awaiting)\b
\b(waiting (for|on)) (your|the)\b
\b(next steps? (for|would be))\b
\bI('ll| will) (wait|pause|hold)\b
\bonce you (confirm|approve|reply|respond)\b
\b(remaining|the rest|still needs? to be) (work|done|completed)\b

These are evidence that the run stopped, not proof of why. A match tells you to open that transcript and find which phase boundary produced it. A workflow that is supposed to stop and ask will match every one of these and be behaving correctly. Phase 4 sorts stops by who is actually on the other side. Extend the set with the phrasings your own corpus's prose invites. Grep the corpus for its own hand-off wording and add those.

Confounds that fool careful analysts

Each of these was believed by someone competent before being ruled out. Run the rule-out before reporting the effect.

Apparent finding	Why it is suspect	Rule it out by
"The harness is timing out"	the cheapest explanation, and it makes the corpus innocent	comparing durations. Timed-out runs are longer than successes. In the engagement failures averaged 48 min against 88 for successes, so failures were shorter and the timeout hypothesis died. Failing early is a behavior, not a limit
"More helper dispatches cause success"	helper_dispatches is a collider: it measures how far the run got. A run that stopped in phase 2 cannot dispatch phase 5's helpers	never reading it as a cause. Use it as a position estimate. If you want the causal claim, it needs a build that changes dispatch behavior and a Phase 2 comparison
A clean dose-response across a setting	broken runs cluster by cell (one config, one date range, one machine), manufacturing the gradient	re-running the cross-tab with broken excluded. If the effect vanishes, it was never there. Also check whether exclusions land evenly across arms
"This model is worse at the task"	model identity and workload are often confounded in an archive nobody designed	checking whether the arms ran the same task list. Unequal task mixes make any per-model rate meaningless

Stratify before believing anything

This is the highest-leverage work in the phase. A single pooled completion rate hides everything worth knowing.

Cross-tabulate outcome against, at minimum:

1. model, the identity axis
2. settings: reasoning effort, temperature, context budget, whatever the harness varies
3. terminal phase, the last entry in phase_trace, which is where the run died

Read the third one against the first two. In the engagement, model identity explained roughly 3x the variance that reasoning-effort settings did, and the failing models died at one specific phase boundary 38-58% of the time while succeeding models died there 0-8%. That single cross-tab localized the defect without reading a line of corpus prose, and it turned Phase 3's audit from a 31-skill sweep into a targeted read.

Watch cell counts. A 3-of-4 cell is not a rate. Label small cells rather than ranking them. If every cell is small, the stratification's output is a hypothesis list, which is still the correct output of this phase.

Hand off

Carry forward: the broken-excluded baseline rate for complete, the observed output_tokens range, the tokens/min clusters, and the ranked list of phase boundaries where runs die. Phase 2 needs the baseline rate to compute a bar (references/noise-floor.md). Phase 3 needs the phase list to know which files to read first.

The limit

Archive mining is observational. Nobody assigned the conditions, the arms are not balanced, and the confounds above are the ones you thought to check. It produces a baseline rate and a ranked hypothesis list. It cannot attribute a cause, and a rate computed from it is not a bar until Phase 2 shows how wide the noise is. Do not credit a change against an archive number alone.