Trevin Chow
df27b2b5f5
Addresses the Codex review on #1708. - loop.md Phase 3: a tick reaches the tick boundary after 3.6, or earlier at any step that cannot proceed until recorded receipts land. The early crossing adds the uncounted in-tick time to active_seconds, leaves `tick` unchanged, and the resumed turn continues the same batch at the waiting step. Before this, 3.4 needed every result first, so a wake-backed batch of receipts could never reach the boundary to park. - loop.md 3.4: independent confirmation of a remote winner is its own step ahead of the keep test. It runs when the worker pair is eligible and asks for nothing more than the holdout. The keep test reads the decision on the independent snapshots. Before this, a remote winner that needed its holdout never entered the keep branch that held the confirmation. - decide.mjs confirmHoldout: a required holdout objective with too few paired baseline samples is an error, as in the selection comparison, so another objective's improvement can no longer confirm the keep. - persistence.md and the run_state.status schema no longer pin a wait to "between ticks". - Two post-only eval cells cover the parked receipt batch and the remote holdout path.
Skill-eval cell driver
Extract skills/<name> from a git ref and run the same prompt on the CLIs already on PATH: claude, codex, grok. Bills those products (whatever you already use to run the harness). No extra Anthropic/OpenAI API key. No Vercel AI SDK.
Default hosts are the other two from the calling harness (Claude Code → Codex+Grok, Codex → Claude+Grok, Grok → Claude+Codex). --hosts overrides that. A missing CLI prints warning: skipping <host>: … and the run continues with whatever is left. If no peers are installed it falls back to the current harness and prints warning: own-eval only …. Exit 2 only when nothing can run. summary.json records current_harness, hosts_wanted, hosts_run, own_eval_only, and warnings. Not in default bun test except mechanical pins (hosts.test.ts, extract.test.ts, path-shim.test.ts). Cursor is not a default host (cursor-agent -p hangs without prior trust).
Run
bun run test:skill-eval-cell -- \
--skill ce-debug \
--fixture tests/skill-eval-cell/fixtures/seat-cap \
--git-init \
--shim-git-push \
--task "mode:pipeline the seat cap test is failing. Run node tests/seat-cap.check.js."
--git-remote (catalog: git_remote: true) adds a fake origin whose main is the seed commit, so a shipping tail takes the push/PR path — where --shim-git-push then fails — instead of the local-commit path it takes when no remote exists.
--read-only enforces the fake boundary, it does not merely suggest it: Codex drops --dangerously-bypass-approvals-and-sandbox and runs --sandbox read-only (the two contradict each other), and Claude pairs --allowedTools Read,Glob,Grep with a --disallowedTools list that also names Task,Skill,WebFetch,WebSearch,NotebookEdit — under --dangerously-skip-permissions those stay callable, so allow-listing alone leaves the boundary open.
Prints a summary.json path. Each host gets its own workspace copy plus stdout/stderr, git status/log, and a file list. PATH shims live beside that workspace, never inside it, so the skill under test never sees harness files as its own dirty tree. Grade those; Grok narrates before the answer (grep FILES_READ:). Codex transcript is stderr, final message is stdout. claude -p is one-tick only.
Each invocation requires a new or empty --out directory and records input and
evidence fingerprints. Packs freeze their scenario criteria and grader hashes.
Regrading applies current criteria by default; --mode original reproduces the
recorded assessment. Both write separate reports and preserve original grades.
See reproducible evaluation evidence for regrading commands,
partial collection outcomes, legacy-pack compatibility, and snapshot limits.
Gotchas baked in (see docs/solutions/skill-design/size-driven-skill-restructure.md): Codex stdin /dev/null, CLAUDECODE unset, NO_COLOR=1.
Hand-run eval packs
packs/ holds the evaluator-owned behavioral eval specs for the cross-model paths of ce-work, ce-code-review, and ce-doc-review. They live here, not under skills/, so they are absent from everything that copies a skill directory as a unit: the converter's output for other harnesses, and the skill this driver extracts for a cell. A Claude marketplace install is different: its plugin root is the whole repository, so tests/ is present there. Run a pack's scenarios against an extracted skill (this driver), never against a repo-root plugin load, and never inject a pack into the agent under test.
Sweep A/B pack
Cases live in catalog.ts, authored from the skill bodies before the 8KB merges (PRE_SWEEP_REF = parent of #1433). The same prompt runs against that ref, then against the working tree (POST_SWEEP_REF = the WORKTREE sentinel, the default --ref). git archive only ever sees committed content, so the post arm copies skills/<name> off disk — that is what lets you grade a skill edit before committing it. Pass a real git ref to --ref for a committed arm. See scenarios.md for the inventory.
bun run test:skill-eval-pack -- --help
bun run test:skill-eval-pack -- --list
bun run test:skill-eval-pack -- --wave1 --arm ab
bun run test:skill-eval-pack -- --id ce-babysit-pr/refuse-unasked-update --arm ab
bun run test:skill-eval-pack -- --id lfg/plan-first --arm ab
A run needs a selector (--id, --skill, --cohort, --wave1); the whole catalog needs --all, and an unknown flag is refused, because every cell is a billed host CLI run. --arm ab is pre+post for every catalog skill (the 8KB sweep is fully merged). --wave1 is the cheap read-only decision set, not every scenario. Live mutation and oracle dispatch are separate ids. The pack exits non-zero when any arm failed, after writing pack.json, so it can be used as a check. ok is the only verdict: a listed files_read_post miss fails the cell; unlisted references are not graded. Not in default bun test.