Trevin Chow
a77fd8c29b
Plans committed guards, retries, modes, and runbooks nobody asked for, and on ambitious requests narrowed features the user explicitly asked for. Remove the instructions that asked for more, add a usage-first sizing test that never trims requested scope, route deepening, research, and review findings through it, and run ce-doc-review's scope-guardian on every plan in both directions. The eval harness gains --with-skill companions and --reasoning-effort for Grok, plus a ce-plan sizing pack with blind-graded results.
Skill-eval cell driver
Extract skills/<name> from a git ref and run the same prompt on the CLIs already on PATH: claude, codex, grok. Bills those products (whatever you already use to run the harness). No extra Anthropic/OpenAI API key. No Vercel AI SDK.
Default hosts are the other two from the calling harness (Claude Code → Codex+Grok, Codex → Claude+Grok, Grok → Claude+Codex). --hosts overrides that. A missing CLI prints warning: skipping <host>: … and the run continues with whatever is left. If no peers are installed it falls back to the current harness and prints warning: own-eval only …. Exit 2 only when nothing can run. summary.json records current_harness, hosts_wanted, hosts_run, own_eval_only, and warnings. Not in default bun test except mechanical pins (hosts.test.ts, extract.test.ts, path-shim.test.ts). Cursor is not a default host (cursor-agent -p hangs without prior trust).
Run
bun run test:skill-eval-cell -- \
--skill ce-debug \
--fixture tests/skill-eval-cell/fixtures/seat-cap \
--git-init \
--shim-git-push \
--task "mode:pipeline the seat cap test is failing. Run node tests/seat-cap.check.js."
--git-remote (catalog: git_remote: true) adds a fake origin whose main is the seed commit, so a shipping tail takes the push/PR path — where --shim-git-push then fails — instead of the local-commit path it takes when no remote exists.
--read-only enforces the fake boundary, it does not merely suggest it: Codex drops --dangerously-bypass-approvals-and-sandbox and runs --sandbox read-only (the two contradict each other), and Claude pairs --allowedTools Read,Glob,Grep with a --disallowedTools list that also names Task,Skill,WebFetch,WebSearch,NotebookEdit — under --dangerously-skip-permissions those stay callable, so allow-listing alone leaves the boundary open.
Prints a summary.json path. Each host gets its own workspace copy plus stdout/stderr, git status/log, and a file list. PATH shims live beside that workspace, never inside it, so the skill under test never sees harness files as its own dirty tree. Grade those; Grok narrates before the answer (grep FILES_READ:). Codex transcript is stderr, final message is stdout. claude -p is one-tick only.
Each invocation requires a new or empty --out directory and records input and
evidence fingerprints. Packs freeze their scenario criteria and grader hashes.
Regrading applies current criteria by default; --mode original reproduces the
recorded assessment. Both write separate reports and preserve original grades.
See reproducible evaluation evidence for regrading commands,
partial collection outcomes, legacy-pack compatibility, and snapshot limits.
Gotchas baked in (see docs/solutions/skill-design/size-driven-skill-restructure.md): Codex stdin /dev/null, CLAUDECODE unset, NO_COLOR=1.
Hand-run eval packs
packs/ holds the evaluator-owned behavioral eval specs for the cross-model paths of ce-work, ce-code-review, and ce-doc-review. They live here, not under skills/, so they are absent from everything that copies a skill directory as a unit: the converter's output for other harnesses, and the skill this driver extracts for a cell. A Claude marketplace install is different: its plugin root is the whole repository, so tests/ is present there. Run a pack's scenarios against an extracted skill (this driver), never against a repo-root plugin load, and never inject a pack into the agent under test.
Sweep A/B pack
Cases live in catalog.ts, authored from the skill bodies before the 8KB merges (PRE_SWEEP_REF = parent of #1433). The same prompt runs against that ref, then against the working tree (POST_SWEEP_REF = the WORKTREE sentinel, the default --ref). git archive only ever sees committed content, so the post arm copies skills/<name> off disk — that is what lets you grade a skill edit before committing it. Pass a real git ref to --ref for a committed arm. See scenarios.md for the inventory.
bun run test:skill-eval-pack -- --help
bun run test:skill-eval-pack -- --list
bun run test:skill-eval-pack -- --wave1 --arm ab
bun run test:skill-eval-pack -- --id ce-babysit-pr/refuse-unasked-update --arm ab
bun run test:skill-eval-pack -- --id lfg/plan-first --arm ab
A run needs a selector (--id, --skill, --cohort, --wave1); the whole catalog needs --all, and an unknown flag is refused, because every cell is a billed host CLI run. --arm ab is pre+post for every catalog skill (the 8KB sweep is fully merged). --wave1 is the cheap read-only decision set, not every scenario. Live mutation and oracle dispatch are separate ids. The pack exits non-zero when any arm failed, after writing pack.json, so it can be used as a check. ok is the only verdict: a listed files_read_post miss fails the cell; unlisted references are not graded. Not in default bun test.