Continuation of the perf(core) Pre-warm security worker pool change —
extends `mockDeps` / inline pack-test plumbing in the three smaller test
files so the default-scope path no longer attempts to spawn a real
worker pool from the test environment.
- tests/core/packager/diffsFunctionality.test.ts: adds
`mockCreateSecurityTaskRunner` to both pack-call sites.
- tests/core/packager/splitOutput.test.ts: same — adds the stub to the
inline mock deps.
- tests/core/security/validateFileSafety.test.ts: updates the
`runSecurityCheck` call assertion to include the new
`{ taskRunner: undefined }` deps argument forwarded by
`validateFileSafety` when no pre-warmed runner is provided.
(See PR description / parent commit for the full perf change rationale,
benchmark numbers, and correctness notes.)
Move worker thread warmup from packager into createMetricsTaskRunner,
which now returns both a taskRunner and warmupPromise. This keeps the
packager clean — it no longer needs to know warmup implementation details.
Also:
- Skip metrics worker pool creation on skill-generation path where
it is unused
- Await warmupPromise in finally block before cleanup to prevent
tearing down workers during initialization
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When includeEmptyDirectories is enabled, buildOutputGeneratorContext
called searchFiles a second time just to obtain emptyDirPaths, despite
these already being computed during the initial file search in packager.
Changes:
- Capture emptyDirPaths from the initial searchFiles result in packager
and thread them through the pipeline (packager → produceOutput →
generateOutput/outputSplit → buildOutputGeneratorContext)
- Guard emptyDirPaths processing with includeEmptyDirectories check to
skip unnecessary work when the feature is disabled
- Fix split output path which was not receiving emptyDirPaths despite
the parameter being declared in produceOutput's signature
- Add tests for cache hit (searchFiles not called) and fallback paths
Local benchmark (repomix on itself, includeEmptyDirectories: true):
main: 696.6ms ± 4.2ms
branch: 637.1ms ± 2.6ms
Improvement: ~60ms (~8.5%)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pipeline optimization that parallelizes independent stages to reduce end-to-end latency:
1. Run security check and file processing concurrently: Security check uses
worker threads while file processing (in default config) runs on the main
thread, so they don't compete for CPU. After both complete, suspicious files
are filtered from the processed results using a Set for O(1) lookups.
2. Overlap output generation with metrics calculation: File metrics and git
metrics don't depend on the generated output, so they start immediately
via the worker pool while output generation runs on the main thread. Only
output token counting waits for the output string, passed as a Promise.
Before (sequential):
collectFiles → securityCheck(284ms) → processFiles(75ms) → produceOutput(185ms) → calculateMetrics(870ms)
After (overlapped):
collectFiles → [securityCheck || processFiles] → [produceOutput || fileMetrics + gitMetrics] → outputMetrics
Benchmark results (repomix repo, 989 files, 10 runs each, back-to-back):
Baseline: avg 2187ms, median 2176ms, p90 2269ms
Optimized: avg 2022ms, median 2018ms, p90 2070ms
Average improvement: 165ms (7.5%)
Median improvement: 158ms (7.3%)
P90 improvement: 199ms (8.8%)
https://claude.ai/code/session_011Sfyivv65pcN4VhFjRnDbs
Pipeline-level optimizations that produce measurable end-to-end improvement:
- Pre-initialize metrics worker pool during file collection phase so tiktoken
WASM loading overlaps with security checks and file processing. First token
count task dropped from 381ms to 22ms (worker already warmed).
- Lazy-load Jiti via dynamic import — only loaded when TS/JS config files are
detected, saving startup time for the common JSON/default config path.
- Fix O(n²) file path re-grouping in packager by using Map + Set for O(1)
membership checks instead of .find() + .includes().
- Move binary extension check before fs.stat in fileRead to skip unnecessary
stat syscalls for binary files.
- Parallelize split output file writes with Promise.all instead of sequential
for-loop.
Benchmark (15 runs each, median ± IQR, packing repomix repo ~1000 files):
main branch: 3515ms (P25: 3443, P75: 3581)
perf branch: 3318ms (P25: 3215, P75: 3383)
Improvement: -197ms (-5.6%)
Pipeline stage breakdown (instrumented):
- Metrics first-file init: 381ms → 22ms (worker pre-warmed)
- Total metrics stage: 793ms → ~450ms
All 1096 tests pass. Lint clean.
https://claude.ai/code/session_01JoNjFe7S2roMfHfNcw6bso
- Remove unused imports (generateFileTree, treeToString) in fileTreeGenerate.test.ts
- Add filePathsByRoot parameter to generateOutput and produceOutput calls in tests
- Update expect assertions to include filePathsByRoot argument
Move split/single output generation and writing logic to
packager/produceOutput.ts to keep packager.ts focused
on the high-level orchestration flow.
- Create produceOutput module handling both output modes
- Simplify packager.ts from 227 to 181 lines
- Update related tests to use new dependency structure
Adds a size-based output splitter via --split-output (kb/mb) and writes numbered parts without splitting within a top-level folder.
Also updates metrics aggregation for multi-part output and adds unit tests.
Kazuki Yamada
Claude
Dango233