Background
----------
On a typical CLI run (`node bin/repomix.cjs --include 'src,tests' --quiet`,
258 files, 4-vCPU host), the metrics worker pool was sized as
`ceil(258 / 100) = 3 workers`. Combined with the security pool's hard cap
of 2 workers (securityCheck.ts:90) and the main thread, the process held
6 active threads on 4 cores during the overlap of `validateFileSafety`
and `calculateMetrics`.
Each metrics worker independently parses gpt-tokenizer's ~2.2 MB
`o200k_base.js` BPE table on its first task — a ~200-300 ms pure-CPU
operation per worker. Spawning 3 cold metrics workers in the warm-up
phase (calculateMetrics.ts:46-48) therefore drove the security workers
off the CPU during their own (concurrent) cold-start, inflating the
critical-path security phase.
Change
------
Raise `TASKS_PER_THREAD` from 100 to 200 so:
- ≤200 file repos: 1 metrics worker (was 1) — no change
- 201-400 file repos: 2 metrics workers (was 3) — -1 worker, the win
- 401-600 file repos: 3 metrics workers (was 4-cap) — -1 worker
- 601-800 file repos: 4 metrics workers (was 4-cap) — no change
- 801+ file repos: 4 metrics workers (was 4-cap) — no change (cap)
For the 258-file benchmark this brings active workers during the
metrics+security overlap to 2 + 2 = 4, matching CPU count, and halves
the parallel BPE-loading work in the warm-up phase.
Tests for `getWorkerThreadCount` and `createWorkerPool` are updated to
reflect the new ratio.
Benchmark
---------
`node bin/repomix.cjs --include 'src,tests' --quiet` (258 files), n=20
paired interleaved (alternating BEFORE-first / AFTER-first ordering):
| | min | p25 | median | p75 | mean | sd |
|--------|---------|---------|---------|---------|---------|--------|
| BEFORE | 1045 ms | 1092 ms | 1109 ms | 1122 ms | 1107 ms | 27 ms |
| AFTER | 937 ms | 973 ms | 991 ms | 1020 ms | 994 ms | 29 ms |
Mean paired Δ: +112.5 ms (10.17 % wall-clock reduction)
Median paired Δ: +115.4 ms (10.66 % wall-clock reduction)
Paired-delta SD: 36.2 ms (paired t = 13.88, p < 0.001)
AFTER faster in 20/20 pairs (100 %)
Regression check — `node bin/repomix.cjs --quiet` (default, 1572 files),
n=15 paired interleaved:
| | min | p25 | median | p75 | mean | sd |
|--------|---------|---------|---------|---------|---------|--------|
| BEFORE | 1933 ms | 1970 ms | 2016 ms | 2102 ms | 2028 ms | 62 ms |
| AFTER | 1955 ms | 1966 ms | 2004 ms | 2131 ms | 2034 ms | 74 ms |
Mean paired Δ: -6.2 ms (-0.31 %) (paired t = -0.29, p > 0.05)
Median paired Δ: -12.7 ms (statistically neutral)
No regression on the large workload — both 100 and 200 saturate the
per-CPU cap at 4 workers for ≥800 file repos, so the dispatch-time
behavior is identical there.
Correctness
-----------
- 1256 / 1256 unit tests pass.
- `npm run lint` clean (only pre-existing warnings unrelated to this change).
- No behavioral change to file processing, tokenization, security checks,
or output. Pool sizing is the only effect.
- validateFileSafety: pin the negative path of `if (config.security.enableSecurityCheck)`
— every other test enabled the check, so a regression that always runs
the security check would have passed silently.
- unifiedWorker:
- Add a positive workerData=securityCheck + ambiguous-task case so the
pair (override + this) distinguishes "inference always wins" from
"inference wins only when it yields a value".
- Stop pretending the handler-cache test verifies caching. Both branches
of `if (cached) return cached;` end with the same Map.set, and Node's
own module cache makes the dynamic import effectively free, so the
cache is unobservable from outside without exposing internals.
Renamed to "repeated calls" with a comment explaining the limitation.
- fileSystemReadDirectoryTool: translate the pre-existing Japanese comment
to English per CLAUDE.md.
- TokenCounter: extract `LoadEncodingFn` type alias instead of the
unusual `typeof loadEncoding`, so a signature drift between the local
function and the deps field would surface at the type level.
- shared/errorHandle: recognize duck-typed OperationCancelledError from
worker boundaries in isRepomixError (it extends RepomixError but the
name was missing from the structured-clone fallback comparison).
Add a regression test for the worker-boundary case.
Test improvements per coderabbit / claude review:
- cliReport: assert skill-directory + relative path on the same log line.
- processConcurrency: restore process.versions.bun by removing the property
when it didn't originally exist, instead of leaving it defined-as-undefined.
- logger: drop the no-op `process.env.REPOMIX_LOG_LEVEL = undefined` (it
coerces to the string "undefined" and is overwritten by the next delete).
- unifiedWorker: replace the tautological cache test with one that proves
cache uniqueness via onWorkerTermination cleanup count; add a test for
task-based inference overriding workerData (bundled-env reuse).
- calculateMetricsWorker: new direct test for the default export's items
vs. single-mode dispatch — unifiedWorker mocks this module so the branch
was otherwise untested.
- packRemoteRepositoryTool: hard-code the expected output path instead of
expect.any(String) to catch arg-swap regressions.
- memoryUtils: tighten getMemoryStats assertions with sanity bounds
(heapUsed <= heapTotal, rss > 0, heapUsagePercent <= 100) so a
unit-conversion regression (bytes vs MB) would fail the test.
intent(asyncMap): tighten doc and test based on coderabbitai review feedback
decision(asyncMap-doc): explicitly note that workers are not cooperatively cancelled after a rejection — sibling workers keep claiming indices
decision(asyncMap-test): replace timing-sensitive `peakActive > 1` with exact `=== 4` — workers spawn synchronously via Promise.all so the cap is hit deterministically
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
intent(empty-dir-check): protect very large repos from FD exhaustion that unbounded Promise.all could trigger
rejected(p-limit): user wants to keep dependencies minimal — built a small in-tree helper instead
decision(asyncMap): single mapWithConcurrency helper rather than a p-limit-style limiter object — only call site is array map
decision(concurrency-limit): 20 in flight — well above libuv default thread pool (4) while still bounded for users who tuned UV_THREADPOOL_SIZE
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
intent(readability): an issue with no `path` (e.g., root-level schema mismatch) previously rendered as `[] message`; emit just `message` when segments are empty. Small quality-of-life for error output
intent(limitation-pin): add an integration test that documents the known ESM unwrap ambiguity — a CJS module shaped like `{ default: {...}, otherKey: ... }` has `otherKey` silently dropped by our heuristic. Non-issue for RepomixConfig today, but worth freezing so the behavior can't drift without someone noticing
intent(interop-consolidation): drop `interopDefault: true` from the jiti setup in configLoad — the explicit ESM namespace unwrap at the call site already handles every module-format case we test (.ts / .mts / .js / .mjs / .cjs). Having both the jiti flag and the manual unwrap was redundant and made the intent fuzzier
intent(error-path-cleanliness): filter out empty path segments before joining in rethrowValidationErrorIfSchemaError — a malformed ValiError item (object without `key`) would otherwise produce `[output..style]`; dropping the empty entry keeps the path readable. Added a dedicated test covering the filter
intent(error-handle): drop the instanceof Error guard in rethrowValidationErrorIfSchemaError so ValiError / ZodError round-tripped through a worker (plain { name, message, issues }) is still recognized — aligns with isError / isRepomixError elsewhere in the file
intent(schema-parity): restore splitOutput's upper bound (Number.MAX_SAFE_INTEGER) in the generated JSON schema so editor hints match the previous zod output; also strip the empty required:[] arrays that @valibot/to-json-schema emits on every object node
intent(esm-unwrap): only unwrap jiti's .default when it's an object, preserving a CJS config that legitimately exports { default: 'plain', ...rest }; plain Symbol.toStringTag === 'Module' was too narrow — jiti returns non-Module namespace wrappers for .ts / .mts files
intent(test-coverage): add tests/shared/errorHandle.test.ts covering the Zod + Valibot + worker-serialized paths through rethrowValidationErrorIfSchemaError; tighten the default-schema assertion in configSchema.test.ts from /expected|invalid/i to toThrow(v.ValiError) + targeted message pattern
decision(path-segment-fallback): return '' for unknown object-shaped path items instead of falling into String(segment) → "[object Object]"; defensive, non-breaking
Add maxWorkerThreads option to WorkerOptions for explicit thread count
capping, then use it to reduce CPU contention when metrics and security
worker pools run concurrently during the pipeline overlap phase.
- Metrics pool: capped at (processConcurrency - 1)
- Security pool: capped at floor(processConcurrency / 2)
On a 4-core machine this reduces concurrent threads from 8 (4+4) to 5
(3+2), avoiding context-switching overhead during gpt-tokenizer warmup.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
File collection was replaced with a promise pool approach in 96ff05dc,
but the worker-related code remained. This removes the now-unused
fileCollectWorker and all references to it from the worker system.
Vitest v4 changed how vi.fn() and vi.mock() work with class constructors.
Arrow functions in mockImplementation no longer work as constructors
when called with 'new' keyword.
Changes:
- Use regular function syntax instead of arrow functions for constructor mocks
- Use vi.hoisted() to define class mocks that can be used in vi.mock() factories
- Replace vi.fn().mockReturnValue() with vi.fn().mockImplementation() for class mocks
- Update mock instance retrieval to use vi.mocked().mock.results[0].value
- Add comprehensive tests for unifiedWorker.ts covering task inference
and worker termination cleanup
- Unify onWorkerTermination to async signature across all worker files
for consistency (fileCollect, securityCheck, calculateMetrics)
Remove code that was added for debugging during development:
- Remove unused isTinypoolWorker function from unifiedWorker.ts
- Remove REPOMIX_DEBUG_WORKER logging from unifiedWorker.ts
- Remove debug logging from defaultActionWorker.ts
- Remove unused getUnifiedWorkerPath export
- Update tests to use workerType instead of workerPath
Add a unified worker entry point that enables full bundling support by
allowing bundled files to spawn workers using themselves. This is a
prerequisite for bundling the website server to improve Cloud Run cold
start times.
Changes:
- Add src/shared/unifiedWorker.ts as single entry point for all workers
- Support both worker_threads and child_process runtimes
- Add REPOMIX_WORKER_TYPE env var for child_process worker type detection
- Add REPOMIX_WORKER_PATH env var for bundled environment worker path
- Add REPOMIX_WASM_DIR env var for WASM file location override
- Update processConcurrency.ts to use unified worker path
- Add debug logging (REPOMIX_DEBUG_WORKER=1) for worker troubleshooting
- Export unified worker handler from main index.ts
Note: This is work in progress. There's a known issue with child_process
runtime where nested worker pools (created inside a worker) may receive
incorrect REPOMIX_WORKER_TYPE environment variable, causing task routing
issues. Investigation ongoing.
- Add test for sizeParse overflow case
- Use RepomixProgressCallback type in outputSplit.ts for consistency
- Improve configuration.md description for splitOutput option
Adds a size-based output splitter via --split-output (kb/mb) and writes numbered parts without splitting within a top-level folder.
Also updates metrics aggregation for multi-part output and adds unit tests.
- Use class names for RepomixError type checking instead of hardcoded strings
- Remove unused RepomixError import from fileProcess.ts
- Simplify comments in errorHandle.ts and fileProcess.ts
- Clean up constructor-based error checking logic
Adjust worker runtime configuration to use child_process for all potentially risky operations, prioritizing stability and isolation over performance.
- Change token-related workers to child_process for better memory isolation:
- calculateGitDiffMetrics: child_process (was worker_threads)
- calculateGitLogMetrics: child_process (was worker_threads)
- calculateOutputMetrics: child_process (was worker_threads)
- calculateSelectiveFileMetrics: child_process (was worker_threads)
- Keep file collection and globby operations as worker_threads (lower risk)
- Remove redundant memory leak risk comments for cleaner code
- Fix test cases to include required runtime parameter and teardown property
- Reorder imports in languageParser.ts for consistency
This conservative approach ensures maximum stability by isolating all token counting operations in separate processes, preventing potential memory leaks from affecting the main process.
- Add WorkerOptions interface to combine numOfTasks, workerPath, and optional runtime
- Update createWorkerPool and initTaskRunner functions to accept WorkerOptions object
- Refactor all usage sites across file processing, metrics, and security modules
- Update corresponding test cases to use new interface
This improves type safety and makes the API more maintainable by avoiding parameter order mistakes.
Add WorkerRuntime type and configurable runtime parameter to createWorkerPool and initTaskRunner functions. This allows choosing between 'worker_threads' and 'child_process' runtimes based on performance requirements.
- Add WorkerRuntime type definition for type safety
- Add optional runtime parameter to createWorkerPool with child_process default
- Add optional runtime parameter to initTaskRunner with child_process default
- Configure fileCollectWorker to use worker_threads for better performance
- Update all test files to use WorkerRuntime type
- Add comprehensive tests for runtime parameter functionality
- Maintain backward compatibility with existing code
The fileCollectWorker now benefits from worker_threads faster startup and shared memory, while other workers continue using child_process for stability.
- Extract git diff token calculation into separate worker and dedicated module
- Parallelize git diff metrics calculation with other metrics computations using Promise.all
- Isolate TokenCounter usage for git diffs within child process worker to prevent memory leaks
- Add comprehensive worker cleanup with exit handler for proper resource management
- Update tests to reflect new worker-based architecture and remove direct TokenCounter mocking
Memory improvements:
- Git diff token calculation now runs in isolated child process
- Enables parallel execution of all three metrics calculations (files, output, git diff)
- Further reduces main process memory footprint by isolating heavy TokenCounter operations
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add generic initTaskRunner function to processConcurrency.ts to eliminate
duplicate initialization logic across multiple modules. This reduces code
duplication and provides consistent worker pool management with proper
type safety through generic parameters.
- Add TaskRunner<T, R> interface and initTaskRunner function
- Remove duplicate createTaskRunner wrappers from 5 modules
- Update all deps parameters to use shared initTaskRunner directly
- Maintain type safety with explicit generic type parameters
- Update corresponding test mocks to match new signature
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Replace the environment variable approach for passing log levels to workers with Tinypool's workerData mechanism, which is more idiomatic for worker thread configuration.
Changes:
- Add setLogLevelByWorkerData() method to handle workerData-based log level setting
- Update Tinypool configuration to use workerData instead of env variables
- Update all 5 worker files to use setLogLevelByWorkerData()
- Remove unused setLogLevelByEnv function and related test mocks
- Update tests to reflect new workerData configuration
This provides better isolation and follows Node.js worker thread best practices.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Set TASKS_PER_THREAD to 100 for better balance between performance and resource usage
- Add comment explaining that worker initialization is expensive
- Update tests to match new thread allocation logic
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Updated all references throughout the codebase:
- Import statements in 5 core modules
- Function calls in file processing, metrics, and security modules
- Test mocks and descriptions
- Maintained backward compatibility and functionality
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Replace Piscina with Tinypool to significantly reduce bundle size (800KB → 38KB) while maintaining full API compatibility and performance. This migration affects all worker thread pools used in file processing, security checks, and metrics calculations.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Claude
Kazuki Yamada
Dango233
Devin AI
Kazuki Yamada
pranshugupta01