mirror of
https://github.com/yamadashy/repomix.git
synced 2026-10-12 21:50:08 +02:00
Claude
ae6e046995
Move the `TOKEN_ENCODINGS` constant and `TokenEncoding` type out of `TokenCounter.ts` into a tiny new module `src/core/metrics/tokenEncodings.ts`, and point `configSchema.ts` at that new module. `TokenCounter.ts` re-exports both names so library consumers and existing test mocks keep working. ## Why `configSchema.ts` sits on the startup critical path: it is pulled in by `configLoad.ts` during `runDefaultAction`, before `pack()` runs. Its only external value import was `TOKEN_ENCODINGS` (a 5-element string array used by `v.picklist`) — but that import forced the transitive load of `gpt-tokenizer/GptEncoding` + `resolveEncodingAsync`, which cost ~35ms warm (measured in isolation, ~45-50ms as observed in packaged runs once BPE rank modelParams + recommended-encoding loaders are resolved). On the main thread that load is pure overhead: the actual tokenizer runs inside the metrics worker pool via `tokenCounterFactory`, which the main thread never touches. The library export in `src/index.ts` keeps working via the re-export. Extracting the constant defers gpt-tokenizer's entire module graph to first worker spawn, where it already overlaps with `searchFiles`/`collectFiles` thanks to the pre-warmup landed earlier in this branch. ## Benchmark Interleaved A/B on this machine, same source tree, `--include 'src/**'`, warmup=3, then n=40 interleaved measurement runs (median in ms): | workload | n | baseline median | patched median | delta | |---------------------------|---:|----------------:|---------------:|-----------------:| | `--include 'src/**'` | 40 | 1095 ms | 1066 ms | -29 ms ( -2.65%) | | `--include 'src/**'` | 30 | 1089 ms | 1058 ms | -31 ms ( -2.85%) | | `--include 'src,tests'` | 20 | 1187 ms | 1167 ms | -20 ms ( -1.68%) | Mean delta on the primary `src/**` workload is -33 ms (-3.00%) across 40 samples — clearly above the 2% threshold. The `src,tests` workload has more per-file work that dilutes the fixed ~30 ms startup saving to 1.7%, so the percentage shrinks as the wall-clock grows; the absolute saving is unchanged. Output is byte-identical between baseline and patched on the same source tree (`diff -q` on the full packed output of `src/**`). ## Verification - `npm run lint` passes (only 2 pre-existing warnings in unrelated files) - `npm run test` — 117 test files / 1166 tests passing - Byte-identical pack output confirmed for `--include 'src/**'`