Lower `EAGER_WARMUP_THREADS` from 2 to 1 when `tokenCountCacheFileExists()`
returns true. With the persistent token-count disk cache populated by a
prior run, `calculateFileMetrics` serves every per-file token count from
the in-memory map and dispatches zero worker tasks. The only worker work
that survives caching on a warm rerun is a small fixed set of dispatches:
- the wrapper-token tokenization (cache hit after run #2)
- git diff staged/worktree token counts (only when
`output.git.includeDiffs` is enabled)
- git log token count (only when `output.git.includeLogs` is enabled)
That worst case is 2-3 short tasks (a few KB each) that fit a single warm
worker serially in well under 30 ms. Spawning a second warm worker means
a redundant ~340 ms BPE table parse that contends with the file-collection
main thread for CPU AND extends the final `pool.destroy()` blocking wait
(BPE-loaded workers take ~21 ms to terminate vs ~3 ms when idle).
Cold-cache (no cache file) behavior is preserved: the unscoped path keeps
3 warm workers and the explicit-scope path keeps 2, so the actual file
tokenizations still parallelise across the original worker counts.
The probe is a coarse heuristic — a cache file written by a previous run
that used a different `tokenCount.encoding` (e.g. cl100k_base instead of
the default o200k_base) yields no hits for the current run, so the metrics
phase pays one BPE parse sequentially on the critical path before
tokenizing files. This is a one-time cost on encoding switches; subsequent
runs rebuild the cache for the new encoding and hit again.
Benchmark (paired, n=25, repomix self-pack on 1068 files):
WARM CACHE (cache file present)
BASELINE mean=968.9ms median=976.0ms sd=40.3ms
AFTER mean=883.2ms median=875.0ms sd=33.1ms
DELTA mean=85.6 ms (8.84%) median=87.0 ms sd=42.7
t=10.02 (df=24) faster=24/25
COLD CACHE (cache file deleted before each run, n=12)
BASELINE mean=1606.3ms median=1588.0ms sd=58.6ms
AFTER mean=1593.2ms median=1598.5ms sd=58.6ms
DELTA mean=13.2 ms (0.82%) t=0.62 faster=9/12 — within noise
Stacks on top of the existing warm-cache wins on this branch (token-count
disk cache, output-wrapper cache, prefetched template, native ignore-file
prescan, etc.); this single change pushes warm-cache wall-clock another
~86 ms below the previous floor.
agent-carnet now serves as the project-local notebook for AI agents
(introduced in #1564). The agent-memory skill is no longer used in
this repository, so its bundled SKILL.md and memories/.gitignore are
removed. Note that auto-memory loaded by Claude Code itself is a
separate, built-in mechanism and is unaffected by this change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduce agent-carnet, a file-based markdown notebook for AI agents,
as an installable skill. Notes live under .carnet/ as personal,
git-ignored content (only .gitignore and README.md are tracked).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Companion to the previous commit. Plumb `tokenCountCacheFileExists` into
the packager `defaultDeps` so the metrics warm-up sizing can be exercised
deterministically from tests, and add a paired test that asserts the
2-warm-up-worker branch is taken when the persistent disk cache exists.
Also rename the cold-cache test to make the new gating explicit and refresh
its docstring with the warm/cold distinction.
https://claude.ai/code/session_01TJqKkJ8n3r6Pa2JdW9Vp2w
The metrics worker pool eagerly spawns N workers at pack startup so each
worker can parse gpt-tokenizer's o200k_base BPE table (~340 ms of pure-CPU
work) in parallel with file search and collection. Previously a fixed
EAGER_WARMUP_THREADS=3 was used on the unscoped (default-scan) path because
3 BPE parses amortize across the file tokenization that follows.
With the persistent token-count disk cache (introduced earlier on this
branch), warm-cache repeat runs serve almost every per-file token count
out of the in-memory cache and dispatch zero worker tasks for them. The
3rd worker's ~340 ms BPE parse becomes pure overhead that contends with
file collection (~360 ms) and security check (~140 ms) for the 4 cores
on a typical host.
Gate the 3rd warm-up worker on `tokenCountCacheFileExists()` (a sync
existsSync on the cache JSON in $TMPDIR). When the cache file exists from
a previous run we treat the run as warm-cache-likely and warm 2 workers;
when it is missing (true cold cache) we keep the original 3-worker warmup
so the actual tokenizations parallelise.
Inject tokenCountCacheFileExists via the packager `deps` object so the
test suite can deterministically exercise both branches without depending
on /tmp filesystem state. Keep the existing `hasExplicitScope` gate
intact — explicit scopes still warm only 2 workers regardless of cache
state, matching the prior tuning for shorter metrics phases on small
file sets.
Benchmark (n=30, paired, NODE_DISABLE_COMPILE_CACHE=1, repomix self-pack
on 1047 files, 4-core host):
Warm cache (cache file present)
BASELINE median=1162.7 mean=1145.9 sd=62.8 ms
AFTER median=1033.7 mean=1035.3 sd=50.4 ms
DELTA mean=110.5 ms (9.65%) median=110.3 ms
t=11.97 (df=29) faster=30/30
Cold cache (cache file deleted before each run, n=20)
BASELINE median=1658.8 mean=1675.0 sd=91.1 ms
AFTER median=1632.0 mean=1652.3 sd=102.9 ms
DELTA mean=22.7 ms (1.36%) median=42.2 ms
t=1.29 (df=19) faster=13/20 — within noise
Test plan:
- All 1261 tests pass (+1 new test for the warm-cache branch)
- Lint clean
- Hosts with `getProcessConcurrency() < 3` are unaffected: the
`Math.min(processConcurrency, EAGER_WARMUP_THREADS)` floor in
`getWorkerThreadCount` already collapses to the host CPU count.
https://claude.ai/code/session_01TJqKkJ8n3r6Pa2JdW9Vp2w
Actions in this repo are SHA-pinned via pinact, so Renovate classifies
SHA bumps as `digest` (and the initial pinning as `pin`). Without
adding them to matchUpdateTypes, those updates would skip the group and
land as individual PRs, defeating the grouping.
Extend the manager-based grouping to dockerfile and nix so base image
bumps across the four Dockerfiles and flake.nix updates each batch into
a single PR per update channel.
Add packageRules for the github-actions manager so workflow dependency
bumps are grouped into one PR per update channel, mirroring how the
package.json updates are already batched.
Previous memoization stored a single boolean at module scope. In any
Node context where the same module instance might be reused across
requests (VitePress SSG, dev server, preview server with `navigator`
polyfilled per request), the first request's UA would silently leak
into subsequent calls.
In production this code is browser-only — Cloud Run's Hono server
doesn't import `botDetect.ts`, and Cloudflare Pages serves the bundle
as static files with one fresh module instance per browser tab — so
the bug was theoretical. But the UA-keyed memo costs nothing extra
and removes the foot-gun: a long-lived process now invalidates the
cache automatically when a different UA shows up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The composite action hard-coded `setup-node` to Node 24, so the
`test-action.yml` matrix `[22, 24, 26]` silently ran every job under
Node 24 — the 22 and 26 cells did not actually exercise those Node
versions.
Add a `node-version` input to the action (default `"24"` to preserve
current behavior for downstream consumers) and pass `${{
matrix.node-version }}` from each `test-action.yml` invocation so the
matrix tests what its name implies.
Four items from gemini and claude reviews:
- botDetect.ts: Memoize isBot() result. navigator.userAgent is immutable
for the page lifetime, so re-running the isbot regex on every Turnstile
pre-mint debounce and post-submit re-mint check is wasted work. SSR
fallback is intentionally not cached so a module instance reused across
SSR/CSR still reaches the real UA check on first CSR call.
- usePackRequest.ts: Disambiguate the "submit-path NOT gated" comment —
it was confusing because the new post-submit re-mint also lives inside
submitRequest's finally. Reworded to "click-path acquireTurnstileToken"
to make clear which call site is intentionally skipped.
- usePackRequest.ts: Update the userTouched comment to reflect autofill
reality — modern Chromium/Firefox DO fire input events on autofill, so
the rationale ("autofill doesn't trigger") was already stale. The new
isBot() guard covers the gap for well-behaved crawler UAs.
- usePackRequest.ts: Add English glosses for the Japanese CF dashboard
labels (提示チャレンジ / 未解決) so non-Japanese-reading maintainers can
follow the comment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`bun install` fires the `prepare` lifecycle hook before `setup-node`
runs, so it executes against the runner's default Node (currently
< 22). `node --run` is unrecognized there and `bun install` aborts
with exit code 9, breaking every Bun matrix job.
Other inner `node --run` script changes (browser, website-client,
website-server lint, browser build-all) are kept as-is — they only
fire when explicitly invoked from CI steps that have already run
`setup-node`, so the Node-version prerequisite is guaranteed.
Reverts the `prepare` portion of 5a1423e.
The CF Turnstile dashboard shows ~17k unsolved challenges over the past
7 days vs ~10k solved — a 2:1 unsolved:solved ratio that's larger than
the post-submit auto re-mint can explain. Most of those are JS-executing
crawlers (Slackbot, Discord card validator, Twitter card validator, X,
Apple link preview, Googlebot, etc.) that render the page, somehow trigger
a DOM input event on the URL field (autofill / accessibility tools /
focus tricks), and pay a CF challenge they can't solve.
Add an `isBot()` short-circuit at the two pre-mint call sites in
`usePackRequest`:
- the debounced `onTrigger` after `markUserTouched` flips
- the post-submit auto re-mint in `submitRequest`'s finally block
The actual security gate is the server-side `siteverify` in
`turnstileMiddleware` — that stays the only authoritative check, so a
crawler that spoofs UA past `isBot()` still gets blocked there. The
submit-path `takeToken()` is intentionally NOT gated to avoid
false-positive lockouts of legitimate users with unusual UAs (e.g. older
clients, accessibility tools).
Net effect:
- "提示チャレンジ" / "未解決" CF dashboard counters drop sharply
(well-behaved crawlers stop minting unsolvable tokens)
- `pack_completed` server logs unaffected (legit users don't change
paths; bots couldn't reach `/api/pack` either way)
- server-side spend on siteverify unchanged
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The hi/vi/id translations of the GitHub Actions guide still showed
`actions/checkout@v3`, `actions/setup-node@v3`,
`actions/upload-artifact@v3`, and `softprops/action-gh-release@v1`,
which are stale relative to both the English doc (already on @v4) and
the project's own CI workflows.
Bump these illustrative examples to @v4 (and @v2 for action-gh-release)
to match the English source and avoid pointing readers at deprecated
action majors.
Follow up to commit 042750c which only converted workflow-level
invocations. With Node.js 22 as the floor, the chained scripts inside
each package.json can also use `node --run` directly, dropping the
intermediate npm process when these scripts run.
- root `prepare`
- browser `build-all`, `lint`
- website/client `lint`
- website/server `lint`
- Migrate the build-and-run job in ci.yml to `node --run build --
--sourceMap --declaration` so the inline command matches the
`node --run` style of the test job (claude review round 2 #1)
- Update the hi github-actions.md matrix example from `[22, 24]` to
`[22, 24, 26]` so the doc mirrors the project's actual CI matrix
(gemini-code-assist / coderabbitai inline comment)
- Bump the stale "(Node 20+)" baseline in reviewer-performance.md to
"(Node 22+)" to track the new engines floor (claude review round 2
minor)
Sweep the multilingual website docs for remaining Node.js 20 references
that survived the initial English-only update:
- installation.md: System Requirements `≥ 20.0.0` → `≥ 22.0.0` across
all 12 languages that include this section (incl. en, which was
missed earlier)
- development/index.md: Prerequisites `≥ 20.0.0` / `v20` / `versi 20`
/ `phiên bản 20` → 22 across all 14 languages
- faq.md: programming-language Q&A narrative "Node.js 20 or later" → 22
across all 14 languages
The version digit substitutions are mechanical and identical across
locales, so updating them in this PR keeps the docs consistent with the
new minimum without requiring the usual translation handoff.
Now that the minimum supported Node.js version is 22, `node --run` is
available everywhere. It avoids the npm process-spawn overhead and
matches the style already used in package.json scripts.
Affects all GitHub Actions workflows that invoke npm scripts and the
website/server Dockerfile bundle step. `npm ci` is left as-is since it
is npm-specific.
Node.js 20 reaches end-of-life on 2026-04-30, so raise the minimum
supported version to 22 (the next active LTS) and add Node.js 26 to the
CI matrix as the current release line.
- Bump engines.node to >=22.0.0 in package.json and scripts/memory
- Update CI matrix to [22.x, 24.x, 26.x] (drop 20.x and 25.x; 25.x EOL 2026-06)
- Update test-action.yml matrix to [22, 24, 26]
- Drop the obsolete `node --run` workaround comment in ci.yml since
`node --run` is supported on all matrix versions
- Update Node.js version mentions in English docs, llms-install.md,
configShard, bug report template, and code samples in hi/vi
github-actions guides
Dockerfile (node:22-slim) is intentionally left at the minimum supported
version so the published image confirms Repomix runs on the floor.
Reuse the content-addressed disk cache used for per-file token counts to
also memoize the "output wrapper" (output minus all file contents) token
count. The wrapper string is byte-stable across runs when neither the
rootDir tree nor headers/instructions change, so the second-run wrapper
tokenization is a guaranteed cache hit.
Why this is on the critical path:
- calculateMetrics blocks on the wrapper tokenization Promise alongside
per-file metrics; with a warm per-file cache, file metrics resolve in
~5 ms and the wrapper worker dispatch (~30 ms on the ~120 KB wrapper)
becomes the longest task in calculateMetrics, which itself is on the
pipeline's critical path after produceOutput.
- A cache hit replaces the worker round-trip with an MD5(wrapper) +
Map.get(), each <1 ms.
Behavior preservation:
- Cache key = `${encoding}:MD5(wrapper)[0:16]`, so any wrapper change
(headers, file set, sort order, template format) automatically misses.
- On miss, the original runTokenCount runs and the result is written
back via setCached so subsequent runs hit. No behavioral difference.
- Falls under the same CACHE_VERSION guard as per-file entries.
Benchmark (paired, warm cache, n=30, repomix self-pack):
BASE mean 1097.2 ms sd 33.5
AFTER mean 1063.7 ms sd 34.7
DELTA mean 33.6 ms (3.06%) median 32.8 ms
t = 4.761 (df=29) faster = 23/30
95% CI [19.1, 48.0] ms
Cold cache (paired, n=15, file deleted before each run):
DELTA mean -1.5 ms (-0.10%) t = -0.12 -- within noise
Tests: all 1260 pass; npm run lint is clean (the two pre-existing
biome warnings in cliSpinner.ts are unrelated).
Introduce a persistent token-count cache keyed by MD5(content) + encoding.
On warm runs (re-packing the same repo), the 600ms BPE metrics phase is
bypassed entirely, cutting total wall-clock time by ~28% (553ms).
- tokenCountCache.ts: new module — load/save JSON from /tmp/, in-memory
Map<string,number> with 100k-entry LRU eviction and CACHE_VERSION guard
- calculateFileMetrics.ts: classify files as cache-hit / cache-miss before
dispatching to workers; merge results back in original order
- packager.ts: fire-and-forget cache load at t=0; await before metrics;
save after metrics completes
Benchmark (n=30, paired t-test on repomix self-pack, warm cache):
Baseline (no cache): 1949.9ms +/- 73.0ms
Candidate (warm): 1396.9ms +/- 72.9ms
Improvement: 553ms (28.4%), t=28.20
https://claude.ai/code/session_01Fm25x51fmGGeFMJyCm1CER
Three targeted improvements to output generation and metrics:
1. **Skip redundant per-file scans in createRenderContext** (`outputGenerate.ts`)
- `calculateMarkdownDelimiter` (backtick scan, ~4 ms) now runs only for
markdown output; other styles use the default ``` fence.
- `calculateFileLineCounts` (newline scan, ~6 ms) now runs only when
the caller passes `needsLineCounts: true`; the regular output path
(XML / plain / markdown) never uses line counts.
- `packSkill.generateSkillReferences` passes the flag to keep its
statistics and tree rendering correct.
2. **Prefetch compiled Handlebars template** (`packager.ts` / `outputGenerate.ts`)
- `prefetchCompiledTemplate(style)` fires a background `getCompiledTemplate`
call immediately after security-worker warm-up, so the ~50 ms Handlebars
compile cost overlaps with `collectFiles` + security-check instead of
sitting on the critical path.
3. **Path-anchored `extractOutputWrapper`** (`calculateMetrics.ts`)
- `getFileContentStart` locates each file's content in the output string
using a style-specific path anchor (`<file path="...">` for XML,
`## File: ...` for markdown, `File: ...` for plain) rather than a raw
`indexOf(content)`.
- Prevents a false match when one file's content appears verbatim inside
an earlier file in the output, which would cause `extractOutputWrapper`
to return `null` and silently fall back to full-output tokenization.
- The fast-path token calculation now passes `config.output.style` to
`extractOutputWrapper` so the anchor logic is always engaged.
https://claude.ai/code/session_01Fm25x51fmGGeFMJyCm1CER
Add fs.readdir mock (returning empty array) to all relevant beforeEach
blocks so collectIgnoreFilePatterns does not fail with "entries is not
iterable". Update globby option assertions to reflect gitignore: false
and ignoreFiles: [] now that patterns are pre-collected by the prescan.
https://claude.ai/code/session_01Fm25x51fmGGeFMJyCm1CER
globby's `gitignore: true` + `ignoreFiles` options each trigger an extra
full-tree traversal to discover and parse .gitignore / .ignore /
.repomixignore files. On the repomix repo itself this adds 200–500 ms to
the searchFiles phase (measured via --verbose [globby] log lines).
Replace both traversals with a single native fs.readdir-based prescan
(`collectIgnoreFilePatterns`) that:
- walks only non-skip directories in parallel (skipping node_modules,
.git, dist, build, lib, etc.)
- reads .gitignore, .repomixignore, and .ignore in one pass
- prefixes each pattern with its directory path and merges into the
main `ignore` array
- skips negation lines (!) to avoid cross-directory semantic issues
globby is then called with `gitignore: false, ignoreFiles: []` so it
performs only a single traversal for file discovery.
Benchmark results (--verbose [globby] elapsed, cache-warm, NODE_DISABLE_COMPILE_CACHE=1):
src,tests scope: 124–156ms → 82–109ms (~35% reduction in searchFiles)
Full repo scan: 664–1153ms → 195–499ms (~60% reduction in searchFiles)
https://claude.ai/code/session_01Fm25x51fmGGeFMJyCm1CER
Earlier push commit (03b8e70b) accidentally dropped the
`expect(actualOutput).toContain('# Directory Structure')` assertion
inside the markdown branch of the integration test's case-style switch.
That line still exists on disk and was present pre-change; restoring it
to keep the markdown-style coverage symmetric with the plain-style
case below it.
Final commit of the perf(core) Pre-warm security worker pool change —
extends the unit packager test and the integration packager test:
- tests/core/packager.test.ts: adds `createSecurityTaskRunner` mock to
the orchestration test's `mockDeps` and to the `parallel error
handling` `baseDeps()` shared fixture, updates the
`validateFileSafety.toHaveBeenCalledWith` assertion to expect the new
6th-argument deps object (`{ taskRunner: <Object> }`), and adds
positive/negative gate assertions —
`expect(deps.createSecurityTaskRunner).toHaveBeenCalled()` for the
default unscoped path, `.not.toHaveBeenCalled()` for the
`--include 'src'` and `explicitFiles` (--stdin) paths.
- tests/integration-tests/packager.test.ts: adds the
`createSecurityTaskRunner` stub so the default-scope path no longer
attempts to spawn a real worker pool (the previous unhandled-rejection
noise from a missing worker file URL is gone with this change).
(See PR description / first source commit for the full perf change
rationale, benchmark numbers, and correctness notes.)
Continuation of the perf(core) Pre-warm security worker pool change —
extends `mockDeps` / inline pack-test plumbing in the three smaller test
files so the default-scope path no longer attempts to spawn a real
worker pool from the test environment.
- tests/core/packager/diffsFunctionality.test.ts: adds
`mockCreateSecurityTaskRunner` to both pack-call sites.
- tests/core/packager/splitOutput.test.ts: same — adds the stub to the
inline mock deps.
- tests/core/security/validateFileSafety.test.ts: updates the
`runSecurityCheck` call assertion to include the new
`{ taskRunner: undefined }` deps argument forwarded by
`validateFileSafety` when no pre-warmed runner is provided.
(See PR description / parent commit for the full perf change rationale,
benchmark numbers, and correctness notes.)
Kazuki Yamada
dependabot[bot]
renovate[bot]
autofix-ci[bot]