Does it find the files
a task actually needs?
4 public JavaScript/TypeScript repos. 8 of their own real historical fix commits. The merged product command. A small first pilot — including the two results that came in below our own target.
Run date: 2026-08-13 · CLI: v0.10.63 · public value corpus release gate
What we can say, precisely
"On a first pilot of 4 public repos and 8 real historical fix commits, CodeLedger's selector achieved 53.8% pooled recall — finding just over half of the files each commit actually touched, from the commit message alone, with zero build or command failures."
We are not claiming this generalizes past these 4 repos, that recall stays at 53.8% at scale, or that precision (which ranged 0.9%-38.3%) is where we want it long-term. Those claims are not supported by this run. The one above is — and we'll publish the next, larger pass here too, whichever way the number moves.
Methodology
How we ran this
Four decisions shaped this pilot. We're sharing them because reproducibility is the product.
We used the merged product command, not a fixture
The public value corpus benchmark shipped in the same release as the selector changes it measures — a real release gate, not a standalone pre-release runner. Every number below came from one command.
Real repos, real commit history — no curated fixtures
Four public JavaScript/TypeScript repos, shallow-cloned fresh for the run. For each, the benchmark samples the repo's own recent commit history, checks out the parent of a real fix commit, builds a context bundle from the commit message alone, then compares the bundle against what that commit actually changed. Nothing here is a designed test scenario.
Repos in this pilot:
chalk/chalk · tj/commander.js · expressjs/express · fastify/fastify
This is a small sample — we're saying that plainly
Only 8 eligible commits total across 4 repos (2-3 per repo), covering 26 changed files. That is not enough to make a durable product claim, and we are not making one. It is enough to publish an honest first data point and commit to expanding it.
We recorded everything, including the two repos below target
chalk (43.8%) and express (33.3%) both came in below our internal quality target — but both cleared our minimum acceptance threshold, so nothing here failed outright. We did not drop them or reweight anything to make the aggregate look better.
The overall release-gate verdict on this run:
status: "warning" (2 of 4 repos below quality target, 0 below the hard floor, 0 command/build failures)
Results
Per-repo results
One realistic task per repo, benchmarked against 2-3 of that repo's own recent fix commits. The bar shows recall — the threshold shown is our internal quality target.
add support for color level detection fallback
fix help wrapping for long option descriptions
fix query parser setting behavior
fix reply serializer content type handling
About the below-target results
chalk (43.8%) — narrowly below our quality target on a single 2-commit sample. High bundle confidence (0.75), so the selector wasn't uncertain — it was confidently missing files that a larger, more diverse commit sample would likely surface.
express (33.3%) — the smallest eligible sample here (1 commit), which makes this single number the least trustworthy in the set. Medium bundle confidence (0.65) — the tool itself flagged this bundle as less certain, which tracked with the lower recall.
Interpretation
What this means for your team
On this sample, a little over half of the files a task actually needed were in the bundle the selector produced — from a commit message alone, with no hints. That's the number that matters for whether an agent has to go hunting for files on its own.
Precision varies a lot by repo (0.9%-38.3%) — most bundles include more context than the exact diff touched, which is often useful (related tests, call sites) but means the ratio of "exactly right" files is low. We report both numbers rather than only the flattering one.
This is a pilot, not a release claim. 4 repos and 8 commits is a start, not a statistically confident result — the same discipline we apply internally (never trust a small sample without a larger confirmation run) applies to what we publish here too.
This measures whether the selector finds the right files (recall/precision). Our Context-Compiler benchmark measures something different — how much smaller the bundle gets once those files are found. Both matter; neither substitutes for the other.
Our current posture
What we're doing with this data
One 4-repo pilot is not a durable product claim. Here's what we decided, and why.
Published as a pilot, labeled as one
53.8% pooled recall on 8 commits is a real, honestly-measured data point — not a claim about retrieval quality at scale. We're not rounding up.
More repos, more commits per repo
8 commits across 4 repos is too small a sample to trust on its own — internally, we don't trust our own results below N=50. This page will update when a larger run lands, in either direction.
Same discipline, public or internal
Every number on this page came from one merged command, run against real repo history, reported exactly as measured — including the results that missed our own internal target.
What we won't claim from this run
Reproducibility
Run it yourself
The benchmark that produced every number on this page is committed to the CodeLedger repo and runs as part of the release gate. It clones each repo fresh into a temp directory, so re-running it will sample whichever commits are most recent at the time — numbers will drift slightly run to run, which is expected and part of why we're calling this a pilot, not a fixed benchmark.
See your own recall number
codeledger benchmark --commits 20 --json runs the same retrospective methodology against your own repo's real history in minutes.
No credit card · Contributor tier always available · Local-only, no data leaves your machine