Coding task
PonyC
66% less input and 1m 10s faster in this test
Raw search
653,348
input tokens
- Task time
- 3m 05s
Codebase Context
223,558
input tokens
- Task time
- 1m 55s
Open-source code-search benchmark
Testing five code-search tools
I compared code-search tools on real coding tasks to see how well they found relevant files. Here are the results and how to try Codebase Context in your own project.
Try it in your project
Use Node.js 22+. Your tool starts CBC when it connects; your agent selects the repository you're working in.
Run once in PowerShell or a macOS/Linux terminal:
Claude Code
claude mcp add --scope user --transport stdio codebase-context -- npx -y codebase-context@2.2.0Start a new Claude Code session in this project. Approve the server if asked.
Client docs ↗Optional: set a fixed project folder
Leave blank to let the agent select its repository. A full path adds a default folder to the command above; Windows Copy as path works.
Then try it in your project
First use may need local indexing and a model download. Ask your agent:
Use Codebase Context to find [feature] in this repository. Pass this repository's absolute path as project when checking get_indexing_status and searching. Wait for indexing if needed, read codebase://context, then search_codebase and open a returned source file. Show me the relevant files.
Replace [feature] with something you want to find.
Check that your agent sees the right project and can open a returned source file.
Relevant files found
0-100% scaleShare of each task's reference files found by each tool, averaged across 20 tasks.
Raw search4.3%
Codebase Context25.7%
jCodeMunch27.1%
Repowise20.1% (failed run = 0)
context-mode16.8%
Useful results returned
0-100% scaleShare of each tool's returned files that were on the reference list, averaged across 20 tasks.
Raw search4.0%
Codebase Context11.5%
jCodeMunch11.0%
Repowise12.0%
context-mode11.7%
The bars count returned files. Raw search uses fixed text-search commands; its result is separate from the two-task agent test. Finding a file does not tell us whether the agent's change was correct.
Two tasks: tokens and time
For each task, I ran each tool three times. Token counts and time each show their own middle value. Token totals include cached text.
Coding task
fmt
About 15% less input; Codebase Context was slightly slower in this test
Raw search
290,186
input tokens
- Task time
- 2m 22s
Codebase Context
247,868
input tokens
- Task time
- 2m 24s
Tokens are pieces of text sent to the AI. These totals include cached text. The full breakdown is in the downloadable results under “How I tested it.” Two tasks cannot tell us whether every task will use less text or finish faster.
How I tested it
How I tested it
Test counts
These counts answer different questions, so I kept them separate instead of combining them into one score.
100
Main benchmark runs
Five tools x 20 tasks, one run each. The evaluator completed 99; one Repowise run returned no usable context and counts as zero.
300
Repeat tests
288 completed, 12 Repowise attempts had no usable context. These stay separate as test history.
30
Two-task test
Two tasks, five tools, and three runs of each. We recorded text used and time taken.
100
Older token-count tests
100 older records. I could not confirm that the tools were used as intended, so they are excluded.
What I tested
All five tools searched the same 20 coding tasks once each. I compared returned files with each task's reference list. The Repowise run with no usable context counts as zero. Separate tests cover repeated searches and agent runs.
Sources and data
The JSON downloads summarize the recorded results; they don't include every run file. You can also inspect the source files and test definitions.
- Original results summary (JSON)
- Download chart data (JSON)
- Download source files (ZIP)
- Benchmark script ↗
- Test rules ↗
- Test task list ↗
The original controller, full task payloads, and run files remain local, so the public materials do not reproduce this result with one documented command.
Source details
The source bundle contains files from original commit 18e0a43a90fe67e9df0d728b2134e9d430469883 (SHA-256 7be72393f93ddab954f0b0a08dfc5fb85d635e10de84d0819f03025a94ad057e). The GitHub files are pinned to cc54fb5aee50ec177402e459b23be0fe8da7836e; the four linked source files match the original revision.
The indexing measurements used the local Xenova/bge-small-en-v1.5 embedding model.
Limits
The main charts compare returned files with the files expected for each task. They don't show whether an agent made a correct code change. Token savings were tested on only two tasks, so they aren't a promise for your project.
Setup costs and search speed aren't directly comparable across tools: they built different indexes, install caches weren't controlled, and timing was measured differently. I also couldn't verify the units on the memory readings, so I haven't used them to rank tools.
I left out the older token tests because I couldn't verify that the tools were used as intended. The request to make at most 12 tool calls wasn't enforced.
How long did indexing take?
Codebase Context took about 12 minutes 31 seconds to build its index in these tests (the middle value). That excludes installation and searches, and doesn't predict how long your project will take.
I could not confirm whether every tool started with an empty model cache or used the GPU.