- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper claims to find a code search method that mitigates the recall-cost dilemma. Given a NL query, the proposed method (CODERETRX) uses a parser to break a repository into fragments and uses embedding models to retrieve the potentially relevant fragments and their corresponding functions. These functions are then provided to an LLM, which determines if the code snippet is relevant to the provided natural language query. The experimental results showed that the proposed method achieved high recall with lower cost.
Critical strengths
-
Proposed a method that combines embedding and generative models for repository code search.
-
Compares the "cost" of different methods based on a practical unit (USD)
Critical weaknesses
-
Very small benchmark size. The repository contains only 18 repositories. After checking the anonymous repo provided at the top of the paper, the number of queries is also very limited. It is unclear whether the recall improvement is due to improved methodology or overfitting on the small benchmark.
-
The evaluated embedding and generative model(s) are limited and outdated. The main experiment is based on OpenAI's text-embedding-3-large (released Jan 2024) and gpt-oss-120b (released Aug 2025). I would recommend testing newer embedding/generative models to demonstrate the generalizability of CODERETRX (please see questions).
-
It is unclear how cost (in USD) in calculated in the experiments, making the "recall-cost dilemma" somewhat confusing. (See questions)
Questions for authors
Benchmarking
- The authors proposed a task on repository-level retrieval, but there are a large number of available repo-level repositories (e.g., LocBench from LocAgent cited in the paper, and RepoBench [1]). Evaluation results on existing benchmarks can help prove the method's generalizability and make it easier to compare with other methodologies.
- The current size of the benchmark (CODERETRBENCH) is very limited. Please consider extending it or including other benchmarks for evaluation.
Application Scenario
- The use case of CODERETRX is confusing. The paper discusses vulnerability detection as an important downstream task. However, if the task requires high accuracy or is serious, why would cost be a big issue? Additionally, the authors claim, "Exhaustive LLM traversal is complete but unaffordable." What about using smaller/cheaper LLMs for traversal?
Experiment
- The embedding model evaluated in the paper (text-embedding-3-large) is 2.5 years old. Based on my experience, there are newer (and potentially better) code retrieval choices on the market (e.g. Voyage-code-3 and newer Voyage-code-4, both are blackbox models; There are also a few competitive open source models from COIR leader board [2]). Similarly, there are also a lot of other choices for the generative model. There are tons of strong open-source/black-box releases since Aug 2025.
[2] https://huggingface.co/spaces/mteb/leaderboard
- It is unclear how "cost" is calculated in the paper, especially when the generative model being utilized is an open-sourced one. Is it simply based on # of tokens being embeded, and # tokens input/output for the generative model? If so, did you consider a reasonable KV-cache technique?