- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes FastCode, an efficient framework for repository-level code understanding and reasoning. It targets the high token overhead that existing coding agents incur by repeatedly processing full code content during multi-round code exploration. The core idea is to decouple repository exploration from full-code reading, which reduces context cost while preserving iterative exploration. Specifically, FastCode first builds a lightweight semantic-structural representation by combining hybrid retrieval, hierarchical code units, and dependency, inheritance, and call graphs. It then progressively locates relevant code through query augmentation, tool-assisted search, and graph expansion, without loading full code implementations. Finally, a cost-aware context management policy dynamically controls the exploration process based on query complexity, context sufficiency, and resource budget, and selects the relevant code for final reasoning.
Critical strengths
- The research problem has practical significance, and the method design is reasonable.
- The experiments are fairly comprehensive, and the results are good.
- The paper is easy to follow.
Critical weaknesses
I have the following concerns about this paper:
- I have some confusion about the optimization objective in the method section. In Equation (1) of Section 3.2, the authors formulate context selection as an optimization problem that balances relevance and computational cost. However, the paper does not seem to clearly specify how the relevance function , the cost function , and the trade-off coefficient are defined and computed in the actual method. Moreover, Section 4.3 mainly relies on heuristic strategies based on confidence and budget constraints to manage the context, and the relationship between these strategies and the optimization objective in Equation (1) remains unclear.
- The method section describes the complementary roles of BM25 and dense retrieval, but it does not specify how the two retrievers are fused, or what content is used to build the vector index. In addition, I have some confusion about the definition of the Information Gain Rate (IGR) in Section 4.3.2. The authors measure the exploration gain as the ratio of the change in the context-sufficiency score to the change in the number of candidate code lines. However, during iterative filtering, the total number of candidate lines may stay the same or even decrease. This could make the denominator of IGR zero, or give it a sign that does not accurately reflect the change in context sufficiency.
- FastCode mainly relies on lightweight structural metadata for exploration, but the authors use the LLM’s self-assessed to decide whether to continue retrieval or terminate early. Without access to the full code implementation, the model may struggle to judge accurately whether the current candidates already contain all the evidence needed to solve the problem.
- In the Cost Calculation paragraph of Appendix A.2, the authors explicitly state: "For baselines, we directly adopt the cost figures reported in their original publications." In other words, FastCode’s cost is computed from the API services and pricing used in this paper, such as Poe and OpenRouter, whereas some baseline costs are taken from numbers reported in prior papers. My biggest concern here is that costs reported in different papers are not necessarily directly comparable. Furthermore, the fairness of the execution protocol on GitTaskBench needs further clarification. FastCode generates an executable
.shscript that is run by a predefined automation script, whereas SWE-agent and OpenHands are interactive coding agents. The cost advantage may therefore come partly from differences in execution protocol, rather than entirely from Structural Scouting. In addition, FastCode needs to build a BM25 index, an embedding index, and an AST relation graph, and it performs multiple rounds of exploration. Do the reported costs include these preprocessing operations? - How much of FastCode’s gain comes from "not reading the full code during exploration," and how much comes from query augmentation, graph retrieval, and other engineering components? The ablation study cannot determine this.
- I have some concerns about whether the backbones are comparable across methods. In Table 7, FastCode’s best result on LOC-BENCH (86.13%) uses Gemini-3-Flash, while all baselines use Claude 3.5 Sonnet. Under the same backbone, FastCode’s Acc@1/3/5 are all lower than those of LocAgent. Similarly, in Section 5.3, FastCode with Gemini-3-Flash outperforms OpenHands with Claude 3.7 Sonnet, but Gemini-3-Flash is a newer-generation model, and "lightweight" does not mean weaker. I suggest the authors compare methods on the same backbone to isolate the framework's contribution.
Minors
- The text in Figure 1 is too small to read comfortably. In addition, Figure 1 uses radar charts to compare Accuracy and Cost across benchmarks, but the left chart lacks a clear numerical scale, and the percentage ticks in the right chart are not evenly spaced.
- The paper uses two spellings, "locBench" and "LOC-BENCH", which should be unified.
- On P4, "Iniitialization" in Figure 2 is misspelled.
- On P5, L238–239, "returns file path and match counts and without loading full files" contains a redundant "and".
Questions for authors
Please see the Weaknesses above; the relevant questions are all included there.
4 comments
Response to Reviewer CRyb (2/2)
4. Cost accounting and execution protocol (W4)
Response to W4: We thank the reviewer for these questions. For the SWE-QA and LongCodeQA costs in Figure 4, we calculate API costs for FastCode and non-commercial baselines using the rates in Table 3. For commercial tools, we use their console-reported costs. For GitTaskBench and LOC-BENCH, matched-backbone comparisons use the same rates across methods: USD 3 per million input tokens and USD 15 per million output tokens for Claude 3.5/3.7 Sonnet [1-2]. This aligns the pricing basis for FastCode and baselines such as OpenHands, SWE-agent, and LocAgent.
On GitTaskBench, the automation script only runs FastCode's generated .sh file; all task-specific operations come from FastCode, with no additional task-solving logic supplied by the executor. SWE-agent and OpenHands can likewise generate and execute scripts through their shell tools to complete tasks. All methods follow the same task requirements and ECR/TPR evaluation criteria.
FastCode's reported API cost includes every task-solving LLM call, covering query augmentation, multi-round exploration, and final reasoning. BM25 and embedding indexes and AST relation graphs are built locally on CPU as repository preprocessing and can be reused across queries. These operations incur no LLM API charges. We will revise Appendix A.2 to distinguish the pricing rules and document the execution protocol and accounting scope.
[1] GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging. arXiv preprint, 2025.
[2] LocAgent: Graph-Guided LLM Agents for Code Localization. ACL, 2025.
5. Component contributions in the ablation study (W5)
Response to W5: We thank the reviewer for this question. The contributions of these components are already examined in Section 5.5 (Ablation Study) and Fig. 5, as detailed below.
(1) Structural scouting reduces repeated token processing by keeping code bodies out of exploration history. Its tool-based navigation contribution is assessed by comparing w/o C with w/o T&C: with query augmentation retained and cost management disabled in both, removing tool-based scouting increases the reported performance drop from 0.92% to 4.31%.
(2) Query augmentation bridges query intent and code terminology through rewriting and implementation hints. Both w/o T&C and w/o N&C disable tool-based scouting and cost management; additionally removing query augmentation increases the drop from 4.31% to 5.06%.
(3) Graph retrieval captures related code beyond initial similarity matches through call, dependency, and inheritance relations. Removing graph expansion alone (w/o G) reduces performance by 0.83%.
(4) For other components, hybrid retrieval supports semantic matching and precise identifier lookup; removing it (w/o R) causes a 0.60% drop. Cost-aware context management regulates exploration according to sufficiency and budgets; removing it (w/o C) causes a 0.92% drop.
6. Comparisons with the same backbone (W6)
Response to W6: We thank the reviewer for raising this point. We already compare FastCode and LocAgent on LOC-BENCH using the same Claude 3.5 Sonnet backbone in Appendix A.3.1 (Table 7). FastCode achieves 76.28% Acc@1 at USD 0.365 per instance, compared with 77.74% at USD 0.660 for LocAgent. The Acc@1 gap is 1.46 percentage points, while FastCode's reported cost per instance is 44.7% lower.
On GitTaskBench, we also report comparisons using the same backbone for FastCode, SWE-agent, and OpenHands in Section 5.3 (Figure 3) and Appendix A.3.1 (Table 6), covering both Claude 3.5 Sonnet and Claude 3.7 Sonnet. FastCode leads both baselines in Execution Completion Rate (ECR) and Task Pass Rate (TPR) with either backbone. With Claude 3.5 Sonnet, FastCode achieves 61.11% ECR and 46.30% TPR, versus 53.70% and 40.74% for OpenHands. With Claude 3.7 Sonnet, the corresponding values are 74.07% and 57.41% for FastCode and 72.22% and 48.15% for OpenHands. The Gemini-3-Flash results serve as a supplementary evaluation, showing FastCode's performance with an additional LLM backbone.
Minor comments on figures and wording
Thank you. We will enlarge Fig. 1's text and add a scale to its left chart. The right chart's uneven ticks show differences in cost. We will refine them for clarity while preserving that contrast. We will use LOC-BENCH consistently, fix Fig. 2's typo, and remove the extra 'and'.
Final Assessment and Rationale for Maintaining Weak Reject
Thank the authors for their detailed responses and additional experiments. However, after careful reconsideration, I remain inclined to maintain my original score (Weak Reject) for the following primary reasons:
- Lack of Technical Novelty. This remains my primary concern. FastCode mainly integrates existing techniques, including BM25, Dense Retrieval, Graph-based Retrieval, Query Augmentation, and Agentic Exploration. The authors have not convincingly demonstrated that their core "exploration-consumption decoupling" provides a substantial methodological contribution. In my view, this work represents an engineering integration rather than a significant methodological innovation.
- Heuristic-based Optimization. The proposed "cost-aware optimization" relies heavily on predefined heuristic confidence thresholds, line budgets, and stopping rules, lacking a principled optimization mechanism.
- Insufficient Validation. The reliability of the LLM-based context sufficiency assessment remains unverified, and the current ablation studies do not adequately isolate and prove the independent contribution of the core mechanisms.
- Limited Evidence for Efficiency and Generalization. The claimed efficiency advantage is not consistent across baselines. On SWE-QA, FastCode achieves a higher score (43.28) than Func RAG (39.62) and Sliding RAG (40.22), but its per-question cost (USD 0.0320) is also substantially higher than theirs (USD 0.0112 and USD 0.0200, respectively). Similarly, on LOC-BENCH under the same Claude 3.5 Sonnet backbone, FastCode achieves lower localization accuracy than LocAgent (Acc@1: 76.28% vs. 77.74%), despite its lower cost (USD 0.365 vs. USD 0.660). These results suggest an accuracy-cost trade-off rather than a consistent efficiency advantage. Furthermore, the cross-language evaluation covers only 22 queries across three repositories, with just one query for greenlet, which is insufficient to establish robust generalization across programming languages.
Finally, to be candid, I respectfully disagree with Reviewer XMa8's assessment. While this paper holds a certain degree of practical engineering value, its technical novelty is insufficient to justify a score of 4 (Clear Accept). The completeness of an engineering implementation and the importance of the research problem should not be conflated with technical innovation.
Response to Reviewer CRyb's Final Assessment
We thank the reviewer for the follow-up and appreciate the opportunity to address the additional points raised in this comment.
1. Novelty.
We do not regard the use or combination of BM25, dense retrieval, graph-based retrieval, and similar techniques as a contribution of this paper. Our contribution is to decouple repository exploration from code consumption, which changes how the agent loop is organized.
In iterative agents such as SWE-agent and OpenHands, code read during exploration stays in the history and is re-processed in every later round, so cost grows with both the amount of code inspected and the number of rounds. In FastCode, the exploration loop sees only metadata (e.g., signatures, docstrings, and dependency relations), and code bodies are loaded once, after exploration ends. This is a change to the method rather than an added component: exploration stays iterative and adaptive, but no code is re-processed across rounds. The established techniques above serve to realize this idea more effectively: they supply and rank the candidates the agent examines.
The paper shows that an agent can explore a repository without loading code bodies into its context, tracing dependencies and pinpointing the code a query needs by navigating a lightweight semantic-structural map, and that this changes how cost scales. For example, on LongCodeQA, FastCode's cost stays within USD 0.041–0.069 as repositories grow from 32K to 256K tokens, while direct input rises from USD 0.083 to 0.666 (Table 5). On GitTaskBench with the same backbone, FastCode costs USD 0.0878 per task vs. 1.38 (SWE-agent) and 8.95 (OpenHands) (Table 6).
2. Cost-aware optimization.
The cost-aware policy aims to gather sufficient query-relevant evidence at low cost, by balancing estimated context sufficiency against exploration depth and code volume. It has three advantages.
(1) Adaptive. Rather than applying fixed values, the policy itself adjusts the line budget, iteration limit, and sufficiency threshold at run time, according to the estimated query complexity and the structural complexity of the repository.
(2) Efficient and training-free. The policy needs no training and no additional model. It ends exploration once the evidence is judged sufficient or added code brings little further gain, and it works across multiple backbones.
(3) Stable. The policy keeps per-query cost in a narrow range as repositories grow (Table 5).
3. Validation.
(1) Sufficiency assessment. For each candidate, the LLM sees its signature and docstring, which indicate what the unit does; retrieval and tool evidence, which show how it matches the query; and call, dependency, and inheritance relations, which show which other units it relies on. From these, it can judge whether the candidates cover the functions and dependencies the query involves. Two results support that the LLM-based context sufficiency assessment is effective. (a) The sufficiency score tracks answer quality: as its mean rises from 83.0 to 91.7, the answer score rises from 40.79 to 43.40 (Table 9). (b) Context selected under this assessment matches or exceeds full-repository input, which contains all evidence, in nearly all settings (Table 2).
(2) Ablation. Sec. 5.5 (Fig. 5) ablates multiple mechanisms with the rest fixed, and each removal lowers the score: hybrid retrieval (w/o R, −0.60%), graph expansion (w/o G, −0.83%), cost-aware control (w/o C, −0.92%), tool-based scouting (a further −3.39%, w/o C to w/o T&C), and query augmentation (a further −0.75%, w/o T&C to w/o N&C).
Details are in our Responses to W3 and W5.
4. Efficiency and generalization.
An accuracy–cost trade-off is our stated claim: the abstract, Sec. 5.4, and Fig. 4 report favorable trade-offs, not the lowest cost on every benchmark. Cost comparisons are meaningful between methods of comparable accuracy.
(1) SWE-QA and LOC-BENCH. The RAG baselines are not at a comparable level: they score 3.06–3.66 points below FastCode. FastCode matches Cursor and exceeds Claude Code (43.28 vs. 43.17 and 42.08) at 55% and 44% lower cost. Against LocAgent, it is within 1.46 points at Acc@1 at 44.7% lower cost.
(2) LongCodeQA and GitTaskBench. FastCode leads on both accuracy and cost (Claude 3.5). On LongCodeQA, it matches or exceeds full-repository input on every subset from 32K to 256K, at 50.6–91.4% lower cost (Table 5). On GitTaskBench, it exceeds SWE-agent and OpenHands in ECR and TPR at 93.6% and 99.0% lower cost (Table 6).
(3) Cross-language. The paper makes no claim of cross-language generalization and therefore does not evaluate it. The 22-question experiment was added at Reviewer fFXD's request for at least one non-Python repository; we reported every LongCodeQA question for three of them.
We hope these responses address the reviewer's concerns and would appreciate a reconsideration.
Response to Reviewer CRyb (1/2)
1. Optimization objective and context management (W1)
Response to W1: We thank the reviewer for the thoughtful question. Eq. (1) expresses our goal of obtaining sufficient query-relevant evidence at low computational cost. The context-selection objective is C∗=argmaxC⊂US(C;q), where S(C;q)=Rel(C∣q)−λΩ(C). For each candidate context, the LLM assesses query coverage and evidence completeness using metadata such as signatures, docstrings, retrieval evidence, and structural relations (Sec. 4.2.2). It assigns a sufficiency score κt∈[0,100], which estimates Rel(C∣q) (Appendix A.2.1). Ω(C) captures two cost components: code volume Lt for final context consumption and exploration depth t for repeated scouting.
The trade-off represented by λ is realized through the confidence and budget controls in Sec. 4.3. Specifically, exploration stops when the current context is judged sufficient, as determined by the threshold τ. When evidence remains insufficient, exploration continues. Information-gain monitoring links relevance and cost by assessing sufficiency gains relative to added code volume, stopping after consecutive low-gain rounds. Finally, the budget bounds code volume and exploration depth, with final selection prioritizing strongly supported units within it. Together, these mechanisms implement the relevance-cost balance in Eq. (1). We will explicitly add the correspondence between the task objective and this policy in the revised manuscript.
2. Hybrid retrieval and IGR definition (W2)
Response to W2: We thank the reviewer for these questions. Hybrid retrieval combines semantic similarity with lexical matching. Embedding and BM25 candidates are merged by unit ID and ranked by the fusion score S(u)=0.5cos(eq,eu)+0.5BM25(q,u)/maxvBM25(q,v), where eq,eu are query/unit embeddings and missing scores are zero. Graph expansion follows fusion. Each file/class/function/documentation unit is embedded from its type, name, signature, docstring, summary, and bounded code text. Paths and line ranges remain metadata.
Due to space constraints, the main-text IGR equation does not detail the handling of unchanged or reduced context volume. IGR measures sufficiency gain per added code volume, so its ratio is evaluated only for positive growth. With Δκ=κt−κt−1 and ΔL=Lt−Lt−1, the implementation uses IGR=Δκ/ΔL when ΔL>0; otherwise it skips division and records zero as a placeholder, avoiding division by zero and sign reversal. This placeholder is assessed jointly with Δκ: a round remains productive when the sufficiency score improves meaningfully, even if context shrinks. Low-progress stopping requires two consecutive rounds with either a substantial confidence decline or both a small gain and low IGR. A single corrective decline is tolerated; sufficiency and resource limits apply separately. We will add the fusion rule, embedding inputs, and guarded IGR and stopping definitions to the appendix.
3. Reliability of context-sufficiency assessment (W3)
Response to W3: We thank the reviewer for this question. The LLM assigns the sufficiency score κt based on the candidate units' structured metadata, including signatures, docstrings, retrieval evidence, and structural relations. Function names, parameters, and type annotations provide cues about functionality and interfaces, while docstrings describe intended behavior. Call and dependency relations reveal how units interact and which related implementations may be needed. Together, these signals support assessing whether the candidate set covers the functional roles and dependencies required by the query, grounding the sufficiency estimate in both semantic and structural evidence.
Table 9 provides empirical support on the three-repository SWE-QA subset: base sufficiency thresholds of 65, 80, and 95 yield answer scores of 40.79, 41.52, and 43.19, respectively. The improvement under stricter sufficiency requirements supports the practical value of metadata-based coverage assessment for context selection.