- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the gap between retrieving relevant repository evidence and actually delivering that evidence to a coding model under a limited context budget. It introduces VITAL-RAG, which groups retrieved views into code objects, selects and orders objects using relevance, file coverage, and estimated token cost, and constructs bounded contexts. Experiments on RepoBench separate the effects of grouping, packing, rendering, and selection. The strongest controlled policy result is a 3.58 percentage-point improvement in rendered-unit availability over rank-only selection under a shared renderer on held-out repositories. A small generation study is inconclusive, while RepoExec evaluates a broader workflow that includes two generated candidates and output selection. The contribution is primarily an empirical analysis of context allocation, with downstream benefits from allocation alone remaining unestablished.
Critical strengths
1.The controlled experiments separate losses caused by repeated views from losses introduced during context construction. Table 1 shows a substantial grouping benefit under full-source containment, while the candidate-count analysis demonstrates that packing becomes more consequential as more objects compete for a fixed budget. This decomposition provides useful knowledge for evaluating repository retrieval systems beyond candidate recall. 2.The shared-renderer evaluation provides credible evidence for a local policy improvement. In Table 3, VITAL-RAG improves rendered-unit availability from 82.47% to 86.05% at nearly unchanged selected recall and with fewer evidence tokens. The repository-held-out evaluation and repository-cluster confidence interval strengthen this finding. The authors also carefully distinguish evidence metrics from code correctness and explicitly acknowledge the inconclusive generation probe and the additional output-selection stage in RepoExec. This transparency makes the supported conclusions easier to assess.
Critical weaknesses
- Appendix A.2 permits a rendered unit to be a query-centered fragment of the original gold snippet. For Table 2, which compares different rendering configurations, it is not sufficiently explicit whether both conditions are evaluated against the same units fixed before allocation. If unit boundaries or lengths depend on the configuration, higher availability could partly reflect shorter evaluation targets rather than better preservation of equivalent evidence. This is a clarification concern, not a demonstrated metric error. Table 3 is better controlled because it shares a renderer, but downstream evidence does not yet resolve the metric’s practical significance: the 100-task generation probe is inconclusive, and the RepoExec improvement includes additional generation and static output selection. Consequently, the reported availability gains do not yet establish the independent utility of the allocator. 2.The main twelve-object policy comparison uses rank-only selection, although the paper itself reports that shortest-first and rank-valued exact knapsack outperform completeness-first packing at twenty objects. Those results use a different protocol and therefore do not directly refute Table 3, but they identify important alternatives that should be compared under its shared candidate pool, renderer, and budget. The MMR comparison in Table 11 uses a different five-object setting and does not settle this question. Moreover, the largest grouping benefit is demonstrated using three explicitly constructed views per snippet, leaving its benefit over a conventional single-view object retriever unclear. Together, these limitations make it difficult to determine how much new capability or generalizable insight the proposed policy adds beyond object deduplication and simple cost-aware packing.
Questions for authors
1.In Table 2, are the gold rendering units identical between Base and Full and fixed before allocation, independently of the remaining token budget? Please specify the window construction rule. If the units differ, can the existing contexts be reevaluated against a common set of units or full-source containment? 2. Under the exact Table 3 protocol, how does VITAL-RAG compare with shortest-first, rank-valued knapsack, and object-level MMR? A focused comparison would help determine whether the reported gain requires the proposed combined utility rather than a simpler allocation rule. 3. Using the already generated RepoExec programs, can you report candidate A, candidate B, and selected-output success separately for each model, with paired comparisons between B and the actual default A? Please also clarify whether the output-selection rules were fixed on separate development data. These results would help distinguish improvements associated with the allocation configuration from those introduced by the selection stage. 4. Is a single-full-source-view retrieval control available under the same scorer and budget as Table 1? Such a comparison would clarify whether the large grouping gain primarily recovers losses introduced by the constructed multi-view expansion or provides an advantage over an ordinary object-level retrieval setup.
3 comments
W2/Q4: Is there a source-only control using the same scorer and budget?
Thank you for suggesting a more detailed control. We have added an experiment that retains only the original source code without expanding it into additional views. The comparison before and after grouping the three views (24.99% → 39.16%) confirms that object-level grouping improves complete original-source containment. Furthermore, the grouped three-view configuration achieves higher complete original-source containment than the source-only configuration (39.16% vs. 32.92%), showing that the grouped configuration outperforms the single-view reference:
| Configuration | Complete original-source containment |
|---|---|
| Source-only retrieval (new) | 32.92% |
| Ungrouped three-view selection (original manuscript) | 24.99% |
| Three views + object-level grouping (original manuscript) | 39.16% |
Thank you for carefully reviewing our paper and for your constructive feedback. As the discussion period draws to a close, we wanted to follow up briefly and ask whether the additional experiments and clarifications above have addressed your main concerns.
This paper and this conference mean a great deal to me. We have invested substantial time and computational resources in this research, and this submission is tied to an important milestone in my life. If you feel that our responses have addressed your concerns, we would be grateful if you could consider updating your rating. We would be happy to clarify any remaining points.
Our detailed responses are above. For ease of reference, we summarize the additions most relevant to your comments below.
-
W1/Q1 (Evaluation of Base and Full). For each task, Base and Full use the same span from the gold snippet as the scoring window. We fix the span before allocating the token budget and count a hit when the final context contains the entire span.
-
W2/Q2 (Strong Baseline Comparison). Following the original Table 3 protocol, we added shortest-first, rank-valued knapsack, and object-level MMR, using the same candidate pool, object limit, renderer, and token budget. VITAL-RAG achieved 86.05% rendered-unit availability, compared with 84.96% for rank-valued knapsack, the strongest baseline in this comparison.
-
W1/Q3 (Candidate Performance and Output Selection). We reported execution pass rates for A, B, and the selected output separately. In a paired comparison before selection, B passed 16, 31, and 23 more tasks than the actual default candidate A (CodeRAG) on GPT-5.4, Claude Sonnet 4.6, and Qwen3-8B, respectively. We determined the selection rules on a development set and fixed them before evaluation on a held-out test set. We also reported results on all 13,121 RepoBench tasks and all 460 CoderEval tasks to address the limited scale of the original 100-task experiment.
-
W2/Q4 (Source-Only Control). We evaluated the source-only control using the same scoring criterion and token budget as the three-view experiments. The source-only control achieved 32.92% complete original-source containment. In the three-view setting, object-level grouping increased containment from 24.99% to 39.16%.
Response to Reviewer v2Aw
We sincerely thank you for carefully reading our paper and raising these important concerns. To demonstrate the generalizability and potential value of our research, we have devoted substantial computational resources to additional code generation experiments across multiple dimensions. We hope these additional results will address your reservations and concerns.
W1/Q1: Do Base and Full use identical evaluation units fixed in advance?
Thank you for pointing out that our description of evaluation fairness was insufficiently clear. We would like to clarify that Base and Full are both scored against the same complete gold snippets fixed in advance, which do not change with the remaining budget. Both contexts are limited to 4,096 tokens, including descriptors and separators. For scoring, we only normalize line endings and remove leading and trailing whitespace. A hit does not require the complete original text to appear verbatim in the final context. Partial retention also counts. The containment rate is the proportion of tasks counted as hits among all tasks. We will explicitly document these rules in the revised manuscript.
W1/Q3: Can you report the results for A, B, and the selected outputs separately, and explain how the selection rules were developed?
W2/Q2: How does the method compare with simple, strong baselines under the same protocol as Table 3?
Thank you for suggesting more targeted comparisons. Following your advice, we have added shortest-first, rank-valued knapsack, and object-level MMR. These methods are compared using the shared candidate pool, object limit of K = 12, identical renderer, and token budget from the original Table 3, changing only the selection policy: