acceptodds
Under review as a conference paper at ICLR 2027

VITAL-RAG: Object-Level Context Allocation for Coding Agents

Abstract

Repository-level coding agents depend on retrieved code, but relevant evidence may still fail to reach the coding model's input. Multiple views of one object can consume several context positions, while sequential packing can discard evidence from objects already selected. We study this retrieval-to-context gap and introduce VITAL-RAG, an object-level context-allocation system for coding agents. Controlled comparisons on 16,490 RepoBench tasks show that object grouping provides a substantial structural gain: full-source evidence containment increases from 24.99% to 39.16%. With candidates fixed task by task, the system's rendering configuration raises complete rendered-unit availability from 39.59% to 63.67% while using 35.63% fewer evidence tokens. Packing gains grow from 1.29 to 14.13 points as the candidate limit increases from five to twenty on a disjoint set of 14,442 tasks. Under a shared renderer and a twelve-object limit, cost-aware selection improves rendered-unit availability by 3.58 points over rank-only selection on 13,121 held-out tasks. A complete two-candidate RepoExec workflow using VITAL-RAG context construction reaches 65.63% Pass@1 with Claude versus 54.65% for the strongest local style control evaluated here; this complete-workflow result includes post-generation output selection. A 100-task controlled generation probe does not establish a statistically reliable code-accuracy gain from allocation alone. Together, these results separate evidence discovery, evidence delivery, and generated-code correctness in repository-level coding agents.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Language Models, AI for Code, Retrieval-Augmented Generation, Repository-Level Code Generation, Code Agents, Context Engineering, Code Retrieval, Software Engineering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_v2AwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the gap between retrieving relevant repository evidence and actually delivering that evidence to a coding model under a limited context budget. It introduces VITAL-RAG, which groups retrieved views into code objects, selects and orders objects using relevance, file coverage, and estimated token cost, and constructs bounded contexts. Experiments on RepoBench separate the effects of grouping, packing, rendering, and selection. The strongest controlled policy result is a 3.58 percentage-point improvement in rendered-unit availability over rank-only selection under a shared renderer on held-out repositories. A small generation study is inconclusive, while RepoExec evaluates a broader workflow that includes two generated candidates and output selection. The contribution is primarily an empirical analysis of context allocation, with downstream benefits from allocation alone remaining unestablished.

Critical strengths

1.The controlled experiments separate losses caused by repeated views from losses introduced during context construction. Table 1 shows a substantial grouping benefit under full-source containment, while the candidate-count analysis demonstrates that packing becomes more consequential as more objects compete for a fixed budget. This decomposition provides useful knowledge for evaluating repository retrieval systems beyond candidate recall. 2.The shared-renderer evaluation provides credible evidence for a local policy improvement. In Table 3, VITAL-RAG improves rendered-unit availability from 82.47% to 86.05% at nearly unchanged selected recall and with fewer evidence tokens. The repository-held-out evaluation and repository-cluster confidence interval strengthen this finding. The authors also carefully distinguish evidence metrics from code correctness and explicitly acknowledge the inconclusive generation probe and the additional output-selection stage in RepoExec. This transparency makes the supported conclusions easier to assess.

Critical weaknesses
  1. Appendix A.2 permits a rendered unit to be a query-centered fragment of the original gold snippet. For Table 2, which compares different rendering configurations, it is not sufficiently explicit whether both conditions are evaluated against the same units fixed before allocation. If unit boundaries or lengths depend on the configuration, higher availability could partly reflect shorter evaluation targets rather than better preservation of equivalent evidence. This is a clarification concern, not a demonstrated metric error. Table 3 is better controlled because it shares a renderer, but downstream evidence does not yet resolve the metric’s practical significance: the 100-task generation probe is inconclusive, and the RepoExec improvement includes additional generation and static output selection. Consequently, the reported availability gains do not yet establish the independent utility of the allocator. 2.The main twelve-object policy comparison uses rank-only selection, although the paper itself reports that shortest-first and rank-valued exact knapsack outperform completeness-first packing at twenty objects. Those results use a different protocol and therefore do not directly refute Table 3, but they identify important alternatives that should be compared under its shared candidate pool, renderer, and budget. The MMR comparison in Table 11 uses a different five-object setting and does not settle this question. Moreover, the largest grouping benefit is demonstrated using three explicitly constructed views per snippet, leaving its benefit over a conventional single-view object retriever unclear. Together, these limitations make it difficult to determine how much new capability or generalizable insight the proposed policy adds beyond object deduplication and simple cost-aware packing.
Questions for authors

1.In Table 2, are the gold rendering units identical between Base and Full and fixed before allocation, independently of the remaining token budget? Please specify the window construction rule. If the units differ, can the existing contexts be reevaluated against a common set of units or full-source containment? 2. Under the exact Table 3 protocol, how does VITAL-RAG compare with shortest-first, rank-valued knapsack, and object-level MMR? A focused comparison would help determine whether the reported gain requires the proposed combined utility rather than a simpler allocation rule. 3. Using the already generated RepoExec programs, can you report candidate A, candidate B, and selected-output success separately for each model, with paired comparisons between B and the actual default A? Please also clarify whether the output-selection rules were fixed on separate development data. These results would help distinguish improvements associated with the allocation configuration from those introduced by the selection stage. 4. Is a single-full-source-view retrieval control available under the same scorer and budget as Table 1? Such a comparison would clarify whether the large grouping gain primarily recovers losses introduced by the constructed multi-view expansion or provides an advantage over an ordinary object-level retrieval setup.

3 comments

Response to Reviewer v2Aw

We sincerely thank you for carefully reading our paper and raising these important concerns. To demonstrate the generalizability and potential value of our research, we have devoted substantial computational resources to additional code generation experiments across multiple dimensions. We hope these additional results will address your reservations and concerns.

W1/Q1: Do Base and Full use identical evaluation units fixed in advance?

Thank you for pointing out that our description of evaluation fairness was insufficiently clear. We would like to clarify that Base and Full are both scored against the same complete gold snippets fixed in advance, which do not change with the remaining budget. Both contexts are limited to 4,096 tokens, including descriptors and separators. For scoring, we only normalize line endings and remove leading and trailing whitespace. A hit does not require the complete original text to appear verbatim in the final context. Partial retention also counts. The containment rate is the proportion of tasks counted as hits among all tasks. We will explicitly document these rules in the revised manuscript.

W1/Q3: Can you report the results for A, B, and the selected outputs separately, and explain how the selection rules were developed?

  1. Thank you for your attention to the sources of the improvement. We have added a controlled analysis using fixed A/B candidates. A uses the CodeRAG context, while B uses the context constructed by our method. Each model is evaluated on the same set of 355 tasks. We separately report the execution pass rates of A, B, and the selected outputs to distinguish the performance of the two candidates from the results after applying the selection policy:
ModelA: CodeRAGB: Our methodAfter selection
GPT-5.452.11%56.62%57.75%
Claude Sonnet 4.653.24%61.97%65.63%
Qwen3-8B14.93%21.41%21.69%
  1. Regarding your concern in W1 about the 100-task generation experiment, we agree that its scale is limited and insufficient to fully assess downstream effectiveness. We have therefore added experiments on the full set of 13,121 RepoBench tasks, reporting exact match (EM) and edit similarity (ES) separately:
MethodQwen3-8B EM (%)Qwen3-8B ESClaude Sonnet 4.6 EM (%)Claude Sonnet 4.6 ESGPT-5.4 EM (%)GPT-5.4 ES
CodeRAG'2510.4352.8728.2063.1126.0561.32
GraphCoder'2410.5754.1226.6166.4825.3164.09
RepoScope'2610.8854.5328.7865.7927.6064.51
VITAL-RAG (direct)10.9654.7629.6367.5729.0966.03
VITAL-RAG11.0955.0830.1767.9629.4166.38
  1. Following your suggestion, we have also added a paired comparison between B and the default A on RepoExec. We count the gains and losses of B relative to A without applying the output selector:
ModelB passes, A fails (gains)A passes, B fails (losses)Net additional tasks passed (B−A)
GPT-5.4392316
Claude Sonnet 4.6481731
Qwen3-8B533023
  1. We also clarify that the selector’s rules were determined on a development set, with a separate test set held out for evaluation. The selection process does not access reference implementations, test results, or execution scores, and test-set results are not used to adjust the rules. In addition, to demonstrate generalizability, we have added evaluations on all 460 CoderEval tasks:
Method / VersionQwen3-8B (%)Claude Sonnet 4.6 (%)GPT-5.4 (%)
CodeRAG-style48.2653.2651.08
GraphCoder-style48.0455.4352.82
RepoScope-style44.7856.0853.69
VITAL-RAG (direct, always B)49.5658.9155.21
VITAL50.4361.7355.86

W2/Q2: How does the method compare with simple, strong baselines under the same protocol as Table 3?

Thank you for suggesting more targeted comparisons. Following your advice, we have added shortest-first, rank-valued knapsack, and object-level MMR. These methods are compared using the shared candidate pool, object limit of K = 12, identical renderer, and token budget from the original Table 3, changing only the selection policy:

MethodRendered-unit availability
Rank-only82.47%
Shortest-first83.04%
Rank-valued knapsack84.96%
Object-level MMR82.37%
VITAL-RAG86.05%

W2/Q4: Is there a source-only control using the same scorer and budget?

Thank you for suggesting a more detailed control. We have added an experiment that retains only the original source code without expanding it into additional views. The comparison before and after grouping the three views (24.99% → 39.16%) confirms that object-level grouping improves complete original-source containment. Furthermore, the grouped three-view configuration achieves higher complete original-source containment than the source-only configuration (39.16% vs. 32.92%), showing that the grouped configuration outperforms the single-view reference:

ConfigurationComplete original-source containment
Source-only retrieval (new)32.92%
Ungrouped three-view selection (original manuscript)24.99%
Three views + object-level grouping (original manuscript)39.16%

Thank you for carefully reviewing our paper and for your constructive feedback. As the discussion period draws to a close, we wanted to follow up briefly and ask whether the additional experiments and clarifications above have addressed your main concerns.

This paper and this conference mean a great deal to me. We have invested substantial time and computational resources in this research, and this submission is tied to an important milestone in my life. If you feel that our responses have addressed your concerns, we would be grateful if you could consider updating your rating. We would be happy to clarify any remaining points.

Our detailed responses are above. For ease of reference, we summarize the additions most relevant to your comments below.

  • W1/Q1 (Evaluation of Base and Full). For each task, Base and Full use the same span from the gold snippet as the scoring window. We fix the span before allocating the token budget and count a hit when the final context contains the entire span.

  • W2/Q2 (Strong Baseline Comparison). Following the original Table 3 protocol, we added shortest-first, rank-valued knapsack, and object-level MMR, using the same candidate pool, object limit, renderer, and token budget. VITAL-RAG achieved 86.05% rendered-unit availability, compared with 84.96% for rank-valued knapsack, the strongest baseline in this comparison.

  • W1/Q3 (Candidate Performance and Output Selection). We reported execution pass rates for A, B, and the selected output separately. In a paired comparison before selection, B passed 16, 31, and 23 more tasks than the actual default candidate A (CodeRAG) on GPT-5.4, Claude Sonnet 4.6, and Qwen3-8B, respectively. We determined the selection rules on a development set and fixed them before evaluation on a held-out test set. We also reported results on all 13,121 RepoBench tasks and all 460 CoderEval tasks to address the limited scale of the original 100-task experiment.

  • W2/Q4 (Source-Only Control). We evaluated the source-only control using the same scoring criterion and token budget as the three-view experiments. The source-only control achieved 32.92% complete original-source containment. In the three-view setting, object-level grouping increased containment from 24.99% to 39.16%.

Reviewer_4jzqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the gap between retrieved repository evidence and the context actually delivered to a coding model. It proposes VITAL-RAG, which groups multiple views of the same code object, selects objects using relevance, file coverage, and estimated token cost, and renders bounded query-centered snippets. Controlled RepoBench experiments separate the effects of object grouping, packing, and selection. Grouping improves full-source containment, while the twelve-object policy improves rendered-unit availability by 3.58 percentage points over rank-only selection. A complete RepoExec workflow also improves execution success for Claude, although it includes two generated candidates and output selection. The controlled generation probe does not establish a statistically reliable accuracy improvement from allocation alone.

Critical strengths

The paper is well motivated and easy to follow. Distinguishing retrieved candidates, selected objects, and evidence actually delivered to the model provides a useful perspective on repository context construction. The controlled experiments are a strength: fixed-candidate comparisons, a shared renderer, repository-held-out evaluation, and repository cluster confidence intervals help identify where gains occur. The authors also report competitive simple alternatives and explicitly distinguish evidence retention improvements from code generation accuracy, avoiding an unsupported attribution of the complete workflow’s gains to the allocator alone. Critical Weaknesses

Critical weaknesses
  1. The main concern is whether improved evidence availability translates into more useful context. The metric counts complete renderer-defined windows from a gold candidate, which need not preserve the information required for generation. The 100-task generation probe remains inconclusive, and newly available evidence does not produce an observable advantage on the reported 22-task subset. A larger controlled study is needed to establish the allocator’s practical value beyond this intermediate metric.

  2. The contribution beyond object deduplication and conventional budget-aware selection remains unclear. The largest gain comes from grouping explicitly constructed views, while the final twelve-object policy is not compared with object-aware MMR or knapsack alternatives under the same renderer. Since shortest-first and exact knapsack outperform completeness-first in the twenty-object experiment, stronger matched comparisons are needed to justify the proposed heuristic.

  3. The RepoExec results combine context allocation with two-candidate generation and output selection, without giving the controls the same generation budget and selector. Consequently, the gains cannot establish the allocator’s independent benefit. The use of local style controls and statistically unresolved improvements for GPT and Qwen further limits the downstream evidence. These limitations are acknowledged, but an equal-budget comparison remains necessary.

Questions for authors
  1. How do you establish that a completely emitted rendering unit retains task-relevant evidence? Could you validate this independently of the renderer or through a larger controlled generation study?

  2. Can you compare the twelve-object policy with object-aware MMR, shortest-first, and knapsack alternatives under the same renderer and budget? Please refer to Weakness 2.

3 comments

W1/Q1: Does completely emitted evidence actually help code generation?

We sincerely thank you for raising this important concern. We agree that window completeness alone does not establish practical usefulness, and we apologize that the original evaluation did not sufficiently address this point. Following your suggestion, we have added full-set evaluations on CoderEval and RepoBench, together with separate reporting of RepoExec results before output selection. Please allow us to clarify how the four result panels address your question.

1. RepoExec: executable correctness before selection.

The newly reported direct results improve over CodeRAG for all three models. For example, Claude increases from 53.24% to 61.97%. This shows that the observed improvement is already present before output selection.

MethodQwen3-8BClaude Sonnet 4.6GPT-5.4
CodeRAG'2514.9353.2452.11
GraphCoder'2421.1354.6552.39
RepoScope'2615.2153.8055.49
VITAL-RAG (direct)21.4161.9756.62
VITAL-RAG21.6965.6357.75

Execution pass rate (%), 355 tasks per model.

2. CoderEval: newly added execution results across Python and Java.

To provide further execution-based evidence, we added evaluation on all 460 CoderEval tasks. Direct generation exceeds the evaluated controls for each model. These results are scored independently of our rendering metric and constitute an exploratory evaluation of the stated ranking-and-rendering variant.

MethodQwen3-8BClaude Sonnet 4.6GPT-5.4
CodeRAG'2548.2653.2651.08
GraphCoder'2448.0455.4352.82
RepoScope'2644.7856.0853.69
VITAL-RAG (direct)49.5658.9155.21
VITAL-RAG50.4361.7355.86

Execution pass rate (%), all 460 tasks.

3. RepoBench: newly added large-scale completion results.

We appreciate your observation that the original 100-task probe was inconclusive. We therefore expanded generation evaluation to 13,121 RepoBench tasks. Direct generation achieves higher observed exact match (EM) and edit similarity (ES) than the evaluated controls across all three models.

MethodQwen3-8B EM (%)Qwen3-8B ESClaude Sonnet 4.6 EM (%)Claude Sonnet 4.6 ESGPT-5.4 EM (%)GPT-5.4 ES
CodeRAG'2510.4352.8728.2063.1126.0561.32
GraphCoder'2410.5754.1226.6166.4825.3164.09
RepoScope'2610.8854.5328.7865.7927.6064.51
VITAL-RAG (direct)10.9654.7629.6367.5729.0966.03
VITAL-RAG11.0955.0830.1767.9629.4166.38

4. RepoClassBench: complementary class-level evidence.

For completeness, we also retain the class-reconstruction evaluation on 227 reference-bearing tasks. We treat this as auxiliary workflow evidence, since it includes output reuse for Claude and Qwen and output selection for GPT.

MethodQwen3-8B Token-F1Qwen3-8B Char.Claude Sonnet 4.6 Token-F1Claude Sonnet 4.6 Char.GPT-5.4 Token-F1GPT-5.4 Char.
CodeRAG'2548.6534.7970.8456.3257.6442.20
GraphCoder'2440.5927.9965.0946.7351.3834.34
RepoScope'2651.6434.5772.0357.0059.4844.91
VITAL-RAG51.6434.5772.0357.0060.1245.61

We hope these additions provide a clearer empirical response to your concern.

W2/Q2: What does the method contribute beyond object deduplication and simple budget-aware selection?

We sincerely thank you for suggesting these more targeted comparisons. Following your advice, we have added shortest-first, rank-valued knapsack, and object-level MMR under the original Table 3 protocol, using the same candidate pool, object limit K = 12, renderer, and token budget.

MethodRendered-unit availability
Rank-only82.47%
Shortest-first83.04%
Rank-valued knapsack84.96%
Object-level MMR82.37%
VITAL-RAG86.05%

VITAL-RAG exceeds the strongest of these alternatives, knapsack, by 1.09 percentage points. This matched comparison supports an additional benefit from the joint selection and ordering strategy within the object-level allocation setting.

W3: Do the RepoExec gains come from additional generation and output selection?

1、We sincerely appreciate your concern about generation budgets and attribution. To clarify the sources of improvement, we separately report candidate A, candidate B, and the selected output on the same 355 tasks per model. A uses CodeRAG context, while B uses the context constructed by our method. A and B each represent a single generation, whereas the selected workflow uses both candidates.

ModelA: CodeRAGB: Our method (direct)After selection
GPT-5.452.11%56.62%57.75%
Claude Sonnet 4.653.24%61.97%65.63%
Qwen3-8B14.93%21.41%21.69%

B exceeds A by 4.51, 8.73, and 6.48 percentage points, respectively. Thus, the observed configuration-level improvements are already present before output selection. The selected-output results describe the two-candidate workflow and are not treated as equal-generation-budget comparisons against A.

2、We have additionally evaluated all 460 CoderEval tasks and 13,121 RepoBench tasks. For the reported CoderEval variant, direct-generation pass rates exceed the strongest evaluated control for each model by 1.30–2.83 percentage points. On RepoBench, direct generation achieves higher EM and ES than all evaluated controls across the three models. These comparisons include the CodeRAG-style, GraphCoder-style, and RepoScope-style implementations. The complete results are provided in our response to W1/Q1.

We hope these additions more clearly distinguish improvements observed in direct generation from the results of the broader selection workflow.

Thank you for carefully reviewing our paper and for your constructive feedback. As the discussion period draws to a close, we wanted to follow up briefly and ask whether the additional experiments and clarifications above have addressed your main concerns.

This paper and this conference mean a great deal to me. We have invested substantial time and computational resources in this research, and this submission is tied to an important milestone in my life. If you feel that our responses have addressed your concerns, we would be grateful if you could consider updating your rating. We would be happy to clarify any remaining points.

Our detailed responses are above. For ease of reference, we summarize the additions most relevant to your comments below.

  • W1/Q1 (Downstream Utility). We evaluated all 13,121 RepoBench tasks and all 460 CoderEval tasks. Across all three models, direct generation achieved higher EM and ES on RepoBench and higher execution pass rates on CoderEval than the evaluated baselines. The CoderEval results come from an exploratory evaluation of our ranking-and-rendering variant.

  • W2/Q2 (Strong Baseline Comparison). Following the original Table 3 protocol, we added shortest-first, rank-valued knapsack, and object-level MMR, using the same candidate pool, object limit, renderer, and token budget. VITAL-RAG achieved 86.05% rendered-unit availability, compared with 84.96% for rank-valued knapsack, the strongest baseline in this comparison.

  • W3 (Generation Budget). For each model, we reported execution pass rates for A, B, and the selected output on the same 355 RepoExec tasks. A and B each use a single generation, and B outperformed A under this equal budget across all three models. The selected-output results describe the full two-candidate workflow.

Reviewer_HgB3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

The authors claimed that in repository-level coding agents, even when agents retrieve relevant code snippets, some of them may still be omitted from the final prompt. The authors claimed this is caused by redundant views of the same code object and truncation under limited context budgets. To address this, this paper proposes to group retrieved views into distinct code objects, select and order them using relevance, file diversity, and estimated token cost, and construct the final context under a fixed token budget.
The authors evaluate the approach on RepoBench and RepoExec, and observe multiple improvements. For example, object grouping improves full-source evidence containment, increasing from 24.99% to 39.16%. These results demonstrate that the proposed approach can preserve more retrieved code in the final prompt. Additionally, the paper also evaluates downstream code generation (a controlled experiment on 100 tasks) and does not find a statistically significant improvement in code accuracy.

Critical strengths
  • I believe this is a practical problem in code agents with RAG systems, where retrieved evidence may be ignored during context construction.

  • The authors conducted controlled experiments, where they separate the effects of different components (object grouping, context packing, and cost-aware selection ...)

Critical weaknesses
  • The paper is very difficult to follow. The fragmented organization, combined with frequent changes to metrics, datasets, and experimental configurations, obscures the relatively simple proposed method and makes the key findings hard to interpret.
  • I am also very confused and concerned that the paper only demonstrates improvements in evidence retention, but not in the downstream code-generation experiment. So, the use case of the method remain unclear to me.
  • The main grouping experiment explicitly constructs three views per source snippet, I am also concerned about its generalizability.
Questions for authors
  • How much of the reported grouping improvement would remain when applied to existing retrieval systems without this artificial view expansion?
  • Can the authors elaborate more on the results on RepoExec? If I understand correctly, it includes additional post-generation output selection?
  • More discussion about the necessity/potential?
4 comments

Response to Reviewer HgB3

We sincerely thank you for carefully reading our paper and providing valuable feedback. We also appreciate your recognition of the relevance of this problem to coding agents. We have updated the PDF with many additional experiments, which we hope will address your concerns and earn your support. All revised and newly added content is highlighted in blue.

W1: The paper is difficult to follow.

  1. We sincerely apologize for the difficulty our presentation caused. We acknowledge that this submission did not follow the conventional structure of most ICLR papers. Instead, we organized it around “observing a phenomenon → analyzing it step by step → developing a method → evaluating it” to address a problem generally overlooked by papers on context construction for coding agents. We intended to explain the rationale behind our design in detail, but this presentation did make the paper harder to follow.

  2. Following your feedback, we have restructured the paper. We would be very grateful if you could give the revised version another brief review. The revised manuscript presents the system and main results earlier, moves the diagnostic experiments, ablations, and sensitivity analyses later, and consolidates the descriptions of datasets, metrics, and configurations to make the central argument clearer. We have also added Figure 2 to make our approach easier to understand.

W2: The downstream code generation benefits remain unclear.

  1. Thank you for your comments on the effectiveness of our RAG approach for code generation. Evidence retention should indeed be evaluated alongside the quality of the generated code. Our original manuscript already included execution results on RepoExec demonstrating the leading performance of our approach, but the paper’s structure may have made these results easy to miss. We apologize again for this. Please see Table 1(a) in the revised manuscript for the results.

  2. In addition, we devoted substantial computational resources to evaluations on all 460 CoderEval tasks and the full set of 13,121 RepoBench tasks. These additions are highlighted in blue in Tables 1 and 2 of the revised manuscript. The results are provided below.

(b) CoderEval: execution pass rate (%), all 460 Python and Java tasks

MethodQwen3-8BClaude Sonnet 4.6GPT-5.4
CodeRAG'2548.2653.2651.08
GraphCoder'2448.0455.4352.82
RepoScope'2644.7856.0853.69
VITAL-RAG (direct)49.5658.9155.21
VITAL-RAG50.4361.7355.86

(a) RepoBench: 13,121 tasks

MethodQwen3-8B EM (%)Qwen3-8B ESClaude Sonnet 4.6 EM (%)Claude Sonnet 4.6 ESGPT-5.4 EM (%)GPT-5.4 ES
CodeRAG'2510.4352.8728.2063.1126.0561.32
GraphCoder'2410.5754.1226.6166.4825.3164.09
RepoScope'2610.8854.5328.7865.7927.6064.51
VITAL-RAG (direct)10.9654.7629.6367.5729.0966.03
VITAL-RAG11.0955.0830.1767.9629.4166.38

Figure 3. Controlled grouping and packing diagnosis

W3/Q1: The generalizability of the three-view experiment is uncertain. How much of the grouping improvement would remain without artificial view expansion?

Thank you for raising this important question.

  1. First, multiple distinct representations of the same object can arise in retrieval pipelines without the three-view expansion described above. We audited the existing retrieval contexts for all 460 CoderEval tasks. Of these, 209/460 tasks (45.43%) contain at least two distinct text windows from the same function, and 90/460 tasks (19.57%) contain at least three. Both counts remain unchanged after exact-text deduplication. Furthermore, 193/460 tasks (41.96%) contain a pair of windows from the same object that partially overlap while each also contains code lines absent from the other. Several other datasets exhibit the same issue, which is common in large-scale software development.

  2. Second, we need to distinguish the controlled construction of three views from multiple representations in real retrieval pipelines. The full source code, signature description, and normalized lexical representation used in our experiments are indeed three retrieval views explicitly constructed from the same code object. This construction allows us to control the number of representations originating from the same object and examine whether their independent competition for candidate slots crowds out other objects, and whether object-level grouping can alleviate this problem.

  3. To provide a reference without view expansion, the table below includes Source-only retrieval. Its complete original-source containment rate is 32.92%, compared with 24.99% for ungrouped selection over three views and 39.16% for three-view retrieval with object-level grouping. This shows that the multi-view grouping configuration outperforms the source-only reference.

ConfigurationComplete original-source containment (%)
Source-only retrieval32.92
Ungrouped view selection24.99
Object-level grouping39.16
Grouping with a companion39.36
Grouping with completeness-first packing40.44
Companion with completeness-first packing40.69
  1. Thank you for reading the paper carefully. Yes, our previous description was somewhat ambiguous. To clarify, for each task, the same model generates two candidates: A uses the CodeRAG context, and B uses the context constructed by our method. A rule-based selector then chooses one program for execution testing. We also clarify that the selection rules were determined on an independent development set, and the test set was used only for evaluation after the rules had been fixed. The selection process does not access reference implementations, test results, or execution scores.

  2. To distinguish the performance of the candidate programs from the contribution of the selector itself, we have added a controlled comparison using fixed A/B candidates:

ModelA: CodeRAGB: Our methodAfter selection
GPT-5.452.11%56.62%57.75%
Claude Sonnet 4.653.24%61.97%65.63%
Qwen3-8B14.93%21.41%21.69%

Q3: The necessity and potential value of the method.

Please allow us to explain how the idea behind VITAL-RAG originated. When using AI to modify a large backend project, you may well have encountered the same situation as we have: you only want to change one API endpoint, yet the agent needs to inspect routing, authentication, business services, and database code along the way. The same code is repeatedly searched and read, and the context keeps growing. Yet when the actual modification is made, critical transaction boundaries or exception-handling logic may not be fully retained. The core insight is that reading a great deal of code does not mean that all the evidence needed for the task has been preserved. We therefore broke down the entire process step by step for controlled analysis and ultimately developed VITAL-RAG.

We believe that coding agents will increasingly rely on active, multi-round agent search. This is precisely why revisiting context allocation matters: the stronger the search capability, the greater the need to organize the evidence it retrieves. Through object-level grouping, cost-aware selection, and budget-constrained rendering, VITAL-RAG reduces the space occupied by repeated representations and makes room for task-relevant code, providing a complementary evidence allocation mechanism for agent search.

Thanks for the additional experiments. The expanded downstream evaluations substantially address my concern about the practical benefit of the method, so I have raised my soundness score from 1 to 2.

My concern about the three-view experiment is only partially addressed. The new analysis shows that multiple views occur naturally, but the reported grouping gain is still based on explicitly constructed three-view inputs. The source-only comparison is useful, but does not isolate how much grouping itself helps on a naturally occurring, unexpanded retrieval pool. Therefore, I believe this generalizability concern still remains.

Thank you for your careful follow-up and for clarifying the remaining concern. We appreciate the distinction between demonstrating naturally occurring redundancy and measuring the benefit of grouping on an unexpanded retrieval pool. To address this directly, we added source-only retrieval + object-level grouping.

ConfigurationComplete original-source containment (%)
Source-only retrieval32.92
Source-only retrieval + object-level grouping34.17
Ungrouped view selection24.99
Object-level grouping39.16
Grouping with a companion39.36
Grouping with completeness-first packing40.44
Companion with completeness-first packing40.69

Grouping yields a 1.25-point net improvement without view expansion. Grouping can remove redundant occupancy while also discarding complementary information, so its net benefit depends on preserving useful evidence. The larger improvement in the three-view setting illustrates its behavior under controlled representation multiplicity, where source relationships are explicit.

As the rebuttal window is closing, we kindly ask whether these results resolve your remaining generalizability concern. This paper and this conference submission are extremely important to me. We have invested considerable time and computational resources into this work, and this submission is critical to a major milestone in my next phase(meltdown). If you believe the concerns have been properly addressed, we sincerely ask you to consider updating your score. Please let us know if there is anything else we should clarify.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.