acceptodds
Under review as a conference paper at ICLR 2027

The Missing Complement: State-Conditioned Minimal Sufficient Evidence for Coding Agents

72 views

Abstract

Retrieval assembles repository context by ranking passages for relevance to the current query. A coding agent halfway through an issue has already read much of what such a ranker returns. Relevance is scored per passage, but sufficiency belongs to the set: independently scored passages can fill the budget with support for one requirement while another goes unmet. We formulate state-conditioned minimal sufficient evidence recovery: given a captured agent state, recover a compact evidence combination supplying what its next decision still lacks. SERBench measures this on 500 held-out states from 45 repositories, recording what the agent has seen, crediting only sets that satisfy every annotated evidence requirement of the current decision, and separating set recovery from candidate discovery. MSS-Complement treats acquisition as set construction, not ranking. Three semantic calls propose a jointly sufficient set, search for what it lacks, and return 4–8 intact source units within 6,144 tokens. One configuration, fixed on calibration data, recovers a complete set for 73.0% of those states at five items and 80.6% at eight, against 61.4% and 72.4% for Qwen3 embedding with reranking. A matched control ranking by similarity alone recovers fewer complete sets, placing the margin over it in the set-level policy, not the computation. The lead persists from frozen repository source with no gold-derived pool. On AMA-Bench it answers from a 76.2% smaller answer prompt, with accuracy 2.08 points above that benchmark's own memory agent. Removing one required group from a complete set costs repair-localization precision under two executors. Retrieval for agents is better posed as recovering what a decision lacks than re-ranking what an issue resembles.

Primary area:
datasets and benchmarks
Keywords:
coding agents, information retrieval, agent memory, evidence selection, context engineering, datasets and benchmarks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sY9EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper explore state conditioned evidence retrieval problem for coding agents. It argues that traditional relevance based retrieval may return evidence already observed by an agent or fail to retrieve a jointly sufficient set of evidence for its current decision. To evaluate this problem, the paper introduces SERBench, which contains intermediate coding agent states and grouped evidence sufficiency annotations. The paper further proposes MSS-Complement, a three stage semantic retrieval framework that constructs, expands, and finalises a compact evidence set based on the agent's current information needs. Experiments show improvements in complete set recovery over retrieval baselines, with additional analysis on repair localisation, long horizon agent QA, etc.

Critical strengths
  1. I think the paper proposes a useful evaluation perspective for agent retrieval, to some degree.

The paper highlights two relevant limitations of traditional relevance based retrieval: 1) redundancy with previously observed information and 2) insufficient coverage of complementary evidence. SERBench attempts to evaluate these issues through captured agent states and grouped evidence, providing a more fine-grained perspective than traditional retrieval metrics.

  1. The experiment and analysis are extensive to some degree.

The paper evaluates multiple retrieval baselines and includes component ablations and downstream coding agent experiments. The human validation of evidence annotations and the attempt to connect retrieval completeness with downstream decisions are valuable aspects of the paper.

Critical weaknesses
  1. I think the paper suffers from a little serious presentation and logical issues that hinder me understand its central contributions. Especially the abstract and introduction, even though I read them multiple times, I am still confused in many places such as the motivation, research gap, research question, and the relationships between the motivation and SERBench and the framework, etc.

The abstract and introduction seem difficult to follow. The paper frequently moves between different concepts, research objectives and experimental findings without clearly establishing their logical connections. For example, the abstract moves from the redundancy of relevance based retrieval to state conditioned evidence recovery, then immediately introduces SERBench and MSS-Complement, without explaining why the identified limitations require the proposed benchmark and method. Similarly, the opening paragraphs (line 40-42) of the introduction move from changing information needs during coding agent execution to long context limitations (lost in the middle etc), and then to retrieval objectives, without clearly establishing how these arguments lead to the proposed research gap. There are still multiple other places in the introduction and abstract that have this problem.

Furthermore, the abstract/introduction introduces several results including complete set recovery, matched similarity controls, etc, without clearly explaining what each evaluation shows or how they support the central contribution.

  1. I am a little confused about methodological novelty and I think the experiment evidence might not sufficiently establish the contribution of MSS-Complement.

The proposed approach relies on evidence-set construction, iterative retrieval based on missing evidence, etc. However, related work such as Evidence Tree Search (ACL 2025) and S2G-RAG (ACL 2026) seems to have already explored closely related mechanisms. Although these works are cited, the paper does not sufficiently clarify what fundamentally distinguishes MSS-Complement beyond applying these principles to captured coding-agent states. What is the advantage of MSS-Complement?

Moreover, the component ablations raise questions about the necessity of the central mechanism. Table 4 shows that the no-feedback adaptive variant achieves 72.80% Complete MSS@5, compared with 73.00% for the full method. This suggests that the independent contribution of selected-set feedback may be limited in the current experiments. But it seems to be emphasised as a central part of missing-complement acquisition.

Questions for authors
  1. Research motivation and method difference.

Could the authors clarify the central research gap and explain how state conditioned minimal sufficient evidence recovery differs from existing retrieval methods such as Evidence Tree Search and S2G-RAG? Especially, what challenge arising from coding-agent states is not adequately addressed by these existing methods?

  1. Contribution of the proposed set level mechnism.

Table 4 shows that removing selected set feedback from the adaptive variant changes Complete-MSS@5 from 73.00% to 72.80%. What is the specific contribution of selected set feedback and explicit missing-complement identification? Why is the three stage semantic controller necessary when a simpler variant achieves nearly identical performance?

  1. Considering the substantially higher online token consumption on AMA-Bench and the limited statistical evidence for downstream QA and code repair improvements, could the authors further clarify the cost-performance trade-off of the additional controller computation?

Actually, I find the research problem of the paper interesting to some degree, but the unclear organisation makes it unnecessarily difficult to identify the precise research question, understand the methodological motivation, and assess the relationship between the claimed contributions and the experimental evidence. I encourage the paper to apply restructuring and clearer explanations in the next version.

7 comments
AuthorsOpenReview ↗

Response to Reviewer sY9E

We thank the reviewer for the detailed comments. Following them, we restructured the paper and rewrote the abstract and the introduction in full. Our answers are below, each with the place in the revised PDF.

W1 and W2. Presentation and logic of the abstract and introduction. Both now follow one chain. The setting comes first, then why relevance ranking fails (it repeats code the agent has read and can miss a piece the decision still needs), then the research question and the gap: existing repository-retrieval benchmarks do not record which code combinations suffice for a specific decision. The task, SERBench, and MSS-Complement follow, each introduced as the answer to the step before it. Every reported result is now followed by what it shows. Section 6 opens with a roadmap of what each subsection tests. Where to look: the Abstract and Section 1 (paragraph 3 holds the research question and the gap).

W3 and Q1. Difference from Evidence Tree Search and S2G-RAG, and the advantage of MSS-Complement. We do not claim that building an evidence set or searching for what is missing is new. The difference lies in the problem. ETS and S2G-RAG serve a reader answering one fixed question, start with no evidence, and judge sufficiency with models trained on answer correctness or teacher labels. A coding agent's need changes at every step, part of the needed code has already been read, and an intermediate decision offers no answer to learn from. These are the challenges that the two methods do not address. Our main novelty is the task and the benchmark, which records what each decision still needs and scores a returned set against it. MSS-Complement needs no training. It is 11.60 points [+6.89, +17.21] above the strongest of eleven retrieval baselines, and 6.40 points [+3.42, +10.46] above a control that makes the same three LLM calls but selects by similarity. We did not run ETS or S2G-RAG. They were built for question answering, and using them here would mean replacing their central parts: a newly trained sufficiency judge, selection over code units in place of sentences or passages, and a way to leave out code the agent has read. The result would be a new method of our own design that shares only their core idea, and it would not be a faithful baseline. The paper claims no advantage over them. Where to look: Section 1, paragraph 3, which now states these differences, and Section 2.

W4 and Q2. Selected-set feedback and the three stages. We agree. Feedback adds 0.20 points [−4.00, +3.14], which is no measurable gain, and the revised paper no longer presents it as a core mechanism. A new ablation separates the two later calls (the "No expansion" row of Table 4). Compared with a single call, they add 3.00 points [+0.99, +4.70] at five units and return 476 fewer tokens of code to the agent per state. They run on the smaller model and account for about a quarter of the cost of the three calls. Finalization supplies most of this gain, and expansion adds 4.20 points [+0.78, +6.30] on the full returned set. The variant the reviewer cites still makes three calls, so it is no simpler than the full method. The evidence supports using more than one call. It does not show that the calls must see each other's selections. Where to look: Section 6.6 and Appendix E.2.

Q3. Cost of the additional computation. Two different token counts are involved. The cost is on the retrieval side: MSS-Complement makes three LLM calls for retrieval, 41.50K tokens per decision. These calls are separate from the coding agent that carries out the task, and none of their tokens enters its context. At DeepSeek's off-peak list prices on 10 November 2026, they cost about 1.5 US cents per decision, and the retrieval baselines make no LLM call. What this cost buys is in the agent's own context: within five units, the share of states with a complete set rises from 61.40% to 73.00%, and under the same 6,144-token budget the code the agent receives is 40% smaller at matched completeness (3,655 against 6,100 tokens). The control with the same three calls shows that the computation alone does not explain the gain. Downstream, the clearest evidence is at a single repair decision (Action52), where removing one required piece from complete evidence lowers precision by 12.28 points [+4.27, +20.34] under DeepSeek-v4-flash and by 11.11 [+5.20, +17.51] under Claude Sonnet 5. We do not claim lower total computation, and we agree that the end-to-end evidence is limited. On Fresh23 every paired interval includes zero, and on AMA-Bench the online total, which adds the retrieval calls to the answer model's tokens, is higher. Where to look: Sections 6.3 to 6.5, and Appendix E.1 for the cost of each call.

We are glad to clarify any point that remains unclear.

AuthorsOpenReview ↗

Gentle reminder: revised paper and response

Thank you again for reviewing our paper. This is a gentle reminder that the paper has been restructured as you suggested:

  • W1, W2 (presentation): the abstract and the introduction are rewritten around one research question.
  • W3, Q1 (novelty): paragraph 3 of the introduction states the difference from Evidence Tree Search and S2G-RAG.
  • W4, Q2 (mechanism): selected-set feedback is no longer presented as a core mechanism.
  • Q3 (cost): Section 6.3 reports the cost of the retrieval calls.

If any concern remains, we would be glad to discuss it.

Authors of Submission 26769

Thank you for the detailed response and revisions. I appreciate the authors' efforts, but I still have concerns about the clarity and coherence of the research motivation, research gap, and overall contribution. Therefore, I will maintain my original rating.

Reviewer_KDZJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies which repository evidence a coding agent still needs at a particular decision point. The authors separate source code the agent has already seen from its unresolved evidence needs, define a complete evidence set through certificates that allow alternatives within each requirement while requiring joint coverage across requirements, and introduce SERBench with complete-set recovery as its primary metric. Their method makes three semantic model calls to propose an evidence set, identify what it lacks, and finalize the evidence. It returns intact source units under item and source-token budgets. On the candidate pools supplied in the test set, its Complete-MSS@5 score is 73.0%, compared with 61.4% for the strongest reranking baseline.

Critical strengths

The evaluation target is well defined, and the annotations receive substantial review. Grouped sufficiency certificates move the evaluation from the relevance of individual passages to complete coverage of the evidence needed for a decision: evidence can substitute within a requirement, distinct requirements must be satisfied jointly, and already observed evidence does not provide new coverage. This design better reflects an agent's information need and prevents redundant evidence from inflating retrieval performance. The multiple layers of review also provide meaningful support for the reliability of the evaluation.

The main comparison shows consistent gains in complete-set recovery, with additional validation of clearly defined scope. Compared with conventional relevance ranking, the proposed method more effectively covers complementary evidence needed for a decision, particularly on tasks with multiple evidence groups. The results provide reasonably strong support for improved evidence-set recovery, although practical gains still depend on candidate discovery quality and the downstream executor.

Critical weaknesses

The advantage on the primary metric has not yet translated into stable gains on real tasks. The candidate pools in the main experiment are supplemented with necessary evidence located through future trajectories and reference patches. Thus, the experiment primarily tests whether the method can construct a complete evidence set when the relevant evidence is largely available. This setting is fair to all methods, but it limits how representative the result is of practical retrieval. In the full-repository experiment, which does not rely on answer-derived sources to construct the candidate pool, complete-set recovery is only 38.8%, indicating that candidate discovery remains an important bottleneck. The downstream evidence is positive but limited: long-trajectory question answering and executed code repair improve, yet the confidence intervals for both performance differences include zero, and the former incurs substantially higher total online token usage. The experiments therefore provide strong evidence for improved evidence-set recovery, but not yet for a stable and cost-effective improvement in real coding-agent task completion.

The evidence for attributing the gains to individual mechanisms remains limited. The full method improves complete-set recovery by 6.4 percentage points over an independent-similarity control that also makes three model calls, with a confidence interval that excludes zero. This supports the effectiveness of the overall policy. However, the comparison changes the sufficiency objective, selected-set feedback, and output size at the same time, making their individual contributions difficult to identify. In the finer ablation, adding selected-set feedback improves the score by only 0.2 percentage points, with a confidence interval that includes zero. The overall gain is well supported, but the selected-set feedback mechanism emphasized in the paper lacks evidence of a stable independent benefit.

Questions for authors

Comparability of the candidate-discovery evaluations. The full-repository diagnostic provides further evidence under practical candidate-discovery conditions, but it uses a separate certificate snapshot, and a complete certificate is reachable in the discovery pool for only 333 of 500 states. These conditions differ from those of the main experiment. For the 434 states shared by the two evaluations, could the authors rescore the existing frozen predictions against the current Test500 certificates and report candidate-pool coverage and Complete-MSS@5 under a common standard? If differences in evidence IDs or inputs make rescoring infeasible, please explain the specific differences between the certificate snapshots and their implications for comparing the results. This analysis would help clarify the method's effectiveness and scope in full-repository retrieval.

Independent contribution of selected-set feedback. In the finer ablation, selected-set feedback adds only 0.2 percentage points, with a confidence interval that includes zero. Could the authors clarify the contributions of the individual mechanisms? If the frozen predictions permit it, could they provide paired comparisons with the same candidate input, output size, and other conditions, separately assessing the sufficiency objective, adaptive output size, and selected-set feedback? Otherwise, I suggest emphasizing the effectiveness of the overall set-selection policy in the conclusion and tempering claims about the independent role of feedback.

Generalization across agent-generated state representations. All Test500 trajectories are generated by the same agent framework and DeepSeek-v4-Flash, which updates the current need, hypothesis, and subgoal after each observation, whereas the development set includes states generated by different planners. This consistent, structured representation helps evaluation, but it may affect applicability to other coding agents. Would the advantage over the baselines persist with another trajectory generator or without the normalized need, hypothesis, and subgoal fields? If relevant frozen results exist, please report them. Otherwise, please state explicitly that dependence on this trajectory generator and state representation has not been systematically tested.

3 comments
AuthorsOpenReview ↗

Response to Reviewer KDZJ

We thank the reviewer for the precise assessment. We followed each suggestion, and the revised paper now contains the analyses that were requested. Our answers are below, each with the place in the revised PDF.

W1. The advantage has not yet translated into stable gains on real tasks. We agree, and the revised paper states its claims in the same terms. It claims improved evidence-set recovery and a measurable effect of complete evidence at a single decision. It does not claim a stable or cost-effective end-to-end improvement. On Action52, removing one required group from complete evidence lowers repair-localization precision by 12.28 points [+4.27, +20.34] under DeepSeek-v4-flash and by 11.11 [+5.20, +17.51] under Claude Sonnet 5, and Section 6.4 now reports these intervals. On Fresh23 every paired interval includes zero. On AMA-Bench the accuracy interval includes zero, and the online total, which adds the retrieval calls to the answer model's tokens, is higher, as the reviewer notes. A new Section 6.3 separates two token counts on Test500. MSS-Complement makes three LLM calls for retrieval, 41.50K tokens per decision, about 1.5 US cents at DeepSeek's off-peak list prices of 10 November 2026. These calls are separate from the coding agent, and none of their tokens enters its context. The code that does enter the agent's context is 40% smaller at matched completeness under the same 6,144-token budget. The abstract, Section 1, Section 4.3, and Section 5 now state that the main benchmark measures selection from supplied pools. The pools are supplied by design: every pool contains a complete set, so a failure is a failure of selection, and experts can check a complete certificate for each state.

W2 and Q2. Attribution to individual mechanisms. We agree. The evidence supports the overall policy and does not support an independent benefit of selected-set feedback. As suggested, the Conclusion now emphasizes the overall set-selection policy, and the paper no longer presents feedback as a core mechanism. The paired comparisons the reviewer asks for could be computed from the frozen predictions. The rows compared share the candidate input, the three-call budget, and the scorer, and each step changes one factor (Complete-MSS@5, paired 95% intervals clustered by repository).

Factor changedOutput sizeDifference
Sufficiency objectivefixed 7 in both+3.20 [−0.25, +8.26]
Adaptive output sizefixed 7 to adaptive 4–8+3.00 [+1.06, +6.50]
Selected-set feedbackadaptive in both+0.20 [−4.00, +3.14]
All threefixed 7 to adaptive 4–8+6.40 [+3.42, +10.46]

Only adaptive sizing and the full step have intervals that exclude zero. A new stage ablation also separates the two later calls. Finalization without expansion adds 2.60 points [0.00, +6.27] over a single call at five units, and expansion adds 4.20 points [+0.78, +6.30] on the full returned set. Where to look: Section 6.6, Table 4, and Appendix E.2.

Q1. Comparability on the 434 shared states. Rescoring was feasible, and we did it as requested (new paragraph in Appendix D.6). We scored the frozen full-repository predictions against the current Test500 certificates. A predicted unit is mapped only when its state, source path, line span, and hash all match.

Setting, five unitsMSS-ComplementQwen3 + RerankerDifference
Full repository, all 434 states44.47%40.55%+3.92 [+1.56, +8.03]
Full repository, 308 coverable states62.66%57.14%+5.52 [+2.44, +11.94]
Released pools, same 434 states71.20%58.76%

The BM25 top-1,000 pool contains a complete certificate for 308 of the 434 states, against 292 under the original snapshot. The visible states are identical in the two cohorts, and the certificate structure differs for 126 of them. Under one standard, the gain at five units remains significant, the gain at eight units does not (+1.15 [−2.16, +7.14]), and finding candidates remains the main bottleneck.

Q3. Other trajectory generators and state representations. We have no frozen result that tests this, so the paper now states it explicitly, as the reviewer suggested. Appendix H reads: "All test states, however, come from one trajectory generator, so other generators and states without the need, hypothesis, and subgoal fields remain untested." The development set contains states from three planners, but it is not held-out evidence, and Prospective44 changes the controller family while keeping the same trajectory generator.

We are glad to clarify any point that remains open.

Thank you for your careful response to my comments. SERBench's evaluation target and annotation work are valuable, but candidate discovery remains the main bottleneck in practical use, and the current end-to-end results do not yet show stable gains. The additional analyses therefore improve the paper's clarity and the interpretation of its evidence, but they do not change my overall assessment of 3 (Weak Accept).

AuthorsOpenReview ↗

Reply to Reviewer KDZJ

Thank you for the response. We agree with the reviewer's overall assessment and appreciate the recognition of SERBench's evaluation target and annotation work. We only add brief context on two details.

End-to-end gains. We agree that our end-to-end runs do not show a clear gain. An end-to-end result depends on many factors, such as the agent's model, the repository environment, and every earlier step of the run. Our work changes one of them, the evidence the agent receives at a decision, so the others dilute its effect. For this reason the paper also tests that step alone: at a single repair decision, removing one required piece from complete evidence lowers precision by 11 to 12 points under both LLMs (Section 6.4). In our view, clear end-to-end gains build up from such component improvements, so a component with a measured effect is a meaningful step even before the end-to-end result changes.

Candidate discovery. We agree that discovery is the main bottleneck in practical use. Retrieval for an agent has two steps: finding candidate code, and selecting a sufficient set from it. Existing retrievers and benchmarks mainly target the first step. The second lacked a benchmark that records what a decision still needs, so our work fixes the candidates in order to study it. The two steps are complementary: any better discovery method can supply the candidates, and selection then decides whether the agent receives a complete set. SERBench's whole-repository track keeps discovery measurable, and it is what shows that discovery is the bottleneck. In our view, the two steps have to improve together, and a benchmark for selection was the missing part.

Thank you again for the constructive review.

Authors of Submission 26769

Reviewer_PNWBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper formulates state-conditioned minimal sufficient evidence recovery for coding agents, aiming to retrieve a compact set of complementary evidence that satisfies the unresolved requirements of an agent's current decision. It introduces SERBench, a benchmark based on captured agent states and grouped evidence-sufficiency certificates, and proposes MSS-COMPLEMENT, a three-stage LLM-based framework for evidence proposal, gap-directed expansion, and final selection.

Critical strengths

1.The paper addresses an important problem in coding-agent retrieval. The decision-specific perspective is meaningful and has the potential to support more effective context construction for coding agents. 2.The paper evaluates the proposed method against a broad range of retrieval baselines, full-repository retrieval, and downstream coding-agent evaluations. The improvements on SERBench are encouraging, and the additional experiments provide useful evidence beyond the proposed benchmark itself.

Critical weaknesses

1.The abstract and introduction do not clearly explain the proposed problem and contributions. In particular, SERBench and MSS-COMPLEMENT are introduced without explicitly identifying them as a newly proposed benchmark and method. The introduction also introduces several abstract concepts, such as state-conditioned evidence recovery, joint sufficiency, and grouped certificates, before providing a concrete explanation of the problem. This makes it difficult to understand what exactly the paper aims to solve and how its contributions differ from existing work. The issue goes beyond writing clarity, as it also makes the novelty and significance of the contributions difficult to assess. The authors should reorganize the motivation and clearly distinguish the problem formulation, benchmark contribution, and proposed method.

2.The ablation results in Table 4 do not sufficiently support the effectiveness of the key mechanism emphasized in the paper. The full MSS-COMPLEMENT method achieves 73.0% Complete-MSS@5, compared with 72.8% for the no-feedback variant with adaptive output sizing, an improvement of only 0.20 percentage points. The reported 95% confidence interval for this difference, [-4.00, +3.14], includes zero. Therefore, the experiments do not establish a reliable benefit from selected-set feedback, despite its central role in the proposed method. Moreover, the current ablations do not separately establish the contributions of gap-directed expansion and set-aware finalization. More targeted ablations are needed to demonstrate which components actually account for the improvements and whether the proposed multi-stage mechanism is necessary.

3.The main SERBench evaluation selects evidence from predefined candidate pools that are constructed using not only standard retrieval but also evidence identified through subsequent agent trajectories, gold patches, and test patches. Although this is a valid setting for evaluating evidence-set selection, it is substantially different from retrieving the required evidence directly from a full repository. This distinction is important because it determines the scope of the problem actually solved by the proposed method. The paper reports a separate gold-blind full-repository experiment, where Complete-MSS@5 decreases from 73.0% in the main evaluation to 38.8%, and the advantage over the reranking baseline decreases from 11.6 to 5.0 percentage points. The introduction and main method description should explicitly state that the primary contribution is evidence-set recovery from a supplied candidate pool, rather than general repository-wide evidence discovery. This limitation should not be left mainly to the appendix, as it directly affects the interpretation of the claimed contribution and experimental results.

Questions for authors

Please see the weaknessess.

4 comments
AuthorsOpenReview ↗

Response to Reviewer PNWB

We thank the reviewer for the careful reading and the concrete suggestions.

W1. The abstract and introduction do not clearly explain the problem and the contributions. We agree, and both are rewritten. They now separate the problem formulation, the benchmark, and the method, and present each as a new contribution (the abstract reads "we formulate…", "we introduce SERBench…", "We propose MSS-Complement…"). The first paragraph of the introduction gives a concrete example (Figure 1): the agent has read one function, its next decision needs two further pieces of code, and relevance ranking returns code it has already read. Paragraph 3 then states the research question and how the work differs from existing benchmarks and methods, before the task, the benchmark, and the method are introduced. The abstract and the introduction no longer use "joint sufficiency" or "grouped certificates", and terms such as "certificate" are defined at first use. Where to look: the Abstract and Section 1.

W2. Table 4 does not support selected-set feedback, and it does not separate expansion from finalization. We agree on both points. Feedback adds 0.20 points [−4.00, +3.14], so the revised paper no longer presents it as a key mechanism and attributes the gain to the overall set-selection policy. We also ran the targeted ablation the reviewer asked for. It reruns finalization on the frozen first-call proposals without the expansion candidates (same 500 states, paired 95% intervals clustered by repository).

VariantProposalExpansionFinalizationUnitsComplete-MSS@5Full returned setContext (tokens)
Single-stage proposal✓870.00%79.40%4,139
No expansion (new)✓✓4 to 872.60%76.40%3,085
MSS-Complement✓✓✓4 to 873.00%80.60%3,663
Fixed top-8 finalization✓✓✓873.00%81.40%4,113

Finalization matters in the first five units: it brings Complete-MSS@5 from 70.00% to 72.60%, a gain of 2.60 points [0.00, +6.27], and expansion adds a further 0.40 [−2.82, +1.93] there. Expansion matters on the full returned set, which is what the agent receives: it brings completion from 76.40% to 80.60%, a gain of 4.20 points [+0.78, +6.30]. Without it the finalizer returns the smallest sets, so that row is below the single call there. Together they put the full policy 3.00 points [+0.99, +4.70] above the single call at five units and 2.00 points [+0.27, +3.85] above it when both return eight units (last row, the same ranking extended to eight). They take about a quarter of the three calls' cost. Returning 4 to 8 units then cuts the context by 11% and changes completion by −0.80 points [−1.57, 0.00]. The full policy is also 6.40 points [+3.42, +10.46] above a control that makes the same three LLM calls but selects by similarity, so the extra calls alone do not explain the gain. Where to look: Section 6.6, Table 4, and Appendix E.2.

W3. The main evaluation selects from supplied candidate pools. The supplied pools are a deliberate design choice, and Appendix B.4 now states its reasons. Retrieval for an agent has two steps, finding candidate code and selecting a sufficient set from it. SERBench fixes the first step in order to measure the second, for three reasons. First, every pool contains a complete set, so a failure is a failure of selection. Over the whole repository, the BM25 top 1,000 contains a complete set for only 333 of the 500 states, and on the others no selection method can succeed. Second, a certificate lists every code unit that can meet each requirement. Experts can check that this list is complete for a pool of 20 to 120 units, which is not feasible for a whole repository, where a valid unit outside the list would be scored as a miss. Third, all methods choose from the same units, so differences in indexing do not enter the comparison. The pools remain hard: five units chosen at random complete 5.17% of states, the strongest baseline completes 61.40%, and an oracle 99.80%.

We agree with the reviewer that this design does not measure discovery, and that the main text must say so. The scope is now stated in the abstract ("All methods select from the same candidate pool, which contains a complete set"), in Section 1 (paragraphs 5 and 8), in Section 4.3, at the opening of Section 5 ("MSS-Complement addresses the selection step of evidence recovery"), and in the Conclusion. Discovery is measured by a separate whole-repository diagnostic, where MSS-Complement reaches 38.8% against 33.8% for the strongest baseline and finding the needed code becomes the main bottleneck. A new analysis rescores it on the 434 states shared with Test500 under the current certificates: 44.47% against 40.55%, a gain of 3.92 points [+1.56, +8.03], against 71.20% and 58.76% on the released pools. The lead persists but narrows. Where to look: Section 4.3, Section 6.5 (paragraph 1), and Appendices B.4 and D.6.

AuthorsOpenReview ↗

Gentle reminder: revised paper and response

Thank you again for reviewing our paper. This is a gentle reminder that the paper has been revised on your three points:

  • W1 (clarity): the abstract and the introduction are rewritten.
  • W2 (ablation): Table 4 now separates expansion from finalization. Our reply on this point was updated on 11 November.
  • W3 (scope): the abstract and Sections 1, 4.3, and 5 state that the benchmark measures selection from supplied pools.

If any concern remains, we would be glad to discuss it.

Authors of Submission 26769

AuthorsOpenReview ↗

Supplement for Reviewer PNWB: the revised text for each request, quoted

To save the reviewer's time, this supplement quotes the revised text for each request in the review, so each change can be checked here without opening the PDF. We also kindly invite the reviewer to read the revised abstract and introduction (pages 1 to 3).

Request in the reviewRevised textWhere
W1. Identify SERBench and MSS-Complement as newly proposed"we formulate state-conditioned minimal sufficient evidence recovery", "we introduce SERBench", "We propose MSS-Complement, a training-free method that chooses code units jointly"Abstract
W1. Explain the problem concretely before abstract termsThe introduction opens with the setting and an example. In Figure 1 the agent has read one function, its next decision needs error-handling code and logging code, and relevance ranking returns code it has already read. "Joint sufficiency" and "grouped certificates" are no longer used.Sec. 1, par. 1 and 2, Fig. 1
W1. State what the paper aims to solve"given an agent's current state, can we retrieve a small set of code that covers everything its next decision still lacks?"Sec. 1, par. 3
W1. State how the contributions differ from existing workExisting repository-retrieval benchmarks "do not record which code combinations suffice for a specific decision". Sufficiency methods from question answering serve one fixed question, start with no evidence, and train their judges on answer correctness or teacher labels.Sec. 1, par. 3
W1. Distinguish the problem formulation, the benchmark, and the method"We define state-conditioned minimal sufficient evidence recovery, the task of retrieving what an agent's next decision still lacks." "We introduce SERBench, with expert-reviewed certificates, a track for selection from candidate pools, and a track for retrieval from whole repositories." "We propose MSS-Complement, a training-free method that selects code units jointly and leads all tested retrievers."Sec. 1, contribution list
W2. Do not claim a benefit from selected-set feedback"The objective and adaptive sizing account for 6.20 of the 6.40 points, and feedback adds no measurable gain."Sec. 6.6
W2. Separate expansion from finalizationA new stage ablation does this. The table in our response above gives each variant and its result.Sec. 6.6, Table 4
W3. State in the abstract and the introduction that the benchmark selects from a supplied pool"All methods select from the same candidate pool, which contains a complete set." "The main benchmark thus measures how well a method selects from given candidates, and a separate track measures how well it finds candidates in the whole repository"Abstract, Sec. 1, par. 5
W3. State it in the method description"MSS-Complement addresses the selection step of evidence recovery. Given the visible state, a supplied candidate pool, and a token budget, it returns an ordered set of code units from the pool for the next decision."Sec. 5, opening
W3. Do not leave the scope to the appendix"each pool contains a complete set and the main benchmark tests selection from given candidates". "With candidates from the whole repository, the method still leads, but finding them becomes the main bottleneck."Sec. 4.3, Conclusion

If any request is still not met, we would be grateful to know which one, and we will revise it.

Authors of Submission 26769

AuthorsOpenReview ↗

Kind follow-up on our response and revision

Dear Reviewer PNWB,

As the discussion period draws to a close, we kindly follow up on our response and the revised paper.

  • What changed: we believe the revision addresses every concern in your review. The supplement above quotes the revised text for each one.
  • Our request: if it resolves your concerns, we would sincerely appreciate it if you could consider updating your rating. If any concern remains, we would be glad to hear which one.

Thank you very much for your time and consideration.

Authors of Submission 26769

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers: summary of the revision

We thank the three reviewers for their careful and constructive reviews. Their concerns converge on four points, and we revised the paper on each. This note lists what changed and where. The individual replies give the answers.

  1. Clarity of the problem and the contributions (PNWB W1; sY9E W1, W2, W3, Q1). The abstract and the introduction are rewritten. The introduction states the research question, presents the new task, the new benchmark (SERBench), and the new method (MSS-Complement) separately, and says what each reported result shows.
  2. Which mechanisms carry the gain (PNWB W2; KDZJ W2, Q2; sY9E W4, Q2). The paper no longer presents selected-set feedback as a core mechanism. It states that feedback adds no measurable gain and attributes the gain to the overall set-selection policy. A new stage ablation separates expansion from finalization (Section 6.6, Table 4).
  3. Scope (PNWB W3; KDZJ W1, Q1). The abstract, Section 1, Section 4.3, and Section 5 now state that the main benchmark and the method address selection from a supplied candidate pool. The pools are supplied by design: every pool contains a complete set, so a failure is a failure of selection. Retrieval from the whole repository is a separate diagnostic, and we rescored it on the 434 shared states, as KDZJ requested (Appendix D.6).
  4. Cost and downstream evidence (KDZJ W1; sY9E Q3). A new Section 6.3 reports the tokens spent by the retrieval calls and the tokens saved in the agent's context. Section 6.4 separates what is established (the effect of complete evidence at a single repair decision) from what is not (end-to-end repair, where every paired interval includes zero).

New evidence (Test500, Complete-MSS@5, paired 95% intervals clustered by repository, unless noted)

QuestionResultWhere
What do the two later calls add over a single call?+3.00 [+0.99, +4.70] at five units, and +2.00 [+0.27, +3.85] when both return eight units. Compared with finalization without it, expansion adds +4.20 [+0.78, +6.30] on the full returned set§6.6, Table 4, App. E.2
Does selected-set feedback help?+0.20 [−4.00, +3.14], which is no measurable gain§6.6
Full repository under the current certificates (434 states)44.47% against 40.55% for the strongest baseline: +3.92 [+1.56, +8.03]App. D.6
What does retrieval cost, and what does the agent save?Cost: MSS-Complement makes three LLM calls for retrieval, 41.50K tokens per decision, about 1.5 US cents at DeepSeek's off-peak list prices of 10 November 2026. The two later calls take about a quarter of this cost. These calls are separate from the coding agent, and none of their tokens enters its context. Saving: under the same budget, the code that enters the agent's context is 40% smaller at matched completeness (3,655 against 6,100 tokens)§6.3, App. E.1

The main text stays within ten pages. The supplementary material is indexed by its README.md, and the data behind the new results are under results/test500_attribution/ and results/full_repository/rescoring_434/.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.