- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper explore state conditioned evidence retrieval problem for coding agents. It argues that traditional relevance based retrieval may return evidence already observed by an agent or fail to retrieve a jointly sufficient set of evidence for its current decision. To evaluate this problem, the paper introduces SERBench, which contains intermediate coding agent states and grouped evidence sufficiency annotations. The paper further proposes MSS-Complement, a three stage semantic retrieval framework that constructs, expands, and finalises a compact evidence set based on the agent's current information needs. Experiments show improvements in complete set recovery over retrieval baselines, with additional analysis on repair localisation, long horizon agent QA, etc.
Critical strengths
- I think the paper proposes a useful evaluation perspective for agent retrieval, to some degree.
The paper highlights two relevant limitations of traditional relevance based retrieval: 1) redundancy with previously observed information and 2) insufficient coverage of complementary evidence. SERBench attempts to evaluate these issues through captured agent states and grouped evidence, providing a more fine-grained perspective than traditional retrieval metrics.
- The experiment and analysis are extensive to some degree.
The paper evaluates multiple retrieval baselines and includes component ablations and downstream coding agent experiments. The human validation of evidence annotations and the attempt to connect retrieval completeness with downstream decisions are valuable aspects of the paper.
Critical weaknesses
- I think the paper suffers from a little serious presentation and logical issues that hinder me understand its central contributions. Especially the abstract and introduction, even though I read them multiple times, I am still confused in many places such as the motivation, research gap, research question, and the relationships between the motivation and SERBench and the framework, etc.
The abstract and introduction seem difficult to follow. The paper frequently moves between different concepts, research objectives and experimental findings without clearly establishing their logical connections. For example, the abstract moves from the redundancy of relevance based retrieval to state conditioned evidence recovery, then immediately introduces SERBench and MSS-Complement, without explaining why the identified limitations require the proposed benchmark and method. Similarly, the opening paragraphs (line 40-42) of the introduction move from changing information needs during coding agent execution to long context limitations (lost in the middle etc), and then to retrieval objectives, without clearly establishing how these arguments lead to the proposed research gap. There are still multiple other places in the introduction and abstract that have this problem.
Furthermore, the abstract/introduction introduces several results including complete set recovery, matched similarity controls, etc, without clearly explaining what each evaluation shows or how they support the central contribution.
- I am a little confused about methodological novelty and I think the experiment evidence might not sufficiently establish the contribution of MSS-Complement.
The proposed approach relies on evidence-set construction, iterative retrieval based on missing evidence, etc. However, related work such as Evidence Tree Search (ACL 2025) and S2G-RAG (ACL 2026) seems to have already explored closely related mechanisms. Although these works are cited, the paper does not sufficiently clarify what fundamentally distinguishes MSS-Complement beyond applying these principles to captured coding-agent states. What is the advantage of MSS-Complement?
Moreover, the component ablations raise questions about the necessity of the central mechanism. Table 4 shows that the no-feedback adaptive variant achieves 72.80% Complete MSS@5, compared with 73.00% for the full method. This suggests that the independent contribution of selected-set feedback may be limited in the current experiments. But it seems to be emphasised as a central part of missing-complement acquisition.
Questions for authors
- Research motivation and method difference.
Could the authors clarify the central research gap and explain how state conditioned minimal sufficient evidence recovery differs from existing retrieval methods such as Evidence Tree Search and S2G-RAG? Especially, what challenge arising from coding-agent states is not adequately addressed by these existing methods?
- Contribution of the proposed set level mechnism.
Table 4 shows that removing selected set feedback from the adaptive variant changes Complete-MSS@5 from 73.00% to 72.80%. What is the specific contribution of selected set feedback and explicit missing-complement identification? Why is the three stage semantic controller necessary when a simpler variant achieves nearly identical performance?
- Considering the substantially higher online token consumption on AMA-Bench and the limited statistical evidence for downstream QA and code repair improvements, could the authors further clarify the cost-performance trade-off of the additional controller computation?
Actually, I find the research problem of the paper interesting to some degree, but the unclear organisation makes it unnecessarily difficult to identify the precise research question, understand the methodological motivation, and assess the relationship between the claimed contributions and the experimental evidence. I encourage the paper to apply restructuring and clearer explanations in the next version.
7 comments
Gentle reminder: revised paper and response
Thank you again for reviewing our paper. This is a gentle reminder that the paper has been restructured as you suggested:
- W1, W2 (presentation): the abstract and the introduction are rewritten around one research question.
- W3, Q1 (novelty): paragraph 3 of the introduction states the difference from Evidence Tree Search and S2G-RAG.
- W4, Q2 (mechanism): selected-set feedback is no longer presented as a core mechanism.
- Q3 (cost): Section 6.3 reports the cost of the retrieval calls.
If any concern remains, we would be glad to discuss it.
Authors of Submission 26769
Thank you for the detailed response and revisions. I appreciate the authors' efforts, but I still have concerns about the clarity and coherence of the research motivation, research gap, and overall contribution. Therefore, I will maintain my original rating.
Response to Reviewer sY9E
We thank the reviewer for the detailed comments. Following them, we restructured the paper and rewrote the abstract and the introduction in full. Our answers are below, each with the place in the revised PDF.
W1 and W2. Presentation and logic of the abstract and introduction. Both now follow one chain. The setting comes first, then why relevance ranking fails (it repeats code the agent has read and can miss a piece the decision still needs), then the research question and the gap: existing repository-retrieval benchmarks do not record which code combinations suffice for a specific decision. The task, SERBench, and MSS-Complement follow, each introduced as the answer to the step before it. Every reported result is now followed by what it shows. Section 6 opens with a roadmap of what each subsection tests. Where to look: the Abstract and Section 1 (paragraph 3 holds the research question and the gap).
W3 and Q1. Difference from Evidence Tree Search and S2G-RAG, and the advantage of MSS-Complement. We do not claim that building an evidence set or searching for what is missing is new. The difference lies in the problem. ETS and S2G-RAG serve a reader answering one fixed question, start with no evidence, and judge sufficiency with models trained on answer correctness or teacher labels. A coding agent's need changes at every step, part of the needed code has already been read, and an intermediate decision offers no answer to learn from. These are the challenges that the two methods do not address. Our main novelty is the task and the benchmark, which records what each decision still needs and scores a returned set against it. MSS-Complement needs no training. It is 11.60 points [+6.89, +17.21] above the strongest of eleven retrieval baselines, and 6.40 points [+3.42, +10.46] above a control that makes the same three LLM calls but selects by similarity. We did not run ETS or S2G-RAG. They were built for question answering, and using them here would mean replacing their central parts: a newly trained sufficiency judge, selection over code units in place of sentences or passages, and a way to leave out code the agent has read. The result would be a new method of our own design that shares only their core idea, and it would not be a faithful baseline. The paper claims no advantage over them. Where to look: Section 1, paragraph 3, which now states these differences, and Section 2.
W4 and Q2. Selected-set feedback and the three stages. We agree. Feedback adds 0.20 points [−4.00, +3.14], which is no measurable gain, and the revised paper no longer presents it as a core mechanism. A new ablation separates the two later calls (the "No expansion" row of Table 4). Compared with a single call, they add 3.00 points [+0.99, +4.70] at five units and return 476 fewer tokens of code to the agent per state. They run on the smaller model and account for about a quarter of the cost of the three calls. Finalization supplies most of this gain, and expansion adds 4.20 points [+0.78, +6.30] on the full returned set. The variant the reviewer cites still makes three calls, so it is no simpler than the full method. The evidence supports using more than one call. It does not show that the calls must see each other's selections. Where to look: Section 6.6 and Appendix E.2.
Q3. Cost of the additional computation. Two different token counts are involved. The cost is on the retrieval side: MSS-Complement makes three LLM calls for retrieval, 41.50K tokens per decision. These calls are separate from the coding agent that carries out the task, and none of their tokens enters its context. At DeepSeek's off-peak list prices on 10 November 2026, they cost about 1.5 US cents per decision, and the retrieval baselines make no LLM call. What this cost buys is in the agent's own context: within five units, the share of states with a complete set rises from 61.40% to 73.00%, and under the same 6,144-token budget the code the agent receives is 40% smaller at matched completeness (3,655 against 6,100 tokens). The control with the same three calls shows that the computation alone does not explain the gain. Downstream, the clearest evidence is at a single repair decision (Action52), where removing one required piece from complete evidence lowers precision by 12.28 points [+4.27, +20.34] under DeepSeek-v4-flash and by 11.11 [+5.20, +17.51] under Claude Sonnet 5. We do not claim lower total computation, and we agree that the end-to-end evidence is limited. On Fresh23 every paired interval includes zero, and on AMA-Bench the online total, which adds the retrieval calls to the answer model's tokens, is higher. Where to look: Sections 6.3 to 6.5, and Appendix E.1 for the cost of each call.
We are glad to clarify any point that remains unclear.