- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes ContextBench, a benchmark for evaluating context retrieval in coding agents beyond final repair success. The framework tracks code observed during execution and agents’ final context declarations, measuring reference overlap at the file, block, and line levels alongside retrieval dynamics and patch correctness. Experiments show strong retrieval performance from simple shell-based agents, and substantial improvements in repair success when gold context is provided.
Critical strengths
- The paper proposes a value practical resource that take a more fine grained look beyond general SWE benches, by characterizing the retrieval aspect as a valuable intermediate signal.
- The paper provides a thorough empirical analysis across coding agents and model backbones, covering retrieval quality, efficiency, redundancy, and gold-context robustness. The oracle experiments complement these analyses by demonstrating the practical value of relevant context for issue resolution.
Critical weaknesses
- Gold-based precision alone may be insufficient to assess the usefulness of diagnostic exploration. Annotators construct compact gold contexts with access to the ground-truth patch, whereas an agent without this knowledge may need to inspect additional code to distinguish plausible causes and rule out incorrect hypotheses. Precision alone may therefore be insufficient to capture the value of such diagnostic exploration. Although this limitation is acknowledged in the paper, the current benchmark design may undermine its central goal of assessing retrieval quality during problem solving.
- The evaluation primarily covers proprietary models, with limited coverage of strong open-weight models beyond Devstral 2. Including additional models from the Qwen and DeepSeek families would broaden the evaluation and help assess whether the observed retrieval patterns generalize across a more diverse set of model families.
Questions for authors
See weakness.
4 comments
Thank you again for your thoughtful review. As the discussion period draws to a close, we wanted to ask whether our response addressed your concerns. If any questions remain, we’d be happy to discuss them further.
Thank you for the clarification. My concern about Q1 remains partially unresolved. If useful diagnostic exploration extends beyond the gold context, an effective agent may receive lower explored-context precision even when the additional inspection is necessary. I would like to see this limitation addressed in the construction of the gold contexts.
Thank you for clarifying this concern. We have revised Section 2.3 and Appendix L(ii) to make this limitation explicit. Starting from a known patch may produce a more compact context than an agent needs during diagnosis, while the patch alone cannot fully reveal the evidence needed to rule out other plausible causes. We now acknowledge that our gold contexts may therefore omit useful diagnostic evidence, causing necessary exploration to receive lower explored-context precision. The changes are marked in blue.
Response to Reviewer PSBf
We thank the reviewer for highlighting the role of diagnostic exploration and suggesting broader coverage of open-weight models.
Q1 Gold-based precision alone may be insufficient to assess the usefulness of diagnostic exploration. Annotators construct compact gold contexts with access to the ground-truth patch, whereas an agent without this knowledge may need to inspect additional code to distinguish plausible causes and rule out incorrect hypotheses. Precision alone may therefore be insufficient to capture the value of such diagnostic exploration. Although this limitation is acknowledged in the paper, the current benchmark design may undermine its central goal of assessing retrieval quality during problem solving. See weakness.
A1 We agree that diagnostic exploration can be useful even when it extends beyond the gold context, as acknowledged in Section 2.4 and Appendix I.4. Appendix L(i) also clarifies that precision reflects agreement with the reference rather than the usefulness of every inspected artifact. Within this scope, we interpret recall as a measure of reference coverage and precision as a proxy for context efficiency. We will clarify in the discussion that inspecting code to rule out a plausible cause can lower precision even when it helps resolve the issue.
Q2 The evaluation primarily covers proprietary models, with limited coverage of strong open-weight models beyond Devstral 2. Including additional models from the Qwen and DeepSeek families would broaden the evaluation and help assess whether the observed retrieval patterns generalize across a more diverse set of model families.
A2 We expanded the mini-SWE-agent evaluation with four models, including GLM-5.1, MiniMax and two Qwen models. They all have publicly available weights.
Line-level F1 remains below 0.35 for all four models, supporting our finding that fine-grained retrieval remains challenging. We will include these results in the revision and clarify that the direction of the precision–recall gap varies across models.