acceptodds
Under review as a conference paper at ICLR 2027

CONTEXTBENCH: A Benchmark for Context Retrieval in Coding Agents

Abstract

LLM-based coding agents have shown strong performance on automated issue resolution benchmarks, yet existing evaluations largely focus on final task success, providing limited insight into how agents retrieve and use code context during problem solving. We introduce CONTEXTBENCH, a process-oriented evaluation of context retrieval in coding agents. CONTEXTBENCH consists of 1,136 issue- resolution tasks from 66 repositories across eight programming languages, each augmented with human-annotated gold contexts. CONTEXTBENCH implements an automated evaluation framework that tracks agent trajectories and measures context recall, precision, and efficiency throughout issue resolution. Comprehensive experiments on CONTEXTBENCH across seven frontier LLMs and seven coding agents reveal that simple shell-based agents achieve the strongest overall context retrieval, while specialized retrieval mechanisms provide no consistent gains, echoing the “Bitter Lesson” for coding-agent design. Most evaluated LLMs show higher recall than precision at the block and line levels, and substantial gaps exist between explored and utilized context. CONTEXTBENCH augments existing end-to-end benchmarks with intermediate gold-context metrics that unbox the issue-resolution process. These contexts offer valuable intermediate signals for guiding LLM reasoning in software tasks

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Context retrieval, Code agents, Repository-level coding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PSBfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ContextBench, a benchmark for evaluating context retrieval in coding agents beyond final repair success. The framework tracks code observed during execution and agents’ final context declarations, measuring reference overlap at the file, block, and line levels alongside retrieval dynamics and patch correctness. Experiments show strong retrieval performance from simple shell-based agents, and substantial improvements in repair success when gold context is provided.

Critical strengths
  1. The paper proposes a value practical resource that take a more fine grained look beyond general SWE benches, by characterizing the retrieval aspect as a valuable intermediate signal.
  2. The paper provides a thorough empirical analysis across coding agents and model backbones, covering retrieval quality, efficiency, redundancy, and gold-context robustness. The oracle experiments complement these analyses by demonstrating the practical value of relevant context for issue resolution.
Critical weaknesses
  1. Gold-based precision alone may be insufficient to assess the usefulness of diagnostic exploration. Annotators construct compact gold contexts with access to the ground-truth patch, whereas an agent without this knowledge may need to inspect additional code to distinguish plausible causes and rule out incorrect hypotheses. Precision alone may therefore be insufficient to capture the value of such diagnostic exploration. Although this limitation is acknowledged in the paper, the current benchmark design may undermine its central goal of assessing retrieval quality during problem solving.
  2. The evaluation primarily covers proprietary models, with limited coverage of strong open-weight models beyond Devstral 2. Including additional models from the Qwen and DeepSeek families would broaden the evaluation and help assess whether the observed retrieval patterns generalize across a more diverse set of model families.
Questions for authors

See weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer PSBf

We thank the reviewer for highlighting the role of diagnostic exploration and suggesting broader coverage of open-weight models.

Q1 Gold-based precision alone may be insufficient to assess the usefulness of diagnostic exploration. Annotators construct compact gold contexts with access to the ground-truth patch, whereas an agent without this knowledge may need to inspect additional code to distinguish plausible causes and rule out incorrect hypotheses. Precision alone may therefore be insufficient to capture the value of such diagnostic exploration. Although this limitation is acknowledged in the paper, the current benchmark design may undermine its central goal of assessing retrieval quality during problem solving. See weakness.

A1 We agree that diagnostic exploration can be useful even when it extends beyond the gold context, as acknowledged in Section 2.4 and Appendix I.4. Appendix L(i) also clarifies that precision reflects agreement with the reference rather than the usefulness of every inspected artifact. Within this scope, we interpret recall as a measure of reference coverage and precision as a proxy for context efficiency. We will clarify in the discussion that inspecting code to rule out a plausible cause can lower precision even when it helps resolve the issue.


Q2 The evaluation primarily covers proprietary models, with limited coverage of strong open-weight models beyond Devstral 2. Including additional models from the Qwen and DeepSeek families would broaden the evaluation and help assess whether the observed retrieval patterns generalize across a more diverse set of model families.

A2 We expanded the mini-SWE-agent evaluation with four models, including GLM-5.1, MiniMax and two Qwen models. They all have publicly available weights.

ModelLine recallLine precisionLine F1Pass@1
GLM-5.10.2360.4330.27246.4%
MiniMax-M2.50.2360.3830.27947.4%
Qwen2.5-32B0.0760.1840.0925.6%
Qwen3-8B0.0480.1800.0683.6%

Line-level F1 remains below 0.35 for all four models, supporting our finding that fine-grained retrieval remains challenging. We will include these results in the revision and clarify that the direction of the precision–recall gap varies across models.

Thank you again for your thoughtful review. As the discussion period draws to a close, we wanted to ask whether our response addressed your concerns. If any questions remain, we’d be happy to discuss them further.

Thank you for the clarification. My concern about Q1 remains partially unresolved. If useful diagnostic exploration extends beyond the gold context, an effective agent may receive lower explored-context precision even when the additional inspection is necessary. I would like to see this limitation addressed in the construction of the gold contexts.

Thank you for clarifying this concern. We have revised Section 2.3 and Appendix L(ii) to make this limitation explicit. Starting from a known patch may produce a more compact context than an agent needs during diagnosis, while the patch alone cannot fully reveal the evidence needed to rule out other plausible causes. We now acknowledge that our gold contexts may therefore omit useful diagnostic evidence, causing necessary exploration to receive lower explored-context precision. The changes are marked in blue.

Reviewer_eQDeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary
  • The paper builds ContextBench, a benchmark to measure how coding agents find and use code context while fixing GitHub issues. It takes 1136 tasks from four existing benchmarks, 66 repos, 8 languages, and adds human-annotated "gold context". A 500-task Lite subset is used for all the main experiments.
  • Gold context is traced by experts starting from the gold patch, then checked by letting GPT-5 write a passing patch from the context alone, then compacted by a second annotator. If GPT-5 fails they fall back to two-annotator agreement plus a third expert.
  • Experiments: 7 agents on a GPT-5 backbone, 7 LLMs inside mini-SWE-agent, an oracle study where agents get the gold context, retrieval dynamics and a gold-context robustness check.
  • Main messages: shell-based agents retrieve best, models favor recall over precision, and agents often see the right code but then drop it from the final context.
Critical strengths
  • The dataset is a real, expensive artifact. Human-verified gold context for 1000+ multi-language tasks, four months of expert annotation, a verify-then-compact loop, and an anonymous release of code and data. This is the kind of thing people will actually reuse, and it gives an intermediate signal that can be turned into a dense reward for training agents, which the paper hints at.
  • The protocol is careful about what it does and does not measure. The authors separate explored context from declared context, define every metric formally, report Pass@1 separately, and the limitations are honest about precision being relative to a non-minimal reference and about evidence drop not proving "forgetting".
  • For a benchmark paper the experiments are broad: 7 agents x 7 models, an oracle study, cost and step analysis, per-language radar plots, and even a protocol-failure case study for Devstral 2. The result is a clean argument that retrieval is a real bottleneck.
Critical weaknesses
  • Related work misses the existing line on code localization evaluation, which is basically the same measurement idea. Agentless and RepoGraph report file/function localization accuracy on SWE-bench; OrcaLoca reports function match rate, file match rate and a function-match precision on SWE-bench Lite and already links localization quality to resolved rate; LocAgent released Loc-Bench for file/module/function-level localization. None of these are cited or compared. The intro claim that existing evaluations "ignore how coding agents arrive at their solutions" is too strong. The paper should say clearly what is new instead of claiming the whole idea.
  • The "shell-based retrieval is sufficient, specialized retrieval is over-engineered" headline is not what the data shows. The two best agents in Pass@1 are Claude Code and Codex, which are closed-source, heavily engineered products, not "simple" scaffolds. The actually simple one, mini-SWE-agent, has the second lowest Pass@1 (0.472, only above Agentless). Agentless has the best line-level F1 but the worst Pass@1. So retrieval-F1 ranking and resolution ranking disagree, and n=7 agents with one backbone is a thin basis for a "Bitter Lesson".
  • There is a circularity in the oracle study. Gold contexts were accepted when GPT-5 could produce a passing patch from them, and then RQ3 shows GPT-5-based agents reach ~0.70 Pass@1 when handed the gold context. Part of the +19 to +23 pp gain is built into how the data was constructed. The paper should discuss this and ideally run the oracle with a model family that was not the verifier.
  • Several setup details that matter are missing: how Claude Code was run with a GPT-5 backbone, what agents were allowed to do with test files, and why all main results are on the 500-task Lite subset while the paper advertises 1000+ tasks.
Questions for authors
  • Clarification: How exactly was Claude Code executed with GPT-5? Was a proxy/router used, and were Claude Code's own system prompt and tools kept unchanged? Same for Codex. How much of the "simple shell" story is really the vendor's internal context management?
  • Clarification: In RQ5, who or what produced the 884 patch-conditioned gold contexts, and were alternative patches filtered in a way that makes block-level overlap 1.0 by construction? Please show the distribution, not only the mean.
  • Clarification: Was GPT-5 also the backbone in the RQ3 oracle study? If yes, can you report the oracle gain for a model that was not used for verification, so the retrieval bottleneck can be separated from the verification bias?
2 comments
AuthorsOpenReview ↗

Response to Reviewer eQDe

We thank the reviewer for pointing us to prior work on code localization and for the detailed questions about our experiments.

Q1 Related work misses the existing line on code localization evaluation, which is basically the same measurement idea. Agentless and RepoGraph report file/function localization accuracy on SWE-bench; OrcaLoca reports function match rate, file match rate and a function-match precision on SWE-bench Lite and already links localization quality to resolved rate; LocAgent released Loc-Bench for file/module/function-level localization. None of these are cited or compared. The intro claim that existing evaluations "ignore how coding agents arrive at their solutions" is too strong. The paper should say clearly what is new instead of claiming the whole idea.

A1 We agree that prior work already evaluates code localization. The localization metrics discussed here compare predicted or actual edit locations with those in the reference patch, whereas CONTEXTBENCH compares agent context with expert-annotated gold context. The latter includes unchanged definitions and dependencies supporting issue resolution. Although both evaluations can operate at file or function granularity, locating the correct edit site does not establish that an agent has retrieved the context needed to understand and implement the change. Our contribution is the context annotations and the evaluation of context retrieval throughout the repair process. Table 2 includes Agentless. We will clarify our contribution in the introduction and discuss RepoGraph, OrcaLoca, and LocAgent in Related Work.


Q2 The "shell-based retrieval is sufficient, specialized retrieval is over-engineered" headline is not what the data shows. The two best agents in Pass@1 are Claude Code and Codex, which are closed-source, heavily engineered products, not "simple" scaffolds. The actually simple one, mini-SWE-agent, has the second lowest Pass@1 (0.472, only above Agentless). Agentless has the best line-level F1 but the worst Pass@1. So retrieval-F1 ranking and resolution ranking disagree, and n=7 agents with one backbone is a thin basis for a "Bitter Lesson".

A2 We agree that Claude Code and Codex are highly engineered systems. We use “simple” to describe the retrieval operations they expose. Anthropic’s context engineering article explicitly describes Claude Code’s on-demand retrieval using glob and grep, while OpenAI’s Codex guidance recommends ripgrep and shell-based inspection. Table 2 shows that all three shell-based agents, including mini-SWE-agent, occupy the top three positions in file-level and block-level F1. Agentless’s lead in line-level F1 and mini-SWE-agent’s lower Pass@1 reflect the distinction between retrieval quality and successful issue resolution, which also depends on reasoning and patch generation. We will revise the RQ1 headline to specify the file- and block-level retrieval result and scope the “Bitter Lesson” discussion to the seven systems evaluated.

AuthorsOpenReview ↗

Response to Reviewer eQDe

Q3 Clarification: Was GPT-5 also the backbone in the RQ3 oracle study? If yes, can you report the oracle gain for a model that was not used for verification, so the retrieval bottleneck can be separated from the verification bias?

A3 RQ3 uses GPT-5 as the backbone. As described in Section 2.3 and Appendix G, gold contexts are annotated by experts, and GPT-5 provides a feasibility check through five patch-generation attempts. Experts can also retain a context when executable verification fails. To test whether the gain extends beyond the verifier, we evaluated DeepSeek-V4-Flash with mini-SWE-agent on the same 500 tasks, using the same gold contexts and evaluation settings. We will report this comparison in RQ3 and clarify GPT-5’s role in verification.

BackboneWithout gold contextWith gold contextGain
GPT-5 — Table 40.4720.702+23.0 pp
DeepSeek-V4-Flash0.4980.740+24.2 pp

Q4 Clarification: How exactly was Claude Code executed with GPT-5? Was a proxy/router used, and were Claude Code's own system prompt and tools kept unchanged? Same for Codex. How much of the "simple shell" story is really the vendor's internal context management?

A4 We ran Claude Code and Codex with GPT-5 through our own proxy, which translates API protocols while preserving each agent’s original system prompt and tools. As described in Section 2.5, explored-context measurements use the source code actually returned by retrieval tools in the recorded trajectories. Section 3.1 includes each agent’s context management in the system comparison. Context compaction was almost never triggered in our runs. All agents could access existing regression tests. The benchmark-provided reproduction tests (test_patch) were withheld. Final evaluation followed each source benchmark’s original protocol. The full benchmark contains 1,136 annotated tasks. For efficient evaluation across agents and models, the main experiments use the 500 tasks ranked most difficult under our combined difficulty criteria, as described in Section 2.4 and Appendix B. We will make the run protocol and test access explicit in the experimental setup.


Q5 Clarification: In RQ5, who or what produced the 884 patch-conditioned gold contexts, and were alternative patches filtered in a way that makes block-level overlap 1.0 by construction? Please show the distribution, not only the mean.

A5 The 884 patch-conditioned contexts were produced using the same expert annotation procedure described in Section 2.3, with each patch annotated independently in isolation. Section 3.5 and Appendix H explain that the 331 tasks were included based on the availability of at least two distinct test-passing patches. Appendix H also states that every evaluated context pair has block-level Jaccard similarity exactly equal to 1 before rounding, so the entire block-level distribution is concentrated at 1. The distribution of per-task average line-level Jaccard scores is shown below.

Line-level JaccardTasksPercentage
J < 0.60236.9%
0.60 ≤ J < 0.703711.2%
0.70 ≤ J < 0.806619.9%
0.80 ≤ J < 0.9011635.0%
0.90 ≤ J ≤ 1.008926.9%

The mean is 0.812, the median is 0.845, and the interquartile range is 0.743–0.906. We will include this distribution and the patch-selection criterion in Section 3.5.

Reviewer_zh1aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper introduces ContextBench, which augments 1,136 existing issue-resolution tasks with patch-derived “gold contexts” and evaluates four LLMs and five coding agents using context overlap metrics. The paper reports low retrieval precision, limited gains from sophisticated agent scaffolds, and a gap between explored and finally declared context.

Critical strengths

The paper draws attention to an important weakness of current coding-agent benchmarks: final patch correctness alone provides little information about the problem-solving process. The trajectory instrumentation may be useful for exploratory diagnosis.

Critical weaknesses
  1. The benchmark lacks a defensible ground truth. Gold contexts are traced backward from a reference patch, so they encode one solution path rather than the context necessary for solving the issue. Valid alternative solutions and exploratory information may therefore be incorrectly penalized. LLM-based verification establishes neither necessity nor minimality.

  2. Several central conclusions are not supported by the measurements. “Context usage” is approximated by an agent’s final self-reported <PATCH_CONTEXT>, which does not reveal what information actually influenced its reasoning. Likewise, the scaffold comparison is confounded by different tools, prompts, budgets, and framework adaptations, yet is used to make a broad “Bitter Lesson” claim.

Questions for authors
  1. How can the benchmark distinguish irrelevant retrieval from context supporting a valid solution different from the reference patch?

  2. What evidence shows that omission from the final <PATCH_CONTEXT> means that retrieved information was not used?

  3. Are tool-call, token, and compute budgets controlled across agents? If not, how can the scaffold comparison support a causal conclusion?

3 comments
AuthorsOpenReview ↗

Response to Reviewer zh1a

We thank the reviewer for examining how we define gold context and what our measurements establish about agent behavior.

Q1 How can the benchmark distinguish irrelevant retrieval from context supporting a valid solution different from the reference patch?

A1 We agree that different valid repairs may rely on different context. Section 3.5 examines how much the annotated context varies across solutions. We derive contexts for 884 distinct test-passing patches across 331 tasks using the same annotation procedure. The contexts contain identical sets of code blocks within each task, with a block-level Jaccard similarity of 1.0 and an average line-level similarity of 0.812. These results support the consistency of gold contexts across the alternative solutions we studied. As described in Section 2.3, annotators trace semantic dependencies beyond the edited regions, with experts reviewing relevance and compactness and executable tests checking sufficiency. We will clarify in Section 3.5 that this consistency result applies to the alternative patches studied.


Q2 What evidence shows that omission from the final <PATCH_CONTEXT> means that retrieved information was not used?

A2 We agree that omission from the final <PATCH_CONTEXT> alone does not establish that the agent did not use the code. In Agentless and Prometheus, the stage boundary does establish which code was directly available to the patch generator. Agentless narrows candidate files and code regions through localization, while Prometheus selects retrieved code for its repair stage. Both pass only the selected code forward, although earlier observations may have informed that selection. Iterative agents share an interaction history across retrieval and editing, so code omitted from the declaration may still have guided the repair (Appendix I.4). We also evaluate explored context directly from the source code returned by tool calls (§2.5). Tables 7 and 8 report that, at the reported precision, explored-context F1 yields the same rankings as patch-context F1 at the file, block, and line levels in both the agent and model comparisons. This agreement concerns retrieval performance across systems. In the revision, we will emphasize the distinction between observed code and reported final selection when interpreting these results.


Q3 Are tool-call, token, and compute budgets controlled across agents? If not, how can the scaffold comparison support a causal conclusion?

A3 Budgets were not matched across agents. We imposed no additional step or token caps, allowing each agent to finish under its own stopping rule. All seven agents used GPT-5 on the same tasks and repository snapshots under a common scoring protocol (Appendix M). Because each retained its own tools and prompts, the comparison measures complete systems and cannot isolate the effect of a retrieval component. The bash-only mini-SWE-agent outperformed SWE-agent, OpenHands, and Prometheus on file-level and block-level F1 for both declared and explored context (Tables 2 and 7), while Agentless led at the line level. We will revise the “Bitter Lesson” discussion to say that specialized retrieval showed no consistent gains in this comparison and report average tool calls, token use, and cost per task for each agent alongside Table 2.

Thank you for the detailed response.

However, I noticed that all reported F1 scores in Tables 7 and 8 match the corresponding scores in Tables 2 and 3 multiplied by 0.95, after rounding to three decimal places.

Could you clarify whether the explored-context F1 scores were computed independently from the observed tool outputs, and explain this uniform scaling pattern?

My concern remains, so I will retain my current score.

AuthorsOpenReview ↗

Response to Reviewer zh1a

We rechecked our Overleaf project against the submitted PDF and confirmed the 0.95 pattern you identified. The submitted PDF appears to contain an earlier placeholder version of Tables 7 and 8. We have not been able to determine how that version was included, and we missed the discrepancy during our final check of the appendix. In the current version, we computed explored-context F1 independently from the source-code regions returned by tool calls, scoring each task against its gold context and then averaging across tasks. We have now submitted the corrected manuscript with the updated Tables 7 and 8, the corresponding Explored F1 columns in Tables 2 and 3, and the revised discussion in Appendix I.2. We appreciate the reviewer for bringing this discrepancy to our attention.

Comments on the paper

1 comment
Area_Chair_8g4DOpenReview ↗

Discussion

Dear Reviewers,

We are reaching the end of the discussion period! If the authors have responded to your review, please read their response, leave a comment, and update your rating if your assessment has changed.

Thank you,

AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.