acceptodds
Under review as a conference paper at ICLR 2027

Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents

Abstract

While coding agents are typically evaluated on end-to-end patch generation, their success heavily depends on an upstream context-acquisition stage: identifying relevant repository files. We introduce Agent Retrieval Bench, a file-level code retrieval benchmark designed for this workflow-driven context discovery problem. Unlike traditional semantic code search, relevance in Agent Retrieval Bench reflects what an agent needs next across four core tasks: code2test, comment2context, trace2code, and edit2ripple, alongside a selective retrieval suite with natural no-gold and counterfactual controls. The benchmark comprises 427 curated samples across 25 repositories over frozen base commits. Evaluating lexical, dense embedding, and RepoMap baselines reveals sharp task-dependent trade-offs and limited selective success on natural no-gold labels. Logged agent trajectories miss all gold files on 27–35% of samples. In a 45-sample, single-run pilot, retrieval-derived seeds achieve higher final File F1 with fewer post-seed calls and read tokens than random non-gold seeds.

Primary area:
datasets and benchmarks
Keywords:
Coding agent, Context Retrieval, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6xvHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Agent Retrieval Bench evaluates how coding agents find repository context across test discovery, code review, failure localization, change impact, and abstention. It contains 427 samples from 25 repositories, using frozen snapshots and task-specific evidence. The main contribution is the workflow coverage and evaluation protocol. Validation includes lexical, structural, and embedding retrieval, interactive agent search, trajectory analysis, and a 45-sample seed intervention.

Critical strengths
  • The task breakdown is useful. Covering these scenarios together captures an important aspect of practical context acquisition. The substantial performance differences across tasks support evaluating them separately.

  • The evaluation goes beyond a retrieval leaderboard. Frozen snapshots, evidence links, and direct leakage checks make the results easier to interpret. Budget, exploration, and abstention analyses also reveal trade-offs that ranking metrics alone would miss.

Critical weaknesses
  • The labels may capture only part of the useful context. Historical edits and task-specific evidence provide reasonable targets, but files that need changing may be only a subset of files worth understanding. High recall against these labels can overstate coverage of the context actually needed, while useful unlabelled outputs reduce measured precision. The same concern applies to abstention: no local fix target does not imply no useful local diagnostic context.

  • The short agent searches and modest scores need more explanation. Closed-tool Codex averages only 3.82 tool calls and reaches 0.3279 File F1. Such limited exploration seems somewhat at odds with what I would expect from everyday coding-agent use. It is unclear whether this reflects task difficulty or the prompts, tool interface, and stopping behavior.

Questions for authors
  1. Did the audit examine useful but unchanged files? Is there evidence showing how missing relevance labels affect method rankings and natural-case abstention results?

  2. What causes the agent searches to stop after so few calls? Could a small set of low-scoring searches be followed by coding attempts with outcomes assessed (tests or manual eval), to check whether low retrieval scores correspond to actual coding failures?

4 comments

Thanks for your response. These are helpful, but my main concerns are not yet fully resolved:

  1. The labels still do not fully align with "what the agent should read next." After human auditing, unchanged method rankings do not resolve the substantial bias in absolute scores, which also matters for a benchmark. I'm not sure if there is an objective ground truth, but manual evaluation is not a suitable standard approach for such benchmark.

  2. I remain concerned about the restricted agent setup. Is removing shell access standard practice? Is the three-file submission cap common for all methods? The agent should be encouraged to gather sufficient evidence before submitting. It remains unclear whether the short searches reflect task difficulty or the interface and stopping instructions.

Thank you very much for engaging in this discussion with us! Constructive dialogue is the best way to sharpen a work, and we truly value this exchange. Below, we address your two remaining concerns concisely.

1. Objective Framing ("Actionable Workflow Context") & Deterministic Score Calibration

We fully agree that (a) manual evaluation is not a suitable standard for benchmark scoring, and (b) absolute scores must align with an objective task definition (our 52-sample human check in Round 1 was purely a diagnostic reply, not a benchmark scoring method). We are making two deterministic revisions:

  • Reframing to Actionable Workflow Context Retrieval: Unlike "what to read next" (which implies subjective browsing order), the unseen files required to execute the workflow action (Gi=Ri∖FigivenG_i = R_i \setminus F^{\text{given}}_i, verified by resolving commits/tests) are 100% objective ground truth. We will revise the paper to frame the task strictly as retrieving Actionable Workflow Context—making recall and BCY against GiG_i exact, unbiased measures of actionable target coverage.

  • Deterministic AST-Graph Calibration for Absolute Precision (Zero Manual Labeling): To avoid penalizing retrievers for surfacing valid unmodified supporting files, we use the frozen snapshot's static AST call/import graph to deterministically extract each sample's 1-hop Structural Bridge Set (BistructB^{\text{struct}}_i) connecting FigivenF^{\text{given}}_i and GiG_i. Reporting Bridge-Tolerant P@5 (treating BistructB^{\text{struct}}_i as non-penalized supporting context) lifts absolute macro P@5 across the 287-sample core by +0.11 to +0.14 (RRF: 0.274 →\rightarrow 0.412, Qwen3-8B: 0.251 →\rightarrow 0.383, Lexical: 0.236 →\rightarrow 0.361) with zero human annotation.


2. Agent Setup Clarifications & Thorough-Exploration Ablation

  • Why No Shell? In benchmarks isolating context localization from test-driven repair (e.g., localization tracks of SWE-bench, Agentless, LocAgent), read-only tools (grep, read_file, list_dir) are standard practice: they provide the exact equivalents of shell search (rg, cat, ls) while preventing test-execution leakage across static snapshots.

  • Is the 3-File Cap Shared, and What If Removed? Static retrievers (Tables 2–6) are evaluated up to top-20 / 8k tokens without a 3-file cap. The 3-file cap applies identically across all 6 agent arms (Table 7 / Appendix F) because 92.3% of samples have ≤3\le 3 gold files (median 1.0). Even if we remove the cap and score all visited files (All Read), unseeded Any-Gold reachability nudges up only from 51.1% to 55.6% (and 54.7% to 58.9% on the 287 core).

  • Ablating Prompt & Stopping Rules: To test whether short searches reflect stopping instructions rather than task difficulty, we prompted the agent to exhaustively cross-check callers/imports before submitting, enforced ≥5\ge 5 tool calls, and relaxed caps (5 submitted files, 2,400 toks/file, 16k budget):

Agent Setup (45-Sample Subset)Avg CallsRead ToksAny-Gold (All Read)Any-Gold (Submitted)File F1
Default Unseeded (3-file cap, 8k)3.712,13755.6%51.1%0.3222
Thorough Prompt + Relaxed Cap (≥5\ge 5 calls, 5-file, 16k)8.644,89260.0%55.6%0.3385
RRF-Seeded (Default prompt, 3-file, 8k)5.421,857 (+3,299 seed)84.4%80.0%0.3967

More than doubling search steps (3.71 →\rightarrow 8.64 calls) raises submitted Any-Gold by only +4.5 pp (51.1% →\rightarrow 55.6%) and F1 by +0.0163, because extra lexical grep calls still fail when de-anchored queries (comment2context, code2test) share no exact identifier strings with the target. By contrast, an RRF seed bridges this semantic/structural gap immediately (80.0% Any-Gold, 0.3967 F1). We will include this ablation in Section 8.4.

Thank you for the further response, and I think that these are promising revision directions. However, I remain uncertain whether such substantial changes can be adequately validated during revision, since they require reassessing the task definition and ground truth. For example, useful context files may fall outside the proposed 1-hop structural bridge. The coverage and validity of the revised labels and metrics therefore need further evaluation. Given these remaining concerns, I will maintain my original scores.

Thank you again for the thoughtful discussion and for recognizing these directions as promising. To ensure complete clarity for the final assessment, we would like to clarify that our core benchmark and main experimental results would remain 100% unchanged and does not require a post-rebuttal overhaul of the dataset or task:

  1. Prose Precision, Not a Task Redefinition: Tightening our phrasing from "what to read next" to "actionable workflow context" is purely a terminological clarification in the text so that our narrative accurately describes the exact resolution-grounded labels (GiG_i) that the benchmark has always used.

  2. Primary Ground Truth Remains Unchanged; Unmodified Context Is an Appendix Diagnostic: We fully agree that useful diagnostic files can fall outside a 1-hop AST bridge. Precisely because no heuristic expansion captures every diagnostic read perfectly, GiG_i remains our sole primary benchmark ground truth. Both the 52-sample human pooling check (where 89.5%89.5\% of the 38 useful diagnostic files actually fell within 1–2 AST hops) and the deterministic bridge analysis are included strictly as auxiliary sensitivity analyses in the Appendix, demonstrating that our main benchmark findings (e.g., RRF complementarity, budget trade-offs, and agent seeding gains) are already fully validated and robust.

We sincerely appreciate your constructive engagement throughout the review process, which has helped us present the scope, interpretation, and diagnostics of our existing benchmark much more clearly.

Reviewer_vyYAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces Agent Retrieval Bench for repository context retrieval in coding workflows. The baseline results vary a lot by task. The methods also struggle with deciding when to abstain on natural no-gold cases.

Critical strengths

I liked the 45-sample pilot. I think this kind of experiment is useful. It gives some idea of how the retrieved context affects what the agent does next.

Critical weaknesses
  1. BCY is my main concern. Reading one non-header token is enough to count a gold file as covered. Say the needed function is at line 800: the agent can read the imports, never see that function, and still get credit for the file. This seems too lenient to me. The paper calls it file exposure, but the gap between finding a file and getting useful code from it is large. I don't think BCY alone tells us enough about retrieval quality.

  2. For the task itself, I am still not sure what is new. It looks quite close to code navigation, and there are already benchmarks for that. I need a clearer comparison before I can see why another benchmark is needed.

Questions for authors
  1. The writing needs more work. I found the paper difficult to follow, and the figures did not make the task and evaluation much clearer.

  2. The difference from existing code-navigation benchmarks also needs to be clearer.

  3. I would also like to see results that check whether the agent actually gets the relevant code. The relevant function or span needs to be visible, using the same token budget. That would help with my concern about BCY.

Reviewer_MuNYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper evaluates context acquisition as a separate part of a coding agent's work. It introduces 427 samples from 25 repositories: 345 positive retrieval samples and 82 no-gold or wrong-repository samples. The four positive tasks ask for tests related to an implementation change, extra context for a review comment, root-cause files for a failure trace, and files affected by an anchored edit. The evaluation includes frozen base commits, leakage checks, ranking and token-budget metrics, abstention, and agent trajectory analysis. A separate pilot uses 45 samples and one run per sample and condition to study how initial context affects file selection and exploration cost. It does not measure repair success.

Critical strengths
  1. The tasks reflect practical coding workflows. They cover test discovery, review context, root-cause retrieval, and change impact. Excluding given files and edit anchors from gold reduces obvious retrieval shortcuts.

  2. The analysis shows where retrieval methods help. Results vary across tasks and repository weighting. RRF improves retrieval by combining structural and semantic methods, while the natural no-gold tests expose limits of threshold-based abstention.

Critical weaknesses
  1. The labels do not fully match the stated retrieval goal. A file changed in a past repair is not always the file an agent should read next. Cases with no local fix may still need local diagnostic context. Independent review is needed to check how often useful files are missing from gold.

  2. BCY does not check whether useful evidence is included. It counts a gold file as exposed after just one content token. The median evidence covers only 4.7% of a gold file (Appendix G.5). A consistent-budget evaluation of the evidence spans would better support the efficiency claims.

Questions for authors
  1. What were the sample size, process, and correction rate for the independent model-assisted audit? Have independent human judges reviewed unlabeled results pooled from the retrievers' top-k lists? This would help estimate missing gold labels and the share of reasonable alternative files. Evidence linking a file to a past change and evidence that the relevance label is correct are different checks.

  2. If the common goal is to find what an agent should read next, which natural no-gold cases still need local diagnostic files? If the common goal is the final resolution target, should the positive tasks be interpreted differently as well? Could both definitions be compared on a small independently reviewed subset?

  3. Under the same tokenizer and token budget, do evidence-span exposure or recall support the main BCY rankings and the RRF gains? For the seed pilot, do the gains remain when total tokens, including seeds, are compared and a subset is run more than once? For example, Table 7 gives about 5,305 total tokens for lexical seeds and 5,156 for RRF. These totals should be considered together with their File F1 scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.