- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Agent Retrieval Bench evaluates how coding agents find repository context across test discovery, code review, failure localization, change impact, and abstention. It contains 427 samples from 25 repositories, using frozen snapshots and task-specific evidence. The main contribution is the workflow coverage and evaluation protocol. Validation includes lexical, structural, and embedding retrieval, interactive agent search, trajectory analysis, and a 45-sample seed intervention.
Critical strengths
-
The task breakdown is useful. Covering these scenarios together captures an important aspect of practical context acquisition. The substantial performance differences across tasks support evaluating them separately.
-
The evaluation goes beyond a retrieval leaderboard. Frozen snapshots, evidence links, and direct leakage checks make the results easier to interpret. Budget, exploration, and abstention analyses also reveal trade-offs that ranking metrics alone would miss.
Critical weaknesses
-
The labels may capture only part of the useful context. Historical edits and task-specific evidence provide reasonable targets, but files that need changing may be only a subset of files worth understanding. High recall against these labels can overstate coverage of the context actually needed, while useful unlabelled outputs reduce measured precision. The same concern applies to abstention: no local fix target does not imply no useful local diagnostic context.
-
The short agent searches and modest scores need more explanation. Closed-tool Codex averages only 3.82 tool calls and reaches 0.3279 File F1. Such limited exploration seems somewhat at odds with what I would expect from everyday coding-agent use. It is unclear whether this reflects task difficulty or the prompts, tool interface, and stopping behavior.
Questions for authors
-
Did the audit examine useful but unchanged files? Is there evidence showing how missing relevance labels affect method rankings and natural-case abstention results?
-
What causes the agent searches to stop after so few calls? Could a small set of low-scoring searches be followed by coding attempts with outcomes assessed (tests or manual eval), to check whether low retrieval scores correspond to actual coding failures?
4 comments
Thank you very much for engaging in this discussion with us! Constructive dialogue is the best way to sharpen a work, and we truly value this exchange. Below, we address your two remaining concerns concisely.
1. Objective Framing ("Actionable Workflow Context") & Deterministic Score Calibration
We fully agree that (a) manual evaluation is not a suitable standard for benchmark scoring, and (b) absolute scores must align with an objective task definition (our 52-sample human check in Round 1 was purely a diagnostic reply, not a benchmark scoring method). We are making two deterministic revisions:
-
Reframing to Actionable Workflow Context Retrieval: Unlike "what to read next" (which implies subjective browsing order), the unseen files required to execute the workflow action (, verified by resolving commits/tests) are 100% objective ground truth. We will revise the paper to frame the task strictly as retrieving Actionable Workflow Context—making recall and
BCYagainst exact, unbiased measures of actionable target coverage. -
Deterministic AST-Graph Calibration for Absolute Precision (Zero Manual Labeling): To avoid penalizing retrievers for surfacing valid unmodified supporting files, we use the frozen snapshot's static AST call/import graph to deterministically extract each sample's 1-hop Structural Bridge Set () connecting and . Reporting Bridge-Tolerant P@5 (treating as non-penalized supporting context) lifts absolute macro P@5 across the 287-sample core by +0.11 to +0.14 (
RRF: 0.274 0.412,Qwen3-8B: 0.251 0.383,Lexical: 0.236 0.361) with zero human annotation.
2. Agent Setup Clarifications & Thorough-Exploration Ablation
-
Why No Shell? In benchmarks isolating context localization from test-driven repair (e.g., localization tracks of SWE-bench, Agentless, LocAgent), read-only tools (
grep,read_file,list_dir) are standard practice: they provide the exact equivalents of shell search (rg,cat,ls) while preventing test-execution leakage across static snapshots. -
Is the 3-File Cap Shared, and What If Removed? Static retrievers (Tables 2–6) are evaluated up to top-20 / 8k tokens without a 3-file cap. The 3-file cap applies identically across all 6 agent arms (Table 7 / Appendix F) because 92.3% of samples have gold files (median 1.0). Even if we remove the cap and score all visited files (
All Read), unseededAny-Goldreachability nudges up only from 51.1% to 55.6% (and 54.7% to 58.9% on the 287 core). -
Ablating Prompt & Stopping Rules: To test whether short searches reflect stopping instructions rather than task difficulty, we prompted the agent to exhaustively cross-check callers/imports before submitting, enforced tool calls, and relaxed caps (5 submitted files, 2,400 toks/file, 16k budget):
| Agent Setup (45-Sample Subset) | Avg Calls | Read Toks | Any-Gold (All Read) | Any-Gold (Submitted) | File F1 |
|---|---|---|---|---|---|
| Default Unseeded (3-file cap, 8k) | 3.71 | 2,137 | 55.6% | 51.1% | 0.3222 |
| Thorough Prompt + Relaxed Cap ( calls, 5-file, 16k) | 8.64 | 4,892 | 60.0% | 55.6% | 0.3385 |
| RRF-Seeded (Default prompt, 3-file, 8k) | 5.42 | 1,857 (+3,299 seed) | 84.4% | 80.0% | 0.3967 |
More than doubling search steps (3.71 8.64 calls) raises submitted Any-Gold by only +4.5 pp (51.1% 55.6%) and F1 by +0.0163, because extra lexical grep calls still fail when de-anchored queries (comment2context, code2test) share no exact identifier strings with the target. By contrast, an RRF seed bridges this semantic/structural gap immediately (80.0% Any-Gold, 0.3967 F1). We will include this ablation in Section 8.4.
Thank you for the further response, and I think that these are promising revision directions. However, I remain uncertain whether such substantial changes can be adequately validated during revision, since they require reassessing the task definition and ground truth. For example, useful context files may fall outside the proposed 1-hop structural bridge. The coverage and validity of the revised labels and metrics therefore need further evaluation. Given these remaining concerns, I will maintain my original scores.
Thank you again for the thoughtful discussion and for recognizing these directions as promising. To ensure complete clarity for the final assessment, we would like to clarify that our core benchmark and main experimental results would remain 100% unchanged and does not require a post-rebuttal overhaul of the dataset or task:
-
Prose Precision, Not a Task Redefinition: Tightening our phrasing from "what to read next" to "actionable workflow context" is purely a terminological clarification in the text so that our narrative accurately describes the exact resolution-grounded labels () that the benchmark has always used.
-
Primary Ground Truth Remains Unchanged; Unmodified Context Is an Appendix Diagnostic: We fully agree that useful diagnostic files can fall outside a 1-hop AST bridge. Precisely because no heuristic expansion captures every diagnostic read perfectly, remains our sole primary benchmark ground truth. Both the 52-sample human pooling check (where of the 38 useful diagnostic files actually fell within 1–2 AST hops) and the deterministic bridge analysis are included strictly as auxiliary sensitivity analyses in the Appendix, demonstrating that our main benchmark findings (e.g., RRF complementarity, budget trade-offs, and agent seeding gains) are already fully validated and robust.
We sincerely appreciate your constructive engagement throughout the review process, which has helped us present the scope, interpretation, and diagnostics of our existing benchmark much more clearly.
Re
Thanks for your response. These are helpful, but my main concerns are not yet fully resolved:
The labels still do not fully align with "what the agent should read next." After human auditing, unchanged method rankings do not resolve the substantial bias in absolute scores, which also matters for a benchmark. I'm not sure if there is an objective ground truth, but manual evaluation is not a suitable standard approach for such benchmark.
I remain concerned about the restricted agent setup. Is removing shell access standard practice? Is the three-file submission cap common for all methods? The agent should be encouraged to gather sufficient evidence before submitting. It remains unclear whether the short searches reflect task difficulty or the interface and stopping instructions.