- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper presents BRIDGE, which pairs a repository-level coding agent with a web-browsing retriever. The retriever may request additional repository context through a WAIT protocol, while an outcome-conditioned prompt memory encourages appropriate tool use. Experiments report improvements over local-only and direct-web-tool agents on SWE-bench Verified and SWE-rebench.
Critical strengths
The paper studies an important practical question and includes more controls than many agent papers, including a direct-tools baseline, matched retriever models, repeated runs, component ablations, and explicit token accounting.
Critical weaknesses
-
Live-web evaluation on public software issues has an unresolved answer-leakage problem. An LLM content judge applied to synthesized responses cannot detect whether the retriever has read the target issue, PR, or commit and paraphrased its solution as a root cause or high-level hint. The temporal policy also relies on uncertain LLM date judgments rather than historically archived sources. Without a source-level audit, the reported gains cannot be attributed to legitimate external knowledge.
-
The comparison with direct tools is not compute-matched. BRIDGE introduces a second agent, a separate context, and almost doubles total tokens from 34.6K to 65.6K. Equal main-model identity and outer interaction limits do not control realized retrieval calls or inference compute. Therefore, the reported advantage does not isolate the value of delegation or the proposed communication protocol.
-
The methodological novelty is limited. “Belief alignment” is implemented as ordinary clarification and search continuation rather than a learned belief representation or alignment objective. Bootstrap-memory sensitivity, paired statistical tests, and an aggregate analysis of retrieved evidence are also missing.
Questions for authors
-
What fraction of retrieved sources mention the target issue, PR, commit, root cause, or modified symbols, based on independent human auditing?
-
Does BRIDGE outperform direct browsing under matched total tokens, model calls, context capacity, and browsing steps?
-
What is the exact SWE-rebench subset, and were all accessed sources demonstrably available before each task’s cutoff date?
3 comments
Response to Reviewer 1nFh (2/3)
W2 & Q2. How does BRIDGE compare with direct tools once compute is taken into account?
BRIDGE's Pass@1 is about 6 points higher than that of direct web tools with adaptation, while it uses slightly fewer total tokens and main-agent steps, so its advantage is not obtained by spending more compute. Specifically, we ran the direct-tools baseline with adaptation in the time-gated setting on the random sample of 112 SWE-bench Verified tasks of the time-gated ablation (Appendix A.6), with GPT-5-mini as main agent and as BRIDGE's retriever, three runs; BRIDGE and local-only with the same memory come from the same runs as that ablation. Pass@1 in %, mean ± std over three runs, with p from Welch's t-test on the three run-level results against BRIDGE, and averages per trajectory:
| Method | Pass@1 | p-value | Total tokens (K) | Main-agent steps | Web actions by the main agent |
|---|---|---|---|---|---|
| Local-only + memory | 53.0 ± 0.52 | 0.01 | 329 | 19.6 | - |
| Direct web tools + adaptation | 53.9 ± 3.61 | 0.09 | 343 | 19.5 | 0.38 |
| BRIDGE | 59.8 ± 1.55 | - | 328 | 18.2 | - |
BRIDGE reaches 59.8 against 53.9 for direct tools (p = 0.09 on the three run-level results), with 328K tokens per trajectory, 13.8K of them retrieval tokens, against 343K, and 18.2 main-agent steps against 19.5. Measured in total tokens and main-agent steps, its realized compute is therefore not higher than that of direct tools. Table 2 gives the earlier comparison in the content-judge setting on SWE-bench Verified: 59.6 ± 1.37 for BRIDGE against 54.2 ± 1.21 for direct tools.
The comparison remains matched in setting, not in realized compute, as Section 4.3 states. Both arms have the same main model, web actions, adaptation procedure, tasks and outer budget: 80 main-agent steps and 3 USD of main-agent cost per trajectory (Appendix A.2), and neither is restricted in how much of this budget it uses. Every web action of the direct-tools agent is one of its steps. In BRIDGE, a retrieval call is one main-agent step; the retriever's own steps, at most 30 per call, are not charged to this budget, and its tokens are included in the totals above. Both arms average under 20 main-agent steps against the limit of 80.
Four facts describe this comparison further. The direct-tools agent rarely browses: about 8% of its trajectories in each run use a web action at all, so in practice this arm is close to a local-only agent with a different prompt; its Pass@1 is 0.9 points above local-only with the same memory (p = 0.71). The three conditions each have their own prompt template, so the comparison is between complete configurations, not a single switched component. Direct tools return more empty patches than BRIDGE, 6, 7 and 2 per run against 0 to 2; when these tasks are excluded, the gap remains. The same time filter applies to the direct-tools agent's own web actions; in these three runs it refused 1 page.
The quoted 34.6K and 65.6K come from Table 5 of the submission, which reported dollars per trajectory under a token heading. With the GPT-5 retriever of that table, local-only uses 280K tokens (0.0346 USD) and BRIDGE 338K (0.0656 USD), so 1.9× is a cost ratio and the token ratio is 1.2×. These figures also compare BRIDGE with local-only; the revised Table 5 reports tokens and cost separately, and Appendix A.6 gives the consumption of direct tools.
On browsing steps, the direct-tools agent has the same web actions as the retriever. How often to browse is what this baseline leaves to the agent, and prescribing it would test a different system. Both main agents have the same context window; the retriever's separate context is part of the organization under study.
The no-WAIT ablation tests the communication protocol separately (see our reply to W3).
See the revised Table 5 and Appendix A.2 and A.6, marked in blue.
Response to Reviewer 1nFh (3/3)
W3. What is new beyond clarification and search continuation?
BRIDGE does not claim a learned belief representation or an alignment objective. Section 3.2 already states that "belief-aligned" describes communicating the main agent's current hypotheses and repository constraints, not a separately learned belief-state model. Mechanically, WAIT is a clarification request combined with search continuation. We present it as a supporting mechanism of the retriever and not as a component of equal weight, because its ablation difference lies within run-to-run variation (see our reply to Reviewer tReN, W3). The revised Section 4.4 presents it this way.
Our contribution is a controlled study of delegated retrieval for repository-level repair inside one scaffold. The retriever sees web sources and the context the main agent sends, but not the repository. It offers an exploratory and a focused retrieval interface, together with a protocol for requesting missing context and resuming a search. A prompt memory adapts when the main agent calls the retriever and what it sends. The evaluation includes a time-gated setting, in which a source- and date-based filter is designed to admit only information published before each task's fix PR. We do not present BRIDGE as a replacement for subagent frameworks.
The aggregate analysis of retrieved evidence is now in the paper: a source audit of the logged retriever visits on 200 SWE-bench Verified tasks, with a domain breakdown and a manual reading of the 33 flagged sources (Appendix B.4; see our replies to W1 & Q1 and to Reviewer tReN, Q2). For the ablations, the revision reports statistical tests on the run-level results in the time-gated setting (see our reply to Reviewer tReN, W3).
See the revised Section 4.4 and the new Appendix B.4, marked in blue.
Q3. Which SWE-rebench subset is used, and were all sources available before each task's cutoff?
We use one fixed subset of 81 SWE-rebench tasks for all comparisons: the first 81 instances of the filtered split of nebius/SWE-rebench on Hugging Face (revision 89cdfbab), in dataset order. The revised Section 4.1 states this rule.
The time-gated SWE-rebench results (Table 9, Appendix B.3) use the same filter and the same cutoff rule as on SWE-bench Verified: the creation time of each task's fix PR. Availability before the cutoff is enforced by the filter's rules (see our reply to W1 & Q1). An automatic audit of one SWE-rebench run made without the time filter (GPT-5 retriever) finds no fix PR, fix commit or benchmark mirror among its 67 logged sources, which come from 8 tasks. As on SWE-bench Verified, the filter narrows, but does not eliminate, the possibility of exposure, and the revision states this scope in Appendix B.3 and B.4.
See the revised Section 4.1 and the new Appendix B.3 and B.4, marked in blue.
Response to Reviewer 1nFh (1/3)
We sincerely thank the reviewer for recognizing the importance of the question and the breadth of our controls, including the direct-tools baseline, matched retriever models, repeated runs, component ablations, and explicit token accounting. We address the concerns and questions below.
W1 & Q1. Can the gains be attributed to legitimate external knowledge?
We agree that a judge applied to the synthesized answer cannot tell what the retriever read. The revision therefore adds a time-gated setting (Appendix B.3), in which a filter that acts on the sources themselves takes the place of the content judge, and an audit of the sources the retriever visited (Appendix B.4). The filter enforces a cutoff at the creation time of each task's fix PR. It dates GitHub issues, PRs, commits and dated pages by rule, date-restricts searches, refuses the fix PR and its commits (by SHA), later-numbered issues and PRs, the current branch of the target repository and benchmark mirrors, and uses an LLM (GPT-5.4) only for pages without a date. In this time-gated setting, BRIDGE stays 3.2 to 7.6 points above local-only for three backbones on both benchmarks (Table 9; see our reply to Reviewer tReN, W1). The audit covers the logged retriever visits on a random sample of 200 SWE-bench Verified tasks (GPT-5-mini as main agent and retriever, one run per setting):
Under the time filter, the fix PR was requested once and refused, and no confirmed fix commit was visited. In total, the filter refused 29 page requests and withheld 67 search results, in 36 tasks (Table 11). For the last row, one of the authors read the logged text of the 33 sources behind the two rows above it (the 20 sources that mention gold-patch files or functions and all sources of the three tasks that read the target issue thread) next to the gold patch. Four of them state the root cause or show the fix: two pages of the target issue itself, one earlier pull request that the problem statement links, and one search-results page. Of the other 29, 23 only name a changed file or function and 6 are other pages or could not be judged. The two tasks with the target-issue pages were resolved; the other two were not.
On dating, rule-based dates cover GitHub and dated pages, and an LLM dates the remaining pages; 77 of the 381 visits were admitted after an "uncertain" LLM judgment. The overlap counts and the manual reading work from search snippets and retriever notes, so both are lower bounds. The filter therefore narrows, but does not eliminate, the possibility of exposure, and the revision states this scope.
See the revised Section 4.5 and the new Appendix B.3 and B.4, marked in blue.