acceptodds
Under review as a conference paper at ICLR 2027

Interactive External Knowledge Acquisition for Software Engineering Agents

Abstract

Repository-level software engineering agents can inspect code and execute tests, yet useful evidence may lie outside their execution environment. Search and browsing tools expand access, but agents must decide when to retrieve, communicate local constraints, and assess external information's applicability. We present BRIDGE, a framework that separates repository-side problem solving from external evidence acquisition. A dedicated retriever provides exploratory and focused interfaces; during focused retrieval, it can request missing repository context and resume its search. An outcome-conditioned prompt-memory procedure helps a parameter-frozen agent use these interfaces, with memory constructed on a disjoint bootstrap set and fixed before evaluation. We evaluate this organization through local-only, direct-tools, and retriever-capacity comparisons. On 450 held-out SWE-bench Verified instances, BRIDGE with GPT-5-mini in both roles achieves , compared with for direct access to the same web tools with the same adaptation (three-run mean standard deviation). Protocol and adaptation ablations, temporal-policy sensitivity experiments, and cost measurements characterize the design's behavior and computational trade-offs. These experiments show how coordinated evidence acquisition can complement repository-local reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Software Engineering Agents, External Knowledge Acquisition, Multi-Agent Coordination, Retrieval-Augmented Reasoning, In-Context Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1nFhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents BRIDGE, which pairs a repository-level coding agent with a web-browsing retriever. The retriever may request additional repository context through a WAIT protocol, while an outcome-conditioned prompt memory encourages appropriate tool use. Experiments report improvements over local-only and direct-web-tool agents on SWE-bench Verified and SWE-rebench.

Critical strengths

The paper studies an important practical question and includes more controls than many agent papers, including a direct-tools baseline, matched retriever models, repeated runs, component ablations, and explicit token accounting.

Critical weaknesses
  1. Live-web evaluation on public software issues has an unresolved answer-leakage problem. An LLM content judge applied to synthesized responses cannot detect whether the retriever has read the target issue, PR, or commit and paraphrased its solution as a root cause or high-level hint. The temporal policy also relies on uncertain LLM date judgments rather than historically archived sources. Without a source-level audit, the reported gains cannot be attributed to legitimate external knowledge.

  2. The comparison with direct tools is not compute-matched. BRIDGE introduces a second agent, a separate context, and almost doubles total tokens from 34.6K to 65.6K. Equal main-model identity and outer interaction limits do not control realized retrieval calls or inference compute. Therefore, the reported advantage does not isolate the value of delegation or the proposed communication protocol.

  3. The methodological novelty is limited. “Belief alignment” is implemented as ordinary clarification and search continuation rather than a learned belief representation or alignment objective. Bootstrap-memory sensitivity, paired statistical tests, and an aggregate analysis of retrieved evidence are also missing.

Questions for authors
  1. What fraction of retrieved sources mention the target issue, PR, commit, root cause, or modified symbols, based on independent human auditing?

  2. Does BRIDGE outperform direct browsing under matched total tokens, model calls, context capacity, and browsing steps?

  3. What is the exact SWE-rebench subset, and were all accessed sources demonstrably available before each task’s cutoff date?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1nFh (1/3)

We sincerely thank the reviewer for recognizing the importance of the question and the breadth of our controls, including the direct-tools baseline, matched retriever models, repeated runs, component ablations, and explicit token accounting. We address the concerns and questions below.

W1 & Q1. Can the gains be attributed to legitimate external knowledge?

We agree that a judge applied to the synthesized answer cannot tell what the retriever read. The revision therefore adds a time-gated setting (Appendix B.3), in which a filter that acts on the sources themselves takes the place of the content judge, and an audit of the sources the retriever visited (Appendix B.4). The filter enforces a cutoff at the creation time of each task's fix PR. It dates GitHub issues, PRs, commits and dated pages by rule, date-restricts searches, refuses the fix PR and its commits (by SHA), later-numbered issues and PRs, the current branch of the target repository and benchmark mirrors, and uses an LLM (GPT-5.4) only for pages without a date. In this time-gated setting, BRIDGE stays 3.2 to 7.6 points above local-only for three backbones on both benchmarks (Table 9; see our reply to Reviewer tReN, W1). The audit covers the logged retriever visits on a random sample of 200 SWE-bench Verified tasks (GPT-5-mini as main agent and retriever, one run per setting):

Time filter (new)Submitted setting (content judge)
Visits / unique sources381 / 274784 / 554
Visits to target-repository GitHub8 (2.1%)173 (22.1%)
Unique sources mentioning gold-patch files or functions20 (7.3%)110 (19.9%)
Tasks in which the target issue thread was read3 (+6 undetermined)44 (+17 undetermined)
Sources read by an author / of these, stating the root cause or showing the fix33 / 4-

Under the time filter, the fix PR was requested once and refused, and no confirmed fix commit was visited. In total, the filter refused 29 page requests and withheld 67 search results, in 36 tasks (Table 11). For the last row, one of the authors read the logged text of the 33 sources behind the two rows above it (the 20 sources that mention gold-patch files or functions and all sources of the three tasks that read the target issue thread) next to the gold patch. Four of them state the root cause or show the fix: two pages of the target issue itself, one earlier pull request that the problem statement links, and one search-results page. Of the other 29, 23 only name a changed file or function and 6 are other pages or could not be judged. The two tasks with the target-issue pages were resolved; the other two were not.

On dating, rule-based dates cover GitHub and dated pages, and an LLM dates the remaining pages; 77 of the 381 visits were admitted after an "uncertain" LLM judgment. The overlap counts and the manual reading work from search snippets and retriever notes, so both are lower bounds. The filter therefore narrows, but does not eliminate, the possibility of exposure, and the revision states this scope.

See the revised Section 4.5 and the new Appendix B.3 and B.4, marked in blue.

AuthorsOpenReview ↗

Response to Reviewer 1nFh (2/3)

W2 & Q2. How does BRIDGE compare with direct tools once compute is taken into account?

BRIDGE's Pass@1 is about 6 points higher than that of direct web tools with adaptation, while it uses slightly fewer total tokens and main-agent steps, so its advantage is not obtained by spending more compute. Specifically, we ran the direct-tools baseline with adaptation in the time-gated setting on the random sample of 112 SWE-bench Verified tasks of the time-gated ablation (Appendix A.6), with GPT-5-mini as main agent and as BRIDGE's retriever, three runs; BRIDGE and local-only with the same memory come from the same runs as that ablation. Pass@1 in %, mean ± std over three runs, with p from Welch's t-test on the three run-level results against BRIDGE, and averages per trajectory:

MethodPass@1p-valueTotal tokens (K)Main-agent stepsWeb actions by the main agent
Local-only + memory53.0 ± 0.520.0132919.6-
Direct web tools + adaptation53.9 ± 3.610.0934319.50.38
BRIDGE59.8 ± 1.55-32818.2-

BRIDGE reaches 59.8 against 53.9 for direct tools (p = 0.09 on the three run-level results), with 328K tokens per trajectory, 13.8K of them retrieval tokens, against 343K, and 18.2 main-agent steps against 19.5. Measured in total tokens and main-agent steps, its realized compute is therefore not higher than that of direct tools. Table 2 gives the earlier comparison in the content-judge setting on SWE-bench Verified: 59.6 ± 1.37 for BRIDGE against 54.2 ± 1.21 for direct tools.

The comparison remains matched in setting, not in realized compute, as Section 4.3 states. Both arms have the same main model, web actions, adaptation procedure, tasks and outer budget: 80 main-agent steps and 3 USD of main-agent cost per trajectory (Appendix A.2), and neither is restricted in how much of this budget it uses. Every web action of the direct-tools agent is one of its steps. In BRIDGE, a retrieval call is one main-agent step; the retriever's own steps, at most 30 per call, are not charged to this budget, and its tokens are included in the totals above. Both arms average under 20 main-agent steps against the limit of 80.

Four facts describe this comparison further. The direct-tools agent rarely browses: about 8% of its trajectories in each run use a web action at all, so in practice this arm is close to a local-only agent with a different prompt; its Pass@1 is 0.9 points above local-only with the same memory (p = 0.71). The three conditions each have their own prompt template, so the comparison is between complete configurations, not a single switched component. Direct tools return more empty patches than BRIDGE, 6, 7 and 2 per run against 0 to 2; when these tasks are excluded, the gap remains. The same time filter applies to the direct-tools agent's own web actions; in these three runs it refused 1 page.

The quoted 34.6K and 65.6K come from Table 5 of the submission, which reported dollars per trajectory under a token heading. With the GPT-5 retriever of that table, local-only uses 280K tokens (0.0346 USD) and BRIDGE 338K (0.0656 USD), so 1.9× is a cost ratio and the token ratio is 1.2×. These figures also compare BRIDGE with local-only; the revised Table 5 reports tokens and cost separately, and Appendix A.6 gives the consumption of direct tools.

On browsing steps, the direct-tools agent has the same web actions as the retriever. How often to browse is what this baseline leaves to the agent, and prescribing it would test a different system. Both main agents have the same context window; the retriever's separate context is part of the organization under study.

The no-WAIT ablation tests the communication protocol separately (see our reply to W3).

See the revised Table 5 and Appendix A.2 and A.6, marked in blue.

AuthorsOpenReview ↗

Response to Reviewer 1nFh (3/3)

W3. What is new beyond clarification and search continuation?

BRIDGE does not claim a learned belief representation or an alignment objective. Section 3.2 already states that "belief-aligned" describes communicating the main agent's current hypotheses and repository constraints, not a separately learned belief-state model. Mechanically, WAIT is a clarification request combined with search continuation. We present it as a supporting mechanism of the retriever and not as a component of equal weight, because its ablation difference lies within run-to-run variation (see our reply to Reviewer tReN, W3). The revised Section 4.4 presents it this way.

Our contribution is a controlled study of delegated retrieval for repository-level repair inside one scaffold. The retriever sees web sources and the context the main agent sends, but not the repository. It offers an exploratory and a focused retrieval interface, together with a protocol for requesting missing context and resuming a search. A prompt memory adapts when the main agent calls the retriever and what it sends. The evaluation includes a time-gated setting, in which a source- and date-based filter is designed to admit only information published before each task's fix PR. We do not present BRIDGE as a replacement for subagent frameworks.

The aggregate analysis of retrieved evidence is now in the paper: a source audit of the logged retriever visits on 200 SWE-bench Verified tasks, with a domain breakdown and a manual reading of the 33 flagged sources (Appendix B.4; see our replies to W1 & Q1 and to Reviewer tReN, Q2). For the ablations, the revision reports statistical tests on the run-level results in the time-gated setting (see our reply to Reviewer tReN, W3).

See the revised Section 4.4 and the new Appendix B.4, marked in blue.

Q3. Which SWE-rebench subset is used, and were all sources available before each task's cutoff?

We use one fixed subset of 81 SWE-rebench tasks for all comparisons: the first 81 instances of the filtered split of nebius/SWE-rebench on Hugging Face (revision 89cdfbab), in dataset order. The revised Section 4.1 states this rule.

The time-gated SWE-rebench results (Table 9, Appendix B.3) use the same filter and the same cutoff rule as on SWE-bench Verified: the creation time of each task's fix PR. Availability before the cutoff is enforced by the filter's rules (see our reply to W1 & Q1). An automatic audit of one SWE-rebench run made without the time filter (GPT-5 retriever) finds no fix PR, fix commit or benchmark mirror among its 67 logged sources, which come from 8 tasks. As on SWE-bench Verified, the filter narrows, but does not eliminate, the possibility of exposure, and the revision states this scope in Appendix B.3 and B.4.

See the revised Section 4.1 and the new Appendix B.3 and B.4, marked in blue.

Reviewer_ZKgyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce BRIDGE, which is a multi-agent framework. Instead of letting the main agent use web search tools directly, they use a dedicated retriever to gather external information. This also helps the retriever and the main agent communicate and work together more effectively. They also introduce an interaction protocol called WAIT, along with a mechanism for handling and sharing context between agents.

In their experiments, BRIDGE improves performance on SWE tasks across different models, including GPT-5-mini, Gemini, and Claude. This shows that the framework is effective and can work with different models.

Critical strengths
  • The paper identifies an important limitation of existing SWE agents: simply giving them web search tools is not always enough. To address this, the authors introduce the WAIT protocol. It allows the retriever to ask the main agent for missing repository context and continue searching with more relevant information. This is a practical way to improve communication between agents.

  • Another strength is the outcome-conditioned prompt memory. The authors extract useful tool-use strategies by comparing successful and failed trajectories. This helps the agent make better decisions about when and how to use external knowledge. More importantly, it improves performance without fine-tuning or updating the model weights.

Critical weaknesses
  • The experiments lack comparisons with existing subagent-based approaches. Frameworks like Claude Code and Codex already support subagents for specialized tasks, including information retrieval. Although the authors compare BRIDGE with direct web search and provide a no-WAIT ablation, they do not compare it with existing subagent-based systems. Such comparisons would help better demonstrate the novelty and effectiveness of BRIDGE over related approaches.

  • The generalization of the learned prompt memory could be further explored. Although the authors evaluate BRIDGE on held-out SWE-bench tasks and SWE-rebench, it is still unclear how well the extracted strategies can transfer to tasks from very different repositories or distributions. It would also be useful to investigate how sensitive the results are to the selection of bootstrap tasks. More experiments on zero-shot transfer could help better demonstrate the generalizability of the approach.

Questions for authors
  • Could the authors provide more details about how WAIT is used in practice? For example, how often is it triggered to request missing repository context, and how often is it used to continue a search due to budget limits? It would also be interesting to see how these two cases contribute to the performance improvement.

  • BRIDGE uses around 1.9× more tokens than the local-only baseline, despite reducing the number of repository interaction steps. Have the authors measured the actual inference latency and monetary cost, especially compared with the direct web-tools baseline? This would help better understand the practical trade-offs of the framework.

1 comment
AuthorsOpenReview ↗

Response to Reviewer ZKgy

We sincerely thank the reviewer for recognizing that web search tools alone are not always enough for SWE agents, the WAIT protocol as a practical way to improve communication between agents, and the outcome-conditioned prompt memory, which improves performance without updating model weights. We address the concerns and questions below.

W1. How does BRIDGE relate to subagent systems such as Claude Code and Codex?

BRIDGE does not claim that delegating retrieval to a subagent is new: Section 1 presents delegation as an existing organization, and the revised Section 5 now discusses the subagents of Claude Code and Codex. These products are complete systems with their own models, prompts, tools and budgets, so a difference in Pass@1 against them could not be attributed to how retrieval is organized. Our comparisons are therefore made inside a single scaffold, with the same main model and tasks. The controlled analogue of a generic retrieval subagent is our no-WAIT variant, in which the main agent sends one request and receives one answer. Pass@1 in %, mean ± std over three runs (GPT-5-mini, content-judge setting of Tables 2 and 3):

VariantSWE-bench VerifiedSWE-rebench
Local-only52.8 ± 1.6942.8 ± 1.23
Direct web tools + adaptation54.2 ± 1.2142.8 ± 0.71
Retrieval subagent without WAIT56.7 ± 1.5746.2 ± 1.38
BRIDGE59.6 ± 1.3748.1 ± 2.14

The subagent without WAIT is 2.5 and 3.4 points above direct tool access, and BRIDGE is a further 2.9 and 1.9 points above it; this last difference is within run-to-run variation (see our reply to Reviewer tReN, W3). We therefore position BRIDGE as a controlled study of delegated retrieval for repository-level repair, covering the retrieval interfaces, the adaptation of the main agent and a time-gated evaluation, and not as a replacement for subagent frameworks.

See the revised Section 5, marked in blue.

W2. Does the prompt memory transfer beyond its bootstrap tasks?

The memory is built from 50 SWE-bench Verified tasks, which are excluded from evaluation, and it is used unchanged on SWE-rebench, so SWE-rebench is a transfer setting for the memory. With it, BRIDGE is 5.3 points above local-only there in the time-gated setting (48.1 ± 2.47, Table 9, against 42.8 ± 1.23, Table 1). The separate contribution of the memory is measured on SWE-bench Verified. Pass@1 in %, mean ± std over three runs on a random sample of 112 tasks (GPT-5-mini, time-gated setting, Appendix A.6):

VariantSWE-bench Verified
BRIDGE59.8 ± 1.55
BRIDGE without memory56.5 ± 2.25

Without the memory, Pass@1 is 3.3 points lower (see our reply to Reviewer tReN, W3, for the statistical tests).

See the new Appendix A.6, marked in blue.

Q1. How is WAIT used in practice?

WAIT has two uses: the retriever asks the main agent for repository context, such as a version or a stack trace, or it hands back a search that exceeded one call's budget so that the search can be continued. The no-WAIT ablation removes both uses together and so measures their combined contribution: without WAIT, Pass@1 is 57.1 ± 1.79, against 59.8 ± 1.55 for full BRIDGE in the time-gated setting (see our reply to Reviewer tReN, W3). We therefore describe WAIT as a supporting mechanism of the retriever.

See the revised Section 4.4, marked in blue.

Q2. What are the latency and monetary cost?

We first correct Table 5: the submitted table reported the cost per trajectory in dollars under a token heading, so the 1.9× figure is a cost ratio and the token ratio is 1.2×. Averages per trajectory (content-judge setting, GPT-5 retriever):

MethodTotal tokens (K)Cost (USD)Main-agent steps
Local-only2800.034621.23
BRIDGE3380.065617.77

BRIDGE costs about three cents more per task, almost all of it on the retrieval side (main-agent cost is 0.0357 USD against 0.0346 USD; retrieval with the content judge adds 0.0299 USD), and takes 16.3% fewer main-agent steps. In the time-gated setting with a GPT-5-mini retriever, on a random sample of 112 SWE-bench Verified tasks (three runs, Appendix A.6), BRIDGE uses 328K tokens and 18.2 main-agent steps per trajectory, against 343K and 19.5 for direct tools with adaptation and 329K and 19.6 for local-only with the same memory.

On SWE-bench Verified, the median wall-clock time per task is 93.6 s for local-only and 89.3 s for time-gated BRIDGE with a GPT-5-mini retriever (86.8 s with a GPT-5 retriever). Wall-clock time depends on API load at the time of a run, so these figures are indicative.

See the revised Table 5, Section 4.6 and Appendix A.6 and E, marked in blue.

Reviewer_tReNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BRIDGE (Belief-aligned Retrieval via Interactive Dialogue with Grounded Evidence)—a framework for agents in the field of software engineering (SWE) that decouples the problem-solving process within a local codebase from the gathering of external evidence. The system consists of a primary agent working in coordination with a specialized retrieval agent; the latter can access the Internet via two interfaces: OVERVIEW_SEARCH and FOCUSED_RETRIEVAL. A core component of the communication protocol is the "WAIT" mechanism. This mechanism allows the retrieval agent to pause its web-browsing process to request missing local contextual information from the primary agent, such as library versions, stack traces, or configuration details, or to seek authorization to continue a task after reaching a step limit, before resuming the search from a previously visited URL. To assist the primary agent (with fixed parameters) in deciding when to employ tools and what context to provide, the paper proposes an in-context reinforcement process based on the outcomes achieved.

Critical strengths

The paper addresses a practical problem: the debugging process often requires information that lies outside the source code repository. The paper defines "internal" and "external" based on accessibility to establish a distinct boundary between what an agent is permitted to access and what constitutes an "information leak." The experimental section introduces a valuable baseline, granting the agent direct access to web search tools through the same fine-tuning process. On the SWE-bench Verified dataset, BRIDGE outperforms this baseline by 5.4 points. The evaluation process is comprehensive: the authors test across three families of foundation models and two benchmark datasets, and also includes a control setup where the same model is used for both the agent and the retrieval system (Table 7), alongside thorough ablation studies. Furthermore, the authors acknowledge about the study's limitations: openly disclose all prompts used for content evaluation and acknowledge that simply removing code diffs is insufficient to definitively claim that improvements stem solely from external knowledge (Appendix B.1); additionally, they treat the "temporal policy" as a sensitivity analysis rather than a fully validated evaluation method; and the paper provides a detailed list of bootstrap tasks and specifies which differences are not statistically significant (Appendix E).

Critical weaknesses

W1. The possibility of answer leakage via a live data retrieval mechanism is not ruled out. The system does not block GitHub or its mirror sites; in fact, the prompt explicitly prioritizes the use of the "official GitHub" repository. Furthermore, the evaluation criteria permit the use of "high-level source context" and "broad hints". Appendix B.1 also acknowledges that identifying the location of a bug and its root cause "can already provide significant information about the solution." Content appearing after the point in time when a bug was introduced is relatively common (a 17.6% filtering rate, per §4.5); however, a temporal check was performed in only one experimental setup (Table 4). Moreover, when the GPT-5 model acts as a retrieval agent, it may inadvertently leak information from its own internal memory. Please document the URLs accessed, measure the degree of overlap with the "gold patch," re-run the experiments in Table 4 with GitHub access blocked, and apply a temporal policy—ensuring that only information available prior to the bug's introduction is accessible—across the relevant configurations and the SWE-rebench dataset.

W2. The current baselines fail to isolate the actual contribution of the proposed method. Half of the performance gain is attributable to "prompt memory"—a component absent in the "local-only" approach. This implies that the combination of data retrieval and the WAIT mechanism yields an improvement of only +3.0 / +2.8 points (p ≈ 0.08 / 0.06, Table 3). Furthermore, the "direct-tools" baseline is not clearly described; it is not computationally equivalent (involving 30.0k retrieved tokens, as shown in Table 5, compared to the context used in the proposed method), and on the SWE-rebench benchmark, it performs no better than the "local-data-only" approach. The "local-data-only" approach itself significantly underperforms compared to publicly reported results using bash commands (56.4% / 58.3% versus ≥73.1% / 62.9%), while no information is provided regarding the inference effort, the number of steps taken, or the computational cost constraints. Please apply the same distilled memory to the "local-data-only" approach, provide a detailed description and ensure computational equivalence for the "direct-tools" baseline, and report results based on standard configurations.

W3. The role of WAIT is unclear. Welch tests give p ≈ 0.08 / 0.28 for removing WAIT and p ≈ 0.03 / 0.18 for the memory, yet the text claims "each component contributes" and "reliably improves"; App. E itself calls the memory ablation "descriptive." WAIT does two things (asks for local context; continues past a call's budget), and the no-WAIT ablation removes both while holding only overall limits fixed. Please add per-task paired tests, report WAIT types and browsing steps/tokens, and add a no-WAIT variant with a matched per-call budget.

Questions for authors

Please clarify and address weaknesses. Besides, answer the following questions:

What is the average number of tokens consumed and web-browsing actions performed by the "Direct web tools + adaptation" baseline? Furthermore, do these web-browsing steps count directly toward the primary agent’s "outer interaction budget"?

Please provide a breakdown of the domains for all URLs returned by the retrieval component across the 450 samples in SWE-bench Verified and SWE-rebench. What percentage of queries access github.com (or the documentation/release notes pages of the target repository itself)?

Please provide the full content of the final memory M\mathcal{M} obtained after executing the BUILD_MEMORY algorithm (Listing 2). Also, specify the number of valid trajectory pairs in P(0)\mathcal{P}^{(0)} across the 50 bootstrap samples.

4 comments
AuthorsOpenReview ↗

Response to Reviewer tReN (1/3)

We sincerely thank the reviewer for the careful reading and for recognizing the practical problem we study, the access-based information boundary, the direct-tools baseline, the breadth of the evaluation, and our disclosure of the paper's limitations. We address the questions below.

W1. How is answer leakage through live retrieval controlled?

We agree that the temporal restriction should not be limited to one setting: in the submission, the Table 1 runs used the GPT-5.4 content judge (Appendix B.1), and the temporal check, with its 17.6% intervention rate, applied only to the Table 4 setting. The revision therefore adds a time-gated setting (Appendix B.3) and re-runs Table 1 in it for all three backbones on both benchmarks (Table 9). Time-gated BRIDGE stays 3.2 to 7.6 points above local-only and within 0.6 points of the submitted numbers.

Specifically, the agent is placed at the time of the human fix: the cutoff is the creation time of the fix PR (the dataset's created_at on both benchmarks). Earlier public information, including GitHub, is admissible, as the developer would consult it; the filter is designed to exclude later information, within the limits stated at the end. We do not cut at the bug's introduction, because the issue report and its discussion come after it. The time-gated setting uses this cutoff in place of the "problem's publication date" of the submitted temporal check. In this setting, a source- and date-based filter takes the place of the content judge, which admitted "broad hints". The filter dates GitHub issues, PRs, commits and dated pages by rule, removes post-cutoff comments from lower-numbered target-repository issues and PRs, date-restricts searches and refuses benchmark mirrors; an LLM dates the remaining undated pages. Pass@1 in %, mean ± std over three runs (SWE-bench Verified / SWE-rebench):

Main agentLocal-only (Table 1)BRIDGE, submitted (Table 1)BRIDGE, time-gated (new, Table 9)Gain over local-only
GPT-5-mini52.8 ± 1.69 / 42.8 ± 1.2359.4 ± 1.32 / 48.3 ± 1.23 (GPT-5 retriever)*59.7 ± 2.36 / 48.1 ± 2.47+6.9 / +5.3
Gemini-3-Flash56.4 ± 1.61 / 43.2 ± 2.1463.8 ± 1.25 / 47.3 ± 1.36 (GPT-5 retriever)64.0 ± 2.65 / 46.9 ± 1.23+7.6 / +3.7
Claude-Haiku-4.558.3 ± 1.18 / 45.8 ± 1.5262.1 ± 1.37 / 50.2 ± 1.44 (GPT-5-mini retriever)61.5 ± 1.50 / 49.8 ± 2.57+3.2 / +4.0

All time-gated runs use a GPT-5-mini retriever, so for the first two rows the comparison with the submitted column is across retrievers. *Submitted with a GPT-5-mini retriever: 59.6 ± 1.37 / 48.1 ± 2.14 (Table 7).

Thus, the improvement is retained under the time filter. Using GPT-5-mini for every retriever also removes the stronger-retriever confound: with the GPT-5-mini main agent, both roles are one model, so the retriever has no parametric knowledge that the local-only agent lacks.

The audit below (Appendix B.4, Table 10) covers the logged retriever visits, and our answer to Q2 gives the domain breakdown.

200 SWE-bench Verified tasks, GPT-5-mini, one run per settingTime filter (new)Submitted setting (GPT-5.4 content judge)
Visits / unique sources381 / 274784 / 554
Tasks with at least one retrieval49 / 20071 / 200
Visits to target-repository GitHub8 (2.1%)173 (22.1%)
Sources / tasks mentioning gold-patch files or functions20 / 12110 / 51

Under the time filter, 8 of 381 visits (2.1%) go to the target repository's GitHub, against 173 of 784 (22.1%) in the submitted setting, and 20 sources in 12 tasks mention files or functions changed by the gold patch, against 110 sources in 51 tasks. The target issue thread, whose pre-cutoff comments are admissible by definition, was read in 3 tasks under the time filter (6 undetermined), against 44 (17 undetermined). These counts come from retriever visits; in the submitted setting, the content judge then replaced the answer in 37 of 78 retrieval calls.

Pre-cutoff GitHub, which the main-agent prompt prefers as an official source (Listing 4), is part of what a developer at the cutoff would use, so the time-gated setting keeps it and restricts potentially answer-bearing sources by source and date: the filter refused the fix PR once, 2 commit pages whose date or SHA could not be verified, 10 views of the target repository's current branch and 4 later-numbered issues or PRs.

The filter therefore narrows, but does not eliminate, the possibility of exposure, and the revision states this scope: 77 of the 381 visits were admitted after an "uncertain" LLM judgment, and the gold-patch overlap is taken from search snippets and retriever notes, so it is a lower bound.

See the revised Section 4.5 and the new Appendix B.3 and B.4, marked in blue.

AuthorsOpenReview ↗

Response to Reviewer tReN (2/3)

W2. Do the baselines isolate the contribution of BRIDGE?

The memory is a component of BRIDGE, not an external advantage of one arm: it is distilled from contrastive bootstrap trajectories, collected with and without forced retrieval, and is prepended to the main agent's prompt. The comparison that holds it fixed is direct tools against BRIDGE, since both use the same adaptation procedure, and there BRIDGE is 5.4 and 5.3 points higher. To answer the request directly, we also ran local-only with the same memory. Pass@1 in %, mean ± std over three runs (content-judge setting of Tables 2 and 3; the new row is in Appendix A.6):

MethodSWE-bench VerifiedSWE-rebench
Local-only52.8 ± 1.6942.8 ± 1.23
Local-only + memory (new)53.2 ± 2.2543.6 ± 2.57
Direct web tools + adaptation54.2 ± 1.2142.8 ± 0.71
BRIDGE without memory55.8 ± 1.4845.6 ± 1.41
BRIDGE59.6 ± 1.3748.1 ± 2.14

With the memory, the local-only agent reaches 53.2 and 43.6, against 52.8 and 42.8 without it; differences of 0.4 and 0.8 points lie within run-to-run variation. The memory alone therefore does not help the local-only agent, and its effect comes through the retriever. Compared at equal memory, BRIDGE is 6.4 and 4.5 points above local-only.

The direct-tools baseline is a matched-setting comparison, not a matched-compute one. It has the same main model, web actions, adaptation procedure, tasks and outer budget as BRIDGE, and its main agent decides itself whether and how often to browse; each web action counts as one of its steps (see Q1). Neither arm is restricted in how much of this budget it uses, so the difference in realized compute is an outcome of the two organizations rather than of what they were given. In the time-gated setting, on the random sample of 112 SWE-bench Verified tasks of Appendix A.6 (three runs), direct tools with adaptation reach 53.9 ± 3.61 and local-only with the same memory 53.0 ± 0.52, against 59.8 ± 1.55 for BRIDGE (p = 0.09 and 0.01, Welch's t-test on the three run-level results), and BRIDGE uses slightly fewer total tokens than direct tools there (see Q1). On SWE-rebench, direct tools stay at the local-only level (42.8 against 42.8): access to the tools is not sufficient by itself, and BRIDGE addresses how they are used. Section 4.3 states this design: the comparison matches tool access and adaptation, while realized computation may differ because tool use is agent-controlled.

See the new Appendix A.6, marked in blue.

W3. What does WAIT contribute, and are the ablation differences established?

We agree. What our runs establish is the comparison of full BRIDGE with local-only; the component ablations do not establish separate contributions, and WAIT has the smaller and less certain effect of the two components. We have therefore replaced "each component contributes" (Section 4.4 and the Table 3 caption) and "reliably improves" (Section 4.2) with descriptive statements, and we now present WAIT as a supporting mechanism that lets the retriever hand a search back and resume it, not as an equally weighted source of the gain. Pass@1 in %, mean ± std over three runs on a random sample of 112 SWE-bench Verified tasks, time-gated setting (Appendix A.6, Table 8); p-values are from Welch's t-test on the three run-level results against full BRIDGE:

VariantPass@1p-value
BRIDGE59.8 ± 1.55
without WAIT57.1 ± 1.790.12
without memory56.5 ± 2.250.12
Local-only + memory53.0 ± 0.520.01

Local-only without memory reaches 52.8 ± 1.69 on all 450 tasks (Table 1).

Without WAIT, Pass@1 is 2.7 points below full BRIDGE, and without memory 3.3 points below; neither difference is established at three runs. BRIDGE is 6.8 points above local-only with the same memory.

The reviewer is also right that WAIT has two functions and that the no-WAIT ablation removes both; the ablation therefore measures their combined effect, as Section 4.4 states.

See the revised Section 4.4 and the new Appendix A.6, marked in blue.

AuthorsOpenReview ↗

Response to Reviewer tReN (3/3)

Q1. What does the direct-tools baseline consume, and do its browsing steps count toward the outer budget?

Yes, they count. In the direct-tools baseline the main agent issues every web action itself, and each one is a main-agent step under the same outer budget as its repository actions: 80 main-agent steps and 3 USD of main-agent cost per trajectory. In BRIDGE, a retrieval request is one main-agent step; the retriever's own steps, at most 30 per call, run in its own context, are not charged to this budget and are reported as retrieval tokens. On average, the direct-tools agent performs 0.38 web actions per trajectory. Averages per trajectory on the random sample of 112 SWE-bench Verified tasks of Appendix A.6 (time-gated setting, GPT-5-mini as main agent and retriever, three runs), tokens in thousands:

MethodMain-agent tokensRetrieval tokensTotal tokensMain-agent stepsWeb actions by the main agent
Local-only + memory329-32919.6-
Direct web tools + adaptation343-34319.50.38
BRIDGE31413.832818.2-

With the same tools available, the direct-tools agent rarely browses: about 8% of its trajectories in each run use a web action at all. BRIDGE uses 328K tokens per trajectory, retrieval included, against 343K for direct tools and 329K for local-only with the same memory, and takes 18.2 main-agent steps against 19.5 and 19.6. All three average under 20 main-agent steps against the limit of 80.

Separately, Table 5 of the submission reported the cost per trajectory in dollars under a token heading; the revised Table 5 gives tokens and cost separately (see our reply to Reviewer ZKgy, Q2).

See the revised Table 5 and Appendix A.6, marked in blue.

Q2. Which domains do the retrieved URLs come from?

Under the time filter, 15 of 381 retriever visits (3.9%) go to github.com, 8 of them to the target repository; in the submitted setting, github.com accounts for 198 of 784 visits (25.3%). The table gives the domain classes of the logged retriever visits on 200 SWE-bench Verified tasks (GPT-5-mini as main agent and retriever, one run per setting), under the time filter and in the submitted setting; shares are percentages of visits:

Domain classTime filter (new)Submitted setting
Target-repository GitHub8 (2.1%)173 (22.1%)
Other GitHub7 (1.8%)25 (3.2%)
Project documentation, trackers, release notes46 (12.1%)175 (22.3%)
Search pages113 (29.7%)243 (31.0%)
Question-and-answer sites87 (22.8%)30 (3.8%)
Other120 (31.5%)138 (17.6%)
Total visits381784

Under the time filter, 46 of 381 visits (12.1%) go to project documentation, trackers and release notes, against 175 of 784 (22.3%) in the submitted setting.

See Table 10 in the new Appendix B.4, marked in blue.

Q3. What is the final memory, and how many valid pairs does P(0)\mathcal{P}^{(0)} contain?

The initial contrastive set P(0)\mathcal{P}^{(0)} contains 11 valid pairs from the 50 bootstrap tasks, that is, tasks where the forced-retrieval trajectory succeeds, the free trajectory fails, and the forced trajectory uses external retrieval (Listing 2). The 50 bootstrap tasks are listed in Table 6 and are excluded from evaluation. The memory M\mathcal{M} distilled from these pairs is prepended to the main agent's prompt and is fixed before evaluation; Appendix A.3 gives its construction and refinement procedure.

Reviewer_tReNOpenReview ↗

Response to the rebuttal

Thank you for the thorough response. The time-gated re-run, the source audit and the memory control address part of my concerns, and the more cautious claims about WAIT and the memory are appropriate. Several issues remain. The local-only baseline is still well below publicly reported results for the same models with standard scaffolds (56.4% and 58.3% vs. 73.1% and 62.9%), and the response does not address this, so it is unclear whether the gains carry over to a strong agent. The time filter narrows but does not remove exposure: 12 tasks still retrieved sources mentioning gold-patch files or functions, and 77 visits were admitted after an uncertain judgment. On the 112-task time-gated sample, the advantage over direct web tools is not significant (p = 0.09 over three runs), and the direct-tools agent performs only 0.38 web actions per trajectory, so this comparison says little about the retriever design itself. Finally, the matched-budget no-WAIT variant was not run, and the content of the final memory that I asked for is not given in the response. I will keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reading. Several comments concern one point, the task setting in which a repository agent may use the live web, so we state it once here. The agent is placed at the time of the human fix: each task's cutoff is the creation time of its fix PR (the dataset's created_at on both benchmarks). Earlier public information is admissible, including GitHub: the issue report, its earlier discussion, earlier issues and PRs, documentation and release notes are what a developer fixing the issue at that time could consult. Not admissible are the fix PR and its commits, later issues and PRs, the repository's current branch, benchmark mirrors and later discussion of the issue. We neither cut at the bug's introduction nor block GitHub altogether, because the issue report and its discussion come after the bug and are part of the developer's situation. This setting matches what a developer has at hand when fixing the issue, and it is the setting our evaluation targets.

The results in the submission already applied leakage controls in this spirit: a content judge screens returned answers for patches, exact edits and complete fix recipes, and one configuration adds a temporal check. The revision adds a time-gated setting (Appendix B.3) that enforces the cutoff at the source level, with a source- and date-based filter applied during retrieval. With Table 1 re-run under it for all three backbones on both benchmarks, BRIDGE stays 3.2 to 7.6 points above local-only and within 0.6 points of the submitted numbers, so the improvement reported in the submission is retained. An audit of the sources the retriever visited (Appendix B.4) documents what was accessed, and the revision states the scope of the filter.

Comparisons hold the setting fixed: the compared methods have the same main model, tasks, web actions where applicable and outer budget, and none is restricted in how much of this it uses. Differences in realized compute therefore reflect how each method uses the same resources. The results under this setting, the source audit and the individual points are in our replies to each reviewer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.