acceptodds
Under review as a conference paper at ICLR 2027

Do Coding Agents Catch Their Own Bugs? (They Don't)

38 views

Abstract

Real software development requires engineers to foresee and fix potential problems with their code that are not explicitly specified in requirements—such as pre- emptively considering vulnerabilities or other bugs from a code change. But current benchmarks often do not measure these future-looking consequences. We introduce **Foresight-SWE**, a benchmark for measuring whether coding agents can anticipate and address unstated problems during software engineering tasks. Foresight-SWE consists of 212 samples spanning 4 languages and 35 popular open-source repositories. Each sample contains two pull requests (PRs), the second of which resolves defects the first one introduced. To measure whether agents exhibit foresight, we define the **foresight resolve rate (FRR)** as the fraction of instances where an agent fully resolves the original task while also avoiding the defect identified by a later PR. We evaluate eleven large language models and find that FRR ranges from only 11.32% to 24.53%, largely failing to address problems introduced by the first PR change. This raises important questions about how models are trained and aligned in coding contexts, as well as how code review tools should be improved for more fail-safe coding. Foresight-SWE shifts evaluations in this direction.

Primary area:
datasets and benchmarks
Keywords:
coding agents, large language models, software engineering, defect prediction, bug prediction, benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_GM37OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Foresight-SWE constructed a dataset of 212 execution-verified GitHub PR pairs: PR1 completes a task but introduces a defect, while a subsequent PR2 fixes that defect. Given the base repository for PR1, an agent must complete the task without introducing the defect (measured by the FRR metric). Across 11 models, the average task resolution rate was 60.7%, while the average FRR was only 14.8%.

Critical strengths

This paper raises a valuable question: whether models can proactively avoid potential flaws when implementing code.

Critical weaknesses

I believe the fundamental flaw of this paper is that the FRR metric may not be as meaningful or convincing as the authors intend.

  1. FRR only evaluates whether an agent can avoid the specific defect that happened to be exposed in the past. We do not know if the defect fixed by PR2 was the only one introduced by PR1, nor do we know if the agent introduced entirely new defects that had never appeared before. In other words, this metric does not seem to reflect actual code quality.

  2. Foresight-SWE builds its benchmark around the core concept of an agent's ability to "foresee the future," yet the training data for current models already contains that "future." The instances are drawn from 35 well-known open-source repositories and public PRs from 2021–2026; the state-of-the-art models being tested have almost certainly encountered these PRs and their corresponding fixes. However, the paper contains no discussion of measures to prevent data contamination. It does not even specify whether the agents have internet access during operation. Regarding the low FRR scores reported: are they due to the models' genuine inability to write adequate code, or are they the result of contamination from the accessible "future"?

Questions for authors

§4.4 states that proactive prompting improved Claude Opus 5's FRR by 13.43 percentage points, yet Table 3 shows 32.55 − 24.53 = 8.02; which figure is correct?

Reviewer_7GfJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that coding agents often satisfy explicit requirements while missing bugs their changes could introduce. It presents Foresight-SWE, a benchmark of 212 tasks across 35 repositories and four programming languages, built from pairs of real pull requests: an initial change and a later fix for a defect it introduced. Agents receive only the original task and are tested on both its requirements and the unstated defect. Across 11 models, agents complete the original task 60.7% of the time on average, but complete it while also avoiding the later defect only 14.8% of the time. In a separate experiment with five models, explicitly describing the defect raises repair success to 91.2%, suggesting that recognizing hidden failure modes is the main bottleneck. Proactive prompting and self-review help some models, but can also introduce regressions. The central implication is that coding agents need training and evaluation that reward anticipating consequences beyond the stated requirement

Critical strengths

A key strength is the paper’s realistic evaluation of software engineering foresight. By pairing an actual code change with a later fix for a defect it introduced, Foresight-SWE tests whether agents can anticipate problems beyond explicit requirements. Its 212 tasks span 35 repositories and four languages, extending coverage beyond a single defect category. Execution checks strengthen the benchmark by verifying that the later defect exists after the original change and that the corresponding fix resolves it.

Another strength is the experimental separation of defect recognition from repair ability. Testing agents both with and without explicit defect descriptions provides evidence that identifying hidden problems is a larger bottleneck than implementing known fixes. The proactive prompting and self-review experiments also expose a useful tradeoff: these interventions can improve anticipation while reducing success on the original task. Together, the experiments offer practical guidance for evaluating coding agents on both task completion and defect prevention.

Critical weaknesses

The paper overlooks relevant literature, weakening its positioning and experimental baselines. ChangeGuard detects unintended behavioral changes, including those introduced by LLM refactoring, while SlopCodeBench studies how agents’ implementation decisions affect subsequent software evolution. CoverUp combines coverage feedback with automated test generation. These approaches address different settings and do not invalidate Foresight-SWE’s contribution, but they motivate comparisons beyond prompting and native self-review. Without testing applicable methods that use execution feedback or generated tests, the paper provides limited evidence about whether stronger verification workflows could address the observed failures.

The benchmark’s metrics provide a narrower assessment than the paper’s framing suggests. FRR combines successful task implementation with avoidance of one historically documented defect, so failures cannot automatically be attributed to poor anticipation. Conversely, passing these tests does not establish that an agent avoided other defects. Probe to Generate demonstrates why this distinction matters: stronger tests can reject patches previously accepted by software repair benchmarks. Historical attribution introduces additional uncertainty, with the paper’s small co-author audit finding approximately 9% of sampled pairs debatable. Independent auditing, stronger test-adequacy checks, and reporting anticipation conditional on successful task completion would strengthen the conclusions.

The experiment separating defect recognition from repair does not fully isolate the proposed bottleneck. Agents repairing an explicitly described defect start from the later PR2 base, whereas agents anticipating it start before PR1 and must also implement the original change. This changes the starting code, available information, and required work simultaneously. A stronger comparison would reveal the defect while holding the starting state and implementation task constant. Different model harnesses and budgets, together with the absence of reported uncertainty from repeated runs, also limit interpretation of model rankings and intervention gains. The results demonstrate a practical performance gap, but establish its cause less decisively than the paper suggests.

  1. ChangeGuard: Validating Code Changes via Pairwise Learning-Guided Execution
  2. SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
  3. CoverUp: Effective High Coverage Test Generation for Python
  4. Probe to Generate: Program Variant-Guided Test Augmentation for Repository-Level Repair Benchmarks
Questions for authors

What fraction of agents’ successfully completed tasks also avoid the later defect? Could you report this conditional rate alongside FRR to separate implementation failures from anticipation failures?

How do you verify that anticipation tests accept alternative correct implementations and specifically expose the historical defect?

How sensitive are the findings to questionable PR attributions? Would excluding ambiguous pairs or expanding independent human validation change the conclusions?

Do the model rankings and prompting improvements remain consistent across repeated runs and comparable computational budgets?

Have you examined whether proactive prompting and self-review introduce new defects beyond those covered by the existing tests?

Reviewer_z4bBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Foresight-SWE, a benchmark evaluating whether coding agents can complete a software task while avoiding unstated defects identified by later fixes. It constructs 212 tasks from 35 repositories and evaluates eleven models.

Critical strengths
  1. Interesting and practical perspective: Evaluating whether agents anticipate defects beyond explicit requirements addresses an important aspect of real-world software development.
  2. Useful construction pipeline: The paper extracts paired changes from real repositories and validates them through test execution, providing a useful contribution to benchmark construction.
Critical weaknesses
  1. Inconsistent annotation procedure: Appendix B and the supplied severity-labeling prompt describe different procedures. This leaves the provenance of the severity labels and the reproducibility of the stratified analysis unclear.
  2. Insufficient explanation of evaluation results: The substantial differences in original-task resolution rates are difficult to interpret given different harnesses and budgets. The absence of per-instance results, trajectories, and aggregate failure statistics limits verification of the reported findings.
Questions for authors
  1. Could you explain the large resolution-rate gap between Qwen3.8 Flash Next and Opus 5? Representative execution trajectories and aggregate failure statistics would help interpret these results.
  2. Table 3 shows substantial drops in resolution rate after proactive prompting or self-review. Do these reflect regressions caused by additional edits, or run-to-run variability? Please provide supporting statistics and representative trajectories, including before/after test results for self-review.
  3. Appendix B describes severity labels as computed from extracted attributes, whereas the supplied severity_judge.j2 asks the model to assign severity directly. Which procedure was actually used to produce the dataset labels?
Reviewer_vSapOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper focuses on the measuring the ability of coding agents to identify and address software defects or possible issues that have not occurred in the first pull request. They introduce foresight resolve rate (FRR), which is the fraction of instances where the agent resolves the original task while avoiding future defects or issues. They specifically focus on evaluating an agent's own implementation using tests from real-world fixes.

To this end, they built a automated pipeline for building a dataset, Foresight-SWE, to measure whether coding agents are capable of foreseeing unstated issues within a specific pull request. Using Foresight-SWE, they evaluate 11 models, 5 closed models and 6 open-weight models, and find that these models are worse at foreseeing issues than fixing them and show that prompting is not a reliable way of improving foresight.

Critical strengths

The paper's main strength would be the introduction of a metric for implying how well models can address future issues and their way of building this metric is grounded in and supported by prior works in the same field. I like that this metric addresses a key gap in evaluating coding agents on the quality of their generated work from the perspective of how future proof the generated fix is. Foresight-SWE is also a strong contribution as the dataset covers multiple languages and is thoroughly collected with a clear criterion that supports their primary task

Critical weaknesses

Although the paper address a key gap in research and is well-written and methodical, there are still key points that still need to be addressed. One is that the basis for the pipeline that built Foresight-SWE is primarily dependent upon SZZ (which is addressed in the appendix), but this technique has known weaknesses and there have been many variants that improve upon the original (such as PR-SZZ: https://arxiv.org/abs/2206.09967?utm_source=chatgpt.com). Another issue is that Foresight-SWE mentions it is built upon 35 repositories, but there is no mention of what kind of repositories are used and what domain they cover. This should be included (at least in the appendix) as we have no idea what kind of software engineering tasks the dataset is built upon.

Another thing I think that needs to be addressed is how usable is FRR outside of 2 pull requests? The study mainly focuses on the scope of PR 1 and PR 2, but what about issues that intertwine across multiple pull requests? How useful is FRR for these kinds of issues where PR 1 introduces something that is fixed in PR 3 but PR 2 also contributes to the same issue in PR 3 and is not caught until a later PR?

You attempted prompting as a way to improve FRR, but why weren't other techniques considered? In your prompting, you also did not attempt to use FRR as a means to improve the results. Then besides measuring foresight in models, what else can FRR be used for?

Sidenote: Not important and does not impact my assessment of the paper, but there is a missing space in section 3.2 "2025).Critical".

Questions for authors
  1. Why was SZZ chosen and not other variations of it?
  2. Was LLM-as-Judge considered instead of SZZ for identifying PR 1 and PR 2?
  3. What kind of repositories were used to construct Foresight-SWE? What was the criteria for said repos?
  4. The study mainly focuses on the scope of PR 1 and PR 2, but what about issues that intertwine across multiple pull requests?
  5. What is the justification for the models chosen (DeepSeek V4 Flash and GLM 5.3 Flash) for validating the dataset? Why weren't code models considered?
  6. You attempted prompting as a way to improve FRR, but why weren't other techniques considered to improve FRR?
  7. Are there other metrics that are comparable to FRR?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.