- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Foresight-SWE constructed a dataset of 212 execution-verified GitHub PR pairs: PR1 completes a task but introduces a defect, while a subsequent PR2 fixes that defect. Given the base repository for PR1, an agent must complete the task without introducing the defect (measured by the FRR metric). Across 11 models, the average task resolution rate was 60.7%, while the average FRR was only 14.8%.
Critical strengths
This paper raises a valuable question: whether models can proactively avoid potential flaws when implementing code.
Critical weaknesses
I believe the fundamental flaw of this paper is that the FRR metric may not be as meaningful or convincing as the authors intend.
-
FRR only evaluates whether an agent can avoid the specific defect that happened to be exposed in the past. We do not know if the defect fixed by PR2 was the only one introduced by PR1, nor do we know if the agent introduced entirely new defects that had never appeared before. In other words, this metric does not seem to reflect actual code quality.
-
Foresight-SWE builds its benchmark around the core concept of an agent's ability to "foresee the future," yet the training data for current models already contains that "future." The instances are drawn from 35 well-known open-source repositories and public PRs from 2021–2026; the state-of-the-art models being tested have almost certainly encountered these PRs and their corresponding fixes. However, the paper contains no discussion of measures to prevent data contamination. It does not even specify whether the agents have internet access during operation. Regarding the low FRR scores reported: are they due to the models' genuine inability to write adequate code, or are they the result of contamination from the accessible "future"?
Questions for authors
§4.4 states that proactive prompting improved Claude Opus 5's FRR by 13.43 percentage points, yet Table 3 shows 32.55 − 24.53 = 8.02; which figure is correct?