acceptodds
Under review as a conference paper at ICLR 2027

PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix

Abstract

Proactive bug discovery is an important software-engineering problem. Even though coding agents are already widely deployed in this domain, their capabilities remain understudied. In particular, the challenging task of finding all bugs that human maintainers would repair across an entire repository has not been studied yet. To close this gap, we introduce PROBE, a benchmark consisting of all 2,457 bugs fixed over the course of 18 months across 20 diverse GitHub repositories, together with a method for matching candidate findings to this partial reference set. We evaluate coding agents across frontier models and find they all achieve a recall of below 21% and a precision of less than 9% with respect to the reference set. While stronger models increase recall substantially from 8% to 17%, they do not improve precision. Surprisingly, despite the low precision, LLM and human evaluators still judge 85% of findings valid. Crucially, we find that found bug distributions are similar across agents but differ substantially from those of bugs fixed by maintainers. The practical relevance of bug discovery and the low recall of frontier models make this benchmark a valuable resource for both model users and developers.

Primary area:
datasets and benchmarks
Keywords:
Bug Finding, Coding Agents, LLMs, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QfntOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper „PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix” consists of two parts. The first part is concerned with the collection of a new bug data set. Here, the authors propose a multi-stage agentic pipeline which analyzes commits first to identify if the fix a bug, then tries to locate the origin of the bug. Using this approach, they collect bug data for twenty projects with about 2500 bugs total. The second part makes use of this data to test if AI agents can find all bugs within a repository. They use a simple splitting mechanism that groups the files such that the context window of the models is not exhausted during the analysis of code. The results show that both recall and precision are not very good with respect to the collected data (both around 20%), but that the low precision is rather an artifact of the incomplete bug labeling because these bugs were not fixed and therefore not labeled within the data set.

Critical strengths
  • Whole repository review is an interesting and important topic and analyzing how complete such a review is, as well as if multiple rounds of review by the same agent could be useful, is an important topic.
  • Results were collected with contemporary models and show that based on this data and approach, a pure AI-based code review for quality assurance is likely not sufficient.
Critical weaknesses

Low data quality

The collected data is likely very noisy and while the own manual analysis clearly showed this shortcoming, as well as that the manual analysis itself is actually of low quality, this is ignored and rather (partially) buried in the appendix. Reporting between the main body and the appendix regarding this seems to disagree with respect to the data. To be honest, this is just bad science.

Concretely: The bottom of Page 3 contains says that 100 samples were validated and that agreement between the LLM and human judge was “barely exceeds chance” and as a conclusion. The text shows that of 100 bugs, the human flagged 12 as not really bugs, the LLM 17, with only six overlapping, i.e., assuming the superset we would have 23% of the bugs not being bugs. Instead of dealing with this noise somehow, the decision was to “just keep all data”, i.e., fully accepting this noise without even accounting for this in any of the subsequent interpretations of the results.

Table 15 in the appendix is presumably related to the same validation, but is for 180 samples, instead of 100. Moreover, apparently there was only a single human judge and this judge was unsure whether the data actually contained a bug for 43/180 = 24% of the data. This indicates that the actual noise rate in the data is likely higher and also that a single rater is not sufficient to establish the data quality.

These problems with the data quality undercut all results and cannot be easily fixed, without using different methods and/or reframing the analysis. Additionally, the severity of this issue together with how this handles does not raise my confidence in the other manual validation, i.e., the validation that shows that precision is not actually low.

Ignores two decades of literature

The paper ignores basically two decades of MSR literature on how to collect bug data, including recent agentic approaches that go well beyond the seminal, but very dated papers by Bird et al. and Kim et al. that are referenced.

Questions for authors
  1. Is my assessment regarding the noise wrong and the actual noise rate lower and the human expertise of a single judge sufficient to establish the quality of the LLM judges?
  2. How does the bug collection to more recent MSR literature, including LLM-based and agentic SZZ variants.
4 comments

Q1: Is the reference set too noisy to support the results, and why do you keep all bugs?

We thank the reviewer for highlighting the opportunity to discuss the effect of label noise: the noise rate is about 4% to 10%, and all results hold when the rejected bugs are removed. For the full analysis, see the general response.

The 23% is the union of the human and the LLM rejections on a stratified sample (failures oversampled), in which 40 of the 100 bugs came from a set that an earlier LLM audit had flagged. Weighted to the reference set, the human annotator rejects 8.4%, mostly borderline cases between a bug fix and a feature request (5.5%). Even at 23%, the conclusions would hold: if 23% of the reference bugs were invalid and no agent had matched any of them, the best recall would rise from 20.8% to 27.0%, and precision would not change.

Q2: Is Table 15 part of the reference-set audit, and is a single rater who is unsure about 24% of the findings sufficient?

No, Table 15 (Table 24 in the revised version) does not concern the reference set. It validates the LLM panel that judges agent-reported bugs, and we changed its caption to say so. We agree that a single rater is a limitation: we added a second human rating and now report a validity rate of 78% instead of 85%.

Table 15 compares one author's blind verdicts with the panel's verdict on 180 agent findings (20 for each of nine configurations). The reference-set audit mentioned by the reviewer is unrelated and concerns a separate sample of 100 bugs from the reference set.

The 43 unsure findings are those for which the author could not determine with confidence whether the agent-reported bug is a real defect. Even if all 43 are counted as invalid, the author alone, without any LLM, confirms 103 of the 180 findings (57.2%). On the 131 findings for which both the author and the LLM panel reach a verdict, they agree in 93% of the cases: the panel confirms all 103 findings that the author confirms, but also 9 of the 28 that the author rejects. Correcting the 84.9% that the panel confirms to the more conservative human judgement (Rogan-Gladen correction, sensitivity 100%, specificity 68%) gives a validity rate of 77.7% (95% CI 68.9 to 83.3). Assuming a sensitivity below 100% would only raise this estimate. Both 57.2% and 77.7% remain far above the precision with respect to the reference set, which is below 9%. We now report 78% in the abstract, the introduction and Section 4.2, with the details in Appendix C.3.

A second human rating is consistent with this (Table 26): the maintainer of one included repository judged 28 of 36 sampled unmatched findings (77.8%) to be real bugs and would merge a fix for 21 of them, again far above the precision of below 9%.

Q3: How does your approach compare to recent MSR literature, including LLM-based and agentic SZZ approaches?

We thank the reviewer for pointing out our relationship to recent MSR work. We added two related-work paragraphs that review this literature and our relationship to it [1-14].

Relation to MSR and SZZ. PROBE builds on the MSR literature but cannot use an existing method unchanged, for three reasons.

First, SZZ and its variants [1-3], including recent LLM-based and agentic ones [4-6], aim to identify a bug-introducing commit by tracing removed and modified lines. In PROBE, 13.2% of bug fixes only add lines. Standard SZZ cannot process these fixes. Extensions fall back to tracing surrounding lines [5, 6] or the enclosing code block [7]. These heuristics yield many false positives [7] and useful candidates for only about half of such fixes [5]. The agentic variants [5, 6] then let an agent search beyond these candidates, which is similar to what our step 4 does. Even so, they report F1 scores of 0.47 to 0.79 [5] and 0.56 to 0.82 [6] across datasets, and [6] recalls only 64% of such add-only fixes. Identifying the introducing commit thus remains error-prone.

Second, some bugs have no well-defined bug-introducing commit at all, for example when an external API or a requirement changes [8]. We leverage a weaker property that remains well-defined in these cases: whether the bug was present at the snapshot T0. We integrate SZZ [2] as evidence to determine this property. The MSR work closest to our setting labels code as defective at a release based on fixes made after a release [9]. PROBE adopts the criterion established by this line of work, that labels must reflect whether a bug actually affected a release [10, 11].

Third, the release-labeling methods [9-11] label whole files or classes rather than individual bugs, and they rely on affected-version fields in issue trackers such as Jira. Even in Apache projects, these fields are unusable for 51% of defects [11], and the GitHub issues and commits used in PROBE do not necessarily have such fields. Known problems of keyword heuristics [10, 12] and tangled commits [13, 14] motivated steps 1 and 2 of our pipeline: we classify each commit from its patch, message, issues, and pull requests, and split it into separate bugs (4,296 commits yield 4,763 candidate bugs).

AG-SZZ vs. our setup. An independent reanalysis with GPT-6.1 Sol under the protocol of the reference-set audit agrees to keep 94.9% of the bugs our agent keeps despite AG-SZZ dating them after T0. Of those candidates dated before T0 and dropped by our agent, it would keep only 26.7%. A reference set built with AG-SZZ alone would thus contain more than 17% of bugs that the audit rejects, compared to 3.5% for ours.

[1] Śliwerski et al., When Do Changes Induce Fixes?, MSR '05
[2] Kim et al., Automatic Identification of Bug-Introducing Changes, ASE '06
[3] Neto et al., The Impact of Refactoring Changes on the SZZ Algorithm: An Empirical Study, SANER '18
[4] Tang et al., LLM4SZZ: Enhancing SZZ Algorithm with Context-Enhanced Assessment on Large Language Models, ISSTA '25
[5] Shi et al., AgenticSZZ: Temporal Knowledge Graph-Guided Agentic Bug-Inducing Commit Identification, arXiv '26
[6] Lyu et al., AgentSZZ: Teaching the LLM Agent to Play Detective with Bug-Inducing Commits, arXiv '26
[7] Sahal and Tosun, Identifying Bug-Inducing Changes for Code Additions, ESEM '18
[8] Rodríguez-Pérez et al., How Bugs Are Born: A Model to Identify How Bugs Are Introduced in Software Components, EMSE '20
[9] Zimmermann et al., Predicting Defects for Eclipse, PROMISE '07
[10] Yatish et al., Mining Software Defects: Should We Consider Affected Releases?, ICSE '19
[11] Vandehei et al., Leveraging the Defects Life Cycle to Label Affected Versions and Defective Classes, TOSEM '21
[12] Herbold et al., Problems with SZZ and Features: An Empirical Study of the State of Practice of Defect Prediction Data Collection, EMSE '22
[13] Herzig and Zeller, The Impact of Tangled Code Changes, MSR '13
[14] Herbold et al., A Fine-Grained Data Set and Analysis of Tangling in Bug Fixing Commits, EMSE '22

Q4: Are you planning to help the maintainers fix your found bugs before dataset publication?

Yes. However, submitting the reports in bulk would shift a significant triage effort to the volunteer maintainers. The 69,352 agent-reported bugs concern almost two-year-old snapshots, and code that has often changed significantly since. For the findings that we confirmed manually and that are still present in the current code, we will notify the maintainers with a response window of 90 days, and will offer support to fix them. Note that PROBE does not target security vulnerabilities and we observe that only around 2% of the findings are security-related (Appendix D.3). We will withhold those from the public release until they have been resolved.

Dear Reviewer Qfnt,

As the discussion phase draws to an end, we kindly ask you to acknowledge our response.

In particular, both key concerns are based on misunderstandings, as summarized below:

  • Data quality: Table 15 validates agent findings, not the reference set. The 23% comes from a sample that oversampled LLM-flagged bugs, and the weighted rate is 8.4%, which three further LLM audits confirm (4–10%). All results hold even when we remove every bug that any auditor flags.
  • Prior work: We build on SZZ directly and now discuss recent agentic variants in detail. Our pipeline gives much higher data quality than SZZ alone (3.5% vs. >17% of bugs rejected by an independent audit).

We hope this addresses your concerns and are happy to answer any follow-up questions.

Thank you for this extensive update and re-write the validation. This is now much improved and clarifies the noise sources and the amount of noise within the analysis. I agree that given the updated descriptions, the remaining noise is justifiable and while it might impact the results somewhat, core conclusions likely hold - at least for this generation of models. Therefore, I will raise my score.

Reviewer_HPNLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Bug detection is a critical task in software engineering (SE). Although coding agents have been widely adopted for SE tasks, their bug-detection capabilities on real-world projects remain underexplored. To address this challenge, this paper presents PROBE, a benchmark covering all 2,457 bugs from 20 diverse GitHub repositories. Evaluations across frontier coding agents show that all attain recall below 21% and precision below 9%.

In my opinion, the benchmark built for practical SE problems carries high practical value. It can expose coding agents' weaknesses in bug detection and facilitate future improvements.

Critical strengths
  1. This work investigates a timely topic. Coding agents are a major research focus in software engineering, and bug detection is a fundamental task in that field.

  2. The benchmark includes 20 Python, JavaScript, and TypeScript repositories, offering good representativeness.

  3. The appendix provides detailed experimental information and discussions from multiple perspectives.

  4. Figures and layouts are carefully organized, greatly improving readability.

Critical weaknesses
  1. The software engineering community already has mature techniques for tracing bug-fixing commits back to their corresponding bug-inducing commits. SZZ is the dominant approach in this line of research. It is unclear why the authors chose an LLM-based pipeline instead of SZZ. The paper lacks a comparison between the proposed design and SZZ, including their respective pros and cons. For instance, what advantages does the LLM-based method offer? Traditional SZZ relies on keyword matching for bug-fixing commits, such as patterns like fix, bug fix, or bug identifiers. Why is this strategy abandoned here? How does the LLM-based approach affect precision or recall when identifying bug-fixing commits? Overall, the process appears to lack human validation, and the above questions remain unanswered.

  2. How do the authors verify whether bugs identified by LLMs match those in the reference set? The authors use both rule-based and LLM-based matching strategies, yet they do not justify either approach's reliability with manual checks. For example, the rule-based method uses Jaccard similarity based on line overlap; is the 0.02 threshold too low?

  3. A large set of LLMs is deployed throughout the pipeline. The rationale for assigning different roles to these LLMs is not clearly explained. For example, Gemini 3.5 Flash serves as the LLM-based Judge in Section 2.3 for evaluation, while GPT-5.6 Luna and Gemini 3.5 Flash are used again in Independent Bug Validation. Does this create undesired reuse of Gemini 3.5 Flash?

Questions for authors
  1. Why not use SZZ? What advantages does the proposed LLM-based method have over SZZ?

  2. What are the precision and recall for bug-fix commit identification and bug matching (as described in Weaknesses 1 and 2)? I strongly suggest adding manual checks to ensure the actual precision and recall.

2 comments

Q1: Why do you identify bug-fix commits with an LLM instead of keyword matching, and what are the precision and recall of this step?

Keyword and issue-label rules would miss 22.1% of our reference bugs, and our classifier rejects 30.4% of the commits that such keywords select.

We compare two rule-based methods with our classifier: a standard list of keywords [1] and issue labels of linked issues and pull requests. The keyword rule selects the fix commit of 71.8% of the 2,457 reference bugs, the issue labels that of 20.6%, and both together that of 77.9%. Conversely, our classifier rejects 1,368 of the 4,499 commits that contain a keyword. While our classifier was based on GPT-5.6 Terra, we use GPT-6.1 Sol for a second, independent analysis. GPT-6.1 Sol agrees with our classifier on 91.4% of these commits, which it also considers not a bug fix. Conversely, it confirms 90.1% of the 1,165 commits that only the classifier selects as bug fixes.

Weighted to all 11,753 classified commits and measured against GPT-6.1 Sol, the classifier reaches a precision of 92.6% and a recall of 88.6%, compared to 67.7% and 67.8% for the keyword rule. We have added a corresponding discussion to Appendix A.6.

[1] Ray et al., A Large Scale Study of Programming Languages and Code Quality in Github, FSE 2014

Q2: Why do you not use SZZ to decide whether a bug was present at the snapshot? What does the agent add?

We do use SZZ: AG-SZZ runs on every fix, and a coding agent verifies its result on the snapshot. An independent audit rejects 3.5% of the resulting reference bugs, compared to more than 17% for AG-SZZ alone. The agent is in particular crucial for the 13.2% of candidates whose fix only adds lines, where standard SZZ has nothing to trace. For a full comparison, please refer to our answer to Q2 in the general response.

Q3: How are findings matched to reference bugs, and how was the matching validated?

We first would like to note that the matching itself is only performed by an LLM. The deterministic rules are only used as a deliberately conservative pre-filter to reduce the cost of the LLM judge, and tuned for high recall. The judge itself is validated against human expert labels (98.8% precision, 97.6% recall).

Manual validation. We manually validated the recall of the deterministic rules in our initial submission, and described it in Section 2.3. On 307 findings from three repositories with 67 expert-validated matches, the filter retains all 67 matches among 17,177 pairs while rejecting 97.3% of the pairs. We now state this more prominently.

Second judge (new). We used GPT-6.1 Sol to judge 500 sampled pairs (250 accepted, 250 rejected) with the same prompt. It confirms 93.2% of the accepted pairs and overturns 1.2% of the rejected ones. Weighted by the shares of accepted and rejected pairs, the two judges agree on 98.4% of all pairs. Relative to GPT-6.1 Sol, our judge thus reaches a precision of 93.2% and a recall of 84.1%. The recall estimate rests on the 3 rejected pairs that GPT-6.1 Sol overturns.

Manual adjudication (new). One author labeled the 20 pairs on which the two judges disagree (17 accepted and 3 rejected by our judge), blind to both verdicts. Our judge is right on 11 of the 17 (3 wrong, 3 unsure) and 2 of the 3. These 20 pairs are the hardest of the 500 by construction. If we count the 233 matches both judges accept as correct, our judge's precision on its 250 sampled matches is 97.6–98.8%, in line with the 98.8% against human labels above.

Q4: Which model is used for which step and why? Does using Gemini 3.5 Flash both as matching judge and as panel judge introduce a bias?

We chose each model by capability, cost, and availability, and validated each step against human labels or a stronger model. We find no evidence for bias from reusing Gemini 3.5 Flash.

Overview (new Table 18). LLM steps of the reference pipeline (1, 2, 4, and 5): GPT-5.6 Terra, with high reasoning effort for the presence check. Matching judge: Gemini 3.5 Flash, since every configuration requires several thousand pair judgments. Validity panel: GPT-5.6 Luna and Gemini 3.5 Flash in the first tier, Claude Opus 5 and GPT-5.6 Terra in the second. Reference-set audit: GPT-6.1 Sol, Claude Opus 5.5, and GPT-6 Astra (GPT-5.6 Terra in the submission). Bug types: Claude Sonnet 5, now also GPT-6.1 Sol.

No evidence of bias. The Gemini judge is in general more lenient than GPT-5.6 Luna, and the Gemini agent’s findings are generally weaker than other agents’ findings. To correct for these differences, we compare the Gemini judge on findings rated the same by GPT-5.6 Luna. On the findings of the eight non-Gemini agents, the Gemini judge confirms as bugs 64% of the findings that the GPT-5.6 Luna judge rejects and 99% of those Luna confirms. Applied to the Gemini agent's 100 findings (33 rejected and 67 confirmed by the Luna judge), these rates predict 87.6 confirmations. The Gemini judge confirms 88.

Thank you for the authors’ response. It addresses most of my concerns, although the rationale for choosing Gemini 3.5 Flash in Question 4 still does not fully convince me. Based on the authors’ reply, I will keep my original evaluation.

Reviewer_yTXHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a benchmark using 2,457 historical bugs across 20 repositories. Evaluated agents detects at most 20.8% of reference bugs. It is also a finding that there is a systematic difference between what agents discover and what human maintainers fix.

Critical strengths

The benchmark addresses an important, realistic problem: discovering bugs without being given their locations.

A very thorough evaluation covering harness ablations, repeated runs, independent validity checks, and bug-type analyses

Critical weaknesses

Benchmark reliability remains insufficiently established. The manual audit rejects 12 of 100 sampled reference bugs, yet the full set is retained. Together with the largely LLM-generated taxonomy, this leaves uncertainty about how annotation errors affect recall and the reported distribution differences.

Questions for authors

Would maintainers, in a blinded comparison, consider valid unmatched agent findings less important or less worth fixing than reference bugs?

1 comment

Q1: The manual audit rejects 12 of 100 sampled reference bugs, yet all bugs are kept. How do annotation errors affect recall and the reported distribution differences?

The 12 rejected bugs come from a stratified sample that oversamples bugs flagged by an LLM audit. Weighted to the reference set, the rejection rate is 8.4%. Removing bugs does not significantly affect our results. For a more detailed discussion, please refer to our answer to Q1 in the general response.

Q2: How reliable is the LLM-generated bug taxonomy, and do the reported distribution differences depend on it?

We relabeled the human-fixed bugs with the same model (Claude Sonnet 5), and all bugs and findings with a different one (GPT-6.1 Sol), and observe that our main conclusion, the distributional difference between agents and maintainers, holds under all labelings.

The bug types were assigned by Claude Sonnet 5. We repeated the assignment of all 2,444 human-fixed bugs with the same model, and assigned them and all 69,352 agent findings with GPT-6.1 Sol. The repeated assignment gives the same family for 80.4% of the human-fixed bugs, while Sol agrees with the repeated assignment for 69.1%.

Under the original, repeated, and Sol labeling, the Jensen-Shannon distance of the human-fixed bugs from the agent configurations is 0.290, 0.248, and 0.252, respectively. The largest distance of any agent configuration from the other agents is 0.144, 0.144, and 0.153, respectively. Part of the difference between original and repeated labeling stems from an inconsistency that we corrected: in the submission, the human-fixed bugs were labeled against an earlier version of the taxonomy. The repeated assignment uses the same 33 categories for both, and we now use it in Table 4 and Figure 6.

Nine of the ten gaps in Table 4 keep their sign under all labelings. For example, agents under-report missing branches for supported input variants (-10.3, -7.7, and -7.3 points) and over-report unhandled degenerate values (+4.8, +3.9, and +4.2).

Q3: Do maintainers consider unmatched findings as less important than historically fixed bugs?

Largely yes: A maintainer judged 28 of 36 sampled unmatched findings of one repository to be real bugs, but they rated none as critical and 67% low-priority or non-bug.

We ran this study for one included repository, which one of the authors maintains. We will disclose the identity of the repository after deanonymization. The maintainer judged 36 agent findings that matched no reference bug and 36 historical bug fixes, blind to the verdict of the LLM panel. For each finding, the maintainer decided whether it was a real bug at the snapshot and what priority they would assign to the bug. They judged 28 of the 36 findings (77.8%) to be real bugs, matching the high validation rate of the LLM-based panel. However, the maintainer deemed no bug to be critical, 4 high priority, 4 medium priority, and 16 low priority (4 real findings received no priority). Judging 36 bugs from the historical reference set under the same protocol, they rated 7 bugs critical, 8 high, 11 medium, and 9 low priority, among the 35 of 36 the maintainer considers real.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback. The reviewers share two concerns: the quality of the reference set (yTXH, Qfnt) and the relationship of our pipeline with SZZ (HPNL, Qfnt). We address both of them below and remaining questions in individual replies. We have further updated the manuscript to incorporate suggested changes and marked any differences in blue.

Q1: How noisy is the reference set, and does this noise affect your conclusions?

In summary, PROBE contains some label noise, like every benchmark mined from repository history, but our conclusions are robust to it, as we will show below. We updated the paper to highlight the potential effect of the noise on concrete results.

Noise rate. One author labeled 100 reference bugs, blind to all LLM verdicts. The sample was stratified: 40 bugs came from a set that an LLM audit had flagged. Weighted by stratum size, the author rejects 8.4% of the reference set bugs, mostly because they could be seen as a feature request (5.5%). The GPT-5.6 Terra audit in the submission rejects 8.5% of all 2,457 bugs. We reran this analysis with three stronger auditors. GPT-6.1 Sol rejects 4.0%, GPT-6 Astra rejects 4.4%, and Claude Opus 5.5 rejects 10.2%. At least one of the three rejects 11.3% of the bugs (278), and all three reject 3.1% (77).

Keeping the full set. The human audit (8.4%) and the Terra audit (8.5%) agree on the rate but not on which bugs are invalid (κ = 0.29). The three LLM auditors agree for 91.8% of the bugs. Of the 201 bugs on which they differ, 159 are rejected only by Opus 5.5. In 112 cases Opus 5.5 labels the fix as a deliberate change of behavior where the other two label it as a bug fix.

To clarify the cause of this disagreement, three authors blindly reassessed a stratified sample of 30 bugs, where GPT-6.1 Sol rejected 4 and Opus 5.5 rejected 24. 17 of these Opus rejections are due to considering the change an intended behavior change rather than a bug fix. One author rejects 10 of the 24, all as an intended behavior change. The other two reject 2 and 1. This highlights that the bugs deemed invalid by models and authors are mostly borderline cases. Filtering on any individual audit would thus mostly remove bugs that can also be considered valid.

Effect on the results. Since agents tend to find these borderline bugs less often, removing them from the reference set raises recall. Precision can only fall, as findings matched to them become unmatched. Without the 278 reference bugs that any auditor rejects, the best recall, 20.8%, becomes 22.7%. Precision falls by 0.1 to 0.2 points across all 14 configurations, and the ranking barely changes (Spearman's ρ = 0.996 on recall, 0.982 on precision). We therefore conclude that our main results are robust to potential noise in the dataset.

Q2: Why does PROBE not build its reference set with SZZ alone, and how do the two approaches compare?

PROBE's presence check is an agentic SZZ variant with a different target, verifying only whether a bug is present at T0. An independent audit rejects far fewer bugs in the resulting reference set than in one built with AG-SZZ alone.

Design. SZZ identifies the bug-introducing commit given a bug-fixing commit, while we only need to know whether the fixed bug was present at the snapshot T0. We run AG-SZZ on every bug fix and provide its result to an agent. The agent inspects the snapshot and determines whether the bug is present, considering also other code changes. Our work is more similar to recent literature on whether releases contain bugs, which we discuss in detail in our response to Q3 of Reviewer Qfnt.

Where they differ. For 443 of the 3,345 candidate bugs (13.2%), the fix only adds lines, so standard SZZ would fail. With our agentic setup, we keep 361 of them. The agent also keeps 112 candidates that SZZ dates after T0, and drops 499 that SZZ dates before T0. Of these 499, 93 are considered not bug fixes. The other 406 are dropped because the faulty code (229) or the code that triggers the bug (177) was added after T0, even though some lines the fix touches are older.

Which is right. We reanalyzed all candidates except the 263 that both methods drop, blindly with GPT-6.1 Sol under the protocol of the reference-set audit. The audit confirms 94.9% of the bugs that SZZ would remove and 96.1% of the bugs that SZZ cannot date, but only 26.7% of the candidates that SZZ would add. A reference set built with SZZ alone would thus contain more than 17% of bugs that the audit rejects, compared to 3.5% for ours (rather than the 4.0% of Q1, which also counts wrong descriptions).

Effect on the results. On the 2,003 reference bugs that both SZZ and the agent accept, recall increases by 0.4 to 1.5 points and the ranking of the configurations is preserved (Spearman's rank correlation of 1.000 for recall and 0.982 for precision).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.