- Rating
- 3(+2)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper „PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix” consists of two parts. The first part is concerned with the collection of a new bug data set. Here, the authors propose a multi-stage agentic pipeline which analyzes commits first to identify if the fix a bug, then tries to locate the origin of the bug. Using this approach, they collect bug data for twenty projects with about 2500 bugs total. The second part makes use of this data to test if AI agents can find all bugs within a repository. They use a simple splitting mechanism that groups the files such that the context window of the models is not exhausted during the analysis of code. The results show that both recall and precision are not very good with respect to the collected data (both around 20%), but that the low precision is rather an artifact of the incomplete bug labeling because these bugs were not fixed and therefore not labeled within the data set.
Critical strengths
- Whole repository review is an interesting and important topic and analyzing how complete such a review is, as well as if multiple rounds of review by the same agent could be useful, is an important topic.
- Results were collected with contemporary models and show that based on this data and approach, a pure AI-based code review for quality assurance is likely not sufficient.
Critical weaknesses
Low data quality
The collected data is likely very noisy and while the own manual analysis clearly showed this shortcoming, as well as that the manual analysis itself is actually of low quality, this is ignored and rather (partially) buried in the appendix. Reporting between the main body and the appendix regarding this seems to disagree with respect to the data. To be honest, this is just bad science.
Concretely: The bottom of Page 3 contains says that 100 samples were validated and that agreement between the LLM and human judge was “barely exceeds chance” and as a conclusion. The text shows that of 100 bugs, the human flagged 12 as not really bugs, the LLM 17, with only six overlapping, i.e., assuming the superset we would have 23% of the bugs not being bugs. Instead of dealing with this noise somehow, the decision was to “just keep all data”, i.e., fully accepting this noise without even accounting for this in any of the subsequent interpretations of the results.
Table 15 in the appendix is presumably related to the same validation, but is for 180 samples, instead of 100. Moreover, apparently there was only a single human judge and this judge was unsure whether the data actually contained a bug for 43/180 = 24% of the data. This indicates that the actual noise rate in the data is likely higher and also that a single rater is not sufficient to establish the data quality.
These problems with the data quality undercut all results and cannot be easily fixed, without using different methods and/or reframing the analysis. Additionally, the severity of this issue together with how this handles does not raise my confidence in the other manual validation, i.e., the validation that shows that precision is not actually low.
Ignores two decades of literature
The paper ignores basically two decades of MSR literature on how to collect bug data, including recent agentic approaches that go well beyond the seminal, but very dated papers by Bird et al. and Kim et al. that are referenced.
Questions for authors
- Is my assessment regarding the noise wrong and the actual noise rate lower and the human expertise of a single judge sufficient to establish the quality of the LLM judges?
- How does the bug collection to more recent MSR literature, including LLM-based and agentic SZZ variants.
4 comments
Q3: How does your approach compare to recent MSR literature, including LLM-based and agentic SZZ approaches?
We thank the reviewer for pointing out our relationship to recent MSR work. We added two related-work paragraphs that review this literature and our relationship to it [1-14].
Relation to MSR and SZZ. PROBE builds on the MSR literature but cannot use an existing method unchanged, for three reasons.
First, SZZ and its variants [1-3], including recent LLM-based and agentic ones [4-6], aim to identify a bug-introducing commit by tracing removed and modified lines. In PROBE, 13.2% of bug fixes only add lines. Standard SZZ cannot process these fixes. Extensions fall back to tracing surrounding lines [5, 6] or the enclosing code block [7]. These heuristics yield many false positives [7] and useful candidates for only about half of such fixes [5]. The agentic variants [5, 6] then let an agent search beyond these candidates, which is similar to what our step 4 does. Even so, they report F1 scores of 0.47 to 0.79 [5] and 0.56 to 0.82 [6] across datasets, and [6] recalls only 64% of such add-only fixes. Identifying the introducing commit thus remains error-prone.
Second, some bugs have no well-defined bug-introducing commit at all, for example when an external API or a requirement changes [8]. We leverage a weaker property that remains well-defined in these cases: whether the bug was present at the snapshot T0. We integrate SZZ [2] as evidence to determine this property. The MSR work closest to our setting labels code as defective at a release based on fixes made after a release [9]. PROBE adopts the criterion established by this line of work, that labels must reflect whether a bug actually affected a release [10, 11].
Third, the release-labeling methods [9-11] label whole files or classes rather than individual bugs, and they rely on affected-version fields in issue trackers such as Jira. Even in Apache projects, these fields are unusable for 51% of defects [11], and the GitHub issues and commits used in PROBE do not necessarily have such fields. Known problems of keyword heuristics [10, 12] and tangled commits [13, 14] motivated steps 1 and 2 of our pipeline: we classify each commit from its patch, message, issues, and pull requests, and split it into separate bugs (4,296 commits yield 4,763 candidate bugs).
AG-SZZ vs. our setup. An independent reanalysis with GPT-6.1 Sol under the protocol of the reference-set audit agrees to keep 94.9% of the bugs our agent keeps despite AG-SZZ dating them after T0. Of those candidates dated before T0 and dropped by our agent, it would keep only 26.7%. A reference set built with AG-SZZ alone would thus contain more than 17% of bugs that the audit rejects, compared to 3.5% for ours.
[1] Śliwerski et al., When Do Changes Induce Fixes?, MSR '05
[2] Kim et al., Automatic Identification of Bug-Introducing Changes, ASE '06
[3] Neto et al., The Impact of Refactoring Changes on the SZZ Algorithm: An Empirical Study, SANER '18
[4] Tang et al., LLM4SZZ: Enhancing SZZ Algorithm with Context-Enhanced Assessment on Large Language Models, ISSTA '25
[5] Shi et al., AgenticSZZ: Temporal Knowledge Graph-Guided Agentic Bug-Inducing Commit Identification, arXiv '26
[6] Lyu et al., AgentSZZ: Teaching the LLM Agent to Play Detective with Bug-Inducing Commits, arXiv '26
[7] Sahal and Tosun, Identifying Bug-Inducing Changes for Code Additions, ESEM '18
[8] Rodríguez-Pérez et al., How Bugs Are Born: A Model to Identify How Bugs Are Introduced in Software Components, EMSE '20
[9] Zimmermann et al., Predicting Defects for Eclipse, PROMISE '07
[10] Yatish et al., Mining Software Defects: Should We Consider Affected Releases?, ICSE '19
[11] Vandehei et al., Leveraging the Defects Life Cycle to Label Affected Versions and Defective Classes, TOSEM '21
[12] Herbold et al., Problems with SZZ and Features: An Empirical Study of the State of Practice of Defect Prediction Data Collection, EMSE '22
[13] Herzig and Zeller, The Impact of Tangled Code Changes, MSR '13
[14] Herbold et al., A Fine-Grained Data Set and Analysis of Tangling in Bug Fixing Commits, EMSE '22
Q4: Are you planning to help the maintainers fix your found bugs before dataset publication?
Yes. However, submitting the reports in bulk would shift a significant triage effort to the volunteer maintainers. The 69,352 agent-reported bugs concern almost two-year-old snapshots, and code that has often changed significantly since. For the findings that we confirmed manually and that are still present in the current code, we will notify the maintainers with a response window of 90 days, and will offer support to fix them. Note that PROBE does not target security vulnerabilities and we observe that only around 2% of the findings are security-related (Appendix D.3). We will withhold those from the public release until they have been resolved.
Dear Reviewer Qfnt,
As the discussion phase draws to an end, we kindly ask you to acknowledge our response.
In particular, both key concerns are based on misunderstandings, as summarized below:
- Data quality: Table 15 validates agent findings, not the reference set. The 23% comes from a sample that oversampled LLM-flagged bugs, and the weighted rate is 8.4%, which three further LLM audits confirm (4–10%). All results hold even when we remove every bug that any auditor flags.
- Prior work: We build on SZZ directly and now discuss recent agentic variants in detail. Our pipeline gives much higher data quality than SZZ alone (3.5% vs. >17% of bugs rejected by an independent audit).
We hope this addresses your concerns and are happy to answer any follow-up questions.
Thank you for this extensive update and re-write the validation. This is now much improved and clarifies the noise sources and the amount of noise within the analysis. I agree that given the updated descriptions, the remaining noise is justifiable and while it might impact the results somewhat, core conclusions likely hold - at least for this generation of models. Therefore, I will raise my score.
Q1: Is the reference set too noisy to support the results, and why do you keep all bugs?
We thank the reviewer for highlighting the opportunity to discuss the effect of label noise: the noise rate is about 4% to 10%, and all results hold when the rejected bugs are removed. For the full analysis, see the general response.
The 23% is the union of the human and the LLM rejections on a stratified sample (failures oversampled), in which 40 of the 100 bugs came from a set that an earlier LLM audit had flagged. Weighted to the reference set, the human annotator rejects 8.4%, mostly borderline cases between a bug fix and a feature request (5.5%). Even at 23%, the conclusions would hold: if 23% of the reference bugs were invalid and no agent had matched any of them, the best recall would rise from 20.8% to 27.0%, and precision would not change.
Q2: Is Table 15 part of the reference-set audit, and is a single rater who is unsure about 24% of the findings sufficient?
No, Table 15 (Table 24 in the revised version) does not concern the reference set. It validates the LLM panel that judges agent-reported bugs, and we changed its caption to say so. We agree that a single rater is a limitation: we added a second human rating and now report a validity rate of 78% instead of 85%.
Table 15 compares one author's blind verdicts with the panel's verdict on 180 agent findings (20 for each of nine configurations). The reference-set audit mentioned by the reviewer is unrelated and concerns a separate sample of 100 bugs from the reference set.
The 43 unsure findings are those for which the author could not determine with confidence whether the agent-reported bug is a real defect. Even if all 43 are counted as invalid, the author alone, without any LLM, confirms 103 of the 180 findings (57.2%). On the 131 findings for which both the author and the LLM panel reach a verdict, they agree in 93% of the cases: the panel confirms all 103 findings that the author confirms, but also 9 of the 28 that the author rejects. Correcting the 84.9% that the panel confirms to the more conservative human judgement (Rogan-Gladen correction, sensitivity 100%, specificity 68%) gives a validity rate of 77.7% (95% CI 68.9 to 83.3). Assuming a sensitivity below 100% would only raise this estimate. Both 57.2% and 77.7% remain far above the precision with respect to the reference set, which is below 9%. We now report 78% in the abstract, the introduction and Section 4.2, with the details in Appendix C.3.
A second human rating is consistent with this (Table 26): the maintainer of one included repository judged 28 of 36 sampled unmatched findings (77.8%) to be real bugs and would merge a fix for 21 of them, again far above the precision of below 9%.