- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the limitations of node attribution when models are trained with graph-level supervision. The authors provide a theoretical analysis showing when different node targets cannot be distinguished from the available observations. They evaluate this problem using real software faults from Defects4J and Codeflaws, finding that around one-third of the eligible faults contain repaired and non-repaired statements that remain indistinguishable under the selected features. They also conduct controlled experiments to examine how additional target information and node-level supervision affect localization accuracy, and propose a finite-sample bound on the best achievable accuracy.
Critical strengths
-
The paper highlights an important distinction between graph-level prediction and node-level attribution, showing why accurate predictions do not necessarily imply accurate identification of the responsible nodes.
-
The study includes real-world evaluations on Defects4J and Codeflaws. The finding that around one-third of eligible faults still contain indistinguishable repaired and non-repaired statements after adding program descriptors is interesting.
-
The evaluation covers multiple GNN architectures, fault-localization methods, and controlled experiments. The analysis of target information and node-level supervision helps explain the factors affecting localization accuracy in the evaluated settings.
Critical weaknesses
-
The theoretical novelty of the paper appears limited, as several results build on established findings in Bayesian hypothesis testing and the 1-WL expressivity limitations of GNNs. It is unclear what new theoretical insights the paper provides beyond applying these existing results to node attribution.
-
The proposed finite-sample bound is evaluated only on synthetic observation channels with a small alphabet size. The paper acknowledges that real-world coverage observations can have a much larger alphabet, but does not demonstrate whether the bound provides meaningful estimates for actual software faults.
-
Proposition 4 requires independent observations collected separately for each candidate target. In real fault-localization scenarios, collecting such observations may be difficult because the actual faulty location is unknown.
-
The real-world experiments identify faulty and non-faulty statements that are indistinguishable under the selected features. However, they do not establish whether these statements remain indistinguishable when richer program information is available, limiting the generalizability of the findings.
-
Table 13 shows that none of the six evaluated molecular or motif datasets contain mixed-relevance WL classes at the selected refinement depth, while only 13 of 9,129 sentiment trees contain such classes. Although these results do not invalidate the theoretical findings, they raise questions about how frequently this particular ambiguity occurs in general graph attribution tasks.
-
Section 3.1 indicates that test outcomes are used both as graph-level labels and in the spectrum features provided to the models. This raises concerns about whether the high test-outcome AUROC reported in Table 2 partly reflects information about the prediction targets already available in the inputs.
-
Table 2 reports pooled node AUROC, which includes comparisons between statements from different faults. Since fault localization aims to rank statements within individual faults, per-fault ranking metrics would be more informative. Although the appendices report macro node AUROC for the spectrum methods and optimistic Top-1 ceilings, these do not directly show the Top-1 performance of the trained GNN models.
-
Some prior work shows that many tests in Defects4J are added after the bug is reported or fixed. Since this paper studies limits imposed by available observations, test provenance is directly relevant. It is unclear whether the reported ambiguity rate holds when using only tests available before debugging.
Questions for authors
-
Theorem 1 builds on established results in Bayesian hypothesis testing, while Lemma 2 follows the known 1-WL expressivity limitations of message-passing GNNs. Proposition 4 also builds on existing concentration inequalities. Could the authors clarify what new theoretical insights these results provide beyond their application to node attribution?
-
Can the authors demonstrate a meaningful bound on Defects4J or Codeflaws, or explain how the required observations could be collected in practice?
-
Could the authors report the Top-1 performance of the trained GNN models, particularly how often the observed ambiguity affects actual fault-localization performance?
Thank you for your detailed review. We evaluated per-fault Top-1 with the original checkpoints and tested richer source observations. Highest-score ties reduce localization success, while source context resolves most original matches, showing how the framework connects ranking errors to the information supplied.
1. What is new beyond Bayesian testing and 1-WL?
Our contribution links observation-level identification to equal scores in shared representations and their evaluation consequences. Building on Bayesian testing and 1-WL, we show that swapping relevance within equal-score classes preserves AUC even when the correct node changes. Thus aggregate performance can miss the localization distinction. For finite samples, the count-based bound uses one sampling radius, compared with two for the mixture-TV relaxation. The real-fault rankings below test the evaluation consequence.
2. Actual trained-model Top-1 and the effect of ambiguity
Equivalent repaired/non-repaired candidates reduce tie-aware Top-1 in all three trained architectures. We use the nine original checkpoints and reduced graphs. Top-1 breaks ties uniformly and succeeds on any repaired statement. Percentages average seeds within each fault, then faults.
As the table shows, actual Top-1 varies across architectures and held-out splits. Highest-score ties reduce root-held-out Top-1 by 4.3, 4.8, and 5.4 percentage points for GraphSAGE, GAT, and Transformer relative to optimistically resolving those ties. Every affected group contains a verified repaired/non-repaired equivalent pair. This quantifies ambiguity in fitted rankings and potential recovery from resolving ties, rather than gains achieved by a new localizer.
3. Richer program information
Real source context distinguishes almost all coverage-equivalent pairs. We tested all 924 original pairs in 115 Defects4J faults using fixed buggy-source versions and features. Removing comments and whitespace while retaining identifiers, literals, and operators leaves 54 matched pairs in 16 faults. A fixed two-line context on either side leaves one pair in one fault. This supports enriching observations before changing the scorer. Distinguishable inputs enable separation, while successful localization remains a separate evaluation.
4. Finite-sample bound and target-specific observations
The bound is informative on compressed execution observations when controlled faults supply independent samples per target. All seven original EPFL designs give non-trivial bounds. A fixed ordering selected two existing injected locations per design, each with 2,000 independently sampled uniform inputs. For an equally likely target and one fresh execution, the observation is either the output-failure bit or the healthy/faulty output bits. All 14 bounds hold jointly with 95% confidence: upper accuracies, rounded upward, range from 54.2–54.9% for the binary channel and 54.8–56.4% for the four-symbol channel. The target is diagnosis from that compressed observation, with input vectors and other traces withheld. Controlled injection supplies per-target samples that a single natural fault does not. The Defects4J rankings above separately measure natural-fault ambiguity.
5. Frequency beyond software and the oracle anchor
The molecular and motif controls show that natural masks can already align with distinguishable classes: zero of the six tested datasets has mixed classes at the reported depth, versus 13 of 9,129 sentiment trees. This supports measuring the observation–target relationship for each task, rather than assuming within-class ambiguity is universal. The oracle anchor separately isolates target information from node supervision. Its role is a controlled intervention, while the new source-context comparison supplies realistic program information.
6. Test-outcome features and test provenance
The high test-outcome AUROC partly reflects outcomes already supplied in the spectrum features. It therefore measures outcome consistency, while localization is evaluated against withheld repair targets through within-fault ranking.
Removing later tests cannot separate an existing coverage-equivalent pair when the buggy program, candidate set, and static descriptors stay fixed: equal coverage columns remain equal after deleting rows, as do their spectrum counts. Therefore the original matched pairs persist for any pre-debug subset of those tests under this fixed candidate universe. The 34.8% figure describes the archived eligible population, rather than a newly filtered historical-test population.