acceptodds
Under review as a conference paper at ICLR 2027

The Score Is Not the Model: Four Protocol Failure Modes in Open Graph Benchmark Reporting

Abstract

Leaderboard scores are read as measurements of a model. On the Open Graph Benchmark (OGB) that reading fails in four independent ways, and we isolate each one by holding the model fixed and changing only the reporting protocol. P1, tail reading: for link prediction Hits@K is, by construction, the fraction of positive pairs that outscore the K-th negative pair, so its value is set by the extreme upper tail of the negative distribution rather than by ranking quality; on ogbl-ddi the K=20 threshold (a Common-Neighbour score of 472) sits above the median positive score (358), and a Common-Neighbour scorer with AUC 0.952 reports Hits@20 = 0.1773. P2, the undeclared sampler: holding the scorer and the number of negatives fixed, swapping OGB's negative set for uniformly drawn pairs drives the same scorer from 0.1773 to 0.0000, and sweeping K from 20 to 95,599 drives it to 0.9997. P3, the two-route pipeline: a pipeline that computes features for positive pairs and a literal zero block for negative pairs scores Hits@20 = 1.000000, and still scores 1.000000 when the features are replaced by the constant 1; the official evaluator returns the value without complaint. P4, reused splits: on ogbn-products validation accuracy exceeds test accuracy by 22.1 points, enabling validation edges as message-passing input raises ogbl-ddi from 0.5800 ± 0.0461 to 0.7792 ± 0.0518, and the validation spread between post-processing configurations is up to 14× narrower than the test spread they produce. We calibrate the instrument before using it: our structural heuristics reproduce published ogbl-ddi values to the precision at which those values are published. Every number is reproducible with the released toolkit, and the paper contributes no new architecture—deliberately so.

Primary area:
datasets and benchmarks
Keywords:
benchmark evaluation, link prediction, Open Graph Benchmark, evaluation protocol, graph neural networks, reproducibility, negative sampling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RYWpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Leaderboard scores are routinely read as measurements of a model. This paper proposes no new architecture — it is deliberately a measurement audit rather than a methods paper: it decomposes a reported number into six protocol stages and, holding the model fixed and varying exactly one stage at a time, isolates four failure modes on the Open Graph Benchmark: tail reading, the undeclared sampler, the two-route pipeline, and reused splits. Every number is produced by the official evaluator, and the instrument is calibrated against published anchor values before use. The paper closes with a checklist split by the role of the reader.

Critical strengths
  1. Methodological discipline that most audit papers lack, together with excellent reproducibility. "Substitution, not ablation" (fix the scorer, change one stage) makes every delta attributable; the instrument is calibrated before use, with three heuristics reproducing published anchor values to the precision at which those values are printed; and all numbers come from the official evaluator, so no claim depends on a re-implemented metric. Engineering is equally checkable: one script writes one artefact, no script reads another's output, a verification script re-checks every printed number and exits non-zero on mismatch, and the structural probes run in tens of seconds without a GPU.

  2. The key quantifications turn "alarming" into "predictable and actionable." The tail-reading identity agrees with the official evaluator to machine precision across several dataset–heuristic pairs, and the corollary derived from it predicts without experiment both the K-sweep and the sampler collapse — two otherwise alarming numbers. The undeclared-sampler mode is quantified into a computable chain: a uniformly drawn pair is an edge with probability equal to the graph density, the measured collision rate equals the density, a small amount of contamination destroys most of the value, and the effect vanishes once density falls by two orders of magnitude.

Critical weaknesses
  1. No evidence of prevalence: the failure modes are demonstrated, never shown to occur. Every headline number comes from zero-parameter structural heuristics or a logistic regression over a few structural columns — never from a real submitted model. The two-route mode rests on a single pipeline (the authors' own) and the sampler mode on a handful of samplers, none drawn from a real submission. The paper explicitly declines to claim that any published entry is wrong, which is fair, but the cost is that we do not know whether any of the four modes actually bites on the leaderboard. The cheapest fix is available and not taken: take a few published submissions with released code or predictions and re-score them under a different sampler and a different K.

  2. Incremental over the existing pitfalls literature, and part of one mode is the definition of the metric. The paper is candid that "what the probes add is not the existence of the problem but a decomposition," yet each component has close precedent: prior work already catalogues link-prediction pitfalls and the sensitivity of Hits@K to the negative set (and is the source of the anchor value used here), other work argues that the default summary statistic is close to uninformative, and splits, leakage and tuning budgets are all covered. Moreover, that Hits@K against a fixed global negative set is a threshold statistic is the definition of the metric; the premise that "readers assume it measures ranking quality" is therefore never tested and may be partly a strawman.

Questions for authors
  1. (Prevalence). Please take two or three published submissions with released code or predictions and re-score them: (a) under a uniform and a degree-weighted negative multiset of the same size, (b) at a set of K values spanning several orders of magnitude. Do real submissions move as far as the heuristics do? If not, please say so — the paper's impact depends entirely on the answer.

  2. (The published anchor that cannot be reproduced). The appendix shows that a published parameter-free anchor value is unreachable in any of the cells the benchmark makes available, and the authors further argue that on one graph the statistic cannot attain that value at any K. Please pursue this: which graph, split or tie convention reproduces it, or is the leaderboard entry not reconstructible? I regard this as the only result in the paper that touches an actual leaderboard row, and it currently sits in an appendix.

  3. (Is the misreading real?). The tail-reading mode rests on the premise that readers take Hits@K as ranking quality. Please either give evidence — a small survey of how the metric is described in benchmark-based papers, or cases where a paper's text interprets a change in it as a ranking improvement — or soften the claim to "the metric is a threshold statistic, and here is what a reader should check," dropping the implication of a widespread misconception.

Reviewer_oZhDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper argues that scores on the OGB leaderboard not only reflect a model's performance but also protocol choices and demonstrate failure modes for 4 specific protocol choices. These are: Hits@K (as defined in the OGB code) can be misleading since an extreme part of the negative distribution can dominate; how we choose the negative samples can change the Hits@K value; how features are processed for positives vs. negatives can impact the results; and reusing validation information can make the performance look better than it actually is. The authors use simple heuristics and the official OGB evaluator to isolate the effect of each of these failure modes and provide a reporting/auditing checklist.

Critical strengths
  • The paper provides several concrete examples illustrating how each of the mentioned protocol failures can be exploited to affect OGB scores. The constant-feature experiment in P3 is particularly interesting as it would suggest that the output is not a good measure of learned information.
Critical weaknesses
  • The contributions are significantly overstated. P1 is a consequence of the definition of Hits@K; P2 is an already-known sensitivity to sampled negatives (as stated in the Related Works section), and P3 demonstrates a straightforward preprocessing bug in the author's own prior work.

  • The paper is excessively long and seems very over-engineered for the observations made. There are several instances of repetition, excessive framing and scaffolding around simple ideas, and plenty of editing inconsistencies (e.g., Proposition 2 is referred to as Proposition 4 in the following paragraph).

  • The paper states that these failure modes can occur, but there is no evidence that this is a prevalent issue in the published literature. This weakens the motivation for this work.

Questions for authors

(See weaknesses)

  • How prevalent are the mentioned failure modes in the literature?

  • How relevant are this checklist and the assessment outside of the OGB benchmark?

  • All of the failure modes seem to be disparate: P1 is a metric definition issue, and P3 is a pipeline coding bug. Could these failure modes better be categorized and expanded to produce a larger assessment of not just OGB, but graph benchmarking in general?

Reviewer_pECfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that benchmark scores reflect the complete experimental pipeline, including the model, feature computation, negative sampling, split handling, and configuration selection. On the Open Graph Benchmark, it demonstrates four issues: Hits@K’s sensitivity to extreme negative scores; score changes under different negative samplers; label leakage from processing positives and negatives differently; and distortions caused by validation/test handling. It proposes a checklist and audit scripts covering protocol disclosure, consistent feature computation, constant/noise-feature controls, split separation, and baseline checks.

Critical strengths

Clear, actionable leakage demonstration. The strongest experiment shows that class-dependent feature processing achieves perfect scores even with constant features. This convincingly isolates an implementation error and motivates a cheap diagnostic.

Critical weaknesses

Limited novelty and uncertain broader impact. The central principle and most of the identified failure modes are already well known. Several demonstrations illustrate explicit protocol violations or expected metric behavior. To establish a stronger contribution, the paper should uncover a less recognized failure mechanism, demonstrate that these issues materially affect published conclusions, or introduce and validate an auditing methodology that goes beyond existing best practices.

Questions for authors

What is the principal new insight beyond established concerns about data leakage, negative sampling, and evaluation protocols? Which findings would be unexpected to researchers familiar with existing best practices?

Can the authors demonstrate that these failure modes affect independently developed, published pipelines or materially change conclusions about competing models? The current examples establish possibility more clearly than prevalence or impact.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.