- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Leaderboard scores are routinely read as measurements of a model. This paper proposes no new architecture — it is deliberately a measurement audit rather than a methods paper: it decomposes a reported number into six protocol stages and, holding the model fixed and varying exactly one stage at a time, isolates four failure modes on the Open Graph Benchmark: tail reading, the undeclared sampler, the two-route pipeline, and reused splits. Every number is produced by the official evaluator, and the instrument is calibrated against published anchor values before use. The paper closes with a checklist split by the role of the reader.
Critical strengths
-
Methodological discipline that most audit papers lack, together with excellent reproducibility. "Substitution, not ablation" (fix the scorer, change one stage) makes every delta attributable; the instrument is calibrated before use, with three heuristics reproducing published anchor values to the precision at which those values are printed; and all numbers come from the official evaluator, so no claim depends on a re-implemented metric. Engineering is equally checkable: one script writes one artefact, no script reads another's output, a verification script re-checks every printed number and exits non-zero on mismatch, and the structural probes run in tens of seconds without a GPU.
-
The key quantifications turn "alarming" into "predictable and actionable." The tail-reading identity agrees with the official evaluator to machine precision across several dataset–heuristic pairs, and the corollary derived from it predicts without experiment both the
K-sweep and the sampler collapse — two otherwise alarming numbers. The undeclared-sampler mode is quantified into a computable chain: a uniformly drawn pair is an edge with probability equal to the graph density, the measured collision rate equals the density, a small amount of contamination destroys most of the value, and the effect vanishes once density falls by two orders of magnitude.
Critical weaknesses
-
No evidence of prevalence: the failure modes are demonstrated, never shown to occur. Every headline number comes from zero-parameter structural heuristics or a logistic regression over a few structural columns — never from a real submitted model. The two-route mode rests on a single pipeline (the authors' own) and the sampler mode on a handful of samplers, none drawn from a real submission. The paper explicitly declines to claim that any published entry is wrong, which is fair, but the cost is that we do not know whether any of the four modes actually bites on the leaderboard. The cheapest fix is available and not taken: take a few published submissions with released code or predictions and re-score them under a different sampler and a different
K. -
Incremental over the existing pitfalls literature, and part of one mode is the definition of the metric. The paper is candid that "what the probes add is not the existence of the problem but a decomposition," yet each component has close precedent: prior work already catalogues link-prediction pitfalls and the sensitivity of
Hits@Kto the negative set (and is the source of the anchor value used here), other work argues that the default summary statistic is close to uninformative, and splits, leakage and tuning budgets are all covered. Moreover, thatHits@Kagainst a fixed global negative set is a threshold statistic is the definition of the metric; the premise that "readers assume it measures ranking quality" is therefore never tested and may be partly a strawman.
Questions for authors
-
(Prevalence). Please take two or three published submissions with released code or predictions and re-score them: (a) under a uniform and a degree-weighted negative multiset of the same size, (b) at a set of
Kvalues spanning several orders of magnitude. Do real submissions move as far as the heuristics do? If not, please say so — the paper's impact depends entirely on the answer. -
(The published anchor that cannot be reproduced). The appendix shows that a published parameter-free anchor value is unreachable in any of the cells the benchmark makes available, and the authors further argue that on one graph the statistic cannot attain that value at any
K. Please pursue this: which graph, split or tie convention reproduces it, or is the leaderboard entry not reconstructible? I regard this as the only result in the paper that touches an actual leaderboard row, and it currently sits in an appendix. -
(Is the misreading real?). The tail-reading mode rests on the premise that readers take
Hits@Kas ranking quality. Please either give evidence — a small survey of how the metric is described in benchmark-based papers, or cases where a paper's text interprets a change in it as a ranking improvement — or soften the claim to "the metric is a threshold statistic, and here is what a reader should check," dropping the implication of a widespread misconception.