The Score Is Not the Model: Four Protocol Failure Modes in Open Graph Benchmark Reporting

Leaderboard scores are read as measurements of a model. On the Open Graph Benchmark (OGB) that reading fails in four independent ways, and we isolate each one by holding the model fixed and changing only the reporting protocol. P1, tail reading: for link prediction Hits@K is, by construction, the fraction of positive pairs that outscore the K-th negative pair, so its value is set by the extreme upper tail of the negative distribution rather than by ranking quality; on ogbl-ddi the K=20 threshold (a Common-Neighbour score of 472) sits above the median positive score (358), and a Common-Neighbour scorer with AUC 0.952 reports Hits@20 = 0.1773. P2, the undeclared sampler: holding the scorer and the number of negatives fixed, swapping OGB's negative set for uniformly drawn pairs drives the same scorer from 0.1773 to 0.0000, and sweeping K from 20 to 95,599 drives it to 0.9997. P3, the two-route pipeline: a pipeline that computes features for positive pairs and a literal zero block for negative pairs scores Hits@20 = 1.000000, and still scores 1.000000 when the features are replaced by the constant 1; the official evaluator returns the value without complaint. P4, reused splits: on ogbn-products validation accuracy exceeds test accuracy by 22.1 points, enabling validation edges as message-passing input raises ogbl-ddi from 0.5800 ± 0.0461 to 0.7792 ± 0.0518, and the validation spread between post-processing configurations is up to 14× narrower than the test spread they produce. We calibrate the instrument before using it: our structural heuristics reproduce published ogbl-ddi values to the precision at which those values are published. Every number is reproducible with the released toolkit, and the paper contributes no new architecture—deliberately so.