acceptodds
Under review as a conference paper at ICLR 2027

The Statistical Resolution of Node-Classification Benchmarks

Abstract

Prediction errors of a message-passing network are correlated along the edges of the graph it is evaluated on, so the nodes of a node-classification test set do not contribute independent observations. We study the consequence: the statistical resolution of a benchmark, the smallest difference it can detect, is far coarser than its test-set size suggests, and most published differences fall below it. We formalize resolution through the design effect and the minimum resolvable gap, show that the classical plug-in estimator of is unidentified on real error fields, where a previously undescribed degree-sampling bias alone flips the sign of its correlation estimate on 14 of 16 configurations on ogbn-arxiv. We instead estimate with a graph-partition block bootstrap validated against known design effects, then train 22,400 models on 14 benchmarks. Resolution attaches to the (dataset, model class) pair: on identical Cora test nodes a graph-free MLP has and a GCN . Depth leaves it flat, and size does not buy it, since the 48,603 test nodes of ogbn-arxiv carry the information of roughly 700. Nothing in the bootstrap needs the statistic to be a mean, so the same machinery gives the design effect of ROC-AUC; auditing all 86 published gaps on the scale each was reported on, 87% fall below their benchmark's measured resolution. A claim about an architecture needs one term more: training randomness is a median 28% of the variance of a paired comparison, and adding it takes the rate at which a seed change alone is called significant from 11.9% to 0.4%. Code, the cached per-node correctness vectors and the audit corpus accompany this submission.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
node classification, graph neural networks, benchmark evaluation, design effect, block bootstrap, statistical power, error correlation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2G7sOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how precisely a single transductive node classification benchmark can distinguish between the performance of two distinct models, given that predictive errors across test nodes are not independent. The paper argues that standard i.i.d. uncertainty estimates are not appropriate, as they can significantly underestimate variance. The paper proposes a graph-partition block bootstrap (GPBB) to estimate the dependency-aware uncertainty and use it to define the minimum resolvable gap (MRG). They apply the framework across 14 graph datasets and 22,400 trained models.

Critical strengths
  • The paper addresses an important problem in evaluating graph models for node classification. Nodes in a graph are not necessarily i.i.d., and evaluating these models as if they were operating on i.i.d. data can lead to misleading estimates of uncertainty. The distinction between within-graph uncertainty, training randomness, and superpopulation uncertainty (graph-to-graph) is useful.

  • The experimental suite is quite large, including both a large number of models, real-world datasets, and synthetic datasets where the true design-effect is known.

Critical weaknesses
  • The proposed GPBB doesn't seem well calibrated enough to support the paper's main quantitative conclusions. In the paper's own experiments, GPBB can be significantly below the nominal 95% target (reaching 49% for one dataset), raising the question of how reliable the reported MRG values can be interpreted.

  • The paper's presentation makes the contributions unnecessarily difficult to assess. The authors combine several distinct concepts (e.g., node-level dependence, variance inflation, design effect, etc.) all in the name of "statistical resolution", without clearly distinguishing between the corresponding quantities and claims. Further, the language is quite strong for what's being presented. For example, the 87% claim comes from a convenience sample of 86 gaps. Further, because MRG is defined using β=0.2\beta = 0.2, falling below the 80%-power threshold for MRG does not imply the reported gap is insignificant.

  • The theoretical justification for GPBB is not entirely clear to me. The propositions require some conditional independence assumptions, which (as the authors admit) do not hold for spatially structured graph data. It is therefore unclear how well the theoretical dependence radius supports the block construction used in the experiments.

Questions for authors

(See weaknesses)

  • How reliable are the MRG estimates given that the bootstrap does not always achieve the expected coverage? How should we interpret a published performance gap that falls below the MRG?

  • How much do the results depend on how the graph is divided into blocks for the bootstrap?

  • How representative is the sample of 86 published performance gaps used in the literature analysis?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 2G7s (1/2)

Thank you for a critical review; it prompted the analyses we think strengthen the paper most.

W1/Q1: calibration and the reliability of the MRG. Your concern is valid: GPBB does not attain nominal coverage everywhere. We now calibrate on all 14 benchmark topologies at the radius, block count and test size of the fixed-recipe audit (App. P). On the six with median block conductance ≤ 0.4 coverage is 0.886–0.934 (moderately below 0.95 on Cora); on the eight whose blocks lose most of their edges it is 0.33–0.79. GPBB beats the i.i.d. interval in all 28 cells. Table 2 flags the eight leaky rows; there GPBB underestimates D on average.

Prop. 4 says why: E[D̂] = (1 − 1/B)·D − κ×, so the expected error is −D/B − κ×, a finite-block term plus the covariance GPBB misses across block boundaries. It is an identity for the expectation, not a bound on any one estimate. κ× ≥ 0 for every field our generator can produce (non-negative smoothing, one threshold direction), and on roman-empire the bootstrap mean lies within the Monte-Carlo error of D. For a paired comparison the relevant field is the difference of the two correctness vectors; for ROC-AUC the identity holds for the linearized statistic. It does not make any audited gap conservative, and the revision no longer says so anywhere.

Do the conclusions depend on the leaky rows? On the six low-conductance topologies the audit gives 44/58 (fixed recipe), 32/58 (strong models from different implementations) and 23/58 (strong one-recipe pairs), in line with the overall shares. The stronger models' partitions are sized to their own receptive fields and lie outside the radius-4 calibration; we state this.

How to read a gap below the MRG. The benchmark, under the protocol that produced the gap, had less than 80% power to detect a true difference of that size. It does not mean the gap is non-significant: with the fixed recipe 55/86 are below the 5% threshold against 66/86 below the MRG. And the MRG depends on the pair compared: with the strongest, most alike reference pairs the share below it falls to 41% overall, while 63% of the rank-1-vs-2 gaps stay below it (§7.5).

AuthorsOpenReview ↗

Response to Reviewer 2G7s (2/2)

W2: presentation and strength of language. §3 opens with a "quantities at a glance" table (dependence → D → n_eff → MRG: what each is a property of, how it is estimated, which claim it supports). "Unresolvable" is replaced by "below the benchmark's resolution (underpowered)" throughout. The abstract calls the corpus a convenience sample, reports the range over reference models (41–77%) rather than one share, and names the part that is robust (rank-1-vs-2 gaps).

W3: theoretical justification for GPBB. GPBB does not rest on Prop. 1's exogenous-noise assumption; Prop. 1 only suggests a block scale. What GPBB estimates is fixed by Prop. 4, the expectation of the cluster-robust variance written out for GPBB: for a fixed partition and per-node statistics with a common mean (regional difficulty treated as a random effect), E[V̂] = (1 − 1/B)·Var(â) − m⁻² Σ_{b≠b'} Cov(U_b, U_b'). The relative bias vanishes when the cross-block covariance is small relative to D and B is large, the low-conductance condition of network cluster-robust inference (Leung, 2023). In 16 synthetic settings the identity predicts the bootstrap mean to Monte-Carlo error, and the cross-block share it isolates orders the topologies by coverage (CiteSeer 14–24%, Cora 29–54%, actor 72–89%).

Q2: dependence on the partition. Up to eleven alternative partitions per benchmark (B from 10 to 100, three BFS seeds, boundary refinement, spectral): the fixed recipe's paired MRG moves ≤ 17% on 11 of 14; the protocol-matched accuracy share ranges 67–78% (single-test-set share 85–89%) (App. N).

Q3: representativeness of the 86 gaps. A convenience sample, as the abstract now says. It is rule-based within each of ten sources (App. F, all now cited): rank-1-vs-2 and rank-1-vs-5 gaps within one table, never across papers, every printed row transcribed. The interval resamples the ten source papers; leaving out any one moves the fixed-recipe share between 74% and 82%. Two systematic samples: on the full ogbn-arxiv leaderboard 16–25 of the 25 state-of-the-art steps are below the protocol MRG across the reference sets (median step 0.17 points; 21 are below even the i.i.d. MRG, so these steps are small whatever the variance model), and in the complete tables of Platonov et al. 87–118 of the 144 adjacent-rank gaps are.

Reviewer_2G7sOpenReview ↗

Response to Rebuttal

I appreciate the authors' efforts to address my concerns and provide a detailed rebuttal, but I still have some reservations regarding the following:

  • GPBB still falls well below nominal coverage on eight of the fourteen topologies. Proposition 4 helps explain the bias, but I'm not convinced this resolves the reliability concerns or sufficiently supports the main quantitative conclusions.
  • The clarification around MRG is helpful, but the results still vary quite a bit across reference models (41–77%). Along with the convenience sample of 86 gaps, this makes the broader conclusions difficult to assess.
  • The additional analysis is useful, but I'm still unsure how well the theoretical arguments support the block construction in practice, particularly when there is substantial cross-block dependence.

Overall, while I appreciate the improvements, my main concerns remain. Thus, I will keep my original score.

Reviewer_xh3QOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the impact of correlated errors on performance evaluation in node-classification benchmarks. Leveraging design effects and minimum resolvable gaps (MRGs), the authors quantify the effective statistical power provided by standard test splits. They introduce a graph block bootstrap method and evaluate it across 14 diverse datasets, demonstrating that many reported performance gains fall below the threshold that these benchmarks can reliably resolve.

Critical strengths

The paper addresses a fundamental, frequently overlooked methodological challenge in evaluating Graph Neural Networks.

The paper conducts an extensive empirical study across 14 datasets, supported by sound experimental controls.

It clearly delineates the boundaries, assumptions, and limitations of the proposed framework.

Critical weaknesses

The operational impact of block size warrants clearer explanation. Because variance estimates are sensitive to this parameter, the paper would benefit from concrete, practical guidelines or heuristics for selecting an optimal block size.

The conceptual interpretation of the MRG should be introduced and clarified earlier in the manuscript. Falling below the MRG threshold does not necessarily imply statistical insignificance; clarifying this distinction early will prevent readers from conflating resolvable effect sizes with null hypothesis rejection.

Questions for authors

See weakness.

2 comments

Thank you for the positive assessment and two concrete suggestions, both adopted.

W1: practical guidance on block size. §5 now gives a four-step recipe, backed by two new analyses:

  1. Choose B from the dependence radius, B = clip(n / (4·|B₂L|), 15, 200).
  2. Check the partition before trusting D̂: report the median block conductance (code/partition_compare.py). In our calibration on all 14 topologies, every topology with conductance ≤ 0.4 covered at 0.886–0.934 at the audit radius; above it coverage fell to 0.33–0.79, and there GPBB underestimates D on average. The 0.4 cut is an empirical diagnostic, not a guarantee.
  3. Average over partition draws (we use 20) and report the spread; one partition into ~15 blocks has ~38% relative noise in its variance estimate.
  4. Report the paired MRG, not only D: for the fixed recipe, across up to 11 partitions (B from 10 to 100, three seeds, three partitioners) it moves ≤ 17% on 11 of 14 benchmarks.

Prop. 4 explains the direction: GPBB keeps covariance inside blocks and misses it across them, so where dependence across blocks is positive, finer partitions lower D̂ on average.

W2: interpreting the MRG early. §1 now says, before any number, that a gap below the MRG is one the benchmark had less than 80% power to detect, not a non-significant gap. After Definition 1 we make this formal: nominal significance needs only 0.70 × MRG (0.79 × MRG with GPBB's t interval at B = 15), and every audit share is reported beside the share below the 5% threshold (fixed recipe, protocol-matched: 77% vs 64%). We also now show that the MRG depends on the pair of models compared (§7.5), which bounds what a single resolution number can say.

I've read the rebuttal and other reviews and am going to keep my score.

Reviewer_ubFUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper shows that correlated errors along graph edges make node-classification test sets far less informative than their size suggests. It proposes the design effect D and the minimum resolvable gap (MRG), shows the classical plug-in estimator fails (including a new degree-sampling bias), and introduces a validated graph-partition block bootstrap (GPBB). Across 22,400 models on 14 benchmarks, it finds that D depends on the model class, that ogbn-arxiv’s 48,603 test nodes are worth about 700 independent ones, and that 87% of 86 published gaps fall below the measured MRG.

Critical strengths

Important question, turned into a number a benchmark can report. Careful validation: known-D simulations, an independent ANOVA check, a seed-change null and network cross-validation. Failure regimes and the refuted depth prediction are reported honestly. Strong reproducibility: code, per-node caches, unit tests and the gap corpus are released.

Critical weaknesses

D depends on the partition scale. Radius-matched partitions give the MLP finer blocks. On a common partition, the Cora MLP-vs-GCN gap shrinks from 2.8× to 1.6×, and on ogbn-arxiv the class ordering reverses. Part of the “model-class” effect may come from the estimator rather than the model. The estimator is used mostly outside its validated regime. The B = 15 floor binds on 11 of 14 datasets, and blocks are smaller than one dependence ball on 10 of 14. Specific D and n_eff values are reported with more precision than this supports. The audit does not match each paper’s protocol. Published gaps are often means over many splits or seeds, but they are compared with a single-run MRG, which may overstate the 87%. The measured models are weaker than the published ones. A fixed untuned recipe leaves some models underfit, and the paper notes underfit models inflate D. The MRGs may not transfer to the tuned models in the audited tables.

Questions for authors

What are D and MRG for all model classes on a common partition? How does the 87% figure change with protocol-matched MRGs? How sensitive are D and the MRG to hyperparameter tuning?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ubFU (1/2)

Thank you for the most detailed reading; each question led to a new analysis, and W4 changed the paper's headline.

W1/Q1: D and MRG for all classes on a common partition. Every configuration is measured on the radius-4 partition the audit uses, averaged over 20 draws (one 15-block partition carries ≈ 38% relative noise). Class medians, mean ± s.d. over draws (Table 2, App. N):

datasetMLPGCNSGCSAGEGATGNN/MLP [5%, 95%]
cora2.5±0.34.2±0.74.2±0.73.8±0.64.5±0.81.66 [1.40, 1.99]
citeseer1.9±0.32.7±0.52.7±0.62.5±0.53.2±0.71.41 [1.21, 1.61]
pubmed1.9±0.22.8±0.33.6±0.42.1±0.23.0±0.31.57 [1.18, 1.85]
tolokers12.3±2.09.5±1.412.1±2.09.6±1.312.3±2.00.89 [0.84, 0.93]
ogbn-arxiv105±3942±13118±4044±1360±200.50 [0.40, 0.60]

You were right that part of the contrast came from the partition scale: Cora's GNN/MLP ratio is 2.8× radius-matched and 1.66× on a common partition, and on ogbn-arxiv the order reverses (it holds with the faithful OGB models: MLP D ≈ 60, GCN/SAGE ≈ 32). We revised claim (i): D varies across classes at a fixed partition on 8 of 14 benchmarks, in a direction that depends on the data. The MRG is a property of a pair through its error correlation: with the tuned families, the median correctness correlation is 0.71 between two GNNs and 0.28 between a GNN and the MLP on Cora, so GNN–MLP MRGs are several times larger (ogbn-arxiv: 2.4–2.7 vs 0.45). Per-pair MRGs for every reference set are in App. R–S and results/*_analysis.json.

W2: the estimator outside its validated regime. (a) Calibration now covers all 14 topologies at the fixed-recipe audit's radius, block count and test size (App. P): 0.886–0.934 at low conductance, 0.33–0.79 on the eight leaky topologies, above the i.i.d. interval everywhere; Table 2 flags those eight rows, where GPBB underestimates D on average. The stronger reference models of §7.5 use partitions sized to their own receptive fields, outside this calibration, and we say so. (b) Prop. 4 gives the expected error exactly. (c) D and n_eff carry their spread over 20 partition draws (App. N; typically ±15–20%, ±30% on ogbn-arxiv); the fixed recipe's paired MRG moves ≤ 17% across up to 11 partitions on 11 of 14 benchmarks.

AuthorsOpenReview ↗

Response to Reviewer ubFU (2/2)

W3/Q2: protocol-matched MRGs. Each gap is scored against the statistic its table reports (App. M): a mean over the 10 official splits, or a seed mean on one test set (Planetoid public split; OGB temporal split). Averaging 10 splits of one graph shrinks the MRG by only 1.2–2.8× (roman-empire 1.98 → 1.01, Cora 3.64 → 2.27, chameleon 8.89 → 6.24), not √10. Fixed recipe: 66/86 (77%) below the protocol MRG, against 88% for one test set; 67/86 if the two sources that report five runs are scored at five. Luo et al.'s Planetoid numbers are a five-seed mean on one random 20-per-class split; we score them on the public split, and on our nine random splits of those sizes the seed-mean MRG is 2.7–4.7 points, above every one of those gaps (≤ 1.22).

W4/Q3: models as strong as the published ones. Three reference sets beyond the fixed recipe: (i) tuned families with residuals and normalization, a 12-trial validation search from the source recipe, plus GCNII and APPNP on Planetoid (1,860 runs); (ii) the OGB reference MLP/GCN/SAGE exactly as released (55.7 / 71.8 / 71.9 on the temporal split; OGB reports 55.5 / 71.7 / 71.5); (iii) GCN/SAGE/GAT with the released configurations of Luo et al. (2024) (733 runs), 0.4–1.7 points below their reported values on the heterophilous benchmarks (roman-empire GCN 91.0 vs 91.3; amazon-ratings 53.5 vs 53.8), though 3–5 points below on Planetoid, which we do not reproduce. Seed counts differ by set (App. R–S); on the fixed grid, a 3-seed mean's MRG exceeds a 10-seed one by 2–7% on Planetoid.

Sensitivity of D: small (Cora GNNs 4.0–4.3 vs 4.2); on ogbn-arxiv stronger GNNs read 29–33 vs 42, still about half the MLP's. Sensitivity of the audit: large, mostly through how alike the two models of a pair are. Gaps below the protocol MRG (Table 3):

reference models, pairsall 86rank-1-vs-2 (43)leaderboard (25)
fixed recipe, all pairs66 (77%)38 (88%)25
fixed recipe, GNN pairs63 (73%)37 (86%)25
tuned, all pairs59 (69%)36 (84%)23
tuned, GNN pairs49 (57%)33 (77%)19
strong, different implementations47 (55%)33 (77%)22
strong, one recipe (Luo et al.)35 (41%)27 (63%)16
all six at once34 (40%)27 (63%)–

The different-implementation row has such pairs on 12 of 15 (benchmark, protocol) cells; the other 10 gaps use the one-recipe row (App. S). So we no longer claim that most published gaps are below resolution: 41–77% are, depending on how alike the compared systems are, which a published table does not reveal; of the rank-1-vs-2 gaps, 27 of 43 are below it under every reference. The abstract, §7.5 and the conclusion say this, and we recommend releasing per-node predictions so that the paired MRG of the actual pair can be computed. Thank you for pushing on this.

A short follow-up on W4. The new App. T re-trains both models of the 14 corpus comparisons whose code is released and computes each pair's own MRG under a common ten-split protocol. The reproduced pairs give the same verdict wherever all reference choices agree, and the most-alike reference's verdict on all 14; as you anticipated, the fixed recipe transfers worst (10/14 gaps below its MRG, 4/14 below the pairs' own). Details are in the general comment above. We would be glad to answer any remaining question before the discussion closes.

I am satisified with authors rebuttal, I would like to maintain my positive score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response and summary for the AC

We thank the reviewers. All three asked how reliable GPBB is where we use it, how much the numbers depend on the partition, and how a gap below the MRG should be read; ubFU also asked for protocol-matched MRGs and models as strong as the published ones. The revision answers each with new analyses (changes in blue; main text within 10 pages). What changed:

quantitysubmissionrevisionwhere
Cora GCN/MLP design-effect ratio2.8× (each class on its own partition)1.66× [1.40, 1.99] on one common partition§7.1, App. N
ogbn-arxiv, MLP vs GNN design effectMLP lower (artefact of its partition)MLP about 2× higher on a common partition, also with the OGB reference models (D ≈ 60 vs 32)§7.1, §7.5
ogbn-arxiv effective test size≈ 700700–1,800 across partition draws§7.3
GPBB calibration4 topologiesall 14 topologies, 28 cells: 0.886–0.934 at low conductance, 0.33–0.79 where blocks leak§6, App. P
what GPBB estimates"a lower bound"exact identity for its expected error, −D/B − κ× (Prop. 4); not a bound on any one estimate§5, App. O
share of 86 published gaps below resolution87% (one test set, fixed recipe)77% with each gap's own protocol; 41–77% across reference-model sets; 63–88% of the 43 rank-1-vs-2 gaps under each set, 27 of 43 under all of them§7.4, §7.5

The last row is the main change, and it changes the paper's headline. Scoring each gap against the protocol that produced it lowers the one-test-set share (88% on the corrected corpus) to 77%. Then, following ubFU, we re-measured every MRG with stronger models: tuned families (1,860 runs), the OGB reference models, and the classic GNNs with the released configurations of Luo et al. (2024) (790 further runs). An MRG belongs to a pair of models: stronger models err less and agree more, so their paired MRG is smaller. With three strong GNNs trained from one recipe, the most alike pairs we have, 41% of the 86 gaps remain below resolution; with strong models from different implementations within 3 points of each other, the closest analogue of a published rank-1-vs-2 pair we can build, 55%. The comparisons that decide a state of the art are the robust part: 63–88% of the rank-1-vs-2 gaps are below resolution under each reference set (77%, CI 62–94%, for different implementations), and 27 of the 43 under all six at once (CI 43–84%), against 19–65% of the rank-1-vs-5 gaps. The abstract and conclusion now say exactly this.

Corrections to the submitted corpus. (i) We had transcribed Luo et al. (2024) from its arXiv v1 rather than the NeurIPS camera-ready we cite, which adds their re-runs of seven transformers to Table 2 and four to Table 3, rows that change several rank-1-vs-2 gaps; we had also left out the seven rows its Table 3 copies from other papers, and labelled its Planetoid columns random 60/20/20, whereas it averages five seeds on one 20-per-class split. (ii) From Polynormer's Table 2 we had kept only the 13 rows not copied from Platonov et al. Both sources are now transcribed in full (24 of the 86 gaps change; the one-test-set share moves from 87% to 88%), and every number in the revision and below uses the corrected corpus. The ten sources of the corpus are now all cited.

what each new analysis shows

  1. Common partition (ubFU W1, 2G7s Q2). Every class on one partition, 20 draws: classes differ on 8 of 14 benchmarks, in a direction that depends on the data (GNN/MLP 1.25–1.66× on six; reversed on ogbn-arxiv, 0.50×, and tolokers, 0.89×).
  2. Calibration on every topology (2G7s W1, ubFU W2). At the fixed-recipe audit's radius GPBB covers 0.886–0.934 on the six topologies with median block conductance ≤ 0.4 (still below nominal on Cora) and 0.33–0.79 on the eight whose blocks leak; it beats the i.i.d. interval in all 28 cells. Table 2 flags the eight leaky rows.
  3. What GPBB estimates (2G7s W3). Prop. 4: E[D̂] = (1 − 1/B)·D − κ×, κ× the cross-block covariance, so the expected error is −D/B − κ×: negative for every field our generator can produce, and a statement about the average, not about any one estimate.
  4. Protocol-matched audit (ubFU W3/Q2). Fixed recipe: 66/86 below the MRG of each gap's own protocol (CI 65–90%, resampling the ten source papers), 55/86 below the 5% threshold; an i.i.d. reading of the same protocols gives 51/86.
  5. Reference models (ubFU W4/Q3). Six (reference set, pair set) combinations, Table 3.
  6. Presentation (2G7s W2, xh3Q W2). §3 has a "quantities at a glance" table; "below the MRG" is defined in §1 as underpowered, beside the 5% threshold (0.70 × MRG); we dropped the replication-probability reading (observed power) and the word "unresolvable".
AuthorsOpenReview ↗

Additional check of reference-model transfer

Additional check of reference-model transfer (new App. T; revised PDF and supplement)

The audit's MRGs come from reference models, and §7.5 reports how the share of gaps below resolution moves across six reference sets (41–77%). Reviewers ubFU and 2G7s asked how close those references are to the comparisons actually audited. We have added a direct check on the subset where this is possible.

What we ran. The 14 corpus comparisons whose two models both come from three released code bases: Luo et al.'s and Polynormer's released implementations and run scripts, and a documented PyTorch port of Platonov et al.'s DGL code over their full depth grid. All are evaluated under a common ten-official-split protocol (Platonov et al.'s published protocol; Polynormer and Luo et al. report ten and five runs and their scripts default to one and three). The pair-specific MRG uses the same GPBB procedure and radius-4 partition as the fixed-reference audit.

What it shows.

  1. Reproduction quality. The 45 model–dataset cells of the nine Platonov models we reproduced lie within 0.25 points of the published values (median 0.04); the 14 published gaps are reproduced within a median 0.14 points.
  2. Consistency with the reference analysis. Wherever all six reference sets agree (8 of the 14 gaps), the reproduced pair gives the same below/above verdict; where they disagree, it sides with the most-alike set, which matches all 14 (classifying published gap magnitudes). In this subset, the four gaps below the MRG under every reference set, the robust part of the audit, are exactly the four below the pair-specific MRG.
  3. Transfer of the fixed recipe. The pair-specific MRG is a median 0.27× the fixed-reference MRG, and the count below it is 4/14 rather than 10/14 under the fixed recipe (4/8 rank-1-vs-2, 0/6 rank-1-vs-5).

Scope. This checks reference transfer, not GPBB's coverage, and it covers 14 comparisons selected by code availability; the other 72 keep their reference-model uncertainty. The tuned and strong references keep their own receptive-field-matched partitions. The per-gap table, the 36-pair analysis within each benchmark and the training repetitions are in App. T; a coarse-partition check of calibration is in App. P.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.