acceptodds
Withdrawn from ICLR 2027

When Can Graph Conformal Sets Transfer? Selective Diagnostics Under Unlabeled Topology Shift

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Unlabeled graph topology can reveal shift but cannot universally certify conformal coverage: observable-equivalent target environments may have different label conditionals and arbitrarily different miscoverage. We turn this identifiability limit into a selective transfer criterion over a declared family of graph-environment shifts. From labeled calibration environments, TOPOTRANSFER fits a cross-fitted monotone envelope from topology–representation discrepancy to excess miscoverage, conformalizes its environment-level residual, and returns a validity decision only inside calibrated support. On GOOD-Cora topology shift, TOPOTRANSFER reaches 0.89 violation AUROC with a 5.6% false-certificate rate at 74.9% acceptance; SNAPS reaches 0.72, 16.8%, and 86.4%, respectively. The support and envelope ablations show that graph observables become useful only after environment-level calibration, yielding a non-vacuous selective decision within the reported environment family rather than an unconditional topology-shift score.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, graph neural networks, uncertainty quantification, distribution shift, graph out-of-distribution, topology shift, selective prediction, calibration

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_quGiOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks whether unlabeled graph observables can indicate when a source-calibrated conformal predictor will continue to achieve acceptable coverage under graph-environment shift. It first gives an identifiability observation: for a fixed observable marginal over (G, X), different label conditionals can induce arbitrarily different miscoverage, so unlabeled graph statistics cannot provide a universal coverage certificate. The proposed TOPOTRANSFER procedure therefore assumes exchangeable labeled calibration environments from a declared shift family, computes topology/representation signatures, fits a cross-fitted monotone predictor of excess miscoverage, conformalizes environment-level residuals, and abstains when a target is outside calibrated support or fails a leave-one-environment-out stability check.

Critical strengths
  1. The paper makes a useful conceptual distinction between a universal coverage certificate and a selective environment-level diagnostic calibrated only within a declared shift family.

  2. The single reported GOOD-Cora experiment at least evaluates the relevant selective trade-off (false certificates jointly with acceptance) and includes support/refusal ablations.

Critical weaknesses
  1. The central positive validity claim is not backed by a theorem for the final procedure. Equation (2) conformalizes residuals, but the actual decision additionally depends on learned signatures, component-wise support checks, leave-one-environment-out stability, the finite-environment correction b_E, and the acceptance threshold. I could not find a formal finite-sample statement showing what error probability the complete selected decision controls, under precisely what exchangeability assumptions, and how the data-dependent selection preserves that guarantee. For a paper whose main contribution is framed as a selective validity decision, this is a fundamental gap rather than a presentation issue.
  2. The empirical evaluation is far below the evidence required by the paper’s own stated hypotheses.
Questions for authors
  1. Can the authors state and prove a finite-sample theorem for the final TOPOTRANSFER decision rule, including cross-fitting, estimation of environment miscoverage, b_E, component-wise support filtering, and the stability gate? Please state exactly what event/probability is controlled and over which randomness.
  2. Are the support ranges and stability thresholds fixed independently of the residual calibration scores, or selected using the same environments? If they are data-dependent, why does the conformal residual argument remain valid after this selection?
Reviewer_QeV3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether unlabeled structure of a target graph can tell us if a source-calibrated conformal predictor still covers on the target. Proposition 1 states that it cannot in general, since the label conditional is unconstrained. The proposed TopoTransfer treats labeled calibration environments as the exchangeable unit: it fits a monotone map from a graph/representation discrepancy to excess miscoverage, conformalizes the environment-level residual, and issues a validity statement only when the target lies inside the calibrated support and the envelope is leave-one-environment-out stable. The only reported evidence is Table 2 on one GOOD-Cora split: violation AUROC 0.89, false-certificate rate 5.6%, acceptance 74.9%, against 0.72/16.8%/86.4% for SNAPS.

Critical strengths
  • The framing is sensible: calibrate across environments rather than nodes, regress excess miscoverage on an observable discrepancy, and abstain outside the covariate's support. Done properly, this would be a useful diagnostic.
  • The paper is candid about what was not done (§8, Table 1, reproducibility statement).
Critical weaknesses
  1. The evidence is a single table with none of the information needed to read it. Table 2 covers one dataset and one split, with point estimates only: no seeds, no intervals. The paper does not state the number of calibration or target environments, how environments are constructed from Cora (the "topology shift split" is not a standard GOOD-Cora split), α, δ, the violation tolerance that defines a false certificate, the form of b_E, the discrepancy D, the signature components, or the backbone behind the table.

  2. The paper's own protocol was not executed. §6 describes a synthetic study (P0), several GOOD/OGB tasks, and a concept-shift stress test; §2 calls WR-CP "a required general-shift baseline." None of these appear. The paper acknowledges this at L154–L156, L186–L188, and L207–L211. Table 1 is a plan of required experiments, presented in place of results.

  3. The baseline numbers are not interpretable as reported. APS, conformalized GNN, and SNAPS construct prediction sets and do not abstain, yet Table 2 gives them acceptance rates of 91.0%, 88.6%, and 86.4%. Whatever abstention rule produced these is not described, so the comparison with TopoTransfer's 74.9% cannot be assessed.

  4. No code. The reproducibility statement says the submission package contains the manuscript source only.

  5. Proposition 1 is trivial and unproved. That no unlabeled statistic can certify coverage follows in one line from the label conditional being arbitrary; the paper lists it as its first contribution and gives no formal argument.

  6. Presentation. The abstract is too dense to parse on a first read. "Unlabeled graph topology" is not standard terminology and is never defined; "registered," "declared family," and "calibrated support" are used before being defined. The ICLR running header is missing. Figure 1 restates §4 without adding information.

Questions for authors
  1. Define "environment" for the GOOD-Cora experiment: how many calibration and target environments, how they were constructed, and what the "topology shift split" is.
  2. Report α, δ, the violation tolerance, b_E, the discrepancy D, the signature components, and the backbone used for Table 2, with seeds and confidence intervals.
  3. How do APS, conformalized GNN, and SNAPS abstain in Table 2?
  4. H1 is stated as a hypothesis. Can you state and prove the coverage guarantee for the envelope under the exchangeability assumption on environments, and say how it relates to hierarchical conformal prediction (e.g., Dunn, Wasserman, Ramdas 2023) and to conformal prediction under covariate shift (Tibshirani et al. 2019; Barber et al. 2023)?
  5. Where are the P0 synthetic results, the remaining GOOD/OGB tasks, and WR-CP?
Reviewer_pz4GOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether conformal prediction sets calibrated on one graph remain valid on a new, unlabeled graph under topology shift. It first presents an observable-equivalence argument showing why unlabeled structure and features cannot universally identify coverage. It then proposes TopoTransfer, which learns a discrepancy-to-miscoverage envelope from labeled environments, conformalizes its residuals, and issues a transfer decision only when support, stability, and risk checks pass. On GOOD-Cora topology shift, it reports a 5.6% false-certificate rate at 74.9% acceptance, versus 16.8% at 86.4% for SNAPS.

Critical strengths
  1. The problem of reusing a calibrated GNN on an unlabeled target graph is practically relevant, and a selective transfer decision is a sensible output.
  2. Using whole environments as the calibration unit and evaluating false certificates jointly with acceptance are reasonable design choices.
Critical weaknesses
  1. Insufficient experiments.
  • The evidence is limited to GOOD-Cora, with one main comparison table and a few briefly reported ablation and edge-rewiring results. The additional GOOD/OGB tasks, backbones, and synthetic experiments described in the protocol are not reported.
  • Signature specificity and safe refusal receive only partial support, while the observable-preserving concept-shift experiment is absent.
  • No error bars, confidence intervals, or number of runs are reported.
  • WR-CP, explicitly described by the authors as a required baseline, is missing.
  • The actual environment counts, environment construction, numerical tolerances, and backbone configuration for the reported experiment are unspecified. The submission package also lacks implementation code, configurations, and raw logs.
  1. Unclear evaluation details. The decision rules and violation scores used to compute baseline acceptance and AUROC are not specified. The selection constraints applied to the label-oracle reference, which accepts 82% of environments, also need clarification.
Questions for authors
  1. How many calibration and target environments are used in the GOOD-Cora experiment, and how are they constructed?
  2. How does TopoTransfer differ from GNNEvaluator (NeurIPS 2023), which also learns to estimate GNN performance from graph-distribution discrepancies? Please clarify the differences in prediction targets, discrepancy representations, and calibration/selection mechanisms.
  3. Can you provide a comparison with WR-CP and results on additional datasets and shift types, with environment-level uncertainty estimates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.