- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies regional imbalance in GNN–LLM feedback for text-attributed graphs: even when average accuracy improves, some graph regions may still receive unreliable preference feedback. The authors construct regions using spectral clustering of frozen GNN representations, the strength of text priors, and local structural consistency. They distinguish three controls: stabilized importance weights regulate regional contributions; prediction sensitivity under degree-preserving rewiring filters preference pairs arising from model disagreements; and harmful-feedback rates estimated on a calibration set, together with structural instability, determine region-specific KL-DRO radii. The final objective is a weighted robust DPO loss over admitted preference pairs, with a small chosen-label NLL term.
Critical strengths
- The problem formulation and the distinct levels of intervention are valuable. Separating which feedback enters training from how each region influences training addresses local failures that global confidence filtering may overlook. C1, C2, and C3 act on regional contributions, the admission support set, and the within-region loss distribution, respectively, making their effects individually testable. The authors also explicitly acknowledge that structural sensitivity does not imply label correctness and that TV diagnostics do not establish KL coverage, avoiding overly strong theoretical claims about empirical mechanisms.
Critical weaknesses
-
The worst-region gains rely on published GNN-as-Judge results without guaranteed common splits or test nodes (Appendix H.1), and their mapping to the frozen (R,T,H) partition is unclear. Rerun Co-teaching lacks worst-region, P10, and CVaR results. Thus, average-accuracy gains are supported, but the central regional-reliability claim remains insufficiently established.
-
The minibatch implementation’s correspondence to the KL-DRO objective is unclear. With batch size 8 and 828 regions on ogbn-arxiv, a region represented by one loss atom has radius-independent risk. Sampling may also alter regional contributions. Appendix K lacks sufficient sampling, cross-batch aggregation, and correction details to connect minibatch risks to Equations (6)/(7).
-
Sparse calibration weakens the interpretation of regional adaptation. On ogbn-arxiv, 590/828 regions use global harmful-rate estimates, while 82% of admitted training pairs under strict kC rely on CF-proxy/global fallback. Aggregate results do not separate gains from regional estimates, global fallback, and structural proxies. Results by support condition and a precise CF-proxy mapping are needed to substantiate the adaptive-radius mechanism.
Questions for authors
-
Could you report worst-region accuracy, P10, CVaR, and intervals for same-seed differences for the rerun Co-teaching baseline, using common test nodes and the same frozen (R,T,H) partition? What node-level or group-level records underlie the “Published” GNN-as-Judge regional results in Table 10, and how are they mapped to your partition?
-
With a batch size of 8, which loss atoms actually enter each regional risk in Equation (7)? Please specify the per-batch objective, sampling distribution, information carried across batches, and treatment of absent regions, and explain how these correspond to Equation (6). What fraction of region occurrences within batches contain only one loss atom, and how does the radius affect those cases?
-
Please distinguish the number of admitted calibration pairs from the number of admitted optimization pairs, and report estimation quality and the gains from adaptive versus fixed radii separately for empirical estimates, shrinkage estimates, and global fallback. How is CF-proxy numerically mapped to a harmful-feedback rate under strict kC, and what is the basis for its parameters? Can this proxy be independently validated in the setting with a held-out label pool?
-
At matched admission coverage and optimization budgets, how does CFdep gating compare with random filtering and filtering based only on calibrated confidence or confidence gaps, in terms of preference quality and final accuracy? Could you also clarify whether the different filters change regional support, so that the additional value of structural information can be assessed?
-
How many full training runs, or how much selection compute, do the 48 candidate configurations actually require, and what are the corresponding budgets for Co-teaching and the other rerun baselines? Does the average-accuracy gap remain stable with comparable selection compute or an increased search budget for Co-teaching?