- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper targets few-shot node classification on Text-Attributed Graphs (TAGs) using GNNs and LLMs. The authors critique existing methods for relying on a fixed teacher (e.g., GNN-as-Judge) and propose \textbf{FairRouter}. FairRouter extracts multi-modal evidence across probability, topology, and gold prototypes, decomposes routing into a trust score () and a preference score (), augments few-shot support views, selects pseudo-labels, and trains a multi-modal residual MLP to fuse expert distributions.
Critical strengths
Symmetrical, node-adaptive routing between GNNs and LLMs is inherently more principled than hardcoding one modality as an infallible judge.
The paper provides extensive breakdowns of pseudo-label precision, agreement/disagreement partitions, and hyperparameter sensitivity.
Critical weaknesses
The main paper presents FairRouter as a unified framework, but the appendix reveals an extreme patchwork of hand-picked heuristics. As shown in Tables 4 and 5: Agreement scoring uses unlearned geometric mean on Cora, raw LLM confidence on Citeseer, logistic regression on PubMed, and Extra Trees ensembles on Arxiv/Products. Disagreement scoring pairs an RBF-SVM on Cora, a transfer neural network on Citeseer, an Extra Trees model on PubMed, and a joint neural net on Arxiv. This is not a generalizable representation learning method; it is a collection of dataset-specific Kaggle-style recipes overfitted to individual benchmarks.
Lines 1085--1088 and 1124--1126 admit that for PubMed and Products, the router classifiers are trained on the other four datasets (Source training), pooling thousands of labeled nodes from large graphs like Arxiv. Calling this a ''few-shot setting'' () while secretly injecting supervised cross-dataset pretraining into the routing classifiers is an unfair evaluation against strictly few-shot baselines.
The router relies on a massive stack of 43 to 70 hand-engineered tabular statistics (Appendix A: Bhattacharyya overlap, Jensen-Shannon divergence, sentinel padding distances of 2.0, empirical rank transforms, and arbitrary feature clipping intervals or ). Such extreme heuristics obscure whether empirical gains stem from principled multi-modal synergy or brittle feature-level hand-crafting.
In Table 7, the validation-selected acceptance ratios exhibit catastrophic instability across random seeds. For instance, on Arxiv 3-shot, swings wildly from 0.25% (Seed 43) to 1.0% (Seed 42) to 100.0% (Seed 44). A selector whose decision swings by two orders of magnitude between different 3-shot splits of the same dataset indicates extreme sensitivity to the few-shot support drawing.
Questions for authors
On ogbn-products, FairRouter consistently loses to GNN-as-Judge by 2.67--3.57 pp across all budgets (Table 1). The authors attribute this to GNN-as-Judge fine-tuning the LLM while FairRouter leaves it frozen. Furthermore, Section D.2 admits that the authors modified GNN-as-Judge's official implementation by disabling its 'retrain_gnn' component before evaluation. Modifying a competing baseline while underperforming against it on complex graphs weakens the core empirical claims.
5 comments
Response for reviewer 6GXE (2/4)
Weakness 2: secretly injecting supervised cross-dataset pretraining.
We thank the reviewer for pointing out this important question. We respectfully clarify that cross-dataset pretraining is confined to the router, not the classifier, transferring expert-reliability patterns rather than class information (Appendix B), and cross-dataset pretraining has also been deployed in prior work on few-shot graph learning [1,2]. We agree that the main text should have described the cross-dataset pretraining more prominently, and we have added it to Section 3.2 of the revised paper (paragraph “Unified Pretraining”, marked in red).
Following the reviewer’s suggestion in Ethics Concern 1, we evaluate FairRouter without cross-dataset pretraining and fitted only on the target k-shot support. As shown in Table W2-1, removing pretraining slightly lowers average accuracy. However, target-only FairRouter still outperforms strongest baseline GNN-as-Judge on average and beats LLM-GNN and Locle in all 15 settings under this strict protocol.
Table W2-1. Removing cross-dataset pretraining results and comparisons.
| Shot | Method | Cora | Citeseer | PubMed | Arxiv | Products | Avg. |
|---|---|---|---|---|---|---|---|
| 3 | LLM-GNN | 73.37±1.76 | 66.80±0.65 | 73.37±0.68 | 47.40±1.48 | 54.93±0.45 | 63.17 |
| 3 | Locle | 74.30±1.90 | 69.30±0.37 | 72.97±1.92 | 51.70±0.98 | 63.80±1.63 | 66.41 |
| 3 | GNN-as-Judge | 74.37±1.19 | 64.07±1.82 | 88.93±1.30 | 53.80±3.82 | 75.00±1.71 | 71.23 |
| 3 | FairRouter (paper) | 79.37±0.17 | 73.37±0.12 | 90.80±1.04 | 58.13±0.12 | 72.30±0.85 | 74.79 |
| 3 | FairRouter (Target-only) | 79.17±0.54 | 72.63±0.92 | 87.47±2.13 | 56.70±0.75 | 71.00±1.30 | 73.39 |
| 5 | LLM-GNN | 74.30±2.06 | 67.30±0.57 | 74.90±1.47 | 48.83±1.05 | 54.73±0.83 | 64.01 |
| 5 | Locle | 75.77±1.44 | 69.07±0.56 | 76.87±1.79 | 52.37±0.84 | 64.07±0.63 | 67.63 |
| 5 | GNN-as-Judge | 76.20±2.33 | 65.90±4.19 | 90.80±1.55 | 59.90±1.22 | 75.10±0.95 | 73.58 |
| 5 | FairRouter (paper) | 80.70±1.91 | 72.60±0.79 | 91.47±1.30 | 60.40±1.56 | 72.43±0.41 | 75.52 |
| 5 | FairRouter (Target-only) | 79.27±0.99 | 73.17±1.11 | 90.40±0.36 | 59.93±2.10 | 71.60±0.65 | 74.87 |
| 10 | LLM-GNN | 76.47±1.50 | 68.33±0.95 | 75.90±1.28 | 52.23±1.82 | 55.13±0.39 | 65.61 |
| 10 | Locle | 78.30±1.30 | 68.87±0.52 | 76.40±1.69 | 56.23±1.13 | 65.37±1.14 | 69.03 |
| 10 | GNN-as-Judge | 79.17±1.23 | 68.17±1.01 | 91.43±1.10 | 58.80±5.40 | 76.37±0.65 | 74.79 |
| 10 | FairRouter (paper) | 81.27±1.36 | 73.47±1.25 | 91.63±0.83 | 61.60±0.86 | 72.80±0.82 | 76.15 |
| 10 | FairRouter (Target-only) | 80.67±1.62 | 72.97±0.85 | 90.53±2.02 | 60.57±0.69 | 72.13±1.65 | 75.37 |
[1] Zhao, Haihong, et al. "All in one and one for all: A simple yet effective method towards cross-domain graph pretraining." Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2024.
[2] Liu, Hao, et al. "One for all: Towards training one graph model for all classification tasks." International conference on learning representations. Vol. 2024. 2024.
Response for reviewer 6GXE (3/4)
Weakness 3: whether empirical gains stem from principled multi-modal synergy or brittle feature-level hand-crafting.
We thank the reviewer for raising this feature-level concern. We would like to clarify that we use a fixed set of interpretable statistics to provide multi-level evidence, allowing the router to learn their relative importance rather than prescribing hand-crafted rules.
To clarify this, we added an ablation study to assess the contribution of each feature group. The ablation shows that more information comes from cross-modal probability evidence (Bhattacharyya overlap, JS divergence), and sentinel padding distances of 2.0 fall into the prototype group, which is not the main source of gains (Table W3-1). This indicates that performance does not come from accumulating features, nor from manually selecting a few effective ones; instead, the router itself learns to weigh which evidence is useful.
Table W3-1. Performance change removing each feature group.
| Removed feature group: accuracy change (pp) | 3-shot | 5-shot | 10-shot |
|---|---|---|---|
| Probability features | −1.71 | −0.77 | −1.04 |
| Prototype features | −0.07 | −0.17 | −0.23 |
| Neighborhood features | −0.05 | −0.37 | −0.35 |
| Representation projections | −0.19 | −0.44 | −0.10 |
The specific constants reviewer mention have simple roles:
- The sentinel value 2.0 is the maximum possible cosine distance (1 − cos ∈ [0, 2]). It encodes "no reference available" as "as far as possible".
- The clipping ranges are outlier guards applied after standardization.
Weakness 4: the validation-selected acceptance ratios exhibit catastrophic instability.
1. Can the authors explain why Arxiv 3-shot acceptance jumps from 0.25% to 100% between Seed 43 and Seed 44?
We thank the reviewer for highlighting this variation. Actually, the jump is in agreement branch and follows the validation-based Wilson criterion. For Seed 43, the top 0.25% is 27/27 correct, giving a Wilson lower bound of 0.9089 versus 0.8179 at full coverage. For Seed 44, the top 0.25% is only 18/24 correct, while full coverage reaches 7,323/8,798 correct and the highest Wilson bound (0.8257), so 100% is chosen. The ratio jump reflects the current selection rule’s sensitivity to validation noise, but this does not imply that the router ranks unreliably.
2. How can users trust this router in deployment?
We thank the reviewer for this practical question. To show that the router is reliable enough for deployment, we removed the validation-based ratio search entirely and ran FairRouter with one fixed ratio ρ_A = ρ_D = 15% across every dataset and settings. The results show that a single fixed ratio already matches or exceeds validation-selected ratios, and the final accuracy is relative stable across seeds. Users can therefore deploy FairRouter without per-dataset or per-split tuning.
Table W4-1. Performance on Arxiv change fixing acceptance ratio (15%).
| Result | 3-shot | 5-shot | 10-shot |
|---|---|---|---|
| Arxiv: original accuracy (%) | 58.13±0.12 | 60.40±1.56 | 61.60±0.86 |
| Arxiv: fixed-ratio accuracy (%) | 58.47±1.08 | 60.90±1.44 | 61.60±0.83 |
| Average accuracy change (pp) | +0.34 | +0.50 | +0.00 |
Response for reviewer 6GXE (1/4)
We sincerely thank the reviewer for recognizing our principled symmetric, node-adaptive routing and extensive analyses of pseudo-label precision and hyperparameter sensitivity. We carefully address each concern below and have incorporated the suggested improvements into our revised manuscript.
Weakness 1: not a generalizable representation learning method.
We thank the reviewer for highlighting the issue with dataset-specific implementations. We respectfully clarify that FairRouter unifies Trust–Preference factorization and node-adaptive routing mechanism, with different scorers serving as alternative implementations within this shared framework.
Following the reviewer's suggestion in Ethics Concern 2, we evaluated a single, unified router universally across all five datasets: logistic regression on the same probability-derived compact features, with both acceptance ratios fixed. As shown in Table W1-1, FairRouter still outperforms competing baselines when scorer families, features, and acceptance ratios are fixed across datasets.
Table W1-1. Uniform configuration results and comparisons.