- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes counterfactual supervision for adaptive GNNs, where each node uses learned switches to control how it receives neighbor information. For a selected switch, the method sets it to each of its two extremes, recomputes the remaining layers, and uses the loss difference as an extra training signal for the controller. To reduce cost, nodes that are far apart are tested together, which gives exactly the same results as testing them one by one. The method is implemented in TriAxis-GNN, which has three switches per node, and experiments on homophilous and heterophilous datasets show small but consistent gains.
Critical strengths
1.Clear motivation. The paper shows that local gradients and endpoint comparisons can prefer opposite settings, and measures 10% to 30% disagreement in real models, which makes the problem concrete.
2.Exact and efficient batching. Testing distant nodes together gives the same results as testing them one by one, while greatly reducing training cost.
3.Applicable to other models. The method only changes training, and also improves another adaptive GNN.
Critical weaknesses
1.The value of the counterfactual supervision signal is unclear. The paper’s core claim is that counterfactual supervision improves adaptive GNNs, but in Table 1 most of the gain comes from the TriAxis architecture, while counterfactual supervision adds only small improvements. Table 2 further shows that random targets already give most of the gain on Mowst, suggesting that improvements come mainly from the additional router update steps.
2.Key control experiment is missing on the main model. The comparison with random and constant targets is only conducted on Mowst with four datasets, and not on TriAxis, the main model of the paper, so it remains unclear whether counterfactual supervision outperforms simple auxiliary targets in the proposed architecture.
3.Limited theoretical support for the core claim. Proposition 1 only shows that endpoint comparisons and local gradients can disagree, but not that endpoint comparisons are the better signal. Figure 3(a) shows that over 90% of loss curves in TriAxis are approximately monotone, where both signals point to the same endpoint, so counterfactual supervision provides little new information beyond the gradient. Only in the remaining cases can it add new information, but Figures 3(b) and 3(c) show that in these cases the best value often lies between the endpoints and can yield notably lower loss, whereas counterfactual supervision pushes the control toward an endpoint and may move it in the wrong direction. This may explain why counterfactual supervision brings only small gains in Table 1.
Questions for authors
(1) Can you report the training time of TriAxis compared with GCN and other adaptive baselines?
Thank you to the authors for their detailed response. My questions have been answered adequately, and I appreciate the additional results provided. I have just a couple of points to discuss further regarding the evaluation setting. Firstly, could the authors explain why they chose not to use the fixed splits provided with the datasets by Platonov et al. 2013? That would make the comparison with other works easier and is considered a standard practice. Secondly, why is TriAxis's evaluation limited to shallow models? I understand that the goal is to show the improvement over the case that does not employ TriAxis rather than achieve SOTA results. However, even for a given architecture, if similar (or even better) performance than the shallow model can be achieved by simply using a deeper model, why would TriAxis need to be employed? I'm trying to understand the scope of TriAxis's applicability and gains.