- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies when graph information can be effectively transferred to sequential models in sequential recommendation. The authors show that direct fusion suffers from an irreducible margin, and that this limitation can be mitigated through knowledge distillation based on output distributions. They theoretically characterize the prediction error of three settings (sequence-only, direct fusion, and graph-to-sequence distillation) by deriving high-probability error bounds for each. Building on this analysis, the authors propose a criterion for assessing the distillability. Empirically, the proposed approach achieves state-of-the-art performance on various real-world datasets.
Critical strengths
[S1] The paper provides both a theoretical analysis and empirical observations on representation learning, which aligns well with the scope of ICLR. To the best of my knowledge, this direction has not yet been explored in the context of graph-enhanced sequential recommendation.
[S2] The paper's attempt to approach the problem from a theoretical perspective is positive and valuable.
[S3] The experimental results are averaged over a sufficient number of random seeds, so the reported performance appears statistically reliable.
Critical weaknesses
[W1] The core contributions (the proposed criterion and its theoretical proofs) rely on the alignment residual measured by the orthogonal map defined in Eq. (1). However, I'm afraid I have to strongly disagree with the claim that a measurement based on an orthogonal map is an appropriate metric for capturing representation conflict. The counterexample is intuitive.
Let and denote the user embedding matrices obtained from the graph and sequence, respectively.
Now suppose that for some constant . Then the Eq (1) is reformulated in matrix form:
where denotes the -th row of .
Since is orthogonal, we have
This implies the optimal orthogonal matrix is an identity, . Therefore,
This counterexample shows that when , the two embeddings have no practical difference in representational capacity, yet the orthogonality constraint can produce an arbitrarily large alignment residual driven purely by the difference in scale.
In other words, the residual can reflect a simple scale difference rather than a genuine conflict between the representations. This is a critical issue because the measure is used to motivate the proposed criterion and support the theoretical analysis. If the measure does not capture the claimed conflict, the conclusions based on it become less convincing. The authors therefore need to show that the measure captures representation conflict beyond scale differences. Without this evidence, the paper’s main contribution is substantially weakened.
[W2] The experiments are conducted on relatively small-scale datasets, and the proposed method is not compared against recent graph-based or sequential models (published in 2025 or later).
[W3] I am concerned about potential scalability issues arising from the matrix operations.
[W4] The theoretically developed criterion and the proposed method do not appear to be well connected; they seem to constitute separate contributions.
Questions for authors
[Q1] In line 54, the paper supports the claim that graph and sequence embeddings conflict with each other by citing a work on multimodal collapse. Although the two embeddings are intuitively different, don't sequence embeddings also learn temporal collaborative signals, at least implicitly?
[Q2] What are the GPU VRAM usage and elapsed time during training and inference?
2 comments
Thank you for your responses and for the additional experiments and analyses.
Considering the purpose of knowledge distillation, the absence of large-scale experiments is a critical weakness. In particular, the memory complexity (Appendix E) is practically infeasible at industrial scale. Therefore, W2 and W3 remain unresolved.
In addition, the claim that direct fusion suffers from the alignment residual is mathematically supported only by the lower bound in Eq. (7). Eq. (6), Eq. (8), and the upper bound in Eq. (7) cannot establish this, since an upper bound containing does not imply that fusion actually incurs the penalty. Below, we show, using only the facts presented in the paper, that this lower bound is practically zero. Please correct me if I have misread any constant.
Let denote the penalty term in Eq. (7), so that the lower bound on is
We proceed in three steps.
(i) (line 1069).
(ii) By the covariance stability condition (Eq. 41) and the item-norm bound , so and hence .
(iii) By the triangle inequality used in the proof of Lemma 8 (line 998), for every . Together with Eq. (42), this gives which holds for any optimal scale .
Combining (i)-(iii) and writing ,
This bound holds uniformly in , , , , and . Since , the lower bound in Eq. (7) is numerically vacuous. In particular, it cannot grow with the residual: since by (iii), the factor decays at least as fast as .
Therefore, the lower bound in Eq. (7) never exceeds and does not scale with . The theory certifies at most penalty for direct fusion, and guarantees that KD outperforms fusion only when KD's own error bound in Eq. (8) is already below that level.
Refining Alignment Analysis and Clarifying Computational Cost and Distillability
Thank you for the helpful comments. We address each point below.
1. Alignment under global rescaling (W1, Q1)
We agree that orthogonal-only alignment could mistake a scale difference for a mismatch. Sections 3.1, 3.4, and 4 now use aQ, with a global scale a≥0 shared by all users and Q⊤Q=I. For your example G=kS, k>0, choosing a=1/k and Q=I gives zero residual. Theorem 1 and Appendix B now account for the scale factor, including its effect on graph-side noise in the fusion bound.
Under the revised assumptions, the fusion lower bound still contains a penalty proportional to the residual after scaled alignment, which can persist as estimation noise decreases. Distillation uses shared item-indexed output distributions and avoids embedding alignment. We also agree that sequential models can implicitly learn collaborative signals; the revised claim concerns representation geometry even when the encoders capture overlapping signals.
We repeated Appendix G's analysis using scaled orthogonal alignment, fitting on 70% of users and evaluating on the remaining 30%. Ten independent pairs of training runs established within-model references. On all nine datasets, the held-out graph–sequence residual exceeded the larger within-model reference on average, with mean paired gaps of 0.055–0.167 (Table 8). This supports mismatch beyond global scaling for the models and datasets studied.
2. Recent graph–sequence baseline (W2)
We added LS4SRec (He et al., 2025), a self-supervised graph neural sequential recommender. The comparison covers all nine datasets, including four public benchmarks and five transaction datasets (Section 5.2, Tables 1 and 9).
3. Training and inference cost (W3, Q2)
Section 5.7 and Table 3 report training costs on all nine datasets using one NVIDIA A10 GPU (24 GB) and sparse graph propagation. Mean epoch time was 2.88–29.02 seconds; total wall-clock time, including graph construction and validation, was about 2–32 minutes; peak PyTorch-allocated GPU memory was 0.17–4.52 GiB. All runs fit on one GPU. Inference uses only the sequential student; the teacher is needed during training. Separate inference latency and VRAM measurements remain to be reported.
4. How the criterion informs FROGS (W4)
The criterion screens whether graph-induced variance reduction can offset propagation bias and the student's matching residual before committing to full distillation. This trade-off motivates using the graph branch as a teacher and transferring item-level probabilities to the student.
εopt is the mean per-user L1 distance between teacher and student item distributions. We would estimate it on a held-out subset after a small pilot distillation run, before full training. C is an unspecified bound constant. We would choose a plausible range using prior experience or validation data, then combine a common C with Section 3.4's data estimates in all three proxies in Eq. (3). We would check whether the KD proxy stays below both the sequential and fusion proxies across that range. This is approximate screening, not a calibrated guarantee of improvement.
Unspecified constants also occur in transfer-learning results: Chen et al. (2026, Theorem 4.5) discuss cross-validating a tuning constant; Fan et al. (2025, Theorem 6) suppress a multiplicative constant; Li et al. (2022, Theorem 3) use an unspecified constant in a sufficient condition. These examples distinguish theoretical characterization from empirical calibration.
For fixed data quantities, Eq. (9) defines a linear boundary in the (εopt,C) plane. For an informative neighborhood, less per-user evidence lowers the C required for the KD bound to improve on the sequential bound at a fixed residual. At a chosen C, the boundary gives the largest admissible residual for each dataset. Larger neighborhoods reduce variance but may increase propagation bias (Corollary 3). Thus the criterion provides qualitative guidance and dataset-specific sensitivity analysis without requiring a single calibrated constant.
References
Chen, E., Chen, X., & Jing, W. (2026). Data-driven knowledge transfer in batch Q∗ learning. JASA, 121(553), 276–288.
Fan, J., Gao, C., & Klusowski, J. M. (2025). Robust transfer learning with unreliable source data. Ann. Stat., 53(4), 1728–1752.
Li, S., Cai, T. T., & Li, H. (2022). Transfer learning for high-dimensional linear regression: Prediction, estimation and minimax optimality. JRSS B, 84(1), 149–173.
He, L., Yan, W., Yi, T., & Sun, H. (2025). Self-supervised graph neural sequential recommendation with disentangling long and short-term interest. ACM Trans. Recomm. Syst., 4(1), 1–25.