- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the effective scaling of sequential recommendation models. The authors point out that simply increasing model depth and parameter count does not necessarily lead to consistent improvements in recommendation performance. Key challenges include the underutilization of deep-layer representations, the tight coupling between model capacity and computational cost, and the difficulty of optimizing large-scale models. To address these issues, the paper proposes TriScaleRec, which consists of three core components: First, "Block Attention" improves cross-layer information reuse by adaptively aggregating representations from different depths. Second, "LatentMoE" performs sparse expert computation within a low-dimensional latent space, allowing for an increase in total parameter capacity while keeping the active computational load under control. Finally, a hybrid optimization strategy—combining Muon and AdamW—employs different optimizers for different types of parameters. The authors conducted experiments on four datasets—ML-1M, Amazon, TencentGR-1M, and Taobao UserBehavior—and analyzed the scaling behavior of the model across six configurations of increasing size. Experimental results demonstrate that TriScaleRec outperforms various sequential recommendation baselines on key metrics and exhibits consistent performance gains as the model scale increases. Ablation studies confirm that each of the three components contributes to the final performance.
Critical strengths
- The research problem addressed in this paper is of practical significance. Rather than merely reporting recommendation accuracy at a single scale, the paper explicitly examines whether consistent marginal gains can be achieved as model size increases, and it compares the scaling trends of different architectures across six distinct model scales. This provides a valuable empirical analysis of the scaling behavior of sequential recommendation models.
- The methodology and experimental framework are comprehensive. Techniques such as Block Attention, LatentMoE, and a hybrid Muon/AdamW optimizer address, respectively, the challenges of deep-level utilization, capacity expansion, and the optimization of large-scale models. The paper presents a robust validation pipeline, encompassing overall performance comparisons, component ablation studies, in-depth analysis, evaluations of the number of experts, and assessments of computational efficiency.
Critical weaknesses
- The novelty of the proposed method is limited. "Block Attention" is essentially a form of cross-depth representation aggregation; "LatentMoE" builds upon existing concepts of low-dimensional sparse expert computation; and the use of Muon/AdamW involves applying existing optimizers with parameter grouping. Consequently, the primary contribution of TriScaleRec is better characterized as the combination and adaptation of several existing techniques to the problem of scaling in sequential recommendation, rather than the proposal of a novel modeling principle. The paper should more clearly distinguish between original design contributions and the application or combination of existing methods.
- The fairness of the scaling comparison is not yet fully established. The appendix provides a useful active-parameter-matched analysis, aligning the number of active encoder parameters per token across six budgets. However, matched active parameter counts do not necessarily imply matched inference FLOPs, training compute, wall-clock cost, memory usage, or total parameter count, particularly because embeddings are excluded and the architectures activate different operations. In addition, the largest SASRec configurations require architecture- and scale-specific stabilization, including Pre-Norm, residual scaling, a lower learning rate, and longer training. It therefore remains unclear whether all baselines received comparably thorough tuning and computational budgets. More strictly compute-matched comparisons would strengthen the claim that the observed scaling advantage comes from the proposed architecture rather than differences in optimization or resource allocation.
Questions for authors
- Appendix D provides an active-parameter-matched scaling comparison. Could the authors additionally report results under matched inference FLOPs, training FLOPs or wall-clock budget, peak memory, and total parameter count including embeddings? Does TriScaleRec retain the same scaling advantage under these stricter compute-matched settings?
- How many random seeds were used for each of the main experiments and ablation studies? Could you report the mean, standard deviation, and statistical significance?
- Could you include stronger baselines that utilize the same training objectives, negative sampling strategies, and hybrid optimization schemes? This would help disentangle the contributions of the architecture itself from those of the optimization configuration.
- Since components such as Block Attention, LatentMoE, and hybrid optimization can be applied independently to other models, could you further clarify what makes TriScaleRec unique, specifically, why it cannot simply be replicated by combining these existing components?
5 comments
Author Rebuttal ( Q3, Q4 )
Q3. Could you include stronger baselines that utilize the same training objectives, negative sampling strategies, and hybrid optimization schemes?
- The comparison includes established and recent baselines: SASRec, BERT4Rec, RecBLR, Mamba4Rec, HSTU, STAR-Rec, and BlossomRec. All models follow a common training and evaluation protocol (Appendix B.2 and Table 6). Our Hybrid Muon/AdamW strategy builds on Muon and is an explicit component of TriScaleRec. Forcing it into SASRec, HSTU, or BlossomRec would create new method variants and test the transferability of our optimizer assignment, rather than preserve those baselines' original optimization designs. We instead isolate the optimizer contribution through a direct control in which the TriScaleRec architecture is kept fixed and Muon is removed, leaving all parameters optimized by AdamW.
- On TencentGR-1M, AdamW-only TriScaleRec outperformed HSTU in NDCG@10 and HR@10 (0.7176/0.8779 vs. 0.6911/0.8626 in Tables 1 and 2 of the initial submission). Hybrid Muon/AdamW further raised these scores to 0.7327/0.8885 (Table 2). The advantage under AdamW supports an architectural contribution, while the fixed-architecture ablation isolates the hybrid optimizer's incremental gain.
Q4. Since Block Attention, LatentMoE, and hybrid optimization can be applied independently to other models, what makes TriScaleRec unique, and why can it not simply be replicated by combining these existing components?
-
We first observe that, in conventional ID-based sequential recommendation, directly increasing model depth, parameter count, and computation does not automatically produce sustained performance improvements. We further formulate this scaling problem as three progressively layered bottlenecks. First, can additional network depth be utilized effectively? Second, once depth is utilized effectively, can model capacity continue to scale without a proportional increase in active computation? Finally, as depth and capacity grow together, can different parameter types receive stable updates suited to their respective optimization characteristics? TriScaleRec's central contribution lies in identifying the factors that impede the scaling of sequential recommendation and designing coordinated architectural, computational, and optimization interfaces for them.
-
Block Attention lets added layers reuse representations from earlier blocks. LatentMoE builds on those representations to expand expert capacity while limiting active computation. As the attention and expert modules grow, Hybrid Muon/AdamW applies Muon to selected matrices and AdamW to the remaining parameters. Table 2 reports component ablations, while Fig. 6 and Tables 3–4 examine cross-depth reuse, the capacity–compute trade-off, and optimizer assignment.
-
We do not claim that TriScaleRec “cannot be reproduced”; on the contrary, clear and complete reproduction is a basic requirement of scientific research. Our point is that simply adding cross-layer connections, sparse experts, and Muon to an arbitrary sequential model is not equivalent to TriScaleRec and does not automatically produce the same scaling behavior. A faithful reproduction must preserve the explicit interfaces, ordering, and division of roles among the three components. In other words, TriScaleRec is reproducible, but it cannot be reduced to an architecture-agnostic checklist of components.
Thank you again for taking the time to review our paper. We hope our responses could clarify your concerns, and hope you will consider increasing your score. If we have left any notable points of concern unaddressed, please do share and we will attend to these points.
References:
[1] C. Liu et al. Behavior-Dependent Linear Recurrent Units for Efficient Sequential Recommendation. CIKM, 2024.
[2] J. Zhai et al. Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations. ICML, 2024.
[3] M. Wang et al. STAR-Rec: Making Peace with Length Variance and Pattern Diversity in Sequential Recommendation. SIGIR, 2025.
[4] M. Ma et al. BlossomRec: Block-level Fused Sparse Attention Mechanism for Sequential Recommendations. The Web Conference, 2026.
[5] K. Jordan. Muon: An Optimizer for Hidden Layers in Neural Networks. 2024.
Summary of Changes in the Revised Manuscript
- Variability across training runs (Table 1 and Appendix E.1, Table 9). We added means and sample standard deviations to the main comparisons, together with per-metric improvements over the strongest baseline. For ML-1M, Table 9 also specifies the three training seeds used in the reported comparison: 4, 42, and 43.
- Capacity and active computation (Sec. 3.3 and Appendix G.3). We clarified the roles of the total expert count, the number of active experts, and latent width. The new analysis links the approximation bound to the same latent width that determines projection and active FFN costs, and specifies when the quadratic expert cost dominates the shared projection cost.
- Scaling conclusions (Sec. 4.3 and Fig. 5). We revised the discussion to distinguish absolute ranking performance at each evaluated scale from the gains between scales. The revised wording ties the conclusion to the six configurations evaluated on TencentGR-1M and UserBehavior.
Author Rebuttal ( W1, W2, Q1, Q2 )
We thank the reviewers for reading our paper and providing detailed review on our submission. We respond to the reviewers’ concerns and questions one by one.
W1. Design Contribution.
TriScaleRec's contribution is a coordinated architecture and optimization design for three bottlenecks in sequential recommendation scaling: underuse of added depth, coupling between capacity and active computation, and heterogeneous optimization needs as the model grows. The design specifies how cross-depth representations feed latent experts and how parameter groups are assigned to the two optimizers. The representation analysis and controlled comparisons were presented in Fig. 6 and Tables 2–4 of the initial submission.
W2. Fairness of the Scaling Comparison.
The six approximately matched active-encoder-parameter budgets were presented in Appendix D of the initial submission. The ML-1M comparison of accuracy against inference MFLOPs was presented separately in Fig. 9(b). Appendix B.2 and Table 7 documented the stabilization and longer training schedule used to help the largest SASRec configurations converge. These experiments support the scaling comparison under active-parameter matching. Matched training FLOPs, wall-clock time, peak memory, and total parameters including embeddings would address additional resource constraints.
Q1. Could the authors additionally report results under matched inference FLOPs, training FLOPs or wall-clock budget, peak memory, and total parameter count including embeddings? Does TriScaleRec retain the same scaling advantage under these stricter compute-matched settings?
The scaling results on TencentGR-1M and UserBehavior were presented in Appendix D of the initial submission across six approximately matched active-encoder-parameter budgets ranging from 2.5M to 80M. Under these budgets, TriScaleRec retained the strongest overall performance and a consistently increasing scaling trajectory. The comparison of NDCG@10 against inference MFLOPs per sequence on ML-1M was presented separately in Fig. 9(b) of Appendix F, where TriScaleRec achieved the highest NDCG@10 while maintaining a competitive inference cost. The current paper therefore evaluates efficiency from both active encoder capacity and measured inference FLOPs rather than comparing only unconstrained model sizes. TriScaleRec retains its scaling advantage under the active-parameter-matched budgets in Appendix D and exhibits a favorable performance-inference-FLOPs trade-off in Fig. 9(b). We do not claim that the current evidence has already demonstrated exactly the same ordering under every stricter resource constraint listed in the question. The two analyses capture complementary aspects of efficiency.
Q2. How many random seeds were used for each of the main experiments and ablation studies? Could you report the mean, standard deviation, and statistical significance?
The original experiments used 5 independent training seeds. Evaluation used a shared candidate pool across methods, generated with a fixed evaluation seed.
To keep the original tables concise, we omitted standard deviations across training runs. We have added means and sample standard deviations for the main comparisons to Table 1 and to Table 9 (Appendix E.1) of the revised manuscript. Table 9 reports ML-1M results over three training seeds. Table 2 reports point estimates for the component ablations.