acceptodds
Under review as a conference paper at ICLR 2027

Bigger Is Not Always Better: Towards Effective Scaling of Sequential Recommendation

Abstract

Scaling model size has become an increasingly important approach to improving sequential recommendation. However, prior studies suggest that simply increasing model size does not necessarily lead to consistent performance improvements. As recommenders scale, additional model capacity does not necessarily yield proportional performance gains, while optimization becomes progressively more challenging. Therefore, a central challenge is to improve scaling efficiency and sustain consistent performance gains as model size increases. To address these, we propose TriScaleRec, a scaling-oriented framework for sequential recommendation. Specifically, TriScaleRec improves depth utilization through adaptive cross-depth representation aggregation. On this basis, it expands model capacity with sparse latent-space expert computation while controlling active computation, and further employs structure-aware optimization to sustain effective training at larger scales. Experiments on multiple public benchmarks show that TriScaleRec achieves strong recommendation performance while scaling experiments over six progressively larger model configurations further show that its performance continues to improve more consistently as model size increases.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Sequential Recommendation, Model Scaling, Block Attention, Latent Mixture-of-Experts, Muon, Compute Efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_H6XNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the effective scaling of sequential recommendation models. The authors point out that simply increasing model depth and parameter count does not necessarily lead to consistent improvements in recommendation performance. Key challenges include the underutilization of deep-layer representations, the tight coupling between model capacity and computational cost, and the difficulty of optimizing large-scale models. To address these issues, the paper proposes TriScaleRec, which consists of three core components: First, "Block Attention" improves cross-layer information reuse by adaptively aggregating representations from different depths. Second, "LatentMoE" performs sparse expert computation within a low-dimensional latent space, allowing for an increase in total parameter capacity while keeping the active computational load under control. Finally, a hybrid optimization strategy—combining Muon and AdamW—employs different optimizers for different types of parameters. The authors conducted experiments on four datasets—ML-1M, Amazon, TencentGR-1M, and Taobao UserBehavior—and analyzed the scaling behavior of the model across six configurations of increasing size. Experimental results demonstrate that TriScaleRec outperforms various sequential recommendation baselines on key metrics and exhibits consistent performance gains as the model scale increases. Ablation studies confirm that each of the three components contributes to the final performance.

Critical strengths
  1. The research problem addressed in this paper is of practical significance. Rather than merely reporting recommendation accuracy at a single scale, the paper explicitly examines whether consistent marginal gains can be achieved as model size increases, and it compares the scaling trends of different architectures across six distinct model scales. This provides a valuable empirical analysis of the scaling behavior of sequential recommendation models.
  2. The methodology and experimental framework are comprehensive. Techniques such as Block Attention, LatentMoE, and a hybrid Muon/AdamW optimizer address, respectively, the challenges of deep-level utilization, capacity expansion, and the optimization of large-scale models. The paper presents a robust validation pipeline, encompassing overall performance comparisons, component ablation studies, in-depth analysis, evaluations of the number of experts, and assessments of computational efficiency.
Critical weaknesses
  1. The novelty of the proposed method is limited. "Block Attention" is essentially a form of cross-depth representation aggregation; "LatentMoE" builds upon existing concepts of low-dimensional sparse expert computation; and the use of Muon/AdamW involves applying existing optimizers with parameter grouping. Consequently, the primary contribution of TriScaleRec is better characterized as the combination and adaptation of several existing techniques to the problem of scaling in sequential recommendation, rather than the proposal of a novel modeling principle. The paper should more clearly distinguish between original design contributions and the application or combination of existing methods.
  2. The fairness of the scaling comparison is not yet fully established. The appendix provides a useful active-parameter-matched analysis, aligning the number of active encoder parameters per token across six budgets. However, matched active parameter counts do not necessarily imply matched inference FLOPs, training compute, wall-clock cost, memory usage, or total parameter count, particularly because embeddings are excluded and the architectures activate different operations. In addition, the largest SASRec configurations require architecture- and scale-specific stabilization, including Pre-Norm, residual scaling, a lower learning rate, and longer training. It therefore remains unclear whether all baselines received comparably thorough tuning and computational budgets. More strictly compute-matched comparisons would strengthen the claim that the observed scaling advantage comes from the proposed architecture rather than differences in optimization or resource allocation.
Questions for authors
  1. Appendix D provides an active-parameter-matched scaling comparison. Could the authors additionally report results under matched inference FLOPs, training FLOPs or wall-clock budget, peak memory, and total parameter count including embeddings? Does TriScaleRec retain the same scaling advantage under these stricter compute-matched settings?
  2. How many random seeds were used for each of the main experiments and ablation studies? Could you report the mean, standard deviation, and statistical significance?
  3. Could you include stronger baselines that utilize the same training objectives, negative sampling strategies, and hybrid optimization schemes? This would help disentangle the contributions of the architecture itself from those of the optimization configuration.
  4. Since components such as Block Attention, LatentMoE, and hybrid optimization can be applied independently to other models, could you further clarify what makes TriScaleRec unique, specifically, why it cannot simply be replicated by combining these existing components?
5 comments
AuthorsOpenReview ↗

Author Rebuttal ( W1, W2, Q1, Q2 )

We thank the reviewers for reading our paper and providing detailed review on our submission. We respond to the reviewers’ concerns and questions one by one.

W1. Design Contribution.

TriScaleRec's contribution is a coordinated architecture and optimization design for three bottlenecks in sequential recommendation scaling: underuse of added depth, coupling between capacity and active computation, and heterogeneous optimization needs as the model grows. The design specifies how cross-depth representations feed latent experts and how parameter groups are assigned to the two optimizers. The representation analysis and controlled comparisons were presented in Fig. 6 and Tables 2–4 of the initial submission.

W2. Fairness of the Scaling Comparison.

The six approximately matched active-encoder-parameter budgets were presented in Appendix D of the initial submission. The ML-1M comparison of accuracy against inference MFLOPs was presented separately in Fig. 9(b). Appendix B.2 and Table 7 documented the stabilization and longer training schedule used to help the largest SASRec configurations converge. These experiments support the scaling comparison under active-parameter matching. Matched training FLOPs, wall-clock time, peak memory, and total parameters including embeddings would address additional resource constraints.

Q1. Could the authors additionally report results under matched inference FLOPs, training FLOPs or wall-clock budget, peak memory, and total parameter count including embeddings? Does TriScaleRec retain the same scaling advantage under these stricter compute-matched settings?

The scaling results on TencentGR-1M and UserBehavior were presented in Appendix D of the initial submission across six approximately matched active-encoder-parameter budgets ranging from 2.5M to 80M. Under these budgets, TriScaleRec retained the strongest overall performance and a consistently increasing scaling trajectory. The comparison of NDCG@10 against inference MFLOPs per sequence on ML-1M was presented separately in Fig. 9(b) of Appendix F, where TriScaleRec achieved the highest NDCG@10 while maintaining a competitive inference cost. The current paper therefore evaluates efficiency from both active encoder capacity and measured inference FLOPs rather than comparing only unconstrained model sizes. TriScaleRec retains its scaling advantage under the active-parameter-matched budgets in Appendix D and exhibits a favorable performance-inference-FLOPs trade-off in Fig. 9(b). We do not claim that the current evidence has already demonstrated exactly the same ordering under every stricter resource constraint listed in the question. The two analyses capture complementary aspects of efficiency.

Q2. How many random seeds were used for each of the main experiments and ablation studies? Could you report the mean, standard deviation, and statistical significance?

  • The original experiments used 5 independent training seeds. Evaluation used a shared candidate pool across methods, generated with a fixed evaluation seed.

  • To keep the original tables concise, we omitted standard deviations across training runs. We have added means and sample standard deviations for the main comparisons to Table 1 and to Table 9 (Appendix E.1) of the revised manuscript. Table 9 reports ML-1M results over three training seeds. Table 2 reports point estimates for the component ablations.

AuthorsOpenReview ↗

Author Rebuttal ( Q3, Q4 )

Q3. Could you include stronger baselines that utilize the same training objectives, negative sampling strategies, and hybrid optimization schemes?

  • The comparison includes established and recent baselines: SASRec, BERT4Rec, RecBLR, Mamba4Rec, HSTU, STAR-Rec, and BlossomRec. All models follow a common training and evaluation protocol (Appendix B.2 and Table 6). Our Hybrid Muon/AdamW strategy builds on Muon and is an explicit component of TriScaleRec. Forcing it into SASRec, HSTU, or BlossomRec would create new method variants and test the transferability of our optimizer assignment, rather than preserve those baselines' original optimization designs. We instead isolate the optimizer contribution through a direct control in which the TriScaleRec architecture is kept fixed and Muon is removed, leaving all parameters optimized by AdamW.
  • On TencentGR-1M, AdamW-only TriScaleRec outperformed HSTU in NDCG@10 and HR@10 (0.7176/0.8779 vs. 0.6911/0.8626 in Tables 1 and 2 of the initial submission). Hybrid Muon/AdamW further raised these scores to 0.7327/0.8885 (Table 2). The advantage under AdamW supports an architectural contribution, while the fixed-architecture ablation isolates the hybrid optimizer's incremental gain.

Q4. Since Block Attention, LatentMoE, and hybrid optimization can be applied independently to other models, what makes TriScaleRec unique, and why can it not simply be replicated by combining these existing components?

  • We first observe that, in conventional ID-based sequential recommendation, directly increasing model depth, parameter count, and computation does not automatically produce sustained performance improvements. We further formulate this scaling problem as three progressively layered bottlenecks. First, can additional network depth be utilized effectively? Second, once depth is utilized effectively, can model capacity continue to scale without a proportional increase in active computation? Finally, as depth and capacity grow together, can different parameter types receive stable updates suited to their respective optimization characteristics? TriScaleRec's central contribution lies in identifying the factors that impede the scaling of sequential recommendation and designing coordinated architectural, computational, and optimization interfaces for them.

  • Block Attention lets added layers reuse representations from earlier blocks. LatentMoE builds on those representations to expand expert capacity while limiting active computation. As the attention and expert modules grow, Hybrid Muon/AdamW applies Muon to selected matrices and AdamW to the remaining parameters. Table 2 reports component ablations, while Fig. 6 and Tables 3–4 examine cross-depth reuse, the capacity–compute trade-off, and optimizer assignment.

  • We do not claim that TriScaleRec “cannot be reproduced”; on the contrary, clear and complete reproduction is a basic requirement of scientific research. Our point is that simply adding cross-layer connections, sparse experts, and Muon to an arbitrary sequential model is not equivalent to TriScaleRec and does not automatically produce the same scaling behavior. A faithful reproduction must preserve the explicit interfaces, ordering, and division of roles among the three components. In other words, TriScaleRec is reproducible, but it cannot be reduced to an architecture-agnostic checklist of components.

Thank you again for taking the time to review our paper. We hope our responses could clarify your concerns, and hope you will consider increasing your score. If we have left any notable points of concern unaddressed, please do share and we will attend to these points.


References:

[1] C. Liu et al. Behavior-Dependent Linear Recurrent Units for Efficient Sequential Recommendation. CIKM, 2024.

[2] J. Zhai et al. Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations. ICML, 2024.

[3] M. Wang et al. STAR-Rec: Making Peace with Length Variance and Pattern Diversity in Sequential Recommendation. SIGIR, 2025.

[4] M. Ma et al. BlossomRec: Block-level Fused Sparse Attention Mechanism for Sequential Recommendations. The Web Conference, 2026.

[5] K. Jordan. Muon: An Optimizer for Hidden Layers in Neural Networks. 2024.

AuthorsOpenReview ↗

Summary of Changes in the Revised Manuscript

  • Variability across training runs (Table 1 and Appendix E.1, Table 9). We added means and sample standard deviations to the main comparisons, together with per-metric improvements over the strongest baseline. For ML-1M, Table 9 also specifies the three training seeds used in the reported comparison: 4, 42, and 43.
  • Capacity and active computation (Sec. 3.3 and Appendix G.3). We clarified the roles of the total expert count, the number of active experts, and latent width. The new analysis links the approximation bound to the same latent width that determines projection and active FFN costs, and specifies when the quadratic expert cost dominates the shared projection cost.
  • Scaling conclusions (Sec. 4.3 and Fig. 5). We revised the discussion to distinguish absolute ranking performance at each evaluated scale from the gains between scales. The revised wording ties the conclusion to the six configurations evaluated on TencentGR-1M and UserBehavior.
Reviewer_vkNMOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

TriScaleRec combines cross-depth Block Attention, latent-space sparse experts, and hybrid Muon/AdamW optimization for sequential recommendation. Evaluation covers four datasets, six model scales, ablations, and active-parameter-matched comparisons.

Critical strengths
  1. Improving accuracy per unit of model capacity is an important recommendation problem.
  2. The ablations and active-parameter-matched comparisons help assess the combined design.
Critical weaknesses
  1. Results fall below published baselines. On ML-1M, HSTU [1, Table 4] reports full-ranking NDCG@10/HR@10 of 0.1720/0.3097, versus 0.0906/0.1715 here. TriScaleRec achieves only 0.1234/0.2164, also below the published SASRec result (0.1603/0.2853). Beating these weak reproductions does not establish state-of-the-art performance.

  2. Figure 5 contradicts the scaling narrative. Section 4.3 claims SASRec has weaker large-scale marginal returns. In Figure 5(a), however, L4→L6 improves NDCG by approximately 5.6 percentage points for SASRec, versus 1.4 for TriScaleRec. TriScaleRec's lead over the strongest baseline shrinks from 15.0 points at L1 to 1.4 at L6. This does not support superior large-scale marginal returns.

  3. The theory assumes the key approximation property. Theorem 5 assumes shared-latent expert residual bounds in Eq. (38); Eq. (39) follows by the triangle inequality. It neither establishes small residuals nor relates error to latent dimension. The claimed “controllable” approximation–compute trade-off therefore lacks an error–dimension characterization.

  4. Relevant scaling-law results are missing. OneRec [2] studies model/data scaling; OpenOneRec [3] studies 0.6B–14B models and reports N_opt ∝ C^0.44, D_opt ∝ C^0.56. Neither is cited. The paper does not explain what new scaling conclusion its ID-based encoder establishes beyond these generative recommendation results.

Questions for authors
  1. Why are the ML-1M baselines far below the published HSTU/SASRec results, and does TriScaleRec outperform those results?
  2. How does Figure 5(a) support weaker large-scale marginal returns for SASRec?
  3. What bounds the residual in Eq. (38) as latent dimension changes?
  4. What scaling conclusion is new relative to OneRec/OpenOneRec?
6 comments
AuthorsOpenReview ↗

Author Rebuttal ( W1, W2, W3, W4 )

We thank the reviewers for reading our paper and providing detailed review on our submission. We respond to the reviewers’ concerns and questions one by one.

W1. Comparability with Published ML-1M Results

We thank the reviewer for pointing out the discrepancy with previously published results. We would like to clarify that all baseline results reported in our paper are independently reproduced using the original authors’ implementations under a unified experimental protocol, rather than directly taken from their respective publications. This ensures consistent data preprocessing, training, and evaluation settings across different methods. Since the original studies may adopt different data splits, training configurations, and evaluation protocols, their reported absolute scores are not necessarily directly comparable to ours, even on the same dataset.

Under our unified protocol, TriScaleRec consistently outperforms all evaluated baselines on both NDCG@10 and HR@10. We emphasize that our primary goal is to demonstrate effective and consistent scaling under controlled experimental conditions, rather than claim superiority over results obtained under different protocols.

W2. Interpretation of the Scaling Curves.

Fig. 5(a) of the initial submission showed that SASRec gained more NDCG@10 from L4 to L6 and that TriScaleRec's lead over the strongest baseline narrowed as scale increased. TriScaleRec nevertheless retained higher absolute NDCG@10 than SASRec at all six scales. We have revised the discussion of Fig. 5(a) in Sec. 4.3 to distinguish the absolute performance advantage from the marginal gains between scaling configurations.

W3. Latent Approximation and Computational Cost.

Eq. (39) bounds the layer-output error given the expert residuals in Eq. (38). We have added a constructive bound for shared latent projections under common routing to Appendix G.3 in the revised manuscript and summarized it in Sec. 3.6. It relates the expected squared output error to the input and expert-output spectral tails and the latent FFNs' approximation error. The same latent width determines the cost of the shared projections and active expert FFNs, so the bound can be evaluated at widths that fit a given compute budget.

W4. Relation to Generative Recommendation Scaling Laws.

  • OneRec studies how model size, data, and compute affect training and recommendation performance in an end-to-end generative system. OpenOneRec characterizes the compute-optimal allocation between model size and recommendation data for generative pretraining. TriScaleRec asks a complementary question for conventional ID-based sequential encoders: under a common scaling schedule, what determines whether added depth and capacity translate into higher ranking accuracy?
  • Scaling raises a sequence of design questions. First, can added layers make effective use of representations from earlier depths? Block Attention gives those layers direct access to earlier block outputs. As we expand feed-forward capacity on top of these representations, the next challenge is to contain the computation each input activates. LatentMoE addresses this by adding expert capacity while routing each input to a small subset of experts operating in a compact latent space. Finally, as the combined attention and expert architecture grows, its parameter groups face different optimization needs. Hybrid Muon/AdamW applies Muon to selected matrices in the attention and expert modules and uses AdamW for the remaining parameters. Across six evaluated scales, TriScaleRec improved monotonically on TencentGR-1M and UserBehavior, while some baselines saturated or showed non-monotonic gains (Sec. 4.3 and Fig. 5). The active-parameter-matched comparison in Appendix D provides a complementary check. These results show that, within the tested range, the ranking gains from added scale depend on how the architecture uses that capacity.

References:

[1] Guorui Zhou et al. OneRec Technical Report. arXiv:2506.13695, 2025.

[2] Guorui Zhou et al. OpenOneRec Technical Report. arXiv:2512.24762, 2025.

AuthorsOpenReview ↗

Author Rebuttal ( Q1, Q2, Q3 )

Q1. Why are the ML-1M baselines far below the published HSTU/SASRec results, and does TriScaleRec outperform those results?

Thank you for pointing out the discrepancy. We reproduced all ML-1M baselines, including HSTU and SASRec, using the original authors’ implementations under a shared experimental protocol. All methods used the same data preprocessing, split, and full-catalog evaluation settings. The HSTU study [1, Table 4] also used full-catalog ranking, but followed a different training recipe. These differences matter when comparing absolute scores across studies.


Q2. How does Figure 5(a) support weaker large-scale marginal returns for SASRec?

The question interprets “weaker marginal returns at larger scales” as requiring SASRec's gain between every pair of adjacent scales to be smaller than TriScaleRec's. However, this is not how Figure 5(a) conducts the comparison. The figure examines how effectively different architectures convert added scale into recommendation performance over the complete six-level scaling trajectory, considering not only relative increases but also absolute performance. The paper never states that TriScaleRec must have a larger raw NDCG@10 increment in every adjacent interval. Therefore, using SASRec's larger gain in an individual interval to discount TriScaleRec's advantage over the full scaling range does not fully reflect the overall scaling trajectories and absolute performance differences shown in Figure 5(a).


Q3. What bounds the residual in Eq. (38) as latent dimension changes?

  • Eq. (38) defines the expert-level residual, and Eq. (39) shows how these residuals propagate to the sparse MoE output. The new result in Sec. 3.6 and Appendix G.3 of the revised manuscript constructs shared input and output projections for Lipschitz experts under common routing and bounds the expected squared layer-output error by Tout(ℓ)+(Tin(ℓ)+δℓ)2.T_{\mathrm{out}}(\ell) + \bigl(\sqrt{T_{\mathrm{in}}(\ell)} + \delta_\ell\bigr)^2.

    The two spectral tails in the bound measure the routing-weighted energy discarded by the shared input and output projections. The remaining term measures how closely the latent FFNs approximate the projected experts. If both spectra decay polynomially with exponent p > 1, the spectral tails are bounded by a constant times the latent width raised to the power 1 − p.

  • With the model dimension, number of active experts, and FFN expansion ratio held fixed, the cost of the shared projections grows linearly with latent width, while the cost of the active expert FFNs grows quadratically. A given budget determines which widths are feasible, and the approximation bound can be evaluated at each feasible width.

  • The latent-width sweep from 64 to 512 was reported in Table 11 of the initial submission, along with total and active FFN parameter counts, NDCG@10, and HR@10. Both parameter counts increased with width, while ranking performance was non-monotonic. The default width of 128 provides an empirical balance between ranking performance and active FFN parameter cost. The same sweep appears in Table 12 of the revised manuscript and measures downstream ranking performance, while the theoretical bound concerns layer-output approximation error.

References:

[1] Jiaqi Zhai et al. Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations. ICML, 2024.

AuthorsOpenReview ↗

Author Rebuttal ( Q4 )

Q4. What scaling conclusion is new relative to OneRec/OpenOneRec?

  • We do not attempt to refit the compute-optimal loss law of a generative foundation model. Instead, we ask whether increasing the depth, width, and parameter count of conventional ID-based sequential recommenders automatically translates into higher ranking accuracy and, if not, which mechanisms prevent the added resources from being used effectively. The paper identifies three concrete bottlenecks: insufficient utilization of added depth when propagation is restricted to adjacent layers, capacity-compute coupling in dense feed-forward transformations, and increasingly heterogeneous optimization dynamics across parameter groups as the model grows. These three problems motivate Block Attention, LatentMoE, and Hybrid Muon/AdamW, respectively. In contrast, OneRec studies how training loss and recommendation metrics change as model size, data, features, codebook size, and inference compute increase, while OpenOneRec primarily concludes that generative recommendation pretraining follows a predictable scaling law that is more data-intensive.

  • Our six-scale experiments show that simply enlarging a conventional sequential model does not guarantee smooth improvements in ranking metrics: competing architectures may saturate, fluctuate, or change non-monotonically, whereas TriScaleRec converts additional depth and capacity into ranking performance more consistently. Thus, OneRec/OpenOneRec characterize how generative recommendation loss changes with model, data, and compute, while TriScaleRec studies which architectural and optimization properties allow conventional sequential recommenders to benefit continuously from increased scale. This distinction is explained in Secs. 1 and 3 and evaluated in Secs. 4.3-4.4 and Appendix D.

Thank you again for taking the time to review our paper. We hope our responses could clarify your concerns, and hope you will consider increasing your score. If we have left any notable points of concern unaddressed, please do share and we will attend to these points.

References:

[1] Guorui Zhou et al. OneRec Technical Report. arXiv:2506.13695, 2025.

[2] Guorui Zhou et al. OpenOneRec Technical Report. arXiv:2512.24762, 2025.

Reviewer_S6swOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses scaling in sequential recommendation. It observes that simply enlarging a SASRec-style model does not reliably improve accuracy and can destabilize training, and it proposes TriScaleRec, which combines three ingredients: Block Attention for more stable deep stacking, Latent Mixture-of-Experts (LatentMoE) in which experts operate in a low-dimensional latent space, and targeted Muon optimization applied only to selected matrices, with AdamW elsewhere. The default configuration uses 8 layers, 8 heads, 16 experts with top-4 routing, and a latent dimension of 128. Experiments on ML-1M, Amazon, TencentGR-1M, and UserBehavior, using HR@10 and NDCG@10, compare TriScaleRec with several sequential-recommendation baselines and with SASRec across six model scales. TriScaleRec is best on all eight dataset-metric combinations in the main table and continues to improve with scale where SASRec plateaus or degrades.

Critical strengths
  1. A clear and practically relevant scaling study with a coherent combined design. The paper does not merely report a single larger model: it tests six scales, reports an active-parameter-matched scaling analysis in the appendix, and shows TriScaleRec best on all eight dataset-metric combinations in Table 1, with large margins on some datasets (for example, ML-1M NDCG@10 of 0.1234 and Amazon NDCG@10 of 0.2096). 2. Useful ablations that separate the components. Table 2 shows that removing Block Attention, Muon, or LatentMoE each reduces NDCG@10 from 0.7327 in the reported ablation setting (to 0.7216, 0.7176, and 0.7273 respectively), and Table 3 frames LatentMoE honestly as a parameter-efficiency trade-off: it uses 13.7M active parameters versus 50.4M for a standard MoE, while remaining close in accuracy and beating a budget-matched MoE.
Critical weaknesses
  1. Evaluation reliability is weaker than the headline margins suggest. Except for ML-1M, which uses full ranking, the protocol uses sampled ranking (1,000 validation negatives and 10,000 test negatives), which can bias HR@10 and NDCG@10 comparisons, and the main tables report single-seed results (training seed 42, evaluation seed 5) with no variance or significance testing. The ablation gaps of about 0.005-0.015 in NDCG@10 therefore cannot be judged as reliable, and the average-improvement figures, which are computed against the average baseline rather than the best baseline, overstate the margin over the strongest competitor. 2. Novelty and baseline confounding limit the strength of the scaling claim. The contribution is mainly a combination of existing ingredients (block-style attention, LatentMoE, and Muon), and the SASRec baseline was itself modified for stability (including Post-Norm and QK-Norm), so part of the SASRec scaling failure may reflect the modified baseline rather than the original model. No overall inference-latency or FLOPs comparison at matched compute against the baselines is reported in the main results, beyond step times in the Muon table.
Questions for authors
  1. Could the authors report full-ranking results (or a full-ranking validation) for the datasets currently evaluated with sampled ranking, and repeat the main and ablation experiments over multiple seeds with error bars or significance tests, so that the 0.005-0.015 ablation gaps can be assessed? 2. How does TriScaleRec compare with the strongest baseline on inference latency and FLOPs at matched accuracy or matched compute, and what is the margin over the best baseline (not the average baseline) in Table 1? 3. To isolate the scaling claim, could the authors compare TriScaleRec with an unmodified SASRec and with a SASRec that receives only the stability modifications, so that the effect of the proposed components is separated from the effect of baseline changes?
4 comments
AuthorsOpenReview ↗

Author Rebuttal ( W1, W2, Q1 )

We thank the reviewers for reading our paper and providing detailed review on our submission. We respond to the reviewers’ concerns and questions one by one.

W1. Evaluation Reliability.

The sampled-ranking protocol in the initial submission followed SASRec and BERT4Rec and used substantially larger candidate sets of 1,000 validation negatives and 10,000 test negatives. All methods shared the same candidate pools, and ML-1M was evaluated using full-catalog ranking. The original experiments used 5 independent training seeds. The original tables omitted standard deviations for concision, and we have added them to Table 1 and to Table 9 (ML-1M) in Appendix E.1 of the revised manuscript. The “Avg. Improv.” row was explicitly defined relative to the average baseline rather than the strongest baseline. We have also added best-baseline improvements to Tables 1 and 9. TriScaleRec achieved higher NDCG@10 and HR@10 on all four datasets.

W2. Novelty, SASRec Stabilization, and Efficiency.

TriScaleRec’s novelty lies not in independently inventing three modules, but in their coordinated design for three progressively emerging scaling bottlenecks: underutilized depth, capacity–active-computation coupling, and heterogeneous optimization. To our knowledge, this is the first sequential recommendation work to study scaling jointly from these perspectives. The stabilized SASRec configurations were included in Fig. 5. The active-parameter-matched comparison was presented in Appendix D, and the inference-MFLOPs comparison was presented in Appendix F.

Q1. Can full-ranking evaluation and multiple training seeds establish the reliability of the main results and ablation gains?

  • We have added full-catalog ranking results to Table 15 (Appendix E.7) of the revised manuscript. We randomly sampled 100,000 test records per dataset and used the same valid records for all reported models within each dataset: 100,000 each for Amazon and TencentGR-1M and 99,993 for UserBehavior. Under full-catalog ranking, TriScaleRec achieved the highest NDCG@10 and HR@10 on all three datasets among the models with completed evaluations. The initial submission already reported full-catalog results on ML-1M.
  • The original experiments used 5 independent training seeds. We omitted standard deviations from the original tables for concision and have added means and sample standard deviations to Table 1 and to Table 9 (ML-1M) in Appendix E.1 of the revised manuscript.

References:

[1] Wang-Cheng Kang and Julian McAuley. Self-Attentive Sequential Recommendation. ICDM, 2018.

[2] Fei Sun et al. BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. CIKM, 2019.

AuthorsOpenReview ↗

Author Rebuttal ( Q2, Q3 )

Q2. What are the gains over the strongest baseline, and how does inference cost compare at matched accuracy or compute?

  • We have added relative improvements over the strongest baseline for each metric to Table 1 and to Table 9 (ML-1M) in Appendix E.1 of the revised manuscript. The results are summarized below:

    DatasetBest BaselineNDCG@10 ImprovementHR@10 Improvement
    ML-1MBlossomRec (NDCG@10), BERT4Rec (HR@10)+1.32%+3.20%
    AmazonRecBLR+6.93%+4.91%
    TencentGR-1MHSTU+5.32%+2.53%
    UserBehaviorHSTU+4.97%+5.05%

    These results demonstrate that TriScaleRec consistently outperforms the strongest baseline on all four datasets and across both ranking metrics. The observed gains therefore remain substantial even when the comparison is made against the best-performing competitor rather than the average baseline.

  • In the initial submission, Fig. 9(b) in Appendix F reported NDCG@10 and inference MFLOPs per sequence for each model’s best-performing configuration on ML-1M. TriScaleRec achieved higher NDCG@10 than BlossomRec, the strongest baseline on ML-1M, at a competitive inference cost. A direct comparison at matched accuracy or matched compute would require a separate benchmark with measured inference latency.

Q3. How does TriScaleRec compare with original SASRec and with SASRec using only the stability modifications?

  • As explained in Appendix B.2, directly scaling the original SASRec configuration led to unstable convergence. At L5–L6, we therefore used Pre-LN and residual scaling of 0.5 (Table 7), reduced the learning rate to 1 × 10⁻⁴, and extended training to 20 epochs (Appendix B.2). Fig. 5 in the initial submission included these stabilized SASRec configurations. TriScaleRec exhibited more consistent gains with increasing scale on TencentGR-1M and UserBehavior even when compared with SASRec receiving these stabilization measures.
  • The ablation results were presented in Table 2 of the initial submission. Removing Block Attention, Muon, or LatentMoE individually reduced NDCG@10 on TencentGR-1M by 0.0111, 0.0151, and 0.0054, respectively. These ablations support the contribution of each component within TriScaleRec. A comparison of original SASRec, stabilized SASRec, and TriScaleRec at the same scales is still needed to quantify the effect of the stability modifications separately.

If any concern still remains that might prohibit a positive recommendation of this work, we would appreciate if you could let us know now.

References:

[1] Wang-Cheng Kang and Julian McAuley. Self-Attentive Sequential Recommendation. ICDM, 2018.

AuthorsOpenReview ↗

Summary of Changes in the Revised Manuscript

  • Main results and variability (Sec. 4.2, Table 1, and Appendix E.1, Table 9). We now report means and sample standard deviations for NDCG@10 and HR@10, along with the relative improvement over the strongest baseline for each metric. Table 1 covers Amazon, TencentGR-1M, and UserBehavior. Table 9 reports the ML-1M full-catalog results across three training seeds.
  • Full-catalog evaluation on the larger datasets (Appendix E.7, Table 15). We added results for all seven baselines and TriScaleRec on fixed random subsets drawn from 100,000 test records per dataset. TriScaleRec achieved the highest NDCG@10 and HR@10 on all three datasets. On UserBehavior, BERT4Rec was evaluated on 100,000 valid records and the other models on 99,993.
  • Interpretation of the SASRec scaling curves (Sec. 4.3 and Fig. 5). We revised the discussion to acknowledge SASRec’s larger gains in some later scaling intervals and to distinguish those gains from TriScaleRec’s higher absolute performance across the six evaluated scales.
AuthorsOpenReview ↗

Follow-up on Our Responses and Revised Manuscript

Dear Reviewer S6sw,

Thank you for your careful review and constructive comments. We have addressed your concerns in our responses and revised manuscript.

With less than one day remaining in the discussion period, we would appreciate any further feedback, particularly on whether any concerns remain unresolved. We would be happy to provide additional clarification.

Thank you again for your time and consideration.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript and Summary of Changes

Dear Area Chair and Reviewers,

Thank you for your careful reading and feedback. We have revised the manuscript to clarify the scaling conclusions, expand the evaluation, and strengthen the LatentMoE analysis. All section, figure, and table numbers below refer to the revised manuscript.

  1. Expanded the reporting of the main results (Reviewers S6sw and H6XN). Section 4.2 and Table 1 now report means and sample standard deviations for NDCG@10 and HR@10 on Amazon, TencentGR-1M, and UserBehavior. Table 1 reports relative gains over the strongest baseline separately for each dataset and metric.
  2. Clarified the ML-1M comparison (Reviewer vkNM). Appendix E.1 and Table 9 now present the ML-1M full-catalog results separately, reporting means and sample standard deviations across training seeds 4, 42, and 43. Appendix E.1 explains how preprocessing, data splits, candidate eligibility, and checkpoint selection affect comparisons with published scores. Under the shared protocol, TriScaleRec achieves the highest mean on both metrics.
  3. Added full-catalog evaluation on three larger datasets (Reviewer S6sw). Appendix E.7 and Table 15 report results for TriScaleRec and all seven baselines on fixed random subsets drawn from 100,000 test records per dataset. TriScaleRec achieves the highest NDCG@10 and HR@10 on Amazon, TencentGR-1M, and UserBehavior. On UserBehavior, BERT4Rec was evaluated on 100,000 valid records and the other models on 99,993.
  4. Clarified the interpretation of the scaling curves (Reviewer vkNM). Section 4.3 now distinguishes gains between scaling levels from absolute performance at each level. The revised discussion of Fig. 5 acknowledges that SASRec makes substantial gains in some later intervals, while TriScaleRec maintains higher absolute performance across all six evaluated scales.
  5. Extended the LatentMoE approximation and computation analysis (Reviewer vkNM). Section 3.3 clarifies how the expert count, active expert count, and latent width affect capacity and computation. Section 3.6 and Appendix G.3 present a new constructive bound on the expected squared layer-output error under shared routing. The bound depends on the input and expert-output spectral tails and the latent FFN approximation error. Appendix G.3 also shows how the same latent width determines projection and active FFN costs, allowing the bound to be evaluated under a specified active-computation budget.

Thank you for the opportunity to revise the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.