acceptodds
Under review as a conference paper at ICLR 2027

One Trajectory, Many Targets: Supervision Density and the Compute-Optimal Regime in Sequential Recommendation

Abstract

Sequential recommendation turns each user history into several next-item prediction tasks. Prefix-wise training encodes overlapping histories repeatedly, so grouping more targets into one update can save computation. The central question is how this supervision density changes the updates and compute needed to reach a fixed ranking quality. We identify an empirical scaling law and make its denser updates executable through exact shared computation. In five-seed MovieLens 1M (ML-1M), with the data, model, objective, initialization, and target-event budget fixed, quality trajectories align after updates are rescaled by density; threshold crossings follow an inverse-density relation after a shared offset of about 202 updates. At 0.966 million target events, m=2 reaches normalized discounted cumulative gain at rank 10 (NDCG@10) of 0.1519 in 102.8 training seconds, versus 0.1498 in 202.5 seconds for m=1. Fixed-state scans and five-seed SASRec sweeps show that the best density depends on the compute regime: the measured frontier peaks at m=4 (0.1201 NDCG@10) on ML-10M and improves through the tested m=16 (0.0463) on Amazon Beauty. *Exact Prefix-Loss Compilation* (EPC) evaluates aligned prefix losses in one causal pass while preserving a fixed update's loss and gradient when the states and training semantics match. Across 48 matched backbone–dataset comparisons, packing shortens train-to-best time; on Beauty, the combined configuration improves NDCG@10 by 52.8% and reaches its selected checkpoint 50.9× sooner. On ML-10M, four backbones reach matched quality in all five seeds with 1.30–4.56× fewer target events and 28.70–87.42× less wall time.

Primary area:
optimization
Keywords:
sequential recommendation, exact prefix-loss compilation, causal sequence modeling, packed training, sample efficiency, contrastive learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_2kqtOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper is about how the number of targets from each user history affects training efficiency in sequential recommendation. It proposes a supervision-density scaling law and an allocation rule based on gradient variance and computational cost. The authors also introduce Exact Prefix-Loss Compilation (EPC), which shares computation across prefix losses.

Critical strengths

a) The paper addresses a useful training-efficiency problem. b) The experiments are extensive, covering multiple backbones and datasets with five seeds.

Critical weaknesses

a) The paper is poorly written, making it difficult for a human reviewer to follow the logic. For example, the introduction does not develop the motivation, research gap, and connections between the main contributions. Technical details and numerical results are introduced before the underlying ideas are explained. This leaves readers to reconstruct the argument themselves. b) The novelty of certain components needs clearer justification. For example, the scaling law (Equation (5)) is similar to the relation in An Empirical Model of Large-Batch Training (McCandlish et al., 2018). Both describe how the required training steps decrease towards a fixed offset as more targets are included per update. The authors should clarify what their formulation adds beyond this existing relation. c) The claimed optimal compute allocation is insufficiently validated. Equation (10) predicts which density should perform best within a specified training-time budget. However, this fixed-budget prediction has not been empirically validated. The comparisons in Table 1 and Figure 4 show useful speed–quality trade-offs, but do not establish whether it selects the best density under the same compute budget.

Questions for authors

Please refer to the weaknesses.

Reviewer_DrzgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors of this paper study how the density of supervision (i.e., the number of targets each user contributes in a given update) affects training efficiency in sequential recommendations. To address this question, they have developed an empirical scaling relationship for the number of updates required for training. They have also developed a fixed-state variance-cost allocation analysis. Additionally, their Exact Prefix-Loss Compilation (EPC) method compiles the computations of all prefixes into one causal pass while maintaining both loss and gradient under prefix consistency and aligned training semantics. Their experimental evaluations analyze training quality and efficiency using density sweep experiments, numerical equivalence tests, and comparative studies using multiple backbone architectures and multiple datasets.

Critical strengths

This work clearly define the circumstances under which shared prefix computation can be used. It also provides a comprehensive analysis of position encoding, padding, candidate sets, and random states using a single framework. As such, the author has established a strong basis for correctly implementing computation reuse by showing that both loss and gradients will remain identical if the required conditions are met. Practical performance improvements result from the sharing of computational resources. The experiments include eight backbone models and six datasets. The experimental comparison included an exact execution alignment.

Critical weaknesses

The method requires backbones to be converted to causal form. However, only one model is checked with its original version. So the accuracy cost of the conversion is not properly evaluated. The core scaling relation is also mostly trivial: the curves can be rescaled by the number of targets to align them, so the faster training arises from shared computation rather than higher supervision density.

Questions for authors
  1. What does it cost in terms of accuracy and training to translate other models into causal form?
  2. What is the accuracy variation in relation to density levels for the same target budget?
Reviewer_ZwQHOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies supervision density in sequential recommendation, how many next-item prediction targets to extract from a single user trajectory per training update. Three contributions are presented: the first one is an empirical scaling law Kq(m)≈K0+Bq/mK_q(m) \approx K_0 + B_q/m relating optimizer updates to density mm on ML-1M; the second one is a gradient variance decomposition that yields a closed-form allocation law m∗=ctraj⋅d/(ctgt⋅a)m^* = \sqrt{c_{\text{traj}} \cdot d / (c_{\text{tgt}} \cdot a)}; the last one is Exact Prefix-Loss Compilation (EPC), a technique that rewrites materialized-prefix training into a shared causal forward pass while provably preserving loss and gradient. Experiments span 8 sequential-recommendation backbones and 6 datasets.

Critical strengths
  • S1: Testing 8 architectures (including 5 that need causal "lifting") is appreciated. The paper doesn't cherry-pick a single architecture.
  • S2: Proposition B.2 gives clean asymptotic savings.
  • S3:  The paper presents a clean and underexplored research question: how to allocate supervision density vs. breadth under a fixed compute budget.
Critical weaknesses
  • W1: Code is not available, impacting the reproducibility of the proposed approach.
  • W2: Statistical tests are not performed, with only standard deviation reported for the experiments.
  • W3: Scaling law is fitted on one dataset/model and tested at sparse granularity. The core law Kq(m)≈K0+Bq/mK_q(m) \approx K_0 + B_q/m is established exclusively on ML-1M with causal FMLP under full CE, fitted to only four density levels (m∈1,2,4,8m \in {1,2,4,8}) with a single held-out point (m=16m=16). No alternative functional forms (logarithmic, power-law) are compared.
  • W4: Under shared-pool InfoNCE, changing mm simultaneously changes target grouping, candidate pools, and execution mode. The paper distinguishes EPC (execution-level) from "practical packing" (policy change) in Section 5, but Table 2 uses practical packing. The reader cannot isolate how much of the reported speedup is from reduced redundant compute vs. a different optimization trajectory.
  • W5: Denser supervision per user means fewer distinct user-gradient updates per epoch. Whether this induces overfitting (especially on small datasets like Beauty or Toys) is never investigated.
  • W6: The predicted optima m∗=40.1m^* = 40.1 (ML-10M) and m∗=49.7m^* = 49.7 (Beauty) fall well outside the tested range (m≤16m \leq 16). The allocation law is the paper's main prescriptive tool, but it is never observed to correctly predict a peak.
Questions for authors

Apart from what I pointed out in the Weaknesses Section, SASRec on Gowalla goes from 724MB (prefix) to 5440MB (packed), which is a 7.5× increase. For long-history datasets this could be prohibitive. The paper reports memory numbers but doesn't discuss mitigation strategies or provide guidance on when memory becomes the binding constraint rather than wall time.

Reviewer_1zrcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to train sequential recommenders more efficiently using several prediction targets from each user history. It reports a scaling relation between supervision density and the updates needed to reach a target quality. It also proposes EPC to share computation across overlapping prefixes while keeping the loss and gradient unchanged under stated conditions. Experiments examine training speed, recommendation quality, and different training choices.

Critical strengths
  1. The paper addresses a useful problem. Processing overlapping histories repeatedly wastes computation, and the experiments show that sharing this work can save considerable training time.
  2. The conditions for keeping the loss and gradient unchanged are clearly stated and tested. The controlled comparisons help separate the benefit of sharing computation from changes to the training setup.
  3. The experiments cover several models and datasets with five paired seeds. They also examine loss functions, candidate pools, learning rates, and user weighting. The authors report both improvements and cases where faster training gives lower recommendation quality.
Critical weaknesses
  1. I wonder whether the speed gains would remain as large in bigger recommendation systems. The method saves time by avoiding repeated work on user histories. But with many more items, scoring those items may take most of the time. The current experiments do not make this clear.
  2. Could using more targets give users with longer histories more weight in training? If so, it is hard to tell how much of the scaling relation comes from supervision density itself. The relation is also tested in relatively few settings, so I am not sure how widely it can be used.
  3. The speed comparisons are harder to judge when the two methods reach different recommendation quality. Reaching a lower best score sooner may not be useful when accuracy matters. The paper does compare time to the same quality, but only for some of the tested settings.
Questions for authors
  1. Would the method still save much time with a much larger item catalog? In practice, which types of recommendation tasks would benefit most?
  2. Does the scaling relation still hold when users keep the same weight across density settings? How stable is it across different models and training choices?
  3. Do the speed gains remain clear when both methods must reach the same recommendation quality? Could this comparison be extended to more of the tested settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.