acceptodds
Under review as a conference paper at ICLR 2027

Improve Discriminative Recommendation through Better Generative Pretraining

Abstract

Generative next-item prediction (NTP) pretraining improves click-through rate (CTR) prediction even when only the item embeddings are transferred. This demonstrates that pretraining yields effective item representations. We decompose embedding gradients and find that positive-target supervision is mostly important. Specifically, we discover that item frequency and attention structure make great effects on gradients. Low item frequency causes the inefficient supervision on embedding while self-attention structure makes the gradient direction sensitive to the most recent interaction. We therefore propose two modifications of NTP pretraining, Frequency Adjustment (FA) and shared-query Cross attention (Cross). First, FA strengthens the output gradients for infrequent targets to mitigate the supervision imbalance. Second, Cross uses a learned shared query to aggregate user history, reducing the model’s reliance on recent interactions. Both methods keep the next-item objective and the transfer pipeline. Experiments on public datasets show that joint method improves mean downstream AUC over standard NTP.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Recommendation Pretraining, Item Embedding Transfer, Click-Through Rate Prediction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eVrDOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how next-item prediction pretraining produces item embeddings that transfer to CTR prediction. It decomposes embedding updates into positive-target, negative-target, and history-input contributions, identifying frequency imbalance and excessive dependence on the latest interaction as potential limitations. It proposes frequency-adjusted positive-target gradients (FA) and shared-query cross-attention (Cross), evaluated through frozen-embedding transfer across eleven datasets.

Critical strengths
  1. The investigation targets an important representation-learning question. Instead of treating generative pretraining as a black box, the paper examines which supervision paths shape transferable item embeddings. The distinction between positive-target, negative-target, and historical-input updates provides a useful framework for understanding transfer.

  2. The proposed changes are simple and connected to the diagnosis. FA modifies only the positive-target embedding gradient, while Cross removes the latest item’s privileged role in initializing the predictive state. Appendix D.4 also compares FA with whole-sample weighting and post-hoc adjustment, helping establish that the specific intervention matters.

  3. The empirical evidence goes beyond overall performance tables. The evaluation includes eleven datasets, five random seeds, frequency-conditioned loss analysis, hidden-state sensitivity measurements, and comparisons across encoder depths. The consistent gains of the combined method support its usefulness within the studied transfer setting.

Critical weaknesses
  1. The evaluation only partially supports the claimed relevance to CTR prediction. Most downstream negatives are uniformly sampled catalog items rather than logged non-clicked impressions. This construction can substantially influence both AUC and the apparent importance of item frequency. The strong popularity baseline on several datasets further motivates validation under realistic exposure-conditioned labels.

  2. The gradient-path analysis establishes endpoint sensitivity more clearly than causal importance during training. Scaling retained updates while holding the downstream reader fixed can introduce distribution shifts and perturb components by different magnitudes. Appendix C correctly acknowledges that removing a path during training would change the trajectory, but the main interpretation remains stronger than the intervention directly establishes.

  3. The evidence for Cross’s proposed mechanism is incomplete. Lower sensitivity to the latest item is expected after removing its direct query/residual contribution, but this does not necessarily mean that the model uses relevant history more effectively. Controlled variation in next-item dependence, or separate query and residual interventions, would better distinguish reduced sensitivity from improved contextual modeling.

Questions for authors
  1. Do the improvements persist with logged impression/non-click labels and with end-to-end fine-tuning of the pretrained embeddings?

  2. Does positive-target dominance survive equal-magnitude perturbations or training-time interventions? Can query-only and residual-only controls clarify why Cross helps?

Reviewer_KVjhOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to improve item embeddings learned through next-item prediction for downstream CTR prediction. It proposes Frequency Adjustment (FA), which strengthens positive-target embedding gradients for infrequent items, and shared-query Cross attention, which replaces the latest-item query and residual initialization with a shared learned vector. Only the pretrained item embeddings are transferred and frozen. The combined method improves downstream AUC over standard pretraining on all 11 datasets.

Critical strengths
  • The focus on transferring only frozen item embeddings is clear and useful. The improvements survive after the pretraining encoder is discarded.
  • The evaluation covers 11 datasets, five seeds, and deeper pretraining encoders, providing fairly broad support for the reported gains.
  • The gradient decomposition offers a useful perspective on embedding training. The whole-sample weighting comparison also helps distinguish FA from ordinary loss reweighting.
Critical weaknesses
  • FA's methodological contribution seems limited relative to existing gradient-adjustment methods. Selective gradient gating for rare embeddings and positive-negative gradient balancing are already established ideas [1, 2]. FA makes a specific modification to these ideas, but its advantage over relevant alternatives is not established.
  • Cross also looks like a relatively small adaptation of existing designs. Shared-query attention has a clear precedent [3], and the connection between residual structure and last-item reliance has already been studied [4]. The paper demonstrates a useful application, but the new architectural contribution remains unclear.
  • The shared-query choice introduces a tradeoff that deserves more discussion. Removing the latest item's privileged role also removes content-dependent query initialization. This may limit how the model selects relevant history, especially in the main single-layer setting.
Questions for authors
  1. My main concern with FA is what is actually new beyond applying selective gradient adjustment to this transfer setting. I understand that FA boosts positive-target embedding gradients, while AGG suppresses selected non-target gradients [1]. But boosting positive gradients to address imbalance also has precedent [2]. The specific path being adjusted is different, yet that alone does not make a strong methodological contribution. Table 6 shows an advantage over the tested whole-sample weighting scheme, but an adapted AGG comparison would be much more relevant. What about this setting makes FA's particular intervention preferable?

  2. Cross replaces the latest-item initialization with a shared learned query and applies this construction in the first layer. The paper acknowledges the connection to an existing query-stream design [3], while residual-driven last-item reliance is also a known issue [4]. I can see why this modification might help, but I'm less convinced by its novelty. What does Cross add beyond adapting a familiar attention design to embedding pretraining? A simple residual-scaling or gating alternative would help establish whether the shared-query construction offers a meaningful advantage. Combining it with FA shows practical value, but does not by itself resolve the limited novelty of the two components.

  3. A smaller concern is why avoiding latest-item dominance calls for a query shared across all users. In the single-layer model, attention scores cannot directly adapt an item's relevance to the rest of the user's history, apart from normalization across the available items. The output is still personalized through the history, so this is not a lack of personalization altogether. Still, why is a shared query the right tradeoff here, rather than a query derived from the broader history? The motivation explains why the latest item may be a poor query, but less clearly why the query should be independent of history content.

[1] Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddings [2] Equalization Loss v2: A New Gradient Balance Approach for Long-Tailed Object Detection [3] XLNet: Generalized Autoregressive Pretraining for Language Understanding [4] Residual Dominance as a Structural Account of Last-Item Reliance in Causal Self-Attention Recommenders

Reviewer_x1HjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to improve next-item prediction pretraining to obtain item embeddings better suited to downstream discriminative recommendation. The authors analyze three sources of updates: positive targets, negative targets, and history inputs. They propose two modifications: Frequency Adjustment increases only the positive-target output-embedding gradients for infrequent items, while Cross uses a shared learnable query in the first layer of the pretraining encoder to construct the predictive state. After pretraining, only the item embeddings are transferred and frozen; all other downstream parameters are reinitialized and trained. The experiments cover eleven public datasets and five random seeds, and also compare deeper pretraining encoders. Downstream evaluation primarily uses constructed negative candidates.

Critical strengths

1.The method addresses a clear and practical transfer problem, with precisely specified interventions. Transferring and freezing only the item table while retraining all other downstream parameters allows the performance changes to be interpreted through the pretrained embeddings. Neither modification requires a change to the downstream architecture.

2.The experimental coverage is broad. FA and the combined method both improve mean AUC over standard NTP on every dataset, and the combined method also yields positive AUC gains averaged across datasets at the deeper pretraining depths evaluated. Update-path diagnostics, frequency-group analyses, and comparisons with alternative frequency-based methods provide support beyond the main performance table

Critical weaknesses

The data boundaries of the evaluation remain insufficiently specified. Section 6.1 and Appendices A and E.2 describe a shared 8:1:1 split, pretraining on training histories, and exclusion of the current downstream target field and label. However, they do not adequately specify the split unit, temporal rules, or whether history windows are constructed before or after splitting. In particular, if source history windows overlap, the relationship between interactions across partitions needs to be explicit.

Questions for authors

Please clarify whether each dataset is split by user, interaction, or history example; what temporal rules are applied; whether history windows are constructed before or after splitting; and how overlapping windows across partitions are handled. How is each downstream example's history truncated relative to its target time?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.