- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates how next-item prediction pretraining produces item embeddings that transfer to CTR prediction. It decomposes embedding updates into positive-target, negative-target, and history-input contributions, identifying frequency imbalance and excessive dependence on the latest interaction as potential limitations. It proposes frequency-adjusted positive-target gradients (FA) and shared-query cross-attention (Cross), evaluated through frozen-embedding transfer across eleven datasets.
Critical strengths
-
The investigation targets an important representation-learning question. Instead of treating generative pretraining as a black box, the paper examines which supervision paths shape transferable item embeddings. The distinction between positive-target, negative-target, and historical-input updates provides a useful framework for understanding transfer.
-
The proposed changes are simple and connected to the diagnosis. FA modifies only the positive-target embedding gradient, while Cross removes the latest item’s privileged role in initializing the predictive state. Appendix D.4 also compares FA with whole-sample weighting and post-hoc adjustment, helping establish that the specific intervention matters.
-
The empirical evidence goes beyond overall performance tables. The evaluation includes eleven datasets, five random seeds, frequency-conditioned loss analysis, hidden-state sensitivity measurements, and comparisons across encoder depths. The consistent gains of the combined method support its usefulness within the studied transfer setting.
Critical weaknesses
-
The evaluation only partially supports the claimed relevance to CTR prediction. Most downstream negatives are uniformly sampled catalog items rather than logged non-clicked impressions. This construction can substantially influence both AUC and the apparent importance of item frequency. The strong popularity baseline on several datasets further motivates validation under realistic exposure-conditioned labels.
-
The gradient-path analysis establishes endpoint sensitivity more clearly than causal importance during training. Scaling retained updates while holding the downstream reader fixed can introduce distribution shifts and perturb components by different magnitudes. Appendix C correctly acknowledges that removing a path during training would change the trajectory, but the main interpretation remains stronger than the intervention directly establishes.
-
The evidence for Cross’s proposed mechanism is incomplete. Lower sensitivity to the latest item is expected after removing its direct query/residual contribution, but this does not necessarily mean that the model uses relevant history more effectively. Controlled variation in next-item dependence, or separate query and residual interventions, would better distinguish reduced sensitivity from improved contextual modeling.
Questions for authors
-
Do the improvements persist with logged impression/non-click labels and with end-to-end fine-tuning of the pretrained embeddings?
-
Does positive-target dominance survive equal-magnitude perturbations or training-time interventions? Can query-only and residual-only controls clarify why Cross helps?