acceptodds
Under review as a conference paper at ICLR 2027

Aligning Where It Matters: Critical-Branch Preference Optimization for LLM-based Generative Recommendation

Abstract

Generative recommendation increasingly represents items with hierarchical Semantic IDs (SIDs) and formulates next-item prediction as autoregressive token generation. While pretrained large language models (LLMs) have shown strong potential for SID-based recommendation, how they exploit the internal structure of SIDs remains insufficiently understood. In this work, we conduct a fine-grained analysis of hierarchical SIDs in LLM-based generative recommendation. We find that the advantage of pretrained LLMs is highly concentrated at early SID levels, which are both sufficient and necessary for recovering item semantics and are more critical for preserving behavior-relevant information. This observation motivates us to explicitly optimize early SID decisions through preference learning. However, we show that standard Direct Preference Optimization (DPO) is mismatched with autoregressive SID generation: suffix tokens introduce both margin-scale and gradient-direction interference, such that improvement of the critical decision cannot be guaranteed. To address this issue, we propose Critical-Branch Direct Preference Optimization (CB-DPO), which applies preference supervision directly at the first-divergence position of each preference pair, providing a cleaner optimization signal and guaranteeing an increase in the critical margin. Experiments on KuaiRec and KuaiRand-1K demonstrate the effectiveness of CB-DPO across both LLM-based and LLM-free generative recommendation models.

Primary area:
generative models
Keywords:
Generative Recommendation, Sequential Recommendation, Preference Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KTkpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the role of hierarchical Semantic ID (SID) levels in LLM-based generative recommendation. It finds that early SID tokens carry more semantic and behavior-relevant information, then proposes Critical-Branch Direct Preference Optimization (CB-DPO). For each chosen/rejected SID pair, CB-DPO applies preference supervision at the first position where the two sequences diverge, rather than optimizing the preference margin over the entire sequence. It also adds a next-token prediction (NTP) regularizer to mitigate entropy collapse. The method is evaluated with Qwen3 and TIGER on KuaiRec and KuaiRand-1K.

Critical strengths
  • The motivation is clear. In autoregressive SID generation, an early token determines the prefix on which all later predictions are conditioned. Focusing preference supervision on the first divergent decision is therefore well motivated.
  • The paper offers useful diagnostic analyses. It examines SID-level performance and tests semantic and behavioral information by perturbing SID levels and removing suffix tokens from histories.
  • The method is simple and directly targeted. CB-DPO isolates the preference margin at the shared-prefix branching point, while NTP regularization is intended to preserve chosen-sequence likelihood and mitigate entropy collapse.
  • The evaluation spans both LLM-based and LLM-free recommenders. Improvements on TIGER suggest that the proposed preference objective may apply beyond pretrained LLMs.
Critical weaknesses
  • The early-level finding may depend on the SID construction. The analysis uses RQ-VAE hierarchical SIDs, whose residual-quantization process naturally assigns coarse information to early levels and refinements to later ones. The paper does not establish whether the same asymmetry holds with other tokenizers or SID constructions.

  • The theoretical guarantee does not cover the complete CB-DPO objective as written. The first-order increase in the critical margin is derived for the critical-branch preference term. The full objective also includes an NTP regularizer, whose gradient is not included in that derivation. Thus, the theorem alone does not guarantee that each update of the combined objective increases the critical margin.

  • The empirical gains vary substantially by dataset. Improvements are modest on KuaiRec—for Qwen3, Hit@10 decreases slightly—while gains are much larger on KuaiRand-1K. This suggests that performance may depend strongly on the quality of the constructed preference pairs and the availability of explicit feedback.

  • The independent contribution of NTP regularization is not fully established. The paper shows that NTP stabilizes chosen-token scores and reduces entropy collapse, but it provides limited evidence about its separate effect on final recommendation quality.

Questions for authors

Please refer to Weaknesses.

Reviewer_MmZWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the roles of hierarchical Semantic ID tokens in generative recommendation. The authors observe that the advantage of a pretrained LLM is concentrated at early SID levels and propose CB-DPO, which applies preference supervision at the first divergence between chosen and rejected IDs, together with NTP regularization. Experiments on KuaiRec and KuaiRand-1K using TIGER and Qwen3 demonstrate that CB-DPO outperforms standard DPO.

Critical strengths
  • The SID-level analysis identifies useful differences between early and later token predictions.
  • The proposed objective is simple and easy to implement.
Critical weaknesses
  • The experiments omit closely related token- and prefix-aware optimization baselines.
  • Two short-video datasets provide insufficient evidence of generality.
Questions for authors

The phenomenon described in the paper occurs in practice, and the proposed method is interesting. However, I have the following concerns:

  • Method. The offline pairs fix the supervised branch throughout training. For [a,b,c] versus [a,d,e], preference supervision remains on b versus d even after that decision is learned. Although NTP still trains the full positive sequence, later decisions receive no direct preference supervision from this pair. Please analyze whether errors shift to later levels and compare against within-prefix supervision under a matched training budget.

  • Baselines. The baseline comparison is limited to Base, DPO, and CB-DPO. Please compare against APAO [R1], BEAR [R2], D²LR [R3], and IGD [R4] (explaining any adaptations to the SID setting) under consistent data, backbones, and comparable training/tuning budgets, while distinguishing training from decoding changes. A discussion of the differences between these methods and CB-DPO would be helpful. In addition, please include an NTP-only continuation baseline on the same positives to isolate the benefit of preference supervision.

  • Datasets and Backbones. Both datasets focus on short-video recommendation, and Qwen3-1.7B is the only LLM backbone evaluated. Please extend the evaluation to Amazon and MovieLens datasets as well as additional LLM families or scales.

References:

  • [R1] APAO: Bridging the Training-Inference Gap in Generative Recommendation via Adaptive Prefix-Aware Optimization. KDD 2026.
  • [R2] BEAR: Towards Beam-Search-Aware Optimization for Recommendation with Large Language Models. SIGIR 2026.
  • [R3] Dual Debiasing in LLM-based Recommendation. SIGIR 2025.
  • [R4] IGD: Token Decisiveness Modeling via Information Gain in LLMs for Personalized Recommendation. NeurIPS 2025.
Reviewer_5YGWOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the non-uniform roles of hierarchical Semantic ID (SID) tokens in LLM-based generative recommendation. Through level-wise prediction analyses, semantic reconstruction experiments, and SID perturbation studies, the authors observe that the advantages of pretrained LLMs are concentrated primarily at early SID positions, which appear to preserve more semantic and behavior-relevant information than later positions. Motivated by these observations, the authors analyze the mismatch between standard Direct Preference Optimization (DPO) and autoregressive SID generation. They argue that sequence-level DPO aggregates preference margins from both the first-divergence position and subsequent suffix tokens, potentially introducing margin-scale and gradient-direction interference. To address this issue, the paper proposes Critical-Branch Direct Preference Optimization (CB-DPO), which applies preference supervision specifically at the first position where chosen and rejected SIDs diverge. An additional next-token prediction regularizer is introduced to mitigate negative squeezing and entropy collapse. Experiments on KuaiRec and KuaiRand-1K evaluate CB-DPO with both an LLM-based recommender (Qwen3-1.7B) and an LLM-free generative recommender (TIGER). The reported results demonstrate improvements over standard DPO and the corresponding base models, particularly on KuaiRand-1K.

Critical strengths
  • The paper investigates an interesting question about how different positions within hierarchical SIDs contribute to semantic representation and sequential recommendation. Rather than viewing an SID as a single item representation, the authors examine the roles of individual positions through teacher-forced prediction, semantic reconstruction, and controlled perturbation experiments. These analyses offer useful insights into how earlier and later SID positions differ in their contributions, and provide motivation for exploring position-aware optimization in generative recommendation.

  • CB-DPO presents a simple extension of sequence-level DPO that applies preference supervision at the first position where the chosen and rejected SIDs differ. By focusing on this branching point, the method directly optimizes the decision between competing continuations without involving suffix probabilities conditioned on different generation paths. Another advantage is that CB-DPO can be applied to both decoder-only LLMs and encoder-decoder generative recommenders. The method also appears relatively easy to implement, without requiring a separate reward model or a more complicated reinforcement learning pipeline.

Critical weaknesses
  • W1. The theoretical guarantee of CB-DPO does not fully apply to the actual training objective. The paper claims that a gradient-descent update of CB-DPO guarantees an increase in the critical margin mtm_t. Specifically, Eq. (5) gives

    mt(θ′)−mt(θ)=ηβσ(−βmt)∥∇θmt∥2+O(η2).m_t(\theta')-m_t(\theta)=\eta\beta\sigma(-\beta m_t)\|\nabla_\theta m_t\|^2+O(\eta^2).

    While this derivation is valid as a first-order approximation for a single preference pair optimized solely with the critical-branch DPO loss, it overlooks several aspects of the proposed method.

    • First, the analysis omits the next-token prediction (NTP) regularization term. Eq. (4) includes an auxiliary NTP loss on the chosen sequence, weighted by λ\lambda. This is not a negligible component, as Figure 5 shows that it helps prevent negative squeezing and entropy collapse. However, its gradient contribution is absent from Eq. (5), and it is unclear whether the critical margin would still increase when this term is included.
    • Second, the derivation considers a single preference pair, whereas CB-DPO is trained using mini-batches. In practice, the parameter update depends on gradients from multiple preference pairs, which may interfere with one another. Therefore, improving the critical margin for one pair does not necessarily imply that every pair in the mini-batch will experience a similar improvement.
    • Third, even under the simplified setting of a single preference pair without NTP regularization, the result is only a local first-order guarantee. A positive margin increase requires a nonzero gradient and a sufficiently small learning rate for the leading term to dominate the higher-order error. These conditions are not explicitly acknowledged in Theorem 2. Consequently, the theorem appears to establish a more limited result than the guarantee claimed for the full CB-DPO objective.
  • W2. The experiments do not adequately isolate the proposed mechanism, and some evaluation choices raise concerns about the reliability of the results.

    • First, the level-wise analysis does not account for differences in the number of valid continuations across SID positions. The authors find that later positions achieve considerably higher teacher-forced Hit@K and interpret this as evidence that earlier positions contain more semantic and behavior-relevant information. However, under trie-constrained decoding, different SID prefixes may have very different numbers of valid next tokens. The higher accuracy at later positions could therefore partly reflect an easier prediction problem rather than differences in semantic importance. Without controlling for the effective branching factor, this interpretation remains insufficiently supported.
    • Second, the semantic perturbation experiments do not clearly separate positional importance from the effects of invalid SID combinations. The authors randomly replace individual SID tokens while keeping the other positions unchanged. However, since hierarchical SIDs are constructed through residual quantization, tokens at different positions are not necessarily independent. Replacing one token may produce an unnatural or invalid SID sequence, making it difficult to determine whether the resulting performance degradation is caused by the semantic importance of that position or by disrupting the hierarchical token structure.
    • Third, the preference optimization protocol appears to incorporate validation data into training. Appendix C states that preference pairs for both KuaiRec and KuaiRand-1K are constructed from the final three days of the training split together with the validation split. Yet the same appendix describes these validation periods as being held out from the original Seq2Item training data. This raises questions about the separation between training and model selection, particularly whether validation information used for preference optimization is also involved in hyperparameter tuning or performance evaluation. The authors should clarify the data partitioning protocol to rule out potential validation leakage.
Questions for authors
  • Could the authors provide a complete derivation of Theorem 2 for the actual objective in Eq. (4), including both the NTP regularization term and mini-batch gradient interactions?

  • Could the authors report the number of valid next-token continuations at each SID level under trie-constrained decoding?

Comments on the paper

1 comment
Area_Chair_p3yFOpenReview ↗

Author- Reviewer Discussion

Dear Reviewers, the authors have submitted their response. Kindly review it and update your review where appropriate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.