acceptodds
Under review as a conference paper at ICLR 2027

Train Fine, Serve Coarse: Tokenizer-Agnostic Item-Level Alignment for Generative Recommendation

Abstract

Generative recommendation models commonly undergo post-training to align their outputs with application-specific objectives such as relevance or user preference. The rewards used for alignment are typically defined on individual items, whereas policy optimization operates on the semantic IDs that encode these items. This creates a mismatch between the granularity of reward evaluation and policy optimization when multiple items share a semantic ID, or node. Items within the same node can receive different rewards for the same input, yet their rewards affect a single shared node likelihood, limiting the policy's ability to learn from these distinctions. We propose Depth-Extended Alignment (DEA), which introduces a training-time refinement of the semantic-ID hierarchy to enable policy optimization at the granularity of individual items. We derive a hierarchical alignment objective that accounts for reward differences across nodes and among items within each node, while shared representations enable item-specific feedback to update the generative model. At inference, DEA preserves the original node-level decoding process without additional item-level computation. Offline experiments on public and real-world industrial datasets demonstrate substantial performance improvements with both RQ-Kmeans and RQ-VAE, two widely used tokenization methods. Validated through rigorous online A/B testing, DEA now operates in full production on a large-scale e-commerce platform serving hundreds of millions of active users.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Generative Recommendation, Item-Level Alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Kg8bOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the granularity mismatch problem in generative recommendation: alignment rewards are based at the item level, but policy optimization is based on shared semantic IDs (SIDs). To address this mismatch, the authors propose a Deep Extended Alignment (DEA) method. During training, DEA enhances the original node policies by using conditional item policies used only for training, thus optimizing the recommendation results. This method uses frozen item embeddings and a shared decoder, referencing only one additional training matrix W, making it deployable in the industry. Its hierarchical joint policy objective decomposes the reward-based advantage into cross-node and intra-node components, applying PPO-style pruning to the node-level and joint importance ratios, respectively. During inference, only the Item-level refinement is removed, while the tokenizer, node-level decoding process, and bucket expansion interface remain unchanged. The author conduced experiment on on industrial search and the publicly available KuaiRec benchmark set. The results demonstrate that DEA outperforms SFT, DPO, and node-level GRPO variants, no matter whether the RQ-Kmeans or RQ-VAE tokenizer is used. In the paper, DEA’s effectiveness through offline experiments on industrial and public datasets and online A/B testing was also presented.

Critical strengths

The paper presents a novel and easy to deploy design for addressing the granularity mismatch in generative recommendation. The author proposes a creative and also deployable approach in the form of the DEA (Decoder-based Embedding Alignment) method. In DEA, the author employs frozen item embeddings and the same decoder backbone, and introduces an additional trainable projection matrix (W) for the item-level alignment. The paper presents a clear argument, and supports it with rigorous mathematical derivations. The structure of the article is clear, and follows logically from the problem statement to the method that is introduced to solve the problem, and then finally to the experimental evaluation of the method. The experimental evaluation is very comprehensive, as it is conducted on two very different benchmarks (an industrial search dataset, and the public KuaiRec benchmark), using two different tokenizers, and with multiple different baselines. The author also provides a thorough set of ablation studies, provides information on the training dynamics, efficiency, and performs online A/B testing in order to provide strong evidence of the effectiveness and practicability of the proposed DEA method.

Critical weaknesses

Ablation studies for the components of DEA are not clear. Although several ablation studies were conducted in this paper, changes to several components were made at the same time in some of the ablation studies. For example, the first DEA ablation study removed the item-level optimization and used a different rule to compute the node values in addition to removing the DEA component. The auxiliary baseline also differed from DEA in several aspects. Therefore, it is unclear whether the improvements are due to the hierarchical joint-policy optimization of DEA or the improved item-level training. More controlled ablation studies are needed to study the contributions of the components of DEA.

Questions for authors

There are some ablation experiments that change multiple components simultaneously. Therefore, it is difficult to identify the contributions of individual key components such as hierarchical joint-policy optimization in the proposed method in Fig. 3. The Training–Serving Objective Mismatch in Eq. (6) and Eq. (18) needs to be addressed. The node value in Eq. (6) is computed by weighting the item rewards using the learned item policy q_{\bar\theta} of the corresponding nodes. However, in serving, this item policy is discarded and instead the relevance of all the item candidates in the expanded buckets is evaluated using Eq. (18). For simplicity of illustration, assume that there are two nodes, each containing four items. Then, Node A: [1.0, 0, 0, 0] and Node B: [0.6, 0.6, 0.6, 0.6]. Using the learned item policy q_{\bar\theta} in Eq. (6), the node value of the first node is computed as 0.97 (i.e., 0.97 1.0 + 0.03 0 + 0.03 0 + 0.03 0) while that of the second node is computed as 0.6. However, after bucket expansion, assuming a relevance threshold of 0.5 for simplicity, Node A has only 25% relevant items while all 4 items of Node B are relevant (i.e., 100%). Thus, higher q-weighted node value in Eq. (6) does not necessarily translate to higher relevance in Eq. (18) after bucket expansion. It would be helpful to clarify how the node value in Eq. (6) aligns with the retrieval objective in Equation (18).

Reviewer_wTzUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses a granularity mismatch in the post-training alignment of generative recommenders: multiple items are frequently encoded into the same semantic-ID "node", whereas alignment rewards are defined on individual items, so within-node reward differences can only influence updates through a single shared node likelihood. The authors propose DEA, which — only during training — introduces an item-level policy inside each node's bucket that reuses the backbone hidden state, and decomposes the return into cross-node and within-node advantages plugged into a GRPO-style objective, so that item-level feedback updates the shared backbone. At inference the item branch is discarded and node-level decoding and serving remain unchanged, incurring no additional inference cost. DEA improves metrics on two offline datasets and yields business gains in a 5%-traffic online A/B test, and has been fully deployed in production.

Critical strengths
  1. DEA is a clean, well-motivated decomposition design for the granularity-mismatch problem: all additional modeling happens only at training time, yet it mitigates the confusion among items that share a node, and the offline ablations — stratified by the degree of within-node label disagreement — show that the benefit grows exactly where a shared node reward obscures item-level differences.
Critical weaknesses
  1. The contribution is incremental in both its theory and its empirical gains. The theoretical analysis is developed for the joint training objective, whereas inference and evaluation use only the node policy; the paper itself notes that the decomposition "does not guarantee improved node-level retrieval under the serving protocol" (L245), yet provides neither a theoretical guarantee nor a mechanistic analysis of why optimizing the joint objective should improve the served node-level ranking (the node-level retrieval results in Table 1 are positive but remain purely empirical). Moreover, Table 2 shows that the fixed-reference auxiliary already recovers most of the gain, and DEA's improvement over it is limited (about +0.83 relevance with RQ-Kmeans).

  2. The baseline comparison is incomplete. It lacks direct comparisons against unique-SID approaches such as TIGER, against simply reducing SID collisions (e.g., a finer tokenizer / smaller buckets), and against the original UniVA and CQ-SID methods. As a result, it is not possible to tell whether the gains come from reducing collision-induced confusion or from DEA's alignment scheme being genuinely superior.

Questions for authors
  1. Given that DEA's improvement over the fixed-reference auxiliary is small, could you provide a finer ablation that isolates the contributions of the hierarchical advantage decomposition and of the separate-clipping design?

  2. Please add comparisons against unique-SID approaches (e.g., TIGER), against reducing the SID collision rate, and against the original UniVA / CQ-SID, so as to verify whether DEA's gain stems from the method design rather than from compensating for a high collision rate. Please also report training cost and inference FLOPs/latency within these comparisons.

  3. Please discuss scalability more thoroughly — across model sizes, bucket sizes, and SID collision rates — to clarify under which settings DEA is the better fit.

  4. Retrieving the top-k nodes rather than individual items makes the number of returned candidates variable; how is this handled in the industrial setting?

  5. No code is currently released. Open-sourcing the code would make the results considerably easier to reproduce and would help validate the approach.

Reviewer_CY4AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Depth-Extended Alignment (DEA) to address the mismatch between item-level rewards and shared semantic-ID optimization in generative recommendation. DEA introduces a training-only item policy and a hierarchical alignment objective while retaining node-level inference. Offline and online experiments show improved retrieval quality without additional inference cost.

Critical strengths
  1. The paper addresses a practical granularity mismatch between item-level rewards and shared-SID optimization in generative retrieval.
  2. DEA incorporates item-specific feedback through a shared backbone while preserving the original inference architecture and serving cost.
  3. Experiments across two tokenizers, industrial search, KuaiRec, and online A/B testing demonstrate consistent gains, supported by useful ablations.
Critical weaknesses
  1. The joint training objective weights items using a learned within-bucket policy, whereas serving expands entire buckets. The theoretical analysis does not establish when improving this objective improves deployed retrieval quality.
  2. The novelty beyond training-only item-level auxiliary learning is insufficiently isolated. The auxiliary comparison changes several optimization components simultaneously, making the source of improvement difficult to identify.
  3. Robustness and reproducibility remain limited: only one backbone size is tested, KuaiRec retrieval details are incomplete, and industrial relevance evaluation uses the same service that provides training rewards.
Questions for authors
  1. Can the authors measure whether gains in the joint training reward correlate with improvements in whole-bucket relevance, particularly for buckets containing few high-reward items?
  2. Can the authors isolate adaptive node valuation, item importance weighting, and clipping through controlled ablations against a matched auxiliary objective?
  3. How sensitive is DEA to bucket size, SID collision rate, item temperature, and the quality of frozen item embeddings?
  4. What are the KuaiRec tokenizer configurations, bucket statistics, retrieval budget, and exact watch-ratio aggregation protocol?
  5. Can the authors provide independent human relevance evaluation and confidence intervals for the online A/B results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.