- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the granularity mismatch problem in generative recommendation: alignment rewards are based at the item level, but policy optimization is based on shared semantic IDs (SIDs). To address this mismatch, the authors propose a Deep Extended Alignment (DEA) method. During training, DEA enhances the original node policies by using conditional item policies used only for training, thus optimizing the recommendation results. This method uses frozen item embeddings and a shared decoder, referencing only one additional training matrix W, making it deployable in the industry. Its hierarchical joint policy objective decomposes the reward-based advantage into cross-node and intra-node components, applying PPO-style pruning to the node-level and joint importance ratios, respectively. During inference, only the Item-level refinement is removed, while the tokenizer, node-level decoding process, and bucket expansion interface remain unchanged. The author conduced experiment on on industrial search and the publicly available KuaiRec benchmark set. The results demonstrate that DEA outperforms SFT, DPO, and node-level GRPO variants, no matter whether the RQ-Kmeans or RQ-VAE tokenizer is used. In the paper, DEA’s effectiveness through offline experiments on industrial and public datasets and online A/B testing was also presented.
Critical strengths
The paper presents a novel and easy to deploy design for addressing the granularity mismatch in generative recommendation. The author proposes a creative and also deployable approach in the form of the DEA (Decoder-based Embedding Alignment) method. In DEA, the author employs frozen item embeddings and the same decoder backbone, and introduces an additional trainable projection matrix (W) for the item-level alignment. The paper presents a clear argument, and supports it with rigorous mathematical derivations. The structure of the article is clear, and follows logically from the problem statement to the method that is introduced to solve the problem, and then finally to the experimental evaluation of the method. The experimental evaluation is very comprehensive, as it is conducted on two very different benchmarks (an industrial search dataset, and the public KuaiRec benchmark), using two different tokenizers, and with multiple different baselines. The author also provides a thorough set of ablation studies, provides information on the training dynamics, efficiency, and performs online A/B testing in order to provide strong evidence of the effectiveness and practicability of the proposed DEA method.
Critical weaknesses
Ablation studies for the components of DEA are not clear. Although several ablation studies were conducted in this paper, changes to several components were made at the same time in some of the ablation studies. For example, the first DEA ablation study removed the item-level optimization and used a different rule to compute the node values in addition to removing the DEA component. The auxiliary baseline also differed from DEA in several aspects. Therefore, it is unclear whether the improvements are due to the hierarchical joint-policy optimization of DEA or the improved item-level training. More controlled ablation studies are needed to study the contributions of the components of DEA.
Questions for authors
There are some ablation experiments that change multiple components simultaneously. Therefore, it is difficult to identify the contributions of individual key components such as hierarchical joint-policy optimization in the proposed method in Fig. 3. The Training–Serving Objective Mismatch in Eq. (6) and Eq. (18) needs to be addressed. The node value in Eq. (6) is computed by weighting the item rewards using the learned item policy q_{\bar\theta} of the corresponding nodes. However, in serving, this item policy is discarded and instead the relevance of all the item candidates in the expanded buckets is evaluated using Eq. (18). For simplicity of illustration, assume that there are two nodes, each containing four items. Then, Node A: [1.0, 0, 0, 0] and Node B: [0.6, 0.6, 0.6, 0.6]. Using the learned item policy q_{\bar\theta} in Eq. (6), the node value of the first node is computed as 0.97 (i.e., 0.97 1.0 + 0.03 0 + 0.03 0 + 0.03 0) while that of the second node is computed as 0.6. However, after bucket expansion, assuming a relevance threshold of 0.5 for simplicity, Node A has only 25% relevant items while all 4 items of Node B are relevant (i.e., 100%). Thus, higher q-weighted node value in Eq. (6) does not necessarily translate to higher relevance in Eq. (18) after bucket expansion. It would be helpful to clarify how the node value in Eq. (6) aligns with the retrieval objective in Equation (18).