- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the inference overhead of autoregressive decoding in SID-based generative recommendation. The authors observe that the number of valid SID continuations decreases rapidly after the first few tokens, suggesting that a full Transformer decoder may be unnecessary. Based on this observation, they propose SID-MLP, which replaces the Transformer decoder with position-specific MLP heads. The model computes user context once and conditions each head on the previously generated SID tokens. The MLP heads are trained through knowledge distillation from a frozen teacher model. Experiments on three Amazon datasets show an 8.74× throughput improvement over TIGER-kv with comparable recommendation accuracy. The authors also introduce SID-MLP++, which further replaces the Transformer encoder.
Critical strengths
- Achieves substantial inference speedup (8.74×) and memory reduction (95.7 %) while preserving recommendation quality.
- Provides clear motivation through analyses of SID branching factors and decoder redundancy.
- Includes comprehensive ablation studies that validate the key architectural choices.
- Evaluates different tokenizers, teacher architectures, and SID configurations.
Critical weaknesses
- Several acceleration baselines require nontrivial adaptations to TIGER, making their reported performance potentially sensitive to implementation choices.
- SID-MLP recovers only 83.3%–97.5% of the teacher's NDCG@10 on LC-Rec 7B, indicating non-negligible performance degradation.
- The position-specific MLP heads limit direct model reuse when SID lengths or tokenization schemes change.
Questions for authors
- Can SID-MLP adapt to longer or modified SID structures without retraining its position-specific MLP heads?
- What causes the accuracy degradation of SID-MLP on LC-Rec 7B?
- How sensitive are the baseline results to the architectural adaptations made for TIGER?
2 comments
W2/Q2. The LC-Rec quality gap. As we discussed in Section 4.3, the gap is mainly caused by 159× teacher–student compression. We agree that some gap remains on 7B LC-Rec, and it should be considered together with model size and inference cost. SID-MLP distills the 7B autoregressive teacher into only 44.1M trainable parameters (about 0.6% of the teacher) and removes online 7B decoder execution. Despite this compression, it recovers 83.18%–97.44% of teacher NDCG@10 with substantial speedup.
AtSpeed-R and NEZHA also show quality gaps on LC-Rec, illustrating the speed–quality trade-off when accelerating large teachers. The key advantage of SID-MLP is that it preserves most of the teacher's performance using only 0.6% of its trainable parameters, while providing substantial verify-free acceleration. This supports our main observation that full 7B Transformer decoding is unnecessary at every SID position. We further added the one-shot diagnostic to Appendix D.2 and the position-wise results to Table 19.
W3/Q1. Low-cost adaptation to a changed teacher. We clarify that changing the SID length or tokenizer first changes the teacher itself: the new structure defines different output positions, label spaces, or token semantics, so the generative teacher must also be trained or adapted accordingly.
SID-MLP then follows this updated teacher through the same lightweight redistillation pipeline, without any architectural redesign. However, this retraining is lightweight, and the performance remains robust across different structures. Appendix D.3 (Tables 20–23) evaluates these adaptations:
| Change | Tested structures | NDCG@10 recovery |
|---|---|---|
| SID length | 2×256 / 3×256 / 6×64 | 102.4% / 102.8% / 99.8% |
| Codebook width | 128 / 256 / 512 | 101.6% / 102.8% / 102.3% |
| Heterogeneous width | [64,128,256] / [256,128,64] | 100.3% / 101.4% |
| Tokenizer | RQ-VAE / RQ-KMeans / PSID-OPQ | 100.0% / 102.8% / 101.5% |
| Multimodal SID | Unified / MACRec dual-branch | 100.1% / 100.2% |
All tested structures recover at least 99.8% of their matched teacher's NDCG@10. Thus, SID-MLP can adapt to teachers with new structures through the same distillation framework.
The cost of redistillation is low. Appendix C.2 reports that training the base SID-MLP student takes ~1 hour per dataset on average. Appendix E.2 (Table 33) further shows that SID-MLP requires only 4.48 PFLOP of additional offline compute, lower than the 6.78–66.17 PFLOP required by the other baselines. It also saves about 4.42 GFLOP per online request, so the offline compute is recovered after ~1M compute-equivalent requests. These results show that the additional cost of training the lightweight student is small, making redistillation practical when the teacher is updated.
Therefore, the plug-and-play advantage of SID-MLP is that it can adapt to an existing teacher with low offline training cost while preserving teacher-level recommendation quality. We have clarified this low-cost adaptation point at the end of Appendix D.3.
We hope the native 7B comparison, LC-Rec diagnosis, and SID-structure results above address your concerns about baseline adaptation, large-teacher fidelity, and adaptation across SID structures. We would be grateful if you would reconsider your assessment. If a specific issue remains, please point us to it and we will address it during the discussion.
We thank the reviewer for the careful assessment.
In the responses below, we show that SID-MLP maintains a strong quality–efficiency advantage across teacher architectures and SID structures, requiring only lightweight redistillation when the structure changes.
W1/Q3. Sensitivity of baselines. We agree that architectural adaptations might affect baseline results, so we additionally tested AtSpeed-S/R on native decoder-only 7B LC-Rec. We used its original 68M LLaMA draft and retained strict verification and relaxed rejection sampling. We profiled all methods on the same A6000 with the batch size 32 protocol used throughout our paper.
AtSpeed performs better on its original 7B backbone than in the TIGER adaptation as expected. However, SID-MLP achieves higher throughput and much lower offline training cost. Section 4.3 (Table 5) also reports NEZHA's LC-Rec results, with 64.96%–82.67% teacher recovery, below SID-MLP's 83.18%–97.44%; Appendices F.1–F.3 document the original TIGER-side baseline adaptations.
Together, these matched comparisons demonstrate that SID-MLP can be readily adapted to larger LLM-based generative recommenders while retaining its quality–efficiency advantage. We have added the native AtSpeed comparison to Section 4.3 (Table 5) and its N=40 operating point to Appendix D.2.