acceptodds
Under review as a conference paper at ICLR 2027

Accelerating Decoding via Autoregressive-to-Parallel Distillation for Architecture-Agnostic Generative Recommendation

Abstract

Generative Recommendation (GR) retrieves items by autoregressively generating semantic identifiers, making beam-search inference costly. Existing speculative-decoding methods for GR either retrain the target with a draft head or train an external draft model while keeping the target frozen. The former incur substantial target retraining costs, whereas the latter still suffer from costly autoregressive draft inference and are limited by architecture-specific verification pipelines. To address these limitations, we propose the Autoregressive-to-Parallel Distillation framework for Generative Recommendation (APD-GR). This framework distills the frozen target's autoregressive prediction knowledge into a lightweight draft that computes SID position representations in parallel. Specifically, we design a two-stage distillation strategy that first establishes parallel prediction through SID supervision and then aligns the draft's prefix-conditioned distributions with those of the frozen target. To further support heterogeneous GR backbones, we propose a unified interface across decoder-only and encoder–decoder targets. In the experiments, to verify effectiveness and versatility of our proposed accelerating framework, we conduct extensive experiments across three datasets and five GR backbones. The results show that our method achieves an average end-to-end inference speedup of while largely preserving recommendation performance. To ease reproducibility, we release the code online https://anonymous.4open.science/r/anonymous-code-release-DC63.

Primary area:
generative models
Keywords:
Generative Recommendation, Inference Acceleration, Parallel Multi-Token Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fwQTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes APD-GR, an inference-acceleration framework for generative recommendation with semantic IDs. Rather than performing autoregressive draft generation and online Target verification, APD-GR extracts representations from a frozen Target once, uses a Context Adapter and Parallel Position Draft to compute representations for all SID positions in parallel, and employs a lightweight recurrent Prefix-Conditioned Head during constrained beam search. The framework is trained with a two-stage objective: supervised SID prediction followed by teacher-forced distribution distillation from the frozen Target.

Critical strengths
  1. The paper identifies a practically relevant bottleneck in generative recommendation: repeated autoregressive decoding over semantic IDs.

  2. The combination of a one-pass parallel position Draft with a recurrent prefix-conditioned Head is a sensible design for reducing repeated backbone computation while retaining prefix information.

Critical weaknesses
  1. The absence of a lightweight autoregressive baseline without online Target verification makes it difficult to isolate the benefit of the parallel Draft from that of removing verification.
  2. Performance degradation is substantial in some settings. For example, MiniOneRec loses approximately 25% of N@10 on Toys, weakening the claim of largely preserving recommendation quality.
Questions for authors

Stage I uses only 50% of the training data, whereas Stage II uses the full dataset. Removing Stage II therefore also reduces data coverage and training steps. Can the authors isolate the benefit of the two-stage design using controls with matched data coverage and training budgets?

1 comment

Thank you for your constructive feedback and for recognizing the practical value of our work. For W1, we added AtSpeed-D, the draft-only AtSpeed baseline, to Table 1 and discussed the results in Section 4.2. For W2, we revised the abstract and conclusion to report average NDCG@20 retention and updated Section 4.2 to discuss the loss on Toys. For Q1, we added two single-stage baselines with matched data coverage and training-example counts in Appendix C.2 (Table 10).

W1. Verification-Free Baseline: We compare APD-GR and AtSpeed w/o verification using the LETTER backbone. With verification disabled, AtSpeed runs beam search under the same SID constraints and returns the top candidates ranked by the draft model’s scores.

MethodInstruments H@10Instruments N@10Instruments Spd.Beauty H@10Beauty N@10Beauty Spd.Toys H@10Toys N@10Toys Spd.
APD-GR0.10870.082111.58×0.07190.041211.35×0.04510.027311.77×
AtSpeed w/o verify0.07630.04738.74×0.03370.01808.56×0.02620.01458.94×

Both methods omit online Target verification, yet APD-GR achieves higher H@10 and N@10 with lower latency across all three datasets. This shows that APD-GR’s advantage over this autoregressive baseline extends beyond removing verification.

W2. Quality Claims: We agree that the quality loss on Toys is substantial for some backbones and that “largely preserving recommendation performance” is too broad. We have revised the abstract and conclusion to report average NDCG@20 retention and updated Section 4.2 to discuss how the quality loss varies across backbone–dataset pairs.

Q1. Two-Stage Design: We compare APD-GR with two single-stage baselines using TIGER. Both baselines train on the full dataset for 14 epochs, using either CE alone or CE+KL throughout. This matches APD-GR’s total training-example count: 4 epochs on 50% of the data followed by 12 epochs on the full dataset (4 × 0.5 + 12 = 14).

MethodInstruments H@10Instruments N@10Beauty H@10Beauty N@10Toys H@10Toys N@10
APD-GR0.10370.07770.06130.03290.06250.0346
Single-stage CE0.09960.07500.05830.03150.06180.0346
Single-stage CE+KL0.10460.07840.05970.03270.06130.0345

Although the two-stage design primarily aims to reduce feature caching and transfer costs, it also maintains competitive recommendation quality under a matched training budget. APD-GR performs better on Beauty and Toys, while single-stage CE+KL performs better on Instruments.

Reviewer_WssDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes APD-GR to accelerate frozen generative recommenders. It computes SID position representations in parallel and combines them with a lightweight recurrent head for sequential decoding, without online target verification. Through supervised adaptation and distribution distillation, the method achieves an average speedup of 9.22× across three datasets and five backbones.

Critical strengths

1 The method addresses a practical inference bottleneck without modifying or retraining the original recommendation model.

2 The framework supports both decoder-only and encoder–decoder architectures, with experiments covering five backbones.

3 The latency reductions are substantial. Some settings, particularly LETTER on Beauty, achieve considerable acceleration with little recommendation quality loss.

Critical weaknesses
  1. The recurrent head already receives user information through the root representation (r) and incorporates previously generated tokens. However, Table 2 does not test a model trained without the parallel position representations (H). Removing this branch and retraining the remaining recurrent head would clarify how much the proposed parallel module contributes beyond a simpler autoregressive decoder.

  2. On Toys, NDCG@10 drops by approximately 25% for MiniOneRec and 18% for both CoFiRec and DIGER. These are substantial losses, and the general claim of largely preserving recommendation performance seems too strong. The paper should explain these differences across backbones and more clearly identify the settings where the accuracy cost is justified by the latency reduction.

  3. SID-MLP (Guo et al., 2026), MLPs are Efficient Distilled Generative Recommenders, appears particularly relevant but is neither discussed nor compared. It also freezes the teacher, computes user context once, and uses lightweight prefix-conditioned heads without online verification. A comparison on one representative backbone, or a clear explanation of the differences that prevent a meaningful comparison, would help establish the benefit of APD-GR over this simpler alternative.

Questions for authors
  1. How do recommendation quality and latency change when the parallel (H) branch is removed and the remaining RNN head is retrained with the same supervision and distillation objectives?

  2. What explains the larger performance losses on Toys, particularly for MiniOneRec, CoFiRec, and DIGER?

  3. Could the authors compare APD-GR with SID-MLP under a shared backbone and evaluation setting, or clarify why such a comparison would not be appropriate?

1 comment

Thank you for your constructive feedback. For W1 and Q1, we added an ablation of the parallel position representations in Appendix C.2 (Table 9). For W2 and Q2, we revised the abstract and conclusion to quantify average quality retention and updated Section 4.2 to discuss the larger losses on Toys. For W3 and Q3, we added the SID-MLP comparison on the LETTER backbone to Section 4.2 (Table 1) and discussed SID-MLP in Section 5.

W1 & Q1. Draft Ablation: We remove H and retrain the recurrent head on Beauty using MiniOneRec and TIGER, retaining user-conditioned initialization and token-based state updates but predicting tokens from WisiW_i s_i. Latency is the average per-sample inference time (ms).

BackboneMethodH@10N@10Latency
MiniOneRecAPD-GR0.04340.023018.99
MiniOneRecw/o H0.03860.020417.09
MiniOneRecRelative Change-11.06%-11.30%1.11×
TIGERAPD-GR0.06130.03297.90
TIGERw/o H0.05670.03036.42
TIGERRelative Change-7.50%-7.90%1.23×

Removing the parallel draft reduces latency but lowers both H@10 and N@10. These results support the contribution of the parallel position representations to recommendation quality.

W2 & Q2. Quality–Latency Trade-off: We agree that these losses are substantial and that “largely preserving recommendation performance” is too broad. We believe the differences may reflect both data limitations and backbone-dependent knowledge transfer.

Dataset. Toys provides fewer training examples than Beauty despite having a similar catalog size. This limited supervision may make it harder for the Draft to generalize from ground-truth prefixes used in distillation to alternative prefixes encountered during beam search.

Backbones. The Target can use attention to access user-history information conditioned on each generated prefix. APD-GR instead computes parallel position representations once and handles subsequent prefix changes through a lightweight recurrent head. Predictions that require fine-grained, prefix-dependent use of user history may therefore be harder to transfer. Differences in SID construction and backbone training may affect this transfer, potentially contributing to the larger losses observed for MiniOneRec, CoFiRec, and DIGER on Toys.

Practical applicability. On Toys, LETTER and TIGER offer more favorable trade-offs, with approximately 5–7% NDCG@10 loss and 8–12× speedup. In contrast, the 18–25% losses for MiniOneRec, CoFiRec, and DIGER make these configurations less suitable when accuracy is a priority; latency reduction alone does not establish that the loss is acceptable. We have revised the abstract and conclusion to report average NDCG@20 retention and updated Section 4.2 to discuss how the quality loss varies across backbone–dataset pairs.

W3 & Q3. SID-MLP Comparison: We compare SID-MLP and APD-GR using the same LETTER backbone and evaluation protocol. We use SID-MLP’s public implementation, adapting only its data interface for LETTER.

MethodInstruments H@10Instruments N@10Instruments LatencyBeauty H@10Beauty N@10Beauty LatencyToys H@10Toys N@10Toys Latency
SID-MLP0.08810.069316.720.04420.023216.930.03930.022317.11
APD-GR0.10870.082119.370.07190.041220.260.04510.027319.13

APD-GR achieves higher H@10 and N@10 than SID-MLP on all three datasets, with the largest gains on Beauty. These gains come with an increase of approximately 2–3 ms in inference latency per sample.

Reviewer_T7saOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes APD-GR, a method for accelerating decoding in generative recommendation models. APD-GR first computes representations for all SID positions in parallel, while using the Prefix-Conditioned Head to handle decoding. The paper also introduces a two-stage distillation strategy to transfer knowledge to the Draft. The paper reports an average end-to-end inference speedup of 9.22× across three datasets and five generative recommendation models, but different configurations exhibit varying degrees of recommendation quality degradation.

Critical strengths
  1. Through parallel position computation and a lightweight Prefix-Conditioned Head, the method reduces inference latency without retraining the original Target model.
  2. The method achieves notable speedups across multiple models and datasets. The paper also provides ablation studies and cost analyses, which help validate the effectiveness of its design.
Critical weaknesses
  1. In some settings, recommendation quality degrades noticeably, which may affect the practical applicability of the method. For example, on the Toys dataset, N@10 decreases from 0.0303 to 0.0228, a relative decrease of approximately 24.8%.
  2. The existing evaluation settings are limited, and it remains unclear whether the speedups can generalize to larger-scale recommendation services.
Questions for authors
  1. What causes the degradation in recommendation quality on some datasets? Will it affect the practical use of the method?
  2. For non-ground-truth SID prefixes encountered during Beam Search, to what extent can the Draft accurately approximate the Target's predictions? Would incorporating candidate prefixes generated by Beam Search into Stage II improve recommendation quality?
  3. Could the authors provide more comprehensive experiments comparing the trade-off between recommendation quality and inference latency for APD-GR and existing acceleration methods such as AtSpeed?
  4. Could the authors provide additional experiments with larger batch sizes, larger item catalogs, and longer SID sequences?
3 comments
AuthorsOpenReview ↗

Response to Reviewer T7sa (1/2)

Thank you for your thoughtful questions. For W1 and Q1, we revised the abstract and conclusion to quantify average quality retention and updated Section 4.2 to discuss the loss on Toys. For Q2, we added the prefix agreement and beam-prefix distillation experiments to Appendix C.7. For Q3, we added the AtSpeed comparison across beam widths to Figure 3 and Appendix C.4. For W2 and Q4, we added results for larger batches and longer item SIDs to Appendix C.6.

W1 & Q1. Recommendation Quality: We agree that the degradation on MiniOneRec–Toys is substantial. We believe two factors may contribute.

Dataset. Toys contains fewer training examples than Beauty despite having a similar catalog size, providing less supervision per item. Since distillation uses ground-truth SID prefixes, while inference also encounters alternative prefixes during beam search, limited supervision may make it harder for the Draft to generalize across different user histories and candidate prefixes.

Backbone. The Target can use attention to access user-history information conditioned on each generated prefix. APD-GR instead computes parallel position representations once and handles subsequent prefix changes through a lightweight recurrent head. Predictions that require fine-grained, prefix-dependent use of user history may therefore be harder to transfer. Differences in SID construction and backbone training may affect this transfer, potentially contributing to the larger loss on MiniOneRec–Toys.

Without online Target verification, prediction differences can alter beam-search candidate selection and ranking. The resulting quality loss therefore matters in practice: configurations such as MiniOneRec–Toys may be unsuitable for applications with strict accuracy requirements. We have revised the abstract and conclusion to report average NDCG@20 retention and updated Section 4.2 to discuss how the quality loss varies across backbone–dataset pairs.

Q2. Beam Prefixes: We evaluate LETTER and MiniOneRec on Toys, comparing the next-token distributions of Draft and Target on non-ground-truth prefixes retained by APD-GR’s beam search.

BackbonePrefix LengthKLTop-1 AgreementTop-10 Overlap
LETTER12.071013.64%29.43%
LETTER21.153565.76%34.09%
LETTER30.855087.44%38.70%
MiniOneRec12.281916.10%36.03%
MiniOneRec21.124561.36%34.27%

KL is DKL(PT∥PD)D_{\mathrm{KL}}(P_T\parallel P_D); Top-1 measures the argmax agreement between Draft and Target; Top-10 overlap is the fraction of shared tokens in their top-10 sets. MiniOneRec uses three-token SIDs, so the longest prefix for next-token prediction has length 2.

We further test beam-prefix distillation using MiniOneRec on Toys. For each training history, the frozen Stage-I Draft runs beam search (beam size 50), retaining the top 10 non-ground-truth prefixes at each prefix length. In Stage II, we add a KL loss matching the Target’s next-token distribution on these prefixes to the original training loss, keeping other training settings unchanged. All additional prefixes come from the training split.

MethodH@10H@20N@10N@20
APD-GR0.04390.06470.02280.0281
+ Beam-prefix distillation0.04260.06290.02200.0271

Adding beam-prefix distillation slightly reduces all four metrics, indicating that this augmentation does not improve recommendation quality in the tested setting.

AuthorsOpenReview ↗

Response to Reviewer T7sa (2/2)

Q3. AtSpeed Comparison: We compare the recommendation quality and inference latency of APD-GR and AtSpeed using LETTER on Beauty with beam sizes from 10 to 100. In the table, Latency is the average per-sample inference time (ms).

Beam SizeAPD-GR Hit@10AtSpeed Hit@10LETTER LatencyAPD-GR LatencyAtSpeed Latency
100.07020.070886.8419.2489.50
200.07220.0729118.2519.1292.81
500.07190.0727229.9020.26107.75
1000.07150.0731430.6819.54140.96

Across all tested beam sizes, APD-GR achieves substantially lower latency than AtSpeed with slightly lower Hit@10. As beam size increases, APD-GR’s latency remains nearly constant, while AtSpeed’s latency rises, widening the gap between the two methods.

W2 & Q4. Scaling: We evaluate APD-GR with TIGER on Instruments across multiple batch sizes. Speedup is measured relative to the original TIGER model at the same batch size. APD-GR maintains its latency advantage as batch size increases, achieving greater speedups at larger batch sizes.

Batch Size141664
Spd.8.00×23.64×82.97×321.94×

For the larger-catalog evaluation, SID quantization is still in progress on Electronics, which contains 159,729 items. We will report the results in a follow-up response in two days.

We evaluate APD-GR on Toys using TIGER as the backbone with item SID lengths of 6 and 8 tokens. APD-GR achieves a larger speedup at 8 tokens than at 6, with lower H@5 and N@5 than the backbone at both lengths.

SID LengthMethodH@5N@5LatencySpd.
6TIGER0.03660.024195.921.00×
6+APD-GR0.03120.01999.1610.47×
8TIGER0.03790.0253125.641.00×
8+APD-GR0.03470.02239.8512.76×
AuthorsOpenReview ↗

Follow-up on W2 & Q4: Larger Item Catalog

Following up on our earlier response to W2 and Q4, we have completed the Electronics experiment and added the results to Appendix C.6 (Table 15) of the revised manuscript.

We evaluate APD-GR using TIGER as the backbone on Amazon 2014 Electronics, with a catalog of 159,729 items. Due to time constraints, evaluation is conducted on a subset of 30,000 test samples.

MethodH@10H@20N@10N@20
TIGER0.06760.08000.05720.0603
+APD-GR0.05960.06950.04980.0523
Retention88%87%87%87%

APD-GR retains approximately 87–88% of TIGER's recommendation performance across all four metrics on this larger item catalog.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.