- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
I understand this paper as two points 1 any order AR model provides conditional probability with respect to tokens at arbitrary locations, while fixed design may not have, e.g., NAR, AR, PAR 2 the dependency between tokens gradually decreases as they are far away. This allows using any-order AR as a powerful drafter and verifier, letting the parallel-generated token far away be less entangled. This formulation is good.
1 As the author claimed in Limitation, this method can only apply in any-order AR, making the scope something small. For example, currently it still lacks t2i any-order AR model, making the application limited
2 The observation of "the dependency between tokens gradually decreases as they are far away" is not novel; please cite LPD and NAR at the proper location.
[1] Locality-aware Parallel Decoding for Efficient Autoregressive Image Generation
[2] Neighboring autoregressive modeling for efficient visual generation
3 The reported cost metric (target forward passes per image) flatters the method, since drafting and verification are fused into one heavier pass while wall-clock is given only at batch size 1 and γ is tuned favorably;
Critical strengths
1 The observation well motivated the method.
- The method is simple and effective. The result is promising and well-verified.
Critical weaknesses
1 As the author claimed in Limitation, this method can only apply in any-order AR, making the scope something small. For example, currently it still lacks t2i any-order AR model, making the application limited
2 The observation of "the dependency between tokens gradually decreases as they are far away" is not novel; please cite LPD and NAR at the proper location.
[1] Locality-aware Parallel Decoding for Efficient Autoregressive Image Generation
[2] Neighboring autoregressive modeling for efficient visual generation
3 The reported cost metric (target forward passes per image) flatters the method, since drafting and verification are fused into one heavier pass while wall-clock is given only at batch size 1 and γ is tuned favorably; a total-FLOPs or latency-vs-batch-size curve would be fairer. The lossy comparison is also framed advantageously: invoking KL to claim exactness is circular (guaranteed by construction), and switching to FDr⁶ inflates the gap to 5.2–10.1×, whereas under FID the practical gap is closer to 3.5×.
I am willing to raise my score if the authors address these concerns
Questions for authors
Please refer to the above
4 comments
W3 - Cost metric and the lossy comparison
We agree that the pass count alone does not give the full picture in terms of cost. We would first like to clarify that, by default DSD adapts SCD, where a pass takes the γ drafts (current window) as input and returns the model's distribution at each of these positions. These distributions are used to verify the current drafts and to draw the drafts of the next step. So, a pass evaluates at most γ positions and we do not perform a separate pass for drafting. The pass therefore has the same computational load as the standard SCD and SJD baselines, which all have the lightest forward passes (lowest computational load) among the speculative samplers we report. For comparison, fused drafting evaluates γ(γ+1) positions per pass (since verification and drafting are fused by predrafting for all potential verification outcomes) and the trained drafters add drafter passes. The target pass count therefore does not favor DSD against any of the speculative decoding baselines. Against sequential decoding, a DSD pass costs 1.2x a sequential pass at batch size 1. We will make the cost clearer in Section 5.
As suggested, we also measured the throughput vs. batch size with the same settings (one H100) on the largest checkpoint of each family (where a pass costs the most). The optimal draft window depends on the batch size (smaller window for larger batch size), so, for each batch size, we report the throughput (images per second) for each speculative sampler at its fastest window (among 8, 16, 32 and 64):
| model | sampler | B = 1 | 4 | 16 | 32 | 64 |
|---|---|---|---|---|---|---|
| RandAR-0.7B | DSD | 1.47 | 4.41 | 8.65 | 9.71 | 10.3 |
| SCD, raster | 0.43 | 1.35 | 3.57 | 4.91 | 5.28 | |
| sequential | 0.16 | 0.64 | 2.57 | 5.10 | 9.50 | |
| ARPG-XXL | DSD | 1.19 | 4.61 | 15.3 | 24.8 | 31.6 |
| SCD, raster | 0.38 | 1.43 | 4.65 | 8.38 | 13.4 | |
| sequential | 0.15 | 0.61 | 2.45 | 4.92 | 9.24 | |
| LPD-XXL | DSD | 1.90 | 6.17 | 11.8 | 14.1 | 15.2 |
| SCD, raster | 0.71 | 2.27 | 4.69 | 6.37 | 6.56 | |
| sequential | 0.22 | 0.88 | 3.51 | 7.06 | 14.0 |
DSD is ~2x to 3.4x faster than standard SCD along raster at every batch size. It also stays ahead of sequential decoding at every batch size (up to 64). Speculative decoding trades computation (FLOPs) for latency, so naturally, this advantage is largest at small batch sizes and shrinks as the batch grows (as we become compute-bound). DSD also needs less FLOPs since, against the other speculative samplers at matched draft window size, it needs fewer passes. We will add the throughput curves for every window to the paper.
Regarding the KL test, we agree that exactness holds by construction. We only use it to measure how far the lossy samplers are from the model's distribution at the step counts where FID no longer distinguishes between them. We will make this explicit in the paper.
Regarding FID and FDr^6, they both are Fréchet distances and differ only in the representations used to compute the distance. FID uses the Inception representation only, while FDr^6 averages over six representations (Inception, ConvNeXt, MAE, DINOv2, SigLIP and CLIP), including ones from more recent models. We understand that FID is still widely used and considered the standard, which is why the paper reports both. We will give both speedup ranges in the abstract to make this clearer.
Full throughput table:
To complement our response to W3, here is the throughput for every draft window and batch size (same setup as in the previous table) where each cell reports DSD / SCD:
| model | window | B = 1 | 4 | 16 | 32 | 64 |
|---|---|---|---|---|---|---|
| RandAR-0.7B | 8 | 0.62 / 0.34 | 2.08 / 1.12 | 5.55 / 3.05 | 7.64 / 4.48 | 8.69 / 5.28 |
| 16 | 0.98 / 0.41 | 3.21 / 1.35 | 7.56 / 3.57 | 9.71 / 4.91 | 10.3 / 5.07 | |
| 32 | 1.20 / 0.41 | 4.22 / 1.11 | 8.65 / 3.44 | 9.50 / 3.75 | 9.35 / 3.93 | |
| 64 | 1.47 / 0.43 | 4.41 / 1.31 | 7.34 / 2.46 | 7.47 / 2.55 | 7.41 / 2.56 | |
| sequential | 0.16 | 0.64 | 2.57 | 5.10 | 9.50 | |
| ARPG-XXL | 8 | 0.58 / 0.31 | 2.21 / 1.11 | 8.94 / 4.47 | 16.7 / 8.31 | 27.6 / 13.4 |
| 16 | 0.92 / 0.35 | 3.46 / 1.17 | 13.0 / 4.65 | 24.8 / 8.38 | 31.6 / 10.3 | |
| 32 | 1.16 / 0.38 | 4.12 / 1.43 | 15.3 / 4.60 | 23.4 / 6.70 | 24.8 / 7.49 | |
| 64 | 1.19 / 0.36 | 4.61 / 1.40 | 13.2 / 4.07 | 14.5 / 4.36 | 14.0 / 3.96 | |
| sequential | 0.15 | 0.61 | 2.45 | 4.92 | 9.24 | |
| LPD-XXL | 8 | 0.89 / 0.51 | 3.02 / 1.68 | 8.20 / 4.36 | 10.7 / 5.85 | 12.1 / 5.27 |
| 16 | 1.34 / 0.66 | 4.37 / 1.99 | 11.8 / 4.69 | 14.1 / 6.37 | 15.2 / 6.56 | |
| 32 | 1.88 / 0.70 | 5.45 / 2.26 | 11.7 / 3.84 | 12.6 / 4.79 | 14.5 / 5.24 | |
| 64 | 1.90 / 0.71 | 6.17 / 2.27 | 9.12 / 2.81 | 10.6 / 3.23 | 10.4 / 3.41 | |
| sequential | 0.22 | 0.88 | 3.51 | 7.06 | 14.0 |
Thank you for the response. I have no more concerns and raise my score.
We thank the reviewer for the very useful feedback and for noting that the method is well motivated, simple and effective, with well-verified results. We address each concern below, with new results on a text-to-image model and a detailed throughput analysis, and we hope that these resolve them.
W1 - Text-to-image model
Any-order autoregressive image models are relatively recent, but that line of work is growing fast (RandAR at CVPR 2025, ARPG and LPD at ICLR 2026) and is gaining attention for zero-shot properties that raster models lack (e.g., parallel decoding, inpainting, outpainting). To complement our results, we also ran DSD on an any-order text-to-image model, RLlamaGen [1]. We report the mean target pass count per image on COCO captions:
DSD needs 3.4x fewer passes than standard SCD along raster, which aligns with the class-conditional models. We agree that DSD is limited to any-order models, as stated in our limitations. We will add the text-to-image results to the paper.
[1] NEP: Autoregressive Image Editing via Next Editing Token Prediction
W2 - Citing LPD and NAR at relevant locations
We agree and thank the reviewer for the references. The paper discusses LPD in the related work (Section 6) but does not cite it (or make a connection with it) where the “locality vs. dependence” observation is introduced. NAR, on the other hand, is missing. We will cite both in Section 3. LPD and NAR use locality to design lossy parallel decoders. In our work it is one part of a broader analysis of speculative decoding, where we separate the disagreement between drafter and target into a part due to the model and another part due to the conditioning. Drafting with the target model itself removes the first, and the remaining disagreement is caused by the dependence between tokens drafted together, which can be reduced with a dispersed order. We show that an any-order model can achieve this without any additional training.