- Rating
- 1
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies knowledge distillation for visual autoregressive (AR) image generation models. The authors first compare supervised KD, SeqKD, and GKD on visual AR models and find that on-policy distillation, which works well in language, yields limited gains on images: decoding sequences are long and visual tokens are ambiguous, so student-generated contexts drift away from the data distribution and teacher predictions on such contexts become unreliable. Based on this, they propose VARKD: the student continues from a partial ground-truth prefix, and the teacher scores the continuation; tokens are reweighted by the teacher's predictive entropy and low-confidence tokens are truncated; the VQ codebook is clustered via k-means and the distillation loss is computed in the compressed codebook space; parallel decoding is used to speed up rollouts. Inference is unchanged. Experiments cover Janus-Pro 7B→1B (GenEval, DPG-Bench) as well as LlamaGen and ARPG (ImageNet 256). VARKD consistently outperforms KD, SeqKD, and GKD in FID.
Critical strengths
- The problem is well-motivated and fills a gap. Efficiency work on visual AR models has focused almost entirely on decoding acceleration.
- Broad experimental coverage: T2I and C2I, three backbones, multiple teacher sizes, with consistent FID improvements.
- The two-stage recipe is practical: KD for fast alignment followed by VARKD for refinement mitigates the throughput disadvantage of on-policy distillation.
Critical weaknesses
- Limited novelty. The mixed ground-truth/student prefix is already part of the GKD baseline (Sec. 4.1; Tab. 3, row 2), truncation follows speculative decoding, and parallel decoding is ZipAR. The gain over GKD comes mainly from entropy reweighting (4.94→4.66); codebook relaxation and parallel decoding add only 0.06 and 0.02 FID (Tab. 3).
- The gains over GKD are small and inconsistent: 2.92→2.84 on LlamaGen-L and 2.20→2.16 on ARPG-L (Tab. 2), GKD is better for XXL→L in Tab. 7 (2.88 vs 2.91), and VARKD has lower IS than GKD and SeqKD on both L students.
- On DPG-Bench, KD, SeqKD, and GKD all score below the undistilled Janus-Pro-1B (81.89/81.51/82.00 vs 82.02, Tab. 1a), which contradicts the claim in Sec. 4.2 that standard KD recovers a substantial fraction of the gap on both benchmarks.
- Results are inconsistent across tables: SeqKD on LlamaGen-B is 5.36 in Tab. 2 but 5.08 in Tab. 5/7, and the LlamaGen-L/ARPG-L rows in Tab. 2 match neither the XL nor the XXL teacher in Tab. 7, while Tab. 2 does not state which teacher is used.
- The baselines copied in Tab. 7 contain errors (LlamaGen-L IS 156.0 vs 256.06; ARPG-L IS/Prec/Rec 309.0/0.80/0.58 vs 297.7/0.82/0.56 in Li et al.), the LlamaGen-B baseline (6.51) matches neither official checkpoint (5.46/6.09), and resolution, CFG, and sampling settings are not reported.
- No teacher numbers are reported, so "substantially narrowing the gap" cannot be checked. Using the official teacher FIDs, VARKD closes about 22% of the gap for XXL→L (3.07→2.91 vs 2.34) and about 32% for ARPG XXL→L (2.30→2.15 vs 1.83).
- Distilled Decoding [1,2] already distills visual AR models (into one-step generators) but is neither cited nor discussed.
- The mandatory ICLR 2027 AI-usage disclosure section is missing.
[1] Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models with Flow Matching [2] Distilled Decoding 2: One-step Sampling of Image Auto-regressive Models with Conditional Score Distillation
Questions for authors
See Weakness.
We thank the reviewer for their feedback and careful reading of our tables.
Novelty
As explained in the general comment, the paper's contribution is the analysis of why on-policy distillation underperforms for visual AR models, together with a recipe that addresses each cause. That several ingredients are adapted from the language literature is by design, and we credit them: speculative decoding inspires truncation, and ZipAR provides parallel decoding. The mixed prefix is our own modification, and we deliberately apply it to the GKD baseline so that the comparison isolates the remaining components. The original fully on-policy GKD degrades FID from 5.20 to 6.02 (Tab. 6), so our gains over GKD are conservative. Codebook relaxation, which has no counterpart in text, improves FID by 0.14 on its own (per-component ablation, reply to nukT).
Consistency of Gains
VARKD achieves the best FID in 6 of the 7 teacher–student pairs in Tab. 7, in every setting of Tab. 2 (p < 0.05 over five seeds), and the best overall score on both T2I benchmarks. The single exception, which the reviewer points out, is XXL → L (2.91 vs. 2.88). Moreover, FID is the primary metric for ImageNet generation, while IS rewards class confidence and can trade off against diversity. The IS differences between VARKD and GKD on the L students in Tab. 2 are within the seed-to-seed spread.
Consistency Across Tables and Evaluation Numbers
We re-evaluated all models ourselves. We used the publicly available code and checkpoints of LlamaGen, ARPG and Janus-Pro, and recomputed every metric (FID, IS, precision/recall, GenEval and DPG-Bench). All models, distilled and original, were therefore evaluated at the same resolution and CFG scale, in the same environment (hardware, software stack, evaluation code). The resulting numbers can differ from those in the original publications, for example our LlamaGen-B baseline (6.51). We believe that evaluating every model, teachers included, under identical conditions is the most reliable way to compare distillation methods.
The differences between Tab. 2 and Tab. 7 come from two different settings. Tab. 2 reports the mean over five seeds, while the other tables report a single seed.
Lastly, the reviewer is right about the undistilled LlamaGen-L and ARPG-L rows in Tab. 7. These are transcription errors on our side: the rows should report our own single-seed evaluations (LlamaGen-L: 3.12 FID, 256.9 IS; ARPG-L: 2.33 FID, 294.8 IS), consistent with the five-seed means in Tab. 2 (3.16 / 257.2 and 2.36 / 296.8). The distilled results in Tab. 7 are unaffected, so the errors do not change any comparison between methods.
We thank the reviewer for the careful check and agree that these settings should be stated explicitly in every caption.
Distilled Decoding
We thank the reviewer for the pointers. Distilled Decoding 1 and 2 reduce the number of decoding steps by distilling an AR model into a one or few-step generator, which gives up autoregressive sampling. Our work reduces model size while keeping standard next-token decoding. The two directions are complementary, and Distilled Decoding could, for example, be applied to a VARKD-distilled student.
AI Use Statement
We acknowledge that the statement was missing. For completeness: we used LLMs to polish the writing of the paper and as coding assistants during implementation. The research idea, the first draft, the method, the experimental design and ablations, the figures, and the analysis of the results are entirely the authors' own work. All LLM-assisted text and code were reviewed and verified by the authors, who take full responsibility for the content of the paper.