acceptodds
Withdrawn from ICLR 2027

Knowledge Distillation for Visual Autoregressive Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Autoregressive (AR) image generation models are highly expressive but computationally intensive, motivating effective model compression. Knowledge distillation (KD) is a natural approach for model compression and has been widely studied in language modeling, yet its behavior in visual AR generation remains underexplored. In this work, we present the first systematic study of distillation strategies for AR image models. Our analysis shows that while standard distillation can yield meaningful gains, recent methods developed for language do not directly transfer to images: long decoding horizons and visual token ambiguity make teacher supervision unreliable especially under student‑conditioned contexts. To address this, we propose VarKD, a distillation framework for visual autoregressive models that distills on student samples while selectively applying teacher supervision and reducing token-level ambiguity. Experiments on text-to-image generation benchmarks and class-to-image generation across multiple autoregressive backbones show that VarKD consistently outperforms prior distillation baselines, substantially narrowing the gap to large-scale models.

Primary area:
generative models
Keywords:
autoregressive, distillation, image generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_r6cyOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies knowledge distillation for visual autoregressive (AR) image generation models. The authors first compare supervised KD, SeqKD, and GKD on visual AR models and find that on-policy distillation, which works well in language, yields limited gains on images: decoding sequences are long and visual tokens are ambiguous, so student-generated contexts drift away from the data distribution and teacher predictions on such contexts become unreliable. Based on this, they propose VARKD: the student continues from a partial ground-truth prefix, and the teacher scores the continuation; tokens are reweighted by the teacher's predictive entropy and low-confidence tokens are truncated; the VQ codebook is clustered via k-means and the distillation loss is computed in the compressed codebook space; parallel decoding is used to speed up rollouts. Inference is unchanged. Experiments cover Janus-Pro 7B→1B (GenEval, DPG-Bench) as well as LlamaGen and ARPG (ImageNet 256). VARKD consistently outperforms KD, SeqKD, and GKD in FID.

Critical strengths
  1. The problem is well-motivated and fills a gap. Efficiency work on visual AR models has focused almost entirely on decoding acceleration.
  2. Broad experimental coverage: T2I and C2I, three backbones, multiple teacher sizes, with consistent FID improvements.
  3. The two-stage recipe is practical: KD for fast alignment followed by VARKD for refinement mitigates the throughput disadvantage of on-policy distillation.
Critical weaknesses
  1. Limited novelty. The mixed ground-truth/student prefix is already part of the GKD baseline (Sec. 4.1; Tab. 3, row 2), truncation follows speculative decoding, and parallel decoding is ZipAR. The gain over GKD comes mainly from entropy reweighting (4.94→4.66); codebook relaxation and parallel decoding add only 0.06 and 0.02 FID (Tab. 3).
  2. The gains over GKD are small and inconsistent: 2.92→2.84 on LlamaGen-L and 2.20→2.16 on ARPG-L (Tab. 2), GKD is better for XXL→L in Tab. 7 (2.88 vs 2.91), and VARKD has lower IS than GKD and SeqKD on both L students.
  3. On DPG-Bench, KD, SeqKD, and GKD all score below the undistilled Janus-Pro-1B (81.89/81.51/82.00 vs 82.02, Tab. 1a), which contradicts the claim in Sec. 4.2 that standard KD recovers a substantial fraction of the gap on both benchmarks.
  4. Results are inconsistent across tables: SeqKD on LlamaGen-B is 5.36 in Tab. 2 but 5.08 in Tab. 5/7, and the LlamaGen-L/ARPG-L rows in Tab. 2 match neither the XL nor the XXL teacher in Tab. 7, while Tab. 2 does not state which teacher is used.
  5. The baselines copied in Tab. 7 contain errors (LlamaGen-L IS 156.0 vs 256.06; ARPG-L IS/Prec/Rec 309.0/0.80/0.58 vs 297.7/0.82/0.56 in Li et al.), the LlamaGen-B baseline (6.51) matches neither official checkpoint (5.46/6.09), and resolution, CFG, and sampling settings are not reported.
  6. No teacher numbers are reported, so "substantially narrowing the gap" cannot be checked. Using the official teacher FIDs, VARKD closes about 22% of the gap for XXL→L (3.07→2.91 vs 2.34) and about 32% for ARPG XXL→L (2.30→2.15 vs 1.83).
  7. Distilled Decoding [1,2] already distills visual AR models (into one-step generators) but is neither cited nor discussed.
  8. The mandatory ICLR 2027 AI-usage disclosure section is missing.

[1] Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models with Flow Matching [2] Distilled Decoding 2: One-step Sampling of Image Auto-regressive Models with Conditional Score Distillation

Questions for authors

See Weakness.

1 comment

We thank the reviewer for their feedback and careful reading of our tables.

Novelty

As explained in the general comment, the paper's contribution is the analysis of why on-policy distillation underperforms for visual AR models, together with a recipe that addresses each cause. That several ingredients are adapted from the language literature is by design, and we credit them: speculative decoding inspires truncation, and ZipAR provides parallel decoding. The mixed prefix is our own modification, and we deliberately apply it to the GKD baseline so that the comparison isolates the remaining components. The original fully on-policy GKD degrades FID from 5.20 to 6.02 (Tab. 6), so our gains over GKD are conservative. Codebook relaxation, which has no counterpart in text, improves FID by 0.14 on its own (per-component ablation, reply to nukT).

Consistency of Gains

VARKD achieves the best FID in 6 of the 7 teacher–student pairs in Tab. 7, in every setting of Tab. 2 (p < 0.05 over five seeds), and the best overall score on both T2I benchmarks. The single exception, which the reviewer points out, is XXL → L (2.91 vs. 2.88). Moreover, FID is the primary metric for ImageNet generation, while IS rewards class confidence and can trade off against diversity. The IS differences between VARKD and GKD on the L students in Tab. 2 are within the seed-to-seed spread.

Consistency Across Tables and Evaluation Numbers

We re-evaluated all models ourselves. We used the publicly available code and checkpoints of LlamaGen, ARPG and Janus-Pro, and recomputed every metric (FID, IS, precision/recall, GenEval and DPG-Bench). All models, distilled and original, were therefore evaluated at the same resolution and CFG scale, in the same environment (hardware, software stack, evaluation code). The resulting numbers can differ from those in the original publications, for example our LlamaGen-B baseline (6.51). We believe that evaluating every model, teachers included, under identical conditions is the most reliable way to compare distillation methods.

The differences between Tab. 2 and Tab. 7 come from two different settings. Tab. 2 reports the mean over five seeds, while the other tables report a single seed.

Lastly, the reviewer is right about the undistilled LlamaGen-L and ARPG-L rows in Tab. 7. These are transcription errors on our side: the rows should report our own single-seed evaluations (LlamaGen-L: 3.12 FID, 256.9 IS; ARPG-L: 2.33 FID, 294.8 IS), consistent with the five-seed means in Tab. 2 (3.16 / 257.2 and 2.36 / 296.8). The distilled results in Tab. 7 are unaffected, so the errors do not change any comparison between methods.

We thank the reviewer for the careful check and agree that these settings should be stated explicitly in every caption.

Distilled Decoding

We thank the reviewer for the pointers. Distilled Decoding 1 and 2 reduce the number of decoding steps by distilling an AR model into a one or few-step generator, which gives up autoregressive sampling. Our work reduces model size while keeping standard next-token decoding. The two directions are complementary, and Distilled Decoding could, for example, be applied to a VARKD-distilled student.

AI Use Statement

We acknowledge that the statement was missing. For completeness: we used LLMs to polish the writing of the paper and as coding assistants during implementation. The research idea, the first draft, the method, the experimental design and ablations, the figures, and the analysis of the results are entirely the authors' own work. All LLM-assisted text and code were reviewed and verified by the authors, who take full responsibility for the content of the paper.

Reviewer_nukTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies knowledge distillation for visual autoregressive models and proposes VAR KD based on GKD, with entropy-based reweighting, low-confidence truncation, and codebook relaxation. Experiments on LlamaGen, ARPG, and Janus-Pro show improvements over existing distillation baselines. However, the gains are relatively small, especially for larger student models, and the proposed components bring only incremental improvements. Some aspects of the method also require further validation.

Critical strengths
  1. The paper studies a relevant problem and evaluates distillation methods across multiple visual autoregressive models.

  2. VAR KD improves FID over existing distillation baselines in the reported ImageNet experiments.

Critical weaknesses
  1. Limited improvements: VAR KD is mainly based on GKD with additional heuristic components. The improvements over existing baselines are relatively small, and the gains become weaker for larger student models. And it remains teacher–student gap.

  2. Insufficient validation of the proposed components: The reliability of entropy-based reweighting and truncation is not sufficiently established. Moreover, codebook relaxation may discard useful differences between visual tokens, and the effect of parallel decoding on the rollout distribution is unclear.

Questions for authors
  1. How much improvement comes from entropy-based reweighting and codebook relaxation individually? Does entropy reliably identify incorrect or unreliable teacher supervision?

  2. How are visual tokens grouped in codebook relaxation, and how sensitive are the results to the grouping strategy? Does parallel decoding preserve the original autoregressive rollout distribution, or does it introduce additional approximation errors?

  3. Although distillation improves the student, a substantial performance gap remains. For example, LlamaGen-B distilled from LlamaGen-L achieves an FID of 4.58 (Table 3), compared with 3.16 reported for LlamaGen-L (Table 2). The authors should discuss why this gap remains and whether it mainly reflects limited student capacity or limitations of the distillation method.

1 comment

We thank the reviewer for their constructive feedback.

Contribution of Each Component

Tab. 3 adds the components one after another. We have also added each component on its own on top of GKD (LlamaGen L to B):

MethodFID ↓
GKD4.94
+ Entropy reweighting only4.66
+ Codebook relaxation only4.80
+ Parallel decoding only4.96
VARKD (all)4.58

Entropy reweighting has the largest effect, and codebook relaxation helps both alone and combined with it. Parallel decoding is not meant to improve quality. It speeds up rollout generation by 1.3× (Tab. 5), and the table confirms it does so at no cost in quality.

Reliability of Entropy

Entropy is a proxy, as we state in Sec. 3.2, and this is exactly why we pair it with truncation. Three observations show that it is a useful one. First, under ground-truth prefixes teacher entropy decreases along the sequence, while under student rollouts it stays high, so it tracks drift. Second, the fraction of retained tokens rises during training (63% → 77%) as the student aligns with the teacher. Third, the threshold sweep in Tab. 6 has a clear optimum: keeping nearly all tokens gives 5.08, an intermediate threshold 4.75, and aggressive truncation 5.02.

Codebook Grouping

Tokens are grouped by k-means over the VQ codebook embeddings, a fixed mapping computed once before training (Appendix A.2, Fig. 4). The grouping is used only inside the loss. The student still predicts over the full vocabulary and decoding is unchanged, so it does not lose the ability to distinguish tokens. The loss only stops penalizing confusions between tokens that are nearly identical in pixel space.

Tab. 6 (last column) already reports a sweep of the compression ratio: a moderate ratio (16384 to 4096) works best, and quality degrades only under extreme compression, where the targets are over-smoothed.

Parallel Decoding

Parallel decoding does introduce approximation errors, as it does not exactly match the autoregressive distribution. However, we do not observe a meaningful degradation in quality, with performance remaining nearly unchanged (4.94 vs. 4.96 see previous table). Inference is always performed using standard next-token decoding.

Remaining Teacher–Student Gap

The remaining gap mainly reflects student capacity. LlamaGen-B has about 3× fewer parameters than LlamaGen-L. VARKD closes 57% of the gap (6.53 to 4.62, five-seed means in Tab. 2, against 3.16 for the teacher), compared with 47% for both KD and GKD. Two observations support the capacity explanation. Larger teachers do not produce better B students (Tab. 4), matching the capacity-gap effect reported for LLM distillation (Zhong et al., 2024). And under the same XL teacher, VARKD closes a larger fraction of the gap for the stronger L student than for the B student (61% vs. 39%).

Reviewer_jegcOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies knowledge distillation for visual autoregressive image generation. It proposes VARKD, which trains a student on rollouts generated from a ground-truth prefix, weights teacher supervision by predictive confidence, and groups similar visual tokens when computing the distillation loss. The method improves upon KD, SeqKD, and GKD on several image-generation settings.

Critical strengths

Studying distillation for compact visual autoregressive models is worthwhile.

Critical weaknesses
  1. The submitted manuscript is incomplete. It refers readers to Appendix A.1 for baseline details and Supplementary Section B for additional results, but these sections are absent from the provided PDF.

  2. The methodological contribution is incremental. Student-generated rollouts are central to GKD, which the paper already uses as a baseline. The remaining contributions are confidence-based loss weighting and token grouping. The paper does not sufficiently establish a distinct insight specific to visual autoregressive distillation.

  3. The gains over GKD are modest, particularly for text-to-image generation. DPG-Bench increases from 82.00 to 82.34 and GenEval from 77.1 to 77.6. On the larger ImageNet students, the FID improvements are also small: 2.92 to 2.84 for LlamaGen-L and 2.20 to 2.16 for ARPG-L. The paper does not report uncertainty for the text-to-image results.

Questions for authors
  1. With the same KD initialization and additional training budget, how does VARKD compare with continued GKD training?

  2. The entropy-based weighting improves FID, but the proposed explanation remains indirect. Can the authors analyze how teacher entropy varies along student rollouts, how many positions are removed by truncation, and whether removing high-entropy positions accounts for the observed gain?

  3. Can the authors provide the referenced appendix and complete implementation details?

2 comments

We thank the reviewer for their feedback.

Appendix

The appendix was submitted as supplementary material and contains every section the reviewer refers to: baseline details (Sec. A.1), VARKD details, training efficiency and hyperparameters (Sec. A.2, Tab. 5–6), and extended results (Sec. B, Tab. 7, Fig. 5–7). Since this was the reviewer's first concern, we believe the assessment would change with the appendix in view.

Incremental Contribution

As explained in the general comment, the paper is mainly an analysis of why on-policy distillation does not transfer to visual AR models. To our knowledge, no prior analysis exists for this setting. Each component of VARKD addresses one identified failure mode, and the per-component ablation (reply to nukT) shows that each component with a quality effect contributes on its own. We openly build on prior mechanisms where they fit, and we do not consider that a weakness for an analysis of transfer.

Magnitude of Gains

All ImageNet results in Tab. 2 are averaged over five seeds, and pairwise t-tests give p < 0.05 for VARKD against every baseline in every setting (Sec. 4.2). Absolute differences shrink for strong students because little headroom remains. For example, measured as the fraction of the teacher–student gap closed, VARKD closes 61% for XL to L, compared with 43% for GKD and 26% for KD. Teacher scores are evaluated under the same conditions as all students.

Moreover, our GKD baseline already includes the mixed prefix formulation (see shared comment). The original fully on-policy GKD degrades FID from 5.20 to 6.02 (Tab. 6) under the same setting, so our comparisons understate the gain over the original on-policy distillation method.

Continued GKD Training

This comparison is already in the paper. GKD and VARKD start from the same KD checkpoint (250k iterations) and are trained for the same additional 20k iterations, with the same data and batch size. Every GKD vs. VARKD comparison in Tab. 2, 3 and 7 is therefore a comparison at equal continued training. Full training and evaluation details are in Sec. A.1–A.2.

Teacher Entropy and Truncation

Under ground-truth prefixes, teacher entropy decreases along the sequence, because more context makes the next token more predictable. Under student rollouts it remains high. The number of truncated positions is reported in Sec. A.2: thresholds of 0.01, 0.05, 0.10 and 0.50 retain about 92%, 79%, 61% and 29% of the generated tokens. The retained fraction also grows during training, from 63% to 77%, as the student aligns with the teacher.

Tab. 6 separates the two effects. Reweighting with almost no truncation (threshold 0.01) improves FID from 5.20 to 5.08. Also removing the least confident positions (0.05) brings it to 4.75. Most of the gain comes from removing unreliable positions, while truncating too aggressively removes useful signal (5.02 at 0.50).

Reviewer_jegcOpenReview ↗

Response of the author's rebuttal

Thank you for the clarification. I apologize for overlooking the supplementary appendix; it addresses several of my concerns, and I have raised my score to 2. I remain below weak accept because VARKD assumes a shared next-token vocabulary and compatible prefixes, limiting its applicability. Moreover, the teacher-size analysis is limited to class-conditional ImageNet generation, its non-monotonic trend may not hold for text-to-image generation. Table 1 and Figure 2 evaluates only one text-to-image teacher and does not report its scores under the same protocol; therefore, it is hard to tell the difference of the student and teacher model after distillation.Futher more, evaluating a stronger teacher such as Emu3.5, potentially through cross-tokenizer distillation, would make the contribution more compelling.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment to All Reviewers

We thank the reviewers for their time and feedback. Before the individual replies, we restate the goal of the paper, since several comments assess it mainly as a new method competing on absolute numbers.

The paper is primarily an analysis of whether on-policy distillation, which has become standard for compressing LLMs, transfers to visual autoregressive models. It asks three questions:

  1. Does it transfer as is? No. GKD is no better than plain KD on LlamaGen-B (4.94 vs. 4.93 FID, Tab. 2). Free-running student rollouts quickly leave the data manifold, and the original fully on-policy GKD is even worse than KD (6.02 vs. 5.20 FID, Tab. 6).
  2. Why does it fail? We identify three causes that are specific to the visual setting or much stronger there than in text: student rollouts drift off the data manifold, teacher feedback on drifted contexts is unreliable (its entropy stays high instead of decreasing along the sequence), and many visual tokens are nearly interchangeable in pixel space.
  3. Which ingredients fix it? We test a targeted remedy for each cause. Most of them are adaptations of ideas from the language literature (mixed prefixes, confidence-based truncation inspired by speculative decoding, parallel rollouts with ZipAR), and we cite them as such. Codebook relaxation addresses the only cause that has no counterpart in text.

Seen this way, the moderate absolute improvements are expected and are themselves part of the finding:

  • Strong baselines. We added our own mixed-prefix modification to the GKD baseline, so every comparison to GKD already contains part of our recipe and understates the gain over the original method.
  • Little headroom. The students are close to their teachers. LlamaGen-L, for example, is about 0.5 FID from its XL teacher. In terms of the teacher–student gap closed, the differences are clear: 61% for VARKD vs. 43% for GKD and 26% for KD on XL to L.
  • Consistency. The improvements hold over five seeds (p < 0.05 against every baseline in Tab. 2), across two model families (LlamaGen, ARPG), in 6 out of 7 teacher–student pairs, and on text-to-image generation.

The main takeaway is therefore not a single FID number. On-policy distillation does not work for visual AR models out of the box. It becomes consistently better than off-policy KD only when rollout drift, teacher reliability and token ambiguity are explicitly controlled, and our analysis shows which component addresses each of them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.