acceptodds
Under review as a conference paper at ICLR 2027

DreamAR: Knowledge Distillation Sampling Using Next Scale Autoregressive Models

Abstract

We introduce DreamAR, an optimization framework that turns next-scale autoregressive (AR) models into priors for 2D and 3D representations, serving as the discrete AR counterpart to Score Distillation Sampling (SDS). While next-scale AR models rival diffusion in image fidelity and scale favorably, they cannot directly drive SDS, which relies on noise perturbation. Instead, these models operate across discrete multi-scale token maps, predicting categorical distributions over codebook indices in a single forward pass. To leverage this, we propose Knowledge Distillation Sampling (KDS), an objective that minimizes the cross-entropy between the model's predictive distribution and a differentiable tokenization of the rendered asset. Optimization progresses from coarse to fine, advancing to each scale after optimizing the preceding ones, with the scale index serving as the discrete analogue to diffusion timesteps. Because the prior's prediction is fully determined by the rendered view, the prompt and the current scale, KDS supervision involves no sampled noise and avoids the gradient variance that diffusion distillation needs dedicated machinery to reduce. Evaluated on text-to-2D generation (MJHQ-30K) using the Infinity-2B backbone, KDS achieves 24.24 FID, outperforming diffusion distillation baselines (36.55 for the strongest), and even surpasses distillation from SD3.5-Medium despite its lower native generation FID. For both text-to-3D generation and TRELLIS.2-based refinement, KDS performs on par with leading distillation sampling methods CFD and RFDS-Rev across appearance and geometry metrics.

Primary area:
generative models
Keywords:
next-scale autoregressive models, visual autoregressive modeling, score distillation sampling, 3D generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QwVwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DreamAR, which adopts available AR image generation models, for example, Infinity, to optimize image latent. DreamAR first encodes rendered images into several fine-grained levels. It then gives lower level latents with textual instructions to AR image generation models, which output the tokens for next finer-grained level. It then calculate the cross-entropy between the originally encoded rendered tokens and the tokens predicted by pretrained AR image generation models at the save next level of granularity. Such penalty is used to optimize the 2D latent, which can also be used to optimize any modality that can be optimized via 2D latents like NeRF.

Critical strengths
  1. Implementing next level AR image generative priors for optimizing 2D image latents is a useful contribution. This paper developed both KDS and a strong PT baseline. The strong PT baseline itself is a evidence for this application. I guess this work is the first work that applies next level AR image generative priors for SDS styled 2D latent optimization. But I am not an experienced expert of this field. This is my first time reviewing a paper related to the broader VAR generation.

  2. Table 1 and table 2 provide useful controled ablations, covering standard sampling of several teacher models; different distillation methods (though the teacher may not align); and 2 PT baselines with the final KDS on both Infinity and Switti. The benchmark covers 5000 prompts so I think it's sufficient.

Critical weaknesses
  1. The benefits of KDS over PT (deterministic) needs stronger evidence. For no matter 2D image generation or 3D generation (for example, using NeRF as a representation), all we want is a high-quality and sharp (single) sample, not a distribution. Methmatically, PT (deterministic) can guide the 2D latents to be optimized towards a deterministic, single sample, while both PT (stochastic) and KDS may guide the 2D latents to be optimized towards a distribution. For PT (stochastic), we can ignore here because it even explicitly optimize the 2D latents towards low-possibility directions, which is much more unstable. Therefore, I am highly concerned about the optimization path across different scales, and also the final sharpness of 2D image or 3D NeRF generation results.

  2. In table 1, KDS show clear advances over previous distillation methods, but in table 3 and table 4, the performance is consistently worse. It seems that KDS can bring better 2D optimization, but such abaility may not help with the 3D geometrical optimization. I noticed in section 3.4, the author also admitted that removing noise sampling may not bring sharpness or geometrical consistency.

  3. The additional benefits of KDS is very modest compared to simple PT baselines, as shown in table 2. Also, in table 2, even PT (stochastic) delivers a FID of 28.57 using Infinity-2B, which is ranked second in table 1's distillation sampling comparison, making the performance contribution of KDS itself more modest.

Questions for authors

please see the weaknesses

Reviewer_7rg6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript proposes distilling the generative prior of a next-scale AR model into an optimized continuous representation, DreamAR, including optimization, lower-scale supervision, and frozen-host regularization to stabilize optimization.

Critical strengths

The paper and method are easy to follow.

The qualitative and quantitative results are promising.

Critical weaknesses

It remains unclear why the diffusion model used in SDS is replaced with a next-scale AR model and what advantages this offers over using a diffusion model. Furthermore, VSD, as an improved variant of SDS, has demonstrated better performance. Why do the authors continue to explore SDS rather than VSD? This raises another question: Does SDS with a next-scale AR model achieve better 3D generation performance than VSD?

KDS repeatedly invokes a frozen AR teacher, making its inference efficiency critical to optimization cost. Could KDS be further combined with acceleration methods for next-scale AR models, such as FastVAR, FasterVAR, and ToProVAR, to further improve optimization efficiency?

Questions for authors

The motivation for replacing the diffusion model used in SDS with a next-scale AR model remains unclear. In theory, the diffusion model could be replaced with any generative model, such as GANs, next-token AR models, or GRNs.

2 comments

Thank you for the authors' responses, which have addressed some of my concerns. However, I still have a question regarding KDS. Although KDS does not involve an explicit noising and denoising process, it adopts an SDS-like approach. I wonder whether KDS could be extended to a VSD-like formulation to achieve better performance, given that VSD has demonstrated advantages over SDS in the original setting.

Dear Reviewer,

Thank you for following up on our responses and for raising this interesting question.

We agree that extending KDS to a VSD-inspired formulation is a promising research direction that could further improve KDS, much like VSD improved SDS. One could envision an analogous formulation for next-scale AR models by learning the token distributions induced by the optimized hosts and incorporating them into the distillation objective.

It is worth noting that KDS operates through a fundamentally different mechanism, which, as demonstrated empirically, already achieves superior distillation performance in 2D without such additions. According to the analysis presented in the SDS-Bridge paper, VSD mitigates the distribution mismatch between the base model's native samples and the host renders. In KDS, we progressively optimize the host from coarse to fine scales, such that predictions at each scale are conditioned on previously optimized lower scales. These conditions are therefore expected to be better aligned with the base model's learned distribution. Thus, KDS might benefit less from such an extension. Nevertheless, we believe this is an interesting direction worthy of further investigation.

Just to be clear, we view our work as establishing the foundations for distillation sampling with next-scale AR models, upon which more advanced distillation formulations can be developed, much like the research that followed the introduction of SDS. We thank the reviewer for this valuable suggestion and will explicitly highlight such extensions to our distillation formulation as promising directions for future work in the paper.

We hope our responses have addressed your previously raised concerns and your latest question, and would be happy to clarify any further concerns or questions you may have before the discussion period concludes.

Reviewer_B2HROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes to improve next-scale autoregressive models by incorporating pseudo-target formulation in a diffusion-style loss (Eq 5) and a weighted scale-wise token probabilities-based distillation (KDS, Eq 8). On a higher level, it addresses issue of conflicting targets by stratifying losses by scale with a "frozen" host regualrization (or a delayed host update IMO).

Critical strengths
  • The proposed distillation method makes intuitive sense.
  • The paper is relatively easy to follow for someone not in this particular field.
Critical weaknesses
  1. In Table 1, distillation sampling perform worse than base models (which undermines the proposed method). Is it because the performance is distilled from those base models? While the concepts may be implied, please clarify the differences among teachers, hosts, and base models in this work.
  2. Infinity performs better than SD3.5 in the original work [1], so Table 1 comparing different methods using different base models is not fair. It would be fair if all sampling methods use the same base model. Similarly for Table 3, the strongest baseline RFDS-Rev uses SD3.5 while the proposed method uses infinity. Naturally one would like to see if RFDS-Rev would benefit from a better base model like infinity.
  3. More recent benchmarks reported in [1] such as GenEval and ImageReward should be used as well. Right now there is only FID and CLIP.
  4. It is unclear how backpropogation works in this method: backprop on (10) requires differentiating through the dequantizer in (4), which could be a constant.

Minor issues.

  • Figure 1 is uninformative without any baseline comparisons.

  • Table 4 is overflowing.

  • Eq (10) unnecessarily abuses notations (or put it mildly, is confusing to read). For example LiL_i is a loss at scale ii in (8), so LL should be a vector in (10). Having subscripts ii for summation index and kk for active scale, when ii already appeared previously, is quite confusing. Then it is noted L=LKDSL=L^{KDS}, making it easily conflated with LiKDSL_i^{KDS}.

[1] Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis. Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng, Xiaobing Liu. CVPR, 2025.

Questions for authors

See weaknesses.

  1. Is this an inference-time method focused on the training of ϕ\phi rather than fine-tuning of the model itself?
  2. Are there any auxiliary models (in addition to base) used in the process compared to previous methods like SDS?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.