- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces DreamAR, which adopts available AR image generation models, for example, Infinity, to optimize image latent. DreamAR first encodes rendered images into several fine-grained levels. It then gives lower level latents with textual instructions to AR image generation models, which output the tokens for next finer-grained level. It then calculate the cross-entropy between the originally encoded rendered tokens and the tokens predicted by pretrained AR image generation models at the save next level of granularity. Such penalty is used to optimize the 2D latent, which can also be used to optimize any modality that can be optimized via 2D latents like NeRF.
Critical strengths
-
Implementing next level AR image generative priors for optimizing 2D image latents is a useful contribution. This paper developed both KDS and a strong PT baseline. The strong PT baseline itself is a evidence for this application. I guess this work is the first work that applies next level AR image generative priors for SDS styled 2D latent optimization. But I am not an experienced expert of this field. This is my first time reviewing a paper related to the broader VAR generation.
-
Table 1 and table 2 provide useful controled ablations, covering standard sampling of several teacher models; different distillation methods (though the teacher may not align); and 2 PT baselines with the final KDS on both Infinity and Switti. The benchmark covers 5000 prompts so I think it's sufficient.
Critical weaknesses
-
The benefits of KDS over PT (deterministic) needs stronger evidence. For no matter 2D image generation or 3D generation (for example, using NeRF as a representation), all we want is a high-quality and sharp (single) sample, not a distribution. Methmatically, PT (deterministic) can guide the 2D latents to be optimized towards a deterministic, single sample, while both PT (stochastic) and KDS may guide the 2D latents to be optimized towards a distribution. For PT (stochastic), we can ignore here because it even explicitly optimize the 2D latents towards low-possibility directions, which is much more unstable. Therefore, I am highly concerned about the optimization path across different scales, and also the final sharpness of 2D image or 3D NeRF generation results.
-
In table 1, KDS show clear advances over previous distillation methods, but in table 3 and table 4, the performance is consistently worse. It seems that KDS can bring better 2D optimization, but such abaility may not help with the 3D geometrical optimization. I noticed in section 3.4, the author also admitted that removing noise sampling may not bring sharpness or geometrical consistency.
-
The additional benefits of KDS is very modest compared to simple PT baselines, as shown in table 2. Also, in table 2, even PT (stochastic) delivers a FID of 28.57 using Infinity-2B, which is ranked second in table 1's distillation sampling comparison, making the performance contribution of KDS itself more modest.
Questions for authors
please see the weaknesses
Thank you for the authors' responses, which have addressed some of my concerns. However, I still have a question regarding KDS. Although KDS does not involve an explicit noising and denoising process, it adopts an SDS-like approach. I wonder whether KDS could be extended to a VSD-like formulation to achieve better performance, given that VSD has demonstrated advantages over SDS in the original setting.