- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper uses diffusion inversion and reconstruction to produce alternative visual inputs for a frozen multimodal model. It considers intermediate reconstructed images, question-conditioned reconstructions, and difference maps between conditioned and unconditioned versions. Configurations are selected on a validation set for each model/task combination. The central idea is to expose diffusion visual priors through pixels, making them usable even by black-box multimodal models without modifying their weights.
Critical strengths
- The interface is appealing: decoded images can be supplied to existing multimodal systems without requiring access to internal representations.
- The experiments cover six multimodal model families and two diffusion backbones. The paper also introduces DV-Bench, a challenging VQA benchmark covering scene understanding, spatial reasoning, counting, and object recognition, providing a focused testbed for evaluating the benefits of diffusion-generated visual inputs.
Critical weaknesses
I find this a compelling findings paper, but it lacks in-depth analysis. My main concerns are as follows:
-
Limited explanation of why DPC works. The paper largely focuses on accuracy, without sufficiently explaining why the diffusion priors introduced by DPC improve model capabilities. This could be explored through deeper representation-level analysis or mathematical derivations. I also have a minor disagreement with the term “imagination”: the model generating the “imagined” images is different from the model answering the questions. The term therefore feels somewhat forced, borrowing from human cognition.
-
Concerns about evaluation and practical deployment. My understanding is that the method uses an auxiliary diffusion model to generate “imagined” images that enhance the answering model’s visual understanding. However, this introduces practical concerns. On the one hand, diffusion preprocessing takes time—approximately 43.43 seconds per sample with Flux 2 or 21.20 seconds with SD 3.5, as reported in Appendix B.4 and Table 5 (pp. 18–19)—and deploying an additional diffusion model incurs memory overhead. On the other hand, the generated images may alter answer-relevant details, such as object counts or spatial relationships, raising reliability concerns for applications that require faithful preservation of the original image.
-
Unmatched input lengths in the experiments. My main experimental concern is that A-O introduces additional visual tokens, while input lengths are not matched across the compared conditions. This raises the question of whether the comparisons isolate the contribution of the diffusion prior.
Questions for authors
-
Can you provide a theoretical explanation or representation-level analysis of how the introduced diffusion priors improve the answering model’s visual understanding?
-
Since image generation and question answering are performed by different models, what specifically justifies interpreting this process as imagination?
-
For A-O, do the gains persist when image count and visual-token budgets are matched, using controls such as duplicate original images, ordinary reconstructions, or crops?
-
Can you quantify changes in object counts, object presence, and spatial relationships, and examine how these changes affect answer correctness?
-
Can you report end-to-end latency and memory requirements, including diffusion preprocessing and multimodal inference, and assess whether the accuracy gains justify these costs?
-
With only 40 validation examples, how sensitive are the selected configurations and resulting gains to validation-set resampling (Page 6, line 276–277)?
-
Minor: in line 277, 349 and 351, What does “e.glet@tokeneonedot” mean?