acceptodds
Withdrawn from ICLR 2027

Machines See Better When They Imagine: Diffusion Imagination Reveals Better Visual Priors for Multimodal Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) are typically evaluated with human-readable images, assuming that representations optimized for human perception are also optimal for machine reasoning. We challenge this assumption by studying whether intermediate images along a diffusion denoising trajectory, which acts like machine imagination, can serve as effective visual priors for visual question answering (VQA). Given an input image, we perform diffusion inversion and reconstruction under different conditional priors, and further derive residual difference maps that highlight question-induced visual changes. Evaluated across top tier MLLMs on our proposed benchmark DV-Bench, the integration of diffusion visual priors consistently improve MLLMs' performance, especially on reasoning-heavy tasks. Our analysis shows that the gains are not only driven by a universal denoising step or conditional priors, but also by task type, MLLM and diffusion backbone. The same diffusion prior can bring significant improvement for one MLLM while harming another, revealing a model-dependent “meat/poison” effect. These results suggest that diffusion priors as visual representation have great potential for enhance the visual reasoning of different MLLMs. The source code will be fully released upon paper acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MLLM, VQA, Visual representation, Generative models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sUDGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper uses diffusion inversion and reconstruction to produce alternative visual inputs for a frozen multimodal model. It considers intermediate reconstructed images, question-conditioned reconstructions, and difference maps between conditioned and unconditioned versions. Configurations are selected on a validation set for each model/task combination. The central idea is to expose diffusion visual priors through pixels, making them usable even by black-box multimodal models without modifying their weights.

Critical strengths
  1. The interface is appealing: decoded images can be supplied to existing multimodal systems without requiring access to internal representations.
  2. The experiments cover six multimodal model families and two diffusion backbones. The paper also introduces DV-Bench, a challenging VQA benchmark covering scene understanding, spatial reasoning, counting, and object recognition, providing a focused testbed for evaluating the benefits of diffusion-generated visual inputs.
Critical weaknesses

I find this a compelling findings paper, but it lacks in-depth analysis. My main concerns are as follows:

  • Limited explanation of why DPC works. The paper largely focuses on accuracy, without sufficiently explaining why the diffusion priors introduced by DPC improve model capabilities. This could be explored through deeper representation-level analysis or mathematical derivations. I also have a minor disagreement with the term “imagination”: the model generating the “imagined” images is different from the model answering the questions. The term therefore feels somewhat forced, borrowing from human cognition.

  • Concerns about evaluation and practical deployment. My understanding is that the method uses an auxiliary diffusion model to generate “imagined” images that enhance the answering model’s visual understanding. However, this introduces practical concerns. On the one hand, diffusion preprocessing takes time—approximately 43.43 seconds per sample with Flux 2 or 21.20 seconds with SD 3.5, as reported in Appendix B.4 and Table 5 (pp. 18–19)—and deploying an additional diffusion model incurs memory overhead. On the other hand, the generated images may alter answer-relevant details, such as object counts or spatial relationships, raising reliability concerns for applications that require faithful preservation of the original image.

  • Unmatched input lengths in the experiments. My main experimental concern is that A-O introduces additional visual tokens, while input lengths are not matched across the compared conditions. This raises the question of whether the comparisons isolate the contribution of the diffusion prior.

Questions for authors
  1. Can you provide a theoretical explanation or representation-level analysis of how the introduced diffusion priors improve the answering model’s visual understanding?

  2. Since image generation and question answering are performed by different models, what specifically justifies interpreting this process as imagination?

  3. For A-O, do the gains persist when image count and visual-token budgets are matched, using controls such as duplicate original images, ordinary reconstructions, or crops?

  4. Can you quantify changes in object counts, object presence, and spatial relationships, and examine how these changes affect answer correctness?

  5. Can you report end-to-end latency and memory requirements, including diffusion preprocessing and multimodal inference, and assess whether the accuracy gains justify these costs?

  6. With only 40 validation examples, how sensitive are the selected configurations and resulting gains to validation-set resampling (Page 6, line 276–277)?

  7. Minor: in line 277, 349 and 351, What does “e.glet@tokeneonedot” mean?

Reviewer_GYJ8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper uses diffusion reconstruction and question-conditioned residual maps to modify visual inputs for MLLMs. Validation-selected configurations improve accuracy across several models and tasks.

Critical strengths

The approach requires no updates to the target MLLM. Experiments cover six MLLMs and two diffusion backbones. Appendix B.1 also demonstrates gains using one fixed configuration per model and dataset.

Critical weaknesses

1. Unclear incremental contribution and insufficient comparative evaluation.

The core idea is to pass the original image through a generator and modify the visual input in a way that may improve downstream performance. Conceptually, this is the multimodal counterpart of modifying a text prompt: changing the input can change a model’s answer and sometimes improve accuracy.

If the intended contribution is that diffusion-based processing provides a particularly effective approach to visual prompting, this claim requires stronger comparative evidence. The main paper compares against the original-image baseline and analyzes variants of its own method. Comparisons with alternative perturbations appear only in Appendix B.3 and are limited to Gaussian noise and random sharpening/blurring. Broader comparisons with competitive image perturbation, visual prompting, and text-prompt optimization methods under comparable tuning budgets are needed to establish the particular value of the diffusion-based design. If the contribution is primarily an empirical observation, the paper should clarify what generalizable insight it provides beyond model-dependent input sensitivity.

2. The persistence of the gains across model capability levels is not established.

The evaluation includes only one variant from each model family. Although gains are reported across these selected models, the experiments do not establish whether the phenomenon persists as models within the same family become more capable. A controlled comparison of weaker and stronger variants would help determine how the gains depend on baseline capability and whether they diminish as models improve.

Questions for authors

See weaknesses.

Reviewer_5riJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper tests whether images taken from the middle of a diffusion denoising run help MLLMs answer visual questions better than the original image. The method inverts an image partway to noise, then rebuilds it with no prompt or with the question as the prompt. A variant gives the model the original image plus a difference map between the two rebuilds. The authors test six MLLMs on a benchmark built from examples two models failed on. For each model and task, they pick the best setting on a small validation set. They report gains over the original image, and cases where the same setting helps one model and hurts another.

Critical strengths

The tests are broad: two diffusion backbones, six current MLLMs, perturbation baselines (Table 4), results on the unfiltered set (Table 2), and full cost numbers (Table 5). The paper also states its failure modes in the open. The coverage is thorough, even though I do not think the results support the claims, for the reasons below.

Critical weaknesses
  1. The motivation does not hold, and the setting makes it worse. I think this paper's fundamental motivation is not convincing to me. Raw pixel image is never optimized for human perception, it is just the images taken by the camera from the real world. The method is also test-time only: all six MLLMs were trained on natural images and are used as-is, and the diffusion images appear only at test time. If intermediate diffusion images were really the better input format for machines, the model should be trained with them, not just tested on them.

  2. The gains can come from selection noise, and the A-O comparison is unfair. Table 1 picks the top-1/top-3 DPCs per model and per task on a 40-example validation set. The baseline gets no such tuning. With this many picks on so few examples, gains of this size can appear by chance. The "meat/poison" pattern points the same way: the same DPC is the best input for one model and the worst for another. A real effect should not flip sign like that; selection noise on 40 examples does exactly that. And A-O gives two images, the original plus the difference map, while the baseline gives one. Test original + original, or original + a noise map, to show the gain comes from diffusion and not from the second image.

Questions for authors

Please refer to above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.