acceptodds
Withdrawn from ICLR 2027

Enabling Visual Reasoning over Generated Images in Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reasoning has become central to LLMs, whereas diffusion models have largely been treated as visual generators. As image generation increasingly requires understanding complex and compositional prompts, existing diffusion models often rely on external LLMs or VLMs for reasoning, leaving instruction understanding decoupled from visual generation. We introduce Seenk, a visual reasoning paradigm for correcting mismatches between generated images and their text prompts. Given an image-description pair, Seenk determines whether any mismatches exist and, if so, generates a mask-annotated image to localize them. This image serves as an explicit visual thought, guiding an SFT-adapted editor to correct the identified mismatches. To equip diffusion models with this capability, we introduce two core visual reasoning modules: VR-Base and VR-Refiner. VR-Base is trained via SFT on VRB-Dataset, a visual reasoning dataset constructed through our data pipeline, to learn mismatch detection and visualization. VR-Refiner is trained via DPO on VRR-Dataset, a preference dataset constructed with a VLM, to further improve visual reasoning and mask localization accuracy. Experiments on visual reasoning and correction demonstrate that Seenk achieves competitive performance against strong open- and closed-source models, as well as pipelines that employ an LLM as an external reasoner. Further experiments show that, like LLMs, Seenk benefits from larger reasoning budgets, generating more visual thoughts to better handle complex prompts.

Primary area:
generative models
Keywords:
visual reasoning, image generation, diffusion model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_W9JKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Seenk, a framework for detecting and correcting inconsistencies between generated images and their text descriptions. The method fine-tunes Qwen-Image-Edit-2511 to generate mask-annotated images as intermediate visual thoughts, which are then used to guide a separately fine-tuned image editor. The visual reasoner is trained through SFT followed by spatially weighted DPO, using automatically constructed training and preference datasets. Experiments show substantial improvements in mismatch localization on the authors' benchmark and competitive image editing performance.

Critical strengths

The paper is generally well written, and the overall pipeline is easy to follow. The automated data construction process is practical and provides a reasonable way to obtain supervision for mismatch localization. The improvement in localization accuracy after SFT is substantial, suggesting that diffusion-based editing models can be adapted effectively to this task.

Critical weaknesses

1. Limited novelty and insufficient discussion of closely related work.

My main concern is the novelty of the proposed framework. Several existing works have already explored very similar ideas, particularly GVCoT (2026), MURE (2025), and D-TIIL (ICLR 2024). GVCoT is especially relevant, as it already explores generating visual spatial cues as intermediate reasoning steps before image editing, using SFT followed by further optimization. MURE also incorporates visual masks into an explicit reasoning process, while D-TIIL addresses text-image inconsistency localization using diffusion models.

These works are not identical to Seenk, and I recognize that this paper focuses specifically on detecting mismatches against a target description. However, the paper does not adequately discuss these closely related approaches or establish why its formulation offers a meaningful advantage. The core framework still appears to be a fairly straightforward combination of mismatch localization, mask generation, and image editing.

The proposed DPO refinement also seems incremental. The spatial weighting strategy is reasonable, but its contribution is limited in the reported experiments: pixel IoU improves only from 0.768 to 0.775, while VQAScore increases from 0.750 to 0.752. Without comparisons to standard DPO or simpler mask refinement techniques, it is difficult to assess the benefit of this additional training stage.

2. Insufficient experiments to establish the necessity and generality of the proposed method.

The most important missing experiment, in my opinion, is a direct editing baseline trained on the same data. Since the method constructs paired inconsistent and corrected images, a natural alternative is to fine-tune Qwen-Image-Edit directly to map the inconsistent image and target description to the corrected image, without generating an intermediate mask. I do not see this comparison in the paper. It is therefore unclear whether the improvement actually comes from the proposed visual reasoning framework or simply from task-specific fine-tuning on the constructed dataset.

Relatedly, most baseline models are not fine-tuned on the same task and data. Comparing a task-specific trained model against mostly prompted general-purpose models does not sufficiently isolate the methodological contribution. A fine-tuned VLM-based localization pipeline followed by the same editor would also be an informative comparison.

The evaluation is relatively limited. The main benchmark contains only 100 curated examples, and several complex error categories, including geometry, physics, and spatial relationships, have very few or no evaluation samples. The experiments on additional reasoning steps are largely qualitative. Moreover, the method is evaluated using only Qwen-Image-Edit as its backbone, without demonstrating transfer to another editing architecture such as FLUX.2.

Another missing aspect is background preservation. The method aims to correct local inconsistencies while retaining correct content, but the paper does not directly measure changes outside the intended editing regions. Global similarity metrics such as DINO and CLIP-I are not sufficient to establish this property.

Finally, the claim of native diffusion-based visual reasoning deserves more investigation. Qwen-Image-Edit already includes a Qwen2.5-VL-based semantic encoder. Without further analysis, it is unclear to what extent the learned mismatch understanding comes from the pretrained vision-language representations rather than the diffusion generator itself. This does not invalidate the method, but it makes the paper's interpretation of the learned capability less convincing.

Overall, I find the technical pipeline reasonable, but the contribution seems largely driven by a standard synthetic data construction and fine-tuning recipe. The experimental evidence does not sufficiently establish the necessity of the proposed intermediate reasoning representation or its advantage over closely related alternatives. Given the limited novelty and missing controls, I do not think the current submission meets the acceptance bar for ICLR.

Refs:

[1] GVCoT: Generative Visual Chain-of-Thought for Image Editing (2026). https://arxiv.org/abs/2603.01893

[2] MURE: Beyond Textual CoT: Interleaved Text-Image Chains with Deep Confidence Reasoning for Image Editing (2025). https://arxiv.org/abs/2510.08157

[3] D-TIIL: Exposing Text-Image Inconsistency Using Diffusion Models (ICLR 2024). https://arxiv.org/abs/2404.18033

Questions for authors
  1. How does Seenk differ fundamentally from GVCoT, MURE, and D-TIIL? In particular, what advantage does the proposed visual reasoning formulation provide beyond the existing mask-based reasoning and editing approaches?

  2. What happens if Qwen-Image-Edit is directly fine-tuned on the same inconsistent/corrected image pairs, without generating intermediate visual thoughts? This comparison seems essential to establish the contribution of the proposed reasoning stage.

  3. How much of the improvement comes specifically from spatially weighted DPO? Could the authors compare against standard DPO and simple mask post-processing methods? Given the small gains, are the improvements consistent across evaluation samples?

  4. Can the authors provide stronger evidence of generalization, including a larger independent benchmark, quantitative background preservation metrics, and results using another image editing backbone?

  5. Since Qwen-Image-Edit contains a pretrained Qwen2.5-VL-based semantic encoder, what evidence supports attributing the learned reasoning capability to the diffusion component? Is there any analysis separating the contributions of the semantic encoder and the generator?

Reviewer_VATKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Seenk, a framework for identifying and correcting inconsistencies between an existing image and its target description. Given an image-description pair, a diffusion-based visual reasoner generates a mask-annotated image in which inconsistent regions are covered in black. This intermediate image, termed a “visual thought,” is then passed to an SFT-adapted image editor that regenerates the marked regions. The visual reasoner is trained in two stages. VR-Base is obtained through supervised fine-tuning on VRB-Dataset, a synthetically constructed dataset of image-text inconsistencies and corresponding masks. VR-Refiner is subsequently trained using a spatially weighted Diffusion-DPO objective on VLM-ranked preference pairs from VRR-Dataset. The paper evaluates mismatch localization, downstream correction, behavior on error-free images, and transfer to MagicBrush. On the authors’ 100-example localization benchmark, Seenk obtains substantially higher IoU than the evaluated general-purpose baselines.

Critical strengths
  1. The intermediate spatial representation is explicit and operationally useful.
 The predicted mask can be inspected independently and directly used to constrain downstream editing. This enables separate evaluation of localization and correction and provides more transparency than an entirely implicit editing pipeline.
  2. The paper presents a reasonably complete training and data construction pipeline.
 The authors construct supervised and preference datasets, provide detailed training objectives, evaluate both localization and correction, and include an initial cross-dataset experiment on MagicBrush. The large localization improvement of VR-Base over the underlying Qwen-Image-Edit model indicates that explicit task-specific supervision is effective.
Critical weaknesses
  1. The novelty relative to prior automatic-mask editing and text-image inconsistency localization work is insufficiently established.
At a high level, Seenk predicts which image regions conflict with a target description and then edits those regions. Closely related pipelines already exist. DiffEdit (Couairon et al., ICLR 2023) automatically estimates an edit mask before diffusion-based editing, while D-TIIL (Huang et al., ICLR 2024) uses diffusion models to localize text-image inconsistencies. InstructEdit also combines automatically obtained masks with mask-guided editing. These works differ in mask construction and model architecture, but they are direct conceptual and experimental comparisons. Neither DiffEdit nor D-TIIL appears in the quantitative localization evaluation. Without these comparisons, it is difficult to assess whether the paper introduces a substantively new reasoning capability or mainly a supervised implementation of an established localization-then-editing pipeline. Calling the generated mask a “visual thought” does not by itself establish that the system performs reasoning beyond learned mismatch segmentation.
  2. The experiments do not cleanly support the claimed contributions or broad generalization.
The main localization benchmark contains only 100 examples manually curated from a held-out subset produced by the same synthetic error-generation and annotation pipeline as the training data. The selection explicitly favors high-quality masks and category coverage, while several difficult error types have very few or no benchmark examples. No uncertainty estimates or repeated-run results are reported.
Furthermore, most of the quantitative improvement comes from SFT and task-specific editor adaptation. VR-Refiner improves pixel IoU only from 0.768 to 0.775 and downstream VQAScore from 0.750 to 0.752, without variance estimates or a comparison against continued SFT using a comparable training budget. The correction comparison also mixes Seenk, whose reasoner and editor are both trained on task-specific paired data, with models used mainly through prompting. Consequently, the current results do not isolate whether the benefit arises from the proposed visual-thought representation, additional task-specific training, the specialized editor, or DPO.
Finally, the claim that Seenk benefits from a larger “reasoning budget” is supported only by a small number of qualitative examples. There is no quantitative curve relating the number of sampled thoughts or correction rounds to accuracy, computational cost, or failure rate.
Questions for authors
  1. Related-work comparison: How does Seenk compare quantitatively with DiffEdit and D-TIIL on the same image-description inputs and ground-truth masks? If these methods cannot be evaluated directly, what precise input or task difference prevents the comparison?
  2. Independent evaluation: What are the localization results on the complete 1,000-example held-out set, rather than the manually curated 100-example subset? Can the authors additionally evaluate on a benchmark whose inconsistencies and masks were not produced by the same synthetic construction pipeline used for training?
  3. Contribution attribution: When all predicted masks are passed to the same editor, how do VR-Base, VR-Refiner, D-TIIL, DiffEdit, a generic VLM-based segmenter, and the ground-truth mask compare? This controlled experiment would separate localization quality from editor adaptation.
  4. DPO significance: Are the improvements from 0.768 to 0.775 IoU and from 0.750 to 0.752 VQAScore statistically stable across random seeds? How does DPO compare with continued SFT or best-of-eight selection under a matched data and computation budget?
  5. Reasoning budget: Across the full multi-error evaluation set, how does correction accuracy change as the number of visual-thought samples or correction rounds increases? What is the associated inference cost, and how often does additional computation introduce false-positive masks or unnecessary edits?
Reviewer_HYGsOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses image–description inconsistencies by fine-tuning a diffusion model to produce an image with mismatched regions marked in black, then passing this intermediate image to a separately adapted editor. The checker is trained through SFT and subsequently refined using spatially weighted diffusion DPO (§3.2). The paper also introduces synthetic localization and preference datasets (§3.3).

The reported results support the usefulness of learned mismatch localization within this pipeline. However, evidence is incomplete for three central claims: a distinct visual reasoning mechanism, benefits beyond equivalent direct-editing training, and generalization beyond the constructed evaluation setting.

Critical strengths
  1. Useful intermediate localization—sufficient evidence within the reported setting. Table 2 shows that adding VR-Base while retaining the untuned editor increases VQAScore from 0.679 to 0.723. This supports the practical value of its spatial guidance, although it does not establish a distinct reasoning mechanism.

  2. Meaningful component controls—partial evidence for attribution. Table 2 separates checker training, editor adaptation, and DPO. Appendix C.2 confirms that the SFT and SFT+DPO checkers use the same adapted editor. Appendix B/Table 4 additionally compares editors receiving identical ground-truth masks, helping isolate their ability to execute spatial guidance.

  3. Clear implementation and evaluation disclosures. The mask representation, objectives, spatial weighting, adapter configuration, prompts, and extraction rules are described in §3.2 and Appendices A, C, and D. Appendix E/Figure 9 explicitly documents mask-format artifacts, while Appendix F evaluates behavior on 100 corresponding error-free images. These details make the method and several evaluation limitations inspectable.

Critical weaknesses
  1. Insufficient evidence for visual reasoning. Section 3.2 and Eq. 5 describe mismatch localization followed by masked editing. The intermediate mask provides useful spatial guidance, but does not itself establish a distinct reasoning mechanism. The paper should clarify what capability distinguishes this approach from a conventional localization–segmentation–editing pipeline.

  2. Missing matched direct-editing baseline. Table 2 does not compare against Qwen-Image-Edit fine-tuned on the same data under a comparable training budget without intermediate masks. Consequently, gains from the proposed decomposition remain confounded with additional training. DPO provides only modest improvements: IoU 0.768→0.775 and VQAScore 0.750→0.752, while DINO decreases 0.891→0.890. Uncertainty estimates and matched continued-SFT controls are needed.

  3. Limited generalization and scaling evidence. The core benchmark contains 100 curated examples sharing the training data’s synthesis procedure (§3.3.1). MagicBrush provides partial external validation, but gains over FLUX.2-dev are only 0.001 DINO and 0.002 CLIP-I. Exact-black mask extraction and box-versus-mask outputs also complicate Pixel IoU comparisons (Appendices A/D/E). Finally, §4.4/Fig. 7 presents only two examples, insufficient to establish systematic benefits from increased inference budgets.

Questions for authors
  1. How does this paper differ functionally from VLM+SAM localization followed by the same editor?

  2. Does it outperform same-data, comparable-budget direct-editing and continued-SFT baselines?

  3. Do gains persist on independent natural errors, across mask-extraction thresholds, and under quantitative, equal-budget inference comparisons?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Clarifying Core Misunderstandings About Seenk

We thank the reviewers for their time and respectfully ask the Area Chair to intervene. The reviews judge Seenk against objectives it does not pursue, leaving no meaningful basis for a rebuttal. Further discussion requires the reviewers to revisit the Introduction and reassess the work based on its stated task, motivation, and contributions. We clarify these three misunderstandings below.

  1. [R-HYGs W1, R-VATK W1 Q1, R-W9JK W1 Q1] Misunderstanding the mismatch correction task. As shown in the table below, Seenk addresses a different problem from conventional mask localization, image editing, and localization-based image editing. The similarity of their inputs and outputs makes the key distinction easy to overlook. In those tasks, users specify what to find or change. In Seenk, the model must make these key decisions without human-specified targets. In the mismatch correction task of Seenk, the text description is the only case-specific guidance in the prompt. The diffusion model itself must reason implicitly over the image and description, identify what is inconsistent, and mark the affected region.

    TasksInputIntermediateOutput
    Mask LocalizationImage + "locate the blue butterfly"--Mask Image
    Image EditingImage + "change the blue butterfly to red"--Edited Image
    Localization-based Image EditingImage + "change the blue butterfly to red"Mask ImageEdited Image
    Reasoning-based Mismatch Correction (Ours)Image + "a blue butterfly rests on a yellow flower"Mask ImageCorrected Image

    This distinction clarifies the comparisons raised in the reviews. DiffEdit derives a mask by contrasting denoising predictions under different prompts. Seenk learns to infer the discrepancy and generate the mask directly. D-TIIL also addresses inconsistency localization, but it obtains the result through multiple stages of alignment, mask generation, and image editing. Its focus is on constructing that procedure, whereas ours is on eliciting the judgment within the diffusion model. GVCoT, MURE, and InstructEdit start from an explicit edit instruction that specifies the intended change. GVCoT is built on a unified multimodal model, MURE includes textual reasoning, and InstructEdit uses a language processor.

  2. Misunderstanding the motivation of Seenk. Large-scale generative pretraining gives LLMs the understanding and generation capabilities on which their reasoning builds. Diffusion models also undergo large-scale generative pretraining, yet they are still viewed primarily as visual generators, and their capacity for higher-order understanding and reasoning remains underexplored. Vision Banana and SenseNova-Vision show that fine-tuning visual generators with suitable data can elicit capabilities such as segmentation and depth estimation. This motivates our question: Can a diffusion model also learn to perform more demanding visual reasoning through its own generation process?

  3. Misunderstanding the contribution of Seenk. We propose a novel visual reasoning paradigm that equips the diffusion model itself to perform implicit visual reasoning and express its judgments directly as visual outputs. Depending on the task, these outputs can take the form of segmentation masks, super-resolved images, depth maps, optical flow maps, or surface normal maps, all of which can guide subsequent processing. Compared with reasoning expressed in text by a VLM, direct visual generation can convey more precise region boundaries, finer visual details, and pixel-level information about depth, motion, and surface orientation. We demonstrate the value of this paradigm on mismatch correction, where the generated mask provides precise spatial guidance for the downstream editor.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.