acceptodds
Under review as a conference paper at ICLR 2027

Local Diffusion Control: A Tool Interface for VLM-Guided Correction

Abstract

Text-to-image generators often produce visually plausible scenes but make local semantic errors, such as omitting an object, assigning an attribute to the wrong object, or introducing unwanted details. In this work, we investigate whether such errors can be corrected within the same sampling trajectory, after they become visible but before denoising is complete. We introduce Direct-CFG, a spatially and temporally bounded intervention interface that uses intermediate clean-image estimates to implement three operations: strengthening visible elements, suppressing unwanted content, and locally re-noising a region followed by its regeneration. We distinguish between two tasks: the effectiveness of the correction tools themselves and the ability of vision-language models (VLMs) to determine when, where, and how to apply them. We evaluated the approach using fixed human-specified interventions and automated VLM control, revealing the capabilities and limitations of local correction of generation trajectories.

Primary area:
generative models
Keywords:
diffusion models, text-to-image generation, trajectory correction, vision-language models, classifier-free guidance, local control

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QpxsOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces Direct-CFG, which adds vlm-based local corrections during the denoising stage of text-to-image diffusion models. Specifically, the VLM inspects the decoded image at chosen timesteps, and conducts a call to different kinds of CFG variants or renoising and repainting operations. They evaluate in a number of generation settings (insertions, attribute change, etc.) and also with human-in-the-loop style evaluation, and show that tools can make local edits during generation.

Critical strengths
  1. Clear problem motivation: The problem of text-to-image model not being entirely faithful to the specified prompt due to local semantic errors is clear and well motivated.

  2. Interesting solution of using tools during sampling

Critical weaknesses
  1. Clarity and writing. The paper is not clear, and is hard to follow. In large parts, it reads like a project progress report rather than a research paper. A few examples: L237 “the originally planned empty-background control”, L265 "Confirmation lacks general artifact ratings but now includes removal patch checks", L417 “A pilot on an earlier stack” etc., with many more such examples. In addition, sentences are dense and overly complex and opaque, which makes reading it very difficult. For example, defining a concept by what it does not mean: "no op adds no intervention. Generation and already active actions continue, so it does not mean that the latent stops changing". In addition, the paper uses terms which are not defined yet, e.g., "GM" and "atoms" are both used in Section 5.1 and Table 1 before Section 5.4 defines them.

  2. Lack of baselines. The authors do not compare to any prior work they cite (most notably PPAD or SLD), or to obvious baselines such as editing the resulting image with Qwen-Image-Edit. They only compare against post-hoc or inpainting with the same calls best-of-N seeds. It is thus very hard to understand if the proposed method helps compared to existing alternatives.

  3. Problematic experiment design. The authors never show that in-trajectory editing beats the simple alternatives. In the one fair comparison (1B), just inpainting the final image or picking the best of a few samples does as well or better. The big gain in Experiment 2 seems to come mostly from rewriting the prompt, since the authors' own control gets the same improvement without any editing. And when everything is automatic (Experiment 3), there's no clear improvement.

  4. Questionable motivation for box-based editing. Many potential errors cannot be fitted around a box, such as counts, spatial arrangements, etc. According to the authors' own experiment (Experiment 2), in 30% of the cases humans provided a bbox larger than 90% or more of the image. This suggests that the central motivation for the method is flawed in many cases.

  5. Many hand-tuned settings and hidden controller logic. The method depends on a long list of hand-set parameters and hidden controller logic.

  6. Very high computational cost. The method requires two extra model evaluations per step and each VLM inspection also pauses generation for 20–26 seconds, which causes a very large slowdown.

  7. All results are VLM judgments with no human calibration.

Questions for authors

Can you compare against a dedicated instruction-based editor applied to the final image (e.g., Qwen-Image-Edit), and against PPAD, the closest prior work, on the same prompts?

Reviewer_z7MZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether semantic errors in text-to-image generation can be corrected within an ongoing sampling trajectory, after becoming visible in intermediate previews but before generation finishes. It introduces Direct-CFG, an interface that allows a human or vision-language model to request spatially and temporally bounded interventions for strengthening content, suppressing unwanted elements, or regenerating a region through local re-noising. A central contribution is the separation of intervention capability from the inspector’s ability to choose appropriate actions. Validation combines prescribed tool tests, held-out oracle-assisted correction, interactive human–agent workflows, and automated VLM inspection. The results support the feasibility of trajectory-level intervention, while revealing substantial limitations in preservation, visual quality, and reliable autonomous correction.

Critical strengths
  1. A well-defined interface with a useful separation of responsibilities. The paper clearly distinguishes error inspection, action selection, controller execution, and the generator’s response. Spatial and temporal constraints make interventions explicit and inspectable. This decomposition is valuable because unsuccessful correction need not imply an incorrect decision: it can also reflect limitations of the available tools or the underlying generator. The interface therefore provides a concrete foundation for studying correction policies.
  2. Controlled evidence beyond selected qualitative examples. The prescribed-tool and oracle-assisted studies include paired generation conditions, frozen configurations, and separate confirmation or held-out prompts. These experiments provide meaningful evidence that interventions can modify unfinished generations, rather than relying entirely on favorable interactive outcomes. Comparing alternative tools for the same requested edit also reveals consequential differences between strengthening content and regenerating a region.
Critical weaknesses
  1. The distinctive benefit of in-trajectory correction remains insufficiently established. The included comparisons do not yet demonstrate a compelling advantage over simpler alternatives: post-generation inpainting achieves comparable alignment improvements at lower reported cost, while resampling offers competitive alignment at the expense of replacing existing content. Although trajectory editing may offer preservation benefits, its overall advantage across correction quality, preservation, and computation remains unclear.
  2. Visual artifacts are not comprehensively evaluated in the main tool-confirmation study. Although removal-specific checks are included, general artifact assessment is absent from the primary success criterion. Consequently, edits that satisfy the semantic and preservation checks may still degrade visual quality within the edited region. This limits the evidence for consistently high-quality local correction.
  3. Evidence for reliable autonomous correction remains limited. Automatic control improves partial prompt alignment but does not establish a reliable net gain in complete-image success. Evaluation on previously seen prompts, the absence of paired preservation assessment, and the lack of independent human calibration further limit confidence in its reliability and generalization.
Questions for authors

Please refer to the weaknesses. I am willing to raise my score if the authors adequately address these concerns.

Reviewer_uo1nOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a method for correcting images midway through the diffusion trajectory. It introduces a set of tools that allow a VLM to inspect intermediate predictions and apply localized corrections during generation. These include CFG Push, which strengthens desired attributes or content within a selected region; CFG Suppress, which weakens unwanted content; and CFG Repaint, which re-noises a selected region and regenerates it. The VLM determines when, where, and which tools to apply, with the goal of correcting errors while preserving the rest of the image.

Critical strengths

The paper is well written, and I find the idea of correcting an image midway through the diffusion trajectory interesting. In particular, the ability to make substantial local changes while preserving the rest of the generated image could be a useful contribution.

Critical weaknesses

My main concern is that the experiments do not clearly establish the advantage of the proposed approach over existing methods. I would like to see the following comparisons:

  1. Local vs. global guidance: Does applying corrections locally provide an advantage over applying the same guidance globally? An ablation comparing localized and global guidance would help establish whether the spatial component of the proposed method contributes to correction success or image preservation.

  2. Comparison to existing methods: How does the proposed approach compare to existing methods for image editing and VLM-guided correction during diffusion? In particular, LEdits++ already performs localized semantic editing, while PPAD performs VLM-guided correction during generation. Although the proposed method differs from both, the paper does not clearly establish the benefits of these differences.

Questions for authors

I believe the proposed approach is interesting and may offer a useful contribution. However, I would like to see stronger evidence that its localized, mid-trajectory correction mechanism provides an advantage over the existing alternatives. Positive results from these comparisons would lead me to raise my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.