- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces Direct-CFG, which adds vlm-based local corrections during the denoising stage of text-to-image diffusion models. Specifically, the VLM inspects the decoded image at chosen timesteps, and conducts a call to different kinds of CFG variants or renoising and repainting operations. They evaluate in a number of generation settings (insertions, attribute change, etc.) and also with human-in-the-loop style evaluation, and show that tools can make local edits during generation.
Critical strengths
-
Clear problem motivation: The problem of text-to-image model not being entirely faithful to the specified prompt due to local semantic errors is clear and well motivated.
-
Interesting solution of using tools during sampling
Critical weaknesses
-
Clarity and writing. The paper is not clear, and is hard to follow. In large parts, it reads like a project progress report rather than a research paper. A few examples: L237 “the originally planned empty-background control”, L265 "Confirmation lacks general artifact ratings but now includes removal patch checks", L417 “A pilot on an earlier stack” etc., with many more such examples. In addition, sentences are dense and overly complex and opaque, which makes reading it very difficult. For example, defining a concept by what it does not mean: "no op adds no intervention. Generation and already active actions continue, so it does not mean that the latent stops changing". In addition, the paper uses terms which are not defined yet, e.g., "GM" and "atoms" are both used in Section 5.1 and Table 1 before Section 5.4 defines them.
-
Lack of baselines. The authors do not compare to any prior work they cite (most notably PPAD or SLD), or to obvious baselines such as editing the resulting image with Qwen-Image-Edit. They only compare against post-hoc or inpainting with the same calls best-of-N seeds. It is thus very hard to understand if the proposed method helps compared to existing alternatives.
-
Problematic experiment design. The authors never show that in-trajectory editing beats the simple alternatives. In the one fair comparison (1B), just inpainting the final image or picking the best of a few samples does as well or better. The big gain in Experiment 2 seems to come mostly from rewriting the prompt, since the authors' own control gets the same improvement without any editing. And when everything is automatic (Experiment 3), there's no clear improvement.
-
Questionable motivation for box-based editing. Many potential errors cannot be fitted around a box, such as counts, spatial arrangements, etc. According to the authors' own experiment (Experiment 2), in 30% of the cases humans provided a bbox larger than 90% or more of the image. This suggests that the central motivation for the method is flawed in many cases.
-
Many hand-tuned settings and hidden controller logic. The method depends on a long list of hand-set parameters and hidden controller logic.
-
Very high computational cost. The method requires two extra model evaluations per step and each VLM inspection also pauses generation for 20–26 seconds, which causes a very large slowdown.
-
All results are VLM judgments with no human calibration.
Questions for authors
Can you compare against a dedicated instruction-based editor applied to the final image (e.g., Qwen-Image-Edit), and against PPAD, the closest prior work, on the same prompts?