Local Diffusion Control: A Tool Interface for VLM-Guided Correction

Text-to-image generators often produce visually plausible scenes but make local semantic errors, such as omitting an object, assigning an attribute to the wrong object, or introducing unwanted details. In this work, we investigate whether such errors can be corrected within the same sampling trajectory, after they become visible but before denoising is complete. We introduce Direct-CFG, a spatially and temporally bounded intervention interface that uses intermediate clean-image estimates to implement three operations: strengthening visible elements, suppressing unwanted content, and locally re-noising a region followed by its regeneration. We distinguish between two tasks: the effectiveness of the correction tools themselves and the ability of vision-language models (VLMs) to determine when, where, and how to apply them. We evaluated the approach using fixed human-specified interventions and automated VLM control, revealing the capabilities and limitations of local correction of generation trajectories.