acceptodds
Under review as a conference paper at ICLR 2027

Images Talk Back: Guiding MMDiTs by Suppressing Image-to-Text Feedback

11 views

Abstract

Multimodal diffusion transformers (MMDiTs) allow the evolving image to update internal text representations through joint attention. These representations then guide subsequent image updates, raising a question: does this feedback help correct deviations from the prompt, or preserve them? We find that image-to-text (I2T) feedback can preserve the objects and attributes already represented in the evolving image, including errors. Transplanting I2T messages can rescue failures or disrupt successes. At matched intervention strengths, suppressing the influence of I2T feedback improves generation, whereas opposite controls degrade it. These findings motivate Semantic Feedback Guidance (SFG), which suppresses the influence of I2T feedback by guiding away from an auxiliary branch that strengthens this feedback and weakens text-to-image conditioning. The in-place variant, iSFG, directly suppresses I2T feedback and strengthens text-to-image conditioning, requiring no extra model evaluations. Temporal interventions motivate running the SFG branch during the first 20% of image sampling steps, requiring about 1.2× the base NFE without classifier-free guidance (CFG). Experiments across three image-generation backbones show gains in alignment and preference-model scores. On FLUX, adding SFG to CFG raises GenEval from 0.6661 to 0.7479; on SD3.5M, iSFG adds +1.33 HPSv2.1 over CFG at zero extra NFE. The control also transfers to text-to-video and image-to-video generation, improving VBench scores.

Primary area:
generative models
Keywords:
multimodal diffusion transformers, image-to-text feedback, training-free guidance, text-to-image generation, text-to-video generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_bc8TOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper shows that image-to-text feedback in mmdits can reinforce generation errors, and proposes a cfg-like guidance method to suppress this feedback and strengthen text conditioning. The method is training-free, improves alignment across several t2i and t2v backbones.

Critical strengths
  1. The paper provides a clear mechanism of how I2T feedback in mmdit can preserve generation errors. The method is well motivated.
  2. SFG requires no model training, which makes the method easy to integrate into existing mmdit pipelines.
  3. The gains across multiple settings are relatively strong, and the generalization experiment on t2b is great
Critical weaknesses
  1. Limitations of older text encoders. The evaluation is limited to earlier mmdits such as sd3/3.5 and flux.1. Testing newer backbones such as flux.2 klein 9b would strengthen the generality claim, especially since modern llm-based text encoders like qwen3 in flux.2 klein may induce different text–image interaction patterns than older T5.
  2. The paper mainly evaluates text-to-image generation, while newer models such as flux.1 kontext, flux.2 klein unify generation and editing. It would be valuable to study whether the identified image-text interactions play a similar role in image-conditioned generation.
  3. Since SFG directly intervenes in internal attention pathways at inference, it would be useful to evaluate whether it introduces visual artifacts. Therefore, a dedicated artifact evaluation using a strong multimodal judge (e.g., Gemini 3.1 Pro) with artifact-focused prompts/criteria, and ideally a human preference study is welcome.
Questions for authors
  1. It seems different models favor different guidance strengths. Have you tried making the guidance strength learnable, or just distilling the effect of SFG into the model, for example via LoRA? This could substantially improve the practical usability of the method.
  2. How localized is the harmful i2t feedback? SFG currently scales the i2t message in all joint-attention blocks. Is the effect concentrated in particular attention heads / frequency areas?
4 comments

Response to Reviewer bc8T

Thank you for reading our paper carefully. We appreciate your comments on the experimental evidence for image-to-text (I2T) feedback and on how easily SFG can be integrated into existing MMDiT pipelines. Your questions prompted us to test SFG on a newer backbone, apply it to image editing, and directly evaluate visible artifacts. Following your suggestion, we also tried distilling SFG into a LoRA adapter.

W1. Does SFG transfer to a backbone with an LLM-based text encoder?

We agree that changing the text encoder may change how the image and text streams interact. Thank you for suggesting that we test FLUX.2 [klein] Base 9B, which uses a Qwen3 text encoder. We ran two 25-step text-to-image comparisons on this model, each with its corresponding classifier-free guidance (CFG) baseline. AM and GM denote the arithmetic and geometric means on GenEval2, respectively.

ComparisonMethodGenEval2 AMGenEval2 GMT2I-CompBenchHPSv2.1
Adding SFG to CFGCFG (scale 4)75.8231.1752.8030.36
Adding SFG to CFGCFG + SFG83.0743.4257.4131.62
Equal number of model evaluationsSFG alone83.6745.2457.5931.56
Larger model referenceFLUX.2-dev (32B)83.3442.0958.1732.05

On FLUX.2, adding SFG to CFG improved all four scores. GenEval2 AM rose by 7.25 points and GM by 12.25 points. For comparison, with SFG, the 9B model even came close to or exceeded the scores of the larger FLUX.2-dev (32B) from the same family.

In the paper, we found that the SFG correction direction was nearly orthogonal to the CFG correction direction. The FLUX.2 results show that SFG can further improve scores when combined with CFG and can also replace CFG when the inference budget is fixed. With two model evaluations per sampling step, SFG alone scored higher than CFG on all four metrics. Thank you for this insightful suggestion, which helped us see more of what SFG can do.

W2. Does SFG help with image editing?

Thank you for suggesting image editing as a test of whether this feedback mechanism generalizes to image-conditioned generation. We evaluated FLUX.2 [klein] Base 9B on all 606 editing instructions in GEdit-Bench-EN, using Qwen2.5-VL-72B as the judge under the VIEScore protocol.

MethodEdit completion ↑Semantic consistency ↑Perceptual quality ↑Overall score ↑
CFG (scale 4)8.0547.1027.4136.894
CFG + SFG8.5647.4897.2637.187

With SFG, the model followed the editing instructions more closely. The overall score rose by 0.293 points.

W3. Does SFG introduce visible artifacts?

Thank you for suggesting that we score artifacts directly to evaluate our method more fully. We used 500 prompts and generated a pair of images per prompt on FLUX.1. Within each pair, we used the same random seed to generate one image with CFG and the other with CFG + SFG. We shuffled the 1,000 images, hid the method labels, and used Claude Sonnet 5.5 to rate structural and rendering artifacts. We counted artifacts rated moderate or worse as visible.

Blinded evaluation of 500 image pairsCFGCFG + SFG
Images with visible artifacts74 / 50037 / 500
Visible artifact rate14.8%7.4%

Adding SFG halved the visible artifact rate in this evaluation. In 64 pairs, only the CFG image had visible artifacts. In 27 pairs, only the CFG + SFG image had visible artifacts. In another 10 pairs, both images had visible artifacts. These results show that SFG clearly reduced visible artifacts compared with the CFG baseline.

Q1. Can a LoRA adapter learn the SFG correction?

We agree with your suggestion. We ran the distillation experiment you suggested. On SD3-Medium, we trained a rank-16 LoRA adapter to match the SFG teacher’s guided velocity prediction in a single model evaluation. The adapter covers the attention projections in all joint-attention blocks. We trained it for 6,000 steps on COCO image–text pairs with a batch size of 16. The teacher controlled only I2T feedback (SFG-X), with ux=−0.40u_x=-0.40, us=0u_s=0, and a guidance scale of ω=8\omega=8.

SD3-Medium, 40 sampling steps, no CFGModel evaluations per stepGenEval
Base model132.95
SFG at inference (teacher)256.23
SFG distilled into LoRA157.41

The LoRA adapter achieved a better result than training-free guidance. It requires only one model evaluation per sampling step, with no auxiliary branch or guidance scale to tune at inference.

Q2. How localized is the harmful I2T feedback?

Thank you for this insightful suggestion. It helped us identify which part of the I2T feedback SFG acts on.

Source of the I2T perturbationGenEval2 AMGenEval2 GM
Full SFG83.6745.24
Low-frequency component of the values only80.1041.54

These results show that attenuating just the low-frequency component of image feedback can already give good results. This may suggest that the errors already present in the image are global and semantic, with high-frequency details playing little role. Suppressing this feedback may therefore prevent much of the harm these errors cause to the semantic information in the text stream.

Thank you again for the constructive feedback. We will incorporate these revisions into the final manuscript. We are glad to discuss and address any remaining question, and if you find your concerns resolved, we would appreciate if you could reflect this in your evaluation.

Thank you for your active response. I would like to raise my rating to 3.

Dear Reviewer bc8T,

Thank you for your follow-up and for your willingness to raise your rating. We will incorporate your suggestions into the final manuscript. Please let us know if you have any further suggestions before the discussion closes.

Thank you again for your time and effort.

Best regards,
The Authors of Submission 2377

Reviewer_oRTSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how image-to-text (I2T) feedback in multimodal diffusion transformers (MMDiTs) can preserve image content that conflicts with the prompt, and proposes Semantic Feedback Guidance (SFG) to control this feedback. SFG guides generation away from an auxiliary prediction obtained by strengthening I2T feedback and weakening text-to-image (T2I) conditioning. The paper also introduces iSFG, which directly scales the messages along both pathways without additional model evaluations. The authors use message transplantation between successful and failed trajectories to examine how I2T messages affect generation outcomes. At failed states, local response analysis shows that the velocity change induced by I2T suppression aligns more closely with successful-message transplantation than with a random control. Two-sided intervention experiments confirm the performance advantage of suppression under the tested settings, while temporal interventions support concentrating guidance early in sampling. Component ablations demonstrate the contribution of I2T control and the additional benefit of strengthening T2I conditioning. Experiments report improvements in alignment and preference scores across multiple image generation models, as well as higher aggregate quality scores in video generation.

Critical strengths
  • Targeted analysis of I2T feedback. The paper directly investigates how image-to-text feedback affects generation through bidirectional message transplantation, local response analysis, and temporal interventions. These experiments provide complementary evidence about the content carried by the feedback, the effect of suppressing it, and when intervention is most effective.

  • Empirical validation of the guidance design. Two-sided interventions test suppression against amplification at matched strengths, connecting the local analysis to final generation outcomes. Component ablations further establish the contribution of I2T control and the additional benefit of T2I conditioning enhancement.

  • Consistent improvements in image generation and text-to-video generation. SFG improves alignment and preference scores across multiple image generation models, including when combined with CFG. Experiments on text-to-video and image-to-video generation also report improved aggregate quality scores, demonstrating applicability beyond image generation.

  • Limited additional inference cost. SFG concentrates auxiliary evaluations in an early sampling window, reducing the overhead of guidance. The iSFG variant provides improvements without additional model evaluations and with a reported runtime overhead of approximately 2.9% in the measured configuration.

Critical weaknesses

Major Weaknesses

  • The motivation for T2I-weakening perturbation and its interaction with I2T amplification remain unclear. Section 2.2 primarily motivates I2T suppression. However, full SFG additionally weakens T2I conditioning in the auxiliary branch, while this design choice does not follow directly from the preceding analysis. The authors demonstrate an empirical benefit from adding this control, while leaving its justification and interaction with I2T amplification unclear. In particular, T2I weakening could reduce the transmission of both prompt information and image-derived information carried by the text stream, potentially counteracting the amplified feedback in the auxiliary branch. A clearer justification for combining these controls, together with an analysis that varies their multipliers independently (especially controlling their signs), would strengthen the connection between the proposed mechanism and the full method.
  • The conditions under which feedback suppression disrupts successful generation remain unclear. The full-schedule SFG-X diagnostic reports a 16.7% success-disruption rate. Although the paper acknowledges this effect, comparisons with other guidance methods on the same setting are needed to determine whether it is specific to feedback suppression or it is observed in general for the other method.
  • The video results leave a quality–motion trade-off unresolved. In image-to-video generation, SFG improves aggregate quality and reference consistency, but dynamic degree decreases from 85.37 for the base model to 70.44. Although this remains higher than CFG’s 66.67, the reduction warrants analysis of whether stronger conditioning limits desirable motion.

Minor Weaknesses

  • Ambiguous notation. The symbol tt denotes both the diffusion timestep and the text modality. The transition from Bi→tB_{i\to t} to Bt,ℓB_{t,\ell} in the transplantation analysis makes the message direction particularly difficult to track. Using distinct symbols for sampling steps and modalities, and retaining an explicit I2T label, would improve readability.
  • Limited evaluation and integration of iSFG. Although Table 3 reports improvements on two image models and Table 20 provides a runtime comparison, iSFG receives substantially less analysis than SFG. Its sensitivity to message-scaling strengths and intervention windows remains unclear, and its connection to the mechanism experiments is less developed. A more systematic comparison with SFG would clarify when direct message scaling is sufficient and when auxiliary-branch guidance provides additional benefits.
Questions for authors
  1. Justification for T2I weakening on auxiliary branch. Section 2.2 supports reducing I2T feedback. Why does SFG also weaken T2I conditioning in the auxiliary branch? Since the text stream contains both prompt information and information from the image, how do these two changes work together? What happens when their strengths are varied separately?

  2. Rescue and disruption under the main settings. What are the failure-rescue and success-disruption rates under the early-step setting used in the main experiments? How do these rates compare with other guidance methods on the same samples, particularly at similar overall performance?

  3. Video quality and motion. What explains the decrease in I2V dynamic degree relative to the base model? How does guidance strength affect the balance between reference consistency and motion? In addition to the reported quality metrics, could the authors provide a direct evaluation of text–video alignment?

  4. Evaluation of iSFG. How sensitive is iSFG to message-scaling strengths and the choice of sampling steps at which it is applied? Under comparable settings, when does iSFG match SFG, and when does the auxiliary branch provide a substantial improvement?

6 comments

Response to Reviewer oRTS

Thank you for reading our paper carefully, for your positive assessment, and for your constructive suggestions. We appreciate your recognition of our analysis of image-to-text (I2T) feedback, the design of our guidance method, and our experiments across different models. Your questions helped us explain the role of each message direction more clearly, analyze failure rescue and success disruption under the main configuration, and directly evaluate how well generated videos follow their prompts. We especially appreciate your suggestion on video evaluation. The new experiments show a more pronounced improvement in semantic alignment when SFG is combined with CFG.

W1 / Q1. Why strengthen I2T feedback and weaken T2I conditioning in the same auxiliary branch?

Thank you for pointing out where our explanation was unclear. We weaken text-to-image (T2I) conditioning in the auxiliary branch so that, when the final prediction extrapolates away from the auxiliary prediction, the text has a stronger influence on the generated image. The two adjustments serve the same purpose. They reduce the influence of the current image on the text stream and help the generated image follow the prompt more closely.

SFG uses the difference between the unmodified prediction and the auxiliary prediction for guidance. In the auxiliary branch, we let the image have more influence on the text and the text have less influence on the image. This makes the auxiliary prediction more likely to deviate from the prompt. As you point out, weakening T2I reduces the information passed from the text stream to the image. We do this in the auxiliary branch because the final guidance extrapolates away from that branch’s prediction. This helps the generated image follow the text more closely.

Prior work supports strengthening the image stream’s access to text information. For example, TACA found that cross-modal attention is suppressed in MMDiTs and showed that strengthening cross-modal (T2I) interaction can improve text–image alignment, attribute binding, and spatial relationships. Lv et al., Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformers, ICCV 2025

With this motivation, we added T2I control to the existing SFG auxiliary branch, so it requires no additional forward pass. The component ablations also show the additional benefit of T2I control. In the revision, we will explain separately the mechanistic evidence for suppressing I2T feedback, the motivation for weakening T2I conditioning in the auxiliary branch, and the experimental evidence for combining the two. We will also specify the message multiplier for each direction and make the sign of each adjustment clear.

We varied one message-scaling parameter at a time while keeping the other parameter and the guidance scale fixed. On SD3.5-Medium, with uS=0.20u_S=0.20 and ω=3.5\omega=3.5, HPS showed a roughly single-peaked trend as we varied uXu_X, with the highest score at uX=−0.25u_X=-0.25.

uXu_X0.00−0.10−0.15−0.20−0.25−0.30−0.35−0.40
HPSv2.132.3432.4132.4332.4932.5632.4832.4932.49

On FLUX.1, we varied only uXu_X, with uS=0.35u_S=0.35, ω=6\omega=6, and a CFG scale of 3.5. HPSv2.1 peaked at uX=−0.40u_X=-0.40.

uXu_X0.00−0.20−0.25−0.30−0.35−0.40−0.45
HPSv2.131.4331.5431.5531.5931.6431.6531.59

In a second FLUX.1 sweep, we fixed uX=−0.10u_X=-0.10 and the CFG scale at 3.5 and varied only uSu_S, with HPSv2.1 peaking at 32.19 for uS=0.25u_S=0.25.

uSu_S0.100.150.200.250.30
HPSv2.132.0532.0732.1132.1932.15

W2 / Q2. Failure rescue and success disruption under the main configuration

Thank you for raising this question. The diagnostic experiment in the appendix applies SFG-X throughout sampling, while the main experiments apply SFG only during the early sampling steps. The two settings therefore intervene differently. We also agree that different guidance methods should be compared on the same samples.

We used the official GenEval prompt generator to create 578 new prompts and generated two images per prompt, for a total of 1,156 images per configuration. We did not select prompts or random seeds based on whether the baseline succeeded or failed. For SD3.5-Medium and FLUX.1, we used the exact configurations from the main experiments. Each model used its own CFG results as the baseline, and the CFG output determined whether a sample initially succeeded or failed. We used the same prompts and random seeds for the paired comparisons across methods.

The failure-rescue rate is the fraction of samples on which CFG failed that become successes after guidance is added. The success-disruption rate is the fraction of samples on which CFG succeeded that become failures after guidance is added.

Model and guidance methodFailure-rescue rate ↑Success-disruption rate ↓Net change in the number of successful images
SD3.5-Medium, SFG20.5% (95/464)7.4% (51/692)+44
SD3.5-Medium, PAG20.7% (96/464)16.9% (117/692)−21
SD3.5-Medium, SEG14.7% (68/464)11.6% (80/692)−12
SD3.5-Medium, S²-Guidance8.4% (39/464)5.2% (36/692)+3
FLUX.1, SFG27.7% (129/465)9.7% (67/691)+62

For PAG, SEG, and S²-Guidance, we used the settings reported in the paper. On SD3.5-Medium, SFG and PAG rescued similar numbers of failures, 95 and 96, respectively. Among the initially successful samples, SFG disrupted 51 and PAG disrupted 117. S²-Guidance had the lowest success-disruption rate and also rescued fewer failures. On both models, SFG rescued more failures than it disrupted successes.

These results show that other guidance methods can also disrupt initially successful samples. One reason is that guidance changes the sampling trajectory even when it is already heading toward a correct result. Extrapolation may change the object layout, which object has a given attribute, or how a spatial relationship is depicted. It may also weaken visual context that was helping generation. Thus, even when a guidance direction improves results overall, it can still disrupt a sample that CFG would have generated correctly. One way to improve guidance is to preserve correct associations as much as possible once they have formed. Thank you for raising this insightful question.

W3 / Q3. The trade-off between video quality and motion, and direct evaluation of text–video alignment

Thank you for pointing out the decrease in Dynamic Degree and for suggesting a direct evaluation of text–video alignment.

One possible explanation is that stronger guidance for text–video alignment encourages the model to preserve the specified subject, its attributes, and its appearance in the reference image, while potentially limiting variation in motion. Free²Guide reports a similar result on VideoCrafter2. The mean text alignment score rose from 41.29 to 44.25, while Dynamic Degree fell from 38.46 to 29.49. Kim et al., Free²Guide, ICCV 2025, Tables 2 and 3

Dynamic Degree measures the proportion of videos whose motion exceeds an optical-flow threshold. When the amount of motion in a video is close to the threshold, even a small change can move it across the threshold and change whether it counts toward the score. The reward-weight experiments in NoisEasier also discuss the sensitivity of this metric to the threshold. Two configurations differed by 12.50 percentage points in Dynamic Degree and by only 0.02 percentage points in Motion Smoothness. Pu and Kong, NoisEasier, Appendix B, Table 10

How to maintain consistency with the reference image while preserving variation in motion deserves further study. We will explore different guidance strengths and intervention windows, as well as the effects of applying guidance to different frequency bands. In our current image-to-video (I2V) results, SFG scores higher on every reported dimension than CFG, which also strengthens conditioning.

We especially appreciate your suggestion to evaluate text–video alignment directly. We added SFG to CFG on HunyuanVideo-1.5 and used 523 official prompts covering six VBench semantic dimensions, generating three video samples per prompt. Both groups used a CFG scale of 6. The SFG parameters were uS=0.25u_S=0.25, uX=−0.25u_X=-0.25, and ω=16\omega=16. We applied SFG only to prompt tokens and only during the first 20% of denoising steps.

VBench semantic dimensionCFGCFG + SFGChange
Object Class90.1997.47+7.28
Multiple Objects44.2162.50+18.29
Human Action92.0092.000.00
Color87.5690.32+2.76
Spatial Relationship60.6267.64+7.03
Overall Consistency68.2269.81+1.59
Aggregate score across the six semantic dimensions73.8079.96+6.16

We normalized the scores following the VBench protocol and calculated the differences from the unrounded values. The aggregate score across the six semantic dimensions increased by 6.16 points. On the 260 prompts that were not used to select the configuration, the aggregate score rose from 72.13 to 78.98.

These experiments directly measure how well generated videos follow their prompts. The largest improvement was in generating multiple specified objects together. Generating objects of the specified classes and depicting the specified spatial relationships also improved. Thank you for suggesting this evaluation. It showed us that SFG can substantially improve direct text–video alignment in video generation.

Comments on the paper

9 comments

Response to Reviewer 2Vts

Thank you for reading our paper carefully and for your constructive suggestions. Following your suggestions, we separated hyperparameter selection from evaluation, measured the failure-rescue and success-disruption rates under the main configuration reported in the paper, and directly evaluated how well the generated videos follow their prompts. We especially appreciate your suggestion on video evaluation. It helped us see that combining SFG with CFG clearly improves semantic alignment between the videos and their prompts.

W1. Separating hyperparameter selection from evaluation

Thank you for suggesting a separate development set. We used the official GenEval prompt generator to create 300 development prompts with no overlap with the prompts used in the original evaluation. We generated two images per prompt and selected the hyperparameters using the GenEval evaluation protocol. We then froze the selected hyperparameters and evaluated SD3.5-Medium and FLUX.1 using the GenEval and HPSv2.1 protocols reported in the paper.

ModelGenEval (CFG)GenEval (CFG + SFG)HPSv2.1 (CFG)HPSv2.1 (CFG + SFG)
SD3.5-Medium0.68910.705231.6532.62
FLUX.10.66610.744130.9631.64

After hyperparameter selection on the separate development set, CFG + SFG still scored higher than CFG on GenEval for both models. The HPSv2.1 scores for CFG + SFG were also higher than those originally reported in the paper. The overall trends on both benchmarks were consistent with the original experiments.

W2. Failure rescue and success disruption under the main configuration

Thank you for pointing out the difference between the diagnostic setting in the appendix and the configuration used in the main experiments. We used the same GenEval prompt generator to create 578 new prompts and evaluated CFG + SFG using the exact hyperparameters reported in the paper. We generated two images per prompt, for a total of 1,156 images. We did not select prompts or random seeds based on whether the baseline succeeded or failed. On both SD3.5-Medium and FLUX.1, SFG was active during the first 20% of sampling steps. We compared each method with its corresponding CFG baseline using the same prompts and random seeds for each pair.

The failure-rescue rate is the fraction of samples on which CFG failed that become successes after guidance is added. The success-disruption rate is the fraction of samples on which CFG succeeded that become failures after guidance is added.

Model and guidance methodFailure-rescue rate ↑Success-disruption rate ↓Net change in the number of successful images
SD3.5-Medium, SFG20.5% (95/464)7.4% (51/692)+44
SD3.5-Medium, PAG20.7% (96/464)16.9% (117/692)−21
SD3.5-Medium, SEG14.7% (68/464)11.6% (80/692)−12
SD3.5-Medium, S²-Guidance8.4% (39/464)5.2% (36/692)+3
FLUX.1, SFG27.7% (129/465)9.7% (67/691)+62

For PAG, SEG, and S²-Guidance, we used the settings reported in the paper. The table below shows the SFG results on SD3.5-Medium by prompt category.

Prompt categoryFailure-rescue rate ↑Success-disruption rate ↓
Two-object presence42.3%1.9%
Counting36.8%6.7%
Attribute binding25.2%17.1%
Single-object color22.7%6.1%
Spatial relations7.7%15.6%

On both models, SFG rescued more failures than it disrupted successes. It also had the largest net gain among the guidance methods compared, measured as the number of rescues minus the number of disruptions. The breakdown suggests that, on tasks such as spatial relations, suppressing feedback may interfere with useful visual context. Overall, however, the benefits were more common across categories and more pronounced.

As for when guidance disrupts useful feedback, the examples from SFG and the other methods suggest that this may be a general effect of guidance. We think one reason is that guidance changes the sampling trajectory even when it is already heading toward a correct result. Extrapolation may change the object layout, which object has a given attribute, or how a spatial relationship is depicted. It may also weaken visual context that was helping generation. Thus, even when a guidance direction improves results overall, it can still disrupt a sample that CFG would have generated correctly. We will also explore how to control SFG more precisely so that it has less effect on successful paths.

W3. Direct evaluation of how well generated videos follow their prompts

Thank you for suggesting that we directly evaluate how well generated videos follow their prompts. We added SFG to CFG on HunyuanVideo-1.5 and evaluated six VBench semantic dimensions using 523 official prompts, with three samples per prompt. Both groups used a CFG scale of 6. We set uS=0.25u_S=0.25, uX=−0.25u_X=-0.25, and ω=16\omega=16 and applied SFG only during the first 20% of denoising steps.

VBench semantic dimensionCFGCFG + SFGChange
Object Class90.1997.47+7.28
Multiple Objects44.2162.50+18.29
Human Action92.0092.000.00
Color87.5690.32+2.76
Spatial Relationship60.6267.64+7.03
Overall Consistency68.2269.81+1.59
Aggregate score across the six semantic dimensions73.8079.96+6.16

We normalized the scores to a 0–100 scale following the VBench protocol. The aggregate score across the six dimensions increased by 6.16 points. On the 260 prompts that were not used to select the configuration, the aggregate score rose from 72.13 to 78.98. Thank you for suggesting this evaluation, which provides stronger evidence for the effectiveness of SFG. The results directly show that SFG helps generated videos follow their prompts more closely, especially when a prompt requires several specified objects to appear together.

Thank you again for the constructive feedback. We will incorporate these revisions into the final manuscript. We are glad to discuss and address any remaining question, and if you find your concerns resolved, we would appreciate if you could reflect this in your evaluation.

This is a gentle and friendly follow-up to our earlier response. Thank you again for suggesting that we report paired prompt-level confidence intervals. We apologize for omitting this result from our initial response. You are right that these intervals are especially important for smaller gains, particularly the gain on GenEval with SD3.5-Medium.

Following your suggested evaluation protocol, we selected the hyperparameters for SD3.5-Medium on 300 separate development prompts, froze them, and evaluated CFG and CFG + SFG on GenEval prompts that were not used for tuning. The table reports the observed gains over CFG on GenEval and HPSv2.1, together with their paired prompt-level 95% confidence intervals.

SD3.5-Medium metricGain over CFG (pp)95% CI for the gain (pp)
GenEval+1.61[+0.047, +3.118]
HPSv2.1+0.97[+0.68, +1.31]

To further test whether these gains are reliable, we evaluated FLUX.2 [klein] Base 9B, which uses a Qwen3 text encoder. We ran two 25-step text-to-image comparisons. AM and GM denote the arithmetic and geometric means on GenEval2, respectively. “SFG alone” means that we used SFG without CFG. We adapted the other guidance methods to FLUX.2 and gave each method the same hyperparameter tuning budget.

MethodGenEval2 AM ↑GenEval2 GM ↑HPSv2.1 ↑
CFG75.8231.1730.36
CFG + PAG75.7230.8230.82
CFG + SEG75.5631.4830.66
CFG + S²-Guidance75.7330.9930.39
CFG + SFG83.0743.4231.62
SFG alone83.6745.2431.56
FLUX.2-dev (32B reference)83.3442.0932.05

These experiments further tested the reliability of the gains across models, and we also observed improvements on FLUX.2 [klein] Base 9B. Adding SFG to CFG improved GenEval2 AM by 7.25 points and GM by 12.25 points, and yielded the highest HPSv2.1 score among the 9B configurations we tested. With SFG alone, the 9B model can even perform comparably to the 32B model from the same family.

We hope this follow-up and our earlier response address all of your concerns. We will also incorporate this important result into the final manuscript. Thank you again for your suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.