acceptodds
Under review as a conference paper at ICLR 2027

Understanding and Enhancing Conditioning Robustness of Text to Image Diffusion Models

Abstract

Recent advances in diffusion models have significantly improved the quality of text-to-image generation. Yet visual fidelity alone is not sufficient: reliable conditioning is essential for predictable control, and its limitations continue to constrain the broader success of these models. In particular, the sensitivity of generated images to non-semantic perturbations of the input prompt is a widely recognized problem. In this work, we develop a simple perturbative model of the mechanism underlying this instability and validate it through targeted experiments. We further introduce dedicated metrics and a benchmark (CoRBen) for systematically evaluating prompt robustness. The resulting theory and empirical findings motivate the sensitivity measure and a family of training-free plug-and-play methods that improve generation stability at inference overheads of 100%, 17%, and 3%. Our code is available at https://anonymous.4open.science/r/conditioning-robustness/.

Primary area:
generative models
Keywords:
text-to-image generation, diffusion models, prompt robustness, paraphrase invariance, training-free inference, variance reduction, classifier-free guidance, flow matching

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_T6vrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the sensitivity of text-to-image diffusion models to semantically equivalent prompt variations. It introduces a conditioning-sensitivity statistic and the CoRBen benchmark, then proposes averaging network predictions across paraphrases to stabilize generation. Early-step averaging and SVRG/SAGA-based estimators reduce the additional network evaluations. Experiments on SD1.5 and FLUX assess image consistency, while full latent averaging is also evaluated through object recall and distributional image-quality metrics.

Critical strengths
  1. The paper addresses a meaningful problem with a well-motivated approach. The early-step sensitivity analysis suggests where additional computation is most useful. The conditional-unbiasedness arguments support the proposed estimators, and the residual-cache formulation provides a sensible adaptation to time-dependent diffusion predictions.

  2. The benchmark construction includes useful methodological improvements. Prompt-derived concept targets are more appropriate than treating every source-image object as required content. The FSP subset directly targets meaning-preserving perturbations, and full LA3 produces a positive object-recall improvement on this subset.

Critical weaknesses
  1. The semantic benefits of the lowest-cost variants remain unverified. Their primary evaluation uses LPIPS and DINO consistency on 100 prompt groups, whereas the object-recall and distributional results concern full LA3. More similar outputs do not necessarily preserve attributes, relations, or appropriate prompt sensitivity. Network-evaluation counts also omit paraphrase-generation and text-encoding costs.

  2. Some benchmark results are difficult to interpret as semantic robustness. Alternative captions and several legacy corruptions can change the requested content while retaining the clean anchor’s targets. Consequently, improved recall may reflect recovery of anchor content. The more relevant FSP subset needs a clearer semantic audit and an explicit description of evaluation-time auxiliary-prompt construction.

Questions for authors
  1. Do the lowest-cost SVRG and SAGA variants improve semantic correctness and image quality on the existing FSP subset?

  2. How are auxiliary prompts constructed, can they access the clean anchor, and what is the measured end-to-end latency?

2 comments

We thank the reviewer for the time invested in assessing our work and their careful evaluation. We address their questions point by point, and then the remaining benchmark concern.

Q1. Both lowest-cost variants improve semantic correctness. We added SVRG35SVRG_{3}^{5} and SAGA3∗5SAGA_{3}^{*5} to the full CoRBen FSP evaluation (Appendix B.2, pp. 17-18, Table 5). For convenience, all new results in the revised manuscript are highlighted in blue. At GLIP threshold 0.5, instance recall increases from 87.41% to 87.82% and 87.88%, respectively. On independently selected unstable concepts, OR0.5∗∗\mathrm{OR}_{0.5}^{\ast\ast} increases from 73.27% to 74.30% and 73.88%.

We also evaluated every efficient variant on all 2,400 T2I-CompBench++ prompts (Appendix B.1, pp. 16-17, Tables 3-4). Every scheme improves all eight category scores and all five atomic recall metrics, covering attributes, spatial and non-spatial relations, counting, and complex compositions. Thus, improved consistency accompanies better realization of requested content.

Q2. The user supplies one prompt. Auxiliary formulations are generated automatically by a few-shot LLM instructed to preserve visual semantics exactly. Exact instructions and settings appear in Appendices G (p. 30), H.4 (pp. 32-33), and J (pp. 37-43). The timed FSP pipeline uses Qwen3-30B-A3B to generate nine candidates and select two.

Appendix B.5 (pp. 19-20) now measures encoding, denoising, and VAE decoding, plus rewriting separately. On A100, baseline generation takes 0.98 s, SVRG 1.00 s, and SAGA 1.01 s. Rewriting adds 0.079 s per prompt at batch size 256, corresponding to approximately 1.08-1.09 s of combined amortized work for one image per prompt. For a detailed discussion, see Appendix B.5 (pp. 19-20). Peak generation memory remains 2.61 GB since VAE decoding determines it.

The answers above address W1. Regarding W2, legacy captions and corruptions can indeed alter semantics. We used them to test the proposed averaging mechanism. The newly introduced FSP represent the practically relevant setting of semantics-preserving reformulations.

We added a quantified FSP audit with Claude Opus 5.5, a higher-capacity judge from a different family than the Qwen generator (Appendix H.4, pp. 32-33, Table 19). It evaluates 849 FSP pairs, 400 alternative-caption pairs, and 50 negative controls from 100 anchors using Listing 5's instruction. Exact preservation is 84.8% [82.1, 87.4] for FSP versus 1.0% [0.2, 2.0] for alternative captions. Approximate preservation is 99.1% [98.3, 99.8] versus 71.5% [64.8, 77.3]. This quantifies both FSP's substantially greater semantic fidelity and its remaining errors. Our expanded comparisons focus on FSP and explicit attribute/relation tests.

Additional evaluations motivated by other reviewers

  • Prompt averaging (PA) baseline: qualitative examples, geometry, T2I-CompBench++, and ImageReward show that PA fails to reproduce LA's gains (Appendix C, pp. 22-25).
  • Ablation of NN involving FSP: OR improves through N=4N=4 on 1,000 FSP prompts (Appendix B.4, p. 19).
  • Statistical association between perturbation norm and concept forgetting: weighted output deviations predict semantic failures within prompt groups with AUC 0.72 [0.65, 0.79] (Appendix B.6, pp. 20-21).

We appreciate the reviewer's attention and valuable suggestions. The revisions have substantially strengthened the evaluation of the proposed schemes and addressed all concerns raised. We hope these improvements provide a basis for reconsidering the score.

Dear Reviewer T6vr,

We thank the reviewer for their thoughtful feedback. We kindly invite them to revisit our rebuttal and revised manuscript. We would welcome their updated assessment and any remaining questions.

Reviewer_PtKaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work analyzes how sensitive the generative diffusion process is to non-semantic perturbations within input prompts. Authors create and analyze a perturbative model, suggesting that, during generation, the non-semantic perturbation term could be averaged over denoising predictions for N prompt rewritings to reduce its variance. They call this method Latent Averaging (LA) and show on their Conditioning Robustness Benchmark that averaging conditional prediction within a classifier-free guidance mechanism can significantly boost diffusion model robustness to alternative prompts. However, as LA incurs significant sampling overhead, the authors propose improvements that maintain good generation consistency while reducing computational cost: using LA only during the first denoising steps and adapting unbiased variance-reduced estimators: SVRG and SAGA.

Critical strengths

I appreciate the theoretical analysis of this work. The authors theoretically ground both their Latent Averaging method and their use of adapted unbiased estimators (SVRG, SAGA). The introduced perturbative sensitivity model is also reasonable. Also, the provided descriptions of the methods are sufficient and readable.

Critical weaknesses

W1. Limitations

There are limitations in the introduced methods that are not disclosed in the paper:

  1. Concerning method setup: LA, SVRG, and SAGA* require from the user, next to the provided prompt, its N−1N-1 paraphrases, which feels like an uncommon scenario. If not provided by the user, potentially, an external LLM could rephrase the prompt, incurring additional computational cost. If access to an LLM is assumed, then your method should be compared against a simple baseline in which the LLM proposes a clear form of the user's instruction, which Diffusion Model inputs.
  2. The authors do not include the Limitations section in their work, where they should mention that:
    1. Latent Averaging adds memory overhead, with accompanying analysis of this overhead.
    2. For guidance-distilled flow matching models, such as FLUX, SAGA does not offer an advantage over SVRG - it performs worse at a bigger computational overhead.

W2. Critical leakage in Fig. 2

I suspect leakage in the experiment shown in Figure 2. To make runs comparable, according to Section 4.1, distances for LA methods need to be averaged over 21 pairs of pp rewrites (pp itself + 6 reformulations). In the case of LA3LA_3, obtaining 7 disjoint 3-prompt sets requires a global pool of 2121 rewrites.

To my understanding, this indicates that rewrite sets in pairs intersect. Hence, better performance of LA methods can be attributed to the intersections of the variant sets that LA averages over. Two runs already share the same seed, so with the same prompts, they produce the same predictions at the start of the denoising process.

Questions for authors

Q1. Could you clarify the setup of your method and compare it with the LLM prompt-rewriting baseline, addressing W1?

Q2. Could you clarify how the experiment in Figure 2 is conducted for the Latent Averaging method, addressing W2?

Q3. I would suggest improving some parts of the manuscript to improve the paper's readiness, in particular:

  • Introduction references the methods (LA, SVRG, SAGA) without providing high-level explanations. I think including them would clarify that section.
  • There are only two image examples in the paper (Figure 1). Could you add a qualitative comparison of: baseline, LA, SVRG, and SAGA?
  • Results in Table 1 and Figure 5 lack statistical significance.

Q4. Your choice of the number of rephrases (N=3N=3) seems to be ad hoc. An experiment that compares methods' downstream gains (OR0.5∗OR_{0.5}^{*}) against NN would clarify role of this hyperparameter.

5 comments

We appreciate the time the reviewer devoted to the paper and their thoughtful evaluation. We address the questions in order, followed by the remaining weaknesses.

Q1. The user supplies one prompt. We obtain its variations automatically with fixed few-shot LLM instructions, reproduced in Appendices G (p. 30), H.4 (pp. 32-33), and J (pp. 37-43). Appendix B.5 (pp. 19-20) now measures rewriting cost explicitly.

The difficulty with canonical rewriting is deciding which correct, semantically equivalent formulation should be canonical. Correcting grammatical errors, omissions, and typos can clearly improve an incorrectly written prompt. Between two correct, semantically equivalent formulations, however, linguistic correctness provides no criterion for selecting the one that the generator will follow more reliably. This baseline is therefore not well-defined for our setting. We also investigated text embedding averaging and report the results in Appendix C (pp. 22-25): qualitative examples (Figure 10, p. 22), T2I-CompBench++ (Table 13, p. 24), ImageReward (Table 14, p. 25), and embedding/prediction geometry (Tables 10-12, pp. 23-24). Prompt averaging underperforms the baseline on 12 of 13 benchmark metrics and on ImageReward, whereas LA improves all reported metrics.

Q2. We redesigned the protocol to exclude overlap between averaging runs. For each of 100 groups, 14 formulations now supply four mutually disjoint triples, using 12 formulations. Each LA run uses one triple. The baseline uses four prompts, one from each triple. Each method produces four images and six pairwise comparisons per group and guidance scale, sharing initial noise. Sections 4.1-4.2 (pp. 4-5) and Appendix G.1 (p. 30) now specify this design for Figures 2 and 5. We replaced both figures with results obtained using these disjoint triples. Shared formulations between averaging runs therefore cannot mechanically explain their agreement.

Q3. We made the requested additions:

  • The Introduction (p. 2) now briefly explains LA's prediction averaging, SVRG's fixed-snapshot correction, and SAGA's updated residual cache.
  • Appendix I (p. 36) adds qualitative comparisons of the baseline, LA, SVRG, and SAGA. Appendix C (pp. 22-25) separately illustrates why embedding averaging fails.
  • Figures 2 and 5 now specify 95% confidence intervals. The revision additionally reports uncertainties for all methods in the full FSP comparison (Appendix B.2, Table 5, p. 18) and distributional measurements (Appendix B.3, Table 6, p. 18).

Q4. We added the requested downstream ablation on 1,000 FSP prompts, varying only NN (Appendix B.4, p. 19). OR$^{**}_{0.5}$ uses independent seeds for target selection and evaluation.

MethodOR$_{0.5}$OR$^{**}_{0.5}$
SD1.586.72 ± 0.7672.63 ± 1.37
LA$_2$87.27 ± 0.7273.89 ± 1.38
LA$_3$87.58 ± 0.7174.40 ± 1.39
LA$_4$87.92 ± 0.7375.33 ± 1.44

Both estimates increase monotonically without observed saturation by N=4N=4. The new Limitations and Future Work section explicitly identifies the joint quality-cost tradeoff in NN and KK as unresolved.

The setup and overlap concerns are addressed above. For the remaining parts of W1:

  • Appendix B.5 (pp. 19-20) measures encoding, denoising, and VAE decoding on RTX 2080 Ti and A100, plus Qwen rewriting separately. SVRG/SAGA add approximately 2% to generation time. Peak denoising memory rises from 2.14 to 2.27 GB, while the full generation peak remains 2.61 GB. Rewriting adds 0.079 s per prompt at batch size 256 in amortized throughput.
  • We added a Limitations and Future Work section (p. 10) covering resource costs beyond NFE, further evaluation of variance-reduction schemes and rewriting LLMs and their instructions, the unresolved quality-cost tradeoff in NN and KK, integration with existing prompt-upsampling pipelines, and robustness distillation.

Additional evaluations motivated by other reviewers

  • Semantic correctness: all suggested schemes improve every category score and atomic recall metric on 2,400 T2I-CompBench++ prompts, including attributes, relations, and counting (Appendix B.1, pp. 16-17).
  • Full FSP comparison: extends OR, OR$^*$ to every efficient variant (Appendix B.2, pp. 17-18).
  • Semantic audit: a different-family LLM judge finds 84.8% exact and 99.1% approximate FSP preservation, versus 1.0% and 71.5% for legacy alternative captions (Appendix H.4, pp. 32-33).
  • Statistical association between perturbation norm and concept forgetting: Appendix B.6 (pp. 20-21).
  • Prompt averaging baseline: qualitative examples, embedding/prediction geometry, T2I-CompBench++, and ImageReward distinguish prompt averaging from LA (Appendix C, pp. 22-25).

We are grateful for the reviewer's engagement in the review process and their attentive reading of the paper. We hope that the revisions and responses have addressed all concerns raised. We believe the expanded evidence substantially strengthens the paper and respectfully request a score reconsideration.

Dear Reviewer PtKa,

We appreciate the reviewer’s time and constructive comments. We would be grateful for their assessment of our response and revised manuscript and are happy to provide any further clarification.

Comments on the paper

5 comments

We thank the reviewer for the time devoted to our work and their careful evaluation. We address the identified weaknesses point by point below.

W1. There is no reason to classify a component shared by all semantically equivalent formulations as non-semantic. Such a component belongs to the common semantic field and is invariant to reformulation, so it does not contribute to the formulation sensitivity studied in this paper. We agree that the N\sqrt{N} reduction arises from ensembling and does not establish independence. However, reducing the response to non-semantic prompt perturbations is precisely the desired effect.

To further address this concern, we added the following experiments to the revised manuscript. For convenience, new material is highlighted in blue:

  • Appendix B.7 (pp. 21-22) examines 8,529 formulation pairs. It finds weak dependence among highly similar formulations and almost no dependence among the others.
  • Appendix B.6 (pp. 20-21) establishes a statistical association between the centered residual norm and concept forgetting, with mean within-group ROC AUC 0.72 [0.65, 0.79].

W2. We had already investigated prompt averaging in our initial experiments and now include its results and discussion in Appendix C (pp. 22-25) of revisiting. Figure 10 (p. 22) shows that it can introduce severe visual artifacts. A comparison based on T2I-CompBench++, covering objects, attributes, relations, and counting (Table 13, p. 24), shows that PA$_3$ underperforms the baseline. PA$_3$ also lowers ImageReward (Table 14, p. 25), whereas LA$_3$ improves every reported metric over the baseline. Finally, we examine the geometry of text embeddings and noise predictions (Tables 10-12, pp. 23-24), offering an explanation for these results. Since this evaluation comprises several experiments, we refer to Appendix C rather than duplicate them here.

For canonical prompt rewriting, the key difficulty is deciding which formulation should be considered canonical. Correcting grammatical errors, omissions, and typos can clearly improve an incorrectly written prompt. Between two correct, semantically equivalent formulations, however, linguistic correctness provides no criterion for selecting the one that the generator will follow more reliably. This baseline is therefore not well-defined for our setting.

W3. The revision adds Appendix B.1 (pp. 16-17), evaluating all 2,400 T2I-CompBench++ test prompts across color, shape, texture, spatial and 3D spatial relations, non-spatial relations, numeracy, and complex compositions, together with object, count, attribute, and relation recall. Notably, a consistent advantage over the baseline is achieved by both full LA$_3$ and its scaled variants LA$_3^5$, SVRG$_3^5$, and SAGA$_3^{*5}$. For convenience, we reproduce method-minus-baseline differences from Table 4 (p. 17) below. Absolute scores appear in Table 3 (p. 17).

MetricLA$_3$LA$_3^5$SVRG$_3^5$SAGA$_3^{*5}$
Color+0.069 ± 0.019+0.070 ± 0.018+0.073 ± 0.019+0.060 ± 0.019
Shape+0.064 ± 0.014+0.053 ± 0.014+0.072 ± 0.014+0.056 ± 0.014
Texture+0.023 ± 0.017+0.020 ± 0.016+0.017 ± 0.016+0.019 ± 0.015
Spatial+0.016 ± 0.015+0.013 ± 0.015+0.018 ± 0.014+0.018 ± 0.016
3D spatial+0.020 ± 0.010+0.006 ± 0.010+0.005 ± 0.009+0.015 ± 0.010
Numeracy+0.047 ± 0.014+0.020 ± 0.013+0.024 ± 0.013+0.018 ± 0.014
Non-spatial+0.004 ± 0.001+0.003 ± 0.001+0.004 ± 0.001+0.003 ± 0.001
Complex+0.010 ± 0.006+0.010 ± 0.006+0.011 ± 0.006+0.009 ± 0.006
Object recall+3.4 ± 1.6+1.5 ± 1.5+3.0 ± 1.6+2.6 ± 1.6
Count recall+3.3 ± 1.7+1.8 ± 1.7+2.0 ± 1.6+1.0 ± 1.7
Attribute recall+3.4 ± 0.7+2.7 ± 0.7+3.3 ± 0.8+3.0 ± 0.7
Relation recall+2.2 ± 1.2+1.0 ± 1.1+1.5 ± 1.1+2.5 ± 1.1
Attribute + relation+3.2 ± 0.6+2.4 ± 0.6+3.0 ± 0.7+2.9 ± 0.6

W4. We added a quantified semantic audit in Appendix H.4 (pp. 32-33), large-scale evaluation of the efficient variants and independent-seed target selection in Appendix B.2 (pp. 17-18).

  • Semantic audit: Claude Opus 5.5 evaluates 849 FSP pairs from 100 anchors, 400 legacy alternative-caption pairs, and 50 negative controls (Table 19, p. 33). FSP preserves semantics exactly in 84.8% [82.1, 87.4] and approximately in 99.1% [98.3, 99.8], versus 1.0% and 71.5% for alternative captions. The judge has higher capacity and belongs to a different family than the Qwen generator. Its complete instruction appears in Listing 5 (p. 43).
  • Full FSP evaluation: the revised comparison covers all efficient methods (Table 5, p. 18). OR$^{**}_{0.5}$ selects unstable concepts using one baseline seed and evaluates the frozen subset using another.
MethodOR$_{0.5}$OR$^*_{0.5}$OR$^{**}_{0.5}$
SD1.587.41 ± 0.3064.70 ± 0.6473.27 ± 0.55
LA$_3$88.24 ± 0.3068.82 ± 0.7474.94 ± 0.60
LA$_3^5$87.92 ± 0.3068.07 ± 0.7174.32 ± 0.57
SVRG$_3^5$87.82 ± 0.3067.88 ± 0.7174.30 ± 0.57
SAGA$_3^{*5}$87.88 ± 0.2968.17 ± 0.7073.88 ± 0.57

All variants retain positive recall gains. The scaled methods give up approximately 39%, 51%, and 43% of the full OR gain, respectively, while reducing its 100% NFE overhead to 16.7%, 3.3%, and 3.3%. The independent-seed LA gain is 1.67 points, larger than ordinary OR's 0.83 but smaller than same-seed OR$^*$'s 4.12. This can be explained by reformulation-induced failure cases varying across generation seeds.

Additional evaluations motivated by other reviewers

  • Resource measurements: wall-clock time and memory on RTX 2080 Ti and A100, together with LLM prompt-rewriting throughput over six batch sizes (Appendix B.5, pp. 19-20).
  • Ablation of NN involving FSP: on 1,000 prompts, both OR and independent-seed OR$^{**}$ increase from N=2N=2 to 4, without observed saturation (Appendix B.4, p. 19).

We appreciate the reviewer's attention and valuable comments. The revision and responses now address every point raised. We believe this substantially strengthens the evidence for the contribution and respectfully ask the reviewer to increase their score in light of these additions.

Dear Reviewer zW7f,

We thank the reviewer again for their careful evaluation. We would appreciate their feedback on our rebuttal and revised manuscript and would welcome clarification of any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.