- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates two failure modes in self-reflective unified multimodal models for compositional image generation: failing to identify errors in generated images and failing to execute corrections after identifying those errors.
To address these issues, the authors propose RECAF, a training-free framework that employs external verification to assess generated images and scales the KV cache states associated with correction instructions to strengthen their influence on subsequent generation.
Experiments on GenEval, GenEval++, and GenEval2 demonstrate consistent performance improvements, with ablation studies supporting the effectiveness of both components.
Critical strengths
-
Clear problem analysis. The paper distinguishes between reflection failures and correction failures and provides empirical evidence for both. This offers a useful perspective on the limitations of unified multimodal models in iterative image generation.
-
Effective training-free approach. RECAF achieves substantial improvements across multiple compositional image generation benchmarks without updating model parameters. The ablation studies also demonstrate the complementary benefits of its two components.
Critical weaknesses
-
Limited methodological novelty. The two core components essentially combine external model-based verification with KV cache scaling, both of which build upon existing techniques. The distinction between reflection failures and correction failures is also relatively intuitive. Despite the promising experimental improvements, the paper does not sufficiently demonstrate that combining these components constitutes a substantial methodological contribution rather than an effective engineering integration. Furthermore, the lack of comparisons with simpler correction enhancement strategies weakens the justification for the proposed design.
-
Insufficient evaluation of generalization. The experiments primarily focus on compositional constraints involving object counts, colors, and spatial relationships, without evaluating more complex semantic reasoning tasks. More importantly, the fallback mechanism can rely on benchmark metadata to obtain the constraints to be verified, whereas real-world user prompts typically do not provide such structured information. Therefore, the applicability of the proposed method to open-ended generation scenarios remains unclear.
Questions for authors
-
What is the key technical novelty of RECAF compared with existing external verification and KV steering methods? Could the authors compare their approach with simpler alternatives, such as directly strengthening correction instructions or modifying attention logits, to justify the necessity of the proposed design?
-
How does RECAF automatically extract verifiable constraints from natural language prompts when benchmark metadata is unavailable? Can the method maintain comparable performance improvements under this setting?
-
Have the authors evaluated RECAF on WISE [1], T2I-ReasonBench [2], or other benchmarks involving complex semantic understanding and reasoning? Such evaluations would help determine whether the proposed method generalizes beyond relatively simple compositional constraint checking.
[1] WISE: World Knowledge-Informed Semantic Evaluation for Text-to-Image Generation
[2] T2I-ReasonBench: Benchmarking Reasoning-Informed Text-to-Image Generation
5 comments
Reply to Reviewer x8WA - Part 2
Q2:
To clarify, we do not use official benchmark evaluator components or outputs. We used the metadata to construct the requirement sets for the verifier. However, we found that such metadata is not necessary. The same requirement sets can be extracted directly from prompts using a rule-based parser, as in [1].
To demonstrate this, we employ a rule-based parser [1]. To evaluate the accuracy of the parser-extracted requirement set against a benchmark metadata-derived requirement set, we follow [2] and calculate the requirement-set exact-match accuracy. The results from all three datasets show that the parser-derived requirement set achieves 100% exact-match accuracy against the metadata requirements, removing the need for the metadata from benchmarks. We revised our paper accordingly, see App. B.2.
[1] Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D. Manning. Generating Semantically Precise Scene Graphs from Textual Descriptions for Improved Image Retrieval. In Proceedings of the Fourth Workshop on Vision and Language, Portugal. Association for Computational Linguistics. 2015
[2] Anderson, Peter, et al. Spice: Semantic propositional image caption evaluation. European conference on computer vision. 2016
W2&Q3:
We thank the reviewer for this suggestion. We would like to clarify that RECAF primarily targets reflection and correction failures in compositional image generation (as stated in the initial submission), rather than improving the underlying UMM's knowledge or reasoning capabilities. Benchmarks such as WISE and T2I-ReasonBench introduce a different challenge: generation prompts may require the model to infer appropriate visual content from implicit world knowledge or abstract semantic descriptions, rather than satisfy explicitly stated object, attribute, and spatial constraints.
Our original SAM-based verifier is designed for directly observable compositional constraints and is therefore not suitable for evaluating such knowledge-intensive requirements. However, the fallback mechanism itself is verifier-agnostic: its role is to identify violations that the UMM's intrinsic reflection misses, and the external verification component can be adapted to the task at hand.
To examine this adaptability, we conduct additional experiments on WISE and T2I-ReasonBench, replacing the original SAM-based verifier with a VLM-based semantic verifier, Qwen3.5 35B. We follow the official repo for evaluation. For correction amplification we use without any hyperparameter search. We evaluate WISE and T2I-ReasonBench across three different random seeds to assess the stability of our results.
The results in the following tables show that RECAF improves the overall WISE score from 71.2 to 73.1, and the overall T2I-ReasonBench score from 52.5 to 54.9. These results suggest that adapting the fallback to semantic verification can help identify and correct errors that the model's intrinsic reflection misses, even in knowledge-intensive generation settings. We have incorporated the results as a generalization to reasoning image generation, along with the experimental configuration in App. F of the revised version.
WISE Results
| Setting | Overall | Cultural | Time | Space | Biology | Physics | Chemistry |
|---|---|---|---|---|---|---|---|
| Uni-CoT | 71.2 ± 0.25 | 72.9 ±0.32 | 68.2 ±1.3 | 74.8 ±1.4 | 67.9 ±1.7 | 73.3 ±0.3 | 65.4 ±1.7 |
| RECAF | 73.1± 0.25 | 73.6± 0.55 | 67.7± 0.1 | 78.3± 0.7 | 71.8± 1.75 | 79.1± 1.53 | 68.0± 1.56 |
T2I-ReasonBench Results
| Setting | Overall | Idiom | Textual | Entity | Scientific |
|---|---|---|---|---|---|
| Uni-CoT | 52.5 ± 0.51 | 45.4±1.37 | 49.8±1.07 | 52.5±0.85 | 62.1±1.87 |
| RECAF | 54.9±1.05 | 48.1±0.94 | 52.8±0.23 | 54±1.87 | 64.6±1.16 |
Dear Reviewer, as the discussion period is coming to an end, we would appreciate hearing from you. Thank you!
Reply to Reviewer x8WA - Part 1
We thank the reviewer for their thoughtful and constructive feedback. We particularly appreciate the reviewer’s recognition of our work with “clear problem analysis and offers a useful perspective on the limitations of unified multimodal models in iterative image generation’’ and acknowledging RECAF as an “effective training-free approach’’ that “achieves substantial improvements across multiple compositional image generation benchmarks without updating model parameters’’. We address the reviewer’s concerns in detail below.
W1&Q1:
We would like to clarify that our key methodological contribution is not the individual use of external verification or KV-cache scaling, but a failure-aware correction framework that identifies and addresses two distinct bottlenecks in iterative generation.
Specifically, our contribution lies in empirically characterizing these failure modes and developing targeted interventions to address them. As shown in Secs. 3.1 and 3.3, successful error identification does not necessarily lead to successful correction, and correction failures can persist even when attention is concentrated on the target region. This reveals an important limitation of approaches that rely solely on improved error detection or spatial attention and remain underexplored.
Motivated by these observations, our framework addresses the two failure modes through complementary mechanisms: external verification provides explicit feedback on unmet requirements, while KV-cache scaling aims to strengthen the influence of corrective instructions during generation. The key insight is that improving error detection and improving correction execution are distinct objectives that require different interventions. Our method systematically connects these interventions.
We therefore emphasize that our contribution is a failure-driven methodological design, rather than a claim of novelty for its individual building blocks. The experimental improvements demonstrate the practical value of this design.
Comparison with simpler correction-enhancement strategies. To further justify our design, we conduct additional experiments with three simpler alternatives in GenEval and GenEval++:
Instruction emphasis: Strengthen the original correction instruction by adding the phrase "Pay attention to the instruction!" at the beginning.
Instruction repetition: Repeat the original correction instruction three times to increase its textual emphasis.
Attention-logit bias: Add a positive bias of ln(2) to the attention logits corresponding to correction-instruction tokens, rather than scaling their key and value representations.
Results on GenEval
Results on GenEval++
The results above show that RECAF consistently achieves higher overall accuracy than all three alternatives. The results suggest that simply emphasizing or repeating correction instructions, or increasing their attention logits, is insufficient to reproduce the improvements achieved by joint K/V amplification in the settings we evaluated. We have incorporated these comparisons into Sec. 4.3 and App. D. To demonstrate RECAF offers better performance under matched generation budgets, we have also conducted experiments to compare verifier-guided resampling and Best-of-N, see Reply to Reviewer WRRa W1&Q2.