- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work addresses whether looped computation, i.e., repeated inference through the middle blocks of the diffusion transformer, can be considered a promising approach for scaling text-to-image models. The authors present that the naive application of looping does not offer consistent improvements. They trace the issue to (1) insufficient supervision during intermediate loops and (2) redundant token updates within looping attention layers, and offer fixes through, respectively, deeply-supervised training loss and self-modulating attention mechanism. The resulting method is compared with prior models across multiple text-to-image benchmarks, demonstrating improvements, also when compared with a non-looped model at matched inference compute.
Critical strengths
-
I consider the main motivation for this work, which is enabling looped computation with image generative models, to have high potential and be a promising direction for the future of visual generation. The proposed improvements over naive looping (deep supervision and self-modulating attention) are well-motivated (with probing and attention update studies), clearly explained in the work, and empirically shown to improve looped diffusion transformers.
-
The authors include a broad comparison, evaluating against multiple prior models on up to 6 alignment benchmarks, analyzing both performance and inference speed.
Critical weaknesses
-
While I consider the model's looped refinement to be a promising phenomenon, I would ask the authors to work on some phrasings in their work to better explain what they mean by "reasoning capability". All qualitative image examples in the paper (Fig. 1, Fig. 5, Fig. 9) refer rather to prompt-following capabilities: visual text adherence (F1,F5), object counting (F9), or object spatial positioning (F5). Also, while I appreciate using multiple benchmarks for evaluation, almost all of them measure prompt-following, often in the form of spatial object composition and counting, in which the prompt explicitly specifies how the image should look. The evaluation that seems particularly interesting here is the reasoning splits from T2I-CoreBench, which includes, i.a., solving puzzles or inferring outcomes from sequences of actions. The Authors use some of those splits in Table 3 (but not all), and their method does not achieve as great gains as the CoT baseline in most cases. Could the authors separately compare against the evaluated models (Table 1), perform ablations (Table 4), and provide more qualitative examples on those splits?
-
I have concerns with some experimental results reported in the paper:
2.1. I would ask the authors to include statistical significance (standard error or std) in their results, particularly in Tables 2 and 4. In Table 2, the difference between their method and the wider MiniT2I, at the same inference FLOP count and higher training FLOPs, is relatively small (0.5) and may not be statistically significant. This is important, given that the authors present their method as an alternative for computational scaling.
2.2. I believe a broader analysis of looped generation is not thoroughly evaluated in this work. Especially, I am not sure whether this approach could work under a choice of different layers that are looped (the current choice seems to be rather ad hoc), and whether the proposed approach can be transferred to diffusion models with different architectures, raising generalization concerns
Questions for authors
-
I appreciate the analysis of the two methods for reducing redundancy of attention updates in Section 2.2. However, I am not sure what role the Gated Attention mechanism plays in the paper, as it does not outperform the parameter-free XSA on any benchmark (Table 4).
-
In Appendix A.1, the authors claim to use the short-prompt split of TIIF, as their "small-scale training setup primarily uses short captions with a maximum length of 256 tokens, making the long-prompt setting less representative of the training regime". Considering this, I'm concerned about the comparison with the CoT-based prompt-rewriting baseline in Table 3. The limited average gain from CoT can stem from Qwen3 rewriting benchmark prompts, pushing their length outside the model's training token-length distribution, making the comparison unfair. I'd ask the authors to report the token-length distributions for: (1) MiniT2I training datasets, (2) original benchmark prompts, and (3) CoT-rewritten prompts.
-
The adaptive looping mechanism seems to be a promising strength of this work. In my opinion, the authors should include more details on this mechanism in the main part of the paper.
3 comments
I thank the authors for their work during the rebuttal.
I believe that your experiments sufficiently answer some of my concerns: statistical significance (2.1), generalization to other architectures (partly 2.2), and my questions (Q1, Q2, Q3). Especially, new experiments on the MoT architecture and image-conditioned generation on VBVR, along with image examples, are highly appreciated.
However, some of my concerns remain unaddressed:
- There are neither quantitative nor qualitative results provided on the Logical Reasoning split from the T2I-CoReBench, which, to me, seems to be the most promising one from the entire benchmark. The paper would benefit from (1) presenting a direct evaluation of your method and (2) showing reasoning refinement over loops on this split.
- Experiments on alternative layer configs are only promised. While I understand the computational constraints, I consider this experiment important to the research community.
New experiment with image-conditioned generation, and evaluation on VBVR-Pro-Bench is a great direction. Given nearly equal scores for Looped-DiT and fine-tuned FLUX, could you share computational and model size differences between the two?
I will be happy to raise my score in case the Authors answer these remaining questions.
Thank you for reviewing our additional results.
1. Logical Reasoning split. Thanks for pointing out this omission. We apologize for omitting the Logical split in our previous response. We had reported Generalization (inductive reasoning from examples) and Hypothetical (reasoning from hypothetical premises and rules), which also require solving an inference problem to determine what should be depicted. We now include the Logical split:
| CoReBench split | CoT | Looping | Both |
|---|---|---|---|
| Generalization | +22.2 | +7.5 | +24.5 |
| Hypothetical | +9.4 | +3.5 | +11.6 |
| Logical | +3.3 | +0.1 | +3.4 |
We view the three splits above as tasks in which a substantial part of the challenge is to infer the intended scene content from examples, rules, or textual premises before rendering it. Our results suggest that textual CoT is particularly helpful for this purpose.
Here's an example of a typical problem in this split:
Generate an image showing three mystical artifacts on pedestals in a triangle formation. Each artifact has a different material (crystal, metal, wood) and a different visual effect (wreathed in flames, shrouded in mist, covered in vines). The artifacts make statements:
(1) The crystal artifact says: “I am covered in vines.”
(2) The metal artifact says: “The crystal artifact is lying.”
(3) The wood artifact says: “I am wreathed in flames, and the metal artifact is telling the truth.”
(4) Additional facts: Exactly one artifact lies, others tell truth.
Lying artifacts have cracks on their surface, truth-telling artifacts have smooth surfaces.
Qwen3 rewrites this prompt as:
Three mystical artifacts are arranged in a triangle formation on pedestals. A crystal artifact shrouded in mist with a cracked surface, a metal artifact covered in vines with a smooth surface, and a wood artifact wreathed in flames with a smooth surface.
This problem can be stated and solved entirely in language, and the solution can then be expressed as an explicit description of the target image. We find CoT-based rewriting doing better in solving problems of this kind. We include the corresponding qualitative comparison in the Appendix A.8 (p. 27).
Where we see promise for looping. We are also interested in reasoning that is closely tied to visual representations and operations: relevant clues may be supplied directly in images, or reaching a consistent solution requires repeatedly constructing and revising a visual configuration. VBVR-Pro-Bench provides more direct examples of these problems, in which we have observed visible benefits from looping.
Our central claim is therefore that looping and textual CoT have complementary strengths across tasks. Textual CoT can help determine what should be depicted, while looping can help construct and revise visual predictions during generation. Their combination achieves the highest performance.
2. Effect of loop configuration.
We have now completed experiments varying both the number and placement of looped blocks. All configurations retain 17 distinct blocks and use Deep Supervision and XSA. Their effective depths are 31–32, approximately matching inference compute; since different number of loops are deep-supervised, training steps are also adjusted to match total training FLOPs.
| Partition [pre, looped, post] | Loops | Effective depth | Avg. |
|---|---|---|---|
| [7,3,7] | 6 | 32 | 57.1 |
| [6,5,6] | 4 | 32 | 59.0 |
| [5,7,5] | 3 | 31 | 58.1 |
| [1,15,1] | 2 | 32 | 57.9 |
| [2,5,10] | 4 | 32 | 58.9 |
| [10,5,2] | 4 | 32 | 58.4 |
Every configuration improves over the non-looped baseline (55.1). Results show that 4 loops of the middle layers perform best in composition with deep supervision under given training compute. The default [6,5,6] ×4 configuration achieves the highest average score, while changing its placement nearer to the front or the end of the network yields closely comparable results (58.4–59.0). Thus, the improvements persist across different choices of looped layers. We have added the full per-benchmark results and experimental details to Appendix A.4 (p. 22).
3. Comparison with fine-tuned FLUX. Thank you for the suggestion. The fine-tuned FLUX baseline uses a 32.22B-parameter denoiser and requires 7.5 PFLOPs per image. In comparison, Looped-DiT B/16 uses 0.26B denoiser parameters and requires 158 TFLOPs per image. Thus, our model achieves a comparable VBVR-Pro-Bench score (0.411 vs. 0.407) with approximately 124× fewer denoiser parameters and 47.5× lower inference FLOPs.
Dear Reviewer wNhq,
Thank you again for your time and valuable feedback on our paper. While we understand that you may have a busy schedule, we would like to confirm whether our responses have addressed your concerns. If you have any further questions or feel that we have misunderstood your feedback, please let us know. We are happy to discuss with you.
Furthermore, if our responses and the revised paper have addressed your concerns, we would be grateful if you would consider updating your evaluation.
Sincerely,
The Authors