acceptodds
Under review as a conference paper at ICLR 2027

Looped Diffusion Transformer

11 views

Abstract

Improving text-to-image models has traditionally relied on increasing model size or the number of denoising steps. In this work, we explore an alternative way to scale computation by repeatedly running shared Transformer blocks within each denoising step, effectively increasing computational depth while keeping the parameter count fixed. This looped computation enables iterative refinement of internal representations without explicit reasoning tokens. However, naive looping fails to consistently improve image quality. We trace this problem to weak supervision across intermediate loops and unregulated attention updates that progressively erode local information. To overcome these challenges, we propose Looped Diffusion Transformer (Looped-DiT), which combines deep supervision across intermediate loops with self-modulating attention to stabilize looped feature updates. Under matched-parameter and matched-compute settings, Looped-DiT consistently outperforms non-looped baselines. Notably, a 260M-parameter looped model can surpass a model 6.5× larger across multiple text-to-image benchmarks while requiring 4.9× lower inference compute. Beyond this performance gain, we find that looped computation can offer a more effective form of iterative computation for diffusion models, with increasing loop depth yielding larger gains than adding more denoising steps under a fixed inference budget. Furthermore, deeper loops can progressively correct mistakes made in earlier loops, exhibiting behaviors suggestive of latent reasoning. Together, these results show that looped computation offers a promising way to scale visual generation models.

Primary area:
generative models
Keywords:
Text-to-Image Generation, Diffusion Transformers, Multimodal Diffusion Transformers, Looped Transformers, Parameter Sharing, Compute Scaling, Iterative Refinement

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wNhqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work addresses whether looped computation, i.e., repeated inference through the middle blocks of the diffusion transformer, can be considered a promising approach for scaling text-to-image models. The authors present that the naive application of looping does not offer consistent improvements. They trace the issue to (1) insufficient supervision during intermediate loops and (2) redundant token updates within looping attention layers, and offer fixes through, respectively, deeply-supervised training loss and self-modulating attention mechanism. The resulting method is compared with prior models across multiple text-to-image benchmarks, demonstrating improvements, also when compared with a non-looped model at matched inference compute.

Critical strengths
  1. I consider the main motivation for this work, which is enabling looped computation with image generative models, to have high potential and be a promising direction for the future of visual generation. The proposed improvements over naive looping (deep supervision and self-modulating attention) are well-motivated (with probing and attention update studies), clearly explained in the work, and empirically shown to improve looped diffusion transformers.

  2. The authors include a broad comparison, evaluating against multiple prior models on up to 6 alignment benchmarks, analyzing both performance and inference speed.

Critical weaknesses
  1. While I consider the model's looped refinement to be a promising phenomenon, I would ask the authors to work on some phrasings in their work to better explain what they mean by "reasoning capability". All qualitative image examples in the paper (Fig. 1, Fig. 5, Fig. 9) refer rather to prompt-following capabilities: visual text adherence (F1,F5), object counting (F9), or object spatial positioning (F5). Also, while I appreciate using multiple benchmarks for evaluation, almost all of them measure prompt-following, often in the form of spatial object composition and counting, in which the prompt explicitly specifies how the image should look. The evaluation that seems particularly interesting here is the reasoning splits from T2I-CoreBench, which includes, i.a., solving puzzles or inferring outcomes from sequences of actions. The Authors use some of those splits in Table 3 (but not all), and their method does not achieve as great gains as the CoT baseline in most cases. Could the authors separately compare against the evaluated models (Table 1), perform ablations (Table 4), and provide more qualitative examples on those splits?

  2. I have concerns with some experimental results reported in the paper:

    2.1. I would ask the authors to include statistical significance (standard error or std) in their results, particularly in Tables 2 and 4. In Table 2, the difference between their method and the wider MiniT2I, at the same inference FLOP count and higher training FLOPs, is relatively small (0.5) and may not be statistically significant. This is important, given that the authors present their method as an alternative for computational scaling.

    2.2. I believe a broader analysis of looped generation is not thoroughly evaluated in this work. Especially, I am not sure whether this approach could work under a choice of different layers that are looped (the current choice seems to be rather ad hoc), and whether the proposed approach can be transferred to diffusion models with different architectures, raising generalization concerns

Questions for authors
  1. I appreciate the analysis of the two methods for reducing redundancy of attention updates in Section 2.2. However, I am not sure what role the Gated Attention mechanism plays in the paper, as it does not outperform the parameter-free XSA on any benchmark (Table 4).

  2. In Appendix A.1, the authors claim to use the short-prompt split of TIIF, as their "small-scale training setup primarily uses short captions with a maximum length of 256 tokens, making the long-prompt setting less representative of the training regime". Considering this, I'm concerned about the comparison with the CoT-based prompt-rewriting baseline in Table 3. The limited average gain from CoT can stem from Qwen3 rewriting benchmark prompts, pushing their length outside the model's training token-length distribution, making the comparison unfair. I'd ask the authors to report the token-length distributions for: (1) MiniT2I training datasets, (2) original benchmark prompts, and (3) CoT-rewritten prompts.

  3. The adaptive looping mechanism seems to be a promising strength of this work. In my opinion, the authors should include more details on this mechanism in the main part of the paper.

3 comments

Dear Reviewer wNhq,

Thank you again for your time and valuable feedback on our paper. While we understand that you may have a busy schedule, we would like to confirm whether our responses have addressed your concerns. If you have any further questions or feel that we have misunderstood your feedback, please let us know. We are happy to discuss with you.

Furthermore, if our responses and the revised paper have addressed your concerns, we would be grateful if you would consider updating your evaluation.

Sincerely,
The Authors

I thank the authors for their work during the rebuttal.

I believe that your experiments sufficiently answer some of my concerns: statistical significance (2.1), generalization to other architectures (partly 2.2), and my questions (Q1, Q2, Q3). Especially, new experiments on the MoT architecture and image-conditioned generation on VBVR, along with image examples, are highly appreciated.

However, some of my concerns remain unaddressed:

  • There are neither quantitative nor qualitative results provided on the Logical Reasoning split from the T2I-CoReBench, which, to me, seems to be the most promising one from the entire benchmark. The paper would benefit from (1) presenting a direct evaluation of your method and (2) showing reasoning refinement over loops on this split.
  • Experiments on alternative layer configs are only promised. While I understand the computational constraints, I consider this experiment important to the research community.

New experiment with image-conditioned generation, and evaluation on VBVR-Pro-Bench is a great direction. Given nearly equal scores for Looped-DiT and fine-tuned FLUX, could you share computational and model size differences between the two?

I will be happy to raise my score in case the Authors answer these remaining questions.

Thank you for reviewing our additional results.

1. Logical Reasoning split. Thanks for pointing out this omission. We apologize for omitting the Logical split in our previous response. We had reported Generalization (inductive reasoning from examples) and Hypothetical (reasoning from hypothetical premises and rules), which also require solving an inference problem to determine what should be depicted. We now include the Logical split:

CoReBench splitCoTLoopingBoth
Generalization+22.2+7.5+24.5
Hypothetical+9.4+3.5+11.6
Logical+3.3+0.1+3.4

We view the three splits above as tasks in which a substantial part of the challenge is to infer the intended scene content from examples, rules, or textual premises before rendering it. Our results suggest that textual CoT is particularly helpful for this purpose.

Here's an example of a typical problem in this split:

Generate an image showing three mystical artifacts on pedestals in a triangle formation. Each artifact has a different material (crystal, metal, wood) and a different visual effect (wreathed in flames, shrouded in mist, covered in vines). The artifacts make statements:

(1) The crystal artifact says: “I am covered in vines.”

(2) The metal artifact says: “The crystal artifact is lying.”

(3) The wood artifact says: “I am wreathed in flames, and the metal artifact is telling the truth.”

(4) Additional facts: Exactly one artifact lies, others tell truth.

Lying artifacts have cracks on their surface, truth-telling artifacts have smooth surfaces.

Qwen3 rewrites this prompt as:

Three mystical artifacts are arranged in a triangle formation on pedestals. A crystal artifact shrouded in mist with a cracked surface, a metal artifact covered in vines with a smooth surface, and a wood artifact wreathed in flames with a smooth surface.

This problem can be stated and solved entirely in language, and the solution can then be expressed as an explicit description of the target image. We find CoT-based rewriting doing better in solving problems of this kind. We include the corresponding qualitative comparison in the Appendix A.8 (p. 27).

Where we see promise for looping. We are also interested in reasoning that is closely tied to visual representations and operations: relevant clues may be supplied directly in images, or reaching a consistent solution requires repeatedly constructing and revising a visual configuration. VBVR-Pro-Bench provides more direct examples of these problems, in which we have observed visible benefits from looping.

Our central claim is therefore that looping and textual CoT have complementary strengths across tasks. Textual CoT can help determine what should be depicted, while looping can help construct and revise visual predictions during generation. Their combination achieves the highest performance.

2. Effect of loop configuration.

We have now completed experiments varying both the number and placement of looped blocks. All configurations retain 17 distinct blocks and use Deep Supervision and XSA. Their effective depths are 31–32, approximately matching inference compute; since different number of loops are deep-supervised, training steps are also adjusted to match total training FLOPs.

Partition [pre, looped, post]LoopsEffective depthAvg.
[7,3,7]63257.1
[6,5,6]43259.0
[5,7,5]33158.1
[1,15,1]23257.9
[2,5,10]43258.9
[10,5,2]43258.4

Every configuration improves over the non-looped baseline (55.1). Results show that 4 loops of the middle layers perform best in composition with deep supervision under given training compute. The default [6,5,6] ×4 configuration achieves the highest average score, while changing its placement nearer to the front or the end of the network yields closely comparable results (58.4–59.0). Thus, the improvements persist across different choices of looped layers. We have added the full per-benchmark results and experimental details to Appendix A.4 (p. 22).

3. Comparison with fine-tuned FLUX. Thank you for the suggestion. The fine-tuned FLUX baseline uses a 32.22B-parameter denoiser and requires 7.5 PFLOPs per image. In comparison, Looped-DiT B/16 uses 0.26B denoiser parameters and requires 158 TFLOPs per image. Thus, our model achieves a comparable VBVR-Pro-Bench score (0.411 vs. 0.407) with approximately 124× fewer denoiser parameters and 47.5× lower inference FLOPs.

Reviewer_vphpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Looped-DiT, which repeatedly applies shared Transformer blocks within each denoising step to improve text-to-image generation without increasing the number of model parameters. The authors introduce deep supervision and self-modulating attention to improve training stability and report promising results on several benchmarks.

Critical strengths

The paper explores an important and timely direction of applying the looped Transformer paradigm to text-to-image generation.

Critical weaknesses

1. Limited contribution and generalizability. The idea of looping has already been explored in visual generation, particularly in ELT. I understand that the authors extend this direction to text-to-image generation, which was not considered in ELT. However, the novelty and scope of the contribution remain somewhat limited, especially given that the experiments focus on a single architecture, resolution, and relatively small model scale.

2. Narrow experimental settings. The authors only evaluate models with approximately 260M parameters at 512×512 resolution. It is unclear whether the proposed approach would provide similar benefits for larger, more practical text-to-image models and higher resolutions. Experiments on larger models, different architectures, and resolutions such as 1024×1024 would significantly strengthen the paper.

3. Inconsistent experimental observations. In Figure 4, Loop ×1 with 100 denoising steps outperforms Loop ×4 with 25 steps. However, Figure 6 suggests the opposite trend, with additional looping outperforming additional denoising steps. I understand that the figures use different model variants (B/16 versus B/32), benchmark sets, and compute configurations, but this raises questions about the generality of the authors conclusions. It would be important to clarify these differences.

4. Experimental improvements are not fully convincing.

  • In Table 2, under matched training-compute settings, looping improves the average score by only 1.0 point (59.1 versus 58.1). It is unclear how statistically significant this difference is. Reporting std or confidence intervals, ideally across independent training runs, would help establish the significance of the improvement. Human evaluation and more qualitative comparisons would also strengthen the conclusions.
  • In Table 5, the full looping approach achieves 59.1 compared with 58.8 for the deeper non-looped baseline with XSA and deep supervision. Although the looping model uses fewer parameters, the performance improvement is only 0.3 points. Moreover, Table 4 shows that looping alone provides a relatively modest improvement, while much of the remaining gain comes from self-modulating attention and deep supervision. This raises questions about how much of the final performance improvement should be attributed specifically to looping rather than the additional components.


5. More analysis of intermediate representations would be useful. The authors provide some qualitative examples of refinement across loops. However, it would be interesting to systematically visualize the intermediate hidden states through the shared post-loop decoder at different denoising steps. Such visualizations could provide better intuition about how looping progressively improves predictions and whether the claimed self-correction behavior consistently emerges during sampling.


The paper addresses an interesting problem and reports promising results in a controlled setting. However, the contribution is limited by its narrow experimental scope, some unclear observations and relatively small improvements over stronger baselines. More extensive experiments and stronger evidence isolating the benefits of looping are needed to establish the significance of the approach.

Questions for authors

see weaknesses

4 comments
AuthorsOpenReview ↗

Additional results on intermediate representations

[W5] Additional visualizations of intermediate predictions. We have added more visualizations of intermediate predictions decoded through the shared post-loop decoder, across different denoising steps and loop iterations, in Figs. 12–14 (pp. 23–25). We observe that early loops often explore multiple candidate solutions, while later loops progressively suppress incorrect alternatives and settle on a single answer. This ambiguity tends to reappear at subsequent denoising steps, as the intermediate sampling states remain noisy, followed by another round of exploration and pruning. These examples show that the exploration-and-pruning behavior appears consistently across different denoising steps.

We hope these additional results and our previous responses have addressed your concerns. If you have any further questions or feel that we may have misunderstood your feedback, we would be happy to provide further clarification and continue the discussion. If you feel that our responses and revisions have satisfactorily addressed your concerns, we would also greatly appreciate your consideration of updating your evaluation.

Thank you again for your time and constructive feedback.

Best regards,
The Authors

I thank the authors for their rebuttal. Some of my concerns regarding the narrow experimental settings and generalizability have been resolved. However, I am still not fully convinced by the claimed performance advantage of looping itself.

Specifically, according to Table 7, applying XSA and DeepSup without looping yields a score of 58.7, while the corresponding looped model achieves 59.0, an improvement of only 0.3 points. This difference is comparable to the reported std. I understand that DeepSup and XSA interact with looping, but they can also be applied to a non-looped model. Therefore, this constitutes a fair baseline against which the improvement from looping does not appear clearly significant under matched compute. This seems at odds with the authors direct claim that “under matched-compute settings, Looped-DiT consistently outperforms non-looped baselines.”

Can the authors clarify this point?

AuthorsOpenReview ↗

Response to concerns regarding benefits of looping

Thank you for the follow-up. There isn't any odds with the central claim. We would like to clarify that the central advantage of Looped-DiT is parameter-efficient scaling of computation, rather than simply maximizing the absolute performance gap.

Importantly, the comparison in Table 7 reinforces this contribution: under matched FLOPs, Looped-DiT achieves higher performance (59.0 vs. 58.7) with 45% fewer parameters (260M vs. 473M), even against a stronger non-looped baseline equipped with both XSA and DeepSup. Thus, the key finding is not the 0.3-point margin in isolation, but the ability to achieve comparable or better performance with substantially fewer parameters.

Furthermore, additional experiments demonstrate the benefits of looping consistently holds across datasets and model scales, beyond this particular comparison.

1. Alternative Dataset (QIB)

ModelQIB Score ↑
Non-looped Depth-32 (B/32) - row4 in Tab. 737.1 ± 0.51
Looped-DiT (B/32) - row6 in Tab. 737.9 ± 0.35

2. Larger Model (B/16)

ModelGenEval ↑DPG ↑
Non-looped Depth-32 - row4 in Tab. 735.7470.48
Looped-DiT (B/16) - row6 in Tab. 738.5671.86
Improvement+2.82+1.38

3. Scaling to 0.91B Parameters and 1024² Resolution

We further scale Looped-DiT to 0.91B parameters (L/16), with results reported in the main table. At 1024² resolution, Looped-DiT L/16 achieves a new SOTA in Table 1:

BenchmarkLooped-DiT L/16
GenEval87.5
DPG87.1
PRISM71.5
CoRe55.6
Spatial55.2
TIIF81.0

Moreover, our experiments on a distinct Mixture-of-Transformers architecture at 2B and 8B scales further demonstrate the generalizability of looping. At 8B/1024², looping improves GenEval by 9.1 points (49.9 → 59.0) and DPG by 4.9 points (78.3 → 83.2).

Taken together, these results directly address the concern: the contribution of looping should not be judged solely by a 0.3-point difference against a substantially larger baseline. Instead, Looped-DiT demonstrates competitive or superior performance with 45% fewer parameters under matched compute, alongside consistent improvements across additional datasets, model scales, and architectures. We believe this collective evidence strongly supports the effectiveness, parameter efficiency, and generalizability of the proposed approach. We have added these results to our main paper.

Thanks for the response. I still have concerns about the performance gap. However, the direction of looping transformers is important in DMs now. So, because the paper is timely and might be interesting for the conference, I raise my score.

Reviewer_nKyPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores looped Diffusion Transformers for text-to-image generation, extending the idea of repeatedly applying shared Transformer blocks, which has recently gained attention in language modeling, to image synthesis. By increasing computational depth without substantially increasing the parameter count, this approach offers a promising direction for improving the efficiency and capability of image generation models. Overall, I find this investigation meaningful, particularly its exploration of how iterative refinement of hidden representations can benefit visual generation.

Critical strengths

The paper has several strengths. First, it is clearly written and well organized, making the motivation and methodology easy to follow. It explores loop depth as an additional axis for scaling computation in image generation, and the figures clearly illustrate the Looped-DiT architecture and its iterative refinement process. Second, the evaluation covers a broad range of text-to-image benchmarks, demonstrating the benefits of the proposed approach across different generation tasks. The paper also provides useful comparisons and analyses of inference loop depth, supervision strategies, attention modulation, and fixed versus adaptive looping. These experiments help explain both the effectiveness of the method and the contributions of its individual components. Overall, I consider this a solid and worthwhile empirical study.

Critical weaknesses
  1. The paper does not include a user study to evaluate the quality of the generated images. Such an evaluation would strengthen the experimental results.

  2. Does Table 5 suggest that looping itself provides only a modest benefit, while XSA plays a larger role? The authors should clarify the relative contributions of these components.

  3. Loop-based refinement has been explored in detection and segmentation as follow [1]. The paper should discuss its connection to these approaches and clarify the differences in paper.

[1] High-Resolution Iterative Feedback Network for Camouflaged Object Detection.

  1. Is the loop mechanism mainly performing progressive refinement, with the proposed components preventing additional iterations from introducing errors? A clearer explanation and analysis would be helpful.

5.The claim of “latent visual reasoning” seems somewhat overstated, as the paper provides limited evidence to support this interpretation.

  1. The experiments remain at a relatively small scale. Does this sufficiently support the paper’s broader scaling narrative? Additional scale-up experiments would strengthen the claim.

  2. Could the authors present the training computational resources and costs more clearly?

Questions for authors

Please address the questions corresponding to the weaknesses listed above.

1 comment

Thank you again for your time and valuable feedback on our paper. We hope that our responses and additional experiments have addressed your concerns. If you have any further questions or feel that we may have misunderstood any of your comments, please do not hesitate to let us know. We would be happy to provide further clarification and continue the discussion.

If you feel that our responses and revisions have satisfactorily addressed your concerns, we would also greatly appreciate your consideration of updating your evaluation.

Thank you again for your time and consideration.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Reviewers and Area Chair,

Thank you for your time and constructive feedback. Looped-DiT explores parameter-efficient scaling of computation for text-to-image generation by repeatedly applying shared Transformer blocks. Unlike prior approaches relying on intra-loop self-distillation, Looped-DiT directly supervises intermediate loops using the original flow-matching objective, while Self-Modulating Attention (XSA) mitigates redundant updates. Our main contributions and additional evidence are:

  • Scalability and generalization. We scaled up Looped-DiT to 1B parameters and 1024 resolution, and we observe consistent gains from looping at larger scales. We also extended Looped-DiT to a different Mixture-of-Transformers (MoT) architecture at 2B and 8B scales, across 512×512 and 1024×1024 resolutions. Looping consistently improves GenEval and DPG-Bench across all four settings. At 8B/1024², GenEval improves by 9.1 points (49.9 vs. 59.0) and DPG-Bench by 4.9 points (78.3 vs. 83.2). On image editing, the 2B model also improves GEdit-EN by 1.0 and ImgEdit by 0.3.

  • Parameter and compute efficiency. In the matched-compute comparison, Looped-DiT improves the six-benchmark average by 1.0 point (59.0 ± 0.19 vs. 58.0 ± 0.27), across three independent training runs. Against a stronger non-looped baseline with both XSA and Deep Supervision, it achieves comparable accuracy (59.0 vs. 58.7) with 45% fewer parameters (260M vs. 473M).

  • Human evaluation. In a study involving 53 participants and 530 prompt-matched image pairs across six benchmarks, Looped-DiT achieves a 75.3% human preference rate over its non-looped counterpart (95% confidence interval: 71.4–78.8%).

  • Visual reasoning and iterative refinement. On VBVR-Pro-Bench, Looped-DiT improves from 0.365 to 0.411 over its non-looped counterpart. Additional visualizations across loops and denoising steps reveal exploration-and-pruning behavior, providing evidence of iterative visual computation beyond conventional prompt following.

  • Effectiveness of XSA. XSA improves performance by 1.7 points with looping, compared with 0.7 points without looping, supporting its role in mitigating redundant updates during repeated computation.

Our rebuttal and revised paper provide further experiments and clarifications addressing the reviewers' concerns:

Reviewer nKyP

  • Added human evaluation, 2B/8B scaling experiments, image-editing results, and dedicated visual reasoning evidence.

  • Clarified the respective contributions of looping and XSA, expanded related work with HitNet, and reported detailed training costs (670 and 3,930 H100 GPU-hours for B/32 and B/16).

Reviewer vphp

  • Added independent-run statistics, stronger matched-compute comparisons, and experiments across architectures, model scales, resolutions, and image editing.

  • Clarified loop-versus-denoising-step comparisons and added intermediate visualizations across loops and denoising steps.

  • Clarified the stronger non-looped baseline comparison, emphasizing comparable accuracy with 45% fewer parameters rather than claiming a significant advantage from the 0.3-point difference.

Reviewer wNhq

  • Expanded reasoning evaluations on T2I-CoReBench and VBVR-Pro-Bench, including qualitative exploration-and-pruning examples.

  • Added statistical analysis and architectural generalization experiments, and clarified the complementarity between looping and textual Chain-of-Thought.

  • Reported prompt-length statistics, clarified Gated Attention, and expanded the adaptive-looping analysis. Additional requested analyses of the Logical Reasoning split, alternative loop configurations, and the FLUX resource comparison remain open.

Together, these additional results strengthen the evidence for Looped-DiT's scalability, parameter efficiency, and iterative refinement capabilities across architectures, model scales, resolutions, and generation tasks.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.