acceptodds
Under review as a conference paper at ICLR 2027

How to Loop Your DiT: Retrofitting Loops on Image Generation Models

Abstract

Scaling inference-time compute has emerged as a prominent avenue for improving neural network performance. Looping models, where transformers are recursively applied to refine predictions, offer a particularly attractive route by allowing a user to control the effective depth of the model at inference time. In the context of image generation, looped transformers trained from scratch offer not only inference-time scaling, but can often match the performance of larger models using significantly fewer parameters. However, training looped models is expensive: an -loop model requires roughly times the FLOPs in both training and inference, negating much of the efficiency advantage. We tackle these limitations by retrofitting loops on top of pretrained models. We focus on the family of ImageNet-scale diffusion models and show that naively looping them often leads to limited improvement in performance. Instead, we draw on recent recursive reasoning literature and present a recipe that uses the looped model to refine a set of context tokens, rather than the main answer stream. We show that this approach can be applied on top of pre-trained models to recover performance competitive with from-scratch looping at a fraction of the training cost. Moreover, models trained in this manner exhibit favorable test-time scaling, and can be easily distilled to lower loop counts with negligible quality loss. As a concrete example, with only a third of the training budget, we retrofit a FLUX.2-style XL/2 model to a -loop variant that matches a larger feed-forward model, and distill it down to loops with little degradation.

Primary area:
generative models
Keywords:
Looped Models, Image Generation, Diffusion Transformers

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_2u5qOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how to retrofit pretrained diffusion Transformers into recurrent models. At each denoising timestep, the method keeps the noisy input fixed and uses the previous prediction and additional scratchpad tokens as context for iterative updates through a weight-tied network. Experiments are conducted across three architectures and two model sizes.

Critical strengths
  1. Retrofitting pretrained models with recurrent computation is a practically meaningful research direction.

  2. The experiments cover three architectures: DiT, SiT, and FLUX.2, providing a certain degree of cross-architecture validation.

Critical weaknesses
  1. The central idea is directly from TRM (arXiv:2510.04871). There is no new theoretical insight, no new architectural primitive, and no demonstration that the same recipe unlocks capabilities that pure parameter scaling or existing inference-time methods cannot.

  2. Each loop processes 640 tokens instead of 256. Table 5 shows that a 4-loop forward pass already costs several times more FLOPs than the baseline or even the ×4 model under comparable conditions. In a fair iso-FLOPs comparison at inference, the advantage largely disappears. The paper never convincingly shows that the method is preferable to simply using a larger feed-forward model or more sampling steps under the same compute budget.

  3. The paper presents the independent scratchpad as one of its main architectural designs, but Table 4 does not provide a direct ablation that retains the fixed noisy input and the previous prediction while removing only the scratchpad. The existing "w/o Previous Answer" experiment demonstrates the importance of prediction feedback, but does not establish the necessity of an independent scratchpad.

  4. The core contribution and motivation of the paper are unclear, and the experimental results do not adequately support the claimed contributions. All the compared methods are implemented by the authors themselves, making it difficult to verify whether the ELT baseline faithfully reproduces the original method (arxiv:2604.09168). The authors should provide detailed baseline pseudocode and implementation details, or compare against the results reported in the original paper.

  5. Table 1 should include results with classifier-free guidance. The results in Table 2 do not warrant a separate table, as precision and recall are generally not considered primary evaluation metrics for image generation. Furthermore, the paper lacks comparisons with current state-of-the-art image generation models, making the experimental results less convincing.

  6. The computational cost analysis is insufficiently emphasized in the main paper, despite efficiency being one of its central motivations. Although the paper includes FLOPs vs. FID curves, the detailed comparisons of training and inference costs are relegated to Appendix. These substantial computational overheads are critical to evaluating the paper's efficiency claims and should be explicitly presented and discussed in the main text.

Questions for authors
  1. The AI Use Statement acknowledges that generative AI tools were used to produce the figures. However, several figures appear to lack sufficient manual refinement in terms of layout, visual consistency, and readability. Directly using AI-assisted figures without adequate polishing undermines the professionalism and credibility of the paper.
Reviewer_acxfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to add recurrent loops to pretrained diffusion transformers. At each denoising timestep, it keeps the noisy input fixed and updates the previous prediction and a separate latent scratchpad through a shared backbone. It uses a specific initialization scheme and random loop counts during training. The authors also study loop-count distillation and loop-skip CFG to reduce compute costs. Experiments cover DiT, SiT, and FLUX.2-style models at B/2 and XL/2 scales on ImageNet 256×256. The results show improved generation quality, with parameter savings and flexible loop counts in some settings.

Critical strengths
  • The pretrained-model setting is useful, and the full method works across several architectures. Reusing pretrained weights to increase effective depth is a practical research direction. The method improves quality across three architectures and two model scales. Without CFG, SiT-XL/2 improves from 12.82 to 9.81 FID, and FLUX.2-style XL/2 improves from 7.44 to 5.25 (Table 1). These results support the effectiveness of the full recipe within the ImageNet setting.

  • Quality improves with limited loss of diversity. The paper reports FID, DINO-FD, precision, and recall. For FLUX.2-style B/2, FID improves from 21.74 to 14.36, while recall changes from 0.748 to 0.744. In comparison, best-of-16 reduces recall to 0.600 (Tables 1 and 2). This suggests that looping offers benefits beyond selecting higher-scoring samples.

Critical weaknesses
  • The contribution over ELT and self-conditioning is not clear. ELT supports shared-weight loops, variable loop counts, and self-distillation across loops. Feeding previous predictions back into the model is a core idea of self-conditioning. Analog Bits also includes same-timestep prediction feedback during training and in sampling variants. The contribution therefore rests mainly on the pretrained-model recipe and the added recurrent state. However, the paper lacks a direct, compute-matched comparison with a simple self-conditioning baseline.

  • The benefit of the scratchpad is not direct. In Table 4, replacing the previous prediction with registers increases FID from 14.36 to 17.39 without CFG. The scratchpad remains present in this comparison, so this result supports the value of prediction feedback. The additive-state and separate-update experiments only compare fusion methods or update schedules. The paper does not test prediction feedback without a scratchpad. It also lacks a control with the same number of registers reset at each loop. These controls are needed to establish the benefit of carrying a separate hidden state across loops and to justify its cost.

Questions for authors
  • Which mechanisms are new relative to ELT and existing self-conditioning methods?
  • Which results separate the benefit of the scratchpad from the benefit of prediction feedback?
  • Starting from the same pretrained checkpoint, could the authors compare the following settings while keeping prediction feedback and other training settings consistent:
    • No independent scratchpad.
    • The same number of learned registers, reset at each loop.
    • The proposed scratchpad, updated and carried across loops.
Reviewer_vf2qOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Looped Transformers enable test-time scaling by repeatedly applying a weight-tied Transformer block, allowing the effective depth of the model to be flexibly adjusted at inference time. This paper proposes a method for retrofitting pretrained image-generation Diffusion Transformers with recursive loops. Rather than directly applying ELT-style looping to pretrained models, the proposed method draws inspiration from TRM's separation of reasoning and answer states. Specifically, it keeps the original noisy latent fixed, provides the previous loop's prediction as additional context, and introduces latent scratchpad tokens to carry intermediate states across loops. During training, the method adopts stochastic loop counts and computes the loss on the output of the sampled final loop, enabling the model to operate at different loop depths. The authors conduct experiments on three architectures—DiT, SiT, and FLUX.2-style models—and compare their approach against the original baselines and their own reimplementation of retrofitted ELT. Under the same number of training steps, with both looping methods using four loops, the proposed method achieves lower FID on most model configurations.

Critical strengths
  1. The paper proposes a recursive looping architecture that incorporates additional latent scratchpad tokens and the previous loop's prediction. By keeping the original noisy latent fixed, the model iteratively updates both its prediction and intermediate states. The proposed architecture is retrofitted onto pretrained models, enabling test-time scaling by adjusting the number of loops at inference time.

  2. The paper adopts a stochastic loop-count training strategy, applying the denoising loss at the sampled loop depth. This reduces the average number of loops executed during training while enabling the model to operate effectively across different inference-time loop counts.

Critical weaknesses
  1. The current experiments are insufficient to fairly evaluate the test-time scaling performance of the two methods. The proposed method introduces 384 additional tokens, increasing the total number of tokens processed per forward pass to 640, whereas ELT processes only 256 tokens. Therefore, the same number of loops does not correspond to the same inference compute budget. Since ELT has a lower computational cost per loop, it can perform more loops under the same inference budget. However, the paper primarily compares models with the same number of loops, and its investigation of higher inference-time loop counts is largely based on ELT models trained with four loops. This cannot replace comparisons with ELT models specifically trained using higher loop counts. Although Figure 1 provides some comparisons in terms of inference FLOPs, these are primarily based on fixed training loop configurations. Similarly, while Table 6 matches training FLOPs, it does not simultaneously match inference compute budgets or adequately account for the effects of different training loop-count configurations. Furthermore, according to Tables 1 and 5, for FLUX.2 XL/2 without CFG, ELT-4 achieves a lower FID (5.06 vs. 5.25) despite having a lower total training cost (301.2 vs. 416.9 EFLOPs, or 172.1 vs. 287.9 EFLOPs after excluding the shared baseline pretraining cost) (The shared baseline pretraining compute can be obtained by subtracting the additional retrofitting compute from the total training compute) and requiring only approximately 38% of the inference compute of the proposed method (44.8 vs. 118.4 TFLOPs/image). This result further suggests that the claimed advantages of the proposed method require validation under strictly matched compute budgets.

  2. The implementation and reliability of the ELT baseline require further verification. The authors' description of the ELT looping mechanism is inconsistent with the original paper. This paper describes ELT as directly feeding the previous loop's prediction into the next loop, whereas the original ELT repeatedly applies a shared Transformer block to hidden representations, rather than feeding the previous denoising prediction back as the input to the next loop. Although the authors state that they reimplemented ELT, the paper does not provide sufficient implementation details to verify whether their implementation faithfully follows the original method. Therefore, the exact implementation of the ELT baseline and the reliability of the corresponding experimental results require further clarification.

  3. The novelty of the proposed method is not clearly positioned relative to prior work. Retrofitting pretrained Transformers with recursive loops is not a new idea. Prior work, such as "Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence" [1], has already explored this direction, and the authors have cited this work. Furthermore, this paper also includes an ELT baseline retrofitted onto pretrained models through fine-tuning. Therefore, the authors should more clearly delineate the specific contributions of their method relative to existing approaches, rather than emphasizing the retrofitting of loops onto pretrained models itself as a primary novelty.

  4. The experimental analysis is insufficiently substantiated, and some explanations lack adequate supporting evidence. The authors' interpretations of certain experimental observations are not sufficiently rigorous. For example, after observing what they describe as a grokking phenomenon, the authors claim that this behavior "strongly hints" that the model could be further improved through better initialization or training schedules, without providing sufficient evidence to support this conclusion. Similarly, the authors hypothesize that FLUX.2 performs better at higher loop counts because RoPE better preserves positional information across deeper loops. However, they say "leave its investigation to future work". These limitations weaken the persuasiveness of the proposed explanations for the observed behavior.

[1] McLeish, Sean, et al. "Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence." arXiv preprint arXiv:2511.07384 (2025).

Questions for authors

Please see the weaknesses.

Reviewer_x8jiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose using retrofitting loops on top of pretrained diffusion transformers on ImageNet. This method can be seen as a cheaper alternative to the trained Elastic Looped Transformers (ELT) (Goyal et al., 2026). The paper argues that trained ELT-style models where each pass output becomes the next pass input are expensive and negate most of the efficiency advantage.

The propose method follows Tine Recursive Models (TRM) (Jolicoeur-Martineau, 2025) and HRM (Wang et al., 2025). It keeps the noisy latent fixed and adds two token streams that he shared backbone updates in every loop. The new output projections are zero-initialized while attention to the new tokens stays active, and during training the loop count is sampled stochastically.

The experiments are done using DiT, SiT and FLUX.2 models at two different size scales B/2 and XL/2. The proposed method is compared against a retrofitted x4 wider ELT-4 model and test time scaling method. The performance is reportes with FID, DINOv2-FD and precision/recall. The paper report consistent gains over the base model and is matching or beating the ELT-4 model at XL/2 without CFG.

Critical strengths

[S1] The experiments show that on DiT and SiT, ELT-style chaining leaves FID unchanged, while the proposed method improves all six model and scale combinations.

[S2] The ablations shown in Table 4 support the main choices in the paper. The split streams beat TRM-like additive states and the stochastic depth matches ILSD at lower training cost.

[S3] The paper presents a broad evaluation with three architectures and two scales, comparisons against test-time scaling and and reports 3 metrics (precision/recall, DINOv2-FD, and FID).

Critical weaknesses

[W1] The paper presents some comparisons with matching parameters but with a large inference cost gap. Each loop processes 640 tokens (instead of 256). Therefore, the 4-loop model seemas to cost 11 times more than the base model per images and about 3 times more the ELT-4 and x4 models. The tables 1 and 7 in the paper present these models side-by-sde without clearing showing these inference costs. This makes the method look far more favorable than a cost-aware comparison would.

[W2] It seems that the retrofit alone costs more than the pretraining it builds on. The total cost is roughly 60% of the x4 model's training compute after inclusing the inherited 750k steps, which contradicts the claim of "a third of the training budget" in the abstract.

[W3] Table 6 matches training FLOPs only among the proposed method, ELT-4 and the x4 model. It omits the base model trained further to the same total FLOPs which it the natural control for a retrofit that costs more than pretraining. This is crucial since the paper's main claim is a cheaper route to larger model quality.

[W4] At lower cost, the simpler baselines are as good or better. For example, on FLUX.2 (XL/2 without CFG), the retrofitted ELT-4 reaches a better FID compared to the proposed method at about 40% of its inference FLOPs and about 70% of its training FLOPs.

[W5] Fig. 1 (right) omits the most relevant cpmarison. The plot shows the FLUX.2-B/2 with CFG, where by Tables 5 and 7 the ×4 model is shown to beat every looped configuration show in the teaser figure.

[W6] On DiT and SiT, the loops seem to contribute little beyond a single refinement pass. Most of the gain comes from one extra pass conditioned on the previous prediction and it's not clear to see how the loops can be seen as test-time scaling for these architectures.

Questions for authors

[Q1] How do the comparisons look like with matching compute? Consider including them: e.g., the base model trained further to the retrofit't total training FLOPs and comparisons with equal inference TF/img (your 2-loop model vs ELT-4 and the x4 model).

[Q2] Which quantities exactly"a third of the training budget" (abstract) and "typically 1/3" (L. 084) compare?

[Q3] How were the x4 models tuned (learinng rate, warmup, etc..)? SiT-XL/2 ×4 is worse than its baseline with CFG and barely better without. If the XL comparisons are based on under-trained ×4 runs, those results need extra examination.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.