acceptodds
Under review as a conference paper at ICLR 2027

R-DiT: Recursive Diffusion Transformer

Abstract

Diffusion models have primarily scaled along two axes: increasing the number of model estimations and increasing the capacity of the denoiser. The former exhibits diminishing returns as sampling depth grows, while the latter improves generation at the cost of increasingly large parameter and memory footprints. We introduce Recursive Diffusion Transformer (R-DiT), which introduces recursive depth as an additional compute-scaling axis. R-DiT repeatedly refines its hidden representation through shared Transformer blocks, increasing within-step computational depth without proportionally increasing the number of unique parameters. To make shared computation effective, we introduce recursion modulation, which conditions the denoiser jointly on diffusion time and recursion time. We further propose adaptive recursion, allocating different amounts of recursive computation across the denoising trajectory under a fixed compute budget. Across image, language, and molecular diffusion, we show that R-DiT significantly improves the quality-compute-parameter trade-off over standard diffusion Transformers. Our results establish recursive computation as a complementary scaling axis alongside sampling depth and model size in training diffusion models.

Primary area:
generative models
Keywords:
Recursive transformer, Diffusion models, Diffusion architecture

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_UzQqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes R-DiT, a Diffusion Transformer that reuses a small set of shared blocks R times within each denoising step, and presents this "recursion depth" as a third compute-scaling axis alongside sampling depth T and model depth L. To apply such recursion, the authors propose (i) architectural choices and (ii) time-adaptive recursion depth, based on empirical analyses. For (i), the authors apply recursion in hidden states, condition DiTs with the recursion depth rr, add a modulation layer to be applied at each recursion, and employ a residual structure for each recursion. For (ii), the authors put more recursion at later sampling steps, improving the generation quality under a fixed computational cost. Extensive experiments are conducted over image, language, and molecular domains.

Critical strengths

Motivating analysis. Treating within-step recursion as a scaling axis distinct from T and L is a useful perspective. The analysis in Sec. 2.3 directly motivates both recursion-aware conditioning and time-adaptive allocation.

Extensive experiments across different settings. The authors show various experiments with different modalities, model width, architectural designs, etc. They clearly demonstrate how recursion improves the generation quality under various settings.

Critical weaknesses

Some experimental results seem to be misleading.

  • In Table 1, FID worsens monotonically from T=50 (4.18) to T=250 (4.82), which is unusual for a flow-matching model and suggests CFG was tuned only for T=50 or the network is not trained enough.
  • In Figure 4 (b), all the baselines except for LightningDiT use a weaker VAE encoder, which is not a fair comparison. Although R-DiT achieves reasonable results with only 64 epochs of training, recursion incurs more computational cost (TFLOPs) and latency, so increasing R with a fixed parameter count is not a fair comparison.
  • Recursion does not reduce latency or activation memory, but no wall-clock, memory, or training-cost measurements are given.

Time-adaptive allocation is a known, axis-agnostic technique, yet it is applied only to the recursion axis. Allocating more computation to low-noise steps has been studied extensively. Prior work uses different-size models per step (OMS-DPM [A], DDSM [B], T-Stitch [C]), timestep-dependent early exit or width (ASE [D], AdaDiff [E], DyDiT [F]), and, for the sampling axis, non-uniform step schedules (EDM [G], AYS [H]). None of these works is cited or discussed. More importantly, Table 1 denotes the recursion axis by the average depth Rˉ\bar{R}, which suggests the time-adaptive schedule is used there, while the depth and sampling baselines use uniform allocation.

[A] Enshu Liu, “OMS-DPM: Optimizing the Model Schedule for Diffusion Probabilistic Models,” ICML 2023.

[B] Shuai Yang, “Denoising Diffusion Step-aware Models,” ICLR 2024.

[C] Zizheng Pan, “T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with Trajectory Stitching,” ICLR 2025.

[D] Taehong Moon, “A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models,” ICML 2024.

[E] Shengkun Tang, “AdaDiff: Accelerating Diffusion Models through Step-Wise Adaptive Computation,” ECCV 2024.

[F] Wangbo Zhao, “Dynamic Diffusion Transformer,” ICLR 2025.

[G] Tero Karras, “Elucidating the Design Space of Diffusion-Based Generative Models,” NeurIPS 2022.

[H] Amirmojtaba Sabour, “Align Your Steps: Optimizing Sampling Schedules in Diffusion Models,” ICML 2024.

Questions for authors
  • Was CFG re-tuned for each T in Table 1?
  • What are wall-clock latency, peak inference memory, and training cost (FLOPs, GPU-hours) for R=1/2/4 vs. L=7/14/28?
  • Does Table 1 use fixed or adaptive recursion? Sec. 3.3 describes a linear schedule, but Table 2 uses piecewise schedules such as (1, 7, 8, 8). Are Table 2 schedules trained separately?
  • How is the token-space recursion in Fig. 3 (a) implemented? What intermediate output is fed back (velocity, noise, etc.)?
  • How does R-DiT compare empirically with ELT, the closest prior work?
9 comments
AuthorsOpenReview ↗

Response to Reviewer UzQq (1/4)

We sincerely thank the reviewer for the detailed assessment and for recognizing the motivation and clear demonstration of our results through various experiments. We address the reviewer's concern in the following.


R3-1. CFG tuning

We thank reviewer for the careful review. Yes, our result in Table 1 is based on the CFG-tuning while we follow other setups the same as in the original sampling protocol as in [1].

For the CFG-tuning, we generate 1,024 samples for the L=7,14,21,28 models (with R=1) and sweep the CFG values and choose the CFG value that results in the lowest FID scores. As a result, we opt CFG=20, 15, 12.5, 10 for the L=7,14,21,28 models accordingly. Accordingly R=1,2,3,4 results (with L=7) are all evaluated with the CFG weight of 20.

For more comparative viewpoint, we have additionally run the sampling under the same fixed CFG value across all models:

Table R3-1. Fixed CFG results (CFG weight: 10).

LTRFID ↓IS ↑sFID ↓Precision ↑Recall ↑
75016.45147.76.430.720.58
710016.11151.85.640.730.58
715016.30150.85.770.730.59
720016.43150.15.930.720.59
725016.47150.26.040.720.59
145013.29199.55.770.780.58
215012.66232.85.510.800.57
285012.47247.85.460.810.58
75023.61193.65.680.780.57
75033.11220.65.820.800.55
75042.98232.76.020.800.55

The result shows that our recursion depth consistently improves the performance compared to increasing the sampling depth even under the fixed CFG scenario.

Moreover, we provide the 50K evaluation result with the independently tuning CFG weights for the different sampling steps (T).

Table R3-2. 50K CFG sweep results under different sampling steps (7 layers, no recursion models).

TCFGFID ↓sFID ↓IS ↑Precision ↑Recall ↑
50154.766.19183.40.740.57
17.54.266.21199.90.750.57
204.186.32212.50.750.56
100154.565.76188.70.740.57
17.54.295.97203.10.750.56
204.336.29213.70.740.56
150154.775.98186.90.740.58
17.54.516.62210.80.740.56
204.586.57212.50.740.56
200154.916.17185.60.730.57
17.54.676.45199.50.740.56
204.756.86209.10.730.56

This shows that merely increasing sampling depth does not improve even under the per sampling step CFG-tuned setup.


R3-2. Implementation details

We appreciate the reviewer for the careful review.

(a) Adaptive recursion in Table 1, 2

Table 1 uses adaptive linear-scheduled recursion where we denote Rˉ\bar{R} as the average recursion numbers during the inference. The ablation conducted in Table 2-(left) uses separte adaptive recursion schedule, independently trained for providing the fair comparison.

(b) Token-space recursion

Our token-space recursion is implemented by passing the output (velocity estimation) in each recursion step and project back to the original noisy latent at the same timestep to estimate the clean latent, then patch-embed that estimate for the next pass through the shared backbone. Only the final velocity prediction is supervised and recursion adds no fresh noise or sampling step.

AuthorsOpenReview ↗

Response to Reviewer UzQq (2/4)

R3-3. Efficiency analysis

R-DiT reuses parameters across recursive passes, increasing effective computational depth without proportionally increasing parameter storage. Beside its memory efficiency, we show that compared to increasing the number of layers, increasing recursion depth (R) proportionally can lead to the improvement on training speed and faster inference in memory-budgeted regime. For all measurements, we use RTX 3090 GPUs with FP32 parameter storage and BF16 autocast.

(a) Peak memory and batch-1 inference latency

Table R3-3. Peak allocated memory and batch-1 inference latency with the optimized implementation.

ModelInference peak memory (MiB)Training peak memory (GiB)Inference latency / step (ms)
R-DiT (R=1)748.04.405.01
DiT (L=7)694.24.304.91
R-DiT (R=2)755.35.599.44
DiT (L=14)1,332.78.359.20
R-DiT (R=4)755.37.6318.21
DiT (L=28)2,609.616.4417.69

At effective depth 28, R-DiT reduces peak inference memory by 71.1% and peak training memory by 53.6%, despite 2.9% higher per-step inference latency than L28 (due to the recursion modulation).

(b) Inference throughput gain under constrained memory.

Table R3-4. Throughput gain under the memory-budegeted scenario.

Cap (GiB)R4 batchXL batchR4 requests/sXL requests/sR4 gain
2.5064—2.222Cannot run (OOM)—
2.5572—2.243Cannot run (OOM)—
2.607232.2361.685+32.8%
2.657242.2381.880+19.0%
2.708082.2392.049+9.3%
2.7572112.2392.117+5.8%
2.8080132.2362.145+4.3%

We independently select each model’s highest-throughput tested batch under the same memory cap. R-DiT’s lower parameter memory allows larger batches, improving throughput by 32.8% at 2.60 GiB. As the budget increases, XL can also use larger batches, narrowing the advantage to 4.3% at 2.80 GiB.

These results show that our recursive architecture provides practical throughput gains when available memory restricts the baseline’s batch size.

(c) Parameter sharing can improve training throughput through better memory–compute trade-offs.

We benchmarked real training updates on four RTX 3090 GPUs using mixed BF16 precision and an effective global batch size of 1,024, both at native capacity and under a 16 GiB allocator cap per GPU. To avoid OOM, in native capacity (24 GiB) experiment, L28 uses checkpointing. Under the 16 GiB cap, R-DiT R1/R2/R4 and DiT L14 use partial checkpointing, while L28 uses full checkpointing. Total GPU-hours are extrapolated estimation from the real training time upto 1K updates.

Table R3-5. Training efficiency comparison at native 24 GiB GPU capacity.

ModelTraining examples/s ↑Training FLOPs (×10¹⁸)Total GPU-hours (estimated) ↓
R-DiT (Rˉ=1\bar R=1)879.814.67103.5
DiT (L=7)889.414.67102.3
R-DiT (Rˉ=2\bar R=2)463.230.34196.5
DiT (L=14)422.529.33215.4
R-DiT (Rˉ=4\bar R=4)231.561.67393.2
DiT (L=28)192.178.08473.7

Table R3-6. Training efficiency under a 16 GiB allocator cap per GPU.

ModelTraining examples/s ↑Training FLOPs (×10¹⁸)Total GPU-hours (estimated) ↓
R-DiT (Rˉ=1\bar R=1)804.016.75113.2
DiT (L=7)829.714.67109.7
R-DiT (Rˉ=2\bar R=2)421.034.50216.2
DiT (L=14)388.534.19234.3
R-DiT (Rˉ=4\bar R=4)209.270.00435.0
DiT (L=28)137.778.08661.0

As in the above tables, R4 achieves 20.5% higher training throughput at native capacity than the XL baseline (L=28), reducing the total training cost from 473.7 to 393.2 GPU-hours. Under more limited memory cap (16 GiB), using R-DiT (Rˉ=4\bar R=4) has training throughput advantage increases to 52.0% compared to the XL model, corresponding to 34.2% fewer projected GPU-hours for the same number of updates.

AuthorsOpenReview ↗

Response to Reviewer UzQq (3/4)

R3-3. Efficiency analysis (continued)

(d) Recursion improves quality-compute-memory trade-offs.

With backbone depth (L) and width fixed, R-DiT decouples the compute allocation between inner neural compute (by increasing Rˉ\bar R) and number of the denoising steps (T) and we argue this improves quality-compute-memory trade-offs in diffusion models.

Table R3-7. quality–compute trade-off under the same L=7.

TRˉT\bar RMethodTTRˉ\bar RParamsTFLOPs ↓FID ↓
100DiT1001173M11.84.33
100R-DiT502186M12.12.87
150DiT1501173M17.74.58
150R-DiT503186M18.32.66
200DiT2001173M23.64.75
200R-DiT504186M24.52.55

At approximately matched inference FLOPs, R-DiT reduces FID by 33.7–46.3%, with 7.5% more parameters. Moreover, T=50T=50, Rˉ=2\bar R=2 achieves better FID than the evaluated T=200T=200 DiT baseline using 48.7% fewer inference FLOPs. These results support allocating computation to recursive refinement under a constrained parameter budget.

Table R3-8. Quality–compute trade-off at XL scale (L=28).

Rˉ\bar RTTDenoising TFLOPs ↓FID ↓
1250118.22.08
210094.21.93
2250236.81.92
3100141.31.90

Increasing recursion improves the observed quality–compute trade-off: (Rˉ,T)=(2,100)(\bar R,T)=(2,100) achieves lower FID than (Rˉ,T)=(1,250)(\bar R,T)=(1,250) with approximately 20% less estimated computation, while (Rˉ,T)=(3,100)(\bar R,T)=(3,100) improves upon (Rˉ,T)=(2,250)(\bar R,T)=(2,250) with approximately 40% less compute.

The above results together clearly indicate that conventional DiT can be suboptimal under the compute-memory-performance trade-off. Therefore, R-DiT can be also compute-efficient in a sense that it achieves better performance with the smaller compute FLOPs required for the sampling using the same number of parameters.


R3-3. Comparison with other time-step dependent compute allocation methods

Thank you for the constructive feedback. We agree that adaptive computation across denoising steps is well established, and we will expand our discussion of these related works. Existing approaches adjust computation by changing the placement of sampling evaluations (EDM [G], AYS[H]), switching between models of different sizes (OMS-DPM[A], T-Stitch[C]), or selectively activating parts of a larger network (DDSM[B], ASE[D], AdaDiff[E], DyDiT[F]).

Compared with these approaches, R-DiT recursively refines its internal features by repeatedly applying the same compact backbone at a fixed noise level, before producing a denoising prediction. This increases effective processing depth (shallower or deeper thinking depends on the timestep) without stacking more transformer blocks.

To quantitatively compare the difference between focusing more on timesteps and adding more neural compute, we run two additional baselines: (1) more fine-grained timestep evaluations matching the number of recursion in that timestep regime, (2) T-Stitch-style [C] depth allocation using separately trained 7/14/21-layer denoisers.

Table R3-9. Comparison with other time-adaptive compute allocation methods.

MethodCoarse TLoaded params (M) ↓TFLOPs/image ↓FID ↓sFID ↓IS ↑Precision ↑Recall ↑
R-DiT, Rˉ=2\bar R=2100186.5¹24.52.995.34243.50.7670.579
R-DiT, Rˉ=4\bar R=450186.5¹24.52.555.07243.7.0.7940.577
Adaptive sampling, R1100173.223.54.767.11210.50.7340.554
T-Stitch-style [C], 7→14→21 layers1001,021.623.52.665.07286.20.8010.567

The result shows that putting more neural compute for one accurate denoising step is often more important than simply increasing the denoising timesteps, while R-DiT avoids training / loading models with different sizes achieving smaller peak-memory during the inference compared to T-stitch [C].

Reviewer_gnCWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes R-DiT, which introduces recursive depth as an additional scaling axis for diffusion transformers. By reusing shared Transformer blocks, R-DiT increases effective computational depth without proportionally increasing model parameters. The paper also introduces recursion-aware modulation and time-adaptive recursion to improve recursive computation. Experiments on image, language, and molecular diffusion demonstrate improvements in generation quality and parameter efficiency.

Critical strengths
  1. Promising parameter efficiency. R-DiT achieves comparable performance to deeper and larger DiTs with significantly fewer parameters. For example, it achieves an FID of 2.55 with 186M parameters and 4 steps, compared to 2.47 with 674M parameters for a deeper and larger baseline under approximately matched inference FLOPs.
  2. Solid Experiments. The paper evaluates recursive scaling across different model capacities and multiple generative domains. The ablation studies also demonstrate the effectiveness of recursion-aware modulation and time-adaptive computation allocation.
Critical weaknesses
  1. Limited novelty and insufficient comparison with related work. Recursive weight sharing in diffusion transformers has already been explored in prior work, particularly ELT. While the proposed modulation and adaptive recursion provide additional contributions, the paper lacks a direct quantitative comparison with ELT under matched settings. This makes it difficult to assess the improvements attributable to the proposed design.
  2. Incomplete efficiency evaluation. The paper mainly reports parameter counts and inference FLOPs, without sufficiently analyzing training costs, inference latency, or memory consumption. Since increasing recursion depth also increases training computation, the overall efficiency advantage remains unclear.
Questions for authors
  1. Could the authors provide a direct comparison with ELT under matched training and inference budgets, and clarify the key architectural differences?
  2. Could the authors report training FLOPs, inference latency, and peak GPU memory for R-DiT and the corresponding deeper DiT baselines?
4 comments
AuthorsOpenReview ↗

Response to Reviewer gnCW (1/2)

We sincerely thank the reviewer for the careful assessment and for recognizing the importance of one-step generation, the motivation for dense early supervision, and the originality of the forward drift. We address the reviewer's concern in the following.


R2-1. Comparison with ELT [1]

Thank you for the careful review.

(a) Implementation and architectural differences.

While both ELT and R-DiT reuse shared transformer layers to refine hidden features ELT directly passes each loop’s features into the next loop and trains a randomly sampled intermediate prediction using both ground-truth supervision and self-distillation from the final prediction. R-DiT additionally conditions each loop on its recursion index and uses a learned controller to gate the update to the persistent hidden state. Therefore, R-DiT supervises only the final prediction. Moreover, R-DiT allocates different computes per denoising timestep thorugh adaptive-recursion

(b) Qualitative comparison between R-DiT and ELT

For direct comparison, we reproduce ELT follwoing [2] by uniformly samples one intermediate exit s∈1,…,R−1s\in{1,\ldots,R-1} per microbatch and optimizes with the loss objectives:

$\mathcal L_{\mathrm{GT}}(v_R) +\lambda_n\mathcal L_{\mathrm{GT}}(v_s) +(1-\lambda_n)\operatorname{MSE}\left(v_s,\operatorname{sg}(v_R)\right)$

Here, (λn\lambda_n) linearly decrease from one to zero over 80K updates. Student and teacher share the same forward trajectory and adapt the prediction objective to our VA-VAE flow-matching recipe, using velocity MSE plus cosine loss for ground-truth supervision.

Table R2-1. Comaparison with the ELT.

ModelL (num_layers)R (recursion)ParametersFID ↓IS ↑
ELT [2]72173M4.37167.8
R-DiT (ours)72186M3.39196.7
ELT [2]74173M3.21186.7
R-DiT (ours)74186M2.05239.2
R-DiT (ours)64163M3.05196.5

At matched backbone depth and recursion count, R-DiT reduces FID by 22.4% at R=2 and 36.1% at R=4, while also improving IS.

The seven-layer R-DiT has more parameters than ELT (186M versus 173M). However, the six-layer R-DiT also achieves lower FID (3.05 versus 3.21) and higher IS (196.5 versus 186.7) while using 5.8% fewer parameters (163M versus 173M). Contribution-focused replacement

These results highlight the benefits of our design choices in practical performance compared to other choices made in [2].

We will update our manuscript to directly incorporate the result.


R2-2. Efficiency anlaysis

We thank the reviwer for the insightful feedback. Here, we provide efficiency analysis of R-DiT comparing with the original DiT.

(a) Peak memory and batch-1 inference latency

Table R2-2. Peak memory and latency comparison.

ModelInference peak memory (MiB)Training peak memory (GiB)Inference latency / step (ms)
R-DiT (Rˉ=1\bar R=1)748.04.405.01
DiT (L=7)694.24.304.91
R-DiT (Rˉ=2\bar R=2)755.35.599.44
DiT (L=14)1,332.78.359.20
R-DiT (Rˉ=4\bar R=4)755.37.6318.21
DiT (L=28)2,609.616.4417.69

At effective depth 28, R-DiT reduces peak inference memory by 71.1% and peak training memory by 53.6%, despite 2.9% higher per-step inference latency than L28 (due to the recursion modulation).

(b) Inference throughput gain under constrained memory.

Table R2-3. Throughput gain under the memory-budegeted scenario.

Cap (GiB)R4 batchXL batchR4 requests/sXL requests/sR4 gain
2.5064—2.222Cannot run (OOM)—
2.5572—2.243Cannot run (OOM)—
2.607232.2361.685+32.8%
2.657242.2381.880+19.0%
2.708082.2392.049+9.3%
2.7572112.2392.117+5.8%
2.8080132.2362.145+4.3%

We independently select each model’s highest-throughput tested batch under the same memory cap. R-DiT’s lower parameter memory allows larger batches, improving throughput by 32.8% at 2.60 GiB. As the budget increases, XL can also use larger batches, narrowing the advantage to 4.3% at 2.80 GiB.

Therefore, R-DiT can provide practical throughput gains in the memory-restricted environments.

AuthorsOpenReview ↗

Response to Reviewer gnCW (2/2)

R2-2. Efficiency anlaysis (continued)

(c) Training thorughput analysis.

We benchmarked real training updates on 4 RTX 3090 GPUs both at native capacity and under a 16 GiB allocator cap per GPU. To avoid OOM, L28 use full checkpointing in native capacity (24 GiB) experiment and for the 16 GiB cap, R-DiT R1/R2/R4 and DiT L14 use partial checkpointing, while L28 uses full checkpointing.

Table R2-4. Training efficiency comparison.

ModelTraining examples/s ↑Training FLOPs (×10¹⁸)Total GPU-hours (estimated) ↓
R-DiT (Rˉ=1\bar R=1)879.814.67103.5
DiT (L=7)889.414.67102.3
R-DiT (Rˉ=2\bar R=2)463.230.34196.5
DiT (L=14)422.529.33215.4
R-DiT (Rˉ=4\bar R=4)231.561.67393.2
DiT (L=28)192.178.08473.7

Table R2-5. Training efficiency under memory cap (16 GiB per GPU).

ModelTraining examples/s ↑Training FLOPs (×10¹⁸)Total GPU-hours (estimated) ↓
R-DiT (Rˉ=1\bar R=1)804.016.75113.2
DiT (L=7)829.714.67109.7
R-DiT (Rˉ=2\bar R=2)421.034.50216.2
DiT (L=14)388.534.19234.3
R-DiT (Rˉ=4\bar R=4)209.270.00435.0
DiT (L=28)137.778.08661.0

As in the above tables, R4 achieves 20.5% higher training throughput at native capacity than the XL baseline (L=28), reducing the total training cost from 473.7 to 393.2 GPU-hours. Under more limited memory cap (16 GiB), using R-DiT (Rˉ=4\bar R=4) has training throughput advantage increases to 52.0% compared to the XL model, corresponding to 34.2% fewer projected GPU-hours for the same number of updates.

(d) Quality-compute-memory trade-offs.

With backbone depth (L) and width fixed, R-DiT decouples the compute allocation between inner neural compute (by increasing Rˉ\bar R) and number of the denoising steps (T) and we argue this improves quality-compute-memory trade-offs in diffusion models.

Table R2-6. quality–compute trade-off under the fixed L=7.

TRˉT\bar RMethodTTRˉ\bar RParamsTFLOPs ↓FID ↓
100DiT1001173M11.84.33
100R-DiT502186M12.12.87
150DiT1501173M17.74.58
150R-DiT503186M18.32.66
200DiT2001173M23.64.75
200R-DiT504186M24.52.55

At approximately matched inference FLOPs, R-DiT reduces FID by 33.7–46.3%, with 7.5% more parameters. Moreover, T=50T=50, Rˉ=2\bar R=2 achieves better FID than the evaluated T=200T=200 DiT baseline using 48.7% fewer inference FLOPs. These results support allocating computation to recursive refinement under a constrained parameter budget.

Table R2-7. Quality–compute trade-off at XL scale (L=28).

Rˉ\bar RTTTFLOPs ↓FID ↓
1250118.22.08
210094.21.93
2250236.81.92
3100141.31.90

Increasing recursion improves the observed quality–compute trade-off: (Rˉ,T)=(2,100)(\bar R,T)=(2,100) achieves lower FID than (Rˉ,T)=(1,250)(\bar R,T)=(1,250) with approximately 20% less estimated computation, while (Rˉ,T)=(3,100)(\bar R,T)=(3,100) improves upon (Rˉ,T)=(2,250)(\bar R,T)=(2,250) with approximately 40% less compute.

The above results together shows that conventional R-DiT pushs the pareto frontier of compute-memory-performance trade-off. In this sense, R-DiT can be also compute-efficient since it achieves better performance with the smaller compute FLOPs required for the sampling given the same parameter budget.


[1] (Goyal et al., 2026) ELT: Elastic Looped Transformers for Visual Generation.


We sincerely thank the reviewer for the constructive feedback. We are happy to provide further details if there are any remaining points we should clarify.

Reviewer_gnCWOpenReview ↗

Response to Authors' Rebuttal

Thank you for the detailed rebuttal and additional experimental results. All my concerns have been fully addressed. I will maintain my positive assessment and recommend this paper for acceptance.

We are glad to hear that reviewer's concern has been fully resolved. We sincerely thank the reviewer again for the helpful comments.

Reviewer_CgRhOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes recursion depth (inspired from looped transformers) as a third scaling axis for DiTs (in addition to sampling and model depth), and proposes a modified architecture of the standard DiTs called R-DiT. The method's design is based on two key components: the (1) recursion modulation (i.e., to control how much the hidden representation changes across the DiT passes), and the (2) time-adaptive recursion (i.e., that allows for different recursive computation depending on the diffusion step). The method has been tested on image, language, and molecule synthesis tasks and the experiments highlight that the method improves the trade-off between the sample quality and sampling depth / model size to vanilla DiT.

Critical strengths
  • The biggest strength is how well this paper has been written: it's very easy to read, the ideas are clear, the paper sections are well-connected, and the story that always discusses about these three distinct compute axes flows naturally.
  • The method has been not only tested on images, but also on text and molecules.
  • The design choices have been backed-up by experiments (Figure 5 + Table 3).
  • The experiments are well-motivated and they do answer the four (good) research questions mentioned in the paper.
Critical weaknesses
  • In Table 1, could you comment on why the FID score is increasing when T increases? If that's the case, then we would not be able to talk sampling depth as a scaling axis, right? You also mentioned that the baseline is "CFG-tuned" line 323, page 6 - were all results from Table 1 also CFG-tuned?
    • An additional question to the previous point: could you mention what CFG you used in each experiment / table / figure?
  • Maybe I missed this, but did you use time-adaptive scheduling in all experiments? Could you mention where you used a fixed R in for all timesteps t?
  • Did you make an analysis regarding peak inference memory / latency / throughput of R-DiT vs. vanilla DiT? Since the paper presents recursion depth as an additional scaling axis (and highlights the drawbacks of the other two), I believe having a small paragraph / section that explains its (obvious limitations) is welcome.
  • Very small details / paper inconsistencies:
    • "recursive depth" (e.g., in the abstract) vs. "recursion depth" (e.g., in the caption of Figure 1) -> i'd stick to only one term throughout the paper.
    • Eq. 3-4 and Fig5d equations show the update rule with different notations.
Questions for authors

See weaknesses section.

5 comments
AuthorsOpenReview ↗

Response to Reviewer CgRh (1/3)

We sincerely thank the reviewer for the thoughtful assessment and for recognizing the clarity of the presentation, the evaluation across multiple modalities, and the experimental support for our design choices. We address the reviewers questions below.


R1-1. Increasing sampling depth and the classifier-free guidance settings.

Thank you for the careful review. By “sampling depth as a scaling axis,” we mean increasing inference computation through additional denoiser evaluations, rather than assuming that FID must improve monotonically at every sampling budget. Previous work similarly treats denoising-step count as an inference-compute axis while observing diminishing returns beyond a certain number of evaluations [1]-[2].

Regarding the CFG values, our main result in Table 1 is based on the CFG-tuning while we follow other setups the same as in the original sampling protocol as in [1]. Here, we tuned CFG values for models with different number of layers (L) by generating 1,024 samples (with Rˉ=1\bar R=1) choosing the value with the lowest FID scores. As a result, we chose CFG weights of w=20,15,12.5,10w=20,15, 12.5, 10 for the L=7,14,21,28 layers respectively (L=7, Rˉ=1,2,3,4\bar R=1,2,3,4 results are all evaluated with the CFG weight of 20).

Here, we further provide CFG result which was originally tuned for the XL (L=28) models in the original Lightning DiT [4] baseline.

Table R1-1. Fixed CFG results (CFG weight: 10).

LTRˉ\bar RFID ↓IS ↑sFID ↓Precision ↑Recall ↑
75016.45147.76.430.720.58
710016.11151.85.640.730.58
715016.30150.85.770.730.59
720016.43150.15.930.720.59
725016.47150.26.040.720.59
145013.29199.55.770.780.58
215012.66232.85.510.800.57
285012.47247.85.460.810.58
75023.61193.65.680.780.57
75033.11220.65.820.800.55
75042.98232.76.020.800.55

The result shows that the benfit of increasing the recursion depth (R) persists even without specific CFG-tuning.

Other than Table 1, we used CFG=1.5 for the SD-VAE expeirment (Fig.4-(a), Table 2) following the original DiT [3]. For Fig. 4-(b), we used CFG=10.0 following the sampling protocol of its corresponding VA-VAE Lightning DiT [4] baseline.

We have updated our manuscript to explicitly state this evaluation protocol for clearness.


R1-2. Where fixed and time-adaptive recursion are used

Thank you for pointing out this.

We have used fixed recursion for the result in Fig.4-(a) for the scaling experiment with original SD-VAE setup and Fig.6 for other domain experiments as described in Appendix D.4-5, while providing direct comaprison with the adaptive recursion for molecules in Appendix D.5.

We have updated our manuscript to explicitly distinguish fixed (R) from average depth (Rˉ\bar R) consistently and label the schedule directly in each table and figure caption to avoid any confusion.


R1-3. Efficiency anlaysis

From the constructive feedback of the reviewer, we provide efficiency analysis of R-DiT in different aspects as follows.

(a) Peak memory and batch-1 inference latency

Table R1-2. Peak allocated memory and batch-1 inference latency.

ModelInference peak memory (MiB)Training peak memory (GiB)Inference latency / step (ms)
R-DiT (R=1)748.04.405.01
DiT (L=7)694.24.304.91
R-DiT (R=2)755.35.599.44
DiT (L=14)1,332.78.359.20
R-DiT (R=4)755.37.6318.21
DiT (L=28)2,609.616.4417.69

At effective depth 28, R-DiT reduces peak inference memory by 71.1% and peak training memory by 53.6%, despite 2.9% higher per-step inference latency than L28 (due to the recursion modulation).

AuthorsOpenReview ↗

Response to Reviewer CgRh (2/3)

R1-3. Efficiency anlaysis (continued)

(b) Inference throughput gain under constrained memory.

Table R1-3. Throughput gain under the memory-budegeted scenario.

Cap (GiB)R4 batchXL batchR4 requests/sXL requests/sR4 gain
2.5064—2.222Cannot run (OOM)—
2.5572—2.243Cannot run (OOM)—
2.607232.2361.685+32.8%
2.657242.2381.880+19.0%
2.708082.2392.049+9.3%
2.7572112.2392.117+5.8%
2.8080132.2362.145+4.3%

We independently select each model’s highest-throughput tested batch under the same memory cap. R-DiT’s lower parameter memory allows larger batches, improving throughput by 32.8% at 2.60 GiB. As the budget increases, XL can also use larger batches, narrowing the advantage to 4.3% at 2.80 GiB.

Therefore, R-DiT provides practical throughput gains when available memory restricts the baseline’s batch size.

(c) Training thorughput analysis.

We benchmarked real training updates on four RTX 3090 GPUs using mixed BF16 precision and an effective global batch size of 1,024, both at native capacity and under a 16 GiB allocator cap per GPU. To avoid OOM, in native capacity (24 GiB) experiment, L28 uses checkpointing. Under the 16 GiB cap, R-DiT R1/R2/R4 and DiT L14 use partial checkpointing, while L28 uses full checkpointing. Total GPU-hours are extrapolated estimation from the real training time upto 1K updates.

Table R1-4. Training efficiency comparison.

ModelTraining examples/s ↑Training FLOPs (×10¹⁸)Total GPU-hours (estimated) ↓
R-DiT (Rˉ=1\bar R=1)879.814.67103.5
DiT (L=7)889.414.67102.3
R-DiT (Rˉ=2\bar R=2)463.230.34196.5
DiT (L=14)422.529.33215.4
R-DiT (Rˉ=4\bar R=4)231.561.67393.2
DiT (L=28)192.178.08473.7

Table R1-5. Training efficiency under memory cap (16 GiB per GPU).

ModelTraining examples/s ↑Training FLOPs (×10¹⁸)Total GPU-hours (estimated) ↓
R-DiT (Rˉ=1\bar R=1)804.016.75113.2
DiT (L=7)829.714.67109.7
R-DiT (Rˉ=2\bar R=2)421.034.50216.2
DiT (L=14)388.534.19234.3
R-DiT (Rˉ=4\bar R=4)209.270.00435.0
DiT (L=28)137.778.08661.0

As in the above tables, R4 achieves 20.5% higher training throughput at native capacity than the XL baseline (L=28), reducing the total training cost from 473.7 to 393.2 GPU-hours. Under more limited memory cap (16 GiB), using R-DiT (Rˉ=4\bar R=4) has training throughput advantage increases to 52.0% compared to the XL model, corresponding to 34.2% fewer projected GPU-hours for the same number of updates.

(d) Quality-compute-memory trade-offs.

With backbone depth (L) and width fixed, R-DiT decouples the compute allocation between inner neural compute (by increasing Rˉ\bar R) and number of the denoising steps (T) and we argue this improves quality-compute-memory trade-offs in diffusion models.

Table R1-6. quality–compute trade-off under the fixed L=7.

TRˉT\bar RMethodTTRˉ\bar RParamsTFLOPs ↓FID ↓
100DiT1001173M11.84.33
100R-DiT502186M12.12.87
150DiT1501173M17.74.58
150R-DiT503186M18.32.66
200DiT2001173M23.64.75
200R-DiT504186M24.52.55

At approximately matched inference FLOPs, R-DiT reduces FID by 33.7–46.3%, with 7.5% more parameters. Moreover, T=50T=50, Rˉ=2\bar R=2 achieves better FID than the evaluated T=200T=200 DiT baseline using 48.7% fewer inference FLOPs. These results support allocating computation to recursive refinement under a constrained parameter budget.

Table R1-7. Quality–compute trade-off at XL scale (L=28).

Rˉ\bar RTTDenoising TFLOPs ↓FID ↓
1250118.22.08
210094.21.93
2250236.81.92
3100141.31.90

Increasing recursion improves the observed quality–compute trade-off: (Rˉ,T)=(2,100)(\bar R,T)=(2,100) achieves lower FID than (Rˉ,T)=(1,250)(\bar R,T)=(1,250) with approximately 20% less estimated computation, while (Rˉ,T)=(3,100)(\bar R,T)=(3,100) improves upon (Rˉ,T)=(2,250)(\bar R,T)=(2,250) with approximately 40% less compute.

The above results together shows that conventional R-DiT pushs the pareto frontier of compute-memory-performance trade-off. In this sense, R-DiT can be also compute-efficient since it achieves better performance with the smaller compute FLOPs required for the sampling given the same parameter budget.

AuthorsOpenReview ↗

Response to Reviewer CgRh (3/3)

R1-4. Notations

We appreciate the careful review. As suggested, we changed our manuscript to use “recursion depth” consistently throughout. Also, we have revised the notations used in Equations (3)–(4) aligned with the Figure 5-(d).


[1] (Ma et al., 2025) Inference-time Scaling of Diffusion Models through Classical Search.

[2] (Karras et al., 2022) Elucidating the Design Space of Diffusion-Based Generative Models.

[3] (Peebles et al., 2022) Scalable Diffusion Models with Transformers

[4] (Yao et al., 2025) Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models.


We sincerely thank the reviewer again for these constructive suggestions and will happy to engage further discussion if any concerns remain.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.