- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes R-DiT, a Diffusion Transformer that reuses a small set of shared blocks R times within each denoising step, and presents this "recursion depth" as a third compute-scaling axis alongside sampling depth T and model depth L. To apply such recursion, the authors propose (i) architectural choices and (ii) time-adaptive recursion depth, based on empirical analyses. For (i), the authors apply recursion in hidden states, condition DiTs with the recursion depth , add a modulation layer to be applied at each recursion, and employ a residual structure for each recursion. For (ii), the authors put more recursion at later sampling steps, improving the generation quality under a fixed computational cost. Extensive experiments are conducted over image, language, and molecular domains.
Critical strengths
Motivating analysis. Treating within-step recursion as a scaling axis distinct from T and L is a useful perspective. The analysis in Sec. 2.3 directly motivates both recursion-aware conditioning and time-adaptive allocation.
Extensive experiments across different settings. The authors show various experiments with different modalities, model width, architectural designs, etc. They clearly demonstrate how recursion improves the generation quality under various settings.
Critical weaknesses
Some experimental results seem to be misleading.
- In Table 1, FID worsens monotonically from T=50 (4.18) to T=250 (4.82), which is unusual for a flow-matching model and suggests CFG was tuned only for T=50 or the network is not trained enough.
- In Figure 4 (b), all the baselines except for LightningDiT use a weaker VAE encoder, which is not a fair comparison. Although R-DiT achieves reasonable results with only 64 epochs of training, recursion incurs more computational cost (TFLOPs) and latency, so increasing R with a fixed parameter count is not a fair comparison.
- Recursion does not reduce latency or activation memory, but no wall-clock, memory, or training-cost measurements are given.
Time-adaptive allocation is a known, axis-agnostic technique, yet it is applied only to the recursion axis. Allocating more computation to low-noise steps has been studied extensively. Prior work uses different-size models per step (OMS-DPM [A], DDSM [B], T-Stitch [C]), timestep-dependent early exit or width (ASE [D], AdaDiff [E], DyDiT [F]), and, for the sampling axis, non-uniform step schedules (EDM [G], AYS [H]). None of these works is cited or discussed. More importantly, Table 1 denotes the recursion axis by the average depth , which suggests the time-adaptive schedule is used there, while the depth and sampling baselines use uniform allocation.
[A] Enshu Liu, “OMS-DPM: Optimizing the Model Schedule for Diffusion Probabilistic Models,” ICML 2023.
[B] Shuai Yang, “Denoising Diffusion Step-aware Models,” ICLR 2024.
[C] Zizheng Pan, “T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with Trajectory Stitching,” ICLR 2025.
[D] Taehong Moon, “A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models,” ICML 2024.
[E] Shengkun Tang, “AdaDiff: Accelerating Diffusion Models through Step-Wise Adaptive Computation,” ECCV 2024.
[F] Wangbo Zhao, “Dynamic Diffusion Transformer,” ICLR 2025.
[G] Tero Karras, “Elucidating the Design Space of Diffusion-Based Generative Models,” NeurIPS 2022.
[H] Amirmojtaba Sabour, “Align Your Steps: Optimizing Sampling Schedules in Diffusion Models,” ICML 2024.
Questions for authors
- Was CFG re-tuned for each T in Table 1?
- What are wall-clock latency, peak inference memory, and training cost (FLOPs, GPU-hours) for R=1/2/4 vs. L=7/14/28?
- Does Table 1 use fixed or adaptive recursion? Sec. 3.3 describes a linear schedule, but Table 2 uses piecewise schedules such as (1, 7, 8, 8). Are Table 2 schedules trained separately?
- How is the token-space recursion in Fig. 3 (a) implemented? What intermediate output is fed back (velocity, noise, etc.)?
- How does R-DiT compare empirically with ELT, the closest prior work?
9 comments
Response to Reviewer UzQq (2/4)
R3-3. Efficiency analysis
R-DiT reuses parameters across recursive passes, increasing effective computational depth without proportionally increasing parameter storage. Beside its memory efficiency, we show that compared to increasing the number of layers, increasing recursion depth (R) proportionally can lead to the improvement on training speed and faster inference in memory-budgeted regime. For all measurements, we use RTX 3090 GPUs with FP32 parameter storage and BF16 autocast.
(a) Peak memory and batch-1 inference latency
Table R3-3. Peak allocated memory and batch-1 inference latency with the optimized implementation.
| Model | Inference peak memory (MiB) | Training peak memory (GiB) | Inference latency / step (ms) |
|---|---|---|---|
| R-DiT (R=1) | 748.0 | 4.40 | 5.01 |
| DiT (L=7) | 694.2 | 4.30 | 4.91 |
| R-DiT (R=2) | 755.3 | 5.59 | 9.44 |
| DiT (L=14) | 1,332.7 | 8.35 | 9.20 |
| R-DiT (R=4) | 755.3 | 7.63 | 18.21 |
| DiT (L=28) | 2,609.6 | 16.44 | 17.69 |
At effective depth 28, R-DiT reduces peak inference memory by 71.1% and peak training memory by 53.6%, despite 2.9% higher per-step inference latency than L28 (due to the recursion modulation).
(b) Inference throughput gain under constrained memory.
Table R3-4. Throughput gain under the memory-budegeted scenario.
| Cap (GiB) | R4 batch | XL batch | R4 requests/s | XL requests/s | R4 gain |
|---|---|---|---|---|---|
| 2.50 | 64 | — | 2.222 | Cannot run (OOM) | — |
| 2.55 | 72 | — | 2.243 | Cannot run (OOM) | — |
| 2.60 | 72 | 3 | 2.236 | 1.685 | +32.8% |
| 2.65 | 72 | 4 | 2.238 | 1.880 | +19.0% |
| 2.70 | 80 | 8 | 2.239 | 2.049 | +9.3% |
| 2.75 | 72 | 11 | 2.239 | 2.117 | +5.8% |
| 2.80 | 80 | 13 | 2.236 | 2.145 | +4.3% |
We independently select each model’s highest-throughput tested batch under the same memory cap. R-DiT’s lower parameter memory allows larger batches, improving throughput by 32.8% at 2.60 GiB. As the budget increases, XL can also use larger batches, narrowing the advantage to 4.3% at 2.80 GiB.
These results show that our recursive architecture provides practical throughput gains when available memory restricts the baseline’s batch size.
(c) Parameter sharing can improve training throughput through better memory–compute trade-offs.
We benchmarked real training updates on four RTX 3090 GPUs using mixed BF16 precision and an effective global batch size of 1,024, both at native capacity and under a 16 GiB allocator cap per GPU. To avoid OOM, in native capacity (24 GiB) experiment, L28 uses checkpointing. Under the 16 GiB cap, R-DiT R1/R2/R4 and DiT L14 use partial checkpointing, while L28 uses full checkpointing. Total GPU-hours are extrapolated estimation from the real training time upto 1K updates.
Table R3-5. Training efficiency comparison at native 24 GiB GPU capacity.
| Model | Training examples/s ↑ | Training FLOPs (×10¹⁸) | Total GPU-hours (estimated) ↓ |
|---|---|---|---|
| R-DiT () | 879.8 | 14.67 | 103.5 |
| DiT (L=7) | 889.4 | 14.67 | 102.3 |
| R-DiT () | 463.2 | 30.34 | 196.5 |
| DiT (L=14) | 422.5 | 29.33 | 215.4 |
| R-DiT () | 231.5 | 61.67 | 393.2 |
| DiT (L=28) | 192.1 | 78.08 | 473.7 |
Table R3-6. Training efficiency under a 16 GiB allocator cap per GPU.
| Model | Training examples/s ↑ | Training FLOPs (×10¹⁸) | Total GPU-hours (estimated) ↓ |
|---|---|---|---|
| R-DiT () | 804.0 | 16.75 | 113.2 |
| DiT (L=7) | 829.7 | 14.67 | 109.7 |
| R-DiT () | 421.0 | 34.50 | 216.2 |
| DiT (L=14) | 388.5 | 34.19 | 234.3 |
| R-DiT () | 209.2 | 70.00 | 435.0 |
| DiT (L=28) | 137.7 | 78.08 | 661.0 |
As in the above tables, R4 achieves 20.5% higher training throughput at native capacity than the XL baseline (L=28), reducing the total training cost from 473.7 to 393.2 GPU-hours. Under more limited memory cap (16 GiB), using R-DiT () has training throughput advantage increases to 52.0% compared to the XL model, corresponding to 34.2% fewer projected GPU-hours for the same number of updates.
Response to Reviewer UzQq (3/4)
R3-3. Efficiency analysis (continued)
(d) Recursion improves quality-compute-memory trade-offs.
With backbone depth (L) and width fixed, R-DiT decouples the compute allocation between inner neural compute (by increasing ) and number of the denoising steps (T) and we argue this improves quality-compute-memory trade-offs in diffusion models.
Table R3-7. quality–compute trade-off under the same L=7.
| Method | Params | TFLOPs ↓ | FID ↓ | |||
|---|---|---|---|---|---|---|
| 100 | DiT | 100 | 1 | 173M | 11.8 | 4.33 |
| 100 | R-DiT | 50 | 2 | 186M | 12.1 | 2.87 |
| 150 | DiT | 150 | 1 | 173M | 17.7 | 4.58 |
| 150 | R-DiT | 50 | 3 | 186M | 18.3 | 2.66 |
| 200 | DiT | 200 | 1 | 173M | 23.6 | 4.75 |
| 200 | R-DiT | 50 | 4 | 186M | 24.5 | 2.55 |
At approximately matched inference FLOPs, R-DiT reduces FID by 33.7–46.3%, with 7.5% more parameters. Moreover, , achieves better FID than the evaluated DiT baseline using 48.7% fewer inference FLOPs. These results support allocating computation to recursive refinement under a constrained parameter budget.
Table R3-8. Quality–compute trade-off at XL scale (L=28).
| Denoising TFLOPs ↓ | FID ↓ | ||
|---|---|---|---|
| 1 | 250 | 118.2 | 2.08 |
| 2 | 100 | 94.2 | 1.93 |
| 2 | 250 | 236.8 | 1.92 |
| 3 | 100 | 141.3 | 1.90 |
Increasing recursion improves the observed quality–compute trade-off: achieves lower FID than with approximately 20% less estimated computation, while improves upon with approximately 40% less compute.
The above results together clearly indicate that conventional DiT can be suboptimal under the compute-memory-performance trade-off. Therefore, R-DiT can be also compute-efficient in a sense that it achieves better performance with the smaller compute FLOPs required for the sampling using the same number of parameters.
R3-3. Comparison with other time-step dependent compute allocation methods
Thank you for the constructive feedback. We agree that adaptive computation across denoising steps is well established, and we will expand our discussion of these related works. Existing approaches adjust computation by changing the placement of sampling evaluations (EDM [G], AYS[H]), switching between models of different sizes (OMS-DPM[A], T-Stitch[C]), or selectively activating parts of a larger network (DDSM[B], ASE[D], AdaDiff[E], DyDiT[F]).
Compared with these approaches, R-DiT recursively refines its internal features by repeatedly applying the same compact backbone at a fixed noise level, before producing a denoising prediction. This increases effective processing depth (shallower or deeper thinking depends on the timestep) without stacking more transformer blocks.
To quantitatively compare the difference between focusing more on timesteps and adding more neural compute, we run two additional baselines: (1) more fine-grained timestep evaluations matching the number of recursion in that timestep regime, (2) T-Stitch-style [C] depth allocation using separately trained 7/14/21-layer denoisers.
Table R3-9. Comparison with other time-adaptive compute allocation methods.
| Method | Coarse T | Loaded params (M) ↓ | TFLOPs/image ↓ | FID ↓ | sFID ↓ | IS ↑ | Precision ↑ | Recall ↑ |
|---|---|---|---|---|---|---|---|---|
| R-DiT, | 100 | 186.5¹ | 24.5 | 2.99 | 5.34 | 243.5 | 0.767 | 0.579 |
| R-DiT, | 50 | 186.5¹ | 24.5 | 2.55 | 5.07 | 243.7. | 0.794 | 0.577 |
| Adaptive sampling, R1 | 100 | 173.2 | 23.5 | 4.76 | 7.11 | 210.5 | 0.734 | 0.554 |
| T-Stitch-style [C], 7→14→21 layers | 100 | 1,021.6 | 23.5 | 2.66 | 5.07 | 286.2 | 0.801 | 0.567 |
The result shows that putting more neural compute for one accurate denoising step is often more important than simply increasing the denoising timesteps, while R-DiT avoids training / loading models with different sizes achieving smaller peak-memory during the inference compared to T-stitch [C].
Response to Reviewer UzQq (1/4)
We sincerely thank the reviewer for the detailed assessment and for recognizing the motivation and clear demonstration of our results through various experiments. We address the reviewer's concern in the following.
R3-1. CFG tuning
We thank reviewer for the careful review. Yes, our result in Table 1 is based on the CFG-tuning while we follow other setups the same as in the original sampling protocol as in [1].
For the CFG-tuning, we generate 1,024 samples for the L=7,14,21,28 models (with R=1) and sweep the CFG values and choose the CFG value that results in the lowest FID scores. As a result, we opt CFG=20, 15, 12.5, 10 for the L=7,14,21,28 models accordingly. Accordingly R=1,2,3,4 results (with L=7) are all evaluated with the CFG weight of 20.
For more comparative viewpoint, we have additionally run the sampling under the same fixed CFG value across all models:
Table R3-1. Fixed CFG results (CFG weight: 10).
The result shows that our recursion depth consistently improves the performance compared to increasing the sampling depth even under the fixed CFG scenario.
Moreover, we provide the 50K evaluation result with the independently tuning CFG weights for the different sampling steps (T).
Table R3-2. 50K CFG sweep results under different sampling steps (7 layers, no recursion models).
This shows that merely increasing sampling depth does not improve even under the per sampling step CFG-tuned setup.
R3-2. Implementation details
We appreciate the reviewer for the careful review.
(a) Adaptive recursion in Table 1, 2
Table 1 uses adaptive linear-scheduled recursion where we denote Rˉ as the average recursion numbers during the inference. The ablation conducted in Table 2-(left) uses separte adaptive recursion schedule, independently trained for providing the fair comparison.
(b) Token-space recursion
Our token-space recursion is implemented by passing the output (velocity estimation) in each recursion step and project back to the original noisy latent at the same timestep to estimate the clean latent, then patch-embed that estimate for the next pass through the shared backbone. Only the final velocity prediction is supervised and recursion adds no fresh noise or sampling step.