- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes StairDrop, a hierarchical token-dropping training method inspired by the progressive contraction and expansion concept of U-Net. During training, token participation varies in a nested pattern of “all → 1/2 → 1/4 → 1/2 → all,” with missing positions restored via skip fusion at each recovery step; during inference, the model remains a complete, dense model. Controlled experiments on SiT-B/2 demonstrate that the intermediate participation levels themselves can improve generation quality. On ImageNet 256×256, StairDrop based on SiT-XL/2 matches SPRINT’s FID of 1.96 with approximately half the training computational cost, achieves 1.47 with comparable computational cost, and can be used in conjunction with REPA.
Critical strengths
-
Important research direction. Reducing diffusion transformer training cost is a pressing practical problem, and training-time token dropping adds no inference overhead.
-
Simple design with negligible overhead. The method adds only two linear fusion layers and no extra Transformer blocks, and the deployed model is a standard dense backbone, making it easy to adopt in existing training pipelines.
Critical weaknesses
-
Limited novelty. The method is essentially SPRINT[1] applied twice in a nested fashion: selection, placeholder restoration, fusion and the two-stage schedule are all inherited, and the only addition is an intermediate participation level. Several of the depth-allocation findings also echo SPRINT's own ablations. The gain from this addition is modest at matched compute, and the headline compute saving is partly confounded by a shorter dense fine-tuning stage than SPRINT's.
-
Confounded compute comparison. The headline claim of matching SPRINT at half the compute is largely driven by the dense fine-tuning stage rather than by the hierarchy: StairDrop uses 20K dense steps, while the quoted SPRINT rows include 100K, and SPRINT's own Fig. 9 shows that 20K steps already recover over 94% of the gain. A SPRINT baseline with an equally short dense stage is needed to attribute the saving to the proposed design. In addition, Table 3 omits the most comparable SPRINT results (the 200-epoch row and the SPRINT+REPA rows), which further inflates the apparent gap.
-
Gains not established under matched compute. The key ablation compares variants at matched training steps, yet the staircase processes more tokens per step, so part of its gain may simply reflect extra computation. At matched FLOPs on SiT-XL/2, the paper itself reports only a 0.4–0.6 FID-10K improvement over one-level dropping.
Minor:
- I found the font size in figure 1 is too small for reading.
[1] Park et al. Sprint: Sparse-dense residual fusion for efficient diffusion transformers. ICLR 2026
Questions for authors
-
What compute and FID does SPRINT reach with the same 20K-step dense stage as StairDrop? If the advantage largely disappears, the main efficiency claim would need revision.
-
Does the gain in Fig. 4A (34.87 to 29.45) persist when the one-level control is trained to the same total FLOPs?
-
Beyond adding an intermediate level, what do the authors consider the key new insight over SPRINT?
-
Why are SPRINT’s 200-epoch and SPRINT+REPA rows omitted from Table 3?
7 comments
W1 / Q3 (novelty):
Limited novelty. The method is essentially SPRINT[1] applied twice in a nested fashion: selection, placeholder restoration, fusion and the two-stage schedule are all inherited, and the only addition is an intermediate participation level. [...]
Beyond adding an intermediate level, what do the authors consider the key new insight over SPRINT?
The key insight is that the schedule of token participation along depth is a design dimension of its own, beyond the retention ratio and the restoration mechanism that prior token-dropping methods vary. It comes from our analysis of one-level dropping (Figure 3, Section 3.2): the sparse span works globally while the dense blocks around it work locally, like the bottleneck of a U-Net, but the bottleneck is reached in a single step, without the intermediate scales of a U-Net. This raises a question that one-level methods, SPRINT included, do not pose: can a multi-level schedule of participation along depth train better? Adding one intermediate level on each side of the core is the simplest way to realize such a schedule. The allocation study then asks how to divide the blocks among these levels, in particular between the intermediate and core levels (Figure 4B), a question that one-level designs cannot pose.
Even this simplest multi-level schedule pays off. It improves on one-level dropping in controlled comparisons at matched FLOPs (W3 / Q2 below), its allocation principles carry from SiT-B/2 to SiT-XL/2, and its gain over one-level dropping holds on other backbones and in text-to-image generation (our answer to Reviewer xXX2, W3 / Q2). At the system level, with classifier-free guidance only and without path-drop guidance, it reaches the lowest FID among the systems in Table 3, 1.47 (W1 above). Since it changes only which tokens take part in which blocks, it can also be combined with other general training improvements such as REPA (Section 4.4). These results show the advantage of scheduling token participation along depth.
W3 / Q2:
The key ablation compares variants at matched training steps, yet the staircase processes more tokens per step, so part of its gain may simply reflect extra computation. At matched FLOPs on SiT-XL/2, the paper itself reports only a 0.4–0.6 FID-10K improvement over one-level dropping.
Does the gain in Fig. 4A (34.87 to 29.45) persist when the one-level control is trained to the same total FLOPs?
Following your suggestion, we trained the one-level control of Figure 4A until it had used the training FLOPs of StairDrop at 100K steps (SiT-B/2, FID-10K, ODE-50, no guidance):
| Model | Steps | FID-10K |
|---|---|---|
| One-level control of Figure 4A, same FLOPs | 104.35K | 33.45 |
| StairDrop 2/2/4/2/2 | 100K | 29.45 |
At matched FLOPs, the control remains well behind StairDrop, so the gain of Figure 4A persists.
On SiT-XL/2, Table 2 is evaluated on the sparse weights, before the dense stage (Table 2 caption); its role in the paper is to carry the allocation principles to the deeper backbone and to choose the XL layout. The training efficiency of StairDrop is established after the full pipeline, at the system level: 1.47 against 1.55 at matched compute, and SPRINT's 1.96 with half of its compute (W1 above). We will report matched-FLOPs results alongside matched steps, and state the role of Table 2 more clearly, in the revised version.
W2 / Q4:
In addition, Table 3 omits the most comparable SPRINT results (the 200-epoch row and the SPRINT+REPA rows), which further inflates the apparent gap.
Why are SPRINT's 200-epoch and SPRINT+REPA rows omitted from Table 3?
The 200-epoch rows are in the table of W1 above; against them, StairDrop is ahead on both axes, 1.96 with 32.8M TFLOPs against 2.01 with 43.7M. The SPRINT+REPA rows are below, with ours:
| Method | Guidance | Training TFLOPs (10⁶) | FID-50K |
|---|---|---|---|
| SPRINT + REPA, 200 epochs | CFG | 44.3 | 1.87 |
| SPRINT + REPA, 200 epochs | PDG | 44.3 | 1.61 |
| SPRINT + REPA, 400 epochs | PDG | 66.7 | 1.59 |
| SPRINT + REPA, 400 epochs | PDG‡ | 66.7 | 1.49 |
| StairDrop | CFG | 32.8 | 1.96 |
| StairDrop | CFG‡ | 32.8 | 1.62 |
| StairDrop | CFG‡ | 64.9 | 1.47 |
| StairDrop + REPA | CFG | 19.2 | 1.89 |
| StairDrop + REPA | CFG‡ | 19.2 | 1.58 |
Our REPA experiment (Section 4.4) is meant to show that StairDrop combines with REPA, not to reach the lowest FID with it, so it uses a short schedule of 96 epochs. REPA adds clearly to StairDrop: under the same guidance, FID falls from 1.96 to 1.89 with CFG and from 1.62 to 1.58 with CFG‡, with 19.2M instead of 32.8M TFLOPs (1.62 and 1.89 are new evaluations). With classifier-free guidance only, 1.58 is also below SPRINT+REPA with path-drop guidance (1.61 and 1.59), with 43% and 29% of its compute. We will add all these rows to Table 3.
Minor:
I found the font size in figure 1 is too small for reading.
We will enlarge the fonts of Figure 1.
Thank you for the detailed response. The rebuttal addresses some of the concerns, but I didn't see a revised manuscript to account for these changes. Additionally, I still have some questions:
-
With the same 20K step of the dense stage (32.8M vs 45.4M TFLOPs), the saving now is about 28%, not half. The current manuscript still claims that it saves half of the training compute in the abstract section, so I think this is overstating the efficiency gain.
-
At matched compute, StairDrop uses interval CFG to achieve 1.47 of FiD, while Sprint achieves 1.55 using scheduled PDG. The guidance used is not the same, so it is hard to see how much of the ratio of the gain comes from the training and how much comes from the guidance.
-
Inconsistent control numbers. The "one level, two fusions" control in Fig. 4A and the "no intermediate level" control in your response to Reviewer xXX2 are described identically (64 tokens through blocks 3–10, two fusions), but their numbers differ: FID-10K at 100K steps is 34.87 vs 37.85, and the FLOPs-matched step count is 104.35K vs 116.9K. Your reported GFLOPs (22.6 vs 26.4) imply about 117K steps. Are these the same model, and which numbers are correct?
Finally, I still consider the novelty incremental over SPRINT. This is a concern about the method itself, which I do not think can be resolved within the rebuttal.
We thank the reviewer for the detailed comments. We answer the points in the following order: the gain at matched compute, the dense stage and novelty, followed by the remaining questions.
W1 (gain at matched compute):
Perhaps the size of Table 3 made the main comparison hard to read, so we extract it: all of SPRINT's results without REPA (the SPRINT+REPA rows are in W2 / Q4 below), and MDTv2, the lowest FID in Table 3 apart from SPRINT and StairDrop, as a reference for the scale of progress at this level of FID (ImageNet 256×256, FID-50K). ‡: interval or scheduled guidance; PDG: path-drop guidance, proposed by SPRINT.
SPRINT's two lowest FIDs, 1.62 and 1.55, use path-drop guidance, which pairs a dedicated training strategy with a sampling scheme specific to SPRINT's architecture. Apart from which tokens take part in which blocks, StairDrop trains with the standard objective and no additional training strategy, and, like MDTv2, it samples with classifier-free guidance only. Even against path-drop guidance, StairDrop reaches a lower FID at matched compute, 1.47 against 1.55 (64.9M against 65.1M TFLOPs). With classifier-free guidance on both sides, the like-for-like comparison, StairDrop reaches SPRINT's 1.96 with half of its compute (32.8M against 65.1M).
At this level of FID, these gains are not modest. Progress here comes in small steps: with about 30% of the compute of MDTv2, SPRINT is 0.03 below MDTv2 with path-drop guidance, and StairDrop 0.11 below it with classifier-free guidance (last column). Small differences are also costly: in SPRINT's own rows above, going from 2.01 to 1.96 takes 49% more compute, about 4M TFLOPs for every 0.01 of FID. These gaps are far above sampling noise: over three sampling seeds, our system-level results vary by at most 0.006 (our answer to Reviewer xXX2, Q4). Measured against this scale, the gain of 0.08 at matched compute is substantial.
W1 / W2 / Q1 (dense stage):
We believe the comparison in our paper is appropriate, as it uses SPRINT's headline results as reported in its Table 3, at the FID and training compute that SPRINT gives. The 20K-step result that the reviewer mentions appears only in Figure 9 of SPRINT (Section 4.5), as a curve without numerical values, so it has to be read from the figure.
Even with the dense stage aligned, the comparison still clearly favours StairDrop. Read from SPRINT's Figure 9, its 400-epoch model reaches FID 1.99 after the same 20K-step dense stage at the same batch size, and the training compute splits as follows (counted as in SPRINT):
With the same dense stage, SPRINT uses 38% more compute than StairDrop for a higher FID, and the whole difference lies in the sparse stage, where the hierarchy acts. Closing the remaining 0.03 would cost SPRINT still more compute, since below FID 2 even 0.01 is costly, about 4M TFLOPs in SPRINT's own results (W1 above). The comparison at matched total compute, 1.47 against 1.55 at 64.9M against 65.1M TFLOPs (W1 above), already includes the dense stage: it compares complete training runs at the same total budget. We will state the dense-stage length of every row in Table 3.