acceptodds
Under review as a conference paper at ICLR 2027

StairDrop: Hierarchical Token Dropping for Diffusion Transformers

Abstract

Recent diffusion transformers have demonstrated that substantial training compute can be saved by allowing only a subset of tokens to traverse intermediate blocks while preserving competitive sample quality. Yet existing token-sparse designs usually treat the resulting sparsity pattern as a fixed computational shortcut, leaving open how information should be organized across depth. Inspired by U-Net's progressive contraction and expansion, we introduce Stairdrop, a hierarchical token-dropping schedule that progressively reduces and restores participation, with skip fusion at each restoration and depth allocated across the resulting stages. The hierarchy is used during training; at inference, all tokens traverse the full backbone with the fusion layers retained. Controlled experiments on SiT-B/2 and SiT-XL/2 show that intermediate participation improves generation and that depth allocation matters. On ImageNet with guided sampling, at half of the training compute of the strongest token-dropping system, Stairdrop matches its classifier-free-guidance FID-50K of 1.96, and at equal compute it reaches 1.47 against its best reported 1.55. These results support hierarchical token participation as an effective design for efficient diffusion transformer training.

Primary area:
generative models
Keywords:
Diffusion Models, Generative Models, Flow Matching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wUj9OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StairDrop, a hierarchical token-dropping training method inspired by the progressive contraction and expansion concept of U-Net. During training, token participation varies in a nested pattern of “all → 1/2 → 1/4 → 1/2 → all,” with missing positions restored via skip fusion at each recovery step; during inference, the model remains a complete, dense model. Controlled experiments on SiT-B/2 demonstrate that the intermediate participation levels themselves can improve generation quality. On ImageNet 256×256, StairDrop based on SiT-XL/2 matches SPRINT’s FID of 1.96 with approximately half the training computational cost, achieves 1.47 with comparable computational cost, and can be used in conjunction with REPA.

Critical strengths
  1. Important research direction. Reducing diffusion transformer training cost is a pressing practical problem, and training-time token dropping adds no inference overhead.

  2. Simple design with negligible overhead. The method adds only two linear fusion layers and no extra Transformer blocks, and the deployed model is a standard dense backbone, making it easy to adopt in existing training pipelines.

Critical weaknesses
  1. Limited novelty. The method is essentially SPRINT[1] applied twice in a nested fashion: selection, placeholder restoration, fusion and the two-stage schedule are all inherited, and the only addition is an intermediate participation level. Several of the depth-allocation findings also echo SPRINT's own ablations. The gain from this addition is modest at matched compute, and the headline compute saving is partly confounded by a shorter dense fine-tuning stage than SPRINT's.

  2. Confounded compute comparison. The headline claim of matching SPRINT at half the compute is largely driven by the dense fine-tuning stage rather than by the hierarchy: StairDrop uses 20K dense steps, while the quoted SPRINT rows include 100K, and SPRINT's own Fig. 9 shows that 20K steps already recover over 94% of the gain. A SPRINT baseline with an equally short dense stage is needed to attribute the saving to the proposed design. In addition, Table 3 omits the most comparable SPRINT results (the 200-epoch row and the SPRINT+REPA rows), which further inflates the apparent gap.

  3. Gains not established under matched compute. The key ablation compares variants at matched training steps, yet the staircase processes more tokens per step, so part of its gain may simply reflect extra computation. At matched FLOPs on SiT-XL/2, the paper itself reports only a 0.4–0.6 FID-10K improvement over one-level dropping.

Minor:

  1. I found the font size in figure 1 is too small for reading.

[1] Park et al. Sprint: Sparse-dense residual fusion for efficient diffusion transformers. ICLR 2026

Questions for authors
  1. What compute and FID does SPRINT reach with the same 20K-step dense stage as StairDrop? If the advantage largely disappears, the main efficiency claim would need revision.

  2. Does the gain in Fig. 4A (34.87 to 29.45) persist when the one-level control is trained to the same total FLOPs?

  3. Beyond adding an intermediate level, what do the authors consider the key new insight over SPRINT?

  4. Why are SPRINT’s 200-epoch and SPRINT+REPA rows omitted from Table 3?

7 comments

We thank the reviewer for the detailed comments. We answer the points in the following order: the gain at matched compute, the dense stage and novelty, followed by the remaining questions.

W1 (gain at matched compute):

The gain from this addition is modest at matched compute [...]

Perhaps the size of Table 3 made the main comparison hard to read, so we extract it: all of SPRINT's results without REPA (the SPRINT+REPA rows are in W2 / Q4 below), and MDTv2, the lowest FID in Table 3 apart from SPRINT and StairDrop, as a reference for the scale of progress at this level of FID (ImageNet 256×256, FID-50K). ‡: interval or scheduled guidance; PDG: path-drop guidance, proposed by SPRINT.

MethodGuidanceTraining TFLOPs (10⁶)FID-50KΔFID vs MDTv2
MDTv2-XL/2CFG‡220.01.58reference
SPRINT, 200 epochsCFG43.72.01+0.43
SPRINT, 400 epochsCFG65.11.96+0.38
SPRINT, 400 epochsPDG65.11.62+0.04
SPRINT, 400 epochsPDG‡65.11.55−0.03
StairDropCFG32.81.96+0.38
StairDropCFG‡64.91.47−0.11

SPRINT's two lowest FIDs, 1.62 and 1.55, use path-drop guidance, which pairs a dedicated training strategy with a sampling scheme specific to SPRINT's architecture. Apart from which tokens take part in which blocks, StairDrop trains with the standard objective and no additional training strategy, and, like MDTv2, it samples with classifier-free guidance only. Even against path-drop guidance, StairDrop reaches a lower FID at matched compute, 1.47 against 1.55 (64.9M against 65.1M TFLOPs). With classifier-free guidance on both sides, the like-for-like comparison, StairDrop reaches SPRINT's 1.96 with half of its compute (32.8M against 65.1M).

At this level of FID, these gains are not modest. Progress here comes in small steps: with about 30% of the compute of MDTv2, SPRINT is 0.03 below MDTv2 with path-drop guidance, and StairDrop 0.11 below it with classifier-free guidance (last column). Small differences are also costly: in SPRINT's own rows above, going from 2.01 to 1.96 takes 49% more compute, about 4M TFLOPs for every 0.01 of FID. These gaps are far above sampling noise: over three sampling seeds, our system-level results vary by at most 0.006 (our answer to Reviewer xXX2, Q4). Measured against this scale, the gain of 0.08 at matched compute is substantial.

W1 / W2 / Q1 (dense stage):

[...] the headline compute saving is partly confounded by a shorter dense fine-tuning stage than SPRINT's.

Confounded compute comparison. [...] StairDrop uses 20K dense steps, while the quoted SPRINT rows include 100K, and SPRINT's own Fig. 9 shows that 20K steps already recover over 94% of the gain. A SPRINT baseline with an equally short dense stage is needed to attribute the saving to the proposed design.

What compute and FID does SPRINT reach with the same 20K-step dense stage as StairDrop? If the advantage largely disappears, the main efficiency claim would need revision.

We believe the comparison in our paper is appropriate, as it uses SPRINT's headline results as reported in its Table 3, at the FID and training compute that SPRINT gives. The 20K-step result that the reviewer mentions appears only in Figure 9 of SPRINT (Section 4.5), as a curve without numerical values, so it has to be read from the figure.

Even with the dense stage aligned, the comparison still clearly favours StairDrop. Read from SPRINT's Figure 9, its 400-epoch model reaches FID 1.99 after the same 20K-step dense stage at the same batch size, and the training compute splits as follows (counted as in SPRINT):

MethodSparse stage, TFLOPs (10⁶)Dense stage (20K steps), TFLOPs (10⁶)Total TFLOPs (10⁶)FID-50K
SPRINT, 400 epochs (its Figure 9)40.54.945.41.99
StairDrop27.94.932.81.96

With the same dense stage, SPRINT uses 38% more compute than StairDrop for a higher FID, and the whole difference lies in the sparse stage, where the hierarchy acts. Closing the remaining 0.03 would cost SPRINT still more compute, since below FID 2 even 0.01 is costly, about 4M TFLOPs in SPRINT's own results (W1 above). The comparison at matched total compute, 1.47 against 1.55 at 64.9M against 65.1M TFLOPs (W1 above), already includes the dense stage: it compares complete training runs at the same total budget. We will state the dense-stage length of every row in Table 3.

W1 / Q3 (novelty):

Limited novelty. The method is essentially SPRINT[1] applied twice in a nested fashion: selection, placeholder restoration, fusion and the two-stage schedule are all inherited, and the only addition is an intermediate participation level. [...]

Beyond adding an intermediate level, what do the authors consider the key new insight over SPRINT?

The key insight is that the schedule of token participation along depth is a design dimension of its own, beyond the retention ratio and the restoration mechanism that prior token-dropping methods vary. It comes from our analysis of one-level dropping (Figure 3, Section 3.2): the sparse span works globally while the dense blocks around it work locally, like the bottleneck of a U-Net, but the bottleneck is reached in a single step, without the intermediate scales of a U-Net. This raises a question that one-level methods, SPRINT included, do not pose: can a multi-level schedule of participation along depth train better? Adding one intermediate level on each side of the core is the simplest way to realize such a schedule. The allocation study then asks how to divide the blocks among these levels, in particular between the intermediate and core levels (Figure 4B), a question that one-level designs cannot pose.

Even this simplest multi-level schedule pays off. It improves on one-level dropping in controlled comparisons at matched FLOPs (W3 / Q2 below), its allocation principles carry from SiT-B/2 to SiT-XL/2, and its gain over one-level dropping holds on other backbones and in text-to-image generation (our answer to Reviewer xXX2, W3 / Q2). At the system level, with classifier-free guidance only and without path-drop guidance, it reaches the lowest FID among the systems in Table 3, 1.47 (W1 above). Since it changes only which tokens take part in which blocks, it can also be combined with other general training improvements such as REPA (Section 4.4). These results show the advantage of scheduling token participation along depth.

W3 / Q2:

The key ablation compares variants at matched training steps, yet the staircase processes more tokens per step, so part of its gain may simply reflect extra computation. At matched FLOPs on SiT-XL/2, the paper itself reports only a 0.4–0.6 FID-10K improvement over one-level dropping.

Does the gain in Fig. 4A (34.87 to 29.45) persist when the one-level control is trained to the same total FLOPs?

Following your suggestion, we trained the one-level control of Figure 4A until it had used the training FLOPs of StairDrop at 100K steps (SiT-B/2, FID-10K, ODE-50, no guidance):

ModelStepsFID-10K
One-level control of Figure 4A, same FLOPs104.35K33.45
StairDrop 2/2/4/2/2100K29.45

At matched FLOPs, the control remains well behind StairDrop, so the gain of Figure 4A persists.

On SiT-XL/2, Table 2 is evaluated on the sparse weights, before the dense stage (Table 2 caption); its role in the paper is to carry the allocation principles to the deeper backbone and to choose the XL layout. The training efficiency of StairDrop is established after the full pipeline, at the system level: 1.47 against 1.55 at matched compute, and SPRINT's 1.96 with half of its compute (W1 above). We will report matched-FLOPs results alongside matched steps, and state the role of Table 2 more clearly, in the revised version.

W2 / Q4:

In addition, Table 3 omits the most comparable SPRINT results (the 200-epoch row and the SPRINT+REPA rows), which further inflates the apparent gap.

Why are SPRINT's 200-epoch and SPRINT+REPA rows omitted from Table 3?

The 200-epoch rows are in the table of W1 above; against them, StairDrop is ahead on both axes, 1.96 with 32.8M TFLOPs against 2.01 with 43.7M. The SPRINT+REPA rows are below, with ours:

MethodGuidanceTraining TFLOPs (10⁶)FID-50K
SPRINT + REPA, 200 epochsCFG44.31.87
SPRINT + REPA, 200 epochsPDG44.31.61
SPRINT + REPA, 400 epochsPDG66.71.59
SPRINT + REPA, 400 epochsPDG‡66.71.49
StairDropCFG32.81.96
StairDropCFG‡32.81.62
StairDropCFG‡64.91.47
StairDrop + REPACFG19.21.89
StairDrop + REPACFG‡19.21.58

Our REPA experiment (Section 4.4) is meant to show that StairDrop combines with REPA, not to reach the lowest FID with it, so it uses a short schedule of 96 epochs. REPA adds clearly to StairDrop: under the same guidance, FID falls from 1.96 to 1.89 with CFG and from 1.62 to 1.58 with CFG‡, with 19.2M instead of 32.8M TFLOPs (1.62 and 1.89 are new evaluations). With classifier-free guidance only, 1.58 is also below SPRINT+REPA with path-drop guidance (1.61 and 1.59), with 43% and 29% of its compute. We will add all these rows to Table 3.

Minor:

I found the font size in figure 1 is too small for reading.

We will enlarge the fonts of Figure 1.

Thank you for the detailed response. The rebuttal addresses some of the concerns, but I didn't see a revised manuscript to account for these changes. Additionally, I still have some questions:

  1. With the same 20K step of the dense stage (32.8M vs 45.4M TFLOPs), the saving now is about 28%, not half. The current manuscript still claims that it saves half of the training compute in the abstract section, so I think this is overstating the efficiency gain.

  2. At matched compute, StairDrop uses interval CFG to achieve 1.47 of FiD, while Sprint achieves 1.55 using scheduled PDG. The guidance used is not the same, so it is hard to see how much of the ratio of the gain comes from the training and how much comes from the guidance.

  3. Inconsistent control numbers. The "one level, two fusions" control in Fig. 4A and the "no intermediate level" control in your response to Reviewer xXX2 are described identically (64 tokens through blocks 3–10, two fusions), but their numbers differ: FID-10K at 100K steps is 34.87 vs 37.85, and the FLOPs-matched step count is 104.35K vs 116.9K. Your reported GFLOPs (22.6 vs 26.4) imply about 117K steps. Are these the same model, and which numbers are correct?

Finally, I still consider the novelty incremental over SPRINT. This is a concern about the method itself, which I do not think can be resolved within the rebuttal.

Reviewer_efvuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes StairDrop, which extends single-level token dropping with nested reductions and restorations during diffusion transformer training. It studies how intermediate participation levels and block allocation affect generation quality and training efficiency.

Critical strengths

The ablations are well aligned with the proposed contribution: they control for the additional fusion parameters when evaluating intermediate participation and systematically vary depth allocation across stages.

Critical weaknesses

The analysis provides limited guidance for the proposed hierarchical design. The motivating observations do not directly establish why progressive reduction and restoration should help, while the design study focuses mainly on two-level schedules, leaving the benefits and trade-offs of additional levels unclear. Configuration selection also relies on empirical generation-quality comparisons, without an inexpensive profiling metric to guide the choice of hierarchy depth and block allocation.

Questions for authors
  1. Figure 3 characterizes spatial interactions in a one-level model, but how do these observations motivate progressive reduction and restoration specifically? Could a controlled analysis using existing checkpoints test whether intermediate stages improve information transfer or reconstruction, as suggested by the U-Net analogy?
  2. For a fixed overall token-reduction ratio, the hierarchy could use different numbers of stages, unequal reduction ratios at each stage, and different block allocations. What motivates the choice of two successive 2× reductions? Is there evidence that this choice offers a better quality–compute trade-off than alternative decompositions?
  3. Does selecting the hierarchy and block allocation require training and evaluating every candidate, or can inexpensive profiling measurements or early-training signals predict their relative performance? Could the existing configuration sweep be used to assess such indicators and provide practical configuration-selection guidance?
3 comments

We thank the reviewer for the positive assessment and the precise questions. The points raised under weaknesses correspond to the three questions, which we answer below.

Q1:

Figure 3 characterizes spatial interactions in a one-level model, but how do these observations motivate progressive reduction and restoration specifically? Could a controlled analysis using existing checkpoints test whether intermediate stages improve information transfer or reconstruction, as suggested by the U-Net analogy?

As requested, we ran a controlled analysis on the existing checkpoints. As in Figure 3, we restrict a stage's attention to a radius of two patches, now for every stage, and report the share of its unrestricted attention benefit that is kept, a transformer analogue of the effective receptive field [1]; lower values mean a more global stage (t = 0.5, 3 probe seeds):

Blocks1–23–45–89–1011–12
Tokens in StairDrop25612864128256
Dense SiT90%96%92%82%94%
One-level (stages 1–2, 3–9, 10–12)95%44%44%44%91%
StairDrop97%83%43%57%94%

Motivation. In dense SiT every stage works locally. One-level dropping keeps the dense ends local (95%, 91%) but turns the sparse span global (44%) in a single step, a global bottleneck reached without intermediate scales. A U-Net [2] reaches its bottleneck step by step: it has "a contracting path to capture context and a symmetric expanding path that enables precise localization", and each subsampling step enlarges the effective receptive field [1]. A classic design principle likewise holds that the representation "should gently decrease from the inputs to the outputs" [3], whereas one-level dropping cuts it to a quarter in one step. Progressive reduction and restoration add the missing intermediate scale.

StairDrop. With two levels, the profile follows the U-Net pattern. The range widens step by step as participation falls (97% → 83% → 43%), the core is the most global stage, and the range narrows again as positions are restored (57% → 94%), as along the expanding path of a U-Net. The blocks at 128 tokens still work locally (83%), so the tokens that continue into the core pass through two local stages before it rather than one. In our understanding, the global core thus works on tokens that already carry the detail of their neighbours, a better balance between global structure and local detail than reaching the core in a single step. We will add the depth profile to Figure 3.

Q2:

For a fixed overall token-reduction ratio, [...] What motivates the choice of two successive 2× reductions? Is there evidence that this choice offers a better quality–compute trade-off than alternative decompositions?

The decomposition follows the U-Net that motivates the design (Q1). A U-Net reduces resolution by the same factor at every step, with 2×2 max pooling at stride 2, and mirrors each step on the way up. We therefore keep the overall reduction of one-level dropping, the 75% drop of SPRINT (256 → 64 tokens), and split it into equal steps with symmetric restorations; two halvings are the smallest such hierarchy beyond one-level dropping, with one intermediate level on each side of the core.

Following your suggestion, we compared alternative decompositions on SiT-B/2, all with the same 12 blocks and the dropped span in blocks 3–9 (100K steps, FID-10K, SDE-250). At the fixed 256 → 64 reduction, we compare one level, unequal ratios and two equal halvings; we also add a deeper level inside the core (32 or 16 tokens):

DecompositionTokens in the dropped spanFID-10K
One level6436.03
Additional 32-token level in the core (blocks 5–6)64 / 32 / 6434.94
Additional 32-token level in the core (blocks 5–7)64 / 32 / 6438.37
Additional 16-token level in the core (blocks 5–7)64 / 16 / 6438.89
Unequal ratios, 4/3× then 3×192 / 64 / 19235.14
Two equal halvings128 / 64 / 12831.95

Two equal halvings give the lowest FID in the table and also cost less per step than the 192-token first level, which supports the choice. A deeper level inside the core improves one-level dropping by at most 1.1 and stays well behind two equal halvings.

These comparisons are on SiT-B/2, where 12 blocks leave room for few levels. How to decompose the reduction on deeper backbones such as SiT-XL/2, where more levels fit, is an interesting question that we hope future work will explore. We will add these decompositions to the appendix.

[1] Luo et al. Understanding the effective receptive field in deep convolutional neural networks. NeurIPS 2016

[2] Ronneberger et al. U-Net: Convolutional networks for biomedical image segmentation. MICCAI 2015

[3] Szegedy et al. Rethinking the inception architecture for computer vision. CVPR 2016

Q3:

Does selecting the hierarchy and block allocation require training and evaluating every candidate, or can inexpensive profiling measurements or early-training signals predict their relative performance? Could the existing configuration sweep be used to assess such indicators and provide practical configuration-selection guidance?

Based on our current results, we believe that not every candidate needs to be trained to the end: the relative performance of the layouts can be estimated in the first half of training, from their FID or from the slope of its decrease.

Before training. The allocation principles of Section 4.1 narrow the candidates to a few layouts; on SiT-XL/2, all three layouts built from them improve on one-level dropping (Table 2).

Early FID. Following your suggestion, we used the existing sweep. The table compares the FID-10K of the eight StairDrop layouts of Figure 4 and Table 6 at 50K steps, half of the training budget, with their FID-10K at 100K steps (ODE-50, rank in parentheses):

StairDrop layout50K steps100K steps
2/2/4/2/257.94 (1)29.45 (1)
2/1/6/1/259.93 (2)30.91 (2)
2/2/3/2/367.88 (6)33.52 (3)
1/2/6/2/162.93 (3)33.74 (4)
3/2/4/2/164.56 (4)35.11 (5)
2/3/2/3/266.04 (5)36.41 (6)
2/2/3/2/3, 192-token first level72.95 (7)36.93 (7)
1/2/4/2/378.08 (8)38.48 (8)

The two rankings agree with Spearman ρ = 0.86, and the two best and the two weakest layouts are the same at both points.

FID slope. On SiT-XL/2, within the first half of training, between 100K and 200K steps, the slope of FID-10K is −13.4 per 100K steps for 3/3/16/3/3, −11.2 for 4/6/8/6/4 and −7.3 for 2/2/20/2/2. Ordered by steepness, this is the same order as their FID-10K at 400K steps (7.52, 7.68 and 8.08; Appendix, Table 9).

Practical guidance. In practice, one can build a few layouts from the principles, train them for half of the budget, keep the few with the lowest FID or the steepest decrease, and choose among them under the intended sampling setting. We will add this analysis to the appendix. How to choose the optimal layout directly is an interesting question that we leave to future work.

The additional experiments have adequately addressed my concerns. The experiment for Q2 does not fully match what I intended, which was to distribute downsampling across multiple stages while keeping the overall downsampling factor fixed. Nevertheless, it provides sufficient justification for the design. I will maintain my positive score.

Reviewer_xXX2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes StairDrop, a hierarchical token participation strategy for improving the training efficiency of Diffusion Transformers (DiTs). Existing token-dropping methods, such as TREAD and SPRINT, typically employ a single sparse span in which only a subset of tokens participates in computation. The authors argue that prior work has primarily treated token sparsity as a computational shortcut, paying relatively little attention to how token participation should be structured across network depth. Motivated by the contraction-expansion hierarchy of U-Net architectures, StairDrop introduces a multilevel token participation schedule (100% → 50% → 25% → 50% → 100%), together with hierarchical skip fusion during token restoration. This hierarchy is applied only during training; at inference time, all tokens traverse the full backbone, with only lightweight fusion layers retained. The paper evaluates StairDrop through a motivation study examining the relationship between token-dropping spans and U-Net bottlenecks (Figure 3), controlled ablations of intermediate participation levels (Figure 4), scaling experiments on SiT-XL/2 (Table 2), system-level efficiency comparisons (Table 3), and compatibility experiments with REPA. Overall, the paper presents a compelling approach to efficient diffusion training. Its main contribution is the introduction of network-depth-aware token participation as an additional design dimension, supported by extensive experiments that demonstrate improvements in both computational efficiency and generation quality.

Critical strengths
  1. A novel perspective on token sparsity The key contribution of StairDrop is not simply another token-dropping mechanism, but a shift in perspective: how should token participation be organized across network depth? Existing approaches, including MaskDiT, MDT, TREAD, and SPRINT, primarily focus on the token retention ratio, the implementation of sparse computation, or the restoration of skipped tokens. StairDrop instead treats token participation as a hierarchical representation organization problem. This is a meaningful conceptual distinction from conventional token-dropping methods and provides a promising direction for further research into efficient diffusion architectures.
  2. A particularly convincing controlled ablation Figure 4A provides the strongest evidence for the proposed design. The authors compare three configurations:
  • SPRINT-style one-level dropping: FID-10K = 39.25.
  • One-level control with the same fusion positions and additional parameters as StairDrop: FID-10K = 34.87.
  • StairDrop with a 2/2/4/2/2 block allocation: FID-10K = 29.45.

The second baseline is particularly important because it incorporates StairDrop's additional fusion structure. The further improvement from 34.87 to 29.45 therefore provides evidence that hierarchical token participation contributes beyond the effects of additional parameters and fusion components. This controlled comparison substantially strengthens the paper's central claim and makes the experimental evidence more convincing than a comparison against a conventional baseline alone.

  1. Systematic analysis of hierarchical depth allocation Figure 4 provides more than a simple validation of StairDrop; it offers useful insights into the allocation of network depth across participation levels. The best configuration, 2/2/4/2/2, achieves an FID-10K of 29.45, outperforming the following alternatives. These results suggest several useful design principles: the sparse core should have sufficient depth, intermediate stages should not be excessively shallow, and the dense stages at the two ends should remain reasonably balanced. Although these observations may require further validation across architectures, they provide actionable guidance for configuring hierarchical token participation and make the paper more valuable than a purely empirical method proposal.

  2. Benefits persist at larger model scales The results on SiT-XL/2 in Table 2 provide evidence that StairDrop's benefits extend beyond the smaller-scale setting. The one-level baseline achieves an FID-10K of 8.75, while StairDrop configurations achieve 8.08 (2/2/20/2/2), 7.52 (3/3/16/3/3), and 7.68 (4/6/8/6/4). Notably, the 2/2/20/2/2 configuration already improves performance over the one-level baseline at every reported training stage. This result supports the hypothesis that introducing hierarchical participation can be beneficial even without adopting a more elaborate depth allocation. It also strengthens the case that the observed gains are not limited to the smaller model setting.

  3. Strong system-level efficiency and quality improvements Table 3 demonstrates the practical relevance of StairDrop under different comparison settings. At comparable generation quality, SPRINT with CFG achieves an FID of 1.96 at 65.1M TFLOPs, whereas StairDrop achieves the same FID at 32.8M TFLOPs, representing approximately a 50% reduction in reported compute. At comparable compute, SPRINT with PDG achieves an FID of 1.55 at 65.1M TFLOPs, while StairDrop achieves an improved FID of 1.47 at 64.9M TFLOPs. Furthermore, StairDrop combined with REPA achieves an FID of 1.58 using only 19.2M TFLOPs. Taken together, these results suggest that StairDrop offers meaningful practical benefits rather than merely marginal improvements in an isolated benchmark. The reported compatibility with REPA further indicates that the method can complement existing techniques for improving diffusion training.

Critical weaknesses
  1. The mechanistic explanation is less conclusive than the empirical evidence The paper motivates StairDrop through an analogy between token-dropping spans and the bottleneck structure of U-Net architectures. Figure 3 provides suggestive evidence for this motivation. At t=0.5t=0.5, dense blocks retain approximately 87% of the unrestricted attention benefit at radius 2, whereas sparse spans retain only about 44%. Following token restoration, radius-1 access recovers approximately 96% of the output benefit. These observations support the intuition that sparse spans and dense stages play different roles in information processing. However, they do not fully establish that U-Net-style hierarchical organization is the causal mechanism responsible for StairDrop's performance gains. In particular, the connection between the observed attention behavior and the benefits of the proposed multilevel participation schedule remains insufficiently explained. Consequently, the empirical effectiveness of StairDrop is more convincingly demonstrated than the underlying mechanism. Further analysis of information flow, representation quality, or the specific contribution of each participation level would strengthen the paper's conceptual claims.
  2. The comparison with hierarchical diffusion architectures is incomplete The paper discusses hierarchical diffusion architectures, including U-DiT, HDiT, and UDT, in the related work section, but provides limited empirical comparisons with these approaches. An important open question is whether the observed improvements arise specifically from hierarchical token participation or whether similar gains could be obtained by introducing hierarchical structures into diffusion transformers more generally. The current experiments do not fully disentangle these possibilities. A more direct comparison with relevant hierarchical diffusion architectures, or an additional control experiment isolating token participation from architectural hierarchy, would help clarify the novelty and specific contribution of StairDrop.
  3. Generalization beyond SiT remains unclear All experiments are conducted on SiT-B/2 and SiT-XL/2 using ImageNet-256. While the results within this setting are extensive and encouraging, the generality of StairDrop remains unverified across other diffusion architectures and application settings. In particular, it is unclear whether the proposed strategy would transfer effectively to conventional DiT variants, larger-scale latent diffusion models, or text-to-image systems such as FLUX. These settings may exhibit different architectural characteristics and computational bottlenecks, potentially affecting the effectiveness of hierarchical token participation. Although this limitation does not undermine the demonstrated results, additional experiments across architectures or model families would strengthen the broader applicability claims.
Questions for authors

Q1. Mechanistic understanding of hierarchical token participation The motivation study suggests that sparse token spans are more dependent on long-range interactions than surrounding dense stages. For example, under a radius-2 restriction, sparse spans retain approximately 44% of the unrestricted attention benefit, compared with approximately 87% for dense blocks. Can the authors provide additional evidence that these differences in interaction patterns causally contribute to StairDrop's performance gains, rather than merely being correlated with them? For example, would targeted interventions on attention range or token interactions help isolate the role of hierarchical participation? Q2. Generality of the depth-allocation principles The best SiT-B/2 configuration (2/2/4/2/2) substantially outperforms alternative depth allocations, including 2/1/6/1/2 and 2/3/2/3/2. Do these results suggest general principles for allocating depth across participation levels, or are the observed preferences specific to the 12-block SiT-B/2 architecture? Could the authors clarify how these principles might transfer to deeper models or different diffusion transformer architectures? Q3. Interaction between depth allocation and classifier-free guidance In Table 2, the relative performance of 3/3/16/3/3 and 4/6/8/6/4 changes depending on the sampling setting. Under unguided evaluation, 3/3/16/3/3 performs better (FID-10K 7.52 vs. 7.68), whereas under classifier-free guidance, 4/6/8/6/4 performs better (6.77 vs. 6.89). Can the authors provide an explanation for this reversal? In particular, does classifier-free guidance change the relative importance of dense and sparse stages, and could this observation inform the selection of participation schedules for different sampling settings? Q4. Statistical robustness of the reported improvements Table 3 reports substantial improvements over SPRINT under comparable compute budgets. Can the authors provide results across multiple random seeds, or confidence intervals where feasible, to establish the statistical robustness of these improvements? This would help clarify whether the reported differences in generation quality are consistent across runs.

3 comments

We thank the reviewer for the careful reading and the questions, and for recognizing the perspective of organizing token participation along depth, the controlled ablation of Figure 4A, the depth-allocation analysis, the gains on SiT-XL/2 and the system-level efficiency.

W1 / Q1:

The mechanistic explanation is less conclusive than the empirical evidence. [...] Can the authors provide additional evidence that these differences in interaction patterns causally contribute to StairDrop's performance gains, rather than merely being correlated with them?

Regarding the underlying mechanism, following the reviewer's suggestion of targeted interventions on attention range, we applied the intervention of Figure 3 to every stage: we restrict one stage's attention to a radius of two patches and report the share of its unrestricted attention benefit that is kept, a transformer analogue of the effective receptive field [1, 2]. Lower values mean a more global stage (t = 0.5, 3 probe seeds):

Blocks1–23–45–89–1011–12
Tokens in StairDrop25612864128256
Dense SiT90%96%92%82%94%
One-level (stages 1–2, 3–9, 10–12)95%44%44%44%91%
No intermediate level (64 tokens in blocks 3–10)93%67%62%50%94%
StairDrop97%83%43%57%94%

With the intermediate level, the range widens as participation falls (97% → 83% → 43%) and narrows again as positions are restored (57% → 94%). Without it, blocks 3–4 already turn global (67%), and the range stays global through block 10 (62%, 50%).

Our understanding. We see the interaction range as a trade-off. Enlarging it in a single step, as one-level dropping does, favours global structure but can lose local detail. With the intermediate level, the blocks at 128 tokens still work locally, so the tokens that continue into the core first gather the detail of their neighbours, in two local steps rather than one. The core then works globally on tokens that carry more local detail, a better balance between global structure and detail. Removing the intermediate level removes this step-by-step profile and loses the gain (W2). We will add the depth profile to Figure 3.

W2:

The comparison with hierarchical diffusion architectures is incomplete. [...] A more direct comparison with relevant hierarchical diffusion architectures, or an additional control experiment isolating token participation from architectural hierarchy, would help clarify the novelty and specific contribution of StairDrop.

StairDrop changes only which tokens take part in which blocks during training. At inference every position passes through every block, and the only change to the SiT architecture is the two linear fusion layers at the restorations. U-shaped diffusion transformers instead build the hierarchy into the network: HDiT merges neighbouring tokens and splits them again on the way up, U-DiT combines a U-shaped backbone with downsampled self-attention, and UDT merges tokens by feature similarity and later unmerges them. In all three, the down- and up-sampling runs every time the network is evaluated, at every denoising step of generation. StairDrop addresses a different question, how to schedule token participation when training a given network, rather than proposing an alternative network.

Following your suggestion, we isolate token participation from architecture with a control trained for this rebuttal. It has exactly the architecture of StairDrop 2/2/4/2/2, with the same 12 blocks, the same two fusions at the same positions (after blocks 8 and 10) and the same parameters, and lacks only the intermediate participation level, so 64 tokens run through blocks 3–10. Since the control is cheaper per step, we compare at the same steps and at the same training FLOPs (ODE-50, no guidance):

GFLOPs per sampleFID-10K at 100K stepsFID-10K at 0.68M training TFLOPs
Control (no intermediate level)22.637.8533.32 (116.9K steps)
StairDrop 2/2/4/2/226.429.4529.45 (100K steps)

With the architecture held fixed, the gain comes from the participation hierarchy. We will add the control to Figure 4 and state this distinction more clearly in Sections 1 and 2 of the revised version.

[1] Luo et al. Understanding the effective receptive field in deep convolutional neural networks. NeurIPS 2016

[2] Raghu et al. Do vision transformers see like convolutional neural networks? NeurIPS 2021

W3 / Q2:

Generalization beyond SiT remains unclear. [...] it is unclear whether the proposed strategy would transfer effectively to conventional DiT variants, larger-scale latent diffusion models, or text-to-image systems such as FLUX.

Do these results suggest general principles for allocating depth across participation levels, or are the observed preferences specific to the 12-block SiT-B/2 architecture? Could the authors clarify how these principles might transfer to deeper models or different diffusion transformer architectures?

The design study on SiT-B/2 is meant to give principles for allocating depth, rather than a layout to be reused as is. As you correctly summarize in the strengths, the sparse core should have sufficient depth, the intermediate stages should not become too shallow, and the dense stages at the two ends should remain balanced (Section 4.1). We tested how these principles carry over to a deeper model, and whether the gain carries over to other architectures and tasks.

Deeper models. As shown in the paper, the same principles already work on SiT-XL/2, which has 28 blocks instead of 12: the three layouts we built from them without further search, 2/2/20/2/2, 3/3/16/3/3 and 4/6/8/6/4, all improve on one-level dropping (Table 2).

Other architectures and text-to-image. To test generality beyond SiT, we trained two further backbones at B/2 scale and a text-to-image model on MS-COCO, each for 100K steps with one-level dropping and with StairDrop, keeping the 2/2/4/2/2 layout unchanged. Since StairDrop costs about 5% more training FLOPs per step, we also trained one-level dropping until it had used at least the same training FLOPs (FID-10K, ODE-50, no guidance):

SettingDense, 100KOne-level, 100KOne-level, matched FLOPsStairDrop, 100K
ImageNet, SiT70.7239.2532.4529.45
ImageNet, SiT + QK-norm, RoPE, logit-normal sampling53.6824.6323.9119.74
ImageNet, LightningDiT [1]51.1229.8728.9721.64
MS-COCO, text-to-image (SiT-B/2, CLIP text, cross-attention)31.1224.0623.7419.12

Text-to-image systems such as FLUX.1, a 12-billion-parameter transformer, are far beyond the compute available to us during the rebuttal period. The MS-COCO experiment examines the same question in a text-conditioned setting at a scale we can train, and it shows the same ordering as on ImageNet. Applying the hierarchy to large text-to-image systems is a natural next step that we leave for future work.

[1] Yao et al. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models. CVPR 2025

Q3:

Can the authors provide an explanation for this reversal? In particular, does classifier-free guidance change the relative importance of dense and sparse stages, and could this observation inform the selection of participation schedules for different sampling settings?

Our reading is that the reversal follows from the trade-off in our answer to W1, between global structure and local detail. When FID is higher, as without guidance, better global structure brings the larger gain, and 3/3/16/3/3, which puts more depth into the global core, is slightly ahead. Guidance strengthens the class-level global structure and lowers FID; the samples then differ less in structure and more in local detail. 4/6/8/6/4 spends 12 blocks in the intermediate stages against 6 in 3/3/16/3/3, and these stages carry local detail into and out of the core, so it gains more from guidance. In this sense guidance does give more weight to the stages at higher participation, and for guided sampling it favours deeper intermediate stages.

That sparse training interacts with guidance has been noted before, and we follow this view. MaskDiT ends masked training with an unmasking stage; TREAD fine-tunes without routing before guided sampling, citing "the well-documented phenomenon that restricting information during training can degrade performance under CFG"; and SPRINT observes that before its dense stage the model "tends to miss some high-frequency details". The guided column of Table 2 is read on the sparse weights, before the dense stage, so it serves as a reference for choosing the layout, and the full comparison is the one after the dense stage. The layout it favoured, 4/6/8/6/4, is the one we carried through the full pipeline to the guided system-level results of Table 3. We will add this explanation next to Table 2.

Q4:

Can the authors provide results across multiple random seeds, or confidence intervals where feasible, to establish the statistical robustness of these improvements?

Following your suggestion, we provide confidence intervals over sampling seeds for both system-level models, together with the consistency of the gain across separately trained models. Repeating the system-level training itself is beyond our compute budget during the rebuttal (about 230 H200 GPU-hours for the 1.47 row and 120 for the 1.96 row).

Sampling seeds. We re-sampled both system-level models with two further seeds, using 50K samples and the same sampler, guidance and batch size; seed 0 is the reported set (FID-50K):

Model (Table 3)Seed 0 (reported)Seed 1Seed 2Mean ± std95% CI
StairDrop, 32.8M TFLOPs1.9621.9561.9561.958 ± 0.003[1.949, 1.967]
StairDrop, 64.9M TFLOPs1.4721.4711.4701.471 ± 0.001[1.469, 1.473]

The spread is far below the differences we report. We will add these intervals to Table 3.

Consistency across separately trained models. The improvement over one-level dropping holds in every comparison we trained: for all three layouts on SiT-XL/2 at 400K steps (Table 2), and on three ImageNet backbones and MS-COCO at B/2, also at matched training FLOPs (W3 / Q2). A gain that appears in each of these separately trained models is unlikely to come from a favourable seed.

Comments on the paper

1 comment

Dear Reviewers,

The authors have provided their rebuttal, but no follow-up comments have been posted yet. As the author-reviewer discussion period is ending soon, please review and acknowledge the rebuttal, and engage in further discussion if any concerns remain unresolved.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.