acceptodds
Under review as a conference paper at ICLR 2027

CSRe: Training-Free Diffusion Transformer Acceleration via Component-Wise Selective Refresh

Abstract

Diffusion Transformers (DiTs) repeatedly evaluate attention and MLP modules across denoising steps, despite temporal redundancy in their internal features. Feature caching reduces this cost, but fine-grained reuse must account for both refresh allocation and execution overhead. We introduce CSRe (Component-Wise Selective Refresh), a training-free inference framework that selectively recomputes output-projected attention-head and MLP hidden-channel contributions. Full anchor steps refresh the caches and measure changes between anchors; these lagged measurements guide component selection for subsequent selective steps. Selected contributions update the cached module outputs through their latest-cache deltas. A shared compute budget coordinates attention and MLP refresh, with the remaining MLP budget distributed across blocks according to anchor-measured hidden-activation changes. Selected channels are executed using hardware-aligned dense submatrices. We evaluate CSRe on DiT-XL/2, PixArt-α, and FLUX.1-dev. On an RTX 5090D, CSRe maintains competitive generation quality while achieving denoiser speedups of 2.292×, 1.963×, and 1.728× on DiT-XL/2, PixArt-α, and FLUX.1-dev, respectively.

Primary area:
generative models
Keywords:
diffusion transformers, inference acceleration, feature caching, selective refresh, training-free acceleration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_F6MdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes CSRe, which selectively refreshes attention heads and MLP hidden channels based on changes measured at completed anchor steps, while coordinating their compute budgets.

Critical strengths

The matched-compute ablations provide useful evidence that change-guided component selection improves over random selection.

Critical weaknesses

The incremental contribution over closely related methods, particularly Chipmunk and DiTFastAttnV2, is not sufficiently established through discussion or controlled comparisons.

Questions for authors
  1. Could the authors clearly identify the claimed advances over Chipmunk and experimentally demonstrate the benefits of those specific design choices, rather than selective refresh in general?
  2. Could the authors isolate the benefits of coordinated attention–MLP budgeting and adaptive block-wise allocation? At a fixed schedule and compute budget, comparisons against a fixed head/channel budget split and uniform per-block MLP widths would help establish these contributions.
  3. Does component-wise refresh provide a better quality–latency trade-off than allocating the same budget to more frequent full refreshes? Given the competitive or stronger results of coarse caching baselines, a budget sweep on one representative model would help establish when CSRe’s additional complexity is justified.
6 comments
AuthorsOpenReview ↗

Response to Reviewer — W1/Q1

We thank the reviewer for pointing out shortcomings in our comparisons with existing methods and controlled experiments. We address W1 and Q1 jointly because both concern our specific contributions relative to existing methods. Q2 and Q3 retain their original numbering.


W1 / Q1 (joint response): Please clarify the specific improvements over Chipmunk and validate the benefits of these designs, rather than only demonstrating the overall effectiveness of selective refresh.

A1: We agree that comparison with full computation alone cannot distinguish the effects of different selective refresh designs. To evaluate attention separately from MLP, we compare Chipmunk-Attn and CSRe-Attn under approximately matched attention compute budgets. Both use dense MLP computation, with their respective MLP acceleration mechanisms disabled.

(1) Specific differences from existing methods. Chipmunk reuses intermediate results through dynamic column-sparse incremental updates. For attention, it selects a subset of KV interactions for each query group for recomputation and reuses the contributions of the remaining interactions. Its MLP uses a separate mechanism for neuron selection and incremental updates. Evaluating attention and MLP separately is therefore appropriate for both Chipmunk and CSRe. Given the limited time, we first compare their attention mechanisms. CSRe refreshes complete attention heads: at selective steps, we recompute the selected heads' QKV projections, attention over all tokens, and output-projected contributions, while reusing cached contributions for the other heads. The key distinction is whether the budget is spent on a subset of KV interactions within each head or on a subset of complete heads, including their projection computations.

(2) Limitations of Chipmunk on short sequences. Chipmunk's main experiments cover high-resolution image and video generation, where long sequences allow greater reductions in attention computation through sparsification. For sequence length NN and hidden dimension dd, the matrix computations in the attention core scale approximately as O(N2d)O(N^2d), whereas QKV projections, output projections, and MLP computation primarily scale as O(Nd2)O(Nd^2). As sequences become shorter, the attention core accounts for a smaller fraction of total computation; even eliminating a substantial proportion of KV interactions has a smaller effect on the total compute cost. Sparse selection, caching, and memory-access overhead may further offset part of the latency benefit.

At 256×256 resolution, FLUX.1-dev has 256 image and 512 text tokens (768 in total). In Full, QK⊤QK^\top and AVAV, excluding QKV/output projections, account for only 3.99% of total matrix FLOPs. Chipmunk-Attn's official attention configuration reduces the 50-step total from 517.38 to 503.10 TFLOPs/image, saving only 2.76%. To compare the methods at a budget with greater computational savings, we use the Chipmunk paper's 768×1280 resolution: 3840 image and 512 text tokens (4352 in total).

Short sequences also matter in practice: the π₀ action expert processes only 50 action tokens, illustrating the need for efficient iterative inference under real-time control constraints. Chipmunk acknowledges limitations on short sequences, motivating acceleration methods that remain effective in this regime.

(3) Controlled comparison. We evaluate each method independently. Chipmunk's reported gains from combination with ToCa suggest that different optimization granularities can be complementary. CSRe's head- and channel-level reuse may likewise complement token-level methods, but combining them requires joint tuning of caching, budgets, execution order, and hyperparameters beyond our current experiments.

All three configurations are run in the official Chipmunk FLUX framework. Chipmunk-Attn retains the official attention configuration, and we match our method to its resulting compute cost.

(4) Latency and generation quality. Results at 768×1280 are shown below.

MethodMatrix computation (TFLOPs/image) ↓Relative to FullEager denoising latency (s/batch) ↓FID-5K ↓
Full3473.64100.%42.7527.251
Chipmunk-Attn3002.6786.44%39.7827.718
CSRe-Attn3004.4086.49%36.9827.231

At approximately matched compute, CSRe-Attn improves FID by 0.487 and reduces eager denoising latency by 7.04% relative to Chipmunk-Attn. This is the shortest-sequence setting in the Chipmunk paper. CSRe can further reduce computation on short sequences when not constrained to match Chipmunk's budget.


AuthorsOpenReview ↗

Response to Reviewer — Q2

Q2: Can the effects of coordinated attention–MLP budget allocation and adaptive allocation across blocks be validated separately? Under a fixed refresh schedule and compute budget, comparisons are needed against a fixed head/channel budget split and a uniform MLP width across blocks.

A2: We refine the score used for block-wise MLP allocation. Within the existing allocation framework, we replace the original block-change score with the relative input–output difference of the MLP mapping at the current anchor step. This score determines the block-wise MLP widths at subsequent selective steps. Following this refinement, ablation results suggest that coordinated attention–MLP budget allocation and adaptive block-wise MLP allocation each contribute to a favorable quality–latency trade-off, enabling finer-grained control over this trade-off.

Following the paper's notation, ama_m and am−1a_{m-1} denote the current and preceding completed anchors, respectively; pp indexes MLP refresh modules, PP is the total number of these modules, and BcB_c is the number of samples in the conditional batch. The hidden activations recorded at an anchor are denoted by rˉmlp\bar r^{\mathrm{mlp}}, i.e., the features after the activation function and before the second projection (FC2). In the new formula, uu follows the MLP input notation in Equation (7) of the paper and denotes the value after timestep-dependent shift and scale modulation.

Original formula (combined form of Equation (11)):

Dam,p=1Bc∑i=1Bcmean⁡n∥rˉi,am,p,n,:mlp−rˉi,am−1,p,n,:mlp∥1∑r=1Pmean⁡n∥rˉi,am,r,n,:mlp−rˉi,am−1,r,n,:mlp∥1+ϵD_{a_m,p}=\frac{1}{B_c}\sum_{i=1}^{B_c}\frac{\operatorname{mean}_{n}\lVert\bar{r}^{\mathrm{mlp}}_{i,a_m,p,n,:}-\bar{r}^{\mathrm{mlp}}_{i,a_{m-1},p,n,:}\rVert_1}{\sum_{r=1}^{P}\operatorname{mean}_{n}\lVert\bar{r}^{\mathrm{mlp}}_{i,a_m,r,n,:}-\bar{r}^{\mathrm{mlp}}_{i,a_{m-1},r,n,:}\rVert_1+\epsilon}

Refined formula:

Dam,p=1Bc∑i=1Bc∥MLPp(ui,am,p)−ui,am,p∥F∥ui,am,p∥F+ϵ,ϵ=10−12.D_{a_m,p}=\frac{1}{B_c}\sum_{i=1}^{B_c}\frac{\lVert\mathrm{MLP}_p(u_{i,a_m,p})-u_{i,a_m,p}\rVert_F}{\lVert u_{i,a_m,p}\rVert_F+\epsilon},\qquad\epsilon=10^{-12}.

The ratio is computed for each sample and then averaged over the conditional batch. The input and output are taken at the following locations:

Residual stream after attention
  → LN → timestep-dependent shift/scale → u [scoring input]
  → FC1 → activation function → r_mlp [hidden activation]
  → FC2 (including bias) → MLP(u) [scoring output]
  → timestep-dependent gate → residual addition → block output

The score therefore compares the MLP input after timestep-dependent shift/scale with the MLP output before the residual gate. These are the input and output of the MLP mapping between the two timestep modulation operations.

This definition is simpler and more directly related to the allocation target. Since we allocate MLP hidden-channel counts across blocks, we score each block using the relative input–output difference of its MLP mapping. The score uses only the current anchor, removing the need for scoring snapshots from the preceding completed anchor and simplifying block scoring.

Experimental results

PixArt 5K: allocation and refresh controls

ConfigurationFID-5K ↓
Adaptive block-wise MLP allocation (ours)31.8889
Reversed MLP width assignment32.0730
Random MLP width assignment32.0750
Uniform MLP widths32.0487
Whole-block refresh, random block selection32.9010
Whole-block refresh, lowest-score block selection33.3827

With the refresh schedule, attention configuration, and MLP compute budget fixed, adaptive allocation yields lower FID than uniform widths or reversed/random width assignments. The reversed and random controls reassign the same MLP width set across blocks. Reversed assignment gives wider MLPs to lower-score blocks, while channel selection within each block still favors the largest anchor-to-anchor changes, helping preserve quality. Together with the DiT comparison in A3 between whole-block refresh and CSRe's fine-grained allocation, these results support our method.

Quality improvements with refined scoring

DiT-XL/2

ConfigurationFID ↓sFID ↓Precision ↑Recall ↑IS ↑
Original configuration2.6113865.2155610.811540.58420260.175781
New score + original allocation formula2.5892925.1952770.812140.58640259.569855

PixArt

ConfigurationFID-30K ↓COCO CLIP ↑Parti CLIP ↑
Original configuration27.22814026.55683227.097962
New score + original allocation formula26.77500026.52885727.117001

FLUX

ConfigurationFID-30K ↓COCO CLIP ↑Parti ImageReward ↑
Original configuration (cost_aware)26.51105425.9054970.921421
New score (io_proportional)26.03037025.8621290.927212

AuthorsOpenReview ↗

Response to Reviewer — Q3

Q3: Under the same budget, does component-wise refresh offer a better quality–latency trade-off than spending the computation on more frequent full refreshes? Given the competitiveness of coarse-grained caching baselines, could a budget sweep on a representative model show when CSRe's additional complexity is justified?

A3: The FORA results already cover reuse of all blocks at selected steps. We therefore compare the fixed CSRe configuration with whole-block refresh on DiT-XL/2 at approximately matched compute. The whole-block baseline concentrates the budget that CSRe distributes across blocks into a smaller set of complete blocks. We report both generation quality and measured latency.

(1) Varying refresh granularity under a fixed refresh schedule and budget. We use ImageNet 50K, 256×256 resolution, 250-step DDPM sampling, CFG=1.5, and seed=0. Whole-block refresh retains CSRe's original 58 anchor steps. At the remaining steps, the budget originally used for selected heads/channels is reassigned to the attention and MLP computation of complete blocks. Blocks are ranked using the original anchor-to-anchor hidden-activation change score, and the ranking is held fixed over the following selective interval. Depending on the budget, 2–8 complete blocks are refreshed at each step; the remaining blocks reuse cached outputs while retaining the current timestep modulation and residual gate.

MethodTFLOPs/image ↓FID ↓sFID ↓IS ↑Precision ↑Recall ↑Latency (ms/image) ↓
Whole-block refresh42.394.129.128231.5630.7710.587812.784 ± 1.592
CSRe (original paper configuration)42.402.615.216260.1760.8120.584982.670 ± 0.326

(2) Quality gains at matched compute. Compared with whole-block refresh, CSRe reduces FID from 4.117 to 2.611 and sFID from 9.128 to 5.216, while also improving IS and Precision, with a slightly lower Recall. At this operating point, selecting heads and channels within blocks therefore uses the compute budget more effectively than whole-block refresh for these quality metrics.

(3) Measured latency cost of the additional complexity. In the current implementation, CSRe requires approximately 20.90% more denoising time than whole-block refresh, while achieving better FID, sFID, IS, and Precision. Fine-grained refresh thus improves quality at this compute budget but incurs additional runtime overhead. It is useful when these quality metrics take priority, whereas whole-block refresh is faster when minimizing latency is the sole objective. Computation over complete blocks is more structured, which improves execution speed at the cost of lower quality in this comparison. Practical implementations of training-free inference acceleration often combine multiple methods, as illustrated by Chipmunk's combined results discussed in our response to W1 and Q1. For applications requiring finer control over quality and latency, CSRe offers a component-wise option that could be combined with methods operating at other levels.

Reviewer_NQRqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CSRe is a training-free method for accelerating diffusion-transformer inference. It caches projected attention-head and MLP-channel contributions, then updates selected components using their current deltas. Full anchor steps measure changes used to choose components for later selective steps. Offline calibration sets the attention-head count. The remaining compute budget is allocated across MLP blocks and implemented with hardware-aligned dense widths. The paper evaluates DiT-XL/2, PixArt-α, and FLUX.1-dev using image-quality metrics, transformer FLOPs, and denoiser latency.

Critical strengths
  • A concrete component-level update. Equations (2)–(3) and Appendix A.1 (Eqs. (6)–(8) and (14)) specify how cached projected contributions are updated. Appendix A.7 describes hardware-aligned widths and packing overhead. This makes the proposed inference path technically inspectable.

  • Useful results on DiT and PixArt. In Table 1, CSRe obtains DiT FID/sFID of 2.611/5.216 at 42.40 TFLOPs, versus 2.637/5.838 for ToCa at 43.42 TFLOPs. In Table 2, the 4.29-TFLOP PixArt setting obtains FID 28.768, versus 30.292 for ECAD at 4.27 TFLOPs. These results support a quality–FLOP benefit, subject to the runtime limitations below.

Critical weaknesses
  • The practical speedup is narrower than the headline results suggest. At similar FLOPs on PixArt, CSRe improves FID over ECAD but has higher normalized latency (50.108 versus 42.969 ms in Table 2). On FLUX, ECAD is better on every reported quality and cost metric (Table 3). The abstract's 1.728×1.728\times FLUX speedup uses the 1822.875 ms framework Full path. Using the 1386.009 ms native dense diagnostic gives about 1.314×1.314\times relative to CSRe (Appendix A.9). The column is labeled "Latency" in Tables 1–3. The values are normalized estimates, as explained below Table 2 and in Eq. (22). The paper should separate quality–FLOP gains, same-framework speedups, and practical runtime comparisons.

  • The main diagnostic does not directly validate the deployed selector. Figure 2 uses projected-change scores and a one-transition lag, whereas Eqs. (9)–(10) select from raw changes between completed anchors. Its "downstream MSE recovery" is defined only in Appendix A.4, Eq. (19). Figures 3–4 also require later text and allocation equations to explain their scores and widths. Table 4 supports the combined method against random controls, but does not link the deployed scores to Figure 2's 47.5% and 77.1% recoveries. Table 5 refreshes every head, unlike the reported kh=2k_h=2 configuration (Appendix A.10). The claims supported by each figure and ablation need clearer limits.

Questions for authors
  1. Runtime comparisons. What implementation difference explains the 1822.875 ms CSRe-framework Full path versus the 1386.009 ms native dense diagnostic in Appendix A.9? Please distinguish the speedups supported by each denominator, discuss ECAD's Table 3 advantage, and clarify whether existing PixArt timing logs change the higher normalized CSRe latency in Table 2.

  2. Selector evidence. How closely do Figure 2's projected, one-transition scores track the raw completed-anchor scores used by Eqs. (9)–(10)? If this was checked in existing runs, please point to it. Otherwise, please present Figure 2 as motivation rather than direct quantitative validation of the deployed policy.

3 comments
AuthorsOpenReview ↗

Response to Reviewer — W1/Q1

We thank the reviewer for pointing out shortcomings in the runtime comparisons and selector validation. We address W1 and Q1 jointly, as both concern practical speedups and runtime comparisons, and W2 and Q2 jointly, as both concern the correspondence between the diagnostic experiments and the deployed selector.


W1 / Q1 (joint response): Practical speedups, runtime comparisons, and speedups relative to different baselines.

W1: The practical speedup is narrower than the headline results suggest. At similar FLOPs on PixArt, CSRe improves FID over ECAD but has higher normalized latency (50.108 versus 42.969 ms in Table 2). On FLUX, ECAD is better on every reported quality and cost metric (Table 3). The abstract's 1.728× FLUX speedup uses the 1822.875 ms CSRe-framework Full path as its baseline; using the 1386.009 ms native dense diagnostic in Appendix A.9 gives a speedup of approximately 1.314× for CSRe. The column is labeled “Latency” in Tables 1–3, but the values are normalized estimates, as explained below Table 2 and in Equation (22). The paper should distinguish quality–FLOPs gains, same-framework speedups, and practical runtime comparisons.

Q1: What implementation difference explains the 1822.875 ms CSRe-framework Full path versus the 1386.009 ms native dense diagnostic in Appendix A.9? Please distinguish the speedups supported by each baseline, discuss ECAD's advantage in Table 3, and clarify whether existing PixArt timing logs change the conclusion that CSRe has higher normalized latency in Table 2.

A1:

We thank the reviewer for identifying distinctions that need clarification in our runtime reporting. Our normalization follows the approach used in Appendix A.11 of ECAD (ICLR 2026) to compare different implementations, including ToCa. We also agree that normalized latency, same-framework speedup, and actual deployment time should be clearly distinguished: an established comparison protocol does not replace reporting actual runtimes.

(1) Basis and scope of the normalization protocol. The “Latency Results” discussion in ECAD Appendix A.11 explicitly states that, because the public implementations use different infrastructure, each method's speedup is first computed relative to its own unaccelerated implementation and then applied to ECAD's unaccelerated baseline. Its Table 15 reports raw latency, speedup within each implementation, and normalized latency. For example, on PixArt-α, ToCa's unaccelerated path takes 948.688 ms/image and its caching configuration takes 403.989 ms/image, giving a 2.348× speedup. Using ECAD's 165.736 ms/image baseline yields a normalized latency of 70.577 ms/image. Equation (22) in our paper uses the same mathematical definition.

(2) Why Full baselines and actual runtimes differ across methods. The same network and similar FLOPs do not imply the same sequence of executed operators. Implementations differ in attention backends, projection fusion, tensor layouts, and cache management; accelerated paths also incur varying amounts of scoring, indexing, data movement, and small-matrix computation. Specifically, FORA's whole-block reuse and ECAD's module-level scheduling can retain regular dense computation when executing the retained modules. ToCa's token selection requires corresponding scoring and cache operations, while CSRe performs scoring, selection, parameter slicing, and cached-contribution updates at the head/channel level. These differences affect operator counts, memory access, and hardware utilization, so FLOPs savings do not translate proportionally into latency reductions. Our existing PixArt operator diagnostics provide a concrete example: to obtain attention maps, the tested ToCa implementation replaces the corresponding memory-efficient attention calls with explicit attention-matrix computation, while CSRe incurs selection, MLP parameter-packing, and additional data-movement overhead. This diagnostic helps explain implementation differences, but its configuration differs from the main table. It therefore cannot precisely decompose the main-table latency or directly explain the FLUX timing gap. The auxiliary operations required by the accelerated paths may be made less costly through operator fusion, improved cache layouts, and reduced intermediate data movement. The current differences in Full latency should therefore not be interpreted as unavoidable, fixed algorithmic costs. ECAD's appendix likewise shows substantially higher framework overhead for ToCa than for ECAD on PixArt-α. Differences in source-code versions make it difficult to port the algorithms directly, and the implementations also differ in their degree of engineering optimization. We therefore regard our algorithm as a viable option for finer-grained selection despite these implementation differences.


AuthorsOpenReview ↗

Response to Reviewer — W2/Q2

W2 / Q2 (joint response): Whether the diagnostic experiments directly validate the deployed selector.

A2:

We thank the reviewer for pointing out the differences between the diagnostic experiments and the deployed configuration. The paper did not adequately connect the motivation, deployed policy, and scope of each ablation. We clarify the temporal span of the scores, the role of projection, and the purpose of using full attention in Table 5.

(1) The role of Figure 2 and its difference from the deployed anchor interval. Figure 2 ranks components for refresh at step tt using the preceding completed adjacent-step change, t−2→t−1t-2\rightarrow t-1, along a dense trajectory. It does not select components using the as-yet unobserved change t−1→tt-1\rightarrow t. The deployed policy instead uses the change between successive completed anchors, am−1→ama_{m-1}\rightarrow a_m, and holds the selected components and MLP widths fixed over the following selective interval. Figure 2 provides motivation: in this controlled diagnostic, historical changes help identify components worth refreshing, and refreshing only a small subset recovers part of the downstream prediction error. Specifically, each intervention modifies one block, then evaluates the network suffix densely; its effect is measured by the MSE of the final CFG denoising prediction relative to the dense reference.In deployed CSRe, consecutive selective steps reuse cached outputs, so the model state and accumulated approximation error depend on earlier refresh decisions. Recovery analysis must therefore account for this history and use pure cache reuse without recomputation as the baseline, making it distinct from Figure 2. Nevertheless, Figure 2 motivates the component-selection strategy, whose effectiveness is supported by the observed improvements in generation quality.

(2) Additional comparison of unprojected and projected scores. A channel's output contribution depends not only on its hidden-activation change but also on the second-projection weights. Figure 2 uses projected-change scores, whereas deployment uses raw activation changes before projection; this distinction was insufficiently explained in the paper. We use raw activation changes to directly measure each channel's change between two completed anchors. Figure 2 motivates change-guided component selection and cannot directly serve as quantitative validation of the deployed scores.

To test whether projection weights improve channel selection in deployment, we add a ixArt COCO-10K comparison. The original score is each channel's absolute activation change between two completed anchors, averaged over tokens. The projection-aware score multiplies this score by the L2 norm of the corresponding FC2 weight column.

Channel-selection scoreFID-10K ↓COCO CLIP ↑TFLOPs/image ↓
Unprojected activation change (original method)29.12526.5355.781
Projected contribution change (FC2 column-norm weighting)29.18226.5315.781

The two scores yield similar measured generation quality at the same compute cost, and projection weighting does not improve quality at this operating point. One possible explanation is that multiple channels are refreshed together: final quality depends on the combined effect of the selected contributions and the remaining cached contributions, so changes in some channel rankings need not produce a clear quality difference. These results support retaining the original score at this operating point. We agree that the paper did not clearly explain the relationship between the diagnostic and deployed scores; the revision will distinguish them explicitly and fully report the settings and results of this experiment. The matched-compute controls in Table 4 support the generation performance of joint head–channel refresh and its selection strategy; they do not directly correspond to the two recovery percentages in Figure 2.

AuthorsOpenReview ↗

Response to Reviewer — W2/Q2

(3) Table 5 uses all attention heads to control for head selection. Table 5 asks which blocks should receive the budget when the MLP refresh budget is held fixed. To avoid introducing head-subset selection as an additional variable, every compared policy refreshes all 16 heads in each self- and cross-attention module at selective steps. The full-cache reuse rule at early steps 1 and 2 is also identical across policies. Only the set of MLP blocks and the associated budget assignment vary across All-28, Top-20, Bottom-20, and Random-20. Full attention is therefore a shared control condition used to isolate the effect of MLP block allocation, rather than a recommendation to refresh all heads in the final deployed configuration. It removes head selection as a variable, but does not imply that error propagation through attention and MLP is fully independent.

The main configuration uses kh=2k_h=2 to address a different question: how to coordinate attention and MLP refresh under a total compute budget. Refreshing fewer attention heads leaves more budget for MLP refresh. The observation in Figure 2 that a small number of heads can recover part of the error motivates this trade-off. To examine the trade-off between the head count and MLP budget directly, we add a PixArt COCO-10K experiment, setting the refresh head count to kh=1,2,3,4,8,16k_h=1,2,3,4,8,16 across attention modules. With the original anchor schedule, kh=16k_h=16 cannot meet the original budget of approximately 5.78 TFLOPs/image, so this experiment uses a new shared budget of 8.25 TFLOPs/image.

Refreshed head count khk_hFID-10K ↓Actual TFLOPs/image
128.5391658.249802
228.5565118.249868
328.6096998.249960
428.7584518.250041
829.1592358.250114
1650.1368108.249940

At this shared budget, the FID values for kh=1k_h=1 and kh=2k_h=2 differ by only 0.017346. Increasing the head count further, while correspondingly reducing the MLP refresh budget, does not improve quality. This supports jointly allocating the budget to a small number of attention heads and more MLP refresh. A small head count remains effective at lower compute budgets, where we likewise favor refreshing fewer heads. The paper's kh=2k_h=2 configuration is more broadly applicable: in our DiT-XL experiments, kh=1k_h=1 performs slightly worse, although the differences between the two settings are consistently small and do not warrant treating them separately. We will also clarify the conclusions supported by each experiment and their scope.

Reviewer_5CRuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CSRe is a training-free framework that accelerates diffusion transformers by selectively refreshing attention heads and MLP hidden channels between full anchor steps. It combines incremental cache updates with block-wise channel allocation and hardware-aligned dense execution. The approach is practical and technically sound, although several design choices require stronger justification.

Critical strengths
  1. The additive formulation provides a clear and mathematically sound mechanism for updating selected components while retaining cached contributions.
  2. The framework requires no retraining and accommodates self-attention, cross-attention, and joint text–image attention.
  3. Evaluations on DiT, PixArt, and FLUX demonstrate useful quality–compute trade-offs across different architectures.
Critical weaknesses
  1. For DiT and PixArt, scores measure raw activation changes, whereas output contributions also depend on the second-projection weights. A channel with a smaller activation change but larger projection weights may have greater output impact. A comparison with projection-aware scoring would strengthen the justification.

  2. Comparing only two anchor snapshots may overlook changes that increase and subsequently decrease within the interval. The robustness of this endpoint-based measure should be evaluated.

  3. Longer intervals reduce refresh costs but can make component rankings stale. An interval-length sweep should quantify the resulting quality–latency trade-off.

  4. Refresh benefits also depend on change concentration, already-covered channels, and downstream sensitivity. Equation (13) minimizes deviation from predefined width targets rather than output error; stronger justification or comparisons with alternative allocation policies are needed.

  5. Appendix A.10 states that Table 5 refreshes all 16 attention heads, whereas the main method typically refreshes two. The experiment does not establish whether the allocation benefit persists under selective attention refresh.

  6. Comparisons with ProCache and SVD-Cache would better contextualize the method’s advantages. If aligned evaluations are infeasible, the implementation barriers should be explained.

Questions for authors

See weaknesses above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer — W1,W2

We thank the reviewer for the comments on the scoring rationale, anchor intervals, budget allocation, and comparisons with related methods. We respond to W1–W6 in their original order below.


W1: Comparing raw activation-change scores with scores that account for output-projection weights.

For DiT and PixArt, the current scores measure raw activation changes, but output contributions also depend on the second-projection weights. A channel with a smaller activation change but larger projection weights may have a greater effect on the output. A comparison with scores that account for projection weights would strengthen the scoring rationale.

A1:

We thank the reviewer for raising this issue. We agree that a channel's output contribution depends on both its hidden-activation change and the second-projection weights. The original method uses unprojected activation changes because they directly measure the magnitude of each hidden channel's change between two completed anchors. However, the paper did not adequately explain how this choice differs from projection-aware scoring or provide a direct comparison. The diagnostic in Figure 2 uses changes in projected contributions to motivate change-guided component selection. Our insufficient distinction between these settings led to the reviewer's concern.

To directly test whether projection weights improve channel selection, we add a PixArt COCO-10K comparison:

Channel-selection scoreFID-10K ↓COCO CLIP ↑TFLOPs/image ↓
Unprojected activation change29.12526.5355.781
Projected contribution change29.18226.5315.781

The two scores yield similar quality, and projection weighting does not improve quality in this setting. One possible explanation is that multiple channels are refreshed together: weighting changes the selection of some channels, but final quality depends on the combined effect of all selected contributions and the remaining cached contributions. Changes in individual channel rankings therefore need not produce a clear difference in generation quality. These results support retaining the straightforward unprojected score at this operating point. In the revision, we will explicitly distinguish the diagnostic scores in Figure 2 from the deployed scores and include this direct comparison.


W2: Whether changes between two anchor endpoints adequately capture changes within the interval.

Comparing only two anchor snapshots may miss changes that first increase and then decrease within the interval. The robustness of this endpoint-based measure should be evaluated.

A2:

We agree that the net change between two anchors may underestimate fluctuations that first increase and then decrease within the interval. Moreover, the change over a completed interval is only a historical proxy for refresh demand in the next interval. It therefore cannot guarantee that each block receives a budget precisely matched to its actual refresh benefit at every step. We should have stated the scope of this approximation more clearly in the paper.

Our motivation is to use the overall variation in computational demand across network depth and denoising stages. Δ-DiT uses different caching schedules for front and rear blocks at different sampling stages, based on their different roles. Following this idea, we refine the coarse front/rear partition into adaptive block-wise MLP width allocation, redistributing the MLP budget for the following interval at each anchor according to observed changes.

The existing statistics support the practical value of this refinement. The diagnostic in Figure 1, based on 5K dense DiT-XL/2 trajectories, shows that the online allocation ranking has an 85.6% overlap with the dense reference's Top-9 blocks with the largest changes. This exceeds the 47.8% overlap of the tested two-stage heuristic with a fixed front/rear partition and the 63.3% overlap of the three-stage heuristic with fixed front/middle/rear partitions. These results indicate that adaptive ranking better captures the overall pattern of change. Together with the generation-quality and compute results in the paper, they support the practical utility of this low-overhead approximation.

We will state this limitation explicitly in the revision and describe endpoint scoring as an empirical proxy for budget allocation. Future work will explore simpler and more accurate scoring and allocation methods that better account for within-interval fluctuations while controlling observation and refresh overhead, improving the correspondence between allocated budgets and actual refresh benefits.


AuthorsOpenReview ↗

Response to Reviewer — W3,W4

W3: Anchor interval length and the quality–latency trade-off.

Longer intervals reduce refresh costs but may also make component rankings stale. An interval-length sweep should quantify the resulting quality–latency trade-off.

A3:

Longer anchor intervals reduce the overhead of full refreshes but also extend the time over which historical scores are used, increasing the possibility of stale component rankings. With no constraint on offline calibration cost, one could search for different intervals at different denoising stages to obtain a better refresh schedule under a given quality or compute budget. Our current interval represents a practical operating point, rather than an optimum at every stage.

The main objective of this work is to test whether, under a given refresh schedule, selecting components according to observed changes and adaptively allocating block-wise widths can use a limited refresh budget effectively. We therefore use a predefined anchor schedule to demonstrate this mechanism, without separately optimizing the length of each interval. Optimizing individual intervals would introduce more calibration parameters and make quality improvements depend on both schedule search and component allocation, complicating the attribution of their respective contributions. Similar schedule optimization could also be applied to other caching methods, so we regard it as a complementary direction for further optimization.

For deployment with sufficient calibration resources, optimizing stage-specific intervals for the target model, sampler, and quality requirements is a reasonable and useful engineering choice. A simpler predefined schedule reduces tuning and maintenance costs. We use the latter to demonstrate the effects of component selection and budget allocation, without claiming that the current interval is optimal for all deployment scenarios.

We also acknowledge that sweeping a uniform interval differs from tuning each interval separately: the former can still assess sensitivity to interval length with relatively few hyperparameters. The considerations above therefore explain our choice of schedule . In the revision, we will clarify the range of schedules covered by the current experiments and this limitation, avoiding generalization from a single schedule to all interval lengths.


W4: Rationale for the width-allocation targets and comparison with other allocation strategies.

Refresh benefits also depend on the concentration of changes, the channels already covered, and downstream sensitivity. Equation (13) minimizes deviations from predefined width targets rather than output error; stronger justification or comparisons with other budget allocation strategies are needed.

A4: We thank the reviewer for this constructive suggestion. We agree that refresh benefits depend not only on a block's total change but also on how that change is concentrated across channels. The same total change may be concentrated in a few channels or distributed across many, so it need not imply the same refresh demand.

This is closely related to channel selection, discussed in W1. Within each block, we rank channels by their activation changes and preferentially refresh those with larger changes, so that the limited budget covers as much of the dominant change as possible. The additional projection-aware scoring comparison in W1 yields generation quality similar to the original score, providing empirical support for the current selection rule in that setting. It does not, however, imply that total change fully characterizes the concentration of changes.

Adaptive block-wise width allocation then aims to provide refresh capacity suited to each block's change-related demand, avoiding a uniform width that limits some blocks' coverage of channels with large changes. Given the overall quality–compute trade-off reported in the paper, we regard this mechanism as a feasible, low-overhead approximation. Its objective is to cover as much of the dominant change within each block as possible. Equation (13) serves only to obtain a feasible discrete width allocation under the total budget constraint; it does not directly minimize output error. The reviewer's suggestion identifies a clear direction for further improvement, which we will explore in future work on this part of the method.

AuthorsOpenReview ↗

Response to Reviewer — W5,W6

W5: Whether the allocation benefits persist with selective attention refresh.

Appendix A.10 states that Table 5 refreshes all 16 attention heads, whereas the main method typically refreshes only 2. This experiment does not establish whether the allocation benefits persist when attention is selectively refreshed.

A5:

Table 5 refreshes all 16 attention heads to control the effect of attention-cache errors and examine MLP block allocation strategies with dense attention computation. All configurations use the same attention setting and MLP compute budget, so this experiment primarily analyzes the effect of MLP budget allocation itself.

To address this question directly, we add a PixArt COCO-10K comparison with 2-head selective refresh. The experiment retains the scoring, anchor schedule, and MLP budget settings used in the actual Table 5 runs. At selective steps, only 2 heads are refreshed in each block's self-attention and cross-attention modules. The findings are consistent with those of the original experiment that refreshes all 16 attention heads.

MLP block policyFID-10K ↓COCO CLIP ↑
Top-2030.44226.579
Bottom-2031.27826.320
Random-2031.12626.373

W6: Comparisons with ProCache and SVD-Cache.

Comparisons with ProCache and SVD-Cache would help establish the advantages of the proposed method more fully. If evaluations under matched conditions are not feasible, the implementation barriers should be explained.

A6: We thank the reviewer for suggesting comparisons with ProCache and SVD-Cache. Under our current experimental conditions, however, it is difficult to complete a fair quantitative comparison under strictly matched conditions. Specifically, SVD-Cache currently has no public implementation, so we would need to reimplement it from the paper and validate its caching mechanism and parameter settings, making reliable reproduction and consistent evaluation challenging. Although ProCache provides a public implementation, its full evaluation pipeline includes offline refresh-schedule search and calibration on a small sample set in addition to inference. A fair comparison would require rerunning these procedures under the same base sampling protocol, ensuring that the resulting refresh schedule matches the target compute budget, and aligning the evaluation and timing settings. For example, ProCache's main DiT experiments use 50-step DDIM sampling, whereas ours use 250-step DDPM sampling. Simply adapting ProCache to 250-step DDPM would raise fairness concerns and could require retuning its parameters according to its algorithm. These sampling differences preclude a direct quantitative comparison between its published results and ours. Given these reproduction and experimental alignment requirements, we were unable to complete reliable comparisons within the rebuttal period. To avoid misleading conclusions caused by differences in implementation, sampling protocols, and budget settings, we do not report quantitative comparisons that have not been rigorously matched.

Thank you for the detailed responses and additional experiments. However, several concerns remain. For example, the block-ranking overlap in A2 does not directly assess within-interval fluctuations or channel-ranking reliability. A3 acknowledges that a uniform interval sweep could assess sensitivity with limited tuning, but provides no such results. A4 clarifies the heuristic nature of width allocation without comparing alternative policies. I will maintain my original score.

Comments on the paper

3 comments

We thank the three reviewers and the Area Chair for their valuable time and thoughtful feedback. We have responded to the comments and added targeted controlled experiments. We discussed Reviewer 1's comments within the limited time available and appreciate their detailed initial questions and willingness to continue the discussion. We did not have a further exchange with Reviewer 2, possibly because our response was submitted close to the deadline. Nevertheless, we believe that our response addresses their concerns, as summarized below.Reviewer 3's comments and guidance were particularly constructive. Their concerns arose because we had not sufficiently explained certain details and distinctions. The reviewer indicated that they would raise their score if the remaining concerns were addressed, and we responded as promptly as possible. As of this writing, with the deadline approaching, we are still awaiting their assessment of whether our response resolves those concerns. We ask the Area Chair to consult the latest discussion for any subsequent updates.

The revised manuscript clarifies the specific distinctions from Chipmunk and DiTFastAttnV2 in Section 2.2 (Related Work), adds one overview sentence and appendix reference for each of the five groups of supplementary experiments in Section 4.3 (Ablation and analysis), and reports their full settings and results in Appendices A.12.1–A.12.5, Tables 8–12.

Reviewer 1 acknowledged the clarity and mathematical soundness of selective component updates and cached-contribution reuse, training-free operation, compatibility with multiple attention structures, and useful quality–compute trade-offs on DiT, PixArt, and FLUX. Reviewer 2 recognized the inspectability of the component-level update equations and implementation details, including hardware-aligned widths and weight packing, as well as the quality–FLOPs benefits demonstrated on DiT and PixArt. Reviewer 3 acknowledged the evidence from matched-compute ablations showing that change-guided component selection outperforms random selection.

Below we summarize the main concerns and give concise responses to the remaining points, identifying the corresponding revision locations in the revised PDF. The full discussions can be found in our exchanges with each reviewer.

I. Shared concerns

1. Diagnostic motivation and deployed scores (R1 W1; R2 W2/Q2). Figure 2 uses changes in projected contributions over a preceding completed adjacent-step transition along a dense trajectory to intervene on one block, evaluates the remaining network densely, and measures the MSE of the final CFG denoising prediction relative to the dense reference. Deployed channel selection instead uses raw activation changes before projection between two completed anchors, with cache reuse across consecutive selective steps. The temporal spans and error-accumulation conditions differ, so Figure 2 motivates change-guided selection and cannot directly serve as quantitative validation of the deployed scores. Appendix A.12.1 of the revised manuscript makes this distinction explicit.

The additional PixArt COCO-10K channel-score comparison fixes each block's refresh width and evaluates raw activation changes against FC2 column-norm weighting. At the same 5.781 TFLOPs/image, FID is 29.125 / 29.182 and COCO CLIP is 26.535 / 26.531, respectively. The two scores yield similar measured generation quality, and projection weighting does not improve quality at this operating point, supporting retention of the original score. Revision locations: Section 4.3 adds the score-comparison overview and reference; Appendix A.12.1 and Table 8 explain the diagnostic/deployment distinction and report the comparison.

2. MLP allocation with selective attention (R1 W5; R2 W2/Q2). The original Table 5 refreshes all 16 attention heads to control for head-subset selection and isolate the effect of MLP block allocation. The additional PixArt COCO-10K two-head comparison retains the scoring, anchor schedule, and MLP-budget settings of the actual Table 5 configuration, but refreshes two heads in both self-attention and cross-attention, comparing All-28, Top-20, Bottom-20, and Random-20. Top-20 achieves FID 30.442, compared with 31.278 for Bottom-20, consistent with the original comparison. Revision locations: Section 4.3 adds the two-head overview and reference; Appendix A.12.2 and Table 9 report the settings and complete comparison.

3. Attention–MLP budget trade-off (R2 W2/Q2; R3 follow-up). The additional PixArt COCO-10K fixed-budget experiment varies the refreshed head count jointly in self- and cross-attention over kh=1,2,3,4,8,16k_h=1,2,3,4,8,16, adjusting MLP width to maintain approximately 8.250 TFLOPs/image. This common budget accommodates the 16-head configuration, which is infeasible at the original approximately 5.780 TFLOPs/image budget while retaining the original Full schedule. One and two heads achieve FID 28.539 / 28.557; allocating more compute to attention at the expense of MLP refresh does not improve quality. This supports combining few-head refresh with more MLP refresh at this operating point and provides evidence for using kh=2k_h=2 together with block-wise MLP budget allocation. Revision locations: Section 4.3 adds the head-count/MLP-budget overview and reference; Appendix A.12.3 and Table 10 report the fixed-budget results.

II. Remaining concerns from Reviewer 1

1. Scope of anchor-endpoint scores (W2). Endpoint differences may miss fluctuations that increase and then decrease within an interval; a completed interval is also only a historical proxy for subsequent refresh demand. In the existing DiT-XL/2 dense-trajectory diagnostic, adaptive rankings overlap the high-change Top-9 blocks by 85.6%, compared with 47.8% and 63.3% for the tested two- and three-stage fixed partitions, supporting their ability to capture overall variation.

2. Anchor intervals and the quality–latency trade-off (W3). We use a preset anchor schedule primarily to demonstrate the effects of component selection and budget allocation under a given refresh schedule. We do not separately optimize each interval or claim that the chosen interval is optimal for every model and denoising stage. We regard optimizing the interval as an engineering refinement rather than a demonstration of our algorithm's advantages; those advantages can still be demonstrated using a preset schedule.

3. Width-allocation objective (W4). Total block change does not fully capture the concentration of channel changes or downstream sensitivity. The method prioritizes high-change channels within blocks and adjusts widths across blocks to accommodate different refresh demands. Equation (13) produces a feasible discrete allocation close to target widths under the total budget; it does not directly minimize output error.

4. ProCache and SVD-Cache comparisons (W6). Strictly aligned quantitative comparisons were not completed. Our response explains the reproduction difficulty due to the lack of a public SVD-Cache implementation and the offline schedule search, small-sample calibration, and sampling-protocol alignment required for ProCache.

III. Remaining concerns from Reviewer 2

Practical acceleration and timing definitions (W1/Q1). We use the same normalization definition as ECAD's appendix: each method's speedup relative to its own Full path is rescaled to a common reference. Normalized latency, same-framework speedup, and measured runtime remain distinct. Attention backends, projection fusion, cache management, scoring, indexing, parameter slicing, and data movement can give different runtimes at similar FLOPs.

For the FLUX baseline raised by the reviewer, 1.728× uses the CSRe-framework Full path of 1822.875 ms; using the native dense diagnostic of 1386.009 ms instead gives approximately 1.314×. ECAD's appendix also shows substantially higher framework overhead for ToCa than for ECAD on PixArt-α. Differences in source-code versions make it difficult to port the algorithms directly, and the implementations also differ in their degree of engineering optimization. We therefore regard our algorithm as a viable option for finer-grained selection despite these implementation differences.

IV. Remaining concerns and follow-up from Reviewer 3

1. Distinctions from Chipmunk and controlled comparisons (W1/Q1; follow-up). Chipmunk refreshes subsets of KV interactions for query groups, whereas CSRe refreshes selected complete heads, including QKV, complete token interactions, and output-projected contributions. Selective MLP-channel refresh is shared by the two methods. Our core contribution builds on this mechanism: given the attention budget, we allocate different numbers of refreshed channels across blocks according to their MLP refresh demands. Revision location: Section 2.2 (Related Work) clarifies the differences from Chipmunk in complete-head refresh and block-wise width allocation.

Two separate comparisons are included:

  • Attention: In the official Chipmunk FLUX framework at 768×1280, with dense MLPs and approximately matched matrix FLOPs, CSRe-Attn and Chipmunk-Attn achieve FID-5K 27.231 / 27.718, and CSRe-Attn reduces eager denoising latency by 7.04%. The additional 256×256 compute diagnostic finds that the attention core accounts for approximately 3.99% of Full matrix FLOPs and that the official Chipmunk attention configuration reduces total compute by only 2.76%, illustrating the limited savings from compressing token interactions on short sequences. This diagnostic and the separate attention comparison are included in Appendix A.12.4, Table 11(a).
  • MLP: In official FLUX.1-dev at 768×1280 and 50 steps, with dense attention and approximately matched main MLP compute, 300 images give CLIP 25.932 / 25.877 and ImageReward 0.847 / 0.780 for CSRe-MLP and Chipmunk-MLP. This reflects the combined effects of their selection mechanisms, refresh schedules, and budget allocation schemes, rather than an ablation that isolates block-wise allocation. The separate MLP comparison is included in Appendix A.12.4, Table 11(b).

Main-text revision for both comparisons: Section 4.3 adds an overview of the separate attention and MLP comparisons and cites Appendix A.12.4; the full settings and results are combined in Table 11(a)/(b).

2. Distinctions from DiTFastAttnV2. DiTFastAttnV2 uses calibrated per-head errors and latency to preassign full, sparse, or cached modes. CSRe dynamically updates the head identities for the following interval from changes between completed anchors on the current sample, retains complete token interactions in the selected heads, and extends selective refresh to their QKV and output-projected contributions. The shared-budget head-count experiment supports combining few-head refresh with MLP capacity. Revision locations: Section 2.2 (Related Work) adds these distinctions and the DiTFastAttnV2 reference; the related budget comparison appears in Section 4.3 and Appendix A.12.3, Table 10.

3. Component-wise versus whole-block refresh (Q3). The additional DiT-XL/2 ImageNet 50K comparison retains the same 58 Full steps and concentrates the remaining head/channel budget into complete blocks at approximately 42.400 TFLOPs/image. CSRe and whole-block refresh achieve FID 2.611 / 4.117 and sFID 5.216 / 9.128. CSRe also improves IS and precision, with slightly lower recall, while requiring approximately 20.90% more measured denoising time. This fixed-schedule, single-budget comparison shows improved generation quality at this operating point together with additional runtime. Revision locations: Section 4.3 adds the quality–runtime trade-off overview and reference; Appendix A.12.5 and Table 12 report the whole-block settings, quality metrics, and measured times.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.