- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes CSRe, which selectively refreshes attention heads and MLP hidden channels based on changes measured at completed anchor steps, while coordinating their compute budgets.
Critical strengths
The matched-compute ablations provide useful evidence that change-guided component selection improves over random selection.
Critical weaknesses
The incremental contribution over closely related methods, particularly Chipmunk and DiTFastAttnV2, is not sufficiently established through discussion or controlled comparisons.
Questions for authors
- Could the authors clearly identify the claimed advances over Chipmunk and experimentally demonstrate the benefits of those specific design choices, rather than selective refresh in general?
- Could the authors isolate the benefits of coordinated attention–MLP budgeting and adaptive block-wise allocation? At a fixed schedule and compute budget, comparisons against a fixed head/channel budget split and uniform per-block MLP widths would help establish these contributions.
- Does component-wise refresh provide a better quality–latency trade-off than allocating the same budget to more frequent full refreshes? Given the competitive or stronger results of coarse caching baselines, a budget sweep on one representative model would help establish when CSRe’s additional complexity is justified.
6 comments
Response to Reviewer — Q2
Q2: Can the effects of coordinated attention–MLP budget allocation and adaptive allocation across blocks be validated separately? Under a fixed refresh schedule and compute budget, comparisons are needed against a fixed head/channel budget split and a uniform MLP width across blocks.
A2: We refine the score used for block-wise MLP allocation. Within the existing allocation framework, we replace the original block-change score with the relative input–output difference of the MLP mapping at the current anchor step. This score determines the block-wise MLP widths at subsequent selective steps. Following this refinement, ablation results suggest that coordinated attention–MLP budget allocation and adaptive block-wise MLP allocation each contribute to a favorable quality–latency trade-off, enabling finer-grained control over this trade-off.
Following the paper's notation, and denote the current and preceding completed anchors, respectively; indexes MLP refresh modules, is the total number of these modules, and is the number of samples in the conditional batch. The hidden activations recorded at an anchor are denoted by , i.e., the features after the activation function and before the second projection (FC2). In the new formula, follows the MLP input notation in Equation (7) of the paper and denotes the value after timestep-dependent shift and scale modulation.
Original formula (combined form of Equation (11)):
Refined formula:
The ratio is computed for each sample and then averaged over the conditional batch. The input and output are taken at the following locations:
Residual stream after attention
→ LN → timestep-dependent shift/scale → u [scoring input]
→ FC1 → activation function → r_mlp [hidden activation]
→ FC2 (including bias) → MLP(u) [scoring output]
→ timestep-dependent gate → residual addition → block output
The score therefore compares the MLP input after timestep-dependent shift/scale with the MLP output before the residual gate. These are the input and output of the MLP mapping between the two timestep modulation operations.
This definition is simpler and more directly related to the allocation target. Since we allocate MLP hidden-channel counts across blocks, we score each block using the relative input–output difference of its MLP mapping. The score uses only the current anchor, removing the need for scoring snapshots from the preceding completed anchor and simplifying block scoring.
Experimental results
PixArt 5K: allocation and refresh controls
| Configuration | FID-5K ↓ |
|---|---|
| Adaptive block-wise MLP allocation (ours) | 31.8889 |
| Reversed MLP width assignment | 32.0730 |
| Random MLP width assignment | 32.0750 |
| Uniform MLP widths | 32.0487 |
| Whole-block refresh, random block selection | 32.9010 |
| Whole-block refresh, lowest-score block selection | 33.3827 |
With the refresh schedule, attention configuration, and MLP compute budget fixed, adaptive allocation yields lower FID than uniform widths or reversed/random width assignments. The reversed and random controls reassign the same MLP width set across blocks. Reversed assignment gives wider MLPs to lower-score blocks, while channel selection within each block still favors the largest anchor-to-anchor changes, helping preserve quality. Together with the DiT comparison in A3 between whole-block refresh and CSRe's fine-grained allocation, these results support our method.
Quality improvements with refined scoring
DiT-XL/2
| Configuration | FID ↓ | sFID ↓ | Precision ↑ | Recall ↑ | IS ↑ |
|---|---|---|---|---|---|
| Original configuration | 2.611386 | 5.215561 | 0.81154 | 0.58420 | 260.175781 |
| New score + original allocation formula | 2.589292 | 5.195277 | 0.81214 | 0.58640 | 259.569855 |
PixArt
| Configuration | FID-30K ↓ | COCO CLIP ↑ | Parti CLIP ↑ |
|---|---|---|---|
| Original configuration | 27.228140 | 26.556832 | 27.097962 |
| New score + original allocation formula | 26.775000 | 26.528857 | 27.117001 |
FLUX
| Configuration | FID-30K ↓ | COCO CLIP ↑ | Parti ImageReward ↑ |
|---|---|---|---|
| Original configuration (cost_aware) | 26.511054 | 25.905497 | 0.921421 |
| New score (io_proportional) | 26.030370 | 25.862129 | 0.927212 |
Response to Reviewer — Q3
Q3: Under the same budget, does component-wise refresh offer a better quality–latency trade-off than spending the computation on more frequent full refreshes? Given the competitiveness of coarse-grained caching baselines, could a budget sweep on a representative model show when CSRe's additional complexity is justified?
A3: The FORA results already cover reuse of all blocks at selected steps. We therefore compare the fixed CSRe configuration with whole-block refresh on DiT-XL/2 at approximately matched compute. The whole-block baseline concentrates the budget that CSRe distributes across blocks into a smaller set of complete blocks. We report both generation quality and measured latency.
(1) Varying refresh granularity under a fixed refresh schedule and budget. We use ImageNet 50K, 256×256 resolution, 250-step DDPM sampling, CFG=1.5, and seed=0. Whole-block refresh retains CSRe's original 58 anchor steps. At the remaining steps, the budget originally used for selected heads/channels is reassigned to the attention and MLP computation of complete blocks. Blocks are ranked using the original anchor-to-anchor hidden-activation change score, and the ranking is held fixed over the following selective interval. Depending on the budget, 2–8 complete blocks are refreshed at each step; the remaining blocks reuse cached outputs while retaining the current timestep modulation and residual gate.
| Method | TFLOPs/image ↓ | FID ↓ | sFID ↓ | IS ↑ | Precision ↑ | Recall ↑ | Latency (ms/image) ↓ |
|---|---|---|---|---|---|---|---|
| Whole-block refresh | 42.39 | 4.12 | 9.128 | 231.563 | 0.771 | 0.587 | 812.784 ± 1.592 |
| CSRe (original paper configuration) | 42.40 | 2.61 | 5.216 | 260.176 | 0.812 | 0.584 | 982.670 ± 0.326 |
(2) Quality gains at matched compute. Compared with whole-block refresh, CSRe reduces FID from 4.117 to 2.611 and sFID from 9.128 to 5.216, while also improving IS and Precision, with a slightly lower Recall. At this operating point, selecting heads and channels within blocks therefore uses the compute budget more effectively than whole-block refresh for these quality metrics.
(3) Measured latency cost of the additional complexity. In the current implementation, CSRe requires approximately 20.90% more denoising time than whole-block refresh, while achieving better FID, sFID, IS, and Precision. Fine-grained refresh thus improves quality at this compute budget but incurs additional runtime overhead. It is useful when these quality metrics take priority, whereas whole-block refresh is faster when minimizing latency is the sole objective. Computation over complete blocks is more structured, which improves execution speed at the cost of lower quality in this comparison. Practical implementations of training-free inference acceleration often combine multiple methods, as illustrated by Chipmunk's combined results discussed in our response to W1 and Q1. For applications requiring finer control over quality and latency, CSRe offers a component-wise option that could be combined with methods operating at other levels.
Response to Reviewer — W1/Q1
We thank the reviewer for pointing out shortcomings in our comparisons with existing methods and controlled experiments. We address W1 and Q1 jointly because both concern our specific contributions relative to existing methods. Q2 and Q3 retain their original numbering.
W1 / Q1 (joint response): Please clarify the specific improvements over Chipmunk and validate the benefits of these designs, rather than only demonstrating the overall effectiveness of selective refresh.
A1: We agree that comparison with full computation alone cannot distinguish the effects of different selective refresh designs. To evaluate attention separately from MLP, we compare Chipmunk-Attn and CSRe-Attn under approximately matched attention compute budgets. Both use dense MLP computation, with their respective MLP acceleration mechanisms disabled.
(1) Specific differences from existing methods. Chipmunk reuses intermediate results through dynamic column-sparse incremental updates. For attention, it selects a subset of KV interactions for each query group for recomputation and reuses the contributions of the remaining interactions. Its MLP uses a separate mechanism for neuron selection and incremental updates. Evaluating attention and MLP separately is therefore appropriate for both Chipmunk and CSRe. Given the limited time, we first compare their attention mechanisms. CSRe refreshes complete attention heads: at selective steps, we recompute the selected heads' QKV projections, attention over all tokens, and output-projected contributions, while reusing cached contributions for the other heads. The key distinction is whether the budget is spent on a subset of KV interactions within each head or on a subset of complete heads, including their projection computations.
(2) Limitations of Chipmunk on short sequences. Chipmunk's main experiments cover high-resolution image and video generation, where long sequences allow greater reductions in attention computation through sparsification. For sequence length N and hidden dimension d, the matrix computations in the attention core scale approximately as O(N2d), whereas QKV projections, output projections, and MLP computation primarily scale as O(Nd2). As sequences become shorter, the attention core accounts for a smaller fraction of total computation; even eliminating a substantial proportion of KV interactions has a smaller effect on the total compute cost. Sparse selection, caching, and memory-access overhead may further offset part of the latency benefit.
At 256×256 resolution, FLUX.1-dev has 256 image and 512 text tokens (768 in total). In Full, QK⊤ and AV, excluding QKV/output projections, account for only 3.99% of total matrix FLOPs. Chipmunk-Attn's official attention configuration reduces the 50-step total from 517.38 to 503.10 TFLOPs/image, saving only 2.76%. To compare the methods at a budget with greater computational savings, we use the Chipmunk paper's 768×1280 resolution: 3840 image and 512 text tokens (4352 in total).
Short sequences also matter in practice: the π₀ action expert processes only 50 action tokens, illustrating the need for efficient iterative inference under real-time control constraints. Chipmunk acknowledges limitations on short sequences, motivating acceleration methods that remain effective in this regime.
(3) Controlled comparison. We evaluate each method independently. Chipmunk's reported gains from combination with ToCa suggest that different optimization granularities can be complementary. CSRe's head- and channel-level reuse may likewise complement token-level methods, but combining them requires joint tuning of caching, budgets, execution order, and hyperparameters beyond our current experiments.
All three configurations are run in the official Chipmunk FLUX framework. Chipmunk-Attn retains the official attention configuration, and we match our method to its resulting compute cost.
(4) Latency and generation quality. Results at 768×1280 are shown below.
At approximately matched compute, CSRe-Attn improves FID by 0.487 and reduces eager denoising latency by 7.04% relative to Chipmunk-Attn. This is the shortest-sequence setting in the Chipmunk paper. CSRe can further reduce computation on short sequences when not constrained to match Chipmunk's budget.