acceptodds
Under review as a conference paper at ICLR 2027

CacheFlow: Unifying Step Distillation and Feature Caching for Few-Step Generation

Abstract

Step distillation and feature caching accelerate diffusion models by reducing sampling steps and per-step computation, respectively. Under extreme acceleration, distillation struggles to approximate teacher trajectories, sacrificing semantic fidelity and fine details; training-free feature caching degrades on the resulting sparse trajectories, amplifying error propagation when combined with distillation. We introduce CacheFlow, which addresses extreme few-step acceleration from a frequency-aware perspective. We identify asynchronous frequency evolution as the primary source of quality degradation: low-frequency semantics stabilize rapidly, while high-frequency details suffer trajectory collapse and cross-step distortion. We therefore propose frequency-aware policy distillation to provide adaptive multi-band supervision for faithful trajectory approximation and detail preservation. A lightweight, learnable cache policy captures rapid cross-step feature dynamics, overcoming rigid extrapolation in sparse regimes. Jointly optimizing the few-step student and cache policy suppresses caching errors. Experiments on FLUX.1-dev, Qwen-Image, Qwen-Image-Edit, and Qwen-Image-2.1 demonstrate competitive visual quality across text-to-image generation and image editing, with a measured speedup of up to 55.31. Code will be released.

Primary area:
generative models
Keywords:
diffusion models, flow models, few-step generation, distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ubjqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CacheFlow, a frequency-aware framework for extreme few-step diffusion acceleration. The authors attribute quality degradation to asynchronous frequency evolution and introduce two main components: frequency-aware policy distillation for adaptive multi-band trajectory supervision, and a lightweight learnable cache policy for modeling cross-step feature dynamics. The two components are jointly optimized to reduce both distillation and caching errors. Experiments on several recent text-to-image and image-editing models demonstrate competitive visual quality with up to 55.31× speedup.

Critical strengths
  1. The proposed framework provides a unified training scheme that integrates policy distillation with learnable feature caching, which is conceptually simple and easy to understand.

  2. The paper conducts extensive experiments on several modern image generation models, including FLUX.1-dev, Qwen-Image, Qwen-Image-2.1, and Qwen-Image-Edit.

Critical weaknesses
  1. The paper points out that the denoising process exhibits a distinction between low- and high-frequency information. However, this phenomenon has already been extensively studied in early DDPM and U-Net-based models. Therefore, it should not be considered a core contribution of this work.

  2. I have some concerns regarding the results presented in Figure 3. First, the paper does not specify how much data was used to obtain these statistics, making it difficult to assess the reliability and generality of the reported results. Moreover, the paper does not provide a clear definition or detailed explanation of the so-called “Normalized Change” metric. Without a precise formulation or justification, this metric appears to be an ad hoc measure introduced by the authors.

  3. As shown in Figure 3, the low-frequency component first decreases and then increases throughout the denoising process. Therefore, the results do not necessarily support the claim that the later stages of denoising primarily focus on high-frequency details. The observed trend seems to suggest a more complex evolution of frequency components during denoising, which warrants further analysis.

  4. The interval approximation method adopted in this paper is somewhat similar to that used in MeanFlow. However, the authors do not appear to provide a direct quantitative comparison with MeanFlow. It would be helpful to include such a comparison to better demonstrate the advantages and differences of the proposed method.

  5. Overall, the proposed method appears to be largely an A+B combination of step distillation and feature caching, with relatively limited methodological novelty.

Questions for authors

Please see the Critical Weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer ubjq Part 1/4

Thank you very much for your detailed review and valuable feedback. We appreciate your suggestions for improving the clarity and quality of our work.

W1. Research scope and contributions of CacheFlow

A1: Thank you for your valuable suggestion. We acknowledge prior studies of frequency evolution in diffusion models. However, our work investigates the joint optimization of step distillation and learnable feature caching in modern Flow Matching models with DiT backbones. The co-adaptation of the distilled student and cache predictor under sparse sampling remains beyond the scope of these earlier studies.

Earlier U-Net architectures extract multiscale features through convolution, downsampling, and skip connections, whereas DiTs aggregate spatial information through attention softmax⁡(QK⊤/d)V\operatorname{softmax}(QK^\top/\sqrt d)V. DDPMs learn noise predictions along a stochastic forward diffusion process, whereas the Flow Matching models learn velocity fields along interpolation paths xt=(1−t)x0+tϵx_t=(1-t)x_0+t\epsilon. These differences affect both feature representations and denoising dynamics.

Motivated by these differences, our work investigates the joint modeling of trajectory distillation and feature caching in modern diffusion models based on Flow Matching and DiT. Our observations of distinct velocity dynamics across frequency bands motivate a frequency-aware policy, and we further jointly optimize frequency-aware policy distillation and learnable feature caching under unified supervision, improving teacher-velocity matching and reducing caching errors in few-step generation.

Due to space constraints, we continue in Response Part 2.

AuthorsOpenReview ↗

Response to Reviewer ubjq Part 2/4

W2/W3. Additional details and interpretation of Figure 3

A2 & A3: Thank you for pointing this out. We apologize for omitting the details of Figure 3 from the main text due to space constraints. Figure 3 examines how the variation of teacher velocity differs across frequency bands and denoising stages. This observation motivates the band-specific velocity policy in CacheFlow, which separately models the temporal evolution of low-frequency structure and high-frequency details, acting as an implicit, dynamic multi-scale reweighting mechanism. We have added these details and analyses in Lines 209–234 and 250–259 in our revised paper.

1. Metric definition and statistics for Figure 3

Normalized Change measures the relative magnitude of velocity changes within each frequency band between adjacent denoising steps. For the velocity vi∈RC×H×W\mathbf v_i\in\mathbb R^{C\times H\times W} along a teacher denoising trajectory, Haar DWT yields the low- and high-frequency components fq,i=Wq[vi]\mathbf f_{q,i}=\mathcal W_q[\mathbf v_i]. We normalize the magnitude of the change between adjacent steps to obtain NormalizedChangeq,i=∣fq,i+1−fq,i∣2/max⁡j∣fq,j∣2\mathrm{Normalized\\ Change}_{q,i}=\\|\mathbf f_{q,i+1}-\mathbf f_{q,i}\\|_2/\max_j\\|\mathbf f_{q,j}\\|_2, and average the scores over the denoising trajectories. A higher Normalized Change indicates a larger normalized cross-step velocity change, whereas a lower value indicates greater stability between adjacent steps.

The statistics cover all prompts from GenEval, DPG-Bench, and OneIG-Bench, totaling 3,972 prompts. These prompts span diverse generation requirements, including compositional relationships, long textual descriptions, text rendering, styles, and reasoning.

2. Denoising progresses from low-frequency structure to high-frequency details

Figure 3 shows the changes of low- and high-frequency components at different denoising stages. Early in denoising, the normalized change of low-frequency components is substantially higher, reflecting the rapid evolution of global layout, contours, and smooth structures. As denoising progresses, the normalized change of high-frequency components gradually exceeds and becomes particularly pronounced in the later stages, reflecting the rapid evolution of edges, textures, and local details. Although the low-frequency components exhibit a late-stage increase, their normalized change remains clearly below that of high-frequency components. Thus, high-frequency components continue to dominate late-stage relative changes, reflecting a transition from establishing global structure early to refining details later in generation.

3. Figure 3 motivates our frequency policy as an implicit dynamic reweighting mechanism

Figure 3 shows that temporal dynamics are highly non-uniform across different noise levels. However, existing monolithic spatial parameterization mixes frequency bands within the same output channels, making it prone to spectral bias and capacity competition. Without carefully designed handcrafted timestep weighting of the loss, the model struggles to accommodate the evolving dynamics across different denoising stages.

Building on this observation, we propose a frequency-aware velocity policy that acts as an implicit, dynamic multi-scale reweighting mechanism that better accommodates the optimization demands of few-step distillation. Specifically, by decomposing the velocity into distinct Haar-wavelet bands and predicting band-specific parameters uθ,q(t)=Bq+[t−ts]Dq\mathbf u_{\boldsymbol\theta,q}(t)=\mathbf B_q+[t-t_s]\mathbf D_q, our policy structurally decouples the gradient paths across frequency bands in the network head. As shown in Figure 3(c)–(d) in our revised paper, our frequency policy supervises the establishment of global structure with higher low-frequency loss weights during early denoising, and the refinement of local details with higher high-frequency loss weights during late denoising, consistent with our observations in Figure 3(a)–(b). CacheFlow adaptively learns the evolution rates at different scales and provides implicit dynamic reweighting during optimization, better accommodating the evolving dynamics across different denoising stages. Tables 8 and 5 in our revised paper further demonstrate that CacheFlow achieves the best velocity fitting accuracy and generation quality.

Due to space constraints, we continue in Response Part 3.

AuthorsOpenReview ↗

Response to Reviewer ubjq Part 3/4

W4. Differences from and comparison with MeanFlow

A4: MeanFlow reduces sampling steps through interval-average velocity prediction, but it faces training instability, additional computational overhead, and challenges in integrating feature caching. CacheFlow instead performs few-step policy distillation by predicting and directly matching the teacher's instantaneous velocity trajectory and integrates learnable feature caching under a unified objective, and its performance has been validated on multiple large-scale generative backbones. Details are provided below. We have added the comparison to Table 2, Lines 282–292 in our revised paper, with discussion in Lines 415–416.

1. Differences between CacheFlow and MeanFlow

MeanFlow represents interval displacement through the average velocity u=1t−r∫rtv[xτ,τ],dτ\mathbf u=\frac{1}{t-r}\int_r^t\mathbf v[\mathbf x_\tau,\tau]\\,\mathrm d\tau and constructs its training target using u=v−[t−r]dudt\mathbf u=\mathbf v-[t-r]\frac{\mathrm d\mathbf u}{\mathrm dt}. However, the additional JVP term amplifies noise in the stochastic conditional velocity, causing training instability and increase in training overhead. Besides, its average-velocity prediction makes it difficult to precisely supervise the cache predictor at individual time points. CacheFlow instead avoids the JVP term, constructs a bandwise velocity policy vθ[t∣xts,ts]=W−1Bq+[t−ts]Dqq∈Q\mathbf v_\theta[t\mid\mathbf x_{t_s},t_s]=\mathcal W^{-1}\\{\mathbf B_q+[t-t_s]\mathbf D_q\\}_{q\in\mathcal Q}, predicts and directly matches the teacher's instantaneous velocity at multiple intermediate time points through Ldistill=E∣vθ[t∣xts,ts]−vT[xt,t]∣22\mathcal L_{\mathrm{distill}}=\mathbb E\\|\mathbf v_\theta[t\mid\mathbf x_{t_s},t_s]-\mathbf v_T[\mathbf x_t,t]\\|_2^2, and advances the trajectory through low-cost substeps. CacheFlow further jointly supervises the student and cache predictor at multiple intermediate times, enabling co-adaptation that reduces caching errors while preserving generation quality.

2. CacheFlow outperforms MeanFlow across multiple settings

We further compare CacheFlow and MeanFlow on FLUX.1-dev. As shown in Table 1, at 4 NFE, MeanFlow incurs drops of 3.0%/1.6% on GenEval/DPG-Bench relative to the original model, preserving its generation quality. However, when NFE is further reduced to 2 and 1, the two metrics drop substantially by 28.8%/11.4% and 53.0%/20.3%, indicating that MeanFlow struggles to maintain generation quality in large-scale diffusion models with very few sampling steps. In contrast, CacheFlow achieves 0.66/84.72 with only 2 NFE, delivering a 22.89× speedup. With caching, CacheFlow-Medium and CacheFlow-Fast further reduce the computational budget to 1.75/1.58 NFE and increase the speedup to 26.34×/29.20×, while maintaining scores of 0.66/84.36 and 0.65/84.08, demonstrating CacheFlow's substantial advantages in generation with very few steps.

Table 1: Comparisons with MeanFlow.

MethodNFE ↓Latency ↓Speed ↑GenEval ↑DPG-Bench ↑
FLUX.1-dev509.8151.00×0.6684.16
MeanFlow40.85311.50×0.6482.79
MeanFlow20.42722.99×0.4774.57
MeanFlow10.21445.97×0.3167.04
CacheFlow (w/o Cache)20.42922.89×0.6684.72
CacheFlow-Medium1.750.37326.34×0.6684.36
CacheFlow-Fast1.580.33629.20×0.6584.08

Due to space constraints, we continue in Response Part 4.

Reviewer_VFUNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CacheFlow combines policy-based step distillation with learnable feature caching for accelerated diffusion-transformer generation. A student predicts affine-in-time velocity coefficients in four Haar-wavelet bands, while a residual MLP predicts the change across a skipped block segment from the preceding step's cached residual. Both components are optimized through teacher-velocity supervision along trajectories that gradually transition from teacher to student integration. Experiments cover text-to-image generation and image editing with FLUX.1-dev and three Qwen variants, including component ablations and comparisons with distillation and caching baselines. The reported maximum 55.31x speedup measures the DiT path, rather than the complete generation pipeline.

Critical strengths
  • Useful acceleration beyond an already distilled student, demonstrated across multiple models and tasks
  • A simple, integrated mechanism makes caching compatible with sparse sampling.
Critical weaknesses
  • The central explanation of the frequency policy is not established by its mathematical formulation. With the linear inverse Haar transform, Eq. 1 and the stated band policies imply v(t) = W(B) + (t - t_s) W(D). This is still an affine spatial-domain velocity policy, and invertibility means unrestricted band coefficients do not enlarge its temporal function class. Moreover, Eq. 3 is ordinary spatial velocity MSE; by Appendix D's Parseval identity, it is equivalent to unweighted bandwise MSE. Consequently, the claim that this removes a restrictive "monolithic temporal parameterization" needs an architectural or optimization explanation beyond changing coordinates.
  • This paper lacks comparisons against some of the most popular few-step baselines, such as DMD2, TDM, and decoupled-DMD.
Questions for authors

See Weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer VFUN Part 1/3

We sincerely thank you for your thoughtful review and insightful comments. Your suggestions have helped us further strengthen our paper.

W1. Frequency-aware parameterization and implicit dynamic reweighting

A1: We sincerely thank you for the rigorous and insightful mathematical analysis, which provides a valuable theoretical perspective for explaining CacheFlow. We agree with your derivation: under an unconstrained linear inverse Haar transform and unweighted Parseval MSE, our formulation does not enlarge the temporal function class, nor does it change the loss value associated with the velocity prediction error. However, we would like to clarify that the core rationale behind our frequency policy lies in its architectural inductive bias and optimization dynamics, which effectively functions as an implicit, scale-aware temporal reweighting mechanism rather than a mere coordinate transformation.

1. Spatial parameterization faces spectral bias and capacity competition

In diffusion distillation, temporal dynamics across different noise levels are highly non-uniform. As analyzed in Figure 3(a)–(b) in our revised paper, low-frequency components exhibit larger relative changes during early denoising, while high-frequency components exhibit larger relative changes during later denoising. Directly modeling a monolithic spatial velocity v(t)=B~+[t−ts]D~v(t)=\tilde{\mathbf B}+[t-t_s]\tilde{\mathbf D} forces high- and low-frequency components to share the same spatial parameterization channels. Because standard MSE gradients are heavily dominated by low-frequency energy, the network struggles to capture the subtle yet crucial temporal evolution D~\tilde{\mathbf D} of high-frequency textures without carefully designed, hand-crafted timestep weighting schedules w(t)w(t).

2. Frequency policy as implicit dynamic multi-scale reweighting

Our frequency policy acts as an implicit, dynamic multi-scale reweighting mechanism that better accommodates the optimization demands of few-step distillation. Specifically, by decomposing the velocity into distinct Haar-wavelet bands and predicting band-specific parameters uθ,q(t)=Bq+[t−ts]Dq\mathbf u_{\boldsymbol\theta,q}(t)=\mathbf B_q+[t-t_s]\mathbf D_q, our policy structurally decouples the gradient paths across frequency bands in the network head:

  • Each band q∈LL,LH,HL,HHq\in\\{LL,LH,HL,HH\\} independently learns its baseline response Bq\mathbf B_q and adaptive temporal drift rate Dq\mathbf D_q.
  • Rather than relying on heuristic temporal loss weighting such as Min-SNR or empirical SNR schedules, the frequency policy naturally acts as an implicit, data-driven dynamic balancer. The network dynamically adjusts the rate of change Dq\mathbf D_q for each band conditioned on the current state xts\mathbf x_{t_s}.

As shown in Figure 3(c)–(d) in our revised paper, our frequency policy supervises the establishment of global structure with higher low-frequency loss weights during early denoising, and the refinement of local details with higher high-frequency loss weights during late denoising, consistent with our observations in Figure 3(a)–(b). CacheFlow adaptively learns the evolution rates at different scales and provides implicit dynamic reweighting during optimization, better accommodating the evolving dynamics across different denoising stages.

Due to space constraints, we continue in Response Part 2.

AuthorsOpenReview ↗

Response to Reviewer VFUN Part 2/3

3. Frequency policies achieve lower fitting errors and better generation quality

To further evaluate the benefits of CacheFlow's implicit, scale-aware temporal reweighting mechanism, we compare the fitting accuracy of the Haar-domain policy and the spatial-domain linear policy on FLUX.1-dev. The experiments use 3,972 prompts from GenEval, DPG-Bench, and OneIG-Bench. We compare the overall velocity error Ev=NMSE(v^,v)E_v=\mathrm{NMSE}(\hat v,v), the low- and high-frequency velocity errors EL=NMSE(v^L,vL)E_L=\mathrm{NMSE}(\hat v_L,v_L) and EH=NMSE(v^H,vH)E_H=\mathrm{NMSE}(\hat v_H,v_H), and the velocity-change errors relative to the interval start ELΔ=NMSE(Δv^L,ΔvL)E_L^{\Delta}=\mathrm{NMSE}(\Delta\hat v_L,\Delta v_L) and EHΔ=NMSE(Δv^H,ΔvH)E_H^{\Delta}=\mathrm{NMSE}(\Delta\hat v_H,\Delta v_H), evaluating the fitting accuracy of the teacher velocity and its evolution within each frequency band. As shown in Table 1, compared with the spatial-domain linear policy, the Haar-domain policy reduces the overall velocity error by 49.2%, the low- and high-frequency velocity errors by 47.1%/56.7%, and the low- and high-frequency velocity-change errors by 11.9%/26.5%, respectively. This demonstrates that CacheFlow fits the teacher velocity field more accurately by explicitly modeling the velocity of each frequency band and its temporal evolution.

In terms of generation quality, Table 5 in our revised paper shows that the Haar-domain velocity policy achieves GenEval and DPG-Bench scores of 0.86 and 88.47, respectively, which decrease to 0.83 and 86.72 with the spatial-domain linear policy. This demonstrates that CacheFlow's adaptive modeling of the evolution of different frequency bands translates into practical gains in generation quality, supporting the effectiveness of frequency-aware parameterization for few-step distillation.

Table 1: Velocity fitting errors.

MethodOverall velocity error EvE_v ↓Low-frequency velocity error ELE_L ↓High-frequency velocity error EHE_H ↓Low-frequency change error ELΔE_L^{\Delta} ↓High-frequency change error EHΔE_H^{\Delta} ↓
Spatial-domain linear policy0.2660.3120.1800.8910.907
Haar-domain policy (Ours)0.1350.1650.0780.7850.667

4. Revisions to the manuscript

To eliminate potential confusion, we have revised the corresponding text:

  • We toned down the claim about removing a monolithic temporal parameterization in Lines 49–51 and 91–95 to avoid the implication of expanding the formal temporal function class.
  • We incorporated the reviewer's theoretical perspective and explained gradient decoupling and implicit dynamic reweighting in Lines 209–234 and 250–259, supported by the frequency dynamics and bandwise loss analysis in Figure 3(a)–(d) in our revised paper.

Due to space constraints, we continue in Response Part 3.

AuthorsOpenReview ↗

Response to Reviewer VFUN Part 3/3

W2. Comparison with additional distribution-matching distillation methods

A2: Thank you for your valuable suggestion. CacheFlow's policy distillation and distribution-matching methods such as DMD2, TDM, and decoupled-DMD follow distinct acceleration approaches. DMD-based methods align student and teacher distributions through complex matching objectives and additional adversarial losses, often causing training instability and mode collapse. CacheFlow instead predicts an interval-wise velocity policy in a single forward pass, advances the trajectory through low-cost substeps, and directly matches the teacher velocity. This simple supervision and optimization enable stable training and efficient generation on large-scale diffusion models.

We further compare CacheFlow, DMD2, TDM, and decoupled-DMD on FLUX.1-dev. As shown in Table 2, CacheFlow substantially outperforms these methods in generation quality. DMD2, TDM, and decoupled-DMD incur drops of 13.6%/2.3%, 22.7%/4.5%, and 1.5%/0.3% on GenEval/DPG-Bench, respectively. In contrast, 2-NFE CacheFlow achieves 0.66/84.72 with a 22.89× speedup. With caching, CacheFlow-Medium and CacheFlow-Fast further increase the speedup to 26.34×/29.20× while maintaining competitive generation quality. We have added these comparisons to Table 2, Lines 282–292 in our revised paper.

Table 2: Comparisons with DMD2, TDM and decoupled-DMD.

MethodNFE ↓Latency ↓Speed ↑GenEval ↑DPG-Bench ↑
FLUX.1-dev509.8151.00×0.6684.16
DMD220.42623.04×0.5782.24
TDM20.42722.96×0.5180.37
decoupled-DMD20.42822.95×0.6583.89
CacheFlow (w/o Cache)20.42922.89×0.6684.72
CacheFlow-Medium1.750.37326.34×0.6684.36
CacheFlow-Fast1.580.33629.20×0.6584.08
Reviewer_58ofOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CacheFlow, which combines policy-based step distillation (pi-flow) with learnable feature caching for few-step generation. Motivated by "asynchronous frequency evolution", the student predicts the velocity policy per Haar band as a linear function of t. At the second step of each step pair, a block segment is skipped and replaced by a lightweight MLP that takes the cached residual of the previous step. Both parts are trained jointly with teacher velocity matching. Experiments on FLUX.1-dev and Qwen-Image variants show teacher-level quality at 1.58 at 2 NFE, with up to 55 times speedup on the DiT path.

Critical strengths
  • Joint training of caching and distillation is simple (no distributional matching loss such as adversarial or dmd loss) and works consistently across several large backbones and tasks.
Critical weaknesses
  • The frequency-aware policy does not reflect the frequency motivation. Since the inverse Haar transform is linear, the policy is equivalent to a spatial linear policy of prior methods (pi-flow). Moreover, unlike the original pi-flow policies, it does not depend on the current state x_t, so the trajectory within each step is a fixed parabola. This is less expressive than pi-Flow, and it is unclear where the gain comes from.

  • The caching part is essentially depth pruning of the second step, i.e., removed blocks are replaced by a small network. This is already explored in block replacement for DiTs (e.g., FastFLUX [1]) and in learnable caching for distilled models (e.g., DisCa). There is no comparison with jointly trained block skipping without the cache, so it is unclear whether cross-step caching actually matters. The gain over the distilled student is also marginal (~1.2x).

[1] Cai, Fuhan, et al. "FastFLUX: Pruning FLUX with Block-wise Replacement and Sandwich Training." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 40. No. 4. 2026.

[2] Zou, Chang, et al. "DisCa: Accelerating Video Diffusion Transformers with Distillation-Compatible Learnable Feature Caching." arXiv preprint arXiv:2602.05449 (2026).

Questions for authors
  • Where does the gain of the Haar-domain policy over the spatial linear policy come from, given that both span the same function class?

  • How does CacheFlow compare with jointly trained block skipping without the cache, under the same NFE?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 58of Part 1/3

Thank you very much for your careful review and constructive suggestions. Your feedback has been invaluable in improving our work.

W1/Q1. Advantages of Haar-domain velocity policies

A1: Thank you for your valuable analysis. We agree that the Haar-domain policy and a spatial linear policy span the same temporal function class, and appreciate this new perspective for analyzing CacheFlow. However, we would like to clarify that the core rationale behind our Haar-domain policy lies in its architectural inductive bias and optimization dynamics, which effectively functions as an implicit, scale-aware temporal reweighting mechanism rather than a mere coordinate transformation, ultimately achieving higher fitting accuracy and better generation quality than the spatial linear policy. We have added this explanation in Lines 209–234 and 250–259, together with Table 8, Lines 459–468 in our revised paper.

1. Spatial parameterization faces spectral bias and capacity competition

In diffusion distillation, temporal dynamics across different noise levels are highly non-uniform. As analyzed in Figure 3(a)–(b) in our revised paper, low-frequency components exhibit larger relative changes during early denoising, while high-frequency components exhibit larger relative changes during later denoising. Directly modeling a monolithic spatial velocity v(t)=B~+[t−ts]D~v(t)=\tilde{\mathbf B}+[t-t_s]\tilde{\mathbf D} forces high- and low-frequency components to share the same spatial parameterization channels. Because standard MSE gradients are heavily dominated by low-frequency energy, the network struggles to capture the subtle yet crucial temporal evolution D~\tilde{\mathbf D} of high-frequency textures without carefully designed, hand-crafted timestep weighting schedules w(t)w(t).

2. Haar-domain policy as implicit dynamic multi-scale reweighting

Our Haar-domain policy acts as an implicit, dynamic multi-scale reweighting mechanism that better accommodates the optimization demands of few-step distillation. Specifically, Haar decomposition maps velocity block z\mathbf z with mean μ\mu to Hz=[L,G1,G2,G3]⊤H\mathbf z=[L,G_1,G_2,G_3]^\top, where L=2μL=2\mu and ∑j=13Gj2=∑i=14[zi−μ]2\sum_{j=1}^3G_j^2=\sum_{i=1}^4[z_i-\mu]^2. The low-frequency component thus encodes the average structure, while the high-frequency components capture local details. By predicting band-specific parameters uθ,q(t)=Bq+[t−ts]Dq\mathbf u_{\boldsymbol\theta,q}(t)=\mathbf B_q+[t-t_s]\mathbf D_q, our policy structurally decouples the gradient paths across frequency bands in the network head:

  • Each band q∈LL,LH,HL,HHq\in\\{LL,LH,HL,HH\\} independently learns its baseline response Bq\mathbf B_q and adaptive temporal drift rate Dq\mathbf D_q.
  • Rather than relying on heuristic temporal loss weighting such as Min-SNR or empirical SNR schedules, the Haar-domain policy naturally acts as an implicit, data-driven dynamic balancer. The network dynamically adjusts the rate of change Dq\mathbf D_q for each band conditioned on the current state xts\mathbf x_{t_s}.

As shown in Figure 3(c)–(d) in our revised paper, our frequency policy supervises the establishment of global structure with higher low-frequency loss weights during early denoising, and the refinement of local details with higher high-frequency loss weights during late denoising, consistent with our observations in Figure 3(a)–(b). CacheFlow adaptively learns the evolution rates at different scales and provides implicit dynamic reweighting during optimization, better accommodating the evolving dynamics across different denoising stages.

Due to space constraints, we continue in Response Part 2.

AuthorsOpenReview ↗

Response to Reviewer 58of Part 2/3

3. Haar-domain policies achieve lower fitting errors and better generation quality

To further evaluate the benefits of CacheFlow's implicit, scale-aware temporal reweighting mechanism, we compare the fitting accuracy of the Haar-domain policy and the spatial-domain linear policy on FLUX.1-dev. The experiments use 3,972 prompts from GenEval, DPG-Bench, and OneIG-Bench. We compare the overall velocity error Ev=NMSE(v^,v)E_v=\mathrm{NMSE}(\hat v,v), the low- and high-frequency velocity errors EL=NMSE(v^L,vL)E_L=\mathrm{NMSE}(\hat v_L,v_L) and EH=NMSE(v^H,vH)E_H=\mathrm{NMSE}(\hat v_H,v_H), and the velocity-change errors relative to the interval start ELΔ=NMSE(Δv^L,ΔvL)E_L^{\Delta}=\mathrm{NMSE}(\Delta\hat v_L,\Delta v_L) and EHΔ=NMSE(Δv^H,ΔvH)E_H^{\Delta}=\mathrm{NMSE}(\Delta\hat v_H,\Delta v_H), evaluating the fitting accuracy of the teacher velocity and its evolution within each frequency band. As shown in Table 1, compared with the spatial-domain linear policy, the Haar-domain policy reduces the overall velocity error by 49.2%, the low- and high-frequency velocity errors by 47.1%/56.7%, and the low- and high-frequency velocity-change errors by 11.9%/26.5%, respectively. This demonstrates that CacheFlow fits the teacher velocity field more accurately by explicitly modeling the velocity of each frequency band and its temporal evolution.

In terms of generation quality, Table 5 in our revised paper shows that the Haar-domain velocity policy achieves GenEval and DPG-Bench scores of 0.86 and 88.47, respectively, which decrease to 0.83 and 86.72 with the spatial-domain linear policy. This demonstrates that CacheFlow's adaptive modeling of the evolution of different frequency bands translates into practical gains in generation quality, supporting the effectiveness of frequency-aware parameterization for few-step distillation.

Table 1: Velocity fitting errors.

MethodOverall velocity error EvE_v ↓Low-frequency velocity error ELE_L ↓High-frequency velocity error EHE_H ↓Low-frequency change error ELΔE_L^{\Delta} ↓High-frequency change error EHΔE_H^{\Delta} ↓
Spatial-domain linear policy0.2660.3120.1800.8910.907
Haar-domain policy (Ours)0.1350.1650.0780.7850.667

Due to space constraints, we continue in Response Part 3.

AuthorsOpenReview ↗

Response to Reviewer 58of Part 3/3

W2/Q2. Comparison with jointly trained block skipping without caching

A2: Thank you for your valuable suggestion. Compared with FastFLUX and DisCa, CacheFlow jointly optimizes step distillation and cross-step caching, enabling the student and cache predictor to adapt together to sparse-step feature dynamics. Following your suggestion, we compare CacheFlow with FastFLUX and DisCa under the same NFE budget to further demonstrate the advantages of our caching mechanism. Details are provided below. We have added these comparisons to Table 2, Lines 282–292 in our revised paper, with accompanying discussion in Lines 421–423.

1. Clarification of CacheFlow's cross-step caching mechanism

CacheFlow enables co-adaptation between the few-step student and the cache predictor through joint optimization, providing a key distinction from FastFLUX and DisCa. Specifically, FastFLUX replaces selected Transformer blocks with linear layers operating on current-step features, without reusing information from preceding steps. DisCa uses historical cached features but trains its cache predictor separately after step distillation, limiting co-adaptation with the student. In contrast, CacheFlow predicts the current-step block-segment residual from the preceding-step cache and jointly optimizes the student and cache predictor under unified teacher-velocity supervision. The predictor thus learns the evolving cross-step feature dynamics while the student adapts to cached feature approximations, reducing caching errors and preserving generation quality under sparse sampling.

2. CacheFlow outperforms FastFLUX and DisCa across multiple settings and delivers substantial speed gains

We further compare CacheFlow, FastFLUX, and DisCa on FLUX.1-dev to demonstrate the advantages of CacheFlow's caching mechanism. Since neither method provides publicly available training code, we reimplement both methods and adapt DisCa, originally designed for video generation, to the text-to-image setting. For both methods, we skip the same Transformer blocks specified in our paper to achieve 1.75 NFE. As shown in Table 2, FastFLUX and DisCa incur drops of 74.2%/46.6% and 25.8%/10.1% on GenEval/DPG-Bench relative to the original model, respectively. In contrast, CacheFlow achieves 0.66/84.36 with a 26.34× speedup while preserving the original model's generation quality. Compared with the substantial degradation in generation quality caused by block replacement methods and stagewise cache training, CacheFlow's joint training and learnable cross-step caching better accommodate feature changes in few-step generation, achieving substantial acceleration while preserving the original model's generation quality.

As shown in Tables 1 and 3 in our revised paper, CacheFlow-Fast reduces inference latency from 0.429/0.407 seconds for the students without caching to 0.336/0.333 seconds on FLUX.1-dev and Qwen-Image, respectively, providing an additional 1.28/1.22× speedup. Relative to the original multi-step models, the acceleration ratios increase from 22.89/45.35× to 29.20/55.31× while maintaining nearly the same generation quality. These further increases of 6.31/9.96 in the acceleration ratios represent substantial gains in inference efficiency.

Table 2: Comparisons with FastFLUX and DisCa.

MethodNFE ↓Latency ↓Speed ↑GenEval ↑DPG-Bench ↑
FLUX.1-dev509.8151.00×0.6684.16
FastFLUX1.750.38125.78×0.1744.98
DisCa1.750.38725.36×0.4975.64
CacheFlow1.750.37326.34×0.6684.36

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response

Summary of Review Responses

We sincerely thank all reviewers for their careful and insightful suggestions and the time and effort devoted to evaluating our work. Your reviews have helped us further improve CacheFlow, including but not limited to clearer explanations of its mechanisms and additional experiments and details to enrich the content.

We thank the reviewers for their insights into frequency-aware modeling and learnable caching, which have helped us better clarify CacheFlow’s motivation and theoretical perspective. CacheFlow’s band-specific velocity parameterization acts as an implicit, scale-aware temporal reweighting mechanism, adapting policy learning to the distinct dynamics of low-frequency structure and high-frequency details. Rather than simply combining step distillation and feature caching, CacheFlow jointly trains the student and cache predictor through a unified and simple velocity-matching objective, enabling co-adaptation to the enlarged sampling intervals of few-step generation, improving teacher-velocity approximation while reducing caching errors.

We also thank the reviewers for encouraging broader comparisons and deeper empirical analysis, which further demonstrate CacheFlow’s quality–efficiency advantages. Additional comparisons with DMD2, TDM, decoupled-DMD, MeanFlow, FastFLUX, and DisCa show that CacheFlow maintains strong generation quality under very low computational budgets. On FLUX.1-dev, it achieves GenEval/DPG-Bench scores of 0.66/84.72 at 2 NFE with a 22.89× speedup. Learnable caching further increases the speedup to 26.34× at 1.75 NFE while maintaining scores of 0.66/84.36. These results support the effectiveness of CacheFlow’s unified design.

Summary of Changes

The following changes have been made in our revised paper and are highlighted in blue:

  • Clarified the frequency-aware policy. Revised the parameterization claims in Lines 49–51 and 91–95, explained the motivation and mechanism of implicit dynamic reweighting in Lines 214–234, and added bandwise loss decomposition and analysis in Figure 3(c)–(d), Lines 250–259. (Reviewers 58of, VFUN, and ubjq)
  • Defined the statistics underlying Figure 3. Added the Normalized Change formula, coefficient definitions, and benchmark sources in Lines 209–213. (Reviewer ubjq)
  • Expanded baseline comparisons. Added comparisons with DMD2, TDM, decoupled-DMD, MeanFlow, FastFLUX, and DisCa in Table 2, Lines 282–292 in our revised paper, with accompanying discussion in Lines 415–416 and 421–423. (Reviewers 58of, VFUN, and ubjq)
  • Added velocity fitting analysis. Compared Haar-domain and spatial-domain linear policies using overall velocity, bandwise velocity, and bandwise velocity-change errors in Table 8, Lines 459–468 in our revised paper. (Reviewers 58of, VFUN and ubjq)
AuthorsOpenReview ↗

Discussion Reminder for Reviewers

Dear Reviewers,

Thank you once again for your constructive and detailed feedback. We have carefully revised the manuscript following your recommendations.

With the discussion deadline approaching in less than three days, we are yet to hear further feedback on our rebuttal. We welcome further discussion and feedback. Please feel free to reach out if you require any clarification or additional information.

Dear Reviewers,

The authors have submitted their responses to your reviews. Please take some time to carefully examine their rebuttals and engage in the discussion by sharing your thoughts, remaining concerns, or follow-up questions.

As the discussion deadline is approaching, I would appreciate your timely participation to allow sufficient time for further exchanges with the authors.

Thank you for your time and valuable contributions.

Best regards, AC

AuthorsOpenReview ↗

Discussion Reminder for Reviewers

Dear Reviewers,

Thank you again for your thoughtful and constructive feedback. We have carefully updated the manuscript in accordance with your suggestions.

With less than one day remaining before the discussion deadline, we would appreciate any further feedback on our responses and revisions. We welcome continued discussion and would be happy to provide any clarification or additional information.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.