acceptodds
Withdrawn from ICLR 2027

CACO: Cost-Aware Dynamic Caching with On-Policy Distillation for Accelerating Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion Transformers (DiTs) achieve strong performance in high-resolution visual generation, but their iterative denoising requires repeated backbone computation, resulting in substantial inference cost. Feature caching accelerates sampling by reusing intermediate features and skipping backbone computation at selected steps. However, their practical effectiveness is limited by scheduling overhead and the accuracy of approximated features, with existing feature predictors further suffering from train–test mismatch caused by teacher-forcing training. In this paper, we propose CACO, a global–local framework that integrates global cache scheduling with local feature prediction to improve both scheduling efficiency and feature estimation accuracy. Specifically, we propose a Cost-Aware Global Scheduler (CAGS) that first allocates a fixed budget of refresh anchors and dynamically relocates refresh positions through sparse probing while preserving the refresh budget, thereby reducing the overhead of dynamic scheduling. For feature estimation, we present Time-Conditioned Feature Predictor (TCFP), which is first trained with supervised fine-tuning (SFT) on full-computation reference trajectories and subsequently refined with On-Policy Distillation (OPD) on cached trajectories to mitigate train–test mismatch during inference. Extensive experiments on image and video generation demonstrate that CACO substantially reduces inference cost while maintaining generation quality, enabling efficient practical deployment.

Primary area:
generative models
Keywords:
Accelerating Diffusion Transformers, DiT Cache, On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1CLkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CACO, a framework for accelerating diffusion models by combining a Cost-Aware Global Scheduler (CAGS) and a Time-Conditioned Feature Predictor (TCFP). CAGS uses predefined refresh anchors and sparse probing to dynamically adjust refresh positions under a fixed computation budget, aiming to reduce scheduling overhead. TCFP predicts intermediate features at reuse steps and is trained through SFT followed by On-Policy Distillation (OPD) to mitigate accumulated prediction errors.

The authors evaluate CACO on image and video generation models, including FLUX.1-dev, Wan2.1, and HunyuanVideo. Experimental results demonstrate improved reconstruction fidelity and competitive inference efficiency compared with existing caching methods. Ablation studies and qualitative comparisons further support the effectiveness of SFT and OPD.

Critical strengths

Strengths

  1. The paper is well-structured and comprehensive, with well-presented figures and tables.
  2. The generated images appear to have good visual quality.
  3. The ablation studies on SFT and OPD are well conducted, with intuitive and informative results that provide convincing evidence of their effectiveness.
Critical weaknesses

Weaknesses

  1. As discussed in Weakness 2 and Question 2, this paper offers only incremental improvements over prior work.
  2. The main difference between TCFP and LESA [1], a paper I previously read, are mainly in the loss functions. Why should these changes lead to improvements from a mechanistic perspective? I noticed that the appendix includes ablation studies on the four loss terms, showing that the second term, Lref-featL_{\mathrm{ref\text{-}feat}}, is the most important. However, the paper does not show how the method performs when using only Lref-featL_{\mathrm{ref\text{-}feat}} (as LESA does), nor does it provide a comparison with LESA.
  3. The evaluation metrics focus more on how closely the generated outputs match those of the full model. There is a lack of metrics evaluating whether the generated images themselves are natural and aligned with human preferences, such as ImageReward for image generation and VBench scores for video generation.
  4. The mathematical notation is a mess and lacks consistency. For example, around line 223, d^a\hat d_a is clearly inconsistent with d^pj\hat d_{p_j} in Equation (5).
  5. The code is not provided, making it difficult to conduct detailed comparisons with prior work.

Errors

  1. Around line 173, the range of tt is incorrectly specified. It should be from 00 to 11.
  2. Around line 368, the paper states, "while improving PSNR from 24.51 dB to 32.28 dB." However, 24.51 dB does not appear anywhere in the table, and 32.28 dB does not match the reported speedup either. I wonder where the value of 24.51 dB comes from. This raises concerns about the reliability of the reported data.

References

[1] Peiliang Cai, Jiacheng Liu, Haowen Xu, Xinyu Wang, Chang Zou, and Linfeng Zhang. "LESA: Learnable Stage-Aware Predictors for Diffusion Model Acceleration." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 43300–43309, 2026.

Questions for authors

Questions

  1. Regarding the "Reuse Risk and Relocation" section starting around line 213, according to the statement around line 681, "These discrete indices increase in sampling execution order and are distinct from the raw timesteps returned by the scheduler," and Algorithm 1 around line 864 (where xt+1x_{t+1} is computed after xtx_t), pjp_j should be executed before aja_j. In this case, when deciding whether to perform a full DiT computation at pjp_j, information from the future step aja_j should not be available. Conversely, if aja_j is executed before pjp_j, as implied by the description around line 215, how can CAGS "move the refresh from aja_j to pjp_j" as stated around line 227, given that the full DiT computation at aja_j has already been performed?
  2. Regarding CAGS, the authors state around line 230 that "l=1l=1 achieves a comprising trade-off between scheduling efficiency and performance." My understanding is that the refresh steps are initially determined, and subsequent adjustments are restricted to a one-step neighborhood around these predefined positions. Under this setting, other reuse steps would not be considered for full DiT computation. This approach seems to depart from heuristic scheduling methods (such as fixed-interval scheduling), but only to a limited extent. What happens when l=2l=2 or other values of ll are used? How do these settings affect computational efficiency and generation quality?
  3. Why are the feature extrapolation coefficients directly compared with the teacher features using MSE in Equation (7)? According to Equation (6), GθG_\theta predicts the extrapolation coefficients rather than the reconstructed features. Should the MSE instead be computed between the reconstructed features and the teacher features?

Summary and Suggestions

  1. Improve the clarity of the presentation, especially the mathematical notation.
  2. Provide a more detailed explanation of the rationale behind the loss design in TCFP, particularly the underlying mechanism that leads to performance improvements, rather than simply adopting these losses because they are effective. Otherwise, the approach seems more like an engineering trick than a mechanistic innovation.
  3. Conduct ablation studies on different values of ll in CAGS.
  4. I believe the above issues should be addressed before the paper can be considered for acceptance. Under the previous six-point rating scale, my assessment would be closer to 2 (Reject) than to 1 (Strong Reject). However, the current rating scale only offers 1 (Clear rejection) to express this level of rejection.
Reviewer_yzgzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CACO to accelerate Diffusion Transformer inference by combining CAGS, a dynamic scheduler under a fixed refresh budget, with TCFP, a lightweight feature predictor. TCFP is initialized through supervised training on full-model trajectories and subsequently refined through on-policy distillation on student-generated cache trajectories. Results on FLUX.1-dev, HunyuanVideo, and Wan2.1 demonstrate improved fidelity to full-model outputs at comparable inference latency.

Critical strengths

1.Consistent improvements across models, supported by measured latency. 2.The training strategy addresses a well-motivated distribution mismatch. 3.The predictor is lightweight, with several implementation details carefully specified

Critical weaknesses

1.CAGS contains an indexing inconsistency in its online definition and is incompletely specified. The probe position satisfies pj<ajp_j<a_j, yet Equation (4) requires features stored at the “actual refresh aja_j,” which has not occurred when pjp_j is reached. This may be a typographical error, but the method cannot be implemented literally as described. Initial-anchor fitting and collision handling are also unclear. Complete causal pseudocode should explain how the fixed refresh budget is preserved. [pp.4–5, Equations (4)–(5)]

2.Essential experimental settings and the complete training objective are missing. The paper omits training-prompt sources, dataset sizes, training–evaluation splits, optimizer, learning rate, training steps, and the actual gradient horizon HH. GPU, resolution, video length, and timing procedures are also unspecified. Furthermore, Equation (11) defines four losses, whereas Table 7 introduces an undefined “innovation regularization.” These omissions hinder reproducibility and leave unclear whether evaluation prompts were used for training or anchor calibration. [pp.6–7, 13, 16–19]

3.OPD’s benefits are not sufficiently separated from additional training and stronger supervision.

Questions for authors

see weakness

Reviewer_Wze1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CACO, a diffusion caching framework that combines cost-aware refresh scheduling with learned feature prediction to accelerate DiT inference. It further introduces a two-stage training strategy that combines supervised fine-tuning with on-policy distillation to improve prediction accuracy along cached inference trajectories. Experiments on image and video generation models demonstrate improved output fidelity over existing caching methods.

Critical strengths
  1. The paper addresses practical challenges in diffusion caching, considering both refresh scheduling and prediction error accumulation.
  2. The evaluation covers multiple image and video generation models and includes ablation studies, providing a reasonable empirical assessment of the proposed framework.
Critical weaknesses
  1. The contributions of individual design choices are not fully established. Although the paper includes component and loss ablations, the effectiveness of the key design choices and their contributions to the overall improvements remain insufficiently isolated. In particular, it is unclear whether the gains arise specifically from the proposed scheduling and on-policy training strategies or could be achieved through simpler alternatives.
  2. The training cost and generalization of the learned predictor remain unclear. The proposed method introduces additional training overhead, but its cost-effectiveness under different deployment scenarios is not sufficiently characterized. Furthermore, it is unclear whether the learned predictor generalizes to different inference configurations or requires retraining when the caching strategy or target acceleration level changes.
  3. The practical efficiency advantage is not sufficiently substantiated. Although the paper claims improvements in both efficiency and accuracy, the reported latency gains over competing caching methods are relatively modest in some experiments. The overhead introduced by feature prediction and refresh scheduling is not sufficiently characterized.
Questions for authors
  1. Can the authors further clarify the contributions of the key design choices and establish their advantages over simpler alternatives?
  2. Does the learned predictor generalize across different settings, and how does the additional training cost affect the overall efficiency?
  3. How does the overhead introduced by CACO compare with that of existing caching baselines?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.