acceptodds
Under review as a conference paper at ICLR 2027

HoltSeer: Stable Feature Forecasting under Sparse Anchors for Accelerated Diffusion Sampling

Abstract

Feature caching offers training-free acceleration for diffusion models by skipping selected network evaluations. Cache-then-reuse methods substitute cached features, whereas cache-then-forecast methods predict their temporal evolution. At high acceleration ratios, computed features become sparse, making forecasts over the intervening timesteps increasingly sensitive to local modeling errors. The challenge is to represent local feature evolution accurately with a lightweight predictor under this limited computation budget. We propose **HoltSeer**, a training-free cache-then-forecast method that models local diffusion feature evolution through a smoothed level–trend state inspired by Holt's exponential smoothing. The level component summarizes recent cached features, the trend component tracks local directional changes, and a trend-scaling coefficient controls extrapolation. We implement this predictor through a bounded sequence of recursive forecasts with periodic full network evaluations; predicted features enter the same state update as computed features. Experiments on FLUX.1-dev, Qwen-Image, and Wan2.1 show improved reference-based fidelity over the evaluated caching baselines at large cache intervals while retaining competitive latency. Our code will be released upon acceptance.

Primary area:
generative models
Keywords:
Diffusion Model, Accelerate inference, Cache

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JpxnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HoltSeer, a training-free diffusion model acceleration method. It addresses the issues of sparse feature calculation and prone-to-false prediction of features under high acceleration. The method draws inspiration from the Holt exponential smoothing. The horizontal state summarizes recent features, the trend state describes the direction of change, and the scaling coefficient controls the extrapolation amplitude. The predicted features participate in subsequent updates and are periodically refreshed through the complete network calculation. The main contribution is to combine these designs into a lightweight feature predictor, which reduces network computation while better preserving the original full-step sampling results. The experiments cover FLUX.1-dev, Qwen-Image, and Wan2.1, including image and video generation tasks. The evaluation uses DrawBench and VBench2 prompts, comparing inference latency, computation volume, and generation result fidelity. The improvement under high acceleration settings is more obvious. The paper also conducts prediction feedback control, window size, and parameter sensitivity experiments. Overall, the results support the full method in improving reference fidelity under large cache intervals. However, the existing ablation studies have not fully separated the role of the smoothing mechanism from other designs.

Critical strengths

The fidelity improvement is significantly enhanced under high acceleration, and there is evidence of matching the computational budget. Under the N=10 setting of Qwen-Image, the report delay and FLOPs of HoltSeer and TaylorSeer are the same, but SSIM increases from 0.4635 to 0.7242, and LPIPS decreases from 0.6128 to 0.4191. This indicates that, at least in this setting, the benefits do not come from increasing the computational load, but rather better preservation of the full-step sampling results at the same cost. This is the most compelling evidence supporting the acceptance of the paper. It reflects the actual speed - fidelity improvement, rather than just intuitive design in terms of method. The application value of the method has been cross-model verified, and the additional memory overhead is low. The main experiments use the same set of smoothing and trend scaling parameters, covering FLUX.1-dev, Qwen-Image, and Wan2.1, including image and video generation tasks. This enhances my confidence in its applicable range, not just the effectiveness on a single model. At the same time, the paper reports on two image models, with only an increase of 0.14 GB and 0.17 GB of memory on the original baseline. This makes its performance improvement more clearly have deployment significance, and is another core basis for my positive evaluation of this work.

Critical weaknesses

The incremental value of the core smoothing design has not been fully verified. Equation (10) indicates that when K = 2, the two smoothing parameters completely cancel out, and the method degenerates into a simple two-point damping extrapolation. However, in Table 6, under the FLUX.1-dev and N = 6 settings, this simplified version of SSIM and LPIPS are superior to the main configuration K = 3. The existing parameter sensitivity experiments have not independently separated the effects of smoothing and damping. Therefore, the experiments support the effectiveness of the complete scheme, but have not yet explained how much additional value the core smoothing design provides compared to the simple damping extrapolation. This directly affects my judgment of the contribution strength of the method, and is the main reason for rejecting the manuscript. There is a substantial logical gap in the theoretical motivation. Proposition 3.2 deduces a scaling relationship between the adjacent network outputs from the local smoothness assumption, but local smoothness does not guarantee this relationship. The authors also explicitly acknowledge this condition limitation. This does not invalidate the reported experimental benefits, but weakens the theoretical basis of the method. Combined with the insufficient mechanism verification, it makes me still inclined to reject the manuscript.

Questions for authors
  1. How much independent gain does the smoothing mechanism provide compared to trend damping? Please explain whether there are fixed window size, trend scaling coefficient, refresh schedule, and calculation budget, and only change the control results of the smoothing mechanism. Especially in the case of the most advantageous Qwen-Image with N=10 settings, which evidence can distinguish "the gain brought by smoothing" from "the gain brought by reducing the extent of extrapolation"?
  2. What necessary conditions does the scalar scaling relationship in Proposition 3.2 depend on? Assuming that Proposition 3.1 only requires local smoothness of residual features, Proposition 3.2 further provides a scaling relationship between adjacent network outputs. Local smoothness itself does not guarantee this conclusion. Please explain whether there are additional unmentioned assumptions and provide the key derivation to obtain this error order. Also, please clarify how to transition from the smoothness of the internal residual features to this relationship between network outputs, and which conclusions in the actual algorithm and subsequent error analysis rely on this proposition.
Reviewer_eTL8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses training-free feature caching for diffusion transformers at high acceleration ratios. HoltSeer treats the evolution of cached features as a time series and forecasts skipped features with a level–trend state inspired by Holt's exponential smoothing. The level smooths recent features, the trend tracks their changes, and a coefficient γ scales the trend during extrapolation. Forecasts use a window of K recent features. Predicted features are fed back into the window, and a full network evaluation is performed every N steps. Experiments on FLUX.1-dev and Qwen-Image (DrawBench) and Wan2.1 (VBench2 prompts) with N up to 10 report better SSIM/PSNR/LPIPS than prior caching methods at large intervals.

Critical strengths

HoltSeer is a simple method that remains robust under aggressive acceleration.

Critical weaknesses
  1. Why rolling prediction approach like HoltSeer should outperform previous methods that explicitly depend on fully computed features remains unexplained.
  • Many previous methods employ traditional numercal predictors for the task of feature caching in diffusion models (e.g., TaylorSeer, HiCache, AB-Cache, etc.). It is important to show why HoltSeer, which uses Holt's exponential smoothing, is preferable compared to these methods.
  • Although the authors show an error anlysis in lines 284~297, RNR_N itself is not bounded and grows as the rolling continues.
  • For K=2, α\alpha and β\beta cancel and each increment is multiplied by γ\gamma, so the forecast saturates at γ1−γ\frac{\gamma}{1-\gamma} times the last computed difference. HoltSeer is thus a damped, saturating extrapolation, whereas Taylor extrapolation grows polynomially with distance. This might explain the gains at large N, but it reflects damping, which existing methods could also implement without rolling prediction. A good example is GOC ("Accelerating Diffusion Transformer via Gradient-Optimized Cache", ICCV 2025).
  1. Some experimental results raise questions.
  • In Table 1, compared baselines change for different acceleration ratios.
  • Latency and speed is not provided for a result in Table 1. (FreqCa at the last block).
  • Many latency values in Talbe 1, 2, and 3 are exactly the same, which is unusal.
Questions for authors
  • γ\gamma looks like an additional hyperparameter that is not used for standard Holt's exponential smoothing. Why is this introduced, and how does HoltSeer perform if the interval size is used instead of γ\gamma?
  • Can you report per-step and cumulative prediction error over the cache interval for HoltSeer, reuse, and TaylorSeer?
  • How does HoltSeer perform with smaller NNs?
Reviewer_cM73OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes HoltSeer, an untrained cache-then-forecast method that simulates the evolution of local diffusion characteristics by leveraging Holt's exponential smoothing level-trend state model.

Critical strengths
  1. At high acceleration ratios, the fidelity improvement with the same computational load compared to the baseline is significant, and the direction is consistent across three models and two modalities.

  2. The readability is good; the methods and the experiment are relatively clear.

Critical weaknesses
  1. The experiment is not sufficiently comprehensive:

    a. Only compared with TeaCache, TC-Padé, and TaylorSeer on Wan2.1, with an obvious lack of baselines;

    b. No VBench dimension scores were reported for the video task,

  2. Is it necessary to smooth the statement? According to Eq. 8–10 and Appendix C.2: When K=2K=2, the predictor degenerates into a damped first-order extrapolation with coefficients, and there is no smoothing at all.

Questions for authors
  1. Why are the Latency and Speed fields missing in FreqCa of Table 1?
  2. Compare with more baselines in Wan2.1 and add VBench results.
  3. Explain the smooth issue mentioned in Weaknesses 2.

Comments on the paper

5 comments

Dear Reviewers,

As we are approaching the end of reviewer-author discussions, please go through authors' responses and raise questions that you would like authors to reply.

Thanks,

AC

AuthorsOpenReview ↗

Response to Concerns Regarding Experimental Reliability

Thank you for your feedback. We apologize for the misunderstanding caused by our earlier wording. First, we confirm that all experimental results reported in the main paper and our rebuttal were obtained from actual runs.

We provided our experimental code in the supplementary material at submission. To address your concerns, we additionally provide the code for all comparison methods to facilitate independent verification.

Although absolute values may vary across hardware environments, the relative ranking of the methods and the core conclusions of our paper remain unchanged in the comparisons we have reported. We sincerely hope you will take our assurance on this point.

We address your two concerns separately below.

1. Differences in Image-Quality Metrics Following a GPU Change

In particular, it remains unconvincing that image quality metrics would change so drastically solely due to a change in GPU hardware (e.g., a PSNR of 28.2043 in Table 1 vs. 25.1593 in the rebuttal table).

We would like to clarify that acceleration methods can exhibit substantial performance differences across hardware and evaluation environments. Related observations have been reported in both public projects and published papers:

  • vLLM-Omni: With the same publicly available TeaCache strategy applied to Qwen-Image 2.1, the RTX PRO 6000 evaluation and an independent H20 evaluation reported PSNR differences of 3.07 dB at CFG=1 and 4.70 dB at CFG=4.

  • Published evaluations of TaylorSeer: For FLUX.1-dev on DrawBench with 50 sampling steps, a resolution of 1024×1024, and N=6, O=1, Spectrum, MeanCache, and CEM reported PSNR values of 20.24, 15.831, and 15.59 dB on A100, H100, and A800 GPUs, respectively—a maximum difference of 4.65 dB.

In our experiments, the second-order term scaling experiment conducted during the rebuttal showed that the original TaylorSeer predictor (λ=1) achieved a latency speedup of 5.90× on the A100, whereas the corresponding configuration in Table 1 achieved 4.00× on the L40S. A substantial accompanying change in PSNR is therefore reasonable.

When the hardware changes, substantial variation in image-quality metrics under otherwise identical settings is normal and reasonable. We respectfully encourage the reviewer to examine related work further when assessing this issue, as this phenomenon is very common.

2. Differences Between the Latest Rerun and Table 1

However, the results for TaylorSeer still do not match those in Table 1.

To address your concern, we reran the experiment using an L40S GPU, matching the GPU model used for Table 1. We can confirm that the GPU model was the same, but we cannot guarantee that the versions of PyTorch, Diffusers, Transformers, and other software dependencies used during the rebuttal were identical to those used for the original experiments. These differences in the execution environment may contribute to variation in the rerun results.

Specifically, the differences between the rerun and the main paper are 0.2046 dB in PSNR, 0.0127 in SSIM, and 0.0107 in LPIPS. We consider these differences modest, and they do not alter the substantive conclusions of the comparison.

Thank you again for your careful review. We respectfully ask you to reconsider the substantive contributions of our work in light of the additional experiments, the complete experimental code we have already provided for both HoltSeer and the comparison methods, and reports of the same phenomenon in public projects and published papers.

We welcome further discussion and will be happy to provide additional information to support independent verification.

AuthorsOpenReview ↗

Discussion Summary (Part 1/2)

Dear Reviewers and Area Chair,

Thank you for reviewing HoltSeer and for the time you devoted to the discussion.

HoltSeer reveals a fundamental limitation of explicit trajectory extrapolation, an approach widely used in predictive caching for diffusion acceleration. Through extensive experiments and theoretical analysis, we demonstrate the potential of time-series methods to address this limitation. HoltSeer substantially mitigates the severe degradation in image quality that occurs at high acceleration ratios and points the community toward further exploration of time-series forecasting as a paradigm for predictive caching.

Following the rebuttal discussion, reviewers cM73 and Jpxn both gave positive assessments. We also had a detailed discussion with reviewer eTL8, who stated that most of their concerns had been addressed. Their remaining concern was the change in image-quality metrics we observed after switching GPUs. We explained that such variation is normal, cited other studies and projects that reported the same phenomenon, and provided the code for all experiments in the paper as supplementary material. We hope this addresses eTL8’s remaining concern.

Strengths recognized by the reviewers

  1. Strong performance at high acceleration ratios. All three reviewers recognized HoltSeer’s substantive contribution to mitigating the severe image-quality degradation that predictive caching methods exhibit at high acceleration ratios.

    eTL8: “HoltSeer is a simple method that remains robust under aggressive acceleration.”

    cM73: “At high acceleration ratios, the fidelity improvement with the same computational load compared to the baseline is significant, and the direction is consistent across three models and two modalities.”

    Jpxn: “The fidelity improvement is significantly enhanced under high acceleration, and there is evidence of matching the computational budget.”

  2. Thorough and extensive experimental evidence. cM73 highlighted the fidelity gain over the baseline at the same computational load and its consistency across three models and two modalities. Jpxn noted that the same smoothing and trend scaling parameters were used across image and video generation and that the controlled ablation showed an additional benefit from smoothing.

    cM73: “… the direction is consistent across three models and two modalities.”

    Jpxn: “The main experiments use the same set of smoothing and trend scaling parameters, covering FLUX.1-dev, Qwen-Image, and Wan2.1, including image and video generation tasks.”

    Jpxn, on the additional ablation: “These results support that smoothing has additional benefits of moderate amplitude and consistent direction.”

  3. Strong performance at the same cost and reliable gains for real-world deployment. cM73 and Jpxn agreed that HoltSeer substantially improves fidelity at the same computational cost, with consistent gains across models. Its additional GPU memory use is less than 0.2 GB on both image models, making these gains valuable for real-world deployment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.