acceptodds
Under review as a conference paper at ICLR 2027

SpectralCache: Accelerating Diffusion-Based World Models via Spectral Feature Caching

Abstract

Diffusion-based world models enable high-quality interactive environment generation but suffer from substantial inference overhead due to repeated Transformer evaluations during denoising. Existing caching methods mainly exploit temporal redundancy at the feature or token level, leaving the underlying mathematical structure of diffusion features largely unexplored. In this work, we reveal that world-model features exhibit highly stable singular subspaces across nearby denoising steps, while their singular values follow predictable evolution patterns. Building on this observation, we propose SpectralCache, a training-free spectral caching framework that reuses stable singular subspaces and estimates only low-dimensional singular values through linear extrapolation. We further exploit the spectral consistency between neighboring full-computation features to skip selected expensive backbone evaluations via singular value scaling. Extensive experiments on representative world models demonstrate that SpectralCache consistently improves inference efficiency while preserving generation quality. On HunyuanWorld-Voyager-13B, SpectralCache achieves 5.22 acceleration while maintaining a WorldScore of 65.90 for static scenes, substantially outperforming existing training-free caching methods in inference efficiency.

Primary area:
generative models
Keywords:
World model, diffusion, efficiency

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WFqHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This study faces a practically significant application involving world models, thus the challenge of balancing inference speed with generation quality. To tackle this, the authors propose SpectralCache, a training-free spectral caching framework that utilizes linear extrapolation and scaling of singular values during generation process. Additionally, based on singular value analysis, the authors uncover two key insights regarding the world model: first, the singular value subspaces and singular vectors exhibit stability across consecutive steps; second, adjacent singular values demonstrate a linear relationship, allowing for a predictable estimate via linear operations. Experimental results show that the proposed framework outperforms competitive baselines, particularly regarding latency and generation quality.

Critical strengths

The paper clearly articulate the motivation behind this line, making their objectives easy to maser, given the limitations of existing research on caching and the growing demand for world models, inference speed is indeed a crucial factor worth addressing. Moreover, the experimental evaluation is adequate, ranging from the experimental settings to ablation studies, demonstrating the significant effort invested in this research. The authors also provide substantial qualitative evidence and algorithm derivations in the appendix to support the proposed framework.

Critical weaknesses

The base models selected by this study, HunyuanVoyager-13B and Aether-5B both generate videos based on image, text, and camera trajectory inputs. While the image condition remains relatively static, serving as the reference for the subsequent video sequence, the camera trajectories should ideally undergo significant dynamic changes rather than remaining fixed during generation and the semantic complexity introduced by different prompts also influences generation quality. I am aware that the authors' primary work focuses on DiT visual representations, but I am curious that how do the authors view and address the impact of these varying conditioning modalities on spectral stability. Could the authors provide a small robustness validation varying prompt complexity and camera trajectory magnitude to examine whether the observed spectral stability remains consistent across different conditioning settings?

Beyond the overall performance gains compared to other training-free methods, I am more curious in seeing a breakdown of performance improvements or quality degradation based on specific categories, e.g., indoor vs. outdoor or simple vs. complex scenes. The authors could design small-scale experiments for this aspect.

As the authors clarify in the introduction, this is a challenge regarding deployment practices. Therefore, it is important to honestly evaluate performance differences across hardware. The original experiments utilized two NVIDIA Pro 6000 96GB GPUs, a very powerful hardware setup, so how would the inference acceleration performance hold up under less powerful hardware conditions?

From a reader's perspective, although these are minor issues, I still feel it is necessary to point them out so that the author can review and revise them (including but not limited to):

  • Please refer to Section 4.1 (CITATIONS WITHIN THE TEXT) in ICLR 2027 publicized template about the use of \citep{ } or \citet{ }
  • The table caption should be placed at the top to maintain consistency with the template.
  • The author should carefully check whether a comma or a period is used at the end of each equation.
  • Lacks a discussion of limitations and future work for a complete paper.
Questions for authors

Please see the questions raised in the Critical Weaknesses, particularly regarding spectral stability, model performance, and stable inference efficiency. I hope to engage in a constructive discussion with the authors during the discussion phase.

Reviewer_QyXsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SpectralCache, a training-free caching method to accelerate diffusion-based world models. By exploiting the temporal stability of feature subspaces and the predictable evolution of singular values across denoising steps, SpectralCache approximates intermediate features through spectral decomposition and selectively performs full computations based on an adaptive error-control mechanism. Experiments on two world models demonstrate consistent improvements in generation quality and inference speed over existing caching methods, with additional evaluations on 3D reconstruction.

Critical strengths
  1. Timely and relevant problem. Efficient inference for diffusion-based world models is an important research direction, given their substantial computational demands and growing potential.
  2. Interesting technical approach. The paper identifies temporal patterns in the spectral structure of intermediate features and leverages these observations to design a caching strategy tailored to diffusion-based world models.
  3. Comprehensive evaluation. The paper compares against multiple baselines and includes ablation studies and computational-overhead analysis to examine the effectiveness and practical feasibility of the proposed method.
Critical weaknesses
  1. Limited ablation of key design components. The paper provides sensitivity analyses of several hyperparameters, but does not fully isolate the contributions of the key components of SpectralCache.
  2. Generality of the spectral observations. The method relies on the temporal stability of feature subspaces and the predictability of singular values. It is unclear how consistently these spectral properties hold across different model architectures and generation settings.
Questions for authors
  1. Could the authors provide a more systematic ablation of the key components? How much does each component contribute to the overall speed–quality trade-off?
  2. Do the observed spectral properties hold consistently across different world model architectures, prompts, and generation settings? Are there scenarios in which the assumptions of subspace stability or singular-value predictability break down?
Reviewer_xhyPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SpectralCache, a training-free feature caching method for accelerating diffusion-based world models. The method reuses the stable singular subspaces of the most recent fully computed feature and estimates only the low-rank singular values. It does this either by linear extrapolation or by scaling the previously computed singular values with a constant factor. For cache scheduling, it adopts the existing approximation-error accumulation strategy. Experiments on two world models show better performance than previous baselines. However, there are concerns about the lack of mathematical justification and the quality drop caused by singular value scaling.

Critical strengths
  • It presents a relatively simple, training-free caching method that effectively accelerates diffusion-based world models, with negligible memory and computational overhead.
  • Experimental results show better quality and speedup than previous methods on both models, including 3D reconstruction metrics.
Critical weaknesses
  • Lack of theoretical justification: The method is supported only by empirical observations. The paper does not explain why the singular subspaces remain stable, why singular values evolve linearly, or how the approximation error propagates.
  • Limited effectiveness of singular value scaling: In Table 5, scaling greatly reduces latency (3.84× → 5.22×). Compared with the variant without scaling, however, the quality drop is relatively large (PSNR 26.09 → 25.06, SSIM 0.832 → 0.818).
  • Missing key ablation: Since the subspaces are nearly fixed, the method is close to a first-order linear extrapolation of the feature itself. The paper should compare against direct feature extrapolation under the same scheduler. Table 3 also shows that rank 32 performs almost as well as rank 128, which raises the question of how much the spectral formulation actually contributes.
Questions for authors
  • Can the same method be applied to diffusion-based video generation models?
  • At L200, the approximation is claimed to introduce only 0.71% error. Exactly which formula is used to compute this value?
  • In Table 6, the Scaling Approximation Step takes about three times longer than the Extrapolation Cache Step, even though scaling is simpler. What causes this difference?
3 comments

[Direct Feature Extrapolation Ablation]

We thank the reviewer for this valuable suggestion. We conducted an additional ablation on HunyuanWorld-Voyager-13B by directly applying linear extrapolation to the feature F under the same scheduler. As shown below, direct feature extrapolation achieves only 22.81 PSNR and 0.738 SSIM, substantially lower than our method (25.06 PSNR and 0.818 SSIM) and the WorldCache baseline (23.49 PSNR and 0.770 SSIM). These results demonstrate that extrapolating singular values yields substantially better generation quality, whereas directly extrapolating the full features performs even worse than the WorldCache baseline.

Table: Comparison of direct feature extrapolation and spectral-based extrapolation on HunyuanWorld-Voyager-13B.

MethodPSNR ↑SSIM ↑
Direct Feature Extrapolation22.810.738
WorldCache23.490.770
Ours25.060.818

[Rank Sensitivity]

RankPSNR ↑SSIM ↑
Voyager∞1.000
Rank=3224.670.806
Rank=6424.880.813
Rank=12825.060.818
Rank=25625.120.820

As shown above, increasing the rank from 32 to 256 consistently improves PSNR from 24.67 to 25.12 and SSIM from 0.806 to 0.820. Notably, even Rank=32 substantially outperforms direct feature extrapolation (24.67 vs. 22.81 PSNR; 0.806 vs. 0.738 SSIM), while higher ranks further improve reconstruction quality. Together, these results demonstrate the effectiveness of singular-value extrapolation over direct feature extrapolation, even at low ranks.

[Generalization to Diffusion-Based Video Generation]

We thank the reviewer for this valuable question. To evaluate the generalizability of SpectralCache beyond diffusion-based world models, we additionally apply our method directly to HunyuanVideo, a diffusion-based video generation model. We follow the same generative experimental settings as TaylorSeer for a fair comparison such as frames and denoising steps. As shown below, SpectralCache achieves a PSNR of 16.85, an SSIM of 0.732, and a 5.27× speedup, demonstrating its applicability to diffusion-based video generation models.

MethodPSNR ↑SSIM ↑Speedup ↑
FORA (N=5)16.0720.63344.74×
ToCa (N=5,R=90%N=5, R=90\%)16.1430.64244.09×
DuCa (N=5,R=90%N=5, R=90\%)16.1690.64434.58×
TeaCache (α\alpha=0.4)16.0720.62164.72×
TaylorSeer (N=5,O=1)16.7960.70394.65×
TaylorSeer (N=6,O=1)16.2410.72155.13×
SpectralCache (Ours)16.8500.73205.27×

We further observe that the average similarities of the U and V subspaces between adjacent denoising steps on HunyuanVideo are 0.94 and 0.92, respectively, which are lower than those observed in diffusion-based world models (0.99 and 0.99, respectively). This relatively weaker subspace stability may explain the more modest PSNR improvement on HunyuanVideo, as our method relies on reusing highly stable singular subspaces across denoising steps. This observation further suggests that the particularly strong singular subspace stability identified in our paper is more pronounced in diffusion-based world models, highlighting the distinctive characteristics of their spectral dynamics and the suitability of SpectralCache for exploiting them.

[Effectiveness of Singular-Value Scaling]

While singular-value scaling introduces a modest reduction in generation quality compared with the variant without scaling, the resulting quality remains substantially higher than that of the strongest baseline, WorldCache (25.06 vs. 23.49 PSNR; 0.818 vs. 0.770 SSIM). More importantly, scaling significantly improves inference speed, achieving a 5.22× speedup compared with WorldCache's 3.65×. Without scaling, the speedup is only 3.84×, offering a relatively marginal improvement over WorldCache. Therefore, singular-value scaling plays a crucial role in achieving a substantially better speed–quality trade-off, delivering significantly faster inference while maintaining superior generation quality compared with existing best baselines.

[Scaling Approximation Overhead]

The Extrapolation Cache Step only requires linear extrapolation to reconstruct the features. In contrast, the Scaling Approximation Step involves additional operations beyond singular value scaling, including computing the velocity, acceleration, and curvature of each token, followed by sorting tokens according to their curvature to re-identify chaotic tokens [1], as described in our Method section and Figure 4. These additional operations account for the higher latency of the Scaling Approximation Step compared with the Extrapolation Cache Step, despite the simplicity of the scaling operation itself.

[1] Feng W, Fan G, Qin H, et al. Worldcache: Accelerating world models for free via heterogeneous token caching[J]. arXiv preprint arXiv:2603.06331, 2026.

[Theoretical Justification of SpectralCache]

We thank the reviewer for this valuable comment. We provide some analysis from three perspectives of stability of the dominant singular subspaces, local linear predictability of singular values, decomposition and propagation of feature approximation errors.

(1) Stability of Dominant Singular Subspaces

Consider the feature matrices at two consecutive denoising steps:

Ft+1=Ft+Et,F_{t+1} = F_t + E_t,

where EtE_t denotes the feature perturbation between adjacent steps. We define the spectral gap at the rank-r truncation boundary as

γt=σr(Ft)−σr+1(Ft).\gamma_t = \sigma_r(F_t) - \sigma_{r+1}(F_t).

According to Wedin's singular subspace perturbation theorem, when the perturbation satisfies ∥Et∥2≤γt/2\|E_t\|_2 \leq \gamma_t/2, the dominant left and right singular subspaces satisfy the following bound:

max⁡(∥sin⁡Θ(Utr,Ut+1r)∥2,∥sin⁡Θ(Vtr,Vt+1r)∥2)≤2∥Et∥2γt\max(\lVert\sin\Theta(U_t^r,U_{t+1}^r)\rVert_2,\lVert\sin\Theta(V_t^r,V_{t+1}^r)\rVert_2)\leq\frac{2\lVert E_t\rVert_2}{\gamma_t}

Here, Θ\Theta denotes the principal angles between the corresponding singular subspaces. This result indicates that when the feature perturbation is sufficiently small relative to the spectral gap, the rotation of the dominant singular subspaces is bounded (We evaluate the Wedin perturbation bound on HunyuanWorld-Voyager and observe an average value of approximately 0.114 for 2∥Et∥2/γt2\|E_t\|_2/\gamma_t, providing empirical support for the stability of the dominant singular subspaces).

(2) Local Linear Predictability of Singular Values

SpectralCache performs linear extrapolation of singular values using two preceding full-computation steps. Let σ(t)\boldsymbol{\sigma}(t) denote the vector containing the top-r singular values.

Assume that the singular-value trajectory is twice continuously differentiable over a local interval [m,p] and satisfies [1,2]:

sup⁡t∈[m,p]∥σ′′(t)∥2≤M.\sup_{t\in[m,p]}\|\boldsymbol{\sigma}''(t)\|_2 \leq M.

where M denotes an upper bound on the local curvature (i.e., the magnitude of the second derivative) of the singular-value trajectory.

Consider extrapolation from two steps m<n<p, with g=n-m and h=p-n. SpectralCache predicts the singular values as $\hat{\boldsymbol{\sigma}}_p=\boldsymbol{\sigma}_n+\frac{h}{g} (\boldsymbol{\sigma}_n-\boldsymbol{\sigma}_m).$

By the remainder bound for two-point linear extrapolation, we obtain

{\||\boldsymbol{\sigma}_p\||_2} \leq \frac{M}{2\||\boldsymbol{\sigma}_p\||_2}h(h+g).$$ This bound shows that the relative singular-value extrapolation error is governed by the normalized local curvature of the singular-value trajectory and the extrapolation horizon. When the singular values exhibit low second-order variation within a local interval, short-horizon linear extrapolation can achieve a small relative prediction error ( We evaluate the relative local curvature, $M/\||\boldsymbol{\sigma}_p\||_2$, on HunyuanWorld-Voyager and observe an average value of approximately **0.0187**, providing empirical support for the local linear predictability of singular values).

Dear authors,

Thank you for the detailed rebuttal addressing my concerns. Most of my concerns have been resolved by the rebuttal. Taking into account the other reviewers' comments together with the authors' rebuttal, I raise my score to 3.

Best regards,

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.