acceptodds
Under review as a conference paper at ICLR 2027

RECAP: Reciprocal Sparse Attention and Hierarchical Caching for Efficient Video Diffusion

Abstract

Diffusion transformers generate high-quality videos but remain computationally expensive. Sparse attention and temporal caching reduce computation within and across denoising steps, respectively, offering a natural composition. However, predictions produced with sparse attention become sources for temporal reuse, coupling their errors along the sampling trajectory. We observe that improving the accuracy of these sources recovers fidelity without changing the reuse schedule, and that sparsification errors remain directionally aligned across nearby steps, suggesting that correction information can be shared rather than recomputed. These observations motivate RECAP, a training-free method that couples sparse attention with hierarchical caching through reciprocal correction, adaptively coordinated by online error feedback. Cached dense refinement corrects omitted interactions in current sparse estimates, while fresh sparse statistics track input changes and regulate the aging correction. The resulting predictions then renew the sources for whole-step prediction reuse, allowing prediction history to advance between dense refreshes. We evaluate RECAP on multiple video diffusion models against sparse-attention methods, caching methods, and their direct compositions. On HunyuanVideo at 720p 5-second generation, RECAP achieves a 3.05 speedup and outperforms all tested direct compositions in both fidelity and inference speed.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
diffusion transformer, video generation, sparse attention, caching

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_o8kaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies supplementing the error of sparse attention using prediction caching. The goal is to accelerate video diffusion. It uses dense refreshes to update the attention cache. It combines coarser tail estimates to compensate for the result of attention omitted by sparse computation. It accumulates the residual between sparse and dense computation, and performs a dense refresh every several steps to compensate. It caches the residual between exact and coarse tail estimates, and updates the cached residual at each dense refresh. The denoiser outputs are also used for the predictions. The method itself is training-free.

The experiments cover CogVideoX1.5, HunyuanVideo, and Wan2.1 for video and FLUX.1 for images. The comparison methods include sparse attention methods, caching methods, and combinations. Besides generation quality benchmarks, it also includes efficiency experiments compared with the non-sparse setting. The results show a good tradeoff between speed and fidelity. But the comparison seems to omit some methods without explanation, and some settings need further clarification.

Critical strengths
  1. The baseline coverage is broad. And the proposed method achieves good speed and fidelity results on the video and image benchmarks and models, which is quite comprehensive.
  2. The ablation of these components is also comprehensive. The ablation study includes refinement, reference updates, and prediction cache updates, which is reasonable.
  3. The experiment section reports the actual latency and GPU memory results, which help to grasp the deployment benefits of the method. These experiments also show the tradeoffs under different speed settings.
  4. The method design is clean and explainable. One dense refresh can provide for multiple sparse computations. And the cached features can be reused by subsequent prediction.
Critical weaknesses
  1. The paper discusses DiTFastAttn, LoSA, and ASTRAEA, but does not compare with them in the experiment section. It needs further explanation of why they are omitted.
  2. Error persistence experiments use cosine similarity, averaged over ten prompts. But similar directions do not mean that the magnitude is also consistent.
  3. One question about the design of the method. Between two dense refreshes, we must fix the selected partition, so that the residual and the coarse estimate correspond to the same omitted keys. This is reasonable, but if important interactions change between dense refreshes, the new important interactions may remain in the reconstructed detail, which can only be recovered approximately. So, in Table 3, if we freeze the initial refinement and partition, PSNR drops, but we can't attribute the result only to either the fixed residual or the fixed partition. The experiment didn't separate their effects.
Questions for authors
  1. The latency reported in Tables 1 to 7. Do these results include text encoding and VAE decoding? Or is it just the denoising step? The appendix states that the cache transfer and the method itself are included, but the definition of generation timing still needs further clarification.
  2. Why are the main CogVideoX and Wan experiments run on an RTX 4090, while HunyuanVideo and the CogVideoX use an A800?
  3. Why not compare with DiTFastAttn, LoSA, and ASTRAEA?
  4. What are the two fixed calibration prompts used for each model? What's the wall-clock time or GPU hours for the whole calibration process?
  5. What's the rationale for the threshold candidates? A small candidate set is acceptable, but I would like to understand whether the candidates cover the optimal speed and fidelity tradeoff.
  6. As mentioned in critical weakness 3, the cached residual can only be updated when a dense refresh is conducted. So, before a dense refresh, if there are drastic changes, how can the coarse statistics and drift weighting detect that the cached residual may not be accurate? If these changes are not detected in time between dense refreshes, the refinement error feedback can be late and fidelity can drop.
  7. The efficiency experiments about RECAP and the sparse-only baseline are impressive, but since RECAP includes many steps, what's the wall-clock time of each step? (For example, dense refresh, coarse tail estimation, residual refinement, and cache transfer.) The time breakdown is helpful to understand the latency of each component, but it's an additional suggestion. It's optional in the rebuttal period if time is limited.
8 comments

Thank you for your detailed reading and the many concrete questions. We are very glad that you recognize the broad baseline coverage, the comprehensive ablations, the reported latency and memory, and the clean and explainable design. Below we respond to your concerns and questions.

Q1: "The latency reported in Tables 1 to 7. Do these results include text encoding and VAE decoding? Or is it just the denoising step?"

A1: Yes, the reported latency includes text encoding and VAE decoding. It is the wall-clock time of the complete pipeline call, measured with CUDA synchronization. It covers text encoding, all denoising steps, VAE decoding, model offloading and the host–device transfers of RECAP, and it excludes model loading, warm-up and file writing. Every method is timed in the same harness throughout our experiments.


Q2: "Why are the main CogVideoX and Wan experiments run on an RTX 4090, while HunyuanVideo and the CogVideoX use an A800?"

A2: The GPU types follow the clusters that were available to us at each stage of the project. In the early stage, we only had access to RTX 4090 GPUs, and CogVideoX-1.5 and Wan2.1-1.3B fit within their 24 GB of memory. All main experiments on these two models therefore ran on RTX 4090 GPUs. HunyuanVideo needs more than 24 GB, and by the time we extended our evaluation to it, we had gained access to A800 GPUs, so its experiments ran there.

During the ablation study, we could not obtain uncontended RTX 4090 GPUs consistently enough to keep all latency measurements fair. We therefore regenerated the dense references on A800 GPUs and ran all CogVideoX-1.5 ablations there.

Throughout the project, every method within a table uses the same GPU type and is timed without contention from other processes, so the speedups within a table are always directly comparable.


Q3: "The paper discusses DiTFastAttn, LoSA, and ASTRAEA, but does not compare with them in the experiment section. It needs further explanation of why they are omitted." "Why not compare with DiTFastAttn, LoSA, and ASTRAEA?"

A3: We now compare RECAP with all three methods.

LoSA. LoSA appeared on arXiv in August 2026, within two months of the submission deadline, so it is concurrent work under the ICLR guidelines. Its code has also not been released, which prevents a faithful comparison. In essence, LoSA is a sparse-attention method. It selects key blocks at an early step and reuses these block indices in later steps. Its paper notes that it can be combined with feature caching methods, in which case the cache schedule, predictor and residuals stay unchanged. The paper pairs LoSA with D2Cache [1]. This is the direct-composition setting that our six compositions in Tables 2 and 7 already represent. For completeness, we still implemented LoSA following its paper, together with D2Cache, and report the results in Table A3a.

Table A3a. Comparison with our implementation of LoSA on Wan2.1-1.3B. The RECAP rows are from Table 7.

MethodPSNR↑SSIM↑LPIPS↓VBench↑Latency↓Speedup↑
LoSA24.120.86320.096581.90219s1.25×
RECAP-speed27.050.91410.074183.39131s2.08×
LoSA + D2Cache16.820.67340.273881.48117s2.34×
RECAP23.510.83230.123882.88105s2.60×

RECAP is both faster and more faithful in each comparison. RECAP-speed exceeds LoSA by 2.93 dB at a 2.08× speedup against 1.25×, and RECAP exceeds LoSA with D2Cache by 6.69 dB while running faster.

[1] Enhuai Liu et al. D2Cache: Second-order delta caching for higher video diffusion acceleration. In CVPR, 2026.

A3 (continued)

DiTFastAttn. The released code of DiTFastAttn supports only OpenSora, and every new model requires its offline calibration. We ported its three mechanisms to our video models, namely window attention with residual sharing, attention sharing across timesteps and attention sharing across CFG. On Wan2.1-1.3B, the calibration takes 16 GPU-hours on an A800 GPU with a peak memory of 30.21 GiB. On CogVideoX-1.5 and HunyuanVideo, the batched calibration of the original code runs out of memory. Even a reduced calibration with a single prompt takes about 41 GPU-hours on CogVideoX-1.5 and more than 280 GPU-hours on HunyuanVideo. We therefore report Wan2.1-1.3B results in Table A3b, with the same prompts and RTX 4090 GPUs as Table 7.

Table A3b. Comparison with DiTFastAttn on Wan2.1-1.3B. The RECAP rows are from Table 7.

MethodPSNR↑SSIM↑LPIPS↓VBench↑Latency↓Speedup↑
DiTFastAttn18.620.72910.218581.97232s1.18×
RECAP-speed27.050.91410.074183.39131s2.08×
RECAP-quality30.290.95200.037283.49158s1.73×

RECAP outperforms DiTFastAttn by a large margin in both fidelity and speed. RECAP-speed is 8.43 dB higher in PSNR while running 1.8 times faster, and RECAP-quality is 11.67 dB higher. We have added the LoSA and DiTFastAttn comparisons to Table 7 and Appendix E.1 of the revised paper.

ASTRAEA. The public release of ASTRAEA provides only its evolutionary search over timesteps. Its token selection module is proprietary and not released, and the paper does not detail it. The search is also costly. The authors report 69 and 139 GPU-hours for 2 s and 4 s videos on Wan2.1, and the search is repeated for each model, video length and computation budget.

A comparison with the full search would therefore not be on equal terms. We instead roughly match its search time to the calibration cost of RECAP. We ran a fresh search on Wan2.1-1.3B, capped at one GPU-hour, with the four prompts supplied with its code. To finish within the cap, we kept the released operators but used a population of two, with one mutation and one crossover offspring per generation. The search evaluated six candidates in 50.6 minutes. The selected schedule computes 26 steps.

Table A3c. Comparison with the public timestep scheduler of ASTRAEA on Wan2.1-1.3B. The RECAP rows are from Table 7. GPU-hours is the search or calibration cost.

MethodPSNR↑SSIM↑LPIPS↓CLIPIQA+↑VBench↑Latency↓Speedup↑GPU-hours↓
ASTRAEA17.410.67130.26620.50580.89164s1.66×0.84
RECAP-speed27.050.91410.07410.54083.39131s2.08×0.53
RECAP-quality30.290.95200.03720.54583.49158s1.73×0.65

With less search time, both RECAP settings are faster and much more faithful than ASTRAEA. RECAP-quality calibrates in 0.65 GPU-hours and exceeds ASTRAEA by 12.88 dB in PSNR, and RECAP-speed calibrates in 0.53 GPU-hours, exceeds it by 9.64 dB and runs 1.25 times faster.

Q4: "What are the two fixed calibration prompts used for each model? What's the wall-clock time or GPU hours for the whole calibration process?"

A4: All three models use the same two calibration prompts from VBench.

  1. "A sleek, black motorcycle with chrome accents leans into a sharp corner on a winding mountain road, the rider clad in a black leather jacket, matching helmet, and dark jeans. The sun casts long shadows, highlighting the bike's polished surface and the rider's focused posture. As the motorcycle rounds the bend, the tires grip the asphalt, kicking up a slight spray of gravel. The surrounding landscape features towering pine trees and a distant view of snow-capped peaks, adding to the sense of adventure and freedom. The rider's movements are fluid and precise, showcasing skill and control as the motorcycle smoothly navigates the curve."

  2. "A mesmerizing splash of turquoise water erupts in extreme slow motion, each droplet suspended in mid-air, creating a captivating dance of liquid. The vibrant turquoise hue shimmers under soft lighting, highlighting the fluid's graceful arcs and intricate patterns. As the splash unfolds, the droplets form delicate, crystalline shapes, almost like a choreographed ballet of water. The background is transparent, allowing the viewer to focus solely on the stunning motion and color of the water. The scene is both tranquil and dynamic, capturing the essence of fluidity and the beauty of nature in exquisite detail."

With the budget BB fixed, calibrating one operating point tests seven tolerance settings on the two prompts. We first vary τpred\tau_{\mathrm{pred}} over the four grid values with τref=0.1\tau_{\mathrm{ref}} = 0.1, and then vary τref\tau_{\mathrm{ref}} over the four grid values at the selected τpred\tau_{\mathrm{pred}}. One setting of the second pass repeats the first, so each operating point needs 14 videos. Table A4 reports the resulting GPU-hours.

Table A4. Calibration cost in GPU-hours. Each operating point runs 7 settings on 2 prompts. Times on CogVideoX-1.5 are estimated from the measured cost of each step type.

ModelQualitySpeedFast
CogVideoX-1.52.392.261.59
HunyuanVideo7.716.765.40
Wan2.1-1.3B0.650.530.45

All nine operating points on three models take 27.7 GPU-hours in total. For comparison, one DiTFastAttn calibration on CogVideoX-1.5 takes about 41 GPU-hours, and ASTRAEA reports 139 GPU-hours for one operating point of Wan2.1-1.3B.


Q5: "What's the rationale for the threshold candidates? A small candidate set is acceptable, but I would like to understand whether the candidates cover the optimal speed and fidelity tradeoff."

A5: Rationale. Both tolerances are relative errors, which do not depend on the scale of the model outputs, so one candidate set can serve both tolerances on all models. Thresholds of this scale are also typical in caching, where the released thresholds of TeaCache, NaviCache, EasyCache [2] and MagCache [3] lie between a few hundredths and a few tenths. Our factor-of-two grid {0.025, 0.05, 0.1, 0.2} covers this scale.

Values beyond the grid. Extending the grid at either end brings no benefit. On Wan2.1-1.3B and the same calibration prompts, Table A5 compares each end of the τpred\tau_{\mathrm{pred}} grid with one more value beyond it, with τref\tau_{\mathrm{ref}} fixed at 0.1.

Table A5. PSNR (dB) on Wan2.1-1.3B for τpred\tau_{\mathrm{pred}} at the two ends of the grid and beyond them.

Operating point0.0125 (beyond)0.025 (grid end)0.2 (grid end)0.4 (beyond)
Quality26.0926.4425.3925.39
Speed21.1123.7423.2223.22
Fast18.3219.6418.1218.03

A stricter τpred\tau_{\mathrm{pred}} of 0.0125 lowers PSNR by 0.35 to 2.63 dB, and a looser 0.4 gives the same or slightly lower PSNR than 0.2. For τref\tau_{\mathrm{ref}}, 0.4 gives the same videos as 0.2, and 0.0125 changes PSNR by at most 0.1 dB relative to 0.025 while making generation 3% to 9% slower than the chosen setting. We therefore did not further consider values beyond the grid.

Finer grids. A finer grid between these values may find slightly better settings, so we do not claim that the selected settings are optimal. We select each operating point by its fidelity on the two calibration prompts of A4, and these settings already give the results in Tables 1, 2 and 7, where RECAP outperforms the compared methods. We therefore prioritize a calibration that is fast and easy to transfer to new models over time-consuming tuning.

[2] Xin Zhou et al. Less is enough: Training-free video diffusion acceleration via runtime-adaptive caching. arXiv preprint arXiv:2507.02860, 2025.

[3] Zehong Ma et al. MagCache: Fast video generation with magnitude-aware cache. In NeurIPS, 2025.

Reviewer_uUo4OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents RECAP, a training‑free inference acceleration method for video diffusion models. Unlike existing approaches that naively stitch sparse attention and caching mechanisms together, RECAP adopts a reciprocal‑correction two‑level cache architecture. At the attention level, cached dense refinement corrects sparse attention outputs, and drift‑weighting controls the effective strength of historical refinement terms. At the prediction level, corrected sparse outputs serve as data sources to enable whole‑step prediction reuse. An error‑guided online scheduling strategy is adopted to adaptively select among three operations: dense refresh, corrected sparse computation, and prediction reuse, without requiring offline search. The authors conduct experiments on three mainstream video generation models: CogVideoX, HunyuanVideo and Wan2.1, and further extend the method to FLUX for image generation. Ablation studies verify the necessity of three components: attention reconstruction, prediction reuse, and online scheduling. Compared with standalone sparse‑attention / caching baselines as well as their direct combinations, RECAP achieves an improved speed‑fidelity trade‑off. It obtains a 3.05× speedup on HunyuanVideo with generation quality superior to all compared composite baselines.

Critical strengths
  1. Clear core innovation: Instead of straightforwardly combining sparse attention and caching, this paper proposes a reciprocal‑correction two‑level cache framework. Leveraging the temporal persistence of sparsification errors, dense residuals are cached at the attention level for correction. Corrected sparse predictions are fed as fresh sources for temporal reuse. The two modules mutually boost performance, which distinguishes RECAP from prior isolated acceleration modules.
  2. Comprehensive experiments and strong generalizability: Evaluations are performed on three video diffusion models of different sizes, and the method is further extended to the FLUX image generation task. Systematic ablations validate contributions from each of the three key components. Extensive comparisons against sparse‑attention and caching baselines are conducted under both matched‑location and matched‑runtime settings, demonstrating RECAP’s advantages on the speed‑fidelity frontier.
Critical weaknesses
  1. Limited evaluation metrics: The current evaluation mainly relies on partial VBench metrics together with PSNR / SSIM / LPIPS. It is worth considering supplementing full VBench results and vision‑reward‑type metrics.
  2. Insufficient discussion on boundary cases: Specialized analysis of error accumulation under long‑duration videos, high‑motion and complex dynamic scenes is missing. It remains undiscussed whether error drift will degrade performance drastically for extremely long sampling sequences.
Questions for authors
  1. For long‑video samples with high dynamics and intensive motion, can the drift‑weighting mechanism consistently suppress error accumulation as denoising steps proceed? Are there certain scenarios where errors can blow up rapidly?
  2. Would you consider using standard VBench prompts and supplementing the complete per‑dimension metrics of VBench?
5 comments

Thank you for your helpful suggestions. We are very glad that you recognize the clear core innovation of our reciprocal-correction cache framework, and our comprehensive experiments with strong generalizability. Below we respond to your concerns and questions.

Q1: Evaluation metrics.

A1: We ran the complete VBench evaluation on the standard prompts and also scored the videos with a vision reward model.

Metrics and prompts in the paper. Training-free acceleration starts from the same prompt and initial noise as dense generation, so its goal is to reproduce the dense video. PSNR, SSIM and LPIPS measure this directly, and the VBench dimensions in Tables 8 and 9 judge how each video looks. The remaining dimensions measure how much a video moves and whether it follows the prompt. These are decided mainly by the base model, which is why acceleration papers such as our baselines SVG, DFSAttn and SVOO usually do not report them. The optimized prompts we used are also official VBench prompts. They are longer versions of the standard prompts with the same meaning, and SVG and SVOO evaluate on them as well.

Complete VBench on standard prompts. For completeness, we still ran the full VBench evaluation on the standard prompts as requested. On Wan2.1-1.3B, we took 12 evenly spaced prompts from each of the 11 VBench prompt suites, 132 prompts in total, and scored all 16 dimensions. All methods keep the normal-regime settings of Table 7.

Table A1a. VBench scores with the official normalization and weights. The Total score is (4 × Quality score + Semantic score) / 5.

MethodQuality score↑Semantic score↑Total score↑
Dense83.2656.1377.84
LeMiCa83.0757.6677.99
NaviCache83.0157.1277.83
RECAP-quality83.3457.2278.11
RECAP-speed83.1657.2777.98

RECAP-quality achieves the highest Quality score, slightly above dense, and also the highest Total score. Every method stays within 0.3 points of dense in the Quality and Total scores, so acceleration preserves the quality of the dense videos, and RECAP preserves it best. Tables A1b and A1c give the individual dimensions. Among the accelerated methods, RECAP is the best in all six Quality dimensions in which the methods differ.

Table A1b. VBench Quality dimensions. IQ denotes imaging quality, SC subject consistency, BC background consistency, AQ aesthetic quality, MS motion smoothness, TF temporal flickering and DD dynamic degree. Bold marks the best accelerated method in each dimension, with dense generation as the reference.

MethodIQ↑SC↑BC↑AQ↑MS↑TF↑DD↑
Dense64.0590.4196.4459.3596.8699.2491.67
LeMiCa63.3190.3996.3459.0596.9199.1691.67
NaviCache63.3190.2996.2258.9596.9399.1291.67
RECAP-quality63.9190.5896.5459.5296.9299.2091.67
RECAP-speed63.4090.6796.3559.1096.9699.1391.67

Table A1c. VBench Semantic dimensions. Obj denotes object class, MultiObj multiple objects, Action human action, Spatial spatial relationship, AStyle appearance style, TStyle temporal style and OC overall consistency.

MethodObj↑MultiObj↑Action↑Color↑Spatial↑Scene↑AStyle↑TStyle↑OC↑
Dense66.6734.9058.3365.2150.4915.1020.5723.3727.32
LeMiCa70.8333.8566.6764.8153.3515.1020.5223.3427.36
NaviCache70.8335.4266.6762.1656.478.3320.6023.5627.52
RECAP-quality71.3535.4266.6762.5850.8913.5420.6023.2627.54
RECAP-speed70.8334.9066.6763.3551.4013.5420.5823.2527.65

Some per-dimension gaps look large only because of the small number of prompts per suite. Dynamic degree and five Semantic dimensions, namely object class, multiple objects, human action, color and scene, count yes or no decisions on each video. With 12 prompts per suite, one borderline video that flips moves such a score by 8 to 10 points. The decisions themselves rarely change. All methods mark the same single video as static, as dense generation does, and in the five Semantic dimensions 45 of the 58 scored videos receive the same decision under dense generation and all four methods.

A1 (continued)

Vision reward. We also scored the same videos with VideoReward [1], a reward model trained on human preferences between generated videos. It rates visual quality (VQ), motion quality (MQ) and text alignment (TA), and their sum gives the overall reward.

Table A1d. VideoReward on the same videos with the released model and settings. Its scores are standardized, so only differences between methods are meaningful. Bold marks the best result.

MethodVQ↑MQ↑TA↑Overall↑
Dense−0.377−0.096−0.520−0.993
LeMiCa−0.386−0.095−0.524−1.005
NaviCache−0.443−0.172−0.535−1.149
RECAP-quality−0.361−0.070−0.519−0.950
RECAP-speed−0.358−0.036−0.541−0.935

RECAP-speed achieves the highest overall reward, and both settings of RECAP score above dense, mostly through better motion quality. NaviCache, which runs at a similar speedup to RECAP-speed, falls clearly below dense.

[1] Jie Liu et al. Improving video generation with human feedback. In NeurIPS, 2025.


Q2: Long videos & high motion.

A2: We ran a new experiment on longer videos, evaluated the prompts with the most motion separately, and recorded the error at every denoising step.

Longer videos. CogVideoX-1.5 generates videos of up to 161 frames, or 10 seconds, twice the 81 frames of our main experiments. We generated videos for ten prompts at both lengths. Five of them are the prompts with the most motion in our prompt set, measured by the mean difference between consecutive frames of their dense videos. All methods keep the settings of the main experiments, and fidelity and speedup are measured against dense generation of the same length.

Table A2a. CogVideoX-1.5 at 81 and 161 frames on one NVIDIA A800 GPU. Each cell gives 81 frames / 161 frames. The first three rows follow the normal regime of Table 1 and the last two the high-speed regime of Table 2. Bold marks the best result within each regime.

MethodPSNR↑SSIM↑LPIPS↓Speedup↑
LeMiCa30.59 / 30.930.9668 / 0.97140.0413 / 0.03951.59× / 1.65×
RECAP-speed30.66 / 30.990.9662 / 0.97040.0419 / 0.04071.65× / 1.80×
RECAP-quality32.50 / 33.070.9745 / 0.97750.0312 / 0.03001.58× / 1.73×
DFSAttn + LeMiCa21.55 / 22.030.8483 / 0.86410.1763 / 0.17012.55× / 2.79×
RECAP-fast23.15 / 23.980.8867 / 0.88140.1258 / 0.13962.58× / 2.84×

RECAP keeps its advantage at twice the length, and the margin grows. RECAP-quality exceeds LeMiCa by 1.91 dB at 81 frames and by 2.14 dB at 161 frames, and RECAP-fast exceeds DFSAttn + LeMiCa by 1.60 and 1.95 dB. The speedup of RECAP also grows more with length than that of LeMiCa, from 1.65× to 1.80× for RECAP-speed against 1.59× to 1.65×. Attention takes a larger share of the computation in longer videos, and RECAP reduces the cost of attention itself, which caching alone does not. At 161 frames, every setting of RECAP is therefore faster than the baseline of its regime, and RECAP-quality and RECAP-fast are also clearly more faithful.

161 frames is the longest video that CogVideoX-1.5 supports. Like HunyuanVideo and Wan2.1, it is a bidirectional model that denoises all frames of a clip jointly, and the clips of such models last about 5 to 10 seconds. Minute-scale videos are mostly generated by autoregressive models instead, which produce a video chunk by chunk with causal attention to the frames already generated [2, 3, 4]. This is a different setting from the bidirectional models that RECAP targets, and extending RECAP to autoregressive generation is a promising direction for future work. We already point to this direction in the Limitations section (Appendix G).

[2] Tianwei Yin et al. From slow bidirectional to fast autoregressive video diffusion models. In CVPR, 2025.

[3] Justin Cui et al. Self-Forcing++: Towards minute-scale high-quality video generation. In ICLR, 2026.

[4] Sand AI et al. MAGI-1: Autoregressive video generation at scale. arXiv preprint arXiv:2505.13211, 2025.

A2 (continued)

High motion. Table A2b repeats the comparison on the five high-motion prompts.

Table A2b. The five high-motion prompts of Table A2a. Each cell gives 81 frames / 161 frames, and bold marks the best result within each regime.

MethodPSNR↑SSIM↑LPIPS↓Speedup↑
LeMiCa28.93 / 29.870.9569 / 0.96790.0568 / 0.04871.58× / 1.65×
RECAP-speed29.00 / 29.650.9580 / 0.96530.0576 / 0.05311.64× / 1.80×
RECAP-quality30.60 / 31.730.9679 / 0.97400.0436 / 0.03841.57× / 1.73×
DFSAttn + LeMiCa20.52 / 21.430.8421 / 0.86880.1979 / 0.18762.60× / 2.79×
RECAP-fast22.98 / 23.330.8755 / 0.88680.1541 / 0.16032.63× / 2.88×

High motion lowers the fidelity of every method, and RECAP keeps a clear advantage. RECAP-quality exceeds LeMiCa by 1.67 dB at 81 frames and by 1.86 dB at 161 frames, and RECAP-fast exceeds DFSAttn + LeMiCa by 2.46 and 1.90 dB. RECAP is again the fastest method in each regime.

Error accumulation over the denoising steps. We saved the latent after every denoising step and measured its relative deviation from the dense trajectory, the quantity used in Fig. 4c.

Table A2c. Deviation from the dense trajectory at the final step, averaged over the ten prompts, and the largest increase of this deviation between two consecutive steps on any prompt, in percentage points. Each cell gives 81 frames / 161 frames.

MethodFinal deviation (%)↓Largest one-step increase↓
LeMiCa6.94 / 6.590.78 / 0.64
RECAP-speed7.67 / 7.231.14 / 1.08
RECAP-quality5.97 / 5.660.66 / 0.63
DFSAttn + LeMiCa18.56 / 17.561.52 / 1.42
RECAP-fast14.84 / 14.851.63 / 1.62

The error of RECAP grows gradually and never jumps. Across all prompts and steps, a single step raises the deviation by at most 1.63 percentage points. The final deviation is the same or lower at 161 frames than at 81 frames, so the error does not build up with video length. RECAP-quality also ends closer to the dense trajectory than LeMiCa, and RECAP-fast ends 15% to 20% closer than DFSAttn + LeMiCa.

Accumulation is limited by renewal. Each dense refresh recomputes the refinement from dense attention (Sec. 3.2), and each computed step renews the prediction history (Sec. 3.3). Between two refreshes, drift weighting lowers the weight of an aging refinement. This moderates error growth compared with full-strength refinement (Fig. 5c), and our response to Reviewer QjnX (Q5) shows that it improves the final videos in both regimes. Errors grow quickly only when the cached state is never renewed, as in the frozen-state ablation of Table 3. An abrupt change between two computed steps also does not carry over, because the next computed step renews the prediction history, as our response to Reviewer ibf5 (Q2) shows.

We have added the longer-video experiment, including the high-motion prompts and the trajectory deviation, to Appendix E.7 and Table 12 of the revised paper.

Summary

Overall, the complete VBench and VideoReward evaluations show that RECAP best preserves the quality of dense generation, and RECAP keeps its advantage on longer and high-motion videos without sudden error growth. We hope these results address your concerns.

Reviewer_ibf5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RECAP, a training-free framework that jointly integrates sparse attention and temporal caching for accelerating diffusion transformers. Its central idea is to address the error propagation that arises when sparse attention outputs are directly reused across denoising steps.

RECAP consists of two complementary components. First, at the attention level, it maintains cached dense refinement residuals to correct the approximation errors of sparse attention. Fresh block-level statistics are recomputed at sparse evaluation steps, while a drift-dependent weighting mechanism controls the contribution of stale refinement information. Second, at the prediction level, RECAP maintains a history of corrected model predictions and uses an online least-squares procedure to extrapolate predictions at skipped denoising steps. An error-guided scheduler coordinates prediction reuse, corrected sparse evaluation, and dense refresh under a prescribed computation budget.

The method is evaluated on CogVideoX-1.5, HunyuanVideo, and Wan2.1 for video generation, with an additional evaluation on FLUX.1-dev for image generation. The reported results show improved fidelity-efficiency trade-offs over individual acceleration methods and direct sparse-attention/cache compositions. In particular, RECAP achieves a reported 3.05× speedup on HunyuanVideo while outperforming the evaluated direct compositions in PSNR, SSIM, and LPIPS.

Critical strengths

A meaningful approach to coupling sparse attention and temporal caching. The paper identifies an important interaction that is often overlooked in straightforward combinations of acceleration techniques: sparse prediction errors can propagate into subsequent cache sources and affect later denoising steps. Rather than treating sparse attention and temporal reuse as independent approximations, RECAP explicitly couples them through cached dense refinement and corrected prediction-history renewal. This provides a coherent motivation for the proposed hierarchical design. Comprehensive empirical evaluation with useful component ablations. The method is evaluated on three video diffusion models and an additional image-generation model. The direct-composition comparisons are particularly relevant to the paper's central motivation. For example, on HunyuanVideo, RECAP achieves 3.05× speedup and 29.67 dB PSNR, compared with 2.83× and 24.30 dB for the strongest PSNR-performing direct composition, DFSAttn + LeMiCa. The ablations also provide evidence for the importance of attention refinement, corrected prediction-history renewal, and adaptive scheduling. The results on different operating points make the empirical evaluation more convincing than relying on a single speed-quality configuration.

Critical weaknesses

The empirical justification for reusing dense refinement across denoising steps remains somewhat limited. A central assumption of RECAP is that sparsification errors remain directionally aligned across nearby denoising steps, allowing dense refinement residuals to be reused. Although Fig. 4d provides supporting evidence, the analysis is conducted along dense trajectories using CogVideoX-1.5, with reuse pairs derived from a reference prompt and evaluated over ten prompts. It is not sufficiently clear whether this error-persistence property holds consistently across different model architectures, sampling stages, and more diverse prompts. Since this assumption directly motivates the attention-level caching mechanism, broader validation would strengthen the technical justification. The reliability of the error-guided scheduler is not fully established. The scheduler estimates prediction errors only at computed steps, while the actual errors at skipped steps are unavailable. Similarly, refinement reliability is checked against dense attention only when a dense refresh occurs. Consequently, the scheduler relies on observed errors to determine future reuse intervals, but it is unclear how reliably these observations predict errors during longer unobserved intervals or under abrupt changes in the denoising trajectory. The paper provides ablations against uniform scheduling, but a more comprehensive analysis of scheduling robustness and sensitivity to the error thresholds would be valuable. The experimental reporting could be more statistically and practically comprehensive. Although the reported speed-fidelity improvements are encouraging, the main quantitative tables do not clearly establish the variability of results across prompts or random seeds. Standard deviations or confidence intervals would help determine whether the reported improvements are consistent. Furthermore, the operating-point parameters are selected using two prompts per model, as described in Appendix C.4. While the selected settings are subsequently fixed across evaluation prompts, additional sensitivity analysis would help establish how robust the method is to this calibration procedure and to different target latency regimes.

Questions for authors

Generalization of sparsification-error persistence: The analysis in Fig. 4d is primarily conducted on CogVideoX-1.5. Have the authors examined whether the directional alignment of sparsification errors also holds for HunyuanVideo and Wan2.1? If so, could they provide quantitative results across different denoising stages and prompt categories? Reliability of online scheduling: Since prediction errors can only be measured at computed steps, how robust is the scheduler when the prediction error changes abruptly between two computed steps? Does the proposed error-rate mechanism remain reliable under highly dynamic prompts or sampling trajectories? Additional evidence comparing actual errors and scheduling decisions would be helpful. Sensitivity to operating-point selection: Appendix C.4 describes selecting the computation budget and error tolerances using two prompts per model. How sensitive are the final speed-fidelity results to these choices? Could the authors provide a sensitivity analysis showing whether comparable performance can be obtained under different calibration prompts and target latency constraints?

7 comments

Thank you for your careful review and helpful suggestions. We are very glad that you recognize the meaningful coupling of sparse attention and caching, and our comprehensive evaluation and ablations. Below we respond to your concerns and questions.

Q1: Generalization of sparsification-error persistence.

A1: We repeated the analysis of Fig. 4d on all three models, over all denoising steps and a broader set of prompts. The protocol follows Fig. 4d, and each prompt now uses its own TeaCache schedule. Besides the cosine, Table A1 reports the error left after reuse, ∥ϵt−ϵs∥/∥ϵt∥\lVert \epsilon_t - \epsilon_s \rVert / \lVert \epsilon_t \rVert, which is below one whenever reusing the source error reduces the current error.

Table A1. Persistence of sparsification error. Means over prompts, with 95% bootstrap intervals over prompts. Early, middle and late cover steps 0–16, 17–33 and 34–49. Each model uses 22 prompts with two prompts from each of the eleven VBench prompt groups.

ModelStageCosineError left after reuse
CogVideoX-1.5Early0.768 [0.746, 0.788]0.66
Middle0.953 [0.950, 0.956]0.35
Late0.945 [0.940, 0.950]0.36
Steps 12–49 (Fig. 4d)0.949 [0.946, 0.953]0.36
Wan2.1-1.3BEarly0.873 [0.853, 0.892]0.49
Middle0.946 [0.938, 0.954]0.32
Late0.965 [0.962, 0.967]0.26
Steps 12–49 (Fig. 4d)0.938 [0.928, 0.948]0.34
HunyuanVideoEarly0.647 [0.622, 0.673]0.85
Middle0.823 [0.806, 0.839]0.58
Late0.861 [0.849, 0.873]0.50
Steps 12–49 (Fig. 4d)0.827 [0.811, 0.842]0.57

Model architectures. The alignment holds on all three models. In the step range of Fig. 4d, the mean cosine is 0.949 on CogVideoX-1.5, 0.938 on Wan2.1 and 0.827 on HunyuanVideo. In all source and reuse pairs, reusing the source error reduces the current error, on average by 64%, 66% and 43%. The cosine is lower on HunyuanVideo, but its sparsification error is also the smallest to begin with, at 2.6% of the norm of the dense output against 5.4% on CogVideoX-1.5 and 8.8% on Wan2.1. After reuse, the error that remains is 1.5% of the dense output on HunyuanVideo, compared with 1.9% and 2.8% on the other two models.

Denoising stages. The alignment is strongest in the middle and late stages, with cosines of 0.945 to 0.965 on CogVideoX-1.5 and Wan2.1 and 0.82 to 0.86 on HunyuanVideo. It is lower only in the early stage. This has little effect in practice, because sparse attention is not used that early. DFSAttn computes steps 0 to 11 densely by default, and we enabled its sparse attention there only for this analysis. RECAP also starts using the cached refinement only after its first refinement check has passed (Appendix B.2).

Prompt categories. The prompts are the same on all models and take two prompts from each of the eleven VBench prompt groups. The alignment is stable across categories. The mean cosine per category lies between 0.940 and 0.962 on CogVideoX-1.5, between 0.916 and 0.950 on Wan2.1, and between 0.807 and 0.860 on HunyuanVideo. Even the least aligned single prompt reaches 0.937, 0.890 and 0.769 on the three models.

These results confirm on all three architectures the property that motivates our attention-level cache.

Q2: Reliability of online scheduling.

A2: We answer the three points in turn, each with a new experiment on all three models.

Actual errors and scheduling decisions. The scheduler does observe real reuse errors. At every computed step, RECAP compares the newly computed prediction with the prediction that reuse would have given at that same step (Eq. 7). For example, if steps 20 and 24 are computed and steps 21 to 23 are skipped, the scheduler measures the reuse error at step 24. What it cannot see are the errors at steps 21 to 23. To measure these hidden errors, we ran RECAP unchanged and also computed the dense prediction at every skipped step, for logging only. All experiments in this answer use 10 prompts on CogVideoX-1.5, 20 on Wan2.1 and 10 on HunyuanVideo.

Table A2a. Observed error against hidden errors. A run is a group of consecutive skipped steps. The last column counts the runs in which the error observed at the computed step that ends the run is at least as large as every hidden error inside the run.

ModelSettingLongest run of skipped stepsRuns where the observed error is the largest
CogVideoX-1.5Speed7140 of 140
CogVideoX-1.5Fast490 of 90
Wan2.1-1.3BSpeed5160 of 160
Wan2.1-1.3BFast10133 of 133
HunyuanVideoSpeed3120 of 120
HunyuanVideoFast3130 of 130

On all three models, the observed error is the largest error of every run, even when ten steps in a row are skipped. The reason is that the reuse error grows with the distance from the last computed step, so the step that the scheduler checks is the worst one. The scheduler therefore decides from the largest error of each run, and the hidden errors are smaller.

Abrupt changes between computed steps. In regular sampling, we found that the error changes smoothly from one step to the next. We therefore created an abrupt change ourselves. At the Fast setting, right after a computed step, we added strong Gaussian noise to the latent, with 30% of the latent's magnitude. This is the worst moment for RECAP, because it then skips several steps before it computes again and cannot notice the change in between. We added the same noise to dense generation and measured how close RECAP stays to it.

Table A2b. Effect of the abrupt change at the Fast setting. The last column is RECAP's PSNR against dense generation with the change, minus its PSNR on the same prompts without the change.

ModelSkipped steps right after the changeChange in PSNR (dB)
CogVideoX-1.54+0.63
Wan2.1-1.3B6 to 7−0.01
HunyuanVideo3−0.04

RECAP follows dense generation as closely with the abrupt change as without it. The change can only affect the few skipped steps that follow it. The next computed step evaluates the model on the current latent and renews the prediction history, so the effect ends there and does not carry over to the rest of sampling.

Highly dynamic prompts. To answer this, we selected the prompts with the most motion for each model before running the experiments, and checked two things on them.

First, the scheduler's observations stay trustworthy. On the high-motion prompts, the observed error is the largest error of every run, exactly as in Table A2a.

Second, the scheduler's decisions still improve the videos. We measure this against a uniform schedule, which computes the same number of steps as RECAP but spaces them evenly and ignores the measured errors. Any gain over it comes only from where the scheduler places the computed steps. Table A2c reports this gain for all prompts and for the high-motion prompts alone.

Table A2c. PSNR gain of the error-guided schedule over the uniform schedule at the Fast setting, in dB. Brackets give the paired 95% bootstrap interval.

ModelAll promptsHigh-motion prompts
CogVideoX-1.5+1.54 [1.27, 1.82]+1.55
Wan2.1-1.3B+1.61 [1.29, 1.91]+1.62
HunyuanVideo+1.12 [0.52, 1.75]+0.80

The scheduler improves every prompt of every model, and on CogVideoX-1.5 and Wan2.1 its gain on the high-motion prompts is as large as on all prompts. The error-rate mechanism therefore remains reliable on highly dynamic prompts.

Q3: Sensitivity to operating-point selection.

A3: An operating point of RECAP consists of the budget BB and the two tolerances τpred\tau_{\mathrm{pred}} and τref\tau_{\mathrm{ref}}. To measure its sensitivity exhaustively, we ran a new study on Wan2.1-1.3B. At each of its three operating points, we generated videos with all 16 combinations of the two tolerances from the candidate set {0.025, 0.05, 0.1, 0.2} on 20 randomly selected prompts, 960 videos in total. We answer the three parts of the question in turn.

Sensitivity to the selected values. The budget BB is the speed setting of RECAP, much like the threshold of a caching method. It is chosen to meet the latency target and is not tuned for fidelity, so the selection only concerns the two tolerances. Table A3a varies one tolerance at a time.

Table A3a. PSNR (dB) when one tolerance is varied and the other keeps its paper value. The paper value is in bold, and parentheses give the change from it.

Operating pointVaried tolerance0.0250.050.10.2
Fastτpred\tau_{\mathrm{pred}}21.73 (−0.49)22.49 (+0.27)22.2221.85 (−0.37)
Fastτref\tau_{\mathrm{ref}}22.30 (+0.07)22.2222.17 (−0.05)22.17 (−0.06)
Speedτpred\tau_{\mathrm{pred}}25.40 (+0.38)25.0225.02 (0.00)25.02 (0.00)
Speedτref\tau_{\mathrm{ref}}25.25 (+0.23)25.11 (+0.09)25.0225.02 (0.00)
Qualityτpred\tau_{\mathrm{pred}}28.1727.85 (−0.32)27.71 (−0.46)27.71 (−0.46)
Qualityτref\tau_{\mathrm{ref}}28.54 (+0.37)28.22 (+0.05)28.1728.17 (0.00)

RECAP is insensitive to both tolerances. Across all 18 alternative values in Table A3a, PSNR stays within 0.5 dB of the paper setting. Several entries coincide because a loose tolerance lets almost every check pass, and the remaining budget then paces the computed steps (Appendix B.2). A stricter τref\tau_{\mathrm{ref}} gains up to 0.37 dB only because more checks fail and more computed steps become dense, which also makes generation slower. Although the paper values were selected on just two prompts, they lie within 0.4 dB of the best candidate in every row on these 20 prompts.

Different calibration prompts. We repeated the selection procedure of Appendix C.4 with every pair of the 20 prompts as calibration prompts, which gives 190 pairs. Table A3b compares each selected setting with the paper setting.

Table A3b. Selection repeated with all 190 pairs of calibration prompts. PSNR changes are relative to the paper setting.

Operating pointPairs that match or exceed the paper settingMean PSNR change (dB)Worst pair (dB)
Fast176 of 190+0.21−0.68
Speed185 of 190+0.260.00
Quality179 of 190+0.17−0.63

The selection is robust to the choice of calibration prompts. In 540 of the 570 repeated selections, it reaches the fidelity of the paper setting or exceeds it, and on average it is 0.17 to 0.26 dB higher. The paper results therefore do not rely on a favorable pair of calibration prompts and, if anything, are slightly conservative. Even the least favorable pair stays within 0.7 dB of the paper setting.

Reviewer_QjnXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

When sparse attention and temporal caching are simply stacked, sparsification errors propagate through the cache along the sampling trajectory. At the attention level, RECAP caches the tail residual computed at dense refreshes to correct subsequent sparse steps, attenuated according to drift; at the prediction level, it skips whole steps by linearly extrapolating from the two most recent computed predictions; and the prediction and refinement errors schedule online when to compute and when to refresh densely. Experiments on three video models and FLUX.1-dev compare against sparse-attention methods, caching methods and their direct combinations; on HunyuanVideo RECAP reaches a 3.05× speedup with higher PSNR than all direct combinations.

Critical strengths

The problem is practical and the method is training-free. How to combine sparse attention and caching is a very practical question in video-generation acceleration, and the authors design the method from the coupling that "sparse outputs become cache sources," which is a clear line of reasoning.

The experiments are broad: three video models plus one image model, two acceleration regimes, 7 individual methods and 6 direct combinations, with fidelity to the dense results, VBench quality, end-to-end latency (including data-transfer overhead) and memory all reported. In the high-speed regime, RECAP is both faster and higher in PSNR than all direct combinations on all three video models (3.78 dB above the best combination on CogVideoX-1.5).

The engineering is solid: custom Triton kernels, the refinement cache kept in host memory and prefetched layer by layer, and peak GPU memory on HunyuanVideo only about 1 GiB above dense inference. The ablations cover every component and are repeated in the fast setting (Table 10), and the comparisons with PISA and Re-ttention are also run at matched runtime.

The decomposition of the reconstruction error in Appendix A (Equations (15)–(18)) and the derivation in Appendix B of how source errors enter the extrapolation (Equation (20)) are clear and help understand the role of each component.

Critical weaknesses

The experiments do not isolate the benefit of sparse attention with cache refinement.  The comparisons are insufficient to establish a reliable advantage over prior methods. Closely related methods combining sparse attention with cross-step reuse, including DiTFastAttn and LoSA, are discussed but not evaluated. The paper also omits evaluation sample sizes and baseline operating-point selection, compares methods at different speedups, and reports some very small quality gains, such as 0.06 dB over NaviCache on HunyuanVideo. Comparisons with the closest methods, using documented evaluation settings and speed–fidelity curves, are needed to determine whether the claimed advantage is robust and attributable to the proposed method.

Questions for authors

When prediction extrapolation and online scheduling are kept but every computed step uses dense attention, what fidelity is obtained at the same latency? For each operating point, what are the numbers of dense refreshes and sparse steps, the sparsity ratio, and the latency breakdown? Does RECAP remain ahead when individual caching methods (LeMiCa, NaviCache) and extrapolation-based caching (e.g., TaylorSeer) are added to the high-speed regime of Table 2? Can it be compared with DiTFastAttn and LoSA? How does end-to-end fidelity change when the drift weighting is removed ()? How many prompts were used for each of Tables 1, 2 and 7? How were the operating points of the baselines chosen? Can speed–fidelity curves be provided for each method over a range of speedups?

6 comments

Thank you for your careful and constructive review. We are very glad that you recognize the practical and clearly reasoned design of RECAP, the broad experiments, the solid engineering and the complete ablations. Below we respond to your concerns and questions.

Q1: "The experiments do not isolate the benefit of sparse attention with cache refinement." "When prediction extrapolation and online scheduling are kept but every computed step uses dense attention, what fidelity is obtained at the same latency?"

A1: We ran this control exactly as described. The variant Dense + reuse keeps the prediction extrapolation, the fitted gain and the error-guided scheduler of RECAP, and differs only in that every computed step uses dense attention. As requested, we compare the two at the same latency instead of the same budget. A dense step costs more than a corrected sparse step, so we adjust the budget BB of this variant to bring its latency to a level similar to RECAP. We follow the ablation protocol of Tables 3 and 10 on CogVideoX-1.5, covering both the normal and the fast regime.

Table A1. RECAP against dense attention with the same prediction reuse and scheduling at similar latency on CogVideoX-1.5.

RegimeVariantLatency↓PSNR↑SSIM↑LPIPS↓
NormalDense + reuse637.32s35.420.97930.0297
NormalRECAP619.69s36.150.98170.0219
FastDense + reuse394.87s24.950.85800.1495
FastRECAP391.16s25.680.87680.1296

RECAP is better on every metric in both regimes while also running faster. It improves PSNR by 0.73 dB in each regime and lowers LPIPS by 26% in the normal regime and by 13% in the fast regime. Both variants share the same prediction reuse and scheduling, so this gain comes entirely from corrected sparse attention. With cached dense refinement, a sparse step stays close to dense accuracy at a lower cost, and the time it saves buys more fidelity than spending the same time on dense attention. This confirms that the attention level of RECAP contributes beyond prediction reuse alone, and that the two levels are most effective together. We have added this comparison to Tables 3d and 10d of the revised paper.

Q2: "For each operating point, what are the numbers of dense refreshes and sparse steps, the sparsity ratio, and the latency breakdown?"

A2: Table A2a reports, for every operating point, the mean number of dense refreshes and corrected sparse steps per video, the fraction of key blocks that a sparse step computes exactly, and the share of end-to-end latency spent in each of the two. Table A2b gives the time of each type of step, measured with GPU-synchronized timers during generation, on RTX 4090 GPUs for CogVideoX-1.5 and Wan2.1-1.3B and on A800 GPUs for HunyuanVideo, as in the paper.

Table A2a. Mean number of dense refreshes and corrected sparse steps per video, sparsity, and their share of end-to-end latency.

SettingDense refreshesCorrected sparse stepsKey blocks computed exactlyLatency share, refresh / sparse
CogVideoX-1.5 Quality4.29.820%18% / 19%
CogVideoX-1.5 Speed4.28.820%19% / 18%
CogVideoX-1.5 Fast5.75.320%38% / 14%
HunyuanVideo Quality4.014.020%21% / 23%
HunyuanVideo Speed4.07.020%24% / 13%
HunyuanVideo Fast4.07.220%34% / 20%
Wan2.1-1.3B Quality4.26.440%19% / 16%
Wan2.1-1.3B Speed3.04.040%17% / 12%
Wan2.1-1.3B Fast3.33.040%23% / 11%

Table A2b. Mean time of one step. Parentheses give the cost relative to a dense step.

SettingDense stepDense refreshCorrected sparse step
CogVideoX-1.5 Quality21.0 s29.3 s (1.39×)13.1 s (0.62×)
CogVideoX-1.5 Speed20.9 s29.3 s (1.40×)13.1 s (0.62×)
CogVideoX-1.5 Fast20.7 s28.3 s (1.36×)11.7 s (0.57×)
HunyuanVideo Quality80.8 s113.0 s (1.40×)34.5 s (0.43×)
HunyuanVideo Speed81.5 s116.4 s (1.43×)34.8 s (0.43×)
HunyuanVideo Fast81.7 s108.2 s (1.32×)35.0 s (0.43×)
Wan2.1-1.3B Quality4.8 s7.2 s (1.49×)4.0 s (0.83×)
Wan2.1-1.3B Speed4.8 s7.1 s (1.47×)4.0 s (0.83×)
Wan2.1-1.3B Fast4.8 s6.8 s (1.40×)4.0 s (0.83×)

Dense refreshes and sparse steps. Only three to six dense refreshes are needed per video on average, and a single refresh serves up to seven corrected sparse steps. Refreshes also become rarer as sampling proceeds, because each passing check extends how long the refinement may be reused, as described in Appendix B.2. A refresh costs 1.3 to 1.5 times a dense step since it also stores and checks the refinement, and the cheaper sparse steps that follow repay this cost.

Sparsity ratio. A corrected sparse step computes attention exactly over 20% of the video key blocks on CogVideoX-1.5 and HunyuanVideo. This level is common for sparse attention on video models. SVG and DFSAttn keep about 25% to 30% of the attention in their released settings. Unlike these methods, RECAP does not drop the remaining key blocks but represents them with the coarse estimate and the cached refinement. Wan2.1-1.3B is a much smaller model in which attention takes only about half of a dense step, so we use 40% there, which adds less than 2% to latency.

Latency breakdown. A corrected sparse step is much cheaper than a dense step, and the saving grows with sequence length. It takes 0.62 of a dense step on CogVideoX-1.5 and 0.43 on HunyuanVideo, which has 2.6 times more video tokens.

Q3: "Does RECAP remain ahead when individual caching methods (LeMiCa, NaviCache) and extrapolation-based caching (e.g., TaylorSeer) are added to the high-speed regime of Table 2?"

A3: Yes, RECAP remains ahead of all three methods. We report the results on Wan2.1-1.3B below.

Individual caching methods. We set LeMiCa and NaviCache to a speedup similar to the high-speed regime and evaluate them with the same setting as Table 7.

Table A3a. Individual caching methods in the high-speed regime on Wan2.1-1.3B. The RECAP row is from Table 7.

MethodPSNR↑SSIM↑LPIPS↓CLIPIQA+↑VBench↑Latency↓Speedup↑
LeMiCa (B=18)23.030.82760.12890.51882.31118s2.32×
NaviCache (threshold = 0.0945)22.710.82510.13460.51382.12115s2.37×
RECAP23.510.83230.12380.53482.88105s2.60×

RECAP achieves the best result on every metric while also being the fastest method. We have added both methods at this speedup to Table 7 of the revised paper.

TaylorSeer. TaylorSeer stores features and their derivatives for every layer, which requires a large amount of memory on video models. It runs out of memory on CogVideoX-1.5 at step 5, when the derivatives are first updated, and on HunyuanVideo already during the initial dense steps. Other works report the same limitation. DisCa [1] measures over 120 GB of memory for TaylorSeer on HunyuanVideo, and BWCache [2] also reports that it runs out of memory beyond short clips on an 80 GB GPU. We therefore evaluate it on Wan2.1-1.3B, where its peak memory still reaches 42.87 GiB. These runs use 50 randomly selected prompts on an A800 GPU, and RECAP is measured under the same setting.

Table A3b. Comparison with TaylorSeer on Wan2.1-1.3B.

MethodPSNR↑SSIM↑LPIPS↓CLIPIQA+↑VBench↑Latency↓Speedup↑
TaylorSeer (interval 2)13.370.54470.45140.44681.77149s1.54×
TaylorSeer (interval 5)12.120.49150.57140.36178.0699s2.32×
RECAP23.210.84580.14590.54082.9193s2.47×

RECAP exceeds TaylorSeer by about 10 dB in PSNR while running faster than both of its settings. This fidelity level of TaylorSeer is consistent with prior reports, such as 18.75 dB on Wan2.1 and 15.33 dB on HunyuanVideo at less than 2× speedup [2].

[1] Chang Zou et al. DisCa: Accelerating video diffusion transformers with distillation-compatible learnable feature caching. In CVPR, 2026.

[2] Hanshuai Cui et al. BWCache: Accelerating video diffusion transformers through block-wise caching. arXiv preprint arXiv:2509.13789, 2025.

Comments on the paper

1 comment

Dear Reviewers and Area Chairs,

We thank all reviewers (R1-QjnX, R2-ibf5, R3-uUo4, R4-o8ka) for their constructive reviews and positive feedback on RECAP. Your insightful comments help us to further improve our paper.

We are pleased that:

  • All reviewers appreciate the design of RECAP, finding it practical and clearly reasoned (R1), a meaningful coupling of sparse attention and caching (R2), a clear core innovation (R3), and clean and explainable (R4).
  • All reviewers recognize our broad and comprehensive experiments, with strong generalizability (R3) and broad baseline coverage (R4).
  • R1, R2 and R4 value our complete ablations.
  • R1 and R4 appreciate the solid engineering and the reported latency and memory.

We have responded to each reviewer individually. Here, we offer a summary:

  • More comparisons. RECAP remains ahead of individual caches in the high-speed regime, TaylorSeer, LoSA, DiTFastAttn and ASTRAEA, and gives the best speed–fidelity trade-off over most of a wide speedup range.
  • Component analysis. Controls at matched latency and computation locations show that corrected sparse attention and drift weighting each improve fidelity beyond prediction reuse. Attention-level measurements show how RECAP responds to drastic changes between dense refreshes.
  • Robustness and generality. We confirm on all three models that sparsification errors stay aligned and that the scheduler remains reliable. RECAP is stable across tolerances, calibration prompts and random seeds, and keeps its advantage on longer and high-motion videos.
  • Evaluation. The complete VBench on standard prompts and VideoReward show that RECAP best preserves the quality of dense generation.
  • Clarifications. We detail the operating points, latency measurement, GPU types, calibration cost and baseline settings.
  • Revised paper. We have added the new ablations to Tables 3 and 10, the new baselines to Table 7, and the speed–fidelity curves, random seeds and longer videos to Appendix E.5 to E.7. All changes are marked in blue.

Thanks again to all reviewers and area chairs. We hope our responses address your concerns, and we are happy to discuss any remaining questions.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.