- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies supplementing the error of sparse attention using prediction caching. The goal is to accelerate video diffusion. It uses dense refreshes to update the attention cache. It combines coarser tail estimates to compensate for the result of attention omitted by sparse computation. It accumulates the residual between sparse and dense computation, and performs a dense refresh every several steps to compensate. It caches the residual between exact and coarse tail estimates, and updates the cached residual at each dense refresh. The denoiser outputs are also used for the predictions. The method itself is training-free.
The experiments cover CogVideoX1.5, HunyuanVideo, and Wan2.1 for video and FLUX.1 for images. The comparison methods include sparse attention methods, caching methods, and combinations. Besides generation quality benchmarks, it also includes efficiency experiments compared with the non-sparse setting. The results show a good tradeoff between speed and fidelity. But the comparison seems to omit some methods without explanation, and some settings need further clarification.
Critical strengths
- The baseline coverage is broad. And the proposed method achieves good speed and fidelity results on the video and image benchmarks and models, which is quite comprehensive.
- The ablation of these components is also comprehensive. The ablation study includes refinement, reference updates, and prediction cache updates, which is reasonable.
- The experiment section reports the actual latency and GPU memory results, which help to grasp the deployment benefits of the method. These experiments also show the tradeoffs under different speed settings.
- The method design is clean and explainable. One dense refresh can provide for multiple sparse computations. And the cached features can be reused by subsequent prediction.
Critical weaknesses
- The paper discusses DiTFastAttn, LoSA, and ASTRAEA, but does not compare with them in the experiment section. It needs further explanation of why they are omitted.
- Error persistence experiments use cosine similarity, averaged over ten prompts. But similar directions do not mean that the magnitude is also consistent.
- One question about the design of the method. Between two dense refreshes, we must fix the selected partition, so that the residual and the coarse estimate correspond to the same omitted keys. This is reasonable, but if important interactions change between dense refreshes, the new important interactions may remain in the reconstructed detail, which can only be recovered approximately. So, in Table 3, if we freeze the initial refinement and partition, PSNR drops, but we can't attribute the result only to either the fixed residual or the fixed partition. The experiment didn't separate their effects.
Questions for authors
- The latency reported in Tables 1 to 7. Do these results include text encoding and VAE decoding? Or is it just the denoising step? The appendix states that the cache transfer and the method itself are included, but the definition of generation timing still needs further clarification.
- Why are the main CogVideoX and Wan experiments run on an RTX 4090, while HunyuanVideo and the CogVideoX use an A800?
- Why not compare with DiTFastAttn, LoSA, and ASTRAEA?
- What are the two fixed calibration prompts used for each model? What's the wall-clock time or GPU hours for the whole calibration process?
- What's the rationale for the threshold candidates? A small candidate set is acceptable, but I would like to understand whether the candidates cover the optimal speed and fidelity tradeoff.
- As mentioned in critical weakness 3, the cached residual can only be updated when a dense refresh is conducted. So, before a dense refresh, if there are drastic changes, how can the coarse statistics and drift weighting detect that the cached residual may not be accurate? If these changes are not detected in time between dense refreshes, the refinement error feedback can be late and fidelity can drop.
- The efficiency experiments about RECAP and the sparse-only baseline are impressive, but since RECAP includes many steps, what's the wall-clock time of each step? (For example, dense refresh, coarse tail estimation, residual refinement, and cache transfer.) The time breakdown is helpful to understand the latency of each component, but it's an additional suggestion. It's optional in the rebuttal period if time is limited.
8 comments
A3 (continued)
DiTFastAttn. The released code of DiTFastAttn supports only OpenSora, and every new model requires its offline calibration. We ported its three mechanisms to our video models, namely window attention with residual sharing, attention sharing across timesteps and attention sharing across CFG. On Wan2.1-1.3B, the calibration takes 16 GPU-hours on an A800 GPU with a peak memory of 30.21 GiB. On CogVideoX-1.5 and HunyuanVideo, the batched calibration of the original code runs out of memory. Even a reduced calibration with a single prompt takes about 41 GPU-hours on CogVideoX-1.5 and more than 280 GPU-hours on HunyuanVideo. We therefore report Wan2.1-1.3B results in Table A3b, with the same prompts and RTX 4090 GPUs as Table 7.
Table A3b. Comparison with DiTFastAttn on Wan2.1-1.3B. The RECAP rows are from Table 7.
| Method | PSNR↑ | SSIM↑ | LPIPS↓ | VBench↑ | Latency↓ | Speedup↑ |
|---|---|---|---|---|---|---|
| DiTFastAttn | 18.62 | 0.7291 | 0.2185 | 81.97 | 232s | 1.18× |
| RECAP-speed | 27.05 | 0.9141 | 0.0741 | 83.39 | 131s | 2.08× |
| RECAP-quality | 30.29 | 0.9520 | 0.0372 | 83.49 | 158s | 1.73× |
RECAP outperforms DiTFastAttn by a large margin in both fidelity and speed. RECAP-speed is 8.43 dB higher in PSNR while running 1.8 times faster, and RECAP-quality is 11.67 dB higher. We have added the LoSA and DiTFastAttn comparisons to Table 7 and Appendix E.1 of the revised paper.
ASTRAEA. The public release of ASTRAEA provides only its evolutionary search over timesteps. Its token selection module is proprietary and not released, and the paper does not detail it. The search is also costly. The authors report 69 and 139 GPU-hours for 2 s and 4 s videos on Wan2.1, and the search is repeated for each model, video length and computation budget.
A comparison with the full search would therefore not be on equal terms. We instead roughly match its search time to the calibration cost of RECAP. We ran a fresh search on Wan2.1-1.3B, capped at one GPU-hour, with the four prompts supplied with its code. To finish within the cap, we kept the released operators but used a population of two, with one mutation and one crossover offspring per generation. The search evaluated six candidates in 50.6 minutes. The selected schedule computes 26 steps.
Table A3c. Comparison with the public timestep scheduler of ASTRAEA on Wan2.1-1.3B. The RECAP rows are from Table 7. GPU-hours is the search or calibration cost.
| Method | PSNR↑ | SSIM↑ | LPIPS↓ | CLIPIQA+↑ | VBench↑ | Latency↓ | Speedup↑ | GPU-hours↓ |
|---|---|---|---|---|---|---|---|---|
| ASTRAEA | 17.41 | 0.6713 | 0.2662 | 0.505 | 80.89 | 164s | 1.66× | 0.84 |
| RECAP-speed | 27.05 | 0.9141 | 0.0741 | 0.540 | 83.39 | 131s | 2.08× | 0.53 |
| RECAP-quality | 30.29 | 0.9520 | 0.0372 | 0.545 | 83.49 | 158s | 1.73× | 0.65 |
With less search time, both RECAP settings are faster and much more faithful than ASTRAEA. RECAP-quality calibrates in 0.65 GPU-hours and exceeds ASTRAEA by 12.88 dB in PSNR, and RECAP-speed calibrates in 0.53 GPU-hours, exceeds it by 9.64 dB and runs 1.25 times faster.
Q4: "What are the two fixed calibration prompts used for each model? What's the wall-clock time or GPU hours for the whole calibration process?"
A4: All three models use the same two calibration prompts from VBench.
-
"A sleek, black motorcycle with chrome accents leans into a sharp corner on a winding mountain road, the rider clad in a black leather jacket, matching helmet, and dark jeans. The sun casts long shadows, highlighting the bike's polished surface and the rider's focused posture. As the motorcycle rounds the bend, the tires grip the asphalt, kicking up a slight spray of gravel. The surrounding landscape features towering pine trees and a distant view of snow-capped peaks, adding to the sense of adventure and freedom. The rider's movements are fluid and precise, showcasing skill and control as the motorcycle smoothly navigates the curve."
-
"A mesmerizing splash of turquoise water erupts in extreme slow motion, each droplet suspended in mid-air, creating a captivating dance of liquid. The vibrant turquoise hue shimmers under soft lighting, highlighting the fluid's graceful arcs and intricate patterns. As the splash unfolds, the droplets form delicate, crystalline shapes, almost like a choreographed ballet of water. The background is transparent, allowing the viewer to focus solely on the stunning motion and color of the water. The scene is both tranquil and dynamic, capturing the essence of fluidity and the beauty of nature in exquisite detail."
With the budget fixed, calibrating one operating point tests seven tolerance settings on the two prompts. We first vary over the four grid values with , and then vary over the four grid values at the selected . One setting of the second pass repeats the first, so each operating point needs 14 videos. Table A4 reports the resulting GPU-hours.
Table A4. Calibration cost in GPU-hours. Each operating point runs 7 settings on 2 prompts. Times on CogVideoX-1.5 are estimated from the measured cost of each step type.
| Model | Quality | Speed | Fast |
|---|---|---|---|
| CogVideoX-1.5 | 2.39 | 2.26 | 1.59 |
| HunyuanVideo | 7.71 | 6.76 | 5.40 |
| Wan2.1-1.3B | 0.65 | 0.53 | 0.45 |
All nine operating points on three models take 27.7 GPU-hours in total. For comparison, one DiTFastAttn calibration on CogVideoX-1.5 takes about 41 GPU-hours, and ASTRAEA reports 139 GPU-hours for one operating point of Wan2.1-1.3B.
Q5: "What's the rationale for the threshold candidates? A small candidate set is acceptable, but I would like to understand whether the candidates cover the optimal speed and fidelity tradeoff."
A5: Rationale. Both tolerances are relative errors, which do not depend on the scale of the model outputs, so one candidate set can serve both tolerances on all models. Thresholds of this scale are also typical in caching, where the released thresholds of TeaCache, NaviCache, EasyCache [2] and MagCache [3] lie between a few hundredths and a few tenths. Our factor-of-two grid {0.025, 0.05, 0.1, 0.2} covers this scale.
Values beyond the grid. Extending the grid at either end brings no benefit. On Wan2.1-1.3B and the same calibration prompts, Table A5 compares each end of the grid with one more value beyond it, with fixed at 0.1.
Table A5. PSNR (dB) on Wan2.1-1.3B for at the two ends of the grid and beyond them.
| Operating point | 0.0125 (beyond) | 0.025 (grid end) | 0.2 (grid end) | 0.4 (beyond) |
|---|---|---|---|---|
| Quality | 26.09 | 26.44 | 25.39 | 25.39 |
| Speed | 21.11 | 23.74 | 23.22 | 23.22 |
| Fast | 18.32 | 19.64 | 18.12 | 18.03 |
A stricter of 0.0125 lowers PSNR by 0.35 to 2.63 dB, and a looser 0.4 gives the same or slightly lower PSNR than 0.2. For , 0.4 gives the same videos as 0.2, and 0.0125 changes PSNR by at most 0.1 dB relative to 0.025 while making generation 3% to 9% slower than the chosen setting. We therefore did not further consider values beyond the grid.
Finer grids. A finer grid between these values may find slightly better settings, so we do not claim that the selected settings are optimal. We select each operating point by its fidelity on the two calibration prompts of A4, and these settings already give the results in Tables 1, 2 and 7, where RECAP outperforms the compared methods. We therefore prioritize a calibration that is fast and easy to transfer to new models over time-consuming tuning.
[2] Xin Zhou et al. Less is enough: Training-free video diffusion acceleration via runtime-adaptive caching. arXiv preprint arXiv:2507.02860, 2025.
[3] Zehong Ma et al. MagCache: Fast video generation with magnitude-aware cache. In NeurIPS, 2025.
Thank you for your detailed reading and the many concrete questions. We are very glad that you recognize the broad baseline coverage, the comprehensive ablations, the reported latency and memory, and the clean and explainable design. Below we respond to your concerns and questions.
A1: Yes, the reported latency includes text encoding and VAE decoding. It is the wall-clock time of the complete pipeline call, measured with CUDA synchronization. It covers text encoding, all denoising steps, VAE decoding, model offloading and the host–device transfers of RECAP, and it excludes model loading, warm-up and file writing. Every method is timed in the same harness throughout our experiments.
A2: The GPU types follow the clusters that were available to us at each stage of the project. In the early stage, we only had access to RTX 4090 GPUs, and CogVideoX-1.5 and Wan2.1-1.3B fit within their 24 GB of memory. All main experiments on these two models therefore ran on RTX 4090 GPUs. HunyuanVideo needs more than 24 GB, and by the time we extended our evaluation to it, we had gained access to A800 GPUs, so its experiments ran there.
During the ablation study, we could not obtain uncontended RTX 4090 GPUs consistently enough to keep all latency measurements fair. We therefore regenerated the dense references on A800 GPUs and ran all CogVideoX-1.5 ablations there.
Throughout the project, every method within a table uses the same GPU type and is timed without contention from other processes, so the speedups within a table are always directly comparable.
A3: We now compare RECAP with all three methods.
LoSA. LoSA appeared on arXiv in August 2026, within two months of the submission deadline, so it is concurrent work under the ICLR guidelines. Its code has also not been released, which prevents a faithful comparison. In essence, LoSA is a sparse-attention method. It selects key blocks at an early step and reuses these block indices in later steps. Its paper notes that it can be combined with feature caching methods, in which case the cache schedule, predictor and residuals stay unchanged. The paper pairs LoSA with D2Cache [1]. This is the direct-composition setting that our six compositions in Tables 2 and 7 already represent. For completeness, we still implemented LoSA following its paper, together with D2Cache, and report the results in Table A3a.
Table A3a. Comparison with our implementation of LoSA on Wan2.1-1.3B. The RECAP rows are from Table 7.
RECAP is both faster and more faithful in each comparison. RECAP-speed exceeds LoSA by 2.93 dB at a 2.08× speedup against 1.25×, and RECAP exceeds LoSA with D2Cache by 6.69 dB while running faster.
[1] Enhuai Liu et al. D2Cache: Second-order delta caching for higher video diffusion acceleration. In CVPR, 2026.