- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CoFiRA, a training-free framework for region-wise compute–fidelity allocation in video diffusion transformers. Instead of restricting each region to exact attention or a single approximation, CoFiRA allocates calibrated means, variable-capacity nested sets of real K/V representatives, or exact attention according to query-conditioned contribution and within-cluster K/V variation. A two-stream runtime executes these heterogeneous representations and combines their outputs through global log-sum-exp normalization.
Critical strengths
-
The central contribution is the combination of a multi-capacity representation space and sensitivity-guided allocation. Nested real K/V representatives provide intermediate operating points between a single centroid and exact attention. The comparisons against Exact/Mean, fixed capacities, and shuffled assignments support the value of allocating different capacities to different regions.
-
The packed approximate stream and variable-length exact stream address the irregular workload introduced by heterogeneous capacities. Table 5 shows a substantial latency reduction from packing and device-side scheduling. Tables 10 and 15–17 further quantify representative construction, component latency, and memory usage, providing useful evidence that the added machinery remains practical.
Critical weaknesses
-
Table 10 shows that residual first-moment selection improves attention-output MSE by approximately 2.2% over uniform real-token sampling, while increasing construction latency from 5.42 to 10.06 ms. The end-to-end cost increase is small, but the corresponding final-video fidelity comparison is not reported there. Thus, the evidence strongly supports the representation-and-allocation framework, while the practical advantage of the particular nested selection rule remains less clear.
-
Sensitivity to the representation ladder and calibration parameters remains insufficiently explored. Table 6 establishes an aggregate capacity–error trend, and Table 9 supports adaptive allocation over fixed-capacity alternatives. However, these experiments do not establish robustness to changing the ladder itself, its maximum capacity, or its granularity. Similarly, Table 12 demonstrates the benefit of enabling mean-mass calibration, but does not show sensitivity to its coefficient and clipping threshold. A small parameter study would clarify how much tuning is required across models.
Questions for authors
-
Under identical routes and capacities, how do uniform sampling and residual first-moment selection compare in final-video PSNR, SSIM, and LPIPS? Are the differences consistent across prompts?
-
How do alternative capacity ladders and maximum capacities affect the quality–latency trade-off under comparable compute budgets? How sensitive are results to the mean-calibration coefficient and clipping threshold?
-
Can the authors explain the Wan T2V discrepancy between improved paired-to-dense fidelity and lower VBench macro scores relative to SVOO and SVG-EAR? Are the differences concentrated in particular quality dimensions or content categories?
-
If a compatible implementation was available before submission, could the authors include a comparison with a residual-reconstruction method such as SparsePR, or explain the practical barriers to a fair comparison?
4 comments
Response to Reviewer xpSY (2/2)
Mean-mass calibration. We use alpha=1 and clipping threshold 2. In the original Table 12, disabling calibration increases mean attention-output MSE from 6.750 × 10⁻⁴ to 7.217 × 10⁻⁴, an increase of 6.92% relative to the calibrated default, with Q/K/V, clustering, routes, and other approximations held fixed.
Separately, the fixed-state coefficient/clipping sensitivity check gives the following changes, each relative to the default within that sensitivity check:
| Alpha | Clipping threshold | Change in mean output MSE |
|---|---|---|
| 0.5 | 2 | +2.47% |
| 1 | 2 | 0% |
| 1.5 | 2 | −0.27% |
| 1 | 1 | +0.04% |
| 1 | 4 | 0% |
These parameter changes have small effects on local output MSE around the chosen default in the evaluated states.
Q3. Wan T2V VBench discrepancy.
Thank you for highlighting the difference between the Wan T2V fidelity and VBench results. The goal of this task is to accelerate the original video generator while reproducing its dense outputs as closely as possible. In this setting, training-free sparse attention methods generally aim to approximate dense attention, and PSNR is a primary metric for evaluating how closely the resulting videos reproduce their dense references. VBench serves a different purpose: it evaluates properties of the completed video without using the paired dense output as a reference. Because the score is computed on the completed video, it does not provide direct guidance on which interactions to retain or approximate in an individual attention call. Dense attention, in contrast, provides a direct reference for those local decisions. Exact reproduction would preserve the dense video's VBench scores under the same evaluation, but higher reconstruction fidelity does not imply monotonically higher VBench scores.
| VBench dimension | CoFiRA | SVOO | SVG-EAR |
|---|---|---|---|
| Subject consistency | 95.77 | 95.70 | 95.70 |
| Background consistency | 96.19 | 96.21 | 96.18 |
| Motion smoothness | 98.71 | 98.71 | 98.71 |
| Dynamic degree | 41.33 | 43.17 | 42.83 |
| Aesthetic quality | 59.53 | 59.69 | 59.65 |
| Imaging quality | 67.74 | 68.14 | 68.02 |
| Six-dimension macro | 76.55 | 76.94 | 76.85 |
On Wan T2V, CoFiRA outperforms SVOO and SVG-EAR on all three paired-reference metrics (PSNR, SSIM, and LPIPS), while its six-dimension VBench macro is slightly lower. Table 13 shows that most of the gap comes from dynamic degree, which accounts for approximately 0.31 of the 0.39-point gap to SVOO and 0.25 of the 0.30-point gap to SVG-EAR. On Wan I2V, CoFiRA achieves both higher reconstruction fidelity and the highest VBench macro among the evaluated sparse methods: 81.14, compared with 81.00 for SVG-EAR (Table 1).
Q4. SparsePR.
Thank you for suggesting a comparison with a residual-reconstruction method. We added a comparison with the official SparsePR implementation on the jointly supported HunyuanVideo-13B backbone. Both methods use the same weights, prompts, sampler, guidance settings, video export, and paired Dense-reference pipeline, retaining their respective default operating points. Mean results over five prompts are:
| Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|
| SparsePR | 25.80 | 0.835 | 0.174 |
| CoFiRA | 30.01 | 0.907 | 0.120 |
CoFiRA improves PSNR by 4.21 dB, SSIM by 0.072, and LPIPS by 0.054, with all three metrics improving on every prompt. These are paired-reference fidelity results at the stated operating points.
The author has addressed most of my questions; I am maintaining my current score and am inclined to accept this paper.
Thank you for taking the time to read our response and share your follow-up. We appreciate your thoughtful feedback and are glad that our response addressed most of your questions. If any further questions come up, we would be happy to discuss them.
Response to Reviewer xpSY (1/2)
Thank you for recognizing the representation, allocation, and runtime contributions. We address the selector comparison, parameter sensitivity, VBench dimensions, and SparsePR below.
Q1. Residual matching versus Uniform across prompts.
Thank you for suggesting a comparison between residual matching and simpler uniform selection. This is a useful control for understanding how much final-video fidelity depends on the representative selection rule itself. We therefore compared the two selectors across five prompts while keeping the rest of the method unchanged.
Before presenting the results, we would like to clarify the role of the selector in CoFiRA. Our core contribution is to provide multiple regional representation capacities and allocate them according to regional demand. The controlled ablations in Tables 2, 4, and 9 support this design as the main source of the fidelity gains. Residual matching and uniform selection differ in how they construct representatives for a given capacity, but both retain the same representation levels and demand-aware allocation policy. The comparison below thus isolates the role of the selector, while CoFiRA's core contribution is unchanged under either choice.
Uniform selection yields slightly higher mean PSNR, while residual matching yields slightly better mean SSIM and LPIPS. The differences are small on all three metrics, indicating comparable final-video fidelity. Residual matching is also efficient: it constructs one ordered sequence and reuses its prefixes across all capacities. In the end-to-end benchmark, the generation times differ by only about 1% (Table 10).
At comparable video fidelity and end-to-end runtime, residual matching offers stronger worst-case control of the first-moment error in the normalized K/V sketch space. Let H be the maximum distance from a token to its cluster mean in this space. When at most half of the cluster's tokens are selected, fixed-position uniform selection can incur a squared first-moment error of H2 in the worst case. In contrast, the deterministic bound in Appendix A.3 for a greedy residual-matching prefix of size m is at most H2/m. This bound holds regardless of token order and tightens as capacity increases.
Table 10 provides complementary empirical evidence: with attention inputs, routes, and capacities held fixed, residual matching reduces attention-output MSE by 2.2%, from 9.11 × 10⁻⁴ to 8.91 × 10⁻⁴. Given comparable video fidelity and generation time, the stronger first-moment guarantee and lower measured local error motivate our default choice. Both selectors preserve CoFiRA's core design: allocating representation capacity according to regional demand.
Q2. Capacity ladder, maximum capacity, and calibration sensitivity.
Thank you for asking about sensitivity to the capacity ladder and mean-mass calibration. We address these two settings separately below.
Capacity allocation. The 2/4/8/16 ladder supplies intermediate capacities through prefixes of one nested construction. Equal quartiles are a simple rank-based default for assigning these capacities to regions ordered by sensitivity.
We tested an alternative proportion allocation and ladders with maximum capacities 8 and 32 on one full video. The alternative 16/8/4/2 proportions of 30/15/20/35% preserve the default's nominal average capacity of 7.5 representatives. All variants use the same generation settings and attention backend. Exact regions are retained, and intermediate coverage is adjusted to fit the default row-wise extra representative-interaction cap; larger capacities generally cover fewer regions.
The alternative proportions give similar fidelity on this video. The 2/4/8 setting yields lower PSNR/SSIM; the 2/4/8/16/32 setting yields better SSIM/LPIPS and slightly lower PSNR than the default. These results support 2/4/8/16 as a practical default among the tested alternatives. This is a fidelity comparison under a common interaction-budget cap; no per-ladder latency measurements are reported.