- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PathSpan, an efficient attention operator for video diffusion transformers. The main idea is that the value of a long-range connection between two frames depends on how the content moves through the frames in between. Most existing sparse and linear attention methods instead look only at the two end frames. PathSpan builds long-range temporal connections from local matches between adjacent frames. Each token attends to an 11x11 window in the neighboring frame, plus a learned "no match" option that lets the path stop, for example under occlusion. These local maps are chained through two recurrent scans over time, one toward the past and one toward the future. A learned per-channel decay controls how far back or ahead each channel reads. Full attention within each frame is then applied on top of the propagated features, so the cost grows linearly with the number of frames.
The authors fine-tune the method on Wan2.2-5B, LTX-Video-13B and Cosmos-Predict2.5-2B. They report 1.5x to 4x faster end-to-end generation than a dense fine-tuned baseline, with very similar VBench scores. PathSpan also compares well against seven efficient attention methods on motion binding and temporal coherence. Ablations suggest that following the motion, the per-channel decay and the option to stop each contribute to the results.
Critical strengths
Clear and well-supported motivation: The idea that a long-range connection should depend on how content moves through the frames in between is intuitive. The DFSAttn experiment backs it up: point accuracy stays flat while track reachability drops from 99.8% to 73.3% and motion binding also drops. This matters because it gives the community a useful way to look at efficient attention beyond endpoint scores, which could matter beyond this specific method.
Strong speedup at similar quality on three backbones: PathSpan gives 1.5x to 4x faster end-to-end generation on Wan2.2, LTX-Video and Cosmos, with VBench scores very close to dense fine-tuning. Timing covers the full pipeline, not just the attention call. This matters most to my assessment because the main goal of the paper is practical efficiency, and the result holds on three quite different models rather than one.
Consistent gains over efficient attention baselines, supported by ablations: Against seven efficient attention methods, PathSpan has better motion binding and temporal coherence in every comparison, and each trained baseline is matched with a PathSpan model trained the same way. The ablations remove each component one at a time, and each removal hurts a different metric in a way that fits its intended role. This matters because it suggests the gains come from the proposed design and are not just a side effect of extra fine-tuning.
Critical weaknesses
-
Missing factorized attention baseline. PathSpan keeps full attention within each frame and adds a cheap temporal operator on top. The closest simple alternative is factorized space-time attention, where full spatial attention is followed by temporal attention across all frames at the same spatial position, e.g. as used in video diffusion models (Ho et al 2022). For these clip lengths the temporal part is cheap. Without this baseline it is hard to assess how much of the gain comes from following routes and how much comes from simply keeping exact spatial attention. Since the paper's main claim is about the value of following routes, this comparison could be quite helpful in supporting this claim.
-
No human evaluation. All results come from automatic metrics. VBench totals are very close to dense fine-tuning, but requested-motion accuracy drops by 2.0 points on LTX and 3.3 points on Cosmos. The paper argues this only affects which motion category is recognized, but that argument relies on other automatic metrics. Since the claimed differences are small, a small paired human study on motion-heavy prompts would help confirm that prompt adherence and perceived quality are preserved. This matters because "matched quality" is a central claim and the quantitative benchmarks are known to be not as reliable.
-
No evaluation on longer videos. The main selling point is that cost grows linearly with the number of frames. However, all experiments use the native clip lengths of each model, and the per-channel decay is initialized relative to the training clip length. It is not clear whether quality holds when the method is applied to longer videos than those seen during adaptation. Measured speed and memory at a few longer lengths would also help show how much of the expected gain carries over end to end, given that the two scans are sequential over frames. Long video generation is where a linear-time method should have the clearest advantage over dense attention, so showing that quality holds there would make the paper's contribution considerably stronger.
-
Novelty and some claims could be framed more carefully. Chaining adjacent-frame affinities, attention along motion paths, and linear-time scans with learned decay all exist in prior work. Some closely related work is not discussed. One example is FLATTEN (Cong et al., ICLR 2024), which makes patches on the same optical flow path attend to each other in video diffusion. The contribution seems to lie mainly in how these parts are combined into an efficient operator, which is still useful, but the paper presents the gap as larger. A few claims also seem stronger than the evidence. For example, "generation follows the routes" rests on two budget settings of one method, where lowering the budget removes many connections, not just routes.
Practical trade-offs are not fully explored. The two scans are sequential over frames, and the paper gives no kernel details or breakdown of the attention cost. This makes the speedups hard to reproduce, and it is unclear how they transfer to other hardware or settings. Also, once a route stops (for example under occlusion), frames after the object reappears cannot directly access the content from before the occlusion, while dense attention can. This case is not evaluated. This matters for judging how widely the method can be adopted.
Questions for authors
-
Factorized attention baseline. Could you compare against factorized space-time attention at similar cost, meaning full spatial attention plus temporal attention across frames at the same position (as in Ho et al., 2022)? If PathSpan clearly beats this, it would strongly support the claim that following routes is what matters.
-
Human evaluation. Could you run a small paired blinded human study (PathSpan vs. Dense-FT) on motion-heavy prompts? This would show whether the drops in requested-motion accuracy on LTX and Cosmos are visible to people.
-
Longer videos. Have you tried running the adapted models on clips longer than those used during adaptation, for example 2x the native length? Does quality hold? Measured end-to-end speed and memory at these lengths would also help.
-
Implementation and timing. How are the windowed matching and the two scans implemented (custom kernel, Triton, or standard PyTorch ops)? Could you give a breakdown of attention time into local matching, scans and within-frame attention? Also, how do the sequential scans work with the multi-GPU setup used for timing?
-
Minor (presentation): Figure 2 is hard to read. Labels are not clearly tied to their segments (for example "Attention Surgery" is far from its point). The bubble sizes for peak memory look nearly the same for all methods, and since this seems to be the only place memory is reported, adding the memory numbers to Table 2 would help. It would also help to mark which end of each segment is PathSpan, since the cheaper end is not always PathSpan (e.g., VMonarch). Also, if I read the figure correctly, the VSA and VMoBA labels appear to be swapped compared to Table 2 (the dark blue segment matches the VSA numbers and the red one matches VMoBA). Could you check this?
4 comments
Response to Reviewer YpRR (2/3): perceived quality and longer videos
Response to Reviewer YpRR (2/3)
W2 / Q2. Does close quality hold where the MAC drops?
You are right that the earlier wording leaned on aggregate automatic scores. The revised paper narrows the claim to close aggregate scores with two task-specific regressions, which it reports (Sec. 1, 6.2, 7), and App. G.1 now tests each difference and locates the two regressions.
- Which differences are real. Table 10 gives 95% root intervals for all 17 scores of Table 1 on all three backbones. Only LTX dynamic-attribute binding and Cosmos requested-motion accuracy lie below zero; LTX's MAC drop is -2.0 [-7.0, +3.0], within root variation.
- The perception-aligned signal moves the right way. VMBench was built as a perception-aligned motion benchmark ("A Benchmark for Perception-Aligned Video Motion Generation"). Its temporal coherence rises on all three backbones with intervals above zero (Wan +0.17 [+0.05, +0.29]).
- What the MAC loss is. EvalCrafter's MAC compares a video's mean optical flow with a fixed threshold. Among 150 paired Cosmos videos, 17 move from correct to incorrect and 12 the other way (LTX: 14 and 11), while the action score and warping error of the same videos stay level (Table 8). The loss concerns how much motion a few prompts receive, not whether the motion looks coherent.
- What it looks like. Figure 1 shows PathSpan generations with displacement, turning and occlusion, and Figure 16 pairs Dense-FT and PathSpan generations from the same condition.
W3 / Q3. Longer videos than those seen in adaptation
We generated at twice the adapted length on all three backbones and at four times on Wan, without retraining (App. G.2, Table 12). Time and memory are measured on all backbones; quality on Wan with root 17 (motion binding on the 200 CompBench motion-binding prompts, coherence on a fixed 100-prompt VMBench subset, native rows recomputed on the same prompts).
| frames | speedup | GiB, PathSpan / Dense-FT | Wan TCS, PathSpan / Dense-FT | |
|---|---|---|---|---|
| native (Wan) | 121 | 1.83x | 43.6 / 43.9 | 94.31 / 94.08 |
| Wan 2x | 241 | 2.70x | 52.4 / 52.9 | 93.86 / 92.71 |
| Wan 4x | 481 | 4.44x | 70.1 / 70.8 | 92.94 / 89.87 |
| LTX 2x | 241 | 2.05x | 67.6 / 68.2 | |
| Cosmos 2x | 153 | 7.10x | 49.1 / 49.6 |
Three things follow. First, the end-to-end gain grows with length as the linear cost predicts (1.83x to 2.70x to 4.44x on Wan), even though the scans are sequential. Second, quality holds better for PathSpan than for Dense-FT: from native to 2x, Wan motion binding drops by 0.5 against 1.8. PathSpan's adjacent scores only see temporal offsets of one frame, so its positional inputs are those of adaptation at any length, while dense attention meets offsets it was not adapted to. Third, PathSpan is not free of degradation at 4x, which we attribute to retention half-lives initialized up to the adaptation length, and peak memory grows with the token count for both methods. App. G.2 states both; a three-root evaluation on all backbones will be added in the final version.
Continued in (3/3).
Response to Reviewer YpRR (3/3): implementation, occlusion, Figure 2
Response to Reviewer YpRR (3/3)
W5 / Q4. Kernels, the time breakdown and the multi-GPU setup
Implementation (App. G.4). The scoring and the scans are Triton kernels. The scoring kernel loads the keys covering a tile of windows into shared memory, computes the 121 logits, joins the no-match logit in the same stable softmax and writes only the local probabilities and the continuation mass (BF16). The scan kernel advances one frame per launch for all heads on a GPU and both directions, gathers the window states of the neighboring frame, accumulates in FP32 and writes the state straight into the buffer that the within-frame read (FlashAttention-2) uses as its value. At inference the maps are freed after the scans.
Multi-GPU. PathSpan keeps each backbone's native head-parallel layout: an all-to-all gives every GPU all frames for a quarter of the heads. Maps, retentions and no-match logits are per head, so both scans run inside one GPU with no communication in the recurrence; the sequential dependency is over frames within a GPU, never across GPUs.
Where the time goes (Table 14, seconds per VBench video). Dense attention takes 105.7 of 205.6 s on Wan, 111.0 of 284.9 s on LTX and 262.5 of 320.9 s on Cosmos. PathSpan replaces it by 12.5, 13.4 and 21.3 s, of which the scans are 3.4, 2.6 and 3.4 s, i.e. 1.4-4.3% of complete time. This also answers how the speedup carries over: the shared stages stay, the attention stage shrinks by 8.3-12.3x, so the complete gain is set by the share of dense attention in the dense pipeline (39% on LTX, 82% on Cosmos). A setting where attention takes a larger share, such as a longer clip, gains more (Table 12). Our timings are on A100s only, and we say so. We will release the kernels with the code.
Occlusion: what happens after a route stops
You are right that after a stop, content seen before the occlusion cannot be read directly, whereas dense attention can. We measured both sides of this (App. G.5, Table 15). The learned stop does fire at occlusions: on TAP-Vid-DAVIS its mass is 0.30-0.33 at annotated occlusion onsets against 0.13-0.16 at visible transitions (AUROC 0.69-0.76). And it has a cost: on target positions that follow a reappearance, chaining gains -0.67 to +0.18 points over direct matching, against +1.89 to +3.56 on continuously visible targets. Such positions are 3,412 of 28,174. Sec. 7 now lists this as a limitation.
Q5. Figure 2
Thank you for reading the figure this closely; you were right on every point. The VSA and VMoBA labels were swapped and are now correct. The caption states which end of each segment is PathSpan (the faster end, except for VMonarch, where it is the higher one), and Table 2 has a GiB column with the memory behind the bubbles: 43.6 GiB for PathSpan, 43.9 for Dense-FT, 42.4 to 45.8 for the operators. That narrow range is why the bubbles look alike, and Sec. 7 states that PathSpan is not designed to reduce memory.
In short
If these additions resolve your concerns, we would be grateful if you would consider raising your score. For the reviewer discussion, the points you asked about now have direct evidence:
- following the route, not exact spatial attention, separates PathSpan from factorized attention at similar cost (Table 2);
- only the two reported regressions exceed root variation, and the MAC loss is an amplitude-class change while coherence rises (Table 10);
- the speedup grows with length, and on Wan coherence holds better than with Dense-FT at 2x and 4x (Table 12);
- the scans take at most 4.3% of complete time on a standard head-parallel layout (Table 14);
- the claims are narrowed where you asked (contribution (ii), Sec. 3.3, Limitations).
Thank you again for the time you put into this review.
Thank you for the detailed response and the revised paper. The factorized and STA baselines address my main concern, and it is good to see that PathSpan stays ahead of both at a similar cost. The longer clip results, the time breakdown and the occlusion analysis are also helpful, and I appreciate that the limitations are now stated more clearly in the paper.
A small human study would still be a good addition, since the quality claims rely on automatic metrics, and it would be good to see the longer clip results with more seeds in the final version. Overall, I think the paper has improved and I remain positive about it, so I am keeping my score of weak acceptance.
Response to Reviewer YpRR (1/3): factorized baseline, novelty and claims
Response to Reviewer YpRR (1/3)
Thank you for a review that reads the paper the way we hoped it would be read. Your summary of the route idea is more compact than ours, and we were glad that you singled out the DFSAttn test (point accuracy flat while reachability drops from 99.8% to 73.3%) as a way to look at efficient attention beyond endpoint scores, and that you noticed each trained baseline has its own matched PathSpan conversion. Your questions are also the most concrete list of what the paper was missing; every one of them led to a change, listed in the general response. Below, sections and tables refer to the revised PDF.
W1 / Q1. Factorized space-time attention at similar cost
We added it, and it does not close the gap. Factorized attention keeps the native projections and replaces PathSpan's scans by exact softmax attention over all frames at the same spatial position, followed by the same exact within-frame read; it is trained with the default recipe (same data, 10k updates, three roots) and evaluated on the CompBench motion-binding prompts and the complete VMBench set. Since Reviewer VTjw asked for STA, we also added it (3x3x3 tiles of 4x4x8 tokens). Wan, Table 2 (last group):
The motion-binding margin over factorized attention is +3.7 [+2.4, +5.0] and the coherence margin +1.88 [+1.51, +2.25] (95% root intervals, Table 11). Factorized attention is 5.5 s faster, so this is the comparison at similar cost you asked for. It keeps exact spatial attention and an exact read across every frame, but each position reads its own coordinate's history, which belongs to other content once the object moves. It sits between the Fixed-address control of Table 3 (MB 35.5, which also reads the same coordinate, through the chained scan rather than exact attention over all frames) and PathSpan. Keeping exact spatial attention therefore explains part of the quality, and following the route explains the rest. Paper: Table 2 and the paragraph "Space-time and local baselines" in Sec. 6.2.
W4. Novelty framing, FLATTEN, and "generation follows the routes"
We agree on both counts and changed the text.
Continued in (2/3).