acceptodds
Under review as a conference paper at ICLR 2027

Follow the Route, Know Where to Stop: Efficient Attention for Video Diffusion

Abstract

Video diffusion transformers denoise all frames of a clip together, and at every step each token attends to every other frame, at a cost quadratic in the number of frames. Efficient attention methods cut this cost for distant frame pairs and decide which pairs to keep, or how to approximate them, by comparing the two frames directly. We show that the value of a distant connection is set in the frames between: by the route the content takes, how far that route carries and where it breaks. When a sparse method keeps fewer connections, point-correspondence accuracy stays level while a quarter of the annotated point tracks can no longer be followed and motion binding drops; when propagation cannot stop, motion grows larger and temporal coherence falls. We propose PathSpan, which builds every distant connection from adjacent-frame matches: each token scores an window of the neighboring frame together with a learned no-match entry that ends propagation, two linear-time scans chain these matches, a per-channel retention sets how far each channel reads, and attention within the frame selects from the carried content. On Wan2.2, LTX-Video and Cosmos-Predict2.5, complete generation is 1.52–4.03× faster than matched dense fine-tuning with VBench totals within 0.05 points, and removing the route, the per-channel reach or the stop lowers motion binding and temporal coherence in all three generators.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video diffusion, efficient attention, local paths, temporal modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YpRROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PathSpan, an efficient attention operator for video diffusion transformers. The main idea is that the value of a long-range connection between two frames depends on how the content moves through the frames in between. Most existing sparse and linear attention methods instead look only at the two end frames. PathSpan builds long-range temporal connections from local matches between adjacent frames. Each token attends to an 11x11 window in the neighboring frame, plus a learned "no match" option that lets the path stop, for example under occlusion. These local maps are chained through two recurrent scans over time, one toward the past and one toward the future. A learned per-channel decay controls how far back or ahead each channel reads. Full attention within each frame is then applied on top of the propagated features, so the cost grows linearly with the number of frames.

The authors fine-tune the method on Wan2.2-5B, LTX-Video-13B and Cosmos-Predict2.5-2B. They report 1.5x to 4x faster end-to-end generation than a dense fine-tuned baseline, with very similar VBench scores. PathSpan also compares well against seven efficient attention methods on motion binding and temporal coherence. Ablations suggest that following the motion, the per-channel decay and the option to stop each contribute to the results.

Critical strengths

Clear and well-supported motivation: The idea that a long-range connection should depend on how content moves through the frames in between is intuitive. The DFSAttn experiment backs it up: point accuracy stays flat while track reachability drops from 99.8% to 73.3% and motion binding also drops. This matters because it gives the community a useful way to look at efficient attention beyond endpoint scores, which could matter beyond this specific method.

Strong speedup at similar quality on three backbones: PathSpan gives 1.5x to 4x faster end-to-end generation on Wan2.2, LTX-Video and Cosmos, with VBench scores very close to dense fine-tuning. Timing covers the full pipeline, not just the attention call. This matters most to my assessment because the main goal of the paper is practical efficiency, and the result holds on three quite different models rather than one.

Consistent gains over efficient attention baselines, supported by ablations: Against seven efficient attention methods, PathSpan has better motion binding and temporal coherence in every comparison, and each trained baseline is matched with a PathSpan model trained the same way. The ablations remove each component one at a time, and each removal hurts a different metric in a way that fits its intended role. This matters because it suggests the gains come from the proposed design and are not just a side effect of extra fine-tuning.

Critical weaknesses
  1. Missing factorized attention baseline. PathSpan keeps full attention within each frame and adds a cheap temporal operator on top. The closest simple alternative is factorized space-time attention, where full spatial attention is followed by temporal attention across all frames at the same spatial position, e.g. as used in video diffusion models (Ho et al 2022). For these clip lengths the temporal part is cheap. Without this baseline it is hard to assess how much of the gain comes from following routes and how much comes from simply keeping exact spatial attention. Since the paper's main claim is about the value of following routes, this comparison could be quite helpful in supporting this claim.

  2. No human evaluation. All results come from automatic metrics. VBench totals are very close to dense fine-tuning, but requested-motion accuracy drops by 2.0 points on LTX and 3.3 points on Cosmos. The paper argues this only affects which motion category is recognized, but that argument relies on other automatic metrics. Since the claimed differences are small, a small paired human study on motion-heavy prompts would help confirm that prompt adherence and perceived quality are preserved. This matters because "matched quality" is a central claim and the quantitative benchmarks are known to be not as reliable.

  3. No evaluation on longer videos. The main selling point is that cost grows linearly with the number of frames. However, all experiments use the native clip lengths of each model, and the per-channel decay is initialized relative to the training clip length. It is not clear whether quality holds when the method is applied to longer videos than those seen during adaptation. Measured speed and memory at a few longer lengths would also help show how much of the expected gain carries over end to end, given that the two scans are sequential over frames. Long video generation is where a linear-time method should have the clearest advantage over dense attention, so showing that quality holds there would make the paper's contribution considerably stronger.

  4. Novelty and some claims could be framed more carefully. Chaining adjacent-frame affinities, attention along motion paths, and linear-time scans with learned decay all exist in prior work. Some closely related work is not discussed. One example is FLATTEN (Cong et al., ICLR 2024), which makes patches on the same optical flow path attend to each other in video diffusion. The contribution seems to lie mainly in how these parts are combined into an efficient operator, which is still useful, but the paper presents the gap as larger. A few claims also seem stronger than the evidence. For example, "generation follows the routes" rests on two budget settings of one method, where lowering the budget removes many connections, not just routes.

Practical trade-offs are not fully explored. The two scans are sequential over frames, and the paper gives no kernel details or breakdown of the attention cost. This makes the speedups hard to reproduce, and it is unclear how they transfer to other hardware or settings. Also, once a route stops (for example under occlusion), frames after the object reappears cannot directly access the content from before the occlusion, while dense attention can. This case is not evaluated. This matters for judging how widely the method can be adopted.

Questions for authors
  • Factorized attention baseline. Could you compare against factorized space-time attention at similar cost, meaning full spatial attention plus temporal attention across frames at the same position (as in Ho et al., 2022)? If PathSpan clearly beats this, it would strongly support the claim that following routes is what matters.

  • Human evaluation. Could you run a small paired blinded human study (PathSpan vs. Dense-FT) on motion-heavy prompts? This would show whether the drops in requested-motion accuracy on LTX and Cosmos are visible to people.

  • Longer videos. Have you tried running the adapted models on clips longer than those used during adaptation, for example 2x the native length? Does quality hold? Measured end-to-end speed and memory at these lengths would also help.

  • Implementation and timing. How are the windowed matching and the two scans implemented (custom kernel, Triton, or standard PyTorch ops)? Could you give a breakdown of attention time into local matching, scans and within-frame attention? Also, how do the sequential scans work with the multi-GPU setup used for timing?

  • Minor (presentation): Figure 2 is hard to read. Labels are not clearly tied to their segments (for example "Attention Surgery" is far from its point). The bubble sizes for peak memory look nearly the same for all methods, and since this seems to be the only place memory is reported, adding the memory numbers to Table 2 would help. It would also help to mark which end of each segment is PathSpan, since the cheaper end is not always PathSpan (e.g., VMonarch). Also, if I read the figure correctly, the VSA and VMoBA labels appear to be swapped compared to Table 2 (the dark blue segment matches the VSA numbers and the red one matches VMoBA). Could you check this?

4 comments
AuthorsOpenReview ↗

Response to Reviewer YpRR (1/3): factorized baseline, novelty and claims

Response to Reviewer YpRR (1/3)

Thank you for a review that reads the paper the way we hoped it would be read. Your summary of the route idea is more compact than ours, and we were glad that you singled out the DFSAttn test (point accuracy flat while reachability drops from 99.8% to 73.3%) as a way to look at efficient attention beyond endpoint scores, and that you noticed each trained baseline has its own matched PathSpan conversion. Your questions are also the most concrete list of what the paper was missing; every one of them led to a change, listed in the general response. Below, sections and tables refer to the revised PDF.

W1 / Q1. Factorized space-time attention at similar cost

We added it, and it does not close the gap. Factorized attention keeps the native projections and replaces PathSpan's scans by exact softmax attention over all frames at the same spatial position, followed by the same exact within-frame read; it is trained with the default recipe (same data, 10k updates, three roots) and evaluated on the CompBench motion-binding prompts and the complete VMBench set. Since Reviewer VTjw asked for STA, we also added it (3x3x3 tiles of 4x4x8 tokens). Wan, Table 2 (last group):

MBVMBench totalTCSs
Factorized40.270.3192.46106.9
STA42.470.6693.71114.9
PathSpan43.970.7894.34112.4

The motion-binding margin over factorized attention is +3.7 [+2.4, +5.0] and the coherence margin +1.88 [+1.51, +2.25] (95% root intervals, Table 11). Factorized attention is 5.5 s faster, so this is the comparison at similar cost you asked for. It keeps exact spatial attention and an exact read across every frame, but each position reads its own coordinate's history, which belongs to other content once the object moves. It sits between the Fixed-address control of Table 3 (MB 35.5, which also reads the same coordinate, through the chained scan rather than exact attention over all frames) and PathSpan. Keeping exact spatial attention therefore explains part of the quality, and following the route explains the rest. Paper: Table 2 and the paragraph "Space-time and local baselines" in Sec. 6.2.

W4. Novelty framing, FLATTEN, and "generation follows the routes"

We agree on both counts and changed the text.

  • Framing. Contribution (ii) now states that chained adjacent-frame affinities, attention along motion paths and linear-time scans with learned decay each exist in prior work, and that our contribution is their combination into one trainable attention operator for pretrained video transformers, together with the two readouts (track reachability, the response bound) that apply to any efficient operator. Sec. 2 says the same.
  • FLATTEN is now in Sec. 2. It makes patches on the same optical-flow path attend to each other in a training-free editor, with the flow estimated from the source video. PathSpan has no source video: its routes are probabilistic adjacent-frame maps computed from the generator's own queries and keys, trained by the generation loss, and they come with a learned stop and a per-channel reach.
  • The two-budget claim. You are right that lowering the budget removes many connections, not only routes. Sec. 3.3 now says only that motion binding falls with reachability rather than with the endpoint readout, that this shows what an endpoint diagnostic misses, and that the route itself is isolated by the controls of Sec. 6.3, which change the route at a fixed budget (Fixed address, Single hop and now the factorized baseline).

Continued in (2/3).

AuthorsOpenReview ↗

Response to Reviewer YpRR (2/3): perceived quality and longer videos

Response to Reviewer YpRR (2/3)

W2 / Q2. Does close quality hold where the MAC drops?

You are right that the earlier wording leaned on aggregate automatic scores. The revised paper narrows the claim to close aggregate scores with two task-specific regressions, which it reports (Sec. 1, 6.2, 7), and App. G.1 now tests each difference and locates the two regressions.

  • Which differences are real. Table 10 gives 95% root intervals for all 17 scores of Table 1 on all three backbones. Only LTX dynamic-attribute binding and Cosmos requested-motion accuracy lie below zero; LTX's MAC drop is -2.0 [-7.0, +3.0], within root variation.
  • The perception-aligned signal moves the right way. VMBench was built as a perception-aligned motion benchmark ("A Benchmark for Perception-Aligned Video Motion Generation"). Its temporal coherence rises on all three backbones with intervals above zero (Wan +0.17 [+0.05, +0.29]).
  • What the MAC loss is. EvalCrafter's MAC compares a video's mean optical flow with a fixed threshold. Among 150 paired Cosmos videos, 17 move from correct to incorrect and 12 the other way (LTX: 14 and 11), while the action score and warping error of the same videos stay level (Table 8). The loss concerns how much motion a few prompts receive, not whether the motion looks coherent.
  • What it looks like. Figure 1 shows PathSpan generations with displacement, turning and occlusion, and Figure 16 pairs Dense-FT and PathSpan generations from the same condition.

W3 / Q3. Longer videos than those seen in adaptation

We generated at twice the adapted length on all three backbones and at four times on Wan, without retraining (App. G.2, Table 12). Time and memory are measured on all backbones; quality on Wan with root 17 (motion binding on the 200 CompBench motion-binding prompts, coherence on a fixed 100-prompt VMBench subset, native rows recomputed on the same prompts).

framesspeedupGiB, PathSpan / Dense-FTWan TCS, PathSpan / Dense-FT
native (Wan)1211.83x43.6 / 43.994.31 / 94.08
Wan 2x2412.70x52.4 / 52.993.86 / 92.71
Wan 4x4814.44x70.1 / 70.892.94 / 89.87
LTX 2x2412.05x67.6 / 68.2
Cosmos 2x1537.10x49.1 / 49.6

Three things follow. First, the end-to-end gain grows with length as the linear cost predicts (1.83x to 2.70x to 4.44x on Wan), even though the scans are sequential. Second, quality holds better for PathSpan than for Dense-FT: from native to 2x, Wan motion binding drops by 0.5 against 1.8. PathSpan's adjacent scores only see temporal offsets of one frame, so its positional inputs are those of adaptation at any length, while dense attention meets offsets it was not adapted to. Third, PathSpan is not free of degradation at 4x, which we attribute to retention half-lives initialized up to the adaptation length, and peak memory grows with the token count for both methods. App. G.2 states both; a three-root evaluation on all backbones will be added in the final version.

Continued in (3/3).

AuthorsOpenReview ↗

Response to Reviewer YpRR (3/3): implementation, occlusion, Figure 2

Response to Reviewer YpRR (3/3)

W5 / Q4. Kernels, the time breakdown and the multi-GPU setup

Implementation (App. G.4). The scoring and the scans are Triton kernels. The scoring kernel loads the keys covering a tile of windows into shared memory, computes the 121 logits, joins the no-match logit in the same stable softmax and writes only the local probabilities and the continuation mass (BF16). The scan kernel advances one frame per launch for all heads on a GPU and both directions, gathers the window states of the neighboring frame, accumulates in FP32 and writes the state straight into the buffer that the within-frame read (FlashAttention-2) uses as its value. At inference the maps are freed after the scans.

Multi-GPU. PathSpan keeps each backbone's native head-parallel layout: an all-to-all gives every GPU all frames for a quarter of the heads. Maps, retentions and no-match logits are per head, so both scans run inside one GPU with no communication in the recurrence; the sequential dependency is over frames within a GPU, never across GPUs.

Where the time goes (Table 14, seconds per VBench video). Dense attention takes 105.7 of 205.6 s on Wan, 111.0 of 284.9 s on LTX and 262.5 of 320.9 s on Cosmos. PathSpan replaces it by 12.5, 13.4 and 21.3 s, of which the scans are 3.4, 2.6 and 3.4 s, i.e. 1.4-4.3% of complete time. This also answers how the speedup carries over: the shared stages stay, the attention stage shrinks by 8.3-12.3x, so the complete gain is set by the share of dense attention in the dense pipeline (39% on LTX, 82% on Cosmos). A setting where attention takes a larger share, such as a longer clip, gains more (Table 12). Our timings are on A100s only, and we say so. We will release the kernels with the code.

Occlusion: what happens after a route stops

You are right that after a stop, content seen before the occlusion cannot be read directly, whereas dense attention can. We measured both sides of this (App. G.5, Table 15). The learned stop does fire at occlusions: on TAP-Vid-DAVIS its mass is 0.30-0.33 at annotated occlusion onsets against 0.13-0.16 at visible transitions (AUROC 0.69-0.76). And it has a cost: on target positions that follow a reappearance, chaining gains -0.67 to +0.18 points over direct matching, against +1.89 to +3.56 on continuously visible targets. Such positions are 3,412 of 28,174. Sec. 7 now lists this as a limitation.

Q5. Figure 2

Thank you for reading the figure this closely; you were right on every point. The VSA and VMoBA labels were swapped and are now correct. The caption states which end of each segment is PathSpan (the faster end, except for VMonarch, where it is the higher one), and Table 2 has a GiB column with the memory behind the bubbles: 43.6 GiB for PathSpan, 43.9 for Dense-FT, 42.4 to 45.8 for the operators. That narrow range is why the bubbles look alike, and Sec. 7 states that PathSpan is not designed to reduce memory.

In short

If these additions resolve your concerns, we would be grateful if you would consider raising your score. For the reviewer discussion, the points you asked about now have direct evidence:

  • following the route, not exact spatial attention, separates PathSpan from factorized attention at similar cost (Table 2);
  • only the two reported regressions exceed root variation, and the MAC loss is an amplitude-class change while coherence rises (Table 10);
  • the speedup grows with length, and on Wan coherence holds better than with Dense-FT at 2x and 4x (Table 12);
  • the scans take at most 4.3% of complete time on a standard head-parallel layout (Table 14);
  • the claims are narrowed where you asked (contribution (ii), Sec. 3.3, Limitations).

Thank you again for the time you put into this review.

Thank you for the detailed response and the revised paper. The factorized and STA baselines address my main concern, and it is good to see that PathSpan stays ahead of both at a similar cost. The longer clip results, the time breakdown and the occlusion analysis are also helpful, and I appreciate that the limitations are now stated more clearly in the paper.

A small human study would still be a good addition, since the quality claims rely on automatic metrics, and it would be good to see the longer clip results with more seeds in the final version. Overall, I think the paper has improved and I remain positive about it, so I am keeping my score of weak acceptance.

Reviewer_VTjwOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed a idea for sparse attn in video gen. It introduce a windowed attention with soft endpoint. The boundary is learned by softmax

Critical strengths
  1. I really like the idea - and it seems that authors made it work. That is incredible given the training difficulty of video diffusion
  2. Many experiments are inlcuded and it also tested on LTX and stated its limitation.
Critical weaknesses

I do not think those are major problems. Just a few points:

  1. How effective is the boundary? maybe the network just leaned a cheat to estimate some fixed trivial boundary? and even in that case the model wont fail so we cannot tell how much improvments it brings right? Or am I missing anything?
  2. Well this paper really need to compare to spatial+temporal attn, or 3D attn like STA [1]?

[1] Peiyuan Zhang, Yongqi Chen, Runlong Su, Hangliang Ding, Ion Stoica, Zhengzhong Liu, Hao Zhang. Fast Video Generation with Sliding Tile Attention. ICML 2025. arXiv:2502.04507.

Questions for authors
  1. why the Wan and Cosmos have same Dense-FT and PathSpan? and their same SA and PC?
  2. How is the kernel devided on 4 GPUs?
2 comments
AuthorsOpenReview ↗

Response to Reviewer VTjw (1/2): is the stop effective, and space-time baselines

Response to Reviewer VTjw (1/2)

Thank you for the kind words. "I really like the idea, and it seems that authors made it work" means a lot to us, because getting a new attention operator to train stably inside three pretrained video generators was the hardest part of this project, and you are the first reader to say so. We also appreciate that you noticed the LTX results and the limitations we reported rather than only the best numbers. Your two questions about the stop and about space-time attention go straight at the two things a skeptical reader would want checked, so we answered both with new experiments. Tables and sections refer to the revised PDF.

W1. Does the learned stop do anything, or could it settle on a fixed, trivial boundary?

It does not settle on a fixed boundary, and a fixed boundary costs coherence. You are right that the model would still produce videos with a trivial stop, so the question has to be answered with a control that keeps everything except the content dependence. We added exactly that, plus a direct readout.

  1. Constant stop (new row in Table 3, on Wan). Each head keeps the mean stop mass that it learned, but the stop no longer depends on how well any candidate matches. The model keeps the same average reach and is retrained under the same protocol (three roots). Amplitude stays within 0.3 points of PathSpan, yet temporal coherence falls from 94.34 to 92.97 and motion binding from 43.9 to 42.6. If the learned stop were a fixed rate, this control would cost nothing; the measured drop is the value of stopping in the right places.
  2. The stop fires where content disappears (App. G.5, Table 15). We read the stop mass at every TAP-Vid-DAVIS track point and label each transition by whether the point becomes occluded. The stop mass is 0.30-0.33 at occlusion onsets against 0.13-0.16 at visible transitions, with AUROC 0.69-0.76 across backbones (a fixed stop gives 0.5 by construction). The network never sees these annotations; it learns this from the generation loss alone.
  3. Removing the stop (No-match removed, already in Table 3) makes every route continue: amplitude rises in all three generators while coherence falls by 2.6-3.3 points.

The learned stop mass at visible transitions is also far from its initial value of 0.5, which rules out the no-match entry simply staying where it started. Together these show the boundary is content-dependent and that its improvement is measurable.

W2. Comparison with spatial + temporal attention and with 3D local attention (STA)

Both are now in Table 2 (last group), trained with our default recipe on Wan (three roots), evaluated on CompBench motion binding and the complete VMBench set, and compared with default PathSpan. Factorized attention replaces the scans by exact attention over all frames at the same position, followed by the same within-frame read. STA uses 3x3x3 tiles of 4x4x8 tokens, the window closest to PathSpan's time; on A100 it runs through its FlexAttention implementation, since its fused kernel targets H100.

MBVMBench totalTCSs
Factorized40.270.3192.46106.9
STA42.470.6693.71114.9
PathSpan43.970.7894.34112.4

STA is close on the VMBench total, because it keeps exact attention in a 3D neighborhood, but frames farther apart than its tile window are reached only by stacking layers, so motion binding and coherence are lower. Factorized attention is faster but reads each position's own history, which belongs to other content once an object moves. Both margins have root intervals above zero (Table 11): STA MB +1.5 [+0.7, +2.2], TCS +0.63 [+0.41, +0.85].

Continued in (2/2).

AuthorsOpenReview ↗

Response to Reviewer VTjw (2/2): identical VideoPhy-2 scores and the 4-GPU layout

Response to Reviewer VTjw (2/2)

Q1. Why do Wan and Cosmos show the same SA, PC (and Joint) for Dense-FT and PathSpan?

Thank you for catching this. It was a copy error: in the submitted Table 1, the PathSpan SA and PC cells, and the joint cells for Wan and Cosmos, were copied from the Dense-FT row. The revised Table 1 shows the PathSpan evaluations, and Table 10 gives the differences with 95% root intervals:

Dense-FT → PathSpanWanLTXCosmos
SA3.61 → 3.63 (+0.017 [-0.053, +0.086])3.68 → 3.66 (-0.024 [-0.103, +0.055])3.64 → 3.63 (-0.009 [-0.075, +0.057])
PC3.20 → 3.21 (+0.007 [-0.067, +0.080])3.17 → 3.18 (+0.008 [-0.050, +0.067])3.36 → 3.34 (-0.026 [-0.106, +0.054])
Joint (%)34.4 → 34.7 (+0.28 [-4.33, +4.89])34.2 → 34.1 (-0.06 [-4.43, +4.32])40.4 → 39.8 (-0.62 [-5.15, +3.91])

Every interval includes zero: with three roots, the evaluation does not resolve a statistically significant difference on these metrics, and we never use VideoPhy-2 to support a gain. App. G.1 notes the correction.

Q2. How is the kernel divided on 4 GPUs?

By heads, using each backbone's native head-parallel layout (App. G.4). An all-to-all before attention gives every GPU all frames for a quarter of the heads (6 of 24 on Wan, 8 of 32 on LTX, 4 of 16 on Cosmos). The adjacent-frame maps, retentions and no-match logits are all per head, so both scans of a head run on one GPU with no communication inside the recurrence, and a second all-to-all restores the sequence layout. The scans are therefore sequential over frames within a GPU, never across GPUs. Dense-FT uses the same layout with FlashAttention-2, so the timing comparison is like for like. The scoring and scan steps are Triton kernels; the within-frame read is FlashAttention-2.

Table 14 gives the resulting stage times: the scans take 1.4-4.3% of complete generation time, and all PathSpan attention stages together are 8.3-12.3x cheaper than dense attention on the same layout.

Summary

If these answers resolve your questions, we would be grateful if you would consider raising your score and supporting the paper in the discussion. Briefly, for that discussion:

  • the stop is content-dependent: a constant stop with the same mean costs coherence and motion binding (Wan), and the learned stop rises at real occlusions (Tables 3, 15);
  • spatial + temporal attention and STA, at similar cost, both trail PathSpan on motion binding and coherence (Tables 2, 11);
  • the identical VideoPhy-2 values were a copy error in Table 1, now corrected, with every difference inside root variation (Table 10);
  • the scans run per head inside one GPU and take a small share of time (Table 14).

Thank you again for your support of the idea and for questions that made the paper more convincing.

Reviewer_5J4POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PathSpan, which replaces video DiT self-attention with adjacent-frame matching, bidirectional propagation, learned retention and stopping, and within-frame attention. Its cost is linear in temporal length. Experiments on three backbones report 1.52–4.03× end-to-end speedups with similar aggregate automatic scores, although perceptual quality preservation needs stronger validation.

Critical strengths
  1. The component ablations are well structured. Removing or modifying one component at a time helps isolate the contributions of motion-following routes, multi-hop propagation, per-channel retention, stopping, and within-frame attention.

  2. Evaluation across three pretrained video-generation backbones provides useful evidence that the approach transfers beyond a single model.

  3. The paper reports substantial end-to-end speedups, including conditioning, guidance, and decoding. The matched Dense-FT baseline and matched adaptation protocols are valuable controls for separating architectural effects from fine-tuning gains.

Critical weaknesses
  1. The perceptual quality claims rely heavily on automatic metrics. I personally find some of those benchmark saturated and not correlate well with human preferences. Although the evaluation covers several benchmarks, many aggregate scores are very close. Such differences do not by themselves establish perceptual equivalence or meaningful superiority. The three-seed averages and selected confidence intervals are helpful, but uncertainty estimates for the principal comparisons would strengthen the conclusions. I consider a blinded, paired human preference evaluation important, particularly for motion consistency, identity preservation, and overall quality.

  2. Some task-specific regressions remain. Requested-motion accuracy decreases from 64.7 to 62.7 on LTX and from 69.3 to 66.0 on Cosmos. Dynamic-attribute binding also declines on LTX. The paper acknowledges these changes, but they qualify the broad claim of matched quality. Examples illustrating the affected behaviors would help assess their practical significance.

  3. Long-duration generation remains insufficiently evaluated. The main experiments use approximately 4–5-second clips. Although Appendix F includes 150-frame Minecraft rollouts through multiple prediction windows, it still do not find it long enough.

  4. The related-work discussion should include LinGen. LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexity.

Questions for authors
  1. Table 6 already explores window sizes and their quality–latency tradeoffs. Could this be expanded into a matched quality–latency–memory Pareto comparison, sweeping both PathSpan’s window size and the competing methods’ sparsity budgets?

  2. Could the authors provide more implementation detail for adjacent-frame scoring and the sequential scans? Useful details include whether custom CUDA or Triton kernels are used, which operations are fused, how intermediate probabilities and states are stored, and a runtime breakdown across local scoring, scanning, and within-frame attention.

  3. A discussion of Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation would also be useful, particularly the distinction between its frame-wise memory updates and PathSpan’s spatial correspondence-based transport. Given its recent release, I view this as a useful addition rather than a required experimental comparison.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 5J4P (1/3): perceptual quality, uncertainty and the two regressions

Response to Reviewer 5J4P (1/3)

Thank you for a careful and fair review. We are glad that you found the one-at-a-time component ablations well structured and the matched Dense-FT baseline and adaptation protocols useful for separating the architecture from the fine-tuning; those are the controls we built the study around. Your main concern, that close automatic scores do not by themselves establish perceptual equivalence, is right, and it is the reason the revised paper now has intervals for every score, an analysis of the two regressions on the paired videos, and more careful wording. Sections and tables refer to the revised PDF.

W1. Perceptual quality and uncertainty for the principal comparisons

Wording. We no longer write "matched quality". The paper now speaks of close aggregate scores and names the two task-specific regressions we report (Sec. 1, 6.2, 7), and the Limitations paragraph states that close aggregate scores do not mean that every task is preserved.

Perception-aligned evidence. Among the six suites, VMBench was built as a perception-aligned motion benchmark, and it covers the aspects you named: temporal coherence and motion smoothness for motion consistency, object integrity for identity. In Table 10, its coherence gain lies above zero on all three backbones, motion smoothness and object integrity have no interval below zero, and image quality (VBench IQ, DOVER) shows no interval below zero either. The frame figures already in the paper (Figures 1, 13-16) show identity kept through displacement, turning and occlusion.

Uncertainty (App. G.1). Table 10 gives the PathSpan minus Dense-FT difference with a 95% root interval for all 17 scores of Table 1 on all backbones, and Table 11 does the same for the motion-binding and coherence margins against all seven operators and the two new space-time baselines. In Table 10, only the two regressions already reported have intervals below zero, and the coherence gains are above zero on all three backbones (e.g. Wan +0.17 [+0.05, +0.29]). In Table 11, 15 of 18 margins are above zero; the three that are not (MB against SLA, TCS against VMoBA and against Surgery) are the smallest ones, and App. G.1 says so rather than counting them as wins.

W2. Task-specific regressions and their practical significance

We agree that they qualify the broad claim, and the paper now says this where the claim is made. The new evidence locates them more precisely:

  • Root level. Requested-motion accuracy on LTX is -2.0 [-7.0, +3.0], within root variation; on Cosmos it is -3.3 [-6.2, -0.5], a real loss. LTX dynamic-attribute binding is -0.17 [-0.24, -0.11] on a score of 9.44.
  • What changes. EvalCrafter's MAC classifies the motion amplitude that the prompt requests. Among the 150 paired Cosmos videos, 17 move from correct to incorrect and 12 the other way (LTX: 14 and 11), while the action score and warping error of the same videos stay level (Table 8). The loss concerns how much motion a few prompts receive, not the coherence of the motion.
  • Examples. MAC compares a video's mean optical flow with a fixed threshold, so the flipped videos differ in how much they move rather than in visible artifacts. The frame figures already in the paper show the behaviors the two scores measure: Figure 15 shows the within-clip attribute change that DAB scores, and Figure 16 shows paired Dense-FT and PathSpan generations from the same condition.

Continued in (2/3).

AuthorsOpenReview ↗

Response to Reviewer 5J4P (2/3): long-duration generation, LinGen and Video DeltaNet

Response to Reviewer 5J4P (2/3)

W3. Long-duration generation

We agree that 4-5 second clips and the Minecraft rollouts leave this open. We now generate at twice the adapted length on all three backbones (Wan and LTX 241 frames, Cosmos 153) and at four times on Wan (481 frames, 20 s at 24 fps), with no retraining (App. G.2, Table 12). Time and memory are measured on all backbones; quality on Wan, the backbone of Table 2, with one root (motion binding on the 200 CompBench motion-binding prompts, coherence and amplitude on a fixed 100-prompt VMBench subset; native values recomputed on the same prompts).

Wan, root 17speedupMB, PathSpan / Dense-FTTCS, PathSpan / Dense-FT
native, 121 frames1.83x43.6 / 44.494.31 / 94.08
2x, 241 frames2.70x43.1 / 42.693.86 / 92.71
4x, 481 frames4.44xnot run92.94 / 89.87

At 2x, LTX and Cosmos are 2.05x and 7.10x faster. The speedup grows with length, as the linear cost predicts, and quality holds better for PathSpan than for Dense-FT: PathSpan's adjacent scores see only one-frame temporal offsets, so its positional inputs are those of adaptation at any length, whereas dense attention meets offsets it was not adapted to. PathSpan does degrade at 4x, since its retention half-lives were initialized up to the adaptation length. We do not claim minute-length generation, which would need training on long clips; the claim we check is that the linear cost carries to longer clips without a quality collapse. A three-root quality evaluation on all backbones will be added in the final version.

W4 and Q3. LinGen and Video DeltaNet

Thank you for both pointers; they sharpen the positioning, and both are now discussed in Sec. 2.

  • LinGen (CVPR 2025) reaches minute-length video with linear cost through its MATE block: a bidirectional Mamba2 branch over rearranged tokens (with review tokens) plus temporal Swin attention, trained from scratch for long videos.
  • Video DeltaNet pairs local softmax attention with a bidirectional linear memory that each frame updates once through a frame-wise delta rule, aimed at livestream generation and brought into pretrained models by teacher alignment.

The difference you asked about is where the long-range state lives. In both works it is a summary that later frames read by content: it keeps what earlier frames held, but not where that content moved. PathSpan's state is kept per spatial position and is carried along adjacent-frame matches, so a token reads the content that arrived at its own position along its route, and a learned stop ends the route when the match fails. The two designs are complementary: a frame-level memory of the Video DeltaNet kind is a natural way to hold content across a stop, the reappearance case that we now list as PathSpan's limitation (Sec. 7, App. G.5). As you suggested, we treat Video DeltaNet as related work rather than an experimental baseline, given its recent release.

Continued in (3/3).

AuthorsOpenReview ↗

Response to Reviewer 5J4P (3/3): quality-latency-memory sweep and implementation

Response to Reviewer 5J4P (3/3)

Q1. A matched quality-latency-memory comparison over budgets

We added this on Wan (App. G.3, Table 13). The table places PathSpan at radii 1, 5 and 9 (matched retraining, Table 6) and every operator at its default budget, each with motion binding, complete CompBench time and peak memory. SVG2 and DFSAttn, which need no training, also run at one cheaper and one more expensive budget on the three Dense-FT roots.

  • No operator point reaches the motion binding of default PathSpan (43.9) in less than its 114.4 s. SLA, the strongest operator, comes closest (44.5 at 118.5 s), against 44.8 at 124.5 s for PathSpan at radius 9.
  • The more expensive SVG2 and DFSAttn budgets add 16.7 and 9.1 s and reach 43.5 and 43.1, still below default PathSpan; their cheaper budgets save time and lose motion binding.
  • Peak memory stays within 42.4-45.8 GiB for every point, so no method buys its speed with memory; Table 2 now has the same memory column.

Retraining the five trained operators at further budgets takes a separate adaptation per point; we will extend the sweep in the final version. We read the current result as PathSpan being on the frontier with SLA close by, a more modest and, we think, more useful statement than winning every point.

Q2. Implementation: kernels, fusion, storage and a runtime breakdown

Kernels and fusion (App. G.4). Adjacent scoring and the scans are Triton kernels. The scoring kernel loads the keys that cover a tile of windows into shared memory, computes the (2r+1)^2 logits, masks invalid offsets, adds the no-match logit to the same stable softmax and writes only the m local probabilities and the continuation mass, in BF16. The scan kernel advances one frame per launch for all heads on a GPU and both directions at once: it reads the step probabilities, gathers the neighbor states in the window, applies the update with FP32 accumulation, and writes the state directly into the buffer that the within-frame read uses as its value. The within-frame read is FlashAttention-2.

Storage. Only the S x m local probabilities per frame, head and direction are stored, never an S x S map. At inference they are freed after the scans; in training, each block keeps its maps and states for the backward pass, which runs the recurrence in reverse frame order.

Runtime breakdown (Table 14, seconds per VBench video on four A100s):

WanLTXCosmos
Dense attention (Dense-FT)105.7111.0262.5
PathSpan adjacent scoring1.81.22.2
PathSpan scans, both directions3.42.63.4
PathSpan within-frame read5.58.013.8
Complete, Dense-FT / PathSpan205.6 / 112.4284.9 / 187.3320.9 / 79.7

The within-frame read is the largest part of PathSpan's attention time, and the sequential scans take 1.4-4.3% of complete time. The complete speedup follows the share of dense attention in the dense pipeline, which is why Cosmos gains most.

Closing

Thank you again. Your review asked for exactly the evidence that a claim of close quality needs, and the paper is more honest and more convincing for it. If the intervals, the regression analysis and the long-clip results resolve your concerns, we would be grateful if you would reconsider your score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2): what changed in the revised paper

General response (1/2): what changed in the revised paper

We thank the three reviewers for their time and for reviews concrete enough to act on directly. Read side by side, they recognize the same core of the paper: the view that a distant connection is decided by the route between its two frames (Reviewer YpRR: "a useful way to look at efficient attention beyond endpoint scores"; Reviewer VTjw: "I really like the idea"), speedups measured on complete generation across three quite different backbones (YpRR, 5J4P), and matched controls with one-at-a-time ablations that separate the operator from the fine-tuning (YpRR, 5J4P).

The concerns overlapped as well, and each one led to a change in the revised PDF. All changes are in blue; the title and abstract are unchanged and the main text stays within 10 pages.

  1. Uncertainty for the main comparisons (App. G.1, Tables 10 and 11). Root intervals for every score of Table 1 and for the motion-binding and coherence margins of Table 2, together with an analysis of the two regressions on the paired videos and pointers to the frame figures that show them. (5J4P, YpRR, VTjw)
  2. Factorized space-time attention and STA as matched baselines (Table 2, last group; Sec. 6.2). (YpRR, VTjw)
  3. Longer clips: twice the adapted length on all backbones (time and peak memory) and, on Wan, quality at twice and four times the length (481 frames) (Sec. 6.2; App. G.2, Table 12). (YpRR, 5J4P)
  4. Quality, latency and memory: PathSpan's radii and every operator's default budget on one chart, with further budgets for the training-free operators (App. G.3, Table 13), and a peak-memory column in Table 2. (5J4P, YpRR)
  5. Implementation: the Triton kernels, what is stored, the head-parallel layout on four GPUs and a stage-by-stage time breakdown (App. G.4, Table 14). (all three)
  6. Is the stop learned or trivial? A Constant-stop control on Wan (Table 3), the stop mass at annotated occlusions, and what stopping costs after a reappearance (App. G.5, Table 15). (VTjw, YpRR)
  7. Positioning and claims. FLATTEN, LinGen and Video DeltaNet are discussed in Sec. 2. Contribution (ii) now says that chained affinities, motion-path attention and decaying scans exist in prior work and that our contribution is their combination into one trainable operator. Sec. 3.3 no longer infers "generation follows the routes" from a two-budget test. "Matched quality" is now close aggregate scores with two reported task-specific regressions, and Sec. 7 has a Limitations paragraph. (YpRR, 5J4P)
  8. Figure 2: the VSA and VMoBA labels were swapped in the submission and are now correct; the caption says which end of each segment is PathSpan, and the memory behind the bubbles is listed in Table 2. (YpRR)
  9. VideoPhy-2 cells of Table 1: the PathSpan SA and PC cells, and the joint cells for Wan and Cosmos, had been copied from the Dense-FT row; they now show the PathSpan evaluations, and every difference lies inside root variation (Table 10). (VTjw)

Every added experiment checks a statement that was already in the submission; none of them introduces a new claim. The second part summarizes the results.

AuthorsOpenReview ↗

General response (2/2): results of the added experiments

General response (2/2): results of the added experiments

QuestionResultWhere
Which Table 1 differences exceed root variation?Only the two regressions already reported (LTX DAB, Cosmos MAC) have intervals below zero; the coherence gains of VMBench, a perception-aligned motion benchmark, are above zero on all three backbones. The MAC loss changes how much a few videos move (amplitude class), while action score and warping error stay level.Table 10
Are the margins over the seven operators reliable?The MB and TCS margins are above zero in 15 of 18 comparisons; the three exceptions are the three smallest margins (MB vs SLA, TCS vs VMoBA and vs Surgery).Table 11
Does exact spatial attention explain the gain?Factorized: MB 40.2, TCS 92.46 at 106.9 s. STA: MB 42.4, TCS 93.71 at 114.9 s. PathSpan: 43.9, 94.34 at 112.4 s.Table 2
Does it hold on longer clips?At 2x length PathSpan is 2.70x / 2.05x / 7.10x faster (Wan / LTX / Cosmos). On Wan its coherence drops by 0.45 at 2x and 1.37 at 4x (481 frames, 4.44x faster), against 1.37 and 4.21 for Dense-FT.Table 12
Is it on the quality-latency frontier?No operator point reaches PathSpan's motion binding (43.9) in less than its 114.4 s of complete CompBench time (Table 2's 112.4 s is VBench time); SLA at its default budget comes closest (44.5 at 118.5 s). Peak memory is 42.4-45.8 GiB for every point.Table 13
Do the sequential scans cost much?They take 1.4-4.3% of complete time; all PathSpan attention stages together are 8.3-12.3x cheaper than dense attention.Table 14
Is the stop a fixed rate?On Wan, Constant stop keeps the mean stop mass yet lowers TCS by 1.37 and MB by 1.3; the learned stop separates annotated occlusion onsets with AUROC 0.69-0.76.Tables 3, 15
What does stopping cost?After a reappearance, chaining gains -0.67 to +0.18 points over direct matching, against +1.89 to +3.56 on continuously visible targets. Now a stated limitation.Table 15, Sec. 7

Taken together, the new evidence supports the paper as submitted rather than a stronger version of it: PathSpan's speed comes with close quality, including the perception-aligned VMBench scores, with two small, now quantified regressions; following the route, not exact spatial attention, is what separates it from factorized and local attention; and its stop is content-dependent and has a measured cost.

We are grateful to all three reviewers. The individual replies below give the details for each point, and we are glad to answer anything further before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.