acceptodds
Under review as a conference paper at ICLR 2027

GAD: Geometry Attention Distillation for Consistent Video Generation

Abstract

Video diffusion models generate visually compelling short-term sequences but often struggle to preserve scene geometry and object structure over time. In this paper, we first demonstrate that current video attention cannot identify reliable correspondences across frames, with responses dispersed across multiple regions even after recognizable scene structure has emerged. We argue that reliable cross-frame associations are essential for information exchange as scene structure forms in video generation. Thus, we introduce Geometry Attention Distillation (GAD), which transfers correspondences from a frozen geometry model's attention to video attention. Extracted from clean videos, these correspondences link the same static surface or moving object across frames. Since structure formation begins in deeper layers at higher noise and shallower layers at lower noise, we apply attention distillation at a student layer selected according to the noise timestep. After training, video attention focuses more consistently on corresponding locations. Experiments across two video backbones demonstrate improved long-range scene consistency and dynamic temporal consistency, with geometric gains across generation lengths. On Wan2.2 5B, GAD improves cross-frame matching accuracy by at least 30% over the Wan baseline on both long-range benchmarks. Notably, GAD achieves these gains without additional inference modules, offering a promising approach to efficient and consistent video-based world simulation.

Primary area:
generative models
Keywords:
Video Generation, 4D Consistency, Attention Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3P6aOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GAD, a method that can improve the geometry consistency in the generated video for diffusion models. The authors identfy that the current model cannot identify reliable geometry correspondences in attention block. Therefore, they propose to distills the correspondences from a pretrained geometry model (VGGT-omega) into the attention of a video diffusion model. To make it work, this paper introduces a few techniques (e.g. confidence-weighted supervision and noise-dependent layer selection) for better distillation. Experiments are conducted to domenstrate the design choice.

Critical strengths

The paper is well-written and easy to follow. The method seems effective, and the visual results show improvements in geometry consistency. I appreciate the careful design, and the ablations provide support for several design choices.

Critical weaknesses

I still have several concerns:

  1. The method is a bit heuristic to me, as it explicitly injects correspondences from a geometry model to constrain the video model’s attention, and the way they are injected (e.g., layer selection based on visualization) is also pretty heuristic. I am wondering whether this affects the general generation ability of the original video model, as the injection may destroy attention patterns naturally learned from data. These cases are not sufficiently evaluated.
  2. There are other ways to introduce geometric information, such as RL with geometry-based rewards. The paper discusses VGGRPO and World-R1 but does not compare against them. They can be viewed as an implicit way of injecting geometry information from a geometry model. I would like to see a comparison under matched settings.
  3. The improvements are only shown on relatively small models, i.e., Wan2.1-1.3B and Wan2.2-5B. In my experience, larger models often already have much better geometric consistency. Geometry modeling ability can also be largely limited by model size. Would this method still provide clear gains on a stronger 14B model?
Questions for authors

How is the camera trajectory encoded and injected into Wan, and which initial checkpoints are used?

4 comments
AuthorsOpenReview ↗

Response to The Reviewer (1 / 3)

We sincerely thank the reviewer for the constructive comments on our work. We greatly appreciate your description of the paper as "well-written and easy to follow", your recognition that "the visual results show improvements in geometry consistency", and your appreciation of the "careful design" and the evidence that "the ablations provide support for several design choices". To address your concerns, we have added 1) quantitative structure-formation analysis, 2) additional noise-timestep routing comparisons, and 3) experiments on the stronger Wan2.1 14B backbone. These results further support our design rationale and its effectiveness at larger model capacity. We address each concern below.

1. Rationale for correspondence supervision and noise-timestep routing.

The method is a bit heuristic to me, as it explicitly injects correspondences from a geometry model to constrain the video model’s attention, and the way they are injected (e.g., layer selection based on visualization) is also pretty heuristic.

Correspondence supervision helps video attention gather information from the correct associated regions for consistent generation. The flow-matching objective supervises denoising but does not explicitly specify which locations should correspond across frames. Our method therefore uses VGGT-Ω\Omega, whose attention provides more reliable cross-frame correspondences, as a teacher to guide the video model’s Q/K relations while retaining the original flow-matching objective. We have added a quantitative evaluation on TAPVid-DAVIS to substantiate the quality of this correspondence supervision. On ground-truth correspondences from all 30 TAPVid-DAVIS videos, our teacher targets achieve 90.88% target coverage, compared with 54.48% for video attention, supporting the teacher as a substantially stronger source of correspondence supervision.

For noise-timestep routing, we have added a quantitative analysis across 80 randomly selected RealEstate10K scenes. The scene-averaged curves show that deeper layers recover spatial structure earlier under high noise and gradually plateau, while middle and shallow layers show greater improvements in structure recovery as noise decreases. The estimated structure formation rates peak at t=988t=988, 884884, and 680680 for the late, middle, and early layer groups, respectively, aligning with our routing intervals of t≥950t\geq950, 850≤t<950850\leq t<950, and t<850t<850. These findings support noise-timestep routing from deeper to shallower layers during denoising. Further details are provided in Appendix E.3, with the quantitative curves shown in Fig. 5.

We have added two controls to further test the effectiveness of our noise-timestep routing under the same experimental settings. Random routing (R9) selects one of L5, L15, and L25 independently of noise level. Inverse routing (R10) uses the same noise intervals but reverses our shallow- and deep-layer assignments. The table below reports all ten configurations on DL3DV-10K with Wan2.2 5B.

Exp. IDAuxiliary supervision strategyR_dist ↓T_dist ↓M_rate ↑M_dist ↓
R1 (Wan)No auxiliary supervision9.5730.1080.25748.90
R2L5 at all timesteps7.4400.0860.29543.89
R3L15 at all timesteps7.3460.0900.28642.71
R4L25 at all timesteps6.5400.0750.28943.55
R5L5 only for t≥900t\geq9008.9320.0980.26246.36
R6L25 only for t<900t<9009.2900.1060.25847.17
R7L25 for t<900t<900; L5 for t≥900t\geq9008.7550.1040.26845.77
R8 (Ours)L5 for t<850t<850; L15 for 850≤t<950850\leq t<950; L25 for t≥950t\geq9504.8900.0650.33538.85
R9Random selection of L5/L15/L25, independent of noise6.8960.0820.30643.18
R10L25 for t<850t<850; L15 for 850≤t<950850\leq t<950; L5 for t≥950t\geq950 (inverse routing)9.0230.0960.26545.64

Our noise-timestep routing outperforms all these alternatives on all four geometric metrics. Its matching hit rate reaches 0.335, compared with 0.286–0.295 for fixed-layer supervision, 0.306 for random routing, and 0.265 for inverse routing. The gains over R9 support conditioning layer selection on noise timestep, while those over R10 support routing from deeper to shallower layers as noise decreases. Together, these experiments and the structural analysis substantiate the design and effectiveness of our noise-timestep routing.

AuthorsOpenReview ↗

Response to The Reviewer (2 / 3)

2. Whether this affects the general generation ability of the original video model?

I am wondering whether this affects the general generation ability of the original video model, as the injection may destroy attention patterns naturally learned from data. These cases are not sufficiently evaluated.

No, our method improves consistency without compromising the original video model’s generation ability. As explained in our response to the previous question, cross-frame attention supports consistent generation by gathering information from correctly associated regions across frames. Our method strengthens this mechanism by using a geometry teacher with more reliable correspondences to guide the video model’s attention. Unlike feature-level alignment approaches such as GeometryForcing and Gen3R, GAD supervises cross-frame relations without requiring the video model to reproduce the teacher’s feature representations. This preserves flexibility in learning representations suited to video generation.

Quantitative and qualitative results under matched training settings support this design. Tables 1–2 show improved FID, FVD, and CLIP over the matched Wan baseline across both backbones and all three datasets. For example, on OmniWorld-Game with Wan2.2 5B, FID improves from 61.29 to 59.52, FVD from 657.6 to 622.3, and CLIP from 27.46 to 28.23; motion smoothness, aesthetic quality, and image quality also improve. The supplementary Wan/Ours comparisons further demonstrate preserved appearance and dynamics in nonrigid flames (Dynamic Scene 1), running with displaced snow (Dynamic Scene 3), running across grass (Dynamic Scene 7), and foliage and grass under viewpoint changes (Static Scenes 1 and 3). Together, these results show that our method preserves diverse appearance and dynamics.

3. Comparison with RL-based methods VGGRPO and World-R1.

There are other ways to introduce geometric information, such as RL with geometry-based rewards. The paper discusses VGGRPO and World-R1 but does not compare against them. They can be viewed as an implicit way of injecting geometry information from a geometry model. I would like to see a comparison under matched settings.

We agree that VGGRPO and World-R1 are valuable baselines. Regarding availability, we could not find an official implementation or pretrained checkpoints linked from the VGGRPO [1]. World-R1 [2] has released training and inference code, but we could not find released model checkpoints or LoRA weights in its official repository. World-R1 reports using 48 and 96 H200 GPUs to train its 1.3B and 14B variants, respectively. These RL-based methods require generating video samples before evaluating their geometry-based rewards, incurring substantial computation and training time. Given these availability and computational constraints, we were unable to reproduce these methods for direct comparison within the rebuttal period.

While these availability constraints limit our ability to conduct direct comparisons, our method offers two key methodological advantages over these RL-based approaches: more direct supervision of cross-frame associations and a simpler training pipeline without online reward evaluation. First, our method provides explicit guidance on where to match across frames. VGGRPO and World-R1 optimize generated videos through aggregated rewards, without explicitly specifying which locations should correspond. Our method instead uses location-specific correspondence targets to supervise the video attention governing cross-frame associations, directly targeting incorrect matches that undermine consistency. Second, our method avoids computationally expensive and time-consuming online geometry-based reward evaluation. These RL-based methods must first generate video samples before evaluating their geometric rewards, adding substantial computation and slowing training. Our teacher correspondences are extracted offline and cached for reuse during flow-matching fine-tuning, avoiding repeated online video sampling and reward computation and simplifying the training pipeline.

Our correspondence supervision is also complementary to RL-based optimization and could be combined with geometry-based rewards to further improve consistency.

[1] VGGRPO: Towards World-Consistent Video Generation with 4D Latent Reward. https://arxiv.org/abs/2603.26599

[2] World-R1: Reinforcing 3D Constraints for Text-to-Video Generation. https://arxiv.org/abs/2604.24764

AuthorsOpenReview ↗

Response to The Reviewer (3 / 3)

4. Would this method still provide clear gains on a stronger 14B model?

The improvements are only shown on relatively small models, i.e., Wan2.1-1.3B and Wan2.2-5B. In my experience, larger models often already have much better geometric consistency. Geometry modeling ability can also be largely limited by model size. Would this method still provide clear gains on a stronger 14B model?

Yes, our additional experiments show that GAD provides consistent gains on the stronger Wan2.1 14B model. Following exactly the same experimental setup as in the original paper, GAD and the Wan baseline use identical initialization, training data, training budget, and evaluation settings, with the baseline trained using flow matching alone.

For long-range generation, GAD improves all seven metrics on both RealEstate10K and DL3DV-10K. Rotation error decreases by 41.7% / 44.7%, and matching hit rate increases by 15.3% / 7.7% relative, respectively, alongside improvements in FID, FVD, and CLIP.

Static datasetMethodFID ↓FVD ↓CLIP ↑R_dist ↓T_dist ↓M_rate ↑M_dist ↓
RealEstate10KWan2.1 14B24.80201.832.004.7240.0780.41330.09
RealEstate10K+ GAD23.15194.832.142.7540.0580.47626.01
DL3DV-10KWan2.1 14B31.02255.834.055.4970.0830.36539.68
DL3DV-10K+ GAD28.66228.834.343.0390.0460.39334.10

These benefits also extend to dynamic generation. On OmniWorld-Game, GAD improves subject and background consistency, motion smoothness, and all reported visual-quality metrics:

MethodFID ↓FVD ↓CLIP ↑Subject consistency ↑Background consistency ↑Motion smoothness ↑Aesthetic quality ↑Image quality ↑
Wan2.1 14B45.94419.728.180.8500.9090.9710.5460.612
+ GAD44.75410.228.880.8630.9170.9780.5520.617

These results show that correspondence supervision remains beneficial even with increased model capacity, improving both consistency and generation quality.

5. How is the camera trajectory encoded and injected into Wan, and which initial checkpoints are used?

How is the camera trajectory encoded and injected into Wan, and which initial checkpoints are used?

We use the camera-conditioned checkpoints released by VideoX-Fun and retain their camera-conditioning mechanism. The exact initializations are:

BackboneInitial checkpoint
Wan2.1 1.3Balibaba-pai/Wan2.1-Fun-V1.1-1.3B-Control-Camera https://huggingface.co/alibaba-pai/Wan2.1-Fun-V1.1-1.3B-Control-Camera
Wan2.2 5Balibaba-pai/Wan2.2-Fun-5B-Control-Camera https://huggingface.co/alibaba-pai/Wan2.2-Fun-5B-Control-Camera
Wan2.1 14Balibaba-pai/Wan2.1-Fun-V1.1-14B-Control-Camera https://huggingface.co/alibaba-pai/Wan2.1-Fun-V1.1-14B-Control-Camera

Specifically, we express camera poses relative to the first-frame camera and combine them with camera intrinsics to construct per-pixel six-dimensional Plücker embeddings, pf(u,v)=[of×df(u,v), df(u,v)]p_f(u,v)=[o_f\times d_f(u,v),\,d_f(u,v)], where ofo_f and df(u,v)d_f(u,v) denote the camera center and unit ray direction in this reference coordinate system. These embeddings are temporally rearranged and passed through the camera adapter, whose output is added elementwise to the video patch embeddings before the Transformer blocks.

Thank you very much for your thoughtful comments and the time you have invested in reviewing our paper. We have provided a detailed response to the concerns you raised earlier. Could you please let us know if our clarification sufficiently addressed your questions? If anything remains unclear, we would be more than happy to continue the discussion.

Reviewer_Swi1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GAD uses a geometry model with fixed parameters to provide matching locations across frames, then uses these matches to train a video generator. It chooses which layer to supervise based on the noise level. Results on two Wan models show better video consistency without extra inference cost.

Critical strengths
  1. Both Wan models improve under comparisons using the same training data and settings. For the 5B model, matching hit rate increases by approximately 33% and 30% on the two static benchmarks (Table 1).
  2. The method adds no inference cost. Human visibility annotations support the design of the supervision weights, and comparisons show that these weights improve the results (Tables 2 and 5; Appendix B.1).
Critical weaknesses
  1. The reliability of the teacher's labels remains a weak point. Using attention instead of static geometric projection helps handle moving objects, but the strongest response can still point to the wrong part. Visibility weighting assesses whether a point can be seen, not whether its location is correct. How the method avoids reinforcing these incorrect labels remains unclear (Eqs. 1 and 9; Appendix B.1).
  2. The motivation for selecting layers by noise is not yet convincing. The aim is to supervise where structure starts to form, but the method selects layers using fixed noise intervals. The same noise level may correspond to different structural states across scenes or training stages. The results show that the rule is useful, but do not establish that it captures the structure-formation process emphasized by the paper (Eq. 2; Appendix F).
  3. Correct auxiliary matches may not be fully used by the actual attention. Training uses an extra MLP to compare locations within the target frame, while generation uses the original multi-head attention across all frames. Supervision affects shared Q/K representations, but a location ranked highly in the auxiliary distribution may still receive little weight in the actual attention. This gap directly affects the central explanation of improved information exchange across frames (Eq. 10; Appendix E.1).
Questions for authors
  1. If the teacher can see a point but matches it to the wrong location, how does the method avoid reinforcing that error?
  2. Why is noise level alone enough to decide which layer to supervise? What directly connects this choice to where structure is forming?
  3. What part of the design turns correct auxiliary matches into effective cross-frame attention during generation?
6 comments
AuthorsOpenReview ↗

Response to The Reviewer (1 / 3)

We sincerely thank the reviewer for the constructive comments on our work. We greatly appreciate your recognition that "Both Wan models improve under comparisons using the same training data and settings" and that "The method adds no inference cost", together with your recognition of the evidence supporting our supervision weights. Regarding your concerns, the original manuscript already provides relevant explanations. To further support them, we have added 1) quantitative teacher–student correspondence comparisons, 2) quantitative structure-formation analysis, and 3) additional noise-timestep routing comparisons. We clarify each point and present the supporting results below.

1. Reliability of the teacher's labels.

Using attention instead of static geometric projection helps handle moving objects, but the strongest response can still point to the wrong part. Visibility weighting assesses whether a point can be seen, not whether its location is correct. How the method avoids reinforcing these incorrect labels remains unclear (Eqs. 1 and 9; Appendix B.1). If the teacher can see a point but matches it to the wrong location, how does the method avoid reinforcing that error?

First, we would like to clarify that our method transfers correspondence knowledge from stronger geometry-model attention to weaker video-model attention without requiring error-free teacher labels. Although teacher matches may be imperfect, they can still guide the student toward better matching beyond what it learns from the generation objective alone. This acts as a general and scalable way to improve video generation consistentyly.

Furthermore, attention-mass weighting and Gaussian smoothing, introduced in Sec. 4.1, both mitigate matching errors even when the queried point is visible. The former reduces the contribution of unreliable correspondences. As illustrated by the dynamic car example in Appendix Fig. 7, the queried wheel remains visible across frames, yet an incorrect match receives substantially lower target-frame attention mass than an accurate match (1.41×10−41.41\times10^{-4} vs. 3.09×10−23.09\times10^{-2}) and consequently receives less supervision weight. The latter distributes supervision over a 3×33\times3 neighborhood around the predicted token, allowing the correct location to receive positive supervision despite small localization errors.

To further substantiate both the teacher’s stronger matching performance and the benefits of these mechanisms, we evaluate the selected teacher on all 30 videos of TAPVid-DAVIS, using 25,507 ground-truth-visible correspondences from 538 tracks, and report teacher and student results under the same evaluation settings:

Metric (%) ↑Teacher (VGGT-Ω\Omega)Teacher (VGGT-Ω\Omega), attention-mass weightedStudent (Wan2.2 5B)
Exact-token matching68.1974.1137.27
Local target coverage (3×33\times3)85.5590.8854.48

For both models, local target coverage measures whether the ground-truth token lies within the 3×33\times3 neighborhood centered on the predicted token. For the teacher, this neighborhood also defines the Gaussian target’s support. Weighted scores use the training attention-mass weights, without refitting.

The teacher achieves 74.11% exact-token matching and 90.88% local coverage, compared with 37.27% and 54.48%, respectively, for the video model, supporting its role as a stronger source of correspondence supervision. The Gaussian neighborhood covers the correct token in 85.55% of cases, compared with 68.19% for the peak token alone, demonstrating its tolerance to small localization errors. Attention-mass weighting yields 74.11% exact-token matching and 90.88% local coverage. Since every evaluated correspondence is ground-truth visible, these weighting gains demonstrate that attention mass helps prioritize accurate matches even among visible points. Together, these results support the effectiveness of attention-mass weighting and Gaussian smoothing in reducing the impact of teacher-label errors, even when the queried point is visible.

AuthorsOpenReview ↗

Response to The Reviewer (2 / 3)

2. Rationale for noise-dependent layer selection.

The aim is to supervise where structure starts to form, but the method selects layers using fixed noise intervals. The same noise level may correspond to different structural states across scenes or training stages. The results show that the rule is useful, but do not establish that it captures the structure-formation process emphasized by the paper (Eq. 2; Appendix F). Why is noise level alone enough to decide which layer to supervise? What directly connects this choice to where structure is forming?

We have added a quantitative analysis across 80 randomly selected RealEstate10K scenes to assess the relationship between noise level and structure formation. The scene-averaged curves show that deeper layers recover spatial structure earlier under high noise and gradually plateau, while middle and shallow layers show greater improvements in structure recovery as noise decreases. The estimated structure formation rates peak at t=988t=988, 884884, and 680680 for the late, middle, and early layer groups, respectively, aligning with our routing intervals of t≥950t\geq950, 850≤t<950850\leq t<950, and t<850t<850. These findings support directing correspondence supervision to progressively shallower layers as their structure actively develops. Further details are provided in Appendix E.3, with the quantitative curves shown in Fig. 5.

Regarding training stages, we have already discussed adapting supervision placement to training progress in Appendix F "Limitations and Future Work". Our method fine-tunes an extensively pretrained Wan model whose structure-forming behavior is already established. Given the comparatively limited fine-tuning, we expect the overall relationship between noise level and structure formation across layers to remain largely unchanged. Moreover, designing a routing strategy that also adapts to changes during fine-tuning requires further investigation. For simplicity, we use noise level to select the supervised layer and keep this mapping fixed throughout training, while identifying joint conditioning on noise timestep and training progress as a direction for future work.

We have added two controls to further test the effectiveness of our noise-timestep routing under the same experimental settings. Random routing (R9) selects one of L5, L15, and L25 for attention supervision independently of noise level. Inverse routing (R10) uses the same noise intervals as our method but swaps the shallow- and deep-layer assignments, supervising L5 at high noise, L15 at medium noise, and L25 at low noise. The table below reports all ten configurations on DL3DV-10K with Wan2.2 5B.

Exp. IDAuxiliary supervision strategyR_dist ↓T_dist ↓M_rate ↑M_dist ↓
R1 (Wan)No auxiliary supervision9.5730.1080.25748.90
R2L5 at all timesteps7.4400.0860.29543.89
R3L15 at all timesteps7.3460.0900.28642.71
R4L25 at all timesteps6.5400.0750.28943.55
R5L5 only for t≥900t\geq9008.9320.0980.26246.36
R6L25 only for t<900t<9009.2900.1060.25847.17
R7L25 for t<900t<900; L5 for t≥900t\geq9008.7550.1040.26845.77
R8 (Ours)L5 for t<850t<850; L15 for 850≤t<950850\leq t<950; L25 for t≥950t\geq9504.8900.0650.33538.85
R9Random selection of L5/L15/L25, independent of noise6.8960.0820.30643.18
R10L25 for t<850t<850; L15 for 850≤t<950850\leq t<950; L5 for t≥950t\geq950 (inverse routing)9.0230.0960.26545.64

Our noise-timestep routing outperforms all these alternatives on all four geometric metrics. Its matching hit rate reaches 0.335, compared with 0.286–0.295 for fixed-layer supervision, 0.306 for random routing, and 0.265 for inverse routing. The gains over R9 support conditioning layer selection on noise timestep, while those over R10 support routing from deeper to shallower layers as noise decreases. Together with the structural analysis, these results validate the effectiveness of our noise-timestep routing during fine-tuning.

AuthorsOpenReview ↗

Response to The Reviewer (3 / 3)

3. Effect of auxiliary correspondence supervision on generation attention.

Training uses an extra MLP to compare locations within the target frame, while generation uses the original multi-head attention across all frames. Supervision affects shared Q/K representations, but a location ranked highly in the auxiliary distribution may still receive little weight in the actual attention. This gap directly affects the central explanation of improved information exchange across frames (Eq. 10; Appendix E.1). What part of the design turns correct auxiliary matches into effective cross-frame attention during generation?

In fact, the attention distillation loss directly optimizes cross-frame attention in the video model. In Eq. (10), the MLP receives the original per-head query–key logits rhr_{h}, predicts head weights α=softmaxh(gϕ(r))\alpha = \mathrm{softmax}_{h}(g_{\phi}(r)), and forms s=1τ∑hαhrhs = \frac{1}{\tau}\sum_{h}\alpha_{h}r_{h}. A softmax over target-frame candidates then produces the matching distribution. Optimizing this distribution updates the shared Q/K representations, guiding the attention used during generation toward more consistent cross-frame associations.

The MLP serves an essential role in combining correspondence evidence across attention heads while preserving their different roles. Directly supervising a single head or imposing the same target on all heads could disrupt the existing specialization of attention heads. The MLP instead learns to combine matching evidence across heads, allowing them to contribute differently to the correspondence objective. At inference, the MLP is removed, while the updated Q/K parameters remain in the original attention mechanism.

Fig. 6 further supports that this supervision improves the attention actually used during generation. It visualizes the model’s original attention responses, rather than the auxiliary matching distribution, and shows better localization around corresponding static and moving regions alongside improved generated structure.

Reviewer_ReyMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

GAD improves geometric and temporal consistency in video generation by distilling correspondences from a geometry foundation model into the attention of a video diffusion model. It extracts correspondences and confidence scores from a frozen VGGT-Ω model’s attention and uses them for pairwise and trackwise supervision. The supervision is applied to deeper layers at higher noise levels and shallower layers at lower noise levels. Since the additional supervision is used only during training, the inference architecture and computational cost remain unchanged.

Critical strengths

1. Clear motivation and a direct solution. The method addresses incorrect cross-frame associations by directly supervising attention with correspondences. Transferring the relevant relations without requiring full feature alignment is a reasonable design choice.

2. Consistent improvements over a matched baseline. GAD consistently improves performance over Wan baselines fine-tuned using the same data and training settings.

Critical weaknesses

1. Limited novelty.

  • The proposed approach does not appear to differ substantially from existing attention distillation methods.

2. Limited analysis of the routing mechanism.

  • Although noise-dependent routing is effective, the rationale for supervising layers where structure begins to form relies primarily on PCA and attention visualizations from a single example. The proposed routing outperforms fixed-layer supervision, but a baseline that randomly selects among the same three layers independently of noise is missing. It therefore remains unclear whether the gains arise from aligning supervision depth with noise level or from supervising multiple layers during training.

3. Lack of controlled comparisons with baseline methods.

  • A controlled comparison with representation distillation methods would help establish the effectiveness of the proposed attention distillation approach. The reported comparisons use different backbones and training setups, so they do not provide a fair comparison of the alignment objectives themselves.

Minor Comments

  • The table captions would benefit from additional information about the experimental settings.
Questions for authors

The questions are included in the weaknesses above.

6 comments
AuthorsOpenReview ↗

Response to The Reviewer (1 / 3)

We sincerely thank the reviewer for the constructive comments on our work. We greatly appreciate your recognition of our work's "Clear motivation and a direct solution" and "Consistent improvements over a matched baseline", as well as your assessment that transferring relevant relations without full feature alignment is "a reasonable design choice". To address your concerns, we have added 1) quantitative structure-formation analysis, 2) further noise-timestep routing comparisons, and 3) a GeometryForcing-style feature-alignment baseline under matched training settings. These experiments provide further support for our noise-timestep routing and attention-distillation design. We address each concern below.

1. Novelty and improvements from prior work.

The proposed approach does not appear to differ substantially from existing attention distillation methods.

Our method advances existing attention distillation approaches in three aspects: the integration of attention guidance, the supervision signal, and the supervision-layer selection strategy. As discussed in Sec. 2.3, Attention Distillation [1] and X2I [2] address appearance transfer and multimodal image generation, while REPA [3], VideoREPA [4], and SARA [5] align features or feature relations. Our method instead targets cross-frame consistency in video generation through geometry-derived attention correspondence supervision and noise-dependent layer selection.

CAMEO [6], the closest related method, focuses on cross-view consistency in static-scene novel-view synthesis, while our method addresses cross-frame consistency in video generation involving both camera and object motion. Compared with CAMEO, our method introduces two fundamental improvements:

  1. Supervision signal: pointmap-to-attention vs. attention-to-attention. CAMEO constructs correspondence labels through nearest-neighbor matching of 3D pointmaps and cycle-consistency filtering. This formulation matches observations of the same 3D position, making it dependent on reconstruction accuracy and unsuitable for directly associating object parts that move across frames. Our method instead transfers correspondence knowledge from a strong geometry teacher’s attention into video attention. Teacher attention associates the same scene element without requiring a fixed world-space position, naturally accommodating both camera and object motion while bypassing explicit pointmap matching and cycle-consistency checks. Our proposed teacher-confidence weighting further downweights unreliable matches using target-frame attention mass, while local Gaussian targets tolerate small localization errors. This reliability mechanism is essential: removing confidence weighting reduces matching hit rate from 0.335 to 0.230, below the Wan baseline’s 0.257 (Table 5).

  2. Supervision layer selection: fixed-layer alignment vs. noise-dependent routing. CAMEO supervises a selected, fixed attention layer. Our routing mechanism follows an analysis of how video features and attention jointly evolve across noise levels and DiT depth. Fig. 2 shows that recognizable structure emerges in deeper layers at higher noise and shallower layers at lower noise, while cross-frame associations remain ambiguous. We therefore route correspondence supervision to these emerging-structure layers, where structural information begins to support matching and guidance can shape subsequent cross-frame interactions. The ablations support this design: our routing achieves a matching hit rate of 0.335, compared with 0.286–0.295 for fixed-layer supervision (Table 4), 0.306 for random layer selection, and 0.265 for inverse routing, with better performance on all four geometric metrics.

[1] Attention Distillation: A Unified Approach to Visual Characteristics Transfer. https://arxiv.org/abs/2502.20235

[2] X2I: Seamless Integration of Multimodal Understanding into Diffusion Transformer via Attention Distillation. https://arxiv.org/abs/2503.06134

[3] Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think. https://arxiv.org/abs/2410.06940

[4] VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models. https://arxiv.org/abs/2505.23656

[5] SARA: Semantically Adaptive Relational Alignment for Video Diffusion Models. https://arxiv.org/abs/2605.07800

[6] CAMEO: Correspondence-Attention Alignment for Multi-View Diffusion Models. https://arxiv.org/abs/2512.03045

AuthorsOpenReview ↗

Response to The Reviewer (2 / 3)

2. Further analysis of the routing mechanism.

Although noise-dependent routing is effective, the rationale for supervising layers where structure begins to form relies primarily on PCA and attention visualizations from a single example. The proposed routing outperforms fixed-layer supervision, but a baseline that randomly selects among the same three layers independently of noise is missing. It therefore remains unclear whether the gains arise from aligning supervision depth with noise level or from supervising multiple layers during training.

Fig. 2 illustrates a pattern observed across multiple scenes. To further substantiate this observation quantitatively, we have added a quantitative analysis across 80 randomly selected RealEstate10K scenes to substantiate the routing rationale beyond qualitative visualizations (Appendix E.3, Fig. 5). The curves show that deeper layers recover spatial structure earlier under high noise and gradually plateau as denoising proceeds, while middle and shallow layers show greater improvements in structure recovery as noise decreases. This progression motivates shifting correspondence supervision toward shallower layers where structure is still actively developing. Specifically, the estimated structure formation rates peak at t=988t=988, 884884, and 680680 for the late, middle, and early layer groups, respectively, aligning with our routing intervals of t≥950t\geq950, 850≤t<950850\leq t<950, and t<850t<850. These results quantitatively support matching supervision depth to the stages of rapid structure formation. Further details are provided in Appendix E.3, with the quantitative curves shown in Fig. 5.

We have added additional controls to test whether the gains arise from matching supervision depth to noise level. Under the same experimental settings, random routing (R9) selects one of L5, L15, and L25 for attention supervision independently of noise level, while inverse routing (R10) reverses our layer assignments over the same noise intervals. The table below summarizes the comparison on DL3DV-10K with Wan2.2 5B; full results appear in Tables 4 and 6.

Exp. IDAuxiliary supervision strategyR_dist ↓T_dist ↓M_rate ↑M_dist ↓
R1 (Wan)No auxiliary supervision9.5730.1080.25748.90
R2L5 at all timesteps7.4400.0860.29543.89
R3L15 at all timesteps7.3460.0900.28642.71
R4L25 at all timesteps6.5400.0750.28943.55
R5L5 only for t≥900t\geq9008.9320.0980.26246.36
R6L25 only for t<900t<9009.2900.1060.25847.17
R7L25 for t<900t<900; L5 for t≥900t\geq9008.7550.1040.26845.77
R8 (Ours)L5 for t<850t<850; L15 for 850≤t<950850\leq t<950; L25 for t≥950t\geq9504.8900.0650.33538.85
R9Random selection of L5/L15/L25, independent of noise6.8960.0820.30643.18
R10L25 for t<850t<850; L15 for 850≤t<950850\leq t<950; L5 for t≥950t\geq950 (inverse routing)9.0230.0960.26545.64

Our routing outperforms all these alternatives on all four geometric metrics. It achieves a matching hit rate of 0.335, compared with 0.286–0.295 for fixed-layer supervision, 0.306 for random routing, and 0.265 for inverse routing. These comparisons show that noise-timestep routing improves cross-frame consistency beyond supervising the same layers independently of noise level. Together with the structural analysis, they support our routing strategy of directing correspondence supervision to layers undergoing rapid structure formation at each noise stage.

AuthorsOpenReview ↗

Response to The Reviewer (3 / 3)

3. Controlled comparisons with baseline methods.

A controlled comparison with representation distillation methods would help establish the effectiveness of the proposed attention distillation approach. The reported comparisons use different backbones and training setups, so they do not provide a fair comparison of the alignment objectives themselves.

To address this concern, we added a GeometryForcing-style feature-alignment baseline on the same Wan2.2 5B video backbone. We implement GeometryForcing’s angular feature-alignment and reconstruction objectives using its official loss weights and configurations, and train this baseline on the same data for the same number of epochs as GAD. The table below reports the results on DL3DV-10K benchmark.

MethodFID ↓FVD ↓CLIP ↑R_dist ↓T_dist ↓M_rate ↑M_dist ↓
Wan2.2 5B38.60322.133.859.5730.1080.25748.90
GeometryForcing-style feature alignment (Wan2.2 5B)38.39351.633.4410.2470.1090.22556.01
GAD (Wan2.2 5B)37.93318.034.244.8900.0650.33538.85

GAD outperforms the GeometryForcing-style feature-alignment baseline on all seven metrics, improving both geometric consistency and generation quality. The qualitative comparison in Fig. 8 further illustrates these gains: compared with the GeometryForcing-style baseline (GF*), GAD better preserves scene layout and object structure across viewpoint changes during long-range video generation. With the video backbone held fixed, these results support the advantages of GAD’s attention distillation design over GeometryForcing’s feature-alignment approach.

The comparison with Gen3R provides additional evidence. Gen3R uses the same Wan2.1 1.3B backbone as our 1.3B model but is trained on more data. Even so, GAD outperforms Gen3R on all seven metrics across RealEstate10K and DL3DV-10K (Table 1), further demonstrating its advantages without relying on a larger backbone or more training data.

4. Clarifying experimental settings in table captions.

The table captions would benefit from additional information about the experimental settings.

Thank you for this suggestion. We have revised the table captions in the manuscript to make the experimental settings clearer and easier to identify.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers

We sincerely thank all reviewers for their time and constructive feedback. We appreciate the recognition of our “Clear motivation” (ReyM), “careful design” (3P6a), and transferring correspondences without full feature alignment as “a reasonable design choice” (ReyM). Reviewers also acknowledge “Consistent improvements over a matched baseline” (ReyM), that “Both Wan models improve” under “the same training data and settings” (Swi1), and visual “improvements in geometry consistency” (3P6a), while noting that “The method adds no inference cost” (Swi1).

We have carefully considered the reviewers’ comments and suggestions and revised our manuscript accordingly, with changes highlighted in blue. In particular, we have added 1) quantitative structure-formation analysis in Appendix E.3, with the corresponding curves shown in Fig. 5, providing direct evidence for the statistical patterns underlying our noise-timestep routing; 2) random and inverse routing comparisons in Tables 4 and 6 to further validate this design; and 3) experiments on the stronger Wan2.1 14B backbone in Tables 1 and 2 to demonstrate its effectiveness at larger model capacity. We hope these clarifications and additional results further strengthen your confidence in our contributions. If any questions remain, we would be more than happy to address them during the discussion stage.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.