acceptodds
Under review as a conference paper at ICLR 2027

Structure From Tracking: Distilling Structure-Preserving Motion for Video Generation

Abstract

Reality is a dance between rigid constraints and deformable structures. For video models, that means generating motion that preserves fidelity as well as . Despite progress in diffusion models, producing realistic structure-preserving motion remains challenging, especially for articulated and deformable objects such as humans and animals. Scaling training data alone, so far, has failed to resolve physically implausible transitions. Existing approaches rely on conditioning with noisy motion representations, such as optical flow or skeletons extracted using an external imperfect model. To address these challenges, we introduce an algorithm to distill structure-preserving motion priors from an autoregressive video tracking model (SAM2) into bidirectional video diffusion models. With our method, we train SAM2VideoX, which contains two innovations: (1) a module that extracts global structure-preserving motion priors from a recurrent model like SAM2; (2) a loss that aligns how local features move together. Experiments on VBench and in human studies show that SAM2VideoX delivers consistent gains (+2.60 on VBench, 21-22% lower FVD, and 71.4% human preference) over prior baselines. Specifically, on VBench, we achieve 95.51%, surpassing REPA (92.91%) by 2.60, and reduce FVD to 360.57, a 21.20% and 22.46% improvement over REPA- and LoRA-finetuning.

Primary area:
generative models
Keywords:
Video Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qbUPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies structure-preserving motion in image-to-video generation, in particular for articulated subjects such as humans and animals. It proposes SAM2VideoX, which distills representations from the SAM2 tracking into a video diffusion model during fine-tuning. The method combines forward and backward SAM2 information through bidirectional Local Gram Flow fusion and aligns local cross-frame feature relations using a KL-divergence objective. Experiments show improvements in temporal consistency and perceptual quality without requiring additional tracking or motion controls at inference.

Critical strengths
  1. The paper addresses a meaningful practical problem. Structural deformation, appearance drift, and inconsistent articulated motion remain important failure modes of video generation.
  2. The teacher choice and technical design sound reasonable. SAM2’s temporally informed tracking representations are a plausible source of supervision for preserving object identity and structure. Aligning local feature relations also avoids requiring coordinate-wise correspondence between heterogeneous teacher and student representations.
  3. The experiments include several useful controls. Comparisons with diffusion-only LoRA fine-tuning, mask supervision, point-trajectory supervision, and alternative teachers help assess different aspects of the approach.
Critical weaknesses
  1. The contribution over closely related methods is kind of incremental, and in some sense, insufficiently established. Track4Gen already demonstrates that tracking supervision can improve video generation, while VideoREPA, CREPA, and MoAlign investigate temporal or motion-oriented representation alignment. Although the paper cites CREPA and MoAlign, it provides neither quantitative comparisons nor a clear justification for excluding them. Its advantages over the closest alternatives need stronger evidence.
  2. The specific benefit of tracking memory is not directly isolated. Comparisons with other foundation-model teachers support the overall approach, but do not establish whether SAM2’s advantage specifically comes from its tracking memory. A control experiment using the same SAM2 image encoder without temporal memory, under the same LGF objective, would directly test the central hypothesis. Ideally, a more systematic comparison of teacher representations under a shared alignment objective would further separate teacher quality from loss design.
  3. The evaluation only indirectly measures the claimed structural improvements. The primary Motion Score combines background consistency, subject consistency, and motion smoothness. These metrics do not directly measure key factors in motion naturalness, such as limb connectivity, part topology, or anatomically plausible motion. Moreover, the gains over VideoREPA are modest: Motion Score increases from 94.98 to 95.51, while FVD decreases from 364.82 to 360.57.
Questions for authors
  1. Could you compare against CREPA and MoAlign under the same backbone, training data, and optimization budget, or explain the technical reasons for excluding them? Which specific capability does LGF provide beyond existing temporal relational alignment objectives?
  2. Could you evaluate SAM2 features with temporal memory disabled while retaining the same image encoder, projector, and LGF loss? Could alternative teachers also be evaluated with the same LGF objective?
  3. Could you provide evaluations targeting articulated structural failures, together with motion magnitude or dynamic degree? Are the gains over VideoREPA consistent across multiple training or generation seeds?
1 comment
AuthorsOpenReview ↗

Response to Reviewer qbUP

We thank the reviewer for the thoughtful review and the precise control experiments suggested.

W1 / Q1. Incremental over Track4Gen, VideoREPA, CREPA and MoAlign; missing comparisons; what does LGF provide beyond existing relational objectives?

CREPA and MoAlign. Neither method has released code, and we could not reimplement them faithfully within the discussion period, so a direct comparison is missing. We explain the technical differences and name the closest evaluated baselines:

  • CREPA aligns each frame's hidden states with DINOv2 features of that frame and neighbouring frames (REPA-style feature alignment). Our REPA baseline uses the same type of teacher and objective without the cross-frame term.
  • MoAlign trains a subspace of VideoMAE V2 features to predict optical flow, then aligns the DiT to it via soft relational alignment, extending VideoREPA's token relation distillation (TRD). We evaluate VideoREPA (VideoMAE V2 with TRD).

What LGF provides. These methods differ from ours along two axes:

  • Teacher. CREPA uses DINOv2 image features, VideoREPA uses VideoMAE V2, and MoAlign uses a flow-trained subspace of VideoMAE V2. None uses a tracking model. In our motion-CKA analysis, SAM2 features score 0.469, compared with 0.163 for V-JEPA 2 and −0.047 for VideoMAE V2 (the encoder used by VideoREPA and MoAlign).
  • Objective. VideoREPA and MoAlign match pairwise token-similarity values. LGF instead normalizes each token's similarities over a local cross-frame neighbourhood into a probability distribution and matches these distributions with KL divergence. Our bidirectional fusion further bridges SAM2's causal memory and the DiT's bidirectional attention. Table R1 varies the teacher with LGF-KL fixed, while Sec. 5.2 (Fig. 8) varies the objective with the SAM2 teacher fixed (G1).

Track4Gen. Track4Gen supervises the generator with explicit point-tracking targets. Our point-trajectory baseline follows Track4Gen on the same backbone (Sec. 5); SAM2VideoX improves on it in Motion Score (95.11 → 95.51 single-seed; 95.04 → 95.50 three-seed), Extended Motion Score (95.69 → 96.03 single-seed; 95.71 → 96.08 three-seed), and HumanScore Extra Limbs (+0.57 motion-matched).

W2 / Q2. The benefit of tracking memory is not isolated; other teachers under the same LGF objective.

Please see G1 (Tables R1 and R2). Under the same LGF-KL objective on CogVideoX, disabling SAM2's temporal memory drops 3-seed VBench Motion Score from 95.50 ± 0.26 to 94.99 ± 0.25 and Dynamic Degree from 52.25 ± 5.12 to 44.14 ± 5.12 (−8.11); in the separate HumanScore evaluation on 70 human prompts, its raw Extra Limbs score drops from 78.71 to 75.26 (−3.45). Replacing SAM2 with CoTracker3 features gives a similar VBench Motion Score (95.48 ± 0.10) and a lower Dynamic Degree (43.69 ± 2.06, −8.56); on HumanScore Extra Limbs, its raw score is 77.08 vs. 78.71 for SAM2VideoX. Table R2 shows what memory conditioning adds in frozen SAM2 features (+36% motion-direction probe, higher temporal-order gap, and higher DAVIS J&F).

W3 / Q3. Evaluation of articulated structural failures; motion magnitude; consistency across seeds.

  • Structural failures (Table R4). On the Extra Limbs metric of HumanScore (Xiang et al., NeurIPS 2026) evaluated on all 70 human videos of our 100-video test set (70 human, 30 animal), SAM2VideoX achieves 78.71, with positive motion-adjusted point estimates against all seven baselines and higher scores than every baseline in Table R4 under both raw and motion-matched comparisons, including mask supervision (+3.75 raw, +4.18 motion-matched; 95% CIs excluding zero), the base model (+4.70 raw, 95% CI excluding zero; +1.02 motion-matched), and VideoREPA (+0.47 raw, +0.99 motion-matched).
  • Motion magnitude (G2, Table R3). On single-seed Table 1, SAM2VideoX reaches DD 47.30 vs. 45.95 for the base model and VideoREPA, and 44.59 for mask supervision. Across three seeds (Table R3), SAM2VideoX achieves a higher mean DD (52.25) than mask supervision (49.55), VideoREPA (48.20), and V-JEPA 2 (43.69).
  • Three-seed VBench comparison (Table R3). Evaluated across three generation seeds on all six fine-tuning methods, SAM2VideoX has higher three-seed means than VideoREPA on Motion Score (95.50 ± 0.26 vs. 95.18 ± 0.16), Extended Motion Score (96.08 ± 0.21 vs. 95.79 ± 0.14), and Dynamic Degree (52.25 vs. 48.20), and also achieves higher three-seed mean Motion Score than V-JEPA 2 (95.38), point trajectory (95.04), and REPA (92.92).

Presentation. We revised the evaluation protocol and the Table 1 caption, in particular the definition of Motion Score, to make the experimental section easier to follow.

Reviewer_QZwROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SAM2VideoX fine-tunes video diffusion models using a frozen SAM2 tracking teacher. It extracts forward and backward memory features, combines their local cross-frame similarities, and matches the resulting distributions with a KL loss. The teacher and projector are removed at inference. Experiments use 9,837 human and animal motion clips, with results on CogVideoX and Wan2.2.

Critical strengths
  1. Using SAM2 memory features as a teacher is a useful way to transfer tracking information to a video generator. Relational alignment already appears in VideoREPA. The new design fuses local similarities from forward and backward tracking, then matches their distributions. Ablations of the loss, tracking direction, fusion method, and projector support these choices.
  2. The gains extend beyond one backbone. On CogVideoX, Motion Score rises from 94.02 with LoRA to 95.51, and FVD falls from 465.00 to 360.57. Wan2.2 also improves. Evaluation on 100 unseen source videos still gives lower FVD than LoRA and REPA. A randomized, double-blind study with 48 participants supports perceptual improvements over the base model, LoRA, and REPA.
Critical weaknesses
  1. Motion Score averages background consistency, subject consistency, and motion smoothness. These metrics do not directly measure limb connectivity or action completion. Without reporting how much the subjects move, it is also hard to tell whether higher consistency partly comes from less movement. The abstract should identify 95.51 as a custom score on 85 selected prompts, rather than the overall VBench score.
  2. The margins over stronger alternatives are small. Motion Score improves by 0.17 over mask supervision, 0.36 over V-JEPA 2, and 0.53 over VideoREPA. FVD improves by about 1.17% over VideoREPA. No uncertainty estimates show how stable these differences are. The human study and evaluation on unseen source videos omit these competitors. The evidence supports the complete training method, but its advantage over the strongest alternatives is less clear.
Questions for authors
  1. Can you assess structural failures, motion magnitude, and action completion on the existing samples, including mask supervision and VideoREPA? A human comparison with these methods would also help.
  2. How stable are the small gains across prompts and generation seeds? Please report paired confidence intervals and clarify the number of judgments behind the 71.4% human preference.
  3. With LGF-KL, the projector, and alignment settings fixed, how do other teachers and SAM2 without temporal memory perform? This would clarify what the tracking features add.
2 comments
AuthorsOpenReview ↗

Response to Reviewer QZwR

We thank the reviewer for the detailed and constructive review, and for noting that our gains hold across two backbones, unseen source videos, and our double-blind user study.

W1 / Q1. Limb connectivity, motion magnitude, action completion (including mask supervision and VideoREPA), and abstract wording.

  • Motion magnitude (Dynamic Degree, DD). We agree that consistency should be evaluated alongside motion amount. The revised Table 1 reports Dynamic Degree for every method (SAM2VideoX 47.30 vs. 45.95 for the base model, 44.59 for mask supervision, and 45.95 for VideoREPA). Across three generation seeds (Table R3 in the general response), SAM2VideoX achieves a higher mean DD (52.25) than all three competing high-consistency baselines, namely mask supervision (49.55), VideoREPA (48.20), and V-JEPA 2 (43.69), confirming that our consistency gains do not come from reduced motion.
  • Extra-limb defects (HumanScore Extra Limbs, Table R4). Our 100-video unseen test set comprises 70 human and 30 animal prompts. On the Extra Limbs metric of HumanScore (Xiang et al., NeurIPS 2026) evaluated on all 70 human videos (G3), SAM2VideoX achieves 78.71, with positive motion-adjusted point estimates against all seven baselines and higher scores than every baseline in Table R4 under both raw and motion-matched comparisons, in particular mask supervision (74.97; +3.75 raw, +4.18 motion-matched, 95% CIs excluding zero), the base model (74.02; +4.70 raw, 95% CI excluding zero; +1.02 motion-matched), and VideoREPA (78.25; +0.47 raw, +0.99 motion-matched).
  • Action completion (VideoVeritas). To evaluate action completion across all 100 test videos (70 human and 30 animal), we prompt VideoVeritas (Tan et al., ICML 2026), a Qwen3-VL-based video judge, to rate on a 1 to 5 scale whether the prompted action is completed. SAM2VideoX, mask supervision, and VideoREPA score 2.67, 2.67, and 2.62, showing that action completion is preserved.
  • Abstract wording. We agree and will revise the abstract to explicitly state that 95.51 is the composite Motion Score (background consistency, subject consistency, and motion smoothness) on the 85 motion-focused VBench-I2V prompts rather than the overall VBench score.

W2 / Q2. Small margins, multi-seed stability, user study details, and comparisons with strong alternatives.

  • Three-seed VBench + Dynamic Degree (Table R3). Evaluated across three generation seeds on all six fine-tuning methods, SAM2VideoX achieves a higher three-seed mean Motion Score than V-JEPA 2 (95.50 vs. 95.38), VideoREPA (95.18), point trajectory (95.04), and REPA (92.92), and is close to mask supervision (95.58) while generating higher Dynamic Degree (DD 52.25 vs. 49.55 for mask supervision, 48.20 for VideoREPA, and 43.69 for V-JEPA 2) and avoiding mask supervision's severe extra-limb defects (+3.75 raw / +4.18 motion-matched on HumanScore Extra Limbs).
  • User study breakdown (G2). The 71.4% overall preference aggregates 3,200 pairwise comparisons across 80 unique prompts (48 participants): 1,092, 1,056, and 1,052 pairs against the base model, LoRA, and REPA, preferring SAM2VideoX in 64%, 66%, and 84% of comparisons. While we could not run a new human study within the discussion window, we directly compare against mask supervision and VideoREPA on three-seed VBench + DD (Table R3), the HumanScore Extra Limbs detector (Table R4), and VideoVeritas action completion above.

Q3. Other teachers and SAM2 without temporal memory under the same LGF-KL objective.

Please see Table R1 and Table R2 in the general response (G1). Under the same training data, schedule, seed (42), alignment layer, and LGF-KL objective on CogVideoX:

  • Disabling SAM2's temporal memory drops 3-seed VBench Motion Score from 95.50 ± 0.26 to 94.99 ± 0.25 (Extended Motion Score from 96.08 ± 0.21 to 95.67 ± 0.22) and Dynamic Degree from 52.25 ± 5.12 to 44.14 ± 5.12 (−8.11); in the separate HumanScore evaluation on 70 human prompts, its raw Extra Limbs score drops from 78.71 to 75.26 (−3.45). Probing frozen SAM2 features before vs. after memory (Table R2) shows that memory conditioning makes the subject's camera-compensated motion direction +36% more decodable, yields smoother feature trajectories (higher temporal-order gap), and improves DAVIS region propagation.
  • Replacing SAM2 with CoTracker3 features under the same LGF-KL objective gives a similar mean VBench Motion Score (95.48 ± 0.10 vs. 95.50 ± 0.26 for SAM2VideoX) and a lower Dynamic Degree (43.69 ± 2.06 vs. 52.25 ± 5.12, −8.56). In the separate HumanScore evaluation on 70 human prompts, its raw Extra Limbs score is 77.08, compared with 78.71 for SAM2VideoX (Table R1).

Thanks, the response addresses my central concern, and I still lean slightly positive.

Reviewer_sdxVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes structure-preserving motion in SAM2 representations and distills it into a video DiT to improve video generation quality. Rather than distilling raw SAM2 features directly, the authors propose LGF, which distills the similarity distribution over neighboring regions. The resulting model outperforms REPA and LoRA fine-tuning baselines.

Critical strengths

Clear presentation. The paper is well written and easy to follow.

Critical weaknesses
  1. Limited justification for choosing SAM2 as the teacher. The proposed LGF appears similar to the local cost volume construction used in point tracking methods (ex. CoTracker3, LocoTrack). If the goal is to distill only local correspondence distributions, features from a point tracking model would likely provide a more accurate signal. However, tracking models are not considered in the initial backbone analysis. The scope of this analysis is therefore too narrow to justify the choice of SAM2, and given the design of the main method, SAM2 does not appear to be the optimal teacher.

  2. Unconvincing motivation for the method. DiffTrack[1] shows that temporal correspondence already emerges in video DiTs, and that query-key matching yields stronger correspondence than intermediate features. Several prior works have also improved video DiTs by applying losses directly to their attention. In terms of correspondence, distilling SAM2 features through LGF alone seems to offer little advantage over what the video DiT already provides. The authors should consider a direction that exploits the strong query-key matching inside the video DiT while also leveraging SAM2 features.

[1] Emergent Temporal Correspondences from Video Diffusion Transformers (NeurIPS'25, Nam et al.)

Questions for authors
  1. The quantitative gains in Table 1 appear marginal, and the comparison relies mainly on qualitative results. Were the video samples generated under identical inference settings (e.g., the same seeds and CFG scale) across all methods? Clarifying this would help rule out concerns about sample selection.
  2. Could a point tracking model be used as the teacher instead? For example, CoTracker could be applied with query points sampled within SAM2 masks, or dense tracking models such as AllTracker or CoWTracker could be used.
  3. How do the internal features of the SAM2-distilled video DiT change compared to the pretrained model? An analysis of these changes and how they relate to the generation results would strengthen the paper.
3 comments
AuthorsOpenReview ↗

Response to Reviewer sdxV

We thank the reviewer for the constructive comments and the concrete suggestions on point-tracking teachers and on the DiT's internal correspondence.

W1 / Q2. Why SAM2 rather than a point-tracking model? LGF resembles a local cost volume.

We agree that LGF is close in form to a local cost volume: both compare a token with its spatio-temporal neighbourhood in another frame. The open question is which features should be compared, and we now test this directly.

  • Distillation. Under the same LGF-KL objective, budget, and seed on CogVideoX, replacing the SAM2 teacher by CoTracker3 features gives a similar 3-seed VBench Motion Score (95.48 ± 0.10 vs. 95.50 ± 0.26 for SAM2VideoX) and a lower Dynamic Degree (43.69 ± 2.06 vs. 52.25 ± 5.12, −8.56). In the separate HumanScore evaluation on 70 human prompts, its raw Extra Limbs score is 77.08, compared with 78.71 for SAM2VideoX (Table R1 in the general response). Dense trackers such as AllTracker or CoWTracker are a natural next step that we could not train within the discussion period.
  • Analysis. In the motion-CKA analysis (Fig. 2(a), Appendix A), SAM2 features are substantially more motion-aligned than CoTracker3 features (0.469 vs. 0.371 on the same 200 videos).
  • Hypothesis. Point trackers are trained to localize individual query points, so their features are optimized for pointwise matching. SAM2 is trained to segment and propagate whole objects and parts through occlusion and deformation. We hypothesize that its local similarity distributions therefore also reflect which regions move together, and that this is what LGF transfers.

The paper's point-trajectory supervision baseline (following Track4Gen) supervises the generator with tracked trajectories rather than tracker features, reaching 95.11 Motion Score single-seed and 95.04 ± 0.24 across three seeds (vs. 95.51 and 95.50 ± 0.26 for SAM2VideoX).

W2. Temporal correspondence already emerges in video DiTs (DiffTrack).

We agree that DiffTrack shows pretrained video DiTs establish temporal correspondence, mainly through query-key matching in a subset of layers. Two points bear on our motivation:

Correspondence alone does not prevent structural failures. CogVideoX already has the emergent correspondence DiffTrack describes, yet its samples show the failures we target, such as duplicated legs and limbs that vanish between frames (Fig. 9 and 13). Matching points across frames does not by itself keep the parts of an articulated body coherent. LGF aligns how each region's local neighbourhood is redistributed across frames, capturing part coherence.

Distillation changes the DiT's representation. Fig. 2(c) shows that after distillation, intermediate DiT features are more aligned with motion than those of both the pretrained model and a step-matched LoRA control (same 200 videos; Appendix A). We have not tested supervising the DiT's query-key affinities directly with SAM2 relations; this is a plausible alternative that we will discuss in the revised paper as future work.

Q1. Quantitative gains in Table 1 and evaluation settings.

  • CogVideoX evaluation settings. All CogVideoX fine-tuning runs used training seed 42. The default quantitative and human evaluations used 50 denoising steps and a guidance scale of 6.0. Each qualitative comparison shows outputs for the same prompt across the methods displayed.
  • Quantitative gains beyond single-seed Motion Score. Please see G2 and G3 in the general response: because composite Motion Score sits in a narrow 93 to 96 range on strong models and trades off against motion magnitude, we evaluate three-seed Motion Score alongside Dynamic Degree and extra-limb detection. Across three generation seeds (Table R3), SAM2VideoX achieves a higher mean Motion Score (95.50) than V-JEPA 2, VideoREPA, point trajectory, and REPA while generating higher Dynamic Degree (52.25 vs. 49.55 for mask supervision, 48.20 for VideoREPA, and 43.69 for V-JEPA 2). On the HumanScore Extra Limbs metric (Table R4), SAM2VideoX reaches 78.71, with positive motion-adjusted point estimates against all seven baselines and higher scores than every baseline in Table R4 on both raw and motion-matched comparisons (notably +3.75 raw / +4.18 motion-matched over mask supervision, 95% CIs excluding zero; and +0.47 raw / +0.99 motion-matched over VideoREPA).

Q3. How do the internal features change?

Fig. 2(c) compares the motion alignment of intermediate DiT features for the pretrained model, a step-matched LoRA control, and SAM2VideoX on the same 200 videos; distillation increases it from 0.031 (base model) and 0.006 (LoRA) to 0.045 (SAM2VideoX). Appendix A gives the full protocol, which we will make more prominent in the revision.

Reviewer_sdxVOpenReview ↗

Thank you for your thoughtful responses

Dear Authors,

Thank you for your thorough and thoughtful responses to my questions, and I apologize for the delay in getting back to you. I appreciate the effort you put into the rebuttal. However, a few of my concerns remain unresolved, and I would like to explain them below.

1. Distinction from prior work. A number of existing studies improve generation quality by applying correspondence losses or mask-based losses to the internal layers of DiTs. I understand that your method does not directly use mask supervision or point trajectories. Still, the underlying mechanism appears similar, with the main difference being the scope over which the correspondence loss is applied. To establish a clear distinction, I believe it would be valuable to analyze how the video DiT changes, and how the generated videos differ, when trained with mask supervision, point trajectories, and LGF, respectively. The quantitative analysis in Figure 2(c) is helpful, but a more multifaceted analysis, such as PCA visualizations or changes in attention distributions, would make the contribution considerably more convincing.

2. Magnitude of improvement. The margins in the tables are relatively small. When I examined the videos, the proposed method outperformed LoRA and mask supervision on some samples but showed comparable quality on others. I think the paper would be much stronger if it demonstrated either a clear performance improvement or a more analytical account of what distinguishes SAM2 cost distillation from the alternatives. Unfortunately, I feel that both aspects are currently insufficient.

For these reasons, I will maintain my score of weak reject. I understand that time is limited at this stage, but if you are able to share any additional results or analyses addressing these points, I would be grateful to see them.

Best regards, Reviewer sdxV

AuthorsOpenReview ↗

Follow-up on LGF supervision and structural artifacts

Thank you for your thoughtful review and follow-up. We would like to draw attention to the PCA visualization of SAM2 memory features included in our original submission (Appendix A, Fig. 10). It shows consistent part-level patterns across frames. LGF uses these features to construct a soft distribution over candidate matches in each local next-frame neighborhood.

We agree that LGF shares a cost-volume-like construction with prior work. The distinction lies in what the loss supervises. Mask supervision specifies subject membership, our point-trajectory baseline supervises tracked point correspondences, and LGF transfers the teacher’s relative affinities across candidates. Fig. 8 isolates the effect of the loss within the same local neighborhoods and with the same SAM2 teacher: LGF-KL achieves a Motion Score of 95.51, compared with 94.51 for LGF-ℓ2.

We extended the student-side analysis in Fig. 2(c) to the mask-supervised Wan2.2 baseline using the same evaluation protocol. The U-centered motion CKA is 0.031 for the pretrained model, 0.006 for matched LoRA, 0.019 for mask supervision, and 0.045 for SAM2VideoX. Under this protocol, the LGF-trained DiT features show stronger motion alignment than the mask-supervised features.

We also agree that the three-seed mean Motion Scores are close for SAM2VideoX and mask supervision (95.50 vs. 95.58). On the 70 human prompts in our separate HumanScore evaluation, the raw Extra Limbs scores are 78.71 and 74.97, respectively. After adjusting for inter-frame motion, SAM2VideoX scores 4.18 points higher than mask supervision (95% paired bootstrap CI [1.42, 7.19]). This supports an improvement on the specific failure mode we target: duplicated or ghost limbs during articulated motion.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers (1/2)

We thank the reviewers for their constructive feedback. Reviewers found the problem practical and meaningful (qbUP), the presentation clear (sdxV, QZwR), the tracking teacher and relational LGF design well-motivated (qbUP, QZwR), and noted that our gains hold across two backbones, unseen source videos, and a double-blind user study (QZwR). In this and the next comment, we address the three shared questions across reviews; reviewer-specific points are answered under each review. We report the additional experimental results in these discussion comments. In the current revision, we clarify the evaluation protocol and the definition of Motion Score, and add single-seed Dynamic Degree to Table 1. These changes are marked in blue.

G1. What do SAM2's tracking features and memory contribute? (sdxV, QZwR, qbUP)

We run controlled teacher ablations on CogVideoX under the same training data, schedule, seed (42), alignment layer, and LGF-KL objective (Table R1, evaluated across three generation seeds).

Table R1. Teacher controls on CogVideoX under the same LGF-KL objective (3-seed VBench on 85 prompts and HumanScore Extra Limbs on 70 human prompts).

Teacher representationMemoryMotion Score ↑Ext. Motion Score ↑Dynamic Degree (DD) ↑Extra Limbs ↑
SAM2 bidirectional (SAM2VideoX, Ours)✓95.50 ± 0.2696.08 ± 0.2152.25 ± 5.1278.71
SAM2 image encoder only✗94.99 ± 0.2595.67 ± 0.2244.14 ± 5.1275.26
CoTracker3 features-95.48 ± 0.1096.08 ± 0.0943.69 ± 2.0677.08
  • Tracking memory (QZwR, qbUP). Disabling SAM2's memory module drops VBench Motion Score from 95.50 to 94.99 (Extended Motion Score from 96.08 to 95.67) and Dynamic Degree (DD) from 52.25 to 44.14 (−8.11); in the separate HumanScore evaluation on 70 human prompts (G3), its raw Extra Limbs score drops from 78.71 to 75.26 (−3.45). Probing frozen SAM2 features before vs. after the memory module (Table R2) explains why: memory conditioning makes the subject's camera-compensated motion direction +36% more decodable via a linear probe, yields smoother feature trajectories (higher temporal-order gap), and improves DAVIS region propagation.
  • Point trackers (sdxV). Replacing SAM2 with CoTracker3 features gives a similar mean VBench Motion Score (95.48 vs. 95.50 for SAM2VideoX) and a lower Dynamic Degree (43.69 vs. 52.25, −8.56). In the separate HumanScore evaluation on 70 human prompts, its raw Extra Limbs score is 77.08, compared with 78.71 for SAM2VideoX (Table R1). Moreover, in our motion-CKA analysis (Fig. 2a), SAM2 features are substantially more motion-aligned (0.469) than CoTracker3 (0.371), V-JEPA 2 (0.163), and VideoMAE V2 (−0.047).
  • Teacher vs. objective (qbUP). Table R1 varies the teacher with LGF-KL fixed, complementing Sec. 5.2 (Fig. 8), which varies the objective with the SAM2 teacher fixed (raw-feature ℓ2: 94.58, LGF-ℓ2: 94.51, LGF-KL: 95.51).

Table R2. Probes of frozen SAM2 features before vs. after the memory module (higher is better).

Probe measureBefore memory (w/o mem.)After memory (w/ mem.)
Motion-direction probe (cosine, subject box) ↑0.2390.325
Temporal-order gap ↑0.6030.650
DAVIS region propagation (J&F) ↑0.4960.523

Metric definitions: (1) Motion-direction probe: a cross-validated linear probe predicts the subject's camera-compensated 2D motion direction (mean optical flow inside the tracked subject box) from the frame-to-frame feature change; we report the cosine similarity between predicted and true directions. (2) Temporal-order gap: (d_far − d_adj) / d_far, where d_adj and d_far are the mean feature distances between consecutive frames and between frames at least four steps apart; higher means features evolve smoothly over time rather than jumping erratically. (3) DAVIS region propagation (J&F): standard region similarity (J) and contour accuracy (F) when propagating the first-frame mask across frames using frozen features.

AuthorsOpenReview ↗

General Response to All Reviewers (2/2)

G2. Multi-seed stability, motion magnitude, and VBench evaluation (sdxV, QZwR, qbUP)

Interpreting Motion Score alongside motion magnitude and limb defects. Composite Motion Score on strong video DiTs sits in a narrow 93 to 96 range, and temporal consistency naturally trades off against motion magnitude (smaller motion is easier to keep consistent). Pairing three-seed Motion Score with Dynamic Degree (DD) and extra-limb detection (G3) provides a fuller picture of motion quality:

  • Three-seed VBench + Dynamic Degree (Table R3). Across three generation seeds on the 85 VBench-I2V prompts (50 denoising steps, CFG 6.0), SAM2VideoX achieves a higher three-seed mean Motion Score (95.50) and Extended Motion Score (96.08) than V-JEPA 2, VideoREPA, point trajectory, and REPA. While mask supervision has a slightly higher mean Motion Score (95.58 vs. 95.50), SAM2VideoX generates more dynamic motion (DD 52.25 vs. 49.55 for mask supervision, 48.20 for VideoREPA, and 43.69 for V-JEPA 2) while avoiding the severe extra-limb defects of mask supervision (G3). Single-seed DD is also added to Table 1 (SAM2VideoX 47.30 vs. base 45.95, VideoREPA 45.95, mask 44.59).
  • User study breakdown. Our 71.4% overall preference (Fig. 6) aggregates 3,200 pairwise comparisons across 80 unique prompts (48 participants): SAM2VideoX is preferred in 64% of comparisons vs. the base model (1,092 pairs), 66% vs. LoRA (1,056 pairs), and 84% vs. REPA (1,052 pairs).

Table R3. Multi-seed VBench evaluation on CogVideoX (mean ± std across 3 generation seeds; best in bold, second-best underlined).

MethodMotion Score ↑Ext. Motion Score ↑Dynamic Degree (DD) ↑
Mask supervision95.58 ± 0.0896.14 ± 0.1049.55 ± 2.06
SAM2VideoX (Ours)95.50±0.26‾\underline{95.50 \pm 0.26}96.08±0.21‾\underline{96.08 \pm 0.21}52.25 ± 5.12
V-JEPA 295.38 ± 0.1496.04 ± 0.1043.69 ± 1.56
VideoREPA95.18 ± 0.1695.79 ± 0.1448.20 ± 4.34
Point trajectory95.04 ± 0.2495.71 ± 0.1853.15±2.81‾\underline{53.15 \pm 2.81}
REPA92.92 ± 0.0993.77 ± 0.0763.51 ± 4.68

G3. Direct evaluation of extra-limb defects (QZwR, qbUP)

To directly measure duplicated or ghost limb defects in articulated human motion, we evaluate the Extra Limbs metric from the HumanScore benchmark (Xiang et al., NeurIPS 2026), which runs the HADM human-anomaly detector on every frame to score duplicated or ghost limbs from 0 to 100 (higher means fewer defects) across all 70 human prompts of our 100-video test set (70 human, 30 animal).

Why and how we report Motion-Matched differences (Δ Motion-Matched). Static videos rarely trigger extra-limb defects, whereas larger body motion naturally makes articulated defects more likely to appear, so raw scores alone favor methods that move less. To compare methods at an identical motion level, we fit a linear model of the Extra Limbs score with per-prompt intercepts, per-method offsets, and a shared slope on inter-frame motion magnitude. Δ Motion-Matched is the resulting motion-adjusted score gap (SAM2VideoX − Baseline) at equal motion magnitude (Table R4).

SAM2VideoX achieves 78.71 on Extra Limbs, with positive motion-adjusted point estimates against all seven baselines and higher scores than every baseline in Table R4 under both raw and motion-matched comparisons. Notably, mask supervision, which ties with SAM2VideoX on VBench Motion Score, suffers severe extra-limb artifacts (74.97, trailing SAM2VideoX by +3.75 raw / +4.18 motion-matched, with 95% CIs excluding zero under both comparisons). SAM2VideoX also scores higher than the base model (+4.70 raw, 95% CI excluding zero; +1.02 motion-matched), VideoREPA (+0.47 raw / +0.99 motion-matched), REPA (+2.72 / +2.22), and LoRA (+2.63 / +1.57). V-JEPA 2 (78.87) and point trajectory (78.94) have slightly higher raw scores while exhibiting lower inter-frame motion on this set (motion magnitude 2.59 and 3.09 vs. 3.41 for SAM2VideoX), whereas SAM2VideoX has higher motion-adjusted scores than both (+1.88 and +0.57), while preserving action completion on VideoVeritas (2.67 vs. 2.67 for mask and 2.62 for VideoREPA).

Table R4. HumanScore Extra Limbs (0-100, higher is better) on the 70 human prompts, CogVideoX.

MethodExtra Limbs Score ↑Δ Raw (SAM2VideoX − Baseline)Δ Motion-Matched (SAM2VideoX − Baseline)
SAM2VideoX (Ours)78.71--
VideoREPA78.25+0.47+0.99
LoRA fine-tuning76.08+2.63+1.57
REPA76.00+2.72+2.22
Mask supervision74.97+3.75+4.18
Base model74.02+4.70+1.02

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.