- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies structure-preserving motion in image-to-video generation, in particular for articulated subjects such as humans and animals. It proposes SAM2VideoX, which distills representations from the SAM2 tracking into a video diffusion model during fine-tuning. The method combines forward and backward SAM2 information through bidirectional Local Gram Flow fusion and aligns local cross-frame feature relations using a KL-divergence objective. Experiments show improvements in temporal consistency and perceptual quality without requiring additional tracking or motion controls at inference.
Critical strengths
- The paper addresses a meaningful practical problem. Structural deformation, appearance drift, and inconsistent articulated motion remain important failure modes of video generation.
- The teacher choice and technical design sound reasonable. SAM2’s temporally informed tracking representations are a plausible source of supervision for preserving object identity and structure. Aligning local feature relations also avoids requiring coordinate-wise correspondence between heterogeneous teacher and student representations.
- The experiments include several useful controls. Comparisons with diffusion-only LoRA fine-tuning, mask supervision, point-trajectory supervision, and alternative teachers help assess different aspects of the approach.
Critical weaknesses
- The contribution over closely related methods is kind of incremental, and in some sense, insufficiently established. Track4Gen already demonstrates that tracking supervision can improve video generation, while VideoREPA, CREPA, and MoAlign investigate temporal or motion-oriented representation alignment. Although the paper cites CREPA and MoAlign, it provides neither quantitative comparisons nor a clear justification for excluding them. Its advantages over the closest alternatives need stronger evidence.
- The specific benefit of tracking memory is not directly isolated. Comparisons with other foundation-model teachers support the overall approach, but do not establish whether SAM2’s advantage specifically comes from its tracking memory. A control experiment using the same SAM2 image encoder without temporal memory, under the same LGF objective, would directly test the central hypothesis. Ideally, a more systematic comparison of teacher representations under a shared alignment objective would further separate teacher quality from loss design.
- The evaluation only indirectly measures the claimed structural improvements. The primary Motion Score combines background consistency, subject consistency, and motion smoothness. These metrics do not directly measure key factors in motion naturalness, such as limb connectivity, part topology, or anatomically plausible motion. Moreover, the gains over VideoREPA are modest: Motion Score increases from 94.98 to 95.51, while FVD decreases from 364.82 to 360.57.
Questions for authors
- Could you compare against CREPA and MoAlign under the same backbone, training data, and optimization budget, or explain the technical reasons for excluding them? Which specific capability does LGF provide beyond existing temporal relational alignment objectives?
- Could you evaluate SAM2 features with temporal memory disabled while retaining the same image encoder, projector, and LGF loss? Could alternative teachers also be evaluated with the same LGF objective?
- Could you provide evaluations targeting articulated structural failures, together with motion magnitude or dynamic degree? Are the gains over VideoREPA consistent across multiple training or generation seeds?
Response to Reviewer qbUP
We thank the reviewer for the thoughtful review and the precise control experiments suggested.
W1 / Q1. Incremental over Track4Gen, VideoREPA, CREPA and MoAlign; missing comparisons; what does LGF provide beyond existing relational objectives?
CREPA and MoAlign. Neither method has released code, and we could not reimplement them faithfully within the discussion period, so a direct comparison is missing. We explain the technical differences and name the closest evaluated baselines:
What LGF provides. These methods differ from ours along two axes:
Track4Gen. Track4Gen supervises the generator with explicit point-tracking targets. Our point-trajectory baseline follows Track4Gen on the same backbone (Sec. 5); SAM2VideoX improves on it in Motion Score (95.11 → 95.51 single-seed; 95.04 → 95.50 three-seed), Extended Motion Score (95.69 → 96.03 single-seed; 95.71 → 96.08 three-seed), and HumanScore Extra Limbs (+0.57 motion-matched).
W2 / Q2. The benefit of tracking memory is not isolated; other teachers under the same LGF objective.
Please see G1 (Tables R1 and R2). Under the same LGF-KL objective on CogVideoX, disabling SAM2's temporal memory drops 3-seed VBench Motion Score from 95.50 ± 0.26 to 94.99 ± 0.25 and Dynamic Degree from 52.25 ± 5.12 to 44.14 ± 5.12 (−8.11); in the separate HumanScore evaluation on 70 human prompts, its raw Extra Limbs score drops from 78.71 to 75.26 (−3.45). Replacing SAM2 with CoTracker3 features gives a similar VBench Motion Score (95.48 ± 0.10) and a lower Dynamic Degree (43.69 ± 2.06, −8.56); on HumanScore Extra Limbs, its raw score is 77.08 vs. 78.71 for SAM2VideoX. Table R2 shows what memory conditioning adds in frozen SAM2 features (+36% motion-direction probe, higher temporal-order gap, and higher DAVIS J&F).
W3 / Q3. Evaluation of articulated structural failures; motion magnitude; consistency across seeds.
Presentation. We revised the evaluation protocol and the Table 1 caption, in particular the definition of Motion Score, to make the experimental section easier to follow.