acceptodds
Withdrawn from ICLR 2027

Structured Motion-Instance Graphs and Persistent Memory for Long-Horizon Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generating long videos in successive chunks requires preserving instance identity, motion, and interactions across chunk boundaries. We propose a framework that couples a Motion-Instance Graph (MIG) with persistent Instance Memory (IM) for controllable long-horizon video generation. MIG organizes subjects, static regions, trajectories, interactions, and event states into a typed graph, which is converted into frame-level motion and spatial conditions. IM retrieves historical instance states before each chunk and updates them from generated observations, allowing subsequent chunks to reuse appearance, trajectory, and interaction information. Lightweight MI-Adapter and LoRA modules inject graph conditions and memory tokens into a frozen video diffusion backbone. First-frame anchoring and frame-aware corruption further support cross-chunk consistency. We evaluate the framework on 160 conditions with three random seeds, conduct incremental ablations, and test generation at 240, 720, and 1,440 frames. On CogVideoX-2B, normalized average displacement error decreases from 0.2108 for the trajectory-control baseline to 0.1850, while background consistency increases from 0.618 for the long-video baseline to 0.750. Evaluations on Wan2.1-1.3B and VideoCrafter2 also show improvements over their paired baselines. The framework provides explicit instance-level control and persistent state for long-video generation without updating the backbone parameters.

Primary area:
generative models
Keywords:
long-horizon video generation, motion-instance graphs, instance memory, video diffusion, parameter-efficient adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CksMOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a framework for the chunk-based generation of long videos, combining a Motion-Instance Graph (MIG) with a persistent Instance Memory (IM):

MIG: Organizes subjects, static regions, trajectories, interactions, and event states into a typed graph, which is then converted by a DMG Planner into 9-channel frame-level motion and spatial conditions. IM: Performs top-k retrieval based on instance identity before the generation of each chunk, and updates the memory with observed results following generation. Injection Mechanism: Employs an MI-Adapter and LoRA to inject information into a frozen CogVideoX-2B model, with 14.87M (0.86%) trainable parameters; the framework also incorporates first-frame anchoring and frame-aware corruption.

Critical strengths
  1. Clear problem definition: The generation of long-form video is framed as the maintenance of a "world state" across chunks, replacing implicit KV history with an explicit, inspectable instance state—offering both interpretability and editability.

  2. Parameter-efficient with low overhead: Only ~1% of parameters are trained; inference time increases by approximately 12.5%, and peak VRAM usage rises from 16 GiB to 19 GiB.

  3. Well-structured evaluation scenarios: The seven stress-test scenarios in Table S2 cover complex cases such as large-scale displacement, reappearance after occlusion, and multi-entity interaction. The evaluation is further supported by a progressive ablation study (A0–A8), transfer experiments across three different backbones, and blind human evaluations.

  4. Transparent discussion of limitations: The authors candidly acknowledge that errors in parsing or association can be written into the memory and subsequently propagated.

Critical weaknesses
  1. The "baseline" mentioned is not an established, previously published method. The configurations labeled "Layout/BBox," "Trajectory," and "History-extension" in Table 1 are custom setups devised by the authors using CogVideoX-2B (Table S3 merely lists "design references"), with no information provided regarding training data, number of steps, or implementation details. Furthermore, the paper lacks direct comparisons with any official implementations of existing methods, such as Tora, MotionCtrl, DragNUWA, Boximator, FreeNoise, StreamingT2V, or LongLive.

  2. The novelty is questionable, and there are significant omissions in the related work. The paper’s approach—using an LLM to plan layouts or trajectories to guide video generation—overlaps substantially with existing research (e.g., LVD, VideoDirectorGPT, and DreamRunner), yet these are neither cited nor discussed. Additionally, the discussion on methods for long-video memory (such as WorldMem, Context-as-Memory, and FramePack) is insufficient.

  3. The evaluation process is circular and unfair. ADE/FDE metrics measure deviations from the system’s own planned target trajectories, yet the "frozen base" model is not provided with any trajectory inputs. The writing of data into the memory relies on detection and association; since the evaluation also employs the same YOLOv7-based association protocol, the data-writing process and the scoring mechanism are coupled.

4 . The methodology is described vaguely, making it difficult to reproduce. The paper fails to specify what kind of model the "SMR Parser" is. It does not explain how the "DMG Planner" generates specific trajectories from text. It does not describe how the key/value pairs for the memory are extracted from the generated videos. There is no algorithmic description for conflict resolution or history compression. The source, annotation method, and licensing of the 96,000 training videos are not disclosed, nor is there any mention of code availability.

  1. Data consistency and credibility are questionable. In Table S6, the incremental improvements from A1 to A8 are suspiciously smooth and monotonic; the unit for "Detection ∆ +86.92" is undefined. In Table S9, the numerical intervals are overly uniform (0.990/0.970/0.950; 0.450/0.510/0.550), and the "mean jump" actually decreases as the length increases. The values ​​for the "full model" are inconsistent between Table 1 and Table 2 (DINO: 0.9580 vs. 0.9583; ADE: 0.1850 vs. 0.1851). A "memory scale" factor of only 8×10⁻⁵ leads to significant performance gains, yet this is left unexplained. The appendix mentions that a grid search for hyperparameters was conducted, but no results are provided.

  2. Insufficient statistics and human evaluation. Although three seeds were used, no variance or significance metrics are reported. The human evaluation involved only three unidentified raters and lacks inter-rater reliability coefficients; furthermore, comparisons were made exclusively against a "frozen" baseline rather than the strongest available baseline.

  3. The experiments on long-form videos lack proper controls. For the 240/720/1440-frame settings, the authors only report diagnostic metrics—such as detection coverage—for the full model itself, with no comparisons to baselines or standard long-video evaluation metrics. Additionally, the backbone model used is outdated (CogVideoX-2B, 720×480).

Questions for authors
  1. How are the three control configurations in Table 1 implemented and trained? Are the training data, number of steps, and input conditions consistent with those used for the method proposed in this paper? Could you provide a direct comparison with the official implementations of Tora, MotionCtrl, StreamingT2V, LongLive, etc.?

  2. When calculating ADE/FDE, what serves as the "target trajectory" for the frozen baseline that lacks trajectory input? If all methods are provided with the same trajectory conditions, how significant is the remaining performance gap?

  3. What are the specific implementations of the SMR Parser and DMG Planner (e.g., based on an LLM, rule-based logic, or a learned model)? What encoder is used to extract the key/value pairs for the memory?

  4. Is the detector used for IM updates the same as the YOLOv7 model used for evaluation? If an independent detector (such as GroundingDINO + SAM2) were used for evaluation, would the results remain consistent?

  5. Could you provide the mean ± standard deviation across 3 seeds, along with the results of significance tests? Additionally, could you specify the background of the human evaluators and the inter-rater reliability coefficients (e.g., Krippendorff’s α)?

  6. Could you explain the discrepancies in the numerical values ​​between Table 1 and Table 2, the definition of "Detection ∆" in Table S6, and why the "mean jump" metric decreases as the sequence length increases? Would it be possible to release the raw, per-video results?

  7. Given that the "memory scale" is only 8×10⁻⁵, how does the IM (Internal Memory) mechanism exert a significant influence? Could you provide a plot illustrating the relationship between the memory scale and performance (i.e., the results from the hyperparameter grid search mentioned in the appendix)?

  8. How do the baseline models and other existing methods for long-video generation perform in the experiments involving 240, 720, and 1440 frames? Could you report metrics such as those from VBench-Long or drift-related evaluations?

  9. What are the fundamental differences and advantages of MIG compared to other approaches—such as LVD, VideoDirectorGPT, and DreamRunner—that utilize LLMs for planning in conjunction with video generation?

1 comment
AuthorsOpenReview ↗

Response to Reviewer CksM

Thank you for the review. We revised baseline comparisons, evaluation, reproducibility, statistics, and long-horizon experiments. Our responses follow.

W1 — Baseline fairness and official comparisons. Our controlled baselines share CogVideoX-2B, 86,400 training videos, 6,000 AdamW updates (4,000+2,000), batch 16, LR 2.5e-5, and LoRA rank 32. Trainable parameters are 14.21M–14.52M versus 14.87M for ours; training costs are 51.4–52.8 versus 54.1 A100 GPU-hours (Sec. 4.1, Table S3). We also evaluated the released implementations/checkpoints of Tora, MotionCtrl, DragNUWA, Boximator, FreeNoise, StreamingT2V, and LongLive using their supported native configurations and a common evaluation benchmark (Tables 1, S8b). Tracking and trajectory metrics favor ours; background/motion results vary by backbone.

W2 — Novelty and related work. Sec. 2 now discusses LVD, VideoDirectorGPT, DreamRunner, FramePack, Context-as-Memory, and WorldMem. Unlike upfront motion scripts and frame/viewpoint-indexed context, MIG–IM associates appearance, trajectories, interactions, and event states with explicit instance identities, then corrects the next chunk's graph and memory from observations of the generated chunk. An A8 open-loop ablation isolates this feedback (Table 2).

W3 — Trajectory and detection fairness. The target trajectory is annotated from each held-out source video, not produced by DMG. The frozen model receives no trajectory and is reported as an unconditional reference; trajectory-conditioned methods share the same reference trajectory. Valid ADE falls from 0.2108 (Trajectory control) and 0.2064 (official Tora) to 0.1850 (ours). Online write-back uses YOLOv7+ByteTrack; independent offline GroundingDINO+SAM 2 rescoring preserves the main relative trends (Secs. 4.1–4.2; Tables 1, S4b, S8b).

W4 — Reproducibility. Secs. 3.2–3.5 and Apps. A.1–A.3 specify the GroundingDINO/SAM 2 + Qwen2.5-VL SMR parser, cubic-Hermite DMG planner and nine channels, DINOv2/Fourier/MLP memory features, 256-D key/value projections, Hungarian matching, confidence gating (0.50), anchored EMA, occlusion handling, and 64-record compression (Eqs. S3–S8; Algorithm 1). The 96,000-video corpus now has sources (OpenVid-1M/Panda-70M), splits, deduplication, and annotation details.

W5 — Numerical and metric consistency. Tables 1 and 2 now share direct 480-video aggregation rather than the earlier rounded intermediate averages. Table S6 defines Detection Delta as relative association-rate gain (51.40% to 96.08%, +86.92%) and Coverage Delta as absolute full-track completion gain (0.350 to 0.920, +0.570). The original detected-seam jump excluded detector dropouts; all-seam jump now includes them and rises 4.31/4.58/4.89 px over 240/720/1,440 frames (Table S9). Norm analysis explains memory scale 8e-5 (3.40% residual); Table S6b/Fig. S2 report the sweep.

W6 — Statistics and human study. Primary controlled comparisons and ablations now report three-seed mean ± SD. Paired Wilcoxon/t-tests use 160 condition-level observations. The blind study was expanded to eight computer-vision researchers, with additional baseline matchups, inter-rater agreement, and paired tests (Sec. 4.4; Tables 4, S10).

W7 — Long-horizon evidence and scaling. Figure 6 and Table S9b compare controlled baselines and supported official long-video methods at 240, 720, and 1,440 frames using drift diagnostics and five VBench-Long metrics. At 1,440 frames, ours achieves 95.1% detection coverage, 0.9382 subject consistency, and 0.9415 background consistency. CogVideoX-5B adds a fourth paired-backbone evaluation with 0.62% trainable parameters (Table 3, S8).

Q1. Training budgets/capacities: Table S3; official implementations and protocols: Tables 1, S8b. Q2. The frozen model is assessed against the held-out ground-truth trajectory only as an unconditioned reference. Shared-trajectory comparisons against Trajectory control and Tora remain favorable (W3). Q3. Parser, planner, and memory encoder are specified in Secs. 3.3–3.5 and Apps. A.2–A.3 (W4). Q4. Online YOLOv7+ByteTrack and independent offline GroundingDINO+SAM 2 are distinguished; both are reported in Tables 1, S4b, S8b. Q5. Three-seed variability, paired tests, eight-rater backgrounds, and agreement are reported in Sec. 4.4 and Table S10. Q6. Aggregation differences, delta definitions, detected/all-seam jumps, and the plan to release per-video logs with code are addressed in W5 and Tables S6, S9. Q7. Norm-ratio explanation: Eq. S2; memory-scale grid: Table S6b, Fig. S2. Q8. Multi-horizon baseline and VBench-Long results: Fig. 6, Table S9b. Q9. Closed-loop re-anchoring differs from upfront LLM planning (W2). Turning off write-back changes association 96.08%→84.65%, DINO 0.9580→0.9504, and ADE 0.1850→0.2008 (Table 2).

Reviewer_LmBsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a framework for long-horizon video generation that combines a Motion-Instance Graph (MIG) with persistent Instance Memory (IM). MIG is a typed graph representing subjects, static regions, trajectories, interactions, and event states, which is converted into frame-level motion and spatial conditions via a Dynamic Motion Guidance (DMG) Planner. IM stores per-instance historical states (appearance, trajectory, interactions) that are retrieved before each generation chunk and updated after generation, enabling cross-chunk state maintenance. These structured conditions are injected into a frozen video diffusion backbone through lightweight MI-Adapter and LoRA modules (≈0.86% trainable parameters). The framework is evaluated on 160 conditions with three seeds on CogVideoX-2B, with cross-backbone transfer to Wan2.1-1.3B and VideoCrafter2, ablations, blind human evaluation, and generation up to 1,440 frames. Reported gains include ADE reduction from 0.2108 to 0.1850 and background consistency increase from 0.618 to 0.750 over the strongest baselines.

Critical strengths

Practical, parameter-efficient integration with frozen backbones. The framework achieves consistent improvements across three different backbones (CogVideoX-2B, Wan2.1-1.3B, VideoCrafter2) while training only ~0.86–1% of parameters. This is consequential because it demonstrates that explicit structured state can be added to existing strong generators without retraining, which materially lowers the barrier to adoption and supports the paper's transferability claim.

Explicit instance-level state as a first-class representation. Organizing subjects, trajectories, interactions, and event states into a typed, updatable graph—and coupling it with a persistent memory indexed by instance identity—provides a principled mechanism for cross-chunk consistency that goes beyond noise rescheduling or history conditioning. The category-wise results (Table S5) showing gains across all seven evaluation categories, including occlusion/reappearance and multi-subject interaction, support the claim that the representation addresses the intended failure modes.

Critical weaknesses

Baseline implementation fidelity is insufficiently documented. The three control configurations are the primary evidence for the method's advantage, yet the paper does not specify how these baselines were trained (loss, capacity, tuning budget) or whether they were adapted to the frozen backbone with comparable effort. If the baselines are under-tuned, the reported margins (e.g., 12–21% relative gains) may not reflect a fair comparison. This is consequential because the central empirical claim—that structured MIG+IM outperforms alternative conditioning routes—rests on these comparisons.

The memory update mechanism is described only qualitatively. Section 3.5 states that observations are merged, conflicts filtered by confidence, and older records compressed, but the actual update equations (how appearance/trajectory fields are written, how confidence is computed/updated, how compression selects records) are not given even in the appendix. Since persistent memory is one of the two core contributions, this limits reproducibility and makes it difficult to assess whether the gains come from the memory design or simply from additional conditioning capacity. This is consequential because it undermines the paper's ability to substantiate its second claimed contribution.

Questions for authors

Baseline training details: How were the layout/BBox, trajectory, and history-conditioned baselines trained? Specifically, were they given the same number of optimizer updates, comparable trainable capacity, and the same two-stage schedule as the full method? Please provide per-baseline trainable parameter counts and training costs.

Memory update equations: Please provide the explicit update rule for memory records—how appearance, trajectory, event state, and interaction fields are written from generated observations, how confidence is computed and updated, and how history compression selects which records to retain. If this is in the supplement, please point to the specific equation/table.

Detection-rate reporting: Can you report per-method detection coverage/association rates on the 160-condition evaluation set? This would clarify whether ADE differences are partly attributable to detection artifacts under occlusion.

1 comment
AuthorsOpenReview ↗

Response to Reviewer LmBs

Thank you for recognizing the complementary roles of Motion-Instance Graph (MIG) and Instance Memory (IM), and for the specific requests on training fairness, memory updates, and detection-related evaluation. We have revised the main paper and supplement accordingly.

W1 — Controlled baseline training and capacity. The three CogVideoX-2B baselines use the same 86,400 training videos, standard diffusion denoising loss, 6,000 AdamW optimizer updates (4,000 single-window + 2,000 continuation), global batch size 16, bf16, learning rate 2.5e-5, and LoRA rank/alpha 32. Their trainable parameter counts and measured training costs are: Layout/BBox 14.21M / 51.4 A100 GPU-h; Trajectory control 14.35M / 51.9 GPU-h; History-conditioned extension 14.52M / 52.8 GPU-h. Ours uses 14.87M / 54.1 GPU-h. These details now appear in Sec. 4.1 and Table S3. The controlled comparisons thus use matched optimization budgets and closely matched capacity. We additionally report official Tora in Table 1 and six other published implementations in Table S8b, distinguished from our controlled configurations.

W2 — Explicit IM write-back and compression. Sec. 3.5 and Appendix A.3 now provide Eqs. S3–S8 and Algorithm 1. After each 49-frame chunk, YOLOv7+ByteTrack extracts tracks; confidence-weighted ROI-aligned DINOv2 features, 64-D Fourier trajectory features, and 64-D MLP relation/event features are projected into 256-D memory keys and values. Hungarian association uses overlap IoU and appearance similarity, with spatial gating for re-entry after occlusion. Composite confidence combines detection confidence (0.4), spatial consistency (0.3), and first-frame appearance similarity (0.3). Observations with score ≥0.50 update memory through first-frame-anchored EMA (anchor weight 0.25) and re-anchor the next chunk's spline. Lower-confidence or fully occluded tracks cannot overwrite memory; their trajectories use velocity extrapolation with 0.90 damping. When more than 64 snapshots remain, a utility based on confidence, recency, and distinctiveness selects redundant same-instance records for merging; first-frame anchors are retained.

Q1 — Baseline details. Table S3 documents the per-method 6,000-update schedule, parameter counts, training GPU-hours, and conditioning paths summarized in W1.

Q2 — Update equations. Main-text Eq. 4 and Appendix A.3 Eqs. S3–S8 / Algorithm 1 specify extraction, association, confidence updates, anchored EMA, occlusion handling, and history compression (W2).

Q3 — Detection/association and ADE validity. We added per-method Detection Coverage (DetCov), Valid Association Rate (AssocRate), Full-Track Completion (TrackComp), valid-track ADE, and unconditional ADE under both YOLOv7+ByteTrack and independent offline GroundingDINO+SAM 2 (Tables 1, S4b). On 160 conditions × three seeds, frozen CogVideoX-2B records 68.4% DetCov / 51.4% AssocRate, versus 96.8% / 96.1% for ours. Against Trajectory control, valid ADE falls 0.2108→0.1850 (12.24%) and unconditional ADE falls 0.2486→0.1924 (22.61%). The unconditional metric penalizes missed tracks rather than excluding them; independent rescoring supports the principal trends. The 24-condition occlusion/reappearance subset is also reported in Table S4b.

Reviewer_HTk9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a framework for long-video generation that combines a Motion-Instance Graph (MIG) with persistent Instance Memory (IM). MIG represents subjects, regions, trajectories, interactions, and event states. A planner converts the graph into frame-level control conditions. IM is designed to retrieve historical instance states before each chunk and update its records using observations after generation. Graph conditions and memory features are incorporated into frozen video models through MI-Adapter/LoRA. The method also uses first-frame anchoring and frame-aware corruption to support consistency across chunks. The authors evaluate the framework on 160 conditions using three random seeds and three backbones, and report human evaluations and video generation up to 1,440 frames. The main comparisons use layout control, trajectory control, and history-conditioned extension baselines built on the same backbone. The authors report improvements in identity consistency, trajectory accuracy, and background consistency, among other metrics, for the full framework.

Critical strengths
  1. The paper links motion plans, identity, interactions, and historical states to the same instance, offering a useful approach to maintaining state across chunks in video generation. The graph specifies control targets for the current chunk, while memory stores previous appearance and state information, giving the two components complementary roles. Integration into frozen video models through lightweight adaptation modules also facilitates the reuse of existing generative models.

  2. The experiments include component ablations and human evaluations of 60 video pairs by three evaluators. The authors also adapt and test the framework on three backbones, CogVideoX-2B, Wan2.1-1.3B, and VideoCrafter2, demonstrating its applicability across different video generation models.

Critical weaknesses
  1. Graph construction and cross-chunk state updates are insufficiently specified. The graph is central to the framework, yet the paper does not adequately describe the parsing model, the specific graph construction rules, or how observations from generated chunks are used to update nodes, relations, and event states.

  2. The experiments do not sufficiently establish the framework’s relative advantages in long-video generation. The paper lacks direct comparisons with existing long-video generation methods, and the main results compare control configurations built on the same backbone. Moreover, clear cross-frame content inconsistencies are visible in Figure 1, and the presented evidence does not adequately support the claimed long-term consistency.

Questions for authors
  1. How is MIG updated after each chunk is generated? Figures 3 and 4 illustrate several key operations, but the main text provides only high-level descriptions of their functions, without sufficient detail on how they are carried out.

  2. How does the framework perform compared with existing long-video generation methods?

  3. What are the full names and specific implementations of SMR and DMG? Please define these abbreviations.

1 comment
AuthorsOpenReview ↗

Response to Reviewer HTk9

Thank you for recognizing the complementary roles of our Motion-Instance Graph (MIG) and Instance Memory (IM). We have clarified the cross-chunk update algorithm, added comparisons with released long-video methods, and corrected the qualitative examples.

W1 — Graph construction and cross-chunk updates. Secs. 3.2–3.5 and Appendices A.2–A.3 now specify the SMR parser and DMG planner. GroundingDINO+SAM 2 detects foreground objects and static regions in the initial frame, with DINOv2 appearance anchors; frozen Qwen2.5-VL-7B-Instruct parses the prompt into event phases, waypoints, states, and typed graph relations. DMG constructs piecewise cubic Hermite trajectories with obstacle and interaction constraints and rasterizes nine frame-level conditioning channels. After each generated chunk, Algorithm 1 updates MIG in four steps: (1) YOLOv7+ByteTrack observations are linked to graph nodes via Hungarian matching over the eight-frame overlap, with re-entry spatial gating; (2) observations passing composite confidence ≥0.50 re-anchor the next spline and update first-frame-anchored appearance memory; (3) missing or low-confidence subjects transition to occluded state, freezing write-back and extrapolating velocity with 0.90 damping; and (4) contact/region-entry predicates advance event phases. Equations S3–S8 provide the corresponding retrieval, association, confidence, and compression rules.

W2 — Published long-video baselines and Figure 1. We added official/released implementations of Tora, MotionCtrl, DragNUWA, Boximator, FreeNoise, StreamingT2V, and LongLive to the 160-condition benchmark (Table 1, Table S8b). At 240, 720, and 1,440 frames, Table S9b compares the supported long-video methods (Tora, FreeNoise, StreamingT2V, LongLive) and controlled baselines using tracking, trajectory error, all-seam jump, static drift, and five VBench-Long metrics. At 1,440 frames, our model retains 95.1% detection coverage, versus 74.2% for LongLive; all-seam jump is 4.89 px versus 6.84 px, with VBench-Long subject/background consistency 0.9382/0.9415. We also re-exported Figure 1 from the final full-model checkpoint using matched prompts, reference frames, and seeds. This corrects the first-row missing tracking overlay and the second-row stray foreground/background artifact in the earlier export. Updated Figure 6 and the Figure S5 contact sheets give longer-range visual context.

Q1 — How MIG changes after each chunk. The four-step process in W1 is specified in Sec. 3.5, Eqs. S4–S8, and Algorithm 1: track association; confidence-gated state/memory update; occlusion and extrapolation; observed-predicate event transitions.

Q2 — Performance against long-video methods. Tables S8b/S9b and Figure 6 provide the requested comparisons at supported horizons. Our method shows stronger tracking, trajectory, and multi-horizon subject/background consistency on the reported benchmark; metric-specific results across backbones are given in Table S8b. Blind human preference versus Tora and LongLive is 70.8% and 70.5% of non-tied overall judgments, respectively (Tables 4, S10).

Q3 — SMR and DMG definitions. SMR is the Structured Motion-Relation Parser (GroundingDINO+SAM 2 for initial visual nodes and Qwen2.5-VL-7B-Instruct for graph/event parsing). DMG is the Dynamic Motion Guidance Planner (constrained cubic Hermite trajectory solver and nine-channel rasterizer). Definitions and implementation details appear in Secs. 3.2–3.3 and Appendix A.2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Official Comment

We sincerely thank the reviewers for the constructive feedback. We have substantially revised the manuscript and supplementary material to address the concerns about fairness, reproducibility, evaluation coupling, statistical reporting, and long-horizon validation.

  1. Controlled baselines and published methods. The three controlled CogVideoX-2B baselines use the same 86,400 training videos, 6,000 AdamW updates, batch size, learning rate, LoRA rank, and two-stage training schedule as our method, with closely matched trainable parameter counts and training cost (Table S3). We additionally evaluate official implementations or released checkpoints of Tora, MotionCtrl, DragNUWA, Boximator, FreeNoise, StreamingT2V, and LongLive (Tables 1, S8b, and S9b). The controlled baselines are the same-backbone comparison. Published methods are evaluated with their official/native backbone and inference protocol, while prompts, initial frames, seeds, output horizons, and evaluation scripts are held fixed whenever supported.

  2. Trajectory fairness and evaluator independence. The reference trajectory is a frame-level ground-truth bounding-box trajectory annotated on each held-out source video before generation; it is not produced by our planner. Trajectory-conditioned methods receive the same reference trajectory. The frozen base model is not trajectory-conditioned, so its displacement error is reported only as an unconditional reference against that same trajectory. Online memory updates use YOLOv7 and ByteTrack, whereas all generated videos are additionally rescored with an independent offline GroundingDINO+SAM 2 evaluator. The major method rankings and relative trends remain consistent across the two evaluators (Table S4b).

  3. Method and data reproducibility. Sections 3.2–3.5 and Appendices A.1–A.3 now specify the SMR Parser, DMG Planner, and Instance Memory, including the graph representation, nine conditioning channels, feature construction, Hungarian matching, confidence gating, anchored EMA write-back, occlusion handling, history compression, equations, and Algorithm 1. We also document the training-data provenance, automatic annotation pipeline, deduplication, and disjoint evaluation split.

  4. Statistical and numerical corrections. We unified the aggregation scripts to average directly over the 480 generated videos and report three-seed means and standard deviations for the primary CogVideoX-2B comparisons and ablations. We corrected the Table 1/Table 2 aggregation discrepancy, clarified Detection Delta and Coverage Delta, added unconditional trajectory error, expanded the memory-scale grid search, and added paired condition-level significance tests.

  5. Long-horizon and human evaluation. We added 240-, 720-, and 1,440-frame comparisons with internal and official long-video baselines, including the five VBench-Long metrics and all-seam boundary-jump analysis (Figure 6 and Table S9b). We also expanded the blind human study to eight independent computer-vision researchers and report pairwise preferences, inter-rater reliability, and condition-level significance tests (Table 4 and Table S10). Figure 1 was re-exported from the final full-model checkpoint, and the corresponding long-horizon contact sheets were checked against the quantitative logs.

For clarity, the reported 0.2108 to 0.1850 Valid ADE improvement is relative to the strongest controlled baseline; the official Tora comparison is reported separately (0.2064 to 0.1850). Cross-backbone results in Table 3 report relative gains over the corresponding frozen baseline. We do not claim to outperform every official method on every metric: background consistency and motion smoothness remain backbone- and protocol-dependent, and these limitations are discussed in Appendix D.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.