- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a framework for the chunk-based generation of long videos, combining a Motion-Instance Graph (MIG) with a persistent Instance Memory (IM):
MIG: Organizes subjects, static regions, trajectories, interactions, and event states into a typed graph, which is then converted by a DMG Planner into 9-channel frame-level motion and spatial conditions. IM: Performs top-k retrieval based on instance identity before the generation of each chunk, and updates the memory with observed results following generation. Injection Mechanism: Employs an MI-Adapter and LoRA to inject information into a frozen CogVideoX-2B model, with 14.87M (0.86%) trainable parameters; the framework also incorporates first-frame anchoring and frame-aware corruption.
Critical strengths
-
Clear problem definition: The generation of long-form video is framed as the maintenance of a "world state" across chunks, replacing implicit KV history with an explicit, inspectable instance state—offering both interpretability and editability.
-
Parameter-efficient with low overhead: Only ~1% of parameters are trained; inference time increases by approximately 12.5%, and peak VRAM usage rises from 16 GiB to 19 GiB.
-
Well-structured evaluation scenarios: The seven stress-test scenarios in Table S2 cover complex cases such as large-scale displacement, reappearance after occlusion, and multi-entity interaction. The evaluation is further supported by a progressive ablation study (A0–A8), transfer experiments across three different backbones, and blind human evaluations.
-
Transparent discussion of limitations: The authors candidly acknowledge that errors in parsing or association can be written into the memory and subsequently propagated.
Critical weaknesses
-
The "baseline" mentioned is not an established, previously published method. The configurations labeled "Layout/BBox," "Trajectory," and "History-extension" in Table 1 are custom setups devised by the authors using CogVideoX-2B (Table S3 merely lists "design references"), with no information provided regarding training data, number of steps, or implementation details. Furthermore, the paper lacks direct comparisons with any official implementations of existing methods, such as Tora, MotionCtrl, DragNUWA, Boximator, FreeNoise, StreamingT2V, or LongLive.
-
The novelty is questionable, and there are significant omissions in the related work. The paper’s approach—using an LLM to plan layouts or trajectories to guide video generation—overlaps substantially with existing research (e.g., LVD, VideoDirectorGPT, and DreamRunner), yet these are neither cited nor discussed. Additionally, the discussion on methods for long-video memory (such as WorldMem, Context-as-Memory, and FramePack) is insufficient.
-
The evaluation process is circular and unfair. ADE/FDE metrics measure deviations from the system’s own planned target trajectories, yet the "frozen base" model is not provided with any trajectory inputs. The writing of data into the memory relies on detection and association; since the evaluation also employs the same YOLOv7-based association protocol, the data-writing process and the scoring mechanism are coupled.
4 . The methodology is described vaguely, making it difficult to reproduce. The paper fails to specify what kind of model the "SMR Parser" is. It does not explain how the "DMG Planner" generates specific trajectories from text. It does not describe how the key/value pairs for the memory are extracted from the generated videos. There is no algorithmic description for conflict resolution or history compression. The source, annotation method, and licensing of the 96,000 training videos are not disclosed, nor is there any mention of code availability.
-
Data consistency and credibility are questionable. In Table S6, the incremental improvements from A1 to A8 are suspiciously smooth and monotonic; the unit for "Detection ∆ +86.92" is undefined. In Table S9, the numerical intervals are overly uniform (0.990/0.970/0.950; 0.450/0.510/0.550), and the "mean jump" actually decreases as the length increases. The values for the "full model" are inconsistent between Table 1 and Table 2 (DINO: 0.9580 vs. 0.9583; ADE: 0.1850 vs. 0.1851). A "memory scale" factor of only 8×10⁻⁵ leads to significant performance gains, yet this is left unexplained. The appendix mentions that a grid search for hyperparameters was conducted, but no results are provided.
-
Insufficient statistics and human evaluation. Although three seeds were used, no variance or significance metrics are reported. The human evaluation involved only three unidentified raters and lacks inter-rater reliability coefficients; furthermore, comparisons were made exclusively against a "frozen" baseline rather than the strongest available baseline.
-
The experiments on long-form videos lack proper controls. For the 240/720/1440-frame settings, the authors only report diagnostic metrics—such as detection coverage—for the full model itself, with no comparisons to baselines or standard long-video evaluation metrics. Additionally, the backbone model used is outdated (CogVideoX-2B, 720×480).
Questions for authors
-
How are the three control configurations in Table 1 implemented and trained? Are the training data, number of steps, and input conditions consistent with those used for the method proposed in this paper? Could you provide a direct comparison with the official implementations of Tora, MotionCtrl, StreamingT2V, LongLive, etc.?
-
When calculating ADE/FDE, what serves as the "target trajectory" for the frozen baseline that lacks trajectory input? If all methods are provided with the same trajectory conditions, how significant is the remaining performance gap?
-
What are the specific implementations of the SMR Parser and DMG Planner (e.g., based on an LLM, rule-based logic, or a learned model)? What encoder is used to extract the key/value pairs for the memory?
-
Is the detector used for IM updates the same as the YOLOv7 model used for evaluation? If an independent detector (such as GroundingDINO + SAM2) were used for evaluation, would the results remain consistent?
-
Could you provide the mean ± standard deviation across 3 seeds, along with the results of significance tests? Additionally, could you specify the background of the human evaluators and the inter-rater reliability coefficients (e.g., Krippendorff’s α)?
-
Could you explain the discrepancies in the numerical values between Table 1 and Table 2, the definition of "Detection ∆" in Table S6, and why the "mean jump" metric decreases as the sequence length increases? Would it be possible to release the raw, per-video results?
-
Given that the "memory scale" is only 8×10⁻⁵, how does the IM (Internal Memory) mechanism exert a significant influence? Could you provide a plot illustrating the relationship between the memory scale and performance (i.e., the results from the hyperparameter grid search mentioned in the appendix)?
-
How do the baseline models and other existing methods for long-video generation perform in the experiments involving 240, 720, and 1440 frames? Could you report metrics such as those from VBench-Long or drift-related evaluations?
-
What are the fundamental differences and advantages of MIG compared to other approaches—such as LVD, VideoDirectorGPT, and DreamRunner—that utilize LLMs for planning in conjunction with video generation?
Response to Reviewer CksM
Thank you for the review. We revised baseline comparisons, evaluation, reproducibility, statistics, and long-horizon experiments. Our responses follow.
W1 — Baseline fairness and official comparisons. Our controlled baselines share CogVideoX-2B, 86,400 training videos, 6,000 AdamW updates (4,000+2,000), batch 16, LR 2.5e-5, and LoRA rank 32. Trainable parameters are 14.21M–14.52M versus 14.87M for ours; training costs are 51.4–52.8 versus 54.1 A100 GPU-hours (Sec. 4.1, Table S3). We also evaluated the released implementations/checkpoints of Tora, MotionCtrl, DragNUWA, Boximator, FreeNoise, StreamingT2V, and LongLive using their supported native configurations and a common evaluation benchmark (Tables 1, S8b). Tracking and trajectory metrics favor ours; background/motion results vary by backbone.
W2 — Novelty and related work. Sec. 2 now discusses LVD, VideoDirectorGPT, DreamRunner, FramePack, Context-as-Memory, and WorldMem. Unlike upfront motion scripts and frame/viewpoint-indexed context, MIG–IM associates appearance, trajectories, interactions, and event states with explicit instance identities, then corrects the next chunk's graph and memory from observations of the generated chunk. An A8 open-loop ablation isolates this feedback (Table 2).
W3 — Trajectory and detection fairness. The target trajectory is annotated from each held-out source video, not produced by DMG. The frozen model receives no trajectory and is reported as an unconditional reference; trajectory-conditioned methods share the same reference trajectory. Valid ADE falls from 0.2108 (Trajectory control) and 0.2064 (official Tora) to 0.1850 (ours). Online write-back uses YOLOv7+ByteTrack; independent offline GroundingDINO+SAM 2 rescoring preserves the main relative trends (Secs. 4.1–4.2; Tables 1, S4b, S8b).
W4 — Reproducibility. Secs. 3.2–3.5 and Apps. A.1–A.3 specify the GroundingDINO/SAM 2 + Qwen2.5-VL SMR parser, cubic-Hermite DMG planner and nine channels, DINOv2/Fourier/MLP memory features, 256-D key/value projections, Hungarian matching, confidence gating (0.50), anchored EMA, occlusion handling, and 64-record compression (Eqs. S3–S8; Algorithm 1). The 96,000-video corpus now has sources (OpenVid-1M/Panda-70M), splits, deduplication, and annotation details.
W5 — Numerical and metric consistency. Tables 1 and 2 now share direct 480-video aggregation rather than the earlier rounded intermediate averages. Table S6 defines Detection Delta as relative association-rate gain (51.40% to 96.08%, +86.92%) and Coverage Delta as absolute full-track completion gain (0.350 to 0.920, +0.570). The original detected-seam jump excluded detector dropouts; all-seam jump now includes them and rises 4.31/4.58/4.89 px over 240/720/1,440 frames (Table S9). Norm analysis explains memory scale 8e-5 (3.40% residual); Table S6b/Fig. S2 report the sweep.
W6 — Statistics and human study. Primary controlled comparisons and ablations now report three-seed mean ± SD. Paired Wilcoxon/t-tests use 160 condition-level observations. The blind study was expanded to eight computer-vision researchers, with additional baseline matchups, inter-rater agreement, and paired tests (Sec. 4.4; Tables 4, S10).
W7 — Long-horizon evidence and scaling. Figure 6 and Table S9b compare controlled baselines and supported official long-video methods at 240, 720, and 1,440 frames using drift diagnostics and five VBench-Long metrics. At 1,440 frames, ours achieves 95.1% detection coverage, 0.9382 subject consistency, and 0.9415 background consistency. CogVideoX-5B adds a fourth paired-backbone evaluation with 0.62% trainable parameters (Table 3, S8).
Q1. Training budgets/capacities: Table S3; official implementations and protocols: Tables 1, S8b. Q2. The frozen model is assessed against the held-out ground-truth trajectory only as an unconditioned reference. Shared-trajectory comparisons against Trajectory control and Tora remain favorable (W3). Q3. Parser, planner, and memory encoder are specified in Secs. 3.3–3.5 and Apps. A.2–A.3 (W4). Q4. Online YOLOv7+ByteTrack and independent offline GroundingDINO+SAM 2 are distinguished; both are reported in Tables 1, S4b, S8b. Q5. Three-seed variability, paired tests, eight-rater backgrounds, and agreement are reported in Sec. 4.4 and Table S10. Q6. Aggregation differences, delta definitions, detected/all-seam jumps, and the plan to release per-video logs with code are addressed in W5 and Tables S6, S9. Q7. Norm-ratio explanation: Eq. S2; memory-scale grid: Table S6b, Fig. S2. Q8. Multi-horizon baseline and VBench-Long results: Fig. 6, Table S9b. Q9. Closed-loop re-anchoring differs from upfront LLM planning (W2). Turning off write-back changes association 96.08%→84.65%, DINO 0.9580→0.9504, and ADE 0.1850→0.2008 (Table 2).