acceptodds
Under review as a conference paper at ICLR 2027

Force-Conditioned Video Generation by Decoupling Motion-State Prediction and Video Synthesis

Abstract

Interacting with the physical world using video diffusion models requires modeling how objects move under force control. Existing methods often map force controls directly to video. Without an explicit motion target, they learn force responses only through per-frame visual appearance. Appearance-based prediction may fail to capture the underlying physics, as different motion directions or strengths can appear similar at small motion amplitudes. Consequently, a generated video may look realistic without following the intended control. To improve physical plausibility and controllability, we propose FoMo, a generation scheme that separates motion-state prediction from video synthesis. Specifically, given an initial frame, relative force control, and initial velocity, instead of directly synthesizing the output video, FoMo first predicts initial-frame-indexed velocity and visibility maps. Velocity explicitly represents direction and magnitude, making motion changes direct training targets. In the next step, FoMo integrates the velocities into trajectories and passes them with the visibility maps to a pretrained trajectory-conditioned video generator. This design has two advantages: (1) direct motion supervision and (2) a generator focused only on visual synthesis. We additionally derive relative 2D controls from motion changes in real videos and use semantic physical priors for context. These controls do not represent exact 3D forces. Controlled ablations support explicit velocity prediction and real-video training. In a pairwise study with 100 participants, FoMo is preferred over Goal Force fine-tuned on the same training dataset.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Generation, Motion Generation, Force-conditioned

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_jpAhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present FoMo, a force conditioned video generation approach that consist of two stages. First, given an initial image, a 2D force, initial-velocity controls, and a prompt, FoMo predicts velocity and visibility maps. It integrates the velocities into trajectories that guide an adapted video generator to create the final video. In my opinion the paper contributions are the motion-first formulation, and the motion-focused losses, the authors validate their method with motion ablations and a human preference study, showing the preference for FoMo over Goal Force.

Critical strengths

Strengths:

1- I think that the use of an intermediate representation is a reasonable idea. Separating motion prediction from rendering provides an explicit target for learning motion direction and magnitude, keeping the appearance out of the prediction target and leaving the strong pretrained renderer to handle the visual part. This representation is well motivated with supporting ablations.

2 - Human preference for FoMo over Goal Force which was trained on the same data goes from 67.5% to 94.5% across different scenarios like collisions, rebounds, slide, roll, multi-object etc.

Critical weaknesses

Weaknesses:

1- In my opinion the paper should show whether changing the force’s direction or strength produces the expected change in motion. The paper does not systematically test different forces on the same scene, therefore the experiments do not clearly show how reliably the model follows the force inputs, in this case i think that these tests matter because the model should generate motion that follows the user’s controls, rather than simply motion that looks plausible.

2- My other doubts are in Appendix, so Appendix I explains what counts as a successful interaction, but not who judged success in this case. On the other hand, although the baseline uses the same training data, it has a different architecture and receives initial-velocity information differently. This makes it difficult to tell how much of the improvement comes from predicting motion before generating video. Together, these gaps make the reported improvements harder to assess confidently.

Questions for authors

Please refer to weaknesses too.

1 comment

We thank the reviewer's positive feedback and address the questions as below:

JP1 · Does changing control direction or magnitude in the same scene produce the expected motion?

Response

Yes. Fixed-scene control edits confirm that FoMo responds consistently to changes in both control direction and magnitude (revised Fig. 5, Sec. 4.5):

  • Direction and magnitude under a fixed prompt. With the baseline prompt held constant, reversing the ball's initial velocity shifts its first shown position left, and increasing control strength moves both the ball and the car out of view sooner.
  • Force-direction edit and prompt alignment. With the edited upward-force vector, the car lifts off even with no prompt; retaining the stale baseline prompt keeps the car on the table due to control-text conflict, whereas updating to the refreshed prompt yields smooth lift in later frames.

JP2 · Who judged interaction success in Appendix I?

Response

Three human annotators independently evaluated whether each interaction met the success criteria in revised Appendix I. Annotators reviewed each video at 0.25× playback speed (4× slow motion) and independently reassessed clips with initial disagreements (only 3 samples) until all three agreed.


JP3 · How much of the matched-data result can be attributed to predicting motion first?

Response

Three experimental controls isolate the contribution of explicit motion prediction from backbone capacity and input interface differences (Sec. 4.2, Table 2, Appendix L):

  1. Comparable 14B Wan visual backbones. Goal Force is built on Wan2.2-A14B-I2V (via a ControlNet branch), and our Stage 2 renderer (Wan-Move) likewise uses a 14B-class Wan image-to-video backbone, whereas FoMo's Stage 1 predictor uses a 1.3B backbone (Wan2.1-Fun-1.3B-InP). Under matched training data and comparable rendering capacity, FoMo's 67.52%–94.52% preference rates in Table 1 reflect the benefit of decoupling motion prediction rather than a stronger visual backbone.
  2. Controlling for initial-velocity conditioning. Because Goal Force lacks a native vector channel for initial velocity, both models receive the exact same control-aware semantic prompt encoding initial velocity and force context, while FoMo additionally receives initial velocity as a sparse vector map (Sec. 4.2, Appendix L).
  3. Same-backbone motion target ablation (Table 2). Within the exact same 1.3B Stage 1 backbone and identical sparse vector inputs, supervising explicit velocity maps reduces trajectory error from 2.317 (direct trajectory target) to 1.662 and raises interaction success from 87.31% to 97.31%.
Reviewer_4SWBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes FoMo, a two-stage method for force-conditioned video generation. Given an initial frame, relative 2D force and initial-velocity controls, and a semantic prompt, FoMo first predicts velocity and visibility maps indexed by points in the initial frame. It integrates the velocities into trajectories and uses them to guide a separate video generator. the motion predictor is trained with both synthetic and real videos. The paper reports improved motion metrics and human preference over several baselines.

Critical strengths

The two-stage approach is reasonable. Predicting an explicit motion representation provides a direct training target for the motion predictor, while the separate video generator can focus on rendering the predicted motion. Using real videos to supervise motion prediction may also broaden the range of objects and motions covered by the training data. The force and initial-velocity controls provide a direct way to specify how an object should move.

The qualitative examples in the supplementary video appear to show smoother motion and better control following than the Goal Force results.

Critical weaknesses

The overall two-stage structure is not entirely novel. FloVD (Jin et al., CVPR 2025) also predicts an intermediate motion representation with a diffusion model and passes it to a separate motion-conditioned video generator. Although FloVD addresses camera-controlled video synthesis rather than force-conditioned generation, the structural similarity is substantial. The paper does not cite or discuss this work. The authors should compare FoMo with FloVD and clarify what is novel beyond applying a similar motion-prediction and video-synthesis framework to force-conditioned generation.

  • Jin et al., FloVD: Optical Flow Meets Video Diffusion Model for Enhanced Camera-Controlled Video Synthesis, CVPR 2025

The paper explains that visibility maps allow the video generator to ignore points that become occluded or unreliable, but it does not show why these maps are necessary or how much they improve the result. There is no ablation comparing generation with and without visibility maps. An ablation or a more direct analysis would clarify the contribution of this component.

Despite the superior quality of the results compared to those of Goal Force, they still look rather simple compared to videos generated by recent state-of-the-art video generation models. As the recent general video generation models show amazing results, I would like to see more detailed discussions about this.

The proposed method seems to be limited to rather simple scenes with only a couple of objects and a fixed camera. I wonder whether the proposed method can handle camera motion or complex scenes with multiple objects, or how it can be extended.

I also wonder whether the proposed method requires an explicit force or initial-velocity signal to generate object motion. For example, can it generate plausible motion for an object that is not assigned any motion control, or does it only animate objects in response to the provided controls? Please clarify the scope of the method on both points.

Questions for authors

Please address the concerns discussed above.

2 comments

We thank the reviewer for the constructive feedback and address each point below:

SW1 · What does FoMo contribute beyond FloVD, how does it handle camera motion, and how could it be extended?

Response

FloVD and FoMo address orthogonal control dimensions, namely prescribed camera-viewpoint synthesis versus force-driven object dynamics, and their representations are complementary (revised Sec. 2.1, lines 129–131):

  • Distinct control and motion representations. FloVD combines 3D-warped camera flow with unconditioned adjacent-frame object flow for viewpoint control. In contrast, FoMo models how objects respond to explicit 2D force and initial-velocity controls by predicting initial-frame-indexed velocity and visibility maps, which prevents temporal drift and reduces trajectory error from 2.317 to 1.662 within the same backbone (Table 2).
  • Camera motion and extension. To avoid conflating camera ego-motion with object-level force responses, we filtered real training videos for static cameras (Sec. 4.1). Incorporating FloVD's camera-flow decomposition provides a direct path to extending FoMo to moving-camera scenes (Sec. 5).

SW2 · Do visibility maps improve rendered videos?

Response

Yes. Explicit visibility conditioning prevents occlusion-induced ghosting and spurious motion propagation to uninvolved objects (revised Sec. 4.5 and Fig. 6). Holding the initial frame, reference trajectories, prompt, and sampling noise identical, replacing reference visibility with a naive all-visible (in-frame) mask degrades visual fidelity in two distinct ways:

  • Suppressing occlusion artifacts (Fig. 6a). Visibility maps mask out the occluded red ring as it passes behind the sphere, whereas the in-frame mask forces the renderer to composite a red ring-shaped artifact onto the sphere's surface.
  • Preventing false motion leakage (Fig. 6b). Visibility conditioning keeps the uninvolved far red domino upright, whereas the in-frame mask induces spurious motion and causes it to fall.

SW3 · How should we interpret the visual gap relative to general video generators?

Response

General video generators prioritize open-domain visual diversity at the expense of fine-grained physical controllability, whereas FoMo achieves both high visual preference on physical interactions and modular compatibility with stronger renderers:

  • Superior visual quality on physical benchmarks. Across all evaluated interaction categories, human raters preferred FoMo's visual quality over prompt-guided Wan2.2-A14B-I2V (82.98%–98.30%) and same-data Goal Force (67.52%–90.88%) (Table 1, Sec. 4.2).
  • Decoupled scalability. Because Stage 1 predicts explicit trajectories independent of pixel synthesis, FoMo's motion predictions can directly drive stronger trajectory-conditioned video backbones as they become available (Sec. 3.4).

SW4 · What evidence covers complex or multi-object scenes?

Response

Both quantitative human evaluations and qualitative rollouts confirm that FoMo reliably models multi-object collisions and chained momentum transfer (Sec. 4.2, Fig. 1, Appendix D). On multi-object test clips involving colliding blocks, dominoes, and Newton's cradle, 100 participants preferred FoMo over same-data Goal Force by a wide margin across all metrics (Table 1):

  • Control consistency: 92.77%
  • Visible physical plausibility: 94.52%
  • Visual quality: 90.88%

SW5 · Can objects move without receiving their own control vectors?

Response

Yes. FoMo does not require per-object control arrows to animate secondary objects, while still preventing unintended background drift:

  • Impulse propagation via dense motion fields. Because Stage 1 predicts a dense velocity field across all grids conditioned on sparse controls and scene context, contact naturally transfers momentum to unprompted objects, such as the ball atop the toy car and the opposite spheres in Newton's cradle (Fig. 1).
  • Operational scope boundary. Conversely, for stationary regions without physical contact or control triggers, the motion mask (MfiltM_{\text{filt}}) suppresses residual velocity noise so scenes remain stable rather than drifting spontaneously (Sec. 3.3–3.4).

I appreciate the authors' response, which has addressed most of my concerns. My only remaining concern is that Fig. 6 has only two examples. I would like to ask the authors to include quantitative analysis if possible. I will raise my score to weak accept.

Reviewer_jNCFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes to add separate motion-state prediction for force-to-video generation. Given an initial frame, relative force control, and initial velocity, it predicts the velocity and visibility maps. Then, it transforms the velocities into trajectories, which are later used for guiding video generation.

Critical strengths

1, The explicit motion-state formulation is well motivated. It formulates the motion states of the video as velocity and visibility maps. During inference, it first predicts the motion states and then renders the video. It provides a meaningful intermediate target for learning motion responses.

2, the framework does not require metric 3D physical parameters or full 3D reconstruction. The ablations also show that real-video training data contributes substantially to motion prediction performance.

Critical weaknesses

1, The interpretation of FrelF^{rel} as a force condition is somewhat questionable. In real videos, the second-order image-space displacement can arise from gravity, friction, collisions, camera motion, depth changes, or other unobserved causes. The proposed relative force appears to be an observed image-space motion change rather than an actual external force. Therefore, it is unclear to what extent the model learns force-conditioned dynamics versus simply extrapolating future motion from an initial change in motion.

2, Potential motion information leakage from the VLM-generated semantic prior. A strong VLM may infer plausible object motion directly from the input image, even without access to the force guidance. Therefore, it is unclear how much of the final motion is determined by the explicit force condition versus the motion prior already encoded in the VLM-generated text. An image-only VLM prior would be a useful baseline to disentangle these two effects.

3, What is the upper bound of the proposed method when using ground-truth trajectories? An oracle-trajectory experiment would help disentangle failures caused by motion prediction from those caused by the trajectory-conditioned renderer.

4, The 2D arrow representation cannot uniquely specify a 3D force, limiting the method to image-space control rather than general force-conditioned dynamics.

5, The method requires additional supervision for intermediate motion states, increasing the cost of data collection and processing.

Questions for authors

See weakness.

1 comment

We appreciate the reviewer's positive feedback and address the questions as follows:

JN1 · Does the relative control represent an external force or an observed motion change? What can the 2D arrow specify about a 3D force?

Response

Our 2D control approximates the effect of a 3D force using an observed relative motion change in 2D image space (revised Secs. 2.2, 3.1–3.2, and 5). While this 2D representation cannot recover calibrated 3D forces or disambiguate depth, scale, and hidden contacts, it avoids 3D scene reconstruction and metric physical annotations, making force-conditioned control more straightforward and broadly applicable across diverse real-world scenes.

  • Training construction. As defined in Eq. (4), Frel:=X2−2X1+X0F^{\text{rel}} := X_2 - 2X_1 + X_0 measures the initial second-order image-plane displacement change from tracked positions. Because monocular projection, depth, scale, and unobserved contacts can map distinct 3D forces to the same 2D observation, FrelF^{\text{rel}} serves as a 2D motion-change proxy rather than a metric 3D force.
  • Inference semantics and practical scope. At test time, the 2D arrow specifies the screen-space direction and relative magnitude of the intended motion change. Operating directly in 2D avoids the fragility of monocular 3D reconstruction and per-object physical parameter estimation, allowing FoMo to learn from ordinary videos and generalize across varied objects and scenes.

JN2 · Could the VLM text prior account for the observed motion response?

Response

The VLM text prior does not by itself account for the observed motion response; rather, the explicit 2D control vectors drive motion direction and actuation, while the semantic prior supplies complementary physical context (revised Sec. 4.5 and Fig. 5).

  • Fixed prompt. Increasing control magnitudes under a fixed prompt moves both the ball and the toy car farther across the scene, showing direct sensitivity to vector magnitude without prompt changes.
  • No prompt. Without any text prompt, the toy car still responds directly to the edited upward-force vector and lifts off the table, confirming that actuation comes from the vector control rather than text leakage.
  • Baseline and refreshed prompts. Under the same edited controls, retaining the stale baseline prompt creates a control-text conflict that keeps the car on the table, whereas updating to the refreshed prompt aligns the physical context and yields smooth lift in later frames.

JN3 · What is the upper bound of the proposed method when using ground-truth trajectories?

Response

Replacing predicted motion states with oracle ground-truth trajectories and visibility masks reduces the fixed renderer's trajectory EPE from 15.90 px to 6.52 px across 11,880 visible object point-frames (480×480480 \times 480 grid), establishing the performance upper bound under ground-truth motion guidance (revised Table 8, Appendix M).

  • Predicted tracks and visibility. 15.90 px mean trajectory EPE against reference object tracks.
  • Oracle reference tracks and visibility. 6.52 px mean trajectory EPE, isolating the remaining rendering error when motion prediction error is removed.

JN4 · What additional cost does intermediate motion supervision introduce?

Response

All intermediate motion labels are extracted offline during dataset preprocessing and introduce zero test-time overhead (revised Table 10, Appendix M). On a single A100 GPU, per-clip preprocessing averages 1.26 s for point tracking and visibility (CoTracker3), 5.75 s for object segmentation and grouping (SAM2.1), and 0.369 ms on the CPU for control-vector selection. Prompt generation is excluded because external GPT call latency is not directly comparable across methods.

Reviewer_AL9VOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose FoMo, a two-stage approach to force-style video control. It predicts velocity and visibility maps from an initial image and relative controls, integrates the velocities into trajectories, and uses them to condition video synthesis. Real-video supervision broadens the training data. Representation ablations, a same-data baseline, and a human preference study evaluate the approach.

Critical strengths
  1. Predicting motion explicitly gives the method a clear intermediate target. Table 2 compares velocity, acceleration, and trajectory targets within the same backbone, giving a clearer account of the target choice than a complete-system comparison alone.
  2. Fine-tuning Goal Force on the same data addresses an important confound. The 100-participant study evaluates the final videos, and the paper clearly limits its claims to relative 2D control and visible plausibility.
Critical weaknesses
  1. Control sensitivity is not tested directly enough. Equation 4 FrelF^{rel} = X2X_2 -2$X_1$ + X0X_0 derives from observed motion, so accurate prediction under these retrospective labels says little about the response to a different control on the same scene. The missing test is a direction/magnitude sweep with the initial image, velocity, and seed fixed. The semantic prompt should also stay fixed: otherwise, language guidance may steer the motion along with the control vector.
  2. The error between predicted motion and rendered motion is unmeasured. Table 2 evaluates predicted states, while the renderer can deviate from the supplied trajectories. The human study evaluates the complete videos, without quantifying how closely their motion follows those trajectories. The control claim ultimately depends on the motion in the final video.
Questions for authors
  1. How does motion change when control direction or magnitude changes? Does this hold with a fixed semantic prompt?
  2. What is the complete two-stage inference cost comparing with previous one-stage methods?
1 comment

We appreciate the reviewer's positive feedback and address the questions as follows:

AL1 · How does motion respond to a change in direction or magnitude under a fixed prompt?

Response

Varying the control vectors while holding the initial frame and baseline semantic prompt fixed produces predictable, physically consistent changes in both speed and direction (Sec. 4.5, revised Fig. 5):

  • Control magnitude. Increasing control strength accelerates object motion along the intended direction, moving both the falling ball and the toy car out of view earlier in the sequence.
  • Direction reversal. Reversing the ball's initial velocity vector shifts its first shown position and subsequent trajectory to the left.

AL2 · How closely does the rendered video follow the supplied trajectories?

Response

Our renderer builds upon Wan-Move, a state-of-the-art trajectory-guided video generator with strong native motion consistency, which we further adapt to physical object interactions via Orthogonal Fine-Tuning (OFT): Rendered videos adhere closely to the supplied trajectories, achieving 3.73 px mean EPE under reference tracks and 5.28 px under FoMo's predicted tracks across 185 clips at 480×480480 \times 480 resolution (revised Table 8, Appendix M).

  • Effect of OFT adaptation. Under identical reference trajectories, OFT reduces trajectory EPE over visible object points and background anchors from 14.79 px to 3.73 px, and full-frame RAFT optical flow EPE from 1.562 px to 0.158 px.
  • End-to-end adherence. When rendering FoMo's own predicted trajectories, the synthesized videos follow the supplied tracks with a mean trajectory EPE of 5.28 px.

AL3 · What is the total inference cost relative to a one-stage method?

Response

Despite using a two-stage pipeline, FoMo runs 34.4% faster overall (122.56 s vs. 186.79 s) and requires 13.8% fewer FLOPs than the one-stage Goal Force baseline (Gillman et al., CVPR 2026) on a single A100 GPU (BF16, batch size 1, 25 frames at 480×480480 \times 480) (revised Table 9, Appendix M):

MethodMotion Prediction (Time / PFLOPs)RGB Generation (Time / PFLOPs)Total Median TimeTotal PFLOPs
Goal Force (1-stage)N/A186.79 s / 19.057 (40 steps)186.79 s19.057
FoMo (2-stage)9.74 s / 1.171 (50 steps)112.82 s / 15.262 (40 steps)122.56 s (-34.4%)16.433 (-13.8%)

Because Stage 1 utilizes a lightweight backbone (1.3B model) and operates at a relatively small resolution (256×256256\times 256), motion prediction takes only 9.74 s (about 8% of total runtime).

Comments on the paper

2 comments

Dear Reviewers and Area Chairs,

Thank you for taking the time to review our paper and for your thoughtful feedback. We have revised and polished both the manuscript and the supplementary material. In both PDFs, text added or revised during the rebuttal period appears in blue.

If you have any further questions or concerns, please feel free to reach out through the OpenReview discussion. We would be happy to clarify.

Best regards, The Authors

Area_Chair_TAfxOpenReview ↗

Discussion Period is About to Finish

Dear reviewers,

Thank you for your reviews and constructive feedback.

The author-reviewer discussion period will be finishing soon. Please take a moment to review it and consider whether your concerns have been adequately addressed. It would be helpful if you could briefly indicate:

  • Whether the rebuttal addresses your main concerns;

  • Whether any significant issues remain unresolved;

  • Whether your overall recommendation has changed.

If appropriate, please also update your review and recommendation in the system.

Thank you for your time and valuable contributions.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.