acceptodds
Under review as a conference paper at ICLR 2027

ForceVid3D: Physical Video Generation with Camera-Space 3D Force Control

Abstract

Force-controlled video generation allows users to specify external forces while the model generates scene-dependent responses. Existing direct force-control methods typically rely on image-plane 2D representations, which do not fully express camera-space 3D forces, while 2D observations provide limited cues about responsive regions and their local geometry. We introduce **ForceVid3D**, a framework for generating videos from an initial image and an explicit camera-space 3D force. Our Projective 3D Force Map exposes the image-space geometry of camera-space 3D forces by combining force support, direction, and magnitude with projection cues. We further introduce Depth-Grounded Response Affinity Supervision (D-RAS), which combines response-affinity and local relative-depth objectives to capture responsive regions and their geometric context. To support learning and evaluation across diverse physical responses, we construct ForceVid3D-Dataset, a 60K-pair force–video dataset spanning seven response groups, and introduce ForceVid3D-Bench for force-to-response evaluation. Under comparable 2D force-control settings, ForceVid3D achieves consistent improvements in force adherence and physical consistency over prior force-conditioned methods while maintaining competitive visual quality. Dedicated 3D evaluations further demonstrate camera-space 3D force control, while controlled ablations support the contributions of the projective representation and D-RAS.

Primary area:
generative models
Keywords:
Video Generation, Controllable Generation, 3D Force Control, Physical Dynamics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_K4JoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ForceVid3D, a framework for generating videos conditioned on an initial image, text, and a camera-space 3D external force. The method uses a Projective 3D Force Map and Depth-Grounded Response Affinity Supervision (D-RAS). The authors also construct a dataset of approximately 60K force–video pairs and an accompanying evaluation benchmark. Experiments demonstrate strong force adherence and physical consistency in the 2D control setting, and further show the potential of camera-space 3D force control.

Critical strengths

1.The problem is practically relevant. The work extends direct force control to camera-space 3D directions and supports different types of interventions, including point forces and wind.

2.The method is clearly designed, addressing both the projection geometry of forces and the geometric context of responsive regions. The auxiliary supervision modules are removed at inference.

3.The dataset and experiments cover a broad range of physical responses and include baseline comparisons, controlled ablations, and failure case analysis. The overall results in the 2D control setting are convincing.

Critical weaknesses

1.The individual contributions of the two supervision objectives in D-RAS remain unclear. The existing ablations support the effectiveness of the joint supervision, but do not include variants using only response-affinity supervision or only relative-depth supervision, making it difficult to assess their individual contributions and complementarity.

2.The baselines are not fine-tuned on the same training data, making it difficult to disentangle the performance differences across methods from the effects of training data and other system differences. Adapting a representative baseline to the same dataset would help clarify the relative gains of the proposed method.

3.The paper does not systematically evaluate how the generated responses vary with force magnitude and duration. When force magnitude and duration are varied separately, while holding the scene and object properties fixed, do the generated responses follow the same trends as those observed in simulation?

Questions for authors

see weaknesses

4 comments
AuthorsOpenReview ↗

Feedback to Reviewer K4Jo

Thank you for your constructive suggestions. We have added individual D-RAS ablations, a matched-backbone retrained baseline, and a controlled force magnitude–duration study.

Q1: Individual and joint contributions of D-RAS supervision

A1: We compare four matched variants, No D-RAS, Affinity-only, Depth-only, and Full D-RAS. Within each independent training repeat, all branches continue from the same 8K SFT initialization to 11K. Automatic results average two runs and three inference seeds per checkpoint; human ratings follow Appendix D.3.

D-RAS ablations — Appendix H.2 and H.4, Tables 15 and 17 (expanded multi-seed counterparts of main-text Tables 2 and 4, respectively)

Objective2D F-Align ↑2D M+ IoU ↑2D H-Physics ↑3D Dir.@45 ↑3D R-Act. IoU ↑P-ADE ↓3D H-Physics ↑
No D-RAS79.9482.2465.1256.1791.490.78177.17
Affinity-only81.2982.8067.0463.3393.020.77177.83
Depth-only81.8082.3167.3161.6792.630.72678.17
Full D-RAS (Ours)81.9282.2868.0065.6794.230.71478.50

Response-affinity supervision associates initial-frame surfaces with their subsequent force-induced responses, helping ground the response in the relevant scene content. Affinity-only achieves the highest 2D M+ IoU (82.80) and improves 3D directional adherence. Relative-depth supervision provides local front–back geometry around responsive surfaces; Depth-only improves trajectory agreement, reducing P-ADE from 0.781 to 0.726. Each objective alone improves H-Physics in both 2D and 3D over No D-RAS. Combining them achieves the best 2D H-Physics (68.00), 3D H-Physics (78.50), 3D Dir.@45 (65.67), R-Act. IoU (94.23), and P-ADE (0.714) among the four variants, while maintaining comparable 2D M+ IoU (82.28). These results support complementary benefits in response grounding, geometric coherence, and physical consistency.

Q2: Fair retraining of the Force Prompting baseline

A2: We retrain Force Prompting (Projected) on ForceVid3D-Dataset with Wan2.2-TI2V-5B for 15K steps, matching ForceVid3D's overall training settings. We construct 2D conditions from the camera-plane XY components of each 3D force, while retaining Force Prompting's moving Gaussian for Point and three-channel magnitude–direction encoding for Wind.

Matched 15K comparison — Appendix H.1, Table 14 (expanded multi-seed counterpart of main-text Table 1); Point + Wind (156 cases)

MethodF-Align ↑Proj.-Resp.@45 ↑M+ IoU ↑H-Force ↑H-Physics ↑
Force Prompting (Projected)79.4567.3182.2884.9459.72
ForceVid3D (Ours)83.8989.1083.1693.3274.20

Point only (90 cases)

MethodF-Align ↑Proj.-Resp.@45 ↑M+ IoU ↑H-Force ↑H-Physics ↑
Force Prompting (Projected)73.7674.4474.0885.9351.20
ForceVid3D (Ours)79.0293.1574.2994.2671.20

ForceVid3D improves Proj.-Resp.@45 by 21.79 points on Point + Wind and 18.71 on Point only; H-Physics improves by 14.48 and 20.00 points. M+ IoU on Point + Wind rises from 82.28 to 83.16, while visual quality remains comparable (full results in Appendix H.1, Table 14; expanded multi-seed counterpart of main-text Table 1). Figure 5 shows qualitative comparisons.

AuthorsOpenReview ↗

Feedback to Reviewer K4Jo

Q3: Independent magnitude–duration control experiments

A3: We test five normalized force magnitudes (0.2–1.0) and three durations (5, 9, 13 frames) in five Supported Rigid-Body scenes, with two force directions per scene. There are 15 conditions per scene–direction group and three inference seeds per condition, for 450 generated videos. Within each scene–direction–seed combination, we hold the image, prompt, force direction/location, camera, and object properties fixed.

We use SAM2.1 to measure signed endpoint displacement along the XY-projected force direction over the same 49-frame window. To reduce scene-scale dominance, the 15 conditions in each scene–direction–seed group share one maximum signed-displacement normalizer, and we average normalized responses over all 30 groups.

Mean normalized force-aligned displacement (dimensionless)

Duration (frames)m=0.2m=0.2m=0.4m=0.4m=0.6m=0.6m=0.8m=0.8m=1.0m=1.0
50.3330.4800.5800.6660.730
90.4490.6170.6730.7620.862
130.5460.6340.7880.8460.965

The mean response increases monotonically with magnitude and duration across the evaluated grid. Figure 8 presents the three response curves and qualitative examples.

Q4: Qualitative consistency with rigid-body simulation trends

A4: The observed monotonic changes in normalized displacement follow the qualitative force–response trends expected from rigid-body simulation under fixed scene conditions. Appendix H.4 (Table 17; expanded multi-seed counterpart of main-text Table 4) additionally evaluates trajectory agreement against paired simulator references. Together, these offer complementary evidence of physically meaningful force controllability, while the sensitivity study focuses on response trends rather than exact numerical matching of simulator trajectories.

Thanks for your response and the additional experiments. They address my concerns and support my initial assessment, so I will maintain my rating of 3

Thank you very much for taking the time to review our response and consider the additional experiments. We are glad that the added analyses and clarifications have addressed your concerns. We sincerely appreciate your constructive feedback and thoughtful assessment, which have helped us further improve the paper. Thank you again for your time and consideration!

Reviewer_kL6JOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents ForceVid3D, a framework for generating videos from an initial image, text, and explicit camera-space 3D forces. It introduces a Projective 3D Force Map that augments direct force encoding with perspective projection cues. A trainable control branch conditions a frozen Wan2.2-TI2V-5B backbone, while Depth-Grounded Response Affinity Supervision (D-RAS) uses training-only response-affinity and relative-depth objectives to associate force interventions with responsive scene content and local geometry. The authors also construct a 59,635-pair synthetic dataset covering seven response groups and ForceVid3D-Bench.

Critical strengths
  1. The authors propose a physically responsive video generation method with camera-space force control. The pipeline and dataset construction are relatively complete. The Projective 3D Force Map and D-RAS enhance 3D force conditioning and response-region association, respectively, and are integrated with the pretrained Wan DiT for direct video generation. The experiments cover multiple types of physical responses and corresponding ablation studies, demonstrating certain methodological novelty and application potential.

  2. The authors construct approximately 59K physical response videos with 3D Force annotations, covering seven types of scenarios, including rigid-body motion, collision, constrained motion, and wind-driven responses. The dataset involves considerable construction effort and research value, and provides a useful foundation for future physically controllable video generation.

Critical weaknesses
  1. The core contribution is 3D Force Control, but the experiments mainly rely on projected and normalized motion metrics and subjective evaluations, lacking sufficient validation of 3D motion and physically meaningful correlations between force and response. In addition, the physical units of force, the magnitude normalization strategy, and the duration sampling rules in the dataset remain unclear. Corresponding controllability experiments are also missing, making it difficult to determine whether the model has truly learned the relationship between force and physical response.

  2. D-RAS constructs Response Affinity from temporal appearance changes in videos, but does not sufficiently distinguish causal responses induced by external forces from independent motion in the scene. Meanwhile, independent ablations of Affinity Loss and Depth Loss, as well as dynamic-background tests, are missing, making it difficult to fully validate the effectiveness and generalization of this module in complex scenes.

Questions for authors

The following are some issues that need discussion and clarification, which I hope will help the authors improve the manuscript.

The authors first encode the normalized force magnitude and the force application interval into the 5C control condition, and further extend it with 2C projective channels. One interesting point is that, as the authors also acknowledge in Appendix G, the design does not guarantee compliance with Newtonian mechanics and does not explicitly recover mass, friction, or other latent physical parameters. However, the claim and motivation of "force-controlled" generation may lead readers to expect that, at least, the momentum increment (impulse) should be positively correlated with the product of the applied force and its duration, namely \Delta p = F * \Delta t. At a minimum, the authors should provide more details at appropriate locations about whether a force magnitude normalization strategy was designed during dataset construction (mass may also need to be considered, since momentum is related to mass, and the same impulse applied over the same duration causes a smaller velocity change for an object with a larger mass), and whether the force duration was aligned with the generated 10 FPS, 49-frame videos.

The manuscript does not explicitly explain what instructions were used when generating Prompts with the VLM, or whether motion-related or physical descriptions were prohibited. I strongly suggest that the authors clarify whether the Prompts contain motion-related terms such as "ejection", "pushing", "blown by wind", and "rolling", to demonstrate that text conditioning does not extensively leak the force-related representations that the model is expected to learn. If the Prompts contain motion descriptions, sufficient qualitative examples or quantitative ablation experiments should be provided for comparison.

I suggest that the authors further explain in Section 5 EXPERIMENTS, together with the validation data split and generalization cases, how the gradients from Flow Matching enable the 1.7B-parameter Controller to learn to convert Force into motion, rather than simply memorizing motion patterns in the videos, given I_0 and Force or Prompt as inputs and the complete GT Video as supervision.

For latent supervision, the authors choose Response Association rather than a frame-by-frame Motion Map. The illustration in Figure 2(c) may lead readers to misunderstand that the Affinity Map uses multi-frame, one-to-one supervision, while the Depth Map uses single-frame supervision. However, according to the main text and appendix, both are supervised with respect to the initial frame I_0. The authors should explicitly indicate in Figure 2 that the response information from multiple frames is mapped back to the corresponding object Masks in the initial frame.

In addition, the current Affinity Map construction considers the accumulated appearance changes within future ground-truth instance masks, meaning that secondary collisions between objects may also be recorded as indirect responses. However, the authors do not clarify whether dynamic backgrounds are included when preparing scenes in Blender. If dynamic backgrounds exist, it remains unclear how the Affinity Map handles spontaneous background motion that is completely unrelated to the external force, such as moving pedestrians, rotating machinery, or weather-related motion. If dynamic backgrounds are not included, the applicability of the method would be limited. The authors should further clarify the motivation, the latent supervision design, and the mechanism used to construct the Affinity training data.

5 comments
AuthorsOpenReview ↗

Feedback to Reviewer kL6J

Thank you for your detailed comments on physical consistency, force conditioning, and the D-RAS supervision design. We have expanded the evaluation and clarified the relevant training and data protocols.

Q1: Automatic evidence for camera-space 3D control

A1: We expand 3D automatic evaluation from 32 cases to all 154 camera-space 3D conditions (88 Point, 66 Wind), each with ∣dz∣>0.6|d_z|>0.6, including purely depth-directed forces (dx=dy=0d_x=d_y=0). Estimated 3D Dir.@45 measures early 3D motion relative to the prescribed direction using CoTracker3 and Depth Anything 3 Metric-Large. We also evaluate the matched 15K Force Prompting (Projected) baseline, which receives only the force's XY components through its 2D interface.

Expanded 3D comparison — Appendix H.3, Table 16 (expanded multi-seed counterpart of main-text Table 3)

ForceMethodF-Align ↑3D Dir.@45 ↑M+ IoU ↑H-Force ↑H-Physics ↑
PointForce Prompting (Projected)64.8636.9363.2278.5062.41
PointForceVid3D (Ours)74.0253.6066.5488.8366.57
WindForce Prompting (Projected)47.5234.3473.7465.5351.64
WindForceVid3D (Ours)68.4442.9396.7567.4254.55

Estimated 3D Dir.@45 improves by 16.67 points for Point and 8.59 for Wind, while F-Align increases by 9.16 and 20.92 points. These results strengthen the automatic evidence beyond the original human ratings. Appendix H.4 (Table 17; expanded multi-seed counterpart of main-text Table 4) additionally reports component ablations on 100 simulator-referenced cases; Section 5.5 addresses force magnitude and duration.

Q2: Force magnitude normalization across objects

A2: For point forces, motivated by F=MaF=Ma, we use an acceleration-scale quantity a=∥F∥/Ma=\|\mathbf F\|/M, then store the dimensionless magnitude m=clip⁡(a/aref,0,1)m=\operatorname{clip}(a/a_{\mathrm{ref}},0,1), where arefa_{\mathrm{ref}} is the maximum value across Point samples. During active intervals, the stored mm is encoded as 2m−12m-1. Wind uses source-relative normalized strengths; mass and friction remain simulator metadata. The exact protocol is in Appendix A.4.

Q3: Force duration sampling and VAE temporal alignment

A3: Videos have 49 RGB frames at 10 FPS. Durations are sampled in 4n+14n+1 RGB-frame units, favoring 5, 9, and 13 frames to align with the VAE structure. Within the retained training window, the active interval is half-open, [ts,te)[t_s,t_e), with duration (te−ts)/10(t_e-t_s)/10 seconds. Latent bin 0 covers frame 0; bin k≥1k\geq1 covers frames 4k−34k-3 through 4k4k and is active when its support overlaps the force interval. Saved active_latent_bins preserve this alignment; inactive and zero-force bins receive zero force maps (Appendix A.4).

Q4: Force magnitude and duration sensitivity

A4: We independently vary five normalized magnitudes (0.2–1.0) and three durations (5, 9, 13 frames) in five Supported Rigid-Body scenes, with two force directions per scene. With three inference seeds per condition and the final 15K checkpoint, this yields 450 videos. Within each scene–direction–seed group, the initial image, prompt, force direction/location, camera, and object properties stay fixed.

Using SAM2.1, we measure signed force-aligned endpoint displacement, D=(p48−p0)⋅f^2DD=(\mathbf p_{48}-\mathbf p_0)\cdot\hat{\mathbf f}_{\mathrm{2D}}. Within each scene–direction–seed group, all 15 conditions share the same maximum signed-displacement normalizer. We average normalized values over the 30 groups.

Mean normalized force-aligned displacement (dimensionless)

Duration (frames)m=0.2m=0.2m=0.4m=0.4m=0.6m=0.6m=0.8m=0.8m=1.0m=1.0
50.3330.4800.5800.6660.730
90.4490.6170.6730.7620.862
130.5460.6340.7880.8460.965

Across the evaluated grid, the mean normalized response increases monotonically with both force magnitude and duration. Figure 8 provides the response curves and qualitative examples.

Q5: Affinity-target construction and secondary collisions

A5: We compute instance-level response weights from future-frame appearance changes within ground-truth instance masks, then map these weights onto the corresponding initial-frame masks. The resulting affinity target links initially visible surfaces to later responses; it does not supervise a separate target at each future frame. Secondary collisions are included as indirect responses. A separate relative-depth target supplies local geometry from the initial depth map (Appendix B.1).

AuthorsOpenReview ↗

Feedback to Reviewer kL6J

Q6: Static-background supervision and dynamic scenes

A6: We deliberately use static backgrounds in synthetic D-RAS scenes so unrelated motion does not contaminate appearance-change-based affinity targets. Appearance changes alone cannot establish causality: independently moving background objects could otherwise introduce spurious response weights.

D-RAS is a training-only auxiliary supervision for force-response grounding, not a requirement that scenes be static at inference. The frozen pretrained backbone retains general motion priors; the control branch learns force-conditioned responses through flow matching. Unlabelled real wind clips contribute flow-matching supervision under null-force conditioning, without D-RAS targets. Supplementary Video S2 shows a dynamic-background case where the predicted response-affinity map remains concentrated on the force-responsive region despite unrelated motion. This is qualitative evidence for that case; explicit disentanglement across diverse moving-background scenes remains a limitation. Appendix G discusses the design choice and its scope.

Q7: Individual and joint effects of D-RAS objectives

A7: We compare No D-RAS, Affinity-only, Depth-only, and Full D-RAS under matched 8K-to-11K continuation. In each training repeat, all four variants start from the same 8K SFT checkpoint. Automatic metrics average two independent training runs × three inference seeds per checkpoint; human evaluation follows Appendix D.3.

Individual and joint supervision effects — Appendix H.2 and H.4, Tables 15 and 17 (expanded multi-seed counterparts of main-text Tables 2 and 4, respectively)

Objective2D F-Align ↑2D M+ IoU ↑2D H-Physics ↑3D Dir.@45 ↑3D R-Act. IoU ↑P-ADE ↓3D H-Physics ↑
No D-RAS79.9482.2465.1256.1791.490.78177.17
Affinity-only81.2982.8067.0463.3393.020.77177.83
Depth-only81.8082.3167.3161.6792.630.72678.17
Full D-RAS (Ours)81.9282.2868.0065.6794.230.71478.50

Response-affinity supervision associates initial-frame surfaces with their subsequent force-induced responses, helping ground the response in the relevant scene content. Affinity-only achieves the highest 2D M+ IoU (82.80) and improves 3D directional adherence. Relative-depth supervision provides local front–back geometry around responsive surfaces; Depth-only improves trajectory agreement, reducing P-ADE from 0.781 to 0.726. Each objective alone improves H-Physics in both 2D and 3D over No D-RAS. Combining them achieves the best 2D H-Physics (68.00), 3D H-Physics (78.50), 3D Dir.@45 (65.67), R-Act. IoU (94.23), and P-ADE (0.714) among the four variants, while maintaining comparable 2D M+ IoU (82.28). These results support complementary benefits in response grounding, geometric coherence, and physical consistency.

Q8: Caption generation and prevention of force leakage

A8: We use Qwen3-VL-30B to caption only the initial frame, describing visible objects, appearance, and spatial/support relations. The captioning instruction excludes future motion, force effects, wind, and per-sample force parameters; captions flagged for motion/force terms are regenerated. We append a fixed Point or Wind suffix, shared within each force family, while the force signal supplies individual support, direction, magnitude, and duration. Appendix A.5 documents the captioning and filtering rules and suffixes.

Q9: Gradient flow through the frozen backbone

A9: The conditional flow-matching loss backpropagates through the injected residuals into the trainable force encoder, control blocks, and residual projections, while the pretrained video backbone stays frozen. Thus, the controller can learn force-dependent changes to the conditional velocity field; D-RAS adds response-aware and geometry-aware supervision.

AuthorsOpenReview ↗

Feedback to Reviewer kL6J

Q10: Force-condition dependence versus motion memorization

A10: The synthetic training set varies force direction, magnitude, duration, object properties, and camera configurations. More directly, Figure 1 compares outputs with the same initial image, prompt, sampler, and inference seed, changing only the force direction. Section 5.5 independently varies magnitude and duration under fixed scene conditions and shows systematic changes in generated motion. ForceVid3D-Bench further evaluates 165 unseen web/AI images with 310 force conditions, including 154 camera-space 3D cases. Together, the counterfactual and unseen-image evaluations support force-dependent behavior beyond reproducing typical training motions.

Q11: Clarification of the supervision in Figure 2(c)

A11: Thank you for identifying the ambiguity. We revised Figure 2(c) and its caption. Future-frame appearance changes and instance masks provide per-instance response weights, which are mapped back to initial-frame masks and aggregated into one affinity target. The relative-depth target also comes from the initial depth map. Both heads supervise the initial-frame token slice, rather than a separate target for each future frame.

Reviewer_JMu6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies force-controlled image-to-video generation in which the user specifies an explicit camera-space 3D force without any scene reconstruction or physics simulation at inference. It conditions a frozen Wan2.2 backbone through a ControlNet-style branch on a seven-channel force map. This map augments a direct force encoding with analytic perspective-projection cues. It also adds training-only auxiliary heads that predict response affinity and local relative depth from intermediate controller features. The authors build a 60K-video synthetic dataset covering seven response modes and a 165-image benchmark. They evaluate against prior force-conditioned methods in a shared 2D setting and report separate 3D results using automatic metrics and human ratings.

Critical strengths

The paper addresses a practically relevant gap between image-plane force control and reconstruction-based or simulation-based physical video generation. It also contributes a sizable synthetic force-video dataset and a benchmark with both 2D and 3D force annotations. This matters because these resources could support future work on physically controllable generation regardless of the specific method.

Critical weaknesses

The evidence for the main claims is not convincing. In Table 1, the baselines use different backbones and are not retrained on the authors’ data, while ForceVid3D is trained on the same seven response modes that the benchmark covers. The large gains, such as F-Align rising from 70.63 to 84.21, therefore cannot be attributed to the method. The camera-space 3D claim rests mostly on human ratings. Automatic 3D metrics cover only 27 point and 5 wind cases, purely depth-directed forces are excluded, the Z-subset angle uses 16 cases and measures image-plane rather than 3D direction, and no 3D baseline is provided. These problems are fixable with retrained baselines and a larger automatic 3D evaluation.

The methodological contribution is small and its effects are not reliably established. The projective channels are an analytic function of the direction and pixel coordinates already given to the model, and real images rely on fixed nominal intrinsics. D-RAS leaves its target grounding metric unchanged (M+ IoU 82.23 versus 82.26 in Table 2). The ablations use a single seed. Within Table 4, the same Projective-7C model is worse at 11K steps than at 5K steps (Z-subset angle 43.64 versus 31.13), which suggests that checkpoint variance may exceed the reported component effects. The projective map also worsens P-ADE from 0.690 to 0.812.

Questions for authors

1.Report a comparison in which at least one baseline, for example Force Prompting or StreamForce, is retrained on ForceVid3D-Dataset with the same backbone. This would separate the effect of the method from that of the training data.

2.Report results over several seeds or checkpoints for Tables 2 and 4. Please also explain why the Projective-7C model at 11K is worse than at 5K on the 3D directional metrics.

3.Suggest automatic 3D results on a larger subset that includes depth-dominant forces. Please also report a simple 3D baseline, such as a prior method given the image-plane projection of the 3D force, along with inter-annotator agreement and annotator independence for the human ratings.

5 comments
AuthorsOpenReview ↗

Feedback to reviewer JMu6

Thank you for your careful review and concrete suggestions on baseline fairness, 3D evaluation, and experimental reliability. We have conducted additional experiments and clarified the relevant protocols in the revised manuscript.

Q1: Fair baseline comparison under matched training settings

A1: We retrain Force Prompting on ForceVid3D-Dataset with the same Wan2.2-TI2V-5B backbone and overall training settings as ForceVid3D. Its 2D conditions use the camera-plane (XY) components of the annotated 3D forces. We retain Force Prompting's moving-Gaussian representation for point forces and its three-channel magnitude–direction encoding for wind. Both models are trained for 15K steps and evaluated on the same shared-2D cases.

Matched 15K comparison — Appendix H.1, Table 14 (expanded multi-seed counterpart of main-text Table 1); Point + Wind (156 cases)

MethodF-Align ↑Proj.-Resp.@45 ↑M+ IoU ↑H-Force ↑H-Physics ↑
Force Prompting (Projected)79.4567.3182.2884.9459.72
ForceVid3D (Ours)83.8989.1083.1693.3274.20

Point only (90 cases)

MethodF-Align ↑Proj.-Resp.@45 ↑M+ IoU ↑H-Force ↑H-Physics ↑
Force Prompting (Projected)73.7674.4474.0885.9351.20
ForceVid3D (Ours)79.0293.1574.2994.2671.20

ForceVid3D improves Proj.-Resp.@45 by 21.79 points on Point + Wind and 18.71 on Point only; H-Physics improves by 14.48 and 20.00 points. M+ IoU on Point + Wind also increases from 82.28 to 83.16. Since the data, backbone, and training budget are matched, these differences cannot be attributed solely to those factors. Figure 5 shows qualitative comparisons.

Q2: Expanded automatic evaluation of 3D force control

A2: We expand the automatic 3D evaluation from 32 cases (27 Point, 5 Wind) to all 154 camera-space 3D conditions (88 Point, 66 Wind). All have unit directions with ∣dz∣>0.6|d_z|>0.6, including purely depth-directed forces with dx=dy=0d_x=d_y=0.

We evaluate Estimated 3D Dir.@45, which tests whether estimated early 3D motion lies within 45∘45^\circ of the prescribed camera-space force direction. Motion is estimated from CoTracker3 tracks and Depth Anything 3 Metric-Large depth estimates. The protocol is detailed in Appendix D.2.

Q3: A simple 3D baseline using projected 2D force inputs

A3: We evaluate the same 15K Force Prompting (Projected) baseline as in Q1 on all 154 3D conditions. It receives the prescribed force's XY components through its native 2D interface, while ForceVid3D receives the full 3D force.

Camera-space 3D evaluation — Appendix H.3, Table 16 (expanded multi-seed counterpart of main-text Table 3)

ForceMethodF-Align ↑3D Dir.@45 ↑M+ IoU ↑H-Force ↑H-Physics ↑
PointForce Prompting (Projected)64.8636.9363.2278.5062.41
PointForceVid3D (Ours)74.0253.6066.5488.8366.57
WindForce Prompting (Projected)47.5234.3473.7465.5351.64
WindForceVid3D (Ours)68.4442.9396.7567.4254.55

Estimated 3D Dir.@45 improves by 16.67 points for Point and 8.59 for Wind; F-Align improves by 9.16 and 20.92 points. Response localization also improves for both force families. These results provide additional automatic evidence of 3D force control beyond human ratings alone.

Q4: Geometric contribution of the projective channels

A4: The channels are analytically derived from the existing force direction, pixel coordinates, and camera intrinsics. Their role is not to add force information, but to make perspective geometry explicit. Direct-5C broadcasts direction and magnitude; Projective-7C also encodes the location-dependent terms (dx−xdz, dy−ydz)(d_x-xd_z,\ d_y-yd_z). This supplies a geometric inductive bias instead of requiring the controller to infer the projection relationship from data alone.

In matched 5K ablations, 7C improves shared-2D F-Align from 75.65 to 78.41 and H-Physics from 55.34 to 59.35 (Appendix H.2, Table 15; expanded multi-seed counterpart of main-text Table 2). On controlled 3D cases, Estimated 3D Dir.@45 rises from 49.17 to 57.50, and Z-subset angular error falls from 44.69° to 31.22° (Appendix H.4, Table 17; expanded multi-seed counterpart of main-text Table 4). These results support the value of the explicit geometry, particularly for forces with substantial depth components.

AuthorsOpenReview ↗

Feedback to reviewer JMu6

Q5: Camera intrinsics for synthetic and real images

A5: The representation accepts camera-specific intrinsics when available. We use stored intrinsics for synthetic scenes and fixed nominal intrinsics when calibration is unavailable, including uncalibrated real images. See Appendix A.4.

Q6: Repeated training runs, inference seeds, and matched budgets (original Tables 2 and 4)

A6: For each newly trained configuration in Appendix H (Tables 14–17; expanded multi-seed counterparts of main-text Tables 1–4, respectively), we use two independent training runs and three inference seeds per checkpoint. Released baselines use six inference seeds. Each automatic metric is computed for each evaluation and then averaged over the six scores. Human evaluation uses one randomly selected output set per configuration, independently rated by three annotators.

We also expand the controlled 3D evaluation to 100 simulator-referenced cases: 40 with ∣dz∣≥0.6|d_z|\geq0.6 and 60 with ∣dz∣<0.6|d_z|<0.6. Our component comparisons use matched budgets: 5C versus 7C at 5K, and the four D-RAS variants continued from the same 8K initialization to 11K.

Updated shared-2D ablation results — Appendix H.2, Table 15 (expanded multi-seed counterpart of main-text Table 2)

IDVariantStepsF-Align ↑Proj.-Resp.@45 ↑M+ IoU ↑H-Force ↑H-Physics ↑Aesth. ↑Imaging ↑
A15C + SFT0–5K75.6574.6881.2278.5855.3459.4566.24
A27C + SFT0–5K78.4176.0782.0483.4459.3559.2165.97
B17C + SFT (No D-RAS)8K–11K79.9479.2782.2485.5265.1259.1866.02
B27C + Affinity-only8K–11K81.2980.7782.8086.2267.0459.1066.22
B37C + Depth-only8K–11K81.8080.6682.3186.0667.3159.1266.37
B47C + Full D-RAS (Ours)8K–11K81.9282.5982.2886.4968.0059.1765.87

Updated controlled 3D results — Appendix H.4, Table 17 (expanded multi-seed counterpart of main-text Table 4)

IDVariantStepsF-Align ↑Z-subset Ang. ↓FS@45 ↑3D Dir.@45 ↑R-Act. IoU ↑H-Force ↑P-ADE ↓H-Physics ↑Aesth. ↑Imaging ↑
A15C + SFT0–5K28.5144.6974.1749.1786.9472.330.69360.4247.6465.61
A27C + SFT0–5K31.1231.2289.1757.5091.4682.750.80468.8349.5368.77
B17C + SFT (No D-RAS)8K–11K27.4142.5885.0056.1791.4987.170.78177.1748.6868.14
B27C + Affinity-only8K–11K30.8634.8388.6763.3393.0289.000.77177.8349.4168.51
B37C + Depth-only8K–11K29.4536.1987.1761.6792.6388.500.72678.1749.1968.67
B47C + Full D-RAS (Ours)8K–11K31.4933.4789.5065.6794.2389.250.71478.5048.4668.57

These repeated evaluations support the same main conclusions as the original Tables 2 and 4. At matched 5K training, Projective-7C improves directional force adherence and human-rated physical consistency over Direct-5C in both 2D and 3D evaluations. Under matched 8K-to-11K continuation, Full D-RAS improves 2D F-Align and H-Physics, as well as 3D directional adherence, region-activation agreement, and simulator-reference trajectory agreement over SFT. The limited additional gain in 2D M+ IoU and the 5K P-ADE trade-off also persist.

AuthorsOpenReview ↗

Feedback to reviewer JMu6

Q7: Directional metrics at 5K versus 11K (original Table 4)

A7: Comparing Projective-7C SFT at 5K with No D-RAS SFT at 11K reveals different trends across benchmarks and metrics. On the shared-2D benchmark, F-Align improves from 78.41 to 79.94, Proj.-Resp.@45 from 76.07 to 79.27, and H-Physics from 59.35 to 65.12 (Appendix H.2, Table 15; expanded multi-seed counterpart of main-text Table 2).

On the controlled 3D benchmark, F-Align decreases from 31.12 to 27.41, Z-subset projected angular error increases from 31.22° to 42.58°, and Estimated 3D Dir.@45 decreases slightly from 57.50 to 56.17. Meanwhile, H-Force improves, H-Physics increases from 68.83 to 77.17, and P-ADE decreases from 0.804 to 0.781 (Appendix H.4, Table 17; expanded multi-seed counterpart of main-text Table 4).

These metrics assess different aspects of generated motion, and the flow-matching objective does not optimize them directly. Improvements in physical coherence and trajectory agreement therefore need not coincide with better directional accuracy. Our component-effect conclusions rely on the matched-budget ablations in Appendix H.2 and H.4, Tables 15 and 17 (expanded multi-seed counterparts of main-text Tables 2 and 4, respectively), as described in Q6, rather than assuming monotonic improvements across checkpoints.

Q8: Directional adherence and the P-ADE trade-off (original Table 4)

A8: The matched 5K comparison in Appendix H.4, Table 17 (expanded multi-seed counterpart of main-text Table 4) shows that Projective-7C increases Estimated 3D Dir.@45 from 49.17 to 57.50 and H-Physics from 60.42 to 68.83, while P-ADE rises from 0.693 to 0.804. The 3D direction score measures adherence to the prescribed force direction; P-ADE instead measures agreement with the simulator-reference trajectory after arc-length resampling. Better direction and human-rated coherence can therefore coexist with larger path error.

This trade-off is discussed further in Appendix H.4. Supplementary Video S1 shows a 5C–7C trajectory comparison illustrating how overshooting can increase P-ADE despite better directional adherence.

Q9: Limited M+ IoU gains from D-RAS (original Table 2)

A9: Base SFT already includes object-region supervision, which may help explain the limited additional improvement in pixel-level localization. Full D-RAS yields a small 2D M+ IoU change (82.24 to 82.28) (Appendix H.2, Table 15; expanded multi-seed counterpart of main-text Table 2). D-RAS also provides additional benefits in response association and geometric consistency: under matched 11K training, Full D-RAS increases R-Act. IoU from 91.49 to 94.23 and 3D Dir.@45 from 56.17 to 65.67, while reducing P-ADE from 0.781 to 0.714 (Appendix H.4, Table 17; expanded multi-seed counterpart of main-text Table 4).

Q10: Independence and agreement of human evaluations

A10: For each configuration, we randomly select one of six checkpoint–seed output sets. Three annotators independently rate each video for H-Force and H-Physics on the original 1–5 scale, using a predefined rubric (Appendix D.3). Candidate videos are randomized within each case and method identities are hidden; ratings are collected separately before aggregation.

Across the study, 93.8% of annotator-pair ratings for H-Force and 88.6% for H-Physics differ by at most one point, measured on the original scores before normalization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.