acceptodds
Under review as a conference paper at ICLR 2027

PhysAtObj: Learning Physics-Grounded 3D Object Interaction from Multi-view Videos

Abstract

Modeling dynamic 3D scenes with interacting objects is crucial for understanding physical environments. However, existing methods struggle to extrapolate future frames because they fail to capture underlying physical dynamics. In this paper, we introduce PhysAtObj, a new framework that learns object interaction dynamics directly from multi-view videos without requiring explicit physical properties. The core of PhysAtObj is an object interaction module that identifies dynamic contact regions and directions, and then learns comprehensive impulse-induced and time-agnostic velocity changes during interactions. By relying on the physical states of interacting entities rather than absolute timestamps, the learned dynamics generalize robustly beyond the observed training window. Extensive experiments on PhysInOne and our newly introduced synthetic and real-world datasets demonstrate that PhysAtObj achieves state-of-the-art performance in future frame extrapolation, significantly outperforming baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Physics Learning, 3D Object Interaction Dynamics Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UoAWOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

PhysAtObj learns scene-specific object dynamics from calibrated multi-view videos using object-labelled Gaussians, motion reconstruction, and a time-independent interaction model. Future-frame results are promising, but the evidence for transferable physical laws remains limited.

Critical strengths
  1. Object-level modeling and the separation of rigid motion from deformation are well motivated.
  2. Table 1 shows substantial gains, including Moving-Static mIoU increasing from 40.105 to 67.255, with foreground metrics reducing background dominance.
  3. The appendix specifies state-aware propagation, and the elastic-cube ablation supports its usefulness for deformation.
Critical weaknesses
  1. The practical use is unclear given costly multi-view capture, scene-specific fitting, and limited evidence of cross-scene generalization.
  2. The external-field model lacks position and time conditioning, while a single contact normal per object pair may limit complex interactions.
  3. The small-scale evaluation, largely conducted in controlled settings with few moving objects, provides limited evidence of robustness in complex scenes.
Questions for authors
  1. Table 1 reports F1 of 87.229 above the precision/recall mean of 85.423, which is incompatible with standard F1 under identical aggregation.
  2. Section 4 reports 7 real scenes, whereas Appendix A.7 reports 6. Maybe reconcile the counts and list scene IDs for each aggregate.
3 comments
AuthorsOpenReview ↗

Response to Reviewer UoAW (Part 1)

Q1: The practical use is unclear given costly multi-view capture, scene-specific fitting, and limited evidence of cross-scene generalization.

A1: We thank the reviewer for this comment. We respectfully highlight that learning physical dynamics is notoriously challenging and remains in its early stages. Consequently, current evaluation settings are necessarily controlled, relying on calibrated cameras and per-scene optimization, akin to the early stages of 3D/4D reconstruction such as NeRF and 3DGS. Despite these constraints, we would emphasize our method's generalization capabilities and discuss practical deployment pathways as follows:

  1. Generalization (Section 4.2): As demonstrated in Section 4.2 of the revised paper, our learned interaction module and state-aware velocity propagation network successfully generalize to unseen geometries, novel initial velocities, and diverse object layouts without retraining. Once interaction priors are learned, the model can predict physical dynamics for novel object configurations and combinations.
  2. Multi-view Capture: With rapid advancements in sparse-view reconstruction, monocular depth estimation, and feed-forward 3D Gaussian priors, the reliance on dense multi-view capture is likely to be alleviated. Because our framework is modular, future progress in upstream feed-forward reconstruction and monocular estimation will directly benefit our interaction and propagation modules, paving the way toward real-world video interaction reasoning.
  3. Downstream Applications: Our method provides precise, physics-grounded 4D representations and trajectory predictions that are essential for applications demanding high physical fidelity, such as dynamic robotic manipulation, embodied AI, and interactive AR/VR content creation.

In the revised paper, we have updated the conclusion section to better highlight these practical considerations and future directions.


Q2: The external-field model lacks position ... a single contact normal per object pair may limit complex interactions.

A2: We appreciate this insightful comment and clarify our design principles as follows:

  1. Our interaction model is intentionally time-independent, which is central to achieving robust future extrapolation. Because physical interaction rules and impulse mechanisms (e.g., collision responses) are inherently invariant to absolute time tt, learning time-agnostic impulse patterns prevents temporal overfitting.
  2. Object interaction patterns are fundamentally invariant to absolute spatial positions; an interaction behaves identically whether it occurs at one corner of a room or another. However, interactions are critically dependent on relative positions. Our framework accounts for this by explicitly encapsulating relative spatial configurations during the identification of contact regions and directions.
  3. Regarding our external-field model, it is formulated to capture overarching environmental trends without rigid time-conditioning, aligning with the time-invariant nature of global baseline forces over short prediction horizons.
  4. Lastly, rather than relying on a single contact normal per object pair, our framework supports multiple simultaneous contact regions. Localized impact responses are computed independently across these regions and systematically aggregated, allowing our method to model complex, multi-point contacts and distributed collision manifolds rather than single-point approximations.

AuthorsOpenReview ↗

Response to Reviewer UoAW (Part 2)

Q3: The small-scale evaluation, largely conducted in controlled settings ... robustness in complex scenes.

A3: We thank the reviewer for this comment and have added a dedicated analysis in Appendix A.16 (Table 11 and Figure 14; also see the following Table 8), showing that:

  1. Across our benchmark datasets, the evaluated scenes feature up to 5 foreground entities, 4 simultaneously moving objects, and 5 distinct multi-body interaction events per scene, encompassing 10 core physical phenomena (e.g., elastic coupling, deformable objects, and oblique projectiles).
  2. Beyond idealized, noise-free settings, our benchmark explicitly incorporates real-world challenges, including scenes with degraded segmentation masks (boundary imperfections) alongside environmental disturbances such as sensor jitter, lighting variations, and visual clutter. Moreover, Figure 14 presents representative frames and masks from the evaluated PhysInOne scenes, demonstrating our model's robust physical reasoning under severe occlusions and intricate contact dynamics.

Overall, our comprehensive benchmark evaluation clearly demonstrates the robustness and breadth of our experiments.

Table 8: The complexity of all evaluated datasets.

Dataset Partition# Scenes# Foreground Objects# Moving Objects# Unique InteractionsKey Characteristics & Phenomena CoveredNoisy Masks & Cameras Settings
Moving-Static Object Interaction Dataset41∼21 \sim 2111∼21 \sim 2∙\bullet Moving–stationary collision ∙\bullet Moving–fixed collision ∙\bullet Moving–moving collision ∙\bullet Vertical fall ∙\bullet Friction stop ∙\bullet Oblique projectile ∙\bullet Balloon tether ∙\bullet Floating balloon ∙\bullet Elastic coupling ∙\bullet Deformable object∙\bullet 6 Mask noise scenes (boundary degradation, segmentation errors) ∙\bullet 6 Lighting & Sensor noise scenes (visual clutter, jitter)
Moving-Moving Object Interaction Dataset62∼42 \sim 4222∼52 \sim 5
PhysInOne Dataset71∼51 \sim 51∼41 \sim 42∼52 \sim 5
Real-world Object Interaction Dataset62∼52 \sim 51∼21 \sim 21∼51 \sim 5
Overall Benchmark Span23Up to 5Up to 4Up to 510 physical phenomenanoisy real world scenes

Q4: Table 1 reports F1 of 87.229 above the precision/recall mean of 85.423, which is incompatible with standard F1 under identical aggregation.

A4: Thank you for pointing out this discrepancy in Table 1.

Upon careful verification, we confirmed that this was an unintentional transcription error during the manual compilation of Table 1. Specifically, the recall and F1-score for one scene in the Moving-Moving Object Interaction Dataset were misrecorded. After incorporating the corrected values, the recall updates from 82.18882.188 to 86.18886.188, and the F1-score updates from 87.22987.229 to 87.27287.272, restoring full numerical consistency across the table.

Additionally, we have thoroughly audited all other experimental results, tables, and metrics throughout the manuscript to ensure complete accuracy.

We have updated Table 1 in the revised paper.


Q5: Section 4 reports 7 real scenes, whereas Appendix A.7 reports 6. Maybe reconcile the counts and list scene IDs for each aggregate.

A5: We sincerely appreciate your careful reading. Yes, that was a typo; there are 6 real scenes in total, which aligns with Appendix A.7.

We have corrected Section 4 and summarized the scene IDs for each aggregate group in Table 12 of the revised paper.

AuthorsOpenReview ↗

Waiting for discussion

Dear Reviewers UoAW,

Since the discussion is closing very soon (in less than 9 hours), we are still waiting for your thoughts on our rebuttal and revised manuscript. We would be grateful if you could share your further feedback. Thank you for your time.

Best,

Authors

Reviewer_HVNJOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the author proposed a method named PhysAtObj which predicts the future condition of a dynamic 3D scene with interaction among objects based multi-view RGB videos. Assuming known camera extrinsics, intrinsics, and object masks, the method first reconstructs a 3D GS of the target scene with extra objects identities for timestamp 0. Second, it estimate the object-aware motion and velocities of each Gaussians using deformation field for the whole scene, the velocity per Gaussian is combined of object-level rigid motion and Gaussian-level deformation residual. Finally, the contacts among objects are detected by the proposed object interaction module. Give current state, contact configuration, current relative velocity, and time step t, the module will estimate the contact regions, directions, and velocity changes, then a propagation network will update the Gaussian state including rigid motion and deformation. To validate their method's improved performance, paper shows comparison experiment results with SOTA baselines on two dataset: PyhsInOne and a new synthetic datasets introduced by the paper.

Critical strengths
  1. The idea from the paper is strong and novel. Instead of predicting a dynamic scene only abased on the time stamp, it including the extra physical states and contact among objects. Physical contact among objects has been seriously ignored by current dynamic 3D scene prediction methods. At the same time, the paper shows their improvement performance in their experiments compared with SOTA baseline which better support the importance of the method.

  2. The paper is well-written and organized which makes the paper easier to understand and potentially have broader influence in the dynamic scene prediction community.

Critical weaknesses
  1. The method highly depends on assumption of accurate camera calibration and object masks, also an accurate scene reconstruction as initialization which will limits the proposed method's performance with many critical scenarios where MVS, segmentation and 3DGS failed.

  2. The method is combined with multi-stage modules and training, and some stages such as 3DGS reconstruction and deformation field are still scene-specific. The contact detection is still a heuristic threshold based. Those make the whole pipeline too complicated and a single stage issue will cause the whole pipeline failed which make the robustness of the proposed method suspicious.

Questions for authors
  1. How sensitive is method to camera calibration error?
  2. How sensitive is method to object masks error?
  3. How is the performance of the proposed method for longer rollouts? Will the error accumulate fast?
  4. How are the static objects treated if they interact with other moving objects?
4 comments
AuthorsOpenReview ↗

Response to Reviewer HVNJ (Part 1)

Q1: The method highly depends on assumption of accurate camera calibration and object masks ...

A1: As requested, we have added a comprehensive sensitivity analysis in Appendix A.14 and Table 9 of the revised paper (also see the following Table 5). Specifically, we evaluated our method under three realistic, imperfect settings:

  1. COLMAP Poses: We replace ground-truth camera poses with estimated poses obtained from COLMAP. Our method exhibits only a graceful performance degradation, demonstrating that our framework does not overfit to ideal camera parameters and effectively accommodates structure-from-motion (SfM) errors.
  2. SAM3 Masks: We evaluate our framework using off-the-shelf masks generated by SAM3. Despite boundary inaccuracies and partial omissions, the model maintains high physical and reconstruction fidelity, confirming its resilience to imperfect segmentation inputs.
  3. 10% Synthetic Mask Noise: In this setting, each pixel is independently selected with a 10% probability and reassigned a uniformly sampled alternative label during training. Performance remains remarkably close to the clean ground-truth setting, highlighting an inherent regularization capability within our formulation that prevents localized annotation errors from corrupting the global scene representation.

Overall, these quantitative results demonstrate that our approach exhibits strong robustness against realistic errors from upstream modules without suffering catastrophic failure.

Table 5: Sensitivity to camera poses and object masks.

Input settingAP$\uparrow$PQ$\uparrow$F1$\uparrow$Pre$\uparrow$Rec$\uparrow$mIoU$\uparrow$mPSNR$\uparrow$mSSIM$\uparrow$mLPIPS$\downarrow$PSNR$\uparrow$SSIM$\uparrow$LPIPS$\downarrow$PMF
COLMAP poses59.28552.03270.61771.09270.88761.27520.4340.7580.43230.9100.8880.1725.325
SAM3 masks55.62747.38569.04368.85869.79558.26019.5570.7520.45529.8420.8720.1855.206
10% mask noise62.45854.87574.10874.27074.19864.48020.2070.7490.42030.9660.8890.1605.459
GT poses and masks66.76059.61577.27877.64877.14067.25521.3440.7570.38531.4750.8840.1635.427

Q2: The method is combined with multi-stage modules and training ... contact detection is still a heuristic threshold based ...

A2: Regarding the contact detection threshold, we have added an additional ablation study on the neighborhood radius in Section 4.3 (Table 8) of the revised paper (also in the following Table 6). Particularly, we scaled our default neighborhood radius by ×0.5\times 0.5 and ×2\times 2. We can see that significantly varying the radius only marginally decreases performance, without causing the pipeline to fail. Notably, we do not perform an extensive search for the optimal radius in our experiments; instead, most scenes share the same hyperparameter setting. Since the neighborhood radius has a clear physical interpretation, it can be intuitively configured based on the scene scale. Within a physically plausible range, our method operates reliably without clear performance degradation.

These results demonstrate that our multi-stage pipeline is robust and does not suffer from brittle cascading failures due to threshold selection.

Table 6: Ablations of the neighborhood radius for contact detection.

ScalingAP ↑\uparrowPQ ↑\uparrowF1 ↑\uparrowPre ↑\uparrowRec ↑\uparrowmIoU ↑\uparrowmPSNR ↑\uparrowmSSIM ↑\uparrowmLPIPS ↓\downarrowPSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrow
×0.5\times 0.564.8557.6775.4475.9875.1565.4521.090.7530.40031.350.8850.162
×1\times 1 (Default)66.7659.6277.2877.6577.1567.2621.340.7570.38531.470.8840.163
×2\times 259.8653.7170.2370.1470.4760.6120.610.7400.44331.140.8840.164

Q3: How sensitive is method to camera calibration error?
Q4: How sensitive is method to object masks error?

A3/A4: Refer to A1.


AuthorsOpenReview ↗

Response to Reviewer HVNJ (Part 2)

Q5: How is the performance of the proposed method for longer rollouts? Will the error accumulate fast?

A5: Thank you for this insightful comment. As requested, we have added a dedicated analysis in Appendix A15 (Figure 13 and Table 10) of the revised paper (also see the following Table 7), evaluating our method when extending the extrapolation rollout horizon from the standard 0.5–1.00.5\text{--}1.0 window to an extended 0.5–1.50.5\text{--}1.5 window.

We can see that the performance of our method indeed degrades over time as the rollout horizon extends. This is fully expected and aligns with the inherent nature of future prediction: operating under conditions of limited initial observations, predicting further into the future naturally encounters increasing environmental uncertainty and compounding ambiguity.

Specifically, instance-level and moving-region metrics exhibit a noticeable yet controlled decline due to increasing environmental uncertainty and compounding motion ambiguity under limited initial observations. Conversely, full-scene structural fidelity remains remarkably stable with only marginal changes. Crucially, this indicates that while errors do accumulate over extended horizons, the degradation is gradual and bounded rather than an abrupt or catastrophic collapse.

Importantly, this limitation is fundamentally manageable, particularly in real-world deployment scenarios. In practical applications, as time moves forward, the system progressively receives new visual observations and streaming data, which can be continuously leveraged to calibrate the model, correct intermediate drifts, and refine subsequent predictions in a closed-loop manner.

Table 7: Quantitative comparison of our proposed method when extending the extrapolation rollout horizon from 0.5–1.0 to 0.5–1.5.

HorizonAP ↑\uparrowPQ ↑\uparrowF1 ↑\uparrowPre ↑\uparrowRec ↑\uparrowmIoU ↑\uparrowmPSNR ↑\uparrowmSSIM ↑\uparrowmLPIPS ↓\downarrowPSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowPMF ↓\downarrow
0.5–1.00.5\text{--}1.066.7659.6277.2877.6577.1567.2621.340.7570.38531.470.8840.1635.43
0.5–1.50.5\text{--}1.541.2434.4451.9252.9051.4943.2617.390.6770.60028.890.8760.1824.93

Q6: How are the static objects treated if they interact with other moving objects?

A6: To clarify, our framework does not treat initially static objects (e.g., an item resting on a table) as passive background; instead, it actively models their physical states and interactions through several key mechanisms:

  1. Continuous Physical Forces and Impulses: Even stationary objects are continuously subject to environmental forces. For instance, an object resting on a desktop remains governed by physical forces and reaction impulses, such as downward gravity paired with the supporting normal reaction from the table to maintain equilibrium and prevent interpenetration.
  2. Mutual Interaction and Collision Response: When a moving object collides with a stationary or resting object, our interaction module computes time-agnostic, impulse-induced velocity updates that affect both entities simultaneously.

To demonstrate this, our datasets include scenarios where static and moving objects interact. For example, as shown in the second scene of Figure 8, when a moving ball strikes two wooden blocks (one stationary and one moving), our method successfully models the mutual collision impulses and physical responses for both.

Thank you for the detailed rebuttal and additional experiments.

  1. The additional ablation study on camera calibration, object masks and contact address some of my concerns.
  2. The performance of the proposed method for longer rollouts is discussed, but still a limitation of the method.
  3. The clarification about static objects answer my question well.

Overall, I will maintain my rating of 3 (Weak Accept).

Dear reviewer HVNJ,

Thank you very much for your positive feedback.

Regarding the performance on longer rollouts, we are actively conducting new continual learning experiments. Specifically, we first train on frames between 0∼0.20 \sim 0.2 timesteps and evaluate extrapolation on 0.2∼0.40.2 \sim 0.4; we then incorporate 0.2∼0.40.2 \sim 0.4 observations for continual training and evaluate extrapolation on 0.4∼0.60.4 \sim 0.6, repeating this progression by adding time units of 0.20.2 up to training on 0∼0.80 \sim 0.8 and evaluating on 0.8∼1.00.8 \sim 1.0. These experiments are being conducted on the Moving-Static Object Interaction Dataset, comparing against baselines: TRACE and FreeGave, both of which have been adapted into a continual learning framework to ensure a fair comparison.

Due to time constraints, we may be unable to complete these results before the end of the Author–Reviewer Discussion window. However, we will share these ongoing findings directly in the comments during the upcoming Reviewer–AC Discussion period, and they will be thoroughly presented and discussed in the next version of our paper.

Thank you again for your concrete feedback. Your thoughtful suggestions have helped us substantially improve the paper.

Best regards,

Authors

Reviewer_AuyhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PhysAtObj predicts future object interactions from calibrated multi-view videos assuming initial object segmentations are provided. It works in stages, canonical 3DGS reconstruction, deformation field learning, and interaction module learning. First, it fits a set of canonical 3DGS (with object codes) and an auxiliary deformation field to recover the observed motion. It then learns a time-independent object interaction module that includes Field-induced Velocity Change, Normal Velocity Change, Frictional Velocity Change. A state-aware propagation network converts these responses into each object's rigid motion and local deformation for future rollout. The experiments mainly evaluate scene-specific future extrapolation on synthetic and real world datasets. The authors also show with additional demonstrations of transfer to ood geometry, initial velocities, and real-world observations.

Critical strengths
  1. PhysAtObj models predict the future velocities of objects conditioning on contact geometry and relative motion in a time independent way, which is a useful design for a lot of real world simulation use cases.
  2. PhysAtObj doesn't require ground truth 3D initial states of objects and is able to learn multi-object dynamics with limited physical supervision (requires just calibrated multi-view videos and segmentations) 3.Quantitatively, PhysAtObj shows big improvement on synthetic interaction benchmarks.
Critical weaknesses
  1. It seems that the state-aware velocity propagation has a lot of components (neighborhood aggregation, restoring and damping responses, stretching constraints, etc), but it is unclear which component improves the improvement and whether all components are necessary. Would it be possible to have some ablation studies on this?
  2. The generalization claims seems limited, in A.10, the model was trained on one scene and tested on an novel scene. Could the authors provide some more comments on the generalization of this method?
  3. In the appendix, the autors mentioned that per scene training takes ~4 hours, which seems quite long given that the scenes don't contact that many object and that the interactions are still quite simple, are there more details about the training (such as the cost of each stage)?
  4. Most of the metrics (such as PSNR, LPIPS,, mIoU) in the paper does not always correlate correlate with physical accuracy, are there more metrics that can be shown related to measuring the physcial accuracy of the predicted interactions?
  5. It seems that the model has a hard time capturing rolling motions and would predict sliding motions (eg the 2 examples in Real-world Object Interaction Dataset from the supplementary videos), could the authors give some explanation about this failure case.
Questions for authors

Please see weaknesses, thanks.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Auyh (Part 1)

Q1: It seems that the state-aware velocity propagation has a lot of components ... some ablation studies on this?

A1: Thank you for this insightful comment. As suggested, in the revised paper, we have added a comprehensive ablation study in Section 4.3 (page 9) and Table 4 (also see the following Table 1) to evaluate each component within our state-aware velocity propagation module.

Based on the new ablation results, we can see that: 1) Each single-component removal lowers the object segmentation metrics. 2) Removing dynamic state causes the largest decline among these ablations, while removing neighborhood aggregation also has a clear effect. 3) The restoring, damping, and stretching components each provide smaller gains. 4) Replacing structured propagation with simple broadcasting reduces AP, while removing local deformation leads to the largest performance drop. 5) Full-frame rendering metrics vary less across the variants, consistent with these components primarily affecting the moving objects.

Overall, our comprehensive ablation studies confirm that each component of the state-aware velocity propagation module contributes uniquely to overall performance and that all of them are jointly necessary.

Table 1: Quantitative results of ablations in state-aware velocity propagation.

Model VariantAP$\uparrow$PQ$\uparrow$F1$\uparrow$Pre$\uparrow$Rec$\uparrow$mIoU$\uparrow$mPSNR$\uparrow$mSSIM$\uparrow$mLPIPS$\downarrow$PSNR$\uparrow$SSIM$\uparrow$LPIPS$\downarrow$
(8) w/o neighborhood aggregation31.20323.64348.07648.78147.87135.88616.7760.6740.61928.8250.9720.065
(9) w/o restoring31.79724.06649.03649.76548.79836.58916.8340.6740.61628.8560.9730.065
(10) w/o damping31.76824.05648.96149.66248.75036.54216.8370.6740.61528.8520.9730.065
(11) w/o stretching constraint31.77724.04348.98949.74148.72636.56016.8310.6740.61628.8560.9730.065
(12) w/o dynamic state30.27822.62747.10148.29146.39734.99816.7020.6740.60428.8070.9710.069
(13) simple contact broadcast23.07717.01137.07038.63335.96527.24115.9420.6780.65628.7060.9730.064
(14) w/o fpropf_{\mathrm{prop}}16.55111.81622.58526.01321.42817.45114.8440.6760.69228.7320.9750.063
The Full PhysAtObj32.00924.38449.22149.93549.00136.78416.8540.6740.61428.8780.9730.064

Q2: The generalization claims seem limited, in A.10, the model ... provide some more comments on the generalization of this method?

A2: Regarding our generalization ability, in the revised paper, we have added more experiments in Section 4.2. In particular, we curate two geometry variants for 2 scenes as shown in Figure 4 of the revised paper, and compare with FreeGave (per-scene trained on the new scenes). As shown in the following Table 2 (also Table 2 in the revised paper), our method yields superior results without retraining the interaction models.

Nevertheless, as shown in Appendix A.10, when our method generalizes from a flat ground plane to an inclined slope, the larger performance gap stems primarily from the unseen friction on the inclined slope, which is absent in the flat ground plane environment.

Table 2: Quantitative results on generalization to unseen geometries.

MethodAP$\uparrow$PQ$\uparrow$F1$\uparrow$Pre$\uparrow$Rec$\uparrow$mIoU$\uparrow$mPSNR$\uparrow$mSSIM$\uparrow$mLPIPS$\downarrow$PSNR$\uparrow$SSIM$\uparrow$LPIPS$\downarrow$
FreeGave35.39528.60537.07038.13337.07032.84019.2260.7320.53531.6260.8940.153
PhysAtObj (Ours)54.81038.94054.64551.37561.26045.52521.3560.7860.44632.5580.8960.146

Beyond this, our evaluation also includes: 1) multi-scene generalization (Section 4.2 of the revised paper), showing that our interaction module can capture physical dynamics across multiple diverse scenes rather than a single environment, and 2) sim-to-real generalization (Appendix A.11 of the revised paper), demonstrating that our object interaction module trained solely on synthetic scenes can successfully transfer to predict real-world object interactions.


AuthorsOpenReview ↗

Response to Reviewer Auyh (Part 2)

Q3: In the appendix, .. training takes ∼\sim 4 hours ... more details about the training (such as the cost of each stage)?

A3: As requested, the following Table 3 (also available in Appendix A.9, Table 6 of the revised paper) provides a detailed breakdown of our training time. Training takes an average of 3.48 hours per scene (conservatively reported as $\sim$4 hours to account for data preprocessing, I/O overhead, and checkpointing) on a single NVIDIA RTX 3090 GPU.

Table 3: Per-scene training-time breakdown.

Training StageIterations / UpdatesWall-Clock TimeShare
Stage A: 4D Gaussian reconstruction40,00040{,}0001,h 55,min 22,s55.4%
Stage B: Interaction-model optimization≈20,000\approx 20{,}0001,h 18,min 29,s37.7%
Stage C: Velocity-propagation module1,0001{,}00014,min 46,s7.1%
Total Optimization Time--3,h 28,min 37,s (≈3.48 h\approx 3.48\text{ h})100%

While jointly optimizing the entire pipeline takes 3.48 hours, our method maintains high inference and rendering efficiency (comparable to FreeGave). Furthermore, peak GPU memory usage requires only 8 GB during inference.


Q4: Most of the metrics (PSNR, LPIPS, mIoU) ... are there more metrics ... to measuring physical accuracy of the predicted interactions?

A4: We appreciate this insightful comment. As requested, we incorporate the following metrics to measure physical motion accuracy:

  1. PMF (proposed in PhysInOne): PMF is a frequency-domain similarity metric that compares the spatiotemporal power spectra of generated and ground-truth videos. Higher PMF values indicate better agreement, demonstrating that the generated video preserves the underlying frequency characteristics of physical motion rather than merely static pixel layouts.
  2. SP IoU / ST IoU / WS IoU (proposed in Physics-IQ): These three metrics evaluate how accurately motion patterns evolve over space and time: 1) SP IoU compares the regions where motion occurs across the entire sequence, 2) ST IoU evaluates motion masks at corresponding time steps, and 3) WS IoU accounts for the fraction of frames exhibiting motion at each pixel.

As reported in the above Table 4 (also Table 1 in the revised paper), our method surpasses all baselines across all synthetic datasets, highlighting the physical accuracy of our predicted object dynamics. In real-world scenes, however, appearance variations such as shadows, reflections, and sensor noise are frequently captured as dynamics during metric calculation, narrowing the performance gap between our method and baselines. In contrast, the available object masks allow segmentation metrics to more directly assess the predicted visible regions of interacting objects.


Table 4: Quantitative results of all methods for future frame extrapolation on all four datasets (4 new metrics).

Moving-Static Object Interaction Dataset

MethodPMF$\uparrow$SP IoU$\uparrow$ST IoU$\uparrow$WS IoU$\uparrow$
DefGS4.7800.3110.0870.202
DefGS$_{nvfi}$4.4140.2330.1150.125
ODE-GS4.9970.3260.2560.291
TRACE4.7560.3140.1290.211
FreeGave4.9360.4070.1930.276
PhysAtObj (Ours)5.4270.6510.2820.554
PhysAtObj$_{multiScenes}$ (Ours)5.2064.700.1840.382

Moving-Moving Object Interaction Dataset

MethodPMF$\uparrow$SP IoU$\uparrow$ST IoU$\uparrow$WS IoU$\uparrow$
DefGS4.7070.1720.0830.100
DefGS$_{nvfi}$4.8800.2730.2140.128
ODE-GS4.7080.2500.0580.175
TRACE5.1540.4290.2330.231
FreeGave4.6420.3000.1580.185
PhysAtObj (Ours)5.1810.4570.2330.258

Real-world Object Interaction Dataset

MethodPMF$\uparrow$SP IoU$\uparrow$ST IoU$\uparrow$WS IoU$\uparrow$
DefGS4.1150.3070.0850.215
DefGS$_{nvfi}$4.0920.0110.0260.007
ODE-GS3.6190.2610.0990.189
TRACE4.0690.0020.0270.001
FreeGave4.1790.2420.1140.197
PhysAtObj (Ours)4.1560.2580.0990.203

PhysInOne Dataset

MethodPMF$\uparrow$SP IoU$\uparrow$ST IoU$\uparrow$WS IoU$\uparrow$
DefGS4.2460.2670.0560.205
DefGS$_{nvfi}$4.6550.2330.1740.264
ODE-GS4.1410.2820.1080.247
TRACE4.6400.3210.1860.317
FreeGave4.5050.2820.1110.274
PhysAtObj (Ours)4.6800.3360.1890.290

AuthorsOpenReview ↗

Response to Reviewer Auyh (Part 3)

Q5: It seems the model has a hard time capturing rolling motions ... give some explanation about this failure case.

A5: We thank the reviewer for this thoughtful comment. Capturing rolling dynamics (such as unconstrained rolling balls or cylinders) solely from videos is inherently challenging due to two primary factors: 1) Without material priors like mass and inertia tensors, modeling torque and moments of inertia is difficult, causing the model to couple rotational inertia with translational velocity. 2) In real-world scenarios, fast-moving rolling objects frequently exhibit motion blur, specular highlights, and texture symmetry (e.g., uniform sports balls). These visual factors hinder the precise tracking of rotational texture shifts from pixels alone, leading the model to favor stable translational trajectories over complex rotational updates.

To address these limitations, potential solutions include scaling up data diversity through large-scale training with a foundation object interaction model, or integrating explicit analytical formulations such as rotational dynamics, torque constraints, and rolling-friction solvers. Nevertheless, these extensions are non-trivial and left for our future work.


Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

Dear reviewers,

We sincerely thank all of you for your constructive feedback and insightful comments. We hope you have had a productive and fruitful CVPR week!

As the discussion period nears its end, we would like to gently follow up on our rebuttal and revised PDF submitted earlier.

In response to all your suggestions, we have conducted comprehensive additional experiments and provided detailed clarifications, all highlighted in blue in the revised PDF and summarized below:

1) Four New Metrics: We have incorporated four new metrics to measure physical motion accuracy (Table 1 in the revised paper).

2) More Detailed Ablations: We have conducted detailed ablations for each component in the State-aware Velocity Propagation module (Section 4.3 and Table 4 in the revised paper).

3) Additional Robustness Analysis: We have added experiments and analysis on our robustness to mask errors and camera errors (Section A.14 in the revised paper).

4) More Generalization Results: We have extended our generalization experiments and also show quantitative comparisons (Section 4.2, Figure 4, and Table 2 in the revised paper).

5) Additional Hyperparameter Robustness: We have added experiments and analysis on our robustness to contact radius (Section 4.3, Table 8 in the revised paper).

6) Additional Analysis of Longer Rollout: We have added experiments and analysis of our performance on long rollout and error accumulation (A.15 in the revised paper).

7) Computational Cost Breakdown: We provided a detailed budget of our computational cost (Section A.9).

8) Discussion on Failure Cases: We discussed our failure cases in rotation prediction (Section 5).

9) Discussion on the Scale and Complexity: We have quantitatively summarized and discussed the complexity of our evaluation datasets.

10) Corrections of Typos: We have corrected the typos in Table 1 and the scene count, and provided detailed scene summarization in Table 12.

We remain available to address any remaining questions or concerns before the discussion phase closes. Thank you again for your time.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.