acceptodds
Withdrawn from ICLR 2027

Differentiable Cross-Object Contact Reconstruction from Real-World Collision Videos

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reconstructing real-world collisions from video requires identifying both how objects deform and how they exchange momentum through contact. Real-world recordings expose these coupled dynamics under occlusion, reconstruction errors, and mismatch between physical behavior and simulation models. We present an object-aware differentiable framework that jointly fits object-level material parameters and inter-object contact parameters from calibrated multi-view video. Separated Material Point Method states preserve each object's constitutive response, while contact geometry derived from reconstructed surfaces drives two-way impulse exchange. Differentiating through a finite projected contact solver connects surface and image observations to material response, friction, and optional adhesion. We also introduce Contact4D, a calibrated multiview dataset of real-world collisions between objects in free fall across diverse material pairings, with persistent instance labels and reconstructed surface observations. Comparisons with adapted reconstruction baselines show improved agreement with observed motion, geometry, and appearance. Matched ablations demonstrate the contribution of inter-object contact and the treatment of ambiguous surface observations, while numerical audits assess the consistency of contact derivatives. Together, the method and dataset support joint material and better contact reconstruction from real-world collision videos.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Physics Simulation, Realworld Dataset

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YS1sOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a pipeline for reconstructing the 3D geometry and motion of interacting objects from multiview videos. The method combines 4D Gaussian reconstruction with differentiable physics to estimate material and contact parameters. The authors also introduce a new dataset to evaluate their approach.

Critical strengths

The paper addresses a challenging problem using modern reconstruction and simulation techniques. The introduction of a real-world multiview collision dataset is also a potentially valuable contribution.

Critical weaknesses
  1. Limited novelty and unclear benefits of the proposed pipeline. The method is relatively complex and primarily combines existing techniques, notably 4D Gaussian reconstruction and differentiable physics. The authors do not convincingly demonstrate the novelty or benefits of this combination compared with simpler approaches. In particular, reconstruction and physical parameter estimation are performed sequentially rather than jointly optimized in an end-to-end differentiable framework. The experiments do not clearly establish what the additional complexity contributes.
  2. Limited generality. The experiments focus on interactions between two objects with prescribed, spatially homogeneous material models. It is unclear how the approach would extend to more complex scenarios involving multiple interacting bodies or heterogeneous materials, such as objects containing cavities or multiple materials. These limitations are particularly relevant when considering the applicability of the proposed reconstruction framework.
  3. Insufficient dataset description. The presentation of the Contact4D dataset is underwhelming. I consider the dataset potentially one of the paper’s main contributions, yet it receives very little attention. Important details about its size, diversity, material properties, and evaluation protocol are insufficiently discussed. A more comprehensive description and characterization would significantly strengthen the paper.
  4. Lack of physical parameter validation. The evaluation focuses primarily on reconstruction quality rather than the accuracy of the estimated physical parameters. Why not perform independent material characterization experiments and compare the recovered Young’s modulus, Poisson’s ratio, or other material parameters against measured values? Reproducing an observed trajectory does not necessarily imply that the inferred material parameters are physically accurate.
Questions for authors

Please address the concerns raised in the weaknesses, particularly the novelty and benefits of the proposed pipeline, the generality of the method, the dataset characterization, and the lack of validation against independently measured material properties.

Reviewer_PdPhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a differentiable framework for reconstructing collisions between two objects from calibrated multiview videos. Each object maintains separate simulation states, while a surface-based contact model exchanges normal and frictional impulses through a projected solver. Differentiating through the simulation enables joint fitting of material and contact parameters using surface, image, and mask losses. The method also filters ambiguous surface correspondences near the contact interface. The authors introduce Contact4D, a real-world collision dataset, and evaluate reconstruction against adapted baselines and matched contact and supervision ablations.

Critical strengths
  • The experiments show the importance of inter-object contact, by removing pair contact reduces IoU from 0.6094 to 0.3832 and increases surface CD from 0.02310 to 0.05477. Because initialization, fitting, and evaluation are matched, this is stronger evidence than the comparison with an independently adapted baseline.

  • The authors support the treatment of uncertain contact surfaces as filtering ambiguous seam correspondences improves IoU from 0.5934 to 0.6127 while retaining the corresponding particles in the simulation. The projected solver also reduces measured penetration compared with smooth-penalty contact, although their reconstruction results are relatively close.

Critical weaknesses
  • The comparison in Table 1 is not very clear to me. The authors report their results on the full dataset, while OPGS-RGB is evaluated only on its accepted subset. The text says their method performs better on the same subset, but the actual scores are not provided. Also, the two methods uses different supervision: OPGS-RGB uses RGB, while the proposed method also uses surfaces and masks. So it is very hard for me to tell how much of the improvement comes from the contact model itself.

  • The results show that the reconstructed motion looks closer to the video, but it is less clear whether the estimated physical parameters are reliable. For example, an error in friction might be compensated by changing the material stiffness, so different parameter values could produce similar motion. The current metrics does not tell us whether this happens. It would help to check whether different initializations give similar parameters, or whether the fitted parameters can predict motion that was not used during fitting.

Questions for authors
  • Would it be possible for providing OPGS-RGB the same surface and mask supervision as your method? This would help show how much of the improvement comes from the contact model. A fair comparison would better compare between the proposed method and the recent work

  • My concern is if I start the optimization with different parameter values, do I end up with similar material and friction estimates? Or can quite different parameters explain the same video equally well?

  • I wonder if the method fit the parameters using only the beginning of the video, can the simulation correctly predict what happens afterward? If the prediction matches what actually happens, it gives stronger evidence that the estimated properties are useful.

  • How sensitive is the method to errors in the reconstructed surfaces? If the surfaces have small gaps or noisy boundaries, can the method still recover the collision correctly?

Reviewer_cqS9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper describes a system for tracking deformable objects from multi-view video. It uses point cloud representations of surfaces, optimized via numerical descent & Gaussian splatting, to reconstruct shape, and then feeds the pieces of geometry at the different time steps into an MPM-model, which is a hybrid particle-grid model for modeling deformations. My understanding is that this approach allows the system to use non-corresponding (only segmented) point clouds of the volumes of the objects as inputs and couple them by a "Eularian", i.e. spatially parametrized, deformation field. The physical model provides several different classes of materials and types of contacts between them (set manually at the beginning). A particular novelty over previous work, which receives special attention in this paper, is detection of contact areas, where an implicit representation, fitted from segmented object images / point clouds is used to detect and track collision events. The overall reconstruction pipeline is quite traditional, following typical approaches that have been proposed in the past 20 years: A numerical optimization scheme that jointly fits geometry and deformation/motion is employed to reconstruct trajectories. The parametrization over volume seems particularly clever, as it avoids the hard and highly non-convex correspondence problems that trouble methods that try to match observed points over frames (in my opinion and experience, this is impossible to reliably fit by just doing local optimization with numerical descent). The paper evaluates the method by several ablation scenarios and by comparison to an earlier method that can be understood as a base-line/ablation of the proposed method, as well as an approach that does not model collisions.

The results look good to me - this is not an easy problem to address in terms of devising a working system. My background knowledge in the current state of the art is too limited to really assess novelty; it seems that collision handling is the main innovation and that part does seem to work very well and is able to handle a lot of different contact models (validated with real-world experiments, which is convincing, as seen in the video). On a conceptual level, the approach is rather straightforward; there are no fundamentally new ideas, but good system engineering.

Remark: I have not worked in this area in recent years; I am broadly familiar with these type of problems but my ability to assess novelty over recent work is limited.

Critical strengths

The main appeal of the paper is a working and flexible system for tracking colliding deformable solids:

  • The method works on real-world data, as seen in the video.
  • It is able to handle multiple different scenarios in a stable fashion.
  • The basic idea is not very difficult, but the hybrid particle-volume formulation navigates around some of the fundamental problems in this area. While not ground-breaking (there are obvious drawbacks - we also do not get any correspondences from the reconstruction), it is a need formulation that leads to a useful system build from rather traditional methods (no fancy pattern recognition or strong generative modeling to build shape and motion priors is required).
Critical weaknesses

In my view, the broader conceptual novelty is not very large and evaluation is limited to one meaningful competing base-line.

  • Rather traditional system, no fundamentally new ideas; structurally similar approaches have already been tried in the past 20 years (various animation-from-video reconstruction systems). The key aspect is adding a better collision modeling approach. While there are not surprises here (engineering a good numerical fit), I am not aware of specific prior work that addresses this specific problem (however, I am not knowledgeable in the latest state of the art).
  • The topic is a bit at the at the edge of the scope of ICLR; a computer vision or computer graphics conference might be a better fit (not a critical issue, but maybe worth a consideration). There is little learning in a strict sense (methodologically, fitting models to data using statistical variational approaches, which is what is done here, is still close to ML, so I would still see a fit to the venue in a broader sense).
  • The paper makes only limited effort to compare against prior work; this is certainly hard in some cases, as some of the early work comes without code (or code that is hard to run today), but a broader discussion would be helpful.
  • The system is complex and likely not easy to reproduce. While the ideas are not that difficult to graps, getting this to actually work is hard (in my experience). This is a strength in terms of contribution but a weakness in terms of reproducibility (consider providing code if possible).
  • Minor: The notation is a bit confusing at times (e.g. n for time) and a bit hard to parse at first-time reading, but no serious issues here. I would still recommend a bit of further polishing.

Overall, I would see the nice and conceptually simple approach be valuable even if the paper leaves some room for improvements, but it is not a very strong submission.

Questions for authors

The paper did not make any statements wrt. to the use of "AI" / LLMs, in particular in terms of implementing the method. Could you comment on this?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.