acceptodds
Withdrawn from ICLR 2027

PIM: A Low-Latency Physical Intuition Model for Predicting Object Interaction Outcomes

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Predicting object interaction outcomes from initial conditions is a central problem in data-driven physical reasoning. Existing methods often use rigid-body-level states or predict the evolution through autoregressive rollouts. Coarse representations cannot represent local deformation, while rollout errors may compound over long prediction horizons. Instead, we introduce PIM, a Physical Intuition Model that directly predicts a physical system's equilibrium state from its initial mesh geometry, kinematics, and physical attributes. PIM combines local k-nearest-neighbor (KNN) attention to encode intra-object geometry, anchor-based cross-object attention to model interactions, and a dual-branch decoder that predicts object-level translations and per-vertex residual displacements. Across MuJoCo scenarios involving both rigid and deformable interactions, PIM achieves lower vertex and centroid errors than the evaluated baselines. By predicting complete equilibrium states in a single forward pass, PIM substantially reduces latency relative to autoregressive learned models and MuJoCo rollouts under the evaluated protocols, enabling low-latency inference when only the final outcome is required.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
learned physical simulation, physical intuition, terminal mesh prediction, multi-object interaction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NfagOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PIM, a model that directly predicts the equilibrium meshes of interacting objects from their initial geometry, kinematics, and physical attributes, without generating intermediate trajectories. Its architecture combines local KNN attention for intra-object geometry, anchor-based cross-object attention for interactions, and a dual-branch decoder that separates object translation from per-vertex residual displacement. Experiments on four controlled MuJoCo scenarios involving rigid and deformable objects show lower terminal-state prediction errors than the evaluated baselines and reduced inference latency compared with autoregressive models and simulator rollouts under the reported protocols.

Critical strengths

Predicting the final configuration directly is a well-motivated problem when intermediate trajectories are not needed. The vertex-level output provides a common representation for rigid motion and local deformation, rather than restricting every object to a rigid pose. This gives the method a useful role as a terminal-state predictor, without requiring it to replace a full dynamics simulator.

The accuracy improvements are consistent across the four evaluated scenarios. PIM also outperforms the force-conditioned rollout baselines in the controlled Flat experiment, and Small-PIM retains much of this advantage at a substantially reduced parameter count. These results suggest that the approach has practical merit within the studied settings. I also appreciate the inclusion of physical-validity diagnostics rather than relying only on vertex error.

Critical weaknesses

My main concern is the gap between predicting a nearby terminal mesh and predicting a valid equilibrium state. On Flat, only 10.90% of scenes satisfy the paper’s inter-object non-penetration criterion, and contact F1 is 13.16% (Table 2). Table 14 reports an average per-scene maximum penetration of 13.967 mm, suggesting that the failures are not merely borderline cases around the 1 mm threshold. The non-penetration extension helps, but even its strongest setting reaches only 53.60% penetration-free scenes, while contact F1 falls to 11.11%. Moreover, the validity evaluation excludes the supporting plane and does not assess force balance or frictional feasibility. The authors acknowledge these limitations, but they remain consequential: lower vertex error alone does not establish that the predicted configuration is a usable approximation to static equilibrium.

I am also not convinced that the experiments establish a sufficiently substantial methodological contribution. Local attention, cross-object communication, and centroid–residual decoding form a sensible design, but the ablations mainly show that objects need to exchange information. Removing cross-object attention does not establish that the proposed FPS-anchor mechanism is better than another reasonable interaction module. The decoder ablation is more informative, although its effect is small on Bumpy and Deformable and is reported from single runs. The full-dataset comparisons use two direct predictors developed for this study, while the published rollout methods are evaluated only on controlled Flat. The direct baselines are not matched to PIM in architecture capacity or training schedule, and their inference times are absent from the runtime table. This makes it difficult to separate the benefit of PIM’s particular design from the benefit of predicting only the terminal state. The restricted evaluation—separate scenario-specific models, fixed object correspondences and mesh topologies—also provides limited evidence that the contribution extends beyond these settings.

Questions for authors

Regarding the equilibrium claim, is there evidence that predicted configurations remain approximately stationary when used to initialize the simulator? Support-plane penetration and displacement during a short settling rollout would help determine whether low geometric error corresponds to a physically usable state. This is a request for evidence supporting the current claim, rather than a suggestion to model full trajectories.

Can the authors provide a comparison with a direct terminal-state predictor matched more closely in capacity, inputs, losses, and training schedule, reporting both accuracy and latency? Replacing the FPS-anchor interaction with another cross-object mechanism under the same setup would be particularly informative. This would directly affect my assessment of how much the proposed architecture contributes beyond the terminal-prediction formulation.

For the shape-OOD experiment, can the authors report quantitative errors separately for the replaced cube? Only one object changes, and it retains the original vertex count and topology, whereas the reported vertex metric averages over the entire scene. Object-specific results would make it easier to assess the demonstrated shape transfer.

Reviewer_yUfgOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

PIM directly predicts the final mesh configuration of interacting rigid and deformable objects from their initial geometry, kinematics, and physical attributes. It combines local KNN attention, FPS-anchor cross-object attention, and a center-relative decoder, supervised by coordinate, centroid, and final edge-length losses. Separate models are evaluated on four MuJoCo scenarios, with additional controlled rollout comparisons, Jenga distribution shifts, ablations, and contact-validity diagnostics.

Critical strengths

From the experimental results, PIM improves vertex and centroid errors over both direct baselines in all four scenarios. The controlled Flat comparison includes shared test scenes, force-adapted rollout baselines, and a smaller PIM variant. The experiment settings, dataset processing, and training details are fully elaborated in the appendix.

Critical weaknesses
  1. The practical value of the paper needs clearer validation. PIM takes initial meshes, velocities, forces, density and friction as given, while the experiments use simulator-generated inputs; robustness when these quantities must be estimated is not evaluated. Terminal prediction is useful when final placement is sufficient, and the latency results support that setting, but it cannot provide intermediate states for trajectory-dependent decisions.

  2. The experiments only validate on 4 settings on MoJoCo, whereas physical reasoning is most prominent in generalization. All baselines achieve predictions close to GT, with errors in centimeters. This may suggest overfitting to the data, where the performance improvements are due to larger parameter size, as Table 1 shows. Transfer across scene compositions, object counts, topologies and broader deformable settings remains untested.

Questions for authors
  • Quantify contact and support validity beyond Flat, or show that the intended final-outcome judgments remain reliable despite the current contact errors? Evidence connecting the reported placement gains to a reliable outcome would materially affect my assessment.

  • How does FPS-anchor cross-object attention compare with pooled-object interaction under matched width, depth, losses, and coarse-to-fine training?

Reviewer_2UE4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PIM, a model that predicts the final equilibrium mesh of interacting objects in one forward pass from their initial meshes, kinematics and physical attributes, without autoregressive rollout. It combines KNN attention within objects, FPS-anchor cross-attention between objects, and a decoder that splits centroid translation from per-vertex residuals. It reports better performance on four MuJoCo scenarios and large speedups compared to MuJoCo. However, the problem setting is too narrow to be practical and it is not compared to low-latency simulation methods.

Critical strengths
  1. Predicting the final state at the mesh-vertex level covers both rigid motion and local deformation in one representation.
  2. The feedforward setup is faster than autoregressive methods.
Critical weaknesses
  1. Narrow setting and limited accuracy. A separate model is trained per scenario, with a fixed number of objects and fixed mesh topology. It is unclear how this can generalize to the claimed practical application. Limited evidence is shown that this works for OOD data.
  2. Weak comparisons. The authors compare the proposed method to MuJoCo and set the precision for MuJoCo to be unnecessarily high, leading to latency that might be exaggerated. Common real-time physics engines such as PhysX also simulate such scenes very fast.
  3. Not explainable. The one-shot prediction is not explainable, and I don't think this is the correct way to model the actual task, i.e., physics simulation, which is inherently sequential in time and sensitive to initial conditions. The fact that it is trained separately per scenario indicates it is more like a scenario-specific mapping between initial conditions and final state, fitted to one fixed setup, rather than really learning the underlying physics.
Questions for authors
  1. How does a lower-fidelity MuJoCo setting with latency similar to PIM perform?
  2. Can one model handle a different number of objects or a different mesh topology without retraining?
  3. How does a feed-forward network predict scenes that is naturally chaotic and sensitive to initial values? It reads like a brute-force average of what happens within a scene given fixed objects, etc.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.