acceptodds
Withdrawn from ICLR 2027

HeteroPhys: Heterogeneous Physical Property Identification from Videos

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recovering physical properties from videos is essential for predicting object dynamics beyond observed interactions, yet direct identification of spatially heterogeneous physical fields remains underexplored. We present HeteroPhys, a framework for recovering heterogeneous physical parameter fields from multi-view videos of deformable objects. HeteroPhys extracts Lagrangian motion and local deformation histories and represents material properties in constitutive coordinates separating volumetric and deviatoric responses. A hierarchical conditional diffusion model reconstructs fields from coarse structure to fine spatial variation, while permutation-invariant fusion integrates evidence from multiple interactions. The predicted dense fields are further refined through differentiable MPM optimization. We construct a benchmark with ground-truth heterogeneous fields across elastic and plasticine materials. Experiments show that HeteroPhys generalizes to unseen geometries, benefits from additional interaction evidence, and remains predictive under unseen interaction conditions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual system identification, heterogeneous physical property fields, inverse physics, multi-interaction inference, material property estimation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UDHEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formalizes the problem of heterogeneous physical system identification from multi-view observations across diverse interaction scenarios and proposes HETEROPHYS, a hybrid framework that combines conditional diffusion models with differentiable physics simulation and rendering. Moving beyond the conventional assumption of global material homogeneity, the framework generates dense, spatially varying physical fields using a conditional diffusion prior conditioned on multi-interaction observation histories via a permutation-invariant fusion mechanism. These fields are subsequently refined through differentiable Material Point Method (MPM) simulation and end-to-end rendering to achieve pixel-level and physical accuracy. Additionally, the authors construct a large-scale benchmark dataset with explicit ground-truth heterogeneous property fields and diverse interaction conditions.

Critical strengths

A major strength of this paper lies in its conceptual departure from the conventional, overly simplistic assumption of global material homogeneity, successfully enabling the recovery of complex, spatially varying internal physical fields. Furthermore, the work makes a compelling case for leveraging multiple interaction observations, demonstrating how they effectively eliminate the severe ambiguity inherent in inverse physical problems to narrow down the vast solution space. To operationalize this, the authors introduce a robust, permutation-invariant fusion mechanism that seamlessly incorporates multi-interaction observation histories, ensuring that complementary physical evidence is rigorously and order-independently aggregated across events.

Critical weaknesses

While the framework is innovative, it faces severe real-world applicability limitations. First, because dense internal physical ground-truth fields are unobservable in the real world, the model cannot be trained end-to-end on real data and relies entirely on synthetic datasets. Consequently, it suffers from a significant sim-to-real generalization gap, where real-world visual noise, motion blur, and mask inaccuracies can easily cause the tracking and inverse pipelines to fail. Furthermore, requiring multiple controlled physical interactions for the same object is an impractical assumption in real-world scenarios. Finally, real-world materials exhibit complex non-ideal behaviors that violate the assumed isotropic constitutive models, leading to severe model mismatch during differentiable refinement.

Questions for authors
  1. The framework heavily relies on front-end 3D Gaussian Splatting (3DGS) and local weighted least squares to extract the 12-dimensional physical history features. In real-world scenarios, front-end tracking inevitably suffers from temporal drift and artifacts. Given that the conditional diffusion model is trained exclusively on clean, noise-free synthetic data, how robust is the network to front-end tracking/velocity/shear changes/… errors, and what defensive mechanisms are in place to prevent catastrophic "garbage in, garbage out" cascading failures when noisy or drifted HobsH_{obs} features are fed into the generation pipeline?

  2. While the current evaluation setup relies on controlled object-to-ground interactions , it remains unclear whether this framework can be extended to handle complex object-to-object collisions (e.g., interactions between two unknown objects) rather than relying on a fixed, predefined plane. If so, how does the system disentangle the coupled, dual-unknown physical properties of both interacting entities without a pre-calibrated reference surface, and how drastically does the underlying solution space expand?

  3. While the method demonstrates strong shape generalization, generative priors can struggle when faced with unseen parameter distributions. To what extent can the diffusion model generalize to out-of-distribution physical parameter combinations (e.g., extreme localized stiffness gradients, rare core-shell thickness ratios, or complex multi-material topologies)?

  4. The pipeline requires pre-specifying the constitutive material family (e.g., isotropic elastic vs. von Mises plastic) and its specific parameterization coordinates. Can this framework successfully identify spatial physical fields without any prior specification of the constitutive material family?

Reviewer_Ui3DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the recovery of spatially varying material properties from multi-view videos of an object undergoing multiple interactions. The method first reconstructs the object, particle trajectories, and local deformation history, then constructs conditions using volumetric and shear decomposition along with corresponding material property coordinates. A coarse-to-fine conditional diffusion model is subsequently employed to predict the parameter fields, followed by projection into feasible domains and differentiable MPM optimization to obtain the final results. The authors create synthetic benchmarks for elastic and plastic materials containing ground-truth heterogeneous property fields, evaluate parameter recovery on unseen geometries, and freeze the identified parameters for dynamic prediction in unobserved interactions.

Critical strengths
  1. The paper distinguishes between motion fitting and true physical property identification, and provides a benchmark for heterogeneous materials with ground-truth parameter fields, enablingevaluation to directly assess property recovery rather than relying solely on similarity in reconstructed images or motion (Table 1). Freezing the identified parameters for unobserved interactions also offers a meaningful test of their transferability (Section 4.4,Table 3). The authors report single-interaction results in Table 4 using the same observation budget as the baseline. Cross-method comparisons use identical target sequences and evaluation metrics, explicitly excluding the use of target interaction frames for fitting, optimization, interaction selection, or early stopping (Appendix B.3).

  2. In Table 2, as the number of observed interactions increases from 1 to 4, the average R² improves from 0.0488 to 0.7255, indicating a significant gain from multiple interactions. Deformation preprocessing, volumetric -shear property coordinates, and multi-resolution structure are also directly supported by ablation studies (Tables 6–8).

Critical weaknesses
  1. The core heterogeneous field task lacks direct comparison. The cross-method comparison in Table 4 is limited to uniform fields. Existing ablation studies support some of the designs, but they cannot demonstrate how much benefit diffusion brings compared to deterministic regression and learning initialization compared to direct optimization.

  2. Table 11 does not clearly indicate whether the output has been projected. The existing results are difficult to distinguish the effects of projection and MPM optimization. The recovery performance of plastic materials also varies depending on the parameters; although the appendix already has mechanism analysis and MAE supplements, they have not fully explained these differences. The conclusion on recovery accuracy should be limited by the parameters.

Questions for authors
  1. Under the same conditions, how much benefit does diffusion bring compared to deterministic regression, and how much does learning initialization bring compared to direct optimization?

  2. Does Table 11 include feasible domain projection? Can you provide the comparison results of the output of the same diffusion at the projection before and after, and after MPM optimization?

  3. Can the existing analysis quantitatively explain the reason for the poor recovery of the Poisson ratio of plastic materials? Why is the R² of the log yield stress of the Gradient group very low, but the MAE is relatively small?

1 comment
Reviewer_Ui3DOpenReview ↗

Thank you for the author's reply. I'll keep my score.

Thank you for the author's reply. I'll keep my score.

Reviewer_vQDZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents HeteroPhys, a framework for identifying spatially heterogeneous physical properties from multi-view videos of deformable objects. The method reconstructs particle motion and local deformation histories using 3D Gaussians, predicts dense material-property fields through hierarchical conditional diffusion, and further refines the fields using differentiable MPM and rendering.

The authors construct a synthetic benchmark with ground-truth heterogeneous material fields for elastic and plasticine objects. Experiments evaluate field recovery on unseen geometries, the benefit of multiple observed interactions, and dynamics prediction under unseen poses and terrains.

Critical strengths

The integration of multi-view Gaussian-based reconstruction with an MPM particle representation, including the extraction of Lagrangian motion and deformation information, is a useful technical contribution, although largely incremental.

The multi-interaction experiments show substantial improvements in field recovery as additional observations are provided. Evaluation under unseen poses and terrains also provides useful evidence of predictive capability.

Critical weaknesses
  1. Limited novelty (see Q2–Q3). The framework largely combines existing techniques, including Gaussian-based motion reconstruction, conditional diffusion, and differentiable MPM refinement. The methodological contribution beyond these components is not sufficiently established.

  2. Limited benchmark diversity (see Q1). Although the dataset contains thousands of synthetic training records, it covers only a small number of geometries, predefined heterogeneous field patterns, and simple free-fall interactions. Its value as a general benchmark remains unclear, particularly without real-world validation.

  3. Inconsistent physical-property recovery (see Q4). Despite encouraging average field-recovery results, some decoded physical parameters, particularly Poisson's ratio and yield stress for Plasticine, show substantially weaker recovery performance (Appendix C.1), raising concerns about the reliability of identifying individual physical properties.

  4. Insufficient baselines and ablations (see Q2–Q3). The experiments do not clearly isolate the contributions of diffusion and MPM refinement.

  5. Insufficient support for the central motivation (see Q5). The paper emphasizes identifying intrinsic physical properties rather than merely reconstructing motion. However, despite direct field-level evaluation, the experiments do not convincingly establish when accurate physical identification provides benefits beyond motion fitting, or how well these benefits extend beyond the controlled synthetic setting.

Questions for authors

Q1. Benchmark generalization. Could the authors evaluate generalization to heterogeneous field distributions beyond the four predefined heterogeneous patterns, or provide evidence that the synthetic benchmark captures sufficiently diverse material behaviors?

Q2. Heterogeneous identification baselines. While Table 4 compares against prior methods on homogeneous materials, the central heterogeneous setting lacks comparable baselines. On heterogeneous test cases, how does the method compare with a fitted homogeneous-material approximation or direct optimization of spatially varying parameters? How much does heterogeneous identification improve unseen-interaction prediction?

Q3. Necessity of diffusion and refinement. Could the authors compare diffusion with a simpler deterministic regression model using the same training data and features? Also, could they clarify the relationship between the raw inference results in Table 11 and the final refined results, and whether MPM refinement actually improves physical-field recovery?

Q4. Physical-property accuracy. Given the weaker decoded results for some Plasticine parameters (Appendix C.1), which physical properties can be reliably identified? Could the authors provide further analysis of errors in the recovered internal fields?

Q5. Physical identification vs. motion fitting. Figure 8 provides an initial analysis of how material-field errors affect simulated motion. Could the authors further demonstrate whether material fields that fit the observed motion similarly but differ in physical accuracy lead to different predictions under unseen interactions? This would better establish the value of physical identification beyond motion fitting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.