acceptodds
Under review as a conference paper at ICLR 2027

LPSLat: Learning Physical Priors on Structured Latents for Single-Image Generated 3D Assets

Abstract

Single-image 3D generators produce textured assets, but physical simulation requires material properties that vary across object parts. We introduce LPSLat, a dual-branch network that predicts material type, density, Young’s modulus, and Poisson’s ratio for single-image generated 3D assets. The network combines complementary information from structured 3D latents and the source image: the latent branch predicts a base physical field using local features and object context, while the image branch refines it through gated appearance-based corrections. The predicted properties are attached directly to the generated mesh, without additional 3D reconstruction or per-object optimization. For training, we transfer reference-part annotations through the shared source view, accommodating geometric differences between reference and generated assets and filtering ambiguous correspondences. Together, these components connect single-image asset generation with spatial physical-property prediction and downstream physics-based simulation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
physical-property prediction, structured latent representations, single-image 3D generation, material estimation, multimodal learning, physics-based simulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5gPPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This method generates material properties like material class, density, Young's modulus and Poisson ratio on the Trellis generated results. The method use a local/global latent predictor to suppply base filed, projected CLIP features for global residual corrections. The method also used view alignments, visiblity checks and boundary filtering to transfer annotations from reference to genearated voxels. The evaluation shows reasonable improvments against existing methods.

Critical strengths

The basic solution is reasonable and practical. The method is also supported by evluations like the macro-F1 accuracy improvements.

The ablation study also shows the benifit of each design, altough the numbers are all closed to each other.

Critical weaknesses

I have some concern on the main objective of the paper, is the phsical properties really need to be 3D native, more importantly, need to be 3D input only? Do we need a 3D based model for this, or we better rely on VL-models?

The evaluation is on CM-25 rather than the full test set. The overall material taxonomy is somewhat coarse.

The improvements are varies on different sources, maybe the authors can have more input on this.

Also, I really don't know what is the proper way to evaluate the accuracy in practice since different outputs are in very different non-linear domain, is MAE really meaningful here?

Questions for authors

I would like the authors to discuss their motivation on the oveall design. Why train a model without leverging the knowledge of LLM-VLs. They can provide much better annotation for the input image or a rendred image, than a train-from-stratch 3D-native model.

I think the authors should at least include a basic VL benchmark here, the method can be simply render the image on multi-views and ask VL model to annotate, then back-project to voxels.

How the method works for occluded parts? Especially on alignment quality?

Having a more reasonable way for the reader understand the accuracy number means? How does this compares with a homogeneous estimation?

What is the relationship to part-segmentation? If the object is well segmented will this material assignment getting better?

11 comments
AuthorsOpenReview ↗

1. Why use a lightweight predictor rather than directly query a VLM?

General-purpose VLM outputs cannot be assumed to provide reliable physical parameters without validation. PhysX-Anything explicitly notes that GPT-5 is not fully reliable for fine-grained physical reasoning when discussing its evaluator. PhysX-Omni uses GPT to produce preliminary physical annotations and then has human annotators verify and refine them. These practices show why direct VLM outputs require additional checking before being used as physical supervision.

Task-specific VLM training also requires substantial computational resources. PhysX-Anything fine-tunes Qwen2.5-VL-7B on 32 A800 GPUs for five days, while PhysX-Omni trains its VLM on 64 A100 GPUs for approximately fourteen days. These models address broader asset-generation tasks, including geometry and physical attributes. Our goal is narrower: predict spatial physical properties for an asset already produced by a single-image generator.

LP²SLat therefore reuses frozen TRELLIS and CLIP representations and trains only a 2.36M-parameter predictor. It is neither a 3D-input-only model nor a model that learns visual knowledge from scratch. TRELLIS provides spatial structure and appearance features, while the source image supplies local material cues. The submitted ablation shows that adding the image branch increases material F1 from 0.429 to 0.585. The prediction head takes 0.008 seconds per asset on an NVIDIA A40, excluding generation, feature extraction, and export.

This design provides a locally deployable physical-property predictor without task-specific VLM fine-tuning or external API calls during prediction.

AuthorsOpenReview ↗

2. Evaluation beyond CM-25 and material taxonomy

CM-25 is the cross-pipeline comparison, not the entire test evaluation. It contains 13 PhysX-Mobility and 12 PixieVerse objects. The representation and supervision-transfer ablations use the larger object-disjoint test split: 225 of the 262 held-out objects have accepted physical targets.

On that larger evaluation, Table 2 reports material F1 of 0.585 for LP²SLat, 0.429 for the latent-only model, and 0.216 for the SLAT-Phys-style reimplementation.

The paper distinguishes two material metrics:

  • Fine Material F1: our 16-class material vocabulary, evaluated where fine-material labels are available.
  • Coarse Constitutive F1: a three-group mapping used for cross-pipeline comparison when the original label spaces differ.
AuthorsOpenReview ↗

3. Differences between data sources

The source-wise differences likely reflect both object composition and how closely each source matches a method’s training data and representation. PhysX-Mobility mainly contains multipart household objects, where physical-property prediction requires distinguishing materials across components. LP²SLat combines spatially organized 3D features with local image cues, which is well suited to this setting. On the PhysX-Mobility portion of CM-25, its Young’s modulus log-MAE is 0.713, compared with 6.480 for PIXIE.

PixieVerse contains a different mix of objects, including vegetation, toys, balls, and granular-appearance assets. PIXIE was developed using PixieVerse, and our comparison uses its released weights and multi-view visual features. This closer match to its training domain may help explain its lower Young’s modulus error on the PixieVerse portion: 1.192 versus 1.590 for LP²SLat.

Reviewer_iyVPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LP²SLat, a method for predicting spatially varying material and physical properties for 3D assets generated from single images. The approach leverages the latent space of TRELLIS to construct a physical property field, which is then refined using CLIP features—extracted from the original image via projection sampling—through learned gated residuals. To address discrepancies between reference assets and generated geometry, the authors propose an annotation transfer pipeline based on shared source views; this pipeline incorporates visibility checks, boundary erosion, and label purity filtering to establish supervision for training. Furthermore, the paper introduces the PhysSLat-227 dataset.

Experimental evaluations include performance assessments on the object-split PhysX–Pixie dataset, cross-method comparisons on the CM-25 dataset, and MPM-based simulations under controlled conditions. The results demonstrate that, compared to baseline methods, this approach improves material recognition, reduces prediction errors for physical properties, and yields produces deformation trajectories closer to reference-annotation-conditioned simulations under the same MPM solver and loading conditions.

Critical strengths

1.Compared to the baseline method, this approach achieves higher accuracy in material property estimation, with significant reductions in prediction errors for density and Young’s modulus. Ablation studies demonstrate that incorporating features from the original images effectively complements the 3D latent variables.

2.This paper employs source-view transfer to mitigate supervision challenges arising from geometric inconsistencies between the reference and generative model. Under otherwise identical conditions, the trajectory error is reduced to 24 out of 25 objects compared to the baseline method, indicating that the simulated deformations are closer to the reference simulations.

Critical weaknesses

1.As shown in Table 5, the voxel coverage of the proposed method is only 15.2%. While this indicates that the predictions for the retained voxels are quite accurate, The authors have not demonstrated whether these covered voxels are representative of the physical parameter predictions for the entire object.

2.The mechanism by which image information is incorporated via the gating approach remains somewhat ambiguous. Since simply concatenating the original images would also introduce a wealth of raw information—such as details regarding different materials—it is difficult to discern whether the performance improvement stems from the gating mechanism itself or merely from the inclusion of the original image data.

Questions for authors

1.When transferring material and physical parameter annotations from a reference model to a generated model, discrepancies inevitably arise due to the inherent differences between the two. In this paper, "boundary erosion" and "label-purity filtering" are employed to mitigate unreliable annotations. To what extent do these techniques improve the accuracy of the annotation correspondence, and what is their impact on the final evaluation results?

2.In Equation (8), the density and Young’s modulus depend explicitly on material category priors; consequently, more accurate material identification can, in itself, reduce errors in these continuous parameters. Thus, the reduction in error stems from two sources: the learning of continuous parameters within local regions and the improved accuracy of material classification. How, then, can we disentangle the respective contributions of these two factors to the final results?

5 comments
AuthorsOpenReview ↗

W1. Prediction coverage and evaluation coverage

The 15.2% in Table 5 measures supervision coverage, not prediction coverage. The SLat branch predicts physical properties at every point. Parameter metrics are computed only at sites with valid annotations.

All transfer variants are evaluated against the same fixed targets. Their accuracy differences therefore do not arise from evaluating each method on its own retained subset.

The submitted simulations provide complementary evidence of downstream usefulness. With geometry, solver, and loading conditions held fixed, LP²SLat reduces trajectory error relative to SLAT-Phys-style on 24 of 25 objects, with reductions of at least 25% on 20 objects. These results support the usefulness of the predicted fields for deformation simulation from whole-object simulation agreement.

AuthorsOpenReview ↗

W2. Contribution of learned gating beyond image features

The submitted image-branch ablation shows that adding source-image information increases material F1 from 0.429 to 0.585. To examine the fusion mechanism separately, we provide completed comparisons with two simpler alternatives:

  • Concatenation + MLP: combines SLat and source-image features before prediction.
  • Fixed-gate residual: retains the image residual branch but removes learned gating.
  • Learned-gate residual: uses the LP²SLat gating mechanism.
Fusion strategyMaterial F1 ↑log⁡ρ\log\rho MAE ↓log⁡E\log E MAE ↓ν\nu MAE ↓
Concatenation + MLP0.4200.3190.8650.0188
Fixed-gate residual0.5450.3390.9620.0209
Learned-gate residual0.5630.3200.8510.0210

Compared with fixed-gate residual fusion, learned gating reduces Young’s modulus error by 11.6%. The paired 95% confidence interval for the error difference is [−0.218,−0.011][-0.218,-0.011]. Its material-F1 improvement is +0.018+0.018, with an interval of [−0.043,0.063][-0.043,0.063].

Compared with concatenation, learned gating has a higher material-F1 point estimate. Concatenation achieves lower Poisson’s ratio error.

AuthorsOpenReview ↗

Q1. Effects of boundary erosion and label-purity filtering

The two filters address different sources of ambiguous supervision. Boundary erosion removes pixels near reference-part boundaries, where small alignment errors can change the assigned part. Label-purity filtering rejects voxel assignments whose supporting pixels disagree on the part label.

The submitted results in Table 5 and Appendix Table C.5 show:

Supervision transferVoxel coverageMaterial F1 ↑log⁡ρ\log\rho MAE ↓log⁡E\log E MAE ↓
ICP50.7%0.3480.4211.094
Source-view transfer without both filters18.2%0.5310.3330.927
Full source-view transfer15.2%0.5850.3180.914

The full strategy improves material F1 over ICP by 0.237, with a paired 95% interval of [0.121,0.311][0.121,0.311]. This supports source-view supervision when reference and generated geometry differ.

Adding the two filters reduces supervision coverage from 18.2% to 15.2% and improves material F1 by 0.054 in point estimate. Density and Young’s modulus errors also decrease slightly.

Reviewer_gQmnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents LP²SLat, a feed-forward network for predicting spatial material types and physical properties of 3D assets generated from a single image. The method combines structured 3D latents generated by TRELLIS with source-image features to predict physical-property fields through gated residual fusion, establishes supervision correspondences between reference and generated assets through their shared source view, and constructs PhysSLat-227, a dataset with part-level physical-property annotations. Together, these components form a framework for predicting spatially varying physical properties of single-image generated 3D assets, enabling the generated objects to be used in downstream MPM-based physical simulations. The experimental results and ablations show that the proposed method outperforms the compared baselines in material classification and physical-parameter prediction, and improves the agreement between MPM simulation trajectories and reference results.

Critical strengths

S1: An efficient feed-forward framework for single-image physical-property prediction.

The method predicts spatially varying material fields in a feed-forward manner without requiring per-object optimization. It combines the 3D structural information from SLat with appearance information from the original image to improve material classification and physical-parameter prediction. The predicted results can be directly used in downstream MPM simulations.

S2: An effective cross-asset supervision strategy for physical properties.

The method establishes correspondences between reference and generated 3D assets through their shared source image, avoiding the difficulties of direct 3D registration when their geometries differ. In addition, boundary erosion and label-purity filtering reduce ambiguous supervision labels. Experiments show that this strategy significantly improves physical-property prediction performance compared with ICP-based label transfer.

Critical weaknesses

W1: A single constitutive model limits the representation of heterogeneous material behavior.

Although the paper predicts 16 material classes and spatially varying physical parameters, all downstream simulations adopt the same Neo-Hookean hyperelastic constitutive model, with different materials characterized only by variations in Young's modulus, Poisson's ratio, and density. This formulation cannot adequately capture complex material behaviors such as plasticity, viscoelasticity, and anisotropy, limiting the applicability of the framework to diverse real-world materials and complex deformation scenarios.

W2: Insufficient description of the construction and physical-parameter annotation process of PhysSLat-227.

The authors present PhysSLat-227 as one of the main contributions, but the main text lacks detailed explanations of part-level material annotations, the sources of physical parameters, and annotation quality control. Therefore, it is difficult to assess the extent to which these parameters reflect the actual physical properties of real objects. Furthermore, the main quantitative experiments are conducted on the PhysX–Pixie dataset, leaving the independent experimental value of the newly constructed PhysSLat-227 dataset unclear.

W3: Insufficient validation of prediction uncertainty and generalization when estimating physical parameters from static visual information.

The paper predicts material properties using only static images and their generated 3D representations, although physical parameters such as density and stiffness generally cannot be uniquely determined from static appearance alone. Therefore, the model largely relies on statistical correlations between visual features and material properties learned from the training data. The paper lacks sufficient analysis of materials with similar appearances but different physical properties, unseen material distributions, and prediction uncertainty, making it difficult to assess the reliability of physical-parameter prediction in out-of-distribution scenarios.

Questions for authors

Q1. Necessity of Gated Fusion

Why are ablation comparisons with simpler fusion strategies, such as Concatenation + MLP and Residual Fusion without Gate, missing? How can the authors demonstrate that the performance improvements come from Gated Fusion itself rather than merely from the additional image features?

Q2. Missing VLM Baselines

The paper discusses VLM-based methods such as PhysX-Anything and PhysX-Omni. Why are these methods not included in the experimental comparisons? If their tasks and output representations differ, could the authors explain why they are not suitable as baselines?

Q3. Annotation Process and Reliability of PhysSLat-227

How are the part-level material classes and physical parameters annotated in the PhysSLat-227 dataset? Are the density, Young’s modulus, and Poisson’s ratio values obtained through VLM-based annotation or manually assigned empirical priors? How do the authors ensure the accuracy and physical plausibility of these annotations?

6 comments
AuthorsOpenReview ↗

Response to Reviewer gQmn: Q2. VLM-based comparisons

PhysX-Anything and PhysX-Omni use task-specific VLM training with substantial computational cost. PhysX-Anything reports fine-tuning Qwen2.5-VL-7B on 32 A800 GPUs for five days, while PhysX-Omni reports training on 64 A100 GPUs for approximately fourteen days. Their scope includes geometry and physical asset generation. Our goal is more focused: adding spatial physical properties to an already generated asset. LP²SLat freezes TRELLIS and CLIP and trains only a 2.36M-parameter predictor, avoiding task-specific VLM fine-tuning.

Directly querying a general-purpose VLM also requires validation of its physical predictions. PhysX-Anything explicitly notes limitations of GPT-5 in fine-grained physical reasoning when discussing its evaluator, and PhysX-Omni uses human verification to refine GPT-generated physical annotations. These observations motivate task-specific supervision rather than treating general-purpose VLM outputs as validated physical parameters.

LP²SLat runs locally with fixed model weights and requires no external API calls during prediction. This provides control over the deployed model and removes dependence on an external inference service. The evidence above motivates our design.

AuthorsOpenReview ↗

Response to Reviewer gQmn: Q1. Concatenation and ungated residual fusion

We provide completed comparisons with two simpler fusion strategies:

  • Concatenation + MLP: combines SLat and source-image features before predicting the physical field.
  • Fixed-gate residual fusion: retains the residual image branch but removes learned gating.
  • Learned-gate residual fusion: uses the LP²SLat gating mechanism.

Within this comparison, all three models share the same configuration and training schedule. They use 1,935 training objects—1,809 eligible PhysX–Pixie objects and 126 PhysSLat objects—with a maximum of 120 epochs, compared with the submitted 50-epoch budget. Validation and accepted test sets contain 229 and 225 objects, respectively. The results are:

Fusion strategyMaterial F1 ↑log⁡ρ\log\rho MAE ↓log⁡E\log E MAE ↓ν\nu MAE ↓
Concatenation + MLP0.4200.3190.8650.0188
Fixed-gate residual0.5450.3390.9620.0209
Learned-gate residual0.5630.3200.8510.0210

The learned gate reduces Young’s modulus error by approximately 11.6% relative to the fixed-gate residual. The paired 95% confidence interval for the error difference is [−0.218,−0.011][-0.218,-0.011]. Its material F1 is also higher in point estimate.

Compared with concatenation, the learned gate has higher material F1 in point estimate, while concatenation achieves lower Poisson’s ratio error. These results support a benefit of learned gating for stiffness prediction relative to an ungated residual and show that fusion design affects the balance between classification and continuous-property accuracy.

AuthorsOpenReview ↗

Response to Reviewer gQmn: W3. Ambiguity of physical properties from appearance

Static appearance does not uniquely determine density or stiffness, but it provides useful cues for estimating material-dependent physical properties. LP²SLat combines these cues with pretrained TRELLIS features, which encode the generated asset’s 3D structure and appearance. The predictor therefore uses both local visual evidence and object-level context to estimate physical priors.

The submitted experiments support the usefulness of this representation across the evaluated objects. The latent-only predictor achieves a material F1 of 0.429, and adding gated source-image features increases it to 0.585. Under controlled simulation conditions, the predicted fields also reduce trajectory error relative to SLAT-Phys-style on 24 of 25 objects. These results show that the learned priors support useful physical-property prediction despite the ambiguity of static appearance. They establish performance on the evaluated distribution, rather than calibrated uncertainty on unseen material distributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.