acceptodds
Under review as a conference paper at ICLR 2027

PhysForge: Material-Controllable 3D Reconstruction with Separable Components

Abstract

Recent advances in physical 3D generation have enabled visually faithful assets with simulatable mechanical behavior. However, existing methods provide limited control over material behavior beyond appearance-based inference and struggle to estimate physical properties from complex interactions between objects and semantic parts. Therefore, we present PhysForge, a Video Diffusion Model (VDM)-based framework for addressing these challenges, comprising two complementary modules for material-controllable physical fitting and componentlevel interaction modeling. The first, PhysFit, leverages material-conditioned VDM dynamics as references for physical-property fitting, introducing windowed fitting to focus on material-revealing interactions and Hallucinated Forcing to disentangle nonphysical motion hallucinated by the VDM from material responses. The second, PhysComp decomposes complex interactions into individual objects and semantic components, enabling their physical parameters to be optimized independently and flexibly composed for joint simulation. Experiments on PhysAssets and Google Scanned Objects demonstrate that, compared with existing methods, PhysForge provides stronger control over material-dependent physical responses while supporting component-level physical modeling for complex interactions. It achieves the best CLIP and physical controllability scores on both common and uncommon materials, with qualitative comparisons, component-level ablations, and a user study further supporting these improvements.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
material-controllable, component separable, 3D reconstruction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5Mw2OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PhysForge, a framework for reconstructing simulation-ready 3D assets with controllable material behavior and separable semantic components. PhysForge integrates two key modules: PhysFit and PhysComp. PhysFit uses dynamics generated by a material-conditioned video diffusion model as reference motion, estimates physical properties through windowed fitting, and handles nonphysical motion through a Hallucinated Forcing mechanism. PhysComp transfers object-part identities from video to the reconstructed 3D Gaussian geometry, organizes these components as simulated rigid bodies, and enables independent optimization of their physical parameters followed by joint simulation. This design allows the same object to exhibit distinct material-dependent responses under different material prompts while preserving the motion and interaction characteristics of individual semantic components.

Critical strengths

PhysForge supports user-specified materials such as wood, silicone, fabric, metal, plastic, rubber, liquid, and sand, and allows the same object to exhibit different material-dependent behaviors. It also preserves component-level motion, e.g., bottle, glass, and table moving separately.

Critical weaknesses
  1. The quality of physical-parameter fitting relies on the generated video’s physical plausibility. Although Hallucinated Forcing mitigates nonphysical motion, it does not fully solve the problem, and severe VDM hallucinations may still bias the fitted parameters.

  2. Each material condition requires additional video generation and physical fitting. This makes PhysForge less efficient than feed-forward or single-pass asset-generation methods, and limits scalability to many material variations or large asset libraries.

Questions for authors
  1. Recently, several methods have used VLMs to achieve soft-body/deformable object deformation, such as PhysX-CoT and PhysX-Omni. The authors should analyze the differences from and relationship to these methods in the related work.

  2. I feel that this pipeline is more like an agentic framework built on existing models, and thus it tends to perform better when compared with methods based on a single model. However, I hope the authors can provide corresponding evaluations, such as efficiency.

5 comments
AuthorsOpenReview ↗

Response to reviewer 5Mw2 (1/3)

Thank you for recognizing material controllability and component-level motion. We address each concern below.

  1. W1: The quality of physical-parameter fitting relies on the generated video’s physical plausibility. Although Hallucinated Forcing mitigates nonphysical motion, it does not fully solve the problem, and severe VDM hallucinations may still bias the fitted parameters.

We would like to clarify that, compared with previous works, PhysForge is the first to explicitly discuss how video hallucinations affect physical-parameter fitting. To address this issue, we propose window selection and Hallucinated Forcing, combined with supervised video generation. Window selection concentrates fitting on contact and deformation, and Hallucinated Forcing aligns the incoming motion state. Eliminating video hallucinations remains challenging for methods that rely on generated dynamics, including Vid2Sim [1], DreamPhysics [2], and MotionPhysics [3], and our approach mitigates their influence rather than fully resolving the problem. Nevertheless, Table 1 and Figure 3 show improved material controllability: PhysForge achieves common/uncommon PC scores of 4.6/4.0, compared with 3.2/2.6 for DreamPhysics. Loss in the following table measures how closely the simulated contact and deformation match the generated reference video; a lower loss indicates better fitting to the reference dynamics.

Window selectionHallucinated ForcingLoss ↓
✗✗0.99154
✓✗0.14242
✓✓0.05864

Window selection isolates the relevant interaction, while Hallucinated Forcing aligns the incoming state; enabling both yields the lowest loss. In Figure 6 in Appendix (newly added), fitting the full video produces watermelons that behave too stiffly, indicating that motion outside the interaction can interfere with material fitting. Window selection alone produces a slow collision with insufficient deformation: a soft material may deform only slightly when the incoming velocity is too low. Hallucinated Forcing addresses this ambiguity by aligning the incoming state, allowing the fitted material to reproduce the requested soft-material behavior. The quantitative and qualitative results therefore support the complementary roles of window selection and Hallucinated Forcing in material fitting.

References

[1] Chen, Chuhao, et al. "Vid2Sim: Generalizable, Video-Based Reconstruction of Appearance, Geometry and Physics for Mesh-Free Simulation." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

[2] Huang, Tianyu, et al. "DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priors." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 4. 2025.

[3] Wang, Miaowei, et al. "MotionPhysics: Learnable Motion Distillation for Text-Guided Simulation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 40. No. 12. 2026.

AuthorsOpenReview ↗

Response to reviewer 5Mw2 (2/3)

  1. W2: Each material condition requires additional video generation and physical fitting. This makes PhysForge less efficient than feed-forward or single-pass asset-generation methods, and limits scalability to many material variations or large asset libraries.

Our generation-and-fitting pipeline supports material-controllable variation for the same asset, a capability absent from the evaluated observation-driven PhysX predictors. PhysX-Omni [1] and PhysX-Anything [2] predict material properties from the appearance of the object, whereas PhysForge fits material properties according to the user’s prompt. Given the same apple image, a user may request hardwood or silicone gel. The PhysX methods produce the same material prediction for both requests, whereas PhysForge fits distinct material properties to produce the requested hard or soft behavior. This use of dynamic priors follows the optimization-based direction of DreamPhysics [3] and MotionPhysics [4].

We report material controllability together with runtime:

MethodMaterial controllabilityRuntime (min)
PhysX-Omni✗16.69 ± 1.89
PhysX-Anything✗11.82 ± 1.90
PhysForge✓11.75 ± 0.56

PhysForge supports user-specified materials with a runtime close to PhysX-Anything and lower than PhysX-Omni. The difference in capability is visible in Figure 7 in Appendix (newly added): the PhysX methods retain the same behavior across the hard and soft conditions, while PhysForge produces a largely rigid apple under the hard condition and pronounced compression under the soft condition.

References

[1] Cao, Ziang, et al. "PhysX-Omni: Unified Simulation-Ready Physical 3D Generation for Rigid, Deformable, and Articulated Objects." arXiv preprint arXiv:2605.21572 (2026).

[2] Cao, Ziang, et al. "PhysX-Anything: Simulation-Ready Physical 3D Assets from Single Image." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

[3] Huang, Tianyu, et al. "DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priors." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 4. 2025.

[4] Wang, Miaowei, et al. "MotionPhysics: Learnable Motion Distillation for Text-Guided Simulation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 40. No. 12. 2026.

  1. Q1: Recently, several methods have used VLMs to achieve soft-body/deformable object deformation, such as PhysX-CoT and PhysX-Omni. The authors should analyze the differences from and relationship to these methods in the related work.

The distinction is that PhysForge is a material-controllable generation method, whereas PhysX methods are observation-driven material predictors. PhysX-CoT [1], PhysX-Omni [2], PhysX-Anything [3], and PhysGM [4] predict an object’s physical parameters from an image that provides an observation of its appearance. PhysForge instead takes images of different objects and fits physical parameters according to the user’s material requests. Its goal is to give the reconstructed objects the requested physical behavior, rather than recover the material implied by their appearance.

We compare with PhysX-Omni and PhysX-Anything. PhysX-CoT is built on PhysX-Anything. Since the dedicated LoRA weights required by PhysX-CoT were unavailable for our experiments, we use PhysX-Anything as the alternative baseline.

MethodMaterial controllabilityCommon CLIP ↑Common PC ↑Uncommon CLIP ↑Uncommon PC ↑
PhysX-Omni✗0.2493.80.2472.6
PhysX-Anything✗0.2492.20.2481.4
PhysForge✓0.2584.60.2644.0

PhysForge achieves higher CLIP and PC scores under both common and uncommon material conditions. In Figure 7 in Appendix (newly added), PhysX-Omni and PhysX-Anything produce unchanged behavior across the two requested materials, and the PhysX-Anything result has incomplete geometry. PhysForge preserves the apple appearance while producing the hard and soft deformation behaviors requested by the user.

References

[1] Huang, Jie, et al. "PhysX-CoT: Structured Physical Reasoning from a Single Image to Simulation-Ready 3D Assets." arXiv preprint arXiv:2608.08053 (2026).

[2] Cao, Ziang, et al. "PhysX-Omni: Unified Simulation-Ready Physical 3D Generation for Rigid, Deformable, and Articulated Objects." arXiv preprint arXiv:2605.21572 (2026).

[3] Cao, Ziang, et al. "PhysX-Anything: Simulation-Ready Physical 3D Assets from Single Image." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

[4] Lv, Chunji, et al. "PhysGM: Large Physical Gaussian Model for Feed-Forward 4D Synthesis." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

AuthorsOpenReview ↗

Response to reviewer 5Mw2 (3/3)

  1. Q2: I feel that this pipeline is more like an agentic framework built on existing models, and thus it tends to perform better when compared with methods based on a single model. However, I hope the authors can provide corresponding evaluations, such as efficiency.

PhysForge centers on fitting of material parameters and component-level simulation; pretrained models only supply perception, geometry, and priors. Using pretrained priors is also part of the evaluated DreamPhysics [1] baseline and the related MotionPhysics [2] approach. Our method contributes PhysFit, which separates incoming-motion alignment from material fitting, and PhysComp, which preserves independent component states and interactions.

The PhysFit ablation in Table 3 (newly added) isolates the two fitting mechanisms. Loss in the following table measures how closely the simulated contact and deformation match the generated reference video; a lower loss indicates better fitting to the reference dynamics.

Window selectionHallucinated ForcingLoss ↓
✗✗0.99154
✓✗0.14242
✓✓0.05864

The reduction in loss shows the contribution of window selection and Hallucinated Forcing. Figure 6 in Appendix (newly added) illustrates the corresponding differences in collision and deformation. Table 4 (newly added) and Figure 5b (refined) further show improved component trajectories and independent motion with PhysComp.

We also report runtime directly:

MethodRuntime (min)
DreamPhysics30.06 ± 11.07
PSIVG15.24 ± 4.74
WMReward12.08 ± 0.08
PhysX-Omni16.69 ± 1.89
PhysX-Anything11.82 ± 1.90
PhysForge11.75 ± 0.56

From the table above, we can see that PhysForge has a runtime close to WMReward and PhysX-Anything, and lower than DreamPhysics, PSIVG, and PhysX-Omni. Together with the ablations, these results show that the improved material control and component separation come from the proposed fitting and simulation mechanisms at a comparable or lower measured runtime than these methods.

References

[1] Huang, Tianyu, et al. "DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priors." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 4. 2025.

[2] Wang, Miaowei, et al. "MotionPhysics: Learnable Motion Distillation for Text-Guided Simulation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 40. No. 12. 2026.

Manuscript revisions

In the revised manuscript, red marks changes to the original content, and blue marks newly added content.

To address your concern about video hallucinations, we have clarified supervised video generation in Section 3.2 and added the fitting ablation in Table 3 (newly added) on page 9, with the qualitative comparison in Figure 6 in Appendix (newly added), Appendix A.2, page 14. For your questions about material-generation cost and the relationship to PhysX methods, we have clarified the distinction in Section 2.1, introduced the PhysX baselines in Section 4.1, and added their results to Table 1, Figure 7 in Appendix (newly added) (Appendix A.2, page 14), and the runtime comparison in Table 6 (newly added) on page 9. To address whether the gains come from our proposed mechanisms, Section 4.2 now analyzes the PhysFit ablation and the PhysComp results in Table 4 (newly added) and Figure 5b (refined), both on page 9, together with the runtime comparison.

Reviewer_ukWnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper addressed reconstructing physically simulatable 3D assets from a single input object, with controllable material behavior and separable semantic components. The proposed PhysForge coupled PhysFit and PhysComp. PhysFit treats material-conditioned VDM outputs as reference dynamics and fits physical parameters. PhysComp propagates part identities from video to the reconstructed 3DGS geometry via pre-trained large models. Physical parameters are optimized for individual components. The method was evaluated on PhysAssets and GSO. PhysForge reports performance gains in CLIP and physical controllability scores across different materials.

Critical strengths

The paper addresses the problem of material-conditioned responses being conflated with hallucinated motion during fitting. It introduces a VDM-based framework for constructing physically simulatable 3D assets with controllable material behavior and separable semantic components. PhysFit leverages material-conditioned VDM dynamics for controllable physical-property estimation and introduces robust generated-video fitting. PhysComp enables independent physical-parameter optimization for individual objects and semantic components. The paper provides diverse qualitative evidence regarding different materials.

Critical weaknesses
  1. The proposed PhysComp enables independent physical-parameter optimization for individual objects. However, there is a lack of quantitative evaluation of component separability. Can the proposed method handle component or part motion trajectories?
  2. The VLM provides the material prior in the model and was used to evaluate the generated video and feedback. Is there a risk of circularity? A description of the rater would be helpful.
  3. This work introduces Hallucinated Forcing to handle VDM hallucinations. However, there is a lack of ablation analysis. It would be helpful to isolate the Hallucinated Forcing term.
  4. The proposed method requires material video generation, depth/mask generation, 3DGS reconstruction, and several fitting rounds. How does the computational complexity compare with that of the compared methods?
Questions for authors

See weakness. This paper introduces the windowed fitting formulation and Hallucinated Forcing, achieving whole-sequence fitting to VDM outputs. The qualitative results show performance gains over the compared methods. However, additional discussion and description are required regarding component separability, VLM-based evaluation, and Hallucinated Forcing.

5 comments
AuthorsOpenReview ↗

Response to reviewer ukWn (1/2)

Thank you for the questions on component trajectories, VLM roles, Hallucinated Forcing, and computational cost. We address them in the order of the review.

  1. W1: The proposed PhysComp enables independent physical-parameter optimization for individual objects. However, there is a lack of quantitative evaluation of component separability. Can the proposed method handle component or part motion trajectories?

PhysComp supports independent component trajectories, and the added ablation quantifies this through per-object trajectory error and its average. Here, trajectory error combines displacement and rotation errors. This evaluates how closely each component follows its reference motion:

Objectw/o PhysComp ↓w/ PhysComp ↓
Table0.0365580.036087
Wine bottle0.1669950.101223
Wine glass0.1502150.116436
Average0.1179230.084582

PhysComp substantially reduces the trajectory errors of the wine bottle and wine glass, improving the motion of the secondary objects placed on the table. In Figure 5b (refined), the bottle and glass separate from the table and translate and rotate independently; without PhysComp, their motion remains coupled to the table.

  1. W2: The VLM provides the material prior in the model and was used to evaluate the generated video and feedback. Is there a risk of circularity? A description of the rater would be helpful.

We would like to clarify the distinct roles of the three VLMs in our framework. The target material is explicitly specified by the user through the original text prompt. Qwen3.5-4B translates this specification into initial physical parameters, which serve as a text-conditioned material prior during subsequent simulation-based fitting. GPT provides feedback to refine reference-video generation. Separately, Gemini serves as the Physical Controllability (PC) rater, operating entirely independently of generation feedback and material initialization.

Qwen3.5-4B identifies objects and semantic components and maps the text-conditioned material prior to material categories and initial physical parameters. These estimates serve as a weak regularization prior, with the parameters subsequently refined through simulation-based fitting.

GPT checks generated reference videos for consistency with the requested material behavior, including object identity, contact, component separation, and floating artifacts, and provides feedback to refine generation.

Gemini uses the video being evaluated, the original user prompt, and the input reference image to assign a five-point PC score (higher is better) based on physical plausibility, material response, and appearance preservation. It is used exclusively for final evaluation; its scores are not used for generation refinement, candidate selection, or parameter tuning.

This separation avoids using the same VLM for generation feedback and final scoring, although model separation alone does not eliminate potential evaluator bias. We additionally report a user study (Section 4.2) in which videos are anonymized and presented in randomized order, providing complementary evidence for material controllability and component separability‌ beyond the VLM-based scores.

AuthorsOpenReview ↗

Response to reviewer ukWn (2/2)

  1. W3: This work introduces Hallucinated Forcing to handle VDM hallucinations. However, there is a lack of ablation analysis. It would be helpful to isolate the Hallucinated Forcing term.

Thank you for this suggestion. We have added an ablation experiment to isolate the contribution of Hallucinated Forcing. Hallucinated Forcing aligns the incoming motion state during fitting; within interaction windows, the physical simulator computes contact and deformation. Loss in the following table measures how closely the simulated contact and deformation match the generated reference video; a lower loss indicates better fitting to the reference dynamics.

Window selectionHallucinated ForcingLoss ↓
✗✗0.99154
✓✗0.14242
✓✓0.05864

Window selection focuses material fitting on contact and deformation, while Hallucinated Forcing prevents incorrect incoming motion from being compensated for by changes in material stiffness. In Figure 6 in Appendix (newly added), without window selection, fitting the full video produces overly stiff behavior, suggesting that motion outside the interaction interferes with material fitting. With window selection alone, the low incoming velocity leads to a slow collision with insufficient deformation: limited deformation can result from low impact velocity rather than high stiffness. Hallucinated Forcing aligns the incoming position and velocity with the reference, allowing material fitting to distinguish these effects. Enabling both yields the lowest loss and behavior closer to the requested soft material.

  1. W4: The proposed method requires material video generation, depth/mask generation, 3DGS reconstruction, and several fitting rounds. How does the computational complexity compare with that of the compared methods?

Thank you for this suggestion. We have added a runtime comparison experiment to evaluate the computational cost of PhysForge relative to the baseline methods. PhysForge has a runtime close to WMReward and PhysX-Anything [1] and lower than DreamPhysics, PSIVG, and PhysX-Omni [2] in the measured comparison. Its pipeline uses pretrained perception and geometry models to construct the scene, then fits physical parameters in interaction windows and simulates the components. The two added baselines, PhysX-Omni and PhysX-Anything, generate simulation-ready 3D assets from a single image and infer physical properties from object appearance. We report runtime as an empirical measure of this computational cost:

MethodRuntime (min)
DreamPhysics30.06 ± 11.07
PSIVG15.24 ± 4.74
WMReward12.08 ± 0.08
PhysX-Omni16.69 ± 1.89
PhysX-Anything11.82 ± 1.90
PhysForge11.75 ± 0.56

PhysForge supports material control and component separation with a runtime comparable to WMReward and PhysX-Anything. The measured runtime is also lower than that of DreamPhysics, PSIVG, and PhysX-Omni.

References

[1] Cao, Ziang, et al. "PhysX-Anything: Simulation-Ready Physical 3D Assets from Single Image." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

[2] Cao, Ziang, et al. "PhysX-Omni: Unified Simulation-Ready Physical 3D Generation for Rigid, Deformable, and Articulated Objects." arXiv preprint arXiv:2605.21572 (2026).

Manuscript revisions

In the revised manuscript, red marks changes to the original content, and blue marks newly added content.

For your question about component trajectories and separability, we have added per-object trajectory errors in Table 4 (newly added) and trajectory visualization in Figure 5b (refined), both on page 9, with their analysis in Section 4.2. To address your concern about the roles of the three VLMs and possible circularity, Sections 3.2 and 3.3.1 now describe their separate roles in reference-video generation and component processing, while Section 4.1 identifies Gemini as the rater of Physical Controllability on a five-point scale. Following your request to isolate Hallucinated Forcing, we have added Table 3 (newly added) on page 9 and Figure 6 in Appendix (newly added) (Appendix A.2, page 14), with the ablation analysis in Section 4.2. For your computational-cost question, we have added the runtime comparison in Table 6 (newly added) on page 9 and its discussion in Section 4.2.

AuthorsOpenReview ↗

Follow-Up to Your Review

Thank you for the time and effort you put into reviewing our work! We understand that this might be a super busy period for you. We wanted to briefly let you know that we have posted point-by-point responses to your questions and concerns and uploaded a revised manuscript with the highlighted changes. Whenever you have a chance, we would be grateful for your feedback on whether the responses address your concerns. We would like to provide additional clarification if you have any further questions.

Reviewer_rYFfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a framework for reconstructing 3D assets with controllable material properties and separable components. It models the response of objects made of different materials to forces, contact, and deformation. The method presented is aimed at overcoming the limitations of the existing methods which use models learned from appearance–material correlations, and therefore lack the ability to (1) modify the material properties of object models, and (2) estimate materials from object-level visual observations of contacts among objects/parts and resulting deformations. To do this, they introduce two modules:

PhysFit: which uses material-conditioned video diffusion priors for physical-property fitting, optimizes physical parameters only over windows manifesting material properties (windows containing contact, collision, deformation). They introduce hallucinated forcing to handle nonphysical motion - generated motion inconsistent with physical motion.

PhysComp: which decomposes scenes into semantic components, for each of which physical-property estimation is done separately and simulation incorporates the intrinsic properties of the objects/parts as well as their and interactions.

Experiments on PhysAssets and Google Scanned Objects report improvements over six baselines in CLIP scores, VLM-rated physical controllability, and human preference (Tables 1–2).

Critical strengths
  1. The combination of material-conditioned video priors, interaction-window fitting, and component-level simulation provides a useful approach to controllable physical 3D reconstruction (Section 3). Isolating different contact/deformation windows significantly lowers physical loss, accelerates convergence rate of optimization, and helps avoid non-physical motion artifacts that would be introduced by treating the entire video as a single track ("Hallucinated Forcing"). Separating contacting objects allows physically contiguous but mechanically distinct parts to move independently. The overall pipeline is clearly illustrated in Figure 2.

  2. Material controllability: PhysForge achieves the highest reported CLIP and physical controllability scores among six baselines for both common and uncommon materials. For uncommon materials, its physical controllability score is 4.0, compared with 3.2 for the strongest baseline (Table 1).

  3. Component-level modeling: PhysComp enables separate simulation of interacting components as shown in Figure 4 on comparisons, and in Figure 5b on the effect of preserving component identities during simulation.

Critical weaknesses
  1. Clarity: Some implementation details remain unclear, particularly how Hallucinated Forcing is incorporated into optimization (Sections 3.2.2 and 3.3.2). The reported results support improved material-conditioned outputs, but the evaluation does not directly establish the accuracy of recovered physical parameters. The ablations also do not isolate all proposed mechanisms (Tables 1–2; Figure 5).

  2. Limited validation of physical accuracy: The quantitative evaluation relies on CLIP similarity and VLM-rated physical controllability, neither of which directly measures errors in physical-parameter estimation. Although PhysAssets provides physical-property annotations, no ground-truth parameter recovery error is reported (Section 4.1; Table 1).

  3. Insufficient isolation of the proposed mechanisms: Figure 5a compares PhysFit against full-video fitting but does not separately evaluate window selection and Hallucinated Forcing. Moreover, Section defines Hallucinated Forcing through reference-motion acceleration, but its precise role in the optimization objective (Equations 4–6) is unclear.

Questions for authors
  1. Can the authors evaluate recovered physical parameters against ground-truth annotations or measure simulation trajectory errors against reference dynamics?

  2. Can the authors provide separate ablations for window selection and Hallucinated Forcing to establish their individual contributions?

  3. How many participants and trials were included in the user study (Table 2), and what is the statistical uncertainty of the reported rankings?

4 comments
AuthorsOpenReview ↗

Response to reviewer rYfF (1/3)

Thank you for the detailed questions on optimization, physical parameters, ablations, and the user study. We group overlapping concerns while following their order in the review.

  1. W1: Clarity: Some implementation details remain unclear, particularly how Hallucinated Forcing is incorporated into optimization (Sections 3.2.2 and 3.3.2).

    W3: Moreover, Section defines Hallucinated Forcing through reference-motion acceleration, but its precise role in the optimization objective (Equations 4–6) is unclear.

Hallucinated Forcing is an auxiliary forcing used during parameter fitting to reproduce nonphysical motion in the generated reference; it is removed for the final simulation. It allows material fitting to start each interaction with the reference position and velocity, so stiffness need not compensate for an incorrect incoming motion state.

The fitting process has three steps. First, we recover the reference trajectory and derive its apparent acceleration. Second, Hallucinated Forcing aligns the simulated motion before each interaction window. Third, within the window, the physical simulator takes over to compute contact and deformation, and the physical regulation loss in Equation (4) updates the material parameters. Hallucinated Forcing acts through the simulation state rather than through an additional loss term. Loss in the following table measures how closely the simulated contact and deformation match the generated reference video; a lower loss indicates better fitting to the reference dynamics.

Window selectionHallucinated ForcingLoss ↓
✗✗0.99154
✓✗0.14242
✓✓0.05864

Window selection isolates the interaction used for material fitting, while Hallucinated Forcing aligns the incoming state so that material parameters are fitted under the intended contact conditions. Without this alignment, low incoming velocity can cause insufficient deformation even for a soft material, confounding motion differences with material stiffness. In Figure 6 in Appendix (newly added), window selection alone produces a slow collision with limited deformation, whereas adding Hallucinated Forcing produces behavior closer to the requested soft material and further reduces the loss in Table 3 (newly added). These results show that window selection and Hallucinated Forcing play complementary roles in material fitting.

AuthorsOpenReview ↗

Response to reviewer rYfF (2/3)

  1. W1: The reported results support improved material-conditioned outputs, but the evaluation does not directly establish the accuracy of recovered physical parameters.

    W2: Limited validation of physical accuracy: The quantitative evaluation relies on CLIP similarity and VLM-rated physical controllability, neither of which directly measures errors in physical-parameter estimation. Although PhysAssets provides physical-property annotations, no ground-truth parameter recovery error is reported (Section 4.1; Table 1).

    Q1: Can the authors evaluate recovered physical parameters against ground-truth annotations or measure simulation trajectory errors against reference dynamics?

We would like to clarify that our task generates material behavior conditioned on text, so the original asset's annotations do not provide ground-truth parameters for its user-specified materials. The same apple appearance may represent hardwood or a soft silicone-gel object. Its original physical annotation describes one material, while the requested conditions define different target behaviors. We apologize for not making this distinction clear in the original manuscript.

To provide a simple validation, we evaluate the change in predicted stiffness using the hard-to-soft Young's modulus ratio, calculated as the modulus under the hard condition divided by that under the soft condition for the same object. A ratio above one indicates the requested ordering of stiffness. The comparison includes methods that output Young's modulus:

MethodCase 1Case 2Case 3
DreamPhysics0.19340.59430.9279
PSIVG111
PhysGM111
PhysForge2074.3636371.77971158.1919

PhysForge produces consistently higher stiffness for the hard material. PSIVG and PhysGM retain the same modulus across the material conditions, while DreamPhysics reverses the requested hard-to-soft ordering in these cases. This shows that PhysForge can adjust material stiffness according to the user’s text, whereas the compared methods do not consistently reflect the requested material differences.

We additionally evaluate mean trajectory loss against the generated reference dynamics across seven methods. This metric measures the root mean square error between normalized two-dimensional displacement trajectories; lower values indicate closer agreement with the reference motion. Results are averaged separately over common and uncommon material conditions.

MethodCommon ↓Uncommon ↓
DreamPhysics0.547650.51775
PSIVG0.528280.50566
PhysRVG0.290460.34566
WMReward0.387230.34931
DeforM0.355830.39554
PhysGM0.408730.46436
PhysForge0.115080.22005

PhysForge achieves the lowest mean trajectory loss under both conditions, showing closer agreement with the requested reference dynamics than the six comparison methods.

AuthorsOpenReview ↗

Response to reviewer rYfF (3/3)

  1. W1: The ablations also do not isolate all proposed mechanisms (Tables 1–2; Figure 5).

    W3: Insufficient isolation of the proposed mechanisms: Figure 5a compares PhysFit against full-video fitting but does not separately evaluate window selection and Hallucinated Forcing.

    Q2: Can the authors provide separate ablations for window selection and Hallucinated Forcing to establish their individual contributions?

We isolate window selection and Hallucinated Forcing by comparing three settings: both disabled, window selection alone, and both enabled. Loss is the physical regulation loss in Equation (4), comprising compression deformation discrepancy, contact duration discrepancy, and a penetration penalty.

Window selectionHallucinated ForcingLoss ↓
✗✗0.99154
✓✗0.14242
✓✓0.05864

Each mechanism reduces loss, and enabling both gives the closest match to the requested material behavior. In Figure 6 in Appendix (newly added), fitting without window selection produces overly stiff behavior. Window selection alone produces a slow collision and insufficient deformation. Together, the mechanisms focus fitting on contact and deformation while aligning the incoming motion state.

  1. Q3: How many participants and trials were included in the user study (Table 2), and what is the statistical uncertainty of the reported rankings?

The user study contains 30 anonymous evaluation records, each covering 12 cases. Each case presents videos from seven methods in anonymized, randomized order, together with the input and target material. Evaluators rank the videos by material consistency and plausible dynamics. Win rate is the proportion of first-place rankings, and mean rank is reported with its standard deviation.

MethodWin rate (%)Mean rank
DreamPhysics5.564.2778 ± 0.4439
PSIVG0.566.2167 ± 0.5974
PhysRVG8.333.9500 ± 0.6090
WMReward12.223.5444 ± 0.4848
DeforM2.783.6833 ± 0.4216
PhysGM10.004.4222 ± 0.3666
PhysForge60.561.9056 ± 0.6155

PhysForge has the highest win rate and the lowest mean rank.

Manuscript revisions

In the revised manuscript, red marks changes to the original content, and blue marks newly added content.

To clarify how Hallucinated Forcing enters optimization, we have revised Sections 3.2.2 and 3.3.2 to explain its effect on the simulation state and its relationship to the loss. For your questions about physical-parameter accuracy and trajectory error, we have clarified the ground-truth issue in the second paragraph of the Introduction and added the stiffness-ratio comparison in Table 7 in Appendix (newly added) (Appendix A.2, page 15) and the trajectory comparison in Table 5 (newly added) on page 9. To address your request to isolate the mechanisms, we have added Table 3 (newly added) on page 9 and Figure 6 in Appendix (newly added) (Appendix A.2, page 14), with their analysis in Section 4.2. For your user-study question, Section 4.1 now specifies the study size, and Table 2 on page 8 includes the standard deviation of mean rank.

AuthorsOpenReview ↗

Thank you for raising your score

Thank you for taking the time to read our rebuttal and for raising your score. We are glad that our response has addressed your concerns. Please feel free to let us know if you have any further questions or would like additional clarification.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers and AC

We thank the reviewers for their constructive suggestions for improving the manuscript and their positive assessment of our work. Our paper introduces PhysForge, which uses PhysFit to select fitting windows and align motion states through Hallucinated Forcing, and PhysComp to perform component-level 3D reconstruction and physical-parameter fitting guided by VDM priors, enabling the reconstruction of material-controllable 3D assets with separable components.

Based on reviewers' feedback, we have made the following revisions in our paper (red marks as changes to the original content, and blue marks as newly added content):

  1. Added comparisons with PhysX-Omni and PhysX-Anything to compare with VLM-based methods (Table 1; Figure 7 in Appendix (newly added)).

  2. Added quantitative and qualitative ablations of window selection and Hallucinated Forcing to assess their individual contributions to material fitting (Table 3 (newly added); Figure 6 in Appendix (newly added)).

  3. Added per-object trajectory errors for the PhysComp ablation, together with component trajectory visualizations, to quantify component separability (Table 4 (newly added); Figure 5b (refined)).

  4. Added hard-to-soft Young’s modulus ratios and trajectory comparisons across seven methods to evaluate material controllability and agreement with reference motion (Table 7 in Appendix (newly added); Table 5 (newly added)).

  5. Added runtime comparisons to assess the computational cost relative to the baselines (Table 6 (newly added)).

  6. Expanded the user-study description and refined ranking uncertainty to clarify the evaluation protocol and the variability in participant rankings (Table 2).

  7. Clarified the material-control task, Hallucinated Forcing, and the distinct VLM roles to explain how user-specified materials guide parameter fitting and distinguish generation feedback from final evaluation (Introduction; Sections 2.1, 3.2, 3.3.1–3.3.2, and 4.1).

  8. Additionally, we updated the title and page numbers in the PhyMAGIC reference to reflect the changes from its arXiv version to the published ECCV version. We also added the PhysX-Omni reference. (Reference)

Our responses have addressed the main concerns of all reviewers 5Mw2, rYfF and ukWn, all of whom raised their scores from 2 to 3. We hope that the area chair considers this context in their evaluation of the PhysForge paper. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.