- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents PhysVGGT, a feedforward method to predict dense maps of object material properties from a single image. The author uses a frozen StreamVGGT backbone, a dense prediction branch with cross-property attention and shared decoder, and a mass head. The authors introduce a pseudo-labeling pipeline combining part-segmentation, VLM reasoning and cross-view refinement to obtain the training data. Experiments on ABO-500 and NeRF2Physics demonstrate the accuracy of the method.
Critical strengths
- Studying feedforward object material property estimating is useful.
- The use of pretrained VGGT model and overall pipeline is sound.
- The authors compared different backbones, frozen strategies, decoder, losses in the experiments.
Critical weaknesses
- How is the methods results applied to real-world inputs?
- WIth the development and effectiveness of VLM models, how does this work compared to directly using VLMs for the task?
- Tables 5 and 11 report substantial mass improvements from cross-property coupling, however the mass head is seperate from DPT, how could that happen?
- The authors didn't explain how multiview training transfers to single-view inference.The backbone comparison also does not isolate whether StreamVGGT’s advantage comes from its causal design or something else.
Questions for authors
Why do the reproduced NeRF2Physics and PhysGS results differ substantially from their original reports? Is the evaluation protocal different?
3 comments
Q3-4: How does multi-view training transfer to single-view inference, and does StreamVGGT's advantage come from its causal design?
Response:
- Multi-view training and single-view inference. StreamVGGT supports variable-length image sequences, and the dense prediction head shares parameters across frames. During causal training, the first frame has no preceding context and is supervised by the same per-frame prediction objective. Therefore, single-view inference is naturally supported by setting the sequence length to one, without architectural modifications. The dense predictions follow the same prediction pipeline described in Section~3.1. We further evaluate PhysVGGT with multiple views for dense prediction. The results show that PhysVGGT maintains comparable performance across different input view counts, but additional views do not necessarily bring performance gains.
| Input Views | Mass ADE (kg) | Density MAE (kg/m$^3$) | MAE (GPa) |
|---|---|---|---|
| 1 | 4.73 | 900 | 33 |
| 2 | 4.65 | 901 | 33 |
| 4 | 4.53 | 908 | 34 |
- Backbone comparison and causal design. We have compared PhysVGGT with various backbones in Table 8, but this comparison does not isolate the effect of causal design from other architectural and pretraining differences. Isolating the contribution of causal design would require a controlled comparison with a non-causal StreamVGGT variant using the same backbone architecture and training configurations. In the revised manuscript, we have clarified that the results demonstrate the effectiveness of StreamVGGT as the backbone in our framework, rather than attributing its advantage exclusively to its causal design.
Q3-5: Why do the reproduced NeRF2Physics and PhysGS results differ substantially from the original reports?
Response: Our evaluation protocol is consistent with that of existing works and benchmarks. We follow the original paper descriptions to reproduce NeRF2Physics and PhysGS independently. Both NeRF2Physics and PhysGS rely on per-scene 3D reconstruction and vision-language-based material/property estimation, making their predictions sensitive to implementation details, model configurations, and reconstruction results. We have clarified this distinction in the revised manuscript. Nevertheless, despite the discrepancies between our independent reproductions and the originally reported results, PhysVGGT still outperforms the originally reported NeRF2Physics and PhysGS results.
Dear Reviewer,
Thank you for your time and consideration. We hope our response has addressed your concerns, particularly regarding the comparisons with direct VLMs. If any questions or concerns remain, we would be grateful for the opportunity to provide further clarification during the discussion period.
Thanks.
We thank the reviewer for the valuable comment. We have revised the manuscript to address the concerns (marked in blue in the revised manuscript). Our responses are detailed below.
Q3-1: How are the method's results applied to real-world inputs?
Response: Thanks for the question. The predicted physical-property maps and object-level mass estimates from PhysVGGT can serve as inputs to various real-world downstream tasks. Specifically, PhysVGGT takes real-world RGB images as input and produces dense physical-property maps and object-level mass estimates through feed-forward inference, without requiring VLM prompting or material-property lookup. For robotic manipulation, the predicted friction maps can be further converted into dense grip-force maps, as illustrated in Figures 8 and 1(c). These maps provide spatial guidance for selecting grasp locations and adjusting grip forces according to local surface friction, favoring high-friction regions while applying larger forces to slippery regions. For outdoor scenes, the predicted friction and Young's modulus maps (Figure 5) can be used as additional environmental perception cues for terrain-aware navigation, traversability estimation, and slip-risk assessment. Thus, PhysVGGT provides dense physical information that can be incorporated into downstream robotic manipulation and autonomous systems.
Q3-2: How does this work compared to directly using VLMs for the task?
Response: Unlike directly prompting VLMs for physical-property estimation, PhysVGGT uses VLMs only during pseudo-label generation and does not require VLM inference at test time. PhysVGGT learns to map image-specific visual features from the geometry encoder backbone to dense physical-property maps through lightweight trainable prediction heads. To resolve this concern, we additionally provide quantitative comparisons against direct VLM inference, as shown below. These results demonstrate that PhysVGGT provides more accurate physical-property estimation than direct VLM prompting, while offering dense feed-forward prediction without requiring VLM inference at deployment.
Q3-3: In Tables 5 and 11 (Tables 6 and 12 in the revised manuscript), why cross-property coupling and dense objective losses improve mass prediction despite the mass head being separate?
Response: Thanks for pointing out this issue. We apologize for the confusion caused by the presentation of the mass results in Tables 5 and 11.