- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces the EfM (Elasticity from Motion) method that focuses on estimating elasticity parameters of moving objects using a multi-view RGB observation setup. The experiments are conducted on PAC-NeRF and Spring-Gaus-Diverse, demonstrating an improvement over the baselines in terms of accuracy and efficiency.
Critical strengths
- A timely method that focuses on leveraging physics and motion-governing paradigms instead of relying on exposing the learning systems to naive losses and hence data-hungry methods (and eventually computationally expensive simulators)
- Well written and motivated piece of research, strong results, easy to follow and with a relevant Appendix
Critical weaknesses
- More about the perception only. It is nudged in the paper that the setup assumption is a multi-view camera set. In my view, this also translates to the later statements in L. 298 (“, whether a camera sees it,” and “displacements over five consecutive frames”), which indicates a reliance on close-to-ideal observation to be able to infer the interior field prediction. Also, the fixed 5 frames seems a bit arbitrary, without much justification. What is the heuristic behind it? What if there are only two cameras with a big baseline? It seems that it is dictated by the specific experimental datasets rather than the design of the method. The general experimental setup of the camera placement is unclear (also in 4.6, it is mentioned that three cameras have been used, but what kind of coverage is provided?). It is hard to infer the robustness wrt the perception system used, e.g., would that work with a single view or 20% overlap two cameras? This is addressed to some degree in A3 and A4 but an important differentiator is the actual baseline setup. E.g., a camera observing an object from the back together with one from the front
- This links up to the assumption of the assumed trajectory provided and as mentioned by the end of the paper (l 485) the error will grow dependent on its accuracy.
Questions for authors
- Could you provide more details on the limits of the system wrt camera setup? Elaborate on the A3 and A4 connecting to my mentioned Weakness above. It would be great to provide an ablation with more specific views’ description (baselines, observation angle, maybe even a sketch if possible - and relation to the overall performance).
- I was wondering about the chosen 3D representation. Why was the tetrahedral mesh (l. 152) chosen? Why not point clouds, voxels, Gaussians or any other representation? Is the watertightness an important aspect? I would welcome a discussion on that matter.
7 comments
Question 1. "Could you provide more details on the limits of the system wrt camera setup? Elaborate on the A3 and A4 connecting to my mentioned Weakness above. It would be great to provide an ablation with more specific views' description (baselines, observation angle, maybe even a sketch if possible - and relation to the overall performance)."
Reply: Thank you for the suggestion. We evaluated every single camera and every camera pair of the benchmark cameras, without selecting any (Table R4, App. C.5, camera sketch in Fig. 11). Two cameras suffice for EfM, and wider, less overlapping pairs work better. In particular, a front/back pair beats every baseline that uses all cameras. A single camera raises the error several times, which we mention in the limitations (Sec. 5). For the real captures (Sec. 4.6), the three cameras stand 89 to 131$^\circ$ apart around the table, about 0.7 m from the object (App. A.2, Fig. 11).
Table R4. Single cameras and camera pairs of the benchmark cameras, grouped by the angle between the two cameras.
| Cameras | Overlap | PAC | SG-Div |
|---|---|---|---|
| 1 camera | - | 0.310 | 0.954 |
| 2 cams, 0-60$^\circ$ | 0.5 | 0.186 | 0.772 |
| 2 cams, 60-120$^\circ$ | 0.3 | 0.168 | 0.648 |
| 2 cams, 120-180$^\circ$ | 0.10-0.14 | 0.123 | 0.517 |
| All cameras (paper) | 0.072 | 0.214 | |
| Best baseline (all cameras) | 0.542 | 0.698 |
Question 2. "I was wondering about the chosen 3D representation. Why was the tetrahedral mesh (l. 152) chosen? Why not point clouds, voxels, Gaussians or any other representation? Is the watertightness an important aspect? I would welcome a discussion on that matter."
Reply: Thank you for the question. We use a tetrahedral mesh because it provides the two things the solve needs, points with volumes and deformation gradients, and it gives the F-GNN interior nodes to pass information through. Watertightness therefore matters only for placing these interior points. The solve itself only needs points that carry a volume and a deformation gradient, so EfM also runs on point clouds. For tracked trajectories, we replace the backbone of the F-GNN with a point-based Transformer (App. C.3), which runs on points alone and still gives the lowest E error of all methods on such input (Sec. 4.7, Table 4). We added this discussion to Sec. 3.
Comprehensive clarifications
I thank the authors for providing a comprehensive answer to my raised questions.
Q1/A1 -- Excellent, this sheds more light on the limitations and required camera setup. I believe this is an insightful addition for a future adoption of the proposed solution.
Q2/A2 -- I see, this explains the design; and the revised discussion in the main paper clarifies and justfies the design as well. It is reasonalbe to use point transformer, I agree.
Thank you for engaging in the discussion.
We thank you for your valuable comments and for appreciating our timely, physics-based approach, its strong results and the clarity of the paper. We revised the manuscript accordingly (changes in blue) and added a supplementary results video. Below, we address the concerns raised in the review.
Reply: Thank you for this insightful comment. Like the multi-view baselines we compare with, EfM works best when several cameras observe the object, and we mention this in the limitations (Sec. 5). Following the suggestion, we measured how much it actually needs. The F-GNN is trained with partial views from only 1 to 3 random directions with noise in half of its training steps (Sec. 3.2) and never sees the benchmark cameras. As a result, two cameras with a wide baseline already beat every baseline that uses all cameras (Table R4, App. C.5), whereas a single camera raises the error several times. As for the five input frames, they are the fewest symmetric frames that give velocity and acceleration at the center frame. Spacing them 2 or 3 frames apart still keeps EfM 2.2 to 5.6 times more accurate than the best baseline (Table R1, App. C.6).
Table R1. Spacing of the five input frames, without retraining. Best baseline: best value of any baseline in Table 2.
Reply: Thank you for the comment. We now measure this dependence with tracked input. Since tracked trajectories are noisier than the given surface trajectory, we keep the same solve and replace the backbone of the F-GNN with a point-based Transformer trained on observations with simulated tracking error (App. C.3). On trajectories tracked from the benchmark videos, EfM then has the lowest E error of all methods (Table R2). On real captures, EfM's material also predicts unseen frames better than 10$\times$ stiffer or softer materials on every object (Table R3).
Table R2. EfM on trajectories tracked from the benchmark videos. Last row: EfM of Table 2 on O, for reference.
Table R3. Real captures: silhouette IoU on the 12 observed (IoUrec) and 8 unseen (IoUpred) frames, with the material estimated from the first 12.