- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces ADYTON, a simulated benchmark for learning deformable object dynamics, and DELPHI, a model that predicts dense 3D motion from monocular RGB-D videos. ADYTON contains 12,000+ simulated scenes with ground-truth motion labels covering visible surfaces, occluded regions, and object interiors. DELPHI uses an implicit displacement field and spatially biased attention to predict motion at arbitrary 3D locations. Experiments show that DELPHI achieves competitive accuracy on visible surfaces and significantly outperforms in full-volume motion prediction. The authors also evaluate generalization to novel scene conditions, 10-step autoregressive prediction, and transfer to real-world videos.
Critical strengths
- ADYTON provides 12,000+ simulated scenes with dense ground-truth motion labels covering visible surfaces, occluded regions, and object interiors. This addresses an important limitation of existing datasets and enables more comprehensive evaluation of deformable object dynamics.
- DELPHI uses an implicit displacement field with spatially biased attention to predict motion at arbitrary 3D locations, extending beyond observed surface points. Its strong full-volume prediction results demonstrate the potential of this representation for learning deformable dynamics.
Critical weaknesses
Although the paper evaluates DELPHI on the real-world Spring-Gaus dataset, the evaluation primarily relies on reconstruction metrics (PSNR, SSIM, and LPIPS). These metrics provide limited evidence of accurate 3D motion prediction, particularly for occluded regions and object interiors. More comprehensive physical evaluations, such as motion trajectory accuracy, contact behavior, and deformation consistency, would strengthen the validation. Furthermore, the paper lacks intuitive side-by-side video comparisons of the predicted dynamics with ground truth and baseline methods, making it difficult to assess motion quality and temporal consistency. Given the focus on deformable dynamics prediction, stronger real-world validation is essential to demonstrate the practical effectiveness of the method.
Questions for authors
- Section 6 mentions a test-time training scheme in which training and testing use the same videos, while Table 5 reports zero-shot results. Could the authors clarify the training and evaluation protocols for each setting, particularly whether test videos or future frames are used for model adaptation?
- Does Table 2 use ground-truth particle positions from the simulator as query points? If so, how would these points, particularly those inside objects, be obtained from real-world RGB-D observations? Would the need for complete object geometry or interior point locations limit the practical applicability of DELPHI's full-volume prediction capability?
4 comments
3. Obtaining Interior Query Points from Real-World RGB-D (Table 2)
DELPHI requires no ground-truth interior geometry at inference: probing DELPHI’s learned occupancy features over a depth-masked grid recovers interior queries in 0.24 s/object while matching or outperforming baselines given true geometry.
- Addressing Table 2: Yes, Table 2 originally queried DELPHI at simulator particle positions. To evaluate practical deployment from monocular RGB-D and masks alone, we sample a 3D grid behind the observed surface and pass candidate queries through a lightweight occupancy filter (trained on ADYTON scenes) that probes DELPHI’s features to separate interior object points from ambient background.
- Whole-Object Benchmark Without Privileged Geometry: Evaluating next-step symmetric Chamfer distance (cm) where every method supplies its own hidden geometry confirms that full-volume prediction remains practical and accurate:
| Method | Input | Heuristic grid + filter (0.24 s/object) | SAM 3D (3.12 s/object) | True geometry (reference) |
|---|---|---|---|---|
| No motion | RGB-D | – | 2.69 | 2.22 |
| Any4D + smooth completion | RGB-D | – | 1.81 | 1.33 |
| Sonata | RGB-D + true 3D tracks of observed points | – | 1.85 | 1.16 |
| DELPHI | RGB-D | 1.31 | 1.73 | 1.04 |
Next-step symmetric Chamfer distance (cm) given context ground-truth masks and depth. Times reflect per-object geometry proposal latency.
- Key Takeaway: Without true geometry, DELPHI (1.31 cm) matches Any4D given true geometry (1.33 cm) and beats all SAM 3D baselines, including Sonata given true 3D tracks (1.85 cm), while remaining the most accurate method given true geometry (1.04 cm). At 0.24 s per object (13× faster than SAM 3D), full-volume prediction transfers directly to real-world RGB-D observations. We will add this experiment to the revision.
Baseline Completion Definitions:
- Rigid completion: Fits a single SE(3) transformation (rotation and translation) per object to the baseline's predicted visible-surface motion and applies it to the true hidden particles.
- Smooth completion: Interpolates the baseline's predicted visible-surface motion into a continuous, non-rigid field across the object volume and evaluates it at the true hidden particles.
- SAM 3D completion: Reconstructs the full 3D object geometry (including occluded regions) from the current RGB-D frame and mask, then propagates those reconstructed points using the rigid or smooth motion rules above.
(Note: Rigid and smooth completion still receive privileged ground-truth hidden particle locations and only infer their motion, whereas SAM 3D completion, like DELPHI's heuristic grid + filter, must also infer where the hidden geometry lies.)
Thank you for the clarifications and additional experiments. My concerns have been largely addressed. I will maintain my score of 3.
Thank you for your follow-up and for confirming that your concerns are largely resolved. We are especially glad that the new monocular RGB-D evaluation addressed the question on inference without privileged interior geometry. Probing DELPHI's occupancy features recovers interior points in 0.24 s/object (13× faster than SAM 3D) at 1.31 cm Chamfer distance, matching baselines given ground-truth geometry (1.33 cm). We will integrate this complete comparison into the revision.
Given that these main concerns have been resolved, we kindly ask if you would consider raising your score. Thanks again for your valuable feedback!
We thank the reviewer for the constructive review and valuable feedback.
1. Real-World Evaluation & Physical Validation
While real-world captures lack ground-truth interior tracks, we directly validate contact, deformation, and hidden-region accuracy on ADYTON, where DELPHI achieves 0.72 deformation alignment (vs. 0.17 for rigid completion)
2. Clarifying Zero-Shot vs. Test-Time Adaptation Protocols (Sec. 6 & Table 5)
Table 5 reports strict zero-shot evaluation without test-time adaptation; no future test frames are ever used during adaptation, and we will separate these two settings cleanly in the revision.