acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Visible: Learning Dense 4D Motion in Contact-Rich Deformable Worlds

Abstract

Predicting 4D dynamics of deformable objects from vision is limited by datasets lacking dense volumetric motion labels and by point-based methods that cannot predict beyond sensor-observed surfaces. We introduce A rena of D eformable dY namics in Temporal Object N exus (ADYTON), a large-scale physics-simulated benchmark with ground-truth material-point trajectories throughout the full 3D volume across ten categories of contact-rich deformable interactions, with systematic generalization tests. We propose D eformable dynamics Evolution Learned via vision-conditioned Predictive H igh-fidelity Implicit fields (DELPHI), a vision-conditioned implicit displacement field that decouples spatial queries from input geometry, enabling amodal prediction from monocular RGB-D. On ADYTON, DELPHI matches the strongest baselines on visible surfaces while extending to unobserved regions with negligible degradation, whereas baselines collapse once denied the privileged correspondence that real sensors cannot provide. These results suggest continuous field representations are a more principled foundation for learned 4D dynamics.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer Vision, 3D Vision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xFaROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces ADYTON, a simulated benchmark for learning deformable object dynamics, and DELPHI, a model that predicts dense 3D motion from monocular RGB-D videos. ADYTON contains 12,000+ simulated scenes with ground-truth motion labels covering visible surfaces, occluded regions, and object interiors. DELPHI uses an implicit displacement field and spatially biased attention to predict motion at arbitrary 3D locations. Experiments show that DELPHI achieves competitive accuracy on visible surfaces and significantly outperforms in full-volume motion prediction. The authors also evaluate generalization to novel scene conditions, 10-step autoregressive prediction, and transfer to real-world videos.

Critical strengths
  1. ADYTON provides 12,000+ simulated scenes with dense ground-truth motion labels covering visible surfaces, occluded regions, and object interiors. This addresses an important limitation of existing datasets and enables more comprehensive evaluation of deformable object dynamics.
  2. DELPHI uses an implicit displacement field with spatially biased attention to predict motion at arbitrary 3D locations, extending beyond observed surface points. Its strong full-volume prediction results demonstrate the potential of this representation for learning deformable dynamics.
Critical weaknesses

Although the paper evaluates DELPHI on the real-world Spring-Gaus dataset, the evaluation primarily relies on reconstruction metrics (PSNR, SSIM, and LPIPS). These metrics provide limited evidence of accurate 3D motion prediction, particularly for occluded regions and object interiors. More comprehensive physical evaluations, such as motion trajectory accuracy, contact behavior, and deformation consistency, would strengthen the validation. Furthermore, the paper lacks intuitive side-by-side video comparisons of the predicted dynamics with ground truth and baseline methods, making it difficult to assess motion quality and temporal consistency. Given the focus on deformable dynamics prediction, stronger real-world validation is essential to demonstrate the practical effectiveness of the method.

Questions for authors
  1. Section 6 mentions a test-time training scheme in which training and testing use the same videos, while Table 5 reports zero-shot results. Could the authors clarify the training and evaluation protocols for each setting, particularly whether test videos or future frames are used for model adaptation?
  2. Does Table 2 use ground-truth particle positions from the simulator as query points? If so, how would these points, particularly those inside objects, be obtained from real-world RGB-D observations? Would the need for complete object geometry or interior point locations limit the practical applicability of DELPHI's full-volume prediction capability?
4 comments

We thank the reviewer for the constructive review and valuable feedback.

1. Real-World Evaluation & Physical Validation

While real-world captures lack ground-truth interior tracks, we directly validate contact, deformation, and hidden-region accuracy on ADYTON, where DELPHI achieves 0.72 deformation alignment (vs. 0.17 for rigid completion)

  • Validating Interior & Contact Dynamics Where Ground Truth Exists: We agree that 2D photometric metrics (PSNR, SSIM, LPIPS) cannot directly measure occluded or interior 3D motion. Because real-world RGB-D captures do not record interior particle trajectories, we evaluate these physical quantities directly on ADYTON’s ground-truth fields during contact impact:
    • Never-Observed Interior Points: During impact, DELPHI reduces error on completely occluded points to 2.17 / 1.32 / 1.49 cm across collision, free-fall, and deformable regimes—outperforming Any4D with smooth completion (3.60 / 2.33 / 2.42 cm).
    • Deformation Consistency: In deformable impacts, DELPHI’s predicted non-rigid residual direction is better than Any4D. (Cosine similarity: DELPHI 0.72 vs. rigid completion 0.17)
  • Side-by-Side Real-World Video Comparisons: To make temporal consistency and motion quality easy to inspect visually, we will include side-by-side video comparisons of DELPHI, baselines, and ground truth on real-world Spring-Gaus sequences in the Appendix.

2. Clarifying Zero-Shot vs. Test-Time Adaptation Protocols (Sec. 6 & Table 5)

Table 5 reports strict zero-shot evaluation without test-time adaptation; no future test frames are ever used during adaptation, and we will separate these two settings cleanly in the revision.

  • Zero-Shot Protocol (Table 5): All metrics in Table 5 are obtained strictly zero-shot by applying the ADYTON-trained model directly to unseen real-world Spring-Gaus sequences with zero fine-tuning.
  • Test-Time Adaptation Protocol: In our optional test-time adaptation experiment, the model is adapted briefly using only the observed context frames from a single camera view (never future frames) and evaluated on held-out novel camera views and future rollout steps. We thank the reviewer for catching the unclear wording in Sec. 6 and will move the test-time adaptation results into a dedicated subsection to clearly distinguish both protocols.

3. Obtaining Interior Query Points from Real-World RGB-D (Table 2)

DELPHI requires no ground-truth interior geometry at inference: probing DELPHI’s learned occupancy features over a depth-masked grid recovers interior queries in 0.24 s/object while matching or outperforming baselines given true geometry.

  • Addressing Table 2: Yes, Table 2 originally queried DELPHI at simulator particle positions. To evaluate practical deployment from monocular RGB-D and masks alone, we sample a 3D grid behind the observed surface and pass candidate queries through a lightweight occupancy filter (trained on ADYTON scenes) that probes DELPHI’s features to separate interior object points from ambient background.
  • Whole-Object Benchmark Without Privileged Geometry: Evaluating next-step symmetric Chamfer distance (cm) where every method supplies its own hidden geometry confirms that full-volume prediction remains practical and accurate:
MethodInputHeuristic grid + filter (0.24 s/object)SAM 3D (3.12 s/object)True geometry (reference)
No motionRGB-D–2.692.22
Any4D + smooth completionRGB-D–1.811.33
SonataRGB-D + true 3D tracks of observed points–1.851.16
DELPHIRGB-D1.311.731.04

Next-step symmetric Chamfer distance (cm) given context ground-truth masks and depth. Times reflect per-object geometry proposal latency.

  • Key Takeaway: Without true geometry, DELPHI (1.31 cm) matches Any4D given true geometry (1.33 cm) and beats all SAM 3D baselines, including Sonata given true 3D tracks (1.85 cm), while remaining the most accurate method given true geometry (1.04 cm). At 0.24 s per object (13× faster than SAM 3D), full-volume prediction transfers directly to real-world RGB-D observations. We will add this experiment to the revision.

Baseline Completion Definitions:

  • Rigid completion: Fits a single SE(3) transformation (rotation and translation) per object to the baseline's predicted visible-surface motion and applies it to the true hidden particles.
  • Smooth completion: Interpolates the baseline's predicted visible-surface motion into a continuous, non-rigid field across the object volume and evaluates it at the true hidden particles.
  • SAM 3D completion: Reconstructs the full 3D object geometry (including occluded regions) from the current RGB-D frame and mask, then propagates those reconstructed points using the rigid or smooth motion rules above.

(Note: Rigid and smooth completion still receive privileged ground-truth hidden particle locations and only infer their motion, whereas SAM 3D completion, like DELPHI's heuristic grid + filter, must also infer where the hidden geometry lies.)

Thank you for the clarifications and additional experiments. My concerns have been largely addressed. I will maintain my score of 3.

Thank you for your follow-up and for confirming that your concerns are largely resolved. We are especially glad that the new monocular RGB-D evaluation addressed the question on inference without privileged interior geometry. Probing DELPHI's occupancy features recovers interior points in 0.24 s/object (13× faster than SAM 3D) at 1.31 cm Chamfer distance, matching baselines given ground-truth geometry (1.33 cm). We will integrate this complete comparison into the revision.

Given that these main concerns have been resolved, we kindly ask if you would consider raising your score. Thanks again for your valuable feedback!

Reviewer_TZ9wOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies motion prediction throughout the entire volume of deformable objects, including occluded surfaces and object interiors, from monocular RGB-D videos. It introduces ADYTON, a physics-simulated benchmark with dense volumetric motion labels and controlled evaluation under changes in initial conditions, object combinations, and backgrounds. It also proposes DELPHI, a vision-conditioned implicit displacement field that predicts motion at queried 3D positions independently of the observed input points. DELPHI achieves substantially stronger full-volume prediction results on ADYTON while remaining competitive on visible-surface prediction. The paper additionally evaluates autoregressive rollouts and transfer to real-world videos.

Critical strengths
  1. A useful benchmark resource for studying volumetric dynamics. ADYTON provides dense simulated motion labels covering visible surfaces, occluded regions, and object interiors, together with controlled tests on initial conditions, object combinations, and backgrounds. This supports systematic evaluation of motion beyond the observed surface.

  2. An effective motion representation within the proposed setting. DELPHI’s queryable displacement field enables predictions at unobserved locations and achieves strong full-volume results while maintaining competitive visible-surface performance. Its improvement over PGND is particularly relevant because both methods receive full-volume supervision, providing evidence for the effectiveness of the proposed architecture within this benchmark.

Critical weaknesses
  1. The assumptions supporting the central prediction objective are insufficiently justified. Although a monocular RGB-D history provides geometric and temporal information, motion throughout the hidden volume is not directly observed. Related approaches use additional views or explicit geometric and physical priors to constrain inference, as illustrated by Spring-Gaus and PhysTwin. Sections 2 and 4 acknowledge a fixed object pool, which allows learned object-specific priors to support prediction. However, beyond this restriction, the paper does not sufficiently explain which assumptions about geometry, material properties, and hidden physical state make accurate full-volume prediction feasible from the available observations. The results demonstrate that the task is learnable within ADYTON, but its information requirements and intended scope remain insufficiently established.

  2. The scientific significance of prioritizing full-volume accuracy remains unclear. The paper does not sufficiently justify why pursuing this objective within a fixed object set advances dynamics prediction more broadly. The substantial full-volume gains do not yield a clear advantage on the other motion-prediction evaluations: visible-surface MDE is 1.78 cm versus 1.52 cm for Sonata (Table 2), while the 10-step rollout MDE improves only from 24.83 to 23.86, with worse results at steps 1 and 5 (Table 4). The Spring-Gaus experiments provide evidence of transfer to unseen real objects, but their image-rendering metrics do not directly validate full-volume motion accuracy (Table 5). Consequently, the broader benefit motivating this research target remains insufficiently demonstrated, and its extension to a large and diverse range of unseen objects remains open.

Questions for authors
  1. Feasibility and assumptions — clarification. Beyond restricting the benchmark to a fixed object set, what assumptions about object geometry, material properties, and hidden physical state make full-volume motion sufficiently constrained by the monocular RGB-D history to support accurate prediction? How are these assumptions reflected in the benchmark and the priors learned during training?

  2. Scientific justification — clarification. Within the stated fixed-object scope, what supports choosing accurate full-volume motion prediction as a central research objective for advancing dynamics modeling? In particular, how should the substantial full-volume gains be interpreted given the comparable visible-surface performance and the modest, mixed improvements in autoregressive prediction?

6 comments

We thank the reviewer for the constructive review and valuable feedback.

1. Feasibility & Physical Assumptions Behind Full-Volume Inference

  • From 2D-to-3D to 4D Field Inference: Prior work shows that predicting complete 3D shape from 2D observations is viable given appropriate data and architecture (e.g., SAM3D [1], Zero-1-to-3 [2]); DELPHI extends this paradigm from 2D-to-3D into 4D dynamics by combining high-fidelity simulation data with a continuous spatial field representation.
  • Geometric & Material Constraints: ADYTON objects are modeled as elastic and elastoplastic MPM continua (parameters in App. B.2). Under continuum mechanics, surface boundary displacements and contact impulses propagate smoothly into the interior rather than leaving internal particles free to move independently.
  • Hidden Physical State from History: While a single frame leaves velocity and stress unobserved, the multi-frame RGB-D history supplies explicit surface velocity, acceleration, and contact onset. Combined with learned structural priors, DELPHI resolves full volume displacements without multi-view cameras or per-object optimization (unlike Spring-Gaus and PhysTwin).
  • Scope & Scaling: We acknowledge that DELPHI does not generalize well to objects or materials completely unseen during training (e.g., cloth or liquids). However, because our field architecture imposes no analytical motion or rigid priors, its modeling capability naturally expands as training data scales.

2. Scientific Significance & Metric Interpretation

Visible-surface tracking and full-volume dynamics address different physical regimes: surface pixels alone miss interior mass and contact forces, whereas DELPHI outperforms all baselines when evaluated fairly on whole-object shape prediction.

  • Why Full-Volume Matters: Pixel-aligned trackers (e.g., Sonata) perform well on visible points (1.52 cm vs. DELPHI’s 1.78 cm) via local appearance matching, but physical collisions and momentum transfer are governed by volumetric mass, not just the camera-facing shell. During contact impact, an oracle rigid fit leaves 0.83 cm of interior error, whereas DELPHI recovers the true non-rigid deformation direction (Cosine similarity: DELPHI 0.72 vs. rigid completion 0.17).
  • Fair Whole-Object Comparison (Table 2): We replaced the zero-padded evaluation with a Chamfer distance protocol where every method supplies its own hidden geometry from context RGB-D and masks:
MethodInputHeuristic grid + filter (0.24 s/object)SAM 3D (3.12 s/object)True geometry (reference)
No motionRGB-D–2.692.22
Any4D + smooth completionRGB-D–1.811.33
SonataRGB-D + true 3D tracks of observed points–1.851.16
DELPHIRGB-D1.311.731.04

Next-step symmetric Chamfer distance (cm) given context ground-truth masks and depth. Times reflect per-object geometry proposal latency.

  • Takeaway: Using a lightweight occupancy filter over a depth-masked grid (0.24 s/object, 13× faster than SAM 3D) to separate object queries from ambient background, DELPHI (1.31 cm) matches Any4D given true geometry (1.33 cm) and outperforms every SAM 3D baseline, including Sonata given true 3D tracks (1.85 cm). Given true geometry, DELPHI is also the most accurate method overall (1.04 cm).

Baseline Completion Definitions:

  • Rigid completion: Fits a single SE(3) transformation (rotation and translation) per object to the baseline's predicted visible-surface motion and applies it to the true hidden particles.
  • Smooth completion: Interpolates the baseline's predicted visible-surface motion into a continuous, non-rigid field across the object volume and evaluates it at the true hidden particles.
  • SAM 3D completion: Reconstructs the full 3D object geometry (including occluded regions) from the current RGB-D frame and mask, then propagates those reconstructed points using the rigid or smooth motion rules above.

(Note: Rigid and smooth completion still receive privileged ground-truth hidden particle locations and only infer their motion, whereas SAM 3D completion, like DELPHI's heuristic grid + filter, must also infer where the hidden geometry lies.)

[1] Chen, Xingyu, et al. Sam 3d: 3dfy anything in images. CVPR 2026
[2] Liu, Ruoshi, et al. Zero-1-to-3: Zero-shot one image to 3d object. ICCV 2023

Thank you for the clarifications and additional experiments. These strengthen the evidence for full-volume prediction within ADYTON. However, SAM 3D and Zero-1-to-3 acknowledge ambiguity in static reconstruction and do not directly establish the conditions for accurate hidden-motion prediction. The new Chamfer results still evaluate whole-object prediction, while the substantial full-volume gains have not been accompanied by clear, consistent improvements in visible-surface prediction or autoregressive rollouts. Consequently, my concerns about the justification and broader predictive benefit of prioritizing accurate full-volume motion prediction from monocular observations remain only partially addressed. I will maintain my rating of 2 (weak rejection).

We thank the reviewer for engaging with the new results. In fact, your point that "SAM 3D and Zero-1-to-3 acknowledge ambiguity in static reconstruction and do not directly establish the conditions for accurate hidden-motion prediction" is precisely the motivation behind DELPHI and ADYTON, and why they move beyond prior art. Inferring complex dynamics from partial observations via in-context prediction is well explored in robotics (e.g., Rapid Motor Adaptation [1] and AnyCar to Anywhere [2]), and we extend this concept to predict full-volume motion by pairing spatiotemporal context with volumetric supervision at scale. We also want to clarify that our full-volume gains do not carry over to the visible surface or autoregressive rollouts. We address your remaining concerns below.

1. Going beyond static ambiguity: Why hidden geometry and motion must be learned jointly

Single-image models (e.g., SAM 3D, Zero-1-to-3) have per-frame reconstruction ambiguity. Reconstructing frames independently and differencing them fails because nothing keeps the hidden geometry consistent over time. Evaluating SAM 3D across 8-frame windows on 30 test sequences confirms this temporal inconsistency:

MetricDELPHISAM 3D (per-frame)
Per-frame Chamfer / IoU*1.30 cm / 0.561.78 cm / 0.48
Temporal consistency t→t+1t \to t+1 (IoU after true motion)0.850.74

*Measured on DELPHI’s per-frame occupancy readout.

2. The value of volumetric prediction

We want to clarify that in the Chamfer experiments, reconstruction serves only to supply spatial query coordinates rather than predicting motion, confirming that DELPHI requires no ground-truth queries at inference. We also want to clarify that we do not claim volumetric supervision improves visible-surface accuracy; it matches surface-only performance without degradation (1.84 cm vs. 1.77 cm for ground-truth surface tracks and 2.53 cm for the monocular baseline). Because our rollouts re-render only the visible surface at each step, rollout accuracy is bounded by this visible-surface accuracy. The core value of learning a full 4D volumetric field lies outside that loop, unlocking three key capabilities:

  • Dense Correspondence Across Time: Predicting a continuous 3D displacement field tracks every surface and interior point over time, maintaining consistent temporal correspondence without brittle frame-to-frame re-association (Response 1).
  • Complete 3D Geometry: Conditioned on a single camera, DELPHI predicts surfaces seen only by three held-out views as accurately as the input view (1.33–1.40 cm vs. 1.41 cm). Totally occluded points reach 1.83 cm, matching the 1.79 cm ground-truth-track surface oracle and beating the 3.02 cm monocular baseline. This provides the full 3D geometry needed for collision checking and grasp planning.
  • Physical Interpretability: Verifying adherence to physical laws, such as energy conservation in deformable objects, requires internal potential energy quantities (strain and stress) that only full-volume motion can provide. For interior strain accuracy (1−∑∥e^−e∥F2/∑∥e∥F21 - \sum \|\hat{e} - e\|_F^2 / \sum \|e\|_F^2, where 1 is exact), volumetric queries score 0.93 versus 0.27 from the closed surface alone (both under ground-truth motion). DELPHI already captures this internal physical state at the body level, with predicted body-mean pressure correlating at r=0.82r = 0.82 with the simulator. This enables future applications in internal stress estimation, deformation-aware manipulation, and fracture prediction.
Reviewer_P1uMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work presets a simulated dataset of multiple elastic objects interacting in a Genesis physics simulator. Using this dataset, the authors train a model, that predicts future scene flow given currently observed video frame. The representation allows arbitrary 3D position queries, thereby allowing for dense scene flow estimation. The method is trained on the introduced benchmark and mostly compared to 3D future point track predictors.

Critical strengths

The overall formulation of predicting dense 3D scene flow, rather than only tracking visible points is important and interesting

Critical weaknesses
  • My biggest concern about this work — does it learn anything useful beyond approximately rigid motion? Even prediction that motion is often incorrect — on real samples it seems to be predicting coherent, yet incorrect, rigid-like object motion. Objects in this dataset are almost rigid (they are not like, cloth or animals), so I’m wondering if we are there yet to learn their subtle elastic motion.

  • I’m not sure what’s going on in the real scenes, Figure 3b. It seems that GT sequences do not abide to the laws physics of physics? The scene with potato on the checkerboard seems to be placed backwards?

  • From the experiments it seems that all models were re-trained on the train set and then evaluated on their test set? I’m not sure if similar performance on the same set of objects makes too much sense though..

  • The only real baseline is PGND and the only main architectural difference is the encoder: in case of PGND it is a PointNet, whereas in this work is a LoRA adapted Any6D visual encoder. Is this the reason PGND basically fails the task?

  • I think the dataset comparison in Table 1 a bit deceiving — for example for Kubric and PointOdyssey, we could extract full scene point tracks too. We do have access to the simulator. I guess in this case author’s materials are elastic. What would make sense is to create a large sim dataset which can be added to the existing articulated and rigid ones.

  • I understand what authors are trying to say about most point tracking models being pixel aligned in their predictions. However, it doesn’t seem to be fair to also benchmark on the full scene flow estimation “Because they lack the ability to predict amodal motion, we set motion of non-visible points to zero at inference.” But then the benchmarking quality will be dominated by these dummy predictions. What are we measuring then?

Overall I struggle to see that this exact benchmark will be found useful by the community. The formulation of the method (i.e tracking occluded 3D points too) is interesting, but in it's current version I don't think this works provides novel insights

Questions for authors

Would be really helpful to understand why does a similar baseline model (Particle-Grid Neural Dynamics) performs so poorly? Are all baseline models retrained from scratch?

5 comments

We thank the reviewer for the constructive review and valuable feedback.

1. Learning Non-Rigid Deformation Beyond Rigid Motion

DELPHI explicitly captures non-rigid interior deformation during contact, achieving a 0.72 directional alignment (Cosine Similarity) with true non-rigid residuals (vs. 0.17 for rigid-completion baselines).

  • All ADYTON objects are simulated using elastic and elastoplastic MPM materials (parameters in App. B.2), and our real-world evaluation uses deformable objects from Spring-Gaus.
  • During the impact phase, even an oracle rigid fit leaves 0.83 cm of interior error. DELPHI accurately recovers the non-rigid deformation field (0.72 cosine similarity between predicted and true non-rigid residuals), whereas Any4D with rigid completion fails to capture local compression (0.17).

2. Real-World Ground Truth & Figure 3b Ordering

Figure 3b shows an actual physical rebound captured in real-world video, not a reversed sequence.

  • The ground-truth frames come directly from real-world captured footage and obey physics by construction.
  • The panels display rollout steps 1, 5, and 10: between steps 5 and 10, the potato impacts the checkerboard and rebounds upward and backward. We will add explicit timestamps and impact-frame labels to Figure 3b, and include the full per-frame sequence in Appendix.

3. Generalization Scope Across Objects and Interactions

Unlike prior per-object deformable models (e.g., Spring-Gaus, PhysTwin), DELPHI trains a single unified model across all objects and generalizes to novel multi-object combinations and physical interactions.

  • Every baseline and DELPHI is trained once across the entire ADYTON training set and evaluated on held-out test scenes featuring unseen object pairings, initial states, and contact dynamics.
  • Scaling from per-object optimization to a single multi-object model is already a substantial step forward; we leave zero-shot generalization to entirely unseen object categories as future work.

4. Why PGND Fails in Practical Settings (Points 4 & 7)

All baselines are retrained from scratch on ADYTON, and PGND collapses because it relies on ground-truth temporal point correspondence—which does not exist in raw RGB-D observations.

  • PGND’s architecture assumes known point-to-point correspondence across frames. When evaluated under standard visual inputs (RGB-D without correspondence), its error jumps from 1.06 cm to 7.13 cm, whereas DELPHI’s vision-grounded spatial queries achieve 1.78 cm without correspondence:
MethodFull points + corr.Trained without corr. (RGB-D)
PGND1.067.13
DELPHI–1.78

5. Dataset Focus: High-DoF Deformables vs. SE(3) Rigid Bodies (Table 1)

Rigid-body datasets reduce to global SE(3) transformations with no interior dynamics, whereas ADYTON isolates the high-DoF, non-uniform motion unique to deformables.

  • While dense points can be extracted from rigid datasets like Kubric or PointOdyssey, rigid motion is fundamentally low-dimensional and requires no interior modeling.
  • Deformable objects offer no SE(3) shortcut: capturing non-uniform local motion demands high-DoF dense representations. Because DELPHI imposes no rigid-body priors, it naturally generalizes across material types, making joint rigid-deformable training an immediate avenue for future work.

6. Fair Evaluation Without Privileged Query Points

We replaced the zero-padded evaluation with a Chamfer distance protocol where every method supplies its own hidden geometry—and DELPHI still outperforms every baseline.

  • We agree that querying DELPHI at true interior particles gave it an unfair advantage over pixel-aligned baselines. In our revised protocol, every model must infer complete 3D object shape from context RGB-D and masks alone, evaluated via next-step symmetric Chamfer distance (cm):
MethodInputHeuristic grid + filter (0.24 s/object)SAM 3D (3.12 s/object)True geometry (reference)
No motionRGB-D–2.692.22
Any4D + smooth completionRGB-D–1.811.33
SonataRGB-D + true 3D tracks of observed points–1.851.16
DELPHIRGB-D1.311.731.04

Next-step symmetric Chamfer distance (cm) given context ground-truth masks and depth. Times reflect per-object geometry proposal latency.

  • Without true geometry, DELPHI (1.31 cm) matches Any4D given true geometry (1.33 cm) and beats all SAM 3D baselines, including Sonata given true 3D tracks (1.85 cm).
  • DELPHI uses a lightweight occupancy filter (trained on ADYTON) over a depth-masked grid to separate object queries from ambient background, improving motion accuracy at 0.24 s/object (13× faster than SAM 3D). Given true geometry, DELPHI remains the most accurate method overall (1.04 cm).

Baseline Completion Definitions:

  • Rigid completion: Fits a single SE(3) transformation (rotation and translation) per object to the baseline's predicted visible-surface motion and applies it to the true hidden particles.
  • Smooth completion: Interpolates the baseline's predicted visible-surface motion into a continuous, non-rigid field across the object volume and evaluates it at the true hidden particles.
  • SAM 3D completion: Reconstructs the full 3D object geometry (including occluded regions) from the current RGB-D frame and mask, then propagates those reconstructed points using the rigid or smooth motion rules above.

(Note: Rigid and smooth completion still receive privileged ground-truth hidden particle locations and only infer their motion, whereas SAM 3D completion, like DELPHI's heuristic grid + filter, must also infer where the hidden geometry lies.)

AuthorsOpenReview ↗

We kindly ask the reviewer to review our clarifications and newly added results.

Dear Reviewer, Please let us know if any further clarification is needed. We would be glad to discuss further! Thanks!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for Area Chair: Addressed Concerns & Revision Commitments

Dear Area Chair,

Thank you for overseeing the review process. Across our rebuttal and follow-up discussions, we have addressed the vast majority of questions and concerns raised by all three reviewers with new experiments, fair evaluation protocols, and clear theoretical clarifications. We will integrate all of these additions into the revised manuscript.

Here is a direct summary of each raised concern and our conclusion:

Reviewer xFaR (Rating: 3 — Confirmed concerns are largely addressed)

  • Real-world physical validation & videos →\rightarrow Resolved: Because real RGB-D captures lack ground-truth interior tracks, we directly validated occluded impact dynamics on ADYTON, confirming DELPHI outperforms baselines across all physical regimes. We will also add side-by-side real-world rollout videos to the Appendix.
  • Zero-shot vs. test-time adaptation protocols (Sec. 6 & Table 5) →\rightarrow Clarified: Table 5 is strictly zero-shot with no fine-tuning on real videos, whereas optional test-time adaptation uses only past context frames (never future frames) and will be separated into its own subsection.
  • Inference without ground-truth query points (Table 2) →\rightarrow Resolved: Querying DELPHI’s learned occupancy readout over a depth-masked grid enables fast, accurate whole-object prediction without privileged geometry, outperforming all SAM 3D baselines.

Reviewer P1uM (Two rounds of discussion completed)

  • Direct 3D position error vs. rigid oracle (Follow-up Q1) →\rightarrow Resolved: Replaced cosine similarity with full 3D Mean Displacement Error (MDE) across all motion phases, confirming DELPHI outperforms the rigid completion baseline before, during, and after impact.
  • "Heuristic grid + filter" vs. out-of-the-box motion prediction (Follow-up Q2) →\rightarrow Clarified: No new motion module was added. DELPHI natively predicts a continuous 4D displacement field out of the box; the occupancy-filtered grid simply supplies spatial query coordinates to discretize that field when ground-truth object points are unavailable at inference.
  • PGND baseline, dataset scope & Fig. 3b ordering (Initial Review) →\rightarrow Resolved: Showed that PGND collapses without ground-truth temporal correspondence, justified ADYTON’s focus on high-DoF deformables over rigid bodies, clarified our unified multi-object generalization scope, and confirmed Fig. 3b depicts a true physical rebound (to be labeled with explicit timestamps).

Reviewer TZ9w (Two rounds of discussion completed)

  • Physical assumptions & scope for hidden-motion prediction (Q1 & Follow-up) →\rightarrow Resolved: Established the three physical conditions making monocular full-volume prediction feasible—object-pool shape/material priors inferred in-context, multi-frame state observability, and passive continuum kinematic coupling—and clearly bounded our scope to passive deformable objects.
  • Joint 4D learning vs. per-frame static reconstruction (Follow-up) →\rightarrow Resolved: Evaluated per-frame SAM 3D across temporal windows and showed it suffers from severe frame-to-frame inconsistency, confirming the necessity of DELPHI’s joint spatiotemporal supervision.
  • Scientific value beyond visible surfaces & rollouts (Q2 & Follow-up) →\rightarrow Resolved: Clarified that DELPHI matches visible-surface accuracy without degradation while unlocking three essential capabilities outside the visible-surface loop:
    1. Dense temporal correspondence across both surface and interior points without brittle frame-to-frame re-association.
    2. Complete 3D geometry across unseen camera views and totally occluded regions for downstream collision checking and grasp planning.
    3. Internal physical quantity derivation (interior strain and pressure) required for continuum mechanics and deformation-aware manipulation.

Commitment to the Revision: We will incorporate all new experimental analyses, side-by-side real-world video comparisons, and explicit clarifications on physical assumptions, scope, and inference querying into the final revision. We respectfully ask the Area Chair to consider these updates in the final decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.