- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AIM to learn deformable-object dynamics from reconstructed particle trajectories and prescribed kinematic controls. It combines history-conditioned relation inference over spatial and retained particle pairs with geometry-conditioned global communication, and trains through autoregressive trajectory supervision. Evaluations on PhysTwin and PGND cover resimulation, future prediction, held-out interactions, and transfer across actions, objects, and manipulation domains.
Critical strengths
The strongest contribution is the empirical improvement across two real-world benchmarks. AIM reduces future-prediction tracking error from 0.025 to 0.020 on PhysTwin, and improves MDE, CD, and EMD in all six PGND categories. The 37-step rollouts also extend beyond the five-step training horizon. Frozen-model transfer improves tracking on all three robot-to-human examples. These results provide useful evidence that the complete system predicts deformation more accurately across several interaction settings.
Critical weaknesses
-
The specific methodological advance is not sufficiently established. History-dependent relation inference and global graph communication have clear precedents. The weighted-cosine similarity component also appears in IDGL (NeurIPS 2020). AIM’s more distinctive choices are relation retention and rescoring, together with geometry-conditioned global redistribution. However, all component ablations use one scene and one seed, with different parameter counts (Appendix D.4). In Fig. 6, replacing adaptive relations with a radius graph while retaining global communication changes tracking error from 6.6 to 6.8 mm; removing global communication increases it to 15.9 mm. Adaptive relations also improve IoU, but the radius variants remove learned relation features as well, while distance ranking preserves relation retention. These comparisons leave the contributions of history conditioning and retained connections unresolved, particularly for the long-horizon and transfer settings motivating the method.
-
Important parts of the transfer protocol remain unspecified. The training and held-out scene composition for Table 3 is not given. Section 6.3 points to Appendix D.2 for per-scene results, but that section instead details the single-scene transfer pairs summarized in Table 2. Cross-dataset evaluation also relies on unspecified coordinate and temporal alignment. Finally, the criterion for selecting source and ablation checkpoints is not stated; Appendix D.4 only mentions individually selected checkpoints. These details are needed to assess the scale of held-out generalization and how model selection is separated from evaluation.
Questions for authors
-
What are the exact training and held-out scenes and counts for Table 3? How are source and ablation checkpoints selected, and which target observations are used to determine the coordinate and temporal alignment?
-
Does AIM retain its advantage over the radius-global variant on held-out rollouts and transfer when parameter counts are matched and variation across training seeds is reported? A separate removal of relation retention would help establish whether that mechanism explains the gains.
4 comments
W2&Q1. Clarity of the transfer protocol. The training and held-out scene composition for Table 3 is not specified, and Appendix D.2 instead describes the single-scene transfer pairs in Table 2. How are checkpoints selected, and which target observations are used for coordinate and temporal alignment?
We sincerely thank the reviewer for identifying these omissions and apologize that the original presentation did not sufficiently clarify the transfer protocols. We have revised all three parts accordingly.
-
Clarify the scene composition for Table 3. We have added Appendix C.4, “Category-Level Generalization Protocols,” to specify the training and held-out sequences for Table 3, with a corresponding reference in Section 6.3. This distinguishes category-level generalization from the single-scene transfer experiments in Table 2. Each category-level setting trains a separate shared model. The held-out cloth setting uses seven training sequences (
cloth_shirt_lift,cloth_skirt_2_fold,cloth_red_lift,cloth_red_fold,cloth_blue_fold,cloth_blue_lift, andcloth_skirt_1_fold) and two test sequences (cloth_shirt_foldandcloth_pant_lift). The lift-to-fold setting trains on the fivecloth_{blue,pant,red,shirt,skirt_1}_liftsequences and tests on their five corresponding_foldsequences. -
Specify the observations used for alignment. We have added the alignment and initialization details to Appendix C.3, “Evaluation Protocol,” under “Coordinate and temporal alignment” and “Cross-dataset rollout initialization.” Coordinate conversion uses fixed axis and scale mappings, with a translation determined solely from target initialization observations and held constant throughout rollout. Source-training normalization statistics are retained, and predictions are converted back to PhysTwin world coordinates for evaluation. Temporal alignment samples the 30 FPS target sequences every three frames to preserve the source model’s 0.1 s time step. For prediction beginning at frame , the model observes frames , , and , then predicts autoregressively under the provided controls. No future target object states are used to determine alignment, and evaluation involves neither subsequent ground-truth state resets nor target-specific parameter updates.
-
Make checkpoint selection explicit. We have clarified checkpoint selection and reuse in Appendix C.3, under “Checkpoint selection” and “Source-model reuse for transfer,” with a summary in Section 5.3. PGND, PhysTwin, and category-level experiments select the saved checkpoint with the lowest training loss under each model’s respective objective. This rule applies to AIM, reproduced baselines, and all ablation variants, with one checkpoint supplying every reported metric for a given run. Cross-object, cross-action, and cross-dataset transfer reuse the corresponding in-domain source checkpoint without reselection based on target performance, keeping source-model selection separate from target evaluation.
Q3. Does AIM retain its advantage when variation across training seeds is reported?
We thank the reviewer for highlighting the importance of initialization sensitivity. We additionally evaluated AIM and the three targeted controls with multiple random seeds on rope_double_hand (1D), double_lift_cloth_3 (2D), and double_stretch_zebra (3D) for reconstruction and resimulation (E1). We also evaluated cross-action transfer from double_lift_cloth_3 to single_clift_cloth_3 across training seeds, reusing each run’s source checkpoint without target-based reselection. The following tables report mean ± standard deviation.
These results have been added to Appendix D.4, under “Variation across Training Seeds,” with source-scene results in Tables 13–15 and cross-action transfer results in Table 16.
Three-seed Reconstruction & Resimulation results: rope_double_hand (1D).
| Model | CD (mm) ↓ | Track (mm) ↓ | IoU (%) ↑ | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|---|---|
| w/o Adaptive Graph (capacity-matched) | 5.631 ± 0.992 | 10.729 ± 1.858 | 73.933 ± 2.276 | 28.382 ± 0.662 | 0.97433 ± 0.00167 | 0.02269 ± 0.00474 |
| w/o Relation Retention | 5.674 ± 0.167 | 10.780 ± 0.137 | 74.015 ± 1.981 | 28.659 ± 0.682 | 0.96806 ± 0.00054 | 0.02286 ± 0.00080 |
| w/o Relation-History Conditioning | 5.422 ± 0.042 | 10.625 ± 0.393 | 73.873 ± 0.566 | 28.573 ± 0.430 | 0.97119 ± 0.00116 | 0.02226 ± 0.00179 |
| AIM | 5.104 ± 0.036 | 10.396 ± 1.830 | 74.058 ± 0.653 | 28.687 ± 0.104 | 0.97583 ± 0.00044 | 0.02197 ± 0.00140 |
Three-seed Reconstruction & Resimulation results: double_lift_cloth_3 (2D).
| Model | CD (mm) ↓ | Track (mm) ↓ | IoU (%) ↑ | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|---|---|
| w/o Adaptive Graph (capacity-matched) | 8.597 ± 0.044 | 6.719 ± 0.178 | 96.431 ± 0.418 | 29.048 ± 0.609 | 0.88577 ± 0.00805 | 0.03832 ± 0.00273 |
| w/o Relation Retention | 8.772 ± 0.123 | 6.350 ± 0.205 | 96.253 ± 0.260 | 28.804 ± 0.142 | 0.88401 ± 0.00115 | 0.03988 ± 0.00181 |
| w/o Relation-History Conditioning | 8.787 ± 0.262 | 6.187 ± 0.042 | 96.371 ± 0.104 | 29.092 ± 0.064 | 0.88617 ± 0.00177 | 0.03850 ± 0.00083 |
| AIM | 8.566 ± 0.210 | 6.066 ± 0.071 | 96.494 ± 0.027 | 29.115 ± 0.110 | 0.88713 ± 0.00246 | 0.03821 ± 0.00007 |
Three-seed Reconstruction & Resimulation results: double_stretch_zebra (3D).
| Model | CD (mm) ↓ | Track (mm) ↓ | IoU (%) ↑ | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|---|---|
| w/o Adaptive Graph (capacity-matched) | 7.638 ± 0.571 | 9.952 ± 2.320 | 77.938 ± 4.246 | 26.871 ± 0.714 | 0.95507 ± 0.00351 | 0.02647 ± 0.00333 |
| w/o Relation Retention | 7.565 ± 0.764 | 9.534 ± 1.727 | 80.203 ± 2.238 | 27.282 ± 0.443 | 0.95704 ± 0.00070 | 0.02547 ± 0.00470 |
| w/o Relation-History Conditioning | 7.412 ± 0.184 | 8.564 ± 0.044 | 80.542 ± 1.013 | 27.349 ± 0.227 | 0.95607 ± 0.00132 | 0.02517 ± 0.00052 |
| AIM | 6.610 ± 0.022 | 6.492 ± 0.336 | 82.968 ± 2.138 | 27.786 ± 0.142 | 0.95775 ± 0.00255 | 0.02219 ± 0.00237 |
Three-seed cross-action transfer: double_lift_cloth_3 → single_clift_cloth_3.
| Model | CD (mm) ↓ | Track (mm) ↓ | IoU (%) ↑ | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|---|---|
| w/o Adaptive Graph (capacity-matched) | 18.829 ± 0.264 | 24.783 ± 0.216 | 92.611 ± 0.279 | 26.962 ± 0.075 | 0.87587 ± 0.01323 | 0.06061 ± 0.00083 |
| w/o Relation Retention | 10.626 ± 0.680 | 21.988 ± 2.739 | 94.483 ± 0.080 | 28.113 ± 0.015 | 0.87402 ± 0.00013 | 0.04673 ± 0.00084 |
| w/o Relation-History Conditioning | 10.550 ± 0.097 | 23.050 ± 0.319 | 94.312 ± 0.307 | 28.018 ± 0.341 | 0.87432 ± 0.00388 | 0.04793 ± 0.00207 |
| AIM | 10.283 ± 0.903 | 21.057 ± 1.446 | 95.097 ± 0.297 | 28.257 ± 0.404 | 0.88367 ± 0.00336 | 0.04650 ± 0.00019 |
AIM consistently achieves the best mean performance across all six metrics in the multi-seed E1 and cross-action comparisons, demonstrating improvements in particle motion, geometric accuracy, and rendering quality. These repeated-run results support the complementary contributions of relation retention and history-conditioned scoring, showing that the observed advantages extend beyond a single initialization.
We sincerely seek your feedback!
Dear Reviewer igdh,
We hope this message finds you well. We understand that the review period is busy and sincerely appreciate the time you have already devoted to our submission. As the discussion period is nearing its end, we wanted to ensure we have addressed your concerns satisfactorily.
We would be grateful for any final feedback and remain happy to address any further questions.
Thank you again for your consideration!
Best regards, The Authors
We thank the reviewer for this suggestion. We added three targeted controls and evaluated them across all 22 PhysTwin scenes, covering reconstruction and resimulation (E1), future prediction (E2), and the paper’s three cross-action and two cross-object transfer pairs:
These controls assess adaptive relations beyond model capacity and isolate the contributions of relation retention and history-conditioned scoring. Their definitions and results are provided in Appendix D.4.2, “Additional Controls for Relation Modeling,” with E1/E2 results in Table 11 and cross-action and cross-object transfer results in Table 12. All transfer evaluations reuse source checkpoints without target-based reselection.
Reconstruction & Resimulation: Scene-averaged results across all 22 PhysTwin scenes.
Future Prediction: Scene-averaged results across all 22 PhysTwin scenes.
Cross-action transfer.
Cross-object transfer.
AIM achieves the best averages across all six metrics in both scene-specific and transfer evaluations. The targeted removals support the complementary roles of retention in keeping useful dependencies available as spatial neighborhoods change and motion history in reassessing their relevance. These improvements extend across particle trajectories, geometric reconstruction, and rendering quality.
When adaptive connectivity matters. We additionally provide an E1 comparison on
double_stretch_zebra.AIM improves all six metrics over the capacity-matched spatial control, supporting the benefit of adaptive relations beyond model capacity and learned edge processing. This comparison is motivated by situations where spatially close regions respond differently to manipulation: radius-based connectivity may introduce inappropriate interactions, while AIM uses motion context to retain and reassess deformation dependencies. We present the corresponding visual analysis in Section 6.5 with Figure 7 illustrating local deformation differences.