acceptodds
Withdrawn from ICLR 2027

AIM: Adaptive Interaction Modeling Networks for Real-to-Sim Soft-Body Simulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Deformable-object manipulation is essential for robotic tasks such as folding laundry and handling food, where robots must control shape changes as well as object motion. Predictive soft-body simulation supports these tasks by anticipating deformation under external interactions. However, spatial neighborhoods can misrepresent deformation dependencies, introducing local errors that accumulate over successive predictions. Models fitted to individual scenes must also accommodate changes in object geometry and manipulation conditions. In this work, we propose \themodel, an Adaptive Interaction Modeling framework that treats real-to-sim soft-body simulation as a local-global interaction modeling problem. \themodel uses motion history and geometry to adapt particle relations over current spatial neighbors and retained connections, while geometry-conditioned global communication coordinates object-wide responses. A unified kinematic control-point interface represents different manipulation configurations, and multi-step autoregressive supervision trains the model on its own predicted trajectories. Experiments on PhysTwin and PGND demonstrate improved motion accuracy and visual fidelity, with a 20.0% reduction in future-prediction tracking error relative to PhysTwin and a 22.8% reduction in mean long-horizon particle error across six object categories relative to PGND. The framework further supports transfer across actions, object instances, and scenes, including zero-shot transfer from robot interactions to human manipulation without target-domain dynamics fitting.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Soft-Body Manipulation, Real-to-Sim, Neural Simulator

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_igdhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AIM to learn deformable-object dynamics from reconstructed particle trajectories and prescribed kinematic controls. It combines history-conditioned relation inference over spatial and retained particle pairs with geometry-conditioned global communication, and trains through autoregressive trajectory supervision. Evaluations on PhysTwin and PGND cover resimulation, future prediction, held-out interactions, and transfer across actions, objects, and manipulation domains.

Critical strengths

The strongest contribution is the empirical improvement across two real-world benchmarks. AIM reduces future-prediction tracking error from 0.025 to 0.020 on PhysTwin, and improves MDE, CD, and EMD in all six PGND categories. The 37-step rollouts also extend beyond the five-step training horizon. Frozen-model transfer improves tracking on all three robot-to-human examples. These results provide useful evidence that the complete system predicts deformation more accurately across several interaction settings.

Critical weaknesses
  1. The specific methodological advance is not sufficiently established. History-dependent relation inference and global graph communication have clear precedents. The weighted-cosine similarity component also appears in IDGL (NeurIPS 2020). AIM’s more distinctive choices are relation retention and rescoring, together with geometry-conditioned global redistribution. However, all component ablations use one scene and one seed, with different parameter counts (Appendix D.4). In Fig. 6, replacing adaptive relations with a radius graph while retaining global communication changes tracking error from 6.6 to 6.8 mm; removing global communication increases it to 15.9 mm. Adaptive relations also improve IoU, but the radius variants remove learned relation features as well, while distance ranking preserves relation retention. These comparisons leave the contributions of history conditioning and retained connections unresolved, particularly for the long-horizon and transfer settings motivating the method.

  2. Important parts of the transfer protocol remain unspecified. The training and held-out scene composition for Table 3 is not given. Section 6.3 points to Appendix D.2 for per-scene results, but that section instead details the single-scene transfer pairs summarized in Table 2. Cross-dataset evaluation also relies on unspecified coordinate and temporal alignment. Finally, the criterion for selecting source and ablation checkpoints is not stated; Appendix D.4 only mentions individually selected checkpoints. These details are needed to assess the scale of held-out generalization and how model selection is separated from evaluation.

Questions for authors
  • What are the exact training and held-out scenes and counts for Table 3? How are source and ablation checkpoints selected, and which target observations are used to determine the coordinate and temporal alignment?

  • Does AIM retain its advantage over the radius-global variant on held-out rollouts and transfer when parameter counts are matched and variation across training seeds is reported? A separate removal of relation retention would help establish whether that mechanism explains the gains.

4 comments

W1&Q2. Does AIM retain its advantage over the radius-global variant on held-out rollouts and transfer when parameter counts are matched and variation across training seeds is reported? A separate removal of relation retention would help establish whether that mechanism explains the gains.

We thank the reviewer for this suggestion. We added three targeted controls and evaluated them across all 22 PhysTwin scenes, covering reconstruction and resimulation (E1), future prediction (E2), and the paper’s three cross-action and two cross-object transfer pairs:

  • w/o Adaptive Graph (capacity-matched): uses spatial-radius connectivity with global communication and learned edge features, containing 3,171,048 parameters versus AIM’s 3,170,980.
  • w/o Relation Retention: excludes previously retained connections from the candidate set while preserving history-conditioned scoring and global communication.
  • w/o Relation-History Conditioning: removes motion history only from the relation scorer, preserving history inputs to the dynamics backbone.

These controls assess adaptive relations beyond model capacity and isolate the contributions of relation retention and history-conditioned scoring. Their definitions and results are provided in Appendix D.4.2, “Additional Controls for Relation Modeling,” with E1/E2 results in Table 11 and cross-action and cross-object transfer results in Table 12. All transfer evaluations reuse source checkpoints without target-based reselection.

Reconstruction & Resimulation: Scene-averaged results across all 22 PhysTwin scenes.

ModelCD (mm) ↓Track (mm) ↓IoU (%) ↑PSNR ↑SSIM ↑LPIPS ↓
w/o Adaptive Graph (capacity-matched)6.9608.33586.66128.7020.948250.02586
w/o Relation Retention7.0588.28786.50829.4250.950200.02612
w/o Relation-History Conditioning6.8878.00186.88829.4360.951610.02584
AIM6.1527.29087.04729.4730.952480.02509

Future Prediction: Scene-averaged results across all 22 PhysTwin scenes.

ModelCD (mm) ↓Track (mm) ↓IoU (%) ↑PSNR ↑SSIM ↑LPIPS ↓
w/o Adaptive Graph (capacity-matched)16.02621.45668.31125.8580.942890.05617
w/o Relation Retention15.19921.47768.84825.0280.933940.05585
w/o Relation-History Conditioning15.53021.76168.99025.7640.943310.05664
AIM14.23120.18769.75025.9160.943430.05507

Cross-action transfer.

ModelCD (mm) ↓Track (mm) ↓IoU (%) ↑PSNR ↑SSIM ↑LPIPS ↓
w/o Adaptive Graph (capacity-matched)23.69852.25470.39824.1110.903690.07301
w/o Relation Retention20.90946.01070.02724.1570.903000.07317
w/o Relation-History Conditioning21.29048.73969.29024.2020.901040.07653
AIM19.84845.06470.59124.4300.904130.07270

Cross-object transfer.

ModelCD (mm) ↓Track (mm) ↓IoU (%) ↑PSNR ↑SSIM ↑LPIPS ↓
w/o Adaptive Graph (capacity-matched)19.09524.31378.68023.8550.933750.07355
w/o Relation Retention18.61626.61279.30024.2950.931030.07440
w/o Relation-History Conditioning18.70725.32179.03823.9550.939750.07344
AIM17.77522.81279.78624.5060.941700.07294

AIM achieves the best averages across all six metrics in both scene-specific and transfer evaluations. The targeted removals support the complementary roles of retention in keeping useful dependencies available as spatial neighborhoods change and motion history in reassessing their relevance. These improvements extend across particle trajectories, geometric reconstruction, and rendering quality.

When adaptive connectivity matters. We additionally provide an E1 comparison on double_stretch_zebra.

ModelTrack ↓CD ↓IoU ↑PSNR ↑SSIM ↑LPIPS ↓
w/o Adaptive Graph (capacity-matched)8.3127.23580.94027.3750.957550.02411
AIM6.5556.59484.58027.8410.959550.02051

AIM improves all six metrics over the capacity-matched spatial control, supporting the benefit of adaptive relations beyond model capacity and learned edge processing. This comparison is motivated by situations where spatially close regions respond differently to manipulation: radius-based connectivity may introduce inappropriate interactions, while AIM uses motion context to retain and reassess deformation dependencies. We present the corresponding visual analysis in Section 6.5 with Figure 7 illustrating local deformation differences.

W2&Q1. Clarity of the transfer protocol. The training and held-out scene composition for Table 3 is not specified, and Appendix D.2 instead describes the single-scene transfer pairs in Table 2. How are checkpoints selected, and which target observations are used for coordinate and temporal alignment?

We sincerely thank the reviewer for identifying these omissions and apologize that the original presentation did not sufficiently clarify the transfer protocols. We have revised all three parts accordingly.

  • Clarify the scene composition for Table 3. We have added Appendix C.4, “Category-Level Generalization Protocols,” to specify the training and held-out sequences for Table 3, with a corresponding reference in Section 6.3. This distinguishes category-level generalization from the single-scene transfer experiments in Table 2. Each category-level setting trains a separate shared model. The held-out cloth setting uses seven training sequences (cloth_shirt_lift, cloth_skirt_2_fold, cloth_red_lift, cloth_red_fold, cloth_blue_fold, cloth_blue_lift, and cloth_skirt_1_fold) and two test sequences (cloth_shirt_fold and cloth_pant_lift). The lift-to-fold setting trains on the five cloth_{blue,pant,red,shirt,skirt_1}_lift sequences and tests on their five corresponding _fold sequences.

  • Specify the observations used for alignment. We have added the alignment and initialization details to Appendix C.3, “Evaluation Protocol,” under “Coordinate and temporal alignment” and “Cross-dataset rollout initialization.” Coordinate conversion uses fixed axis and scale mappings, with a translation determined solely from target initialization observations and held constant throughout rollout. Source-training normalization statistics are retained, and predictions are converted back to PhysTwin world coordinates for evaluation. Temporal alignment samples the 30 FPS target sequences every three frames to preserve the source model’s 0.1 s time step. For prediction beginning at frame bb, the model observes frames b−9b-9, b−6b-6, and b−3b-3, then predicts b,b+3,…b,b+3,\ldots autoregressively under the provided controls. No future target object states are used to determine alignment, and evaluation involves neither subsequent ground-truth state resets nor target-specific parameter updates.

  • Make checkpoint selection explicit. We have clarified checkpoint selection and reuse in Appendix C.3, under “Checkpoint selection” and “Source-model reuse for transfer,” with a summary in Section 5.3. PGND, PhysTwin, and category-level experiments select the saved checkpoint with the lowest training loss under each model’s respective objective. This rule applies to AIM, reproduced baselines, and all ablation variants, with one checkpoint supplying every reported metric for a given run. Cross-object, cross-action, and cross-dataset transfer reuse the corresponding in-domain source checkpoint without reselection based on target performance, keeping source-model selection separate from target evaluation.

Q3. Does AIM retain its advantage when variation across training seeds is reported?

We thank the reviewer for highlighting the importance of initialization sensitivity. We additionally evaluated AIM and the three targeted controls with multiple random seeds on rope_double_hand (1D), double_lift_cloth_3 (2D), and double_stretch_zebra (3D) for reconstruction and resimulation (E1). We also evaluated cross-action transfer from double_lift_cloth_3 to single_clift_cloth_3 across training seeds, reusing each run’s source checkpoint without target-based reselection. The following tables report mean ± standard deviation.

These results have been added to Appendix D.4, under “Variation across Training Seeds,” with source-scene results in Tables 13–15 and cross-action transfer results in Table 16.

Three-seed Reconstruction & Resimulation results: rope_double_hand (1D).

ModelCD (mm) ↓Track (mm) ↓IoU (%) ↑PSNR ↑SSIM ↑LPIPS ↓
w/o Adaptive Graph (capacity-matched)5.631 ± 0.99210.729 ± 1.85873.933 ± 2.27628.382 ± 0.6620.97433 ± 0.001670.02269 ± 0.00474
w/o Relation Retention5.674 ± 0.16710.780 ± 0.13774.015 ± 1.98128.659 ± 0.6820.96806 ± 0.000540.02286 ± 0.00080
w/o Relation-History Conditioning5.422 ± 0.04210.625 ± 0.39373.873 ± 0.56628.573 ± 0.4300.97119 ± 0.001160.02226 ± 0.00179
AIM5.104 ± 0.03610.396 ± 1.83074.058 ± 0.65328.687 ± 0.1040.97583 ± 0.000440.02197 ± 0.00140

Three-seed Reconstruction & Resimulation results: double_lift_cloth_3 (2D).

ModelCD (mm) ↓Track (mm) ↓IoU (%) ↑PSNR ↑SSIM ↑LPIPS ↓
w/o Adaptive Graph (capacity-matched)8.597 ± 0.0446.719 ± 0.17896.431 ± 0.41829.048 ± 0.6090.88577 ± 0.008050.03832 ± 0.00273
w/o Relation Retention8.772 ± 0.1236.350 ± 0.20596.253 ± 0.26028.804 ± 0.1420.88401 ± 0.001150.03988 ± 0.00181
w/o Relation-History Conditioning8.787 ± 0.2626.187 ± 0.04296.371 ± 0.10429.092 ± 0.0640.88617 ± 0.001770.03850 ± 0.00083
AIM8.566 ± 0.2106.066 ± 0.07196.494 ± 0.02729.115 ± 0.1100.88713 ± 0.002460.03821 ± 0.00007

Three-seed Reconstruction & Resimulation results: double_stretch_zebra (3D).

ModelCD (mm) ↓Track (mm) ↓IoU (%) ↑PSNR ↑SSIM ↑LPIPS ↓
w/o Adaptive Graph (capacity-matched)7.638 ± 0.5719.952 ± 2.32077.938 ± 4.24626.871 ± 0.7140.95507 ± 0.003510.02647 ± 0.00333
w/o Relation Retention7.565 ± 0.7649.534 ± 1.72780.203 ± 2.23827.282 ± 0.4430.95704 ± 0.000700.02547 ± 0.00470
w/o Relation-History Conditioning7.412 ± 0.1848.564 ± 0.04480.542 ± 1.01327.349 ± 0.2270.95607 ± 0.001320.02517 ± 0.00052
AIM6.610 ± 0.0226.492 ± 0.33682.968 ± 2.13827.786 ± 0.1420.95775 ± 0.002550.02219 ± 0.00237

Three-seed cross-action transfer: double_lift_cloth_3 → single_clift_cloth_3.

ModelCD (mm) ↓Track (mm) ↓IoU (%) ↑PSNR ↑SSIM ↑LPIPS ↓
w/o Adaptive Graph (capacity-matched)18.829 ± 0.26424.783 ± 0.21692.611 ± 0.27926.962 ± 0.0750.87587 ± 0.013230.06061 ± 0.00083
w/o Relation Retention10.626 ± 0.68021.988 ± 2.73994.483 ± 0.08028.113 ± 0.0150.87402 ± 0.000130.04673 ± 0.00084
w/o Relation-History Conditioning10.550 ± 0.09723.050 ± 0.31994.312 ± 0.30728.018 ± 0.3410.87432 ± 0.003880.04793 ± 0.00207
AIM10.283 ± 0.90321.057 ± 1.44695.097 ± 0.29728.257 ± 0.4040.88367 ± 0.003360.04650 ± 0.00019

AIM consistently achieves the best mean performance across all six metrics in the multi-seed E1 and cross-action comparisons, demonstrating improvements in particle motion, geometric accuracy, and rendering quality. These repeated-run results support the complementary contributions of relation retention and history-conditioned scoring, showing that the observed advantages extend beyond a single initialization.

AuthorsOpenReview ↗

We sincerely seek your feedback!

Dear Reviewer igdh,

We hope this message finds you well. We understand that the review period is busy and sincerely appreciate the time you have already devoted to our submission. As the discussion period is nearing its end, we wanted to ensure we have addressed your concerns satisfactorily.

We would be grateful for any final feedback and remain happy to address any further questions.

Thank you again for your consideration!

Best regards, The Authors

Reviewer_6HbbOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AIM, an Adaptive Interaction Modeling framework for predicting the dynamics of deformable objects such as ropes and cloth. Traditional particle-based models often connect particles based on spatial proximity, which can misrepresent deformation dependencies: stretching separates regions that remain mechanically coupled, while folding brings structurally distant regions close together. AIM addresses this limitation by using motion history, local geometry, and control context to infer connection strengths and relation embeddings, while retaining previously informative connections. It combines relation-weighted local message passing with a global latent token that aggregates object and control information and redistributes it according to particle geometry. The resulting model predicts future particle positions under prescribed control trajectories and is trained using multi-step autoregressive supervision. Experiments demonstrate improved prediction accuracy and transfer across objects and manipulation settings.

Critical strengths

Consistent predictive gains across two real-world benchmarks, which is also visualized in supplementary materials.

Critical weaknesses

None

Questions for authors

None

1 comment

We sincerely thank you for the time and effort you devoted to reviewing our submission. We appreciate your positive assessment of our work, recognition of our contributions, and encouraging feedback. We would be happy to answer any additional questions or address any further concerns you may have.

Reviewer_xp71OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes AIM, an adaptive graph-based neural simulator for predicting the dynamics of deformable objects under external interactions. The method combines learned particle connections, global communication, and autoregressive training to improve motion prediction. Experiments on real-world deformable-object datasets show promising results, outperforming several existing baselines in prediction accuracy and generalization across objects and actions.

Critical strengths

The paper applies adaptive graph modeling to deformable-object dynamics prediction and achieves strong results, improving prediction accuracy and generalization over existing baselines.

Critical weaknesses

Adaptive graph modeling has already been explored in previous work, and GNN-based deformable simulation is well established. While applying these ideas to this setting is useful, the proposed method appears to be an incremental improvement rather than a substantial methodological contribution.

Questions for authors
  1. Figure 6 shows only a small improvement in tracking error from adaptive graph modeling. Could the authors provide more comprehensive ablations across different objects, actions, and multiple seeds to demonstrate its consistent benefits?

  2. Could the authors compare the proposed adaptive graph against simpler alternatives, such as fixed spatial graphs, or standard learned edge weights, under matched model capacity and training settings? This would help clarify whether the added complexity is necessary.

4 comments

W1. The proposed method appears to be an incremental improvement.

We appreciate this concern. However, we respectfully disagree that AIM is merely an incremental combination. Building on adaptive graph learning and graph-based simulation does not by itself anticipate our contribution: maintaining and revising deformation dependencies as spatial neighborhoods change, while translating local manipulation into coordinated, geometry-dependent responses.

This problem arises naturally in deformable manipulation. Folding can bring regions with different motion responses close together, while stretching can separate regions that remain coupled. Distance-based connectivity can consequently introduce inappropriate interactions or lose useful dependencies, producing local errors that accumulate during autoregressive rollout. Changes in object geometry and control contacts further alter how these dependencies transmit motion. AIM addresses these connected challenges by retaining relations beyond current spatial neighborhoods and rescoring them using motion history, learning their persistence and relevance from trajectories without material-connectivity supervision. Geometry-conditioned global updates then convey object-wide information according to each particle’s position relative to the object and its controls. The resulting formulation connects which dependencies remain relevant over time with how manipulation effects propagate across the object, supporting coherent deformation under evolving interaction conditions.

Q1. Could the authors provide more comprehensive ablations across different objects and actions to demonstrate its consistent benefits?

We thank the reviewer for this suggestion. We have therefore expanded AIM and all five original ablations to all 22 PhysTwin scenes, covering diverse object geometries and manipulation actions. The following tables report equally weighted scene averages for reconstruction and resimulation (E1) and future prediction (E2).

These results have been incorporated into Section 6.4 and Figure 6 of the revised manuscript, with complete numerical results in Appendix D.4.1, Table 10.

Reconstruction & Resimulation: Scene-averaged results across all 22 PhysTwin scenes.

ModelCD (mm) ↓Track (mm) ↓IoU (%) ↑PSNR ↑SSIM ↑LPIPS ↓
w/o Adaptive Graph & Global Token13.20818.14976.46027.1990.934220.04305
w/o Global Token9.84913.21481.93628.4910.939540.03237
w/o Adaptive Graph7.0738.32886.32829.2510.941210.02670
w/o Geometry Conditioning6.8088.05986.39328.8380.942350.02600
w/o Learned Weight Assignment6.8737.92686.77429.1990.943960.02585
AIM6.1527.29087.04729.4730.952480.02509

Future Prediction: Scene-averaged results across all 22 PhysTwin scenes.

ModelCD (mm) ↓Track (mm) ↓IoU (%) ↑PSNR ↑SSIM ↑LPIPS ↓
w/o Adaptive Graph & Global Token27.31149.51151.90722.8980.921220.09505
w/o Global Token20.07929.77263.57725.1440.939800.06417
w/o Adaptive Graph14.87220.72468.93325.3790.942530.05567
w/o Geometry Conditioning14.72821.59468.31525.4800.942910.05873
w/o Learned Weight Assignment15.02721.20669.28525.6130.942230.05562
AIM14.23120.18769.75025.9160.943430.05507

AIM achieves the best scene-averaged results across all six metrics in both E1 and E2. We place particular emphasis on CD and Track because they directly assess predicted 3D geometry and the motion of corresponding points, respectively. Image-space metrics such as SSIM, PSNR, and LPIPS provide complementary evidence of rendered fidelity, but also depend on appearance and rendering; high image similarity alone does not establish accurate particle motion or local deformation. We therefore interpret these metrics jointly while prioritizing geometric and tracking accuracy when assessing dynamics. The expanded comparisons support the complementary roles of adaptive relations, learned interaction weights, and geometry-conditioned global communication beyond the original single-scene example.

Q2. Could the authors provide more comprehensive ablations across multiple seeds to demonstrate its consistent benefits?

We thank the reviewer for highlighting the importance of variation across training seeds. We additionally evaluated AIM and all five original ablations with multiple random seeds on rope_double_hand (1D), double_lift_cloth_3 (2D), and double_stretch_zebra (3D), covering distinct object structures and deformation patterns. The following tables report E1 mean ± standard deviation across independent training runs.

These results have been incorporated into Appendix D.4, under “Variation across Training Seeds,” with Tables 13–15 reporting the rope, cloth, and zebra comparisons, respectively.

Three-seed Reconstruction & Resimulation results: rope_double_hand (1D).

ModelCD (mm) ↓Track (mm) ↓IoU (%) ↑PSNR ↑SSIM ↑LPIPS ↓
w/o Adaptive Graph & Global Token28.455 ± 5.95248.834 ± 0.07231.100 ± 1.98222.256 ± 0.0460.95266 ± 0.003590.07292 ± 0.00557
w/o Global Token17.982 ± 4.20338.757 ± 4.32339.382 ± 1.40523.589 ± 0.1480.95597 ± 0.002100.07072 ± 0.00089
w/o Adaptive Graph5.651 ± 0.07810.678 ± 0.01873.600 ± 0.32028.375 ± 0.0320.97495 ± 0.000100.02207 ± 0.00008
w/o Geometry Conditioning5.376 ± 0.28610.710 ± 1.11973.486 ± 0.54728.659 ± 0.4110.97412 ± 0.000770.02230 ± 0.00096
w/o Learned Weight Assignment5.545 ± 0.10510.447 ± 0.68872.564 ± 3.85528.343 ± 0.7970.97522 ± 0.000990.02323 ± 0.00139
AIM5.104 ± 0.03610.396 ± 1.83074.058 ± 0.65328.687 ± 0.1040.97583 ± 0.000440.02197 ± 0.00140

Three-seed Reconstruction & Resimulation results: double_lift_cloth_3 (2D).

ModelCD (mm) ↓Track (mm) ↓IoU (%) ↑PSNR ↑SSIM ↑LPIPS ↓
w/o Adaptive Graph & Global Token9.246 ± 0.0466.427 ± 0.10196.129 ± 0.42929.038 ± 0.3040.88458 ± 0.000030.04007 ± 0.00157
w/o Global Token8.818 ± 0.0076.124 ± 0.03596.288 ± 0.00828.984 ± 0.0080.88514 ± 0.000150.03877 ± 0.00013
w/o Adaptive Graph8.771 ± 0.3586.152 ± 0.08196.121 ± 0.43628.880 ± 0.3480.88459 ± 0.000420.04069 ± 0.00214
w/o Geometry Conditioning8.588 ± 0.0056.356 ± 0.04796.393 ± 0.03229.064 ± 0.0850.88662 ± 0.000880.03853 ± 0.00021
w/o Learned Weight Assignment8.728 ± 0.0326.150 ± 0.06196.193 ± 0.11929.093 ± 0.1510.88500 ± 0.001940.03871 ± 0.00086
AIM8.566 ± 0.2106.066 ± 0.07196.494 ± 0.02729.115 ± 0.1100.88713 ± 0.002460.03821 ± 0.00007

Three-seed Reconstruction & Resimulation results: double_stretch_zebra (3D).

ModelCD (mm) ↓Track (mm) ↓IoU (%) ↑PSNR ↑SSIM ↑LPIPS ↓
w/o Adaptive Graph & Global Token18.561 ± 0.22534.572 ± 0.51761.836 ± 1.26625.319 ± 0.4290.95202 ± 0.003840.05428 ± 0.00172
w/o Global Token19.194 ± 0.41435.276 ± 2.92960.813 ± 0.16025.004 ± 0.1800.95041 ± 0.001000.05592 ± 0.00055
w/o Adaptive Graph7.989 ± 0.14610.219 ± 2.54078.495 ± 0.80426.888 ± 0.1150.95530 ± 0.001360.02624 ± 0.00051
w/o Geometry Conditioning7.453 ± 0.3429.011 ± 0.66071.602 ± 2.23526.825 ± 0.4240.95015 ± 0.002290.03224 ± 0.00314
w/o Learned Weight Assignment7.533 ± 0.5489.286 ± 0.73680.924 ± 2.64727.469 ± 0.4900.95429 ± 0.002750.02449 ± 0.00104
AIM6.610 ± 0.0226.492 ± 0.33682.968 ± 2.13827.786 ± 0.1420.95775 ± 0.002550.02219 ± 0.00237

AIM consistently achieves the best mean performance across all six metrics in the scene-averaged, transfer, and multi-seed E1 comparisons, demonstrating improvements in particle trajectories, geometric reconstruction, and rendering quality. The repeated runs across 1D, 2D, and 3D objects further show reproducible geometric accuracy, supporting the complementary contributions of adaptive relations, learned interaction weights, and geometry-conditioned global communication beyond a single scene or initialization.

Q3. Could the authors compare the proposed adaptive graph against simpler alternatives, such as fixed spatial graphs, or standard learned edge weights, under matched model capacity and training settings?

We thank the reviewer for highlighting this gap. Following the reviewer’s suggested alternatives, we added three controls, all retaining global communication and geometry conditioning:

  • w/o Adaptive Graph (capacity-matched): uses current spatial-radius connectivity with learned edge features and an active residual edge encoder, matching AIM’s capacity at 3,171,048 versus 3,170,980 parameters.
  • w/o Graph Updates (fixed initial): fixes object–object connectivity at the beginning of each training window or rollout, while updating edge features and manipulator interactions.
  • w/ Standard Edge Weighting: uses a symmetric pairwise MLP to predict edge features and weights over current spatial candidates, retaining motion-history inputs but excluding retained connections.

These controls examine whether comparable capacity, preserved initial neighborhoods, or conventional feature-dependent weighting can recover AIM’s performance. The distinction matters because a useful deformation dependency can outlast spatial proximity, while a retained connection must still be reassessed as motion changes. We evaluate all three alternatives across 22 PhysTwin scenes and the paper’s cross-action and cross-object transfer pairs. All models, including AIM, use matched training settings: 5-step autoregressive supervision for 20,000 optimizer updates, with the same training data, temporal sampling, optimizer, learning-rate schedule, batch size, and checkpoint-selection procedure.

These controls and their results have been added to Appendix D.4, under “Additional Controls for Relation Modeling,” with reconstruction and resimulation and future prediction results in Table 11, and cross-action and cross-object transfer results in Table 12.

Reconstruction & Resimulation: Scene-averaged results across all 22 PhysTwin scenes.

ModelCD (mm) ↓Track (mm) ↓IoU (%) ↑PSNR ↑SSIM ↑LPIPS ↓
w/o Adaptive Graph (capacity-matched)6.9608.33586.66128.7020.948250.02586
w/o Graph Updates (fixed initial)7.2148.68986.24929.2010.947170.02650
w/ Standard Edge Weighting7.0148.20186.74829.3090.951680.02661
AIM6.1527.29087.04729.4730.952480.02509

Future Prediction: Scene-averaged results across all 22 PhysTwin scenes.

ModelCD (mm) ↓Track (mm) ↓IoU (%) ↑PSNR ↑SSIM ↑LPIPS ↓
w/o Adaptive Graph (capacity-matched)16.02621.45668.31125.8580.942890.05617
w/o Graph Updates (fixed initial)14.40820.68769.22325.8860.935870.05640
w/ Standard Edge Weighting14.36220.38669.35525.1870.936930.05616
AIM14.23120.18769.75025.9160.943430.05507

Cross-action transfer.

ModelCD (mm) ↓Track (mm) ↓IoU (%) ↑PSNR ↑SSIM ↑LPIPS ↓
w/o Adaptive Graph (capacity-matched)23.69852.25470.39824.1110.903690.07301
w/o Graph Updates (fixed initial)25.75650.15469.19523.8340.902350.07816
w/ Standard Edge Weighting21.81645.90469.67324.2010.901170.07510
AIM19.84845.06470.59124.4300.904130.07270

Cross-object transfer.

ModelCD (mm) ↓Track (mm) ↓IoU (%) ↑PSNR ↑SSIM ↑LPIPS ↓
w/o Adaptive Graph (capacity-matched)19.09524.31378.68023.8550.933750.07355
w/o Graph Updates (fixed initial)19.19826.84279.34623.9150.937990.07345
w/ Standard Edge Weighting18.85726.51579.72924.2320.939440.07600
AIM17.77522.81279.78624.5060.941700.07294

AIM achieves the best averages across all six metrics in both E1/E2 and both transfer settings, demonstrating improvements in particle trajectories, geometric reconstruction, and rendering quality under the reported configurations. These results are consistent with the intended role of retaining useful dependencies while revising their relevance, rather than relying exclusively on fixed connectivity or weighting current spatial neighbors.

AuthorsOpenReview ↗

We sincerely seek your feedback!

Dear Reviewer xp71,

We hope this message finds you well. We understand that the review period is busy and sincerely appreciate the time you have already devoted to our submission. As the discussion period is nearing its end, we wanted to ensure we have addressed your concerns satisfactorily.

We would be grateful for any final feedback and remain happy to address any further questions.

Thank you again for your consideration!

Best regards, The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

We sincerely thank you for the careful assessments and constructive feedback. We are encouraged that reviewers recognized (i) the strong predictive performance of AIM across real-world deformable-object benchmarks (Reviewers xp71, 6Hbb, and igdh), (ii) improved generalization across objects, actions, and manipulation domains, including frozen-model robot-to-human transfer (Reviewers xp71 and igdh), and (iii) evidence of prediction beyond the training horizon and qualitative support for the reported improvements (Reviewers igdh and 6Hbb, respectively).

The main concerns center on clarifying the contribution of adaptive interaction modeling and broadening the ablation evidence across objects, interactions, and training seeds. We summarize our responses to these themes below; detailed results and reviewer-specific clarifications are provided in our point-by-point responses.

1. Isolating the contribution of adaptive interaction modeling.

Following the suggestions of Reviewers xp71 and igdh, we added a capacity-matched spatial baseline that retains global communication and learned edge features, fixed-initial connectivity and standard learned edge-weighting alternatives, and separate removals of relation retention and history conditioning. These comparisons support the value of preserving useful deformation dependencies as spatial neighborhoods change and reassessing their relevance using motion context, complemented by a qualitative case study illustrating local deformation differences. Detailed results and analyses are provided in our point-by-point responses.

2. Extending the ablation studies across objects, interactions, and training seeds.

To address the requests from Reviewers xp71 and igdh, we expanded the original component ablations to all 22 PhysTwin scenes, covering reconstruction and resimulation as well as future prediction, and evaluated the additional relation-modeling controls under cross-action and cross-object transfer. We further report means and standard deviations from repeated training runs on rope, cloth, and zebra, including a cloth cross-action evaluation. The results support the complementary contributions of adaptive relations and geometry-conditioned global communication, with AIM retaining the best mean performance across the evaluated settings. These findings extend the evidence for our design beyond a single scene or initialization. Detailed results are provided in our point-by-point responses.

We have incorporated additional experimental results and accompanying analyses into the revised manuscript and appendix. The corresponding additions and revisions are highlighted in blue for ease of reference.

We thank the reviewers again for helping us strengthen the manuscript. We believe that the revised version, incorporating the clarifications and additional analyses described below, presents a technically solid and empirically validated contribution.

We look forward to an active and productive discussion!

Best regards,
The Authors

Dear Reviewers,

Thank you again for your reviews of this submission. The authors have now posted their responses, and the author–reviewer discussion period ends on November 18, 2026 (AoE).

Please read the authors' rebuttal and let them know whether it addresses your concerns. A short reply is enough, for example:

  • which of your concerns are now resolved;
  • which remain open, and what evidence or clarification would resolve them;
  • whether you plan to update your rating.

If the rebuttal changes your assessment, please update your review and rating before the discussion period ends. If you have already engaged, thank you, and no further action is needed unless the authors reply again.

Your participation is important for a fair and well-informed decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.