acceptodds
Under review as a conference paper at ICLR 2027

RiCo: Neural Simulation of Rigid-Body Interactions via Local Contact Reasoning

14 views

Abstract

Accurate simulation of rigid-body interactions is essential for predictive physical world models. Despite recent progress in modeling object dynamics, capturing how local contacts between surfaces shape object motion remains challenging. While end-to-end world models predict interactions across entire scenes or objects, in practice, rigid-body contact is inherently local, and only nearby surfaces can directly exchange contact forces. Motivated by this observation, we introduce Rigid-body Contact Reasoning (RiCo), which represents interactions between objects through sparse neighborhoods of contact surface points. RiCo combines each point’s state with the relative geometry, motion, and physical properties of nearby surfaces, then reasons across the object’s points to determine how these local contacts jointly affect its motion. By confining cross-object reasoning to nearby surfaces while propagating contact information within each rigid body, RiCo retains fine-grained interaction details without the cost of modeling every pair of scene points. Such properties enable RiCo a higher accuracy and contact fidelity. Experiments on MOVi-benchmark demonstrate that RiCo reduces 100-frame position and orientation errors by 31–35% and approximately 38%, respectively, compared with baselines. Moreover, RiCo achieves high contact fidelity, with ground-truth-relative penetration-time and mean-depth differences of 11.0% and 2.22 mm, respectively. RiCo further generalizes zero-shot from small-scale training scenarios to scenes containing 270 objects. Our real-world multi-ball collision experiments further provide preliminary evidence of sim-to-real transfer

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Neural Simulation, Learned Dynamics, Point Cloud Dynamics, Mesh-Free Simulation, Physical Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RoXvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RiCo, a mesh-free neural simulator for multi-object rigid-body dynamics. Motivated by the observation that rigid-body contact occurs only locally between adjacent surfaces, the authors model inter-object interactions through sparse, point-level local contacts rather than mesh-based or object-level attention-based formulations. Specifically, for each surface point, the model retrieves contact candidates via K-nearest-neighbor (KNN) search and constructs interaction descriptors that include an estimated signed distance. A point Transformer then aggregates this contact information within each object to infer how local contacts affect the object’s overall motion. Finally, the rigid transformation is recovered with the Kabsch algorithm from motion predicted at a small set of anchor points. On the MOVi benchmark, RiCo substantially reduces position and orientation errors, as well as interpenetration, compared with existing methods. The authors further demonstrate scalability with respect to point resolution, zero-shot generalization to scenes with 270 objects, and preliminary sim-to-real transfer in a real-world billiards setting.

Critical strengths

S1. Significant performance improvement

Using only irregular surface points without mesh connectivity, RiCo reduces position errors by 31–35% and orientation errors by approximately 38% on MOVi-A/B. Notably, it achieves substantially lower penetration depth than both mesh-based and point-based baselines, showing that precise contact modeling is achievable with a purely point-based representation.

S2. Resolution scalability

Contact fidelity improves consistently as the number of points increases (Fig. 4), and the runtime at each resolution is reported (Fig. 5). This gives practitioners the flexibility to choose a resolution that fits their computational budget.

Critical weaknesses

W1. Insufficient analysis of contact fidelity

RiCo’s Δ\DeltaMPD is more than 100× smaller than those of the baselines. Even without the signed distance feature, a ~30$\times$ gap remains, yet the source of this gap is not analyzed. Because Δ\DeltaMPD is a mean, it may be dominated by catastrophic failures in a few sequences; however, the supplementary video does not suggest that the baselines fail this severely. In addition, the values reported for RigidFormer under seemingly identical settings differ between Fig. 3a (267.70 mm) and Fig. 4a (402 mm).

W2. Limited novelty of the main idea

The key change relative to RigidFormer is replacing object-level attention with point-level KNN contact. However, decoupling local contact from intra-object propagation has already been adopted in FIGNet and SDF-Sim. The paper also lacks analysis of the specific benefits this change brings across different contact scenarios.

Questions for authors

Q1. Contact fidelity analysis

Could you provide the per-sequence distribution of the baselines’ Δ\DeltaMPD (e.g., median), statistics separated into object–object and object–ground penetration, and the absolute ground-truth PTR/MPD values? Also, why do the RigidFormer values differ between Fig. 3a and Fig. 4a?

Q2. Novelty of the main idea

What are the structural differences between RiCo and FIGNet or SDF-Sim, and why is SDF-Sim not included as a baseline? Is there an ablation that isolates the contribution of point-level KNN contact?

Q3. Training strategy

Since the model is trained only with teacher forcing, how is stability ensured in long-horizon rollouts? Moreover, does applying robustness-oriented training techniques (e.g., autoregressive fine-tuning) lead to a significant drop in performance?

4 comments
AuthorsOpenReview ↗

Author response to Reviewer cQV5 - Part 1

Dear Reviewer, thank you for your constructive feedback. We address your questions below.

  1. "Question: Could you provide the per-sequence distribution of the baselines’ Δ\Delta MPD (e.g., median), statistics separated into object–object and object–ground penetration, and the absolute ground-truth PTR/MPD values? Also, why do the RigidFormer values differ between Fig. 3a and Fig. 4a?"

Our response. We report penetration statistics and per-sequence MPD distributions, separating object–object and object–ground interactions.

Table 1. Penetration statistics

MethodOverall PTR (%)Overall MPD (mm)Obj–obj PTR (%)Obj–obj MPD (mm)Obj–ground PTR (%)Obj–ground MPD (mm)
GT (120)10.703.910.9310.919.933.22
GT (RF subset, 116)10.723.910.9510.709.933.24
HOPNet (120)53.39289.7711.99122.5748.92305.04
RigidFormer (116)50.2750.7615.01151.6043.117.85
RiCo (120)21.706.133.1515.7119.394.52

Table 2. Per-sequence MPD distributions (mm)

MethodOverall MedianOverall Q1–Q3Overall P90Obj–obj MedianObj–ground Median
HOPNet261.28200.43–359.51497.5557.35272.55
RigidFormer20.848.80–57.31105.9490.707.76
RiCo4.673.73–6.598.578.563.96

RiCo outperforms both baselines across penetration metrics. Relative to RigidFormer, overall PTR drops from 50.27% to 21.70% and MPD from 50.76 to 6.13 mm (87.9% reduction), with gains in both contact types. Per-sequence median/P90 MPD is 4.67/8.57 mm versus 20.84/105.94 mm, indicating fewer severe-penetration trajectories. GT itself has nonzero penetration (10.70% PTR; 3.91 mm MPD).

Dataset-level MPD aggregates penetrated object-frame observations; per-sequence statistics treat each trajectory separately and exclude those without penetration. RigidFormer has 116/120 valid predictions, so its GT reference uses the same subset.

For scale, cubes have 1 m sides and spheres a 0.5 m radius. Per-trajectory PTR/MPD statistics are in our supplementary materials.

RigidFormer's results differ because Fig. 3a uses the official vertex-based implementation, while Fig. 4a uses our point-cloud reimplementation with RiCo-matched point-cloud observations but RigidFormer's original features. We will clarify this in the revision.

  1. "Question: Since the model is trained only with teacher forcing, how is stability ensured in long-horizon rollouts? Moreover, does applying robustness-oriented training techniques (e.g., autoregressive fine-tuning) lead to a significant drop in performance?"

Our response. Although RiCo is trained with teacher forcing, Kabsch projection enhances long-horizon stability by fitting rigid transforms to predicted anchors at each rollout step, preserving intra-object geometry. HOPNet (Wei & Fink, 2025) likewise uses shape matching to maintain rigidity.

We also performed 10 epochs of autoregressive fine-tuning on MOVi-A. Starting from two GT frames, predictions are recursively fed back, with intermediate steps detached from gradient tracking and the final step supervised by the original losses. Varying rollout depth exposes the model to accumulated errors.

Table. MOVi-A autoregressive validation performance during fine-tuning.

EpochFramePosition RMSE (m) ↓Rotation RMSE (°) ↓
0500.03102.86
0750.06706.49
01000.115011.40
5500.03082.87
5750.06586.49
51000.114511.58
10500.03102.85
10750.06546.31
101000.115211.33

Across epochs 0–10, position and rotation RMSE show no substantial degradation at 50, 75, or 100 frames. At 75 frames, position RMSE improves from 0.0670 to 0.0654 m and rotation RMSE from 6.49° to 6.31°. Thus, fine-tuning maintains stability without clear overall accuracy gains.

AuthorsOpenReview ↗

Author response to Reviewer cQV5 - Part 2

  1. "Weakness: RiCo’s Δ\Delta MPD is more than 100$\times$ smaller than those of the baselines. Even without the signed distance feature, a ~30 ×\times gap remains, yet the source of this gap is not analyzed. ... In addition, the values reported for RigidFormer under seemingly identical settings differ between Fig. 3a (267.70 mm) and Fig. 4a (402 mm)." "Weakness: The key change relative to RigidFormer is replacing object-level attention with point-level KNN contact. However, decoupling local contact from intra-object propagation has already been adopted in FIGNet and SDF-Sim. The paper also lacks analysis of the specific benefits this change brings across different contact scenarios." "Question: What are the structural differences between RiCo and FIGNet or SDF-Sim, and why is SDF-Sim not included as a baseline? Is there an ablation that isolates the contribution of point-level KNN contact?"

Our response.

a. Architectural differences and SDF-Sim comparison.

FIGNet relies on mesh connectivity and face-face interaction hyperedges, while SDF-Sim uses pretrained signed-distance functions to construct surface-node-to-object collision edges. In contrast, RiCo operates directly on unstructured point clouds without requiring meshes or pretrained SDFs. It models sparse point-level contacts using relative geometry, motion, and physical properties, followed by intra-object Point Transformer reasoning.

SDF-Sim was initially omitted primarily due to the lack of results at multiple prediction horizons on MOVi-B. Furthermore, its official implementation is not publicly available to our knowledge, preventing reproduction under our evaluation protocol. For completeness, we provide a comparison using its published 50-step MOVi-B results.

Table 1. MOVi-B comparison at 50-frame rollouts.

MethodTranslation RMSE (m) ↓Rotation Error (°) ↓
FIGNet0.14015.00
SDF-Sim0.16018.03
RiCo (Ours)0.0322.59

RiCo demonstrates a substantial advantage in prediction accuracy over both FIGNet and SDF-Sim. Specifically, it achieves 77.1% and 80.0% lower translation errors, and 82.7% and 85.6% lower rotation errors, respectively. These results highlight the effectiveness of RiCo's sparse point-level contact reasoning in predicting rigid-body interactions without requiring explicit mesh connectivity or pretrained SDFs.

2. Point-level contact ablation.

We conducted two complementary component-swap ablations on MOVi-B:

  • RF Features + RiCo Transformer: Replaces RiCo's sparse KNN contact descriptors with RigidFormer-style point-wise features, while retaining RiCo's intra-object Point Transformer.
  • RiCo Contact + RF Transformer: Retains RiCo's KNN contact descriptors but replaces its intra-object Transformer with RigidFormer's scene-level Transformer.

Table 2. Autoregressive rollout performance on MOVi-B.

MethodFrameTranslation RMSE (m) ↓Rotation RMSE (°) ↓
RF Features + RiCo Transformer500.0554.70
750.12710.55
1000.21217.46
RiCo Contact + RF Transformer500.0333.34
750.0727.42
1000.12112.10
RiCo (Full)500.0322.59
750.0655.45
1000.1119.54

Replacing RiCo's local contact descriptors with RigidFormer features substantially degrades long-horizon prediction, increasing the 100-frame translation RMSE from 0.111 m to 0.212 m and rotation RMSE from 9.54° to 17.46°. In contrast, retaining RiCo's contact descriptors with RigidFormer's Transformer produces smaller errors (0.121 m / 12.10°), suggesting that fine-grained contact encoding is a major contributor to prediction accuracy. Full RiCo achieves the lowest errors across all three horizons, indicating an additional benefit from intra-object Point Transformer reasoning.

c. Penetration analysis of component ablations.

Table 3. Geometric penetration on MOVi-B.

MethodPTR (%) ↓MPD (mm) ↓
Ground Truth10.703.91
RF Features + RiCo Transformer36.3925.93
RiCo Contact + RF Transformer27.056.82
RiCo (Full)21.706.13

The penetration results further support the importance of local contact encoding. With RiCo's Transformer unchanged, replacing the KNN contact representation with RigidFormer features increases PTR from 21.70% to 36.39% and MPD from 6.13 mm to 25.93 mm. Retaining RiCo's contact representation while using RigidFormer's Transformer yields substantially lower penetration (27.05% PTR / 6.82 mm MPD). Full RiCo achieves the best results, suggesting that local contact representation is particularly important for reducing geometric penetration, while intra-object reasoning provides an additional improvement.

Thank you for the detailed response and the revisions. My concerns have been well addressed through the inclusion of updated contact-fidelity statistics, an ablation study involving component swaps, results from autoregressive fine-tuning, and a controlled comparison with the official RigidFormer code. I have confirmed that these additions have been incorporated into the revised manuscript. Accordingly, I am raising my evaluation score to 3.

Dear reviewer, thank you very much for your positive feedback and the valuable suggestions you provided during the rebuttal period. We have learned a lot from them based on your comments. Thank you!

Reviewer_ePFROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RiCo predicts rigid-body interactions using sparse local contact reasoning and anchor-based motion prediction. Although it reports improved trajectory and contact metrics, the rigid-body assumption makes the setting relatively narrow. It remains unclear what advantages the method offers over existing contact-capable simulators, whether it can extend to moderately deformable objects, and how its computational cost changes in dense or complex collision scenarios.

Critical strengths
  1. Well-motivated contact representation. RiCo models interactions between nearby surfaces and propagates their effects across each rigid body. This directly addresses the local nature of contact while avoiding dense interactions across all scene points.
  2. Strong results on the stated task. RiCo substantially reduces long-horizon position and orientation errors on MOVi-A and MOVi-B, and reports lower penetration measures than the compared neural simulators. These results support its effectiveness within the rigid-body setting, even though they do not resolve my concerns about broader applicability.
Critical weaknesses
  • Limited scope of the setting. The paper assumes rigid-body motion, which imposes a strong constraint and makes the problem setting relatively narrow. I am not sure how much this result tells us about more general physical interactions.

  • Advantage over existing simulators. Contact is already handled by established simulators, including MPM-based methods. The paper should explain what RiCo does better than these simulators and whether it can be used together with them.

  • Applicability to deformation. Can the proposed contact-reasoning approach be extended to objects that undergo moderate deformation? This would help clarify whether the method is specific to rigid bodies or has broader applicability.

  • Computational complexity. I would like to know which scenarios are most expensive for RiCo. For example, how does its runtime change when collisions are frequent or when objects have complex shapes?

Questions for authors
  1. Scope: The Kabsch step enforces rigid-body motion. Is RiCo intended only for passive rigid bodies, or is there evidence that its contact representation remains useful for moderately deformable objects? This affects how broadly I can interpret the contribution.
  2. Scaling: How do contact-search time and total inference time change when many objects are simultaneously in contact? The point-resolution benchmark does not show the cost of dense collision scenarios or complex shapes, which is important to the efficiency claim.
3 comments

Dear Reviewer, thank you for your thoughtful review. We address your concerns below.

  1. "Weakness: Limited scope of the setting ... which imposes a strong constraint and makes the problem setting relatively narrow..." "Applicability to deformation. Can the proposed contact-reasoning approach be extended to objects that undergo moderate deformation? ..." "Question: Scope: The Kabsch step enforces rigid-body motion. Is RiCo intended only for passive rigid bodies, or is there evidence that its contact representation remains useful for moderately deformable objects? "

Our response. RiCo targets passive rigid-body dynamics, with Kabsch enforcing rigidity. However, our local contact representation is not inherently limited to rigid bodies, so we tested moderate deformation.

We relax Kabsch via weighted interpolation between raw node predictions and rigid projections, retaining the original contact representation. With α\alpha = 0.99, we achieve an autoregressive node RMSE of 0.0549 m and a free-node RMSE of 0.0576 m. The rollout shows deformable motion against FEM ground truth.

We provide qualitative rollout video comparing our predicted deformable motion with FEM ground truth, along with additional experimental details, on our anonymous supplementary website.

While these preliminary results suggest that RiCo's contact representation can potentially extend beyond rigid-body dynamics, we acknowledge that comprehensive deformable-body modeling remains outside the scope of this work. Our main contribution focuses on rigid-body interactions, with this experiment providing initial evidence of broader applicability.

  1. " Weakness: Computational complexity. I would like to know which scenarios are most expensive for RiCo. For example, how does its runtime change when collisions are frequent or when objects have complex shapes?" "Question: Scaling: How do contact-search time and total inference time change when many objects are simultaneously in contact? "

Our response. We evaluated RiCo on three 270-object scenarios (Spheres, Spots, and Knots), despite training only on 3–10 objects. The measured throughput was 5.02, 6.50, and 6.54 FPS, respectively. We additionally profiled inference latency and contact interactions during a 270-object rollout.

Table 1. Inference efficiency and profiling results for a 270-object rollout.

MetricProfiled Rollout
Throughput (FPS) ↑5.58
Total latency (ms/step) ↓179.3
Contact search (ms)18.4
Network forward (ms)166.5
Rigid-state update (ms)1.18
Contact pairs (early)0
Contact pairs (middle)89.0
Contact pairs (late)353.9

Profiling components were measured separately and are therefore non-additive. Contact pairs represent the mean number of eligible contact-candidate object pairs at different rollout stages.

RiCo maintains a throughput of 5.02–6.54 FPS across all three 270-object scenarios, demonstrating its ability to perform inference on substantially larger scenes than those encountered during training. Profiling indicates that network computation (166.5 ms) dominates inference time compared with contact search (18.4 ms), even as eligible contact-candidate pairs increase from 0 to 353.9 during the rollout. This suggests that inference efficiency is primarily constrained by the increasing object-wise processing workload rather than contact search alone.

  1. "Weakness: Advantage over existing simulators. Contact is already handled by established simulators, including MPM-based methods."

Our response. RiCo offers several potential advantages as a neural simulator for contact dynamics. First, its differentiable dynamics could support model-based policy optimization, as illustrated by recent work such as SuperMimic, which uses differentiable world models for policy learning. Second, RiCo’s explicit local contact representations could be incorporated into downstream policies to provide contact-aware geometric information for manipulation and control. Third, RiCo can be trained on trajectories generated by conventional simulators, allowing it to complement existing physics-based simulation tools. These applications are promising directions for future work and are beyond the scope of the present study.

Having these benefits, we position RiCo as a complementary learning-based approach to modeling contact dynamics, rather than a replacement for established physics simulators.

Thank you for the response. My concerns are fully addressed, so I keep my score.

AuthorsOpenReview ↗

Thanks for reply

Thank you for your thoughtful assessment. We truly appreciate the time and effort you have invested during the review process, as well as your valuable suggestions.

Reviewer_1AAsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RiCo, a neural simulator for multi-object rigid-body dynamics represented by surface point clouds. RiCo is motivated by the observation that contact forces are exchanged locally between nearby surfaces, while their effects must be integrated over an entire rigid body. For each surface point, the method constructs a sparse neighborhood of nearby points from other objects or static surfaces and encodes their relative geometry, motion, surface normals, and physical properties. These contact-conditioned point features are then propagated within each object using a shared point Transformer. An anchor-based decoder predicts second-order displacements for a small set of persistent surface anchors, from which a single rigid transformation is recovered using Kabsch alignment and applied to the entire object. Experiments on MOVi-Sphere, MOVi-A, and MOVi-B report improved long-horizon position and orientation prediction over several mesh- and point-based baselines. The paper additionally evaluates cross-geometry generalization, penetration-based contact statistics, point-cloud resolution and runtime, zero-shot transfer from scenes with 3–10 objects to 270-object scenes, and preliminary sim-to-real transfer on multi-ball billiard collisions.

Critical strengths
  1. The separation between sparse cross-object contact encoding, object-wide integration of contact effects, and rigid projection is well motivated. It reflects the fact that contact exchange is spatially local while the response of a rigid body is global. The resulting architecture preserves rigid motion by construction and provides a reasonable mesh-free alternative to explicit face- or topology-based message passing. It offers a conceptually simple way to retain surface-level contact information without requiring mesh connectivity.

  2. RiCo reports substantial reductions in both position and orientation error on MOVi-A and MOVi-B across several rollout horizons. The resolution experiment also suggests that the model can use denser surface samples to improve penetration-related behavior while maintaining competitive runtime. If these improvements persist under controlled input and training conditions, they would constitute meaningful empirical progress.

Critical weaknesses
  1. The current ablations do not isolate the contribution of the proposed local-contact reasoning mechanism. RiCo differs from the closest point-based baseline in several coupled dimensions, including the number and content of contact candidates, the use of surface normals and signed offsets, preservation of point-level features, intra-object self-attention, model capacity, and surface resolution. The existing ablations vary the sign feature, neighborhood size, and resolution, but do not separate explicit cross-object contact encoding from intra-object point reasoning. As a result, the experiments demonstrate that the complete RiCo system performs well, but do not establish which proposed component is responsible for the improvement. A particularly important comparison would keep the input features and resolution fixed while replacing the point Transformer with object-level aggregation, and conversely augment an object-level baseline with RiCo’s contact descriptors.

  2. Table 1 is not a resolution-matched comparison between RiCo and RigidFormer. The default RiCo model uses 1024 surface points per object, whereas the reported RigidFormer results were obtained using point counts matched to the corresponding mesh resolution, including substantially fewer points for simple shapes. Figure 4 partially addresses this concern on MOVi-B by comparing both methods across matched nominal resolutions, where RiCo remains better. However, this experiment uses an independent RigidFormer reimplementation and does not establish parity in input features, training budget, or model capacity. Therefore, the complete empirical evidence supports the effectiveness of RiCo, but Table 1 alone does not isolate the effect of the proposed local-contact architecture.

Questions for authors
  1. What exact point-cloud resolution and per-point input features are used for RiCo and RigidFormer in Table 1 and Figure 4? Please clarify whether the RigidFormer entries in Table 1 were retrained using RiCo’s 1024-point representation or copied from the mesh-resolution-matched setting of the original RigidFormer paper, and whether the Figure 4 reimplementation receives the same surface normals, physical attributes, and contact information as RiCo.

  2. Can the authors provide a controlled RiCo–RigidFormer comparison using identical point clouds, input features, train/test splits, optimization budgets, checkpoint-selection criteria, and evaluation code? Reporting parameter counts, training compute, and mean ± standard deviation over multiple seeds would help determine whether the improvement persists independently of representation resolution and implementation differences.

  3. Can the authors separately ablate the two central components of RiCo? Specifically, please evaluate (i) RiCo’s complete contact descriptors with the intra-object point Transformer replaced by permutation-invariant object-level aggregation, and (ii) the point Transformer retained while replacing the top-KK contact descriptors with a RigidFormer-style single nearest-neighbor displacement. These comparisons would determine whether the gains primarily arise from richer local contact encoding, point-level intra-object reasoning, or their combination.

6 comments
AuthorsOpenReview ↗

Author response to Reviewer 1AAs - Part 1

Dear Reviewer, we sincerely value the time and effort you have devoted to evaluating our work. To address each point you raised, we have prepared comprehensive responses and clarifications. We hope these responses can resolve your concerns.

  1. "Question: What exact point-cloud resolution and per-point input features are used for RiCo and RigidFormer in Table 1 and Figure 4? Please clarify whether the RigidFormer entries in Table 1 were retrained using RiCo’s 1024-point representation or copied from the mesh-resolution-matched setting of the original RigidFormer paper, and whether the Figure 4 reimplementation receives the same surface normals, physical attributes, and contact information as RiCo."

Our response.

a. Point-cloud resolution and input features.

RiCo consistently uses the same feature design across all experiments. In Table 1, RiCo operates on 1024 uniformly sampled surface points per object. Each point is represented by a 15-dimensional state feature, consisting of:

  • COM-relative position (3D)
  • Displacement (3D)
  • Surface normal (3D)
  • Physical properties: inverse mass, friction, and restitution (3D)
  • Reference-frame offset (3D)

In addition, RiCo explicitly models inter-object interactions through sparse point-level contacts. For each surface point, we retrieve the top-K nearest contact candidates (K = 4) and construct a 14-dimensional contact descriptor for each candidate, incorporating relative geometric, motion, and physical information. These contact descriptors are processed separately from the 15D point-state features and subsequently integrated into intra-object Point Transformer reasoning.

RigidFormer, in contrast, consistently follows the original feature design described in its paper. Each point uses a 12-dimensional representation, consisting of:

  • Nearest-neighbor displacement (3D)
  • Velocity (3D)
  • Reference-frame offset (3D)
  • Physical properties: mass, friction, and restitution (3D)

Importantly, we do not introduce RiCo-specific features into RigidFormer. In particular, RigidFormer does not receive RiCo's explicit surface-normal features or 14D top-K contact descriptors. Although both methods incorporate physical attributes and geometric interaction information, they encode these through their respective original representations.

b. Clarification of Table 1.

The RigidFormer results in Table 1 are directly taken from the original RigidFormer paper, rather than obtained by retraining RigidFormer with RiCo's 1024-point representation.

In the original RigidFormer experiments, the input point resolution is matched to the mesh vertex count of each object, which varies across object geometries. Therefore, unlike RiCo's fixed 1024-point sampling, RigidFormer does not use a uniform number of input points per object in Table 1.

Consequently, Table 1 compares the two methods under their respective original experimental configurations, rather than enforcing identical input resolutions or feature representations.

c. Clarification of Figure 4.

For Figure 4, we independently reimplemented RigidFormer to investigate the influence of point-cloud resolution under more comparable geometric sampling conditions. Specifically, we evaluate the methods at 128, 256, 512, and 1024 surface points per object, using the same sampling resolution wherever corresponding results are available.

However, only the geometric sampling resolution is controlled; the original feature designs of the two methods remain unchanged:

  • RiCo retains its 15D point-state features and 14D top-K contact descriptors.
  • RigidFormer retains its original 12D point features, without receiving RiCo's additional surface-normal or explicit contact-descriptor inputs.

Therefore, Figure 4 provides a comparison under matched point-cloud resolutions while preserving the representation and interaction-modeling mechanisms of each method. This allows us to examine how the methods behave as input resolution changes, although it is not a fully feature-matched architectural comparison.

AuthorsOpenReview ↗

Author response to Reviewer 1AAs - Part 2

  1. "Question: Can the authors provide a controlled RiCo–RigidFormer comparison using identical point clouds, input features, train/test splits, optimization budgets, checkpoint-selection criteria, and evaluation code? Reporting parameter counts, training compute, and mean ± standard deviation over multiple seeds would help determine whether the improvement persists independently of representation resolution and implementation differences."

Our response. We conducted additional controlled RiCo–RigidFormer comparisons at 256, 512, and 1024 points per object, using identical point clouds and input features (including surface normals and COM), the official RigidFormer data splits, matched training configurations, and a shared autoregressive evaluation protocol.

Both methods were trained for 60 epochs under matched optimization settings, with the final checkpoint selected for evaluation. We report prediction errors, parameter counts, and training GPU-hours, with detailed experimental configurations provided below.

Table 1. Controlled RiCo–RigidFormer comparison on MOVi-B at 100-frame rollouts.

MethodPointsPosition RMSE (m) ↓Rotation RMSE (°) ↓Parameters (Trainable / Total)GPU-hours
RigidFormer2560.2157 ± 0.019818.19 ± 1.69105.05M / 146.45M132.99
RiCo2560.1382 ± 0.014511.69 ± 1.0830.36M / 30.36M39.67
RigidFormer5120.1994 ± 0.019319.00 ± 1.78105.05M / 146.45M137.39
RiCo5120.1284 ± 0.011911.21 ± 1.1530.36M / 30.36M78.85
RigidFormer10240.1925 ± 0.018918.25 ± 1.73105.05M / 146.45M148.34
RiCo10240.1173 ± 0.011310.67 ± 1.0930.36M / 30.36M162.77

All runs use eight RTX 5090 GPUs. GPU-hours are calculated as eight times the recorded training duration. RMSE statistics are reported across three evaluation seeds.

Table 2. Controlled experimental configurations for RiCo and RigidFormer.

SettingConfiguration
Point-cloud resolution256 / 512 / 1024 points per object
Point-cloud samplingIdentical point clouds for paired experiments;
Input featuresMatched features, including surface normals and center of mass (COM)
Dataset splitsOfficial RigidFormer train / validation / test splits ids
Temporal setup2 context frames; forecast horizon 1; sequence length 3
Training budget60 epochs; batch size 128; micro-batch size 4; BF16 precision
Training objectiveTeacher-forced velocity-change and rigid velocity-change losses (weight 1.0 each); anchor-position loss weight 0
OptimizerCosine LR schedule (1e-4 to 1e-5); weight decay 0; gradient clipping 1.0
Checkpoint selectionSaved every 10 epochs; final checkpoint used for evaluation
EvaluationAutoregressive rollout at 50 / 75 / 100 frames; interval 1
Hardware8 × NVIDIA RTX 5090 GPUs per run

Under matched experimental settings, RiCo consistently outperforms RigidFormer across all three point-cloud resolutions. At 1024 points, RiCo reduces position RMSE by 39.1% (0.1925 m to 0.1173 m) and rotation RMSE by 41.5% (18.25° to 10.67°). Similar improvements are observed at 256 and 512 points, demonstrating that RiCo's advantage persists across different input resolutions rather than arising solely from differences in point-cloud sampling or input features.

Furthermore, RiCo uses approximately 71.1% fewer trainable parameters (30.36M vs. 105.05M). Its training compute is substantially lower at 256 and 512 points, although it becomes slightly higher at 1024 points (162.77 vs. 148.34 GPU-hours). These results demonstrate that RiCo achieves better prediction accuracy with a considerably smaller model, while its computational advantage depends on point-cloud resolution.

The differences from our original submission primarily arise from the availability of RigidFormer's official implementation after submission. Our updated experiments are built directly upon the official codebase, providing a more faithful reproduction and improved RigidFormer performance compared with our previous reimplementation. Importantly, RiCo maintains a substantial accuracy advantage even under this more rigorous controlled comparison.

The complete validation curves for all six experiments are provided in our anonymous supplementary materials .

AuthorsOpenReview ↗

Author response to Reviewer 1AAs - Part 3

  1. "The current ablations do not isolate the contribution of the proposed local-contact reasoning mechanism. ... A particularly important comparison would keep the input features and resolution fixed while replacing the point Transformer with object-level aggregation, and conversely augment an object-level baseline with RiCo’s contact descriptors." "Question: Can the authors separately ablate the two central components of RiCo? Specifically, please evaluate (i) RiCo’s complete contact descriptors with the intra-object point Transformer replaced by permutation-invariant object-level aggregation, and (ii) the point Transformer retained while replacing the top-(K) contact descriptors with a RigidFormer-style single nearest-neighbor displacement. These comparisons would ..."

Our response. We conducted two complementary component-swap ablations on MOVi-B to investigate the contributions of local contact encoding and intra-object reasoning:

  • RF Features + RiCo Transformer: Replaces RiCo's top-K contact descriptors with RigidFormer-style point-wise features while retaining RiCo's intra-object Point Transformer.
  • RiCo Contact + RF Transformer: Retains RiCo's top-K contact descriptors but replaces its intra-object Point Transformer with RigidFormer's scene-level Transformer.

Table 1. Autoregressive rollout performance on MOVi-B.

MethodFramePosition RMSE (m) ↓Rotation RMSE (°) ↓
RF Features + RiCo Transformer500.055 ± 0.0704.70 ± 5.73
750.127 ± 0.13110.55 ± 10.60
1000.212 ± 0.19517.46 ± 15.72
RiCo Contact + RF Transformer500.033 ± 0.0413.34 ± 4.44
750.072 ± 0.0727.42 ± 8.28
1000.121 ± 0.10712.10 ± 12.23
RiCo (Full)500.032 ± 0.0402.59 ± 4.04
750.065 ± 0.0855.45 ± 7.32
1000.111 ± 0.1229.54 ± 10.43

The reported standard deviations reflect variation across evaluation trajectories, not across different seeds.

Table 2. Geometric penetration statistics on MOVi-B.

MethodPTR (%) ↓MPD (mm) ↓
Ground Truth10.703.91
RF Features + RiCo Transformer36.3925.93
RiCo Contact + RF Transformer27.056.82
RiCo (Full)21.706.13

The results demonstrate the importance of RiCo's local contact representation for long-horizon motion prediction. Replacing RiCo's contact descriptors with RigidFormer-style features substantially increases prediction errors, particularly at longer horizons. At 100 frames, position RMSE increases from 0.111 m to 0.212 m, while rotation RMSE increases from 9.54° to 17.46°. In contrast, retaining RiCo's contact representation with RigidFormer's Transformer yields considerably smaller errors (0.121 m / 12.10°), suggesting that fine-grained contact encoding contributes substantially to prediction accuracy.

The penetration results further support this observation. Replacing RiCo's contact representation increases PTR from 21.70% to 36.39% and MPD from 6.13 mm to 25.93 mm. Meanwhile, replacing only the Transformer while retaining RiCo's contact representation results in substantially smaller increases (27.05% PTR / 6.82 mm MPD).

Overall, local contact encoding appears to be a major contributor to both motion accuracy and penetration reduction, while RiCo's intra-object Point Transformer provides additional improvements. Full RiCo consistently achieves the best performance across all prediction horizons and penetration metrics, supporting the effectiveness of combining sparse point-level contact modeling with intra-object reasoning.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

We thank the AC and reviewers for their constructive feedback. We conducted additional experiments and clarifications to address major concerns, summarized below.

We addressed four main areas:

  • Fair Baseline Comparisons and Experimental Reproducibility (Reviewers 1AAs, cQV5): We conducted controlled RiCo–RigidFormer comparisons at 256, 512, and 1024 points per object, matching point clouds, input features, dataset splits, training configurations, and evaluation protocols using the official RigidFormer codebase. RiCo outperforms RigidFormer across all resolutions. At 1024 points, RiCo reduces position RMSE by 39.1% and rotation RMSE by 41.5%, with 71.1% fewer trainable parameters. We provided configurations, parameter counts, GPU-hours, and validation curves, clarified representation differences between Table 1 and Figure 4, and added published FIGNet and SDF-Sim comparisons on MOVi-B.

  • Component Ablations and Physical Plausibility (Reviewers 1AAs, cQV5): Two component-swap ablations isolate the contributions of sparse local contact encoding and intra-object Point Transformer reasoning. At 100 frames, replacing RiCo's contact descriptors with RigidFormer features increases position RMSE from 0.111 m to 0.212 m, while retaining RiCo's contact representation with RigidFormer's Transformer yields 0.121 m. These results emphasize the importance of local contact encoding. We also reported object–object and object–ground penetration statistics, absolute GT PTR/MPD, and per-sequence MPD distributions. RiCo achieves 21.70% PTR and 6.13 mm MPD, versus RigidFormer's 50.27% and 50.76 mm, demonstrating substantially reduced penetration.

  • Long-Horizon Stability and Robustness (Reviewer cQV5): We explained how Kabsch-based rigid projection preserves geometry during autoregressive rollouts. A 10-epoch autoregressive fine-tuning experiment on MOVi-A showed nearly unchanged position and rotation errors across 50-, 75-, and 100-frame horizons, indicating stable long-horizon performance without substantial improvement or degradation.

  • Scalability, Generalization, and Scope (Reviewer ePFR): Despite training on 3–10 objects, RiCo achieves 5.02–6.54 FPS across three 270-object scenarios. Profiling shows that network computation dominates latency even as contact pairs increase. We also explored a preliminary deformable-body extension using weighted interpolation to relax Kabsch rigidity, achieving 0.0549 m autoregressive node RMSE, with qualitative comparisons against FEM ground truth. This suggests potential beyond rigid-body dynamics, although comprehensive deformation modeling remains future work. We position RiCo as complementary to established physics simulators rather than a replacement.

The additional experiments reinforce four central contributions:

  • Mesh-free, physically informed contact modeling: RiCo models sparse point-level interactions without mesh connectivity or pretrained SDFs, supported by component ablations. (Reviewers 1AAs, cQV5)

  • Predictive accuracy and physical plausibility: Controlled comparisons demonstrate consistent accuracy gains, while penetration analyses confirm reduced geometric inconsistency. (Reviewers 1AAs, cQV5)

  • Efficient, scalable neural simulation: RiCo handles hundreds of objects despite training on smaller scenes, using considerably fewer trainable parameters than RigidFormer. (Reviewers ePFR, 1AAs)

  • Long-horizon stability and extensibility: Autoregressive fine-tuning maintains stable performance, while preliminary deformation experiments suggest broader applicability. (Reviewers cQV5, ePFR)

We appreciate the reviewers' feedback, which helped strengthen our experiments and clarify RiCo's contributions and limitations. We believe the controlled comparisons, component ablations, penetration analyses, and generalization experiments substantially address the major concerns and provide stronger evidence of RiCo's effectiveness.

We hope this summary assists the AC in evaluating our submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.