acceptodds
Under review as a conference paper at ICLR 2027

OrbiSim: World Models as Differentiable Engines for Embodied Intelligence

15 views

Abstract

We present OrbiSim, a novel robotic simulation paradigm that redefines world models as a fully differentiable physics engine for embodied intelligence. Unlike prior world models that focus on unconstrained imagination in latent or visual domains, OrbiSim establishes a unified, physically-grounded pathway that bridges structured scene assets, neural dynamics, and downstream reinforcement learning. By enabling end-to-end differentiability throughout the entire simulation loop—spanning from explicit state transitions to visual observation generation—OrbiSim supports tasks traditionally intractable for classical simulators, such as differentiable contact modeling, gradient-based policy optimization under sparse rewards, and intuitive physical inference. Empirical results demonstrate that OrbiSim significantly outperforms state-of-the-art world models in both predictive fidelity and control performance. Furthermore, its consistent responsiveness to asset configurations and physical parameters suggests its potential as a differentiable tool for enhancing robot simulation and policy training.

Primary area:
reinforcement learning
Keywords:
Embodied world models, differentiable physical simulation, gradient-based policy optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wWGkOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OrbiSim is an asset- and parameter-conditioned world model that separates explicit state dynamics from visual generation. The differentiable pipeline supports prediction, physical-parameter identification, and gradient-based policy or action optimization. The paper evaluates these capabilities in simulation and on a real cable-manipulation task.

Critical strengths
  1. Separating explicit dynamics from visual generation makes physical parameters available for direct optimization. Connecting this interface to both system identification and control is practically interesting.
  2. The paper includes downstream evidence beyond prediction metrics, notably the Newton initial-velocity experiment and Cable Lift.
Critical weaknesses
  1. The Push success criteria require reconciliation. Appendix A.1 describes reaching a designated region while satisfying constraints against objects falling off the table, whereas Table 9 and Appendix C.11 count success when the second cube moves more than 5 cm. The reported 42.71% versus 25.00% therefore supports the latter metric, but cannot directly establish completion of the former task. Pretraining data, environment interactions, and update budgets should also be accounted for explicitly when interpreting the control comparison.
  2. The evidence for useful gradients has a narrower scope than a general replacement for a differentiable physics solver. Appendix C.10 optimizes a ball's initial velocity: Newton reaches zero loss within 60 steps, whereas OrbiSim reduces its loss below 10⁻⁴ within 800 steps. This demonstrates useful optimization in that example, rather than equivalent optimization efficiency or generally accurate contact gradients. Cable Lift provides further evidence, but its alternating identification and optimization should be distinguished from fixed-policy baselines in terms of test-time adaptation. Equation 12 also uses geometrically shaped rewards delivered at termination, which differs from binary success feedback.
Questions for authors
  1. Are the different success criteria in Appendix A.1 and Table 9 intentional? If trajectories are available, please report both criteria on the same trajectories and clarify each method's data, interaction, update, and adaptation budgets.
  2. For the existing Push or Cable Lift experiments, can the authors relate improvements predicted during model-based optimization to improvements after execution, including failure cases? This would directly strengthen the central claim about useful gradients without requiring a new benchmark suite.
4 comments
AuthorsOpenReview ↗

Response to Reviewer wWGk (Part 1)

W1 & Q1: Success definition and reports of the robosuite Push task. Pretraining data, environment interactions and update budgets.

1. Success criteria and new success rate results

The original manuscript stated inconsistent success criteria. We redesign the success criteria and re-evaluate success rates to fit our initial task objective. In our evaluation, a robosuite Push task rollout is successful when r3≥0.2r_3 \geq 0.2, where r3r_3 is the relative reduction in the distance from the second cube to the table edge over the rollout, corresponding to a reduction of at least 20%. Under this criterion, OrbiSim achieves a success rate of 29.17%, compared with 4.92% for DreamerV3 and 0% for PPO, SAC, and behavior cloning.

Revision: New success criteria and success reports are updated in paper's Appendix A.1, C.11 sections and Table 8.

2. Pretraining data, environment interaction and update budget statistics

Regarding pretraining data, collecting the robosuite dataset of approximately 234k frames (11,740 seconds of video) takes about one hour, and training OrbiSim-Dynamics takes about 30 hours on a single NVIDIA A800 GPU. We use the world model as a simulator and report this construction cost separately from policy training, analogous to setting up a simulation environment for downstream learning.

The table below distinguishes ground-truth simulator samples, imagined transitions, and policy optimizer updates. The model-free methods, PPO and SAC, collect their policy-training samples directly from robosuite and use no imagined transitions. OrbiSim uses the pre-collected dataset to train its world model, then trains its policy entirely within that model without further interaction with robosuite. DreamerV3 receives the same pretraining sample budget, but additionally collects simulator samples during training and uses imagined rollouts for policy learning. Optimizer-update counts are algorithm-specific and do not imply equal data or computational budgets.

MethodGround-truth simulator samples (pretraining + policy training)Imagined transitionsPolicy optimizer updates
OrbiSim2.34 × 10⁵ + 02.56 × 10⁷1.6 × 10⁴
PPO0 + 3.2 × 10⁶01.6 × 10⁵
SAC0 + 3.2 × 10⁶01.6 × 10⁴
DreamerV32.34 × 10⁵ + 3.2022 × 10⁶≈3.075 × 10⁹2.00225 × 10⁵

Revision: We added computational costs and training budgets in Appendix C.14 and Table 14.

AuthorsOpenReview ↗

Response to Reviewer wWGk (Part 2)

W2.1: The Newton comparison leaves optimization efficiency and contact-gradient reliability unclear.

We agree that the ball experiment alone cannot establish comparable efficiency across tasks or gradient accuracy at all contact events. In the revised paper, we distinguish optimization outcomes from computational cost in Appendix C.13 and examine gradients near contact separately in Appendix C.9.

Optimization outcomes. The official Newton diffsim_ball task tests whether OrbiSim can optimize a task supported by an existing differentiable simulator using learned gradients. With the same learning rate, Newton reaches zero loss within 60 steps, while OrbiSim reduces the loss below 10−410^{-4} within 800 steps. OrbiSim thus reaches a comparably low loss, although it converges more slowly on this task. We additionally implement a differentiable Newton baseline for Cable Lift and compare state-based system identification and action-sequence optimization on the same test cases. Figure 23 and Table 11 of the revised paper report the full results:

MethodTarget distance sum: before →\rightarrow after (cm)Physical-parameter error: before →\rightarrow afterAverage time per iteration (s)
Newton (ID)24.62 →\rightarrow 3.560.2086 →\rightarrow 0.1044100.3
OrbiSim (ID)24.62 →\rightarrow 1.220.2086 →\rightarrow 0.029377.5
Newton (OOD)18.96 →\rightarrow 4.070.4518 →\rightarrow 0.1666100.3
OrbiSim (OOD)18.96 →\rightarrow 1.530.4518 →\rightarrow 0.067377.5

Both methods improve the target distance and parameter estimates under ID and OOD conditions. In this Cable Lift experiment, OrbiSim achieves lower final errors with less average time per optimization iteration.

Differentiable computation and runtime. Table 12 of the revised paper details the computation and gradient path within each integration substep across three tasks; Table 13 reports the corresponding rollout-update times. An update includes the forward rollout, backpropagation, and optimizer step.

TaskMethodControl stepsIntegration substepsRollout update time (s)Amortized time per substep (ms)
diffsim_ballNewton362880.230.81
OrbiSim36360.4913.50
Cable LiftNewton402,5602.000.78
OrbiSim40401.5538.75
PushNewton804,000182.8145.70
OrbiSim80802.9036.31

On diffsim_ball, Newton advances one particle with contact against static surfaces, yielding a lighter differentiable computation and a faster update than OrbiSim. Cable Lift requires gradients through a flexible particle chain, while Push additionally involves robot kinematics, control, rigid-body contacts, and their gradients. The cost of Newton's differentiable rollouts increases substantially across these implementations. OrbiSim instead optimizes through learned Dynamics transitions. The Cable Lift comparison provides optimization results; for Push, we report a runtime analysis, with an approximately 63-fold difference in single-rollout update time under the current implementations.

Gradients near contact. We do not interpret these optimization results as evidence that learned gradients are accurate at every contact event. Appendix C.9 and Figure 14 separately compare OrbiSim Jacobians with MuJoCo finite-difference estimates over a representative 40-step trajectory containing contact phases. Action-Jacobian cosine similarity is positive at 35 of 40 steps, and state-Jacobian cosine similarity is positive at all 40 steps, although local fluctuations occur near contact-mode transitions. Our claim is that the learned gradients provide useful local directions in the evaluated settings, not that they exactly reproduce derivatives at every instantaneous contact.

Revision: We clarify the purpose of the ball experiment and add the Cable Lift optimization comparison in Appendix C.13.1, the three-task computational analysis in C.13.2, and the contact-gradient analysis in C.9.

AuthorsOpenReview ↗

Response to Reviewer wWGk (Part 3)

W2.2: Distinguishing alternating test-time adaptation from fixed-policy baselines.

We agree that adaptation budgets should be matched. In the real-robot experiments in Sec. 4.3.1 of the revised manuscript, OrbiSim uses a single adaptation round: we collect eight real-robot RGB/action trajectories to identify the cable’s physical parameters, fix the inferred parameters, and optimize an action sequence for deployment. No additional observations are used to update the model or actions during deployment.

For a matched comparison, we additionally fine-tuned the π0.5\pi_{0.5} baseline on the same eight trajectories and kept the resulting policy fixed during deployment. Both methods therefore receive the same real-robot data and one round of adaptation before evaluation. Using the same mean pixel-distance metric averaged over the two cable endpoints as in Table 10, we obtain:

MethodMean distance to target (px) ↓
π0.5\pi_{0.5} (original)62.2
π0.5\pi_{0.5} (fine-tuned on the same real-robot data)45.1
OrbiSim17.5

Fine-tuning improves π0.5\pi_{0.5} from 62.2 to 45.1 px, while OrbiSim retains a lower mean distance of 17.5 px under the matched adaptation-data budget.

The multi-round alternating procedure is evaluated separately in the controlled simulation diagnostics in Sec. 4.3.2 of the revised manuscript, with eight rounds of physical-parameter identification and action optimization under matched ID and OOD conditions.

Revision: We clarify the single-round real-robot adaptation in Sec. 4.3.1 and report the fine-tuned π0.5\pi_{0.5} result in Appendix C.12 and Table 10.

W2.3: Limited evidence for OrbiSim under binary sparse rewards.

We thank the reviewer for raising this concern. We added an experiment on the robosuite Push task in which all three reward components, r1,r2,r3r_1,r_2,r_3, were changed from the original dense rewards to sparse binary rewards defined by annotation thresholds. We relabeled the existing world-model training dataset with these sparse rewards and trained a reward predictor on the resulting labels. We then repeated the same policy optimization experiments, replacing the original reward signal with the predictor’s estimate of r1+r2+r3r_1+r_2+r_3.

Figure 18 of the revised manuscript shows the training curves. Metric 1, Metric 2, and Metric 3 in the figure are computed using the original definitions of r1r_1, r2r_2, and r3r_3, respectively, and Total is their sum.

As expected, policy performance is lower under sparse reward supervision than in the original dense-reward setting. Thresholding removes fine-grained reward information, while the reward predictor introduces approximation error and an additional stage in gradient propagation. Nevertheless, OrbiSim still achieves policy improvement in this setting: the learned predictor provides a smooth, differentiable surrogate for the sparse rewards, enabling gradient-based optimization through OrbiSim-Dynamics. These results demonstrate that OrbiSim can support policy learning with sparse binary reward supervision, with reduced performance relative to dense rewards.

Revision: Policy optimization results with binary rewards are updated in Appendix C.11.3.

Q2: Relating model-predicted gains to execution outcomes.

We compare the rewards of policies trained by model-based method in OrbiSim-Dynamics between rollouts in robosuite and in OrbiSim-Dynamics. The training curves of evaluation rollouts in robosuite and OrbiSim-Dynamics are shown in Figure 16 in the revised manuscript. Although detailed rewards calculated in OrbiSim-Dynamics have deviation comparing to rewards in robosuite, they have same growing trends and similar total rewards. While model-based policy optimization in OrbiSim do not requires accurate rewards, policies trained with gradients from OrbiSim-Dynamics gain improvements in robosuite evaluation.

Revision: Comparison settings and results are updated to Appendix C.11.2.

Thanks for the response. I’ve raised my score.

Reviewer_QstyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces OrbiSim, a differentiable world-model framework for embodied intelligence that aims to transform learned world models into physics simulation engines suitable for gradient-based optimization. The framework consists of two main components: OrbiSim-Dynamics, an object-centric recurrent dynamics model that predicts future physical states based on historical states, control actions, and physical attributes; and OrbiSim-Vision, a state-conditioned latent diffusion model that generates future visual observations. By decoupling physical dynamics prediction from visual generation, OrbiSim supports gradient-based system identification and policy optimization. The authors evaluate the framework across diverse tasks involving robotic manipulation, rigid-body interactions, articulated objects, and deformable objects, and demonstrate its potential for real-world deployment through a Cable Lift task. The experimental results show competitive performance on several prediction metrics and downstream control tasks.

Critical strengths
  1. The paper integrates physical state prediction and visual observation generation into a differentiable framework, providing a unified approach to gradient-based system identification and policy optimization using learned dynamics.

  2. OrbiSim represents robots and scene objects as separate entities and conditions the dynamics on their physical attributes and scene configurations. This design facilitates the modeling of different object types and varying physical parameters, with evaluations covering rigid, articulated, and deformable objects.

  3. Beyond evaluating future-state and visual prediction, the paper investigates gradient-based system identification and policy optimization. The real-world Cable Lift experiments further demonstrate the potential of the proposed framework for downstream robotic control.

Critical weaknesses

1、The paper emphasizes the end-to-end differentiability between OrbiSim-Dynamics and OrbiSim-Vision. However, the strategy optimization in Section 4.3 seems to mainly rely on the backward propagation of the predicted physical state trajectories. Could the author please clarify further whether OrbiSim-Vision directly participates in the strategy optimization process or is mainly used for visual prediction and system identification based on visual observations?

2、In Section 3.2, OrbiSim-Dynamics is defined as a cyclic model that depends on historical states, and the gradient calculation in Equation (8) adopts the form of a single-step state transition. Could the author further explain how the gradient is propagated through the cyclic hidden states and the historical state dependencies during the policy optimization process? Was the actual implementation explicitly considering these dependencies by using Time Backward Propagation?

3、The paper evaluated OrbiSim under different object types and physical parameter configurations. Could the author please further explain how this model demonstrates its generalization ability for geometric shapes and physical parameter combinations of objects that were not seen during training, especially when multiple factors change simultaneously?

Questions for authors

See the weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Qsty

W1: The role of OrbiSim-Vision in policy optimization was unclear.

OrbiSim-Vision is not used for policy optimization. As clarified in the new Appendix B.4 and Figure 9 of the revised paper, control objectives are evaluated on physical-state trajectories predicted by OrbiSim-Dynamics, and their gradients are propagated through the Dynamics rollout to the controls. OrbiSim-Vision is used for visual prediction and image-based system identification.

Revision: We clarified the gradient propagating path and roles of OrbiSim-Vision and OrbiSim-Dynamics in policy optimization in Appendix B.4 of the revised paper.

W2: Clarifying gradient propagation through recurrent states and BPTT.

During multi-step optimization, OrbiSim-Dynamics is unrolled across the rollout, and the hidden states are not detached. Gradients therefore propagate through both predicted physical states and recurrent hidden-state updates across time, using BPTT. We clarify this temporal gradient path in the new Appendix B.4 and Figure 9.

Revision: We clarified gradient propagation through recurrent hidden states and the use of BPTT in Appendix B.4 and Figure 9.

W3: Generalization to unseen geometries and combined OOD shifts.

We have evaluated OrbiSim under OOD conditions; please see Appendix C.7 of the revised manuscript. The evaluation covers both Push and Drape. In Push, we test object-count OOD using numbers of cubes unseen during training, and physical-parameter OOD using unseen friction combinations. Our OOD test set also includes cases where both the object count and friction combination are out of distribution. In Drape, we test object-count OOD by varying the number of ground objects beyond the training configurations, and shape OOD by using ground objects with unseen geometries. The OOD test set also includes cases where both object count and shape are simultaneously out of distribution.

Table 5 of the revised manuscript reports OrbiSim’s results under the ID and OOD evaluations. Overall, its predictive performance remains stable under the tested distribution shifts. These results show that OrbiSim largely retains its predictive capability under the tested shifts in object count, shape, and physical parameters, including cases where multiple factors are simultaneously out of distribution.

MethodPSNR10 ↑PSNR100 ↑LPIPS10 ↓LPIPS100 ↓FVD ↓
OrbiSim (ID)26.710519.98190.10780.1428533.9
OrbiSim (OOD)27.186720.11090.11880.1847597.3
Reviewer_QstyOpenReview ↗

reply

Thans for the author's reply. I will keep my rating.

Reviewer_yUtJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents OrbiSim, a learned world model designed to function as a differentiable physics engine for robotic manipulation. Unlike conventional generative world models that primarily predict future visual observations, OrbiSim explicitly represents physical states, object attributes, and scene parameters, allowing gradients to propagate through simulated dynamics for downstream optimization.

The framework consists of two main components: (1) OrbiSim-Dynamics, an object-centric recurrent model combining Transformer-based interaction modeling with physical-parameter conditioning, and (2) OrbiSim-Vision, a state-conditioned diffusion model that generates visual observations from predicted physical states. The resulting framework supports gradient-based system identification and policy or action optimization.

The authors evaluate OrbiSim across rigid-body manipulation, articulated objects, and deformable-object scenarios. The results demonstrate improvements over existing world models in long-horizon prediction and physical trajectory consistency. On the robosuite Push task, OrbiSim achieves a 42.71% success rate, compared with 25.00% for DreamerV3. Real-robot experiments on a dual-arm cable manipulation task further demonstrate the potential of vision-based physical parameter identification and gradient-based action optimization.

Critical strengths
  • The separation of explicit physical dynamics and visual rendering, together with asset-conditioned object-centric modeling, provides a practical interface for physical-state prediction, parameter adjustment, and gradient-based optimization. This extends the role of world models beyond visual generation toward physically grounded control and system identification, addressing an important limitation of existing generative world models.

  • The paper evaluates not only visual fidelity and long-horizon trajectory consistency, but also downstream policy optimization, physical-parameter identification, and real-robot manipulation. In particular, the cable manipulation experiments demonstrate a practical application of the proposed framework, providing encouraging evidence that learned differentiable dynamics can support real-world control.

Critical weaknesses
  • Although OrbiSim introduces static scene descriptors and geometry encoders, it remains unclear how collision geometries (e.g., tables and walls) are represented and whether physical constraints such as non-penetration are explicitly enforced or implicitly learned. Moreover, the evaluations primarily focus on prediction quality, without sufficient quantitative analysis of contact consistency under unseen geometric configurations. This limits the evidence supporting OrbiSim's claim as a general-purpose physics engine.

  • The reported computational statistics do not establish an efficiency advantage over conventional physics simulators. Since OrbiSim requires simulator-generated training data and additional model training, its practical benefits should be evaluated in terms of total computational cost and downstream optimization efficiency.

Questions for authors
  • While the paper introduces static scene descriptors, it remains unclear how collision geometries such as tables and walls are represented, and whether non-penetration constraints are explicitly enforced or implicitly learned. Could the authors evaluate physical consistency under counterfactual changes in table height or wall position?

  • Could the authors compare gradient-based and gradient-free optimization using the same OrbiSim-Dynamics model? Additionally, how reliable are the learned gradients near contact events?

  • Could the authors provide a total compute–performance scatter plot comparing OrbiSim with PPO, SAC, and DreamerV3?

4 comments
AuthorsOpenReview ↗

Response to Reviewer yUtJ (Part 1)

W1 & Q1: Scene collision-geometry representation and non-penetration constraints

To address this concern, we conducted two additional analyses: first, we include table height and tilt as scene-geometry conditions to evaluate OrbiSim’s prediction capability under unseen table configurations; second, we introduce a penetration constraint to reduce object interpenetration in predicted trajectories.

1. Scene collision-geometry representation.

For scene modeling, we generally consider only the factors that affect the dynamics of the current scene. In the original setting, we model the table as a fixed and sufficiently large plane. Therefore, we include only the table friction coefficient as part of the World configurations xˉw\bar{x}^{w}, without explicitly modeling the table height and tilt.

To further verify whether OrbiSim can adjust its contact behavior according to scene geometry, we add the table height and tilt to xˉw\bar{x}^{w}, collect a dataset containing variations in table height and tilt, and use it to fine-tune OrbiSim. We then evaluate the fine-tuned OrbiSim separately on the ID and OOD test data. In the ID setting, the table height and tilt follow the training-data distribution, with a table-height variation of 5 cm and a tilt variation of 5∘5^\circ. In the OOD setting, the table height and tilt come from more challenging distributions beyond the training range, with a table-height variation of 15 cm and a tilt variation of 15∘15^\circ. The results are shown below.

MethodPSNR10 ↑\uparrowPSNR100 ↑\uparrowLPIPS10 ↓\downarrowLPIPS100 ↓\downarrowFVD ↓\downarrow
OrbiSim (original)26.710519.98190.10780.1428533.9
OrbiSim (table height/tilt, ID)24.162518.67750.12150.1720689.5
OrbiSim (table height/tilt, OOD)23.879518.07530.14160.2079723.5

Under the new table height/tilt setting, the visual metrics show a small degradation from ID to OOD, while the model maintains stable long-horizon prediction performance. This shows that, after explicitly modeling the table height and tilt, OrbiSim can use the explicit conditions in the World configurations to adjust its state and visual predictions even when these geometric parameters undergo changes unseen during training, rather than relying only on the fixed table geometry in the training scenes.

2. Non-penetration constraints.

The original OrbiSim formulation does not impose a hard non-penetration projection or an analytical constraint on the predicted states. This design preserves end-to-end differentiability from state prediction to visual generation and downstream optimization. As an additional analysis, we design a discriminator to determine whether any pair of objects in a scene is interpenetrating, use its output as an auxiliary constraint term, and fine-tune OrbiSim for 10 epochs. The changes in penetration rates before and after fine-tuning are shown below.

Object pairBefore fine-tuning (%)After fine-tuning (%)
Gripper / table0.360.00
Gripper / object2.700.48
Table / object9.440.04
Object / object2.700.00

As shown in the table, the penetration rates for all four object-pair categories decrease substantially after fine-tuning. These results show that OrbiSim can reduce interpenetration through the penetration constraint while retaining end-to-end gradient propagation. This constraint is soft: it substantially reduces the penetration rate but does not provide an absolute non-penetration guarantee in the sense of an analytical physics solver.

Revision: Non-penetration constraint settings and results are add to Appendix C.10 of our revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer yUtJ (Part 2)

W2 & Q3: Total Computational Cost and Compute–Performance Trade-offs

We thank the reviewer for raising this concern. For robosuite, collecting the 234k frames (11,740 seconds video) dataset takes approximately one hour, and training OrbiSim-Dynamics takes approximately 30 hours on a single NVIDIA A800 GPU.

However, we use the world model as a simulator and therefore report its training cost separately from policy training. This cost is analogous to setting up the simulation environments required by baselines that interact with simulation software during policy training.

We remeasured all methods on a single NVIDIA RTX 4090D; throughput and reward are reported as mean ± standard deviation over three training seeds. As shown in Table 15 and Figure 24 of the revised manuscript, OrbiSim achieves the highest mean reward, with moderately lower throughput than PPO and SAC and 14.3× higher throughput than DreamerV3. Its memory use is also lower than DreamerV3’s (258.3 vs. 426.4 MiB).

MethodTraining efficiency (steps/s)Performance:total rewardMemory(MiB)
OrbiSim82.25 ± 0.251.200 ± 0.149258.3
SAC144.32 ± 1.130.063 ± 0.02835.6
PPO135.53 ± 2.170.026 ± 0.03820.4
DreamerV35.74 ± 0.110.662 ± 0.275426.4

Revision: We added details about computation efficiency and resource usage in Appendix C.14 of the revised manuscript.

Q2.1: Gradient-Based vs. Gradient-Free Optimization

We thank the reviewer for suggesting this comparison. We added a PPO baseline that, like our method, learns a policy by using the same pretrained OrbiSim-Dynamics world model as its environment for the robosuite Push task.

During policy training, both methods interact exclusively with the learned world model; neither interacts with the original robosuite simulator. PPO updates its policy from world-model-generated trajectories without backpropagating through the dynamics model. Our method propagates reward gradients through the world-model rollout to optimize the policy.

The training curves are shown in Figure 19 of the revised manuscript. The results show that PPO can learn within the world model, but achieves a lower final total reward than our method. This comparison supports the benefit of directly using the learned dynamics gradients for policy optimization, beyond using the model to generate training rollouts.

Revision: Comparison of gradient-free and gradient-based methods in OrbiSim-Dynamics are added in Appendix C.11.4 in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer yUtJ (Part 3)

Q2.2: Gradient Reliability near Contact

To further evaluate the reliability of OrbiSim's learned gradients, we added a step-by-step comparison between OrbiSim's learned gradients and the central finite-difference Jacobians computed from MuJoCo, as described in Appendix C.9 of the revised manuscript.

We selected a representative episode from real control and analyzed 40 control steps containing contact events. For each control step, we report the Jacobian cosine similarity, sign agreement, and scaled Jacobian discrepancy, as shown in Figure 14 of the revised manuscript.

Across the 40-step evaluation window, OrbiSim's learned gradients generally preserve the direction of the MuJoCo finite-difference gradients. The cosine similarity of the action Jacobian is positive at 35 out of 40 time steps, while the cosine similarity of the state Jacobian is positive at all time steps. The mean sign agreement for the action Jacobian is approximately 0.720.72, and the mean cosine similarity for the state Jacobian is approximately 0.700.70. Overall, these results indicate that the learned gradients retain useful local directional information at most time steps, rather than completely failing near contact phases.

As shown in Figure 14, the sign agreement of the state Jacobian exhibits larger fluctuations near contact-mode transitions, particularly at time steps where the positive and negative finite-difference perturbations cross different contact modes. However, the cosine similarity of the state Jacobian remains positive in these regions, and we do not observe a systematic reversal of the gradient direction relative to the MuJoCo finite-difference estimate.

We believe that these local discrepancies are related to the inherent nonsmoothness of contact dynamics. When the positive and negative finite-difference perturbations fall into different contact modes or constraint branches, the central finite-difference estimate no longer strictly corresponds to a local derivative on a single smooth dynamics branch. Therefore, the finite-difference estimate in cross-contact regions may mix multiple local dynamics branches and should not be treated as an exact ground truth for a smooth Jacobian. Accordingly, we do not claim that the learned gradients exactly reproduce the MuJoCo Jacobian at every instantaneous collision point. Rather, our claim is that OrbiSim provides useful local directional information for downstream optimization and has practical value for control tasks involving contact.

Furthermore, we added comparisons between optimization using Newton's simulator gradients and OrbiSim's learned gradients. Detailed results are provided in our response to Reviewer wWGk (W2.1) and Appendix C.13.1--C.13.2 of the revised manuscript. OrbiSim reaches a comparably low loss on Newton's official diffsim_ball task and achieves lower final errors with a shorter average optimization iteration time on Cable Lift under the tested conditions. These results further support the practical usefulness of OrbiSim's learned gradients for optimization on the evaluated tasks involving contact.

Revision: We added the contact-gradient analysis in Appendix C.9 and Newton comparisons in Appendix C.13.1--C.13.2.

Thank you for your detailed and constructive responses. My questions have been adequately addressed.

The paper proposes a novel differentiable physics simulation and rendering framework and demonstrates promising generalization to unseen geometric configurations. Nevertheless, extending the framework to more diverse objects, scenes, and robot embodiments remains an important direction for future work.

Considering both the strengths and remaining limitations, I maintain my original rating of 3: Weak Accept.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful evaluation of OrbiSim and their constructive feedback. In response, we added or substantially revised experiments and explanations addressing the task-success criterion, computational budgets, real-robot adaptation, gradient propagation and reliability near contact, and comparisons with differentiable simulators. The table below summarizes these revisions and their locations in the revised manuscript.

Additional experiments or revisionsSummaryLocation in revised manuscript
Push task success criterion and success ratesWe clarify that success is defined by r3≥0.2r_3 \geq 0.2 and update the success-rate evaluation for all methods accordingly.Appendix A.1 and C.11.1; Table 8
Training data, environment interactions, and computational budgetsWe distinguish ground-truth simulator samples, imagined transitions, and policy optimizer updates; report data-collection and Dynamics pretraining costs; and provide training throughput, reward, and memory results measured on unified hardware.Appendix C.14; Tables 14–15 and Figure 24
Real-robot test-time adaptation baselineUsing the same real-robot data budget, we add a π0.5\pi_{0.5} baseline fine-tuned on RGB/action data and kept fixed during deployment, and compare its real-world target distance with OrbiSim.Section 4.3.1 and Appendix C.12; Figure 5 and Table 10
Gradient propagation through recurrent dynamicsWe clarify how gradients propagate through recurrent hidden states over multi-step rollouts, identify the procedure as BPTT, and describe the gradient paths for control optimization, state-based SysID, and image-based SysID.Appendix B.4; Figure 9
Comparison with differentiable Newton simulation and computation analysisWe compare initial-velocity optimization on Newton's official diffsim_ball task and system identification/action-sequence optimization on Cable Lift; we also analyze per-substep computation and rollout-update time across three tasks.Appendix C.13.1–C.13.2; Figures 21–23 and Tables 11–13
Reward comparison between model rollouts and robosuiteWe compare rewards from OrbiSim-Dynamics rollouts and robosuite evaluation to assess whether policy improvements predicted by the model are reflected in simulator evaluation.Appendix C.11.2; Figure 16
Policy optimization with binary sparse rewardsWe add an experiment that trains a reward predictor from binary sparse labels and uses its differentiable predictions for policy optimization.Appendix C.11.3; Figure 18
Gradient-based versus gradient-free policy optimizationWe compare gradient-based policy optimization with PPO in the same pretrained OrbiSim-Dynamics environment.Appendix C.11.4; Figure 19
Gradient reliability near contactWe compare OrbiSim Jacobians step by step with central finite-difference estimates from MuJoCo and discuss discrepancies near contact-mode transitions.Appendix C.9; Figure 14
Non-penetration constraint analysisWe add an auxiliary penetration constraint, fine-tune OrbiSim, and report penetration rates for four object-pair categories before and after fine-tuning; we also clarify that this is a soft constraint.Appendix C.10; Table 7

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.