- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces PhysTrace, a benchmark for evaluating physical simulation quality across six platforms. It standardizes experimental conditions and observation protocols, combines controlled experiments with real-world references, and uses process-level measurements to investigate discrepancies. Experiments cover rigid-body dynamics, deformable objects, and articulated mechanisms, alongside supplementary VLA learning and cross-platform execution studies. The findings highlight that simulation fidelity depends on the task and interaction stage, and that similar overall trajectories can conceal differences in contact processes.
Critical strengths
The standardized six-platform evaluation and process-level diagnostics provide a useful framework for investigating simulation discrepancies. Controlled experiments, such as the Newton’s cradle gap analysis, connect motion differences to contact activation, offering explanatory value beyond aggregate trajectory errors.
Critical weaknesses
1. Insufficient validation of the physical-fidelity comparisons. Although the benchmark standardizes experimental conditions, it does not sufficiently establish whether observed discrepancies arise from simulator limitations, asset and parameter mismatches, or uncertainty in the real-world references. This limits confidence in the interpretation of its cross-platform comparisons.
2. Limited evidence for policy learning and real-world transfer. The VLA experiments do not convincingly demonstrate the relative effectiveness of training data from different simulators. The relationship between the benchmark’s physical-fidelity metrics and real-world policy performance also remains unvalidated, leaving its transfer-oriented motivation insufficiently supported by the experiments.
Questions for authors
1. Insufficient Sim-to-Real Asset and Environment Alignment. How are simulation assets and physical parameters validated against their real-world counterparts? For example, deformable objects such as bread may not be adequately represented by shared meshes, elastic-modulus priors, and zero material damping. What evidence supports these choices, and how can discrepancies caused by the simulator be distinguished from mismatches in asset and material properties?
2. Insufficient Validation of the Cross-Platform Policy Comparison. The 3,000-update policies in Section 6 exhibit command-to-nominal-path RMS errors of 96.30–129.17 mm and zero recorded approach events, raising concerns about insufficient training or data/action-interface issues. How is the nominal path constructed, and are its corresponding task configurations seen during training or held out? What evidence establishes that these policies can perform the tasks on their source platforms? Could the authors repeat the comparison using task-competent policies, such as the more successful 20,000-update policies in Appendix F, to determine whether the observed differences persist?
3. Insufficient Validation of Real-World Policy Transfer. The paper emphasizes real-world transfer, but its policy evaluations are conducted in simulation. Does “real-world transfer” refer to reproducing real physical responses or transferring learned policies to real robots? To support the latter, could the authors compare real-robot success rates of policies trained on different simulators under matched task, data-budget, and training conditions? Such an evaluation would assess the combined effects of visual and physical discrepancies and test whether closer agreement between simulated and real-world physical responses, as measured by the benchmark’s metrics, predicts better real-world policy performance.
4. Insufficient Quantification of Measurement Uncertainty and Real-World Variability. Appendix A.2 states that most rigid-body conditions have only one real recording, while repeated software restarts primarily assess numerical reproducibility. Yet the reported median pushing errors of 2.20, 2.15, 2.15, and 2.14 mm differ by at most 0.06 mm, potentially less than measurement uncertainty or real-world trial variability. How are calibration, tracking, temporal-alignment uncertainty, and trial variability reflected in the reported errors and threshold-based pass rates? What uncertainty estimates or repeated real-world measurements establish which cross-platform differences remain distinguishable?
We explain how we rebuilt assets and traced physical differences, report cross-backend and real-robot tests, and clarify how we score pushing trials.
Response to Weakness 1 on attribution of physical differences We inspect assets and effective parameters, check inputs and timestamps, locate divergence in contact, constraints and motion, then change one factor and check against independent mechanics calculations. With fixed inertia and input, eccentric-hinge endpoint deviation stayed near 47.7% as iterations rose from 10 to 400. Changing linear/angular relaxation from 0.7/0.4 to 0.7/0.7 reduced it to -0.326%. This intervention locates the discrepancy in the tested solver configuration. Section 5 and Appendix D.3 will list controls, interventions, continuous readings and scope.
Response to Weakness 2 on learning and transfer evidence Same-platform training and execution test learning; frozen model weights across execution backends test execution conditions. Appendix F's 20,000-update group completed 13/18 platform-task combinations. Section 6's pre-contact failures locate policy-output deviations. Real-robot tests align future prediction, currently adopted target and measured feedback. Our focus is process-level diagnosis; training-source effects are interpreted by task, budget and execution conditions. Section 6 and Appendix F will list these by model group.
Response to Question 1 on deformable asset alignment Scans and photographs establish bread geometry; repeated pressing and unloading with tactile and depth acquisition record force response, deformation profiles and recovery for adjusting and checking soft-body parameters. Haply provides an additional force-feedback interaction check. Shared 40/200 kPa values and zero material damping are engineering settings. Zero material damping adds no internal material damping term; contact, friction, constraints and numerical integration still affect response. This compares platforms under declared conditions. Effects on rebound, recovery time and dissipation are considered separately against physical process records. Appendices C/H will connect acquisition and reconstruction and list parameter sources.
Response to Question 2 on training budget and the nominal path Section 6 reports 96.30-129.17 mm command-to-nominal-path errors and zero approach events in 72 episodes. The path uses a frozen teacher's geometry and smoothstep interpolation; evaluation seed 10000 is outside TRAIN/VAL. The 3,000-update budget is per round, with weights carried through three rounds: 9,000 updates. Additional same-platform evaluation completed 20/30: 4, 4, 3, 4, 3 and 2 out of 5 for Isaac, MuJoCo, Bullet, Drake, Newton and Gazebo. Cross-platform evaluation of frozen older models completed 100/150. This group is separate from Appendix F's 20,000-update group. Section 6 and F.1/F.5 will list budgets, path construction, held-out indices and results.
Response to Question 3 on real-robot transfer tests We deployed simulation-trained policies on the real robot. Measured feedback and the currently adopted target reached wrist-height peaks of 79.01 and 78.97 mm in a lifting episode: independent peaks over the last ten queries relative to the lowest wrist position after closure. Measured feedback records arm lift; visual records confirm block lift. On 23 October 2026, six models each underwent three real-robot tests. All 18 performed approach and gripper closure; nine completed grasping and lifting, with the block leaving the table. Grasping and lifting were completed in 1/3, 3/3, 0/3, 3/3, 0/3 and 2/3 tests for Isaac, MuJoCo, Bullet, Drake, Newton and Gazebo: 9/18 . Closure and effective grasp are assessed separately; completion requires the block to leave the table with the gripper. New training/simulation results are listed by checkpoint; real-robot stage rates describe deployed models. The 150 simulation trials and 18 real-robot tests provide stage diagnosis. Section 6 and F.5 will list models, conditions, stages, measured feedback and simulation comparison criteria.
Response to Question 4 on the 0.06 mm difference For each physical pushing condition, all six platforms replayed the same recorded probe trajectory and used the same tracked block trajectory for scoring. The eight clips span different materials and shapes. Within-condition physical variability was not quantified. We do not rank the four platforms by the 0.06 mm spread between their median errors. Figure 4 scores 72 conditions per platform using a tolerance of max(2 mm, 5% of the reference displacement scale). The 0.06 mm spread does not enter scoring. We will report margins to the threshold, repeat selected physical trials, quantify calibration, tracking, and timing errors, and recompute pass rates.
Evaluation code, task configurations, numerical records, scene assets and reproduction instructions have been released on GitHub https://github.com/hoiyaipac4-lab/PhysTrace.