acceptodds
Under review as a conference paper at ICLR 2027

PhysTrace: A Standardized Benchmark for Evaluating Sim-to-Real Physical Fidelity

Abstract

Physical simulation provides scalable interaction environments for robot learning, yet understanding how physical fidelity affects the transfer of robot policies to the real world requires systematic investigation through comparable physical experiments, real-world observations, and task evaluations. We establish a benchmark for evaluating physical simulation quality with a focus on real-world transfer. Using six simulation platforms as experimental testbeds, the benchmark integrates fundamental physics experiments, everyday interaction tasks, and real-world experimental comparisons into a hierarchical evaluation framework. It establishes consistent experimental procedures for asset representation, physical properties, initial conditions, loading protocols, and observation specifications. Platform adaptation, native configuration verification, and temporal alignment of data ensure reproducible experimental conditions and traceable data pipelines, while explicitly documenting differences in physical implementations and parameter semantics across platforms. Through multifactor experimental matrices, controlled single-variable comparisons, and paired simulation–real-world comparisons, we evaluate the accuracy, stability, and parameter sensitivity of physical responses. We further use process data from control, contact, motion, and sensing to locate the stages at which discrepancies arise and analyze their propagation mechanisms. Under the tested conditions, agreement between simulation and real-world observations depends on the task, interaction stage, and configuration, with platforms exhibiting distinct responses during frictional transitions, transient collisions, and support transitions. Process observations and controlled comparisons provide evidence of how initial states, native numerical configurations, and contact responses contribute to some of these discrepancies. Building on these findings, supplementary short-horizon manipulation experiments with vision-language-action policies examine the relationships among physical discrepancies, task performance, and real-world transfer. The benchmark reveals that simulation fidelity is task- and stage-dependent: agreement in overall responses can coexist with discrepancies in critical interaction processes. This finding offers an important perspective on real-world transfer performance and supports a shift in simulation-based training for embodied models, from evaluating overall realism toward targeted calibration of task-critical physical processes.

Primary area:
datasets and benchmarks
Keywords:
Physics simulation, Standardized benchmarking, Cross-platform evaluation, Sim-to-real transfer, Contact dynamics, Mechanism attribution.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_X7mmOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces PhysTrace, a benchmark for evaluating physical simulation quality across six platforms. It standardizes experimental conditions and observation protocols, combines controlled experiments with real-world references, and uses process-level measurements to investigate discrepancies. Experiments cover rigid-body dynamics, deformable objects, and articulated mechanisms, alongside supplementary VLA learning and cross-platform execution studies. The findings highlight that simulation fidelity depends on the task and interaction stage, and that similar overall trajectories can conceal differences in contact processes.

Critical strengths

The standardized six-platform evaluation and process-level diagnostics provide a useful framework for investigating simulation discrepancies. Controlled experiments, such as the Newton’s cradle gap analysis, connect motion differences to contact activation, offering explanatory value beyond aggregate trajectory errors.

Critical weaknesses

1. Insufficient validation of the physical-fidelity comparisons. Although the benchmark standardizes experimental conditions, it does not sufficiently establish whether observed discrepancies arise from simulator limitations, asset and parameter mismatches, or uncertainty in the real-world references. This limits confidence in the interpretation of its cross-platform comparisons.

2. Limited evidence for policy learning and real-world transfer. The VLA experiments do not convincingly demonstrate the relative effectiveness of training data from different simulators. The relationship between the benchmark’s physical-fidelity metrics and real-world policy performance also remains unvalidated, leaving its transfer-oriented motivation insufficiently supported by the experiments.

Questions for authors

1. Insufficient Sim-to-Real Asset and Environment Alignment. How are simulation assets and physical parameters validated against their real-world counterparts? For example, deformable objects such as bread may not be adequately represented by shared meshes, elastic-modulus priors, and zero material damping. What evidence supports these choices, and how can discrepancies caused by the simulator be distinguished from mismatches in asset and material properties?

2. Insufficient Validation of the Cross-Platform Policy Comparison. The 3,000-update policies in Section 6 exhibit command-to-nominal-path RMS errors of 96.30–129.17 mm and zero recorded approach events, raising concerns about insufficient training or data/action-interface issues. How is the nominal path constructed, and are its corresponding task configurations seen during training or held out? What evidence establishes that these policies can perform the tasks on their source platforms? Could the authors repeat the comparison using task-competent policies, such as the more successful 20,000-update policies in Appendix F, to determine whether the observed differences persist?

3. Insufficient Validation of Real-World Policy Transfer. The paper emphasizes real-world transfer, but its policy evaluations are conducted in simulation. Does “real-world transfer” refer to reproducing real physical responses or transferring learned policies to real robots? To support the latter, could the authors compare real-robot success rates of policies trained on different simulators under matched task, data-budget, and training conditions? Such an evaluation would assess the combined effects of visual and physical discrepancies and test whether closer agreement between simulated and real-world physical responses, as measured by the benchmark’s metrics, predicts better real-world policy performance.

4. Insufficient Quantification of Measurement Uncertainty and Real-World Variability. Appendix A.2 states that most rigid-body conditions have only one real recording, while repeated software restarts primarily assess numerical reproducibility. Yet the reported median pushing errors of 2.20, 2.15, 2.15, and 2.14 mm differ by at most 0.06 mm, potentially less than measurement uncertainty or real-world trial variability. How are calibration, tracking, temporal-alignment uncertainty, and trial variability reflected in the reported errors and threshold-based pass rates? What uncertainty estimates or repeated real-world measurements establish which cross-platform differences remain distinguishable?

1 comment
AuthorsOpenReview ↗

We explain how we rebuilt assets and traced physical differences, report cross-backend and real-robot tests, and clarify how we score pushing trials.

Response to Weakness 1 on attribution of physical differences We inspect assets and effective parameters, check inputs and timestamps, locate divergence in contact, constraints and motion, then change one factor and check against independent mechanics calculations. With fixed inertia and input, eccentric-hinge endpoint deviation stayed near 47.7% as iterations rose from 10 to 400. Changing linear/angular relaxation from 0.7/0.4 to 0.7/0.7 reduced it to -0.326%. This intervention locates the discrepancy in the tested solver configuration. Section 5 and Appendix D.3 will list controls, interventions, continuous readings and scope.

Response to Weakness 2 on learning and transfer evidence Same-platform training and execution test learning; frozen model weights across execution backends test execution conditions. Appendix F's 20,000-update group completed 13/18 platform-task combinations. Section 6's pre-contact failures locate policy-output deviations. Real-robot tests align future prediction, currently adopted target and measured feedback. Our focus is process-level diagnosis; training-source effects are interpreted by task, budget and execution conditions. Section 6 and Appendix F will list these by model group.

Response to Question 1 on deformable asset alignment Scans and photographs establish bread geometry; repeated pressing and unloading with tactile and depth acquisition record force response, deformation profiles and recovery for adjusting and checking soft-body parameters. Haply provides an additional force-feedback interaction check. Shared 40/200 kPa values and zero material damping are engineering settings. Zero material damping adds no internal material damping term; contact, friction, constraints and numerical integration still affect response. This compares platforms under declared conditions. Effects on rebound, recovery time and dissipation are considered separately against physical process records. Appendices C/H will connect acquisition and reconstruction and list parameter sources.

Response to Question 2 on training budget and the nominal path Section 6 reports 96.30-129.17 mm command-to-nominal-path errors and zero approach events in 72 episodes. The path uses a frozen teacher's geometry and smoothstep interpolation; evaluation seed 10000 is outside TRAIN/VAL. The 3,000-update budget is per round, with weights carried through three rounds: 9,000 updates. Additional same-platform evaluation completed 20/30: 4, 4, 3, 4, 3 and 2 out of 5 for Isaac, MuJoCo, Bullet, Drake, Newton and Gazebo. Cross-platform evaluation of frozen older models completed 100/150. This group is separate from Appendix F's 20,000-update group. Section 6 and F.1/F.5 will list budgets, path construction, held-out indices and results.

Response to Question 3 on real-robot transfer tests We deployed simulation-trained policies on the real robot. Measured feedback and the currently adopted target reached wrist-height peaks of 79.01 and 78.97 mm in a lifting episode: independent peaks over the last ten queries relative to the lowest wrist position after closure. Measured feedback records arm lift; visual records confirm block lift. On 23 October 2026, six models each underwent three real-robot tests. All 18 performed approach and gripper closure; nine completed grasping and lifting, with the block leaving the table. Grasping and lifting were completed in 1/3, 3/3, 0/3, 3/3, 0/3 and 2/3 tests for Isaac, MuJoCo, Bullet, Drake, Newton and Gazebo: 9/18 . Closure and effective grasp are assessed separately; completion requires the block to leave the table with the gripper. New training/simulation results are listed by checkpoint; real-robot stage rates describe deployed models. The 150 simulation trials and 18 real-robot tests provide stage diagnosis. Section 6 and F.5 will list models, conditions, stages, measured feedback and simulation comparison criteria.

Response to Question 4 on the 0.06 mm difference For each physical pushing condition, all six platforms replayed the same recorded probe trajectory and used the same tracked block trajectory for scoring. The eight clips span different materials and shapes. Within-condition physical variability was not quantified. We do not rank the four platforms by the 0.06 mm spread between their median errors. Figure 4 scores 72 conditions per platform using a tolerance of max(2 mm, 5% of the reference displacement scale). The 0.06 mm spread does not enter scoring. We will report margins to the threshold, repeat selected physical trials, quantify calibration, tracking, and timing errors, and recompute pass rates.

Evaluation code, task configurations, numerical records, scene assets and reproduction instructions have been released on GitHub https://github.com/hoiyaipac4-lab/PhysTrace.

Reviewer_RJEnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PhysTrace compares six simulation platforms through controlled mechanics, real-motion references, and household interactions. Native configuration checks and synchronized process records support evaluations of physical response and discrepancy attribution, complemented by policy-execution studies.

Critical strengths

The framework carefully separates shared physical specifications from native implementations and aligns input, contact, and state observations. Explicit eligibility and failure denominators improve the interpretability of the comparisons.

The cradle gap controls and independent eccentric-hinge tests provide useful mechanistic evidence. Contact timing, isolated impacts, and relaxation sweeps help distinguish geometric and numerical causes beyond aggregate trajectory error.

Critical weaknesses

Real-reference validation remains uneven: most rigid conditions have one physical recording, and uncertainty in measurement and initial-state reconstruction is not propagated into the real-reference pass fractions in Figure 4 (Appendix A). Shared elastic priors and uncalibrated tactile measurements further limit quantitative validation of household deformation and contact forces (Appendix C).

The connection to policy transfer remains preliminary: the successful policies in Appendix F are evaluated on their respective training platforms, coupling learned behavior with execution physics, while the crossed cohort fails before contact. These results support stage-specific diagnosis, but the abstract’s connection to real-world transfer exceeds the demonstrated evidence.

Questions for authors

How stable are Figure 4’s real-reference pass fractions under plausible measurement and initial-state uncertainty, and under alternative screening thresholds?

Which household cases have independently measured material parameters or calibrated force/deformation references, and what quantitative simulation–reality agreement do they achieve?

Can a successful policy or fixed action sequence be evaluated across backends to test whether the identified contact discrepancies explain differences in task-stage outcomes?

2 comments
AuthorsOpenReview ↗

We address acquisition, uncertainty, threshold sensitivity, asset construction and cross-platform evaluation. Revisions will distinguish training and execution conditions. We will release the corresponding experimental protocols, data, assets and code upon publication.

Response to Weakness 1 on real-world references and uncertainty Most rigid-body conditions use one physical recording as the common six-platform reference; each tactile condition has ten independent operations. Supplementary materials contain 33 bread tactile-and-depth acquisition packages and 15 cushion pressing acquisitions. A fixed reference aligns conditions but retains measurement and initial-state errors. Figure 4's tolerance scan tests scoring boundaries; measurement-uncertainty propagation will be reported separately from the corresponding records. Scans, dimensions, pressing and depth video supply geometry and interaction references. Shared elastic settings support comparison under common conditions, distinct from individual material calibration. Uncalibrated tactile intensity retains its original units for contact-stage and relative-change analysis. Revisions will list acquisition counts, measured parameters, shared settings and ranges, with pass-rate intervals where data support them.

Response to Weakness 2 on learning and execution physics Training and execution on the same platform examine learning; switching the execution backend with frozen model weights examines execution conditions. Appendix F's 20,000-update group completed 13/18 platform-task combinations. In a stalled real-robot test, despite a future prediction of lifting, the currently adopted target and measured feedback peaked at 3.42 and 2.84 mm: independent peaks over the last ten queries relative to the lowest wrist position after closure. Low motion was already present in the adopted command, requiring consideration of action adoption and physical execution. Question 3 adds cross-platform evaluation and fixed-command comparisons. Section 6 and Appendix F will identify models, training platforms and execution backends.

Response to Question 1 on Figure 4 sensitivity We rescored the first-peak and energy columns of 72 trajectories at six tolerance levels. In MuJoCo, Bullet, Gazebo, Drake, Isaac and Newton order, first-peak pass counts change from 1/6/1/3/2/0 at 10% tolerance to 5/6/5/4/4/0 at 20% (12 per platform). Raising the energy-ratio limit from 1.05 to 1.10 changes 6/12/12/11/9/12 to 9/12/12/12/9/12. Counts change, while the scanned thresholds preserve the process-level interpretation: the controlled gap intervention's 122.57% first-peak increase, contact timing and contact topology remain observed differences. Pushing medians of 2.14-2.20 mm are treated as similar error magnitudes. A supplementary reference-perturbation bound of 0.03 mm RMS no longer guarantees their relative order; this is an analytical assumption, not measured camera error. For scale, time alignment, initial pose and velocity, the corresponding calibration and reconstruction records will define perturbation ranges for rescoring within fixed windows. Figure 4 and A.4-A.5 will list ranges, counts, denominators, continuous readings and reference conditions.

Response to Question 2 on household asset measurements Scans and photographs establish household asset geometry. Repeated pressing and unloading with tactile and depth acquisition record force response, deformation profiles and recovery for adjusting and checking soft-body parameters. Haply provides an additional force-feedback interaction check. For bread and cushion, these records support geometry and deformation checks. Force agreement remains unquantified. Appendices C/H will connect acquisition, reconstruction and checking.

Response to Question 3 on cross-platform evaluation Yes. The same frozen Newton-trained policy succeeded in 4/5 paired scenes in Gazebo and 0/5 in MuJoCo. In one pair, both backends recorded robot-object contact. Gazebo achieved a stable hold for 1.23 s, with the block bottom ending 73.4-74.0 mm above the table. In MuJoCo, the block ended near the table. This locates the difference at lift and hold. If insufficient finger-block friction contributed to the MuJoCo outcome, increasing it alone should reduce slip and improve lift and hold. Overall, 100/150 cross-platform and 20/30 same-platform trials succeeded. Success varied across policy-backend pairs.

Evaluation code, task configurations, numerical records, scene assets and reproduction instructions have been released on GitHub https://github.com/hoiyaipac4-lab/PhysTrace.

Thank you for the clarifications and additional experiments. The cross-backend and real-robot evaluations strengthen the paper, but concerns about measurement uncertainty, quantitative force validation, and the link between fidelity metrics and real-world policy performance remain. I will maintain my rating.

Reviewer_7EgMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks how physics simulators differ in physical fidelity and where those differences actually arise, with sim-to-real transfer for robot policies as the motivation. The key contribution is PhysTrace benchmark. The evidence suggests fidelity depends on the task and the interaction stage, therefore the authors conclude that the field should calibrate the physical processes critical to a task rather than pursue overall realism.

Critical strengths
  • The paper shows that matching trajectories can hide different physics. In the pushing contrast, endpoints agree while the contact forces differ. This is a useful warning against judging simulators by trajectory error alone.

  • The findings are grounded in real references across a wide range. The tasks run from elementary mechanics to household objects and compliant bodies. The paper keeps numerical replay separate from real trial-to-trial variability and keeps failed runs in the reported totals.

Critical weaknesses
  • The sim2real claim goes beyond the evidence. The title and abstract frame the work around real world transfer, but no policy is ever run on a real robot. The real references are mostly passive physics clips. The link between physical discrepancies and transfer performance is more likely to be asserted not shown.

  • Thresholds are post hoc and there is no reusable benchmark protocol. The response-map tolerances are described as organizing a retrospective interpretation, with no reason given for why each matters for a task. There is no defined task suite, metric or reporting format that another group could run on a new engine, which makes it read more as a study than a benchmark.

  • The scope is too aboard and there is no clear fit with ICLR. The paper is mainly a physics-simulation benchmarking study, and learning enters only as a supplementary component. Its many studies (rigid, soft, articulated, tactile, asset-scan, VLA and LLM) each get only brief treatment. The main text uses abstract, jargon-heavy prose and depends on a very long appendix for its evidence.

Questions for authors
  • The title and abstract center on sim2real transfer, but the policy experiments run only in simulation. The paper itself treats real-robot transfer as a separate endpoint. What claim about transfer do you think the current evidence supports?

  • Would calibrating each engine to the same real data be expected to close many of the reported gaps?

  • The findings are tied to specific engine versions and defaults. Which conclusions do you expect to reflect stable properties of the contact models, and which are specific to a configuration?

2 comments
AuthorsOpenReview ↗

We address transfer evidence, tolerances, reproducibility, scope, common real-world references and configuration dependence. We will clarify these points and release the corresponding experimental protocols, data, assets and code upon publication.

Response to Weakness 1 on real-robot transfer evidence The abstract created expectations of real-robot transfer beyond our direct evidence. Our contribution is reusable physical simulation evaluation, process-level diagnosis and controlled mechanism tests. In real-robot tests, measured feedback reached 79.01 mm of wrist lift, with the block leaving the table. In a stalled episode, the future prediction included 75.89 mm of lift; the currently adopted target was 3.42 mm. These records distinguish prediction, action adoption and measured feedback. The paper also includes active probe pushing: the measured probe trajectory supplies the input; the independently extracted block trajectory supplies the reference. The abstract and Sections 1 and 7 will reflect this scope.

Response to Weakness 2 on tolerances and a reusable protocol Appendix A.4 uses retrospective screening tolerances for distinct physical questions: support force tests static equilibrium, and contact timing characterizes interaction. We rescored the first-peak and energy columns of 72 saved trajectories at six tolerance levels. First-peak tolerances of 10% and 20% change pass counts (12 trials per platform) and some relative orders. They do not change the observed 122.57% first-peak increase under controlled gap intervention, contact timing or the contact-topology mechanism. Continuous records remain the basis for interpreting these differences. The paper specifies task conditions, input construction, metrics, eligible and failed trial counts, and process-record fields. These support native adaptation and rescoring on a new engine; Section 3 will consolidate the procedure.

Response to Weakness 3 on scope and readability We detect differences using common rules and real-world references, locate sources in process records, then examine complex interactions and policy execution. Mechanics tests mechanisms, household and deformable objects test complex contact, and policies test learned behavior. The link to ICLR is bias in simulation data used for robot learning. Terms distinguish physical objects, native implementations and observations. Sections 1 and 3 will clarify this sequence, replace selected terms with physical descriptions, and bring key continuous records into the main text.

Response to Question 1 on the supported transfer claim Our evidence supports execution diagnosis in real-robot tests, without establishing direct prediction of final success rates from simulation metrics. The same policy produced lifting and stalled episodes. In the latter, future prediction, currently adopted target and measured feedback reached independent wrist-height peaks of 75.89, 3.42 and 2.84 mm over the last ten queries. The adopted target remained low in this window. Section 6 and Appendix F.5 will compare these episodes.

Response to Question 2 on calibration to a common reference Calibration to a common real-world reference can reduce asset or parameter mismatches. We retain common physical conditions while adapting native parameter definitions and combination rules. For friction, a material coefficient can differ from the effective contact-pair coefficient; the same physical contact can therefore require different parameter expressions. We inspect loaded materials and effective contact parameters, then compare responses with the common reference. Geometry correction also uses physical measurements: Appendices A.6 and H.3 report a local scan-based PiPER finger mounting-origin correction from 135.8 to 137.5 mm, reducing RMS error on the selected scan surface from 0.642 to 0.332 mm. This supports local geometric correction, with its scope distinguished from overall six-engine calibration.

Response to Question 3 on mechanisms and configuration dependence In Bullet's Newton's cradle, changing the inter-ball gap from 0 to 0.1 mm while fixing size, mass, pendulum length and release angle changes contact activation and increases the terminal-ball first peak by 122.57%. The experiment supports the contact-activation mechanism; its magnitude belongs to that configuration. Across 12 matching native scenes in MuJoCo 3.10.0 and 3.13.0, first peaks, energy-related metrics and motion-onset times reproduced to numerical precision, with unchanged pass counts. This establishes repeatability for those scenes and environments. Versions, defaults and explicit overrides define the reported operating conditions. Section 5 will state the tested scope of each mechanism and numerical result.

Evaluation code, task configurations, numerical records, scene assets and reproduction instructions have been released on GitHub https://github.com/hoiyaipac4-lab/PhysTrace.

Thank you for your response. Some of my concerns were addressed, but it's hard for me to buy your sim2real statements. I appreciate your work, but I still think this may not be the best fit. I will keep my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.