acceptodds
Under review as a conference paper at ICLR 2027

When the Simulator Fails, Not the Policy: Auditing Contact Artifacts in Simulated Robot Benchmarks

Abstract

Simulation benchmarks score robot policies by success rate, and every failed episode counts against the policy. Testing a vision language action model and a world action model in simulation, we find that failures in which the policy has placed the object and the simulator then ejects it are frequent, and that the benchmark's own scripted demonstrations fail the same way. We present SimuGuard, an auditing plugin for LIBERO, RoboTwin, RoboCasa and RoboDojo. It leaves the evaluation unchanged. At every physics step, it records the state and flags an object that starts moving at a contact faster than the robot's contact or gravity account for. It replays a recorded episode bit for bit, and it reruns the episode with one solver parameter changed. With it we trace each such episode step by step to the contact solver's response at a resting contact: within one physics step the solver reports an overlap of about two centimeters between the placed object and the thin wall of its container, removes the whole overlap in that same step, and the separation speed throws the object out at several meters per second. In SimuGuard's controlled reruns, a limit on the separation speed slows the ejection and lets most of the flagged failures pass. The artifacts change the score, and every way of scoring them rests on an assumption. Reported rates assume every ejection is the policy's fault. Replaying the recorded actions with the separation speed limited on every body from before each artifact assumes those actions remain the policy's choice once the ejection is gone. Excluding the flagged episodes assumes they are as hard as the rest. Running the policy closed loop with the limit on for the whole rollout assumes the limit changes nothing the task needs, and our data refute this one: it lowers the score of a task the artifacts barely touch. SimuGuard lets a benchmark separate what the policy did from what the solver did.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot manipulation, policy evaluation, simulation benchmarks, physical validity, contact dynamics, benchmark auditing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HxnnOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduce the explosion problem with physics simulation in the context of simulated robotics benchmarks.

Critical strengths

The paper introduces a solution to the explosion problem, where it looks for specific huge deltas with manipulated objects and it limits the explosion.

Critical weaknesses

To me this paper is missing a key benchmark, robolab, which since its publication in June 2026 is now the standard for evaluating robotics policies. Moreover the explosion problem is a well known problem and thus not necessarily novel even though the applications to robotics simulation is important. I would have liked the authors to address this problem directly into the simulation rather than introducing an extra layer. Moreover the paper is hard to follow, the abstract and introduction need considerable amount of work.

Questions for authors

Here are my raw notes as I was reading.

ICLR 12374 Why is robolab not there? Ah not even cited! Three pages for an intro? Already I feel like without robolab it is hard to take this work seriously, I know it is built on top of PhysX or newton. “Testing a vision language action model and a world action model in simulation, we find that failures in which the policy has placed the object and the simulator then ejects it are frequent, and that the benchmark’s own scripted demonstrations fail the same way.” hard to parse. Ejecting like it is an airplane pilot or rejecting the condition for success? abstract is quite hard to follow and read. So I think the problem is physics explosion. The abstract does not serve its purpose well, I did not get some of the main findings, I did not read anything regarding how it improved things, just the problem and its technical solution.

Introduction. I think the problem observed is physics explosion or clipping explosion. This happens when clipping occurs in rigid body simulation. It is a well known problem in simulation and authors should refer to it as a known problem not as a surprise. The second paragraph should focus on related work at high level and what you are proposing as a solution. there is a lot of numbers for an intro, keep that for the experiments please. So reading the intro the authors propose a solution to the physics explosion problem.

I like the contact simulation in the related work.

7 comments
AuthorsOpenReview ↗

Response to Reviewer Hxnn

Thank you for your review. We reply to each point below. All changes in the revised PDF are in blue.

W1. RoboLab is missing.

We agree that RoboLab is now a standard benchmark for robot policies, and we apologize for missing it. The revised paper cites RoboLab and evaluates on it: two policies, each on 120 tasks × 20 episodes, 4,800 episodes in total. Success rates before and after the correction with SimuGuard and a human check:

PolicyEpisodesReportedCorrected
Pi0.5-DROID2,40028.1%28.5%
GR00T N1.7-DROID2,4008.8%9.1%

SimuGuard can now identify and detect three classes of artifacts:

ArtifactWhat happens
Abnormal collisionAn object touches another body and suddenly flies off at high speed
Object stuck to the gripperThe gripper is open, but the object still sticks to the fingers and moves with them
Passing throughThe object passes into the gripper

Videos of one example of each class are in the supplementary material.

With RoboLab, we cover 5 benchmarks on the two main physics engines, PhysX and MuJoCo: RoboLab, RoboTwin 2.0, RoboDojo, LIBERO and RoboCasa. All 5 show artifacts.

W2. Physics explosion is a well-known problem, so the work is not novel.

An object flying off in simulation, often called a physics explosion, is a known effect. Building on this, we measure how often it happens in policy evaluation and how much it changes the scores, and we find its cause in the engine. Prior work compared with this paper:

Prior workWhat it showsWhat this paper adds
PhysX docs (NVIDIA 2024)Overlapping bodies get pushed apartContacts are also reported when nothing overlaps
Erez et al. 2015; Acosta et al. 2022; Le Lidec et al. 2023; PhyFu (Xiao et al. 2023); PhysiXFails (Li et al. 2025)Engine contacts are not accurate; errors can be found automaticallyWe check every episode of a benchmark run
Baker et al. 2020; Krakovna's specification gaming list; Afzal et al. 2021RL agents exploit simulator bugsHow often errors occur in evaluation and how much they change scores
Jiang et al. 2026; LIBERO-PRO; RoboEvalQuestion what benchmark scores measureWhether the result of each episode can be trusted

W3. The problem should be fixed in the simulator itself, not with an extra layer.

We agree. After the submission, we went on to fix the simulator itself, and the revised paper reports this. Take RoboTwin as an example. We used SimuGuard to find the source of the false contact in three steps:

  1. Read the log. In the 35 ejections on Place Can Basket, the can and the basket wall are still apart one step before the event, by 15 mm at the median. At the next step, the engine reports that they overlap. The objects never went into each other; the engine got the contact wrong.
  2. Redo one step with one change. We go back to one step before the event and redo only that step, changing one setting at a time. With the contact cache off, or with a smaller contact distance, the false contact is gone. So the problem is in how the engine computes contacts.
  3. Reproduce it alone. We give the recorded object poses to the PhysX contact code outside the simulator and get exactly the same false contact. We then follow the code to the cause.

The cause: when PhysX computes the distance between two shapes, it uses a fixed threshold, 2^-23, to decide whether a triangle is degenerate, that is, so flat that it is almost a line. This value is too large. Normal triangles on thin container walls are treated as degenerate, and the engine reports a contact that does not exist.

The PhysX docs limit how fast bodies are pushed apart. The false contact stays, and objects just fly off more slowly. Our fix only makes the threshold above seven orders of magnitude smaller, and the false contact is gone. The speed limit is tested on Place Object Basket, and our fix on five tasks:

FixPolicyEpisodes with artifactsSuccess
PhysX docs: speed limitLingBot-VA2/20 → 1/2090% → 50%
SimuGuard: smaller thresholdLingBot-VA12/100 → 0/10089% → 98%
SimuGuard: smaller thresholdHyVLA10/100 → 1/10090% → 94%

W4. The abstract and introduction are hard to follow.

We rewrote both. The abstract now gives the main results: the cause of the artifacts, the fix, and the gain after the fix. We replaced "ejects" with "throws it out". The introduction now says that such ejections, often called physics explosions, are a known effect of rigid body solvers, and most of its numbers moved to the experiments. We will keep polishing the whole paper.

Dear Reviewer Hxnn,

Thank you again for your review. We ran two policies on the 120 RoboLab tasks, 4,800 episodes in total. We also fixed the simulator itself by making a threshold in PhysX's collision detection smaller: LingBot-VA rises from 89% to 98%, and no episode has an artifact.

If you have other questions, we are happy to discuss them.

Thank you for looking into my comments into such details. "numerical instability in physics simulation" is the problem name, I did a little bit more research on it using chatgpt, I only prompt about the problem no information about your submission was shared.

in soft body simulations https://publications.ri.cmu.edu/storage/publications/pub_files/pub1/baraff_david_1998_1/baraff_david_1998_1.pdf https://dl.acm.org/doi/10.1145/311535.311548

in rigid bodies https://cims.nyu.edu/gcl/papers/2021-RigidIPC.pdf "guarantees intersection-free configurations at every time step." I think the proposed benchmark here would be interesting to check against your proposed method.

https://www.cs.ubc.ca/labs/sensorimotor/projects/sp_sigasia08/ "accurate frictional contact resolution"

I am inclined to increase my score. I appreciate the effort that was put into the paper. Sorry for the late reply, CVPR took a lot of my attention. But I feel like the body of work that was put into rigid body simulation is very present, I am not an expert but I feel like a better coverage would help the language of the paper. I just did a 10 min research and found older relevant work, I am specially curious about the benchmark as well.

Reviewer_uZ2mOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how contact artifacts in physics simulators can confound the evaluation of robot policies. It introduces SIMUGUARD, an auditing framework that records physics-step states and controls, detects candidate artifacts, reproduces recorded executions, and performs controlled reruns with modified solver settings. The main experiments identify abrupt object ejections in RoboTwin container tasks, including failures of the benchmark’s scripted expert. Limiting depenetration velocity during replay changes LingBot-VA’s aggregate success rate on five affected tasks from 82.3% to 91.8%. The paper distinguishes reported, replay-based, exclusion-based, and modified closed-loop scores, explaining their different assumptions, and presents additional audits on LIBERO, RoboCasa, and RoboDojo.

Critical strengths
  1. A consequential benchmark failure mode supported by multiple forms of evidence. The paper connects physics-step traces, unusually large velocity changes, controlled parameter interventions, and episode outcomes. Observing similar failures in the benchmark’s ground-truth scripted expert is particularly valuable: it shows that the phenomenon is not specific to the learned policies. This makes the work relevant to how the community interprets simulation benchmark results.

  2. A useful auditing workflow with substantial attention to reproducibility. Recording actuation and simulator state, validating exact replay, and retaining the benchmark’s success check make the analysis more actionable than a collection of failure videos. The authors also report successful episodes that become failures under intervention and cases where closed-loop performance deteriorates. This balanced reporting strengthens the empirical contribution.

Critical weaknesses
  1. The interpretation of the replay score remains limited. The intervention starts 0.1 seconds before the detected artifact, affects every dynamic body and articulation link, and remains active until termination, while recorded actions are held fixed. Outcome changes therefore cannot be attributed exclusively to suppressing the identified artifact. The authors acknowledge these limitations in §5, but they constrain whether the replay score estimates policy performance under artifact-free physics.

  2. Detector error rates are not fully characterized across the benchmark audits. The 37/40 precision, hard-negative evaluation, and 58/60 agreement across six additional tasks provide useful validation. However, they do not establish recall or comprehensive false-positive and false-negative rates. This matters because missed artifacts and misclassified episodes can affect the reported score intervals and replay estimates.

Questions for authors
  1. Can the existing recordings help distinguish the effect of removing the detected ejection from subsequent effects of the persistent, all-body intervention? Would limiting the intervention to a short window, or applying matched interventions to unflagged episodes, materially change the replay-score conclusions?

  2. What evidence is available about missed artifacts beyond agreement on selected clips? How sensitive are the reported score intervals and replay estimates to plausible detector false-negative and false-positive rates?

1 comment
AuthorsOpenReview ↗

Response to Reviewer uZ2m

Thank you for your review. We reply to each point below. All changes in the revised PDF are in blue.

W1 and Q1. The replay intervention acts on every body until the end, so the outcome changes cannot be attributed to the artifact alone. Would a short window, or the same intervention on unflagged episodes, change the conclusions?

Thank you for both suggestions. We ran an experiment for each.

  1. Shorter intervention: the original replay limits every body until the episode ends. The new replay limits only the ejected object to 1 m/s, and lifts the limit after 0.3 s. Success on the five tasks:
PolicyReportedOriginal replayNew replay
LingBot-VA83.191.987.3
HyVLA90.093.091.0

Two other short versions: also limit the body the object touched, which gives 88.1 for LingBot-VA and 90.0 for HyVLA; or limit the object only until the two bodies no longer overlap, which gives 86.2 and 93.0.

The new replay is still above the reported score, so the artifacts cost the policy points. It is below the original replay, so part of the original gain came from the intervention itself. The revised paper uses the new replay.

  1. Control: apply the same short intervention to unflagged episodes, and count the episodes whose outcome changes:
PolicyFlaggedUnflagged
LingBot-VA31/590/113
HyVLA7/100/20

No unflagged episode changes, so the limit itself does not change outcomes. The changes in flagged episodes come from removing the artifact.

W2 and Q2. Detector misses and false flags are not fully measured. How sensitive are the scores to them?

How many misses? We checked unflagged episodes by hand:

BenchmarkCheckedMissedCause
RoboLab1001An object stuck to the gripper; SimuGuard now detects this
RoboDojo1000—

On RoboTwin, we checked misses in another way. We replayed all 260 LingBot-VA episodes on the five tasks with their recorded actions, in the engine with the smaller threshold. Of the 201 unflagged episodes, 4 failed, and 3 of these succeed in the replay. So at most 3 unflagged episodes, 1.5%, are missed artifacts that changed the outcome.

How many false flags? We checked flagged episodes by hand:

BenchmarkCheckedReal artifacts
RoboLab10026%
RoboDojo10030%

Most false flags are objects falling at the start of an episode.

How much can detector errors shift the scores? The paper's scores are based on the human check. On each benchmark we checked 100 flagged episodes, and each kind of flag counts at the share of real artifacts we found. If we removed all flagged episodes without this check, false flags would raise the scores by at most 1.4 points:

BenchmarkPolicyReportedSimuGuard onlyWith human checkDifference
RoboLabPi0.5-DROID28.129.928.51.4
RoboLabGR00T N1.7-DROID8.89.29.10.1
RoboDojoInternW039.741.840.61.2
RoboDojoOpenWAM19.421.219.81.4
RoboDojoX-VLA15.817.516.70.8
RoboDojoPi0.517.117.617.00.6

With the measured miss and false-flag rates, the final RoboLab success rate is between 26.8 and 32.6 for Pi0.5, and between 8.7 and 12.7 for GR00T. Without the human check, the ranges are 23.8–44.4 and 8.2–19.4. Halving or doubling the detector threshold keeps the scores within a range of 1.6 points.

Reviewer_WvHdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SimuGuard, a tool that audits robot simulations for contact artifacts, where the physics solver rather than the policy causes a failure during evaluation. The authors audit benchmarks built on PhysX and MuJoCo, validate their detector against human judgments, and trace the artifacts to the contact solver through controlled reruns. They then propose re-scoring a benchmark by replaying each flagged episode with the artifact removed, and compare this with alternative scoring rules.

Critical strengths
  • The finding of a real and common problem in the simulated robot environment is convincing and valuable.
Critical weaknesses
  • W1: The scope is narrow. It covers one artifact class in two families of simulators, and didn't reason about if the same problem could appear in simulators in general.

  • W2: There is no evidence that SimGuard is more accurate compared to other baselines. The conclusion of policy ranking shift after applying SimGuard should be grounded by showing an improvement in policy ranking of real world performance.

Questions for authors
  • Q1: Can you show that Simguard improves policy evaluation, e.g., by comparing how well the reported and corrected scores rank policies against their real-world performance on RoboDojo’s sim-and-real tasks?
8 comments
AuthorsOpenReview ↗

Response to Reviewer WvHd (1/2)

Thank you for your review. We reply to each point below, in two comments. All changes in the revised PDF are in blue.

W1. The scope is narrow: one artifact class in two families of simulators, and no reasoning about simulators in general.

We study three artifact classes on two physics engines, PhysX and MuJoCo. All 5 benchmarks we tested show artifacts: RoboLab, RoboTwin 2.0, RoboDojo, LIBERO and RoboCasa.

Most robot manipulation simulators are built on one of these two engines:

EngineSimulatorsMain benchmarksTested by usArtifacts found
PhysXSAPIEN, Isaac SimRoboLab, RoboTwin, ManiSkill, SimplerEnv, RoboDojo, BEHAVIOR-1KRoboLab, RoboTwin 2.0, RoboDojoYes
MuJoCoMuJoCo, robosuiteLIBERO, RoboCasa, robosuite, Meta-WorldLIBERO, RoboCasaYes

The three artifact classes SimuGuard detects and corrects:

ArtifactWhat happens
Abnormal collisionAn object touches another body and suddenly flies off at high speed
Object stuck to the gripperThe gripper is open, but the object still sticks to the fingers and moves with them
Passing throughThe object passes into the gripper

Videos of one example of each class are in the supplementary material.

Success rates before and after the correction with SimuGuard and a human check:

BenchmarkPolicyEngineEpisodesReportedCorrected
RoboTwin 2.0LingBot-VAPhysX2,06090.8%93.0%
RoboTwin 2.0HyVLAPhysX22078.2%80.0%
RoboDojoXR-1PhysX4040.0%48.5%
RoboDojoInternW0PhysX36039.7%40.6%
RoboDojoOpenWAMPhysX36019.4%19.8%
RoboDojoPi0.5PhysX36017.1%17.0%
RoboDojoX-VLAPhysX36015.8%16.7%
RoboDojoXiaomi-Robotics-0PhysX3605.2%5.3%
RoboDojoPi0PhysX3603.6%3.7%
RoboDojoInternVLA-A1PhysX3602.1%2.1%
RoboDojoGR00T N1.7PhysX3601.3%1.3%
RoboLabPi0.5-DROIDPhysX2,40028.1%28.5%
RoboLabGR00T N1.7-DROIDPhysX2,4008.8%9.1%
LIBEROPi0.5MuJoCo40095.2%95.5%
RoboCasaXR-1MuJoCo30061.7%62.7%

Why the change differs across rows. An artifact changes the success rate only when it breaks an episode that would have succeeded. So the change depends on how many artifacts there are and which episodes they hit. A weak policy fails mostly on its own, so its rate barely moves, e.g. GR00T N1.7. A strong policy is more often close to success, so its rate moves more, e.g. LingBot-VA gains 2.2 points. Artifacts also occur in successful episodes, e.g. an object gets a sudden push but the task still succeeds. We remove these episodes too, because we cannot tell whether they would succeed without the artifact. So the rate can drop slightly, e.g. Pi0.5 on RoboDojo drops 0.1 points.

Both engines show artifacts. Every rigid-body engine does two steps. First, collision detection uses a tolerance, and a poor tolerance reports contacts that do not exist. Second, the engine pushes overlapping bodies apart. PhysX does this in one step and MuJoCo over several steps, so objects fly off faster in PhysX. In the same scene with 20 mm of overlap, the object leaves at 4.20 m/s in PhysX and 1.05 m/s in MuJoCo. The detector reads only the body states and contacts at each step, so a new engine needs only an adapter. We support SAPIEN, Isaac Sim and MuJoCo.

AuthorsOpenReview ↗

Response to Reviewer WvHd (2/2)

W2. There is no evidence that SimuGuard is more accurate than other baselines.

Each simple rule looks at one physical quantity, such as speed or penetration depth, and flags an episode when the value passes a threshold. All methods flag the same number of episodes. We score them on the 200 RoboLab episodes checked by hand:

DetectorPrecisionRecallF1
SimuGuard26%84%39%
Max speed20%65%30%
Velocity jump19%62%29%
Penetration depth13%34%19%

SimuGuard has the highest precision, recall and F1.

W2 and Q1. Does the correction improve policy evaluation? Compare how the reported and corrected scores rank policies against real-world performance on RoboDojo.

The correction does not change the order of the 8 models, and the Spearman correlation with real success stays 0.81. But SimuGuard still corrects the scores: X-VLA, for example, goes from 15.8% to 16.7%. It also finds and fixes the error in the engine: after the fix on RoboTwin, LingBot-VA's closed-loop success rises from 89% to 98%. RoboDojo, 8 models, on the 7 tasks found both in sim and on the real robot, sorted by real success. XR-1 has no public real-robot results, so it is not in the table.

PolicyReportedCorrectedReal robot
InternW039.7%40.6%31.7%
OpenWAM19.4%19.8%24.4%
Pi0.517.1%17.0%12.8%
InternVLA-A12.1%2.1%7.2%
Xiaomi-Robotics-05.2%5.3%3.9%
X-VLA15.8%16.7%3.3%
GR00T N1.71.3%1.3%1.7%
Pi03.6%3.7%1.7%

Dear Reviewer WvHd,

Thank you again for your review. We extended the study to three artifact classes on 5 benchmarks and two engines. We compared SimuGuard with three simple detectors on RoboLab. We also compared the reported and corrected scores of 8 models with their real-robot results on RoboDojo.

If you have other questions, we are happy to discuss them.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank the three reviewers for their comments. Main changes in the revised PDF, all in blue:

  1. Additional experiments: 10,261 episodes in total, 4,800 of them on RoboLab, where two policies each run 120 tasks × 20 episodes.
  2. We use SimuGuard to locate and analyze the cause of the artifacts in the physics engine, and fix the simulator.
  3. We make the replay correction shorter and add a control, and we compare the corrected scores with real-robot results.
  4. We measure the misses and false flags of the detector and compare it with other detectors.
  5. We revised the writing to make the paper easier to read.
BenchmarkAdditional experimentEpisodes
RoboLabTwo policies, 120 tasks × 20 episodes each4,800
RoboDojo8 models on the 7 tasks with real-robot results2,880
RoboTwin 2.0Short replays and the control2,121
RoboTwin 2.0Replays with the smaller threshold260
RoboTwin 2.0Closed-loop reruns with the smaller threshold200
Total10,261

Videos of each artifact class are in the supplementary material.

Thank you for reading our response and for confirming that it addressed your main concerns. If any smaller point remains, we are happy to answer it.

AuthorsOpenReview ↗

Summary of the discussion and added experiments

Dear Area Chair and Reviewers,

We thank the Area Chair and all three reviewers for the discussion. It made the paper stronger. As the discussion closes, we give a short summary of the main concerns and what we added for each. All of it is in the revised PDF, with changes marked in blue.

1. Whether the correction matters for ranking policies. We tested five models in RoboTwin, in RoboTwin repaired with SimuGuard, and on a real dual-arm Piper-X robot.

PolicyRoboTwinRoboTwin repaired with SimuGuardReal robot
LingBot-VA89/10098/10024/30
InternVLA-A186/10098/10023/30
HyVLA90/10094/10020/30
LingBot-VLA82/10095/10021/30
StarVLA60/10071/10010/30

In RoboTwin, HyVLA comes first, although it is fourth on the real robot. Once RoboTwin is repaired with SimuGuard, the five models fall into the same order as on the real robot.

2. The problem should be fixed in the simulator. SimuGuard records the step where the error starts, and with it we traced the false contact to one threshold in the collision code and repaired it. In the repaired engine LingBot-VA rises from 89 to 98 of 100, and no episode has an artifact.

3. Relation to known work. Section 2 now places the error as a case of numerical instability and cites the suggested works. The appendix runs the Rigid IPC unit tests in three simulators with SimuGuard attached. Our part is to measure how the error changes benchmark scores and to find its cause in a released benchmark. We see this as a step toward simulators that agree with the real world, which matters more as more policies are trained and tested in simulation.

4. RoboLab. We added this benchmark with two policies on all 120 tasks, 20 episodes per task, 4800 episodes in total.

We are grateful for the time that the reviewers and the Area Chair gave to this paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.