acceptodds
Under review as a conference paper at ICLR 2027

ManiPhysicsBench: Physics-Based Assesment of Object Preservation In VLA Manipulation

Abstract

Vision-language-action (VLA) models aim to perform diverse manipulation tasks, but task success in existing rigid-body benchmarks does not indicate whether they preserve objects. We introduce ManiPhysicsZoo, which consolidates literature-supported material properties, 3D meshes, and supporting references into reusable object assets. Using these assets, a solver-based assessment computes grasp-specific damage thresholds from object geometry, material properties, and recorded grasp conditions and compares them with recorded contact forces to assess potential deformation and fracture. Building on these components, ManiPhysicsBench evaluates object preservation in LIBERO and SimplerEnv across three physics axes and three difficulty levels. Public VLA checkpoints show a substantial gap between task success and safe success, defined as task completion while preserving the object. Their gripper commands concentrate near full opening and closure, with largely similar aggregate distributions across objects, consistent with binary gripper supervision. We examine how object-specific continuous gripper labels change model behavior by retraining a VLA model. The retrained model shows more object-dependent gripping and higher safe success, but lower task success and limited generalization of object-preserving behavior.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Robot Manipulation, Benchmark, Safety

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6GxqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ManiPhysicsBench, a physics-based benchmark for evaluating object preservation in VLA manipulation. The authors build ManiPhysicsZoo, a library of 91 object assets with geometry, material properties and structural information. Using these assets, they compute grasp-specific damage thresholds, and compare these thresholds with contact forces recorded during policy rollouts to predict potential deformation or fracture. ManiPhysicsBench instantiates tasks on LIBERO and SimplerEnv across three physics axes—brittle fracture, soft object deformation, and thin-walled container deformation—each with high, medium, and low difficulty 3 levels. Evaluating a broad set of public checkpoints, the authors report a gap between task success and safe success (task completion while preserving the object). They further find that gripper commands concentrate near fully open and fully closed states, with similar distributions across objects. Finally, they retrain QwenGR00T by replacing binary closing labels with object-specific continuous targets, and observe more object-dependent grasping and higher Safe SR, but lower task success and limited generalization.

Critical strengths
  1. The paper identifies object preservation as an requirement for VLA evaluation and provides a reusable tool kit. Existing manipulation benchmarks such as LIBERO treat objects as rigid bodies and measure only task success rate, thus a policy that damages an object during a task may still be counted as successful. To address this problem, the authors construct ManiPhysicsZoo, which consolidates literature-supported material properties, meshes and structural information into traceable assets. They also construct a solver-based assessment pipeline which computes grasp-specific damage thresholds using established solvers. These are useful as shared tools for the community.
  2. The gripper analysis provides an illustration of the preservation gap, which shows that gripper command distribution of policies is nearly binary across objects with very different physical properties. This observation is consistent with the fact that most VLA pretraining datasets use binary gripper annotations. The relabeling experiment, which replaces binary closing labels with object-specific continuous targets and retrains QwenGR00T, is a well-motivated attempt to test whether post train can alleviate this problem.
Critical weaknesses
  1. Task coverage is too narrow to support the paper’s broad claims about VLA object-preservation ability, and the technical novelty beyond benchmark construction is limited. The benchmark evaluates only pick-and-place tasks, and the main evaluation involves only nine object–difficulty configurations across three physics axes which is a much smaller subset of their object library. While pick-and-place is a natural starting point, object preservation is relevant to a much wider range of manipulation behaviors: pushing, sliding, opening, inserting, tool use, and other contact-rich tasks where forces are applied in different directions and magnitudes. The paper’s conclusion that current VLAs do not preserve objects would be much stronger if it were supported by a more diverse task suite. Thus, the benchmark cannot serve as a comprehensive assessment of VLA preservation ability. Combined with the fact that the core idea—physics-based damage assessment for manipulation—is a natural extension of existing benchmarks, this limits the paper’s informativeness for readers already familiar with rigid-body evaluation.
  2. The experimental of object-specific gripper supervision is insufficient. The paper’s main experiment is a single ablation: it replaces only the closing labels of a subset of Bridge demonstrations with object-specific continuous targets, leaving arm actions and opening labels unchanged, LoRA fine-tunes QwenGR00T, and compares the result against the original public checkpoint. This comparison is confounded by multiple factors. The retrained model uses Bridge-only data, and a different training configuration than the public checkpoint which was trained on a Bridge–RT-1 mixture. The authors acknowledge this in App. E.4. More importantly, given that most VLA pretraining datasets use binary gripper annotations, the paper should include more experiments with policies fine-tuned or trained on continuous gripper data, not just a single LoRA run. Without more experiments, the claim that object-specific supervision encourages object-dependent gripping remains suggestive rather than established.
Questions for authors
  1. The object library contains 91 objects, but the benchmark uses a much smaller subset. What criteria determined which objects were selected for the benchmark, and how do you expect the results to generalize to the broader library? Would including more objects from the library change the result?
  2. Since object deformation can alter contact forces, have you conducted any experiments to quantify the gap between the rigid-body contact forces recorded during rollout and the deformable-body contact response modeled by the solver? A comparison on a small set of objects would help establish how much the solver-based assessment depends on the rigid-body rollout assumption.
  3. To support the paper’s conclusion that current VLAs do not preserve objects, the benchmark needs evidence beyond pick-and-place. Can you provide experiments on other contact-rich tasks with a wider range of manipulation behaviors, such as insertion, pushing, and tool use, where forces are applied in different directions and magnitudes? Without such evidence, the claim that VLAs fail at object preservation remains confined to a narrow task family and may not generalize to contact-rich manipulation.
  4. You acknowledged in Appendix E.4 that this experiment fails to fully isolate the effect of gripper annotation changes. Could you justify the rationality of this experimental design and demonstrate that the observed performance change exactly reflects the effects of the newly proposed gripper annotations?
  5. In the retraining experiment, the paper generates synthetic continuous gripper labels by setting the target gripping force as the midpoint of lifting force and damage threshold. While this works for objects with a narrow safe force window, for objects with a wide safe-force window, this midpoint may exceed the proper force needed for lifting, potentially encouraging unnecessarily strong grips. Could you justify this design choice, or provide a variant that uses a lower target force within the safe interval?
  6. Most of the checkpoints used in your experiments are pretrained on datasets whose gripper annotations are binary. Could you provide more experiments with checkpoints that are retrained on datasets with object-specific continuous gripper labels?
9 comments
AuthorsOpenReview ↗

Author Responses (1/5)

We thank the reviewer for the careful review and detailed questions. We address object and task coverage, gripper supervision, deformation feedback, and the lower-force labeling variant below.

W1, Q1, Q3 (1/2)

Thank you for these questions. We address the coverage concern through the object-admission criteria, additional objects from ManiPhysicsZoo, and evaluations on tool-use and insertion tasks.

Q1: why these objects were selected. Sec. 5.1 and App. B.1 (Task Admission) describe the selection procedure. A candidate must fit the robot’s gripper opening and allow the task to be completed while preserving the object within the robot’s operating range. Objects whose damage thresholds are effectively unreachable under that setting contribute little additional discrimination between task success and safe success. We therefore retain one Low-difficulty control in each physics axis and avoid filling the benchmark with redundant high-strength cases.

We admit a task only when a scripted-oracle setting achieves both task completion and a preserved-object verdict. From the eligible candidates, we select objects representative of groups with similar mechanical responses and clearly separated gripping-force windows, forming High, Mid, and Low conditions within each axis. This gives a structured comparison of preservation demands rather than an arbitrary sample of the library. We have stated the admission criterion more clearly in Sec. 5.1 and added this selection rationale to the admission procedure in App. B.1.

Additional library objects and their difficulty. We evaluated quail eggs and eggplant with the public QwenGR00T, the binary-label control, and the reported object-specific model, using 12 episodes per object and policy. The additional objects have the following difficulty indices under the definition in App. G.

Added objectDifficulty index D
Quail egg (A)0.242
Eggplant (B)0.742

Quail egg lies between the original High egg (0.357) and Mid cherry tomato (0.185). Eggplant extends the difficult end of axis B, above the original High boiled potato (0.242). The SR / Safe SR / PR results are:

PolicyQuail egg (SR / Safe SR / PR, %)Eggplant (SR / Safe SR / PR, %)
Public QwenGR00T100.0 / 16.7 / 16.791.7 / 0.0 / 0.0
Binary-label control66.7 / 8.3 / 12.541.7 / 0.0 / 0.0
Object-specific relabel (α=0.5)83.3 / 8.3 / 10.050.0 / 33.3 / 66.7

The public checkpoint completes 12/12 quail-egg tasks but preserves the object in only 2/12, and completes 11/12 eggplant tasks with no safe successes. The task-success versus preservation gap in Sec. 6.1 therefore remains on both additional objects.

The relabeled model increases preservation on eggplant, while quail-egg preservation remains low. Eggplant appears in the Bridge episodes, so its relabeling target is explicitly included, whereas the quail egg has no Bridge episode, so no target is included for it. This is consistent with the limited generalization beyond the supplied distinctions discussed in Sec. 6.3.

The original objects were chosen to form the High/Mid/Low comparison; the selection and assessment procedure itself can be applied across the Zoo library, with robot-specific feasibility determining which tasks are admitted. Applying it to these two additional objects retains the central finding in Sec. 6.1. We report the added-object results and their difficulty values in App. I.1, and Sec. 6.1 refers to them.

AuthorsOpenReview ↗

Author Response (2/5)

W1, Q1, Q3 (2/2)

Q3: tool-use and insertion tasks. The paper targets pick-and-place tasks and judges damage that arises while the object is grasped. To examine whether the same resources apply beyond this setting, we applied the Zoo assets and the solver-based assessment to tool-use and insertion tasks in three existing benchmarks, retaining the selected policies and the host task structure. In the Coffee_D0 task of MimicGen [1], the robot picks up a coffee capsule, inserts it into the holder of a coffee machine, and closes the lid. The capsule is already an object in ManiPhysicsZoo, so we applied its material properties directly. In the scoop_with_spatula task of THE COLOSSEUM [2], the robot grasps a spatula, scoops up an object on the table, and lifts it. In the ScrubBowl task of RoboCasa365 [3], the robot picks up a sponge and scrubs an object in the sink. In these two tasks, we replaced the object of the task with an object from the Zoo: the boiled potato for spatula scooping and the PET bottle for scrubbing. Task completion follows the success check of each task, adapted to the replaced object where needed. In these tasks, we assess not only the grasp but also the recorded contacts of the object with the hand, the tool, the support, and the insertion fixture.

Replacing the object places the policy outside the conditions of its training data, so task success becomes infrequent. In THE COLOSSEUM, we therefore continued the rollouts until ten successful episodes were collected, which took 283 rollouts, so that there are successful episodes to assess. These ten episodes are a sample for the assessment and not an estimate of the task success rate.

BenchmarkPolicyTaskObjectRolloutsSuccessful episodesDamaged by graspDamaged by other contacts
MimicGen [1]WMPOCoffee_D0Coffee capsule128954
THE COLOSSEUM [2]BridgeVLAscoop_with_spatulaBoiled potato28310N/A0
RoboCasa365 [3]GR00T N1.5ScrubBowlPET bottle501201

In MimicGen, all nine successful episodes exceed the damage criterion of the capsule. Five are damaged by the grasp and four by the contact after release, when the capsule lands in the holder. The gap between task success and object preservation reported in Sec. 6.1 therefore also appears in an insertion task. In THE COLOSSEUM, the ten successful episodes show no damage. This is expected from the nature of the task: the potato is not grasped, and a successful scoop only carries it on the spatula, a contact that stays below the damage criterion. Damage can be seen instead in failed episodes: in two failed episodes that we assessed, the criterion is exceeded when the spatula presses the potato and when the potato lands on the table. These are cases of damage caused by contacts other than a grasp, namely contact with the tool and with the support. In RoboCasa365, damage is identified in one of the twelve successful episodes, where the palm presses the bottle while the robot scrubs it with the sponge.

Across the three benchmarks, damage arises in different ways: during the grasp and after release in insertion, under the tool and on the support in scooping, and under a hand contact other than a grasp in scrubbing. These applications have not been validated to the same extent as the pick-and-place benchmark, so we do not present them as a completed extension of the benchmark. The results nevertheless indicate that the Zoo records, together with a solver-based assessment grounded in material response, can be applied readily to tool-use and insertion tasks once the relevant contacts and boundary conditions are specified, and that they make it possible to examine damage arising from contacts other than the grasp. The resources are therefore not limited to our benchmark: they can serve as a basis for adding object-preservation assessment to other existing benchmarks. We report these results in App. I.2 and refer to them from Sec. 5.1.

Contribution and scope. The contribution combines the evidence-linked object resources in Sec. 4.1, grasp-specific mechanical assessment across three material-response categories in Sec. 4.2, and integration with existing evaluation platforms in Sec. 5. The command analysis in Sec. 6.2 and supervision experiments in Sec. 6.3 further investigate the behavior behind preservation failures. The additional objects and task studies broaden this evidence and illustrate how the resources can be reused.

[1] Mandlekar, Ajay, et al. "Mimicgen: A data generation system for scalable robot learning using human demonstrations." arXiv preprint arXiv:2310.17596 (2023).

[2] Pumacay, Wilbert, et al. "The colosseum: A benchmark for evaluating generalization for robotic manipulation." arXiv preprint arXiv:2402.08191 (2024).

[3] Nasiriany, Soroush, et al. "Robocasa365: A large-scale simulation framework for training and benchmarking generalist robots." International Conference on Learning Representations. Vol. 2026. 2026.

AuthorsOpenReview ↗

Author Response (3/5)

W2, Q4, Q6

Thank you for these questions. We agree that the public-checkpoint comparison in Sec. 6.3 does not isolate the label change. We have added a binary-label control for the reported QwenGR00T experiment and an additional continuous-label experiment with GR00T N1.7.

W2/Q4: a matched QwenGR00T control. The new control uses the same Bridge data, initialization, language LoRA configuration, full-action-model training, and training budget as the relabeled model already reported in Sec. 6.3 and App. E.4, while retaining the original binary gripper labels. As in the reported relabeling experiment, we select the control checkpoint with the highest task SR and apply the same preservation assessment. The table compares this new control with the existing relabeled-model results, using Table 3’s six High/Mid objects and 12 episodes per object. All values are percentages.

QwenGR00T training settingSRSafe SRPR
Matched binary-label control43.118.141.9
Object-specific continuous labels (Sec. 6.3)38.925.064.3

With the data and training recipe matched, Safe SR increases from 18.1% to 25.0% and PR from 41.9% to 64.3%, while task SR decreases from 43.1% to 38.9%. Thus, the observation in Sec. 6.3 persists under the matched comparison: object preservation improves, while task completion decreases.

QwenGR00T and the pretraining concern. QwenGR00T’s recipe enters robot-action training directly, with a newly initialized action model and no separate preceding robot-action pretraining stage (App. E.4). Our object-specific targets are supplied from the beginning of this stage. The concern about inheriting binary-gripper supervision from an earlier robot-action pretraining stage therefore does not apply to this initialization.

Q6: additional continuous-label training of GR00T N1.7. Repeating large-scale robot pretraining with continuous gripper labels is beyond the compute and time available during the rebuttal. We therefore evaluated object-specific continuous-label fine-tuning of a GR00T N1.7 policy pretrained with binary gripper labels. We use its full action-model training recipe without LoRA and evaluate the checkpoint selected by task SR. The comparison below uses the public N1.7 results in Table 3 and the same six-object averaging convention.

GR00T N1.7SRSafe SRPR
Public checkpoint (Table 3)62.515.324.4
Object-specific continuous-label fine-tuning61.137.561.4

Safe SR increases from 15.3% to 37.5% and PR from 24.4% to 61.4%, supporting the preservation finding in Sec. 6.3 on another policy. Here, task SR remains close to the public checkpoint (62.5% vs. 61.1%). The additional experiment therefore shows that preservation can also increase with little change in task completion.

Implications for pretraining. We agree that pretraining with binary gripper labels may limit how readily a policy adapts to subsequent supervision. In our N1.7 experiment, however, a policy already trained with binary gripper labels still shows the increase in Safe SR and PR observed in Sec. 6.3 after object-specific relabeling. The same qualitative result is therefore present even with this potential constraint from pretraining.

This gives us reason to expect a similar qualitative response to subsequent object-specific relabeling in a model pretrained with continuous gripper labels: that model would not inherit the same restriction to binary closing targets.

Nevertheless, we have not performed continuous-label robot pretraining or isolated how the original pretraining labels affect subsequent relabeling. The N1.7 subsequent training uses Bridge and also does not match the original pretraining data mixture or total training exposure. We have stated these limitations in the Limitations paragraph (Sec. 7), added the matched control to Table 3 and Sec. 6.3, and report the N1.7 results together with the training and assessment details in App. E.5, referenced from Sec. 6.3.

Reviewer_8WgDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces ManiPhysicsBench, a benchmark for evaluating object damage during VLA manipulation tasks. It uses physics solvers to estimate damage thresholds based on object properties and grasp configurations. Several VLA models are evaluated in LIBERO and SimplerEnv, showing a large gap between task success and safe success. The paper also explores continuous gripper supervision, which improves Safe SR but reduces task success in the reported experiments.

Critical strengths

1.The paper considers object damage in addition to task success. This is an important issue that is often overlooked in existing VLA benchmarks.

2.The damage thresholds take object properties and grasp configurations into account, rather than relying on fixed force limits. The integration with LIBERO and SimplerEnv also makes the benchmark easier to use for evaluating existing VLA models.

Critical weaknesses

1.The validation of the damage thresholds could be improved. For example, the paper uses half-eggshell compression tests to validate the solver, while the benchmark evaluates whole eggs under gripper grasps. Since the contact and loading conditions are different, these tests may not fully validate the grasp damage thresholds used to calculate Safe SR.

2.The relabeling experiment uses different training data and settings for the two models. The original model uses Bridge and RT-1 data, while the relabeled model only uses Bridge data. A comparison under the same training setup would help show whether the improvement in Safe SR comes from the new gripper labels.

Questions for authors

Please see the weakness sections.

8 comments
~Bowen_Jing2OpenReview ↗

Prior work omission: SoftVTBench (arXiv:2607.04234, 5 Jul 2026; arXiv:2608.18701, 19 Aug 2026)

I am the first author of SoftVTBench, the work Reviewer PqKw refers to. I am posting to put the timeline and the overlap on record, and to request an explanation from the authors.

Timeline. SoftVTBench v1 was posted on arXiv on 5 July 2026 (2607.04234) and a revised version on 19 August 2026 (2608.18701), 82 and 37 days before the ICLR deadline respectively. Code and project page have been public since July (github.com/TuojingAI/SoftVTBench, softvtbench.github.io).

Overlap. The submission’s central claim, evaluation paradigm, and headline finding coincide with those of SoftVTBench:

Claim (Sec. 1): task success does not indicate whether the object was physically harmed. SoftVTBench’s abstract opens with the same statement — most benchmarks evaluate task success alone, and a policy may complete the task while causing excessive compression. Metric (Sec. 5.2): Safe SR counts an episode as successful only if the task is completed and the object’s physical constraint is not violated. This is the construction of Goal Success / Safety Success in our July version and DSR in our August version. Sec. 5.2 presents it without attribution to any prior safety-conditioned success metric, including SafeVLA-Bench, which the submission otherwise cites. Finding (Sec. 6.1): a substantial gap between task success and safe success across policies. The same gap is SoftVTBench’s primary result. Gripper encoding (Secs. 6.2–6.3): binary versus continuous gripper supervision as the mechanism behind the gap. SoftVTBench releases both encodings and analyzes this factor.

Request. Related Work discusses DefGraspSim, SoftGym, PlasticineLab, SafeVLA-Bench, OopsieVerse, and tactile VLAs, but not SoftVTBench, which is closer to the submission than any of them on all four points above. I ask the authors to (a) state whether they were aware of SoftVTBench at submission time, (b) add an explicit comparison in Related Work and Table 1, and (c) attribute the safety-conditioned success-rate structure to prior work rather than presenting it as new.

I recognize that both SoftVTBench versions fall within ICLR’s contemporaneous-work window and that the submission’s technical route — offline solver-based assessment of rigid-body rollouts with literature-grounded material models, extended to brittle and thin-walled objects — differs from SoftVTBench’s closed-loop visuo-tactile evaluation. I am not alleging misconduct. I am asking that the closest prior work be acknowledged and the overlap be described accurately.

AuthorsOpenReview ↗

Author Response (1/3)

We thank the reviewer for the careful review and the suggestions on validation and comparison fairness. We address the points below.

W1 (1/2)

Thank you for this important question. New physical robot experiments and destructive-grasp tests are not feasible within the rebuttal period. We therefore strengthened the available evidence through literature-based cross-checks on tests in which an object is compressed between two opposing surfaces, as it is in a gripper grasp.

Reproducing compression tests between opposing surfaces. We reproduced two published tests in which a whole object is compressed from both sides. Hahn et al. [1] compress an intact egg between opposing rubber pads (Sec. 2.2, Fig. 3a), unlike the half-shell tests already reported in App. A.3. This closely resembles our assessment’s whole-egg, opposed-padded-contact configuration (App. C.3). An et al. [2] compress a whole strawberry between two flat metal plates. This test also loads the fruit from both sides, and differs from the egg test in that the contacting surfaces are metal rather than rubber. In both cases we used our assessment pipeline as it is, with the material parameters and the damage criterion unchanged and without any experiment-specific fitting. For the egg, we matched only the egg geometry and the pad specification to the test; for the strawberry, only the fruit geometry.

ObjectMeasured mean (N)Model (N)Difference
Egg745.99829.53+11.20%
Strawberry9.949.46−4.80%

With the material parameters and damage criteria unchanged, the pipeline reproduces both tests to within about 11%. This addresses the concern about relying only on half-shell validation: the additional comparisons use whole objects under loading closer to a padded-gripper grasp. They therefore strengthen the physical support for applying the models under the benchmark’s grasp conditions.

Why the assessment includes allowances favorable to the robot. Material models and contact calculations inevitably have uncertainty. We therefore avoid making preservation failure depend only on the earliest modeled microscopic event. The tomato criterion does not count cuticle microcracking alone. For potato and strawberry, damage is declared only when tissue with PEEQ ≥ 0.002 occupies 1% of the initial volume (App. A.3), which allows more deformation than declaring damage at first yield. Thus, a preservation failure under these settings is not merely the consequence of choosing the earliest possible onset criterion. In addition, the damage verdicts for the High objects do not lie close to the thresholds: among damaged episodes, the median ratio of recorded force to threshold ranges from 2.0 to 6.0 (App. A.3; App. C.4). Because preservation failures remain even under assumptions that favor the robot, the concern that these grips could also damage objects during physical handling becomes more credible.

[1] Hahn, Eric N., et al. "Nature's technical ceramic: the avian eggshell." Journal of the Royal Society Interface 14.126 (2017): 20160804.

[2] An, Xue, et al. "Characterization of textural failure mechanics of strawberry fruit." Journal of Food Engineering 282 (2020): 110016.

AuthorsOpenReview ↗

Author Response (2/3)

W1 (2/2)

Sensitivity of the conclusion to more permissive criteria. To examine this argument directly, we rescored the recorded episodes with relaxed damage criteria, holding task outcomes and recorded actions fixed. We show the original and 2× criteria below, with each entry giving SR / Safe SR / PR (%).

Object (SR / Safe SR / PR)1×2×
A Egg (High)45.0 / 0.8 / 1.945.0 / 13.3 / 29.6
A Cherry tomato (mid)73.3 / 65.0 / 88.673.3 / 65.8 / 89.8
B Boiled potato (High)60.0 / 0.8 / 1.460.0 / 1.7 / 2.8
B Strawberry (Mid)55.0 / 9.2 / 16.755.0 / 39.2 / 71.2
C Paper cup (High)33.3 / 3.3 / 10.033.3 / 6.7 / 20.0
C Portion cup (Mid)23.3 / 1.7 / 7.123.3 / 3.3 / 14.3
C Yakult bottle (Low)70.0 / 68.3 / 97.670.0 / 69.2 / 98.8

With the criteria relaxed by 2×, task SR remains above Safe SR for every object in this comparison. The persistence of preservation failures under this more permissive judgment supports the robustness of the assessment: its interpretation does not rest on a single narrowly chosen damage criterion.

Separately, the observation that gripping is only weakly differentiated across objects does not depend on the damage threshold. Sec. 6.2 directly examines the executed gripper commands and finds concentration near full opening and closure, with largely similar aggregate distributions across objects despite their different physical constraints. Changing a damage threshold can change the preservation verdict, but it leaves this observed command pattern unchanged. The finding of limited object-dependent gripping in Sec. 6.2 therefore remains supported independently of which of these damage criteria is used.

We have added the whole-egg and whole-strawberry comparisons and the original-versus-2× criterion comparison to App. A.3, and Sec. 6.1 now refers to the criterion comparison. Direct validation of force–displacement responses and damage onset under matched physical robot grasps remains the limitation stated in Sec. 7.

Reviewer_PqKwOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ManiPhysicsBench to evaluate whether VLA policies complete manipulation tasks while preserving the manipulated objects. It combines ManiPhysicsZoo, a library of object geometries and documented material properties, with an offline assessment pipeline that computes grasp-specific damage thresholds using physics solvers. Evaluations on LIBERO and SimplerEnv cover brittle fracture, soft-object deformation, and thin-walled container deformation. The results reveal substantial gaps between task success and safe success. The paper also analyzes gripper commands and explores object-specific continuous gripper supervision.

Critical strengths
  1. An important evaluation problem. Task completion alone provides an incomplete picture of manipulation quality, particularly for fragile and deformable objects. Explicitly assessing object preservation addresses a meaningful limitation of existing evaluation practices and offers a useful direction for developing more reliable manipulation policies.

  2. A substantive resource and assessment framework. The combination of reusable object assets, documented material-property sources, and grasp-specific mechanical assessment is valuable. Accounting for geometry and contact configuration provides a more informative basis for evaluation than a single object-level force threshold. The literature-response comparisons and explicit documentation of modeling assumptions strengthen the framework’s transparency.

  3. Useful empirical findings across models and platforms. Evaluating multiple public VLA checkpoints on two platforms demonstrates the diagnostic value of separating task success from safe success. The gripper-command analysis and continuous-supervision experiment provide additional insights into potential limitations of current control representations, while acknowledging the associated task-success and generalization trade-offs.

Critical weaknesses
  1. The physical validity of the final damage judgments remains insufficiently established. The literature-based calibration and response comparisons are useful, but they do not directly validate the complete assessment pipeline under matched physical grasps. Agreement with a material response or a compression measurement does not necessarily establish accurate damage-onset prediction for a whole object under the benchmark’s contact conditions. Representative physical tests comparing force–displacement responses and damage onset would substantially strengthen confidence in the reported preservation scores.

  2. The assessment depends on controller settings and an offline approximation of interaction. Policies execute in rigid-body simulation, while deformation and damage are assessed afterward. Although the authors account for servo-load reduction, deformed geometry does not affect subsequent contacts or policy observations. Moreover, the gain sensitivity analysis rescales recorded forces while holding trajectories and task outcomes fixed, rather than rerunning the closed-loop interaction. These assumptions limit how directly the scores can be interpreted as properties of the policies themselves or generalized to physical deployment.

  3. Positioning relative to closely related recent benchmarks is incomplete. SoftVTBench also distinguishes task completion from physically constrained interaction quality, uses FEM states for deformation-aware evaluation, and examines gripper control granularity. It is not discussed in the current manuscript. A substantive comparison would clarify the additional contribution of grasp-specific damage prediction and literature-supported material modeling, including their scope and trade-offs relative to closed-loop deformable-object evaluation.

Overall assessment

Weak Accept. I lean toward acceptance because the paper addresses a meaningful problem and contributes a useful resource and coherent assessment framework. The empirical findings provide motivation for evaluating manipulation beyond task completion. My main reservations concern the real-world validity of the damage judgments and the incomplete positioning against closely related work. These limitations constrain the interpretation of the results, but do not outweigh the potential value of the benchmark.

Questions for authors
  1. What evidence supports the accuracy of the complete grasp-specific damage-onset prediction pipeline, beyond matching individual material responses or compression measurements? Please clarify the scope of the existing validation and any matched physical-grasp evidence already available.

  2. How should preservation scores be interpreted given that policies execute in rigid-body simulation and deformation is assessed offline? In particular, please clarify what the fixed-trajectory controller-gain sensitivity analysis can and cannot establish about closed-loop policy behavior.

  3. Please clarify the relationship to SoftVTBench (https://arxiv.org/abs/2608.18701), particularly the scope and trade-offs of grasp-specific damage prediction and literature-supported material models versus closed-loop deformable-object assessment.

7 comments
AuthorsOpenReview ↗

Author Response (1/5)

We thank the reviewer for the thoughtful review and helpful questions. Below, we address the physical validation, offline assessment, and relationship to SoftVTBench.

W1 & Q1 (1/2)

Thank you for this important question. New physical robot experiments and destructive-grasp tests are not feasible within the rebuttal period. We therefore strengthened the available evidence through literature-based cross-checks on tests in which an object is compressed between two opposing surfaces, as it is in a gripper grasp.

Reproducing compression tests between opposing surfaces. We reproduced two published tests in which a whole object is compressed from both sides. Hahn et al. [1] compress an intact egg between opposing rubber pads (Sec. 2.2, Fig. 3a), unlike the half-shell tests already reported in App. A.3. This closely resembles our assessment’s whole-egg, opposed-padded-contact configuration (App. C.3). An et al. [2] compress a whole strawberry between two flat metal plates. This test also loads the fruit from both sides, and differs from the egg test in that the contacting surfaces are metal rather than rubber. In both cases we used our assessment pipeline as it is, with the material parameters and the damage criterion unchanged and without any experiment-specific fitting. For the egg, we matched only the egg geometry and the pad specification to the test; for the strawberry, only the fruit geometry.

ObjectMeasured mean (N)Model (N)Difference
Egg745.99829.53+11.20%
Strawberry9.949.46−4.80%

With the material parameters and damage criteria unchanged, the pipeline reproduces both tests to within about 11%. This addresses the concern about relying only on half-shell validation: the additional comparisons use whole objects under loading closer to a padded-gripper grasp. They therefore strengthen the physical support for applying the models under the benchmark’s grasp conditions.

Why the assessment includes allowances favorable to the robot. Material models and contact calculations inevitably have uncertainty. We therefore avoid making preservation failure depend only on the earliest modeled microscopic event. The tomato criterion does not count cuticle microcracking alone. For potato and strawberry, damage is declared only when tissue with PEEQ ≥ 0.002 occupies 1% of the initial volume (App. A.3), which allows more deformation than declaring damage at first yield. Thus, a preservation failure under these settings is not merely the consequence of choosing the earliest possible onset criterion. In addition, the damage verdicts for the High objects do not lie close to the thresholds: among damaged episodes, the median ratio of recorded force to threshold ranges from 2.0 to 6.0 (App. A.3; App. C.4). Because preservation failures remain even under assumptions that favor the robot, the concern that these grips could also damage objects during physical handling becomes more credible.

[1] Hahn, Eric N., et al. "Nature's technical ceramic: the avian eggshell." Journal of the Royal Society Interface 14.126 (2017): 20160804.

[2] An, Xue, et al. "Characterization of textural failure mechanics of strawberry fruit." Journal of Food Engineering 282 (2020): 110016.

AuthorsOpenReview ↗

Author Response (2/5)

W1 & Q1 (2/2)

Sensitivity of the conclusion to more permissive criteria. To examine this argument directly, we rescored the recorded episodes with relaxed damage criteria, holding task outcomes and recorded actions fixed. We show the original and 2× criteria below, with each entry giving SR / Safe SR / PR (%).

Object (SR / Safe SR / PR)1×2×
A Egg (High)45.0 / 0.8 / 1.945.0 / 13.3 / 29.6
A Cherry tomato (mid)73.3 / 65.0 / 88.673.3 / 65.8 / 89.8
B Boiled potato (High)60.0 / 0.8 / 1.460.0 / 1.7 / 2.8
B Strawberry (Mid)55.0 / 9.2 / 16.755.0 / 39.2 / 71.2
C Paper cup (High)33.3 / 3.3 / 10.033.3 / 6.7 / 20.0
C Portion cup (Mid)23.3 / 1.7 / 7.123.3 / 3.3 / 14.3
C Yakult bottle (Low)70.0 / 68.3 / 97.670.0 / 69.2 / 98.8

With the criteria relaxed by 2×, task SR remains above Safe SR for every object in this comparison. The persistence of preservation failures under this more permissive judgment supports the robustness of the assessment: its interpretation does not rest on a single narrowly chosen damage criterion.

Separately, the observation that gripping is only weakly differentiated across objects does not depend on the damage threshold. Sec. 6.2 directly examines the executed gripper commands and finds concentration near full opening and closure, with largely similar aggregate distributions across objects despite their different physical constraints. Changing a damage threshold can change the preservation verdict, but it leaves this observed command pattern unchanged. The finding of limited object-dependent gripping in Sec. 6.2 therefore remains supported independently of which of these damage criteria is used.

We have added the whole-egg and whole-strawberry comparisons and the original-versus-2× criterion comparison to App. A.3, and Sec. 6.1 now refers to the criterion comparison. Direct validation of force–displacement responses and damage onset under matched physical robot grasps remains the limitation stated in Sec. 7.

AuthorsOpenReview ↗

Author Response (3/5)

W2 & Q2

Thank you for this question. Our preservation score is a post hoc assessment of recorded interactions under the stated material, contact, and controller models (Sec. 4.2; App. D.6). The gain analysis in App. C.4 keeps the trajectories and task outcomes fixed. It measures how the verdicts change when the recorded loads are rescaled, rather than how the policy would act under a different gain.

To test whether a policy responds to the deformed object and whether the updated shape changes later contacts, we added a closed-loop mesh-feedback mode. After each control step, the displacement field is applied to the collision and visual geometry before the policy receives its next observation.

We focus on the two axis-B High/Mid objects and the three axis-C containers. For axis-A objects, substantial fracture-related shape changes already correspond to damage, while the axis-B Low object undergoes negligible deformation under the evaluated loads. We evaluate the same public QwenGR00T checkpoint used in Table 3, with 12 episodes per object. Each entry is SR / Safe SR / PR (%).

Object (SR / Safe SR / PR)Main table (Table 3)Closed-Loop
B Boiled potato (High)91.7 / 0.0 / 0.091.7 / 0.0 / 0.0
B Strawberry (Mid)75.0 / 8.3 / 11.183.3 / 0.0 / 0.0
C Paper cup (High)41.7 / 0.0 / 0.08.3 / 0.0 / 0.0
C Portion cup (Mid)58.3 / 0.0 / 0.033.3 / 8.3 / 25.0
C Yakult bottle (Low)83.3 / 83.3 / 100.083.3 / 83.3 / 100.0

The results retain the preservation gap reported in Sec. 6.1 and Table 3. The potato and paper cup have no safe successes in either setting. The portion cup has a small increase, Yakult remains fully preserved among successful episodes, and the strawberry still has successful episodes without preservation. Thus, the main finding also holds when the updated shape is returned to the policy: task completion does not ensure object preservation.

Task SR is lower for the two cups. Updating the mesh changes the contact dynamics: the gripper can lose contact with the updated surface, allowing the object to slip or drop. Our aim is to add object-preservation assessment while keeping the platform's task-evaluation conditions unchanged, so that the task success of released models remains comparable under the original protocol. Using geometry feedback as the default would instead evaluate them under a different interaction setting.

Mesh feedback updates the shape after each control step, while the rigid-body engine of the host simulator still resolves the contact. We therefore also ran a two-way coupled simulation on RoboLab [1], an independent Isaac Lab platform with a different gripper and policy. In this simulation, the FEM solver resolves the contact and deformation of the object at every physics step, returns the reaction forces to the robot, and updates the shape that the policy observes. We replaced the object of an existing pick-and-place task with the Zoo boiled potato, keeping its material record and damage criterion, and ran the public π0.5-DROID checkpoint without further training. In all 12 runs, the potato exceeds the damage criterion during the first grasp. The damage found by the offline assessment therefore also arises when the object deforms within the rollout, and the Zoo assets and the criterion apply in this setting as well. This simulation is costly, however: each run took about 36 minutes on average for about 11 s of simulated time.

Our purpose is to assess released VLA models. The default therefore keeps the original rollouts of LIBERO and SimplerEnv, the widely used platforms on which these models are evaluated, and applies the assessment offline. Both closed-loop settings are consistent with the main finding. However, mesh feedback changes the conditions under which a task is evaluated, and the coupled simulation requires far more computation and departs from the pipelines on which released models are commonly evaluated. We therefore keep offline assessment as the default and will release mesh feedback as an optional mode. We have added both comparisons and their procedures to App. D, which Sec. 6.1 refers to.

[1] Yang, Jenai Xuning, et al. "RoboLab: A high-fidelity simulation benchmark for analysis of task generalist policies." arXiv preprint arXiv:2604.09860 (2026).

Comments on the paper

3 comments
~Bowen_Jing2OpenReview ↗

Prior work omission: SoftVTBench (arXiv:2607.04234, 5 Jul 2026; arXiv:2608.18701, 19 Aug 2026)

Comment:

I am the first author of SoftVTBench, the work Reviewer PqKw refers to. I am posting to put the timeline and the overlap on record, and to request an explanation from the authors.

Timeline. SoftVTBench v1 was posted on arXiv on 5 July 2026 (2607.04234) and a revised version on 19 August 2026 (2608.18701), 82 and 37 days before the ICLR deadline respectively. Code and project page have been public since July (github.com/TuojingAI/SoftVTBench, softvtbench.github.io).

Overlap. The submission’s central claim, evaluation paradigm, and headline finding coincide with those of SoftVTBench:

Claim (Sec. 1): task success does not indicate whether the object was physically harmed. SoftVTBench’s abstract opens with the same statement — most benchmarks evaluate task success alone, and a policy may complete the task while causing excessive compression. Metric (Sec. 5.2): Safe SR counts an episode as successful only if the task is completed and the object’s physical constraint is not violated. This is the construction of Goal Success / Safety Success in our July version and DSR in our August version. Sec. 5.2 presents it without attribution to any prior safety-conditioned success metric, including SafeVLA-Bench, which the submission otherwise cites. Finding (Sec. 6.1): a substantial gap between task success and safe success across policies. The same gap is SoftVTBench’s primary result. Gripper encoding (Secs. 6.2–6.3): binary versus continuous gripper supervision as the mechanism behind the gap. SoftVTBench releases both encodings and analyzes this factor.

Request. Related Work discusses DefGraspSim, SoftGym, PlasticineLab, SafeVLA-Bench, OopsieVerse, and tactile VLAs, but not SoftVTBench, which is closer to the submission than any of them on all four points above. I ask the authors to (a) state whether they were aware of SoftVTBench at submission time, (b) add an explicit comparison in Related Work and Table 1, and (c) attribute the safety-conditioned success-rate structure to prior work rather than presenting it as new.

I recognize that both SoftVTBench versions fall within ICLR’s contemporaneous-work window and that the submission’s technical route — offline solver-based assessment of rigid-body rollouts with literature-grounded material models, extended to brittle and thin-walled objects — differs from SoftVTBench’s closed-loop visuo-tactile evaluation. I am not alleging misconduct. I am asking that the closest prior work be acknowledged and the overlap be described accurately.

AuthorsOpenReview ↗

Author Response

Thank you for the detailed comment and for setting out the timeline and the points of overlap. We answer your three requests first and then address the four points.

(a) Awareness at submission time. We were not aware of SoftVTBench at submission time. Thank you for bringing it to our attention.

(b) Comparison in Related Work and Table 1. We have added a comparison with SoftVTBench [1] to Related Work and Table 1, and we also cite it in the Introduction.

(c) Attribution of the success-rate structure. We would first like to make one point clear: we have not presented the separation of task completion from safe completion as new, nor as a contribution of our work. The Related Work states that SafeVLA-Bench and OopsieVerse "share our premise that task completion should be separate from safe completion when evaluating VLAs" (Sec. 2), and our listed contributions are the object library, the solver-based assessment, the benchmark, and the evaluation with behavioral analysis (Sec. 1). Nevertheless, to clarify this further, we now cite SafeVLA-Bench, OopsieVerse, and SoftVTBench at the point you raised, where the metrics are defined (Sec. 5.2).

Claim and finding (Secs. 1 and 6.1). That task success can hide harm to the object is a motivation shared across this line of work, including OopsieVerse, and it is the motivation of our paper rather than its contribution. Our contribution lies in the physical basis of the preservation verdict. For each recorded grasp, physics solvers compute a damage threshold from the object's geometry and literature-supported material properties, covering brittle fracture, soft-object deformation, and thin-walled container deformation. SoftVTBench describes its tolerance as "an operational benchmark criterion for deformation compliance in simulation, not a material-failure or real-world damage threshold" [1] (Sec. 3.5). The two verdicts therefore measure different things. Accordingly, the gap in Sec. 6.1 counts completed tasks in which the recorded grasp exceeds a damage threshold derived from material properties, measured on public checkpoints as released, rather than completions that exceed a calibrated tolerance among policies trained for the benchmark. The aims also differ: SoftVTBench provides a visuo-tactile dataset and studies how policies trained on it, with and without touch, keep deformation within a calibrated tolerance, whereas we assess, on the basis of each object's physical properties, whether public VLA checkpoints complete tasks without damaging the object when evaluated without retraining on LIBERO and SimplerEnv.

Gripper encoding (Secs. 6.2–6.3). We do not claim that comparing binary and continuous gripper commands is new. The two analyses differ in purpose, in the policies analyzed, and in the basis of the labels. SoftVTBench compares the two encodings to separate the benefit of tactile sensing from that of control granularity [1] (Sec. 5.4). It does so on policies trained on its own demonstrations, whose continuous closures are drawn from per-object ranges calibrated by scripted grasps on its simulation assets [1] (Secs. 3.4–3.5). Our analysis instead follows from the research question in Sec. 1: "Can current VLAs account for objects' physical properties to complete manipulation tasks without causing damage?" Sec. 6.2 therefore examines whether public checkpoints adapt their gripper commands to objects with different damage thresholds, and finds that the commands concentrate near full opening and closure with similar distributions across objects. Sec. 6.3 then examines whether supervision grounded in physical properties changes this behavior: the continuous labels set the target grip between each object's lifting load and its damage load estimated from material properties, and are applied by relabeling existing robot demonstrations.

In conclusion, as you note, the technical routes of the two works differ. Thank you again for helping us describe the relationship between them accurately.

[1] Jing, Bowen, et al. "SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation." arXiv preprint arXiv:2608.18701 (2026).

AuthorsOpenReview ↗

Summary of Author-Reviewer Discussions

Dear Reviewers and Area Chair,

We thank the reviewers for their constructive feedback and for recognizing the importance of assessing object preservation and the value of ManiPhysicsZoo and the solver-based assessment. All results and clarifications have been incorporated into the revised manuscript, with revisions highlighted in red. Below, we summarize the main concerns and how we addressed them.

Physical Validity of the Damage Judgments (W1 & Q1 of Reviewer PqKw, W1 of Reviewer 8WgD): We reproduced two published tests in which a whole object, an intact egg and a whole strawberry, is compressed between opposing surfaces, as in a gripper grasp. With the material parameters and damage criteria unchanged, the model differs from the measured loads by +11.2% and −4.8%. With the criteria relaxed by 2×, task SR remains above Safe SR for every object (App. A.3).

Offline Assessment and the Rigid-Body Rollout (W2 & Q2 of Reviewer PqKw, Q2 of Reviewer 6Gxq): We added two closed-loop comparisons. With mesh feedback on SimplerEnv, the gap between task success and safe success is retained. In a two-way coupled FEM simulation on RoboLab, the boiled potato exceeds the damage criterion during the first grasp in all 12 runs. Both are consistent with the main finding, and we keep offline assessment as the default to preserve the original evaluation protocol of each platform (App. D).

Relationship to SoftVTBench (W3 & Q3 of Reviewer PqKw, the public comment): We added a comparison to Related Work and Table 1, and cite it where the metrics are defined (Sec. 5.2). Our work differs in computing a damage threshold for each grasp from literature-supported material properties, in treating three material-response categories with separate solvers, and in assessing released checkpoints without retraining.

Supervision Experiments (W2 of Reviewer 8WgD, W2 & Q4–Q6 of Reviewer 6Gxq): Against a binary-label control trained with the same data, model, and training recipe, object-specific labels raise Safe SR from 18.1% to 25.0% and PR from 41.9% to 64.3%, while task SR falls from 43.1% to 38.9% (Table 3, Sec. 6.3). We also added continuous-label fine-tuning of GR00T N1.7, which raises Safe SR from 15.3% to 37.5% with task SR nearly unchanged (App. E.5), and a lower-target labeling variant (App. E.6).

Object and Task Coverage (W1, Q1 & Q3 of Reviewer 6Gxq): We clarified the object-admission criteria (Sec. 5.1, App. B.1) and evaluated two additional Zoo objects, quail egg and eggplant, on which the gap remains (App. I.1). We also applied the Zoo assets and the assessment to insertion and tool-use tasks in MimicGen, THE COLOSSEUM, and RoboCasa365 (App. I.2).

Scope of the Conclusions (Reviewer 6Gxq follow-up): Our conclusions concern damage that arises while the object is grasped, and they rest on around ten public checkpoints per platform across many policy families (Secs. 6.1–6.2). Extension to other contact-rich tasks is stated as future work, for which the added case studies provide a basis (Sec. 7).

Findings and Real-Robot Validation (Reviewer 8WgD follow-up): Alongside the object assets and the assessment pipeline, the paper contributes findings about current VLA policies: task success hides object damage (Sec. 6.1), gripper commands differ little across objects (Sec. 6.2), and object-specific supervision does not close the gap (Sec. 6.3). Validation on a physical robot has not been carried out and is stated as a limitation (Sec. 7), but the central conclusions do not rest on the exact thresholds: the gap persists with the criteria relaxed by 2×, and the gripper analysis involves no damage threshold.

We thank the reviewers and the Area Chair again for their time and consideration.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.