- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ManiPhysicsBench, a physics-based benchmark for evaluating object preservation in VLA manipulation. The authors build ManiPhysicsZoo, a library of 91 object assets with geometry, material properties and structural information. Using these assets, they compute grasp-specific damage thresholds, and compare these thresholds with contact forces recorded during policy rollouts to predict potential deformation or fracture. ManiPhysicsBench instantiates tasks on LIBERO and SimplerEnv across three physics axes—brittle fracture, soft object deformation, and thin-walled container deformation—each with high, medium, and low difficulty 3 levels. Evaluating a broad set of public checkpoints, the authors report a gap between task success and safe success (task completion while preserving the object). They further find that gripper commands concentrate near fully open and fully closed states, with similar distributions across objects. Finally, they retrain QwenGR00T by replacing binary closing labels with object-specific continuous targets, and observe more object-dependent grasping and higher Safe SR, but lower task success and limited generalization.
Critical strengths
- The paper identifies object preservation as an requirement for VLA evaluation and provides a reusable tool kit. Existing manipulation benchmarks such as LIBERO treat objects as rigid bodies and measure only task success rate, thus a policy that damages an object during a task may still be counted as successful. To address this problem, the authors construct ManiPhysicsZoo, which consolidates literature-supported material properties, meshes and structural information into traceable assets. They also construct a solver-based assessment pipeline which computes grasp-specific damage thresholds using established solvers. These are useful as shared tools for the community.
- The gripper analysis provides an illustration of the preservation gap, which shows that gripper command distribution of policies is nearly binary across objects with very different physical properties. This observation is consistent with the fact that most VLA pretraining datasets use binary gripper annotations. The relabeling experiment, which replaces binary closing labels with object-specific continuous targets and retrains QwenGR00T, is a well-motivated attempt to test whether post train can alleviate this problem.
Critical weaknesses
- Task coverage is too narrow to support the paper’s broad claims about VLA object-preservation ability, and the technical novelty beyond benchmark construction is limited. The benchmark evaluates only pick-and-place tasks, and the main evaluation involves only nine object–difficulty configurations across three physics axes which is a much smaller subset of their object library. While pick-and-place is a natural starting point, object preservation is relevant to a much wider range of manipulation behaviors: pushing, sliding, opening, inserting, tool use, and other contact-rich tasks where forces are applied in different directions and magnitudes. The paper’s conclusion that current VLAs do not preserve objects would be much stronger if it were supported by a more diverse task suite. Thus, the benchmark cannot serve as a comprehensive assessment of VLA preservation ability. Combined with the fact that the core idea—physics-based damage assessment for manipulation—is a natural extension of existing benchmarks, this limits the paper’s informativeness for readers already familiar with rigid-body evaluation.
- The experimental of object-specific gripper supervision is insufficient. The paper’s main experiment is a single ablation: it replaces only the closing labels of a subset of Bridge demonstrations with object-specific continuous targets, leaving arm actions and opening labels unchanged, LoRA fine-tunes QwenGR00T, and compares the result against the original public checkpoint. This comparison is confounded by multiple factors. The retrained model uses Bridge-only data, and a different training configuration than the public checkpoint which was trained on a Bridge–RT-1 mixture. The authors acknowledge this in App. E.4. More importantly, given that most VLA pretraining datasets use binary gripper annotations, the paper should include more experiments with policies fine-tuned or trained on continuous gripper data, not just a single LoRA run. Without more experiments, the claim that object-specific supervision encourages object-dependent gripping remains suggestive rather than established.
Questions for authors
- The object library contains 91 objects, but the benchmark uses a much smaller subset. What criteria determined which objects were selected for the benchmark, and how do you expect the results to generalize to the broader library? Would including more objects from the library change the result?
- Since object deformation can alter contact forces, have you conducted any experiments to quantify the gap between the rigid-body contact forces recorded during rollout and the deformable-body contact response modeled by the solver? A comparison on a small set of objects would help establish how much the solver-based assessment depends on the rigid-body rollout assumption.
- To support the paper’s conclusion that current VLAs do not preserve objects, the benchmark needs evidence beyond pick-and-place. Can you provide experiments on other contact-rich tasks with a wider range of manipulation behaviors, such as insertion, pushing, and tool use, where forces are applied in different directions and magnitudes? Without such evidence, the claim that VLAs fail at object preservation remains confined to a narrow task family and may not generalize to contact-rich manipulation.
- You acknowledged in Appendix E.4 that this experiment fails to fully isolate the effect of gripper annotation changes. Could you justify the rationality of this experimental design and demonstrate that the observed performance change exactly reflects the effects of the newly proposed gripper annotations?
- In the retraining experiment, the paper generates synthetic continuous gripper labels by setting the target gripping force as the midpoint of lifting force and damage threshold. While this works for objects with a narrow safe force window, for objects with a wide safe-force window, this midpoint may exceed the proper force needed for lifting, potentially encouraging unnecessarily strong grips. Could you justify this design choice, or provide a variant that uses a lower target force within the safe interval?
- Most of the checkpoints used in your experiments are pretrained on datasets whose gripper annotations are binary. Could you provide more experiments with checkpoints that are retrained on datasets with object-specific continuous gripper labels?
9 comments
Author Response (2/5)
W1, Q1, Q3 (2/2)
Q3: tool-use and insertion tasks. The paper targets pick-and-place tasks and judges damage that arises while the object is grasped. To examine whether the same resources apply beyond this setting, we applied the Zoo assets and the solver-based assessment to tool-use and insertion tasks in three existing benchmarks, retaining the selected policies and the host task structure. In the Coffee_D0 task of MimicGen [1], the robot picks up a coffee capsule, inserts it into the holder of a coffee machine, and closes the lid. The capsule is already an object in ManiPhysicsZoo, so we applied its material properties directly. In the scoop_with_spatula task of THE COLOSSEUM [2], the robot grasps a spatula, scoops up an object on the table, and lifts it. In the ScrubBowl task of RoboCasa365 [3], the robot picks up a sponge and scrubs an object in the sink. In these two tasks, we replaced the object of the task with an object from the Zoo: the boiled potato for spatula scooping and the PET bottle for scrubbing. Task completion follows the success check of each task, adapted to the replaced object where needed. In these tasks, we assess not only the grasp but also the recorded contacts of the object with the hand, the tool, the support, and the insertion fixture.
Replacing the object places the policy outside the conditions of its training data, so task success becomes infrequent. In THE COLOSSEUM, we therefore continued the rollouts until ten successful episodes were collected, which took 283 rollouts, so that there are successful episodes to assess. These ten episodes are a sample for the assessment and not an estimate of the task success rate.
| Benchmark | Policy | Task | Object | Rollouts | Successful episodes | Damaged by grasp | Damaged by other contacts |
|---|---|---|---|---|---|---|---|
| MimicGen [1] | WMPO | Coffee_D0 | Coffee capsule | 128 | 9 | 5 | 4 |
| THE COLOSSEUM [2] | BridgeVLA | scoop_with_spatula | Boiled potato | 283 | 10 | N/A | 0 |
| RoboCasa365 [3] | GR00T N1.5 | ScrubBowl | PET bottle | 50 | 12 | 0 | 1 |
In MimicGen, all nine successful episodes exceed the damage criterion of the capsule. Five are damaged by the grasp and four by the contact after release, when the capsule lands in the holder. The gap between task success and object preservation reported in Sec. 6.1 therefore also appears in an insertion task. In THE COLOSSEUM, the ten successful episodes show no damage. This is expected from the nature of the task: the potato is not grasped, and a successful scoop only carries it on the spatula, a contact that stays below the damage criterion. Damage can be seen instead in failed episodes: in two failed episodes that we assessed, the criterion is exceeded when the spatula presses the potato and when the potato lands on the table. These are cases of damage caused by contacts other than a grasp, namely contact with the tool and with the support. In RoboCasa365, damage is identified in one of the twelve successful episodes, where the palm presses the bottle while the robot scrubs it with the sponge.
Across the three benchmarks, damage arises in different ways: during the grasp and after release in insertion, under the tool and on the support in scooping, and under a hand contact other than a grasp in scrubbing. These applications have not been validated to the same extent as the pick-and-place benchmark, so we do not present them as a completed extension of the benchmark. The results nevertheless indicate that the Zoo records, together with a solver-based assessment grounded in material response, can be applied readily to tool-use and insertion tasks once the relevant contacts and boundary conditions are specified, and that they make it possible to examine damage arising from contacts other than the grasp. The resources are therefore not limited to our benchmark: they can serve as a basis for adding object-preservation assessment to other existing benchmarks. We report these results in App. I.2 and refer to them from Sec. 5.1.
Contribution and scope. The contribution combines the evidence-linked object resources in Sec. 4.1, grasp-specific mechanical assessment across three material-response categories in Sec. 4.2, and integration with existing evaluation platforms in Sec. 5. The command analysis in Sec. 6.2 and supervision experiments in Sec. 6.3 further investigate the behavior behind preservation failures. The additional objects and task studies broaden this evidence and illustrate how the resources can be reused.
[1] Mandlekar, Ajay, et al. "Mimicgen: A data generation system for scalable robot learning using human demonstrations." arXiv preprint arXiv:2310.17596 (2023).
[2] Pumacay, Wilbert, et al. "The colosseum: A benchmark for evaluating generalization for robotic manipulation." arXiv preprint arXiv:2402.08191 (2024).
[3] Nasiriany, Soroush, et al. "Robocasa365: A large-scale simulation framework for training and benchmarking generalist robots." International Conference on Learning Representations. Vol. 2026. 2026.
Author Response (3/5)
W2, Q4, Q6
Thank you for these questions. We agree that the public-checkpoint comparison in Sec. 6.3 does not isolate the label change. We have added a binary-label control for the reported QwenGR00T experiment and an additional continuous-label experiment with GR00T N1.7.
W2/Q4: a matched QwenGR00T control. The new control uses the same Bridge data, initialization, language LoRA configuration, full-action-model training, and training budget as the relabeled model already reported in Sec. 6.3 and App. E.4, while retaining the original binary gripper labels. As in the reported relabeling experiment, we select the control checkpoint with the highest task SR and apply the same preservation assessment. The table compares this new control with the existing relabeled-model results, using Table 3’s six High/Mid objects and 12 episodes per object. All values are percentages.
| QwenGR00T training setting | SR | Safe SR | PR |
|---|---|---|---|
| Matched binary-label control | 43.1 | 18.1 | 41.9 |
| Object-specific continuous labels (Sec. 6.3) | 38.9 | 25.0 | 64.3 |
With the data and training recipe matched, Safe SR increases from 18.1% to 25.0% and PR from 41.9% to 64.3%, while task SR decreases from 43.1% to 38.9%. Thus, the observation in Sec. 6.3 persists under the matched comparison: object preservation improves, while task completion decreases.
QwenGR00T and the pretraining concern. QwenGR00T’s recipe enters robot-action training directly, with a newly initialized action model and no separate preceding robot-action pretraining stage (App. E.4). Our object-specific targets are supplied from the beginning of this stage. The concern about inheriting binary-gripper supervision from an earlier robot-action pretraining stage therefore does not apply to this initialization.
Q6: additional continuous-label training of GR00T N1.7. Repeating large-scale robot pretraining with continuous gripper labels is beyond the compute and time available during the rebuttal. We therefore evaluated object-specific continuous-label fine-tuning of a GR00T N1.7 policy pretrained with binary gripper labels. We use its full action-model training recipe without LoRA and evaluate the checkpoint selected by task SR. The comparison below uses the public N1.7 results in Table 3 and the same six-object averaging convention.
| GR00T N1.7 | SR | Safe SR | PR |
|---|---|---|---|
| Public checkpoint (Table 3) | 62.5 | 15.3 | 24.4 |
| Object-specific continuous-label fine-tuning | 61.1 | 37.5 | 61.4 |
Safe SR increases from 15.3% to 37.5% and PR from 24.4% to 61.4%, supporting the preservation finding in Sec. 6.3 on another policy. Here, task SR remains close to the public checkpoint (62.5% vs. 61.1%). The additional experiment therefore shows that preservation can also increase with little change in task completion.
Implications for pretraining. We agree that pretraining with binary gripper labels may limit how readily a policy adapts to subsequent supervision. In our N1.7 experiment, however, a policy already trained with binary gripper labels still shows the increase in Safe SR and PR observed in Sec. 6.3 after object-specific relabeling. The same qualitative result is therefore present even with this potential constraint from pretraining.
This gives us reason to expect a similar qualitative response to subsequent object-specific relabeling in a model pretrained with continuous gripper labels: that model would not inherit the same restriction to binary closing targets.
Nevertheless, we have not performed continuous-label robot pretraining or isolated how the original pretraining labels affect subsequent relabeling. The N1.7 subsequent training uses Bridge and also does not match the original pretraining data mixture or total training exposure. We have stated these limitations in the Limitations paragraph (Sec. 7), added the matched control to Table 3 and Sec. 6.3, and report the N1.7 results together with the training and assessment details in App. E.5, referenced from Sec. 6.3.
Author Responses (1/5)
We thank the reviewer for the careful review and detailed questions. We address object and task coverage, gripper supervision, deformation feedback, and the lower-force labeling variant below.
W1, Q1, Q3 (1/2)
Thank you for these questions. We address the coverage concern through the object-admission criteria, additional objects from ManiPhysicsZoo, and evaluations on tool-use and insertion tasks.
Q1: why these objects were selected. Sec. 5.1 and App. B.1 (Task Admission) describe the selection procedure. A candidate must fit the robot’s gripper opening and allow the task to be completed while preserving the object within the robot’s operating range. Objects whose damage thresholds are effectively unreachable under that setting contribute little additional discrimination between task success and safe success. We therefore retain one Low-difficulty control in each physics axis and avoid filling the benchmark with redundant high-strength cases.
We admit a task only when a scripted-oracle setting achieves both task completion and a preserved-object verdict. From the eligible candidates, we select objects representative of groups with similar mechanical responses and clearly separated gripping-force windows, forming High, Mid, and Low conditions within each axis. This gives a structured comparison of preservation demands rather than an arbitrary sample of the library. We have stated the admission criterion more clearly in Sec. 5.1 and added this selection rationale to the admission procedure in App. B.1.
Additional library objects and their difficulty. We evaluated quail eggs and eggplant with the public QwenGR00T, the binary-label control, and the reported object-specific model, using 12 episodes per object and policy. The additional objects have the following difficulty indices under the definition in App. G.
Quail egg lies between the original High egg (0.357) and Mid cherry tomato (0.185). Eggplant extends the difficult end of axis B, above the original High boiled potato (0.242). The SR / Safe SR / PR results are:
The public checkpoint completes 12/12 quail-egg tasks but preserves the object in only 2/12, and completes 11/12 eggplant tasks with no safe successes. The task-success versus preservation gap in Sec. 6.1 therefore remains on both additional objects.
The relabeled model increases preservation on eggplant, while quail-egg preservation remains low. Eggplant appears in the Bridge episodes, so its relabeling target is explicitly included, whereas the quail egg has no Bridge episode, so no target is included for it. This is consistent with the limited generalization beyond the supplied distinctions discussed in Sec. 6.3.
The original objects were chosen to form the High/Mid/Low comparison; the selection and assessment procedure itself can be applied across the Zoo library, with robot-specific feasibility determining which tasks are admitted. Applying it to these two additional objects retains the central finding in Sec. 6.1. We report the added-object results and their difficulty values in App. I.1, and Sec. 6.1 refers to them.