- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This work introduces RoboPhyscan, an object dataset with nearly 1.1k objects for simulation based on purchasable everyday objects. Each object requires 60min of human effort for 3D scanning, texture photographing, asset cleanup and annotations. These annotations include physical properties required for simulation, like mass, inertia and articulated joints with friction and dampening, and optimized collision meshes. But also "semantic" annotations such as Affordances and Parallel-Jaw Gripper annotations.
The dataset quality is evaluated for sim readiness in Isaac Sim by a test prototol looking at stability on tests such as drop, push or grasp.
Imitation learning experiments demonstrate that simualted trajectories based on these assets can improve downstream task success.
Critical strengths
- Large scale dataset with multiple annotations. Mesh and Texture Quality are very high, which enables multiple downstream applications beyond robot simulations. I see potential interest for Computer Vision community with point cloud processing, grasp prediction and other applications, where a library of high quality object scans are helpful.
- Imitation Learning Experiments explore downstream usage. However, I recommend explicitly reporting statistical significance.
Critical weaknesses
-
Data Quality. The test protocol is limited to only 200 random samples of the nearly 1.1k assets. I quickly explored some of the assets provided as supplementary material in the 3D software Blender.
batch_03_0001_StorageFurniture_draweris modeled upside down. I could also not find how the physical parameters were validated, and if the found errors were corrected in any way. -
I want to mention it for completeness sake: As I understand this work is mainly a "raw" dataset contribution. The collection required substantial human effort to 3D scan and process the objects into simulation ready digital assets. It is not a fully automated pipeline. And it is a collection of 3D assets for downstream tasks, not a suite of test ready benchmark tasks. I strongly believe that such work is helpful to the scientific community and should be recognized. However, one could discuss if ICLR is the best venue for such a dataset contribution.
Questions for authors
Questions
-
Assessment: Mesh Performance. While the high triangle count is great for downstream vision tasks requiring this fidelity, does it have an impact on simulation performance? Right now I suspect that most assets are at approx. 300k triangles due to an artifact of the pipeline, not because this is the optimal setting for the asset. Quick tests in blender indicate, that 1%-10% of the triangle count is sufficient without obvious visual degradation.
1.1 What is the impact on simulation / render speed of these high quality assets in cluttered scenes, e.g. 10-20 random RoboPhyscan assets vs 10-20 random PartNet-Mobility assets?
1.2 How about the quality / speed impact of collision meshes? For example, the glasses in Fig 6. show a case were the collision mesh construction is close to failure with the thin, curved surfaces of the lenses.
Minor Notes and Typos:
- Software Packages are cited inconsistently. Sometimes as footnotes, sometimes in the references. I recommend making it more concise.
- I did not find any information about the Manipulation Knowledge annotations, such as number of Grasps per Object. The definition of affordance is also unclear to me. I believe this work uses the robot centric affordance definition of "which area is graspable". But I also am familiar with what is sometimes called "Visual Affordances": Any interactions an object offers, such as grasp, but also pour, cut, clean, sit, (e.g 3D AffordanceNet or Artiverse). I recommend making this clear once.
7 comments
Rebuttal by Authors
W1: The test protocol covers only 200 random samples; one asset is modeled upside down; it is unclear how physical parameters were validated and whether errors were corrected.
Thank you for examining the released assets so closely, and especially for finding the upside-down drawer.
(1) Full-dataset results and sampling. We ran the test protocol on the full dataset and added the results as a new last row of Table 5. Static, Settling, Push, and Grasp test the whole object, so they cover all 1,079 assets; Articulation and Actuation apply only to movable parts and cover the 617 articulated assets. Results: 88.4% / 85.6% / 92.1% / 90.4% / 91.3% / 72.9% (Static / Settling / Push / Articulation / Grasp / Actuation).
(2) The upside-down asset. We confirmed and corrected the orientation of batch_03_0001_StorageFurniture_drawer. The error came from asset format conversion and version management. We then rechecked all 1,079 assets with an automated orientation check and manual visual inspection, and repaired 9 assets in total, including this drawer. The assets in our anonymous Hugging Face release have been replaced with the repaired versions.
(3) Physical-parameter validation and corrections. We are sorry that this was not clear in the submission. In our pipeline, physical parameters are validated in simulation for stability and executability: each exported asset is checked in simulation before release (Sec. 3.5). The check exercises the asset through a range of simulated interactions, including actuating its joints, testing collisions, applying external forces, and manipulating it with a robot arm, and these checks are highly automated with scripts. Problems found in this check are corrected in the final verification, which takes about 5 of the roughly 60 minutes of human time per asset (Table 8).
In addition, the original submission did not compare the parameters with real measurements, so we added three comparisons. On 40 weighed objects from 19 categories, whole-object mass has a median absolute percentage error of 14.61% (mean 27.55%). In cloth-pulling tests on 60 objects, friction has leave-one-object-out mean absolute errors of 0.106 (static) and 0.101 (dynamic). Force-replay tests on a drawer, a pump head, and a magnetic door give normalized-position root-mean-square errors (RMSEs) of 0.025, 0.116, and 0.030, with parameters set before the recordings and not tuned to them. These errors are small relative to the ranges that SIMPLER found tolerable: the median relative errors of mass and friction (15–32%) are far smaller than the eightfold variations under which SIMPLER [2] found sim-to-real ranking agreement unchanged. The protocols and errors are in Appendix C.2 (Table 9), Appendix C.3 (interaction parameters), and Appendix C.4 (Figures 8 and 9).
Rebuttal by Authors
Q1: (1.1) Simulation and render speed of high-polygon assets in cluttered scenes, compared with PartNet-Mobility; (1.2) quality and speed of collision meshes, e.g., the thin, curved lenses of the glasses in Fig. 6 of the submission.
Thanks for your questions. We added three sets of experiments to answer. First, we timed rendering and physics in scenes of 16 random assets from our dataset and from PartNet-Mobility. Second, we reduced the visual triangles of 10 assets by about 90% while keeping their collision meshes, and measured the effect on appearance, geometry, and speed. Third, we varied the collision-mesh budget (16, 64, and 256 convex hulls per part) and checked the released collision mesh of the glasses in Fig. 6 against its visual mesh. In short, in Isaac Sim the high triangle count has little measurable effect on rendering and almost none on physics, because physics uses only the separate collision meshes. The extra physics cost of our assets comes from their finer collision decomposition, which improves contact accuracy. We agree that the ~300k-face budget is a uniform pipeline setting rather than a per-asset optimum. We chose it so that the released assets keep as much geometric detail as users may need, and we also provide a mesh decimation script so that users can reduce the face count for their own needs.
(1.1) Rendering and physics speed with 16 objects. In scenes with 16 random assets, rendering time is about the same for the two datasets (26.20 vs. 26.50 ms per frame), and physics stepping is 4.3 times slower with our assets (11.15 vs. 2.58 ms per step). We consider this speed acceptable; users who need faster stepping can choose a smaller hull budget (see (1.2)).
We randomly selected 20 assets from each of RoboPhyScan and PartNet-Mobility and built three scenes with 16 different assets each, plus single-object scenes. Rendering was timed in Isaac Sim 6.0 on an RTX 5060 Laptop GPU at 512×512 without stepping physics. Physics was timed with the same three object sets, at original scale, in Isaac Sim 5.1 with CPU physics at 120 Hz on an Intel Core Ultra 5 225H, with rendering off; each dataset kept its own collision geometry, mass, materials, and joints. Values are means over the three scenes; triangle counts are scene medians (Appendix F, Tables 18 and 20):
| 16-object scene | RoboPhyScan | PartNet-Mobility |
|---|---|---|
| Visual triangles | 4,779k | 176k |
| Render time per frame | 26.20 ms | 26.50 ms |
| Rigid bodies | 57.0 | 42.3 |
| Collision shapes | 4,477 | 132 |
| Physics time per step | 11.15 ms | 2.58 ms |
Rendering is similar despite 27 times as many triangles (single objects: 26.02 vs. 26.18 ms). An empty scene takes 26.99 ms per frame, so in this setup frame time is dominated by fixed overhead. The test shows that 4.8M triangles add no measurable cost here, but it cannot resolve smaller differences, and larger scenes or other renderers may show a measurable cost.
Physics runs at about 90 steps per second with our assets, or 0.75× real time at 120 Hz on this laptop CPU. The difference most likely comes from collision geometry: our scenes have 34 times as many collision shapes (about 79 convex pieces per rigid body, compared with about 3) and 35% more rigid bodies, while removing about 90% of visual triangles changes physics-only throughput by less than 1% (see below). The finer decomposition lets collision meshes follow the scanned surface, so contacts are computed on the actual shape of handles, rims, and thin parts rather than on a few coarse hulls; Figure 11 shows the convex pieces for three assets.
Effect of visual decimation. On 10 assets, we reduced visual triangles by about 90% while keeping collision meshes and all physical settings. Many assets keep their overall appearance (mean foreground SSIM 0.9573 over 120 view pairs), but local details are visibly lost at USB openings, box-lid edges, and parts of a chair (Figure 13). Geometrically, on three 3D-printed objects whose exact geometry is known from their source files (Table 3), mean surface distance rises from 0.17 mm at 300k faces to 0.26 mm at 10% (30k) and 1.05 mm at 1% (3k). For speed, with 1, 16, and 64 replicas (RTX 5070 Ti Laptop GPU, Isaac Sim 5.1.0), the speedup is 1.0071×, 1.0010×, and 0.9996× for physics only, and 1.0120×, 1.0236×, and 1.0053× with RGB rendering (Appendix G).
Together with the 16-object scenes above, the decimation test shows that decimating visual meshes changes neither rendering nor physics speed much in Isaac Sim, while physics speed depends on the collision budget. As you noted, the high triangle count is useful for vision tasks, and detail cannot be recovered from a decimated release, so we keep the full-resolution visual meshes. The decimation script is in our anonymous code repository (https://github.com/RoboPhyScan/RoboPhyScan).
Rebuttal by Authors
Thank you for the supportive and hands-on review. Your inspection of our assets in Blender uncovered an orientation error that led us to check the full dataset and repair all affected assets. Our responses follow.
On the venue question. Thank you for stating that such work should be recognized; we should have explained the fit to ICLR more clearly. We agree that RoboPhyScan is primarily a dataset contribution, not a fully automated pipeline or a suite of benchmark tasks. We see three reasons why it fits ICLR:
Statistical significance of the imitation-learning results. Following your recommendation in the strengths section, we added significance tests. Each task–setting pair has 20 real-world trials, so we pooled the four tasks with a Cochran–Mantel–Haenszel test stratified by task (Table 26). Here R-n and S-n denote n real and n simulated demonstrations. For the original experiments, R-150+S-150 with our assets significantly outperforms R-150 for both DP and π0.5 (p = 0.006 for both). Our added experiments also show that R-150+S-150 with our assets significantly outperforms the same setting with PartNet-Mobility assets (p = 0.014 and 0.013), and that its difference from R-300 is not significant (p = 0.63 and 0.86).