acceptodds
Under review as a conference paper at ICLR 2027

RoboPhyScan: A Large-Scale Sim-Ready Real-World Object Dataset with Manipulation Annotations

Abstract

Robotic manipulation requires object assets that are not only visually realistic, but also sim-ready and reproducible as real-world instances. Existing resources typically provide either scan-derived real-world appearance or simulation metadata, but rarely both for large-scale purchasable objects. Thus, we introduce RoboPhyScan, a large-scale high-quality sim-ready real-world object dataset that contains 1,079 purchasable everyday object instances across 65 object types. Each asset aligns high-fidelity scanned geometry, reconstructed texture, precise articulation, rich physical properties, and manipulation annotations covering part pose, affordance, and grasp pose. Furthermore, we design an efficient real-to-sim assetization pipeline to reduce active human production time to approximately 60 minutes per asset on average. We also systematically evaluate our assets, including visual quality, sim-ready stability, and utility in downstream robotic applications. With RoboPhyScan, we provide a bridge from purchasable real objects to high-quality sim-ready assets, establishing a basis for reproducible sim-real manipulation research.

Primary area:
datasets and benchmarks
Keywords:
Sim-Ready, Real-World, Manipulation Annotations

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_6tESOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This work introduces RoboPhyscan, an object dataset with nearly 1.1k objects for simulation based on purchasable everyday objects. Each object requires 60min of human effort for 3D scanning, texture photographing, asset cleanup and annotations. These annotations include physical properties required for simulation, like mass, inertia and articulated joints with friction and dampening, and optimized collision meshes. But also "semantic" annotations such as Affordances and Parallel-Jaw Gripper annotations.

The dataset quality is evaluated for sim readiness in Isaac Sim by a test prototol looking at stability on tests such as drop, push or grasp.

Imitation learning experiments demonstrate that simualted trajectories based on these assets can improve downstream task success.

Critical strengths
  • Large scale dataset with multiple annotations. Mesh and Texture Quality are very high, which enables multiple downstream applications beyond robot simulations. I see potential interest for Computer Vision community with point cloud processing, grasp prediction and other applications, where a library of high quality object scans are helpful.
  • Imitation Learning Experiments explore downstream usage. However, I recommend explicitly reporting statistical significance.
Critical weaknesses
  • Data Quality. The test protocol is limited to only 200 random samples of the nearly 1.1k assets. I quickly explored some of the assets provided as supplementary material in the 3D software Blender. batch_03_0001_StorageFurniture_drawer is modeled upside down. I could also not find how the physical parameters were validated, and if the found errors were corrected in any way.

  • I want to mention it for completeness sake: As I understand this work is mainly a "raw" dataset contribution. The collection required substantial human effort to 3D scan and process the objects into simulation ready digital assets. It is not a fully automated pipeline. And it is a collection of 3D assets for downstream tasks, not a suite of test ready benchmark tasks. I strongly believe that such work is helpful to the scientific community and should be recognized. However, one could discuss if ICLR is the best venue for such a dataset contribution.

Questions for authors

Questions

  1. Assessment: Mesh Performance. While the high triangle count is great for downstream vision tasks requiring this fidelity, does it have an impact on simulation performance? Right now I suspect that most assets are at approx. 300k triangles due to an artifact of the pipeline, not because this is the optimal setting for the asset. Quick tests in blender indicate, that 1%-10% of the triangle count is sufficient without obvious visual degradation.

    1.1 What is the impact on simulation / render speed of these high quality assets in cluttered scenes, e.g. 10-20 random RoboPhyscan assets vs 10-20 random PartNet-Mobility assets?

    1.2 How about the quality / speed impact of collision meshes? For example, the glasses in Fig 6. show a case were the collision mesh construction is close to failure with the thin, curved surfaces of the lenses.

Minor Notes and Typos:

  • Software Packages are cited inconsistently. Sometimes as footnotes, sometimes in the references. I recommend making it more concise.
  • I did not find any information about the Manipulation Knowledge annotations, such as number of Grasps per Object. The definition of affordance is also unclear to me. I believe this work uses the robot centric affordance definition of "which area is graspable". But I also am familiar with what is sometimes called "Visual Affordances": Any interactions an object offers, such as grasp, but also pour, cut, clean, sit, (e.g 3D AffordanceNet or Artiverse). I recommend making this clear once.
7 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for the supportive and hands-on review. Your inspection of our assets in Blender uncovered an orientation error that led us to check the full dataset and repair all affected assets. Our responses follow.

On the venue question. Thank you for stating that such work should be recognized; we should have explained the fit to ICLR more clearly. We agree that RoboPhyScan is primarily a dataset contribution, not a fully automated pipeline or a suite of benchmark tasks. We see three reasons why it fits ICLR:

  1. We submitted to the primary area "datasets and benchmarks", and the ICLR 2027 call for papers lists both "datasets and benchmarks" and "applications to robotics, autonomy, planning" as topics.
  2. The dataset provides data that learning methods need and that existing resources rarely combine: sim-ready articulated assets, clean mesh topology, real textures, and manipulation annotations. Each asset is paired with a purchasable object, so simulation and real-world experiments can use the same objects. This supports reproducible sim-to-real manipulation research, building on YCB [1], which shares the same real objects across labs, and on real-to-sim evaluation such as SIMPLER [2] and REALM [3]. As you noted, the same assets can also serve vision tasks such as point-cloud processing and grasp prediction.
  3. The experiments evaluate the data for learning. In the original experiments, adding 150 simulated demonstrations from our assets to 150 real ones raises mean real-world success from 31.3% to 53.8% for Diffusion Policy [4] (DP) and from 47.5% to 70.0% for π0.5 [5], and policies trained only on the simulated demonstrations already reach 17.5% and 33.8% mean success on the real robot. Our added experiments also show that the gain comes from the assets rather than from data volume alone: at an equal 300-demonstration budget, our simulated demonstrations recover 82% (DP) and 90% (π0.5) of the gain from extra real demonstrations, versus 9% and 10% with PartNet-Mobility [6] assets under the same scenes, planner, and budget. Both the gain from adding simulated demonstrations and the difference from PartNet-Mobility assets are significant (see the next paragraph).

Statistical significance of the imitation-learning results. Following your recommendation in the strengths section, we added significance tests. Each task–setting pair has 20 real-world trials, so we pooled the four tasks with a Cochran–Mantel–Haenszel test stratified by task (Table 26). Here R-n and S-n denote n real and n simulated demonstrations. For the original experiments, R-150+S-150 with our assets significantly outperforms R-150 for both DP and π0.5 (p = 0.006 for both). Our added experiments also show that R-150+S-150 with our assets significantly outperforms the same setting with PartNet-Mobility assets (p = 0.014 and 0.013), and that its difference from R-300 is not significant (p = 0.63 and 0.86).

AuthorsOpenReview ↗

Rebuttal by Authors

W1: The test protocol covers only 200 random samples; one asset is modeled upside down; it is unclear how physical parameters were validated and whether errors were corrected.

Thank you for examining the released assets so closely, and especially for finding the upside-down drawer.

(1) Full-dataset results and sampling. We ran the test protocol on the full dataset and added the results as a new last row of Table 5. Static, Settling, Push, and Grasp test the whole object, so they cover all 1,079 assets; Articulation and Actuation apply only to movable parts and cover the 617 articulated assets. Results: 88.4% / 85.6% / 92.1% / 90.4% / 91.3% / 72.9% (Static / Settling / Push / Articulation / Grasp / Actuation).

(2) The upside-down asset. We confirmed and corrected the orientation of batch_03_0001_StorageFurniture_drawer. The error came from asset format conversion and version management. We then rechecked all 1,079 assets with an automated orientation check and manual visual inspection, and repaired 9 assets in total, including this drawer. The assets in our anonymous Hugging Face release have been replaced with the repaired versions.

(3) Physical-parameter validation and corrections. We are sorry that this was not clear in the submission. In our pipeline, physical parameters are validated in simulation for stability and executability: each exported asset is checked in simulation before release (Sec. 3.5). The check exercises the asset through a range of simulated interactions, including actuating its joints, testing collisions, applying external forces, and manipulating it with a robot arm, and these checks are highly automated with scripts. Problems found in this check are corrected in the final verification, which takes about 5 of the roughly 60 minutes of human time per asset (Table 8).

In addition, the original submission did not compare the parameters with real measurements, so we added three comparisons. On 40 weighed objects from 19 categories, whole-object mass has a median absolute percentage error of 14.61% (mean 27.55%). In cloth-pulling tests on 60 objects, friction has leave-one-object-out mean absolute errors of 0.106 (static) and 0.101 (dynamic). Force-replay tests on a drawer, a pump head, and a magnetic door give normalized-position root-mean-square errors (RMSEs) of 0.025, 0.116, and 0.030, with parameters set before the recordings and not tuned to them. These errors are small relative to the ranges that SIMPLER found tolerable: the median relative errors of mass and friction (15–32%) are far smaller than the eightfold variations under which SIMPLER [2] found sim-to-real ranking agreement unchanged. The protocols and errors are in Appendix C.2 (Table 9), Appendix C.3 (interaction parameters), and Appendix C.4 (Figures 8 and 9).

AuthorsOpenReview ↗

Rebuttal by Authors

Q1: (1.1) Simulation and render speed of high-polygon assets in cluttered scenes, compared with PartNet-Mobility; (1.2) quality and speed of collision meshes, e.g., the thin, curved lenses of the glasses in Fig. 6 of the submission.

Thanks for your questions. We added three sets of experiments to answer. First, we timed rendering and physics in scenes of 16 random assets from our dataset and from PartNet-Mobility. Second, we reduced the visual triangles of 10 assets by about 90% while keeping their collision meshes, and measured the effect on appearance, geometry, and speed. Third, we varied the collision-mesh budget (16, 64, and 256 convex hulls per part) and checked the released collision mesh of the glasses in Fig. 6 against its visual mesh. In short, in Isaac Sim the high triangle count has little measurable effect on rendering and almost none on physics, because physics uses only the separate collision meshes. The extra physics cost of our assets comes from their finer collision decomposition, which improves contact accuracy. We agree that the ~300k-face budget is a uniform pipeline setting rather than a per-asset optimum. We chose it so that the released assets keep as much geometric detail as users may need, and we also provide a mesh decimation script so that users can reduce the face count for their own needs.

(1.1) Rendering and physics speed with 16 objects. In scenes with 16 random assets, rendering time is about the same for the two datasets (26.20 vs. 26.50 ms per frame), and physics stepping is 4.3 times slower with our assets (11.15 vs. 2.58 ms per step). We consider this speed acceptable; users who need faster stepping can choose a smaller hull budget (see (1.2)).

We randomly selected 20 assets from each of RoboPhyScan and PartNet-Mobility and built three scenes with 16 different assets each, plus single-object scenes. Rendering was timed in Isaac Sim 6.0 on an RTX 5060 Laptop GPU at 512×512 without stepping physics. Physics was timed with the same three object sets, at original scale, in Isaac Sim 5.1 with CPU physics at 120 Hz on an Intel Core Ultra 5 225H, with rendering off; each dataset kept its own collision geometry, mass, materials, and joints. Values are means over the three scenes; triangle counts are scene medians (Appendix F, Tables 18 and 20):

16-object sceneRoboPhyScanPartNet-Mobility
Visual triangles4,779k176k
Render time per frame26.20 ms26.50 ms
Rigid bodies57.042.3
Collision shapes4,477132
Physics time per step11.15 ms2.58 ms

Rendering is similar despite 27 times as many triangles (single objects: 26.02 vs. 26.18 ms). An empty scene takes 26.99 ms per frame, so in this setup frame time is dominated by fixed overhead. The test shows that 4.8M triangles add no measurable cost here, but it cannot resolve smaller differences, and larger scenes or other renderers may show a measurable cost.

Physics runs at about 90 steps per second with our assets, or 0.75× real time at 120 Hz on this laptop CPU. The difference most likely comes from collision geometry: our scenes have 34 times as many collision shapes (about 79 convex pieces per rigid body, compared with about 3) and 35% more rigid bodies, while removing about 90% of visual triangles changes physics-only throughput by less than 1% (see below). The finer decomposition lets collision meshes follow the scanned surface, so contacts are computed on the actual shape of handles, rims, and thin parts rather than on a few coarse hulls; Figure 11 shows the convex pieces for three assets.

Effect of visual decimation. On 10 assets, we reduced visual triangles by about 90% while keeping collision meshes and all physical settings. Many assets keep their overall appearance (mean foreground SSIM 0.9573 over 120 view pairs), but local details are visibly lost at USB openings, box-lid edges, and parts of a chair (Figure 13). Geometrically, on three 3D-printed objects whose exact geometry is known from their source files (Table 3), mean surface distance rises from 0.17 mm at 300k faces to 0.26 mm at 10% (30k) and 1.05 mm at 1% (3k). For speed, with 1, 16, and 64 replicas (RTX 5070 Ti Laptop GPU, Isaac Sim 5.1.0), the speedup is 1.0071×, 1.0010×, and 0.9996× for physics only, and 1.0120×, 1.0236×, and 1.0053× with RGB rendering (Appendix G).

Together with the 16-object scenes above, the decimation test shows that decimating visual meshes changes neither rendering nor physics speed much in Isaac Sim, while physics speed depends on the collision budget. As you noted, the high triangle count is useful for vision tasks, and detail cannot be recovered from a decimated release, so we keep the full-resolution visual meshes. The decimation script is in our anonymous code repository (https://github.com/RoboPhyScan/RoboPhyScan).

Reviewer_g4m5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper introduces RoboPhyScan, a dataset of 1,079 purchasable object spanning 65 object categories, each converted into a simulation-ready asset. All of these assets combines scanned geometry, reconstructed texture, part segmentation and articulation, physical parameters (mass, inertia...), and manipulation annotation such as grasps pose, regions of affordance, and part pose. The authors also introduce a real-to-sim assetization pipeline described as follow : i) handheld scanning followed by mesh cleaning and Poisson reconstruction, ii) multi-view texture and PBR reconstruction, iii) template-based part segmentation and articulation built on ConceptFactory, with human correction, iv) physical parameters from direct measurement or material priors, v) manipulation annotations propagated from category-level concept templates.

Critical strengths
  1. The dataset provides manipulation annotation for each assets that is very useful for robotic manipulation.
  2. The quality of the mesh provided.
Critical weaknesses
  1. Mesh quality evaluation : the authors state that triangle count characterizes “mesh complexity and geometric accuracy” (Sec. 4.1). However, triangle count mainly reflects mesh complexity, not fidelity to the real object. The near-zero boundary-edge ratio is also partly expected by construction: Screened Poisson reconstruction produces watertight surfaces, so this metric reflects the pipeline design rather than geometric quality. No metric measures geometric accuracy with respect to the real object, for example dimensional measurements on a subset of objects, or the deviation introduced by Poisson reconstruction and Laplacian smoothing.

  2. Texture quality evaluation : the authors compare their textures with assets reconstructed by SAM3D and Hunyuan3D 2.1. This comparison seems unfair: SAM3D and Hunyuan3D 2.1 are zero-shot methods that produce an asset from a single image. The proposed pipeline relies on a high-precision scan and many photographs from multiple views (Sec. 3.3). The advantage of RoboPhyScan is therefore expected and not very informative about the quality of the texture reconstruction itself.

  3. Policy evaluation : The authors train Diffusion Policy from scratch and fine-tune π0.5 using real and simulated demonstrations, both generated by their motion planner with the manipulation annotations. This evaluation raises several issues: i)there is no comparison with another baseline, such as simulated demonstrations generated from another object asset dataset, so the gains cannot be attributed to the quality of RoboPhyScan assets. ii)Adding simulated data improves real-world success rates, but this may simply be due to adding more training data. A real-only setting with the same total number of demonstrations (e.g., R-300) is missing. iii)Overall, this evaluation does not demonstrate the specific value of RoboPhyScan assets.

4.Physical fidelity. Despite the emphasis on physical properties, the physical parameters are not validated against the real objects. Friction coefficients come from material priors, and the values of the modular interaction parameters (damping, spring stiffness, magnetic thresholds; Appendix C.2) are not justified. No comparison of dynamic behavior between simulation and the real objects is provided.

Questions for authors
  1. Did you compare the texture quality of your assets with other scanned object datasets (e.g., GSO, OmniObject3D, DTC)?
  2. For Diffusion Policy training, did the loss converge well? Could the image resolution (96×96) explain the low real-world success rates, especially for the InsertUSB task?
  3. Did you conduct an ablation study on the quality of your manipulation annotations? Why is it better to use your precomputed annotations than a model that directly predicts them from the object asset?
8 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for the detailed and rigorous reviews. Your points on geometric accuracy, fair texture baselines, data-volume confounds, and physical fidelity are well taken, and we added an experiment for each of them. The main new results are:

  • geometric accuracy: against 3D-printed references, a mean surface distance of 0.170 mm and an [email protected] mm of 99.08% (W1);
  • texture: better scores than DTC [1], GSO [2], and OmniObject3D [3] on all four metrics for each of six matched purchased objects (W2 and Q1);
  • asset value: at an equal 300-demonstration budget, simulated demonstrations from our assets recover 82% (Diffusion Policy [4], DP) and 90% (π0.5 [5]) of the gain from extra real demonstrations, versus 9% and 10% with PartNet-Mobility [6] assets, a significant difference (W3);
  • physical fidelity: a median mass error of 14.61% on 40 objects, mean absolute friction-coefficient errors of about 0.10 on 60 objects, and force-replay errors of 3.8 mm, 2.3 mm, and 2.1° on three interaction types (W4).

Our responses follow the order of your review.

W1: Triangle count reflects mesh complexity, not fidelity; no metric measures geometric accuracy; the near-zero boundary-edge ratio is partly expected by construction.

We thank the reviewer for the insightful viewpoint that triangle count reflects mesh complexity rather than fidelity to the real object. We revised how the existing metrics are presented in Sec. 4.1 and added a direct accuracy metric.

Geometric accuracy. We designed a reference-geometry evaluation that measures the deviation of the full pipeline, including Poisson reconstruction, and separately isolates Laplacian smoothing. We used 3D-printed references because their source STL files provide exact ground-truth surfaces, whereas measuring a retail object gives only a few dimensions. We printed three reference STL models (a cat, a bunny, and an owl; nominal sizes 120, 140, and 150 mm) on a Bambu Lab H2D with the Vision Encoder, using PLA, a 0.4 mm nozzle, and 0.20 mm layers. We then scanned and reconstructed them with our pipeline and compared the results with the STLs after rigid registration without scaling. Results averaged over the three objects (Table 3):

Target faces1000k300k (release)100k30k3k
Mean surface distance (mm)0.16370.17020.18980.25821.0485

At 300k faces, P95 distance is at most 0.35 mm in both directions and [email protected] mm is 99.08%. Laplacian smoothing alone adds 0.0007 mm to the mean distance on undecimated surfaces. Printing error is not subtracted (the manufacturer specifies 50 µm motion accuracy with the Vision Encoder; Bambu Lab), so these distances are conservative estimates of reconstruction error. Figure 10 (left) shows the reference STLs, printed objects, scans, and the five face budgets.

The full per-object results and protocol are in Appendix D.2 and Table 11.

AuthorsOpenReview ↗

Rebuttal by Authors

W2 and Q1: The texture comparison with SAM3D and Hunyuan3D 2.1 seems unfair. Did you compare texture quality with other scanned datasets such as GSO, OmniObject3D, or DTC?

Thank you for pointing out that single-image generators and multi-view scanning start from very different inputs. We compare against leading generative methods, SAM 3D [7] and Hunyuan3D 2.1 [8], to show why reconstruction is necessary. The comparison is restricted to the view from which each object is generated. For a comparison in which both sides start from real captures, we followed your suggestion in Q1 and compared with the three scanned datasets you named.

Comparison with scanned datasets. DTC, GSO, and OmniObject3D do not publish purchase links, so matching objects are hard to find. After searching retail listings, we bought six objects that are identical or closely matched to dataset assets: one from DTC (birdhouse), two from GSO (crayon box, eagle), and three from OmniObject3D (two calculators, a table-tennis bat). We scanned and reconstructed each object with our pipeline. We then rendered both our reconstruction and the original dataset asset under the same Blender Cycles PBR settings and compared them against the same 421 calibrated photographs we took. Results (ours / original dataset; Table 4):

DatasetObjectsPhotosLPIPS↓DISTS↓CLIP↑DINOv2↑
DTC1720.275 / 0.4370.186 / 0.2800.937 / 0.8920.868 / 0.836
GSO21440.279 / 0.4240.182 / 0.2620.930 / 0.8920.840 / 0.766
OmniObject3D32050.286 / 0.4130.214 / 0.3190.931 / 0.8720.845 / 0.782

Our reconstructions score better on all four metrics for each of the six objects (Table 12). Figure 10 (right) shows real photographs, our renderings, and dataset renderings from the same view. We have added this experiment to Appendix D.3.

AuthorsOpenReview ↗

Rebuttal by Authors

W3: (i) No comparison with another asset dataset, (ii) gains may come from more training data and a real-only R-300 is missing, (iii) the value specific to RoboPhyScan assets is not shown.

Thank you for raising these reasonable concerns. To answer these questions, we added two baselines: an equal-budget real-only baseline (R-300) and an asset-source baseline in which our assets are replaced by PartNet-Mobility (PNM) assets.

Setup. For R-300, we collected 150 more real demonstrations per task. For the PNM baseline, we took the PNM instance closest to each real task object from the matching category (Bottle, Dispenser, USB, Stapler), scaled it to the real object's size, and manually annotated the grasp pose, contact region, and motion axis that PNM does not provide. Scenes, cameras, randomization, the motion planner, and the budget of 150 successful trajectories were identical to our setting. Real-world success rates (DP: Diffusion Policy; 20 trials per cell):

PolicyDataSim. assetsTwistCapPressShampooInsertUSBOpenStaplerMean
DPR-150–20%40%25%40%31.3%
DPR-300–50%60%50%75%58.8%
DPS-150PNM0%5%0%10%3.8%
DPS-150Ours10%25%10%25%17.5%
DPR-150+S-150PNM15%50%20%50%33.8%
DPR-150+S-150Ours45%60%40%70%53.8%
π0.5R-150–35%60%40%55%47.5%
π0.5R-300–60%75%75%80%72.5%
π0.5S-150PNM5%15%0%20%10.0%
π0.5S-150Ours25%45%25%40%33.8%
π0.5R-150+S-150PNM35%70%35%60%50.0%
π0.5R-150+S-150Ours55%80%65%80%70.0%

(i) Comparison with another asset dataset. Scenes, planner, and budget are identical in the two settings. For S-150, replacing our assets with PNM lowers mean success from 17.5% to 3.8% (DP) and from 33.8% to 10.0% (π0.5). For R-150+S-150, it lowers success from 53.8% to 33.8% and from 70.0% to 50.0%. All four differences are significant under a Cochran–Mantel–Haenszel (CMH) test pooled over the four tasks (S-150: p = 0.010 for DP and p < 0.001 for π0.5; R-150+S-150: p = 0.014 and 0.013; Table 26).

(ii) Data volume. The two mixed settings have the same 300-demonstration budget and differ only in the asset source. With our assets, the 150 simulated demonstrations recover 82% (DP) and 90% (π0.5) of the gain that 150 additional real demonstrations give over R-150; with PNM assets they recover 9% and 10%. If data volume alone explained the gain, the two mixed settings would perform similarly. R-300 remains the strongest setting (58.8% for DP, 72.5% for π0.5), above R-150+S-150 with our assets (53.8% and 70.0%), although this difference is not significant (CMH test, p = 0.63 and 0.86).

The two kinds of demonstrations also differ in cost. Each real demonstration needs the physical robot and a manual scene reset, whereas simulated demonstrations from our assets are generated by the motion planner without robot time and can be generated in parallel at scale. Simulated data with our assets that recovers most of the gain of real data is therefore a practical way to scale up training data.

(iii) Value specific to RoboPhyScan assets. A PNM instance shares the category and rough shape of the real object; a RoboPhyScan asset is a scan of the evaluated object itself, with its geometry, articulation, appearance, and physical parameters. Because the two settings ("Ours" and "PNM") differ only in the asset source, the gaps in (i) and (ii) measure the value of assets that match the real objects.

We added these rows to Table 6, the controls and a fourth finding to Sec. 4.3, and the baseline protocol to Appendix H.3.

Reviewer_qQiSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RoboPhyScan contains 1,079 real-world object instances from 65 object types. The assets include scanned geometry, reconstructed textures, articulated parts, physical properties, and annotations for part poses, affordances, and grasps. The authors also describe a pipeline for turning these objects into simulation-ready assets, with human correction and simulation checks. They report an average of about 60 minutes of active human work per validated asset. The paper compares RoboPhyScan with ArtVIP and PartNet-Mobility and demonstrates its use in motion planning and policy learning.

Critical strengths

One useful aspect of RoboPhyScan is that each scanned asset is linked to a real object that can be purchased. The dataset also provides the joint structure, physical properties, and manipulation annotations, so it offers more than just the 3D models themselves. This could be useful for researchers who want to use the same objects in simulation and real-world experiments.

Critical weaknesses

Table 4 shows that RoboPhyScan performs better than ArtVIP on all six sim-ready tests, but I am not sure how much this tells us about the actual physical accuracy of the assets. These tests mainly check whether the objects are stable and can be manipulated in simulation. ArtVIP (Jin et al., 2026), on the other hand, also compares simulated joint motion with motion captured from real objects. Since RoboPhyScan still involves estimating physical properties and manually adjusting joints, collision geometry, and motion limits, it is hard to tell what leads to the higher pass rates. Is it the scanned geometry, the parameter assignment, or the manual tuning?

Questions for authors

1.The paper reports about 60 minutes of human work per asset. While some stages have individual time estimates, it is unclear how much time is spent on physical parameter assignment, adjusting joints and collision meshes, and checking the assets in simulation. Are these steps included in the 60 minutes?

2.In Sec. 3.5, mass is estimated from mesh volume and material-density priors when individual parts cannot be weighed, while friction coefficients are assigned using material priors inferred by GPT-5.5. Have the authors compared these estimates with real-world measurements? If so, what are the errors in mass and friction coefficients?

6 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for the careful and balanced review. Your questions on what the sim-ready tests measure, what drives the higher pass rates, the 60-minute estimate, and the accuracy of the physical parameters are well taken. The main additions are:

  • physical accuracy: force-replay tests on a real drawer, pump head, and magnetic door, with position errors of about 3.8 mm, 2.3 mm, and 0.036 rad (W1);
  • attribution: a physical-parameter ablation on the same 200 objects as Table 5 shows that physical parameters have a larger effect on Grasp, Articulation, and Actuation, and that the other differences may come from geometry and collision meshes (W1);
  • time: a per-stage breakdown showing that the 60 minutes include parameter assignment, joints, collision meshes, and simulation checks (Q1);
  • mass and friction: against real measurements, a median whole-object mass error of 14.61% on 40 objects and mean absolute friction-coefficient errors of about 0.10 on 60 objects (Q2).

Below, we provide detailed responses to your comments and concerns.

W1: Table 4 shows higher pass rates, but how much does it say about physical accuracy? Is the gain due to scanned geometry, parameter assignment, or manual tuning?

What Table 4 measures. Thank you for separating stability in simulation from physical accuracy. We agree that the sim-ready tests in Table 4 of the submission (Table 5 in the revision) do not, by themselves, measure physical accuracy. They test stability and executability: whether an asset can be placed, settled, pushed, articulated, grasped, and actuated in simulation without failure. Unlike ArtVIP's [1] joint-trajectory validation, they do not compare simulated motion with real motion. Sec. 4.2 now states this explicitly.

Physical accuracy against real objects. To measure physical accuracy directly, we added comparisons with real-world measurements. We ran force-replay tests, which are close in design to ArtVIP's joint-trajectory validation. We applied force by hand to a real drawer, pump head, and magnetic door through a digital force gauge, recorded the force with the gauge and the joint position from video, replayed the recorded force on the simulated asset, and compared the resulting motion. The interaction parameters were set before these recordings from force measurements on other objects (Appendix C.3) and were not tuned to them. Normalized-position root-mean-square errors (RMSEs) are 0.025, 0.116, and 0.030, about 3.8 mm, 2.3 mm, and 0.036 rad (Sec. 4.2; Appendix C.4, Figures 8 and 9). We also compared mass on 40 objects and friction on 60 objects with real-world measurements; Q2 below gives the numbers.

AuthorsOpenReview ↗

Rebuttal by Authors

What drives the higher pass rates. We address the three factors you listed:

  • Parameter assignment. Physical parameters can be removed without changing the rest of the asset, so we ablated them. In the 245 ArtVIP models we audited, mass or density and friction are each missing from about 79% (Appendix E.1, Table 14). On the same 200 articulated objects as Table 5, we reset these settings of our assets to Isaac Sim defaults and kept geometry, collision meshes, joints, interaction modules, and annotations unchanged (new "default physics" row of Table 5; Appendix E):
DatasetStaticSettlingPushArticulationGraspActuation
RoboPhyScan89.5%86.0%91.0%89.0%92.5%72.5%
RoboPhyScan (default physics)89.5%85.0%89.0%83.0%83.0%67.0%
ArtVIP75.0%82.5%86.0%72.5%80.0%64.5%

Grasp, Articulation, and Actuation drop by 9.5, 6.0, and 5.5 points, while Static, Settling, and Push change by at most 2.0 points. Physical parameters therefore have a larger effect on contact-rich manipulation; these drops correspond to 9.5 of the 12.5-point Grasp gap and 5.5 of the 8.0-point Actuation gap to ArtVIP.

  • Scanned geometry and collision meshes. With default parameters, our assets still score higher than ArtVIP on all six tests, and the Static gap (89.5% vs. 75.0%) is unchanged. Since Static checks physical configuration and collision geometry, this gap and the remaining Articulation gap (10.5 of 16.5 points) may come from geometry, collision meshes, and joint structure. Our geometry comes from scans of the real objects, whereas ArtVIP assets are modeled and assembled by hand by professional 3D modelers [1], and PartNet-Mobility [2] assets come from existing 3D model repositories. Hand assembly can leave joints inconsistent with part poses: in an 80-model subset of ArtVIP screened for joint anchors, 6 have residuals above 1 mm; in one of them, a dishwasher, the solver moves the shelf by 12.4 mm at the first physics step to satisfy the joint constraint.
  • Manual correction. We would like to clarify that there is no separate manual tuning step: joints are instantiated when the annotator fits the concept template to the scanned geometry, without looking at simulation results. Simulation results are used only in final verification, which takes about 5 minutes per asset; only a small number of objects need manual correction in this step (Q1).
AuthorsOpenReview ↗

Rebuttal by Authors

Q1: Are physical parameter assignment, joint and collision-mesh adjustment, and simulation checks included in the 60 minutes?

Yes. The 60 minutes is the average active human time per validated asset and includes all of these steps:

Geometry capture & cleanupTexture photographySegmentation & conceptualizationPhysical parametersFinal verificationTotal
25–30 min5 min20 min5 min5 minabout 60 min

The 20 minutes for segmentation and conceptualization are the 12 minutes for conceptualization and 8 minutes for correcting segmentation that Sec. 3.4 already reports. Physical parameters cover weighing and reviewing the material priors. The final verification (5 minutes) covers inspecting the simulation checks and correcting any issues they reveal.

We apologize that our wording in Sec. 3.4 was unclear, and thank you for raising this question. The sentence "We then adapt joint placement, link geometry, collision support, and motion limits to the individual object" reads as if joints were tuned by hand in a separate step. What it describes is part of conceptualization: joint types, axes, and motion ranges are defined on the category-level concept template, and they are instantiated for each object when the annotator fits the template to that object. There is no separate joint-adjustment stage, so the time spent on joints is included in the 12 minutes of conceptualization. Collision meshes are generated automatically with V-HACD; checking them is part of the final verification.

Unattended runs (Poisson reconstruction, PBR fitting, material inference, Isaac Sim validation scripts) are excluded; inspecting and correcting their outputs is included. The breakdown is in Table 8, and the Sec. 3.4 sentence has been clarified.

Q2: Have the mass and friction estimates been compared with real measurements? What are the errors?

Yes. We compared the assigned masses with scale readings on 40 objects from 19 categories, and the assigned friction coefficients with cloth-pulling measurements on 60 objects. The errors are below (MAE: mean absolute error; APE: absolute percentage error; Table 9), and the protocols follow the table.

QuantityMAEMean APEMedian APE
Whole-object mass (40 objects, 19 categories)58.99 g27.55%14.61%
Component mass54.25 g24.55%16.16%
Static friction (leave-one-object-out)0.10630.77%21.73%
Dynamic friction (leave-one-object-out)0.10128.75%32.29%

Mass. We weighed 40 objects on a scale with 0.1 g resolution, whole when they cannot be disassembled and by component otherwise, and compared the weights with the masses computed from mesh volume and the manually reviewed material priors. For half of the objects, the estimate from scanned geometry and material priors alone is within 15% of the scale reading. The mean error is about twice the median because a few objects account for much of it.

Friction. We pulled 60 objects with representative contact materials (plastic, metal, glass, ceramic) horizontally on the same cloth, measuring the force at sliding onset (static) and at near-constant speed (dynamic). We kept the asset coefficients fixed, fitted the shared cloth parameters on the other 59 objects, and evaluated the prediction on the held-out object. This test covers contact with cloth only; other surface pairs use the same material priors but were not measured separately.

Do errors of this size matter? SIMPLER [3], which also assigns density and friction from material priors, varied object mass from 10 to 80 g and gripper friction from 0.25 to 2.0, both eightfold. Across these ranges, sim-to-real policy rankings remained consistent (a mean maximum rank violation of 0.031 and Pearson r of 0.957–0.990), and the SIMPLER authors report at most a moderate (≤15%) effect on success rates. Our median relative errors, 15–32% across the four quantities above, are far smaller than these eightfold variations. Policies trained only on simulated demonstrations with these parameters reach 17.5% (Diffusion Policy [4]) and 33.8% (π0.5 [5]) mean real-world success (S-150 in Table 6). At the same 300-demonstration budget, replacing 150 real demonstrations with 150 simulated ones (R-150+S-150) reaches 53.8% and 70.0%, close to 58.8% and 72.5% with 300 real demonstrations (R-300; Cochran–Mantel–Haenszel test, p = 0.63 and 0.86). Simulated data generated with these parameters thus recovers 82% and 90% of the gain from the extra real demonstrations.

Sec. 3.5 now summarizes the mass and friction comparison and the SIMPLER reference. The full protocol is in Appendix C.2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.