- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work introduces AnnotateAnything, a framework for annotating 3D assets for robot interactions. Object assets are annotated with keypoints and grasp candidates. Articulated objects are labeled with grasp candidates focussing on the handles and movement trajectories for articulating the joints, e.g. opening doors. Deformable cloth assets are annotated with keypoints. And room level scenes get labeled with occupancy maps and navigation targets.
This annotation is performed in an automated manner by using foundation modesl for 3D part segmentation and VLMs as knowledge base for retrieving possible "skills", which are used to run the specific annotation procedure. The generated grasp candidates and trajectories are validated in Isaac Sim.
The authors demonstrate downstream applications such as supervised keypoint detection, VQA, but also robot manipulation.
Critical strengths
- Fully automated annotation pipeline which can handle multiple asset classes, such as rigid and deformable objects. The annotations also include target trajectories for different atomic skills. The annotations are validated in physics simulation.
Critical weaknesses
-
The real robot experiments seem suboptimal in demonstrating the sim-to-real transfer claimed as a contribution. To be fair, I have limited experience with dexterous grasping. However, cube pickup and toast insertion appear very simple to me, due to the object symmetry and abundance of grasp candidates. Same with towel folding. I believe tasks with clearer constraints such as picking up a mug, or pouring a bottle, which are examples mentioned in the method section, would make more convincing experiments. For deformable objects, T-Shirt folding would also introduce more object level constraints, demonstrating working keypoint-transfer.
I also missed experiments for the other two claims: Articulation and Room Scale Navigation.
Questions for authors
If possible, I would like to see how sim-to-real transfer performs on more objects, such as mug pick and place, or t shirt folding.
Minor Notes
- I recommend another revision focusing on consistency. For example, Fig. 2 and the Appendix mentions Hunyuan Part Segmentation. However, this is not mentioned in the main text. Critical information such as the used object libraries are only detailed in the appendix.
4 comments
Dear Reviewer yXoT, thank you again for the concrete suggestions. Following them, the revision adds the two tasks you proposed, mug pick-and-place (13/20) and T-shirt folding (4/10), together with a real-robot test of the grasp validator (Sec. 5.6, App. H.7; videos in the supplementary). As the discussion period ends soon, we would be glad to know whether this addresses your concerns.
I apologize for my delayed response. The newly added real-robot experiments on less forgiving tasks strengthen the contribution and address my main concerns.
I acknowledge the other reviewers’ concerns about the novelty of the individual components. However, I believe that a good pipeline is a valuable contribution in itself. But I think the paper’s presentation undersells this, which may explain why it reads as incremental. I therefore maintain my score of 3 (weak accept) and support acceptance.
Thank you for taking the time to reply, and for your support. We agree with your point about the presentation. In the final version we will put the pipeline as a whole at the center of the introduction and Sec. 3.
Thank you for the positive assessment and the concrete suggestions.
W1 / Q (more constrained real tasks). We agree that the cube and the towel are forgiving because many grasps work, and added the two cases you suggested (Sec. 5.6, App. H.7; videos in the supplementary):
Since a mug is constrained by its handle, we also tested the validator on the real robot (our reply to xkZN, Q3): 300 single grasps on two mugs, two bottles, a screwdriver and an apple. On the mug handles, simulation and robot agree in every trial: all 25 handle grasps the validator accepts succeed, and all 25 it rejects fail (App. H.7).
W2 (articulation and room-scale navigation). Real-world articulation results are in the paper, though only in the appendix; Sec. 5.6 now points to them: with pi0.5 fine-tuned only on our simulation data, Open Drawer succeeds in 33/40 trials and Close Lid in 53/60 (Table 42). Room-scale navigation is evaluated in simulation: on 418 audited pairs, 94.2% are ready and 96.5% execute on the full robot in simulation (Table 3; rollouts in Table 38). We did not deploy navigation on hardware, and Sec. 6 now states this.
Hunyuan naming. Fixed: Sec. 3.1 now names "the Hunyuan3D part-segmentation model, which builds on P3-SAM and X-Part", as in Fig. 2 and App. D.
Asset libraries. Sec. 5.5 now lists the nine sources by asset type (rigid: Objaverse, ShapeNet, OmniObject3D; articulated: PartNet-Mobility, PhysX-Mobility; garments: GarmentLab, ClothesNet; rooms: AgentWorld, GRScenes) with a pointer to Table 49 for counts and licenses.
Thank you again. We hope the new real-robot results address your concerns; if so, we would appreciate it if you would consider raising your score.