acceptodds
Under review as a conference paper at ICLR 2027

AnnotateAnything: Automatic Annotation of 3D Assets for Robot Manipulation

Abstract

Simulation enables scalable robot data collection, but 3D assets often lack the interaction annotations needed to specify where and how robots should act. We present AnnotateAnything, an automatic framework that converts passive 3D assets into manipulation-ready assets with language, 3D visual, and executable action annotations. A vision-language model supplies interaction priors—functional parts, keypoints, affordance regions, and compatible skills—that are grounded in each asset's 3D geometry. A physics annotation pipeline converts these priors into action candidates through skill-specific generation, geometric optimization, and GPU-parallel validation, retaining multiple solutions in reusable candidate banks. These banks drive an asynchronous parallel simulation data-collection system. Applied to rigid, articulated, deformable, and room-scale assets, the framework produces about M validated action annotations across nine skill families. On an audited simulation suite, it achieves higher readiness and full-robot execution success than ablations and framework-adapted baselines, as well as higher annotation and collection throughput than the latter. Pretrained policies fine-tuned solely on the generated simulation demonstrations achieve mean success across six real-world tasks on three robot platforms, with trials per task. Project page: https://gilded-cupcake-c582c1.netlify.app/.

Primary area:
datasets and benchmarks
Keywords:
Automatic 3D Asset Annotation, Manipulation-Ready Assets, Visual-Language-Action Annotation, Actionable Robot Labels, Robot Manipulation, Simulation-based Robot Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yXoTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces AnnotateAnything, a framework for annotating 3D assets for robot interactions. Object assets are annotated with keypoints and grasp candidates. Articulated objects are labeled with grasp candidates focussing on the handles and movement trajectories for articulating the joints, e.g. opening doors. Deformable cloth assets are annotated with keypoints. And room level scenes get labeled with occupancy maps and navigation targets.

This annotation is performed in an automated manner by using foundation modesl for 3D part segmentation and VLMs as knowledge base for retrieving possible "skills", which are used to run the specific annotation procedure. The generated grasp candidates and trajectories are validated in Isaac Sim.

The authors demonstrate downstream applications such as supervised keypoint detection, VQA, but also robot manipulation.

Critical strengths
  • Fully automated annotation pipeline which can handle multiple asset classes, such as rigid and deformable objects. The annotations also include target trajectories for different atomic skills. The annotations are validated in physics simulation.
Critical weaknesses
  • The real robot experiments seem suboptimal in demonstrating the sim-to-real transfer claimed as a contribution. To be fair, I have limited experience with dexterous grasping. However, cube pickup and toast insertion appear very simple to me, due to the object symmetry and abundance of grasp candidates. Same with towel folding. I believe tasks with clearer constraints such as picking up a mug, or pouring a bottle, which are examples mentioned in the method section, would make more convincing experiments. For deformable objects, T-Shirt folding would also introduce more object level constraints, demonstrating working keypoint-transfer.

    I also missed experiments for the other two claims: Articulation and Room Scale Navigation.

Questions for authors

If possible, I would like to see how sim-to-real transfer performs on more objects, such as mug pick and place, or t shirt folding.

Minor Notes

  • I recommend another revision focusing on consistency. For example, Fig. 2 and the Appendix mentions Hunyuan Part Segmentation. However, this is not mentioned in the main text. Critical information such as the used object libraries are only detailed in the appendix.
4 comments

Thank you for the positive assessment and the concrete suggestions.

W1 / Q (more constrained real tasks). We agree that the cube and the towel are forgiving because many grasps work, and added the two cases you suggested (Sec. 5.6, App. H.7; videos in the supplementary):

  • Mug pick-and-place (PiPER-X), policy trained only on simulation data: 13/20 real trials (95% CI [43.3, 81.9]%).
  • T-shirt folding (PiPER-X), simulation data collected with the same setup as towel folding: 4/10 real trials ([16.8, 68.7]%), against 15/20 for the towel. The T-shirt is clearly harder.

Since a mug is constrained by its handle, we also tested the validator on the real robot (our reply to xkZN, Q3): 300 single grasps on two mugs, two bottles, a screwdriver and an apple. On the mug handles, simulation and robot agree in every trial: all 25 handle grasps the validator accepts succeed, and all 25 it rejects fail (App. H.7).

W2 (articulation and room-scale navigation). Real-world articulation results are in the paper, though only in the appendix; Sec. 5.6 now points to them: with pi0.5 fine-tuned only on our simulation data, Open Drawer succeeds in 33/40 trials and Close Lid in 53/60 (Table 42). Room-scale navigation is evaluated in simulation: on 418 audited pairs, 94.2% are ready and 96.5% execute on the full robot in simulation (Table 3; rollouts in Table 38). We did not deploy navigation on hardware, and Sec. 6 now states this.

Hunyuan naming. Fixed: Sec. 3.1 now names "the Hunyuan3D part-segmentation model, which builds on P3-SAM and X-Part", as in Fig. 2 and App. D.

Asset libraries. Sec. 5.5 now lists the nine sources by asset type (rigid: Objaverse, ShapeNet, OmniObject3D; articulated: PartNet-Mobility, PhysX-Mobility; garments: GarmentLab, ClothesNet; rooms: AgentWorld, GRScenes) with a pointer to Table 49 for counts and licenses.

Thank you again. We hope the new real-robot results address your concerns; if so, we would appreciate it if you would consider raising your score.

Dear Reviewer yXoT, thank you again for the concrete suggestions. Following them, the revision adds the two tasks you proposed, mug pick-and-place (13/20) and T-shirt folding (4/10), together with a real-robot test of the grasp validator (Sec. 5.6, App. H.7; videos in the supplementary). As the discussion period ends soon, we would be glad to know whether this addresses your concerns.

I apologize for my delayed response. The newly added real-robot experiments on less forgiving tasks strengthen the contribution and address my main concerns.

I acknowledge the other reviewers’ concerns about the novelty of the individual components. However, I believe that a good pipeline is a valuable contribution in itself. But I think the paper’s presentation undersells this, which may explain why it reads as incremental. I therefore maintain my score of 3 (weak accept) and support acceptance.

Thank you for taking the time to reply, and for your support. We agree with your point about the presentation. In the final version we will put the pipeline as a whole at the center of the introduction and Sec. 3.

Reviewer_EBF4OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Annotate-Anything is an automated framework for annotating 3D assets with linguistic, visual, and executable annotations.

  • It leverages Vision-Language Models (VLMs) to generate manipulation-related information and annotation candidates, which are subsequently validated by a physics annotation pipeline and stored as an executable annotation bank to support parallel simulation.
  • Applied to 17,005 assets, the framework has produced over 100 million validated action annotations spanning nine skill families.
  • It outperforms competing annotation frameworks on a simulation asset suite. Furthermore, policies fine-tuned on demonstrations collected by leveraging its annotations demonstrate robust performance in sim-to-real experiments.
Critical strengths
  1. This work shows a substantial workload.
  • It introduces a retrieval-based annotation bank and provides a unified annotation schema covering four types of assets.
  • The annotation results involve the processing of 17K assets and the generation of 100M annotations.
  1. The framework effectively combines the semantic proposing capabilities of VLMs with physical validation, aligning executable annotations with semantic labels such as compatible skills. This design significantly facilitates the downstream usage of the annotation results.
Critical weaknesses
  1. The work appears derivative and lacks novelty. The key annotation processes do not differ substantially from previous works and fail to address the core challenges in the corresponding fields, leading to limited improvements in method comparisons. a. For articulated objects, the framework does not independently annotate joint parameters and continues to rely on external annotation sources (e.g., use an asset source that already has the articulation annotation). The omission of articulated object-specific attribute annotations limits the contribution of this work. b. Regarding dexterous manipulation, the absence of specialized annotation forms or processing methods makes the framework’s performance in this domain questionable. c. Furthermore, the room/scene annotation process differs significantly from the object annotation process (covering rigid, articulated, and deformable objects), making it appear as a disjointed component of the framework.
  2. The manuscript's writing is of poor quality and is not suitable for publication. a. In the experimental section, many results are reported without adequate analysis, making the source of the observed improvements unclear and hindering a proper assessment of the framework's contributions. b. As a central contribution of the work, the trajectory generation and physical validation pipeline is not described with sufficient detail in either the figures or the main text. c. The paper suffers from poor presentation quality, including issues with overly condensed text and figures, which significantly compromises its readability and comprehensibility.
Questions for authors
  1. Line 171: Could the authors clarify the specific role that geometry plays in the skill inference process?
  2. Section 5.4: Regarding the trajectory counts per hour results, could you provide further clarification or a breakdown of these metrics?
  3. Physical validation pipeline: The current pipeline employs a coarse filtering step with floating grippers to primarily filter out infeasible annotations, followed by full-robot execution for further filtering. Could the authors report the filtering efficiency or rejection rates at each of these two stages?
  4. End-effector details: What specific gripper is used for physical validation? How many types of end-effectors are employed in this phase, and are dexterous hands also included?
  5. Time consumption of physical validation: Could the authors report the time consumption or efficiency of the physical validation process? Given its similar simulation interaction to the subsequent full-robot execution, its computational overhead may be non-trivial.
  6. Baseline comparison in Table 4: Could the processing times of the other baselines in Table 4 be reported? Does the proposed framework still demonstrate superior time efficiency when the annotation time is taken into account?
  7. Section 5.4 baselines: Could a baseline comparison for deformable objects be provided? Additionally, what are the detailed comparison results with the used baselines for each category, such as Grasp, Articulation, and Navigation, respectively?
  8. The authors do not provide analysis for the reported experiment results. Experiment-related questions can only be discussed after enough analysis is provided.

I also identified several writing and presentation issues that may confuse readers or compromise the readability of the paper:

  1. Fig. 6: The figures are excessively shrunk, making the text within them difficult to read.
  2. Line 401: This statement contradicts the reported precision of this work.
  3. Around line 327: This sentence presents neither an experimental result nor an analysis, which is confusing.
  4. Section 5.2: C1 claims the importance of the interaction prior in the introduction. However, the ablation experiments for C1 retain the VLM step in each ablation competitor, resulting in a methodological misalignment.
  5. From line 323: Results are reported without adequate analysis. It remains unclear why Annotate-Anything achieves the reported superior performance over other baselines.
  6. Around line 422: Regarding the “Protocol matched DexGrasp baseline,” results are presented but lack necessary analysis.
  7. Line 172: The sentence starting with “Rooms…” is overly condensed.
6 comments

Thank you for the review. We first clarify the contribution, because it bears on W1 and on the later questions.

W1 (derivative, core challenges). We agree that several of our annotation steps build on existing methods, and the revised paper does not present them as the contribution. Existing methods already produce large numbers of assets and annotations. What limits simulation data collection is that many of these annotations fail when executed, so collection spends its time on failed attempts: with annotations converted from existing systems, collection needs 1.35-2.53 attempts per successful trajectory, against 1.11-1.14 with our validated banks (Table 4), and 4.17 without annotations (Table 3). Testing annotations in simulation is not new by itself (ManiTwin and DexGraspNet 3.0 do it for grasps; Table 1); our main contribution is applying it to every candidate of every skill family before collection: each candidate is validated once, in parallel, under skill-specific tests (a floating end-effector for object-centric skills), and the surviving bank is reused across scenes and robots. Removing the validator, and with it the validated augmentation, lowers full-robot execution from 91.9% to 63.2% (Table 3). On standard part and affordance benchmarks our grounding is close to specialized models rather than above them (Sec. 5.2); its role is to give the generators the right anchors, and its effect shows in execution (Geometry-only vs. ours, Table 3).

W1a (joint parameters). We read joint type, axis and limits from the asset (App. J.1; Sec. 3 now states it), which we think is appropriate for a data-generation pipeline. An articulated simulation asset is defined by its joint model; without one it cannot be simulated as articulated. Articulated assets with joint models are now generated procedurally or from images at scale (Infinite Mobility, Lian et al., 2025; Joshi et al., 2025; PhysX-Anything, Cao et al., 2025; App. I), so the joint parameters come with the asset. What we annotate is the interaction given the joint model (which part to contact, how, and a joint-consistent trajectory), validated in simulation (Exec. 93.2%, Table 3) and on hardware (Open Drawer 33/40, Close Lid 53/60, Table 42).

W1b (dexterous hands). The dexterous skills have their own annotation form and processing, described in App. F; Sec. 3 now mentions them. Each DexGrasp candidate stores the wrist pose, the hand joint configuration and a semantic contact assignment, in pregrasp, grasp and squeeze stages; the generator builds on BODex. BiDexGrasp adds category-conditioned contact patches, a staged SDF objective over labeled hand collision spheres and pairing constraints between the hands. Validation uses a floating Dex3-1, XHand or Sharpa hand through pregrasp, grasp, squeeze, settling, a gravity hold and a lift. DexGrasp reaches Ready 88.7% and BiDexGrasp 82.4%, and on the complete robots Exec. 93.9% and 88.5% (Table 3); on hardware, policies trained on this data succeed in 16/20 dexterous pick-and-place and 11/20 toast insertion trials (Fig. 8).

W1c (rooms). The room pipeline differs in its perception stage because navigation needs different information (free space, obstacles at several heights, approach regions). Its output uses the same schema: scene-level regions are anchors in the same set H(A) as parts, keypoints and affordance regions, and yield banks of approach poses in the same format (Sec. 3.2.1, App. E.1). The collection system composes them with object-level banks into mobile-manipulation tasks (Sec. 4.1). Sec. 3 now states this.

We agree that the results needed more analysis, and added it to Sec. 5.2-5.4. In brief:

Table 2. Direct VLM is close to the full pipeline on semantics (85.1 vs. 93.7) but far lower on 3D grounding (62.5 vs. 90.2) and actionability (65.2 vs. 91.9): a VLM can name the right part but cannot localize it on the asset. Removing the 3D refinement lowers part IoU from 91.5 to 82.6.

Table 3. The ablations separate three failure sources. Without VLM priors (Geometry-only), many pairs get no valid candidate (Ready 58.5%) and the rest are rated far lower (61.7 vs. 85.5). Without optimization and validation (VL-only), candidates sit on the right part but often fail to execute (Exec. 67.5%). Without validation and augmentation, Exec. drops from 91.9% to 63.2%. Exec. is lowest where tolerances are tight or contact is deformable (Deformable 84.1%, BiDexGrasp 88.5%, Hanging 88.7%, Insertion 89.3%).

Table 4. Most of the gap already arises in Ready, i.e., whether a pair gets any validated candidate at all: 45.2-58.0% for RoboGen-, MolmoBot- and GenManip-adapted vs. 90.0-93.4% for ours. These systems output task-level targets, programs or interaction proposals that are not checked against the contact, collision and stability behavior of the specific asset under our end-effectors. ManiTwin's grasps are already simulation-verified, so its gap is smaller (82.6% vs. 96.4%).

W2b (validation under-described). Sec. 3 now states the grasp tests: a parallel-jaw grasp must hold the object under gravity and through a lift with an external force applied in turn along five axis directions; a dexterous grasp goes through a gravity hold and a lift. Validation settings and criteria per family are in Tables 10-11 (App. E) and App. F, which we corrected (see our reply to xkZN, Q5).

W2c / Writing. We rewrote the sentences you list (L171-172, L323-327, L398-401, L421-424; below), added analysis, and give Fig. 6 at readable size as Fig. 18 in the appendix (a pointer is in the Fig. 6 caption). The additional page is used for these additions.

L401 vs. the reported precision. The two statements answer different questions. The intervals quantify sampling variability over asset-skill pairs under our protocol. "Estimates" referred to a systematic uncertainty: each baseline's output is converted into our interface, and conversion may lose information the original system would use. The revised Sec. 5.4 states this second point explicitly, and the Table 4 caption no longer uses the word.

L327. Now an analysis: of 3,240 VLM queries, 94 (2.9%) fail structured parsing or segment linking and are rejected before action generation; the remaining errors are semantic (App. H.3).

C1 ablations keep the VLM. The ablation without the VLM is Geometry-only in Table 3, and Sec. 5.2 now points to it. It selects anchors from surface normals, curvature, free space, collision margins, reachability and traversability (App. H.8), and reaches Ready 58.5% vs. 87.4%, Exec. 58.4% vs. 91.9%, an action human score of 61.7 vs. 85.5, and collection success 48.2% vs. 86.0%. Table 2 asks how best to use the VLM, which is why all its rows keep it.

L323. These results are not superior, and the text now explains why. On PartNeXt our 44.1 mIoU is above P3-SAM (41.01), the segmenter we build on, and below SAMPart3D (46.70): object-macro mIoU scores every annotated part, whereas we target task-relevant functional parts. On LASO our 41.0/54.3 is above Direct VLM (35.8/48.9) and below the released LASO model (43.6/57.2), which is trained on LASO; no component of ours is.

L422 (BODex). Both use the same hand, pairs, validator and G1 execution, so the difference comes from the candidates alone. End to end, 83.3% vs. 76.5% of all 520 pairs execute (+6.7 pp, 95% CI [1.9, 11.6]); the separate gains in Ready (88.7% vs. 84.8%) and Exec. (93.9% vs. 90.2%) are smaller (Table 48). Since our generator builds on BODex and the baseline is the released BODex with default parameters, the gain measures our changes on top of BODex; Sec. 5.4 now says so.

L172 (rooms). Expanded: occupancy maps at several heights and a VLM-cleaned floor plan define the scene-level anchors (free space and approach regions) from which navigation candidates are generated in the same bank schema (App. D).

Q1 (geometry in skill inference, L171). The VLM proposes the skills an anchor supports from its affordance and language priors; geometry and task constraints then decide whether each can be instantiated on the asset. For example, a parallel-jaw grasp needs antipodal contacts on the anchor within the gripper aperture, insertion and hanging need an opening, hook or support edge of compatible cross-section and axis, articulation needs a movable part with a joint, and navigation needs reachable free space. A skill with no feasible candidate leaves the pair unready (counted in Table 3, never used for collection). Skill sets are exactly correct for 300/360 anchors (P/R/F1 88.4/84.6/86.5, App. H.3). Sec. 3.1 now says this.

Q2 (Traj./h). Successful trajectories per wall-clock hour of collection, from candidate retrieval to trajectory validation (goal-set IK, collision-aware planning, full-robot execution), on one RTX 4090 (Table 25, App. J.2). It excludes bank construction, which Table 40 adds for the real tasks. Per-family values are in Table 38 (e.g., Grasp: 300 with our bank, 150 with VL-only annotations, 70 without). Values are averages over assets, because time per trajectory depends mainly on collision complexity (an eight-drawer cabinet takes far longer than one drawer).

Q3 (rejection at each stage). Floating stage: Pass and Ready in Table 3 (e.g., Grasp 60.3% / 96.4%), split by stage in Table 28: of 2,315 raw candidates per pair, 1,383 pass geometry checks, 842 IK/trajectory checks, 615 disturbance tests, and 538 enter the bank. Full-robot stage: Exec. in Table 3 (e.g., Grasp 98.1%, Articulation 93.2%); for DexGrasp, App. H.4 gives 2,278/5,200 candidates passing floating validation, and on G1-Dex3-1 the top-ranked candidate succeeds for 402/461 ready pairs (87.2%), 433/461 (93.9%) with one fallback.

Q4 (end-effectors). In the evaluation, Grasp, Articulation, Insertion and Hanging are validated with a floating FrankaUMI gripper and executed on the full Franka-FrankaUMI robot; DexGrasp with a floating Dex3-1 hand and executed on G1-Dex3-1; BiGrasp runs on dual Franka arms and BiDexGrasp on the G1 with two Dex3-1 hands; garment skills are validated on the full manipulator and navigation on a base-aware G1 (App. H.4). The banks cover four parallel-jaw grippers and three dexterous hands (Table 26, Table 27), so dexterous hands are included. A new gripper needs its USD and about six values; a new hand its USD and a configuration file.

Q5 (validation time). App. J.2 reports the time from candidate generation to serialization, which includes validation and therefore bounds it. Per 10^4 validated annotations on one RTX 4090: Grasp 13 min, DexGrasp 50, BiGrasp 95, BiDexGrasp 105, Articulation 91, Insertion 143, Hanging 167, Deformable 222, Navigation 33, including generation, optimization and serialization. Floating validation is cheaper than full-robot execution because it simulates only the end-effector and object, batches many candidates (400 cloned environments per batch for Grasp), and runs once per asset.

Q6 (baseline processing time). Both are in Table 4: Ann./min (annotation throughput, generation to serialization) and Traj./h. For the three baselines run from official code (RoboGen, MolmoBot, GenManip) ours is faster on both, e.g., 455 vs. 18 Ann./min and 225 vs. 42 Traj./h against RoboGen-adapted, so including annotation time does not change the ordering. ManiTwin-adapted uses released annotations; its Ann./min is the rate at which those pass our validator, not ManiTwin's own generation time (App. H.8 now states this).

Q7 (deformable and per-family baselines). None of the compared systems produces garment or cloth annotations that can be converted to our interface; for deformables we compare against ablations under the same protocol (Table 38: data-collection success 15.0% without annotations, 52.6% VL-only, 80.3% ours). Each Table 4 row is matched by skill family: ManiTwin is a Grasp comparison (Table 4) and BODex a DexGrasp one (Sec. 5.4), and for multi-family baselines "Ours (matched)" aggregates exactly the same families. Our per-family results are in Tables 36 and 38.

We hope the analysis added in the revision allows the experimental questions to be assessed, and we are glad to clarify further.

Reviewer_xkZNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AnnotateAnything, which automatically converts 3D assets without interaction annotations into action annotations for robot data collection. Qwen3.5-27B labels functional parts, keypoints, affordance regions and applicable skills on multi-view renderings, and these labels are grounded on 3D parts from P3-SAM/X-Part and on Skeleton Merger keypoint candidates. For each skill family, hand-written templates generate candidate actions. The candidates are refined by trajectory optimization, validated in parallel in Isaac Sim with a floating gripper or hand, and expanded by symmetry and perturbation augmentation. The method covers nine skill families and four asset types, processes 17,005 assets, and yields about 100M validated annotations.

Critical strengths

On three real robots, π0.5 fine-tuned solely on simulation data succeeds on six tasks. On the three rigid insertion tasks, policies trained on the paper's data clearly outperform those trained on MoMaGen data, and the paper's data require no human demonstrations.

Critical weaknesses
  1. Limited novelty and inaccurate positioning against prior work. The pipeline integrates existing components: VLM point selection, off-the-shelf 3D part segmentation, hand-written skill templates, trajectory optimization and simulation-based validation. The appendix acknowledges that each new skill family requires a hand-written generator and validator, and that joint information for articulated objects is read from the asset's USD joint model rather than inferred. Contribution 1 contrasts the candidate bank with "a single verified grasp set per object", but prior work is already function-conditioned and one-to-many. ManiTwin, described as the closest work, has a VLM select functional and grasp points from numbered FPS candidates on multi-view renderings. It keeps about 62 simulation-verified grasps per object on average, and each grasp inherits the semantic label of its nearest functional point. DexGraspNet 3.0 (DexVLG, ICCV 2025) provides 170M dexterous grasps over 174K objects, aligned with semantic parts and paired with part descriptions. The relation to BODex is not stated: the dexterous-grasp generator builds on BODex (Appendix F), yet the main text presents BODex only as a baseline. That baseline uses BODex's default parameters, while the paper's generator is tuned for the Dex3-1 hand. InternData-A1 validates its trajectories in physics simulation and generates them in parallel on 8 RTX 4090 GPUs, yet Table 1 marks both Physics Validation and Parallel Generation as ✗ for it. RoboTwin 2.0 has "strong domain randomization" in its title, yet Table 1 gives it only a partial mark for Data Augmentation. The introduction cites these two works as examples of manual annotation with one label per object, whereas InternData-A1 generates many grasp candidates automatically with AnyGrasp. ManiTwin, the closest work, is missing from Table 1.

  2. The experiments do not demonstrate the value of the candidate bank itself, which is the paper's central claim. The main problem is that Pass/Ready/Exec are measured with the same validator and simulator that filter candidates during generation. External baselines are converted to the paper's interface and scored with the paper's validator, and the authors themselves call these numbers "estimates". There is also no downstream evidence for diversity or scale. The real-robot tasks use objects hand-modeled for each task rather than the 17,005 assets, a separate policy is trained per task, and there is no cross-object generalization experiment. On the real robots, policies trained on candidates that use only VLM anchors and pass the same simulation validation reach essentially the same success rate as the full method (81/120 vs. 85/120).

Questions for authors
  1. ManiTwin's released annotations are defined on its own image-generated assets, which are not among the nine asset sources in Table 47. On which assets was ManiTwin-adapted evaluated? The "Ours (matched)" row it is compared with in Table 4 is identical to the Grasp row of Table 3 (760 pairs). In what sense is this comparison paired?
  2. With the same number of demonstrations, how do policies trained on data from the multi-candidate bank compare on unseen objects with policies trained on data from a single candidate per anchor?
  3. Can you provide executability evidence that does not depend on your validator? For example, re-evaluate Exec with realistic friction and a modeled support surface, use a different simulator, or execute bank grasps open-loop on a real robot. Can the floating-to-full-robot audit be extended to the other skill families?
  4. Please break down the ~100M annotations by skill family, by end-effector, and into original candidates and augmented variants, and state which level of validation the augmented variants pass.
  5. The grasp validator in the anonymous code (annotation-generation/Grasp/Object_grasp_pose_pipeline.py) uses standard gravity, a 0.30 m lift that passes if the object drops by at most 0.05 m, and external forces of 2 N in five directions. Appendices E.5 and F instead describe gravity perturbations of 1.5g in eight random directions and a 1.0 m lift in which the object must rise at least 0.95 m. These settings differ completely; please explain why.
7 comments

Thank you for the detailed review and for reading the released code.

Built from existing components. We agree that VLM point selection, 3D part segmentation, skill templates, trajectory optimization and simulation are existing tools, and we do not claim them. As stated in the general response, testing annotations in simulation is not new by itself (ManiTwin and DexGraspNet 3.0 do it for grasps); the contribution is a pipeline that validates every candidate of every skill family before use and keeps the survivors as reusable banks. The validator's effect is measured in Table 3 (Exec. 91.9% -> 63.2% without validation and the validated augmentation) and on the real robot (Table 41), and it is affordable on every candidate (App. J.2: 13-222 min per 10^4 annotations from candidate generation to serialization). Contribution 2 in Sec. 1 now says this.

A generator and validator per family. Each skill family needs one generator and one validator, as our Limitations state. There are few families, and each is written once for a class of embodiments, not per robot or asset. The parallel-jaw generator and validator serve four grippers (Franka, FrankaUMI, X-Trainer, PiPER-X), and a new gripper needs its USD and about six values (finger joints, open and close targets, aperture, fingertip standoff). The dexterous generator serves three hands (Dex3-1, XHand, Sharpa) through a per-hand configuration file. All 17,005 assets were processed with these nine families.

Joints read from the asset. This is by design (App. J.1; Sec. 3 now states it). In simulation the joint model is part of an articulated asset, like its mass and collision geometry; without one the asset cannot be simulated as articulated. Such assets are no longer scarce: articulated assets with joint models are now generated procedurally or from images at scale (Infinite Mobility, Lian et al., 2025; Joshi et al., 2025; PhysX-Anything, Cao et al., 2025; cited in App. I). Methods that infer kinematics from meshes are complementary, since our generators consume their output unchanged. What we annotate is the interaction given the joint model (which part, handle grasp / edge pull / panel push, a joint-consistent trajectory), validated in simulation (Articulation Exec. 93.2%, Table 3) and on hardware (Open Drawer 33/40, Close Lid 53/60, Table 42).

ManiTwin and DexGraspNet 3.0 (Contribution 1). We re-read both and agree. By our own definition (App. E.1: per anchor-skill pair, a bank of validated candidates), ManiTwin is function-labeled and one-to-many for grasping (about 62 verified grasps per object, each labeled by its nearest functional or grasp point), and DexGraspNet 3.0 (ICCV 2025) provides many part-aligned grasps per object. We removed "rather than a single verified grasp set per object" from Contribution 1 and describe both accurately in Sec. 2. What we claim instead is the scope of the bank: one schema across nine skill families (not grasping alone), four asset types and several grippers and hands.

BODex. You are right that the main text did not state this. Sec. 2 and 3 now say that our DexGrasp generator builds on BODex's optimization. We changed it where it did not fit our use: contact targets are restricted to the VLM-grounded functional region, initial hand-object penetrations are resolved, pinch grasps are added for tabletop objects, and the optimization parameters are set per hand (App. F now lists these). Every candidate is then checked by our own floating-hand validator. The Sec. 5.4 baseline is the released BODex with default parameters (Table 45), so the +6.7 pp gain measures our changes on top of BODex as a whole; Sec. 5.4 now says so.

Table 1. We re-checked the papers and corrected the table. The caption now defines Physics Validation: full marks when candidates are checked in simulation before use, partial when only executed trajectories are kept or rejected during collection. InternData-A1 executes planned trajectories in simulation and renders only the successful ones, while its grasp candidates are ranked by confidence and not simulated, so its mark changes from none to partial. ManiTwin and DexGraspNet 3.0, which test each grasp in simulation, have full marks. InternData-A1 now also has full marks for Parallel Generation, Articulation, Deformable and Data Augmentation, and RoboTwin 2.0 for Data Augmentation and Articulation. The caption also defines the other two pipeline columns, and ManiTwin and DexGraspNet 3.0 are new rows.

Intro (L067-069). Agreed; the sentence misdescribed both works. It now reads: "Existing data pipelines label a fixed object library with keypoint-axis annotations that are expanded into grasps (RoboTwin 2.0) or generate generic grasp candidates automatically (InternData-A1), but do not provide reusable, physics-validated per-asset banks across skill families."

Pass/Ready/Exec. and the validator. We agree that Pass and Ready are statistics of our validator: they measure how many candidates survive it, not whether the survivors are good actions. Sec. 5.1 now says so. The quality claims rest on measurements the validator does not make: Exec. runs the complete robot with goal-set IK and collision-aware planning, which the floating validator does not check (it uses the same simulator, so it is a stricter test rather than an independent one); human scores come from five blinded non-author raters (App. H.9); and real robots (Sec. 5.6, Table 42, and the new grasp test in part 3).

"Estimates." The word was meant narrowly. Each system's native output is converted into our candidate interface and evaluated with our validator and rollout protocol (Table 44), so the numbers measure that system's converted outputs under our protocol, not an end-to-end reproduction of the system. The confidence intervals cover sampling variability only, not conversion error. The revised Sec. 5.4 states that the numbers measure converted outputs and are not reproductions.

Diversity and scale. The bank's purpose is to keep a feasible action available whatever state the object is in. In tabletop collection, initial object poses are randomized (Sec. 4.1), and objects tip over or move during failed attempts. A single canonical candidate per anchor (e.g., a top-down grasp of an upright bottle) then often has no feasible solution, whereas the bank, which is not filtered by the object's resting orientation (App. F), still holds grasps that work in the new pose. The same property enables failure recovery during collection: the planner retries with another validated candidate. The paper measures this directly: using only the top-ranked candidate, i.e., one candidate per anchor, lowers DexGrasp execution from 93.9% to 87.2% even from nominal poses (App. H.4), and the median asset keeps 176 (Grasp) and 153 (DexGrasp) distinct modes (Sec. 5.5). The paper does not claim a policy-level effect beyond this; Sec. 6 lists it as future work.

Real-robot objects. The real-world experiments test one question: whether policies trained only on bank-driven simulation data transfer zero-shot. Each real object therefore needs a simulation counterpart, hence digital twins (App. H.7). Generality across assets is evaluated in simulation on the audited suite drawn from the 17,005 assets (Sec. 5.3-5.5). Cross-object policy generalization is a different question that the paper does not claim (Sec. 6, App. H.7).

VL-only + validation (81 vs. 85/120). We agree, and Sec. 5.6 reports the same reading: when VL-only candidates must pass our validator during collection, success is similar but collection takes 2.7-5.5x longer. We see this as support for the main claim: validation is what makes the data executable, and optimized, pre-validated candidates are what make collection efficient. Sec. 5.6 now says so.

Q1 (ManiTwin-adapted). It was evaluated on ManiTwin's own released objects, because its annotations are defined on them. Those meshes are not in our asset pool or release, which is why the source table (Table 49 in the revision, Table 47 in the submission) does not list them. You are right that the comparison is therefore not paired: "matched" meant matched by skill family. We removed "paired" from Sec. 5.4, state there that each baseline is evaluated on the assets its outputs are defined on (ManiTwin on its own released objects), and removed the ManiTwin paired intervals from Sec. 5.4 and Table 48.

Q2 (bank vs. one candidate per anchor at equal demos). We did not train this comparison. With one candidate per anchor, many randomized or perturbed initial states admit no feasible demonstration at all (e.g., a tipped-over bottle whose only stored grasp approaches from the top), so the two datasets would differ in which states they can cover, and the single-candidate dataset could not include recovery with a different grasp. The top-1 vs. one-fallback comparison above measures this at the execution level; a policy-level study on unseen objects is listed as future work.

Q3 (evidence independent of the validator). Of the options you list, we chose execution on a real robot. We executed 300 grasps with a PiPER-X arm on six real objects (two mugs, two bottles, a screwdriver, an apple), 50 different grasp poses per object, including poses the validator accepts (156) and rejects (144). Each object is scanned into simulation; on the robot, its pose is estimated from the point cloud and the arm moves to the grasp. A grasp succeeds if the object is lifted stably without being dropped. No trial was rerun or left out.

ObjectSim ok, real okSim ok, real failSim fail, real okSim fail, real fail
Mug 11821020
Mug 2213521
Bottle 1340214
Bottle 2290120
Screwdriver221423
Apple206816
Total1441230114
  • Agreement: 258/300 (86.0%, 95% CI [81.6, 89.5]).
  • Accepted grasps that succeed on the robot: 144/156 (92.3%, [87.0, 95.5]).
  • Rejected grasps that fail on the robot: 114/144 (79.2%, [71.8, 85.0]).
  • The disagreements are mostly conservative: 30 rejected grasps succeed, against 12 accepted grasps that fail.
  • On the mug handles, simulation and robot agree in every trial: all 25 accepted handle grasps succeed and all 25 rejected ones fail.

The full table is Table 41 (App. H.7); example videos are in the supplementary. On extending the floating-to-full-robot audit: Exec. in Table 3 is already the full-robot execution of validated candidates for every family (floating-validated for the object-centric families).

Q4 (breakdown of the ~100M). The breakdown is now Table 27. The total is 97.3M validated annotations: 58.4M (60%) directly generated and 38.9M (40%) augmented variants (origin counts derived from per-family shares, as the Table 27 caption states).

FamilyValidatedOriginalAugmented
Grasp36.8M28.2M8.6M
Articulation: open by handle8.7M5.3M3.4M
Articulation: open on edge21.4M15.2M6.2M
Articulation: close by push26.1M8.8M17.3M
Articulation: rotate3.1M0.4M2.7M
BiDexGrasp785K188K597K
DexGrasp262K211K51K
Deformable106K56K50K
BiGrasp31.3K9.3K22.0K
Insertion25.7K20.4K5.3K
Hanging21.8K20.5K1.3K
Navigation / Approach11.9K4.4K7.5K
Total97.3M58.4M38.9M
  • By family. Grasp and Articulation account for 98.7%; Sec. 5.5 now says so.
  • By end-effector. FrankaUMI 41.1M, X-Trainer 18.0M, Franka 17.2M, PiPER-X 17.1M, Dex3-1 3.8M, Sharpa 0.17M, XHand 0.07M (per-family shares in Table 27).
  • Validation of augmented variants. Every augmented variant passes the same physics validation as an original candidate of its family and is counted only if it passes. App. E.6 and H.2 described a lighter recheck for pose-level variants; that was wrong and is corrected.
  • Why Articulation is largest. Its annotations are attached to joints, not assets: cabinets have several drawers and doors, appliances several knobs, and each joint is annotated for every sub-skill and end-effector that fits it. The density in Table 26 is per joint and before augmentation; the caption now says so and lists all five end-effectors used for articulation.
  • As Sec. 5.5 states, the total counts validated instances, not distinct strategies. After merging within 2 cm / 10 deg, the median asset keeps 176 (Grasp) and 153 (DexGrasp) distinct modes, and the median joint 21 (Articulation); the submission gave the last figure as per asset, and Sec. 5.5 and Table 30 now say per joint.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response and summary of revisions

We thank the reviewers for their careful reading. We first restate the contribution, because several comments turn on it, and then list the revision.

What the paper contributes. AnnotateAnything is a data-generation pipeline submitted to the Datasets and Benchmarks area. Its claim is not a new segmenter, VLM or grasp optimizer; we use existing ones and say so. Testing annotations in simulation is not new by itself: ManiTwin and DexGraspNet 3.0 do it for grasps, and generators such as InternData-A1 keep only trajectories that succeed during collection (Table 1). What is new is where we apply it: passive 3D assets are turned, automatically and at scale, into banks of action annotations in which every candidate of every skill family is validated for execution before it is used, under one schema across nine skill families, four asset types and several embodiments. GPU-parallel validation (a floating end-effector for object-centric skills, the full manipulator for garments, a base-aware robot for navigation) is what makes this affordable. Its effect is measured directly in the paper:

  • removing the validator, and with it the validated augmentation, lowers full-robot execution from 91.9% to 63.2% and rollout success from 86.0% to 57.6% (Table 3; components in Table 43);
  • collecting from our optimized, pre-validated bank is 2.7-5.5x faster than collecting with unoptimized VL-only candidates validated during collection, at similar real-robot success (85/120 vs. 81/120; Sec. 5.6, Table 40);
  • validation is affordable on every candidate: action annotation from candidate generation to serialization, validation included, takes 13-222 min per 10^4 validated annotations on one RTX 4090 (App. J.2).

New evidence in the revision.

  • A real-robot test of the validator (xkZN Q3): 300 grasps on six real objects, both accepted and rejected by the validator. Simulation and robot agree on 258/300; 144 of 156 accepted grasps succeed on the robot, and 114 of 144 rejected ones fail (Sec. 5.6, Table 41).
  • Two more constrained real tasks (yXoT): mug pick-and-place 13/20 and T-shirt folding 4/10 (Sec. 5.6, App. H.7).
  • A breakdown of the ~100M annotations by family, end-effector and origin (xkZN Q4; Table 27).
  • Analysis paragraphs for the experimental results (EBF4; Sec. 5.2-5.4).

Corrections. We corrected the description of InternData-A1, RoboTwin 2.0, ManiTwin and DexGraspNet 3.0 (Sec. 1, Sec. 2, Table 1), stated our DexGrasp generator's dependence on BODex (Sec. 2, 3, App. F), and corrected the description of the grasp-validation settings (App. E.5, F; xkZN Q5) and of how augmented variants are validated (App. E.6, H.2; xkZN Q4) to what was used for the released banks. No reported number changes. Update: Reviewer xkZN pointed out that our w/o-validation ablation also removes augmentation (Table 43). Sec. 5.3 and App. H.8 now state that none of the ablations uses augmentation; no numbers change.

All changes are marked in blue in the revised PDF. The main text now uses the additional page allowed for the discussion phase. New videos are in the supplementary (video/rebuttal_videos/).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.