AnnotateAnything: Automatic Annotation of 3D Assets for Robot Manipulation

Simulation enables scalable robot data collection, but 3D assets often lack the interaction annotations needed to specify where and how robots should act. We present AnnotateAnything, an automatic framework that converts passive 3D assets into manipulation-ready assets with language, 3D visual, and executable action annotations. A vision-language model supplies interaction priors—functional parts, keypoints, affordance regions, and compatible skills—that are grounded in each asset's 3D geometry. A physics annotation pipeline converts these priors into action candidates through skill-specific generation, geometric optimization, and GPU-parallel validation, retaining multiple solutions in reusable candidate banks. These banks drive an asynchronous parallel simulation data-collection system. Applied to $17,005$ rigid, articulated, deformable, and room-scale assets, the framework produces about $100$M validated action annotations across nine skill families. On an audited simulation suite, it achieves higher readiness and full-robot execution success than ablations and framework-adapted baselines, as well as higher annotation and collection throughput than the latter. Pretrained policies fine-tuned solely on the generated simulation demonstrations achieve $70.8\%$ mean success across six real-world tasks on three robot platforms, with $20$ trials per task. Project page: https://gilded-cupcake-c582c1.netlify.app/.