- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
AffordCraft constructs simulation assets from a single RGB image and a task instruction by retrieving and adapting existing assets. It selects candidates with compatible parts and joints, preserves their articulation during adaptation, and checks their physical validity in simulation; if construction fails, it tries another candidate. The paper addresses an important bottleneck in embodied intelligence: obtaining diverse, functional assets for robot learning. It reports an 85.15% physical acceptance rate on 2,000 photographs, substantially outperforming the evaluated generative methods. Further experiments examine cluttered images and simulated manipulation, although transfer to real robots is not evaluated.
Critical strengths
-
The approach directly addresses a practical limitation of robot simulation. Reusing established articulation structures reduces the uncertainty of inferring mechanisms from a single image. Combined with checks for collision geometry, inertia, and support, this produces a substantial improvement in simulator acceptance. These results support the method’s value as a tool for building usable simulation assets.
-
The construction results are supported by a broad evaluation. The paper evaluates 2,000 photographs, several external baselines, cluttered inputs, and individual pipeline components. In particular, the ablations demonstrate the importance of physical-feedback reselection and collision processing. The detailed reporting of failures and protocol differences also helps establish the scope of the findings.
Critical weaknesses
-
The evaluation does not yet connect physical acceptance to useful embodied learning. An asset can satisfy the simulator’s physical checks while still having the wrong actionable part, joint configuration, or motion range for the intended task. These properties are not independently validated across the benchmark. The downstream experiments provide only limited evidence: policy success is 57% on single-asset tasks and 11% on composed scenes. Moreover, all evaluated assets were seen during development, and there are no real-robot experiments or controlled comparisons showing that the constructed assets improve policy learning. The evidence therefore supports reliable asset construction more strongly than the broader claim of task readiness.
-
The contribution of task-aware selection is not clearly established. Removing multimodal selection slightly increases category-correct physical passes from 169/200 to 173/200, although this difference is not statistically significant. Meanwhile, the task-conditioning ablation removes both the instruction and category-restricted retrieval, making it difficult to distinguish affordance reasoning from the benefit of knowing the object category. A comparison with simple category-conditioned retrieval, using the same adaptation and physical reselection, is needed to establish what the proposed selection mechanism adds.
Questions for authors
-
What fraction of accepted assets independently matches the intended operated part, joint configuration, and usable motion range? How does the physical gate handle unspecified parts or unknown motion families?
-
With adaptation and physical checks held fixed, does multimodal selection improve mechanism compatibility or manipulation performance over category-conditioned retrieval?
-
Is there additional evidence of real-robot transfer or generalization to unseen assets? Do AffordCraft assets improve policy performance relative to ordinary category-matched library assets?
6 comments
Response to Reviewer 6r8p (W1 continued, W2)
W1 (continued)
Policy success of 57% and 11%. These rates come from a lightweight recurrent head on frozen OpenVLA-OFT features, so we trained stronger learners on the same assets (Appendix A.16). On the five door- and drawer-closing tasks, a PPO policy that reads the simulator state succeeds in 279 of 300 held-out episodes, against 57 of 100 for our head's recorded episodes replayed in the same simulator port (Table 26). On the five pushing tasks, it succeeds in 155 of 300, against 46 of 100 for our head in the port. This shows that the door, drawer, camera, clock and display tasks can be learned on these assets.
Unseen assets. On library assets never used in development (other entries AffordCraft delivered on the benchmark, split from the training pool by visual-similarity clusters; closing a door, closing a drawer, pushing), the head succeeds in 46.5% when trained on up to 40 assets per task type, against 28.7% with one (p < 0.001; Tables 22 and 23). The comparison with ordinary library entries is under Q3.
Real-robot experiments (Appendix A.17). On four real objects (a two-drawer unit, a bowl, a plate, and a basket), with our assets as built, the bowl and the plate were picked up in all 6 runs; with the assets scaled to the measured size, the drawer unit was opened and closed again in all 3 runs, the plate was picked up in all 3 runs, and all 3 runs with the basket succeeded. In all, 7 of 12 runs succeeded as built and 9 of 12 scaled (Appendix A.17). A Franka Research 3 arm with a two-finger gripper ran the motions taken from the assets. These motions succeeded on real objects that differ from the assets in appearance and size. We have uploaded videos of these experiments to the supplementary material. More real-robot experiments are under way; we will add their results to this thread shortly.
Controlled comparison. We held out 65 PartNet-Mobility objects with a door or drawer, rendered a photograph of each from its mesh, and removed each object and its near-duplicates from the library. Policies trained on our assets succeed in 34.9% of the episodes on these objects, 27.7 points more often than with a random entry of the same category regardless of mechanism (p < 0.001; Appendix A.16, Table 24), which usually lacks the needed part (it has one for 13 of the 65 objects, against 54 for ours).
Title. We keep "Task-Ready" in the title: the revised paper defines task readiness and measures it for every method, and AffordCraft yields a task-ready asset for 53.5% of the labeled inputs, against 27.5% for the next best method (Table 1, Appendix A.12, Table 18).
W2. Contribution of task-aware selection
A factorial study separates the two parts of the task condition (Section 4.4). When a part is named (without task verification), the selector's first choice more often has that part and more of its deliveries have a named part that moves, and where the selector and Table 2's category-conditioned retrieval deliver different entries, annotators preferred the selector's asset (22 against 7, p = 0.008).
The baseline. Table 2's "Category-conditioned retrieval (no selector)" row uses the same candidate pool, adaptation, repair, reselection, decomposition cascade and gate, building candidates in retrieval order (Section 4.4).
New settings (Section 4.4, Appendix A.14, Table 20). They add task verification, a catalog-metadata rule applied before selection (when the instruction names a part, only candidates whose metadata carry it with the implied joint type and a usable range are kept, and the system abstains if none is), a per-input time budget of 1,800 s, and movable-part labels for the selector. The primary metric is the human label, with exact two-sided McNemar tests paired by input.
Category knowledge versus the selector. Crossing retrieval (category-restricted or category-blind) with selection (instructed selector, selector without the instruction, retrieval order) on the 200 inputs shows where the effect of the task condition comes from. After category-blind retrieval, the instructed selector delivers 117 category-correct assets against 99 for retrieval order (p = 0.003), and 118 without the instruction: the selector recovers the category by reading catalog categories, which shows that Table 2's task-condition effect (125 against 169) reflects the category restriction.
Response to Reviewer 6r8p (W2 continued, Q1-Q3)
W2 (continued)
Part-specified instructions (118 inputs). Without task verification, more of the selector's deliveries have a named part that moves when driven (47 against 37, p = 0.006), and on 565 held-out objects its first choice more often has the named part (85.1% against 79.8%, p < 0.001; reworded beyond the verification word lists, 83.5% against 79.3%, p = 0.0012). With verification in both arms, which isolates the selector, the results suggest a further small gain (53 against 49 inputs with a task-correct asset and a sufficient measured range, p = 0.22).
Missing mechanism. With every movable entry of the requested category removed from the library, task verification abstains on 57 of the 118 part-specified inputs, whereas retrieval order without it delivers 101 assets; by human labels, 108 against 17 inputs end without a substitute (p < 0.001). The abstention comes from verification, which retrieval order with the same rule shares.
Q1. Independent fractions; unspecified parts and motions
Independent labels give the fraction of accepted assets with the intended part, joint type and range for every method, and we clarify that the physical gate does not use the grounded part or motion. Of AffordCraft's 83 accepted assets for the 101 inputs that need a movable part, 68.7% have the labeled part, 62.7% also its correct joint type, and 60.2% also the requested category and a sufficient measured range; per accepted asset, the GPT-6 Astra agent is higher on all three and Articulate-Anything on the first two, each on 38 accepted assets against our 83 (W1; Table 18).
Eq. (5) and Section 3.3 now describe the gate's structural joint term (every revolute or prismatic joint keeps valid limits and consistent frames and axes after adaptation, and an entry with a movable joint keeps at least one enabled). The grounded part and motion go, with the instruction, to the selector, which judges mechanism compatibility from the photograph, the candidate renders and each candidate's catalog category, part labels and joint types; when the part is unspecified or the motion unknown, it compares the candidates with the requested category and the photographed object. That is the common case on the benchmark (part unspecified for 75.7% of the localized inputs, motion unknown for 61.0%), because the grounding prompt does not invent a manipulation task when the instruction asks only for an asset (Appendix A.2). In the configuration of Section 4.4, task verification turns a part named in the instruction into a requirement; category-only instructions set none.
Q2. Selection versus category-conditioned retrieval
With adaptation and checks fixed, selection improves mechanism compatibility when the instruction names a part, and in Table 2 it is on par on category-only instructions. Without task verification, more of the selector's deliveries have a named part that moves (47 against 37, p = 0.006), and its first choice on 565 held-out objects more often has the named part (85.1% against 79.8%, p < 0.001); with verification in both arms, the results suggest a further small gain (53 against 49, p = 0.22; W2, Appendix A.14, Table 20). As for manipulation, annotators compared the two assets with the source hidden wherever both conditions pass but deliver different entries, and preferred the selector's asset for 22 of 57 inputs and the baseline's for 7 (exact sign test, p = 0.008); the actuation test of W1 measures each asset on its own.
Q3. Real-robot experiments, unseen assets, category-matched library assets
Appendix A.17 adds real-robot experiments with our assets (videos in the supplementary material), and the policy studies show that success on unseen assets rises with the number of our training assets and that policies trained on our assets succeed far more often than with ordinary category-matched library entries, and on par with entries matched in mechanism as well.
- Real-robot experiments: On four real objects, a bowl and a plate were picked up in all 6 runs with the assets as built; with the assets scaled to the measured size, a two-drawer unit was opened and closed again in all 3 runs, and all 3 runs with a basket succeeded; in all, 7 of 12 runs succeeded as built and 9 of 12 scaled (W1, Appendix A.17).
- Unseen assets: on library assets never used in development, success rises from 28.7% with one training asset per task type to 46.5% with up to 40 (p < 0.001; W1; Appendix A.16, Tables 22 and 23).
- Category-matched library assets: policies trained on our assets succeed 27.7 points more often than with a random entry of the same category regardless of mechanism, which usually lacks the needed part (p < 0.001; Appendix A.16, Table 24).
We sincerely thank the reviewer for the constructive feedback and hope that the new experiments and clarifications have resolved these concerns. We are happy to discuss any further questions.
Response to Reviewer 6r8p (Summary, Update to Table 1, W1)
We thank the reviewer for recognizing that reusing established articulation structures addresses a practical limitation of robot simulation and that the construction results rest on a broad evaluation, and for pressing on the two points that matter most: whether accepted assets are ready for the intended task, and what task-aware selection contributes. We have uploaded the revised paper, with every change marked in blue, and updated the code at the anonymous link accordingly. The revised paper now:
Update to Table 1
Table 1 now evaluates the external methods with their native up axis and retries builds that the infrastructure had interrupted; the best other physical pass rate changes from 44.55% to 46.55%. Our row reports one complete run on all 2,000 inputs with timed-out decompositions cached, the setting of its timing columns (1,713 passes; 1,703 as submitted).
W1. Physical acceptance versus task readiness
The revised paper defines task readiness, validates part, joint type and range independently, and shows that 53.5% of the labeled inputs yield a task-ready asset of AffordCraft, against 27.5% for the next best method (Section 4.1, Appendix A.12, Table 18, and the new Corr. and Ready columns of Table 1). An asset is task-correct if it has the requested category and the intended operated part with the correct joint type, and task-ready if it is also physically valid, moves through the range the task needs, and can be opened and closed in simulation by a Franka arm whose gripper is attached to the part's handle, within the arm's reach and torque limits.
Results. Share of the inputs that yield an asset of each kind:
Over all 2,000 photographs, 33.2% yield an asset of AffordCraft with a joint that opens and closes, against 6.2–8.6% for PhysX-Anything, PhysX-Omni and PAct.
Independent validation. Three annotators who are not authors labeled the operated part and its joint type in the 200 photographs before seeing any output, then judged every accepted asset of every method with the source hidden; two annotators judged each item, and a separate adjudicator resolved disagreements (agreement in Appendix A.12). They judged the range from the measured motion of an actuation test in the gate's simulator, which drives the labeled part's joint to its limits and then has the arm open and close it, a reach, collision and torque check. The items, the images shown to the annotators, every annotator's answers, and the adjudicated labels are available at the anonymous link.