acceptodds
Under review as a conference paper at ICLR 2027

AffordCraft: Scalable Construction of Task-Ready Simulation Assets from Single Images

Abstract

Robot learning in simulation depends on the objects the simulator offers. Many tasks need objects with separate parts, joints that allow the required motion, and physical properties that remain valid under contact. Existing methods recover this structure anew for every image: generative models predict parts and joints that mostly fail to settle or move in simulation, and general-purpose agents need a long session of model calls for each photograph. AffordCraft builds such an asset from a single RGB image and a task instruction by retrieval instead of generation: it locates the object and the part to operate, selects a matching entry from a library of articulated assets, and fits it to the image while keeping its parts and joints intact. Without any box or mask marking the object, AffordCraft produces a physically valid asset for 1,703 of 2,000 photographs from 31 categories. Five generative methods pass on at most 45% of the same photographs and, at the median, need 10 to 78 times our GPU time per valid asset. On 50 cluttered images, 162 of 237 annotated objects pass the same physical test after automatic detection. Growing the library from 141 to 11,372 entries needs no change to the method and raises category coverage from 46% to 100% and the share of selections with the requested label from 18% to 51%. We also build manipulation tasks from the constructed assets, both with single objects and in composed scenes; policies trained on scripted demonstrations complete both kinds of tasks from initial states unseen in training. Code is available at https://anonymous.4open.science/r/AffordCraft-D2C5.

Primary area:
applications to robotics, autonomy, planning
Keywords:
real-to-sim, articulated objects, simulation assets, asset retrieval, robot manipulation, embodied AI

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6r8pOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

AffordCraft constructs simulation assets from a single RGB image and a task instruction by retrieving and adapting existing assets. It selects candidates with compatible parts and joints, preserves their articulation during adaptation, and checks their physical validity in simulation; if construction fails, it tries another candidate. The paper addresses an important bottleneck in embodied intelligence: obtaining diverse, functional assets for robot learning. It reports an 85.15% physical acceptance rate on 2,000 photographs, substantially outperforming the evaluated generative methods. Further experiments examine cluttered images and simulated manipulation, although transfer to real robots is not evaluated.

Critical strengths
  1. The approach directly addresses a practical limitation of robot simulation. Reusing established articulation structures reduces the uncertainty of inferring mechanisms from a single image. Combined with checks for collision geometry, inertia, and support, this produces a substantial improvement in simulator acceptance. These results support the method’s value as a tool for building usable simulation assets.

  2. The construction results are supported by a broad evaluation. The paper evaluates 2,000 photographs, several external baselines, cluttered inputs, and individual pipeline components. In particular, the ablations demonstrate the importance of physical-feedback reselection and collision processing. The detailed reporting of failures and protocol differences also helps establish the scope of the findings.

Critical weaknesses
  1. The evaluation does not yet connect physical acceptance to useful embodied learning. An asset can satisfy the simulator’s physical checks while still having the wrong actionable part, joint configuration, or motion range for the intended task. These properties are not independently validated across the benchmark. The downstream experiments provide only limited evidence: policy success is 57% on single-asset tasks and 11% on composed scenes. Moreover, all evaluated assets were seen during development, and there are no real-robot experiments or controlled comparisons showing that the constructed assets improve policy learning. The evidence therefore supports reliable asset construction more strongly than the broader claim of task readiness.

  2. The contribution of task-aware selection is not clearly established. Removing multimodal selection slightly increases category-correct physical passes from 169/200 to 173/200, although this difference is not statistically significant. Meanwhile, the task-conditioning ablation removes both the instruction and category-restricted retrieval, making it difficult to distinguish affordance reasoning from the benefit of knowing the object category. A comparison with simple category-conditioned retrieval, using the same adaptation and physical reselection, is needed to establish what the proposed selection mechanism adds.

Questions for authors
  1. What fraction of accepted assets independently matches the intended operated part, joint configuration, and usable motion range? How does the physical gate handle unspecified parts or unknown motion families?

  2. With adaptation and physical checks held fixed, does multimodal selection improve mechanism compatibility or manipulation performance over category-conditioned retrieval?

  3. Is there additional evidence of real-robot transfer or generalization to unseen assets? Do AffordCraft assets improve policy performance relative to ordinary category-matched library assets?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 6r8p (Summary, Update to Table 1, W1)

We thank the reviewer for recognizing that reusing established articulation structures addresses a practical limitation of robot simulation and that the construction results rest on a broad evaluation, and for pressing on the two points that matter most: whether accepted assets are ready for the intended task, and what task-aware selection contributes. We have uploaded the revised paper, with every change marked in blue, and updated the code at the anonymous link accordingly. The revised paper now:

  • defines task readiness and measures it for every method: 53.5% of the labeled inputs yield a task-ready asset of AffordCraft, against 27.5% for the next best method (Section 4.1, Appendix A.12);
  • separates the selector from category-restricted retrieval in a factorial study: after category-blind retrieval, the selector recovers the requested category (117 against 99 category-correct assets, p = 0.003), and when a part is named, more of its deliveries have that part moving (47 against 37, p = 0.006; Section 4.4, Appendix A.14);
  • adds policy studies on unseen assets, against category-matched library entries and with stronger learners: policies trained on our assets succeed 27.7 points more often than with category-matched entries chosen regardless of mechanism (p < 0.001), and a PPO policy succeeds in 279 of 300 held-out episodes of the door- and drawer-closing tasks (Appendix A.16);
  • adds real-robot experiments with our assets: a bowl and a plate were picked up in all 6 runs with the assets as built; with the assets scaled to the measured size, a two-drawer unit was opened and closed again in all 3 runs, and all 3 runs with a basket succeeded; in all, 7 of 12 runs succeeded as built and 9 of 12 scaled (Appendix A.17).

Update to Table 1

Table 1 now evaluates the external methods with their native up axis and retries builds that the infrastructure had interrupted; the best other physical pass rate changes from 44.55% to 46.55%. Our row reports one complete run on all 2,000 inputs with timed-out decompositions cached, the setting of its timing columns (1,713 passes; 1,703 as submitted).

W1. Physical acceptance versus task readiness

The revised paper defines task readiness, validates part, joint type and range independently, and shows that 53.5% of the labeled inputs yield a task-ready asset of AffordCraft, against 27.5% for the next best method (Section 4.1, Appendix A.12, Table 18, and the new Corr. and Ready columns of Table 1). An asset is task-correct if it has the requested category and the intended operated part with the correct joint type, and task-ready if it is also physically valid, moves through the range the task needs, and can be opened and closed in simulation by a Franka arm whose gripper is attached to the part's handle, within the arm's reach and torque limits.

Results. Share of the inputs that yield an asset of each kind:

InputsAssetAffordCraftBest other method
200 labeledtask-ready53.5% (107)27.5%
200 labeledphysically valid, task-correct62.5% (125)31.5% (GPT-6 Astra agent)
101 that need a movable parttask-correct51.5%31.7% (Articulate-Anything)
101 that need a movable partwith the requested category and a sufficient measured range as well49.5% (50)26.7% (27)
101 that need a movable partlabeled part moves through its range43.6% (44)26.7% (27)
71 whose part opens and closesthe arm opens and closes the part54.9% (39)19.7% (14)

Over all 2,000 photographs, 33.2% yield an asset of AffordCraft with a joint that opens and closes, against 6.2–8.6% for PhysX-Anything, PhysX-Omni and PAct.

Independent validation. Three annotators who are not authors labeled the operated part and its joint type in the 200 photographs before seeing any output, then judged every accepted asset of every method with the source hidden; two annotators judged each item, and a separate adjudicator resolved disagreements (agreement in Appendix A.12). They judged the range from the measured motion of an actuation test in the gate's simulator, which drives the labeled part's joint to its limits and then has the arm open and close it, a reach, collision and torque check. The items, the images shown to the annotators, every annotator's answers, and the adjudicated labels are available at the anonymous link.

AuthorsOpenReview ↗

Response to Reviewer 6r8p (W1 continued, W2)

W1 (continued)

Policy success of 57% and 11%. These rates come from a lightweight recurrent head on frozen OpenVLA-OFT features, so we trained stronger learners on the same assets (Appendix A.16). On the five door- and drawer-closing tasks, a PPO policy that reads the simulator state succeeds in 279 of 300 held-out episodes, against 57 of 100 for our head's recorded episodes replayed in the same simulator port (Table 26). On the five pushing tasks, it succeeds in 155 of 300, against 46 of 100 for our head in the port. This shows that the door, drawer, camera, clock and display tasks can be learned on these assets.

Unseen assets. On library assets never used in development (other entries AffordCraft delivered on the benchmark, split from the training pool by visual-similarity clusters; closing a door, closing a drawer, pushing), the head succeeds in 46.5% when trained on up to 40 assets per task type, against 28.7% with one (p < 0.001; Tables 22 and 23). The comparison with ordinary library entries is under Q3.

Real-robot experiments (Appendix A.17). On four real objects (a two-drawer unit, a bowl, a plate, and a basket), with our assets as built, the bowl and the plate were picked up in all 6 runs; with the assets scaled to the measured size, the drawer unit was opened and closed again in all 3 runs, the plate was picked up in all 3 runs, and all 3 runs with the basket succeeded. In all, 7 of 12 runs succeeded as built and 9 of 12 scaled (Appendix A.17). A Franka Research 3 arm with a two-finger gripper ran the motions taken from the assets. These motions succeeded on real objects that differ from the assets in appearance and size. We have uploaded videos of these experiments to the supplementary material. More real-robot experiments are under way; we will add their results to this thread shortly.

Controlled comparison. We held out 65 PartNet-Mobility objects with a door or drawer, rendered a photograph of each from its mesh, and removed each object and its near-duplicates from the library. Policies trained on our assets succeed in 34.9% of the episodes on these objects, 27.7 points more often than with a random entry of the same category regardless of mechanism (p < 0.001; Appendix A.16, Table 24), which usually lacks the needed part (it has one for 13 of the 65 objects, against 54 for ours).

Title. We keep "Task-Ready" in the title: the revised paper defines task readiness and measures it for every method, and AffordCraft yields a task-ready asset for 53.5% of the labeled inputs, against 27.5% for the next best method (Table 1, Appendix A.12, Table 18).

W2. Contribution of task-aware selection

A factorial study separates the two parts of the task condition (Section 4.4). When a part is named (without task verification), the selector's first choice more often has that part and more of its deliveries have a named part that moves, and where the selector and Table 2's category-conditioned retrieval deliver different entries, annotators preferred the selector's asset (22 against 7, p = 0.008).

The baseline. Table 2's "Category-conditioned retrieval (no selector)" row uses the same candidate pool, adaptation, repair, reselection, decomposition cascade and gate, building candidates in retrieval order (Section 4.4).

New settings (Section 4.4, Appendix A.14, Table 20). They add task verification, a catalog-metadata rule applied before selection (when the instruction names a part, only candidates whose metadata carry it with the implied joint type and a usable range are kept, and the system abstains if none is), a per-input time budget of 1,800 s, and movable-part labels for the selector. The primary metric is the human label, with exact two-sided McNemar tests paired by input.

Category knowledge versus the selector. Crossing retrieval (category-restricted or category-blind) with selection (instructed selector, selector without the instruction, retrieval order) on the 200 inputs shows where the effect of the task condition comes from. After category-blind retrieval, the instructed selector delivers 117 category-correct assets against 99 for retrieval order (p = 0.003), and 118 without the instruction: the selector recovers the category by reading catalog categories, which shows that Table 2's task-condition effect (125 against 169) reflects the category restriction.

AuthorsOpenReview ↗

Response to Reviewer 6r8p (W2 continued, Q1-Q3)

W2 (continued)

Part-specified instructions (118 inputs). Without task verification, more of the selector's deliveries have a named part that moves when driven (47 against 37, p = 0.006), and on 565 held-out objects its first choice more often has the named part (85.1% against 79.8%, p < 0.001; reworded beyond the verification word lists, 83.5% against 79.3%, p = 0.0012). With verification in both arms, which isolates the selector, the results suggest a further small gain (53 against 49 inputs with a task-correct asset and a sufficient measured range, p = 0.22).

Missing mechanism. With every movable entry of the requested category removed from the library, task verification abstains on 57 of the 118 part-specified inputs, whereas retrieval order without it delivers 101 assets; by human labels, 108 against 17 inputs end without a substitute (p < 0.001). The abstention comes from verification, which retrieval order with the same rule shares.

Q1. Independent fractions; unspecified parts and motions

Independent labels give the fraction of accepted assets with the intended part, joint type and range for every method, and we clarify that the physical gate does not use the grounded part or motion. Of AffordCraft's 83 accepted assets for the 101 inputs that need a movable part, 68.7% have the labeled part, 62.7% also its correct joint type, and 60.2% also the requested category and a sufficient measured range; per accepted asset, the GPT-6 Astra agent is higher on all three and Articulate-Anything on the first two, each on 38 accepted assets against our 83 (W1; Table 18).

Eq. (5) and Section 3.3 now describe the gate's structural joint term (every revolute or prismatic joint keeps valid limits and consistent frames and axes after adaptation, and an entry with a movable joint keeps at least one enabled). The grounded part and motion go, with the instruction, to the selector, which judges mechanism compatibility from the photograph, the candidate renders and each candidate's catalog category, part labels and joint types; when the part is unspecified or the motion unknown, it compares the candidates with the requested category and the photographed object. That is the common case on the benchmark (part unspecified for 75.7% of the localized inputs, motion unknown for 61.0%), because the grounding prompt does not invent a manipulation task when the instruction asks only for an asset (Appendix A.2). In the configuration of Section 4.4, task verification turns a part named in the instruction into a requirement; category-only instructions set none.

Q2. Selection versus category-conditioned retrieval

With adaptation and checks fixed, selection improves mechanism compatibility when the instruction names a part, and in Table 2 it is on par on category-only instructions. Without task verification, more of the selector's deliveries have a named part that moves (47 against 37, p = 0.006), and its first choice on 565 held-out objects more often has the named part (85.1% against 79.8%, p < 0.001); with verification in both arms, the results suggest a further small gain (53 against 49, p = 0.22; W2, Appendix A.14, Table 20). As for manipulation, annotators compared the two assets with the source hidden wherever both conditions pass but deliver different entries, and preferred the selector's asset for 22 of 57 inputs and the baseline's for 7 (exact sign test, p = 0.008); the actuation test of W1 measures each asset on its own.

Q3. Real-robot experiments, unseen assets, category-matched library assets

Appendix A.17 adds real-robot experiments with our assets (videos in the supplementary material), and the policy studies show that success on unseen assets rises with the number of our training assets and that policies trained on our assets succeed far more often than with ordinary category-matched library entries, and on par with entries matched in mechanism as well.

  • Real-robot experiments: On four real objects, a bowl and a plate were picked up in all 6 runs with the assets as built; with the assets scaled to the measured size, a two-drawer unit was opened and closed again in all 3 runs, and all 3 runs with a basket succeeded; in all, 7 of 12 runs succeeded as built and 9 of 12 scaled (W1, Appendix A.17).
  • Unseen assets: on library assets never used in development, success rises from 28.7% with one training asset per task type to 46.5% with up to 40 (p < 0.001; W1; Appendix A.16, Tables 22 and 23).
  • Category-matched library assets: policies trained on our assets succeed 27.7 points more often than with a random entry of the same category regardless of mechanism, which usually lacks the needed part (p < 0.001; Appendix A.16, Table 24).

We sincerely thank the reviewer for the constructive feedback and hope that the new experiments and clarifications have resolved these concerns. We are happy to discuss any further questions.

Reviewer_GfjFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

AffordCraft constructs simulation assets from an RGB image and a task instruction by retrieving and adapting existing articulated assets rather than reconstructing each mechanism from scratch. The pipeline grounds the target, retrieves candidates, applies multimodal selection, preserves bodies and joints during geometric adaptation, and uses physical checks with bounded repair and reselection. It reports a physical pass rate of 85.15% on 2,000 photographs and demonstrates manipulation tasks using the resulting assets.

Critical strengths
  1. The method addresses concrete simulation failure modes. Preserving articulation, scaling prismatic limits consistently, auditing collision geometry, and reloading the final artifact in a separate process are practically important. This is a more useful target than visual plausibility alone.
  2. The asset-construction advantage is large under the stated gate.
  3. The reporting distinguishes several commonly conflated protocols.
Critical weaknesses
  1. The main physical pass metric does not independently verify task correspondence. 2.The system’s behavior under mechanism unavailability is insufficiently characterized.
  2. A clearer scope and cost accounting is preferred for the efficiency and downstream-utility conclusions
Questions for authors
  1. What fraction of inputs produces an asset that is both physically valid and independently verified as task-correct?
  2. How does the full system compare with a complete retrieval-plus-physical-reselection baseline?
  3. When the library lacks the required mechanism, how often does the system reject correctly versus accept a physically valid but functionally mismatched substitute? Can existing logs support an error analysis?
  4. What are the online and amortized costs under a clearly specified deployment workload?
5 comments
AuthorsOpenReview ↗

Response to Reviewer GfjF (Summary, Update to Table 1, W1/Q1)

We thank the reviewer for recognizing that the method targets concrete simulation failure modes, that its construction advantage is large under the stated gate, and that the reporting separates commonly conflated protocols, and for asking for independent task verification, behavior under a missing mechanism, and cost and scope under a stated workload. We have uploaded the revised paper, with every change marked in blue, and updated the code at the anonymous link accordingly. The main additions are:

  • a human annotation and an actuation test of every method's assets: 62.5% of the labeled inputs yield a physically valid, task-correct asset of AffordCraft, against at most 31.5% for another method (Section 4.2, Appendix A.12);
  • a study with the needed mechanism removed from the library: our system delivers no mismatched substitute for 108 of 118 inputs, against 17 for retrieval order (p < 0.001; Section 4.4, Appendix A.14);
  • total compute under a stated workload, also reported in Table 1 next to the median and the mean time per input: online, 3.48 GPU-minutes per accepted asset for AffordCraft, against at least 5.37 for the other local methods, and within five minutes per input, 71.5% of the inputs accepted, against at most 28.0% (Table 1, Appendix A.15);
  • three policy studies, with the scope of the policy results stated: with up to 40 training assets per task type, success on unseen assets rises to 46.5%, from 28.7% with one (p < 0.001; Section 4.5, Appendix A.16);
  • real-robot experiments with our assets: a bowl and a plate were picked up in all 6 runs with the assets as built; with the assets scaled to the measured size, a two-drawer unit was opened and closed again in all 3 runs, and all 3 runs with a basket succeeded; in all, 7 of 12 runs succeeded as built and 9 of 12 scaled (Appendix A.17).

Update to Table 1

Table 1 now evaluates the external methods with their native up axis and retries builds that the infrastructure had interrupted; the best other physical pass rate changes from 44.55% to 46.55%. Our row reports one complete run on all 2,000 inputs with timed-out decompositions cached, the setting of its timing columns (1,713 passes; 1,703 as submitted).

W1 / Q1. Task correspondence

Human annotation and an actuation test now verify task correspondence for every method, and they show that AffordCraft yields a physically valid, task-correct asset for about twice as many inputs as any other method (Section 4.2, Appendix A.12, Tables 1 and 18). Answer to Q1, as the share of the inputs that yield an asset of each kind (task-correct: requested category, labeled part, correct joint type; task-ready: also moves through the needed range and can be opened and closed by the arm):

InputsAssetAffordCraftBest other method
200 labeledphysically valid, task-correct62.5% (125)31.5% (GPT-6 Astra agent)
200 labeledtask-ready (Table 1, Ready)53.5% (107)27.5%
101 whose photograph needs a movable parttask-correct51.5%31.7% (Articulate-Anything)

Three annotators who are not authors first labeled every photograph of the 200-input subset (Appendix A.8) with the part a manipulation would operate and its joint type, before seeing any output. They then judged every accepted asset of every method on these inputs with the source hidden; two annotators judged each item, and a separate adjudicator resolved disagreements (agreement in Appendix A.12). The items, the images shown to the annotators, every annotator's answers, and the adjudicated labels are available at the anonymous link.

The actuation test runs in the gate's simulator on the joint of the part labeled in the photograph. Driven to its limits, the joint moves through its range for 44 of the 101 inputs that need a movable part (43.6%), against 27 (26.7%) for the best other method. A Franka arm whose gripper is attached to the handle opens and closes the part, within reach and torque limits, for 39 of the 71 inputs whose labeled part opens and closes (54.9%), against 14 (19.7%).

Eq. (5) and Section 3.3 now describe the gate's joint term exactly: it is structural (every revolute or prismatic joint keeps valid limits and consistent frames and axes after adaptation, and an entry with a movable joint keeps at least one enabled). Part and motion are matched at selection, and the measurements above verify them.

AuthorsOpenReview ↗

Response to Reviewer GfjF (W2/Q3, W3/Q4)

W2 / Q3. Missing mechanisms

With the needed mechanism removed from the library, task verification abstains on about half of the inputs, and our system ends without a substitute on 108 of 118 inputs, against 17 for retrieval order (p < 0.001). With every movable entry of the requested category removed, our system with task verification abstains on 57 of 118 part-specified inputs (48.3%), while retrieval order delivers an asset for 101 (85.6%), 92 of them of the requested catalog category; by human labels, the 108 inputs without a substitute end with no delivery or a task-correct one. The abstentions come from task verification (Section 3.2): before selection, a rule on catalog metadata keeps only candidates whose part metadata carry the named part with the implied joint type and a usable range, and the system abstains if none does; retrieval order with the same rule abstains likewise. Without the rule, the selector still ends without a substitute on more inputs than retrieval order (23 against 17, p = 0.031).

Design. The 118 inputs are those of the 200-input subset whose instruction names the operated part ("… so that a robot can open its door"; parts from an automatic LLM annotation). Same-category entries without a movable part and other-category entries with a same-named part remain in the library, so substitutes stay possible. Abstention requires a named part that the verification word lists recognize, and the lists were written against these instructions (Section 4.4); on category-only instructions there is nothing to verify.

Existing logs. The logs record each input's ordered candidate pool, the selector's verdict on every candidate and the construction and gate outcome of each attempt, and every library entry carries part and joint metadata, which supports an error analysis of failures (Figure 4). They contain no case of a missing mechanism: all 54 benchmark inputs that exhausted the candidate budget (Appendix A.3) had a movable entry of the requested category in their pool. The controlled removal above therefore answers Q3.

W3 / Q4. Cost under a stated workload

Under a stated workload, AffordCraft is the cheapest method per accepted asset online, at the median and in total, and under time budgets of two, five and ten minutes per input. The workload is the construction of assets for N new photographs with the library already built, N = 2,000 (our benchmark) and 20,000 (Appendix A.15, Table 21). Table 1 now reports the mean time per input and the total GPU-minutes per accepted asset next to the median-based measure. We timed every local method on 200 inputs, one job per dedicated RTX 5090. Totals are in GPU-minutes per accepted asset; each cap column gives the share of inputs accepted when every input stops at the cap, at the GPU-minutes per accepted asset:

MethodOnline totalAmortized, N = 2,000Amortized, N = 20,0002-min cap5-min cap10-min cap
AffordCraft3.4812.554.3957.5% at 1.8071.5% at 2.3478.5% at 2.84
PAct5.375.375.3727.0% at 4.9728.0% at 5.2828.0% at 5.37
PhysX-Anything19.4719.4719.473.0% at 65.5612.5% at 36.8333.5% at 22.46
PhysX-Omni25.2525.2525.253.5% at 56.7613.0% at 35.8826.5% at 27.87
TRELLIS.230.6230.6230.620.0%3.5% at 139.0821.0% at 35.65
PartCrafter76.8376.8376.833.5% at 55.326.5% at 63.257.5% at 74.04
  • Online, at the median input: AffordCraft needs 44 s per input and 0.86 GPU-minutes per accepted asset, against 80–525 s and 5.0–83 GPU-minutes for the generative methods; per task-ready asset it needs 1.3 GPU-minutes, against 41.4 for the cheapest other method (Table 1).
  • Online, counting every input in full (wall-clock time over all timed inputs, failures and every candidate attempt included, divided by accepted assets): 3.48 GPU-minutes per accepted asset with timed-out decompositions cached, as in Table 1, against 5.37 to 76.83 for the other local methods. AffordCraft's mean, median and P90 latency per input are 2.92, 0.73 and 7.9 minutes.
  • One-time: 1,190 s for the visual index and 254 worker-hours for the collision-geometry cache, including the time that first produced the cached timeouts.
  • Amortized: the one-time cost spread over the assets accepted among N inputs, plus the online total; the other methods' totals do not depend on N.
  • Budgeted: with a five-minute cap per input, AffordCraft accepts 71.5% of the timed inputs, against at most 28.0% for any other method; it also leads with caps of two and ten minutes (Figure 5, Table 21).

These figures use a warm geometry cache with timed-out decompositions cached, as in Table 1; without caching them, the online total is 10.74 and the amortized cost 19.35 and 11.60, and with an empty cache the online total is 23.79 (Table 21).

AuthorsOpenReview ↗

Response to Reviewer GfjF (W3 continued, Q2)

W3. Scope of the downstream results

Three new studies show that policies trained on our assets succeed on unseen library assets and on held-out simulated objects, and Section 4.5 now states the scope of the policy results: they are in simulation. On its ten single-asset tasks, our head succeeds in 114 of 200 episodes from unseen initial states (untrained: 1 of 200). Appendix A.16 adds three studies:

StudySuccess (episodes)Comparison
Unseen assets of the same library (closing a door, closing a drawer, pushing into a goal region): the head with up to 40 training assets per task type, against one (p < 0.001; Tables 22 and 23)46.5%28.7%
65 held-out simulated objects (renders of PartNet-Mobility objects with a door or drawer): policies trained on our assets, against random entries of the category alone, which mostly lack the needed part (p < 0.001; Table 24)34.9%7.1%
Stronger learners (Appendix A.16, Table 26): a state-reading PPO policy learns the five door- and drawer-closing tasks, against our head replayed in the same simulator port279 of 30057 of 100
Stronger learners: the five pushing tasks, of which the PPO policy learns three, against the same head155 of 30046 of 100

Q2. Retrieval-plus-reselection baseline

The complete baseline is Table 2's category-conditioned retrieval with the same pool, adaptation and physical-feedback reselection; in Table 2 the two are on par on category-only instructions, and the full system adds most where selection has more to check. The baseline is the "Category-conditioned retrieval (no selector)" row: the same candidate pool (at most twelve entries of the requested category by DINOv2 similarity, then entries of any label), adaptation, repair, physical-feedback reselection, decomposition cascade and gate, with candidates built in retrieval order. The construction advantage over the generative methods comes from what the two share, library retrieval with articulation-preserving adaptation and physical-feedback reselection: in Table 2, removing the articulation adapter costs 40 percentage points of correct assets, the task condition 22 and reselection 11.

The full system (selector) against the baseline (retrieval order), where the two deliver different entries and where selection has more to check (Section 4.4, Appendix A.14, Table 20: task verification, a per-input time budget of 1,800 s and movable-part labels for the selector; human labels of W1 as the primary metric):

ComparisonSelectorRetrieval orderp
Where the full system and the baseline deliver different entries: annotators' preferred asset2270.008
Category-blind retrieval (visual similarity over the whole library), where the baseline cannot filter by category: category-correct assets117990.003
Instructions that name the operated part (118 inputs), without verification: deliveries with a named part that moves47370.006
The same instructions, on 565 held-out objects: first choice has the named part85.1%79.8%< 0.001

After category-blind retrieval, the selector delivers its category-correct assets by reading each candidate's catalog category, part labels and joint types.

We sincerely thank the reviewer for the constructive feedback and hope that the new experiments and clarifications have resolved these concerns. We are happy to discuss any further questions.

Reviewer_nP9dOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AffordCraft, a framework to construct functional simulation environments from raw RGB image observation and language instruction. The pipeline first retrieves exciting assets and select potential candidates by visual and structural evidence, adapt to preserve joints, parts for the articulation. The final results are validated in the simulation. The results are robust, across single-object reconstruction and cluttered scenes. They also ablate performance against agent pipelines and modular designs, retrieve library scaling. The simulation manipulation experiments show that the pipeline enables policy learning on the constructed articulated assets.

Critical strengths
  • Reusing well-built articulation structure from library is a good method for robustness. It clearly distinguishes functional substitutes from exact geometry reconstruction. It is a good problem formulation for simulation usage.
  • The experiment parts are comprehensive, expanding from native/adapted baselines, to paired ablations, separate construction and simulation validation. Many details are included in the appendix, showing the careful design and the robustness of the design.
  • The policy learning part is sufficient. It shows that exported assets from the pipeline's construction can support learned interaction and has relatively high success rate. The articulation-preserving adapter and physical feedback reselection have significant effects.
Critical weaknesses
  • The paper claims task-correct substitution. However physical acceptance does not validate it sufficiently. The physical gates does not check the object identity and the joint test uses grounded motion quires. The correct metric checks selected catalog categories and they do not provide independent ground truth that can deliver asset supports the motion that required by observations and conditions.
  • The multimodal structural selection argument is not well-supported. The results show that removing it increases the correct counts, but it does not show that remove it helps or providing no demonstrated benefit for retraining it.
  • In Tab 1 and 9, they show the headline cost ratio. It is a median-based surrogate, not necessarily a good estimate cost for asset construction. The median latency divided by success prob does not equal total compute divided by successful outputs, especially when there are multiple attempts.
Questions for authors

Please see above. Additionally,

  • how often does the physically valid result fail the compatibility check? Can independent labels verify operated parts, notions and geometry for accepted outputs?
  • How does the whole pipeline compare with category-conditioned retrieval with same adapter/budget/gate? what capability was enabled by multimodal selector as shown in Tab 2?
6 comments
AuthorsOpenReview ↗

Response to Reviewer nP9d (Summary, Update to Table 1, W1)

We thank the reviewer for recognizing that reusing library articulation structure makes construction robust and that the paper separates functional substitutes from exact reconstruction, and for three precise questions: whether physical acceptance shows task-correct substitution, what multimodal selection adds, and whether a median-based ratio measures cost. We have uploaded the revised paper, with every change marked in blue, and updated the code at the anonymous link accordingly. For this review, the revision adds:

  • an independent check of task correspondence for every method: by human labels, 62.5% of the labeled inputs yield a physically valid, task-correct asset of AffordCraft, against at most 31.5% for another method (Section 4.2, Appendix A.12);
  • tests of the selector where it has more to check: its first choice more often has the part named in the instruction (85.1% against 79.8%, p < 0.001), and after category-blind retrieval it recovers the requested category (117 against 99 category-correct assets, p = 0.003; Section 4.4, Appendix A.14);
  • the mean time per input and the total GPU-minutes per accepted asset, failures and every candidate attempt included, next to the median in Table 1: in total, 3.48 GPU-minutes per accepted asset for AffordCraft with its library built, against 5.37 to 76.83 for the other local methods (Table 1, Appendix A.15);
  • real-robot experiments with our assets: a bowl and a plate were picked up in all 6 runs with the assets as built; with the assets scaled to the measured size, a two-drawer unit was opened and closed again in all 3 runs, and all 3 runs with a basket succeeded; in all, 7 of 12 runs succeeded as built and 9 of 12 scaled (Appendix A.17).

Update to Table 1

Table 1 now evaluates the external methods with their native up axis and retries builds that the infrastructure had interrupted; the best other physical pass rate changes from 44.55% to 46.55%. Our row reports one complete run on all 2,000 inputs with timed-out decompositions cached, the setting of its timing columns (1,713 passes; 1,703 as submitted).

W1. Task-correct substitution

Independent human labels and an actuation test now show that AffordCraft yields task-correct and task-ready assets for about twice as many inputs as any other method (Section 4.2, Appendix A.12, Table 18). Share of the inputs that yield an asset of each kind (task-correct: requested category, labeled part, correct joint type; task-ready: also moves through the needed range and can be opened and closed by a simulated arm):

InputsAssetAffordCraftBest other method
200 labeledphysically valid, task-correct62.5% (125)31.5% (GPT-6 Astra agent)
200 labeledtask-ready53.5% (107)27.5%
101 that need a movable parttask-correct51.5%31.7% (Articulate-Anything)
101 that need a movable partwith the requested category and a sufficient measured range as well49.5% (50)26.7% (27)
101 that need a movable partlabeled part's joint, driven to its limits, moves through its range43.6%26.7%
71 whose labeled part opens and closesa simulated arm opens and closes the part54.9%19.7%

Human labels. Three annotators who are not authors labeled the part to operate and its joint type in every photograph of the 200-input subset before seeing any output, then judged every accepted asset of every method with its source hidden: category, that part, its joint type, plausible motion, whether the part reaches the position the task needs (from the measured motion), and whether the movable parts match the photograph in number and arrangement. Two annotators judged each item, and a separate adjudicator resolved disagreements; agreement is reported in Appendix A.12. The items, the images shown to the annotators, every annotator's answers, and the adjudicated labels are available at the anonymous link.

Actuation test. In the gate's simulator, each joint of every accepted articulated asset is driven to its limits; a Franka arm whose gripper is attached to the part's handle then opens and closes it, a reach, collision and torque check. Over all 2,000 photographs, 33.2% yield an asset of AffordCraft with a joint the arm opens and closes, against at most 8.6% for the generative methods.

The gate. We clarify that the gate's joint term is structural (every revolute or prismatic joint keeps valid limits, frames and axes after adaptation, and an entry with a movable joint keeps one enabled) and does not read the grounded part or motion; Eq. (5) and Section 3.3 now say so. Part and motion are matched at selection, and the measurements above verify them.

AuthorsOpenReview ↗

Response to Reviewer nP9d (W2-W3)

W2. Multimodal selection

The new studies show what the selector adds, reading named parts and catalog text (Contribution 2, Section 4.4, Appendix A.14, Table 20). We clarify that the difference in Table 2 (173 category-correct assets without the selector against 169 with it) is not significant (p = 0.12): the benchmark's instructions name only the category, which retrieval already enforces. The selector against retrieval order:

ComparisonSelectorRetrieval orderp
Instruction names a part and no other rule checks it: first choice has that part (565 held-out PartNet-Mobility objects; Table 28)85.1%79.8%< 0.001
Same condition: deliveries with a named part that moves (118 inputs)47370.006
After category-blind retrieval: category-correct assets (of 200), the requested category recovered by reading catalog text117990.003
Table 2, where the selector and retrieval order deliver different entries: annotators' preferred asset2270.008

Setup. The new runs add task verification (Section 3.2: when the instruction names a part, a catalog-metadata rule keeps only candidates whose part metadata carry it with the implied joint type and a usable range, and abstains if none does), a 1,800 s budget per input, and movable-part labels for the selector, which reads each candidate's catalog category, part labels and joint types. They are reported apart from Tables 1 and 2. Tests are exact two-sided McNemar tests paired by input, with the human labels of W1 as the primary metric.

  • Category-blind retrieval (visual similarity over the whole library): the instructed selector delivers 117 category-correct assets against 99 for retrieval order (p = 0.003). The capability is reading catalog text: shown only the candidate images, a selector delivers 97 against 94 by catalog category (p = 0.55).
  • Part-specified instructions (118 inputs whose parts come from an automatic annotation): without task verification, the named part moves in more of the selector's deliveries (47 against 37, p = 0.006); with verification in both arms, which isolates the selector, the results suggest a further small gain (53 against 49 inputs with a task-correct asset and a sufficient measured range, p = 0.22).
  • Missing mechanism (every movable entry of the requested category removed): task verification abstains on 57 of the 118 inputs, while retrieval order delivers 101 assets; by human labels, 108 against 17 inputs end without a substitute (p < 0.001). The abstention comes from the rule, which acts before selection.

W3. Cost metric

Table 1 now reports, next to the median, the mean time per input and the total GPU-minutes per accepted asset, failures and every candidate attempt included; by this total, AffordCraft is the cheapest method per accepted asset in the configuration of Table 1 and under time budgets of two, five and ten minutes per input (Section 4.2, Table 1, Appendix A.15, Table 21). All three measures come from one timing run of every local method on 200 inputs, one job per dedicated RTX 5090:

MethodMedian time (s)Mean time (s)Median-based GPU-minutes per accepted assetTotal GPU-minutes per accepted asset
AffordCraft44.1175.50.863.48
PAct79.790.25.05.37
PhysX-Anything525.2601.818.819.47
PhysX-Omni489.6545.522.425.25
TRELLIS.2447.8541.927.130.62
PartCrafter304.4345.783.276.83
  • Mean and total: AffordCraft's mean time per input is 175.5 s; PAct's mean is lower (90.2 s), but PAct accepts 28.0% of the timed inputs, against 84.0% for AffordCraft, and per accepted asset the total is lowest for AffordCraft once its library is built.
  • Under a time budget: within five minutes per input it accepts 71.5% of the timed inputs at 2.34 GPU-minutes per accepted asset, against at most 28.0% for any other method, and it also leads with budgets of two and ten minutes.
  • At the median: 0.86 GPU-minutes per accepted asset, against 5.0 to 83 for the generative methods, and 1.3 per task-ready asset, against at least 41.4 for the other methods.
  • One-time and amortized: the one-time cost is 254 worker-hours of collision geometry, including the time that first produced the cached timeouts, and 1,190 s of visual indexing; spread over the assets accepted among N inputs, the total is 12.55 GPU-minutes per accepted asset at N = 2,000 and 4.39 at N = 20,000.

Without caching timed-out decompositions, the online total is 10.74 (amortized 19.35 and 11.60), above PAct's 5.37, as 20 of the 200 inputs take over 24 minutes each (Appendix A.15, Table 21). Every row of Table 1 now counts every timed input (Appendix A.9), and the derived throughput row of Table 9 is removed.

AuthorsOpenReview ↗

Response to Reviewer nP9d (Q1-Q2)

Q1. Incompatible accepted results; independent labels

Independent labels now measure, for every method, how many accepted assets have the labeled part, its joint type and a sufficient range. Of AffordCraft's 83 accepted assets for the 101 labeled inputs that need a movable part, 68.7% have the labeled part, 62.7% also its correct joint type, and 60.2% also the requested category and a sufficient measured range; per accepted asset, the GPT-6 Astra agent is higher on all three and Articulate-Anything on the first two, each on 38 accepted assets against our 83 (Table 18). Over all 200 labeled inputs, 168 get an accepted asset of AffordCraft and 125 (62.5%) a task-correct one.

The labels verify operated parts and motions: annotators who are not authors labeled the part, its joint type and plausible motion, and whether the part reaches the needed position, judged from the measured motion of the actuation test (W1). For geometry, they judged structure, the number and arrangement of movable parts, rather than surface shape, because AffordCraft builds a functional substitute, not a copy (Section 1). On 565 held-out PartNet-Mobility objects with annotated structure, the selector's first choice matches it for 61.9%, well above a random entry of the category (34.3%; Appendix A.14, Table 28).

Q2. Comparison with category-conditioned retrieval; what the selector adds

The comparison is Table 2's "Category-conditioned retrieval (no selector)" row, defined in Section 4.4: the same candidate pool, adaptation, reselection, candidate budget and gate, with candidates built in retrieval order. Annotators preferred the selector's asset where the two differ (22 against 7, p = 0.008). The capability the selector adds is measured in W2: it reads catalog text, recovering the requested category after category-blind retrieval (117 against 99, p = 0.003), and it reads a named part, choosing and delivering entries that have it (85.1% against 79.8%, p < 0.001; 47 against 37, p = 0.006). Abstaining on a missing mechanism is a property of task verification, a catalog-metadata rule applied before selection (Section 3.2).

We sincerely thank the reviewer for the constructive feedback and hope that the new experiments and clarifications have resolved these concerns. We are happy to discuss any further questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Follow-up to All Reviewers on Our Responses and the Revised Paper

Dear Reviewers,

Thank you again for your careful reviews. We are writing to follow up on our responses and the revised paper (changes marked in blue). Its main additions are:

  • Task readiness, measured for every method. 53.5% of the labeled inputs yield a task-ready asset of AffordCraft, against 27.5% for the next best method, and 62.5% a physically valid, task-correct one, against at most 31.5% (human labels and an actuation test; Sections 4.1 and 4.2, Appendix A.12).
  • What the selector adds. Its first choice more often has the part named in the instruction (85.1% against 79.8%, p < 0.001), and after category-blind retrieval it recovers the requested category (117 against 99 category-correct assets, p = 0.003; Section 4.4, Appendix A.14).
  • Missing mechanisms. With the needed mechanism removed from the library, our system delivers no mismatched substitute for 108 of 118 inputs, against 17 for retrieval order (p < 0.001; Section 4.4, Appendix A.14).
  • Cost. In total, failures and every candidate attempt included, AffordCraft needs 3.48 GPU-minutes per accepted asset with its library built, against 5.37 to 76.83 for the other local methods (Table 1, Appendix A.15).
  • Policy studies and real-robot experiments. Policies trained on our assets succeed 27.7 points more often than with category-matched library entries chosen regardless of mechanism (p < 0.001; Appendix A.16), and on seven real objects the motions taken from our assets succeeded in 15 of 21 runs as built and 18 of 21 scaled (Appendix A.17).

We would be grateful to know whether our responses address your concerns. If any point remains open, we will gladly clarify it or add further analysis during the discussion period.

Thank you for your time and consideration.

AuthorsOpenReview ↗

Summary of Our Rebuttal and the Revised Paper

Dear Reviewers and Area Chair,

We sincerely thank all reviewers for their careful and constructive reviews, and the Area Chair for handling our submission. As the author-reviewer discussion closes, we summarize how our rebuttal and the revised paper (changes marked in blue) address the main concerns.

Strengths noted by the reviewers. Reusing library articulation addresses practical limitations of robot simulation (nP9d, GfjF, 6r8p); the asset-construction advantage is large and the improvement in simulator acceptance substantial (GfjF, 6r8p); the evaluation is comprehensive and broad (nP9d, 6r8p).

Main concerns and what the revision adds.

Concern (reviewer points)What the revision addsMain result
Physical acceptance versus task readiness (nP9d W1, Q1; GfjF W1, Q1; 6r8p W1, Q1)Human labels and an actuation test of every method's assets (Sections 4.1 and 4.2, Appendix A.12)A task-ready asset for 53.5% of the labeled inputs, against 27.5% for the next best method; a physically valid, task-correct one for 62.5%, against at most 31.5%
What task-aware selection adds over category-conditioned retrieval (nP9d W2, Q2; GfjF Q2; 6r8p W2, Q2)A pairwise preference with the source hidden, and a factorial study that separates the selector from category-restricted retrieval (Section 4.4, Appendix A.14)Annotators preferred the selector's asset where the two differ (22 against 7, p = 0.008); after category-blind retrieval, 117 against 99 category-correct assets (p = 0.003)
Missing mechanisms (GfjF W2, Q3)The needed mechanism removed from the library (Section 4.4, Appendix A.14)No mismatched substitute for 108 of 118 inputs, against 17 for retrieval order (p < 0.001)
Cost metric (nP9d W3; GfjF W3, Q4)Mean time per input and total GPU-minutes per accepted asset in Table 1 (Appendix A.15)3.48 GPU-minutes per accepted asset with the library built, against 5.37 to 76.83 for the other local methods
Downstream value and unseen assets (GfjF W3; 6r8p W1, Q3)Policy studies on unseen assets and against category-matched library entries (Section 4.5, Appendix A.16)Success 27.7 points higher than with category-matched library entries chosen regardless of mechanism (p < 0.001); on unseen assets, success rises from 28.7% with one training asset per task type to 46.5% with up to 40 (p < 0.001)
Real-robot evidence (6r8p W1, Q3)Real-robot experiments on seven real objects (Section 4.5, Appendix A.17, supplementary video)The motions taken from our assets succeeded in 15 of 21 runs as built and 18 of 21 scaled to the measured size

Further updates. Videos of the real-robot experiments are in the supplementary material. The code at the anonymous link has been updated, and the annotation items, the images shown to the annotators, every annotator's answers, and the adjudicated labels are available there.

Discussion. Reviewer nP9d has replied that our rebuttal addressed the reviewer's major concerns. Our point-by-point responses under each review give the details behind every row of the table.

We sincerely thank the reviewers and the Area Chair for their time and constructive feedback, and hope that the new experiments and clarifications have resolved the concerns raised.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.