- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
USDCraft reconstructs simulation-ready articulated 3D assets from a source mesh and a reference image using a pretrained LLM without task-specific fine-tuning. The method converts source geometry into a metric textual representation that distinguishes observed surfaces from unknown space. An agent uses this representation, local mesh inspection, and iterative geometric and visual feedback to construct and revise an executable asset program. The output includes geometry, articulation, and physical parameters for simulation. The paper evaluates reconstruction on USDCraft-bench and the public Lightwheel dataset, studies text- and image-conditioned generation, demonstrates real-to-sim-to-real manipulation, and introduces the USDCraft-10k asset library.
Critical strengths
- Preserving measured geometry while allowing completion of unobserved regions is appropriate for imperfect meshes. The distinction between observed surface support and unknown space is useful. The executable representation also makes geometry, joints, and physical properties explicitly editable.
- Mini Workflow uses the same backbone and image/mesh inputs, providing an informative comparison. With Astra, USDCraft improves part F1 from 74.48% to 84.83% and reduces joint-axis angular error from 21.14 to 6.83 degrees (Table 3). These are substantial gains, and the repeated-run results in Table 9 provide additional support for their consistency.
- The experiments include the external Lightwheel benchmark with 243 objects across 14 categories, multiple model families, cumulative and removal ablations, and sensitivity analyses excluding particular sources of benchmark assets. This coverage strengthens the empirical case beyond the authors’ own generated examples.
- The robot experiments connect asset reconstruction to policy transfer rather than relying solely on visual quality. The paper also reports runtime and API costs, and discusses limitations involving fine structures and deformable objects, helping readers assess the method’s practical scope.
Critical weaknesses
- Section 3.2 and Appendix A.4 describe the sampling procedure and grid resolutions, and Appendix G.6 provides a useful encoding example. However, the algorithms for detecting and summarizing protrusions, openings, and observed interior structures are not described sufficiently. The relevant geometric criteria, thresholds, and treatment of noisy or incomplete surfaces are unclear. Since these summaries provide measurement anchors for reconstruction, this limits reproducibility of a central component.
- Tables 6 and 7 support the usefulness of source geometry analysis as a module, but this module combines axial slices, local numerical summaries, and on-demand mesh inspection. The complete system also uses images and subsequent geometric feedback. Consequently, the results do not isolate how much geometric information the fixed encoding preserves or how effectively the LLM uses it. The largest incremental improvement in Table 6 occurs when adding the authoring harness, making a more targeted analysis of the encoding particularly valuable.
- Although Lightwheel provides external validation, Table 2 does not include Mini Workflow, leaving the direct comparison against a general-purpose agent primarily on the authors’ benchmark.
Questions for authors
- Could you evaluate the same fixed encoding under three reconstruction settings: (a) rule-based reconstruction, (b) GPT reconstruction without an image, and (c) GPT reconstruction with the reference image? Additional source-mesh queries and source-based geometric feedback should be disabled, with the same encoding information available across settings and matched model, tools, and inference budgets between (b) and (c). Surface-based evaluation of the observed geometry would help distinguish encoding/decoding distortion, the model’s ability to use the representation, and the additional contribution of the image. Since the encoding is lossy, the target should be fidelity at the represented resolution; the rule-based result would serve as a reference, not a theoretical reconstruction upper bound.
- Could you report a Mini Workflow comparison on Lightwheel, or a representative subset, ideally with comparable inference budgets? This would help establish whether the gains over the general-purpose agent extend to the external benchmark and clarify the quality-cost trade-off.
3 comments
Response to Reviewer QQcK [2/2]
W3 and Q2. Mini Workflow on Lightwheel
We ran Mini Workflow–Astra on the full Lightwheel benchmark, with the same image and mesh, the same model, the same low reasoning effort and the same cap of 80 turns as USDCraft–Astra. All objects completed.
| Method | Evaluation | F1 ↑ | Static gIoU ↑ | Static PC ↓ | mIoU ↑ | Artic. gIoU ↑ | Artic. PC ↓ | OC ↓ | AE° ↓ | LE ↓ |
|---|---|---|---|---|---|---|---|---|---|---|
| Mini Workflow–Astra | in the source frame | 18.8 | -0.347 | 0.258 | 0.088 | -0.349 | 0.294 | 0.084 | 17.0 | 0.193 |
| USDCraft–Astra | in the source frame | 87.0 | 0.590 | 0.050 | 0.683 | 0.555 | 0.088 | 0.011 | 10.2 | 0.009 |
| Mini Workflow–Astra | centered and scale-normalized | 70.1 | 0.235 | 0.129 | 0.477 | 0.223 | 0.182 | 0.015 | 12.2 | 0.023 |
| USDCraft–Astra | centered and scale-normalized | 87.0 | 0.580 | 0.049 | 0.674 | 0.546 | 0.087 | 0.011 | 10.1 | 0.010 |
The two evaluations answer different questions. In the source frame, each output is scored where it is placed, which tells whether the asset can directly replace the input mesh. With centering and scale normalization, prediction and reference are each centered and scaled to unit size before scoring, which forgives a global offset and measures only part structure and articulation.
Mini Workflow does not keep its output in the frame of the input mesh, so its assets need an extra alignment step before they can replace the source, and its F1 in the source frame is 18.8%. USDCraft builds the asset in the measured frame of the source and scores almost the same under both evaluations. With the offset forgiven, USDCraft still leads by 16.9 F1 points, so the gain over the general-purpose agent extends to the external benchmark. This normalized row is the one added to Table 2 and Section 4.2.
Quality and cost. Mini Workflow costs 0.38 USD per object and USDCraft 0.91 USD (Appendix D.11). USDCraft thus stays below one dollar per object, while it reconstructs the source mesh in place and with clearly better geometry. Overall, we believe USDCraft is the more practical choice when an asset has to match a given object.
We hope this addresses the reviewer's concerns, and we welcome further discussion.
We sincerely thank you for your constructive feedback. As the discussion period is closing soon, we would like to kindly ask whether our response has addressed your questions, and we are happy to discuss further.
Thank you for your time and consideration.
Response to Reviewer QQcK [1/2]
We thank the reviewer for the detailed and constructive review. The three points are addressed below and in the revised paper.
W1. Detection and summarization of protrusions, openings and interior structures
Because of the page limit, the submission kept this part brief. The revision states the criteria in Section 3.2 and lists the main criteria and thresholds in Appendix A.4 (Table 9). The detectors are deterministic and involve no learned model.
The thresholds are physical lengths, chosen once during development and shared by all objects and benchmarks. Each detection is summarized as one line of text with its position, size and depth, without a semantic label, and Appendix G.6 now shows this output for a scanned toaster.
Noisy or incomplete surfaces. The source mesh is never repaired or filled, and unsampled cells stay unknown. Reference depths are medians or percentiles, so burrs and seams do not shift them, and detections on small or sparse fragments are dropped. Cavity depths are reported as lower bounds. Since a missing detection does not establish a flat face, the agent can remeasure any region with a mesh query that bypasses the detectors.
W2 and Q1. Fidelity of the fixed encoding
We ran the requested experiment on the 100 USDCraft-bench objects (Appendix D.3). Every setting starts from the same fixed encoding and has no further access to the source mesh.
In (b) and (c), source renders, source mesh queries and source-based geometric feedback are disabled. The agent keeps renders of its own candidate, which involve no source information. The two settings share the model, reasoning effort, tools and the budget of 80 turns, and differ only in the image. Outputs are compared with the observed source surface in the source frame. The slice threshold is the pitch of the slice grid, the resolution that the encoding represents, and the fine threshold is the pitch of the 192-cell grid. CD is the symmetric Chamfer distance in percent of the longest source extent. As the reviewer suggests, the rule-based rows are references and not upper bounds.
The table separates the three effects the reviewer asks about.
The remaining gap to the complete system comes from mesh inspection and iterative geometric rechecking, which the fixed encoding cannot replace.