- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether coding agents can build physically valid, moving LEGO assemblies from functional and kinematic specifications. Because prior LEGO benchmarks only evaluate static shapes and raw LDraw files only store part coordinates without connectivity, the authors: (1) build a decompiler and grader that recover typed connections, rigid bodies, and joint ranges of motion across 1,464 official LEGO sets; (2) automatically derive the Brick Dance benchmark across four tracks (T1–T4), including required motion arcs and half-built model editing; and (3) show that under a fixed model (claude-opus-5-5) and budget (20 min / $2), providing connection and fit-checking tools (Brick Dance Copilot) boosts physical assembly validity from 47% to 89% and requirement satisfaction from 46% to 70%.
Critical strengths
- Compelling Core Insight and Novel Scope: The paper clearly demonstrates that "a renderer is not a checker"—due to 3D occlusion, builds with floating parts, unseated pins, or deep interpenetrations still render as plausible models (Figure 5). Introducing verifiable kinematic motion specifications and half-built assembly editing (Track T4) over the real LDraw library fills an important gap between static shape generation and mechanical engineering.
- Solid Decompiler and Automated Task Pipeline: Automatically extracting verifiable requirements and grading criteria from 1,464 official LEGO sets without human authoring is well-executed, and the controlled ablation cleanly isolates the value of physical connectivity feedback.
- Clarity and Scientific Honesty: The paper is very well-written with informative figures. The authors candidly report benchmark-side flaws in Section 4 and provide a complete, unedited 21-turn agent execution trace in Appendix B.
Critical weaknesses
- Incomplete Coverage of Joint and Motion Types: The current decompiler and grader primarily focus on single-axis revolute bearings (hinges/axles) swept one joint at a time against a static rest of the model. This does not cover all joint and mechanism types—such as prismatic/sliding joints, multi-DOF ball joints, or mechanisms requiring coupled multi-joint motion—and leaves ~16% of seatings as well as unmodeled/flexible parts (e.g., tyres, hoses) unverifiable.
- Small Evaluation Sample Size and Single Model Family: The agent evaluation is conducted on only 9 sampled sets (72 runs total) using a single model (
claude-opus-5-5). Testing additional frontier or open-weight models would better show how the tool-use gap generalizes across model families. - Minor Benchmark Artifacts Affecting Small-Sample Statistics: Out of only 9 evaluated sets, 1 set produced no submissions on T3 due to a missing-requirement generator bug, and 2 sets on T4 were marked invalid because they inherited pre-existing interpenetrations/floating parts from the official LEGO seed files (despite Section 3.1 stating that submissions are only penalized for new defects).
Questions for authors
- Incomplete Coverage of Joint and Motion Types: The current decompiler and grader primarily focus on single-axis revolute bearings (hinges/axles) swept one joint at a time against a static rest of the model. This does not cover all joint and mechanism types—such as prismatic/sliding joints, multi-DOF ball joints, or mechanisms requiring coupled multi-joint motion—and leaves ~16% of seatings as well as unmodeled/flexible parts (e.g., tyres, hoses) unverifiable.
- Small Evaluation Sample Size and Single Model Family: The agent evaluation is conducted on only 9 sampled sets (72 runs total) using a single model (
claude-opus-5-5). Testing additional frontier or open-weight models would better show how the tool-use gap generalizes across model families. - Minor Benchmark Artifacts Affecting Small-Sample Statistics: Out of only 9 evaluated sets, 1 set produced no submissions on T3 due to a missing-requirement generator bug, and 2 sets on T4 were marked invalid because they inherited pre-existing interpenetrations/floating parts from the official LEGO seed files (despite Section 3.1 stating that submissions are only penalized for new defects).
4 comments
Thank you to the authors for the additional experiments and revisions. The concerns about evaluation scale and model coverage (original W2) have largely been addressed. However, the core problem of the paper remains unresolved, and the revised manuscript exposes it more clearly: of the 20 official sets used in the experiments, the official models of 6 fail the authors' own base checks. The official models are references known to be physically buildable, and "no piece inside another" is the most basic physical constraint. Given that the joint and motion types supported by the paper are still fairly limited, a discrepancy of this magnitude between the base checks and the reference models significantly undermines the credibility of the benchmark and the paper's core contribution, especially for the large sets that the paper emphasizes.
This problem has the same root as W3 in the original review, which pointed out that the T4 sets were invalid because they inherited interpenetrations from the official files. The authors responded by removing the track concerned rather than by correcting the check itself. The original review listed this as a minor issue, but the numbers reported in the revision show that its scope is far larger than was judged at the time.
The authors are advised to resolve this problem first. Whether the cause is part geometry modeling (hollow parts drawn as solid), real overlaps in the transcribed files, or the collision test itself, each case should be identified, corrected and validated, and the evaluation then repeated on the original 20 sets. Excluding the failing sets is not an acceptable solution. This work should take priority over any further expansion of the experiments.
"All conditions are scored by the same grader" is not a sufficient answer to this problem. Agents in the tool condition can call a checker that also reports collisions (Section 3.4): with the checker, the average number of pieces inside another piece per build is 0.0, compared with 1.5 with no tools and 6.0 with the checker removed (Table 3). In other words, agents with tools can see and work around such flags, possibly including false positives, whereas agents without tools cannot, so the grader's error does not affect the two conditions symmetrically. In addition, repeated agent runs can only estimate the randomness of model outputs; they cannot remove systematic error in the grader itself. At present only Opus 5.5 has three repetitions, and every other model was run once per condition.
The current problem bears directly on the paper's most central claim of "physical validity" and cannot be set aside merely by stating it as a limitation in the text. On these grounds, the rating is lowered to 2. The rating will be reconsidered if the authors (1) identify the 6 failing sets and explain the cause for each, (2) manually audit the collision flags in the 160 builds of the main experiment and report the false-positive rate, and (3) after correction, report the Table 2 results again on the original 20 sets.
Response to the follow-up
Thank you for the follow-up.
An agent is never given the official file, and a build is not compared with it piece by piece; the official file is used to write the specification. So the six official files that fail the collision check do not by themselves make any build's grade wrong. What matters for our results is whether the check penalises builds wrongly, and we have now measured this (new Appendix F).
In four of the six sets the check is wrong: parts that nest in reality, such as a steering hub seated in a wheel, are drawn solid in our part library. In the other two the transcribed file has an error, such as a brick entered twice. We corrected the check by excusing every pair it flags in an official model (the same two parts at the same spacing). No build's verdict changes, so Table 2 is the same under the corrected check.
As the most lenient reading possible, we also switched the collision check off entirely. Tools still raise the share of valid builds for every model: Opus 5.5 from 50% to 75%, GPT-6 Astra from 35% to 70%, Sonnet 5.5 from 25% to 50%, Haiku 5.5 from 35% to 70%. The reason is that most invalid builds do not depend on this check: 72 of the 87 have a piece joined to nothing, which no official model has.
One result does depend on the check, and Section 5 now says so: with the check off, GPT-6 Astra's gap in mistakes narrows from 21.9 against 7.2 to 9.6 against 7.0.
I thank the authors for their additional response. Appendix F has partially alleviated my concerns that interpenetration misclassifications could affect the main experimental results. However, there remains room to further improve the reliability of the collision check and address the paper’s other limitations. Taking these considerations into account, I am adjusting my score from 2 back to 3.
Response to Reviewer e5Ha
Thank you for the careful review. We answer each point below.
W1 / Q1. Coverage of joint and motion types. We agree, and the revision now claims only what is measured: joints that turn. Two things changed. First, parts that cannot be joined rigidly, such as hoses and strings, are taken out of a set and out of its parts list before the task is written, and a set is not used at all if a wheel or tyre would have to be left out. Every piece that stays is joined through a connection the decompiler reads, so an agent is never given something the grader cannot judge. Second, we say plainly what is still not measured: sliding joints, ball joints, motions that must happen together, and gear trains. To give a sense of scale, the 20 tasks ask for 58 functions in all; 54 of them are turning joints and are measured, and 4 are gear trains, which stay in the task's text but are left out of the score. We did not have time to build the coverage table over the whole corpus, and will add it to the final version.
W2 / Q2. Nine sets, one model family. The revision has 20 sets of very different sizes, four models from two families, and three repetitions of the main comparison. The effect of tools holds for the second family: with tools, OpenAI's GPT-6 Astra hands in a valid build on 65% of the sets, against 20% without.
W3 / Q3. The T3 set with no submissions, and the T4 sets penalised for inherited defects. Thank you for catching this; it was an inconsistency with what Section 3.1 promised. Both problems came from the four-track design: one task was issued incomplete, and on the editing track a build that kept the given half-built model also kept that model's own flaws. The revision avoids both by studying one simpler setting. The agent is given only a written task and the parts, never the official model, and a build is graded only on its own pieces, so there is nothing to inherit. The four-track grid is therefore no longer in the paper, and we no longer claim editing as a contribution of this version. We still think editing is worth evaluating, and when we return to it the grader will subtract the given model's own defects, as Section 3.1 of the submitted version said it should.
On the trace. You noted the unedited trace in Appendix B. It belonged to the earlier experiments, so the revision replaces it with a step-by-step account of one session of the new ones (Appendix D); the complete traces of all 320 builds will be released with the code.
We are glad to answer anything further.