acceptodds
Under review as a conference paper at ICLR 2027

Brick Dance: Choreographing LEGO Builds from Motion Specifications

Abstract

Large language models write working software, yet they cannot reliably build working machines. The hard part of mechanical engineering is rarely a single part: it is an assembly of many parts whose connections must produce the motion a specification demands. We study this gap in LEGO, since its parts are fixed, expert-designed assemblies are plentiful, and, as we show, a machine can check whether an assembly holds together and moves as asked. Prior work on LEGO treats a build as a static shape, examining only whether its bricks connect and stand. We treat it as a mechanism, whose parts must move as specified: a door that opens 140° and carries twenty-five pieces with it. Such requirements are much closer to real mechanical engineering. To evaluate them, we build a decompiler that recovers a LEGO model's connections, its rigid bodies, and how far each joint turns before it collides, from its raw geometry. It reads 1,464 human-built models, and from any of them it derives a functional requirement specification without human authoring, which is what Brick Dance grades against, to our knowledge the first LEGO benchmark to evaluate motion and editing. Finally, we propose Brick Dance Copilot, a coding agent that uses our decompiler as tools and feedback: holding the model and the budget fixed, it submits a valid assembly on 89% of runs against 47% for an unmodified coding agent, and the gap is almost entirely physical.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, coding agents, benchmark, mechanical engineering, LEGO, assembly, kinematics, tool use, verification, program synthesis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_e5HaOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether coding agents can build physically valid, moving LEGO assemblies from functional and kinematic specifications. Because prior LEGO benchmarks only evaluate static shapes and raw LDraw files only store part coordinates without connectivity, the authors: (1) build a decompiler and grader that recover typed connections, rigid bodies, and joint ranges of motion across 1,464 official LEGO sets; (2) automatically derive the Brick Dance benchmark across four tracks (T1–T4), including required motion arcs and half-built model editing; and (3) show that under a fixed model (claude-opus-5-5) and budget (20 min / $2), providing connection and fit-checking tools (Brick Dance Copilot) boosts physical assembly validity from 47% to 89% and requirement satisfaction from 46% to 70%.

Critical strengths
  1. Compelling Core Insight and Novel Scope: The paper clearly demonstrates that "a renderer is not a checker"—due to 3D occlusion, builds with floating parts, unseated pins, or deep interpenetrations still render as plausible models (Figure 5). Introducing verifiable kinematic motion specifications and half-built assembly editing (Track T4) over the real LDraw library fills an important gap between static shape generation and mechanical engineering.
  2. Solid Decompiler and Automated Task Pipeline: Automatically extracting verifiable requirements and grading criteria from 1,464 official LEGO sets without human authoring is well-executed, and the controlled ablation cleanly isolates the value of physical connectivity feedback.
  3. Clarity and Scientific Honesty: The paper is very well-written with informative figures. The authors candidly report benchmark-side flaws in Section 4 and provide a complete, unedited 21-turn agent execution trace in Appendix B.
Critical weaknesses
  1. Incomplete Coverage of Joint and Motion Types: The current decompiler and grader primarily focus on single-axis revolute bearings (hinges/axles) swept one joint at a time against a static rest of the model. This does not cover all joint and mechanism types—such as prismatic/sliding joints, multi-DOF ball joints, or mechanisms requiring coupled multi-joint motion—and leaves ~16% of seatings as well as unmodeled/flexible parts (e.g., tyres, hoses) unverifiable.
  2. Small Evaluation Sample Size and Single Model Family: The agent evaluation is conducted on only 9 sampled sets (72 runs total) using a single model (claude-opus-5-5). Testing additional frontier or open-weight models would better show how the tool-use gap generalizes across model families.
  3. Minor Benchmark Artifacts Affecting Small-Sample Statistics: Out of only 9 evaluated sets, 1 set produced no submissions on T3 due to a missing-requirement generator bug, and 2 sets on T4 were marked invalid because they inherited pre-existing interpenetrations/floating parts from the official LEGO seed files (despite Section 3.1 stating that submissions are only penalized for new defects).
Questions for authors
  1. Incomplete Coverage of Joint and Motion Types: The current decompiler and grader primarily focus on single-axis revolute bearings (hinges/axles) swept one joint at a time against a static rest of the model. This does not cover all joint and mechanism types—such as prismatic/sliding joints, multi-DOF ball joints, or mechanisms requiring coupled multi-joint motion—and leaves ~16% of seatings as well as unmodeled/flexible parts (e.g., tyres, hoses) unverifiable.
  2. Small Evaluation Sample Size and Single Model Family: The agent evaluation is conducted on only 9 sampled sets (72 runs total) using a single model (claude-opus-5-5). Testing additional frontier or open-weight models would better show how the tool-use gap generalizes across model families.
  3. Minor Benchmark Artifacts Affecting Small-Sample Statistics: Out of only 9 evaluated sets, 1 set produced no submissions on T3 due to a missing-requirement generator bug, and 2 sets on T4 were marked invalid because they inherited pre-existing interpenetrations/floating parts from the official LEGO seed files (despite Section 3.1 stating that submissions are only penalized for new defects).
4 comments
AuthorsOpenReview ↗

Response to Reviewer e5Ha

Thank you for the careful review. We answer each point below.

W1 / Q1. Coverage of joint and motion types. We agree, and the revision now claims only what is measured: joints that turn. Two things changed. First, parts that cannot be joined rigidly, such as hoses and strings, are taken out of a set and out of its parts list before the task is written, and a set is not used at all if a wheel or tyre would have to be left out. Every piece that stays is joined through a connection the decompiler reads, so an agent is never given something the grader cannot judge. Second, we say plainly what is still not measured: sliding joints, ball joints, motions that must happen together, and gear trains. To give a sense of scale, the 20 tasks ask for 58 functions in all; 54 of them are turning joints and are measured, and 4 are gear trains, which stay in the task's text but are left out of the score. We did not have time to build the coverage table over the whole corpus, and will add it to the final version.

W2 / Q2. Nine sets, one model family. The revision has 20 sets of very different sizes, four models from two families, and three repetitions of the main comparison. The effect of tools holds for the second family: with tools, OpenAI's GPT-6 Astra hands in a valid build on 65% of the sets, against 20% without.

W3 / Q3. The T3 set with no submissions, and the T4 sets penalised for inherited defects. Thank you for catching this; it was an inconsistency with what Section 3.1 promised. Both problems came from the four-track design: one task was issued incomplete, and on the editing track a build that kept the given half-built model also kept that model's own flaws. The revision avoids both by studying one simpler setting. The agent is given only a written task and the parts, never the official model, and a build is graded only on its own pieces, so there is nothing to inherit. The four-track grid is therefore no longer in the paper, and we no longer claim editing as a contribution of this version. We still think editing is worth evaluating, and when we return to it the grader will subtract the given model's own defects, as Section 3.1 of the submitted version said it should.

On the trace. You noted the unedited trace in Appendix B. It belonged to the earlier experiments, so the revision replaces it with a step-by-step account of one session of the new ones (Appendix D); the complete traces of all 320 builds will be released with the code.

We are glad to answer anything further.

Thank you to the authors for the additional experiments and revisions. The concerns about evaluation scale and model coverage (original W2) have largely been addressed. However, the core problem of the paper remains unresolved, and the revised manuscript exposes it more clearly: of the 20 official sets used in the experiments, the official models of 6 fail the authors' own base checks. The official models are references known to be physically buildable, and "no piece inside another" is the most basic physical constraint. Given that the joint and motion types supported by the paper are still fairly limited, a discrepancy of this magnitude between the base checks and the reference models significantly undermines the credibility of the benchmark and the paper's core contribution, especially for the large sets that the paper emphasizes.

This problem has the same root as W3 in the original review, which pointed out that the T4 sets were invalid because they inherited interpenetrations from the official files. The authors responded by removing the track concerned rather than by correcting the check itself. The original review listed this as a minor issue, but the numbers reported in the revision show that its scope is far larger than was judged at the time.

The authors are advised to resolve this problem first. Whether the cause is part geometry modeling (hollow parts drawn as solid), real overlaps in the transcribed files, or the collision test itself, each case should be identified, corrected and validated, and the evaluation then repeated on the original 20 sets. Excluding the failing sets is not an acceptable solution. This work should take priority over any further expansion of the experiments.

"All conditions are scored by the same grader" is not a sufficient answer to this problem. Agents in the tool condition can call a checker that also reports collisions (Section 3.4): with the checker, the average number of pieces inside another piece per build is 0.0, compared with 1.5 with no tools and 6.0 with the checker removed (Table 3). In other words, agents with tools can see and work around such flags, possibly including false positives, whereas agents without tools cannot, so the grader's error does not affect the two conditions symmetrically. In addition, repeated agent runs can only estimate the randomness of model outputs; they cannot remove systematic error in the grader itself. At present only Opus 5.5 has three repetitions, and every other model was run once per condition.

The current problem bears directly on the paper's most central claim of "physical validity" and cannot be set aside merely by stating it as a limitation in the text. On these grounds, the rating is lowered to 2. The rating will be reconsidered if the authors (1) identify the 6 failing sets and explain the cause for each, (2) manually audit the collision flags in the 160 builds of the main experiment and report the false-positive rate, and (3) after correction, report the Table 2 results again on the original 20 sets.

AuthorsOpenReview ↗

Response to the follow-up

Thank you for the follow-up.

An agent is never given the official file, and a build is not compared with it piece by piece; the official file is used to write the specification. So the six official files that fail the collision check do not by themselves make any build's grade wrong. What matters for our results is whether the check penalises builds wrongly, and we have now measured this (new Appendix F).

In four of the six sets the check is wrong: parts that nest in reality, such as a steering hub seated in a wheel, are drawn solid in our part library. In the other two the transcribed file has an error, such as a brick entered twice. We corrected the check by excusing every pair it flags in an official model (the same two parts at the same spacing). No build's verdict changes, so Table 2 is the same under the corrected check.

As the most lenient reading possible, we also switched the collision check off entirely. Tools still raise the share of valid builds for every model: Opus 5.5 from 50% to 75%, GPT-6 Astra from 35% to 70%, Sonnet 5.5 from 25% to 50%, Haiku 5.5 from 35% to 70%. The reason is that most invalid builds do not depend on this check: 72 of the 87 have a piece joined to nothing, which no official model has.

One result does depend on the check, and Section 5 now says so: with the check off, GPT-6 Astra's gap in mistakes narrows from 21.9 against 7.2 to 9.6 against 7.0.

I thank the authors for their additional response. Appendix F has partially alleviated my concerns that interpenetration misclassifications could affect the main experimental results. However, there remains room to further improve the reliability of the collision check and address the paper’s other limitations. Taking these considerations into account, I am adjusting my score from 2 back to 3.

Reviewer_kGmbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

the authors propose Brick Dance as a first LEGO benchmark that evaluates mechanical motion and editing. A decompiler is built to recover LEGO model's connections, rigid bodies, and kinematics for each body. They define four different evaluation tracks each with more information than previous. The benchmark is tested using Claude Opus across nine different LEGO sets.

Critical strengths
  1. evaluating functional motion of the assemblies using kinematic requirements is important as visual feedback alone does not guarantee the assembly's going to work as intended.

  2. the four tracks used for the benchmarking process allow examining how agents perform under different assembly generation and editing conditions.

Critical weaknesses
  1. a single model is evaluated on nine LEGO sets under four related tracks. there are no trials with repetitions or different sampling. the experiments also exclude assemblies not in the 80-400 part range. This matters as it limits evidence of reliability across design complexities.

  2. for Brick Dance Copilot, the source of improvement is not isolated and it ends up receiving a bunch of features - connector queries, placement testing, rendering etc. I can see the combined benefit but nothing pertaining to which components are responsible for improvements.

Questions for authors
  1. what portion of the validity improvement can be attributed to say placement testing, rendering etc? have you tried an ablation for this?

  2. how does the benchmark ensure that the generated joint satisfies the intended semantic function and not just matching the numerical motion thresholds?

  3. when you recover joint limits, do you compare them with manually validated versions?

2 comments
AuthorsOpenReview ↗

Response to Reviewer kGmb

Thank you for the review. Point by point, with pointers into the revised PDF:

W1. One model, nine sets, no repetitions, 80 to 400 parts only. All four are addressed: four models from two families (Table 2), three repetitions of the main comparison with the result of each trial (Table 6), and 20 sets drawn with a fixed seed from five size bands, under 50 to over 1,000 pieces (Section 3.2; results by size in Figure 7 and Table 7).

W2 / Q1. Which component is responsible? Table 3 removes one group of tools at a time (Opus 5.5, 20 sets). Valid builds: all tools 75%; without part lookup 75%; without rendering 75%; without the build tool 50%; without the checker 0%; no tools 45%. So part lookup and rendering account for none of the validity gain at this sample size, the build tool's 25 points are inside the trial-to-trial variation (11 to 15 of 20 valid across three trials of one condition), and the checker accounts for the gain. Placement by connections without the checker is worse than no tools, because it places pieces without testing for collisions. Figure 5 shows the same thing from the other side: what tools remove is loose and colliding pieces, not mistakes of design.

Q2. Semantic function or only a numeric threshold? In the new grader a function is not "some joint that turns far enough" (Section 3.3). A judge model, shown the build and its joints, names the joint that is the specification's function (the rear wing, the crane's claws), and the grader measures that joint. A joint that only carries the named thing counts for nothing, and a joint can answer one function at most. Figure 4 shows this on one build, and Appendix E gives the judge's instructions and each check exactly. What is still not checked is the direction of motion: a lever that rocks up and down where the specification says side to side currently passes. We state this as a limit.

Q3. Are recovered joint limits compared with manually validated ones? We did not address this in the time we had. Each limit in a specification comes from the range measured on the official model (80% of it), so the official model meets its own specification by construction, but we have not compared the measured ranges with the physical sets, and we list this as a limitation. We believe the paper's findings stand without it, since every agent is graded against the same limits.

On the four tracks. You listed the four tracks as a strength, so we want to be direct that this revision sets them aside: we chose to study one setting in depth, with more sets, models and repetitions, and we plan to return to the tracks, including editing, in later work.

We hope the ablation and the repetitions answer the main concerns, and we are glad to clarify further.

i thank the authors for their response. i have increased my score.

Reviewer_cBBvOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Brick Dance, a benchmark designed for generating and editing LEGO assemblies based on functional specifications. To this end, the authors create a decompiler which, by means of a corpus consisting of 1,464 sets, extracts the connections, the rigid bodies, and the ranges of joint motion from LDraw models and uses them to produce tasks. Experiments carried out on nine sets compare the same coding agent when using specialized tools and when not using them, showing an increase in assembly validity from 47% to 89%.

Critical strengths

1: The paper looks at a problem that goes beyond simple LEGO generation, since it involves checking whether an assembly allows for the required motion and whether an agent is able to carry out an existing build, thus offering a more rigorous assessment than mere visual similarity.

2: It is a considerable amount of engineering work to reconstruct the connection and motion data from the existing LDraw models, and by accepting raw LDraw submissions the benchmark becomes accessible to a variety of generation methods without the need for a specific internal representation.

3: The comparison retains the backbone, the time limit, and the spending cap. The traces that are reported together with the examples of failure help to explain how connection feedback can prevent incorrect placements which are difficult to identify from the rendered images.

Critical weaknesses

1: The paper's writing and organization need to be significantly improved. The figures (Figures 1 and 2) are cluttered, with small labels and insufficient visual emphasis on the comparisons they are meant to support. The writing is often repetitive and makes broad claims without enough supporting analysis, while key methodological details are scattered across the paper and appendix.

2: What do the sets "1773-1, 1775-1, ..." mean in Table 2?

3: The authors didn't analyze the performance of some existing open-source and commercial models on the proposed LEGO tasks.

4: The paper also doesn't analyze the effect of the proposed different modules. It's hard for me to verify the results after reading the main text.

Questions for authors

Please see the weaknesses and ensure the quality before the submission.

1 comment
AuthorsOpenReview ↗

Response to Reviewer cBBv

Thank you for the review. Point by point, with pointers into the revised PDF:

W1. Writing, organisation, Figures 1 and 2. We agree and have rewritten for it. The introduction now states the question, the benchmark and the findings in that order. Figure 1 is redrawn as one flow: an official set, the generated task and parts inventory, the coding agent with or without our tools, the submitted file, and the grade. Section 3 is reorganised into Data, Tasks, Grader and Models. Figures 2 and 3 are each cut to one row, and the decompiler's algorithm moves to Appendix B. A new Figure 4 walks through one task end to end: the official model, the specification, a build, and what the judge and the grader found, line by line.

W2. "1773-1, 1775-1, ..." in Table 2. These are official LEGO set numbers; each row is one set. The table that lists the sets now gives each set's name and piece count beside its number (Table 8), and the figures refer to sets by name.

W3. Other commercial and open-source models. Added (Table 2): four models from two families, each with our tools and with none, on 20 sets. With tools, valid builds rise from 45% to 75% for Opus 5.5 and from 20% to 65% for OpenAI's GPT-6 Astra; Sonnet 5.5 and Haiku 5.5 are clearly weaker, so the benchmark separates models. We have not yet run an open-weight model and say so in the limitations.

W4. The effect of the different modules; verifying results. Table 3 removes one group of tools at a time. The checker is what matters: no build is valid without it. Part lookup and rendering have no measurable effect, and placement by connections without the checker is harmful, because parts are placed without a collision test. Figure 5 adds where the mistakes come from: tools remove loose and colliding pieces and leave mistakes of design unchanged. For verification, the results are in one table in the main text, every build's score is listed set by set (Table 8), every build is pictured (Figures 8 to 12), one agent session is followed step by step (Appendix D), and each build is archived with its model file, checks, the judge's written findings, grade and full traces, to be released with the code.

We hope the new experiments and the rewriting address the concerns about soundness and presentation, and we are glad to answer anything further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.