acceptodds
Under review as a conference paper at ICLR 2027

BrickBench: Evaluating Agentic Brick Design

Abstract

We propose BrickBench, a benchmark for agentic text-conditioned LEGO-set design. Given a prompt, an agent is tasked with producing an assembly that not only satisfies semantic and design criteria, but that also can be physically built. To do so, it must select parts from a discrete library and reason jointly about local and global constraints. We provide BrickAgent, an environment in which coding agents build, and we score validity, alignment, and design across three settings that vary scale and part availability. We find that leading agents largely satisfy the verifiable physical and semantic specifications, but fall substantially short of human design quality.

Primary area:
datasets and benchmarks
Keywords:
3D, brick, agent, design

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QAqNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper introduces BrickBench, a benchmark for text-conditioned LEGO design by coding agents. It contains 300 prompts across three settings: Model, Set, and Alt-Build. The accompanying BrickAgent environment provides tools for part placement, connector-based construction, rendering, collision checking, and simulated stability evaluation. The benchmark evaluates physical validity, semantic alignment using dependency-aware VQA, and relative alignment and design quality using VLM as judge. The authors evaluate nine coding agents and two specialized generative baselines. Several agents achieve near-perfect validity with BrickAgent, while design quality remains more strongly differentiated. Removing BrickAgent sharply reduces validity without a comparable drop in visual quality. A human study validates the relative design ranking among agents, while a separate study shows that human and agent designs are readily distinguishable.

Critical strengths
  1. The benchmark goes beyond feasibility-only evaluation. The paper makes a useful distinction between satisfying explicit construction constraints and producing a well-designed artifact. Several agents achieve very high validity, while their outputs remain clearly separated by design judgments. Evaluating both aspects is more informative than validity alone.

  2. BrickAgent is a useful engineering contribution. The environment combines part placement, connector-aware construction, rendering, and interpretable validation in a practical interface for coding agents. The environment ablation also shows that these tools are important for producing valid assemblies.

Critical weaknesses
  1. The additional diagnostic value of BrickBench relative to existing agentic design benchmarks could be better established. LEGO provides a compelling combination of discrete part selection, connector compatibility, physical constraints, and restricted inventories. However, the current experiments mainly establish system-level performance differences and the usefulness of the BrickAgent environment. It is less clear which of these LEGO-specific challenges are responsible for the observed differences, or what broader conclusions about agent capabilities can be drawn from them. The finding that agents handle explicit constraints more reliably than open-ended design quality is interesting, but similar questions can also be studied in graphics (BlenderGym), CAD (BenchCAD), and construction benchmarks. I would therefore like to see the paper more clearly articulate what additional diagnostic insight BrickBench provides beyond these existing settings.

  2. The human comparison does not directly establish a gap in design quality. Section 5.4 asks raters to identify which assembly was made by a human, rather than which assembly is better designed. The human examples are also matched by part count but not by prompt. The result therefore shows that human and agent designs are distinguishable, but distinguishability is not the same as lower design quality. Section 5.3 validates the design ranking among agents, but human designs are never placed on the same design-quality scale. A more direct comparison would evaluate human and agent designs using the same pairwise design criterion under comparable tasks and constraints.

Questions for authors
  1. What capability is BrickBench primarily intended to diagnose beyond existing graphics, CAD, and construction benchmarks? Do the authors see inventory constraints, connector reasoning, or the validity–design tradeoff as the key distinction, and which experiments most directly demonstrate this?

  2. Do the authors have direct human preference judgments comparing human and agent designs using the same design-quality criterion as Design ELO?

3 comments

We thank the reviewer for their thoughtful comments and suggestions, which motivated the direct-preference study and a clearer statement of what BrickBench evaluates.

Diagnostic Value

The reviewer asks:

Do the authors see inventory constraints, connector reasoning, or the validity–design tradeoff as the key distinction, and which experiments most directly demonstrate this?

We thank the reviewer for the question. The capability we target is that of producing a good design while satisfying discrete physical assembly constraints. We therefore see the distinction between validity and design as key. Validity is a simulator-verifiable floor while design quality has an open ceiling (Sec. 7). The environment ablation demonstrates this most directly (Sec. 5.2) where, without BrickAgent, Astra's validity falls from 100% to 40%, and Luna's from 100% to a single valid build, yet alignment and design do not fail (Sec. 5.2, Tab. 2). Regarding connector use, we additionally ablate the two components of BrickAgent individually, and find that Luna's validity drops by 13 percentage points without connector placement, compared to 98 without validator feedback (see Component Ablation in our response to GFZh, Tab. R3). Regarding inventory constraints, we additionally ran Luna on the 100 Alt-Build prompts with the inventory restriction lifted, and found that its VQA rises from 0.80 to 0.93. Separately, we analyzed what agents leave out in Set, and find that they include fewer of the requested entities, while the relations between the entities they do include hold about as often as in Model. Both are reported under Set and Alt-Build in our response to Vr6V. Agents also differ in how often a repair of a failed verification leads to a build with fewer parts (see Failures and Repair in our response to GFZh). While the checks are available to all of the agents, validity ranges from 80--100% (Tab. 1), and, among the four agents that reach 100%, Design ELO spans 420 points. We will clarify this distinction in the introduction [C].

See also Open-Ended Design Under Physical Constraints in our comment to all reviewers.

Direct Preference Judgments

The reviewer asks:

Do the authors have direct human preference judgments comparing human and agent designs using the same design-quality criterion as Design ELO?

Following the reviewer's suggestion, we collected direct preference judgments using the design question of Sec. 4.4. Raters prefer the human design in each row of Tab. R1, with official sets preferred in about 94% of comparisons and fan designs in about 76%. See Human Comparison in our comment to all reviewers. We will add the experiment to Sec. 5.4 as the basis for the gap to human design [E].

Thank you for the response and additional experiments. In the new human comparison, the authors compare the same subject and use human preferences, which provides a more convincing assessment. The authors also clarify the distinction between BrickBench and several existing works. The component ablation also makes the contribution of BrickAgent clearer. In short, the authors' response largely addresses my concerns. I appreciate the authors' efforts.

We thank the reviewer for their response. We will include the additional experiments and clarifications in the revision.

Reviewer_GFZhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BrickBench, a benchmark for text-conditioned LEGO assembly with 300 prompts across three settings: Model, Set, and Alt-Build. It also provides BrickAgent, an environment for constructing, inspecting, and validating assemblies. The evaluation separates physical validity under the proposed checks, semantic alignment, and design preferences. Experiments compare nine coding agents, with two task-specific baselines evaluated in the Model setting. The paper reports that several agents achieve high validity and semantic alignment, while design preferences distinguish their outputs. An environment ablation and two human studies examine the role of the tools, agreement with the design judge, and differences between agent-generated and human-designed assemblies.

Critical strengths
  1. An interesting task with complementary constraints. The combination of discrete part selection, spatial composition, and open-ended design makes LEGO assembly an interesting evaluation domain for coding agents. The three settings vary assembly scale and inventory availability, extending evaluation beyond small, single-object generation. Separating validity, alignment, and design also helps expose differences that a single semantic score would miss.
  2. A broad empirical comparison. The evaluation covers nine agents and two task-specific baselines, and reports results by setting alongside generation costs and intervals for the ELO estimates.
Critical weaknesses
  1. Limited scientific insights and insufficient attribution of BrickAgent's gains. The paper partly motivates BrickBench by the saturation of the BrickNet evaluation, yet four agents already achieve 100% Valid, and the best overall VQA reaches 0.954 on BrickBench. ELO distinguishes agents, but the rankings offer limited insight into the capabilities responsible for performance differences. In particular, the reported analyses do not systematically characterize failures in spatial reasoning, structural planning, or iterative error correction. Consequently, it remains unclear what broader insights BrickBench provides into agentic capabilities beyond measuring task-specific LEGO construction performance. The ablation in Sec. 5.2 removes the entire BrickAgent toolset, making it difficult to isolate the contributions of its individual components. In particular, agents have access to the same physical checks used in the final Valid evaluation. The observed improvements therefore cannot be clearly attributed to connector-based construction, explicit validation feedback, or their combination. These components should be ablated independently. Additionally, the description of BrickAgent lacks concrete API examples, validator-output specifications, and representative agent-tool interaction traces. These omissions make it difficult to assess which aspects of the environment contribute generalizable insights beyond the integration of existing tools and resources.
  2. Insufficient evidence for design-quality evaluation and the claimed gap to human design. The operational definition of design quality underlying Design ELO remains insufficiently specified. The judge is instructed to assess "how well the pieces are used," but the evaluation does not explicitly distinguish among different aspects of LEGO design, such as part utilization, structural organization, and visual appearance. The judge sees only four rendered views, leaving unclear how much the scores reveal about the quality of the underlying construction. Furthermore, Astra has the highest average part count in every setting and achieves the highest Design ELO (Tables 1 and 3), raising the possibility that visual complexity or part count may act as confounding factors. The human comparison in Sec. 5.4 does not control for task specifications. Human-designed assemblies are matched to agent-generated outputs by part count, without matching prompts. Participants are asked to identify the human-made assembly; the study does not collect direct ratings of relative design quality. Differences in subject matter, design objectives, or construction style may therefore influence the judgments independently of design quality. Distinguishing human-designed from agent-generated assemblies does not directly establish superior design quality under comparable task requirements.
Questions for authors
  1. What recurring failures in spatial reasoning, structural planning, and iterative error correction explain the differences between agents? What broader insights into agent capabilities can be drawn from these failures?
  2. Which aspects of agents' design capabilities does Design ELO actually measure, and what evidence supports this interpretation? Do the rankings remain consistent when all agents are required to use the same fixed number of parts within each setting?
  3. What are the independent contributions of connector-based placement and validator feedback to the observed validity gains? Please provide component-level ablations on a representative subset, together with an interaction trace showing API calls and validator responses.
  4. When human and agent designs are produced under the same prompts and part constraints, how do they compare in direct human judgments of design quality?
4 comments

We thank the reviewer for their thoughtful comments and suggestions, which motivated the environment-component ablation and the failure and part-count analyses.

Benchmark Contribution

See Open-Ended Design Under Physical Constraints and Adherence and Design in our comments to all reviewers.

Component Ablation

The reviewer asks:

What are the independent contributions of connector-based placement and validator feedback to the observed validity gains?

Following the reviewers suggestion of performing component-level ablations "on a representative subset," we ablated the two components (Sec. 4.3) individually with GPT-5.6 Luna on 100 prompts drawn across the three evaluation settings, using the same prompts, budget requirements, and remaining tools in each condition. To ablate connector placement, we remove the connector functions and their examples, and the agent places each part by absolute position and rotation through the same library. The validator is unchanged, and reports colliding parts and unstable components as before. To ablate the effect of validator feedback, we remove the agent's access to the validator, but keep connector placement, which on its own only rejects incompatible connector pairs and does not check for collisions, connectivity, or stability. We report the results in Tab. R3.

For Luna, validity drops by 98 percentage points without validator feedback, and by 13 without connector placement. We clarify that the validator is part of the environment by design, as a repository's tests might be for a coding agent. Access to it is not, however, sufficient for validity on its own, as the nine agents range from 80--100% validity with it (Tab. 1). Without BrickAgent (Sec. 5.2), Luna is valid on only one of these prompts. We will report this ablation in Sec. 5.2 and we thank the reviewer for their suggestion [E].

Table R3: Component Ablation. Build validity of GPT-5.6 Luna on 100 prompts

ConditionValid
BrickAgent100%
without connector placement87%
without validator feedback2%

BrickAgent Documentation

The reviewer requests:

an interaction trace showing API calls and validator responses.

We include an abridged excerpt from Luna's trace for model-018 (Model setting) below. In the excerpt, the validator reports a collision after Luna attaches a rock, which Luna resolves by sliding the rock along its connector:

arm.attach("10178", to=holder.pin[1], by=("socket", 0))             # rock
check(scene)
46 parts; 1 collision pairs; 1 connected components; stability not simulated until the collisions are resolved
collision: #38 4261 in arm_with_claw[2] at (-64, 46, -80) LDU; attached cross[0] to arm_with_claw[0].bar[4]
  <> #39 10178 in arm_with_claw[3] at (-88, 46, -80) LDU; attached socket[0] to arm_with_claw[2].pin[1]; bounds overlap 10 x 16.5 x 20 LDU
arm.attach("10178", to=holder.pin[1], by=("socket", 0), slide=-5)   # revised line
check(scene)
46 parts

We will add a link to the API reference and complete traces for all builds [C].

Failures and Repair

The reviewer asks:

What recurring failures in spatial reasoning, structural planning, and iterative error correction explain the differences between agents?

Following the reviewer's suggestion, we classified each of the failed validator checks in the nine agents' runs, and report the results in Tab. R4. Collisions are the most common failure during construction for every agent, at 78--96% of failed checks. The agents differ, however, in stability, with the check failing 5--6% of the time for Luna, Qwen, and GLM, and at most 1% of the time for Muse, Sol, Opus 5, and Gemini. Counted per run, an unstable component is reported in 18--32% of runs for the former and in at most 8% for the latter. We additionally analyze the agents' repairs, and find that the build ends with fewer parts in 24% of Astra's, compared to 31--55% for the other agents. Across agents, the VQA judge finds 89% of requested entities in builds with such a repair, and 92% in builds without one. We will expand upon this analysis in the updated paper [E].

Table R4: Failures and Repair. Collisions: share of failed checks that report one. Unstable: share of stability checks that report an unstable component. A repair runs from a failed check to the next passing one. In parentheses: the number of failed checks, stability checks, and repairs that each share is taken over (300 runs per agent).

AgentRuns with a failed checkCollisionsUnstableRuns with an unstable checkRepairs ending with fewer parts
GPT-6 Astra28%89% (303)2.9% (1,197)9%24% (140)
Claude Opus 531%92% (592)0.7% (4,964)8%47% (305)
Gemini 3.8 Flash13%96% (98)0.0% (11,870)1%36% (61)
Qwen 3.8 Flash50%79% (2,280)5.2% (7,477)28%47% (648)
GPT-5.6 Sol17%93% (166)0.8% (1,503)4%31% (78)
DeepSeek V4.1 Flash43%85% (1,511)3.9% (5,351)25%47% (408)
GPT-5.6 Luna68%79% (1,802)5.9% (5,771)32%42% (768)
Muse Spark 1.319%81% (366)1.0% (6,563)5%49% (107)
GLM 5.3 Flash38%78% (1,140)5.0% (4,206)18%55% (319)

Design ELO

The reviewer asks:

Which aspects of agents' design capabilities does Design ELO actually measure, and what evidence supports this interpretation?

We clarify that Design ELO measures overall design preference from the renders. The judge is not given the build prompt, instead being shown four views of each assembly and asked "which assembly is better designed by the standard of an official set, judged on how the pieces are used rather than the subject" (Sec. 4.4; full instructions in Fig. 9). It does not, however, account for physical validity which we report separately as Valid. Regarding evidence for our interpretation, we validated Design ELO against human ratings in Sec. 5.3. The rankings agree on 34 of 36 agent pairings, with a Kendall τ\tau of 0.89 (Fig. 5b). We will clarify this further in the text where Design ELO is introduced [C].

Part Count

The reviewer asks:

Do the rankings remain consistent when all agents are required to use the same fixed number of parts within each setting?

We thank the reviewer for their question. As formulated, part count is a design decision that the agent must make within the limits of each setting (Sec. 4.1), which fixing it would remove. To evaluate whether it drives the rankings, we add a part-count term to the Bradley--Terry model and refit it on all 20,906 design verdicts. We find that part count is associated with preferences differently in each of the settings, with a term of −0.25±0.07-0.25\pm0.07 in Model, +0.19±0.09+0.19\pm0.09 in Set, and +0.87±0.17+0.87\pm0.17 in Alt-Build (log-odds per e-fold of the part ratio and 95% CI). With the term included, Astra remains ranked first in every setting. The Set ranking is unchanged, and the Model and Alt-Build rankings agree with the unadjusted ones with a Kendall τ\tau of 0.89 and 0.78. We additionally refit on only the 2,2982{,}298 comparisons between builds that are within 10% of each others' part count. Astra remains ranked first in every setting, and the full rankings agree with the unadjusted one. Bootstrapping this subset across tasks, we find Astra first in 100% of Alt-Build resamples, 99% in Set, and 84% in Model. We will include both analyses in the updated paper [E].

Human Comparison

See Human Comparison in our comment to all reviewers (Tab. R1).

Thank you for the detailed response, which has addressed my previous questions. I have a remaining concern about the scoring cost, particularly the multimodal pairwise comparisons for ELO. Could the authors share the time, computational resources, or cost needed to fully evaluate a new system, to help clarify its practical affordability?

We thank the reviewer for their engagement.

Scoring Cost

The reviewer writes:

Could the authors share the time, computational resources, or cost needed to fully evaluate a new system, to help clarify its practical affordability?

Evaluating a new system requires rendering its 300 assemblies and making 11,100 calls to the judge, Gemma 4 31B (Sec. 4.4). Each assembly is compared with those of the nine evaluated agents on both questions and in both orders (10,800 calls), and receives one VQA call. In a pairwise comparison, the views of each assembly are tiled into one image and the judge outputs only its choice. The calls amount to about 9M tokens, which cost under $1 at current batch API prices. Run locally, they take under an hour on a single H100. Rendering also takes under an hour. We will add these costs to the paper [C].

We again thank the reviewer for their feedback, which has improved the paper.

Reviewer_Vr6VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces BrickBench, a benchmark for evaluating whether coding agents can turn a text description into a LEGO assembly that satisfies construction requirements and works as a visual design. Its 300 prompts span small models, larger scenes, and builds restricted to a fixed retail-set inventory. The accompanying BrickAgent environment lets agents place parts, inspect renders, and repair failures reported by a validator. The evaluation measures simulated validity, prompt alignment, and design preference separately. Experiments compare general agents with task-specific generators, and an ablation tests the effect of the environment. Human studies assess the automatic design ranking and whether people can distinguish generated assemblies from human designs. The authors conclude that current agents largely satisfy physical and semantic specifications but fall short of human design quality.

Critical strengths
  • BrickAgent makes it possible to evaluate the interaction between construction constraints and visual design in a working generation loop. It extends construction to larger scenes and fixed inventories, where agents can inspect assemblies, identify failed checks, and repair particular parts. The environment ablation is especially useful here: removing the tools can preserve or improve visual scores while severely reducing validity. This demonstrates a concrete failure of evaluating generated constructions through appearance alone and supports including executable checks in this task.

  • The design evaluation has meaningful human validation. People compare assemblies generated for the same prompt, and the resulting agent rankings closely agree with the automatic rankings. The design question also separates the use of pieces from coverage of the requested content, which helps avoid treating prompt adherence as a complete measure of design quality. Bradley–Terry fitting is a reasonable way to summarize these pairwise judgments. Together, these choices support using the metric to compare the evaluated agents and make the design evaluation more credible than an unvalidated VLM score.

Critical weaknesses
  • The paper needs a stronger account of what the new settings teach us about agentic design. It motivates larger assemblies through retail-set scale and Alt-Build through scarce inventory, both plausible extensions of the earlier small-object task. However, the analysis does not adequately connect those motivations to the behaviors observed. For example, most agents build near the minimum allowed size in the larger-scene setting. The paper reports this pattern without establishing whether the scenes omit requested structure, use overly simple constructions, or successfully express the brief with fewer pieces. These possibilities have different implications for the intended scale challenge. The fixed-inventory setting similarly receives little analysis of how scarcity changes design decisions or causes failures. Such analysis would help explain why these tasks are informative and what researchers should learn from performance on them. The scientific purpose of the expanded benchmark is therefore less developed than its implementation.

  • The conclusion that prompt adherence is saturated and design quality is the remaining bottleneck exceeds the evidence from the metrics. VQA is a bounded score from generated question graphs and one VLM. The paper supplies example graphs and a breakdown of question types, but no independent check of coverage or answer accuracy. A high average score therefore does not establish that important semantic errors have become rare, particularly if failures on a small subset of requirements materially change a scene. Bradley–Terry scores summarize relative preferences within the evaluated pool; their spread does not measure absolute design headroom. Comparing that spread with bounded VQA scores cannot establish which capability is more nearly solved. Although the human study supports the design ranking, it does not validate the semantic ceiling. The paper's proposed distinction between largely solved adherence and unsolved design quality consequently needs stronger support.

  • The human-versus-agent study measures source recognition, yet the paper interprets it as evidence of better design. A participant could recognize familiar human conventions while preferring the generated assembly. The references are matched by part count but not by subject, so subject matter and style can also influence the identification decision. Figure 6 obscures this distinction further: its axis says “Win Rate vs. Human” and marks chance as “no preference,” although the task does not ask for a preference. The separate human validation compares agents with one another and cannot establish the claimed gap to human design. A direct preference comparison under matched requirements would provide evidence for that claim.

  • The physical evaluation is too limited to establish the buildability claimed in the abstract and examples. Each connected component becomes a single rigid body, so hinges, axles, and other connections cannot move or fail within that component; high friction also suppresses sliding between components. An assembly can pass while a physical version sags or comes apart. The authors acknowledge this limitation, but still describe the examples as buildable and conclude that agents largely meet the task's physical requirements. Since physical feasibility is a main reason to distinguish this task from visual generation, the reliability of that conclusion matters. The ablation demonstrates compliance with the implemented checks, while their correspondence to actual construction remains untested. Even a limited examination of representative outputs for failures omitted by the simulator would help assess the gap and establish how broadly the physical claims can be made.

  • The baseline results establish a quality–cost trade-off that the discussion does not analyze sufficiently. General agents obtain much higher scores with an interactive environment and substantial opportunities to inspect and revise, while the trained generators produce assemblies at much lower inference cost. The paper reports this cost difference, but still characterizes task-specific models as “subsumed” without developing when the additional quality justifies the resources or what explains the gap. The strong effect of the environment in the ablation makes access to feedback especially relevant. Performance at comparable budgets, or evidence about how much refinement helps a trained generator, would clarify the relative benefits of these approaches. The current comparison leaves researchers with a ranking of complete systems but little guidance about the choice between specialized generation and general agents.

Questions for authors
  1. What capability or design limitation do the larger-scene and fixed-inventory settings reveal that is not evident from the earlier evaluation? Does the existing output analysis connect minimum-sized scenes or inventory restrictions to specific failures in satisfying the task?

  2. Were the question graphs and VLM answers checked for semantic coverage and correctness? What supports interpreting the near-ceiling VQA score as saturation?

  3. What evidence connects source identification to design preference in the human-versus-agent study? Please also clarify the preference terminology in Figure 6.

  4. Were any reported assemblies checked beyond the rigid-component simulation to support the claim that they can be physically built?

4 comments

We thank the reviewer for their thoughtful comments and suggestions, which motivated the analysis of the Set and Alt-Build settings, the adherence and physical evaluations, and the cost-matched comparison reported below.

Set and Alt-Build

The reviewer asks:

What capability or design limitation do the larger-scene and fixed-inventory settings reveal that is not evident from the earlier evaluation? Does the existing output analysis connect minimum-sized scenes or inventory restrictions to specific failures in satisfying the task?

To further investigate what each setting reveals, we analyzed the VQA judge's answers (Sec. 4.4) on all nine agents' builds. In the Set setting (Sec. 4.1), agents include fewer of the requested entities than in the Model setting (86% vs. 94%). When entities are present, however, relations are satisfied nearly as often as in Model (91% vs. 92%). The entities left out most often are small details and secondary objects, such as a character's mustache, a coin, or a dog bed. Omissions are also more frequent in the smallest builds, with the third of Set builds that are within 50 parts of the minimum including only 80% of the requested entities while larger builds contain 90%.

In the Alt-Build setting, which restricts the agent to the 223 lots contained in set 10698 (Sec. 4.1), VQA falls for eight of the nine agents relative to that in the Model setting, although the prompts are also relatively shorter (Tab. 1, Sec. 4.2). We do not find that agents omit entities because they run out of parts. Builds with an entity that is omitted use less of the inventory than complete builds. To measure the effect of the restriction, we additionally ran Luna on the 100 Alt-Build prompts under the rules of the Model setting, in which any part may be used (Sec. 4.1), with the same tools and budget. All 100 of the generated builds are valid, and they remain of similar size, with a median of 41 parts compared to 39. Without the restrictions, Luna's VQA rises from 0.80 to 0.93, satisfying 98% of the entity requirements instead of 85%, and its ELO rises from 874 to 1226 when rated against the nine agents' Alt-Build assemblies, where only Astra rates higher (Tab. 1). The observed gain is greatest on the 59 prompts for which Luna uses minifigure parts which the set does not contain (0.76 to 0.94 compared to 0.85 to 0.91 on the other 41). To separate the restriction on the kinds of parts from that on their number, we also ran Luna with only the parts of the set but without limits on their quantity. Its VQA rises to only 0.81 and its ELO to 962, with a median build of 49 parts. This suggests that agents are limited more by the kinds of parts in the set than by their number. We again thank the reviewer for suggesting the analysis, and we will include both experiments in Sec. 5.1 [E].

See also "Open-Ended Design Under Physical Constraints" in our aggregate comment.

Question Graphs and Adherence

The reviewer asks:

Were the question graphs and VLM answers checked for semantic coverage and correctness? What supports interpreting the near-ceiling VQA score as saturation?

Following the suggestion of the reviewer, we checked both of these by hand on 50 randomly drawn Astra builds (17 Model, 17 Set, and 16 Alt-Build). Regarding coverage, we clarify that the graphs are not intended to list every detail of a prompt. They cover its entities, attributes, and relations (Sec. 4.2) and leave out redundant defaults. We compared each build against its full prompt, independently of its graph, and found that 49 of the 50 builds fully satisfy it. The one exception we identified is a pose detail, where a kneeling ape's arms were spread toward a slab rather than stretched toward it, which is contained in the graph and which the judge also marked as unmet. Regarding correctness, the builds satisfy 878 of the 879 questions we reviewed, and the judge's answers agree with ours on 836 (95%). The judge errs on the side of strictness, as it is instructed to "not assume a detail is present because the brief asks for it" (Fig. 9), so its score is, if anything, a conservative measure of prompt adherence. We take this as support for our claim of saturation, which we intend for the best-performing agent (Sec. 6) [C,E].

Human Comparison

The reviewer writes:

Please also clarify the preference terminology in Figure 6.

We agree with the reviewer that the terminology should be clarified. Fig. 6 measures identification, and we will relabel its axis and chance line accordingly in the updated figure. See "Human Comparison" in our aggregate comment to all reviewers for a direct preference comparison (Tab. R1) [C].

Physical Validity

The reviewer asks:

Were any reported assemblies checked beyond the rigid-component simulation to support the claim that they can be physically built?

We thank the reviewer for the question. To evaluate assemblies beyond the simulation, we physically built 30 valid Alt-Build assemblies from the parts of set 10698, which were randomly drawn across the nine agents. We followed each generated design exactly. All 30 of them stand on their own, including those with hinges and axles, with no sagging or connections coming apart. In the updated text, we will describe assemblies as passing the collision and stability checks of Sec. 3.2 [C,E].

Cost-Matched Comparison

The reviewer writes:

Performance at comparable budgets, or evidence about how much refinement helps a trained generator, would clarify the relative benefits of these approaches.

The reviewer's suggestion is much appreciated. To perform an evaluation at a comparable budget, we tested BrickNet-14B with best-of-NN sampling. We filtered its NN samples with the validity checks applied in Tab. 1 and selected the valid sample closest to the prompt by SigLIP 2, an alignment metric of BrickNet's own evaluation (Sec. 3.3). Reported costs include sampling and selection and, as in Sec. 4.4, we score VQA and ELO regardless of validity, using the first sample that parses if none are valid. We report the results in Tab. R2. Validity rises from 26% at N=1N=1 (Tab. 1) to 100% at N=1,000N=1{,}000, but BrickNet's VQA remains below that of every agent. At a cost comparable to GLM 5.3 Flash, BrickNet reaches a VQA of 0.35 and an ELO of 816, compared to the 0.66 and 848 of GLM. We will add the results to the paper and reword "subsumed" in Sec. 6 and "displacing" in Sec. 7 accordingly [C,E].

Table R2: Cost-Matched Comparison. BrickNet-14B best-of-NN vs. GLM 5.3 Flash (both Model setting). As in Tab. 1, BrickNet's Valid does not include stability.

NNCost per buildValid*VQAELO
1$0.000626%0.18723
16$0.01168%0.19754
256$0.1898%0.28804
1,000$0.69100%0.35816
GLM 5.3 Flash (lowest-VQA agent)$0.6990%0.66848

Thank you for the detailed rebuttal. The new experiments address several of my specific concerns. I’m still not fully convinced by the human study, but this is less of a concern to me now.

Related to my first point, I’m still not sure what the benchmark as a whole is meant to measure. I’d like to hear the authors’ thoughts:

  • The paper says the task has a floor a simulator can check and an open ceiling on design. I’m still not sure what this benchmark tells us that other agentic coding or text-to-3D benchmarks don’t. Does doing well here say anything about the model beyond LEGO building?
  • The categories come from the BrickLink Designer Program, so they are themes rather than capabilities. It’s not clear to me what doing well on, say, “Pirate” or “Train” actually means. A breakdown by prompt properties (number of entities and relations, characters vs. objects, scale) might say more about where agents fail.

We thank the reviewer for their engagement.

Open-Ended Design Under Physical Constraints

The reviewer writes:

I'm still not sure what the benchmark as a whole is meant to measure.

The capability we target is that of producing a good design while satisfying discrete physical assembly constraints. We evaluate whether an assembly is physically valid, whether it depicts what the prompt asks for, and how well it is designed (Sec. 4.4).

They write also:

I'm still not sure what this benchmark tells us that other agentic coding or text-to-3D benchmarks don't. Does doing well here say anything about the model beyond LEGO building?

To our knowledge, BrickBench is the first benchmark to evaluate the design quality of agents' open-ended 3D design under physical constraints (see also Open-Ended Design Under Physical Constraints in our comment to all reviewers). While four agents reach 100% validity, Design ELO separates them by 420 points (Tab. 1). Regarding whether doing well says anything "beyond LEGO building," BrickBench requires an agent to design an assembly for an open-ended prompt and to build it from a fixed part library under connection and stability constraints. These requirements are not specific to LEGO. We will state in the introduction that BrickBench evaluates them in LEGO alone [C].

Categories

The reviewer writes:

The categories come from the BrickLink Designer Program, so they are themes rather than capabilities. It's not clear to me what doing well on, say, "Pirate" or "Train" actually means.

We agree with the reviewer that the categories are themes. They are intended to cover the range of subjects that LEGO sets depict, and the ranking of the agents within each theme agrees with the overall ranking (Kendall τ\tau of 0.78--1.00). We distinguish the categories from the three settings (Sec. 4.1), which vary the constraints on the build. We will clarify this in Sec. 4.2 [C].

They write also:

A breakdown by prompt properties (number of entities and relations, characters vs. objects, scale) might say more about where agents fail.

Following the reviewer's suggestion, we analyzed the VQA judge's answers by prompt property. We labeled each requested entity as a character (a person or creature) or an object. In the Model and Set settings, the judge finds requested characters as often as objects (96% vs. 95% and 88% vs. 87%). These rates are for whole entities, while those in Set and Alt-Build in our response (94% and 86%) also include the parts of entities, which the judge finds less often (93% and 80%). In the Alt-Build setting, however, it finds characters less often (72% vs. 92%), as the agent is restricted to the pieces of set 10698, which are mostly basic bricks and plates and include no minifigure parts. In the Set and Alt-Build settings, the judge also finds fewer of the requested entities for the prompts with the most relations than for those with the fewest (85% vs. 89% and 84% vs. 92%). In a regression that also includes the number of entities, the length of the prompt, the agent, and the setting, VQA decreases by 0.010 per relation (95% CI: 0.002 to 0.018), and we observe no association with the number of entities. When entities are present, however, relations are satisfied 91--95% of the time in every group. Regarding scale, see also Set and Alt-Build. We will add this analysis alongside Fig. 13 [E].

We again thank the reviewer for their feedback, which has improved the paper.

Comments on the paper

2 comments

We thank each of the reviewers for their constructive comments and suggestions, which we will use to improve the quality of our paper. They recognize that in going "beyond feasibility-only evaluation" (QAqN), our benchmark exposes "differences that a single semantic score would miss" (GFZh), demonstrating "a concrete failure" of evaluating "through appearance alone" (Vr6V). They find our task "interesting" (GFZh) and its combination of constraints "compelling" (QAqN), and our key distinction between validity and design a "useful" one (QAqN). They describe our environment ablation as "especially useful" (Vr6V), our human validation as "meaningful" (Vr6V), and our comparison as "broad" (GFZh). We address shared points below, respond to individual suggestions in part, and will update the paper accordingly, adding clarifications [C], discussions [D], and experiments [E] to the text.

Open-Ended Design Under Physical Constraints (QAqN, GFZh, Vr6V)

We thank the reviewers for their questions about how BrickBench relates to existing benchmarks in graphics, CAD, and construction. To our knowledge, BrickBench is the first benchmark to evaluate the design quality of agents' open-ended 3D design under physical constraints. We clarify that BlenderGym and BenchCAD are reference-based -- the result is scored against a ground-truth scene or artifact -- and leave no design to be evaluated. BuildArena does not prescribe a target, but it scores machines by their simulated behavior (Sec. 2) and includes no evaluation of design. In BrickBench, we give the agent a prompt that has no canonical instantiation and require the assembly to be physically valid while evaluating its design using a human-validated judge (Secs. 4.4 and 5.3). Both are needed as removing the BrickAgent environment sharply reduces physical validity even when alignment and design ELO remain largely intact (Sec. 1, Tab. 2). While four agents reach 100% validity, Design ELO separates them by 420 points (Tab. 1). In our new preference study (see Human Comparison below), raters in turn prefer human designs to Astra's builds of the same subjects (Tab. R1). We will clarify this in the introduction and expand upon the comparison in the related-work section [C,D].

Human Comparison (QAqN, GFZh, Vr6V)

We thank all three reviewers for their suggestion of a direct preference comparison between human and agent designs. We randomly drew 100 official LEGO sets from the LDraw Official Model Repository and 100 fan designs from BrickNet, all of which were built from parts available to Astra and with half having at most 400 parts and the other half having between 401 and 4,000 parts. For each, we prompted GPT-6 Astra, the top-performing agent, to build the same subject once from a one-sentence description of the model's subject and contents, such as "A tow truck in orange and red on black wheels, with a crane boom and hook." We generated the descriptions from each models' name and renders using Claude Opus 5.5 -- which neither builds nor performs the judging -- and we individually checked them. Astra's builds are part-count matched relative to the human models (90--110%) and all 200 of them pass the validator. We then asked raters from the same pool as in Fig. 6 (Sec. 5.4) to compare the two builds which we rendered identically and which were shown without their source or description. As our aim was to measure preference, we did not ask the raters to identify the human design, instead asking the same design question (Sec. 4.4) used in our validation study (Sec. 5.3). Each pair was judged by two raters, and they agree on 170 of the 200 (85%). We report all 200 pairs in Tab. R1.

We find that raters prefer the human design in every row, with official sets being preferred in about 94% of comparisons and fan designs in about 76%. The fan designs -- which are not products -- are preferred to Astra's builds as well in both size bands (82% and 69%). We agree that Fig. 6 (Sec. 5.4) reports identification rather than preferences, which we will clarify in the text. We clarify also that the comparison is to existing human designs, each of which Astra builds once from a description of the subject. We will add the experiment to Sec. 5.4 as the basis for the gap to human design, and describe the gap in these terms [C,E].

Table R1: Human Comparison. Astra vs. human designs; 400 judgments; 95% CI bootstrapped over reference designs

Human designsPartsHuman design preferred
Official sets≤400\le 40091% (85--96)
401--4,00096% (92--99)
Fan designs≤400\le 40082% (72--91)
401--4,00069% (57--80)

Adherence and Design (Vr6V, GFZh)

We appreciate the reviewers' questions about saturation. We clarify that, while leading agents largely satisfy the defined physical and semantic specifications -- which are largely verifiable -- they differ in design (Tab. 1). To verify adherence, we evaluated 50 of Astra's builds by hand and found that 49 of them fully satisfy the prompt, suggesting that the near-ceiling VQA performance reflects saturation (see also Question Graphs and Adherence in response to Vr6V). Design, however, is open-ended, and there is not a target to saturate. We will clarify this distinction in the introduction and discussion [C].

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.