- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper introduces BrickBench, a benchmark for text-conditioned LEGO design by coding agents. It contains 300 prompts across three settings: Model, Set, and Alt-Build. The accompanying BrickAgent environment provides tools for part placement, connector-based construction, rendering, collision checking, and simulated stability evaluation. The benchmark evaluates physical validity, semantic alignment using dependency-aware VQA, and relative alignment and design quality using VLM as judge. The authors evaluate nine coding agents and two specialized generative baselines. Several agents achieve near-perfect validity with BrickAgent, while design quality remains more strongly differentiated. Removing BrickAgent sharply reduces validity without a comparable drop in visual quality. A human study validates the relative design ranking among agents, while a separate study shows that human and agent designs are readily distinguishable.
Critical strengths
-
The benchmark goes beyond feasibility-only evaluation. The paper makes a useful distinction between satisfying explicit construction constraints and producing a well-designed artifact. Several agents achieve very high validity, while their outputs remain clearly separated by design judgments. Evaluating both aspects is more informative than validity alone.
-
BrickAgent is a useful engineering contribution. The environment combines part placement, connector-aware construction, rendering, and interpretable validation in a practical interface for coding agents. The environment ablation also shows that these tools are important for producing valid assemblies.
Critical weaknesses
-
The additional diagnostic value of BrickBench relative to existing agentic design benchmarks could be better established. LEGO provides a compelling combination of discrete part selection, connector compatibility, physical constraints, and restricted inventories. However, the current experiments mainly establish system-level performance differences and the usefulness of the BrickAgent environment. It is less clear which of these LEGO-specific challenges are responsible for the observed differences, or what broader conclusions about agent capabilities can be drawn from them. The finding that agents handle explicit constraints more reliably than open-ended design quality is interesting, but similar questions can also be studied in graphics (BlenderGym), CAD (BenchCAD), and construction benchmarks. I would therefore like to see the paper more clearly articulate what additional diagnostic insight BrickBench provides beyond these existing settings.
-
The human comparison does not directly establish a gap in design quality. Section 5.4 asks raters to identify which assembly was made by a human, rather than which assembly is better designed. The human examples are also matched by part count but not by prompt. The result therefore shows that human and agent designs are distinguishable, but distinguishability is not the same as lower design quality. Section 5.3 validates the design ranking among agents, but human designs are never placed on the same design-quality scale. A more direct comparison would evaluate human and agent designs using the same pairwise design criterion under comparable tasks and constraints.
Questions for authors
-
What capability is BrickBench primarily intended to diagnose beyond existing graphics, CAD, and construction benchmarks? Do the authors see inventory constraints, connector reasoning, or the validity–design tradeoff as the key distinction, and which experiments most directly demonstrate this?
-
Do the authors have direct human preference judgments comparing human and agent designs using the same design-quality criterion as Design ELO?
3 comments
Thank you for the response and additional experiments. In the new human comparison, the authors compare the same subject and use human preferences, which provides a more convincing assessment. The authors also clarify the distinction between BrickBench and several existing works. The component ablation also makes the contribution of BrickAgent clearer. In short, the authors' response largely addresses my concerns. I appreciate the authors' efforts.
We thank the reviewer for their response. We will include the additional experiments and clarifications in the revision.
We thank the reviewer for their thoughtful comments and suggestions, which motivated the direct-preference study and a clearer statement of what BrickBench evaluates.
Diagnostic Value
The reviewer asks:
We thank the reviewer for the question. The capability we target is that of producing a good design while satisfying discrete physical assembly constraints. We therefore see the distinction between validity and design as key. Validity is a simulator-verifiable floor while design quality has an open ceiling (Sec. 7). The environment ablation demonstrates this most directly (Sec. 5.2) where, without BrickAgent, Astra's validity falls from 100% to 40%, and Luna's from 100% to a single valid build, yet alignment and design do not fail (Sec. 5.2, Tab. 2). Regarding connector use, we additionally ablate the two components of BrickAgent individually, and find that Luna's validity drops by 13 percentage points without connector placement, compared to 98 without validator feedback (see Component Ablation in our response to GFZh, Tab. R3). Regarding inventory constraints, we additionally ran Luna on the 100 Alt-Build prompts with the inventory restriction lifted, and found that its VQA rises from 0.80 to 0.93. Separately, we analyzed what agents leave out in Set, and find that they include fewer of the requested entities, while the relations between the entities they do include hold about as often as in Model. Both are reported under Set and Alt-Build in our response to Vr6V. Agents also differ in how often a repair of a failed verification leads to a build with fewer parts (see Failures and Repair in our response to GFZh). While the checks are available to all of the agents, validity ranges from 80--100% (Tab. 1), and, among the four agents that reach 100%, Design ELO spans 420 points. We will clarify this distinction in the introduction [C].
See also Open-Ended Design Under Physical Constraints in our comment to all reviewers.
Direct Preference Judgments
The reviewer asks:
Following the reviewer's suggestion, we collected direct preference judgments using the design question of Sec. 4.4. Raters prefer the human design in each row of Tab. R1, with official sets preferred in about 94% of comparisons and fan designs in about 76%. See Human Comparison in our comment to all reviewers. We will add the experiment to Sec. 5.4 as the basis for the gap to human design [E].