acceptodds
Under review as a conference paper at ICLR 2027

LMBuild: Evaluating LLM Agents for Generating Buildable and Functional Structures

17 views

Abstract

Large Language Model (LLM)-based agents are increasingly capable of generating complex 3D structures, with the potential to reshape how objects are designed and realized in the physical world. Yet, producing elegant geometry is fundamentally different from producing objects that can actually be built and perform their intended functions. Existing evaluations largely focus on geometric or visual quality while overlooking structural soundness, functional affordances, and physical realizability. We introduce LMBuild, a benchmark for evaluating LLM agents on generating buildable and functional structures. LMBuild represents generated objects as assembled structures comprising meaningful part decompositions, joints, materials, and assembly sequences. To support reproducible evaluation, we provide a unified framework consisting of: (1) an interactive environment in which agents can use tools to retrieve, create, and place components to construct an object; (2) a curated benchmark that repurposes established CAD datasets and augments them with real-world object knowledge from Wikipedia; and (3) a four-level evaluation framework covering structural soundness, functional affordance, design quality, and physical realization. Evaluations across 30 domain-specific models, open-source LLMs, and frontier APIs reveal several intriguing findings: (a) Structural soundness and visual alignment are no longer the primary bottlenecks for frontier closed-source models, while functional affordance and physical operability remain substantially more challenging; (b) stronger models more effectively exploit part creation to construct structure-specific components, whereas weaker models tend to rely on retrieval and can struggle with the additional flexibility introduced by creation; and (c) providing explicit functional specifications substantially improves functional-part completeness, kinematic correctness, and overall physical operability. These results show that generating real-world structures requires capabilities beyond geometric synthesis, including deeper reasoning about functional affordances, mechanics, and physical constraints, as well as the ability to design and create novel components when existing ones are insufficient. We expect LMBuild to provide a foundation for measuring progress and incentivizing research toward agents that generate buildable and functional structures, while helping narrow the gap between open-source communities and frontier labs.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Structure Generation, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DJLbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LMBuild aims to evaluate agents that construct structured 3D objects, rather than merely generate visually plausible geometry. Its environment allows agents to retrieve or create parts, position them, specify joints and materials, and define an assembly sequence. The benchmark combines existing geometry datasets with evidence-grounded knowledge about real-world objects and defines twelve metrics covering structural soundness, functional affordance, design quality, and physical realization. Experiments evaluate 30 system configurations on a core set of 50 objects. The reported results show that systems generally perform better on structural and visual criteria than on kinematic correctness and operability. Providing explicit functional specifications improves functional-part completeness and operability, while iterative revision yields relatively modest gains.

Critical strengths

1.Benchmark provides a useful evaluation target beyond visual appearance, by considering part structure, joints, materials, and assembly order, it captures requirements that visual similarity alone cannot assess. The 50-object core set covers several construction regimes. Experiments on part retrieval versus creation and on explicit task specifications make the benchmark useful both for comparing system performance and for diagnosing model weaknesses.

2.LMBuild documents the tool protocols, metric definitions, and evidence-grounded knowledge extraction procedure in detail, and clearly states its limitations. The design-quality evaluator is calibrated using generated designs for eight objects and tested on a disjoint set of seven objects, with ratings provided by four annotators. Reporting item-level correlations alongside system-level agreement helps assess how well the evaluator reflects human judgments.

Critical weaknesses

1.Table 5 validates the Design Quality metrics but does not establish the validity of the Affordance or Realization metrics. R.3 combines role matching, graph relationships, human-use heuristics, and selected rigid-body motion tests. The presence and connection of components identified as an oven’s heating subsystem or a fan’s airflow subsystem therefore do not demonstrate that these functions have actually been realized. Simulation with uniform density and convex-hull collision geometry is a reasonable standardized proxy, but convexification can remove cavities that are important for motion. The effect of this approximation on the evaluation conclusions has not been quantified. Low R.3 scores can therefore reflect both generator failures and errors introduced by evaluator approximations, with their respective contributions remaining unclear.

2.A.1 metric evaluates outputs without semantic roles using only whole-object shape, but includes an additional component-level term when sufficiently informative roles are available. The same metric can therefore measure different quantities across systems. To illustrate the stated definition, a whole-shape F-score of 0.8 and a mean component F-score of 0.2 yield 56/100 when the component term is included, compared with 80/100 when it is omitted. In addition, closed-source agents run through provider CLIs, whereas open-source agents use a custom serialized tool-call protocol and associated termination rules. Section 3.5 itself attributes many part-creation failures to tool-specific conventions. These differences are relevant when evaluating deployed systems, but they weaken the attribution of performance gaps to intrinsic construction or reasoning ability.

Questions for authors

1.Provide an independent validation of the A/R scores using expert mechanical assessments or known-valid assemblies and deliberately modified invalid variants? Also, the main conclusions remain robust to changes in collision representations and geometric tolerances?

2.How do the rankings change when all eligible systems are compared using only the whole-object shape term of A.1? Is there evidence that the performance gaps persist when systems use the same tool-calling interface?

3.For Figure 4, specify the models, objects, and paired observations included in the analysis. The caption states that confidence intervals are obtained by bootstrapping over models and that Wilcoxon signed-rank tests are performed over episodes, whereas A.5 describes resampling over objects. Clarify the actual resampling unit and how dependencies among observations involving the same object are handled. Functional prompting improves A.2 and R.3 by 10.3 and 12.0 points, respectively, but also reduces R.1 by 9.0 points. Does this trade-off in assembly performance also appear in the per-object results? The explicit-specification experiment is reasonable in itself; its gains should be interpreted as better specification following, rather than improved reasoning under unchanged information.

5 comments
AuthorsOpenReview ↗

Response to Reviewer DJLb

We sincerely thank the reviewer for the careful assessment and constructive suggestions, and for recognizing the value of evaluating generated structures beyond visual appearance and the diagnostic insights provided by LMBuild. We appreciate the concerns regarding the validity and comparability of the Affordance and Realization metrics, the effect of different execution interfaces, and the statistical analysis of Figure 4. To address them, we validated the Affordance and Realization metrics with controlled errors, tested their sensitivity to the evaluator's settings, clarified what R.3 measures, refined A.1 so that one rule applies to every system, studied the effect of the execution interface, and redid the statistical analysis of Figure 4. In addition, the main evaluation in the revised paper now covers 200 objects, up from 50 in the submission. We address each concern below.

Validation of the Affordance and Realization Metrics (W1, Q1)

We understand the reviewer's request for a validation showing that the Affordance and Realization metrics respond correctly to actual designs. To provide this validation, we conducted an additional experiment in which we take known designs, break each of them in one controlled way at a time, and re-score every variant with the unchanged evaluator. The base designs that we break are (i) the 50 reference assemblies, which are valid by construction, and (ii) the designs generated by six models (GPT-5.6 Sol, Claude Haiku 4.5, Qwen3.5-27B, Qwen3.5-35B-A3B, GPT-OSS-120B, Gemma-4-31B). No design was selected or filtered: of the 350 base designs, 340 are non-empty and all of them are used. Each of the 10 perturbations is applied to every design that has the property it breaks. For example, material and relabelling perturbations apply to all 340 designs, sequence reversal to the 314 that declare an assembly sequence, and joint perturbations to the 271 that contain a moving joint. This gives 3,020 broken variants, each compared with its own unperturbed copy. The tables below list, for each metric, the errors we injected and the resulting change in score.

A.1 Geometry

Injected errorReference designs: ΔModel designs: Δ
Part roles permuted, geometry unchanged−27.3−7.0

A.2 Parts

Injected errorReference designs: ΔModel designs: Δ
Moving subassemblies deleted−29.5−16.0
Moving subassemblies pulled 15% away−16.2−6.3

A.3 Kinematics

Injected errorReference designs: ΔModel designs: Δ
Moving joints made fixed−61.7−27.4
Joint axes turned by 90°−57.3−13.9
Joint pivots moved by 25% of the diagonal−56.9−24.2
Moving subassemblies deleted−60.4−27.2

R.1 Sequence

Injected errorReference designs: ΔModel designs: Δ
Assembly order reversed−59.1−51.2
All parts pushed 35% apart−36.1−40.2

R.2 Material

Injected errorReference designs: ΔModel designs: Δ
Implausible materials assigned−46.3−75.0

R.3 Operability

Injected errorReference designs: ΔModel designs: Δ
Moving joints made fixed−10.8−9.7
Moving subassemblies deleted−20.9−14.8

Control (all twelve metrics)

Change that should not matterReference designs: mean |Δ|Model designs: mean |Δ|
Ids renamed, parts reordered, object translated≤ 0.5≤ 0.6

We observe three key results from these tables: (1) Every metric detects the errors it is designed for, not only on the valid reference assemblies but also on generated designs. (2) The metrics are specific. An error leaves the unrelated metrics unchanged: for example, changing the materials or the assembly order does not move the S, A or R.3 scores. (3) The metrics do not react to irrelevant changes. Renaming, reordering and translating a design moves no metric by more than 0.6 points on average.

With this added experiment, we hope to assure the reviewer that the A and R metrics are validated and work as intended on both reference and generated designs. This validation is reported in Appendix A.6.4 of the revised paper, with the full results in Table 13.

AuthorsOpenReview ↗

Response to Reviewer DJLb

What R.3 Measures, and Why Its Composite Design Is Appropriate (W1)

We agree with the reviewer on the scope of R.3: the presence and connection of, for example, an oven's heating subsystem do not prove that the oven heats. Below we explain why R.3 is designed this way and why its relation to the other metrics is intended.

R.3 tests standardized preconditions for function. Verifying heating or airflow would require thermal, fluid and electrical simulation specific to each object, which is not feasible in a standardized benchmark over 200 heterogeneous objects. R.3 instead tests standardized preconditions of each capability: the required components exist, they are connected into the functional chain, a person can reach and operate them, and the required motions can be actuated in simulation without collision or toppling. A design that passes is not thereby certified, and a design may occasionally realize a function through a mechanism the reference description does not anticipate. Our conclusion uses R.3 only as evidence of what is missing: operability is low for every system (the best R.3 is 22.1), and these low scores persist under the collision and tolerance settings we tested (point 3). The definition of R.3 and the limitations now state this scope explicitly.

Why R.3 is a composite. A capability is a conjunction: a fan provides airflow only if the blades exist, are mounted on the driven shaft, can rotate freely, and the fan stands. Each element calls for its own kind of evidence (simulation for motion, geometry for human use, presence and connection for non-mechanical subsystems), and R.3 scores a capability by its weakest element.

Why the overlap with A.2 and A.3 is intended. The levels of LMBuild are cumulative. A.2 and A.3 ask whether each ingredient is correct on its own (the parts are present; each joint matches the reference in type, axis and pivot), and R.3 asks whether the ingredients work together under physics. The validation above shows that they are related but not redundant: turning a joint axis by 90° lowers A.3 by 57 points and R.3 by only 1.4, because the motion can still be actuated; fixing the joints lowers both; changing materials or the assembly order lowers neither. This is why the paper reports the twelve metrics separately. R.3 identifies components through the roles an agent declares, since roles are part of an assembly specification; a variant in which the evaluator alone identifies them preserves the ranking of the 19 LLM agents exactly (Kendall τ = 1.000 on all 200 objects).

Do low R.3 scores come from the generators or from the evaluator's approximations? The two can be separated: R.3 moves when the designs change and does not move when the approximations change.

  • Changing the evaluator: a convex decomposition instead of convex hulls moves no system's R.3 by more than 0.5 points.
  • Changing the designs: deleting the moving subassemblies lowers R.3 by 14.8 points.
  • No evaluated capability succeeds on 30–40% of the tasks for five closed models and on 51% for Claude Haiku 4.5, and missing components are a frequent failure mode (Table 8).

Low R.3 scores therefore reflect what the generated designs lack. An assessment by mechanical experts is a natural next step when current R.3 could be reached, and is listed as future work.

In the revised paper, the definition of R.3 in Appendix A.4 (lines 3177–3178) states this scope, and the paragraph "A.3 and R.3 are complementary" in Appendix A.6.4 explains how the two metrics differ.

AuthorsOpenReview ↗

Response to Reviewer DJLb

3. Robustness to the Collision Representation and Geometric Tolerances (W1, Q1)

The reviewer asks whether our conclusions hold when the collision representation and the geometric tolerances change. They do. We changed each setting separately and re-scored the designs of all 30 systems on 50 core objects. For each setting, the table reports how much any system's score changes, and how well the ranking of the 19 LLM agents agrees with the default ranking. Agreement is measured by Kendall's τ, the rank correlation between the two orderings: τ = 1 means the order of the agents is identical, and values close to 1 mean that only a few neighbouring agents swap places.

Setting changedMetricLargest score change of any systemRanking agreement with default (τ)
Collision shapes that keep cavities, instead of one convex hull per partR.30.51.00
Distance tolerances halvedA.1 / A.2 / R.319 / 10 / 10.97 / 0.97 / 0.98
Distance tolerances doubledA.1 / A.2 / R.318 / 9 / 10.95 / 0.97 / 0.99
Insertion clearance for assembly, halvedR.10.71.00
Insertion clearance for assembly, doubledR.10.80.99

Convex hulls and cavities. The reviewer is right that a convex hull fills the cavities of a part. We therefore replaced each hull with a convex decomposition, which splits a concave part into several convex pieces that follow its actual shape, so cavities are preserved (computed with the CoACD library). R.3 is essentially unchanged: no system moves by more than 0.5 points, and the ranking is identical. The reason is that the simulator already ignores contact between parts that overlap in the designed pose, so a filled cavity cannot block the part inside it.

Tolerances. A.1 is an F-score at a distance threshold, so its absolute level moves with the threshold, as expected. The comparison between systems does not: the ranking of the agents is nearly the same under every setting (τ between 0.95 and 1.00), closed models remain ahead of open models, and every agent still scores lower on Affordance and Realization than on Soundness. R.1 and R.3 barely change at all.

In short, these choices change the level of some scores but not the comparison between systems, and the low R.3 scores are not caused by the collision model. This analysis is reported in Appendix A.6.5 of the revised paper, with the full results in Table 14.

Reviewer_hLTGOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes LMBuild, a benchmark for evaluating whether LLM agents can generate buildable and functional 3D structures. Agents retrieve or create parts, assemble them, and specify joints, materials, and assembly sequences. The authors evaluate 30 systems on 50 objects using twelve metrics. The results show that current models perform relatively well on structural and visual criteria, but still struggle with functional completeness, kinematics, and operability.

Critical strengths

1: This paper studies an important problem. A generated object can look reasonable but still fail to assemble or function as intended. Evaluating these aspects together helps clarify what current 3D generation agents can actually achieve.

2: The environment and evaluation framework are a substantial effort. Besides geometry, they consider parts, joints, materials, and assembly sequences. This makes the benchmark useful for identifying specific failures that visual evaluation alone would miss.

3: The experiments provide several interesting observations. In particular, weaker models struggle with the additional freedom of part creation, while explicit functional descriptions improve completeness and operability. The analysis of tool-use and scene-state errors also helps explain some of these results.

Critical weaknesses

1: The model comparison is partly affected by the execution setup. Closed-source models use command-line agents, while open-source models use a different tool-calling loop. Since many reported failures concern tool use and state tracking, it is unclear how much of the gap comes from the models versus their interfaces. How do the harness or different modes (like the goal command generally used in current coding agents' harness) affect the results?

2: The evaluation is relatively small, with 50 objects and generally one generation per model-task pair. I understand the cost of running these agents, but a few repeated runs on representative models would help determine whether the observed differences are consistent.

Overall, I value the authors' effort in building an evaluation framework that goes beyond the appearance of generated 3D objects. The experiments reveal useful limitations of current agents, and I think the benchmark can support further work in this direction. While the physical metrics and some experimental conclusions would benefit from additional validation, I lean toward weak acceptance.

Questions for authors

Please refer to the weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer hLTG

We are sincerely grateful to the reviewer for the supportive and careful review. We are especially encouraged that the reviewer recognizes what we care most about in this work: that an object which looks reasonable can still fail to assemble or to function, that an environment and evaluation framework covering parts, joints, materials and assembly sequences exposes failures that visual evaluation alone would miss, and that the experiments offer useful observations about current agents. This is the role we hope LMBuild can play for the community: a shared, reproducible testbed that shows where 3D generation agents stand on buildability and function. Both of the reviewer's suggestions were very helpful, and we followed each with new experiments that we believe make the paper considerably stronger. First, we ran three controlled studies of the execution environment, a comparison with native CAD programming in Codex and Claude Code, tool ablations, and a swap of the agent harness. Second, the main evaluation now covers 200 objects, up from 50 in the submission, together with a repeated-run analysis.

A unified execution environment for a fair comparison of all agents (W1)

We thank the reviewer for this important question: since many of the failures we report concern tool use and state tracking, the model must be separated from its execution environment. Every LLM agent receives the same task text and image, the same 19 tools and documentation, the same budgets and the same feedback. The new experiments below examine this environment from three directions:

  • The strongest agent tested scores about the same with and without the LMBuild tools. With free native CAD code, GPT-5.6 Sol scores 58.9 against 61.5 with the LMBuild tools (−2.6, 95% CI [−5.8, +1.3]), and no model scores higher without the tools.
  • The environment is robust to its individual tools. Removing any one tool family changes the Overall score by at most 3.2 points, so the conclusions do not depend on a particular tool.
  • Each model is run with the harness built for it. Moving a closed model into the open models' loop lowers its score, so the harness of the open models is not what holds them back.

In Appendix A.6.2 of the revised paper, the paragraph "Each model works best in its own harness" reports the harness comparison, Table 10 presents the tool ablations, and Table 11 compares LMBuild with native CAD; Section 3.5 (line 485) now states that using the interface is part of the evaluated task.

AuthorsOpenReview ↗

Response to Reviewer hLTG

The environment does not degrade the capabilities of state-of-the-art agents (W1)

We first replaced the whole environment with the agent's native way of working. In the native-CAD condition the agent receives the same task text, image, catalogue meshes, material vocabulary and required declarations, but no LMBuild tools. It works in a sandboxed shell with a Python CAD stack (build123d, CadQuery, trimesh, manifold3d; pip allowed) and delivers a standard URDF plus an assembly file. GPT-5.6 Sol runs in Codex, Claude Haiku 4.5 in Claude Code, and the open models in their usual loop with shell and file tools, under a matched budget. Every setting is paired, per object, with a fresh re-run of the full LMBuild environment and scored by the unchanged evaluator.

ModelLMBuild toolsNative CADΔ Overall [95% CI]ΔSΔAΔDΔRValid deliverable
GPT-5.6 Sol61.558.9−2.6 [−5.8, +1.3]−4.4+2.3+0.9−9.298%
Claude Haiku 4.553.040.8−12.3 [−16.2, −8.1]−18.9−5.4−13.1−11.7100%
Qwen3.5-27B53.15.3−47.8 [−52.2, −43.0]−70.4−24.8−45.2−50.812%
Qwen3.5-35B-A3B45.122.7−22.4 [−28.7, −16.0]−36.4−8.5−25.3−19.552%
GPT-OSS-120B49.216.0−33.2 [−39.1, −27.0]−52.4−12.6−30.4−37.334%
Gemma-4-31B37.825.6−12.2 [−20.4, −3.7]−19.1−3.8−14.6−11.552%
Mean of six models−21.7 [−34.6, −9.9], p = 9e-4

No model scores higher with native CAD than with the LMBuild tools. For the strongest coding agent tested, the two interfaces are close: GPT-5.6 Sol reaches the same Affordance and Design scores with free CAD code (+2.3 and +0.9), so the tools do not cap what it can model, while it loses on Soundness (−4.4) and Realization (−9.2), mainly on the executability of the assembly sequence, which is where the environment's geometry feedback helps. Claude Haiku 4.5 loses 12 points and the open models lose 12 to 48 points, mostly because they fail to produce a valid deliverable (12–52% valid). A shared tool interface thus narrows the gap between closed and open models; it does not create it.

This comparison is reported in Table 11 of Appendix A.6.2 in the revised paper.

Controlled tool ablations (W1)

We then removed one tool family at a time for the same six models and 50 objects. Removing geometry checking takes away check, measure and the contact and collision reports returned by each placement. Removing visual feedback takes away render. Removing structured placement takes away attach, so parts are placed by coordinates only. Δ is the paired difference in the Overall score against the full environment, averaged over the six models.

SettingΔ Overall95% CIp
− geometry checking−3.2[−6.3, −0.2]0.019
− visual feedback+0.0[−3.0, +2.5]0.98
− structured placement+1.7[−1.3, +3.8]0.24
− all three−4.0[−7.0, −1.3]0.004
+ physics simulation scoring tool+0.6[−2.0, +2.8]0.64
Native CAD (no LMBuild tools)−21.7[−34.6, −9.9]9e-4

Geometry checking is the component that matters. Removing it lowers connectivity (S.1 −6.1), collision avoidance (S.2 −8.2) and the executability of the assembly sequence (R.1 −12.2). Visual feedback and structured placement have no measurable effect when removed alone, because agents compensate with the remaining tools, for example by placing parts by coordinates and verifying with check. The environment is therefore robust: it does not depend on any single tool, removing all three costs 4 points, and removing the interface altogether costs 22.

These ablations are reported in Table 10 of Appendix A.6.2 in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer hLTG

Agent harness (W1)

Within the environment, only the transport of the tool calls differs between model families: closed models call the tools as shell commands from their vendor CLI, and open models write <tool_call> blocks that our loop parses and executes. To examine this difference, we ran a closed model, Claude Haiku 4.5, inside the open models' loop with exactly what an open model receives, on the same 50 objects, and compared it with the same model in its native harness.

Haiku 4.5 episodes in the open-model text loopnOverallΔ vs. its native harness95% CIp
all4943.9−9.4[−15.2, −4.3]0.003
protocol followed3549.7−1.6[−5.1, +1.8]0.29
first reply invents tool results or submits at once1429.4−28.8[−41.0, −16.3]0.001

Haiku scored lower in the serialized loop (−9.4, 95% CI [−15.2, −4.3]; one of the 50 episodes lacks a complete set of metric records), with the largest losses in the 14 episodes that failed to follow the protocol: its first reply invented tool results or submitted at once, which none of the 200 episodes of the four open models in the same loop did. In the 35 episodes that followed the protocol the difference is small (−1.6), but this split is made after the fact, so it describes the failure mode and does not establish that the two harnesses are equivalent. Moving a closed model to the open models' loop therefore does not help it. Every model in Table 2 is evaluated with the best interface we know for it to do the task, and Table 2 measures each system together with its execution harness.

Regarding harness modes such as a goal command, the closed models were run non-interactively (claude -p, codex exec) with one instruction that states the completion condition: keep working until a submit returns episode_done: true. This is the same contract a goal mode enforces, and it was met: all six closed models submitted a design on all 200 tasks. We did not run a separate goal-mode configuration, so we cannot quantify its effect beyond this; the three studies above are our evidence on the sensitivity to the execution environment.

The harness comparison is reported in Appendix A.6.2 of the revised paper, in the paragraph "Each model works best in its own harness".

AuthorsOpenReview ↗

Response to Reviewer hLTG

Expansion of the core evaluation set and consistency after sampling (W2)

We appreciate the reviewer's understanding of the cost of running these agents, and we agree that the differences we report should be shown to be consistent. First, all main results in the revised paper are now on 200 objects (Table 2), four times the submitted set. The new objects are harder (closed models 68.0 on the original 50 and 61.8 on the new 150; open models 40.6 and 31.7), yet the conclusions hold: system-level scores on the original 50 and the new 150 correlate at Spearman ρ = 0.90–0.98 for each of the twelve metrics, and two disjoint stratified 50-object samples agree at Pearson r = 0.94–0.99 (revised Tables 6 and 7). Second, following the reviewer's suggestion, we generated every design a second time, independently, for six models spanning the range of Table 2, on a stratified sample of 50 of the 200 objects (300 repeated episodes).

ModelOverall, run 1Overall, run 2Δ
GPT-5.6 Sol63.061.5−1.6
Claude Haiku 4.553.753.0−0.7
Qwen3.5-27B53.953.1−0.8
GPT-OSS-120B49.349.2−0.1
Qwen3.5-35B-A3B45.245.1−0.1
Gemma-4-31B39.937.8−2.1

The observed differences are consistent: the ranking of the six models is identical in both runs (Kendall τ = 1.000), every 95% interval of the per-model difference includes zero, and system means differ by 0.9 points in absolute value on average (the second run is lower by 0.9 on average). Single episodes are noisy (the per-object difference between two runs has a standard deviation of about 14 points), so conclusions should rest on means over objects: the standard error of a system's Overall mean from generation noise is 0.7 points on 200 objects, and generation noise alone can account for differences of up to about 2 points on the fixed core set (an estimate of run-to-run noise, not a threshold for generalization to new objects). The closed–open gap (63.4 vs. 33.9) and the differences between model tiers are far larger, while the four frontier models (66.5, 65.9, 65.4, 64.9) lie within this margin and we do not claim an order among them. We could not afford to repeat the other frontier models and state this as a limitation. We thank the reviewer for this suggestion, which gives the benchmark a stated resolution and will help future users of LMBuild decide which differences are meaningful.

In the revised paper, Table 2 reports the results on 200 objects, Tables 6 and 7 show the consistency across sub-samples, and Table 12 in Appendix A.6.3 reports the repeated runs.

Overall, we are truly grateful for the reviewer's encouraging assessment and support of our work, and for suggestions that directly improved the paper. With the native-CAD comparison, the tool ablations and the harness swap, and with the evaluation expanded to 200 objects and repeated runs, we believe the main findings (current agents do well on structural and visual criteria, yet remain far from functional completeness, correct kinematics and operability) now rest on firmer evidence. We hope these additions address the reviewer's remaining concerns and that the reviewer will maintain the positive evaluation of our work. We would be glad to answer any further questions.

Reviewer_hbSeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LMBuild introduces a benchmark for evaluating LLM agents on generating buildable and functional 3D structures. It provides an interactive environment with 19 construction tools and a hierarchical evaluation framework covering structural soundness, functional affordance, design quality, and physical realization. The paper evaluates 30 systems on a 50-object core benchmark and studies part retrieval, creation, iterative revision, and functional prompting.

Critical strengths
  1. Broader evaluation scope. The benchmark goes beyond visual quality to assess structural soundness, functional components, kinematics, and physical operability.
  2. Interactive evaluation environment. The framework supports part retrieval, creation, assembly operations, and iterative feedback, enabling systematic comparisons across diverse models.
Critical weaknesses
  1. Unclear value of the restricted tool environment. The framework limits agents to 19 predefined tools without comparing against capable coding agents directly using native CAD APIs. For systems such as Codex or Claude Code, these restrictions may limit rather than enhance their capabilities. The current ablations do not establish the benefit of this tool interface or its individual components.

  2. Limited physics-grounded interaction and experimental validation. Although physical simulation is used for evaluation, agents cannot directly conduct simulation experiments to guide their design decisions. Feedback is primarily geometric and rule-based. Furthermore, all main experiments use only 50 objects, limiting the strength of the generalization claims.

Questions for authors
  1. How does LMBuild compare with Codex or Claude Code directly using native CAD APIs under matched computational budgets?
  2. Can the authors quantify the contributions of geometry checking, visual feedback, and structured placement through controlled tool ablations?
  3. Why is physics simulation limited to evaluation rather than exposed to agents as an interactive tool? How might this affect conclusions about agent capabilities?
  4. Can the authors provide broader evaluation or repeated trials beyond the 50-object core set?
5 comments
AuthorsOpenReview ↗

Response to Reviewer hbSe

We sincerely thank the reviewer for the thoughtful and constructive feedback, and for recognizing both the broader evaluation scope of LMBuild beyond visual quality and the interactive environment that enables systematic comparison across diverse models. We truly appreciate the reviewer's questions, which led to changes that we believe substantially strengthen the paper. First, the main evaluation in the revised paper now covers 200 objects. Second, we ran four new controlled studies: a direct comparison with coding agents (Codex and Claude Code) that use native CAD libraries under matched budgets, controlled tool ablations, an agent-side physics simulation tool, and a repeated-run analysis. We would also like to highlight the unified framework as a central contribution of our work: a single construction interface under which 19 LLM agents, from 4B open models to frontier coding agents, build and receive feedback in the same way, and a single output specification under which all 30 systems, including 11 domain-specific generators that run their native pipelines, are scored on the metrics that apply to them. We address each concern below.

Tool environment for a unified, fair comparison for all models and existing solutions (W1)

The tool environment is not proposed as the best way for a strong coding agent to do CAD. It is the common contract that makes heterogeneous systems comparable. LMBuild scores an assembly specification (parts with roles, joints, materials and an assembly sequence), not a mesh. The 19 tools are the smallest action space that produces this specification in one metric world frame and returns the same feedback to every agent, from a 4B open model to GPT-6 Astra. The interface does not restrict geometry: create_part accepts a constructive-solid-geometry program (extrude, revolve, booleans, hull, transforms) compiled by a CAD kernel, so an agent can model any part it could write in a CAD script. Without a shared interface, a score would mix modelling ability with each agent's success at installing libraries, exporting meshes and writing a valid deliverable.

The new experiments below support this contribution from three directions:

  • The strongest agent tested scores about the same with and without the framework. With free native CAD code, GPT-5.6 Sol scores 58.9 against 61.5 with the LMBuild tools (−2.6, 95% CI [−5.8, +1.3]), and no model scores higher without the tools.
  • The framework enables systems below the state of the art to complete the task. Without it, open models deliver a valid design in only 12–52% of episodes and lose 12 to 48 points; with it, they are measured on construction ability and not on tool-chain handling.
  • The framework is robust. Removing any one tool family changes the Overall score by at most 3.2 points, and removing all three feedback channels together costs 4.0, so the conclusions do not depend on a particular tool.

Together, these results should show that the unified framework is what makes a fair comparison across the 19 LLM agents possible, at a small and non-significant cost for the strongest agent we tested.

AuthorsOpenReview ↗

Response to Reviewer hbSe

Our environment does not degrade the capabilities of state-of-the-art models (W1, Q1)

We ran the comparison the reviewer asks for. In the native-CAD condition the agent receives the same task text, image, catalogue meshes, material vocabulary and required declarations, but no LMBuild tools. It works in a sandboxed shell with a Python CAD stack (build123d, CadQuery, trimesh, manifold3d; pip allowed) and delivers a standard URDF plus an assembly file. A validator checks the file format only. The study covers two closed and four open models on the stratified sample of 50 objects and is scored by the unchanged evaluator. Every setting is paired, per object, with a fresh re-run of the full LMBuild environment.

ModelLMBuild toolsNative CADΔ Overall [95% CI]ΔSΔAΔDΔRValid deliverable
GPT-5.6 Sol61.558.9−2.6 [−5.8, +1.3]−4.4+2.3+0.9−9.298%
Claude Haiku 4.553.040.8−12.3 [−16.2, −8.1]−18.9−5.4−13.1−11.7100%
Qwen3.5-27B53.15.3−47.8 [−52.2, −43.0]−70.4−24.8−45.2−50.812%
Qwen3.5-35B-A3B45.122.7−22.4 [−28.7, −16.0]−36.4−8.5−25.3−19.552%
GPT-OSS-120B49.216.0−33.2 [−39.1, −27.0]−52.4−12.6−30.4−37.334%
Gemma-4-31B37.825.6−12.2 [−20.4, −3.7]−19.1−3.8−14.6−11.552%
Mean of six models−21.7 [−34.6, −9.9], p = 9e-4

No model scores higher with native CAD than with the LMBuild tools. For the strongest coding agent tested, the two interfaces are close: GPT-5.6 Sol reaches the same Affordance and Design scores with free CAD code (+2.3 and +0.9), so the tools do not cap what it can model, while it loses on Soundness (−4.4) and Realization (−9.2), mainly on the executability of the assembly sequence, which is where the environment's geometry feedback helps. Claude Haiku 4.5 loses 12 points and the open models lose 12 to 48 points; most of the open models' loss comes from never producing a valid deliverable (12–52% valid), which confirms that an unrestricted setting would measure tool-chain handling more than construction ability. Regarding scope, this comparison covers one frontier coding agent and one smaller one: we did not run Claude Opus 5, Claude Fable 5.1 or GPT-6 Astra in the native-CAD condition for cost reasons, and we state this in the paper.

Table 11 in Appendix A.6.2 of the revised paper reports the updated results.

Our environment keeps the cost of state-of-the-art models comparable and makes their behaviour traceable (W1, Q1)

Both interfaces were given the same budget: 160 tool calls in LMBuild, and 160 shell or file actions within 60 minutes with native CAD. Neither closed model reached a limit in any episode. The table reports the median per episode on the same 50 objects.

ModelInterfaceActionsTime (min)Input tokens (k)Output tokens (k)
GPT-5.6 SolLMBuild tools515.55409.4
GPT-5.6 SolNative CAD165.956516.4
Claude Haiku 4.5LMBuild tools413.61,75613.7
Claude Haiku 4.5Native CAD212.280914.1

For the strongest coding agent tested, the cost of the two interfaces is similar. GPT-5.6 Sol takes about the same time and reads a similar number of input tokens, and it writes fewer output tokens with the LMBuild tools, because a tool call is shorter than a CAD script. For the smaller Claude Haiku 4.5 we do observe a moderate increase: the output tokens are similar, but the episode takes 1.4 minutes longer and re-reads about twice as many cached input tokens, which raises the price of an episode from $0.21 to $0.31. In return, Haiku gains 12 points in the Overall score. For the open models, native CAD is the more expensive interface: 79 of 200 episodes ran into the 60-minute limit, against 29 of 200 that used all 160 tool calls in LMBuild.

The larger number of actions is itself a benefit of the framework. A native-CAD episode consists of a few long scripts, and the decisions about parts, placement and joints are hidden inside them. In LMBuild every decision is one tool call, logged with the feedback it returned and the resulting scene state. This makes the reasoning of a state-of-the-art model visible step by step: which parts it retrieves or creates and in what order, where it stops to render or check the scene, and whether it repairs a reported collision or continues past it. Because every model acts through the same tools, these traces can be compared across all agents, which is what makes the analysis of tool-use and scene-state errors in the paper possible.

Appendix A.6.2, paragraph "The shared interface does not limit strong agents" of the revised paper reports the updated results.

AuthorsOpenReview ↗

Response to Reviewer hbSe

Additional controlled tool ablations (W1, Q2)

We removed one tool family at a time for the same six models and 50 objects. Removing geometry checking takes away check, measure and the contact and collision reports returned by each placement. Removing visual feedback takes away render. Removing structured placement takes away attach, so parts are placed by coordinates only. Δ is the paired difference in the Overall score against the full environment, averaged over the six models.

SettingΔ Overall95% CIp
− geometry checking−3.2[−6.3, −0.2]0.019
− visual feedback+0.0[−3.0, +2.5]0.98
− structured placement+1.7[−1.3, +3.8]0.24
− all three−4.0[−7.0, −1.3]0.004
+ physics simulation scoring tool+0.6[−2.0, +2.8]0.64
Native CAD (no LMBuild tools)−21.7[−34.6, −9.9]9e-4

Geometry checking is the component that matters. Removing it lowers connectivity (S.1 −6.1), collision avoidance (S.2 −8.2) and the executability of the assembly sequence (R.1 −12.2). Visual feedback and structured placement have no measurable effect when removed alone, because agents compensate with the remaining tools, for example by placing parts by coordinates and verifying with check. The environment is therefore robust: it does not depend on any single tool, removing all three costs 4 points, and removing the interface altogether costs 22. A fresh re-run of the full environment reproduces the original scores (Δ +0.9 [−1.2, +3.3], p = 0.49), so these differences are not run-to-run noise.

Table 10 in Appendix A.6.2. of the revised paper reports the updated results.

Physical simulation as an evaluation signal rather than ready for interactive feedback (W2, Q3)

Agents already receive physically grounded feedback during construction: each placement returns contacts, collisions and ground penetration computed by the same geometry kernel as the evaluator, and check reports disconnected parts, interpenetration, stability under gravity and joint problems. Following the reviewer's suggestion, we also exposed the simulation itself. In the "+ simulate" setting, the agent can call a simulate tool that runs the evaluator's rigid-body engine on its own declared joints and reports whether the object stands still under gravity and how much of the commanded travel each joint achieves. The agents used the tool (GPT-5.6 Sol in 47 of 49 episodes, Qwen3.5-27B in 25 of 50), but it gave limited help: Overall changes by +0.6 [−2.0, +2.8] and R.3 by +0.6, and the operability of GPT-5.6 Sol stays the same (R.3 15.4 → 15.0).

We believe simulation is not yet ready to serve as interactive feedback, for the following reasons. A simulation result is a number that current models hardly know how to optimise with their existing knowledge. A geometric message names the two parts that collide, whereas a simulation score does not indicate which part or joint should change, so supplying it currently provides limited help. In addition, a simulator can only test the joints that exist, and the failure analysis in the revised appendix shows that most failures come earlier: among the failed motion targets of the closed models, 57% lack the moving component itself and a further 17% have the part but no joint. Finally, a full score in simulation does not mean that the object is buildable and functional. The simulator tests the joints that the agent declares, so a score exposed as an optimisation target could be satisfied by simplified or placeholder mechanisms, which is a form of reward hacking. For the same reason we do not expose the list of capabilities that R.3 tests.

We therefore keep simulation as an evaluation signal in the baseline protocol and report the "+ simulate" setting in the appendix. Our conclusions about agent capability do not change: the bottleneck is inferring which components and mechanisms an object needs, which is also what the functional-prompt experiment shows (R.3 +9.9 when the mechanisms are specified). This result also inspires us to investigate, in future work, stable reinforcement with verifiable feedback that is grounded in real-world construction.

Appendix A.6.2, paragraph "Simulation feedback does not help"; failure modes in Table 8 of the revised paper reports the updated results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.