- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LMBuild aims to evaluate agents that construct structured 3D objects, rather than merely generate visually plausible geometry. Its environment allows agents to retrieve or create parts, position them, specify joints and materials, and define an assembly sequence. The benchmark combines existing geometry datasets with evidence-grounded knowledge about real-world objects and defines twelve metrics covering structural soundness, functional affordance, design quality, and physical realization. Experiments evaluate 30 system configurations on a core set of 50 objects. The reported results show that systems generally perform better on structural and visual criteria than on kinematic correctness and operability. Providing explicit functional specifications improves functional-part completeness and operability, while iterative revision yields relatively modest gains.
Critical strengths
1.Benchmark provides a useful evaluation target beyond visual appearance, by considering part structure, joints, materials, and assembly order, it captures requirements that visual similarity alone cannot assess. The 50-object core set covers several construction regimes. Experiments on part retrieval versus creation and on explicit task specifications make the benchmark useful both for comparing system performance and for diagnosing model weaknesses.
2.LMBuild documents the tool protocols, metric definitions, and evidence-grounded knowledge extraction procedure in detail, and clearly states its limitations. The design-quality evaluator is calibrated using generated designs for eight objects and tested on a disjoint set of seven objects, with ratings provided by four annotators. Reporting item-level correlations alongside system-level agreement helps assess how well the evaluator reflects human judgments.
Critical weaknesses
1.Table 5 validates the Design Quality metrics but does not establish the validity of the Affordance or Realization metrics. R.3 combines role matching, graph relationships, human-use heuristics, and selected rigid-body motion tests. The presence and connection of components identified as an oven’s heating subsystem or a fan’s airflow subsystem therefore do not demonstrate that these functions have actually been realized. Simulation with uniform density and convex-hull collision geometry is a reasonable standardized proxy, but convexification can remove cavities that are important for motion. The effect of this approximation on the evaluation conclusions has not been quantified. Low R.3 scores can therefore reflect both generator failures and errors introduced by evaluator approximations, with their respective contributions remaining unclear.
2.A.1 metric evaluates outputs without semantic roles using only whole-object shape, but includes an additional component-level term when sufficiently informative roles are available. The same metric can therefore measure different quantities across systems. To illustrate the stated definition, a whole-shape F-score of 0.8 and a mean component F-score of 0.2 yield 56/100 when the component term is included, compared with 80/100 when it is omitted. In addition, closed-source agents run through provider CLIs, whereas open-source agents use a custom serialized tool-call protocol and associated termination rules. Section 3.5 itself attributes many part-creation failures to tool-specific conventions. These differences are relevant when evaluating deployed systems, but they weaken the attribution of performance gaps to intrinsic construction or reasoning ability.
Questions for authors
1.Provide an independent validation of the A/R scores using expert mechanical assessments or known-valid assemblies and deliberately modified invalid variants? Also, the main conclusions remain robust to changes in collision representations and geometric tolerances?
2.How do the rankings change when all eligible systems are compared using only the whole-object shape term of A.1? Is there evidence that the performance gaps persist when systems use the same tool-calling interface?
3.For Figure 4, specify the models, objects, and paired observations included in the analysis. The caption states that confidence intervals are obtained by bootstrapping over models and that Wilcoxon signed-rank tests are performed over episodes, whereas A.5 describes resampling over objects. Clarify the actual resampling unit and how dependencies among observations involving the same object are handled. Functional prompting improves A.2 and R.3 by 10.3 and 12.0 points, respectively, but also reduces R.1 by 9.0 points. Does this trade-off in assembly performance also appear in the per-object results? The explicit-specification experiment is reasonable in itself; its gains should be interpreted as better specification following, rather than improved reasoning under unchanged information.
5 comments
Response to Reviewer DJLb
What R.3 Measures, and Why Its Composite Design Is Appropriate (W1)
We agree with the reviewer on the scope of R.3: the presence and connection of, for example, an oven's heating subsystem do not prove that the oven heats. Below we explain why R.3 is designed this way and why its relation to the other metrics is intended.
R.3 tests standardized preconditions for function. Verifying heating or airflow would require thermal, fluid and electrical simulation specific to each object, which is not feasible in a standardized benchmark over 200 heterogeneous objects. R.3 instead tests standardized preconditions of each capability: the required components exist, they are connected into the functional chain, a person can reach and operate them, and the required motions can be actuated in simulation without collision or toppling. A design that passes is not thereby certified, and a design may occasionally realize a function through a mechanism the reference description does not anticipate. Our conclusion uses R.3 only as evidence of what is missing: operability is low for every system (the best R.3 is 22.1), and these low scores persist under the collision and tolerance settings we tested (point 3). The definition of R.3 and the limitations now state this scope explicitly.
Why R.3 is a composite. A capability is a conjunction: a fan provides airflow only if the blades exist, are mounted on the driven shaft, can rotate freely, and the fan stands. Each element calls for its own kind of evidence (simulation for motion, geometry for human use, presence and connection for non-mechanical subsystems), and R.3 scores a capability by its weakest element.
Why the overlap with A.2 and A.3 is intended. The levels of LMBuild are cumulative. A.2 and A.3 ask whether each ingredient is correct on its own (the parts are present; each joint matches the reference in type, axis and pivot), and R.3 asks whether the ingredients work together under physics. The validation above shows that they are related but not redundant: turning a joint axis by 90° lowers A.3 by 57 points and R.3 by only 1.4, because the motion can still be actuated; fixing the joints lowers both; changing materials or the assembly order lowers neither. This is why the paper reports the twelve metrics separately. R.3 identifies components through the roles an agent declares, since roles are part of an assembly specification; a variant in which the evaluator alone identifies them preserves the ranking of the 19 LLM agents exactly (Kendall τ = 1.000 on all 200 objects).
Do low R.3 scores come from the generators or from the evaluator's approximations? The two can be separated: R.3 moves when the designs change and does not move when the approximations change.
- Changing the evaluator: a convex decomposition instead of convex hulls moves no system's R.3 by more than 0.5 points.
- Changing the designs: deleting the moving subassemblies lowers R.3 by 14.8 points.
- No evaluated capability succeeds on 30–40% of the tasks for five closed models and on 51% for Claude Haiku 4.5, and missing components are a frequent failure mode (Table 8).
Low R.3 scores therefore reflect what the generated designs lack. An assessment by mechanical experts is a natural next step when current R.3 could be reached, and is listed as future work.
In the revised paper, the definition of R.3 in Appendix A.4 (lines 3177–3178) states this scope, and the paragraph "A.3 and R.3 are complementary" in Appendix A.6.4 explains how the two metrics differ.
Response to Reviewer DJLb
3. Robustness to the Collision Representation and Geometric Tolerances (W1, Q1)
The reviewer asks whether our conclusions hold when the collision representation and the geometric tolerances change. They do. We changed each setting separately and re-scored the designs of all 30 systems on 50 core objects. For each setting, the table reports how much any system's score changes, and how well the ranking of the 19 LLM agents agrees with the default ranking. Agreement is measured by Kendall's τ, the rank correlation between the two orderings: τ = 1 means the order of the agents is identical, and values close to 1 mean that only a few neighbouring agents swap places.
| Setting changed | Metric | Largest score change of any system | Ranking agreement with default (τ) |
|---|---|---|---|
| Collision shapes that keep cavities, instead of one convex hull per part | R.3 | 0.5 | 1.00 |
| Distance tolerances halved | A.1 / A.2 / R.3 | 19 / 10 / 1 | 0.97 / 0.97 / 0.98 |
| Distance tolerances doubled | A.1 / A.2 / R.3 | 18 / 9 / 1 | 0.95 / 0.97 / 0.99 |
| Insertion clearance for assembly, halved | R.1 | 0.7 | 1.00 |
| Insertion clearance for assembly, doubled | R.1 | 0.8 | 0.99 |
Convex hulls and cavities. The reviewer is right that a convex hull fills the cavities of a part. We therefore replaced each hull with a convex decomposition, which splits a concave part into several convex pieces that follow its actual shape, so cavities are preserved (computed with the CoACD library). R.3 is essentially unchanged: no system moves by more than 0.5 points, and the ranking is identical. The reason is that the simulator already ignores contact between parts that overlap in the designed pose, so a filled cavity cannot block the part inside it.
Tolerances. A.1 is an F-score at a distance threshold, so its absolute level moves with the threshold, as expected. The comparison between systems does not: the ranking of the agents is nearly the same under every setting (τ between 0.95 and 1.00), closed models remain ahead of open models, and every agent still scores lower on Affordance and Realization than on Soundness. R.1 and R.3 barely change at all.
In short, these choices change the level of some scores but not the comparison between systems, and the low R.3 scores are not caused by the collision model. This analysis is reported in Appendix A.6.5 of the revised paper, with the full results in Table 14.
Response to Reviewer DJLb
We sincerely thank the reviewer for the careful assessment and constructive suggestions, and for recognizing the value of evaluating generated structures beyond visual appearance and the diagnostic insights provided by LMBuild. We appreciate the concerns regarding the validity and comparability of the Affordance and Realization metrics, the effect of different execution interfaces, and the statistical analysis of Figure 4. To address them, we validated the Affordance and Realization metrics with controlled errors, tested their sensitivity to the evaluator's settings, clarified what R.3 measures, refined A.1 so that one rule applies to every system, studied the effect of the execution interface, and redid the statistical analysis of Figure 4. In addition, the main evaluation in the revised paper now covers 200 objects, up from 50 in the submission. We address each concern below.
Validation of the Affordance and Realization Metrics (W1, Q1)
We understand the reviewer's request for a validation showing that the Affordance and Realization metrics respond correctly to actual designs. To provide this validation, we conducted an additional experiment in which we take known designs, break each of them in one controlled way at a time, and re-score every variant with the unchanged evaluator. The base designs that we break are (i) the 50 reference assemblies, which are valid by construction, and (ii) the designs generated by six models (GPT-5.6 Sol, Claude Haiku 4.5, Qwen3.5-27B, Qwen3.5-35B-A3B, GPT-OSS-120B, Gemma-4-31B). No design was selected or filtered: of the 350 base designs, 340 are non-empty and all of them are used. Each of the 10 perturbations is applied to every design that has the property it breaks. For example, material and relabelling perturbations apply to all 340 designs, sequence reversal to the 314 that declare an assembly sequence, and joint perturbations to the 271 that contain a moving joint. This gives 3,020 broken variants, each compared with its own unperturbed copy. The tables below list, for each metric, the errors we injected and the resulting change in score.
A.1 Geometry
A.2 Parts
A.3 Kinematics
R.1 Sequence
R.2 Material
R.3 Operability
Control (all twelve metrics)
We observe three key results from these tables: (1) Every metric detects the errors it is designed for, not only on the valid reference assemblies but also on generated designs. (2) The metrics are specific. An error leaves the unrelated metrics unchanged: for example, changing the materials or the assembly order does not move the S, A or R.3 scores. (3) The metrics do not react to irrelevant changes. Renaming, reordering and translating a design moves no metric by more than 0.6 points on average.
With this added experiment, we hope to assure the reviewer that the A and R metrics are validated and work as intended on both reference and generated designs. This validation is reported in Appendix A.6.4 of the revised paper, with the full results in Table 13.