- Rating
- 1(−1)
- Confidence
- 3(+1)
- Informativeness
- 1(−1)
- Presentation
- 2
- Soundness
- 1(−1)
Read review
Paper summary
The paper studies whether multimodal language models can construct brick assemblies from instructions, decomposing each step into selecting a candidate part and predicting its global translation and rotation. BC-Bench renders multiview instructions, partial assemblies, and part information from design files, covering diverse part shapes with object-level train/test splits. Brick-Composer combines Human Design Sparks, World Feedback, and Synthetic Experience in supervised fine-tuning: human-designed examples, correction trajectories following erroneous placements, and procedurally generated structures. Experiments on two model backbones measure selection accuracy, pose errors, and joint single-step success. For example, Qwen's strict single-step success rises from 0.36% to 14.27%. These results primarily measure single-step assembly under reference-history contexts; the paper also reports that complete-object success remains zero.
Critical strengths
-
Diverse parts make this more than a placement task with regular bricks. Separating selection from pose, while accounting for interchangeable parts and rotational symmetries, helps identify where the models fail.
-
Task-specific training clearly improves single-step performance on both backbones. The disclosure of reference histories and zero complete-object success helps distinguish local capability gains from complete autonomous assembly.
Critical weaknesses
-
The standard prompt defines local axes through rotation-matrix rows, while the feedback prompt uses columns (pp. 16–17). If the experiments share a parser, this affects comparability between baseline and feedback settings. The examples alone do not establish an implementation error, but the actual convention needs clarification.
-
Feedback examples include errors relative to the ground-truth pose (pp. 17–18). These are legitimate training supervision, but their test-time availability is unclear. The distinction matters for whether World Feedback is observable execution feedback or correction information requiring ground truth. This alone does not establish test-time leakage.
-
Synthetic expands training data from 3,873 to over 40K steps, with the same number of training epochs. Component effects are therefore confounded with additional optimization updates (Tables 3 and 10). Without a matched-budget comparison, the contribution of each signal remains uncertain, leaving me leaning toward weak rejection.
Questions for authors
-
Is the row/column inconsistency confined to the examples, or is it present in the actual prompts?
-
Which test settings receive ground-truth pose errors, and what are the results with rendered feedback alone?
-
Under a matched training budget, how does Designer + Synthetic perform without Feedback?
8 comments
Response to Reviewer zgtb
Additional assembly supervision improves performance beyond repeated training (W3).
We appreciate the concern that the larger synthetic corpus also entails more optimization updates. We added two complementary analyses: a comparison of designer and synthetic examples at the same dataset size, and a one-versus-five-epoch training-duration control.
The same-scale comparison uses 3,873 assembly steps from each source, as specified in Appendix D, p. 21, lines 1120–1127. Table 8 shows:
| Supervision | Selection accuracy (%) ↑ | Translation error (LDU) ↓ | Rotation error (°) ↓ | Strict step success (%) ↑ |
|---|---|---|---|---|
| Same-scale Synthetic | 44.15 | 180.07 | 59.12 | 4.44 |
| Designer Supervision | 48.32 | 162.82 | 57.81 | 5.44 |
| Synthetic Full | 62.38 | 107.64 | 47.98 | 12.33 |
Human-designed examples are more effective at the same scale, while expanding synthetic experience substantially improves performance. The new training-duration results in Table 9, are:
| Supervision | Selection, 1 epoch (%) | Selection, 5 epochs (%) | Step success, 1 epoch (%) | Step success, 5 epochs (%) |
|---|---|---|---|---|
| Designer only | 48.32 | 48.41 | 5.44 | 5.44 |
| Synthetic only | 62.38 | 62.38 | 12.33 | 12.33 |
| Designer + Synthetic | 66.55 | 66.82 | 13.69 | 13.87 |
The small changes suggest limited benefit from repeatedly training on the existing examples in these settings. This experimental result actually drives us to train all the systems with 1 epoch instead of more from the very beginning. We hypothesize that current MLLMs can learn efficiently from the available examples, but remain hungry for more diverse, domain-specific data.
Designer + Synthetic without feedback, and the additional benefit of learned feedback (Q3).
Following the reviewer's suggestion, we explicitly report the results Designer + Synthetic without feedback. Table 8 also includes synthetic-only training with and without feedback:
| Configuration | Translation error (LDU) ↓ | Rotation error (°) ↓ | Strict step success (%) ↑ |
|---|---|---|---|
| Synthetic Full | 107.64 | 47.98 | 12.33 |
| Synthetic Full + Feedback (L) | 82.53 | 43.05 | 12.96 |
| Designer + Synthetic | 72.83 | 40.39 | 13.69 |
| Brick-Composer | 64.92 | 37.25 | 14.23 |
The combined supervision already accounts for most of the improvement. Learned feedback provides a further reduction in pose errors and a smaller increase in strict step success. However, we describe learned feedback as a complementary gain rather than the primary source of improvement.
These results show that learned feedback provides a complementary benefit even after synthetic supervision is included. Moreover, extending Designer + Synthetic training to five epochs does not close the observed performance gap, supporting the usefulness of feedback-based correction examples.
Overall, we sincerely appreciate the reviewer's careful evaluation and the suggestions that motivated these revisions. We respectfully ask the reviewer to reconsider the weak-reject recommendation and consider raising the score in light of the corrected prompt descriptions, explicit training/evaluation distinction, and added ablation evidence. We hope the revised manuscript makes the contribution more convincing while remaining transparent about its scope and limitations.
Thank you for the detailed response and additional experiments. The correction of the rotation-prompt inconsistency and the clarification that ground-truth pose errors are excluded from test-time feedback address my concerns about the prompt specification and privileged information during evaluation. The additional Designer + Synthetic ablation is also helpful in assessing the contribution of synthetic supervision.
However, after examining new predicted-history rollout results, I have an additional concern about the gap between the demonstrated capability and the paper’s framing. Even on the best object selected separately for each configuration, Brick-Composer achieves correct initial prefixes of only 1 out of 16 steps with Gemma and 5 out of 96 steps with Qwen.
I understand the diagnostic value of evaluating individual steps under ground-truth assembly histories, given concurrent MLLMs may have poor ability to self roll out a whole LEGO assembly (maybe GPT6-Astra could do better? Anyway, I do not require authors to do such experiments in this rebuttal period). I also do not expect the authors to solve reliable long-horizon assembly during the rebuttal period. But my concern is, how this GT-conditioned task can serve as an effective proxy task for a true self rollout, for MLLMs' ability to construct a LEGO from scratch? I want to see more explanations about this, especially when you titled your paper as "MLLMs Construct Everything from Building Blocks".
Response to Reviewer zgtb
We sincerely thank the reviewer for the detailed feedback and constructive comments, and for recognizing the value of our benchmark and the improvements across both backbones. We especially appreciate your careful examination of the rotation conventions, feedback signals, and training budgets. Your thoughtful questions have helped us clarify the experimental protocol and strengthen the evidence supporting our learning framework. We welcome the opportunity to address these concerns in detail. Below, we explain the corrected prompt convention and the distinction between training supervision and inference-time feedback, and present additional data-source and training-duration experiments with references to the corresponding revisions. Unless explicitly labeled as original, all line numbers refer to the revised manuscript.
Correcting the rotation-convention description and updated results show consistent findings (W1, Q1, Q2).
We thank the reviewer for identifying this inconsistency. The original feedback prompt described the local-axis directions using “columns” (original Appendix B, p. 17, lines 893–895), whereas the standard pose-estimation prompt used “rows” (original p. 16, lines 811–816). We agree that these descriptions are not interchangeable and that the inconsistency should be corrected rather than dismissed as a formatting issue despite being small. Both prompt descriptions now consistently specify rows as the local axes expressed in world coordinates: Appendix B, p. 16, lines 826–831, and p. 17, lines 905–912.
The inconsistency was present in the affected feedback (Learning) prompts, rather than being only a typographical error in the appendix. We corrected the affected training and evaluation prompts, retrained the affected configurations, and reran evaluation. The updated overall results appear in Table 3 and are reproduced below. Parentheses give the absolute change from the previously reported result; pp denotes percentage points.
For these configurations, the absolute changes in overall strict step success are less than 0.1 percentage points. The main comparisons remain unchanged: learned feedback improves over designer supervision, and Brick-Composer provides the strongest performance among the main learning configurations. The revised full-framework results are also reflected in Table 4, Table 8, and Table 13.
We believe the limited impact could be explained by the localized nature of the inconsistency: it concerned only a small portion of an approximately 1,500-token instructional prompt describing rows versus columns (3 words different), while the supervised visual inputs and pose examples provided consistent task information. The model may therefore have relied more strongly on these supervision signals and visual relationships than on the conflicting instructional wording.
Relative pose-error feedback is supervision used only during training (W1, Q2).
We incorporate relative pose-error feedback into training trajectories to help the model relate visual discrepancies to spatial corrections. These ground-truth-relative error values are not included in any evaluation prompts, including inference-time feedback prompts. Otherwise it would cause a leakage. At inference, the feedback consists of the rendered predicted assembly state alongside the visual instructions, without ground-truth translation or rotation errors. We have made this distinction explicit in Appendix B, p. 18, lines 965–970.