acceptodds
Under review as a conference paper at ICLR 2027

Brick-Composer: MLLMs Construct Everything from Building Blocks

Abstract

We dream of AI agents that can read arbitrary designs and construct real-world objects from reusable building blocks. As a first step toward this vision, we study whether multimodal large language models (MLLMs) possess the visual grounding and spatial reasoning capabilities required for brick assembly. We formulate brick assembly as a sequential decision-making problem, where each step involves two subtasks: brick selection, identifying the target brick from candidate components, and brick pose estimation, predicting where and how the selected brick should be placed. To support this study, we introduce BC-Bench (Brick Construction Benchmark), the first benchmark for evaluating MLLMs on assembly with diverse bricks. Experiments show that current state-of-the-art MLLMs remain far from reliable builders, struggling with fine-grained brick selection and failing at precise pose estimation. To bridge this gap, we propose Brick-Composer, a learning framework that equips MLLMs with assembly skills through three complementary signals: Human Design Sparks, which provide affordance-rich construction demonstrations; World Feedback, which grounds predicted actions in visual and physical consequences; and Synthetic Experience, which scales learning beyond existing object designs. Brick-Composer improves brick selection accuracy by over three times, substantially reduces pose estimation errors, and raises strict step-level assembly success from less than 1% to around 15%. After training, a Qwen-3-VL-8B can correctly compose up to 42% of the steps for a complete object, suggesting that MLLMs can acquire assembly capabilities through targeted, physically grounded learning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
MLLMs, Brick Assembly

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zgtbOpenReview ↗
Rating
1(−1)
Confidence
3(+1)
Informativeness
1(−1)
Presentation
2
Soundness
1(−1)
Read review
Paper summary

The paper studies whether multimodal language models can construct brick assemblies from instructions, decomposing each step into selecting a candidate part and predicting its global translation and rotation. BC-Bench renders multiview instructions, partial assemblies, and part information from design files, covering diverse part shapes with object-level train/test splits. Brick-Composer combines Human Design Sparks, World Feedback, and Synthetic Experience in supervised fine-tuning: human-designed examples, correction trajectories following erroneous placements, and procedurally generated structures. Experiments on two model backbones measure selection accuracy, pose errors, and joint single-step success. For example, Qwen's strict single-step success rises from 0.36% to 14.27%. These results primarily measure single-step assembly under reference-history contexts; the paper also reports that complete-object success remains zero.

Critical strengths
  1. Diverse parts make this more than a placement task with regular bricks. Separating selection from pose, while accounting for interchangeable parts and rotational symmetries, helps identify where the models fail.

  2. Task-specific training clearly improves single-step performance on both backbones. The disclosure of reference histories and zero complete-object success helps distinguish local capability gains from complete autonomous assembly.

Critical weaknesses
  1. The standard prompt defines local axes through rotation-matrix rows, while the feedback prompt uses columns (pp. 16–17). If the experiments share a parser, this affects comparability between baseline and feedback settings. The examples alone do not establish an implementation error, but the actual convention needs clarification.

  2. Feedback examples include errors relative to the ground-truth pose (pp. 17–18). These are legitimate training supervision, but their test-time availability is unclear. The distinction matters for whether World Feedback is observable execution feedback or correction information requiring ground truth. This alone does not establish test-time leakage.

  3. Synthetic expands training data from 3,873 to over 40K steps, with the same number of training epochs. Component effects are therefore confounded with additional optimization updates (Tables 3 and 10). Without a matched-budget comparison, the contribution of each signal remains uncertain, leaving me leaning toward weak rejection.

Questions for authors
  1. Is the row/column inconsistency confined to the examples, or is it present in the actual prompts?

  2. Which test settings receive ground-truth pose errors, and what are the results with rendered feedback alone?

  3. Under a matched training budget, how does Designer + Synthetic perform without Feedback?

8 comments
AuthorsOpenReview ↗

Response to Reviewer zgtb

We sincerely thank the reviewer for the detailed feedback and constructive comments, and for recognizing the value of our benchmark and the improvements across both backbones. We especially appreciate your careful examination of the rotation conventions, feedback signals, and training budgets. Your thoughtful questions have helped us clarify the experimental protocol and strengthen the evidence supporting our learning framework. We welcome the opportunity to address these concerns in detail. Below, we explain the corrected prompt convention and the distinction between training supervision and inference-time feedback, and present additional data-source and training-duration experiments with references to the corresponding revisions. Unless explicitly labeled as original, all line numbers refer to the revised manuscript.

Correcting the rotation-convention description and updated results show consistent findings (W1, Q1, Q2).

We thank the reviewer for identifying this inconsistency. The original feedback prompt described the local-axis directions using “columns” (original Appendix B, p. 17, lines 893–895), whereas the standard pose-estimation prompt used “rows” (original p. 16, lines 811–816). We agree that these descriptions are not interchangeable and that the inconsistency should be corrected rather than dismissed as a formatting issue despite being small. Both prompt descriptions now consistently specify rows as the local axes expressed in world coordinates: Appendix B, p. 16, lines 826–831, and p. 17, lines 905–912.

The inconsistency was present in the affected feedback (Learning) prompts, rather than being only a typographical error in the appendix. We corrected the affected training and evaluation prompts, retrained the affected configurations, and reran evaluation. The updated overall results appear in Table 3 and are reproduced below. Parentheses give the absolute change from the previously reported result; pp denotes percentage points.

BackboneConfigurationTranslation error (LDU) ↓Rotation error (°) ↓Strict step success (%) ↑
Gemma-3-12BWorld Feedback (L)172.15 (change: <2 LDU)51.32 (change: <1°)1.99 (unchanged)
Gemma-3-12BBrick-Composer122.31 (change: <2 LDU)45.11 (change:<1°)4.35 (unchanged)
Qwen3-VL-8BWorld Feedback (L)137.51 (change: <1 LDU)53.14 (change: <1°)6.26 (change <0.1 pp)
Qwen3-VL-8BBrick-Composer64.92 (change: <1 LDU)37.25 (change: <1°)14.23 (change <0.1 pp)

For these configurations, the absolute changes in overall strict step success are less than 0.1 percentage points. The main comparisons remain unchanged: learned feedback improves over designer supervision, and Brick-Composer provides the strongest performance among the main learning configurations. The revised full-framework results are also reflected in Table 4, Table 8, and Table 13.

We believe the limited impact could be explained by the localized nature of the inconsistency: it concerned only a small portion of an approximately 1,500-token instructional prompt describing rows versus columns (3 words different), while the supervised visual inputs and pose examples provided consistent task information. The model may therefore have relied more strongly on these supervision signals and visual relationships than on the conflicting instructional wording.

Relative pose-error feedback is supervision used only during training (W1, Q2).

We incorporate relative pose-error feedback into training trajectories to help the model relate visual discrepancies to spatial corrections. These ground-truth-relative error values are not included in any evaluation prompts, including inference-time feedback prompts. Otherwise it would cause a leakage. At inference, the feedback consists of the rendered predicted assembly state alongside the visual instructions, without ground-truth translation or rotation errors. We have made this distinction explicit in Appendix B, p. 18, lines 965–970.

AuthorsOpenReview ↗

Response to Reviewer zgtb

Additional assembly supervision improves performance beyond repeated training (W3).

We appreciate the concern that the larger synthetic corpus also entails more optimization updates. We added two complementary analyses: a comparison of designer and synthetic examples at the same dataset size, and a one-versus-five-epoch training-duration control.

The same-scale comparison uses 3,873 assembly steps from each source, as specified in Appendix D, p. 21, lines 1120–1127. Table 8 shows:

SupervisionSelection accuracy (%) ↑Translation error (LDU) ↓Rotation error (°) ↓Strict step success (%) ↑
Same-scale Synthetic44.15180.0759.124.44
Designer Supervision48.32162.8257.815.44
Synthetic Full62.38107.6447.9812.33

Human-designed examples are more effective at the same scale, while expanding synthetic experience substantially improves performance. The new training-duration results in Table 9, are:

SupervisionSelection, 1 epoch (%)Selection, 5 epochs (%)Step success, 1 epoch (%)Step success, 5 epochs (%)
Designer only48.3248.415.445.44
Synthetic only62.3862.3812.3312.33
Designer + Synthetic66.5566.8213.6913.87

The small changes suggest limited benefit from repeatedly training on the existing examples in these settings. This experimental result actually drives us to train all the systems with 1 epoch instead of more from the very beginning. We hypothesize that current MLLMs can learn efficiently from the available examples, but remain hungry for more diverse, domain-specific data.

Designer + Synthetic without feedback, and the additional benefit of learned feedback (Q3).

Following the reviewer's suggestion, we explicitly report the results Designer + Synthetic without feedback. Table 8 also includes synthetic-only training with and without feedback:

ConfigurationTranslation error (LDU) ↓Rotation error (°) ↓Strict step success (%) ↑
Synthetic Full107.6447.9812.33
Synthetic Full + Feedback (L)82.5343.0512.96
Designer + Synthetic72.8340.3913.69
Brick-Composer64.9237.2514.23

The combined supervision already accounts for most of the improvement. Learned feedback provides a further reduction in pose errors and a smaller increase in strict step success. However, we describe learned feedback as a complementary gain rather than the primary source of improvement.

These results show that learned feedback provides a complementary benefit even after synthetic supervision is included. Moreover, extending Designer + Synthetic training to five epochs does not close the observed performance gap, supporting the usefulness of feedback-based correction examples.

Overall, we sincerely appreciate the reviewer's careful evaluation and the suggestions that motivated these revisions. We respectfully ask the reviewer to reconsider the weak-reject recommendation and consider raising the score in light of the corrected prompt descriptions, explicit training/evaluation distinction, and added ablation evidence. We hope the revised manuscript makes the contribution more convincing while remaining transparent about its scope and limitations.

Thank you for the detailed response and additional experiments. The correction of the rotation-prompt inconsistency and the clarification that ground-truth pose errors are excluded from test-time feedback address my concerns about the prompt specification and privileged information during evaluation. The additional Designer + Synthetic ablation is also helpful in assessing the contribution of synthetic supervision.

However, after examining new predicted-history rollout results, I have an additional concern about the gap between the demonstrated capability and the paper’s framing. Even on the best object selected separately for each configuration, Brick-Composer achieves correct initial prefixes of only 1 out of 16 steps with Gemma and 5 out of 96 steps with Qwen.

I understand the diagnostic value of evaluating individual steps under ground-truth assembly histories, given concurrent MLLMs may have poor ability to self roll out a whole LEGO assembly (maybe GPT6-Astra could do better? Anyway, I do not require authors to do such experiments in this rebuttal period). I also do not expect the authors to solve reliable long-horizon assembly during the rebuttal period. But my concern is, how this GT-conditioned task can serve as an effective proxy task for a true self rollout, for MLLMs' ability to construct a LEGO from scratch? I want to see more explanations about this, especially when you titled your paper as "MLLMs Construct Everything from Building Blocks".

Reviewer_aqKhOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether multimodal large language models can follow LEGO-style assembly instructions. BC-Bench divides each assembly step into brick selection from a remaining inventory and prediction of the selected brick's absolute translation and rotation. Inputs include rendered before/after instructions, highlighted target components, and views of the current assembly and candidate or selected parts. The core evaluation contains 1,013 steps from 20 held-out objects; human-designed training data contain 3,873 steps from 82 objects.

Brick-Composer combines three sources of supervision: demonstrations from human-designed assemblies, simulator-rendered feedback on erroneous model predictions, and procedurally generated assemblies with feasible attachments, collision filtering, and a compactness preference. The synthetic corpus contains approximately 700 objects and over 40,000 steps. Training uses autoregressive supervised objectives, including correction trajectories; the world-feedback component is not presented as reinforcement learning. The language-model backbone is fine-tuned while the vision encoder and multimodal projector remain frozen.

Critical strengths
  1. Authors introduce a novel benchmark for assembly with diverse building blocks. BC-Bench brings manual understanding, fine-grained part selection, and full 3D pose estimation into a unified evaluation framework, providing a concrete testbed for studying how MLLMs translate visual instructions into assembly actions.

  2. Human designs provide meaningful structures and connection examples, while synthetic data expand local assembly patterns through feasible attachments, collision filtering, and connectivity checks. Their combination offers a scalable approach to overcoming limited demonstration data. Moreover, this work supports the learnability of visual correspondence and spatial placement skills for assembly.

Critical weaknesses
  1. The core test set contains only 20 objects, and an object-level split does not necessarily imply unseen part types or connection mechanisms. Quantifying part and connection overlap, together with evaluations on unfamiliar parts, complex structures, occlusion, and fewer views, would better establish generalization.
  2. Evaluation uses reference assembly contexts and does not adequately test error accumulation or recovery when subsequent decisions depend on the model’s own predictions. Full-object success is 0% for all methods. Therefore, the reported improvements demonstrate better single-step prediction but do not establish reliable sequential assembly, limiting the evidence for practical applicability to autonomous construction.
Questions for authors
  1. Could the authors evaluate closed-loop assembly using states produced by the model’s own previous actions? Reporting successful-prefix lengths, error recovery rates, and full-object completion would help establish whether the single-step improvements translate into sustained assembly capability.
  2. Can the predicted part selections and poses be integrated into a physical robotic-arm assembly system?
3 comments
AuthorsOpenReview ↗

Response to Reviewer aqKh

We sincerely thank the reviewer for the clear-accept recommendation and the strong support for our work. We are especially grateful for your recognition of BC-Bench's value and of the potential of combining human-designed and synthetic assembly experience. Your encouraging assessment means a great deal to us. We also appreciate the thoughtful questions about generalization, accumulated errors, and physical deployment. These questions helped us clarify what the current results establish and add a preliminary evaluation beyond reference-history prediction. All line numbers below refer to the revised manuscript.

Quantifying part overlap and clarifying generalization to unseen structures (W1).

We agree that holding out objects does not, by itself, establish generalization to unseen part types or connection mechanisms. We added explicit part-overlap statistics and clarified the scope of generalization in Appendix A, p. 14, lines 746–755.

Part-type occurrence in the analyzed human-designed splitsNumber of unique part types
Shared between training and evaluation158
Training only322
Evaluation only103
Total583

The 103 evaluation-only types account for 279 of 1,103 evaluation steps (25.29%). However, the synthetic training data include the full part inventories. We therefore explicitly clarify that these counts do not establish that the final model has never encountered those part types. The main generalization claim concerns held-out object designs, assembly-state patterns, and geometric configurations, rather than entirely unseen parts and connection types, which are limited connection types (Studs, Axles, Hinge, Ball, and Fixed) from LEGO.

We also provide the evaluated object detail Table 5, with the dataset description in lines 727–735. Connection-disjoint generalization, controlled occlusion, and fewer-view evaluation remain outside the present experiments; we do not use the part-overlap statistics as evidence that these stronger forms of generalization have been established.

Additional assembly results under accumulated prediction errors (W2, Q1).

We thank the reviewer for suggesting evaluation using states produced by the model's own actions. We added a preliminary rollout diagnostic in which later observations are rendered from previous model placements, without resetting to the reference assembly. The protocol and limitations are described in Appendix D, p. 24, lines 1242–1257, with results in Table 11, lines 1259–1270.

BackboneMethodSelected object lengthReference-history step success (%)Predicted-history step success (%)
Gemma-3-12BDirect Prompting452.220.00
Gemma-3-12BBrick-Composer1618.756.25
Qwen3-VL-8BDirect Prompting454.440.00
Qwen3-VL-8BBrick-Composer9641.6713.54

Brick-Composer achieves correct initial prefixes of 1 step out of 16 for Gemma-3-12B and 5 steps out of 96 for Qwen3-VL-8B (lines 1250–1251). The predicted-history success rates correspond to 1/16 and 13/96 successful steps, respectively. These observations show that some step-wise capability persists under accumulated state errors, while also revealing a substantial decline from reference-history performance.

This experiment uses each method's best object selected under reference-history evaluation, so the rows are not a matched-object comparison or an estimate of full-test-set rollou performance.

We state this explicitly in lines 1246–1253 and the table caption. We also distinguish later successful placements from recovery of the whole assembly state: the renderer can retain intersecting or unsupported parts, and recovery and full-object completion are not evaluated as separate rollout metrics (lines 1253–1257).

AuthorsOpenReview ↗

Response to Reviewer aqKh

Reliable assembly of complex structures remains challenging for current MLLMs (W2, Q1).

We agree with the reviewer's interpretation that improved reference-context predictions do not establish reliable autonomous construction. The short correct prefixes and reduced predicted-history performance reinforce this limitation. We preserve the distinction between within-object step success and complete-object success in Appendix F, p. 26, lines 1402–1403, and p. 27, lines 1433–1438, and clarify the main step-success and best-object definitions in Section 4.2, p. 7, lines 358–363. Our contribution is a diagnostic benchmark and evidence that these local assembly capabilities can be improved through targeted learning; reliable long-horizon assembly remains an open problem.

Integration with robotic assembly systems and remaining sim-to-real challenges (Q2).

The predicted part identity and target translation and rotation can serve as a high-level interface to a robotic assembly system after alignment to the robot's coordinate frame. However, this interface is not itself a validated physical assembly system. Deployment additionally requires part localization, camera and robot calibration, grasp selection, motion and contact control, and recovery from execution failures. Clean multiview renderings and highlighted target parts also simplify the perception problem relative to a physical workspace. While previous works such as BrickGPT and Manual2Skill demonstrate initial progress toward robotic assembly, substantial challenges remain before reliable autonomous assembly can be achieved.

We explicitly discuss these requirements in Appendix G, p. 28, lines 1470–1475. We therefore present physical deployment as a direction enabled by our output representation, rather than as an experimentally demonstrated capability of the current system. We are eager to pursue this direction toward automated assembly in future work.

Thank you again for the clear support and constructive suggestions. We are grateful that the reviewer recognizes the value of this first step toward MLLM-based assembly, and we hope the added analyses and more precise scope make the contribution and its limitations clearer.

Thank you for the additional experiments, especially the predicted-history rollouts, which show how quickly assembly errors can accumulate. I am keeping my rating at 4 because BC-Bench provides a valuable testbed for spatial reasoning, and the results show that training improves step-level assembly performance on held-out designs.

Reviewer_XrRKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces BC-Bench, a benchmark that decomposes multimodal brick assembly into brick selection and 3D pose estimation. It also proposes Brick-Composer, which combines supervision from human-designed assemblies, simulator-generated feedback, and synthetic assembly configurations. For Qwen3-VL-8B, Table 3 reports an increase in selection accuracy from 22.76% to 68.21%, a reduction in translation error from 210.14 to 65.63 LDU, and an increase in strict step-level success from 0.36% to 14.27%. The results primarily support improved single-step, manual-conditioned prediction under reference assembly contexts. As the paper acknowledges, full-object success remains 0%, and the reported 41.63% best-object result measures the fraction of successful steps within one object

Critical strengths

s1: this is an interesting research topic. Combining fine-grained part recognition, coordinate transformations, and assembly geometry exposes limitations that conventional visual question-answering benchmarks may miss.

s2: task and metric design with careful consideration. The benchmark accounts for interchangeable parts, rotational symmetries, changing candidate inventories, and the distinction between strict evaluation and pose snapping.

s3: substantial reported learning gains. Improvements appear across both evaluated model families, suggesting that task-specific supervision can improve assembly-related capabilities.

s4: acknowledge of important limitations.The appendices discuss reference-context evaluation, the additional requirements for physical deployment, and the limitations of the CAD search baseline.

Critical weaknesses

w1: The pose conventions are inconsistent, and the information available through feedback is insufficiently specified. The standard pose-estimation prompt in Appendix B, page 16, lines 811–816 states that each row of the rotation matrix represents a local axis expressed in world coordinates. The feedback prompt on page 17, lines 893–895 instead assigns this interpretation to each column. These conventions generally differ by a transpose and can affect both predictions and rotation-error calculations.

The manuscript therefore contains a definite protocol inconsistency, although the available material does not establish whether the implementation has the same problem. The authors should clarify the conventions used in training, inference, rendering, and evaluation; identify any explicit conversion; and establish whether correcting the prompts changes the comparisons.

w2: The ablations do not sufficiently isolate the contributions of the three learning signals. the experiments establish that the complete training configuration outperforms smaller configurations, but do not adequately distinguish:

  • Benefits from synthetic-data composition versus increased training volume.
  • The independent contribution of feedback after synthetic data are included.
  • Benefits from feedback content versus additional inference rounds and inputs. Two focused comparisons would be particularly informative: a comparison of data sources under matched training budgets, and Human + Synthetic versus Human + Synthetic + Feedback.

w3: The positioning relative to closely related work needs correction and stronger support. Table 1 marks Wang et al.’s Translating a Visual LEGO Manual to a Machine-Executable Plan (ECCV 2022) as not supporting “Pose Est. w/ Rotation.” However, that work explicitly predicts component translations and rotations from manual images, and MEPNet includes pose inference and orientation refinement. The current characterization is therefore inaccurate. BC-Bench can instead emphasize its particular combination of MLLM evaluation, diverse remaining-part inventories, candidate selection, and joint step-level metrics. Differences in input assumptions, part coverage, and permitted rotations should be stated explicitly.

Questions for authors
  1. Which rotation convention was actually used throughout the experiments?

  2. Does harmonizing the prompts change any reported results? During evaluation, does World Feedback include ground-truth-relative error magnitudes, or use ground truth to decide when feedback is triggered?

  3. How are the reported percentages aggregated from the stated 1,013 evaluation steps? Can per-object counts and denominators be provided?

  4. Does feedback retain an independent benefit when synthetic data, training updates, and inference budgets are controlled?

  5. How should Table 1 and the novelty claims be revised in light of MEPNet’s translation and rotation prediction capabilities?

9 comments
AuthorsOpenReview ↗

Response to Reviewer XrRK

We sincerely thank the reviewer for the detailed feedback and constructive comments, and for recognizing the value of our problem setting, metric design, and learning framework. We especially appreciate the careful examination of our prompts and the identification of the rotation-convention inconsistency. Your comments have helped us improve both the clarity of the evaluation protocol and the evidence supporting our learning signals. Below, we address each concern and point to the corresponding revisions. Unless explicitly labeled as original, all line numbers refer to the revised manuscript.

Correcting the rotation-convention description and updated results show consistent findings (W1, Q1, Q2).

We thank the reviewer for identifying this inconsistency. The original feedback prompt described the local-axis directions using “columns” (original Appendix B, p. 17, lines 893–895), whereas the standard pose-estimation prompt used “rows” (original p. 16, lines 811–816). We agree that these descriptions are not interchangeable and that the inconsistency should be corrected rather than dismissed as a formatting issue despite being small. Both prompt descriptions now consistently specify rows as the local axes expressed in world coordinates: Appendix B, p. 16, lines 826–831, and p. 17, lines 905–912.

The inconsistency was present in the affected feedback (Learning) prompts, rather than being only a typographical error in the appendix. We corrected the affected training and evaluation prompts, retrained the affected configurations, and reran evaluation. The updated overall results appear in Table 3 and are reproduced below. Parentheses give the absolute change from the previously reported result; pp denotes percentage points.

BackboneConfigurationTranslation error (LDU) ↓Rotation error (°) ↓Strict step success (%) ↑
Gemma-3-12BWorld Feedback (L)172.15 (change: <2 LDU)51.32 (change: <1°)1.99 (unchanged)
Gemma-3-12BBrick-Composer122.31 (change: <2 LDU)45.11 (change:<1°)4.35 (unchanged)
Qwen3-VL-8BWorld Feedback (L)137.51 (change: <1 LDU)53.14 (change: <1°)6.26 (change <0.1 pp)
Qwen3-VL-8BBrick-Composer64.92 (change: <1 LDU)37.25 (change: <1°)14.23 (change <0.1 pp)

For these configurations, the absolute changes in overall strict step success are less than 0.1 percentage points. The main comparisons remain unchanged: learned feedback improves over designer supervision, and Brick-Composer provides the strongest performance among the main learning configurations. The revised full-framework results are also reflected in Table 4, Table 8, and Table 13.

We believe the limited impact could be explained by the localized nature of the inconsistency: it concerned only a small portion of an approximately 1,500-token instructional prompt describing rows versus columns (3 words different), while the supervised visual inputs and pose examples provided consistent task information. The model may therefore have relied more strongly on these supervision signals and visual relationships than on the conflicting instructional wording.

Relative pose-error feedback is supervision used only during training (W1, Q2).

We incorporate relative pose-error feedback into training trajectories to help the model relate visual discrepancies to spatial corrections. These ground-truth-relative error values are not included in any evaluation prompts, including inference-time feedback prompts. At inference, the feedback consists of the rendered predicted assembly state alongside the visual instructions, without ground-truth translation or rotation errors. We have made this distinction explicit in Appendix B, p. 18, lines 965–970.

At evaluation, feedback is provided at every step, regardless of correctness, for exactly one revision round using the rendered predicted assembly state. Ground-truth correctness therefore does not determine whether feedback is triggered or how many revision rounds are performed.

AuthorsOpenReview ↗

Response to Reviewer XrRK

Clarifying aggregation, evaluation denominators, and best-object performance (Q3).

We thank the reviewer for requesting a clearer account of the evaluation set. This helped us find a typo in the original manuscript. We corrected the previously reported evaluation-set size: the core evaluation contains 1,103 steps, instead of 1,013 steps, as stated in Appendix A, p. 14, lines 727–735. Table 5 now lists all 21 objects and their individual step counts.

Our overall step-success rate is computed by avreaging successful steps across 1,103 steps, and we understand that an object-balanced success rate could be different. Thus, we additionally report an object-balanced rate that first computes the successful-step fraction within each object and then averages those fractions equally across the evaluation objects. The formulas and interpretation are given in Appendix D, p. 23, lines 1236–1241, and the full results appear in Table 10.

BackboneConfigurationPooled step success (%)Object-balanced step success (%)
Gemma-3-12BDirect Prompting0.090.11
Gemma-3-12BDesigner Supervision0.730.73
Gemma-3-12BWorld Feedback (L)1.992.01
Gemma-3-12BBrick-Composer4.355.13
Qwen3-VL-8BDirect Prompting0.360.38
Qwen3-VL-8BDesigner Supervision5.445.34
Qwen3-VL-8BWorld Feedback (L)6.266.43
Qwen3-VL-8BBrick-Composer14.2314.54

We show that the results computed with different metrics generally aligns, and the method ranking is unchanged under object-balanced aggregation. We also clarified that best-object performance is selected separately for each metric, rather than reporting all metrics from one common object (Section 4.2, p. 7, lines 358–363).

Human-designed data are more effective at the same scale, while synthetic data provide scalable additional supervision (W2).

We agree that comparing the small designer dataset with the full synthetic dataset alone cannot distinguish data-source quality from scale. We therefore added a same-scale comparison using 3,873 assembly steps from each source. The sampling setup is described in Appendix D, p. 21, lines 1120–1127, and the results appear in Table 8.

SupervisionSelection accuracy (%) ↑Translation error (LDU) ↓Rotation error (°) ↓Strict step success (%) ↑
Same-scale Synthetic44.15180.0759.124.44
Designer Supervision48.32162.8257.815.44
Synthetic Full62.38107.6447.9812.33

At the same number of assembly examples, human-designed supervision performs better on all four metrics. Increasing the synthetic dataset to over 40K assembly steps substantially improves performance. We therefore interpret synthetic experience as a scalable complement to human-designed examples, rather than claiming that synthetic examples are intrinsically superior. This interpretation is discussed in p. 23, lines 1191–1198.

Additional training epochs provide only limited gains on the existing datasets (W2, Q4).

We examined that simply training with more epochs on existing examples cannot provide extra help, which explains why we choose epoch = 1 for our standard setting. To show this finding, we added a one-versus-five-epoch comparison result in Table 9, with analysis in lines 1208–1214.

SupervisionSelection, 1 epoch (%)Selection, 5 epochs (%)Step success, 1 epoch (%)Step success, 5 epochs (%)
Designer only48.3248.415.445.44
Synthetic only62.3862.3812.3312.33
Designer + Synthetic66.5566.8213.6913.87

Five epochs of designer-only training yield only a minimal improvement in selection accuracy and no additional successful steps. For Designer + Synthetic, the gains are similarly small. These results suggest that repeatedly optimizing on the same data provides limited benefit in the tested settings. We hypothesize that current MLLMs can learn efficiently from the available examples, but remain hungry for more diverse, domain-specific data.

AuthorsOpenReview ↗

Response to Reviewer XrRK

Correcting the comparison with MEPNet and clarifying the task distinction (W3, Q5).

We thank the reviewer for pointing out the inaccurate characterization. We agree that MEPNet predicts both component translations and rotations, and our original intention was that MEPNet makes the rotation prediction task a classification task with limited rotations, instead of freeform prediction.We have corrected Table 1, where the revised column explicitly concerns general 3D rotation. MEPNet receives partial support because its LEGO setting restricts orientations to four rotations about the vertical axis; this does not mean that MEPNet lacks rotation prediction.

The revised table also now further distinguishes diverse part selection, manual-based tasks, and MLLM evaluation. Here, “Diverse Part Selection” means the task require selecting the next required component from an available inventory, rather than receiving the component to place as an input. “MLLM Eval.” refers to evaluating MLLMs on the complete task defined by each benchmark; partial support includes their use for an intermediate stage. BC-Bench's contribution is the combination of remaining-inventory selection, numerical pose prediction for diverse bricks, and joint step-level evaluation of MLLMs.

Overall, we sincerely appreciate the reviewer's time and the concrete suggestions, which have led to improvements in the paper and additional evidence. In light of the corrected comparison and prompts, the clarified evaluation protocol, and the new ablations and aggregation analysis, we respectfully ask the reviewer to reconsider the weak-reject recommendation and consider raising the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.