acceptodds
Under review as a conference paper at ICLR 2027

VISTA-Brick: Visually Faithful and Physically Stable Brick Program Generation

Abstract

Reproducing an object's geometry does not guarantee a physically stable assembly. We present VISTA-Brick, a framework for generating visually faithful and physically stable brick programs from multi-view RGB-D observations and text. To support this task, we construct a multimodal extension of StableText2Brick that pairs source-object observations with reference brick programs. VISTA-Brick combines multimodal supervised fine-tuning with online GRPO, using program-level geometric agreement, structural cues, and static-analysis feedback to evaluate the policy's own predictions. During post-training, the physical solver serves as an evaluator rather than repairing sampled programs. We evaluate the learned policy both through direct generation and with optional budgeted physical search. Compared with the supervised baseline, post-training improves program legality and static stability without inference-time correction, while modestly improving geometric fidelity. Under the same physical search procedure and sampling-budget cap, post-trained policies also achieve higher joint geometric–physical success with lower average search cost. These results support learning structural feasibility alongside geometric fidelity, with physical verification complementing rather than substituting for the learned generation capability.

Primary area:
generative models
Keywords:
Reinforce Learning, 3d Generation, Multimodal

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Y3teOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents VISTA-Brick, a framework for generating geometrically faithful and physically stable brick programs from multi-view RGB-D observations and text descriptions. The authors extend StableText2Brick with rendered multi-view observations and adopt a two-stage training pipeline consisting of multimodal supervised fine-tuning and online GRPO. The GRPO stage combines geometric agreement, structural cues, and static-analysis feedback to improve program generation. The framework supports both direct generation and optional verifier-assisted physical search. Experiments on 250 objects show improvements in program validity, static stability, joint geometric-physical success, and search efficiency over the supervised baseline. Additional ablations examine visual conditioning, reward design, and inference-time search budgets.

Critical strengths
  1. A meaningful problem setting combining geometric fidelity and physical stability. The paper addresses the challenge of generating brick assemblies that both resemble an observed object and satisfy physical constraints. Conditioning on multi-view RGB-D observations provides instance-specific geometric information beyond text-only generation. The comparison between text-only and multimodal inputs supports the importance of visual conditioning for geometric fidelity and joint success.

  2. Comprehensive experiments separating learned generation from inference-time verification. The paper evaluates both direct generation and verifier-assisted physical search, allowing the improvements from GRPO to be distinguished from those obtained through external correction. The reported gains in direct stability and joint success, together with budget sensitivity experiments, reward ablations, and search-cost analysis, provide useful evidence for the effectiveness of the proposed framework.

Critical weaknesses
  1. The contributions of individual reward components and the sources of efficiency improvements are not fully characterized. Although the reward ablation in Table 3 shows consistent improvements from full GRPO, the geometry-only and full configurations differ in their reward formulations, making it difficult to isolate the contributions of static-analysis feedback and structural proxies. Furthermore, Appendix Table 9 shows that, under verifier-assisted generation, GRPO reduces the average number of returned bricks from 145.32 to 125.04. While this may indicate more efficient assembly generation, it could also reflect changes in structural complexity or geometric detail. The paper does not sufficiently analyze whether the improvements in stability and search efficiency result from better structural arrangements, shorter programs, or a combination of both.

  2. The evaluation of geometric fidelity and robustness remains limited. The geometric reward and evaluation metrics rely on the occupied geometry of a single paired reference brick program, although the paper acknowledges that a source object may admit multiple valid brick realizations. While different brick decompositions with identical occupied geometry are evaluated equivalently, valid realizations with different occupancy patterns may receive lower scores even when they preserve the intended source-object shape. Furthermore, the evaluation relies on synthetic observations rendered from ShapeNet objects using fixed camera configurations and clean depth and foreground-mask inputs. Without additional evaluation under varying viewpoints or imperfect observations, the robustness of the proposed framework beyond this controlled setting remains uncertain.

Questions for authors
  1. Table 3 compares geometry-only and full GRPO, but the two settings differ in several aspects of their reward formulations. Could the authors further isolate the effects of static-analysis feedback, structural proxies, and geometric gating? It would also be helpful to see how reward, direct-generation stability, and joint success change over the course of training.

  2. I noticed that the average number of returned bricks decreases from 145.32 to 125.04 after GRPO in Appendix Table 9. Does this mainly reflect more compact assemblies, or could some geometric details be lost? A comparison with the reference brick counts, along with stability results for valid programs and objects of different complexities, would help clarify this. I am also curious why the average verification time increases from 25.45 to 33.51 seconds despite fewer verifier calls.

  3. The reward function uses several fixed parameters, including the geometric threshold τ=0.40\tau = 0.40 and the geometry–physics weight λ=0.70\lambda = 0.70. How sensitive are the results to these choices? In particular, I wonder whether the discontinuity at the geometric threshold affects training stability and how performance changes when these parameters are varied.

  4. Since a single source object may have multiple valid brick realizations, I wonder whether evaluating the generated structures directly against the original source mesh would provide a more complete picture of geometric fidelity. It would also be useful to know how well the method performs under small changes in camera viewpoints or perturbations to the depth maps and foreground masks, rather than only under the fixed synthetic observation setting.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Y3te: A. Reward decomposition and training trajectory

Thank you for distinguishing the overall post-training benefit from the contribution of each feedback component. We agree that the original geometry-only versus Full comparison does not isolate every change.

Q1 and related W1: Reward Ablation

Response. We compare Full with three reward ablations: T1 removes only the static term S without renormalization; T2 substitutes STABLE's reward recipe; T3 (Geometry+Gate) removes all physical feedback P while retaining the gate and geometry coefficient. All use the same multimodal policy, SFT start, data/order, training seed, and effective prompts per update, and are evaluated at matched steps on the same 250 objects under the same sampled-Direct settings.

PolicyValid %Stable %Stable/Valid %[email protected] %Source F@2%SourceJoint(0.3) %
Full52.440.477.1038.00.452934.8
T1 without static term54.034.864.4432.40.450129.6
T2 STABLE recipe53.234.063.9132.40.456128.8
T3 Geometry + Gate, all P removed54.834.062.0432.40.450930.4

F@2%. We sample 32,768 points per surface from the exposed brick-union boundary and the source mesh, using fixed source-defined coordinates. With tolerance �0��0�, where DD is the source bounding-box diagonal, precision pp is the fraction of predicted points within �3��3� of a source sample, and recall rr reverses the direction. F@2%=2pr/(p+r)F@2\%=2pr/(p+r), or zero when p+r=0p+r=0. The mean includes all 250 objects: parseable invalid or non-EOS outputs retain their geometry scores; unavailable geometry scores zero.

SourceJoint(t). This is the percentage of all 250 objects with completed (EOS), valid, statically stable outputs and F@2%>=tF@2\%>=t. Thus, SourceJoint(0.3) requires F@2% of at least 0.3; 2% is the point-matching distance tolerance, whereas 0.3 is the per-object F-score threshold. Incomplete, invalid, unstable, or static-audit-failed outputs do not count as successes.

The clearest distinction is in stability among valid outputs. Validity ranges from 52.4% to 54.8% across the four policies, whereas Stable/Valid reaches 77.10% for Full compared with 62.04%-64.44% for the ablations. Full also achieves the highest overall stability, at 40.4%. The improvement therefore lies primarily in the physical quality of legal assemblies, rather than in producing more legal programs.

The sequence T3 → T1 → Full separates the feedback components while keeping the gate fixed. Adding C/I/J to T3 increases stability from 34.0% to 34.8%, with [email protected] unchanged at 32.4%. Adding S on top of these proxies then raises stability to 40.4% and [email protected] to 38.0%. This progression shows a small observed stability gain from proxies alone and a clearer joint benefit when explicit static feedback is included. The gate is unchanged throughout, so the latter improvement reflects feedback beyond the geometric threshold.

The source-mesh columns connect this stability gain to reconstruction quality. T2 obtains the highest mean source F@2% (0.4561, versus 0.4529 for Full), but Full achieves the highest SourceJoint(0.3), at 34.8%, compared with 28.8% for T2, 29.6% for T1, and 30.4% for T3. Thus, the advantage of Full is that more outputs are both geometrically acceptable and stable, even when another reward recipe achieves a higher average geometric score. This is the joint reconstruction objective that motivates combining static and structural feedback.

Q1: How do stability and joint success evolve during training?

Response. The saved training logs and original greedy direct evaluations provide the following trajectory on the same 250 test objects. Training reward is averaged over the 500 steps ending at each checkpoint; it is not a test-set score.

Full training stepMean training rewardDirect Stable %Direct [email protected] %
SFT startN/A44.442.8
5000.039245.643.6
1,0000.095649.648.8
3,5000.225154.052.8
4,0000.271653.251.6
6,0000.242459.657.2
8,0000.240662.860.8
10,0000.237164.863.2

At 10K steps, GRPO adds 20.4 percentage points to both stability and [email protected] over SFT, a net 51 additional joint successes on 250 objects. The matching gains indicate that improved stability also satisfies the geometric criterion, rather than merely favoring stable but mismatched shapes. Since these are direct outputs, the benefit is learned by the policy rather than supplied by search.

AuthorsOpenReview ↗

Response to Reviewer Y3te: B. Sensitivity complexity and robustness

Q3 and related W1: Threshold discontinuity and parameter sensitivity

Response. Thank you for pointing out the missing sensitivity analysis. The G = 0.4 threshold is an empirical safeguard against geometrically poor candidates receiving substantial physical reward. The 0.7/0.3 geometry/physics split aims to retain the SFT policy's instance fidelity while improving physical behavior. This is a design rationale, not a validated optimum or an argument that the two reward scales are directly comparable.

A retraining-based threshold/weight sweep is not yet complete. Regarding the threshold discontinuity, GRPO treats the reward as an external scalar signal and updates the policy through token log-probabilities; it does not differentiate through the geometric score or the gate. The hard threshold therefore does not require a differentiable reward function. Nevertheless, crossing the threshold produces a discrete reward change that can alter the relative advantages of candidates within a group, particularly when candidates fall on both sides of the threshold. Its practical effect depends on the distribution of candidate geometry scores near the boundary. Our component ablations hold the threshold fixed and isolate the contribution of physical feedback under this configuration; sensitivity to the threshold itself remains a separate empirical question. Given the cost of online training within the rebuttal period, we prioritize matched component evaluations and will acknowledge the remaining sensitivity study.

Q2 and W1: Fewer bricks: better arrangements or missing detail?

Response. We agree that shorter output does not automatically imply lossless compaction. GT assemblies average 131.24 bricks, SFT plus search 145.32, and Full-10K plus search 125.04. In the high-complexity group, stability increases from 59.04% to 83.13%, while Visual decreases from 0.7257 to 0.7150.

Independent source-mesh evaluation shows the same tradeoff: SFT/Full search mean F@2% is 0.4522/0.4397, while SourceJoint(0.3) improves from 75.6% to 80.0%. F@2% measures surfaces at a 2%-diagonal tolerance, independent of brick decomposition. Thus, the evidence supports more frequent faithful-and-stable reconstruction, but not preservation of every geometric detail or improvement solely through arrangement.

Q2: Why does verification time increase despite fewer calls?

Response. Thank you for highlighting this interesting discrepancy; we examined the verification-time distribution in detail to understand it. The mean is affected by a long tail. From SFT to Full with search, median verification time decreases from 2.07 to 1.61 seconds, while the 95th percentile increases from 119.04 to 220.76 seconds. Mean time therefore rises from 25.45 to 33.51 seconds, despite mean calls falling from 5.336 to 3.412. We will report the distribution rather than infer lower solver time from fewer calls or bricks.

Q4 and W2: Alternative realizations and imperfect observations

Response. Thank you for these complementary suggestions. Source-mesh evaluation compares exposed union surfaces rather than one exact brick layout, using source-defined coordinates and no GT fitting. On the matched direct outputs, SFT-to-Full SourceJoint at F@2% thresholds 0.2/0.3/0.4 increases from 41.6/36.4/26.8% to 61.6/50.0/36.0%. This supports the joint objective under a different geometric target.

We also evaluated the same full policy under observation perturbations:

ConditionStable %[email protected] %
Canonical clean RGB-D64.863.2
Camera yaw -5 degrees46.041.6
Camera yaw +5 degrees43.240.0
Depth noise 3%58.856.8
Depth missing 30%56.855.2
Mask dilation 3 pixels54.452.0

Noise is measured relative to the encoded depth range, not metric distance. With 3% depth noise, Visual remains 0.7893 versus clean 0.7916, but Joint drops 6.4 points. At the tested strengths, camera shifts have a larger effect than depth noise or missing depth. This identifies camera configuration as a priority for improvement through explicit view modeling, calibration information, and broader training-view coverage. The joint metric reveals failures that average geometry alone can obscure.

Reviewer_bLDAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

VISTA-Brick generates brick programs from four-view RGB-D observations and text using a Qwen3-VL-2B model trained with SFT and geometry-and-physics GRPO. It supports direct generation and optional physical search. The authors report improved unassisted stability and joint geometric–physical success, alongside reduced search cost.

Critical strengths
  1. The evaluation is clearly documented, including checks for train–test overlap, parsing and completion rules, and how failed outputs are counted. The appendix also explains how diagnostic examples were selected.

  2. Reporting direct and search-assisted results separately helps assess the policy’s unassisted performance and the benefit of search. Table 9 provides a useful breakdown of verifier calls, regenerations and rejected lines.

Critical weaknesses
  1. Visual fidelity is measured against one reference brick assembly in a fixed grid. Sensitivity to alignment and alternative valid realizations remains unclear; source-mesh comparisons and aligned scores would strengthen the evaluation.

  2. STABLE receives mesh-sampled points instead of its native target-brick occupancy, while TreeSBA uses different training data and fewer brick types. Native-input STABLE and RGB-D fusion followed by Mesh2Brick would provide more informative comparisons.

  3. The methodological contribution appears incremental. STABLE already uses SFT and GRPO with geometric and structural rewards, while PVPO combines geometry with physical-validity feedback. VISTA-Brick adds RGB-D conditioning, a rendered dataset and modified rewards. However, the ablations neither isolate the solver’s contribution nor compare existing reward recipes under the same multimodal setup, leaving the advance beyond adapting prior approaches unclear.

  4. The visual ablation removes all images together, leaving the contributions of RGB, depth and multiple views unclear. Testing these separately, including missing or perturbed views, would help assess robustness beyond clean canonical renders.

Questions for authors

See weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer bLDA: A. Independent geometry and baseline inputs

Thank you for recommending source-mesh evaluation. It directly addresses whether we recover the observed object rather than only one paired brick realization.

Q1/W1: Single-reference metrics and alignment

Response. We add an evaluation against the source-object meshes. For each object, we sample 32,768 points from the source mesh and 32,768 points from the exposed surface of the predicted brick union, using surface-area-weighted sampling in both cases.

F@2%. Let DD be the source-mesh bounding-box diagonal and �1��1�. Precision pp is the fraction of predicted surface samples whose nearest source sample is within �3��3�; recall rr is the fraction of source samples whose nearest predicted sample is within �5��5�. Their harmonic mean is F@2%=2pr/(p+r)F@2\%=2pr/(p+r), with zero assigned when p+r=0p+r=0. Precision measures surface agreement, while recall measures source-surface coverage. The score lies in [0,1] and uses fixed source-defined coordinates rather than fitting predictions to the reference.

SourceJoint(t) is the percentage of all 250 objects whose outputs are completed (EOS for autoregressive outputs), valid, statically stable, and satisfy F@2%≥tF@2\% ≥ t. Equivalently, it is 100 times the number of outputs meeting all four conditions divided by 250. SourceJoint(0.3) therefore requires F@2% of at least 0.3: 2% is a distance tolerance for point matching, whereas 0.3 is a per-object F-score threshold. Incomplete, invalid, unstable, or static-audit-failed outputs remain in the denominator but do not count as successes.

Method / inferenceMean F@2%SourceJoint(0.2)SourceJoint(0.3)SourceJoint(0.4)
RGB-D SFT / Direct0.453941.6%36.4%26.8%
Full-GRPO / Direct0.438061.6%50.0%36.0%
STABLE, RGB-D points / no search0.551942.0%40.0%33.2%
RGB-D fusion + Mesh2Brick0.490764.4%58.4%48.0%
RGB-D SFT / 32K search0.452282.0%75.6%57.6%
Full-GRPO / 32K search0.439792.4%80.0%60.8%

Under this independent target, the controlled SFT-to-GRPO comparison improves joint success in both Direct and Search. STABLE has higher mean F@2% while full plus search has higher SourceJoint at all three displayed thresholds. We examine supplementary F@2% thresholds 0.2/0.3/0.4 mainly to show the tradeoff from looser to stricter fidelity requirements. GT brick approximations themselves have mean F@2% 0.4485, which contextualizes this range.

A bounded-alignment diagnostic allows yaw within +/-5 degrees and translation within one grid unit, without scaling or mirroring. SFT/Full F@2% becomes 0.4699/0.4594, preserving the mean-score ordering. Fixed coordinates remain primary. Mean F@2% includes all 250 objects. Parseable invalid or non-EOS outputs retain their geometric scores; missing or unparseable geometry scores zero. Static-audit failure does not erase an available geometric score, but prevents SourceJoint success.

Q2/W2: Native and adapted STABLE and RGB-D fusion

Response. We agree that STABLE should first be recognized in its intended setting. Native target-brick occupancy yields excellent geometric fidelity. However, our task starts from source-object observations. So we additionally test reconstructed input, explicitly labeling this as an adaptation rather than STABLE's native task.

Method / inputInferenceStable %VisualVoxel IoU[email protected] %
STABLE / target-brick occupancy(not ranked)Native, no search66.00.92720.877266.0
STABLE / source-mesh occupancyAdapted, no search58.00.64540.494041.6
STABLE / RGB-D-recovered pointsAdapted, no search42.40.61710.421126.4
RGB-D fusion + Mesh2BrickPhysical conversion65.20.60680.459340.0
VISTA-Brick / RGB-D + textDirect64.80.79160.673363.2
VISTA-Brick / RGB-D + text32K search94.00.76480.636787.2

Privileged geometry references are not ranked. Target occupancy and the complete source mesh provide more geometry than four views. RGB-D fusion uses calibration and depth-range information, without GT occupancy or source-mesh completion. Together, these comparisons recognize native STABLE's geometric strength, while locating our benefit in successful stable reconstruction from observations.

AuthorsOpenReview ↗

Response to Reviewer bLDA: B. Reward and observation ablations

Q3/W3: Does static feedback contribute beyond structural proxies?

Response. Thank you for recommending a more specific test. We compare Full with three reward ablations in the same multimodal policy: T1 removes only S, leaving 0.25C + 0.25I + 0.20J without renormalization; T2 substitutes STABLE's reward recipe, rather than retraining its model; T3 (Geometry+Gate) removes all P while retaining the gate, validity treatment, and geometry coefficient. T1 tests the static term beyond proxies, T2 compares reward recipes, and T3 tests the whole physical term with the gate fixed.

Full and T1/T2/T3 share the SFT starting checkpoint, data/order, training seed, and effective prompts per update. All are evaluated at matched steps on the same 250 objects with the same decoding seed setting, temperature 0.8, top-p 0.95, top-k 0, an 8,192-new-token cap, and no search. These are sampled direct results, not the archived greedy trajectory. Final static verification is identical. Bold marks the best value in each column.

PolicyValid %Stable %Stable/Valid %VisualVoxel IoU[email protected] %
Full52.440.477.100.75730.624738.0
T1 without static term54.034.864.440.75160.618432.4
T2 STABLE recipe53.234.063.910.76280.631632.4
T3 Geometry + Gate, all P removed54.834.062.040.76370.632532.4
PolicySource F@2%SourceJoint(0.2) %SourceJoint(0.3) %SourceJoint(0.4) %
Full0.452939.634.824.4
T1 without static term0.450134.029.623.2
T2 STABLE recipe0.456133.228.820.4
T3 Geometry + Gate, all P removed0.450933.230.422.4

Static feedback beyond proxies (Full vs. T1). Adding S to the unchanged C/I/J reward raises stability from 34.8% to 40.4%, [email protected] from 32.4% to 38.0%, and SourceJoint(0.3) from 29.6% to 34.8%. This supports an incremental contribution of explicit static feedback beyond structural proxies, rather than a gain explained by generating more legal programs.

Overall physical feedback (Full vs. T3). With the gate and geometry coefficient fixed, adding the full physical term improves stability by 6.4 points, [email protected] by 5.6 points, and SourceJoint(0.3) by 4.4 points. The benefit is therefore more frequent jointly faithful and stable outputs, rather than uniformly higher geometric similarity.

Reward formulation (Full vs. T2). Under the same multimodal policy, Full improves stability and [email protected] by 6.4 and 5.6 points, respectively, and SourceJoint(0.3) by 6.0 points. This favors our reward formulation for the joint reconstruction objective in this setting.

Proxies without static feedback (T1 vs. T3). Adding C/I/J alone slightly raises metric by 0.8 points (two objects). Thus, proxies alone show a small observed gain in stability.

Q4/W4: What do depth and multiple views contribute?

Response. Thank you for suggesting finer observation tests. We retain text and the same 10K policy while changing its test-time observations:

ObservationsStable %VisualVoxel IoU[email protected] %
Four-view RGB-D64.80.79160.673363.2
Four-view RGB60.80.72270.595152.8
Two-view RGB-D61.20.77460.649458.8
One-view RGB-D64.40.64890.512843.6

Relative to four-view RGB-D input, removing depth and retaining only one RGB-D view reduce [email protected] by 10.4 and 19.6 percentage points, respectively. These results indicate that the policy makes use of the visual observations.

We also jointly re-render RGB images, depth maps, and masks under camera yaw offsets not provided to the policy. [email protected] falls to 41.6% and 40.0% at −5° and +5°, and to 20.8% and 27.2% at −10° and +10°, respectively. Thus, the policy retains useful reconstruction capability without depth or with fewer views, whereas unreported camera shifts cause greater degradation, consistent with a mismatch between the input views and the fixed four-view configuration used for training.

AuthorsOpenReview ↗

Response to Reviewer bLDA: C. Further results on robustness to camera-view shifts

Thank you for highlighting the limitations of training and evaluating with fixed camera views. Our earlier response showed that even a 5° shift can reduce joint reconstruction success. We have since tested whether training with a broader range of views can reduce this sensitivity.

Starting from the same Full-10K checkpoint, we fine-tuned two models for 1,000 SFT updates. The control used only canonical views; in the augmented condition, half of the training observations were re-rendered with camera yaw offsets within ±10°. Neither model received camera poses. We evaluated both without search on the same 250 objects.

Evaluation viewsCanonical-only training: [email protected]View-augmented training: [email protected]
Canonical44.4%48.4%
Yaw −5°25.6%40.4%
Yaw +5°24.4%44.0%

The gain was larger on the shifted views than on the canonical views: 17.2 percentage points averaged over the two shifts, compared with 4.0 points on canonical views. As a result, the average drop from canonical to shifted views narrowed from 19.4 to 6.2 points. This paired comparison suggests that training-view coverage is one practical way to reduce the sensitivity identified in your review, without changing the model architecture or requiring camera poses at inference.

We hope this result helps address your question. If any aspect of this experiment or our earlier responses remains unclear, we would be happy to clarify.

AuthorsOpenReview ↗

Thank you, and looking forward to your thoughts

Thank you again for your careful review. Your question about relying on a single reference assembly prompted us to evaluate the generated structures against the original source meshes. Your suggestion to isolate static feedback also led to a matched ablation: with the structural proxies held fixed, adding the static term raised [email protected] from 32.4% to 38.0% under direct generation.

As the discussion draws to a close, we would be grateful to hear whether these analyses address your concerns or whether there is a point you would like us to clarify.

Reviewer_1mmfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VISTA-Brick, a framework for generating visually faithful and physically stable brick programs from multi-view RGB-D data and text. The framework combines supervised fine-tuning, which maps multi-view RGB-D observations and text to brick programs, with online GRPO, which evaluates raw programs using geometric agreement, structural cues, and static-analysis feedback. During inference, VISTA-Brick supports direct generation and optional budgeted physical search. To support observation-conditioned brick program generation, the authors construct a multi-view RGB-D extension of the StableText2Brick dataset.

Critical strengths
  • The paper extends text-to-brick generation to an observation-conditioned setting.
  • The results show that the proposed method produces better qualitative results and faithful alignment with input conditions.
Critical weaknesses
  • The paper's novelty is limited. It extends text-to-brick generation to a multimodal setting by incorporating RGB-D observations. The proposed method integrates existing components, including off-the-shelf GRPO and a physical analyzer inherited from BrickGPT. Moreover, the key technical difference between VISTA-Brick and STABLE is not sufficiently clarified.
  • Although the method outperforms previous approaches, the paper offers limited new insights. Moreover, the choice of GRPO for post-training is not adequately motivated or justified.
  • Equations (7)–(9) contain many hyperparameters, but it is unclear how they were chosen.
  • The comparison with BrickGPT is limited to standard greedy sampling. Does BrickGPT support alternative decoding strategies, such as beam search or rejection sampling, and could these strategies reduce autoregressive errors?
  • The qualitative results in Figure 4 do not clearly show how the proposed method outperforms the Hunyuan model.
Questions for authors
  • What is the core technical novelty compared to previous work?
  • What are the key insights of the paper?
  • How are the hyperparameters in Equations (7)–(9) chosen?
  • Does BrickGPT support alternative decoding strategies, such as beam search or rejection sampling, and could these strategies reduce autoregressive errors?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 1mmf : A. Contribution and task motivation

We thank the reviewer for the careful reading and constructive questions. Our goal is to infer a discrete brick approximation of an observed object that preserves its instance-specific geometry and satisfies the adopted static-stability criteria. We clarify below how this task and our training feedback differ from prior work.

Q1/W1: What distinguishes VISTA-Brick from STABLE and PVPO?

Response. STABLE studies rollback-free generation conditioned on complete occupied-voxel coordinates of the target brick structure. Its training reward combines collision, shape, interlocking, and connectivity terms. VISTA-Brick infers the brick approximation from source-object observations and text. Its physical feedback includes a solver-based static term that directly evaluates sampled assemblies, complementing structural proxies.

PVPO studies text-conditioned generation using rewards based on the fraction of valid bricks and voxel-Chamfer alignment. It also uses Gurobi for stability evaluation and regeneration. Our contribution concerns the integration of static feedback into observation-conditioned post-training, alongside geometric and structural feedback.

Specifically, our physical score is P=0.30S+0.25C+0.25I+0.20JP = 0.30S + 0.25C + 0.25I + 0.20J, where SS is the static-analysis term, and CC, II, and JJ measure ground reachability, multiple lower-layer supports, and offset seams, respectively. These structural proxies do not independently certify static stability. The geometric score GG combines voxel, silhouette, and depth agreement with the paired reference brick assembly.

We examine the feedback through three matched reward ablations: T1 removes only SS, without renormalizing the remaining terms; T2 adapts STABLE's reward to our policy; and T3 (Geometry + Gate) removes the entire physical term PP while retaining the gate and geometry coefficient. At matched checkpoints under sampled direct generation, Full achieves 38.0% [email protected], compared with 32.4% for each of T1, T2, and T3.

Q2/W2: Why this task, and what is the central insight?

Response. A caption such as “a chair with a backrest” leaves the backrest height, seat proportions, and leg layout unspecified. A low IoU with a single reference may therefore reflect a different instance that is still consistent with the caption. Our objective is to reconstruct the observed instance as a stable brick assembly. RGB-D observations provide instance-specific geometric evidence without requiring complete target-brick geometry at inference.

InputPolicyInferenceStable %[email protected] %
Text-onlyGRPODirect84.021.6
Text + RGB-DSFTDirect44.442.8
Text + RGB-DGRPODirect64.863.2

The fixed-policy modality ablation illustrates why stability alone is insufficient. Using the same GRPO checkpoint, text-only input yields 84.0% stability and 21.6% [email protected], whereas text plus RGB-D yields 64.8% stability and 63.2% [email protected]. Visual observations therefore increase the frequency of outputs that satisfy both the geometric and static criteria, even though the stability rate alone decreases.

To assess fidelity independently of the paired brick realization, we additionally compare the exposed surface of the predicted brick union with the source-object mesh. We denote the surface F-score at a distance tolerance of 2% of the source bounding-box diagonal by F@2%. SourceJoint(t) is the percentage of all 250 test objects whose outputs are completed (EOS for autoregressive outputs), valid, statically stable, and satisfy F@2% ≥ t.

InferencePolicyMean F@2%SourceJoint(0.3) %
DirectSFT0.453936.4
DirectGRPO0.438050.0

In a separate direct generation, SourceJoint(0.3) increases from 36.4% for SFT to 50.0% for GRPO. This improvement concerns joint reconstruction success.

Verification also provides a complementary benefit in the main result of original experiments. Under the same physical-search procedure and 32K sampled-token cap, GRPO improves [email protected] from 79.2% to 87.2% relative to SFT, while reducing mean sampled-token consumption from 7.56K to 5.02K. Together, these results support complementary roles for observations, post-training, and verification: observations constrain the instance, program-level feedback improves the generated candidates, and verification addresses remaining static failures within the search budget.

Why GRPO? SFT can learn structural regularities from reference programs, but teacher-forced token loss does not directly evaluate the geometry and stability of the policy's freely generated programs. GRPO compares multiple candidates for the same observation using computable program-level rewards, without a separate learned critic. It assigns credit to alternative brick sequences according to their geometric and physical properties.

AuthorsOpenReview ↗

Response to Reviewer 1mmf: B. Parameters decoding and qualitative evidence

Q3/W3: How were the reward parameters chosen?

Response. Thank you for asking us to clarify the parameter-selection process. The reward parameters were manually specified based on the roles of the constituent terms, rather than selected through a systematic hyperparameter search.

We set the geometric threshold to 0.4 as a safeguard against geometrically poor candidates receiving substantial physical reward. The geometry/physics weights of 0.7/0.3 give greater emphasis to instance fidelity while allowing physical feedback to influence candidate preferences. Within the geometric score, voxel IoU, silhouette IoU, and depth agreement receive weights of 0.40/0.35/0.25, respectively, placing the greatest weight on volumetric agreement while retaining complementary projected-shape and depth information. Within the physical score, the weights for S/C/I/J are 0.30/0.25/0.25/0.20, giving explicit static analysis the largest individual coefficient while also incorporating structural proxies. These values constitute a heuristic configuration used consistently in the main experiments, not an empirically established optimum.

Q4/W4: Could better BrickGPT decoding reduce failures?

Response. Enabling physical rollback substantially improves BrickGPT's stability. Its “no physical rollback” setting already uses brick-line grammar constraints and local rejection for unsupported dimensions, out-of-bounds placements, and collisions. This setting disables static-verification-based rollback and suffix regeneration while retaining the local checks. We will correct the overly broad description of the direct-generation panel so that it accurately distinguishes the constraints used by each method.

BrickGPT settingStable[email protected]
Local rejection, no physical rollback28.8%8.4%
Local rejection + physical rollback86.0%17.6%

These results demonstrate the benefit of physical rollback for BrickGPT: stability increases from 28.8% to 86.0%, while [email protected] increases from 8.4% to 17.6%. The remaining gap between stability and joint success reflects the additional requirement of matching the observed instance.

In the supplementary source-mesh evaluation, BrickGPT with search achieves 43.6% SourceJoint(0.3), compared with 80.0% for Full GRPO with search. This comparison reflects the performance of the complete pipelines, which differ in conditioning and training. The contribution of visual observations is examined more directly by the fixed-policy modality ablation in our response to Q2.

W5: Hunyuan appears similarly good in the qualitative figure

Response. Thank you for this observation. Hunyuan produces plausible overall shapes, while the differences from our method are more apparent in geometric details and proportions. The Visual and voxel IoU scores are reported beneath each result in the qualitative comparison. For example, in panel (d), Hunyuan's bookshelf is too thin relative to the ground truth, whereas our result better preserves its depth.

AuthorsOpenReview ↗

Response to Reviewer 1mmf: C. Update on reward-parameter sensitivity

Thank you for asking how we chose the reward parameters. We set the defaults before the main experiments: the gate limits physical reward to candidates with sufficient geometric agreement, and the 0.7/0.3 split gives geometry more weight once that condition is met. We subsequently tested two alternatives, changing one setting at a time and keeping the weights within each score fixed. Both new runs used the same SFT starting point, training data order, and training seed as the default. We compared all three policies after 500 GRPO updates on the same 250 held-out objects using sampled direct generation without search.

Geometry thresholdGeometry / physics weightsStable %Visual[email protected] %SourceJoint(0.3) %
0.40.7 / 0.3 (default)36.80.759935.631.6
0.60.7 / 0.336.80.763934.831.2
0.40.5 / 0.535.20.758432.830.4

Raising the threshold to 0.6 left the observed stability rate unchanged and yielded two fewer joint successes against the reference brick assembly (87 vs. 89 of 250) and one fewer against the source mesh (78 vs. 79). Giving physics equal weight did not improve stability at this checkpoint; [email protected] fell by 2.8 percentage points. The default thus gave the highest joint success on both targets among the three settings tested.

These 500-step comparisons suggest that joint reconstruction was only weakly affected by the tested threshold change: the stricter gate slightly improved mean Visual, but that gain did not translate into more outputs that were both faithful and stable. The weight comparison supports retaining the larger geometry weight in this setting. Shifting reward weight from geometry to physics produced no compensating gain in static stability.

Thank you again for the constructive review and we would be glad to hear whether these address your questions, and to clarify anything further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion for the Area Chair and Reviewers

We thank the Area Chair and all three reviewers for their time and thoughtful questions. We have submitted the revised PDF. The discussion helped us clarify what VISTA-Brick learns from visual observations, what static feedback contributes beyond structural proxies, and how the method compares with reconstruction-based alternatives. The main revisions and results are summarized below.

1.Contribution (1mmf). We clarify the distinction between reconstructing from source-object observations and generating from target-brick occupancy. With RGB-D input fixed, GRPO raises direct [email protected] from 42.8% to 63.2% over SFT.

2.Independent geometry and baselines (1mmf, bLDA, Y3te). SourceJoint(0.3) is 50.0% for VISTA-Brick Direct, 58.4% for RGB-D fusion + Mesh2Brick, and 80.0% for VISTA-Brick with search. We report generation costs, native and adapted STABLE inputs, and category-level Hunyuan results (Tables 11–12, 19–20).

3.Reward design (1mmf, bLDA, Y3te). Removing only static feedback lowers stability among valid outputs from 77.1% to 64.4%. We also report sensitivity tests for the geometry threshold, reward weights, and branch constants (Tables 13, 15–16).

4.Camera sensitivity (bLDA, Y3te). At the same fine-tuning budget, view augmentation raises mean [email protected] under ±5° yaw shifts from 25.0% to 42.2% (Tables 17–18).

We are grateful for the reviewers’ suggestions, particularly the independent source-mesh evaluation, the matched reward ablations, and the view-augmentation control. These additions make it clearer where the method helps.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.