- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents VISTA-Brick, a framework for generating geometrically faithful and physically stable brick programs from multi-view RGB-D observations and text descriptions. The authors extend StableText2Brick with rendered multi-view observations and adopt a two-stage training pipeline consisting of multimodal supervised fine-tuning and online GRPO. The GRPO stage combines geometric agreement, structural cues, and static-analysis feedback to improve program generation. The framework supports both direct generation and optional verifier-assisted physical search. Experiments on 250 objects show improvements in program validity, static stability, joint geometric-physical success, and search efficiency over the supervised baseline. Additional ablations examine visual conditioning, reward design, and inference-time search budgets.
Critical strengths
-
A meaningful problem setting combining geometric fidelity and physical stability. The paper addresses the challenge of generating brick assemblies that both resemble an observed object and satisfy physical constraints. Conditioning on multi-view RGB-D observations provides instance-specific geometric information beyond text-only generation. The comparison between text-only and multimodal inputs supports the importance of visual conditioning for geometric fidelity and joint success.
-
Comprehensive experiments separating learned generation from inference-time verification. The paper evaluates both direct generation and verifier-assisted physical search, allowing the improvements from GRPO to be distinguished from those obtained through external correction. The reported gains in direct stability and joint success, together with budget sensitivity experiments, reward ablations, and search-cost analysis, provide useful evidence for the effectiveness of the proposed framework.
Critical weaknesses
-
The contributions of individual reward components and the sources of efficiency improvements are not fully characterized. Although the reward ablation in Table 3 shows consistent improvements from full GRPO, the geometry-only and full configurations differ in their reward formulations, making it difficult to isolate the contributions of static-analysis feedback and structural proxies. Furthermore, Appendix Table 9 shows that, under verifier-assisted generation, GRPO reduces the average number of returned bricks from 145.32 to 125.04. While this may indicate more efficient assembly generation, it could also reflect changes in structural complexity or geometric detail. The paper does not sufficiently analyze whether the improvements in stability and search efficiency result from better structural arrangements, shorter programs, or a combination of both.
-
The evaluation of geometric fidelity and robustness remains limited. The geometric reward and evaluation metrics rely on the occupied geometry of a single paired reference brick program, although the paper acknowledges that a source object may admit multiple valid brick realizations. While different brick decompositions with identical occupied geometry are evaluated equivalently, valid realizations with different occupancy patterns may receive lower scores even when they preserve the intended source-object shape. Furthermore, the evaluation relies on synthetic observations rendered from ShapeNet objects using fixed camera configurations and clean depth and foreground-mask inputs. Without additional evaluation under varying viewpoints or imperfect observations, the robustness of the proposed framework beyond this controlled setting remains uncertain.
Questions for authors
-
Table 3 compares geometry-only and full GRPO, but the two settings differ in several aspects of their reward formulations. Could the authors further isolate the effects of static-analysis feedback, structural proxies, and geometric gating? It would also be helpful to see how reward, direct-generation stability, and joint success change over the course of training.
-
I noticed that the average number of returned bricks decreases from 145.32 to 125.04 after GRPO in Appendix Table 9. Does this mainly reflect more compact assemblies, or could some geometric details be lost? A comparison with the reference brick counts, along with stability results for valid programs and objects of different complexities, would help clarify this. I am also curious why the average verification time increases from 25.45 to 33.51 seconds despite fewer verifier calls.
-
The reward function uses several fixed parameters, including the geometric threshold and the geometry–physics weight . How sensitive are the results to these choices? In particular, I wonder whether the discontinuity at the geometric threshold affects training stability and how performance changes when these parameters are varied.
-
Since a single source object may have multiple valid brick realizations, I wonder whether evaluating the generated structures directly against the original source mesh would provide a more complete picture of geometric fidelity. It would also be useful to know how well the method performs under small changes in camera viewpoints or perturbations to the depth maps and foreground masks, rather than only under the fixed synthetic observation setting.
2 comments
Response to Reviewer Y3te: B. Sensitivity complexity and robustness
Q3 and related W1: Threshold discontinuity and parameter sensitivity
Response. Thank you for pointing out the missing sensitivity analysis. The G = 0.4 threshold is an empirical safeguard against geometrically poor candidates receiving substantial physical reward. The 0.7/0.3 geometry/physics split aims to retain the SFT policy's instance fidelity while improving physical behavior. This is a design rationale, not a validated optimum or an argument that the two reward scales are directly comparable.
A retraining-based threshold/weight sweep is not yet complete. Regarding the threshold discontinuity, GRPO treats the reward as an external scalar signal and updates the policy through token log-probabilities; it does not differentiate through the geometric score or the gate. The hard threshold therefore does not require a differentiable reward function. Nevertheless, crossing the threshold produces a discrete reward change that can alter the relative advantages of candidates within a group, particularly when candidates fall on both sides of the threshold. Its practical effect depends on the distribution of candidate geometry scores near the boundary. Our component ablations hold the threshold fixed and isolate the contribution of physical feedback under this configuration; sensitivity to the threshold itself remains a separate empirical question. Given the cost of online training within the rebuttal period, we prioritize matched component evaluations and will acknowledge the remaining sensitivity study.
Q2 and W1: Fewer bricks: better arrangements or missing detail?
Response. We agree that shorter output does not automatically imply lossless compaction. GT assemblies average 131.24 bricks, SFT plus search 145.32, and Full-10K plus search 125.04. In the high-complexity group, stability increases from 59.04% to 83.13%, while Visual decreases from 0.7257 to 0.7150.
Independent source-mesh evaluation shows the same tradeoff: SFT/Full search mean F@2% is 0.4522/0.4397, while SourceJoint(0.3) improves from 75.6% to 80.0%. F@2% measures surfaces at a 2%-diagonal tolerance, independent of brick decomposition. Thus, the evidence supports more frequent faithful-and-stable reconstruction, but not preservation of every geometric detail or improvement solely through arrangement.
Q2: Why does verification time increase despite fewer calls?
Response. Thank you for highlighting this interesting discrepancy; we examined the verification-time distribution in detail to understand it. The mean is affected by a long tail. From SFT to Full with search, median verification time decreases from 2.07 to 1.61 seconds, while the 95th percentile increases from 119.04 to 220.76 seconds. Mean time therefore rises from 25.45 to 33.51 seconds, despite mean calls falling from 5.336 to 3.412. We will report the distribution rather than infer lower solver time from fewer calls or bricks.
Q4 and W2: Alternative realizations and imperfect observations
Response. Thank you for these complementary suggestions. Source-mesh evaluation compares exposed union surfaces rather than one exact brick layout, using source-defined coordinates and no GT fitting. On the matched direct outputs, SFT-to-Full SourceJoint at F@2% thresholds 0.2/0.3/0.4 increases from 41.6/36.4/26.8% to 61.6/50.0/36.0%. This supports the joint objective under a different geometric target.
We also evaluated the same full policy under observation perturbations:
| Condition | Stable % | [email protected] % |
|---|---|---|
| Canonical clean RGB-D | 64.8 | 63.2 |
| Camera yaw -5 degrees | 46.0 | 41.6 |
| Camera yaw +5 degrees | 43.2 | 40.0 |
| Depth noise 3% | 58.8 | 56.8 |
| Depth missing 30% | 56.8 | 55.2 |
| Mask dilation 3 pixels | 54.4 | 52.0 |
Noise is measured relative to the encoded depth range, not metric distance. With 3% depth noise, Visual remains 0.7893 versus clean 0.7916, but Joint drops 6.4 points. At the tested strengths, camera shifts have a larger effect than depth noise or missing depth. This identifies camera configuration as a priority for improvement through explicit view modeling, calibration information, and broader training-view coverage. The joint metric reveals failures that average geometry alone can obscure.
Response to Reviewer Y3te: A. Reward decomposition and training trajectory
Thank you for distinguishing the overall post-training benefit from the contribution of each feedback component. We agree that the original geometry-only versus Full comparison does not isolate every change.
Q1 and related W1: Reward Ablation
Response. We compare Full with three reward ablations: T1 removes only the static term S without renormalization; T2 substitutes STABLE's reward recipe; T3 (Geometry+Gate) removes all physical feedback P while retaining the gate and geometry coefficient. All use the same multimodal policy, SFT start, data/order, training seed, and effective prompts per update, and are evaluated at matched steps on the same 250 objects under the same sampled-Direct settings.
F@2%. We sample 32,768 points per surface from the exposed brick-union boundary and the source mesh, using fixed source-defined coordinates. With tolerance �0�, where D is the source bounding-box diagonal, precision p is the fraction of predicted points within �3� of a source sample, and recall r reverses the direction. F@2%=2pr/(p+r), or zero when p+r=0. The mean includes all 250 objects: parseable invalid or non-EOS outputs retain their geometry scores; unavailable geometry scores zero.
SourceJoint(t). This is the percentage of all 250 objects with completed (EOS), valid, statically stable outputs and F@2%>=t. Thus, SourceJoint(0.3) requires F@2% of at least 0.3; 2% is the point-matching distance tolerance, whereas 0.3 is the per-object F-score threshold. Incomplete, invalid, unstable, or static-audit-failed outputs do not count as successes.
The clearest distinction is in stability among valid outputs. Validity ranges from 52.4% to 54.8% across the four policies, whereas Stable/Valid reaches 77.10% for Full compared with 62.04%-64.44% for the ablations. Full also achieves the highest overall stability, at 40.4%. The improvement therefore lies primarily in the physical quality of legal assemblies, rather than in producing more legal programs.
The sequence T3 → T1 → Full separates the feedback components while keeping the gate fixed. Adding C/I/J to T3 increases stability from 34.0% to 34.8%, with [email protected] unchanged at 32.4%. Adding S on top of these proxies then raises stability to 40.4% and [email protected] to 38.0%. This progression shows a small observed stability gain from proxies alone and a clearer joint benefit when explicit static feedback is included. The gate is unchanged throughout, so the latter improvement reflects feedback beyond the geometric threshold.
The source-mesh columns connect this stability gain to reconstruction quality. T2 obtains the highest mean source F@2% (0.4561, versus 0.4529 for Full), but Full achieves the highest SourceJoint(0.3), at 34.8%, compared with 28.8% for T2, 29.6% for T1, and 30.4% for T3. Thus, the advantage of Full is that more outputs are both geometrically acceptable and stable, even when another reward recipe achieves a higher average geometric score. This is the joint reconstruction objective that motivates combining static and structural feedback.
Q1: How do stability and joint success evolve during training?
Response. The saved training logs and original greedy direct evaluations provide the following trajectory on the same 250 test objects. Training reward is averaged over the 500 steps ending at each checkpoint; it is not a test-set score.
At 10K steps, GRPO adds 20.4 percentage points to both stability and [email protected] over SFT, a net 51 additional joint successes on 250 objects. The matching gains indicate that improved stability also satisfies the geometric criterion, rather than merely favoring stable but mismatched shapes. Since these are direct outputs, the benefit is learned by the policy rather than supplied by search.