- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Vibe Building, a task for generating structural building designs from textual or visual specifications, and proposes PE-Loop, which combines LLM-based discrete design revisions with deterministic column-section search and physics-based acceptance. Constraint violations guide revisions to sections, cores, walls, and structural systems. Returned designs undergo additional seismic, wind, and selected detailing checks. VB-Bench contains 3,577 instances across six code families; the main experimental campaign uses 260 instances, with the principal comparison focusing on 130 topology-decision instances. Across four backbone models and five agent systems, PE-Loop achieves the highest certified success rate in three backbone settings. The paper attributes its reliability primarily to external adjudication, although the current ablations and numerical reporting do not fully substantiate this interpretation.
Critical strengths
- The task addresses a meaningful problem in engineering agents. The separation between language-based design decisions, deterministic numerical search, and externally computed acceptance criteria is sensible. The explicit treatment of evaluation budgets, constraint violations, and escalation makes the system more interpretable than an unconstrained generate-and-assess workflow.
- The distinction between static compliance and certified success is particularly valuable. Table 1 shows that many designs passing the in-loop checks fail additional dynamic or detailing checks. Under DeepSeek-V4-Flash, PE-Loop achieves 91.28% static PASS but only 35.64% certified success. The constructability projection in Table 6 raises certified success to 45.38% while reducing static PASS to 61.54%. These results illustrate how optimizing an incomplete constraint set can produce misleading assessments of progress.
- The experimental campaign covers five systems, four backbone models, and three repetitions, and includes comparisons with LLM-free search on a separate reference set. The authors also disclose important negative results and limitations, including the absence of a feasibility improvement over pure search on the reference set, increased material consumption, infrastructure asymmetries, and the inability of held-out checks using the same physical model to detect model-form errors. These disclosures help establish the boundaries of the reported results.
Critical weaknesses
- The central ablation confounds acceptance authority with access to physical information. Section 5.2 and Table 5 state that the LLM judge receives the brief, code limits, and design parameters, but no simulator output. Consequently, this experiment changes both the adjudicator and the information available for adjudication. The reported 58.43% false-positive rate demonstrates the limitations of judging structural compliance without physical response data, but does not isolate the effect of transferring acceptance authority to an LLM that receives the same numerical evidence. Moreover, “Verified PASS” in Table 5 denotes a static oracle recheck, whereas the identically named metric in Table 1 includes dynamic and detailing checks. The oracle’s zero false-positive rate in Table 5 follows from reusing its acceptance criterion and should not be interpreted as independent evidence of generalization.
- The validity and scope of the physics oracle require stronger justification. Appendix B, lines 962–966, states that load-combination factors are omitted because the envelope of the selected gravity, wind, and seismic cases already provides a conservative upper bound. Taking an envelope over evaluated cases does not, by itself, establish conservatism relative to unevaluated factored combinations. The paper should specify the combinations and factors used for each graded constraint and justify any simplification. The certificate also covers a restricted set of checks: the seismic recheck uses one record in one horizontal direction with an elastic response model, and the detailing gate currently covers one clause family concerning beam clear-span-to-depth ratio. These limitations are acknowledged, but the broad language of code compliance and constructability needs to reflect this restricted scope. A representative comparison against trusted reference calculations would strengthen the benchmark’s validity.
- Several reported quantities are internally inconsistent. In the DeepSeek-V4-Flash block of Table 1, PE-Loop has PASS = 91.28% over 390 answers, corresponding to 356 successes and 34 failures. Since these answers come from 130 instances evaluated three times, at most 34 instances can fail at least once. Therefore, at least 96 instances must pass all three repetitions, giving . The reported value is 68.46%, which cannot arise from the same records under the stated definition. Figure 15 and Table 7 also disagree: for Openclaw, the figure reports 98 detailing-compliant designs among 157 feasible designs, whereas the table reports only 85 compliant designs among 170 feasible designs. A subset interpretation cannot explain a larger compliant count in the smaller set. These discrepancies require reconciliation using the underlying episode records.
Questions for authors
- Can you provide the episode-level records and calculation scripts needed to reconcile Table 1’s PASS and , as well as Figure 15 and Table 7? Were these quantities computed from different experiment versions or acceptance definitions?
- What happens when the LLM judge receives exactly the same physical response quantities and code limits as the deterministic judge? Can acceptance authority be evaluated separately from access to simulator feedback?
- On the same 130 topology-decision instances, how does LLM-based C1 compare with rule-based or budget-matched search alternatives when C2, the initial conditions, the action space, and the projection are held fixed?
5 comments
Response to Reviewer WZyU (2/2): W2, Q3
W2: The validity and scope of the physics oracle require stronger justification.
Thank you very much. We withdraw the claim that the unfactored envelope is a conservative bound; you are right that an envelope over evaluated cases does not bound unevaluated factored combinations. Appendix B.1 now specifies the load basis, and Table 2 lists every limit.
- Loads and factors. Acceptance takes the envelope of a self-weight case, a seismic case, and a wind case, without live load or load-combination factors. This matches the unit-factor drift checks of GB 50011-2010 Clause 5.5.1 and IS 1893 (Part 1):2016 Clause 7.11.1.1, and omits the displacement amplification of EN 1998-1 Clause 4.3.4 () and of the NEHRP provisions ().
- Limits. The scored instances of the main split carry performance targets defined by the benchmark, fixed per difficulty tier and uniform across code families; Table 2 marks them as such. Table 2 lists the reference set's GB limits and their clauses.
- What the certificate covers. Verified PASS means passing the implemented checks: one ground-motion record in one horizontal direction on an elastic model, a wind time-history drift check, and one detailing clause on beam clear-span-to-depth ratio. The abstract now states that these checks are necessary but not sufficient for code compliance.
- Reference calculations. We have not compared against independent reference calculations. Because the held-out checks reuse the same structural model, they cannot detect model-form errors. The limitations state this and list independent-solver validation as the next step.
Q3: On the same 130 topology-decision instances, how does LLM-based C1 compare with rule-based or budget-matched search?
Thank you very much. The new control runs this comparison on the same 130 instances, holding C2 and the action space fixed; no arm uses the projection (General Response, G1; Appendix G.2, Table 11). The fully rule-based variant (R3) and the fixed-order enumeration (R4) trail LLM-based PE-Loop by 3.08 and 4.62 pp of Verified PASS, which is not significant after Holm correction. Rule-based C1 on the LLM's replayed intent (R1) trails by 6.15 pp [−11.28, −1.28], and random C1 (R2) loses 16.41 pp. The repair step therefore needs a structured choice, and rule-based variants without an LLM come within 5 pp of LLM-based C1. The revision replaces our claim about what the language model buys accordingly.
We thank the reviewer again. If our responses and revisions address your concerns, we would be grateful if you would reconsider your score accordingly. Thank you very much!
Follow-up Before the Discussion Deadline (Less Than 1 Day Left)
Dear Reviewer,
As the author–reviewer discussion period will close in less than 1 day, we would like to follow up on our previous responses.
We have carefully addressed each concern raised in your review and provided clarifications and additional evidence. If our responses have resolved your concerns, we would be sincerely grateful if you would consider updating your evaluation accordingly. We sincerely hope that, in light of our responses, the paper may now merit a score of 3 or above. If any concerns remain, please let us know. We would be glad to address them before the discussion closes.
Thank you again for your time and thoughtful review.
Best regards, The Authors
Response to Reviewer WZyU (1/2): W3 & Q1, W1 & Q2
We thank the reviewer for the thorough, insightful comments and for checking the arithmetic. We address each point below.
W3 & Q1: Several reported quantities are internally inconsistent; can the episode records and scripts reconcile them?
Thank you very much. Your derivations are correct. pass³. The submitted PASS (91.28%) and pass³ (68.46%) for the DeepSeek-V4-Flash PE-Loop row cannot come from one set of records, as you show, and we could not trace them to the per-episode records. This was our error. We reran the row (130 instances × 3 seeds, on the model snapshot DeepSeek-V4-Flash-0731) and now take its PASS, pass³, NFE@pass, Seis., Wind, Build, and Verified PASS from this one run: PASS 92.31% (360/390) and pass³ 88.46% (115/130) and Verified PASS 41.54% (162/390). Update every number derived from this row in the text.
Table 7 and Fig. 15. Fig. 15 is generated by a script from the per-episode records of the reference set. In contrast, we filled in the submitted Table 7 by hand, and it contained errors. This was also our error. The table is now generated by script from the same records (Table 8 of the revision) and agrees with Fig. 15 in every cell:
For Openclaw, 98 of 157 feasible designs comply, as shown in the figure. On versions and definitions: neither discrepancy involves a different acceptance definition. The Table 1 row is now one rerun, Table 8 is regenerated from the records behind Figure 15, and all corrected numbers use the definitions stated in the paper. The per-episode records and scripts behind the PE-Loop row of Table 1 (PASS, pass³, and the held-out columns) and behind Table 8 and Figure 15 are in the anonymous repository https://anonymous.4open.science/r/records-41130-2774. The projection ablation you cite (Table 6 of the submission, Table 7 of the revision) is rerun in the same batch as this row: with Πbuild, static PASS is 77.69% and Verified PASS is 56.41%.
W1 & Q2: The judge ablation confounds acceptance authority with access to physical evidence; Verified PASS in Table 5 is a static recheck.
Thank you very much. We agree on both points. We renamed the column "Oracle-rechecked PASS" and added a note that the oracle's zero false-positive rate follows from reusing its acceptance criterion. In the new study, every judge-accepted design also underwent the held-out time-history and detailing checks, yielding a Verified PASS in the sense of Table 1 (General Response, G2; Appendix G.1, Table 10).
On Q2: when the judge receives the oracle's computed values and limits, it makes no false negatives and its violation sets match the oracle's (Jaccard 0.999). Its false-positive rate falls from 89.53% to 10.27% (11.26% with DeepSeek-V4-Pro as judge, 8.90% on Qwen3.8-Flash), and every remaining false positive is a tie at the oracle's display precision. Acceptance authority can therefore be evaluated separately from access to simulator feedback: on this evidence, access matters far more than who decides, and the deterministic comparison is the only judge that does not depend on how the values are presented.