acceptodds
Under review as a conference paper at ICLR 2027

Vibe Building

Abstract

Automated building design must comply with seismic and wind codes and satisfy structural mechanics constraints, yet most existing agents produce visually plausible models without verification grounded in mechanical analysis and code compliance. We introduce the Vibe Building task and propose **PE-Loop** (**P**hysics-**E**ngine-in-the-**Loop**), an agent in which a deterministic physics engine is the sole source of evaluation signals, mapping code constraints to a physics process reward, while the language model is confined to proposing discrete revisions (section menu, topology, and lateral system). Designs are verified by held-out seismic and wind time-history checks and a constructability gate. On VB-Bench, 3,577 physics-adjudicated building instances across six code families, PE-Loop achieves the highest verified success rate under three of four backbone LLMs, the highest held-out seismic pass rate under all four, and the highest held-out wind pass rate under three. Replacing the physics verdict with a language-model judge, all else fixed, leaves 58.43% of accepted designs noncompliant. These results suggest that reliable structural design rests less on a stronger LLM proposer than on an adjudicator the proposer cannot influence, a division of labor for agents whose outputs must hold up in the physical world.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, simulation-based verification, execution-based verification, structural design automation, AI for engineering design

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_WZyUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Vibe Building, a task for generating structural building designs from textual or visual specifications, and proposes PE-Loop, which combines LLM-based discrete design revisions with deterministic column-section search and physics-based acceptance. Constraint violations guide revisions to sections, cores, walls, and structural systems. Returned designs undergo additional seismic, wind, and selected detailing checks. VB-Bench contains 3,577 instances across six code families; the main experimental campaign uses 260 instances, with the principal comparison focusing on 130 topology-decision instances. Across four backbone models and five agent systems, PE-Loop achieves the highest certified success rate in three backbone settings. The paper attributes its reliability primarily to external adjudication, although the current ablations and numerical reporting do not fully substantiate this interpretation.

Critical strengths
  1. The task addresses a meaningful problem in engineering agents. The separation between language-based design decisions, deterministic numerical search, and externally computed acceptance criteria is sensible. The explicit treatment of evaluation budgets, constraint violations, and escalation makes the system more interpretable than an unconstrained generate-and-assess workflow.
  2. The distinction between static compliance and certified success is particularly valuable. Table 1 shows that many designs passing the in-loop checks fail additional dynamic or detailing checks. Under DeepSeek-V4-Flash, PE-Loop achieves 91.28% static PASS but only 35.64% certified success. The constructability projection in Table 6 raises certified success to 45.38% while reducing static PASS to 61.54%. These results illustrate how optimizing an incomplete constraint set can produce misleading assessments of progress.
  3. The experimental campaign covers five systems, four backbone models, and three repetitions, and includes comparisons with LLM-free search on a separate reference set. The authors also disclose important negative results and limitations, including the absence of a feasibility improvement over pure search on the reference set, increased material consumption, infrastructure asymmetries, and the inability of held-out checks using the same physical model to detect model-form errors. These disclosures help establish the boundaries of the reported results.
Critical weaknesses
  1. The central ablation confounds acceptance authority with access to physical information. Section 5.2 and Table 5 state that the LLM judge receives the brief, code limits, and design parameters, but no simulator output. Consequently, this experiment changes both the adjudicator and the information available for adjudication. The reported 58.43% false-positive rate demonstrates the limitations of judging structural compliance without physical response data, but does not isolate the effect of transferring acceptance authority to an LLM that receives the same numerical evidence. Moreover, “Verified PASS” in Table 5 denotes a static oracle recheck, whereas the identically named metric in Table 1 includes dynamic and detailing checks. The oracle’s zero false-positive rate in Table 5 follows from reusing its acceptance criterion and should not be interpreted as independent evidence of generalization.
  2. The validity and scope of the physics oracle require stronger justification. Appendix B, lines 962–966, states that load-combination factors are omitted because the envelope of the selected gravity, wind, and seismic cases already provides a conservative upper bound. Taking an envelope over evaluated cases does not, by itself, establish conservatism relative to unevaluated factored combinations. The paper should specify the combinations and factors used for each graded constraint and justify any simplification. The certificate also covers a restricted set of checks: the seismic recheck uses one record in one horizontal direction with an elastic response model, and the detailing gate currently covers one clause family concerning beam clear-span-to-depth ratio. These limitations are acknowledged, but the broad language of code compliance and constructability needs to reflect this restricted scope. A representative comparison against trusted reference calculations would strengthen the benchmark’s validity.
  3. Several reported quantities are internally inconsistent. In the DeepSeek-V4-Flash block of Table 1, PE-Loop has PASS = 91.28% over 390 answers, corresponding to 356 successes and 34 failures. Since these answers come from 130 instances evaluated three times, at most 34 instances can fail at least once. Therefore, at least 96 instances must pass all three repetitions, giving pass3≥73.85\mathrm{pass}^{3}\geq73.85%. The reported value is 68.46%, which cannot arise from the same records under the stated definition. Figure 15 and Table 7 also disagree: for Openclaw, the figure reports 98 detailing-compliant designs among 157 feasible designs, whereas the table reports only 85 compliant designs among 170 feasible designs. A subset interpretation cannot explain a larger compliant count in the smaller set. These discrepancies require reconciliation using the underlying episode records.
Questions for authors
  1. Can you provide the episode-level records and calculation scripts needed to reconcile Table 1’s PASS and pass3\mathrm{pass}^{3}, as well as Figure 15 and Table 7? Were these quantities computed from different experiment versions or acceptance definitions?
  2. What happens when the LLM judge receives exactly the same physical response quantities and code limits as the deterministic judge? Can acceptance authority be evaluated separately from access to simulator feedback?
  3. On the same 130 topology-decision instances, how does LLM-based C1 compare with rule-based or budget-matched search alternatives when C2, the initial conditions, the action space, and the projection are held fixed?
5 comments
AuthorsOpenReview ↗

Response to Reviewer WZyU (1/2): W3 & Q1, W1 & Q2

We thank the reviewer for the thorough, insightful comments and for checking the arithmetic. We address each point below.


W3 & Q1: Several reported quantities are internally inconsistent; can the episode records and scripts reconcile them?

Thank you very much. Your derivations are correct. pass³. The submitted PASS (91.28%) and pass³ (68.46%) for the DeepSeek-V4-Flash PE-Loop row cannot come from one set of records, as you show, and we could not trace them to the per-episode records. This was our error. We reran the row (130 instances × 3 seeds, on the model snapshot DeepSeek-V4-Flash-0731) and now take its PASS, pass³, NFE@pass, Seis., Wind, Build, and Verified PASS from this one run: PASS 92.31% (360/390) and pass³ 88.46% (115/130) and Verified PASS 41.54% (162/390). Update every number derived from this row in the text.

Table 7 and Fig. 15. Fig. 15 is generated by a script from the per-episode records of the reference set. In contrast, we filled in the submitted Table 7 by hand, and it contained errors. This was also our error. The table is now generated by script from the same records (Table 8 of the revision) and agrees with Fig. 15 in every cell:

ArmInstancesAnswersFeasibleViolating ln/hb≥4l_n/h_b\ge 4
Codex5114912568 (54.40%)
Openclaw5715715759 (37.58%)
DeepSeek Harness5817517289 (51.74%)
Claude Code5817216973 (43.20%)
PE-Loop w/o Πbuild\Pi_{\mathrm{build}}58177174126 (72.41%)

For Openclaw, 98 of 157 feasible designs comply, as shown in the figure. On versions and definitions: neither discrepancy involves a different acceptance definition. The Table 1 row is now one rerun, Table 8 is regenerated from the records behind Figure 15, and all corrected numbers use the definitions stated in the paper. The per-episode records and scripts behind the PE-Loop row of Table 1 (PASS, pass³, and the held-out columns) and behind Table 8 and Figure 15 are in the anonymous repository https://anonymous.4open.science/r/records-41130-2774. The projection ablation you cite (Table 6 of the submission, Table 7 of the revision) is rerun in the same batch as this row: with Πbuild\Pi_{\mathrm{build}}, static PASS is 77.69% and Verified PASS is 56.41%.


W1 & Q2: The judge ablation confounds acceptance authority with access to physical evidence; Verified PASS in Table 5 is a static recheck.

Thank you very much. We agree on both points. We renamed the column "Oracle-rechecked PASS" and added a note that the oracle's zero false-positive rate follows from reusing its acceptance criterion. In the new study, every judge-accepted design also underwent the held-out time-history and detailing checks, yielding a Verified PASS in the sense of Table 1 (General Response, G2; Appendix G.1, Table 10).

On Q2: when the judge receives the oracle's computed values and limits, it makes no false negatives and its violation sets match the oracle's (Jaccard 0.999). Its false-positive rate falls from 89.53% to 10.27% (11.26% with DeepSeek-V4-Pro as judge, 8.90% on Qwen3.8-Flash), and every remaining false positive is a tie at the oracle's display precision. Acceptance authority can therefore be evaluated separately from access to simulator feedback: on this evidence, access matters far more than who decides, and the deterministic comparison is the only judge that does not depend on how the values are presented.

AuthorsOpenReview ↗

Response to Reviewer WZyU (2/2): W2, Q3

W2: The validity and scope of the physics oracle require stronger justification.

Thank you very much. We withdraw the claim that the unfactored envelope is a conservative bound; you are right that an envelope over evaluated cases does not bound unevaluated factored combinations. Appendix B.1 now specifies the load basis, and Table 2 lists every limit.

  • Loads and factors. Acceptance takes the envelope of a self-weight case, a seismic case, and a wind case, without live load or load-combination factors. This matches the unit-factor drift checks of GB 50011-2010 Clause 5.5.1 and IS 1893 (Part 1):2016 Clause 7.11.1.1, and omits the displacement amplification of EN 1998-1 Clause 4.3.4 (qdq_d) and of the NEHRP provisions (Cd/IeC_d/I_e).
  • Limits. The scored instances of the main split carry performance targets defined by the benchmark, fixed per difficulty tier and uniform across code families; Table 2 marks them as such. Table 2 lists the reference set's GB limits and their clauses.
  • What the certificate covers. Verified PASS means passing the implemented checks: one ground-motion record in one horizontal direction on an elastic model, a wind time-history drift check, and one detailing clause on beam clear-span-to-depth ratio. The abstract now states that these checks are necessary but not sufficient for code compliance.
  • Reference calculations. We have not compared against independent reference calculations. Because the held-out checks reuse the same structural model, they cannot detect model-form errors. The limitations state this and list independent-solver validation as the next step.

Q3: On the same 130 topology-decision instances, how does LLM-based C1 compare with rule-based or budget-matched search?

Thank you very much. The new control runs this comparison on the same 130 instances, holding C2 and the action space fixed; no arm uses the projection (General Response, G1; Appendix G.2, Table 11). The fully rule-based variant (R3) and the fixed-order enumeration (R4) trail LLM-based PE-Loop by 3.08 and 4.62 pp of Verified PASS, which is not significant after Holm correction. Rule-based C1 on the LLM's replayed intent (R1) trails by 6.15 pp [−11.28, −1.28], and random C1 (R2) loses 16.41 pp. The repair step therefore needs a structured choice, and rule-based variants without an LLM come within 5 pp of LLM-based C1. The revision replaces our claim about what the language model buys accordingly.


We thank the reviewer again. If our responses and revisions address your concerns, we would be grateful if you would reconsider your score accordingly. Thank you very much!

AuthorsOpenReview ↗

Follow-up Before the Discussion Deadline (Less Than 1 Day Left)

Dear Reviewer,

As the author–reviewer discussion period will close in less than 1 day, we would like to follow up on our previous responses.

We have carefully addressed each concern raised in your review and provided clarifications and additional evidence. If our responses have resolved your concerns, we would be sincerely grateful if you would consider updating your evaluation accordingly. We sincerely hope that, in light of our responses, the paper may now merit a score of 3 or above. If any concerns remain, please let us know. We would be glad to address them before the discussion closes.

Thank you again for your time and thoughtful review.

Best regards, The Authors

Reviewer_hefuOpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper introduces Vibe Building, the task of producing a code-compliant structural design for a building from a text brief, a sketch or photographs, together with VB-Bench, 3,577 building instances across six code families, and PE-Loop, a harness in which a deterministic physics engine holds sole acceptance authority. The LLM agent picks one of five action types per round, the column side is found by bisection, and accepted designs are re-checked by held-out seismic and wind time-history analyses and one detailing clause. The paper compares PE-Loop with four harnesses driven by the same four backbone LLMs on a subset of the benchmark, reports Verified PASS as the main metric, and ablates the verdict holder by replacing the physics check with an LLM judge.

Critical strengths
  1. The paper gives a usable evaluation pipeline for LLM agents on agentic building design, with a clear task definition and a physics engine as the judge, which has its practical value for the AI for engineering community. The reproducibility statement promises a release of the instances and data.

  2. PE-Loop exploits two properties of code-compliant design, that acceptance is decidable by a deterministic engine and that feasibility is monotone (under the paper's assumption) in the column side. The engine decides acceptance, the column side is searched by bisection, and the agent only chooses among discrete structural changes.

Critical weaknesses
  1. The main comparison does not isolate the conclusion the paper ends on, that reliability comes from who holds the verdict and not from the proposer. The four baselines receive the same margin oracle as a callable tool, so they can hold the same verdict. Where they fall behind PE-Loop in Table 1, they do so by timing out, running the budget to zero or never delivering a design, which are differences between harnesses. The control that would settle it is PE-Loop with its LLM replaced by a fixed rule, run as a comparison experiment on the topology-decision instances. The paper only has an LLM-free search on the reference set, where it ties with PE-Loop. The ablation that swaps the verdict for an LLM judge does not settle it either. The judge never sees a simulator quantity, so it measures the absence of simulation output and not who holds the verdict.

  2. The lead on Verified PASS is small. Under DeepSeek-V4-Flash PE-Loop reaches 35.64 against 31.03 for Claude Code, under GLM 5.2 Fast it comes fourth, and the paper notes that both exceptions lie within about one standard error. The raw PASS lead is larger, but six to seven of every ten designs PE-Loop accepts in the loop fail its own Verified PASS. The standard errors treat the three seeds of one instance as independent answers. It still does not tell me whether a gain this small is worth giving up an extensible general harness for this agentic building task.

  3. Nothing is released at review time, so the paper has to stand on its own, and it currently does not. The reproducibility statement promises a release upon publication. Until then the paper does not give the task prompts or the prompts provided to PE-Loop and to the baseline harnesses, a sample brief and a few of the 1,311 tasks, the limit values checked under each code family and the list of the 86 codes behind the knowledge base, how the synthetic instances were generated and which buildings the literature-grade reconstructions and the real anchors come from, who performed the expert review, with what qualification and what relation to the authors, the model identifiers with dated versions and the reasoning effort of each backbone, or the OpenSeesPy version. This is fixable, but at this stage the reported results cannot be checked.

Questions for authors

Clarification

  1. Section 5.3 reports an LLM-free pure search that ties with PE-Loop on the reference set. Was it run on the topology-decision instances of the main split, where the systems are separated? For the baselines, Table 1 gives timeouts and Appendix E gives non-deliveries for the GLM block only. For each block, how much of each failure rate is timeouts and non-delivery, and how much is delivered designs that failed?

  2. The abstract says PE-Loop has the highest held-out seismic pass rate under all four backbones. That holds for the run without the constructability projection. Section 4 writes the projection into the method and also calls it an ablation. With it, in Table 6, the held-out seismic and wind rates under DeepSeek fall below all four baselines. Why does the main table run without it, and which version is the method?

  3. In the ablation that swaps the verdict for an LLM judge, how many designs did the judge reject that the physics check would have accepted? How is Jaccard defined? Why does the judge get no simulator output, and was the ablation run on any other backbone?

  4. The detailing set of each instance is said to be drawn from its code family, but the only sourced detailing clause comes from GB 50011 and JGJ 3. For EC8, ASCE 7, BSL, IS and NZS instances, is this clause applied, is the detailing set empty, or is the Build check unevaluated? Please report the Build column by code family.

  5. What is a task, as opposed to an instance? Section 3 counts 1,311 gradable tasks of three kinds, intent, design and modeling, but every result in the paper is on instances, and no example of a task or of how it is graded appears. What are they for in this paper, if none of them enters an experiment?

  6. Which model performs perception on the sketch and image instances in each backbone block? From my experience, DeepSeek-V4-Flash itself does not take images. What do the baselines receive on those instances mixing text and images? If the perception model differs from the backbone, the statement that the driving model is the only free variable does not hold there.

I think this paper is decent, and I can see it going to a 3 if the concerns above are answered. Further suggestions below.

Suggestions

  1. The title tells little about what the paper does. In nine pages the paper tries to position a task, present a benchmark and present a method at once, and none of the three gets the room it needs. Especially when read from the title and the opening paragraph, the paper seems more of a position paper, but from the bold PE-Loop in the abstract and most of the content, it seems more like a method paper, so I would suggest a title built around the method and agentic building design, narrowed by what the method actually does, such as code compliance and physics adjudication, and one main line through the paper. The paper's naming of harness, agent and LLM is also somewhat mixed. In my experience an agent is an LLM plus a harness, and an action, with or without tools, is performed by the agent and not by the LLM, so PE-Loop reads as a harness, and the terms could be straightened out, though I am open on this definition.

  2. There is no cost information in tokens or money. A cost against verified success plot in the appendix, per system and backbone, would be welcome. I think the community could further benefit from it and I am also interested in it.

  3. The appendix is long and has no table of contents. Please add one.

8 comments
AuthorsOpenReview ↗

Response to Reviewer hefu (1/2): W1, W2, Q1, Q2

We thank the reviewer for the thorough, insightful comments and for identifying exactly what would change the assessment. We address each point below.


W1: The main comparison does not isolate the verdict holder.

Thank you very much. We ran the control you proposed: PE-Loop with the LLM replaced by fixed rules, on the 130 topology-decision instances, with C2, the oracle, and the action space unchanged (General Response, G1; Appendix G.2). With the LLM removed entirely (R3, R4), Verified PASS falls by 3.08 and 4.62 pp, which is not significant, whereas random repairs lose 16.41 pp. This supports your reading: PE-Loop's lead comes mainly from its harness, not from its LLM, and the revision says so. We also agree that the judge ablation measured the absence of simulator output. With the oracle's computed values and limits (G2; Appendix G.1), the judge's false-positive rate falls from 89.53% to 10.27%. Every remaining false positive is a tie at display precision, and there are no false negatives.


W2: The Verified PASS lead is small, and the standard errors treat seeds as independent.

Thank you very much. You are right about the errors. With 130 instances, the instance-clustered standard error of a rate pp cannot exceed p(1−p)/130\sqrt{p(1-p)/130}, which is the value it takes when the three seeds for every instance agree. For the row we recomputed from per-episode records (PE-Loop, DeepSeek-V4-Flash), the actual clustered errors are close to this bound (PASS 2.08 vs 2.34 pp; Verified PASS 4.01 vs 4.32 pp). Under this bound, PE-Loop minus the best baseline of each block, in pp (z of an unpaired two-sided test, no multiplicity correction):

BlockStatic PASSHeld-out seismicHeld-out windVerified PASS
DeepSeek-V4-Flash+8.46 (2.12)+14.87 (2.57)+19.49 (3.70)+10.51 (1.77)
Qwen3.8-Flash+11.54 (2.08)+18.72 (3.07)+18.72 (3.13)+4.10 (0.81)
GLM 5.2 Fast−10.00 (−2.11)+2.31 (0.38)−0.26 (−0.04)−3.33 (−0.59)
Kimi K3+28.98 (5.30)+26.67 (4.46)+31.54 (5.39)+6.93 (1.30)

The Verified PASS lead does not reach the 5% level in any block (closest: DeepSeek-V4-Flash, p = 0.08), while the leads in static PASS and at both held-out checks do in three of the four blocks. We therefore do not claim that PE-Loop's Verified PASS exceeds that of the best baseline. Our claims rest on the held-out checks, on the smaller variation of PE-Loop across the four backbones (static PASS spans 15.39 pp, against 32.82 pp for the best baseline of each block), and on the two controls above.

On in-loop acceptances that fail, Verified PASS adds checks that no system optimizes for, and the biggest loss is the single detailing clause. When that clause is moved into the loop (Πbuild\Pi_{\mathrm{build}}, Appendix E.1), Verified PASS rises from 41.54% to 56.41%, so the gap measures how incomplete the in-loop checks are, not noise.

On giving up a general harness: we do not propose replacing general harnesses. The control in G1 shows what makes the oracle reliable: termination gated by the oracle, bisection on the monotone coordinate, and escalation. A general harness would need these elements for this task.


Q1: Was LLM-free search run on the main split? How much of each failure rate is due to timeouts and non-delivery?

Thank you very much. The submitted pure search ran only on the reference set; R3 and R4 now run on the 130 topology-decision instances (R4 is a fixed-order enumeration). We classified every failed topology-decision answer of the four baselines. Timeouts and non-deliveries account for 44%, 81%, 88%, and 81% of the failures under DeepSeek-V4-Flash, Qwen3.8-Flash, GLM 5.2 Fast, and Kimi K3; the rest are delivered designs that failed. Outside the DeepSeek-V4-Flash block, the baselines therefore mostly fail to finish or to deliver, and their raw PASS mainly measures whether a harness completes the task within the cap. PE-Loop delivers no infeasible design in any block: it escalates instead. Section 5.1 now says so, and Appendix E gives the five failure modes.


Q2: Which version is the method, with or without Πbuild\Pi_{\mathrm{build}}?

Thank you very much. The method is PE-Loop without Πbuild\Pi_{\mathrm{build}}, the configuration of Table 1 and of the abstract's claims. Section 4 now presents the projection as an optional variant (Appendix E.1). It stays out of the main comparison for fairness: the baselines never see the detailing clause during search. Its lower absolute held-out rates come mainly from accepting fewer designs (303 of 390 answers instead of 360). Among statically compliant designs, held-out seismic rises from 79.7% to 83.2%, wind falls from 91.4% to 89.1%, and the full certificate rises from 45.0% to 72.6%.

AuthorsOpenReview ↗

Response to Reviewer hefu (2/2): W3, Q3–Q6, S1–S3

W3: Nothing is released at review time; the reproducibility details are missing.

Thank you very much. The revised manuscript now contains these items.

  • Prompts: Appendix F gives the PE-Loop prompts (intent, C1), the perception prompt, and the blind and informed judge prompts, abridged; F.4 states what the baselines receive, and F.5 gives the brief of one instance.
  • Limits and sources: Appendix B.1 and Table 2 list every limit with its clause or mark it as a performance target, and summarize the 86 source documents, including the eight design standards by designation.
  • Instances: Appendix B. T1 is generated by code without an LLM (a deterministic stratified sampler). An LLM wrote T2 descriptions from public sources. T3 descriptions were compiled from the source documents we collected (station records and reconnaissance reports) and public online material, with an LLM assisting the writing.
  • Expert review: civil-engineering graduate students of the authors' research group, assisted by AI tools (Appendix B, AI use statement).
  • Models and software: deepseek-v4-flash, qwen3.8-flash, glm-5.2-fast-preview, and kimi-k3, accessed in September 2026. The platforms expose no dated snapshots for these endpoints. Reasoning effort is the provider default. OpenSeesPy 3.8.0.0 (Appendix H, Tables 12 and 13).

Q3: Judge false negatives, the Jaccard definition, simulator output, and other backbones.

Thank you very much. In the rerun, the blind DeepSeek judge rejected 33 of the 290 evaluations the oracle accepted (11.38%), and the blind Qwen3.8-Flash judge rejected 252 of 724 (34.81%); the informed judges rejected none. Jaccard is computed on each evaluation where both the judge and the oracle report a violation: the intersection of the two violated-check sets over their union, averaged over these evaluations. The submitted judge stood for an LLM-only harness that certifies its own design from the specification; that conflates evidence with authority, which the informed arms now separate. The ablation now also runs on Qwen3.8-Flash, with the same conclusion (G2).


Q4: Is the detailing clause applied to non-GB families? Build by code family.

Thank you very much. The clause ln/hb≥4l_n/h_b \ge 4 is applied to every instance regardless of its code family; no instance has an empty set, and Build is never unevaluated. It is a benchmark rule taken from GB 50011-2010 Clause 6.3.1 and GB 50010-2010 Clause 11.3.5, not a claim that EC8 or ASCE 7 impose the same limit. The 130 topology-decision instances cover GB (56), EC8 (39), ASCE 7 (32), and BSL (3). Across the four backbones, PE-Loop's Build is 42–74% on GB, 31–47% on EC8, and 18–26% on ASCE 7; GB is the highest in every system × backbone cell. Appendix D.3 gives the breakdown. We also corrected the count: the instance layer has five code families. NZS occurs only in the task layer.


Q5: What is a task, as opposed to an instance?

Thank you very much. An instance is an observation of a massing model paired with a site; the agent designs it end-to-end. The task layer is a separately collected set of 1,311 records, each a text brief of a real building, and a task is a question posed on one record: intent (L1), design (L2), or modeling (L3). Tasks do enter one experiment: the 59-instance reference set of Section 5.3 consists of 59 GB records of the task layer, posed as L2 questions and adjudicated by the same physics engine. No L1 or L3 question enters an experiment. Appendix B now defines the three kinds and their grading and shows two records that cover all three.


Q6: Which model performs perception?

Thank you very much. One vision-language model, Qwen3-VL-Plus, performs perception on all 21 sketch and image instances, in every block and for every system, because DeepSeek-V4-Flash and GLM 5.2 Fast do not accept images. Every system receives its text output, and adjudication runs on the ground-truth massing, so the driving model remains the only free variable within each block (Section 5).


S1–S3: Title and terminology, cost, table of contents.

Thank you very much. We agree that the title should say what the method does, and the revised title is "Vibe Building: Towards Physics-Engine-in-the-Loop Structural Design". We adopt your terminology: an agent is an LLM plus a harness, and PE-Loop is a harness (Section 1). Appendix H (Table 13) reports LLM calls, tokens, and cost per answer for all 20 cells: PE-Loop makes 2.0–2.3 calls and about 16k tokens per answer, against 9.8–42.9 calls and 151–2,669k tokens for the baselines. The appendix now has a table of contents.


We thank the reviewer again. If our responses and revisions address your concerns, we would be grateful if you would reconsider your score accordingly. Thank you very much!

Thank you for the point by point responses and the revised manuscript.

The rule-based variants on the 130 topology-decision instances and the informed judges isolate the source of PE-Loop's reliability. The revised claims now follow from the new controls.

Your responses on the clustered standard errors, the withdrawn claim on Verified PASS and the reproducibility details resolve my other concerns. With the new controls and the cost table, the paper looks comprehensive and solid. And I need some more time to go through the revised manuscript.

So far, I am willing to raise my rating to 3.

Reviewer_nL8BOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Vibe Building, a task for producing structural designs from textual or visual requirements, and VB-Bench, a benchmark spanning several building-code families. Its proposed PE-Loop combines deterministic section search with LLM-proposed structural revisions, while a physics engine determines whether each candidate satisfies the modeled constraints. Evaluation compares five agent systems across four LLM backbones and distinguishes in-loop compliance from held-out dynamic and detailing checks. The results report generally stronger verified performance and lower evaluation cost for PE-Loop, while ablations examine language-model judging, constructability constraints, and the contribution of the LLM relative to numerical search.

Critical strengths
  1. The distinction between optimization-time compliance and subsequent verification is valuable. The substantial drop from raw to verified success shows that passing the checks used during search is an incomplete measure of agent reliability, making the evaluation protocol a useful contribution in its own right.

  2. The evaluation across four backbones and a shared simulator budget provides meaningful evidence for the practical effectiveness of PE-Loop on the evaluated tasks. The paper also reports exceptions, material-cost tradeoffs, and constructability failures, which helps establish the actual benefits and limits of the approach.

Critical weaknesses
  1. The experiments do not isolate the claim that acceptance authority is the decisive source of reliability. The LLM judge in Table 5 receives no simulator outputs, so the intervention changes access to physical evidence as well as the acceptance rule. The main comparisons also combine deterministic search and structured repair with the proposed gate, while the LLM-free comparison uses a separate, easier setting. These results do not establish how much of the improvement comes specifically from the proposed allocation of decision-making responsibility.

  2. The scope of verification is narrower than the framing of code-compliant structural design. The held-out checks share the same physical model, the seismic check uses one record in one direction, and constructability covers only one clause family. Appendix B also treats an unfactored load envelope as conservative without establishing that it bounds the relevant combined-load effects. These limitations leave important failure modes outside the certificate and restrict what the reported success rates establish about structural compliance.

Questions for authors
  1. In the judge ablation, is the feedback used for structural repair also generated by the judge? Does any reported comparison preserve the same physical evidence and repair feedback while changing only the acceptance decision?

  2. What loads and factors are actually included in the oracle’s load cases, and what justifies treating their envelope as a conservative substitute for load combinations?

5 comments
AuthorsOpenReview ↗

Response to Reviewer nL8B

We thank the reviewer for the thorough and insightful comments. We address each point below.


W1 & Q1: The experiments do not isolate acceptance authority; is the repair feedback generated by the judge?

Thank you very much. On Q1: yes, in the submitted judge arms, the judge generated the repair feedback itself. The new arms A3 and A4 (General Response, G2; Appendix G.1, Table 10) keep the same physical evidence and take the repair feedback from the oracle, so only the acceptance decision changes. In that comparison, the judge reads the numbers correctly: it makes no false negatives, and its violation sets match the oracle's (Jaccard 0.999). All of its 10–11% false positives are ties at the oracle's display precision. Without the numbers, 89.53% of its acceptances are noncompliant. The decisive ingredient is therefore the simulator evidence, and the abstract and Section 5.2 now say so.

On the main comparison bundling search and structured repair with the gate, and on the LLM-free comparison using an easier set, two LLM-free variants now run on the 130 topology-decision instances of the main split (General Response, G1; Appendix G.2, Table 11). R3 uses rule intent and rule repairs; R4 enumerates repairs in a fixed order. Neither differs significantly from PE-Loop on Verified PASS (−3.08 and −4.62 pp, Holm p ≥ 0.07). In contrast, uniformly random repairs lose 16.41 pp. What carries the result is the allocation the harness enforces: the oracle decides acceptance and drives the column search. An LLM in the loop is not needed to come within 5 pp of PE-Loop's Verified PASS on this split.


W2 & Q2: The scope of verification is narrow; what loads and factors does the oracle include?

Thank you very much. The oracle evaluates three load cases separately (self-weight, a seismic case, and a wind case, each built from the loading provisions of the instance's code family). It takes the most unfavourable value of each response quantity, without live load and without load-combination factors. We withdraw the statement that this envelope is a conservative upper bound: an envelope over evaluated cases does not bound unevaluated factored combinations. Appendix B.1 now states where this basis agrees with the codes and where it does not. For elastic drift, it matches GB 50011-2010 Clause 5.5.1 and IS 1893 (Part 1):2016 Clause 7.11.1.1, which prescribe unit load factors; EN 1998-1 Clause 4.3.4 and the NEHRP provisions amplify elastic displacements by qdq_d or Cd/IeC_d/I_e, which the oracle does not apply.

In addition, every scored instance of the main split carries performance targets defined by the benchmark, fixed per difficulty tier and uniform across code families (Table 2: drift ratio 0.0015, torsional displacement ratio 1.5, stability index 0.15, static displacement 0.02 m, and an instance-specific minimum first frequency). The scores therefore measure compliance with these targets under this load basis, not with each family's clauses.

The other limits hold as you state them: the held-out checks reuse the same structural model, the seismic recheck applies one record in one direction to an elastic model, and the detailing check covers one clause. The abstract now says that the verified checks are necessary but not sufficient for code compliance, and the limitations list independent-solver validation, factored load combinations, and broader clause coverage as the next steps.


We thank the reviewer again. If our responses and revisions address your concerns, we would be grateful if you would reconsider your score accordingly. Thank you very much!

Thanks for the authors‘ detailed response and paper revisions. I have also read other reviewers' comments and the following discussions. The added informed-judge experiments, and the rule-based controls on the main topology-decision split addressed the weakness W1 of me. I have two simple remaining concerns.

  1. I agree with Reviewer hefu for the benchmark release statement. The revision now commits to a curated set of high-quality examples. I also noticed that Reviewer WZyU pointed out the inconsistency of the original results, and the authors have corrected the results and provided constructive records. Nevertheless, this makes the scope of the benchmark materials very important. The authors should clarify which of these content will be released. If only a subset of the data will be released, what is the selection criteria? Since VB-Bench is a key contribution, the release of it is important for reproduction and using by the community.

  2. Table 2 and Appendix B.1 clarify that the main split uses benchmark targets across code families. But the current phrase “necessary but not sufficient for code compliance” may still make readers think that all benchmark targets are necessary conditions under the applicable codes.

I remain the positive score (3) based on the improved controls and the value of separating the in-loop compliance from subsequent verification.

AuthorsOpenReview ↗

Response to Reviewers nL8B (Continued)

Thank you for the follow-up and for your positive assessment. We address the two remaining concerns below.


Release scope of VB-Bench.

You and Reviewer hefu are right: in shortening the reproducibility statement, we narrowed the release commitment, which we should not have done for a benchmark contribution. We have restored it. We will release, upon publication:

  • VB-Bench: the instances, the task records with their grader, the limits of Table 2, the screening and difficulty labels, the 260-instance main split, and the 59-record reference set;
  • The margin oracle with the held-out seismic and wind checks and their record libraries, and the detailing check;
  • PE-Loop and the harness that drives all five systems.

This is the full benchmark, not a subset, so no selection criterion applies. The only material we cannot release is the text and page images of the source standards, which remain under their publishers' copyright; the knowledge base is released as document designations, clause numbers, and limit values (Appendix B.1). The per-episode records and scripts behind the corrected numbers are already available in the anonymous repository cited in our reply to Reviewer WZyU.


"Necessary but not sufficient for code compliance".

We agree. The main split scores benchmark-defined targets, which are uniform across code families and are not clauses of any code, so passing them is not a necessary condition under the applicable code. We removed the phrase from the abstract, Section 5, and the limitations. The abstract now reads: "Designs are verified by held-out seismic and wind time-history checks and a detailing check against benchmark-defined targets; passing them does not establish code compliance." Section 5 and the limitations say the same.


Thank you again for the careful reading.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their careful and constructive reviews. All three point to the same gap: the experiments did not isolate the source of PE-Loop's reliability. We ran the controls you requested (2,300 new episodes, every returned design rechecked with the held-out time-history and detailing checks). We revised our claims to match what the results support. The revised manuscript marks all changes in blue.


G1: Rules in place of the LLM on the 130 topology-decision instances (hefu W1/Q1, WZyU Q3, nL8B W1).

We fixed C2 (bisection over the column side), the margin oracle, the action space, B=15B=15, and K=3K=3, used no Πbuild\Pi_{\mathrm{build}} in any arm, and replaced the LLM in four ways. The reference arm is the PE-Loop row of Table 1 under DeepSeek-V4-Flash. We froze the rule table before any scored run and tuned it only on instances outside the scored split.

VariantIntentRepair (C1)Static PASSVerified PASSΔVerified vs. PE-Loop [95% CI]Holm p (bootstrap / McNemar)
PE-Loop (3 seeds)LLMLLM92.3141.54reference—
R1 (3 seeds)LLM (replayed)frozen rule table82.3135.38−6.15 [−11.28, −1.28]0.055 / 0.022
R2 (3 seeds)LLM (replayed)uniform random91.5425.13−16.41 [−23.08, −10.00]<0.001 / <0.001
R3rulefrozen rule table90.0038.46−3.08 [−8.97, 2.56]0.30 / 0.14
R4rulefixed-order enumeration88.4636.92−4.62 [−10.00, 0.51]0.17 / 0.070

Differences are paired by instance with 10,000 instance-level bootstrap resamples; Holm correction over the four variants.

Three observations: (1) With no LLM at all, the rule-based variants (R3, R4) do not differ significantly from PE-Loop, on Verified PASS or on static PASS (−2.31 and −3.85 pp). (2) Which action is taken still matters: random repairs keep static PASS but lose 16.41 pp of Verified PASS at the detailing check (Build 38.46% vs 53.59%) and the held-out seismic check. (3) The rule table on the LLM's replayed intent (R1) trails PE-Loop by 6.15 pp in Verified PASS and 10.00 pp in static PASS, so we do not claim that a frozen rule replaces the LLM's repair choices without loss.

Revised claim. We removed the sentence in Section 4 that the model supplies cross-instance experience for discrete repairs. PE-Loop's lead over the baselines comes mainly from the harness: acceptance gated by the oracle, bisection on the monotone coordinate, and escalation. Rule-based variants without any LLM come within 5 pp of PE-Loop's Verified PASS. Full results: Appendix G.2, Table 11.

AuthorsOpenReview ↗

General Response to All Reviewers (Continued)

G2: Acceptance authority versus access to evidence (nL8B W1/Q1, WZyU W1/Q2, hefu W1/Q3).

You are right that the submitted judge saw no simulator quantity, so Table 5 measured missing evidence as much as who holds the verdict; its repair feedback also came from the judge itself. We ran a 2 × 2 design (judge evidence × feedback source) on 60 pre-registered topology-decision instances × 3 seeds. Informed judges receive the oracle's computed value and limit for every check, without the pass/fail flags, and the repair feedback comes from the oracle, so only the acceptance decision changes. Every judge-accepted design went through the held-out checks.

ArmJudge seesRepair feedbackFalse-positive rateFalse negativesOracle-rechecked PASSVerified PASS
A0 margin oracle—oracle0 (by construction)086.6730.00
A1 blind judgebrief, limits, designjudge89.53% (154/172)11.38% (33/290)10.005.56
A1b blind judgebrief, limits, designoracle89.35% (151/169)10.93% (33/302)10.004.44
A3 informed judge+ computed valuesoracle10.27% (15/146)0/63072.7825.56
A4 informed judge (DeepSeek-V4-Pro)+ computed valuesoracle11.26% (17/151)0/64174.4426.67
A1, Qwen3.8-Flashbrief, limits, designjudge61.49% (99/161)34.81% (252/724)34.440.56
A3, Qwen3.8-Flash+ computed valuesoracle8.90% (13/146)0/63473.8929.44

Three observations: (1) Without simulator quantities, an LLM verdict fails on both backbones: 61–90% of its acceptances are noncompliant, and oracle feedback does not help (A1b vs A1), so the failure lies in acceptance, not in repair. (2) With the computed values, the judge reads them correctly: it makes no false negatives, and its violation sets match the oracle's (Jaccard 0.999–1.000). All 45 remaining false positives occur where the displayed value of the violated check equals its limit at the oracle's display precision, while the oracle decides on unrounded values; a stronger judge (A4) does not remove these ties. They cost A3 and A4 13.89 and 12.22 pp of static PASS against A0 (Holm p ≤ 0.004); their Verified PASS differences (−4.44 and −3.33 pp) are not significant. (3) The LLM verdict costs more: per episode, the informed judges take 2.7–2.9 min and the blind ones 10.4–10.9 min, compared with 1.7 min for the oracle (DeepSeek arms).

Revised claim. Reliability rests on adjudication computed from simulator quantities, not on which model proposes. Given those quantities, an LLM verdict adds cost and an error mode that depends on how the numbers are presented; a deterministic comparison has neither. We do not claim that a judge given full-precision values would fail, because we did not measure that. The abstract no longer says "all else fixed"; the judge table's column is renamed "Oracle-rechecked PASS" with a note that the oracle's zero false-positive rate holds by construction. Full results: Appendix G.1, Table 10.


Other revisions, with their locations in the revised manuscript.

  • Corrected numbers (WZyU W3): the DeepSeek-V4-Flash PE-Loop row of Table 1 and the reference-set coverage table (Table 8) are recomputed from per-episode records.
  • Scope of the checks (nL8B W2, WZyU W2): we withdraw the claim that the unfactored envelope is conservative; Appendix B.1 and Table 2 state the load basis and every limit.
  • Reproducibility (hefu W3): prompts in Appendix F; environment, model identifiers, and cost in Appendix H; instance provenance and the task layer in Appendix B.
  • Failure modes and Build by code family (hefu Q1, Q4): Appendix E and Appendix D.3.

We thank the reviewers again and welcome further questions during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.