- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper evaluates LLM-generated models, developing a new testbench for this process. The paper argues that reliable Simulink-generation agents need a closed loop of specification-driven construction, native simulation, multi-condition validation, and correction—not just code or diagram generation.
The paper shows that structural similarity is not a reliable quality measure. Structurally distinct models could score well, while reference-aligned models could still fail engineering tests completely. The paper identifies robustness and dynamic behavior as major weaknesses: Output quality scored relatively highest overall, while performance under operating variations and recovery/transient conditions was weakest.
The paper shows that failures differ by domain. While some domains are held back mainly by inability to form a qualifying model, others have reasonable qualification rates but weak post-qualification performance.
Finallly, they show that simulation feedback matters substantially. On ten otherwise solvable tasks, removing agent-side simulation reduced the score from 78.46 to 7.88. Removing the full MATLAB MCP tool interface reduced it to 18.23.Engineering correctness does not ensure readable diagrams. Of 31 high-scoring models, experts judged eight severely disordered visually, with overlapping blocks and hard-to-follow signal routes.
Critical strengths
the benchmark contribution is strong and likely publishable. The evaluator, scenario design, and emphasis on engineering requirements rather than structural similarity are all valuable contributions.
The examination of the use of LLM generators for engineering models is an interesting topic and worthy of exploration.
The set of generarated models are also of value.
Critical weaknesses
The developed system is very useful for engineering domains. However, as an ML paper, the empirical analysis is currently too shallow. The paper demonstrates substantial differences between LLMs but provides almost no explanation of those differences---for a venue like ICML this is a real issue. The absence of human baselines is a serious limitation, because there is no meaningful reference point for interpreting scores such as 42.86 or 28.40.
My main review comment is: The benchmark is compelling, but the paper spends far more effort measuring performance than understanding it. For an ML conference, the most important missing ingredient is an analysis of why some models succeed and others fail, ideally supported by failure taxonomies, capability attribution studies, and comparisons against human-generated Simulink models.
From the ICML perspective, the nain concern is that the paper does benchmarking without learning anything about the models. The paper evaluates six agent systems and reports large performance differences: • Opus 4.8: 42.86 • GPT-5.5: 41.72 • DeepSeek-V4-Pro: 28.40 • Qwen3.8-Max: 25.01 • GLM-5.3-Flash: 4.98 • Qwen3.8-27B-FP8: 1.60 These differences are enormous, yet the paper never attempts to explain: • Why Opus and GPT are strong. • Why DeepSeek is intermediate. • Why GLM and Qwen-27B nearly collapse. • Which modeling capabilities differentiate the strong systems from the weak ones. This is particularly striking because Tables 2 and 3 contain exactly the sort of signal one would want to analyze. The paper effectively stops at Model A > Model B. For an ML conference, that is insufficient as a primary contribution.
The most interesting question is never asked: The benchmark naturally enables a capability decomposition. For example, the paper develops 5 failure modes in detail; None of these are analyzed. Consequently we learn very little about what the LLMs actually know. There is a lot of information potentially of interest in examining these failure modes in more detail. The questions include:
Failure mode 1: architecture synthesis Can the model infer: • plant structure, • signal flow, • feedback topology, • subsystem decomposition?
Failure mode 2: control design Can it construct: • PID loops, • state feedback, • observers, • compensators?
Failure mode 3: physical modeling Can it encode: • mass balances, • dynamics, • energy conservation, • kinematics?
Failure mode 4: Simulink tool usage Can it correctly use: • blocks, • ports, • libraries, • solvers?
Failure mode 5: debugging Can it diagnose simulation failures?
The paper treats the benchmark as the science: This is a common benchmark paper issue, but does not provide an ML audience with the necessary insights. The evaluation framework is detailed: • A/Q/M/C/R/D dimensions, • 595 scenarios, • qualification gates, • domain coverage, • native execution.
However, the scientific conclusions are mostly: • current models are not very good, • simulation feedback helps, • structural similarity is unreliable. These are useful observations, but they do not justify the enormous volume of experimental data collected. A reviewer may reasonably ask: What new understanding of LLM reasoning emerges from these experiments? I do not think the paper presently answers that.
Missing error analysis: For me this is the largest weakness. The qualification gate already separates failures into:
- not delivered,
- not executable,
- executable but unqualified,
- qualified but poor. This should lead naturally to an error taxonomy, comparing Failure against Possible cause. Yet the paper never constructs such a taxonomy.
Human baseline is badly needed: The paper includes only reference systems and LLM-generated systems. But there is no benchmark involving human engineers. This creates a major interpretability problem. Suppose Opus scores 42.86. Is that terrible? decent? near junior engineer level? near graduate student level? We have no idea. A much stronger evaluation would include a wide range of provided capabilities: Novice engineers Undergraduate students given the same specification. Graduate students Control or systems students. Professional engineers Experienced Simulink practitioners. Then we could place LLM capability on a meaningful scale. Without such baselines, the reported numbers lack context.
Human models may expose benchmark artifacts: This is perhaps even more important. The benchmark is entirely based on LLM-generated outputs. But LLM-produced Simulink models may have characteristic pathologies,e.g., excessive subsystem nesting, poor naming, unusual routing, templates copied from training data, overuse of certain block types, etc. If the benchmark is tuned, even unintentionally, to these pathologies, we may not know whether the scores reflect engineering quality or LLM-specific quirks.
In contrast, human-generated models would answer: Does the evaluator reward good engineering practice generally? or merely Does the evaluator reward certain machine-generated artifacts? The reported expert-validation study only correlates evaluator scores with expert judgments on 30 Opus-generated models. That is not sufficient validation. I'd like to see: (a) human-built models, (b) expert-built alternative implementations, (c) independently developed student solutions, all passed through the same evaluator.
The control-theoretic content is surprisingly thin Many benchmark tasks involve: • feedback control, • power electronics, • robotics, • aerospace, • automotive systems. However the analysis rarely discusses control design itself. For example: • Why are aerospace tasks difficult? • Why are battery tasks difficult? • Why do robustness and dynamic-response scores lag? The natural hypothesis would be that: LLMs are relatively good at assembling static architecture but relatively poor at dynamic closed-loop reasoning. But the paper never investigates this. That feels like a missed opportunity, especially for the controls community.
Structural similarity section is not entirely convincing: The paper argues that structural similarity is a poor proxy for engineering quality. I generally agree. However there are several issues beyond this, including: (a) similarity labels are manually assigned, (b) only Opus models are analyzed, (c) sample sizes are small, (d) categories are coarse (Reference-Aligned, Partially Similar, Structurally Distinct).
I would like a deeper analysis, addressing questions such as: (a) Which alternative architectures emerge? (b) Do stronger models generate more diverse architectures? (c) Do weaker models overfit to canonical templates? Those are much more interesting ML questions.
The tool-ablation result is obvious: The paper shows that removing simulation access causes performance to collapse. I do not find this surprising. An engineer without the ability to run simulations would also perform much worse. The more interesting question is: How is simulation feedback being used? For example: (a) debugging? (b) gain tuning? (c) architecture correction? (d)parameter estimation? The paper demonstrates the importance of simulation but not the mechanism by which simulation improves outcomes.
Questions for authors
Can you add four analyses.
-
Failure taxonomy Characterize failures by root cause: • topology errors, • control errors, • parameterization errors, • physical-model errors, • execution errors.
-
Human baseline Compare: • students, • practitioners, • LLMs. This would dramatically improve interpretability.
-
Capability attribution Explain why Opus and GPT outperform others. For example: • better planning, • better debugging, • better control synthesis, • better Simulink API usage.
-
Domain-specific analysis Investigate why certain domains are difficult: • aerospace, • batteries, • communication systems, • robotics.
This would convert the benchmark from a leaderboard into a scientific study.
16 comments
Response to Reviewer ACQz (2/10)
(continued: 1. ACQz-1 (W1+W3+Q3))
1.2 Why Opus and GPT outperform the other systems (Q3)
We extracted process metrics from the event logs of all 1,878 sessions (1,818 formal runs and 60 ablation runs):
Table R2. Process metrics by system
| Capability | Metric | Opus | GPT | DeepSeek | Qwen-Max | GLM | Qwen-27B |
|---|---|---|---|---|---|---|---|
| Planning | Decomposes requirements, interfaces, and operating conditions early in the session | 69.3% | 66.3% | 56.8% | 50.2% | 27.4% | 26.7% |
| Debugging | Fraction of reported errors eventually resolved | 92.9% | 89.9% | 77.6% | 74.9% | 25.7% | 28.0% |
| Re-verification | Re-simulates after the final edit prior to delivery | 94.7% | 93.4% | 89.1% | 83.2% | 45.9% | 39.9% |
| Control synthesis | Invokes systematic design tools on closed-loop tasks | 4.8% | 9.5% | 5.7% | 3.8% | 0.0% | 0.0% |
| API usage | Error rate of modeling API calls | 9.0% | 8.3% | 9.5% | 9.8% | 17.0% | 17.2% |
Planning, debugging, and re-verification decline tier by tier and correspond to the measured outcomes: Opus has the highest planning rate and the highest G-pass rate given executable; GPT has high repair and re-verification rates and the highest executable-given-delivered rate; GLM re-verifies in only 45.9% of runs, and only 50.2% of its delivered models are executable. Simulink API usage separates only the two weakest systems: GLM and Qwen-27B have error rates of about 17%, against 8.3–9.8% for the others. Control synthesis is a shortcoming shared by all systems: none uses systematic control-design tools in more than 9.5% of runs. The advantage of Opus and GPT therefore comes from planning, debugging, and re-verification; in control synthesis they are as weak as the other systems.
1.3 New insights into LLM reasoning (W3)
- Capability is staged: from weak to strong, the systems are limited successively by task completion, Simulink construction, structural compliance, and dynamic performance: Qwen-27B fails to deliver in 211 runs; only 50.2% of GLM's delivered models are executable; DeepSeek and Qwen-Max pass G in 66.5–69.7% of executable runs; Opus and GPT average about 55 points after G. Capability does not improve uniformly across the stages; the constraints are lifted one stage at a time.
- The bottleneck is dynamics and robustness: the mean of C/R/D among qualified candidates (57.66) is about 11 points below that of A/Q/M (69.09), and in nine of the ten domains the lowest dimension is D, R, or C (see ACQz-5).
- Simulation feedback is decisive: the G-pass rate of executable candidates is 12/15 with simulation and 4/10 without; No-Simulation MCP, which keeps the interactive modeling interface but disables simulation (7.88), scores below Batch-only, which removes the interface but allows batch simulation (18.23); what matters is the simulation feedback itself.
- Correct output does not imply correct mechanism: in the battery domain, 47.3% of qualified candidates have Q ≥ 90 but M < 40, and 7 of the 39 Reference-Aligned candidates in Table 4 score 0; judging by output shape alone would overestimate capability, so mechanistic checks are indispensable.
- Off-nominal conditions go unverified: at most 23.3% of runs simulate boundary conditions on their own before delivery, and at most 15.7% simulate fault conditions; correspondingly, 46.2% of qualified candidates fall below threshold on robustness (R).
- Where structural innovation comes from: valid structural innovation comes mainly from changes of modeling paradigm (62% of all structural differences), and 7 structurally different candidates score no lower than their references; cross-run diversity is similar across the four main systems (31.0–35.7%), while the weaker systems reuse fixed substructures more (recurring-substructure share 35.7–51.1%) (see ACQz-7).
Response to Reviewer ACQz (3/10)
2. ACQz-2 (W2+W4+Q1)
W2: "The benchmark naturally enables a capability decomposition. For example, the paper develops 5 failure modes in detail; None of these are analyzed. Consequently we learn very little about what the LLMs actually know. … Failure mode 1: architecture synthesis Can the model infer: plant structure, signal flow, feedback topology, subsystem decomposition? Failure mode 2: control design Can it construct: PID loops, state feedback, observers, compensators? Failure mode 3: physical modeling Can it encode: mass balances, dynamics, energy conservation, kinematics? Failure mode 4: Simulink tool usage Can it correctly use: blocks, ports, libraries, solvers? Failure mode 5: debugging Can it diagnose simulation failures?"
W4: "Missing error analysis: For me this is the largest weakness. The qualification gate already separates failures into: 1. not delivered, 2. not executable, 3. executable but unqualified, 4. qualified but poor. This should lead naturally to an error taxonomy, comparing Failure against Possible cause. Yet the paper never constructs such a taxonomy."
Q1: "Characterize failures by root cause: topology errors, control errors, parameterization errors, physical-model errors, execution errors."
2.1 Failure taxonomy
Anchored to the four failure stages defined by the qualification gates, we assigned every one of the 1,818 runs to one of the five root causes specified in Q1:
- Execution failure: not delivered, not executable, or simulation timeout (stages 1–2);
- Topology error: executable, but missing required functional roles, signal paths, or output contracts, thus failing G (stage 3);
- Physical-model error: passes G, but mechanistic fidelity (M) falls below threshold (stage 4);
- Control error: passes M, but C, D, Q, or R falls below threshold for reasons unrelated to plant parameters (stage 4);
- Parameterization error: falls below threshold only on objective accuracy (A), or failure stems from plant parameters (stage 4).
Attribution proceeds sequentially in the order execution → topology → physical model → control → parameterization, prioritizing more fundamental failures so that each run receives a single primary root cause. Control errors are further refined using task-level engineering quantities: of the 368 runs initially flagged as control errors, 118 were traced to incorrect plant parameters and reclassified as parameterization errors. Across all 1,672 failed runs (runs with at least one dimension below the 60-point pass threshold), the root-cause breakdown by system is:
Table R3. Root-cause distribution of failed runs by system
| Root cause | Opus | GPT | DeepSeek | Qwen-Max | GLM | Qwen-27B | All six systems |
|---|---|---|---|---|---|---|---|
| Execution failure | 15.1% | 6.3% | 13.4% | 20.9% | 52.5% | 86.5% | 570 (34.1%) |
| Topology error | 10.7% | 23.0% | 30.1% | 29.5% | 35.5% | 7.3% | 378 (22.6%) |
| Physical-model error | 33.9% | 25.0% | 28.6% | 24.8% | 7.0% | 3.0% | 331 (19.8%) |
| Control error | 24.4% | 31.7% | 18.2% | 14.0% | 4.0% | 1.3% | 250 (15.0%) |
| Parameterization error | 15.9% | 13.9% | 9.7% | 10.8% | 1.0% | 2.0% | 143 (8.6%) |
| Failed runs | 271 | 252 | 269 | 278 | 299 | 303 | 1,672 |
The failure distribution shifts progressively toward later stages as system capability increases: Qwen-27B and GLM fail primarily in execution (86.5% and 52.5%), DeepSeek and Qwen-Max face their largest bottleneck in topology (30.1% and 29.5%), and Opus and GPT fail predominantly in physical modeling and control (accounting for 58.3% and 56.7% of their failures, respectively).
Response to Reviewer ACQz (1/10)
Owing to the 5,000-character limit per comment, our response is divided into ten parts (1/10–10/10), posted as consecutive comments below.
We thank the reviewer for the thorough and constructive critique, and for the positive assessment that "the benchmark contribution is strong and likely publishable." We have added detailed analyses, clarifications, and new experiments to address every question raised; our point-by-point responses follow. (Throughout, the six agent systems are abbreviated as Opus, GPT, DeepSeek, Qwen-Max, GLM, and Qwen-27B.)
1. ACQz-1 (W1+W3+Q3)
1.1 Differences among the LLMs
From Tables 2 and 3 of the paper, we compute stage-by-stage conditional pass rates and pair them with the root-cause breakdown developed in ACQz-2:
Table R1. Stage-wise pass rates and main failure types