acceptodds
Under review as a conference paper at ICLR 2027

SimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation

Abstract

Existing Simulink benchmarks mainly evaluate whether generated models compile, execute, or resemble a reference model. These criteria do not establish whether a model satisfies its engineering requirements. We introduce SimuVerity, a benchmark of 101 text-to-executable Simulink model-generation tasks across ten engineering domains. For each task, executable-system profiles ground the engineering specification and four families of native simulation scenarios. A hierarchical evaluator first checks artifact delivery, native executability, and engineering qualification, then scores qualified models across six dimensions covering accuracy, output quality, mechanistic fidelity, control and causal integrity, operating-domain robustness, and dynamic response. We evaluate six agent systems with SimuVerity. The best system achieves an overall score of only 42.86. The results show that structural similarity is a poor proxy for engineering performance: capability bottlenecks arise both in producing qualified implementations and in satisfying multidimensional requirements after qualification. Meanwhile, some high-scoring models still exhibit severe visual-layout disorder. SimuVerity provides a systematic basis for assessing agents' engineering capabilities and diagnosing failures in executable Simulink model generation.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Benchmark, Simulink Model Generation, Engineering Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ACQzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper evaluates LLM-generated models, developing a new testbench for this process. The paper argues that reliable Simulink-generation agents need a closed loop of specification-driven construction, native simulation, multi-condition validation, and correction—not just code or diagram generation.

The paper shows that structural similarity is not a reliable quality measure. Structurally distinct models could score well, while reference-aligned models could still fail engineering tests completely. The paper identifies robustness and dynamic behavior as major weaknesses: Output quality scored relatively highest overall, while performance under operating variations and recovery/transient conditions was weakest.

The paper shows that failures differ by domain. While some domains are held back mainly by inability to form a qualifying model, others have reasonable qualification rates but weak post-qualification performance.

Finallly, they show that simulation feedback matters substantially. On ten otherwise solvable tasks, removing agent-side simulation reduced the score from 78.46 to 7.88. Removing the full MATLAB MCP tool interface reduced it to 18.23.Engineering correctness does not ensure readable diagrams. Of 31 high-scoring models, experts judged eight severely disordered visually, with overlapping blocks and hard-to-follow signal routes.

Critical strengths

the benchmark contribution is strong and likely publishable. The evaluator, scenario design, and emphasis on engineering requirements rather than structural similarity are all valuable contributions.

The examination of the use of LLM generators for engineering models is an interesting topic and worthy of exploration.

The set of generarated models are also of value.

Critical weaknesses

The developed system is very useful for engineering domains. However, as an ML paper, the empirical analysis is currently too shallow. The paper demonstrates substantial differences between LLMs but provides almost no explanation of those differences---for a venue like ICML this is a real issue. The absence of human baselines is a serious limitation, because there is no meaningful reference point for interpreting scores such as 42.86 or 28.40.

My main review comment is: The benchmark is compelling, but the paper spends far more effort measuring performance than understanding it. For an ML conference, the most important missing ingredient is an analysis of why some models succeed and others fail, ideally supported by failure taxonomies, capability attribution studies, and comparisons against human-generated Simulink models.

From the ICML perspective, the nain concern is that the paper does benchmarking without learning anything about the models. The paper evaluates six agent systems and reports large performance differences: • Opus 4.8: 42.86 • GPT-5.5: 41.72 • DeepSeek-V4-Pro: 28.40 • Qwen3.8-Max: 25.01 • GLM-5.3-Flash: 4.98 • Qwen3.8-27B-FP8: 1.60 These differences are enormous, yet the paper never attempts to explain: • Why Opus and GPT are strong. • Why DeepSeek is intermediate. • Why GLM and Qwen-27B nearly collapse. • Which modeling capabilities differentiate the strong systems from the weak ones. This is particularly striking because Tables 2 and 3 contain exactly the sort of signal one would want to analyze. The paper effectively stops at Model A > Model B. For an ML conference, that is insufficient as a primary contribution.

The most interesting question is never asked: The benchmark naturally enables a capability decomposition. For example, the paper develops 5 failure modes in detail; None of these are analyzed. Consequently we learn very little about what the LLMs actually know. There is a lot of information potentially of interest in examining these failure modes in more detail. The questions include:

Failure mode 1: architecture synthesis Can the model infer: • plant structure, • signal flow, • feedback topology, • subsystem decomposition?

Failure mode 2: control design Can it construct: • PID loops, • state feedback, • observers, • compensators?

Failure mode 3: physical modeling Can it encode: • mass balances, • dynamics, • energy conservation, • kinematics?

Failure mode 4: Simulink tool usage Can it correctly use: • blocks, • ports, • libraries, • solvers?

Failure mode 5: debugging Can it diagnose simulation failures?

The paper treats the benchmark as the science: This is a common benchmark paper issue, but does not provide an ML audience with the necessary insights. The evaluation framework is detailed: • A/Q/M/C/R/D dimensions, • 595 scenarios, • qualification gates, • domain coverage, • native execution.

However, the scientific conclusions are mostly: • current models are not very good, • simulation feedback helps, • structural similarity is unreliable. These are useful observations, but they do not justify the enormous volume of experimental data collected. A reviewer may reasonably ask: What new understanding of LLM reasoning emerges from these experiments? I do not think the paper presently answers that.

Missing error analysis: For me this is the largest weakness. The qualification gate already separates failures into:

  1. not delivered,
  2. not executable,
  3. executable but unqualified,
  4. qualified but poor. This should lead naturally to an error taxonomy, comparing Failure against Possible cause. Yet the paper never constructs such a taxonomy.

Human baseline is badly needed: The paper includes only reference systems and LLM-generated systems. But there is no benchmark involving human engineers. This creates a major interpretability problem. Suppose Opus scores 42.86. Is that terrible? decent? near junior engineer level? near graduate student level? We have no idea. A much stronger evaluation would include a wide range of provided capabilities: Novice engineers Undergraduate students given the same specification. Graduate students Control or systems students. Professional engineers Experienced Simulink practitioners. Then we could place LLM capability on a meaningful scale. Without such baselines, the reported numbers lack context.

Human models may expose benchmark artifacts: This is perhaps even more important. The benchmark is entirely based on LLM-generated outputs. But LLM-produced Simulink models may have characteristic pathologies,e.g., excessive subsystem nesting, poor naming, unusual routing, templates copied from training data, overuse of certain block types, etc. If the benchmark is tuned, even unintentionally, to these pathologies, we may not know whether the scores reflect engineering quality or LLM-specific quirks.

In contrast, human-generated models would answer: Does the evaluator reward good engineering practice generally? or merely Does the evaluator reward certain machine-generated artifacts? The reported expert-validation study only correlates evaluator scores with expert judgments on 30 Opus-generated models. That is not sufficient validation. I'd like to see: (a) human-built models, (b) expert-built alternative implementations, (c) independently developed student solutions, all passed through the same evaluator.

The control-theoretic content is surprisingly thin Many benchmark tasks involve: • feedback control, • power electronics, • robotics, • aerospace, • automotive systems. However the analysis rarely discusses control design itself. For example: • Why are aerospace tasks difficult? • Why are battery tasks difficult? • Why do robustness and dynamic-response scores lag? The natural hypothesis would be that: LLMs are relatively good at assembling static architecture but relatively poor at dynamic closed-loop reasoning. But the paper never investigates this. That feels like a missed opportunity, especially for the controls community.

Structural similarity section is not entirely convincing: The paper argues that structural similarity is a poor proxy for engineering quality. I generally agree. However there are several issues beyond this, including: (a) similarity labels are manually assigned, (b) only Opus models are analyzed, (c) sample sizes are small, (d) categories are coarse (Reference-Aligned, Partially Similar, Structurally Distinct).

I would like a deeper analysis, addressing questions such as: (a) Which alternative architectures emerge? (b) Do stronger models generate more diverse architectures? (c) Do weaker models overfit to canonical templates? Those are much more interesting ML questions.

The tool-ablation result is obvious: The paper shows that removing simulation access causes performance to collapse. I do not find this surprising. An engineer without the ability to run simulations would also perform much worse. The more interesting question is: How is simulation feedback being used? For example: (a) debugging? (b) gain tuning? (c) architecture correction? (d)parameter estimation? The paper demonstrates the importance of simulation but not the mechanism by which simulation improves outcomes.

Questions for authors

Can you add four analyses.

  1. Failure taxonomy Characterize failures by root cause: • topology errors, • control errors, • parameterization errors, • physical-model errors, • execution errors.

  2. Human baseline Compare: • students, • practitioners, • LLMs. This would dramatically improve interpretability.

  3. Capability attribution Explain why Opus and GPT outperform others. For example: • better planning, • better debugging, • better control synthesis, • better Simulink API usage.

  4. Domain-specific analysis Investigate why certain domains are difficult: • aerospace, • batteries, • communication systems, • robotics.

This would convert the benchmark from a leaderboard into a scientific study.

16 comments
AuthorsOpenReview ↗

Response to Reviewer ACQz (1/10)

Owing to the 5,000-character limit per comment, our response is divided into ten parts (1/10–10/10), posted as consecutive comments below.

We thank the reviewer for the thorough and constructive critique, and for the positive assessment that "the benchmark contribution is strong and likely publishable." We have added detailed analyses, clarifications, and new experiments to address every question raised; our point-by-point responses follow. (Throughout, the six agent systems are abbreviated as Opus, GPT, DeepSeek, Qwen-Max, GLM, and Qwen-27B.)


1. ACQz-1 (W1+W3+Q3)

W1: "The paper demonstrates substantial differences between LLMs but provides almost no explanation of those differences … the paper never attempts to explain: Why Opus and GPT are strong. Why DeepSeek is intermediate. Why GLM and Qwen-27B nearly collapse. Which modeling capabilities differentiate the strong systems from the weak ones. This is particularly striking because Tables 2 and 3 contain exactly the sort of signal one would want to analyze. The paper effectively stops at Model A > Model B."

W3: "The paper treats the benchmark as the science … the scientific conclusions are mostly: current models are not very good, simulation feedback helps, structural similarity is unreliable. These are useful observations, but they do not justify the enormous volume of experimental data collected. A reviewer may reasonably ask: What new understanding of LLM reasoning emerges from these experiments?"

Q3: "Explain why Opus and GPT outperform others. For example: better planning, better debugging, better control synthesis, better Simulink API usage."

1.1 Differences among the LLMs

From Tables 2 and 3 of the paper, we compute stage-by-stage conditional pass rates and pair them with the root-cause breakdown developed in ACQz-2:

Table R1. Stage-wise pass rates and main failure types

SystemDelivery rateExecutable given deliveredG pass given executablePost-G meanMain failure types
Opus94.7%91.3%88.9%55.73Physical model, control
GPT98.7%96.0%79.8%55.21Control, physical model
DeepSeek97.0%90.8%69.7%46.26Topology, physical model
Qwen-Max89.8%90.1%66.5%46.50Topology, physical model
GLM96.0%50.2%27.4%37.70Execution, topology
Qwen-27B30.4%44.6%46.3%25.56Execution (mostly non-delivery)
  • Sources of the advantage of Opus and GPT: the two have nearly identical post-G means, but their advantages arise at different stages: GPT's lies in the reliability of its artifacts (the highest delivery rate and the highest executable-given-delivered rate), Opus's in satisfying the requirements (the highest G-pass rate given executable). They also excel on different tasks: on 12 tasks Opus scores ≥ 50 while GPT scores < 20, and the reverse holds on 6; taking the better of the two on each task would give a mean of 53.85.
  • Where DeepSeek falls short: both at the qualification gate (a G-pass rate given executable of 69.7%) and in post-G performance (44.8 vs. 52.5 for Opus on the 63 tasks on which both have qualified candidates).
  • Where GLM and Qwen-27B fail: GLM delivers in 96.0% of runs, but 128 of its 291 delivered models are not executable and 17 time out in simulation; 211 of Qwen-27B's 303 runs terminate without delivery.
AuthorsOpenReview ↗

Response to Reviewer ACQz (2/10)

(continued: 1. ACQz-1 (W1+W3+Q3))

1.2 Why Opus and GPT outperform the other systems (Q3)

We extracted process metrics from the event logs of all 1,878 sessions (1,818 formal runs and 60 ablation runs):

Table R2. Process metrics by system

CapabilityMetricOpusGPTDeepSeekQwen-MaxGLMQwen-27B
PlanningDecomposes requirements, interfaces, and operating conditions early in the session69.3%66.3%56.8%50.2%27.4%26.7%
DebuggingFraction of reported errors eventually resolved92.9%89.9%77.6%74.9%25.7%28.0%
Re-verificationRe-simulates after the final edit prior to delivery94.7%93.4%89.1%83.2%45.9%39.9%
Control synthesisInvokes systematic design tools on closed-loop tasks4.8%9.5%5.7%3.8%0.0%0.0%
API usageError rate of modeling API calls9.0%8.3%9.5%9.8%17.0%17.2%

Planning, debugging, and re-verification decline tier by tier and correspond to the measured outcomes: Opus has the highest planning rate and the highest G-pass rate given executable; GPT has high repair and re-verification rates and the highest executable-given-delivered rate; GLM re-verifies in only 45.9% of runs, and only 50.2% of its delivered models are executable. Simulink API usage separates only the two weakest systems: GLM and Qwen-27B have error rates of about 17%, against 8.3–9.8% for the others. Control synthesis is a shortcoming shared by all systems: none uses systematic control-design tools in more than 9.5% of runs. The advantage of Opus and GPT therefore comes from planning, debugging, and re-verification; in control synthesis they are as weak as the other systems.

1.3 New insights into LLM reasoning (W3)

  • Capability is staged: from weak to strong, the systems are limited successively by task completion, Simulink construction, structural compliance, and dynamic performance: Qwen-27B fails to deliver in 211 runs; only 50.2% of GLM's delivered models are executable; DeepSeek and Qwen-Max pass G in 66.5–69.7% of executable runs; Opus and GPT average about 55 points after G. Capability does not improve uniformly across the stages; the constraints are lifted one stage at a time.
  • The bottleneck is dynamics and robustness: the mean of C/R/D among qualified candidates (57.66) is about 11 points below that of A/Q/M (69.09), and in nine of the ten domains the lowest dimension is D, R, or C (see ACQz-5).
  • Simulation feedback is decisive: the G-pass rate of executable candidates is 12/15 with simulation and 4/10 without; No-Simulation MCP, which keeps the interactive modeling interface but disables simulation (7.88), scores below Batch-only, which removes the interface but allows batch simulation (18.23); what matters is the simulation feedback itself.
  • Correct output does not imply correct mechanism: in the battery domain, 47.3% of qualified candidates have Q ≥ 90 but M < 40, and 7 of the 39 Reference-Aligned candidates in Table 4 score 0; judging by output shape alone would overestimate capability, so mechanistic checks are indispensable.
  • Off-nominal conditions go unverified: at most 23.3% of runs simulate boundary conditions on their own before delivery, and at most 15.7% simulate fault conditions; correspondingly, 46.2% of qualified candidates fall below threshold on robustness (R).
  • Where structural innovation comes from: valid structural innovation comes mainly from changes of modeling paradigm (62% of all structural differences), and 7 structurally different candidates score no lower than their references; cross-run diversity is similar across the four main systems (31.0–35.7%), while the weaker systems reuse fixed substructures more (recurring-substructure share 35.7–51.1%) (see ACQz-7).
AuthorsOpenReview ↗

Response to Reviewer ACQz (3/10)

2. ACQz-2 (W2+W4+Q1)

W2: "The benchmark naturally enables a capability decomposition. For example, the paper develops 5 failure modes in detail; None of these are analyzed. Consequently we learn very little about what the LLMs actually know. … Failure mode 1: architecture synthesis Can the model infer: plant structure, signal flow, feedback topology, subsystem decomposition? Failure mode 2: control design Can it construct: PID loops, state feedback, observers, compensators? Failure mode 3: physical modeling Can it encode: mass balances, dynamics, energy conservation, kinematics? Failure mode 4: Simulink tool usage Can it correctly use: blocks, ports, libraries, solvers? Failure mode 5: debugging Can it diagnose simulation failures?"

W4: "Missing error analysis: For me this is the largest weakness. The qualification gate already separates failures into: 1. not delivered, 2. not executable, 3. executable but unqualified, 4. qualified but poor. This should lead naturally to an error taxonomy, comparing Failure against Possible cause. Yet the paper never constructs such a taxonomy."

Q1: "Characterize failures by root cause: topology errors, control errors, parameterization errors, physical-model errors, execution errors."

2.1 Failure taxonomy

Anchored to the four failure stages defined by the qualification gates, we assigned every one of the 1,818 runs to one of the five root causes specified in Q1:

  • Execution failure: not delivered, not executable, or simulation timeout (stages 1–2);
  • Topology error: executable, but missing required functional roles, signal paths, or output contracts, thus failing G (stage 3);
  • Physical-model error: passes G, but mechanistic fidelity (M) falls below threshold (stage 4);
  • Control error: passes M, but C, D, Q, or R falls below threshold for reasons unrelated to plant parameters (stage 4);
  • Parameterization error: falls below threshold only on objective accuracy (A), or failure stems from plant parameters (stage 4).

Attribution proceeds sequentially in the order execution → topology → physical model → control → parameterization, prioritizing more fundamental failures so that each run receives a single primary root cause. Control errors are further refined using task-level engineering quantities: of the 368 runs initially flagged as control errors, 118 were traced to incorrect plant parameters and reclassified as parameterization errors. Across all 1,672 failed runs (runs with at least one dimension below the 60-point pass threshold), the root-cause breakdown by system is:

Table R3. Root-cause distribution of failed runs by system

Root causeOpusGPTDeepSeekQwen-MaxGLMQwen-27BAll six systems
Execution failure15.1%6.3%13.4%20.9%52.5%86.5%570 (34.1%)
Topology error10.7%23.0%30.1%29.5%35.5%7.3%378 (22.6%)
Physical-model error33.9%25.0%28.6%24.8%7.0%3.0%331 (19.8%)
Control error24.4%31.7%18.2%14.0%4.0%1.3%250 (15.0%)
Parameterization error15.9%13.9%9.7%10.8%1.0%2.0%143 (8.6%)
Failed runs2712522692782993031,672

The failure distribution shifts progressively toward later stages as system capability increases: Qwen-27B and GLM fail primarily in execution (86.5% and 52.5%), DeepSeek and Qwen-Max face their largest bottleneck in topology (30.1% and 29.5%), and Opus and GPT fail predominantly in physical modeling and control (accounting for 58.3% and 56.7% of their failures, respectively).

Reviewer_W5XtOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SimuVerity, a benchmark for text-to-executable Simulink model generation comprising 101 tasks across ten engineering domains. It constructs native simulation tests from reference systems and engineering specifications, checks artifact delivery, executability, and engineering qualification, and then evaluates six dimensions of engineering performance. The authors assess six agent systems and analyze structural similarity, tool access, and diagram layout.

Critical strengths
  1. The benchmark uses engineering specifications and native simulation evidence to distinguish executable models from those that satisfy engineering requirements, capturing failures that basic compilation or structural checks may miss.

  2. The evaluation spans multiple engineering domains and reports repeated runs, dimension-level results, statistical analyses, and comparisons with expert ratings, supporting a more detailed assessment of agent capabilities.

Critical weaknesses
  1. The evaluator's engineering validity and coverage require further examination. Task-specific qualification gates, utility functions, and weights are derived from reference-system profiles and expert-defined rules, without systematic tests of alternative valid implementations or threshold sensitivity. Expert validation covers only 30 candidates from one run of one agent. Moreover, eight of 31 high-scoring models have severely disordered diagrams, while readability and maintainability are absent from the formal score. These limitations constrain the scope of the claimed engineering-grade assessment.

  2. The claim that structural similarity is a poor predictor of engineering performance is not sufficiently supported. In Table 4, reference-aligned candidates have higher engineering-qualification pass rates and end-to-end mean scores than structurally distinct candidates. Individual counterexamples show that similarity does not guarantee correctness, but do not establish weak predictive value overall. The analysis is limited to 95 candidates from one run of one agent and lacks a direct comparison with quantitative structural-similarity metrics.

Questions for authors
  1. How do the reference-derived qualification criteria and scoring rules fairly assess other valid implementations? Can the authors test multiple independent implementations and the sensitivity to scoring thresholds?

  2. Can expert validation be extended to other agents and borderline candidates? Could readability and maintainability be evaluated separately for models with high engineering scores but severely disordered diagrams?

  3. Can the authors quantify the correlation and predictive value of structural similarity for engineering performance across multiple agents and domains to substantiate this central claim?

  4. Can the authors explain the absence of domain-specific systems such as SimuAgent and report token usage, tool calls, and runtime for each agent system to contextualize the rankings and reproduction cost?

16 comments
AuthorsOpenReview ↗

Response to Reviewer W5Xt (1/5)

Owing to the 5,000-character limit per comment, our response is divided into five parts (1/5–5/5), posted as consecutive comments below.

We thank the reviewer for the constructive assessment and for recognizing how SimuVerity uses engineering specifications and native simulation evidence to distinguish merely executable models from models that satisfy engineering requirements. In response to your comments, we have conducted additional experiments and full-population analyses; our point-by-point responses follow. (Throughout, the six agent systems are abbreviated as Opus, GPT, DeepSeek, Qwen-Max, GLM, and Qwen-27B.)


1. W5Xt-1 (W1+Q1)

W1: "Task-specific qualification gates, utility functions, and weights are derived from reference-system profiles and expert-defined rules, without systematic tests of alternative valid implementations …"

Q1: "How do the reference-derived qualification criteria and scoring rules fairly assess other valid implementations? Can the authors test multiple independent implementations …?"

1.1 How the evaluator fairly assesses alternative valid implementations

The engineering qualification gate G verifies three properties: whether the functional roles required by the specification are present, whether the required signal paths are connected, and whether the outputs satisfy the output contract. None of these checks requires the model's internal structure to match the reference. Likewise, the utility functions operate strictly on final engineering quantities extracted from native simulation (e.g., tracking error and settling time); the reference system is used solely to establish the task profile—namely, the system boundary, interfaces, operating envelope, and dynamic characteristics. Empirical results already in the paper reflect this design: in Table 4, Structurally Distinct candidates achieve a post-G mean score of 61.20, which is no lower than that of Reference-Aligned candidates (59.80), with structurally distinct solutions on XMECH-11 and XAERO-07 scoring 85.10 and 80.07, respectively.

1.2 Independent-implementation experiment

We invited two experts who had not participated in designing the evaluator to construct 30 alternative implementations across 20 tasks spanning all ten domains (one easy and one hard task per domain). Each implementation was deliberately designed to depart from the reference in modeling paradigm (e.g., replacing a signal-flow diagram with a Simscape physical network, or implementing the plant via MATLAB Function code), control structure, or both. All 30 models were evaluated by the frozen evaluator:

Table R1. Scores of expert-built alternative implementations

Type of alternativeCountPass GMean score
Different modeling paradigm1212/1283.18
Different control structure1010/1090.11
Both88/885.39
Total3030/3086.08

Structurally distinct implementations achieve high scores

All 30 implementations that deliberately depart from the reference architecture pass G, achieving an overall mean score of 86.08; for example, the paradigm-substituted implementation of XTHERM-09 scores 94.67, closely matching the reference score of 94.86.

Scores do not decrease with greater structural departure

The group departing furthest from the reference ("Both") achieves a mean score of 85.39, higher than the "Different modeling paradigm" group (83.18), while the "Different control structure" group averages 90.11. Thus, the evaluator fairly assesses valid engineering implementations regardless of structural departure from the reference.

AuthorsOpenReview ↗

Response to Reviewer W5Xt (2/5)

2. W5Xt-2 (W1+Q1)

W1: "… without systematic tests of … threshold sensitivity."

Q1: "Can the authors test … the sensitivity to scoring thresholds?"

Performance thresholds in SimuVerity are grounded in the tolerances and performance indices specified for each task. Using the native engineering quantities logged during evaluation, we uniformly tightened and relaxed all performance tolerance bands by 10%, 20%, and 30%, and rescored all 1,818 runs (requiring no re-simulation):

Table R2. Overall scores under perturbed tolerance bands

SystemTightened 30%Tightened 20%Tightened 10%OriginalRelaxed 10%Relaxed 20%Relaxed 30%
Opus38.6240.3041.7742.8643.8944.7045.39
GPT38.0039.5340.7941.7242.9243.8044.53
DeepSeek25.9026.8527.6428.4029.1630.1830.83
Qwen-Max22.8923.7024.3525.0125.8026.6027.30
GLM4.574.754.884.985.075.405.52
Qwen-27B1.291.491.551.601.781.932.00

Across all six perturbations, the system ranking remains completely invariant relative to the original ranking (Kendall τ = 1.00 in all cases), preserving all four performance tiers reported in the paper. All systems shift monotonically in the same direction, the largest change is about 4.2 points, and G decisions are unaffected.


3. W5Xt-3 (W1+Q2)

W1: "Expert validation covers only 30 candidates from one run of one agent."

Q2: "Can expert validation be extended to other agents and borderline candidates?"

3.1 Design

Following the blind-rating protocol in the paper, we expanded the validation sample to cover all six systems across all three runs, totaling 140 candidates (the original 30 from Opus plus 30, 30, 30, 10, and 10 newly sampled candidates). Each of the four main systems contributes 30 candidates sampled via the paper's five-stratum scheme (8 high-scoring, 5 mid-scoring, 5 low-scoring nonzero, 5 G-passing with score 0, and 7 failing G), with Opus retaining its original 30. Because GLM and Qwen-27B have only 25 and 9 nonzero-scoring candidates in total, 10 candidates were drawn from each. Crucially, the expanded sample deliberately targets the two types of borderline cases that are most challenging to grade: near-gate candidates (failing G despite achieving ≥ 50 on all diagnostic dimensions; 19 sampled out of 23 in the full population) and mid-range candidates scoring 40–70 (25 candidates). Two domain experts who had not participated in evaluator design and were blind to evaluator scores independently rated each candidate on a 1–5 scale using only the task statement, model structure, simulation curves, and logs.

3.2 Results

Table R3. Agreement statistics of the expanded blind rating

StatisticValue
Spearman ρ between evaluator score and mean expert rating0.944 (95% CI [0.925, 0.957])
Spearman ρ between the two experts (quadratic-weighted κ)0.920 (0.919)
Mean absolute error between evaluator (rescaled to 1–5) and mean expert rating0.339
Mean absolute disagreement between the two experts0.300

Evaluator–expert agreement is at least as strong as inter-expert agreement (permutation test, p < 0.001), with ρ = 0.945 on the original 30 candidates and ρ = 0.950 on the 110 newly added candidates. High agreement holds consistently across every system:

Table R4. Evaluator–expert agreement by system

SystemCandidatesρ (evaluator vs. experts)
Opus300.945
GPT300.958
DeepSeek300.958
Qwen-Max300.969
GLM100.966
Qwen-27B100.888

On the borderline candidates, the 19 near-gate candidates (all failing G) receive a mean expert rating of at most 2, and 14 of them receive 1; the conclusion holds for any "unqualified" cutoff between 2 and 3. On the 25 mid-range candidates scoring 40–70, ρ = 0.751; within this narrow score range, the agreement between the two experts also decreases (κ = 0.639).

Overall, mean expert ratings increase monotonically across evaluator score bands:

Table R5. Mean expert rating by evaluator score band

Evaluator score bandCandidatesMean expert rating (1–5)
0581.18
0–40 (excluding 0)212.74
40–70253.56
≥ 70364.31
AuthorsOpenReview ↗

Response to Reviewer W5Xt (3/5)

4. W5Xt-4 (W1+Q2)

W1: "… eight of 31 high-scoring models have severely disordered diagrams, while readability and maintainability are absent from the formal score. These limitations constrain the scope of the claimed engineering-grade assessment."

Q2: "Could readability and maintainability be evaluated separately for models with high engineering scores but severely disordered diagrams?"

4.1 Scope of "engineering-grade"

In the paper, engineering-grade refers to the function and performance of the generated simulation model with respect to the engineering requirements. Readability and maintainability are equally important, but they do not change simulation behavior: the layout-only reorganizations in Appendix F adjust only block positions and line routing, keeping the blocks, parameters, connections, and simulation settings fixed, yet readability improves markedly. If readability were folded into the engineering score, such layout-only adjustments would change the engineering score. We therefore report the engineering score, readability, and maintainability as three separate scores.

4.2 Readability

We evaluate readability on a 1–5 scale using the block overlap, routing-detour ratio, and line-crossing density metrics defined in Appendix F.1, with thresholds calibrated against the 101 human-built references: scores of 4–5 indicate layouts within the human-reference range; scores of 2–3 indicate that one to three metrics fall outside the human-reference range without reaching severe disorder; and a score of 1 denotes severe disorder (block overlap ≥ 0.2, or line-crossing density exceeding twice the human-reference maximum). Across all high-scoring candidates (passing G with an overall score ≥ 70), the readability distribution is as follows:

Table R6. Readability distribution of high-scoring candidates

SystemHigh-scoringScore 1 (severely disordered)Scores 2–3Scores 4–5 (within human range)
Opus7935.4%34.2%30.4%
GPT8633.7%27.9%38.4%
DeepSeek5527.3%20.0%52.7%
Qwen-Max4835.4%22.9%41.7%
GLM1154.5%9.1%36.4%

4.3 Maintainability

From each system we selected the candidates with an engineering score ≥ 70 and a readability score of 1 (36 in total: 8 from each of the four main systems and 4 from GLM; Qwen-27B has no high-scoring candidate), together with the corresponding human references (26 after de-duplication by task). Two raters independently scored four items on a 1–5 scale (inter-rater ρ = 0.873), and maintainability is the mean of the four items:

Table R7. Maintainability of severely disordered high-scoring candidates and their references

Source (n)MaintainabilityNamingHierarchy and modularitySignal labelingComments
Opus (8)3.063.314.062.622.25
GPT (8)2.972.564.313.002.00
DeepSeek (8)2.731.884.502.811.75
Qwen-Max (8)2.672.124.502.621.44
GLM (4)2.341.754.252.121.25
Corresponding human references (26)4.072.604.634.714.35

These candidates are close to the human references in hierarchy and modularity; the gap concentrates in signal labeling and comments: they are functionally correct but hard to read and to hand over. Such issues are independent of engineering function and are appropriately reported as a separate dimension.

Taken together, the two assessments show that readability and maintainability are properties independent of engineering performance, yet deficient in a substantial fraction of high-scoring candidates: 27.3–35.4% of high-scoring candidates from the four main systems have severely disordered diagrams (54.5% for GLM), while 30.4–52.7% lie within the human-reference range; the severely disordered high-scoring candidates have a maintainability of 2.34–3.06 against 4.07 for the human references, with the gap concentrated in signal labeling and comments and with hierarchy and modularity close to the references. Both properties can be improved without changing engineering behavior: after an explicit readable-layout requirement was added to the prompts, the readability of the tasks of the eight severely disordered models rose from 1 to 3 while the engineering score remained essentially unchanged (76.33 → 75.31). We therefore report both as separate dimensions and list them as directions for future improvement.

Reviewer_ufGjOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SimuVerity, a benchmark of 101 tasks across ten engineering domains along with an evaluation framework for measuring the engineering quality of the LLM generated Simulink models. Prior benchmarks mainly evaluate the generated Simulink files on whether they execute or on their similarity to reference models, neither of which necessarily correlates to engineering quality. This paper's main contributions are introducing diverse benchmark tasks in the form of executable system profiles derived from reference models, designing a hierarchical performance evaluation method with task-specific scenarios, and evaluating various LLM models according to their framework and readability of the generated Simulink models.

Critical strengths
  1. The paper is well motivated and fills a clear gap by testing whether LLMs can only generate .slx models that compile and run or whether there is actually engineering substance and quality.

  2. The independent expert validation of the evaluator showing a high agreement with their performance evaluation framework is a reasonable sanity check.

  3. The readability analysis of the generated Simulink models is an interesting addition and shows that human usability matters. The authors also make a reasonable choice to keep it separate from the evaluation dimensions since readability does not directly affect engineering quality.

  4. The evaluations were each run three times and associated with confidence intervals making the results more reliable.

  5. The conclusion about structural similarity not being a reliable indicator of engineering quality is a surprising but useful addition. However, similarity does seem to predict if the models pass the initial engineering qualification gate, but among the models that pass, there seems to be no correlation.

Critical weaknesses
  1. This paper does not evaluate the newest generation of LLMs. Additionally, only the max effort of the models is tested. Comparing effort levels could be interesting to show whether the LLMs improve with more compute or if they are really limited by capability.

  2. There should be more of an attempt to explain why the LLMs fail at generating Simulink models with high engineering quality or what specifically makes the problem so hard, given that the LLMs are clearly capable of generating executable models. For example, what are the most common failures such as wrong dynamics/parameters, etc.

  3. The readability analysis is somewhat qualitative, given that the experts call the diagrams "severely disordered", and the quantitative part seems to only be over 8 of the Simulink models and is not compared to human built files. Additionally, only 8 out of 31 Simulink models being classified as "severely disordered" does not seem entirely unreasonable. A plot showing the full range of the quality could be helpful, for example, to show whether most of the models were only slightly disordered, but still generally readable.

  4. The results are all based on only one prompt per task. Varying the prompt slightly would be a good experiment to show the sensitivity of results to prompt wording.

  5. The approach relies on simulation. Given some discussion of verification/verifying aspects, it would be useful to consider Simulink Design Verifier (SDV) for formal verification of properties in the models, especially as verification of generated models (or code) is now a crucial task given the ease of generation.

Minor Comments / Typos:

  • Figure 1, 2, and 5 do not appear to be referenced in the text.

  • The A/Q/M/C/R/D should first be introduced as Objective Accuracy (A), Output Quality (Q), etc or something along those lines.

  • Table 1 showing the task distribution is repetitive with the left panel of Figure 1. I think just keeping the figure would be better.

  • The upper case V_t(x) showing the evaluation and the lower case v_t could be confusing. Consider changing one of the variables.

  • Defining z_t,s,k in more detail could be helpful.

Questions for authors
  1. How does the originality of the generated Simulink models compare to human built versions? Do the LLMs come up with valid ideas that humans would not and are some LLMs more creative than others?

  2. What are the hypotheses for why the models do not perform as well as expected? Why is the problem so hard? What could be added or done differently to improve performance?

  3. How do you eliminate the bias of the experts since they design the scenarios and their errors could propagate into the model evaluation?

  4. The LLMs do not seem to naturally have an incentive to create organized block diagrams. If the prompt explicitly asks for a readable layout, would the LLMs produce equivalent, but better organized diagrams?

  5. Why does the reference system not get a perfect 100?


Thanks for the responses, I have updated the evaluation score.

11 comments
AuthorsOpenReview ↗

Response to Reviewer ufGj (1/8)

Owing to the 5,000-character limit per comment, our response is divided into eight parts (1/8–8/8), posted as consecutive comments below.

We thank the reviewer for the careful evaluation and for recognizing our decision to assess readability separately from engineering quality. We have conducted additional analyses and experiments to address every point raised; our point-by-point responses follow. (Throughout, the six agent systems are abbreviated as Opus, GPT, DeepSeek, Qwen-Max, GLM, and Qwen-27B.)


1. ufGj-1 (W1)

W1: "This paper does not evaluate the newest generation of LLMs. Additionally, only the max effort of the models is tested. Comparing effort levels could be interesting to show whether the LLMs improve with more compute or if they are really limited by capability."

Our formal experiments were conducted in stages between June and September 2026. At the time of evaluation, each model was the latest release from its provider and was run at the maximum reasoning effort supported (Appendix D.1, Table 12). We selected maximum effort to measure each system's capability ceiling: even under this setting, the top-performing system achieves an overall score of only 42.86, with only 26.1% of its runs scoring 70 or above. Lower effort levels would only reinforce the conclusion that current agents remain far from meeting engineering-grade requirements.

Regarding evaluation cost, each additional effort level or model requires 303 generations; for Opus, this corresponds to approximately 693 hours of agent runtime. We prioritized full coverage of all six systems across three independent runs each to enable statistical testing of between-system differences.

We release the complete task set, test scenarios, and scoring code with a frozen evaluator, ensuring a uniform evaluation protocol and reproducible results. The community can independently evaluate newly released models and alternative reasoning-effort levels; the resulting scores are directly comparable to those reported in the paper and can be integrated into the public leaderboard.

AuthorsOpenReview ↗

Response to Reviewer ufGj (2/8)

2. ufGj-2 (W2+Q2)

W2: "There should be more of an attempt to explain why the LLMs fail at generating Simulink models with high engineering quality or what specifically makes the problem so hard, given that the LLMs are clearly capable of generating executable models. For example, what are the most common failures such as wrong dynamics/parameters, etc."

Q2: "What are the hypotheses for why the models do not perform as well as expected? Why is the problem so hard? What could be added or done differently to improve performance?"

2.1 Failure attribution

We assigned each of the 1,818 runs to a single primary root cause across five categories, evaluated sequentially in the order: execution → topology → physical model → control → parameterization. Across the four main systems (Opus, GPT, DeepSeek, and Qwen-Max), there are 1,070 failed runs (defined as runs with at least one dimension below the 60-point pass threshold), distributed as follows:

Table R1. Root-cause composition of failed runs (four main systems)

Root causeCriterionShare of failed runs
Execution failureNot delivered, not executable, or simulation timeout14.1%
Topology errorExecutable, but missing required functional roles, signal paths, or output contracts (fails G)23.4%
Physical-model errorPasses G, but mechanistic fidelity (M) falls below threshold28.1%
Control errorPasses M, but C, D, R, or Q falls below threshold for reasons unrelated to plant parameters21.9%
Parameterization errorFalls below threshold only on objective accuracy (A), or failure stems from plant parameters12.5%

Dynamics errors encompass two categories: physical-model errors (incorrect plant mechanisms) and control errors (valid plant mechanisms, but closed-loop dynamics that fail to meet requirements). Physical-model errors (28.1%) and control errors (21.9%) together make dynamics errors the most common cause of failure, followed by topology errors (23.4%), whereas parameterization errors account for a smaller share (12.5%). Among failed runs from Opus and GPT, dynamics errors account for an even larger share (58.3% and 56.7%, respectively).

2.2 Why the problem is hard (Q2)

Executability is merely a prerequisite. Engineering qualification requires four conditions to hold simultaneously, each associated with a substantial failure rate:

  • Structural completeness: A model must implement all functional roles, signal paths, and output contracts required by the specification. Overall, 23.6% of executable candidates fail G; among these, 69% score 0 on all diagnostic dimensions.
  • Dynamic performance: Among qualified candidates, 18.9% fall below threshold on output quality (Q), compared with 49.4% on dynamic response (D). Within the same candidate, Q exceeds the mean of R and D by 21.8–25.4 points on average.
  • Robustness across operating conditions: Among qualified candidates, 46.2% fall below threshold on operating-domain robustness (R). Test scenarios are hidden from the agents, and only 18.0–23.3% of runs proactively simulate boundary conditions before delivery; even for the human-built references, 64.89% of their score loss occurs in boundary scenarios (see ufGj-10).
  • Simultaneous multi-dimensional constraints: Overall, 55.9% of qualified candidates fall below threshold on two or more dimensions, and geometric-mean aggregation compounds deficiencies across dimensions.

2.3 Directions for improvement

  • Control design and tuning: Provide agents with professional control-design and tuning tools to support iterative tuning in closed-loop simulation; currently, at most 9.5% of runs on closed-loop tasks invoke systematic design tools.
  • Operating-condition coverage: Automatically synthesize self-verification simulations for boundary and fault conditions based on the operating ranges specified in the prompt.
  • Structural and execution verification: Verify required functional roles, signal paths, and output contracts item by item prior to delivery, and integrate block-library and API retrieval to reduce execution failures.

These directions are supported by our ablation and trajectory analyses: removing simulation feedback in the ablation reduces the overall score from 78.46 to 7.88, and the error-repair rates (89.9–92.9%) and pre-delivery re-verification rates (93.4–94.7%) of Opus and GPT are substantially higher than those of GLM and Qwen-27B (25.7–28.0% and 39.9–45.9%, respectively).

AuthorsOpenReview ↗

Response to Reviewer ufGj (3/8)

3. ufGj-3 (W3)

W3: "The readability analysis is somewhat qualitative, given that the experts call the diagrams 'severely disordered', and the quantitative part seems to only be over 8 of the Simulink models and is not compared to human built files. Additionally, only 8 out of 31 Simulink models being classified as 'severely disordered' does not seem entirely unreasonable. A plot showing the full range of the quality could be helpful, for example, to show whether most of the models were only slightly disordered, but still generally readable."

3.1 Scoring method

We computed the three layout metrics defined in Appendix F.1 across all delivered candidates:

  • Block overlap: the fraction of top-level blocks that overlap another block with positive area (boundary contact alone is excluded);
  • Routing-detour ratio: for directed signal lines, the ratio of routed path length to the Euclidean distance between endpoints;
  • Line-crossing density: the number of crossings between distinct signal networks per 100 line segments.

All scoring thresholds are anchored to the metric distributions of the 101 human-built reference models, so that each score reflects layout quality relative to human expert designs:

Table R2. Readability scoring criteria

ScoreCriterion
5All three metrics no worse than the 75th percentile of the human references
4All three metrics within the human-reference range (no worse than the worst reference)
3One metric outside the human-reference range
2Two or three metrics outside the human-reference range
1 (severely disordered)Block overlap ≥ 0.2, or line-crossing density exceeding twice the maximum of the human references

The grading rule first checks whether all three metrics lie within the human-reference range (scores 4–5), next checks whether the severe-disorder criterion is triggered (score 1), and otherwise assigns a score of 2 or 3 according to the number of out-of-range metrics. All eight models identified as severely disordered by experts in Appendix F (block overlap 0.240–0.971) receive a score of 1 under this rule.

3.2 Full distribution

Table R3. Readability distribution and correlation with the engineering score

SourceCandidatesScore 1Scores 2–3Scores 4–5ρ with engineering score
Human references1010%0%100%—
Opus28734.8%29.3%35.9%−0.048
GPT29928.4%32.8%38.8%−0.041
DeepSeek29430.6%29.3%40.1%0.026
Qwen-Max27234.9%23.9%41.2%0.079
GLM29144.0%28.9%27.1%−0.087
Qwen-27B9248.9%27.2%23.9%−0.195

Across the four main systems, 35.9–41.2% of delivered candidates fall within the human-reference range, while 28.4–34.9% are severely disordered. The 8-of-31 proportion reported in the paper is therefore not an isolated case: among high-scoring candidates across the four main systems, 27.3–35.4% are severely disordered (54.5% for GLM).

Readability is essentially unrelated to the engineering score: |ρ| ≤ 0.087 for five of the six systems, and the exception, Qwen-27B, arises because most of its candidates score 0. This supports reporting readability separately from the engineering score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Discussion and Revisions

We thank the three reviewers and the area chair for their time and constructive feedback during the discussion period.

Reviewers W5Xt and ACQz both acknowledged that our responses and the revision addressed most of their concerns and raised their ratings to 3, Weak acceptance (Reviewer ACQz has committed to raising the score but has not yet updated it in the system). We have also responded to all comments of Reviewer ufGj, and the corresponding changes are included in the revised paper; we have not yet received the reviewer's feedback on our responses and the revision.

Following the reviewers' comments, we have substantially revised the paper, keeping the main text within the 10-page limit and marking all new content in blue. The main issues and changes are:

  • Validity of the evaluator (all three reviewers). The original expert validation covered only 30 Opus candidates. We extended the blind expert rating to 140 candidates from all six systems, including borderline cases, and the evaluator agrees with the experts at ρ = 0.944, above the 0.920 between the experts. Rankings are unchanged under ±30% tolerance changes, and all 30 expert-built alternative implementations pass G (Section 3.4; Appendices C.4–C.6).
  • Failure causes and system differences (Reviewers ufGj and ACQz). A root-cause classification of all 1,818 runs, with secondary causes and process metrics, explains why the agents fail and why the systems differ (Section 4.4, Table 3; Appendix D.6).
  • Closed-loop dynamics and domain difficulty (Reviewers ACQz and W5Xt). Among qualified candidates, the dynamic dimensions score 11.43 points below the static ones, a gap absent for the references, expert-built implementations, and graduate students; the shortfall thus lies in the agents' closed-loop dynamic reasoning and is unrelated to scenario difficulty. Root causes by domain explain the difficulty of domains such as aerospace and batteries (Section 4.4; Appendices D.6–D.7).
  • Human baseline (Reviewers ACQz and W5Xt). On a 20-task subset, references, expert-built implementations, and graduate-student solutions were evaluated under the same evaluator. Graduate students outscore the best agent, mainly on the hard tasks, which the paper notes were selected by agent scores (Section 4.2; Appendix D.4).
  • Structural similarity (all three reviewers). We extended the structural-similarity analysis to all 1,535 delivered candidates of the six systems and quantified similarity from block composition and connection structure: it predicts G passage only weakly (AUC = 0.565) and is unrelated to post-G scores and expert ratings, which shows that it does not reliably reflect engineering quality (Section 4.3; Appendix E.3).
  • Readability and maintainability (Reviewers ufGj and W5Xt). Readability is now scored for all delivered models against the 101 human references; 27.3–35.4% of the high-scoring candidates of the four strongest systems are severely disordered. We also added maintainability ratings and an explicit layout-requirement experiment (Section 4.5; Appendix F.3).
  • Other requests (all three reviewers). We reported that 92.25% of the scoring indicators have a basis in the public prompts and that the tolerances are set by domain experts and not taken from the reference outputs (Appendix C.8); analyzed how simulation feedback is used (Section 4.6; Appendix G.3); added an SDV study (Appendix C.9), per-run cost (Appendix D.2), and model versions (Section 4.1); and fixed figure references and notation (Sections 1, 3, and 4).

In total, the revision adds one table to the main text and 12 subsections with 22 tables to the appendix. The remaining materials will be released with the benchmark.

We thank the reviewers and the area chair again for their engagement, which has substantially strengthened the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.