acceptodds
Under review as a conference paper at ICLR 2027

EVOSYSML: TRAINING A SPECIALIZED MODEL FOR SYSML V2 CODE GENERATION

Abstract

Automatically generating SysML v2 code from natural-language requirements can reduce the cost of model-based systems engineering, but progress is hindered by scarce requirement–code data and the limited SysML knowledge of general-purpose language models. We present EvoSysML, a framework for training and deploying a specialized SysML generator, which consists of four stages. (1) An API teacher constructs cold-start data through breadth and depth evolution. (2) Local role models continue both processes, reconstruct aligned requirements, and filter candidates into a validated 50K training corpus. (3) Supervised fine-tuning and syntax-directed direct preference optimization train the final generator. (4) At deployment, formal validation selects an output from multiple sampled candidates. We also introduce EvoSysML-204, a manually verified bilingual benchmark containing 204 aligned requirements per language across three difficulty levels. Against 12 strong baselines, EvoSysML achieves the highest average bilingual syntax pass rate using single-pass inference. Furthermore, compared to strong API models, our method can sample ten reasoning paths under an equivalent time budget, increasing syntax pass rates to 39.22% in English and 37.25% in Chinese. This outperforms the best-performing baselines in each language (Kimi-K2.7-Code and Claude-Sonnet-5) by 18.14 and 19.11 percentage points, respectively. Among syntax-valid outputs, EvoSysML achieves conditional semantic averages of 86.90% and 77.34%, ranking first in English and tying for first in Chinese. Ablation studies confirm that data evolution, two-stage post-training, and validation-guided candidate selection all contribute to the final performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
SysML code generation, model-based systems engineering, breadth–depth self-evolution, syntax-directed DPO, supervised fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_a11gOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EvoSysML, a domain-specific model training framework designed for generating SysML v2 code from natural language requirements. The approach employs an API-based model to evolve a small set of human-curated seed data—both in breadth and depth—and subsequently trains local "role-based" models to continuously expand the dataset, resulting in a training corpus of approximately 50,000 samples. During the training phase, the authors utilize Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), while the inference phase incorporates multiple-sampling and a formal validator to filter for valid code. Additionally, the authors constructed EvoSysML-204, a bilingual (Chinese-English) test set comprising 204 tasks, and benchmarked their approach against 12 baseline models. Experiments demonstrate that the method achieves high syntactic correctness in a single-generation setting, with multiple-sampling further improving the syntactic pass rate.

Critical strengths

Addressing the challenge of data scarcity in SysML v2 code generation, this paper proposes a complete framework—encompassing data synthesis, model training, and inference deployment—that provides a viable path for building domain-specific training data. The experimental evaluation is thorough, covering 12 baselines. The results indicate that domain adaptation enhances syntactic correctness, while multiple-sampling further boosts the success rate of code generation.

Critical weaknesses

1. The innovation of the core methodology and the necessity of its individual components remain insufficiently demonstrated.

The paper primarily integrates existing techniques—namely data evolution, SFT, DPO, and validator-based filtering. Although the authors have designed strategies specific to SysML, the work lacks a direct comparison against simpler data synthesis methods. Table 4 focuses primarily on the evolution ratio; it lacks a thorough ablation study to isolate the individual contributions of requirement reconstruction, semantic filtering, and multi-role evolution. Furthermore, the optimal DPO configuration only improves the average syntactic pass rate from 17.40% to 19.12%, and performance actually declines when the volume of preference data is increased. The study also lacks multiple random seeds or statistical analysis to verify the stability of these gains.

2. There are shortcomings regarding the reliability and semantic evaluation of the synthetic data.

The paper fails to provide a detailed explanation of the implementation and accuracy of the semantic validator (VsemV_{\mathrm{sem}}). While reconstructing requirements from generated code can enhance pair consistency, it does not necessarily reflect the distribution of requirements as defined by actual users. The authors also do not adequately describe the decontamination process between the full 50K training set and the test set. Moreover, metrics such as "Requirement Coverage" and "Entity Recall" focus primarily on recall, failing to sufficiently penalize the generation of erroneous entities or relationships. The semantic metrics in Table 3 are calculated only on syntactically valid outputs, meaning the evaluation sample sets differ across models. For instance, the reported English SemAvg is 86.90% for the proposed method versus 86.88% for "GPT-5.6-Sol"—a difference insufficient to demonstrate a clear semantic advantage.

3. The baseline setup and the comparison of inference budgets are inadequate to fully substantiate the claimed performance superiority.

The paper posits that general-purpose models lack domain-specific knowledge of SysML; however, it reports no experiments in which baselines were provided with official SysML v2 documentation, syntactic specifications, or relevant examples. Consequently, it remains unclear whether the performance gap stems from a lack of domain knowledge or from inherent limitations in code generation capabilities—a point that weakens the argument for the necessity of specialized training.

Questions for authors

See "Weaknesses."

  1. Could you provide a comparison against simpler data synthesis methods, keeping the total data volume and training budget constant? Additionally, could you isolate and verify the individual contributions of requirement reconstruction, semantic filtering, and multi-role evolution?
  2. How is VsemV_{\mathrm{sem}} implemented, and has its accuracy been independently verified by humans? Furthermore, what measures were taken to ensure there is no data contamination between the full training set and the test set? 3. Is it possible to incorporate SysML documentation, syntax examples, or a retrieval-augmented general model baseline, and then evaluate the effectiveness of verifier-based filtering or code repair under a comparable inference budget?
4 comments

Answer to Critical Weakness 1 / Q1

We thank the reviewer for distinguishing overall performance from evidence for the individual components. Our contribution is a SysML-specific data-synthesis algorithm, rather than a claim that SFT, DPO, or validation is new in isolation. Starting from a small seed pool, the breadth-evolution role creates new requirement scenarios, while the depth-evolution role enriches SysML programs with more complex modeling structures. A requirement-authoring role then rewrites each requirement from its validated program. Only pairs passing syntax, requirement–code consistency, and duplicate checks enter SFT. DPO addresses syntax errors remaining after SFT, and the deployment validator selects a syntactically valid candidate.

Comparison with simpler synthesis methods. We adapted five data-synthesis methods to SysML and trained each resulting model with 3K pairs. All models in this comparison were tested with a zero-shot, single-pass prompt. Our method achieves the highest syntax pass rate on SysMBench and EvoSysML-204 English, and the highest mean SysMEval-F1 and mean SemAvg across the three benchmarks. Instruct-SkillMix performs better on the Chinese benchmark, which we report explicitly.

Table 1. Zero-shot syntax pass rate for models trained on 3K synthesized pairs (%).

MethodSysMBenchEN-204ZH-204
Direct few-shot21.8510.786.86
Self-Instruct9.932.942.45
Code Evol-Instruct31.1310.6711.24
Instruct-SkillMix27.8111.2713.24
CoT-Self-Instruct33.778.8211.76
Ours35.1014.7011.30

Table 2. Zero-shot semantic performance for models trained on 3K synthesized pairs (%). Mean is the arithmetic mean across the three benchmarks.

GroupModel/settingSysMBench F1SysMBench SemAvgEN-204 F1EN-204 SemAvgZH-204 F1ZH-204 SemAvgMean F1Mean SemAvg
BaselineDirect few-shot29.2170.3467.7781.1862.3979.3653.1276.96
BaselineSelf-Instruct31.5269.6460.5570.7858.4770.9650.1870.46
BaselineCode Evol-Instruct34.1376.2065.7284.1262.8382.2554.2380.86
BaselineInstruct-SkillMix37.2079.8268.7587.5768.2885.2758.0884.22
BaselineCoT-Self-Instruct34.8675.0868.0486.5266.6683.9756.5281.86
OursEvoSysML data synthesis43.6883.2170.1487.5463.8684.6759.2385.14

Contribution of preference tuning and evolution roles. We evaluate preference-tuning settings and breadth–depth compositions using syntax pass rates on SysMBench and the EN and ZH EvoSysML-204 sets, together with mean F1 and mean SemAvg across the three evaluation sets. Table 3 reports these results; the equal-weight macro-average SPR is stated in the text rather than shown as a separate column. DPO-200 performs best among the preference-tuning settings, and the B:D 1:3 mixture provides the strongest overall balance between syntax and semantic performance. DPO-200 achieves the highest macro-average SPR in its group (27.98%) and the highest mean F1 and mean SemAvg (67.45% and 86.10%). Among the evolution-role settings, Depth-only has a slightly higher macro-average SPR than Breadth-only (20.53% vs. 11.10%), while Breadth-only has higher mean F1 and mean SemAvg (59.45% and 85.34% vs. 52.64% and 80.85%). The 1:3 mixture achieves a macro-average SPR of 20.36%, close to Depth-only, and semantic means of 59.23% and 85.14%, close to Breadth-only. We select 1:3 as the best overall trade-off among the tested compositions.

Table 3. Ablations of preference tuning and breadth–depth composition. SPR is reported separately for SysMBench and the EN and ZH EvoSysML-204 sets; semantic metrics report means across the three evaluation sets (%).

FactorSettingSPR: SysMBenchSPR: ENSPR: ZHMean F1Mean SemAvg
Preference tuningSFT42.5518.6316.1866.8185.64
Preference tuningDPO-10043.7118.6317.6566.7585.80
Preference tuningDPO-20045.7019.6118.6367.4586.10
Preference tuningDPO-40036.4217.6517.6566.3585.40
B:D compositionDepth-only37.0914.2210.2952.6480.85
B:D compositionB:D 1:335.1014.7111.2759.2385.14
B:D compositionB:D 1:130.4612.7511.7658.6484.72
B:D compositionB:D 3:133.1112.7510.2959.1185.22
B:D compositionBreadth-only11.2610.7811.2759.4585.34

Contribution of requirement reconstruction and semantic filtering. We compare the full 3K synthesis pipeline with the pipeline before requirement reconstruction and semantic filtering. Table 4 reports SPR for each evaluation set and mean semantic metrics across the three sets. Adding reconstruction and filtering improves mean F1 from 55.76% to 59.23% and mean SemAvg from 81.11% to 85.14%. Mean SPR increases slightly from 20.14% to 20.36%: it rises on SysMBench and ZH, with a small decrease on EN. This comparison measures the joint contribution of reconstruction and filtering; it does not separate their individual effects.

Table 4. Ablation of requirement reconstruction and semantic filtering at the 3K budget. SPR is reported per evaluation set; semantic metrics are means across the three sets (%).

SettingSPR: SysMBenchSPR: ENSPR: ZHMean F1Mean SemAvg
Before reconstruction and semantic filtering34.4415.2010.7855.7681.11
Full EvoSysML data synthesis35.1014.7111.2759.2385.14

DPO and stability. We agree that DPO’s observed gain is modest and non-monotonic. DPO-200 raises mean SPR from 17.40% to 19.12%, an improvement of 1.72 percentage points, but DPO-400 falls to 17.65%. DPO-100, rather than DPO-200, has the highest conditional SemAvg. We therefore describe DPO as a syntax-focused refinement whose outcome depends on the preference-data budget. These are single-run point estimates: without multiple seeds or uncertainty intervals, we cannot claim that the DPO gain is statistically stable. We will state this limitation in the revised ablation discussion.

Expanded benchmarks, prompts, and semantic evaluation. We contribute the bilingual EvoSysML-204 test set and extend evaluation to SysMBench, the external 151-scenario benchmark introduced by Jin et al. (2025). We adopt its SysMEval precision, recall, and F1 measures alongside requirement coverage (Req.), entity coverage (Ent.), and SemAvg = (Req. + Ent.)/2. Jin et al., A System Model Generation Benchmark from Natural Language Requirements.

We use two clearly specified test-time prompt settings:

  • Zero-shot: The system instruction asks the model to act as an MBSE engineer and return only a complete SysML v2 textual model. The user message contains the target natural-language requirement. The prompt contains no SysML documentation and no requirement–code demonstration.
  • SysML documentation + 3-shot: The prompt retains the target requirement and adds excerpts from the official SysML v2 textual-notation documentation, a local syntax checklist, compact notation examples checked with the SysML parser, and three requirement–code demonstrations that we wrote and checked for both syntax and semantic alignment. The target requirement appears after these materials and is explicitly prioritized over the demonstrations. The demonstrations are training examples, not benchmark answers.

This distinction matters because “Direct few-shot” in Table 1 names a training-data synthesis baseline; its evaluation is zero-shot, like every other row in that 3K comparison. In the revised semantic evaluation, we score every available saved generation, including syntax-invalid outputs, rather than restricting semantic scores to syntax-valid outputs as in the original manuscript. Missing generations are not treated as scored outputs. The original conditional SemAvg values in Table 3 and the revised all-output semantic values are therefore different evaluation scopes.

Human reliability check. Separately from automatic scoring of the full evaluation, two human reviewers checked 30 outputs against frozen lists containing 441 requirement facts and 351 entity roles. Req. and Ent. agreement was assessed at the underlying label level. Because reviewers can segment atomic claims differently, agreement for SysMEval-F1 was assessed through the ranking of the same 30 outputs using Spearman correlation, rather than by claiming identical atomic-claim labels.

Reliability measureReviewer 1Reviewer 2Mean
Req. label agreement88.89%85.49%87.19%
Ent. label agreement92.88%92.88%92.88%
SysMEval-F1 ranking correlation (Spearman)0.8170.8190.818

The broader model comparison is summarized below. All entries are percentages; mean semantic scores are arithmetic means across SysMBench, EvoSysML-204 English, and EvoSysML-204 Chinese.

Table 5. Syntax pass rate under zero-shot and SysML documentation + 3-shot prompting.

GroupModel/settingZero-shot SysMBenchZero-shot ENZero-shot ZHDocs + 3-shot SysMBenchDocs + 3-shot ENDocs + 3-shot ZH
General-purpose APIClaude-Sonnet-53.9719.1218.1451.6622.5525.49
General-purpose APIGPT-5.6-Sol11.2616.1817.1648.3419.6122.06
General-purpose APIDeepSeek-V4-Pro1.325.886.3750.7214.7124.02
General-purpose APIGLM-5.25.967.356.3762.2517.1619.61
Code-specializedKimi-K2.7-Code5.3021.0816.6739.0718.6320.10
Code-specializedQwen3.5-9B-Coder0.000.550.0027.017.296.19
Code-specializedSeed-Coder-8B-Instruct0.000.000.5022.5210.405.39
Locally deployableQwen3-8B0.000.000.0012.165.614.50
Locally deployableQwen3-5.9B0.000.500.5030.008.386.95
Locally deployableQwen3-8.27B0.000.981.4749.0115.6912.25
Locally deployableGemma-4-E4B-it0.000.490.0017.888.466.37
Locally deployableGLM-Z1-9B-04140.000.000.002.762.111.33
OursSingle-pass inference45.7019.6118.6363.1226.1930.90
OursRepeated sampling (10)92.0539.2237.2598.6256.4654.55

Table 6. Mean semantic performance across the three benchmarks.

GroupModel/settingZero-shot Mean F1Zero-shot Mean SemAvgDocs + 3-shot Mean F1Docs + 3-shot Mean SemAvg
General-purpose APIClaude-Sonnet-565.0390.8569.6267.63
General-purpose APIGPT-5.6-Sol63.0587.5166.9168.11
General-purpose APIDeepSeek-V4-Pro67.1285.6057.3467.12
General-purpose APIGLM-5.265.4189.4964.4268.62
Code-specializedKimi-K2.7-Code64.1889.3170.5466.45
Code-specializedQwen3.5-9B-Coder51.0567.2657.2961.17
Code-specializedSeed-Coder-8B-Instruct51.3280.3944.7361.73
Locally deployableQwen3-8B62.4159.1855.8462.80
Locally deployableQwen3-5.9B54.3764.3455.7563.27
Locally deployableQwen3-8.27B61.1280.4761.9863.58
Locally deployableGemma-4-E4B-it56.9366.6755.2064.10
Locally deployableGLM-Z1-9B-041466.1859.6551.1359.37
OursRepeated sampling (10)67.4586.1066.1769.88

Under ten-candidate sampling, our method has the highest reported syntax pass rate on all three benchmarks in both prompt settings. It also has the highest mean SysMEval-F1 in zero-shot and the highest mean SemAvg with documentation + 3-shot. These results establish broad performance; the matched synthesis comparisons and component ablations above provide the more direct evidence for the method’s design. We will revise the method, ablation, and evaluation sections to state the prompt contents, semantic-scoring denominator, component contributions, and remaining stability limitation explicitly.

Answer to Weakness 2/Q2

We agree that the original manuscript should explain semantic validation and decontamination more clearly. Our ten author-written seeds were grounded in reference cases based on real user requirements; breadth evolution preserves their core intent while creating diverse scenarios. For each synthesized SysML program, we reconstruct the requirement from the code and check its entities, attributes, relations, behaviors, constraints, values, and units before retaining the pair. The EvoSysML-204 test programs were selected separately from the official SysML v2 Pilot Implementation repository. We manually checked that their paired requirements express realistic user needs and align with the code, and screened the full 50K training corpus against the test set for exact and structural overlap. We will describe these checks and their results in the revision. To address the recall-oriented nature of Req. and Ent., we added SysMEval precision, recall, and F1, where precision penalizes unsupported generated claims. The revised evaluation scores every available generation, including syntax-invalid outputs, rather than only syntax-valid ones. In a separate 30-output human audit, the reported Req. and Ent. label agreements were 87.19% and 92.88%, and the mean Spearman correlation for F1 rankings was 0.818.

Answer to Weakness 3/Q3

We added a SysML documentation + 3-shot setting for all baselines, using official textual-notation excerpts, a syntax checklist, and three author-written requirement–code examples checked for syntax and semantic alignment. SysML-specific guidance substantially improves the API models on SysMBench: Claude-Sonnet-5 rises from 3.97% to 51.66% SPR, GPT-5.6-Sol from 11.26% to 48.34%, DeepSeek-V4-Pro from 1.32% to 50.72%, and GLM-5.2 from 5.96% to 62.25%. This indicates that limited SysML domain knowledge contributes to weak zero-shot performance. Under the same documentation + 3-shot, single-pass setting, our method achieves 63.12% / 26.19% / 30.90% SPR on SysMBench / EvoSysML-204 English / Chinese, exceeding the strongest baseline on each dataset (62.25% / 22.55% / 25.49%). We report ten-candidate results separately and assess semantic performance with the expanded metrics described above.

Reviewer_TyViOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EvoSysML, a four-stage framework for training a specialized model for SysML v2 code generation. First, it constructs EvoSysML-50K, a high-quality dataset containing over 50K validated requirement-code pairs generated through breadth and depth evolution. Next, it trains the model using supervised fine-tuning and syntax-directed preference optimization. Finally, it employs Syntax-Guided Multi-Sampling Deployment to generate syntactically valid SysML v2 code from natural language requirements.

Critical strengths
  1. Given the scarcity of high-quality requirement-code pairs for SysML v2, the paper introduces EvoSysML-50K for model training and EvoSysML-204 for bilingual evaluation, providing valuable resources for further research in SysML v2 code generation.
  2. The paper presents a comprehensive pipeline for both code dataset construction and code generation model training, offering valuable insights for code generation in other low-resource scenarios.
Critical weaknesses
  1. The introduction identifies syntactic correctness and semantic completeness as two key requirements for code generation. However, in Section 3.3, the preference optimization process relies on syntax-checking results without explicitly incorporating semantic supervision. It is unclear how the proposed method ensures that the generated code faithfully captures the requirements described in the input.

  2. Although the paper provides a data scaling analysis, the syntax pass rate continues to improve up to 50K training pairs, with no clear sign of saturation. The rationale for selecting 50K as the target dataset size therefore remains unclear, as larger datasets may yield further improvements.

Questions for authors
  1. How does EvoSysML ensure that the generated SysML v2 code semantically captures the input requirements?
  2. What is the rationale for selecting 50K training pairs as the target dataset size? Could scaling up the dataset further lead to additional improvements in code generation performance?
3 comments

Answer to Weakness 1

We thank the reviewer for pointing out that the sources of semantic and syntactic supervision were not sufficiently distinguished. Semantic alignment is established during data construction, before DPO. For each synthesized SysML v2 program, our requirement-authoring role rewrites the natural-language requirement from the validated code. The reconstructed requirement must cover all verified requirement facts and required modeling entities, including their applicable attributes, relations, behaviors, constraints, numerical values, and units. We retain a requirement–code pair for SFT only after syntax validation, semantic-consistency checking, and duplicate filtering. DPO then addresses a different remaining problem: syntactic reliability. Its positive example is a validated requirement–code pair from the SFT data, while its negative example is an SFT-generated output that fails the syntax checker. Thus, SFT learns from semantically reviewed pairs; syntax-directed DPO does not itself provide the semantic supervision. We will make this division of responsibilities explicit in Sections 3.2–3.3. To evaluate whether generated models preserve the input requirements, we extended our evaluation to SysMBench, the 151-scenario natural-language-to-SysML v2 benchmark introduced by Jin et al. (2025). We also adopted its SysMEval precision, recall, and F1 measures and report them alongside requirement coverage (Req.), entity coverage (Ent.), and SemAvg = (Req. + Ent.)/2. Jin et al., A System Model Generation Benchmark from Natural Language Requirements.

We checked the reliability of the semantic measures on 30 outputs using frozen lists of 441 requirement facts and 351 entity roles. The reported mean agreement was 87.19% for Req. labels and 92.88% for Ent. labels. The mean Spearman correlation between the two human reviewers’ SysMEval-F1 rankings was 0.818.

Reliability measureReviewer 1Reviewer 2Mean
Req. label agreement88.89%85.49%87.19%
Ent. label agreement92.88%92.88%92.88%
SysMEval-F1 ranking correlation (Spearman)0.8170.8190.818

**Evaluation prompts.**We evaluated the models with two test-time prompts. Zero-shot supplied an MBSE role instruction and the target requirement. SysML documentation + 3-shot additionally supplied official SysML v2 textual-notation excerpts, a syntax checklist, and three requirement–code examples that we wrote and checked for syntactic validity and semantic alignment. The target requirement followed these materials. Unlike the original manuscript, which calculated semantic scores only for syntax-valid outputs, the revised evaluation scores every available generated output, including syntax-invalid outputs. Thus, the semantic results describe the full set of saved generations rather than the subset that passed syntax checking. Table 1 reports syntax pass rate (SPR) for every model, and Table 2 reports the corresponding three-benchmark mean semantic scores. All values are percentages.

Table 1. Zero-shot syntax pass rate (SPR).

GroupModel/settingSysMBenchOursData-ENOursData-ZH
General-purpose APIClaude-Sonnet-53.9719.1218.14
General-purpose APIGPT-5.6-Sol11.2616.1817.16
General-purpose APIDeepSeek-V4-Pro1.325.886.37
General-purpose APIGLM-5.25.967.356.37
Code-specializedKimi-K2.7-Code5.3021.0816.67
Code-specializedQwen3.5-9B-Coder0.000.550.00
Code-specializedSeed-Coder-8B-Instruct0.000.000.50
Locally deployableQwen3-8B0.000.000.00
Locally deployableQwen3-5.9B0.000.500.50
Locally deployableQwen3-8.27B0.000.981.47
Locally deployableGemma-4-E4B-it0.000.490.00
Locally deployableGLM-Z1-9B-04140.000.000.00
OursSingle-pass inference45.7019.6118.63
OursRepeated sampling (10)92.0539.2237.25

Table 2. SysML documentation + 3-shot syntax pass rate (SPR).

GroupModel/settingSysMBenchOursData-ENOursData-ZH
General-purpose APIClaude-Sonnet-551.6622.5525.49
General-purpose APIGPT-5.6-Sol48.3419.6122.06
General-purpose APIDeepSeek-V4-Pro50.7214.7124.02
General-purpose APIGLM-5.262.2517.1619.61
Code-specializedKimi-K2.7-Code39.0718.6320.10
Code-specializedQwen3.5-9B-Coder27.017.296.19
Code-specializedSeed-Coder-8B-Instruct22.5210.405.39
Locally deployableQwen3-8B12.165.614.50
Locally deployableQwen3-5.9B30.008.386.95
Locally deployableQwen3-8.27B49.0115.6912.25
Locally deployableGemma-4-E4B-it17.888.466.37
Locally deployableGLM-Z1-9B-04142.762.111.33
OursSingle-pass inference63.1226.1930.90
OursRepeated sampling (10)98.6256.4654.55

Table 3. Zero-shot semantic performance: SysMEval-F1 and SemAvg.

GroupModel/settingSysMBench F1SysMBench SemAvgOursData-EN F1OursData-EN SemAvgOursData-ZH F1OursData-ZH SemAvgMean F1Mean SemAvg
General-purpose APIClaude-Sonnet-542.3893.9775.9988.8676.7389.7365.0390.85
General-purpose APIGPT-5.6-Sol42.5591.5371.5683.7275.0487.2763.0587.51
General-purpose APIDeepSeek-V4-Pro52.6681.8875.9087.6672.8187.2567.1285.60
General-purpose APIGLM-5.247.3392.8775.0988.0273.8087.5865.4189.49
Code-specializedKimi-K2.7-Code47.0692.8277.7988.0167.6987.1064.1889.31
Code-specializedQwen3.5-9B-Coder15.1868.1468.4767.6969.5165.9651.0567.26
Code-specializedSeed-Coder-8B-Instruct22.1082.0365.2278.2266.6480.9351.3280.39
Locally deployableQwen3-8B46.4647.4768.4363.6372.3466.4562.4159.18
Locally deployableQwen3-5.9B34.8760.7665.6564.2262.6068.0354.3764.34
Locally deployableQwen3-8.27B42.2188.9172.6777.8968.4774.6061.1280.47
Locally deployableGemma-4-E4B-it44.6783.8166.8857.5759.2458.6356.9366.67
Locally deployableGLM-Z1-9B-041456.4649.7073.6971.3568.4057.9166.1859.65
OursRepeated sampling (10)44.7684.6778.7388.2178.8585.4367.4586.10

Table 4. SysML documentation + 3-shot semantic performance: SysMEval-F1 and SemAvg.

GroupModel/settingSysMBench F1SysMBench SemAvgOursData-EN F1OursData-EN SemAvgOursData-ZH F1OursData-ZH SemAvgMean F1Mean SemAvg
General-purpose APIClaude-Sonnet-555.3137.2277.7486.6875.8279.0069.6267.63
General-purpose APIGPT-5.6-Sol45.4736.0577.7787.5377.4880.7466.9168.11
General-purpose APIDeepSeek-V4-Pro40.0738.9660.2684.8271.6877.5757.3467.12
General-purpose APIGLM-5.237.1439.0970.9386.7785.1880.0164.4268.62
Code-specializedKimi-K2.7-Code49.0133.1387.3386.7875.2979.4470.5466.45
Code-specializedQwen3.5-9B-Coder33.5127.4273.8281.4364.5574.6557.2961.17
Code-specializedSeed-Coder-8B-Instruct25.8430.5460.0280.8848.3373.7844.7361.73
Locally deployableQwen3-8B26.7229.8762.6582.3778.1476.1655.8462.80
Locally deployableQwen3-5.9B35.7630.0571.4584.4560.0375.3255.7563.27
Locally deployableQwen3-8.27B40.9430.0978.7784.1266.2276.5461.9863.58
Locally deployableGemma-4-E4B-it38.6033.4064.6983.1562.3175.7655.2064.10
Locally deployableGLM-Z1-9B-041459.8229.0643.8977.9949.6771.0551.1359.37
OursRepeated sampling (10)40.3746.8982.5485.3375.6077.4166.1769.88

Under ten-candidate sampling, our method has the highest reported syntax pass rate on all three benchmarks in both prompt settings. Its semantic results are also strong: it has the highest mean SysMEval-F1 in the zero-shot table and the highest mean SemAvg in the documentation + 3-shot table. We report the individual metrics because no method leads every semantic column.

We further compared data-synthesis methods under the same 3K training-pair budget, using zero-shot, single-pass testing for every trained model. “Direct few-shot” describes how that baseline synthesized its training pairs; it does not mean that its evaluation prompt contained demonstrations.

Table 5. Zero-shot syntax pass rate for 3K data-synthesis methods (%).

MethodSysMBenchEN-204ZH-204
Direct few-shot21.8510.786.86
Self-Instruct9.932.942.45
Code Evol-Instruct31.1310.6711.24
Instruct-SkillMix27.8111.2713.24
CoT-Self-Instruct33.778.8211.76
Ours35.1014.7011.30

Table 6. Zero-shot semantic performance of models trained on 3K synthesized pairs (%, single-pass inference). Mean denotes the arithmetic mean across the three benchmarks.

GroupModel/settingSysMBench F1SysMBench SemAvgOursData-EN F1OursData-EN SemAvgOursData-ZH F1OursData-ZH SemAvgMean F1Mean SemAvg
BaselineDirect few-shot29.2170.3467.7781.1862.3979.3653.1276.96
BaselineSelf-Instruct31.5269.6460.5570.7858.4770.9650.1870.46
BaselineCode Evol-Instruct34.1376.2065.7284.1262.8382.2554.2380.86
BaselineInstruct-SkillMix37.2079.8268.7587.5768.2885.2758.0884.22
BaselineCoT-Self-Instruct34.8675.0868.0486.5266.6683.9756.5281.86
OursEvoSysML data synthesis43.6883.2170.1487.5463.8684.6759.2385.14

At the matched 3K budget, our synthesis method achieves the highest mean F1 and mean SemAvg, as well as the highest SysMBench and English F1. Together with the requirement-reconstruction procedure, these results provide evidence for the semantic value of the constructed SFT data, while the syntax-directed DPO stage specifically targets grammatical validity.

Answer to Weakness 2

We agree that 50K is not a saturation point. EvoSysML is an iterative data-synthesis method that can continue generating and validating requirement–code pairs beyond the current corpus. The 50K validated pairs have already produced substantial improvements, as shown by the scaling results in the appendix. Further iterations may yield additional gains, but they also require more generation, validation, and training time. We therefore chose approximately 50K pairs as a practical scale for this study, and will explore larger validated corpora and their performance–cost trade-offs in future work.

Reviewer_9mTPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EVOSYSML to address the lack of training data and the limited ability of general LLMs to generate valid SysML v2 code. It builds a 50K synthetic training dataset through breadth and depth evolution and trains a specialized model with SFT and DPO. This paper also introduces EvoSysML-204, a bilingual benchmark, and reports competitive results against larger LLMs, with further gains from repeated sampling.

Critical strengths
  1. The paper addresses a practical problem in SysML v2 code generation, where training data is limited. The construction of a 50K training dataset and a bilingual benchmark could be useful for future research in this area.

  2. The experimental results are promising. Despite its relatively small size, the specialized 9B model achieves competitive single-pass syntax pass rates compared with frontier models.

Critical weaknesses
  1. The novelty is limited: Previous studies have already explored automatic generating SysML code from natural language requirements. This paper mainly combines existing techniques, including breadth, depth evolution, SFT and DPO. I didn't find a clear insight beyond applying these methods to SysML v2.

  2. The evaluation is insufficient: The experiments are conducted on the EvoSysML-204 benchmark, without testing on existing benchmarks or comparing against specialized SysML generation methods. In addition, the implementation of the semantic evaluation metrics is not clearly explained, and their reliability has not been adequately validated.

  3. Some issues in citation: There are also several incorrect or mismatched model references. In particular, I could not find an official checkpoint or corresponding paper for the Qwen3-5.9B-Coder baseline in Table 2.

Questions for authors
  1. How do the authors justify the claim in Section 5.1 that this is the first automated method for high-quality SysML generation, given previous work in this area?

  2. How are normalized facts extracted and entities matched in the semantic evaluation? Have these metrics been validated against human judgments?

  3. Could the authors add results on an existing benchmark such as SysMBench and comparisons with previous SysML v2 generation methods?

  4. Which specific model is Qwen3-5.9B-Coder in Table 2 referring to, and why does the citation point to the Qwen3.5-Omni Technical Report?

4 comments

Answer to Weakness 1

We thank the reviewer for raising this concern. Prior work has explored natural-language-to-SysML generation, but directly comparable studies have generally relied on relatively small paired corpora or focused on prompting and inference-time refinement (An et al., 2026; Dehn et al., 2025). Our contribution is the complete LLM-driven self-evolution procedure for constructing large-scale, validated native SysML v2 requirement–code pairs. We designed the functional roles, SysML-specific prompt templates, breadth and depth evolution paths, and the transition from API-generated cold-start data to iterative local-model synthesis. Crucially, after each code evolution, a requirement-authoring role reconstructs the requirement from the validated program; syntax, semantic-consistency, and duplicate checks then control admission to the next training pool. This closed-loop procedure produced 50,677 aligned pairs for training. To our knowledge, this is the first use of LLM-driven self-evolution to construct a native SysML v2 requirement–code corpus at this scale. SFT, syntax-directed DPO, and validation-guided inference build on that corpus to train and deploy the generator. We will revise the Introduction and Method sections to state this contribution and the role of each stage explicitly. Elmouhi et al., 2026.

Comparison with simpler synthesis methods. We adapted five data-synthesis methods to SysML and trained each resulting model with 3K pairs. All models in this comparison were tested with a zero-shot, single-pass prompt. Our method achieves the highest syntax pass rate on SysMBench and EvoSysML-204 English, and the highest mean SysMEval-F1 and mean SemAvg across the three benchmarks. Instruct-SkillMix performs better on the Chinese benchmark, which we report explicitly.

Table 1. Zero-shot syntax pass rate for models trained on 3K synthesized pairs (%).

MethodSysMBenchEN-204ZH-204
Direct few-shot21.8510.786.86
Self-Instruct9.932.942.45
Code Evol-Instruct31.1310.6711.24
Instruct-SkillMix27.8111.2713.24
CoT-Self-Instruct33.778.8211.76
Ours35.1014.7011.30

Table 2. Zero-shot semantic performance for models trained on 3K synthesized pairs (%). Mean is the arithmetic mean across the three benchmarks.

GroupModel/settingSysMBench F1SysMBench SemAvgEN-204 F1EN-204 SemAvgZH-204 F1ZH-204 SemAvgMean F1Mean SemAvg
BaselineDirect few-shot29.2170.3467.7781.1862.3979.3653.1276.96
BaselineSelf-Instruct31.5269.6460.5570.7858.4770.9650.1870.46
BaselineCode Evol-Instruct34.1376.2065.7284.1262.8382.2554.2380.86
BaselineInstruct-SkillMix37.2079.8268.7587.5768.2885.2758.0884.22
BaselineCoT-Self-Instruct34.8675.0868.0486.5266.6683.9756.5281.86
OursEvoSysML data synthesis43.6883.2170.1487.5463.8684.6759.2385.14

Contribution of the evolution roles. Depth-only training yields a higher mean syntax pass rate than breadth-only training (12.25% versus 11.03%), whereas breadth-only training yields a higher conditional SemAvg (83.66% versus 81.82%). The breadth:depth ratio of 1:3 gives the highest mean syntax pass rate in this ablation (12.99%). The roles therefore contribute differently: depth evolution favors formal validity in these results, while breadth evolution favors semantic coverage.

Table 3. Preference-tuning and breadth–depth ablations. Each cell reports ZH / EN / Avg (%). SemAvg in this original manuscript ablation is conditional on syntax-valid outputs.

FactorSettingSPR: ZH / EN / AvgConditional SemAvg: ZH / EN / Avg
Preference tuningSFT16.18 / 18.63 / 17.4076.83 / 85.90 / 81.37
Preference tuningDPO-10017.65 / 18.63 / 18.1477.57 / 85.43 / 81.50
Preference tuningDPO-20018.63 / 19.61 / 19.1276.53 / 84.87 / 80.70
Preference tuningDPO-40017.65 / 17.65 / 17.6577.04 / 85.10 / 81.07
B:D compositionDepth-only10.29 / 14.22 / 12.2578.25 / 85.40 / 81.82
B:D compositionBreadth:Depth 1:311.27 / 14.71 / 12.9978.85 / 85.98 / 82.42
B:D compositionBreadth:Depth 1:111.76 / 12.75 / 12.2576.87 / 85.38 / 81.13
B:D compositionBreadth:Depth 3:110.29 / 12.75 / 11.5276.96 / 88.03 / 82.50
B:D compositionBreadth-only11.27 / 10.78 / 11.0381.47 / 85.85 / 83.66

Contribution of requirement reconstruction and semantic filtering. We additionally compared the full 3K synthesis pipeline with the pipeline measured before the requirement-reconstruction and semantic-filtering stage. Adding this stage improves mean SysMEval-F1 from 55.76% to 59.23% and mean SemAvg from 81.11% to 85.14%, gains of 3.47 and 4.03 percentage points. All three benchmarks improve. This comparison measures the joint contribution of reconstruction and filtering; it does not separate their individual effects.

Table 4. Semantic ablation of requirement reconstruction and semantic filtering at the 3K budget (%).

SettingSysMBench F1SysMBench SemAvgEN-204 F1EN-204 SemAvgZH-204 F1ZH-204 SemAvgMean F1Mean SemAvg
Before reconstruction + semantic filtering39.7279.5665.7483.4261.8280.3655.7681.11
Full EvoSysML data synthesis43.6883.2170.1487.5463.8684.6759.2385.14

Role of DPO. DPO provides a narrower syntax-focused refinement: DPO-200 raises mean syntax pass rate from 17.40% after SFT to 19.12%, while DPO-400 falls to 17.65%. We therefore do not treat DPO alone as the source of the method’s novelty or semantic supervision.

Expanded benchmarks, prompts, and semantic evaluation. We contribute the bilingual EvoSysML-204 test set and extend evaluation to SysMBench, the external 151-scenario benchmark introduced by Jin et al. (2025). We adopt its SysMEval precision, recall, and F1 measures alongside requirement coverage (Req.), entity coverage (Ent.), and SemAvg = (Req. + Ent.)/2. Jin et al., A System Model Generation Benchmark from Natural Language Requirements.

We use two specified test-time prompt settings:

  • Zero-shot: The system instruction asks the model to act as an MBSE engineer and return only a complete SysML v2 textual model. The user message contains the target requirement. The prompt contains no SysML documentation or requirement–code demonstration.
  • SysML documentation + 3-shot: The prompt adds official SysML v2 textual-notation excerpts, a local syntax checklist, parser-checked notation examples, and three requirement–code demonstrations that we wrote and checked for syntax and semantic alignment. The target requirement follows these materials and is explicitly prioritized over the demonstrations. These are training examples, not benchmark answers.

“Direct few-shot” in Table 1 names a training-data synthesis method; its test-time evaluation is zero-shot, like every other row in that comparison. In the revised semantic evaluation, we score every available saved generation, including syntax-invalid outputs, rather than only syntax-valid outputs as in the original manuscript. Missing generations are not treated as scored outputs. Consequently, the original conditional SemAvg values in Table 3 and the revised all-output semantic values have different evaluation scopes.

Human reliability check. Separately from automatic scoring of the full evaluation, two human reviewers checked 30 outputs against frozen lists containing 441 requirement facts and 351 entity roles. Req. and Ent. agreement was assessed at the underlying label level. Because reviewers can segment atomic claims differently, agreement for SysMEval-F1 was assessed using the ranking of the same 30 outputs.

Reliability measureReviewer 1Reviewer 2Mean
Req. label agreement88.89%85.49%87.19%
Ent. label agreement92.88%92.88%92.88%
SysMEval-F1 ranking correlation (Spearman)0.8170.8190.818

The broader model comparison is summarized below. All entries are percentages; mean semantic scores are arithmetic means across SysMBench, EvoSysML-204 English, and EvoSysML-204 Chinese.

Table 5. Syntax pass rate under zero-shot and SysML documentation + 3-shot prompting.

GroupModel/settingZero-shot SysMBenchZero-shot ENZero-shot ZHDocs + 3-shot SysMBenchDocs + 3-shot ENDocs + 3-shot ZH
General-purpose APIClaude-Sonnet-53.9719.1218.1451.6622.5525.49
General-purpose APIGPT-5.6-Sol11.2616.1817.1648.3419.6122.06
General-purpose APIDeepSeek-V4-Pro1.325.886.3750.7214.7124.02
General-purpose APIGLM-5.25.967.356.3762.2517.1619.61
Code-specializedKimi-K2.7-Code5.3021.0816.6739.0718.6320.10
Code-specializedQwen3.5-9B-Coder0.000.550.0027.017.296.19
Code-specializedSeed-Coder-8B-Instruct0.000.000.5022.5210.405.39
Locally deployableQwen3-8B0.000.000.0012.165.614.50
Locally deployableQwen3-5.9B0.000.500.5030.008.386.95
Locally deployableQwen3-8.27B0.000.981.4749.0115.6912.25
Locally deployableGemma-4-E4B-it0.000.490.0017.888.466.37
Locally deployableGLM-Z1-9B-04140.000.000.002.762.111.33
OursSingle-pass inference45.7019.6118.6363.1226.1930.90
OursRepeated sampling (10)92.0539.2237.2598.6256.4654.55

Table 6. Mean semantic performance across the three benchmarks.

GroupModel/settingZero-shot Mean F1Zero-shot Mean SemAvgDocs + 3-shot Mean F1Docs + 3-shot Mean SemAvg
General-purpose APIClaude-Sonnet-565.0390.8569.6267.63
General-purpose APIGPT-5.6-Sol63.0587.5166.9168.11
General-purpose APIDeepSeek-V4-Pro67.1285.6057.3467.12
General-purpose APIGLM-5.265.4189.4964.4268.62
Code-specializedKimi-K2.7-Code64.1889.3170.5466.45
Code-specializedQwen3.5-9B-Coder51.0567.2657.2961.17
Code-specializedSeed-Coder-8B-Instruct51.3280.3944.7361.73
Locally deployableQwen3-8B62.4159.1855.8462.80
Locally deployableQwen3-5.9B54.3764.3455.7563.27
Locally deployableQwen3-8.27B61.1280.4761.9863.58
Locally deployableGemma-4-E4B-it56.9366.6755.2064.10
Locally deployableGLM-Z1-9B-041466.1859.6551.1359.37
OursRepeated sampling (10)67.4586.1066.1769.88

Under ten-candidate sampling, our method has the highest reported syntax pass rate on all three benchmarks in both prompt settings. It also has the highest mean SysMEval-F1 in zero-shot and the highest mean SemAvg with documentation + 3-shot. The broad comparisons establish the method’s performance; the equal-budget synthesis comparison and component ablations provide direct evidence for its design. We will revise the contribution statement and related-work discussion to make clear how code-grounded reconstruction and complementary evolution roles advance prior natural-language-to-SysML approaches.

Answer to Weakness 2

We thank the reviewer for identifying these gaps. We have extended evaluation beyond EvoSysML-204 to SysMBench, an existing benchmark of 151 human-curated scenarios introduced by Jin et al.. We now specify how Req. measures coverage of fixed requirement-fact lists, how Ent. matches required modeling roles, and how SemAvg averages them. We additionally report SysMEval precision, recall, and F1; precision penalizes unsupported generated claims. The revised semantic evaluation scores every available generated output, including syntax-invalid outputs, rather than selecting only syntax-valid cases. To assess metric reliability, two human reviewers checked 30 outputs against 441 facts and 351 entity roles: reported mean agreement was 87.19% for Req. and 92.88% for Ent., with 0.818 mean Spearman correlation for F1 rankings. We also compare five data-synthesis methods under the same 3K training budget.

Answer to Weakness 3

The reviewer is correct: “Qwen3-5.9B-Coder” is an incorrect model name in Table 2. The evaluated checkpoint was the community-released armand0e/Qwen3.5-9B-Coder, which our generation records identify by its armand0e checkpoint path. It is a fine-tune of Qwen3.5-9B, not an official Qwen Coder checkpoint with a corresponding model paper. We will correct the table and model description, cite the checkpoint page itself, and use the Qwen reference only when discussing its underlying model family. This corrects the attribution without changing the evaluated outputs.

Answer to Q1. We agree that previous work has explored natural-language-to-SysML generation, often using manually annotated examples or relatively small paired datasets. To our knowledge, EvoSysML is the first LLM-driven, large-scale data-synthesis algorithm designed specifically for native SysML v2 requirement-to-code generation. Its role-specific breadth and depth evolution, code-grounded requirement reconstruction, and syntax and semantic filtering expand ten author-written seeds into 50,677 validated requirement–code pairs. We will revise Section 5.1 to make this specific contribution clear and distinguish it from the broader claim of being the first to automate SysML generation.

Answer to Q2. We use fixed lists of requirement facts and required modeling roles. Req. measures the proportion of facts recovered from generated code; Ent. uses normalized, one-to-one matches between required roles and generated entities. In a 30-output human audit covering 441 facts and 351 roles, reported mean label agreement was 87.19% for Req. and 92.88% for Ent. The mean Spearman correlation for SysMEval-F1 rankings was 0.818. We will specify the extraction and matching rules in the revision.

Answer to Q3. We have added SysMBench, an existing benchmark with 151 scenarios, and compared five data-synthesis methods under a matched 3K training budget.

Answer to Q4. The name and citation in Table 2 are incorrect. The evaluated checkpoint was the community model armand0e/Qwen3.5-9B-Coder, not “Qwen3-5.9B-Coder.” The Qwen3.5-Omni report does not document this fine-tune. We will correct the name and cite the checkpoint page directly.

Reviewer_LAJUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an automated framework for generating SysML v2 code through a 4-stage process, EvoSysML, that involves:

  1. Synthetic data generation using a API model to cover both depth and breadth versions of requirement-code pairs
  2. Using the API-generated traces to train local models to generate the similar requirement-code pairs to form EvoSysML-50K, the training data
  3. Training final generator model through SFT and syntax-DPO using LLaMa-Factory and LoRA
  4. Final generator generates 10 samples and the first sample that passes the syntax checker is passed as the final output

They also contribute a test dataset, EvoSysML-204, which collects 204 SysML programs from the official SysML v2 repository and pair each program with aligned English and Chinese requirements. To compare their final generator model from their framework, they compared their model against 12 models (general-purpose models, code models, and local models) on EvoSysML-204.

Critical strengths

The main strength of the paper is their automated framework for generating SysML v2 requirement-code pairs, training dataset EvoSysML-50K, test dataset EvoSysML-204, and their evaluation of their framework by comparing their generator model with 12 other models that cover genera-purpose models, code models, and local models. They state that their framework addresses the gap in current SysML datasets by creating a way to automatically generate requirement-code that is formally and semantically validated. Their framework also addresses a gap in current LLMs for generating SysML code and requirements by introducing a training framework for finetuning models to generate SysML requirement-code pairs. I agree that these two contributions are important problems that need to be addressed and this paper does provide meaningful contributions towards addressing these gaps.

Critical weaknesses

In general, this paper lacks citations or basis throughout the paper supporting how and why they chose certain experimental design decisions.

4.1 Test Data: The curation method for EvoSysML-204 is very vague and does not base or reference existing work on how they curated the test data and do not mention how the dataset was validated. They also mention constructing the english and chinese requirements, but how were these requirements generated? How were the pairs actually checked for semantic alignment and what was the criteria for determining alignment?

Compared models: What is the basis for choosing these models? Have they been used in previous studies on SysML datasets/benchmarks?

Implementation details describes the training setup but does not reference any basis for the setup used. Also missing a citation for LoRA and how many GPUs are exactly used.

4.2 Small nitpick, but this section could be better written as the second line is quite confusing because it mentions the repeated sampling strategy and mentions single-pass is used like it is an after thought. It would be more forward to just say you evaluated "using single-pass inference and repeated sampling strategy..."

4.3 Mentions syntax-valid outputs from EvoSysML were compared against the top 5 performing baselines, but it is unclear how they actually "measures whether the successfully parsed code accurately reflects the original textual constraints and instantiates the necessary structural elements." They state they do so but do not mention any criteria or how they verified it.

Questions for authors
  1. Where are the citations to support certain design decisions in the method and experimental sections?
  2. how were the requirements for EvoSysML-204 generated or retrieved? How were the pairs actually checked for semantic alignment and what was the criteria for determining alignment?
  3. What is the basis for choosing these models? Have they been used in previous studies on SysML datasets/benchmarks? These questions applies to the models used in EvoSysML.
  4. How were the implementation details selected? Were they referenced from previous studies?
  5. How many GPUs were actually used to run these experiments? This is needed for reproducibility
  6. How were the syntax-valid outputs from EvoSysML actually verified against the top 5 baselines? Were they manually verified or automatically checked?
4 comments

Answer to Weakness 1

We thank the reviewer for identifying details that Section 4.1 should make explicit. EvoSysML-204 is constructed from existing SysML v2 programs, rather than from model-generated test code. We selected 204 programs from the SysML v2 Pilot Implementation repository, screened them for overlap with our training seeds, and checked their syntax. A fixed structural-complexity rubric assigned 69 programs to Low, 68 to Medium, and 67 to High difficulty before model evaluation.

Requirement generation and validation. For each selected program, DeepSeek-V4-Flash first drafted an English requirement from the code. We manually reviewed the draft against the program and revised or excluded descriptions that were inaccurate, incomplete, or too vague to specify the modeled system. We then translated the verified English requirement–code pairs into Chinese, maintained a one-to-one correspondence between language versions, and reran syntax validation on the Chinese SysML programs.

Our semantic-alignment check was performed requirement by requirement, rather than by matching entity names alone. A retained description had to cover all annotated requirement facts and required modeling entities. We checked entity roles and types; attributes; relation types and endpoints; behaviors and constraints; and specified values, units, directions, and multiplicities. We also rejected descriptions that introduced facts unsupported by the code. For example, a statement that two components are connected was accepted only when the program represented that connection with the appropriate endpoints. Thus, the 100% coverage criterion refers to the annotated facts and entities checked for each retained test pair. We will add this construction and acceptance procedure to Section 4.1.

To test performance beyond our curated benchmark, we added SysMBench, the 151-scenario benchmark of Jin et al. (2025), and its SysMEval precision, recall, and F1 measures. We report these alongside requirement coverage (Req.), entity coverage (Ent.), and SemAvg = (Req. + Ent.)/2. Jin et al., A System Model Generation Benchmark from Natural Language Requirements.

Separately from validating the 204 benchmark pairs, we checked the reliability of the semantic evaluation measures on 30 model outputs. Two human reviewers assessed frozen lists containing 441 requirement facts and 351 entity roles. The reported mean label agreement was 87.19% for Req. and 92.88% for Ent.; the mean Spearman correlation between their SysMEval-F1 rankings was 0.818.

Reliability measureReviewer 1Reviewer 2Mean
Req. label agreement88.89%85.49%87.19%
Ent. label agreement92.88%92.88%92.88%
SysMEval-F1 ranking correlation (Spearman)0.8170.8190.818

We evaluated the models with two test-time prompts. Zero-shot supplied an MBSE role instruction and the target requirement. SysML documentation + 3-shot additionally supplied official SysML v2 textual-notation excerpts, a syntax checklist, and three requirement–code examples that we wrote and checked for syntactic validity and semantic alignment. The target requirement followed these materials. Unlike the original manuscript, which calculated semantic scores only for syntax-valid outputs, the revised evaluation scores every available generated output, including syntax-invalid outputs. Thus, the semantic results describe the full set of saved generations rather than the subset that passed syntax checking. Table 1 reports syntax pass rate (SPR) for every model, and Table 2 reports the corresponding three-benchmark mean semantic scores. All values are percentages.

Table 1. Syntax pass rate under zero-shot and SysML documentation + 3-shot prompting.

GroupModel/settingZero-shot SysMBenchZero-shot ENZero-shot ZHDocs + 3-shot SysMBenchDocs + 3-shot ENDocs + 3-shot ZH
General-purpose APIClaude-Sonnet-53.9719.1218.1451.6622.5525.49
General-purpose APIGPT-5.6-Sol11.2616.1817.1648.3419.6122.06
General-purpose APIDeepSeek-V4-Pro1.325.886.3750.7214.7124.02
General-purpose APIGLM-5.25.967.356.3762.2517.1619.61
Code-specializedKimi-K2.7-Code5.3021.0816.6739.0718.6320.10
Code-specializedQwen3.5-9B-Coder0.000.550.0027.017.296.19
Code-specializedSeed-Coder-8B-Instruct0.000.000.5022.5210.405.39
Locally deployableQwen3-8B0.000.000.0012.165.614.50
Locally deployableQwen3-5.9B0.000.500.5030.008.386.95
Locally deployableQwen3-8.27B0.000.981.4749.0115.6912.25
Locally deployableGemma-4-E4B-it0.000.490.0017.888.466.37
Locally deployableGLM-Z1-9B-04140.000.000.002.762.111.33
OursSingle-pass inference45.7019.6118.6363.1226.1930.90
OursRepeated sampling (10)92.0539.2237.2598.6256.4654.55

Table 2. Mean semantic performance across SysMBench, EvoSysML-204 English, and EvoSysML-204 Chinese.

GroupModel/settingZero-shot Mean F1Zero-shot Mean SemAvgDocs + 3-shot Mean F1Docs + 3-shot Mean SemAvg
General-purpose APIClaude-Sonnet-565.0390.8569.6267.63
General-purpose APIGPT-5.6-Sol63.0587.5166.9168.11
General-purpose APIDeepSeek-V4-Pro67.1285.6057.3467.12
General-purpose APIGLM-5.265.4189.4964.4268.62
Code-specializedKimi-K2.7-Code64.1889.3170.5466.45
Code-specializedQwen3.5-9B-Coder51.0567.2657.2961.17
Code-specializedSeed-Coder-8B-Instruct51.3280.3944.7361.73
Locally deployableQwen3-8B62.4159.1855.8462.80
Locally deployableQwen3-5.9B54.3764.3455.7563.27
Locally deployableQwen3-8.27B61.1280.4761.9863.58
Locally deployableGemma-4-E4B-it56.9366.6755.2064.10
Locally deployableGLM-Z1-9B-041466.1859.6551.1359.37
OursRepeated sampling (10)67.4586.1066.1769.88

Under repeated sampling, our method has the highest reported SPR on all three benchmarks in both prompt settings. It also has the highest mean SysMEval-F1 in zero-shot and the highest mean SemAvg with documentation + 3-shot. We report both semantic measures because no system leads every semantic column.

Finally, we compared synthesis methods using the same 3K training-pair budget and zero-shot, single-pass testing. Table 3 reports syntax, while Table 4 combines SysMEval-F1 and SemAvg. “Direct few-shot” names a training-data synthesis method; its test prompt is zero-shot like the other rows.

Table 3. Zero-shot syntax pass rate for models trained on 3K synthesized pairs (%).

MethodSysMBenchEN-204ZH-204
Direct few-shot21.8510.786.86
Self-Instruct9.932.942.45
Code Evol-Instruct31.1310.6711.24
Instruct-SkillMix27.8111.2713.24
CoT-Self-Instruct33.778.8211.76
Ours35.1014.7011.30

Table 4. Zero-shot semantic performance for models trained on 3K synthesized pairs (%). Mean is the arithmetic mean across the three benchmarks.

GroupModel/settingSysMBench F1SysMBench SemAvgEN-204 F1EN-204 SemAvgZH-204 F1ZH-204 SemAvgMean F1Mean SemAvg
BaselineDirect few-shot29.2170.3467.7781.1862.3979.3653.1276.96
BaselineSelf-Instruct31.5269.6460.5570.7858.4770.9650.1870.46
BaselineCode Evol-Instruct34.1376.2065.7284.1262.8382.2554.2380.86
BaselineInstruct-SkillMix37.2079.8268.7587.5768.2885.2758.0884.22
BaselineCoT-Self-Instruct34.8675.0868.0486.5266.6683.9756.5281.86
OursEvoSysML data synthesis43.6883.2170.1487.5463.8684.6759.2385.14

At equal data scale, our synthesis method achieves the highest mean F1 and mean SemAvg. These comparisons support the usefulness of the curated requirement–code pairs, while the source, generation, and pair-level checks above directly answer how EvoSysML-204 itself was constructed and validated.

Answer to Weakness 2

We thank the reviewer for requesting clarification. We selected strong contemporary baselines covering three complementary groups: general-purpose API models with strong reasoning and instruction-following capabilities, code-specialized models with strong programming capabilities, and locally deployable models for comparison with our generator. This selection allows us to assess whether general reasoning and code-generation capabilities transfer to SysML and to quantify the benefits of task-specific training.

Previous SysML benchmarking provides precedent for evaluating these model families. For example, SysMBench evaluated GPT-4.1, Claude 3 Opus, DeepSeek R1, and Qwen3-32B. Our experiments include newer versions and additional baselines to reflect the capabilities available when conducting our study. We will clarify this selection rationale and cite SysMBench in Section 4.1.

Answer to Weakness 3

We thank the reviewer for highlighting these omissions. Our setup follows established parameter-efficient fine-tuning practice using LLaMA-Factory (Zheng et al., 2024) and LoRA (Hu et al., 2022). We chose LoRA to reduce trainable parameters and memory requirements, and used gradient accumulation to accommodate long SysML sequences with small per-device batches. The lower learning rate during DPO was chosen to limit changes to the SFT model while refining syntax preferences.

All training was conducted on a single NVIDIA RTX PRO 6000 GPU. We will add the LoRA citation and clarify the configuration rationale and exact GPU count in Section 4.1.

Answer to Weakness 4

We thank the reviewer for this helpful suggestion. We will revise the opening of Section 4.2 to read: “We evaluate syntactic performance using both single-pass inference and repeated sampling with up to ten candidates per requirement.” We will present the single-pass results first, followed by the repeated-sampling results, so that both evaluation settings are introduced clearly and in a consistent order.

Answer to Weakness 5

We thank the reviewer for pointing out that Section 4.3 did not specify the semantic criteria or verification procedure. We have expanded the evaluation from the five selected baselines to all evaluated models and added SysMBench (151 cases) alongside EvoSysML-204. Unlike the original analysis, the revised semantic evaluation scores every available generated output, including syntax-invalid outputs, rather than only those that pass syntax checking. For each requirement, we use a fixed checklist of facts and required modeling roles: Req. measures the proportion of facts correctly expressed in the code, including applicable entities, attributes, relations and endpoints, behaviors, constraints, values, and units; Ent. measures one-to-one coverage of required modeling roles; and SemAvg is their average. We additionally report SysMEval precision, recall, and F1 for atomic claims in generated and reference models. As a reliability check, two human reviewers assessed 30 outputs against 441 requirement facts and 351 entity roles. The reported mean label agreement was 87.19% for Req. and 92.88% for Ent., and the mean Spearman correlation between their F1 rankings was 0.818. We will add these criteria, the all-output evaluation scope, and the reliability procedure to Section 4.3.

Answer to Q1

We will make the basis for our design choices explicit throughout the method and experimental sections, including references to LoRA (Hu et al., 2022), LLaMA-Factory (Zheng et al., 2024), the official SysML v2 repository, and SysMBench. We will distinguish established techniques from our task-specific configuration choices and explain the rationale for the latter.

Answer to Q2

We selected programs from the official SysML v2 Pilot Implementation repository and used DeepSeek-V4-Flash to generate English requirements from their code. We manually compared each description with its program, checking whether the specified entities, attributes, relationships, behaviors, and constraints were faithfully represented. Inconsistent or materially incomplete descriptions were excluded. The Chinese requirement–program pairs were translated from the corresponding English pairs.

Answer to Q3

We selected strong contemporary models covering general-purpose APIs, code-specialized models, and locally deployable models to assess how their capabilities transfer to SysML generation. Previous SysML benchmarks provide precedent: SysMBench evaluated GPT-4.1, Claude 3 Opus, DeepSeek R1, and Qwen3-32B. Our selection includes newer versions and additional baselines, rather than exactly reproducing that model list.

Answer to Q4

We adopted LoRA-based fine-tuning through LLaMA-Factory. LoRA reduces trainable parameters and memory requirements, while gradient accumulation accommodates long SysML sequences with small per-device batches. A lower DPO learning rate limits updates to the SFT model during preference refinement. The cited methods support this training approach; the exact hyperparameters are our implementation choices. We will clarify this distinction and add the missing LoRA citation.

Answer to Q5

All local training and inference experiments used a single NVIDIA RTX PRO 6000 GPU. API baselines were accessed through their providers. We will explicitly state the GPU count in Section 4.1.

Answer to Q6

The outputs in the original top-five comparison were scored automatically, not manually verified one by one. Syntax validity was checked with the SysML validation kernel. For semantic evaluation, the same procedure was applied to every model: Req. measures coverage of a fixed list of requirement facts, Ent. measures one-to-one matches against required modeling roles, and SemAvg averages the two. We now additionally report SysMEval precision, recall, and F1, which compare atomic claims in generated and reference models. We have expanded the comparison to all evaluated models and SysMBench, and changed the semantic evaluation to include every available generated output, including syntax-invalid outputs. Human review was a separate reliability check, not the scoring method for the full evaluation: on 30 outputs, the reported mean agreement was 87.19% for Req. labels and 92.88% for Ent. labels, with a mean Spearman correlation of 0.818 for F1 rankings. We will clarify the automatic scoring procedure, the human audit, and the revised evaluation scope in Section 4.3.

Comments on the paper

2 comments

Dear Reviewers,

A friendly reminder to check the authors’ responses and the other reviewers’ comments. It would be helpful to hear your thoughts on the differing assessments and whether they affect your evaluation.

Thank you for your time and effort!

AC

AuthorsOpenReview ↗

Global Response to the Reviewers and ACs

We sincerely thank all four reviewers for their thoughtful feedback and for recognizing the importance of addressing data scarcity in SysML v2 generation, as well as the potential value of EvoSysML and the bilingual EvoSysML-204 benchmark. Their comments helped us clarify the method’s contribution, strengthen the evaluation, and make the study’s scope and limitations more explicit.

We have provided detailed, point-by-point responses. Below is a summary of the main issues addressed.

Reviewer LAJU

  • Dataset construction and semantic alignment: We clarified how EvoSysML-204 was curated from existing SysML v2 programs, how its English and Chinese requirements were prepared, and how each pair was checked for syntax and semantic alignment.
  • Experimental design and reproducibility: We explained the rationale for the baseline groups and training setup, specified that local experiments used one NVIDIA RTX PRO 6000 GPU, and clarified the single-pass and repeated-sampling settings.
  • Semantic evaluation: We specified the scoring criteria and automatic evaluation procedure, expanded evaluation to SysMBench and all available generated outputs, and reported a separate human reliability check.

Reviewer 9mTP

  • Novelty and component contributions: We clarified that the contribution is a SysML-specific, LLM-driven data-synthesis procedure. We added comparisons with five synthesis baselines at a matched 3K-pair budget and ablations of evolution roles and the combined effect of requirement reconstruction and semantic filtering.
  • Evaluation and limitations: We added SysMBench, expanded semantic metrics, clarified the prompt settings, and described training–test overlap screening. We also explain that 50K is the scale used in this study, not a demonstrated saturation point, and that DPO results are single-run estimates.

Reviewer TyVi

  • Semantic and syntactic supervision: We clarified that requirement reconstruction and semantic filtering guide training-data construction, while syntax-directed DPO targets syntactic errors.
  • Benchmarking and data scale: We added matched-budget synthesis comparisons and expanded evaluation with SysMBench, broader semantic metrics, and both zero-shot and documentation-plus-3-shot prompts. We also explained why the current 50K corpus is a practical study scale and identified larger-scale evaluation as future work.

Reviewer a11g

  • Method evidence and ablations: We added matched-budget comparisons and component ablations. We clarify that the ablation measures the joint contribution of requirement reconstruction and semantic filtering; it does not isolate their individual effects.
  • Evaluation reliability and fairness: We clarified decontamination checks, added precision-oriented semantic metrics and a human reliability audit, and scored all available outputs, including syntax-invalid ones. We also evaluated baselines with SysML documentation and examples, and corrected the mislabeled Qwen checkpoint.

We thank the reviewers again for their constructive feedback. Their comments helped us present EvoSysML’s contribution, supporting evidence, and remaining limitations more clearly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.