- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces EvoSysML, a domain-specific model training framework designed for generating SysML v2 code from natural language requirements. The approach employs an API-based model to evolve a small set of human-curated seed data—both in breadth and depth—and subsequently trains local "role-based" models to continuously expand the dataset, resulting in a training corpus of approximately 50,000 samples. During the training phase, the authors utilize Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), while the inference phase incorporates multiple-sampling and a formal validator to filter for valid code. Additionally, the authors constructed EvoSysML-204, a bilingual (Chinese-English) test set comprising 204 tasks, and benchmarked their approach against 12 baseline models. Experiments demonstrate that the method achieves high syntactic correctness in a single-generation setting, with multiple-sampling further improving the syntactic pass rate.
Critical strengths
Addressing the challenge of data scarcity in SysML v2 code generation, this paper proposes a complete framework—encompassing data synthesis, model training, and inference deployment—that provides a viable path for building domain-specific training data. The experimental evaluation is thorough, covering 12 baselines. The results indicate that domain adaptation enhances syntactic correctness, while multiple-sampling further boosts the success rate of code generation.
Critical weaknesses
1. The innovation of the core methodology and the necessity of its individual components remain insufficiently demonstrated.
The paper primarily integrates existing techniques—namely data evolution, SFT, DPO, and validator-based filtering. Although the authors have designed strategies specific to SysML, the work lacks a direct comparison against simpler data synthesis methods. Table 4 focuses primarily on the evolution ratio; it lacks a thorough ablation study to isolate the individual contributions of requirement reconstruction, semantic filtering, and multi-role evolution. Furthermore, the optimal DPO configuration only improves the average syntactic pass rate from 17.40% to 19.12%, and performance actually declines when the volume of preference data is increased. The study also lacks multiple random seeds or statistical analysis to verify the stability of these gains.
2. There are shortcomings regarding the reliability and semantic evaluation of the synthetic data.
The paper fails to provide a detailed explanation of the implementation and accuracy of the semantic validator (). While reconstructing requirements from generated code can enhance pair consistency, it does not necessarily reflect the distribution of requirements as defined by actual users. The authors also do not adequately describe the decontamination process between the full 50K training set and the test set. Moreover, metrics such as "Requirement Coverage" and "Entity Recall" focus primarily on recall, failing to sufficiently penalize the generation of erroneous entities or relationships. The semantic metrics in Table 3 are calculated only on syntactically valid outputs, meaning the evaluation sample sets differ across models. For instance, the reported English SemAvg is 86.90% for the proposed method versus 86.88% for "GPT-5.6-Sol"—a difference insufficient to demonstrate a clear semantic advantage.
3. The baseline setup and the comparison of inference budgets are inadequate to fully substantiate the claimed performance superiority.
The paper posits that general-purpose models lack domain-specific knowledge of SysML; however, it reports no experiments in which baselines were provided with official SysML v2 documentation, syntactic specifications, or relevant examples. Consequently, it remains unclear whether the performance gap stems from a lack of domain knowledge or from inherent limitations in code generation capabilities—a point that weakens the argument for the necessity of specialized training.
Questions for authors
See "Weaknesses."
- Could you provide a comparison against simpler data synthesis methods, keeping the total data volume and training budget constant? Additionally, could you isolate and verify the individual contributions of requirement reconstruction, semantic filtering, and multi-role evolution?
- How is implemented, and has its accuracy been independently verified by humans? Furthermore, what measures were taken to ensure there is no data contamination between the full training set and the test set? 3. Is it possible to incorporate SysML documentation, syntax examples, or a retrieval-augmented general model baseline, and then evaluate the effectiveness of verifier-based filtering or code repair under a comparable inference budget?
4 comments
Contribution of requirement reconstruction and semantic filtering. We compare the full 3K synthesis pipeline with the pipeline before requirement reconstruction and semantic filtering. Table 4 reports SPR for each evaluation set and mean semantic metrics across the three sets. Adding reconstruction and filtering improves mean F1 from 55.76% to 59.23% and mean SemAvg from 81.11% to 85.14%. Mean SPR increases slightly from 20.14% to 20.36%: it rises on SysMBench and ZH, with a small decrease on EN. This comparison measures the joint contribution of reconstruction and filtering; it does not separate their individual effects.
Table 4. Ablation of requirement reconstruction and semantic filtering at the 3K budget. SPR is reported per evaluation set; semantic metrics are means across the three sets (%).
| Setting | SPR: SysMBench | SPR: EN | SPR: ZH | Mean F1 | Mean SemAvg |
|---|---|---|---|---|---|
| Before reconstruction and semantic filtering | 34.44 | 15.20 | 10.78 | 55.76 | 81.11 |
| Full EvoSysML data synthesis | 35.10 | 14.71 | 11.27 | 59.23 | 85.14 |
DPO and stability. We agree that DPO’s observed gain is modest and non-monotonic. DPO-200 raises mean SPR from 17.40% to 19.12%, an improvement of 1.72 percentage points, but DPO-400 falls to 17.65%. DPO-100, rather than DPO-200, has the highest conditional SemAvg. We therefore describe DPO as a syntax-focused refinement whose outcome depends on the preference-data budget. These are single-run point estimates: without multiple seeds or uncertainty intervals, we cannot claim that the DPO gain is statistically stable. We will state this limitation in the revised ablation discussion.
Expanded benchmarks, prompts, and semantic evaluation. We contribute the bilingual EvoSysML-204 test set and extend evaluation to SysMBench, the external 151-scenario benchmark introduced by Jin et al. (2025). We adopt its SysMEval precision, recall, and F1 measures alongside requirement coverage (Req.), entity coverage (Ent.), and SemAvg = (Req. + Ent.)/2. Jin et al., A System Model Generation Benchmark from Natural Language Requirements.
We use two clearly specified test-time prompt settings:
- Zero-shot: The system instruction asks the model to act as an MBSE engineer and return only a complete SysML v2 textual model. The user message contains the target natural-language requirement. The prompt contains no SysML documentation and no requirement–code demonstration.
- SysML documentation + 3-shot: The prompt retains the target requirement and adds excerpts from the official SysML v2 textual-notation documentation, a local syntax checklist, compact notation examples checked with the SysML parser, and three requirement–code demonstrations that we wrote and checked for both syntax and semantic alignment. The target requirement appears after these materials and is explicitly prioritized over the demonstrations. The demonstrations are training examples, not benchmark answers.
This distinction matters because “Direct few-shot” in Table 1 names a training-data synthesis baseline; its evaluation is zero-shot, like every other row in that 3K comparison. In the revised semantic evaluation, we score every available saved generation, including syntax-invalid outputs, rather than restricting semantic scores to syntax-valid outputs as in the original manuscript. Missing generations are not treated as scored outputs. The original conditional SemAvg values in Table 3 and the revised all-output semantic values are therefore different evaluation scopes.
Human reliability check. Separately from automatic scoring of the full evaluation, two human reviewers checked 30 outputs against frozen lists containing 441 requirement facts and 351 entity roles. Req. and Ent. agreement was assessed at the underlying label level. Because reviewers can segment atomic claims differently, agreement for SysMEval-F1 was assessed through the ranking of the same 30 outputs using Spearman correlation, rather than by claiming identical atomic-claim labels.
| Reliability measure | Reviewer 1 | Reviewer 2 | Mean |
|---|---|---|---|
| Req. label agreement | 88.89% | 85.49% | 87.19% |
| Ent. label agreement | 92.88% | 92.88% | 92.88% |
| SysMEval-F1 ranking correlation (Spearman) | 0.817 | 0.819 | 0.818 |
The broader model comparison is summarized below. All entries are percentages; mean semantic scores are arithmetic means across SysMBench, EvoSysML-204 English, and EvoSysML-204 Chinese.
Table 5. Syntax pass rate under zero-shot and SysML documentation + 3-shot prompting.
| Group | Model/setting | Zero-shot SysMBench | Zero-shot EN | Zero-shot ZH | Docs + 3-shot SysMBench | Docs + 3-shot EN | Docs + 3-shot ZH |
|---|---|---|---|---|---|---|---|
| General-purpose API | Claude-Sonnet-5 | 3.97 | 19.12 | 18.14 | 51.66 | 22.55 | 25.49 |
| General-purpose API | GPT-5.6-Sol | 11.26 | 16.18 | 17.16 | 48.34 | 19.61 | 22.06 |
| General-purpose API | DeepSeek-V4-Pro | 1.32 | 5.88 | 6.37 | 50.72 | 14.71 | 24.02 |
| General-purpose API | GLM-5.2 | 5.96 | 7.35 | 6.37 | 62.25 | 17.16 | 19.61 |
| Code-specialized | Kimi-K2.7-Code | 5.30 | 21.08 | 16.67 | 39.07 | 18.63 | 20.10 |
| Code-specialized | Qwen3.5-9B-Coder | 0.00 | 0.55 | 0.00 | 27.01 | 7.29 | 6.19 |
| Code-specialized | Seed-Coder-8B-Instruct | 0.00 | 0.00 | 0.50 | 22.52 | 10.40 | 5.39 |
| Locally deployable | Qwen3-8B | 0.00 | 0.00 | 0.00 | 12.16 | 5.61 | 4.50 |
| Locally deployable | Qwen3-5.9B | 0.00 | 0.50 | 0.50 | 30.00 | 8.38 | 6.95 |
| Locally deployable | Qwen3-8.27B | 0.00 | 0.98 | 1.47 | 49.01 | 15.69 | 12.25 |
| Locally deployable | Gemma-4-E4B-it | 0.00 | 0.49 | 0.00 | 17.88 | 8.46 | 6.37 |
| Locally deployable | GLM-Z1-9B-0414 | 0.00 | 0.00 | 0.00 | 2.76 | 2.11 | 1.33 |
| Ours | Single-pass inference | 45.70 | 19.61 | 18.63 | 63.12 | 26.19 | 30.90 |
| Ours | Repeated sampling (10) | 92.05 | 39.22 | 37.25 | 98.62 | 56.46 | 54.55 |
Table 6. Mean semantic performance across the three benchmarks.
| Group | Model/setting | Zero-shot Mean F1 | Zero-shot Mean SemAvg | Docs + 3-shot Mean F1 | Docs + 3-shot Mean SemAvg |
|---|---|---|---|---|---|
| General-purpose API | Claude-Sonnet-5 | 65.03 | 90.85 | 69.62 | 67.63 |
| General-purpose API | GPT-5.6-Sol | 63.05 | 87.51 | 66.91 | 68.11 |
| General-purpose API | DeepSeek-V4-Pro | 67.12 | 85.60 | 57.34 | 67.12 |
| General-purpose API | GLM-5.2 | 65.41 | 89.49 | 64.42 | 68.62 |
| Code-specialized | Kimi-K2.7-Code | 64.18 | 89.31 | 70.54 | 66.45 |
| Code-specialized | Qwen3.5-9B-Coder | 51.05 | 67.26 | 57.29 | 61.17 |
| Code-specialized | Seed-Coder-8B-Instruct | 51.32 | 80.39 | 44.73 | 61.73 |
| Locally deployable | Qwen3-8B | 62.41 | 59.18 | 55.84 | 62.80 |
| Locally deployable | Qwen3-5.9B | 54.37 | 64.34 | 55.75 | 63.27 |
| Locally deployable | Qwen3-8.27B | 61.12 | 80.47 | 61.98 | 63.58 |
| Locally deployable | Gemma-4-E4B-it | 56.93 | 66.67 | 55.20 | 64.10 |
| Locally deployable | GLM-Z1-9B-0414 | 66.18 | 59.65 | 51.13 | 59.37 |
| Ours | Repeated sampling (10) | 67.45 | 86.10 | 66.17 | 69.88 |
Under ten-candidate sampling, our method has the highest reported syntax pass rate on all three benchmarks in both prompt settings. It also has the highest mean SysMEval-F1 in zero-shot and the highest mean SemAvg with documentation + 3-shot. These results establish broad performance; the matched synthesis comparisons and component ablations above provide the more direct evidence for the method’s design. We will revise the method, ablation, and evaluation sections to state the prompt contents, semantic-scoring denominator, component contributions, and remaining stability limitation explicitly.
Answer to Weakness 2/Q2
We agree that the original manuscript should explain semantic validation and decontamination more clearly. Our ten author-written seeds were grounded in reference cases based on real user requirements; breadth evolution preserves their core intent while creating diverse scenarios. For each synthesized SysML program, we reconstruct the requirement from the code and check its entities, attributes, relations, behaviors, constraints, values, and units before retaining the pair. The EvoSysML-204 test programs were selected separately from the official SysML v2 Pilot Implementation repository. We manually checked that their paired requirements express realistic user needs and align with the code, and screened the full 50K training corpus against the test set for exact and structural overlap. We will describe these checks and their results in the revision. To address the recall-oriented nature of Req. and Ent., we added SysMEval precision, recall, and F1, where precision penalizes unsupported generated claims. The revised evaluation scores every available generation, including syntax-invalid outputs, rather than only syntax-valid ones. In a separate 30-output human audit, the reported Req. and Ent. label agreements were 87.19% and 92.88%, and the mean Spearman correlation for F1 rankings was 0.818.
Answer to Weakness 3/Q3
We added a SysML documentation + 3-shot setting for all baselines, using official textual-notation excerpts, a syntax checklist, and three author-written requirement–code examples checked for syntax and semantic alignment. SysML-specific guidance substantially improves the API models on SysMBench: Claude-Sonnet-5 rises from 3.97% to 51.66% SPR, GPT-5.6-Sol from 11.26% to 48.34%, DeepSeek-V4-Pro from 1.32% to 50.72%, and GLM-5.2 from 5.96% to 62.25%. This indicates that limited SysML domain knowledge contributes to weak zero-shot performance. Under the same documentation + 3-shot, single-pass setting, our method achieves 63.12% / 26.19% / 30.90% SPR on SysMBench / EvoSysML-204 English / Chinese, exceeding the strongest baseline on each dataset (62.25% / 22.55% / 25.49%). We report ten-candidate results separately and assess semantic performance with the expanded metrics described above.
Answer to Critical Weakness 1 / Q1
We thank the reviewer for distinguishing overall performance from evidence for the individual components. Our contribution is a SysML-specific data-synthesis algorithm, rather than a claim that SFT, DPO, or validation is new in isolation. Starting from a small seed pool, the breadth-evolution role creates new requirement scenarios, while the depth-evolution role enriches SysML programs with more complex modeling structures. A requirement-authoring role then rewrites each requirement from its validated program. Only pairs passing syntax, requirement–code consistency, and duplicate checks enter SFT. DPO addresses syntax errors remaining after SFT, and the deployment validator selects a syntactically valid candidate.
Comparison with simpler synthesis methods. We adapted five data-synthesis methods to SysML and trained each resulting model with 3K pairs. All models in this comparison were tested with a zero-shot, single-pass prompt. Our method achieves the highest syntax pass rate on SysMBench and EvoSysML-204 English, and the highest mean SysMEval-F1 and mean SemAvg across the three benchmarks. Instruct-SkillMix performs better on the Chinese benchmark, which we report explicitly.
Table 1. Zero-shot syntax pass rate for models trained on 3K synthesized pairs (%).
Table 2. Zero-shot semantic performance for models trained on 3K synthesized pairs (%). Mean is the arithmetic mean across the three benchmarks.
Contribution of preference tuning and evolution roles. We evaluate preference-tuning settings and breadth–depth compositions using syntax pass rates on SysMBench and the EN and ZH EvoSysML-204 sets, together with mean F1 and mean SemAvg across the three evaluation sets. Table 3 reports these results; the equal-weight macro-average SPR is stated in the text rather than shown as a separate column. DPO-200 performs best among the preference-tuning settings, and the B:D 1:3 mixture provides the strongest overall balance between syntax and semantic performance. DPO-200 achieves the highest macro-average SPR in its group (27.98%) and the highest mean F1 and mean SemAvg (67.45% and 86.10%). Among the evolution-role settings, Depth-only has a slightly higher macro-average SPR than Breadth-only (20.53% vs. 11.10%), while Breadth-only has higher mean F1 and mean SemAvg (59.45% and 85.34% vs. 52.64% and 80.85%). The 1:3 mixture achieves a macro-average SPR of 20.36%, close to Depth-only, and semantic means of 59.23% and 85.14%, close to Breadth-only. We select 1:3 as the best overall trade-off among the tested compositions.
Table 3. Ablations of preference tuning and breadth–depth composition. SPR is reported separately for SysMBench and the EN and ZH EvoSysML-204 sets; semantic metrics report means across the three evaluation sets (%).