acceptodds
Under review as a conference paper at ICLR 2027

XGenerator: Skill-Guided Synthesis of Verifiable Training Data for a Low-Resource Modeling Language

Abstract

Low-resource domain-specific languages (DSLs) often lack training data linking explicit requirements, executable models, and verification evidence. In industrial system modeling, constructing such data requires compilation, simulation, and requirement checking within a unified workflow. We introduce XGenerator, a skill-guided framework for synthesizing verifiable training data that keeps base-model parameters fixed during generation. Versioned procedural skills organize language specifications, modeling knowledge, and verified experience for on-demand retrieval. The language model produces a compact executable intermediate design describing components, states, and event interactions. Deterministic tools translate it into source code and perform compilation, simulation, and requirement checking against frozen behavioral contracts. Failure evidence guides scoped design repair and revalidation while requirements and shared skills remain fixed. The resulting artifacts link requirements, designs, code, and execution evidence; subsequent skill revisions require separate development validation. We evaluate XGenerator on 80 discrete-event modeling tasks across eight domains and five difficulty levels in X-language, a low-resource DSL. XGenerator improves first-pass requirement satisfaction over prompt-only baselines by 88.75 and 40.00 percentage points on GPT-6-astra and DeepSeek-flash, respectively. With at most one feedback-guided repair, the final requirement satisfaction rate reaches 97.5% and 80.0%. We further construct a model dataset for fine-tuning. Across four Qwen3-8B runs, requirement satisfaction on 410 tasks with executable criteria reaches 68.78–70.24%, compared with 0% for the base model under raw-output evaluation. These results support constructing traceable training data for low-resource modeling DSLs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, synthetic training data, code generation, domain-specific languages, procedural skills, execution-guided repair

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XRFqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces XGenerator, a framework for synthesizing training data for X-language, a low-resource modeling DSL. An LLM uses procedural skills to produce an executable intermediate design, which deterministic tools translate into source code. Compilation, simulation, and frozen requirement checks provide feedback for one repair attempt. The evaluation includes generation experiments on 80 tasks, subsystem ablations on 40 tasks, and downstream fine-tuning. The reported results show improved first-pass generation and executable delivery, although attribution, data-quality validation, and downstream evaluation remain unresolved.

Critical strengths
  • The paper addresses a concrete data-scarcity problem in a specialized modeling language. Linking requirements, executable implementations, and verification evidence is a useful objective because isolated syntax examples provide limited supervision for models that must represent interacting components and dynamic behavior.

  • The separation between model-generated semantic design and deterministic source translation is a sensible engineering choice. The intermediate design explicitly represents components, states, actions, and connections, providing a structured interface for validation and repair while leaving the application behavior to the language model.

  • The validation protocol distinguishes compilation, successful simulation, and requirement satisfaction. Freezing requirements and acceptance criteria during repair, and reevaluating the revised implementation across the prescribed scenarios, provides a clearer basis for assessing repairs than compilation success alone.

  • The evaluation includes multiple language models, tasks across eight domains and five difficulty levels, subsystem ablations, and downstream fine-tuning. The stage-specific results expose meaningful differences between syntactic delivery and behavioral correctness, while the cost reporting makes the additional computational demands visible.

Critical weaknesses
  • The methodological contribution is not sufficiently distinguished from combining procedural skills, structured intermediate representations, deterministic translation, and execution-guided repair. The ablations remove several capabilities together, making it difficult to identify which proposed mechanism explains the improvements. The contribution of reusable experience is particularly unclear: the full ablation configuration retrieves no matching experience during first-pass generation, and the frozen-skill experiments do not evaluate the skill-evolution process illustrated in the overview. The evidence therefore establishes the usefulness of a bundled workflow more clearly than the value of its individual proposed mechanisms.

  • The comparisons do not adequately separate workflow improvements from additional computation. For Astra, the documentation-and-examples baseline reaches 74/80 final requirement successes, compared with 78/80 for the action-patch variant, while total token consumption increases from approximately 28.3 million to 74.2 million. The first-pass improvement is substantial, and DeepSeek shows a larger final gain, but equal repair opportunities do not establish comparable computational budgets. Budget-matched baselines or performance-versus-cost results are needed to assess the practical advantage of the more elaborate workflow.

  • The validation does not sufficiently establish the quality and generalization of the resulting training data. All generation-benchmark scenarios are public, and passing them does not demonstrate mechanism fidelity or robustness to unseen conditions. The broader corpus also includes historical hybrid and continuous models that were not validated through the same complete frozen-contract procedure. In addition, the paper leaves item-level split isolation and semantic near-duplicate verification unresolved. These gaps limit the evidence that the corpus provides diverse, reliable supervision and that downstream performance reflects generalization.

  • The downstream fine-tuning comparison is substantially confounded by output formatting and incomplete model provenance. The evaluator passes raw predictions directly to the compiler, and 527 of the 590 base-model outputs contain Markdown fences that cause parsing failures. The remaining unfenced outputs also fail, so formatting is not the sole explanation, but a normalized evaluation is needed to separate delivery-format gains from modeling improvements. The unverified 27B reference identity and incomplete linkage between loaded checkpoints and test outputs further weaken the interpretation and reproducibility of the training results.

Questions for authors
  1. Can the proposed components be compared while holding the available domain information, execution tools, and computational budget fixed? What evidence specifically establishes the contribution of skill organization and reusable experience beyond structured design and deterministic translation?

  2. How do the full workflow and the documentation-and-examples baseline compare under matched token or time budgets? Can performance-versus-cost results clarify whether the main benefit is higher first-pass reliability, better final correctness, or reduced repair demand?

  3. Can the authors provide held-out scenario or parameter tests, together with a mechanism-fidelity assessment of generated models? Please also clarify the split-group definition and provide an overlap and near-duplicate analysis supporting train–test isolation.

  4. What are the base and fine-tuned results after applying the same minimal output normalization, such as removing Markdown fences? Can the authors establish the identities of the evaluated reference models and checkpoints and link them to the reported prediction artifacts?

6 comments
AuthorsOpenReview ↗

Respond to XRFq review

Thank you, XRFq, for your comments on skill organization and computational cost.

Q1. With information, tools, and budget held fixed, what do skill organization and reusable experience each contribute?

With information, tools and budget fixed, changing the same rules from ID order to stage order increased first-pass requirement successes from 6/40 to 12/40 on Astra-low, but reduced them from 7/40 to 4/40 on Sol-low, from 6/40 to 3/40 on Sol-medium, and from 9/40 to 2/40 on Sol-high. Only the Astra-low run benefited from stage ordering. AE7 retrieved no past cases during first generation. The logs record case use during repair.

Methodological contribution. XGenerator turns the task-by-task work of designing, coding, and verifying a domain model into an automated data-generation workflow guided by procedural skills. Experts still specify the language rules, tools, and acceptance criteria. For each sample, the model draws on skills to produce a design; tools translate it to source and link execution results to the original requirements. The training sample thus contains both code and evidence for judging whether it meets the specified requirements. Skill evolution is the part of this workflow designed to carry verified experience into later batches; it is distinct from repairing the current task.

Fixed information and budget; only rule order changes. Both groups received the same 18 rules and stage labels in one model call and used the same CE-ID, translator, and evaluation tools. The only change was to list the rules by ID or by modeling stage (understand the requirements, write the design, check the design). Each task had a 900-second limit and a 65,536-token ceiling, with no feedback or repair. The single-prompt rows in the table below report first submissions on the same 40 tasks.

Three settings in the table. (1) Single prompt, ID order: all 18 rules appear in one prompt, ordered by a fixed identifier unrelated to the modeling steps. (2) Stage order: the same 18 rules and stage labels appear in a single prompt, reordered as “understand the requirements, write the design, check the design.” In both groups, the model is called once per task and submits one CE-ID. The program scores the submission afterward; the model sees no execution feedback. (3) M9: during modeling, the model consults skills as needed and uses tools to check its CE-ID. A tool translates the design into source code. After the first submission, the program compiles the code, runs the simulations, and checks the behavior against the frozen requirements contract. If the submission fails, the model may make one local repair based on the feedback; the program then re-evaluates it against the original contract.

\hline \text{Model} & \text{Invocation method} & \text{Compile pass} & \text{Simulation pass} & \text{Requirements met} \cr \hline \text{Astra-Low} & \text{Single prompt (ID order)} & 9/40 & 7/40 & 6/40 \cr \text{Astra-Low} & \text{Single prompt (stage order)} & 17/40 & 17/40 & 12/40 \cr \text{Astra-Low} & \text{M9 (first submission)} & 39/40 & 39/40 & 38/40 \cr \text{Sol-Low} & \text{Single prompt (ID order)} & 8/40 & 8/40 & 7/40 \cr \text{Sol-Low} & \text{Single prompt (stage order)} & 7/40 & 7/40 & 4/40 \cr \text{Sol-Medium} & \text{Single prompt (ID order)} & 7/40 & 7/40 & 6/40 \cr \text{Sol-Medium} & \text{Single prompt (stage order)} & 5/40 & 4/40 & 3/40 \cr \text{Sol-Medium} & \text{M9 (first submission)} & 40/40 & 39/40 & 32/40 \cr \text{Sol-High} & \text{Single prompt (ID order)} & 9/40 & 9/40 & 9/40 \cr \text{Sol-High} & \text{Single prompt (stage order)} & 2/40 & 2/40 & 2/40 \cr \text{DeepSeek-Medium} & \text{M9 (first submission)} & 34/40 & 33/40 & 16/40 \cr \hline \end{array}$$ Paired by task, 11 passed only in stage order and five only in ID order on Astra-low. On Sol-high, one passed only in stage order and eight only in ID order. The Sol-high difference arose before compilation: nine fixed-ID designs and two stage-ordered designs passed CE-ID admission and produced source code. All eleven programs then passed compilation, simulation and requirement checks. Both groups received all the rules; only their order changed. Of the four settings, only Astra-low passed more tasks in stage order. **Full-workflow results.** Rescoring the same 40 tasks with the frozen evaluator gave the following requirement passes from first generation to at most one repair: Astra-low, 38/40 to 40/40; DeepSeek-medium, 16/40 to 33/40; and Sol-medium, 32/40 to 37/40. For both Astra-low and Sol-medium, the rule-order controls used the same model and reasoning setting as the corresponding full M9 workflow. Here we report requirement passes for the full workflow by model. The rule-order experiment above compares the two orderings at fixed model, tools, and budget.
AuthorsOpenReview ↗

Respond to XRFq review

Q2. What does the extra compute buy?

On Astra, the main benefit was fewer failed first attempts. Of the same 80 tasks, M9 passed 71 on the first submission versus 46 for M3, so it needed 9 repairs instead of 34. After repair the difference narrowed to four tasks (78 versus 74). On DeepSeek, the gain persisted after repair: first-pass successes rose from 9 to 32 and final successes from 34 to 64. Each group comprises 80 tasks. Token counts are in millions; hours are summed across calls.

\hline \text{Model / method} & \text{Initial req.} & \text{Repairs} & \text{Final req.} & \text{R0 (M)} & \text{R1 (M)} & \text{Total (M)} & \text{Hours} \cr \hline \text{Astra / M3} & 46/80 & 34 & 74/80 & 17.831 & 10.482 & 28.313 & 6.124 \cr \text{Astra / M9} & 71/80 & 9 & 78/80 & 67.509 & 6.675 & 74.183 & 8.833 \cr \text{DeepSeek / M3} & 9/80 & 71 & 34/80 & 67.025 & 76.744 & \text{143.769†} & 16.878 \cr \text{DeepSeek / M9} & 32/80 & 48 & 64/80 & 394.682 & 178.707 & 573.388 & 30.069 \cr \hline \end{array}$$ † One DeepSeek M3 call lacks a token count; the total shown is the recorded subtotal. On Astra, M9 spent more on the first pass (67.509M versus 17.831M tokens) and less on repairs (6.675M versus 10.482M). Its total was 2.62 times M3’s. Both methods had the same repair allowance. The table reports their observed costs rather than pass counts at a fixed token or time budget. We will add the first-pass and repair costs separately in Appendix B. **Q3. What conclusions about generalization do the data splits, overlap checks, and unseen scenarios support?** In the original evaluation, the scenarios were public and available to XGenerator. Here, we kept the original requirements and archived M9/Astra business logic and changed the test inputs to check whether the same programs still behaved as required under new conditions. We included all 78 programs that passed every public requirement check. The other two (D03-L2-B and D05-L5-B) remain failures in the original evaluation. **How were the new cases constructed and scored?** We selected an observable rule from the original requirement, changed an input that exercises it, and specified which outcomes should change or stay the same. In D02-L2-B, a precharge response at time 2 was within the deadline; at time 4 it was late. The energy request followed the response in both cases. The package compiled, and both simulations completed. The contactor should close once in the first case and never in the second; the observed counts were one and zero. Zero is the correct behavior in the late case. Conversely, giving D02-L1-B one extra energy request after the battery was fully charged should leave the number of settlements at one, as observed. We changed test drivers or runtime inputs; we did not rewrite or repair the business logic in response to these tests. **Four worked examples.** Each value in the last column is both the expected and observed result. $$\begin{array}{llccl} \hline \text{Model / case} & \text{Change to the scenario} & \text{Compile} & \text{Sim.} & \text{Requirement: expected = observed} \cr \hline \text{D02-L2-B/1} & \begin{array}{l}\text{Delay precharge response from 1 to 2} \cr \text{time units, within deadline}\end{array} & \text{Pass} & \text{Ran} & \text{main closures: 1} \cr \text{D02-L2-B/2} & \begin{array}{l}\text{Delay precharge response to time 4,} \cr \text{past the deadline}\end{array} & \text{Pass} & \text{Ran} & \text{main closures: 0} \cr \text{D02-L1-B/1} & \begin{array}{l}\text{Request energy again after a full charge} \cr \text{but before settlement}\end{array} & \text{Pass} & \text{Ran} & \text{settlements: 1} \cr \text{D03-L3-B/2} & \begin{array}{l}\text{Reorder the same six requests} \cr \text{as A,B,C,A,A,A}\end{array} & \text{Pass} & \text{Ran} & \text{cache misses: 4} \cr \hline \end{array}$$ **Results.** We tested 52 business-input cases on the first 24 programs and report one final, interpretable business-input case on each of the remaining 54, for 106 cases. Among the latter 54, 40 changed the workload or entity count, 1 changed reserve capacity, and 13 changed events or input conditions. At least one checked output had to change in 45 cases; 9 tested outcomes that should remain invariant. All 106 cases compiled, completed simulation, and met their specified behavioral checks. These are responses of archived programs to particular inputs, not 106 new modeling tasks or proof of all internal mechanisms. D01-L5-B covers only premature ground resets. Test designers could inspect the archived source.
AuthorsOpenReview ↗

Respond to XRFq review

Semantic near-duplicates problems: We compared each of the 590 test tasks against the training set (4,712 items) and the validation set (590 items). We found no verbatim matches for a complete code-generation prompt or reference solution. To detect implementation reuse, we consistently renamed identifiers in the reference solutions and compared the full programs, preserving constants, strings, operators, and token order. Of the 410 tasks with executable checks, 306 matched at least one training task. All 264 tasks with a validation match were among those 306. The remaining 104 had no such match in either set. We therefore tested how often the generated code passed on these 104 tasks without a matching implementation template. The three columns report passes out of 104 (percentage). The 8B runs use the original code-generation prompt and the scores after code-fence removal. The 27B LoRA run also uses the original code-generation prompt. All three rates retain the planned denominator of 104.

\hline \text{Model} & \text{Compiled} & \text{Completed simulation} & \text{Met requirements} \cr \hline \text{8B base} & \text{0/104 (0.0)} & \text{0/104 (0.0)} & \text{0/104 (0.0)} \cr \text{8B full, 1e−5} & \text{79/104 (76.0)} & \text{54/104 (51.9)} & \text{15/104 (14.4)} \cr \text{8B full, 5e−6} & \text{82/104 (78.8)} & \text{61/104 (58.7)} & \text{15/104 (14.4)} \cr \text{8B LoRA r128} & \text{86/104 (82.7)} & \text{63/104 (60.6)} & \text{15/104 (14.4)} \cr \text{8B LoRA r64} & \text{85/104 (81.7)} & \text{60/104 (57.7)} & \text{16/104 (15.4)} \cr \text{27B base} & \text{0/104 (0.0)} & \text{0/104 (0.0)} & \text{0/104 (0.0)} \cr \text{27B LoRA} & \text{99/104 (95.2)} & \text{77/104 (74.0)} & \text{25/104 (24.0)} \cr \hline \end{array}$$ All 104 outputs from the original 27B base group failed compilation, so none completed simulation or passed the requirement checks. **On the same 104 tasks without a matching training template, 79–86 programs from the four 8B fine-tunes compiled and 15–16 met the requirements, compared with 0 for the 8B base model. The fine-tunes therefore solved some tasks without a matching training template, although they performed markedly better on tasks with a match**. The focus of this paper is data generation; we use supervised fine-tuning only to assess the usefulness of the generated data.
Reviewer_LHuwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes XGenerator, a skill-guided framework for synthesizing verifiable training data for the low-resource modeling DSL X-language. The framework combines procedural skill retrieval, a compact executable intermediate design, deterministic code translation, and compilation, simulation, and requirement-based validation. Failed candidates can undergo evidence-guided repair. Experiments on 80 generation tasks demonstrate improvements over prompt-based baselines, while downstream fine-tuning experiments on Qwen models suggest that the generated data can improve executable model generation.

Critical strengths
  1. The problem is important. Generating reliable training data for low-resource domain-specific modeling languages.
  2. The integration of compilation, simulation, and requirement checking provides a meaningful verification pipeline beyond syntactic correctness.
  3. The experiments include multiple foundation models, subsystem ablations, and downstream fine-tuning, offering evidence of practical utility. I think the validation is good.
Critical weaknesses
  1. The fine-tuning results are difficult to interpret because semantic train-test overlap has not been fully audited. Moreover, the reported improvement from 0% to approximately 70% requirement satisfaction is confounded by output formatting: most base-model predictions contain Markdown code fences and are evaluated without preprocessing. 
  2. Limited Isolation of the Proposed Methodological Contributions. I think framework combines several established techniques, including skill retrieval, intermediate representations, deterministic translation, and execution-guided repair. However, the main ablation removes multiple components simultaneously, making it difficult to attribute improvements to individual innovations. A more detailed ablation study is needed. For me, the necessity and distinct contribution of the proposed components remain unclear.
Questions for authors
  1. Can the authors provide a format-normalized base-model comparison and verify the absence of semantic near-duplicates between training and test tasks?

  2. Can the authors isolate the effects of CE-ID, deterministic translation, skill retrieval, and evidence-constrained repair through more targeted ablations?

  3. Since all requirement checks use predefined scenarios, how do the authors ensure that accepted models implement the intended behavior rather than merely satisfying scenario-specific checks?

For me, once these problems are resolved/I am convinced of the novelty or I dont get some new questions/concern via seeing other reviewers' review, I should see an improvement in my grades.

6 comments
AuthorsOpenReview ↗

Respond to LHuw reviewer

Thank you again for your comment, LHuw. We have revised our response based on your last comment.

Q1. Does the gap between the base and fine-tuned models remain after removing the effect of formatting? Are there semantic near-duplicates between the training and test sets?

Formatting question: After code-fence removal, all four 8B fine-tunes still had higher requirement pass rates than the base model. After re-scoring, the base still compiled 0/590, simulated 0/590 and passed 0/410 contract tasks. None of the 2,360 outputs from the four fine-tunes changed.

Semantic near-duplicates problems: We compared each of the 590 test tasks against the training set (4,712 items) and the validation set (590 items). We found no verbatim matches for a complete code-generation prompt or reference solution. To detect implementation reuse, we consistently renamed identifiers in the reference solutions and compared the full programs, preserving constants, strings, operators, and token order. Of the 410 tasks with executable checks, 306 matched at least one training task. All 264 tasks with a validation match were among those 306. The remaining 104 had no such match in either set. We therefore tested how often the generated code passed on these 104 tasks without a matching implementation template. The three columns report passes out of 104 (percentage). The 8B runs use the original code-generation prompt and the scores after code-fence removal. The 27B LoRA run also uses the original code-generation prompt. All three rates retain the planned denominator of 104.

\hline \text{Model} & \text{Compiled} & \text{Completed simulation} & \text{Met requirements} \cr \hline \text{8B base} & \text{0/104 (0.0)} & \text{0/104 (0.0)} & \text{0/104 (0.0)} \cr \text{8B full, 1e−5} & \text{79/104 (76.0)} & \text{54/104 (51.9)} & \text{15/104 (14.4)} \cr \text{8B full, 5e−6} & \text{82/104 (78.8)} & \text{61/104 (58.7)} & \text{15/104 (14.4)} \cr \text{8B LoRA r128} & \text{86/104 (82.7)} & \text{63/104 (60.6)} & \text{15/104 (14.4)} \cr \text{8B LoRA r64} & \text{85/104 (81.7)} & \text{60/104 (57.7)} & \text{16/104 (15.4)} \cr \text{27B base} & \text{0/104 (0.0)} & \text{0/104 (0.0)} & \text{0/104 (0.0)} \cr \text{27B LoRA} & \text{99/104 (95.2)} & \text{77/104 (74.0)} & \text{25/104 (24.0)} \cr \hline \end{array}$$ All 104 outputs from the original 27B base group failed compilation, so none completed simulation or passed the requirement checks. **On the same 104 tasks without a matching training template, 79–86 programs from the four 8B fine-tunes compiled and 15–16 met the requirements, compared with 0 for the 8B base model. The fine-tunes therefore solved some tasks without a matching training template, although they performed markedly better on tasks with a match**. The focus of this paper is data generation; we use supervised fine-tuning only to assess the usefulness of the generated data.
AuthorsOpenReview ↗

Respond to LHuw reviewer

Thank you again for your comment, LHuw. We have revised our response based on your last comment.

Q2. What do the ablations isolate?

XGenerator turns the steps of designing, coding, and checking each model by hand into a workflow guided by procedural skills: the model produces an intermediate design (CE-ID), and tools translate it to source and check it against fixed scenarios and criteria. The ablations distinguish two kinds of comparisons. We can vary rule order or repair scope separately. Removing CE-ID, however, also removes deterministic translation and diagnostics that depend on the design, so this comparison measures their joint effect.Below are the results of the ablation experiments from the original paper.

\hline \text{Setting} & \text{Change from AE7} & \text{Compiled} & \text{Simulated} & \text{Met requirements} \cr \hline \text{AE7} & \text{Full method: generate CE-ID, then translate} & 39/40 & 39/40 & 38/40 \cr \text{AE8} & \text{Generate X-language source directly} & 29/40 & 29/40 & 28/40 \cr \text{AE9} & \text{Remove enhanced static checks and required self-review} & 40/40 & 40/40 & 36/40 \cr \text{AE11} & \text{Remove the requirement to retrieve and use past cases} & 38/40 & 38/40 & 36/40 \cr \hline \end{array}$$ The CE-ID generation path improves first-pass reliability and reduces the number of tasks requiring repair, while both methods achieve the same final requirement satisfaction. This comparison establishes the contribution of the combined design–translation–diagnosis pipeline, rather than isolating skill retrieval or experience reuse. AE7 achieved 38/40 first-pass requirement satisfaction, compared with 28/40 for AE8, a 25-percentage-point improvement. In the paired comparison, 12 tasks passed only with AE7, two only with AE8, and 26 with both. Of AE8’s 12 failures, 11 failed to compile, indicating that compilation failures accounted for most of its unsuccessful first submissions. After repair, both methods reached 40/40; AE7 required repair on only two tasks, compared with 12 for AE8. Removing CE-ID also removes the translator and the design-dependent diagnostics and repair procedures, so their effects are assessed jointly. Likewise, AE7–AE9 evaluates enhanced static diagnostics and mandatory self-checking together. **To further compare the effects of different subcomponents on the overall generation performance, we further designed ablation experiments, as detailed below.** $$\begin{array}{llrrr} \hline \text{Model} & \text{Invocation method} & \text{Compile pass} & \text{Simulation pass} & \text{Requirements met} \cr \hline \text{Astra-Low} & \text{Single prompt (ID order)} & 9/40 & 7/40 & 6/40 \cr \text{Astra-Low} & \text{Single prompt (stage order)} & 17/40 & 17/40 & 12/40 \cr \text{Astra-Low} & \text{M9 (first submission)} & 39/40 & 39/40 & 38/40 \cr \text{Sol-Low} & \text{Single prompt (ID order)} & 8/40 & 8/40 & 7/40 \cr \text{Sol-Low} & \text{Single prompt (stage order)} & 7/40 & 7/40 & 4/40 \cr \text{Sol-Medium} & \text{Single prompt (ID order)} & 7/40 & 7/40 & 6/40 \cr \text{Sol-Medium} & \text{Single prompt (stage order)} & 5/40 & 4/40 & 3/40 \cr \text{Sol-Medium} & \text{M9 (first submission)} & 40/40 & 39/40 & 32/40 \cr \text{Sol-High} & \text{Single prompt (ID order)} & 9/40 & 9/40 & 9/40 \cr \text{Sol-High} & \text{Single prompt (stage order)} & 2/40 & 2/40 & 2/40 \cr \text{DeepSeek-Medium} & \text{M9 (first submission)} & 34/40 & 33/40 & 16/40 \cr \hline \end{array}$$
AuthorsOpenReview ↗

Respond to LHuw reviewer

Q3. How do you guard against a model passing the prescribed scenarios by chance without implementing the intended behavior?

In the original evaluation, the scenarios were public and available to XGenerator. Here, we kept the original requirements and archived M9/Astra business logic and changed the test inputs to check whether the same programs still behaved as required under new conditions. We included all 78 programs that passed every public requirement check. The other two (D03-L2-B and D05-L5-B) remain failures in the original evaluation.

How were the new cases constructed and scored? We selected an observable rule from the original requirement, changed an input that exercises it, and specified which outcomes should change or stay the same. In D02-L2-B, a precharge response at time 2 was within the deadline; at time 4 it was late. The energy request followed the response in both cases. The package compiled, and both simulations completed. The contactor should close once in the first case and never in the second; the observed counts were one and zero. Zero is the correct behavior in the late case. Conversely, giving D02-L1-B one extra energy request after the battery was fully charged should leave the number of settlements at one, as observed. We changed test drivers or runtime inputs; we did not rewrite or repair the business logic in response to these tests.

Four worked examples. Each value in the last column is both the expected and observed result. Model / caseChange to the scenarioCompileSim.RequirementD02-L2-B/1Delay precharge response from 1 to 2time units, within deadlinePassRanmain closures: 1D02-L2-B/2Delay precharge response to time 4,past the deadlinePassRanmain closures: 0D02-L1-B/1Request energy again after a full chargebut before settlementPassRansettlements: 1D03-L3-B/2Reorder the same six requestsas A,B,C,A,A,APassRancache misses: 4\begin{array}{llccl}\hline\text{Model / case} & \text{Change to the scenario} & \text{Compile} & \text{Sim.} & \text{Requirement} \cr\hline\text{D02-L2-B/1} & \begin{array}{l}\text{Delay precharge response from 1 to 2} \cr \text{time units, within deadline}\end{array} & \text{Pass} & \text{Ran} & \text{main closures: 1} \cr\text{D02-L2-B/2} & \begin{array}{l}\text{Delay precharge response to time 4,} \cr \text{past the deadline}\end{array} & \text{Pass} & \text{Ran} & \text{main closures: 0} \cr\text{D02-L1-B/1} & \begin{array}{l}\text{Request energy again after a full charge} \cr \text{but before settlement}\end{array} & \text{Pass} & \text{Ran} & \text{settlements: 1} \cr\text{D03-L3-B/2} & \begin{array}{l}\text{Reorder the same six requests} \cr \text{as A,B,C,A,A,A}\end{array} & \text{Pass} & \text{Ran} & \text{cache misses: 4} \cr\hline\end{array}

Results. We tested 52 business-input cases on the first 24 programs and report one final, interpretable business-input case on each of the remaining 54, for 106 cases. Among the latter 54, 40 changed the workload or entity count, 1 changed reserve capacity, and 13 changed events or input conditions. At least one checked output had to change in 45 cases; 9 tested outcomes that should remain invariant. All 106 cases compiled, completed simulation, and met their specified behavioral checks. These are responses of archived programs to particular inputs, not 106 new modeling tasks or proof of all internal mechanisms. D01-L5-B covers only premature ground resets. Test designers could inspect the archived source.

Why report the final results? Of the 53 variants in the first run of the final batch, 41 passed directly (including 3 identifier-only controls), 6 failed the checks used in that run, and the logs did not confirm delivery of the new inputs for the other 6. For 11 tasks, we corrected the test driver or added input-delivery logging and reran the test against the original expected behavior. For battery derating, a separate test checked the charging level after the temperature reading rather than the peak over the entire run. For the 54th task, we used acknowledgments and counters from the business components to check for premature reset, without requiring a normal-completion signal from a process that should not complete. We retained the original records. Identifier changes, controls, and corrective reruns do not count as additional business-input cases.

Reviewer_srtFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes XGenerator, a skill-guided framework for synthesizing verifiable training data for low-resource DSLs. It combines a versioned skill library, a compact executable intermediate design (CE-ID), deterministic code translation, and execution-based validation (compilation, simulation, and requirement checking), with evidence-constrained repair for failed candidates. Experiments demonstrate improved generation reliability over prompting-based baselines, while downstream fine-tuning results provide preliminary evidence of the utility of the synthesized data.

Critical strengths

1.Verification-oriented data synthesis pipeline. The paper addresses a key bottleneck in low-resource DSLs where syntactic validity and simulation completion do not guarantee behavioral correctness. Embedding compilation, simulation, and frozen requirement checks into the synthesis loop—while preserving requirement–design–code–evidence traceability—provides a rigorous and auditable mechanism for curating synthetic training data.

2.Decoupled intermediate representation and scoped repair. Separating semantic design (via CE-ID) from deterministic code translation reduces low-level implementation errors, while binding repairs to failure evidence localizes edits and avoids unconstrained regeneration. Downstream fine-tuning results further demonstrate that the synthesized artifacts effectively transfer executable modeling capabilities to smaller models.

Critical weaknesses
  1. Limited methodological novelty and unclear motivation for skill abstraction.The proposed framework primarily integrates existing techniques, including skill-based knowledge retrieval, intermediate representations, deterministic code generation, execution-based validation, and feedback-guided repair. Although the overall pipeline is practically useful, it remains unclear what substantial methodological novelty distinguishes XGenerator from existing agentic code generation and verification frameworks. In particular, the motivation for introducing the skill abstraction is insufficiently established. It is unclear what unique benefits skills provide over conventional documentation retrieval, structured prompting, or RAG for domain-specific modeling tasks.

  2. Limited comparison with existing methods.The experiments mainly compare XGenerator against internally constructed prompting configurations and subsystem ablations, without direct comparisons to closely related X-language generation or execution-feedback frameworks. This limits the ability to assess the empirical advantages and novelty of the proposed approach. Comparisons with relevant existing methods, such as template-guided generation approaches, would strengthen the evaluation.

Questions for authors

1.How does XGenerator compare with existing X-language generation approaches? It would be helpful to include a lightweight comparison with relevant prior methods, such as template-guided generation (e.g., GenAI for Simulation Model in Model-Based Systems Engineering, arXiv:2503.06422), or other suitable approaches, evaluated using the existing benchmark and validation pipeline.

2.Which parts of the workflow are programmatically enforced by the framework, and which rely on the LLM following natural-language skill instructions? How does the framework ensure that the prescribed workflow is consistently executed?

3.What is the key motivation for introducing the skill abstraction in this framework? What unique benefits do skills offer over conventional documentation retrieval, structured prompting, or RAG, particularly for domain-specific modeling tasks?

2 comments
AuthorsOpenReview ↗

Respond to srtF reviewer

We thank Reviewer srtF for their suggestions on the related methods and the division of responsibilities between models and tools.

Q1. Why did you not run an experiment with the existing template-based method for X-language? How do the two methods differ?

The public repository of Zhang et al. (commit 67c1a8e) provides data, examples, adapter configurations, and training logs. It does not include the trained entity extractor, the code-completion weights, or the end-to-end inference script. We therefore cite the results reported in their paper.

Zhang et al. extract a component template from a design document and use a fine-tuned model to complete it. On one aircraft electrical system, they ran each method 20 times and reported weighted quality scores of 0.825 with templates and 0.156 without (Table VII). For each task, we check whether the complete model compiles, completes simulation in the specified scenarios, and meets its requirements. After at most one repair, Astra passed 78/80 tasks and DeepSeek passed 64/80. Zhang et al. report a weighted quality score for a single system; we report how many of 80 tasks met their requirements.

Q2. Which steps does software enforce, and which depend on the LLM?

Following the skill instructions, the model makes two decisions: how to represent the system in CE-ID and which part of the design to change after a failure. The program determines whether the design can be translated, whether the simulation completes, and whether the requirements are met.

First submission

\to \mathop{\text{Validate + translate}}\limits_{\text{Program}} \to \mathop{\text{Compile + simulate}}\limits_{\text{Program}} \to \mathop{\text{Check contract}}\limits_{\text{Program}}$$ **If the first submission fails: one repair** $$\mathop{\text{Localize fault}}\limits_{\text{Program; LLM fallback}} \to \mathop{\text{Propose patch}}\limits_{\text{LLM}} \to \mathop{\text{Check scope + replay}}\limits_{\text{Program}} \to \mathop{\text{Translate + retest}}\limits_{\text{Program}}$$ The controller freezes all initial designs in the batch before releasing feedback. The model steps in to locate a failure only if automatic localization fails; it must quote the diagnostic and identify a class present in the design. The controller checks the scope of the modification and replays the patch on the frozen design. If either check fails, it rejects the patch. If both pass, it translates the revised CE-ID back into source code, recompiles, reruns the simulation, and evaluates it against the original contract. For example, on D05-L3-B, DeepSeek’s initial design compiled and ran but completed none of the 6 orders. The model changed the replenishment delay from 1 to 3. The contract requires replenishment to follow a shortage but does not prescribe a delay. When re-evaluated under the same criteria, the design completed all 6 orders (Appendix E).
AuthorsOpenReview ↗

Respond to srtF reviewer

Q3. Why use skills? Would document retrieval, structured prompts, or RAG suffice?

We use a skill to specify both when a rule applies and where the model must record its decision. For example, a document can explain how to handle a shortage. Our skill also directs the model to record the response as an event action on a named component in CE-ID. If an order then fails to complete, the diagnostic points to a part of the design the model can edit. Each generated sample records the skill version; a proposed revision is checked on separate development tasks before it replaces that version (§4.2).

On the same 40 tasks, Astra-low met the requirements on 38 first submissions with the full M9 workflow, compared with 6 when all rules were supplied in one prompt in fixed ID order. We also used the single-prompt group’s frozen public evaluator to re-evaluate three complete M9 batches: 40 initial source programs from Astra-low, 40 source programs from DeepSeek-medium, and 40 from Sol-medium.

Three settings in the table. (1) Single prompt, ID order: all 18 rules appear in one prompt, ordered by a fixed identifier unrelated to the modeling steps. (2) Stage order: the same 18 rules and stage labels appear in a single prompt, reordered as “understand the requirements, write the design, check the design.” In both groups, the model is called once per task and submits one CE-ID. The program scores the submission afterward; the model sees no execution feedback. (3) M9: during modeling, the model consults skills as needed and uses tools to check its CE-ID. A tool translates the design into source code. After the first submission, the program compiles the code, runs the simulations, and checks the behavior against the frozen requirements contract. If the submission fails, the model may make one local repair based on the feedback; the program then re-evaluates it against the original contract.

\hline \text{Model} & \text{Invocation method} & \text{Compile pass} & \text{Simulation pass} & \text{Requirements met} \cr \hline \text{Astra-Low} & \text{Single prompt (ID order)} & 9/40 & 7/40 & 6/40 \cr \text{Astra-Low} & \text{Single prompt (stage order)} & 17/40 & 17/40 & 12/40 \cr \text{Astra-Low} & \text{M9 (first submission)} & 39/40 & 39/40 & 38/40 \cr \text{Sol-Low} & \text{Single prompt (ID order)} & 8/40 & 8/40 & 7/40 \cr \text{Sol-Low} & \text{Single prompt (stage order)} & 7/40 & 7/40 & 4/40 \cr \text{Sol-Medium} & \text{Single prompt (ID order)} & 7/40 & 7/40 & 6/40 \cr \text{Sol-Medium} & \text{Single prompt (stage order)} & 5/40 & 4/40 & 3/40 \cr \text{Sol-Medium} & \text{M9 (first submission)} & 40/40 & 39/40 & 32/40 \cr \text{Sol-High} & \text{Single prompt (ID order)} & 9/40 & 9/40 & 9/40 \cr \text{Sol-High} & \text{Single prompt (stage order)} & 2/40 & 2/40 & 2/40 \cr \text{DeepSeek-Medium} & \text{M9 (first submission)} & 34/40 & 33/40 & 16/40 \cr \hline \end{array}$$ On the same 40 tasks, Astra-low passed 32 more tasks on its first submission with the full M9 workflow than with a single prompt in ID order (80 percentage points). This difference arose before public execution feedback was available. One repair per failed task recovered the remaining two, a further gain of 5 percentage points across the cohort. With full M9, DeepSeek and Sol passed 16/40 and 32/40 on the first submission, respectively, and 33/40 and 37/40 after repair.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.