- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces XGenerator, a framework for synthesizing training data for X-language, a low-resource modeling DSL. An LLM uses procedural skills to produce an executable intermediate design, which deterministic tools translate into source code. Compilation, simulation, and frozen requirement checks provide feedback for one repair attempt. The evaluation includes generation experiments on 80 tasks, subsystem ablations on 40 tasks, and downstream fine-tuning. The reported results show improved first-pass generation and executable delivery, although attribution, data-quality validation, and downstream evaluation remain unresolved.
Critical strengths
-
The paper addresses a concrete data-scarcity problem in a specialized modeling language. Linking requirements, executable implementations, and verification evidence is a useful objective because isolated syntax examples provide limited supervision for models that must represent interacting components and dynamic behavior.
-
The separation between model-generated semantic design and deterministic source translation is a sensible engineering choice. The intermediate design explicitly represents components, states, actions, and connections, providing a structured interface for validation and repair while leaving the application behavior to the language model.
-
The validation protocol distinguishes compilation, successful simulation, and requirement satisfaction. Freezing requirements and acceptance criteria during repair, and reevaluating the revised implementation across the prescribed scenarios, provides a clearer basis for assessing repairs than compilation success alone.
-
The evaluation includes multiple language models, tasks across eight domains and five difficulty levels, subsystem ablations, and downstream fine-tuning. The stage-specific results expose meaningful differences between syntactic delivery and behavioral correctness, while the cost reporting makes the additional computational demands visible.
Critical weaknesses
-
The methodological contribution is not sufficiently distinguished from combining procedural skills, structured intermediate representations, deterministic translation, and execution-guided repair. The ablations remove several capabilities together, making it difficult to identify which proposed mechanism explains the improvements. The contribution of reusable experience is particularly unclear: the full ablation configuration retrieves no matching experience during first-pass generation, and the frozen-skill experiments do not evaluate the skill-evolution process illustrated in the overview. The evidence therefore establishes the usefulness of a bundled workflow more clearly than the value of its individual proposed mechanisms.
-
The comparisons do not adequately separate workflow improvements from additional computation. For Astra, the documentation-and-examples baseline reaches 74/80 final requirement successes, compared with 78/80 for the action-patch variant, while total token consumption increases from approximately 28.3 million to 74.2 million. The first-pass improvement is substantial, and DeepSeek shows a larger final gain, but equal repair opportunities do not establish comparable computational budgets. Budget-matched baselines or performance-versus-cost results are needed to assess the practical advantage of the more elaborate workflow.
-
The validation does not sufficiently establish the quality and generalization of the resulting training data. All generation-benchmark scenarios are public, and passing them does not demonstrate mechanism fidelity or robustness to unseen conditions. The broader corpus also includes historical hybrid and continuous models that were not validated through the same complete frozen-contract procedure. In addition, the paper leaves item-level split isolation and semantic near-duplicate verification unresolved. These gaps limit the evidence that the corpus provides diverse, reliable supervision and that downstream performance reflects generalization.
-
The downstream fine-tuning comparison is substantially confounded by output formatting and incomplete model provenance. The evaluator passes raw predictions directly to the compiler, and 527 of the 590 base-model outputs contain Markdown fences that cause parsing failures. The remaining unfenced outputs also fail, so formatting is not the sole explanation, but a normalized evaluation is needed to separate delivery-format gains from modeling improvements. The unverified 27B reference identity and incomplete linkage between loaded checkpoints and test outputs further weaken the interpretation and reproducibility of the training results.
Questions for authors
-
Can the proposed components be compared while holding the available domain information, execution tools, and computational budget fixed? What evidence specifically establishes the contribution of skill organization and reusable experience beyond structured design and deterministic translation?
-
How do the full workflow and the documentation-and-examples baseline compare under matched token or time budgets? Can performance-versus-cost results clarify whether the main benefit is higher first-pass reliability, better final correctness, or reduced repair demand?
-
Can the authors provide held-out scenario or parameter tests, together with a mechanism-fidelity assessment of generated models? Please also clarify the split-group definition and provide an overlap and near-duplicate analysis supporting train–test isolation.
-
What are the base and fine-tuned results after applying the same minimal output normalization, such as removing Markdown fences? Can the authors establish the identities of the evaluated reference models and checkpoints and link them to the reported prediction artifacts?
6 comments
Respond to XRFq review
Q2. What does the extra compute buy?
On Astra, the main benefit was fewer failed first attempts. Of the same 80 tasks, M9 passed 71 on the first submission versus 46 for M3, so it needed 9 repairs instead of 34. After repair the difference narrowed to four tasks (78 versus 74). On DeepSeek, the gain persisted after repair: first-pass successes rose from 9 to 32 and final successes from 34 to 64. Each group comprises 80 tasks. Token counts are in millions; hours are summed across calls.
\hline \text{Model / method} & \text{Initial req.} & \text{Repairs} & \text{Final req.} & \text{R0 (M)} & \text{R1 (M)} & \text{Total (M)} & \text{Hours} \cr \hline \text{Astra / M3} & 46/80 & 34 & 74/80 & 17.831 & 10.482 & 28.313 & 6.124 \cr \text{Astra / M9} & 71/80 & 9 & 78/80 & 67.509 & 6.675 & 74.183 & 8.833 \cr \text{DeepSeek / M3} & 9/80 & 71 & 34/80 & 67.025 & 76.744 & \text{143.769†} & 16.878 \cr \text{DeepSeek / M9} & 32/80 & 48 & 64/80 & 394.682 & 178.707 & 573.388 & 30.069 \cr \hline \end{array}$$ † One DeepSeek M3 call lacks a token count; the total shown is the recorded subtotal. On Astra, M9 spent more on the first pass (67.509M versus 17.831M tokens) and less on repairs (6.675M versus 10.482M). Its total was 2.62 times M3’s. Both methods had the same repair allowance. The table reports their observed costs rather than pass counts at a fixed token or time budget. We will add the first-pass and repair costs separately in Appendix B. **Q3. What conclusions about generalization do the data splits, overlap checks, and unseen scenarios support?** In the original evaluation, the scenarios were public and available to XGenerator. Here, we kept the original requirements and archived M9/Astra business logic and changed the test inputs to check whether the same programs still behaved as required under new conditions. We included all 78 programs that passed every public requirement check. The other two (D03-L2-B and D05-L5-B) remain failures in the original evaluation. **How were the new cases constructed and scored?** We selected an observable rule from the original requirement, changed an input that exercises it, and specified which outcomes should change or stay the same. In D02-L2-B, a precharge response at time 2 was within the deadline; at time 4 it was late. The energy request followed the response in both cases. The package compiled, and both simulations completed. The contactor should close once in the first case and never in the second; the observed counts were one and zero. Zero is the correct behavior in the late case. Conversely, giving D02-L1-B one extra energy request after the battery was fully charged should leave the number of settlements at one, as observed. We changed test drivers or runtime inputs; we did not rewrite or repair the business logic in response to these tests. **Four worked examples.** Each value in the last column is both the expected and observed result. $$\begin{array}{llccl} \hline \text{Model / case} & \text{Change to the scenario} & \text{Compile} & \text{Sim.} & \text{Requirement: expected = observed} \cr \hline \text{D02-L2-B/1} & \begin{array}{l}\text{Delay precharge response from 1 to 2} \cr \text{time units, within deadline}\end{array} & \text{Pass} & \text{Ran} & \text{main closures: 1} \cr \text{D02-L2-B/2} & \begin{array}{l}\text{Delay precharge response to time 4,} \cr \text{past the deadline}\end{array} & \text{Pass} & \text{Ran} & \text{main closures: 0} \cr \text{D02-L1-B/1} & \begin{array}{l}\text{Request energy again after a full charge} \cr \text{but before settlement}\end{array} & \text{Pass} & \text{Ran} & \text{settlements: 1} \cr \text{D03-L3-B/2} & \begin{array}{l}\text{Reorder the same six requests} \cr \text{as A,B,C,A,A,A}\end{array} & \text{Pass} & \text{Ran} & \text{cache misses: 4} \cr \hline \end{array}$$ **Results.** We tested 52 business-input cases on the first 24 programs and report one final, interpretable business-input case on each of the remaining 54, for 106 cases. Among the latter 54, 40 changed the workload or entity count, 1 changed reserve capacity, and 13 changed events or input conditions. At least one checked output had to change in 45 cases; 9 tested outcomes that should remain invariant. All 106 cases compiled, completed simulation, and met their specified behavioral checks. These are responses of archived programs to particular inputs, not 106 new modeling tasks or proof of all internal mechanisms. D01-L5-B covers only premature ground resets. Test designers could inspect the archived source.Respond to XRFq review
Semantic near-duplicates problems: We compared each of the 590 test tasks against the training set (4,712 items) and the validation set (590 items). We found no verbatim matches for a complete code-generation prompt or reference solution. To detect implementation reuse, we consistently renamed identifiers in the reference solutions and compared the full programs, preserving constants, strings, operators, and token order. Of the 410 tasks with executable checks, 306 matched at least one training task. All 264 tasks with a validation match were among those 306. The remaining 104 had no such match in either set. We therefore tested how often the generated code passed on these 104 tasks without a matching implementation template. The three columns report passes out of 104 (percentage). The 8B runs use the original code-generation prompt and the scores after code-fence removal. The 27B LoRA run also uses the original code-generation prompt. All three rates retain the planned denominator of 104.
\hline \text{Model} & \text{Compiled} & \text{Completed simulation} & \text{Met requirements} \cr \hline \text{8B base} & \text{0/104 (0.0)} & \text{0/104 (0.0)} & \text{0/104 (0.0)} \cr \text{8B full, 1e−5} & \text{79/104 (76.0)} & \text{54/104 (51.9)} & \text{15/104 (14.4)} \cr \text{8B full, 5e−6} & \text{82/104 (78.8)} & \text{61/104 (58.7)} & \text{15/104 (14.4)} \cr \text{8B LoRA r128} & \text{86/104 (82.7)} & \text{63/104 (60.6)} & \text{15/104 (14.4)} \cr \text{8B LoRA r64} & \text{85/104 (81.7)} & \text{60/104 (57.7)} & \text{16/104 (15.4)} \cr \text{27B base} & \text{0/104 (0.0)} & \text{0/104 (0.0)} & \text{0/104 (0.0)} \cr \text{27B LoRA} & \text{99/104 (95.2)} & \text{77/104 (74.0)} & \text{25/104 (24.0)} \cr \hline \end{array}$$ All 104 outputs from the original 27B base group failed compilation, so none completed simulation or passed the requirement checks. **On the same 104 tasks without a matching training template, 79–86 programs from the four 8B fine-tunes compiled and 15–16 met the requirements, compared with 0 for the 8B base model. The fine-tunes therefore solved some tasks without a matching training template, although they performed markedly better on tasks with a match**. The focus of this paper is data generation; we use supervised fine-tuning only to assess the usefulness of the generated data.
Respond to XRFq review
Thank you, XRFq, for your comments on skill organization and computational cost.
Q1. With information, tools, and budget held fixed, what do skill organization and reusable experience each contribute?
With information, tools and budget fixed, changing the same rules from ID order to stage order increased first-pass requirement successes from 6/40 to 12/40 on Astra-low, but reduced them from 7/40 to 4/40 on Sol-low, from 6/40 to 3/40 on Sol-medium, and from 9/40 to 2/40 on Sol-high. Only the Astra-low run benefited from stage ordering. AE7 retrieved no past cases during first generation. The logs record case use during repair.
Methodological contribution. XGenerator turns the task-by-task work of designing, coding, and verifying a domain model into an automated data-generation workflow guided by procedural skills. Experts still specify the language rules, tools, and acceptance criteria. For each sample, the model draws on skills to produce a design; tools translate it to source and link execution results to the original requirements. The training sample thus contains both code and evidence for judging whether it meets the specified requirements. Skill evolution is the part of this workflow designed to carry verified experience into later batches; it is distinct from repairing the current task.
Fixed information and budget; only rule order changes. Both groups received the same 18 rules and stage labels in one model call and used the same CE-ID, translator, and evaluation tools. The only change was to list the rules by ID or by modeling stage (understand the requirements, write the design, check the design). Each task had a 900-second limit and a 65,536-token ceiling, with no feedback or repair. The single-prompt rows in the table below report first submissions on the same 40 tasks.
Three settings in the table. (1) Single prompt, ID order: all 18 rules appear in one prompt, ordered by a fixed identifier unrelated to the modeling steps. (2) Stage order: the same 18 rules and stage labels appear in a single prompt, reordered as “understand the requirements, write the design, check the design.” In both groups, the model is called once per task and submits one CE-ID. The program scores the submission afterward; the model sees no execution feedback. (3) M9: during modeling, the model consults skills as needed and uses tools to check its CE-ID. A tool translates the design into source code. After the first submission, the program compiles the code, runs the simulations, and checks the behavior against the frozen requirements contract. If the submission fails, the model may make one local repair based on the feedback; the program then re-evaluates it against the original contract.
\hline \text{Model} & \text{Invocation method} & \text{Compile pass} & \text{Simulation pass} & \text{Requirements met} \cr \hline \text{Astra-Low} & \text{Single prompt (ID order)} & 9/40 & 7/40 & 6/40 \cr \text{Astra-Low} & \text{Single prompt (stage order)} & 17/40 & 17/40 & 12/40 \cr \text{Astra-Low} & \text{M9 (first submission)} & 39/40 & 39/40 & 38/40 \cr \text{Sol-Low} & \text{Single prompt (ID order)} & 8/40 & 8/40 & 7/40 \cr \text{Sol-Low} & \text{Single prompt (stage order)} & 7/40 & 7/40 & 4/40 \cr \text{Sol-Medium} & \text{Single prompt (ID order)} & 7/40 & 7/40 & 6/40 \cr \text{Sol-Medium} & \text{Single prompt (stage order)} & 5/40 & 4/40 & 3/40 \cr \text{Sol-Medium} & \text{M9 (first submission)} & 40/40 & 39/40 & 32/40 \cr \text{Sol-High} & \text{Single prompt (ID order)} & 9/40 & 9/40 & 9/40 \cr \text{Sol-High} & \text{Single prompt (stage order)} & 2/40 & 2/40 & 2/40 \cr \text{DeepSeek-Medium} & \text{M9 (first submission)} & 34/40 & 33/40 & 16/40 \cr \hline \end{array}$$ Paired by task, 11 passed only in stage order and five only in ID order on Astra-low. On Sol-high, one passed only in stage order and eight only in ID order. The Sol-high difference arose before compilation: nine fixed-ID designs and two stage-ordered designs passed CE-ID admission and produced source code. All eleven programs then passed compilation, simulation and requirement checks. Both groups received all the rules; only their order changed. Of the four settings, only Astra-low passed more tasks in stage order. **Full-workflow results.** Rescoring the same 40 tasks with the frozen evaluator gave the following requirement passes from first generation to at most one repair: Astra-low, 38/40 to 40/40; DeepSeek-medium, 16/40 to 33/40; and Sol-medium, 32/40 to 37/40. For both Astra-low and Sol-medium, the rule-order controls used the same model and reasoning setting as the corresponding full M9 workflow. Here we report requirement passes for the full workflow by model. The rule-order experiment above compares the two orderings at fixed model, tools, and budget.