acceptodds
Under review as a conference paper at ICLR 2027

Specification-Driven Generation and Evaluation of Discrete-Event World Models via the DEVS Formalism

Abstract

World models are central to LLM agents that must evaluate actions over long horizons. Yet much existing work focuses on environments governed by physical dynamics or spatial structure, whereas many high-impact domains, including supply chains, procurement networks, and business processes, evolve through discrete events, timing constraints, and causal dependencies. These settings call for discrete-event world models. Hand-engineered simulators provide consistency but are costly to build and adapt, while neural models can accumulate inconsistencies over long rollouts. We introduce DEVS-Gen, an LLM pipeline that generates discrete-event simulators from natural-language specifications using the Discrete Event System Specification (DEVS) formalism. It first plans the component hierarchy and interfaces, then synthesizes component behavior under these interface constraints. For evaluation, we develop a benchmark based on completed specifications that state the required interface and trace-observable behavior. We validate simulator traces against these operational and behavioral requirements, enabling automated scoring and rule-level diagnostics. When averaged across four backbones, DEVS-Gen achieves the highest operational and behavioral scores among the evaluated methods, although its relative performance varies by backbone. Its behavioral gains over both non-executing baselines are statistically significant, whereas those over the execution-enabled baselines are not. It also requires less generation time and fewer tokens to reach higher operational and behavioral performance than the baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
World Modeling, LLM Agents, DEVS Formalism, Discrete-Event Simulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XpidOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces DEVS-Gen, a pipeline that uses large language models (LLMs) and the DEVS formalism to automatically generate discrete-event simulators from natural language specifications, targeting complex domains like supply chains and business processes. The approach structures and synthesizes simulator components to ensure consistency and compliance with requirements. Evaluation on a new benchmark shows DEVS-Gen outperforms baselines in both behavioral accuracy and efficiency.

Critical strengths

I believe this paper is novel within its field. Strengths: The writing is well-organized, the theoretical analysis is sound, and the experimental results support the authors' claims.

Critical weaknesses

a) Since this paper focuses on discrete event systems, its application scope may be somewhat limited. b) The evolution of discrete event systems is generally straightforward to enumerate and quantify; therefore, the necessity of building a world model for simulation in such cases may not be particularly strong. c) Reasoning about the environment in discrete event systems and training agents in such environments seem to be essentially similar. Since the evolution rules can be derived during training, constructing a world model in this context is largely equivalent to generating verifiable signals. There may not be a fundamental difference between these approaches.

Questions for authors

What is the connection between the baselines (including OpenHands-Lite, SWE-Agent-Lite, and OpenHands) and the methods designed for the current discrete event system scenarios? Are there potentially more effective approaches for these tasks?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Xpid(1/2)

Thank you for encouraging us to clarify the motivation, scope and role of executable discrete-event world models. To further examine the value of executable simulators, we additionally compared direct trajectory prediction with generated simulators across five backbones, examining both behavioral accuracy and the cost of repeated prediction ([A2] and [Common Response 3]).

[Q1] Since this paper focuses on discrete event systems, its application scope may be somewhat limited.

[A1] Discrete-event modeling covers a broad range of important applications, including communication networks, manufacturing, healthcare services and supply chains. Common Response 1 discusses its importance and broad applicability, including established approaches for incorporating continuous dynamics. The key is that event timing, resource constraints and interactions among entities provide an effective abstraction for many decision problems. Our work makes these environments easier to construct from natural-language specifications and reuse for prediction and policy optimization.

When continuous processes also affect task outcomes, the generated model can also coordinate tasks and resources while interacting with continuous-dynamics modules. DEVS-Gen can therefore also support the task-coordination layer of hybrid environments.

[Q2] The evolution of discrete event systems is generally straightforward to enumerate and quantify; therefore, the necessity of building a world model for simulation in such cases may not be particularly strong.

[A2] The value of an executable world model lies in reliably executing interacting rules and reusing it to compute. Even when individual rules are simple, predicting a complete trajectory using language models can be hard.

Our additional experiments, reported in Common Response 3 and Appendix E.7 of the revised PDF, compare direct trace prediction with generated simulators across five backbones: gpt-6.1-sol and the four used in our paper. We evaluated both approaches on the same cases. The experiments show two benefits of executable simulators:

  1. Stronger models are unreliable on the direct rollouts. Simulators achieve higher mean scores than direct prediction. Explicit rules therefore still leave a substantial challenge in accurately predicting their combined effects.

  2. Direct trace prediction incurs a new output-generation cost for every rollout.

Direct trajectory prediction requires tracking component states, event times, and interactions across successive events. Errors can accumulate as each prediction depends on the states produced by earlier events. Related temporal-reasoning research also reports declining performance as the number of entities and facts increases [1].

DEVS-Gen addresses this challenge by generating executable simulators that maintain states and schedule events according to the specified rules. Once constructed, a simulator can run under different initial conditions, parameters, and policies. This makes executable world models particularly valuable for policy optimization, where methods such as grid search and TPE require repeated environment evaluations.

Reference: [1] Fatemi B, Kazemi S M, Tsitsulin A, et al. Test of time: A benchmark for evaluating llms on temporal reasoning[C]//International Conference on Learning Representations. 2025, 2025: 94426-94447.

[Q3] Reasoning about the environment in discrete event systems and training agents in such environments seem to be essentially similar. Since the evolution rules can be derived during training, constructing a world model in this context is largely equivalent to generating verifiable signals. There may not be a fundamental difference between these approaches.

[A3] Providing verifiable feedback for agent training and policy optimization is precisely a motivation for our work to construct simulators. In many settings, environment rules and requirements remain ambiguous or incomplete textual descriptions. Turning them into executable state transitions and feedback signals requires modeling and implementation, often involving substantial manual effort. DEVS-Gen addresses this need by translating natural-language specifications into executable programs through structural planning and behavioral synthesis, without training a separate predictive model for each target environment.

Environment generation and agent training or optimization therefore form complementary stages: DEVS-Gen constructs the environment, and subsequent training or optimization uses its feedback to improve policies. Within the executable environment, an agent can try different policies, observe event traces, state changes, and costs, and use these outcomes to improve its decisions. The paper’s existing downstream experiments (Section 5.5) illustrate this use by comparing direct policy generation with decision-making supported by generated simulators.

AuthorsOpenReview ↗

Response to Reviewer Xpid(2/2)

[Q4] What is the connection between the baselines (including OpenHands-Lite, SWE-Agent-Lite, and OpenHands) and the methods designed for the current discrete event system scenarios? Are there potentially more effective approaches for these tasks?

[A4] The coding-agent baselines and DEVS-Gen address the same construction problem: turning a scenario’s natural-language specifications into an executable DES simulator. OpenHands, SWE-Agent and their Lite variants use general-purpose code-development workflows to implement the required dynamics. DEVS-Gen specializes this process through explicit component planning, component-wise implementation and interface integration.

Other specialized modeling approaches provide related representations. As discussed in Appendix A.2, methods that generate PDDL planning domains or SysML models translate natural language into action preconditions, effects and goals, or system structure and behavioral design. Using them for our task would additionally require the event-execution mechanisms and trace interfaces needed to run and evaluate DES simulators.

We also investigated combining DEVS-Gen's structural planning with a general-purpose coding agent. Our additional OH + PlanTree experiment, reported in Common Response 2, provides OpenHands with the complete PlanTree while allowing code execution and debugging. The results show that a strong coding agent can use PlanTree to produce high-quality simulators. Compared with OH + PlanTree, DEVS-Gen maintains comparable quality with stronger backbones, performs better with smaller backbones, and consumes fewer tokens. These findings highlight the value of structural planning and the role of the implementation workflow in generation quality and efficiency.

Thank you again for the discussion. To address your questions about the value of executable world models, we added matched comparisons between direct trajectory prediction and simulator execution across five backbones. These results and the revised discussion explain the benefits of programmatic execution and simulator reuse. Please let us know whether any questions remain.

Reviewer_32KPOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce an LLM pipeline, DEVS-GEN, for generating discrete-event simulators from natural language specifications using the Discrete Event System Specification (DEVS) formalism. The goal is to guide LLM simulator generation by first generating structured scaffolding with defined input-output schemes, then running full generation. The approach exploits modularity: breaking the simulator into subtasks; predefining the input-output schemes for each of these modules allows checking system behavior to ensure certain specifications are satisfied. Authors argue that evaluating LLM simulator generation via matching correct code or against key performance indicators alone is insufficient. The two key components of the DEVS architecture, atomic modules and coupled modules, provide the framework for breaking the simulator generation task down into individual subcomponents and routing them together. The authors create a new benchmark with 10 scenarios covering banking, network protocols, transportation, logistics, security, and epidemic models. Each scenarios has a natural language specification consisting of an operational contract and behavioral requirements, a test suite, and a checker package that reads event traces to calculate the scores. The pipeline takes input specifications and uses DEVS-GEN to create a structured project plan and implement the plan through bottom-up behavioral synthesis. The authors’ benchmark has 66 rules of which 50 micro level rules examine the change of one component while the remaining 16 macro level rules examine causal and time relations between components.

Critical strengths
  1. Applying rigorous software engineering practices to LLM simulator development can be useful, although the concept is not that novel.
  2. Empirical results are positive. The DEVS-GEN framework gets the highest mean scores across the four backbones. Additionally, DEVS-GEN completes the most generations and makes the most fully operational simulators out of all of the methods tested.
  3. The DEVS-GEN pipeline uses less time and fewer tokens to get to higher score levels.
Critical weaknesses
  1. There does not appear to be a definitive backstop to prevent the model from incorrectly implementing the formalism. Because an LLM ultimately creates the plan tree and behavioral components there is still the risk of errors in the pipeline.

  2. A key use case for rapid simulator generation is the ability to adapt policies to new variables. This paper would be stronger if it closed the loop by demonstrating the full pipeline of a policy encountering an out-of-distribution (OOD) event and failing, followed by deploying DEVS-GEN to build a simulation that successfully trains the policy to overcome that event.

  3. Among the fully operational simulators generated by each of the tested methods, the DEVS-GEN approach has the lowest value, although it is noted that the scores are very close. However, this result suggests that the DEVS-GEN pipeline does not make the behavior of a working simulator more correct, it just helps the LLMs generate working simulators to begin with. Hence the main contribution is operational validity.

  4. At lower score levels the baselines are less expensive for simulator generation which means using DEVS-GEN requires paying an upfront cost. Hence it may not be the right tool for quick, low cost simulator generation.

  5. While exploratory, the downstream planning study benchmark is small which makes it difficult to draw conclusions. With only three scenarios, two backbones, where each setting has one policy and one evaluation seed, the study cannot estimate the variance in its findings.

Questions for authors
  1. How does one explicitly verify that the LLM constructed the DEVS scaffolding correctly before moving on to behavioral synthesis? Alternatively, is there empirical evidence showing that the LLM rarely generates undesired formalisms from the text input?

  2. Since the key problem is imposing structure on LLM generation for simulation, can DEVS-GEN be extended to handle hybrid environments that feature both discrete and continuous cases?

  3. Based on the paper it seems like a certain strength of backbone is needed for simulator generation to work. Can you say more about the minimum viable backbone capacity that DEVS-GEN needs?

  4. Are simultaneous events covered by the checker rules to make sure the LLM implementation is correct?

  5. Why are the baselines not incorporating dedicated simulator or world model generation techniques beyond coding agent frameworks?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 32KP(1/4)

Thank you for your suggestions on generation reliability and the evidence for our method's usefulness. We expanded the policy-generation and multi-seed evaluations ([A5] and [Common Response 4]), examined smaller backbones to find minimum viable backbone capacity ([A7]), developed an environment-change case illustrating simulator updates and policy adaptation ([A2]), and compared behavioral score on common operational settings ([A3] and [Common Response 5]).

[Q1] Weakness: There does not appear to be a definitive backstop to prevent the model from incorrectly implementing the formalism. Because an LLM ultimately creates the plan tree and behavioral components there is still the risk of errors in the pipeline.

Question: How does one explicitly verify that the LLM constructed the DEVS scaffolding correctly before moving on to behavioral synthesis? Alternatively, is there empirical evidence showing that the LLM rarely generates undesired formalisms from the text input?

[A1] DEVS-Gen supports correct implementation of the formalism through explicit interface contracts, plan revision and interface reconciliation during generation. These mechanisms are described in Sections 3.1–3.2. While they do not constitute independent formal verification, they provide consistency checks before and during behavioral synthesis. In our experiments, we rarely observed violations of the DEVS component hierarchy and interface conventions.

Before behavioral synthesis, the planner first drafts the root plan and then revises it. The revision prompt asks the model to audit event flows and interfaces, correct omissions or contradictions, and confirm that each coupling connects an event-producing source to an event-consuming destination. The resulting parent plans establish the component types, constructor arguments and port contracts inherited by their children. Behavioral synthesis therefore starts from an explicit component hierarchy and interface specification.

During synthesis, each component is generated under its local contract. Before generating a parent component, the pipeline extracts the actual constructor and interfaces from the generated child code. The parent uses these implemented interfaces when instantiating and connecting its children, reducing inconsistencies between the plan and the code.

[Q2] A key use case for rapid simulator generation is the ability to adapt policies to new variables. This paper would be stronger if it closed the loop by demonstrating the full pipeline of a policy encountering an out-of-distribution (OOD) event and failing, followed by deploying DEVS-GEN to build a simulation that successfully trains the policy to overcome that event.

[A2] Thank you for the suggestion. We added a policy-adaptation case (presented at https://demo2.devs-demo.workers.dev/ ). After the decision-making agent made the first decision with the help of DEVS simulator, we changed the environment to introduce an OOD situation. Then the decision-making agent inspected the log, identified the error, told DEVS-Gen to update the simulator, and adjusted the replenishment policy using its feedback. The adapted policy reduced real cost in the changed environment by 96.9% compared with the original policy, demonstrating successful policy adaptation in this case.

  1. Settings: The case involves changes to both the supply-network structure and demand patterns. One of two warehouses is closed, and its stores are reassigned to the remaining warehouse or direct factory supply. The directly supplied store also faces a different replenishment lead time and a new weekday–weekend demand pattern, requiring the policy to adapt to both the changed network and demand mechanism.
  2. Process of the case: We provided GPT-6.1-sol with 28 days of operating records from the changed environment. The agent successfully identified increased shortages and a changed demand pattern. It revised the specifications and called DEVS-Gen to construct a new simulator. Then the agent used new simulator's feedback to test and select the best policy.
  3. How we evaluate the policies: The original policy retains its replenishment rules, with only its supply-node assignments adapted to the new network. After policy selection, we measured real cost for each policy–environment combination on the same 20 environment seeds, with 100 days per episode from the corresponding specified initial state. The table reports mean real cost and sample standard deviation across environment seeds.
PolicyEnvironmentCost
OriginalOriginal68,540.25±1,410.42
OriginalChanged1,442,662.51±76,818.26
AdaptedChanged44,491.23±1,920.36

This case illustrates how operating records can reveal a mismatch between the model and the changed environment, prompting simulator updates through DEVS-Gen, and how simulator feedback can support policy adaptation and improve performance. We will update App. K.2 with the case and its results.

AuthorsOpenReview ↗

Response to Reviewer 32KP(2/4)

[Q3] Among the fully operational simulators generated by each of the tested methods, the DEVS-GEN approach has the lowest value, although it is noted that the scores are very close. However, this result suggests that the DEVS-GEN pipeline does not make the behavior of a working simulator more correct, it just helps the LLMs generate working simulators to begin with. Hence the main contribution is operational validity.

[A3] DEVS-Gen improves the behavioral quality of working simulators as well as the success rate of generating them. The lowest conditional average reported in Appendix E.1 does not establish that its contribution is limited to operational validity, because each method’s average uses its own fully operational subset. These subsets contain different tasks and cannot be directly ranked as a comparison of behavioral accuracy.

To compare working simulators on common tasks, we added the analysis reported in Common Response 5 and Appendix E.1 of the revised PDF. We retain the 21 backbone–scenario settings where all five methods produce at least one fully operational simulator. For each method, we average its fully operational generations within each setting, then average equally across settings. DEVS-Gen achieves the highest mean behavioral score. This shows a behavioral advantage even when the comparison includes only fully operational simulators on common settings.

[Q4] At lower score levels the baselines are less expensive for simulator generation which means using DEVS-GEN requires paying an upfront cost. Hence it may not be the right tool for quick, low cost simulator generation.

[A4] DEVS-Gen’s upfront planning cost should be assessed against the practical goal of reliably obtaining a usable simulator. The baselines’ advantage at lower cumulative score levels does not by itself establish a reliable low-cost route to that goal.

Here, a lower score level refers to the cumulative outcome achieved within a resource budget, rather than the quality of a single successfully generated simulator. A method can reach an early positive score through a small number of inexpensive generations by chance, even if that outcome is difficult to reproduce.

To assess the practical value of these early results, we analyzed their distribution across scenarios and repetitions in Appendix E.2. They are concentrated in easier scenarios such as HouseHeating, SEIRD and StoreCashier. Among baseline settings with at least one low-cost, positive-scoring generation, only 31.0% under time budgets and 26.5% under token budgets produce such results in all three repetitions. These results indicate that low-cost positive outcomes are not consistently reproducible across repetitions.

DEVS-Gen invests in the component hierarchy and interfaces before implementation. Once these are established, components can be generated in parallel, and each call processes only its local context. The resource–outcome curves show that this investment pays off at higher cumulative score levels, where DEVS-Gen generally requires less time and fewer tokens.

[Q5] While exploratory, the downstream planning study benchmark is small which makes it difficult to draw conclusions. With only three scenarios, two backbones, where each setting has one policy and one evaluation seed, the study cannot estimate the variance in its findings.

**[A5] ** Thank you for the suggestion. In Common Response 4, we added repeated policy generation and multi-seed evaluation to quantify policy-generation variability and environmental randomness. Using GPT-5.2, GLM-4.7, and GPT-6.1-sol, we made five independent policy-generation attempts per method and scenario, evaluating each valid policy on the same 20 environment seeds.

The 20-seed evaluations provide stable estimates of individual policy costs. Across the 130 valid policies, the estimated relative standard error of the mean cost has a median of 0.62% and a maximum of 3.98%, indicating limited uncertainty from environmental randomness.

The repeated experiments also support our original conclusion in Section 5.5. DEVS-Augmented achieves the lowest mean cost in eight of the nine backbone–scenario settings. This further supports DEVS-Gen’s practical value: its generated simulators provide useful environment feedback that helps agents find lower-cost policies.

AuthorsOpenReview ↗

Response to Reviewer 32KP(3/4)

[Q6] Since the key problem is imposing structure on LLM generation for simulation, can DEVS-GEN be extended to handle hybrid environments that feature both discrete and continuous cases?

[A6] Yes. DEVS-Gen can be extended to hybrid environments by using its component structure and event interfaces to connect discrete task logic with continuous-dynamics modules. Common Response 1 discusses established approaches for continuous simulation and hybrid integration. Section 3.3 outlines this extension through physics coupling.

Specifically, DEVS-Gen's structural planning stage can be extended to identify discrete and continuous components, assign their responsibilities, and define their interactions. For example, DEVS components can manage task sequencing and resource allocation, while continuous modules handle physical dynamics. Through interface adaptation, DEVS task events can drive physical controllers, and relevant physical outcomes, such as collisions or arrivals, can be translated into DEVS input events. This preserves DEVS-Gen’s benefit of defining component responsibilities and interfaces before implementation.

[Q7] Based on the paper it seems like a certain strength of backbone is needed for simulator generation to work. Can you say more about the minimum viable backbone capacity that DEVS-GEN needs?

[A7] Thank you for the suggestion. We added experiments with Qwen3.5 4B, 9B, and 27B from the same model family to examine how backbone capacity affects simulator generation. We compare DEVS-Gen with single, which generates the complete simulator in one LLM call.

ModelDEVS-Gen: Score_opesingle: Score_opeDEVS-Gen: Score_behsingle: Score_beh
Qwen3.5-4B0.233 ± 0.1530.159 ± 0.1330.117 ± 0.0630.118 ± 0.081
Qwen3.5-9B0.237 ± 0.1520.252 ± 0.0600.173 ± 0.1270.149 ± 0.090
Qwen3.5-27B0.661 ± 0.0430.503 ± 0.0780.569 ± 0.1000.418 ± 0.124

The 4B and 9B models score relatively low under both settings, with limited gains from DEVS-Gen over single. At 27B, both operational validity and behavioral conformance improve markedly, and DEVS-Gen exceeds single on both metrics.

These differences across model sizes reflect the backbone’s implementation capabilities. DEVS-Gen’s requirement organization and structural planning stages still need to integrate the complete specification and establish component responsibilities and interfaces. Although DEVS-Gen's component-wise generation reduces the local context burden, its benefits depend on the quality of the initial plan. This may explain why smaller models benefit less, while stronger backbones can make more effective use of the staged generation process.

[Q8] Are simultaneous events covered by the checker rules to make sure the LLM implementation is correct?

[A8] Yes. The checker includes rules and test cases for simultaneous events. For example, in SAA, requests arriving while AlarmAdmin is busy must be ignored, whereas a request arriving exactly when authentication completes must be accepted. The Busy_And_Exact_Boundary test case explicitly checks this boundary behavior. ABP also includes a check for an ACK and a retransmission occurring at the same timestamp. An invalid retransmission is still flagged even if its record appears before the ACK record in the trace.

These checks evaluate the behavior required by the scenario specification, so changing the order of same-timestamp log entries alone does not change the correctness judgment.

Reviewer_LDMROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates a method for generating executable discrete-event simulators based on natural language specifications. We propose the DEVS-Gen framework, which first constructs a hierarchical "PlanTree" to define the responsibilities and interfaces of individual components, and subsequently synthesizes and assembles these components into a fully functional simulator. Furthermore, we introduce a benchmark suite comprising ten distinct scenarios, utilizing execution traces to evaluate the operational validity of the generated simulators and their adherence to specified behavioral requirements. Experiments conducted with four different base language models demonstrate that this approach outperforms agents lacking code-execution capabilities, striking an effective balance between resource consumption and output quality. Meanwhile, comparisons with agents that possess code-execution capabilities reveal more nuanced and complex distinctions.

Critical strengths

1 The evaluation design is a key contribution, valuable even independently of the generation method itself. By separating the operational and behavioral layers and employing a micro/macro rule structure, the framework allows for precise localization of failures, providing far more diagnostic information than a simple binary "does it run" assessment. 2 The pipeline addresses a specific integration challenge, going beyond mere decomposition of large prompts into smaller ones. It mandates defining interfaces before implementing behaviors, aligns parent nodes using actual code excerpts from their child nodes, and, in doing so, enables parallel generation. 3 The reporting is transparent. The authors conducted a stratified permutation test and explicitly noted that, compared to execution-based baselines, the results were not statistically significant and remained dependent on the underlying backbone model—avoiding any misleading framing that would suggest their method is superior in every respect.

Critical weaknesses
  1. Whether the benefits come from the formalization of DEVS or from "planning first and then generating blocks" is currently unclear. The transition from single to single-xdevs involves simultaneous changes to the representation, the API, and the generation process (i.e., the number of steps). A controlled experiment—such as using a "staged" plain-Python baseline or feeding the PlanTree directly to a coding agent—is missing. Furthermore, on the GPT-5.2 model, the Score_beh for single (0.893) is nearly identical to that of DEVS-Gen (0.894), suggesting that the performance gap narrows as the underlying backbone model improves; this observation should be explicitly discussed.
  2. The performance advantage is primarily driven by a higher success rate in generating executable code, rather than by greater accuracy in implementing the underlying dynamics. As shown in Table 4, the Score_beh for all methods falls within the 0.86–0.88 range when evaluated solely on the subset of "fully operational" programs. It is recommended to perform a comparative analysis, broken down by specific rules on the subset of cases that are operational across all methods, and to incorporate this distinction into the main text.
  3. The benchmark and downstream evidence are somewhat limited, and some claims are overstated. All 10 scenarios are adapted from publicly available Cadmium models, yet the issue of data contamination is not addressed. The result of 0.200 ± 0.000 on "Llama 4 Scout" requires further explanation. Additionally, the "physics coupling" mentioned in Section 3.3 is not actually implemented, yet it is included under the "Methods" section.
Questions for authors

please address the weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer LDMR(1/3)

Thank you for your comments on the contributions of the DEVS representation and generation workflow, and on the interpretation of our results. To better answer your questions, we added PlanTree-guided coding-agent experiments (In [A1] and [Common response 2]), ran the benchmark on stronger models (In [A2]), and compared behavioral conformance on common operational settings (In [A3] and [Common Response 5]).

[Q1] Whether the benefits come from the formalization of DEVS or from "planning first and then generating blocks" is currently unclear. The transition from single to single-xdevs involves simultaneous changes to the representation, the API, and the generation process (i.e., the number of steps). A controlled experiment—such as using a "staged" plain-Python baseline or feeding the PlanTree directly to a coding agent—is missing.

[A1] Our results suggest that the benefits come from combining DEVS structure with a generation workflow designed to use it. DEVS separates component behavior from interactions through explicit interfaces and hierarchical composition. It enables PlanTree planning, local component implementation and parallel generation. Although adopting xDEVS alone introduces implementation burden, DEVS-Gen uses these structural properties to improve generation quality and efficiency.

To better show this, following your suggestion, we added the OH + PlanTree experiment(provide plantree to OpenHands), reported in Common Response 2. All the ablations clarify the five aspects:

  1. Adopting xDEVS introduces implementation burden. Both single and single-xdevs use one LLM call, but single-xdevs scores lower on both metrics with every backbone.

  2. DEVS-Gen’s workflow makes this structure easier to implement. Under the same xDEVS requirement, adding PlanTree planning and component-wise implementation improves scores.

  3. PlanTree provides useful structural guidance. Giving OpenHands the complete PlanTree raises GPT-5.2’s behavioral score from 0.855 to 0.9831.

  4. DEVS-Gen performs better with smaller backbones. DEVS-Gen scores higher than OH + PlanTree with Qwen3-Coder, GLM-4.7 and Llama-4-Scout, while maintaining high quality with GPT-5.2.

  5. DEVS modularity supports efficient generation. DEVS-Gen uses fewer tokens than OH + PlanTree. Local component contexts reduce repeated project-wide processing, and explicit interfaces permit parallel generation.

[Q2] Furthermore, on the GPT-5.2 model, the Score_beh for single (0.893) is nearly identical to that of DEVS-Gen (0.894), suggesting that the performance gap narrows as the underlying backbone model improves; this observation should be explicitly discussed.

[A2] Thank you for highlighting this observation. We evaluated the methods with GPT-6.1-sol using the benchmark we constructed. It does not show a further narrowing or reversal of the mean performance gap. The results are summarized below and reported in Appendix E.5 of the revised PDF. Entries are mean behavioral scores ± sample standard deviations across three repetitions.

MethodGPT-5.2: Score_opeGPT-5.2: Score_behGPT-6.1-sol: Score_opeGPT-6.1-sol: Score_beh
DEVS-Gen0.985 ± 0.0270.894 ± 0.0170.9451 ± 0.05810.9050 ± 0.0765
single0.944 ± 0.0190.893 ± 0.0450.9098 ± 0.09470.8855 ± 0.0956
single-xdevs0.650 ± 0.0450.594 ± 0.0830.7674 ± 0.11490.6822 ± 0.0569

We also expect single-pass generation to face greater difficulty in more complex scenarios, where more component behaviors and interfaces must be implemented together. We will discuss these observations in Section 5.4 of the revised PDF.

AuthorsOpenReview ↗

Response to Reviewer LDMR(2/3)

[Q3] The performance advantage is primarily driven by a higher success rate in generating executable code, rather than by greater accuracy in implementing the underlying dynamics. As shown in Table 4, the Score_beh for all methods falls within the 0.86–0.88 range when evaluated solely on the subset of "fully operational" programs. It is recommended to perform a comparative analysis, broken down by specific rules on the subset of cases that are operational across all methods, and to incorporate this distinction into the main text.

[A3] Table 4 reports conditional averages over each method’s own fully operational generations. These generations contain different tasks, so their averages cannot be directly compared.

DEVS-Gen’s advantage includes more accurate implementation of the underlying dynamics, alongside a higher success rate in generating operational simulators. To show this, we added a comparison on "common tasks" following your suggestion. The full settings and results are in Common Response 5 and Appendix E.1 of the revised PDF.

In the comparision, we retain the 21 backbone–scenario settings where all five methods produce at least one fully operational simulator. DEVS-Gen achieves the highest mean behavioral score, 0.9118, compared with 0.8402–0.9037 for the baselines. This shows a behavioral advantage even when the comparison includes only fully operational simulators on common settings.

We also examined the per-rule scores to understand where dynamics implementation improves. We compare all 59 behavioral rules represented in the 21 common settings. The results show that DEVS-Gen’s behavioral improvements vary across rules, does not show a clear pattern. The average scores of the methods on each rule are listed in Appendix E.1 of the revised PDF.

[Q4] The benchmark and downstream evidence are somewhat limited, and some claims are overstated.

[A4] Thanks for your advice. Our main claims concern improved simulator generation, generation efficiency, and the usefulness of simulator feedback for policy optimization. We will state these claims, and show what we did to strengthen them.

Simulator generation. Our claim is that DEVS-Gen improves operational validity and behavioral conformance on the benchmark. DEVS-Gen achieves the highest scores across the four backbones. The analysis in Common Response 5 further identifies DEVS-Gen's behavioral improvements among fully operational simulators. Together, these results support improvements in both producing usable simulators and implementing their dynamics.

Generation efficiency. Our claim is that DEVS-Gen reaches higher generation-quality levels with less time and fewer tokens than the coding-agent baselines. The resource-versus-outcome analysis in Section 5.3 compares the scores attained within the same time or token budget. These results support the efficiency of our component-wise implementation, which reduces repeated processing through local contexts and permits parallel generation.

Downstream usefulness. Our claim is that generated DEVS simulator feedback can help improve policy search. To test whether this benefit persists across independent decisions and environmental randomness, we added repeated experiments, reported in Common Response 4, Table 2 and Appendix K of the revised PDF. We evaluated GPT-5.2, GLM-4.7 and GPT-6.1-sol on three supply-chain scenarios, with five independent policy-generation attempts per method and scenario and 20 common environment seeds per evaluable policy. The repeated results still support the practical benefit of DEVS simulator feedback.

[Q5] All 10 scenarios are adapted from publicly available Cadmium models, yet the issue of data contamination is not addressed.

[A5] We agree that the open-source Cadmium may have been included in the pretraining data. We will acknowledge this potential contamination risk in the Limitations section.

However, several aspects of our benchmark construction may mitigate this risk. The Cadmium models serve as source material for constructing the benchmark, rather than being used directly as the task statements or reference simulators for scoring. We construct the task specifications from their source code and descriptions, and evaluate generated simulators against the resulting operational and behavioral requirements. Familiarity with the original code therefore does not by itself ensure a high score.

We also observe that DEVS-Gen's generated PlanTrees and implementations differ from the original Cadmium models in structure and code, rather than simply reproducing their implementations verbatim.

AuthorsOpenReview ↗

Response to Reviewer LDMR(3/3)

[Q6] The result of 0.200 ± 0.000 on "Llama 4 Scout" requires further explanation.

[A6] We checked the original records and confirmed that the reported value is correct. For each repetition, we average the operational scores across the ten scenarios, then report the mean and standard deviation across the three repetitions. In each repetition, Llama 4 Scout happened to achieve an operational score of 1 on exactly two scenarios and 0 on the remaining eight. Thus, all three repetitions yielded an average of 0.200, resulting in a standard deviation of 0.000.

[Q7] Additionally, the "physics coupling" mentioned in Section 3.3 is not actually implemented, yet it is included under the "Methods" section.

[A7] Physics coupling in Section 3.3 is a natural use of DEVS’s existing event interfaces rather than a new simulation architecture. As discussed in Common Response 1, a discrete-event model can serve as the task-coordination layer of a hybrid environment. It needs interface adaptation: DEVS task events can drive physical controllers, and relevant physical outcomes, such as collisions or arrivals, can be translated into DEVS input events. This connects task logic with continuous dynamics without changing DEVS-Gen’s generation pipeline.

The current demo in Fig. 2 illustrates event-driven visualization using simulator outputs. We will revise Section 3.3: clarify its role as an interface-based extension and describe the demo.

Thank you again for your thoughtful review. Following your suggestions, we added the PlanTree-guided coding-agent experiment and behavioral comparisons on common operational settings, and clarified the backbone-dependent gains and potential contamination risk in the revised paper. We would appreciate hearing whether our responses address your questions, or whether any points would benefit from further clarification. We would be happy to discuss them.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Common Response

[Common Response 1] The application scope of discrete-event world models, extensions to hybrid environments, and the role of physics coupling.

[Reviewer Xpid Q1, Reviewer 32KP Q6, Reviewer LDMR Q7]

Discrete-event modeling captures the dynamics relevant to a broad range of important decision problems. Problems in communication networks, manufacturing, healthcare services, transportation and supply chains all involve event timing, resource constraints and interactions among entities. Models of these processes support decisions about scheduling, capacity allocation and congestion, with consequences for cost, delay and service quality. For example, network protocol behavior can be studied through packet transmission, reception, timeout and retransmission events without resolving the underlying continuous signal dynamics. Discrete-event world models thus provide an effective abstraction for evaluating system behavior and policy choices.

Our motivation is to make these environments easier to construct and adapt. New workflows, resource configurations and operating rules require corresponding models before agents can evaluate candidate policies. DEVS-Gen translates natural-language specifications into executable simulators through structural planning and component-wise synthesis. The resulting models provide reusable feedback for repeated prediction and policy optimization.

When continuous dynamics also affect the outcomes of interest, discrete-event modeling offers established ways to incorporate them:

  • Represent continuous dynamics through discrete-event approximations. Quantized State Systems (QSS), for example, approximate continuous dynamics using discrete-event models within the DEVS framework (Kofman and Junco, 2001).

  • Couple the task model with a continuous simulator. DEVS manages task sequencing, resource allocation and coordination, while a physics engine handles motion and contact. An event interface connects the two: task events drive physical actions, and relevant physical outcomes return as DEVS input events. The generated model can thus serve as the coordination layer of a hybrid environment.

These extensions build on the generated model’s explicit event interfaces. The event-driven visualization in Fig. 2 illustrates a related use: simulator outputs drive scene updates. Section 3.3 outlines how these interfaces can further connect task logic with continuous dynamics.

Reference:

[1] Kofman E, Junco S. Quantized-state systems: A DEVS approach for continuous system simulation[J]. Transactions of the Society for Modeling and Simulation International, 2001, 18(3): 123–132.

[Common Response 2] Contributions of the DEVS representation and generation workflow, and their relationship to general-purpose coding-agent baselines.

[Reviewer LDMR Q1, Reviewer Xpid Q4]

DEVS-Gen has two main features: (1) using xDEVS.py to organize the simulator; (2) structural planning and component-wise implementation. We did ablations to examine how they affect generation quality and efficiency.

To examine the contributions of these two factors, we bring together the original ablation of single, single-xdevs and DEVS-Gen, and the additional experiment comparing OpenHands and DEVS-Gen with OH + PlanTree. These experiments are reported in Section 5.4 and Appendix E.5 of the revised PDF. We present their settings and results below, followed by an analysis of generation quality and efficiency.

We use the following generation settings.

  • single directly generates a free-form simulator in one LLM call.

  • single-xdevs directly generates a xDEVS.py-based simulator in one LLM call.

  • DEVS-Gen uses xDEVS with PlanTree planning, component-wise implementation and integration. It uses no execution feedback during generation.

  • OpenHands implements the specifications through a general-purpose coding-agent workflow, with code execution and debugging, without an externally supplied PlanTree.

  • OH + PlanTree receives the complete PlanTree from the corresponding DEVS-Gen run and implements that architecture through OpenHands. It can choose the implementation library and use execution feedback for debugging.

Each backbone is evaluated on ten scenarios with three repetitions per setting. Score_ope measures operational validity and Score_beh measures behavioral conformance, with higher values indicating better performance. Tokens are in unit of 10410^4 (continue in the next comment)

AuthorsOpenReview ↗

Common Responses (2/4)

BackboneMethodScoreope\mathrm{Score}_{ope}Scorebeh\mathrm{Score}_{beh}Planning time (s)Implementation time (s)Planning tokensImplementation tokens
GPT-5.2DEVS-Gen0.9850.894132.231.79.5976.847
single0.9440.893-71.4-0.412
single-xdevs0.6500.594-95.7-0.706
OpenHands0.9610.855-599.5†-43.239†
OH + PlanTree1.00000.9831132.2388.49.597130.172
Qwen3-CoderDEVS-Gen0.6100.494309.160.67.9645.582
single0.3920.316-43.9-0.376
single-xdevs0.0030.003-53.0-0.530
OpenHands0.6230.488-355.5†-73.094†
OH + PlanTree0.36320.2663-155.7†-73.024†
GLM-4.7DEVS-Gen0.8820.771294.970.410.0145.888
single0.5840.549-155.5-0.349
single-xdevs0.0840.065-141.3-0.538
OpenHands0.8710.809-213.2-66.286
OH + PlanTree0.71620.6301-1512.9†-406.871†
Llama-4-ScoutDEVS-Gen0.2000.146162.473.36.4316.123
single0.0700.043-30.5-0.297
single-xdevs0.0000.000-36.0-0.444
OpenHands0.0920.057-158.6†-5.643†
OH + PlanTree0.00000.0000-29.0†-1.516†

† Resource means only use time/token records of completed generations.

These comparisons identify five aspects of DEVS-Gen:

  1. the burden of adopting xDEVS: single versus single-xdevs. Both use direct generation without execution feedback, and single scores higher on both metrics for every backbone. The results show that this implementation requirement makes direct generation harder.

  2. DEVS-Gen make it easier to follow xDEVS.py requirement: single-xdevs versus DEVS-Gen. These two settings both implement DEVS using xDEVS and receive no execution feedback. DEVS-Gen adds PlanTree planning and component-wise implementation and achieves higher scores on both metrics for every backbone. Comparing DEVS-Gen with single also shows the net benefit of the complete design: DEVS-Gen has larger gains on the three smaller backbones.

  3. PlanTree as structural guidance benefits: OpenHands versus OH + PlanTree. With GPT-5.2, the behavioral score increases from 0.855 to 0.9831. This result shows that PlanTree provides useful component-level guidance that a capable coding agent can turn into high-quality simulators.

  4. Component-wise workflow supports more backbones: OH + PlanTree versus DEVS-Gen. DEVS-Gen maintains high quality with GPT-5.2 and achieves higher scores than OH + PlanTree with the weaker backbones. DEVS-Gen implements each component within its local context and uses the implemented child interfaces when integrating parent components. This follows DEVS’s hierarchical composition, helping models carry the plan through to executable code.

  5. **DEVS modularity supports efficient implementation. **DEVS-Gen requires less time and token than any other methods. Each DEVS-Gen call focuses on a component and its relevant interfaces, reducing repeated processing of the growing project. Once interface contracts are established, independent components can be generated through parallel LLM calls. DEVS’s separation of component behavior and interactions thus supports both local implementation and concurrent generation.

The ablation results show that requiring xDEVS alone makes generation harder. DEVS-Gen nevertheless benefits from this structure by designing its generation workflow around DEVS’s modularity, explicit interfaces and hierarchical composition. The resulting workflow helps models implement the component structure more accurately and supports efficient parallel generation.

AuthorsOpenReview ↗

Common Responses (3/4)

[Common Response 3] Given explicit environment rules, can a foundation model directly predict system evolution instead of constructing an executable simulator?

[Reviewer Xpid Q2, Reviewer 32KP Q9]

Explicit rules do not make their combined effects straightforward to predict. A complete rollout requires maintaining component states, event times and interactions throughout the trajectory consistently, which is difficult for foundation models [1]. Generating an executable simulator turns these dependencies into programmatic state updates and event scheduling, providing a model that can be repeatedly queried under different conditions.

To test whether direct prediction can replace this executable model, we added experiments comparing foundation models used directly as world models with simulators generated by DEVS-Gen. The results are provided below and in Appendix E.7 of the revised PDF.

We compared two settings. (1) direct prediction, each backbone receives the scenario specification and case inputs and is asked to output a complete event trace. We repeat this process in three independent LLM calls per case. (2) simulator execution, we run the three independently generated DEVS-Gen simulators for that backbone and scenario on the same case, obtaining one trace from each simulator.

For each of the ten scenarios, we selected one representative test cases from our benchmark.

For each setting, we first average behavioral scores across the ten scenarios within each repetition. The table reports the mean and sample standard deviation of these three repetition-level averages. Higher scores indicate better behavioral conformance. We also report mean token cost in one direct prediction rollout.

"Direct" is for direct prediction, "Simulator" is for simulator execution. Qwen3-Coder is Qwen3-Coder-30B-A3B-Instruct.

BackboneDirect ScoreSimulator ScoreDirect TokensDirect Time (s)
GPT-5.20.698 ± 0.0580.821 ± 0.0289,756 ± 12,41675.8 ± 123.8
GLM-4.70.642 ± 0.0310.734 ± 0.09631,289 ± 20,180676.1 ± 436.7
Qwen3-Coder0.320 ± 0.0840.466 ± 0.06720,667 ± 24,086507.0 ± 644.2
Llama4-Scout0.084 ± 0.0210.138 ± 0.0084,637 ± 1,94537.0 ± 26.0
GPT-6.1-sol0.643 ± 0.0560.839 ± 0.1448,060 ± 3,526106.6 ± 118.2

Stronger models are unreliable on the direct rollouts. Simulators achieve higher mean scores than direct prediction.

Direct prediction also requires generating substantial output tokens for every rollout. Longer horizons, more entities, and finer logging increase the output required for each prediction.

Moreover, an executable simulator makes this computation reusable. Once constructed, it can run under different initial conditions, or parameters, and can avoid the uncertainty of LLM-based generation of complete trajectories. Optimization methods such as grid search and TPE require many environment queries, so executable simulators are useful for them.

Reference:

[1] Fatemi B, Kazemi S M, Tsitsulin A, et al. Test of time: A benchmark for evaluating llms on temporal reasoning[C]//International Conference on Learning Representations. 2025, 2025: 94426-94447.

AuthorsOpenReview ↗

Common Responses (4/4)

[Common Response 4] Limited downstream evidence and unmeasured variability across policy generation and environment seeds.

[Reviewer LDMR Q4, Reviewer 32KP Q5]

The downstream study tests whether generated simulator feedback helps agents find lower-cost policies. The original evaluation used one policy and one environment seed per setting, leaving two sources of variability unmeasured: whether independent policy searches find similarly good policies, and whether a policy’s measured cost changes substantially across environment seeds.

We added repeated policy generation and multi-seed evaluation to address both questions. The results are provided below and in Table 2 and Appendix K of the revised PDF.

Specifically, we evaluated GPT-5.2, GLM-4.7 and GPT-6.1-sol on three supply-chain scenarios: branching supply chain, multi-product, and assembly and distribution. We compared Direct-Policy, which directly proposes a policy; Native-Python, which can construct and use its own simulation code; and DEVS-Augmented, which uses a simulator generated by DEVS-Gen. For each scenario, we generated policy for 5 times. Each policy, we tested it on 20 seeds. Each policy's score is the average cost over the 20 seeds. The table reports the mean and sample standard deviation across these policy-level averages. Costs are in units of 10410^4; lower is better.

BackboneMethodBranching supply chainMulti-productAssembly and distribution
GPT-5.2Direct-Policy4822.84 ± 1275.3122.45 ± 20.155.61 ± 1.60
Native-Python4252.60 ± 15.5827.32 ± 21.206.94 ± 2.45
DEVS-Augmented14.00 ± 8.4919.29 ± 18.666.40 ± 3.14
GLM-4.7Direct-Policy2613.63 ± 3744.0342.93 ± 28.967.57 ± 1.97
Native-Python5110.33 ± 3521.8326.71 ± 11.997.41 ± 6.43
DEVS-Augmented68.85 ± 111.6411.34 ± 3.584.84 ± 0.61
GPT-6.1-solDirect-Policy2561.92 ± 2300.278.56 ± 1.185.12 ± 1.58
Native-Python18.33 ± 25.348.03 ± 0.345.74 ± 2.21
DEVS-Augmented6.88 ± 2.897.74 ± 0.224.40 ± 2.80
  1. The repeated decisions still support the benefit of simulator feedback. DEVS-Augmented achieves the lowest mean cost in most of the settings. This shows that its benefit extends across independent policy searches.

  2. The multi-seed evaluations provide precise policy-cost estimates for these comparisons. Across the 130 evaluable policies, the estimated relative standard error of the 20-seed mean has a median of 0.62% and a maximum of 3.98%.

Note: In GPT-5.2 assembly-and-distribution setting, DEVS-Augumented does not have the lowest average cost. But in this setting, DEVS-Augmented produces the best observed policy in the GPT-5.2 assembly-and-distribution setting, at a cost of 3.76×1043.76 \times 10^4 versus 4.33×1044.33 \times 10^4 for Direct-Policy's best policy.

[Common Response 5] Does DEVS-Gen improve the behavioral conformance of operational simulators, or mainly increase the success rate of producing operational simulators?

[Reviewer LDMR Q3, Reviewer 32KP Q3]

To assess behavioral accuracy separately from operational success, we added comparisons of fully operational simulators on common settings. The results are provided below and in Appendix E.1 of the revised PDF.

The original conditional averages in Appendix E.1 Table 4 describe behavioral scores after successful execution, complementing the main results. However, each method contributes its own fully operational subset, so these averages cover different tasks. Directly ranking them therefore does not establish which method implements the dynamics more accurately on the same tasks.

We addressed this difference by selecting common settings across methods. A setting is a LLM–scenario combination: four LLMs and ten scenarios give 40 settings, each with three generations per method. We selected the 21 settings in which every method produces at least one fully operational simulator. Within each setting, we average each method’s fully operational generations.

The table reports mean behavioral scores; higher is better.

MethodDEVS-GenOpenHands-LiteSWE-Agent-LiteOpenHandsSWE-Agent
Score0.91180.84020.90370.85840.8626

DEVS-Gen’s gains also include behavioral improvements. It achieves the highest mean behavioral score on the 21 common settings, showing that its advantage extends beyond producing operational code.

To identify which aspects of the dynamics improve, we also compared all 59 behavioral rules represented in the 21 common settings. The behavioral gains vary across rules. The average scores of the methods on each rule are listed in Appendix E.1 of the revised PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.