- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces DEVS-Gen, a pipeline that uses large language models (LLMs) and the DEVS formalism to automatically generate discrete-event simulators from natural language specifications, targeting complex domains like supply chains and business processes. The approach structures and synthesizes simulator components to ensure consistency and compliance with requirements. Evaluation on a new benchmark shows DEVS-Gen outperforms baselines in both behavioral accuracy and efficiency.
Critical strengths
I believe this paper is novel within its field. Strengths: The writing is well-organized, the theoretical analysis is sound, and the experimental results support the authors' claims.
Critical weaknesses
a) Since this paper focuses on discrete event systems, its application scope may be somewhat limited. b) The evolution of discrete event systems is generally straightforward to enumerate and quantify; therefore, the necessity of building a world model for simulation in such cases may not be particularly strong. c) Reasoning about the environment in discrete event systems and training agents in such environments seem to be essentially similar. Since the evolution rules can be derived during training, constructing a world model in this context is largely equivalent to generating verifiable signals. There may not be a fundamental difference between these approaches.
Questions for authors
What is the connection between the baselines (including OpenHands-Lite, SWE-Agent-Lite, and OpenHands) and the methods designed for the current discrete event system scenarios? Are there potentially more effective approaches for these tasks?
3 comments
Response to Reviewer Xpid(2/2)
[Q4] What is the connection between the baselines (including OpenHands-Lite, SWE-Agent-Lite, and OpenHands) and the methods designed for the current discrete event system scenarios? Are there potentially more effective approaches for these tasks?
[A4] The coding-agent baselines and DEVS-Gen address the same construction problem: turning a scenario’s natural-language specifications into an executable DES simulator. OpenHands, SWE-Agent and their Lite variants use general-purpose code-development workflows to implement the required dynamics. DEVS-Gen specializes this process through explicit component planning, component-wise implementation and interface integration.
Other specialized modeling approaches provide related representations. As discussed in Appendix A.2, methods that generate PDDL planning domains or SysML models translate natural language into action preconditions, effects and goals, or system structure and behavioral design. Using them for our task would additionally require the event-execution mechanisms and trace interfaces needed to run and evaluate DES simulators.
We also investigated combining DEVS-Gen's structural planning with a general-purpose coding agent. Our additional OH + PlanTree experiment, reported in Common Response 2, provides OpenHands with the complete PlanTree while allowing code execution and debugging. The results show that a strong coding agent can use PlanTree to produce high-quality simulators. Compared with OH + PlanTree, DEVS-Gen maintains comparable quality with stronger backbones, performs better with smaller backbones, and consumes fewer tokens. These findings highlight the value of structural planning and the role of the implementation workflow in generation quality and efficiency.
Thank you again for the discussion. To address your questions about the value of executable world models, we added matched comparisons between direct trajectory prediction and simulator execution across five backbones. These results and the revised discussion explain the benefits of programmatic execution and simulator reuse. Please let us know whether any questions remain.
Response to Reviewer Xpid(1/2)
Thank you for encouraging us to clarify the motivation, scope and role of executable discrete-event world models. To further examine the value of executable simulators, we additionally compared direct trajectory prediction with generated simulators across five backbones, examining both behavioral accuracy and the cost of repeated prediction ([A2] and [Common Response 3]).
[Q1] Since this paper focuses on discrete event systems, its application scope may be somewhat limited.
[A1] Discrete-event modeling covers a broad range of important applications, including communication networks, manufacturing, healthcare services and supply chains. Common Response 1 discusses its importance and broad applicability, including established approaches for incorporating continuous dynamics. The key is that event timing, resource constraints and interactions among entities provide an effective abstraction for many decision problems. Our work makes these environments easier to construct from natural-language specifications and reuse for prediction and policy optimization.
When continuous processes also affect task outcomes, the generated model can also coordinate tasks and resources while interacting with continuous-dynamics modules. DEVS-Gen can therefore also support the task-coordination layer of hybrid environments.
[Q2] The evolution of discrete event systems is generally straightforward to enumerate and quantify; therefore, the necessity of building a world model for simulation in such cases may not be particularly strong.
[A2] The value of an executable world model lies in reliably executing interacting rules and reusing it to compute. Even when individual rules are simple, predicting a complete trajectory using language models can be hard.
Our additional experiments, reported in Common Response 3 and Appendix E.7 of the revised PDF, compare direct trace prediction with generated simulators across five backbones: gpt-6.1-sol and the four used in our paper. We evaluated both approaches on the same cases. The experiments show two benefits of executable simulators:
Stronger models are unreliable on the direct rollouts. Simulators achieve higher mean scores than direct prediction. Explicit rules therefore still leave a substantial challenge in accurately predicting their combined effects.
Direct trace prediction incurs a new output-generation cost for every rollout.
Direct trajectory prediction requires tracking component states, event times, and interactions across successive events. Errors can accumulate as each prediction depends on the states produced by earlier events. Related temporal-reasoning research also reports declining performance as the number of entities and facts increases [1].
DEVS-Gen addresses this challenge by generating executable simulators that maintain states and schedule events according to the specified rules. Once constructed, a simulator can run under different initial conditions, parameters, and policies. This makes executable world models particularly valuable for policy optimization, where methods such as grid search and TPE require repeated environment evaluations.
Reference: [1] Fatemi B, Kazemi S M, Tsitsulin A, et al. Test of time: A benchmark for evaluating llms on temporal reasoning[C]//International Conference on Learning Representations. 2025, 2025: 94426-94447.
[Q3] Reasoning about the environment in discrete event systems and training agents in such environments seem to be essentially similar. Since the evolution rules can be derived during training, constructing a world model in this context is largely equivalent to generating verifiable signals. There may not be a fundamental difference between these approaches.
[A3] Providing verifiable feedback for agent training and policy optimization is precisely a motivation for our work to construct simulators. In many settings, environment rules and requirements remain ambiguous or incomplete textual descriptions. Turning them into executable state transitions and feedback signals requires modeling and implementation, often involving substantial manual effort. DEVS-Gen addresses this need by translating natural-language specifications into executable programs through structural planning and behavioral synthesis, without training a separate predictive model for each target environment.
Environment generation and agent training or optimization therefore form complementary stages: DEVS-Gen constructs the environment, and subsequent training or optimization uses its feedback to improve policies. Within the executable environment, an agent can try different policies, observe event traces, state changes, and costs, and use these outcomes to improve its decisions. The paper’s existing downstream experiments (Section 5.5) illustrate this use by comparing direct policy generation with decision-making supported by generated simulators.