acceptodds
Withdrawn from ICLR 2027

PLCWorld: Benchmarking LLM-Generated PLC Programs in Closed-Loop Plant Simulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Programmable logic controllers (PLCs) coordinate industrial equipment by reading sensor inputs and issuing control commands. Evaluating whether large language model (LLM)-generated PLC programs satisfy task requirements and safety constraints requires observing how their commands affect device and workpiece states. We introduce PLCWorld, a common closed-loop execution environment and benchmark that couples Structured Text (ST) execution with simulated plant responses and sensor feedback. Grounded in control relations identified in industrial PLC programs and engineering documentation, PLCWorld contains 100 synthetic tasks and 473 registered task–condition pairs across Motion Control and Material Handling, with difficulty defined by control-dependency scope. A common protocol reports Task Success and Safety Violation separately. Validation combines practitioner review, reference and alternative programs, targeted counterexamples, specification–evaluator alignment checks, and comparisons with independent ST runtimes. Reference and alternative programs satisfy their applicable cases, while all 542 targeted counterexamples activate their designated evaluator rules under at least one registered condition. Execution Gap relates submission-profile acceptance to subsequent task failure or observed Safety Violation. Across the constructed task groups, direct GPT-5.5 achieves 82.70% Task Success on Easy cases but 25.10% on Hard cases. Evaluations of six LLMs and four adapted generation-and-verification workflows further expose differences between completion, safety, and generation cost.

Primary area:
datasets and benchmarks
Keywords:
Large language models, Code generation, Programmable logic controllers, Structured Text, Closed-loop simulation, Execution-based evaluation, Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_345kOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents PLCWorld, a benchmark focuses on evaluating PLC programs generated by LLMs via closed-loop plant simulation. The benchmark contains two type of tasks: Motion Control and Material Handling. Within each task, the benchmark also contains easy, medium, and hard samples. This paper also reports evaluation results on six LLMs, and several generation-and-verfication frameworks using GPT-5.5.

Critical strengths
  • The tasks are using closed-loop simulations to provide verifiable evaluations.
  • Test both direct ST generation and generation-and-verfication frameworks are useful.
Critical weaknesses
  • The scope of the benchmark is a bit limited, the benchmark only contains two use-cases in PLC, while neglecting many other PLC use cases. How diverse of tasks within each domain is also not analyzed.
  • The source of tasks are also not well-documented in Section 4.1.
  • The simulation-based evaluation is interesting, but also is not novel.
  • The evaluation results need more clarifications. For example, some Generation-and-verification frameworks performs worse than direct ST generation. What are the failure modes there?
Questions for authors

Please see the section above.

Reviewer_FrS1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PLCWorld, a closed-loop plant simulation environment and benchmark comprising 100 industrially grounded tasks (473 task-condition pairs) to evaluate LLM-generated Structured Text (ST) PLC programs. The authors establish an evaluation protocol decoupling Task Success from Safety Violation to quantify the "Execution Gap," validated via practitioner review, 542 targeted counterexamples, and cross-runtime checks across six LLMs and four verification frameworks.

Critical strengths
  1. Coupling PLC scan execution with dynamic plant feedback effectively exposes how open-loop checks fail to guarantee physical task completion or safety.

  2. Rigorous multi-stage benchmark validation.

  3. Well writing.

Critical weaknesses
  1. Over 60% of tasks in LLM4PLC and Agents4PLC terminated in timeouts (Table 27) and all 473 cases are inexplicably marked as "Interrupted" in Table 20, making it unclear whether their poor performance stems from algorithmic limitations or brittle tool-adaptation bottlenecks.

  2. Several verification frameworks significantly underperform the single-turn direct GPT-5.5 baseline without sufficient root-cause ablation, and the discrete-event plant abstraction omits continuous physical dynamics (e.g., contact forces, friction), limiting evaluation to sequential logic.

Questions for authors
  1. Table 20 shows that all 473 cases for LLM4PLC and Agents4PLC were "Interrupted" (Observation complete = 0), yet Table 22 reports 58.77% and 52.64% Task Success Rates despite Appendix E.3 stating that program-attributable interruptions yield Ue=0U_e = 0; could you clarify this apparent contradiction?

  2. Why do frameworks like Agents4PLC and SemaPLC achieve notably lower Task Success than the matched single-turn GPT-5.5 baseline, and is this degradation driven by information loss during planning or destructive edits during repair loops?

  3. Have you considered feeding closed-loop plant simulation traces from development conditions back into the agent repair loop to test whether closed-loop feedback resolves the Execution Gap?

Reviewer_zBo1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PLCWorld is a closed-loop benchmark for LLM-generated Structured Text (ST) PLC programs. Each program runs scan by scan against a simulated plant whose sensor feedback comes from the evolving device and workpiece states. The benchmark provides 100 synthetic tasks in two suites. Evaluations of six LLMs and four generation-and-verification frameworks demonstrate distinct completion and safety outcomes across task groups

Critical strengths
  1. It systematically develops an execution environment and benchmark for evaluating LLM-generated ST programs on industrial control tasks.
  2. The established tasks are challenging even for the latest LLMs like GPT-5.5.
Critical weaknesses
  1. The contribution is insufficient: 1) No new models/methods are proposed, and the contribution is mainly the benchmark itself. 2) Compared to previous PLC benchmarks like LLM4PLC, which already use compilation, model checking, and runtime tests, the modification of PLCWorld is relatively weak.
Questions for authors
  1. Completion timing accounts for 73% of adverse cases. How much slack does each deadline leave over the reference program's completion time, and how does SR change if deadlines are relaxed?

  2. In Table 7, Average I/O count rises from 8.9 to 128.4 between Motion Easy and Motion Hard. Does dependency scope predict failure beyond I/O count or specification length?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.