- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents PLCWorld, a benchmark focuses on evaluating PLC programs generated by LLMs via closed-loop plant simulation. The benchmark contains two type of tasks: Motion Control and Material Handling. Within each task, the benchmark also contains easy, medium, and hard samples. This paper also reports evaluation results on six LLMs, and several generation-and-verfication frameworks using GPT-5.5.
Critical strengths
- The tasks are using closed-loop simulations to provide verifiable evaluations.
- Test both direct ST generation and generation-and-verfication frameworks are useful.
Critical weaknesses
- The scope of the benchmark is a bit limited, the benchmark only contains two use-cases in PLC, while neglecting many other PLC use cases. How diverse of tasks within each domain is also not analyzed.
- The source of tasks are also not well-documented in Section 4.1.
- The simulation-based evaluation is interesting, but also is not novel.
- The evaluation results need more clarifications. For example, some Generation-and-verification frameworks performs worse than direct ST generation. What are the failure modes there?
Questions for authors
Please see the section above.