- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Factorio Design Benchmark, a benchmark for evaluating LLM agents on hierarchical design synthesis. The benchmark contains 32 factory-design tasks requiring agents to generate complete declarative JSON designs under production, spatial, logistics, and power constraints. Evaluation combines deterministic static validation with execution in Factorio to measure feasibility, production throughput, and design cost. Experiments on four LLMs demonstrate substantial challenges in meeting target production rates. The authors further introduce a five-dimensional task-difficulty profile and analyze common design failures and cost efficiency.
Critical strengths
Firstly, from my sides, this paper is interesting. It doesnt just use agent/llm to design a single task/product/satifsfy some constraints.
Here are the strengths
- The paper addresses an practically motivated problem involving system-level design under coupled constraints. The problem is very important in transportation engineering/ OR.
- The benchmark provides an objective and executable evaluation framework combining static constraint checking with simulated production throughput.
- The experiments include repeated runs, task-difficulty analysis, failure diagnostics, and cost comparisons, offering useful insights into current agents' limitations.
- The task specifications, validation rules, and implementation details support reproducibility.
Critical weaknesses
-
Insufficient Evidence of Task Feasibility. The paper reports that 12 out of 32 tasks remain unsolved, but does not sufficiently establish that all tasks admit feasible solutions satisfying their production and spatial constraints. Without independently verified reference solutions or feasibility evidence, it is difficult to determine whether failures on the hardest tasks reflect agents' limitations or potentially infeasible task specifications. This directly affects the validity of the reported success rates and difficulty analysis.
-
Missing Planning and Optimization Baselines.The benchmark involves structured production planning, equipment allocation, spatial layout, and logistics optimization, yet the evaluation only considers LLM agents. Comparisons with heuristic, constraint-programming, or hybrid optimization approaches would provide an important reference for interpreting task difficulty and assessing the significance of agent performance. Without such baselines, it remains unclear how much of the observed difficulty is specific to LLM-based hierarchical reasoning rather than the underlying combinatorial optimization problem. I believe if you dont use some traditional baseline like hybrid optimization to verify your concern/claim, the benchmark is not so meaningful. The proposed benchmark involves combinatorial optimization problems closely related to known NP-hard problems, like equipment allocation, spatial packing, and logistics network design. Please show the shortcoming/strength with quantifiable results of LLM/Agent of this problem, the paper will be better.
Questions for authors
-
Have the authors independently verified the feasibility of all 32 tasks? In particular, can reference designs or feasibility evidence be provided for the 12 tasks that none of the evaluated agents solved?
-
How would the evaluated agents compare with conventional heuristic or constraint-based optimization methods under the same task specifications?
-
How does the benchmark distinguish hierarchical design reasoning from general spatial placement and constraint satisfaction? Is there evidence that the explicit subsystem structure is necessary for solving the tasks?
Overall, I find the proposed platform and problem formulation interesting and potentially valuable. However, without establishing the feasibility of all benchmark tasks or providing comparisons with conventional optimization methods, it is difficult to assess the broader significance of the benchmark. Otherwise, the study risks remaining a relatively narrow testbed for LLM performance rather than offering deeper insights into hierarchical design and optimization. I would encourage the authors to strengthen the experimental validation and include reproducible optimization baselines, so that the benchmark can provide meaningful insights to the operations research and engineering communities, rather than simply demonstrating how well LLMs perform on these tasks. I think it will take some time, but it is worthy.
Response to Reviewer oY6y
[Q1] Have all 32 tasks been independently verified as feasible, particularly the 12 tasks unsolved by the evaluated models?
[A1] All 32 tasks now have verified successful reference designs. Please see our General Response section "[CQ3] Feasibility of All 32 Tasks" and the task-by-task summary in Appendix B.2 of the revised manuscript.
[Q2] How do the evaluated agents compare with conventional heuristic, constraint-based, or hybrid optimization methods under the same task specifications? Such baselines would help distinguish limitations of LLM agents from the difficulty of the underlying combinatorial optimization problem.
[A2] Thank you for suggesting a planning and optimization baseline. We added an end-to-end non-LLM method that met the production-rate criterion on 28 of the 32 tasks (87.5% task coverage), providing a concrete planning and optimization reference for interpreting agent performance. It combines mixed-integer linear programming (MILP) with domain-specific construction heuristics.
The MILP selects production-equipment types and integer counts to minimize their cost subject to material-balance and capacity constraints derived from recipes and the target production rate.
Reusable production-unit templates, greedy placement, and logistics and power construction rules then generate a complete factory.
The solver makes no LLM calls and does not read human reference designs or prior model-generated layouts.
After development and trial runs on the released suite, we froze the shared rules and parameters. The final evaluation generated and executed one design per task under the original task specifications and success criterion. All 32 designs passed static validation, executed successfully, and produced the target item. In the original experiments, DeepSeek V4 Flash, V4 Pro, GPT-5 mini, and Gemini 3 Flash Preview solved 19, 17, 12, and 12 tasks, respectively, counting a task as solved if any of its three independent runs succeeded. These task-coverage figures use the respective attempt counts stated above and do not constitute a controlled comparison. Code, designs, and evaluation records are available in the anonymous baseline package.
The method encodes machine-inserter-chest interfaces, construction order, and infrastructure rules, but still computes equipment quantities and placements for each task. It also uses explicit environment parameters, including the mining-productivity bonus, so its prior knowledge and information conditions differ from the original model experiments. MILP optimality applies only to the capacity subproblem, not the complete factory.
The comparison does not isolate hierarchical reasoning as the source of performance differences or separate the contributions of domain priors and individual algorithmic components.
[Q3] How does the benchmark distinguish hierarchical design reasoning from general spatial placement and constraint satisfaction? Is the explicit subsystem structure necessary for solving the tasks?
[A3] The production-rate requirement is one of our task constraints, so satisfying all task constraints constitutes success. Meeting them requires coordinating production dependencies, capacities, material interfaces, and shared infrastructure across the complete factory. A particular subsystem decomposition is not required or separately rewarded. Please see the General Response section "[CQ1] How Does the Benchmark Evaluate Hierarchical Design Synthesis, and What Is the Role of Explicit Subsystems?" for the distinction between placement validity and complete-task success, the interpretation of constraint satisfaction, and the role of explicit subsystem structure.