Factorio Design Benchmark: Evaluating LLM Agents on Hierarchical Design Synthesis

Designing a circuit to meet functional requirements or a factory to meet production targets exemplifies hierarchical design synthesis. These tasks require selecting component types and quantities, configuring and placing instances, organizing them into subsystems, and coordinating interfaces and shared infrastructure under coupled feasibility constraints and system-level objectives. Existing benchmarks evaluate important aspects of engineering design, but provide limited evidence on how well large language model (LLM) agents jointly coordinate these decisions across a complete system. We introduce Factorio Design Benchmark, which is based on Factorio, a deterministic factory-construction game. To complete the task, agents are required to submit complete declarative JSON factory designs for static validation, game execution, and production monitoring to decide whether it reaches the target production rate. We evaluate four LLMs on 32 tasks (8 target products $\times$ 2 product-specific target production rates $\times$ 2 map settings), with three independent runs per model–task pair. Across all runs, 73.7% produce the target product, but only 40.9% reach the target rate. Beyond these aggregate results, we examine how success varies with task demands. We derive a five-dimensional difficulty profile from task specifications, covering recipe complexity, capacity, logistics, spatial layout, and power infrastructure. The average of its five scores correlates negatively with the proportion of successful runs per task for each model, with Spearman $\rho =-0.90$ when runs are pooled across all four models. This association supports the profile as a proxy for relative task difficulty within the suite. To identify recurring design failures, we also analyze validation feedback. Entity overlap and unpowered consumers have the highest reported violation counts, with larger counts at high target production rates. These failures highlight challenges in coordinating spatial layouts and shared infrastructure across the complete factory.