FlowCompile: An Optimizing Compiler for Structured LLM Workflows

Structured LLM workflows, in which specialized LLM sub-agents are executed according to a predefined execution graph, have become a powerful abstraction for solving complex tasks. Optimizing such workflows, i.e., selecting configurations for each sub-agent to balance accuracy and latency, is fundamentally challenging due to the combinatorial design space over model choices, reasoning budgets, and workflow structures. Routing-based cost-aware methods select a configuration at inference time for each query according to an accuracy–latency objective. We argue that, beyond runtime routing, structured LLM workflows can also be optimized from a compilation perspective: before deployment, the system can globally explore the workflow design space and construct a reusable set of workflow-level configurations spanning diverse accuracy–latency trade-offs. Drawing inspiration from machine learning compilers, we introduce FlowCompile, a structured LLM workflow compiler that performs compile-time design space exploration to identify such a high-quality, reusable trade-off set. FlowCompile decomposes a workflow into sub-agents, profiles each sub-agent under diverse configurations, and composes these measurements through a structure-aware proxy to estimate workflow-level accuracy and latency. It then identifies a diverse set of high-quality configurations in a single compile-time pass, without retraining or online adaptation. Experiments across diverse workflows and challenging benchmarks show that FlowCompile consistently outperforms heuristically optimized workflow configurations and routing-based baselines by a large margin, delivering up to 6.4× speedup while maintaining strong task performance. Furthermore, the compiled configuration set serves as a reusable optimization artifact, enabling flexible deployment under varying runtime preferences and naturally supporting downstream selection or routing for additional gains.