acceptodds
Withdrawn from ICLR 2027

FlowCompile: An Optimizing Compiler for Structured LLM Workflows

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Structured LLM workflows, in which specialized LLM sub-agents are executed according to a predefined execution graph, have become a powerful abstraction for solving complex tasks. Optimizing such workflows, i.e., selecting configurations for each sub-agent to balance accuracy and latency, is fundamentally challenging due to the combinatorial design space over model choices, reasoning budgets, and workflow structures. Routing-based cost-aware methods select a configuration at inference time for each query according to an accuracy–latency objective. We argue that, beyond runtime routing, structured LLM workflows can also be optimized from a compilation perspective: before deployment, the system can globally explore the workflow design space and construct a reusable set of workflow-level configurations spanning diverse accuracy–latency trade-offs. Drawing inspiration from machine learning compilers, we introduce FlowCompile, a structured LLM workflow compiler that performs compile-time design space exploration to identify such a high-quality, reusable trade-off set. FlowCompile decomposes a workflow into sub-agents, profiles each sub-agent under diverse configurations, and composes these measurements through a structure-aware proxy to estimate workflow-level accuracy and latency. It then identifies a diverse set of high-quality configurations in a single compile-time pass, without retraining or online adaptation. Experiments across diverse workflows and challenging benchmarks show that FlowCompile consistently outperforms heuristically optimized workflow configurations and routing-based baselines by a large margin, delivering up to 6.4× speedup while maintaining strong task performance. Furthermore, the compiled configuration set serves as a reusable optimization artifact, enabling flexible deployment under varying runtime preferences and naturally supporting downstream selection or routing for additional gains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM workflow

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kN8gOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

FlowCompile optimizes structured LLM workflows by jointly selecting models, reasoning budgets, and permitted structural choices. It profiles sub-agents using reference-model traces, analytically composes their measurements into workflow-level accuracy and latency estimates, and constructs a reusable trade-off set. Experiments on four benchmarks evaluate deployment performance, proxy quality, and compatibility with query-level routing.

Critical strengths
  1. Separating reusable sub-agent profiling from workflow-level search is practically appealing and allows the resulting configuration set to accommodate different deployment preferences.

  2. The reported accuracy–latency improvements across four benchmarks are encouraging. Restricted-space frontier comparisons and reference-model ablations provide additional empirical support.

Critical weaknesses
  1. The methodological novelty is insufficient. LLMSelector already uses module-level evaluations to optimize model assignments in compound AI systems. FlowCompile extends this direction to reasoning budgets, permitted structural choices, and reusable accuracy–latency trade-off sets. Its optimization procedure combines analytical composition rules, local Pareto pruning, configuration enumeration, and non-dominated sorting. While the joint formulation is useful, I find the additional algorithmic or conceptual insight beyond this integration too limited to constitute a sufficiently substantial contribution. This is my primary concern, rather than a claim that the method is technically invalid.

  2. The experiments do not sufficiently isolate the value of the proposed optimization approach. Several baselines optimize fewer configuration dimensions, and Table 4 shows substantial gains from expanding the search space itself. Appendix D provides useful restricted-space comparisons, but does not fully establish the advantage over alternative optimization approaches operating in the same space under comparable offline evaluation budgets. This makes it difficult to determine whether the reported gains demonstrate a substantial advance beyond the additional configuration flexibility.

Questions for authors
  1. What is the principal new technical insight beyond existing module-level model selection, expanded configuration choices, and Pareto-based trade-off construction?

  2. Which baselines share the full configuration space, and how do their offline optimization budgets compare? Is there additional matched-space, budget-controlled evidence that isolates the benefit of the proposed approach?

Reviewer_pv1DOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Structured LLM workflows call several specialized models through a fixed execution graph, with a different model and reasoning budget per call. The paper optimizes such workflows before deployment. A reference model runs the workflow on a small validation set, and successful sub-agent calls become per role training sets. Each role is profiled over a grid of model sizes and reasoning budgets. Simple composition rules estimate workflow accuracy and latency for any assignment, and non dominated sorting returns configurations that span the trade-off. On four reasoning benchmarks the compiled set is compared with workflow and routing baselines at a small one time profiling cost.

Critical strengths
  1. The system outputs a set of operating points, and the evaluated protocol selects by predicted latency rank alone. Accuracy estimates decide which configurations enter the set, not what runs. Latency is what the compositional proxy estimates most reliably, so the division of labor is sound. The latency constrained mode does use predicted accuracy, so this holds for the evaluated rule only.

  2. The cost and error accounting can be checked. Profiling scales with roles and candidate settings, dominated choices are pruned with an exact preservation argument, and the selection error analysis says it bounds a deterministic comparison, not generalization.

Critical weaknesses
  1. Accuracy is agreement with pseudo ground truth, and no reliability is reported. The targets come from a GPT-5 reference run, an LLM filter keeps the calls, and an LLM judge scores candidates where matching is impossible. Neither the filter nor the judge carries a reliability figure. The reference model ablation argues against a single model accident, but it measures neither stage. If they favor the reference style, the frontier optimizes the wrong objective.

  2. The accuracy proxy assumes independent parallel branches. The generators receive the same input, and the paper never reports how often they fail together. End to end validation covers only restricted spaces for HotpotQA and LiveCodeBench. GSM8K and MATH-500 are checked through about twenty sampled configurations, and MATH-500 agreement is the weakest of the four.

  3. The baselines omit methods that related work names as representative. DAAO, MasRouter, and LLMSELECTOR are cited but not evaluated. The routing set is MaAS, a KNN router, and two in house variants.

Questions for authors
  1. What is the human audited agreement of the filtering judge and of the profiling judge on a sample of retained calls, and how often does the filtering step keep calls from traces whose final answer was wrong.

  2. On HotpotQA, how often are both generators correct at the same time, and how does that compare with the product the proxy uses. How many compiled configurations change if the composition rule is corrected.

  3. Can you run the same validation protocol with twenty to fifty evaluated configurations on GSM8K and MATH-500 rather than only the restricted spaces.

  4. Can you release the measured accuracy and latency of every compiled configuration. The low latency LiveCodeBench selection loses nearly forty points of Pass@1 against the full workflow (80.97 to 41.34 in Table 5), which suggests the frontier is thin at that end.

Reviewer_ori7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FlowCompile treats the configuration of a structured LLM workflow as a compile-time optimization problem: which model each role uses, how much reasoning budget each call gets, and whether optional stages run. It profiles each sub-agent on its own, combines the profiles with a structure-aware proxy for accuracy and latency, and extracts a set of Pareto configurations that can be reused at deployment. Experiments on math, QA, and code benchmarks evaluate the proxy and compare the compiled sets against single-model, fixed-workflow, and routing baselines. The method is practical and the frontier results are convincing, but the preference-aware evaluation and the baseline set are weaker than the claims built on them.

Critical strengths
  1. The compiler framing fits the problem well. Sections 3.1-3.4 break the method into clear steps: inducing data from reference traces, profiling each role, composing the profiles according to the workflow graph, and extracting a Pareto set. Algorithm 1 is specific enough to implement. The main payoff is cost, since profiling scales with a few hundred role/model/budget settings rather than with a design space that can reach millions or billions of configurations.

  2. The proxy is shown to be useful for selection, not only accurate in isolation. In restricted spaces where every configuration can be run end to end, the proxy-selected points lie on the measured frontier (Figure 2, Figure 6, Appendix D). Table 1 adds high rank agreement within the compiled sets, with average Spearman correlations of 0.96 for latency and 0.92 for accuracy.

  3. The held-out results hold across quite different workloads. Table 2, Figure 3, and Table 5 cover four workflow types on math, QA, and code, which suggests the method is not tied to one particular graph.

Critical weaknesses
  1. The preference-aware results measure a latency-rank heuristic rather than utility maximization. In Appendix J.1, a preference α\alpha is mapped to the configuration at position round(α(N−1))\mathrm{round}(\alpha(N-1)) in predicted-latency order, not to arg⁡max⁡c∈F^U^α(c)\arg\max_{c \in \widehat{\mathcal F}} \widehat U_\alpha(c). Section 3.4 is open about the fact that this rule can leave a nonzero shortfall in Equation 2, and the two preference-aware baselines use the same rule, so the comparison itself is even. Still, the rank rule ignores predicted accuracy and the spacing between latencies, so Table 2 and Figure 4 mostly reflect how each candidate set is spread along the latency axis. They show that FlowCompile works well with this heuristic. They do not yet show that the compiled set supports good deployment for a given accuracy-latency preference, which is the use case the paper motivates.

  2. The routing comparison leaves out the methods the paper itself identifies as closest. Section 2 names MasRouter, DAAO, and LLMSelector as representative methods and describes LLMSelector as closely related because it also relies on module-level assessments. The experiments in Section 4.1, however, compare only against a KNN router, MaAS, and two in-house preference-aware variants. Since the paper claims consistent gains over routing-based approaches, at least one of these methods should be included under matched models, budgets, and latency measurement, or the paper should explain why they cannot be run in this setting.

  3. The single-model baselines do not test the premise that a compiled multi-agent workflow is the better choice. Section 4.1 includes only Qwen3-32B and QwQ-32B used directly as single models. Recent work argues that a single LLM run over multiple turns can match homogeneous multi-agent workflows at lower cost (e.g., Xu et al., 2026, "Rethinking the Value of Multi-Agent Workflow: A Strong Single Agent Baseline"). Because FlowCompile assigns different models to different roles, it is well placed to show when this heterogeneity pays off, but the current baselines cannot answer that question.

Questions for authors
  1. Do the main conclusions of Table 2 and Figure 4 still hold if every method selects arg⁡max⁡cU^α(c)\arg\max_{c} \widehat U_\alpha(c) over its own candidates instead of using the latency rank? The answer would show whether the advantage comes from the compiled set itself or from how it interacts with the rank rule.

  2. For Pref-Aware Router and Pref-Aware MaAS, how large are the candidate menus, and do they cover the same model and budget ranges as FlowCompile's compiled set? Without this, it is hard to tell whether the utility gap reflects compilation or differences in the candidate sets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.