acceptodds
Under review as a conference paper at ICLR 2027

FlowMaker: From Tasks to Cost-Efficient Generative AI Workflow Deployments

Abstract

Generative AI workflows execute user tasks by composing sequences of LLM calls, where each call performs a sub-task (e.g., summarization or translation), forming a workflow. Executing such workflows requires decisions over workflow structure, model assignments, and deployment configurations, which jointly determine latency, cost, and accuracy under dynamic workloads and resource constraints. Existing systems optimize these decisions in isolation or fix them early, leading to suboptimal cost and goodput. We make two key observations. First, efficient serving requires jointly exploring all combinations of workflows, model choices, and deployment configurations under dynamic cluster state. Second, reuse of deployed LLMs is critical for reducing cost, but must be performed on-the-fly, as reuse depends on current load and resource availability. Together, these make selecting a workflow-variant—a combination of workflow, models, and deployment configuration—a time-critical problem. We present FlowMaker, which exposes a task-level interface and selects workflow-variants on behalf of users. To realize this, we design a constraint-guided load-aware pruning algorithm for fast, on-the-fly decisions. FlowMaker improves goodput by up to 10.14× and reduces cost by up to 2.77× compared to state-of-the-art baselines, including Murakkab.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Inference serving, generative AI workflows, search algorithm

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_s43DOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

FlowMaker is a serving system for LLM workflows. Users provide a task and its latency and accuracy SLOs. They also confirm that the generated candidate workflows are semantically acceptable. At deployment time, the system jointly selects the workflow, models for each stage, and parallelism configurations. It also decides whether to reuse existing instances based on their current load. The selector uses three pruning stages: accuracy, incremental deployment footprint, and system constraints. It is a heuristic and does not guarantee global optimality. Experiments use eight A6000 GPUs and a reconstructed Murakkab baseline. At high load, the mean FlowMaker-to-Murakkab goodput ratio is 4.73. The mean Murakkab-to-FlowMaker cost ratio is 1.95. Selection takes about one second on the seven evaluated tasks. The selected variants match brute force at all comparable points, including the scaling experiments. Selection time reaches 210 seconds at seven stages.

Critical strengths
  1. The problem is clearly formulated and supported by experiments. The system jointly considers workflows, models, deployment configurations, and reuse under the current load. The motivation experiments show the benefits of alternative workflows and load-aware reuse.

  2. The implementation and evaluation are reasonably complete. The paper includes an ablation of reuse across tasks and separate evaluations of selector speed and selection quality. The experiments also support joint selection, although differences in reuse policy are not fully controlled.

Critical weaknesses
  1. Output quality is not directly evaluated. Accuracy uses average leaderboard scores and their minimum across stages. This leaves quality preservation in shorter workflows unverified.
  2. Murakkab receives one default workflow, while FlowMaker has three candidates. Its offline reuse policy and periodic reconfiguration settings are unclear. These gaps limit attribution of the gains.
Questions for authors
  1. Can output quality be evaluated on one or two tasks?
  2. How much of FlowMaker's advantage remains if Murakkab uses the best approved workflow? Does its reuse policy use OfflineReuseAvg, OfflineReuseMax, or another strategy?
  3. Which setting and baseline underlie the reuse gains in Appendix H.4 (1.13× goodput and 1.42× cost reduction)?
3 comments
AuthorsOpenReview ↗

Answers to Reviewer s43D's Weaknesses & Questions

R3-W1/Q1: Measured output quality

We agree with the reviewer that the proxy-accuracy metric does not evaluate true output quality. Nevertheless, we note that we have used the same proxy-accuracy metric across all baselines to enable fair comparison. On top of that, we post-process all the generated outputs across the end-to-end experiments (Fig. 5), through an LLM evaluator (gpt-oss-120b), which scored each of those from 1-100. Across all the experiments, we report the median scores for each baseline: FlowMaker, FlowMakerNoReuse & RouteLLM-maxquality → 55/100, Murakkab → 48/100, LangChain, RouteLLM-balanced & RouteLLM-mincost → 35/100 (see R2-W1), which indicate that FlowMaker achieves similar or better quality compared to the rest of the systems.

In addition, we analyse the per-task scores pooled over all traces (see R1-W1) and find that across 5/7 tasks (T1, T3, T4, T5, T6), FlowMaker achieves the top-ranking scores, except T2 and T7, where their BASIC-SLO configuration (see App. G; Proxy-accuracy SLOs) does not require high-quality models.

Finally, we evaluate the effect of merging operations on output quality (see R1-W1). We analyse 3 tasks (merged and unmerged operations) using 3 fixed models and measure their quality through an LLM evaluator. When merging, T2 preserves quality with the 14b and 32Bb models but hurts the 7b model; T6 maintains its accuracy across workflows; T7 improves drastically when merged, since the LLM sees the full input at once instead of passing it through multiple steps.

In conclusion, we show that FlowMaker meets every task’s accuracy requirement, and merging or shortening workflows does not degrade the E2E task quality.

Please see our responses in R1-W1 & R2-W1 for the detailed results.

R3-W2/Q2: FlowMaker vs Murakkab comparison

We note that in our experiments, we have disabled Murakkab’s periodic reconfiguration to enable fair comparison. The reconfiguration part is orthogonal to our work, as we focus on the workflow variant selection and deployment phase of the serving pipeline. Nevertheless, FlowMaker’s modular design allows integration with autoscaling/reconfiguring approaches with minimal source code modifications.

We specify that, as the size of our cluster did not allow for provisioning at the maximum load, we use the OfflineReuseAvg as Murakkab’s default policy, similar to the average load that FlowMaker uses to make decisions.

Finally, we provide detailed results regarding Murakkab’s performance when using FlowMaker’s workflow choice. We run the code trace under medium (50%) and high (100%) loads and measure each system’s performance. Each cell in the table shows the result for 50%-load followed by 100%-load, separated by ‘/’.

MetricFlowMakerMurakkab (default workflows)Murakkab (FlowMaker's workflows)
Goodput (RPM)11.77 / 12.418.09 / 1.2211.61 / 14.91
Cost ($)0.540 / 0.9031.016 / 2.5040.654 / 0.985
p95 latency (s)89 / 413581 / 1577101 / 329

We notice that even under FlowMaker’s workflow, Murakkab increases operational costs across all rate scales, while FlowMaker’s goodput is similar (medium load) or slightly worse (high load). Thus, the advantages in cost efficiency come not only from full search-space exploration but from the on-the-fly reuse as well.

More results regarding a comparison between the baselines with reuse disabled are located in R2-W2, where we show that full search space exploration has a drastic effect on end-to-end performance. Also, a detailed comparison of FlowMaker with Murakkab’s online reconfiguration scheme is provided in R2-Q2, where we show that even under this setting, FlowMaker outperforms Murakkab.

R3-Q3: Appendix H.4 clarification

The 1.13x goodput and 1.42x cost reduction in Appendix H.4 come from the comparison between FlowMaker vs FlowMaker-No-Reuse (i.e., the gains achieved by on-the-fly reuse) on the code trace at high loads (i.e., 100% and 75%); we will make this clarification in the text.

Thanks for the detailed response. The new experiments address most of my concerns. I suggest including the comparison with the same workflows in the paper. FlowMaker costs less at both tested loads, but Murakkab has higher goodput at full load. I have no further questions, and my original score stands.

AuthorsOpenReview ↗

Author Response to Official Reviewer s43D's Comment

Thank you for the prompt follow-up: we appreciate your constructive feedback and are glad the new experiments addressed most of your concerns. We will include the comparison using the same workflows in the revised paper and clarify the cost–goodput tradeoff. We hope the additional evidence gives you confidence to consider stronger support for the paper.

Reviewer_FmAwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents FlowMaker, a serving system that selects and deploys generative AI workflows from task descriptions and user-specified latency and accuracy SLOs. The system jointly considers user-approved workflows, model assignments, parallelism configurations, and reuse of existing model instances. A three-stage heuristic filters candidates using benchmark-derived accuracy scores, incremental deployment size, and system constraints. Experiments on seven tasks with Azure-derived arrival traces report substantial goodput and serving-cost improvements over the evaluated baseline configurations, alongside lower selection overhead than exhaustive and Bayesian search.

Critical strengths
  • The paper identifies a useful interaction between workflow structure and deployment reuse. A longer workflow can be cheaper when it uses existing model instances, while sharing those instances can introduce contention. The example in Section 5.2, where FlowMaker selects a three-stage workflow reusing Qwen2.5-3B while FlowMakerNoReuse selects a two-stage alternative, makes this tradeoff concrete. Coordinating these decisions is a worthwhile systems problem.

  • The implementation and evaluation provide practical evidence for the proposed design. All evaluated configurations use the same TensorRT-LLM backend on actual GPU hardware, reducing differences due to the inference runtime. The comparison with FlowMakerNoReuse provides useful evidence for enabling cross-task reuse within the proposed selector: serving cost decreases by 1.04–1.42× at higher loads while goodput remains similar. The selector also matches the exhaustive-search selections at the tested comparable points with substantially lower decision overhead.

Critical weaknesses
  • The evaluation does not establish that the efficiency improvements preserve task quality. Section 4 and Appendix G explicitly state that generated outputs are not evaluated. Instead, Eq. 2 assigns each workflow the minimum of its models’ average scores across six general benchmarks. For the same model, merging several operations into one call leaves this score unchanged, regardless of whether the combined operation is executed correctly. Yet even the number of LLM calls alone changes output quality non-monotonically [R1], which is why systems that search jointly over models and inference-time structure select configurations by measured task performance [R2]; LLMSelector, which the paper cites to justify the proxy, likewise estimates per-module performance from task data with known answers rather than from general leaderboard averages [R3]. This is particularly consequential because Section 5.2 attributes substantial latency improvements to selecting shorter workflows, while every fixed-workflow baseline is given a workflow whose stages each perform a single operation (Appendix G). The restriction of smaller models to single-operation stages in Appendix G also acknowledges that compound-operation capability matters, but does not validate the quality of the remaining choices. User approval of a decomposition establishes acceptable intended semantics, not the quality of its generated outputs. The proxy also gates goodput (Eq. 4), so RouteLLM’s 34.9–36.8% accuracy compliance in its minimum-cost setting reflects the proxy thresholds rather than its outputs. Consequently, the reported goodput improvements do not demonstrate more efficient execution at comparable task quality.

  • The claimed benefit of joint selection is not sufficiently isolated from differences in the baseline configuration. The 4.34× average goodput improvement attributed to “joint search alone” compares FlowMakerNoReuse with Murakkab, changing both workflow choice and reuse policy. Section 5.2 acknowledges this confounding but characterizes the comparison as conservative because Murakkab retains reuse. That argument does not hold for goodput: Appendix H.1 specifically reports that Murakkab reuses an instance already at capacity, causing queues to grow. Disabling such reuse can itself improve goodput independently of workflow selection. Moreover, Appendix G states that the reconstructed Murakkab receives a default workflow, while Appendix B acknowledges the original system’s workflow-parameter optimization and periodic reconfiguration [R4]. The results therefore establish an advantage over the evaluated configuration, but leave the incremental contribution of the proposed joint selection unresolved. A comparison holding reuse and deployment logic fixed while varying workflow choice, together with an offline-selected fixed-workflow baseline, would address this issue directly.

  • The parameter-size pruning can discard feasible deployments before their feasibility is checked. In Algorithm 1, the budget β is computed from the smallest accuracy-admissible models, and assignments exceeding it are removed before evaluating latency and capacity. For a single-stage workflow without reuse, only models of the smallest eligible size survive. If none of their configurations satisfies the system constraints, but a larger eligible model has a feasible configuration, that workflow is discarded despite admitting a valid deployment. The paper acknowledges that its heuristic can miss lower-cost assignments, but losing feasible solutions has a different consequence, particularly given the subsequent SLO relaxation in Appendix E. The observed agreement with brute force in Section 5.3 is encouraging, but does not quantify this risk across changing load and placement conditions. The evaluation needs to distinguish failure of the pruned search from actual infeasibility.

References

[R1] (NeurIPS 2024) L. Chen et al. Are More LLM Calls All You Need? Towards the Scaling Properties of Compound AI Systems. arXiv:2403.02419.

[R2] (ICML 2025) J. Saad-Falcon et al. Archon: An Architecture Search Framework for Inference-Time Techniques. arXiv:2409.15254.

[R3] (arXiv 2025) L. Chen et al. Optimizing Model Selection for Compound AI Systems (LLMSelector). arXiv:2502.14815.

[R4] (OSDI 2026) G. I. Chaudhry et al. Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platforms.

Questions for authors
  • How do the selected variants compare with the baselines on held-out, task-specific output quality, particularly when multiple operations are merged into one call? Please evaluate the final outputs of the deployed workflows and report whether the cost and goodput improvements persist under comparable measured quality.

  • How much of the reported improvement remains when workflow choice is the only changed factor, with identical model candidates, load information, reuse policy, and deployment logic? Please also compare against a fixed workflow selected offline from the same candidate set, and specify which periodic reoptimization and runtime adaptation mechanisms are retained in the Murakkab reconstruction. Since FlowMaker selects once per task deployment, how do its selections hold up when load shifts after deployment?

  • When the assignments retained by Stage 2 are infeasible, does FlowMaker expand β before relaxing the SLOs? Please compare against brute force over the original candidate set across different load and memory states, reporting how often the heuristic finds a feasible deployment when one exists and whether any SLO relaxations were avoidable.

  • What assumptions connect the incremental memory–latency objective in Eq. 1 to the deployment-residency cost reported in Eq. 3? Persistent deployments need not have residence times proportional to individual request latency, so the two objectives can rank configurations differently. Please validate the cost rankings on representative feasible variants and clarify the implications of the reported zero objective gap for measured serving cost.

5 comments
AuthorsOpenReview ↗

Answers to Reviewer FmAw's Weaknesses

R2-W1/Q1: Task quality under measured outputs

Measuring the output quality of LLMs on real-world tasks remains an open problem (see R1-W1), and hence we follow the methodology of prior works using proxy-based accuracy metrics [MaverIQ]. Since we use the same proxy metric across all baselines, the results remain consistent. Nevertheless, FlowMaker’s modular design allows replication for any appropriate quality estimator (see §3.2).

However, to provide more details on the output quality of the generated results, we use an LLM evaluator (gpt-oss-120b) as a judge that scores each output from 1-100; a common methodology used in prior work [A Survey on LLM-as-a-Judge; arXiv’25]. Across all served requests pooled over all traces, each baseline achieved the following median scores: FlowMaker, FlowMakerNoReuse & RouteLLM-maxquality → 55/100, Murakkab → 48/100, LangChain, RouteLLM-balanced & RouteLLM-mincost → 35/100. We also measure the per-task quality across baselines (see R1-W1). The results indicate that (i) across all settings, FlowMaker consistently meets the accuracy requirement of each task and (ii) tasks where we merge still have high output quality (e.g., in T4 & T6, FlowMaker achieves the highest scores while using merged operations, while the other baselines do not merge).

We also investigate the quality implications of merged operations in isolation (see R1-W1). We fix the model selection and evaluate, again through the LLM evaluator, the quality of the generated outputs across 3 tasks using merged and unmerged operations. The effect of merging on quality is task-specific. On T2, merging preserves quality with the 14b and 32b models but hurts the 7b model. T6 maintains its accuracy across workflows, while T7 improves sharply when merged, since the LLM sees the full input at once instead of passing it through multiple steps.

So, FlowMaker maintains the required end-to-end task quality even when it merges or shortens workflows.

R2-W2: Isolating the benefit of joint selection

We agree with the reviewer that comparing FlowMakerNoReuse with Murakkab does not isolate the joint search exploration. To test this, we disable Murakkab’s reuse and provide it with (i) its default workflow and (ii) FlowMaker’s selected workflow. We collect results over 2 rate scales (100% and 50% load) using the code trace. Each cell in the table shows the result for 100%-load followed by 50%-load (<100%-load / 50%-load>).

MetricFlowMaker-NoReuseMurakkab-NoReuse (default workflows)Murakkab-NoReuse (FlowMaker's workflows)
Goodput (RPM)10.35/11.921.63/7.0610.42/11.95
Cost ($)1.285/0.8201.668/1.5251.342/1.136
p95 latency (s)660/1251153/765457/124

These results establish our key insight by showing that exploring the full search space leads to picking better workflows, which greatly affects performance; both FlowMakerNoReuse and Murakkab-NoReuse with FlowMaker’s workflows outperform Murakkab-NoReuse with its default static workflow definition.

R2-W3: Does FlowMaker’s algorithm discard feasible solutions?

We clarify that while our algorithm may miss min-cost solutions, as it prunes the search space, it never misses feasible solutions. As we set the β parameter budget (Alg. 1, line 5), we identify the smallest accuracy-admissible models (i.e., β shows the size of the cheapest deployment that meets the accuracy SLO without reusing any deployed model); everything that surpasses this budget requires extra GPU memory (already deployed models do not count towards the deployment cost). Thus, when we start evaluating the system constraints (lines 8-17), we know that the β-constrained variant is the threshold after which latency and memory requirements increase; if this variant cannot be deployed, none of the pruned ones will be able to be deployed in the online cluster, due to memory availability. Thus, we look over the sorted pruned search space and stop when we find the first feasible one.

AuthorsOpenReview ↗

Answers to Reviewer FmAw's Questions

R2-Q1: Task quality under measured outputs

Please see R2-W1/Q1

R2-Q2: Workflow-only ablation and load shifts

To isolate the workflow choice, we disable reuse across FlowMaker and Murakkab and rerun the code trace under high (100%) and medium (50%) load. The results (see R2-W2) confirm that full search-space exploration (including the workflow choice) plays a very important role, leading to better workflow variants that improve goodput, cost, and latency.

To isolate the model and deployment configuration impact, we run the code trace under medium load but only use Qwen-2.5-14b with TP1/PP1 instances. The results are as follows:

MetricFlowMakerFlowMaker-NoReuseMurakkab
Goodput (RPM)8.4110.581.21
Cost ($)1.0510.9782.232
p95 latency (s)361.5260.91137.6

With a single model and a single TP/PP configuration available, FlowMaker-NoReuse achieves the best goodput and lowest operational cost, as it deploys separate replicas for each operation. FlowMaker falls a bit behind, as it reuses some instances, causing resource contention. On the other hand, Murakkab performs the worst across all baselines, as it makes reuse choices offline, without reasoning over the serving load of the deployed instances.

Note that in our experiments, we disabled the periodic reconfiguration spec of Murakkab to enable fair comparison with our system, as this work focuses mainly on the deployment phase. Adding an autoscaler or applying reconfiguration is orthogonal, and FlowMaker’s modular design allows easy addition with minimal changes.

But to answer the reviewer’s question, we implement Murakkab’s periodic online reconfiguration and compare it with FlowMaker. As Murakkab requires deploying new instances and removing old ones, to decrease the downtime, we use 6 serving GPUs and 2 for those transitions. We run the code trace under low (25%) and medium (50%) load using the same experimental setup, serving 5 tasks in total. Each cell holds two values separated by ‘/’: the result at 25% load, then at 50% load:

MetricFlowMakerMurakkab (paper implementation)Murakkab+reconfig (default workflows)Murakkab+reconfig (FlowMaker's workflows)
Goodput (RPM)10.28/13.297.88/7.238.97/5.0710.59/12.44
Cost ($)0.327/0.4370.626/1.1040.996/1.0180.695/0.748

The results show that in every scenario, FlowMaker outperforms all variants of Murakkab. As we increase the load, Murakkab's reconfiguration downtime adds extra overhead, decreasing the goodput by 1.42x (7.23 vs 5.07 RPM). The only variant that is comparable to FlowMaker is Murakkab with FlowMaker’s workflows and periodic reconfiguration; however, FlowMaker is significantly cheaper.

R2-Q3: SLO relaxation and brute-force comparison

SLO Relaxation. FlowMaker relaxes the SLOs progressively as follows: Initially, it tries to use all SLOs and constraints with β If this fails, it drops serving capacity constraints and keeps β If the previous step fails, it drops accuracy SLOs and β and explores the full search space Finally, it drops both latency and accuracy SLOs, β, and explores the full search space If the final iteration does not lead to a choice, then we inform the user that no deployable workflow exists under those conditions. We will clarify this in the paper as well.

Across all reported experiments in the paper, FlowMaker never invoked this relaxation, while FlowMaker-NoReuse invoked it only on the 14-task scalability experiment, where it was not able to deploy 7/14 tasks due to space limitations (App. H.2).

Brute-force Comparison. We note that our algorithm does not miss feasible solutions (see R2-W3). To compare a brute-force strawman approach with FlowMaker, we replayed the code trace at the highest load with brute-force search making each deployment decision. The results (see R1-W2) showed that it made the same choices as FlowMaker for every task.

R2-Q4: Eq.1 vs. Eq.3 & zero objective gap

Eq.1 vs. Eq.3. We agree with the reviewer that Eq.1 and Eq.3 may lead to different choices; but, we clarify that: Eq.3 captures the operational cost (which depends on the duration models stay deployed in the GPUs), while Eq.1 is used as the algorithm’s cost function in Alg.1. As the amount of time models remain deployed in the server is not known prior to execution (it depends on the trace and the number of requests served), we can’t use Eq.3 as the cost function in our algorithm. We use Eq. 1 as a proxy to capture residency cost.

Zero Objective Gap. §5.3 reports a “0% observed objective gap” between FlowMaker’s and BF’s choices, as both led to the same selection. We will clarify this in writing.

Thank you for the additional experiments. However, the LLM-judge evaluation is not sufficiently specified or validated, and it remains unclear whether the merged 7B T2 configuration was ever selected in the end-to-end experiments. The no-reuse ablation supports the importance of workflow choice, but without an offline-selected baseline, it does not establish that online joint search is necessary. The inconsistent descriptions of when assignments beyond β are explored also leave the claimed feasibility guarantee unclear. As these issues bear directly on the paper's central claims, I will maintain my original score.

Reviewer_TtH8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FlowMaker, a workflow deployment system for generative AI tasks that jointly optimizes workflow structure, stage-specific models, deployment configurations, and the reuse of existing model instances. Given user-defined accuracy and latency constraints, along with proxy accuracy scores for individual stage models, offline performance profiles, and the current cluster state, the system employs a multi-stage pruning algorithm to select a low-cost feasible workflow variant. The authors evaluate the system on an eight-GPU prototype built on TensorRT-LLM using seven tasks, various request arrival patterns, and different load levels. The evaluation compares goodput, serving costs, and search time against several baselines and includes component-level ablation studies.

Critical strengths
  1. By incorporating workflow structure selection into the joint optimization of models and deployment configurations, the system can exploit cost and latency trade-offs across different workflows that methods relying on fixed workflow structures cannot explore.

  2. The system makes reuse decisions based on the current load and remaining capacity of existing model instances, reducing unnecessary deployments and their associated serving costs.

Critical weaknesses

W1. The paper does not evaluate actual generated outputs and instead uses proxy accuracy scores derived from leaderboards of individual stage models to represent workflow quality. Consequently, it remains unclear whether the system maintains the required end-to-end task quality after merging or shortening workflows.

W2. The use of parameter-size threshold pruning, combined with a strategy that stops searching after finding the first feasible deployment configuration for each candidate model assignment, may discard better solutions or even miss solutions that satisfy the specified latency constraints.

W3.The evaluation covers only seven end-to-end tasks, each with a limited set of three candidate workflows containing one to three stages. Given that the selection process for a seven-stage workflow takes approximately 210 seconds in the scalability experiment, the current approach may not scale effectively to support the rapid deployment of more complex workflows.

Questions for authors

See weakness.

2 comments
AuthorsOpenReview ↗

Answers to Reviewer TtH8's Weaknesses

R1-W1: Measured end-to-end output quality

Evaluating the output quality of LLMs on real-world tasks remains an open problem [A Survey on Evaluation of LLM-based Agents; ACL’26]. Hence, we follow prior works’ methodology: we use proxy accuracy scores [MaverIQ] and approximate workflow quality as the minimum accuracy across its stages, motivated by the correlation between per-stage and E2E quality [LLMSelector, Aragog]. Nevertheless, FlowMaker’s modular design allows the quality estimator to be replaced with any suitable alternative, including estimators developed in the future (§3.2).

To further evaluate FlowMaker's efficacy on output quality, we use an LLM evaluator (gpt-oss-120b) as a judge to score all generated outputs from 1-100, a common methodology in prior work [A Survey on LLM-as-a-Judge; arXiv’25]. The table gives the scores per task and baseline for all E2E experiments (Fig. 5). T2,T7 have an accuracy SLO set to BASIC, T1,T3,T5 GOOD, T4,T6 BEST (App. G).

BaselineT1T2T3T4T5T6T7
FlowMaker25357855828535
FlowMakerNoReuse25327855828535
Murakkab25357845785235
LangChain2535784568-35
RouteLLM-maxquality25587848825055
RouteLLM-balanced22555835823255
RouteLLM-mincost15455835683035

Across all non-BASIC tasks, FlowMaker achieves the top quality despite merging and shortening workflows. For T2 and T7, it meets the accuracy SLO, but not necessarily top quality, to reduce operational cost and latency.

We also investigate whether merging operations affects quality. With the same evaluator, we report the median scores of merged and unmerged workflows over 100 requests with 3 Qwen-2.5 models:

TaskWorkflow7b14b32b
T2translation→summarization5556.559
T2summarization→translation53.57876
T2merged (translation-summarization)225566.5
T6aspect→sentiment→opinion summary325553.5
T6sentiment→opinion summary7576.579
T6merged (review-analysis)788882
T7keyphrase→topic→tags355855
T7topic→tags47.55845
T7merged (auto-tagging)928888

The effect of merging on quality is task-specific; T2 achieves similar quality on the 14b and 32b variants, while merging operations hurt performance on the 7b variant. T6 preserves its accuracy across workflows, while T7 improves drastically when merged, as the LLM receives the full information without iterating over it across multiple steps.

So, FlowMaker maintains the required end-to-end task quality even when it merges or shortens workflows.

R1-W2: FlowMaker’s algorithm selections

The reviewer is correct that our heuristic may miss the globally optimal (min-cost) solution. However, FlowMaker always finds a feasible workflow variant when one exists. A variant is feasible if it (1) meets the accuracy SLO, (2) meets the E2E latency SLO, (3) has enough serving capacity for the offered load, and (4) fits in the available GPU memory.

This derives from the algorithm. Early termination (Alg.1, line 17) triggers only after a feasible variant has been found for a workflow; until then, FlowMaker checks every TP/PP configuration of every candidate model assignment (lines 8–16). If no assignment within the parameter budget is feasible, FlowMaker searches the remaining assignments before relaxing any SLO. Pruning and early stopping therefore affect only the cost of the selected variant, never whether a feasible workflow is found. In all deployment decisions in our experiments, the first pass found a variant satisfying every requirement.

Finding the globally optimal variant requires brute-force search, which is too expensive at deployment time (>15 min at ≥6 stages, Fig.6). Nonetheless, we applied BF to the code trace at the highest load; FlowMaker made the same choice as BF search for every task deployment.

R1-W3: Support for complex workflows

We target typical GenAI workflows, which are short: nearly half of deployed agents execute fewer than 5 steps before human intervention [Measuring Agents in Production; ICML'26], and representative agentic workflows make about 5 LLM calls per request [Aragog]. For those workflows, FlowMaker selects variants in ~1s. For longer workflows, it remains the only approach that finds a variant within a few minutes; BF and BO exceed 15 min at ≥ 6 stages (Fig.6).

This overhead can be significantly reduced: iterations of our algorithm are independent, so they can run in parallel. We re-implemented it with optimizations including cached per-model profiles, vectorized checks, and a process pool, and find that even with 9 stages or 125 models to choose from, our algorithm completes in <30 seconds. Search time (sec) when scaling stages and model catalog size is shown below:

SettingFlowMaker (paper)1 thread, cached128 threads, cached
7 stages2092.640.22
9 stages>9005097.2
75 models @ 5-stages>9001112.8
100 models @ 5-stages>9004569.52
125 models @ 5-stages>900>90026.1

Hello Reviewer TtH8. We appreciate the constructive feedback you provided in your initial review. Based on your feedback and questions, we collected additional results, as presented in our response. We hope our additional experiments and results address all your feedback. Given that the review deadline is wrapping up in a day, we would appreciate it if you could let us know if there are any remaining concerns. We look forward to hearing from you.

Comments on the paper

1 comment

We thank the reviewers for their careful and constructive reviews. We ran extensive new experiments to address all the comments raised in the reviews and described them in detail in our responses to each reviewer. Additionally, we have revised the paper to include all the new data, results, insights, and clarifications that came out of the reviews and the discussion so far. All new or modified text is marked in red. The main text remains within the 10-page limit; the new results are in a new appendix (App. I).

The new experiments include:

  • LLM-judge scores for every output of the end-to-end experiments, and a merged vs. unmerged workflow comparison (App. I.1)
  • controlled ablations against Murakkab: the same workflows with and without reuse, and a single-model, single-configuration run (App. I.2)
  • Murakkab with periodic reconfiguration (App. I.3)
  • a brute-force replay and an optimized, parallel selector scaled to 9 stages and 125 models (App. I.4)

We will include any additional clarifications or results that follow-up comments call for.

Given that the review deadline is wrapping up in a day, we would appreciate it if you could let us know of any remaining concerns. We look forward to hearing from you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.