- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
FlowMaker is a serving system for LLM workflows. Users provide a task and its latency and accuracy SLOs. They also confirm that the generated candidate workflows are semantically acceptable. At deployment time, the system jointly selects the workflow, models for each stage, and parallelism configurations. It also decides whether to reuse existing instances based on their current load. The selector uses three pruning stages: accuracy, incremental deployment footprint, and system constraints. It is a heuristic and does not guarantee global optimality. Experiments use eight A6000 GPUs and a reconstructed Murakkab baseline. At high load, the mean FlowMaker-to-Murakkab goodput ratio is 4.73. The mean Murakkab-to-FlowMaker cost ratio is 1.95. Selection takes about one second on the seven evaluated tasks. The selected variants match brute force at all comparable points, including the scaling experiments. Selection time reaches 210 seconds at seven stages.
Critical strengths
-
The problem is clearly formulated and supported by experiments. The system jointly considers workflows, models, deployment configurations, and reuse under the current load. The motivation experiments show the benefits of alternative workflows and load-aware reuse.
-
The implementation and evaluation are reasonably complete. The paper includes an ablation of reuse across tasks and separate evaluations of selector speed and selection quality. The experiments also support joint selection, although differences in reuse policy are not fully controlled.
Critical weaknesses
- Output quality is not directly evaluated. Accuracy uses average leaderboard scores and their minimum across stages. This leaves quality preservation in shorter workflows unverified.
- Murakkab receives one default workflow, while FlowMaker has three candidates. Its offline reuse policy and periodic reconfiguration settings are unclear. These gaps limit attribution of the gains.
Questions for authors
- Can output quality be evaluated on one or two tasks?
- How much of FlowMaker's advantage remains if Murakkab uses the best approved workflow? Does its reuse policy use OfflineReuseAvg, OfflineReuseMax, or another strategy?
- Which setting and baseline underlie the reuse gains in Appendix H.4 (1.13× goodput and 1.42× cost reduction)?
3 comments
Thanks for the detailed response. The new experiments address most of my concerns. I suggest including the comparison with the same workflows in the paper. FlowMaker costs less at both tested loads, but Murakkab has higher goodput at full load. I have no further questions, and my original score stands.
Author Response to Official Reviewer s43D's Comment
Thank you for the prompt follow-up: we appreciate your constructive feedback and are glad the new experiments addressed most of your concerns. We will include the comparison using the same workflows in the revised paper and clarify the cost–goodput tradeoff. We hope the additional evidence gives you confidence to consider stronger support for the paper.
Answers to Reviewer s43D's Weaknesses & Questions
R3-W1/Q1: Measured output quality
We agree with the reviewer that the proxy-accuracy metric does not evaluate true output quality. Nevertheless, we note that we have used the same proxy-accuracy metric across all baselines to enable fair comparison. On top of that, we post-process all the generated outputs across the end-to-end experiments (Fig. 5), through an LLM evaluator (gpt-oss-120b), which scored each of those from 1-100. Across all the experiments, we report the median scores for each baseline: FlowMaker, FlowMakerNoReuse & RouteLLM-maxquality → 55/100, Murakkab → 48/100, LangChain, RouteLLM-balanced & RouteLLM-mincost → 35/100 (see R2-W1), which indicate that FlowMaker achieves similar or better quality compared to the rest of the systems.
In addition, we analyse the per-task scores pooled over all traces (see R1-W1) and find that across 5/7 tasks (T1, T3, T4, T5, T6), FlowMaker achieves the top-ranking scores, except T2 and T7, where their BASIC-SLO configuration (see App. G; Proxy-accuracy SLOs) does not require high-quality models.
Finally, we evaluate the effect of merging operations on output quality (see R1-W1). We analyse 3 tasks (merged and unmerged operations) using 3 fixed models and measure their quality through an LLM evaluator. When merging, T2 preserves quality with the 14b and 32Bb models but hurts the 7b model; T6 maintains its accuracy across workflows; T7 improves drastically when merged, since the LLM sees the full input at once instead of passing it through multiple steps.
In conclusion, we show that FlowMaker meets every task’s accuracy requirement, and merging or shortening workflows does not degrade the E2E task quality.
Please see our responses in R1-W1 & R2-W1 for the detailed results.
R3-W2/Q2: FlowMaker vs Murakkab comparison
We note that in our experiments, we have disabled Murakkab’s periodic reconfiguration to enable fair comparison. The reconfiguration part is orthogonal to our work, as we focus on the workflow variant selection and deployment phase of the serving pipeline. Nevertheless, FlowMaker’s modular design allows integration with autoscaling/reconfiguring approaches with minimal source code modifications.
We specify that, as the size of our cluster did not allow for provisioning at the maximum load, we use the OfflineReuseAvg as Murakkab’s default policy, similar to the average load that FlowMaker uses to make decisions.
Finally, we provide detailed results regarding Murakkab’s performance when using FlowMaker’s workflow choice. We run the code trace under medium (50%) and high (100%) loads and measure each system’s performance. Each cell in the table shows the result for 50%-load followed by 100%-load, separated by ‘/’.
We notice that even under FlowMaker’s workflow, Murakkab increases operational costs across all rate scales, while FlowMaker’s goodput is similar (medium load) or slightly worse (high load). Thus, the advantages in cost efficiency come not only from full search-space exploration but from the on-the-fly reuse as well.
More results regarding a comparison between the baselines with reuse disabled are located in R2-W2, where we show that full search space exploration has a drastic effect on end-to-end performance. Also, a detailed comparison of FlowMaker with Murakkab’s online reconfiguration scheme is provided in R2-Q2, where we show that even under this setting, FlowMaker outperforms Murakkab.
R3-Q3: Appendix H.4 clarification
The 1.13x goodput and 1.42x cost reduction in Appendix H.4 come from the comparison between FlowMaker vs FlowMaker-No-Reuse (i.e., the gains achieved by on-the-fly reuse) on the code trace at high loads (i.e., 100% and 75%); we will make this clarification in the text.