- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FlowCompile optimizes structured LLM workflows by jointly selecting models, reasoning budgets, and permitted structural choices. It profiles sub-agents using reference-model traces, analytically composes their measurements into workflow-level accuracy and latency estimates, and constructs a reusable trade-off set. Experiments on four benchmarks evaluate deployment performance, proxy quality, and compatibility with query-level routing.
Critical strengths
-
Separating reusable sub-agent profiling from workflow-level search is practically appealing and allows the resulting configuration set to accommodate different deployment preferences.
-
The reported accuracy–latency improvements across four benchmarks are encouraging. Restricted-space frontier comparisons and reference-model ablations provide additional empirical support.
Critical weaknesses
-
The methodological novelty is insufficient. LLMSelector already uses module-level evaluations to optimize model assignments in compound AI systems. FlowCompile extends this direction to reasoning budgets, permitted structural choices, and reusable accuracy–latency trade-off sets. Its optimization procedure combines analytical composition rules, local Pareto pruning, configuration enumeration, and non-dominated sorting. While the joint formulation is useful, I find the additional algorithmic or conceptual insight beyond this integration too limited to constitute a sufficiently substantial contribution. This is my primary concern, rather than a claim that the method is technically invalid.
-
The experiments do not sufficiently isolate the value of the proposed optimization approach. Several baselines optimize fewer configuration dimensions, and Table 4 shows substantial gains from expanding the search space itself. Appendix D provides useful restricted-space comparisons, but does not fully establish the advantage over alternative optimization approaches operating in the same space under comparable offline evaluation budgets. This makes it difficult to determine whether the reported gains demonstrate a substantial advance beyond the additional configuration flexibility.
Questions for authors
-
What is the principal new technical insight beyond existing module-level model selection, expanded configuration choices, and Pareto-based trade-off construction?
-
Which baselines share the full configuration space, and how do their offline optimization budgets compare? Is there additional matched-space, budget-controlled evidence that isolates the benefit of the proposed approach?