- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how to decompose complex reasoning problems into task graphs and execute them with different LLMs under resource constraints. The paper proposes JOVE, in which a planner LLM first decomposes each problem into a DAG of subtasks, and then, for each node, JOVE jointly decides two things, (i) which executor LLM to use, and (ii) whether to call a verifier LLM to check the node's output. The verification results do not affect answer generation and are only used to update the estimates of each model's quality, so as to improve future allocations. JOVE formulates the decisions for each query as a mixed-integer linear program that maximizes the average node correctness under a long-term average budget and a per-query latency constraint, where model quality is estimated online with LinUCB, the value of verification is measured by information gain, and the budget is controlled by dynamic pricing. The paper proves sublinear quality-learning regret under certain assumptions and conducts experiments on Bamboogle, MMLU-Pro, GPQA, and LiveBench-Reasoning, claiming substantially lower cost and latency with accuracy comparable to the baselines.
Critical strengths
The contribution of the paper is an optimization framework with theoretical guarantees, which jointly models executor selection and verification decisions, considers both a long-term budget and a per-query latency constraint, and provides a sublinear regret bound.
Critical weaknesses
-
The proposed idea of using a planner to decompose a problem into subtasks and then selecting an executor for each node is not necessarily general or reasonable for all tasks, and the paper does not discuss its scope of applicability. (i) Some problems cannot be decomposed into subtasks, or the resulting subtasks are rather abstract. However, the number of nodes in the main experiments is manually fixed at 2–5 (only the appendix tries letting the planner decide on MMLU-Pro), so problems that are hard to decompose must still be decomposed, which may not be the most appropriate approach. (ii) Selecting an executor for each node makes almost no use of language understanding. It only encodes the node description with a very small sentence-embedding model (MiniLM) and scores it with a linear model, so the more abstract the subtask, the less reliable the selection. (iii) The paper shows no actually generated task graphs and does not evaluate the quality of the decomposition. This framework may only be suitable for multi-step tasks that can be naturally decomposed.
-
The planner used in the experiments appears too strong for the evaluated benchmarks, and the paper provides no further experiments comparing different planners. The planner in the experiments is Gemini 2.5 Flash-Lite, which is already a capable commercial model (relative to the evaluated benchmarks). Since the proposed method has to call it once for every question anyway, why not let the planner answer the question directly? Then neither the executors nor the verifier would be needed. The converse question is whether the planner needs to be a strong model at all. Would a smaller planner still achieve this performance? The paper conducts no experiments on these questions, so it is unclear whether the decomposition is reasonable and whether the whole pipeline is necessary.
-
The baseline comparison is insufficient, and the baselines are rather weak. (i) The inference-level baselines use only a single model, Qwen3-32B, while JOVE can choose from a pool that includes stronger models such as GPT-OSS-120B and Llama-3.3-70B. The paper does not compare with the strongest single model in the pool answering directly, or with query-level routing over the same pool under the same budget. (ii) Among the resource-aware baselines, WR-Online is the only external method, and the two Knapsack variants are heuristics. (iii) The paper cites many related methods, e.g., in Paragraph 2 of Section 2, but does not compare with most of them.
Questions for authors
- Does the proposed framework rely on a strong planner? Would a smaller planner still achieve this performance? If the planner answers the question directly without decomposition, what are the accuracy, cost, and latency? If the planner were not an LLM but a random or rule-based decomposition (the authors are free to choose the rules, e.g., splitting by commas or by syntax), what would the accuracy be?
- What happens when problems that cannot be naturally decomposed are forced into 2-5 nodes? Could the authors show examples of actually generated task graphs and evaluate the quality of the decomposition?
- What are the results if the strongest single model in the pool (e.g., GPT-OSS-120B) answers directly, or if query-level routing is performed over the same pool under the same budget? And could the authors also provide a broader comparison with other SOTA baselines?
8 comments
W3 and Q3 Extra baselines
Direct answers from stronger models: We have additionally evaluated GPT-OSS-120B and Llama-3.3-70B-Instruct from JOVE’s executor pool as single-call baselines. As shown in Table 7 (link above), GPT-OSS-120B achieves higher accuracy but at 2.47-8.78 times JOVE’s API cost and 1.88-3.74 times its latency. Llama-3.3-70B-Instruct exceeds JOVE’s accuracy only on MMLU-Pro and incurs higher costs on all four datasets.
Query-level routing: We added two online routing baselines beyond WR-Online: B.A. LinUCB [1] and LimBanFeed [2] (Table 1, https://anonymous.4open.science/r/JOVE/Table1.png). Across the four benchmarks, JOVE achieves higher accuracy and lower latency than both, with costs moderately higher than B.A. LinUCB but substantially lower than LimBanFeed.
Interpretation: B.A. LinUCB uses successive routing attempts and free answer-level feedback, while JOVE learns from paid node-level verification and can execute independent subtasks in parallel. LimBanFeed selects verifications after execution, whereas JOVE jointly optimizes both decisions under cost and latency constraints.
[1] Online Multi-LLM Selection via Contextual Bandits Under Unstructured Context Evolution, AAAI 2026
[2] Online Learning with LLM Experts from Limited Feedback, preprint Sept 2026
We thank the reviewer again for their time and feedback and would be happy to answer any further questions.
Thank you once again for your valuable comments on our submission. As the author-reviewer discussion period is approaching its end, we would like to kindly confirm whether we have sufficiently addressed your concerns. If any questions remain, please let us know, and we would be happy to clarify. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly. We sincerely look forward to your feedback.
We thank the reviewer for their feedback and address each concern below. We have also highlighted the corresponding revisions in blue in the updated paper.
W1 and Q2 Clarification on the applicability of JOVE
We have added examples of planner-generated task graphs to the revised paper (see Appendix A.3.3). These examples illustrate how the planner identifies intermediate reasoning steps and dependencies not explicitly stated in the query. We have also revised Figure 1 (https://anonymous.4open.science/r/JOVE/overview_revised.pdf) to better illustrate task decomposition when the intermediate reasoning steps are not explicitly stated in the query. JOVE is designed for problems where decomposition yields useful, executable subtasks. For simple queries, or those that do not benefit from decomposition, JOVE’s planner may generate a single-node graph that reduces JOVE to query-level routing with optional verification, under the same resource constraints. We acknowledge that vague or poorly decomposed subtasks can degrade performance. We have clarified this in the revised paper (see Remark 3.4).
Quality predictor: We use MiniLM embeddings of node and candidate-model descriptions with a linear predictor to limit allocation overhead and support the regret analysis (Theorem 3.7). JOVE’s MILP can also use scores from other quality predictors, although the current regret guarantee relies on the stated linear model assumptions.
W1 Task graph sizes
We agree that fixing a graph size can introduce unnecessary work when a query admits a simpler plan. In the main experiments (Table 1), we generated four graphs per query (2-5 nodes) to increase the number of online learning rounds, particularly on smaller datasets.
Planner-selected graph sizes: We have evaluated one planner-selected graph per query across all four datasets in Appendix A.4.1 (Table 2, https://anonymous.4open.science/r/JOVE/Table2.png). JOVE achieves 49.3% mean accuracy, compared with 49.5% when using four graphs per query, suggesting that fixing multiple graph sizes is not necessary to maintain accuracy. On a mixed workload with one graph per query, JOVE achieves the highest accuracy among the considered resource-aware methods (50.5%, see Table 3, https://anonymous.4open.science/r/JOVE/mixed.png). Appendix A.3.3 also illustrates that, for a naturally two-hop question, requesting additional nodes produces finer-grained retrieval and answer extraction steps rather than necessarily introducing vague subtasks.
W2 and Q1 Planner ablation
We thank the reviewer for suggesting these comparisons. We chose Gemini 2.5 Flash-Lite for its low cost, fast inference, and reliable adherence to the task-graph JSON schema. We evaluated three alternative planners (Table 7, https://anonymous.4open.science/r/JOVE/ablation-planner.png): Llama 3.1 8B Instruct, Gemma 3 4B-IT, and a rule-based planner that splits queries at punctuation and conjunctions. Gemini 2.5 Flash-Lite achieves the highest accuracy on all four datasets, with Llama 3.1 8B Instruct as the closest alternative. Gemma 3 4B-IT achieves 6.25 percentage points lower mean accuracy and produces invalid task graphs for 5.1% of queries (scored as incorrect). The rule-based planner achieves only 11-29% accuracy across the four datasets, suggesting that simply splitting the question into parts often fails to capture the reasoning needed to solve it.
Why not let the planner answer the question directly? Table 7 (see the link above) shows that JOVE achieves 11.0 percentage points higher mean accuracy than direct calls to Gemini 2.5 Flash-Lite across the four datasets. Alongside this accuracy gain, JOVE gives the system greater control over end-to-end latency and API cost.