- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ProRe, a Propose-and-Refine multi-turn routing framework designed for long-horizon tasks, and OnVir, an online data-collection strategy that augments Monte Carlo Tree Search (MCTS) with virtual rollouts. ProRe utilizes a generative proposer to sample future routing plans comprising variable-duration macro-actions, and a plan-conditioned refiner to rank these plans based on predicted terminal performance and cumulative inference costs according to specific user preferences. OnVir mitigates the high costs of real-environment MCTS rollouts by substituting them with ProRe-generated virtual rollouts.
Critical strengths
-
Innovative Data Collection Strategy: The introduction of OnVir to replace expensive real-environment MCTS rollouts with virtual rollouts—evaluated by the evolving ProRe router itself—is a clever and cost-effective approach to mitigating the data collection bottleneck in long-horizon routing.
-
Explicit Future Modeling: Unlike existing multi-turn routers that model future trajectories implicitly, ProRe explicitly generates and evaluates complete future routing plans. This provides a more interpretable and grounded estimation of long-horizon performance-cost trade-offs.
Critical weaknesses
-
Unquantified Training Overhead: While the paper successfully argues that OnVir lowers rollout costs compared to traditional MCTS, the computational overhead of the iterative training itself (updating the proposer, performance predictor, and cost predictor over multiple rounds) is not systematically compared against the training costs of the baselines. The absolute experimental cost (~3000 USD) is buried in the appendix, leaving the comparative efficiency of training ProRe unclear.
-
Method Presentation and Figure Clarity: The method presentation, particularly Figure 2, is highly abstract and confusing. It relies heavily on color-coded blocks without providing a concrete grounding example (e.g., an actual interaction trace in a task like "Fix the CSV bug"), which obscures the practical mechanism of macro-actions and plan refinement, negatively impacting reproducibility. Furthermore, the title of Figure 1(b) ("Lack of high-quality traning data") does not align with its visual content, which actually illustrates "High Real-Environment Rollout Costs".
-
Cross-Domain Generalization & Fine-Tuning Dependency: It is unclear if the trained router (specifically the task-conditioned predictors) requires complete retraining or fine-tuning when transferred to entirely new domains. A strong dependency on domain-specific training data severely limits the practical out-of-the-box utility of the proposed framework.
Questions for authors
- Training Overhead Comparison: Could you provide a quantitative comparison of the total training costs (compute and API costs) between ProRe/OnVir and the primary multi-turn baselines (e.g., SeqRouter, MTRouter)? How does the iterative optimization of the three predictors impact the overall training budget compared to offline baseline methods?
- Predictor Convergence and Dynamics: How does the accuracy of the predictors (performance and cost) evolve during the online collection rounds? Does the predictor effectiveness heavily bottleneck the final routing quality, and do they converge early or require continuous optimization?
- Cross-Domain Fine-Tuning: Does ProRe require domain-specific fine-tuning to maintain its routing accuracy in entirely new domains? How robust are the predictors to zero-shot out-of-domain evaluation without any in-domain bootstrap data?
- Role of Macro-actions: In the ablation study (Table 2), removing the macro-action design only resulted in a marginal utility drop (e.g., Overall from 0.34 to 0.33). The text mentions its main role is to "reduce the number of routing decisions". Does this reduction translate to measurable improvements in wall-clock time or significant savings in the router's own inference overhead? Please quantify this benefit in the rebuttal.
Gentle Reminder to Reviewer acAN
Dear Reviewer acAN,
Thank you again for your careful review and helpful comments. We would be grateful if you could take a look at our responses when you have a moment.
Your first concern is that OnVir discards previous virtual rollouts. This is an intentional design, because a virtual rollout is not actually executed: the refiner directly estimates its performance and cost. Following AlphaGo, we take the current state as the root and roll out from there. Because it is virtual, it cannot be reused in the next state or inserted into the Monte Carlo tree of next state.
Discarding them does not remove long-range dependency, because each state already uses many virtual rollouts and ProRe considers the future through an explicit plan. Instead, keeping them would make the design unreasonable. We have also compared alternative MCTS simulators in our work.
Your second concern is sensitivity to hyperparameters. All hyperparameters are fixed. ProRe has only two core hyperparameters, and OnVir uses common hyperparameters for online data collection. We have ablated the key ones in extensive experiments.
Your third concern is the out-of-distribution issue in offline Q-learning. That discussion is only an example to explain our motivation. We do not claim to solve that problem. Higher-quality training data by Onvir can alleviate this concern.
Please let us know if you have any further comments. If anything is still unclear, we would be happy to discuss it.
Thank you again.