- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper focuses on multi-turn LLM routing, where each turn an LLM from a pool is selected to continue the conversation. The method they proposed leverages a value model to select the LLM given previous conversations and the LLM pool. By introducing hindsight response as training-time privileged information, their router achieves better performance compared to previous methods and surpasses the best LLM in the pool and a lower cost.
Critical strengths
- The method they proposed is well-motivated and easy to follow. The performance is competitive and has a lower cost compared to previous methods.
Critical weaknesses
- The most critical weakness is the limited novelty. The main contribution is a value model trained with response information. However, this is a not a new idea as the authors discussed in the related works. And I don't see much difference of the proposed method compared to the previous ones in introducing privileged information during the training of a value model.
Questions for authors
- What's the cost for training such a router compared to other methods? How many LLM calls are required to amortize the training cost?
- I wonder whether or not the method can be applied to a pool that is much larger than the current one.
- How's the cost computed? Why would a cost larger than 1 if deepseek-v4-pro is the most expensive model? (An educated guess is that some model may produce longer responses and as a result it costs more than simply using ds-v4-pro. But for now, it's a bit unclear.)
Thanks for your response, and I have raised my score.