acceptodds
Under review as a conference paper at ICLR 2027

Responses Reveal What Outcomes Hide: Response-Informed Value Learning for Multi-Turn LLM Routing

Abstract

Multi-turn language-model routing must select each model before its response is available, although training trajectories retain that response and reveal task success only after the interaction ends. This information timing leaves each intermediate choice without an independent task-quality label while exposing post-call evidence about the return that follows it. We introduce RIVET (Response-Informed Value Estimation for Turn-Level Routing), which jointly trains two value estimators over a shared state–model representation: a deployable Routing Value Estimator (RVE) that scores candidate models from pre-call information, and a training-only Response-Informed Value Estimator (RIVE) that additionally observes the realized response. Both estimators estimate the same discounted return under different information conditions, regress to the same scalar return targets, and are coupled through a shared state–model representation, while inference uses RVE alone. Conditioning on the response turns response-induced return variation from residual noise into an observed covariate, and we show that this cleaner auxiliary regression problem provably reduces the estimation error of the shared representation used by the deployed pre-call router. On 141 held-out MINT tasks with six candidate models, RIVET solves 124 tasks and uses 74.7% less total call cost than MTRouter, which solves 120. The resulting system reaches a high-accuracy, lower-cost operating point without adding a post-call routing path at deployment.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Routing, Reinforment Learning, Value-based Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_gPN7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper focuses on multi-turn LLM routing, where each turn an LLM from a pool is selected to continue the conversation. The method they proposed leverages a value model to select the LLM given previous conversations and the LLM pool. By introducing hindsight response as training-time privileged information, their router achieves better performance compared to previous methods and surpasses the best LLM in the pool and a lower cost.

Critical strengths
  1. The method they proposed is well-motivated and easy to follow. The performance is competitive and has a lower cost compared to previous methods.
Critical weaknesses
  1. The most critical weakness is the limited novelty. The main contribution is a value model trained with response information. However, this is a not a new idea as the authors discussed in the related works. And I don't see much difference of the proposed method compared to the previous ones in introducing privileged information during the training of a value model.
Questions for authors
  1. What's the cost for training such a router compared to other methods? How many LLM calls are required to amortize the training cost?
  2. I wonder whether or not the method can be applied to a pool that is much larger than the current one.
  3. How's the cost computed? Why would a cost larger than 1 if deepseek-v4-pro is the most expensive model? (An educated guess is that some model may produce longer responses and as a result it costs more than simply using ds-v4-pro. But for now, it's a bit unclear.)
Reviewer_So1oOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors presented RIVET, a system for choosing which language model to call at each turn of a multi-turn task. The main idea was simple and useful: during deployment, the router must choose a model before seeing its answer, but during training, the answer is already available and can provide extra information about whether that call was useful. RIVET was a promising approach to multi-turn LLM routing, and the reported cost reductions made it worth further study.

Critical strengths

Overall speaking, the idea is practical (as well as easy to understand) and did have meanings in applied situation. It also treated different turns as connected decisions and showed promising cost savings without reducing accuracy.

Critical weaknesses

The main weakness was that the paper did not clearly show why the method worked or whether the gains were reliable through statistical importance measurements. The experiments were small, important setup details were missing, and the tests could not tell whether the model learned from response meaning or simple clues such as length, cost, and format. More experiments in ablations are needed.

Questions for authors
  1. The main results came from one small run. (muli-seeds runs maybe? if plausible?).
  2. I am curious the value of response content (extra ablation study on this?).
  3. The out-of-distribution and turn-level claims were empirical, more solid evidence are needed.
1 comment

Thanks for your response, and I have raised my score.

Reviewer_rvLyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RIVET picks which model an agent should use at each step. It learns from past runs and a short online training phase. A second training-only head looks at model responses, while the deployed router makes its choice before the response. On 141 MINT tasks, it gets 87.94% accuracy at 0.199 relative cost, versus 85.11% at 0.788 for MTRouter.

Critical strengths

RIVET is much cheaper than MTRouter across all four tests, with similar or better accuracy. The paper also tests whether the training-only head actually needs to read the response.

Critical weaknesses
  1. The MINT gain is just four tasks: 124 versus 120, from a single random seed. There is no variance estimate. Figure 3 does not clearly support the claimed cost trend. Three of four online checkpoints do worse than the offline model, and the final one is chosen using just 35 validation tasks.
  2. The fixed-model comparison is incomplete. Avengers-Pro gets 85.82% at 0.133 cost; RIVET gets 87.94% at 0.199. That is about 50% more cost for three additional tasks. The authors already ran six fixed models but report only the most expensive one in Table 1. Baseline training data and access to the 140 online trajectories are unclear.
Questions for authors
  1. My main issue is: Mean and spread over at least five seeds for the RIVET rows of Table 1, MTRouter, the Table 2 rows and the Figure 3 sweep.
  2. We miss the experiments to run all 6 fixed model policies on the 4 test sets.
Reviewer_Mo7aOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RIVET, a method for cost-aware LLM routing in multi-turn interactions. The key idea is to use generated responses as additional information during training to improve pre-call routing decisions. RIVET jointly trains two value estimators with a shared state-model representation: RVE predicts the expected return of selecting a candidate model, while RIVE additionally conditions on the generated response. Only RVE is used at inference time.

The training procedure combines Monte Carlo initialization, off-policy value learning with Double DQN and CQL, and limited online adaptation. The paper also provides a theoretical analysis showing that response-informed auxiliary learning can reduce the estimation error of the shared representation under certain assumptions.

Experiments on MINT and three OOD benchmarks demonstrate promising accuracy-cost trade-offs. On MINT, RIVET achieves 87.94% accuracy with 74.7% lower model-call cost than MTRouter. Ablation studies further show that removing RIVE or its response input leads to a noticeable performance drop.

Critical strengths
  1. A well-motivated approach to response-informed value learning. Multi-turn routing suffers from delayed task-level feedback, making it difficult to learn the value of individual model-selection decisions. RIVET uses responses already available in logged trajectories as an additional training signal, without requiring extra model calls at inference time. The shared representation between RVE and RIVE provides a reasonable way to transfer this information to pre-call routing.

  2. Promising accuracy-cost improvements with supporting ablations. RIVET substantially reduces model-call cost compared with MTRouter while achieving slightly higher accuracy on MINT. The cost advantage also holds across the evaluated OOD benchmarks. The ablation results are particularly useful: removing RIVE reduces MINT accuracy from 87.94% to 80.14%, and removing its response input produces the same accuracy drop. These results support the effectiveness of response-informed auxiliary learning in the evaluated setting.

Critical weaknesses
  1. The accuracy-cost trade-off is not fully characterized. The paper reports substantial cost reductions, but the comparisons are largely based on individual operating points. For example, Avengers-Pro achieves 85.82% accuracy at a lower cost (0.133) than RIVET (87.94%, 0.199) on MINT. While RIVET achieves higher accuracy, it is unclear whether it consistently provides a better trade-off across different cost budgets. A more complete accuracy-cost frontier comparison would strengthen the main empirical claim.

  2. The experimental results lack robustness analysis. The MINT test set contains only 141 tasks, and the accuracy improvement over MTRouter (85.11% to 87.94%) corresponds to four additional successful tasks. Appendix B.1 reports a single random seed, and the online adaptation results are based on one run. Although the cost reductions are substantial, results across multiple seeds would help establish whether the improvements are stable, particularly given the relatively small evaluation set.

Questions for authors
  1. Could the authors provide a more complete accuracy-cost frontier comparison, or compare the baselines under matched cost or accuracy constraints? This would help clarify whether RIVET consistently improves the trade-off rather than achieving a different operating point.

  2. Could the authors report results across multiple random seeds, including variance or confidence intervals for accuracy and cost? This would help assess the stability of the reported improvements.

  3. Could the authors provide a breakdown of the cost savings in terms of model selection, number of turns per episode, and token usage? This would help explain which factors contribute most to the reported cost reduction.

  4. Can the authors further isolate which aspects of the response contribute to RIVE's improvement? For example, would simpler response-level features such as length or token cost provide similar benefits to the full response embedding?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.