acceptodds
Withdrawn from ICLR 2027

R2V Agent: Teaching SLMs When to Ask for Help

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

135 views

Abstract

Efficient agentic systems should incur expensive frontier-model costs only on decisions where a cheaper local model is likely to fail. Existing LLM cascades usually route whole queries before execution, but task difficulty shifts mid-trajectory - after flaky tool calls, truncated observations, or compounding local errors - making pre-execution routing brittle. We introduce R2V-Agent, a risk-calibrated SLM-LLM routing framework for interactive agents. R2V combines four components: a distilled small language model (SLM) policy, a stronger teacher LLM, a lightweight process verifier that scores candidate actions at each step, and a calibrated step-level router. The router is our central contribution: after the SLM is trained, it estimates residual failure risk at each step and escalates only when teacher intervention is warranted. To make the routing problem well-defined, we first train a stable local SLM using a standard offline pipeline: behavioral cloning (BC) on teacher trajectories, followed by verifier-guided Direct Preference Optimization (DPO) with consistency regularization. The router is then trained on this fixed policy's residual failures using Brier-calibrated probability estimation and a Conditional Value-at-Risk (CVaR)-constrained objective that penalizes worst-case failures across perturbation seeds. Across HumanEval+, TextWorld, and TerminalBench with four SLM backbones, R2V improves the reliability-cost frontier: it achieves HumanEval+ success with LLM escalation, recovers TextWorld from SLM-only success to at escalation, and reaches TerminalBench success at LLM calls, roughly half the heuristic-router cost.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
tool use, planning in agents, environment interaction, LLM-based controllers, agent evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GLbBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces R2V-Agent, a risk-calibrated step-level routing framework for interactive LLM agents. Unlike traditional query-level routers that assign an entire task to a single model, R2V operates mid-trajectory, allowing a cheap, local Small Language Model (SLM) to execute routine steps while escalating to an expensive frontier LLM only when the estimated residual failure risk is high. The framework consists of a fixed, distilled SLM (trained via BC and verifier-guided DPO), a local process verifier, and an MLP-based router. The core contribution is the router, which is trained using Brier calibration and a CVaR-constrained objective to manage tail risks across perturbation seeds.

Critical strengths
  • Insightful Abstraction of Oracle vs. Heuristic Efficiency: The paper beautifully illustrates the necessity of fine-grained, step-level routing. For instance, on TerminalBench, the Oracle requires only 18.4% LLM escalation to achieve 97.8% success, proving that SLMs are perfectly capable of handling the vast majority (>80%) of agentic steps independently. This strongly validates the core motivation that monolithic LLM execution or heuristic escalations (66.0% calls) are highly wasteful.
  • Rigorous Risk-Calibrated Formulation: The introduction of Brier calibration combined with a CVaR-constrained objective to train the router is mathematically sound. It effectively prevents the system from learning a policy that is merely cheap on average but catastrophically brittle under edge-case perturbations.
Critical weaknesses
  • Ignored Latency and Token Inflation Costs (Wall-clock Reality): The paper claims cost efficiency purely based on a simplified formula (SLM=1,LLM=50SLM=1, LLM=50). However, step-level routing requires the SLM to sample K=5K=5 candidates, run a local verifier, and execute an MLP at every single step. This introduces massive wall-clock latency. Furthermore, if the SLM flails for several steps before escalating, the accumulated error logs will severely inflate the context window, driving up the actual API token costs for the Teacher LLM. This practical overhead is entirely unquantified.
  • Imbalanced Writing and Presentation Focus: The paper spends a disproportionate amount of space (Section 3.1 and Appendix A) detailing the Behavioral Cloning (BC) and verifier-guided DPO processes. As the authors admit, this is a standard offline pipeline used merely to stabilize the base SLM. This distracts heavily from the actual innovation: the router itself, which is ultimately just a lightweight MLP. The presentation should shift focus to the step-level risk features and routing dynamics.
  • Missing Comparisons with Recent Step/Token-Level Routers: The authors contrast their work primarily against query-level routers (e.g., RouteLLM, FrugalGPT). However, the concept of fine-grained routing is rapidly evolving. Recent contemporary works (e.g., TwinRouterBench[1] for step-level multi-turn agent routing, or R2R[2] at NeurIPS 2026) have already introduced step/token-level routing concepts. The authors must explicitly differentiate their CVaR-calibrated approach from these highly relevant, fine-grained routing counterparts to establish novelty.

[1] arXiv:2605.18859

[2] T. Fu, Y. Ge, Y. You, E. Liu, Z. Yuan, G. Dai, S. Yan, H. Yang, and Y. Wang, “R2R: Efficiently navigating divergent reasoning paths with small-large model token routing,” in Advances in Neural Information Processing Systems, vol. 38, Main Conference, 2025, pp. 124108–124145, doi: 10.52202/085713-4135.

Questions for authors
  1. Wall-clock Latency and Token Overhead: Your cost analysis only evaluates the binary choice of invoking the LLM vs. SLM. What is the actual wall-clock latency overhead introduced by sampling K=5K=5 candidates, running the verifier, and executing the router at every step? Additionally, does deferring to the LLM mid-trajectory result in inflated API costs due to the accumulation of lengthy, SLM-generated error contexts?
  2. Context Handoff Mechanism: When the router escalates to the Teacher LLM, do you use specific prompt templates to seamlessly "hand off" the task, or do you simply append the SLM's trajectory to the context window? Does the Teacher LLM ever get confused by the SLM's prior suboptimal reasoning steps?
  3. Comparison with Recent Fine-Grained Routers: While your critique of query-level routers is well-founded, recent works also explore step-level and token-level LLM routing. Could you explicitly discuss how your risk-calibrated, stateful routing differs from or improves upon these contemporary fine-grained routing methodologies?
Reviewer_kV1YOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a routing system for multi-step LLM tasks where steps are taken by a small LM by default but routed to a large LM when needed. The router is a classifier trained on 15 features, optimized to predict whether the small LM is able to successfully complete the task at the current point. Evaluation is performed on 3 benchmarks, showing a performance-cost trade-off.

Critical strengths

The core idea is interesting; routing specific steps in a trajectory could potentially achieve a better cost-accuracy balance.

Critical weaknesses

The main proposed hypothesis of the paper is that routing specific steps to the LLM is better and more cost-effective than routing whole tasks. But this hypothesis is never investigated or evaluated. There is quite a lot of existing work on estimating the difficulty and routing whole tasks. Some of these need to be included as baselines, in order to determine whether the task-specific routing actually outperforms them. Comparing against doing nothing or against trivial baselines is not really a fair comparison.

The Brier + CVaR training on the classifier is emphasised as a contribution but there is no evaluation to show its benefit. There should be a baseline without CVaR and a baseline that just trains logistic regression on the same 15 features.

The result numbers are suspicious. For example, the highest recorded result according to the official Terminal Bench 2.0 leaderboard is currently 84.7%. The paper reports the LLM-only baseline as 98.1 and even the Heuristic Router baseline at 95%.

Measuring the performance in steps and assuming an LLM step is 50x more expensive than SLM step is quite arbitrary. Real-life LLM work is measured and charged in tokens. The need to encode context when switching to a different LM, running various verifier tests after each step and considering K options at every step likely add quite a substantial overhead. Please provide generated token counts and wall clock measurements to assess efficiency.

The 15 hand-specified features are not sufficiently described at the moment. Given that they are the core to the method, they should each be fully listed somewhere, at least in the appendix.

The hand-crafted verifiers seem to be doing a lot of heavy lifting, including in terms of choosing the next best step out of K options. There seems to be no evaluation that compares their impact on the performance, compared to the learned routing.

It is mentioned that for HumanEval+ the verifier runs a "smoke-test harness". It should be clarified what that actually is. Because if that involves running the grader then that will leak information from the grader and the benchmark.

Questions for authors

Please see above

Reviewer_oV99OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Step-level SLM→LLM router for agents. Pipeline: BC + verifier-guided DPO + JSD consistency → frozen SLM; then a 15-feature MLP router trained with Brier + CVaR on labels y_t = 1[episode failed] from SLM-only rollouts. Three benchmarks (HumanEval+, ALFWorld-as-“TextWorld”, Terminal-Bench 2.0), four 7–14B backbones. Claims router-only gains isolate compute allocation.

Critical strengths

Clean experimental separation: freezing the SLM and varying only the router is the right design for isolating allocation.

Critical weaknesses

SLM-only 81.1% and LLM-only 98.1% on Terminal-Bench 2.0 with 7–14B models and Gemini Flash — public TB2 leaderboards have top frontier agents well below that. The verifier “operates without Docker or subprocess execution” (B.2), and C.4 says “TerminalBench trajectories were unavailable for reconstruction.” Strongly suggests the “environment” is a replay of logged HF trajectories, in which case “success” isn’t task completion and the whole TB column is uninterpretable.

Questions for authors

Is TerminalBench executed live in containers, or replayed from logged trajectories? If replayed, what does “success” mean for an SLM action that diverges from the log?

What does the HumanEval+ test action execute — the hidden EvalPlus tests, a subset, or model-written tests?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.