acceptodds
Under review as a conference paper at ICLR 2027

Think Short, Defer Smart, Act, and Repeat: Calibrated Reasoning and Uncertainty-Aware Deferral for Edge LLM Agents

Abstract

LLM agents following the ReAct paradigm are promising enablers of complex multi-step tasks, including multi-hop question answering, code generation, and control of physical AI systems. Yet, when deployed at the edge, they must tightly manage their reasoning budget while remaining reliable and deferring to a cloud-side model only when local uncertainty is too high to act safely. We propose Think Short, Defer Smart (TSDS), a framework that synergistically integrates a lightweight convergence probe, which halts on-device reasoning once the intended action has stabilized, with a perplexity-based deferral rule that escalates uncertain actions to a cloud-side model. Both mechanisms are jointly calibrated on end-to-end episode trajectories via a multi-objective Learn-Then-Test (LTT) procedure, providing simultaneous finite-sample guarantees on expected episode reward and cloud-call rate. We evaluate TSDS on four ReAct benchmarks spanning arithmetic reasoning (GSM8K), multi-hop question answering (HotpotQA), code generation (MBPP), and multi-step embodied planning (household robot), and compare against thought-calibration-only and calibrated-deferral-only standalone baselines. TSDS reduces per-episode thinking compute by 43%-73% over deferral-only baselines across HotpotQA, MBPP, and the household robot task, while maintaining certified reward and cloud-call rate guarantees.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
LLM Agents, Edge AI, Uncertainty-aware deferral, Finite-sample calibration, Adaptive reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_S3YnOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework, Think Short, Defer Smart (TSDS), that uses a convergence probe to stop reasoning on an edge device early and a perplexity-based uncertainty score to selectively defer action decisions to a more powerful model in the cloud. They use Learn-Then-Test (LTT) to select convergence and deferral thresholds that certify expected reward above a target floor and expected deferral rate below a target budget. They evaluate the framework on code generation, multi-hop QA, simulated household planning, and arithmetic reasoning benchmarks, showing reduced thinking cost on the edge device.

Critical strengths
  1. The paper addresses a useful problem: reducing thinking on the edge while maintaining task performance and controlling cloud use. Combining early stopping with selective cloud deferral is a reasonable approach.
  2. The use of LTT is good and more convincing than simple grid search. It jointly certifies the reward floor and cloud-deferral budget, providing statistical guarantees for the selected thresholds.
  3. The paper has good benchmark coverage, including arithmetic reasoning, multi-hop question answering, code generation, and multi-step household planning.
  4. The convergence probe is clearly defined, and its performance is separately evaluated in the appendix. The comparison across probe architectures and layers helps support the choice of the EMA probe.
  5. Figure 1 is intuitive and well presented. It clearly illustrates the pipeline and the proposed interaction between early stopping and uncertainty-based deferral.
Critical weaknesses
  1. The claim that early stopping improves deferral routing by mitigating overconfidence is not sufficiently supported. The main comparisons use different calibration constraints and deferral rates, making it difficult to isolate the effect of early stopping. In Figure 3, even at comparable edge thinking costs, higher reward could reflect greater cloud use. Overall, the comparisons are not clearly presented, making it difficult to distinguish the contributions of the individual components of the framework.
  2. Using PPL as an uncertainty proxy is reasonable, but textual uncertainty should not be assumed to imply decision uncertainty. Further validation should show that high-PPL actions are more likely to be wrong and benefit more from cloud deferral.
  3. The efficiency evaluation counts only edge reasoning tokens, excluding cloud computation and probing overhead. This limits conclusions about overall computational savings.
  4. The baseline reporting is incomplete: ReDAct is defined but omitted from the main results, and Figure 2 hides some uncertified baselines’ empirical performance, making the comparisons harder to assess.
  5. The contribution is mainly combining existing thought calibration and PPL-based deferral with joint LTT certification. The added methodological novelty appears limited, although this is not my main concern.
Questions for authors
  1. Does early stopping improve deferral routing when the methods are compared under the same calibration constraints and at matched deferral rates?
  2. Are high-PPL actions more likely to be incorrect, and does deferring them improve performance more than deferring low-PPL actions?
  3. How much computation does TSDS save when cloud reasoning and probing overhead are also included?
  4. How is the target deferral budget chosen, and how does changing it affect the selected thresholds and task performance?
Reviewer_xquHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Think Short, Defer Smart (TSDS), which integrates a convergence probe for halting local reasoning with an action-level uncertainty score for deciding whether to defer to a cloud model. The stopping and deferral thresholds are jointly calibrated on end-to-end episode trajectories via a multi-objective Learn-Then-Test (LTT) procedure, providing simultaneous finite-sample guarantees on expected episode reward and cloud-call rate.

Critical strengths

The proposed TSDS may reduce the number of tokens used at the edge.

The idea on the trade-off between local reasoning and cloud deferral is interesting.

Critical weaknesses

The novelty of the method is limited, and the contribution of this paper is mainly an integration of existing methods, and its generality needs stronger evidence.

More related work should be discussed.

Several results provided in the paper are inconsistent which undermine confidence in the evaluation.

  1. The values for the average deferral rate for the MBPP ReDAct-CD in Figure2 and Table 3 are not consistent (around 0.7 in Figure 2 and 0.786 in Table 3).
  2. Table 2 and Table 4 provide different results for TSDS and edge model without deferral.
  3. In Appendix F.6, three fixed budgets multiplied by five deferral thresholds provide 15 pairs, but the stated grid size is 20.

For the multi-object calibration, a finite grid is constructed in the paper. This is too simple, and not optimal.

Questions for authors

What is the criteria to set up the size of the finite grid?

Why do Table 2 and Table 4 provide different results for TSDS and edge model without deferral for what is described as the same GSM8K setup?

What happens with fixed 16- and 32-token budgets, using independently optimized deferral thresholds?

How often is the certified set empty? In this case, the certified set may violate the constraints.

Reviewer_C2FYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TSDS, which jointly calibrates early stopping of reasoning by an edge model and uncertainty-based deferral to the cloud. It reduces reasoning tokens on the edge while meeting requirements on task reward and the cloud-call rate. Experiments cover mathematics, coding, multi-hop question answering, and household tasks in a text-based simulator, demonstrating computational savings from combining the two mechanisms.

Critical strengths

1.Jointly considering when to stop reasoning and when to defer to the cloud addresses a practical problem. The paper also recognizes that early truncation can affect subsequent uncertainty estimates. 2.Calibrating parameters on complete task trajectories and comparing configurations with only early stopping or only cloud deferral helps clarify the roles of the two mechanisms (Sections 4 and 5).

Critical weaknesses

1.The need for the probe over simple fixed truncation is not sufficiently established. In Appendix F.6, TSDS uses only 17.4 reasoning tokens on average, but the fixed-truncation baselines use budgets of 48, 80, and 112 tokens, leaving no comparison at a similar cost. Moreover, in Table 4, fixed truncation at 80 tokens achieves a reward of 0.933, higher than TSDS's 0.913, which conflicts with the subsequent claim of matching the best reward.

2.The scope of the guarantees needs to be corrected. In Section 4.3, when no parameter setting passes calibration, the method falls back to E-ReAct without cloud calls. This ensures only the cloud-call rate constraint; it does not automatically ensure the reward lower bound. Proposition 1 should explicitly distinguish between selecting a qualifying parameter setting and triggering the fallback.

3.The efficiency evidence mainly concerns the number of reasoning tokens on the edge. Appendix F.4 explicitly excludes cloud-generated tokens, and the paper does not report actual latency or energy consumption including action generation, the probe, and communication overhead. The deployment benefits of these token savings therefore remain unclear.

Questions for authors

1.What advantages does the probe retain over fixed truncation at a budget close to TSDS's average reasoning cost? Do these advantages also hold on multi-step tasks such as HotpotQA?

2.Could you report end-to-end task latency for a representative edge configuration, with a breakdown of local inference, cloud inference, and communication time?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.