- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework, Think Short, Defer Smart (TSDS), that uses a convergence probe to stop reasoning on an edge device early and a perplexity-based uncertainty score to selectively defer action decisions to a more powerful model in the cloud. They use Learn-Then-Test (LTT) to select convergence and deferral thresholds that certify expected reward above a target floor and expected deferral rate below a target budget. They evaluate the framework on code generation, multi-hop QA, simulated household planning, and arithmetic reasoning benchmarks, showing reduced thinking cost on the edge device.
Critical strengths
- The paper addresses a useful problem: reducing thinking on the edge while maintaining task performance and controlling cloud use. Combining early stopping with selective cloud deferral is a reasonable approach.
- The use of LTT is good and more convincing than simple grid search. It jointly certifies the reward floor and cloud-deferral budget, providing statistical guarantees for the selected thresholds.
- The paper has good benchmark coverage, including arithmetic reasoning, multi-hop question answering, code generation, and multi-step household planning.
- The convergence probe is clearly defined, and its performance is separately evaluated in the appendix. The comparison across probe architectures and layers helps support the choice of the EMA probe.
- Figure 1 is intuitive and well presented. It clearly illustrates the pipeline and the proposed interaction between early stopping and uncertainty-based deferral.
Critical weaknesses
- The claim that early stopping improves deferral routing by mitigating overconfidence is not sufficiently supported. The main comparisons use different calibration constraints and deferral rates, making it difficult to isolate the effect of early stopping. In Figure 3, even at comparable edge thinking costs, higher reward could reflect greater cloud use. Overall, the comparisons are not clearly presented, making it difficult to distinguish the contributions of the individual components of the framework.
- Using PPL as an uncertainty proxy is reasonable, but textual uncertainty should not be assumed to imply decision uncertainty. Further validation should show that high-PPL actions are more likely to be wrong and benefit more from cloud deferral.
- The efficiency evaluation counts only edge reasoning tokens, excluding cloud computation and probing overhead. This limits conclusions about overall computational savings.
- The baseline reporting is incomplete: ReDAct is defined but omitted from the main results, and Figure 2 hides some uncertified baselines’ empirical performance, making the comparisons harder to assess.
- The contribution is mainly combining existing thought calibration and PPL-based deferral with joint LTT certification. The added methodological novelty appears limited, although this is not my main concern.
Questions for authors
- Does early stopping improve deferral routing when the methods are compared under the same calibration constraints and at matched deferral rates?
- Are high-PPL actions more likely to be incorrect, and does deferring them improve performance more than deferring low-PPL actions?
- How much computation does TSDS save when cloud reasoning and probing overhead are also included?
- How is the target deferral budget chosen, and how does changing it affect the selected thresholds and task performance?