- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces Nudgeability, a framework for measuring how confidence and doubt signals affect tool delegation in reasoning models. The main finding is that such signals strongly change delegation behavior, but the induced changes are only weakly aligned with whether the model actually needs help.
Critical strengths
- The focused question is unusual.
- The motivation is clear.
- It provides empirical evidence.
Critical weaknesses
- The practical significance is not sufficiently demonstrated. The paper motivates Nudgeability through realistic tool-routing scenarios, but the experiment only studies an externally injected confidence sentence followed by a SELF/TOOL decision. The tool is never actually executed, so the study does not show whether delegation improves final accuracy, utility, latency, or cost. The confidence signal is also artificial rather than generated by an actual uncertainty estimator or self-reflection mechanism.
- The paper provides limited mechanistic insight. The experiments establish that confidence language changes delegation behavior, but do not explain why this happens, why sensitivity and targeting are weakly related, or why different models show substantially different behavior. The contribution is therefore primarily behavioral rather than mechanistic.
- Several main findings are intuitive or largely descriptive. It is unsurprising that explicitly expressing doubt increases tool use while expressing confidence decreases it. The more interesting observation is that high sensitivity does not imply competence-aware targeting, but the paper mainly documents this phenomenon without providing a deeper explanation for why it arises.
- Generalization to realistic agent settings remains unclear. The experiments cover only two QA tasks with search- or retrieval-style tools and a specific natural-language intervention at a particular reasoning boundary. It is unclear whether the findings extend to coding, planning, multi-step tool use, long-horizon agents, or realistic environments involving heterogeneous tool costs and failures.
- Evidence from frontier systems is still limited. Although the paper includes two larger provider-served models, most of the evaluation focuses on smaller open-weight systems. Broader evaluation on contemporary frontier models would be needed to establish that the sensitivity–targeting pattern is general.
Questions for authors
- What pratical value will the paper provide?
- Do the authors have any machanistic insight beyond empirical observation?
- Do the findings generalize beyond QA and search settings?
- Can the authors evaluate a broader set of frontier reasoning models?
3 comments
(Continued from the previous comment.)
Generalization beyond QA and search.
We agree; our tasks are multi-hop QA (MuSiQue) and StrategyQA with a search-style tool, and the revision states this scope explicitly. The measurement itself is task-agnostic, since it needs only a binary delegation decision and a competence label. We thank the reviewer and list code-execution and calculator tools as next settings.
More frontier models.
We thank the reviewer for this suggestion, and we would have liked to include more frontier systems. The constraint is the intervention itself, not cost. Nudgeability requires taking the model's own reasoning prefix, splicing a sentence into it at a fixed point, and having the model continue from the edited trace. The closed frontier APIs do not permit this. As noted in the introduction (Section 1), OpenAI, Google (Gemini) and Anthropic (Claude) do not return the full raw reasoning trace, returning summaries or encrypted content instead. Any reasoning that is passed back is encrypted or cryptographically signed and validated, so an edited trace cannot be submitted for continuation. Open-weight frontier models served by third-party providers are the only remaining route, and even there, access is uneven. Before running the provider cohort, we probed each candidate model and provider for byte-exact delivery of the edited prefix, continuation without restarting the reasoning, and deterministic replay. Many providers do not offer raw completions, or they silently prepend tokens to the supplied prefix. GLM-5.2 restarted its reasoning instead of continuing the edited trace, and Kimi-K3 passed the structural checks but not determinism. DeepSeek-V4-Pro (CoreWeave) and MiniMax-M3 (Together) were the large models whose providers passed all checks. That is why the provider cohort is small.
The two provider models nonetheless show that the pattern holds at a larger scale: high sensitivity (53–70 pp swings), near-random targeting, and the largest oracle gains (+47.4 pp median lift with a perfect estimator). The revision states the access constraint explicitly in the limitations (Section 5) and documents the route checks in Appendix I.2. Should closed providers expose editable reasoning in the future, the same measurement applies without change.
Thank you for the detailed response and additional analyses. I appreciate the new estimator-quality experiments and the clarification of the framework's scope. However, my main concerns remain. The reported improvements are largely based on simulated delegation decisions, without demonstrating actual downstream benefits. The findings also remain primarily descriptive, with limited mechanistic insight and unclear generalization beyond QA settings. Therefore, I will maintain my original rating.
Thank you for the review and for the suggestions on practical evaluation, generalization and frontier coverage. We respond to each concern.
Not executing the tool is deliberate. It isolates the delegation decision from the quality of any particular tool. The paper also includes a supporting condition in which the scoped tool is replaced by a free oracle that the prompt describes as always correct (Section 3, Appendix I.8). If the tool is correct when called, the decision maps directly onto outcomes. A well-targeted switch to TOOL on a failing problem converts an error into a correct answer. A well-targeted switch to SELF on a solvable problem saves a call at no accuracy cost.
The new oracle analysis turns this into concrete guidance for practitioners (General Response §1–3):
This tells a system designer which models' trace channels are worth connecting an estimator to, and how good that estimator must be.
This is also intentional. The exogenous signal is what lets us measure the model's response separately from estimator quality. The new estimator-quality analysis then connects the measurement to real estimators: lift is computed exactly from an estimator's hit and false-alarm rates, so any estimator with a known operating point can be evaluated. Nor is the framework tied to natural-language signals. A hidden-state correctness predictor such as Gnosis (Ghasemabadi & Niu, 2025), or a learned confidence token, can be evaluated on exactly the same footing (General Response §4). As a first check, three simple label-free signals read from the trace (AUROC about 0.6) realise a median lift of +4.9 to +7.7 pp against +2.0 pp, matching the lift the analysis predicts from their hit and false-alarm rates (Appendix G).
Our contribution is a measurement framework, and we agree that explaining why models respond as they do is the important next step. We thank the reviewer for pushing on it. The new analysis does add one structural insight that the descriptive results did not show: the model's own selectivity, not the channel, is what fails. The same paired continuations become well targeted when the signal is, and the remaining loss is attributable to sensitivity. Explaining that sensitivity is listed as future work: through internal representations, matched checkpoints before and after RL, graded signal intensity, and independently annotated analysis of how the continuations refer to the inserted sentence.
[continued]