- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies when agents should use external tools and when they should rely on their own reasoning. It introduces ToA-Eval, an evaluation protocol that compares each model’s answers without tools and with forced tool use, identifying cases where tools help or harm answer correctness. The proposed method, ToA-Steer, extracts a tool-use direction from 50 pairs of contrasting responses. It then groups inputs by their internal representations and uses a development set to choose how strongly to encourage or discourage tool use for each group, without updating the model’s parameters. Across four models and four reasoning benchmarks, the authors report the highest accuracy in all settings.
Critical strengths
A simple control method. ToA-Steer uses the model’s internal representations to encourage or discourage tool use for different groups of inputs, without updating the model’s weights. The paper reports the highest accuracy across all model–benchmark settings.
Critical weaknesses
- Limited evidence of robustness to new settings. The controller assigns new inputs to existing clusters and reuses their calibrated steering strengths. This assumes that similar internal representations imply similar tool-use needs, which may not hold when question types, difficulty, or tool reliability change. An unfamiliar input is still assigned to the nearest cluster, even if it is far from every cluster, and may receive harmful steering. The paper provides no explicit fallback for this situation. Although the web-search transfer experiment is encouraging, it covers only one model and one target environment. This limits confidence in the method’s reliability beyond its calibration settings.
- Unclear data construction. The paper does not sufficiently explain the sources of the contrastive queries or report the size of each data split, particularly for AIME. This makes the reported sample counts difficult to reconcile and limits reproducibility in the evaluation.
Questions for authors
Could you clarify where the 50 contrastive pairs come from and how many examples are in each data split?
5 comments
Response to Reviewer f7rD Part B
How the controller behaves when the setting changes (W1)
We thank the reviewer for naming the three kinds of change. Without re-calibration, ToA-Steer stays above Base under all three shifts, and the web-search transfer now holds on a second model. ToA-Steer is calibrated per setting by design. To see what happens without re-calibration, we calibrate on one setting and evaluate on another for each of the three changes. “Transferred” applies the source calibration unchanged; “reference” is the controller calibrated on the target itself and equals Table 2 where applicable.
Table F2a: Change in question type (MATH / Python → GPQA / Wikipedia)
| Model | Base Acc | ToA-Steer Acc (transferred) | Δ | Reference |
|---|---|---|---|---|
| Qwen3-8B | 0.58 ± 0.05 | 0.64 ± 0.06 | +0.06 | 0.74 ± 0.05 |
| Qwen3.5-27B | 0.62 ± 0.07 | 0.64 ± 0.08 | +0.02 | 0.72 ± 0.07 |
| Gemma3-12B | 0.55 ± 0.05 | 0.60 ± 0.06 | +0.05 | 0.69 ± 0.05 |
| Gemma4-31B | 0.65 ± 0.07 | 0.67 ± 0.07 | +0.02 | 0.73 ± 0.06 |
Table F2b: Change in difficulty (MATH → AIME)
| Model | Base Acc | ToA-Steer Acc (transferred) | Δ | Reference |
|---|---|---|---|---|
| Qwen3-8B | 0.38 ± 0.09 | 0.50 ± 0.11 | +0.12 | 0.62 ± 0.10 |
| Qwen3.5-27B | 0.45 ± 0.15 | 0.48 ± 0.16 | +0.03 | 0.59 ± 0.14 |
| Gemma3-12B | 0.42 ± 0.09 | 0.51 ± 0.10 | +0.09 | 0.62 ± 0.09 |
| Gemma4-31B | 0.48 ± 0.18 | 0.52 ± 0.19 | +0.04 | 0.60 ± 0.17 |
Table F2c: Change in tool reliability (GPQA / Wikipedia → BFCL V4 Web Search, 50% of page fetches fail)
| Model | Base | GRPO | ToA-Steer (transferred) | ToA − Base |
|---|---|---|---|---|
| Qwen3.5-27B (paper) | 31.0 ± 3.0 | 34.9 ± 3.1 | 36.0 ± 2.7 | +5.0 |
| Qwen3-8B (new) | 22.4 ± 3.4 | 26.1 ± 3.6 | 26.9 ± 3.2 | +4.5 |
Tables F2a–b: accuracy on the ToA-Eval test split of the target, mean ± SD over five runs.
Key findings.
- Transfer without re-calibration stays above Base under all three changes: +0.05 to +0.12 for the 8B and 12B models, +0.02 to +0.04 for the 27B and 31B models whose in-domain gains are also smaller.
- The live-environment result holds on a second model: under 50% fetch failure, Qwen3-8B gains +4.5 over Base and matches GRPO within one SD, the same pattern as Qwen3.5-27B.
- Per-setting calibration is still preferable: the target-calibrated reference is higher in every case.
Overall, the controller degrades gracefully rather than harmfully when the setting moves away from its calibration.
Revision. Add Tables F2a–c to Appendix O with performed analysis.
Response to Reviewer f7rD Part C
Where the contrastive queries come from, and the size of each split (Q1, W2)
We thank the reviewer for this request. The contrastive queries are unlabeled questions outside Test, so they never overlap the labeled pool, and the counts reconcile even on AIME. The full split sizes are below.
1. Contrastive pairs and the router
Each pair is one question with two fixed template completions: “Use [tools] to address this query.” (ACT) and “Answer directly from your own knowledge.” (THINK). We use 50 questions per benchmark, taken from the source questions that are not in Test. The same kind of unlabeled questions form the Router-train set on which PCA and K-means are fit. Only the development set and Test are drawn from the labeled pool. For AIME, the labeled pool uses AIME 2024–2026 (90 problems); the 50 pair queries and the N > 50 points in Fig. 8 use AIME 2000–2023.
Small pools use 5-fold cross-validation: where the labeled pool has at least 50 instances, Dev is 25 ACT + 25 THINK and Test is the rest. Where it has fewer, a single split would leave a test set of a few questions, so we calibrate with 5-fold cross-validation: each fold is held out in turn, the other four form Dev, and the held-out predictions are pooled. Every labeled instance is then a test instance exactly once.
| Model | Benchmark | Labeled pool | Vector-train | Router-train | Dev | Test |
|---|---|---|---|---|---|---|
| Qwen3-8B | MATH | 309 | 50 | 50 | 50 | 259 |
| Qwen3-8B | AIME | 30 | 50 | 50 | 24* | 30* |
| Qwen3-8B | GPQA | 84 | 50 | 50 | 50 | 34 |
| Qwen3-8B | MMLU-Pro | 627 | 50 | 50 | 50 | 577 |
| Qwen3.5-27B | MATH | 415 | 50 | 50 | 50 | 365 |
| Qwen3.5-27B | AIME | 14 | 50 | 50 | 11* | 14* |
| Qwen3.5-27B | GPQA | 42 | 50 | 50 | 34* | 42* |
| Qwen3.5-27B | MMLU-Pro | 555 | 50 | 50 | 50 | 505 |
| Gemma3-12B | MATH | 315 | 50 | 50 | 50 | 265 |
| Gemma3-12B | AIME | 34 | 50 | 50 | 27* | 34* |
| Gemma3-12B | GPQA | 95 | 50 | 50 | 50 | 45 |
| Gemma3-12B | MMLU-Pro | 650 | 50 | 50 | 50 | 600 |
| Gemma4-31B | MATH | 365 | 50 | 50 | 50 | 315 |
| Gemma4-31B | AIME | 12 | 50 | 50 | 10* | 12* |
| Gemma4-31B | GPQA | 45 | 50 | 50 | 36* | 45* |
| Gemma4-31B | MMLU-Pro | 623 | 50 | 50 | 50 | 573 |
Table F3: Unique question counts. Vector-train and Router-train are unlabeled source questions outside Test. *5-fold cross-validated calibration: Dev is the size of the four calibration folds, Test is the whole labeled pool with each instance held out once.
Revision. Add Table F3 and the cross-validation protocol to Appendix G; rewrite the split description there (two unlabeled splits from source questions, Dev and Test from the labeled pool); correct the AIME year ranges in Section 2 and Appendix E; state the benchmark versions used; mark the cross-validated cells in Table 2.
Response to Reviewer f7rD Part A
What happens to inputs outside the calibrated regions (W1)
We thank the reviewer for this concern. Inputs far from every cluster receive no harmful steering, and setting m = 0 for them gives an explicit fallback.
We call a test input “outside” if its distance to the nearest centroid is larger than that of 95% of the calibration data, and compare Base and ToA-Steer on the two groups.
Table F1: Qwen3-8B, ToA-Eval test split, four benchmarks pooled, mean ± SD over five runs.
Key findings.
Overall, inputs the controller has not seen the like of receive little benefit but no damage.
Revision. Add Table F1 and the m = 0 option to Section 3.2.