- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes Tool Vector, an inference-time activation steering method to predict and intervene an LLM’s tool choice before tool-call generation. It constructs tool prototype vectors by averaging final-layer hidden states from a small set of demonstrations, then steers activations using the difference between two tool vectors without fine-tuning. The method achieves 56–78% steering success on clean tool pairs but degrades substantially on ToolBench and T-Bench. A central open question is whether the effect arises from semantic representation shifts or is merely equivalent to logit bias.
Critical strengths
- Novel perspective: it predicts and intervenes tool choice before decoding, different from most post-hoc correction methods.
- Lightweight: training-free, only vector addition at inference time with low overhead.
- Multi-model evaluation: tested across a range of instruction-tuned models from 4B to 27B.
Critical weaknesses
- Weak mechanistic evidence. Head-to-head comparison against logit bias is missing, so semantic steering cannot be distinguished from token-level bias.
- Flawed evaluation: only measuring whether the model switches to the target tool, without checking whether the resulting tool call matches the query, enabling invalid forced steering.
- Missing ablations and validation: the core clustering assumption is untested; ablations over demo count, steering strength and token positions are absent; failure modes on real benchmarks are not analyzed.
Questions for authors
Same as weakness