- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
If you want to benchmark a model in tool use, you will roughly want to split it into: if the model can tell when it has to use a tool vs if the model is too reluctant or eager to use it. The paper builds a test set for this where each question asks whether a tool is a wise choice in a physical situation (such as a humidifier in a very dry or a very wet room), with the right answer worked out by some fixed rule. Across 44 open models, larger models within a family are better at telling the cases apart, and most models are too reluctant to say yes. The authors then push models toward acting, by steering their activations and by training each model to copy its steered version, which makes them act more on other tool-use benchmarks without making them any better at telling the cases apart.
Critical strengths
- It seems like a good idea to have a criterion comparable across benchmarks, and to try to use convergent evidence such as from steering or graded evidence.
Critical weaknesses
- One big piece of feedback is that a main selling point is a criterion comparable across benchmarks, but it doesn’t carry over to other benchmarks: FLIPACT’s criterion is uncorrelated with the criteria measured on When2Call, QuestBench and When2Tool. The authors reframe it as “a stable reading of each model on FLIPACT rather than a trait,” but this importantly concedes the comparability that motivated the fixed scale!
- I found the prose very confusing, and hard to parse. The abstract, for example, reads as entirely or almost entirely AI generated.
Questions for authors
NA