- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies meta-reasoning and safety associated with tool calls. The authors define "tool-admission" as the decision of a model to invoke a tool, instead of answering from its memory. From the lens of safety, the authors attempt to study what pieces of an unsafe input cause a model to call a tool. To this end, they decompose the request into its operation, topic, and modifier. To understand why the model made a decision (to call or not call), the authors also present a mechanistic study by analysing the alignment request-operation direction, text-refusal direction, and admission-outcome direction.
Critical strengths
- Interesting problem statement, and worth studying.
Critical weaknesses
- The paper is very badly written and reeks of AI at this point, with some technical jargon and adjectives. The AI generated contents seems to dominate, especially toward the end of the paper. The sentences and paragraphs are super disconnected, and I fail to see a logical flow. Due to this, I am very unclear about the contributions of the paper, why they are performing the experiments they mentioned, and what their true findings. Similarly, it's also very unclear what the takeaways of this work are, and the contributions mentioned at the end of the introduction don't seem to fit. Overall, I find no causal flow in the work, and I suggest a full, proper rewrite of the paper for a better presentation at this point.
- For instance, the "factorial" technique is quite unclear, even to the point of why it is named so, or what it is testing. The title also has a heavy/overclaiming adjective "Mastering", but I don't see any formal proof, or a one-shot solution to tool admission. I urge that word be toned down. Also, for example, it's hard to understand what Eq 2 physically represents.
- The analysis just seems to be limited to just URL-based web search. It would be good to try other tools as well. So, I feel the scope of the paper is a bit limited in that way.
- I also believe multiple papers deal with tool call reasoning/meta-reasoning, so how is your work different from them, and what extra do you provide beyond them? I ask this as I am unsure about the novelty of the studied problem.
Questions for authors
- Why was the GPT-4o judgement on a scale of 1-5 and cut at 3? Why not use a binary decision? won't a single binary decision also help the model as be more specific?