- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a method for preventing LLM agents from using unsupported identifiers in tool calls. The authors argue that existing constrained decoding methods can either allow unsupported values or restrict valid tool calls, leading to lower task success rates. NoWit allows agents to generate tool calls freely and then checks whether the referential arguments have appeared in previous observations, such as user messages, tool responses, or documentation. If an argument is not supported, the tool call is blocked, and the agent is guided to retrieve more information or ask for clarification. The results show that NoWit reduces unsupported tool arguments and state-changing calls compared with unconstrained generation, while maintaining comparable task performance.
Critical strengths
1.The paper addresses an important issue in tool-using agents, where models may generate unsupported identifiers that lead to incorrect tool calls.
2.The paper compares different intervention strategies and evaluates NoWit across multiple seeds, model sizes, and environments. The results show consistent reductions in unsupported tool arguments while maintaining comparable task performance.
Critical weaknesses
1.The paper mainly evaluates whether tool arguments have appeared in previous observations, but this does not necessarily mean the arguments are correct. While NoWit reduces unwitnessed emissions, task success shows little improvement. It is unclear whether the proposed method actually improves tool-call correctness and overall agent reliability.
2.The paper lacks a comparison with alternative recovery methods using the same provenance check. For example, would ordinary prompting or retrying after detecting an unsupported argument achieve similar results? Without this comparison, it is difficult to assess the additional benefit of the proposed constrained recovery mechanism.
Questions for authors
1.Would a simpler approach that uses the same detector but allows ordinary prompting or retrying achieve similar results? Such a comparison would help clarify the contribution of the proposed recovery mechanism.
2.How does NoWit handle valid arguments that have not appeared explicitly in previous observations? For example, an agent may need to derive a file path or construct an identifier based on available information. Could the witness check incorrectly block these calls, and how often does this happen in the evaluation?