- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ScrambleToolBench, a benchmark for evaluating whether LLM agents can discover unknown tools and adapt to environmental changes. It removes semantic information from tool interfaces and introduces mapping drift, stochastic failures, and temporal execution windows. Experiments across 15 models show large performance drops after scrambling and under dynamic conditions. The authors also find that agents often rely on exhaustive exploration instead of using previously discovered tool mappings for efficient recovery.
Critical strengths
-
Evaluating how agents adapt their previously learned knowledge to changing environments is an interesting problem, especially for long-running agents. The benchmark provides a controlled setting to study this behavior.
-
I find the behavioral analysis more interesting than the overall success rates. The behavioral analysis is useful. Particularly the observation that increasing reasoning effort improves task completion but does not necessarily lead to more efficient recovery strategies.
Critical weaknesses
-
I have some doubt about the motivation of this benchmark. Why do we need to completely remove semantic information from tool interfaces? Even undocumented tools usually have meaningful names, parameters or response structures, and using these information is a reasonable part of agent capability. The mapping drift is also quite artificial, where existing identifiers suddenly execute unrelated functions. I understand these are controlled settings, but the connection to real agent robustness is not very clear. The large performance drop mainly shows agents struggle when useful interface information is deliberately removed. Whether this reflects a meaningful limitation in practical tool use remains unclear. For a benchmark paper, I think this is quite important.
-
The conclusion about agents failing to exploit their own mapping knowledge is not fully convincing. The cycle-tracing reference assumes accurate tool mappings and a particular permutation structure, while agents have incomplete knowledge and are not informed about this structure. Table 2 measures whether agents follow displacement pointers for previously discovered tools, but having discovered a tool does not mean the agent has correctly maintained its mapping. Also, without knowing the cyclic structure, following the displaced identifier is just one possible exploration strategy. The observed inefficiency could come from inaccurate memory or uncertainty about environment changes. Hard to conclude that agents fail at structured reasoning from these results.
Questions for authors
-
Is there any evidence that performance on ScrambleToolBench correlates with agents' ability to handle real undocumented tools or API changes? What practical capability should a high score on this benchmark indicate?
-
For Table 2, can you report the pointer-following rate conditioned on the agent actually having the correct previous mapping available in its memory? Currently, previously discovered tools and correctly maintained mappings seem mixed together.
-
How would a simple online tracker perform if it maintains observed tool fingerprints and updates mappings after drift, without knowing the underlying cyclic permutation? This could help understand how much of the recovery gap is due to reasoning.
-
If agents are provided with a correct but incomplete mapping table, without revealing the permutation rule, would their recovery behavior change substantially? This may help distinguish knowledge maintenance from reasoning over existing knowledge.