- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper explores how to encapsulate a segment of an agent's execution process into a reusable tool. The author uses controlled replay to examine which information affects the agent's decision-making, and then determines which steps the tool should include, as well as what needs to be passed in and returned. Finally, another model generates the tool code. Experiments use synthetic programs to check if the dependency judgments are correct, and then compare the task completion rate and interface size on AppWorld.
Critical strengths
S1. Considering which steps a tool includes and its input/output together, this approach is reasonable. When the scope of encapsulation changes, the information needed from the outside will also change. The two sides cannot be processed completely separately.
S2. The synthetic experiment has a clear correct answer and also takes into account the situation where multiple pieces of information must be used together and different pieces of information can be substituted for each other. This can directly check whether the method has identified the correct dependencies.
S3. Compared with the control group that retains all the observation information, Suffice uses a shorter interface under the same encapsulation scope, and the task completion rates measured on both sides are also relatively close. This comparison supports the value of simplifying the interface.
Critical weaknesses
W1. I still don't have a clear understanding of how much manual preparation is actually required for this process. For instance, how should the information in the execution record be broken down into variables, and how should a reasonable new history be constructed during replay? The appendix provides some requirements, but it doesn't clarify exactly how to do it in AppWorld. These are prerequisites for the subsequent search to run and will affect my judgment on the degree of automation of this method.
W2. Retaining sufficient information does not mean that the tool written by another model will make the original decision. Passing the interface check does not mean that the internal logic is correct. The current task completion rate indicates that the tool is useful, but it has not directly verified the decision preservation emphasized in the paper.
W3. The paper states that these tools can be reused in new tasks, but I didn't clearly understand if the execution records of the test tasks were used when generating the tools. If these records have been reviewed and the same task is run again, it still cannot be said that the tool can perform well on unseen tasks.
W4. The theoretical definition of sufficiency uses the action-distribution reconstruction error , but the replay check measures , the maximum total variation distance between action distributions induced by histories that agree on the retained information. The appendix gives bounds relating the two, but when a nonzero error is allowed, why does this recover all the minimal sufficient sets as defined?
Questions for authors
See weaknesses.
Can you run the original agent and the generated tool multiple times from the same initial state, and compare their action distributions? This way, we can directly check how much of the original decision-making has been retained after encapsulation.