- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
To improve deployed agent’s performance it is becoming a common practice to augment the agent with its past experience. The key challenge is to know what experience will be beneficial for a new query/task. Prior works have focussed on relevancy of an experience, but this paper argues that relevancy is not sufficient and aims to explore mechanisms through which we can establish reusability of an experience even when it might be relevant.
Critical strengths
The paper studies an important problem of construction context for a deployed agent from its past experience. The idea to separate relevance of an experience from reusability seems useful. The authors proposal to structure the raw experience into a form that makes assessing reusability seems useful.
Critical weaknesses
- The paper is hard to read. Terminology used does not seem to be standard and also definitions are missing before the terms are used. Especially in the abstract and the introduction ; for example
prose rendering,evidence,same-task state, etc. - The system description in section 3 seems very abstract and hard to follow. Again there are terms that are not defined. For example line 153 – artifact bindings, host-supplied receipts, etc. A simple running example should make it easier to follow. Similarly, equation 2 and 3 in admission gate discussion, it is not clear what Rank_hybrid and Tack_B are.
- The results are a limited set of datasets such as HotPotQA and a few others. And it appears the proposed method does not out perform the ablations studied. Additionally baselines mentioned in related works are not compared with. Agentic and reasoning tasks may be better tasks to evaluate on to see the method in action. The results are often presented without providing any insights, so it is hard to judge the significance of their results.
Questions for authors
See weaknessess above.
Also, how are the different stages of the experience curation computed, more examples on this will be useful