- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes to train an AI agent when to do memory management (read, write, etc.) via hindsight process reward. Specifically, the hindsight process reward is assigned by checking whether a memory was retrieved, cited, or necessary for a correct answer through deletion tests. Experiments show that this feedback improves memory organization and QA accuracy compared with rewarding only final answer correctness.
Critical strengths
It is a quite interesting idea to use a micro-level ablation study to assign credit to each memory edition operation.
Critical weaknesses
- A major weakness is the soundness of the experimental setting. The model is tested on the in-domain tasks as the training tasks. It might be the case that the model has absorbed patterns related to the dataset instead of truly understanding how to do memory edition.
- A second weakness is that the reward is computed based on answer flips. In long-horizon tasks, re-rolling for each memory operation is very expensive and there might be noise as the flipping signal could simply be due to randomness in model reasoning.
Questions for authors
- Under the same total API/GPU budget, how does the method compare with more teacher-generated SFT data or iterative teacher feedback?
- How much improvement survives on a new domain without changing the memory schema or retraining?