- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper explores how to reduce the memory verification cost of long-term LLM intelligent agents. The author proposes CERP: Firstly, perform Full-CMI on a small number of candidate memories, and use the verification results to train a lightweight classifier that combines historical evaluation information with the similarity to the current task; subsequently, reject the expected useless memories in advance, and only perform cheaper utility verification for the remaining candidates, thereby amortizing the initial cost. The paper has been verified on Causal-LoCoMo. CERP saves the number of calls while resulting in a decrease in the Useful-Memory F1 score and an 8.8% reduction in the error rate of early rejection.
Critical strengths
- The paper focuses on the often overlooked verification cost in memory-enhanced agents: Even though verifying each item can improve memory selection, the additional calls may undermine system efficiency. This issue is framed as the cost allocation across sequences for decision-making, providing a useful perspective for evaluating memory verification methods.
- CERP employs a small number of historical features and logistic regression to achieve early rejection, with a simple structure. The paper explicitly includes the initial Full-CMI supervision cost and derives the net savings condition , distinguishing between the reduction in the number of calls for relative complete validation and the net gain from relative utility validation.
- The experiment separately examined the effects of current similarity, historical information, and their combination, and set a random baseline with the same number of rejections. Under the same call cost, CERP reduced the rejection error rate from 23.7% of the random strategy to 8.8%, indicating that the benefits come from selectively rejecting candidates rather than simply reducing the number of verifications.
- The paper analyzed the influence of the initial supervision budget, sequence length, rejection threshold, candidate order, and different responders on the results, and provided validation on a second benchmark. The authors clearly reported a decrease in quality, the inability to recover some orders, and that the utility labels could achieve similar performance, making the contribution boundaries more clear.
Critical weaknesses
- Using only the utility verification label can also achieve similar performance. This makes the value of performing additional perturbation verification during the initial stage less clear: if cheaper supervision can train an equally effective rejector, then is the additional cost of Full-CMI worth it?
- The main experiments were conducted by replaying the sequences while keeping the candidate set fixed and pre-calculating the verification results. This approach failed to reflect how memory selection affects current behavior, subsequent tasks, and future memories. The task scores of Causal-LoCoMo were derived from the concatenation of retained memories rather than the actual generation of responses.
- The main experiment saved 12.7% of the calls compared to Always-Verify, but the F1 score on Useful-Memory decreased by approximately 0.051, and it achieved positive returns only in 7 out of 10 sequences. When the initial budget was increased to , the average net savings became -10.5%. These results indicate that the method has conditional cost advantages, but the stable and predictable deployment benefits have not yet been established.
- Classifier fitting and threshold selection Use the same batch of initial samples. The 2% rejection error target set during the training stage does not guarantee the subsequent risk. The actual RER of the main experiment reached 8.8%. The model and threshold were then fixed, while the historical features and task distribution might change; the candidates that were rejected prematurely did not generate feedback, thus lacking a mechanism to discover and correct persistent false rejections.
Questions for authors
See weaknesses.