- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
CMP randomizes retrieval exposure to measure memories that ordinary retrieval never surfaces. Exposed and unexposed outcomes estimate their contribution. Experiments improve required-memory discrimination, while showing that exposure-pool construction and aggregation across queries remain obstacles to retention.
Critical strengths
- Distinguishing untested memories from genuinely unhelpful ones is useful. The support-stratified results locate where randomized exposure adds information.
- The paper is candid about oracle pool construction and unsuccessful cross-query transfer. These negative results help define the method's actual scope.
Critical weaknesses
- The store-level effect U is zero when retrieval remains unchanged; that effect is identified. It is forced-exposure utility U_R that is unidentified. The paper repeatedly conflates the two. Proposition 1 also needs a retrieval-stability assumption beyond Definition 1 to justify a single replacement.
- Balanced assignment fixes exposure counts, but Eq. 6 defines Z as actual retrieval, which can include deterministic top-ranked memories. Appendix E.4 confirms that some have no excluded arm. Thus inclusion positivity alone does not support Proposition 2 for every pool member; the indicator and both-arm support conditions need correction.
- Poor transfer of the tested aggregations does not establish that no aggregation can predict future utility. That conclusion requires assumptions about the future query distribution. The oracle exposure pool further limits what the empirical gains establish for deployment.
Questions for authors
- Clarification: which intervention does the implemented estimator target, and how are deterministic inclusions handled?
- Clarification: under what query-distribution assumptions is the general cross-query impossibility claim valid?
Response to Reviewer Lcru
We thank the reviewer for the precise comments.
W1(a) (U vs. UR). We agree. The revision states after Proposition 1 that U(m∣q,c) is identified and equals 0 when m is not retrieved, and that the unidentified quantity is UR (Sec. 2.3).
W1(b) (retrieval stability). We agree. Proposition 1 now rests on Assumption 1 (score-separable retrieval, new) and Lemma 1 (new, App. C.1): adding or removing one memory changes the retrieved set by at most one element. Under this assumption, Proposition 1 is an exact identity at a fixed store. Cosine top-B retrieval satisfies Assumption 1; LLM-mediated retrieval, as in Mem0, need not.
W2 and Q1 (indicator, both-arm support, target). We agree. Assumption 2 (the submission's Assumption 1) now requires 0<P(m∈Rt)<1, and the design satisfies it for every pool member (Sec. 3.1):
On all 936 pool memories, CMP reaches 0.661 AUC, compared with 0.541 for store-level randomization (paired 95% CI [0.05, 0.19]; Table 2).
W3(a) and Q2 (cross-query claim). The claim fails under a stationary query distribution, where the query average estimates expected future utility. It would hold only if the observed queries carried no information about future ones, an assumption the paper neither stated nor tested. We removed the claim. Corollary 1 (new, Sec. 3.3) instead bounds the sample cost of the query average. On LoCoMo, a memory required by one question in thirty needs about 7,200 draws per arm, compared with 8 for the per-query contrast. The transfer results are reported as consistent with this cost.
W3(b) (oracle pool). We built an annotation-free pool from the query and the retriever alone: the six retrieved memories and cosine ranks 7–50 (Sec. 4.2, App. H.9). It contains 88% of the required memories and 76% of those never retrieved.
The gain is smaller than with the annotated pool, as the limitations state (App. J).