- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes ReMind, a recurrent memory reranking framework for personalized LLMs. Unlike conventional single-pass rerankers, ReMind iteratively selects memories while conditioning each subsequent ranking step on previously selected evidence. A jointly trained sufficiency predictor determines when enough information has been collected. Experiments across multiple memory benchmarks demonstrate consistent improvements over existing rerankers, particularly for queries requiring multiple memories.
Critical strengths
Effective Recurrent Ranking: Conditioning memory selection on previously retrieved evidence improves retrieval quality, especially for complex queries requiring multiple relevant memories.
Adaptive Memory Selection: The sufficiency predictor enables query-dependent stopping, improving downstream QA performance compared to fixed Top-K selection.
Comprehensive Experimental Validation: Extensive comparisons across multiple benchmarks, retrievers, and backbone sizes, along with detailed ablations, support the effectiveness of the proposed framework.
Critical weaknesses
Limited Methodological Novelty: Iterative reranking and sufficiency-based stopping are conceptually straightforward extensions of existing retrieval approaches. The contribution primarily lies in their integration.
Computational Overhead: Repeatedly reranking the remaining candidates introduces additional inference costs, potentially limiting scalability to large memory collections.
Dependence on Gold Memory Annotations: Training the sufficiency predictor requires annotated sets of necessary memories, which may be difficult to obtain in realistic applications. Furthermore, unanswerable queries are excluded from evaluation.
Questions for authors
How does ReMind fundamentally differ from existing iterative retrieval approaches, such as RF-Mem? Could the authors provide a more direct comparison isolating the benefits of recurrent reranking versus iterative retrieval?
How could the sufficiency predictor be trained without complete gold-memory annotations, and how would ReMind handle queries for which no sufficient evidence exists in the memory collection?
3 comments
Response to Reviewer HhHZ [Part 2]
W3: Dependence on Gold Memory Annotations: Training the sufficiency predictor requires annotated sets of necessary memories, which may be difficult to obtain in realistic applications. Furthermore, unanswerable queries are excluded from evaluation.
Q2: How could the sufficiency predictor be trained without complete gold-memory annotations, and how would ReMind handle queries for which no sufficient evidence exists in the memory collection?
We thank the reviewer for these questions and respond to them in turn.
(1) Training without gold-memory annotations. We use datasets with gold annotations because they allow an accurate evaluation of ranking. When such annotations are not available, the sufficiency labels can be obtained from an LLM annotator. To verify this, we run ReMind on PersonaMem-v1 (32k), which provides no gold memory labels. For each training and validation question, we give GPT-4o the question, the options, the correct answer, and the top-50 candidates from the dense retriever. It returns a minimal ordered set of memories that together support the correct answer, and the memories that are confirmed to be unhelpful. The training trajectories and the relevance and sufficiency labels are then built from these annotations in the same way as in Section 2.1. The test set is not annotated. At test time, the selected memories are passed to a reader (Qwen3-8B), and we report the multiple-choice accuracy. From the results, ReMind achieves the best accuracy, outperforming the strongest baseline, while passing fewer memories to the reader. This shows that ReMind can be trained with labels from an LLM annotator and does not depend on gold memories.
| Method | Accuracy (%) | Avg. #Mem |
|---|---|---|
| Base (Dense) | 59.49 | 5 |
| Qwen3-Reranker-0.6B | 62.66 | 5 |
| Qwen3-Reranker-4B | 60.76 | 5 |
| Qwen3-Reranker-8B | 62.66 | 5 |
| KaLM-Nano | 63.92 | 5 |
| KaLM-Small | 62.03 | 5 |
| KaLM-Large | 62.66 | 5 |
| RankGPT | 63.92 | 5 |
| MemReranker-0.6B | 60.76 | 5 |
| MemReranker-4B | 61.39 | 5 |
| QRRanker | 63.29 | 5 |
| ConvMemory | 63.29 | 5 |
| ReMind-SFT | 65.19 | 5 |
| ReMind | 66.46 | 3.47 |
(2) Queries without sufficient evidence. Our evaluation focuses on queries whose required evidence exists in the memory collection. ReMind predicts whether the selected memories are sufficient, rather than whether sufficient evidence exists anywhere in the collection. The latter is a complementary problem. A possible extension is to add an evidence-availability classifier that uses the query and retrieved candidates to decide whether to proceed with memory selection or abstain. Such a classifier would require separate training and evaluation, which we leave for future work.
We have clarified this evaluation scope in Section 3.1, Section 3.2, Appendix B, and Appendix N of the revised manuscript.
Dear Reviewer, thank you for reading our paper and offering such insightful comments. We hope our responses and the corresponding updates have resolved your concerns. With the end of the reviewer-author discussion phase drawing near, we would like to kindly check whether there are any further questions we could help answer. We would be glad to address them before the deadline. If you find our responses satisfactory, we would greatly appreciate it if you could reconsider your evaluation and possibly raise the score. Thank you very much for your time and support!
Response to Reviewer HhHZ [Part 1]
We sincerely thank the reviewer for recognizing that conditioning on previously selected evidence improves retrieval quality, and that the sufficiency predictor improves downstream QA performance compared with fixed top-K selection.
We thank the reviewer for this comment. We first clarify the relation between ReMind and iterative retrieval, and then restate our contribution.
(1) ReMind and RF-Mem work at different stages and are complementary. RF-Mem is a memory retriever: it decides which memories enter the candidate pool. ReMind is a memory ranker: given the candidate pool, it decides which memories enter the context and when to stop. ReMind can therefore be applied on top of RF-Mem, and RF-Mem is one of the three upstream retrievers in our experiments (Appendix G).
(2) Technical differences. First, RF-Mem is training-free, while ReMind consists of trajectory construction, joint training of relevance and sufficiency, and recurrent inference. Second, RF-Mem mixes the retrieved memories with the query in the embedding space, while ReMind adds the selected memories to the context as text, so the ranker reads the query, the selected memories, and the candidate jointly. Third, RF-Mem has no notion of sufficiency, while ReMind learns when the selected memories are enough.
(3) Direct comparison. We show the relevant results on PersonaBench below.
ReMind improves RF-Mem and outperforms Qwen3-Reranker-4B, the single-pass ranker that ReMind is built on, on the same RF-Mem candidates. Iterative retrieval thus does not replace recurrent ranking, and the two can be combined.
(4) On novelty. Our contribution is how to make iterative selection and stopping work for memory ranking, which is not a direct combination. As Table 2 shows, applying recurrent inference to a standard ranker lowers NDCG@3 by 3.52 (rows 1 and 2). It becomes effective only when the ranker is trained on selection states that contain wrong selections, and the sufficiency supervision further improves ranking itself (rows 4 and 6).
We thank the reviewer for this comment and respond from three aspects. (1) The cost of ReMind does not grow with the size of the memory collection. ReMind ranks the candidates returned by the upstream retriever, not the whole collection. A larger collection only affects the retriever, and the number of candidates that ReMind scores stays the same. (2) The additional cost is bounded and parallelizable. Within each step, the candidates are scored independently of each other, so they can be processed in parallel. The number of steps is limited by the selection budget and further reduced by sufficiency stopping, which ends the loop after 3.81 steps on average on PersonaBench. (3) The ranking time is small compared with answer generation. As analyzed in Appendix I, when the candidates within a step are scored in parallel, five recurrent steps take 297.48 ms in total, which is about 6% of the 4.83 s that the reader needs to generate an answer. With sufficiency stopping, ReMind runs 3.81 steps per query on average, so the ranking time in practice is lower than this five-step value. We have added more measurements on latency and analysis in Appendix I.