acceptodds
Under review as a conference paper at ICLR 2027

Recall Begets Recall: Recurrent Memory Ranking for Personalized LLMs

Abstract

Memory ranking is the stage in personalized Large Language Models (LLMs) that decides which of the candidate memories returned by a retriever enter the limited context, so that the answer is grounded in the right parts of the user's history. Existing memory rankers operate in a single pass, scoring each candidate once against the raw query and cutting the ranked list at a fixed top-. The single pass misses memories that the answer needs but that look unrelated to the query on their own, and the fixed cut ignores how many memories the query needs. To address both problems, we propose ReMind, a memory ranker that runs as a recurrent and self-terminating loop, following the cognitive theory of memory search: each recalled item joins the cue for the next recall (compound cueing), and recall stops once enough has been collected for the task (metamemory monitoring). For compound cueing, ReMind adds the top-ranked memory to the query context at each step and reranks the remaining candidates under the expanded cue, so that each recall makes the cue more specific and later steps look for what is still missing rather than for what merely resembles the query. For metamemory monitoring, a sufficiency predictor estimates at each step whether the collected memories are enough to answer the query and stops the loop once they are, so that the number of selected memories adapts to each query. Experiments on public benchmarks show that ReMind achieves the best results compared to state-of-the-art memory rankers.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Personalized LLM, Agent Memory, Ranking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HhHZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ReMind, a recurrent memory reranking framework for personalized LLMs. Unlike conventional single-pass rerankers, ReMind iteratively selects memories while conditioning each subsequent ranking step on previously selected evidence. A jointly trained sufficiency predictor determines when enough information has been collected. Experiments across multiple memory benchmarks demonstrate consistent improvements over existing rerankers, particularly for queries requiring multiple memories.

Critical strengths

Effective Recurrent Ranking: Conditioning memory selection on previously retrieved evidence improves retrieval quality, especially for complex queries requiring multiple relevant memories.

Adaptive Memory Selection: The sufficiency predictor enables query-dependent stopping, improving downstream QA performance compared to fixed Top-K selection.

Comprehensive Experimental Validation: Extensive comparisons across multiple benchmarks, retrievers, and backbone sizes, along with detailed ablations, support the effectiveness of the proposed framework.

Critical weaknesses

Limited Methodological Novelty: Iterative reranking and sufficiency-based stopping are conceptually straightforward extensions of existing retrieval approaches. The contribution primarily lies in their integration.

Computational Overhead: Repeatedly reranking the remaining candidates introduces additional inference costs, potentially limiting scalability to large memory collections.

Dependence on Gold Memory Annotations: Training the sufficiency predictor requires annotated sets of necessary memories, which may be difficult to obtain in realistic applications. Furthermore, unanswerable queries are excluded from evaluation.

Questions for authors

How does ReMind fundamentally differ from existing iterative retrieval approaches, such as RF-Mem? Could the authors provide a more direct comparison isolating the benefits of recurrent reranking versus iterative retrieval?

How could the sufficiency predictor be trained without complete gold-memory annotations, and how would ReMind handle queries for which no sufficient evidence exists in the memory collection?

3 comments
AuthorsOpenReview ↗

Response to Reviewer HhHZ [Part 1]

We sincerely thank the reviewer for recognizing that conditioning on previously selected evidence improves retrieval quality, and that the sufficiency predictor improves downstream QA performance compared with fixed top-K selection.

W1: Limited Methodological Novelty: Iterative reranking and sufficiency-based stopping are conceptually straightforward extensions of existing retrieval approaches. The contribution primarily lies in their integration.

Q1: How does ReMind fundamentally differ from existing iterative retrieval approaches, such as RF-Mem? Could the authors provide a more direct comparison isolating the benefits of recurrent reranking versus iterative retrieval?

We thank the reviewer for this comment. We first clarify the relation between ReMind and iterative retrieval, and then restate our contribution.

(1) ReMind and RF-Mem work at different stages and are complementary. RF-Mem is a memory retriever: it decides which memories enter the candidate pool. ReMind is a memory ranker: given the candidate pool, it decides which memories enter the context and when to stop. ReMind can therefore be applied on top of RF-Mem, and RF-Mem is one of the three upstream retrievers in our experiments (Appendix G).

(2) Technical differences. First, RF-Mem is training-free, while ReMind consists of trajectory construction, joint training of relevance and sufficiency, and recurrent inference. Second, RF-Mem mixes the retrieved memories with the query in the embedding space, while ReMind adds the selected memories to the context as text, so the ranker reads the query, the selected memories, and the candidate jointly. Third, RF-Mem has no notion of sufficiency, while ReMind learns when the selected memories are enough.

(3) Direct comparison. We show the relevant results on PersonaBench below.

RetrieverRankerR@3R@5N@3N@5MAPMRRF1
RF-MemNone26.5242.7227.5432.7931.8739.2419.40
RF-MemQwen3-Reranker-4B56.7166.5057.5259.7256.2163.6138.33
RF-MemReMind63.4273.3366.5067.7365.3072.3443.21

ReMind improves RF-Mem and outperforms Qwen3-Reranker-4B, the single-pass ranker that ReMind is built on, on the same RF-Mem candidates. Iterative retrieval thus does not replace recurrent ranking, and the two can be combined.

(4) On novelty. Our contribution is how to make iterative selection and stopping work for memory ranking, which is not a direct combination. As Table 2 shows, applying recurrent inference to a standard ranker lowers NDCG@3 by 3.52 (rows 1 and 2). It becomes effective only when the ranker is trained on selection states that contain wrong selections, and the sufficiency supervision further improves ranking itself (rows 4 and 6).

W2: Computational Overhead: Repeatedly reranking the remaining candidates introduces additional inference costs, potentially limiting scalability to large memory collections.

We thank the reviewer for this comment and respond from three aspects. (1) The cost of ReMind does not grow with the size of the memory collection. ReMind ranks the candidates returned by the upstream retriever, not the whole collection. A larger collection only affects the retriever, and the number of candidates that ReMind scores stays the same. (2) The additional cost is bounded and parallelizable. Within each step, the candidates are scored independently of each other, so they can be processed in parallel. The number of steps is limited by the selection budget and further reduced by sufficiency stopping, which ends the loop after 3.81 steps on average on PersonaBench. (3) The ranking time is small compared with answer generation. As analyzed in Appendix I, when the candidates within a step are scored in parallel, five recurrent steps take 297.48 ms in total, which is about 6% of the 4.83 s that the reader needs to generate an answer. With sufficiency stopping, ReMind runs 3.81 steps per query on average, so the ranking time in practice is lower than this five-step value. We have added more measurements on latency and analysis in Appendix I.

AuthorsOpenReview ↗

Response to Reviewer HhHZ [Part 2]

W3: Dependence on Gold Memory Annotations: Training the sufficiency predictor requires annotated sets of necessary memories, which may be difficult to obtain in realistic applications. Furthermore, unanswerable queries are excluded from evaluation.

Q2: How could the sufficiency predictor be trained without complete gold-memory annotations, and how would ReMind handle queries for which no sufficient evidence exists in the memory collection?

We thank the reviewer for these questions and respond to them in turn.

(1) Training without gold-memory annotations. We use datasets with gold annotations because they allow an accurate evaluation of ranking. When such annotations are not available, the sufficiency labels can be obtained from an LLM annotator. To verify this, we run ReMind on PersonaMem-v1 (32k), which provides no gold memory labels. For each training and validation question, we give GPT-4o the question, the options, the correct answer, and the top-50 candidates from the dense retriever. It returns a minimal ordered set of memories that together support the correct answer, and the memories that are confirmed to be unhelpful. The training trajectories and the relevance and sufficiency labels are then built from these annotations in the same way as in Section 2.1. The test set is not annotated. At test time, the selected memories are passed to a reader (Qwen3-8B), and we report the multiple-choice accuracy. From the results, ReMind achieves the best accuracy, outperforming the strongest baseline, while passing fewer memories to the reader. This shows that ReMind can be trained with labels from an LLM annotator and does not depend on gold memories.

MethodAccuracy (%)Avg. #Mem
Base (Dense)59.495
Qwen3-Reranker-0.6B62.665
Qwen3-Reranker-4B60.765
Qwen3-Reranker-8B62.665
KaLM-Nano63.925
KaLM-Small62.035
KaLM-Large62.665
RankGPT63.925
MemReranker-0.6B60.765
MemReranker-4B61.395
QRRanker63.295
ConvMemory63.295
ReMind-SFT65.195
ReMind66.463.47

(2) Queries without sufficient evidence. Our evaluation focuses on queries whose required evidence exists in the memory collection. ReMind predicts whether the selected memories are sufficient, rather than whether sufficient evidence exists anywhere in the collection. The latter is a complementary problem. A possible extension is to add an evidence-availability classifier that uses the query and retrieved candidates to decide whether to proceed with memory selection or abstain. Such a classifier would require separate training and evaluation, which we leave for future work.

We have clarified this evaluation scope in Section 3.1, Section 3.2, Appendix B, and Appendix N of the revised manuscript.

Dear Reviewer, thank you for reading our paper and offering such insightful comments. We hope our responses and the corresponding updates have resolved your concerns. With the end of the reviewer-author discussion phase drawing near, we would like to kindly check whether there are any further questions we could help answer. We would be glad to address them before the deadline. If you find our responses satisfactory, we would greatly appreciate it if you could reconsider your evaluation and possibly raise the score. Thank you very much for your time and support!

Reviewer_HbEuOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReMind, a recurrent memory ranker for personalized LLMs. Starting from a fixed retrieved candidate pool, ReMind repeatedly scores the remaining memories conditioned on the query and previously selected memories, adds the highest-scoring candidate, and predicts whether the accumulated evidence is sufficient. Relevance and sufficiency prediction share a language-model backbone and are jointly trained using LoRA. Training examples are constructed from sampled trajectories containing positive memories and hard-negative distractors. Experiments cover LongMemEval-S/M, PersonaBench, and ConvoMem with three upstream retrievers. The paper reports improvements over general-purpose and memory-specific rankers, provides component ablations, and evaluates adaptive stopping through downstream question answering on PersonaBench.

Critical strengths
  1. The ranking evaluation provides encouraging evidence of practical value. All rankers receive the same frozen top-50 candidates, which helps isolate improvements after retrieval. The evaluation spans multiple memory benchmarks and three retrievers. On PersonaBench with dense retrieval, ReMind improves NDCG@3 from 66.08 for ReMind-SFT to 74.77, suggesting that the overall training and selection recipe is useful.
  2. The ablations reveal a useful interaction between training and recurrent inference. Simply applying recurrent inference to the standard SFT model reduces NDCG@3 from 66.08 to 62.56, whereas recurrent inference improves the jointly trained model from 71.67 to 75.06 (Table 2). This supports the importance of training on intermediate selection states rather than merely repeating an existing ranker.
Critical weaknesses
  1. The incremental methodological contribution is insufficiently established. Prior methods such as Beam Retrieval [1] already perform sequential evidence selection conditioned on previously selected passages, with adaptive termination. ReMind-SFT also uses only one positive memory per query, whereas ReMind receives richer trajectory-based supervision, making it difficult to attribute the overall gains to recurrent selection.
  2. Evidence for adaptive stopping remains limited. Evaluation uses only one dataset and one reader. Gains over the best fixed cutoff are small (EM: 14.81 vs. 13.58; F1: 31.91 vs. 30.88), with identical LLM-judge scores and no uncertainty estimates. This provides limited support for a central contribution.

[1] Jiahao Zhang et al. 2024. End-to-End Beam Retrieval for Multi-Hop Question Answering. In Proceedings of NAACL 2024, pages 1718–1731.

Questions for authors
  1. How does ReMind differ from existing sequential evidence selection methods, particularly End-to-End Beam Retrieval (NAACL 2024)? A direct comparison with this closely related approach would help clarify ReMind's technical contribution.
  2. ReMind introduces both recurrent selection and additional supervision compared with ReMind-SFT. Can the authors provide a single-pass baseline with matched supervision and training budget to isolate the contribution of recurrence?
  3. The improvement over the best fixed cutoff in Table 11 is relatively small. Can the authors report statistical significance or confidence intervals for this improvement and analyze premature versus delayed stopping, particularly given the gap between ReMind's average selection count (3.81) and the oracle (2.12)?
  4. Appendix I estimates ranking latency under ideal parallelism. Can the authors report the actual end-to-end latency of ReMind, including recurrent ranking and sufficiency checks, compared with a single-pass ranker on the same hardware?
6 comments
AuthorsOpenReview ↗

Response to Reviewer HbEu [Part 1]

We sincerely thank the reviewer for recognizing the practical value of our work and the useful interaction between training and recurrent inference revealed by our ablations.

W1&Q1: How does ReMind differ from existing sequential evidence selection methods, particularly End-to-End Beam Retrieval (NAACL 2024)? A direct comparison with this closely related approach would help clarify ReMind's technical contribution.

We thank the reviewer for pointing Beam Retrieval [1]. We clarify the task setting and the specific contribution of ReMind below.

(1) Different problem settings. Beam Retrieval targets multi-hop QA, where the evidence forms a reasoning chain linked by bridge entities, and the candidates are the 10 to 20 passages provided by the dataset. ReMind targets memory ranking for personalized LLMs. The required memories are independent facts scattered across a user's history, their number varies across queries, and the candidates come from an upstream retriever, so they contain many memories that resemble the query but do not help answer it.

(2) Explicit supervision of evidence completeness. Beam Retrieval terminates by thresholding the highest score among expanded passage hypotheses. ReMind jointly learns two distinct targets: whether a candidate is relevant and whether the selected memories, after adding that candidate, cover the required evidence. A relevant candidate can extend an incomplete evidence set without completing it. ReMind explicitly distinguishes these cases through separate relevance and sufficiency labels, with both tasks sharing all model parameters.

(3) Contributions of supervision and recurrent inference. Table 2 separates their effects. With the same training trajectories, adding sufficiency supervision improves performance without recurrent inference (rows 3 and 5) and with it (rows 4 and 6). Sufficiency stopping is disabled in all four rows, so these gains show that the additional objective improves ranking itself. Using the same jointly trained model, recurrent inference further improves the metrics (rows 5 and 6).

ReMind's contribution is therefore the joint training of conditional memory ranking and explicit evidence-completeness prediction on intermediate selection states, with the latter also providing the stopping criterion. Beam search could be combined with this design to maintain multiple selection paths; we leave this extension for future work. We have added Beam Retrieval and this discussion to the related work in the revision.

[1] Zhang, J., Zhang, H., Zhang, D., Yong, L., & Huang, S. (2024, June). End-to-end beam retrieval for multi-hop question answering. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 1718-1731).

W1&Q2: ReMind introduces both recurrent selection and additional supervision compared with ReMind-SFT. Can the authors provide a single-pass baseline with matched supervision and training budget to isolate the contribution of recurrence?

We thank the reviewer for this question. We would first like to clarify that ReMind-SFT is trained on all positive memories of each query, not only one of them. Each training instance pairs one positive memory with several hard negatives, and a query with several positives yields several instances. We have revised the description in Section 3.1, Section 3.4, and Appendix C to avoid this ambiguity.

The single-pass baselines with matched supervision are rows 3 and 5 of Table 2. Both are trained on exactly the same trajectories as ReMind and rank all candidates in a single pass at inference. We show the relevant rows below to make the comparison explicit.

MethodTrainingInferenceR@1R@3R@5N@1N@3N@5MAPMRRF1
ReMind-SFT (row 1)Single-passSingle-pass41.7765.0075.8459.2666.0867.8465.5070.8343.58
Matched supervision, w/o SL (row 3)TrajectoriesSingle-pass42.3965.6277.7060.4966.8469.1366.3272.2444.07
Matched supervision, w/ SL (row 5)Trajectories + SLSingle-pass49.8166.3676.2872.8471.6772.1771.0177.8645.62
ReMind w/o SS (row 6)Trajectories + SLRecurrent49.8171.6079.9272.8475.0675.0173.0679.6248.44

Rows 5 and 6 use the same trained model and differ only in whether inference is recurrent, so their gap isolates the contribution of recurrence. R@1 and N@1 are identical because the first step has an empty context, and the gains appear from the second selection onward, where the selected memories join the cue. The richer supervision also contributes on its own (rows 1, 3, and 5), which shows that the two sources of gain are complementary.

AuthorsOpenReview ↗

Response to Reviewer HbEu [Part 2]

W2: Evidence for adaptive stopping remains limited. Evaluation uses only one dataset and one reader. Gains over the best fixed cutoff are small (EM: 14.81 vs. 13.58; F1: 31.91 vs. 30.88), with identical LLM-judge scores and no uncertainty estimates. This provides limited support for a central contribution.

We thank the reviewer for this comment. Following the suggestion, we further extend the adaptive stopping experiment to two datasets (PersonaBench and ConvoMem) and two readers (Llama3-8B and Qwen3-8B), and report 95% confidence intervals for all results. All methods use the same ranking from ReMind and differ only in where the selection stops. On ConvoMem, the answers are long free-form sentences that rarely match the reference exactly, so Answer EM is close to 0 for all methods and is omitted.

PersonaBench + Llama3-8B

MethodAnswer EMAnswer F1Answer ROUGE-LAnswer LLM-judge#Mem
Fixed top-18.64 [2.47, 14.81]26.49 [19.30, 34.00]25.23 [18.16, 32.55]25.93 [16.05, 35.80]1
Fixed top-211.11 [4.94, 18.52]28.87 [21.44, 36.62]29.95 [21.99, 38.19]28.40 [18.52, 38.27]2
Fixed top-313.58 [7.40, 20.98]30.88 [22.69, 39.45]30.56 [22.80, 38.53]30.86 [20.98, 40.74]3
Fixed top-49.88 [3.70, 16.05]24.83 [17.63, 32.62]23.32 [16.25, 30.97]25.93 [17.28, 35.80]4
Fixed top-58.64 [3.70, 14.81]23.12 [16.23, 30.52]20.51 [14.12, 27.56]22.22 [13.58, 30.86]5
ReMind14.81 [7.41, 23.46]31.91 [23.90, 40.05]31.24 [23.67, 39.06]30.86 [20.98, 40.74]3.81
Oracle stop16.05 [8.64, 24.69]33.41 [25.23, 41.84]34.17 [26.08, 42.53]32.10 [22.22, 41.98]2.12
Gold memories16.05 [8.64, 24.69]34.79 [26.40, 43.51]34.81 [26.69, 43.27]33.33 [23.46, 43.21]2.02

PersonaBench + Qwen3-8B

MethodAnswer EMAnswer F1Answer ROUGE-LAnswer LLM-judge#Mem
Fixed top-14.94 [1.23, 9.88]20.91 [13.97, 28.49]18.55 [12.47, 25.28]17.28 [9.88, 25.93]1
Fixed top-27.41 [2.47, 13.58]19.43 [13.15, 26.21]20.70 [14.29, 27.65]16.05 [8.64, 24.69]2
Fixed top-37.41 [2.47, 13.58]23.26 [15.76, 31.02]22.48 [15.50, 29.86]18.52 [11.11, 27.16]3
Fixed top-43.70 [0.00, 8.64]20.34 [13.69, 27.49]18.87 [12.70, 25.41]17.28 [9.88, 25.93]4
Fixed top-54.94 [1.23, 9.88]18.80 [11.95, 26.16]18.09 [11.79, 24.90]16.05 [8.64, 24.69]5
ReMind8.64 [2.47, 14.81]23.96 [17.04, 31.33]24.94 [18.06, 32.37]20.99 [12.35, 29.63]3.81
Oracle stop8.64 [2.47, 14.81]28.38 [20.57, 36.61]26.53 [19.38, 34.00]22.22 [13.58, 30.86]2.12
Gold memories9.88 [3.70, 17.28]28.84 [21.04, 36.99]26.62 [19.36, 34.04]22.22 [13.58, 30.86]2.02

ConvoMem + Llama3-8B

MethodAnswer F1Answer ROUGE-LAnswer LLM-judge#Mem
Fixed top-113.40 [11.00, 16.94]11.38 [9.47, 14.22]3.50 [1.49, 8.33]1
Fixed top-225.53 [23.41, 27.99]21.17 [19.46, 23.43]13.50 [10.00, 20.37]2
Fixed top-330.72 [28.25, 33.20]24.59 [22.46, 26.73]24.00 [18.24, 29.63]3
Fixed top-431.24 [29.79, 32.45]24.50 [22.92, 25.73]24.00 [14.29, 27.50]4
Fixed top-531.23 [28.42, 33.61]24.35 [21.59, 26.62]29.00 [24.66, 34.72]5
ReMind34.80 [31.79, 37.53]27.26 [24.23, 30.01]34.00 [28.42, 40.00]4.11
Oracle stop36.88 [33.16, 39.05]29.21 [25.56, 31.70]38.00 [32.20, 46.79]3.56
Gold memories37.51 [33.71, 39.67]29.66 [26.03, 31.99]43.00 [34.94, 50.00]3.65

ConvoMem + Qwen3-8B

MethodAnswer F1Answer ROUGE-LAnswer LLM-judge#Mem
Fixed top-114.24 [12.07, 20.30]11.78 [10.43, 16.22]8.00 [4.85, 12.82]1
Fixed top-229.45 [27.06, 31.93]24.61 [22.08, 26.02]23.50 [17.02, 28.40]2
Fixed top-336.33 [33.12, 37.92]29.52 [26.20, 30.95]33.00 [25.00, 37.18]3
Fixed top-440.60 [38.27, 42.01]33.24 [30.46, 34.34]42.00 [25.71, 47.65]4
Fixed top-540.57 [37.39, 42.02]33.06 [29.35, 34.51]49.50 [45.10, 57.14]5
ReMind42.54 [39.26, 44.02]34.50 [30.97, 35.57]53.00 [48.28, 59.58]4.11
Oracle stop42.86 [40.10, 44.08]34.67 [31.26, 35.63]53.50 [48.00, 59.78]3.56
Gold memories43.58 [40.22, 45.44]35.22 [31.31, 36.55]58.00 [53.12, 63.16]3.65

From the results, we can draw the observation that the gain of adaptive stopping is consistent across datasets and readers. Across the four settings and all metrics, ReMind outperforms the best fixed cutoff in 13 of 14 comparisons and ties in the remaining one.

AuthorsOpenReview ↗

Response to Reviewer HbEu [Part 3]

To answer gains over the best fixed cutoff are small, we can draw the following conclusions based on the above results. (1) ReMind recovers a large part of the remaining headroom. The oracle stop bounds what any stopping rule can achieve on the same ranking. Measured against this bound, ReMind closes 41% and 14% of the F1 gap left by the best fixed cutoff on PersonaBench (Llama3-8B and Qwen3-8B), and 63% and 86% on ConvoMem. (2) It is worth noting that the best fixed cutoff is not known in advance and changes across datasets. The best K is 3 on PersonaBench but 4 or 5 on ConvoMem, and a cutoff tuned on one dataset transfers poorly to the other. ReMind reaches or exceeds the best cutoff on both datasets without tuning K, which is the practical benefit of adaptive stopping. We have added these results to Section 3.3 and Appendix H.1.

Q3: The improvement over the best fixed cutoff in Table 11 is relatively small. Can the authors report statistical significance or confidence intervals for this improvement and analyze premature versus delayed stopping, particularly given the gap between ReMind's average selection count (3.81) and the oracle (2.12)?

We thank the reviewer for these suggestions and respond to them in turn.

  • We have added 95% confidence intervals to Table 12 (Table 11 in the original submission) and extended the experiment to another dataset and another reader. Please see our response to W2 for the complete results.

  • Premature versus delayed stopping. For each query, let kk be the number of memories ReMind selects and k∗k^* be the length of the shortest prefix of the ReMind ranking that covers all required evidence. We group the PersonaBench test queries as follows.

CategoryCriterionShare
Exact stoppingk=k∗k = k^*25.93%
Premature stoppingk<k∗k < k^*4.94%
Delayed stoppingk>k∗k > k^*35.80%
Insufficient evidenceNo prefix within the top-5 covers all required evidence33.33%

To measure the cost of the two types of errors, we replace the memories selected by ReMind with the oracle prefix of length k∗k^* on the affected queries and compare the downstream qa performance. Based on the results, we can draw the following conclusions. (1) First, ReMind rarely stops too early. Premature stopping accounts for only 4.94% of the queries, while delayed stopping accounts for 35.80%. (2) Second, the two errors have very different costs. Stopping early removes evidence the answer needs and lowers metrics substantially. Stopping late only adds distractors. Erring on the late side is therefore the safer behavior for a stopping rule. We have added this analysis to Appendix H.4.

CategoryInputAvg. #MemAnswer F1Answer LLM-judge
Premature stoppingReMind1.318.330.00
Premature stoppingOracle stop3.2537.5025.00
Delayed stoppingReMind4.8718.0924.14
Delayed stoppingOracle stop2.3623.5127.59

Q4: Appendix I estimates ranking latency under ideal parallelism. Can the authors report the actual end-to-end latency of ReMind, including recurrent ranking and sufficiency checks, compared with a single-pass ranker on the same hardware?

We thank the reviewer for this suggestion. Following it, we measure the actual latency of ReMind and a single-pass ranker (qwen3-reranker-4b) on the same hardware (10 NVIDIA V100 GPUs), with the same queries and the same candidate sets from PersonaBench. In each step, the remaining candidates are distributed over the GPUs and scored in parallel. The timing covers the complete ranking pipeline from the candidates to the final ranked list.

MethodMean (ms/query)95% CI (ms)P95 (ms)Avg. steps
Single-pass141.99[140.05, 143.94]156.261
ReMind (adaptive stopping)750.33[721.23, 774.62]875.913.81

Based on the above results, we can draw the following conclusions. (1) First, ReMind takes 750.33 ms per query on average, costing more time than the latency of the single-pass ranker. This overhead comes from running 3.81 ranking steps per query on average, each followed by a sufficiency check. (2) Second, the ranking time remains small compared with answer generation. The reader takes 4.83 s on average to generate an answer, so the ranking of ReMind accounts for about 15.5% of the generation time. We have added these measurements and analysis to Appendix I.

Reviewer_GQkXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This manuscript proposes ReMind, a framework based on compound cueing and recurrent ranking, to address the limitations of single-pass ranking and fixed top-K selection in memory rankers in personalized LLMs. ReMind utilizes compound cueing and employs recurrent ranking to incorporate the selected memories into the context, dynamically adjusting the subsequent ranking, and achieving adaptive stopping through a sufficiency predictor. Extensive experiments verify the effectiveness of the proposed method in memory ranking and downstream question-answering tasks.

Critical strengths
  1. The manuscript addresses the limitations of existing memory rankers in single-pass ranking and fixed top-K selection. The motivation is clear, and the method design is intuitive.
  2. Extensive experiments were conducted on multiple memory benchmarks and different retrievers, which effectively verified the effectiveness of the method.
Critical weaknesses
  1. ReMind relies on existing gold memories and constructs training instances through hard-negative sampling and manually set selection trajectories. These training states may have distributional differences from the memory-selection trajectories generated by the model during actual recurrent inference. In Sec. 3.4, Table 2 indicates that the sufficiency loss is an important part of performance improvement, but the independent contributions of training data construction, additional supervision information, and SL itself have not been fully verified.
  2. The experiment in Sec. 3.3 of sufficiency stopping was only conducted on PersonaBench, and it mainly compared with ReMind's own fixed top-K variants, without direct comparison with existing methods.
Questions for authors
  1. The paper has not analyzed the impact of different topic concentrations and negative difficulties on the method. Could the authors, based on the existing data, further compare the performance under different topic similarities, hard-negative ratios, or naturally generated selection trajectories to verify the generalization ability of ReMind?
  2. Can SL be applied to single-pass rerankers or other existing methods under the same training data, supervision, and computational budget, in order to further distinguish the performance improvement brought by the construction of training data from that of SL itself?
  3. Sec. 3.3 only verified the downstream QA performance on PersonaBench. Could Answer EM and Answer F1 be added to the benchmark for verification, and then compared with the existing methods and different ablation variants under the same memory budget?
8 comments
AuthorsOpenReview ↗

Response to Reviewer GQkX [Part 1]

We sincerely thank the reviewer for recognizing our clear motivation, intuitive method design, and extensive experiments we conducted.

W1: ReMind relies on existing gold memories and constructs training instances through hard-negative sampling and manually set selection trajectories. These training states may have distributional differences from the memory-selection trajectories generated by the model during actual recurrent inference. In Sec. 3.4, Table 2 indicates that the sufficiency loss is an important part of performance improvement, but the independent contributions of training data construction, additional supervision information, and SL itself have not been fully verified.

We thank the reviewer for this comment. It raises two concerns and we respond to them in turn.

(1) Gap between constructed training trajectories and inference trajectories.

ReMind uses constructed trajectories during training and its own selections during inference. Our trajectory construction is designed to reduce this gap. With the negative ratio ρ\rho, hard negatives are placed into the selected context and stand in for the wrong memories the ranker may select at inference, so the ranker learns to find the remaining positives from imperfect states. Appendix E confirms that this design matters: the Positive-only variant (ρ=0\rho = 0), whose training contexts never contain a negative memory, falls behind ReMind.

Another way to reduce this gap is to train on the trajectories generated by the model itself [1]. Following the reviewer's suggestion, we have additionally tested whether such trajectories bring further gains. We split the PersonaBench training set into two halves, A and B, and compare three models trained from scratch:

  • A-only: trained on the constructed trajectories of A.
  • A + B on-policy: we run the A-only model on the queries of B, which it has not seen, and collect the trajectories it generates. The model is then trained on the constructed trajectories of A and the model-generated trajectories of B.
  • Full (ReMind): trained on the constructed trajectories of A and B.
MethodR@1R@3R@5N@1N@3N@5MAPMRRF1
A-only42.1266.8576.4662.9668.0468.9467.0171.9245.31
A + B on-policy48.1768.3378.0970.3772.5973.3472.0177.4046.45
Full (ReMind)49.8171.3078.7472.8474.7774.2872.6179.2548.09

Adding model-generated trajectories from B improves all nine metrics over A-only. However, Full performs best on every metric and exceeds the mixed variant. Thus, model-generated trajectories provide useful training states, but replacing the constructed trajectories of B with them does not improve performance in this experiment. These results support the effectiveness of our trajectory construction, while leaving open whether further rounds of model-generated training could bring additional gains. We have added this experiment to Appendix E.2 and referenced it in Section 3.1 of the revised manuscript.

[1] Ross, S., Gordon, G., & Bagnell, D. (2011, June). A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics (pp. 627-635). JMLR Workshop and Conference Proceedings.

AuthorsOpenReview ↗

Response to Reviewer GQkX [Part 2]

(2) Contributions of training data construction, supervision, and SL

We discuss the three factors in turn. For convenience, we show the ablation results below, where RT is recurrent training, SL is the sufficiency loss, RI is recurrent inference, and SS is sufficiency stopping.

IDRTSLRISSR@1R@3R@5N@1N@3N@5MAPMRRF1
1✗✗✗✗41.7765.0075.8459.2666.0867.8465.5070.8343.58
2✗✗✓✗41.7759.4072.1459.2662.5665.3364.0469.1740.39
3✓✗✗✗42.3965.6277.7060.4966.8469.1366.3272.2444.07
4✓✗✓✗42.3967.8878.1160.4968.9470.4267.5673.9545.60
5✓✓✗✗49.8166.3676.2872.8471.6772.1771.0177.8645.62
6✓✓✓✗49.8171.6079.9272.8475.0675.0173.0679.6248.44
7✓✓✓✓49.8171.3078.7472.8474.7774.2872.6179.2548.09

Training data construction. Training on the constructed trajectories improves performance under single-pass inference (rows 1 and 3) and under recurrent inference (rows 2 and 4). In addition, we also study the individual choices in the construction with four variants:

  • Single-pass (SP): no trajectories are built, and each training instance pairs the query with positive memories and the hard negatives.
  • Positive-only (PO): the trajectories contain only positive memories (ρ=0\rho = 0).
  • Negative-end (NE): negative memories are also used as selection targets.
  • Single-order (SO): only one ordering of the positive memories is sampled for each query.
VariantR@1R@3R@5N@1N@3N@5MAPMRRF1
SP41.7765.0075.8459.2666.0867.8465.5070.8343.58
PO41.9366.9576.0360.4967.8668.7465.9872.1045.33
NE45.6867.2277.2065.4369.5470.8268.6774.6745.49
SO40.2366.2375.9959.2667.5368.7166.2072.3344.78
ReMind49.8171.3078.7472.8474.7774.2872.6179.2548.09

ReMind outperforms all four variants, which shows that each construction choice (negatives in the context, negatives never used as selection targets, and multiple positive orderings) contributes to the final performance. We have highlighted these findings in Section 3.4 of the revised version.

Supervision. We would like to clarify that ReMind does not use any additional data or external supervision. All rows of ablation study use the same queries, the same gold memory annotations, and the same retrieved candidate pools. Trajectory construction only reorganizes these annotations into selection states, and the sufficiency label is computed directly from them, zt,m=1[Mq+⊆set(St⊕m)]z_{t,m} = \mathbb{1}[\mathcal{M}_q^+ \subseteq \mathrm{set}(S_t \oplus m)]. Regarding the combination without recurrent training but with SL (RT ✗, SL ✓), we kindly note that SL is defined on selection states. The sufficiency label asks whether the memories collected so far are enough to answer the query, so it needs a state StS_t that holds the collected memories. Single-pass training data has no such state, which is why ablation study does not include this combination.

SL itself. We would like to clarify that the ablation study isolates the contribution of SL. Rows 3 and 5, as well as rows 4 and 6, are trained on exactly the same trajectories with the same relevance labels, and differ only in whether Lsuf\mathcal{L}_{suf} is added to the objective. Under this controlled comparison, SL improves performance consistently. We attribute this gain to the multi-task effect of the two objectives. To decide whether the evidence is complete, the model must judge what the selected memories already cover and what is still missing, which is the same ability the relevance task needs when it asks whether a candidate adds needed evidence.

AuthorsOpenReview ↗

Response to Reviewer GQkX [Part 3]

W2: The experiment in Sec. 3.3 of sufficiency stopping was only conducted on PersonaBench, and it mainly compared with ReMind's own fixed top-K variants, without direct comparison with existing methods.

We thank the reviewer for this constructive suggestion. Following the suggestion, we extend this experiment in two directions.

(1) Adaptive stopping on an additional dataset. We repeat the experiment on ConvoMem with the same reader (Llama3-8B). Answer EM is omitted because ConvoMem answers are long free-form sentences that rarely match the reference exactly, and EM is 0 for all methods.

MethodAnswer F1Answer ROUGE-LAnswer LLM-judge#Mem
Fixed top-113.4011.383.501
Fixed top-225.5321.1713.502
Fixed top-330.7224.5924.003
Fixed top-431.2424.5024.004
Fixed top-531.2324.3529.005
ReMind34.8027.2634.004.11
Oracle stop36.8829.2138.003.56
Gold memories37.5129.6643.003.65

The trend is consistent with PersonaBench. ReMind outperforms the best fixed cutoff on all metrics while using fewer memories than top-5.

(2) Comparison with existing stopping strategies. We further compare sufficiency stopping with two commonly used stopping strategies. All methods use the same trained ReMind ranker and differ only in the stopping rule:

  • Relevance threshold: select at least one memory, and stop when the highest relevance score among the remaining candidates falls below a threshold.
  • Score gap: at each step, select the top-ranked candidate, and stop when the gap between the highest and the second-highest relevance scores reaches a threshold.

We report both the ranking metrics and the downstream QA results with two readers.

PersonaBench (ranking)

Stopping ruleR@1R@3R@5N@1N@3N@5MAPMRRF1
Relevance threshold49.8168.8378.5072.8473.3873.7372.0978.7846.85
Score gap49.8167.5978.2572.8472.7673.4771.9378.6746.23
Sufficiency (ReMind)49.8171.3078.7472.8474.7774.2872.6179.2548.09

PersonaBench (downstream QA)

Stopping ruleLlama EMLlama F1Llama LLM-judgeQwen EMQwen F1Qwen LLM-judge
Relevance threshold9.8824.9428.403.7020.3418.52
Score gap9.8824.6527.163.7020.3418.52
Sufficiency (ReMind)14.8131.9130.868.6423.9620.99

ConvoMem (ranking)

Stopping ruleR@1R@3R@5N@1N@3N@5MAPMRRF1
Relevance threshold29.9776.2793.1799.0097.0296.2594.6999.2979.62
Score gap29.9776.1593.0899.0096.9196.1594.6399.2579.47
Sufficiency (ReMind)29.9776.7593.7799.0097.5196.7595.1099.5080.08

ConvoMem (downstream QA)

Stopping ruleLlama F1Llama LLM-judgeQwen F1Qwen LLM-judge
Relevance threshold31.1729.0040.4749.50
Score gap31.2129.0040.4949.50
Sufficiency (ReMind)34.8034.0042.5453.00

Based on the above results, we can draw the conclusion that sufficiency stopping outperforms both strategies on both datasets and both readers. The sufficiency predictor judges the selected set as a whole, which is what the stopping decision requires. We have revised Appendix H to include the ConvoMem experiment and comparisons with stopping variants.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Reviewers, AC, SAC, and PC,

We sincerely thank you for the time, expertise, and thoughtful feedback you have provided in reviewing our manuscript. We greatly appreciate the reviewers' recognition of the clear motivation and intuitive design of ReMind, the effectiveness of recurrent ranking and adaptive memory selection, and the comprehensive experimental validation. Your constructive comments have been invaluable in helping us strengthen both the technical presentation and the experimental evaluation. For convenience, we summarize below the major strengths, followed by the key revisions we have made in response to the concerns.


Key strengths noted by the reviewers:

  • S1: Clear motivation and intuitive design, addressing single-pass ranking and fixed top-K selection in existing memory rankers (R1).
  • S2: Effective recurrent ranking, especially for queries needing multiple memories, with ablations showing a useful interaction between recurrent training and inference (R2, R3).
  • S3: Adaptive memory selection, where the sufficiency predictor improves downstream QA over fixed top-K (R1, R3).
  • S4: Comprehensive and fair experiments across multiple benchmarks, retrievers, and backbone sizes, with a shared frozen candidate pool (R1, R2, R3).
  • S5: Clear presentation. All reviewers found the paper clear and easy to follow (R1, R2, R3).

Key revisions in response to the comments:

We have revised the paper carefully, with major changes summarized below:

  • [R2, R3] To clarify the novelty of ReMind relative to sequential evidence selection and iterative retrieval, we added Beam Retrieval and RF-Mem to the related work in Section 4, and clarified that RF-Mem is a retriever while ReMind is a ranker, so the two are complementary. We point to the results of ReMind on top of RF-Mem candidates in Appendix G, where it outperforms the single-pass ranker it is built on. We emphasize that the contribution lies in jointly training conditional ranking and explicit evidence-completeness prediction on intermediate selection states, since naively applying recurrent inference to a standard ranker hurts performance (Table 2, rows 1 and 2).
  • [R1, R2] To isolate the contributions of training data construction, supervision, sufficiency loss, and recurrence, we clarified in Section 3.4 that the sufficiency loss is defined on selection states and is isolated by rows 3 vs. 5 and 4 vs. 6 of Table 2, and that rows 5 and 6 isolate the effect of recurrent inference under matched supervision. we summarized the trajectory construction variants of Appendix E.1 in Section 3.4, and clarified in Section 3.1, Section 3.4, and Appendix C that ReMind-SFT is trained on all positive memories of each query.
  • [R1] To address the gap between constructed training trajectories and inference trajectories, we added an experiment in Appendix E.2 that trains on model-generated trajectories, referenced in Section 3.1.
  • [R1, R2] To strengthen the evidence for adaptive stopping, we extended the fixed-cutoff experiment to ConvoMem and a second reader in Section 3.3 and Appendix H.1. We further compared sufficiency stopping with relevance-threshold and score-gap stopping in Appendix H.2.
  • [R2] Following the suggestion on premature versus delayed stopping, we added an error analysis in Appendix H.4.
  • [R1] To evaluate downstream QA across all methods, we added answer-quality results for all baselines and ablation variants on PersonaBench and ConvoMem with two readers in Appendix L and Appendix M, summarized in Section 3.2 and Section 3.4.
  • [R1] To verify generalization across topics and retrieval difficulties, we added breakdowns of PersonaBench queries by topic and by retriever recall in Section 3.6, Appendix K.2, and Appendix K.3.
  • [R2, R3] To address computational overhead, we added measured end-to-end ranking latency on the same hardware in Appendix I.2.
  • [R3] To address the dependence on gold memory annotations, we added an experiment on PersonaMem-v1 in Appendix N (referenced in Section 3.2), where ReMind is trained with memory labels from an LLM annotator.
  • [R3] Regarding queries without sufficient evidence, we clarified the evaluation scope in Section 3.1 and Appendix B, and discussed an evidence-availability classifier as a future extension.

All the revised content has been marked in red\color{#9b2c2c}{\text{red}} in our revised paper.

(* We refer to Reviewer GQkX as R1, Reviewer HbEu as R2, and Reviewer HhHZ as R3.)


Thank you again for your time, careful evaluation, and for managing the review process. We appreciate your effort, and we hope this summary is helpful.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.