- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper shows an interesting phenomenon: if we RL-train personal agents with QA rewards, the model will learn the shortcut to the correct answer that fits some shallow patterns, instead of learning to memorize the user preferences that guide reasoning towards the correct answer. Motivated by this, they created a de-biased version of the PersonaMem-V2 (PMv2) dataset and redesigned the RL reward to train a preference extractor directly, independent of the QAs.
Critical strengths
- They demonstrate that their trained preference extractor provides useful memories that can be used by many different LLMs to answer personalized questions in Table 3.
- They identify issues (that potentially cause reward hacking) in the PMv2 benchmark and propose a modified version, FairPMv2.
Critical weaknesses
- In Table 2, why do all the retrieval methods have identical scores on PMv2 and FairPMv2?
- The core idea of this paper, rewarding the model directly by comparing the memorized preferences against the chunked gold preferences, resembles [1], which rewards each dialogue turn by having an LLM judge determine whether it covers any sentence of the masked ground-truth information. It is therefore suggested to cite the paper and discuss the differences and this paper's novel contribution.
- Instead of semantic-F1, have the authors considered other cheaper metrics like BLEU scores / ROUGE / BERTScore? I wonder how much semantic-F1 outperforms these alternatives.
- How good are the questions in FairPMv2 compared to PMv2? In Figure 1, (b) and (c) indicate that base models answer FairPMv2 questions better. What is the performance of base Qwen3-4B?
- In Table 3, what will the performance be if the answer model is Qwen3-4B? That would serve as a fair baseline for comparing against the QA-RL-Joint baseline, which uses Qwen3-4B only.
[1] Huang, Tenghao, et al. "Teaching Language Models To Gather Information Proactively." EMNLP (Findings). 2025.
Questions for authors
- The paper stated that QA rewards for tuning personalized agents via reinforcement learning are very common. However, it seems that in the cited papers, only [2] actually used QA rewards for RL on a personalization task. If QA rewards are not widely adopted in this domain, the practical impact of showing their shortcomings may be limited. Why are QA rewards rarely used for personalization? Is the limited adoption a sign that they are already known to be unreliable for this task? Does QA shortcutting also exist in domains beyond PersonaMem-V2 for memory agents?
- Do the reconstructed QA pairs provide better signals? If so, how does the model trained with reconstructed QA rewards perform on FairPersonaMem-V2 and the original PersonaMem-V2?
- How to interpret that on the OOD benchmark PMv1, most RL baselines (except QUIET-diff) are weaker than Mem0 (which requires no training)?
[2] Jiang, Bowen, et al. "PersonaMem-v2: Towards personalized intelligence via learning implicit user personas and agentic memory." arXiv preprint arXiv:2512.06688 (2025).
4 comments
Author Rebuttal (Part 2/2)
- For W5: The results corresponding to Table 3, using vanilla Qwen3-4B-Instruct as the answer model on FairPMv2 with a 32k context length, are shown below. Using preferences extracted by QUIET yields meaningfully higher accuracy than using those from the QA-RL methods, although the absolute scores are lower than those obtained with larger LLM answer models. Since vanilla Qwen3-4B-Instruct has not been additionally trained for personalized responses or MCQ answering, it may perform worse as an answer model than QA-RL-Joint.
| Extractor | Qwen3-4B-Instruct |
|---|---|
| QA-RL-Swap | 33.7 |
| QA-RL-Frozen | 35.2 |
| QUIET-plain | 39.4 |
| QUIET-diff | 39.1 |
-
For Q1 : We would like to clarify that our paper does not claim that QA-reward RL is already “very common” for personalized agents. Rather, we note that QA accuracy is a common evaluation metric for preference and memory systems, and that recent work has increasingly explored using downstream QA outcomes as RL rewards for learning memory extraction and management. This direction has also begun to extend to personalized preference extraction, including MemAgent[2] and Memory-R1[3]. Our intent is therefore not to suggest that QA-reward RL is already standard in preference extraction, which remains an emerging research area, but to identify an important weakness of this increasingly explored training paradigm. We appreciate the reviewer’s question, as it gives us an opportunity to clarify the intended scope of our claim. [2] Yu, Hongli, et al. "Memagent: Reshaping long-context llm with multi-conv rl-based memory agent." ICLR. 2026. [3] Yan, Sikuan, et al. "Memory-r1: Enhancing large language model agents to manage and utilize memories via reinforcement learning." ACL (Volume 1: Long Papers). 2026.
-
For Q2 : We thank the reviewer for suggesting this important control. FairPersonaMem-V2 was constructed as an evaluation benchmark, and we did not reconstruct the training QA pairs. Extending the reconstruction procedure to the large training split would require substantial additional data generation, which we were unable to complete within the rebuttal period. We therefore cannot determine from the current experiments whether training with reconstructed QA rewards would close the gap with QUIET. We agree that improved QA construction is a plausible alternative and do not claim that it would be insufficient. Improved QA construction and direct preference-level rewards may offer complementary remedies.
-
For Q3: This can be understood from two perspectives. First, the experimental settings differ. Complete-form extraction methods must compress all extracted information into a fixed token budget (), whereas retrieval methods can retrieve the top segments of the original history for each query without a token limit on the retrieved text. As user histories grow longer, complete-form extraction requires increasingly lossy compression, placing these methods at a disadvantage relative to retrieval. Second, there is the QA bias discussed in our paper. PMv1 is an OOD benchmark that the models have not seen during training. QA-RL models tend to be strongly biased toward their training data, and their performance therefore drops more sharply on such OOD benchmarks.
Thank you for the response. The authors have addressed some of my concerns. I have decided to maintain my positive score.
Thank you very much for your thoughtful feedback and for maintaining your positive score. We are glad that the key strengths of our work came through, particularly our identification of shortcut signals in PMv2 and the reconstruction of FairPMv2, as well as the transferability of our learned preference extractions across various answer models. Your questions and suggestions were also very helpful in sharpening our analysis and the presentation of the paper.
Author Rebuttal (Part 1/2)
We thank Reviewer K1T2 for the thoughtful and constructive feedback and for recognizing the significance of the QA shortcutting phenomenon and the utility of QUIET’s extracted preferences across different answer models. We appreciate the reviewer’s questions regarding novelty, generalizability, and additional baselines, which we address below.
For W1: There were numerical inconsistencies and copy-paste errors in the reported experimental results. We apologize for the confusion these errors caused regarding the reliability of our results. Due to space constraints, we cannot include the corrected Table 2 here; please refer to the revised manuscript, which we have uploaded. We thank the reviewer for identifying these inconsistencies, and we have audited all reported values against the raw evaluation outputs. These corrections do not change the method ranking or our main conclusions. The corrected values are now consistent across all occurrences in the figures, tables, main text, and appendix.
For W2 : Thank you for pointing us to this relevant work. [1] shares a high-level motivation with our approach in providing intermediate supervision for implicit user information, rather than relying solely on downstream outcomes. However, the learning objectives are distinct: [1] trains models to proactively elicit missing information through questions, whereas QUIET directly optimizes the fidelity of preferences already expressed in conversations using semantic F1 as the RL reward. The per-turn reward in [1] is binary, whereas QUIET’s semantic F1 balances preference precision and recall. We will add this work to Related Work section and clarify this connection and distinction. [1] Huang, Tenghao, et al. "Teaching Language Models To Gather Information Proactively." EMNLP (Findings). 2025.
For W3: This is a very interesting suggestion. Our current semantic matching approach compares preferences at the level of meaning, but requires an LLM call for each reward computation, making it costly. It is therefore reasonable to consider simpler metrics such as BLEU, ROUGE, and BERTScore.
We explored two approaches to test this idea. The first is to replace the LLM judge’s binary semantic matching with thresholding based on this metric. We determined the thresholds using a subset of previous training logs to fit original semantic matching. The second is to use Hungarian one-to-one matching between extracted and GT bullets, taking the maximum total metric score over all possible one-to-one assignments as the reward. In this approach, matching and reward computation are performed together, so no separate calculation such as an F1 score is needed. We trained models using BLEU, ROUGE-2 and BERTScore under both approaches, but the latter approach (Hungarian one-to-one matching) did not provide an adequate training signal, and the models did not train successfully. The results for the former approach (thresholding) are shown below.
Base model : Qwen3-4B-Instruct, context length=32k / BLEU & ROUGE-2 & BERTScore with thresholding method
Training with BLEU and BERTScore slightly improved F1, but reduced performance on the other MCQ evaluations compared with the baseline. Training with ROUGE-2 slightly improved MCQ accuracy and substantially improved F1 over the baseline, demonstrating the potential applicability of simple metrics. However, these metrics performed substantially worse than QUIET, which uses semantic-level matching. We suspect that this is because metrics based on direct string matching or shallow semantic matching provide rewards that are too sparse. Unlike explicit tasks such as summarization, implicit extraction often involves target words that do not appear verbatim in the input context. Models therefore have the freedom to record the content in their own words. These metrics struggle with matching at an abstract level and may fail to assign rewards even when the extracted and GT preferences are semantically related, potentially resulting in a sparse reward signal.