acceptodds
Under review as a conference paper at ICLR 2027

QUIET: QA-Free Unbiased Implicit Preference Extraction with Semantic F1 Rewards

Abstract

Personal LLM agents rely on preferences extracted from user–agent interactions to provide personalized responses. However, such preferences are often expressed only implicitly across lengthy interactions, making reliable extraction challenging. Although specialized small-scale extractors offer an efficient solution, we observed that question-answering (QA) rewards used in reinforcement learning (RL) can bias extractors toward QA-specific patterns at the expense of preference fidelity. Specifically, even on the rigorous PersonaMem-V2 benchmark, the QA-reward-trained model performs worse when conditioned on its own extracted preferences than without them, highlighting the need for improved extraction methods and new benchmarks that do not overlook such biases. To this end, we introduce FairPersonaMem-V2, a bias-aware reconstruction of the benchmark, and QUIET (QA-free Unbiased Implicit preference ExTraction), a reward design for RL to train a preference extractor directly from ground-truth preferences, without requiring QA supervision. On FairPersonaMem-V2, the small-scale Qwen3-4B-Instruct extractor trained with QUIET achieves a 32% relative MCQ accuracy gain and an 11 higher semantic F1, with 21 fewer training context tokens compared to the QA-reward trained baseline.

Primary area:
reinforcement learning
Keywords:
agent, personalization, preference, extraction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_K1T2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows an interesting phenomenon: if we RL-train personal agents with QA rewards, the model will learn the shortcut to the correct answer that fits some shallow patterns, instead of learning to memorize the user preferences that guide reasoning towards the correct answer. Motivated by this, they created a de-biased version of the PersonaMem-V2 (PMv2) dataset and redesigned the RL reward to train a preference extractor directly, independent of the QAs.

Critical strengths
  • They demonstrate that their trained preference extractor provides useful memories that can be used by many different LLMs to answer personalized questions in Table 3.
  • They identify issues (that potentially cause reward hacking) in the PMv2 benchmark and propose a modified version, FairPMv2.
Critical weaknesses
  • In Table 2, why do all the retrieval methods have identical scores on PMv2 and FairPMv2?
  • The core idea of this paper, rewarding the model directly by comparing the memorized preferences against the chunked gold preferences, resembles [1], which rewards each dialogue turn by having an LLM judge determine whether it covers any sentence of the masked ground-truth information. It is therefore suggested to cite the paper and discuss the differences and this paper's novel contribution.
  • Instead of semantic-F1, have the authors considered other cheaper metrics like BLEU scores / ROUGE / BERTScore? I wonder how much semantic-F1 outperforms these alternatives.
  • How good are the questions in FairPMv2 compared to PMv2? In Figure 1, (b) and (c) indicate that base models answer FairPMv2 questions better. What is the performance of base Qwen3-4B?
  • In Table 3, what will the performance be if the answer model is Qwen3-4B? That would serve as a fair baseline for comparing against the QA-RL-Joint baseline, which uses Qwen3-4B only.

[1] Huang, Tenghao, et al. "Teaching Language Models To Gather Information Proactively." EMNLP (Findings). 2025.

Questions for authors
  • The paper stated that QA rewards for tuning personalized agents via reinforcement learning are very common. However, it seems that in the cited papers, only [2] actually used QA rewards for RL on a personalization task. If QA rewards are not widely adopted in this domain, the practical impact of showing their shortcomings may be limited. Why are QA rewards rarely used for personalization? Is the limited adoption a sign that they are already known to be unreliable for this task? Does QA shortcutting also exist in domains beyond PersonaMem-V2 for memory agents?
  • Do the reconstructed QA pairs provide better signals? If so, how does the model trained with reconstructed QA rewards perform on FairPersonaMem-V2 and the original PersonaMem-V2?
  • How to interpret that on the OOD benchmark PMv1, most RL baselines (except QUIET-diff) are weaker than Mem0 (which requires no training)?

[2] Jiang, Bowen, et al. "PersonaMem-v2: Towards personalized intelligence via learning implicit user personas and agentic memory." arXiv preprint arXiv:2512.06688 (2025).

4 comments
AuthorsOpenReview ↗

Author Rebuttal (Part 1/2)

We thank Reviewer K1T2 for the thoughtful and constructive feedback and for recognizing the significance of the QA shortcutting phenomenon and the utility of QUIET’s extracted preferences across different answer models. We appreciate the reviewer’s questions regarding novelty, generalizability, and additional baselines, which we address below.

  • For W1: There were numerical inconsistencies and copy-paste errors in the reported experimental results. We apologize for the confusion these errors caused regarding the reliability of our results. Due to space constraints, we cannot include the corrected Table 2 here; please refer to the revised manuscript, which we have uploaded. We thank the reviewer for identifying these inconsistencies, and we have audited all reported values against the raw evaluation outputs. These corrections do not change the method ranking or our main conclusions. The corrected values are now consistent across all occurrences in the figures, tables, main text, and appendix.

  • For W2 : Thank you for pointing us to this relevant work. [1] shares a high-level motivation with our approach in providing intermediate supervision for implicit user information, rather than relying solely on downstream outcomes. However, the learning objectives are distinct: [1] trains models to proactively elicit missing information through questions, whereas QUIET directly optimizes the fidelity of preferences already expressed in conversations using semantic F1 as the RL reward. The per-turn reward in [1] is binary, whereas QUIET’s semantic F1 balances preference precision and recall. We will add this work to Related Work section and clarify this connection and distinction. [1] Huang, Tenghao, et al. "Teaching Language Models To Gather Information Proactively." EMNLP (Findings). 2025.

  • For W3: This is a very interesting suggestion. Our current semantic matching approach compares preferences at the level of meaning, but requires an LLM call for each reward computation, making it costly. It is therefore reasonable to consider simpler metrics such as BLEU, ROUGE, and BERTScore.

We explored two approaches to test this idea. The first is to replace the LLM judge’s binary semantic matching with thresholding based on this metric. We determined the thresholds using a subset of previous training logs to fit original semantic matching. The second is to use Hungarian one-to-one matching between extracted and GT bullets, taking the maximum total metric score over all possible one-to-one assignments as the reward. In this approach, matching and reward computation are performed together, so no separate calculation such as an F1 score is needed. We trained models using BLEU, ROUGE-2 and BERTScore under both approaches, but the latter approach (Hungarian one-to-one matching) did not provide an adequate training signal, and the models did not train successfully. The results for the former approach (thresholding) are shown below.

Base model : Qwen3-4B-Instruct, context length=32k / BLEU & ROUGE-2 & BERTScore with thresholding method

MethodPMv2FairPMv2F1PMv1
QA-RL-Frozen43.644.010.754.2
BLEU40.941.318.750.1
ROUGE-246.644.526.355.0
BERTScore39.638.211.144.7
QUIET-diff51.149.847.564.2

Training with BLEU and BERTScore slightly improved F1, but reduced performance on the other MCQ evaluations compared with the baseline. Training with ROUGE-2 slightly improved MCQ accuracy and substantially improved F1 over the baseline, demonstrating the potential applicability of simple metrics. However, these metrics performed substantially worse than QUIET, which uses semantic-level matching. We suspect that this is because metrics based on direct string matching or shallow semantic matching provide rewards that are too sparse. Unlike explicit tasks such as summarization, implicit extraction often involves target words that do not appear verbatim in the input context. Models therefore have the freedom to record the content in their own words. These metrics struggle with matching at an abstract level and may fail to assign rewards even when the extracted and GT preferences are semantically related, potentially resulting in a sparse reward signal.

  • For W4: The PMv2 and FairPMv2 results using vanilla Qwen3-4B-Instruct as both the extractor and the answer model are shown below. Compared with the QA-RL-Joint version of Qwen3-4B-Instruct, performance differs substantially on PMv2, particularly in the Empty Pref and Extracted Pref conditions, whereas the differences on FairPMv2 are minor.
BenchmarkEmpty PrefExtracted PrefGT Pref
PMv236.433.265.7
FairPMv236.935.560.6
AuthorsOpenReview ↗

Author Rebuttal (Part 2/2)

  • For W5: The results corresponding to Table 3, using vanilla Qwen3-4B-Instruct as the answer model on FairPMv2 with a 32k context length, are shown below. Using preferences extracted by QUIET yields meaningfully higher accuracy than using those from the QA-RL methods, although the absolute scores are lower than those obtained with larger LLM answer models. Since vanilla Qwen3-4B-Instruct has not been additionally trained for personalized responses or MCQ answering, it may perform worse as an answer model than QA-RL-Joint.
ExtractorQwen3-4B-Instruct
QA-RL-Swap33.7
QA-RL-Frozen35.2
QUIET-plain39.4
QUIET-diff39.1
  • For Q1 : We would like to clarify that our paper does not claim that QA-reward RL is already “very common” for personalized agents. Rather, we note that QA accuracy is a common evaluation metric for preference and memory systems, and that recent work has increasingly explored using downstream QA outcomes as RL rewards for learning memory extraction and management. This direction has also begun to extend to personalized preference extraction, including MemAgent[2] and Memory-R1[3]. Our intent is therefore not to suggest that QA-reward RL is already standard in preference extraction, which remains an emerging research area, but to identify an important weakness of this increasingly explored training paradigm. We appreciate the reviewer’s question, as it gives us an opportunity to clarify the intended scope of our claim. [2] Yu, Hongli, et al. "Memagent: Reshaping long-context llm with multi-conv rl-based memory agent." ICLR. 2026. [3] Yan, Sikuan, et al. "Memory-r1: Enhancing large language model agents to manage and utilize memories via reinforcement learning." ACL (Volume 1: Long Papers). 2026.

  • For Q2 : We thank the reviewer for suggesting this important control. FairPersonaMem-V2 was constructed as an evaluation benchmark, and we did not reconstruct the training QA pairs. Extending the reconstruction procedure to the large training split would require substantial additional data generation, which we were unable to complete within the rebuttal period. We therefore cannot determine from the current experiments whether training with reconstructed QA rewards would close the gap with QUIET. We agree that improved QA construction is a plausible alternative and do not claim that it would be insufficient. Improved QA construction and direct preference-level rewards may offer complementary remedies.

  • For Q3: This can be understood from two perspectives. First, the experimental settings differ. Complete-form extraction methods must compress all extracted information into a fixed token budget (L=2,048L=2{,}048), whereas retrieval methods can retrieve the top k=4k=4 segments of the original history for each query without a token limit on the retrieved text. As user histories grow longer, complete-form extraction requires increasingly lossy compression, placing these methods at a disadvantage relative to retrieval. Second, there is the QA bias discussed in our paper. PMv1 is an OOD benchmark that the models have not seen during training. QA-RL models tend to be strongly biased toward their training data, and their performance therefore drops more sharply on such OOD benchmarks.

Thank you for the response. The authors have addressed some of my concerns. I have decided to maintain my positive score.

Thank you very much for your thoughtful feedback and for maintaining your positive score. We are glad that the key strengths of our work came through, particularly our identification of shortcut signals in PMv2 and the reconstruction of FairPMv2, as well as the transferability of our learned preference extractions across various answer models. Your questions and suggestions were also very helpful in sharpening our analysis and the presentation of the paper.

Reviewer_yHipOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies implicit preference extraction for personal LLM assistants: a small extractor splits a long user–assistant conversation into 5k-token chunks and recurrently updates a length-bounded, query-independent preference memory, which is then consumed by a downstream answer model. The authors argue that the answer-correctness reward used by existing RL methods allows the model to exploit regularities between questions and options in PersonaMem-V2. The contributions are: (1) FairPersonaMem-V2, which retains the conversation histories and ground-truth (GT) preferences of PMv2 but regenerates every multiple-choice question, replacing the distractors with two closely related preferences from the same category and making the question neutral among all three; (2) QUIET, a reward that does not rely on question answering: after each conversation chunk, an LLM judge performs one-to-one semantic matching between the extracted memory and the GT preferences revealed up to that chunk, and the resulting F1 score is used as the reward for GRPO optimization. The paper conducts a progressive comparison based on Qwen3-4B. QUIET-diff reaches 49.8% on FairPMv2-32k, compared with 37.6% for QA-RL-Joint and 44.0% for QA-RL-Frozen, while reducing the number of training tokens by about 21×.

Critical strengths

S1. The paper provides a clear and controlled diagnosis of a real failure mode. The empty-memory probe in Figure 1 and the progressive comparison in Table 1 separate two factors: the co-adaptation between the extractor and the answerer, and the inherent insufficiency of the QA reward. Showing that a QA-RL model performs worse with its own memory than with no memory at all is a useful warning for the community. If released, FairPMv2 would also be a practical tool for examining this issue. S2. The method is simple, does not depend on a specific answer model, and yields broad improvements: QUIET-diff ranks first on FairPMv2 with all six answer models, and it also leads across three backbones, on PMv1 with contexts of up to 1M tokens, and on AlpsBench, while using about 21× fewer training tokens. The advantage of the Diff format grows with context length, which is a finding of practical significance.

Critical weaknesses

W1. The conclusion that removing QA from the reward leads to better performance may instead be attributable to the substantial additional supervision that QUIET introduces. QUIET is trained with chunk-level ground-truth preference states P*_i, whereas the QA-RL baselines receive only a single multiple-choice correctness signal per trajectory. The performance gain may therefore come from much stronger supervision rather than from removing QA from the reward. W2. The reliability of the reported numbers is insufficient to judge the true magnitude of the improvement. (a) In Table 2, the accuracies of the three retrieval-based baselines on PMv2 and FairPMv2 are identical for all three backbones and both context lengths (e.g., Mem0 with Qwen3-4B scores 39.9/39.2 on both benchmarks). (b) The same configuration is reported with inconsistent values across tables: the semantic F1 of QUIET-diff (4B, 32k) is 47.5 (Tables 1, 2, 6), 45.9 (Tables 9, 10), and 44.9 (Table 4); the FairPMv2 accuracy of QA-RL-Frozen with GPT-5.4-mini is 44.0 (Tables 1, 2) and 40.3 (Table 3). (c) FairPMv2 is generated by GPT-5.4 and evaluated with GPT-5.4-mini without any human validation; the distractor preferences generated by “altering the attribute” may coincide with other true preferences of the same user.

Questions for authors

Q1. How are the chunk-level ground-truth preference states P*_i obtained from PMv2? Q2. Are the identical numbers for Mem0, A-MEM, and LangMem on PMv2 and FairPMv2 in Table 2 an error? Q3. I suggest adding an SFT baseline that uses P*_i as targets, as well as a variant that combines QA-RL-Frozen with the F1 reward, to separate the respective effects of “removing QA” and “adding ground-truth preference supervision.” Q4. To verify that FairPMv2 indeed removes the query–choice bias without introducing new labeling errors, could the authors conduct a human evaluation on a sample of FairPMv2 items, checking whether the distractors are plausible, whether they conflict with the user’s other true preferences, and whether the questions are truly neutral?

7 comments
AuthorsOpenReview ↗

Author Rebuttal (Part 1/2)

We thank Reviewer yHip for the thoughtful and constructive feedback. We appreciate the reviewer’s recognition of the issue of QA shortcutting and QUIET’s consistent improvements across settings. We agree that disentangling the benefit of preference-level supervision from that of our reward design is essential for correctly interpreting the results. We address each concern below.

  • For W1 : We agree that QUIET receives more fine-grained supervision by using utterance-grounded preference annotations as chunk-level training targets. However, the number of preferences in each conversation is fixed. QUIET assigns these preferences to chunks using their evidence grounding, whereas QA-RL uses as many QA pairs as there are preferences in the conversation (i.e., multiple QAs per trajectory). Thus, the underlying preference content is shared, but the form and granularity of supervision differ. When preference annotations and evidence grounding are available, QUIET uses them at a finer granularity to provide a more efficient learning objective. We provide an additional control for stronger supervision through the Gold-SFT experiment in our response to Q3.

  • For W2(a) and (b) : We thank the reviewer for catching these errors. The identical retrieval numbers across PMv2 and FairPMv2 and the inconsistent values reported across tables were reporting/copy-paste errors, rather than results produced by identical evaluations. We apologize that these errors reduced confidence in the reported results. We have now audited the reported values against the raw evaluation outputs and corrected all affected tables, figures, main-text references, and appendix entries. These corrections do not change the relative method rankings or the main conclusions of the paper. The corrected results are included in the revised manuscript.

  • For W2(c): To address concerns about the quality of FairPMv2, we conducted a human validation of both PMv2 and FairPMv2. We evaluated 300 items from each benchmark, corresponding to approximately 10% of the eligible items in which the GT preference applied to the user. (Excluding items concerning third-party attributes or scenarios in which the user requested that the information be forgotten.) Each item contained three personalized answer choices, with the generic option removed. Here we excluded generic option and the items where the generic option was correct, as including them would require a more complex choice-selection protocol beyond the scope of this validation. Our goal was to assess whether participants could achieve high accuracy when given the GT preference and accuracy close to chance (33.3%) without preference information. Thirteen participants were randomly assigned items without knowing which benchmark each item came from, and the three answer choices were randomly shuffled. Half of the assigned items were evaluated under the Empty Pref condition and the other half under the GT Pref condition. The table below reports human three-choice MCQ accuracy (%) under each condition.

BenchmarkEmpty PrefGT Pref
PMv256.296.9
FairPMv232.395.8

In the GT Pref condition, PMv2 and FairPMv2 achieved accuracies of 96.9% and 95.8%, respectively, indicating that the correct choices could be clearly identified from the provided preference information in both benchmarks. In the Empty Pref condition, PMv2 achieved 56.2% accuracy, above the chance level (33.3%), whereas FairPMv2 achieved 32.3%, close to the chance. These findings suggest that correct choices in PMv2 can often be identified without preference information, whereas FairPMv2 reduces this source of bias.

We also used an LLM judge (GPT-6 Luna) to check whether the altered preferences in FairPMv2 coincided with any other GT preferences in the same conversation. No such overlaps were found in this check.

  • For Q1: PMv2 provides utterance-level evidence grounding for each target preference. We use this grounding to construct $P_i^* $ : at each chunk ii, the gold state contains all preferences whose supporting utterances have appeared up to that point. We therefore do not introduce additional preference annotations when constructing Pi∗P_i^*; rather, we derive the intermediate states from the evidence grounding already provided by PMv2.

  • For Q2 : Yes. The identical retrieval numbers were reporting errors; please see our response to W2(a,b). The corrected values are included in the revised manuscript.

AuthorsOpenReview ↗

Author Rebuttal (Part 2/2)

  • For Q3: The reviewer asks us to disentangle the effects of using GT preferences as training targets and removing the QA reward. We constructed the warmup SFT dataset for QUIET-diff by providing Sonnet-4.6 with each chunk and the gold preference state expected after processing it, and asking it to generate the corresponding oracle edit actions in JSON format. Thus, the checkpoint trained on this dataset can be regarded as Gold-SFT with a diff output format. Since QUIET-diff starts RL training from this checkpoint, comparing Gold-SFT with QUIET-diff allows us to assess the additional contribution of the RL stage.
ModelMethodPMv2 32kPMv2 128kFairPMv2 32kFairPMv2 128kF1 32kF1 128kPMv1 32kPMv1 128kPMv1 1M
Qwen3-4B-InstQA-RL-Frozen43.6354440.510.71.154.245.239.3
Qwen3-4B-InstGold-SFT46.845.846.642.918.18.659.843.439.9
Qwen3-4B-InstQUIET-diff51.151.849.844.247.527.164.258.353.6
Llama3.2-3BQA-RL-Frozen35.535.738.537.30.71.452.538.235
Llama3.2-3BGold-SFT44.439.142.838.514.58.153.840.637.9
Llama3.2-3BQUIET-diff48.844.747.544.332.115.360.149.842.6
Qwen3-1.7BQA-RL-Frozen37.633.63839.22.71.745.736.430
Qwen3-1.7BGold-SFT41.335.741.438.511.43.658.647.436.7
Qwen3-1.7BQUIET-diff43.639.442.941.324.46.457.653.953.1

Gold-SFT provides a strong baseline using gold preference supervision. Importantly, QUIET-diff further improves semantic F1 across all three extractor backbones at both context lengths, and improves downstream accuracy in most settings, with particularly large gains in long-context evaluation. These results demonstrate the additional benefit of the RL stage beyond the shared gold-supervised SFT initialization. We will add details of the Gold-SFT baseline and the construction of the QUIET-diff SFT dataset to the revised manuscript.

Regarding QA-RL-Frozen+F1, QA correctness in QA-RL-Frozen is computed after processing the entire conversation, whereas QUIET computes the F1 reward and updates the model after each chunk. Specifically, QUIET evaluates the candidates generated at each chunk using F1, updates the model, and carries only one candidate state forward to the next chunk. Consequently, a corresponding conversation-level QA reward is not available for each candidate scored by F1, making it difficult to directly combine the two rewards within the current training procedure. These results support the conclusion that direct preference-level RL provides additional gains after gold-supervised SFT and can improve downstream performance without QA supervision during QUIET training. We agree that an objective combining QA and F1 rewards would provide a useful additional comparison. Since we have not evaluated this combination, our results do not rule out the possibility that adding a QA reward could be beneficial.

  • For Q4: Following the reviewer’s suggestion, we conducted human validation of both PMv2 and FairPMv2. Please see our response to W2(c) for the protocol and results.

Thanks — the Gold-SFT baseline, the seeded reruns, the human validation and the table audit are the controls I asked for, and they narrow W1. But they do not move my score. Against Gold-SFT the downstream gap is +3.2 on FairPMv2-32k (46.6 to 49.8), and the large F1 gap comes from the same judge family that supplies the reward, so the headline still mixes "no QA reward" with "preference-level supervision plus its own metric"; the human validation dropped the generic option and every item where it was correct, which is exactly the subset my concern applies to; and the five seeds cover only Qwen3-4B at 32k. The rebuttal is not sufficient for me to change my rating.

Reviewer_whKNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies RL training of small LLMs that extract implicit user preferences from long conversations. The authors argue that QA-correctness rewards, as in MemAgent-style training, let the model exploit query–choice regularities instead of improving extraction. On PersonaMem-V2, a QA-RL-trained Qwen3-4B answerer reaches 61.9% with empty preferences, which is higher than the 57.8% it gets with its own extracted preferences.

They make two contributions:

FairPersonaMem-V2, a rebuilt version of the PMv2 test set. It keeps the original chats and GT preferences but uses same-category “plausible” distractors and queries that are neutral across the correct answer and the distractors. QUIET, a QA-free reward. Each per-chunk preference state is scored with an LLM-judged, bipartite-matching semantic F1 against chunk-level gold preferences. There are two output formats: Plain (full rewrite) and Diff (Mem0-style ADD/UPDATE/DELETE with a format reward).

Across three extractor backbones, QUIET improves MCQ accuracy on FairPMv2, PMv1 and AlpsBench over QA-reward baselines, and uses about 21× fewer training context tokens.

Critical strengths
  1. A clear, well-designed diagnosis of QA-reward shortcutting. The progressive comparison in Table 1 (Joint → Swap → Frozen) cleanly separates answerer co-adaptation from extractor quality. The evidence is striking:

Figure 1a shows the QA-RL model doing better with empty preferences than with its own extractions. Qwen3-1.7B QA-RL-Joint reaches 80.4% on PMv2 with a semantic F1 of 0.0.

This matters for the community because QA-reward RL is becoming a standard recipe for memory and preference agents, and the paper shows that in-distribution QA accuracy can be a misleading training and evaluation signal.

  1. A simple reward that improves results consistently across many settings. The reward-design ablation (Table 4, Figure 4) justifies the bipartite constraint: precision-only and many-to-one rewards collapse to a single bullet, and recall-only inflates to the token budget. QUIET’s ranking over QA-reward baselines holds across three backbones, six answer models (Table 3), six judges (Table 10), and out-of-distribution benchmarks. The Diff format gives growing gains at long context (up to 1M tokens). This breadth is the main reason I do not recommend clear rejection.
Critical weaknesses
  1. The comparison mixes up the reward design with stronger supervision, and the key baseline is missing.

QUIET needs gold preference sets aligned to every chunk (P*_i), while the QA-RL baselines use only QA labels. The “QA-free” framing hides that QUIET relies on strictly stronger supervision, which is rarely available outside synthetic benchmarks. Two related concerns compound this:

Missing SFT-on-gold baseline. With chunk-level gold states available, the obvious baseline is SFT that distills P*_i directly. The existing SFT baseline uses the PMv2 SFT data and reaches 4.8% semantic F1. A.1 also says QUIET-diff uses its “own dataset” for SFT warm-start, without saying how it was built. Circular metric. The same judge (GPT-5.4-mini) with the same prompt produces both the training reward and the reported semantic F1. The 47.5 vs. 4.4 gap is therefore largely expected, not independent evidence of extraction quality. Baseline outputs are also bulletized by a separate LLM, which may inflate their bullet counts (N = 40–91) and depress their precision. There is no human evaluation of extraction quality.

As a result, I cannot tell whether the gains come from RL with semantic F1 or simply from access to gold preferences. This is the central claim of the paper.

  1. Evaluation reliability: the new benchmark is not validated, and there are internal inconsistencies.

Benchmark validation. FairPMv2 has no human validation of distractor correctness or query neutrality. Empty-preference accuracy is still 36.4%, above chance. The correct option is always personalized, while the answer prompt tells the model to pick the generic option when no match is found. Both the benchmark and QUIET’s targets come from the same GT preferences, which favours extractors that copy GT phrasing. Algorithm 1 vs. Section 3.1. Algorithm 1 (line 7) gives the query q to the QA-RL extractor, which contradicts the query-independent formulation in Section 3.1. This leaves unclear how the baseline was run at inference and how its “shared extraction” F1 was computed. Numerical inconsistencies. Table 2 retrieval results are identical on PMv2 and FairPMv2 for every method and backbone, even though the queries differ. QA-RL-Joint on FairPMv2 is 38.7 in Figure 1a vs. 37.6 in Tables 1 and 2. QA-RL-Frozen with GPT-5.4-mini is 40.3 in Table 3 vs. 44.0 in Tables 1 and 2. QUIET-diff semantic F1 is reported as 47.5, 44.9 and 45.9 in different tables. The text says QUIET-diff beats QA-RL-Joint on PMv1 32k by 7.9pp; the table gives 7.7.

Together these lower my confidence in the reported margins. All results are also single-run, with no seeds or variance, while several conclusions rest on 1–3 point differences.

Questions for authors

Clarifications (could change my assessment):

How does SFT that directly distills the chunk-level gold states P*_i perform with the same data budget? Does the RL stage of QUIET still give a clear gain over it? (This is my most important question.) Does the QA-RL extractor receive the query q during training and/or inference, as Algorithm 1 suggests? Which extraction is used to compute its semantic F1? How were the chunk-level gold sets P*_i built and aligned across tokenizers? How was the SFT warm-start data for QUIET-diff built? Please explain the identical PMv2/FairPMv2 retrieval numbers in Table 2 and the other inconsistencies listed above. Was any human validation done on FairPMv2? What is the accuracy of a choice-only (no-query) baseline? How many seeds were used, and what are the standard deviations for Tables 1–3? What is the full training cost, including about 44.5K judge calls per epoch, compared with QA-RL-Frozen?

Suggestions (would not change my score on their own):

Add a small human evaluation of extraction quality that does not rely on an LLM judge. Moderate the framing. “Unbiased” in the title overstates the claim. The 32% relative gain in the abstract is measured against the weakest baseline; against QA-RL-Frozen it is about 13%. Retrieval methods outperform QUIET on AlpsBench EI. Cite related work on context-free shortcut baselines: hypothesis-only NLI (Gururangan et al., 2018; Poliak et al., 2018), shortcut learning (Geirhos et al., 2020), and choice-only MCQ artifacts (Balepur et al., 2024). Fix minor issues: “bounded by = 2,048” in A.1 is missing the symbol L. b_{i,j} in Section 3.3 is undefined. w_fmt and λ_fmt are used for the same quantity. Several citations use \citet where \citep is needed. The “(AI, 2024)” bib entry should read “Meta AI”. “givne” in A.5. The y-axis in Figure 1 starts at 20.

8 comments
AuthorsOpenReview ↗

Author Rebuttal (Part 1/3)

We thank Reviewer whKN for the careful and constructive review. We appreciate the reviewer’s positive assessment of our diagnosis of QA-reward shortcutting and the breadth of our evaluation, as well as the important concern about disentangling the effect of our reward design from the stronger preference-level supervision available to QUIET. We address each concern below.

  • For W1 :

First, we agree that QUIET uses finer-grained supervision than the QA-RL baselines. While QA-RL uses downstream QA correctness as its training signal, QUIET directly targets chunk-level gold preference states constructed from utterance-grounded preference annotations. For PMv2, we construct these intermediate states using the existing preference annotations and utterance-level grounding, without collecting additional chunk-level annotations. This gold supervision is required only during training; at inference time, QUIET extracts preferences solely from the conversation history. QUIET therefore applies to training settings where grounded preference annotations are available, and we will explicitly acknowledge the cost of obtaining these annotations as a limitation when applying the method to real-world data. We do not attribute the performance gap relative to QA-RL solely to reward design. To assess the additional contribution of RL starting from the same gold-supervised SFT checkpoint, we present the Gold-SFT comparison below.

Second, the reviewer points out that the lack of an SFT-on-gold baseline makes it difficult to assess the additional benefit of F1-score RL post-training. We constructed the warmup SFT dataset for QUIET-diff by providing Sonnet-4.6 with each chunk and the gold preference state expected after processing that chunk, and asking it to generate oracle edit actions in JSON format. We evaluated the checkpoint trained on this dataset, before applying RL, as Gold-SFT. Since QUIET-diff starts RL training from this same checkpoint, comparing Gold-SFT with QUIET-diff allows us to assess the additional contribution of RL while keeping the gold-supervised SFT initialization and output format fixed.

ModelMethodPMv2 32kPMv2 128kFairPMv2 32kFairPMv2 128kF1 32kF1 128kPMv1 32kPMv1 128kPMv1 1M
Qwen3-4B-InstructQA-RL-Frozen43.635.044.040.510.71.154.245.239.3
Qwen3-4B-InstructGold-SFT46.845.846.642.918.18.659.843.439.9
Qwen3-4B-InstructQUIET-diff51.151.849.844.247.527.164.258.353.6
Llama3.2-3BQA-RL-Frozen35.535.738.537.30.71.452.538.235.0
Llama3.2-3BGold-SFT44.439.142.838.514.58.153.840.637.9
Llama3.2-3BQUIET-diff48.844.747.544.332.115.360.149.842.6
Qwen3-1.7BQA-RL-Frozen37.633.638.039.22.71.745.736.430.0
Qwen3-1.7BGold-SFT41.335.741.438.511.43.658.647.436.7
Qwen3-1.7BQUIET-diff43.639.442.941.324.46.457.653.953.1

Gold-SFT provides a strong baseline using gold preference supervision. Importantly, QUIET-diff further improves semantic F1 across all three extractor backbones at both context lengths, and improves downstream accuracy in most settings, with particularly large gains in long-context evaluation. These results demonstrate the additional benefit of the RL stage beyond the shared gold-supervised SFT initialization. We will add details of the Gold-SFT baseline and the construction of the QUIET-diff SFT dataset to the revised manuscript.

Third, circular metric of semantic F1. We agree that improvement on the same semantic-F1 evaluator used for training is insufficient, by itself, to establish better extraction quality. We therefore support our interpretation with two complementary analyses. Table 10 evaluates fixed extracted preference states using five alternative judges. Both QUIET variants outperform the QA-reward baselines under every judge, including Qwen, Gemini, and Sonnet models. This reduces the concern that the improvement is specific to GPT-5.4-mini. The gains also extend beyond semantic F1 to downstream MCQ accuracy. In Table 3, QUIET-diff improves over QA-RL-Frozen with diverse answer models including Qwen3-14B, Gemini-3.5-Flash, and Sonnet-4.6.

AuthorsOpenReview ↗

Author Rebuttal (Part 2/3)

Fourth, the reviewer raises the possibility that bulletization may have underestimated the F1 scores of the QA-RL methods. Our internal checks confirmed that F1 scores vary with the number of bullets, and that the number of bullets also varies depending on the LLM used for bulletization. We therefore concluded that splitting model outputs into sentences is a more appropriate approach than relying on an external LLM for bulletization. We thank the reviewer for this constructive suggestion. We have revised the paper to compute F1 by treating each sentence as a bullet, without a separate bulletization step. The largest increase is for QA-RL-Frozen (4B, 32k), 3.2%->10.7%, and the largest decrease is for QA-RL-Joint (3B, 128k), 5.0%->1.1%. Although this change affects the reported F1 scores, the QA-RL methods still score substantially below QUIET variants, so the main conclusions of the paper remain unchanged.

  • For W2 :

First, to address concerns about the quality of FairPMv2, we conducted human validation of both PMv2 and FairPMv2. For each benchmark, we evaluated approximately 10% of the items whose correct answers reflect GT user preferences, excluding cases involving third-party attributes or requests to forget information. This yielded 300 items per benchmark. Each item contained three answer choices, with the generic option removed because it does not reflect any user preference. Here we excluded generic option and the items where the generic option was correct, as including them would require a more complex choice-selection protocol beyond the scope of this validation. Our goal was to assess whether participants could achieve high accuracy when given the GT preference, while performing near chance (33.3%) without preference information. Thirteen participants were randomly assigned items without being told whether they came from PMv2 or FairPMv2, and the three answer choices were randomly shuffled. Each participant evaluated half of their assigned items under the Empty Pref condition and the other half under the GT Pref condition. The table below reports human three-choice MCQ accuracy (%) under each condition.

Human three-choice MCQ accuracy (%). The chance level is 33.3%.

BenchmarkEmpty PrefGT Pref
PMv256.296.9
FairPMv232.395.8

In the GT Pref condition, PMv2 and FairPMv2 achieved accuracies of 96.9% and 95.8%, respectively, indicating that the correct choices could be clearly identified from the provided preference information in both benchmarks. In the Empty Pref condition, PMv2 achieved 56.2% accuracy, above the chance level (33.3%), whereas FairPMv2 achieved 32.3%, close to the chance. These findings suggest that correct choices in PMv2 can often be identified without preference information, whereas FairPMv2 reduces this source of bias.

Second, thank you for catching this typo. The QA-RL extractor never receives the query qq, during either training or inference. Section 3.1 is correct: the extractor input is (P^i−1,ci)(\hat{P}_{i-1}, c_i). The qq in line 7 of Algorithm 1 was included by mistake and has been removed.

Third, there were numerical inconsistencies and copy-paste errors in the reported experimental results. We apologize for the confusion these errors caused regarding the reliability of our results. Due to space constraints, we cannot include the corrected Table 2 here; please refer to the revised manuscript, which we have uploaded. We thank the reviewer for identifying these inconsistencies, and we have audited all reported values against the raw evaluation outputs. These corrections do not change the method ranking or our main conclusions. The corrected values are now consistent across all occurrences in the figures, tables, main text, and appendix.

Fourth, the reviewer notes that conducting all experiments with a single seed makes it difficult to assess statistical significance. We therefore repeated the main experiments five times using different training seeds. The results below were obtained using Qwen3-4B-Instruct with a 32k context length.

Base model : Qwen3-4B-Instruct, context length=32k, 5 repetitions, unit: %

MethodPMv2FairPMv2Semantic F1PMv1
SFT34.23 ± 1.1528.23 ± 0.769.60 ± 0.7548.76 ± 1.10
QA-RL-Joint59.36 ± 0.4539.65 ± 0.944.80 ± 0.8756.77 ± 0.80
QA-RL-Swap38.02 ± 0.9838.90 ± 1.604.80 ± 0.8755.35 ± 1.46
QA-RL-Frozen46.43 ± 1.1343.64 ± 0.749.38 ± 0.7955.21 ± 0.64
QUIET-plain50.14 ± 1.5546.50 ± 1.5138.31 ± 0.8660.99 ± 1.39
QUIET-diff52.08 ± 1.0548.69 ± 1.4549.47 ± 2.9564.87 ± 1.18

These results across five training seeds support our conclusion that QUIET outperforms the QA-RL baselines on FairPMv2 and PMv1. We will conduct additional runs and include the results in the camera-ready version.

AuthorsOpenReview ↗

Author Rebuttal (Part 3/3)

For questions :

Most of the questions are addressed in our responses to the weaknesses above. We provide additional clarification below.

  • How were the chunk-level gold sets P*_i built and aligned across tokenizers? : The gold preferences come with utterance-level grounding in the conversation. We construct P*_i at the text level by including each preference once its grounded utterance has appeared by chunk i. Because this mapping is defined over the underlying dialogue text rather than tokenizer-specific token indices, the preference-to-evidence alignment does not depend on the extractor tokenizer.

  • How was the SFT warm-start data for QUIET-diff built? : We divided the PersonaMem-V2 training split into 5k-token chunks. For each chunk, we provided Sonnet-4.6 with both the chunk and the gold preference state expected after processing it, and asked it to generate oracle edit actions in JSON format. We have added details of the QUIET-diff SFT dataset construction to the appendix.

  • What is the accuracy of a choice-only (no-query) baseline? : We evaluated the Query+choice condition (equivalent to Empty Pref) and the Choice-only condition on 500 of the 5,000 items in each of PMv2 and FairPMv2. The table below reports the selection rates (%) for the correct option / generic option. In the Choice-only condition, we replaced the original query with the instruction “Choose the correct answer from the following choices.” The answer choices were randomly shuffled, and GPT-5.4-mini was used as the answer model.

BenchmarkQuery+choiceChoice-only
PMv220.8 / 56.229.0 / 56.2
FairPMv218.6 / 28.427.4 / 50.4

In the Query+choice condition, accuracy was below the 25% chance level on both benchmarks, with 95% confidence intervals of 17.5–24.6% for PMv2 and 15.4–22.2% for FairPMv2. In the Choice-only condition, the confidence interval included chance level only for FairPMv2. The generic option was selected at a high rate in all conditions except Query+choice on FairPMv2. For FairPMv2, providing the query substantially reduces generic-option selection, but does not improve accuracy; instead, the model appears to shift its selections toward the personalized distractors. We also observed a position bias: the model selected A least often and increasingly favored later options. (A:B:C:D=13.6:21.9:29.7:34.8)

  • What is the full training cost, including about 44.5K judge calls per epoch, compared with QA-RL-Frozen? : We thank the reviewer for raising the cost of semantic-F1 judging. We recomputed the training cost to include not only the training context and policy-generated tokens, but also all LLM API calls. QA-RL-Frozen use API calls for answer generation, while QUIET uses for semantic judgement. Per-epoch RL post-training cost is as follows.
MetricQA-RL-JointQA-RL-FrozenQUIET-plainQUIET-diff
Input context tokens588M588M27M27M
Policy-generated tokens241M317M31M8M
LLM API Input0142M24M25M
LLM API Output04M1M1M
LLM cost$0$126.7$22.5$23.2
Total processed tokens829M1,051M83M61M
  • Moderate the framing. “Unbiased” in the title overstates the claim. : We agree that “unbiased” would be too broad if interpreted as claiming the absence of all sources of bias. In QUIET, we use the term specifically to refer to independence from a predefined downstream QA/query set during training. We will clarify this scope explicitly throughout the paper and avoid broader claims of unbiasedness. If acceptable, we would therefore like to retain “Unbiased” in the QUIET acronym, while explicitly defining it as QA/query-independent and avoiding any broader use of the term elsewhere in the paper.

  • Retrieval methods outperform QUIET on AlpsBench EI. : Complete-form extraction methods must compress all extracted information into a fixed token budget (L=2,048L=2{,}048), whereas retrieval methods retrieve the top k=4 segments of the original history for each query without a token cap on the retrieved text. As user histories grow longer, complete-form extraction requires increasingly lossy compression, placing it at a disadvantage relative to retrieval. Retrieval methods may therefore perform better on benchmarks where the relevant evidence consists of direct cues in the original history or details that an extractor is likely to omit. This difference may help explain their stronger performance on AlpsBench EI.

  • We have added more citations and fixed typos in response to the reviewer’s feedback. We appreciate the reviewer’s helpful suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.