- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies adaptive personalization under preference epistemic uncertainty, where LLMs must adjust their response strategies as user preferences become uncertain or change. The authors introduce a multilingual benchmark with 250 personas and approximately 25K conversational turns, revealing a substantial gap between recognizing appropriate strategies and generating responses that follow them. To address this, they propose a GRPO reward that aligns strategy distributions decoded from hidden representations with human preferences, demonstrating improvements in strategy calibration and generalization.
Critical strengths
-
The paper goes beyond preference retrieval to study whether models adapt their response strategies as preference reliability changes. The recognition–generation comparison and oracle-strategy experiments provide compelling evidence that the main difficulty lies in selecting appropriate strategies rather than executing them.
-
The benchmark covers diverse personas and languages, with human validation supporting its quality. The proposed Link reward also consistently improves strategy calibration and user acceptability across multiple models.
Critical weaknesses
-
Unclear contribution of representation-level supervision. Unlike the binary strategy calibration reward, the Link reward provides a continuous distributional training signal. It is therefore unclear whether the improvements arise from supervising internal representations or simply from providing richer strategy-level feedback. A baseline using the same KL reward computed from an output-level strategy classifier is needed to establish the specific benefit of representation-level supervision.
-
Limited causal evidence and potentially restrictive evaluation criteria. The strategy probe uses hidden activations from the first eight generated tokens, which may primarily encode strategy cues already expressed in the output rather than an internal strategy-selection mechanism. Moreover, the state-dependent binary acceptability criteria may incorrectly penalize valid responses, particularly when a model directly acts on newly revised preferences. These issues raise questions about the interpretation of both the observed failures and the proposed improvements.
Questions for authors
-
How does the Link reward compare against an output-level strategy-distribution reward using the same KL objective? This comparison would help isolate the contribution of representation-level supervision.
-
Can the strategy be reliably decoded from hidden activations before generation begins? Does Link training increase the causal influence of the identified state direction on subsequent strategy selection?
-
How are responses that directly act on newly established preferences handled in the Revised state? Do the reported findings hold under human evaluation of generated responses rather than LLM-based strategy judgments?
We thank the reviewer for the comments and address the main questions and weaknesses below.
Question 1: How does link reward compare against an output-level strategy-distribution reward
To isolate what activations add, we trained GRPO on Gemma-3-12B with the same KL objective, human target Pref(s), λ and training budget, but with the strategy distribution predicted from the generated response by an output-level classifier instead of from activations by the probe (Link′). In-distribution, Link′ is close to Link (SC 75 vs. 77; R 3.94 vs. 3.95; UA 63 vs. 65; GRPO: 70, 3.89, 60). On the benchmark’s unseen personas, Link′ falls further behind (SC 64 vs. 70; R 3.36 vs. 3.45; UA 57 vs. 60; GRPO: 59, 3.31, 53). Its generalization gap (in-distribution minus benchmark SC) is 11, the same as GRPO’s, while Link’s is 7 (Link vs. Link′ on benchmark SC: [McNemar p < 0.05]). We conclude that an output-level distributional reward accounts for much of the in-distribution improvement, while computing the reward from activations improves generalization to unseen personas.
Question 2: Can the strategy be reliably decoded from hidden activations before generation begins?
We refit the probe on the last prompt token, right before the first reply token, at the same layer, on all four open-weight models. Strategy is decodable at 49-54% (chance 20%), versus 70.0% from generated tokens, so a signal exists before any output. We further experiment with causal ablation to address the reviewer’s point on the causal influence of the identified state direction on subsequent strategy selection. On Gemma-3-12B, over the full benchmark (5.6K prompts), removing the state subspace lowers the SC metric from 59 to 55 in the GRPO model and from 70 to 62 in the Link reward model, compared with 57 and 67 when a random subspace of the same rank is removed. The larger state-specific drop for Link (8 vs. 4 points; difference p < 0.05, paired test) is consistent with Link training increasing the subspace’s influence on strategy selection. This experiment shows supporting evidence that Link reward training does increase the model’s reliance on the preference epistemic state subspace.
Question 3: How are responses that directly act on newly established preferences handled in the Revised state?
In the Revised epistemic state, a response that acts directly on the newly stated preference is labeled recalibrate, because it updates behavior once new evidence contradicts the earlier preference. Thus, recalibrate is an acceptable strategy for this state (Table 1). Strategy exploit in Revised means continuing to act on the superseded (old) preference, which is not an acceptable strategy. To check that the findings do not depend on LLM judgments, we had two annotators label 300 generated responses from Gemma-3-12B (100 each, from the untrained, GRPO and Link reward models, on the same prompts), blind to the jury labels and to which model wrote each response. They agreed with the jury at κ = 0.74 for strategy and κ = 0.80 for acceptability. Under their labels, 38% of the untrained model’s responses, 57% of GRPO’s and 66% of Link’s were acceptable, the same ordering as our jury-based results.
We hope these responses and the additional results address your concerns. If they do, we would appreciate it if you would consider updating your assessment of the paper. We are happy to answer any further questions during the discussion period.