- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Persona Learning from Feedback (PLF), a framework for personalized AI assistants that learns reusable persona-specific LoRA adapters from natural-language feedback. Each adapter is trained through feedback-gated on-policy self-distillation, where a frozen teacher conditions on persona specifications and response-specific feedback to provide token-level supervision. At inference time, a supervised router selects or composes the learned adapters based on the user's request or dialogue context, without requiring user-specific fine-tuning.
The authors evaluate PLF on single-turn personalization and multi-turn tutoring with simulated students. The results show that PLF improves model-judged preference alignment over fixed-persona policies and router-based persona prompting. Additional experiments investigate persona composition, adaptation to changing preferences, and the effects of feedback gating and KL direction.
Critical strengths
1. A practical framework for reusable and adaptive personalization.
The paper presents a modular framework that separates persona learning from context-dependent persona selection. By learning independent LoRA adapters and selecting or composing them at inference time, the approach enables personalization without user-specific fine-tuning. This is a practical and potentially scalable direction for personalized AI assistants.
2. Evaluation across different personalization settings.
The paper evaluates the proposed approach in both single-turn and multi-turn settings, including experiments on combined preferences and preference switching. The comparisons against fixed-persona policies and routed persona prompting, together with ablation studies and a small human evaluation, provide useful empirical evidence for the effectiveness of the overall framework.
Critical weaknesses
1. Limited methodological novelty and insufficient baselines to establish the proposed contributions.
The proposed framework largely combines existing techniques, including modular LoRA-based personalization, contextual adapter routing, and feedback-conditioned on-policy distillation. Prior works such as Personalized Soups (Jang et al., 2023), PROPER (Zhang et al., 2025), and recent feedback-based self-distillation methods (Hübotter et al., 2026; Song et al., 2026) have already explored closely related components. While their integration is reasonable, the paper does not sufficiently clarify the methodological novelty beyond combining these existing techniques.
More importantly, the experimental comparisons do not adequately isolate the contribution of the proposed feedback-based learning mechanism. The main comparison against routed persona prompting demonstrates the benefits of learned adapters, but does not establish whether response-specific feedback or on-policy distillation is necessary. In particular, comparisons against persona-only distillation without response-specific feedback, conventional persona-conditioned LoRA SFT, and feedback-revised response SFT are missing. Without these baselines, it is difficult to determine whether the observed improvements stem from the proposed learning mechanism or simply from training persona-specific adapters.
2. Limited effectiveness and generality of persona composition and preference adaptation.
Although persona composition and adaptation are important contributions of the paper, their effectiveness appears inconsistent across different settings. For example, the impersonal-concise composition provides little improvement over the concise head alone, while compositions involving the thorough persona underperform prompting with both persona specifications. Moreover, the composition coefficients are selected using validation data, leaving the generality of the proposed composition strategy unclear.
Similarly, the preference-switching experiments reveal substantial limitations in adaptation. The router selects the newly matching persona at its first opportunity in only 39.8% of dialogues and fails to select it before the end of 33.8% of dialogues. Although the overall preference fit improves over fixed-persona policies, these results suggest that the proposed approach is not yet consistently effective at adapting to changing preferences. This weakens the empirical support for the claimed generality and adaptability of the framework.
[1] Personalized Soups
Jang, J., Kim, S., Lin, B. Y., et al. (2023). Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging. arXiv:2310.11564.
https://arxiv.org/abs/2310.11564
[2] PROPER
Zhang, L., Wu, J., Zhou, D., & He, Y. (2025). PROPER: A Progressive Learning Framework for Personalized Large Language Models with Group-Level Adaptation. Proceedings of ACL 2025, pp. 16399–16411.
https://aclanthology.org/2025.acl-long.800/
[3] Reinforcement Learning via Self-Distillation (SDPO)
Hübotter, J., Lübeck, F., Behric, L., et al. (2026). Reinforcement Learning via Self-Distillation. arXiv:2601.20802.
https://arxiv.org/abs/2601.20802
[4] Expanding the Capabilities of Reinforcement Learning via Text Feedback (RLTF-SD)
Song, Y., Chen, L., Tajwar, F., et al. (2026). Expanding the Capabilities of Reinforcement Learning via Text Feedback. arXiv:2602.02482.
Questions for authors
-
Could the authors provide comparisons against persona-only on-policy distillation (without response-specific feedback) and conventional persona-conditioned LoRA SFT under comparable training budgets? These comparisons would help clarify the specific contribution of feedback-conditioned distillation beyond standard persona adaptation.
-
Could the authors provide a more detailed held-out comparison between the proposed signed adapter composition and prompting with both persona specifications across different preference combinations? In particular, what explains the weaker performance for some combinations, and how sensitive are the results to the validation-selected composition coefficients?
-
For the preference-switching experiments, could the authors further analyze the causes of routing failures and adaptation delays? For example, how much of the performance gap is attributable to the router rather than the learned persona adapters? A comparison with an oracle that switches at the first opportunity after observing the changed preference would help isolate this effect.
4 comments
Response to Reviewer pySd (Part 2/2)
Response to Weakness 2 and Question 2: Persona Composition
Thanks for the suggestion. We provide a detailed held-out comparison of signed adapter composition against prompting with both persona specifications. The table reports PLF wins, ties, and losses on correctly routed paired-preference requests.
| Preference pair | Samples | W / T / L | Win rate |
|---|---|---|---|
| Warm + Concise | 48 | 30 / 8 / 10 | 71% |
| Warm + Thorough | 59 | 30 / 14 / 15 | 63% |
| Impersonal + Concise | 55 | 30 / 6 / 19 | 60% |
| Impersonal + Thorough | 60 | 28 / 11 / 21 | 56% |
| Overall | 222 | 118 / 39 / 65 | 62% |
Held-out performance. Signed composition achieves more wins than losses against both-persona prompting across all four combinations, including both thorough pairs. The overall win rate remains 62% across all 247 paired-preference requests when routing errors are included.
Variation across combinations. We manually inspect the saved responses and judge assessments and find two main reasons for weaker gains over individual heads of certain combinations. Impersonal–concise offers limited improvement over the concise head (53%) because both often produce similarly brief, neutral responses. For warm–thorough, the thorough head already provides detailed explanations, while the mixture sometimes adds repetitive or overly sentimental language without improving clarity or coverage. These observations explain why composition does not consistently outperform individual heads, despite its overall advantage over both-persona prompting.
Coefficient sensitivity. We use the same signed composition rule across all preference combinations, with coefficients selected on validation. For voice coefficients of {1, 0.5, 0.25}, validation win rates against both-persona prompting are 32%/56%/64% for warm–thorough and 50%/56%/66% for impersonal–thorough. Both combinations favor the same coefficient (0.25), which is fixed before held-out evaluation.
Together, these results support signed composition across the evaluated preference combinations, while showing that gains depend on behavioral complementarity and coefficient calibration. We will include the held-out comparisons and coefficient analysis in the revised paper.
Response to Weakness 2 and Question 3: Analyzing Routing Failures and Adaptation Delays
Thanks for the suggestion. We conducted additional analyses to distinguish routing failures from insufficient observable preference evidence and assess their impact on adaptation.
First-opportunity oracle. We added a lagged oracle that uses the same learned persona adapters as PLF but selects the matching head at the first post-change opportunity (j+1) using the hidden preference. PLF achieves a 42.5% preference win rate against this oracle overall, increasing to 48.9% in the 398 dialogues where PLF selects the matching head at the first opportunity. Since both use the same adapters, this comparison isolates the effect of persona selection.
Causes of adaptation delays. We used Claude Opus 5.5 to identify explicit preference requests from student replies and their preceding tutor turns, without access to hidden preferences or routing decisions. Among the 602 first-opportunity misses, 493 contained no clearly expressed new preference before PLF switched or the dialogue ended. Similarly, 275 of the 338 no-detection dialogues lacked explicit preference evidence. Of the remaining 109 first-opportunity misses with explicit requests, 107 involved changes toward direct instruction.
Routing with explicit evidence. Following a clear preference request, PLF selects the matching head at the next turn in 97.6% of cases toward interactive guidance and 58.7% toward direct instruction. Restricting the same frozen router to the last two exchanges improves responsiveness to late direct-instruction requests from 40.9% to 65.9%, while maintaining comparable fixed-preference accuracy (92.8% vs. 92.0%).
These findings show that many adaptation delays reflect insufficient observable preference evidence, while the remaining failures are concentrated in routing toward direct instruction and sensitivity to dialogue history. We will include these analyses in the revised paper.
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns, particularly by introducing the three missing baselines. These results substantially strengthen the empirical evaluation and help clarify the contribution of feedback-conditioned learning.
However, my main concern regarding methodological novelty remains. In particular, the relatively small improvement over Feedback imitation, especially in the multi-turn setting, leaves it unclear whether the proposed on-policy distillation mechanism provides a substantial advantage over conventional feedback-conditioned SFT.
The additional composition and routing analyses are also helpful, but they primarily clarify the limitations rather than fully resolving my concerns about generality and adaptation reliability.
Overall, while the rebuttal has improved my assessment of the empirical support, I remain insufficiently convinced of the methodological contribution to change my recommendation. I will therefore maintain my current score.
Thank you for the follow-up and for recognizing the stronger empirical support. We would like to clarify the methodological contribution underlying PLF.
The central question PLF addresses is how to turn response-specific, preference-dependent feedback into reusable behaviors that can be selected and combined across different interactions. Feedback identifies what should change in a particular response, but does not directly determine how that correction should generalize to future requests or when the resulting behavior should be applied. PLF connects these two decisions by learning independent preference-specific adapters from hindsight feedback and selecting or composing them using observable interaction context, without user-specific finetuning.
This learning-and-reuse framework, rather than the on-policy distillation objective alone, is our claimed methodological contribution. Feedback imitation retains the same modular persona architecture and contextual selection, so its comparison isolates the additional benefit of on-policy training rather than the contribution of the overall framework. Persona distillation and shared-policy distillation (in Appendix E.2) provide complementary evidence for response-specific feedback and independent persona learning, while the composition and switching experiments demonstrate how these learned behaviors can be reused beyond their individual training settings.
Together, these results support a framework for converting response-specific feedback into reusable, contextually selected behaviors, rather than an isolated improvement over feedback-conditioned SFT. We hope this clarification better highlights the distinct contributions of our work, and we would sincerely appreciate your reconsideration.
Response to Reviewer pySd (Part 1/2)
Thank you for your detailed and constructive feedback, each of your concerns is addressed in our response below.
Response to Weakness 1 and Question 1: Methodology Novelty and Additional Baselines
Methodological novelty. Thanks for highlighting these connections. While PLF builds on existing techniques, our contribution is learning reusable preference-specific behaviors from response-specific feedback and adapting their selection to users' turn-specific needs. The key challenge is that the same response may require different corrections under different preferences, while the appropriate behavior must be inferred from the current interaction.
Personalized Soups [1] learns separate policies for predefined preference objectives and merges them according to user-specified preferences. PLF instead learns behaviors from response-specific feedback and automatically selects or composes them based on the observable request or dialogue. PROPER [2] routes among group- and user-level adaptations learned from user histories, whereas PLF learns behavior-specific adapters reusable across users and selects them at each turn without user-specific finetuning. SDPO [3] and RLTF-SD [4] develop feedback-conditioned learning for improving task policies. PLF applies this learning principle to preference-dependent behavior, using persona specifications and response-specific feedback to supervise separate adapters that can later be selected or composed without feedback at inference.
These distinctions are reflected in our additional persona-only and shared-policy comparisons. All four works are discussed in Related Work, and we will clarify PLF's methodological contribution in the revision.
Additional baselines. We added three training alternatives: Persona imitation, which trains persona adapters using conventional persona-conditioned LoRA SFT without feedback; Persona distillation, which uses persona-only on-policy distillation without response-specific feedback; and Feedback imitation, which trains persona adapters using SFT on feedback-conditioned teacher responses. These comparisons assess the contributions of response-specific feedback and on-policy distillation beyond standard persona adaptation.
The table reports raw PLF wins, ties, and losses on the test sets.
The results show that PLF's improvements extend beyond standard persona adaptation. PLF achieves more wins than losses against both Persona imitation and Persona distillation in both settings, supporting the value of response-specific feedback beyond persona-only training. Feedback imitation is more competitive, indicating that feedback-conditioned supervision contributes substantially, while on-policy distillation provides additional gains, particularly in single-turn personalization. All baselines use the same backbone, LoRA configuration, and nominal training schedule. Feedback-free baselines do not use feedback-dependent gating. We will incorporate these comparisons and implementation details into the revised paper.