acceptodds
Under review as a conference paper at ICLR 2027

Persona Learning from Feedback for Personalized AI Assistants

Abstract

Personalizing AI assistants requires adapting their behavior to users’ prefer- ences and task contexts. Existing work elicits assistant personas through persona prompting or finetuning on persona-conditioned generations. However, persona descriptions alone do not identify what to change or preserve in a particular re- sponse, or which behaviors suit the current request. We introduce Persona Learn- ing from Feedback (PLF), a framework that learns reusable, preference-grounded assistant personas as independent LoRA adapters on a shared frozen backbone. During training, a frozen self-teacher conditions on persona specifications and response-specific feedback to provide token-level supervision on each adapter’s own generated prefixes. A feedback gate focuses corrective updates on responses judged to need improvement. At deployment, a supervised router uses the request or dialogue to select a single adapter or a fixed composition, without user-specific finetuning. We evaluate PLF on single-turn personalization with model-generated critiques and multi-turn tutoring with simulated students’ reactions. Across both settings, PLF improves model-judged preference fit over fixed-persona policies and persona prompting with the same router, demonstrating how feedback-trained behaviors can be reused for adaptive assistant personalization.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
personalization, alignment, learning from feedback

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pySdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Persona Learning from Feedback (PLF), a framework for personalized AI assistants that learns reusable persona-specific LoRA adapters from natural-language feedback. Each adapter is trained through feedback-gated on-policy self-distillation, where a frozen teacher conditions on persona specifications and response-specific feedback to provide token-level supervision. At inference time, a supervised router selects or composes the learned adapters based on the user's request or dialogue context, without requiring user-specific fine-tuning.

The authors evaluate PLF on single-turn personalization and multi-turn tutoring with simulated students. The results show that PLF improves model-judged preference alignment over fixed-persona policies and router-based persona prompting. Additional experiments investigate persona composition, adaptation to changing preferences, and the effects of feedback gating and KL direction.

Critical strengths

1. A practical framework for reusable and adaptive personalization.

The paper presents a modular framework that separates persona learning from context-dependent persona selection. By learning independent LoRA adapters and selecting or composing them at inference time, the approach enables personalization without user-specific fine-tuning. This is a practical and potentially scalable direction for personalized AI assistants.

2. Evaluation across different personalization settings.

The paper evaluates the proposed approach in both single-turn and multi-turn settings, including experiments on combined preferences and preference switching. The comparisons against fixed-persona policies and routed persona prompting, together with ablation studies and a small human evaluation, provide useful empirical evidence for the effectiveness of the overall framework.

Critical weaknesses

1. Limited methodological novelty and insufficient baselines to establish the proposed contributions.

The proposed framework largely combines existing techniques, including modular LoRA-based personalization, contextual adapter routing, and feedback-conditioned on-policy distillation. Prior works such as Personalized Soups (Jang et al., 2023), PROPER (Zhang et al., 2025), and recent feedback-based self-distillation methods (Hübotter et al., 2026; Song et al., 2026) have already explored closely related components. While their integration is reasonable, the paper does not sufficiently clarify the methodological novelty beyond combining these existing techniques.

More importantly, the experimental comparisons do not adequately isolate the contribution of the proposed feedback-based learning mechanism. The main comparison against routed persona prompting demonstrates the benefits of learned adapters, but does not establish whether response-specific feedback or on-policy distillation is necessary. In particular, comparisons against persona-only distillation without response-specific feedback, conventional persona-conditioned LoRA SFT, and feedback-revised response SFT are missing. Without these baselines, it is difficult to determine whether the observed improvements stem from the proposed learning mechanism or simply from training persona-specific adapters.

2. Limited effectiveness and generality of persona composition and preference adaptation.

Although persona composition and adaptation are important contributions of the paper, their effectiveness appears inconsistent across different settings. For example, the impersonal-concise composition provides little improvement over the concise head alone, while compositions involving the thorough persona underperform prompting with both persona specifications. Moreover, the composition coefficients are selected using validation data, leaving the generality of the proposed composition strategy unclear.

Similarly, the preference-switching experiments reveal substantial limitations in adaptation. The router selects the newly matching persona at its first opportunity in only 39.8% of dialogues and fails to select it before the end of 33.8% of dialogues. Although the overall preference fit improves over fixed-persona policies, these results suggest that the proposed approach is not yet consistently effective at adapting to changing preferences. This weakens the empirical support for the claimed generality and adaptability of the framework.

[1] Personalized Soups

Jang, J., Kim, S., Lin, B. Y., et al. (2023). Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging. arXiv:2310.11564.

https://arxiv.org/abs/2310.11564

[2] PROPER

Zhang, L., Wu, J., Zhou, D., & He, Y. (2025). PROPER: A Progressive Learning Framework for Personalized Large Language Models with Group-Level Adaptation. Proceedings of ACL 2025, pp. 16399–16411.

https://aclanthology.org/2025.acl-long.800/

[3] Reinforcement Learning via Self-Distillation (SDPO)

Hübotter, J., Lübeck, F., Behric, L., et al. (2026). Reinforcement Learning via Self-Distillation. arXiv:2601.20802.

https://arxiv.org/abs/2601.20802

[4] Expanding the Capabilities of Reinforcement Learning via Text Feedback (RLTF-SD)

Song, Y., Chen, L., Tajwar, F., et al. (2026). Expanding the Capabilities of Reinforcement Learning via Text Feedback. arXiv:2602.02482.

https://arxiv.org/abs/2602.02482

Questions for authors
  1. Could the authors provide comparisons against persona-only on-policy distillation (without response-specific feedback) and conventional persona-conditioned LoRA SFT under comparable training budgets? These comparisons would help clarify the specific contribution of feedback-conditioned distillation beyond standard persona adaptation.

  2. Could the authors provide a more detailed held-out comparison between the proposed signed adapter composition and prompting with both persona specifications across different preference combinations? In particular, what explains the weaker performance for some combinations, and how sensitive are the results to the validation-selected composition coefficients?

  3. For the preference-switching experiments, could the authors further analyze the causes of routing failures and adaptation delays? For example, how much of the performance gap is attributable to the router rather than the learned persona adapters? A comparison with an oracle that switches at the first opportunity after observing the changed preference would help isolate this effect.

4 comments
AuthorsOpenReview ↗

Response to Reviewer pySd (Part 1/2)

Thank you for your detailed and constructive feedback, each of your concerns is addressed in our response below.

Response to Weakness 1 and Question 1: Methodology Novelty and Additional Baselines

Methodological novelty. Thanks for highlighting these connections. While PLF builds on existing techniques, our contribution is learning reusable preference-specific behaviors from response-specific feedback and adapting their selection to users' turn-specific needs. The key challenge is that the same response may require different corrections under different preferences, while the appropriate behavior must be inferred from the current interaction.

Personalized Soups [1] learns separate policies for predefined preference objectives and merges them according to user-specified preferences. PLF instead learns behaviors from response-specific feedback and automatically selects or composes them based on the observable request or dialogue. PROPER [2] routes among group- and user-level adaptations learned from user histories, whereas PLF learns behavior-specific adapters reusable across users and selects them at each turn without user-specific finetuning. SDPO [3] and RLTF-SD [4] develop feedback-conditioned learning for improving task policies. PLF applies this learning principle to preference-dependent behavior, using persona specifications and response-specific feedback to supervise separate adapters that can later be selected or composed without feedback at inference.

These distinctions are reflected in our additional persona-only and shared-policy comparisons. All four works are discussed in Related Work, and we will clarify PLF's methodological contribution in the revision.


Additional baselines. We added three training alternatives: Persona imitation, which trains persona adapters using conventional persona-conditioned LoRA SFT without feedback; Persona distillation, which uses persona-only on-policy distillation without response-specific feedback; and Feedback imitation, which trains persona adapters using SFT on feedback-conditioned teacher responses. These comparisons assess the contributions of response-specific feedback and on-policy distillation beyond standard persona adaptation.

The table reports raw PLF wins, ties, and losses on the test sets.

PLF versusSingle-turn QA: wins / ties / lossesMulti-turn Tutoring: wins / ties / losses
Base model332 / 78 / 89589 / 107 / 304
Persona prompting264 / 88 / 147494 / 118 / 388
Persona imitation245 / 92 / 162522 / 117 / 361
Persona distillation244 / 107 / 148494 / 136 / 370
Feedback imitation199 / 145 / 155455 / 127 / 418

The results show that PLF's improvements extend beyond standard persona adaptation. PLF achieves more wins than losses against both Persona imitation and Persona distillation in both settings, supporting the value of response-specific feedback beyond persona-only training. Feedback imitation is more competitive, indicating that feedback-conditioned supervision contributes substantially, while on-policy distillation provides additional gains, particularly in single-turn personalization. All baselines use the same backbone, LoRA configuration, and nominal training schedule. Feedback-free baselines do not use feedback-dependent gating. We will incorporate these comparisons and implementation details into the revised paper.

AuthorsOpenReview ↗

Response to Reviewer pySd (Part 2/2)

Response to Weakness 2 and Question 2: Persona Composition

Thanks for the suggestion. We provide a detailed held-out comparison of signed adapter composition against prompting with both persona specifications. The table reports PLF wins, ties, and losses on correctly routed paired-preference requests.

Preference pairSamplesW / T / LWin rate
Warm + Concise4830 / 8 / 1071%
Warm + Thorough5930 / 14 / 1563%
Impersonal + Concise5530 / 6 / 1960%
Impersonal + Thorough6028 / 11 / 2156%
Overall222118 / 39 / 6562%

Held-out performance. Signed composition achieves more wins than losses against both-persona prompting across all four combinations, including both thorough pairs. The overall win rate remains 62% across all 247 paired-preference requests when routing errors are included.

Variation across combinations. We manually inspect the saved responses and judge assessments and find two main reasons for weaker gains over individual heads of certain combinations. Impersonal–concise offers limited improvement over the concise head (53%) because both often produce similarly brief, neutral responses. For warm–thorough, the thorough head already provides detailed explanations, while the mixture sometimes adds repetitive or overly sentimental language without improving clarity or coverage. These observations explain why composition does not consistently outperform individual heads, despite its overall advantage over both-persona prompting.

Coefficient sensitivity. We use the same signed composition rule across all preference combinations, with coefficients selected on validation. For voice coefficients of {1, 0.5, 0.25}, validation win rates against both-persona prompting are 32%/56%/64% for warm–thorough and 50%/56%/66% for impersonal–thorough. Both combinations favor the same coefficient (0.25), which is fixed before held-out evaluation.

Together, these results support signed composition across the evaluated preference combinations, while showing that gains depend on behavioral complementarity and coefficient calibration. We will include the held-out comparisons and coefficient analysis in the revised paper.


Response to Weakness 2 and Question 3: Analyzing Routing Failures and Adaptation Delays

Thanks for the suggestion. We conducted additional analyses to distinguish routing failures from insufficient observable preference evidence and assess their impact on adaptation.

First-opportunity oracle. We added a lagged oracle that uses the same learned persona adapters as PLF but selects the matching head at the first post-change opportunity (j+1) using the hidden preference. PLF achieves a 42.5% preference win rate against this oracle overall, increasing to 48.9% in the 398 dialogues where PLF selects the matching head at the first opportunity. Since both use the same adapters, this comparison isolates the effect of persona selection.

Causes of adaptation delays. We used Claude Opus 5.5 to identify explicit preference requests from student replies and their preceding tutor turns, without access to hidden preferences or routing decisions. Among the 602 first-opportunity misses, 493 contained no clearly expressed new preference before PLF switched or the dialogue ended. Similarly, 275 of the 338 no-detection dialogues lacked explicit preference evidence. Of the remaining 109 first-opportunity misses with explicit requests, 107 involved changes toward direct instruction.

Routing with explicit evidence. Following a clear preference request, PLF selects the matching head at the next turn in 97.6% of cases toward interactive guidance and 58.7% toward direct instruction. Restricting the same frozen router to the last two exchanges improves responsiveness to late direct-instruction requests from 40.9% to 65.9%, while maintaining comparable fixed-preference accuracy (92.8% vs. 92.0%).

These findings show that many adaptation delays reflect insufficient observable preference evidence, while the remaining failures are concentrated in routing toward direct instruction and sensitivity to dialogue history. We will include these analyses in the revised paper.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns, particularly by introducing the three missing baselines. These results substantially strengthen the empirical evaluation and help clarify the contribution of feedback-conditioned learning.

However, my main concern regarding methodological novelty remains. In particular, the relatively small improvement over Feedback imitation, especially in the multi-turn setting, leaves it unclear whether the proposed on-policy distillation mechanism provides a substantial advantage over conventional feedback-conditioned SFT.

The additional composition and routing analyses are also helpful, but they primarily clarify the limitations rather than fully resolving my concerns about generality and adaptation reliability.

Overall, while the rebuttal has improved my assessment of the empirical support, I remain insufficiently convinced of the methodological contribution to change my recommendation. I will therefore maintain my current score.

Thank you for the follow-up and for recognizing the stronger empirical support. We would like to clarify the methodological contribution underlying PLF.

The central question PLF addresses is how to turn response-specific, preference-dependent feedback into reusable behaviors that can be selected and combined across different interactions. Feedback identifies what should change in a particular response, but does not directly determine how that correction should generalize to future requests or when the resulting behavior should be applied. PLF connects these two decisions by learning independent preference-specific adapters from hindsight feedback and selecting or composing them using observable interaction context, without user-specific finetuning.

This learning-and-reuse framework, rather than the on-policy distillation objective alone, is our claimed methodological contribution. Feedback imitation retains the same modular persona architecture and contextual selection, so its comparison isolates the additional benefit of on-policy training rather than the contribution of the overall framework. Persona distillation and shared-policy distillation (in Appendix E.2) provide complementary evidence for response-specific feedback and independent persona learning, while the composition and switching experiments demonstrate how these learned behaviors can be reused beyond their individual training settings.

Together, these results support a framework for converting response-specific feedback into reusable, contextually selected behaviors, rather than an isolated improvement over feedback-conditioned SFT. We hope this clarification better highlights the distinct contributions of our work, and we would sincerely appreciate your reconsideration.

Reviewer_e4djOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes Persona Learning from Feedback (PLF), which separates assistant personalization into learning reusable personas and selecting them from context. Each persona is represented by an independent LoRA adapter on a frozen backbone. A frozen self-teacher conditions on persona specifications and response-specific feedback to supervise student-generated prefixes, while feedback gating excludes satisfactory responses from distillation. At deployment, a supervised router selects an adapter or a signed weight-space composition without user-specific fine-tuning. The authors evaluate PLF on PRISM-based single-turn preference adaptation and multi-turn tutoring with simulated students, examining persona composition, preference switching, feedback gating, and KL direction. Model-based judgments and a small human evaluation generally favor PLF over the base model, fixed-persona policies, and persona prompting with the same router. Against routed prompting, model-judged preference win rates are approximately 55% in both settings. Composition benefits depend on the personas combined, and routing remains a bottleneck when preferences change.

Critical strengths
  1. The same-router prompting baseline provides direct evidence that the learned personas add value. PLF achieves preference win rates of 54.8% and 55.3% against routed persona prompting in single-turn personalization and multi-turn tutoring, respectively, with directionally consistent results from a small human evaluation (Figure 2; Tables 6–7). In particular, the single-turn comparison holds persona selection fixed through identical requests and routing, so the improvement cannot be attributed solely to better preference identification. Although the gains are modest, this controlled comparison is consequential evidence for the usefulness of the learned behaviors.

  2. The framework clearly separates behavior learning from contextual selection and demonstrates feasibility with two feedback sources. Independent persona adapters are learned through a common distillation procedure using either explicit critiques or subsequent student reactions, then reused through routing and parameter composition without user-specific fine-tuning (Figure 1). Composition and preference-switching experiments illustrate the practical purpose of this design (Tables 1–2). This gives the work significance beyond training a fixed response style and provides a modular framework that facilitates extension and analysis.

Critical weaknesses
  1. Missing controls make the central methodological contribution difficult to establish. The same-router prompting comparison demonstrates additional value from trained adapters, but simultaneously changes parameter training, distillation, and feedback use; it does not isolate the benefit of response-specific feedback (Figure 2; Table 6). The ablations focus on gating and KL direction (Tables 18–22), without persona-only distillation or a shared conditional model trained with equivalent supervision. These controls would test whether response-specific feedback and independent persona modules are necessary. Their absence materially weakens the case for technical advancement: the complete system works, but its core design choices are insufficiently justified.

  2. The validation is narrow and provides limited evidence of reliable adaptive personalization. Single-turn evaluation uses requests rewritten to express assigned preferences, while tutoring relies primarily on simulated students with two predefined preferences, restricting the evidence to controlled adaptation within a small persona catalog (Appendices B.1–B.2). The human evaluation judges generated outputs rather than real user interactions (Table 7). Even under controlled preference switching, only 39.8% of dialogues select the newly matching persona at the first opportunity, and 338/1,000 never do before termination (Table 16). These limitations substantially weaken confidence that the framework can reliably accommodate naturally expressed, evolving user needs.

Questions for authors
  1. What is the independent contribution of response-specific feedback? The prompting comparison in Figure 2 and Table 6 changes both parameter training and feedback use. Do the authors have persona-only distillation results that match the training budget and gating setup while removing only the feedback from the teacher input? If so, does PLF retain its advantage? This comparison would directly affect my assessment of response-specific feedback as a central contribution.
  2. What advantage do independent persona adapters provide over a shared conditional model? Do the authors have results for a shared model trained with the same feedback supervision and conditioned on persona labels? If so, please clarify the parameter counts, training budgets, and performance differences. Otherwise, what current evidence distinguishes the benefit of independent modules from that of conditional training more generally? This bears directly on the architectural contribution.
  3. How much routing failure reflects insufficient preference evidence in the observable input? Appendix B.1 states that preference-readability checks were recorded for rewritten requests but were not used for filtering. What were their pass rates, and do routing and preference-evaluation results differ substantially according to these checks? For the 338 dialogues without a detected switch in Table 16, is there evidence that the student clearly expressed the new preference in the observable replies? This would help distinguish inference failures from evaluation targets that are insufficiently supported by the input, materially affecting the interpretation of adaptation performance.
6 comments
AuthorsOpenReview ↗

Response to Reviewer e4dj (Part 1/3)

We appreciate your thorough and helpful feedback. Below, we respond to each of your concerns and questions.

Response to Weakness 1 and Questions 1-2: Additional Baseline Comparisons

Thanks for the questions. We added two targeted baselines: Persona distillation follows PLF's on-policy distillation setup but removes response-specific feedback from the teacher input; Shared-policy distillation trains a shared conditional model using the same feedback supervision as PLF, with a single adapter conditioned on persona labels.

The table reports raw PLF wins, ties, and losses on the test sets.

PLF versusSingle-turn QA: wins / ties / lossesMulti-turn Tutoring: wins / ties / losses
Base model332 / 78 / 89589 / 107 / 304
Persona prompting264 / 88 / 147494 / 118 / 388
Persona distillation244 / 107 / 148494 / 136 / 370
Shared-policy distillation219 / 110 / 170498 / 118 / 384

Response-specific feedback. PLF achieves more wins than losses against Persona distillation in both settings, supporting the contribution of response-specific feedback beyond persona-only distillation.

Independent persona adapters. PLF also achieves more wins than losses against Shared-policy distillation in both settings under equivalent feedback supervision, supporting the advantage of independent persona adapters over a shared conditional model.

Both baselines use rank-32 LoRA and the same nominal training schedule as PLF. Feedback-dependent gating is omitted when feedback is unavailable. PLF uses four adapters in single-turn (264M vs. 66M parameters) and two in tutoring (168M vs. 84M), so total parameter counts differ. Shared-policy comparisons use the same selected preferences for both systems. We will incorporate these comparisons and implementation details into the revised paper.

AuthorsOpenReview ↗

Response to Reviewer e4dj (Part 2/3)

Response to Weakness 2: Evaluation Scope and Human Validation.

Evaluation design. Our controlled settings test whether learned behaviors transfer across held-out requests and topics and adapt to combined or changing preferences. Single-turn request rewriting follows the evaluation approach in APM [1], while controlled preference following is also studied in PrefEval [2]. Our multi-turn simulation builds on Education Dialogue [3], with related simulated-user evaluation approaches studied in [4, 5]. These protocols allow us to evaluate adaptation against defined preference targets without revealing their labels to the assistant.

Our persona choices are also grounded in prior evidence. The warm–impersonal and concise–thorough contrasts correspond to preference dimensions identified in PRISM-X [6], with descriptions constructed from human-written PRISM preferences [7]. The tutoring preferences follow the direct and interactive teaching modes in Education Dialogue [3].

Human-guided calibration. We iteratively refine both the student simulator and judge prompts using human annotations. Three annotators assess 50 tutor–student contexts for consistency with the assigned learning preference, while another 50 response pairs are used to calibrate the judge against human preference judgments. Both prompts are frozen before test evaluation, with calibration examples excluded from independent validation.

Independent human validation. We additionally evaluated 96 single-turn response pairs and 100 tutoring dialogue pairs comparing PLF with same-router persona prompting. Three independent annotators assessed each pair using anonymized outputs and randomized presentation order. We report raw agreement and Cohen's κ, with human–human agreement as a reference:

SettingLLM–human agreementLLM–human κHuman–human agreementHuman–human κ
Single-turn (96 pairs)71.5%0.45476.0%0.517
Tutoring (100 pairs)78.3%0.62880.3%0.659

Values average the three annotator comparisons. The LLM judge achieves agreement close to the human–human reference, supporting its use for preference-fit evaluation. We will clarify that these evaluations provide controlled evidence of behavioral adaptation and human validation of preference judgments, rather than a direct evaluation of live-user interactions.


References

[1] APM: Evaluating Style Personalization in LLMs with Arbitrary Preference Mappings. https://arxiv.org/abs/2605.21063, 2026.

[2] Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs. ICLR, 2025.

[3] Multi-turn Reinforcement Learning from Preference Human Feedback. NeurIPS, 2024.

[4] Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning. NeurIPS, 2025.

[5] Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment. NeurIPS, 2025.

[6] PRISM-X: Experiments on Personalised Fine-Tuning with Human and Simulated Users. https://arxiv.org/abs/2605.13307, 2026.

[7] The PRISM Alignment Dataset. NeurIPS, 2024.

AuthorsOpenReview ↗

Response to Reviewer e4dj (Part 3/3)

Response to Question 3: Single-turn Preference Readability

On the test set, the preference-readability check passes for 78% of rewritten requests and the same-task check for 98% (training: 76% and 97%; validation: 72% and 99%).

Routing accuracy is lower when preferences are less readable (62% vs. 92%), while preference-evaluation results remain relatively stable. On requests passing versus failing the readability check, PLF's win rates are 63% vs. 59% against Persona prompting and 75% vs. 73% against the Base model. Only 11 test requests fail the same-task check.

These results suggest that insufficient observable preference evidence contributes to routing errors, while PLF maintains similar preference fit across readability groups. We will add this analysis to Appendix B.1.


Response to Weakness 2 and Question 3: Distinguishing Preference Evidence from Routing Failures.

Thanks for the question. We conducted an additional analysis to distinguish insufficient observable preference evidence from routing failures. Using only the preceding tutor turn and student reply, Claude Opus 5.5 labels whether each student reply expresses direct instruction, interactive guidance, or neither, without access to hidden preferences or routing decisions. Among replies expressing a clear preference, 99.1% match the assigned student preference.

Of the 338 dialogues without a detected switch, 275 (81%) contain no reply clearly expressing the new preference, and 210 involve preference changes at turn 6 or 7, leaving limited opportunities to adapt. Similarly, 546 of the 602 first-opportunity misses (91%) follow replies without clear preference evidence.

When the new preference is explicitly expressed, PLF selects the matching persona at the next turn in 97.6% of cases toward interactive guidance and 58.7% toward direct instruction. This shows that many missed switches reflect insufficient observable evidence, while routing toward direct instruction remains more challenging.

These results clarify the interpretation of the aggregate switching rates and identify where routing can be improved. We will include this analysis in the revised paper.

Reviewer_bK9iOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a new task of adapting an assistant’s behavior to users’ turn-specific needs and proposes a router-based approach that assigns inputs to different personas, with each persona learned separately through SDPO-style training. Results show improvements in preference fit over fixed persona heads and persona prompting with the same router across single-turn personalization and multi-turn tutoring.

Critical strengths
  1. The paper addresses the important problem of adapting an assistant’s behavior to users’ turn-specific needs. The proposed method, learning reusable behaviors separately and selecting them according to the context, appears to be an appropriate approach to this problem.
  2. The modular design separating persona learning from selection, along with the individual design choices, is technically sound. The method shows consistent improvements over persona prompting with the same router in both evaluation settings, and the small-scale human evaluation shows some alignment with the model-based judgments.
Critical weaknesses
  1. Several obvious baselines and technical justifications are missing. The paper presents a router-based approach that assigns inputs to different personas, with each persona learned separately through SDPO-style training, as a main contribution. However, it does not compare against straightforward alternatives: training a single policy on data covering all preferences using SFT on feedback-edited or filtered responses, training such a policy using SDPO, or retaining the router while training the persona adapters using SFT. These comparisons would help establish how much routing and feedback-based on-policy distillation each contribute. The persona catalog appears somewhat heuristic; although some motivation is provided, further justification for these choices (quantitative evidence if possible), would be helpful. Also, human alignment of LLM-based judge is reported only through raw agreement rates. Supplementing these with chance-corrected measures, such as Cohen’s kappa, and human–human agreement as a reference would be fundamental.
  2. The writing could be improved for clarity. The paper introduces a new task setting, which could be stated more explicitly early on (more strongly than line 112). The importance of this task and the construction of its evaluation pipeline should also be described more clearly. The term “feedback” refers to signals with different roles and forms during training and inference, which can be confusing; please consider distinguishing these more explicitly in the presentation. I would also find the main results easier to assess if the numerical results underlying the figures were presented in a table in the main text.

I am willing to raise the score if all points of the first weakness (comparison w/ baselines, justification for persona choices, human alignment report) are properly addressed.

Questions for authors

See weaknesses. Also, are the results comparing PLF with each fixed persona head on the evaluation subset corresponding to that persona, rather than on the full test set? For example, how does PLF compare with the warm head on warm-only requests? (Not highly important, but to understand better about the task formulation) Please let me know if I have misunderstood any aspect of the paper, particularly the evaluation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer bK9i (Part 1/2)

Thank you for your detailed and constructive feedback! We address your concerns and questions as follows.

Response to Weakness 1: Additional Baselines and Technical Justifications

Baseline comparisons. Thanks for the suggestion. We added three baselines: Feedback imitation retains separate persona adapters and the router but trains the adapters using SFT on feedback-conditioned teacher responses; Shared-policy distillation trains a single adapter conditioned on persona labels using PLF's feedback-conditioned on-policy objective; and Shared-policy SFT trains the same shared architecture using SFT.

The table reports raw PLF wins, ties, and losses on the test sets.

PLF versusSingle-turn QA: wins / ties / lossesMulti-turn Tutoring: wins / ties / losses
Base model332 / 78 / 89589 / 107 / 304
Persona prompting264 / 88 / 147494 / 118 / 388
Feedback imitation199 / 145 / 155455 / 127 / 418
Shared-policy distillation219 / 110 / 170498 / 118 / 384
Shared-policy SFT224 / 118 / 157471 / 107 / 422

The results support the contributions of feedback-conditioned on-policy training and separate persona learning. PLF achieves more wins than losses against Feedback imitation in both settings, with a clearer advantage in single-turn personalization. PLF also outperforms both shared conditional policies, supporting the benefit of separately learned persona behaviors. Together with the existing fixed-head comparisons, these findings support PLF's combination of adaptive selection and feedback-trained persona adapters. We will incorporate these results into the revised paper.


Persona choices. Our single-turn personas are grounded in human-written preferences from PRISM [1]. The warm–impersonal and concise–thorough contrasts correspond to the warmth/personability and brevity/expansiveness dimensions identified in PRISM-X [2]. We construct the persona descriptions by summarizing recurring response-style preferences expressed by PRISM training users and fix them before evaluation. For tutoring, direct instruction and interactive guidance correspond to two modes explicitly included in the Education Dialogue environment [3]. We will make these sources and the persona-construction procedure explicit in the main text.


Human alignment. Thanks for the suggestion. We conducted additional human evaluations on 96 single-turn response pairs and 100 multi-turn tutoring dialogue pairs comparing PLF with persona prompting. Three independent annotators assessed preference fit using anonymized outputs with randomized A/B assignments. We report Cohen's κ alongside raw agreement, with human–human agreement as a reference.

Single-turn QA (96 pairs):

ComparisonAgreementCohen's κ
LLM vs Human 170.8%0.429
LLM vs Human 275.0%0.516
LLM vs Human 368.8%0.419
Mean LLM–Human71.5%0.454
Mean Human–Human76.0%0.517

Multi-turn tutoring (100 pairs):

ComparisonAgreementCohen's κ
LLM vs Human 181.0%0.658
LLM vs Human 278.0%0.626
LLM vs Human 376.0%0.599
Mean LLM–Human78.3%0.628
Mean Human–Human80.3%0.659

Mean values average the three LLM–human or human–human comparisons. These results suggest that the LLM judge generally reflects human preference judgment, where its agreement with humans is close to human–human agreement.


References

[1] The PRISM Alignment Dataset. NeurIPS, 2024.

[2] PRISM-X: Experiments on Personalised Fine-Tuning with Human and Simulated Users. https://arxiv.org/abs/2605.13307, 2026.

[3] Multi-turn Reinforcement Learning from Preference Human Feedback. NeurIPS, 2024.

AuthorsOpenReview ↗

Response to Reviewer bK9i (Part 2/2)

Response to Weakness 2: Clarifying the Task, Evaluation Pipeline, and Feedback Mechanism

Task definition and importance. We will state the task more explicitly in the introduction: given a user request and observed dialogue history, an assistant must fulfill the request while matching the user's current preferences for tone, detail, and interaction style. These preferences may be combined within a request or change during a conversation. This matters because effective assistance depends not only on the content but also on how it is delivered. For example, the same user may prefer concise instructions under time pressure but detailed explanations when learning, while a student seeking guided reasoning may not benefit from receiving the final answer directly.

Evaluation construction. For single-turn personalization, we use 499 held-out PRISM requests, each assigned one of eight preference conditions (four individual and four combined preferences) and minimally rewritten to express that preference while preserving the original task. All methods receive the same request. For multi-turn tutoring, we use 500 held-out Education Dialogue openings, each evaluated under direct and interactive student preferences. Compared methods share the initial context, while a preference-conditioned student simulator generates subsequent replies. We also evaluate preference switching within conversations. In both settings, the LLM judge compares outputs against the assigned preference in both presentation orders. We will clarify this evaluation pipeline and add a main-text table with the numerical results underlying the figures.

Meaning of feedback. During training, a critic's assessment in single-turn personalization or a student's subsequent reply in tutoring provides hindsight feedback on the preceding response. This feedback conditions the frozen teacher's token-level supervision for updating persona adapters. During inference, the router and adapters remain frozen; observed requests and replies instead provide preference evidence for selecting subsequent behaviors, without parameter updates. We will distinguish these roles explicitly.


Response to Questions: Clarifying Evaluation Comparisons

The main comparisons evaluate each fixed head on the full mixed-preference test set, rather than only its matching preference subset. They measure the benefit of adapting behavior across different preferences instead of using one fixed persona throughout. For example, on warm-only requests, correctly routing to the warm head activates the same adapter as the fixed-warm baseline, producing identical responses under the same decoding settings. Differences on this subset can arise from routing errors, whereas the overall comparison captures the benefit of selecting appropriate behaviors across heterogeneous requests. We will clarify this distinction in the evaluation description.

Thank you for your response and the additional experiments. I appreciate the new baseline comparisons and the expanded human evaluation, which substantially address my concerns. The clarification of how the persona choices are grounded in prior work is also helpful. I have raised the overall rating from 2 to 3 and informativeness from 2 to 3.

Thank you for your careful rebuttal review and for raising your score after considering our response. We are very glad that our rebuttal addressed your concerns, and we sincerely appreciate your time and support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.