acceptodds
Under review as a conference paper at ICLR 2027

User Alignment Requires More Than Following Feedback

Abstract

Aligning language models from user interactions offers a direct path to personal assistants and agents that continually adapt to their users. Existing methods generally treat actionable feedback as ground-truth supervision: once a follow-up indicates a direction for improvement, the model is optimized toward the behavior inferred from it. This assumption is ill-suited to real user feedback, which is often incomplete, noisy, or open to revision. Such updates can prematurely commit the policy to weak evidence or unresolved choices that later feedback may undo. Therefore, learning from such naturally occurring feedback requires answering two core unresolved questions: \emph{how strongly should a feedback turn influence the policy, and which policy changes does it support? To address the first gap, we formalize it through feedback sufficiency, the extent to which a change implied by feedback is supported as a policy update. We then introduce Hindsight of Hindsight (HOH), which assesses this support by examining how the original user responds after that change is carried out. These retrospective assessments supervise an online sufficiency model that operates when feedback arrives. To address the second question, we propose User-Centric Multi-Teacher On-Policy Distillation (UMOPD), which constructs multiple feedback-compatible revisions rather than distilling a single feedback-conditioned teacher. UMOPD compares the policy shifts these revisions induce, reinforcing changes they share while preserving alternatives where they disagree. Predicted sufficiency then scales the resulting policy update. We validate UMOPD across three interaction scenarios: learning a user's preference, adapting when it changes, and integrating multiple preferences introduced over time. Under a naturalistic user simulator, UMOPD exceeds the strongest baseline in final preference win rate by points when learning an initial preference and by points after it reverses. When three preferences are introduced sequentially, UMOPD is the only method to exceed on each. Our code is available at https://anonymous.4open.science/r/umopd_hoh-2864.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Human-AI Alignment, User Feedback, Real-World Interactions

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pMujOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper considers the problem of aligning LLMs with user intent for personal assistant and agentic applications. The main problem the authors identify is that user feedback is noisy, incomplete, or open to revision, therefore, using a single-turn as a direction for update might be misleading on the long-term. To address this challenge, the authors propose the following contributions

  1. Establish feedback sufficiency to measure the extent and reliability of the user feedback as a policy update signal.
  2. Introduce Hindsight of Hindsight (HOH), a new supervision approach grounded in feedback sufficiency
  3. Propose the UMOPD algorithm for model alignment that can adapt based on feedback sufficiency
Critical strengths
  1. Well-motivated, empirically grounded problem framing. The finding that most fully executed feedback is still refined afterwards is replicated across two corpora, partially validated by humans, and shown not to reduce to task vagueness (Cramér’s V = 0.133). This directly challenges the core assumption of current hindsight-distillation methods, which makes the contribution relevant beyond the proposed algorithm.
  2. Large gains under naturalistic feedback, with informative ablations. The method’s advantage appears exactly where baselines degrade (UserLM rather than prompted Qwen). Removing HOH, the feedback contrast (β=0\beta = 0), adaptive agreement, or multiple teachers each causes failure in at least one protocol, which supports the claim that the components are complementary rather than redundant.
Critical weaknesses
  1. Writing Clarity: Section 3.1. is missing a clear explanations of some terminology such as domains for y_t, u_t, O_t, C_t, or G_t. Without them, it is hard to understand the notation. Consequently, feedback sufficiently, c_t, is only informally defined. An example here could improve clarity by much.
  2. The paper assumes that the revision turn y_{t+1} is meant to only correct y_t, however, in real interactions the user might give implicit or explicit feedback on y_t in u_t but also request further actions in y_{t+1} which were not part of the history H_t. This is a gap between the model's framework and the real user interactions.
  3. The experimental section is limited. It considers a single model family (Qwen3), small scales (4B and 8B), and simple clean datasets (TL;DR, HelpSteer2). Despite the authors use agentic applications as motivating examples and real-world datasets in Section 3, Section 5 is much simpler leaving it ambiguous whether UMOPD can really scale up to these tasks.
Questions for authors
  1. The paper extends previous works which consider only the immediate turn feedback to a second-turn feedback. Can this framework be extended to multiple consecutive turns, potentially extracting the feedback from the whole remaining interaction between an agent/assistant and a user?
  2. In Figure 4, there is a clear difference in results between the Qwen and UserLM simulations. Do the authors know why this is the case?
  3. Could the authors provide confidence estimates for the results in Section 5? Without them it is hard to estimate whether methods are really significantly different.
  4. How does UMOPD compare to other algorithms used for comparison in terms of compute requirements?
6 comments
AuthorsOpenReview ↗

Response to Reviewer pMuj (Part 1/5)

We sincerely thank the reviewer for the thoughtful and constructive feedback.

1. Clarifying Notation and Examples (W1)

Thank you for pointing this out. We clarify the notation and formal definition in Section 3.1 together with a concrete example below (further interaction examples are also provided in Appendix A.1).

Let V∗\mathcal{V}^\ast denote the space of natural-language text sequences over vocabulary V\mathcal{V}. In HOH, we use offline, in-the-wild interaction logs to retrospectively assess whether a user feedback turn provides sufficient guidance for learning. Each offline window is a 5-tuple wt=(Ht,yt,ut,yt+1,ut+1)w_t = (H_t, y_t, u_t, y_{t+1}, u_{t+1}):

  • Ht∈(V∗)∗H_t \in (\mathcal{V}^\ast)^\ast: the conversation history prior to turn tt;
  • yt∈V∗y_t \in \mathcal{V}^\ast: the assistant's initial response given HtH_t;
  • ut∈V∗u_t \in \mathcal{V}^\ast: the user's follow-up turn containing feedback on yty_t;
  • yt+1∈V∗y_{t+1} \in \mathcal{V}^\ast: the assistant's next response following utu_t;
  • ut+1∈V∗u_{t+1} \in \mathcal{V}^\ast: the original user's subsequent turn after seeing yt+1y_{t+1}.

When an online learner receives feedback, only the current context (Ht,yt,ut)∈(V∗)∗×(V∗)2(H_t, y_t, u_t) \in (\mathcal{V}^\ast)^\ast \times (\mathcal{V}^\ast)^2 is available. The later turns (yt+1,ut+1)(y_{t+1}, u_{t+1}) are used only offline to verify whether the feedback on yty_t is addressed in yt+1y_{t+1} and observe how the user reacts in ut+1u_{t+1}.

Before examining ut+1u_{t+1}, we anchor the assessment to the feedback-implied change in (Ht,yt,ut)(H_t, y_t, u_t), represented as a triple of open-vocabulary text descriptions χt=(Ot,Ct,Gt)∈(V∗)3\chi_t = (O_t, C_t, G_t) \in (\mathcal{V}^\ast)^3:

  • Ot∈V∗O_t \in \mathcal{V}^\ast (the object changed): the specific part or property of yty_t targeted by the feedback;
  • Ct∈V∗C_t \in \mathcal{V}^\ast (the requested change): the modification requested on OtO_t;
  • Gt∈V∗G_t \in \mathcal{V}^\ast (the intended outcome): the outcome that applying CtC_t to OtO_t aims to achieve.

Conditioned on yt+1y_{t+1} fully executing the explicit requirements of χt\chi_t, feedback sufficiency ctc_t is formally defined as a retrospective evidence mapping ct=Φ(χt,ut+1)∈S∪{⊥}c_t = \Phi(\chi_t, u_{t+1}) \in \mathcal{S} \cup \lbrace \bot\rbrace from the anchored change χt\chi_t and the subsequent user turn ut+1u_{t+1} to the ordered support domain S={Insufficient≺Partially sufficient≺Sufficient}\mathcal{S} = \lbrace \text{Insufficient} \prec \text{Partially sufficient} \prec \text{Sufficient}\rbrace, with ⊥\bot denoting abstention (No evidence):

ct=Φ(χt,ut+1)={Sufficient,if ut+1 accepts Ct, confirms Gt, or reuses yt+1 with Ct retained,Partially sufficient,if ut+1 preserves (Ct,Gt) on Ot but adds or refines constraints left open by ut,Insufficient,if ut+1 reverses or rejects Ct, or indicates that Gt was not met,⊥ (No evidence),if ut+1 provides no identifiable evidence concerning χt.c_t = \Phi(\chi_t, u_{t+1}) = \begin{cases} \text{Sufficient}, & \text{if } u_{t+1} \text{ accepts } C_t, \text{ confirms } G_t, \text{ or reuses } y_{t+1} \text{ with } C_t \text{ retained}, \\\\ \text{Partially sufficient}, & \text{if } u_{t+1} \text{ preserves } (C_t, G_t) \text{ on } O_t \text{ but adds or refines constraints left open by } u_t, \\\\ \text{Insufficient}, & \text{if } u_{t+1} \text{ reverses or rejects } C_t, \text{ or indicates that } G_t \text{ was not met}, \\\\ \bot \text{ (No evidence)}, & \text{if } u_{t+1} \text{ provides no identifiable evidence concerning } \chi_t. \end{cases}

Windows with ct=⊥c_t = \bot are filtered out after annotation, so predictor training only uses the three ordinal levels in S\mathcal{S} to supervise ρϕ(Ht,yt,ut)\rho_\phi(H_t, y_t, u_t).

For example, consider a real WildChat window (from Appendix A.1):

  • yty_t: "connection between masses and particles"
  • utu_t: "make it like a textbook section title"
  • χt=(Ot,Ct,Gt)\chi_t = (O_t, C_t, G_t):
    • OtO_t: the topic phrase "connection between masses and particles";
    • CtC_t: rewrite the phrase in the style of a textbook section title;
    • GtG_t: obtain a formal title suitable for a textbook section.
  • yt+1y_{t+1}: "The Relationship Between Masses and Particles: Avogadro's Number and Molar Mass" (which fully executes CtC_t).
  • ut+1u_{t+1}: "use an existing textbook section title"
  • ctc_t: Here, ut+1u_{t+1} preserves the requested direction CtC_t and intended outcome GtG_t (still wanting a textbook section title for OtO_t), but adds a realization constraint that utu_t left open. Therefore, ct=Φ(χt,ut+1)c_t = \Phi(\chi_t, u_{t+1}) is Partially sufficient (whereas reversing CtC_t yields Insufficient, and accepting or reusing CtC_t yields Sufficient).
AuthorsOpenReview ↗

Response to Reviewer pMuj (Part 2/5)

2. Clarifying Real-World Interaction Assumptions (W2)

We fully agree that in real-world interactions, users often give implicit or explicit feedback on yty_t while simultaneously requesting further actions or moving on to new steps. We would like to clarify that our framework does not assume that yt+1y_{t+1} or subsequent turns only correct yty_t.

First, we clarify how (yt+1,ut+1)(y_{t+1}, u_{t+1}) is used in our framework. During online learning, the policy updates on-policy directly from the current tuple (Ht,yt,ut)(H_t, y_t, u_t) and never uses yt+1y_{t+1}. In offline HOH, our goal is not to train the policy on the revision yt+1y_{t+1} itself, but to read the user's subsequent reaction in ut+1u_{t+1} to assess whether the feedback in utu_t provides sufficient guidance for learning (which supervises our online predictor ρϕ(Ht,yt,ut)\rho_\phi(H_t, y_t, u_t)).

Precisely because real conversations often mix feedback with additional requests or follow-up actions across turns, offline HOH never treats an entire turn as a monolithic signal. Instead, it tracks only the feedback-implied change χt=(Ot,Ct,Gt)\chi_t = (O_t, C_t, G_t) across all three elements:

  • In the feedback utu_t: We extract only the specific feedback information directed at yty_t to anchor χt=(Ot,Ct,Gt)\chi_t = (O_t, C_t, G_t), separating it from any additional actions requested in the same turn.
  • In the revision yt+1y_{t+1}: We examine yt+1y_{t+1} only to verify that χt\chi_t was executed, ruling out assistant non-compliance as a confounder. We do not require yt+1y_{t+1} to only revise yty_t, so executing additional requested actions in yt+1y_{t+1} does not affect this check.
  • In the subsequent turn ut+1u_{t+1}: We extract only the evidence concerning the same change χt\chi_t. If the user only follows up on the additional actions or moves on to a new topic without providing identifiable evidence about χt\chi_t, the window is labeled as No evidence (⊥\bot) and excluded from predictor training.

This is also why we learn HOH from large-scale in-the-wild logs: even after strictly filtering for verified execution of χt\chi_t and attributable evidence in ut+1u_{t+1}, the corpus provides ample clean supervision to train ρϕ(Ht,yt,ut)\rho_\phi(H_t, y_t, u_t).

3. Clarification on HOH and Extension to Multiple Consecutive Turns (Q1)

We appreciate this thoughtful question. We would first like to gently clarify the role of the second turn in HOH. Whereas prior work directly treats the immediate feedback utu_t as a learning target for yty_t, HOH uses the user's subsequent reaction ut+1u_{t+1} in offline logs to assess whether utu_t itself provides sufficient guidance for learning. During online interaction, the model therefore still learns continuously from each feedback turn utu_t as it arrives.

At the same time, offline HOH can indeed be naturally extended to multiple consecutive future turns (or the remaining conversation). Because our assessment is anchored to the specific change χt=(Ot,Ct,Gt)\chi_t = (O_t, C_t, G_t), we can track (ut+1,ut+2,… )(u_{t+1}, u_{t+2}, \dots) and extract evidence whenever a later turn refers back to χt\chi_t. In fact, we explored this multi-turn extension when designing our work, and found that looking beyond ut+1u_{t+1} successfully recovers delayed evidence that has not yet appeared in ut+1u_{t+1} (for example, when a user briefly asks a side question in ut+1u_{t+1} before returning to confirm or refine χt\chi_t in ut+2u_{t+2}).

To examine the trade-off between recovering delayed evidence and maintaining clean attribution, we tracked ut+1u_{t+1}, ut+2u_{t+2}, and ut+3u_{t+3} across 5,000 sampled windows in WildChat:

Subsequent TurnAttributable Evidence for χt\chi_tNo evidence Among Active TurnsNewly Recovered from No evidence at ut+1u_{t+1}
ut+1u_{t+1} (5-tuple)62.1% (3,103 / 5,000)37.9%-
ut+2u_{t+2}10.4% (519 / 5,000)75.5%3.5% (173 / 5,000)
ut+3u_{t+3}5.3% (266 / 5,000)75.0%0.6% (28 / 5,000)

As shown in the table, extending to ut+2u_{t+2} and ut+3u_{t+3} recovers delayed evidence for an additional 201 windows (173 at ut+2u_{t+2} and 28 at ut+3u_{t+3}) that were No evidence at ut+1u_{t+1}. Meanwhile, over 93% of all recovered evidence already appears immediately at ut+1u_{t+1}, whereas later turns are separated from χt\chi_t by intermediate model outputs and new user requests (with over 75% of active turns at ut+2u_{t+2} and ut+3u_{t+3} being No evidence). Given the scale of in-the-wild logs, we adopt the 5-tuple as our default in this paper to prioritize unconfounded attribution, while multi-turn aggregation offers a natural extension when delayed evidence is more prevalent.

AuthorsOpenReview ↗

Response to Reviewer pMuj (Part 3/5)

4. Broader Models and Tasks (W3)

We expanded our evaluation across model families and scales, real-world user queries, and a personal agent scenario, comparing UMOPD against all three baselines (SDPO, OpenClaw-RL, and Mem0):

  • Across model families and larger scales: Building on Qwen3-4B and Qwen3-8B in the main text, we further evaluated Llama-3.1-8B-Instruct and Qwen3-32B under the preference reversal setting (Figure 4b, 480 online steps under UserLM-8B feedback):
Policy ModelUMOPDOpenClaw-RLMem0SDPO
Qwen3-4B98.2% ± 1.3%36.6% ± 5.7%50.9% ± 4.8%7.3% ± 2.3%
Qwen3-32B98.5% ± 0.9%46.9% ± 4.4%52.9% ± 4.2%48.3% ± 4.6%
Llama-3.1-8B-Instruct76.9% ± 4.5%70.2% ± 4.5%34.6% ± 4.6%6.7% ± 2.0%

On Qwen3-32B, larger model capacity mitigates the collapse of SDPO, yet all baselines remain around 50% due to phase-1 policy inertia, whereas UMOPD reaches 98.5% (+45.6 points). Across model families on Llama-3.1-8B-Instruct, SDPO again degrades to 6.7%, while OpenClaw-RL and UMOPD remain above 70%, with UMOPD achieving the highest win rate.

  • Real-world user queries: Beyond TL;DR and HelpSteer2, we further evaluated on real-world user queries. Specifically, we sampled 1,200 real-world user requests (spanning coding debugging, technical analysis, and everyday advice) from WildFeedback, which is derived from real WildChat conversations, for long-horizon online training, and evaluated on 512 held-out requests (Qwen3-4B, UserLM-8B feedback, target preference: concise, casual, beginner-friendly):
WildFeedbackStep 240Step 600Step 1200 Final Win Rate
UMOPD64.0% ± 3.0%80.0% ± 2.4%92.1% ± 1.4%
SDPO51.4% ± 3.2%56.7% ± 3.2%64.4% ± 2.9%
OpenClaw-RL50.0% ± 3.3%48.4% ± 3.1%52.7% ± 3.1%
Mem055.8% ± 3.0%51.1% ± 3.0%49.8% ± 3.0%

On these diverse real-world queries with frequent follow-up questions, Mem0 and OpenClaw-RL plateau around 50% and SDPO improves slowly, whereas UMOPD steadily climbs above 92% over 1,200 online steps.

  • Personal agent scenario: We further extended our evaluation to a personal agent setting, following the personal homework agent setup in OpenClaw-RL. In each session, a student assigns a GSM8K math problem to the agent (Qwen3-4B) using a fixed initial prompt that does not reveal their preference. The student requires the solution to preserve complete step-by-step calculations and arrive at the correct final answer while reading like natural student homework without AI formatting cues (such as bold text, headers, or numbered lists), requesting a rewrite otherwise. Following OpenClaw-RL, we report two metrics: (i) Personalization Score ↑\uparrow (∈[0,1]\in [0, 1], measuring whether the agent's first-turn solution simultaneously satisfies natural student style, complete step-by-step derivation, and the correct mathematical answer), and (ii) Sessions to Convergence ↓\downarrow (the minimum number of sessions to reach 3 consecutive sessions where the first response is directly accepted, up to 72 sessions):
SimulatorMethodPersonalization Score ↑\uparrowSessions to Convergence ↓\downarrow
Prompted QwenUMOPD0.8914.2
Prompted QwenOpenClaw-RL0.7918.8
Prompted QwenSDPO0.8223.4
Prompted QwenMem00.4831.6
UserLM-8BUMOPD0.8719.6
UserLM-8BSDPO0.14>72.0
UserLM-8BOpenClaw-RL0.00>72.0
UserLM-8BMem00.00>72.0

Under explicit Prompted Qwen feedback, OpenClaw-RL, SDPO, and UMOPD all adapt effectively, with UMOPD converging fastest (14.2 sessions) and achieving the highest score (0.89). Under naturalistic UserLM-8B interaction, however, baselines are disrupted by step-level follow-up questions and fail to converge within 72 sessions, whereas UMOPD converges in 19.6 sessions with a score of 0.87 while maintaining standalone test-set math accuracy at 91.4% to 93.8%, matching π0\pi_0 at 92.2%. This demonstrates that UMOPD generalizes effectively to multi-session personal agent scenarios without sacrificing underlying reasoning accuracy.

Reviewer_eiypOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies online personalization from incomplete user feedback. It introduces HOH to estimate feedback sufficiency and UMOPD to distill multiple feedback-compatible revisions. Experiments cover preference acquisition, reversal, and sequential integration using simulated users.

Critical strengths
  • Relevant observation. The distinction between actionable feedback and a sufficiently specified learning target is meaningful. The analysis of real conversations illustrates why following an instruction does not necessarily resolve the user’s expectations.

  • Clear method. The method clearly separates update strength from update direction. Multiple revision-conditioned teachers represent alternative interpretations, while the sufficiency predictor regulates how strongly each feedback turn affects learning.

  • Broad adaptation scenarios. The experiments cover retaining, reversing, and accumulating preferences, with substantial improvements under UserLM feedback. The ablations provide useful evidence about the contribution of individual components.

Critical weaknesses
  • Unclear need for immediate parameter updates. Feedback about a particular response does not necessarily express a durable preference that should generalize to future tasks. The paper does not sufficiently distinguish these cases or establish when immediate parameter updates are preferable to retaining feedback in context, updating memory, or waiting for clarification. Including Mem0 is useful, but does not fully resolve this question about the learning setup.

  • Uncertain interpretation of baseline collapse. Repeatedly distilling a model’s own interpretations can amplify errors and destabilize training. The large performance drops may therefore reflect overly aggressive online updates rather than a specific failure to model feedback sufficiency. Without update-matched or more conservative baseline configurations, it is difficult to determine how much of UMOPD’s advantage supports the proposed account of incomplete feedback.

  • Limited evidence for the interaction assumptions. The strongest gains occur with UserLM, whereas several baselines work well with role-prompted Qwen. The paper does not sufficiently characterize which differences between these feedback sources drive the gap. Moreover, evaluation focuses on stylistic preference adherence and excludes factual correctness, leaving uncertain whether the resulting adaptation improves overall usefulness for real users.

Questions for authors

When should feedback about one response become a persistent parameter update? How does the method distinguish a local correction from a preference intended to apply across future interactions?

Would retaining feedback in context or delaying updates until further clarification achieve similar results? Do the gains persist against baselines with matched update magnitude or more conservative learning rates?

What properties of UserLM feedback cause the baselines to deteriorate so sharply? Are these failures also observed with real users, and does UMOPD preserve task quality while improving preference adherence?

9 comments
AuthorsOpenReview ↗

Response to Reviewer eiyp (Part 1/6)

We sincerely appreciate the reviewer for the time and effort in reviewing our paper, and for the constructive questions.

1. Immediate Parameter Updates and Alternatives (W1, Q1, Q2)

We appreciate this important question and fully agree that feedback on a single response does not automatically constitute a global rule for all future tasks. Below, we would like to clarify three aspects and provide additional experimental comparisons.

(1) Local Feedback and Durable Preferences in Context-Conditioned On-Policy Updates

In natural conversations, user feedback on a specific response yty_t falls into three cases according to its generalizable meaning for future tasks:

  • Durable preferences expressed incrementally through local feedback: Ideally, if users explicitly labeled each feedback turn as either a future-oriented preference or a one-off adjustment to the current answer, updating parameters only on the former would be straightforward. In everyday interactions, however, users rarely state abstract global declarations upfront, and no such explicit markers exist. Durable preferences on level of detail, step-by-step presentation, task focus, or style typically emerge incrementally through reactions to individual responses, consistent with observations in real-world interaction studies such as WildFeedback and LENS. Waiting only for explicit global declarations would therefore miss the main channel through which preferences are expressed.
  • Context-grounded instance corrections: Unlike unconditional memorization of response content into model weights, on-policy self-distillation on a conditional policy πθ(y∣Ht)\pi_\theta(y \mid H_t) remains tied to the input context HtH_t. Even when feedback points out an omission in the current task, such as "you missed that the house's rent was discounted" in summarization or QA, both student and teacher distributions are conditioned on the current input HtH_t, and the shift Δi,j\Delta_{i,j} measures the change induced by the feedback relative to that same context HtH_t. Distilling a well-supported correction into πθ(⋅∣Ht)\pi_\theta(\cdot \mid H_t) does not make the model memorize "discounted rent" on unrelated future inputs Ht′H_{t'} where rent never appears. Instead, it reinforces the generalizable habit of faithfully preserving key conditions from the input context.
  • Non-directive continuations, unsettled trials, or one-off requests: Follow-up questions, tentative edits that conflict with existing task constraints, or one-off requests specific to a single prompt should indeed not become global behavioral shifts, and constraining such signals is a central goal of UMOPD.
AuthorsOpenReview ↗

Response to Reviewer eiyp (Part 2/6)

(2) Regulating Update Strength and Update Direction in UMOPD (Q1)

Two properties underlie how online parameter updates operate in UMOPD. (i) In online on-policy learning, a single immediate parameter update does not lock in permanent behavior in one step, but takes only a small controlled step. Durable parameter changes form only when consistent feedback repeatedly reinforces the same direction across subsequent tasks. (ii) When the user does not specify the intended scope, a single turn (Ht,yt,ut)(H_t, y_t, u_t) alone cannot determine in advance how often a reasonable request will recur. Accordingly, UMOPD neither relies on a brittle single-turn classifier between local and global intent nor treats every turn as complete ground truth, and instead regulates both update strength and update direction:

  • When feedback triggers a parameter update (update strength): Conversational continuations that do not provide actionable improvement guidance for the preceding response, including follow-up questions, topic switches, and brief acknowledgments, receive zero update weight. For turns that provide improvement guidance, the HOH predictor ρϕ(Ht,yt,ut)\rho_\phi(H_t, y_t, u_t), trained on subsequent user reactions in real-world logs, estimates whether executing the feedback in context (Ht,yt)(H_t, y_t) will be accepted and retained rather than reversed or re-patched next turn. Vague complaints or local trials that conflict with existing context constraints are downweighted immediately. In Appendix A.3, for example, requesting a "more natural tone" when the prompt requires preserving an author's formal voice receives a low weight of ρϕ=0.258\rho_\phi = 0.258.
  • How durable preferences separate from local adjustments (update direction and cross-turn accumulation): Within each update, UMOPD constructs multiple plausible revisions compatible with the feedback, applying mode-seeking reverse KL only along shared adjustment directions (high directional agreement aja_j) while using mass-covering forward KL on instance-specific wording or unresolved choices (low aja_j). An isolated local adjustment therefore produces only a bounded, coverage-preserving step that is not reinforced on unrelated future tasks Ht′H_{t'}. Conversely, a durable preference repeatedly pushes the policy in a consistent direction across tasks while remaining revisable if preferences later change, reaching 98.2% after preference reversal in Figure 4b compared with 1.9% for w/o HOH.

To test directly whether one-off local requests spill over to unrelated tasks, we injected a post-specific instruction ("for this specific post only, end the summary with [Case-Note]") into 20% of turns (every 5th turn) across the 240-step interaction stream and evaluated on 256 unrelated held-out test prompts. The cross-task spillover rate of the local marker on held-out prompts is 0.0% (0 / 256) for UMOPD (as well as for SDPO). This confirms that under context-conditioned small-step updates, occasional one-off instructions are not unconditionally memorized across tasks, whereas the core challenge in online updates lies in filtering insufficient feedback and preserving unresolved choices.

AuthorsOpenReview ↗

Response to Reviewer eiyp (Part 3/6)

(3) Comparison with Context Retention, Memory, and Delayed Updates (W1, Q2)

Retaining feedback in context, updating external memory, and waiting for clarification each suit specific scenarios: context retention fits multi-turn refinement within a single task, external memory fits explicit discrete facts that require direct inspection or deletion, and waiting for clarification fits high-stakes actions.

To evaluate whether they achieve similar results for continual alignment from naturalistic feedback, we added two baselines alongside Mem0 on the same UserLM-8B benchmark. Retaining in Context keeps π0\pi_0 frozen and prepends the 6 most recent user feedback turns from the stream to the system context. Delaying until Clarification withholds updating upon receiving utu_t, generates the revised response yt+1y_{t+1}, observes the user's subsequent turn ut+1u_{t+1} from UserLM-8B, and updates only when ut+1u_{t+1} confirms or further clarifies the change under the HOH rubric, skipping turns without relevant evidence.

ApproachPreference Acquisition (Fig. 4a)Preference Reversal (Fig. 4b)
Retaining in Context46.8% ± 6.3%58.2% ± 5.2%
Updating Memory (Mem0)26.6% ± 5.5%50.9% ± 4.8%
Delaying until Clarification62.3% ± 6.6%44.9% ± 4.8%
UMOPD98.7% ± 0.9%98.2% ± 1.3%

Under naturalistic feedback, each alternative faces a distinct bottleneck:

  • Retaining in Context: Raw feedback turns are coupled with details and follow-up questions from earlier tasks, causing context interference when prepended to unrelated new prompts while adding around 90 to 130 context tokens per query.
  • Updating Memory (Mem0): First, when everyday feedback is brief or implicit, extracting clean rules and retrieving them by similarity is prone to misses or mismatches, and old and new memory entries conflict after preference reversal. Second, because model weights remain frozen, concatenating multiple rules in the prompt often fails to override the base model's default generation habits (for instance, Mem0 reaches only 40.4% to 56.8% under compound constraints in Figure 5 and reverts to long formatted lists with emojis in Appendix D.3, Case 2).
  • Delaying until Clarification: Updating must wait until observing the user's reaction ut+1u_{t+1} to the revised response, yet satisfied users in natural conversations typically move on to new questions rather than staying to confirm an edit, leaving around 40% of next turns with No evidence in both WildChat and this UserLM stream. Online waiting therefore discards substantial valid feedback and slows adaptation after preference reversal, whereas offline pretraining of ρϕ\rho_\phi captures subsequent support patterns without online delay.
Reviewer_6CVqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies continual alignment of LLMs from user feedback, where a follow-up turn serves as privileged hindsight for on-policy self-distillation. It argues that such feedback is often incomplete, and that treating it as full supervision makes the model commit to choices the user never specified. It makes two contributions:

  1. Feedback sufficiency, measured by HOH. Each feedback turn is labelled from the user's next reaction (sufficient / partially sufficient / insufficient). On WildChat and LMSYS-Chat-1M, ~57% of fully executed feedback is only partially sufficient. A Qwen3-4B scorer trained on these labels predicts sufficiency online and scales each update.

  2. UMOPD. The method generates K feedback-compatible corrections, each defining a teacher. Their agreement selects, token by token, between reverse KL to a consensus target and forward KL to a mixture.

Experiments use TL;DR (preference acquisition and reversal) and HelpSteer2 (sequential composition of three preferences), with baselines SDPO, OpenClaw-RL and Mem0. Under the UserLM simulator, UMOPD shows large gains on TL;DR. On sequential composition, its advantage over the parametric baselines is small; it is large only over Mem0.

Critical strengths

Critical strengths:

  1. Feedback sufficiency is a useful new angle, backed by data. The finding that most executed feedback is only partially sufficient replicates across two corpora (Sec. 3.2). It motivates the problem well beyond the proposed method. Also, the writing is clear and motivates the method well.

  2. Strong results under the naturalistic simulator. Under UserLM, UMOPD clearly outperforms the baselines (98.7% vs 72.4% for acquisition, 98.2% vs 50.9% after reversal), while the baselines degrade.

Critical weaknesses

Critical weaknesses:

  1. The empirical evidence is not strong enough for the claims. Every curve is a single run, in a noisy setting: sampling temperature, simulated multi-turn feedback, and only 256 or 128 evaluation prompts. Training is visibly unstable: K=3 collapses to 0.5%, and several ablations end near 2%. Win rates exclude ties, and tie counts are never reported. With many ties, the win rate rests on only a subset of the prompts and doesn't show how often the trained model is actually preferred. It is then less reliable, and harder to compare across methods. From what I read, UMOPD uses different learning rates for the two simulators, while the baselines follow their published configurations. It is unclear whether the baselines were run with the same learning rates or tuned under UserLM. On sequential composition, the advantage over SDPO is only a few points per preference from a single run. Together, these make it hard to separate the headline gains (+26 and +47 points) from seed variance or tuning differences.

  2. The contribution and cost of each component are not isolated. The method stacks many components: gate, sufficiency scorer, proposer, contrastive decoding, teacher weighting, agreement-based mixed KL. This makes the paper hard to follow. The value of the sufficiency signal is never separated from the scorer's errors. The proposer that generates the K revision plans is not specified.

Questions for authors

Questions for authors:

  1. (Material) Can you report several seeds with confidence intervals and tie rates? Also, were the baselines' learning rates tuned under UserLM, as UMOPD's was?

  2. (Clarification) Which model is the proposer (Sec. 4.1), and with what prompt? If it differs from the trained policy, the baselines should have access to it. Can you show examples of the K plans and how much they differ on the unresolved choices?

  3. (Material) Update strength prediction: could you add a run with ρ derived from the actual next simulator turn (so a delayed update labelled with the HOH protocol)? It would show how much of the gain is lost to the scorer's prediction errors.

  4. (Material) Could you compare against a simple multi-teacher extension of SDPO, which averages K teachers conditioned on independently sampled revisions? This would show whether the gain comes from UMOPD's specific design or simply from using several teachers, at similar compute.

  5. (Clarification) Have you evaluated UMOPD on offline tasks too? Or on preferences beyond the stylistic ones from the SDPO setup?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 6CVq (Part 1/6)

We sincerely thank the reviewer for the careful and detailed feedback.

1. Multi-Seed Evaluation, Ties, and Tuning (W1, Q1)

Thank you for these thorough questions on our empirical evaluation. We address each point regarding multi-seed stability, learning-rate configurations, tie rates, and the evaluation setup below.

(1) Multi-Seed Evaluation

We evaluated all methods across three independent random seeds (seeds 42, 43, 44) on fixed preference acquisition, preference reversal, and sequential preference composition. We report the mean and standard deviation (Mean±SD\text{Mean} \pm \text{SD}) across seeds below:

Experimental Setting (Section 5, UserLM-8B)UMOPDOpenClaw-RLMem0SDPO
Preference Acquisition (Fig. 4a, Step 240)92.8% ± 6.8%39.4% ± 28.7%26.6% ± 3.2%9.0% ± 9.1%
Preference Reversal (Fig. 4b, Step 480)95.2% ± 2.7%63.7% ± 24.0%48.0% ± 2.9%6.2% ± 5.8%
Sequential Composition, Phase-End Mean (Fig. 5)86.4% ± 2.4%50.8% ± 4.9%53.7% ± 2.5%74.9% ± 2.7%
Sequential Composition, Final Three-Attribute Mean (Fig. 5, Step 720)88.0% ± 3.7%53.8% ± 13.8%56.3% ± 8.2%84.9% ± 2.6%

Across all tasks, UMOPD consistently achieves high win rates with low variance across seeds. In particular, on sequential composition (Fig. 5), where SDPO is the strongest baseline, UMOPD leads SDPO by 11.5 points on phase-end acquisition across seeds (86.4% vs. 74.9%) and also achieves the highest final three-attribute mean at Step 720.

(2) Baseline Learning-Rate Configuration and Tuning

For a fair comparison, all baselines in the main paper use their published default configurations. Among them, Mem0 is a non-parametric memory method without gradient updates, and OpenClaw-RL already uses the exact same default learning rate as UMOPD (lr=10−5\text{lr} = 10^{-5}). Only SDPO has a different published default (lr=5×10−6\text{lr} = 5\times 10^{-6}). We therefore evaluated SDPO across a grid of learning rates (lr∈{10−5,5×10−6,1.25×10−6}\text{lr} \in \lbrace 10^{-5}, 5\times 10^{-6}, 1.25\times 10^{-6}\rbrace) under UserLM-8B:

Experimental Setting (UserLM-8B)SDPO (lr=10−5\text{lr} = 10^{-5}, matched to UMOPD)SDPO (lr=5×10−6\text{lr} = 5\times 10^{-6}, published default)SDPO (lr=1.25×10−6\text{lr} = 1.25\times 10^{-6}, conservative)UMOPD (Ours, lr=10−5\text{lr} = 10^{-5})
Preference Acquisition (Fig. 4a, Step 240)2.6% ± 1.5%18.2% ± 4.7%38.9% ± 6.6%98.7% ± 0.9%
Preference Reversal (Fig. 4b, Step 480)0.0% ± 0.0%7.3% ± 2.3%41.0% ± 4.8%98.2% ± 1.3%

When SDPO uses the same learning rate as UMOPD (10−510^{-5}), the larger un-gated updates accelerate policy drift and reduce win rates to 2.6% and 0.0%. Lowering the learning rate to a conservative 1.25×10−61.25\times 10^{-6} mitigates collapse and raises win rates to 38.9% and 41.0%, which still leaves a gap of over 55 percentage points relative to UMOPD. This confirms that the performance gains come from our algorithmic design rather than learning-rate tuning.

AuthorsOpenReview ↗

Response to Reviewer 6CVq (Part 2/6)

(3) Clarification on Ties and Full-Sample Preference Metrics

We would also like to clarify how ties arise and report the complete tie statistics. Following the pairwise evaluation protocol of prior work, each test prompt is judged twice by swapping the presentation order of the two responses to eliminate position bias, and a win or loss is recorded only when both orders agree, with all remaining pairs conservatively recorded as ties. Examining the win/tie/loss (W / T / L) breakdown shows that UMOPD not only achieves the highest win rate, but also substantially reduces ties and yields much clearer, more decisive separation over π0\pi_0 than the baselines:

  • In preference acquisition (Fig. 4a), UMOPD achieves 151 / 103 / 2 (40.2% tie rate), whereas baselines remain mostly tied with π0\pi_0 (74.2% to 77.3% tie rates).
  • In preference reversal (Fig. 4b), UMOPD achieves 111 / 143 / 2 (55.9% tie rate), losing on only 2 of 256 prompts compared with 45 to 115 losses for the baselines.

To further verify how often each model is preferred when ties are included across all test prompts, we report two additional metrics below: (i) the tie-adjusted win rate (W+0.5T)/N(W + 0.5T)/N across the full evaluation set, and (ii) direct head-to-head comparisons between UMOPD and each baseline:

Experimental Setting (UserLM-8B)Evaluation MetricUMOPDOpenClaw-RLMem0SDPO
Preference Acquisition (Fig. 4a)vs. π0\pi_0 (Decisive / Tie-Adjusted)98.7% / 79.1%72.4% / 55.1%26.6% / 44.1%18.2% / 41.8%
Preference Acquisition (Fig. 4a)Direct Head-to-Head: UMOPD vs. Baseline (Decisive / Tie-Adjusted)-98.1% / 79.3%99.5% / 85.2%98.7% / 80.1%
Preference Reversal (Fig. 4b)vs. π0\pi_0 (Decisive / Tie-Adjusted)98.2% / 71.3%36.6% / 46.3%50.9% / 50.4%7.3% / 29.3%
Preference Reversal (Fig. 4b)Direct Head-to-Head: UMOPD vs. Baseline (Decisive / Tie-Adjusted)-99.2% / 73.8%99.2% / 73.6%100.0% / 84.8%

Across decisive win rates, full-sample tie-adjusted win rates, and direct head-to-head comparisons, UMOPD consistently holds a substantial lead over all baselines.

(4) Clarification on Evaluation Setup and Expanded Evaluation

Finally, we would like to clarify that the online decoding settings and evaluation splits in Section 5 directly follow the published benchmark protocol of SDPO so that all methods are compared under identical conditions on the same fixed evaluation prompts. To further evaluate performance on a larger test set, we conducted an additional evaluation on 1,024 TL;DR test prompts:

Experimental Setting (UserLM-8B, N=1,024N = 1,024)Evaluation MetricUMOPDOpenClaw-RLMem0SDPO
Preference Acquisition (Fig. 4a, Step 240)Decisive (±\pm SE) / Tie-Adjusted95.1% ± 1.0% / 69.7%88.6% ± 2.2% / 58.3%28.7% ± 3.0% / 45.4%7.2% ± 1.5% / 37.3%
Preference Reversal (Fig. 4b, Step 480)Decisive (±\pm SE) / Tie-Adjusted98.6% ± 0.4% / 86.4%48.1% ± 2.8% / 49.4%42.4% ± 2.3% / 46.6%0.8% ± 0.4% / 19.8%
AuthorsOpenReview ↗

Response to Reviewer 6CVq (Part 3/6)

2. Clarification on Ablation Dynamics and Sequential Composition (W1)

We appreciate the opportunity to clarify the results on component ablations, teacher-count sensitivity, and sequential preference composition.

(1) Component Ablations Ending Near 2% (Appendix D.1)

First, we would like to respectfully clarify that these low win rates do not stem from numerical or optimization instability, as the training KL loss converges smoothly across all runs. Instead, they reflect generation drift when core regularizers are removed, evaluated pairwise against the coherent initial policy π0\pi_0. Once an unconstrained model drifts into repetitive endings or exceeds the generation limit and gets truncated, π0\pi_0 is consistently preferred across prompts, driving the win rate near zero. Specifically, the two variants ending near 2% each remove a necessary constraint against online drift:

  • Pure forward KL (a=0a = 0): Without reverse-KL consensus anchoring, mass-covering forward KL forces the student to cover all candidate elaborations simultaneously, causing response length to grow unchecked until truncation.
  • Removing HOH sufficiency weighting (w/o HOH): Updating at full strength on every naturalistic turn after a preference shift, including brief follow-ups and partially satisfied reactions, leads to over-updating and repetitive outputs, similar to un-gated SDPO (7.3%) on the same reversal task.

These results show that adaptive mixed-KL geometry and HOH weighting are essential to prevent online drift, rather than suggesting instability in full UMOPD.

(2) Sensitivity to Teacher Count KK (Appendix D.2)

Similarly, K=3K = 3 is not optimization-unstable. It converges smoothly to 90.7% under a fixed preference and adapts rapidly right after the preference switch, reaching 95.5% at Step 288. Its late-stage drop reflects how KK balances forward-KL multi-teacher expansion and reverse-KL consensus anchoring via directional agreement aja_j when shifting from concise to detailed:

  • Conservative updates (K=2K = 2): With only two candidates, if one carries low weight qiq_i or induces negligible shift, the update effectively reduces to single-teacher reverse KL. Even when both contribute, two candidates capture limited variation and keep aja_j high, so updates stay short and conservative, adapting more slowly at first (60.0% at Step 288) while avoiding length runaway and reaching 93.8% at Step 480.
  • Expansion outpacing anchoring (K=3K = 3): K=3K = 3 typically retains multiple active teachers expanding in different directions, sustaining forward-KL mass covering for fast initial adaptation (95.5% at Step 288). Across hundreds of steps toward the detailed target, however, forward KL keeps encouraging added detail from each branch while aja_j from at most three candidates provides limited anchoring on when to stop, gradually leading to overly long, repetitive, or truncated endings similar to a=0a = 0 and losing to π0\pi_0.
  • Balanced coverage and anchoring (K=4K = 4, default): Four candidates preserve multi-teacher coverage for fast switching while yielding a steadier aja_j signal to separate shared edits from individual elaborations, helping curb length drift and maintaining win rates above 98% in both settings.

(3) Sequential Preference Composition (Section 5.4, Figure 5)

Unlike preference reversal in Figure 4b, where conflicting targets cause mode collapse below 10% in single-teacher baselines, Figure 5 introduces three compatible preferences sequentially (No emojis, Less filler praise, and Direct answers). Parametric baselines therefore do not collapse to zero here, and instead show gradual cross-phase interference and slower late-stage learning. Even on the main-text curves alone, UMOPD's advantage over SDPO goes well beyond a few points at the final step in two ways:

  • Joint retention across all preferences: Compositional personalization requires maintaining all learned preferences simultaneously, since high adherence on an early preference cannot compensate for a drop on a later one. UMOPD is the only method finishing above 80% on every preference (98.6%, 83.1%, and 83.3%), whereas SDPO declines across successive preferences from 93.8% to 82.0% and 74.5%.
  • Faster acquisition without interference: Because UMOPD subtracts baseline logits log⁡T0,j\log T_{0,j} to adjust only feedback-relevant tokens rather than rewriting entire responses, it learns new preferences much faster mid-training, leading SDPO by +24.3 and +22.7 points on the second preference at Steps 400 and 480, and by +14.7 points on the third preference at Step 640. Together with the three-seed results above, this advantage is consistent across runs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.