- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper considers the problem of aligning LLMs with user intent for personal assistant and agentic applications. The main problem the authors identify is that user feedback is noisy, incomplete, or open to revision, therefore, using a single-turn as a direction for update might be misleading on the long-term. To address this challenge, the authors propose the following contributions
- Establish feedback sufficiency to measure the extent and reliability of the user feedback as a policy update signal.
- Introduce Hindsight of Hindsight (HOH), a new supervision approach grounded in feedback sufficiency
- Propose the UMOPD algorithm for model alignment that can adapt based on feedback sufficiency
Critical strengths
- Well-motivated, empirically grounded problem framing. The finding that most fully executed feedback is still refined afterwards is replicated across two corpora, partially validated by humans, and shown not to reduce to task vagueness (Cramér’s V = 0.133). This directly challenges the core assumption of current hindsight-distillation methods, which makes the contribution relevant beyond the proposed algorithm.
- Large gains under naturalistic feedback, with informative ablations. The method’s advantage appears exactly where baselines degrade (UserLM rather than prompted Qwen). Removing HOH, the feedback contrast (), adaptive agreement, or multiple teachers each causes failure in at least one protocol, which supports the claim that the components are complementary rather than redundant.
Critical weaknesses
- Writing Clarity: Section 3.1. is missing a clear explanations of some terminology such as domains for y_t, u_t, O_t, C_t, or G_t. Without them, it is hard to understand the notation. Consequently, feedback sufficiently, c_t, is only informally defined. An example here could improve clarity by much.
- The paper assumes that the revision turn y_{t+1} is meant to only correct y_t, however, in real interactions the user might give implicit or explicit feedback on y_t in u_t but also request further actions in y_{t+1} which were not part of the history H_t. This is a gap between the model's framework and the real user interactions.
- The experimental section is limited. It considers a single model family (Qwen3), small scales (4B and 8B), and simple clean datasets (TL;DR, HelpSteer2). Despite the authors use agentic applications as motivating examples and real-world datasets in Section 3, Section 5 is much simpler leaving it ambiguous whether UMOPD can really scale up to these tasks.
Questions for authors
- The paper extends previous works which consider only the immediate turn feedback to a second-turn feedback. Can this framework be extended to multiple consecutive turns, potentially extracting the feedback from the whole remaining interaction between an agent/assistant and a user?
- In Figure 4, there is a clear difference in results between the Qwen and UserLM simulations. Do the authors know why this is the case?
- Could the authors provide confidence estimates for the results in Section 5? Without them it is hard to estimate whether methods are really significantly different.
- How does UMOPD compare to other algorithms used for comparison in terms of compute requirements?
6 comments
Response to Reviewer pMuj (Part 2/5)
2. Clarifying Real-World Interaction Assumptions (W2)
We fully agree that in real-world interactions, users often give implicit or explicit feedback on while simultaneously requesting further actions or moving on to new steps. We would like to clarify that our framework does not assume that or subsequent turns only correct .
First, we clarify how is used in our framework. During online learning, the policy updates on-policy directly from the current tuple and never uses . In offline HOH, our goal is not to train the policy on the revision itself, but to read the user's subsequent reaction in to assess whether the feedback in provides sufficient guidance for learning (which supervises our online predictor ).
Precisely because real conversations often mix feedback with additional requests or follow-up actions across turns, offline HOH never treats an entire turn as a monolithic signal. Instead, it tracks only the feedback-implied change across all three elements:
- In the feedback : We extract only the specific feedback information directed at to anchor , separating it from any additional actions requested in the same turn.
- In the revision : We examine only to verify that was executed, ruling out assistant non-compliance as a confounder. We do not require to only revise , so executing additional requested actions in does not affect this check.
- In the subsequent turn : We extract only the evidence concerning the same change . If the user only follows up on the additional actions or moves on to a new topic without providing identifiable evidence about , the window is labeled as No evidence () and excluded from predictor training.
This is also why we learn HOH from large-scale in-the-wild logs: even after strictly filtering for verified execution of and attributable evidence in , the corpus provides ample clean supervision to train .
3. Clarification on HOH and Extension to Multiple Consecutive Turns (Q1)
We appreciate this thoughtful question. We would first like to gently clarify the role of the second turn in HOH. Whereas prior work directly treats the immediate feedback as a learning target for , HOH uses the user's subsequent reaction in offline logs to assess whether itself provides sufficient guidance for learning. During online interaction, the model therefore still learns continuously from each feedback turn as it arrives.
At the same time, offline HOH can indeed be naturally extended to multiple consecutive future turns (or the remaining conversation). Because our assessment is anchored to the specific change , we can track and extract evidence whenever a later turn refers back to . In fact, we explored this multi-turn extension when designing our work, and found that looking beyond successfully recovers delayed evidence that has not yet appeared in (for example, when a user briefly asks a side question in before returning to confirm or refine in ).
To examine the trade-off between recovering delayed evidence and maintaining clean attribution, we tracked , , and across 5,000 sampled windows in WildChat:
| Subsequent Turn | Attributable Evidence for | No evidence Among Active Turns | Newly Recovered from No evidence at |
|---|---|---|---|
| (5-tuple) | 62.1% (3,103 / 5,000) | 37.9% | - |
| 10.4% (519 / 5,000) | 75.5% | 3.5% (173 / 5,000) | |
| 5.3% (266 / 5,000) | 75.0% | 0.6% (28 / 5,000) |
As shown in the table, extending to and recovers delayed evidence for an additional 201 windows (173 at and 28 at ) that were No evidence at . Meanwhile, over 93% of all recovered evidence already appears immediately at , whereas later turns are separated from by intermediate model outputs and new user requests (with over 75% of active turns at and being No evidence). Given the scale of in-the-wild logs, we adopt the 5-tuple as our default in this paper to prioritize unconfounded attribution, while multi-turn aggregation offers a natural extension when delayed evidence is more prevalent.
Response to Reviewer pMuj (Part 3/5)
4. Broader Models and Tasks (W3)
We expanded our evaluation across model families and scales, real-world user queries, and a personal agent scenario, comparing UMOPD against all three baselines (SDPO, OpenClaw-RL, and Mem0):
- Across model families and larger scales: Building on Qwen3-4B and Qwen3-8B in the main text, we further evaluated Llama-3.1-8B-Instruct and Qwen3-32B under the preference reversal setting (Figure 4b, 480 online steps under UserLM-8B feedback):
| Policy Model | UMOPD | OpenClaw-RL | Mem0 | SDPO |
|---|---|---|---|---|
| Qwen3-4B | 98.2% ± 1.3% | 36.6% ± 5.7% | 50.9% ± 4.8% | 7.3% ± 2.3% |
| Qwen3-32B | 98.5% ± 0.9% | 46.9% ± 4.4% | 52.9% ± 4.2% | 48.3% ± 4.6% |
| Llama-3.1-8B-Instruct | 76.9% ± 4.5% | 70.2% ± 4.5% | 34.6% ± 4.6% | 6.7% ± 2.0% |
On Qwen3-32B, larger model capacity mitigates the collapse of SDPO, yet all baselines remain around 50% due to phase-1 policy inertia, whereas UMOPD reaches 98.5% (+45.6 points). Across model families on Llama-3.1-8B-Instruct, SDPO again degrades to 6.7%, while OpenClaw-RL and UMOPD remain above 70%, with UMOPD achieving the highest win rate.
- Real-world user queries: Beyond TL;DR and HelpSteer2, we further evaluated on real-world user queries. Specifically, we sampled 1,200 real-world user requests (spanning coding debugging, technical analysis, and everyday advice) from WildFeedback, which is derived from real WildChat conversations, for long-horizon online training, and evaluated on 512 held-out requests (Qwen3-4B, UserLM-8B feedback, target preference: concise, casual, beginner-friendly):
| WildFeedback | Step 240 | Step 600 | Step 1200 Final Win Rate |
|---|---|---|---|
| UMOPD | 64.0% ± 3.0% | 80.0% ± 2.4% | 92.1% ± 1.4% |
| SDPO | 51.4% ± 3.2% | 56.7% ± 3.2% | 64.4% ± 2.9% |
| OpenClaw-RL | 50.0% ± 3.3% | 48.4% ± 3.1% | 52.7% ± 3.1% |
| Mem0 | 55.8% ± 3.0% | 51.1% ± 3.0% | 49.8% ± 3.0% |
On these diverse real-world queries with frequent follow-up questions, Mem0 and OpenClaw-RL plateau around 50% and SDPO improves slowly, whereas UMOPD steadily climbs above 92% over 1,200 online steps.
- Personal agent scenario: We further extended our evaluation to a personal agent setting, following the personal homework agent setup in OpenClaw-RL. In each session, a student assigns a GSM8K math problem to the agent (Qwen3-4B) using a fixed initial prompt that does not reveal their preference. The student requires the solution to preserve complete step-by-step calculations and arrive at the correct final answer while reading like natural student homework without AI formatting cues (such as bold text, headers, or numbered lists), requesting a rewrite otherwise. Following OpenClaw-RL, we report two metrics: (i) Personalization Score (, measuring whether the agent's first-turn solution simultaneously satisfies natural student style, complete step-by-step derivation, and the correct mathematical answer), and (ii) Sessions to Convergence (the minimum number of sessions to reach 3 consecutive sessions where the first response is directly accepted, up to 72 sessions):
| Simulator | Method | Personalization Score | Sessions to Convergence |
|---|---|---|---|
| Prompted Qwen | UMOPD | 0.89 | 14.2 |
| Prompted Qwen | OpenClaw-RL | 0.79 | 18.8 |
| Prompted Qwen | SDPO | 0.82 | 23.4 |
| Prompted Qwen | Mem0 | 0.48 | 31.6 |
| UserLM-8B | UMOPD | 0.87 | 19.6 |
| UserLM-8B | SDPO | 0.14 | >72.0 |
| UserLM-8B | OpenClaw-RL | 0.00 | >72.0 |
| UserLM-8B | Mem0 | 0.00 | >72.0 |
Under explicit Prompted Qwen feedback, OpenClaw-RL, SDPO, and UMOPD all adapt effectively, with UMOPD converging fastest (14.2 sessions) and achieving the highest score (0.89). Under naturalistic UserLM-8B interaction, however, baselines are disrupted by step-level follow-up questions and fail to converge within 72 sessions, whereas UMOPD converges in 19.6 sessions with a score of 0.87 while maintaining standalone test-set math accuracy at 91.4% to 93.8%, matching at 92.2%. This demonstrates that UMOPD generalizes effectively to multi-session personal agent scenarios without sacrificing underlying reasoning accuracy.
Response to Reviewer pMuj (Part 1/5)
We sincerely thank the reviewer for the thoughtful and constructive feedback.
1. Clarifying Notation and Examples (W1)
Thank you for pointing this out. We clarify the notation and formal definition in Section 3.1 together with a concrete example below (further interaction examples are also provided in Appendix A.1).
Let V∗ denote the space of natural-language text sequences over vocabulary V. In HOH, we use offline, in-the-wild interaction logs to retrospectively assess whether a user feedback turn provides sufficient guidance for learning. Each offline window is a 5-tuple wt=(Ht,yt,ut,yt+1,ut+1):
When an online learner receives feedback, only the current context (Ht,yt,ut)∈(V∗)∗×(V∗)2 is available. The later turns (yt+1,ut+1) are used only offline to verify whether the feedback on yt is addressed in yt+1 and observe how the user reacts in ut+1.
Before examining ut+1, we anchor the assessment to the feedback-implied change in (Ht,yt,ut), represented as a triple of open-vocabulary text descriptions χt=(Ot,Ct,Gt)∈(V∗)3:
Conditioned on yt+1 fully executing the explicit requirements of χt, feedback sufficiency ct is formally defined as a retrospective evidence mapping ct=Φ(χt,ut+1)∈S∪{⊥} from the anchored change χt and the subsequent user turn ut+1 to the ordered support domain S={Insufficient≺Partially sufficient≺Sufficient}, with ⊥ denoting abstention (No evidence):
ct=Φ(χt,ut+1)=⎩⎨⎧Sufficient,Partially sufficient,Insufficient,⊥ (No evidence),if ut+1 accepts Ct, confirms Gt, or reuses yt+1 with Ct retained,if ut+1 preserves (Ct,Gt) on Ot but adds or refines constraints left open by ut,if ut+1 reverses or rejects Ct, or indicates that Gt was not met,if ut+1 provides no identifiable evidence concerning χt.
Windows with ct=⊥ are filtered out after annotation, so predictor training only uses the three ordinal levels in S to supervise ρϕ(Ht,yt,ut).
For example, consider a real WildChat window (from Appendix A.1):