acceptodds
Under review as a conference paper at ICLR 2027

Learning Personalized Agents from Human Feedback

Abstract

Modern AI agents are powerful but often fail to align with the idiosyncratic, evolving preferences of individual users. Prior approaches rely on static datasets, either training implicit preference models on interaction history or encoding user profiles in external memory. However, these approaches struggle with new users and with preferences that change over time. We introduce Personalized Agents from Human Feedback (PAHF), a framework for continual personalization in which agents learn online from live interaction using explicit per-user memory. PAHF operationalizes a three-step loop: (1) seeking pre-action clarification to resolve ambiguity, (2) grounding actions in preferences retrieved from memory, and (3) integrating post-action feedback to update memory when preferences drift. To evaluate this capability, we develop a four-phase protocol and two benchmarks in embodied manipulation and online shopping. These benchmarks quantify an agent's ability to learn initial preferences from scratch and adapt to persona shifts. Our theoretical analysis and empirical results show that integrating explicit memory with dual feedback channels is critical: PAHF learns substantially faster and consistently outperforms both no-memory and single-channel baselines, reducing initial personalization error and enabling rapid adaptation to preference shifts.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
personalized AI agents, continual personalization, agent memory, human-AI interaction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mF6jOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces PAHF, which continuously updates a memory of user preferences through pre-action clarification and post-action correction. In object-selection and product-selection tasks, the authors separately evaluate initial learning, held-out testing, adaptation after preference changes, and subsequent testing. The experiments compare feedback channels, memory backends, and interaction strategies, and also examine feedback noise and variation in how preferences are expressed.

Critical strengths
  1. Separating learning phases from tests without feedback helps establish whether information acquired through interaction is retained and applied to new situations. Evaluations before and after preference changes also provide a clear way to examine adaptation separately.

  2. The ablations examine the distinct roles of pre-action clarification and post-action correction, and the comparisons are repeated across multiple memory backends. The appendix results with A-MEM and Mem0 strengthen confidence that the findings extend beyond a simple storage implementation.

  3. The analyses of Always-ask, read-only profiles, ambiguous preference expressions, and variation in simulated user language make the experimental study reasonably comprehensive. In particular, comparing response accuracy during interaction with performance in later memory tests helps distinguish the quality of immediate feedback from its contribution to longer-term memory updates.

Critical weaknesses
  1. The Pre-action Only baseline generally stops asking questions once it retrieves relevant memory, so its difficulties after preference drift may partly reflect the rule used to trigger clarification. The Always-ask variant in the appendix still includes post-action feedback and therefore does not provide a direct comparison with a purely pre-action strategy that actively rechecks existing memories. A baseline that periodically reconfirms preferences under a comparable interaction budget would help clarify how much the benefit of post-action feedback depends on the current questioning rule.

  2. Preference changes are implemented mainly by replacing or resampling entire user profiles at once, while the adaptation phase reuses earlier training scenarios. This is a useful way to control changes in the inputs, but it offers limited evidence about partial preference updates and the retention of unchanged preferences during continuous personalization. Allowing subsets of preferences to change at different times, and testing both changed and unchanged preferences, would make it possible to assess adaptation speed and the risk of incorrectly overwriting existing memories.

  3. The noise experiments perturb feedback only in the first round of initial learning, after which interactions return to clean feedback. The ambiguous expressions and language rewrites also preserve the correct preference information. These experiments leave open how the method would behave under persistent feedback errors, omissions, or delays, which could affect both drift detection and memory updates. Applying such perturbations throughout learning and adaptation would provide a more complete assessment of robustness during online use.

  4. I would appreciate clarification of how the definition of “unambiguous” relates to the construction in the proof of Proposition 1. In that construction, preferences can change at an unknown time while the observable inputs remain unchanged. A Bayes-optimal decision-maker may still face uncertainty given the actual interaction history, which raises the question of whether the construction satisfies the stated unambiguity condition. Could the authors clarify whether the term refers to the model's subjective confidence or to the identifiability of the true preference, and revisit the assumptions and necessity claim accordingly?

Questions for authors

My questions are included in the weaknesses above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer mF6j [1/4]

We thank the reviewer for the thoughtful review. The reviewer's questions are contained in the four weaknesses, so we answer W1–W4 one by one, in the order of the review. Every answer names the place in the revised paper where the change is (new text is in blue). SR is the success rate in %, and ACPE is the paper's average cumulative personalization error during a learning phase (lower is better).

W1. "The Pre-action Only baseline generally stops asking questions once it retrieves relevant memory … The Always-ask variant in the appendix still includes post-action feedback … A baseline that periodically reconfirms preferences under a comparable interaction budget would help clarify how much the benefit of post-action feedback depends on the current questioning rule."

We added exactly this baseline, at PAHF's interaction budget and at lower and higher re-confirmation rates. Unlike Always-ask, it never receives post-action feedback, so it shows what pre-action feedback can do on its own. Before the agent acts on a note it retrieved from memory, it asks the user, with probability q, to confirm the note. It writes the answer to memory, revises the note if the user reports a change, and then acts. For the comparable budget, we set q = 0.81, which gives the same number of interactions per task as PAHF (0.77: its questions plus the post-action feedback that led to a memory update). Embodied domain, seed 1 (Table 15 in the revised paper):

questions per learning taskPhase 2 SRPhase 3 ACPE ↓Phase 4 SR
Pre-action only (original baseline)0.0655.867.235.3
Pre-action only + re-confirmation, q = 0.20.2262.249.854.2
Pre-action only + re-confirmation, q = 0.40.3867.837.059.5
Pre-action only + re-confirmation, same budget as PAHF (q = 0.81)0.7673.221.663.9
Pre-action only + re-confirmation, q = 10.9376.516.265.7
Post-action only077.437.467.8
PAHF0.0776.836.468.2

Re-confirmation recovers from drift without post-action feedback, so the failure of Pre-action only in Table 1 is tied to its rule for when to ask, as the reviewer suspected. But a pre-action agent has no way to know when a silent change has happened (Prop. 1), so it can recover only by asking even when it is confident, and its accuracy follows the number of questions it asks. At the comparable budget the reviewer asks for, re-confirmation spends the whole budget on questions (0.76 per task, against 0.07 for PAHF). It makes fewer errors while re-learning, because each avoided error is replaced by a question, but it is less accurate than PAHF in both test phases (−3.5 and −4.3 points, p ≤ 0.013). With a smaller budget it falls 9 to 15 points below PAHF in both test phases (all p < 10⁻⁴), and even when it asks before 93% of its actions it does not exceed PAHF.

So, in the embodied domain, the benefit of post-action feedback does not disappear when the questioning rule is improved. Prop. 1′ (under W4) shows why: without post-action feedback, every pre-action policy must pay for drift either in mistakes or in questions. Revised paper: App. G.4 and Table 15 report this baseline, and Remark 2 in App. A states the claim precisely: post-action feedback is necessary for agents that ask only when their memory does not already answer the request, and re-confirmation is an alternative that trades errors for interruptions.

AuthorsOpenReview ↗

Response to Reviewer mF6j [2/4]

W2. "Preference changes are implemented mainly by replacing or resampling entire user profiles at once … Allowing subsets of preferences to change at different times, and testing both changed and unchanged preferences, would make it possible to assess adaptation speed and the risk of incorrectly overwriting existing memories."

We ran this experiment, and PAHF handles partial changes well: it re-learns changed preferences within about two iterations and largely keeps the unchanged ones. The setup (embodied domain, seed 1): for each user, 25% of the situations change at iteration 1 of Phase 3 (group 1), another 25% change at iteration 3 (group 2), and 50% never change. As in the paper, Phase 3 reuses the Phase 1 scenarios and Phase 4 uses held-out tasks. We score the three groups separately, and for the unchanged situations we also compare Phase 4 with the same tasks in Phase 2. The Phase 3 columns show the success rate per iteration (only the first and the last iteration where it stays flat). The last two rows add a consolidation step that we explain below (Table 17 in the revised paper, App. G.6):

Group 1 (changes at it. 1), Phase 3Group 2 (changes at it. 3), Phase 3: it. 1, 2 → 3 → 4 → 5Unchanged, Phase 3Phase 4: group 1 / group 2 / unchangedUnchanged: Phase 4 − Phase 2
No memory–––45.7 / 44.9 / 36.9+0.5
Pre-action only29.9 → 34.557.6, 58.7 → 31.9 → 33.0 → 31.961.6 → 63.038.0 / 34.3 / 57.4+2.1
Post-action only14.1 → 73.9 → 83.1 → 86.6 → 89.181.5, 79.0 → 13.0 → 70.3 → 80.884.7 → 78.369.6 / 63.4 / 73.6−1.7
PAHF18.7 → 74.3 → 79.9 → 84.9 → 89.484.4, 82.2 → 15.9 → 64.9 → 82.282.8 → 77.764.1 / 67.9 / 71.9−5.8
Post-action only + consolidation step17.6 → 82.7 → 88.0 → 88.4 → 90.583.0, 76.1 → 14.9 → 76.1 → 83.081.7 → 79.173.2 / 72.1 / 72.8−3.5
PAHF + consolidation step19.7 → 84.2 → 83.5 → 89.4 → 89.481.9, 81.5 → 17.0 → 73.2 → 85.583.4 → 79.470.7 / 70.6 / 74.4−0.6

Adaptation speed. With post-action feedback, the agent re-learns each group within about two iterations after it changes (group 1, PAHF: from 18.7 to 74.3 after one iteration, and 89.4 at the end).

Changed vs unchanged preferences. Pre-action only does not re-learn the changed preferences (below 40 in Phase 4), although it keeps the unchanged ones. Post-action feedback is what makes the difference.

Risk of overwriting. Most unchanged preferences are kept. On the unchanged tasks, Phase 4 is 1.7 points below Phase 2 for Post-action only and 5.8 points below for PAHF (the difference between the two methods is not significant, p = 0.10).

A consolidation step does not add to this risk. In our response to omsy (W1) we add a consolidation step that rewrites stored notes when a new note contradicts them. One could expect it to overwrite preferences that did not change, so we reran the partial-drift experiment with it (last two rows). It does not: with the step, Phase 4 accuracy on unchanged preferences does not drop (74.4 for PAHF, against 71.9 without it), and changed preferences are re-learned better (PAHF: 70.5 vs 65.7, p = 0.02) and faster.

We also added gradual drift. During Phase 3, the user follows the new preference with a probability that rises from 0.2 to 1.0, so the user's own feedback switches between old and new (Table 16 in the revised paper, App. G.5). Agents with post-action feedback follow the change as it happens and end about 5 points below the abrupt-drift setting in Phase 4 (PAHF: 62.4). Pre-action only does not adapt (32.5).

W3. "The noise experiments perturb feedback only in the first round of initial learning … These experiments leave open how the method would behave under persistent feedback errors, omissions, or delays, which could affect both drift detection and memory updates."

We added all three. In the embodied domain, PAHF keeps its advantage under each of them, and both methods still recover after drift. Each perturbation is applied to the post-action feedback in every learning iteration of Phases 1 and 3, so it affects both the initial learning and the adaptation after drift:

  • Omissions: the user gives no feedback 30% of the time.
  • Errors: 20% of the feedback is wrong. The user approves a wrong action, or complains about a right action and names another option. Unlike the rewrites in the paper, this feedback carries wrong information about the preference.
  • Delays: the feedback reaches the agent one task later.
AuthorsOpenReview ↗

Response to Reviewer mF6j [3/4]

Embodied domain, seed 1, PAHF / Post-action only (Table 18 in the revised paper, App. G.7):

Phase 1 first pass SRPhase 1 ACPE ↓Phase 2 SRPhase 3 ACPE ↓Phase 4 SR
Clean feedback50.8 / 36.324.3 / 28.276.8 / 77.436.4 / 37.468.2 / 67.8
30% omitted50.6 / 34.426.4 / 31.576.8 / 78.641.0 / 42.066.7 / 68.8
20% inconsistent50.0 / 36.030.9 / 34.270.7 / 70.840.5 / 41.965.5 / 65.6
Delayed by one task51.4 / 35.724.6 / 27.777.3 / 75.536.8 / 36.869.8 / 68.8

In every condition, PAHF keeps its cold-start advantage over Post-action only (at least +14 points in the first pass of Phase 1, p < 10⁻⁴). This is expected, because the answers to pre-action questions do not depend on post-action feedback. Omissions slow learning down but leave Phase 2 unchanged. Errors slow learning down more and cost about 6 points in Phase 2. Delays have no measurable effect. Drift detection and memory updates keep working: after drift, both methods still recover, Phase 4 stays within 3 points of clean feedback, and the two methods are not significantly different. Revised paper: Table 18 and the paragraph "Persistent omissions, errors, and delays" in App. G.7.

We also tested feedback that carries less information: the user never says that a preference changed, or only rejects the action. When the user only rejects, PAHF's advantage over Post-action only is much larger (13.5 points in Phase 2). The details, and two small changes to the memory update that these tests led to, are in our response to omsy (W1) and in App. G.7 (Tables 18–20).

W4. "I would appreciate clarification of how the definition of 'unambiguous' relates to the construction in the proof of Proposition 1. … Could the authors clarify whether the term refers to the model's subjective confidence or to the identifiability of the true preference, and revisit the assumptions and necessity claim accordingly?"

We meant the agent's subjective confidence: what the agent believes given its memory. The reviewer is right that the definition as written does not say this. It speaks of the posterior conditioned on the transcript and of the Bayes-optimal error. If that posterior is computed with knowledge of the proof's prior over the switch time, then P(τ ≤ t | transcript) = t/T, the error in round t is min(t/T, 1 − t/T), which is at least ε₀ for most t, and the rounds are not unambiguous.

The fix is in the definition, not in the result. The prior over the switch time is the environment's randomization, the standard device for proving a worst-case bound. It is not knowledge that the agent has. Once γ is defined as instruction ambiguity (below), the proof of Prop. 1 goes through unchanged. An agent that does reason about possible drift is a different agent. It is outside the scope of Prop. 1, and we now cover it with Prop. 1′: it can reduce its mistakes only by asking the user again and again. So the paper's conclusion holds for both kinds of agent: without post-action feedback, the agent pays for drift either in mistakes (Prop. 1) or in questions (Prop. 1′). The revised paper makes this explicit:

(i) We define γ as instruction ambiguity: the uncertainty about what the instruction refers to, if the agent's current memory is taken as the user's true preference. This is the uncertainty that a clarifying question removes. Whether the memory itself is outdated is a different kind of uncertainty (drift uncertainty). The definition is changed in Sections 3.1 and 3.3 and in App. A.

(ii) We state Prop. 1 for policies that ask pre-action questions only when the instruction is ambiguous. Prop. 1 now reads "unambiguous given the agent's memory", and a new subsection at the end of App. A states its scope.

(iii) We add Prop. 1′ for every pre-action policy, including policies that ask about drift. It is in the same subsection, with a full proof, and Remark 1 in Section 3.3 points to it.

(iv) We state the necessity claim for agents that ask only when their memory does not already answer the request, and we describe re-confirmation as an alternative that trades errors for interruptions (Remark 2 in App. A; App. G.4).

AuthorsOpenReview ↗

Response to Reviewer mF6j [4/4]

Prop. 1′. Without post-action feedback, any (possibly adaptive, randomized) policy facing K switches that asks B re-confirmation questions in expectation makes E[mistakes] ≥ KT / (16(2B + K)) − K/4. If a question costs c, E[mistakes] + c·B ≥ √(cKT/8) − (2c + 1)K/4, and keeping the mistakes at O(K) takes Ω(T) questions. With post-action feedback, PAHF makes at most K post-switch mistakes using at most K corrections.

Proof sketch. Split the horizon into K blocks of length T′ = T/K. Each block contains one switch at a uniformly random time, independently of the other blocks. Until the user first answers "changed", the only thing the agent knows is that nothing had changed up to its last question. So, as long as it keeps hearing "no change", its questions follow a fixed schedule. Let n be the number of questions that this schedule places in the first half of a block. With probability ½ the switch falls in the second half, and then all n questions are asked, so E[questions] ≥ n/2. The n questions split the first half into n + 1 segments of lengths ℓ_j, which sum to T′/2. Inside a segment the agent learns nothing until its next question, so by the argument of Prop. 1 the switch times in that segment cause at least (ℓ_j² − 1)/4 mistakes in total. Therefore E[mistakes] ≥ Σ_j (ℓ_j² − 1)/(4T′) ≥ T′/(16(n + 1)) − 1/4 by Cauchy–Schwarz. This gives E[mistakes] ≥ T′/(16(2E[questions] + 1)) − 1/4 per block. Summing over the blocks with Jensen's inequality gives the bound, and minimizing over B gives the √(cKT) form. The full proof is in App. A of the revised paper.

The new re-confirmation experiments (W1; Table 15) show the same trade-off: the fewer questions the agent asks, the more mistakes it makes while re-learning, and it comes close to PAHF's final accuracy only by asking before most of its actions, including when nothing has changed.

Presentation. All of these changes are in blue in the revised paper, and Table 15 now reports questions and interactions per task next to the errors. If the reviewer has other passages in mind, we will revise them as well.

Reviewer_omsyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes how agents can learn each user's dynamic preferences from live interaction and adapt their changes. The authors introduce PAHF, which maintains a user preference memory updated through two feedback channels. Pre-action interaction is done before the action, the agent asks a clarifying question when ambiguity remains. After a wrong action, the user's correction is merged into memory to revise outdated preferences, called post-action feedback integration.

On embodied manipulation and online shopping benchmarks with four-phase protocol that seperates initial learning from adaptation to persona shifts, the authors report that PAHF outperforms no-memory and single-channel baselines.

Critical strengths
  1. The problem framing is very clear, and the evaluation protocol supports it well. Section 3.1 splits errors into two types, partial observability and non-stationarity, which gives an intuitive motivation for the two feedback channels. The four-phase protocol carries this framing into the evaluation. Because the training and testing phases are kept separate, the cost incurred during learning and the quality of the learned memory can be measured separately. In particular, Phase 3 reuses the Phase 1 inputs with only the ground-truth labels changed, which helps isolate the agent’s sensitivity to preference drift.
Critical weaknesses
  1. The benchmark’s post-action feedback may be more informative than real-user feedback. For example, the simulator states the preference change explicitly (e.g., “I now like Sprite most”), and in shopping, the rule-based judge identifies the offending feature. Drift is also designed as a relatively clear shift, via logical inversion or profile resampling. Real feedback can be implicit or absent, leaving the agent to judge whether it signals a preference change or a one-off request. It remains unclear whether the results hold under less diagnostic feedback or partial, gradual drift.
Questions for authors
  1. In the released shopping data, “do not buy” becomes the correct answer far more often after drift (about 33% → 68%). As a result, a policy that always chooses this option would perform comparably to Post-action only and PAHF in Phase 4. The large post-drift gains of memory-based methods may therefore partly reflect this shift rather than adaptation to new preferences alone. Could the authors report an always-“do not buy” baseline and each method’s rate of choosing this option per phase?

  2. Would pre-action feedback still be brittle under drift with a stronger Pre-action-only variant, for example one that occasionally re-confirms stored notes before acting? A small experiment or discussion would help clarify whether the gap to methods with post-action feedback reflects the channel itself or the current implementation.

I would be open to raising my score if these two points are adequately addressed in the rebuttal.

5 comments
AuthorsOpenReview ↗

Response to Reviewer omsy [1/4]

We thank the reviewer for reading the released data so carefully and for the concrete requests. We ran the controls that both questions ask for, and the paper's main conclusions hold under both. The reviewer wrote that Q1 and Q2 decide the score, so we answer them first, and then the weakness (W1) in four parts (W1a–W1d). Every answer names the place in the revised paper where the change is (new text is in blue). SR is the success rate in %, and ACPE is the paper's average cumulative personalization error during a learning phase (lower is better).

Q1. "In the released shopping data, 'do not buy' becomes the correct answer far more often after drift (about 33% → 68%). … Could the authors report an always-'do not buy' baseline and each method's rate of choosing this option per phase?"

We ran these controls, and the paper's conclusion holds: the methods with post-action feedback adapt to the changed preferences, and Pre-action only does not. Both requested numbers are in the table below: the always-D baseline is the first row, and the share of items where each method chooses "do not buy" (D) is the last column.

The reviewer's observation is correct. D is the correct answer for 33.3% of the items in Phases 1 and 2, but for 68.8% and 67.8% in Phases 3 and 4, because the options are reused and fewer of them remain acceptable under the changed preferences. This raises the absolute Phase 4 numbers in Table 1. To separate real adaptation from this effect, we add two measures: accuracy on the items whose correct answer is a product, and a balanced Phase 4, with the same users, products and requests but with the options re-drawn so that the mix of correct answers matches Phase 2 (3 seeds for Post-action only and PAHF, 1 seed for the other methods):

MethodPhase 2 SRPhase 4 SRbalanced Phase 4 SRPhase 4 SR, items whose answer is a productchooses D (%), Phase 1 / 2 / 3 / 4
Always D33.367.833.30.0100 / 100 / 100 / 100
No memory27.731.629.620.736 / 38 / 38 / 38
Pre-action only36.053.831.811.470 / 73 / 72 / 73
Post-action only37.3 ± 0.369.3 ± 0.737.2 ± 0.331.0 ± 1.465 / 65 / 79 / 78
PAHF38.6 ± 1.269.0 ± 0.938.9 ± 0.532.2 ± 0.569 / 64 / 77 / 77

With the shift removed, the adaptation remains. On the balanced Phase 4, PAHF is 9.4 points above no memory and 5.6 points above the always-D policy (both p ≤ 10⁻⁴; it beats always-D for 19 of 20 users), and Post-action only is 7.6 and 3.9 points above them (both p < 10⁻³), while Pre-action only does not adapt. The agents also choose products better: when they choose one, the choice is right in 40.5% (PAHF) and 34.2% (Post-action only) of the cases, against 24.0% without memory (p ≤ 2 × 10⁻⁴), and on the original Phase 4 items whose answer is a product, where always-D scores 0, both are about 10 points above no memory (p < 0.004). So the ranking of the methods and the paper's conclusion are unchanged. What the shift affects is the absolute size of the Phase 4 gain in Table 1.

We also balanced Phase 3 in the same way, because in the table above the agents still learn on the original Phase 3, where D is correct for 68.8% of the items. The result is the same (PAHF 39.8, Post-action only 37.2; 13.0 and 10.4 points above no memory, both p < 10⁻⁴), so it does not depend on learning under the shifted answer mix, and PAHF is again ahead of Post-action only (+2.6 points, p = 0.05). Revised paper: App. G.3, Tables 13 and 14.

Q2. "Would pre-action feedback still be brittle under drift with a stronger Pre-action-only variant, for example one that occasionally re-confirms stored notes before acting?"

The gap in Table 1 is tied to the rule of asking only when memory does not already answer the request, but this rule is not an implementation detail: when a preference changes silently, an agent that trusts its memory has no reason to ask (Prop. 1). The variant the reviewer suggests asks even when the agent is confident, and it does recover from drift, but only as far as the number of questions it asks allows.

We implemented it. Before the agent acts on a note it retrieved from memory, it asks the user, with probability q, to confirm the note, and it updates its memory with the answer. There is no post-action feedback. We ran q = 0.2 and q = 0.4 (occasional re-confirmation, as the reviewer suggests), q = 0.81 (the same interaction budget as PAHF, 0.77 per task: its questions plus the post-action feedback that led to a memory update) and q = 1. Embodied domain, seed 1:

AuthorsOpenReview ↗

Response to Reviewer omsy [2/4]

questions per learning taskPhase 2 SRPhase 3 ACPE ↓Phase 4 SR
Pre-action only (original baseline)0.0655.867.235.3
Pre-action only + re-confirmation, q = 0.20.2262.249.854.2
Pre-action only + re-confirmation, q = 0.40.3867.837.059.5
Pre-action only + re-confirmation, same budget as PAHF (q = 0.81)0.7673.221.663.9
Pre-action only + re-confirmation, q = 10.9376.516.265.7
Post-action only077.437.467.8
PAHF0.0776.836.468.2

Occasional re-confirmation is not enough: at q = 0.2 and q = 0.4 the agent stays 9 to 15 points below PAHF in both test phases (all p < 10⁻⁴). With the same budget as PAHF, it asks before 76% of its actions and is still less accurate in both test phases (−3.5 and −4.3 points, p ≤ 0.013). It comes close to PAHF only at q = 1, where it asks before 93% of its actions, while PAHF asks before about 7%. At these two high rates it makes fewer errors while re-learning, because each avoided error is replaced by a question. Prop. 1′ proves that this trade-off is not specific to our variant: the total cost of re-confirmation grows at least as √(cKT), while correcting after an action costs O(K). Our benchmark is also the best case for re-confirmation, because all preferences change at once. When changes are rare, most confirmation questions are wasted.

So, in the embodied domain, post-action feedback lets an agent recover from drift with about 10 times fewer questions before acting (0.07 vs 0.76 per task). Revised paper: App. G.4 and Table 15 report this baseline, and Prop. 1′ with its full proof is at the end of App. A. The claim is now stated precisely (Remark 2 in App. A): post-action feedback is necessary for agents that ask only when their memory does not already answer the request, and re-confirmation is an alternative that trades errors for interruptions.

W1. "The benchmark's post-action feedback may be more informative than real-user feedback. … Real feedback can be implicit or absent, leaving the agent to judge whether it signals a preference change or a one-off request. It remains unclear whether the results hold under less diagnostic feedback or partial, gradual drift."

We tested each part of this concern directly, in every learning iteration of Phases 1 and 3, and the results hold. In the embodied domain, PAHF keeps its advantage over Post-action only at cold start under every feedback variant (at least +13.5 points in the first pass of Phase 1, p < 10⁻⁴), and the advantage is larger when post-action feedback is weak. So less informative feedback makes the case for combining the two channels stronger, not weaker. Where it did hurt, the cause was a detail of the memory update, which two small changes fix, and after drift PAHF and Post-action only are tied under every variant (bare rejections: with the first change; implicit feedback: with both). We answer the four parts of the concern below (W1a–W1d). Revised paper: App. G.5–G.7 (Tables 16–20) have all of these results, and the limitations (App. B) discuss how learning degrades when feedback carries less information.

The embodied variants are implicit (the user only says what they want instead, and never mentions a change or a reason), minimal (the user only rejects the action: "No, that's not what I wanted."), 30% omitted, 20% inconsistent (20% of the feedback contradicts the user's real preference) and delayed (the feedback arrives one task later). In shopping, the implicit judge names the feature that was wrong but does not say that a preference changed, and the minimal judge does not name the feature either. Each cell shows PAHF / Post-action only for seed 1, and the "context-aware update" rows use a fix that we explain under W1a (Tables 18 and 19 in the revised paper):

EmbodiedPhase 1 first pass SRPhase 1 ACPE ↓Phase 2 SRPhase 3 ACPE ↓Phase 4 SR
Standard feedback50.8 / 36.324.3 / 28.276.8 / 77.436.4 / 37.468.2 / 67.8
30% omitted50.6 / 34.426.4 / 31.576.8 / 78.641.0 / 42.066.7 / 68.8
20% inconsistent50.0 / 36.030.9 / 34.270.7 / 70.840.5 / 41.965.5 / 65.6
Delayed by one task51.4 / 35.724.6 / 27.777.3 / 75.536.8 / 36.869.8 / 68.8
Implicit48.2 / 34.838.4 / 46.560.1 / 55.252.8 / 50.649.4 / 53.0
Minimal54.9 / 34.841.0 / 64.956.1 / 36.867.1 / 49.835.9 / 47.7
Implicit, context-aware update51.7 / 34.824.2 / 29.370.4 / 69.144.6 / 44.456.3 / 61.4
Minimal, context-aware update49.9 / 33.432.4 / 44.259.8 / 46.249.8 / 58.252.7 / 50.7
No memory; Pre-action only35.2; 52.064.8; 42.436.9; 55.850.0; 67.246.5; 35.3
AuthorsOpenReview ↗

Response to Reviewer omsy [3/4]

ShoppingPhase 2 SRPhase 4 SRbalanced Phase 4 SR
Standard feedback36.4 / 37.970.8 / 70.738.0 / 37.6
Implicit39.9 / 39.668.8 / 69.438.9 / 40.2
Minimal32.3 / 31.263.1 / 63.331.6 / 32.6
No memory; always D27.7; 33.331.6; 67.829.6; 33.3

W1a. Implicit or less diagnostic feedback. Shopping. Whether the user announces the change makes no difference (implicit row). When the judge does not name the wrong feature either (minimal row), memory helps less: Phase 2 lies between no memory and standard feedback.

Embodied. The loss under implicit and minimal feedback comes from one prompt in the memory update, not from the framework. The agent wrote its memory note from the feedback text alone, so with implicit feedback the note lost the situation, and a bare rejection left nothing useful to write down. We changed one line of the prompt, so that the agent also sees the scenario and its own action when it writes the note (the context-aware update). With it, Phase 2 under implicit feedback recovers most of the loss (PAHF: 70.4, up from 60.1). Under bare rejections, where the pre-action channel matters most, PAHF is 13.5 points above Post-action only in Phase 2 (p < 10⁻⁴) and recovers from drift again (Phase 4: 52.7, up from 35.9).

After drift, recovery under implicit feedback was still slower, and slower for PAHF than for Post-action only. The cause is outdated notes: the memory revised a stored note only when the feedback said that a preference changed, which implicit feedback never does, and PAHF carries more outdated notes because it also stores the answers to its clarifying questions. We therefore added a consolidation step, similar to the UPDATE step of memory systems such as Mem0: each time the agent writes a note, one extra LLM call rewrites the most similar stored notes that state a different preference for the same situation. The table shows PAHF / Post-action only, and the last column is the share of Phase 4 tasks with the old, pre-drift answer:

ConditionPhase 1 ACPE ↓Phase 2 SRPhase 3 ACPE ↓Phase 4 SRpicks the old answer in Phase 4 (%) ↓
Implicit, context-aware update (3 seeds)23.9 / 28.970.6 / 68.745.1 / 45.356.7 / 59.920.3 / 18.1
+ consolidation step (3 seeds)25.3 / 29.869.8 / 69.434.9 / 36.264.9 / 65.412.7 / 12.2
Standard feedback (seed 1)24.3 / 28.276.8 / 77.436.4 / 37.468.2 / 67.812.2 / 12.3
+ consolidation step (seed 1)26.4 / 30.075.7 / 75.233.5 / 35.472.5 / 72.49.6 / 9.3

With the step, errors during re-learning drop by about 10 points for both methods, Phase 4 rises by 8.2 points for PAHF (p < 10⁻⁴), and the two methods are tied again. The step also helps under standard feedback (Phase 4: +4.3 points for PAHF, p = 0.004), and its cost is small (Phase 1 ACPE rises by at most 2.1 points). Revised paper: App. G.7, Tables 18–20.

W1b. Feedback that is absent. Missing feedback slows learning down but does not change what is learned: with 30% of the feedback missing, Phase 2 is unchanged (76.8 for PAHF, as with clean feedback), and Phase 4 ends at about the same level. Feedback that arrives one task late has no measurable effect. Revised paper: App. G.7, Table 18.

W1c. Is the feedback a preference change or a one-off request? The paper tests this directly (App. F.3): for 100 one-off requests, 100 context-dependent preferences and 100 permanent changes, the memory is correct after the update in 89%, 97% and 100% of the cases. PAHF merges the old and the new content when it revises a note, so a one-off request that is taken for a change usually keeps both scopes. The new results extend this to harder settings: with 20% inconsistent feedback, where some feedback is not a real change, Phase 2 drops by only about 6 points, and under gradual drift (W1d), where a real change looks like one-off requests, the agents still follow the change. When a change is never announced, the consolidation step (W1a) removes the dependence on the user's wording. Revised paper: App. G.7 (Tables 18 and 20) and App. G.5.

W1d. Partial or gradual drift. PAHF handles both (embodied domain). Partial drift. For each user, 25% of the situations change at the start of Phase 3, another 25% change halfway through, and 50% never change. With post-action feedback, the agent re-learns each group of changed preferences within about two iterations (first group, PAHF: from 18.7 right after the change to 74.3 one iteration later) and largely keeps the unchanged ones, and the consolidation step does not hurt them. Pre-action only does not learn the changed preferences (below 40 in Phase 4). Revised paper: App. G.6, Table 17.

Reviewer_vPaMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PAHF, a framework for continual agent personalization. It combines pre-action clarification, explicit user memory, and post-action feedback to learn evolving user preferences. The authors also introduce two benchmarks and a four-phase evaluation protocol to test initial personalization and preference drift.

Critical strengths
  1. The paper is well-structured and easy to follow. The problem of continual personalization is meaningful and practical.
  2. The four-phase evaluation protocol is well-designed. Experiments cover two domains, different LLMs, and multiple memory backends.
Critical weaknesses
  1. The theoretical analysis relies on strong assumptions. For example, Proposition 1 claims a general lower bound, but the proof mainly considers a worst-case preference switch. More clarification is needed.

  2. All experiments rely on simulated users. Although the authors test different simulators, the effectiveness of PAHF with real human feedback remains unclear.

  3. The improvement over Post-action Only is relatively limited in some settings. For example, PAHF achieves 68.8% versus 68.3% in Phase 4 of embodied manipulation (Table 1). The statistical significance of these gains is unclear.

Questions for authors
  • Could the authors clarify whether the lower bound in Proposition 1 applies to all environments or only the worst case?
  • Could the authors provide statistical significance tests for the improvements over Post-action Only?
  • How would PAHF perform with gradual preference drift or inconsistent human feedback?
3 comments
AuthorsOpenReview ↗

Response to Reviewer vPaM [1/3]

We thank the reviewer for the review and the precise questions. Below we answer each weakness (W1–W3) and each question (Q1–Q3) in the order of the review. W1 and Q1 are both about Prop. 1, and W3 and Q2 are both about significance. In each pair, we give our position under the weakness and the full detail under the question. Every answer names the place in the revised paper where the change is (new text is in blue). SR is the success rate in %, and ACPE is the paper's average cumulative personalization error during a learning phase (lower is better).

W1. "The theoretical analysis relies on strong assumptions. For example, Proposition 1 claims a general lower bound, but the proof mainly considers a worst-case preference switch."

Worst case. Prop. 1 is a necessity result, and a necessity result is by nature a worst-case statement: to show that post-action feedback is necessary, one shows that there are environments in which no policy can succeed without it. This is the standard form of a lower bound in online learning. Our bound is also stronger than a single adversarial case. For any policy without post-action feedback, at least half of all possible switch times lead to at least T/4 expected mistakes (the argument is under Q1). Revised paper: App. A, new subsection "Scope of Proposition 1 and Re-confirmation Policies", paragraph "The lower bound is not a corner case".

Assumptions. The two assumptions only give the bounds a simple form; the qualitative conclusions do not depend on them. A1 says that each answer to a clarifying question is noise-free and rules out a fixed fraction of the wrong hypotheses. A2 says that a correction after a wrong action reveals the right action. A2 can be relaxed directly: if a correction reveals the right action only with probability p, the number of mistakes after the switches is O(K/p) instead of K. The experiments do not use the assumptions at all: questions, answers and feedback are free-form text. The new experiments even break A2 on purpose in every learning iteration, with missing feedback, wrong feedback, and feedback that arrives one task late. PAHF's cold-start advantage is unchanged in all three cases. Missing and late feedback cost little, and wrong feedback costs about 6 points in Phase 2. (See Q3 below. The full table is in our response to omsy, W1, which also covers less informative feedback.) Revised paper: the relaxation of A2 (paragraph "Imperfect corrections") and the scope of Prop. 1 are in the same subsection of App. A, and the runs with imperfect feedback are in App. G.7 (Table 18).

W2. "All experiments rely on simulated users. … the effectiveness of PAHF with real human feedback remains unclear."

Using simulated users is a deliberate choice, and for this problem we believe it is the right one. Continual personalization can only be evaluated interactively: the agent chooses its own questions and actions, so the user must be able to answer any question and react to any action. The evaluation also needs the user's true preference in every situation, to score each action, and a preference change at a known time, to measure adaptation. Recorded human data provides none of these, and a human study cannot provide the last two at this scale: one run of the embodied benchmark is 14,400 interactive tasks. This is why LLM-simulated users are common practice in benchmarks for interactive agents (e.g. τ-bench).

What matters is that the conclusions do not depend on one particular simulator, and we test this in several ways. The paper uses three different user models (GPT-4.1, GPT-4o, Llama-4-Scout; App. F), users who never name the item they want, and user language taken from real Amazon reviewers (App. F.3). In the rebuttal we add the ways in which real feedback is most likely to differ from a simulator: feedback that is wrong, missing, late, or less informative, in every learning iteration (Q3 below, and our response to omsy, W1). In the embodied domain, the main conclusions hold in all of them (for less informative feedback, with two small changes to the memory update). Nothing in PAHF itself assumes a simulated user. Revised paper: these experiments are in App. G.7 (Tables 18–20), and the limitations (App. B) name a study with real users as the main direction for future work.

AuthorsOpenReview ↗

Response to Reviewer vPaM [2/3]

W3. "The improvement over Post-action Only is relatively limited in some settings. For example, PAHF achieves 68.8% versus 68.3% in Phase 4 of embodied manipulation (Table 1). The statistical significance of these gains is unclear."

The reviewer's example is the comparison in which no difference is expected. The theory predicts this: both methods have the post-action feedback that drives recovery from drift (Prop. 1), so with enough rounds of feedback they converge to similar memory and should reach similar accuracy in the test phases. The paper reports it as well. App. F.3 notes that Phase 2 success is "nearly identical" for PAHF and Post-action only, and Section 5 says that after drift PAHF "matches the rapid adaptation of Post-action Only". Our three-seed rerun confirms it (Phase 4: 68.2 for PAHF and 68.9 for Post-action only, p = 0.41).

What the paper claims for PAHF over Post-action only is that it "minimizes initial personalization error", and this gain is large and highly significant. In the first pass of Phase 1, PAHF's success rate is 14.5 points higher (p < 10⁻⁴, and higher for all 40 users), which means about 23% fewer errors for the user. Its cumulative error is also significantly lower in both learning phases. Over the other single-channel baseline, Pre-action only, the gain is in the test phases: 21 points in Phase 2 and 33 points in Phase 4. So, among the paper's four methods, PAHF is the only one that is strong in both respects, and this is the paper's claim. The full tests are under Q2. Revised paper: App. G.2 and Table 12 report these tests and say where the gain is, and Section 5 points to them.

Q1. "Could the authors clarify whether the lower bound in Proposition 1 applies to all environments or only the worst case?"

It is a worst-case bound, as lower bounds are, but the worst case here is common, not a rare special case. In the proof, the switch time τ is drawn uniformly at random. This gives E_τ E[mistakes] ≥ ⌊T²/4⌋/T for every policy that does not use post-action feedback. A stronger statement also holds. Let ℓ(τ) be the expected number of mistakes when the switch happens at time τ. For any policy and any τ, ℓ(τ) + ℓ(τ + T/2) ≥ T/2, because in every round between the two switch times the two environments need different actions, and the agent has no way to tell them apart. So for at least half of all switch times, the policy makes at least T/4 mistakes in expectation.

The bound applies under two conditions, which are those of the setting the paper studies: the change is not visible in the agent's inputs, and the agent asks pre-action questions only when the instruction is ambiguous (this is how PAHF and the Pre-action only baseline work). If the user announces the change, post-action feedback is not needed. If the agent instead checks for changes by asking the user to re-confirm its memory, the new Prop. 1′ gives the general form of the result: with B re-confirmation questions, at least KT / (16(2B + K)) − K/4 expected mistakes remain, so the agent must pay for drift either in mistakes or in questions. The statement and proof sketch of Prop. 1′ are in our response to mF6j (W4), and the matching experiment is in our response to omsy (Q2). Revised paper: the scope of Prop. 1, the result for half of all switch times, and Prop. 1′ with its full proof are in App. A ("Scope of Proposition 1 and Re-confirmation Policies").

AuthorsOpenReview ↗

Response to Reviewer vPaM [3/3]

Q2. "Could the authors provide statistical significance tests for the improvements over Post-action Only?"

Yes, and the improvements that the paper claims are highly significant. We ran three seeds of each method. The p-values come from a paired permutation (sign-flip) test over the 40 users, and the 95% confidence intervals from a bootstrap over users, with the three seeds averaged per user. We also ran McNemar's test on the paired tasks within each seed. The table shows the mean ± standard error over seeds in the embodied domain:

Phase 1 first pass SRPhase 1 ACPE ↓Phase 2 SRPhase 3 ACPE ↓Phase 4 SR
Post-action only36.0 ± 0.528.1 ± 0.076.1 ± 0.837.7 ± 0.268.9 ± 0.6
PAHF50.5 ± 0.624.4 ± 0.176.8 ± 0.436.5 ± 0.168.2 ± 0.3
Difference (PAHF − Post) [95% CI]+14.5 [13.0, 16.0]−3.8 [−4.5, −3.0]+0.8 [−0.6, 2.2]−1.2 [−1.9, −0.5]−0.7 [−2.2, 0.8]
p (users better / worse)< 10⁻⁴ (40 / 0)< 10⁻⁴ (38 / 2)0.32 (20 / 16)0.002 (29 / 10)0.41 (20 / 19)

McNemar's test gives the same result in every seed. In the first pass, PAHF makes about 23% fewer errors (an error rate of 49.5% against 64.0%). It also has lower cumulative error in both learning phases, and it reaches the same accuracy in the test phases, as the theory predicts once both methods have received enough rounds of post-action feedback.

Shopping (3 seeds, 20 users) shows the same pattern with a smaller gain at cold start (+3.4 points in the first pass of Phase 1, p = 0.044), and the two methods again reach the same accuracy in the test phases. Revised paper: App. G.2 and Table 12 (both domains). The tests are described in App. G.1.

Q3. "How would PAHF perform with gradual preference drift or inconsistent human feedback?"

PAHF handles both. We ran both experiments in the embodied domain.

(a) Gradual drift. We follow the usual meaning of gradual drift: for a while old and new behavior are mixed, and the share of the new one grows. In iteration i of Phase 3, the user follows the new preference with probability i/5 (0.2, 0.4, …, 1.0) and the old preference otherwise. This is drawn separately for every task, and the draws are the same for all methods. So during most of Phase 3, the user's feedback switches between the old and the new preference. Phases 1–2 are shared with the main runs. We ran 3 seeds for PAHF and Post-action only and 1 seed for the other methods. The last column also gives Phase 4 under the original abrupt drift, in brackets:

Phase 3 SR in iterations 1 → 5 (share of the new preference 0.2 → 1.0)Phase 3 ACPE ↓Phase 4 SR (abrupt drift)
No memory40.0 (no learning)60.047.0 (46.5)
Pre-action only55.4 → 48.3 → 44.5 → 35.7 → 30.857.032.5 (35.3)
Post-action only67.9 → 52.1 → 47.3 → 53.6 → 66.442.564.1 (68.9)
PAHF69.7 → 53.9 → 50.3 → 53.6 → 67.041.162.4 (68.2)

Agents with post-action feedback follow the drift as it happens, and Pre-action only does not adapt: in Phase 4, PAHF is 30 points above it. During Phase 3, PAHF's accuracy on tasks where the user holds the new preference rises from 16 to 67. Compared with Post-action only, PAHF makes significantly fewer errors during the transition (p < 0.001), and the two are not significantly different in Phase 4. Both end about 5 points below the abrupt-drift setting, probably because the user follows the new preference consistently only in the last iteration. Revised paper: App. G.5, Table 16.

(b) Inconsistent feedback. In every learning iteration of Phases 1 and 3, 20% of the feedback contradicts the user's real preference: the user approves a wrong action, or complains about a right action and names another option. The table shows PAHF / Post-action only for seed 1:

Phase 1 first pass SRPhase 1 ACPE ↓Phase 2 SRPhase 3 ACPE ↓Phase 4 SR
Clean feedback50.8 / 36.324.3 / 28.276.8 / 77.436.4 / 37.468.2 / 67.8
20% inconsistent50.0 / 36.030.9 / 34.270.7 / 70.840.5 / 41.965.5 / 65.6

PAHF's cold-start advantage does not change (+14.0 points, p < 10⁻⁴). This is expected, because the answers to pre-action questions do not depend on post-action feedback. Learning becomes slower, and Phase 2 drops by about 6 points for both methods. After drift, both methods still recover, and Phase 4 ends about 3 points below clean feedback. Revised paper: App. G.7, Table 18.

The other reviewers asked for related settings. Partial drift, where only some preferences change and at different times, is in our response to mF6j (W2) and in App. G.6. Missing, late and less informative feedback are in our response to omsy (W1) and in App. G.7.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers [1/2]

We thank the reviewers for their careful reviews. Each response answers every weakness (W) and question (Q) one by one. We ran about 80 new runs on the full benchmarks. The results are in the revised paper (new and changed text is in blue), and every answer names the place of the change. The new evidence supports the three main claims of the paper:

  • Pre-action clarification removes errors at cold start. In the embodied domain, PAHF's success rate in the first pass of Phase 1 is 14.5 points higher than that of Post-action only (p < 10⁻⁴, and higher for all 40 users), and at least 13.5 points higher under every kind of imperfect feedback we tested.
  • Post-action feedback is what lets an agent adapt to drift without constant questions. Pre-action only does not adapt, whether the drift is abrupt, gradual or partial. A pre-action agent that re-confirms its memory comes close to PAHF only when it asks before 93% of its actions (PAHF: about 7%), and the new Prop. 1′ shows that this trade-off cannot be avoided.
  • The two channels complement each other. PAHF has the cold-start benefit of the first channel and the adaptation of the second, and its advantage during learning grows when post-action feedback is weak.

Setup of the new experiments. All new runs use the Table 1 setting (GPT-4o as agent and simulated user, SQLite memory, 5 learning iterations) on the full benchmarks (40 users in the embodied domain, 20 in shopping). As with the separate runs in App. F.3, absolute numbers differ somewhat from Table 1 (embodied Phase 2 is higher), and the pattern is the same (App. G.1).

1. PAHF's gain over Post-action only is where the paper and the theory place it, and it is highly significant (vPaM). The paper claims that PAHF "minimizes initial personalization error" (Section 5), and its App. F.3 already reports that Phase 2 success is "nearly identical" for PAHF and Post-action only. Three seeds confirm both: in the embodied domain, PAHF is significantly better in the first pass of Phase 1 and in cumulative error in both learning phases, and the two methods reach the same accuracy in the test phases. This is consistent with the theory: both have the post-action feedback that drives recovery from drift (Prop. 1), so with enough rounds of feedback they end up with similar memory, and pre-action questions avoid the errors before that point. Against Pre-action only, the gain is in the test phases (21 points in Phase 2, 33 in Phase 4). PAHF is the only one of the four methods that is strong in both respects. Revised paper: App. G.2, Table 12.

2. In shopping, the adaptation result holds after controlling for the shift in correct answers (omsy). Reviewer omsy observed that "do not buy" (D) is the correct answer for about 33% of the items before the drift and about 68% after it, so that a policy that always answers D scores 67.8% in Phase 4. This is correct, and we ran the controls the reviewer asked for. On a balanced Phase 4 (the same users, products and requests, with the options re-drawn so that the mix of correct answers matches Phase 2), PAHF scores 38.9 and Post-action only 37.2. Both are significantly above no memory (29.6) and above the always-D policy (33.3; all p < 10⁻³), while Pre-action only does not adapt. When they choose a product, the choice is also right far more often than without memory, and balancing Phase 3 as well gives the same result. So the gain is not explained by answering D all the time. Revised paper: App. G.3, Tables 13 and 14.

3. A pre-action agent recovers from drift only by asking far more questions (omsy, mF6j). Both reviewers asked for a Pre-action-only baseline that re-confirms its memory. We built it: before the agent acts on a note it retrieved from memory, it asks the user, with probability q, whether the note is still correct, and it never receives post-action feedback. This baseline does recover from drift, but only as far as the number of its questions allows. In the embodied domain, with occasional re-confirmation it stays 9 to 15 points below PAHF in both test phases. With the same number of interactions as PAHF, it asks before 76% of its actions and is still significantly less accurate, and it comes close to PAHF only when it asks before 93%. Prop. 1′ shows that this is not a matter of implementation: without post-action feedback, every policy must pay for drift either in mistakes or in questions. Revised paper: App. G.4, Table 15. Prop. 1′ with its full proof is at the end of App. A, and Remark 2 there states the necessity claim in this sharper form.

AuthorsOpenReview ↗

General response to all reviewers [2/2]

4. The results hold under harder drift and harder feedback (vPaM, omsy, mF6j).

  • Gradual drift. During Phase 3, the user follows the new preference with a probability that rises from 0.2 to 1.0. Agents with post-action feedback follow the change as it happens, and PAHF makes significantly fewer errors than Post-action only during the transition. After the drift, PAHF reaches 62.4, and Pre-action only does not adapt (32.5). Revised paper: App. G.5, Table 16.

  • Partial drift at different times. For each user, 25% of the situations change at the start of Phase 3, another 25% change halfway through, and 50% never change. Agents with post-action feedback adapt to each change within about two iterations and keep most of the unchanged preferences. Pre-action only never learns the changed ones. Revised paper: App. G.6, Table 17.

  • Wrong, missing and late feedback in every learning iteration of Phases 1 and 3 (embodied domain). Missing feedback slows learning but leaves Phase 2 unchanged, wrong feedback costs about 6 points in Phase 2, and feedback that arrives one task late has no measurable effect. After drift, both methods still recover (Phase 4 stays within 3 points of clean feedback). Revised paper: App. G.7, Table 18.

  • Less informative feedback. In shopping, it makes no difference whether the user announces a change. In the embodied domain, when the user only names the wanted item or only rejects the action, the original method loses accuracy. The cause is two details of the memory update, not the framework, and two small changes fix them. First, the agent now sees the scenario and its own action when it writes a note: Phase 2 with feedback that only names the wanted item recovers from 60.1 to 70.4 (standard feedback: 76.8). Second, a consolidation step rewrites stored notes that a new note contradicts: Phase 4 rises from 56.7 to 64.9 for PAHF, level with Post-action only. Revised paper: App. G.7, Tables 18–20 (details in our response to omsy, W1).

Theory. Prop. 1 and its proof stand, and we make the statement more precise (vPaM, mF6j). It is a worst-case bound, as lower bounds are, but not a corner case: it holds for at least half of all possible switch times. "Unambiguous" was meant as the agent's own confidence given its memory. The paper's definition did not say this clearly, so we now define γ explicitly as instruction ambiguity, separate from the uncertainty about whether preferences have drifted, and the proof is unchanged. The new Prop. 1′ extends the result to every pre-action policy, including re-confirmation: without post-action feedback, a policy that asks B re-confirmation questions still makes at least KT / (16(2B + K)) − K/4 mistakes in expectation, so if a question costs c, the total cost is Ω(√(cKT)). With post-action feedback, at most K mistakes and K corrections are needed. Revised paper: Sections 3.1 and 3.3 (the definition), and the new subsection "Scope of Proposition 1 and Re-confirmation Policies" at the end of App. A (the scope of Prop. 1, and Prop. 1′ with its full proof).

Where the changes are in the revised paper:

ChangePlace in the revised paperAsked by
Three seeds and significance tests, both domainsApp. G.2, Table 12 (protocol in App. G.1)vPaM W3, Q2
Shopping: always-D baseline, how often each method chooses D, balanced Phases 3 and 4App. G.3, Tables 13–14omsy Q1
Pre-action only with re-confirmation, with questions per taskApp. G.4, Table 15omsy Q2, mF6j W1
Gradual driftApp. G.5, Table 16vPaM Q3, omsy W1
Partial drift at different times, changed and unchanged preferencesApp. G.6, Table 17mF6j W2, omsy W1
Missing, wrong, late and less informative feedback; context-aware update; consolidation stepApp. G.7, Tables 18–20vPaM Q3, omsy W1, mF6j W3
"Ambiguous" defined relative to the agent's memory (instruction ambiguity)Sections 3.1 and 3.3; notation in App. AmF6j W4
Scope of Prop. 1; the bound holds for half of all switch times; relaxed A2App. A, new subsection "Scope of Proposition 1 and Re-confirmation Policies"vPaM W1, Q1; mF6j W4
Prop. 1′ with full proof; the necessity claim in its precise form (Remark 2)Same subsection of App. A; Remark 1 in Section 3.3 points to itmF6j W4, omsy Q2
Limitations: simulated users, less informative feedbackApp. BvPaM W2, omsy W1

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.