- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces PAHF, which continuously updates a memory of user preferences through pre-action clarification and post-action correction. In object-selection and product-selection tasks, the authors separately evaluate initial learning, held-out testing, adaptation after preference changes, and subsequent testing. The experiments compare feedback channels, memory backends, and interaction strategies, and also examine feedback noise and variation in how preferences are expressed.
Critical strengths
-
Separating learning phases from tests without feedback helps establish whether information acquired through interaction is retained and applied to new situations. Evaluations before and after preference changes also provide a clear way to examine adaptation separately.
-
The ablations examine the distinct roles of pre-action clarification and post-action correction, and the comparisons are repeated across multiple memory backends. The appendix results with A-MEM and Mem0 strengthen confidence that the findings extend beyond a simple storage implementation.
-
The analyses of Always-ask, read-only profiles, ambiguous preference expressions, and variation in simulated user language make the experimental study reasonably comprehensive. In particular, comparing response accuracy during interaction with performance in later memory tests helps distinguish the quality of immediate feedback from its contribution to longer-term memory updates.
Critical weaknesses
-
The Pre-action Only baseline generally stops asking questions once it retrieves relevant memory, so its difficulties after preference drift may partly reflect the rule used to trigger clarification. The Always-ask variant in the appendix still includes post-action feedback and therefore does not provide a direct comparison with a purely pre-action strategy that actively rechecks existing memories. A baseline that periodically reconfirms preferences under a comparable interaction budget would help clarify how much the benefit of post-action feedback depends on the current questioning rule.
-
Preference changes are implemented mainly by replacing or resampling entire user profiles at once, while the adaptation phase reuses earlier training scenarios. This is a useful way to control changes in the inputs, but it offers limited evidence about partial preference updates and the retention of unchanged preferences during continuous personalization. Allowing subsets of preferences to change at different times, and testing both changed and unchanged preferences, would make it possible to assess adaptation speed and the risk of incorrectly overwriting existing memories.
-
The noise experiments perturb feedback only in the first round of initial learning, after which interactions return to clean feedback. The ambiguous expressions and language rewrites also preserve the correct preference information. These experiments leave open how the method would behave under persistent feedback errors, omissions, or delays, which could affect both drift detection and memory updates. Applying such perturbations throughout learning and adaptation would provide a more complete assessment of robustness during online use.
-
I would appreciate clarification of how the definition of “unambiguous” relates to the construction in the proof of Proposition 1. In that construction, preferences can change at an unknown time while the observable inputs remain unchanged. A Bayes-optimal decision-maker may still face uncertainty given the actual interaction history, which raises the question of whether the construction satisfies the stated unambiguity condition. Could the authors clarify whether the term refers to the model's subjective confidence or to the identifiability of the true preference, and revisit the assumptions and necessity claim accordingly?
Questions for authors
My questions are included in the weaknesses above.
4 comments
Response to Reviewer mF6j [2/4]
W2. "Preference changes are implemented mainly by replacing or resampling entire user profiles at once … Allowing subsets of preferences to change at different times, and testing both changed and unchanged preferences, would make it possible to assess adaptation speed and the risk of incorrectly overwriting existing memories."
We ran this experiment, and PAHF handles partial changes well: it re-learns changed preferences within about two iterations and largely keeps the unchanged ones. The setup (embodied domain, seed 1): for each user, 25% of the situations change at iteration 1 of Phase 3 (group 1), another 25% change at iteration 3 (group 2), and 50% never change. As in the paper, Phase 3 reuses the Phase 1 scenarios and Phase 4 uses held-out tasks. We score the three groups separately, and for the unchanged situations we also compare Phase 4 with the same tasks in Phase 2. The Phase 3 columns show the success rate per iteration (only the first and the last iteration where it stays flat). The last two rows add a consolidation step that we explain below (Table 17 in the revised paper, App. G.6):
| Group 1 (changes at it. 1), Phase 3 | Group 2 (changes at it. 3), Phase 3: it. 1, 2 → 3 → 4 → 5 | Unchanged, Phase 3 | Phase 4: group 1 / group 2 / unchanged | Unchanged: Phase 4 − Phase 2 | |
|---|---|---|---|---|---|
| No memory | – | – | – | 45.7 / 44.9 / 36.9 | +0.5 |
| Pre-action only | 29.9 → 34.5 | 57.6, 58.7 → 31.9 → 33.0 → 31.9 | 61.6 → 63.0 | 38.0 / 34.3 / 57.4 | +2.1 |
| Post-action only | 14.1 → 73.9 → 83.1 → 86.6 → 89.1 | 81.5, 79.0 → 13.0 → 70.3 → 80.8 | 84.7 → 78.3 | 69.6 / 63.4 / 73.6 | −1.7 |
| PAHF | 18.7 → 74.3 → 79.9 → 84.9 → 89.4 | 84.4, 82.2 → 15.9 → 64.9 → 82.2 | 82.8 → 77.7 | 64.1 / 67.9 / 71.9 | −5.8 |
| Post-action only + consolidation step | 17.6 → 82.7 → 88.0 → 88.4 → 90.5 | 83.0, 76.1 → 14.9 → 76.1 → 83.0 | 81.7 → 79.1 | 73.2 / 72.1 / 72.8 | −3.5 |
| PAHF + consolidation step | 19.7 → 84.2 → 83.5 → 89.4 → 89.4 | 81.9, 81.5 → 17.0 → 73.2 → 85.5 | 83.4 → 79.4 | 70.7 / 70.6 / 74.4 | −0.6 |
Adaptation speed. With post-action feedback, the agent re-learns each group within about two iterations after it changes (group 1, PAHF: from 18.7 to 74.3 after one iteration, and 89.4 at the end).
Changed vs unchanged preferences. Pre-action only does not re-learn the changed preferences (below 40 in Phase 4), although it keeps the unchanged ones. Post-action feedback is what makes the difference.
Risk of overwriting. Most unchanged preferences are kept. On the unchanged tasks, Phase 4 is 1.7 points below Phase 2 for Post-action only and 5.8 points below for PAHF (the difference between the two methods is not significant, p = 0.10).
A consolidation step does not add to this risk. In our response to omsy (W1) we add a consolidation step that rewrites stored notes when a new note contradicts them. One could expect it to overwrite preferences that did not change, so we reran the partial-drift experiment with it (last two rows). It does not: with the step, Phase 4 accuracy on unchanged preferences does not drop (74.4 for PAHF, against 71.9 without it), and changed preferences are re-learned better (PAHF: 70.5 vs 65.7, p = 0.02) and faster.
We also added gradual drift. During Phase 3, the user follows the new preference with a probability that rises from 0.2 to 1.0, so the user's own feedback switches between old and new (Table 16 in the revised paper, App. G.5). Agents with post-action feedback follow the change as it happens and end about 5 points below the abrupt-drift setting in Phase 4 (PAHF: 62.4). Pre-action only does not adapt (32.5).
W3. "The noise experiments perturb feedback only in the first round of initial learning … These experiments leave open how the method would behave under persistent feedback errors, omissions, or delays, which could affect both drift detection and memory updates."
We added all three. In the embodied domain, PAHF keeps its advantage under each of them, and both methods still recover after drift. Each perturbation is applied to the post-action feedback in every learning iteration of Phases 1 and 3, so it affects both the initial learning and the adaptation after drift:
- Omissions: the user gives no feedback 30% of the time.
- Errors: 20% of the feedback is wrong. The user approves a wrong action, or complains about a right action and names another option. Unlike the rewrites in the paper, this feedback carries wrong information about the preference.
- Delays: the feedback reaches the agent one task later.
Response to Reviewer mF6j [3/4]
Embodied domain, seed 1, PAHF / Post-action only (Table 18 in the revised paper, App. G.7):
| Phase 1 first pass SR | Phase 1 ACPE ↓ | Phase 2 SR | Phase 3 ACPE ↓ | Phase 4 SR | |
|---|---|---|---|---|---|
| Clean feedback | 50.8 / 36.3 | 24.3 / 28.2 | 76.8 / 77.4 | 36.4 / 37.4 | 68.2 / 67.8 |
| 30% omitted | 50.6 / 34.4 | 26.4 / 31.5 | 76.8 / 78.6 | 41.0 / 42.0 | 66.7 / 68.8 |
| 20% inconsistent | 50.0 / 36.0 | 30.9 / 34.2 | 70.7 / 70.8 | 40.5 / 41.9 | 65.5 / 65.6 |
| Delayed by one task | 51.4 / 35.7 | 24.6 / 27.7 | 77.3 / 75.5 | 36.8 / 36.8 | 69.8 / 68.8 |
In every condition, PAHF keeps its cold-start advantage over Post-action only (at least +14 points in the first pass of Phase 1, p < 10⁻⁴). This is expected, because the answers to pre-action questions do not depend on post-action feedback. Omissions slow learning down but leave Phase 2 unchanged. Errors slow learning down more and cost about 6 points in Phase 2. Delays have no measurable effect. Drift detection and memory updates keep working: after drift, both methods still recover, Phase 4 stays within 3 points of clean feedback, and the two methods are not significantly different. Revised paper: Table 18 and the paragraph "Persistent omissions, errors, and delays" in App. G.7.
We also tested feedback that carries less information: the user never says that a preference changed, or only rejects the action. When the user only rejects, PAHF's advantage over Post-action only is much larger (13.5 points in Phase 2). The details, and two small changes to the memory update that these tests led to, are in our response to omsy (W1) and in App. G.7 (Tables 18–20).
W4. "I would appreciate clarification of how the definition of 'unambiguous' relates to the construction in the proof of Proposition 1. … Could the authors clarify whether the term refers to the model's subjective confidence or to the identifiability of the true preference, and revisit the assumptions and necessity claim accordingly?"
We meant the agent's subjective confidence: what the agent believes given its memory. The reviewer is right that the definition as written does not say this. It speaks of the posterior conditioned on the transcript and of the Bayes-optimal error. If that posterior is computed with knowledge of the proof's prior over the switch time, then P(τ ≤ t | transcript) = t/T, the error in round t is min(t/T, 1 − t/T), which is at least ε₀ for most t, and the rounds are not unambiguous.
The fix is in the definition, not in the result. The prior over the switch time is the environment's randomization, the standard device for proving a worst-case bound. It is not knowledge that the agent has. Once γ is defined as instruction ambiguity (below), the proof of Prop. 1 goes through unchanged. An agent that does reason about possible drift is a different agent. It is outside the scope of Prop. 1, and we now cover it with Prop. 1′: it can reduce its mistakes only by asking the user again and again. So the paper's conclusion holds for both kinds of agent: without post-action feedback, the agent pays for drift either in mistakes (Prop. 1) or in questions (Prop. 1′). The revised paper makes this explicit:
(i) We define γ as instruction ambiguity: the uncertainty about what the instruction refers to, if the agent's current memory is taken as the user's true preference. This is the uncertainty that a clarifying question removes. Whether the memory itself is outdated is a different kind of uncertainty (drift uncertainty). The definition is changed in Sections 3.1 and 3.3 and in App. A.
(ii) We state Prop. 1 for policies that ask pre-action questions only when the instruction is ambiguous. Prop. 1 now reads "unambiguous given the agent's memory", and a new subsection at the end of App. A states its scope.
(iii) We add Prop. 1′ for every pre-action policy, including policies that ask about drift. It is in the same subsection, with a full proof, and Remark 1 in Section 3.3 points to it.
(iv) We state the necessity claim for agents that ask only when their memory does not already answer the request, and we describe re-confirmation as an alternative that trades errors for interruptions (Remark 2 in App. A; App. G.4).
Response to Reviewer mF6j [4/4]
Prop. 1′. Without post-action feedback, any (possibly adaptive, randomized) policy facing K switches that asks B re-confirmation questions in expectation makes E[mistakes] ≥ KT / (16(2B + K)) − K/4. If a question costs c, E[mistakes] + c·B ≥ √(cKT/8) − (2c + 1)K/4, and keeping the mistakes at O(K) takes Ω(T) questions. With post-action feedback, PAHF makes at most K post-switch mistakes using at most K corrections.
Proof sketch. Split the horizon into K blocks of length T′ = T/K. Each block contains one switch at a uniformly random time, independently of the other blocks. Until the user first answers "changed", the only thing the agent knows is that nothing had changed up to its last question. So, as long as it keeps hearing "no change", its questions follow a fixed schedule. Let n be the number of questions that this schedule places in the first half of a block. With probability ½ the switch falls in the second half, and then all n questions are asked, so E[questions] ≥ n/2. The n questions split the first half into n + 1 segments of lengths ℓ_j, which sum to T′/2. Inside a segment the agent learns nothing until its next question, so by the argument of Prop. 1 the switch times in that segment cause at least (ℓ_j² − 1)/4 mistakes in total. Therefore E[mistakes] ≥ Σ_j (ℓ_j² − 1)/(4T′) ≥ T′/(16(n + 1)) − 1/4 by Cauchy–Schwarz. This gives E[mistakes] ≥ T′/(16(2E[questions] + 1)) − 1/4 per block. Summing over the blocks with Jensen's inequality gives the bound, and minimizing over B gives the √(cKT) form. The full proof is in App. A of the revised paper.
The new re-confirmation experiments (W1; Table 15) show the same trade-off: the fewer questions the agent asks, the more mistakes it makes while re-learning, and it comes close to PAHF's final accuracy only by asking before most of its actions, including when nothing has changed.
Presentation. All of these changes are in blue in the revised paper, and Table 15 now reports questions and interactions per task next to the errors. If the reviewer has other passages in mind, we will revise them as well.
Response to Reviewer mF6j [1/4]
We thank the reviewer for the thoughtful review. The reviewer's questions are contained in the four weaknesses, so we answer W1–W4 one by one, in the order of the review. Every answer names the place in the revised paper where the change is (new text is in blue). SR is the success rate in %, and ACPE is the paper's average cumulative personalization error during a learning phase (lower is better).
W1. "The Pre-action Only baseline generally stops asking questions once it retrieves relevant memory … The Always-ask variant in the appendix still includes post-action feedback … A baseline that periodically reconfirms preferences under a comparable interaction budget would help clarify how much the benefit of post-action feedback depends on the current questioning rule."
We added exactly this baseline, at PAHF's interaction budget and at lower and higher re-confirmation rates. Unlike Always-ask, it never receives post-action feedback, so it shows what pre-action feedback can do on its own. Before the agent acts on a note it retrieved from memory, it asks the user, with probability q, to confirm the note. It writes the answer to memory, revises the note if the user reports a change, and then acts. For the comparable budget, we set q = 0.81, which gives the same number of interactions per task as PAHF (0.77: its questions plus the post-action feedback that led to a memory update). Embodied domain, seed 1 (Table 15 in the revised paper):
Re-confirmation recovers from drift without post-action feedback, so the failure of Pre-action only in Table 1 is tied to its rule for when to ask, as the reviewer suspected. But a pre-action agent has no way to know when a silent change has happened (Prop. 1), so it can recover only by asking even when it is confident, and its accuracy follows the number of questions it asks. At the comparable budget the reviewer asks for, re-confirmation spends the whole budget on questions (0.76 per task, against 0.07 for PAHF). It makes fewer errors while re-learning, because each avoided error is replaced by a question, but it is less accurate than PAHF in both test phases (−3.5 and −4.3 points, p ≤ 0.013). With a smaller budget it falls 9 to 15 points below PAHF in both test phases (all p < 10⁻⁴), and even when it asks before 93% of its actions it does not exceed PAHF.
So, in the embodied domain, the benefit of post-action feedback does not disappear when the questioning rule is improved. Prop. 1′ (under W4) shows why: without post-action feedback, every pre-action policy must pay for drift either in mistakes or in questions. Revised paper: App. G.4 and Table 15 report this baseline, and Remark 2 in App. A states the claim precisely: post-action feedback is necessary for agents that ask only when their memory does not already answer the request, and re-confirmation is an alternative that trades errors for interruptions.