acceptodds
Under review as a conference paper at ICLR 2027

MemCtrl: Do users have control over their agent's memory?

17 views

Abstract

As everyday tasks are increasingly delegated to AI agents, large amounts of personal information are disclosed and stored in memory to personalize future assistance. This raises a fundamental question: **do users actually have control over their agent's memory?** Prior work largely focuses on improving memory retention and personalization, implicitly assuming that everything should be memorized whenever capacity permits. We instead start from the premise that users may want some information forgotten while keeping other information in memory. We observe such behavior in the wild: in real-world user–ChatGPT conversations, users explicitly request that specific information be removed or disregarded, which we term *Memory Control Instructions*. Motivated by this observation, we introduce **MemCtrl**, a benchmark for evaluating memory control across three common settings: Incognito (do not store newly disclosed information), Forget (remove previously stored information), and Withhold (temporarily disregard stored information). We evaluate six state-of-the-art memory systems and two production chatbots (ChatGPT and Claude). No system reliably follows all three types of instructions: high utility consistently comes at the cost of low control, and even ChatGPT and Claude surface restricted information in 43.6%–62.4% of controlled cases on average. Through thorough analysis and targeted interventions, we trace these violations to failure modes at four levels and distill our findings into design guidelines for faithful memory control.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Agent Memory, Memory Control, Instruction following, Privacy

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QVszOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MemCtrl studies whether assistants obey user instructions not to store newly disclosed information (Incognito), to forget stored information (Forget), or to suppress it temporarily and restore access after release (Withhold). The benchmark uses synthetic multi-session conversations, counterfactually filtered recall and application MCQs, four scope levels, and separate measures of allowed-information utility and control. Six memory systems and two production chatbots are evaluated. The reported results show limited selective control, especially for Withhold, and operation traces suggest failures in executing control actions and identifying affected records.

Critical strengths
  1. The paper studies an important user-facing problem: users should be able to express memory-control requests through ordinary language without locating or manipulating individual memory records.

  2. The benchmark covers non-storage, permanent forgetting, and reversible withholding while measuring retained utility. Its scope, phase-wise, and operation-level analyses provide useful diagnostics beyond aggregate accuracy.

Critical weaknesses
  1. The evaluation relies on each system’s default pipeline and provides no standardized layer for translating natural-language control requests into structured memory operations. For configurations without native support, some failures are therefore predictable by construction. Although the paper provides useful failure analyses and an in-context baseline, it lacks a positive persistent-memory reference that implements this control. It is therefore unclear whether the observed failures reflect limitations of the underlying memory mechanisms or simply a missing control interface.

  2. Two measurement issues limit interpretation. Application probes can conflate fact availability with recommendation preference; although the paper reports counterfactual filtering and human checks, it does not report application-specific ambiguity rates or agreement. Moreover, the Withhold reference U(1−U)U(1-U) assumes independent hold/release outcomes with identical availability and is not validated on matched no-control runs, so the resulting zero/negative-JJ interpretations are not justified.

Questions for authors
  1. Can the authors add a standardized natural-language-to-operation adapter or native-API baseline and a positive persistent-memory reference system? How much failure remains once the requested control is explicitly supported?

  2. Can the authors report a no-control run on the same Withhold facts at both active and release checkpoints, along with raw UU, raw CC, and uncertainty, to test whether the joint blind rate matches U(1−U)U(1-U)?

  3. Can the authors report application-specific results from the existing human checks and assess a stratified sample for ambiguity and annotator agreement? How do the results change after removing items with multiple reasonable responses?

  4. Can the authors recompute Table 2 using fact-level L1 Incognito compliance and provenance? Does a write count as noncompliant only when it stores the restricted fact, and how do the conclusions change?

7 comments
AuthorsOpenReview ↗

(1/4) To Reviewer QVsz

Thank you for all the constructive and insightful feedback, which greatly helped us improve the quality of our paper. Our point-by-point responses follow.

W1 & Q1: The evaluation uses each system's default pipeline, with no standardized NL-to-operation adapter and no positive persistent-memory reference. How much failure remains once the requested control is explicitly supported?

Thank you for the good question. We answer it in two points.

Control is natively supported, just not reliable. Every system evaluated in this paper natively supports memory controls. As mentioned in Line 300-303, all systems explicitly support adding/updating/deleting memories and are equipped with the required tools, and each one uses an LLM to interpret user conversations and decide which operation to run. Their own prompts cover user-directed changes; mem0's memory-managing prompt, for example, says "if the direction is to delete the memory, then you have to delete it." A user's instruction is therefore an input these systems are designed to handle, and the LLM manager, the first stop of a memory system, already plays the role of a natural-language-to-operation adapter.

Memory-control failure is therefore more than a missing adapter. To illustrate this, in Appendix G.2 (Table 16), we added a standardized natural-language-to-operation adapter as requested on mem0 for Withhold, the most challenging setting. It’s an LLM layer that identifies whether a user turn contains a control instruction and its type, then maps it to the set of proper operations, with explicit guidance for the memory manager (prompt in Appendix G.2). The adapter identifies instruction types with 99.87% accuracy and substantially increases the use of the correct operation (compliance in Table 2): from 0.21 to 0.72 when the restriction is issued, and from 0.50 to 0.93 at release. Yet end-to-end control stays low (0.03 →\rightarrow 0.04). This is exactly what failure point 2 in our failure analysis points to: the operation does not reach the memory records that store the controlled facts (effective coverage is only 0.15 and 0.08). As Appendix G.3 further shows, once the right operation is chosen, the system still has to resolve which stored entries the instruction covers and edit them at the right granularity (e.g., when only part of an entry should be controlled).

AuthorsOpenReview ↗

(2/4) To Reviewer QVsz

W2(a) & Q3: how we ensure the application probes behavior reflecting the real usage of a fact

Thanks for this great question. We fully understand the reviewer’s concern, and this is indeed a central consideration in our benchmark construction. First, we would like to make it clear that multiple recommendations can be reasonable for an application probe; but what matters in memory control is whether the target fact changes which response models prefer.

There are two potential ambiguities: (1) a model selects the with_fact choice without actually using the fact, and (2) a model selects the without_fact choice but still remembers the fact. Our benchmark uses two designs to reduce both:

Using plausible_distractor option to reduce false positives in case (1): each MCQ has three candidate choices: with_fact, without_fact, and a plausible_distractor. The plausible_distractor is similar to the with_fact choice but relies on an incorrect personal fact. For example, a with_fact option may explicitly mention the seafood allergy, whereas the plausible_distractor mentions a different, fabricated peanut allergy. In Prompt 12 (Line 2120-2123), a distractor is generated to "rest on a made-up fact of the same kind, one you never said but that could plausibly be true of you." Its role is to distinguish between a model selecting a recommendation based on the actual fact and one selecting a merely plausible recommendation.

Use counterfactual tests to filter out both (1) and (2). For both recall and application probes, we conduct a leave-one-fact-out test. We provide a backend model (gpt-5.4-mini) with the full conversation and then with the same conversation excluding only the target fact. We retain only questions for which the model selects with_fact in the first condition and without_fact in the second, as described in Section 3.2 and Appendix D.2. During the benchmark construction, there are many raw questions where the model does not recommend the with_fact choice even with the target fact, as expected by the reviewer, but all these questions were already filtered out. We deliberately use a weak model for this filtering so that the retained questions have a sufficiently clear fact-dependent discrepancy without relying on strong reasoning capability.

Note that this is a leave-one-fact-out test: the only variable changed between the two conditions is the presence of the target fact. Therefore, memory use of the fact is the only explanation for the observed change in answer. To further validate that the observed preference is not specific to the model used for filtering, we expand the leave-one-fact-out test to 8 models from 6 different families, none of which was used for filtering:

ModelFact present: selects with_factFact removed: selects without_fact
Claude Opus 5.594.2%95.0%
Claude Sonnet 5.596.2%91.5%
Gemini 4 Argon96.9%95.8%
GPT-5.6 Sol92.7%96.9%
GPT-5.6 Luna92.7%93.4%
DeepSeek-V4-Pro95.8%88.8%
Qwen3.8100.0%89.2%
GLM-5.398.5%87.7%
Average95.9%92.3%

The results show that the probability of recommending without_fact when the fact is present is low (and vice versa), and deviations from the expected answer are rare. With the fact present, only 4.1% of answers on average (at most 7.3% for any model) are not with_fact; with the fact removed, only 7.7% on average (at most 12.3%) are not without_fact. Since these models also commonly serve as the decision-making backends of existing memory systems, their agreement under full context is a direct measure of whether a probe is ambiguous.

AuthorsOpenReview ↗

(3/4) To Reviewer QVsz

Q3: Application-specific results from the existing human checks; ambiguity and annotator agreement; results after removing ambiguous items.

What the existing human checks cover. The spot-check in Appendix D.3 targets option quality: whether the distractor is close enough to the with_fact option, and whether any option is obviously wrong. It does not judge whether the with_fact and without_fact responses are each appropriate, because annotators tend to judge a recommendation by their own preferences and not those of the synthetic persona. For application probes, human effort also goes into revision: a probe that fails the counterfactual test is manually revised and re-tested, and dropped if it still fails (Line 255-260, Section 3.2 and Appendix D.2).

Ambiguity, agreement, and results after removing ambiguous items. For the reason above, we measure ambiguity and agreement with the 8-model counterfactual test reported in the table above. Treating any probe on which at least one model disagrees as ambiguous and evaluating only on the remaining ones, average application JJ (over the six memory systems and two production chatbots) changes by at most 0.056 (on Forget):

IncognitoForgetWithhold
All probes0.2960.1540.029
Ambiguous probes removed0.3170.2100.019

Rationale analysis as an additional ambiguity check. In our evaluation, we always ask for a rationale along with the MCQ answer (Prompt 18, Appendix H.3). This lets us verify that when a model selects with_fact, it does mention the fact in its rationale. We added a rationale analysis in Appendix E.3: among application probes where the system selects the with_fact option, 95.2% on average across the six systems explicitly mention the target fact (90.0%–96.0% across control and utility types). This is direct evidence that when a system selects the with_fact option, it makes that choice because of the fact.

Reviewer_wPvVOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds MemCtrl, a benchmark of memory control instructions in three settings: Incognito (do not store new information), Forget (remove stored information) and Withhold (temporarily disregard stored information). It evaluates six open-source memory systems (mem0, A-Mem, MemOS, Letta, and two LangMem variants) and two production chatbots (ChatGPT and Claude, through browser automation), finds that no system follows all three instruction types reliably, and analyzes where the failures come from.

Critical strengths
  1. The benchmark points out a real problem: users have little control over what an agent's memory keeps or forgets.
  2. Some of the findings are interesting, and the topic overall is interesting.
Critical weaknesses
  1. The evaluation of production chatbots is hard to trust and hard to reproduce. ChatGPT and Claude are tested by browser automation that logs into the web products and types messages (Section 4.1, Appendix). I doubt this procedure can reliably measure these systems, and no packaged evaluation toolkit is provided that would let others quickly run the same tests on ChatGPT or Claude. For a benchmark paper, how the evaluation is run and whether it can be guaranteed is the key part; such a toolkit would be a real plus. Without it, the contribution is mainly a set of test questions.
  2. The paper reads mainly as an experiment report. It measures failures and offers design guidelines, but does not propose a method that solves the problem. I am not convinced the contribution reaches the ICLR bar.
  3. Presentation quality is below the ICLR bar. The figures are raster images (their text cannot be selected) rather than vector graphics.
Questions for authors
  1. The reproducibility statement offers the benchmark data, evaluation logs and code upon reviewers' request. Would you consider releasing them, including the browser-automation harness for ChatGPT and Claude, as a packaged toolkit that others can run directly?
  2. How reliable is testing ChatGPT and Claude by simulating logged-in web sessions? How do you make sure the measurements are valid?
  3. A suggestion, not a weakness: the memory scoping failure (Section 5, Figure 6) can also be seen as an in-context issue. If the information is still in the conversation context, whether the model can disregard it when asked depends on the model's own in-context ability. Since there is no established definition of what memory system design covers, treating this as part of it is reasonable; it might still help to discuss this view. Relatedly, recent work on Context Language Models lets the model edit its own context through tool calls, so a request to forget can actually remove the content from context. It is very recent concurrent work, and I do not penalize the authors for not discussing it.
6 comments
AuthorsOpenReview ↗

(1/3) To Reviewer wPvV

Thank you for your review and for the helpful pointer to Context Language Models. Our responses to each of your questions follow.

W1 & Q1 (Toolkit release): The evaluation of production chatbots is hard to trust and hard to reproduce. No packaged toolkit is provided for running the evaluation on ChatGPT and Claude. Would the authors release them?

Q2. How reliable is testing ChatGPT and Claude by simulating logged-in web sessions? How do you make sure the measurements are valid?

We have updated the anonymized link in the revision (Line 560) to include the automation toolkit, together with the benchmark, code, evaluation interface, and system implementations. In short, for each trial, the harness

  1. clears/resets the account's memory, chat history, library, and artifacts, so that every trial starts from a blank account;
  2. plays the user's role: it types each query into the chat box, waits for the full reply, then sends the next query, and opens a new session whenever a session ends;
  3. checks that all previous sessions have been indexed in memory;
  4. sends each test probe in a new session and saves the full response to the log.

We also include step-by-step screenshots in the toolkit README for reference.

Re reliability (Q2): Our evaluation process reproduces what a real user would do (it’s not a simulation); the systems under test are the deployed products themselves, including their memory features. To our knowledge, this is the most realistic way to evaluate the memory of a commercial chatbot, since these memory features are not available through the API. It’s possible that memory behavior differs for long-lived accounts with extensive history, but using a blank user with shorter interaction should actually make memory control simpler, as there are fewer conversations in the memory. Therefore, we expect memory control to potentially be even lower in the wild.

We are happy to try any alternative evaluation strategy for commercial chatbots that the reviewer suggests.

AuthorsOpenReview ↗

(2/3) To Reviewer wPvV

W2: The paper reads mainly as an experiment report: it measures failures and offers guidelines but does not propose a method. I am not convinced the contribution reaches the ICLR bar.

We respectfully disagree. The paper identifies a new problem, quantifies the severity of the problem with a systematically designed benchmark, explains why current systems fail at it, and proposes targeted solutions at each failed step. Its contributions go beyond experiment report, and we believe they are more useful to the field than proposing another memory system:

A previously neglected problem.

We show that existing memory systems, including commercial products, largely fail at memory control. We built a systematic benchmark for diagnosis. It covers (a) three mainstream control types and (b) four control scopes of increasing difficulty: same-session to cross-session, single-fact to multi-fact, and (c) recall to application. Each is built with a leave-one-fact-out test, so that the presence or absence of the fact leads to different choices. Scoring therefore is reliable and reproducible, and does not rely on any LLM judge. This design lets us tell failures apart, e.g., recall-level vs. application-level, same-session vs. cross-session (i.e., failure to rack instructions over time), and a single fact vs. a broader semantic scope (i.e., reliance on verbatim mentions).

Root causes and targeted fixes.

We locate failure in execution: (i) the system stores the instruction as a separate entry instead of executing it, and later fails to retrieve that entry together with the controlled fact (Failure point 1; Appendix G.2); (ii) it executes the instruction but misses some of the related entries (Failure point 2; Appendix G.3); (iii) it finds the entries but cannot control them selectively when only part of an entry is affected (Appendix G.3). We then test a targeted fix at each step. An instruction-to-operation adapter raises mem0's compliance on Withhold from 0.21 to 0.72 (Table 16); an LLM-based matcher raises mem0's Forget control from 30% to 98%, at a cost in utility (Figure 12); and a stronger answerer raises A-Mem's Forget control from 40% to 64% (Figure 11). Each fix repairs its own step, but none restores control end to end.

Why not a single new method.

Memory systems use different interface designs and fail for different reasons, so no single method fixes all of them. Production chatbots are black boxes whose memory design changes every few weeks, and outside researchers cannot fix them at all. What we can do is raise the alarm: control is neglected in memory system design, and recent progress, which optimizes for retention alone, makes memory even harder to control. We also give designers concrete levers at each step where control fails.

Identifying a widespread failure and tracing it to its causes is a contribution in its own right, and a prerequisite for any method that aims to solve it. We hope it leads future memory systems to treat control as a design requirement from the start.

W3. Presentation quality is below the ICLR bar. The figures are raster images (their text cannot be selected) rather than vector graphics.

We have updated all figures with vector graphics in the revision. If there are other presentation issues beyond the figure format, we would appreciate it if the reviewer could point them out, and we will fix them.

AuthorsOpenReview ↗

(3/3) To Reviewer wPvV

Q3: The memory scoping failure in chatbots can also be seen as an in-context issue. If the information is still in the conversation context, whether the model can disregard it when asked depends on the model's own in-context ability. Since there is no established definition of what memory system design covers, treating this as part of it is reasonable; it might still help to discuss this view. Relatedly, recent work on Context Language Models lets the model edit its own context through tool calls.

Thank you for raising this good question; it was in fact part of our design considerations, and we have added a more explicit discussion to Section 5 and Appendix G.4 of the revision.

To separate memory control from in-context reasoning, the chatbot evaluation is cross-session (Section 4.1): every probe is issued in a new session, so the controlled fact is never in the same-session context as the probe. To use it, the chatbot has to retrieve it from saved memories or past conversations through its memory tools (e.g., personal_context). The memory-scoping failures are therefore about which sources the control instruction is applied to: a chatbot's memory draws on several sources (saved memories, past conversations, artifacts, libraries), and the chatbot does not honor the instruction equally across them. Content retrieved from past conversations is different from content that sits in the current context, so using it despite the instruction is a memory-control failure.

We have also isolated the model's own in-context ability. The in-context baseline (Table 1) shows that the backend model does disregard a fact on request when the full conversation is in its context, and the memory-scoping analysis is conducted only on probes for which the backend model follows the instruction in context but the chatbot violates it (Lines 497-500, Section 5 and Appendix G.4).

Added experiment: Context Language Models (Zero-shot). Thanks for the wonderful suggestion. CLM is a very good and strong baseline, although it was released after the submission deadline. We have added it as a second baseline next to the in-context one, with the same backbone (GPT-5.6 Sol). See Tables 1 and 9 for the full results alongside the other memory systems.

Incognito (J)Forget (J)Withhold (J)Utility drop
Recall
In-context0.8830.9030.7769.0%
CLM0.8270.8150.61316.4%
Application
In-context0.7690.8340.7284.7%
CLM0.7840.8020.7199.4%

Both baselines perform better than every evaluated memory system. Note that this is partly by design: to separate memory control from long-context failures (e.g., lost-in-the-middle), we keep the conversations short enough (15–54 user turns) to fit well within the context window, so the in-context setting is deliberately favorable.

However, the ability to edit its context does not give CLM better control: its JJ is lower than the in-context baseline on recall and about the same on application, but with much more severe harm on utility. A general tool for managing context is therefore not enough for selective memory control. These results make us more confident that memory control is hard and has to be designed explicitly; it does not come for free when context management is left to the agent.

Reviewer_uHt8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MemCtrl, a benchmark for evaluating whether LLM agents follow users' instructions about their memory. It covers three types of memory control: Incognito (not storing new information), Forget (removing previously stored information), and Withhold (temporarily not using certain information and restoring access later).

MemCtrl includes 40 user profiles and 834 evaluation questions, with both recall and application probes. The authors evaluate six agent memory systems and two production chatbots, ChatGPT and Claude. They also examine where memory control fails in the underlying memory operations. The results show that current systems struggle to respect users' memory preferences while keeping useful information available.

Critical strengths

1.The paper looks at something existing memory benchmarks often overlook: whether users can control what their agents remember, forget, or temporarily stop using. I like the distinction between Incognito, Forget, and Withhold. Including both recall and application probes also makes the evaluation more informative.

2.Rather than just reporting benchmark scores, the authors trace memory operations to understand where things go wrong. They distinguish failures to take the right memory action from failures to cover all relevant memory entries. The experiments on ChatGPT and Claude, along with the additional analysis of memory matching and control scope, make the findings more useful.

Critical weaknesses

1.Application probes may rely too much on predefined answers.

The paper uses three-option multiple-choice questions to check whether a model uses certain memories in downstream tasks. This works reasonably well for factual recall, but it is less straightforward for recommendations and decisions, where more than one answer can be reasonable.

For example, in the headphone recommendation task in Appendix D.2.1, knowing that a user has a relatively high income does not necessarily mean the model should recommend the most expensive headphones. A different recommendation could still be reasonable even if the model used that information. I would like to see more evidence that the application probes reliably capture actual memory use.

2.The baseline assumption for the Withhold metric needs more justification.

The paper uses the J metric to measure memory control relative to an instruction-blind system. For Withhold, the baseline is defined as U(1−U), but this seems to rely on an assumption about the relationship between the withholding and release stages.

In persistent memory systems, behavior across these two stages may be strongly correlated, so it is not clear whether this is the right baseline. An empirical no-control baseline or a sensitivity analysis would help clarify how to interpret the Withhold results.

Questions for authors

1.Have the authors considered using open-ended responses or human evaluation to validate the application probes? In recommendation tasks, how can we distinguish a model that ignored the memory from one that used it but made a different, reasonable choice?

2.Could the authors explain the choice of U(1−U) as the instruction-blind baseline for Withhold? Does it still hold if behavior during withholding and after release is correlated? An empirical no-control baseline would be helpful here.

5 comments
AuthorsOpenReview ↗

(1/2) To Reviewer uHt8

Thank you for the encouraging and constructive review. They greatly help improve the quality of our paper. We respond to each point below.

W1 & Q1: evidence on application probes reliably captures actual memory use.

Thanks for this great question. We fully understand the reviewer’s concern, and this is in fact a central consideration in our benchmark construction. First, we would like to make it clear that application probes do not necessarily have a unique or most reasonable answer. Multiple recommendations can be reasonable; what matters in memory control is whether the target fact changes which response models prefer.

There are two potential ambiguities: (1) a model selects the with_fact choice without actually using the fact, and (2) a model selects the without_fact choice but still remembers the fact. Our benchmark uses two designs to reduce both:

Using plausible_distractor option to reduce false positives in case (1): each MCQ has three possible options: with_fact, without_fact, and a plausible_distractor. The plausible_distractor is similar to the with_fact choice but relies on an incorrect personal fact. For example, in the headphone recommendation task mentioned by the reviewer, with_fact explicitly mentions the user's actual salary, whereas plausible_distractor mentions a different, fabricated salary. In the prompt used for generating the application probe and its possible answers (Prompt 12, Line 2120-2123), we explicitly note that a distractor is generated to "rest on a made-up fact of the same kind, one you never said but that could plausibly be true of you." Its role is to distinguish between a model selecting a recommendation based on the actual fact and one selecting a merely plausible recommendation.

Use counterfactual tests to filter out both (1) and (2). For both recall and application probes, we conduct a leave-one-fact-out test. We provide a backend model (gpt-5.4-mini) with the full conversation and then with the same conversation excluding only the target fact. We retain only questions for which the model selects with_fact in the first condition and without_fact in the second, as described in Section 3.2 and Appendix D.2. There were indeed many questions where the model does not recommend the with_fact choice even with the target fact, but these were all filtered out. We deliberately use a weak model for this filtering so that the retained questions have a sufficiently clear fact-dependent discrepancy without relying on strong reasoning capability.

Note that this is a leave-one-fact-out test—the only variable changed between the two conditions is the presence of the target fact. Therefore, memory use of the fact is the only explanation for the observed change in answer. To further validate that the observed preference is not specific to the model used for filtering or due to response randomness, we expand the leave-one-fact-out tests to 8 models from 6 different families:

ModelFact present: selects with_factFact removed: selects without_fact
Claude Opus 5.594.2%95.0%
Claude Sonnet 5.596.2%91.5%
Gemini 4 Argon96.9%95.8%
GPT-5.6 Sol92.7%96.9%
GPT-5.6 Luna92.7%93.4%
DeepSeek-V4-Pro95.8%88.8%
Qwen3.8100.0%89.2%
GLM-5.398.5%87.7%
Average95.9%92.3%

The results show that the probability of recommending without_fact when the fact is present is low (and vice versa), and deviations from the expected answer are rare. With the fact present, only 4.1% of answers on average (at most 7.3% for any model) are not with_fact; with the fact removed, only 7.7% on average (at most 12.3%) are not without_fact. Even if we drop every probe on which any of the eight models deviates in either condition and re-score the systems, the average application JJ moves by at most 0.056 (from 0.296 / 0.154 / 0.029 to 0.317 / 0.210 / 0.019 for Incognito / Forget / Withhold). Our conclusions therefore do not depend on the few probes where models disagree.

Importantly, this result is more direct than human judgments. Since these models also commonly serve as the decision-making backends of existing memory systems, their behavior under full context is a straightforward baseline. Our rationale analysis below further shows that when a model selects the with_fact choice, it always explicitly mentions the fact.

AuthorsOpenReview ↗

(2/2) To Reviewer uHt8

Why MCQs rather than open-ended evaluation?

Q1: Have the authors considered using open-ended responses or human evaluation to validate the application probes? In recommendation tasks, how can we distinguish a model that ignored the memory from one that used it but made a different, reasonable choice?

Our first version did use open-ended responses scored by an LLM judge. We moved to MCQs because that setup was not reliable enough.

Issues with open-ended responses. In free-form recommendations, what the system recommends and which facts it relies on vary widely across runs. When a response did not use the target fact, we could not tell whether the question was poor (e.g., only weakly related to the fact) or the system had failed to use it. Our human validation of that version also showed that the LLM judge was reliable for only one thing: whether a response cites a given fact as a reason for its recommendation.

Issues with Human evaluation. Annotators tend to judge a recommendation by their own preferences rather than those of the synthetic persona, so we do not treat human preference as ground truth. Instead, we validate each question with a counterfactual check across model families: given the fact, backend models should pick the with_fact option; without it, the without_fact option. If models disagree, either the option is not clearly better or the question is not relevant enough to the fact, and we revise or drop the question. So "used the fact but reasonably chose otherwise" is rare by construction.

Rationale analysis as an additional check While we evaluate with MCQs, we still ask for a rationale along with the answer (Prompt 18, Appendix H.3). This lets us verify that when a model selects with_fact, it does mention the fact in its rationale (one thing an LLM judge can assess reliably). We added a rationale analysis in Appendix E.3: among application probes where the system selects the with_fact option, 95.2% on average across the six systems explicitly mention the target fact (90.0%–96.0% across control and utility types). This is direct evidence that when a system selects the with_fact option, it makes that choice because of the fact.

We also added a more detailed discussion on the leave-one-fact-out filtering in the revision (Section 3.2) and added the discussion of cross-model validation and ambiguity checks (Appendix D.2.1) and the rationale analysis (Appendix E.3).

W2 & Q2: The Withhold baseline U(1−U)U(1-U) seems to assume independence between the withholding and release stages. Does it still hold if the two stages are correlated? An empirical no-control baseline would help.

Thanks for raising this point. Our original reference in Eq. (3) was meant to be a random baseline that surfaces a fact with a fixed probability pp. Its two-stage performance is p(1−p)p(1-p) because its decisions in the two stages are independent, and p=Up=U in one stage by the definition of utility. The purpose was to rule out trivial solutions such as always or never surfacing the fact. We therefore did not assume that the two stages are independent for any evaluated system.

We agree with the reviewer's suggestion that reporting a no-control baseline score for each system as a reference is more direct and can capture the correlation between the two stages. Importantly, we can measure it empirically without any estimation. We have updated the metric in revision (Line 201-222) to measure the effective control minus the utility drop (both relative to the no-control baseline): J=(C−Cblind)−(Ublind−U)J = (C - C_{\text{blind}}) - (U_{\text{blind}} - U), where "blind" here denotes the no-control baseline. All JJ scores are updated accordingly in the revision (Sections 3.1 and 4.2, with utility and control reported separately in Appendix F.1). This does not change any of our previous conclusions.

Dear Reviewer uHt8,

Thank you again for your constructive feedback and insightful suggestions. To make the revised manuscript easier to read, we summarize below the main changes related to your comments (details are in our response above):

  1. Cross-model validation of application probes (Appendix D.2.1, Table 4, p. 18-19), across 8 models from 6 families, to show that models commonly agree on with_fact as the best option when the fact is there and on without_fact when it is not.

  2. Rationale analysis (Appendix E.3, Table 8, p. 23-24), to show that when a system selects the with_fact option, its rationale explicitly grounds the choice in the target fact in 95.2% of cases.

  3. Empirical no-control baseline for JJ (Section 3.1, Eq. 3, p. 5; Appendix F.1, Table 9, p. 24-25), as you suggested, to account for the correlation between the withholding and release stages. All results are updated, and the conclusions do not change.

We hope this summary is helpful, and we are happy to discuss anything that remains unclear.

Best regards,

Authors of Submission 14406

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.