acceptodds
Under review as a conference paper at ICLR 2027

ICM-bench: A Benchmark for Evaluating the In-context Modification Ability of Large Language Models

Abstract

Large language model (LLM) agents continually accumulate conversation histories, yet the contextual state relevant to the current situation is not identical to the full history: some previously valid information may become obsolete, while other information should remain unchanged. We formalize the ability to selectively transform such contextual information during inference, without parameter updates, as In-Context Modification (ICM). Existing benchmarks on unlearning and long-term memory primarily evaluate forgetting, retrieval, or conflict resolution, but do not systematically test whether models can update the current contextual state while preserving unaffected information. To fill this gap, we introduce ICM-Bench, a controlled benchmark for evaluating ICM in multi-turn conversations. It contains over 2.8K multi-turn conversations and 26K queries, covering deletion and update across single-attribute, multi-hop, and entity-level settings, together with explicit retain and unmodified evaluations. All evaluated knowledge is purely contextual, isolating ICM from parametric knowledge. Experiments across different LLMs reveal a striking gap between using and modifying context: unmodified success ranges from 94.50% to 100.00%, whereas modified success ranges from only 0.25% to 35.75%. Existing inference-time methods can improve modification success, but often at the cost of retained or unmodified information. These results show that reliable ICM requires selective contextual state transformation rather than indiscriminate forgetting, and establish ICM-Bench as a benchmark for developing LLM agents that can maintain an accurate current state across evolving conversations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM inference-time modify, contextual state tranformation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AihfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studied a question that conversational agent needs to keep the "vaild status", rather than repeat the whole history. So we need to modify the context information. The authors build a benchmark called ICM-BENCH to measure this problem.

Critical strengths

None. To be honest, the logic of this paper is super hard to follow. I didn't see clear evidence that this paper should be accepted.

Critical weaknesses
  1. this paper studied a trivial problem, almost all the current agent memory management libraries/tools should do by default, in context modification. I cannot see the justification why the authors claim that they formalize this as In-Context Modification problem.
  2. the proposed method to generate the benchmark data is trival and also didn't provide any detail. No support that this benchmark is able to evaluate the key problems of the existing methods. The paper didn't point out the key issues of the existing methods either.
Questions for authors

see the weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Aihf. (Part 1)

Weaknesses

W1. this paper studied a trivial problem, almost all the current agent memory management libraries/tools should do by default, in context modification. I cannot see the justification why the authors claim that they formalize this as In-Context Modification problem.

Thanks for time and efforts in reviewing this work. This work study in-context modification (ICM), an important but not well-studied problem. ICM requires selective modification while preserving unaffected information. A method that successfully handles ICM should not destroy the rest of the context.

We did evaluate a LLM memory management method, Sculptor[1]. However, contrary to the reviewer’s expectations, it does not perform well. The results for Sculptor[1] in Table 6 provide counter-evidence. Moreover, to further evaluate the effectiveness of in-context memory management methods. we evaluate two memory management baselines following Mem0 [2]. Their core mechanisms are summarized below.

MethodCore Mechanisms
SculptorEquips LLMs with three categories of tools, including (1) context fragmentation, (2) summary, hide, and restore, and (3) precise search. LLMs actively manage their in-context with these tools.
Mem0 (Graph)Converts context into a local knowledge graph. It extracts new relations, marks matched old edges as invalid, and adds new ones. Only valid edges are passed to the LLM.
Mem0 (Triplet)Stores context as (subject, relation, object) triples. The model uses ADD, UPDATE, DELETE, or NONE to process new facts. Only active triples are passed to the LLM.

We evaluate these baselines using our benchmark. The results are shown in the table below. Higher is better for all metrics.

ModelMethodTaskMS ↑\uparrowUMS ↑\uparrowMSR ↑\uparrowRS ↑\uparrow
Gemma 4-31BBaseDelete10.49%99.65%10.31%99.89%
Gemma 4-31BSculptorDelete87.85%84.79%87.96%55.92%
Gemma 4-31BMem0 (Graph)Delete95.56%49.03%92.21%42.44%
Gemma 4-31BMem0 (Triplet)Delete45.07%57.85%11.04%60.41%
Gemma 4-31BBaseUpdate1.74%99.86%1.67%99.82%
Gemma 4-31BSculptorUpdate66.81%85.14%66.48%62.91%
Gemma 4-31BMem0 (Graph)Update84.17%57.36%94.43%46.14%
Gemma 4-31BMem0 (Triplet)Update31.18%71.60%14.74%66.59%
Qwen 3.8-27BBaseDelete20.83%99.72%20.75%99.94%
Qwen 3.8-27BSculptorDelete63.96%73.96%62.25%57.58%
Qwen 3.8-27BMem0 (Graph)Delete100.00%0.00%0.00%0.25%
Qwen 3.8-27BMem0 (Triplet)Delete97.15%1.39%50.00%4.49%
Qwen 3.8-27BBaseUpdate0.21%100.00%0.21%99.86%
Qwen 3.8-27BSculptorUpdate40.00%67.99%39.94%89.30%
Qwen 3.8-27BMem0 (Graph)Update44.31%2.01%75.86%4.47%
Qwen 3.8-27BMem0 (Triplet)Update51.04%2.22%34.38%3.81%

As shown in the results, achieving a higher MS consistently degrades UMS and RS. On Gemma 4-31B, Mem0 (Graph) reaches Delete MS 95.56% but RS falls to 42.44%, and Update MS 84.17% with RS 46.14%. Mem0 (Triplet) is conservative (Delete: MS 45.07%, RS 60.41%; Update: MS 31.18%, RS 66.59%). Mem0 (Graph) on Qwen 3.8-27B Delete reaches 100.00% MS with 0.00% UMS and 0.25% RS. For Mem0(Graph), the entire context is suppressed.

If ICM were already solved by default, these metrics should all be close to 100%; however, high MS cannot coexist with high UMS and RS. Thus these baselines do not solve ICM by default, and we frame ICM as an open problem rather than a solved default.

[1] Li M, Xu L H, Tan Q, et al. Sculptor: Empowering llms with cognitive agency via active context management. ICLR 2026.

[2] Chhikara P, Khant D, Aryan S, et al. Mem0: Building production-ready ai agents with scalable long-term memory. ECAI 2025.

AuthorsOpenReview ↗

Response to Reviewer Aihf. (Part 2)

W2. the proposed method to generate the benchmark data is trival and also didn't provide any detail. No support that this benchmark is able to evaluate the key problems of the existing methods. The paper didn't point out the key issues of the existing methods either.

the proposed method to generate the benchmark data is trival and also didn't provide any detail.

(1) Our method used to generate benchmark data is simple but effective in demonstrating the limitation of ICM. Besides, we invested substantial human effort and time in constructing and validating the dataset.

We have described the detail of generating data in details. In Sections 3.4 and 3.5, the paper describes the data construction process of ICM-Bench, yielding 2,880 instances and 26,195 deterministic evaluation queries across single-attribute, multi-hop, and entity-level tasks. Appendix A further provides the details of construction for every scenario. Appendix B reports the details of rule-based and human audits. Each scenario has 10 or more designed rules, with pass rates mostly above 98%. The audit further includes three independent human reviewers (Gwet’s AC1 0.9069 to 0.9511, Majority pass rate 95.83% to 100%) in Table 2. Samples that fail review are discarded or regenerated. The construction module allows ICM-Bench to generate high-quality samples meeting the task requirements.

No support that this benchmark is able to evaluate the key problems of the existing methods. The paper didn't point out the key issues of the existing methods either.

(2) ICM requires an LLM to delete what should be deleted, update what should be updated, and retain what should be retained, all within the current context. However, the paper's experiments indicate that the core key issue of current methods is precisely their inability to selectively modify target information without destroying unaffected context. This problem leads to severe real-world failures: (i) If an agent fails to delete a cancelled order, it will continue to process the shipment, causing direct financial loss. (ii) If a medical agent fails to update a patient's state to not fasting, it may proceed with an inaccurate examination, causing medical consequences.

ICM-Bench evaluates three situations of existing methods:

(i) Modification at the cost of context destruction. In each instance, paired retain and unmodified queries measure RS and UMS.

(ii) Evaluation of different task scenarios. We design single-attribute, multi-hop, and entity-level context information to comprehensively evaluate.

(iii) Entity-level consistency. Entity-level requires all queries for an entity to succeed.

This design is supported by the results in Table 6. Current baselines either fail to improve MS sufficiently or raise MS at the cost of RS. For example, Sculptor in Gemma 4-31B raises Delete MS to 87.85% while RS drops to 55.92%. This clearly demonstrates that ICM-Bench can evaluate the correct modification capability of LLMs and existing methods.


Questions

see the weakness.

Our responses to the questions are provided under W1 and W2 above.

Dear Reviewer,

Thank you for your efforts in evaluating this submission. The authors have uploaded their reply. Please take your responsibility to review the authors' response, revisit your evaluation,and update your score and recommendation as appropriate. Please add comments on whether the authors have sufficiently addressed your raised points. If you have further questions,please discuss them with the authors.

Thanks, Area Chair

Thanks the authors provided the explanations and new data. However, I’m not convinced by the new responses and also I agree with another reviewer’s comment. The authors added too many new data in the rebuttal, which is also the evidence that this paper is not ready for publication. I would keep my original rating score.

Reviewer_AfgcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper formalizes In-Context Modification (ICM): the ability of an LLM, at inference time and without parameter updates, to selectively transform previously accumulated conversational context — deleting information that has become obsolete, updating values that changed, and preserving everything still valid (the partition into deletion / update / retention sets in Eq. 3). To measure it, the authors build ICM-Bench: 2,880 synthetic multi-turn conversations and 26,195 queries constructed from TOFU fictitious-author QA, chosen so that all target knowledge is contextual rather than parametric. The benchmark crosses {Delete, Update} with {single-attribute, multi-hop, entity-level} targets, and probes each instance with direct queries, reverse queries, retain queries about unaffected facts, and a paired unmodified control. Reported metrics are Modified Success (MS), Unmodified Success (UMS), MS-Retention (MSR) and Retain Success (RS). Five LLMs (Llama3.1-8B, GLM4-32B, Gemma4-31B, Qwen3.8-27B, GPT5.6-light) and five inference-time methods (ICUL, ECO, Divergence Decoding, Sculptor, DEER) are evaluated under a deterministic rule-based judge (normalized substring matching plus task-specific logical rules). The authors report three findings: (i) models are near-perfect when nothing changes (UMS 94.50%–100.00%) but fail when a change must be applied (MS 0.25%–35.75%); (ii) inference-time methods do not reliably help — some give small gains, others raise MS at the cost of retention (Sculptor: Gemma delete MS 10.49% → 87.85% while RS drops 99.89% → 55.92% and UMS 99.65% → 84.79%); (iii) ICM-Bench is harder than ICF-Bench and MemoryAgentBench. A short final experiment (RQ3) fine-tunes a Sheared-LLaMA-1.3B "locator" on human [MASK] annotations, suppresses attention on the located tokens, and reports MS 71.46% (Gemma, delete) with RS 95.64%. Quality control combines automated rule audits (mostly >98% pass, a few checks at 90–93%), three independent human reviewers (Gwet's AC1 0.907–0.951), and a judge-validation study (rule-based judge vs. human majority vote: Cohen's κ = 0.958; GPT5.6-light judge vs. human: κ = 0.619).

Critical strengths
  1. "The paired "retain query + unmodified control" design is the paper's real contribution, and it is what makes the benchmark diagnostic rather than another accuracy leaderboard." Scoring RS and UMS alongside MS is what separates selective modification from context destruction: it shows Sculptor's MS gain (10.49% → 87.85%) comes with RS collapsing 99.89% → 55.92%, which any benchmark without these controls would report as a 77-point improvement.
  2. "The benchmark is carefully engineered, cleanly isolated from parametric knowledge, and transparent about quality control." TOFU's fictitious entities keep target knowledge out of the weights, and the Locator–Applier–Backend abstraction runs five heterogeneous methods under one deterministic protocol. Validation goes beyond automated audits: three reviewers (Gwet's AC1 0.907–0.951) and an explicit rule-judge vs. LLM-judge comparison on the same human reference (κ = 0.958 vs. 0.619).
Critical weaknesses
  1. The success criterion does not validly measure ICM, and the headline gap rests entirely on it. Delete is scored as "target string absent," so refusals and degenerate replies count as success; Update fails whenever the old value appears, so "now leadership — earlier you said drama" is marked wrong; and multi-hop Update only requires suppressing the old value, never producing the new one — likely why Qwen scores 19.90% delete vs. 0.00% update. The κ = 0.958 validation does not repair this, since reviewers and the GPT judge applied the same rule: it shows reproducibility, not construct validity.
  2. The one constructive result is unvalidated, and key comparisons rest on thin statistics. RQ3's locator is trained on human MASK labels of ICM-Bench and evaluated on ICM-Bench with no split disclosed and no ablation (random span, oracle mask, or simply suppressing the whole modification turn), on only 2 of 5 models. Entity splits are n = 100 per requirement and multi-hop n = 240, all single greedy runs without confidence intervals, and baselines cover only 3 of 5 models.
Questions for authors
  1. Re-scoring. Please re-score a stratified sample with a judge that credits Delete only for well-formed non-disclosure (not degenerate refusal) and credits Update answers that state the new value while mentioning the old one. What fraction of failed Delete responses are refusals/non-answers, and what fraction of failed Update responses already contain the updated value?
  2. Multi-hop Update. Confirm that it is graded solely on suppressing the obsolete value; if so, how do the Update columns of Tables 5–6 change when the updated answer must actually be produced?
  3. Locator validity. Were the MASK-labeled training instances disjoint from those in Table 7? Please report split sizes and add random-span, oracle-mask, and whole-turn-suppression variants, plus per-scenario N and confidence intervals.
  4. Scope checks. Please probe the TOFU facts with no context to verify they are not parametrically known, and report how results change when the change is conveyed implicitly rather than by an explicit instruction.
9 comments
AuthorsOpenReview ↗

Response to Reviewer Afgc. (Part 1)

Weaknesses

W1. The success criterion does not validly measure ICM, and the headline gap rests entirely on it. Delete is scored as "target string absent," so refusals and degenerate replies count as success; Update fails whenever the old value appears, so "now leadership — earlier you said drama" is marked wrong; and multi-hop Update only requires suppressing the old value, never producing the new one — likely why Qwen scores 19.90% delete vs. 0.00% update. The κ = 0.958 validation does not repair this, since reviewers and the GPT judge applied the same rule: it shows reproducibility, not construct validity.

Thanks for the reviewer's valueful insight. We address the concerns by analyzing and verifying (with human analyzers) the scenarios in our benchmark as follows.

In the Delete scenario, a refusal “I don't know” is a correct response and it does not affect our results. For degenerate replies such as unrelevant or hallucinated replies, we analyze these cases raised by the reviewer through three independent human analyzers. The analyzed cases are sampled proportionally to each scenario's size, yielding overall 291 responses across the five models. These statistics are listed as follows.

For the deleted scenario, there are 4 degenerated cases out of 291 responses.

ModelDegenerate Reply
Gemma 4-31B0
Qwen 3.8-27B1
GLM 4-32B0
Llama 3.1-8B1
GPT 5.6-light2

For the update scenario, there are two cases where the old value appears with the new value (e.g., "now leadership — earlier you said drama"). Such cases are rare.

Modelold value with new value
Gemma 4-31B0
Qwen 3.8-27B1
GLM 4-32B0
Llama 3.1-8B0
GPT 5.6-light1

For Multi-hop Update scenario, we construct a new Multi-hop Update scenario in which each sample requires the model to actually produce the updated answer along the multi-hop updated chain. The new set contains 240 samples, the same size as the original multi-hop update scenario. We report results on original and new scenarios as follows.

Llama's MS suffers more in the new scenario than the old scenario. Gemma 4-31B's and Qwen 3.8-27B's MS also remain low. This indicates that the finding that today's LLMs perform poorly for the ICM problem.

ModelDataMS ↑\uparrowUMS ↑\uparrowMSR ↑\uparrowRS ↑\uparrow
Llama 3.1-8BOriginal Multi-hop Update11.25%96.67%10.34%95.98%
Llama 3.1-8BNew Multi-hop Update6.67%85.00%7.35%98.62%
Gemma 4-31BOriginal Multi-hop Update6.67%99.58%6.28%99.79%
Gemma 4-31BNew Multi-hop Update8.33%87.50%6.67%99.86%
Qwen 3.8-27BOriginal Multi-hop Update1.25%100.00%1.25%99.90%
Qwen 3.8-27BNew Multi-hop Update5.00%84.58%4.93%99.72%

In summary, these new results does not change the main conclusions of this work.

The κ = 0.958 validation does not repair this, since reviewers and the GPT judge applied the same rule: it shows reproducibility, not construct validity.

In our work, there are three judges: human judge, Rule (Python) judge, and GPT judge. The three human (AI researchers) judges labeled samples independently, without given any rules (prompts). They evaluate the LLM responses accroding to their common sense. Thus, their judge results can be used as the ground truth. The κ = 0.958 validation measure the agreement between the results among human judge and rule judge. Such metric κ is construct validity rather than reproducibility. The high Human–Rule agreement indicates that the rule implementation matches independent human judgments, providing evidence for the correctness of the rules. The GPT-5.6-light judge is a separate evaluator with its own prompt (Appendix B.3). As the GPT judge has weaker alignment than the rule judge. We use the rule judge to evaluate the response of LLMs.

AuthorsOpenReview ↗

Response to Reviewer Afgc. (Part 2)

W2. The one constructive result is unvalidated, and key comparisons rest on thin statistics. RQ3's locator is trained on human MASK labels of ICM-Bench and evaluated on ICM-Bench with no split disclosed and no ablation (random span, oracle mask, or simply suppressing the whole modification turn), on only 2 of 5 models. Entity splits are n = 100 per requirement and multi-hop n = 240, all single greedy runs without confidence intervals, and baselines cover only 3 of 5 models.

Thanks for your suggestions. We describe more details and design multiple variants of the proposed method to validate it.

RQ3's locator is trained on human MASK labels of ICM-Bench and evaluated on ICM-Bench with no split disclosed and no ablation (random span, oracle mask, or simply suppressing the whole modification turn)

(1) The locator is a Sheared-LLaMA-1.3B model fine-tuned for token-level binary classification. Given the full context, it outputs whether the token should be suppressed. Attention scores to these tokens is suppressed during the backbone model responses a query. The locator is trained on human-annotated [MASK] labels.

We ensure that every task scenario appears in both train and test in the same proportion. The MASK-labeled contexts are split within each of different task scenario at a 6:4 ratio. The train parts of task scenarios are merged into the train set, and the test parts into the test set. The resulting sizes are 1,728 train and 1,152 test samples. No context appears in both splits. The result indicates that the locator generalizes to unseen contexts rather than memorizing training contexts.

ModelSplitthe number of samplesMSUMSMSRRS
Gemma 4-31BTrain1,72871.47%99.25%71.37%97.03%
Gemma 4-31BTest1,15267.62%98.61%67.25%96.48%
Qwen 3.8-27BTrain1,72854.34%99.59%54.21%97.87%
Qwen 3.8-27BTest1,15254.77%99.83%54.70%97.66%

Following the reviewer's suggestion, we evaluate three ablation variants. These specific forms are summarized in the table below:

Ablation VariantCore Description
Oracle-maskLocates tokens using human-annotated [MASK] labels, then suppresses their attention scores.
Random-spanLocates random 40% spans within the context, then suppresses their attention scores.
Whole-turn suppressionUses our trained locator to find target tokens, then suppresses the attention scores of all tokens within the corresponding turn.
OursUses our trained Sheared-LLaMA-1.3B classifier to auto locate target tokens, then suppresses their attention scores.

The results show that random span is ineffective, while whole-turn suppression achieves higher MS but much lower RS due to over-suppression. In contrast, our method approaches Oracle-mask across all metrics, indicating the effective of ours.

VariantModelTaskMS ↑\uparrowUMS ↑\uparrowMSR ↑\uparrowRS ↑\uparrow
Oracle-maskGemma 4-31BDelete81.25%99.65%81.18%99.91%
Random-spanGemma 4-31BDelete36.25%99.65%36.17%78.29%
Whole-turn suppressionGemma 4-31BDelete79.86%99.44%79.82%59.14%
OursGemma 4-31BDelete71.46%99.65%71.43%95.64%
Oracle-maskGemma 4-31BUpdate74.31%99.86%74.27%99.66%
Random-spanGemma 4-31BUpdate35.21%99.86%35.12%83.26%
Whole-turn suppressionGemma 4-31BUpdate71.25%99.79%71.26%73.73%
OursGemma 4-31BUpdate68.40%98.33%68.01%97.97%

on only 2 of 5 models. ... all single greedy runs without confidence intervals

(2) For the confidence interval of reviewer concern, we add Llama 3.1-8B based on our method by training the SFT-based locator with five independent random seeds and evaluate it on our benchmark. We report the mean and standard deviation on the train and test set. The results show improvements in MS and MSR, with limited effects on UMS and RS.

ModelSplitthe number of samplesMS ↑\uparrowUMS ↑\uparrowMSR ↑\uparrowRS ↑\uparrow
Llama 3.1-8Btrain1,72857.91 ±\pm 1.9794.32 ±\pm 0.0556.50 ±\pm 2.1994.25 ±\pm 0.93
Llama 3.1-8Btest1,15253.37 ±\pm 2.4394.79 ±\pm 0.0651.98 ±\pm 2.4593.69 ±\pm 1.10
AuthorsOpenReview ↗

Response to Reviewer Afgc. (Part 3)

W2. The one constructive result is unvalidated, and key comparisons rest on thin statistics. RQ3's locator is trained on human MASK labels of ICM-Bench and evaluated on ICM-Bench with no split disclosed and no ablation (random span, oracle mask, or simply suppressing the whole modification turn), on only 2 of 5 models. Entity splits are n = 100 per requirement and multi-hop n = 240, all single greedy runs without confidence intervals, and baselines cover only 3 of 5 models.

baselines cover only 3 of 5 models

(3) We have now extended the baseline comparisons to cover all 5 models, including GLM 4-32B and Llama 3.1-8B. As shown in the supplementary tables below, the results are fully consistent with our original conclusions: existing baselines either fail to achieve high MS or achieve it at the cost of UMS and RS.

ModelMethodTaskMS ↑\uparrowUMS ↑\uparrowMSR ↑\uparrowRS ↑\uparrow
GLM 4-32BBaseDelete3.54%98.33%2.68%99.62%
GLM 4-32BICULDelete2.99%98.19%1.98%99.63%
GLM 4-32BECODelete66.39%25.21%12.12%38.68%
GLM 4-32BDDDelete32.50%62.99%13.34%87.07%
GLM 4-32BSculptorDelete83.06%30.97%76.46%13.23%
GLM 4-32BDEERDelete31.39%94.86%30.16%98.40%
GLM 4-32BBaseUpdate3.89%99.37%3.78%99.43%
GLM 4-32BICULUpdate5.69%99.51%5.58%99.56%
GLM 4-32BECOUpdate66.18%23.26%28.36%37.16%
GLM 4-32BDDUpdate22.71%63.47%5.25%70.18%
GLM 4-32BSculptorUpdate29.24%26.18%30.77%7.62%
GLM 4-32BDEERUpdate18.89%98.40%18.77%98.87%
Llama 3.1-8BBaseDelete3.89%96.39%2.65%97.19%
Llama 3.1-8BICULDelete11.53%91.53%7.66%96.67%
Llama 3.1-8BECODelete70.00%21.60%20.26%29.51%
Llama 3.1-8BDDDelete26.67%68.47%8.42%92.52%
Llama 3.1-8BSculptorDelete88.96%16.46%84.39%8.21%
Llama 3.1-8BDEERDelete7.01%90.07%4.32%97.43%
Llama 3.1-8BBaseUpdate2.01%98.33%1.79%97.80%
Llama 3.1-8BICULUpdate10.76%95.90%9.12%74.66%
Llama 3.1-8BECOUpdate38.54%18.75%11.48%28.00%
Llama 3.1-8BDDUpdate13.26%85.35%8.06%79.41%
Llama 3.1-8BSculptorUpdate22.57%11.74%38.46%11.17%
Llama 3.1-8BDEERUpdate10.62%94.65%9.83%95.11%
Reviewer_RaYWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces In-Context Modification (ICM): the ability of an LLM to delete or update information stated earlier in a conversation, without parameter updates, while keeping unaffected information usable. The authors build ICM-Bench from TOFU fictitious-author profiles. It contains 2,880 multi-turn conversations and 26,195 queries, covering deletion and update across single-attribute, multi-hop, and entity-level targets, with modified, retain, and paired unmodified queries. Responses are scored using substring matching and task-specific rules. Across five LLMs, unmodified success ranges from 94.5% to 100%, while modified success ranges from only 0.25% to 35.75%. Existing inference-time methods either provide limited gains or improve modification at the cost of retained information. A small trained locator combined with attention suppression substantially improves modification success.

Critical strengths

S1. Explicit evaluation of knowledge preservation. Retain queries and paired unmodified controls help distinguish selective modification from methods that indiscriminately suppress contextual information.

S2. Clean isolation from parametric knowledge. The benchmark uses fictitious profiles with facts supplied through the conversation, reducing interference from pretrained factual knowledge. Data construction is documented in detail, including generation prompts, automated audits, and human agreement statistics.

Critical weaknesses

W1. The Update scoring rule may penalize correct responses. Appendix A.3 marks an answer incorrect whenever it mentions the old value, even if it correctly states the update (e.g., "drama was updated to leadership"). This may partly explain the near-zero single-attribute Update MS (0.00–0.30%) and entity-direct Update MS (0.00% across all models) in Tables 21 and 23.

W2. The evaluation can reward incorrect answers as successful modification. In Delete and Multi-hop Update tasks, success primarily requires the obsolete answer to be absent (Appendix A.3). Irrelevant or hallucinated responses may therefore pass, conflating correct contextual modification with simple answer suppression.

W3. The evaluation lacks a straightforward state-updating baseline. The paper mainly compares inference-time unlearning and context-management methods, without evaluating a simple approach that explicitly maintains and updates the current facts. Such a baseline is important for determining whether the reported failures reflect fundamental ICM difficulties or limitations of the tested approaches.

Questions for authors

Q1. Could the authors address the scoring concerns in W1–W2 by examining how often correct updates are rejected and incorrect responses are accepted?

Q2. Could the authors evaluate a simple explicit state-updating baseline to address W3?

3 comments
AuthorsOpenReview ↗

Response to Reviewer RaYW. (Part 1)

Weakness

W1. The Update scoring rule may penalize correct responses. Appendix A.3 marks an answer incorrect whenever it mentions the old value, even if it correctly states the update (e.g., "drama was updated to leadership"). This may partly explain the near-zero single-attribute Update MS (0.00–0.30%) and entity-direct Update MS (0.00% across all models) in Tables 21 and 23.

Thanks for the reviewer's advice.

(1) We analyaze the cases pointout by the reviewer in the single-attribute and entity-direct scenarios, through three independent human analyzers, regarding the case that an old value was mentioned in the answer (e.g., "drama was updated to leadership"). We sample proportionally to each scenario's size, yielding overall 291 responses across the five models.

Scenariomentioning the old value
Single-attribute2
Entity-direct0

In total, there is only 2 out of the 291 responses from our benchmark. This indicates that such cases are rare.

(2) We re-evaluate LLMs with improved judgement with such cases using the modified success (MS) score. Δ\Delta denotes the change from original MS to re-scoring MS. The changes to existing results are close to zero. The updated results confirm that the tested LLMs still struggle on both Single-attribute and Entity-direct scenario. The updated evaluation results do not change the conclusion that ICM is difficult for current LLM models.

ScenarioModelOriginal MSRe-scoring MSΔ\Delta
Single-attributeGemma 4-31B0.00%0.00%+0.00
Single-attributeQwen 3.8-27B0.00%0.10%+0.10
Single-attributeGLM 4-32B0.20%0.20%+0.00
Single-attributeLlama 3.1-8B0.20%0.20%+0.00
Single-attributeGPT 5.6-light0.30%2.60%+2.30
Entity-directGemma 4-31B0.00%0.00%+0.00
Entity-directQwen 3.8-27B0.00%0.00%+0.00
Entity-directGLM 4-32B0.00%0.00%+0.00
Entity-directLlama 3.1-8B0.00%0.00%+0.00
Entity-directGPT 5.6-light0.00%0.00%+0.00

W2. The evaluation can reward incorrect answers as successful modification. In Delete and Multi-hop Update tasks, success primarily requires the obsolete answer to be absent (Appendix A.3). Irrelevant or hallucinated responses may therefore pass, conflating correct contextual modification with simple answer suppression.

Thanks for reviewer's valueful comments.

(1) We analyzed these cases that may include irrelevant or hallucinated response, through three independent human analyzers for the Delete and Multi-hop Update tasks. The analyzed responses are sampled proportionally to each scenario's size, yielding overall 291 responses across 5 LLMs. The human-evaluated results are depicted in the following table. In the benchmarks, there are only a few irrelevant answers, which account for 2.5% percent of the responses.

TaskIrrelevantHallucinated
Delete40
Multi-hop Update30

(2) We reevaluated the LLM responses through updated judgements with such cases in mind. The results are depicted as follows.

Delete. The table below compares the original MS and the re-scoring MS. Δ\Delta denotes the change from original MS to re-scoring MS. The updating result of re-scoring is shown the following:

ModelOriginal MSRe-scoring MSΔ\Delta
Gemma 4-31B10.49%10.07%-0.42
Qwen 3.8-27B20.83%20.56%-0.28
GLM 4-32B3.54%2.15%-1.39
Llama 3.1-8B3.89%2.78%-1.11
GPT 5.6-light11.18%9.86%-1.32

For the Multi-hop Update cases, we construct a new scenario, consisting of Multi-hop Update samples. In this scenario, each sample requires the model to produce the answer along the new multi-hop update chain. The new scenario contains 240 samples, the same number of samples as the original multi-hop update scenario. We report results on both the original and the new Multi-hop Update scenarios as follows.

ModelDataMS ↑\uparrowUMS ↑\uparrowMSR ↑\uparrowRS ↑\uparrow
Llama 3.1-8BOriginal Multi-hop Update11.25%96.67%10.34%95.98%
Llama 3.1-8BNew Multi-hop Update6.67%85.00%7.35%98.62%
Gemma 4-31BOriginal Multi-hop Update6.67%99.58%6.28%99.79%
Gemma 4-31BNew Multi-hop Update8.33%87.50%6.67%99.86%
Qwen 3.8-27BOriginal Multi-hop Update1.25%100.00%1.25%99.90%
Qwen 3.8-27BNew Multi-hop Update5.00%84.58%4.93%99.72%

Llama's MS suffers more in the new scenario than the old scenario. Gemma 4-31B's and Qwen 3.8-27B's MS also remain low. We show that our findings regarding the multi-hop update challenge for LLMs do not change.

AuthorsOpenReview ↗

Response to Reviewer RaYW. (Part 2)

W3. The evaluation lacks a straightforward state-updating baseline. The paper mainly compares inference-time unlearning and context-management methods, without evaluating a simple approach that explicitly maintains and updates the current facts. Such a baseline is important for determining whether the reported failures reflect fundamental ICM difficulties or limitations of the tested approaches.

Thanks for your advice. We design two simple state-updating baselines: Mem0(Graph) and Mem0 (Triplet), based on Mem0[1]. Mem0 is a well-performing memory management method that supports add, update, delete, and retrieval operations for LLM agents. We describe the two baselines as follows.

Mem0 (Graph). The historical context is converted into a local knowledge graph with typed entity nodes and directed relation edges. Entities and relations are extracted by LLM, and relations are added as edges with deduplication by normalized node IDs. For the modified state, we extract entities and relations from the modified context, pass the current valid edges together with the instruction to the model, and mark the matched edges as invalid. New relations are then added to the graph. Invalid edges are retained in the trace but never serialized into the answering prompt, so only valid edges are used at answer time.

Mem0 (Triplet). The historical context is stored as (subject, relation, object) triples. For the modified state, we extract new triples from the modified turn and pass them together with the current active triples to Mem0, which selects ADD, UPDATE, DELETE, or NONE for each new fact. ADD appends a triple; UPDATE deactivates old triples with the same subject and relation before appending the new one; DELETE marks matching triples as inactive; NONE leaves the store unchanged. As in Graph-Mem0, inactive triples remain in the trace but only active triples are serialized into the answering prompt.

We evaluate these two baselines against ICM-bench with two LLMs. The results of these state-updating baselines are consistent with our original evaluation. These methods raise MS at the cost of UMS and RS, so they do not resolve existing defects of ICM.

ModelMethodTaskMS ↑\uparrowUMS ↑\uparrowMSR ↑\uparrowRS ↑\uparrow
Gemma 4-31BMem0 (Graph)Delete95.56%49.03%92.21%42.44%
Gemma 4-31BMem0 (Triplet)Delete45.07%57.85%11.04%60.41%
Gemma 4-31BMem0 (Graph)Update84.17%57.36%94.43%46.14%
Gemma 4-31BMem0 (Triplet)Update31.18%71.60%14.74%66.59%
Qwen 3.8-27BMem0 (Graph)Delete100.00%0.00%0.00%0.25%
Qwen 3.8-27BMem0 (Triplet)Delete97.15%1.39%50.00%4.49%
Qwen 3.8-27BMem0 (Graph)Update44.31%2.01%75.86%4.47%
Qwen 3.8-27BMem0 (Triplet)Update51.04%2.22%34.38%3.81%

[1] Chhikara P, Khant D, Aryan S, et al. Mem0: Building production-ready ai agents with scalable long-term memory. ECAI 2025.


Questions

Q1. Could the authors address the scoring concerns in W1–W2 by examining how often correct updates are rejected and incorrect responses are accepted?

We have addressed the scoring cocerns in W1-W2. The wrong cases are rare (about 2%). We find that existing LLMs and baselines perform poorly on ICM.

Q2. Could the authors evaluate a simple explicit state-updating baseline to address W3?

We have designed two simple baselines in our response to W3.

Dear Reviewer,

Thank you for your efforts in evaluating this submission. The authors have uploaded their reply. Please take your responsibility to review the authors' response, revisit your evaluation,and update your score and recommendation as appropriate. Please add comments on whether the authors have sufficiently addressed your raised points. If you have further questions,please discuss them with the authors.

Thanks, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.