- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studied a question that conversational agent needs to keep the "vaild status", rather than repeat the whole history. So we need to modify the context information. The authors build a benchmark called ICM-BENCH to measure this problem.
Critical strengths
None. To be honest, the logic of this paper is super hard to follow. I didn't see clear evidence that this paper should be accepted.
Critical weaknesses
- this paper studied a trivial problem, almost all the current agent memory management libraries/tools should do by default, in context modification. I cannot see the justification why the authors claim that they formalize this as In-Context Modification problem.
- the proposed method to generate the benchmark data is trival and also didn't provide any detail. No support that this benchmark is able to evaluate the key problems of the existing methods. The paper didn't point out the key issues of the existing methods either.
Questions for authors
see the weakness.
4 comments
Response to Reviewer Aihf. (Part 2)
W2. the proposed method to generate the benchmark data is trival and also didn't provide any detail. No support that this benchmark is able to evaluate the key problems of the existing methods. The paper didn't point out the key issues of the existing methods either.
the proposed method to generate the benchmark data is trival and also didn't provide any detail.
(1) Our method used to generate benchmark data is simple but effective in demonstrating the limitation of ICM. Besides, we invested substantial human effort and time in constructing and validating the dataset.
We have described the detail of generating data in details. In Sections 3.4 and 3.5, the paper describes the data construction process of ICM-Bench, yielding 2,880 instances and 26,195 deterministic evaluation queries across single-attribute, multi-hop, and entity-level tasks. Appendix A further provides the details of construction for every scenario. Appendix B reports the details of rule-based and human audits. Each scenario has 10 or more designed rules, with pass rates mostly above 98%. The audit further includes three independent human reviewers (Gwet’s AC1 0.9069 to 0.9511, Majority pass rate 95.83% to 100%) in Table 2. Samples that fail review are discarded or regenerated. The construction module allows ICM-Bench to generate high-quality samples meeting the task requirements.
No support that this benchmark is able to evaluate the key problems of the existing methods. The paper didn't point out the key issues of the existing methods either.
(2) ICM requires an LLM to delete what should be deleted, update what should be updated, and retain what should be retained, all within the current context. However, the paper's experiments indicate that the core key issue of current methods is precisely their inability to selectively modify target information without destroying unaffected context. This problem leads to severe real-world failures: (i) If an agent fails to delete a cancelled order, it will continue to process the shipment, causing direct financial loss. (ii) If a medical agent fails to update a patient's state to not fasting, it may proceed with an inaccurate examination, causing medical consequences.
ICM-Bench evaluates three situations of existing methods:
(i) Modification at the cost of context destruction. In each instance, paired retain and unmodified queries measure RS and UMS.
(ii) Evaluation of different task scenarios. We design single-attribute, multi-hop, and entity-level context information to comprehensively evaluate.
(iii) Entity-level consistency. Entity-level requires all queries for an entity to succeed.
This design is supported by the results in Table 6. Current baselines either fail to improve MS sufficiently or raise MS at the cost of RS. For example, Sculptor in Gemma 4-31B raises Delete MS to 87.85% while RS drops to 55.92%. This clearly demonstrates that ICM-Bench can evaluate the correct modification capability of LLMs and existing methods.
Questions
see the weakness.
Our responses to the questions are provided under W1 and W2 above.
Dear Reviewer,
Thank you for your efforts in evaluating this submission. The authors have uploaded their reply. Please take your responsibility to review the authors' response, revisit your evaluation,and update your score and recommendation as appropriate. Please add comments on whether the authors have sufficiently addressed your raised points. If you have further questions,please discuss them with the authors.
Thanks, Area Chair
Thanks the authors provided the explanations and new data. However, I’m not convinced by the new responses and also I agree with another reviewer’s comment. The authors added too many new data in the rebuttal, which is also the evidence that this paper is not ready for publication. I would keep my original rating score.
Response to Reviewer Aihf. (Part 1)
Weaknesses
Thanks for time and efforts in reviewing this work. This work study in-context modification (ICM), an important but not well-studied problem. ICM requires selective modification while preserving unaffected information. A method that successfully handles ICM should not destroy the rest of the context.
We did evaluate a LLM memory management method, Sculptor[1]. However, contrary to the reviewer’s expectations, it does not perform well. The results for Sculptor[1] in Table 6 provide counter-evidence. Moreover, to further evaluate the effectiveness of in-context memory management methods. we evaluate two memory management baselines following Mem0 [2]. Their core mechanisms are summarized below.
We evaluate these baselines using our benchmark. The results are shown in the table below. Higher is better for all metrics.
As shown in the results, achieving a higher MS consistently degrades UMS and RS. On Gemma 4-31B, Mem0 (Graph) reaches Delete MS 95.56% but RS falls to 42.44%, and Update MS 84.17% with RS 46.14%. Mem0 (Triplet) is conservative (Delete: MS 45.07%, RS 60.41%; Update: MS 31.18%, RS 66.59%). Mem0 (Graph) on Qwen 3.8-27B Delete reaches 100.00% MS with 0.00% UMS and 0.25% RS. For Mem0(Graph), the entire context is suppressed.
If ICM were already solved by default, these metrics should all be close to 100%; however, high MS cannot coexist with high UMS and RS. Thus these baselines do not solve ICM by default, and we frame ICM as an open problem rather than a solved default.
[1] Li M, Xu L H, Tan Q, et al. Sculptor: Empowering llms with cognitive agency via active context management. ICLR 2026.
[2] Chhikara P, Khant D, Aryan S, et al. Mem0: Building production-ready ai agents with scalable long-term memory. ECAI 2025.