acceptodds
Under review as a conference paper at ICLR 2027

When Personal Memory Has No Single Answer: Evaluating LLM Agents Facing Underdetermined Conflicts

Abstract

Large Language Model (LLM)-based agents increasingly retain personal memory across sessions, but such memory can contain inconsistencies involving context-dependent preferences, evolving behaviors, and conflicting sources. When a query omits the context, time, or source authority needed to interpret these inconsistencies, treating one memory as definitive produces unjustified, overconfident actions. Existing benchmarks reward designated outcomes, overlooking whether agents recognize underdetermination, preserve alternatives, seek missing information, and express evidence-compatible behavior. We introduce Testing Agents' Navigation of Genuine, Latent, and Entangled Memory Conflicts (TANGLE), a controlled benchmark for evaluating cognitive behavior when agents face underdetermined conflicts in personal memory. TANGLE comprises 541 fully synthetic instances that span 40 diverse personas and three conflict types: Context-Partitioned Conflict (CPC), Behavior-Oscillation Conflict (BOC), and Source-Contradiction Conflict (SCC). We evaluate matched oracle and pipeline tracks, using curated memory and system-retrieved memories from multi-session dialogues, respectively. Results show that explicit conflict recognition is more frequent than calibrated-action or targeted-clarification behavior under the rubric; retrieved memories can also omit relations needed for downstream reasoning. Fixed resolution rules show structural limitations; we therefore introduce Conflict-Aware Action Policy (CAAP) as a two-stage prompting-based policy baseline that selects evidence-grounded actions case by case. Together, TANGLE and our findings frame conflict handling as recognizing underdetermination, retaining conflicting evidence, and selecting an evidence-compatible response.

Primary area:
datasets and benchmarks
Keywords:
underdetermined memory conflicts, benchmark evaluation, LLM agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yRoxOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TANGLE, a synthetic benchmark of 541 personal-memory conflicts across three types where the query withholds the information needed to resolve the conflict.

Critical strengths

The problem is well chosen: conflicts where no record is wrong and no single answer exists are common in long-term memory and under-studied compared to stale-fact updates.

Critical weaknesses
  • No negative class, so the benchmark can’t separate “recognizes underdetermination” from “always hedges.” All 541 instances are conflicts by construction, and D3/D4 reward expressing uncertainty and asking. Your own Always-ask baseline (Fig. 4) scores higher than every real model on D3, D4, and D5. The headline finding—models perceive conflict but don’t calibrate or clarify reads just as well as “models don’t hedge unconditionally,” which may be correct behavior.
  • Underdetermination is assumed, not validated. The generator was instructed to make conflicts unresolvable, but no human ever checked whether a reasonable person actually finds them unresolvable; Appendix L says the review is “not a human gold label for the best action.” If some instances are resolvable with ordinary judgment, committing is right and gets penalized on D3.
  • The data is 100% single-model synthetic with no quantified human review. Everything—memories, distractors, queries, dialogues—comes from Claude Sonnet 4.6. The human review covers only coherence and non-leakage, with no reviewer count, sample size, rejection rate, or agreement reported. Memory items are stylistically uniform (18–32-word third-person statements), so D1 may partly reflect surface-pattern detection.
  • Scoring rests on one human annotator and weak judge–human agreement. With a single annotator there is no human IAA. Judge–human exact agreement is 46–55% on a 0–4 scale, and D4—the dimension the main conclusion depends on is the worst: on SCC, MAE 1.36–1.49 and within-one agreement only 54–58%.
  • Same-family loop. Sonnet 4.6 generates, Opus 4.7 judges, Sonnet 5 wins. CAAP is a prompt that tells the model to pick from the actions the rubric rewards, scored by the same rubric, with no cost for asking so its D4 gains are expected and don’t show it beats Always-ask, which already outscores it.
Questions for authors

please see above

7 comments
AuthorsOpenReview ↗

# Response to Reviewer yRox - Part 1

We thank the reviewer for the careful and probing feedback. We are grateful that the reviewer recognizes TANGLE's focus on an important and underexplored class of underdetermined long-term-memory conflicts. In response to the reviewer’s concerns, we would like to provide the necessary clarifications and explain why they do not undermine our claim. We have considered every point and revised the manuscript accordingly, with changes marked in blue. We address the reviewer’s concerns point by point below.

W1. Negative class and evidence-sensitive action selection.

To test whether the observed caution is sensitive to the evidence state rather than a result of indiscriminate hedging, we added 60 matched controls for 30 original cases to the revision—10 cases from each conflict type (BOC, CPC, and SCC)—yielding 90 inputs in total across the unresolved, resolved, and no-conflict conditions. The resolved controls supply the missing context, phase, or source authority, while the no-conflict controls remove or rewrite one side of the conflict. Both controls preserve the persona, domain, query form, memory bank, and distractors, including their order and count. The variants are constructed with Claude Sonnet 4.6. As a construction-quality check, two independent annotators review six sampled controls each. All 12 judgments confirm that the intended edit succeeds (100%).

We evaluate the five response models used in the paper—Claude Sonnet 5, Gemini 3.1 Pro, GPT-4o, DeepSeek-V3.2, and GLM-4.7—on the 60 control inputs, yielding 300 responses. GPT-5.4 scores D3 and D4 using the paper's original rubric. The table reports the mean scores by condition; D3 measures confidence calibration and D4 measures decision-relevant clarification seeking.

Response modelUnresolved D3Unresolved D4Resolved D3Resolved D4No-conflict D3No-conflict D4
Claude Sonnet 52.632.233.800.003.530.67
Gemini 3.1 Pro2.031.933.700.173.370.97
GPT-4o2.371.373.470.003.500.33
DeepSeek-V3.22.201.673.670.103.670.43
GLM-4.72.201.803.370.132.930.50

Across all five models, moving from the unresolved condition to the resolved controls raises D3 from 2.03–2.63 to 3.37–3.80 and lowers D4 from 1.37–2.23 to 0.00–0.17. The resolved controls therefore show high calibration and near-zero clarification seeking when the missing context, phase, or source authority is supplied. No-conflict controls show the same pattern, with D3 scores of 2.93–3.67 and D4 scores of 0.33–0.97.

In resolved and no-conflict cases, a low D4 is the appropriate behavior because no unresolved variable remains for a question to adjudicate; D4 should be high when targeted clarification can resolve an underdetermined conflict. Thus, the matched benchmark conditions separate recognizing underdetermination from always hedging, and the D3–D4 rubric distinguishes them accordingly: it rewards calibrated confidence and direct response when the evidence is sufficient, while assessing whether the model expresses calibrated uncertainty and seeks decision-relevant information when the evidence remains underdetermined. We have added new content to §4.2 and a new appendix subsection, Appendix N, “Evidence-Sufficiency Controls.”

As for the headline finding, we would like to respectfully clarify that our paper does not claim that caution or clarification is always preferable, nor that every commitment is inappropriate. Rather, it reports that models often identify and describe the conflict, but less consistently calibrate their commitment or request the specific missing information that could change the recommendation. The negative-class results provide the corresponding evidence-sufficient contrast: the same models receive high D3 and near-zero D4 when the evidence is sufficient. This supports interpreting the observed calibration and clarification gap as a behavior profile specific to underdetermined conflict settings.

AuthorsOpenReview ↗

# Response to Reviewer yRox - Part 2

W2. Underdetermination validation and rubric interpretation.

During generation, we have manually checked that the intended conflict for each conflict type is present and that the query omits the context, time, or source authority needed to determine which record applies. To further validate underdetermination, we conduct an independent audit in which two human annotators assess a 10-case subset spanning all three conflict types, yielding 20 case-level judgments. The 20 judgments are positive (100%) for retaining an unresolved decision-relevant variable. This audit validates the intended underdetermination. We have added the underdetermination validation details to §3.3 and Appendix L, “Generation, Session Synthesis, and Quality Control,” with the added text highlighted in blue.

Appendix L uses the phrase “not a human gold label for the best action” to define the scope of the construction review. The review checks narrative coherence, non-leakage, and whether the target conflict remains reconstructable only from the aggregate evidence; it does not assign a unique correct response. This follows the paper's rubric design: underdetermined conflicts do not have a single licensed answer, and multiple evidence-compatible strategies can be reasonable.

Within this scope, we would like to respectfully clarify that when the available evidence supports resolution through ordinary judgment, a corresponding commitment is appropriate and is not penalized by D3; what D3 penalizes is unsupported over-commitment.

W3. Quantified human validation and memory style.

We used a gated, iterative construction protocol. We first generated a 30-instance pilot, balanced across the three conflict types (10 BOC, 10 CPC, and 10 SCC). Two reviewers manually checked coherence, non-leakage, and whether each target conflict remained reconstructable only from the aggregate evidence; issues were revised and re-reviewed until the pilot was jointly acceptable. The schema, coverage, conflict-structure, query, and distractor checks followed the deterministic and model-based quality-control procedures described in Appendix L.

Once the pilot was jointly accepted, we scaled the validated construction procedure to the full benchmark. The full-generation pool contained 550 instances; we filtered out 9 instances (1.6%) whose conflicts mixed axes and were not sufficiently clear, leaving 541 retained instances. We then conducted a final stratified spot check of 30 retained instances (10 per conflict type), and all 30 passed (100%; 0 rejected). We have added these details to §3.3 and Appendix L.

As for memory style, across the 541 instances, 4,893 core memory items were analyzed: 3,647 (74.5%) contain 18–32 words, lengths range from 8–83 words (median 29), and 52 items contain first-person pronouns while 17 contain second-person pronouns. We describe them precisely in §3.3, “Data Construction,” with the revision highlighted in blue.

AuthorsOpenReview ↗

# Response to Reviewer yRox - Part 3

W4. Evaluator agreement and robustness of the behavioral profile.

We would like to respectfully clarify that judge–human agreement is not weak when evaluated using the ordinal structure of the 0–4 rubric. On the 518-response cohort, GPT-5.4 and Opus achieve 84.4% and 82.1% within-one agreement with the human annotator, and the two independent LLM judges agree within one point on 90.5% of scores. Exact matching requires identical integer scores on a 0–4 scale and is therefore a particularly strict criterion; the high within-one agreement provides the appropriate reliability evidence. We report both judges separately, and the same behavioral profile is preserved across evaluators.

Related conflict-evaluation work likewise reports reliability with graded or chance-corrected metrics rather than requiring perfect exact agreement: WikiContradict reports Cohen’s κ=0.58–0.88 across prompt templates (Hou et al., 2024), and CONFLICTS validates an automatic evaluator at 0.89 accuracy on 100 human-labeled examples (Cattan et al., 2025). These values are not directly comparable to our ordinal rubric, but they support treating exact integer matching as a strict diagnostic and reporting ordinal tolerance and MAE alongside it.

To further verify this evaluator agreement, we conduct a blinded multi-rater audit. Five independent human raters score 30 benchmark query–response cases spanning all three conflict types and five response models, without seeing model identities or judge scores; each rater scores 10 queries (50 D1–D5 values). We compare each of these five raters with the original human judge, giving 50 comparisons per rubric dimension and 250 comparisons overall. By rubric dimension, the inter-rater agreement (IAA) is:

DimensionnMAEWithin-one
D1: Conflict Perception500.3298.0%
D2: Causal Reasoning500.28100.0%
D3: Confidence Calibration500.6284.0%
D4: Clarification Seeking500.4294.0%
D5: Memory Faithfulness500.8686.0%
All D1–D52500.5092.4%

The direct human–human comparison is MAE=0.50 and within-one=92.4% across 250 comparisons; D4 has within-one agreement of 94.0%. These results show that the reported evaluator agreement is strong under the ordinal rubric and that the behavioral profile is not an artifact of a single judge. We have added this supplementary experiment to §4.1 and a new appendix subsection, Appendix M.2, “Blinded Human Re-rating.”

As for the paper’s main conclusion, we would also like to respectfully clarify that it does not depend on D4 alone. §4.2 and the Conclusion report a multidimensional behavioral profile: conflict perception and causal diagnosis are observed more often than calibrated action and targeted clarification. D4 remains the lowest-scoring dimension for every response model under all three raters, including the human annotator, and this cross-rater ordering is preserved across conflict types.

For SCC, the lower agreement reflects the narrower verification judgment required by D4: the score depends on whether a response requests the specific source-verification fact that could change the recommendation. The SCC-specific MAE=1.36–1.49 and within-one agreement of 54–58% therefore do not overturn the consistent cross-evaluator ordering; together, these results support the consistency of the paper’s multidimensional finding.

Reviewer_3BqNOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how memory-augmented LLM agents act when responding to queries with conflicting evidence. Authors create a benchmark they name TANGLE (Testing Agents Navigation of Genuine, Latent, and Entangled memory conflicts) that consists of 541 scenarios that test context-dependent preferences, user behavior oscillation, and contradictory evidence in their past interactions. Since there is no single ‘correct answer’, the benchmark evaluates behavioral rubrics: whether the agent recognizes the conflict, do they express uncertainty if the evidence is conflicting, etc. Authors evaluate several open-source and proprietary LLMs augmented with Mem0, Letta, A-mem, and MemOS memory, with two human-calibrated LLM judges (gpt5.4, opus 4.7). Among other findings, their experiments suggest that memory frameworks differ significantly in how often they preserve conflict, and that models overall recognize conflict more often than they calibrate their action based on it. Authors also propose CAAP (conflict-aware action policy), a 2-stage prompt that helps the agent respond faithfully when it has conflicting memory evidence.

Critical strengths
  • The paper addresses a genuine and important problem currently not fully covered by existing benchmarks. Modern LLM assistants and agents do often use memory, their memory does collect conflicts, including of the three types (CPC, BOC, SCC) that authors focus on, though there are others out of scope of this work (e.g. hallucinated memories). The analysis in this benchmark can guide future LLMs to better handle conflicting memories, as evidenced by their own proposed prompting policy.

  • The other strength is rigorous experiments: authors test 5 families of LLMs with 4 memory frameworks. The rubrics-based evaluation appears to be the natural way to test behavior under conflicts but depends on the faithfulness of LLM judges. The paper covers this by having two separate LLM judges and calibrating them against a human annotator (Appendix M.1). Some results still exhibit significant disagreement, but, from my experience researching memory-augmented agent behavior, this is an inherent problem with the subject area, not this specific work.

Critical weaknesses

To the best of my knowledge, there is no single glaring weakness, but several relatively minor ones that I took into consideration. Note that these weaknesses are not critical.

Synthetic benchmark. While authors evaluate their judges against a human annotator, there is no guarantee that their scenarios match the distribution of conflicts that occur in real user interactions.

Minor gaps in the experimental setup:

(L296) We evaluate Claude Sonnet 5, Gemini 3.1 Pro, DeepSeek-V3.2, GLM-4.7, and GPT-4o in the oracle and pipeline tracks. Unless stated, models generate responses with temperature 0 and a 4,096-token limit.

To the best of my knowledge, Claude Sonnet 5 does not support temperature, and generation services either ignore the temperature or return an error [1]. Also, most models explicitly recommend a different temperature because greedy inference degrades their accuracy.

Also, to the best of my knowledge, the D6 rubric (defined in appendix only) does not offer clear mapping from labels (safe-partial-unsafe) to numbers reported in Table 17 (app. N), or at least I failed to locate their definition.

On another note, there are several related prior works [2,3, possibly more] that analyze context-based conflict resolution and at least one memory system designed for them [4]. I’d argue it would be nice to better analyze the overlap between these scenarios.

[1] Claude docs for sonnet 5 mention that it does not support temperature, platform.claude.com/docs/en/models/sonnet-5/overview

[2] Shutova, Alina, et al. "Evaluating memory structure in llm agents." arXiv preprint arXiv:2602.11243 (2026). [state transition tasks]

[3] Yoon, Sangyeon, et al. "BenchPreS: A benchmark for context-aware personalized preference selectivity of persistent-memory LLMs." arXiv preprint arXiv:2603.16557 (2026). The entire benchmark.

[4] Gao, Pengyu, et al. "An efficient context-dependent memory framework for llm-centric agents." Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: Industry Track). 2025.

Questions for authors

Based on weaknesses: please clarify the usage of temperature. If you have capacity to reproduce some results using the models’ recommended configuration, can you please report if it affects the model’s ability to handle conflicts?

6 comments
AuthorsOpenReview ↗

# Response to Reviewer 3BqN - Part 1

We sincerely thank the reviewer for the careful and constructive feedback, and we are especially grateful that the reviewer recognizes that TANGLE addresses a genuine and important problem currently not fully covered by existing benchmarks, and that our study is supported by rigorous experiments. We have considered every point and revised the manuscript accordingly, with changes marked in blue. Below, we address the reviewer's concerns point by point.

W1. Synthetic Benchmark We designed TANGLE to approximate the range and structure of conflicts that arise in real personal interactions. Its three conflict families correspond to recurring patterns in personal memory: context-dependent preferences, changing behaviors, and disagreements between sources; we instantiate them across diverse personas, life aspects, and domains in multi-session episodes with natural user requests, conflict-bearing memories spread across sessions, and same-domain distractors.

We chose synthetic, controlled construction because real user traces are difficult to obtain responsibly and to annotate for latent context, phase, or source authority; controlled cases let us create auditable conflict structures, match oracle and pipeline inputs, hold relevant factors fixed, and isolate model behavior. The closest related literature is predominantly synthetic or controlled hybrid: conflict-focused benchmarks such as MemConflict, STALE, and SelectiveQA use controlled synthetic scenarios, while long-term-memory benchmarks such as LoCoMo, LongMemEval, PersonaMem/PersonaMem-v2, and HaluMem also rely on generated or hybrid interactions.

While the controlled synthetic construction is important for the reasons above, we also sought a check that the conflict patterns represented in TANGLE occur in real user interactions. We therefore constructed a 20-case real-trace evaluation subset from the publicly released AlpsBench data, including examples of the same three conflict families: behavior oscillation (e.g., reducing exercise from three sessions to two and then trying to return to four), context-partitioned conflict (e.g., weighing meaningful work against income, funding, and time when choosing among a master's program, a PhD, and employment), and source contradiction (e.g., a dispute over whether a structural-inspection fee should be included in a broader project cost).

Two independent human annotators labeled all 20 cases for conflict presence, conflict type, and underdetermination. Among the 17 cases both annotators marked as conflict-positive, the type distribution was 5 BOC, 9 CPC, and 3 SCC. Conflict-type exact agreement was 90.0%, and underdetermination agreement among these 17 cases was 82.4% (14/17). These results indicate high direct agreement on both the conflict taxonomy and the underdetermination judgment. We have added a new subsection, Appendix A.1, "Real-Trace Sanity Check," with the new material highlighted in blue.

W2. Experimental Setup Our response-generation configuration uses temperature=0.0 and max_tokens=4096 for endpoints that expose temperature. Claude Sonnet 5 does not expose this parameter, so its requests follow the provider-supported behavior. We thank the reviewer for pointing this out and therefore revise the "Setup" subsection in Section 4, "Experiments," to state these settings clearly; the changes are highlighted in blue.

W3. D6 Rubric The intended D6 outcome is the ternary SAFE/PARTIAL/UNSAFE commitment flag. The s=2/3/4 columns in the earlier table come from the Action Quality rubric and are not numeric safety labels. We thank the reviewer for pointing out this ambiguity and therefore remove these legacy columns from the Appendix Q "High-Risk Commitment Results" and the table titled "D6 commitment-appropriateness labels on 62 high-risk SCC instances," reporting only the defined D6 commitment labels; the changes are highlighted in blue.

AuthorsOpenReview ↗

# Response to Reviewer 3BqN - Part 2

W4. How does TANGLE differ from the suggested related work? We thank the reviewer for pointing us to these three relevant works. We read all three papers and add them to the related-work discussion. The additions are highlighted in blue.

WorkMain focusDifference from TANGLE
StructMemEval (Shutova et al., arXiv:2602.11243)Organizing long-term knowledge into structures such as trees, ledgers, state records, and recommendation summaries.StructMemEval mainly asks whether the agent produces the correct answer for a structured-memory task, with analyses of organization failures such as incorrect structures, memory hallucinations, and duplication. Its state-tracking cases reconstruct a designated current state from an ordered history. TANGLE BOC instead models temporal preference drift, where multiple behaviors can remain valid while the phase or trigger needed to select among them is hidden, and uses TANGLE's diagnostic profile to separate evidence observability, conflict reasoning, calibration, information seeking, and memory faithfulness.
BenchPreS (Yoon et al., arXiv:2603.16557)Context-aware preference application and suppression with supplied context and binary preference labels.BenchPreS measures whether each preference should be applied or suppressed, summarized by Misapplication Rate and Appropriate Application Rate. BenchPreS closely resembles our CPC subset, and we explicitly acknowledge the overlap. TANGLE CPC withholds the governing context that would make a single apply/suppress label appropriate; it evaluates the evidence observability, conflict reasoning, calibration, information seeking, and memory faithfulness involved in handling that unresolved setting. TANGLE also includes BOC and SCC, oracle/pipeline tracks, and multiple response behaviors.
CDMem (Gao et al., NAACL HLT 2025)A context-dependent memory architecture for online LLM-centric agents.CDMem evaluates whether its memory architecture improves downstream task success on ALFWorld and ScienceWorld. Its context-dependent indexing retrieves knowledge for task solving, whereas TANGLE treats the applicable context as unresolved and evaluates conflict handling through evidence observability, conflict reasoning, calibration, information seeking, and memory faithfulness. The two are complementary.

The central distinction is the evaluation target: these works primarily assess whether a final answer, preference decision, or downstream task succeeds, whereas TANGLE is a controlled diagnostic benchmark for three underdetermined personal-memory conflict structures. It separates evidence observability, conflict reasoning, calibration, information seeking, and memory faithfulness, and compares complete evidence with retrieved evidence through matched oracle and pipeline tracks. We add the citations and revise "Related Work" accordingly.

AuthorsOpenReview ↗

# Response to Reviewer 3BqN - Part 3

Q1. Does using the models' recommended generation configuration affect conflict handling?

We run a matched DeepSeek-V3.2 comparison on 60 cases (20 each BOC, CPC, and SCC), holding the prompt, memory, query, order, and max_tokens=4096 fixed while changing only temperature from 0.0 to the provider-recommended 1.3. GPT-5.4 scores all generations using the same D1-D5 rubric.

The comparison focuses on two effects: whether the temperature changes the overall behavioral profile, and whether it changes the relative difficulty of the three conflict types. D1-D5 are scored from 0 to 4 and the total from 0 to 20.

Overall behavioral dimensions.

DimensionGPT-5.4 judge; DeepSeek-V3.2 generation t=0.0GPT-5.4 judge; DeepSeek-V3.2 generation t=1.3
D1 Conflict Perception3.0503.017
D2 Causal Reasoning2.6672.650
D3 Confidence Calibration2.1832.000
D4 Clarification Seeking1.4331.617
D5 Memory Faithfulness3.0172.783
Total D1-D512.35012.067

At t=1.3, the D1-D3 scores are modestly lower, indicating less explicit conflict articulation, causal diagnosis, and confidence calibration. Relative to t=0.0, t=1.3 responses more often introduce concrete details absent from the provided memory, which accounts for the D5 reduction. D4 increases modestly, consistent with a higher frequency of clarifying questions, but remains the lowest-scoring dimension overall. The overall behavioral profile is consistent with TANGLE's finding that agents recognize underdetermination more readily than they express calibrated uncertainty or seek targeted clarification.

Conflict-type aggregate scores.

Conflict type (total D1-D5)GPT-5.4 judge; DeepSeek-V3.2 generation t=0.0GPT-5.4 judge; DeepSeek-V3.2 generation t=1.3
CPC (20)16.20015.800
SCC (20)10.30010.450
BOC (20)10.5509.850

Across both temperatures, CPC remains the highest-scoring conflict type, while BOC and SCC remain lower. The conflict-type results are consistent with TANGLE's finding that capability coverage is highest for CPC, whereas BOC and SCC pose greater challenges. We have added a new paragraph in § 4.2 and a new appendix section, Appendix P, "Response-Generation Temperature Sensitivity".

We again sincerely thank you for engaging so carefully with our work and for the constructive suggestions. We sincerely hope that our responses can address your concerns, and, if you find our responses satisfactory, we would be very grateful if you would consider raising your evaluation of our paper.

Reviewer_YnRWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a benchmark to evaluate how LLM Agents handle conflicts in personal long-term memory that cannot be directly resolved. This benchmark consists of 541 composite cases, covering three types of conflicts: context-dependent conflict, behavior recurrent conflict, and information source conflict. The author evaluated the model from five dimensions: conflict recognition, causal reasoning, uncertainty calibration, active clarification, and memory fidelity, and compared its performance under complete memory and real memory retrieval systems. The experiment found that the model is usually able to identify conflicts, but it is difficult to handle uncertainty reasonably and actively seek key information.

Critical strengths
  1. Unlike existing benchmarks that assume a single correct answer, TANGLE focuses on underdetermined memory conflicts where multiple responses may be valid.

  2. The benchmark covers three distinct conflict types and evaluates five complementary behavioral dimensions, providing a systematic framework for analyzing memory-related failures.

  3. The paper evaluates five LLMs and four memory systems, with oracle–pipeline comparisons and additional ablation studies that help identify different sources of failure.

Critical weaknesses
  1. The evaluation relies heavily on LLM-as-a-Judge, with substantial disagreement with human annotations, particularly for clarification seeking (D4). Moreover, the rubric measures prescribed behaviors rather than directly establishing whether an agent's action is appropriate.

  2. CAAP is essentially a two-stage prompting strategy. Its comparison against direct prompting does not control for additional inference computation or provide sufficiently strong prompting baselines, making it difficult to attribute improvements to the proposed policy itself.

  3. The benchmark relies entirely on synthetic memories and dialogues generated using the same LLM, with 40 personas. Without validation on real user interactions, its generalizability remains questionable.

Questions for authors

Please see above.

5 comments
AuthorsOpenReview ↗

# Response to Reviewer YnRW - Part 1

We sincerely thank the reviewer for the detailed and probing feedback, and we are especially grateful that the reviewer recognizes that TANGLE focuses on underdetermined conflicts, provides a systematic framework spanning three conflict types and five behavioral dimensions, and offers broad empirical coverage across five LLM families and four memory systems. We have considered every point and revised the manuscript accordingly, with changes marked in blue. Below, we address the reviewer's concerns point by point.

W1. Evaluator agreement, profile consistency, and action appropriateness. We would like to respectfully point out the agreement evidence reported in Appendix M.1. Across 518 responses, GPT-5.4 and Opus are within one point of the human annotator on 84.4% and 82.1% of all D1--D5 scores, respectively, while the two LLM judges are within one point on 90.5% of scores. More importantly, the relative D1--D5 profile is preserved across evaluators: D4 is the lowest-scoring dimension for every response model under the human annotator and both judges, and this ordering is preserved across conflict types. Thus, human and LLM evaluators agree on the behavioral pattern and on the paper’s reported behavioral finding, while allowing for small differences in the granularity of individual scores.

Related conflict-evaluation work likewise reports reliability with graded or chance-corrected metrics rather than requiring perfect exact agreement: WikiContradict reports Cohen’s κ=0.58–0.88 across prompt templates (Hou et al., 2024), and CONFLICTS validates an automatic evaluator at 0.89 accuracy on 100 human-labeled examples (Cattan et al., 2025). These values are not directly comparable to our ordinal rubric, but they support treating exact integer matching as a strict diagnostic and reporting ordinal tolerance and MAE alongside it.

To further verify this profile-level consistency, we conducted a blinded re-rating by five independent human raters who provide D1--D5 scores for 30 benchmark query--response cases spanning the three conflict types and five response models. Each rater contributes scores for 10 queries (50 dimension scores). Raters score the five dimensions without seeing the model identity or any judge scores. The resulting profiles are computed over the 30 benchmark query--response cases (150 D1--D5 scores per source), with human scores pooled at the case level:

SourceD1: Conflict PerceptionD2: Causal ReasoningD3: Confidence CalibrationD4: Clarification SeekingD5: Memory FaithfulnessTotal
Pooled human ratings3.243.062.441.943.1813.86
Human judge3.333.172.371.873.2714.00
GPT-5.43.803.302.632.233.4715.43
Opus3.573.002.672.433.8715.53

The blinded human re-rating reproduces the paper’s behavioral profile: D4 remains the lowest-scoring dimension, while D1 and D2 remain relatively high. This pattern supports the reported finding across evaluator types that models more often recognize and reason about conflicts than explicitly seek the missing information needed to resolve the conflict. We have added new content to §4.1 and a new appendix subsection, Appendix M.2, "Blinded Human Re-rating".

Action appropriateness. TANGLE constructs conflicts in which the evidence does not determine a unique gold output (y^*), so multiple response strategies can be appropriate. We therefore do not assign a gold-action label, because evaluating overall appropriateness would require a normative policy that specifies how to trade off the user's goal, consequence stakes, reversibility, and clarification cost. The behavior-centered rubric instead evaluates five separable capabilities of conflict-sensitive behavior, allowing comparison across cases with multiple defensible strategies.

AuthorsOpenReview ↗

# Response to Reviewer YnRW - Part 2

W2. CAAP's additional inference computation and prompting baselines. We appreciate the concern, however, we would like to respectfully clarify the role of CAAP in the paper. In the underdetermined memory conflicts studied by TANGLE, the visible evidence can leave the applicable state unresolved. In such cases, the agent should select an evidence-compatible action before realizing its response. We introduce CAAP to make this action-selection step explicit: its decision stage selects the action, and its realization stage expresses that action in the final response. The decision stage is therefore an explicit component of the policy under study, rather than an incidental source of computation.

This design has close precedents in related work. CONFLICTS uses taxonomy-aware prompting to make conflict-sensitive response behavior explicit (Cattan et al., 2025), while Matsnev (2026) separates request uncertainty from action confidence and routes the model to clarification when the request is underspecified. These studies support our characterization of CAAP as a two-stage, evidence-conditioned prompting policy that makes explicit which action the model selects from the visible evidence and allows descriptive analysis of the resulting behavioral trade-offs.

The CAAP--Direct comparisons in the paper are descriptive profile comparisons. They examine the behavioral profiles and trade-offs that emerge when evidence-conditioned action selection is made explicit across conflict types and response backbones; they do not claim causal superiority or universal action utility. We add the corresponding clarification in Subsection 4.4, Heuristic and Policy Baselines, and mark the revision in blue.

W3. Synthetic Benchmark We designed TANGLE to approximate the range and structure of conflicts that arise in real personal interactions. Its three conflict families correspond to recurring patterns in personal memory: context-dependent preferences, changing behaviors, and disagreements between sources; we instantiate them across diverse personas, life aspects, and domains in multi-session episodes with natural user requests, conflict-bearing memories spread across sessions, and same-domain distractors.

We chose synthetic, controlled construction because real user traces are difficult to obtain responsibly and to annotate for latent context, phase, or source authority; controlled cases let us create auditable conflict structures, match oracle and pipeline inputs, hold relevant factors fixed, and isolate model behavior. The closest related literature is predominantly synthetic or controlled hybrid: conflict-focused benchmarks such as MemConflict, STALE, and SelectiveQA use controlled synthetic scenarios, while long-term-memory benchmarks such as LoCoMo, LongMemEval, PersonaMem/PersonaMem-v2, and HaluMem also rely on generated or hybrid interactions.

While the controlled synthetic construction is important for the reasons above, we also sought a check that the conflict patterns represented in TANGLE occur in real user interactions. We therefore constructed a 20-case real-trace evaluation subset from the publicly released AlpsBench data, including examples of the same three conflict families: behavior oscillation (e.g., reducing exercise from three sessions to two and then trying to return to four), context-partitioned conflict (e.g., weighing meaningful work against income, funding, and time when choosing among a master's program, a PhD, and employment), and source contradiction (e.g., a dispute over whether a structural-inspection fee should be included in a broader project cost).

Two independent human annotators labeled all 20 cases for conflict presence, conflict type, and underdetermination. Among the 17 cases both annotators marked as conflict-positive, the type distribution was 5 BOC, 9 CPC, and 3 SCC. Conflict-type exact agreement was 90.0%, and underdetermination agreement among these 17 cases was 82.4% (14/17). These results indicate high direct agreement on both the conflict taxonomy and the underdetermination judgment. We have added a new subsection, Appendix A.1, "Real-Trace Sanity Check," with the new material highlighted in blue.

We again sincerely thank you for taking the time to engage with our work and for the insightful suggestions. We sincerely hope that our responses can address your concerns, and, if you find our responses satisfactory, we would be very grateful if you would consider raising your evaluation of our paper.

We sincerely hope that our responses and revisions have addressed your concerns. We would be very grateful if you could let us know whether the issues you raised have been adequately resolved. If so, we would be deeply grateful if you would consider raising your evaluation of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.