- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces TANGLE, a synthetic benchmark of 541 personal-memory conflicts across three types where the query withholds the information needed to resolve the conflict.
Critical strengths
The problem is well chosen: conflicts where no record is wrong and no single answer exists are common in long-term memory and under-studied compared to stale-fact updates.
Critical weaknesses
- No negative class, so the benchmark can’t separate “recognizes underdetermination” from “always hedges.” All 541 instances are conflicts by construction, and D3/D4 reward expressing uncertainty and asking. Your own Always-ask baseline (Fig. 4) scores higher than every real model on D3, D4, and D5. The headline finding—models perceive conflict but don’t calibrate or clarify reads just as well as “models don’t hedge unconditionally,” which may be correct behavior.
- Underdetermination is assumed, not validated. The generator was instructed to make conflicts unresolvable, but no human ever checked whether a reasonable person actually finds them unresolvable; Appendix L says the review is “not a human gold label for the best action.” If some instances are resolvable with ordinary judgment, committing is right and gets penalized on D3.
- The data is 100% single-model synthetic with no quantified human review. Everything—memories, distractors, queries, dialogues—comes from Claude Sonnet 4.6. The human review covers only coherence and non-leakage, with no reviewer count, sample size, rejection rate, or agreement reported. Memory items are stylistically uniform (18–32-word third-person statements), so D1 may partly reflect surface-pattern detection.
- Scoring rests on one human annotator and weak judge–human agreement. With a single annotator there is no human IAA. Judge–human exact agreement is 46–55% on a 0–4 scale, and D4—the dimension the main conclusion depends on is the worst: on SCC, MAE 1.36–1.49 and within-one agreement only 54–58%.
- Same-family loop. Sonnet 4.6 generates, Opus 4.7 judges, Sonnet 5 wins. CAAP is a prompt that tells the model to pick from the actions the rubric rewards, scored by the same rubric, with no cost for asking so its D4 gains are expected and don’t show it beats Always-ask, which already outscores it.
Questions for authors
please see above
7 comments
# Response to Reviewer yRox - Part 2
W2. Underdetermination validation and rubric interpretation.
During generation, we have manually checked that the intended conflict for each conflict type is present and that the query omits the context, time, or source authority needed to determine which record applies. To further validate underdetermination, we conduct an independent audit in which two human annotators assess a 10-case subset spanning all three conflict types, yielding 20 case-level judgments. The 20 judgments are positive (100%) for retaining an unresolved decision-relevant variable. This audit validates the intended underdetermination. We have added the underdetermination validation details to §3.3 and Appendix L, “Generation, Session Synthesis, and Quality Control,” with the added text highlighted in blue.
Appendix L uses the phrase “not a human gold label for the best action” to define the scope of the construction review. The review checks narrative coherence, non-leakage, and whether the target conflict remains reconstructable only from the aggregate evidence; it does not assign a unique correct response. This follows the paper's rubric design: underdetermined conflicts do not have a single licensed answer, and multiple evidence-compatible strategies can be reasonable.
Within this scope, we would like to respectfully clarify that when the available evidence supports resolution through ordinary judgment, a corresponding commitment is appropriate and is not penalized by D3; what D3 penalizes is unsupported over-commitment.
W3. Quantified human validation and memory style.
We used a gated, iterative construction protocol. We first generated a 30-instance pilot, balanced across the three conflict types (10 BOC, 10 CPC, and 10 SCC). Two reviewers manually checked coherence, non-leakage, and whether each target conflict remained reconstructable only from the aggregate evidence; issues were revised and re-reviewed until the pilot was jointly acceptable. The schema, coverage, conflict-structure, query, and distractor checks followed the deterministic and model-based quality-control procedures described in Appendix L.
Once the pilot was jointly accepted, we scaled the validated construction procedure to the full benchmark. The full-generation pool contained 550 instances; we filtered out 9 instances (1.6%) whose conflicts mixed axes and were not sufficiently clear, leaving 541 retained instances. We then conducted a final stratified spot check of 30 retained instances (10 per conflict type), and all 30 passed (100%; 0 rejected). We have added these details to §3.3 and Appendix L.
As for memory style, across the 541 instances, 4,893 core memory items were analyzed: 3,647 (74.5%) contain 18–32 words, lengths range from 8–83 words (median 29), and 52 items contain first-person pronouns while 17 contain second-person pronouns. We describe them precisely in §3.3, “Data Construction,” with the revision highlighted in blue.
# Response to Reviewer yRox - Part 3
W4. Evaluator agreement and robustness of the behavioral profile.
We would like to respectfully clarify that judge–human agreement is not weak when evaluated using the ordinal structure of the 0–4 rubric. On the 518-response cohort, GPT-5.4 and Opus achieve 84.4% and 82.1% within-one agreement with the human annotator, and the two independent LLM judges agree within one point on 90.5% of scores. Exact matching requires identical integer scores on a 0–4 scale and is therefore a particularly strict criterion; the high within-one agreement provides the appropriate reliability evidence. We report both judges separately, and the same behavioral profile is preserved across evaluators.
Related conflict-evaluation work likewise reports reliability with graded or chance-corrected metrics rather than requiring perfect exact agreement: WikiContradict reports Cohen’s κ=0.58–0.88 across prompt templates (Hou et al., 2024), and CONFLICTS validates an automatic evaluator at 0.89 accuracy on 100 human-labeled examples (Cattan et al., 2025). These values are not directly comparable to our ordinal rubric, but they support treating exact integer matching as a strict diagnostic and reporting ordinal tolerance and MAE alongside it.
To further verify this evaluator agreement, we conduct a blinded multi-rater audit. Five independent human raters score 30 benchmark query–response cases spanning all three conflict types and five response models, without seeing model identities or judge scores; each rater scores 10 queries (50 D1–D5 values). We compare each of these five raters with the original human judge, giving 50 comparisons per rubric dimension and 250 comparisons overall. By rubric dimension, the inter-rater agreement (IAA) is:
| Dimension | n | MAE | Within-one |
|---|---|---|---|
| D1: Conflict Perception | 50 | 0.32 | 98.0% |
| D2: Causal Reasoning | 50 | 0.28 | 100.0% |
| D3: Confidence Calibration | 50 | 0.62 | 84.0% |
| D4: Clarification Seeking | 50 | 0.42 | 94.0% |
| D5: Memory Faithfulness | 50 | 0.86 | 86.0% |
| All D1–D5 | 250 | 0.50 | 92.4% |
The direct human–human comparison is MAE=0.50 and within-one=92.4% across 250 comparisons; D4 has within-one agreement of 94.0%. These results show that the reported evaluator agreement is strong under the ordinal rubric and that the behavioral profile is not an artifact of a single judge. We have added this supplementary experiment to §4.1 and a new appendix subsection, Appendix M.2, “Blinded Human Re-rating.”
As for the paper’s main conclusion, we would also like to respectfully clarify that it does not depend on D4 alone. §4.2 and the Conclusion report a multidimensional behavioral profile: conflict perception and causal diagnosis are observed more often than calibrated action and targeted clarification. D4 remains the lowest-scoring dimension for every response model under all three raters, including the human annotator, and this cross-rater ordering is preserved across conflict types.
For SCC, the lower agreement reflects the narrower verification judgment required by D4: the score depends on whether a response requests the specific source-verification fact that could change the recommendation. The SCC-specific MAE=1.36–1.49 and within-one agreement of 54–58% therefore do not overturn the consistent cross-evaluator ordering; together, these results support the consistency of the paper’s multidimensional finding.
# Response to Reviewer yRox - Part 1
We thank the reviewer for the careful and probing feedback. We are grateful that the reviewer recognizes TANGLE's focus on an important and underexplored class of underdetermined long-term-memory conflicts. In response to the reviewer’s concerns, we would like to provide the necessary clarifications and explain why they do not undermine our claim. We have considered every point and revised the manuscript accordingly, with changes marked in blue. We address the reviewer’s concerns point by point below.
W1. Negative class and evidence-sensitive action selection.
To test whether the observed caution is sensitive to the evidence state rather than a result of indiscriminate hedging, we added 60 matched controls for 30 original cases to the revision—10 cases from each conflict type (BOC, CPC, and SCC)—yielding 90 inputs in total across the unresolved, resolved, and no-conflict conditions. The resolved controls supply the missing context, phase, or source authority, while the no-conflict controls remove or rewrite one side of the conflict. Both controls preserve the persona, domain, query form, memory bank, and distractors, including their order and count. The variants are constructed with Claude Sonnet 4.6. As a construction-quality check, two independent annotators review six sampled controls each. All 12 judgments confirm that the intended edit succeeds (100%).
We evaluate the five response models used in the paper—Claude Sonnet 5, Gemini 3.1 Pro, GPT-4o, DeepSeek-V3.2, and GLM-4.7—on the 60 control inputs, yielding 300 responses. GPT-5.4 scores D3 and D4 using the paper's original rubric. The table reports the mean scores by condition; D3 measures confidence calibration and D4 measures decision-relevant clarification seeking.
Across all five models, moving from the unresolved condition to the resolved controls raises D3 from 2.03–2.63 to 3.37–3.80 and lowers D4 from 1.37–2.23 to 0.00–0.17. The resolved controls therefore show high calibration and near-zero clarification seeking when the missing context, phase, or source authority is supplied. No-conflict controls show the same pattern, with D3 scores of 2.93–3.67 and D4 scores of 0.33–0.97.
In resolved and no-conflict cases, a low D4 is the appropriate behavior because no unresolved variable remains for a question to adjudicate; D4 should be high when targeted clarification can resolve an underdetermined conflict. Thus, the matched benchmark conditions separate recognizing underdetermination from always hedging, and the D3–D4 rubric distinguishes them accordingly: it rewards calibrated confidence and direct response when the evidence is sufficient, while assessing whether the model expresses calibrated uncertainty and seeks decision-relevant information when the evidence remains underdetermined. We have added new content to §4.2 and a new appendix subsection, Appendix N, “Evidence-Sufficiency Controls.”
As for the headline finding, we would like to respectfully clarify that our paper does not claim that caution or clarification is always preferable, nor that every commitment is inappropriate. Rather, it reports that models often identify and describe the conflict, but less consistently calibrate their commitment or request the specific missing information that could change the recommendation. The negative-class results provide the corresponding evidence-sufficient contrast: the same models receive high D3 and near-zero D4 when the evidence is sufficient. This supports interpreting the observed calibration and clarification gap as a behavior profile specific to underdetermined conflict settings.