- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies what the authors call tool-memory conflict (TMC): cases where a tool-augmented LLM gives one answer when relying only on its parametric knowledge and a different answer when it is forced to use tools. The authors measure how often such conflicts occur across a set of proprietary and open-source models on math, medical, factual QA and general knowledge benchmarks. They then test whether several existing knowledge-conflict resolution methods can pick the right answer once a conflict is exposed, and probe whether models systematically lean toward memory or toward the tool. The main takeaways are that conflicts are common and vary a lot by model and domain, that conflict frequency and resolution ability are largely unrelated, that giving the model the full tool trace helps more than longer reasoning, and that most models show a mild preference for their own memory.
Critical strengths
- The problem is timely and practically relevant. As more agents call tools autonomously, how a model arbitrates between its own knowledge and tool outputs is a real question, and it has received surprisingly little direct attention.
- The distinction from context-memory conflict in Appendix C is well thought out. The point that tool evidence is produced by the model’s own actions, and therefore carries its own error chain and provenance, is a useful conceptual contribution and deserves a place in the main text.
- The finding that low conflict rate does not imply good conflict resolution is interesting. The observation that the strongest model rarely conflicts but resolves conflicts poorly, largely by sticking with its own answer, raises a real question about self-trust calibration in strong models.
Critical weaknesses
- The text and the tables are inconsistent in many places, to the point where I’m not sure which results are current. Section 5.3 refers to thirteen models, including one that never appears in any table, and nearly all of the bias numbers quoted there differ from Table 4. Several other statements also contradict the tables: the conflict rate given for Claude Opus 4.7 in Section 5.1, and the claim that Full Trace is the best method for every model, which does not hold for GPT-5.5 or Qwen3 8B. Together with the unresolved “Table ??” / “Fig. ??” references, this makes it hard to trust the analysis as written.
- The way conflicts are counted may not match the definition in Eq. 3. On SimpleQA, the “both wrong” share of conflicts is exactly zero for every model, while “both wrong” cases among non-conflicts are not. For free-form answers it is very unlikely that two wrong answers always coincide, which suggests conflicts were determined from correctness labels rather than from whether the two answers actually differ. If so, the conflict statistics need to be recomputed.
- The resolution results lack simple reference points. Since memory-wrong/tool-correct cases make up roughly half of all conflicts, a trivial “always pick the tool answer” rule would land around 50% accuracy, which already matches or beats several of the evaluated methods. Table 5 also shows that Full Trace has a strong tool bias, so part of its advantage may come from deferring to the tool rather than from better reasoning. Always-tool, always-memory and oracle baselines are needed before any claims about method quality can be made.
- The experimental setup doesn’t really test the scenarios that motivate the paper. The introduction emphasizes stale knowledge, code execution and unreliable tools, but the experiments use static benchmarks where many conflicts come from weak tool use (e.g., smaller models losing a lot of accuracy on SimpleQA once they search). This makes TMC easy to read as a tool-reliability problem that better engineering could fix, rather than an arbitration problem. Controlled experiments that vary the tool error rate, or use oracle and corrupted tool outputs, would separate these much more clearly.
Questions for authors
- How are instances where both answers are wrong handled in resolution accuracy? Under Eq. 5 they can never count as appropriate, so the achievable ceiling differs by model. What do the resolution numbers look like with these cases excluded?
- In tool-only mode, how often does the model’s final answer actually differ from what the tool returned? It would help to know how much of a_t reflects tool output versus the model’s own interpretation layered on top. A breakdown of tool-side errors (wrong tool, wrong arguments, wrong result, misreading) would also be informative.
- Do you have any explanation for GPT-5.5’s strong memory bias? For example, is it more likely to override the tool when its memory answer is confident or consistent across samples, and does this hold across domains?