- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes MemUQ, a memory-centric uncertainty quantification framework for LLM agents in multi-step interactions. MemUQ represents accumulated agent memory as atomic claims, aggregates claims across independently sampled trajectories, and constructs a conflict graph to capture inconsistencies in retrieved evidence. Claim-level extrinsic uncertainty is quantified by combining graph centrality with frequency-based support and is further combined with intrinsic generation uncertainty. Experiments on HotpotQA and StrategyQA across three LLMs demonstrate consistent improvements over existing single-step and agentic UQ baselines.
Critical strengths
-
The paper identifies an important limitation of existing UQ methods in agentic settings and provides a well-motivated solution. Instead of quantifying uncertainty only from final outputs or individual reasoning steps, MemUQ explicitly models inconsistencies in the accumulated evidence stored in agent memory. This provides a meaningful perspective on uncertainty propagation in multi-step agent interactions.
-
The proposed claim-level formulation is interpretable and empirically effective. Decomposing memory into atomic claims and combining conflict-graph centrality with frequency-based support provides fine-grained uncertainty signals while maintaining a clear interpretation. The experiments consistently demonstrate improvements across different models, datasets, and uncertainty baselines, with additional analyses validating the individual components and the reliability of the claim-processing pipeline.
Critical weaknesses
-
The current evaluation is still limited in terms of agent tasks and interaction environments. The experiments focus on two multi-hop QA benchmarks, HotpotQA and StrategyQA, both of which are primarily retrieval/reasoning tasks. Since the paper motivates MemUQ as a general framework for uncertainty quantification in LLM agents, evaluation on more diverse agentic settings, such as longer-horizon planning or heterogeneous tool-use tasks, would provide stronger evidence for its generality.
-
MemUQ remains dependent on LLM-based semantic processing. Claim decomposition, deduplication, and conflict detection are themselves performed by an LLM. Although the paper provides useful validation of this pipeline, errors or biases in these semantic judgments may directly affect the constructed conflict graph and consequently the estimated uncertainty, particularly in domains with more ambiguous or specialized evidence.
-
The related-work discussion could be further expanded to cover recent approaches that explicitly model different sources of uncertainty. In addition to sampling-, semantic-, and representation-based UQ methods currently discussed in the paper, several recent studies have investigated complementary formulations of epistemic and evidential uncertainty, e.g., Detecting Misbehaviors of Large Vision-Language Models by Evidential Uncertainty Quantification (ICLR 2026), Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification (ICLR 2026), and Epistemic Uncertainty Quantification To Improve Decisions From Black-Box Models (ICLR 2026). Discussing these works and their relationship to the proposed intrinsic/extrinsic uncertainty decomposition would provide a more complete positioning of MemUQ within the recent UQ literature.
Questions for authors
See the weaknesses.
3 comments
Rebuttal Response to Reviewer UroQ (2/2)
MemUQ remains dependent on LLM-based semantic processing...
We agree that the reliability of the claim pipeline is essential, since its errors could in principle be mistaken for agent uncertainty. We address this from three aspects.
Each pipeline stage is validated. We evaluate decomposition, deduplication, and conflict detection separately (Appendix G):
| Stage | Validation | Result |
|---|---|---|
| Decomposition | Self-contained claims | 95.7% |
| Decomposition | Information loss from previous memory | 0% |
| Decomposition | Claims contradicting the input | 0.7% |
| Deduplication + conflict detection | Agreement with human labels (pipeline / GPT-5.5) | 80% / 80% |
| Deduplication | Duplicate pairs not merged (NLI audit, all pairs, Gemma3-27B, StrategyQA) | 0.91% (1,704 / 187,805) |
| Conflict detection | Conflict edges not confirmed as contradictions (NLI audit, all edges) | 1.88% (119 / 6,343) |
The pipeline, run with the same backbone as the agent (Gemma3-27B), agrees with human judgment at the same rate as a frontier model. Over the full claim set, fewer than 1% of claim pairs are duplicates that the pipeline failed to merge, and fewer than 2% of conflict edges are not confirmed as contradictions by an NLI model.
The score limits the influence of pipeline errors. By Proposition 3.1, the conflict graph does not inject uncertainty of its own. It only reweights the frequency evidence produced by the agent through the covariance term . In particular, when the agent's support is uniform across claims, this covariance is zero for any conflict graph, so a spurious edge cannot change the score.
The pipeline is modular. Decomposition, deduplication, and conflict detection are independent of the uncertainty computation. Each stage can be replaced by a stronger or domain-specific model, e.g., an NLI model or a specialized LLM judge, without changing the method. In the additional AgentBench-OS evaluation, evidence consists of shell outputs rather than Wikipedia text, and MemUQ maintains its advantage there. This further indicates that the pipeline transfers beyond the QA setting.
The related-work discussion could be further expanded to cover recent approaches that explicitly model different sources of uncertainty...
We appreciate the literature suggestions. These works refine the uncertainty of a model's own predictions through evidential, epistemic, or cross-model formulations, which corresponds to the intrinsic component of MemUQ. Since the intrinsic estimator is modular (we already combine the extrinsic score with five intrinsic methods, Appendix I), they can be adopted without modifying the extrinsic score. MemUQ's extrinsic component instead targets a source specific to agentic settings: uncertainty from external evidence accumulated over multi-step interactions.
We have added these works to Section 4 and discussed their relationship to the intrinsic/extrinsic decomposition.
Thank you again for the insightful comments and we have incorporated your suggestions in the revised version of the manuscript, highlighted in blue. If there are any additional questions or concerns, we would be happy to address them.
Should you find that our responses sufficiently address your feedback, we would sincerely appreciate your consideration of raising the score, if deemed appropriate.
I appreciate the authors’ response and clarifications, which have addressed some of my concerns. However, my overall assessment remains unchanged, and I will maintain my original rating.
Rebuttal Response to Reviewer UroQ (1/2)
We sincerely thank the reviewer for the positive assessment and for recognizing that modeling inconsistencies in the accumulated evidence gives a meaningful view of uncertainty propagation in multi-step agents. We address each point below.
To evaluate MemUQ beyond retrieval and reasoning, we add the OS Interaction task of AgentBench (Liu et al., 2024). In this task, the agent operates in an interactive Ubuntu environment through multi-turn shell commands, its actions change the environment state, and many tasks require executing operations rather than answering a question. We evaluate Qwen3-30B and Gemma3-27B with the same setup as the main experiments (K=10 trajectories, 10 samples per step).
MemUQ achieves the best performance with every centrality metric on both models. With the default PageRank centrality, it improves over the baselines by 19.0% (Qwen3-30B) and 13.7% (Gemma3-27B) on average in AUROC, and over the agentic baseline UProp by 19.1% and 23.1% in AUROC. Applying single-step UQ methods to paragraph-style memory (Mem_Ext) falls below the corresponding trajectory-level baseline in four of five cases for each model. The gains therefore stem from the claim-level decomposition and conflict graph rather than from the memory representation alone.
These results indicate that MemUQ generalizes from retrieval-based QA to interactive tool use with changing environment state. We agree that longer-horizon planning tasks would further strengthen the evidence, and we consider them an important direction for future work.
We have added the AgentBench-OS results and setup details to the revised paper (Section 5, Table 2).