acceptodds
Under review as a conference paper at ICLR 2027

Belief Memory: Agent Memory Under Partial Observability

Abstract

LLM agents that operate over long context depend on external memory to accumulate knowledge over time. However, existing methods typically store each observation as a single deterministic conclusion (e.g., inferring "API X failed” from temporary errors), even though such observations are inherently partial and potentially ambiguous. By committing to one conclusion and discarding uncertainty, these methods results in *self-reinforcing error*: the agent acts on the stored conclusion, never revisits alternatives, and reinforces the conclusion over time. To address this issue, we propose *BeliefMem*, which shifts the memory paradigm from committing to a single conclusion per observation to retaining multiple candidate conclusions with their probabilities. Concretely, BeliefMem stores the candidate conclusions as separate memory entries, each carrying a probability that is updated via Noisy-OR rules as new observations arrive. At retrieval, all candidates surface together with their probabilities, keeping alternatives visible to the agent. Since each conclusion in memory retains its probability, BeliefMem preserves the uncertainty that the deterministic paradigm discards, enabling the agent to act with high confidence on well-evidenced knowledge while retaining the capacity to update its confidence when new evidence arrives. Empirical evaluations on LoCoMo and ALFWorld benchmarks show that, even with limited data, BeliefMem achieves the best average performance, remarkably outperforming well-known baselines. More broadly, such probabilistic memory produces substantial gains and explores a new direction for agent memory in partially observable environments.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent memory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tp79OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This project presents BeliefMem, a memory framework for LLM agents operating under partial observability. The authors identify self-reinforcing errors as a limitation of conventional memory systems, where uncertain observations are stored as deterministic conclusions and may subsequently bias agent behavior. To address this issue, BeliefMem maintains multiple candidate conclusions with associated confidence scores and dynamically updates them as new evidence arrives. Experiments on LoCoMo and ALFWorld demonstrate improvements over several existing memory baselines.

Critical strengths

Important and well-motivated problem. The paper provides a meaningful discussion of self-reinforcing errors in agent memory. Understanding how incorrect experiences persist and influence future decisions is an important direction for self-evolving agents.

Promising methodological design. Maintaining multiple candidate conclusions with explicit confidence scores, rather than committing to a single interpretation, is a reasonable and potentially useful approach. The dynamic confidence update mechanism adds some methodological novelty.

Encouraging empirical results. The method achieves competitive improvements across the evaluated benchmarks, suggesting that preserving uncertainty can benefit long-term memory utilization.

Critical weaknesses

Misalignment between evaluation and core motivation. LoCoMo primarily evaluates long-term conversational memory and question answering, rather than agent self-evolution or self-reinforcing errors. It lacks the closed-loop interaction required to investigate how incorrect memories influence subsequent actions and observations. Although ALFWorld provides an interactive setting, the experiments do not sufficiently establish the claimed self-reinforcing error mechanism.

Insufficient analysis of noise from competing hypotheses. Preserving multiple candidate conclusions may introduce conflicting or misleading information during retrieval, potentially degrading downstream decisions. While the paper examines retrieval size, it does not systematically investigate how the number of conflicting candidates or their confidence distributions affect task performance. More targeted ablations are necessary.

Incomplete discussion of related work. The paper would benefit from a more comprehensive comparison with recent studies on persistent memory corruption, poisoned experience retrieval, and error propagation in self-evolving agents. The distinction between BeliefMem and these approaches should be clarified.

Unclear recovery mechanism under incorrect high-confidence beliefs. When an initially incorrect belief already dominates an agent's actions, the agent may avoid behaviors that could generate contradictory evidence. BeliefMem primarily updates confidence after receiving new observations, but does not explain how corrective evidence can be actively obtained in such situations.

Questions for authors

Could the authors provide a closed-loop evaluation demonstrating how incorrect memories influence future actions, reinforce themselves, and are eventually corrected by BeliefMem?

How does performance change when varying the number of distracting hypotheses and assigning incorrect or closely competing confidence scores?

How does BeliefMem differ from existing work on persistent memory poisoning and self-reinforcing agent errors?

If an incorrect high-confidence belief prevents the agent from exploring alternatives, what mechanism enables BeliefMem to escape this feedback loop?

Reviewer_nZa7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that existing agent memory methods store a single conclusion for each observation, which can lead to self-reinforcing errors. To address this, the authors propose BeliefMem. It keeps several candidate conclusions for each attribute. Each candidate has a score, which is updated using a Noisy-OR rule as new observations arrive. At retrieval, all candidates are returned together with their scores. The method is tested on LoCoMo and ALFWorld. The paper also includes ablation studies, experiments with varying memory corpus sizes, an analysis of belief convergence, and adversarial experiments.

Critical strengths
  1. The paper addresses a real problem in agent memory. Storing only one conclusion can make an early mistake hard to correct, and the API example in Figure 1 makes this motivation easy to follow.

  2. The method is simple and needs no training. The experiments are comprehensive, covering both a conversational benchmark LoCoMo and an interactive benchmark ALFWorld. The authors also do ablation studies and several further analyses.

Critical weaknesses

Overall: it feels more like an output of automated research. Some symbols are defined twice, some appear without any definition, and several equations do not match the text. The experimental results also have problems.

  1. The notation is inconsistent in several places. First, the formula above eq. 3 is written as a_t ∼ π(· | o_t, z_t), but z_t is never defined. Second, the letter M is used for two different things. M_t in Eq. 2 denotes the external memory, while M_c in H(c) = {h_1^(c), ..., h_{M_c}^(c)} denotes the number of hypotheses. Third, Eq. 3 updates the memory as Update(M_t, o_t, o_{t+1}) without a_t. However, the paragraph on self-reinforcing error says that the transition (o_t, a_t, o_{t+1}) is written back to memory via Eq. 3. Fourth, in the paragraph on belief update, the authors keep scores unnormalised so that the score of a supported h is not affected by the number of alternative candidates for the same c. However, in Merge, a competing candidate is reset to 1/|H_sub(c)|, which depends directly on the number. Moreover, the retrieval score α_t(c) in Eq. 9 depends only on semantic similarity and time decay but p is not used. So retrieval ranking is not belief-aware.

  2. The ALFWorld success rates contain values that cannot be obtained from the stated number of episodes. SR is the number of solved episodes divided by the total number and times 100. Appendix C.1 states that the Seen set has 140 episodes and the Unseen set has 134. So the number of solved episodes equals 1.4×SR on Seen and 1.34×SR on Unseen, and this must be an integer. In Table 5, w/o Add has a Seen SR of 19.87. It gives 19.87×1.4=27.82 solved episodes. However, 27 solved episodes give 19.29% and 28 give 20.00%, so 19.87% is not possible.The same problem appears in 25.29, 42.32, 27.65 and 29.76 of Table 5, and in the 53.75 for Table 2. Moreover, B.3 reports only one seed 42, and the paper does not mention averaging over multiple runs. These values therefore cannot come from the reported setting. Furthermore, in Table 8, w/o decay and λ = 1.0 should be identical according to Eq. 9. However, the authors report different results.

Questions for authors
  1. How were the success rates in Table 2 and Table 5 computed? Why do most values correspond to an integer number of solved episodes while some do not?

  2. In Table 8, how does w/o decay differ from λ = 1.0 in the implementation? According to Eq. 9, the two settings should be identical.

3.Which reset value is used for a competing candidate in Merge? Section 3.3 states 1/|H_sub(c)|, while Appendix B.2 states 0.25.

  1. Since the retrieval score in Eq. 9 does not use the candidate scores p, does the “w/o belief-aware retrieval” ablation only remove the scores from the retrieved text given to the agent?
Reviewer_Wh7dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes BeliefMem, a memory framework for LLM agents operating under partial observability. Instead of storing a single deterministic conclusion, BeliefMem maintains multiple candidate conclusions with confidence scores for each memory attribute. These scores are updated using Noisy-OR evidence aggregation, and the candidate conclusions are retrieved together to support decision-making under uncertainty. The authors evaluate BeliefMem on LoCoMo and ALFWorld, demonstrating improvements over existing memory methods.

Critical strengths

The paper addresses a meaningful challenge in agent memory: storing deterministic conclusions from incomplete or noisy observations can lead to persistent errors. And the method is interesting to maintain different conclusions with explicit confidence scores and supports incremental memory updates.

Critical weaknesses
  1. Storing multiple conclusions with confidence scores can improve performance. However, BeliefMem relies on uncalibrated LLM confidence scores and heuristic Noisy-OR updates. It is unclear whether these updates reliably represent uncertainty, particularly when observations are correlated or contradictory.
  2. BeliefMem is motivated by interactive agent settings, where memory influences future actions and observations. However, the evaluation primarily focuses on conversational QA, with ALFWorld as the only interactive agent benchmark. This provides limited evidence of its effectiveness in broader agentic settings.
Questions for authors
  1. Could the authors evaluate the calibration and robustness of belief scores, particularly under correlated or contradictory observations?
  2. Could the authors evaluate BeliefMem in more complex online agent settings where memory influences the agent's actions and subsequent observations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.