- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes an expectation-driven memory framework, termed ExpMem, that aims to mitigate hallucinations in LLM agents. In the context of this paper, hallucinations appear when generic priors given by LLMs conflict with the specific rules and operational constraints of real environments in which the LLM agents are deployed. ExpMem builds an environmental memory to encode heterogeneous environmental rules, and performs memory-augmented reasoning prior to give real-time decision-making. The authors verify the effectiveness and efficiency of ExpMem via experiments on ALFWorld and ScienceWorld. They show that ExpMem reduces repeated ineffective decisions while improving task success and interaction efficiency.
Critical strengths
The paper is well written. The idea is clearly presented and easy to follow. The issue to be addressed, namely LLM hallucinations involved in LLM agents, is a crucially important issue for real deployments of LLM agents.
Critical weaknesses
The proposed ExpMem framework is difficult to apply to complex real-world problems because its utility relies on several idealized assumptions:
- The environment is fully observable. For instance, in the authors' example, the state of the wall switch—critical information affecting task progress—is fully observable.
- The agent possesses sufficiently strong reasoning capabilities to instantly grasp causal relationships. Again, using the authors' example, this assumption implies that the agent can recognize that the state of the wall switch plays a decisive role in whether the door lock can be opened.
In the context of complex real-world problems, these assumptions often cannot be met.
Questions for authors
How can the weaknesses listed above be resolved?