- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies how the LLM infers the user's state under personal scenarios. The author uses evidence from Personamem-v2 and represents each state by type. The performance of Qwen3-4B reaches 19.81% and is raised to 65.28% by SFT.
Critical strengths
-
The paper distinguishes the retrieval and interpretation phases, which makes the retrieval error easy to identify.
-
The ablation study is well-designed and only changes user states in the experiments, which makes comparison clearer.
-
The paper reports the confidence interval at the user level, which can deal with the relevance with the same personal sample.
Critical weaknesses
-
The experiment part is weak. The author only uses one dataset and doesn't compare with other memory baselines. I think this paper is below the ICLR bar.
-
Only a weak 4B model is tested under the zero-shot setting. The experiments can not show the real problem of LLM.
-
The paper says retrieval is not enough, but no experiment uses retrieval phase. Moreover, the paper also does not report how many errors come from the retrieval stage.
-
The main motivation is model cannot understant user, but the results cannot support this view.
Questions for authors
-
Could the authors add experiments on more datasets and compare with more memory baselines?
-
Could the authors test stronger LLMs and also report their results?
-
Could the authors add experiments with retrieval?