- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces STILL-Bench to evaluate whether embodied agents can recognize when previously observed information becomes outdated. It distinguishes descriptive uncertainty from belief validity and proposes EWM, a calibrated survival model that estimates whether stored beliefs remain valid. Experiments on real and simulated environments show improvements in validity prediction and selective answering, while revealing limitations in cross-dataset generalization.
Critical strengths
-
Clear and meaningful research question. The paper distinguishes descriptive uncertainty from belief validity, highlighting an important limitation of persistent memory in dynamic environments. This provides a well-motivated perspective on when embodied agents should trust previously observed information.
-
Rigorous diagnostic evaluation. The paper introduces a carefully designed benchmark with matched fresh/stale belief pairs and provides systematic experiments on real-world and simulated environments. The findings offer useful insights into belief expiration, particularly the importance of observation context and provenance.
Critical weaknesses
-
Limited novelty over prior work. Predicting scene changes and modeling the reliability of persistent spatial memory have been explored in prior work, such as 3D VSG [1] and HitMem [2]. While the distinction between descriptive and validity uncertainty is useful, the proposed method largely builds on existing survival modeling and calibration techniques, leaving its methodological novelty somewhat limited.
-
Limited cross-domain generalization (major concern). The proposed validity model relies heavily on dataset-specific object-location statistics and performs worse than simple recency baselines when transferred across real-world datasets. This raises concerns about whether EWM learns generalizable principles of belief validity or primarily captures environment-specific patterns, limiting its practical applicability to persistent embodied memory.
[1] Looper et al., 3D VSG: Long-term Semantic Scene Change Prediction through 3D Variable Scene Graphs, ICRA 2023.
[2] Tang et al., HitMem: Hierarchical Temporal 3D Memory with Multi-modal Context-aware Retrieval for Dynamic Environments, ECCV 2026.
Questions for authors
-
What fundamentally distinguishes EWM from prior work on scene change prediction and temporal memory reliability, beyond the use of calibrated survival modeling?
-
Given the poor cross-dataset transfer results, how can the authors justify that EWM captures generalizable belief validity rather than dataset-specific location statistics?