acceptodds
Under review as a conference paper at ICLR 2027

Knowing What You No Longer Know: Calibrated Belief Validity for Persistent Perceptual World Memory

Abstract

Persistent world memories let embodied agents answer questions about objects they saw minutes or hours ago. Every such system that reports a confidence reports descriptive uncertainty, how sure it is what an object is, and none reports validity uncertainty, how sure it is that a belief is still true, as a fitted or evaluated quantity. The consequence is systematic and invisible to current benchmarks: asked where an object is after it moved off-camera, models from 14B to 235B answer with the stale location on every such real-kitchen item and never once flag it as stale, at confidently-wrong rates of 1.000 to 0.970 and still 0.879 when told that declining is acceptable, while correctly declining about objects they never saw. We prove that any confidence which is a function of object class and elapsed time is at exactly chance on an elapsed-time-matched pair, and we introduce STILL-Bench, a benchmark with derived epistemic ground truth over real changing kitchens and controlled ones, built from such pairs. Fitting an interval-censored hazard to 9,120 real observation gaps, we find that validity is predictable, at AUROC 0.765 against 0.587 for fitted recency, and that the signal is almost entirely where the object was last seen: expiry runs from 0.029 on a counter to 0.614 in a drawer. A place-conditioned validity model separates stale from fresh beliefs where every published time-based heuristic is provably at chance, at PVD 0.710 (p = 0.029), answers 98.0% of queries at 5% risk or less where recency answers 61.3%, and ranks places well enough to be the best budget-limited re-observation policy we tested on procedurally generated houses. Three cautions follow: descriptive uncertainty does not predict validity and is anti-predictive on pooled pairs; a simulator whose injected changes lack the real place regularity inverts a validity evaluation; and the fitted hazard does not transfer between two real datasets, because a belief formed by seeing an object in an enclosure and one formed by putting it there carry opposite hazards.

Primary area:
datasets and benchmarks
Keywords:
belief validity, epistemic uncertainty, calibration, selective prediction, embodied memory, benchmark, survival analysis, abstention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WYXkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces STILL-Bench to evaluate whether embodied agents can recognize when previously observed information becomes outdated. It distinguishes descriptive uncertainty from belief validity and proposes EWM, a calibrated survival model that estimates whether stored beliefs remain valid. Experiments on real and simulated environments show improvements in validity prediction and selective answering, while revealing limitations in cross-dataset generalization.

Critical strengths
  1. Clear and meaningful research question. The paper distinguishes descriptive uncertainty from belief validity, highlighting an important limitation of persistent memory in dynamic environments. This provides a well-motivated perspective on when embodied agents should trust previously observed information.

  2. Rigorous diagnostic evaluation. The paper introduces a carefully designed benchmark with matched fresh/stale belief pairs and provides systematic experiments on real-world and simulated environments. The findings offer useful insights into belief expiration, particularly the importance of observation context and provenance.

Critical weaknesses
  1. Limited novelty over prior work. Predicting scene changes and modeling the reliability of persistent spatial memory have been explored in prior work, such as 3D VSG [1] and HitMem [2]. While the distinction between descriptive and validity uncertainty is useful, the proposed method largely builds on existing survival modeling and calibration techniques, leaving its methodological novelty somewhat limited.

  2. Limited cross-domain generalization (major concern). The proposed validity model relies heavily on dataset-specific object-location statistics and performs worse than simple recency baselines when transferred across real-world datasets. This raises concerns about whether EWM learns generalizable principles of belief validity or primarily captures environment-specific patterns, limiting its practical applicability to persistent embodied memory.

[1] Looper et al., 3D VSG: Long-term Semantic Scene Change Prediction through 3D Variable Scene Graphs, ICRA 2023.

[2] Tang et al., HitMem: Hierarchical Temporal 3D Memory with Multi-modal Context-aware Retrieval for Dynamic Environments, ECCV 2026.

Questions for authors
  1. What fundamentally distinguishes EWM from prior work on scene change prediction and temporal memory reliability, beyond the use of calibrated survival modeling?

  2. Given the poor cross-dataset transfer results, how can the authors justify that EWM captures generalizable belief validity rather than dataset-specific location statistics?

Reviewer_Q7DUOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is concerned with persistent world memory. It makes the claim that current systems consisting of embodied agents endowed with persistent world memories provide confidence levels indicating how sure it is what an object is, but does not provide similar levels of confidence as to whether that belief still holds (given that some time has passed).

The author(s) mention three contributions in this regard:

  1. A result proving that any confidence value which is a function of object class and elapsed time, is at exactly 50% on an elapsed-time-matched pair.

  2. The introduction of a benchmark (STILL-Bench) with what is referred to as derived epistemic ground truth over real changing kitchens and controlled ones.

  3. What is referred to as three cautions. Perhaps the best way to describe this is as three aspects to take into account when developing such systems.

Critical strengths

The paper considers an important topic that is relevant for the design of any system which instantiates an embodied agent.

Critical weaknesses

Unfortunately I've found the paper really hard to follow, mostly because, in my view, it fails to place matters in context. For someone such as myself, who is an interested outsider in this particular subfield, the lack of context, makes it virtually impossible to judge the novelty and significance of the contributions. As an example, early in the introductory section, there is a reference to "models from 14B to 235B" without any explanation of what this refers to. Similarly, in Proposition 1, which is listed as one of the three contributions of the paper, there are references to important aspects of the results (such as the variables f_FRESH and f_STALE) without any proper indication of what these refer to.

Questions for authors

I don't have any specific technical questions for the author(s) of the paper. This is mostly because, as mentioned above, the paper does not provide sufficient context for an interested outsider (such as myself) to evaluate it properly.

My rating of "Weak rejection" rather than "Clear rejection" is based on the fact that i) I am not an expert in the fields and ii) the paper on its own doesn't provide sufficient information for me to judge its merits.

Reviewer_hbjiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when a robot should trust a remembered object location. STILL-Bench tests answers using HD-EPIC kitchen records and simulated changes in AI2-THOR. EWM learns how reliability varies with time and location, then uses this estimate to answer cautiously or choose where to look again.

Tests on EPIC-KITCHENS-100 examine transfer to another dataset. Results show gains over age alone, but less consistent benefits over simple checking rules.

Critical strengths
  1. The benchmark tests whether systems recognize outdated information. A system can remember a sighting correctly even though the object has moved. Comparing memories of the same object at similar ages helps test this distinction.

  2. Location helps predict whether a memory is still reliable. On kitchens excluded from training, adding location improves the ranking score (AUROC) from 0.587 to 0.705; full EWM reaches 0.765. The benefit also appears outside the balanced pairs.

Critical weaknesses
  1. The benefit over a simple checking rule is limited. In Table 6, checking the place with the most remembered objects nearly matches EWM in one simulator and beats it in the other. It remains unclear whether EWM saves travel or helps complete a user's task.

  2. The advance over earlier work is unclear. Rosen et al. (ICRA 2016) already estimate whether mapped features still exist, while 3D VSG predicts changes to guide inspection. STILL-Bench appears to be the clearest new contribution, but what it teaches beyond those earlier tasks needs stronger support.

Questions for authors

None.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.