- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(−1)
Read review
Paper summary
This paper studies how an agent should assess whether a stored memory remains valid before receiving contradictory evidence. It formalizes current validity as the probability that an observation was correct when recorded and has not subsequently been superseded. The proposed Transition-Calibrated Memory (TCM) estimates invalidation hazards conditioned on memory type, content, provenance, and available events, and combines them over elapsed time to obtain a validity probability. This estimate supports cost-sensitive decisions about whether to reuse or verify a memory. Experiments on TicToc, PhysioNet-2012, MIMIC-IV, and an exact-event simulator examine calibration, the contribution of transition context, the role of survival factorization, and downstream verification decisions. The results support conditioning on transition context over age-only heuristics, while showing that the additional benefit of survival factorization depends on the supervision regime. The paper also compares TCM with a direct validity predictor, TCM-D, to distinguish these two sources of improvement.
Critical strengths
The memory expiration issue in this paper is significant, which is important for applying this kind of memory system to daily assistant systems. And also I feel that the challenge proposed by the paper is suitable and insightful for solving this kind of problem: Two different memories with the same age may have different kinds of states because of their memory types and different objects. I also think that TCM solves this well; the solution is intuitive.
And I think the paper is well written and coherent, and also it's clear to read. I also like your experimental part. I think the experiments are solid. And in Section 5, they are well organized into different questions and provide enough findings. I think this is very interesting, and it covers the full process across different domains.
I like the problem formulation; it provides coherent ground truth for the paper.
Critical weaknesses
The relationship between the evaluation labels and the formal definition of current validity deserves further examination. TicToc provides human re-query judgments, which may reflect verification preferences as well as beliefs about whether an observation remains valid. The clinical datasets assess agreement at the next measurement, without observing all intervening changes. Although the paper acknowledges these limitations, they constrain how directly the calibration results can be interpreted as estimates of uninterrupted memory validity.
Questions for authors
What evidence supports interpreting TicToc re-query judgments as current-validity probabilities rather than preferences over verification actions?
For the clinical datasets, can the authors assess how departures from the original state followed by a return before the next measurement would affect the interpretation of the reported calibration results?
8 comments
Follow-up on the revised target interpretation
Thank you again for your time and consideration of our work. We hope our response has helped connect the observed labels to the validity question that motivated the paper.
First, we now distinguish uninterrupted state survival from endpoint agreement explicitly. A value can change and later return, so agreement at the next measurement does not imply that it remained valid throughout. To check whether our finding depends on this distinction, we evaluated both targets in the simulator. Context improves prediction over typed-age under both targets in every seed. The advantage therefore does not depend on treating the two labels as interchangeable.
For the real data, we have revised the interpretation accordingly. TicToc supports calibration to observed human reuse judgments, while the clinical labels measure endpoint agreement and can miss a departure followed by a return. The simulator check does not recover those unobserved clinical transitions. This keeps each empirical conclusion tied to what its labels can establish.
We would be happy to discuss any remaining question about how these targets relate to the decision problem.
Thanks for the clear response and the added simulator analyses. Separating human reuse judgments, endpoint agreement and state survival makes the paper more precise, and I appreciate the honest reframing.
That reframing is also what changes my view. The quantity defined in Definition 1, uninterrupted validity, is now measured only in the simulator. TicToc measures human reuse rates, and the clinical cohorts measure agreement at the next observation. So the real-data calibration results no longer speak directly to the paper's core estimand.
In the simulator, where that estimand is observed, the results are less favorable to TCM than I expected:
TCM-D beats TCM in every matched setting and every seed. TCM's worst-group Brier is clearly worse than the typed baselines: 0.305 against 0.171 for typed-age in Table 3, and 0.408 against 0.229 for typed constant decay with exact episodes. The paper is motivated by calibration across memory types, so this matters. The original event features used episode-end information, which an early interval should not see. That was only fixed during the rebuttal, so I am less sure how far the earlier simulator results can be trusted. On the real data, TCM ties the flexible typed-age hazard on TicToc (0.1368 overall), and its worst-group Brier is slightly worse on both TicToc and MIMIC-IV. The clinical decision gains are real, but they target endpoint agreement rather than the validity notion the paper sets out to estimate.
I still think the problem is important and context conditioning is a useful finding. However, the main estimator and the paper's central estimand are less well supported after the revision than I had assumed in my review. I am lowering my score to 2.
Response to Reviewer tPQm
We appreciate your encouraging assessment of the problem formulation and experiments.
Consider glucose that is normal, becomes high, and returns to normal before the next measurement. The clinical label records endpoint agreement, although the original validity episode has ended. Agreement is the relevant target when the loss depends on the value at use. Uninterrupted survival is relevant when continuity itself matters.
What the human judgments support
The votes measure when people would reuse or re-check an observation. We now describe calibration there as calibration to the human reuse rate, one minus the re-query rate. Those votes combine judgments about validity with preferences about verification cost, so they do not identify a latent survival probability. TicToc tests the action-level prediction, while the simulator directly tests survival.
What the simulator lets us check
Exact simulator trajectories let us check whether the context result holds under either target interpretation.
The mapping after Definition 1 now makes this explicit:
We quantified departures followed by returns in the simulator: 0.30% of original probes and 1.9% of observation-window probes in the seed-0 audit (Table 25). These frequencies are specific to that simulator and sampling design; intervening returns in the clinical cohorts remain unobserved.
The revised simulator also distinguishes seamless record renewals from actual state changes, using state survival S′ as its primary label and corrected interval-prefix event features. The comparison is:
The ordering holds in every seed. Context improves prediction under both target interpretations, while the predicted probability has a distinct meaning in each (Figure 6b).
How this affects the clinical result
The clinical decision experiments address their observed endpoint target. At the same 20% verification budget, TCM improves utility by 0.052 and 0.093 on the two cohorts. The revision thus preserves the distinction you ask for while showing that the decision benefit and the simulator context result do not rely on treating these targets as interchangeable.