acceptodds
Under review as a conference paper at ICLR 2027

When Will a Model's Knowledge Expire?

Abstract

A language model may answer a factual query correctly at deployment and later become wrong even though its parameters have not changed, simply because the world has. We study how long such currently correct knowledge can be expected to remain valid. We cast this as a survival problem in calendar time, with left truncation because many facts have already been valid for some time at the evaluation date, interval censoring when the change time is known only within a window, and right censoring when no change is observed. We introduce WIKILIFETIME, a rolling-origin benchmark containing only facts that are both valid in the world and recoverable from a point-in-time language model at the anchor date. We propose EXPIRE (EXpiration Prediction from Internal Representations and Event histories), which combines relation-specific duration patterns, entity-level histories, frozen hidden states, and calendar drift in a low-rank time-varying hazard model. The analysis separates model fitting, tuning, recalibration, retrieval-risk certification, and final testing chronologically. We derive estimation and calibration guarantees, characterize the limits of relation-recency rules, and obtain finite-sample control of missed stale answers for retrieval decisions. Numerical studies show that EXPIRE improves remaining-lifetime prediction, within-relation discrimination, and calibration, while reducing missed stale answers in retrieval decisions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Knowledge staleness, Large language models, Retrieval-augmented generation, Survival analysis, Temporal factuality

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_R8a3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how long a fact will remain valid after a frozen language model can answer it correctly. It defines this as a remaining-lifetime prediction problem with interval and right censoring. The paper introduces WIKILIFETIME and proposes EXPIRE. The model combines relation-level patterns, fact age, event history, calendar effects, and frozen LM representations. The authors evaluate it through simulations, historical forecasting, and retrieval decisions. They also give theoretical results for estimation and risk control under stated assumptions.

Critical strengths
  1. The problem is meaningful and the survival formulation fits it well. A model can keep giving the same answer while the world changes. Predicting this risk before a change is observed has clear value for deciding when to retrieve fresh evidence. The method also uses the observed event intervals directly. This avoids treating uncertain change dates as exact dates.

  2. The representation experiments address an important alternative explanation. The authors do more than compare EXPIRE with a relation-only baseline. They use within-relation matching, representation permutation, and relation-centroid controls. In Table 2, adding the representation improves NLL from .352 to .331 and matched-pair accuracy from .636 to .699. These results support an instance-level signal beyond relation identity. This is the strongest empirical reason to consider acceptance, although it still needs confirmation under a corrected time setup.

Critical weaknesses

1. The chronological split does not establish that development labels were available before deployment.

Table C.4 places validation origins in 2021, recalibration in January 2022, certification in July 2022, and testing in January and July 2023. Page 32 says earlier blocks also use up to 24 months of follow-up. The final hazard model is refitted on both training and validation data.

This creates a timing problem. Full 24-month outcomes for July 2021 validation records are not available until July 2023. January 2022 recalibration records need follow-up until January 2024. Even the full 12-month certification outcomes from July 2022 are not available before the January 2023 test date.

Future follow-up is needed to evaluate test predictions. My concern is about using future outcomes to fit the model, recalibrator, or retrieval threshold before those test predictions are made. Restricting input features to the anchor date and keeping subjects separate do not solve this issue.

Please state the latest outcome date used by every stage at each deployment date. Either leave enough time for the required outcomes to become available, or censor development records at the deployment date. Then rerun fitting, selection, recalibration, certification, and testing. Without this check, the results support an offline historical study more clearly than a strict prospective deployment claim.

2. The main censoring-adjusted scores need stronger justification.

Appendix E.1 estimates unresolved outcomes using a reference model based on relation, anchor quarter, and age group. The candidate predictors use richer history and semantic features. The reference model may therefore miss outcome information that is present in the candidate prediction.

This matters for the reported Brier scores and calibration. Suppose the reference model assigns a survival probability of .5 to an unresolved record. A candidate that correctly predicts its true status can receive a worse pseudo-Brier score than a constant prediction of .5. Cross-fitting reduces overfitting, but it does not remove this missing-information problem.

The test missed-stale estimate in Appendix E.3 uses the same pseudo-outcomes. Its interpretation also needs checking. This concern applies to unresolved outcomes. Observed-data NLL, matched pairs with known status, and fully observed horizon scores still provide useful evidence.

Please give the conditions under which these scores estimate the intended prediction risk. Report the unresolved fraction at each horizon. Also check the conclusions with a justified censoring-adjusted estimator that accounts for the relevant predictors. A fully resolved subset would be useful as an additional sensitivity check, with its selection limits clearly stated.

Questions for authors
  1. Label availability. At the January 2023 test date, which validation, recalibration, and certification outcomes were actually available? Was follow-up cut off at that date? If such a rule was used, please show it and explain how it fits the 24-month follow-up description. If not, please provide results after correcting the timing.

  2. Score validity. What assumptions allow the reference posterior based on relation, quarter, and age group to evaluate predictions that also use history and LM states? How many records have unresolved status at each reported horizon? Do the main comparisons hold under an evaluation that addresses this issue?

  3. Scope of the risk guarantee. Certification uses only records with fully resolved outcomes. Why should this selected group match the full deployment population? I appreciate that the paper already limits Theorem 4.6 to exchangeable data and does not claim protection under arbitrary calendar drift. Please also clarify the effect of selecting only resolved records, and keep the theorem’s expected-risk guarantee separate from the later-year test results.

  4. Event-interval filter. Pages 28 and 32 state that intervals wider than 180 days are excluded from the primary analysis. Table C.6 still reports 513 events with intervals of 7–12 months. Which rule was used? Please correct the counts and confirm that all main tables use the same primary cohort.

  5. Model source and reproducibility. What are the exact sources and versions of the 7B checkpoints in Table C.2? The cited Kelly et al. paper describes 1.5B and 4B models, so the mapping is unclear. Please explain whether the 7B models were separately trained or came from another source. An anonymous manifest with model identifiers, hashes, data cutoffs, relation IDs, and schema filters would help verify the benchmark. See the cited point-in-time model paper.

Reviewer_ntZBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks how long knowledge that a model can currently recall correctly will remain valid, and frames this as a remaining-lifetime prediction problem. The authors introduce the WikiLifetime benchmark and propose EXPIRE, which combines fact duration, entity history, internal model representations, and calendar-time changes to predict when facts may expire and guide retrieval decisions.

Critical strengths
  1. EXPIRE distinguishes between facts a model never knew and facts it knows that become outdated as the world changes. Moving beyond current correctness to future validity is useful for knowledge maintenance and deciding when to retrieve external information.

  2. The method accounts for how long a fact has already been valid and directly handles interval and right censoring, avoiding simple imputation of expiration times. Combining relation-level patterns, historical information, and semantic features also makes sense.

  3. The same-relation matching, within-relation representation shuffling, and history-only baselines help rule out the possibility that the model is simply identifying which relations tend to change more often.

  4. The paper examines the approach through experiments and theoretical analysis from several angles, providing fairly broad coverage.

Critical weaknesses
  1. The event being predicted is essentially a change in an external fact, while the language model mainly serves to select samples and provide features. The paper could explain more clearly what studying the model’s internal knowledge tells us beyond what a general fact-lifetime prediction approach would offer.

  2. Although the paper focuses on facts changing over time, its retrieval-risk guarantee relies on exchangeability between the certification data and future data. This does not directly cover sudden events or shifts in how facts change; the authors acknowledge this limitation, but it still limits what the guarantee can offer for long-term deployment.

  3. The method assumes that a fact is confirmed to be valid at the prediction point and that the model can answer it correctly. Neither condition is usually guaranteed in practice, so the paper should explain how these assumptions would be handled in a real deployment.

Questions for authors

Please refer to the weaknesses.

Reviewer_74XUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates a potential blind spot in LLM deployments: a model can provide a factually correct answer at deployment and later become wrong simply because the world has changed, even though the model parameters remain frozen. The authors formulate this as a prospective survival analysis problem in calendar time—handling left truncation, interval censoring, and right censoring. They introduce WIKILIFETIME, a rolling-origin benchmark of valid and correctly recoverable facts, and propose EXPIRE, a low-rank time-varying hazard model that integrates relation-specific duration structures, entity-level histories, calendar drift, and frozen hidden states.

Contributions:

  1. Formulates knowledge expiration as a prospective survival target under left truncation and censoring
  2. Introduces WIKILIFETIME, featuring strictly separated chronological blocks for fitting, tuning, recalibration, risk certification, and testing; 3. Proposes EXPIRE, combining duration, event-history, and frozen LM representations in a nested framework;
  3. Establishes theoretical guarantees
Critical strengths
  1. Framing factual staleness as a proper survival analysis problem, complete with rigorous handling of left truncation, interval censoring, and chronological data separation

  2. The paper provides an end-to-end theoretical analysis, a carefully curated empirical benchmark, and ablation studies.

Critical weaknesses
  1. The paper is difficult to read and follow because it jumps straight into dense mathematical formulations without a clear narrative structure.. It would greatly benefit for an intuitive running example (e.g., tracing over time a politician's career or sports transfer over time) that would reduce the cognitive load of the reader.

  2. Relying on WIKILIFETIME to both quantify the "blind spot" and evaluate the proposed solution creates a circular setup where the metric is co-designed with the tool. Because the benchmark relies on heavy programmatic filtering, it remains unclear whether the measured decay reflects natural real-world dynamics or is an artifact of a heavily sanitized pipeline.

  3. The paper lacks an independent estimate of the scale of the factual expiration problem (e.g., an out-of-distribution study based on naturalistic queries evaluated across general time windows). Without an independent baseline measured outside of their custom curation pipeline, the actual scale of the problem remains unknown.

Questions for authors

n/a

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.