- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how long a fact will remain valid after a frozen language model can answer it correctly. It defines this as a remaining-lifetime prediction problem with interval and right censoring. The paper introduces WIKILIFETIME and proposes EXPIRE. The model combines relation-level patterns, fact age, event history, calendar effects, and frozen LM representations. The authors evaluate it through simulations, historical forecasting, and retrieval decisions. They also give theoretical results for estimation and risk control under stated assumptions.
Critical strengths
-
The problem is meaningful and the survival formulation fits it well. A model can keep giving the same answer while the world changes. Predicting this risk before a change is observed has clear value for deciding when to retrieve fresh evidence. The method also uses the observed event intervals directly. This avoids treating uncertain change dates as exact dates.
-
The representation experiments address an important alternative explanation. The authors do more than compare EXPIRE with a relation-only baseline. They use within-relation matching, representation permutation, and relation-centroid controls. In Table 2, adding the representation improves NLL from .352 to .331 and matched-pair accuracy from .636 to .699. These results support an instance-level signal beyond relation identity. This is the strongest empirical reason to consider acceptance, although it still needs confirmation under a corrected time setup.
Critical weaknesses
1. The chronological split does not establish that development labels were available before deployment.
Table C.4 places validation origins in 2021, recalibration in January 2022, certification in July 2022, and testing in January and July 2023. Page 32 says earlier blocks also use up to 24 months of follow-up. The final hazard model is refitted on both training and validation data.
This creates a timing problem. Full 24-month outcomes for July 2021 validation records are not available until July 2023. January 2022 recalibration records need follow-up until January 2024. Even the full 12-month certification outcomes from July 2022 are not available before the January 2023 test date.
Future follow-up is needed to evaluate test predictions. My concern is about using future outcomes to fit the model, recalibrator, or retrieval threshold before those test predictions are made. Restricting input features to the anchor date and keeping subjects separate do not solve this issue.
Please state the latest outcome date used by every stage at each deployment date. Either leave enough time for the required outcomes to become available, or censor development records at the deployment date. Then rerun fitting, selection, recalibration, certification, and testing. Without this check, the results support an offline historical study more clearly than a strict prospective deployment claim.
2. The main censoring-adjusted scores need stronger justification.
Appendix E.1 estimates unresolved outcomes using a reference model based on relation, anchor quarter, and age group. The candidate predictors use richer history and semantic features. The reference model may therefore miss outcome information that is present in the candidate prediction.
This matters for the reported Brier scores and calibration. Suppose the reference model assigns a survival probability of .5 to an unresolved record. A candidate that correctly predicts its true status can receive a worse pseudo-Brier score than a constant prediction of .5. Cross-fitting reduces overfitting, but it does not remove this missing-information problem.
The test missed-stale estimate in Appendix E.3 uses the same pseudo-outcomes. Its interpretation also needs checking. This concern applies to unresolved outcomes. Observed-data NLL, matched pairs with known status, and fully observed horizon scores still provide useful evidence.
Please give the conditions under which these scores estimate the intended prediction risk. Report the unresolved fraction at each horizon. Also check the conclusions with a justified censoring-adjusted estimator that accounts for the relevant predictors. A fully resolved subset would be useful as an additional sensitivity check, with its selection limits clearly stated.
Questions for authors
-
Label availability. At the January 2023 test date, which validation, recalibration, and certification outcomes were actually available? Was follow-up cut off at that date? If such a rule was used, please show it and explain how it fits the 24-month follow-up description. If not, please provide results after correcting the timing.
-
Score validity. What assumptions allow the reference posterior based on relation, quarter, and age group to evaluate predictions that also use history and LM states? How many records have unresolved status at each reported horizon? Do the main comparisons hold under an evaluation that addresses this issue?
-
Scope of the risk guarantee. Certification uses only records with fully resolved outcomes. Why should this selected group match the full deployment population? I appreciate that the paper already limits Theorem 4.6 to exchangeable data and does not claim protection under arbitrary calendar drift. Please also clarify the effect of selecting only resolved records, and keep the theorem’s expected-risk guarantee separate from the later-year test results.
-
Event-interval filter. Pages 28 and 32 state that intervals wider than 180 days are excluded from the primary analysis. Table C.6 still reports 513 events with intervals of 7–12 months. Which rule was used? Please correct the counts and confirm that all main tables use the same primary cohort.
-
Model source and reproducibility. What are the exact sources and versions of the 7B checkpoints in Table C.2? The cited Kelly et al. paper describes 1.5B and 4B models, so the mapping is unclear. Please explain whether the 7B models were separately trained or came from another source. An anonymous manifest with model identifiers, hashes, data cutoffs, relation IDs, and schema filters would help verify the benchmark. See the cited point-in-time model paper.