- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates whether LLMs can predict the time until the first independent, functionally equivalent alternative to a scientific work emerges. Through a post-hoc audit of an author-constructed dataset and saved predictions from three models, the study examines whether waiting-time labels are consistent with their target works and decomposes the C-index into observed–observed and observed–lower-bound comparisons. The results show that at least 8 of 34 eligible held-out observed samples have inconsistent labels. Although adding 15 curated lower bounds improves the pooled C-index, the gain is driven primarily by cross-source comparisons and does not reflect improved ranking of observed waiting times.
Critical strengths
-
The paper addresses the relatively underexplored problem of when independent, functionally equivalent scientific alternatives emerge, and its decomposition into observed–observed and observed–lower-bound comparisons makes the evaluation framework more transparent and convincing.
-
The paper reports label inconsistencies, model output failures, and post-hoc analyses transparently, which strengthens the reproducibility and credibility of its conclusions.
Critical weaknesses
-
My main concern is that the motivation for this work remains insufficiently clear, particularly regarding the practical significance of evaluating independent, functionally equivalent scientific alternatives. In addition, the evaluation set is quite small, consisting of only 34 eligible observed held-out samples and 15 N3 samples, which I believe is insufficient to support broad conclusions.
-
I think a substantial portion of the paper is devoted to dataset construction and auditing, while the analysis relies only on saved predictions from three models, without a systematic evaluation on a broader set of mainstream models. In addition, the reinvention-time target itself has not been externally validated, making it unclear to what extent the identified label and scoring issues generalize to broader scientific forecasting benchmarks.
Questions for authors
-
Could the authors evaluate a broader set of current mainstream models and include repeated runs to verify that the main findings are not driven by a small number of models or failed calls?
-
It would be helpful to validate the observed score-composition phenomenon on a larger dataset or on scientific forecasting benchmarks with similar censored-label structures, in order to assess its generalizability.
-
Would the authors consider more strictly controlling for temporal leakage, for example through time-restricted data and model settings, to verify that the forecasting results do not rely on future information?