acceptodds
Withdrawn from ICLR 2027

When Do Independent Scientific Alternatives Emerge? Evaluating LLM Predictions of Waiting Times

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The wait from a scientific contribution to its first independent, functionally equivalent alternative describes when another route to the same capability becomes available. We examine whether evaluation scores reflect how well large language models rank these reinvention times. In a post-hoc audit of an author-built dataset and saved predictions from three models, we trace waiting-time labels to their target works and decompose the ranking score by comparison type. At least 8 of 34 eligible observed held-out examples have labels inconsistent with the target work's interval under the dataset's own rules. Adding curator-specified lower bounds (claims of no alternative before a cutoff) raises the C-index ranking score from 0.42–0.45 to 0.61–0.67 on the models' respective valid subsets, without changing any observed-interval prediction ordering. The increase comes from comparisons between observed intervals and lower bounds. A diagnostic using only record type scores higher still, without ranking waiting times within either type. The score increase persists after recomputing the known mismatches from retained records, including on samples valid for all three models. Neither excluding the known mismatches nor recomputing their labels establishes above-chance observed-interval ranking. Two models exceed the specified output-failure threshold, so their affected results remain exploratory. This case shows why assessing waiting-time ranking requires checking both what each label measures and which comparisons produce the score.

Primary area:
datasets and benchmarks
Keywords:
large language model evaluation, scientific forecasting, benchmark auditing, ranking evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_rNzWOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether LLMs can predict the time until the first independent, functionally equivalent alternative to a scientific work emerges. Through a post-hoc audit of an author-constructed dataset and saved predictions from three models, the study examines whether waiting-time labels are consistent with their target works and decomposes the C-index into observed–observed and observed–lower-bound comparisons. The results show that at least 8 of 34 eligible held-out observed samples have inconsistent labels. Although adding 15 curated lower bounds improves the pooled C-index, the gain is driven primarily by cross-source comparisons and does not reflect improved ranking of observed waiting times.

Critical strengths
  1. The paper addresses the relatively underexplored problem of when independent, functionally equivalent scientific alternatives emerge, and its decomposition into observed–observed and observed–lower-bound comparisons makes the evaluation framework more transparent and convincing.

  2. The paper reports label inconsistencies, model output failures, and post-hoc analyses transparently, which strengthens the reproducibility and credibility of its conclusions.

Critical weaknesses
  1. My main concern is that the motivation for this work remains insufficiently clear, particularly regarding the practical significance of evaluating independent, functionally equivalent scientific alternatives. In addition, the evaluation set is quite small, consisting of only 34 eligible observed held-out samples and 15 N3 samples, which I believe is insufficient to support broad conclusions.

  2. I think a substantial portion of the paper is devoted to dataset construction and auditing, while the analysis relies only on saved predictions from three models, without a systematic evaluation on a broader set of mainstream models. In addition, the reinvention-time target itself has not been externally validated, making it unclear to what extent the identified label and scoring issues generalize to broader scientific forecasting benchmarks.

Questions for authors
  1. Could the authors evaluate a broader set of current mainstream models and include repeated runs to verify that the main findings are not driven by a small number of models or failed calls?

  2. It would be helpful to validate the observed score-composition phenomenon on a larger dataset or on scientific forecasting benchmarks with similar censored-label structures, in order to assess its generalizability.

  3. Would the authors consider more strictly controlling for temporal leakage, for example through time-restricted data and model settings, to verify that the forecasting results do not rely on future information?

Reviewer_A9CvOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper performs a post-hoc audit of predictions of waiting times between scientific contributions and their first independent, functionality equivalent alternatives. The predictions are performed by LLMs, and evaluated using the C-index ranking score. The main contribution is evidence showing that when assessing waiting-time ranking, we need to check both label consistency and score composition.

Critical strengths

None.

Critical weaknesses

I found this paper exceedingly difficult to read. It is insufficiently motivated (e.g., lines 36--41 do not actually address why we should study the arrival of alternatives), its prose is filled with unexplained jargon, its equations contain symbols that are not explained anywhere in the text, the flow of the narrative is incoherent, and the main claims and contributions are unclear.

Even if the writing is substantially improved, there are other critical issues that need to be addressed before this paper is ready for publication. There is no reference to the 'author-built dataset' that the authors perform their post-hoc audit on. If it is not a public dataset, it is difficult for me to imagine why we should care about this audit. If it is not public and it is a dataset constructed by the authors of this paper, then the first step would be to publish the dataset itself as a standalone paper, or perhaps combine the dataset with this audit into one paper.

Additionally, there are multiple instances where the authors bring up issues that need to be addressed before publication, e.g. "no external validation" (line 165), "unverified curator assertation" (lines 231--232), "temporal leakage unresolved" (line 471)

Questions for authors

The questions below are prefixed with line numbers from the submitted PDF. These are all clarification requests.

  • 50:

    • What is the 'reference interval'?
    • 'Evaluation' -> What is being evaluated?
  • 51:

    • 'comparisons' -> what is being compared?
    • 'label' - what do you mean by 'label' here? This suggests a classification task, but the previous paragraph suggests a regression task, since the output (days) is an ordinal variable
    • How are the predictions scored? Is it on a numeric scale?
  • 52:

    • What does it mean for the papers to be 'linked'?
    • What do you mean by 'it' here?
  • 53:

    • What is the 'different temporal question' here?
    • What is an 'observed interval'?
    • What is a 'record' here? Is a record a paper?
    • What does it mean to pool intervals with records?
  • Figure 1:

    • Figure 1(b)
      • What are 'A', 'B', and 'C' in this example? If they are papers, why are the names of the papers not shown in the figure?
  • 79: 'comparisons involving lower bounds' -> what is being compared here?

  • 82: What do you mean by 'author-built'? Do you mean that the authors of the dataset are the same as the authors of this paper?

  • 86: What is a 'source-only diagnostic'?

  • 165: 'no external validation' - is that not a cause for concern?

  • 193: 'The target is a work' - what does this mean?

  • 195: What is a 'row' here?

  • 213: What are 'the dataset's rules'?

  • 222: What are tu,tv,Ye,Ee,Bet_u, t_v, Y_e, E_e, B_e? A number of symbols in the paper do not have any explanation.

  • 224: 'graph-derived reference interval within the retained component' - what is this?

  • 225: What is this notation: 1{Ye=Ze(a)}\mathbf{1}\{Y_e = Z_e(a)\}?

  • 229--230: What is Ze+(a)Z_e^+(a)? Is it the same as ZeZ_e?

  • 231--232: "The separate claim that no qualifying substitute occurred before tct_c remains an unverified curator assertion." -- Is this not a cause for concern?

  • 253--254:

    • What is 'source separation' here? How is it defined?
    • 'ties every OO prediction' - what does this mean?
  • 281--282: "A later Cox gradient-boosting baseline fitted to the 77 development samples is exploratory." - Why is this sentence here? There are no other instances of the word 'gradient' in this paper.

  • 285: "projection onto each model’s observed subset."- what does this mean?

  • 471: "temporal leakage unresolved" - this seems like a serious issue, rather than an acceptable limitation. Can you explain further?

  • 504: What is this 'anonymous artifact"?

Reviewer_FMPoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the validity of an LLM benchmark for predicting scientific reinvention times, identifying issues with label consistency and evaluation metrics.

Critical strengths

The paper proposes an interesting research question and conducts extensive experiments.

Critical weaknesses
  1. The paper is very difficult to follow. For example, the abstract contains many complex sentences that require multiple readings to understand. The introduction is poorly structured and lacks coherence. The authors motivate reinvention time as an interesting research topic, but the actual focus is on LLM benchmark validity, creating a mismatch. Moreover, the introduction reads very much like an abstract, without sufficiently developing the research motivation or contributions. As the introduction is an important part of the paper, I strongly suggest that the authors reorganize it to improve readability. I think for now, this form is not human-readable.
  2. Also, many symbols are not explained when first introduced, requiring readers to infer their meanings from context (e.g., t, B, and N). I find it very difficult to read this paper.
  3. I have serious concerns about the validity of the results. (1) The data collection process relies on papers addressing the same task, but this does not necessarily mean they represent equivalent discoveries. (2) The authors use LLMs for adjudication but do not sufficiently discuss the errors they may introduce. (3) The candidate set may be incomplete; failing to find an alternative does not necessarily mean that none exists.
Questions for authors
  1. How do the authors ensure the validity of the data collection process? Is the assumption that papers addressing the same task represent equivalent discoveries too strong? Could LLM-based adjudication introduce errors?
  2. What is the primary conclusion of this paper? Can a general conclusion be drawn from evaluating only three LLMs? Given that the findings are based on a single benchmark, how generalizable are they?
  3. Is the small sample size sufficient to support conclusions about LLM predictive ability?
  4. The introduction initially motivates reinvention-time forecasting, while the main contribution appears to be a post-hoc audit of benchmark validity. Could the authors clarify the central research question and more explicitly distinguish the forecasting task from the benchmark evaluation problem?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.