acceptodds
Under review as a conference paper at ICLR 2027

Ephemeral Models, Durable Claims: Closed-Model Dependence and the Preservation of AI Research

Abstract

Large language models (LLMs) are routinely used inside modern AI experiments: to generate or annotate data, to provide comparison points, and increasingly to evaluate other systems. When the exact model artifact can be retained locally, later researchers can often preserve that dependency. When access is mediated only through a provider-controlled service, they usually cannot. Using an end-to-end measurement pipeline validated on a blinded random sample (F1=0.967), we measure this dependence across 53,538 final archival papers from ACL, EMNLP, ICLR, ICML, and NeurIPS (2023–2026) and assess whether the referenced model artifacts persist. We find that 17,841 papers (33.32%) contain a covered closed model in at least one scientific role. Across the common five-venue window, prevalence rises from 14.96% in 2023 to 37.98% in 2025, while use of closed models as judges rises from 1.47% to 11.50%. At a September 24, 2026 lifecycle cutoff, seven of the 20 most frequently used normalized models are already retired; several others are deprecated, on explicit snapshot-shutdown paths, or named too coarsely to identify a unique served artifact. We characterize this exposure and derive practical preservation norms for durable scientific evidence, spanning model reporting, peer review, archival traces, and post-commercial research access.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, scientific reproducibility, closed models, model deprecation, LLM evaluation, LLM-as-a-judge, model preservation, research infrastructure

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JhPYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies the prevalence and parctices of using closed AI models in research acorss ML/NLP conferences in the past few years.

Critical strengths
  • this is a hugely interesting research question
Critical weaknesses
  • the scope and depth is limited
Questions for authors

Honestly I hugely appreciate the authors doing this: doing AI research with open, reproducible models should be encouraged and be the norm, and the research question of 1) how prevalent is closed models in research, and 2) what does that mean to AI research, is hugely important and critical. It is ``heartbreaking'' to see this important question not studied to its fullest in here to warrant acceptance.

  • Are the 4 categories in Table 1 exhaustive and non-overlapping? For example, if a paper on synthetic data generation uses a closed model for it, does it count as direct or data generation?

  • People are still using Gemini 1 in 2025, according to Figure 4? How is model version deteceted in this pipeline? Is that accurate? Can the authors give a few specific example papers where they said they used Gemini 1 in 2025?

  • The result sections felt hollow, just like the huge blank spaces on page 5 and throughout the paper. There is so much left out in this study, a few of them include:

  1. When closed models are used, do people preserve their outputs and share them? This could serve as a compromise? How many of those outputs are still accessible?

  2. How does drift impact things? For example, for usages of the same closed models, any cases where the numbers/conclusions changed between different timed usages?

The authors are encouraged to significantly beef up the analysis part.

  • Would some policy-level levers to advocate for open models in research make sense? For example, the main conclusions in every paper has to be supported by at least one open-weight model, or risk desk rejection?
1 comment
AuthorsOpenReview ↗

Clarifications, Additional Analyses, and Revision Plans

Thanks for the thoughtful review. We share the concern and appreciate your suggestions.

Our 53,538-paper census is complete at paper × model resolution, linked to year, venue/track, roles, providers and reported identifiers. The submitted ZIP contains frozen outputs, detailed tables, validation/lifecycle data, registry and pipeline code. We intend to release the pipeline and processed data publicly, especially for a paper about preservation.

1. Are the four roles exhaustive and non-overlapping?

They are non-exclusive (Sections 2.2 and 3.1). Using a closed model to synthesize study material is Data Generation; it can also be Direct if the model separately serves as an experimental system. 61.6% of Scientific Use papers have at least two roles and 25.1% have three or four. The four roles define Scientific Use; writing/coding assistance is separate Supporting Use.

2. Gemini 1 in 2025 / version detection / examples

Figure 4 plots generational families (text and caption). "Gemini 1" means Gemini 1.x, including 1.5, not 1.0 alone. The registry/tables retain finer identities. In 2025, Gemini 1.5 Pro appears in 680 papers, 1.5 Flash in 393 and 1.0 Pro in 219 (non-exclusive).

For example: Training Language Models to Self-Correct via Reinforcement Learning (ICLR 2025) uses 1.0 Pro and 1.5 Flash; Towards Adapting Open-Source Large Language Models for Expert-Level Clinical Note Generation (Findings of ACL 2025) uses 1.0 Pro as a teacher. Both are in our corpus. Publication year is not invocation year (Section 4.4). We can relabel Figure 4 "Gemini 1.x".

3. On the depth of the results

The paper already goes beyond the 33.32% headline: role trends/overlap, venue/track differences, multi-provider use, model-family footprints, lifecycle and identifier specificity. Only 16.3% of Scientific Use papers are baseline-only; Judge prevalence rises from 1.47% to 11.50%; NeurIPS 2025 Datasets & Benchmarks has 63.98% Scientific Use versus 21.32% in Main. These show how closed models enter scientific evidence, not just their frequency.

We prioritized broad field-level readings because of the nine-page conference format, not because the analysis stops there. The full paper × model × year × venue/track analysis already exists, with frozen outputs and code submitted. We can recombine it for model trajectories, role × venue/year, provider combinations, identifier-level patterns, lifecycle × role and more, without redoing the census. We would welcome suggestions on additional analyses that could strengthen the study, and will try to incorporate them in the revision.

3.1. Are outputs preserved and shared?

Our objects of study were the 53,538 archival papers. Their preservation and availability made this scale possible, although building a reliable paper × model census still took substantial work. Experimental artifacts are another research object: code, prompts, generated data and outputs are not archived or discoverable as consistently as papers. Assessing what was preserved would be a separate study with different coverage and access challenges. Even a repository/link audit would tell us only what is publicly discoverable, not whether full experimental traces were ever retained or made shareable.

Section 5 recommends archiving material inputs/outputs for closed judges and generators. Traces preserve evidence from the original run, but not the ability to reuse a retired judge on new systems or regenerate material under corrected protocols. We can emphasize this in revision.

3.2. Has drift changed numbers or conclusions?

Model changes can affect results, especially for judges. But papers alone cannot tell us which historical results changed. That requires original inputs/settings and access to comparable historical model states. Authors may report only a broad name or mutable alias, leaving the checkpoint unidentified; identified checkpoints may also be retired. Comparing distinct released checkpoints is not evidence of drift in the same checkpoint. Our census maps identification and availability problems, not a rate of changed conclusions. We can clarify this scope.

4. Policy levers

Table 3 proposes preservable references for acceptance-critical evidence and closed comparisons where the claim requires that model/capability. A blanket open-weight rule can change the question when the proprietary model is the object of study. We favor avoiding unnecessary non-preservable dependencies and documenting necessary ones.

The urgency is concrete. Our lifecycle cutoff was September 24, 2026. On October 23, OpenAI shut down gpt-4-0613, gpt-4-1106-preview, gpt-4o-2024-05-13 and gpt-4-turbo-2024-04-09. These occur in 480, 393, 376 and 233 papers in our submitted data (non-exclusive). The artifacts disappeared during review.

Thanks again. We welcome further suggestions and will do our best to incorporate them in revision. Looking forward to further discussions.

Reviewer_GLmYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A census of 53,538 ACL, EMNLP, ICLR, ICML, and NeurIPS papers (2023–2026) finds that a third depends on closed models in a scientific role, and that this share is rising fast, especially for LLM-as-judge use. Furthermore, many of the most-used models are already retired or were never identified precisely enough to be preserved.

Critical strengths
  1. The drift/extinction/underspecification framing is useful.
  2. The authors explicitly decline to report an “irreproducibility rate.”
Critical weaknesses
  1. With about 33% prevalence, the 200-paper sample has roughly 65 positives, and at 11.5% prevalence, probably fewer than 20 Judge cases. The headline judge trend therefore has little direct validation.
  2. No analysis shows that any study actually became unreproducible as a result.
Questions for authors
  1. What explains ACL’s jump from 16.5% to 62.6% between 2023 and 2024?
  2. Can you report paper-level exposure using each paper’s exact reported identifier?
2 comments
AuthorsOpenReview ↗

Judge Validation, ACL Trends, and Paper-Level Reproducibility Loss

Thanks for the thoughtful review. We pulled the paper-level retirement figures directly from the existing outputs of our submitted pipeline. The results are striking.

1. Is the Judge trend sufficiently validated?

We believe so. The blinded 200-paper validation sample contains 29 human-labeled Judge positives, not fewer than 20. Appendix E, Table 11 gives 24 TP, 2 FP and 5 FN, with precision 0.923, recall 0.828, and F1 0.873. Scientific Use F1 is 0.967, and four-role micro-F1 is 0.888. The human labels and validation are in the submitted ZIP. This validates Judge classification overall; the yearly rates come from the full census. We'll bring these figures into the main text in the revised PDF.

2. Have any papers actually become irreproducible?

Yes, thousands of papers. At our September 24, 2026 lifecycle cutoff, 6,918 unique papers already had at least one scientifically used model whose first-party API had retired. By November 11, that rose to 13,197 papers, or 73.97% of the 17,841 Scientific Use papers. That's 6,279 additional papers in less than seven weeks, using the same frozen census, largely due to the retirement of legacy OpenAI GPT endpoints (details in point 4). That is 24.65% of the entire 53,538-paper corpus, with no new papers added to the census.

When a required historical model can no longer be run, the original experiment cannot be reproduced as performed. A retired judge cannot evaluate new systems with the original scoring procedure; a retired baseline cannot be rerun to verify the original comparison; a retired generator cannot produce further samples using the original model. Preserved outputs may document the past run, but they cannot make the original model callable again. Even if we set aside retired baseline dependencies entirely, 9,697 papers still used a retired model directly, as a judge, or for data generation, including 2,734 with a retired Judge.

These numbers capture one source of irreproducibility, not the overall rate across the field. Missing repositories, unavailable data, code/paper mismatches, incomplete methods and other proprietary components were already problems long before closed LLM APIs became common. Those failures lie outside this census, and papers without retired models may be irreproducible for those reasons too. We will include the paper-level analysis and role breakdown in the revised PDF.

3. What explains the ACL 2023 to 2024 jump?

The submission calendar holds much of the answer. The October 15, 2022 ARR cycle closed 46 days before ChatGPT launched on November 30. The final ACL 2023-eligible ARR deadline, December 15, was only 15 days after launch, and the January 20 direct submission deadline was 51 days after. GPT-4 launched March 14, 2023, after those submission deadlines. ACL 2024, meanwhile, accepted ARR submissions through February 15, 2024. EMNLP 2023's later June 23 direct deadline already corresponds to 44.06% Scientific Use in our census. The cohorts align with rapid ChatGPT/GPT-4 adoption.

The change is not an artifact of track composition. ACL Scientific Use rises 16.46% to 62.61% overall, but also within Findings (13.65% to 61.95%), Main Long (19.89% to 64.81%), and Main Short (12.80% to 46.05%). The increase spans multiple scientific roles and is already visible at the initial model-reference retrieval stage, before any role adjudication.

4. Can you report paper-level exposure from reported identifiers?

Yes. We joined the submitted paper × model records, their reported identifiers and scientific roles, to dated provider retirement records, then counted each affected paper once regardless of how many retired models it used. The pipeline had already recovered those relationships; this is an additional aggregation of the frozen data, not a new classification run. It retains paper-level evidence, roles, models and retirement dates.

The affected-paper count rose from 6,918 at the September 24 cutoff to 13,197 by November 11. The October 23 retirement of legacy OpenAI endpoints accounts for over six thousand of the newly affected papers. Those endpoints include gpt-3.5-turbo, gpt-4-0613, gpt-4-1106-preview, gpt-4-turbo-2024-04-09, and gpt-4o-2024-05-13. GPT-4o is the most-used normalized model in our corpus, appearing in 7,340 papers; one of its historical snapshots has already retired, with others scheduled for January 2027.

The union also includes papers naming a model without a dated snapshot when its whole reported generation has retired, such as GPT-3/3.5. Papers with multiple affected models still count once. These records show how many original research procedures have lost their historical model access, and how quickly that number is growing.

We'll bring the paper-level retirement analysis and Judge validation into the revised PDF during the current discussion period. Thanks again for the specific questions. They've helped us identify useful additional results to surface.

AuthorsOpenReview ↗

Update on Validation Clarifications and Reproducibility Analysis

Thank you again for the careful review. Following our earlier author response, we wanted to highlight the updates in the revised manuscript PDF addressing your questions about measurement validation and the reproducibility implications of model retirement.

The validation was already part of the original submission, including the blinded sample of 200 human-annotated papers and the detailed role-level results in Appendix E. To make this evidence immediately accessible, the revised manuscript PDF now brings these existing validation results into the main Methods section, explicitly reporting the number of human-labeled positive examples for each scientific role alongside its corresponding F1 score.

Of the 200 papers in the blinded validation sample, 32 were labeled positive for Direct Use, for which the pipeline achieved an F1 score of 0.906. 48 papers were labeled positive for Baseline Use, with an F1 score of 0.913. 29 papers were labeled positive for Judge Use, with an F1 score of 0.873, while 27 papers were labeled positive for Data Generation, with an F1 score of 0.844. These gold-positive counts are not mutually exclusive, as a paper may involve multiple scientific roles. The overall Scientific Use classification achieved F1 = 0.967. These are the original validation results, now presented more prominently rather than newly conducted experiments.

The largest empirical update is the paper-level retirement analysis. Using the frozen paper-model-role records and dated first-party lifecycle evidence, we identify 13,197 of 17,841 Scientific Use papers (73.97%) with at least one retired historical closed-model dependency. This includes 2,734 papers that used a now-retired closed-model judge.

The revised manuscript PDF now gives these results dedicated subsections and figures in Sections 4.4-4.6. Section 4.7 also reports the separate model-identity problem: 17,109 of 17,841 Scientific Use papers (95.90%) lack an explicitly fixed snapshot or revision for at least one scientific closed-model dependency. These findings establish the loss of access needed to rerun original scientific procedures, rather than inferring changes in published conclusions.

The targeted updates are highlighted in blue in the revised manuscript PDF, and the updated supplementary archive includes the paper-level retirement evidence, identifier audit, and executable analysis.

Our consolidated comment, "Consolidated Review Clarifications and Targeted Manuscript Updates," summarizes the complete set of updates across the reviews.

Reviewer_Co1VOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines a unique research question about the dependence of recent AI research papers on proprietary AI models, especially given that these models are regularly deprecated over time. It reports 33.32% scientific use, increasing prevalence and judge use over 2023–2025, and the retirement of seven of the twenty most-used normalized models. These results establish substantial documented dependence and lifecycle exposure within the covered corpus and registry, rather than a paper-level irreproducibility rate or the effectiveness of the proposed preservation interventions.

Critical strengths
  1. The core motivation of the paper on the dependencies of AI research papers on proprietary AI models is a timely topic, especially regarding the increasing use of AI in many different aspects of research.

  2. A comprehensive data analysis across model families from different companies and different venues provides statistical insights for the paper.

Critical weaknesses
  1. Limited insight beyond descriptive measurement: The paper provides a broad descriptive analysis of proprietary-model use in AI research, but the significance of the resulting empirical insights remains insufficiently established. For example, the growth in judge use and the high prevalence of dependence in dataset and benchmark papers document the scale and distribution of the phenomenon, but the paper does not clearly explain how these findings materially advance our understanding beyond the general concern that provider-controlled models may become unavailable. While quantifying this exposure is useful, the preservation recommendations largely follow from that general concern, leaving the additional implications of the specific empirical findings underdeveloped.

  2. Underdeveloped trade-offs in preservation recommendations: Section 5 appropriately conditions its preference for preservable models on their ability to serve the same scientific role, but leaves the practical meaning of this condition underexplored. Depending on the task and available resources, proprietary models may offer capabilities or lower operational costs that materially affect the feasibility of using preservable alternatives. A qualitative discussion of these trade-offs would better establish when the preservation benefits justify the recommendation, since increasing proprietary-model use alone does not establish that such use should be reduced.

Questions for authors

The questions below mostly come from the weaknesses above.

  1. How should “serving the same role” in Section 5 be interpreted when preservable alternatives differ in capability, cost, or practical accessibility?

  2. Which specific findings most substantially advance our understanding beyond the established concern that provider-controlled models may become unavailable?

2 comments
AuthorsOpenReview ↗

Empirical Significance and Practical Preservation Trade-offs

Thanks for the review. We appreciate the push to explain what these findings tell us, and how the practical model-choice trade-offs work.

1. What do the findings add beyond the known risk that closed models may disappear?

People have raised this concern before, including in studies we cite. But before this study, the evidence was largely qualitative or limited to narrower audits, such as the 255-paper GPT-3.5/GPT-4 audit discussed in our Introduction. To our knowledge, there was no comparable large-scale, multi-venue census measuring scientific roles alongside model identity and lifecycle state. Our validated census covers 53,538 archival papers.

Some key findings:

  • Of the 17,841 Scientific Use papers, only 16.3% use closed models only as baselines. The other 83.7% use them directly, as judges, or to generate study data. They are part of how the experiment is carried out, not just a comparison in a table.
  • Judge use rises from 1.47% of papers in 2023 to 11.50% in 2025. A growing part of the field is using these models to measure results, not only as systems being measured.
  • NeurIPS 2025 Datasets & Benchmarks papers have 63.98% Scientific Use versus 21.32% in Main; Judge use alone is 19.1% versus 6.3%. The irony is that work building benchmarks depends heavily on models that may disappear. If a benchmark’s judge is retired, future systems can no longer be scored with its original evaluation procedure, undermining comparability as well as reproducibility.
  • 61.6% of Scientific Use papers involve multiple roles. Multi-provider use also grows from 7.3% to 33.6% between 2023 and 2025. Using several providers can be useful, but each historical model still has its own lifecycle.
  • Seven of the twenty most-used normalized models had already retired by our September 24, 2026 cutoff.

And this is happening now. GPT-4o is the most-used normalized model in the census, appearing in 7,340 Scientific Use papers. On October 23, OpenAI shut down gpt-4o-2024-05-13 (reported in 376 papers in our frozen data), along with gpt-4-0613 (480), gpt-4-1106-preview (393), and gpt-4-turbo-2024-04-09 (233). Other widely used GPT-4o snapshots, including gpt-4o-2024-08-06 (640 papers), now have a January 14, 2027 shutdown date. These identifier counts are from our submitted data. Models used by hundreds of papers have gone offline between our measurement cutoff and this review.

The scientific consequence also depends on the role. If a baseline disappears, we lose the original comparison point. If a judge disappears, it can't score new systems. If a generator disappears, we may not be able to recreate or correct the study material. That is what the role-level analysis adds: not just how much use there is, but what is at stake when access changes.

The full paper × model analysis is done. The ZIP includes its frozen tables, registry, and pipeline. We plan to release the pipeline and processed data publicly, and develop a public tracker of model lifecycles and the research that depends on them. The paper establishes the research method and empirical baseline for that effort.

2. What does it mean for a preservable model to serve the same role?

We mean the actual scientific task being performed. Many tasks can be done with either an open-weight model or a closed API. If both can do the job well enough for the claim being made, choosing the model we can keep makes that part of the experiment easier to reproduce later.

We made exactly this choice in our own pipeline. We use gpt-oss-120b served locally to adjudicate ambiguous model-use evidence (Section 3.3). Using a closed API would have been easier operationally, but the task was to read bounded evidence and assign the right scientific roles, not to study a particular commercial model. We chose to keep that step locally runnable, took on the compute cost, and checked the pipeline against blinded human annotations (Scientific Use F1=0.967). So this isn't just a recommendation we're making to others; it's a trade-off we made in the study.

Of course the choice still depends on the experiment. A judge has to measure what the paper intends to measure; a generator has to produce suitable data; a baseline has to be a meaningful comparison. Capability, cost, and access all matter. And if the research is specifically about a closed model like GPT-4, or needs a capability only available through that system, then that model is part of the research question. Using something else wouldn't answer it.

This is what we mean by “when it can serve the same role” in Section 5. The aim isn't to tell the field to use fewer closed models. It's to avoid giving up the ability to rerun an experiment when the scientific task didn't require that trade-off, and to document the necessary closed-model uses as well as possible.

We will expand on these points in the revised PDF during discussion. Thanks again for engaging with the work. We'd welcome any further thoughts.

AuthorsOpenReview ↗

Update on Scientific Consequences and Preservation Results

Thank you again for your review. Following our earlier author response, we wanted to share the targeted updates now incorporated into the revised manuscript PDF, particularly concerning the scientific significance and practical consequences of closed-model dependence.

The revised PDF now quantifies how extensively original experimental procedures have already lost their historical closed-model dependencies. As of November 11, 2026, 13,197 of the 17,841 Scientific Use papers (73.97%) have lost first-party access to at least one closed model used in their research. Even excluding baseline-only cases, 9,697 papers have a retired Direct, Judge, or Data Generation dependency.

This establishes the preservation consequence at the paper level. A published baseline score cannot be recomputed with a retired model, and a historical judge cannot evaluate subsequent systems once the original model is unavailable.

Sections 4.1 and 4.4-4.6 of the revised manuscript PDF now present these findings alongside the historical accumulation of affected papers and the role-specific losses. Section 4.7 separately quantifies model identity loss: 17,109 Scientific Use papers (95.90%) lack an explicit fixed snapshot or revision for at least one closed-model dependency.

The preservation recommendations in Section 5 retain the distinction explained in our earlier response: when a locally retainable model can adequately serve the scientific task, it provides a more durable reference; when the scientific question requires a particular closed model, preserving its exact identity and historical accessibility becomes essential.

These targeted additions are highlighted in blue in the revised manuscript PDF. The updated supplementary archive includes the corresponding paper-level evidence and analysis code.

For the complete cross-review summary, please see our consolidated comment, "Consolidated Review Clarifications and Targeted Manuscript Updates."

Comments on the paper

2 comments
AuthorsOpenReview ↗

Follow-up on Additional Analyses and Venue Fit

Thanks for following up, and for the suggestions on developing the paper further. We have quite a few additional analyses available from the submitted data, and intend to use the revised PDF submission option during the current discussion period to bring more of these results into the paper in response to your feedback. We'll prioritize those that add the most insight within the available page budget.

On the venue question, perhaps it helps to explain the nature of the work a little more. This is an empirical measurement study built around a substantial computational pipeline. We assembled and processed 53,538 final archival papers across ACL, EMNLP, ICLR, ICML, and NeurIPS, constructed a model registry and evidence-based classification system, adjudicated ambiguous model-use cases, validated the measurements against blinded human annotations, and independently curated model lifecycle information. The pipeline code, processed data, and validation artifacts are included in the submitted supplementary materials.

The research questions are empirical: how prevalent closed-model dependence is, what scientific roles these models serve, how this dependence evolves across venues and years, and how model identity and lifecycle status affect the preservation of these dependencies. The measurements and findings are the contribution. The preservation recommendations are a relatively small part of the Discussion, arising from what we observed rather than forming the premise of the study.

We also feel this is directly relevant to the ICLR community. Closed models are themselves learned computational artifacts, and they increasingly participate in ML research as experimental systems, baselines, judges, and data generators. Yet the community is losing access to the exact historical models that published results depend on. Understanding the scale and structure of this dependence, and which parts of the research workflow are affected, is particularly relevant to the community developing and relying on these models.

Our longer-term intention is to develop the measurement pipeline into a public tracker of model dependencies and lifecycles. The phenomenon evolves too quickly for a single archival snapshot to capture indefinitely. The present study establishes the measurement methodology, validation, and empirical baseline for such ongoing monitoring. Our immediate motivation is to bring the scale and structure of this emerging preservation problem to the research community's attention through systematic evidence.

Thanks again for taking the time to engage with the work. We appreciate the feedback and will take it into account as we prepare the revised PDF. We'd be happy to hear any further thoughts.

AuthorsOpenReview ↗

Consolidated Review Clarifications and Targeted Manuscript Updates

We thank all three reviewers and the Area Chair for their time and effort. Across the reviews, three concerns emerged: empirical depth, realized reproducibility loss, and measurement reliability. The revised manuscript PDF addresses each, alongside updated supplementary code and data. Targeted updates responding to reviewer feedback are highlighted in blue; wording-only edits are not.

1. From dependence to realized loss

The census found scientific closed-model use in 17,841 of 53,538 papers (33.32%) across ACL, EMNLP, ICLR, ICML, and NeurIPS. Reviewers asked what this dependence means in practice:

13,197 of those 17,841 papers (73.97%) have already lost first-party access to at least one closed model used in their original scientific procedure, or 24.65% of the full corpus.

  • 9,697 papers have a retired Direct, Judge, or Data Generation dependency, excluding baseline-only losses.
  • 2,734 papers have lost a closed-model judge used as an evaluation instrument.
  • 891 papers entered archival proceedings in a calendar year after a scientifically used closed model had already retired.

These are deduplicated paper-level losses, not forecasts. This is a present preservation failure, not a hypothetical future risk. Saved scores cannot rerun baselines, judges, or generators. Sections 4.1 and 4.4-4.6 of the revised manuscript PDF show the consequences.

2. Historical and identity evidence

Historical loss of access. The revised manuscript PDF charts the share of all archival papers with a retired scientific closed-model dependency: 0.26% (2023), 5.04% (2024), 8.92% (2025), and 24.65% by November 11, 2026. A second main-text figure reports 13,197 of 17,841 Scientific Use papers (73.97%) affected, with retirement rates among papers using each role: Direct 5,168/7,939 (65.10%), Judge 2,734/5,087 (53.74%), Data Generation 5,371/9,292 (57.80%), and Baseline 8,589/11,755 (73.07%). Roles overlap. The lifecycle cutoff is November 11, 2026; October 23 OpenAI endpoint retirements alone affected over 6,000 papers.

Identity loss, separately quantified. A corpus-wide identifier audit finds 17,109 of 17,841 Scientific Use papers (95.90%) without an explicitly fixed snapshot/revision for at least one scientific closed-model dependency, including 4,492 of 5,087 Judge papers (88.30%). The revised manuscript PDF includes a dedicated subsection and role-level table. Retirement and missing model identity are separate, widespread failures.

3. Measurement clarifications and validation

Scientific Use comprises non-exclusive Direct, Baseline, Judge, and Data Generation roles; writing/coding assistance is separate Supporting Use. Among Scientific Use papers, 61.6% use closed models in at least two scientific roles, and 25.1% in three or four.

The original submission already reported blinded validation on 200 human-annotated papers (Scientific Use F1 = 0.967), with role-level details in Appendix E. The revised manuscript PDF now brings those existing results into the main Methods text, reporting human-labeled gold-positive counts alongside F1: Direct 32 (0.906), Baseline 48 (0.913), Judge 29 (0.873), Data Generation 27 (0.844). The role-specific gold-positive counts can overlap.

The retirement analysis measures whether the original procedure can still be run with the same closed model, not the correctness of its conclusions.

The revised manuscript PDF also highlights role and track findings: judging rose from 1.47% of archival papers (2023) to 11.50% (2025); scientific closed-model use reaches 63.98% of NeurIPS 2025 Datasets & Benchmarks papers versus 21.32% of Main-track papers.

4. Updated code, evidence, and camera-ready commitment

We uploaded the updated supplementary code-and-data ZIP, including the original multi-stage pipeline: proceedings acquisition, PDF/OCR processing, model retrieval, evidence classification, local LLM adjudication, deterministic reconciliation, blinded validation, and analysis. Included are the registry, launchers/tests, frozen outputs, human labels, retirement evidence, paper-level joins, identifier audit, and regression checks.

Launchers specify digest-pinned Docker images for GROBID, OCR/vLLM, and local gpt-oss-120b adjudication, with serving settings and environment versions. Weights/images are not bundled; frozen outputs and scripts reproduce aggregates without rerunning those services.

Camera-ready commitment: We will extend the five-venue census through proceedings available at camera-ready, refresh lifecycle evidence to that cutoff, and regenerate the counts, figures, tables, and processed data. We will publicly release the full pipeline and complete processed data for the research community.

The revised manuscript PDF and updated artifacts now document a preservation problem already affecting the published research record.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.