- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work studies the prevalence and parctices of using closed AI models in research acorss ML/NLP conferences in the past few years.
Critical strengths
- this is a hugely interesting research question
Critical weaknesses
- the scope and depth is limited
Questions for authors
Honestly I hugely appreciate the authors doing this: doing AI research with open, reproducible models should be encouraged and be the norm, and the research question of 1) how prevalent is closed models in research, and 2) what does that mean to AI research, is hugely important and critical. It is ``heartbreaking'' to see this important question not studied to its fullest in here to warrant acceptance.
-
Are the 4 categories in Table 1 exhaustive and non-overlapping? For example, if a paper on synthetic data generation uses a closed model for it, does it count as direct or data generation?
-
People are still using Gemini 1 in 2025, according to Figure 4? How is model version deteceted in this pipeline? Is that accurate? Can the authors give a few specific example papers where they said they used Gemini 1 in 2025?
-
The result sections felt hollow, just like the huge blank spaces on page 5 and throughout the paper. There is so much left out in this study, a few of them include:
-
When closed models are used, do people preserve their outputs and share them? This could serve as a compromise? How many of those outputs are still accessible?
-
How does drift impact things? For example, for usages of the same closed models, any cases where the numbers/conclusions changed between different timed usages?
The authors are encouraged to significantly beef up the analysis part.
- Would some policy-level levers to advocate for open models in research make sense? For example, the main conclusions in every paper has to be supported by at least one open-weight model, or risk desk rejection?
Clarifications, Additional Analyses, and Revision Plans
Thanks for the thoughtful review. We share the concern and appreciate your suggestions.
Our 53,538-paper census is complete at paper × model resolution, linked to year, venue/track, roles, providers and reported identifiers. The submitted ZIP contains frozen outputs, detailed tables, validation/lifecycle data, registry and pipeline code. We intend to release the pipeline and processed data publicly, especially for a paper about preservation.
1. Are the four roles exhaustive and non-overlapping?
They are non-exclusive (Sections 2.2 and 3.1). Using a closed model to synthesize study material is Data Generation; it can also be Direct if the model separately serves as an experimental system. 61.6% of Scientific Use papers have at least two roles and 25.1% have three or four. The four roles define Scientific Use; writing/coding assistance is separate Supporting Use.
2. Gemini 1 in 2025 / version detection / examples
Figure 4 plots generational families (text and caption). "Gemini 1" means Gemini 1.x, including 1.5, not 1.0 alone. The registry/tables retain finer identities. In 2025, Gemini 1.5 Pro appears in 680 papers, 1.5 Flash in 393 and 1.0 Pro in 219 (non-exclusive).
For example: Training Language Models to Self-Correct via Reinforcement Learning (ICLR 2025) uses 1.0 Pro and 1.5 Flash; Towards Adapting Open-Source Large Language Models for Expert-Level Clinical Note Generation (Findings of ACL 2025) uses 1.0 Pro as a teacher. Both are in our corpus. Publication year is not invocation year (Section 4.4). We can relabel Figure 4 "Gemini 1.x".
3. On the depth of the results
The paper already goes beyond the 33.32% headline: role trends/overlap, venue/track differences, multi-provider use, model-family footprints, lifecycle and identifier specificity. Only 16.3% of Scientific Use papers are baseline-only; Judge prevalence rises from 1.47% to 11.50%; NeurIPS 2025 Datasets & Benchmarks has 63.98% Scientific Use versus 21.32% in Main. These show how closed models enter scientific evidence, not just their frequency.
We prioritized broad field-level readings because of the nine-page conference format, not because the analysis stops there. The full paper × model × year × venue/track analysis already exists, with frozen outputs and code submitted. We can recombine it for model trajectories, role × venue/year, provider combinations, identifier-level patterns, lifecycle × role and more, without redoing the census. We would welcome suggestions on additional analyses that could strengthen the study, and will try to incorporate them in the revision.
3.1. Are outputs preserved and shared?
Our objects of study were the 53,538 archival papers. Their preservation and availability made this scale possible, although building a reliable paper × model census still took substantial work. Experimental artifacts are another research object: code, prompts, generated data and outputs are not archived or discoverable as consistently as papers. Assessing what was preserved would be a separate study with different coverage and access challenges. Even a repository/link audit would tell us only what is publicly discoverable, not whether full experimental traces were ever retained or made shareable.
Section 5 recommends archiving material inputs/outputs for closed judges and generators. Traces preserve evidence from the original run, but not the ability to reuse a retired judge on new systems or regenerate material under corrected protocols. We can emphasize this in revision.
3.2. Has drift changed numbers or conclusions?
Model changes can affect results, especially for judges. But papers alone cannot tell us which historical results changed. That requires original inputs/settings and access to comparable historical model states. Authors may report only a broad name or mutable alias, leaving the checkpoint unidentified; identified checkpoints may also be retired. Comparing distinct released checkpoints is not evidence of drift in the same checkpoint. Our census maps identification and availability problems, not a rate of changed conclusions. We can clarify this scope.
4. Policy levers
Table 3 proposes preservable references for acceptance-critical evidence and closed comparisons where the claim requires that model/capability. A blanket open-weight rule can change the question when the proprietary model is the object of study. We favor avoiding unnecessary non-preservable dependencies and documenting necessary ones.
The urgency is concrete. Our lifecycle cutoff was September 24, 2026. On October 23, OpenAI shut down
gpt-4-0613,gpt-4-1106-preview,gpt-4o-2024-05-13andgpt-4-turbo-2024-04-09. These occur in 480, 393, 376 and 233 papers in our submitted data (non-exclusive). The artifacts disappeared during review.Thanks again. We welcome further suggestions and will do our best to incorporate them in revision. Looking forward to further discussions.