acceptodds
Under review as a conference paper at ICLR 2027

Science Utopia? Closed-Loop LLM Simulation of Academic Research Ecosystems

Abstract

Scientific progress emerges from a longitudinal ecosystem in which researchers, institutions, funding agencies, collaboration networks, and the scientific literature co-evolve. As AI becomes increasingly involved throughout the scientific research cycle, understanding these interconnected and evolving processes becomes increasingly important. We introduce SUTO, a persistent, closed-loop LLM-agent simulation framework for studying academic research ecosystems. SUTO models interconnected scientific processes such as research-direction choice, collaboration, submission, peer review, resubmission, citation, funding, and researcher attrition, while maintaining evolving states across simulated years. Its configurable institutional mechanisms and information channels provide a controlled testbed for matched counterfactual experiments and targeted interventions. Across 61 simulation worlds, SUTO simulates over 40,000 researchers from 8,000 institutions, producing over 400,000 publication decisions and 1.2 million LLM-generated peer reviews. Using these longitudinal simulations, we find that rejection-driven resubmission substantially amplifies reviewer burden beyond population growth alone, cautious exploration balances citation impact with career success and long-term topic diversity, and resource inequality can emerge even without detectable cumulative advantage from narrowly winning early funding. Code is available at https://anonymous.4open.science/r/ScienceUtopia.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Social Simulation, LLM Agents, Science of Science, Scientific Ecosystems, Ethics, Multi-Agent Systems, Equality

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gDdDOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces Suto as a persistent, closed-loop LLM-agent simulation framework for studying academic research ecosystems. The authors claim that Suto can simulate various research activities through persistent researcher states and a shared scientific record, which can support matched counterfactual experiments. Using the framework, the authors conduct empirical analysis to explore research questions related to rejection-driven resubmission, cautious exploration, and resource inequality.

Critical strengths
  1. The proposed framework is potentially interesting and impactful, which I believe could help researchers explore various questions in science of science. Specifically, I think there are multiple choices in the design of framework that can improve the simulation of research communities compared with previous work.
  2. The presented experiments are quite comprehensive, since the authors claim that they cover 61 simulation worlds, simulate over 40,000 researchers from 8,000 institutions, producing ~400,000 publication decisions and 1.2 million LLM-generated peer-reviewed. I believe such large-scale data could potentially benefit some downstream research.
Critical weaknesses
  1. The fatal problem in the paper is that there is no validation that the agents in the simulated world behave like researchers in the real world, which leads to significant concerns about the reliability of the findings.
  2. The proposed framework Suto presents limited novelty, as simulation systems of research communities driven by LLM agents have been extensively studied in previous work and there are no fundamental difference in the proposed framework. Also, multiple types of research activities are mixed together in the framework without sufficient discussions, which create doubts regarding whether the system can actually reflect the workflows of research activities in the real world.
  3. The empirical experiments also seem fragile and problematic. First, the authors only use Qwen3-32B for different simulations, which leaves significant concerns regarding the robustness of the findings. Second, the key claims in the paper lack sufficient statistical support, and many simulations actually don't use the full scale simulations as reported.
Questions for authors
  1. What is the methodological novelty of the proposed Suto framework compared with previous work?
  2. How can these findings be connected to the real data?
13 comments
AuthorsOpenReview ↗

Concerns about methodological novelty and the coupled research cycle (1/4)

The proposed framework Suto presents limited novelty ... there are no fundamental difference in the proposed framework.

Also, multiple types of research activities are mixed together in the framework without sufficient discussions, which create doubts regarding whether the system can actually reflect the workflows of research activities in the real world.

What is the methodological novelty of the proposed Suto framework compared with previous work?

We respectfully disagree with the claim that SUTO has "no fundamental difference" from prior work.

Section 4 (Related Work) already explicitly states what distinguishes SUTO from prior work. SUTO connects research decisions, publication, resubmission, citation, and funding to researchers’ resources and continued participation within a persistent, closed-loop, literature-grounded ecosystem. Sections 2.1–2.2 specify these dependencies, and Appendix F.6 explains why carrying their consequences into subsequent years is necessary for studying the long-term effects of institutional interventions.

Both the distinction from prior work and the rationale for coupling these activities are therefore already present in the manuscript.

Our primary contribution is two-fold: 1) a framework for controlled longitudinal experiments on research ecosystems, and 2) the scientific findings obtained through those experiments. These findings show how rejection-driven resubmission amplifies reviewer burden, how cautious exploration balances career outcomes with scientific diversity, and how aggregate resource inequality can emerge alongside a near-zero local estimate of the effect of a marginal grant on subsequent funding.

Such combination of a proposed framework and substantive findings is well established in prior work ResearchTown [1], AgentReview [2], and agent-based studies of publication systems and scientific incentives [9, 10].

SUTO’s contribution should accordingly be assessed through the distinct experimental capabilities it introduces and the scientific insights those capabilities enable.

AuthorsOpenReview ↗

Concerns about methodological novelty and the coupled research cycle (2/4)

The Related Work section distinguishes SUTO from prior work:

Prior workFocus and mechanismMethodological distinction in SUTO
ResearchTown [1]Agent–data graphs and TextGNN model paper reading, collaborative writing, and review generation.SUTO is a closed-loop feedback system. Publication and funding outcomes continuously update budgets, research opportunities, and researcher participation across simulated years.
AgentReview [2]Author–reviewer–chair interactions examine how review mechanisms and participant characteristics affect publication decisions.Review decisions carry forward into resubmission histories, resource expenditure, funding assessment, and continued participation.
VirSci [3]Collaborator selection and multi-round discussion support scientific idea generation, evaluation, and refinement.Research strategies are evaluated through their longer-term publication, funding, and participation outcomes under resource constraints.
AgentRxiv [4]A shared preprint repository allows autonomous laboratories to build cumulatively on their own and others’ findings.The evolving scientific record is coupled to funding allocation, research costs, and researcher attrition.
CycleResearcher [5]Research–review–refinement feedback improves research models through iterative preference training.Institutional interventions change persistent researcher and ecosystem states, allowing their downstream consequences to be followed over time.
The AI Scientist / The AI Scientist-v2 [6, 7]Autonomous ideation, experimentation, manuscript writing, and reviewing, with accumulated findings or iterative experimental and review feedback.The simulation examines how publication and funding rules shape future research opportunities and the sustainability of an evolving researcher population.
AI-Researcher [8]Research production from literature analysis to implementation and manuscript preparation, with recursive feedback between ideas and implementations.Literature-grounded research activity is embedded in a population whose budgets and participation evolve through institutional decisions and research costs.
Agent-based models of science [9, 10]Rule-based models study publication, resubmission, and resource feedback [9], or the evolution of research practices under publication incentives [10].Content-dependent LLM choices and evaluations operate on scientific texts and researcher histories within an explicitly controlled institutional system.
AuthorsOpenReview ↗

Concerns about methodological novelty and the coupled research cycle (3/4)

These comparisons concern the mechanisms implemented and evaluated in the cited works. In particular, ResearchTown’s Algorithm 1 produces paper and review content for a target paper node; its Appendix D.4 identifies richer institutional and reputational dynamics as future work [1]. SUTO implements and evaluates the feedback from publication and funding decisions to subsequent resources, activity, and participation. AgentRxiv, CycleResearcher, and the AI Scientist systems already provide important forms of iteration and feedback [4–7]; SUTO’s distinction is the coupling of scientific activity to persistent institutional and population states.

2. The connections among activities are already specified in the methodology.

Sections 2.1–2.2 and Appendix F.6 specify the dependencies among these activities. Researchers retain their expertise, reputation, budget, memory, and conflicts of interest, while the environment maintains the scientific record. The activities interact through three concrete mechanisms:

  • Review decisions affect subsequent submissions and resource use. Rejected manuscripts retain their reviews and submission histories. Later resubmission decisions depend on this history, available resources, and resubmission costs; successful resubmissions and new submissions compete within the same review process.
  • Publication decisions affect subsequent research and funding. Accepted papers enter the citable literature and the authors’ publication records. These records inform later research choices and funding evaluations.
  • Funding and research costs affect subsequent participation. Grants replenish budgets, while research activities and resubmissions consume resources. Researchers who exhaust their resources become inactive, changing the population that generates subsequent submissions and reviews.

These dependencies are the reason for modeling the activities jointly. An intervention in publication policy can alter manuscript histories, funding opportunities, resource consumption, and the future author and reviewer population. Holding those downstream states fixed would exclude precisely the consequences that SUTO is designed to study.

The framework also specifies temporal ordering: projects take time and consume resources, retrieved artifacts are restricted by simulated year, and accepted papers become available for citation in subsequent years. LLMs provide context-dependent choices and evaluations, while explicit institutional rules govern acceptance quotas, resource accounting, and attrition. This separation allows institutional mechanisms to be changed while retaining the same agent decision framework.

Reviewer_U5S1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors have introduced SUTO, a LLM based world aiming to simulate, how research ecosystem will evolve, considering factors like collaboration, funding, extend of exploration by researchers, research direction choice, citation, paper submissions etc. Researchers retain memories and resources, allowing earlier outcomes to influence later decisions. Across 61 simulated worlds, the authors find that resubmissions increase review pressure, higher publication output can coincide with declining researcher participation, and exploring topics close to existing expertise balances citation impact, career success, and long term topic diversity. Its main contribution is a unified platform for testing how academic rules affect an evolving research ecosystem.

Critical strengths
  1. The paper is well written and easy to follow.
  2. The metrics used to explain the dynamics are carefully selected and well explained.
  3. The simulation framework allows us to study the impact of different factors within the research ecosystem at a high level.
  4. The authors have done comprehensive studies and presented the trends in a clear way.
  5. The main results indicate cautious explorers gained more citation impact than the other groups and produced a world with more topic variety down the lane.
Critical weaknesses
  1. Closed loop simulation assumes that research outcomes shape the subsequent research. This may not be fully true in a real setting as authors can pursue small and low hanging fruit ideas alongside their major research topics. Sometimes the researchers might even abandon their current line of work and pursue something that is more in trend or something which can provide more funding at that point of time.
  2. The collaboration assumes there is an overlap in research topic or publication overlap. In real life, the collaboration can also be tied to funding (when multiple agencies write proposals together), or each collaborator has complementing research expertise, which when combined can be useful etc. These real affinities are not represented in the simulation.
  3. The funding institutions systems simulated here are purely LLM based and does not use any characteristic or trends from the past, looking at topics they have approved, given a call for proposals.
  4. Results are presented only for a single LLM. It would be interesting to see if the trends change if a more powerful model is used.
  5. The paper does not provide any examples of initial state or trajectory used in simulation.
  6. The idea of using a closely related paper as submission seems to be a very crude way of simulation. The LLMs already have the knowledge of this paper, acceptance and review . It is unclear how that will influence the outcome of the review.
  7. The only real worlds data seems to come from SciEvo corpus. I am wondering it the results would change if historical data was used in informing any of the agents used here, before they take a decision or provide score.
  8. No new knowledge is really tested here.
  9. The paper shows trends, but a practical explanation of why such trends occur is not discussed in great detail.
Questions for authors
  1. Can the authors report some of the main metrics or plots with a different LLM?
  2. Can the author provide some examples of trajectories of a simulation?
  3. Can the authors show some experiments to demonstrate how the prior knowledge of LLM about the papers which are being used as submission, influences the decision?
  4. Can the authors provide some comparison of the simulated results with some historical data on any of these metrics? Doing this can help establish confidence in this simulation framework or it it differs, we can understand why.
  5. Can the authors provide some experiments on how some of the key metrics change with small changes in prompt?
  6. Can the authors comment on or demonstrate, why the results from this simulation are reliable?
11 comments
AuthorsOpenReview ↗

Considering more scenarios in the real-world (1/2)

  1. Closed loop simulation assumes that research outcomes shape the subsequent research ... Sometimes the researchers might even abandon their current line of work and pursue something that is more in trend or something which can provide more funding at that point of time.
  1. The collaboration assumes there is an overlap in research topic or publication overlap ... In real life, the collaboration can also be tied to funding ...

We thank the reviewer for highlighting these scenarios.

We study a clearly specified set of research activities through controlled counterfactual comparisons. Researchers may pursue smaller projects alongside a main agenda, move toward emerging or better-funded topics, form teams around joint funding opportunities, take visiting appointments in industry to seek research support, or leave faculty positions for industry. These possibilities involve distinct behavioral mechanisms.

SUTO's closed-loop design allows researchers to change direction as their resources, opportunities, and research histories evolve. We distinguish behaviors already supported by the framework from additional mechanisms that can be studied through controlled extensions.

  1. Research direction changes are already modeled.

As described in §2.1–2.2, researchers select new directions at project boundaries based on their expertise, strategy, resources, and accumulated memory, including prior publication, review, and funding outcomes. They can also abandon, revise, or resubmit rejected work. Thus, researchers can change their research directions as their circumstances evolve. Concurrent side projects and explicit responses to emerging trends or funding priorities can be added as separate, controlled interventions.

  1. Collaboration does not require topical overlap.

Our collaboration mechanism combines four signals: topical similarity, recent-publication overlap, preferential attachment, and triadic closure (§2.2). Neither shared topics nor shared publications is a mandatory condition for forming a collaboration. This design is empirically motivated: prior studies identify the importance of research interests and existing collaboration networks [1–3]. Notably, Smith et al. [4] find that collaboration is most likely at low-to-moderate topic overlap, consistent with the value of combining related but distinct expertise. We agree that explicitly optimizing for complementary skills or joint funding opportunities would capture additional collaboration incentives.

AuthorsOpenReview ↗

Considering more scenarios in the real-world (2/2)

  1. Funding is governed by explicit institutional characteristics.

SUTO distinguishes foundation-oriented and mission-oriented programs, with specified topical priorities, eligibility requirements, and ranked funding-allocation rules. Evaluations consider proposal fit, research potential, and applicants' publication histories. LLM evaluations inform funding allocation within explicit eligibility, topical-priority, and ranking rules. The current experiments use fixed program priorities. Historical award portfolios provide a basis for future experiments with evolving priorities.

  1. Controlled counterfactuals isolate the effects of institutional interventions.

SUTO's primary contribution is to study how specified institutional mechanisms propagate through a persistent, closed-loop research ecosystem. Within each matched comparison, we maintain consistent initial conditions and background behavioral rules while varying the intervention of interest. Agents still adapt to resulting changes in their research outcomes, resources, and opportunities.

Holding background behavioral rules fixed makes the institutional comparisons interpretable: differences in publication, reviewing, funding and participation can be traced through the specified intervention and its subsequent feedback. Each suggested behavior can then be varied in its own matched comparison.

References

  1. Zeng et al. Impactful scientists have higher tendency to involve collaborators in new topics. PNAS, 2022.
  2. Feng et al. Mixing Patterns in Interdisciplinary Co-Authorship Networks at Multiple Scales. Scientific Reports, 2020.
  3. Guimerà et al. Team Assembly Mechanisms Determine Collaboration Network Structure and Team Performance. Science, 2005.
  4. Smith et al. Great minds think alike, or do they often differ? Research topic overlap and the formation of scientific teams. Journal of Informetrics, 2021.
AuthorsOpenReview ↗

Temporal / historical grounding and paper familiarity (1/2)

  1. The idea of using a closely related paper as submission ... The LLMs already have the knowledge of this paper, acceptance and review . It is unclear how that will influence the outcome of the review.
  1. The only real worlds data seems to come from SciEvo corpus. I am wondering it the results would change if historical data was used ...

We would like to clarify that our simulation already includes controls on temporal information and access to real review outcomes.

Submissions and citations are restricted to literature available by the simulated year (Appendix F). For example, a simulation in 2016 accesses literature available by 2016. Acceptance is determined from generated review scores under the conference rule. The source paper’s historical acceptance decision is not provided as an input.

Our use of real scientific artifacts follows ResearchTown [1], VirSci [2], and AgentReview [3]. Related safeguards include restricting publication histories chronologically in ResearchTown [1], testing potential temporal leakage in VirSci [2], and combining temporal instructions with restricted retrieval in TimeChara [4]. These controls limit explicit access to future information.

Hiding titles produces small average score changes. Following prior work on familiarity and contamination checks [5, 6], we compared reviews of 600 papers with and without titles, keeping abstracts and three reviewer profiles per paper fixed. Each paper’s score is averaged across the profiles.

ModelMean score change (95% CI)Spearman ρ with original rankings (95% CI)
Qwen3-32B−0.021 [−0.043, 0.001]0.789 [0.751, 0.823]
Qwen3.5-397B-A17B-FP8−0.002 [−0.016, 0.012]0.764 [0.721, 0.801]

Both models retain substantial rank agreement with minimal changes in mean scores. This robustness to title masking provides empirical support for using retrieved papers as simulated submissions, with evaluations remaining broadly consistent when paper-title cues are removed.

Publication history improves next-topic prediction. To evaluate historical conditioning [7], we predicted the primary topic of 1,000 researchers' first 2021 paper using expertise alone or adding their 2016–2020 publication records. Target papers were withheld from the inputs.

Reviewer_C8tGOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents SUTO, a persistent LLM agent simulation of the academic ecosystem that couples topic choice, collaboration, submission, peer review, resubmission, citation, funding, and researcher attrition in an annual closed loop. Agents select real arXiv papers as research artifacts and carry state across simulated years. Across 61 worlds, the authors run matched counterfactual experiments on resubmission, output per project, exploration strategies, funding cutoffs, and author disclosure in review.

Critical strengths

The main contribution is the integrated framework. Previous agent-based models of science, with or without LLMs, address isolated mechanisms such as peer review, funding, or citation. To my knowledge, this is the first simulation that couples the full research cycle in a single longitudinal loop, at a scale of tens of thousands of agents. The matched counterfactual design makes it possible to trace how an institutional rule propagates into long-term outcomes. The released corpus of trajectories and 1.2 million reviews is a useful resource for the science-of-science community. These points support acceptance, because the framework opens a line of work that others can build on.

Critical weaknesses
  1. Most findings follow directly from the simulation rules. Each submission receives three reviews, so more resubmissions produce more reviews by construction, as the authors acknowledge in Appendix B.4. Population growth leaves per-reviewer load unchanged because reviewers and submissions grow together. Attrition follows from fixed costs against a fixed budget. The rubric asks reviewers to place most papers at 2.0 to 2.5, which explains why scores remain borderline. The few nontrivial results lack power or validation. The null effect of narrow funding wins has a confidence interval of the same magnitude as the grant itself. The selection and impact trade-off for distant papers reproduces a known empirical bias against novelty, but the paper does not present it as validation. No result is calibrated against real scientometric data, unlike closely related work such as ResearchTown. This limits what the simulation can tell us about real academia.

  2. The action space excludes the strategic decisions that drive real dynamics. All venues share a fixed 30% acceptance rate and have no prestige. Venue choice in practice is multifactorial and depends on topic, acceptance probability, and prestige. Without this, a resubmission is a repeated draw in an equivalent venue, which conditions the resubmission and reviewer load findings. Collaboration is never analyzed as a strategy to increase output. Agents do not produce content, so paper quality is fixed by the corpus. This design cannot address the volume-versus-quality question that motivates the paper through AI-assisted productivity. Resubmitted papers also receive fresh reviews, although the acceptance rates used come from ACL, which adopted rolling review to reuse reviews across venues. Testing such mechanisms would make better use of the framework than the current experiments.

Questions for authors

How do resubmission cascades and reviewer load results change when venues differ in acceptance rates and prestige, and when agents can take both into account when choosing where to submit?

Are the main dynamics invariant to world size, given that experiments use worlds from 300 to 5,000 researchers, and can any result be compared against empirical data?

11 comments
AuthorsOpenReview ↗

Concerns about the score-distribution instruction

The rubric asks reviewers ... why scores remain borderline.

The scoring instruction is intentional: it aims to improve simulation realism by counteracting LLM reviewers’ tendency toward overly generous scores. Empirical studies find that general-purpose LLMs produce more favorable assessments than human reviewers [1,2]. For example, [1] reports average ratings of 6.9–8.1 for general-purpose LLMs versus the borderline score 5.4 for human reviewers on the same 400 papers, normalized to a ten-point scale. Prior work also uses venue-specific guidelines and historical statistics to mitigate inflated review scores [3]. Our instruction similarly provides a conservative scoring baseline, reserving high scores for strong contributions.

Acceptance depends on relative rankings within each venue at its specified acceptance rate. We tested sensitivity to the instruction using an identical set of 600 papers and three fixed expert profiles per paper:

ModelMean score change after removing the instructionSpearman rank agreement with the original
Qwen3-32B+0.4420.810
Qwen3.5-397B-A17B-FP8+0.3550.832

The Spearman correlation with respect to the original remains very high at over 0.8.

Thus, removing the instruction retains substantial agreement in paper rankings. The guidance therefore adjusts absolute score levels as intended for simulation realism, with relative assessments remaining substantially consistent across the ablation.

References

  1. Idahl and Ahmadi. “OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews.” NAACL System Demonstrations, 2025.
  2. Sharma, Joachims, and Dean. Do LLMs Favor LLMs? Quantifying Interaction Effects in Peer Review. FAccT, 2026.
  3. Tyser et al. AI-Driven Review Systems: Evaluating LLMs in Scalable and Bias-Aware Academic Reviews. arXiv:2408.10365, 2024.
AuthorsOpenReview ↗

Concerns about empirical grounding and the novelty–impact relationship

Concerns about empirical grounding and the novelty–impact relationship

The selection and impact trade-off for distant papers reproduces a known empirical bias against novelty, but the paper does not present it as validation.

No result is calibrated against real scientometric data, unlike closely related work such as ResearchTown.

We agree that the novelty–impact relationship should be explicitly presented as qualitative validation.

This pattern is already reported in Figure 3c of the original manuscript. We will clarify its validation role and incorporate additional empirical evaluations of institutional selection and research-direction choice.

  1. Qualitative agreement with established empirical patterns.

As reported in Figure 3c, SUTO's acceptance rate decreases from 48.0% to 21.8% with increasing topical departure, while age-3 citations among accepted papers increase from 0.95 to 1.98. This reproduces the qualitative tension between evaluation risk and potential impact documented in empirical science-of-science studies [1–4].

  1. Empirical calibration and behavioral evaluation.

We further evaluate two components of SUTO against observed academic decisions, alongside ResearchTown's task-level validation of research and review outputs [6].

Institutional selection. Using 13,738 decided papers from the ICLR 2026 OpenReview archive [7], we calibrate SUTO's acceptance quota on 2,748 papers and evaluate its existing mean-rating ranking rule on 10,990 held-out papers. The calibrated evaluation yields 81.00% decision accuracy (95% Wilson CI: 80.26–81.72%) and 79.87% balanced accuracy, compared with 80.38% and 77.32% under the default 30% quota. This validates the institutional selection rule.

Research-direction choice. We evaluate whether publication history improves prediction of real researchers' subsequent topics. For 1,000 researchers, we predict the primary topic of their first 2021 paper using expertise alone or additionally providing their 2016–2020 publication histories. Target papers are withheld.

ModelExpertise onlyWith publication historyGain in percentage points (95% CI)
Qwen3-32B50.1%57.0%+6.9 [4.0, 9.8]
Qwen3.5-397B-A17B-FP854.6%59.5%+4.9 [2.5, 7.3]

With publication histories, both models also outperform the 51.7% previous-topic baseline. Confidence intervals use paired bootstrap resampling over researchers. These results provide empirical evidence that historical publication records improve the prediction of subsequent research choices.

  1. SUTO tests the longitudinal consequences of these mechanisms.

Beyond reproducing known individual-level novelty–impact patterns, SUTO examines how research strategies interact with repeated publication selection, funding, and researcher attrition to shape ecosystem-level outcomes. As already reported in Figure 3a–b, across ten matched composition comparisons, cautious-heavy populations achieve 0.11 higher normalized topic entropy and 8.5 more active research directions at year ten than exploiter-heavy populations. They also exceed explorer-heavy populations in ten-year diversity AUC by 0.156 on average. These findings extend the individual strategy–impact relationship studied by Yang et al. [5] to the long-term persistence of scientific diversity under institutional selection and resource constraints.

References

  1. Foster et al. Tradition and Innovation in Scientists’ Research Strategies. American Sociological Review, 2015.
  2. Boudreau et al. Looking Across and Looking Beyond the Knowledge Frontier: Intellectual Distance, Novelty, and Resource Allocation in Science. Management Science, 2016.
  3. Wang et al. Bias against novelty in science: A cautionary tale for users of bibliometric indicators. Research Policy, 2017.
  4. Uzzi et al. Atypical combinations and scientific impact. Science, 2013.
  5. Yang et al. Conservative adventurers have more future academic impact. Scientometrics, 2026.
  6. Yu et al. ResearchTown: Simulator of Human Research Community. ICML, 2025.
  7. Yang et al. Paper Copilot: Tracking the Evolution of Peer Review in AI Conferences. ICLR 2026.
AuthorsOpenReview ↗

Concerns about simulation rules and findings (1/2)

Most findings follow directly from the simulation rules ...

Each submission receives three reviews, so more resubmissions produce more reviews by construction, as the authors acknowledge in Appendix B.4.

Our research question concerns the magnitude, timing, shape, and distribution of review demand, together with its joint evolution with publication output, first-submission activity, and researcher participation.

In SUTO, publication and funding outcomes update researchers’ records and resources, which in turn affect later submissions, resubmissions, and continued participation. The three-review rule fixes reviews per submission, but it does not determine how many submissions occur over time, when review demand peaks or contracts, or how the active researcher population evolves. These longitudinal interactions are the substantive findings of the simulation.

  1. Resubmission increases publication output while reducing first-submission activity and continued participation. We conducted a 10-year comparison with the same set of initial researchers and no population influx. We use identical venues and complete, consistent funding rankings. Both conditions retain the same funding and annual research-cost rules.
OutcomeResubmission offResubmission onChange
Accepted papers, ten-year total5311,045+96.8%
First-submitted papers, ten-year total1,7771,501−15.5%
New review reports, ten-year total5,33110,431+95.7%
Active researchers at year ten240154−35.8%
Annual reviews / year-end active researcher, year ten2.1255.2792.48×

Higher publication output accompanies reduced first-submission activity and participation. By year ten, repeat submissions account for 173 of 271 attempts, while first submissions number 98, compared with 170 in the control.

Annual reviews also peak and then contract, falling 49.5% from 1,611 in year six to 813 in year ten. Despite this decline, endpoint demand remains 1.59× the control in total and 2.48× per year-end active researcher. Thus, declining aggregate reviewing accompanies a smaller participating population with persistently elevated normalized demand (annual review volume divided by year-end active researchers).

2. Population growth moderates review demand relative to the active population.

Population growth moderates normalized demand while amplifying total submission volume when combined with resubmission. A 10-year comparison measures this interaction:

Year-ten outcomeBaselinePopulation growthResubmissionBoth
Submissions193364489867
Annual reviews / year-end active researcher2.252.076.645.48
Active / all researchers257/300527/600221/300475/600

Population growth alone leaves normalized demand close to baseline. With resubmission enabled, however, adding growth increases annual submissions by 77.3% (489→867) while reducing normalized demand by 17.5% (6.64→5.48, calculated from unrounded counts). The expanding active population therefore offsets part of the additional demand, but normalized demand remains well above baseline.

The joint condition also produces 867 submissions, exceeding the additive reference of 364 + 489 − 193 = 660 by 207 submissions, or 31.4%. This is a descriptive interaction on the submission-count scale in these matched trajectories. Together, the results show how the same population-growth intervention can increase system-wide demand while moderating demand relative to the active population.

Within heterogeneous venues with hidden attributes, enabling resubmission increases accepted papers from 285 to 950 while reducing first-submitted papers by 8.9% and the year-ten active share by 32.0 percentage points. Review demand peaks at 1,848 reports in year six, then falls 48.1% to 960 in year ten. At that endpoint, total demand remains 2.05× the matched control (960 versus 468 reports), while demand per year-end active researcher is 3.40× the control (6.58 versus 1.93). This comparison identifies the peak, subsequent contraction and participation response under unequal venue selectivity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.