- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper introduces Suto as a persistent, closed-loop LLM-agent simulation framework for studying academic research ecosystems. The authors claim that Suto can simulate various research activities through persistent researcher states and a shared scientific record, which can support matched counterfactual experiments. Using the framework, the authors conduct empirical analysis to explore research questions related to rejection-driven resubmission, cautious exploration, and resource inequality.
Critical strengths
- The proposed framework is potentially interesting and impactful, which I believe could help researchers explore various questions in science of science. Specifically, I think there are multiple choices in the design of framework that can improve the simulation of research communities compared with previous work.
- The presented experiments are quite comprehensive, since the authors claim that they cover 61 simulation worlds, simulate over 40,000 researchers from 8,000 institutions, producing ~400,000 publication decisions and 1.2 million LLM-generated peer-reviewed. I believe such large-scale data could potentially benefit some downstream research.
Critical weaknesses
- The fatal problem in the paper is that there is no validation that the agents in the simulated world behave like researchers in the real world, which leads to significant concerns about the reliability of the findings.
- The proposed framework Suto presents limited novelty, as simulation systems of research communities driven by LLM agents have been extensively studied in previous work and there are no fundamental difference in the proposed framework. Also, multiple types of research activities are mixed together in the framework without sufficient discussions, which create doubts regarding whether the system can actually reflect the workflows of research activities in the real world.
- The empirical experiments also seem fragile and problematic. First, the authors only use Qwen3-32B for different simulations, which leaves significant concerns regarding the robustness of the findings. Second, the key claims in the paper lack sufficient statistical support, and many simulations actually don't use the full scale simulations as reported.
Questions for authors
- What is the methodological novelty of the proposed Suto framework compared with previous work?
- How can these findings be connected to the real data?
13 comments
Concerns about methodological novelty and the coupled research cycle (2/4)
The Related Work section distinguishes SUTO from prior work:
| Prior work | Focus and mechanism | Methodological distinction in SUTO |
|---|---|---|
| ResearchTown [1] | Agent–data graphs and TextGNN model paper reading, collaborative writing, and review generation. | SUTO is a closed-loop feedback system. Publication and funding outcomes continuously update budgets, research opportunities, and researcher participation across simulated years. |
| AgentReview [2] | Author–reviewer–chair interactions examine how review mechanisms and participant characteristics affect publication decisions. | Review decisions carry forward into resubmission histories, resource expenditure, funding assessment, and continued participation. |
| VirSci [3] | Collaborator selection and multi-round discussion support scientific idea generation, evaluation, and refinement. | Research strategies are evaluated through their longer-term publication, funding, and participation outcomes under resource constraints. |
| AgentRxiv [4] | A shared preprint repository allows autonomous laboratories to build cumulatively on their own and others’ findings. | The evolving scientific record is coupled to funding allocation, research costs, and researcher attrition. |
| CycleResearcher [5] | Research–review–refinement feedback improves research models through iterative preference training. | Institutional interventions change persistent researcher and ecosystem states, allowing their downstream consequences to be followed over time. |
| The AI Scientist / The AI Scientist-v2 [6, 7] | Autonomous ideation, experimentation, manuscript writing, and reviewing, with accumulated findings or iterative experimental and review feedback. | The simulation examines how publication and funding rules shape future research opportunities and the sustainability of an evolving researcher population. |
| AI-Researcher [8] | Research production from literature analysis to implementation and manuscript preparation, with recursive feedback between ideas and implementations. | Literature-grounded research activity is embedded in a population whose budgets and participation evolve through institutional decisions and research costs. |
| Agent-based models of science [9, 10] | Rule-based models study publication, resubmission, and resource feedback [9], or the evolution of research practices under publication incentives [10]. | Content-dependent LLM choices and evaluations operate on scientific texts and researcher histories within an explicitly controlled institutional system. |
Concerns about methodological novelty and the coupled research cycle (3/4)
These comparisons concern the mechanisms implemented and evaluated in the cited works. In particular, ResearchTown’s Algorithm 1 produces paper and review content for a target paper node; its Appendix D.4 identifies richer institutional and reputational dynamics as future work [1]. SUTO implements and evaluates the feedback from publication and funding decisions to subsequent resources, activity, and participation. AgentRxiv, CycleResearcher, and the AI Scientist systems already provide important forms of iteration and feedback [4–7]; SUTO’s distinction is the coupling of scientific activity to persistent institutional and population states.
2. The connections among activities are already specified in the methodology.
Sections 2.1–2.2 and Appendix F.6 specify the dependencies among these activities. Researchers retain their expertise, reputation, budget, memory, and conflicts of interest, while the environment maintains the scientific record. The activities interact through three concrete mechanisms:
- Review decisions affect subsequent submissions and resource use. Rejected manuscripts retain their reviews and submission histories. Later resubmission decisions depend on this history, available resources, and resubmission costs; successful resubmissions and new submissions compete within the same review process.
- Publication decisions affect subsequent research and funding. Accepted papers enter the citable literature and the authors’ publication records. These records inform later research choices and funding evaluations.
- Funding and research costs affect subsequent participation. Grants replenish budgets, while research activities and resubmissions consume resources. Researchers who exhaust their resources become inactive, changing the population that generates subsequent submissions and reviews.
These dependencies are the reason for modeling the activities jointly. An intervention in publication policy can alter manuscript histories, funding opportunities, resource consumption, and the future author and reviewer population. Holding those downstream states fixed would exclude precisely the consequences that SUTO is designed to study.
The framework also specifies temporal ordering: projects take time and consume resources, retrieved artifacts are restricted by simulated year, and accepted papers become available for citation in subsequent years. LLMs provide context-dependent choices and evaluations, while explicit institutional rules govern acceptance quotas, resource accounting, and attrition. This separation allows institutional mechanisms to be changed while retaining the same agent decision framework.
Concerns about methodological novelty and the coupled research cycle (1/4)
We respectfully disagree with the claim that SUTO has "no fundamental difference" from prior work.
Section 4 (Related Work) already explicitly states what distinguishes SUTO from prior work. SUTO connects research decisions, publication, resubmission, citation, and funding to researchers’ resources and continued participation within a persistent, closed-loop, literature-grounded ecosystem. Sections 2.1–2.2 specify these dependencies, and Appendix F.6 explains why carrying their consequences into subsequent years is necessary for studying the long-term effects of institutional interventions.
Both the distinction from prior work and the rationale for coupling these activities are therefore already present in the manuscript.
Our primary contribution is two-fold: 1) a framework for controlled longitudinal experiments on research ecosystems, and 2) the scientific findings obtained through those experiments. These findings show how rejection-driven resubmission amplifies reviewer burden, how cautious exploration balances career outcomes with scientific diversity, and how aggregate resource inequality can emerge alongside a near-zero local estimate of the effect of a marginal grant on subsequent funding.
Such combination of a proposed framework and substantive findings is well established in prior work ResearchTown [1], AgentReview [2], and agent-based studies of publication systems and scientific incentives [9, 10].
SUTO’s contribution should accordingly be assessed through the distinct experimental capabilities it introduces and the scientific insights those capabilities enable.