- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the prediction of outcomes of automatic research by modeling the multi-agent interactions as a stochastic growth process on a shared knowledge hypergraph. In the proposed framework, each agent is described by the phase-type distribution of its action duration and outcome, and the coarse-grained dynamics are averaged over the individual agents. The authors use formal theorem-proving tasks to evaluate the proposed model, and demonstrate that the model can predict how the completion time of many agents sharing a record changes with their number.
Critical strengths
- Predicting the outcomes of multi-agent interactions in automatic research is interesting and important. I also agree that statistical physics can provide valuable methods and insights for modeling the processes in the theoretical sense.
- The paper covers comprehensive theoretical analysis and empirical validations, and yields meaningful insights regarding predicting the outcomes of multi-agent systems in automatic research.
Critical weaknesses
- The theoretical contributions of the paper are significantly limited in the sense that the empirical implications do not match the theoretical complexity. I think most of the analysis, although reasonable, is conceptual, without deterministic insights regarding the behaviors of multi-agent systems.
- The empirical validations are also thin. The multi-agent test uses a single ETP world, and a single agent. The replication at large is also quite limited since only several values are tested, leaving significant concerns regarding the results of the paper. Also, the only baseline compared in the paper is with the own zeroth-order model, which is known to fail with a delay at synchronized start.
- Another big issue of the paper is the weak connection to automatic research. From the theoretical analysis to empirical validations, I think the methods used in the paper can generalize to many different multi-agent systems powered by LLMs, without pointing out the fundamental challenges in automatic research. Empirically, the paper also only studies theorem proving, which is not equivalent to many settings of automatic research studied in previous work.
Questions for authors
N/A
We thank the reviewer for the helpful comments and for recognizing the importance of predicting multi-agent automatic research.
The theory yields quantitative predictions computed from the single-agent action law alone, with nothing fitted to multi-agent campaigns. It predicts when adding agents stops dividing the completion time by N (the onset of competition, Eq. (17)), the floor of the completion time set by the fixed delay, and the scaling exponents above the onset. These predictions match the observed T50 and T80 exponents (Table 2). The same equations also give a new result for finite worlds: when difficulty stays with each question, the late stage slows down, and the slowdown grows with the number of agents per question. This prediction is supported out of sample (Table 1).
We extended the validation in three ways. First, we ran new campaigns at N=64,128,256 with a changed agent: under the same model identifier, it was about five times slower on single-agent attempts. Calibrated only on its own single-agent campaigns, the model predicts all three T50 values within the 90% intervals, and with question-specific difficulty two of the three T90 values (Table 1). Second, we compared against baselines that use the same single-agent data: an empirical-duration simulator, which resamples the recorded single-agent attempts, and ideal 1/N scaling with a fitted delay. The phase-kept model matches the simulator and reproduces the observed T50 and T80 exponents, while the fitted 1/N baseline does not (Table 2). Third, ablations show that the fixed delay and the phase structure jointly produce the departure from 1/N scaling. Beyond ETP, the single-agent action law is validated on three kinds of task with three agent frameworks and three models (Section 6.1), and the crowding predictions are tested in two Gymnasium worlds (Appendix E.5).
The model is built on features that distinguish research from other multi-agent tasks. Knowledge accumulates in a shared record, and one result can settle many other questions through logical closure; in ETP, a single refuting table can settle over a hundred implications. A question becomes answerable only once its premises are established. Researchers duplicate each other's work on the same open question. And a finite research program ends with its hardest questions, where additional agents help least. These features set the scaling laws, and generic multi-agent task models do not contain them. The experiments are also not limited to theorem proving: the action laws are tested on scientific law discovery in NewtonBench as well as on formal mathematics in Lean and ETP, and ETP is itself a collaborative mathematical research project. We will make these connections explicit in the revised paper.
Table 1. New campaigns with the changed agent
Times in seconds. Difficulty is estimated from the original agent's campaigns; all inputs were fixed before the new campaigns.
Table 2. Predictors using the same single-agent data