acceptodds
Under review as a conference paper at ICLR 2027

Where the Growth of Peer Review Goes: A Decade of ICLR Discussions on OpenReview

Abstract

Peer review in AI and machine learning conferences now runs at a size that no reviewing system was designed for. ICLR itself grew between 2017 and 2026 with its review process in public on OpenReview. In this paper, we use this public record to understand how each part of peer review scales with the size of a venue, what drives the growth, and what the discussion still tells us about the reviewers' judgment. Each part is summarized by a power law in the number of submissions, with the exponent showing who absorbs the growth. We find that authors absorbed most of it, as their rebuttals grew faster per paper than the reviews that they answer, while public attention did not grow at all. A small group of prolific authors holds a growing share of all submissions, which exhibits the property of cumulative advantage. We also find that the discussion became routine, with reviewer follow-ups that are frequent, short, and polite, and the review wording predicts scores somewhat less well than it did. These findings show where the load of a growing venue lands and which rules affect it.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Peer review, AI conferences, data analysis, scaling laws

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_9ix4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper provides an in-depth analysis of ICLR peer review data from 2017 to 2026. Multiple conclusions and suggestions were made based on quantitative results. The paper tried to make its analysis independent of the growth of venue size. The paper first analyzed the number of submissions by existing authors and new authors joining the field. Sub-analyses such as public commenters, time zone, and tone of the reply were conducted. The second type of analysis is based on topic groups. The paper concludes with suggestions for the community.

Critical strengths
  • The analysis is in-depth and covers a wide range of relevant topics (although, as noted in the weaknesses, I also see this breadth as a potential drawback).

  • The source data is described thoroughly, and the choice of data is well justified.

  • The paper provides quantitative evidence for issues that have so far been discussed largely on an anecdotal basis.

  • The reviewer appreciates the suggestions made in the conclusion by the author.

Critical weaknesses
  • The major weakness in this paper is also its breadth of analysis. There are simply too many conclusions and analyses for a single paper. The findings cover a range of topics, including the number of authors per year, per team, topic coverage, reviewer tone, team size, etc. The reviewer acknowledges that the analyses conducted are important, and it is nice to see quantitative results. However, with the number of analyses and specific statistical analyses conducted for each analysis, it is simply too much for a 9-page conference paper. The paper reads more like a professional report, but it simply covers too many topics for a single paper.

  • Some analyses do not appear to contribute meaningfully to the paper’s main argument. For instance, the authors present the sub-finding that “Reviewers assess the safety group least positively in every year, and the LLM and RL groups most positively.” However, the authors themselves acknowledge that this result may simply reflect vocabulary that is common in the topic (e.g., attack, bias, and harm). While I appreciate the authors’ transparency, a finding that is likely driven by such a confound offers limited insight, and in its current form it reads more as a report of an analysis that was performed than as evidence supporting the paper’s claims. These kinds of analyses simply blur the topic and feel like they are throwing information at the reader. I would suggest streamlining the paper, or either removing such analysis (moving it to the appendix, or clarifying how it connects to the paper’s research questions).

  • It was very hard to understand “From the model to the decision quantities.” I would recommend providing more intuitive explanations and placing the math in the appendix.

Questions for authors

(Suggestion) It took me a great deal of time to understand how to analyze the table and figures. Currently, there are so many numbers that are aggregated into a single table (Tab 1). I would suggest splitting the table, or extracting the numbers that are only necessary to understand the author’s analysis. Also, it seems that comparing the exponents α\alpha is the key to understanding the paper’s analysis. As the direction and comparison of α\alpha play an important role, I would highly recommend that the authors provide descriptive notes (a direction of up or down) on how to analyze the α\alpha in the main paper.

  1. The analysis based on exponents is something that I am not very familiar with. Have there been works that uses a similar approach? If so, please provide some references for guidance.

  2. Please consider breaking up long sentences or providing additional details, as several sentences pack too much information into a single sentence. For instance, it took me some time to understand the following sentence: “The review form changed three times, so we train a text-to-score model on the reviews of years that share a form and evaluate it out of sample in every year, next to a tone-score fit on the assessment fields.” This one sentence explains that the form changed, how the model was trained, how it was evaluated, that a second model was fitted, and what the assessment fields are. Splitting it into two or three sentences would make it much easier to follow.

  3. The phrase “next to a tone-score fit” is ambiguous. It is unclear whether the tone-score fit serves as a baseline for comparison or as a separate, supplementary analysis. Please clarify the relationship between the two models.

  4. The link between the findings and their implications could be stronger. For example, it is left to the reader to interpret what it means that reviewer follow-ups became frequent, short, and polite. A sentence explaining why this matters (whether it suggests less substantive discussion) would strengthen the argument.

  5. Similarly, “e.g., who adds the extra papers or words and why” this short sentence could encompass two meanings. Is it the newcomers who are increasing the number of papers, or is it the existing researchers who are adding the papers? The paper is full of abstraction, which makes it really hard to understand. It also places the burden of interpreting the results on the reader. I think a more direct conclusion would substantially benefit the paper.

  6. How did the author define the busiest day?

  7. I might have missed the detail for this. What is the purpose of analyzing bullet lines per word?

Reviewer_zxZ2OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates how the peer-review process has evolved alongside the rapid growth of ICLR submissions from 2017 to 2026. Using historical OpenReview data, the authors analyze submission concentration, reviewer and author workloads, discussion behaviors, review language, and decision reliability. The paper introduces power-law scaling analyses to characterize the growth of different components, alongside text-based analyses of reviewer judgments and a statistical model of rating reliability. The authors report several findings regarding increasing rebuttal effort, concentrated authorship, declining public engagement, changes in reviewer discussions, and decreasing agreement in reviewer judgments.

Critical strengths

The paper addresses an important and timely problem concerning the sustainability and reliability of peer review under rapidly increasing submission volumes. The use of historical ICLR data provides a valuable opportunity to investigate how different aspects of academic reviewing have evolved over nearly a decade. The authors collect a substantial amount of data and examine several potentially interesting dimensions, including reviewer workload, rebuttal behavior, public participation, and rating consistency. The paper also makes an effort to account for changes in review forms and conference policies when comparing results across years. These observations could provide useful empirical evidence for future research on improving peer-review systems.

Critical weaknesses
  1. Lack of a coherent research focus and narrative. My primary concern is that the paper presents too many loosely connected research directions without establishing a clear central contribution. Although the stated goal is to understand how peer review scales, what drives its growth, and how reviewer judgment evolves, the analysis continuously introduces different observations without sufficiently developing their relationships. The paper reads more like a collection of independent statistical findings than a systematic investigation of a well-defined research question. Consequently, I found it difficult to identify which findings are the most important and what fundamental insight the paper intends to establish.

  2. Insufficient connections between findings and their interpretations. The paper investigates submission concentration, rebuttal length, public participation, reviewer politeness, text-score alignment, topic distributions, and rating reliability. However, these analyses are largely presented independently, with limited explanation of how one finding supports, contradicts, or explains another. Although the authors propose mechanisms for individual observations, many remain conjectural, and the broader relationships are not sufficiently established. As a result, the extensive analysis does not develop into a convincing explanation of how the peer-review system evolves as a whole.

  3. Unclear significance of the conclusions. The conclusion introduces another collection of suggestions for authors, reviewers, and area chairs, rather than synthesizing the findings into a small number of meaningful conclusions. Furthermore, the paper acknowledges that venue size and calendar year are highly correlated, making it difficult to establish whether the observed changes are driven by growth itself or other evolving factors. This further limits the interpretation of the proposed mechanisms. Overall, despite the large amount of analysis, I am not convinced that the paper provides sufficiently focused and well-supported insights beyond documenting historical trends.

Questions for authors
  1. What is the single most important research finding that the authors intend to establish? How do the different analyses collectively support this central contribution rather than functioning as independent observations?

  2. Could the authors clarify the relationships between the different findings, particularly how the changes in reviewer workload, discussion behavior, and rating reliability are connected? What evidence supports these relationships beyond their simultaneous evolution over time?

  3. Given the strong correlation between conference size and calendar year, how can the authors distinguish changes driven by submission growth from changes caused by conference policies, reviewer behavior, or broader developments in the research community?

Reviewer_Tf5AOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines how peer review has evolved as ICLR submissions increased 40-fold, from 490 in 2017 to 19,814 in 2026. Using public OpenReview records, the authors fit power-law relationships (q≈cNαq \approx cN^{\alpha}) to various review activities and interpret the exponents as indicators of how the growing workload is distributed.

The main findings suggest that rebuttal effort has grown faster than reviewing effort, publication activity has become increasingly concentrated among prolific authors, and discussions have become shorter and more formulaic. A signal-to-noise model further estimates that the proportion of meaningful information in reviewer scores declined from 0.56 to 0.33, leading the authors to characterize more than one-third of decisions as nearly random.

Critical strengths
  • The paper examines how conference growth affects peer review using comprehensive public OpenReview records. Publicly available code supports reproducibility.

  • Informative workload analysis. The finding that rebuttal effort grows faster than reviewing effort, primarily through the number rather than the length of messages, provides useful insight into where the additional workload falls.

  • Promising signal-ratio model. This is the paper's strongest contribution. The model is simple, interpretable, and potentially applicable to other conferences with publicly available review scores.

Critical weaknesses
  • Conference size and review activities naturally change together over time. Yet the paper interprets power-law exponents fitted to only 9–10 annual observations as evidence of "scaling" and workload redistribution.

  • Year and log submission count are almost perfectly correlated (r = 0.98). The authors themselves acknowledge that these exponents do not establish size effects and cannot distinguish power laws from alternative trends.

  • Consequently, the power-law formulation adds little explanatory value beyond descriptive temporal trends.

  • The estimated signal ratio (ρ) essentially measures inter-reviewer agreement, which may reflect shared biases rather than true paper quality. Using final post-discussion scores may also inflate agreement through reviewer convergence.The claimed 40% decline depends heavily on 2017, the smallest and earliest sample. From 2018 onward, the decline is much smaller (0.40 to 0.33). The "coin-flip" region, defined by acceptance probabilities between 25% and 75%, is somewhat arbitrary. Moreover, projecting the trend to 2035 and approximately 448,000 submissions is an extreme extrapolation from only nine observations.

  • Although the fixed-productivity model is rejected, the results do not distinguish cumulative advantage from productivity growth associated with academic career progression. The proposed mechanism, in which senior researchers increase productivity by supervising more students, is also not empirically tested.

  • Analyses of public comments, increasingly formulaic discussions, declining score predictability from review text, and topic clusters are presented without sufficiently explaining their implications or connections to the main research question. The conclusions and operational recommendations are therefore not clearly derived from the empirical findings.

Questions for authors
  • If the estimated exponents do not establish conference-size effects, what is gained by using power-law scaling rather than simply describing temporal trends?
  • How does the estimated decline change when excluding 2017 or using pre-rebuttal scores (available externally for 2024–2025)? How sensitive is the "coin-flip" region to its probability thresholds?
  • Does the evidence remain after controlling for academic age, measured by the year of an author's first appearance?
  • How sensitive is the difference between rebuttal and review effort to measurement choices, such as including all tracks or removing quoted text (Table C.8)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.