- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper provides an in-depth analysis of ICLR peer review data from 2017 to 2026. Multiple conclusions and suggestions were made based on quantitative results. The paper tried to make its analysis independent of the growth of venue size. The paper first analyzed the number of submissions by existing authors and new authors joining the field. Sub-analyses such as public commenters, time zone, and tone of the reply were conducted. The second type of analysis is based on topic groups. The paper concludes with suggestions for the community.
Critical strengths
-
The analysis is in-depth and covers a wide range of relevant topics (although, as noted in the weaknesses, I also see this breadth as a potential drawback).
-
The source data is described thoroughly, and the choice of data is well justified.
-
The paper provides quantitative evidence for issues that have so far been discussed largely on an anecdotal basis.
-
The reviewer appreciates the suggestions made in the conclusion by the author.
Critical weaknesses
-
The major weakness in this paper is also its breadth of analysis. There are simply too many conclusions and analyses for a single paper. The findings cover a range of topics, including the number of authors per year, per team, topic coverage, reviewer tone, team size, etc. The reviewer acknowledges that the analyses conducted are important, and it is nice to see quantitative results. However, with the number of analyses and specific statistical analyses conducted for each analysis, it is simply too much for a 9-page conference paper. The paper reads more like a professional report, but it simply covers too many topics for a single paper.
-
Some analyses do not appear to contribute meaningfully to the paper’s main argument. For instance, the authors present the sub-finding that “Reviewers assess the safety group least positively in every year, and the LLM and RL groups most positively.” However, the authors themselves acknowledge that this result may simply reflect vocabulary that is common in the topic (e.g., attack, bias, and harm). While I appreciate the authors’ transparency, a finding that is likely driven by such a confound offers limited insight, and in its current form it reads more as a report of an analysis that was performed than as evidence supporting the paper’s claims. These kinds of analyses simply blur the topic and feel like they are throwing information at the reader. I would suggest streamlining the paper, or either removing such analysis (moving it to the appendix, or clarifying how it connects to the paper’s research questions).
-
It was very hard to understand “From the model to the decision quantities.” I would recommend providing more intuitive explanations and placing the math in the appendix.
Questions for authors
(Suggestion) It took me a great deal of time to understand how to analyze the table and figures. Currently, there are so many numbers that are aggregated into a single table (Tab 1). I would suggest splitting the table, or extracting the numbers that are only necessary to understand the author’s analysis. Also, it seems that comparing the exponents is the key to understanding the paper’s analysis. As the direction and comparison of play an important role, I would highly recommend that the authors provide descriptive notes (a direction of up or down) on how to analyze the in the main paper.
-
The analysis based on exponents is something that I am not very familiar with. Have there been works that uses a similar approach? If so, please provide some references for guidance.
-
Please consider breaking up long sentences or providing additional details, as several sentences pack too much information into a single sentence. For instance, it took me some time to understand the following sentence: “The review form changed three times, so we train a text-to-score model on the reviews of years that share a form and evaluate it out of sample in every year, next to a tone-score fit on the assessment fields.” This one sentence explains that the form changed, how the model was trained, how it was evaluated, that a second model was fitted, and what the assessment fields are. Splitting it into two or three sentences would make it much easier to follow.
-
The phrase “next to a tone-score fit” is ambiguous. It is unclear whether the tone-score fit serves as a baseline for comparison or as a separate, supplementary analysis. Please clarify the relationship between the two models.
-
The link between the findings and their implications could be stronger. For example, it is left to the reader to interpret what it means that reviewer follow-ups became frequent, short, and polite. A sentence explaining why this matters (whether it suggests less substantive discussion) would strengthen the argument.
-
Similarly, “e.g., who adds the extra papers or words and why” this short sentence could encompass two meanings. Is it the newcomers who are increasing the number of papers, or is it the existing researchers who are adding the papers? The paper is full of abstraction, which makes it really hard to understand. It also places the burden of interpreting the results on the reader. I think a more direct conclusion would substantially benefit the paper.
-
How did the author define the busiest day?
-
I might have missed the detail for this. What is the purpose of analyzing bullet lines per word?