- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors propose a framework for automatically testing hypotheses on a collection of articles. They describe two applications: how the submissions and reviews changed before and after 2024 (the first year after ChatGPT), and how the review content correlates with the acceptance or rejection of the paper. For both, the submissions and reviews from ICLR (2017-2026) are used. The framework starts with a predefined condition to test. Several hypotheses are automatically generated by repeatedly sampling articles from the two groups and prompting an LLM to return the hypotheses that discriminate the two groups best. A smaller LLM is then used to decide for each article whether the hypothesis matches the abstract of the article. A validation partition is used to select the hypotheses that discriminate best between the 2 groups. This is done by fitting a regression to the measured condition. The effect of the remaining set of hypotheses is measured on a separate test partition.
Critical strengths
- The proposed pipeline is fully automatic and general enough to be applied to any testing condition. The decisions behind the experimental setup are sound without any obvious issues.
Critical weaknesses
- The main issue with the paper is the lack of any kind of human validation of the steps involved in the pipeline. While the initial hypotheses generation step is difficult to evaluate, the pair classification task could be easily replicated. This is a critical step in the pipeline, and it is performed by a smaller model.
- The paper seems unnecessarily convoluted and difficult to follow. Once you get past this, the pipeline itself is not that complex.
Questions for authors
- The initial hypotheses generation step is not detailed enough. Are all selected abstracts added in the prompt? How many are selected? How does the number of documents influence the quality of generated hypotheses? Are the sampled articles completely random? Would clustering help by comparing more similar articles?
- Will the sets of hypotheses and labels be released publicly? They could represent an important result of this paper.
4 comments
Question about the corpus
Thanks for recognizing that our testbed is important. Yes. To preserve anonymity, we will release the full 2017–2026 corpus after the review process, which links 42,123 accepted or rejected submissions to 161,469 reviews, together with research-area annotations (model-imputed labels for earlier years), LLM-generated marker, their accompanied hypothesis, document-level labels, etc., which in detail shown in the field case in the updated Appendix A Table 2.
The release will also include reconstruction scripts, fixed paper-level proposal/validation/test splits, and code reproducing the reported analyses. We want to reclaim these linked resources provide a reusable testbed for comparing hypothesis-generation and verification methods under shared data and evaluation targets. They also support longitudinal studies of research topics and peer-review judgments, making them valuable for the ICLR and whole ML community.
Question about details
Prompt selection and sampling. In total, 12,549 abstracts and 48,376 reviews (30% of the corpus) are selected into the proposal split. GPT-5.6-Terra generates candidates from subsampled batches detailed in Appendices A.1 to A.3. Sampling is structured; the proposal pool is stratified by year and decision (Table 1), and each prompt conditions on the target contrast while alternating presentation orders to prevent position bias, drawing randomly across subfields within each group. We will surface these summary numbers directly in Section 4.1.
Effect of document count and hypothesis quality. Per-prompt document count balances small-sample noise against long-context attention dilution, whereas accumulating batches across the proposal pool governs search recall (Figures 7 and 8; Proposition 2). Crucially, because every proposed predicate is subsequently screened across thousands of validation documents (7,906 to 14,678 texts) before held-out testing, finite prompt sizes affect only recall rather than statistical validity. Our expert evaluation (Section 4.4 and Appendix K) further attests to the quality of hypotheses surfaced under this setup: ten domain experts achieved 95.4% initial agreement (1,831 of 1,920 pairs) when annotating the 24 discovered predicates and reproduced the LLM effect directions on 23 of the 24 claims.
Role of clustering. Grouping similar submissions prior to prompting is a promising way to suppress topic-shift confounders and isolate fine-grained, subfield-specific distinctions. In this study, we sampled broadly across subfields because our estimands target conference-wide norms, and we evaluated sensitivity to topic composition downstream using research-area fixed effects (Section 4.2). Consistent with this broad scope, experts spanning diverse ML specialties reliably validated the resulting claims (Table 7). When researchers instead target within-area estimands, cluster-conditioned prompting offers a natural extension for the future work of our paper.
We believe this response addresses your concerns, and we would be glad to provide further clarification if needed.
Thank you for the additional human validation of the predicates, this really helps. The updates made in the text also help by making the story more clear. Because of this, I have raised my overall score to 3.
Dear Reviewer NdE6,
Thank you very much for taking the time to review our revisions and for your positive feedback! This really helps strengthen our paper.
Best regards, The Authors
We thank the reviewer for recognizing the generality of our framework and the soundness of setup, and the suggestions for the further improvement.
Human validation
Building on our pre-submission spot checks, we now added an independent human validation study covering all 24 reported predicates (Section 4.4 and Appendix K according to the revised version). Ten domain experts with graduate-level training assessed 1,920 text–predicate pairs, with two independent judgments per pair, yielding 3,840 initial judgments. Annotators saw only the predicate and its associated text and were blinded to the model labels, metadata (e.g. year) and expected effect directions. Initial agreement was 95.4%; 89 pairs required adjudication, and three remained uncertain. Expert backgrounds, assignments, instructions, and adjudication details are provided in Appendix K.
The estimates underlying Figure 9 are reproduced below. S/R effects are in percentage points; C effects are in (10−3) nats per review.
* R5 and R6 retain one and two unresolved labels, respectively; the two estimates reflect alternative assignments of these labels, represented by the filled and open red points in Figure 9.
Human measurements largely preserve the direction of the reported effects. Human- and LLM-based estimates have the same sign for 23 of the 24 hypotheses, with 18 of the automated point estimates falling within human 95% confidence intervals.
For temporal analysis in Application 1, human judgments broadly corroborate shifts in contribution framing and novelty assessment. Human estimates preserve the combination of reduced theory-related framing (S5–S6) and fewer novelty-related criticisms (R1–R5), with intervals below zero for R1 and R5, suggesting that changes in contribution narratives accompany changes in the emphasis of reviewer scrutiny. Positive estimates for research-enabling datasets (S8) and contributions spanning data, methods, and evaluation (R8) are also consistent with a broader contribution narrative, although the integrated-pipeline estimate (S7) is close to zero.
Similarly, in the predictive study (Application 2), contribution clarity and evidential support remain informative beyond ratings and year. Judgments concerning contribution clarity (C4) and evidential support (C6) retain positive predictive gains beyond ratings and year, with human-based intervals above zero. The same holds for unjustified design choices (C1) and unclear novelty relative to prior work (C5). This reinforces the value of moving from (M_0) to (M_2).
We accordingly soften the increased emphasis on research-enabling contributions as tentative (revised manuscript, lines 23, 71, and 330).
Writing concern
We appreciate the advice. In this revised paper, we have improved the presentation by clarifying the workflow: explaining the steps before introducing notation and using concrete examples to explain the statistical targets. We have also reduced technical detail in the main text by moving the temporal-estimand derivation and the conditional mutual information discussion to the appendix.