acceptodds
Under review as a conference paper at ICLR 2027

Estimand-Aligned Natural-Language Hypothesis Discovery: Probing What the ML Community Recognizes as a Contribution

Abstract

Natural-language hypothesis discovery can reveal interpretable patterns in text, but the statistical objectives used to select them may not align with the population claims researchers seek to investigate. We introduce estimand-aligned natural-language hypothesis discovery, a framework in which a declared research design guides candidate evaluation and selection. Document-level measurements make predicates testable, and effect estimates with uncertainty quantify their support under the specified design. We develop a testbed for natural-language discovery about the machine-learning community, drawing on a decade of public submissions and peer reviews from a broad-scope venue, including accepted and rejected papers. We use this testbed to probe what the machine-learning community recognizes as a contribution. Across the transition to the LLM era, our temporal analysis suggests that enabling further research is gaining prominence as a claimed contribution through reusable datasets and integrated pipelines. Papers foreground theoretical contributions less, yet reviews express fewer novelty objections and more requests for specific forms of empirical support. Additional decision analysis finds that judgments about contribution clarity and evidential support improve acceptance prediction beyond a rating-and-year baseline. Accounting for rating-dependent associations reveals additional predictive value in novelty criticism. Together, the applications illustrate how an explicit statistical target shapes what can be learned about scientific contributions and their evaluation.

Primary area:
interpretability and explainable AI
Keywords:
Natural-language hypothesis discovery, Estimand alignment, Science of science, Peer review

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NdE6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose a framework for automatically testing hypotheses on a collection of articles. They describe two applications: how the submissions and reviews changed before and after 2024 (the first year after ChatGPT), and how the review content correlates with the acceptance or rejection of the paper. For both, the submissions and reviews from ICLR (2017-2026) are used. 
The framework starts with a predefined condition to test. Several hypotheses are automatically generated by repeatedly sampling articles from the two groups and prompting an LLM to return the hypotheses that discriminate the two groups best. A smaller LLM is then used to decide for each article whether the hypothesis matches the abstract of the article. A validation partition is used to select the hypotheses that discriminate best between the 2 groups. This is done by fitting a regression to the measured condition. The effect of the remaining set of hypotheses is measured on a separate test partition.

Critical strengths
  • The proposed pipeline is fully automatic and general enough to be applied to any testing condition. The decisions behind the experimental setup are sound without any obvious issues.
Critical weaknesses
  • The main issue with the paper is the lack of any kind of human validation of the steps involved in the pipeline. While the initial hypotheses generation step is difficult to evaluate, the pair classification task could be easily replicated. This is a critical step in the pipeline, and it is performed by a smaller model.
  • The paper seems unnecessarily convoluted and difficult to follow. Once you get past this, the pipeline itself is not that complex.
Questions for authors
  • The initial hypotheses generation step is not detailed enough. Are all selected abstracts added in the prompt? How many are selected? How does the number of documents influence the quality of generated hypotheses? Are the sampled articles completely random? Would clustering help by comparing more similar articles?
  • Will the sets of hypotheses and labels be released publicly? They could represent an important result of this paper.
4 comments

We thank the reviewer for recognizing the generality of our framework and the soundness of setup, and the suggestions for the further improvement.

Human validation

Building on our pre-submission spot checks, we now added an independent human validation study covering all 24 reported predicates (Section 4.4 and Appendix K according to the revised version). Ten domain experts with graduate-level training assessed 1,920 text–predicate pairs, with two independent judgments per pair, yielding 3,840 initial judgments. Annotators saw only the predicate and its associated text and were blinded to the model labels, metadata (e.g. year) and expected effect directions. Initial agreement was 95.4%; 89 pairs required adjudication, and three remained uncertain. Expert backgrounds, assignments, instructions, and adjudication details are provided in Appendix K.

The estimates underlying Figure 9 are reproduced below. S/R effects are in percentage points; C effects are in (10−310^{-3}) nats per review.

PredicateLLM estimateLLM 95% CIHuman estimateHuman 95% CI
S1-5.81[-8.38, -3.25]-2.77[-6.12, -0.40]
S2-9.93[-13.24, -6.61]-8.22[-14.50, -2.78]
S3-7.44[-10.43, -4.45]-8.33[-11.78, -5.50]
S4-9.06[-12.38, -5.73]-6.31[-11.34, -1.47]
S5-8.10[-10.54, -5.65]-4.47[-10.10, 0.00]
S6-6.42[-8.18, -4.66]-2.43[-5.79, 0.94]
S74.44[2.17, 6.71]-0.56[-10.76, 9.53]
S85.82[3.55, 8.10]1.89[-3.59, 5.95]
R1-17.00[-18.85, -15.15]-8.64[-14.43, -2.99]
R2-12.21[-13.80, -10.63]-6.05[-18.13, 4.64]
R3-10.85[-12.59, -9.12]-1.81[-4.19, 0.64]
R4-6.60[-7.96, -5.24]-2.20[-4.73, 0.82]
R5-4.29[-5.83, -2.75]-8.56 (-8.83)*[-14.06, -2.35]
R61.47[1.04, 1.91]5.90 (6.31)*[-1.49, 14.40]
R71.53[1.03, 2.04]1.08[-0.20, 2.05]
R82.27[1.30, 3.24]8.96[-2.84, 21.48]
C13.39[2.58, 4.21]7.82[5.66, 9.00]
C22.35[1.80, 2.91]0.52[-2.28, 3.17]
C32.19[1.60, 2.78]0.69[-2.17, 4.08]
C44.17[3.34, 5.01]8.62[3.72, 13.79]
C52.73[1.97, 3.48]2.85[2.16, 3.53]
C63.80[3.00, 4.60]1.98[0.98, 2.73]
C71.36[0.80, 1.93]1.73[-1.12, 3.51]
C83.08[2.55, 3.62]1.56[-1.49, 5.27]

* R5 and R6 retain one and two unresolved labels, respectively; the two estimates reflect alternative assignments of these labels, represented by the filled and open red points in Figure 9.

Human measurements largely preserve the direction of the reported effects. Human- and LLM-based estimates have the same sign for 23 of the 24 hypotheses, with 18 of the automated point estimates falling within human 95% confidence intervals.

For temporal analysis in Application 1, human judgments broadly corroborate shifts in contribution framing and novelty assessment. Human estimates preserve the combination of reduced theory-related framing (S5–S6) and fewer novelty-related criticisms (R1–R5), with intervals below zero for R1 and R5, suggesting that changes in contribution narratives accompany changes in the emphasis of reviewer scrutiny. Positive estimates for research-enabling datasets (S8) and contributions spanning data, methods, and evaluation (R8) are also consistent with a broader contribution narrative, although the integrated-pipeline estimate (S7) is close to zero.

Similarly, in the predictive study (Application 2), contribution clarity and evidential support remain informative beyond ratings and year. Judgments concerning contribution clarity (C4) and evidential support (C6) retain positive predictive gains beyond ratings and year, with human-based intervals above zero. The same holds for unjustified design choices (C1) and unclear novelty relative to prior work (C5). This reinforces the value of moving from (M_0) to (M_2).

We accordingly soften the increased emphasis on research-enabling contributions as tentative (revised manuscript, lines 23, 71, and 330).

Writing concern

We appreciate the advice. In this revised paper, we have improved the presentation by clarifying the workflow: explaining the steps before introducing notation and using concrete examples to explain the statistical targets. We have also reduced technical detail in the main text by moving the temporal-estimand derivation and the conditional mutual information discussion to the appendix.

Question about the corpus

Thanks for recognizing that our testbed is important. Yes. To preserve anonymity, we will release the full 2017–2026 corpus after the review process, which links 42,123 accepted or rejected submissions to 161,469 reviews, together with research-area annotations (model-imputed labels for earlier years), LLM-generated marker, their accompanied hypothesis, document-level labels, etc., which in detail shown in the field case in the updated Appendix A Table 2.
The release will also include reconstruction scripts, fixed paper-level proposal/validation/test splits, and code reproducing the reported analyses. We want to reclaim these linked resources provide a reusable testbed for comparing hypothesis-generation and verification methods under shared data and evaluation targets. They also support longitudinal studies of research topics and peer-review judgments, making them valuable for the ICLR and whole ML community.

Question about details

Prompt selection and sampling. In total, 12,549 abstracts and 48,376 reviews (30% of the corpus) are selected into the proposal split. GPT-5.6-Terra generates candidates from subsampled batches detailed in Appendices A.1 to A.3. Sampling is structured; the proposal pool is stratified by year and decision (Table 1), and each prompt conditions on the target contrast while alternating presentation orders to prevent position bias, drawing randomly across subfields within each group. We will surface these summary numbers directly in Section 4.1.

Effect of document count and hypothesis quality. Per-prompt document count balances small-sample noise against long-context attention dilution, whereas accumulating batches across the proposal pool governs search recall (Figures 7 and 8; Proposition 2). Crucially, because every proposed predicate is subsequently screened across thousands of validation documents (7,906 to 14,678 texts) before held-out testing, finite prompt sizes affect only recall rather than statistical validity. Our expert evaluation (Section 4.4 and Appendix K) further attests to the quality of hypotheses surfaced under this setup: ten domain experts achieved 95.4% initial agreement (1,831 of 1,920 pairs) when annotating the 24 discovered predicates and reproduced the LLM effect directions on 23 of the 24 claims.

Role of clustering. Grouping similar submissions prior to prompting is a promising way to suppress topic-shift confounders and isolate fine-grained, subfield-specific distinctions. In this study, we sampled broadly across subfields because our estimands target conference-wide norms, and we evaluated sensitivity to topic composition downstream using research-area fixed effects (Section 4.2). Consistent with this broad scope, experts spanning diverse ML specialties reliably validated the resulting claims (Table 7). When researchers instead target within-area estimands, cluster-conditioned prompting offers a natural extension for the future work of our paper.

We believe this response addresses your concerns, and we would be glad to provide further clarification if needed.

Thank you for the additional human validation of the predicates, this really helps. The updates made in the text also help by making the story more clear. Because of this, I have raised my overall score to 3.

Dear Reviewer NdE6,

Thank you very much for taking the time to review our revisions and for your positive feedback! This really helps strengthen our paper.

Best regards, The Authors

Reviewer_PUawOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper develops a framework in which LLM-generated hypotheses are explicitly linked to statistical estimands and independently tested. It applies this framework to changes in research contributions and peer-review judgments across ten years of ICLR submissions.

Critical strengths

The alignment between research question, measurement, estimand, and inference is structurally sound. Independent discovery and confirmation samples, simultaneous uncertainty bounds, and explicit treatment of alternative explanations do constitute a coherent statistical methodology. I appreciated that.

Critical weaknesses

The framework establishes statistical properties of LLM-generated labels, not necessarily their semantic validity: however, independent human annotation is missing. The 2024 structural-break specification rests on restrictive temporal assumptions, and the incremental predictive gains for acceptance are small.

Questions for authors

How robust are the substantive discoveries to human-validated semantic measurements and more flexible temporal models? What does estimand alignment achieve beyond existing hypothesis-generation methods combined with conventional held-out statistical testing?

3 comments

We appreciate your thoughtful engagement with our analysis for the ICLR community and your recognition of the structural soundness and coherence of our statistical methodology, and we address your specific concerns below.

Semantic validity and human validation

To directly evaluate the semantic fidelity of our automated measurements and the robustness of our substantive conclusions, we added a blinded expert validation covering all 24 reported hypotheses (Section 4.4, Figure 9, and Appendix K). Ten domain experts with graduate-level training independently annotated 1,920 text-predicate pairs (3,840 initial judgments, or 80 pairs per hypothesis), blinded to LLM labels, publication year, review scores, decisions, and expected effect directions. Initial inter-annotator agreement reached 95.4% (1,831 of 1,920 pairs); 89 disagreements were referred for adjudication, with only three pairs remaining uncertain.

To verify that temporal shifts are not artifacts of changing review style or labeler drift across the 2024 cutoff, the audit balanced 960 pre-2024 and 960 post-2024 pairs. Against adjudicated human labels, Qwen3-8B maintains high accuracy and stability across both periods:

MetricBefore 2024 (n=960n = 960)2024 onward (n=960n = 960)
False-positive rate5.45%6.94%
False-negative rate2.58%2.41%
Precision94.38%92.71%
F1 score95.87%95.09%

Re-estimating all 24 target effects directly on the human annotations confirms strong directional concordance (full numerical comparison below; S and R in percentage points, C in 10−310^{-3} nats per review):

PredicateLLM estimateLLM 95% CIHuman estimateHuman 95% CI
S1-5.81[-8.38,-3.25]-2.77[-6.12,-0.40]
S2-9.93[-13.24,-6.61]-8.22[-14.50,-2.78]
S3-7.44[-10.43,-4.45]-8.33[-11.78,-5.50]
S4-9.06[-12.38,-5.73]-6.31[-11.34,-1.47]
S5-8.10[-10.54,-5.65]-4.47[-10.10,0.00]
S6-6.42[-8.18,-4.66]-2.43[-5.79,0.94]
S74.44[2.17,6.71]-0.56[-10.76,9.53]
S85.82[3.55,8.10]1.89[-3.59,5.95]
R1-17.00[-18.85,-15.15]-8.64[-14.43,-2.99]
R2-12.21[-13.80,-10.63]-6.05[-18.13,4.64]
R3-10.85[-12.59,-9.12]-1.81[-4.19,0.64]
R4-6.60[-7.96,-5.24]-2.20[-4.73,0.82]
R5-4.29[-5.83,-2.75]-8.56(-8.83)*[-14.06,-2.35]
R61.47[1.04,1.91]5.90(6.31)*[-1.49,14.40]
R71.53[1.03,2.04]1.08[-0.20,2.05]
R82.27[1.30,3.24]8.96[-2.84,21.48]
C13.39[2.58,4.21]7.82[5.66,9.00]
C22.35[1.80,2.91]0.52[-2.28,3.17]
C32.19[1.60,2.78]0.69[-2.17,4.08]
C44.17[3.34,5.01]8.62[3.72,13.79]
C52.73[1.97,3.48]2.85[2.16,3.53]
C63.80[3.00,4.60]1.98[0.98,2.73]
C71.36[0.80,1.93]1.73[-1.12,3.51]
C83.08[2.55,3.62]1.56[-1.49,5.27]

* R5 and R6 retain one and two unresolved labels, respectively; the two estimates reflect alternative assignments of these labels, represented by the filled and open red points in Figure 9.

Human and LLM estimates share the same directional sign on 23 of the 24 hypotheses, with 18 LLM point estimates falling within the human 95% intervals. Although intervals on n=80n = 80 audited pairs per predicate are naturally wider than on the full test archive, ten hypotheses still exclude zero at the 95% level on human data alone (S1 to S4, R1, R5, and C1, C4, C5, C6). Human coding corroborates the retreat from method- and theory-centric framing alongside fewer novelty objections, as well as positive gains for research-enabling datasets (S8) and multi-faceted contributions (R8). Because the human point estimate for integrated pipelines (S7) is near zero, we explicitly tempered our wording regarding research-enabling contributions in the revised text (lines 23, 71, and 330).

Question about the robustness to more flexible temporal models (Appendix G.1)

In our discovery workflow, pre-2024 placebo break checks at 2020 and 2022 already filter out 635 of 1,026 review candidates exhibiting prior nonlinear drift (lines 279 to 284), while Figure 3 plots unrestricted annual means alongside the fitted breaks. In the revised manuscript, we additionally re-evaluated all 16 reported Application 1 predicates on the held-out test labels under three alternative temporal specifications, retaining equal document weights and pointwise 95% paper-clustered confidence intervals (Appendix G.1, Figure 13 and Table 4):

Temporal specificationDirection retainedPointwise 95% CI excludes zero
Original common slope (Yh∼1+Ut+PtY_h \sim 1 + U_t + P_t, 2017 to 2026)16/1616/16
Varying pre/post slopes: immediate 2024 departure16/1616/16
Varying pre/post slopes: 2024 to 2026 mean departure16/1615/16
Local window (Yh∼1+Ut+PtY_h \sim 1 + U_t + P_t, 2020 to 2026)15/1612/16
Quadratic background trend (Yh∼1+Ut+Ut2+PtY_h \sim 1 + U_t + U_t^2 + P_t)14/1612/16

Across every model specification, the declines in method- and theory-centric framing (S1 to S6), the increase in research-enabling datasets (S8), and reduced novelty objections (R1 to R4, plus R5 across all immediate-break specifications) remain robust, with no predicate exhibiting a statistically significant reversal. While the four smaller positive effects (S7 and R6 to R8) exhibit larger 2024 to 2026 mean departures under varying slopes, their 2024 break intervals cross zero under the local-window and quadratic models, indicating greater sensitivity to how the background trend is parameterized (consistent with our tempered framing of S7 in the revised text).

Question on interpreting the magnitude of incremental predictive gains

The small numerical scale of the log-loss reductions reflects the stringency of the conditional target rather than weak substantive relevance, for three reasons. First, reviewers assign numerical scores specifically to summarize their written evaluations, and acceptance tracks those scores steeply (Figure 5); conditioning on rating bin and year (M0M_0) intentionally absorbs nearly all first-order signal, isolating the residual conditional mutual information I(A;Yh∣Z)I(A; Y_h \mid Z) carried by a single binary indicator beyond scores (Appendix H.2). Second, corpus-wide averages pool across clear rejects and strong accepts where decisions are nearly deterministic and residual entropy is minimal. Allowing rating-specific effects (M0→M2M_0 \to M_2) quadruples the set-level certified margin from 0.430.43 to 1.80×10−31.80 \times 10^{-3} nats per review (Table 6) precisely because textual arguments exert concentrated leverage in borderline rating bins where Area Chairs adjudicate competing reviews. Third, automated measurement noise attenuates predictive gains: on clean human labels, point gains for contribution clarity (C4) and design justification (C1) more than double to 8.628.62 and 7.82×10−37.82 \times 10^{-3} nats per review.

Question: What estimand alignment achieves beyond existing generation plus held-out testing?

Decoupling open-ended generation from the target research design fails in one of two ways:

  • Validating the generator’s native objective confirms the wrong claim: Pairing contrastive or predictive generation (e.g., D5 or NLParam) with held-out multiplicity control validly confirms marginal distributional differences or raw correlations, which need not answer the scientific question. A property growing purely linearly over time will reliably pass held-out pre- versus post-2024 testing while having zero 2024 structural break (θh=0\theta_h = 0); likewise, a harsh critique will pass held-out acceptance testing merely by correlating with low ratings while adding zero predictive value beyond scores.

  • Applying design-specific tests only after unaligned selection starves confirmation: If conventional design testing is applied only post hoc to candidates ranked by marginal contrasts, upstream selection wastes the finite confirmation budget KK on confounded signals. Our own empirical funnel illustrates this directly: in Application 1, 62% (635 of 1,026) of review candidates with strong break signals failed pre-2024 placebo diagnostics due to prior drift; in Application 2, novelty criticism (C5) fails confirmation under a common text effect (Lh=−0.32L_h = -0.32) and is recovered only when evaluated under the rating-varying estimand M2M_2 (Lh=+1.53L_h = +1.53). Estimand alignment brings the research design and its diagnostics upstream into candidate scoring and selection before freezing claims, ensuring the confirmed set answers the intended population question.

Thank you again for your constructive feedback. We hope this clarifies the matter and welcome any further suggestions you may have.

Dear Reviewer PUaw,

We sincerely appreciate your time and constructive feedback. With the discussion phase coming to a close soon, we want to gently check whether our updates have resolved your questions. If you have any further questions, we would be glad to answer them before the deadline.

Best regards, The Authors

Reviewer_L86AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a workflow that specifies a target estimand, uses LLMs to propose text predicates, measures them with an NLI model, and separates discovery, validation, and confirmation. Applied to ICLR papers and reviews from 2017–2026, it identifies changes in contribution framing and examines whether review text improves acceptance prediction beyond numerical ratings.

Critical strengths

The statistical workflow is carefully specified. Paper-level splitting reduces leakage, while explicit effect sizes and multiplicity-adjusted confirmation make the conclusions more interpretable than an unrestricted search for interesting patterns.

Critical weaknesses

The substantive findings depend on machine-generated predicate labels without blinded human calibration. Agreement between language models does not establish that the predicates are measured correctly or consistently across years. The methodological contribution also appears to rely substantially on established statistical components, and the paper does not clearly demonstrate an advantage over comparable hypothesis-discovery workflows through budget-matched evaluation.

Questions for authors

How accurate and stable are the predicate labels across years, particularly for predicates driving the largest reported changes? What evidence distinguishes better discovery from more disciplined validation?

3 comments

We appreciate your recognition of our statistical specification for uncovering interesting interpretable patterns, and thank you for the thoughtful feedback for sharpening our paper.

Without blinded human calibration

Building on our pre-submission spot checks, we now added an independent human validation study covering all 24 reported predicates (Section 4.4 and Appendix K according to the revised version). Ten domain experts with graduate-level training assessed 1,920 text–predicate pairs, with two independent judgments per pair, yielding 3,840 initial judgments. Annotators saw only the predicate and its associated text and were blinded to the model labels, metadata (e.g. year) and expected effect directions. Initial agreement was 95.4%; 89 pairs required adjudication, and three remained uncertain. Expert backgrounds, assignments, instructions, and adjudication details are provided in Appendix K.

The estimates underlying Figure 9 are reproduced below. S/R effects are in percentage points; C effects are in (10−310^{-3}) nats per review.

PredicateLLM estimateLLM 95% CIHuman estimateHuman 95% CI
S1-5.81[-8.38, -3.25]-2.77[-6.12, -0.40]
S2-9.93[-13.24, -6.61]-8.22[-14.50, -2.78]
S3-7.44[-10.43, -4.45]-8.33[-11.78, -5.50]
S4-9.06[-12.38, -5.73]-6.31[-11.34, -1.47]
S5-8.10[-10.54, -5.65]-4.47[-10.10, 0.00]
S6-6.42[-8.18, -4.66]-2.43[-5.79, 0.94]
S74.44[2.17, 6.71]-0.56[-10.76, 9.53]
S85.82[3.55, 8.10]1.89[-3.59, 5.95]
R1-17.00[-18.85, -15.15]-8.64[-14.43, -2.99]
R2-12.21[-13.80, -10.63]-6.05[-18.13, 4.64]
R3-10.85[-12.59, -9.12]-1.81[-4.19, 0.64]
R4-6.60[-7.96, -5.24]-2.20[-4.73, 0.82]
R5-4.29[-5.83, -2.75]-8.56 (-8.83)*[-14.06, -2.35]
R61.47[1.04, 1.91]5.90 (6.31)*[-1.49, 14.40]
R71.53[1.03, 2.04]1.08[-0.20, 2.05]
R82.27[1.30, 3.24]8.96[-2.84, 21.48]
C13.39[2.58, 4.21]7.82[5.66, 9.00]
C22.35[1.80, 2.91]0.52[-2.28, 3.17]
C32.19[1.60, 2.78]0.69[-2.17, 4.08]
C44.17[3.34, 5.01]8.62[3.72, 13.79]
C52.73[1.97, 3.48]2.85[2.16, 3.53]
C63.80[3.00, 4.60]1.98[0.98, 2.73]
C71.36[0.80, 1.93]1.73[-1.12, 3.51]
C83.08[2.55, 3.62]1.56[-1.49, 5.27]

* R5 and R6 retain one and two unresolved labels, respectively; the two estimates reflect alternative assignments of these labels, represented by the filled and open red points in Figure 9.

Human measurements largely preserve the direction of the reported effects. Human- and LLM-based estimates have the same sign for 23 of the 24 hypotheses, with 18 of the automated point estimates falling within human 95% confidence intervals.

For temporal analysis in Application 1, human judgments broadly corroborate shifts in contribution framing and novelty assessment. Human estimates preserve the combination of reduced theory-related framing (S5–S6) and fewer novelty-related criticisms (R1–R5), with intervals below zero for R1 and R5, suggesting that changes in contribution narratives accompany changes in the emphasis of reviewer scrutiny. Positive estimates for research-enabling datasets (S8) and contributions spanning data, methods, and evaluation (R8) are also consistent with a broader contribution narrative, although the integrated-pipeline estimate (S7) is close to zero.

Similarly, in the predictive study (Application 2), contribution clarity and evidential support remain informative beyond ratings and year. Judgments concerning contribution clarity (C4) and evidential support (C6) retain positive predictive gains beyond ratings and year, with human-based intervals above zero. The same holds for unjustified design choices (C1) and unclear novelty relative to prior work (C5). This reinforces the value of moving from (M_0) to (M_2).

We accordingly soften the increased emphasis on research-enabling contributions as tentative (revised manuscript, lines 23, 71, and 330).

How accurate and stable are the predicate labels across years

The above audit sampled 960 pairs from before 2024 and 960 from 2024 onward, which also examines the temporal stability. Model performance against the human reference labels was:

MetricBefore 20242024 onward
False-positive rate5.45%6.94%
False-negative rate2.58%2.41%
Precision94.38%92.71%
F195.87%95.09%

These audit-sample results show high precision and F1 in both periods, with an almost unchanged false-negative rate and a modest increase in false positives after 2024. They provide direct human-based evidence of broadly similar aggregate labeling performance across the cutoff.

The methodological contribution …

We deliberately build upon mature statistical primitives (e.g. regression, sample splitting) since they provide established tools for implementation; non-standard estimators here would be a liability. Our methodological contribution lies instead in estimand-aligned natural-language discovery: bridging open-ended LLM predicate search with a formally declared research design. Rather than treating statistical inference as a post-hoc filter after text mining, our framework makes the target population quantity (estimand) and design diagnostics the governing objective throughout candidate scoring, selection, and claim formulation, which returns each hypothesis alongside its explicit measurement rule and simultaneous lower bounds on directional effect sizes.

Existing discovery pipelines optimize well-defined specific objectives, but succeeding at those objectives does not answer design-specific scientific questions, for example (the references are already included in the paper):

WorkflowNative Search ObjectiveStatistical Claim SupportedLimitation for Design-Specific Questions
D5 (Zhong et al., 2023)Corpus-level distributional contrastProperty X occurs more frequently in corpus A than BConflates structural shifts with secular background trends or confounders
HypotheSAEs (Movva et al., 2025)Predictive feature selection and interpretationInterpretable feature X predicts target Y on held-out dataPredictive utility does not isolate controlled or counterfactual parameters
OursEstimand-governed search and diagnosticsPopulation claims under an explicit research design, with simultaneous directional boundsRequires researchers to specify an explicit statistical target upfront

Let us consider this instance: a review property whose prevalence grows purely linearly over time. Both D5 (contrasting pre- vs. post-2024 corpora) and HypotheSAEs (predicting era membership) will rightfully flag this property as a strong, replicable signal. But under our Application 1 estimand, a post-2024 structural level shift net of a shared linear trend, its true effect is strictly zero. Perfectly discovering and validating distributional differences or predictive features still leaves our scientific question unanswered.

What distinguishes better discovery from more disciplined validation?

If an upstream discovery engine optimizes purely for marginal differences or raw predictability, bolting on stricter validation downstream merely confirms answers to the wrong statistical question. In other words, it will “starve” the pipeline when confounded candidates systematically fail the target design on held-out data. In our framework, the declared estimand governs discovery itself. Analytically, the linear-trend example demonstrates why disciplined post-hoc validation cannot substitute for estimand-aware search. Empirically, our two applications provide end-to-end evidence that steering open-ended search via explicit research designs reliably uncovers non-trivial, independently confirmed claims (note that we claim fidelity to design-specific questions instead of superior candidate-generation yield per token on generic tasks).

About budget-matched yield comparisons

Meaningful head-to-head benchmarking at matched compute requires a common statistical target and success criterion. Comparing works like D5 or HypotheSAEs against our workflow by counting "statistically significant discoveries" at equal cost would tally answers to incomparable statistical questions, which rewards baseline methods for surfacing confounded signals (such as purely linear trends) that represent nulls under our design. Conversely, retrofitting those baselines to score and select candidates under our regression-adjusted estimands and diagnostics would no longer test their original formulations; it would evaluate adapted instantiations of our own estimand-aligned principle. While we are glad to report our exact API and compute costs for transparency upon archived, resource parity alone cannot equate workflows designed around fundamentally different scientific claims.

We hope these clarifications resolve your concerns, and we are glad to address any follow-up questions you may have.

Dear Reviewer L86A,

Thank you again for your time and thoughtful review. As the discussion phase is coming to an end, we want to briefly follow up to see whether you have received our response and our clarifications have addressed your concerns. Please feel free to let us know if there is anything else we can clarify before the discussion phase concludes.

Best regards, The Authors

Comments on the paper

1 comment

Dear Reviewers and Area Chair,

Thank you all for your constructive feedback and for helping us improve the paper. We appreciate your recognition of our coherent statistical workflow and interpretable conclusions, as well as the general pipeline and the value of insights alongside resources.

To address your questions and verify our findings, we provided the clarifications and refinements including:

  • Human Verification (L86A, PUaw, NdE6): Added a blinded expert audit across all reported predicates, confirming strong agreement with our automated measurements and refining our wording where appropriate.
  • Stability & Methodological Clarifications (L86A, PUaw): Verified cross-year labeling stability and robustness to alternative temporal specifications, and clarified how upstream estimand alignment differs from post-hoc validation.
  • Clarity & Open Corpus (NdE6): Streamlined the presentation for readability and confirmed the public release of our linked 2017–2026 corpus (42k submissions and 161k reviews) and code.

Together, these verifications confirm our substantive findings on post-2024 shifts in contribution framing and novelty scrutiny, as well as the specific textual arguments (such as contribution clarity and evidential support) that inform acceptance beyond numerical scores. We have incorporated these refinements into the revised manuscript. We believe the responses adequately resolve the remaining concerns, and we hope both the framework and longitudinal testbed serve as useful resources for the ICLR and broader ML community.

Thank you again for your time and dedication throughout the review process.

Sincerely,
The Author Team

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.