acceptodds
Under review as a conference paper at ICLR 2027

The Scientific Research Taste Benchmark (SciTaste-Bench): A Multi-Disciplinary Benchmark for Assessing Large Language Models' Judgment of Research Taste in Scientific Papers

Abstract

Whether large language models can assess the innovativeness of academic publications is important in literature reviews, research funding screening, and academic recommendations. We introduce the Scientific Research Taste Benchmark (SciTaste), a gold-standard benchmark covering six disciplines: physics, mathematics, chemistry, biology, medicine, and finance. SciTaste measures research taste, that is, the ability to judge the scientific value and potential of a paper, of which innovativeness is only one dimension among the seven assessed. SciTaste integrates metadata from NASA ADS, OpenAlex, and arXiv, aligns papers to field-specific classification systems, and evaluates each paper using the seven-dimensional Academic Taste Index. The weights for each dimension are derived from a bi-objective machine-learning derivation (five regressors, including random forests) and are shared across all six disciplines. This benchmark includes 300 selected papers (50 per discipline) and a backup pool of 5,700 papers (950 papers per field) covering 1996–2023. To allow external knowledge to influence model decisions, SciTaste supports four evaluation modes: pure_qa, native_search, our self-built structured tool-calling agent DimAgent (9 bibliometric tools in ADS/OpenAlex), and dimagent_v3_native. Model outputs are evaluated with two complementary schemes: one combines correlation with dimension-level accuracy, and the other is based on the WAPE relative error. In the evaluation of ten state-of-the-art large language models, the tool-based structured evaluation generally outperformed the toolless and raw web search modes, but the improvement was model-dependent. Physics is the easiest subject, while chemistry, biology, and medicine are still more challenging. By measuring how well large language models judge the value potential of academic work across fields, SciTaste supplies the value-judgment substrate that auto-research systems need to screen, rank, and prioritize candidate ideas. We release the dataset, the evaluation harness, and the scoring code to support reproducible research on scientific-innovation assessment; all data and code will be made publicly available on Hugging Face.

Primary area:
datasets and benchmarks
Keywords:
large language models, research-paper innovativeness, scientific benchmark, scientometrics, multi-disciplinary evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_36GjOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SciTaste-Bench, a multi-disciplinary benchmark designed to assess large language models' judgment of research taste. Specifically, the benchmark integrates data from four platforms across six disciplines, and comprises 300 selected papers, plus a backup pool of 5,700 papers covering a wide date range. The authors propose Academic Taste Index covering seven dimensions, whose weights are derived from a bi-objective machine-learning derivation with five regressors and shared across all six disciplines. It also supports four evaluation modes and adopts dual scoring schemes combining correlation with dimension-level accuracy and WAPE relative error. Evaluations on ten state-of-the-art LLMs show that structured tool-based evaluation generally outperforms tool-less and raw web-search modes. Further analysis show that the improvement is model-dependent, and that physics is the easiest discipline while chemistry, biology, and medicine remain more challenging.

Critical strengths
  • Broad discipline coverage and extensive model evaluation: SciTaste-Bench benchmark integrates data from multiple platforms across six disciplines, with 300 selected papers and a large backup pool covering a wide date range, enabling cross-discipline comparison under a unified scoring framework. Additionally, experiments cover ten state-of-the-art LLMs under four evaluation modes with dual scoring schemes, providing a systematic comparison across both models and disciplines.
Critical weaknesses
  • Bias in design of the taste index and limited dataset size: The paper applies identical weight distribution among seven-dimensions across all six scientific disciplines, and the weights are derived from a corpus of a single discipline solely based on regression outcomes. Also, the distribution of the weights are identical across all six disciplines with no refined calibration for each field. Such design overlooks the inductive bias among different disciplines, and derivation of weights from a single discipline bears the risk of propagating the bias to all disciplines. Finally, the author mentioned that although there are a total of 300 samples in their dataset, there are only 50 papers within each discipline. The limited dataset scale plus the rigid design of taste index all bears the risk that yields unreliable evaluation results.

  • Limited human-in-the-loop supervision of dataset quality: The pipeline of data fusion, discipline classification, and the computation of the ground-truth scores are fully automated, and human involvement is minimal to manual verification of the metadata consistency of the selected papers. This poses the risk that errors in the automated pipeline propagate unchecked into the ground truth, and that the benchmark ends up measuring agreement with bibliometric formulas rather than with human judgment of research taste.

Questions for authors

Unless further justification of the taste index as well as the absence of human evaluation are provided, the evaluation results are not convincingly reliable.

Reviewer_SVbZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

SciTaste-Bench is a multidisciplinary benchmark for evaluating LLM assessments of scientific papers using a seven-dimensional Academic Taste Index. It includes 300 evaluation papers across six disciplines and a backup pool of 5,700 papers. Reference scores are constructed from bibliometric and metadata features, with shared weights derived from 998 quantum-physics papers. The authors evaluate ten models under four settings, comparing direct scoring, native web search, structured academic tools, and their combination. Structured tools generally improve agreement with the reference scores, although the gains depend on the model.

Critical strengths
  1. The paper provides an operational framework for comparing paper assessments across six disciplines, with both overall and dimension-level reference scores. This makes the evaluation more informative than using citation counts alone.

  2. The four evaluation settings provide a useful comparison of information access. Reporting model-specific failures and the near-zero correlations on novelty-related dimensions is consequential because it exposes limitations that an aggregate score could obscure.

Critical weaknesses
  1. Closely related work is omitted, and the novelty of the main contributions is limited.

    The related-work section and reference list omit SchNovel / RAG-Novelty, AI Can Learn Scientific Taste, OpenNovelty, and NovBench. These are directly relevant to the paper's claimed contributions. Examining those contributions separately shows substantial overlap with existing work. Dates below refer to each work's first arXiv submission (v1).

    Multidisciplinary benchmarking: SchNovel / RAG-Novelty (September 2024) already evaluates scholarly novelty using 15,000 paper pairs across six fields. SciTaste-Bench uses a different subject mix and scoring target, but multidisciplinary evaluation itself is an established setting.

    Citation-based evaluation of scientific taste: AI Can Learn Scientific Taste (March 2026) already constructs SciJudgeBench using field- and time-matched citation-based paper pairs. Using community citation signals to evaluate scientific judgment is therefore not a new contribution of this submission. Its distinctive scoring element is the particular seven-dimensional composite, whose validity requires separate evidence.

    Tool-assisted paper assessment: SchNovel includes retrieval-augmented novelty assessment, while OpenNovelty (January 2026) retrieves prior work and performs contribution-level full-text comparisons to produce evidence-grounded novelty reports. DimAgent-v3 adapts tool-assisted assessment to nine predefined bibliometric queries followed by an LLM scoring step. This provides a specific implementation, but the paper does not clearly demonstrate a new retrieval, reasoning, or learning mechanism beyond existing tool-assisted assessment workflows.

    Benchmarking novelty judgments against reference evaluations: NovBench (April 2026) already evaluates novelty-review generation using 1,684 paper–review pairs with reference evaluations extracted from expert reviews. Systematically testing LLM novelty judgments and their relationship to human assessments is also an established research direction.

    I regard the proposed approach as a limited integration of established methods. The paper does not substantiate a stronger benchmark contribution through validated reference labels or sufficiently distinctive empirical findings; the construction problems in the next weakness further undermine that contribution.

  2. The benchmark does not sufficiently validate either its evaluation population or its reference labels.

    The 300-paper evaluation set is too limited to substantiate the breadth of the conclusions without further validation. Each discipline contributes only 50 papers while covering seven or eight sub-disciplines, leaving roughly six or seven papers per sub-discipline on average if all are covered, before accounting for year and journal tier. The 5,700 backup papers are not included in the main model comparisons. The paper does not report uncertainty intervals or demonstrate that model rankings and discipline-level differences persist across independently sampled evaluation sets. The concern is the stability and coverage of the reported conclusions, rather than a universal minimum dataset size.

    The selection procedure also leaves important representativeness questions unresolved. Non-physics candidates are collected in descending citation order and sampled across composite-score quartiles (§3.3), while the selected set emphasizes 2000–2020 papers and balances journal tiers (§4). These choices define a particular evaluation population, whose relationship to the broader literature is not quantified. The paper does not establish how citation-ranked collection and subsequent selection affect coverage of less-cited or recent work. Physics and the other fields also use different database backends, so observed discipline gaps may reflect sample composition and evidence availability as well as model capability. Stratified sampling can improve coverage, but the reported strata alone do not establish that the comparisons are representative or robust.

    The quantum-physics-derived weights define the answers against which all models are judged. Section 3.4 converts feature contributions from predicting citation and journal impact on 998 quantum-physics papers into dimension weights, then applies them to all six disciplines. This requires two distinct justifications: why predictive importance for these bibliometric outcomes should determine importance in scientific-value assessment, and why the resulting relative importance should hold in medicine, chemistry, finance, and the other fields. Predictive performance within quantum physics establishes neither. A common weight vector supplies a common calculation; it does not establish that the resulting scores have the same scientific meaning across disciplines.

    Section 8 acknowledges that transferability is an assumption, but does not quantify its effect on the benchmark. Although multidisciplinary data have been collected, there is no comparison with balanced multidisciplinary calibration, field-specific weights, or equal weights, and no analysis of how these alternatives change reference paper rankings, model rankings, or agreement with independent experts. Appendix C mentions expert inspection of final paper scores in the journal-score construction, but does not report the number of experts, the inspection procedure, or quantitative agreement. The reported inspection is therefore insufficient to establish the validity of the full seven-dimensional target; independent expert annotations remain future work. For a benchmark contribution, these checks are needed to establish the reference labels themselves. Their absence leaves open whether model rankings reflect scientific judgment or agreement with a scoring rule calibrated on quantum physics.

    The temporal definition further weakens the interpretation of the labels. The dataset spans 1996–2023, but Appendix C's citation formula uses cumulative counts without specifying year-conditioned reference cohorts or a common observation window. The observation horizon for disruptiveness and the historical cutoff for semantic comparisons are also unclear. These choices determine whether the benchmark evaluates realized influence or potential at publication. Publication-year and field effects have already been investigated in related research evaluation work, including Thelwall and Kurt (November 2024), but this study is not discussed.

    Finally, the tools retrieve quantities used to construct the reference scores, including citation and author counts. Better agreement may therefore reflect improved access to the scoring rule's inputs. Without simple bibliometric baselines and independent validation, the tool gains do not establish an improvement in scientific judgment.

    Internal inconsistencies also affect reproducibility and interpretation. Section 3.4 derives weights by averaging feature contributions across five regressors, whereas Appendix B and Table 5 describe a derivation from two random-forest analyses. Section 7.2 also calls DeepSeek's pure_qa Scheme-2 score of 30.2 its best result, although Table 2 reports higher scores for dimagent_v3 (34.5) and dimagent_v3_native (33.7). The authors should reconcile these descriptions with the actual implementation and results.

    Appendix F also states that reasoning_effort and temperature were disabled for GPT's tool calls. The effect of this configuration change needs to be examined before attributing its poor tool-mode performance to an intrinsic model difference.

    These issues indicate shortcomings in the manuscript's consistency checks. The weight derivation and experimental configuration affect reproducibility and interpretation, so they cannot be treated merely as language-editing problems. However, the PDF alone does not establish the full extent of AI involvement or the thoroughness of human verification.

    The PDF provides no access link for the dataset or code. The abstract (p. 1) says that the data and code will be made available on Hugging Face, while Appendix I (p. 15) describes the materials as published. However, the PDF provides no dataset link, code repository, or anonymous download address. For a paper whose main contribution is a benchmark, this prevents reviewers from checking the complete paper collection and reference scores or reproducing the weight derivation and model evaluations. If these materials are supplied separately through the submission system, their location should be specified.

Questions for authors
  1. Relative to SchNovel, SciJudgeBench, OpenNovelty, and NovBench, what specific capability or insight does the proposed benchmark add? Please distinguish the new evaluation target from the established use of retrieval and citation signals.

  2. Why derive the shared weights only from 998 quantum-physics papers when collecting six-discipline data? Is their transfer supported by independent expert assessments or comparisons with balanced multidisciplinary calibration, field-specific weights, and equal weights? Were the final weights obtained by averaging five regressors or from the random-forest analyses reported in Table 5?

  3. How are papers from different fields and publication years compared? For example, does being in the top 10% by citations mean among all papers, papers in the same field, or papers in the same field and publication year? Please state when citations were counted, how many years after publication were used to assess disruptiveness, and whether novelty comparisons use only literature available when the target paper was published. Would these choices materially change paper or model rankings?

  4. Which scoring dimensions improve when tools are added? Are the gains mainly in directly checkable information such as citations and journal indicators, or do novelty and disruptiveness assessments also improve? How does the models' ranking of papers compare with simply sorting them by citation count, in terms of agreement with the reference ranking?

  5. What population is the 300-paper set intended to represent, and how does it differ from the candidate and backup pools in publication year, citations, journal tier, and sub-discipline? What evidence shows that model rankings and discipline gaps persist across independently selected samples and after accounting for database-backend differences?

Reviewer_ytJ7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Constructs a multidisciplinary benchmark intended to measure the scientific judgment of LLMs using seven bibliometric and structural dimensions. Evaluates models with and without access to structured scientific metadata and retrieval tools.

Critical strengths

The cross-disciplinary dataset construction and structured bibliometric retrieval infrastructure could be useful independently. The comparison of tool-assisted and tool-free evaluations addresses an interesting practical question.

Critical weaknesses

There is a fundamental construct-validity problem: citation counts, journal prestige, team size, and metadata-derived novelty are (quite certainly) not equivalent to scientific taste or research quality. The supposedly gold-standard labels are constructed from these proxies, with weights learned from citation and journal outcomes in quantum physics; models are therefore evaluated on reproducing the authors' index rather than independently established scientific judgment.

Questions for authors

What independent evidence establishes that this index measures scientific taste rather than bibliometric conformity? Can the benchmark predict blinded expert assessments of research novelty, significance, or long-term contribution, especially after controlling for field, publication year, citations, and journal prestige?

Comments on the paper

1 comment

Thank you for the detailed response and additional analyses. The weight-sensitivity experiments and bootstrap confidence intervals are useful additions and address some of my concerns regarding ranking stability.

However, my main concerns remain unresolved. Stable model rankings under different weights do not establish the validity or cross-disciplinary transferability of the reference scores. More importantly, the lack of independent expert validation makes it difficult to determine whether the benchmark measures scientific research taste rather than agreement with a bibliometric scoring rule.

The citation-only baseline and dimension-level analysis further highlight this distinction. While I appreciate the authors' willingness to acknowledge these limitations and narrow their claims, I do not believe the current evidence sufficiently supports the benchmark's central claims. I therefore maintain my original assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.