acceptodds
Under review as a conference paper at ICLR 2027

ResearchMirror: Can AI scientists Match the Research Taste of Human Researchers?

Abstract

Research taste, the judgment of what to read, which problems to pursue, and which methods to try, increasingly shapes the work of AI scientists, yet it remains far less understood than their ability to execute research. We introduce ResearchMirror, a temporal benchmark for research taste that freezes the literature through 2025, generates research proposals from that snapshot, and compares them with work published by the same communities in 2026. Our experiments span six contemporary language models and four research areas from established to emerging fields, covering more than 90,000 proposals in total. AI scientists fall well short of human communities in both the literature they cite and the ideas they state, reaching about 26% of the human level on citation matching and only 18% on idea matching. Their populations can cite as diversely as a community and take up its tasks and methods, yet rarely arrive at the concrete approaches and research plans it pursues. Neither richer field context nor larger populations substantially close this gap. In blinded expert review, AI proposals read nearly as well as the 2026 papers yet are judged far less likely to succeed, and those whose ideas no human paper shares fare worst, suggesting weaker directions rather than overlooked opportunities. These results indicate that the main bottleneck in AI research taste lies less in finding the directions a community works on than in turning them into the concrete approaches it goes on to pursue. We will maintain ResearchMirror as an open, continuously updated leaderboard that benchmarks contemporary AI proposals against future publication cycles.

Primary area:
foundation or frontier models, including LLMs
Keywords:
ai scientists, agent, idea generation, benchmark, multi agent

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_B13xOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces ResearchMirror, a temporal benchmark that compares research proposals generated from literature available through 2025 with papers published by human communities in 2026. Across six models, four agent configurations, four research areas, and over 90,000 proposals, the results show that AI scientists remain substantially below the human reference, particularly in concrete approaches and research plans rather than problems and motivations.

Critical strengths

The benchmark is novel, timely, and carefully executed at an impressive scale. In particular, the temporal controls, population-scaling experiments, slot-level analysis, and blinded expert evaluation provide strong and complementary evidence. The benchmark not only identifies a human–AI gap but also offers an interpretable diagnosis of where that gap arises.

Critical weaknesses

ResearchMirror operationalizes research taste as agreement with the realized choices of a future research community, which is informative but not equivalent to scientific quality or normative research taste. Moreover, human papers may access 2026 literature and report completed results, while AI agents are restricted to the pre-2026 corpus and a 20-step proposal process. The broader interpretation of the human–AI gap should therefore be appropriately qualified.

Questions for authors

Could the authors provide a deeper analysis of the agents’ research trajectories, including their search queries, retrieved papers, intermediate hypotheses, and proposal revisions? In particular, for proposals with high MATCH-IDEA scores, it would be valuable to understand whether the final match results from a coherent and human-like refinement process or merely from convergence in the final text. A comparison between high- and low-matching trajectories could offer additional insight into how AI research ideas are formed.

Reviewer_Xz4sOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ResearchMirror, a temporal benchmark that evaluates the research taste of AI scientists against the papers human communities publish after a frozen literature snapshot. The paper indicates that the main bottleneck in AI research taste lies less in finding the directions a community works on than in turning them into the concrete approaches it goes on to pursue.

Critical strengths
  1. The paper is easy to read and established a useful and meaningful benchmark for evaluating AI research agents.
  2. The paper has some good empirical evaluation and informative analyses. The additional analyses on population size, temporal leakage, research-direction coverage, and individual idea components provide a comprehensive study of the gaps.
  3. The paper handles knowledge leakage very well.
Critical weaknesses
  1. The connection between research taste and research citation. The paper defines research taste as the judgment of what literature to read, which problems to pursue, and which methods to try, but evaluates it primarily as agreement with the research choices subsequently made by human communities. But this does not necessarily imply better scientific judgment. A valuable research idea might not be pursued by the human community during the evaluation window.
  2. The bias of human reviewers. Table 3 shows that reviewers identify 86.46% of AI proposals as AI-generated, so the blinding may not be effective. It is unclear whether reviewers' perceptions of document origin influence their assessments. This potential bias could affect the conclusions drawn from the human evaluation.
  3. Lack of inter-reviewer agreement analysis. The paper does not report inter-reviewer agreement for the human evaluation of research idea quality. Since each document receives only two independent reviews, it is unclear how consistently reviewers assess these defined dimensions. Could the authors report inter-reviewer agreement for the human evaluation results?
Questions for authors

See weakness.

Reviewer_gPu5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper propose ResearchMirror benchmark that evaluate how research choices from given context of AI scientists matches with human research community. Authors collected published paper until 2025 within selected research area (e.g. machine learning system, automated research, …). Then, generate research proposals from AI Scientists and compare with published work from human in 2026. Using LLM-as-judge, the authors evaluate matchless and diversity of (1) citations and (2) idea. Throughout 4 research area, 6 language models, and 4 generation setting, AI’s citation and idea coherentness is around 26% and 18%. Also, the authors provides ablations regarding expanding group size, human evaluation on 192 documents regarding validation in LLM-as-judge. In overall, the paper propose AI scientists are capable of direction of human community, but struggling on detailed directions and planning.

Critical strengths
  1. The paper provides a population-level analysis of AI scientists’ capabilities. Experiments compare AI-generated research proposals with human research across multiple areas. By using published research from human community as an external reference, the benchmark complements evaluations based on individual expert or LLM ratings.

  2. The experiments clarify how context shapes AI-generated research proposals. The authors compare access to research topics, a field-level research landscape, and peer proposals with the controlled agent implementation and budget control. These comparisons help distinguish changes in proposal diversity from improvements in alignment with human research.

Critical weaknesses
  1. Interpretation of Match-ideas as measure of research-test is inefficiently validated. Matching idea measure community alignment, yet Secion 3 interprets low matching as lack of taste (L350-354). The experiment of Llama scout models shows high matching score (L313-314), but low human ratings challenges use of match-ideas as proxy of research taste. Independent quality validation to connect matching to research taste or narrower claims are needed.

  2. Population size and missing informations may distort the proposed metrics. The study compares 300 AI proposals with 1,136-5,188 human papers (Table 1; L256-257). Figure 5 shows that matching gaps persist at equal sizes in selected settings, but leaves the diversity comparison unresolved. Missing fields also count as disagreement, allowing absent information to increase measured diversity (Figure 8, L1353–1355; Eq. 3). Equal-size comparisons and sensitivity analysis for such missing field are needed in order to distinguish research differences with measurement effects.

  3. The evaluation does not establish whether the findings hold for more capable research agents (e.g. Claude Opus,Fable, GPT Sol) . Although the study compares six models, it does not systematically examine model scaling or reasoning effort, and all generators use a 20-step budget (L256–257). The claimed bottleneck in approach development (L473–476) may therefore reflect the tested configurations by authors rather than a general limitation of AI scientists. Evaluation with stronger generators and larger per-proposal budgets would help distinguish these explanations.

Questions for authors
  1. What evidence supports interpreting Match–Idea as research-taste quality, given that Llama achieves the highest average matching score but the lowest human overall rating?
  2. Do the diversity conclusions persist when human and AI populations are matched in size and missing fields are treated separately from semantic disagreement?
  3. Does the bottleneck in approach development persist with stronger generators or larger per-proposal reasoning and revision budgets?
  4. Does the use of DeepSeek-V4-Pro for extraction and GPT-5.6 Luna for judging affect the results, and do the main conclusions remain stable when either component is replaced with a model from another family?
Reviewer_PfuvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces ResearchMirror, a temporal benchmark for evaluating the research taste of AI scientists through comparisons with future human research publications, supported by extensive experiments across four research areas and six language models. The proposed evaluation framework is interesting and the experimental analysis is comprehensive, particularly in examining the gap between identifying research problems and formulating concrete approaches. However, the benchmark primarily measures alignment with human research choices rather than scientific value or genuine research taste, while the asymmetry between AI-generated proposals and completed human papers raises concerns about the fairness and validity of the evaluation. Overall, this work provides a valuable contribution to evaluating AI scientists, but its central claims would benefit from a clearer conceptual foundation and more controlled experimental validation.

Critical strengths

-Novel and timely research problem: this paper addresses an important yet underexplored aspect of AI scientists: their ability to make research decisions, rather than merely execute predefined research tasks. The proposed temporal evaluation perspective provides a meaningful way to assess AI research taste against actual future research activities of human communities.

  • Comprehensive benchmark with rigorous empirical analysis: ResearchMirror covers four research areas, six language models, and four agent configurations, with over 90,000 generated research proposals. Beyond the comprehensive evaluation of literature selection and idea formulation, the authors conduct extensive analyses, including research direction coverage, slot-level decomposition, population scaling, and blinded expert evaluation, providing valuable insights into the gap between AI-generated and human research.
Critical weaknesses
  • Limited construct validity: ResearchMirror primarily measures alignment with future human research choices, which may not fully capture scientific value or genuine research taste. Non-matching AI proposals may represent novel opportunities rather than inherently weaker ideas, a distinction not fully resolved by the limited expert evaluation.
  • Limitations of evaluation method: MATCH-IDEA relies on abstract-based idea extraction, top-5 candidate retrieval, and LLM-based judgments, while its human validation is limited to a small sample from a single research area. Consequently, the reported matching scores may be affected by extraction and retrieval errors, potentially weakening the reliability of the main findings.
Questions for authors
  • How would ResearchMirror evaluate an AI-generated proposal that does not match any subsequent human publication but is later experimentally validated as scientifically valuable? Can the benchmark distinguish genuine research novelty from poor research judgment?
  • Since MATCH-IDEA relies on abstract-based extraction and top-5 candidate retrieval, have the authors investigated how retrieval recall and candidate pool size affect the reliability of the final matching scores?
  • Could the authors clarify the rationale for placing Related Work at the end of the paper, rather than integrating relevant comparisons into the Introduction or condensing less central content, as a more direct discussion of prior work could better establish the research gap and improve the overall narrative flow?

Comments on the paper

2 comments

Thank you for your detailed response. I believe the additional evidence you provided sufficiently addresses my concerns, and I would like to raise my score accordingly.

Dear Reviewers, thank you for your thoughtful reviews. As the authors have now responded, please review their responses and the other reviews, and leave follow-up comments on whether the concerns have been addressed. Thank you for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.