- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces ResearchMirror, a temporal benchmark that compares research proposals generated from literature available through 2025 with papers published by human communities in 2026. Across six models, four agent configurations, four research areas, and over 90,000 proposals, the results show that AI scientists remain substantially below the human reference, particularly in concrete approaches and research plans rather than problems and motivations.
Critical strengths
The benchmark is novel, timely, and carefully executed at an impressive scale. In particular, the temporal controls, population-scaling experiments, slot-level analysis, and blinded expert evaluation provide strong and complementary evidence. The benchmark not only identifies a human–AI gap but also offers an interpretable diagnosis of where that gap arises.
Critical weaknesses
ResearchMirror operationalizes research taste as agreement with the realized choices of a future research community, which is informative but not equivalent to scientific quality or normative research taste. Moreover, human papers may access 2026 literature and report completed results, while AI agents are restricted to the pre-2026 corpus and a 20-step proposal process. The broader interpretation of the human–AI gap should therefore be appropriately qualified.
Questions for authors
Could the authors provide a deeper analysis of the agents’ research trajectories, including their search queries, retrieved papers, intermediate hypotheses, and proposal revisions? In particular, for proposals with high MATCH-IDEA scores, it would be valuable to understand whether the final match results from a coherent and human-like refinement process or merely from convergence in the final text. A comparison between high- and low-matching trajectories could offer additional insight into how AI research ideas are formed.
Thank you for your detailed response. I believe the additional evidence you provided sufficiently addresses my concerns, and I would like to raise my score accordingly.