- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces DISCERN, a benchmark for evaluating whether AI agents can assess the reliability of evidence and appropriately constrain their conclusions accordingly. The benchmark comprises 203 tasks across three independently evaluated levels: data integrity assessment (L1), analysis verification (L2), and hypothesis generation and revision (L3). The paper evaluates models from the GPT, Claude, Kimi, GLM, and DeepSeek families and provides extensive experimental analyses.
Critical strengths
- The paper is motivated by a clear and practically relevant problem. Prior work has identified misalignment between claims and supporting evidence as a recurring failure mode of agents conducting autonomous research [1]. This paper develops a systematic evaluation framework around this issue and analyzes the capabilities underlying such failures.
- The paper presents extensive experiments and detailed analyses. The evaluation metrics are thoughtfully designed and cover multiple complementary aspects of scientific reasoning and evidence handling.
[1] "How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks", Yanlin Fei, Nazhou Liu, et.al.
Critical weaknesses
Although the three levels are evaluated independently to isolate capabilities and weaknesses at different stages, the paper does not evaluate an end-to-end workflow connecting L1, L2, and L3. Consequently, it does not directly assess whether agents can correct errors across stages, how errors accumulate throughout the workflow, or how agents perform long-horizon planning. I regard this as a limitation of the evaluation scope, rather than an essential addition required for the current paper.
Questions for authors
None
3 comments
Thank the authors for the thorough rebuttal. I will maintain my score. Good luck.
Thank you for your thoughtful review and for considering our rebuttal. We appreciate your time and constructive feedback, and would be happy to clarify or discuss any remaining questions!
We thank the reviewer for their thoughtful assessment and recognition of DISCERN's motivation, comprehensive evaluation, and detailed analysis. We appreciate the distinction between stage-wise diagnostic evaluation and the additional capabilities required for end-to-end scientific research.
End-to-end evaluation and benchmark scope.
We agree that independently evaluating L1, L2, and L3 does not directly capture error propagation, recovery from earlier mistakes, or long-horizon planning in a continuous scientific workflow.
DISCERN evaluates these stages independently to isolate failures in different scientific reasoning capabilities. However, the stages are connected through their underlying data and reference analyses. Within each track, L2 tasks use data derived from the same source datasets as L1, while L3 evidence dossiers are constructed from the corresponding reference analyses and methodological context (Section 2; Appendix E). By providing fixed reference inputs between stages, DISCERN allows failures to be attributed to specific scientific decisions without confounding downstream performance with errors introduced earlier. Each track's composite score is calculated as the geometric mean of its L1–L3 scores, emphasizing balanced performance across levels rather than continuous workflow execution (Section 3; Appendix A).
To further examine performance across connected stages, we conducted an additional matched-trajectory analysis using existing model outputs. We identified 16 trajectories across eight scientific tracks, each linking an L1 data-integrity task, a corresponding L2 analysis, and the L3 evidence-evaluation task(s) derived from that analysis. Each trajectory is scored by the minimum of its three stage scores (0–3 scale), with success requiring every stage to score at least 2 of 3. Only 40 of 128 model–trajectory evaluations (31%) met this criterion, and only one achieved a perfect score across all three stages. These results reveal a substantial gap between performance on individual stages and consistent performance across linked scientific decisions.
Importantly, these trajectories are constructed from independently executed tasks and therefore do not measure error propagation or recovery during continuous execution. We will incorporate the matched-trajectory analysis into Table 1, document the task mappings in Appendix A, and clarify the distinction between stage-wise diagnostic evaluation and end-to-end scientific workflows in the Introduction and Discussion (Sections 1 and 6). Continuous evaluation using agents' own intermediate outputs remains a complementary direction for future work.
Relation to prior work.
AutoResearchEval (Fei et al., 2026) identifies misalignment between scientific claims and supporting evidence during end-to-end research execution. DISCERN examines a related failure mode under controlled conditions: among challenged L2 evaluations receiving full credit for issue recognition, 29.6% receive only half the available credit for conclusion calibration (Section 4.2). This distinction illustrates how DISCERN complements end-to-end evaluations by quantifying specific scientific reasoning failures under controlled conditions. We will incorporate AutoResearchEval into the related-work discussion (Section 5).