acceptodds
Under review as a conference paper at ICLR 2027

DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?

Abstract

Reliable automated research requires agents to vet data, verify analyses, and generate hypotheses grounded in trustworthy evidence, potentially reducing routine scientific workload while allowing scientists to focus on interpretation and discovery. Existing benchmarks often only assess analytical task completion or hypothesis generation separately rather than testing whether reliable evidence supports valid and novel claims. We introduce DISCERN (Data Integrity and Scientific Capability: Evidence, Reasoning, and Novelty), a controlled benchmark on real, publicly available datasets that evaluates three key levels of an automated research workflow. The first two levels test data integrity and analysis verification under confounds and tool traps, while the third tests hypothesis generation and revision under adversarial review, including counterfactual cases in which evidence consistent with real data and documented scientific phenomena conflicts with established expectations, motivating alternative explanations and testable hypotheses. Across 203 tasks, eight life-science tracks, and eight models, DISCERN shows that strong aggregate performance can mask level-specific weaknesses. Agents earn perfect scores in only 60.8% of Level 1, 34.2% of Level 2, and 0.6% of Level 3 evaluations, with penalties attributed to rejection of sound data, failure to carry recognized limitations into conclusions, and wide variation in hypothesis production. Cross-track rankings by token and code use are substantially more stable than rankings by evidence judgment, suggesting greater consistency in computational effort than in evidence-based reasoning. These profiles identify opportunities for supervised scientific assistance, but current agents do not yet demonstrate reliable autonomous analysis or discovery.

Primary area:
datasets and benchmarks
Keywords:
AI for Science, scientific discovery, AI agents, automated scientific research, hypothesis generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9MQUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces DISCERN, a benchmark for evaluating whether AI agents can assess the reliability of evidence and appropriately constrain their conclusions accordingly. The benchmark comprises 203 tasks across three independently evaluated levels: data integrity assessment (L1), analysis verification (L2), and hypothesis generation and revision (L3). The paper evaluates models from the GPT, Claude, Kimi, GLM, and DeepSeek families and provides extensive experimental analyses.

Critical strengths
  1. The paper is motivated by a clear and practically relevant problem. Prior work has identified misalignment between claims and supporting evidence as a recurring failure mode of agents conducting autonomous research [1]. This paper develops a systematic evaluation framework around this issue and analyzes the capabilities underlying such failures.
  2. The paper presents extensive experiments and detailed analyses. The evaluation metrics are thoughtfully designed and cover multiple complementary aspects of scientific reasoning and evidence handling.

[1] "How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks", Yanlin Fei, Nazhou Liu, et.al.

Critical weaknesses

Although the three levels are evaluated independently to isolate capabilities and weaknesses at different stages, the paper does not evaluate an end-to-end workflow connecting L1, L2, and L3. Consequently, it does not directly assess whether agents can correct errors across stages, how errors accumulate throughout the workflow, or how agents perform long-horizon planning. I regard this as a limitation of the evaluation scope, rather than an essential addition required for the current paper.

Questions for authors

None

3 comments

We thank the reviewer for their thoughtful assessment and recognition of DISCERN's motivation, comprehensive evaluation, and detailed analysis. We appreciate the distinction between stage-wise diagnostic evaluation and the additional capabilities required for end-to-end scientific research.

End-to-end evaluation and benchmark scope.

We agree that independently evaluating L1, L2, and L3 does not directly capture error propagation, recovery from earlier mistakes, or long-horizon planning in a continuous scientific workflow.

DISCERN evaluates these stages independently to isolate failures in different scientific reasoning capabilities. However, the stages are connected through their underlying data and reference analyses. Within each track, L2 tasks use data derived from the same source datasets as L1, while L3 evidence dossiers are constructed from the corresponding reference analyses and methodological context (Section 2; Appendix E). By providing fixed reference inputs between stages, DISCERN allows failures to be attributed to specific scientific decisions without confounding downstream performance with errors introduced earlier. Each track's composite score is calculated as the geometric mean of its L1–L3 scores, emphasizing balanced performance across levels rather than continuous workflow execution (Section 3; Appendix A).

To further examine performance across connected stages, we conducted an additional matched-trajectory analysis using existing model outputs. We identified 16 trajectories across eight scientific tracks, each linking an L1 data-integrity task, a corresponding L2 analysis, and the L3 evidence-evaluation task(s) derived from that analysis. Each trajectory is scored by the minimum of its three stage scores (0–3 scale), with success requiring every stage to score at least 2 of 3. Only 40 of 128 model–trajectory evaluations (31%) met this criterion, and only one achieved a perfect score across all three stages. These results reveal a substantial gap between performance on individual stages and consistent performance across linked scientific decisions.

Importantly, these trajectories are constructed from independently executed tasks and therefore do not measure error propagation or recovery during continuous execution. We will incorporate the matched-trajectory analysis into Table 1, document the task mappings in Appendix A, and clarify the distinction between stage-wise diagnostic evaluation and end-to-end scientific workflows in the Introduction and Discussion (Sections 1 and 6). Continuous evaluation using agents' own intermediate outputs remains a complementary direction for future work.

Relation to prior work.

AutoResearchEval (Fei et al., 2026) identifies misalignment between scientific claims and supporting evidence during end-to-end research execution. DISCERN examines a related failure mode under controlled conditions: among challenged L2 evaluations receiving full credit for issue recognition, 29.6% receive only half the available credit for conclusion calibration (Section 4.2). This distinction illustrates how DISCERN complements end-to-end evaluations by quantifying specific scientific reasoning failures under controlled conditions. We will incorporate AutoResearchEval into the related-work discussion (Section 5).

Thank the authors for the thorough rebuttal. I will maintain my score. Good luck.

Thank you for your thoughtful review and for considering our rebuttal. We appreciate your time and constructive feedback, and would be happy to clarify or discuss any remaining questions!

Reviewer_tkAkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DISCERN is an evaluation suite that aims to test whether agents can make sound scientific decision. The authors split the testing on 3 levels, first being data validity (L1), second analysis validity (L2) and third hypothesis validity (L3). The benchmark contains 203 primary tasks from public life-science datasets. The 203 are from biology with a small addition of calorimetry and nuclear decay separately. The evaluation probes the baseline examples with deliberately introduced perturbations and counterfactuals. As the result of the evaluation, authors report that the tested models achieved perfect scores in 60.8% of L1 evaluations, 34.2% of L2 evaluations, and 0.6% of L3 evaluations. From this, they conclude that current LLM agents can assist with specific tasks, but their scientific judgment remains inconsistent and requires human oversight.

The paper comes with a dataset and code for reproducing the experiments.

Critical strengths
  1. An important and timely evaluation problem. As AI agents increasingly support scientific research, it is important to understand how much of the scientific process we can delegate to agents. DISCERN is an attempt to addresses these decisions directly. The authors provide a substantial public benchmark package, including prepared tasks, evaluation code which is easily extensible to other models.

  2. The benchmark combines real datasets with clean controls and perturbations. Evaluating the data in such way helps identify where weaknesses occur. This design produces useful findings that aggregate performance alone doesn't show and gives researchers concrete targets for improvement.

  3. The paper includes robustness checks, examining alternative ways of combining scores, one versus three recorded LLM judgments, and different subsets of scientific tracks. These help assess how sensitive the findings are to scoring choices and task selection to remove the influence of the evaluation setting. They also tracked token budgets, albeit across different models, where they show that greater token budgets don't necessarily lead to better performance on their benchmark.

Critical weaknesses
  1. The paper’s title suggests a broad evaluation of scientific agents, but its main results come entirely from biology tasks. Making the biological focus more prominent would better align the framing with the evidence

  2. High L3 score does not mean that a model is good at generating hypotheses, so the framing it as 'testing hypothesis generation' is lacking a bit. Astra scores 72.8/100, despite proposing a hypothesis on only 1 of 20 eligible tasks. This is possible because a decision not to propose a hypothesis can earn up to 83.3/100. The authors report hypothesis counts separately, which helps clarify this limitation, but it somehow diminishes the importance of the main L3 score. I understand the difficulty in constructing this but it leaves something to be desired and devalues

  3. DISCERN evaluation suite as a self contained work is valuable, but the relevance and positioning in the overall body of work in this space is a bit lackluster in my opinion. Prior works in biology and other domains have demonstrated this jaggedness of models, if we can call it that, and I don't think that additional confirmation was really necessary. Framing the evaluation across different levels (L1-3) is a nice addition but as stated previously, in 2. point, I believe it could be developed a bit more.

Questions for authors
  1. How were the counterfactual and perturbation scenarios designed and selected? What roles did domain experts or LLMs play in choosing what modification to make to the clean controls?

  2. Thinking about future work; could LLM generated counterfactuals be used to train agents to handle conflicting scientific evidence better? Given the weaknesses observed in DISCERN and your data generating process, how would you verify that the generated examples are scientifically sound?

4 comments

Part 1/2: Scope and L3 scoring

We thank the reviewer for recognizing DISCERN's motivation, controlled design, and extensible benchmark. We respond with analyses of existing outputs and clarifications of scope.

1. Biological focus and applicability beyond life sciences (Weakness #1)

We agree that DISCERN's main empirical findings concern life sciences and that this scope should be more explicit. The benchmark contains 203 primary tasks across eight life-science tracks (Section 2).

The framework addresses data-integrity assessment, analysis verification, and evidence-grounded hypothesis evaluation across empirical sciences. The original submission includes an initial extension to calorimetry and nuclear decay using the same three-level structure and grading (Appendix F). Mean L2 performance is lower in the physics tracks (Section 6; Appendix F), motivating broader evaluation.

We will clarify this scope in the Abstract, Introduction, and Discussion, retain the broader framework title, and describe physics as a preliminary extension rather than validation of generalizability.

2. L3 scoring and hypothesis generation (Weakness #2)

We agree that the headline L3 score does not fully distinguish evidence-grounded reasoning from productive hypothesis generation. Under the original rubric, abstentions can receive partial credit for evidence fidelity and reasoning, capped at 83.3/100, even without generating a hypothesis (Section 3; Appendix J.3).

This explains Astra's high score despite committing on only 1 of 20 eligible tasks. Of its 19 abstentions, 14 provide an explicit evidentiary rationale (Section 4.3), but 18 occur on tasks where the evidence supports a hypothesis. Thus, Astra receives credit for evidence interpretation without making the expected hypothesis commitment. Thus, the aggregate score alone does not measure hypothesis-generation ability.

To quantify this distinction, we conducted two additional scoring analyses using existing outputs. The strict score assigns zero novelty credit to all abstentions, while the appropriateness-aware score preserves abstention credit only when withholding a hypothesis is the expected response.

ModelOriginal L3 (rank)Strict L3 (rank)Appropriateness-aware (rank)Committed / Eligible
GPT-6 Astra72.8 (1)50.3 (6)51.7 (6)1/20
GLM-5.371.1 (2)62.4 (1)64.1 (1)15/22
GPT-5.6-sol70.0 (3)51.7 (5)52.9 (5)4/22
Kimi K369.1 (4)62.3 (2)63.8 (2)16/21
Claude Opus 563.0 (6)59.0 (3)60.2 (3)17/21

Rankings are calculated across all eight models; five are shown.

Under both alternative scoring approaches, Astra falls from first to sixth, while GLM-5.3 ranks first. Among the 22 L3 tasks, 21 expect a hypothesis commitment, and all eight models correctly abstain on the remaining task (Appendix E.3). We will report these complementary scores alongside commitment rate, conditional hypothesis quality, and novelty yield (Table 1; Appendix H), distinguishing evidence interpretation, abstention appropriateness, and hypothesis generation.

Blinded human rescoring of 30/172 eligible L3 outcomes showed strongest agreement on novelty (weighted κ\kappa = 0.86; composite Pearson r = 0.76). See our jpWZ response; we will add details to Appendix J.

3. Positioning relative to prior work (Weakness #3)

Prior work documents uneven scientific-agent capabilities: ScienceAgentBench (Chen et al., 2025) and BixBench (Mitchener et al., 2025) evaluate scientific analyses; BixBench3 (Koch et al., 2026) examines study-scale workflows; GeneBench-Pro (Li et al., 2026) examines statistical reasoning and diagnostic decisions; and AutoResearchEval (Fei et al., 2026) diagnoses failures in end-to-end research.

DISCERN complements these benchmarks through matched clean controls, controlled analytical challenges, and model-specific counterfactuals (Appendix C, Table 3). The comparisons isolate three gaps: (i) defect detection versus acceptance of valid data (64.3% perfect on corrupted datasets versus 29.7% on clean controls; Section 4.3, Figure 4A); (ii) issue recognition versus conclusion calibration (29.6% of challenged L2 evaluations with full recognition earn only half calibration credit; Section 4.2); and (iii) evidence interpretation versus hypothesis commitment (Section 2 above). Unlike post-hoc analysis of continuous trajectories, DISCERN quantifies these decisions under predefined, controlled conditions. We will clarify this complementary positioning in Section 5 and Appendix C.

Part 2/2: Challenge validation and counterfactual training

4. Construction and validation of perturbations and counterfactuals (Question #1)

DISCERN's perturbations and counterfactuals were designed by domain experts on our team based on established scientific phenomena, quality-control practices, and common analytical pitfalls (Section 2; Appendix E.3). These include documented challenges involving TWAS effect directions, bivalent chromatin, clinical variant reclassification, and context-dependent gene essentiality. Rather than introducing arbitrary contradictions, injected counterfactuals modify upstream data and rerun the corresponding analyses to maintain consistency between the modified evidence and downstream results.

Each L1/L2 challenge undergoes reference-solver validation to verify that the intended perturbation produces a measurable analytical effect and that the correct analysis recovers the reference result. For example, in the Histone input-normalization challenge, unnormalized H3K27me3 coverage produces an association with gene expression in the opposite direction to the input-normalized reference analysis. Applying input normalization recovers the reference relationship.

Challenges that failed these checks were redesigned. Five of 12 initially designed Histone L2 challenges did not produce their intended effects, and one Protein L2 challenge required reconstruction. In the final benchmark, all 13 Enhancer L2 challenges passed their checks; four Protein mechanism checks and all eight released Protein task instances also passed re-verification.

LLMs were used to elicit model-specific prior expectations and test whether candidate scenarios produced the intended evaluation challenge during development. Domain experts selected the scientific modifications and verified their validity through reference analyses, rather than relying on LLM judgments to establish ground truth. We will expand the appendix to document these construction and validation procedures, including unsuccessful initial challenges.

5. LLM-generated counterfactuals for training (Question #2)

LLM-generated counterfactuals could provide a useful approach for training agents to evaluate evidence that conflicts with their prior expectations. However, generated examples require both statistical consistency and biological plausibility. The modified data must support the reported analytical results while representing a scientifically defensible phenomenon.

DISCERN's construction process provides a potential validation framework. Candidate counterfactuals would be grounded in documented scientific phenomena, followed by modifications to upstream data and regeneration of downstream analyses. Additional checks would assess whether the resulting patterns are biologically plausible and whether unrelated relationships remain consistent.

For example, our injected bivalent-promoter scenario uses H3K27me3 and expression values observed at a real bivalent promoter (DUSP4), while the MHC class II scenario introduces coordinated changes across related genes rather than an isolated single-gene effect. These examples illustrate how biological structure can inform counterfactual construction.

Domain-expert review would remain necessary to ensure that the modified evidence supports the intended scientific conclusion without implying an unsupported mechanism. Training should also include matched clean controls and prior-consistent examples so that agents learn when evidence warrants revising a prior belief and when existing conclusions should be retained.

We have not evaluated counterfactual-based training in this study and will discuss it as a potential extension in Section 6.

I appreciate you acknowledging the second issue with L3 scoring and hypothesis generation. I think it's important to be included in the main text.

Thank you for clarifying the rest of the points related to perturbations and counterfactuals.

Thank you for your follow-up and constructive feedback! We will ensure that the L3 scoring limitations and the distinction between evidence-grounded judgment and hypothesis generation are clearly addressed in the main text. We would be happy to discuss or clarify any remaining questions!

Reviewer_jpWZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DISCERN, a benchmark for evaluating LLM agent across different stages of scientifc research, including data processing, analysis verification and hypothesis generation. This benchmark includes 203 tasks across life-science domains and evaluate 8 state-of-the-art LLMs for that.

Critical strengths
  1. comprehensive benchmark design: The benchmark covers 203 tasks ranging from different tracks of life-science and three types of scientific reasoning. The evaluation also includes intermediate scientific decisions.

  2. detailed analysis: The paper evaluated 8 models and provides fine-grained analysis for scientific reasoning, hypothesis generation and ocmputational efficiency.

  3. inclusion of controlled challenges: The benchmark includs clean controls and counterfactual evidence provides a useful way to identify failure modes in specific LLMs.

  4. clear writing: the overall paper is clearly written and easy to understand.

Critical weaknesses
  1. hypothesis-generation evaluation validity: The L3 evaluation contains only 22 tasks and relies heavily on LLM-as-a-Judge. GPT-6 Astra achieves the highest L3 score while generating only one hypothesis across 20 eligible tasks. Although the authors analyze judge stability, they do not provide sufficient validation against independent domain experts. It remains unclear whether the scores reliably reflect scientific reasoning and hypothesis-generation capabilities, especially regarding novelty.

  2. saturated benchmark scores: The benchmark, as a newly release one, already shows substantial ceiling effects in L1. The strongest model achieves 86.1 points, and 60.8% of L1 evaluations receive perfect scores. This makes the benchmark less meaningful since a good benchmark should point out an ability of LLMs that are not good yet but worth working on.

  3. lack end-to-end evaluation: Although the benchmark covers three stages of scientific research, these stages are evaluated independently. It therefore remains unclear whether agents can reliably complete an integrated scientific workflow or how errors in earlier stages affect downstream analysis and hypothesis generation.

  4. lack quantified results on leakage analysis: The benchmark relies on publicly available scientific datasets and published findings that may have been encountered during model pretraining. Newly released models naturally have advantages over older models. Although the authors acknowledge this limitation and introduce controlled perturbations, they do not quantitatively assess the impact of potential data contamination. Additional analysis would help distinguish reasoning capabilities from familiarity with previously published results.

Questions for authors

How reliable are the LLM-as-a-Judge evaluations compared with human expert assessments?

Given that 60.8% of L1 evaluations already receive perfect scores, how can the benchmark remain challenging and discriminative as model capabilities improve?

What type of harness does the evaluation process rely on? A better harness can have different effects on the model performance.

5 comments

Part 1/3: L3 validity and human-expert agreement

We thank the reviewer for the constructive feedback and recognition of DISCERN's comprehensive design, controlled challenges, detailed analysis, and clear writing. We address the concerns with additional human-expert evaluation, quantitative analyses of existing results, and clarifications of the benchmark design.

1. L3 validity and human-expert agreement (Weakness #1; Question #1)

L3 scoring and hypothesis generation. DISCERN's headline L3 score measures evidence-grounded scientific judgment rather than hypothesis generation alone. Under the original rubric, abstentions can receive partial credit for evidence fidelity and reasoning, capped at 83.3/100 (Section 3; Appendix J.3). Astra provides an evidentiary rationale in 14 of its 19 abstentions (Section 4.3), contributing to its high score despite proposing only 1 hypothesis across 20 eligible tasks (Table 13; Figure 4B). However, 18 of its 19 abstentions occur on tasks where the evidence supports a hypothesis. Thus, providing a rationale for abstention does not necessarily indicate that abstention is scientifically appropriate.

To quantify this distinction, we conducted a sensitivity analysis removing novelty credit for abstentions:

ModelOriginal L3 (rank)Strict L3 (rank)Committed / Eligible
GPT-6 Astra72.8 (1)50.3 (6)1/20
GLM-5.371.1 (2)62.4 (1)15/22
GPT-5.6-sol70.0 (3)51.7 (5)4/22
Kimi K369.1 (4)62.3 (2)16/21
Claude Opus 563.0 (6)59.0 (3)17/21

Strict L3 removes novelty credit for abstentions. Rankings are calculated across all eight models; five are shown.

Astra falls from first to sixth under strict scoring, while GLM-5.3 ranks first. We further assessed abstention appropriateness: 21 of the 22 L3 tasks expect a hypothesis, and all eight models correctly abstain on the remaining task (Appendix E.3). We will report strict scoring, commitment rate, conditional hypothesis quality, and abstention appropriateness alongside the headline score (Table 1; Appendix H), distinguishing evidence-grounded judgment from productive hypothesis generation.

Blinded human validation. We conducted an additional blinded rescoring of 30 of 172 eligible L3 model--task outcomes (17%), covering 19 of 22 tasks, all eight scientific tracks, and all eight models. A domain expert from our team evaluated 18 committed hypotheses and 12 abstentions using the original rubric, without access to model identities or LLM-judge scores.

Agreement was strongest for novelty, with 78% exact agreement, 100% within-one-point agreement, and quadratic-weighted Cohen's κ\kappa = 0.86 (95% CI: 0.67--0.96). Across 162 paired axis ratings, exact agreement was 61%, within-one-point agreement was 88%, and weighted κ\kappa = 0.58. Composite scores were correlated (Pearson r = 0.76; Spearman ρ\rho = 0.79), with a mean absolute difference of 0.31 on the 0--3 scale.

Disagreements revealed specific limitations of LLM-based scoring. For citation integrity (κ\kappa = 0.22), the expert identified five outcomes containing bibliographic errors that the LLM judge scored highly; the judge lacked bibliographic lookup capability. For causal calibration (κ\kappa = 0.30), the expert was stricter in 9 of 10 disagreements, particularly when responses made causal overclaims before correcting them. These findings support the consistency of novelty scoring while highlighting dimensions where human review remains important. DISCERN already includes domain-expert review following LLM scoring (Section 3; Appendix J.1). We will add the agreement analysis to Appendix J, introduce bibliographic verification, and clarify that causal calibration considers the full response trajectory. This validation remains limited to a subset of outcomes assessed by one domain expert per task.

Part 2/3: L1 saturation and evaluation harness

2. L1 saturation and discriminative power (Weakness #2; Question #2)

Although 60.8% of L1 model--task evaluations receive perfect scores (Section 4.1), this aggregate rate masks substantial variation in task difficulty. Only 7 of 79 tasks (8.9%) are solved perfectly by all eight models. Thirty tasks (38%) are solved perfectly by at most half of the models and account for 72% of imperfect evaluations, while six tasks are solved perfectly by no model. Even on challenged datasets, 21.7% of evaluations identify the wrong cause of the defect (Appendix G).

At the model level, the strongest model solves 61 of 79 tasks perfectly (77.2%), while the remaining models achieve 20.3--75.9%. The reported maximum L1 score of 86.1 differs because it includes partial credit and track-level averaging. These results indicate that high aggregate performance does not imply consistent reliability across data-integrity challenges.

Importantly, L1 also evaluates whether agents recognize valid data. Perfect-score rates are substantially lower for clean controls (29.7%) than for datasets with introduced defects (64.3%; Section 4.3; Figure 4A). This distinction matters because unnecessary rejection or modification of valid data can compromise downstream analyses. Furthermore, L2 and L3 remain challenging, with perfect-score rates of 34.2% and 0.6%, respectively. We will add the task-difficulty breakdown to Appendix G (Figure 17) and discuss expanding task difficulty as models improve (Section 6).

3. Evaluation harness (Question #3)

DISCERN uses a standardized, minimal tool-calling harness across models, with Python execution in isolated sandboxes containing only permitted data and tools, without internet access (Section 2; Appendix I.1). Gold answers, graders, and other task worlds are inaccessible. The harness provides no additional retrieval, persistent memory, specialized planning, or multi-agent coordination. Models receive identical task prompts and reporting schemas within each analysis family.

We recognize that more capable harnesses could affect both absolute performance and model rankings. However, several failures occur despite successful tool execution, including rejecting valid data and failing to incorporate recognized analytical limitations into conclusions (Sections 4.2--4.3). These failures involve scientific judgment in addition to tool execution. We will clarify the harness configuration in Section 2 and discuss evaluating stronger scaffolding as future work.

Part 3/3: Connected stages and data contamination

4. Connected stages and end-to-end evaluation (Weakness #3)

We agree that evaluating L1--L3 independently does not directly measure error propagation, recovery, or continuous workflow execution. Each track's composite score is the geometric mean of its L1--L3 scores, emphasizing balanced performance across levels; the Overall Score averages these composite scores across tracks and is not an end-to-end execution score (Section 3; Appendix A).

The three levels are nevertheless connected through their scientific data and reference analyses. Within each track, L2 tasks use data derived from the same source datasets as L1, while L3 evidence dossiers are constructed from corresponding reference analyses and methodological context (Section 2; Appendix E). Each stage receives fixed inputs rather than an agent's earlier outputs, allowing failures to be attributed to specific scientific decisions without confounding downstream performance with upstream errors.

To further assess performance across connected stages using existing results, we identified 16 matched trajectories across eight tracks, comprising eight clean and eight issue-based trajectories. Each links an L1 data-integrity task, a corresponding L2 analysis, and the corresponding L3 evidence-evaluation task(s). We define the trajectory score as the minimum of the three stage scores (each on a 0--3 scale), requiring every stage to score at least 2 of 3 for success.

Only 40 of 128 model--trajectory evaluations (31%) meet this criterion, and only one achieves a perfect score across all three stages. This indicates that strong performance on individual levels does not necessarily translate into consistent performance across related scientific decisions.

These trajectories are constructed from independently executed tasks and therefore do not measure error propagation or recovery. We will add this analysis to Table 1, document the matched tasks in Appendix A, and clarify in the Introduction and Discussion (Sections 1 and 6) the distinction between stage-wise diagnostic evaluation and continuous end-to-end workflows. Evaluating agents using their own intermediate outputs remains a complementary direction for future work.

5. Potential data contamination (Weakness #4)

We agree that models may have encountered public scientific datasets and findings during pretraining, as acknowledged in Section 6. DISCERN incorporates researcher-introduced L1/L2 perturbations, isolated execution without internet access, and L3 counterfactuals evaluated only when they conflict with each model's elicited prior (Sections 2--3). These controls reduce reliance on familiar published answers but cannot exclude pretraining exposure.

We further analyzed existing L3 results without additional model runs. Mean scores were similar across real-data tasks (65.5; n = 64), native counterfactuals (63.6; n = 46), and injected counterfactuals (64.2; n = 62). Among 108 counterfactual outcomes, 12 (11.1%) reverted to prior expectations or failed to maintain the direction of the supplied evidence. Restricting to 92 outcomes where prior probing indicated familiarity with the published finding, 10 (10.9%) exhibited this behavior.

These results suggest that models frequently follow supplied counterfactual evidence even when it conflicts with their elicited prior knowledge. However, this analysis assesses responses to conflicting evidence rather than pretraining exposure directly and therefore cannot rule out contamination. We will include these quantitative findings and clarify their limitations in the revised manuscript.

Comments on the paper

1 comment

Dear Reviewers,

Thank you again for your thoughtful reviews and constructive feedback.

As the discussion period approaches its conclusion, we would appreciate hearing whether our rebuttal responses and additional analyses have addressed your concerns. We would be happy to clarify any remaining questions.

We appreciate your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.