- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MAP-Eval, a benchmark for testing whether research agents can detect methodological flaws in medical AI projects without being told that a flaw exists. The authors construct 189 workspaces by injecting one flaw into each project and have human experts verify the flaws and their consequences. They evaluate ten models across nine harness settings, including full-context review, tool-based investigation, and review at different research stages. Models generally perform better with the full workspace provided, while tool use helps some models but hurts others.
Critical strengths
-
The paper contributes a benchmark for detecting methodological flaws in medical AI research. Human experts check the constructed instances, and human annotations are used to validate the automated judge.
-
The evaluation covers ten models and nine harness settings, allowing comparisons across models, tool use, and different stages of the research workflow.
Critical weaknesses
-
Although the study focuses on medical AI, prior work has reported similar findings about methodological failures in autonomous research, the limitations of evaluating papers without checking supporting artifacts, and models’ difficulty detecting scientific errors [1,2,3]. The authors should compare these findings with their own results and discuss the similarities, differences, and possible reasons behind them. This would help clarify what the medical setting adds and provide a deeper analysis beyond reporting performance differences.
-
Each model–instance–harness combination is evaluated using a single low-temperature generation, without repeated runs or uncertainty estimates. The authors should at least repeat evaluations on a representative subset to demonstrate that the observed performance differences are robust to sampling variability.
[1] Zhang et al. How Far Are We From True Auto-Research? 2026.
[2] Luo et al. The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systems 2025.
[3] Xi et al. FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers 2025.
Questions for authors
See Weaknesses.
5 comments
Response to Reviewer SrCE (Part 2/2)
Q2: Single-run evaluation and statistical uncertainty
“Each model–instance–harness combination is evaluated using a single low-temperature generation, without repeated runs or uncertainty estimates. The authors should at least repeat evaluations on a representative subset to demonstrate that the observed performance differences are robust to sampling variability.”
A2: The original experiments cover ten models, nine conditions, and 189 instances under uniform settings to evaluate proactive detection in medical research workflows and compare whether the same flaw is found under different conditions. The full results show consistent identification gaps across research stages and opposite tool effects for different models.
Your suggested repeated runs further test whether these patterns are sensitive to sampling and tool-trajectory variation. Thank you for this suggestion. We sampled two instances from each of the 33 pitfall types and ran three trials of full-workspace review, tool investigation, and PRA for three models, retaining the original decoding settings.
Consistent with the original observations, Qwen and Kimi retain the same directions of tool effects in every run, while Claude and Kimi consistently score lower in PRA than in full-workspace review. For NanoResearch's PRA stage (Nano-PRA), the gap from L1 is 32.3 points for Claude (95% paired bootstrap CI [18.7, 46.0]) and 23.2 points for Kimi (CI [10.1, 36.9]), both p < 0.001. Qwen's corresponding gap is 2.5 points (CI [−8.1, 13.1], p = 0.71).
We also ran paired bootstrap analyses on all 189 instances. Across ten models, L1 exceeds the six-framework-state mean by 22.7 points (95% CI [19.2, 26.2]) and PTR by 39.7 points (CI [36.0, 43.3]).
Repeated runs directly assess decoding and tool-trajectory variation; the full-instance analysis quantifies uncertainty in aggregate differences. The General Response summarizes the repeated-run table, with full results in Sections 3.2, 4.2–4.3 and Appendix M, Table 17.
References
[1] Zhang et al. How Far Are We From True Auto-Research? 2026.
[2] Luo et al. The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systems. 2025.
[3] Xi et al. FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers. 2025.
Following up on our responses and additional experiments
Dear Reviewer SrCE:
We sincerely appreciate your valuable time devoted to reviewing our manuscript. We would like to gently remind you of the approaching deadline for the discussion phase. We have diligently addressed the issues you raised in your feedback, providing detailed explanations and statistical analyses. For instance, we conducted three independent repeated runs across all 33 pitfall types and paired bootstrap analyses over all 189 instances, confirming that both the workflow-stage drops (a 22.7-point gap between full-workspace review and workflow states) and the model-specific tool effects are highly consistent against sampling variability (run-to-run SDs of 0.9%–2.3%). Moreover, we expanded Sections 2.5 and 7 and Table 1 to compare our findings with Zhang et al., Luo et al., and Xi et al., elucidating how tool-based evidence acquisition, cross-file clinical reasoning, and unsafe deployment decisions distinguish proactive pitfall detection in medical research. Would you kindly take a moment to look at it?
We are very enthusiastic about engaging in more in-depth discussions with you.
Response to Reviewer SrCE (Part 1/2)
We thank you for your insightful comments on the relationship to prior work and statistical robustness. We provide detailed responses below.
Q1: Prior findings and the medical setting
A1: MAP-EVAL places these questions in medical autonomous research workflows, testing whether agents proactively identify problems without an error cue and judge whether the results support clinical deployment.
Prior studies show that code, data, and logs help identify errors in papers. Our experiments further hold the same methodological flaw fixed to observe whether agents actively seek and use this evidence at different research stages. Sections 2.5, 7 and Table 1 now include the following comparisons.
The difference from Luo et al. can be understood through evidence acquisition. Their reviewer directly receives code, traces, and issue-specific detection rules; in L2, agents must locate relevant files through tools. Appendix H shows traces where agents accept successful execution or high metrics without examining the evidence containing the methodological problem. How evidence is provided and how agents investigate it both affect detection.
The medical setting requires clinical reasoning across files and a judgment about deployment. Checking patient-level independence requires connecting split code to repeated encounters, radiographic views, or ECG segments. Detecting scanner shortcuts requires relating acquisition metadata to labels; checking prevalence mismatch requires comparing the evaluation cohort with the target deployment population. These problems can coexist with successful execution and consistent logs.
Deployment decisions also provide information beyond identification rates. Claude Sonnet-4.6 identifies more flaws than Gemini-3.1-Pro (62% vs. 56%), yet has higher unflagged unsafe deployment (19% vs. 6%). How many problems an agent detects and whether it approves unreliable clinical claims therefore need separate evaluation. Sections 2.5, 3.3, 5.2 and 7 explain these requirements of the medical setting.