acceptodds
Under review as a conference paper at ICLR 2027

When We Evaluate AI Scientists, What Are We Evaluating ?

Abstract

The highly complex and multifaceted nature of scientific research makes it particularly challenging to accurately assess AI agents in research workflows. Specifically, frontier AI trained in reward-seeking environment tend to learn shortcuts that achieve higher scores without improving the underlying science. We term this failure mode evaluation hacking where AI might learn to game proxy metric, leverage memorized data or look for easy answers online when presented with highly challenging tasks. In this study, we first present a systematic taxonomy on how evaluation hacking could occur and demonstrate their failure case on MLR-Bench, ResearchClawBench, and SciAgentGym. We find that susceptibility differs substantially across models and scientific domains. We observed that score pressure induce less faithful reports with forged numbers and sometimes in direct contradiction to objective evidence. These strategic misrepresentation can occur even when agents do not explicitly describe a plan to game the evaluation metric. Adding rules that account for evidence support can reduce, but not eliminate such harmful behavior across all 3 scenarios. In conclusion, this work seeks to serve as a pilot study that uncovers the covert failure mode of eval hacking in many AI for Research and RSI evaluation, even though no graders are broken and no questions are factually incorrect, frontier agents have shown increasingly stronger capability to infer and game eval metric without breaking/altering any code.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
evaluation hacking, reward hacking, AI agents, scientific agents, AI safety, benchmark evaluation, evaluator robustness, evidence faithfulness, LLM agents, scientific benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gJayOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper tries to answer the question can AI research agents get higher scores on science benchmarks without doing better science? The paper calls this “evaluation hacking”.

The authors test this by giving the AI the same tasks twice, once asking the AI to be responsible (focus on accuracy, evidence, admit uncertainties) and once asking the AI to be 'score-maximizing'.

Then they grade the results with the benchmarks' scores and use an AI checker to verify the evidence behind the AI's answers.

The benchmarks are ResearchClawBench, MLR-Bench, and SciAgentGym.

Their main results are: 'score-maximizing' increases the benchmark's grade but lowers the support; the pattern differs by model; the models notice the metric but rarely plan to cheat; a support-aware scoring rule helps in two of three benchmarks; and a model whose refusal ability was removed doesn’t chase the score more.

Critical strengths

The idea is intuitive and easy to follow, and the underlying question the authors are trying to answer is important to ask.

Critical weaknesses
  1. The writing is hard to follow due to: repetitive words; constant references to earlier versions that the reader never sees (“Earlier analyses… are kept separate”, “replace the earlier unreproduced value of 0.0014”); numbers printed with too many digits (0.8135593220, 0.9751007825); the sudden introduction of concepts that were not defined earlier (“RSI evaluations”, “utilize memorized data or look for easy answers online”, ε_S and ε_Q, “IRL science reward”, etc.); and several invented terms such as “fixed-science vulnerability,” “score pressure,” “frozen-path audit,” “strict unsupported-success endpoint” and “harmful switch.” Throughout the paper, the text also reads defensively, with many disclaimers following the results, making the reader wonder why the authors raise them.
  2. The prompts of the AI checker are missing, so a reader cannot determine whether the main finding is real or comes from biased prompting.
  3. The authors test a multi-model system throughout the paper, but they keep changing the models behind the multi-model system, the graders, and the test types across benchmarks. The lack of consistency makes their evidence weaker.
  4. There are existing related works that the authors do not discuss, such as: a. Hidden Pitfalls of AI Scientist Systems. It finds AI-scientist systems misusing metrics and picking results after the fact. b. LMs Learn to Mislead Humans via RLHF (ICLR 2025). Models become more convincing without becoming more correct. c. Reward Hacking Benchmark (RHB) (May 2026). Agents skip verification steps in tool use, like the SciAgentGym shortcuts. RHB also links reinforcement-learning post-training to exploits, which is the training claim this paper makes but cannot show.
  5. Although the paper contains many studies, the scope of most of them is small: ResearchClawBench: 20 tasks, or 39 in the choice study MLR-Bench: 10 tasks SciAgentGym: 13 tasks ResearchClawBench and MLR-Bench mainly use one agent model each.
  6. The abstract blames training, but only the instruction is tested, and the body admits this five times.
  7. The authors say their human audits don’t validate the checker’s quality scores (l.755).
Questions for authors
  1. Is it possible to share the prompts you used for the AI checkers?
  2. Is it possible to rerun the experiments with a consistent model configuration?
  3. Is it possible to adjust the scope by going deeper into each study and narrowing the main claims, while discussing the existing literature more thoroughly?
  4. Is it possible to define the terms that are uncommon in this field before using them?
4 comments

Dear Reviewer,

This is just a gentle reminder that we have 8 days left, please kindly let us know if you have any remaining concerns.

Best, The Authors

Dear reviewer,

This is just a gentle reminder to kindly read our rebuttals. We have roughly 4 days left. If you have any remaining concerns please let us know.

Best, The Authors

Respected Reviewer,

Please let us know if your comments are addressed or not. We just have 1.5 days left we would be very grateful if you can let us know what you think of our rebuttals.

Best, The authors

Respected Reviewer,

Please let us know if you have any additional questions. We have less than 48 hours.

Best,

The Authors

Reviewer_7XRvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the reliability of evaluation procedures in scientific-agent benchmarks. Through controlled modifications to model outputs and comparisons under different instructions, it examines whether benchmark scores can increase as independently audited evidence support declines, and whether score-maximizing instructions encourage such outcomes. The paper also incorporates evidence support into scoring rules and tests whether these revisions reduce score advantages associated with weaker evidence support and encourage better-supported outputs when disclosed to agents. Experiments on ResearchClawBench, MLR-Bench, and SciAgentGym reveal cases where scores increase as evidence support declines, or remain unchanged despite weaker support. Evidence-aware scoring mitigates some of these failures but does not eliminate them.

Critical strengths
  1. The paper addresses an important problem: whether scientific-agent benchmark scores reflect the evidence supporting their outputs.
  2. The Observe–Infer–Act framework helps distinguish evaluator vulnerabilities, instruction-induced behavior, and visible exploit planning. Separate scoring-repair experiments examine changes in both scores and subsequent behavior.
  3. Holding scientific artifacts fixed helps isolate score improvements that do not reflect better underlying research.
Critical weaknesses
  1. The paper is difficult to follow. Different samples and metrics are interleaved without clear subsection headings, and the tolerance symbols in line 109 are introduced without sufficient explanation.

  2. The main findings largely confirm the expected mismatch between benchmark scores and evidential support. The paper provides useful examples, but offers limited additional insight into when these failures arise or how to prevent them reliably. The proposed repair mainly incorporates the same support criterion used to diagnose the failures.

  3. Although task-level statistical tests are reported, the small task sets limit generalization, particularly for comparisons across scientific domains.

Questions for authors
  1. The evidence-support scores lack direct validation. Human checks assess option selection and reasoning labels, but do not validate the scientific-support scores underlying the main conclusions. Do the authors have additional data validating these scores?

  2. Explicit score-maximizing prompts demonstrate susceptibility under pressure, but provide limited evidence about how often these failures occur in ordinary use. Do the authors have data or statistics on the frequency of these failures in ordinary use?

2 comments

Dear Reviewer,

This is just a gentle reminder that we have 8 days left, please kindly let us know if you have any remaining concerns.

Best, The Authors

Dear reviewer,

This is just a gentle reminder to kindly read our rebuttals. We have roughly 4 days left. If you have any remaining concerns please let us know.

Best, The Authors

Reviewer_brjeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether scientific-agent benchmarks reward outputs that are less supported by the underlying evidence. It distinguishes evaluator vulnerability under fixed-output edits from behavioural changes under score-maximizing instructions, using an Observe–Infer–Act framework across ResearchClawBench, MLR-Bench, and SciAgentGym. The experiments hold artifacts or candidate reports fixed where possible, compare official scores with separate support audits, and examine hidden regrading and disclosed evidence-aware objectives. The clearest findings concern report generation: ResearchClawBench has harmful score/support changes on most tasks, while MLR-Bench has them on all tasks. SciAgentGym provides a more qualified result: score pressure increases direct/no-tool behaviour and lowers trace support, but neither official SPL nor the stricter unsupported-success endpoint significantly increases. Additional studies examine report choices, visible decision notes, annotation reliability, and refusal suppression.

Critical strengths
  1. The interventions separate several commonly conflated questions. Fixed-science edits test an evaluator's vulnerability; instructed generation tests behaviour; hidden regrading tests a changed scoring rule; disclosure tests adaptation. Keeping code and logs fixed makes fabricated numerical claims and unsupported experiments much more interpretable than an uncontrolled difference between complete agent runs. The ResearchClawBench and MLR-Bench results therefore identify a practically relevant failure mode rather than merely a correlation between scores and writing style (Sections 3–5; Tables 4–6).

  2. The analysis preserves important negative evidence and avoids several statistical shortcuts. Repeated runs are grouped by task, including the exact ten-task MLR-Bench sign-flip test (Appendix A.4). The paper reports that SciAgentGym's strict unsupported-success endpoint does not significantly increase, that no episode passes the full six-stage inference-to-action test, and that ResearchClawBench's hidden-regrading improvement is not significant. These qualifications materially improve the credibility of the narrower conclusions.

Critical weaknesses
  1. The central evidence-quality construct has limited independent validation. The primary support scores are model-audited, while the separate audit subsets introduce another rubric rather than a demonstrated human reference for scientific support. The two human studies resolve selected options and classify visible reasoning; the manuscript explicitly notes that neither validates the scientific-quality scores (Section 4; Appendix B). This does not establish that the audits are incorrect; their error rate and sensitivity to condition remain unclear. A prompt-induced change in wording could affect both the official judge and the support auditor. Concrete mismatched numbers are compelling examples, but do not calibrate the aggregate judgments. Because evidence-aware scoring also incorporates these judgments, mitigation can improve the measured construct without independently establishing more faithful science. A blinded claim-to-artifact audit of matched outputs would strengthen both conclusions.

  2. The broad interpretation exceeds the strongest measured endpoints. The main contrast explicitly asks for score maximization without new scientific work; it establishes susceptibility under that stress test, not its prevalence in ordinary research workflows or acquisition through reward training. In SciAgentGym, direct answers can remain semantically correct, and the strict unsupported-success endpoint is only 20/390 trajectories with no significant score-pressure increase (Appendix A.7). Likewise, the six-stage test has 0/59 complete successes (Appendix E). The paper often acknowledges these boundaries, but the abstract's inference/gaming language and cross-benchmark narrative should consistently reflect them. The Observe–Infer–Act framing is useful, but its “Infer” component is not comparably supported.

Questions for authors
  1. Please specify the principal and independent support-audit rubrics, aggregation rules, grader identities, and repeated-grading procedures. Were auditors blinded to the prompting condition and official scores? Was an auditor independent of the disclosed repair rule used to assess patched outputs? Please distinguish the principal support measure, earlier artifact-faithfulness checks, and independent audit subsets, and explain how grading variation near the zero-tolerance thresholds affects strict harmful-event counts (Section 4; Appendices A and F). These details are needed to assess measurement reliability and the evidence for mitigation.

  2. Do you have a blinded human audit of the actual claim-to-artifact or trace-support judgments, stratified across responsible, score-maximizing, and patched outputs? If not, could you provide such validation on matched outputs, reporting agreement on the direction of support changes, errors by condition, and corrected harmful-event rates? The option-resolution and visible-decision-note studies validate different annotation tasks, as the manuscript acknowledges (Section 4; Appendix B). This additional evidence could materially change my assessment; more annotations of those existing tasks alone would not resolve the support-measure concern.

  3. Is the main claim limited to evaluator vulnerabilities and behaviour under the tested instructions? Please align the abstract and conclusions with the nonsignificant increase in SciAgentGym's strict unsupported-success endpoint, the 0/59 complete inference-to-action chains in ResearchClawBench, and the distinction between hidden regrading and changed behaviour after disclosure (Section 5; Appendices E–F). The 0/59 result does not negate the measured behavioural effects, and the strict harmful-event definition does not require intent. Which stronger claims about deliberate strategy, ordinary-workflow prevalence, or reward-training effects are intended, and what evidence supports them?

  4. Could you reconcile Table 1's N=242 model-versus-consensus entries with Appendix B.1's 237-case full human-consensus reference and 236-case majority comparison, as well as Table 3's 192- and 191-case concrete-reference comparisons? The statement that every model supplied 242 usable labels also appears inconsistent with the 241 responses having all three labels. Please give the actual denominator for each comparison, exclusions or missing-label reasons, and whether corrections change agreement estimates or statistical conclusions. I regard this as a correctable reporting inconsistency unless reconciliation changes the substantive results.

  5. Could you correct the statement on page 5, line 246 that the 11 variants gain both judge score and faithfulness, given the reported negative faithfulness change? Also, Figure 6's assertion that refusal suppression does not change sensitivity appears stronger than confidence intervals spanning zero establish (Appendix D.1). Was an equivalence analysis with a justified margin conducted? If not, please describe the result as no consistent difference detected and retain the uncertainty. These corrections should distinguish a wording error from an unsupported equivalence interpretation.

  6. There are several citation issues detected by AI: Refusal in Language Models Is Mediated by a Single Direction (NeurIPS 2024), ScienceAgentBench (ICLR 2025), and Language Models Don't Always Say What They Think (NeurIPS 2023); these are published conference papers and cited using arXiv links

2 comments

Dear Reviewer,

This is just a gentle reminder that we have 8 days left, please kindly let us know if you have any remaining concerns.

Best, The Authors

Dear reviewer,

This is just a gentle reminder to kindly read our rebuttals. We have roughly 4 days left. If you have any remaining concerns please let us know.

Best, The Authors

Comments on the paper

2 comments

Dear Reviewers,

As the discussion phase draws to a close, please acknowledge the author's response and let them know whether it has addressed some or all of your concerns.

Thank you for the time and effort you and the other reviewers have put into this discussion.

Thank you for taking the time, we'd like to emphasize that this study seeks to uncover problems in current AI scientist benchmarks (which is what the field currently uses in real application) that provide guidance for future evaluation paradigm.

1. What do we learn beyond the score–support mismatch?

Our aim is not just to show that current benchmark scores can be misleading and uncover the underlying mechanism, which is critical for construct validity of AI evaluation (in itself is an important contribution to the field's collective understanding!)

We saw different issues across the three benchmarks. In ResearchClawBench we can see stronger claims being awarded a higher score even when the evidence is the same. In MLR-Bench the issue arises when agents are producing full reports (not in the 20 simple fixed edits). In SciAgentGym agents can get credit for a shorter path even when their tool steps don't actually support the answer.

We also tried two approaches to improve the evaluation: changing the scores of the existing outputs and informing the agents of the new scoring rule prior to them acting. Our results show that the new scoring rule can eliminate unsupported shortcuts whilst still rewarding useful shorter paths (Sec. 5.4; Appendix F). We think these distinctions are useful in general beyond the three benchmarks that we've looked at, even if we don't claim to have uncovered the internal mechanisms that are responsible for the behaviour.

2. How consequential are these failures in practice?

We find this comment to be nitpicking and invalid (which we have reported to SAC/AC as such), this kind of comments apply to the vast majority of papers in our field so long as they don't offer field experiments (and even when they do, such nitpicking can be about their sample size)

Our focus is on AI benchmarks which is used by the field to assess how AI would perform in real research workflow, where we have shown actual cases where the scientific conclusions can be very misleading. In ResearchClawBench one report shows to have a 6.4 sigma tension when the results actually should support 3.4 sigma. In MLR-Bench we have reports that talk about repeated experiments and additional baselines that the agent didn't run. All these instances aren't differences in the way the text is written, but these issues can change what the reader would conclude from the experiments that are reported.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.