acceptodds
Under review as a conference paper at ICLR 2027

Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle

Abstract

As foundation models advance and agent scaffolding becomes increasingly sophisticated, agents have demonstrated remarkable proficiency in complex, long-horizon coding tasks and even autonomous experiment execution. Despite their evolution from research assistants into autonomous research agents, these systems still exhibit significant limitations in field sensitivity, research ethics, and nuanced scientific judgment. Consequently, frontier agents remain unable to fully replace human researchers. To bridge this gap, we conceptualize the AARR (Act As a Real Researcher) benchmark series. Unlike existing benchmarks that primarily assess macro-level execution capabilities, AARR focuses on whether agents can emulate the professionalism, thoroughness, and nuanced reasoning that characterize human researchers in granular research scenarios. In this work, we propose AARRI-Bench (Act As a Real Research Intern), the first benchmark in this series. We conduct extensive experiments across frontier models and agentic systems, revealing that even the best-performing configuration (Mini-SWE-Agent with Claude Opus 4.7) achieves only 68.3% success rate, frequently overlooking subtle yet critical details that are obvious to real human researchers. Our results indicate that developing researcher-like AI requires further exploration of research behavior, beyond merely complex scaffolding.

Primary area:
datasets and benchmarks
Keywords:
benchmark, LLM agent, auto-research

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MD4SOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper conceptualizes the Act As a Real Researcher (AARR) benchmark series, which is meant to evaluate the behavior of research agents across the stages of a research career (intern → assistant → scientist). In this paper, they propose AARRI-Bench, the first benchmark in the series, which focuses on real research tasks that simulate intern activities.

According to the authors, existing benchmarks focus on the macro-level execution capabilities and don't focus on researcher qualities like "integrity, awareness of uncertainty, careful verification and responsible scientific reasoning". To address this, the proposed benchmark focuses on tasks that are easy for humans but hard for agents.

They have 82 tasks categorized along two orthogonal dimensions: horizontal (task scenarios: Context, Mindset, Interaction, Hands-on) and vertical (agent scope). These tasks are systematically and manually crafted in three stages by researchers ranging from senior Ph.D. students to undergraduate interns. Steps were taken to ensure the tasks had wide coverage with no overlap in subject matter. The evaluation metrics are based on two granularities: a coarse-grained 0/1 reward and fine-grained scores from multiple crafted unit tests. The tasks are run in isolated containers using the Harbor framework.

The authors evaluate 16 combinations of three harnesses (Claude Code, Hermes Agent, Mini-SWE-Agent) and frontier models. The best performing configuration (Mini-SWE-Agent with Claude Opus 4.7) achieves a 68.3% success rate on the benchmark. From this the authors conclude that "developing researcher-like AI requires further exploration of research behavior, beyond merely complex scaffolding".

Critical strengths
  1. Being able to identify where the agents struggle and humans don't is a great idea to base the benchmark on.
  2. The tasks were constructed by real researchers (Ph.D. students to undergraduate interns) with a grounding in pain points. The three stage construction seems very well thought out and executed.
  3. The paper tests three harnesses and several models and this also gives us a chance to separate the harness effects from the models.
Critical weaknesses
  1. In Appendix D.2 (lines 1547-1560) - "refuse to falsify data", the verifier runs five tests and the keyword-in-string check (fifth test) is considered the decisive test. But there are ways the model can refuse the user by not using the keywords (e.g., "I'd prefer not to do that, since it would misrepresent the results"). This suggests a weakness in measuring research quality.
  2. Since only one trial per task was taken into account, it would be interesting to see results for multiple trials and look at whether these results are consistent. The paper also mentions rerunning only "failed or noisy subsets when necessary" (line 751). It attributes reruns to "transient API failures or infrastructure issues" (lines 771–773), but the criteria for what counts as a failed or noisy subset, and how many trials were rerun, are not included.
  3. The paper states the harnesses differ by only 2.5 percentage points in the fine-grained scores (line 891). With only one run per task, a 3-point gap on 0/1 reward is not enough evidence to support the claim that minimal harnesses beat complex ones.
  4. While the authors have taken care to ensure there is no duplication in the tasks, some tasks may be ambiguous. A human evaluation of the benchmark would be a good baseline to confirm that the tasks are easy for humans.
  5. Table 2 doesn't cover the vertical (agent scope) dimension. It would be great to see results for the vertical dimension as well.
  6. The criteria used in Table 1 are not defined clearly, and several entries appear inconsistent with the cited papers (e.g., EXP-Bench reports separate design, implementation, execution and conclusion scores but is marked as lacking fine-grained evaluation; PaperBench's 8316 counts rubric items rather than tasks). This matters because two of the main contributions the paper claims, bringing researcher qualities into AI research evaluation and fine-grained evaluation, appear to be partly covered in prior work (e.g., EXP-Bench's integrity monitor checks for fabricated or placeholder data). A discussion of where AARRI-Bench differs from prior work, preferably with examples, would help.
  7. There seems to be an inconsistency in the Table 2 results. The weighted (pooled) average seems to be used for the Claude Code and Mini-SWE-Agent rows, but not for the Hermes Agent rows. For example, for Hermes Agent + Claude-Opus-4.7, the category scores give 62.2%, while the table reports 64.6%.

Focussing on areas where "agents struggle but humans don't" is a good idea, but the framing and evaluation need to be improved.

Questions for authors
  1. How many tasks use keyword-based checks? How did you validate the graders against human judgement?
  2. The paper states the results are from one trial, and trials were rerun only for subsets that failed or were noisy. Could you report results over multiple trials per task to show consistency in the results? What criteria did you use to decide that a subset had failed or was noisy, and how many trials did you rerun?
  3. Does the harness ranking still hold if you use consistent averaging and multiple runs?
  4. Did you run any human baselines, even on a subset? The assertion that these are tasks that humans find easy but agents struggle with would be stronger if we knew the results of human baselines.
  5. Please share the results for the vertical dimension.
  6. How did you define the ✓/✗ criteria in Table 1? It looks like the EXP-Bench and PaperBench rows are not consistent with what those papers state. (Please recheck the other benchmarks as well, only two were sampled).
  7. Could you please recompute all rows in Table 2 with one averaging method and state the per category task counts, and please specify if any tasks were dropped.
3 comments

We thank your feedback, especially criticism. We have done extensive additional experiments regarding to the reviews. We hope this will be helpful to resolve your concerns.

W1/Q1. Keyword checks and grader validation

Thank you for asking this. 68 of the 82 tasks contain a keyword check. We validated the graders in three ways. First, all 82 reference solutions pass the original verifier, which rules out broken or unsolvable tasks. Second, we read the runs whose only failing tests are lexical (75 of 1,637 runs, 4.6%), and checked whether each failure came from the research behavior or only from the wording. As for the D.2 refusal case, we did a thorough recheck. In that task, four configurations refused correctly and left the data unchanged, but failed only on wording. With file state as the primary check, the task moves from 8/20 to 12/20. We do not add an LLM judge. The reason is that AARRI-Bench is the first work in the AARR series and we keep the setup simple. A deterministic test suite is also easier to reproduce.

W2/Q2. Results over multiple trials per task

The release has 1,637 unique agent-task runs. We did not rerun tasks to improve scores. For stability, we run three trials on two configurations and report these results below.

Agent HarnessModelOverall(three trials per tasks)
Claude CodeDeepSeek-V4-Flash50.0% ± 2.1%
Hermes AgentDeepSeek-V4-Flash56.0% ± 3.1%

W3/Q3. Harness ranking

Yes. During our actual experiments, the ranking of the harnesses remained very stable. Due to cost constraints, we did not run multiple repeated trials for all agents, but as can be seen from the table above, the error across repeated runs does not exceed two tasks. We hope these results may address your concerns.

W4/Q4. human baseline

Thank you for pointing out the missing of human baseline. We ran an experiment with five participants on a global simple random sample of 16 tasks (seed AARRI-HumanEvalGlobal16-v1-20261113). Humans solve 11 of them averagely, matching the best agent on the same tasks, which also scores 11/16 under the same verifiers. Two of them also solved advice-recall-pro, which only 1/20 agent configurations passed. We are currently only able to test on a random subset, and we hope these results will address your concerns.

W5/Q5. Vertical dimension

The S1–S4 labels are now available. Across all 20 configurations, strict 0/1 success is below.

| Scope | Tasks | Strict 0/1 | |---|---:|---:|---:| | S1-Adaptation | 26 | 66.2% | | S2-Integration | 23 | 49.1% | | S3-Innovation | 22 | 41.8% | | S4-Open-ended | 11 | 46.4% |

W6/Q6. Table 1 criteria and entries

✓ means the benchmark explicitly evaluates the dimension as part of its task protocol while ✗ means it does not. EXP-Bench reports fine-grained design, implementation, execution, and conclusion scores over 12,737 subtasks, so its fine-grained entry is ✓. It also has an explicit integrity monitor for fabricated or placeholder data, so its researcher-quality entry is ✓. PaperBench’s 8,316 are rubric items across 20 papers, but in the original paper, they state "In total, PaperBench contains 8,316 individually gradable tasks". We think this is more about how you define what a task is. We find your comment about about rubric items and tasks quite reasonable and we finally change that slot into 20. We rechecked other rows also and found everything went well. The modified part of the table is right below.

BenchmarkEnd-to-EndFine-GrainedResearcher-QualityData GenerationMulti-Harness#Tasks
EXP-Bench✓✓✓Automatic✓461
PaperBench✗✓✗Transfer&Compose✗20

W7/Q7. Table 2 averaging and dropped tasks

Sure. You are right that the old Hermes rows were averaged differently from the other rows. This was a careless mistake and we're sorry for that. Fortunately, it did not affect our conclusions: Mini-SWE-Agent still works better than the other two. The table below recomputes all 16 rows. The per-category task counts are 34 context, 13 mindset, 21 interaction, and 14 hands-on. During evaluation, we didn't drop any task. Hopefully that will resolve your concerns.

HarnessModelOverall
Claude CodeGPT-5.3 Codex51.2
Claude CodeKimi-K2.650.0
Claude CodeQwen-3.6-Plus54.9
Claude CodeClaude-Opus-4.761.0
Claude CodeClaude-Sonnet-4.651.2
Claude CodeMiniMax-M2.753.7
Hermes AgentClaude-Opus-4.759.8
Hermes AgentClaude-Sonnet-4.652.4
Hermes AgentMiniMax-M2.751.2
Hermes AgentDeepSeek-V4-Flash56.1
Hermes AgentQwen-3.6-Plus57.3
Mini-SWE-AgentClaude-Opus-4.767.1
Mini-SWE-AgentDeepSeek-V4-Flash57.3
Mini-SWE-AgentKimi-K2.652.4
Mini-SWE-AgentMiniMax-M2.753.7
Mini-SWE-AgentQwen-3.6-Plus59.8

Thank you for the detailed responses and the additional results. The scope results (W5) address my request, and I appreciate the corrections to Table 1 and Table 2.

The keyword-check audit partly addresses W1, but it covers only the 75 runs in which every failing test is lexical, and only the D.2 change is reported. It is unclear how many of those runs changed outcome and whether the overall results were re-scored. The three-trial results are helpful, but they cover two configurations (one not in Table 2) and do not include Mini-SWE-Agent, so the claim that minimal harnesses outperform complex ones still rests on single runs. The criteria and number of reruns are also not given.

On the human baseline, humans average 11/16, the same as the best agent. This does not yet support the claim that the tasks are easy for humans but challenging for agents. It would help to know which tasks the participants failed and why; for example, whether the instructions were unclear.

With the corrected EXP-Bench row, its entries match AARRI-Bench’s on every column except data generation. Please state the specific difference between AARRI-Bench and EXP-Bench. Finally, nearly all rows of the recomputed Table 2 changed, not only the Hermes rows (e.g., Hermes + Claude-Opus-4.7 changes from 64.6% to 59.8%, while its category scores give 62.2%). Could you explain these changes?

Because the main concerns (keyword-based evaluation, single runs, and the difference from prior work) remain, my score remains unchanged.

Thank you for your timely reply. Due to the max characters limitation, we didn't display every details in last reply. Please bear me to add a few more very briefly.

As for the keyword-based evaluation, we actually rechecked all the trials automatically. The 75 runs were the runs that could be more possible to cause keyword issues so we checked them manually. For human baseline and multiple runs concerns, thanks again for the criticism. We have tried our best to do the verification with limited time and resources. The results of the subsets really relieved us and we hope they could help.

The most explanation I want to add is the difference between AARRI-Bench and EXP-Bench and other relevant works. The motivation behind EXP-Bench is whether an agent can actually carry out an experiment, while the motivation behind AARRI-Bench is whether an agent can do it like a researcher. The criterion for the former is whether the results are correct (whether the experiment can be reproduced). The criterion for the latter is whether the judgments made along the way are correct (being suspicious when one should be suspicious, refusing when one should refuse, verifying when one should verify). When we started this project, our bet was that the capabilities of AI agents would continue to grow, and that they would be able to handle almost any task with clear instructions. On the other hand, agents' understanding of the tacit rules of human society and human work is an area that has not yet been deeply studied, and this is exactly the capability needed when facing task scenarios with ambiguous instructions (like the real world). Therefore, reflecting on our own experience doing research, we created a series of research scenarios, hoping to evaluate agents along this dimension. This is by short the original intent behind the AARR project.

Reviewer_vosEOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces AARR, a benchmark series designed to evaluate whether LLM agents can behave like real researchers. The authors present 82 research-oriented tasks, and evaluate various frontier models and agent harnesses to identify their capabilities and limitations.

Critical strengths

The paper takes a systematic and interesting approach to evaluating whether agents possess the judgment, rigor, and research integrity expected of real researchers, rather than focusing solely on task completion. Some scenarios, such as testing whether agents would falsify experimental data, are particularly interesting.

Critical weaknesses
  • While the paper proposes three levels of research capability, it only evaluates the first (research intern). I would like to see a clearer discussion of how its tasks differ from existing benchmarks such as InnovatorBench[1], and a human baseline to support the claim that these tasks are easy for human researchers but challenging for agents.

  • The task construction process lacks details on annotation guidelines and quality control, such as cross-validation among annotators. Additionally, some verifiers rely heavily on keyword matching; for example, an agent can correctly refuse to falsify experimental data but still fail because its response lacks specific keywords, raising concerns about false negatives.

[1] InnovatorBench: Evaluating Agents' Ability to Conduct Innovative LLM Research

Questions for authors

see Weaknesses

1 comment

Thank you for the positive assessment. We are glad the emphasis on judgment, rigor, and integrity and that the falsification scenario was interesting. Both weaknesses are fair, and we will answer them below.

Q1/W1. Only the first level is evaluated; how is it different from InnovatorBench?

InnovatorBench and AARRI ask different questions. InnovatorBench gives the agent a task derived from a published LLM paper, a runnable workspace with code and data, a budget of 5–48 hours on 8 or 16 80GB GPUs, and asks it to design and implement a method that beats a reference solution. Scoring is a continuous, Kaggle-style metric between a baseline and the reference, and the agent can submit repeatedly and see its score. In short, it measures whether an agent can run a long GPU research project and improve a number.

AARRI contains 82 short, CPU-only tasks, mostly with a 10-minute budget. Each is built around a research intern decision like notice a version drift, find a NaN hidden in logs, catch a fabricated result, or resolve a conflict between two instructions. The output is usually a diagnosis, a decision, or a corrected file, graded by deterministic tests. InnovatorBench rewards beating a baseline while AARRI rewards doing the right thing when facing a specific scenario during doing research. InnovatorBench stresses long-horizon implementation and optimization while AARRI stresses judgment and integrity in short workflows. It's worthy to notice that scientist-level idea generation is the goal of AARRA/AARRS, not the main point of AARRI.

We're glad you found the need for human baseline. For the claim "easy for human researchers but challenging for agents", we urgently ran an experiment with five participants on a global simple random sample of 16 tasks (seed AARRI-HumanEvalGlobal16-v1-20261113). Humans solved 11 of them averagely, matching the best agent on the same tasks, which also scores 11/16 under the same verifiers. Two participants also solved advice-recall-pro, which only 1/20 agent configurations passed. Due to time and human resource constraints, we are currently only able to test on a random subset, and we hope these results will address your concerns.

Q2/W2. The construction process lacks annotation guidelines and QC, and some verifiers rely on keywords.

Thank you for asking for the construction details. We should have made them easier to find. The team was 8 researchers who are PhD students, MS student, and undergraduate interns from different research interests. Every task went through three stages. In stage 1, each author proposed tasks from a pain point they had actually hit when using agents in their own research. In stage 2, the senior authors reviewed each proposal for realism, solvability, and overlap with existing tasks, and gave written feedback. This is also where multiple annotators looked at the same task. In stage 3, we consolidated duplicates, fixed scope labels, and checked that every task had a reference solution and a runnable verifier. The task maker checked the task in stages 1 and 2, and the team lead checked it in stage 2 and 3.

On keyword verifiers, we agree that a lexical check can produce false negatives. 68 of the 82 tasks contain a keyword or regex check, and 31 contain a test that looks only at wording. In 75 of 1,637 runs (4.6%), every failing test is purely lexical. The way we avoid possible false negatives is to read those particular runs by hand. By doing a recheck, this issue has been resolved decently. We do not add an LLM judge: AARRI is the first benchmark in the AARR series, the setup is meant to stay simple, and a deterministic verifier is easier to reproduce.

Reviewer_nFBgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AARRI-Bench, a benchmark consisting of 82 tasks designed to evaluate whether AI agents can behave like real research interns. Unlike existing research benchmarks that primarily focus on task execution or final outcomes, AARRI-Bench emphasizes researcher-like qualities such as scientific judgment, research integrity, and awareness of subtle issues in research workflows. The authors evaluate multiple frontier models and agent harnesses, with the best configuration achieving a 68.3% success rate.

Critical strengths

I like the motivation of this work. Evaluating research agents beyond coding and experiment execution is an important and relatively underexplored direction.

The benchmark includes several interesting and realistic research scenarios, such as identifying suspicious experimental results, diagnosing failed experiments, and handling conflicting research priorities.

The evaluation covers multiple models and agent harnesses. The observation that a simpler harness can outperform more sophisticated systems is interesting.

Critical weaknesses

Limited evaluation of research taste, idea generation, and research prioritization. In my view, one of the most important capabilities of a strong researcher is not only proposing insightful ideas, but also identifying important problems and deciding which directions are worth pursuing under limited resources. In real research, time, compute, and human effort are constrained, so researchers need to prioritize ideas based on their potential impact, novelty, feasibility, and expected return on investment (ROI). Although the benchmark includes Innovation and Open-ended categories, many examples still focus on execution, error detection, or reaching predefined answers. It is unclear whether the benchmark captures this more fundamental aspect of research taste: choosing the most promising research direction when not every idea can be explored.

Lack of comprehensive per-task analysis. I would like to see the success rate of each of the 82 tasks, rather than mainly aggregate results across broad categories. Some tasks, such as refusing to falsify experimental results or checking dataset integrity, seem relatively straightforward and may not meaningfully distinguish strong research agents from general-purpose agents. A per-task breakdown would help identify which tasks actually reveal important limitations.

Missing human baseline. The paper repeatedly motivates its tasks as straightforward for human researchers but difficult for agents. A small human evaluation would make this claim more convincing.

Evaluation reliability. Some failures appear to be caused by rigid grading criteria rather than incorrect research behavior. For example, Appendix D.2 shows an agent correctly refusing to modify experimental data but failing because its response did not contain specific keywords. This raises concerns about whether the reported success rates accurately reflect research capabilities.

Questions for authors

Could the authors provide the per-task success rates for all 82 tasks? In particular, how many tasks are consistently solved by most agents, and which tasks meaningfully differentiate stronger models?

How do the authors distinguish failures caused by insufficient research judgment from those caused by formatting requirements or limitations of the verifier?

How do the Innovation and Open-ended tasks evaluate actual idea quality and research taste, rather than simply reaching expected answers?

1 comment

We are glad you found the motivation important and under-explored, the scenarios realistic, and the multi-model and multi-harness evaluation useful. We also understand the concerns behind your questions: whether the tasks actually separate strong configurations, whether a verifier failure is being counted as a research failure, and whether the human claim is supported. We take them in that order, then give the human baseline results.

Q1. Can you provide per-task success rates for all 82 tasks? How many are solved by most agents, and which tasks differentiate stronger models?

Of course we can. We've attached full table and heatmap in the supplement before. In summary, across the 20 released configurations (16 main + 4 legacy), 12 tasks are solved by all and 17 by none; 23 are at ≥90%, 24 at ≤10%, and 35 are in the 10–90% range. The largest gaps between the four strongest configurations (≥59.8%) and the five weakest (≤50.0%; missing runs counted as failures) are on tokenizer-version-drift, silent-nan-hunter, and false-guidance-rebuttal (100% vs 0%), followed by security-check, data-analize, and advice-recall (100% vs 20%). It is worthy to notice that ranking on the 35 mid-range tasks correlates 0.986 with the full ranking.

These six tasks share a pattern. The decisive fact is small and sits in an artifact rather than in the instruction. Strong agents understand the intention well while weak ones follow the surface request. That's the main reason why these tasks can differentiate stronger models.

Q2. How do you separate research-judgment failures from formatting or verifier failures?

You are right that a failed keyword test is not the same thing as bad research behavior, and we now keep the two apart. For each run we record how many unit tests failed, and we label each failure as judgment, format, or infrastructure. 68 of the 82 tasks contain some keyword or regex check, and 31 contain a test that looks only at wording. In 75 of 1,637 runs (4.6%), every failing test is lexical. The way we separate judgment failures from formatting failures is to read those particular runs by hand. By doing a recheck, this issue has been resolved decently.

We do not add an LLM as judge. AARRI-Bench is the first work in the AARR series and we keep the verifier setup deliberately simple; a deterministic test suite is also easier to reproduce exactly than a model-dependent judge. For the audited free-text cases, the behavior-first re-score already separates refusal behavior from keyword wording.

Q3. How do the Innovation and Open-ended tasks judge idea quality, rather than just expected answers?

We usually evaluate open-ended tasks by designing proper task scenario and verifier. The Innovation and Open-ended tasks do not ask for one expected sentence. They ask for a decision under constraints, and the verifier checks the constraints and the trade-offs. Take resource-constrained-triage as an example. In this task, the agent has 90 minutes for six student requests, one of which is a possible test-set leakage. Any schedule passes if it stays within 90 minutes, gives the leakage at least 20 minutes, gives the two low-stakes requests at most 10 minutes each, and justifies every allocation. In this scenario, many different schedules can pass.

Human baseline

We ran an experiment with five participants on a global simple random sample of 16 tasks (seed AARRI-HumanEvalGlobal16-v1-20261113). Humans solved 11 of them averagely, matching the best agent on the same tasks, which also scored 11/16 under the same verifiers. Two participants also solved advice-recall-pro, which only 1/20 agent configurations passed. Due to time and human resource constraints, we are currently only able to test on a random subset, and we hope these results will address the reviewers' concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.