- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper conceptualizes the Act As a Real Researcher (AARR) benchmark series, which is meant to evaluate the behavior of research agents across the stages of a research career (intern → assistant → scientist). In this paper, they propose AARRI-Bench, the first benchmark in the series, which focuses on real research tasks that simulate intern activities.
According to the authors, existing benchmarks focus on the macro-level execution capabilities and don't focus on researcher qualities like "integrity, awareness of uncertainty, careful verification and responsible scientific reasoning". To address this, the proposed benchmark focuses on tasks that are easy for humans but hard for agents.
They have 82 tasks categorized along two orthogonal dimensions: horizontal (task scenarios: Context, Mindset, Interaction, Hands-on) and vertical (agent scope). These tasks are systematically and manually crafted in three stages by researchers ranging from senior Ph.D. students to undergraduate interns. Steps were taken to ensure the tasks had wide coverage with no overlap in subject matter. The evaluation metrics are based on two granularities: a coarse-grained 0/1 reward and fine-grained scores from multiple crafted unit tests. The tasks are run in isolated containers using the Harbor framework.
The authors evaluate 16 combinations of three harnesses (Claude Code, Hermes Agent, Mini-SWE-Agent) and frontier models. The best performing configuration (Mini-SWE-Agent with Claude Opus 4.7) achieves a 68.3% success rate on the benchmark. From this the authors conclude that "developing researcher-like AI requires further exploration of research behavior, beyond merely complex scaffolding".
Critical strengths
- Being able to identify where the agents struggle and humans don't is a great idea to base the benchmark on.
- The tasks were constructed by real researchers (Ph.D. students to undergraduate interns) with a grounding in pain points. The three stage construction seems very well thought out and executed.
- The paper tests three harnesses and several models and this also gives us a chance to separate the harness effects from the models.
Critical weaknesses
- In Appendix D.2 (lines 1547-1560) - "refuse to falsify data", the verifier runs five tests and the keyword-in-string check (fifth test) is considered the decisive test. But there are ways the model can refuse the user by not using the keywords (e.g., "I'd prefer not to do that, since it would misrepresent the results"). This suggests a weakness in measuring research quality.
- Since only one trial per task was taken into account, it would be interesting to see results for multiple trials and look at whether these results are consistent. The paper also mentions rerunning only "failed or noisy subsets when necessary" (line 751). It attributes reruns to "transient API failures or infrastructure issues" (lines 771–773), but the criteria for what counts as a failed or noisy subset, and how many trials were rerun, are not included.
- The paper states the harnesses differ by only 2.5 percentage points in the fine-grained scores (line 891). With only one run per task, a 3-point gap on 0/1 reward is not enough evidence to support the claim that minimal harnesses beat complex ones.
- While the authors have taken care to ensure there is no duplication in the tasks, some tasks may be ambiguous. A human evaluation of the benchmark would be a good baseline to confirm that the tasks are easy for humans.
- Table 2 doesn't cover the vertical (agent scope) dimension. It would be great to see results for the vertical dimension as well.
- The criteria used in Table 1 are not defined clearly, and several entries appear inconsistent with the cited papers (e.g., EXP-Bench reports separate design, implementation, execution and conclusion scores but is marked as lacking fine-grained evaluation; PaperBench's 8316 counts rubric items rather than tasks). This matters because two of the main contributions the paper claims, bringing researcher qualities into AI research evaluation and fine-grained evaluation, appear to be partly covered in prior work (e.g., EXP-Bench's integrity monitor checks for fabricated or placeholder data). A discussion of where AARRI-Bench differs from prior work, preferably with examples, would help.
- There seems to be an inconsistency in the Table 2 results. The weighted (pooled) average seems to be used for the Claude Code and Mini-SWE-Agent rows, but not for the Hermes Agent rows. For example, for Hermes Agent + Claude-Opus-4.7, the category scores give 62.2%, while the table reports 64.6%.
Focussing on areas where "agents struggle but humans don't" is a good idea, but the framing and evaluation need to be improved.
Questions for authors
- How many tasks use keyword-based checks? How did you validate the graders against human judgement?
- The paper states the results are from one trial, and trials were rerun only for subsets that failed or were noisy. Could you report results over multiple trials per task to show consistency in the results? What criteria did you use to decide that a subset had failed or was noisy, and how many trials did you rerun?
- Does the harness ranking still hold if you use consistent averaging and multiple runs?
- Did you run any human baselines, even on a subset? The assertion that these are tasks that humans find easy but agents struggle with would be stronger if we knew the results of human baselines.
- Please share the results for the vertical dimension.
- How did you define the ✓/✗ criteria in Table 1? It looks like the EXP-Bench and PaperBench rows are not consistent with what those papers state. (Please recheck the other benchmarks as well, only two were sampled).
- Could you please recompute all rows in Table 2 with one averaging method and state the per category task counts, and please specify if any tasks were dropped.
3 comments
Thank you for the detailed responses and the additional results. The scope results (W5) address my request, and I appreciate the corrections to Table 1 and Table 2.
The keyword-check audit partly addresses W1, but it covers only the 75 runs in which every failing test is lexical, and only the D.2 change is reported. It is unclear how many of those runs changed outcome and whether the overall results were re-scored. The three-trial results are helpful, but they cover two configurations (one not in Table 2) and do not include Mini-SWE-Agent, so the claim that minimal harnesses outperform complex ones still rests on single runs. The criteria and number of reruns are also not given.
On the human baseline, humans average 11/16, the same as the best agent. This does not yet support the claim that the tasks are easy for humans but challenging for agents. It would help to know which tasks the participants failed and why; for example, whether the instructions were unclear.
With the corrected EXP-Bench row, its entries match AARRI-Bench’s on every column except data generation. Please state the specific difference between AARRI-Bench and EXP-Bench. Finally, nearly all rows of the recomputed Table 2 changed, not only the Hermes rows (e.g., Hermes + Claude-Opus-4.7 changes from 64.6% to 59.8%, while its category scores give 62.2%). Could you explain these changes?
Because the main concerns (keyword-based evaluation, single runs, and the difference from prior work) remain, my score remains unchanged.
Thank you for your timely reply. Due to the max characters limitation, we didn't display every details in last reply. Please bear me to add a few more very briefly.
As for the keyword-based evaluation, we actually rechecked all the trials automatically. The 75 runs were the runs that could be more possible to cause keyword issues so we checked them manually. For human baseline and multiple runs concerns, thanks again for the criticism. We have tried our best to do the verification with limited time and resources. The results of the subsets really relieved us and we hope they could help.
The most explanation I want to add is the difference between AARRI-Bench and EXP-Bench and other relevant works. The motivation behind EXP-Bench is whether an agent can actually carry out an experiment, while the motivation behind AARRI-Bench is whether an agent can do it like a researcher. The criterion for the former is whether the results are correct (whether the experiment can be reproduced). The criterion for the latter is whether the judgments made along the way are correct (being suspicious when one should be suspicious, refusing when one should refuse, verifying when one should verify). When we started this project, our bet was that the capabilities of AI agents would continue to grow, and that they would be able to handle almost any task with clear instructions. On the other hand, agents' understanding of the tacit rules of human society and human work is an area that has not yet been deeply studied, and this is exactly the capability needed when facing task scenarios with ambiguous instructions (like the real world). Therefore, reflecting on our own experience doing research, we created a series of research scenarios, hoping to evaluate agents along this dimension. This is by short the original intent behind the AARR project.
We thank your feedback, especially criticism. We have done extensive additional experiments regarding to the reviews. We hope this will be helpful to resolve your concerns.
W1/Q1. Keyword checks and grader validation
Thank you for asking this. 68 of the 82 tasks contain a keyword check. We validated the graders in three ways. First, all 82 reference solutions pass the original verifier, which rules out broken or unsolvable tasks. Second, we read the runs whose only failing tests are lexical (75 of 1,637 runs, 4.6%), and checked whether each failure came from the research behavior or only from the wording. As for the D.2 refusal case, we did a thorough recheck. In that task, four configurations refused correctly and left the data unchanged, but failed only on wording. With file state as the primary check, the task moves from 8/20 to 12/20. We do not add an LLM judge. The reason is that AARRI-Bench is the first work in the AARR series and we keep the setup simple. A deterministic test suite is also easier to reproduce.
W2/Q2. Results over multiple trials per task
The release has 1,637 unique agent-task runs. We did not rerun tasks to improve scores. For stability, we run three trials on two configurations and report these results below.
W3/Q3. Harness ranking
Yes. During our actual experiments, the ranking of the harnesses remained very stable. Due to cost constraints, we did not run multiple repeated trials for all agents, but as can be seen from the table above, the error across repeated runs does not exceed two tasks. We hope these results may address your concerns.
W4/Q4. human baseline
Thank you for pointing out the missing of human baseline. We ran an experiment with five participants on a global simple random sample of 16 tasks (seed
AARRI-HumanEvalGlobal16-v1-20261113). Humans solve 11 of them averagely, matching the best agent on the same tasks, which also scores 11/16 under the same verifiers. Two of them also solvedadvice-recall-pro, which only 1/20 agent configurations passed. We are currently only able to test on a random subset, and we hope these results will address your concerns.W5/Q5. Vertical dimension
The S1–S4 labels are now available. Across all 20 configurations, strict 0/1 success is below.
| Scope | Tasks | Strict 0/1 | |---|---:|---:|---:| | S1-Adaptation | 26 | 66.2% | | S2-Integration | 23 | 49.1% | | S3-Innovation | 22 | 41.8% | | S4-Open-ended | 11 | 46.4% |
W6/Q6. Table 1 criteria and entries
✓ means the benchmark explicitly evaluates the dimension as part of its task protocol while ✗ means it does not. EXP-Bench reports fine-grained design, implementation, execution, and conclusion scores over 12,737 subtasks, so its fine-grained entry is ✓. It also has an explicit integrity monitor for fabricated or placeholder data, so its researcher-quality entry is ✓. PaperBench’s 8,316 are rubric items across 20 papers, but in the original paper, they state "In total, PaperBench contains 8,316 individually gradable tasks". We think this is more about how you define what a task is. We find your comment about about rubric items and tasks quite reasonable and we finally change that slot into 20. We rechecked other rows also and found everything went well. The modified part of the table is right below.
W7/Q7. Table 2 averaging and dropped tasks
Sure. You are right that the old Hermes rows were averaged differently from the other rows. This was a careless mistake and we're sorry for that. Fortunately, it did not affect our conclusions: Mini-SWE-Agent still works better than the other two. The table below recomputes all 16 rows. The per-category task counts are 34 context, 13 mindset, 21 interaction, and 14 hands-on. During evaluation, we didn't drop any task. Hopefully that will resolve your concerns.