- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Authors show that "retire-and-replace" approach might not be the best way to treat saturated benchmarks. Model evaluations on those kind of benchmarks can still contain valuable insights such as model's efficiency, reliability, exploitation of shortcuts vs real task mastery. They argue that some task-level threats such as false positives or reward-hacking can only be revealed once the benchmark is saturated. Moreover, better benchmark performance might be attributed to better adaptation to this benchmark's specific scaffold, prompts, format etc. Authors also perform the study on the uplift in duration of agent-human collaboration in research paper reproduction tasks.
Critical strengths
Interesting view on benchmark saturation. Old benchmarks can still be useful to measure capabilities beyond just accuracy.
Critical weaknesses
- No error bars. Especially considering the small size of the benchmarks - 39 and 19 tasks. So for these tasks (tables 1 and 3) 10pp might be well within the std of the scores.
- The second part (the uplift) study is vaguely related to the first part (benchmark might have tasks with threats that might make saturated performance misleading and saturated benchmarks are still useful).
Questions for authors
- Why only when benchmark is saturated one can find tasks with threats? One can imagine a set of hard tasks that are not fully solved by an agent due to their nature and at the same time have flaws such as over reliance on scaffolding, incorrect verifier etc.
- How were the reproductions evaluated in human-agent reproduction uplift?
- Was the goal of the uplift study to understand whether agents improve paper reproduction performance of humans or that the agents can reproduce papers autonomously? How is it related to the benchmark saturation problem?
2 comments
Thank you for addressing my concerns, I will revise my score.
Thank you for providing feedback on the error bars and uplift study. We appreciate that you found our submission to be an interesting view on benchmark saturation. We have revised the paper in response to feedback with all text changes in blue.
1. Benchmark saturation motivation
Before accuracy saturation, identifying all potential threats to benchmark validity is difficult and often infeasible across all tasks. For example, in a paper with a large codebase, manually investigating hundreds of data files and scripts to identify potential shortcuts an agent could exploit may be impractical. Running highly capable agents on these tasks, however, can help surface such shortcuts by revealing the various avenues these agents were able to find to exploit them.
2. Human-agent reproduction uplift evaluation
The human-agent uplift study was evaluated in two main ways. First, we estimated uplift in duration using the fixed effects model in Section A.5.8. Second, we created a questionnaire, provided in Section A.5.6, to understand where agents were useful, what blockers human-agent collaborative runs encountered and resolved compared to human-only runs, where agents struggled, and where agents required human intervention. We provide qualitative observations in Section 4 and Section A.5.10. In addition, we conducted a post-hoc correctness audit of all 50 reproductions in Section A.5.11, which showed human-only runs had lower correctness rates, but the differences were not statistically significant.
3. Goal of the uplift study
The goal of the uplift study was to understand whether agents are useful for human computational reproducibility workflows. In Section 4, we state the following:
The uplift study is related to benchmark saturation because when agents are obtaining low accuracy, the question is still whether agents can solve the task at all. However, once accuracy reaches saturation, this is used as evidence to suggest that the agent is ready for deployment. However, high benchmark accuracy may not always translate to utility when deployed alongside humans. We give a few examples of why this could be the case in Section 4:
Thus, Section 4 asks whether an agent's near-ceiling benchmark performance (saturation) actually translates to real-world usefulness when deployed alongside humans in the same domain. Saturation therefore makes measuring uplift in task duration important to investigate whether benchmark success reflects real-world value when the task is collaborative between humans and agents.
4. Error bars
We agree that the analyses in Section 3 should have error bars. In Tables 1 and 3, we show the accuracy values with 95% Wilson CI bounds. We have updated the paper to add error bars to the consistency plots (Figure 1) based on the standard errors added in Table 25.
Once again, we appreciate your time and attention to detail. Our work has already improved thanks to your feedback. We hope that our rebuttal has addressed your concerns and remain available for further discussion throughout the rebuttal process.