- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduce measurement-regime diagnosis, a post-pilot audit that evaluates task difficulty as a proxy for ranking fidelity under fixed budgets and discrimination among frontier systems. This paper observes that the hardest problems in SWE Bench Verified are not useful to distinguish agentic systems. However, HFI problem are more useful than hardest problem.
Critical strengths
- This paper does some experiments
- The experimental results shows that hardest problem in one benchmark are not very useful
- The problem studied in this paper is important
Critical weaknesses
- The results shown in this paper are not very interesting
- This paper lacks novelty and this paper does not propose any novel solutions
- The problem studied in this paper is important, however, this paper does not propose any solutions to solve this paper. The problem is how to design tasks to distinguish two llm agentic systems
Questions for authors
I believe this paper falls well below the acceptance bar for ICLR 2027. One open question is how to automatically generate tasks that distinguish two agent systems