- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes SAR-Arena, a benchmark consisting of 10 systems and architecture optimization tasks. Each task can be evaluated in three different modes: full cycle research, guided research, and engineering optimization. The difference between the three modes is the amount of expert information that is given for each subtask that the agent is responsible for (“bottleneck analysis”, “method design”, and “engineering iteration”).
Critical strengths
- Tasks are timely and important for the current generation of LLMs to improve upon, benchmarked models and harnesses are relevant and practical.
Critical weaknesses
- The writing is very unclear, not concise, and hard to follow. It is riddled with odd phrasing (e.g. "stage-end") and it is not clear why the tasks are structured and scored the way that they are (e.g. in line 252, why are those specific weights chosen?).
Questions for authors
- What exactly distinguishes this benchmark from existing ones and what exactly are the findings and takeaways from the failure analysis?