- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
RSIBench-Data is a benchmark that tests whether LLM agents can do data-centric post-training research: find where a model is weak, design training data to fix it, and revise that data based on feedback. Everything except the data is fixed. The authors tested 4 agents on 6 benchmarks. The main finding is a "discovery–reliability gap”.
Critical strengths
-
The topic is important now; we need to judge the model's ability like RSI
-
The analysis is interesting in Section 4 and the appendix.
Critical weaknesses
-
The idea is quite similar to DatasetResearch and InnovatorBench. (Just different tasks)
-
There is a risk of reward hacking. (Search data online? Just modify the test set a little bit?)
-
The scope is small.
Questions for authors
- What did the agent really do? Sourcing data online or something else? Can the agent just download something online?
I'm really concerned about whether the score reflects ability or is just reward hacking. I may raise my score if my concern is solved.