- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates the scientific rigor behind data curation methods, and questions whether their gains are attributed to algorithm novelty or retraining noise. It conducts extensive experiments on 6 different data curation tasks, and reports results on their own reimplementation of these various methods. Overall, it finds that there is a lot of noise in performance depending on the seed that is used.
Critical strengths
The paper does highlight an important aspect of robotics research, which is the variance in seed to seed training. It does extensive training, and releases their code.
Critical weaknesses
Overall, my biggest concern is that I feel like this paper is an agglomeration of multi-seed experiments and reproductions of existing methods, without much novelty of it's own. While statistical rigor is indeed an important aspect of research, the contributions listed by the authors do not assist in actually advancing such rigor. The suite that they release is not practically usable, and there is no method that is clearly set for for the community, such as what was done for RL works in (Rliable)[https://araffin.github.io/post/rliable/].
Additionally, I have some concerns to the validity of the findings as a whole. Specifically, the authors mention starting ln387 that all evaluations happen on their implementations. It is an odd choice to not run on any published curation code, and that attributions of the findings to be up to reimplementation noise rather than the method family is a fair, yet unaddressed, critique of the paper. Also, ln467-471 very clearly state a defect in the results -- this seems highly unpolished.
As a side note, the manuscript, writing and figures feel heavily AI generated, without much attempt to improve legibility nor appearance. For example Figure 1 is very challenging to parse, with vague pairings between the symbols in ln 68-69 and the actual figure, and overlapping caption text in the figure itself. The key message is easily lost. I would highly encourage authors to re-do these plots, as well as the accompanying captions. Another example: the contributions list is long and scattered -- it reads more like a list of findings, instead of actual contributions. The full paper reads like a technical progression of experiments conducted.
Questions for authors
- What is the concrete takeaway the authors hope readers will take from their paper? How do authors envision their findings furthering the community as a whole? Please take into account that a lot of the findings were based on reimplementations, and thus there is likely confounding noise underlying all findings.
- What is the plan for authors to address the limitations and defect stated in the paper? (Such as ln 467-471?)