- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper points out that, self-improvement is not necessarily a recursive synthetic-data training pipeline that outperforms a certain training baseline, and then proposes five gates for identifying real self-improvement: untouched base, matched exposure, verifier validity, controller activation, decontamination. Then the authors apply it on DataEvolve and find out that, although DataEvolve claims to outperform a trained baseline by over 10 percent, it falls short on even outperforming a simple untouched base, and thus the self-improvement claim doesn't hold for this particular case. The authors also perform very fine-grained and careful audit on the results.
Critical strengths
(1) Timely and important topic. The AI community is suffering from various kinds of slop papers which essentially become a DDoS attack and a huge challenge to academic integrity. The paper establishes standards for identifying real claims from massive hypes in the popular subarea of RSI (recursive self-improvement) and showcases a real example of hype or faulty claims. It is meaningful to the entire AI community for restoring the order, integrity and fostering real scientific advances, and the establishment of such standards and application to existing works are timely, correct, important, and should be encouraged.
(2) Fine-grained analysis and protocol. The failure diagnosis is fine-grained and comprehensive.
Critical weaknesses
The main weakness is scope and novelty. I'm a fan of the work but I'm not sure whether or not it is appropriate as an ICLR paper. As a blogpost, I would be willing to offer my strongest support, but the amount of work and novelty remains questionable when it comes to determining whether the paper merits ICLR main conference publication. All the proposed gates, from my perspective, are common sense to real research practitioners and should be the most basic and fundamental rules that a researcher should adhere to, which has limited novelty, although I have to acknowledge that, unfortunately, it has not been the case for some works these years.
Questions for authors
I am willing to support the paper and increase my score to 4 if the authors (or other reviewers/AC) can convince me this paper is not out of scope and appropriate for main conference publication.
Q1: Could you offer some justification that the work is more suitable as a technical paper rather than a blogpost?
Q2: DataEvolve looks ambiguous in the manuscript without a proper citation or reference. Could you cite or give a pinpoint to that work (e.g. as a footnote in the paper)?
Dear Reviewer Gq5B, thank you for your detailed feedback. We appreciate your recognition of the importance of evaluating recursive self-improvement and the value of the failure analysis. We have clarified the technical contribution and scope of the case study and plan to incorporate these changes into the revised manuscript.
Thank you again for your review.