- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors augmented the vanilla-flavor evolution strategies (ES) algorithm for LLM posttraining with 1) difficulty-based reward weighting and 2) prioritized sample reply in hopes of more efficiently allocating the perturbation sampling budget compared to baseline ES to achieve better performance.
Critical strengths
- The method proposed is sound and the experimental comparison is straightforward and extensive, primarily following the structure of the original ES-at-scale paper from Qiu et. al.;
- I appreciate the development of the concept of the population pass rate which describes how "good" the neighborhood is for a given problem, which provides concrete measure that can be leveraged for reweighting;
- The concept of replay has been prominent in the RL literature, I appreciate how the authors was able to apply it to ES.
Critical weaknesses
- While the proposed tricks are sound and intuitively appealing, the experimental outcomes do not show that big of a gain compared to the baseline methods (ES, ESSAM, variants of GRPO). My suggestion for the authors is to instead of just showing the numerics, dig into analysis of case specific examples of how DART-ES actually helped significantly improved performance of individual problems compared to the prior methods. The paper would benefit from re-framing the contribution to how this method improves the low-pass-rate problems rather than improving the overall average pass-rate of all problems.
- The results on the instruction tuning benchmarks as well as the coding benchmark seems quite interesting. The authors should consider digging into why ES and DART-ES perform better in these benchmarks compared to the math ones. Countdown specifically is interesting, what about its structure might make it a better suited candidate for ES and especially DART-ES? Can you look into the model outputs pre- and post-optimization?
- Fig2 is a bit overblown in my opinion. It is also not entirely fair to not include ES and ESSAM, and maybe even EGGROLL in this figure. Also I don't believe the runtime/memory save is a special contribution of the proposed tricks in this current paper, so the authors should be honest about that.
- Overall, I find the contribution to be a bit incremental, especially if the current results are just mostly focusing on comparing numbers with prior methods. The paper has more potential if the authors instead focus on more qualitative analysis rather than just the quantitative comparison. I think interesting thing to plot could be a distribution of problem pass@k rate of base model compared to DART-ES optimized model for these benchmarks/datasets. There could be some interesting/surprising findings there.
Questions for authors
All my questions are already included in my weaknesses section