acceptodds
Withdrawn from ICLR 2027

DART-ES: Difficulty-Aware Reweighting and Targeted Replay for Fine-Tuning LLMs with Evolution Strategies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Evolution Strategies (ES) enable memory efficient full parameter fine-tuning of large language models (LLMs) using only forward computation. However, standard ES uniformly averages rewards across problems and compresses problem level population feedback into a single scalar, making it difficult to capture how the learning value of each problem changes with model capability. To address this limitation, we propose Difficulty-Aware Reweighting and Targeted Replay for Evolution Strategies (DART-ES). DART-ES estimates the local solvability of each problem from its pass rate across the perturbation population and aggregates historical observations to construct a dynamic difficulty state. This shared state jointly guides continuous difficulty reweighting and rare solvable sample replay, thereby improving perturbation direction evaluation and training data allocation without introducing an additional difficulty model or backpropagation. Extensive experiments show that DART-ES achieves good fine-tuning performance. DART-ES outperforms ES on all five base models and improves the average accuracy from 72.07% to 73.53%, exceeding the 73.26% achieved by GRPO on GSM8K. Across five challenging mathematical reasoning benchmarks, DART-ES achieves an average accuracy of 49.20%, compared with 48.34% for ES and remains competitive with strong 7B models trained with RL. Further experiments show consistent gains in instruction tuning, code generation and the Countdown task with a 14B model, demonstrating strong generalization across tasks and scalability to larger models. Beyond performance gains, DART-ES also shows clear advantages in system efficiency. It reduces runtime per step by 15.2%–50.2% and peak memory usage per GPU by 21.1%–51.1% compared with GRPO. Despite performing full parameter fine-tuning, DART-ES also requires less runtime and GPU memory than GRPO+LoRA. Overall, DART-ES achieves strong performance while maintaining generalization, scalability and system efficiency. Code: https://anonymous.4open.science/r/DART-ES/

Primary area:
foundation or frontier models, including LLMs
Keywords:
Evolution Strategies, Large Language Model Fine-Tuning, Difficulty-Aware Reweighting, Targeted Replay, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WvoGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors augmented the vanilla-flavor evolution strategies (ES) algorithm for LLM posttraining with 1) difficulty-based reward weighting and 2) prioritized sample reply in hopes of more efficiently allocating the perturbation sampling budget compared to baseline ES to achieve better performance.

Critical strengths
  1. The method proposed is sound and the experimental comparison is straightforward and extensive, primarily following the structure of the original ES-at-scale paper from Qiu et. al.;
  2. I appreciate the development of the concept of the population pass rate which describes how "good" the neighborhood is for a given problem, which provides concrete measure that can be leveraged for reweighting;
  3. The concept of replay has been prominent in the RL literature, I appreciate how the authors was able to apply it to ES.
Critical weaknesses
  1. While the proposed tricks are sound and intuitively appealing, the experimental outcomes do not show that big of a gain compared to the baseline methods (ES, ESSAM, variants of GRPO). My suggestion for the authors is to instead of just showing the numerics, dig into analysis of case specific examples of how DART-ES actually helped significantly improved performance of individual problems compared to the prior methods. The paper would benefit from re-framing the contribution to how this method improves the low-pass-rate problems rather than improving the overall average pass-rate of all problems.
  2. The results on the instruction tuning benchmarks as well as the coding benchmark seems quite interesting. The authors should consider digging into why ES and DART-ES perform better in these benchmarks compared to the math ones. Countdown specifically is interesting, what about its structure might make it a better suited candidate for ES and especially DART-ES? Can you look into the model outputs pre- and post-optimization?
  3. Fig2 is a bit overblown in my opinion. It is also not entirely fair to not include ES and ESSAM, and maybe even EGGROLL in this figure. Also I don't believe the runtime/memory save is a special contribution of the proposed tricks in this current paper, so the authors should be honest about that.
  4. Overall, I find the contribution to be a bit incremental, especially if the current results are just mostly focusing on comparing numbers with prior methods. The paper has more potential if the authors instead focus on more qualitative analysis rather than just the quantitative comparison. I think interesting thing to plot could be a distribution of problem pass@k rate of base model compared to DART-ES optimized model for these benchmarks/datasets. There could be some interesting/surprising findings there.
Questions for authors

All my questions are already included in my weaknesses section

Reviewer_bjHYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This review was entirely human-written, GPT-6 Astra was used after the fact to adversarially contradict my review. I manually corrected some of my statements that I judged to have been misrepresented based on this.

This paper introduces Difficulty-Aware Reweighting and Targeted Replay for Evolution Strategies (DART-ES), a method that combines difficulty-awareness with Evolution Strategies for fine-tuning LLMs. They use a scheme that downweights problems that are often solved and upweights difficult problems. They combine this with a specific problem sampling scheme that uses this difficulty as an input. This enables difficult problems to be sampled more often without being sampled too often either. Using this method they, on average, improve on GRPO, GRPO-LoRA, standard ES and ESSAM on GSM8K across 2 model families (LLaMa3.2 and Qwen2.5) and 2-3 model sizes. They additionally measure performance against other existing trained baselines on challenging Math benchmarks and outperform and on average are within 0.4% of the top performing baseline. Finally, they evaluate their method against ES in instruction tuning, code generation and countdown and show that they outperform it there too. They claim that DART-ES is cheaper in runtime and GPU-memory. Overall, this paper has nice results and is well written, but is lacking justification of choices, experimental detail and comparison to prior work and to meet the bar for acceptance.

Critical strengths
  • S1 the paper is clear and well written, (until the Experiments section, where the writing becomes much denser and tough to parse, but this is ok since the tables and plots are clear). The method is easy to understand.
  • S2 the breadth of evaluation is impressive, spanning multiple fine-tuning algorithms, model sizes, families and other fine-tuned models.
Critical weaknesses

W1 Experimental hygiene/reporting is poor, and the codebase is missing crucial details, leading me to doubt how reproducible the results are.

  • Hyperparameters are reported to be different for different experiment/dataset/model. The authors should be transparent about how tuning was done for these hyperparameters. No validation split is reported in the main text or appendix.
  • The choice of baselines is unjustified, switching between baselines for different datasets, this isn't justified in the main text, the appendix informs that the scores from some of the baselines were simply copied from other papers, but the main text or caption does not make this explicit. Specifically, we do not know how the training was done for those methods compared to DART-ES.
  • Noise and variance quantification: no error bars in any table. The gains on prior work are modest, and ES methods (and LLMs) are fundamentally stochastic. At least reporting what are the error bars across different training seeds for GSM8K would help in assessing how noisy the method is and how significant the gains are.
  • Figure 2 shows GRPO-LoRA training is longer than simple GRPO. LoRA is a method for efficiency, therefore should be faster in wall-clock time than GRPO. The authors need to clarify this and adjust their claims of gains over GRPO-LoRA.
  • The codebase is too limited to reproduce their results, the non-GSM8K task implementations, benchmark evaluators, RL/ESSAM baselines and efficiency measurement code are missing. The authors should push the missing code to the repository.

W2 Framing of the paper and framing within prior work.

  • The authors cite relevant prior work but (i) are not explicit about how their method is different (ii) do not compare their method in experiments to the cited prior work such as Foster et al. 2025, Dai et al. 2026.
  • The method is similar to Dai et al. 2026 (this is not addressed in the main text). Dai et al. 2026 develop Difficulty-Aware Group Policy Optimization (DGPO), the difficulty-awareness mechanism has many similarities to DART-ES. The main difference is the use of ES instead of GRPO. If the core mechanism is that ES perturbations have more diversity (for weighing the problem difficulty) than members of a group of GRPO then the authors should run that ablation. This would help in giving scientific insights about the method, which are currently very light.
  • No limitations section.
Questions for authors

See Weaknesses section.

Reviewer_eygJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DART-ES changes how evolution strategies (ES) score parameter perturbations and select training problems. For each problem, it measures the pass rate across the perturbation population and smooths this rate with an exponential moving average. Problems with lower historical pass rates receive higher weights when scoring perturbations. The same history guides replay of problems that are not yet mastered but have been solved by some perturbations. The method keeps the base ES perturbation and update procedure and requires no additional difficulty model or backpropagation. Experiments cover mathematical reasoning with several base models, code generation, instruction tuning, and Countdown with a 14B model.

Critical strengths
  1. The method uses feedback that ES already collects when evaluating its perturbation population. The added history and weighting rules are lightweight, and using the same history for scoring and replay makes the approach easy to incorporate into an ES training pipeline.

  2. DART-ES improves over standard ES on all five GSM8K base models, with an average gain of about 1.46 percentage points. Improvements are also reported for code generation, instruction tuning, and 14B Countdown. The ablations and analyses of update coefficients, replayed problems, and EMA prediction error give some insight into how the method works.

Critical weaknesses
  1. All five GSM8K models improve over standard ES, but the paper does not report variation across training seeds or confidence intervals. The average GSM8K gains over GRPO and ESSAM are about 0.27 and 0.25 percentage points, and EMA adds only 0.23–0.31 percentage points in the paired ablations. Results from several training seeds are needed to judge whether these small differences are reproducible.

  2. The ablations cover only GSM8K with a 0.5B model. Without random replay under the same total budget, it is unclear how much of the gain comes from the targeted rule and how much comes from repeating problems. Simple pass-rate filtering and alternative replay priorities would also be useful comparisons. The paper does not test sensitivity to EMA decay, replay ratio, or mastery threshold. Checking the main ablations on a larger model or harder task would help show whether the findings extend beyond this setting.

  3. Each epoch continues until all fresh problems have been visited, and replay takes up part of each batch. Matching epoch counts can therefore give DART more updates and generations. The paper says the ablations use the same training budget, but does not report the actual update counts, problem visits, and generated tokens for each setting. Training steps are explicitly matched for instruction tuning and code generation; the same information is needed for GSM8K, MATH, and the 14B experiments. The MATH baselines are taken from prior work, so it is also unclear whether their training and evaluation settings are comparable to DART's.

  4. The runtime and memory comparison includes GRPO and GRPO+LoRA but omits standard ES. It therefore does not show the overhead of adding weighting and replay to ES. Per-step timings are also difficult to compare when the amount of generation per step and the total number of steps differ. The authors should report runtime and memory against standard ES, along with performance at a fixed total compute budget or the total time and generation needed to reach the same score.

Questions for authors
  1. See Weaknesses.

  2. Are the reported scores from the final checkpoint or one selected on validation data? What data were used to tune the hyperparameters? For the MATH baselines taken from prior work, please specify the training budgets, evaluation decoding settings, and checkpoint selection rules, and provide a comparison using the same protocol.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.