- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DARE, a difficulty-adaptive reinforcement learning framework for improving the efficiency and effectiveness of LLM reasoning. The method combines policy-aligned difficulty estimation using self-normalized importance sampling over replayed rollouts, difficulty-guided prompt sampling that favors medium-difficulty examples while retaining easy and hard ones, and difficulty-specific optimization strategies that allocate different rollout budgets and reward shaping across difficulty levels. The authors first show that existing difficulty estimators become inaccurate under policy drift, while difficulty-based filtering alone provides limited final-accuracy gains and does not improve inference efficiency. Experiments across multiple model scales and reasoning benchmarks show that DARE improves training efficiency, final accuracy, and inference-time token usage.
Critical strengths
-
The paper is comprehensive and well executed, with solid experiments and generally promising empirical results.
-
The paper is clearly presented and logically organized. It first analyzes the limitations of existing difficulty-aware RL frameworks, and then motivates and introduces the proposed method accordingly, followed by a clear presentation of the experimental results.
Critical weaknesses
My main concern with this paper can be summarized with a simple analogy. Suppose the original method is a grilled steak. The proposed method replaces the beef with a better cut, changes the cooking procedure to slow cooking, and further adds an appetizer, soup, and dessert. The resulting meal may certainly be better than the original steak, but this raises two issues (W1-2) regarding what exactly has been contributed and what the comparison demonstrates.
-
In my view, the most clearly novel component is the application of SNIS to obtain a policy-aligned difficulty estimate, which I agree is a sensible idea. Beyond this, however, most components appear relatively incremental. The cold-start mechanism in Section 3.1 largely follows DOTS; the modification in Section 3.2 is fairly straightforward; and much of Section 3.3 concerns efficient reasoning and adaptive computation allocation. There has also been substantial recent work on efficient reasoning and adaptive computation since last year, while this line of work receives little discussion in the related-work section. Overall, I see DARE primarily as a combination of several reasonable incremental modifications rather than a substantially new methodological contribution, which limits my enthusiasm for the paper.
-
The paper combines all of these components into DARE and then compares the resulting full system against prior methods that are primarily designed around difficulty-aware data selection. In particular, the main results in Figure 4 and Table 2 include not only the proposed difficulty estimator and sampling strategy, but also the additional difficulty-adaptive optimization mechanisms in Section 3.3. I understand that better data selection may remain beneficial even if the competing selection methods were equipped with similar additional mechanisms, but the current comparison does not establish this. As a result, I do not consider Figure 4 and Table 2 to be fully fair apple-to-apple comparisons, and I believe they somewhat amplify the apparent advantage of the proposed method.
-
Additionally, the method introduces many additional hyperparameters (e.g,. SNIS clipping threshold, easy/hard difficulty thresholds, tier-specific rollout counts, easy/hard reward-shaping coefficients, asymmetric clipping ranges), which may limit its practicality. Although the paper provides some ablations, the overall framework still involves a fairly large number of tuning knobs. I view this partly as a consequence of the highly compositional nature of the method.
Questions for authors
-
Why is clipping necessary for the SNIS estimator? What happens if the importance weights are not clipped, or if clipping is applied only to extremely large values?
-
I find the results in Table 3 somewhat subtle. Level 1 sees only a 4% reduction in output tokens, while Levels 2–4 see reductions of around 10–11%. Given that the method explicitly encourages concise reasoning on easy problems, why is the reduction smallest for Level 1?
-
What is the practical takeaway of Proposition 3.1? I currently find its significance unclear.
3 comments
Thank you for the response. I appreciate the authors’ clarifications, but I am still not particularly convinced by the rebuttal.
For W1, while the authors responded to the individual components of my concern, I still find the overall methodological novelty limited. In fact, I do not think the rebuttal provides a particularly strong argument for what is fundamentally novel beyond the individual incremental modifications. I therefore maintain my original concern regarding novelty.
For W2, the evidence provided is still indirect. The response shows that SNIS itself has advantages, but it does not provide the controlled full-system comparison that I was asking for. This makes the current presentation somewhat awkward to me. One possibility would be to restructure the paper around difficulty estimation as the primary contribution: first compare and improve difficulty estimation, present the corresponding empirical results, and then introduce the additional components in a separate section. Alternatively, if the paper wishes to present DARE as a unified system and position this as the main contribution, I still think a matched system-level comparison is needed.
The response to W3 also does not substantially change my concern. The method still contains a large number of hyperparameters and design choices, which I believe limits its practicality.
I also did not find the responses to Q2–Q3 particularly informative. In particular, regarding Q3, I still do not find the result very useful. It does not provide a practically meaningful guarantee, or substantially support the empirical method.
Overall, without further evidence addressing these concerns, I plan to maintain my original score.
We sincerely thank the reviewer for the prompt and detailed reply, and appreciate your patient follow-up and explanations. The two options in W2 are concrete and very helpful, and we answer with both below.
W2. Estimator-only and matched system-level comparisons
We greatly thank the reviewer for spelling out exactly which comparison would settle this.
The response shows that SNIS itself has advantages, but it does not provide the controlled full-system comparison
We apologize: our previous response did not lay out the existing comparisons clearly, so they read as indirect. We now give both options the reviewer proposes.
Difficulty estimation as the primary contribution. This is already the order of the paper. Sec. 2 compares all estimators under the same GRPO pipeline: "More accurate difficulty estimation improves training-step efficiency," but selection alone "yields limited final accuracy gains" and "does not improve inference efficiency." Sec. 3.3 targets these two gaps using the estimate , and Sec. 4.2 reports the estimator first and the full system after it. We will further highlight this separation in the introduction and Sec. 4.
DARE as a unified system. Below, the full DARE pipeline on Qwen2.5-Math-1.5B, with only the estimator replaced and all Sec. 3.3 components unchanged.
| Estimator inside DARE | MATH-500 | GSM8K | AIME-AMC | Minerva | Olympiad | Avg. |
|---|---|---|---|---|---|---|
| SNIS (DARE) | 73.2 | 84.5 | 50.6 | 19.9 | 26.4 | 50.9 |
| Embedding (DOTS) | 72.2 | 83.3 | 49.4 | 18.4 | 25.5 | 49.8 |
| Previous FR | 71.0 | 81.0 | 48.2 | 17.3 | 24.8 | 48.4 |
SNIS is best on all five benchmarks, and the order follows the estimation error in Table 5 (0.1889 < 0.2775 < 0.2988). Previous FR reads the same replay buffer as SNIS but without the importance weights (App. F), so its row isolates the weights: dropping them costs 1.6–3.5 points on every benchmark. This measures the case the reviewer anticipated, that "better data selection may remain beneficial even if the competing selection methods were equipped with similar additional mechanisms." We will add this table to Sec. 4.
W1. Novelty
We thank the reviewer for weighing the contribution as a whole.
what is fundamentally novel beyond the individual incremental modifications
The reviewer named the core novelty in the first review: "the application of SNIS to obtain a policy-aligned difficulty estimate, which I agree is a sensible idea." The paper is built around it: contribution 2 is the estimator, and contribution 3 is the difficulty-adaptive training that uses it (Sec. 1).
Accepted work in this line also builds training strategies on the difficulty signal and reports the full system: DOTS with rollout replay (NeurIPS 2025), CurES with rollout reallocation (ICLR 2026), DEPO with online rollout pruning (ICLR 2026), ExGRPO with experience replay (ICLR 2026), Liao et al. with temperature scheduling (EMNLP 2025), and Zhang et al. with expert-guided reformulation (EMNLP 2025).
W3. Hyperparameters
We thank the reviewer for the focus on practicality.
The method still contains a large number of hyperparameters
DARE tunes five values, all swept in the paper: (Table 7), (Table 8), and the easy-prompt clip bounds (Table 9). The rest are fixed: is set "for numerical stability" (Sec. 3.1), sampling and buffer settings follow the baselines (App. F), and the tier bounds define the tiers. Accepted methods tune as many or more:
- DEPO (ICLR 2026), eight: pruning ratio, mean and width of the difficulty prior, window size, initial sampling rate, decay rate, replay ratio, entropy threshold.
- GRESO (NeurIPS 2025), six: two target zero-variance ratios, probability step, initial and minimum probability, batch-size safety factor.
- ExGRPO (ICLR 2026), five: replay ratio, mean and width of the bucket weight, shaping constant, delayed-start threshold.
Q3. Role of Proposition 3.1
We thank the reviewer for pressing on this point.
It does not provide a practically meaningful guarantee, or substantially support the empirical method.
Prop. 3.1 is the reason for the ESS test, not a stand-alone guarantee. The bound loosens when the importance weights are uneven, which is what ESS measures, so Sec. 3.1 states: "The worst-case concentration term can be loose for large c, so we use ESS (see App. C) in practice." Table 7 checks the test: at every threshold from 1 to 5, the estimator beats the best prior MAE of 0.2775. We will state this link directly in Sec. 3.1.
Hope the matched comparison above addresses the main concern, we realize that your time is valuable, but should you have any further questions, please do let us know; we will provide as detailed a response as possible.
We thank the reviewer for the thorough review. All new results below will be added to the revised manuscript.
Weaknesses
W1. Contribution beyond SNIS
We greatly thank the reviewer for the vivid framing.
Every dish is cooked from the new cut. Sec. 3.2 samples by d^q, and "Rollout allocation precedes generation and therefore uses the estimate d^q of Sec. 3.1" (Sec. 3.3), so extra rollouts and hints help only if that estimate is right. Each component answers one finding of Sec. 2: SNIS fixes drifting estimates, the hard tier targets the accuracy ceiling of selection alone, and the shaping targets token usage that selection leaves unchanged.
Only the similarity-based predictor follows DOTS, for "prompts without buffered rollouts" (Sec. 3.1). Its rollout covers 256 prompts and runs only "at the start of training" (App. C). From the second epoch, to further cut cost, we skip this rollout and use the 256 latest prompts in the replay buffer as the reference set (code:
ray_trainer.py,fit,epoch > 0branch); DOTS rolls out a new reference set every step (App. E). SNIS overrides the predictor wherever the buffer has enough history (Table 7).It is simple on purpose, because it pairs with a more accurate estimator. DOTS samples ∝exp(−∣d^q−0.5∣/τ) with τ=10−3 (DOTS, Table 6), so a prompt at 0.49 gets only 4.5×10−5 of the peak weight. That sharpness cannot help when the estimate is off: existing estimators "retain many samples whose ground-truth difficulty is far from 0.5" (Sec. 2). SNIS tracks the current failure rate, so we use a wider Beta(51,51). It keeps 98% of the peak weight at 0.49, still peaks at 0.5, and "assigns nonzero probability across the full difficulty range" (Sec. 3.2).
Sec. 3.3 is not uniform compression: it shortens Levels 1–4 by 4–11% and spends 7% more on Level 5 to gain 8.21 points (Table 3). We will add this related work to App. A.
W2. Full-system comparison
We greatly thank the reviewer for this careful reading.
Our core claim is more accurate difficulty estimation, and the paper tests it with SNIS only: MAE 0.1889 versus 0.2775 for the best prior estimator (Table 5), and higher accuracy than Embedding and LLM-Judge at every rollout budget (App. K). On code, App. I: "the estimator-level advantage transfers from mathematical reasoning to code generation." On 7B, GRPO + SNIS alone gives 3.4 of DARE's 5.8-point MATH-500 gain (Table 4).
All baselines run on the same code, data, and models. Sec. 4.1: "each difficulty-aware baseline follows the training pipeline of its source paper." Each is its own branch of
RayPPOTrainer.fit(e.g., DOTS'sTeacherModelWorker).W3. Hyperparameter count
We greatly thank the reviewer for raising practicality.
Each is a standard RL knob: weight truncation (V-trace), clip ratios (PPO), decoupled clips (DAPO). The thresholds mirror the target band of selection methods (App. F), and the tier counts are G/2, G, 2G.
One default setting serves all main results (Sec. 4.1) and is not fragile: every ESS threshold from 1 to 5 beats the best prior estimator's MAE (Table 7), and every relaxed upper clip matches or beats the symmetric one on all six benchmarks (Table 9). Baselines have comparable knobs, e.g., four for MoPPS (App. F). We will list all defaults in a table.
Questions
Q1. Why clip SNIS weights?
We greatly thank the reviewer for this insightful question.
A weight sums log-ratios over a whole response (Eq. 4), so it can overflow or vanish; "clipping is applied for numerical stability" (Sec. 3.1). Prop. 3.1 needs both clips: without the upper one there is no finite Cw, and without the lower one bmin loses its floor e−c. The upper clip already acts only on extreme values: e4≈55 is hit by 0.2–0.6% of weights in our 1.5B runs.
Q2. Level 1 token reduction
We greatly thank the reviewer for the close reading of Table 3.
Level 1 has the least to remove: it starts shortest (357 tokens). The easy tier follows the policy's failure rate, not MATH levels (Sec. 3.3), and Levels 2–4 start longer.
Q3. Takeaway of Prop. 3.1
We greatly thank the reviewer for this question.
Prop. 3.1 shows the trade-off in c: larger c loosens concentration, smaller c widens the clipping gap. Sec. 3.1: "The worst-case concentration term can be loose for large c, so we use ESS (see App. C) in practice." This gives MAE 0.1889 at 88.7% coverage (Table 7), and bmin stays close to 1 (Q1).