acceptodds
Under review as a conference paper at ICLR 2027

DARE: Difficulty-Adaptive Reinforcement Learning with Co-Evolved Difficulty Estimation

Abstract

Reinforcement learning improves the reasoning ability of large language models but remains costly and sample-inefficient, as many rollouts provide weak learning signals. Difficulty-aware data selection methods attempt to address this by prioritizing moderately difficult prompts, yet our analysis reveals three limitations: difficulty estimates become inaccurate under policy drift, data selection alone yields limited final-performance gains, and inference efficiency remains largely unchanged. These findings suggest that efficient and effective RL requires more than filtering by difficulty: the policy should learn to solve hard tasks while producing concise responses for easy ones. To this end, we propose **DARE**, a unified framework that co-evolves difficulty estimation with the policy via self-normalized importance sampling, maintains diverse difficulty coverage through a symmetric Beta sampling distribution, and applies tailored training strategies across difficulty tiers with adaptive compute allocation. Extensive experiments across multiple models and domains demonstrate that **DARE** consistently outperforms existing methods in training efficiency, final effectiveness, and inference efficiency, producing more concise responses on easy tasks while improving correctness on hard ones.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Large Language Models, Difficulty-Adaptive Training, Efficient Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_42UZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DARE, a difficulty-adaptive reinforcement learning framework for improving the efficiency and effectiveness of LLM reasoning. The method combines policy-aligned difficulty estimation using self-normalized importance sampling over replayed rollouts, difficulty-guided prompt sampling that favors medium-difficulty examples while retaining easy and hard ones, and difficulty-specific optimization strategies that allocate different rollout budgets and reward shaping across difficulty levels. The authors first show that existing difficulty estimators become inaccurate under policy drift, while difficulty-based filtering alone provides limited final-accuracy gains and does not improve inference efficiency. Experiments across multiple model scales and reasoning benchmarks show that DARE improves training efficiency, final accuracy, and inference-time token usage.

Critical strengths
  1. The paper is comprehensive and well executed, with solid experiments and generally promising empirical results.

  2. The paper is clearly presented and logically organized. It first analyzes the limitations of existing difficulty-aware RL frameworks, and then motivates and introduces the proposed method accordingly, followed by a clear presentation of the experimental results.

Critical weaknesses

My main concern with this paper can be summarized with a simple analogy. Suppose the original method is a grilled steak. The proposed method replaces the beef with a better cut, changes the cooking procedure to slow cooking, and further adds an appetizer, soup, and dessert. The resulting meal may certainly be better than the original steak, but this raises two issues (W1-2) regarding what exactly has been contributed and what the comparison demonstrates.

  1. In my view, the most clearly novel component is the application of SNIS to obtain a policy-aligned difficulty estimate, which I agree is a sensible idea. Beyond this, however, most components appear relatively incremental. The cold-start mechanism in Section 3.1 largely follows DOTS; the modification in Section 3.2 is fairly straightforward; and much of Section 3.3 concerns efficient reasoning and adaptive computation allocation. There has also been substantial recent work on efficient reasoning and adaptive computation since last year, while this line of work receives little discussion in the related-work section. Overall, I see DARE primarily as a combination of several reasonable incremental modifications rather than a substantially new methodological contribution, which limits my enthusiasm for the paper.

  2. The paper combines all of these components into DARE and then compares the resulting full system against prior methods that are primarily designed around difficulty-aware data selection. In particular, the main results in Figure 4 and Table 2 include not only the proposed difficulty estimator and sampling strategy, but also the additional difficulty-adaptive optimization mechanisms in Section 3.3. I understand that better data selection may remain beneficial even if the competing selection methods were equipped with similar additional mechanisms, but the current comparison does not establish this. As a result, I do not consider Figure 4 and Table 2 to be fully fair apple-to-apple comparisons, and I believe they somewhat amplify the apparent advantage of the proposed method.

  3. Additionally, the method introduces many additional hyperparameters (e.g,. SNIS clipping threshold, easy/hard difficulty thresholds, tier-specific rollout counts, easy/hard reward-shaping coefficients, asymmetric clipping ranges), which may limit its practicality. Although the paper provides some ablations, the overall framework still involves a fairly large number of tuning knobs. I view this partly as a consequence of the highly compositional nature of the method.

Questions for authors
  1. Why is clipping necessary for the SNIS estimator? What happens if the importance weights are not clipped, or if clipping is applied only to extremely large values?

  2. I find the results in Table 3 somewhat subtle. Level 1 sees only a 4% reduction in output tokens, while Levels 2–4 see reductions of around 10–11%. Given that the method explicitly encourages concise reasoning on easy problems, why is the reduction smallest for Level 1?

  3. What is the practical takeaway of Proposition 3.1? I currently find its significance unclear.

3 comments

We thank the reviewer for the thorough review. All new results below will be added to the revised manuscript.

Weaknesses

W1. Contribution beyond SNIS

We greatly thank the reviewer for the vivid framing.

Suppose the original method is a grilled steak ...

Every dish is cooked from the new cut. Sec. 3.2 samples by d^q\hat d_q, and "Rollout allocation precedes generation and therefore uses the estimate d^q\hat d_q of Sec. 3.1" (Sec. 3.3), so extra rollouts and hints help only if that estimate is right. Each component answers one finding of Sec. 2: SNIS fixes drifting estimates, the hard tier targets the accuracy ceiling of selection alone, and the shaping targets token usage that selection leaves unchanged.

The cold-start mechanism in Section 3.1 largely follows DOTS

Only the similarity-based predictor follows DOTS, for "prompts without buffered rollouts" (Sec. 3.1). Its rollout covers 256 prompts and runs only "at the start of training" (App. C). From the second epoch, to further cut cost, we skip this rollout and use the 256 latest prompts in the replay buffer as the reference set (code: ray_trainer.py, fit, epoch > 0 branch); DOTS rolls out a new reference set every step (App. E). SNIS overrides the predictor wherever the buffer has enough history (Table 7).

the modification in Section 3.2 is fairly straightforward

It is simple on purpose, because it pairs with a more accurate estimator. DOTS samples ∝exp⁡(−∣d^q−0.5∣/τ)\propto\exp(-|\hat d_q-0.5|/\tau) with τ=10−3\tau=10^{-3} (DOTS, Table 6), so a prompt at 0.49 gets only 4.5×10−54.5\times10^{-5} of the peak weight. That sharpness cannot help when the estimate is off: existing estimators "retain many samples whose ground-truth difficulty is far from 0.5" (Sec. 2). SNIS tracks the current failure rate, so we use a wider Beta(51,51). It keeps 98% of the peak weight at 0.49, still peaks at 0.5, and "assigns nonzero probability across the full difficulty range" (Sec. 3.2).

much of Section 3.3 concerns efficient reasoning ... little discussion

Sec. 3.3 is not uniform compression: it shortens Levels 1–4 by 4–11% and spends 7% more on Level 5 to gain 8.21 points (Table 3). We will add this related work to App. A.

W2. Full-system comparison

We greatly thank the reviewer for this careful reading.

better data selection may remain beneficial ... the comparison does not establish this

Our core claim is more accurate difficulty estimation, and the paper tests it with SNIS only: MAE 0.1889 versus 0.2775 for the best prior estimator (Table 5), and higher accuracy than Embedding and LLM-Judge at every rollout budget (App. K). On code, App. I: "the estimator-level advantage transfers from mathematical reasoning to code generation." On 7B, GRPO + SNIS alone gives 3.4 of DARE's 5.8-point MATH-500 gain (Table 4).

not ... fully fair apple-to-apple comparisons

All baselines run on the same code, data, and models. Sec. 4.1: "each difficulty-aware baseline follows the training pipeline of its source paper." Each is its own branch of RayPPOTrainer.fit (e.g., DOTS's TeacherModelWorker).

W3. Hyperparameter count

We greatly thank the reviewer for raising practicality.

many additional hyperparameters

Each is a standard RL knob: weight truncation (V-trace), clip ratios (PPO), decoupled clips (DAPO). The thresholds mirror the target band of selection methods (App. F), and the tier counts are G/2G/2, GG, 2G2G.

a fairly large number of tuning knobs

One default setting serves all main results (Sec. 4.1) and is not fragile: every ESS threshold from 1 to 5 beats the best prior estimator's MAE (Table 7), and every relaxed upper clip matches or beats the symmetric one on all six benchmarks (Table 9). Baselines have comparable knobs, e.g., four for MoPPS (App. F). We will list all defaults in a table.

Questions

Q1. Why clip SNIS weights?

We greatly thank the reviewer for this insightful question.

A weight sums log-ratios over a whole response (Eq. 4), so it can overflow or vanish; "clipping is applied for numerical stability" (Sec. 3.1). Prop. 3.1 needs both clips: without the upper one there is no finite CwC_w, and without the lower one bmin⁡b_{\min} loses its floor e−ce^{-c}. The upper clip already acts only on extreme values: e4≈55e^{4}\approx55 is hit by 0.2–0.6% of weights in our 1.5B runs.

Q2. Level 1 token reduction

We greatly thank the reviewer for the close reading of Table 3.

Level 1 has the least to remove: it starts shortest (357 tokens). The easy tier follows the policy's failure rate, not MATH levels (Sec. 3.3), and Levels 2–4 start longer.

Q3. Takeaway of Prop. 3.1

We greatly thank the reviewer for this question.

Prop. 3.1 shows the trade-off in cc: larger cc loosens concentration, smaller cc widens the clipping gap. Sec. 3.1: "The worst-case concentration term can be loose for large c, so we use ESS (see App. C) in practice." This gives MAE 0.1889 at 88.7% coverage (Table 7), and bmin⁡b_{\min} stays close to 1 (Q1).

Thank you for the response. I appreciate the authors’ clarifications, but I am still not particularly convinced by the rebuttal.

For W1, while the authors responded to the individual components of my concern, I still find the overall methodological novelty limited. In fact, I do not think the rebuttal provides a particularly strong argument for what is fundamentally novel beyond the individual incremental modifications. I therefore maintain my original concern regarding novelty.

For W2, the evidence provided is still indirect. The response shows that SNIS itself has advantages, but it does not provide the controlled full-system comparison that I was asking for. This makes the current presentation somewhat awkward to me. One possibility would be to restructure the paper around difficulty estimation as the primary contribution: first compare and improve difficulty estimation, present the corresponding empirical results, and then introduce the additional components in a separate section. Alternatively, if the paper wishes to present DARE as a unified system and position this as the main contribution, I still think a matched system-level comparison is needed.

The response to W3 also does not substantially change my concern. The method still contains a large number of hyperparameters and design choices, which I believe limits its practicality.

I also did not find the responses to Q2–Q3 particularly informative. In particular, regarding Q3, I still do not find the result very useful. It does not provide a practically meaningful guarantee, or substantially support the empirical method.

Overall, without further evidence addressing these concerns, I plan to maintain my original score.

We sincerely thank the reviewer for the prompt and detailed reply, and appreciate your patient follow-up and explanations. The two options in W2 are concrete and very helpful, and we answer with both below.

W2. Estimator-only and matched system-level comparisons

We greatly thank the reviewer for spelling out exactly which comparison would settle this.

The response shows that SNIS itself has advantages, but it does not provide the controlled full-system comparison

We apologize: our previous response did not lay out the existing comparisons clearly, so they read as indirect. We now give both options the reviewer proposes.

Difficulty estimation as the primary contribution. This is already the order of the paper. Sec. 2 compares all estimators under the same GRPO pipeline: "More accurate difficulty estimation improves training-step efficiency," but selection alone "yields limited final accuracy gains" and "does not improve inference efficiency." Sec. 3.3 targets these two gaps using the estimate d^q\hat d_q, and Sec. 4.2 reports the estimator first and the full system after it. We will further highlight this separation in the introduction and Sec. 4.

DARE as a unified system. Below, the full DARE pipeline on Qwen2.5-Math-1.5B, with only the estimator replaced and all Sec. 3.3 components unchanged.

Estimator inside DAREMATH-500GSM8KAIME-AMCMinervaOlympiadAvg.
SNIS (DARE)73.284.550.619.926.450.9
Embedding (DOTS)72.283.349.418.425.549.8
Previous FR71.081.048.217.324.848.4

SNIS is best on all five benchmarks, and the order follows the estimation error in Table 5 (0.1889 < 0.2775 < 0.2988). Previous FR reads the same replay buffer as SNIS but without the importance weights (App. F), so its row isolates the weights: dropping them costs 1.6–3.5 points on every benchmark. This measures the case the reviewer anticipated, that "better data selection may remain beneficial even if the competing selection methods were equipped with similar additional mechanisms." We will add this table to Sec. 4.

W1. Novelty

We thank the reviewer for weighing the contribution as a whole.

what is fundamentally novel beyond the individual incremental modifications

The reviewer named the core novelty in the first review: "the application of SNIS to obtain a policy-aligned difficulty estimate, which I agree is a sensible idea." The paper is built around it: contribution 2 is the estimator, and contribution 3 is the difficulty-adaptive training that uses it (Sec. 1).

Accepted work in this line also builds training strategies on the difficulty signal and reports the full system: DOTS with rollout replay (NeurIPS 2025), CurES with rollout reallocation (ICLR 2026), DEPO with online rollout pruning (ICLR 2026), ExGRPO with experience replay (ICLR 2026), Liao et al. with temperature scheduling (EMNLP 2025), and Zhang et al. with expert-guided reformulation (EMNLP 2025).

W3. Hyperparameters

We thank the reviewer for the focus on practicality.

The method still contains a large number of hyperparameters

DARE tunes five values, all swept in the paper: τESS\tau_{\mathrm{ESS}} (Table 7), λeasy,λhard\lambda_{\mathrm{easy}}, \lambda_{\mathrm{hard}} (Table 8), and the easy-prompt clip bounds ϵq−,ϵq+\epsilon_q^{-}, \epsilon_q^{+} (Table 9). The rest are fixed: cc is set "for numerical stability" (Sec. 3.1), sampling and buffer settings follow the baselines (App. F), and the tier bounds define the tiers. Accepted methods tune as many or more:

  • DEPO (ICLR 2026), eight: pruning ratio, mean and width of the difficulty prior, window size, initial sampling rate, decay rate, replay ratio, entropy threshold.
  • GRESO (NeurIPS 2025), six: two target zero-variance ratios, probability step, initial and minimum probability, batch-size safety factor.
  • ExGRPO (ICLR 2026), five: replay ratio, mean and width of the bucket weight, shaping constant, delayed-start threshold.

Q3. Role of Proposition 3.1

We thank the reviewer for pressing on this point.

It does not provide a practically meaningful guarantee, or substantially support the empirical method.

Prop. 3.1 is the reason for the ESS test, not a stand-alone guarantee. The bound loosens when the importance weights are uneven, which is what ESS measures, so Sec. 3.1 states: "The worst-case concentration term can be loose for large c, so we use ESS (see App. C) in practice." Table 7 checks the test: at every threshold from 1 to 5, the estimator beats the best prior MAE of 0.2775. We will state this link directly in Sec. 3.1.

Hope the matched comparison above addresses the main concern, we realize that your time is valuable, but should you have any further questions, please do let us know; we will provide as detailed a response as possible.

Reviewer_UgTgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed an RL algorithm called DARE. Its motivation comes from the observation of inaccurate difficulty estimation when policy drift occurs, limitations on convergence speed, and accuracy issues when only doing data selection. The method includes three parts: co-evolved difficulty estimation, symmetric beta sampling, and adaptive difficulty policy optimization. Their experiment results show that DARE improves in multiple aspects like difficulty estimation accuracy, final performance, training efficiency, inference efficiency, and can generalize to coding tasks like humaneval and MBPP.

Critical strengths

This paper addresses a major limitation in RL training: how to balance the difficulty of data points in a training batch. The method is meaningful compared with other difficulty estimations, such as LLM judges or embedding-based methods. It works on math and easy coding tasks.

Critical weaknesses

The Beta sampling distribution is very sharp, so the claim that easy and hard prompts are “retained” may not hold in practice. With κ=100κ=100, the sampler is Beta(51,51){Beta}(51, 51), whose density is proportional to (x(1−x))50(x(1−x))^{50}. Relative to d=0.5d=0.5, a prompt at d=0.3d=0.3 gets about 1.6e−41.6e-4 of the weight, and a prompt at d=0.2d=0.2 or 0.80.8 gets about 2e−102e-10. If there are enough prompts near 0.50.5, prompts with d^<0.3\hat{d} < 0.3 or d^>0.8\hat{d} > 0.8 will almost never be sampled. Then the pre-rollout branches (fewer rollouts for easy prompts, more rollouts and hints for hard prompts) are rarely used. The reward shaping is probably triggered mostly by sampling noise in the realized failure rate. For a prompt with true difficulty 0.50.5 and G=8G=8, the chance of less or equals than 22 failures is about 1414%, and the chance of greater or equals than 77 failures is about 3.53.5%. The authors should report how often each tier is actually triggered during training. Without this, it is hard to know which component really contributes to the gains in Table 4 and Table 10.

The length shaping may not work as a small perturbation. If advantages are normalized by the group standard deviation as in standard GRPO, then in an all-correct easy group the shaped rewards differ only by about 1e-5, but after normalization the advantages become O(1) and depend only on length. For example, four correct responses with 200/300/400/500200/300/400/500 tokens get advantages of about +1.34/+0.45/−0.45/−1.34+1.34/+0.45/−0.45/−1.34. So λλ does not control the strength of the signal in these groups. In an all-wrong hard group, the same thing happens, and the model is trained to prefer longer wrong answers. Table 8 seems consistent with this.

The hint mechanism is not clearly described. Section 3.3 says the hint is a reasoning skeleton from a successful trajectory. Appendix D also has a “think longer” hint appended to all hard prompts. Appendix H.2 says correct responses from hint-augmented rollouts replace incorrect ones. It is not clear whether these are used together or are different versions. Also, hint-augmented rollouts are sampled under a different prompt, but they are put in the same group as the original prompt and used to compute advantages. Their probability under the original prompt is not the probability they were sampled with, so this is an off-policy, distillation-like signal. The paper does not discuss this bias, or whether these rollouts are also stored in the buffer and used for SNIS difficulty estimation.

Questions for authors

N/A

3 comments

We thank the reviewer for the thorough review. All new results below will be added to the revised manuscript.

Weaknesses

W1. Beta sampling and observed tier frequencies

We thank the reviewer for the careful probability analysis.

The authors should report how often each tier is actually triggered during training.

We will add these frequencies to the revised manuscript. In our 1.5B logs, both tiers are active in every batch: 16–23% of prompts per step are easy and about 10% hard, most hard ones with a memory-guided hint. Eq. 6 is built for this: it "assigns nonzero probability across the full difficulty range, so no prompt is permanently excluded from training" (Sec. 3.2).

The reward shaping is probably triggered mostly by sampling noise in the realized failure rate.

Any finite rollout is noisy, but a realized outcome is not noise for the update that uses it. If 7 of 8 rollouts fail, the policy did fail on that prompt in that step, and the group has almost no correct answer to learn from. This is the case the hard tier is built for. So the shaping uses the actual failure rate, not the estimate: an estimate can be wrong, but the rewards are the same signal the advantage is computed from. The weight wqeasyw_q^{\mathrm{easy}} also scales the penalty: at 2 failures out of 8 it is only 1/61/6 (Sec. 3.3).

it is hard to know which component really contributes to the gains in Table 4 and Table 10.

Both tables add one component at a time on top of GRPO + SNIS. Both pre-rollout branches help on 1.5B: tiered allocation raises MATH-500 from 69.8 to 71.6, and the memory-guided hint raises Minerva from 14.3 to 16.2 (Table 10).

W2. Length shaping and group-standard-deviation normalization

We thank the reviewer for highlighting the interaction between shaping and normalization.

in an all-correct easy group ... the advantages become O(1) and depend only on length

The example needs a homogeneous group, and such groups are excluded before optimization in our runs (we will state this in App. F). Sec. 3.3 notes they carry no signal: "Hard prompts often produce rollout groups in which every response is incorrect, collapsing the group-relative advantage." Every retained group has both correct and incorrect answers, so its reward standard deviation is at least 0.25 (group size at most 16). The easy penalty acts on easy groups with at least one failure. There, shaping changes a reward by at most 10−410^{-4}, about 10−310^{-3} in normalized advantage, and every correct answer still beats every incorrect one. Length only ranks answers with the same outcome; it cannot become an O(1)O(1) signal.

λ\lambda does not control the strength of the signal ... the model is trained to prefer longer wrong answers

Table 8 shows the coefficients do control length, and longer answers come with accuracy. Doubling λhard\lambda_{\mathrm{hard}} lengthens answers on all five benchmarks (AIME-AMC: 956$\to$1842 tokens, 47.0$\to$48.2%); doubling λeasy\lambda_{\mathrm{easy}} shortens them on all five (GSM8K: 351$\to$248 tokens, 82.5$\to$89.0%).

W3. Hint mechanism and off-policy bias

We greatly thank the reviewer for the careful reading of the hint design.

It is not clear whether these are used together or are different versions.

They are two separate steps at different stages (Figure 1), not two hints in one prompt.

  • Before rollout: memory hint. Triggered by the predicted difficulty: "GG standard rollouts alongside Gq−GG_q-G hint-augmented ones, where the hint is a successful historical trajectory retrieved from the replay buffer" (Sec. 3.3).
  • After rollout: think-longer re-rollout. If a hard group still has very low accuracy, DARE appends a fixed "think longer" instruction (App. D) and draws extra rollouts, "replacing incorrect responses with correct ones" (App. H.2), so the group keeps a nonzero advantage.

They are separate branches in the code (inspiration.py L22–38, L391–403; ray_trainer.py L1964–1977, L2143–2161). The paper describes the first in Sec. 3.3 and Alg. 1, and the second in App. D and H.2. The revision will link both to Figure 1 in Sec. 3.3.

Their probability under the original prompt is not the probability they were sampled with … whether these rollouts are also stored in the buffer and used for SNIS.

Hinted trajectories are never scored under the unhinted prompt. Both the behavior and the current-policy log-probabilities are computed on the prompt actually used, hint included (vllm_rollout.py L263–273; dp_actor.py L311–359). Sharing the group advantage is intended: on hard prompts, these trajectories supply the correct answers that standard rollouts rarely find. They are stored in the buffer (ray_trainer.py L2263–2294), and SNIS weights them under the same prompt (is_data_selector.py L263–336). We will add this to Sec. 3.3.

We hope our response has resolved the questions you raised. We understand your time is limited, and any further feedback or suggestions would be greatly appreciated.

AuthorsOpenReview ↗

Thank You for Your Valuable Comment :) !

As we approach the end of the response period, I would like to sincerely thank you for your valuable suggestions, which have helped us further improve the clarity of our paper. We genuinely hope that our responses have adequately addressed your concerns. If time permits and you have any further questions, we would be more than happy to respond promptly. Regardless, we wish you a wonderful day!

Reviewer_tQA7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper opens as a critique of difficulty-aware prompt selection for RLVR. Sec. 2 argues three things on a 1.5B math model: existing difficulty estimators drift away from the current policy during training, six selection strategies end up within about two points of each other, and none of them changes response length. DARE's answer has three parts. Difficulty is re-estimated for the current policy by reweighting buffered rollouts with truncated importance weights, with an effective-sample-size test and an embedding fallback when the buffer is uninformative. Prompts are then drawn from a symmetric Beta density over estimated difficulty. Easy, medium and hard prompts are finally trained differently: 4, 8 and 16 rollouts, a length penalty on correct easy rollouts, hints and length-rewarding on hard ones, and a looser upper clipping bound on easy ones. Experiments cover four models including a code setting, and the claims are faster progress per training step, 1.7 to 3.6 points of final accuracy over the strongest baseline at 1.5B, and shorter answers on easy MATH levels together with better accuracy on the hardest one.

Critical strengths
  1. The estimator claim is measured against a current-policy oracle on held-out prompts instead of asserted, which is the experiment this line of work usually skips.

  2. Table 12 also reports the cases DARE loses, and Sec. 4.2 repeats the GSM8K exception in the main text rather than burying it.

Critical weaknesses
  1. Neither mechanism behaves as described at the paper's own settings. Proposition 3.1 needs its concentration term below b_min; at c = 4 and the buffer size implied by Table 7 it is roughly 28.6 against a b_min of order 1, and neither quantity is ever measured. And kappa = 100 means Beta(51,51), whose density relative to difficulty 0.5 is 1.6e-4 at 0.3 and 2e-10 at 0.8 - so the easy and hard tiers of Sec. 3.3, the paper's own answer to its Sec. 2 critique, are nearly unreachable. No kappa ablation, no tier-occupancy numbers.

  2. Nothing is budget-matched, which matters when the claims are about efficiency. Baselines follow their source pipelines with on-policy fractions of 0.5 or 1.0, so a step is not the same generation across rows, and rollouts and tokens per step are never reported - 8.71x is a step ratio. The token reduction on easy problems is also the direct target of Eq. 9, with no length-control baseline to separate the two.

Questions for authors
  1. What fraction of each batch actually lands in the easy and hard tiers under Beta(51,51)?
  2. What are the measured b_min and clipping bias at c = 4, and does any non-vacuous form of Proposition 3.1 hold at the settings you ran?
  3. Please report total rollouts and tokens per method for Table 12. Does the 8.71x speedup on 7B MATH-500 survive equal-token accounting, given that the on-policy fraction differs by row?
  4. Is the GRPO row in Tables 2-4 the same run as Random Selection in Sec. 2 (67.40, 626.9 tokens)? If so, what does fully on-policy GRPO reach at the same step count?
  5. App. F merges the MATH train and test splits minus MATH-500. What decontamination was done against Minerva, OlympiadBench and AIME?
5 comments

We thank the reviewer for the thorough review. All new results below will be added to the revised manuscript.

Weaknesses

W1. Proposition 3.1 and tier exposure

We greatly thank the reviewer for the careful check.

Proposition 3.1 needs its concentration term below bmin⁡b_{\min} ... roughly 28.6

Proposition 3.1 is a worst-case bound. Sec. 3.1: "The worst-case concentration term can be loose for large c, so we use ESS (see App. C) in practice." In practice we measure the error directly: MAE 0.1889 versus 0.2775 for the best baseline (Table 5).

neither quantity is ever measured

Both are tracked at every estimation step. ESS measures the weight spread that the concentration term bounds. App. H.1: "the SNIS estimate is accepted only if ESSq≥τESS\mathrm{ESS}_q\geq\tau_{\mathrm{ESS}}"; 88.7% of prompts pass at τESS=3\tau_{\mathrm{ESS}}=3 (Table 7). Only the upper clip can pull bmin⁡b_{\min} below 1 (App. B). In our 1.5B runs at c=4c=4, only 0.2–0.6% of weights hit the upper clip. So bmin⁡b_{\min} stays close to 1.

κ=100\kappa=100 ... easy and hard tiers ... nearly unreachable

Reward shaping and the easy-prompt clipping do not use the sampler; they use each group's realized failure rate (Eq. 7, Sec. 3.3). A prompt sampled near 0.5 can still fail 7 of 8 rollouts, and for that update it is a hard prompt. In our 1.5B logs, both tiers are active in every batch: 16–23% of prompts are easy and about 10% hard. Tiered allocation raises 7B MATH-500 from 75.8 to 77.0 (Table 4).

W2. Budget matching

We greatly thank the reviewer for the attention to training cost.

Nothing is budget-matched

The paper measures training efficiency on two axes, Sec. 1: "training efficiency (measured by steps and wall-clock time)". Wall-clock time is one budget for all rows: it counts all generation, replay, and estimation, whatever the on-policy fraction. Sec. 4.2: "DARE reaches the highest accuracy in the fewest steps and shortest time among difficulty-aware methods" (Fig. 4, right). The same holds on SmolLM3-3B (App. H) and LiveCodeBench (App. I).

a step is not the same generation across rows

GRPO, the reference for the 8.71x, uses the same B=512B=512 and σ=0.5\sigma=0.5 as DARE (App. F). Baselines with σ=1.0\sigma=1.0 generate twice as many fresh rollouts per step, so a per-step comparison favors them (Q3).

rollouts and tokens per step are never reported - 8.71x is a step ratio

Rollouts per step are in App. F: "on-policy fraction σ=0.5\sigma = 0.5 (i.e., 256 fresh rollouts and 256 replay samples per step)"; all are fresh at σ=1.0\sigma=1.0. App. J defines 8.71x as a "step speedup"; in generated tokens the gap is 7.8x (Q3).

The token reduction ... direct target of Eq. 9, with no length-control baseline

Shorter easy answers are not the goal in themselves. Eq. 9 and Eq. 10 together move reasoning budget by difficulty. Sec. 3.3: "the policy learns to be concise on prompts it solves reliably and to reason more deeply on prompts where it struggles." A length control would shorten answers at every level; DARE does not. Sec. 4.2: "The token savings hold across Levels 1–4 (up to 11%) ... while Level 5 is the only level where DARE spends more tokens (+7%) and it is also where the accuracy gain is largest (+8.21)" (Table 3). Against the no-shaping row of Table 8, Minerva answers get longer and more accurate (826→882 tokens, 14.3→17.3%).

Questions

Q1. Tier fractions under Beta(51,51)

We thank the reviewer for this insightful question. In our 1.5B logs, 16–23% of each batch is easy and about 10% hard (W1).

Q2. bmin⁡b_{\min} and clipping bias

We thank the reviewer for the close reading. Only 0.2–0.6% of weights hit the upper clip, the only side that lowers bmin⁡b_{\min}, so bmin⁡b_{\min} stays close to 1 (W1). Each lower-clipped weight adds less than e−ce^{-c} to the clipping bias (App. B).

Q3. Equal-token speedup

We thank the reviewer for the thoughtful question. Yes. Generation cost to reach 72% on 7B MATH-500 (Table 12), with measured response lengths of 744–977 tokens:

Methodσ\sigmaStepsRolloutsTokensvs. DARE
GRPO0.5122250K186–244M7.8x
LLM-Judge0.552106K79–104M3.3x
Previous FR Estimation1.0118483K360–472M15.1x
DARE0.51432K24–31M1.0x

Q4. GRPO row and fully on-policy GRPO

We thank the reviewer for the careful cross-check. Yes, the GRPO row is the Random Selection run (identical per-level results, Tables 1 and 3). The fully on-policy row in Table 12 is Previous FR Estimation (σ=1.0\sigma=1.0): it reaches 72% at step 118, versus 122 for GRPO and 14 for DARE.

Q5. Decontamination

We thank the reviewer for raising this important point. We checked all 35,512 training prompts for exact matches and near-duplicates (≥50% shared normalized 13-grams). Overlaps: 0/272 Minerva, 3/674 OlympiadBench, 0/83 AIME-AMC, 0/60 AIME24/25. The three can change OlympiadBench by at most 0.45 points, equally for all methods.

We hope our response has addressed your concerns. We know your time is valuable, and we would be very grateful for any further questions or suggestions.

AuthorsOpenReview ↗

Thank You for Your Valuable Comment :) !

As we approach the end of the response period, I would like to sincerely thank you for your valuable suggestions, which have helped us further improve the clarity of our paper. We genuinely hope that our responses have adequately addressed your concerns. If time permits and you have any further questions, we would be more than happy to respond promptly. Regardless, we wish you a wonderful day!

Comments on the paper

1 comment
AuthorsOpenReview ↗

With Gratitude to the AC and Reviewers: Summary of the Discussion

We thank the AC and all three reviewers for their careful reviews. We answered every weakness and question, mostly with quotes from the submitted paper, and added new evidence during the rebuttal: a matched system-level comparison, tier frequencies and weight-clipping statistics from the training logs, an equal-token accounting of the speedup, a decontamination check, and a hyperparameter comparison with accepted methods. All of it will be added to the revised manuscript.

1. Fair comparison of the full system (42UZ W2). The reviewer asked whether DARE's gains come from the estimator or from the added training components. We ran the full DARE pipeline on Qwen2.5-Math-1.5B with only the difficulty estimator replaced. SNIS is best on all five benchmarks, and the ranking follows the estimation error in Table 5 (0.1889 < 0.2775 < 0.2988). Previous FR reads the same replay buffer without the importance weights, so its row isolates the weights: removing them costs 1.6–3.5 points on every benchmark. The reviewer confirmed that this is "exactly the type of comparison I had in mind" and that "I do not think additional experiments are necessary."

2. Easy and hard tiers under Beta(51,51) (tQA7 W1/Q1, UgTg W1). Both reviewers computed that the sampler rarely draws prompts far from 0.5. Reward shaping and the easy-prompt clipping do not depend on the sampler; they use each group's realized failure rate (Eq. 7, Sec. 3.3). The 1.5B training logs show both tiers active in every batch: 16–23% of prompts are easy and about 10% hard. The one-at-a-time ablation shows each branch helps (Table 10): tiered allocation raises MATH-500 from 69.8 to 71.6, and the memory hint raises Minerva from 14.3 to 16.2.

3. Budget matching (tQA7 W2/Q3). Wall-clock time, already reported for all methods (Fig. 4), counts all generation, replay and estimation. We added the requested equal-token accounting for Table 12: to reach 72% on 7B MATH-500, GRPO generates 7.8x the tokens of DARE, and the fully on-policy baseline 15.1x. We also checked contamination against all evaluation sets: 0/272 Minerva, 3/674 OlympiadBench, 0/83 AIME-AMC, 0/60 AIME24/25 overlap, which changes no conclusion.

4. Proposition 3.1 and hyperparameters (tQA7 W1/Q2, 42UZ W3/Q3). Prop. 3.1 is the reason for the ESS test, and every ESS threshold from 1 to 5 beats the best prior estimator (Table 7). For the quantities tQA7 asked to be measured, only 0.2–0.6% of weights hit the upper clip, the only side that lowers bmin⁡b_{\min}, so bmin⁡b_{\min} stays close to 1. DARE tunes five values, all swept in the paper (Tables 7–9); accepted methods in this line tune as many or more (DEPO, ICLR 2026: eight; GRESO, NeurIPS 2025: six; ExGRPO, ICLR 2026: five).

Where the discussion ended. Reviewer tQA7 raised the score after our responses on tier exposure, Proposition 3.1, budget matching and decontamination. Reviewer 42UZ engaged in two rounds. After the matched comparison, the reviewer wrote that the paper is "empirically quite comprehensive and very clearly written" and "I recognize the merits of the proposed method", and narrowed the remaining reservation to methodological novelty. On this point, the reviewer's first review identifies the policy-aligned SNIS estimate as "the most clearly novel component ... a sensible idea", and the matched comparison shows that the full system's gains follow the accuracy of this estimator. Building training strategies on the difficulty signal and reporting the full system is also the norm in accepted work in this line, including DOTS (NeurIPS 2025), CurES, DEPO and ExGRPO (ICLR 2026). On scope, the paper covers four models and goes beyond math: App. I trains on code and finds that "the estimator-level advantage transfers from mathematical reasoning to code generation."

We are grateful to all three reviewers for their time and constructive engagement, and to the AC for overseeing this discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.