- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes a data sampling strategy to improve the efficiency of LLM RL training, particularly for reasoning models. It designs a Langevin-based strategy to update sampling weights of data, which are organized into buckets for more accurate estimation of utility. Moreover, it includes a diffusion mechanism to low-weight buckets can also be sampled. Experiments are performed on multiple datasets covering math and science domains. Analysis of data sampling is also presented like non-zero advantage ratios of data batches.
Critical strengths
- The efficiency of RL is key for its scaling and final performance. This work provides a reasonable approach to address this.
- The Lengevin-based strategy provides a new idea in data sampling.
Critical weaknesses
The main weaknesses lie in the experiments.
- Not evaluated on coding tasks.
- Training separate model for each domain. What if training on mixed domains? Will the proposed method still work? If not, it would be a severe limitation in both its effectiveness and impact.
- Not validated on agentic tasks.
- Marginal improvement over uniform sampling accoding to Table 1. The gaps are ~2%, which can be accidental. It is unsure whether there are multiple runs for each of these numbers. The results in Table 2 are also similar with small gaps between different items.
- Only tested with <=8B models.
- No concrete analysis and comparison of sampled data by different methods. What kind of data will be preferred during different stages of learning?
- Lack of comparison with relevant methods discussed in intro or related work, e.g. DAPO, LILO, CurES.
Questions for authors
What is the most direct motivation to use Langevin MC?
8 comments
Response to Weaknesses 1 and 3
Weakness 1&3
Not evaluated on coding and agentic tasks.
We have extended our evaluation to widely adopted coding and agentic benchmarks using Qwen3-1.7B-Base, comparing LACO against GRPO, DAPO, and CurES.
- Coding. We train on the Eurus coding dataset and evaluate on three established coding benchmarks: HumanEval+, MBPP+, and LiveCodeBench.
- Agentic tasks. We train on ALFWorld and evaluate on its seen and unseen splits, as well as WebShop, two widely used benchmarks for evaluating interactive agents.
Across both domains, LACO achieves the best overall performance among the compared methods, further demonstrating its effectiveness and generalizability beyond mathematical and scientific reasoning tasks.
| Method | HumanEval+ | MBPP+ | LiveCodeBench |
|---|---|---|---|
| GRPO | 57.93 | 53.44 | 10.86 |
| DAPO | 62.80 | 57.94 | 15.43 |
| CurES | 62.20 | 56.61 | 11.43 |
| LACO | 65.24 | 62.17 | 17.71 |
| Method | ALFWorld Seen | ALFWorld Unseen | WebShop |
|---|---|---|---|
| GRPO | 82.84 | 78.36 | 35.80 |
| DAPO | 84.33 | 80.60 | 37.20 |
| CurES | 82.09 | 79.10 | 36.60 |
| LACO | 85.07 | 82.09 | 39.40 |
Response to Weakness 2
Weakness 2
Training separate model for each domain. What if training on mixed domains? Will the proposed method still work? If not, it would be a severe limitation in both its effectiveness and impact.
we conduct additional mixed-domain experiments by jointly training a single Qwen3-1.7B-Base model on 3,000 OpenRS mathematical problems and 3,000 Eurus coding problems. Since LACO constructs its curriculum based on policy-estimated difficulty rather than domain-specific information, it can be directly applied to mixed-domain training without architectural modifications. We compare LACO against uniform sampling under the same GRPO framework, as existing curriculum baselines were primarily evaluated in task-specific settings and their mixed-domain adaptations would require additional validation for a reliable comparison.
Both methods are trained for 200 steps. Table below reports mathematical and coding performance separately.
| Domain | Benchmark | GRPO | LACO | Δ (pp) |
|---|---|---|---|---|
| Math | MATH500 | 64.60 | 66.80 | +2.20 |
| Math | AIME2025 | 4.17 | 7.08 | +2.91 |
| Math | AMC | 35.47 | 39.53 | +4.06 |
| Math | MinervaMath | 23.16 | 23.97 | +0.81 |
| Coding | HumanEval+ | 57.93 | 62.80 | +4.87 |
| Coding | MBPP+ | 53.97 | 57.94 | +3.97 |
| Coding | LiveCodeBench | 13.14 | 14.86 | +1.72 |
LACO achieves the best performance across all six evaluation settings, outperforming the strongest competing baseline by 2.28–4.23 percentage points on coding tasks and 0.74–2.20 points on agentic tasks. These additional results demonstrate that LACO remains effective in joint multi-domain training without requiring separate domain-specific models, further supporting its generalizability across diverse reasoning tasks.
Response to Question
We thank the reviewer for these valuable suggestions. We would like to emphasize that our original evaluation follows established experimental protocols in prior RLVR studies[1–4] and provides a comprehensive and sufficiently rigorous assessment of our core contribution, covering mathematical and scientific reasoning, models with 1.5B, 1.7B, 7B, and 8B parameters, and training with GRPO, PPO, and REINFORCE++, and extensive ablation studies. To further strengthen the empirical evidence, we have followed the reviewer's suggestions and conducted additional experiments on coding, agentic tasks, mixed-domain training, and larger models, with detailed results presented in our point-by-point responses below.
[1] Parashar et al., 2026. Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning.
[2] Qu et al., 2026. Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?
[3] Zeng et al., 2026. CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs.
[4] Bae et al., 2026. Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning. EACL 2026.
The key challenge in RLVR curriculum learning is that sampling allocation and utility estimation are inherently coupled: prioritizing currently informative prompts reduces observations from under-sampled regions, potentially leaving their utility estimates stale as the policy evolves. LACO addresses the estimation challenge through bucket-level feedback aggregation, which reduces the sparsity of utility observations and enables more stable, lower-dimensional curriculum optimization. However, reliable estimation alone cannot resolve the self-reinforcing bias caused by adaptive sampling.
Langevin dynamics are particularly suited to this setting because they combine utility-driven optimization with stochastic feedback renewal. Specifically, the drift optimizes an entropy-regularized utility objective, while diffusion perturbs the sampling logits independently of their current weights, enabling under-sampled buckets to regain sampling opportunities and refresh their utility estimates. The key contribution is therefore not simply applying Langevin dynamics, but designing a feedback-driven curriculum optimization framework that jointly addresses estimation reliability and sampling-induced feedback bias, allowing the sampling distribution to adapt as the policy's capability frontier shifts. Our theoretical analysis (Appendix B) characterizes the conditions under which diffusion increases feedback acquisition, while the ablations (Table 2 and Figure 5) empirically support its effectiveness.