acceptodds
Under review as a conference paper at ICLR 2027

Learning What to Learn: Langevin-based Curriculum Optimization for RLVR

Abstract

Reinforcement learning with verifiable rewards has substantially improved the reasoning capabilities of large language models, but its efficiency is often limited by uniform prompt sampling that ignores differences in training utility under the evolving policy. Curriculum learning offers a natural way to adapt training data to the evolving policy, yet existing methods either rely on predefined difficulty schedules or estimate utility at the individual-prompt level, where sparse rollout feedback can lead to unreliable estimates and self-reinforcing sampling bias. To address these challenges jointly, we propose Langevin-based Adaptive Curriculum Optimization (LACO). LACO groups prompts into buckets based on their initial difficulty and pools online rollout feedback within each bucket, enabling more reliable estimation of their current training utility. It then uses Langevin Monte Carlo to adapt the sampling weights across buckets: utility-driven updates favor currently informative buckets, while stochastic perturbations preserve exploration of under-sampled buckets whose utility may change as the policy learns. This enables LACO to continuously track the policy's moving capability frontier while avoiding premature sampling concentration. Extensive experiments across multiple reasoning benchmarks demonstrate consistent improvements over uniform sampling and existing curriculum methods.

Primary area:
reinforcement learning
Keywords:
reinforcement learning with verifiable rewards, curriculum learning, Langevin dynamics, adaptive prompt sampling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_H9CsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a data sampling strategy to improve the efficiency of LLM RL training, particularly for reasoning models. It designs a Langevin-based strategy to update sampling weights of data, which are organized into buckets for more accurate estimation of utility. Moreover, it includes a diffusion mechanism to low-weight buckets can also be sampled. Experiments are performed on multiple datasets covering math and science domains. Analysis of data sampling is also presented like non-zero advantage ratios of data batches.

Critical strengths
  1. The efficiency of RL is key for its scaling and final performance. This work provides a reasonable approach to address this.
  2. The Lengevin-based strategy provides a new idea in data sampling.
Critical weaknesses

The main weaknesses lie in the experiments.

  • Not evaluated on coding tasks.
  • Training separate model for each domain. What if training on mixed domains? Will the proposed method still work? If not, it would be a severe limitation in both its effectiveness and impact.
  • Not validated on agentic tasks.
  • Marginal improvement over uniform sampling accoding to Table 1. The gaps are ~2%, which can be accidental. It is unsure whether there are multiple runs for each of these numbers. The results in Table 2 are also similar with small gaps between different items.
  • Only tested with <=8B models.
  • No concrete analysis and comparison of sampled data by different methods. What kind of data will be preferred during different stages of learning?
  • Lack of comparison with relevant methods discussed in intro or related work, e.g. DAPO, LILO, CurES.
Questions for authors

What is the most direct motivation to use Langevin MC?

8 comments
AuthorsOpenReview ↗

Response to Question

We thank the reviewer for these valuable suggestions. We would like to emphasize that our original evaluation follows established experimental protocols in prior RLVR studies[1–4] and provides a comprehensive and sufficiently rigorous assessment of our core contribution, covering mathematical and scientific reasoning, models with 1.5B, 1.7B, 7B, and 8B parameters, and training with GRPO, PPO, and REINFORCE++, and extensive ablation studies. To further strengthen the empirical evidence, we have followed the reviewer's suggestions and conducted additional experiments on coding, agentic tasks, mixed-domain training, and larger models, with detailed results presented in our point-by-point responses below.

[1] Parashar et al., 2026. Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning.

[2] Qu et al., 2026. Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?

[3] Zeng et al., 2026. CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs.

[4] Bae et al., 2026. Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning. EACL 2026.


Response to question "What is the most direct motivation to use Langevin MC?"

The key challenge in RLVR curriculum learning is that sampling allocation and utility estimation are inherently coupled: prioritizing currently informative prompts reduces observations from under-sampled regions, potentially leaving their utility estimates stale as the policy evolves. LACO addresses the estimation challenge through bucket-level feedback aggregation, which reduces the sparsity of utility observations and enables more stable, lower-dimensional curriculum optimization. However, reliable estimation alone cannot resolve the self-reinforcing bias caused by adaptive sampling.

Langevin dynamics are particularly suited to this setting because they combine utility-driven optimization with stochastic feedback renewal. Specifically, the drift optimizes an entropy-regularized utility objective, while diffusion perturbs the sampling logits independently of their current weights, enabling under-sampled buckets to regain sampling opportunities and refresh their utility estimates. The key contribution is therefore not simply applying Langevin dynamics, but designing a feedback-driven curriculum optimization framework that jointly addresses estimation reliability and sampling-induced feedback bias, allowing the sampling distribution to adapt as the policy's capability frontier shifts. Our theoretical analysis (Appendix B) characterizes the conditions under which diffusion increases feedback acquisition, while the ablations (Table 2 and Figure 5) empirically support its effectiveness.

AuthorsOpenReview ↗

Response to Weaknesses 1 and 3

Weakness 1&3

Not evaluated on coding and agentic tasks.

We have extended our evaluation to widely adopted coding and agentic benchmarks using Qwen3-1.7B-Base, comparing LACO against GRPO, DAPO, and CurES.

  • Coding. We train on the Eurus coding dataset and evaluate on three established coding benchmarks: HumanEval+, MBPP+, and LiveCodeBench.
  • Agentic tasks. We train on ALFWorld and evaluate on its seen and unseen splits, as well as WebShop, two widely used benchmarks for evaluating interactive agents.

Across both domains, LACO achieves the best overall performance among the compared methods, further demonstrating its effectiveness and generalizability beyond mathematical and scientific reasoning tasks.

MethodHumanEval+MBPP+LiveCodeBench
GRPO57.9353.4410.86
DAPO62.8057.9415.43
CurES62.2056.6111.43
LACO65.2462.1717.71
MethodALFWorld SeenALFWorld UnseenWebShop
GRPO82.8478.3635.80
DAPO84.3380.6037.20
CurES82.0979.1036.60
LACO85.0782.0939.40
AuthorsOpenReview ↗

Response to Weakness 2

Weakness 2

Training separate model for each domain. What if training on mixed domains? Will the proposed method still work? If not, it would be a severe limitation in both its effectiveness and impact.

we conduct additional mixed-domain experiments by jointly training a single Qwen3-1.7B-Base model on 3,000 OpenRS mathematical problems and 3,000 Eurus coding problems. Since LACO constructs its curriculum based on policy-estimated difficulty rather than domain-specific information, it can be directly applied to mixed-domain training without architectural modifications. We compare LACO against uniform sampling under the same GRPO framework, as existing curriculum baselines were primarily evaluated in task-specific settings and their mixed-domain adaptations would require additional validation for a reliable comparison.

Both methods are trained for 200 steps. Table below reports mathematical and coding performance separately.

DomainBenchmarkGRPOLACOΔ (pp)
MathMATH50064.6066.80+2.20
MathAIME20254.177.08+2.91
MathAMC35.4739.53+4.06
MathMinervaMath23.1623.97+0.81
CodingHumanEval+57.9362.80+4.87
CodingMBPP+53.9757.94+3.97
CodingLiveCodeBench13.1414.86+1.72

LACO achieves the best performance across all six evaluation settings, outperforming the strongest competing baseline by 2.28–4.23 percentage points on coding tasks and 0.74–2.20 points on agentic tasks. These additional results demonstrate that LACO remains effective in joint multi-domain training without requiring separate domain-specific models, further supporting its generalizability across diverse reasoning tasks.

Reviewer_Q3WeOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LACO, an adaptive prompt-sampling method for RLVR. It groups prompts by initial difficulty, updates bucket-level success rates during training, and uses Langevin updates to favor informative buckets while maintaining exploration.

Critical strengths

The paper addresses the practical problem of allocating rollout computation efficiently and reports consistent improvements over the evaluated sampling baselines.

Critical weaknesses

The novelty appears limited given existing intermediate-difficulty selection and entropy-regularized sampling methods, such as PCL [1] and CurES [2]. The main additions are bucket-level feedback pooling and Langevin updates, but comparisons with these closely related methods and simpler exploration strategies are missing. This makes their added value difficult to assess.

[1] Prompt Curriculum Learning for Efficient LLM Post-Training

[2] CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs

Questions for authors
  1. The paper motivates stochastic diffusion as a way to refresh feedback from under-sampled buckets. Why is Langevin updating preferable to simpler alternatives, such as softmax utility weights mixed with uniform exploration?
  2. Can you compare against PCL and CurES under the same total compute budget, including initial difficulty calibration?
5 comments
AuthorsOpenReview ↗

Response to Critical Weakness

We appreciate the reviewer's feedback. Please find our point-by-point response below.

The novelty appears limited given existing intermediate-difficulty selection and entropy-regularized sampling methods, such as PCL [1] and CurES [2]. The main additions are bucket-level feedback pooling and Langevin updates, but comparisons with these closely related methods and simpler exploration strategies are missing. This makes their added value difficult to assess.

We would like to clarify that LACO's novelty lies not in intermediate-difficulty selection or entropy regularization alone, but in addressing the coupling between sampling allocation and utility estimation under sparse, evolving feedback. PCL relies on an auxiliary value model for prompt selection, introducing additional training costs and dependence on prediction accuracy, while CurES combines entropy-regularized sampling with prompt-level Bayesian estimates. However, prompt-level utility estimates can become stale when low-priority prompts are rarely revisited, creating a self-reinforcing bias as the policy evolves.

LACO addresses this challenge through two complementary mechanisms. First, bucket-level feedback pooling enables more reliable utility estimation from sparse rollouts while reducing the dimensionality of curriculum optimization. Second, Langevin dynamics combine utility-guided drift with stochastic diffusion over persistent sampling logits. Unlike entropy regularization alone, diffusion can perturb low-weight buckets even when deterministic gradients become weak, creating opportunities to refresh their utility estimates and adapt the curriculum to the evolving policy. The contribution is therefore a feedback-driven curriculum optimization framework that jointly improves utility estimation and mitigates sampling-induced feedback bias, rather than simply incorporating Langevin updates into existing sampling strategies.

This design is further supported by our theoretical analysis (Appendix B), which establishes conditions under which diffusion increases feedback acquisition, and our ablations (Table 2 and Figure 5), which demonstrate its benefits for performance and feedback freshness.

AuthorsOpenReview ↗

Response to Question 1

  1. The paper motivates stochastic diffusion as a way to refresh feedback from under-sampled buckets. Why is Langevin updating preferable to simpler alternatives, such as softmax utility weights mixed with uniform exploration?

The motivation for Langevin updates is to address the coupling between curriculum allocation and feedback acquisition as the policy evolves. Unlike directly mapping estimated utilities to sampling weights, LACO combines history-dependent allocation updates with stochastic exploration, allowing under-sampled buckets to be revisited and their utility estimates refreshed.

First, LACO maintains a history-dependent curriculum state. A softmax utility rule mixed with uniform exploration takes the form

=(1-\varepsilon)\operatorname{softmax}(U_{t+1}/\tau) +\varepsilon u,$$ where $u$ is the uniform distribution over buckets. For fixed utility estimates, this rule produces the same allocation at every step, with low-utility buckets relying primarily on their fixed exploration share. In contrast, LACO updates the sampling logits incrementally: $$z_{t+1} =z_t-h\nabla_z\widetilde F_{t+1}^{(c)}(z_t) +\sqrt{\frac{2h}{\beta_L}}\xi_t, \qquad w_{t+1}=\operatorname{softmax}(z_{t+1}),$$ where $\widetilde F_{t+1}^{(c)}(z)=F_{t+1}^{(c)}(\operatorname{softmax}(z))$ denotes the implemented utility-scaled objective, as clarified in our response to Reviewer 4P9c. The logit state retains the effects of previous updates and stochastic perturbations. Consequently, exploration-induced changes in allocation can persist across iterations, creating opportunities to acquire fresh feedback from previously under-sampled buckets. Once their utility estimates are refreshed, subsequent utility-driven updates can respond to changes in their learning value. **Second, diffusion enables exploration even when deterministic updates become weak.** Under the softmax parameterization, the gradient of the objective with respect to a bucket's logit contains a multiplicative factor $w_{t,k}$. As $w_{t,k}\to0$, this gradient vanishes, weakening the direct deterministic adjustment of that logit. In contrast, Gaussian diffusion perturbs each logit with a fixed standard deviation $\sqrt{2h/\beta_L}$, independent of its current sampling weight. Thus, stochastic perturbations can change the relative logits of under-sampled buckets even when their deterministic updates are small. Through the softmax transformation, these perturbations create opportunities to increase their sampling weights and obtain fresh feedback. Subsequent utility-driven drift can then exploit the updated information. **Our theoretical analysis and experiments support this mechanism.** Appendix B shows that, for sufficiently small diffusion strength, the expected sampling weight of a bucket increases when $v_k < \lVert v \rVert_2^2$, where $v$ denotes the sampling weights after deterministic drift. It further establishes that diffusion increases the probability of obtaining new feedback in the next batch under the sufficient condition $v_k < \lVert v \rVert_2^2/(B+1)$, where $B$ is the batch size. Empirically, Figure 5 shows that diffusion reduces the average age of observations from an initially low-weight bucket, supporting its role in feedback renewal. Moreover, Table 2 shows that **removing diffusion reduces Avg@16 from 28.57% to 27.99%**, a decrease of **0.58 percentage points**, while retaining online utility updates and entropy regularization. Together, these results support the role of Langevin diffusion in LACO. **Its distinction from fixed uniform exploration lies in the interaction between persistent stochastic logit updates, feedback renewal, and subsequent utility-driven adaptation**, rather than simply allocating a fixed exploration probability to under-sampled buckets.
AuthorsOpenReview ↗

Response to Question 2

  1. Can you compare against PCL and CurES under the same total compute budget, including initial difficulty calibration?

LACO's advantage persists when the full cost of constructing and updating the curriculum is included.

Specifically, we compare GRPO, PCL, CurES, and LACO using Qwen3-1.7B-Base on OpenRS, with the same initial checkpoint, training framework, and eight NVIDIA A800 80GB GPUs. The common budget is 56.34 GPU-hours, corresponding to LACO's complete run with 200 training steps. All methods use the same total rollout budget per training step; their numbers of completed steps depend on their actual computational costs.

We measure elapsed time from model loading through checkpoint saving, including calibration, rollout generation, reward computation, policy optimization, and curriculum updates. This accounts for LACO's initial difficulty calibration, PCL's value-model scoring and training, and CurES's calibration rollouts, gradient estimation, and stage transitions. We evaluate the last checkpoint saved within the budget and report its actual GPU-hours below. Evaluation is performed separately for all methods.

MethodTraining stepsAIME24AIME'25MATH500AMCMinervaOlympiadAverage
GRPO2356.464.7965.8638.4422.8230.7128.18
PCL2147.715.6266.7439.0623.2830.4928.82
CurES1876.675.0066.2739.6923.0230.6528.55
LACO2008.756.6767.5041.2524.1331.8630.03

All benchmarks except MATH500 report Avg@16 (%); MATH500 reports Pass@1 (%). Average is the arithmetic mean across the six benchmarks.

At this budget, LACO achieves the highest average accuracy of 30.03%, exceeding PCL and CurES by 1.21 and 1.48 percentage points, respectively, and outperforming both methods on all six benchmarks. We will include this comparison and the compute-accounting protocol in the revised paper.

Reviewer_4P9cOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Langevin-based Adaptive Curriculum Optimization (LACO) for reinforcement learning with verifiable rewards. LACO groups prompts by their initial difficulty, pools online feedback to estimate bucket-level utility, and updates sampling weights through utility-driven gradients and Gaussian noise. Experiments show improved reasoning accuracy and rollout utilization compared with uniform sampling and existing curriculum methods.

Critical strengths
  1. The paper addresses a meaningful problem: adaptive sampling can leave utility estimates outdated by repeatedly overlooking certain prompts. Connecting curriculum allocation with opportunities to refresh feedback gives the method a clear motivation.
  2. The method is simple and practical. Bucket-level aggregation reduces the need to estimate each prompt’s utility independently, and curriculum updates reuse training rollouts without requiring a separate prediction model.
  3. Results show consistent average improvements across four models. The component ablations and feedback-freshness analysis provide useful evidence for the proposed mechanisms, while additional experiments suggest compatibility with different RL optimizers.
Critical weaknesses
  1. A bucket’s average success rate can hide important differences in its prompts’ learning value. For example, an equal mixture of fully solved and completely unsolved prompts has an average success rate of 0.5 and receives the highest utility, although neither group provides useful GRPO advantages. Figure 2 supports bucket coherence in the studied setting, but the argument in Appendix C depends on within-bucket variation remaining small. Since assignments stay fixed, LACO cannot adapt when initially similar prompts develop differently. A variance-aware utility or occasional rebucketing could address this limitation.

  2. The reported update scales raise a question about how strongly utility actually drives the curriculum. From Equations 9–11 and Table 5, the cumulative utility-driven change in each logit is at most 0.125 over 200 steps, while the accumulated Gaussian noise has a standard deviation of 2. This suggests that random perturbations may dominate the sampling trajectory, making the explanation of utility-driven tracking less convincing despite the positive ablations. The authors should clarify whether the implementation uses additional scaling and explain how the stated update balances adaptation with exploration.

Questions for authors

Please response the weakness points.

6 comments
AuthorsOpenReview ↗

Response to Weakness 1 (Part 1/2)

We appreciate the reviewer's feedback. Please find our point-by-point response below.

  1. A bucket’s average success rate can hide important differences in its prompts’ learning value. For example, an equal mixture of fully solved and completely unsolved prompts has an average success rate of 0.5 and receives the highest utility, although neither group provides useful GRPO advantages. Figure 2 supports bucket coherence in the studied setting, but the argument in Appendix C depends on within-bucket variation remaining small. Since assignments stay fixed, LACO cannot adapt when initially similar prompts develop differently. A variance-aware utility or occasional rebucketing could address this limitation.

Our additional longitudinal analysis shows that within-bucket heterogeneity does not materially affect bucket prioritization in our experiments. After accounting for within-bucket variance, the highest-utility bucket and the top-three ranking remain unchanged at all evaluated checkpoints, with 49 out of 50 pairwise bucket comparisons preserved.

Quantifying within-bucket heterogeneity. The effect of heterogeneity on our quadratic utility can be characterized exactly. At a given checkpoint, let p(x)p(x) denote the current policy's success probability for prompt xx, and define

\qquad v_k=\operatorname{Var}_{x\sim D_k}(p(x)),$$ where the expectation is uniform over prompts in bucket $D_k$. For $U(p)=-(p-0.5)^2$, we have $$U(m_k)-\mathbb{E}_{x\sim D_k}[U(p(x))]=v_k.$$ Thus, **the discrepancy between the mean-based utility used by LACO and the average prompt-level utility is exactly the within-bucket variance**. This identity allows us to directly assess whether within-bucket heterogeneity affects bucket prioritization. **Additional experiments.** We evaluated Qwen3-1.7B-Base at training steps 40, 80, 120, 160, and 200, using 256 prompts per bucket and 8 responses per prompt at each checkpoint. Table R1 reports the evolution of bucket-level success rates. As the policy improves, success rates generally increase across buckets, shifting the buckets closest to the target success probability of $0.5$. This illustrates why curriculum allocation needs to adapt throughout training, even when bucket assignments remain fixed. **Table R1. Mean prompt success rate (%)** | Bucket | Step 40 | Step 80 | Step 120 | Step 160 | Step 200 | |---|---:|---:|---:|---:|---:| | 1 | 62.50 | 72.27 | 77.10 | 79.20 | 79.83 | | 2 | 50.93 | 66.06 | 70.07 | 71.92 | 71.58 | | 3 | 45.85 | 61.13 | 64.60 | 65.19 | 66.06 | | 4 | 30.81 | 42.38 | 45.21 | 44.48 | 44.58 | | 5 | 14.45 | 21.14 | 22.27 | 22.17 | 25.39 | To estimate genuine within-bucket heterogeneity, we correct for the sampling noise arising from the finite number of rollouts per prompt. Let $q_{k,i}$ denote the empirical success rate of the $i$-th evaluated prompt in bucket $D_k$ at the current checkpoint, computed from $G$ responses, and define $\bar q_k=\frac{1}{n}\sum_{i=1}^{n}q_{k,i}$. We estimate $$\hat v_k= \frac{1}{n-1}\sum_{i=1}^{n}(q_{k,i}-\bar q_k)^2 -\frac{1}{n(G-1)}\sum_{i=1}^{n}q_{k,i}(1-q_{k,i}),$$ where $n=256$ and $G=8$. The correction removes the expected contribution of finite-rollout Bernoulli sampling noise to the empirical variance. **Table R2. Within-bucket sample variance corrected for rollout sampling noise** | Bucket | Step 40 | Step 80 | Step 120 | Step 160 | Step 200 | |---|---:|---:|---:|---:|---:| | 1 | 0.028112 | 0.030425 | 0.026697 | 0.024839 | 0.021987 | | 2 | 0.026984 | 0.031820 | 0.027795 | 0.025992 | 0.023635 | | 3 | 0.025481 | 0.033876 | 0.029190 | 0.026738 | 0.027480 | | 4 | 0.020340 | 0.029855 | 0.029601 | 0.028913 | 0.023701 | | 5 | 0.011297 | 0.018642 | 0.019913 | 0.021468 | 0.013236 | Table R2 shows that within-bucket heterogeneity remains relatively limited throughout training. The maximum estimated variance is **0.0339**, substantially below the variance of $0.25$ in the reviewer's example of an equal mixture of fully solved and completely unsolved prompts. More importantly, we next examine whether the observed heterogeneity actually changes bucket prioritization.
AuthorsOpenReview ↗

Response to Weakness 1 (Part 2/2)

Impact on bucket prioritization. We compare the original mean-based utility with its variance-corrected counterpart:

\qquad \hat U_k^{\mathrm{corr}}=\hat U_k^{\mathrm{mean}}-\hat v_k.$$ where $\bar q_k$ is the mean empirical success rate over the $n=256$ prompts evaluated in bucket $D_k$ at the current checkpoint. Table R3 reports the resulting bucket rankings at each checkpoint, ordered from highest to lowest utility. **Table R3. Bucket rankings before and after variance correction** | Checkpoint | Mean-based ranking | Variance-corrected ranking | Preserved pairwise comparisons | |---|---|---|---:| | 40 | 2 > 3 > 1 > 4 > 5 | 2 > 3 > 1 > 4 > 5 | 10/10 | | 80 | 4 > 3 > 2 > 1 > 5 | 4 > 3 > 2 > 1 > 5 | 10/10 | | 120 | 4 > 3 > 2 > 1 > 5 | 4 > 3 > 2 > 5 > 1 | 9/10 | | 160 | 4 > 3 > 2 > 5 > 1 | 4 > 3 > 2 > 5 > 1 | 10/10 | | 200 | 4 > 3 > 2 > 5 > 1 | 4 > 3 > 2 > 5 > 1 | 10/10 | **The highest-utility bucket and the entire top-three ordering are preserved at every checkpoint.** Overall, $49$ of $50$ pairwise comparisons remain unchanged. The only reversal occurs between the two lowest-utility buckets at step $120$. Notably, both criteria identify the shift in the highest-utility bucket from Bucket 2 at step $40$ to Bucket 4 at subsequent checkpoints. These results demonstrate that the observed within-bucket heterogeneity has little effect on the identification of high-utility buckets along the evaluated training trajectory. **Regarding fixed bucket assignments**, LACO does not assume that prompt success probabilities remain constant during training. Although bucket membership is fixed, bucket utilities and sampling weights are continuously updated using online rollout feedback, allowing the curriculum to track changes in the policy's capability. Fixed grouping cannot capture arbitrary divergence among prompts within the same bucket, but our analysis shows that such heterogeneity does not materially alter the estimated priorities of high-utility buckets in the evaluated setting. These findings provide direct empirical support for fixed difficulty grouping and complement the sufficient-condition analysis in Appendix C. We will incorporate the variance identity and additional longitudinal results into Appendix C.
AuthorsOpenReview ↗

Response to Weakness 2

  1. The reported update scales raise a question about how strongly utility actually drives the curriculum. From Equations 9–11 and Table 5, the cumulative utility-driven change in each logit is at most 0.125 over 200 steps, while the accumulated Gaussian noise has a standard deviation of 2. This suggests that random perturbations may dominate the sampling trajectory, making the explanation of utility-driven tracking less convincing despite the positive ablations. The authors should clarify whether the implementation uses additional scaling and explain how the stated update balances adaptation with exploration.

The scale comparison does not account for the utility-scaling coefficient c=100c=100 used in our implementation, which was inadvertently omitted from the manuscript. Accounting for this coefficient substantially changes the scale of the utility-driven drift. Our additional ablation experiments further confirm that utility guidance makes a substantial contribution to LACO's performance.

Clarification of the implemented update. Equation (9) defines the unscaled bucket utility as Ut+1,k=−(p^t+1,k−0.5)2U_{t+1,k}=-(\hat p_{t+1,k}-0.5)^2. The curriculum objective used in our implementation is

=-c\sum_{k=1}^{K}w_kU_{t+1,k}-\lambda H(w), \qquad H(w)=-\sum_{k=1}^{K}w_k\log w_k,$$ where $c=100$. The corresponding Langevin update is $$z_{t+1} =z_t-h\nabla_z\left[F_{t+1}^{(c)}(\operatorname{softmax}(z))\right]_{z=z_t} +\sqrt{\frac{2h}{\beta_L}}\\,\xi_t, \qquad \xi_t\sim\mathcal N(0,I_K).$$ Letting $\bar U_{t+1}=\sum_jw_{t,j}U_{t+1,j}$, the utility-driven logit increment is $$\Delta z^U_{t,k} =hc\\,w_{t,k}(U_{t+1,k}-\bar U_{t+1}).$$ Since $U_{t+1,k}\in[-1/4,0]$, we have $$\lvert \Delta z^U_{t,k} \rvert \leq \frac{hc}{4}w_{t,k}(1-w_{t,k}) \leq \frac{hc}{16}.$$ With $h=0.01, \beta_L=1, T=200$, and $c=100$, this yields $$\left\lvert \sum_{t=0}^{T-1}\Delta z^U_{t,k} \right\rvert \leq \frac{Thc}{16}=12.5,$$ whereas the standard deviation of the accumulated Gaussian noise is $$\sqrt{\frac{2hT}{\beta_L}}=2.$$ Thus, **including the omitted coefficient increases the upper bound on the cumulative utility-driven contribution from $0.125$ to $12.5$**. However, this worst-case bound does not establish the realized relative contributions of drift and diffusion, which depend on the evolving utilities and sampling weights. We therefore conducted additional experiments to directly assess the effect of utility guidance. **Additional utility-guidance ablations.** We compare full LACO ($c=100$) with two variants: removing utility drift ($c=0$) and using unscaled utility ($c=1$). Both variants retain entropy regularization and Gaussian diffusion. All experiments use **Qwen3-1.7B-Base**, with $h=0.01, \beta_L=1$, and $\lambda=0.01$. The three settings share the same initial model checkpoint, bucket construction, initial sampling weights, training data, 200-step rollout budget, and evaluation protocol. We use **three paired seeds**, with identical Langevin noise sequences within each matched comparison. **Table R4. Utility-guidance ablations with entropy regularization and Gaussian diffusion retained** | Setting | Utility coefficient $c$ | Six-benchmark average Avg@16 (%), mean ± std | Mean nonzero-advantage ratio (%) | |---|---:|---:|---:| | Full LACO | 100 | 28.82 ± 0.28 | 56.79 | | No utility drift | 0 | 26.36 ± 0.33 | 35.11 | | Unscaled utility | 1 | 26.70 ± 0.23 | 36.23 | The Avg@16 results are averaged across the six mathematical reasoning benchmarks in Table 1, with the mean and standard deviation computed over three training seeds. Nonzero-advantage ratios are averaged over $200$ training steps within each run and then across seeds. **The results demonstrate the substantial contribution of utility-driven adaptation.** Removing utility drift reduces average accuracy by **2.46 percentage points** and the mean nonzero-advantage ratio by **21.68 percentage points**. Using unscaled utility ($c=1$) similarly reduces accuracy by **2.12 percentage points** and the nonzero-advantage ratio by **20.56 percentage points**. Importantly, both ablations retain Gaussian diffusion and entropy regularization. Their substantially weaker performance shows that **stochastic exploration alone cannot explain LACO's improvements**. The utility-driven drift guides sampling toward informative buckets, while diffusion supports exploration and feedback renewal. The controlled comparisons confirm that utility guidance is a major contributor to LACO's effectiveness. We will revise Eq. (10) to include the utility-scaling coefficient, specify $c=100$ in Table 5 and Algorithm 1, and incorporate these additional ablation results into the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.