acceptodds
Under review as a conference paper at ICLR 2027

Small Steps on the Geodesic: Ability-Anchored Progressive Curricula for RL on Reasoning

Abstract

Curriculum design for RL with verifiable rewards (RLVR) is usually framed as a per-step problem: keep each batch maximally informative by admitting only prompts of intermediate pass rate, where the Bernoulli variance peaks. We argue that this objective is incomplete. Viewed as transport of the policy's ability distribution toward a target, training accumulates deviation along the whole path, which no per-step criterion can see. The direction of travel must be estimated from the model's own pass rates, and no estimate is exact: a step of length leaves the Wasserstein geodesic by an amount linear in , whereas covering the same displacement in re-anchored steps lets independent errors cancel, so the deviation grows only as . Bounded, repeatedly re-anchored increments therefore stay closer to the geodesic than greedy jumps to the currently best difficulty, which gives a formal reason why Krashen's "" hypothesis calls for small increments. Whether greedy filtering actually loses depends on how the estimation error scales with step length, an assumption we state but do not prove. The premise of the per-step view is nonetheless sound: under group-relative advantages, a prompt whose rollouts are all correct or all wrong contributes exactly zero gradient, and 61–62% of Hendrycks-MATH lies at one of these two extremes. We instantiate the analysis as Ri1A and compare it with ODF, AdaCuRL, DAPO, and DAPO with static difficulty ordering (DAPO+static). At equal wall-clock (five hours on Qwen3-4B with identical data, batch size and response limit), Ri1A outperforms ODF and AdaCuRL by 2.9 and 2.0 pp on 502 pooled competition problems (), at and lower cost per optimizer step. Against DAPO and DAPO+static trained on the full dataset for their complete schedules, Ri1A matches or exceeds their accuracy in 43–46% of the wall-clock time; its Hard-Avg margin over DAPO is +0.12/+0.79/+3.04 pp at 1.7B/4B/8B.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning with verifiable rewards (RLVR), Curriculum learning, Wasserstein geodesic, Mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PTVqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies curriculum design for mathematical reasoning with RLVR. Using an idealized transport model in one-dimensional Wasserstein space, the authors show that smaller steps with repeated recalibration can reduce deviation from an ideal learning path. Motivated by this analysis, they propose Ri1A, which ranks problems by initial success rates and progressively selects them using an adaptive difficulty threshold. Experiments across multiple models and benchmarks report efficiency gains and performance improvements in some settings.

Critical strengths
  • Ri1A uses a single initial difficulty scan and stage-wise local evaluations to update the curriculum. This limits the additional cost of curriculum selection and makes the method practical to integrate into RLVR training.
  • The paper provides a fairly comprehensive experimental evaluation, covering both reasoning performance and training efficiency across multiple models and benchmarks, along with detailed ablation studies.
Critical weaknesses
  • The connection between the theory and the practical algorithm remains insufficiently established. Ri1A uses the fraction of problems selected per stage as a proxy for geometric step length and interprets threshold updates based on the model’s current performance as geometric recalibration. However, it does not establish how these choices control the actual Wasserstein displacement or directional error. Although the paper discusses these limitations and provides related ablations, it does not directly validate the proposed geometric mechanism.
  • The algorithmic novelty appears limited. Ri1A combines initial difficulty ranking, gradually introducing harder problems, adjusting difficulty using model performance, and deferring difficult problems. However, similar ideas of initial difficulty ranking and gradually introducing harder problems have been explored in AdaCuRL [1], while AdaRFT [2] considers adjusting difficulty using model performance. An extension proposed in the same AdaCuRL paper also revisits problems initially excluded as too difficult. Ri1A therefore appears to offer an incremental combination of existing ideas.

[1] AdaCuRL: Adaptive Curriculum Reinforcement Learning with Invalid Sample Mitigation and Historical Revisiting. arXiv, 2025.
[2] Efficient Reinforcement Finetuning via Adaptive Curriculum Learning. arXiv, 2025.

Questions for authors
  • Could you address the concerns raised in the Weaknesses section?
  • The threshold in Eq. (7) uses the current model’s average difficulty on the previous stage’s problems, but candidate problems are evaluated using difficulty scores from the initial model. For fixed problem sets, better current performance lowers the threshold while candidate scores remain unchanged. This can defer more problems even if the model has become better at solving them. Is this behavior intended?
Reviewer_eBKvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a stylised transport model to investigate per-step updates in RLVR. The core idea is to model RLVR training as moving along a geodesic in Wasserstein space, and error amplification is linear in step length. Therefore, it does not seek to maximize the gain in each step, but considers a progressive curriculum that takes small steps. To this end, this paper proposes a small-step Ri1A framework, which repeatedly re-anchors difficulty thresholds and defers hard samples instead of discarding them immediately. The experimental results on math benchmarks show trends that Ri1A boosts the performance of several base models (Qwen, Llama, and GLM) and achieves state-of-the-art performance.

Critical strengths
  1. The problem of zero gradient under group-relative advantage is important, and the proposed method provides a practical solution to it.

  2. The Wasserstein geodesic transport perspective is interesting, as it compensates for current methods that mainly investigate global model performance (success rate).

  3. The proposed Ri1A framework is evaluated across multiple model sizes and settings (fixed wall-clock time and fixed optimizer steps).

Critical weaknesses
  1. The geometric model itself is stylized, not a theoretically verified dynamical model. This creates a gap between the mathematical assumptions of the paper and the real RLVR dynamics, and makes it harder for readers to follow.

  2. The ablations are about settings like increment size, leaving the performance gains of key components (e.g., re-anchoring) in Ri1A remain unclear.

  3. Some ablations and hyperparameter sweeps (L477) use only one or two training seeds.

Questions for authors
  1. Can you verify that larger steps indeed lead to larger directional deviation (L200)? This would strengthen the theoretical claims of this paper substantially.

  2. The core claim that smaller increments are better is verified only at 1.7B. Why does this not reproduce the predicted monotonic trend on these other models (non-monotonic at 4B)? Please clarify.

1 comment

As the discussion period draws to a close, we would appreciate any further feedback on our responses and additional experiments. In particular, please let us know which concerns remain unresolved so that we can address them directly.

Reviewer_ufo7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies curriculum design for RL with verifiable rewards, arguing that maximizing the informativeness of individual batches need not produce the best training trajectory. It introduces a stylized Wasserstein transport model and proposes Ri1A, which ranks problems once using base-policy pass rates, trains on progressively admitted subsets, and updates a deferral threshold using the current policy. Experiments report improvements over ODF and AdaCuRL under a shared wall-clock budget, comparable performance to full-schedule DAPO at lower runtime on smaller models, and a performance gain at 8B.

Critical strengths
  1. The motivation is clear. Distinguishing the usefulness of an individual training batch from the quality of the resulting curriculum is a worthwhile question, and the geometric model gives a concrete way to discuss estimation error and incremental updates.

  2. The method is relatively simple and addresses the practical cost of repeatedly estimating problem difficulty. The comparisons against both adaptive curricula and full-schedule DAPO provide useful evidence of its potential.

Critical weaknesses
  1. The ability-anchoring interpretation is not justified by the threshold update.

Equation (7) constructs the next threshold from the current policy’s failure rate on the trained subset, whereas Algorithm 1 applies it to base-policy failure rates on candidate problems. The Rasch assumption preserves problem ordering, but does not make these numerical scales interchangeable. Under the stated model, δi(x)=σ(logit⁡(δ0(x))−(ai−a0)).\delta_i(x) = \sigma\left(\operatorname{logit}(\delta_0(x)) - (a_i - a_0)\right). Moreover, for a fixed probe subset, improving the policy lowers the threshold and makes admission stricter in the base-policy coordinates. This behavior needs an explanation. The rule may still be useful as a heuristic, but Appendix B.3 does not establish that updating it is equivalent to updating ability. A derivation or a more limited interpretation, supported by stage-wise thresholds and deferral-cap binding rates, would clarify what the anchor actually controls.

  1. The incremental benefit over the closest adaptive curriculum remains unclear.

AdaRFT (arXiv:2504.05520) already combines precomputed problem difficulties with a reward-driven scalar target and bounded difficulty updates, without repeatedly rescoring the entire pool. Ri1A also retains a fixed per-problem ordering, so Section 2’s observation about AdaRFT’s fixed ordering does not distinguish the two methods. Ri1A introduces meaningful differences—staged admission, removal of trained examples, and deferral—but the experiments do not establish their benefit over this closely related, low-overhead alternative. The comparisons with ODF and AdaCuRL demonstrate practical gains over those implementations; a comparison with AdaRFT using the same initial difficulty scores, DAPO backbone, and training budget would better establish the incremental contribution.

Questions for authors

See Critical Weaknesses above.

1 comment

Thank you for your time and feedback. We have posted additional results and clarifications addressing your concerns. We would appreciate your assessment of whether these resolve the issues raised, and would be happy to clarify any remaining points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.