acceptodds
Under review as a conference paper at ICLR 2027

Frontier Learning: Training LLM Reasoners at the Edge of Capability

Abstract

Reinforcement Learning-based post-training of Large Language Models (LLM) has been successfully applied to improve their reasoning capabilities. Existing pipelines primarily finetune LLMs on a fixed pool of problems specified prior to training using the GRPO loss. This is fundamentally limiting, as learning signal arises only when policy rollouts mix successes and failures, causing the useful portion of any fixed pool to quickly become stale as the model improves. To address this, we propose frontier learning, an open-ended post-training approach in which procedural generators are used online to continually produce informative training problems. It treats the generator's task-specific parameters as a search space and uses a regret signal to prioritize and explore frontier difficulty levels in order to focus training at the edge of the model's evolving reasoning capabilities. Across several reasoning tasks and model families, our approach consistently achieves higher relative gains over fixed-pool baselines, demonstrating that effective post-training requires not only selecting useful problems, but continually generating them at the edge of capability.

Primary area:
reinforcement learning
Keywords:
Generative Models, Reasoning, online post-training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CyY1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes frontier learning, an open-ended RL post-training framework for LLM reasoning that dynamically adapts the training distribution to the evolving capability of the model. Compared with curriculum learning, which trains on a fixed pool of problems, frontier learning assumes access to a task-specific procedural generator whose configuration parameters define a level space. The method maintains a dynamic buffer of levels, estimates their learning potential using a regret signal derived from existing GRPO rollouts, prioritizes informative frontier levels, and expands the buffer through random exploration and state-dependent mutation. The method is instantiated with GRPO and evaluated on five procedurally generated reasoning tasks spanning puzzle, math, and graph domains. The experiments show consistent improvements over fixed-buffer curriculum and sampling baselines.

Critical strengths
  • The paper addresses an important limitation of RLVR with fixed training data, i.e., group-relative learning signals vanish as model improves.

  • The proposed framework is relatively simple and computationally practical.

  • The empirical results are reasonably broad within the procedural-generation setting.

Critical weaknesses
  • The proposed frontier learning relies heavily on procedural data generation and rule-based or programmatic verification, which somewhat limites the scope of this paper. It remains unexplored whether models trained with frontier learning have good transferability to other domains, especially those without procedural data generation.

  • The allocation of difficulty levels rely on a posthoc empirical success rate, which makes the online generation essentially equivalent to a vast amount of offline generation with posthoc selection. This limits the novelty of the proposed frontier learning from offline curriculum learning.

  • It remains unclear whether the empirical advantage of frontier learning comes from online generation. Finer ablations are required.

  • (Minor) Notations ri,tr_{i,t} and A^i,t\hat{A}_{i,t} in Eq (2) are not defined.

Questions for authors
  • Can the models trained with frontier learning transfer across domains? For example, can a model trained on Countdown transfer to Sokoban and other domains without procedural data generation?

  • How much of the improvement comes specifically from online generation? Could the authors compare against a baseline that pre-generates a large and diverse procedural level pool offline and then applies the same regret-based adaptive sampling during training, ideally under a matched rollout and generation budget?

  • In Eq (2), what is the relationship between AiA_i and A^i,t\hat{A}_{i,t}?

4 comments

We thank the reviewer for recognising frontier learning's simplicity, computational practicality as well as the broadness of our evaluation. We address each concern below.

W1 / Q1 — Scope and cross-domain transfer

Scope. Frontier learning is explicitly framed for tasks that come with a procedural generator and a verifiable reward. This is the setting of the paper, and we already mention it throughout, including in Section 6. Given the growing use of such generators in fields like RLVR or Recursive Self Improvement, we consider this setting increasingly important.

Transfer. This is an interesting question, but it is not what frontier learning is designed for. The method makes RL post-training on a given task more efficient: it keeps the GRPO signal alive by training on the levels where the current policy is still learning. Accordingly, our objective (Eq. 3) and our evaluation measure performance on unseen configurations of the same task. The paper makes no claim about transfer across domains, and nothing in the method is designed to produce it. Whether training on Countdown helps on Sokoban depends on the relationship between the two tasks, not on how the Countdown levels were scheduled.

W2 / W3 / Q2 — Online vs offline generation

We note that online generation is not the contribution. Every method in the paper, baselines included, generates problems online at every step. Since the generator is fixed, pre-generating a level's problems instead would change nothing in training. It would only be less efficient, because it is not known in advance which levels of our space L\mathcal{L} will be used or how many problems each will need. What distinguishes the methods is which levels they train on.

Hence, given that generating problems online or offline is equivalent up to efficiency, the baseline requested in Q2 is already covered in the paper. PLR+Explore applies our regret prioritisation to up to 100 levels drawn uniformly from L\mathcal{L}. It reaches 33.5 ± 1.8 on Dice, against 71.8 ± 6.2 for frontier learning under matched rollouts and optimizer updates.

Frontier learning treats the generator's configuration space as a search space with no assumed difficulty ordering. It keeps training at the capability frontier through (i) a regret signal computed from the GRPO rollouts already collected, (ii) a dynamic level buffer prioritised by this regret, and (iii) exploration and state-dependent mutation, which admit new frontier levels to the buffer as the policy improves. Post-hoc selection over L\mathcal{L} is infeasible: it contains far more configurations than a run can evaluate, and their informativeness changes as the policy improves. Additionally, as said above, one does not know in advance how many problems will be needed for each level. Frontier learning makes this search efficient, reusing the training rollouts for regret and focusing mutation on currently informative levels. Unlike offline curriculum learning, neither the levels nor their order is fixed before training. Our ablations show how each of our designed components concretely add up to achieve substantial gains over the baselines.

W4 / Q3 — Notation in Eq. 2

We thank the reviewer for spotting the notation inconsistencies.

ri,t=πθ(yi,t∣x,yi,<t)/πθold(yi,t∣x,yi,<t)r_{i,t} = \pi_\theta(y_{i,t} \mid x, y_{i,<t}) / \pi_{\theta_{\text{old}}}(y_{i,t} \mid x, y_{i,<t}) is the token-level importance ratio and A^i,t=Ai\hat{A}_{i,t} = A_i is the advantage of response ii (Eq. 1), which is shared by all its tokens.

We updated the revision renaming the ratio to wi,tw_{i,t} to avoid the clash with the reward rir_i and using AiA_i consistently to avoid any confusion.

We hope these corrections and clarifications resolve the expressed concerns, and would be grateful if the reviewer would reconsider their score.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer,

As the discussion period is coming to an end, we wanted to follow up and ask whether our response has addressed your concerns. We would be happy to provide any further clarifications if needed.

Thank you in advance for your time and consideration.

Thank you for the clarifications. I appreciate the distinction between online problem generation and adaptive level discovery.

However, the clarified scope makes the contribution relatively narrow, with the main novelty lying in combining existing sampling and mutation techniques for procedurally generated tasks. Moreover, the incremental benefit of adaptive level discovery over a sufficiently large fixed level pool remains insufficiently validated.

I maintain my rating of 2.

Thanks for the reply. We disagree with the characterization of our contribution as narrow or merely combining existing techniques. As such there are no existing works that have a regret-based priority signal that identifies and prioritizes frontier levels using only the GRPO rollouts already collected and uses random exploration with state-dependent mutation to continually scout for new frontier levels suited to the model’s evolving capability during RL post-training of LLMs.

Moreover, the Domain Randomization baseline already samples fresh levels at every training step from the full configuration space of each task, providing access to a much larger and more diverse set of levels than any practical pre-generated fixed pool. Our results therefore demonstrate that access to a larger pool alone is insufficient, and that adaptive frontier discovery is crucial.

We hope these points will be considered in the final assessment.

Reviewer_eRGCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper uses procedural generators to produce reasoning problems near the model’s evolving capability frontier during RL. It uses rollout outcomes to prioritize informative configurations and introduces new configurations through exploration and mutation. The trained models are evaluated on held-out configurations of the same tasks.

Critical strengths
  • The method reuses training rollouts to identify problems that the model can sometimes solve but has not yet mastered. Combining this signal with configuration mutation allows the training distribution to adapt as the model improves, without requiring a separate teacher to generate problems.

  • The improvements on Dice and Largest Island extend to harder configurations and continue after the baselines plateau. The gains on Dice also remain under a matched wall-clock budget, supporting the effectiveness of adaptive generation beyond simply spending more compute.

Critical weaknesses
  • The method closely follows ACCEL’s replay-and-mutation principle, while the fixed-buffer baselines retain only eight initial configurations. What additional benefits do the proposed regret signal and state-dependent mutation provide? Could the authors compare the method against an ACCEL-style baseline under matched training budgets? Would a larger, well-covered fixed pool with adaptive sampling substantially reduce the gap?

  • Evaluation relies on a small set of predefined anchor configurations, whereas the stated objective averages performance uniformly over the full level space. Do the gains persist on independently sampled configurations and attribute combinations outside the predefined difficulty ladders? Could the authors provide broader configuration-level evaluations to establish that the improvements extend across the intended search space?

  • Important implementation details remain underspecified, including the parameter ranges, validity constraints, mutation step sizes for each task, and handling of out-of-range or invalid configurations. How sensitive are the gains to these choices, and do all methods operate over the same effective search space?

  • Although the evaluation covers several model families, all models are limited to 3B–7B parameters. Do the gains persist with larger, more capable reasoning models? When a stronger model can already solve many generated problems, can the method still discover informative configurations and sustain further improvement?

Questions for authors

Same as the “Critical Weaknesses” section.

3 comments

We thank the reviewer for recognising our method’s ability to find informative configurations from GRPO's own rollouts, without a separate teacher, and highlighting that our gains on Dice and Largest Island hold under matched wall-clock time. We address each concern below.

W1 — Relation to ACCEL and to larger fixed pools.

We note that ACCEL is an unsupervised environment design approach for training RL agents from scratch with PPO on navigation and continuous-control environments. It was published more than 4 years ago and is not applicable to RL post-training of LLMs for reasoning. We cite it as a related curriculum-based approach, but our method is not a direct adaptation given the substantial differences in setting and methodology.

​​In contrast, frontier learning targets a core limitation of GRPO: it provides no learning signal on problems whose sampled responses all receive the same reward. Our design keeps the training distribution where this signal exists. To answer the reviewer’s concern about what the regret signal and state-dependent mutation add, we isolate them with two new ablations under the same training budget as all other methods (Dice, 500 steps, 4 seeds):

  • SEC+Explore+Mut replaces our regret with the SEC score as the priority signal. Exploration and state-dependent mutation are unchanged.
  • Frontier learning, flat mutation mutates every replayed level with the same probability regardless of its success rate. Our regret and exploration are unchanged.

Both are already included in Table 2 of the revised paper and summarized here for convenience:

Change from frontier learningDice (%)
None (frontier learning)71.8 ± 6.2
Regret → SEC score (new)61.4 ± 6.0
State-dependent mutation → flat mutation (new)61.7 ± 7.7
State-dependent → flat mutation, and regret → uniform sampling36.9 ± 2.9

With mutation held flat, regret-guided replay alone raises accuracy from 36.9 to 61.7, and state-dependent mutation raises it further to 71.8. With exploration and state-dependent mutation unchanged, replacing our regret with the SEC score lowers accuracy to 61.4. Both components contribute.

On larger fixed pools: Table 2 already contains them. Unif+Explore and PLR+Explore grow the buffer to 100 levels drawn uniformly from the full space L, which is a larger and broader pool than the 8 seed levels. They reach only 30.8 and 33.5, on par with the fixed-pool baselines. Broader coverage, with or without adaptive sampling, does not close the gap. Regret-guided mutation does: PLR+Mut reaches 62.5.

W2 — Evaluation set vs. Eq. 3.

Eq. 3 reflects the broad objective one would ideally use to assess the trained model’s capability. However, evaluating on all levels is computationally infeasible as some tasks can have 10610^6 levels (Countdown and Largest Island). In practice, we estimate it on pre-defined, held-out anchor levels that span the same attribute ranges as the seeding grid for the level buffer (Sec. 4). However, these levels are excluded from seeding, exploration, and mutation, so no method ever trains on them (see Sec. 5).

Moreover, we point out that these anchor levels also cover regions where our method has minimal advantage, (Hard for Countdown and Decimal Arithmetic, Fig. 7; Easy on Largest Island, Fig. 3) as well as regions where we have large gains (Hard on Dice and Largest Island, Medium on Countdown and Easy on Decimal Arithmetic).

W3 — Level-space details.

The revision adds a table to Appendix D listing each attribute's range, type and mutation step size for all the tasks. A configuration is valid if the Reasoning Gym generator can produce a problem for it. Configurations it cannot produce problems for are discarded and never enter the buffer. All methods use the same space L: seeding, Domain Randomization, exploration and mutation draw from identical ranges under identical validity checks.

W4 — Larger models.

Two of our settings already start from post-trained models with strong capabilities, Olmo3-7B-Instruct and Qwen3-4B with thinking enabled, and frontier learning remains best in both: 73.7 vs. 52.0 on Largest Island and 46.3 vs. 43.3 on Sokoban. We have no reason to believe this would not hold for larger models. We trained up to the limit of our compute budget, with more of 6k GPU hours (H200/GH200) employed to get the presented results, corresponding to a cost of around $25k on Lambda compute provider. We leave larger scales to future work.

We hope these clarifications address the reviewer's concerns, and would be grateful if the reviewer would reconsider their score.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer,

As the discussion period is coming to an end, we wanted to follow up and ask whether our response has addressed your concerns. We would be happy to provide any further clarifications if needed.

Thank you in advance for your time and consideration.

Thank you for the response and clarifications. After considering the authors’ replies, I have decided to maintain my original score.

Reviewer_MtY9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Frontier Learning, an online curriculum generation framework for RLVR. Rather than training on a fixed set of problem configurations, the method searches the configuration space of procedural reasoning generators to identify and produce problems near the model's evolving capability frontier.

The approach combines a dynamic level buffer, a regret-based priority measure estimated from GRPO rollout outcomes, random exploration, and state-dependent mutation of existing configurations. The regret signal favors configurations containing problems that can be solved by some, but not all, policy rollouts.

The authors evaluate the approach on five procedurally generated reasoning tasks spanning arithmetic, puzzles, and graph reasoning, using models from the Qwen3, Llama, and OLMo families. The strongest reported results are obtained under extended training, including 71.8% versus 33.3% accuracy on Dice and 73.7% versus 52.0% on Largest Island. Ablations suggest that regret-guided mutation is particularly important for sustained improvements.

Critical strengths
  1. The paper demonstrates substantial improvements over fixed-level sampling and adaptive curriculum baselines, particularly under longer training horizons. On Dice, Frontier Learning achieves 71.8% accuracy compared with 33.3% for the strongest standard baseline, while on Largest Island it achieves 73.7% compared with 52.0%. The continued improvements after competing methods plateau provide meaningful evidence that expanding the set of training configurations can be beneficial.

  2. The ablation results help identify the main source of improvement. On Dice, PLR+Explore achieves 33.5%, whereas PLR+Mut reaches 62.5%, and the full method reaches 71.8%. This supports the importance of mutation-driven configuration expansion combined with regret-based prioritization, rather than random exploration or replay alone.

Critical weaknesses
  1. The central framework closely resembles ACCEL [1], which already combines regret-prioritized level replay with local mutation to generate increasingly challenging environments near an agent's capability frontier. The present work adapts this paradigm to LLM post-training using GRPO-derived binary-reward statistics and state-dependent mutation probabilities. Although this adaptation is potentially useful, the distinction between the proposed algorithm and an ACCEL-style adaptation to procedural RLVR is insufficiently established. The existing ablations demonstrate the importance of mutation and exploration, but do not clearly isolate the added value of the new regret formulation and mutation policy relative to this particularly close prior framework. Comparison with simpler adaptive difficulty-generation approaches would also help establish the necessity of the proposed mechanism.

  2. The Sokoban results in Figure 7 do not reconcile with the aggregate accuracies in Table 1. According to Appendix C, the Sokoban evaluation uses three Easy, two Medium, and two Hard levels, with the same number of evaluation problems per level. Consequently, the aggregate accuracy should equal the level-count-weighted average of the reported difficulty-bin accuracies. Using Figure 7 gives approximately 46.3% for Frontier Learning, 43.3% for PLR, and 38.4% for SEC, whereas Table 1 reports 45.2%, 41.1%, and 34.3%, respectively. These differences, particularly the approximately 4.1-point discrepancy for SEC, cannot be explained by numerical rounding. The inconsistency also changes the apparent magnitude of the improvement over the strongest baseline. Because these results contribute to the paper's empirical claims, the discrepancy should be resolved before the conclusions can be fully assessed.

[1] Parker-Holder J, Jiang M, Dennis M, et al. Evolving curricula with regret-based environment design[C]//International Conference on Machine Learning. PMLR, 2022: 17473-17498.

Questions for authors
  1. Could the authors reconcile the Sokoban accuracies in Table 1 with the per-difficulty results in Figure 7?

  2. What is the incremental contribution over an ACCEL-style adaptation to the same procedural RLVR setting?

2 comments

We thank the reviewer for recognizing our method’s strong performance over baselines and highlighting the importance of our ablations. Both concerns are addressed below.

Q1 — Sokoban: Table 1 vs. Figure 7

We thank the reviewer for the careful check of our numbers: Table 1's Sokoban column was computed when only 2 of the 4 seeds had finished. Figure 7 was later updated with all 4 seeds while Table 1 was not. We updated the revision with the correct values, which we also report here for convenience:

MethodOld Table 1 (2 seeds)Corrected (4 seeds)
Domain Rand.37.6 ± 2.637.4 ± 1.7
Uniform39.5 ± 1.339.8 ± 2.6
SEC34.3 ± 6.638.4 ± 3.6
PLR41.1 ± 3.943.3 ± 2.1
Frontier learning45.2 ± 1.946.3 ± 1.4

The ranking is unchanged: frontier learning remains best, with a +6.9% relative gain over PLR. We re-audited every other table and no other entry is affected. Moreover, in case of sokoban the main difference among methods is observed in the easy category as the model's capability primarily expands in this region within 200 training steps.

MethodAccuracy
Domain Rand.51.4 ± 3.1
Uniform52.8 ± 5.0
SEC53.7 ± 5.2
PLR59.7 ± 3.3
Frontier learning65.6 ± 2.1

Q2 — Contribution over ACCEL

We note that ACCEL is an unsupervised environment design approach for training RL agents from scratch with PPO on navigation and continuous-control environments. It was published more than 4 years ago and is not applicable to RL post-training of LLMs for reasoning. We cite it as a related curriculum-based approach, but our method is not a direct adaptation given the substantial differences in setting and methodology.

In contrast, our frontier learning targets a core limitation of GRPO: it provides no learning signal on problems whose sampled responses all receive the same reward. Our design keeps the training distribution where this signal exists. Upon the request of the reviewer, we further isolate the contribution of our regret and mutation policy through two new ablations in the Dice, 500 steps setting:

  • SEC+Explore+Mut replaces our regret with the SEC score as the priority signal, keeping exploration and state-dependent mutation unchanged.
  • Frontier learning, flat mutation mutates every replayed level with the same probability, regardless of its success rate, keeping our regret and exploration unchanged.

Both are already included in Table 2 of the revised paper and summarized here for convenience:

Change from frontier learningDice (%)
None (frontier learning)71.8 ± 6.2
Regret → SEC score (new)61.4 ± 6.0
State-dependent mutation → flat mutation (new)61.7 ± 7.7
State-dependent → flat mutation, and regret → uniform sampling36.9 ± 2.9

With mutation held flat, our regret-guided replay alone raises accuracy from 36.9 to 61.7. Our state-dependent mutation raises it further to 71.8. Additionally, keeping the same exploration and state-dependent mutation policies and replacing our regret with the SEC score lowers the accuracy to 61.4.

This shows how each of our designed components concretely adds up to achieve substantial gains over the baselines.

We hope these corrections and clarifications resolve both concerns, and would be grateful if the reviewer would reconsider their score.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer,

As the discussion period is coming to an end, we wanted to follow up and ask whether our response has addressed your concerns. We would be happy to provide any further clarifications if needed.

Thank you in advance for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.