- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes frontier learning, an open-ended RL post-training framework for LLM reasoning that dynamically adapts the training distribution to the evolving capability of the model. Compared with curriculum learning, which trains on a fixed pool of problems, frontier learning assumes access to a task-specific procedural generator whose configuration parameters define a level space. The method maintains a dynamic buffer of levels, estimates their learning potential using a regret signal derived from existing GRPO rollouts, prioritizes informative frontier levels, and expands the buffer through random exploration and state-dependent mutation. The method is instantiated with GRPO and evaluated on five procedurally generated reasoning tasks spanning puzzle, math, and graph domains. The experiments show consistent improvements over fixed-buffer curriculum and sampling baselines.
Critical strengths
-
The paper addresses an important limitation of RLVR with fixed training data, i.e., group-relative learning signals vanish as model improves.
-
The proposed framework is relatively simple and computationally practical.
-
The empirical results are reasonably broad within the procedural-generation setting.
Critical weaknesses
-
The proposed frontier learning relies heavily on procedural data generation and rule-based or programmatic verification, which somewhat limites the scope of this paper. It remains unexplored whether models trained with frontier learning have good transferability to other domains, especially those without procedural data generation.
-
The allocation of difficulty levels rely on a posthoc empirical success rate, which makes the online generation essentially equivalent to a vast amount of offline generation with posthoc selection. This limits the novelty of the proposed frontier learning from offline curriculum learning.
-
It remains unclear whether the empirical advantage of frontier learning comes from online generation. Finer ablations are required.
-
(Minor) Notations and in Eq (2) are not defined.
Questions for authors
-
Can the models trained with frontier learning transfer across domains? For example, can a model trained on Countdown transfer to Sokoban and other domains without procedural data generation?
-
How much of the improvement comes specifically from online generation? Could the authors compare against a baseline that pre-generates a large and diverse procedural level pool offline and then applies the same regret-based adaptive sampling during training, ideally under a matched rollout and generation budget?
-
In Eq (2), what is the relationship between and ?
4 comments
Gentle Reminder
Dear Reviewer,
As the discussion period is coming to an end, we wanted to follow up and ask whether our response has addressed your concerns. We would be happy to provide any further clarifications if needed.
Thank you in advance for your time and consideration.
Thank you for the clarifications. I appreciate the distinction between online problem generation and adaptive level discovery.
However, the clarified scope makes the contribution relatively narrow, with the main novelty lying in combining existing sampling and mutation techniques for procedurally generated tasks. Moreover, the incremental benefit of adaptive level discovery over a sufficiently large fixed level pool remains insufficiently validated.
I maintain my rating of 2.
Thanks for the reply. We disagree with the characterization of our contribution as narrow or merely combining existing techniques. As such there are no existing works that have a regret-based priority signal that identifies and prioritizes frontier levels using only the GRPO rollouts already collected and uses random exploration with state-dependent mutation to continually scout for new frontier levels suited to the model’s evolving capability during RL post-training of LLMs.
Moreover, the Domain Randomization baseline already samples fresh levels at every training step from the full configuration space of each task, providing access to a much larger and more diverse set of levels than any practical pre-generated fixed pool. Our results therefore demonstrate that access to a larger pool alone is insufficient, and that adaptive frontier discovery is crucial.
We hope these points will be considered in the final assessment.
We thank the reviewer for recognising frontier learning's simplicity, computational practicality as well as the broadness of our evaluation. We address each concern below.
W1 / Q1 — Scope and cross-domain transfer
Scope. Frontier learning is explicitly framed for tasks that come with a procedural generator and a verifiable reward. This is the setting of the paper, and we already mention it throughout, including in Section 6. Given the growing use of such generators in fields like RLVR or Recursive Self Improvement, we consider this setting increasingly important.
Transfer. This is an interesting question, but it is not what frontier learning is designed for. The method makes RL post-training on a given task more efficient: it keeps the GRPO signal alive by training on the levels where the current policy is still learning. Accordingly, our objective (Eq. 3) and our evaluation measure performance on unseen configurations of the same task. The paper makes no claim about transfer across domains, and nothing in the method is designed to produce it. Whether training on Countdown helps on Sokoban depends on the relationship between the two tasks, not on how the Countdown levels were scheduled.
W2 / W3 / Q2 — Online vs offline generation
We note that online generation is not the contribution. Every method in the paper, baselines included, generates problems online at every step. Since the generator is fixed, pre-generating a level's problems instead would change nothing in training. It would only be less efficient, because it is not known in advance which levels of our space L will be used or how many problems each will need. What distinguishes the methods is which levels they train on.
Hence, given that generating problems online or offline is equivalent up to efficiency, the baseline requested in Q2 is already covered in the paper. PLR+Explore applies our regret prioritisation to up to 100 levels drawn uniformly from L. It reaches 33.5 ± 1.8 on Dice, against 71.8 ± 6.2 for frontier learning under matched rollouts and optimizer updates.
Frontier learning treats the generator's configuration space as a search space with no assumed difficulty ordering. It keeps training at the capability frontier through (i) a regret signal computed from the GRPO rollouts already collected, (ii) a dynamic level buffer prioritised by this regret, and (iii) exploration and state-dependent mutation, which admit new frontier levels to the buffer as the policy improves. Post-hoc selection over L is infeasible: it contains far more configurations than a run can evaluate, and their informativeness changes as the policy improves. Additionally, as said above, one does not know in advance how many problems will be needed for each level. Frontier learning makes this search efficient, reusing the training rollouts for regret and focusing mutation on currently informative levels. Unlike offline curriculum learning, neither the levels nor their order is fixed before training. Our ablations show how each of our designed components concretely add up to achieve substantial gains over the baselines.
W4 / Q3 — Notation in Eq. 2
We thank the reviewer for spotting the notation inconsistencies.
ri,t=πθ(yi,t∣x,yi,<t)/πθold(yi,t∣x,yi,<t) is the token-level importance ratio and A^i,t=Ai is the advantage of response i (Eq. 1), which is shared by all its tokens.
We updated the revision renaming the ratio to wi,t to avoid the clash with the reward ri and using Ai consistently to avoid any confusion.
We hope these corrections and clarifications resolve the expressed concerns, and would be grateful if the reviewer would reconsider their score.