- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Paper summary: This paper tries to address the problem of "learning frontier non-stationarity", which is the problem of the RL agents improving the reward consistently and then suddenly dropping and being unable to fully recover. The cause of this problem, as identified by the paper, comes from the master prompt (which is the prompt that results in success in the RL rollout) receives no gradient during the training phase and gradually starts to drift its behavior as long as the model is training. In order to solve this problem, this paper proposes AgentFrontier, which consists of
- Frontier retention: adding an auxiliary loss, which is a next-token prediction LLM loss, for updating the network gradient, circumventing the problem of zero gradient for the master prompt.
2.Frontier Stabilization: Adjust the proportion of the batch such that it has the balancing of master prompt, frontier prompt and hard prompt during the training.
Contribution: This paper's main contributions include:
- Problem identification and analysis: The paper points out and conducts several analyses of the problem of "learning frontier non-stationarity".
- Methods for addressing the problem: This paper proposes Frontier retention and Frontier Stablization to address the problems.
Critical strengths
There are several strengths of this paper:
- Clear and simple problem formulation: The problem this paper tries to address is the degradation of RL rewards during training. The analysis is simple, explaining that successful and easy samples do not have gradient updates; therefore, they can be degraded in later training iterations.
- Experimental results: The experiments strongly suggest a consistent improvement of the proposed method against other baselines, and the method is generalized across different RL algorithms, including GRPO, GSPO,...
Critical weaknesses
There are several weaknesses that need to be addressed:
-
Clarity in the sampling within a batch: Equation 6 introduces the sampling proportions of master, frontier, and unsolved prompts in a training batch. Taking the default and in Table 4, in the late training of the RL, when more prompts become master prompts, the proportion ; therefore, . If we conduct uniform sampling in this case for the batch , on average, we will get the same proportion of master, frontier, and unsolved prompts. Therefore, keeping this sampling mechanism will likely result in the exact phenomenon of uniform sampling, stated in lines 304, 305, and 306. Further analysis is required to clarify this problem.
-
Unclear batch ratio composition: In the ablation study of Figure 4 and the config parameters shown in Table 4, this paper uses the fixed default batch mixtures of 20/60/20 for master, frontier, and unsolved prompts, which makes it confusing when compared to the adaptive sampling strategy proposed in Equation 6.
-
PPO integration: This paper claims the method can be integrated with PPO (Line 107, 108 and 109), but Table 1 does not include the results with PPO.
-
Ablation study: The paper lacks an ablation study of , , , and .
Questions for authors
See the weaknesses section.
3 comments
Response to Reviewer BcNx (2/3)
Thanks for working through the late-stage 80/10/10 example and asking for a PPO endpoint. Thanks also for highlighting the simple analysis connecting sparse reward-gradient support to degradation of successful prompts.
Follow-up on Eq. 6. Realized quotas and PPO endpoints.
R3. We hold retention fixed and change the sampler. Table 1 shows the batches each rule actually produces. Table 2 changes one sampler setting at a time. Table 3 checks the PPO endpoint with the same critic.
Table 1. Fixed versus population-proportional allocation, with identical retention.
| Rule | Late-window realized mastered/frontier/unsolved | Batches with a shortage | CD mean ± sample SD | Cohort retention |
|---|---|---|---|---|
| Retention + proportional quota | 73% / 16% / 11% | 3.4% | 65.50% ± 0.36% | 91.70% |
| Retention + fixed quota | 24% / 55% / 21% | 17.6% | 67.76% ± 0.51% | 94.00% |
Table 2. One-factor sampler sensitivity around the fixed-target reference.
| Factor | Lower setting and CD | Default setting and CD | Higher setting and CD |
|---|---|---|---|
| Mastered threshold | 0.85 → 67.25% | 0.875 → 67.76% | 0.90 → 67.47% |
| Unsolved threshold | 0.10 → 67.40% | 0.125 → 67.76% | 0.15 → 67.52% |
| Target frontier share | 50% → 67.10% | 60% → 67.76% | 70% → 67.41% |
Table 3. PPO integration, five matched training seeds.
| Arm | CD mean ± sample SD | Cohort retention |
|---|---|---|
| Vanilla PPO | 55.56% ± 0.72% | 75.6% |
| PPO + AgentFrontier | 62.30% ± 0.63% | 91.2% |
A batch of 64 shows what 20/60/20 means in practice. Rounding by the largest remainder gives 13 mastered, 38 frontier, and 13 unsolved. If two groups tie we give priority to mastered, then frontier, then unsolved. If a group comes up short we share its empty slots among the groups that still have prompts, in proportion to their target weights. We keep the frontier cap and show any slots still empty. The 70% check allows frontier to take 70%. The pool-based default caps it at 60%. In the PPO comparison we keep the critic and optimizer fixed and add the auxiliary loss.
Revision. Name the two allocation rules beside Eq. 6, give realized quotas beside the composition sweep, and include the matched PPO endpoint.
Response to Reviewer BcNx (3/3)
Thanks for working through the quota arithmetic. Thanks also for highlighting the clear training-instability formulation and the consistent improvements across RL algorithms.
Additional experimental results. What batch does each quota rule actually produce when a group runs short?
R4. We start both rules from the same pool. The rows below show the integer draw counts and where the extra slots go when a group runs out.
Table S1. Next-batch sampler trace at a global size of 64.
| Snapshot | Eligible M / F / U | Proportional draws M / F / U | Fixed-target draws M / F / U | Reallocated fixed slots |
|---|---|---|---|---|
| Target feasible | 110 / 54 / 27 | 37 / 18 / 9 | 13 / 38 / 13 | 0 |
| Frontier shortage | 110 / 19 / 27 | 45 / 8 / 11 | 23 / 19 / 22 | 19 |
| Unsolved shortage | 110 / 54 / 7 | 41 / 20 / 3 | 19 / 38 / 7 | 6 |
Round by largest remainder. The frontier can get at most 38 of 64 slots. Move shortfall slots to eligible groups by the fixed weights and use the stated group order for ties. If frontier is short then mastered gets 10 of the 19 slots and unsolved gets 9. If unsolved is short then mastered gets all six because frontier is at its cap. The pool-based rule uses the available counts and draws without replacement.
Manuscript addition. We’ll use a batch of 64 to show how the targets in Eq. 6 turn into actual draws. The table will show how many prompts each group has, how many we draw without replacement, and which group gets any spare slots. We’ll give the same breakdown next to the PPO critic results.
Response to Reviewer BcNx (1/3)
Thanks for your careful review and the specific questions. Thanks also for highlighting the clear training-instability formulation and the consistent improvements across RL algorithms.
R1. The 20/60/20 test keeps the target mix fixed as prompts move between groups. Eq. 6 is a different sampler. Its fractions depend on the current pool. With the cap inactive and γm=1 it gives the same expected mix as uniform sampling. On CD the fixed mix scores 67.8 and frontier-only sampling scores 64.9.
What is adaptive in the 20/60/20 setting.
One training step with fixed quotas.
When a category is short of prompts, the reported implementation redistributes the shortage while respecting the frontier cap.
Here’s how we round the fixed quotas in the new comparison. Start with ⌊Bwc⌋ slots for each group using w=(0.2,0.6,0.2). Give the spare slots to the groups with the largest fractional remainders.
For the population-proportional policy, wF=min(∣F∣/N,γmax), wM=γm∣M∣/N, and wU=1−wF−wM. With the cap inactive and γm=1, the expected mix equals the current pool composition. The 20/60/20 targets are constant even as prompts move between groups.
Revision. We will name the fixed-target and population-proportional rules beside Eq. 6 and label the composition sweep.
R2. Table 1 reports final scores for GRPO and GSPO. The PPO plots show how rollout outcomes change during training. The matched PPO score comparison below uses the same critic and rollout budget.
PPO integration. The retention loss is added to the inner optimizer's RL objective, while the status estimator reads rollout outcomes. Table 1 reports AgentFrontier endpoints for GRPO and GSPO. The PPO panels diagnose training dynamics, where critic-based advantages can remain nonzero for equal rewards. We will label the PPO panels as outcome-diversity diagnostics in the manuscript. The matched PPO endpoint fixes initialization, critic, prompts, rollouts, and updates, and gives 55.56% CD for PPO and 62.30% with AgentFrontier in the example. At the same retention weight, changing the GSPO sampler from proportional to fixed allocation increases the realized frontier share from 16% to 55% in the late window.
Which settings matter in the reported sweeps. On CD, 20/60/20 scores 67.8, frontier-only sampling scores 64.9, and 40/40/20 scores 66.0. Cache refresh every 25, 50, or 100 updates gives 67.3, 67.8, and 67.0. The composition sweep shows how allocating some updates to the other groups improves over a frontier-only batch.