- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper argues that the observation horizon (L) and the rollout horizon (H) do not need to be the same in RLVR. The motivating study on Qwen3-4B-Base/DAPO-Math-17k reveals that (1) higher rollout horizons raise cost and instability without better pass@1, and (2) continuing 2k-truncated responses to 8k turns up to 33.56% of incorrect boundary responses to correct ones. The proposed NCBR continues H-truncated responses up to L under same policy, replaces its reward with the outcome at L, and updates only the original H tokens. Experiments on Qwen3-4B/8B-Base with GRPO and DAPO report the performance gains on AIME24/25 and AMC23. The paper also reports that NCBR beats full-8k training with 54.5% fewer GPU-hours.
Critical strengths
- Informative diagnostics. The boundary analysis (Table 1: 0$\rightarrow$1 transitions) and the prefix probes (Figure 3b, 3c) give concrete evidence that H-truncated responses are often mislabeled. This evidence is useful beyond this paper.
- Favorable cost-performance trade-off. H=2k/L=8k NCBR outperforms full-L on all three benchmarks with 54.5% fewer GPU-hours and 62.4% fewer actor-update tokens (Table 3). Continuation tokens are never trained on, which keeps the update cost at H=2k level.
Critical weaknesses
W1. The empirical evidence does not support the central claims.
Both the motivating study in Section 4 and the main results (Tables 2 and 3) evaluate at one fixed checkpoint on 30-40 problem benchmarks. Several specific problems:
- In Figure 2, the H=8k baseline looks collapsed, not merely "less stable". Figure 6 shows its policy entropy climbing from ~0.5 to >2 after ~180 steps with repeated gradient spikes. H=4k baseline also spikes at ~290 steps, right before the step-300 evaluation. Hyperparameters tuned for H=2k are reused at 4k/8k, so Finding 1 cannot be drawn from an untuned, single run that diverged.
- The overlong buffer width and 4k/8k hyperparameters are not stated anywhere in the paper.
- 47.6% of the H=8k model's evaluation responses are truncated at 8k (Figure 1.b), so its pass@1 is a lower bound that the H=2k model is not subject to. Results at a non-truncating budget (e.g. H_eval=16k) are needed.
- Only three benchmarks of about 30-40 problems each were used, and seeds and CIs are not reported. Other larger-size benchmarks such as MATH500, Minerva, and OlympiadBench are absent.
- Step 300 is an arbitrary cut. There is no process of selecting the best checkpoint based on the validation set. I believe that if a checkpoint represents the best performance (even if it occurred before one epoch), it should be selected. (e.g., For the H=8k DAPO baseline, if the model in fact collapses around step 150, you should choose the checkpoint from step ~140 as the final model.)
W2. Baselines are limited to vanilla GRPO and DAPO.
- DAPO's 'hard' overlong filtering (masking truncated responses, not 'soft' overlong punishment) is mentioned in the introduction but not compared. It costs nothing, whereas NCBR roughly doubles training cost. I think this is the minimum bar.
- PPPO is mentioned in the Related Work as the closest method, yet never compared. Same for VinePPO.
- The partial rollouts (Kimi K1.5) or horizon curriculum (DeepScaleR or Skywork-OR1) should be compared, especially in the cost claim since they are cheaper alternatives.
- In the supplementary materials, GSPO recipe exists, but the paper does not report GSPO results.
Questions for authors
Q1. Please clarify how the evaluated checkpoint was selected. Is there a specific reason for using the step-300 checkpoint without any validation-based selection? (Note: I am not asking for the separation of a validation set and a test set. It is perfectly fine to use AMC23, AIME24, and AIME25 here. For example, as I mentioned in W1, if a checkpoint from before step 300 yields better performance on amc23/aime24/aime25, shouldn't that checkpoint be chosen instead?)
Q2. In the motivating experiment, do the 4k/8k conclusions hold for the best pre-collapse checkpoint selected on validation?
Q3. Please report pass@1 and truncation rates at a non-truncating budget (e.g., H_eval=16k), seeds/CIs, and more benchmarks including MATH500, Minerva Math, and OlympiadBench.
Q4. How does NCBR compare with DAPO’s overlong filtering, PPPO, partial rollouts (Kimi K1.5), and horizon curricula (DeepScaleR, Skywork-OR1) in both performance and GPU-hour cost?
Q5. Why are the GSPO results omitted, given that a GSPO recipe is included in the supplementary material?
Q6. Please report, over training, (a) the fraction of continuations that also hit L, (b) the 0$\rightarrow$0 transitions split into "terminated wrong" and "truncated at L", and (c) the dynamics beyond 300 steps. Does the L-cap rate stay bounded once the H cap-hit ratio reaches ~40%?
[Minor Suggestions/Questions]
S1. Please state the model precisely. The supplementary and the Table 2 base-row numbers indicate Qwen3-4B-Base, not Qwen3-4B. Is the 8B model also the Qwen3-8B-Base, not Qwen3-8B?
S2. What is the difference between Table 1 and Table 6? It seems that Table 6 is dupilcate, which is unnecessary.
S3. Without Figure 6, the counts in Table 1 are hard to interpret: as training goes, the number of truncated responses goes 9.2k$\rightarrow$0.8k$\rightarrow$11.3k$\rightarrow$0.7k. Since the populations differ so much, the recovery rate trend across steps is not meaningful (the paper does not overclaim this, but the table invites it). One recommended option is presenting step 0 and step 200, which have enough samples, as the key points.
S4. Typo in the third line from the bottom of the Introduction: "DAP".
S5. What is the temperature during evaluation? The supplementary suggests 1.0, as in training. Please state it in the paper.
11 comments
Thank you for your follow-up and for clarifying your main concerns. We appreciate your patience as we conducted additional experiments under computational constraints. Below, we address the points for which results are currently available. Below, we address the points for which results are currently available.
Additional Evaluation Benchmarks and Budgets (Key Point 2 and Q3)
We evaluated the Step-300 checkpoints on AIME'24, AIME'25, AMC'23, and MATH500 under two evaluation budgets: 8K (8192 tokens) and 16K (16384 tokens). All values below are Pass@1 (%).
Evaluation budget: H_eval = 8K (8192 tokens).
| Training configuration | AIME24 | AIME25 | AMC23 | MATH500 |
|---|---|---|---|---|
| DAPO H=2K | 10.52 | 11.46 | 61.64 | 56.06 |
| DAPO H=4K | 15.83 | 16.25 | 56.25 | 54.08 |
| DAPO H=8K | 9.90 | 11.35 | 58.28 | 55.36 |
| NCBR H=2K L=8K | 16.56 | 14.48 | 69.84 | 59.03 |
Evaluation budget: H_eval = 16K (16384 tokens).
| Training configuration | AIME24 | AIME25 | AMC23 | MATH500 |
|---|---|---|---|---|
| DAPO H=2K | 11.15 | 10.83 | 60.70 | 56.01 |
| DAPO H=4K | 15.42 | 16.15 | 56.80 | 54.01 |
| DAPO H=8K | 8.96 | 11.56 | 56.72 | 55.27 |
| NCBR H=2K L=8K | 15.52 | 13.75 | 71.25 | 59.18 |
NCBR achieves the highest Pass@1 among these configurations on AIME24, AMC23, and MATH500 at both evaluation budgets, while DAPO H=4K achieves the highest Pass@1 on AIME25. On MATH500, NCBR achieves 59.03% at 8K and 59.18% at 16K, compared with 56.06% and 56.01% for DAPO H=2K, respectively.
Comparison with other baselines (Q4).
Comparison with Hard Overlong Filtering
Thank you for suggesting this additional baseline. We trained DAPO with hard overlong filtering on Qwen3-4B-Base for 300 steps, using a 2K training horizon. Hard filtering masks the actor loss for boundary-truncated responses while retaining DAPO's dynamic sampling and group-relative advantage computation.
All methods were evaluated under the same protocol described in the paper, with an 8K evaluation budget.
| Method | AIME'24 P@1 / P@32 | AIME'25 P@1 / P@32 | AMC'23 P@1 / P@32 |
|---|---|---|---|
| Vanilla DAPO 2K | 10.52 / 33.33 | 11.46 / 36.67 | 61.64 / 85.00 |
| DAPO Hard-only 2K | 17.29 / 43.33 | 8.54 / 30.00 | 62.42 / 95.00 |
| DAPO + NCBR 2K→8K | 16.56 / 46.67 | 14.48 / 40.00 | 69.84 / 95.00 |
Hard filtering improves performance on some benchmarks, notably AIME'24 Pass@1. However, simply masking the actor loss of truncated responses does not consistently reproduce NCBR's downstream performance across the three benchmarks. NCBR achieves higher Pass@1 on AIME'25 and AMC'23, while matching or improving Pass@32 on all three benchmarks.
Comparison with PPPO
Response to the comparison with PPPO and incremental novelty
We thank the reviewer for highlighting PPPO and agree that continuation-based prefix evaluation and suffix-gradient masking are not unique to NCBR. To strengthen the empirical evidence, we implemented PPPO with DAPO dynamic sampling and conducted a compute-matched comparison on Qwen3-4B-Base using 16 A100 GPUs. PPPO at Step 20 consumed 641.94 GPU-hours, compared with 628.65 GPU-hours for NCBR at Step 300.
Computational cost analysis. The main computational overhead of PPPO arises from its multiple-continuation sampling. Under our configuration, each problem generates 8 original responses, each spawning 8 independently sampled continuations, resulting in 64 additional continuations per problem. With 768 candidate problems per training step, PPPO generates 6,144 original responses and 49,152 continuations. In contrast, NCBR continues only boundary-truncated responses, with approximately 615 continuations per step in our experiments.
Consequently, PPPO generates approximately 80 times more continuations per step than NCBR. Its continuation stage consumes 600.41 GPU-hours, accounting for 94.26% of its measured training-step time, whereas actor updates account for only 1.51%. For NCBR, continuation accounts for 45.07% of training-step time. The average training-step duration is 119.43 minutes for PPPO versus 7.63 minutes for NCBR.
Under the same 8K evaluation horizon and 32 samples per problem, the results are:
| Method (GPU-h) | AIME24 P@1 | P@32 | AIME25 P@1 | P@32 | AMC23 P@1 | P@32 |
|---|---|---|---|---|---|---|
| PPPO (641.94) | 9.90 | 36.67 | 7.81 | 36.67 | 39.61 | 90.00 |
| NCBR (628.65) | 16.56 | 46.67 | 14.48 | 40.00 | 69.84 | 95.00 |
All results are percentages. Pass@1 is estimated by the average accuracy across 32 samples per problem.
Under comparable end-to-end GPU-hour budgets, NCBR achieves higher Pass@1 and Pass@32 on all three benchmarks, supporting the practical effectiveness of selective boundary continuation. We have incorporated this comparison into Section 6.3 (Method Comparison) of the revised manuscript.
Continuation truncation and training dynamics (Q6)
We thank the reviewer for requesting a more detailed characterization of continuation behavior. We analyzed the per-step training logs of Qwen3-4B-Base with DAPO+NCBR (, ) over Steps 1–300.
(a) Continuations reaching the observation horizon.
We define the conditional L-cap rate as the number of continuations truncated at divided by the number of original responses that strictly hit .
| Training steps | H-cap continuations | L-cap continuations | Conditional L-cap rate |
|---|---|---|---|
| 1–100 | 29,999 | 2,011 | 6.70% |
| 101–200 | 8,170 | 71 | 0.87% |
| 201–300 | 146,206 | 2,387 | 1.63% |
| 251–300 | 89,191 | 1,206 | 1.35% |
Although the H-cap rate increases substantially during later training, the conditional L-cap rate remains low in the observed period.
In particular, at Step 264, the actor-batch H-cap rate reaches 44.24%, while only 40 of 2,092 continuations (1.91%) reach . Across the five steps where the actor-batch H-cap rate exceeds 40%, only 170 of 10,122 continuations (1.68%) reach .
These observations indicate that provides sufficient observation length for the vast majority of boundary-truncated responses during the first 300 training steps.
(b) Decomposition of 0→0 transitions.
We agree that 0→0 transitions should distinguish naturally terminated incorrect responses from responses truncated again at .
The existing aggregate logs record both the total number of 0→0 transitions and the total number of L-cap events, but do not retain their exact intersection.
Nevertheless, they provide a conservative bound. Across Steps 201–300, there are 104,615 candidate-level 0→0 transitions and only 2,387 L-cap events. Therefore, at least 102,228 (97.72%) of these 0→0 transitions correspond to naturally terminated incorrect responses.
At Step 300 alone, at least 1,289 of 1,295 (99.54%) candidate-level 0→0 transitions are naturally terminated errors.
S1. Clarification of Model Initialization
Thank you for pointing this out. We confirm that all experiments use Qwen3-4B-Base and Qwen3-8B-Base as the initialization models. We have clarified in the Introduction that Qwen3-4B and Qwen3-8B are used as shorthand throughout the paper, and specified the full model names in Appendix A.1.
S2 & S3. Boundary-Recovery Table Revision and Interpretation
Thank you for the helpful suggestions. We have revised Table 1 in Section 4.1 to highlight Steps 0 and 200, while retaining the complete checkpoint-wise results in Appendix B.1, with explicit recovery denominators and clarification that the rates should not be interpreted as a monotonic training trend, thereby eliminating the redundancy between the two tables.
S4. Correction of Typographical Error
Thank you for pointing out this typo. We have corrected “DAP” to “DAPO” in the Introduction.
S5. Clarification of Evaluation Settings
Thank you for the suggestion. We have explicitly specified the evaluation temperature of 1.0 in Appendix A.2, along with the other evaluation settings.
----------Ongoing Experiments and Planned Follow-up---------
Regarding Key Point 1, both Q1 and Q2 concern the interpretation of Finding 1. We are currently conducting additional experiments to strengthen the motivation and provide a more comprehensive analysis. We expect to have the results and corresponding analysis within approximately two days.
Ongoing experiments. We have completed the training runs for the horizon curriculum experiments and are currently evaluating the checkpoints. We expect to have the results shortly. Regarding GSPO, we had already implemented the method before the original submission, but due to time constraints, we were unable to complete the experiments and therefore did not include them in the paper. We have now resumed these experiments and expect to obtain the results within approximately two days.
We sincerely appreciate your constructive suggestions, which have helped us strengthen the experimental evidence and improve the clarity of our manuscript. We welcome any further questions or concerns.
Clarifiation on the key concerns affecting my rating
I understand that preparing rebuttal responses, particularly those involving additional experiments, requires subtantial time and resources. To help prioritize the discussion, I would like to clarify which concern are most important to my assessment. I remain open to increasing my rating if the following key concerns can be adequately addressed.
Throughout the paper, the authors use the checkpoint at training step 300 for evaluation. However, in RL training, it is common to monitor performance at intermediate training steps (e.g., AIME accuracy over training steps in Figure 1 of the DAPO paper). Such evaluations are not reported in this paper, which is particularly important given Finding 1 in the motivating experiment (Sec. 4.1).
From Fig. 2, I agree that the large gradient spikes observed for H=4k and H=8k before step 300 indicate optimization instability. However, I think this does not necessarily imply that these configurations consistently underperform throughout training. For example, if the H=8k configuration achieves higher performance than the competing configurations at step 140, before the gradient spike, the interpretation of Finding 1 would need to be reconsidered or qualified. If this is the case, the additional training cost with a longer horizon may be justified by the improved performance, rather than being an inherent drawback. Therefore, I believe that evaluating intermediate checkpoints, particularly those before the observed gradient spikes, is important for substantiating Finding 1.
The evaluation is restricted to only three benchmarks, each containing 30 or 40 questions. Although pass@1 is estimated using 32 sampled responses per problem, the small number of distinct evaluation problems still raises concerns about the statistical reliability and generalizability of the results.
I believe that a broader evaluation using additional benchmarks, such as MATH-500 or OlympiadBench, is necessary to provide more convincing evidence for the effectiveness of the proposed method.
I confirmed that even my minor suggestions were incorporated into the revisions. I do not necessarily expect extensive new experiments if these concerns can be adequately addressed (e.g., multiple additional baselines). I hope these clarifications are helpful in identifying which concerns are most important to my assessment. I look forward to the authors' responses and remain open to discuss.