acceptodds
Under review as a conference paper at ICLR 2027

Decoupling Outcome-Observation and Actor-Update Horizons in Reinforcement Learning with Verifiable Rewards

Abstract

Long rollout horizons are increasingly used in reinforcement learning with verifiable rewards (RLVR), yet their effect on policy optimization remains unclear. Existing methods mainly modify how boundary-truncated responses are filtered or penalized, while retaining the assumption that outcome observation and policy optimization should share the same horizon. We find that longer rollouts substantially delay the final answer without consistently improving task performance, while earlier response prefixes can already support correct answers. At the same time, short horizons may miss useful outcome feedback beyond the truncation boundary. Motivated by this mismatch, we propose Natural-Continuation Boundary Return (NCBR), which naturally continues truncated responses for verification while updating only the original responses. Under a frozen continuation policy, the extended outcome estimates the continuation value of the original response, providing a basis for short-response updates. Across three mathematical reasoning benchmarks, NCBR consistently improves both DAPO and GRPO. With DAPO, it achieves a 57.4% relative improvement in AIME'24 Pass@1 over the 2K DAPO baseline, while outperforming Full 8K training with 54.5% fewer GPU-hours.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning with Verifiable Rewards, Large Language Models, Finite-Horizon Reinforcement Learning, Rollout Boundary, Natural Continuation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_w3r3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper argues that the observation horizon (L) and the rollout horizon (H) do not need to be the same in RLVR. The motivating study on Qwen3-4B-Base/DAPO-Math-17k reveals that (1) higher rollout horizons raise cost and instability without better pass@1, and (2) continuing 2k-truncated responses to 8k turns up to 33.56% of incorrect boundary responses to correct ones. The proposed NCBR continues H-truncated responses up to L under same policy, replaces its reward with the outcome at L, and updates only the original H tokens. Experiments on Qwen3-4B/8B-Base with GRPO and DAPO report the performance gains on AIME24/25 and AMC23. The paper also reports that NCBR beats full-8k training with 54.5% fewer GPU-hours.

Critical strengths
  1. Informative diagnostics. The boundary analysis (Table 1: 0$\rightarrow$1 transitions) and the prefix probes (Figure 3b, 3c) give concrete evidence that H-truncated responses are often mislabeled. This evidence is useful beyond this paper.
  2. Favorable cost-performance trade-off. H=2k/L=8k NCBR outperforms full-L on all three benchmarks with 54.5% fewer GPU-hours and 62.4% fewer actor-update tokens (Table 3). Continuation tokens are never trained on, which keeps the update cost at H=2k level.
Critical weaknesses

W1. The empirical evidence does not support the central claims.

Both the motivating study in Section 4 and the main results (Tables 2 and 3) evaluate at one fixed checkpoint on 30-40 problem benchmarks. Several specific problems:

  • In Figure 2, the H=8k baseline looks collapsed, not merely "less stable". Figure 6 shows its policy entropy climbing from ~0.5 to >2 after ~180 steps with repeated gradient spikes. H=4k baseline also spikes at ~290 steps, right before the step-300 evaluation. Hyperparameters tuned for H=2k are reused at 4k/8k, so Finding 1 cannot be drawn from an untuned, single run that diverged.
  • The overlong buffer width and 4k/8k hyperparameters are not stated anywhere in the paper.
  • 47.6% of the H=8k model's evaluation responses are truncated at 8k (Figure 1.b), so its pass@1 is a lower bound that the H=2k model is not subject to. Results at a non-truncating budget (e.g. H_eval=16k) are needed.
  • Only three benchmarks of about 30-40 problems each were used, and seeds and CIs are not reported. Other larger-size benchmarks such as MATH500, Minerva, and OlympiadBench are absent.
  • Step 300 is an arbitrary cut. There is no process of selecting the best checkpoint based on the validation set. I believe that if a checkpoint represents the best performance (even if it occurred before one epoch), it should be selected. (e.g., For the H=8k DAPO baseline, if the model in fact collapses around step 150, you should choose the checkpoint from step ~140 as the final model.)

W2. Baselines are limited to vanilla GRPO and DAPO.

  • DAPO's 'hard' overlong filtering (masking truncated responses, not 'soft' overlong punishment) is mentioned in the introduction but not compared. It costs nothing, whereas NCBR roughly doubles training cost. I think this is the minimum bar.
  • PPPO is mentioned in the Related Work as the closest method, yet never compared. Same for VinePPO.
  • The partial rollouts (Kimi K1.5) or horizon curriculum (DeepScaleR or Skywork-OR1) should be compared, especially in the cost claim since they are cheaper alternatives.
  • In the supplementary materials, GSPO recipe exists, but the paper does not report GSPO results.
Questions for authors

Q1. Please clarify how the evaluated checkpoint was selected. Is there a specific reason for using the step-300 checkpoint without any validation-based selection? (Note: I am not asking for the separation of a validation set and a test set. It is perfectly fine to use AMC23, AIME24, and AIME25 here. For example, as I mentioned in W1, if a checkpoint from before step 300 yields better performance on amc23/aime24/aime25, shouldn't that checkpoint be chosen instead?)

Q2. In the motivating experiment, do the 4k/8k conclusions hold for the best pre-collapse checkpoint selected on validation?

Q3. Please report pass@1 and truncation rates at a non-truncating budget (e.g., H_eval=16k), seeds/CIs, and more benchmarks including MATH500, Minerva Math, and OlympiadBench.

Q4. How does NCBR compare with DAPO’s overlong filtering, PPPO, partial rollouts (Kimi K1.5), and horizon curricula (DeepScaleR, Skywork-OR1) in both performance and GPU-hour cost?

Q5. Why are the GSPO results omitted, given that a GSPO recipe is included in the supplementary material?

Q6. Please report, over training, (a) the fraction of continuations that also hit L, (b) the 0$\rightarrow$0 transitions split into "terminated wrong" and "truncated at L", and (c) the dynamics beyond 300 steps. Does the L-cap rate stay bounded once the H cap-hit ratio reaches ~40%?

[Minor Suggestions/Questions]

S1. Please state the model precisely. The supplementary and the Table 2 base-row numbers indicate Qwen3-4B-Base, not Qwen3-4B. Is the 8B model also the Qwen3-8B-Base, not Qwen3-8B?

S2. What is the difference between Table 1 and Table 6? It seems that Table 6 is dupilcate, which is unnecessary.

S3. Without Figure 6, the counts in Table 1 are hard to interpret: as training goes, the number of truncated responses goes 9.2k$\rightarrow$0.8k$\rightarrow$11.3k$\rightarrow$0.7k. Since the populations differ so much, the recovery rate trend across steps is not meaningful (the paper does not overclaim this, but the table invites it). One recommended option is presenting step 0 and step 200, which have enough samples, as the key points.

S4. Typo in the third line from the bottom of the Introduction: "DAP".

S5. What is the temperature during evaluation? The supplementary suggests 1.0, as in training. Please state it in the paper.

11 comments
Reviewer_w3r3OpenReview ↗

Clarifiation on the key concerns affecting my rating

I understand that preparing rebuttal responses, particularly those involving additional experiments, requires subtantial time and resources. To help prioritize the discussion, I would like to clarify which concern are most important to my assessment. I remain open to increasing my rating if the following key concerns can be adequately addressed.

  1. Evaluation of intermediate training checkpoints.

Throughout the paper, the authors use the checkpoint at training step 300 for evaluation. However, in RL training, it is common to monitor performance at intermediate training steps (e.g., AIME accuracy over training steps in Figure 1 of the DAPO paper). Such evaluations are not reported in this paper, which is particularly important given Finding 1 in the motivating experiment (Sec. 4.1).

From Fig. 2, I agree that the large gradient spikes observed for H=4k and H=8k before step 300 indicate optimization instability. However, I think this does not necessarily imply that these configurations consistently underperform throughout training. For example, if the H=8k configuration achieves higher performance than the competing configurations at step 140, before the gradient spike, the interpretation of Finding 1 would need to be reconsidered or qualified. If this is the case, the additional training cost with a longer horizon may be justified by the improved performance, rather than being an inherent drawback. Therefore, I believe that evaluating intermediate checkpoints, particularly those before the observed gradient spikes, is important for substantiating Finding 1.

  1. Limited evaluation benchmarks.

The evaluation is restricted to only three benchmarks, each containing 30 or 40 questions. Although pass@1 is estimated using 32 sampled responses per problem, the small number of distinct evaluation problems still raises concerns about the statistical reliability and generalizability of the results.

I believe that a broader evaluation using additional benchmarks, such as MATH-500 or OlympiadBench, is necessary to provide more convincing evidence for the effectiveness of the proposed method.


I confirmed that even my minor suggestions were incorporated into the revisions. I do not necessarily expect extensive new experiments if these concerns can be adequately addressed (e.g., multiple additional baselines). I hope these clarifications are helpful in identifying which concerns are most important to my assessment. I look forward to the authors' responses and remain open to discuss.

Thank you for your follow-up and for clarifying your main concerns. We appreciate your patience as we conducted additional experiments under computational constraints. Below, we address the points for which results are currently available. Below, we address the points for which results are currently available.

Additional Evaluation Benchmarks and Budgets (Key Point 2 and Q3)

We evaluated the Step-300 checkpoints on AIME'24, AIME'25, AMC'23, and MATH500 under two evaluation budgets: 8K (8192 tokens) and 16K (16384 tokens). All values below are Pass@1 (%).

Evaluation budget: H_eval = 8K (8192 tokens).

Training configurationAIME24AIME25AMC23MATH500
DAPO H=2K10.5211.4661.6456.06
DAPO H=4K15.8316.2556.2554.08
DAPO H=8K9.9011.3558.2855.36
NCBR H=2K L=8K16.5614.4869.8459.03

Evaluation budget: H_eval = 16K (16384 tokens).

Training configurationAIME24AIME25AMC23MATH500
DAPO H=2K11.1510.8360.7056.01
DAPO H=4K15.4216.1556.8054.01
DAPO H=8K8.9611.5656.7255.27
NCBR H=2K L=8K15.5213.7571.2559.18

NCBR achieves the highest Pass@1 among these configurations on AIME24, AMC23, and MATH500 at both evaluation budgets, while DAPO H=4K achieves the highest Pass@1 on AIME25. On MATH500, NCBR achieves 59.03% at 8K and 59.18% at 16K, compared with 56.06% and 56.01% for DAPO H=2K, respectively.

Comparison with other baselines (Q4).

Comparison with Hard Overlong Filtering

Thank you for suggesting this additional baseline. We trained DAPO with hard overlong filtering on Qwen3-4B-Base for 300 steps, using a 2K training horizon. Hard filtering masks the actor loss for boundary-truncated responses while retaining DAPO's dynamic sampling and group-relative advantage computation.

All methods were evaluated under the same protocol described in the paper, with an 8K evaluation budget.

MethodAIME'24 P@1 / P@32AIME'25 P@1 / P@32AMC'23 P@1 / P@32
Vanilla DAPO 2K10.52 / 33.3311.46 / 36.6761.64 / 85.00
DAPO Hard-only 2K17.29 / 43.338.54 / 30.0062.42 / 95.00
DAPO + NCBR 2K→8K16.56 / 46.6714.48 / 40.0069.84 / 95.00

Hard filtering improves performance on some benchmarks, notably AIME'24 Pass@1. However, simply masking the actor loss of truncated responses does not consistently reproduce NCBR's downstream performance across the three benchmarks. NCBR achieves higher Pass@1 on AIME'25 and AMC'23, while matching or improving Pass@32 on all three benchmarks.

Comparison with PPPO

Response to the comparison with PPPO and incremental novelty

We thank the reviewer for highlighting PPPO and agree that continuation-based prefix evaluation and suffix-gradient masking are not unique to NCBR. To strengthen the empirical evidence, we implemented PPPO with DAPO dynamic sampling and conducted a compute-matched comparison on Qwen3-4B-Base using 16 A100 GPUs. PPPO at Step 20 consumed 641.94 GPU-hours, compared with 628.65 GPU-hours for NCBR at Step 300.

Computational cost analysis. The main computational overhead of PPPO arises from its multiple-continuation sampling. Under our configuration, each problem generates 8 original responses, each spawning 8 independently sampled continuations, resulting in 64 additional continuations per problem. With 768 candidate problems per training step, PPPO generates 6,144 original responses and 49,152 continuations. In contrast, NCBR continues only boundary-truncated responses, with approximately 615 continuations per step in our experiments.

Consequently, PPPO generates approximately 80 times more continuations per step than NCBR. Its continuation stage consumes 600.41 GPU-hours, accounting for 94.26% of its measured training-step time, whereas actor updates account for only 1.51%. For NCBR, continuation accounts for 45.07% of training-step time. The average training-step duration is 119.43 minutes for PPPO versus 7.63 minutes for NCBR.

Under the same 8K evaluation horizon and 32 samples per problem, the results are:

Method (GPU-h)AIME24 P@1P@32AIME25 P@1P@32AMC23 P@1P@32
PPPO (641.94)9.9036.677.8136.6739.6190.00
NCBR (628.65)16.5646.6714.4840.0069.8495.00

All results are percentages. Pass@1 is estimated by the average accuracy across 32 samples per problem.

Under comparable end-to-end GPU-hour budgets, NCBR achieves higher Pass@1 and Pass@32 on all three benchmarks, supporting the practical effectiveness of selective boundary continuation. We have incorporated this comparison into Section 6.3 (Method Comparison) of the revised manuscript.

Continuation truncation and training dynamics (Q6)

We thank the reviewer for requesting a more detailed characterization of continuation behavior. We analyzed the per-step training logs of Qwen3-4B-Base with DAPO+NCBR (H=2048H=2048, L=8192L=8192) over Steps 1–300.

(a) Continuations reaching the observation horizon.

We define the conditional L-cap rate as the number of continuations truncated at LL divided by the number of original responses that strictly hit HH.

Training stepsH-cap continuationsL-cap continuationsConditional L-cap rate
1–10029,9992,0116.70%
101–2008,170710.87%
201–300146,2062,3871.63%
251–30089,1911,2061.35%

Although the H-cap rate increases substantially during later training, the conditional L-cap rate remains low in the observed period.

In particular, at Step 264, the actor-batch H-cap rate reaches 44.24%, while only 40 of 2,092 continuations (1.91%) reach LL. Across the five steps where the actor-batch H-cap rate exceeds 40%, only 170 of 10,122 continuations (1.68%) reach LL.

These observations indicate that L=8192L=8192 provides sufficient observation length for the vast majority of boundary-truncated responses during the first 300 training steps.

(b) Decomposition of 0→0 transitions.

We agree that 0→0 transitions should distinguish naturally terminated incorrect responses from responses truncated again at LL.

The existing aggregate logs record both the total number of 0→0 transitions and the total number of L-cap events, but do not retain their exact intersection.

Nevertheless, they provide a conservative bound. Across Steps 201–300, there are 104,615 candidate-level 0→0 transitions and only 2,387 L-cap events. Therefore, at least 102,228 (97.72%) of these 0→0 transitions correspond to naturally terminated incorrect responses.

At Step 300 alone, at least 1,289 of 1,295 (99.54%) candidate-level 0→0 transitions are naturally terminated errors.

S1. Clarification of Model Initialization

Thank you for pointing this out. We confirm that all experiments use Qwen3-4B-Base and Qwen3-8B-Base as the initialization models. We have clarified in the Introduction that Qwen3-4B and Qwen3-8B are used as shorthand throughout the paper, and specified the full model names in Appendix A.1.

S2 & S3. Boundary-Recovery Table Revision and Interpretation

Thank you for the helpful suggestions. We have revised Table 1 in Section 4.1 to highlight Steps 0 and 200, while retaining the complete checkpoint-wise results in Appendix B.1, with explicit recovery denominators and clarification that the rates should not be interpreted as a monotonic training trend, thereby eliminating the redundancy between the two tables.

S4. Correction of Typographical Error

Thank you for pointing out this typo. We have corrected “DAP” to “DAPO” in the Introduction.

S5. Clarification of Evaluation Settings

Thank you for the suggestion. We have explicitly specified the evaluation temperature of 1.0 in Appendix A.2, along with the other evaluation settings.

----------Ongoing Experiments and Planned Follow-up---------

Regarding Key Point 1, both Q1 and Q2 concern the interpretation of Finding 1. We are currently conducting additional experiments to strengthen the motivation and provide a more comprehensive analysis. We expect to have the results and corresponding analysis within approximately two days.

Ongoing experiments. We have completed the training runs for the horizon curriculum experiments and are currently evaluating the checkpoints. We expect to have the results shortly. Regarding GSPO, we had already implemented the method before the original submission, but due to time constraints, we were unable to complete the experiments and therefore did not include them in the paper. We have now resumed these experiments and expect to obtain the results within approximately two days.

We sincerely appreciate your constructive suggestions, which have helped us strengthen the experimental evidence and improve the clarity of our manuscript. We welcome any further questions or concerns.

Reviewer_ztGWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the distinction between the response length needed to observe a verifiable outcome and the tokens included in an actor update. Natural-Continuation Boundary Return (NCBR) continues boundary-truncated responses under the frozen rollout policy, reassesses their outcomes, and updates only the original bounded responses. The paper combines horizon comparisons, prefix probes, a frozen-continuation analysis, and experiments with DAPO and GRPO on Qwen3-4B/8B across three mathematics benchmarks.

Critical strengths
  1. Separating outcome observation from actor updates is useful and supported by targeted evidence.

  2. The appendix, ablations, and public data strengthen the evaluation. The 16-page appendix (Appendices A-H, pages 12-27) specifies token-preserving continuation, boundary detection, score replacement before dynamic sampling, probe controls, proof assumptions, and cost accounting. These details improve completeness and clarity.

Critical weaknesses
  1. The empirical scope is narrower than the abstract's consistency claim.

  2. The incremental contribution and reproducibility need sharper substantiation. PPPO already updates prefix tokens using rewards from multiple continuations and explicitly masks subsequent-token gradients. Section 2 identifies meaningful differences, but the experiments do not directly establish the advantage of one continuation at a fixed boundary with the original update tokens preserved. A controlled comparison or clearer account of the different operational objectives would strengthen the novelty case.

  3. The submission provides neither a code attachment nor an accessible repository, weakening independent reproducibility despite the detailed appendix.

Questions for authors

See Critical Weaknesses

2 comments

Weakness 1. The empirical scope is narrower than the abstract's consistency claim.

We thank the reviewer for this suggestion. We have narrowed the empirical claim in the abstract to the consistent improvements achieved by NCBR with both DAPO and GRPO on Qwen3-4B-Base across three mathematical reasoning benchmarks.

Weakness 2. Compare with PPPO

Response to the comparison with PPPO and incremental novelty

We thank the reviewer for highlighting PPPO and agree that continuation-based prefix evaluation and suffix-gradient masking are not unique to NCBR.

The key distinction lies in their operational objectives. PPPO aims to improve reasoning initiation by optimizing a progressively increasing fraction of prefix tokens, using accumulated rewards from multiple continuations. In contrast, NCBR addresses outcome censoring induced by a short training horizon: it generates original responses up to H=2048H=2048, selectively continues only boundary-truncated responses once to L=8192L=8192, corrects their outcome rewards, and preserves the original actor-update token set. Thus, NCBR decouples outcome observation from policy optimization rather than prioritizing prefix tokens.

Additional comparison. To strengthen the empirical evidence, we implemented PPPO with DAPO dynamic sampling and conducted a compute-matched comparison on Qwen3-4B-Base using 16 A100 GPUs. PPPO at Step 20 consumed 641.94 GPU-hours, compared with 628.65 GPU-hours for NCBR at Step 300.

Computational cost analysis. The main computational overhead of PPPO arises from its multiple-continuation sampling. Under our configuration, each problem generates 8 original responses, each spawning 8 independently sampled continuations, resulting in 64 additional continuations per problem. With 768 candidate problems per training step, PPPO generates 6,144 original responses and 49,152 continuations. In contrast, NCBR continues only boundary-truncated responses, with approximately 615 continuations per step in our experiments.

Consequently, PPPO generates approximately 80 times more continuations per step than NCBR. Its continuation stage consumes 600.41 GPU-hours, accounting for 94.26% of its measured training-step time, whereas actor updates account for only 1.51%. For NCBR, continuation accounts for 45.07% of training-step time. The average training-step duration is 119.43 minutes for PPPO versus 7.63 minutes for NCBR.

Under the same 8K evaluation horizon and 32 samples per problem, the results are:

Method (GPU-h)AIME24 P@1P@32AIME25 P@1P@32AMC23 P@1P@32
PPPO (641.94)9.9036.677.8136.6739.6190.00
NCBR (628.65)16.5646.6714.4840.0069.8495.00

All results are percentages. Pass@1 is estimated by the average accuracy across 32 samples per problem.

These measurements explain why reducing the number of actor-optimized tokens does not necessarily reduce end-to-end training cost: PPPO incurs substantial additional inference overhead from repeated continuation sampling, whereas NCBR selectively extends only truncated trajectories.

Under comparable end-to-end GPU-hour budgets, NCBR achieves higher Pass@1 and Pass@32 on all three benchmarks, supporting the practical effectiveness of selective boundary continuation. We have incorporated this comparison into Section 6.3 (Method Comparison) of the revised manuscript.

Weakness 3. provides code attachment nor an accessible repository

We thank the reviewer for highlighting the importance of reproducibility. We have provided the NCBR implementation as part of the OpenReview supplementary material. The release includes the core natural-continuation mechanism, boundary-return replacement, DAPO dynamic-sampling integration, and GRPO integration, together with training configurations, executable training and evaluation scripts, dependency specifications, and installation instructions.

In particular, the implementation preserves the original response tokens for actor optimization, performs continuation under the same rollout-policy snapshot, and reassesses boundary outcomes before group-relative advantage computation and, where applicable, dynamic group filtering. The supplementary material also documents the training and evaluation protocols, including the rollout horizons, batch configurations, sampling parameters, and optimization settings.

Thank you for your reply. Having read the reviews and replies, I’ve decided to keep my score.

Reviewer_LHTmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Short generation limits may truncate the reasoning process before the final answer is reached, yet including full-length, long-form responses in training is computationally expensive. NCBR decouples the length of the sequence used for observation from the length used for parameter updates. For responses that hit the short generation limit, the model continues generating (with parameters frozen) to reach the full answer. This complete response is then re-evaluated to calculate an adjusted reward, which is used to recompute the in-group advantage. Crucially, the loss is applied only to the original (truncated) response, excluding the appended suffix.

Critical strengths
  • The motivation is clear, and the methodology is well-defined. It directly addresses the reward misestimation caused by truncation by focusing on the final correctness, preserving the original length-based reward, and performing re-evaluation prior to in-group filtering. These design choices allow for seamless integration with methods like DAPO and GRPO.
  • The approach demonstrates performance gains in reasoning tasks.
Critical weaknesses
  • The efficiency analysis is based on a fixed number of updates (300) rather than a fixed total computational budget. The current results do not clarify whether the same additional compute—if applied to the short-limit baseline (e.g., through more training steps or increased sampling)—would yield comparable improvements.
  • Performance on AIME24 and AMC23 (Pass@1) appears to decline for the 8B+DAPO configuration. While less truncation might explain a reduction in opportunities for reward correction, it does not, by itself, account for an actual performance regression.
Questions for authors
  1. When normalized by total GPU-hours, does NCBR still outperform the 2K-limit baseline when the latter is granted additional training steps or increased sampling volume?
  2. Please report the number of independent training runs and the results for each individual run. Additionally, please clarify the performance gains observed for the 4B model and the specific case of 8B+DAPO.
2 comments

Q1. Performance under Comparable Computational Budgets (Weakness 1 & Q1)

Thank you for this suggestion. We conducted additional experiments to examine whether allocating a comparable computational budget to Vanilla DAPO with a 2K horizon can recover NCBR’s downstream performance. Following the reviewer’s suggestions, we considered two ways of increasing the baseline’s computational budget:

  • More training steps: We extended Vanilla DAPO with group size G=8G=8 to 600 steps, evaluating intermediate checkpoints at steps 400 and 500.
  • More sampling: We increased the group size from G=8G=8 to G=16G=16 and trained for 300 steps.

Our NCBR configuration uses H=2048H=2048, L=8192L=8192, and G=8G=8, with 300 training steps. All experiments use Qwen3-4B-Base.

The following table reports computational costs and performance. Each benchmark cell shows Pass@1 / Pass@32 (%).

MethodStepsGPU-hoursAIME'24AIME'25AMC'23
Vanilla DAPO, G=8G=8300314.6210.52 / 33.3311.46 / 36.6761.64 / 85.00
Vanilla DAPO, G=8G=8400425.0214.38 / 43.3311.35 / 30.0068.44 / 90.00
Vanilla DAPO, G=8G=8500536.9012.29 / 30.009.27 / 30.0055.94 / 87.50
Vanilla DAPO, G=8G=8600631.067.40 / 23.338.02 / 20.0055.00 / 85.00
Vanilla DAPO, G=16G=16300610.6011.77 / 36.6711.35 / 20.0069.69 / 92.50
DAPO + NCBR, G=8G=8200270.4516.46 / 43.3315.63 / 50.0068.59 / 90.00
DAPO + NCBR, G=8G=8300614.3816.56 / 46.6714.48 / 40.0069.84 / 95.00

Extending training does not close the performance gap. At step 600, Vanilla DAPO consumes slightly more computation than NCBR (631.06 vs. 614.38 GPU-hours), while NCBR achieves higher Pass@1 by 9.17, 6.46, and 14.84 percentage points on AIME'24, AIME'25, and AMC'23, respectively.

Increasing sampling also does not match NCBR’s overall performance. At a comparable computational cost, the G=16G=16 baseline achieves nearly identical Pass@1 on AMC'23, but remains below NCBR on AIME'24 and AIME'25. NCBR also achieves higher Pass@32 on all three benchmarks.

These results show that, even when Vanilla DAPO with a 2K horizon receives approximately the same computational budget through additional training steps or increased sampling, it does not match NCBR’s overall downstream performance in our experiments.

Why NCBR outperforms compute-matched baselines. We attribute NCBR's advantage to improving the informativeness of reward feedback rather than simply increasing computation. Extending training steps or increasing group size provides additional optimization or sampling, but does not directly address outcome censoring at the 2K boundary, where potentially successful responses may be prematurely judged incorrect. In contrast, NCBR naturally continues boundary-truncated responses to 8K, recovers delayed outcome feedback, and reconstructs group-relative advantages while preserving the original 2K actor-update horizon. This also allows previously all-zero groups to become informative for DAPO's dynamic sampling. Consequently, NCBR makes more effective use of the available computational budget by improving the learning signals rather than merely increasing the amount of training under truncated feedback. Our compute-matched results support this interpretation, showing that neither additional training steps nor increased sampling reproduces NCBR's overall performance under the evaluated settings.

Manuscript revision. We have added these compute-matched experiments, including the training configurations, GPU-hour costs, and downstream performance comparisons, to the revised main text (Appendix F.3 EXTENDED TRAINING AND SAMPLING BASELINES).

Weakness 2. Performance Regression on Qwen3-8B with DAPO (Weakness 2 & Q2)

We agree that the low truncation rate alone cannot explain the performance regression on Qwen3-8B-Base. At Step 300, NCBR decreases Pass@1 from 22.60% to 18.96% on AIME'24 and from 80.31% to 71.72% on AMC'23, while slightly improving AIME'25 (18.54% to 18.85%). In contrast, NCBR consistently improves both metrics across all three benchmarks on Qwen3-4B-Base in two independent DAPO training runs.

Training dynamics. We further compared critic/advantages/mean over the first 300 training steps for both 4B and 8B DAPO (anonymized training dynamics). On Qwen3-4B-Base, NCBR does not exhibit sustained negative fluctuations comparable to those observed on 8B. In contrast, Vanilla DAPO on Qwen3-8B-Base already exhibits substantial fluctuations, which become more pronounced and persistent with NCBR. Across 298 shared steps, the standard deviation increases from 0.0249 to 0.0359, while the number of steps with mean advantage below −0.10 increases from 7 to 23.

Possible mechanism. NCBR's stochastic boundary reassessment changes group-relative advantages and DAPO's dynamic sampling decisions, potentially amplifying existing optimization fluctuations through the redistribution of learning signals across response groups. For the 8B model, the low truncation rate further limits the benefits of additional outcome feedback. The contrasting 4B and 8B dynamics suggest that NCBR's effectiveness depends on the interaction between the model's optimization state and the underlying optimizer, with NCBR potentially exacerbating DAPO's existing instability in the 8B configuration. This offers a plausible explanation for the observed regression.

Manuscript revision. We have added the 8B advantage dynamics and potential instability analysis to Appendix H, Advantage Dynamics Across Model Scales, with an explicit reference in Section 6.1, Main Results.

Q2. Report the number of independent training runs and the results for each individual run.

We clarify that the Qwen3-4B DAPO experiments (Vanilla H=2K and NCBR H=2K, L=8K) were each conducted with two independent training runs, while all other model–optimizer configurations were evaluated using one training run each. The original single-run results for these configurations are reported in Tables 2 and 3.

To assess reproducibility, we conducted an additional independent training run for both 4B-DAPO variants under the same training configuration and evaluated their Step-300 checkpoints using an 8K inference budget.

We provide the individual results for both runs in the supplementary table, rather than reporting only their averages.

BenchmarkMetricVanilla Run 1Vanilla Run 2Vanilla Mean ± SDVanilla Var.NCBR Run 1NCBR Run 2NCBR Mean ± SDNCBR Var.
AIME'24Pass@110.5213.2311.88 ± 1.923.6716.5615.9416.25 ± 0.440.19
Pass@3233.3336.6735.00 ± 2.365.5846.6740.0043.34 ± 4.7222.24
AIME'25Pass@111.4611.4611.46 ± 0.000.0014.4814.4814.48 ± 0.000.00
Pass@3236.6730.0033.34 ± 4.7222.2440.0053.3346.67 ± 9.4388.84
AMC'23Pass@161.6461.8861.76 ± 0.170.0369.8466.9568.40 ± 2.044.18
Pass@3285.0080.0082.50 ± 3.5412.5095.0097.5096.25 ± 1.773.13

All performance values are Pass@1 / Pass@32 (%). Variances are sample variances across two independent training runs, expressed in squared percentage points (pp²).

Due to time and computational resource constraints, we limited the additional independent training runs to the Qwen3-4B DAPO configurations. Nevertheless, NCBR consistently outperforms Vanilla DAPO in both Pass@1 and Pass@32 across all three benchmarks in both independent runs, demonstrating reproducible performance improvements under the evaluated settings. We have included the individual-run results, in the revised *Appendix F.4 INDEPENDENT TRAINING RUNS ON QWEN3-4B-BASE.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal

Dear AC, SAC and PC

We sincerely thank you and the reviewers for the careful assessment and constructive discussion. We would like to summarize the scientific contributions of our work, the additional evidence provided in response to all three reviewers, and our commitment to implementing the remaining revisions.

1. Valuable findings and a practical contribution to RLVR

Our work investigates a fundamental question in RLVR: must the horizon needed to observe an informative outcome also determine how many tokens participate in policy optimization?

Our boundary analysis shows that short training horizons can hide recoverable successes and change the learning signals available to group-relative optimization. These findings identify a concrete consequence of truncation: it can affect both reward assignment and which groups are retained for training.

Our prefix-probing analysis further distinguishes earlier answer capability from final-answer emission. Earlier prefixes can already support correct short answers even when the original response continues substantially further. Together, these findings provide useful evidence for understanding how generation length, outcome feedback, and policy optimization interact.

NCBR translates these observations into a simple mechanism: it selectively continues boundary-truncated responses under the same frozen policy, uses the extended outcomes to reassess the original responses, and restricts actor updates to the original tokens. The frozen-continuation analysis provides a principled basis for using later outcomes to guide these shorter updates. We believe that treating outcome observation and actor optimization as separate design decisions is a useful contribution to future RLVR training methods.

2. Substantial additional evidence addressing all three reviewers

During the discussion, we conducted additional experiments and revised our interpretation in response to the reviewers’ concerns:

  • For Reviewer LHTm, we added comparisons against short-horizon DAPO with extended training and increased sampling at comparable GPU-hour budgets, reported two independent training runs for each main 4B-DAPO variant, and analyzed the regression observed with 8B+DAPO. These additions strengthen the evidence in the main 4B setting while clarifying the method’s dependence on the model and optimization configuration.

  • For Reviewer ztGW, we clarified the distinction from PPPO, added a comparison at comparable computational cost, narrowed the empirical claims, and provided the implementation, configurations, and training and evaluation scripts in the supplementary material.

  • For Reviewer w3r3, we added intermediate-checkpoint evaluations, MATH-500 and 16K-budget evaluations, comparisons with hard overlong filtering, PPPO, GSPO, and horizon curricula, and further analysis of continuation behavior. We also revised Finding 1 to acknowledge the meaningful intermediate gains of longer horizons and incorporated the requested presentation clarifications.

These additions have substantially expanded the empirical evidence and made the scope and limitations of our conclusions more explicit.

3. Clear commitment to Reviewer w3r3’s remaining recommendations

We acknowledge Reviewer w3r3’s remaining concern about consistent checkpoint selection across the paper. The additional evaluations show that longer-horizon baselines can achieve strong intermediate performance, and we have withdrawn the unqualified claim that NCBR outperforms Full 8K with 54.5% fewer GPU-hours. That percentage describes the original fixed-Step-300 comparison and should not be presented as a general computational-efficiency advantage.

If the paper is accepted, we will fully implement Reviewer w3r3’s recommendations in the revised manuscript. We will apply periodic checkpoint evaluation and a common checkpoint-selection rule consistently across all methods and comparison tables, including Tables 3 and 16, with clearly stated selection criteria and computational budgets. We will revise Table 3 and the corresponding experimental descriptions, and update the Abstract, figures, and conclusions accordingly. We will also distinguish end-to-end computational cost from actor-update token counts and ensure that all claims reflect the resulting comparisons. We recognize that this requires substantive revisions throughout the manuscript and are fully committed to completing them.

We appreciate that Reviewer w3r3’s final comment explicitly recognizes the value of our central observation: long-horizon outcome feedback can improve performance without directly optimizing the entire extended trajectory. We respectfully ask that the paper be considered in light of this scientific contribution, the informative diagnostic findings, the substantial additional evidence, and our firm commitment to addressing the remaining evaluation concerns.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.