acceptodds
Under review as a conference paper at ICLR 2027

TraceBand: Prefix Reweighting under Finite-Sample Uncertainty for RLVR

Abstract

In reinforcement learning with verifiable rewards (RLVR), sampled continuations can be used to estimate the values of intermediate reasoning prefixes, but finite sampling leaves these estimates uncertain. This uncertainty makes it unclear whether observed differences reflect true value differences and therefore justify reweighting the sampled prefixes. We show that directly reweighting sampled prefixes using these finite-sample estimates can decrease the sampled-prefix objective. We introduce TraceBand, which constructs simultaneous confidence bands for prefix values and reweights sampled prefixes to maximize worst-case objective improvement. The resulting maximin problem has a closed-form solution that leaves the sampled-prefix distribution unchanged whenever the intervals have a nonempty intersection, that is, whenever all prefixes may have the same value, in which case reweighting could only incur a cost without any gain. Whenever the intervals contain the true prefix values, this reweighting has nonnegative gain in the sampled-prefix objective relative to the unchanged prefix weights. Experiments across mathematical reasoning and code generation show that TraceBand improves over finite-sample plug-in reweighting and achieves strong performance relative to representative RLVR baselines across multiple models.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning with verifiable rewards, LLM Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NNY6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the uncertainty of intermediate reasoning prefix value estimates in reinforcement learning with verifiable rewards (RLVR), showing that directly reweighting sampled prefixes based on finite-sample estimates can decrease the sampled-prefix objective. It proposes TraceBand, a principled reweighting method that constructs simultaneous confidence bands for prefix values and maximizes worst-case objective improvement through a closed-form maximin solution. TraceBand avoids unnecessary reweighting when the confidence intervals overlap and guarantees nonnegative gain in the sampled-prefix objective when the intervals contain the true prefix values. Experiments on mathematical reasoning and code generation demonstrate improvements over finite-sample plug-in reweighting and competitive performance against representative RLVR baselines.

Critical strengths
  1. Rigorous treatment of estimation uncertainty. The paper identifies a key limitation of finite-sample prefix value estimation and demonstrates that naive reweighting can reduce the optimization objective. This provides a clear motivation for explicitly accounting for uncertainty in prefix-level optimization.
  2. Principled optimization with theoretical guarantees. TraceBand formulates prefix reweighting as a maximin optimization problem with a closed-form solution. Its uncertainty-aware design avoids unnecessary reweighting when the evidence does not establish meaningful value differences and guarantees nonnegative objective gain under the stated coverage condition.
  3. Broad empirical validation. Experiments across mathematical reasoning and code generation evaluate TraceBand across multiple models and against representative RLVR baselines. The results demonstrate improvements over plug-in reweighting and support the practical value of uncertainty-aware prefix optimization.
Critical weaknesses
  • The paper argues that directly reweighting prefixes based on finite-sample value estimates may degrade the objective, but does not establish how prevalent or severe this issue is in practical RLVR training. The authors should evaluate its frequency across sampling budgets, task difficulties, and training stages, and assess its impact on final reasoning performance.
    • TraceBand maximizes worst-case objective improvement, which mitigates potential losses due to uncertainty but may sacrifice average performance gains. The authors should analyze its impact on optimization efficiency and examine whether wide confidence intervals or small sample sizes lead to excessive conservative decisions that hinder exploration and learning.
    • The claimed non-negative gain from reweighting relies on confidence intervals covering the true prefix values, but the scope of this guarantee remains unclear. The authors should clarify whether it applies only to the sampled-prefix objective or can extend to policy updates, expected returns, and final task performance.
    • Relevant prior methods should be included as baselines to enable a fair comparison. The authors should also discuss their similarities and differences with TraceBand in terms of methodology, theoretical guarantees, and empirical performance.
      • Shrinking the Variance: Shrinkage Baselines for Reinforcement Learning with Verifiable Rewards
      • On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation
      • Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
Questions for authors

Explain the concerns in Weaknesses Part

3 comments

W1. Prevalence and Severity of Harmful Prefix Reweighting.

Plug-in is the natural finite-sample implementation of optimal prefix reweighting, replacing unknown prefix values with Monte Carlo estimates from sampled continuations. However, estimation errors can make such reweighting harmful.

We extend the independent reference diagnostic in Appendix I.3 to examine how frequently and severely this occurs across continuation budgets, task difficulties, and training stages. Using 256 independent reference continuations per prefix, we classify a Plug-in update as harmful when the reference upper confidence bound on its improvement is negative. Following Figure 1(a), harmful rates are measured among activated pools. Severity is measured by the mean magnitude of the negative upper bound, providing a conservative lower bound on the loss.

AnalysisSettingHarmful among Activated (%)Mean Loss Lower Bound
Sampling BudgetM=2M=231.00.0144
M=4M=428.30.0098
M=8M=828.40.0068
M=16M=1625.00.0037
Task Difficulty (M=8M=8)Level 324.90.0052
Level 428.60.0068
Level 531.70.0081
Training Stage (M=8M=8)Early32.10.0085
Middle28.50.0065
Late24.60.0049

Harmful Plug-in updates occur across all examined settings. Increasing MM from 2 to 16 reduces the harmful-update rate from 31.0% to 25.0% and the mean loss lower bound from 0.0144 to 0.0037, but does not eliminate harmful updates. Both prevalence and severity increase with task difficulty and decrease across training stages. These observations demonstrate that finite-sample estimation errors remain consequential even when more continuations are available.

Importantly, this does not imply that Plug-in reweighting is detrimental on average. Plug-in improves Pass@1 over Continuation-only at both M=8M=8 (42.1 versus 41.3) and M=16M=16 (43.9 versus 42.4), while TraceBand further improves performance to 44.9 and 45.1, respectively (Table 3). Thus, prefix reweighting provides a useful learning signal, but directly substituting uncertain value estimates can introduce harmful updates that uncertainty-aware reweighting is designed to avoid.


W2. Conservatism and Optimization Efficiency.

We quantify the cost of conservative reweighting by comparing the optimal improvement on each realized prefix pool with the improvement achieved by TraceBand, using 256 independent reference continuations per prefix.

Table 1. Available and unrealized prefix-level improvement.

Training PoolsFrequency (%)Available ImprovementTraceBand ImprovementUnrealized Improvement
Active13.30.022370.010460.01191
Inactive86.70.001370.000000.00137
Overall100.00.004160.001390.00277

Although TraceBand activates on only 13.3% of pools at M=8M=8, these pools account for approximately 71% of the total available improvement. Their mean available improvement is 0.02237, compared with 0.00137 for inactive pools. Thus, conservative reweighting leaves some improvement unrealized, but activation is concentrated on pools offering substantially greater improvement opportunities. Moreover, TraceBand achieves higher mean reference-estimated improvement than Plug-in on identical pools (0.00139 versus 0.00041).

Table 2. Activation rate and six-benchmark average Pass@1 (%), from Figure 1(b) and Table 3.

MMTraceBand Activation (%)Continuation-onlyPlug-inTraceBand
40.040.039.340.0
813.341.342.144.9
1637.142.443.945.1

At M=4M=4, wide confidence intervals prevent prefix reweighting, and TraceBand reduces to Continuation-only. Importantly, inactive reweighting does not prevent continuation learning, as all sampled prefixes and continuations remain available for training. As MM increases, activation rises to 13.3% and 37.1% at M=8M=8 and M=16M=16, respectively, while TraceBand outperforms both Plug-in and Continuation-only at both budgets.

These results demonstrate that conservatism has a measurable opportunity cost but does not necessarily hinder downstream learning. TraceBand concentrates reweighting on pools with greater improvement opportunities while retaining continuation learning when confidence bounds do not justify prefix updates.

W3. Scope of the Nonnegative Improvement Guarantee.

The guarantee in Eq. (14) applies to the sampled-prefix objective in Eq. (5), conditional on simultaneous confidence coverage, which holds with probability at least 1−δ1-\delta for the realized prefix pool. On the coverage event,

\ge B(w^{\mathrm{TB}}) \ge B(\mathbf{1})=0.$$ This guarantees nonnegative improvement for the **reweighted prefix target**, but not directly for the parameterized policy. Since Eq. (17) fits this target through policy optimization, the guarantee does not automatically extend to parameter updates, expected returns, or final task performance. We examine this transfer empirically at two levels. First, **Appendix I.4 (Table 8)** evaluates the projected policy using independent reference confidence intervals. Among 1,000 active prefix pools, **92.1% show positive improvement, 7.7% remain unresolved, and 0.2% show negative improvement**. Second, **Figure 2(c)** evaluates downstream performance on 300 Level 4–5 MATH test problems using 512 fresh rollouts per policy and prompt, showing a **mean success-rate improvement of +2.28 percentage points**. Thus, the theoretical guarantee is restricted to the sampled-prefix objective, while improvements in the projected policy and downstream performance are supported empirically. We will clarify this distinction in the revised manuscript. --- **W4. Comparison with Related RLVR Methods.** **Downstream performance comparison.** We additionally compare TraceBand with the three suggested methods under matched training and evaluation protocols, using the same initial model, training data, and verifier budget. We report the six-benchmark average Pass@1 on Qwen2.5-Math-7B. | Method | Average Pass@1 (%) | |---|---:| | Shrinkage Baseline | 43.7 | | Direction of RLVR Updates | 43.2 | | ProGRPO | 43.9 | | **TraceBand** | **44.9** | **TraceBand achieves the highest average Pass@1**, outperforming Shrinkage Baseline, Direction of RLVR Updates, and ProGRPO by 1.2, 1.7, and 1.0 percentage points, respectively. **Methodological distinctions and theoretical guarantees.** All three methods improve RLVR optimization, but address different sources of inefficiency: - **Shrinkage Baselines** combines per-prompt and cross-prompt reward estimates to reduce policy-gradient variance, with a theoretical variance-reduction guarantee. - **Direction of RLVR Updates** identifies reasoning-critical token changes through signed log-probability differences and introduces probability-aware token-level advantage reweighting. Its theoretical analysis of test-time extrapolation considers an idealized natural-policy-gradient setting. - **ProGRPO** modifies response-level advantages using generative probabilities to promote diverse correct reasoning trajectories, with theoretical analysis of confidence balancing and exploration dynamics. In contrast, **TraceBand addresses uncertainty in prefix-value estimation from finitely sampled continuations**. Rather than modifying gradient baselines or token-/response-level advantages, it constructs simultaneous confidence bands and maximizes worst-case improvement over prefix weights. This yields **a nonnegative improvement guarantee for the realized-prefix objective under the confidence-coverage event (Eq. 14)**. These results establish TraceBand's empirical competitiveness while distinguishing its optimization objective and theoretical guarantee from those of the cited methods. We will incorporate these studies into Related Work and clarify these distinctions in the revised manuscript.

We thank the authors for their response. I will maintain my positive scores.

Reviewer_LBWzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the credit assignment in RLVR. While existing prefix-level methods use multiple continuation samples to reduce variance, they still suffer from finite-sample uncertainty, the plug-in estimates can often lead to noisy policy updates when true value differences are poorly resolved. The author introduce a framework called TraceBand. The framework builds confidence bands for prefix values and frames reweighting as a maximin optimization problem for worst-case improvement.

Critical strengths
  1. The author theoratically derive the TraceBand threshold rule, which determines which prefix to reweight and by how much under simultaneous confidence bands. The traceBand is not restricted to the Plug-in direction and maximizes worst-case improvement over all feasible prefix weights.
  2. The authors compare TraceBand with Continuation-only, Worst-case Interpolation, and Matched Plug-in under the same sampling budget and optimization setup. The comparison with Worst-case Interpolation is useful because it also guarantees nonnegative true pool improvement on the simultaneous-coverage event, but remains restricted to the Plug-in direction.
Critical weaknesses
  1. The authors report improved fresh-rollout success in Figure 2(c), but the comparison does not indicate whether the contribution is from prefix reweighting or from continuation learning.
  2. The authors show that TraceBand reduces harmful updates, but it remains unclear how much available improvement is lost through conservative reweighting on the prefix pools encountered during training.
  3. The authors should clarify whether the prefix sampling procedure in Appendix J.2 supports the importance weighting argument connecting the marginal objective in Eq(3) to the empirical marginal objective in Eq(4).
Questions for authors
  1. Can author compare TraceBand, continuation-only, and matched plug-in starting from the same πold\pi_{old}, using the same sampled prefixes and continuation rewards for three optimizer updates, and then evaluate fresh-rollout success?
  2. Can author compare different choices of K and M while keeping K×MK\times{M} (keep the verifier budget fixed) ?
  3. Can the author explain clearly how the prefix distribution is induced by the common cut-off?
  4. Can the author explain how the empirical marginal objective in Eq(4) relates to the marginal objective in Eq(3)?
2 comments

W1/Q1. Contribution of Prefix Reweighting.

Table 3 already compares TraceBand, Continuation-only, and Matched Plug-in under matched training protocols and verifier budgets. All three methods use the same continuation-learning formulation but differ in their prefix-weight construction. TraceBand achieves 44.9 average Pass@1, compared with 42.9 for Matched Plug-in and 41.3 for Continuation-only.

To further assess whether these differences transfer to the learned policy, we compare the three methods after 1,000 rollout-update iterations. Starting from the same initial policy, we evaluate their trained checkpoints on the same 300 Level 4–5 MATH test problems, using 512 fresh rollouts per policy and prompt.

MethodInitial (%)Final (%)Δ Success (pp)Improved Prompts (%)
Continuation-only46.0047.05+1.0566.7
Matched Plug-in46.0047.68+1.6871.3
TraceBand46.0048.28+2.2874.3

Continuation-only achieves a 1.05 percentage-point gain and improves 66.7% of prompts without prefix reweighting. In comparison, TraceBand achieves a 2.28-point gain and improves 74.3% of prompts, providing evidence of an additional contribution beyond continuation learning alone. TraceBand also outperforms Matched Plug-in (1.68 points; 71.3%) despite matched prefix-KL, indicating that the benefit is not explained solely by the magnitude of prefix updates.


W2. Cost of Conservative Reweighting.

We quantify the improvement left unrealized by conservative reweighting on prefix pools encountered during training. Following Eq. (7), let piTB=wiTB/Kp_i^{\mathrm{TB}}=w_i^{\mathrm{TB}}/K and pi⋆=Zi/∑jZjp_i^\star=Z_i/\sum_j Z_j. The gap between the optimal improvement on the realized pool and the improvement achieved by TraceBand is exactly

=\frac{1}{\beta}D_{\mathrm{KL}}(p^{\mathrm{TB}}\\|p^\star).$$ We estimate this gap using **256 independent reference continuations per prefix**, separately for active and inactive training pools. The reported improvements are reference-based estimates, and overall averages are weighted by the observed pool frequencies. | Training Pools | Frequency (%) | Available Improvement | TraceBand Improvement | Unrealized Improvement | |---|---:|---:|---:|---:| | Active | 13.3 | 0.02237 | 0.01046 | 0.01191 | | Inactive | 86.7 | 0.00137 | 0.00000 | 0.00137 | | **Overall** | **100.0** | **0.00416** | **0.00139** | **0.00277** | Overall, TraceBand leaves **0.00277 of the 0.00416 available improvement unrealized**. However, the improvement opportunities are highly concentrated: active pools account for only 13.3% of training pools but approximately **71.5% of the total available improvement**. On these pools, TraceBand captures 46.8% of the available improvement, while inactive pools offer substantially smaller gains (0.00137 versus 0.02237). These results quantify the cost of conservatism. Rather than maximizing improvement based on uncertain point estimates, TraceBand maximizes the worst-case improvement supported by finite-sample confidence bounds, trading potential additional gains for protection against harmful reweighting.

W3/Q3/Q4. Common-Cutoff Sampling and the Scope of Guarantees.

For each prompt xx, we sample KK independent initial rollouts Y1,…,YK∼πold(⋅∣x)Y_1,\ldots,Y_K\sim\pi_{\mathrm{old}}(\cdot\mid x). The feasible cutoff set T(Y1:K)\mathcal T(Y_{1:K}) is determined by the shortest rollout, subject to minimum prefix and continuation lengths. A common cutoff TT is then sampled uniformly from this set, yielding prefixes si=Yi,1:Ts_i=Y_{i,1:T}.

Because the cutoff depends on sampled rollout lengths, this procedure induces a joint prefix-pool distribution that need not coincide with the policy's prefix distribution at an independently fixed cutoff.

Eq. (3) defines the population-level objective, whereas Eq. (4) defines its finite-pool counterpart. Given the realized pool S=(s1,…,sK)S=(s_1,\ldots,s_K), Eq. (4) assigns uniform reference mass p0,i=1/Kp_{0,i}=1/K to each sampled occurrence. With pi(w)=wi/Kp_i(w)=w_i/K and ∑iwi=K\sum_i w_i=K, the objective is

=\frac{1}{\beta}\sum_{i=1}^K p_i(w)\log Z_i -\frac{1}{\beta}D_{\mathrm{KL}}(p(w)\|p_0).$$ Thus, Eq. (4) is **a realized-pool objective, not an unbiased estimator of Eq. (3)** under the length-dependent common-cutoff procedure. Importantly, **our theoretical guarantees are conditional on the realized prefix pool**. Given the sampled prefixes and frozen rollout policy, the $M$ fresh continuations per prefix are independently sampled from $\pi_{\mathrm{old}}(\cdot\mid x,s_i)$, preserving the simultaneous confidence coverage in Eq. (8). On the coverage event, $$\Delta\bar J(w^{\mathrm{TB}};Z) \ge B(w^{\mathrm{TB}}) \ge B(\mathbf 1)=0.$$ Therefore, the nonnegative improvement guarantee in Eq. (14) remains valid for the realized-pool objective under the length-dependent common-cutoff procedure. We will clarify the sampling procedure, the relationship between Eqs. (3) and (4), and the scope of the guarantee in the revised manuscript. --- **Q2. Sensitivity to K and M under a Fixed Verifier Budget.** We compare different allocations of the fixed verifier budget $KM=32$ between the number of prefixes $K$ and continuations per prefix $M$, using the same training protocol and confidence-bound settings. | K | M | Activation (%) | Continuation-only | TraceBand | Gain (pp) | |---:|---:|---:|---:|---:|---:| | 2 | 16 | 17.4 | 42.1 | 43.8 | +1.7 | | **4** | **8** | **13.3** | **41.3** | **44.9** | **+3.6** | | 8 | 4 | 0.0 | 40.9 | 40.9 | 0.0 | At $K=8,M=4$, the confidence intervals overlap for every possible success-count configuration under the specified confidence settings. Consequently, TraceBand never activates and reduces to Continuation-only, consistent with Appendix F. At $K=2,M=16$, TraceBand activates more frequently than at $K=4,M=8$ (17.4% versus 13.3%), yet achieves a smaller improvement over Continuation-only (+1.7 versus +3.6 percentage points). These results show that **activation frequency alone does not explain downstream performance**. The observed differences are consistent with a trade-off between the diversity of candidate prefixes and the statistical precision of their continuation-value estimates. Among the tested allocations, $K=4,M=8$ achieves the strongest average Pass@1 under the fixed verifier budget.
Reviewer_DRx6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to mitigate the unreliability of prefix weights estimated from a small number of continuations in reinforcement learning with verifiable rewards (RLVR). The proposed method constructs confidence intervals for the estimated prefix weights and selects weights that guarantee improvement across the values covered by these intervals. The paper provides theoretical guarantees for a fixed pool of sampled prefixes and evaluates the approach on mathematical reasoning and code generation tasks.

Critical strengths
  1. Precise and well-scoped theoretical contribution. The theoretical analysis is precise and appropriately scoped. Equation (7) characterizes when estimation error can make plug-in reweighting harmful, while Equations (10)–(14) establish a baseline-relative robust objective, a unique weight solution, and an explicit condition under which the weights remain unchanged. Importantly, the appendices distinguish the target-level guarantee from a global improvement guarantee for the neural policy, avoiding an overstatement of the theoretical results. All six supplied mathematical tests passed, and independent enumeration reproduced the key Plug-in and TraceBand results reported in Tables 2 and 4. Together, these results provide a verifiable technical foundation for the proposed method.

  2. Carefully controlled and broad empirical evaluation. The experimental design addresses several important alternative explanations for the reported gains. The paper evaluates five models across two domains and uses three independent training seeds. Tables 3 and 6 compare TraceBand against a range of informative controls, including Continuation-only, fixed shrinkage, Worst-case Interpolation, activation/KL-matched, and randomly gated variants. Additional ablations investigate the fitting objective, cutoff, and hyperparameters. Under the default mathematical setting, the reported six-task averages are 44.9 for TraceBand, 42.1 for Plug-in, and 42.9 for Matched Plug-in. These comparisons provide more direct evidence for the contribution of the proposed prefix-weight construction than comparisons against GRPO alone, although the evaluation protocol should be clarified to support a more definitive interpretation.

Critical weaknesses
  1. Inconsistencies in the central diagnostics. The central diagnostics require clarification. Figure 2(b) reaches approximately 0.0015, whereas the stated metric has a numerically verified upper bound of approximately 0.000391 under the default configuration. In addition, the paper classifies harmful updates using reference confidence intervals, while the corresponding code appears to use the sign of a reference point estimate. These discrepancies may be explained by unreported settings, scaling, or postprocessing, but the paper does not currently provide enough information to reconcile them. Since these diagnostics are central to the paper's safe-improvement argument, clarifying their definitions and implementation is important for assessing the supporting evidence.

  2. Insufficiently specified mathematical evaluation protocol. The mathematical evaluation protocol is not fully verifiable from the supplied code. The evaluation script does not explicitly enable greedy decoding, and the pinned dependency defaults to a temperature of 1. This does not establish that the reported results are incorrect, but the actual evaluation configuration—including the decoding strategy and relevant generation parameters—is needed to reconcile the implementation with the paper and assess the reproducibility of the reported results.

Questions for authors
  1. Figure 2(b) and confidence-bound classification. What configuration, scaling, and aggregation procedure were used to produce Figure 2(b)? Could the authors reconcile the plotted values with the KL upper bound discussed above and provide the underlying plotting data or computation? Additionally, which implementation of confidence-bound-based classification was used for Figure 1(a) and Table 8? If a separate postprocessing pipeline was involved, providing the relevant code path and corrected statistics would help establish whether the main conclusions remain unchanged.

  2. Reproducibility of the mathematical evaluations. What exact commands, temperatures, generation limits, and dependency versions were used for the mathematical evaluations reported in Tables 1, 3, and 6? Could the authors provide the evaluation configurations or result metadata for each training seed, reconcile these settings with the defaults in the supplied script, and confirm that all compared methods were evaluated under the same protocol?

Rebuttal Consideration

The proposed method is promising, but these issues concern important evidence supporting its safety and performance claims. I would consider raising my score if the authors' clarifications or corrections resolve these concerns while preserving the paper's main conclusions.

2 comments

Response to Weakness 1 and Question 1

1. Figure 2(b): KL scaling and computation.

Figure 2(b) does not violate the stated KL upper bound. Its y-axis is scaled by 10−410^{-4}, so a plotted value of approximately 1.5 represents 1.5×10−41.5 \times 10^{-4}, rather than 0.00150.0015.

Figure 2(b) reports the target alignment gain

G=DKL(pTB∥p0)−DKL(pTB∥qθ),G=D_{\mathrm{KL}}(p^{\mathrm{TB}}\|p_0)-D_{\mathrm{KL}}(p^{\mathrm{TB}}\|q_\theta),

where piTB=wiTB/Kp_i^{\mathrm{TB}} = w_i^{\mathrm{TB}}/K, p0,i=1/Kp_{0,i} = 1/K, and qθ,i∝πθ(si∣x)/πold(si∣x)q_{\theta,i} \propto \pi_\theta(s_i \mid x)/\pi_{\mathrm{old}}(s_i \mid x) denotes the normalized prefix distribution induced by the policy after three optimizer updates.

Since the second KL is nonnegative, G≤DKL(pTB∥p0)G\leq D_{\mathrm{KL}}(p^{\mathrm{TB}}\|p_0). Under the default configuration (K=4K=4, M=8M=8, β=1\beta=1, δ=0.05\delta=0.05), enumerating all 949^4 possible success-count configurations yields a maximum target KL of 3.9101×10−43.9101\times10^{-4}. The plotted values therefore satisfy the upper bound.

Both KL terms are computed on the same activated prefix pools. At each iteration, their sums and the active-pool counts are aggregated across distributed processes, and both means use the same global count. We provide the underlying 1,000-step plotting data (figure2b_plotting_data.jsonl) and the corresponding computation in the updated supplementary code.


2. Figure 1(a) and Table 8: Reference confidence-bound classification.

Both diagnostics use the independent reference procedure described in Appendix I.3. For each prefix, Mref=256M_{\mathrm{ref}}=256 reference continuations are used to construct simultaneous confidence intervals [Liref,Uiref][L_i^{\mathrm{ref}}, U_i^{\mathrm{ref}}] for log⁡Zi\log Z_i, with δref=0.05\delta_{\mathrm{ref}}=0.05 and per-tail failure probability δref/(2K)\delta_{\mathrm{ref}}/(2K).

Let mi=(Liref+Uiref)/2m_i=(L_i^{\mathrm{ref}}+U_i^{\mathrm{ref}})/2 and ri=(Uiref−Liref)/2r_i=(U_i^{\mathrm{ref}}-L_i^{\mathrm{ref}})/2. The improvement bounds corresponding to Eq. (5) are

=\frac{1}{\beta K}\sum_{i=1}^{K} \left[(w_i-1)m_i\pm r_i|w_i-1|-w_i\log w_i\right].$$ An update is classified as **harmful only if** $\Delta\bar J_{\mathrm{ref}}^{+}(w)<0$, positive if $\Delta\bar J_{\mathrm{ref}}^{-}(w)>0$, and unresolved otherwise. Figure 1(a) applies this criterion to the constructed prefix weights, whereas Table 8 applies it to the policy-induced weights $w_i^{\mathrm{proj}}=Kq_{\theta,i}$ after projection. **Supplementary code clarification.** We inadvertently uploaded an outdated version of the diagnostic code with the original supplementary materials instead of the implementation used for the reported results. We apologize for this oversight. The updated supplementary code provides the confidence-bound classification implementation used for both mathematical reasoning and code generation. **The reported diagnostic statistics remain unchanged.** Figure 1(a) reports no TraceBand target updates classified as harmful among activated pools at M=8 and M=16, while Table 8 reports **92.1% positive, 7.7% unresolved, and 0.2% harmful** projected updates on active pools, as classified using independent reference confidence bounds. This correction does not affect the training algorithm, theoretical guarantees, or reported benchmark results. --- ### **Response to Weakness 2 and Question 2** **Mathematical evaluation protocol.** All mathematical evaluations in Tables 1, 3, and 6 used **greedy decoding (temperature = 0)** with a maximum generation length of 2,048 tokens and a maximum model context length of 4,096 tokens. Evaluations were conducted using LightEval with vLLM. The supplementary `requirements.txt` specifies the evaluation dependencies, including the LightEval commit and vLLM version. The evaluation script initially included in the supplementary materials did not explicitly specify the decoding parameters, making the reported evaluation settings difficult to verify. The updated supplementary materials include the evaluation script used for the reported results, which explicitly specifies temperature = 0, top-p = 1, and the 2,048-token generation limit. All compared methods were evaluated under the **same protocol across three independent training seeds**, with each resulting checkpoint evaluated deterministically. We provide the evaluation commands and explicit configurations for all six mathematical benchmarks, together with the **per-seed Pass@1 results** underlying Tables 1, 3, and 6. These clarifications resolve the ambiguity in the originally submitted evaluation script **without changing the evaluation protocol or reported results**.

Dear Reviewer,

As the discussion period is nearing its end, we would like to follow up on our responses to your review.

If you have any remaining questions or concerns, please feel free to let us know. We would be happy to provide further clarification before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.