acceptodds
Under review as a conference paper at ICLR 2027

CBSD: Cross-Batch Self-Distillation for Reinforcement Learning with Verifiable Rewards

Abstract

In language-model post-training with verifiable rewards, successful responses provide evidence for subsequent learning. GRPO-style response weighting, however, is restricted to current-batch gradients and may miss useful directions revealed by earlier successes. We introduce Cross-Batch Self-Distillation (CBSD), which separates rollout generation from evidence retention. An acting policy generates responses, while a non-acting carrier learns from verified successes across batches. The actor reads the carrier on its own sampled prefixes through on-policy distillation, obtaining token-wise guidance without replaying source responses or supplying them as context. A control-variate write and pre-write reading support stable training. Our analysis characterizes the ideal success-conditioned carrier and interprets its log-ratio read as token-level credit. In a binary-verifier family at a fixed actor, sufficient source sampling and reuse yield larger expected first-order gains than fresh-batch response weighting under matched total response and local KL budgets. With Qwen3-8B on SciKnowEval, CBSD achieves average maximum Avg@16, surpassing reported SDPO/GRPO by percentage points and leading in three of four domains. On 27 Physics prompts rarely solved initially, it exceeds GRPO by points at step 200.

Primary area:
reinforcement learning
Keywords:
Large Language Model Reasoning, Reinforcement Learning with Verifiable Rewards, Online Policy Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_THpfOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors propose a novel training algorithm with verifiable rewards, which use past successful trajectories in a novel way: do a (variation of) SFT on positive examples on the "carrier" model, and then run an On-Policy Distillation from this carrier model to train an actor. The carrier update employs a correction for the divergence between actor and the carrier in order to stabilize carrier learning, which authors call 'Write Control Variate'. Authors offer two theoretical analysis for the insights. First, authors show that the token-level credit in on-policy distillation can be viewed as log-advantage. Authors also show with the toy problem (learning first m bits of n-bit string should be turned on) that reusing previous samples leads to faster learning. Empirically, authors experiment on SciKnowEval-L3 with Qwen 3-8B models to demonstrate the advantage over GRPO and SDPO, which is consistent across domains.

Critical strengths

Authors propose a novel method of leveraging past positive trajectories. This topic is gathering a lot of attention in the literature, as the popular GRPO baseline is clearly very sample-inefficient on problems with low success probability. Authors leverage token-level supervision for both carrier and actor, which is a relative strength against GRPO baseline. On the other hand, while SDPO provides token-level supervision, SDPO requires careful prompt engineering to incorporate prior trajectories into input context; the proposed method is free of such ad-hoc prompt engineering.

Also, authors provide some theoretical foundation for the proposed method. While the theory does not directly provide guarantees or applicable to realistic settings, it does provide some insight on why the proposed method is advantageous.

Critical weaknesses

The idea of reusing past successful trajectories is on a very active area of research, and authors cite some of them in their literature review. The major concern I have is that it is neither conceptually nor empirically clear the proposed method is advantageous over existing methods. Conceptually, the carrier is only trained on positive trajectories, so it is unclear how much carrier can learn without its access to negative trajectories. Indeed, in Table 1, carrier is behind actor. How then can it be a strong teacher for the actor? Also, empirically, I believe STaR is a clear baseline that should've been compared against, which is essentially "only training the carrier" baseline. In addition, comparisons against other methods which reuse past trajectories, some of which are cited in the literature review and some of which are included in recent papers like https://arxiv.org/abs/2509.25808 , would further strengthen the empirical evidence of the paper, as the proposed algorithm introduces the complexity of having to maintain a carrier model.

Also, authors experiment on a single training setup, hence it is unclear how well does the proposed method generalize to other RLVR setups. In particular, it would be much strengthen the empirical evidence of the paper if authors experimentd on hard math or graph/combinatorial problems, which success probability is very low and the reuse of past positive trajectories is critical.

Questions for authors

It isn't clear in Section 3 that whether Carrier updates only use positive trajectories or negative trajectories are included, because the batch size is not conditional on how many success trajectories there were.

In line 153, it would be great to provide some insight on how D_{t,-b} is calculated.

Proposition 1 is very interesting, equation (7) looks like an advantage computed with log rewards rather than original rewards. Is this form of "advantage" previously known in the RL literature?

In line 214, "one logit per decision, all fixed at zero" -> what do authors mean by "all fixed at zero" if independent decisions are made?

In line 249, I suggest to first describe the intent of setting up an "oracle" (how we characterize it) before going into the mechanis of what it does.

In line 253, Align_J(d) is defined but the actual discussion does not use this definition but rather use "alignment" verablly?

In line 258, \mathcal{S}_{\mathcal{B}} seems to be used without definition?

In line 281, why do authors use Avg@16? Is this a standard metric in SciKnowEval-L3? For reasoning problems, Best@K/Hit@K is more common.

In line 374, it is unclear how to interpret R^2=0.985 without knowing the baseline; it would be easier to interpret if we compute R^2 with the baseline, for ex untrained model.

5 comments
AuthorsOpenReview ↗

Responses to Weaknesses

Learning from negative trajectories.

You raised the concern that training the carrier on successful trajectories may limit its learning without access to negative trajectories. While the carrier's successful-response fitting targets verified correct trajectories, its token-level guidance trains the actor on all rollouts, including failed ones. Thus, the CBSD framework as a whole already provides a pathway for using failed trajectories to update the actor.

We also considered your suggestion to incorporate failed trajectories directly into carrier updates. Simply negating the SFT loss on failed responses yields an objective that is unbounded below, so we used an auxiliary DPO loss on verified correct–incorrect response pairs from the same prompt as an alternative. All other settings remain the same as in the main Physics experiment.

MethodFinal actor accuracy (%)Peak actor accuracy (%)Final carrier accuracy (%)Peak carrier accuracy (%)
CBSD77.8180.1676.8078.05
CBSD + auxiliary carrier DPO73.0574.6171.4174.84

We have not yet observed an accuracy gain from adding auxiliary DPO to explicitly incorporate failed trajectories into carrier updates. These results support the effectiveness of our current approach. However, we acknowledge that they do not rule out the potential benefits of incorporating negative trajectories into the carrier's learning objective.

Carrier accuracy and teaching value.

Indeed, in Table 1, carrier is behind actor. How then can it be a strong teacher for the actor?

Thank you for raising this question. The carrier's standalone generation accuracy does not fully capture its teaching value: it provides token-level guidance on actor-generated prefixes. A carrier with lower overall accuracy may still offer useful guidance at particular prefixes where the actor can improve.

Both branches also evolve throughout training. Through finite updates, the actor learns from successive carrier states while retaining its existing capabilities. We hypothesize that this complementary interaction helps explain why the actor can outperform an individual carrier checkpoint. Table 1 reports independently selected peak accuracies, which do not imply that the carrier consistently trails the actor; the carrier also achieves higher peaks in Chemistry and Materials.

Comparisons with STaR and trajectory-reuse baselines.

To address the missing baseline comparisons you identified, we evaluated STaR and AR3PO with advantage-only reuse. All methods use Qwen3-8B on Physics and report Avg@16, consistent with the paper.

MethodTraining scheduleRollout budget (responses)Training GPU-hoursFinal accuracy (%)Peak accuracy (%)
CBSD800 steps204,800 (1×)169.6577.8180.16
CBSD400 steps102,400 (0.50×)87.3774.9276.41
STaR20 iterations, 800 SFT updates153,816 (0.75×)41.3362.1162.34
AR3PO (advantage-only reuse)800 steps115,164 (0.56×)218.5368.5970.70

Rollout budgets count newly generated training responses, including STaR rationalization; 1× denotes CBSD's 204,800 responses. AR3PO uses adaptive sampling.

At 800 steps, CBSD improves final accuracy over STaR and AR3PO by 15.70 and 9.22 percentage points, respectively, and also achieves higher peak accuracy. To compare with AR3PO at a similar rollout budget, we also report CBSD at 400 steps, where it still achieves better performance. These results support the effectiveness of using a persistent carrier to retain knowledge from past successes and reuse it to guide the actor's current rollouts.

Generalization to harder RLVR tasks.

Following your suggestion, we additionally report hard-math results for Qwen3-8B after 100 training steps on DAPO-Math-17k. We evaluate on AIME2025 using Avg@4 with thinking disabled.

Due to computational and time constraints, we could not retrain SDPO for this comparison. We therefore evaluated the released step-100 checkpoint from Kim et al., Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?. We limit this comparison to step 100 and report only the accuracy at that checkpoint for each method, without selecting peak results over training.

MethodAccuracy (Avg@4, %)
SDPO19.17
GRPO44.17
CBSD33.33

CBSD achieves higher hard-math accuracy than the evaluated SDPO model, but still trails the established GRPO baseline. These results suggest that, in this setting, our approach can strengthen self-distillation for mathematical reasoning, while a gap to direct reward optimization remains.

We hypothesize that the remaining gap to GRPO may arise from the demands of hard mathematical reasoning, which often requires exploring alternative derivations and combining known techniques in unfamiliar ways. While CBSD improves the retention and reuse of successful trajectories, its carrier-based guidance may favor previously learned reasoning patterns, providing less support for discovering new solution paths than direct outcome-based optimization.

AuthorsOpenReview ↗

Responses to Questions For Authors

Q1. Do carrier updates include failed trajectories, and what does BB count?

BB counts all sampled responses, including both successful and failed ones. Without WCV, tokens from failed responses are masked out of the carrier loss. With WCV, they can contribute gradients through the control-variate correction. We have further clarified this point in the main text.

Q2. What motivates the design of Dt,−bD_{t,-b}?

Thank you for this suggestion. Normalizing by the total batch token count introduces dependence on the current response’s random length, which can break the zero-mean property of the WCV correction. We therefore replace that response’s length with the mean length of the other responses. Under independent sampling, this removes the dependence while maintaining the scale of the full-batch token count. We have clarified this rationale in the main text.

Q3. Is the log-ratio advantage in Proposition 1 related to existing RL formulations?

Thank you for this question. The identity π+(a∣h)=π(a∣h)Qπ(h,a)/Vπ(h)\pi^+(a\mid h)=\pi(a\mid h)Q^\pi(h,a)/V^\pi(h) is established in success-conditioning literature. Eq. (7) is its logarithmic form, which Proposition 1 uses to interpret the token-level guidance of an ideally fitted carrier. Here, the logarithms apply to conditional success probabilities, not sampled binary rewards.

Q4. What does “one logit per decision, all fixed at zero” mean?

Each decision has its own parameter: Yi∼Bernoulli(σ(θi))Y_i\sim\mathrm{Bernoulli}(\sigma(\theta_i)), independently across ii. The local comparison is evaluated at θ=(0,…,0)\theta=(0,\ldots,0), so each bit equals one with probability 1/21/2; the sampled bits themselves are not fixed to zero. We have replaced the ambiguous phrase with “we evaluate local updates at the zero-logit actor, with a separate logit for each decision.”

Q5. Clarifying the oracle’s purpose before describing its mechanics.

Thank you for this suggestion. We have revised the main text to explain the oracle’s purpose before describing its mechanics. The oracle represents the best direction obtainable by weighting current-batch response gradients, providing an idealized benchmark for testing whether retained successes offer useful directions beyond those available from the current batch.

Q6. How is Align⁡J(d)\operatorname{Align}_{J}(d) used after its definition?

Thank you for pointing this out. We acknowledge an inconsistency in our presentation of the notation: after defining Align⁡J(d)\operatorname{Align}_{J}(d), we switched to αcar\alpha_{\mathrm{car}} and αorc\alpha_{\mathrm{orc}} without explicitly stating that

αcar=Align⁡J(dcar(1)),αorc=Align⁡J(dorc).\alpha_{\mathrm{car}}=\operatorname{Align}_{J}(d_{\mathrm{car}}^{(1)}), \qquad \alpha_{\mathrm{orc}}=\operatorname{Align}_{J}(d_{\mathrm{orc}}).

The theorem compares the expectations of these two quantities. We have stated these identities explicitly in the main text and made the notation consistent.

Q7. Defining SB\mathcal S_{\mathcal B} before its use.

Thank you for pointing this out. We omitted an explicit introduction of SB\mathcal S_{\mathcal B} before using it in the theorem. We have defined it before its first use and clarified its connection to SBt\mathcal S_{\mathcal B_t} in Section 2. It denotes the span of the current batch's response log-probability gradients, with the time index omitted for the fixed-actor analysis.

Q8. Why use Avg@16 instead of Best@K or Hit@K?

We use Avg@16 to follow the SDPO evaluation protocol on SciKnowEval, keeping our metric consistent with its published baselines. Averaging over 16 responses per question also reduces sampling noise when estimating expected single-response accuracy.

Q9. Providing a reference for interpreting R2=0.985R^2=0.985.

Thank you for this suggestion. We did not provide a clear reference for interpreting R2R^2. Our statistic measures training-induced changes in log-probability. An untrained model has zero change by definition, so we use shuffled correspondence as a reference for this change-based analysis.

As a reference, we shuffled the carrier residuals within groups sharing the same sampling step and verifier outcome. Across 20,000 permutations, mean R2R^2 is 0.00345, compared with 0.985 for the actual correspondence. This supports the interpretation that the actor tracks response-specific carrier changes. We have added this clarification and the permutation baseline to the revised manuscript.

Reviewer_gnWwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Cross-Batch Self-Distillation (CBSD), which jointly trains two policies with different roles. An actor generates responses, while a separate carrier learns from the actor’s verified successes using supervised learning with a variance-reduction correction. The actor then learns through on-policy distillation from the carrier on its own generated responses, including unsuccessful ones. This allows successful experience to inform later updates without retaining the source responses as demonstrations or replay data. The paper provides an idealized theoretical analysis and evaluates Qwen3-8B on four SciKnowEval science domains, reporting improvements over GRPO and SDPO.

Critical strengths

A creative and coherent approach to reusing successful experience. Separating experience retention in the carrier from supervision of the actor provides a clear mechanism for transferring earlier discoveries to fresh trajectories. In particular, the carrier can supply token-level guidance even when the current rollout group contains no successful responses. This is an interesting design for addressing sparse feedback in RLVR.

Critical weaknesses

The contribution of the separate actor–carrier architecture is insufficiently established against simpler alternatives. The carrier is trained to fit successful actor responses, but it is unclear how much the separate actor and distillation stage add over a single model that generates its own rollouts and learns from its verified successes. Reporting carrier accuracy within CBSD does not resolve this question, since that carrier receives trajectories from the jointly evolving actor. Other informative comparisons would include a carrier trained on its own rollouts using GRPO and subsequently distilled into an actor, as well as a sequential train-then-distill approach. Without such comparisons under comparable sampling and computational budgets, it is difficult to determine whether the proposed coupling is essential or whether simpler training procedures would achieve similar gains.

The empirical analysis does not sufficiently isolate the mechanisms responsible for the improvements. The theoretical results establish an idealized interpretation and a constructed setting in which retention helps, but the experimental diagnostics do not demonstrate that these mechanisms explain the observed gains. Substantial within-response variation in carrier advantages does not by itself establish useful token-level credit assignment, and improvements on difficult questions do not isolate the effect of retaining earlier successes. The gains could instead arise primarily from WCV, distillation, or the different treatment of failed trajectories. This matters because, on Physics, removing WCV reduces CBSD from 80.16% to 75.63%, essentially matching the reported SDPO result of 75.60%. Stronger controlled evidence is needed to attribute the improvement to the paper’s central architectural claims.

Questions for authors
  1. Separate actor versus standalone learning: Have you compared CBSD with a single model that generates its own responses and performs online successful-response SFT, with the same applicable variance-reduction correction? How does its performance compare under a matched rollout budget? This would help establish whether the separate actor provides a material benefit.

  2. Choice of carrier training objective: Have you evaluated a carrier trained on its own rollouts using GRPO, while retaining on-policy distillation into a separate actor? Such a comparison would help distinguish the benefit of distillation from that of successful-response fitting and the proposed actor–carrier coupling.

  3. Evidence for retention and token-level credit: Do you have controlled results that remove accumulated carrier history or replace token-specific carrier advantages with their response-level means during training? These comparisons would materially affect my confidence that cross-batch retention and token-level guidance explain the gains, rather than merely accompanying them.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for your constructive comments. As you pointed out, our original submission lacked comparisons with several relevant baselines. Consistent with the paper, we evaluate on the Physics dataset using Avg@16 and report both the peak accuracy during training and the final checkpoint accuracy.

1. Separate actor versus standalone learning.

Thank you for suggesting a direct comparison between our actor–carrier architecture and a single model learning from its own verified successes. To address this, we implemented a single model that generates its own rollouts and learns from verified successes with the same applicable WCV correction, initial model, and 800-step rollout budget. It achieves 71.56% final / 72.19% peak, versus 77.81% / 80.16% for CBSD, giving CBSD improvements of 6.25 / 7.97 percentage points, respectively.

2. Sequential training and online GRPO teachers.

Following your suggestion, we compare CBSD with two alternatives: sequential teacher training followed by actor distillation, and online GRPO training of the carrier on its own rollouts alongside on-policy actor distillation.

In the sequential controls, the trained teacher is frozen, and the student starts from the original initialization and learns through on-policy distillation.

MethodRollout budget (responses)Training GPU-hoursFinal accuracy (%)Peak accuracy (%)
Standalone successful-response fitting + WCV, 800 steps204,800 (1×)139.6371.5672.19
GRPO 400 steps → distillation 400 steps204,800 (1×)266.5863.9867.03
Successful-response fitting + WCV 400 steps → distillation 400 steps204,800 (1×)119.6166.5670.78
GRPO 800 steps → distillation 800 steps409,600 (2×)717.3768.2071.25
Online GRPO carrier + on-policy actor distillation, 400 joint steps204,800 (1×)231.7358.0560.78
Online GRPO carrier + on-policy actor distillation, 600 joint steps307,200 (1.5×)396.3060.1662.34
CBSD, 800 steps204,800 (1×)169.6577.8180.16

Rollout budgets count newly generated training responses; 1× denotes CBSD's 204,800 responses. Each rollout batch contains 32 prompts × 8 responses. Two-stage budgets include both stages; each online joint step generates 512 responses across separate carrier and actor rollouts.

For two-model methods, accuracy refers to the actor/student. CBSD outperforms the sequential alternatives in both final and peak accuracy, including GRPO-800 → distillation-800 with twice CBSD's rollout budget. The online control used each model's own rollouts and remained below CBSD at both 400 joint steps (matched rollout budget) and 600 joint steps (1.5× rollout budget).

We hypothesize that CBSD benefits from continuous actor–carrier interaction. The carrier accumulates the evolving actor’s verified successes through SFT, while on-policy distillation reuses this knowledge to guide subsequent actor responses, including failed attempts.

3. Carrier reset and response-mean advantages.

Thank you for highlighting the need for direct evidence of the contributions of cross-batch retention and token-level guidance. Following your suggestions, we ran two 800-step ablations: resetting the carrier's parameters and Adam state to the current actor's state each batch while preserving the original teaching timing, and replacing token-specific carrier advantages with their within-response means. Both retain WCV and distillation on successful and failed responses.

MethodFinal accuracy (%)Peak accuracy (%)
Refreshed carrier57.6662.03
Response-mean carrier advantages57.9761.64
CBSD77.8180.16

With WCV and distillation on both successful and failed responses retained, carrier reset and response-mean substitution reduce final accuracy by 20.16 and 19.84 percentage points, respectively, and also lower peak accuracy. These results support the contributions of persistent carrier state and token-level guidance beyond WCV and failed-response distillation alone.

AuthorsOpenReview ↗

Kindly requesting for feedback on our rebuttal

Dear Reviewer,

Thank you again for your detailed review. Following your suggestions, our rebuttal includes comparisons with standalone successful-response fitting using the same WCV correction, sequential train-then-distill alternatives, and an online GRPO-trained carrier with on-policy actor distillation (W1/Q1–Q2). We report final and peak accuracy alongside rollout budgets and training GPU-hours.

We also completed the refreshed-carrier and response-mean advantage controls you requested, retaining WCV and distillation on both successful and failed responses (W2/Q3).

We would be grateful to know whether these comparisons address your concerns about the separate actor–carrier architecture and the contributions of persistence and token-level guidance. If any concerns remain, we would welcome further discussion.

If time permits, we would appreciate your feedback during the discussion period.

Thank you for your time and consideration.

Thank you for the substantial additional experiments. The new comparisons and ablations address my main concerns about the actor–carrier architecture and strengthen the evidence for persistent carrier state and token-level guidance. Although the additional results are limited to Physics, they substantially improve my assessment, and I am raising my rating. Please incorporate these results into the revised paper.

Reviewer_grCSOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CBSD trains a persistent carrier on successful responses and uses its token log-ratios to supervise later policy updates, including unsuccessful groups. On four science domains, the reported average of selected checkpoint maxima is 76.22%, versus 72.93% for SDPO.

Critical strengths
  • The carrier reuses successful experience without additional carrier rollouts or storing source trajectories.
  • On 27 Physics prompts preselected using separate initial generations, the step-200 gain over GRPO is 29.40 points, with a 95% prompt-cluster interval of [16.44,42.59][16.44,42.59].
  • Timing, write-control-variate ablations, and GPU-hour measurements are useful.
Critical weaknesses
  1. Mechanism: Existing ablations retain carrier persistence. A matched reset/refreshed-carrier control is needed to establish the distinctive benefit.
  2. Selection: Headline results combine checkpoint maxima, domain-specific coefficient selection, and mostly literature baselines. Independent training-run uncertainty and comparable selection budgets are unclear.
  3. Theory: Equation (3)'s token normalization need not recover the ideal target: two equally likely successful responses of lengths 1 and 2, with batch size 2, yield optimum (1017,717)(\frac{10}{17},\frac{7}{17}), rather than (12,12)(\frac12,\frac12).
Questions for authors
  1. Can you provide a matched carrier-persistence control, including rollout and update budgets?
  2. Which validation split and search budgets selected checkpoints and write-control coefficients? Please report independent-run uncertainty.
  3. How does Equation (3)'s normalization connect to the ideal target, and what sampling assumptions justify the control variate?
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

1. Carrier reset control.

Thank you for pointing out that our original ablations retained carrier persistence and therefore did not directly isolate its contribution. Following your suggestion, we ran a refreshed-carrier control on Physics: each batch, we replace the carrier's parameters and Adam state with the current actor's state before the carrier write. We cache the carrier's guidance before refreshing it to preserve the original teaching timing. CBSD and the refreshed-carrier control use the same initialization and matched rollout and update budgets. All other settings remain the same as those used in the paper.

MethodStepRollout budget (responses)Final accuracy (%)Peak accuracy (%)
GRPO800204,80070.6371.25
Refreshed carrier800204,80057.6662.03
CBSD800204,80077.8180.16

Refreshing the carrier reduces final and peak accuracy by 20.16 and 18.13 percentage points, respectively. These results support the benefit of maintaining carrier-specific parameter and optimizer state across batches.

2. Model selection and training-run uncertainty.

We did not reserve a separate validation set for model selection. We follow SDPO's public SciKnowEval-L3 90/10 split (seed 42), use the same rollout batch size (32 prompts × 8 responses), and select checkpoints by peak Avg@16 on the evaluation split. Each CBSD configuration is trained for 800 steps and evaluated every 20 steps.

WCV introduces an additional coefficient, for which we tested λcv∈{0,0.5,1}\lambda_{\mathrm{cv}}\in\{0,0.5,1\}. Using a uniform λcv=0.5\lambda_{\mathrm{cv}}=0.5 across all four domains yields an average accuracy of 75.67%, versus 76.22% with domain-specific selection, and still exceeds SDPO's reported 72.93%.

To quantify variability across independent training runs, we completed a three-seed experiment on SciKnowEval-L3 Materials with fixed λcv=1\lambda_{\mathrm{cv}}=1 and the same data split. Due to limited computational resources, each run was limited to 400 training steps.

SeedFinal accuracy (%)Peak accuracy (%)
4276.5376.53
4376.9376.99
4474.4776.86
Mean ± sample SD75.98 ± 1.3276.80 ± 0.24

These results suggest that CBSD maintains stable performance across training seeds.

3. Token normalization, the ideal target, and WCV assumptions.

Thank you for pointing this out. Your counterexample is correct: Eq. (3)'s random denominator reweights responses by length, giving (10/17,7/17)(10/17,7/17) instead of (1/2,1/2)(1/2,1/2). We have revised the main text to clarify the scope of the ideal-target guarantee and avoid misleading readers.

The revised Proposition 1 explicitly states the ideal population objective Eτ∼π(⋅∣x)[−R(x,τ)log⁡q(τ∣x)]\mathbb E_{\tau\sim\pi(\cdot\mid x)}[-R(x,\tau)\log q(\tau\mid x)], whose optimum is π+\pi^+. Eq. (3) has the same optimum when its token denominator is deterministic, because normalization then only rescales the objective. With variable lengths, this equivalence need not hold.

WCV preserves the expected update of Eq. (3), including any length reweighting: E[W^WCV∣Ht]=E[W^+∣Ht]\mathbb E[\widehat W_{\mathrm{WCV}}\mid\mathcal H_t]=\mathbb E[\widehat W_+\mid\mathcal H_t]. This result assumes conditionally independent responses given prompts and history, full actor support, bounded response lengths, unclipped importance ratios, and a WCV coefficient chosen before sampling. Conditioning on the other responses fixes the leave-one-response baseline and denominator. The importance-weighted score has zero conditional mean, so the correction also averages to zero (Appendix B.3).

AuthorsOpenReview ↗

Request for feedback on our rebuttal

Dear Reviewer,

Thank you again for your detailed review. Our rebuttal addresses your concerns with a refreshed-carrier control under matched rollout and update budgets (W1/Q1), clarification of checkpoint and WCV coefficient selection, results with a uniform coefficient, and a three-seed Materials experiment (W2/Q2). We also address your normalization counterexample and clarify the assumptions underlying WCV’s expectation-preservation guarantee (W3/Q3). Proposition 1 has been revised to state the expected loss explicitly and restrict the ideal-target guarantee to the stated normalization assumptions.

We would be grateful to know whether these additions address your concerns, particularly regarding carrier persistence, model selection and training-run uncertainty, and the scope of the theoretical guarantees. If any concerns remain, we would welcome further discussion.

If time permits, we would appreciate your feedback during the discussion period.

Thank you for your time and consideration.

Reviewer_grCSOpenReview ↗

Review to the rebuttal

Thank you for the thorough rebuttal. The refreshed-carrier and response-mean controls substantially address my concerns about carrier persistence and token-specific guidance, and the theoretical clarifications resolve my normalization objection. Although I still think checkpoint and coefficient selection on the reported test split, together with limited full-configuration run uncertainty, should be clearly acknowledged, the additional controls and clarifications sufficiently strengthen the paper for me. I have therefore raised my rating from 2 to 3 (weak acceptance). Please include the new controls and clearly describe the evaluation-selection protocol and remaining limitations in the revision.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank all reviewers for their thorough and constructive feedback. Your insights have been immensely helpful, and we will incorporate all of these results into the main text of the revised manuscript. We are grateful that all three reviewers found our rebuttal responsive and raised their scores.

Summary of our rebuttal:

  • Isolating Core Mechanisms (Reviewers grCS & gnWw): On SciKnowEval-L3 Physics (matched budget, retaining WCV and failed-trajectory distillation), resetting the carrier each batch reduces peak Avg@16 from 80.16% to 62.03%, and replacing token-level carrier advantages with response means reduces it to 61.64%. These controls confirm that persistent cross-batch carrier memory and token-level guidance drive our gains beyond WCV and failed-response distillation alone.

  • Architecture & Trajectory-Reuse Baselines (Reviewers gnWw & THpf): On Physics (relative to CBSD's 800-step, 1×1\times budget and 80.16% peak Avg@16), continuous actor–carrier coupling substantially outperforms alternative designs and trajectory-reuse methods under matched or larger rollout budgets:

    • Alternative designs: Standalone successful-response SFT with WCV reaches 72.19% (1×1\times); two-stage distillation reaches 67.03%–70.78% (1×1\times) and 71.25% (2×2\times); online GRPO carrier with distillation reaches 60.78% (1×1\times) and 62.34% (1.5×1.5\times).

    • Trajectory reuse: STaR reaches 62.34% (0.75×0.75\times) and AR3PO reaches 70.70% (0.56×0.56\times), whereas 400-step CBSD achieves 76.41% (0.50×0.50\times).

  • Negative Trajectories & Carrier Teaching Value (Reviewer THpf):

    • CBSD already leverages failed rollouts by distilling carrier guidance across all actor trajectories; as a naive attempt, adding an auxiliary DPO loss on same-prompt correct–incorrect pairs to carrier updates lowers Physics actor peak Avg@16 from 80.16% to 74.61% (carrier: 78.05% to 74.84%), supporting our design of fitting the carrier solely on verified successes.

    • We explained why a carrier with lower standalone Avg@16 can still effectively guide the actor: instead of generating full solutions, it provides localized token-level credit conditioned on actor-generated prefixes, while finite co-evolution allows the actor to integrate successive carrier states while retaining its own capabilities.

  • Robustness, Uncertainty & Hard Math (Reviewers grCS & THpf): A uniform λcv=0.5\lambda_{\mathrm{cv}}=0.5 achieves 75.67% four-domain average peak Avg@16 (vs. SDPO's 72.93%), and three seeds on Materials yield 76.80±0.24%76.80 \pm 0.24\% peak Avg@16, demonstrating insensitivity to domain-specific tuning and low cross-seed variance. On DAPO-Math-17k →\rightarrow AIME2025 (step-100 Avg@4), CBSD (33.33%) substantially improves self-distillation over SDPO (19.17%) while trailing GRPO (44.17%), suggesting that trajectory reuse strengthens self-distillation on hard math, though we hypothesize that favoring learned patterns limits novel path exploration relative to direct reward optimization.

  • Theory & Clarifications (Reviewers grCS & THpf): We revised Proposition 1 to specify the expected-loss / deterministic-denominator conditions for recovering π+\pi^+ and detailed WCV's conditional independence assumptions, clarifying the exact scope of our theoretical guarantees. We also added a permutation baseline confirming that the actor tracks response-specific carrier changes (R2R^2), and clarified BB, Dt,−bD_{t,-b}, Align⁡J\operatorname{Align}_J, SB\mathcal{S}_{\mathcal{B}}, and the oracle setup.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.