- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Authors propose a novel training algorithm with verifiable rewards, which use past successful trajectories in a novel way: do a (variation of) SFT on positive examples on the "carrier" model, and then run an On-Policy Distillation from this carrier model to train an actor. The carrier update employs a correction for the divergence between actor and the carrier in order to stabilize carrier learning, which authors call 'Write Control Variate'. Authors offer two theoretical analysis for the insights. First, authors show that the token-level credit in on-policy distillation can be viewed as log-advantage. Authors also show with the toy problem (learning first m bits of n-bit string should be turned on) that reusing previous samples leads to faster learning. Empirically, authors experiment on SciKnowEval-L3 with Qwen 3-8B models to demonstrate the advantage over GRPO and SDPO, which is consistent across domains.
Critical strengths
Authors propose a novel method of leveraging past positive trajectories. This topic is gathering a lot of attention in the literature, as the popular GRPO baseline is clearly very sample-inefficient on problems with low success probability. Authors leverage token-level supervision for both carrier and actor, which is a relative strength against GRPO baseline. On the other hand, while SDPO provides token-level supervision, SDPO requires careful prompt engineering to incorporate prior trajectories into input context; the proposed method is free of such ad-hoc prompt engineering.
Also, authors provide some theoretical foundation for the proposed method. While the theory does not directly provide guarantees or applicable to realistic settings, it does provide some insight on why the proposed method is advantageous.
Critical weaknesses
The idea of reusing past successful trajectories is on a very active area of research, and authors cite some of them in their literature review. The major concern I have is that it is neither conceptually nor empirically clear the proposed method is advantageous over existing methods. Conceptually, the carrier is only trained on positive trajectories, so it is unclear how much carrier can learn without its access to negative trajectories. Indeed, in Table 1, carrier is behind actor. How then can it be a strong teacher for the actor? Also, empirically, I believe STaR is a clear baseline that should've been compared against, which is essentially "only training the carrier" baseline. In addition, comparisons against other methods which reuse past trajectories, some of which are cited in the literature review and some of which are included in recent papers like https://arxiv.org/abs/2509.25808 , would further strengthen the empirical evidence of the paper, as the proposed algorithm introduces the complexity of having to maintain a carrier model.
Also, authors experiment on a single training setup, hence it is unclear how well does the proposed method generalize to other RLVR setups. In particular, it would be much strengthen the empirical evidence of the paper if authors experimentd on hard math or graph/combinatorial problems, which success probability is very low and the reuse of past positive trajectories is critical.
Questions for authors
It isn't clear in Section 3 that whether Carrier updates only use positive trajectories or negative trajectories are included, because the batch size is not conditional on how many success trajectories there were.
In line 153, it would be great to provide some insight on how D_{t,-b} is calculated.
Proposition 1 is very interesting, equation (7) looks like an advantage computed with log rewards rather than original rewards. Is this form of "advantage" previously known in the RL literature?
In line 214, "one logit per decision, all fixed at zero" -> what do authors mean by "all fixed at zero" if independent decisions are made?
In line 249, I suggest to first describe the intent of setting up an "oracle" (how we characterize it) before going into the mechanis of what it does.
In line 253, Align_J(d) is defined but the actual discussion does not use this definition but rather use "alignment" verablly?
In line 258, \mathcal{S}_{\mathcal{B}} seems to be used without definition?
In line 281, why do authors use Avg@16? Is this a standard metric in SciKnowEval-L3? For reasoning problems, Best@K/Hit@K is more common.
In line 374, it is unclear how to interpret R^2=0.985 without knowing the baseline; it would be easier to interpret if we compute R^2 with the baseline, for ex untrained model.
5 comments
Comparisons with STaR and trajectory-reuse baselines.
To address the missing baseline comparisons you identified, we evaluated STaR and AR3PO with advantage-only reuse. All methods use Qwen3-8B on Physics and report Avg@16, consistent with the paper.
| Method | Training schedule | Rollout budget (responses) | Training GPU-hours | Final accuracy (%) | Peak accuracy (%) |
|---|---|---|---|---|---|
| CBSD | 800 steps | 204,800 (1×) | 169.65 | 77.81 | 80.16 |
| CBSD | 400 steps | 102,400 (0.50×) | 87.37 | 74.92 | 76.41 |
| STaR | 20 iterations, 800 SFT updates | 153,816 (0.75×) | 41.33 | 62.11 | 62.34 |
| AR3PO (advantage-only reuse) | 800 steps | 115,164 (0.56×) | 218.53 | 68.59 | 70.70 |
Rollout budgets count newly generated training responses, including STaR rationalization; 1× denotes CBSD's 204,800 responses. AR3PO uses adaptive sampling.
At 800 steps, CBSD improves final accuracy over STaR and AR3PO by 15.70 and 9.22 percentage points, respectively, and also achieves higher peak accuracy. To compare with AR3PO at a similar rollout budget, we also report CBSD at 400 steps, where it still achieves better performance. These results support the effectiveness of using a persistent carrier to retain knowledge from past successes and reuse it to guide the actor's current rollouts.
Generalization to harder RLVR tasks.
Following your suggestion, we additionally report hard-math results for Qwen3-8B after 100 training steps on DAPO-Math-17k. We evaluate on AIME2025 using Avg@4 with thinking disabled.
Due to computational and time constraints, we could not retrain SDPO for this comparison. We therefore evaluated the released step-100 checkpoint from Kim et al., Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?. We limit this comparison to step 100 and report only the accuracy at that checkpoint for each method, without selecting peak results over training.
| Method | Accuracy (Avg@4, %) |
|---|---|
| SDPO | 19.17 |
| GRPO | 44.17 |
| CBSD | 33.33 |
CBSD achieves higher hard-math accuracy than the evaluated SDPO model, but still trails the established GRPO baseline. These results suggest that, in this setting, our approach can strengthen self-distillation for mathematical reasoning, while a gap to direct reward optimization remains.
We hypothesize that the remaining gap to GRPO may arise from the demands of hard mathematical reasoning, which often requires exploring alternative derivations and combining known techniques in unfamiliar ways. While CBSD improves the retention and reuse of successful trajectories, its carrier-based guidance may favor previously learned reasoning patterns, providing less support for discovering new solution paths than direct outcome-based optimization.
Responses to Questions For Authors
Q1. Do carrier updates include failed trajectories, and what does count?
counts all sampled responses, including both successful and failed ones. Without WCV, tokens from failed responses are masked out of the carrier loss. With WCV, they can contribute gradients through the control-variate correction. We have further clarified this point in the main text.
Q2. What motivates the design of ?
Thank you for this suggestion. Normalizing by the total batch token count introduces dependence on the current response’s random length, which can break the zero-mean property of the WCV correction. We therefore replace that response’s length with the mean length of the other responses. Under independent sampling, this removes the dependence while maintaining the scale of the full-batch token count. We have clarified this rationale in the main text.
Q3. Is the log-ratio advantage in Proposition 1 related to existing RL formulations?
Thank you for this question. The identity is established in success-conditioning literature. Eq. (7) is its logarithmic form, which Proposition 1 uses to interpret the token-level guidance of an ideally fitted carrier. Here, the logarithms apply to conditional success probabilities, not sampled binary rewards.
Q4. What does “one logit per decision, all fixed at zero” mean?
Each decision has its own parameter: , independently across . The local comparison is evaluated at , so each bit equals one with probability ; the sampled bits themselves are not fixed to zero. We have replaced the ambiguous phrase with “we evaluate local updates at the zero-logit actor, with a separate logit for each decision.”
Q5. Clarifying the oracle’s purpose before describing its mechanics.
Thank you for this suggestion. We have revised the main text to explain the oracle’s purpose before describing its mechanics. The oracle represents the best direction obtainable by weighting current-batch response gradients, providing an idealized benchmark for testing whether retained successes offer useful directions beyond those available from the current batch.
Q6. How is used after its definition?
Thank you for pointing this out. We acknowledge an inconsistency in our presentation of the notation: after defining , we switched to and without explicitly stating that
The theorem compares the expectations of these two quantities. We have stated these identities explicitly in the main text and made the notation consistent.
Q7. Defining before its use.
Thank you for pointing this out. We omitted an explicit introduction of before using it in the theorem. We have defined it before its first use and clarified its connection to in Section 2. It denotes the span of the current batch's response log-probability gradients, with the time index omitted for the fixed-actor analysis.
Q8. Why use Avg@16 instead of Best@K or Hit@K?
We use Avg@16 to follow the SDPO evaluation protocol on SciKnowEval, keeping our metric consistent with its published baselines. Averaging over 16 responses per question also reduces sampling noise when estimating expected single-response accuracy.
Q9. Providing a reference for interpreting .
Thank you for this suggestion. We did not provide a clear reference for interpreting . Our statistic measures training-induced changes in log-probability. An untrained model has zero change by definition, so we use shuffled correspondence as a reference for this change-based analysis.
As a reference, we shuffled the carrier residuals within groups sharing the same sampling step and verifier outcome. Across 20,000 permutations, mean is 0.00345, compared with 0.985 for the actual correspondence. This supports the interpretation that the actor tracks response-specific carrier changes. We have added this clarification and the permutation baseline to the revised manuscript.
Responses to Weaknesses
Learning from negative trajectories.
You raised the concern that training the carrier on successful trajectories may limit its learning without access to negative trajectories. While the carrier's successful-response fitting targets verified correct trajectories, its token-level guidance trains the actor on all rollouts, including failed ones. Thus, the CBSD framework as a whole already provides a pathway for using failed trajectories to update the actor.
We also considered your suggestion to incorporate failed trajectories directly into carrier updates. Simply negating the SFT loss on failed responses yields an objective that is unbounded below, so we used an auxiliary DPO loss on verified correct–incorrect response pairs from the same prompt as an alternative. All other settings remain the same as in the main Physics experiment.
We have not yet observed an accuracy gain from adding auxiliary DPO to explicitly incorporate failed trajectories into carrier updates. These results support the effectiveness of our current approach. However, we acknowledge that they do not rule out the potential benefits of incorporating negative trajectories into the carrier's learning objective.
Carrier accuracy and teaching value.
Thank you for raising this question. The carrier's standalone generation accuracy does not fully capture its teaching value: it provides token-level guidance on actor-generated prefixes. A carrier with lower overall accuracy may still offer useful guidance at particular prefixes where the actor can improve.
Both branches also evolve throughout training. Through finite updates, the actor learns from successive carrier states while retaining its existing capabilities. We hypothesize that this complementary interaction helps explain why the actor can outperform an individual carrier checkpoint. Table 1 reports independently selected peak accuracies, which do not imply that the carrier consistently trails the actor; the carrier also achieves higher peaks in Chemistry and Materials.