- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes CBPO, a training framework that unifies both SFT and RL. The intuition is to fit the policy point-wise towards the optimal policy under the RL objective, and the new objective is independent to the behavior policy, thus allowing off-policy and on-policy data source. The experiment is done on Qwen3-1.7B and 8B on math benchmarks, compared with offline, on-policy, off-policy and joint SFT/RL baselines. CBPO-unified outperforms all baselines on all benchmarks.
Critical strengths
- The paper is rigorous in experiment design. Each configuration is repeated on 5 training seeds, 4 benchmarks and 2 model scales are tested. There is a careful ablation study, and all methods use similar compute.
- The paper proposes a simple framework to use all data available during post-training, making a good contribution to deal with data efficiency for model post-training.
Critical weaknesses
- The "behavior policy independence" in Line 193-198 is not that strong as claimed. Though the point-wise loss Equation (4) is independent on the behavior policy, the actual loss optimized is the expectation of it over , so the behavior policy affects the policy that is eventually learned. The theory in appendix implicitly acknowledges this, but I would expect it to be claimed in the main text as well.
- CBPO-Unified uses positive offline demonstrations, negative offline demonstrations, reference-policy responses, current and historical rollouts as training data. No other baselines in the experiments use all of them. It is therefore questionable whether the performance gain arises from more data source or from the method itself. Nonetheless, the ability of CBPO to use all source of data can be regarded as the contribution itself.
Questions for authors
Is there any known training workflow that uses all source of data as in CBPO-Unified? It might not be a single stage, but consists of multiple separate stages, e.g., SFT + DPO + RL. If there is, a comparison with such a workflow is beneficial to show the effect of CBPO from the algorithm side.
3 comments
Thanks for the authors' response. These address my concern. I will keep my score.
Thank you for the careful follow-up and for verifying the new analyses. We appreciate the constructive assessment and will incorporate all of the requested clarifications and results into the revision.
Dear Reviewer Xw9p
W1
We thank the reviewer for pointing out that our claim should distinguish the pointwise loss from its expectation under the training distribution. Although μ does not appear in the target or pointwise loss for a fixed record, it determines which completions are observed and how they are weighted in the training objective. Therefore, changing the behavior policy can affect the policy learned by optimization.
We will revise the main-text discussion in "behavior policy independence" to clarify that behavior-policy independence applies to the per-record regression target and per-record loss, not to the expected training objective. This distinction is also reflected in the Appendix, where the buffer-sampling distribution determines the weights of records in the aggregate objective.
W2/Q1
We thank the reviewer for highlighting this attribution issue. We agree that the original experiments do not separate the effect of broader data-source coverage from the effect of the training method. CBPO-Unified is the only original baseline trained on all four source categories.
To assess whether CBPO offers an additional advantage when data-source coverage is held fixed, we added a comparison with SPO and AGRO. We train SPO and AGRO from base policy Qwen3-1.7B using the same source pool under matched settings and evaluation protocol on CBPO.
With source coverage and training budget matched, AGRO and SPO obtain aggregate avg@32/pass@32 scores of 43.21/65.22 and 43.45/65.36, respectively, whereas CBPO-Unified obtains 44.57/67.40. CBPO-Unified therefore retains its aggregate advantage under matched source coverage and training budget, with gains of 1.36/2.18 points over AGRO and 1.12/2.04 points over SPO.
We clarified in Section 4 that behavior-policy independence applies to the per-record target and loss, while the behavior policy determines which completions are observed and the buffer-sampling distribution determines their weights in the aggregate objective. We added the corresponding source-mixture analysis to Appendix A. We expanded the Section 2 discussion of AGRO and SPO and reported their comparison with CBPO-Unified under matched data sources and training budgets in Appendix C.3.