acceptodds
Under review as a conference paper at ICLR 2027

Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target

Abstract

Reasoning post-training draws on offline demonstrations and policy-generated rollouts, yet conventional supervised fine-tuning (SFT) followed by reinforcement learning (RL) separates their objectives and limits continued reuse of collected responses. We propose Cross-Behavior Policy Optimization (CBPO), which makes these sources compatible with one regression loss through a fixed-reference target. Because the generating policy does not enter this target, positive and negative offline completions, reference responses, and current and historical rollouts can share a common reward-labeled record format and replay buffer. As new responses enter the buffer, optimization retains the same regression objective without coordinating separate SFT and RL objectives or requiring stored behavior-policy probabilities, importance weighting, or a learned critic. Restricting the available sources gives offline, on-policy, and off-policy variants, while CBPO-Unified combines them. Across two model scales and four mathematical reasoning benchmarks, CBPO delivers leading avg@32 and pass@32 performance across benchmarks and data regimes. On Qwen3-1.7B, CBPO-Unified improves benchmark-averaged avg@32 and pass@32 over SFTDAPO by 2.33 and 4.44 percentage points, respectively. In the replay ablation, retaining historical rollouts improves these metrics by 1.87 and 3.72 percentage points over the no-history configuration.

Primary area:
reinforcement learning
Keywords:
offline-to-online reinforcement learning, shared regression target, reinforcement learning with verifiable rewards (RLVR)

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_Xw9pOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CBPO, a training framework that unifies both SFT and RL. The intuition is to fit the policy point-wise towards the optimal policy under the RL objective, and the new objective is independent to the behavior policy, thus allowing off-policy and on-policy data source. The experiment is done on Qwen3-1.7B and 8B on math benchmarks, compared with offline, on-policy, off-policy and joint SFT/RL baselines. CBPO-unified outperforms all baselines on all benchmarks.

Critical strengths
  1. The paper is rigorous in experiment design. Each configuration is repeated on 5 training seeds, 4 benchmarks and 2 model scales are tested. There is a careful ablation study, and all methods use similar compute.
  2. The paper proposes a simple framework to use all data available during post-training, making a good contribution to deal with data efficiency for model post-training.
Critical weaknesses
  1. The "behavior policy independence" in Line 193-198 is not that strong as claimed. Though the point-wise loss lCBPO(x,y;θ)l_{\text{CBPO}}(x,y;\theta) Equation (4) is independent on the behavior policy, the actual loss optimized is the expectation of it over y∼μ(⋅∣x)y\sim \mu(\cdot\mid x), so the behavior policy affects the policy that is eventually learned. The theory in appendix implicitly acknowledges this, but I would expect it to be claimed in the main text as well.
  2. CBPO-Unified uses positive offline demonstrations, negative offline demonstrations, reference-policy responses, current and historical rollouts as training data. No other baselines in the experiments use all of them. It is therefore questionable whether the performance gain arises from more data source or from the method itself. Nonetheless, the ability of CBPO to use all source of data can be regarded as the contribution itself.
Questions for authors

Is there any known training workflow that uses all source of data as in CBPO-Unified? It might not be a single stage, but consists of multiple separate stages, e.g., SFT + DPO + RL. If there is, a comparison with such a workflow is beneficial to show the effect of CBPO from the algorithm side.

3 comments
AuthorsOpenReview ↗

Dear Reviewer Xw9p

W1

On the scope of behavior-policy independence

We thank the reviewer for pointing out that our claim should distinguish the pointwise loss from its expectation under the training distribution. Although μ\mu does not appear in the target or pointwise loss for a fixed record, it determines which completions are observed and how they are weighted in the training objective. Therefore, changing the behavior policy can affect the policy learned by optimization.

We will revise the main-text discussion in "behavior policy independence" to clarify that behavior-policy independence applies to the per-record regression target and per-record loss, not to the expected training objective. This distinction is also reflected in the Appendix, where the buffer-sampling distribution determines the weights of records in the aggregate objective.


W2/Q1

On comparison to existing workflow using all source of data

We thank the reviewer for highlighting this attribution issue. We agree that the original experiments do not separate the effect of broader data-source coverage from the effect of the training method. CBPO-Unified is the only original baseline trained on all four source categories.

To assess whether CBPO offers an additional advantage when data-source coverage is held fixed, we added a comparison with SPO and AGRO. We train SPO and AGRO from base policy Qwen3-1.7B using the same source pool under matched settings and evaluation protocol on CBPO.

MethodsAIME25 (avg@32 / pass@32)MinervaMath (avg@32 / pass@32)AMC23 (avg@32 / pass@32)MATH500 (avg@32 / pass@32)Avg. (avg@32 / pass@32)
AGRO19.17 / 43.3323.56 / 41.5458.20 / 90.0071.89 / 86.0043.21 / 65.22
SPO19.27 / 43.3323.76 / 41.9158.44 / 90.0072.32 / 86.2043.45 / 65.36
CBPO-Unified20.63 / 46.6724.48 / 42.6560.16 / 92.5073.01 / 87.8044.57 / 67.40

With source coverage and training budget matched, AGRO and SPO obtain aggregate avg@32/pass@32 scores of 43.21/65.22 and 43.45/65.36, respectively, whereas CBPO-Unified obtains 44.57/67.40. CBPO-Unified therefore retains its aggregate advantage under matched source coverage and training budget, with gains of 1.36/2.18 points over AGRO and 1.12/2.04 points over SPO.

We clarified in Section 4 that behavior-policy independence applies to the per-record target and loss, while the behavior policy determines which completions are observed and the buffer-sampling distribution determines their weights in the aggregate objective. We added the corresponding source-mixture analysis to Appendix A. We expanded the Section 2 discussion of AGRO and SPO and reported their comparison with CBPO-Unified under matched data sources and training budgets in Appendix C.3.

Thanks for the authors' response. These address my concern. I will keep my score.

Thank you for the careful follow-up and for verifying the new analyses. We appreciate the constructive assessment and will incorporate all of the requested clarifications and results into the revision.

Reviewer_P7QDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Cross-Behavior Policy Optimization (CBPO), a training method/pipeline that could jointly train the model from demonstrations, responses from the reference model, and previously generated rollouts. Theoretically, the paper provides theoretical analysis of normalizer estimation and mixed-sourced training. Empeirically, the paper studies the math reasoning setting with Qwen3-1.7B/8B models and observes improvements compared with selected baselines.

Notably, the core training objectice and pipeline losely resemble A*-PO, with the principal algorithmic extension being joint training on multiple data sources through a persistent replay.

Critical strengths
  1. Important topic. Continuously using demonstrations and previously generated, reward-labeled responses is a important research question. The fixed-target record format provides a simple way to train on these sources together. This practical motivation remains valid even if the basic regression objective is closely related to the prior work A*-PO.

  2. Interesting ablation. The path-length ratio and cumulative turning analysis makes differences around the SFT-to-RL transition observable. I find this observation very interesting as a diagnostic, while recognizing that a straighter projected path alone does not establish that objective conflict causes the final quality difference.

Critical weaknesses
  1. Despite theoretical extensions, the training algorithm appears to be an A-PO variant that changes the replay buffer's contents and retention. CBPO provides additional theoretical analysis, but its practical training pipeline and objective closely resemble A*-PO. Both methods first sample from a fixed reference policy to precompute prompt-specific targets, then alternate collecting reward-labeled current-policy responses with regression updates against those fixed targets. Using CBPO's notation, the basic A*-PO loss (Algorithm 1 and §3 of Brantley et al., 2025) is

ℓA∗-PO(x,y;θ)=[βlog⁡πθ(y∣x)πref(y∣x)−(r(x,y)−V^(x))]2,\ell_{\mathrm{A^*\text{-}PO}}(x,y;\theta)=\left[\beta\log\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}-\left(r(x,y)-\widehat V(x)\right)\right]^2,

where V^(x)=βlog⁡Z^ref(x).\widehat V(x)=\beta\log\widehat Z_{\mathrm{ref}}(x).

CBPO's Eq. (4) is

ℓCBPO(x,y;θ)=[log⁡πθ(y∣x)πref(y∣x)−r(x,y)β+log⁡Z^ref(x)]2.\ell_{\mathrm{CBPO}}(x,y;\theta)=\left[\log\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}-\frac{r(x,y)}{\beta}+\log\widehat Z_{\mathrm{ref}}(x)\right]^2.

Therefore ℓA∗-PO=β2,ℓCBPO.\boxed{\ell_{\mathrm{A^*\text{-}PO}}=\beta^2\\,\ell_{\mathrm{CBPO}}.} where β\beta is a chosen hyper-parameter.

At the algorithmic level, compared to. A*-PO, the principal extension appears to be a persistent multi-source replay buffer: teacher demonstrations and reference responses become training records, previous rollouts are retained. The changes therefore primarily concern data inclusion, retention, sampling, and reuse. I acknowledge the additional theoretical analysis in §4.4 and Appendix A. These analytical contributions should be assessed separately; the concern here is that the algorithmic extension and its empirical value relative to A*-PO are not sufficiently isolated or established.

  1. The generation budget and baseline weaken the reliability of the experiment results. Appendix B explicitly set the max generation length as 2,048 tokens for teacher and training rollouts with thinking model enabled. I think 2k tokens is too short for modern reasoning models. Meanwhile response-length distributions, cap-hit/unfinished rates and longer-generation-budget controls for demonstration data and eval are not reported. The original Qwen3 checkpoints are also absent from the main comparison, making it difficult to establish the actual gain over initialization. The current exp results need further validation under an explicit evaluation protocol and reasonable generation choices.

  2. Missing citation, comparaison or dicussion with related works I could not find a discussion of OAPL, which studies off-policy advantage regression with a buffer, or Prefix-RFT, which integrates demonstrations and exploration through prefix-conditioned trajectories. These methods are not identical to CBPO, but explaining the distinctions with these methods is necessary to establish what is new about CBPO's training. A*-PO is already cited; the issue there is substantive differentiation and missing direct comparisons, rather than a missing citation.

Questions for authors
  1. Please refer to Weaknesses 1 and 2 regarding the contribution relative to A*-PO, as well as generation limits chosen, and robustness to longer generation budgets.

  2. Training on positive demonstrations only with the same loss. What performance would CBPO achieve if its objective were used to train only on the same positive demonstrations used by SFT, without reference responses or rollouts as training data? How would this compare with standard SFT?

  3. Replay age versus source composition (§5.3 and Table 4). Full replay improves aggregate avg@32/pass@32 from 43.07/65.80 with FIFO-1 to 44.56/68.72. However, the buffer definition retains the offline/reference responses while increasing the number of rollout records; uniform record sampling therefore also increases the probability of sampling training rollout. My question is does full-history replay still outperform FIFO if each training batch uses the same proportions of demonstrations, reference responses, and rollouts in both methods? In particular, can progressively reducing demonstration/reference training shares in FIFO recover the improvement, while leaving the frozen normalizer-estimation bank unchanged?

  4. SFT learning-rate tuning (§5.1). Section 5.1 states that actor learning rates are matched across methods, so SFT and RL use the same learning rate in the reported comparisons if I understood right. Since SFT may benefit from a higher learning rate than RL, did you independently search SFT's learning rate, including larger values than the RL setting? Do the comparisons in Table 3 remain consistent with an independently tuned SFT baseline?

**Additional validation suggestion:**I think AIME 2025 for qwen3-series model is quite old. AIME26 could be a better choice. Evaluation in additional settings and on other model families would also be appreciated.

5 comments
AuthorsOpenReview ↗

Dear Reviewer P7QD(1)

W1/Q1

On the contribution relative to A*-PO

We thank the reviewer for highlighting the need to clarify the difference between CBPO and A*-PO.

The A*-PO's loss presented in their paper is a simplified version. Its actual loss has two different β\beta, β1=0.5\beta_1=0.5 for value estimation β1log⁡Z^ref(x)\beta_1\log\widehat{Z}_{\mathrm{ref}}(x),

and β2=10−3\beta_2=10^{-3} for policy regression β2log⁡πθ(y∣x)πref(y∣x)\beta_2\log\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}, while CBPO uses only one single β\beta for both value estimation and policy regression. Therefore, the actual A*-PO and CBPO losses are not related by a constant rescaling.

With different coefficients in the two stages, A*-PO estimates Vβ1⋆(x)V^\star_{\beta_1}(x) using β1\beta_1, but uses β2\beta_2 in policy regression, which causes A*-PO generally retains a residual of Vβ1⋆(x)−Vβ2⋆(x)V^\star_{\beta_1}(x)-V^\star_{\beta_2}(x) at the optimal policy:

βlog⁡πβ⋆(y∣x)πref(y∣x)=r(x,y)−Vβ⋆(x),Vβ⋆(x)=βlog⁡Zβ(x).\beta\log\frac{\pi^\star_\beta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)} =r(x,y)-V^\star_\beta(x), \qquad V^\star_\beta(x)=\beta\log Z_\beta(x).

CBPO uses a single β\beta to estimate Zβ(x)Z_\beta(x) and form the regression target, so its target remains aligned with πβ⋆\pi^\star_\beta, without this coefficient-mismatch residual. Consistent with this, CBPO theoretically converges faster and achieves better performance. We will incorporate the comparison with A*-PO into the Section 2.

We also train A*-PO from base policy Qwen3-1.7B under matched settings. Table below reports the results across the same evaluation protocol.

MethodsAIME25 (avg@32 / pass@32)MinervaMath (avg@32 / pass@32)AMC23 (avg@32 / pass@32)MATH500 (avg@32 / pass@32)Avg. (avg@32 / pass@32)
A*-PO17.08 / 33.3321.54 / 39.3456.33 / 85.0070.41 / 84.6041.34 / 60.57
CBPO-OnPolicy17.40 / 36.6722.00 / 39.7157.19 / 87.5070.71 / 85.0041.82 / 62.22
CBPO-Unified20.63 / 46.6724.48 / 42.6560.16 / 92.5073.01 / 87.8044.57 / 67.40

Across the four benchmarks, A*-PO achieves 41.34 avg@32 and 60.57 pass@32, compared with 41.82 avg@32 and 62.22 pass@32 for CBPO-OnPolicy. CBPO-Unified achieves 44.57 avg@32 and 67.40 pass@32. We will add the matched comparison and experimental details to the revised manuscript.


W2/Q1

On the max generation length and the evaluations of base policy models

We thank the reviewer for raising these concerns about the generation budget and baseline comparison. For teacher-generated offline data, DeepSeek-R1-Distill-Qwen-32B generated 32 candidates per prompt under the 2,048-token response limit. We retained prompts with at least three reward-1 candidates, and the candidate bank for every retained prompt contained sufficient negative candidates with rewards 0 and 0.1. This provided the positive and negative offline trajectories required by CBPO-Unified for every retained prompt.

Table D.2 reports the completion ratio for each prompt, which provides an indication of cap-hit or unfinished-response rates under the 2,048-token generation limit. We set the maximum generation length to 2,048 tokens with reference to prior work. For example, A$^{\star}$-PO uses a 1,024-token limit. This provides context for the generation budget used in our experiments.

We additionally evaluated the base-policy Qwen3-1.7B and Qwen3-8B under the same evaluation protocol. Their results are shown below. These comparisons provide direct references for assessing the gains over the initial policies.

ModelAIME25 (avg@32 / pass@32)MinervaMath (avg@32 / pass@32)AMC23 (avg@32 / pass@32)MATH500 (avg@32 / pass@32)Avg. (avg@32 / pass@32)
Qwen3-1.7B1.88 / 10.0018.53 / 31.6227.66 / 37.5053.09 / 70.8025.29 / 37.48
Qwen3-8B10.21 / 13.3329.15 / 42.2863.91 / 67.5068.68 / 78.2042.99 / 50.33
AuthorsOpenReview ↗

Dear Reviewer P7QD(2)

W3

On the missing related works

We thank the reviewer for pointing out that the relationship between CBPO, OAPL, and Prefix-RFT was not sufficiently discussed.

OAPL addresses policy lag between the trainer and inference engine. It uses the lagged inference policy to generate rollouts and as the KL reference. At each synchronization, it clears the rollout buffer so that each synchronization interval uses samples from a single inference-policy version. CBPO estimates a prompt-level normalizer from a fixed reference policy before optimization and freezes it. Its replay buffer retains complete responses with their verifier rewards and reference-policy log probabilities, including offline demonstrations, reference responses, current-policy rollouts, and historical rollouts. OAPL organizes data use around inference-policy synchronization intervals. CBPO organizes it around persistent replay of complete records from multiple sources and policy ages.

Prefix-RFT replaces one standard on-policy rollout with a hybrid trajectory formed by a sampled demonstration prefix and a continuation generated by the current policy. It estimates advantages over the resulting group of trajectories and uses PPO-style clipped updates, with entropy-based clipping to select demonstration-prefix tokens for updating. CBPO stores demonstrations and rollouts as complete response records. It does not splice a demonstration prefix into a current-policy trajectory. We will add this comparison to the revised main text.


Q2

On positive-demonstration-only training

We thank the reviewer for requesting this ablation. We trained CBPO-Offline (PosCoT) using only the positive demonstrations used by SFT, with the same Qwen3-1.7B base policy and data source pool as SFT.

MethodAIME25 (avg@32 / pass@32)MinervaMath (avg@32 / pass@32)AMC23 (avg@32 / pass@32)MATH500 (avg@32 / pass@32)Avg. (avg@32 / pass@32)
SFT14.17 / 26.6719.27 / 33.4648.83 / 77.5064.99 / 78.0036.81 / 53.91
CBPO-Offline (PosCoT)14.38 / 26.6719.30 / 33.4648.91 / 75.0065.22 / 78.0036.95 / 53.28

With the positive demonstrations, base policy, and data source pool matched, SFT obtains aggregate avg@32/pass@32 scores of 36.81/53.91, whereas CBPO-Offline (PosCoT) obtains 36.95/53.28. CBPO-Offline (PosCoT) therefore improves aggregate avg@32 by 0.14 points but lowers aggregate pass@32 by 0.63 points relative to SFT. We will extend to 8B and add results in the revision.


Q3

On performance gains from historical replay

We thank the reviewer for pointing out that the Full replay–FIFO comparison may conflate rollout age with the proportion of rollouts in training batches. Table D.8 reports age-balanced experiments that keep the proportion of rollout data fixed across age conditions. The results show that older rollouts are not uniformly beneficial and that performance does not increase monotonically with rollout age. Instead, gains arise when training reintroduces forgotten hard cases, suggesting that the benefit comes from recovering these cases rather than simply increasing the rollout share or using older rollouts.


Q4

On SFT learning-rate

We thank the reviewer for asking whether the SFT baseline in Table 3 was independently tuned or simply assigned the RL learning rate. We independently explored the SFT learning rate on SFT baseline and found the student-model SFT learning rate in BRIDGE was already a good choice. As a result, the SFT baseline in Table 3 uses the same learning rate in BRIDGE as a reference. In the SFT→DAPO experiment, we also use the same learning rate for the SFT stage to align with BRIDGE’s mixed SFT/RL objective.

Reviewer_P7QDOpenReview ↗

Response to Author's rebuttal message

I thank the authors for their reponse including additional comparisons and calrification. My problems are partially resolved, while two major concerns remain:

Authors highlight the A*-PO’s two-temperature implementation, while CBPO only use one. But CBPO’s core loss and training loop still remain close to A*-PO’s formulation in Algo. 1. The main extension appears to be multi-source replay, which I view as a limited algorithmic contribution.

Novelty is subjective, so I may change my score if the my concern about the evaluation setup is addressed. A 2k-token cap in thinking mode may substantially affect baseline accuracy and the reliability of the results. Table D.2 mentioned in reponse reports reward outcomes but does not show how often responses hit the cap. Even if 2K is a practical budget for training rollouts, please evaluate the original Qwen3-1.7B and Qwen3-8B checkpoints with an 8K or 16K output limit and report the fraction of responses that hit the limit. Comparision in a more reasonable budget could significantly strengthen the paper.

Reviewer_f2CdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CBPO trains a reasoning model with a single loss applied to several data sources at once. The sources are:

  • positive and negative offline demonstrations;
  • samples from the reference policy;
  • current and historical rollouts. The loss is a squared regression of the sequence log-ratio log π_θ/π_ref onto r/β − log Ẑ_ref(x). Here Ẑ_ref is a Monte Carlo estimate computed from K_ref = 24 reference samples per prompt and frozen before training. All records can share one replay buffer, with no importance weights or critic. Restricting which sources are used gives Offline, OnPolicy, OffPolicy and Unified variants.
Critical strengths
  • The implementation is easy to implement: loss and one record format cover offline, on-policy and off-policy data, with no critic, no stored behavior probabilities
  • The paper has broad baseline coverage for each dataset.
  • The paper does extensive ablations, which cover K_ref, rollout count, budget allocation, number and type of negatives, replay horizon and sampling, β, and teacher source.
Critical weaknesses
  • Lack of novelty compared to existing work: CBPO-OnPolicy appears to be A*-PO itself; even K_ref follows A*-PO’s choice. Behavior-independence is also not new. The paper builds on the fact that a KL-regularized consistency target does not depend on the behavior policy, so off-policy and offline data need no importance weights. The paper cites but does not compare against any of the single-algorithm on/off-policy methods (AGRO, SPO)
  • Based on Table 3, most of the gain comes from extra data instead not from the shared objective.
Questions for authors
  • Can you provide per-seed results for Tables 1–3?
  • Are these the post-trained Qwen3 checkpoints or the Base models? What do the pre-trained models score under your protocol?
  • How does CBPO-OnPolicy differ from A*-PO? How does CBPO relate to DRO and PCL?
8 comments
AuthorsOpenReview ↗

Dear Reviewer f2Cd(1)

Response to Reviewer f2Cd

We thank the reviewer for the questions about novelty, the role of additional data, checkpoint initialization, and per-seed results.

Scores in the tables below are avg@32 / pass@32 (%); averages weight the four benchmarks equally.


W1-1/Q3-1

W1 (excerpt): CBPO-OnPolicy appears to be A*-PO itself; even K_ref follows A*-PO’s choice. Behavior-independence is also not new.

Q3 (excerpt): How does CBPO-OnPolicy differ from A*-PO?

Response: Our core algorithm is CBPO-Unified. CBPO-OnPolicy is only an ablation variant that restricts policy updates to current-policy rollouts; it is not the main method embodying our contribution. We therefore distinguish the implementation difference of this ablation from the substantive advances of the full method.

When comparing CBPO-OnPolicy with A*-PO alone, the implementation difference is the use of two coefficients versus one: A*-PO's reported experiments use β1=0.5\beta_1=0.5 for value estimation and β2=10−3\beta_2=10^{-3} for policy regression, whereas CBPO-OnPolicy uses a single β\beta for both, keeping the two stages aligned with the same KL-regularized objective. Regarding the reference-sampling budget, A*-PO samples N=8N=8 reference responses per prompt, whereas we use Kref=24K_{\mathrm{ref}}=24. These budgets are different, and we did not adopt A*-PO's choice.

To assess the substantive advances of our work over A*-PO, however, the relevant comparison is with our core method, CBPO-Unified. CBPO-Unified enables joint offline, on-policy, and off-policy training with persistent data reuse, with advances in the following three aspects.

Training paradigm: from on-policy updates to unified post-training. A*-PO's Algorithm 1 uses offline reference samples to estimate values, then fits the policy only on newly collected current-policy responses. Although Section 3 notes that, with an exact normalizer, the optimal policy attains zero regression loss under any sampling distribution, the algorithm does not use this property for multi-source joint training and persistent replay of historical data. CBPO-Unified makes more direct use of this property by incorporating offline positive and negative demonstrations, reference responses, and historical rollouts into the same objective. This enables joint optimization of offline supervision and RL experience, replacing the objective switch in a separate SFT→RL pipeline with one continuing training process.

Theoretical analysis: from on-policy guarantees to mixed-source analysis. A*-PO's Section 5 establishes guarantees for on-policy collection through no-regret online learning. Our analysis studies finite-data fitting under different source distributions: Appendix A.4, Theorem 1, relates the KL-regularized objective gap to function approximation, normalizer-estimation, sampling, and optimization errors, and explicitly includes distribution-shift and coverage terms; Appendix A.5, Proposition 3, treats adaptively collected records. This characterizes the effects of changing the data-use regime even when the per-record target remains fixed.

Empirical benefits: CBPO-Unified achieves clearly higher scores. CBPO-Unified achieves aggregate avg@32/pass@32 of 44.57/67.40, clearly higher than A*-PO (41.34/60.57) and CBPO-OnPolicy (41.82/62.22). The same unified configuration also outperforms sequential SFT→DAPO (42.22/63.57; Tables B1 and B3). The CBPO entries report Seed 1.

AuthorsOpenReview ↗

Dear Reviewer f2Cd(2)

W1-2

W1 (excerpt): The paper cites but does not compare against any of the single-algorithm on/off-policy methods (AGRO, SPO).

Response: We added comparisons with AGRO and SPO. CBPO-Unified obtains higher aggregate avg@32/pass@32 in the reported runs: gains of 1.36/2.18 points over AGRO and 1.12/1.41 points over SPO (Table A).

Centering and fixed targets. AGRO centers the current policy's regularized reward under the behavior distribution μ\mu (Tang et al., 2025, Section 3.1). CBPO uses a frozen reference-policy normalizer. Changing the source distribution changes AGRO's centering and CBPO's aggregate sample weights, while CBPO's target for each record stays fixed.

Shared regression and the replay procedure. SPO also estimates fixed prompt values, includes squared-error and cross-entropy variants, and supports mixed online/offline training (Cohen et al., 2025, Sections 2.4 and 3.1). With matched rewards, reference policy, temperature, normalizer estimate, and complete-response log probabilities, its unwarped terminal-Q squared loss satisfies

(Q_T^\theta-r(x,y))^2=\beta^2\ell_{\mathrm{CBPO}}(x,y;\theta).$$ CBPO specifies how data enter and remain in this regression process: positive and negative demonstrations and reference responses initialize the buffer, new rollouts are appended, and earlier records remain available. Reference responses serve both to estimate the frozen normalizer and to train the policy. The finite-data analysis above addresses the resulting evolving sampling distribution. Table A compares this complete training procedure with the evaluated AGRO and SPO configurations. All three methods start from Qwen/Qwen3-1.7B and use the same mathematical evaluation protocol. The **CBPO-Unified** entry is Seed 1 from Q1. **Table A.** Comparison with AGRO and SPO. The **CBPO-Unified** row reports Seed 1. | Methods | MATH500 (avg@32 / pass@32) | MinervaMath (avg@32 / pass@32) | AMC23 (avg@32 / pass@32) | AIME25 (avg@32 / pass@32) | Avg. (avg@32 / pass@32) | | --- | --- | --- | --- | --- | --- | | AGRO | 71.89 / 86.00 | 23.56 / 41.54 | 58.20 / 90.00 | 19.17 / 43.33 | 43.21 / 65.22 | | SPO | 72.32 / 86.20 | 23.76 / 41.91 | 58.44 / 90.00 | 19.27 / 43.33 | 43.45 / 65.36 | | **CBPO-Unified** | **73.01** / **87.80** | **24.48** / **42.65** | **60.16** / **92.50** | **20.63** / **46.67** | **44.57** / **67.40** | --- ## W2 > **W2:** Based on Table 3, most of the gain comes from extra data instead not from the shared objective. **Response:** **Effectively using additional data is precisely a benefit enabled by the shared optimization objective.** Even if standard on-policy RL methods seek to incorporate the same offline demonstrations and historical rollouts, they cannot readily do so without changing their original update procedure. The shared objective keeps each record's target fixed as new data are added, allowing heterogeneous supervision and policy experience to contribute without switching training stages. Paper Table 3 varies data-source inclusion and history retention, evaluating the gains from these data-use choices within the unified procedure.
AuthorsOpenReview ↗

Dear Reviewer f2Cd(3)

Q1

Q1: Can you provide per-seed results for Tables 1–3?

Response: The complete results for seeds 1–5 are provided in Tables B1–B15 below. Each seed includes paper Table 1 (Qwen3-1.7B), Table 2 (Qwen3-8B), and Table 3 (source and replay ablations). Each Avg. entry averages the four benchmarks for that seed.

Table B1. Seed 1, paper Table 1: Qwen3-1.7B.

RegimeMethodMATH500 (avg@32 / pass@32)MinervaMath (avg@32 / pass@32)AMC23 (avg@32 / pass@32)AIME25 (avg@32 / pass@32)Avg. (avg@32 / pass@32)
OfflineSFT64.99 / 78.0019.27 / 33.4648.83 / 77.5014.17 / 26.6736.81 / 53.91
OfflineDPO65.34 / 78.2019.29 / 33.4648.98 / 75.0014.48 / 26.6737.02 / 53.33
OfflineCBPO-Offline66.98 / 80.4020.29 / 35.2950.08 / 77.5015.73 / 26.6738.27 / 54.97
On-policyPPO69.71 / 83.6021.34 / 38.6056.48 / 85.0017.40 / 30.0041.23 / 59.30
On-policyDAPO70.38 / 84.6021.58 / 39.7156.25 / 85.0017.19 / 36.6741.35 / 61.49
On-policyGSPO70.53 / 84.8021.85 / 39.7155.63 / 82.5017.19 / 36.6741.30 / 60.92
On-policyREINFORCE++68.58 / 81.8021.19 / 38.2452.58 / 80.0016.88 / 30.0039.81 / 57.51
On-policyCBPO-OnPolicy70.71 / 85.0022.00 / 39.7157.19 / 87.5017.40 / 36.6741.82 / 62.22
Off-policyTBRM71.46 / 83.6022.08 / 40.0757.50 / 90.0018.13 / 40.0042.29 / 63.42
Off-policyCBPO-OffPolicy71.46 / 85.8022.63 / 40.0757.97 / 92.5018.65 / 40.0042.68 / 64.59
Joint SFT/RLBRIDGE72.02 / 85.8023.44 / 41.5457.97 / 87.5018.96 / 43.3343.10 / 64.54
UnifiedCBPO-Unified73.01 / 87.8024.48 / 42.6560.16 / 92.5020.63 / 46.6744.57 / 67.40

Table B2. Seed 1, paper Table 2: Qwen3-8B.

RegimeMethodMATH500 (avg@32 / pass@32)MinervaMath (avg@32 / pass@32)AMC23 (avg@32 / pass@32)AIME25 (avg@32 / pass@32)Avg. (avg@32 / pass@32)
OfflineSFT84.64 / 88.6045.20 / 52.5783.20 / 87.5033.13 / 36.6761.54 / 66.34
OfflineDPO85.11 / 89.8045.58 / 53.3183.28 / 90.0033.44 / 36.6761.85 / 67.44
OfflineCBPO-Offline85.62 / 90.6047.20 / 55.1584.14 / 90.0033.96 / 36.6762.73 / 68.10
On-policyPPO89.42 / 96.6050.31 / 58.4689.77 / 95.0041.25 / 46.6767.69 / 74.18
On-policyDAPO89.59 / 96.8049.08 / 58.0988.52 / 95.0039.38 / 40.0066.64 / 72.47
On-policyGSPO89.16 / 96.6049.13 / 58.4687.11 / 97.5038.96 / 40.0066.09 / 73.14
On-policyREINFORCE++88.95 / 95.2048.77 / 57.3583.52 / 95.0035.94 / 40.0064.29 / 71.89
On-policyCBPO-OnPolicy89.90 / 97.0050.94 / 59.1990.70 / 97.5042.19 / 46.6768.43 / 75.09
Off-policyTBRM90.18 / 97.2052.15 / 60.2991.33 / 97.5043.23 / 46.6769.22 / 75.42
Off-policyCBPO-OffPolicy90.17 / 97.2051.64 / 59.9391.72 / 97.5043.44 / 46.6769.24 / 75.32
Joint SFT/RLBRIDGE91.63 / 97.2053.77 / 63.2492.19 / 97.5044.17 / 46.6770.44 / 76.15
UnifiedCBPO-Unified94.90 / 97.8056.65 / 66.9193.20 / 100.0046.46 / 50.0072.80 / 78.68

Table B3. Seed 1, paper Table 3: source and replay ablations.

MethodMATH500 (avg@32 / pass@32)MinervaMath (avg@32 / pass@32)AMC23 (avg@32 / pass@32)AIME25 (avg@32 / pass@32)Avg. (avg@32 / pass@32)
SFT → DAPO71.40 / 85.6021.99 / 41.1857.27 / 87.5018.23 / 40.0042.22 / 63.57
CBPO w/o History (PosCoT)71.67 / 85.8022.32 / 40.4457.34 / 85.0018.13 / 40.0042.37 / 62.81
CBPO w/o History71.54 / 86.2022.52 / 41.1857.97 / 87.5018.75 / 40.0042.70 / 63.72
CBPO-Unified FIFO-171.94 / 86.0022.86 / 41.1858.52 / 92.5018.96 / 43.3343.07 / 65.75
CBPO-Unified73.01 / 87.8024.48 / 42.6560.16 / 92.5020.63 / 46.6744.57 / 67.40
Reviewer_jwbNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Post-training of LLMs can draw from multiple data regimes: (1) offline, independently collected demonstrations, (2) on-policy and (3) off-policy demonstrations from the current policy and the older versions of it, respectively. SOTA fine-tuning algorithms either discard parts of these data regimes or use them in isolation in a sequence of potentially conflicting training phases.

This paper introduces Cross-Behavior Policy Optimization (CBPO), a unified framework that can aggregate data from all three regimes and fine-tune the LLM accordingly under a common objective. Compared to the baselines with multiple training phases, models trained with CBPO show improved performance under matched computational budget.

The authors also evaluate the proposed algorithm in three isolated data regimes (CBPO-Offline, CBPO-OnPolicy, CBPO-OffPolicy) and compare them with corresponding baselines under matched settings. All three variants match or outperform the SOTA baselines in their respective settings. The unified CBPO (CBPO-Unified) learns from all data regimes and outperforms each model post-trained in one of the individual data regimes.

Critical strengths

The paper is well-written with a strong motivation and clear arguments. The experiments are extensive, the results are discussed in detail, and the ablations ask the right questions to isolate the benefits of CBPO from any specific hyperparameter choice.

The main strength of the method is that it allows post-training of LLMs from multiple data regimes under a unified training objective, and hence achieves better final performance than any other compared baseline that trains in only a limited data regime. This enables efficient reuse of previously collected question-answer datasets to speed up and improve post-training.

The secondary strength of the method is that it matches or outperforms corresponding baselines even when only trained under isolated data regimes. This establishes CBPO as a strong general-use post-training algorithm, regardless of which of the three domains available data comes from.

Critical weaknesses

Authors acknowledge similarity to A*-PO, however, the exact methodological differences, as well as direct performance comparison with CBPO is missing. The training objective of CBPO is equivalent to that of A*-PO up to a constant, which further necessitates an explicit comparison to verify any performance differences.

Secondly, as acknowledged by the authors, the experiments evaluate and compare CBPO only for mathematical reasoning tasks and leave other problem domains unexplored. The method also assumes access to a verifier to evaluate and assign reward values to model outputs. Although this is a natural assumption for mathematical question and answers, the extension of the algorithm to more general problem domains is not established.

Questions for authors

The following concerns about the current presentation limit my final assessment score.

  1. Can the authors explicitly clarify the novelty of their method in the manuscript, especially in comparison to the related work A*-PO?
  2. Can the authors run baseline experiments with A*-PO under matched settings and report the results?
  3. Do authors have any intuition on how CBPO affects the performance of the LLMs in answering questions from domains that post-training doesn't focus on? Perhaps a direct comparison can be made between the performances of the base and post-trained LLMs in answering a held-out evaluation dataset from a different problem domain, e.g. the original benchmark datasets the base LLMs were evaluated on.
2 comments
AuthorsOpenReview ↗

Dear Reviewer jwbN(1)

Response to Reviewer jwbN

We thank the reviewer for the questions about CBPO's contribution, applicability beyond mathematics, and effects on other capabilities.


W1/Q1

W1 (excerpt): The training objective of CBPO is equivalent to that of A*-PO up to a constant, which further necessitates an explicit comparison to verify any performance differences.

Q1: Can the authors explicitly clarify the novelty of their method in the manuscript, especially in comparison to the related work A*-PO?

Response: Compared with A*-PO, CBPO-Unified enables joint offline, on-policy, and off-policy training with persistent data reuse with single-β\beta loss. The advances concern the training paradigm, its theoretical analysis, and the empirical benefits of using data beyond current-policy rollouts.

Training paradigm: from on-policy updates to unified post-training. A*-PO's Algorithm 1 uses offline reference samples to estimate values, then fits the policy only on newly collected current-policy responses. Although Section 3 notes that, with an exact normalizer, the optimal policy attains zero regression loss under any sampling distribution, the algorithm does not use this property for multi-source joint training and persistent replay of historical data. CBPO-Unified makes more direct use of this property by incorporating offline positive and negative demonstrations, reference responses, and historical rollouts into the same objective. This enables joint optimization of offline supervision and RL experience, replacing the objective switch in a separate SFT→RL pipeline with one continuing training process.

Theoretical analysis: from on-policy guarantees to mixed-source analysis. A*-PO's Section 5 establishes guarantees for on-policy collection through no-regret online learning. Our analysis studies finite-data fitting under different source distributions: Appendix A.4, Theorem 1, relates the KL-regularized objective gap to function approximation, normalizer-estimation, sampling, and optimization errors, and explicitly includes distribution-shift and coverage terms; Appendix A.5, Proposition 3, treats adaptively collected records. This characterizes the effects of changing the data-use regime even when the per-record target remains fixed.

Empirical benefits: gains from a broader range of training data. The direct comparison in Q2 (Table B) shows close aggregate scores for A*-PO and CBPO-OnPolicy, while CBPO-Unified achieves higher avg@32 and pass@32. This comparison evaluates the practical benefit of the complete multi-source configuration.

Implementation difference. The A*-PO's practical loss has two β\beta, where β1=0.5\beta_1=0.5 for value estimation and β2=10−3\beta_2=10^{-3} for policy regression, while CBPO uses single β\beta for both of them. Therefore, the actual A*-PO and CBPO losses are not related by a constant rescaling.

With different coefficients in the two stages, A*-PO estimates Vβ1⋆(x)V^\star_{\beta_1}(x) using β1\beta_1, but uses β2\beta_2 in policy regression during training process, which causes A*-PO generally retains a residual of Vβ1⋆(x)−Vβ2⋆(x)V^\star_{\beta_1}(x)-V^\star_{\beta_2}(x) at the optimal policy:

βlog⁡πβ⋆(y∣x)πref(y∣x)=r(x,y)−Vβ⋆(x),Vβ⋆(x)=βlog⁡Zβ(x).\beta\log\frac{\pi^\star_\beta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)} =r(x,y)-V^\star_\beta(x), \qquad V^\star_\beta(x)=\beta\log Z_\beta(x).

CBPO uses a single β\beta to estimate Zβ(x)Z_\beta(x) and form the regression target, so its target remains aligned with πβ⋆\pi^\star_\beta, without this coefficient-mismatch residual. Consistent with this, CBPO theoretically converges faster and achieves better performance.


W1/Q2

Q2: Can the authors run baseline experiments with A*-PO under matched settings and report the results?

Response: CBPO-Unified reaches 44.57/67.40 aggregate avg@32/pass@32, compared with 41.34/60.57 for A*-PO and 41.82/62.22 for CBPO-OnPolicy (Table B).

All three use Qwen/Qwen3-1.7B initialization and the same evaluation protocol. The added A*-PO run uses its reported double-β\beta configuration (β1=0.5\beta_1=0.5 for value estimation and β2=10−3\beta_2=10^{-3} for policy regression). CBPO uses one consistent β\beta. The CBPO entries are Seed 1 results and averages are over the four benchmarks.

Table B. Comparison with A*-PO. Scores are avg@32 / pass@32 (%); Average is the unweighted mean across four benchmarks. CBPO entries report Seed 1.

MethodMATH500 (avg@32 / pass@32)MinervaMath (avg@32 / pass@32)AMC23 (avg@32 / pass@32)AIME25 (avg@32 / pass@32)Average (avg@32 / pass@32)
A*-PO70.41 / 84.6021.54 / 39.3456.33 / 85.0017.08 / 33.3341.34 / 60.57
CBPO-OnPolicy70.71 / 85.0022.00 / 39.7157.19 / 87.5017.40 / 36.6741.82 / 62.22
CBPO-Unified73.01 / 87.8024.48 / 42.6560.16 / 92.5020.63 / 46.6744.57 / 67.40
AuthorsOpenReview ↗

Dear Reviewer jwbN(2)

W2

W2 (excerpt): The method also assumes access to a verifier to evaluate and assign reward values to model outputs. [...] The extension of the algorithm to more general problem domains is not established.

Response: We added general-domain post-training with a learned reward model. In this experiment, CBPO-Unified improves both Arena-Hard v2 and MT-Bench scores over the initial policy and PPO (Table A).

We train on the General subset of HelpSteer3-Preference, starting both methods from Qwen3-1.7B. Both use Qwen3-Nemotron-32B-GenRM-Principle under the same helpfulness principle, with its Yes/No score difference mapped to a reward in [0,1][0,1]. We evaluate the initial policy and both trained policies using GPT-6-Luna-Max as the judge. Relative to PPO, CBPO-Unified gains 0.6 percentage points in Arena-Hard win rate and 0.56 points in MT-Bench mean score.

Table A. General-domain post-training with a learned reward model.

ModelArena-Hard v2 win rate (%)MT-Bench mean judge score (1–10)
Qwen3-1.7B2.84.66
PPO7.56.35
CBPO-Unified8.16.91

Q3

Q3 (excerpt): Do authors have any intuition on how CBPO affects the performance of the LLMs in answering questions from domains that post-training doesn't focus on?

Response: The observed cross-domain score changes range from −0.51 to 0.00 percentage points after math-focused post-training (Table C).

We compare each initial checkpoint with its corresponding CBPO-Unified checkpoint on held-out coding, instruction-following, and scientific question-answering evaluations. We report LiveCodeBench v5 pass@1, IFEval prompt-level strict accuracy, and GPQA-Diamond accuracy. The small changes describe these benchmarks under the evaluated generation settings.

Table C. Cross-domain evaluation before and after math-focused post-training. Scores are percentages.

Model sizeCheckpointLiveCodeBench v5 (pass@1)IFEval (prompt-level strict accuracy)GPQA-Diamond (accuracy)
1.7BBase Policy15.0234.755.56
1.7BCBPO-Unified14.8534.385.05
8BBase Policy33.2045.1012.12
8BCBPO-Unified33.1644.9212.12

Manuscript revisions. We clarified in Section 2 how CBPO-Unified differs from A*-PO, including its single-β\beta implementation and persistent joint use of offline, on-policy, and off-policy records. We added the matched-setting A*-PO results to Appendix C.3. To address generalization beyond mathematical reasoning, we added the HelpSteer3-Preference experiment and cross-domain evaluations to Section 5, with configurations and results in Appendices C.1 and C.2.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We thank the reviewers for their careful evaluations and constructive feedback. We summarize the contribution of CBPO and the additional analyses and manuscript revisions made in response.

1. Our contributions and their research significance

The target for an individual response in fixed-reference regression is independent of the policy that generated it. Prior work has noted this property, but existing post-training procedures have not fully used it to combine offline demonstrations, reference responses, and current and historical rollouts throughout training. We build on this property in Cross-Behavior Policy Optimization (CBPO), with three contributions:

  • A unified learning paradigm. We apply one response-level regression objective to positive and negative offline demonstrations, reference responses, and current and historical rollouts in a persistent buffer.
  • A theoretical analysis. We characterize the finite-sample effect of estimating the frozen prompt normalizer, analyze its induced gradient bias, and relate regression error to the KL-regularized objective gap.
  • Mechanism experiments and learning dynamics. Controlled ablations isolate the contributions of negative demonstrations, reference responses, and historical replay. The learning curves show how performance develops during training, while evaluations on Qwen3-1.7B and Qwen3-8B show that CBPO-Unified achieves the best avg@32 and pass@32 on all four mathematical reasoning benchmarks.

2. Reviewer concerns and our responses

C1. Method novelty and comparisons with prior work (jwbN W1/Q1–Q2; f2Cd W1-1/Q3-1, W1-2, and Q3-2; P7QD W1/Q1 and W3; Xw9p W2/Q1). Reviewers asked what distinguishes CBPO from A*-PO and related methods. A*-PO estimates values from reference-policy samples and updates its policy using newly collected on-policy responses. CBPO-OnPolicy is our current-rollout-only variant, whereas CBPO-Unified applies one response-level objective to offline demonstrations, reference responses, and current and historical rollouts in a persistent buffer. We clarified this distinction and expanded Section 2 to position CBPO relative to A*-PO, DRO, PCL, Prefix-RFT, OAPL, AGRO, and SPO. In the single-run comparison, CBPO-Unified achieves aggregate avg@32/pass@32 of 44.57/67.40, compared with 41.34/60.57 for A*-PO and 41.82/62.22 for CBPO-OnPolicy.

C2. Data-source contributions and behavior-policy independence (f2Cd W1-1 and W2; Xw9p W1 and W2/Q1). Reviewers asked whether the gains came from adding data rather than from the shared objective, and whether behavior-policy independence applies to the overall training objective. We clarified in Section 4 that independence applies to each record’s target and loss, not to the expected objective. Appendix A analyzes how the buffer-sampling distribution weights records in that objective. Section 5 reports the data-source ablations: adding negative demonstrations, reference responses, and historical rollouts improves performance within the shared training procedure. We also added the matched-source and matched-budget AGRO/SPO comparisons to Appendix C.3.

C3. Generalization and evaluation length (jwbN W2 and Q3; P7QD W2/Q1 and follow-ups U65w73tBIX and ndD0q0rNcB). Reviewers asked about general-domain performance, cross-domain effects, and the generation limit. Section 5 now reports HelpSteer3-Preference post-training results: CBPO-Unified scores 8.1% on Arena-Hard v2 and 6.91 on MT-Bench, compared with PPO’s 7.5% and 6.35. After math-focused training, scores on LiveCodeBench v5, IFEval, and GPQA-Diamond change by −0.51 to 0.00 percentage points across the two model sizes. We also evaluated the original and CBPO-Unified checkpoints at 8,192- and 16,384-token output limits. At 16,384 tokens, CBPO-Unified’s average avg@32 exceeds the original checkpoint by 17.30 points for 1.7B and 22.96 points for 8B. Appendices C.1–C.8 provide the experiment details, filtered metrics, and fully truncated-problem counts.

C4. Additional controls and reproducibility (f2Cd Q1–Q2; P7QD Q2–Q4). We provided per-seed results for Tables 1–3 and clarified the checkpoint initialization and evaluation protocol in our responses. Appendix C.3 includes the positive-demonstration-only SFT and CBPO-Offline comparison. Appendix B documents baseline configurations, and Appendix E reports age-balanced replay and age- and residual-stratified results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.