acceptodds
Under review as a conference paper at ICLR 2027

Bellman Policy Optimization

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a central approach to improving the reasoning capabilities of large language models (LLMs). We introduce Bellman Policy Optimization (BPO), a critic-free policy optimization method derived from Policy Mirror Descent (PMD). For autoregressive generation with terminal rewards, we use the Bellman equations to reformulate PMD into a trajectory-level objective that avoids estimating intermediate state values. We prove that the reformulated objective has the same unique optimal solution as the original PMD objective. By approximating the trajectory-level objective, we derive the BPO loss, which replaces the importance-sampling ratio in Group-Relative Policy Optimization (GRPO) with a smoothed ratio of complementary token probabilities. In long chain-of-thought and multi-turn tool-integrated mathematical reasoning tasks, BPO achieves the highest average accuracy over the evaluated benchmarks among all compared methods.

Primary area:
optimization
Keywords:
Policy Mirror Descent, Policy Optimization, Large Language Model, RLVR

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FBwzOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Bellman Policy Optimization (BPO), a critic-free policy optimization method for RLVR. The starting point is Policy Mirror Descent (PMD), whose update requires advantage estimates at every intermediate token. Using the Bellman equation and the terminal-reward structure of autoregressive generation, the authors rewrite PMD as a trajectory-level least-squares objective that only involves the final reward and the prompt-level value, and prove that it has the same unique optimal solution as PMD. They then derive a practical loss through a sequence of approximations. Experiments demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The derivation builds on two fairly elementary observations: the log-partition function can be replaced by DKL(μ∣∣π+)D_{\mathrm{KL}}(\mu ||\pi^+), and the token-level advantages telescope to R(x,y)−Vμ(x)R(x,y) - V^\mu(x). Similar likelihood-ratio reparameterizations already appear in the direct-alignment literature (e.g., DPO and its token-level variants), so these steps are not new in themselves. The interesting part is the martingale argument in the necessity part of Theorem 1. Note that a zero trajectory-level residual does not by itself imply a zero residual at each token, since positive and negative terms may cancel along a response. This point is handled by observing that, under the rollout policy μ\mu, each token-level residual has zero conditional mean given the preceding tokens, so that the partial sums of the token-level residuals form a martingale. Because the response length is bounded by Tmax⁡T_{\max}, the martingale property implies that every partial sum equals the terminal value, which is zero, and hence every token-level residual vanishes. In this sense, the trajectory-level objective recovers the PMD update at each token, rather than only on average over a response.
  2. From an implementation perspective, the resulting method is simple. Compared with GRPO, the importance-sampling ratio π/μ\pi/\mu is replaced by the weight (1+ϵ−μ)/(1+ϵ−π)(1+\epsilon-\mu)/(1+\epsilon-\pi), truncated at a constant CC, while the group-normalized advantage and the clipping mask are kept unchanged. In practice this amounts to modifying only a few lines of code. Note that no value model is introduced and no additional forward pass is required, since the weight depends only on the probabilities of the sampled token, which are already available in GRPO. Therefore, the method can be readily incorporated into existing RLVR pipelines and compared against GRPO under identical settings.
Critical weaknesses
  1. Relation to Kimi k1.5 and VIMPO. Objective (18) is nearly identical in form to two existing methods, neither of which is compared against. Written side by side (with τ=1/η\tau = 1/\eta and Rˉ\bar{R} the group-mean reward), the residuals are:

    • Kimi k1.5 (arXiv:2501.12599): (R−Rˉ)−τ∑tlog⁡πμ(R - \bar{R}) - \tau \sum_t \log \frac{\pi}{\mu}
    • VIMPO (arXiv:2606.20008, cited as [11]): (R−Rˉ)−β∑t(log⁡ππref−sg[DKL(π∣∣πref)])(R - \bar{R}) - \beta \sum_t \big(\log \frac{\pi}{\pi_{\mathrm{ref}}} - \mathrm{sg}[D_{\mathrm{KL}}(\pi || \pi_{\mathrm{ref}})]\big)
    • BPO, Eq. (18): (R−Rˉ)−τ∑t(log⁡πμ+DKL(μ∣∣π))(R - \bar{R}) - \tau \sum_t \big(\log \frac{\pi}{\mu} + D_{\mathrm{KL}}(\mu || \pi)\big)

    All three minimize the squared residual over sampled responses. Thus BPO differs from Kimi k1.5 only by the per-token KL term, and from VIMPO mainly in the anchor policy (μ\mu versus πref\pi_{\mathrm{ref}}) and the direction of the KL term. The authors should explain what these differences bring, both in theory (e.g., whether they change the target policy, and whether Theorem 1 would still hold without them) and in practice (whether these differences survive the linearization in Section 3.2, since BPO ultimately optimizes a linearized surrogate, whereas Kimi k1.5 and VIMPO optimize the squared loss directly). Both methods should also be included as baselines under the same settings.

  2. Binary KL approximation. Most approximations in Section 3.2 are standard, but replacing the full KL with its binary version is, at least to me, less common, and this is the step that yields the weight (1−μ)/(1−π)(1-\mu)/(1-\pi). The only justification is that the binary KL lower-bounds the full KL, which says little about how close their gradients are. The authors should justify this step and compare it with a more faithful alternative, e.g., a top-kk approximation of the full KL.

Questions for authors

see weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer FBwz (Part 1)

Thank you for your comments. We have addressed all your concerns below.

W1

Relation to Kimi k1.5 and VIMPO. Objective (18) is nearly identical in form to two existing methods, neither of which is compared against. Written side by side (with τ=1/η\tau = 1/\eta and Rˉ\bar R the group-mean reward), the residuals are:

  • Kimi k1.5 (arXiv:2501.12599): (R−Rˉ)−τ∑tlog⁡πμ(R-\bar R) - \tau\sum_t \log\frac{\pi}{\mu}
  • VIMPO (arXiv:2606.20008, cited as [11]): (R−Rˉ)−β∑t(log⁡ππref−sg[DKL(π∥πref)])(R-\bar R) - \beta\sum_t\left(\log\frac{\pi}{\pi_{\mathrm{ref}}} - \mathrm{sg}[D_{\mathrm{KL}}(\pi\|\pi_{\mathrm{ref}})]\right)
  • BPO, Eq. (18): (R−Rˉ)−τ∑t(log⁡πμ+DKL(μ∥π))(R-\bar R) - \tau\sum_t\left(\log\frac{\pi}{\mu} + D_{\mathrm{KL}}(\mu\|\pi)\right)

All three minimize the squared residual over sampled responses. Thus BPO differs from Kimi k1.5 only by the per-token KL term, and from VIMPO mainly in the anchor policy (μ\mu versus πref\pi_{\mathrm{ref}}) and the direction of the KL term. The authors should explain what these differences bring, both in theory (e.g., whether they change the target policy, and whether Theorem 1 would still hold without them) and in practice (whether these differences survive the linearization in Section 3.2, since BPO ultimately optimizes a linearized surrogate, whereas Kimi k1.5 and VIMPO optimize the squared loss directly). Both methods should also be included as baselines under the same settings.

Thank you for this comment.

Experiments. Following the reviewer's suggestion, we compare BPO with Kimi k1.5 and VIMPO on Qwen3-4B-Base (batch size 128, group size 16, maximum response length 8k, i.e., the setting in Table 3). BPO (PMD equivalent) denotes the unlinearized critic-free objective, which has the same optimal solution as PMD (Theorem 1). For VIMPO, we compare with two variants. VIMPO denotes the main algorithm of VIMPO (Eq. (12) of VIMPO) with cA=5×10−3c_A=5\times10^{-3}, the setting used in VIMPO. VIMPO (cA=0c_A=0) denotes the operational form of the value-loss component of VIMPO (Eq. (8) of VIMPO), which is equivalent to the main algorithm of VIMPO with cA=0c_A=0. We report Avg@32 after 200 training steps

AIME24AIME25AIME26AIME24-26 Avg.
BPO25.825.3‾\underline{25.3}21.924.3
BPO (PMD equivalent)26.3‾\underline{26.3}25.619.0‾\underline{19.0}23.6‾\underline{23.6}
Kimi k1.527.122.317.622.3
VIMPO22.518.916.819.4
VIMPO (cA=0c_A=0)19.816.414.716.9
GRPO-ClipHigher19.018.413.617.0

BPO achieves the highest average accuracy, 2.0 percentage points higher than Kimi k1.5. Both BPO and BPO (PMD equivalent) outperform Kimi k1.5, VIMPO, and VIMPO (cA=0c_A=0) in average accuracy.

Theory: BPO, Kimi k1.5, and VIMPO have fundamentally different theoretical foundations. For notational simplicity, let η=1/τ\eta=1/\tau.

  • The Kimi k1.5 RL algorithm performs sequence-level PMD from the rollout policy μ\mu, with the KL penalty on complete responses.
  • BPO (PMD equivalent) performs token-level PMD from μ\mu with the advantage function AμA^\mu, i.e., one PMD step at every state.
  • VIMPO (Eq. (1) and (2) of VIMPO) solves a token-level KL-regularized problem with respect to a fixed reference policy πref\pi_{\mathrm{ref}} instead of μ\mu.

The key difference lies in the log-partition function. Kimi k1.5 has a single sequence-level term τlog⁡Z(x)\tau\log Z(x) and approximates it by the group-mean reward Rˉ\bar R, which is accurate only as τ→∞\tau\to\infty. BPO has a log-partition function log⁡Zμ(st)\log Z_\mu(s_t) at every state, and we show in our submission that log⁡Zμ(st)=KL(μ(⋅∣st)∥π+(⋅∣st)).\log Z_\mu(s_t)=\mathrm{KL}\big(\mu(\cdot|s_t)\Vert\pi^+(\cdot|s_t)\big). In VIMPO, the corresponding term −βsg[KL(π∥πref)]-\beta\mathrm{sg}[\mathrm{KL}(\pi\Vert\pi_{\mathrm{ref}})] is defined with respect to πref\pi_{\mathrm{ref}}, and has the opposite sign and the opposite KL direction to the BPO term +KL(μ∥π)+\mathrm{KL}(\mu\Vert\pi).

Therefore, BPO is fundamentally different from Kimi k1.5 and VIMPO in theory.

AuthorsOpenReview ↗

Response to Reviewer FBwz (Part 2)

W2

Binary KL approximation. Most approximations in Section 3.2 are standard, but replacing the full KL with its binary version is, at least to me, less common, and this is the step that yields the weight (1−μ)/(1−π)(1-\mu)/(1-\pi). The only justification is that the binary KL lower-bounds the full KL, which says little about how close their gradients are. The authors should justify this step and compare it with a more faithful alternative, e.g., a top-kk approximation of the full KL.

Thank you for this comment.

Efficiency. Our primary consideration for the binary KL approximation is efficiency. The binary KL approximation only requires π(yt∣st)\pi(y_t|s_t) and μ(yt∣st)\mu(y_t|s_t) of the sampled token yty_t. It requires neither storing the top-k tokens of the rollout engine nor computing the KL divergence over the full vocabulary.

Instability of both approximations. Both the top-k and the binary approximations involve ratios that can become very large. The top-k approximation keeps yty_t and the kk most probable tokens under μ(⋅∣st)\mu(\cdot|s_t), and aggregates all remaining tokens into a single category "other". The gradient of psμ(other)log⁡psμ(other)psπ(other)p^\mu_s(\mathrm{other})\log\frac{p^\mu_s(\mathrm{other})}{p^\pi_s(\mathrm{other})} involves the ratio psμ(other)psπ(other)\frac{p^\mu_s(\mathrm{other})}{p^\pi_s(\mathrm{other})}, which can become very large when psπ(other)→0p^\pi_s(\mathrm{other})\to 0 and introduce large noise into the update. Similarly, the weight 1−μ(yt∣st)1−π(yt∣st)\frac{1-\mu(y_t|s_t)}{1-\pi(y_t|s_t)} given by the binary approximation can become very large when 1−π(yt∣st)→01-\pi(y_t|s_t)\to 0, and we have observed this value above 10710^7 for some tokens.

Binary KL approximation fits the unified framework. An advantage of the binary approximation is that it gives a single ratio. We mitigate its instability by the ϵ\epsilon smoothing, and by substituting the smoothed weight for the importance-sampling ratio rtir_t^i in the unified RLVR framework ∇L(π)=−E[1G∑i=1G∑t=1∣yi∣Mtimin⁡rti,CA^i∇log⁡π(yti∣sti)]\nabla\mathcal{L}(\pi)=-\mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G}\sum_{t=1}^{|y^i|}M_t^i\min\\{r_t^i,C\\}\hat A^i\nabla\log\pi(y_t^i|s_t^i)\right], where MtiM_t^i is the clipping mask and CC caps the ratio. GRPO, CISPO, and DPPO are instances of the unified framework that differ only in MtiM_t^i and CC

In contrast, the gradient of the top-k KL divergence involves the ratio psμ(a)psπ(a)\frac{p^\mu_s(a)}{p^\pi_s(a)} for every token aa in the top-k set and for "other", together with the gradients of the log-probabilities of all tokens in the top-k set. It is therefore not of the form of the unified framework, where a single weight multiplies ∇log⁡π(yt∣st)\nabla\log\pi(y_t|s_t). How to apply smoothing and clipping to the top-k KL divergence is not straightforward, and we leave it for future study

Moreover, the top-k approximation requires storing the top-k tokens of the rollout engine, which adds computation and storage cost. On Qwen3-30B-A3B-Base with 32 GPUs, the wall-clock time to reach step 199 is 33.8 hours for BPO, compared with 56.8 hours for BPO (Top-16 KL Approx) and 152.6 hours for BPO (Top-128 KL Approx). Therefore, we choose the binary KL approximation

Ablation on ϵ\epsilon. We train Qwen3-4B-Base in the same setting as in W1 and report Avg@32 after 200 training steps. BPO (ϵ=0\epsilon=0) denotes the practical BPO loss with ϵ=0\epsilon=0

AIME24AIME25AIME26AIME24-26 Avg.
BPO25.8‾\underline{25.8}25.321.924.3
BPO (ϵ=0\epsilon=0)27.025.0‾\underline{25.0}20.6‾\underline{20.6}24.2‾\underline{24.2}
GRPO-ClipHigher19.018.413.617.0

Appendix H.1 and H.2 of our original submission further vary ϵ\epsilon over 0.05,0.1,0.2,0.3\\{0.05,0.1,0.2,0.3\\} and CC over 2.0,3.0,4.0\\{2.0,3.0,4.0\\}. These results show that BPO is insensitive to ϵ\epsilon and CC, and the ϵ\epsilon smoothing slightly improves the average accuracy. We therefore keep the ϵ\epsilon smoothing in the large-scale experiments on Qwen3-30B-A3B-Base

Comparison with the top-k approximation. Our original submission already compares BPO with the top-k approximation on Qwen3-30B-A3B-Base (Appendix H.5, Table 9). The table below reports Avg@32 after 200 steps

AMC23AIME24AIME25AIME26HMMT25HMMT26Avg.
BPO93.553.235.845.716.611.842.8
BPO (Top-16 KL Approx)84.745.731.939.011.38.636.9
BPO (Top-128 KL Approx)89.8‾\underline{89.8}47.6‾\underline{47.6}34.4‾\underline{34.4}42.6‾\underline{42.6}13.2‾\underline{13.2}11.3‾\underline{11.3}39.8‾\underline{39.8}

These results are expected from the above analysis. The binary approximation fits the unified framework, whose clipping improves stability, and the ϵ\epsilon smoothing slightly improves the average accuracy (in the ablation on Qwen3-4B-Base). The top-k approximation does not fit the unified framework, and it is unclear how to apply smoothing and clipping to it, so the noise from the tail probability psπ(other)p^\pi_s(\mathrm{other}) is harder to remove

In summary, considering both performance and efficiency, we choose the binary KL approximation

Thank you for clarification

Reviewer_mKxgOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces BPO, a critic-free method for reinforcement learning with verifiable rewards. It derives a trajectory-level objective from Policy Mirror Descent (PMD), then approximates it into a practical token-level update that replaces GRPO’s importance ratio with a complementary-probability ratio. Experiments on mathematical reasoning and tool use show gains over several strong baselines. Evaluated on long-CoT mathematical reasoning and multi-turn tool-integrated reasoning, BPO outperforms GRPO-ClipHigher, GSPO, CISPO, and DPPO on all six evaluated competition math benchmarks

Critical strengths
  • The practical update is simple to implement and requires no extra memory or vocabulary-wide logits computation. Results show consistent gains over GRPO-ClipHigher, GSPO, CISPO, and DPPO, with useful ablations.
  • The martingale proof gives a rigorous equivalence between the exact trajectory-level objective and state-wise PMD on reachable states.
  • The paper is easy to read and in good writing.
Critical weaknesses
  • Step 1 freezes the outer residual at π=μ\pi = \mu, removing the trajectory residual and step size from the update. Theorem 1 therefore does not establish equivalence for practical BPO. The paper also provides no experiments optimizing the exact objective.
  • The role of KL term is unclear, For positive advantages, Eq. 14 rewards increasing log⁡π+DKL(μ∥π)\log\pi+D_{\mathrm{KL}}(\mu\|\pi), so the kl no longer acts as a conventional penalty toward μ\mu. Top-k KL reduces accuracy by 3.0–5.9 relative to Binary KL, raising questions about the proposed derivation. These results alone do not establish the reason.
  • The complementary ratio changes clipping behavior substantially, yet BPO inherits GRPO’s thresholds. With smoothing 0.1, upper clipping allows large relative increases for low-probability tokens. The contribution of this change needs isolation.
Questions for authors

see weeknesses

  • How does the unlinearized objective perform, and which PMD guarantees survive the practical approximations?
  • How do you interpret the KL term after linearization and the weaker Top-k KL results?
  • Can you report clipping rates by rollout probability, tuned baseline comparisons?
6 comments
AuthorsOpenReview ↗

Response to Reviewer mKxg (Part 1)

Thank you very much for your valuable feedback. We have addressed all your concerns as follows.

W1 & Q1:

Step 1 freezes the outer residual at π=μ\pi=\mu, removing the trajectory residual and step size from the update. Theorem 1 therefore does not establish equivalence for practical BPO. The paper also provides no experiments optimizing the exact objective.

How does the unlinearized objective perform, and which PMD guarantees survive the practical approximations?

Thank you for this comment. Theorem 1 establishes the equivalence between the critic-free objective Eq. (11) and PMD, and that it does not directly apply to the practical BPO loss Eq. (9). Eq. (11) requires computing the KL divergence over the full vocabulary at every token, which incurs a huge computational cost and is difficult to apply in practice. We therefore apply the practical approximations, which give the final loss objective the simple form of Eq. (9).

As with GRPO, GSPO, CISPO, and DPPO, rigorous theoretical guarantees are difficult to establish for practical LLM algorithms. Establishing a rigorous theoretical guarantee for Eq. (9) is an interesting theoretical question, which we leave for future work. Eq. (11), which has a rigorous theoretical guarantee, mainly serves to motivate the practical loss Eq. (9). Moreover, Eq. (9) requires neither computing the full-vocabulary KL divergence nor storing the top-k tokens of the rollout engine. It is therefore much faster in practice, while maintaining good performance.

Which PMD guarantees survive the practical approximations? The practical approximations preserve the following property: at the rollout policy π=μ\pi=\mu, the gradient descent step on the BPO loss Eq. (9) and the gradient ascent step on the PMD objective Eq. (19) move in the same direction, in expectation over y∼μy\sim\mu. That is, on each rollout batch, the practical BPO update starts from the same direction as the PMD update. We show this below.

At π=μ\pi=\mu, we have π(a∣st)/μ(a∣st)=1\pi(a|s_t)/\mu(a|s_t)=1 for every token aa. Hence the log-ratios and the KL divergences in Eq. (12) vanish, so the residual equals δ(x,y;μ,μ)=η(R(x,y)−Vμ(x))\delta(x,y;\mu,\mu)=\eta(R(x,y)-V^\mu(x)) and the linear approximation Eq. (13) is exact, with the step size η\eta canceling out. Moreover, the gradients of both the full KL divergence and the binary KL divergence vanish, the weights 1−μ(yt∣st)1−π(yt∣st)\frac{1-\mu(y_t|s_t)}{1-\pi(y_t|s_t)} and ωt\omega_t both equal 11, and therefore Mt=1M_t=1. As a result, at π=μ\pi=\mu, the gradient of the BPO loss Eq. (9) is the same as the gradient of the critic-free objective Eq. (11), with Vμ(x)V^\mu(x) and ϕ(x)\phi(x) replaced by their group estimates.

We next compare it with the PMD objective Eq. (19), which is maximized. Since the gradient of KL(π(⋅∣st)∥μ(⋅∣st))\mathrm{KL}(\pi(\cdot|s_t)\Vert\mu(\cdot|s_t)) vanishes at π=μ\pi=\mu, the gradient of Eq. (19) at π=μ\pi=\mu is the policy gradient Eyt∼μ(⋅∣st)[Aμ(st,yt)∇log⁡π(yt∣st)]\mathbb{E}_{y_t\sim\mu(\cdot|s_t)}\left[A^\mu(s_t,y_t)\nabla\log\pi(y_t|s_t)\right]. Since E[R(x,y)∣st,yt]=Qμ(st,yt)\mathbb{E}[R(x,y)\mid s_t,y_t]=Q^\mu(s_t,y_t), Vμ(x)V^\mu(x) and Vμ(st)V^\mu(s_t) do not depend on yty_t, and Eyt∼μ(⋅∣st)[∇log⁡π(yt∣st)]=0\mathbb{E}_{y_t\sim\mu(\cdot|s_t)}\left[\nabla\log\pi(y_t|s_t)\right]=0 at π=μ\pi=\mu, we have Ey∼μ[ϕ(x)(R(x,y)−Vμ(x))∇log⁡π(yt∣st)]=Ey∼μ[ϕ(x)Aμ(st,yt)∇log⁡π(yt∣st)]\mathbb{E}_{y\sim\mu}\left[\phi(x)(R(x,y)-V^\mu(x))\nabla\log\pi(y_t|s_t)\right]=\mathbb{E}_{y\sim\mu}\left[\phi(x)A^\mu(s_t,y_t)\nabla\log\pi(y_t|s_t)\right] at π=μ\pi=\mu, where the left-hand side is the expectation of the negative BPO gradient and the right-hand side is the gradient of Eq. (19), weighted by ϕ(x)\phi(x) and averaged over the states sts_t visited by μ\mu.

How does the unlinearized objective Eq. (11) perform? We train Qwen3-4B-Base (batch size 128, group size 16, maximum response length 8k, i.e., the setting in Table 3) with the unlinearized objective Eq. (11) and with each intermediate approximation:

  • BPO Eq. (11) denotes Eq. (11) in the submission.
  • BPO Eq. (14) denotes Eq. (14) in the submission, i.e., the expression obtained by applying the linear approximation (Step 1) and group estimation (Step 2) to BPO Eq. (11). The linear approximation avoids sequence-level computation and gives the loss a simpler form.
  • From Eq. (14) to Eq. (16), we apply the binary KL approximation, which gives the weight 1−μ(yt∣st)1−π(yt∣st)\frac{1-\mu(y_t|s_t)}{1-\pi(y_t|s_t)}. However, directly training with this weight is unstable. Since π(yt∣st)\pi(y_t|s_t) can approach 1, we have observed 1−μ(yt∣st)1−π(yt∣st)>107\frac{1-\mu(y_t|s_t)}{1-\pi(y_t|s_t)}>10^7 for some tokens. We therefore apply the ϵ\epsilon smoothing, and substitute 1−μ(yt∣st)1−π(yt∣st)\frac{1-\mu(y_t|s_t)}{1-\pi(y_t|s_t)} or 1+ϵ−μ(yt∣st)1+ϵ−π(yt∣st)\frac{1+\epsilon-\mu(y_t|s_t)}{1+\epsilon-\pi(y_t|s_t)} for the importance-sampling ratio in the unified RLVR framework Eq. (1).
  • BPO (ϵ=0\epsilon=0) denotes the BPO loss Eq. (9) with ϵ=0\epsilon=0.

The table below reports Avg@32 after 200 training steps.

AuthorsOpenReview ↗

Response to Reviewer mKxg (Part 2)

AIME24AIME25AIME26AIME24-26 Avg.
BPO Eq. (11)26.3‾\underline{26.3}25.619.023.6
BPO Eq. (14)26.025.220.223.8
BPO (ϵ=0\epsilon=0)27.025.020.6‾\underline{20.6}24.2‾\underline{24.2}
BPO25.825.3‾\underline{25.3}21.924.3
GRPO-ClipHigher19.018.413.617.0

As shown in the table, BPO performs comparably to the unlinearized objective BPO Eq. (11), as well as to BPO Eq. (14) and BPO (ϵ=0\epsilon=0), and all of them outperform GRPO-ClipHigher. This shows that the practical approximations in Section 3.2 of our submission have a small effect on performance, while avoiding the computation of the full-vocabulary KL divergence.

W2 & Q2:

The role of KL term is unclear, For positive advantages, Eq. 14 rewards increasing , so the kl no longer acts as a conventional penalty toward . Top-k KL reduces accuracy by 3.0–5.9 relative to Binary KL, raising questions about the proposed derivation. These results alone do not establish the reason.

How do you interpret the KL term after linearization and the weaker Top-k KL results?

Thank you for this comment.

Role of the KL term. The KL term in Eq. (12) is not a penalty toward μ\mu. It comes from the log partition function in PMD. The PMD solution π+\pi^+ satisfies Eq. (21): log⁡π+(yt∣st)−log⁡μ(yt∣st)−ηAμ(st,yt)+log⁡Zμ(st)=0.\log\pi^+(y_t|s_t)-\log\mu(y_t|s_t)-\eta A^{\mu}(s_t,y_t)+\log Z_{\mu}(s_t)=0. Taking the expectation over yt∼μ(⋅∣st)y_t\sim\mu(\cdot|s_t) gives log⁡Zμ(st)=KL(μ(⋅∣st)∥π+(⋅∣st))\log Z_\mu(s_t)=\mathrm{KL}(\mu(\cdot|s_t)\Vert\pi^+(\cdot|s_t)) (Eq. (22)). Substituting it back gives Eq. (23): log⁡π+(yt∣st)−log⁡μ(yt∣st)+KL(μ(⋅∣st)∥π+(⋅∣st))=ηAμ(st,yt).\log\pi^+(y_t|s_t)-\log\mu(y_t|s_t)+\mathrm{KL}(\mu(\cdot|s_t)\Vert\pi^+(\cdot|s_t))=\eta A^{\mu}(s_t,y_t). Summing over tt, the advantages telescope to R(x,y)−Vμ(x)R(x,y)-V^\mu(x) by the Bellman equations, which gives the residual δ\delta in Eq. (12). Therefore, the KL term plays the role of log⁡Zμ(st)\log Z_\mu(s_t), which makes δ=0\delta=0 at π=π+\pi=\pi^+.

After linearization, the gradient of log⁡π(yt∣st)+KL(μ(⋅∣st)∥π(⋅∣st))\log\pi(y_t|s_t)+\mathrm{KL}(\mu(\cdot|s_t)\Vert\pi(\cdot|s_t)) with respect to the logits is eyt−μ(⋅∣st)e_{y_t}-\mu(\cdot|s_t), while the standard policy gradient gives eyt−π(⋅∣st)e_{y_t}-\pi(\cdot|s_t). Thus, the KL term only replaces the reference point π\pi with μ\mu, and the two coincide at π=μ\pi=\mu.

Discussion of the weaker Top-k KL results. The top-k approximation is closer to the full KL divergence than the binary approximation, but BPO (Top-k KL Approx) achieves lower Avg@32 than BPO (Table 9). We attribute this gap to the clipping: BPO clips the weight given by the binary approximation, while BPO (Top-k KL Approx) has no clipping for the top-k KL divergence.

Both approximations involve ratios that can become very large. The top-k approximation in Eq. (18) keeps yty_t and the kk most probable tokens under μ(⋅∣st)\mu(\cdot|s_t), and aggregates all remaining tokens into a single category "other". The gradient of psμ(other)log⁡psμ(other)psπ(other)p^\mu_s(\mathrm{other})\log\frac{p^\mu_s(\mathrm{other})}{p^\pi_s(\mathrm{other})} involves the ratio psμ(other)psπ(other)\frac{p^\mu_s(\mathrm{other})}{p^\pi_s(\mathrm{other})}, which can become very large when psπ(other)→0p^\pi_s(\mathrm{other})\to 0. Similarly, the weight 1−μ(yt∣st)1−π(yt∣st)\frac{1-\mu(y_t|s_t)}{1-\pi(y_t|s_t)} given by the binary approximation in Eq. (15) can become very large when 1−π(yt∣st)→01-\pi(y_t|s_t)\to 0, and we have observed values above 10710^7 for some tokens. In BPO, we apply the ϵ\epsilon smoothing in Eq. (10) and substitute the smoothed weight for the importance-sampling ratio in the unified RLVR framework Eq. (1), which applies the clipping mask MtiM_t^i. BPO (Top-k KL Approx) has no such clipping for the ratio of "other".

As shown in our response to W1 & Q1 above, BPO (ϵ=0\epsilon=0) performs close to BPO. Therefore, the stability of BPO mainly comes from the clipping rather than the ϵ\epsilon smoothing. How to clip the top-k KL divergence is an interesting question for future work.

Moreover, the binary approximation is much cheaper. On 32 GPUs, the wall-clock time to reach step 199 is 33.8 hours for BPO, compared with 56.8 hours for BPO (Top-16 KL Approx) and 152.6 hours for BPO (Top-128 KL Approx). Together with its higher Avg@32 in Table 9, this lower cost motivates us to use the binary approximation rather than the top-k approximation.

AuthorsOpenReview ↗

Response to Reviewer mKxg (Part 3)

W3 & Q3:

The complementary ratio changes clipping behavior substantially, yet BPO inherits GRPO’s thresholds. With smoothing 0.1, upper clipping allows large relative increases for low-probability tokens. The contribution of this change needs isolation.

Can you report clipping rates by rollout probability, tuned baseline comparisons?

Thank you for this comment.

Clipping rates by rollout probability. For the long-CoT experiment on Qwen3-30B-A3B-Base (Figure 1), we divide the rollout probability μ\mu into bins of width 0.1 and report the fraction of tokens masked by clipping in each bin for GRPO-ClipHigher and BPO, over all tokens of the rollout batch at training steps 99, 199, 299, and 399.

μ\mu│GRPO-ClipHigher│BPO
│Step 99Step 199Step 299Step 399│Step 99Step 199Step 299Step 399
[0.0, 0.1)│2.80×10−32.80\times10^{-3}1.52×10−31.52\times10^{-3}1.17×10−31.17\times10^{-3}1.26×10−11.26\times10^{-1}│0000
[0.1, 0.2)│9.99×10−49.99\times10^{-4}6.10×10−46.10\times10^{-4}3.42×10−43.42\times10^{-4}9.56×10−39.56\times10^{-3}│01.45×10−51.45\times10^{-5}5.12×10−55.12\times10^{-5}5.14×10−55.14\times10^{-5}
[0.2, 0.3)│3.30×10−43.30\times10^{-4}1.84×10−41.84\times10^{-4}1.33×10−41.33\times10^{-4}1.00×10−21.00\times10^{-2}│2.46×10−52.46\times10^{-5}8.36×10−58.36\times10^{-5}1.39×10−41.39\times10^{-4}2.44×10−42.44\times10^{-4}
[0.3, 0.4)│1.85×10−41.85\times10^{-4}1.13×10−41.13\times10^{-4}6.37×10−56.37\times10^{-5}1.20×10−21.20\times10^{-2}│2.25×10−42.25\times10^{-4}2.98×10−42.98\times10^{-4}9.11×10−49.11\times10^{-4}7.99×10−47.99\times10^{-4}
[0.4, 0.5)│1.12×10−41.12\times10^{-4}9.19×10−59.19\times10^{-5}3.54×10−53.54\times10^{-5}1.51×10−21.51\times10^{-2}│4.35×10−44.35\times10^{-4}8.75×10−48.75\times10^{-4}2.04×10−32.04\times10^{-3}2.39×10−32.39\times10^{-3}
[0.5, 0.6)│9.31×10−59.31\times10^{-5}3.94×10−53.94\times10^{-5}2.64×10−52.64\times10^{-5}2.14×10−22.14\times10^{-2}│5.69×10−45.69\times10^{-4}1.31×10−31.31\times10^{-3}2.96×10−32.96\times10^{-3}3.25×10−33.25\times10^{-3}
[0.6, 0.7)│1.68×10−51.68\times10^{-5}1.26×10−51.26\times10^{-5}1.09×10−51.09\times10^{-5}3.66×10−23.66\times10^{-2}│4.22×10−34.22\times10^{-3}8.23×10−38.23\times10^{-3}1.12×10−21.12\times10^{-2}1.11×10−21.11\times10^{-2}
[0.7, 0.8)│8.91×10−68.91\times10^{-6}1.18×10−51.18\times10^{-5}8.17×10−68.17\times10^{-6}8.39×10−28.39\times10^{-2}│5.03×10−35.03\times10^{-3}8.62×10−38.62\times10^{-3}1.18×10−21.18\times10^{-2}1.15×10−21.15\times10^{-2}
[0.8, 0.9)│6.34×10−66.34\times10^{-6}4.66×10−64.66\times10^{-6}9.05×10−79.05\times10^{-7}2.19×10−12.19\times10^{-1}│3.84×10−33.84\times10^{-3}7.21×10−37.21\times10^{-3}9.95×10−39.95\times10^{-3}9.65×10−39.65\times10^{-3}
[0.9, 1.0]│2.57×10−72.57\times10^{-7}2.21×10−72.21\times10^{-7}2.57×10−72.57\times10^{-7}5.89×10−25.89\times10^{-2}│2.65×10−52.65\times10^{-5}7.47×10−57.47\times10^{-5}1.22×10−41.22\times10^{-4}1.25×10−41.25\times10^{-4}
  • Up to step 299, GRPO-ClipHigher masks mainly tokens with μ<0.1\mu<0.1 (2.80×10−32.80\times10^{-3}, 1.52×10−31.52\times10^{-3}, and 1.17×10−31.17\times10^{-3} at steps 99, 199, and 299), and masks almost no tokens with μ≥0.6\mu\ge 0.6. At step 399, it masks a large fraction of tokens in every bin, up to 0.219 for μ∈[0.8,0.9)\mu\in[0.8,0.9). This coincides with the drop of its mean accuracy over AIME24–26 in Figure 1, from 39.5% at step 349 to 34.7% at step 399.
  • BPO masks almost no tokens with μ<0.3\mu<0.3 (at most 2.44×10−42.44\times10^{-4} in each bin), and masks mainly tokens with μ\mu between 0.6 and 0.9, at most 0.0118 in each bin. Its masking rates at step 399 are nearly the same as at step 299.

Tuned baselines. We further vary the clipping thresholds of GRPO on Qwen3-4B-Base (batch size 128, group size 16, maximum response length 8k, i.e., the setting in Table 3). The upper threshold 1+ϵhigh1+\epsilon_{\mathrm{high}} only applies to tokens with A^i>0\hat A^i>0, and the lower threshold 1−ϵlow1-\epsilon_{\mathrm{low}} only applies to tokens with A^i<0\hat A^i<0. We report Avg@32 at training step 199.

Method1+ϵhigh1+\epsilon_{\mathrm{high}} (A^i>0\hat A^i>0)1−ϵlow1-\epsilon_{\mathrm{low}} (A^i<0\hat A^i<0)AIME24AIME25AIME26Avg.
GRPO-ClipHigher1.280.819.018.413.617.0
GRPO1.10.918.818.114.117.0
GRPO2.00.515.416.112.314.6
BPO1.280.825.825.321.924.3

BPO outperforms GRPO with all three clipping thresholds on each benchmark, and its average accuracy is 7.3 percentage points higher than the best GRPO setting.

Reviewer_atRuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Bellman Policy Optimization (BPO), a critic-free policy optimization method for RL with verifiable rewards. Starting from policy mirror descent and the Bellman equations for terminal rewards, the authors derive a trajectory-level objective that avoids estimating intermediate state values. The practical method uses group-normalized rewards, a binary-KL approximation, additive smoothing, and a complementary-token-probability mismatch correction in place of the usual importance-sampling ratio. Experiments on long-chain-of-thought mathematical reasoning and multi-turn tool-integrated reasoning show gains over GRPO-ClipHigher, GSPO, CISPO, and DPPO.

Critical strengths
  1. The paper identifies a real engineering problem in RLVR: value/critic estimation is expensive and can be unreliable for long reasoning trajectories. The Bellman telescoping argument gives a clean motivation for eliminating intermediate value estimates.

  2. The empirical comparison is reasonably broad for an RLVR method. BPO is evaluated on both long-CoT mathematical reasoning and tool-integrated reasoning, with several recent policy-optimization baselines and ablations for smoothing, truncation, mismatch correction, clipping, and KL approximation.

Critical weaknesses
  1. The strongest theoretical claim applies to the critic-free trajectory-level objective, but the practical BPO loss is obtained through several additional approximations: local linearization, group estimation, binary KL, smoothing, clipping, and truncation. The paper does not provide a bound or convergence argument relating the practical loss to the PMD objective. Thus, the PMD equivalence does not by itself justify the algorithm actually trained.

  2. The empirical gains are not accompanied by multi-seed uncertainty or a cost comparison. The reported differences may be meaningful, but the paper should report variation across runs and wall-clock/GPU-memory/throughput costs to support the claim that critic-free training is practically advantageous.

Questions for authors
  1. Can the authors quantify the bias introduced by each approximation between the PMD gradient and the practical BPO gradient, especially binary-KL replacement and additive smoothing?

  2. Are the reported results averaged over multiple random seeds? If so, please report mean, standard deviation or confidence intervals, and per-seed results for the main table.

  3. How do BPO and the baselines compare in wall-clock time, GPU-hours, memory usage, and rollout/training throughput under matched hardware? The paper’s practical motivation depends on this comparison.

  4. How sensitive are the conclusions to group size, rollout-policy lag, reward sparsity, and response-length cap? These factors may affect the mismatch-correction weight substantially.

3 comments
AuthorsOpenReview ↗

Response to Reviewer atRu (Part 1)

Thank you very much for your valuable feedback. We have addressed all your concerns as follows.

W1 & Q1: Gap between Theorem 1 and the practical loss

The strongest theoretical claim applies to the critic-free trajectory-level objective, but the practical BPO loss is obtained through several additional approximations: local linearization, group estimation, binary KL, smoothing, clipping, and truncation. The paper does not provide a bound or convergence argument relating the practical loss to the PMD objective. Thus, the PMD equivalence does not by itself justify the algorithm actually trained

Can the authors quantify the bias introduced by each approximation between the PMD gradient and the practical BPO gradient, especially binary-KL replacement and additive smoothing?

Thank you for this comment. Theorem 1 proves that the critic-free objective Eq. (11) is equivalent to the original PMD. However, Eq. (11) requires computing the KL divergence over the full vocabulary, which incurs a huge computational cost and is difficult to apply in practice. We therefore apply the practical approximations in Section 3.2, which give the final loss objective the simple form of Eq. (9).

As with GRPO, GSPO, CISPO, and DPPO, rigorous theoretical guarantees are difficult to establish for practical LLM algorithms. Establishing a rigorous theoretical guarantee for Eq. (9) is an interesting theoretical question, which we leave for future work. Eq. (11), which has a rigorous theoretical guarantee, mainly serves to motivate the practical loss Eq. (9). Moreover, Eq. (9) requires neither computing the full-vocabulary KL divergence nor storing the top-k tokens of the rollout engine. It is therefore much faster in practice than computing the full-vocabulary KL divergence or taking the top-k tokens from the rollout engine, while maintaining good performance. This makes Eq. (9) more practical, and we therefore present Eq. (9) as the main algorithm.

The effect of each approximation step on performance is described below. We train Qwen3-4B-Base (batch size 128, group size 16, maximum response length 8k, i.e., the setting of the ablation on BPO hyperparameters in Table 3) and report Avg@32 after 200 training steps.

  • BPO Eq. (11) denotes Eq. (11) in the submission.
  • BPO Eq. (14) denotes Eq. (14) in the submission, i.e., the expression obtained by applying the linear approximation (Step 1) and group estimation (Step 2) to BPO Eq. (11). The linear approximation avoids sequence-level computation and gives the loss a simpler form.
  • From Eq. (14) to Eq. (16), we apply the binary KL approximation, which gives the weight 1−μ(yt∣st)1−π(yt∣st)\frac{1-\mu(y_t|s_t)}{1-\pi(y_t|s_t)}. However, directly training with this weight is unstable. Since π(yt∣st)\pi(y_t|s_t) can approach 1, we have observed 1−μ(yt∣st)1−π(yt∣st)>107\frac{1-\mu(y_t|s_t)}{1-\pi(y_t|s_t)}>10^7 for some tokens. We therefore apply the ϵ\epsilon smoothing, and substitute 1−μ(yt∣st)1−π(yt∣st)\frac{1-\mu(y_t|s_t)}{1-\pi(y_t|s_t)} or 1+ϵ−μ(yt∣st)1+ϵ−π(yt∣st)\frac{1+\epsilon-\mu(y_t|s_t)}{1+\epsilon-\pi(y_t|s_t)} for the importance-sampling ratio in the unified RLVR framework Eq. (1).
  • BPO (ϵ=0\epsilon=0) denotes the BPO loss Eq. (9) with ϵ=0\epsilon=0.
AIME24AIME25AIME26AIME24-26 Avg.
BPO Eq. (11)26.3‾\underline{26.3}25.619.023.6
BPO Eq. (14)26.025.220.223.8
BPO (ϵ=0\epsilon=0)27.025.020.6‾\underline{20.6}24.2‾\underline{24.2}
BPO25.825.3‾\underline{25.3}21.924.3
GRPO-ClipHigher19.018.413.617.0

As shown in the table, BPO, BPO Eq. (11), BPO Eq. (14), and BPO (ϵ=0\epsilon=0) achieve close average accuracy, and all of them outperform GRPO-ClipHigher. This shows that the practical approximations in Section 3.2 have a small effect on performance, while avoiding the computation of the full-vocabulary KL divergence and saving computation.

AuthorsOpenReview ↗

Response to Reviewer atRu (Part 2)

W2 & Q3: Cost comparison

The empirical gains are not accompanied by multi-seed uncertainty or a cost comparison. The reported differences may be meaningful, but the paper should report variation across runs and wall-clock/GPU-memory/throughput costs to support the claim that critic-free training is practically advantageous.

How do BPO and the baselines compare in wall-clock time, GPU-hours, memory usage, and rollout/training throughput under matched hardware? The paper’s practical motivation depends on this comparison.

Thank you for this suggestion. We compare the training cost of all methods in the main experiment (Figure 1). All methods use the same configuration and trained on 32 GPUs.

Accuracy at the same wall-clock time. The table below reports the AIME24-26 average accuracy after the same wall-clock training time, linearly interpolated between adjacent evaluation points.

20 h40 h60 h
BPO37.9‾\underline{37.9}47.148.0
GRPO-ClipHigher27.036.436.2
GSPO36.840.341.7
CISPO38.343.1‾\underline{43.1}45.7‾\underline{45.7}
DPPO29.542.445.5

At 40 h and 60 h, BPO achieves the highest accuracy, 4.0 and 2.3 points higher than the strongest baseline, respectively. At 20 h, BPO is second only to CISPO (37.9 vs. 38.3).

Rollout and training throughput. The table below reports the average throughput of all 32 GPUs in thousand tokens per second. The rollout throughput counts only the generated response tokens, and the training throughput counts the prompt and response tokens used in the actor updates. Each is divided by the time of the corresponding stage, excluding evaluation, checkpointing, and waiting time. The average is the total number of tokens divided by the total stage time.

Rollout (k tokens/s)Training (k tokens/s)
BPO48.391.0
GRPO-ClipHigher45.087.4
GSPO44.186.6
CISPO51.593.8
DPPO47.684.0

BPO achieves rollout and training throughput comparable to the baselines. This is expected. For example, when compared with GRPO, BPO only changes the per-token weight, which is computed from π(yt∣st)\pi(y_t|s_t) and μ(yt∣st)\mu(y_t|s_t), the same quantities already used by the importance ratio in GRPO. BPO therefore requires no critic, no additional forward pass, and no full-vocabulary computation.

GPU memory. The table below reports the average per-GPU memory (GiB) over the whole training run, covering the rollout, training, and switching phases.

BPOGRPO-ClipHigherGSPOCISPODPPO
Memory (GiB)98.097.998.6101.799.7

BPO uses comparable GPU memory to the other baselines.

Q2: Results over multiple random seeds

Are the reported results averaged over multiple random seeds? If so, please report mean, standard deviation or confidence intervals, and per-seed results for the main table.

Thank you for this suggestion. Due to limited computational resources, we conduct the multi-seed study on Qwen3-1.7B. For each of GRPO-ClipHigher and BPO, we run 5 independent trials with random seeds 0, 1, 2, 3, and 4, keeping all other settings identical across runs (batch size 128, group size 8, 4 minibatches per batch, maximum response length 8k, and 200 training steps). We report the per-seed AIME24/25/26 Avg@32 of last iterate, together with the mean and standard deviation across seeds.

│GRPO-ClipHigher│BPO
│AIME24AIME25AIME26Avg.│AIME24AIME25AIME26Avg.
seed 0│29.430.026.328.5│35.931.331.532.9
seed 1│35.029.927.830.9│36.831.928.232.3
seed 2│29.828.323.027.0│34.332.226.931.1
seed 3│28.827.224.826.9│35.527.329.030.6
seed 4│32.828.527.729.7│38.129.626.831.5
Mean ± std│31.1 ± 2.428.8 ± 1.125.9 ± 1.828.6 ± 1.5│36.1 ± 1.330.4 ± 1.828.5 ± 1.731.7 ± 0.8

The results show that the advantage of BPO is consistent across random seeds rather than an artifact of a particular seed:

  • BPO achieves a higher AIME24-26 average than GRPO-ClipHigher under every one of the 5 seeds
  • In terms of the mean over seeds, BPO improves over GRPO-ClipHigher by 5.0, 1.6, 2.5, and 3.1 points on AIME24, AIME25, AIME26, and the average, respectively. The 3.1-point gain in the average is about twice the standard deviation of GRPO-ClipHigher (1.5) and nearly four times that of BPO (0.8).
  • BPO also exhibits a smaller seed-to-seed variation in the average accuracy (std 0.8 vs. 1.5).
AuthorsOpenReview ↗

Response to Reviewer atRu (Part 3)

Q4: Sensitivity Analysis

How sensitive are the conclusions to group size, rollout-policy lag, reward sparsity, and response-length cap? These factors may affect the mismatch-correction weight substantially.

Thank you for this question. We study the sensitivity to group size, rollout-policy lag, and response-length cap on Qwen3-1.7B. Unless otherwise stated, all runs use the same setting as in our response to Q2 (128 prompts per step, group size 8, 4 minibatches per rollout batch, response-length cap 8k), and we vary one factor at a time. We report the AIME24/25/26 Avg@32 after 200 training steps.

Group size. We vary the number of responses per prompt among 4, 8, and 16, keeping all other settings the same.

Group size│GRPO-ClipHigher│BPO
│AIME24AIME25AIME26Avg.│AIME24AIME25AIME26Avg.
4│31.827.925.928.5│35.129.027.230.4
8│29.430.026.328.5│35.931.331.532.9
16│31.728.425.728.6│36.729.930.532.4

BPO outperforms GRPO-ClipHigher on every benchmark under all three group sizes.

Response-length cap. We vary the maximum response length among 4k, 8k, and 16k tokens, which is used for both training and evaluation.

Response-length│GRPO-ClipHigher│BPO
│AIME24AIME25AIME26Avg.│AIME24AIME25AIME26Avg.
4k│24.327.020.523.9│26.823.021.723.8
8k│29.430.026.328.5│35.931.331.532.9
16k│38.534.935.136.2│46.038.139.241.1

With the 8k and 16k caps, BPO outperforms GRPO-ClipHigher on every benchmark. With the 4k cap, the two methods are comparable in average accuracy. The gain of BPO in average accuracy increases with the response-length cap.

Rollout-policy lag. We vary the number of policy updates performed on each rollout batch among 2, 4, and 8. For a fair comparison, we fix the minibatch size to 256 responses, so that every policy update uses the same number of responses.

Policy lag│GRPO-ClipHigher│BPO
│AIME24AIME25AIME26Avg.│AIME24AIME25AIME26Avg.
2│30.728.526.028.4│36.027.026.629.9
4│29.430.026.328.5│35.931.331.532.9
8│31.429.727.829.6│35.831.828.932.2

BPO achieves a higher average accuracy under all three lags.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.