acceptodds
Under review as a conference paper at ICLR 2027

Off-Policy Value-Based Reinforcement Learning for Large Language Models

Abstract

Improving data utilization efficiency is critical for scaling reinforcement learning (RL) for long-horizon tasks where generating trajectories is expensive. However, the dominant RL methods for LLMs are largely on-policy: they update each batch of data only once, discard it, and then collect fresh samples, resulting in poor sample efficiency. In this work, we explore an alternative value-based RL framework for LLMs that naturally enables off-policy learning. We propose ReVal, a Bellman-update-based method that combines stepwise signals capturing internal consistency with trajectory-level signals derived from outcome verification. ReValnaturally supports replay-buffer-based training, allowing efficient reuse of past trajectories. Experiments on standard mathematical reasoning benchmarks show that ReVal not only converges faster but also outperforms GRPO in final performance. On DeepSeek-R1-Distill-1.5B, ReVal improves training efficiency and achieves improvement of 2.7% in AIME24 and 4.5% in out-of-domain benchmark GPQA over GRPO. These results suggest that value-based RL is a practical alternative to policy-based methods for LLM training.

Primary area:
reinforcement learning
Keywords:
Large Language Model, Value-Based RL, Off-Policy RL

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MvAdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReVal, a value-based, off-policy RL method for LLM post-training. Building on TBRM, which treats LLM logits as soft Q-values, ReVal adds a reward-shaping term with the reference value V_ref as the potential, so that the loss vanishes at the reference policy when the reward is zero (“Calibrated Initialization”), and trains from a FIFO replay buffer. Trained on DeepScaleR, ReVal achieves higher average accuracy than GRPO and TBRM on DeepSeek-R1-Distill-1.5B (45.6 vs. 43.4 for GRPO) and Qwen2.5-Math-7B.

Critical strengths
  1. A real initialization issue with a simple fix. With unnormalized logits, TBRM is not stationary at the reference model even under zero reward (Fig. 9). Subtracting V_ref(s₁) makes the reference policy a zero-loss point at no extra cost, which is useful for any logits-as-Q method.

  2. Consistent empirical signal. ReVal stays above GRPO for most of training on DeepSeek-R1-Distill-1.5B (Fig. 10a) and clearly beats TBRM on both models (Table 1).

  3. Informative ablations and failure cases. Reference resets avoid saturation (Fig. 5), β must be set according to response length (Sec. 4.5.2), and normalized rewards beat 0/1 and ±1 rewards (Fig. 7); collapse at β = 0.002 and the unstable regression variant are also reported (Figs. 6, 11).

Critical weaknesses
  1. The theory conflates optimality with stationarity, and the contribution over TBRM is narrower than claimed. Def. 1 and Prop. 1 are stated in terms of the optimal policy, but the paper notes that the shaping term does not change the optimal solution (L205–208); since ReVal is TBRM plus this term, the two losses share the same optimum. What the argument shows is that the pretrained model is not a stationary point of TBRM, which is the property ReVal fixes. Off-policy training is also not new: TBRM already claims convergence from arbitrary off-policy data, and Eq. 2 is the trajectory balance loss with R = π_ref·e^(r/β) that TBA[1] trains from a replay buffer. What remains new is using V_θ(s₁) − V_ref(s₁) as log Z, and its benefit needs to be shown.

  2. The experiments do not isolate the source of the gains. ReVal adds both calibration and replay (plus reward normalization and reference resets on Qwen), yet Table 1 compares it only with on-policy GRPO and TBRM, and the paper does not report the baselines' update counts. Without TBRM with the same buffer, ReVal without it, and GRPO with matched updates or the same buffer, the gains cannot be attributed to the value-based objective; on Qwen, ReVal without resets (Fig. 5) or with the 0/1 reward of Eq. 2 (Fig. 7) ends below GRPO's 38.4.

  3. The evaluation cannot support gains of 2–4 points. No seeds or confidence intervals are reported, benchmarks are subsampled to at most 100 problems (App. A), and hyperparameters such as β are tuned with ablations on the same test benchmarks (Sec. 4.5). GPQA is four-way multiple choice, yet the DPSK base model (15.8) and all four Qwen results (12.8–24.8) fall below the 25% chance level, so the GPQA gains may partly reflect answer formatting.

[1] Bartoldson, B., Venkatraman, S., Diffenderfer, J., Jain, M., Ben-Nun, T., Lee, S., ... & Kailkhura, B. (2026). Trajectory balance with asynchrony: Decoupling exploration and learning for fast, scalable llm post-training. Advances in Neural Information Processing Systems, 38, 113901-113931.

Questions for authors
  1. Regarding W1: do you agree that, with a fully expressive parameterization, TBRM's minimizer at r = 0 is π_ref, so that Prop. 1 is really about non-stationarity at initialization rather than about the optimum? If not, what property of the actual parameterization prevents it?

  2. Regarding W2: Sec. 3.3, App. A, and Alg. 1 describe the update schedule differently. Which one was used, and how many gradient updates per rollout does each method perform? Do the DPSK runs in Table 1 also use reward normalization and reference resets? If GRPO gets at least as many updates and the gap persists, this concern would be largely resolved.

  3. Regarding W3: why are the GPQA scores below chance? Please report the answer-extraction failure rate, and state which checkpoint Table 1 reports and how many seeds were run.

7 comments

Weaknesses 1 and Question 1:The theory conflates optimality with stationarity, and the contribution over TBRM is narrower than claimed.

Thank you for pointing this out. Our original claim about TBRM’s optimal policy was incorrect, and we apologize for the confusion. When the task reward is zero, the reference policy is already optimal for KL-regularized RL, so further policy optimization is unnecessary. However, TBRM can have a nonzero loss and can still produce parameter updates; Our zero-reward experiment in Figure 9 illustrates the resulting policy drift in that run. ReVal instead ensures zero loss and zero loss gradient at this initialization, consistent with Calibrated Initialization. We have corrected Definition 1, Proposition 1, and the accompanying proof accordingly.

Although TBRM supports off-policy learning, effectively incorporating a replay buffer into LLM training is not straightforward. We systematically investigate replay frequency, reference-policy updates, reward scaling, and the treatment of incorrect responses. Directly adding replay to TBRM yields comparatively modest improvements: its average score increases by 0.47 percentage points on Qwen2.5-Math-7B and 1.82 percentage points on DPSK-R1-Distill-1.5B. Even with the same replay-buffer configuration and update schedule, TBRM remains 2.23 and 1.49 percentage points below ReVal, respectively, showing that adding replay alone does not reproduce ReVal’s gains.

[1] Yuan et al. (2025). Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning.

[2] Bartoldson et al. (2025). Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training.

Weakness 2:The experiments do not isolate the source of the gains.

Thank you for suggesting this controlled comparison. Although TBRM supports off-policy learning, effectively incorporating a replay buffer into LLM training is not straightforward. The original TBRM training procedure uses an on-policy schedule without replay-buffer updates [1, 2], which our original TBRM baseline followed.

Our work develops an initialization-calibrated objective and systematically investigates replay frequency, reference-policy updates, reward scaling, and the treatment of incorrect responses in Sections 4.2 and 4.5. These experiments guide a practical training recipe combining fresh-data and replay updates, periodic reference resets, model-appropriate reward scaling, and mean-centered rewards. To isolate the contribution of reward shaping within this recipe, we reran TBRM using the same FIFO replay-buffer configuration and update schedule as ReVal, with the other settings held constant. The results are summarized below.

Qwen2.5-Math-7B

MethodAIME24AIME25AMCMATHMINERVAOlympiadGPQAAvg
TBRM30.81058.374.427.337.819.936.9
TBRM + buffer28.7513.3359.4874.7529.8738.7516.8737.4
ReVal3413.360.275.230.739.224.839.6

DPSK-R1-Distill-1.5B

MethodAIME24AIME25AMCMATHMINERVAOlympiadGPQAAvg
TBRM26.922.165.181.328.844.627.342.3
TBRM + buffer27.523.9566.1183.1230.6844.9332.5644.12
ReVal32.123.868.684.630.346.633.345.6

Directly adding replay to TBRM yields comparatively modest gains, improving its average score by approximately 0.5 percentage points on Qwen2.5-Math-7B and 1.8 percentage points on DPSK-R1-Distill-1.5B. Under matched replay settings, ReVal still outperforms TBRM with replay by approximately 2.2 and 1.5 percentage points, respectively. These controlled comparisons support the benefit of ReVal’s reward shaping beyond additional data reuse.

We also tested GRPO with replay-buffer updates. In this run, training collapsed and the model’s evaluated performance dropped to zero, further illustrating that effective off-policy training cannot be assumed to follow from simply adding replay updates.

[1] Yuan et al. (2025). Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning, Section 3, Algorithm 1.

[2] TBRM official implementation: verl/tbrm/ray_trainer.py, lines 90–165.

Weakness 3: The evaluation cannot support gains of 2–4 points. No seeds or confidence intervals are reported, benchmarks are subsampled to at most 100 problems (App. A), and hyperparameters such as β are tuned with ablations on the same test benchmarks (Sec. 4.5).

Thank you for requesting these clarifications. The evaluation cap also applies to Table 1. All methods use exactly the same fixed evaluation subset. For smaller benchmarks, such as AIME24, AIME25, and AMC, we report avg@16 to reduce evaluation variability. Other benchmarks, such as MATH, contain substantially more questions, making repeated evaluation with 16 responses per question very time-consuming. To accelerate evaluation, we randomly select a fixed subset of 100 questions from each larger benchmark following Knapsack RL [1]. Our preliminary comparisons using 100-question subsets versus the full benchmarks showed only small differences in the resulting scores, supporting this choice as a practical balance between evaluation cost and reliability.

Regarding independent training runs, LLM RL experiments can exhibit limited variation across training seeds, and single-seed training is used in practice [2, 3]. Our original experiments used one training run per configuration. To address the reviewer’s concern, we conducted an additional independent ReVal run on DPSK-R1-Distill-1.5B, bringing the total to two runs for this configuration.

MethodAIME24AIME25AMCMATHMINERVAOlympiadGPQAAvg
GRPO29.4024.4065.0082.6027.6046.3028.8043.44
ReVal32.1023.8068.6084.6030.3046.6033.3045.61
ReVal (Additional Run)31.4522.0869.1285.3729.0647.0034.3145.48

The additional results yield an average of 45.48%, compared with 45.61% for the original ReVal results and 43.44% for GRPO. The two ReVal results differ by only 0.13 percentage points, suggesting consistent average performance across the two runs.

[1] Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocation. arXiv:2509.25849, 2025.

[2] ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models. ICML, 2024.

[3] Generalizable End-to-End Tool-Use RL with Synthetic CodeGym. ICLR, 2026.

Question 2:Regarding W2: Sec. 3.3, App. A, and Alg. 1 describe the update schedule differently. Which one was used, and how many gradient updates per rollout does each method perform? Do the DPSK runs in Table 1 also use reward normalization and reference resets? If GRPO gets at least as many updates and the gap persists, this concern would be largely resolved.

Thank you for pointing out these inconsistencies. The experiments followed the schedule described in Appendix A. After each generation round, ReVal performs one update on the newly collected batch, followed by one update on a batch sampled from the FIFO replay buffer. GRPO and the original TBRM baseline each perform one update on the newly collected batch. Thus, ReVal performs two gradient updates per generation round, while GRPO and TBRM each perform one.

Section 3.3 incorrectly describes both ReVal updates as replay-buffer updates, while Algorithm 1 omits the update on newly collected data. We apologize for these reporting errors and will align both descriptions with the actual implementation.

The DPSK-R1-Distill-1.5B ReVal experiments in Table 1 also use reward centering and periodic reference resets, following the same procedure as Qwen2.5-Math-7B.

To assess the contribution beyond additional data reuse, we additionally evaluated TBRM with the same FIFO replay buffer and update schedule as ReVal. This comparison addresses the concern for TBRM; a GRPO comparison with a matched update budget remains necessary to fully address the corresponding concern.

Question 3:Regarding W3: why are the GPQA scores below chance? Please report the answer-extraction failure rate, and state which checkpoint Table 1 reports and how many seeds were run.

We found that the below-chance GPQA scores are largely explained by answer-extraction failures. For example, approximately 69% of responses from DeepSeek-R1-Distill-1.5B have no extractable answer, with almost all such failures arising from responses truncated at the generation limit. Among responses with an extractable answer, accuracy is 49.8%, well above chance. However, this subset differs across methods and checkpoints, making accuracy conditioned on successful extraction unsuitable for direct comparison. We therefore report the overall score, counting responses without an extractable answer as incorrect, so that every method is evaluated on the same full question set.

Thank you for the detailed response. Could you explain on why training collapsed when replay-buffer updates were added?

Reviewer_MeQiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReVal, a value-based off-policy reinforcement learning framework for LLM post training that aims to overcome the high generation cost and data inefficiency of on-policy RL methods. Conceptually, the work shows that value-based off-policy RL can be a practical and sample-efficient alternative to policy-based methods for LLM post-training, especially in long-horizon or generation-limited settings.

Critical strengths
  1. Addresses a real and increasingly important inefficiency: The paper targets a genuine bottleneck in LLM post-training: on-policy methods like GRPO and DAPO discard every batch after a single update, and generation dominates training cost. As LLM training moves toward long-horizon, agentic tasks, this inefficiency becomes more severe. Framing the problem around the coupling Kgeneration=KupdateK_{\text{generation}} = K_{\text{update}} and the cost asymmetry Tgeneration≫TupdateT_{\text{generation}} \gg T_{\text{update}} is clear and well motivated.

  2. Identifies and fixes a concrete failure mode in prior work: The calibrated-initialization analysis is the paper's strongest technical contribution. The authors pinpoint a specific flaw in TBRM, spurious policy drift when rrule=0r_{\text{rule}} = 0, and propose a principled reward-shaping correction that guarantees zero loss and zero gradient at the reference policy when no reward is available. This is a well-motivated diagnostic and repair, not a vague heuristic.

Critical weaknesses
  1. The single most significant weakness is that the training objective is gauge-dependent, and the propose method cannot solve it. TBRM/ReVal loss is:

L=(log⁡πθ(τ)−log⁡πref(τ)−rrule/β+Vθ(s1))2\mathcal{L} = \big(\log \pi_\theta(\tau) - \log \pi_{\text{ref}}(\tau) - r_{\text{rule}}/\beta + V_\theta(s_1)\big)^2

Under a global shift of all logits by a constant cc:

  • log⁡πθ(τ)\log \pi_\theta(\tau) is invariant (softmax normalizes it away),
  • Vθ(s1)→Vθ(s1)+cV_\theta(s_1) \to V_\theta(s_1) + c,
  • log⁡πref(τ)\log \pi_{\text{ref}}(\tau) and rruler_{\text{rule}} are fixed.

So the residual changes by cc. The optimizer can therefore reduce the loss by adjusting the logit offset, even though this has zero effect on the policy πθ\pi_\theta. This is wasted gradient, or worse, it drives logit magnitudes up or down (entropy collapse / overconfidence) without any corresponding policy improvement.

The calibrated-initialization correction adds a shaping term so that when rrule=0r_{\text{rule}} = 0 and πθ=πref\pi_\theta = \pi_{\text{ref}}, the loss is exactly zero. This handles the degenerate case where there is no reward at all. But even when rrule>0r_{\text{rule}} > 0, the gauge freedom remains: the optimizer can still shift the logit offset to fit the residual, and the loss provides no principled constraint against it. The reference-policy term log⁡πref(τ)\log \pi_{\text{ref}}(\tau) anchors the relative logits, but not the absolute offset that enters Vθ(s1)V_\theta(s_1). In other words, the paper diagnoses a real calibration failure, but its remedy addresses a measure-zero case rather than the general instability. Training stability in practice relies on empirical knobs: the temperature β\beta, gradient clipping, and the anchoring effect of the reference policy rather than a theoretically grounded solution.

  1. Secondary drawback: No off-policy correction analysis. The replay buffer reuses trajectories from old policies, but the paper does not analyze the distribution shift between buffered data and the current policy, nor does it discuss target networks or other standard stabilizers for Bellman-residual training with function approximation. In off-policy RL, the gradient of the Bellman residual is only an unbiased estimate of the true gradient if you correct for the fact that actions were sampled from a different policy.

  2. Minor problems. In Section 3.2, several sentences are redundant or repetitive—for example, Proposition 1 and the sentence immediately following it restate essentially the same point, and this passage could be tightened. In addition, because the paper builds directly on TBRM, it would benefit from more explanation of TBRM’s method before presenting the proposed modifications. In particular, the paper should clarify why TBRM designs its loss function in that form, why the V1V_1 term is incorporated, and what role it plays in the objective. Providing this background would make the motivation for ReVal’s changes easier to follow and would help readers who are unfamiliar with TBRM understand exactly what is being inherited versus what is being fixed.

Questions for authors
  1. The calibrated-initialization property is enforced only for the degenerate case rrule=0r_{\text{rule}} = 0 and πθ=πref\pi_\theta = \pi_{\text{ref}}. When rrule>0r_{\text{rule}} > 0, the loss remains sensitive to a global shift of all logits (since Vθ(s1)V_\theta(s_1) shifts with the offset while log⁡πθ(τ)\log \pi_\theta(\tau) does not). Do you have evidence that this residual gauge freedom does not drive logit-scale drift or entropy collapse during training? Can you measure the logit norm over training and correlate it with performance? The shaping term Vθ(sh)−Vref(sh)V_\theta(s_h) - V_{\text{ref}}(s_h) partially cancels a common offset, but VrefV_{\text{ref}} is fixed while VθV_\theta moves. Does this term fully remove the gauge dependence, or only attenuate it? Is there a formal argument that the loss is gauge-invariant when rrule>0r_{\text{rule}} > 0?

  2. ReVal trains a Bellman residual with function approximation, bootstrapping, and off-policy replay data: the classical "deadly triad." You do not use a target network or importance correction. What mechanism, if any, guarantees that this combination remains stable? Is the reference-policy term acting as an implicit regularizer, and if so, can you make that argument precise?

  3. Trajectories in the buffer were generated by older policies, yet the loss evaluates log⁡πθ(τ)\log \pi_\theta(\tau) under the current policy. Why is no importance-sampling correction, or a bound on staleness, needed? Can you bound the bias introduced by this mismatch?

  4. All experiments use 1.5B and 7B models on math/reasoning benchmarks. How do the method's stability properties and gains change at larger scale or on general-purpose tasks, where logit drift and off-policy staleness would likely be more severe?

5 comments

Weakness 1 and Question 1: The single most significant weakness is that the training objective is gauge-dependent, and the propose method cannot solve it.

Thank you for raising this concern. We agree that ReVal remains gauge-dependent. However, compared with TBRM, ReVal can mitigate this issue by removing the reward-independent logit shift associated with VrefV_{\mathrm{ref}}. This benefit is not limited to zero-reward initialization: it also applies to the Bellman-consistent solution when rewards are nonzero, as we explain below.

When task rewards are positive, the same correction remains useful. Appendix C.1 derives the Bellman-consistent solution:

QReVal∗(s,a)=Qref(s,a)+gr(s,a),Q_{\mathrm{ReVal}}^*(s,a)=Q_{\mathrm{ref}}(s,a)+g_r(s,a),

QTBRM∗(s,a)=Qref(s,a)+gr(s,a)−Vref(s),Q_{\mathrm{TBRM}}^*(s,a)=Q_{\mathrm{ref}}(s,a)+g_r(s,a)-V_{\mathrm{ref}}(s),

where

gr(s,a)=r(s,a)β+log⁡Z(s′),g_r(s,a)=\frac{r(s,a)}{\beta}+\log Z(s'),

Z(s)=Eπref[eGs/β∣s],Z(s)=\mathbb{E}_{\pi_{\mathrm{ref}}}\left[e^{G_s/\beta}\mid s\right],

and GsG_s denotes the remaining task reward. Relative to the same pretrained initialization, ReVal therefore requires the reward-dependent correction grg_r, while TBRM additionally includes the reward-independent recentering term −Vref-V_{\mathrm{ref}}. Removing this additional term can mitigate the extra displacement associated with large initial logit offsets, even when rewards are nonzero.

Empirically, at the evaluated checkpoint, ReVal exhibits a policy entropy of approximately 0.67, which is slightly lower than TBRM (0.74) but still higher than GRPO (0.50), indicating greater policy diversity than GRPO. Together with the observed benchmark improvements, these results provide no indication of entropy or performance collapse in this comparison.

Weakness 2 and Question 3: No off-policy correction analysis. The replay buffer reuses trajectories from old policies, but the paper does not analyze the distribution shift between buffered data and the current policy, nor does it discuss target networks or other standard stabilizers for Bellman-residual training with function approximation.

Thank you for raising this concern. Importance-sampling correction is not a universal requirement for value-based off-policy RL. Standard on-policy policy-gradient estimators compute an expectation under the current policy’s trajectory distribution [1]. When trajectories come from an older policy, directly averaging their gradient contributions generally gives a biased estimate of that expectation. Importance sampling accounts for this mismatch by reweighting samples according to their relative probabilities under the target and behavior policies [2].

ReVal is derived from the soft Bellman optimality operator, which does not require training trajectories to be sampled from the current policy distribution. Following TBRM [3], it directly minimizes a trajectory-level residual objective defined over replay-buffer data. With the buffer, reference model, and reward targets fixed, uniform minibatch sampling provides an unbiased estimate of the full gradient of this replay-buffer objective. Therefore, importance-sampling correction is unnecessary for this estimator.

[1] Sutton et al. (1999). Policy Gradient Methods for Reinforcement Learning with Function Approximation.

[2] Degris, White, and Sutton (2012). Off-Policy Actor-Critic.

[3] Yuan et al. (2025). Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning.

Weakness 3: Minor problems.

Thank you for these helpful suggestions. We have streamlined Section 3.2 by removing redundant restatements around Proposition 1. The subsequent explanation focuses on how the initialization residual arises from the pretrained model’s raw logits. We have also added Appendix C, “Derivation of the TBRM Objective,” immediately after the maximum-entropy RL background in Appendix B. This appendix introduces TBRM’s modified reward and soft Bellman operator, then derives its trajectory-level loss by summing the token-level Bellman residuals. In particular, it explicitly shows that the intermediate value terms cancel, leaving Vθ(s1)V_\theta(s_1) as the initial-state boundary term when the terminal value is zero. We also explain its equivalence to TBRM’s original expression involving the first-token raw logit and subsequent token log-probabilities. Section 3.1 now points readers to this derivation before introducing ReVal’s modifications.

Question 2:ReVal trains a Bellman residual with function approximation, bootstrapping, and off-policy replay data: the classical "deadly triad.

Thank you for raising this concern. The intermediate bootstrap value terms cancel when we sum the Bellman residuals over a complete trajectory. At step hh, the residual Qθ(sh,ah)−(TβQθ)(sh,ah)Q_{\theta}(s_h,a_h)-(\mathcal{T}_{\beta}Q_{\theta})(s_h,a_h) contains the next-state term −Vθ(sh+1)-V_{\theta}(s_{h+1}) from the Bellman backup. At the following step, substituting

Qθ(sh+1,ah+1)=log⁡πθ(ah+1∣sh+1)+Vθ(sh+1)Q_\theta(s_{h+1},a_{h+1})=\log\pi_\theta(a_{h+1}\mid s_{h+1})+V_\theta(s_{h+1})

introduces the corresponding +Vθ(sh+1)+V_\theta(s_{h+1}). These terms cancel across consecutive steps, giving

∑h=1H[Qθ(sh,ah)−Vθ(sh+1)]=log⁡πθ(τ)+Vθ(s1)−Vθ(sH+1).\sum_{h=1}^{H}\left[Q_\theta(s_h,a_h)-V_\theta(s_{h+1})\right]=\log\pi_\theta(\tau)+V_\theta(s_1)-V_\theta(s_{H+1}).

With zero terminal values, only the initial-state value remains. The reference-value shaping terms telescope similarly, leaving the correction Vθ(s1)−Vref(s1)V_\theta(s_1)-V_{\mathrm{ref}}(s_1).

ReVal therefore optimizes a complete-response objective without a separate next-state bootstrapped target. We update the policy through this objective and differentiate all terms, including the initial-state value. This differs from traditional stepwise semi-gradient TD, so its associated deadly-triad instability mechanism cannot be directly assumed to apply to ReVal.

Question 4:All experiments use 1.5B and 7B models on math/reasoning benchmarks.

Thank you for this suggestion. To evaluate ReVal beyond mathematical reasoning, we conducted additional experiments on ALFWorld, an interactive agent benchmark, using Qwen2.5-7B-Instruct. ALFWorld requires agents to interact with an environment and execute sequences of actions to accomplish household tasks. [1]

The following table reports performance on the seen and unseen validation splits:

MethodSeenUnseen
GRPO0.380.46
TBRM0.350.38
ReVal0.480.54

ReVal achieves higher mean reward than both baselines on both splits at this checkpoint, providing preliminary evidence of its applicability to interactive agent tasks beyond mathematical reasoning.

[1] Shridhar et al. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. ICLR 2021.

Reviewer_zJq6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors investigate how off-policy data reuse can improve sample efficiency in LLM reinforcement learning when trajectory generation is expensive. They propose ReVal, a value-based method that uses LLM logits as Q-values. Specifically, ReVal (1) introduces reward shaping to obtain Calibrated Initialization and (2) uses a FIFO replay buffer for multiple updates between data collection rounds. Experiments report average avg@16 scores of 45.6 and 39.6 on DPSK-R1-Distill-1.5B and Qwen2.5-Math-7B, compared with GRPO's 43.4 and 38.4, respectively.

Critical strengths

The motivation for reusing expensive trajectories is well justified. ReVal uses a FIFO replay buffer to perform multiple updates between data collection rounds, while representing the policy and Q-values within the same model. This is a practical approach to improving sample efficiency without introducing a separate Q-value network.

The reported performance improvements extend across two models. Table 1 shows average gains over GRPO of 2.2 percentage points on DPSK-R1-Distill-1.5B and 1.2 on Qwen2.5-Math-7B, with GPQA gains of 4.5 and 4.4 points. These results provide useful empirical support for ReVal on mathematical reasoning and the out-of-domain benchmark.

Critical weaknesses

Proposition 1 does not establish its claim about TBRM's optimal policy. If Q_theta(s,a) = log pi_ref(a|s) is representable, it gives V_theta(s) = 0 and pi_theta = pi_ref, making the zero-reward TBRM loss zero. The argument does not rule out this solution, and the policy drift in Figure 9 does not establish a different optimum. ReVal's zero loss when initialized with the reference model remains useful, but supports a claim about initialization rather than the stated distinction between optimal policies.

The experiments do not isolate ReVal's improvement over TBRM under comparable data reuse. Yuan et al. (2025, §3) already formulate TBRM for off-policy data, but §3.3 here describes TBRM as updating newly collected trajectories once, while ReVal adds replay buffer updates. A TBRM comparison using the same FIFO replay buffer and number of updates, with other settings held constant, is needed to establish the benefit of ReVal's reward shaping beyond additional data reuse.

Questions for authors

Gradient dynamics: Eq. (5) omits the gradient of V_theta(x), which appears in Eq. (2). Is V_theta(x) held fixed during each update, or should Eq. (5) also include its gradient? Please clarify which gradient is used in the experiments.

Training efficiency: What performance threshold is used for each benchmark in Figure 4, and is it the same across methods? Please provide the timing breakdown for step=8, including all parameter updates, and clarify whether Figure 2's horizontal axis counts generation rounds or parameter updates.

Evaluation: Appendix A caps evaluation benchmarks at 100 samples. Does this also apply to Table 1, and are the same subsets used across methods? Please report the number of independent training runs and uncertainty for the average improvements.

3 comments

Weakness

Q1:Proposition 1 does not establish its claim about TBRM's optimal policy.

Thank you for pointing this out. Our original claim about TBRM’s optimal policy was incorrect, and we apologize for the confusion. When the task reward is zero, the reference policy is already optimal for KL-regularized RL, so further policy optimization is unnecessary. However, TBRM can have a nonzero loss and can still produce parameter updates; Our zero-reward experiment in Figure 9 illustrates the resulting policy drift in that run. ReVal instead ensures zero loss and zero loss gradient at this initialization, consistent with Calibrated Initialization. We have corrected Definition 1, Proposition 1, and the accompanying proof accordingly.

Q2:The experiments do not isolate ReVal's improvement over TBRM under comparable data reuse.

Thank you for suggesting this controlled comparison. We agree that TBRM already supports off-policy learning and that matching data reuse is necessary to assess the contribution of ReVal’s reward shaping. To clarify, the original TBRM [1, 2] training procedure uses an on-policy training schedule without replay-buffer updates. Our original TBRM baseline followed this on-policy training procedure. To isolate the contribution of reward shaping, we reran TBRM using the same FIFO replay-buffer configuration and update schedule as ReVal, with the other settings held constant. The results are summarized below.

Qwen2.5-Math-7B

MethodAIME24AIME25AMCMATHMINERVAOlympiadGPQAAvg
TBRM30.81058.374.427.337.819.936.9
TBRM + buffer28.7513.3359.4874.7529.8738.7516.8737.4
ReVal3413.360.275.230.739.224.839.6

DPSK-R1-Distill-1.5B

MethodAIME24AIME25AMCMATHMINERVAOlympiadGPQAAvg
TBRM26.922.165.181.328.844.627.342.3
TBRM + buffer27.523.9566.1183.1230.6844.9332.5644.12
ReVal32.123.868.684.630.346.633.345.6

Adding replay improves TBRM’s average performance on both models. ReVal nevertheless retains higher average scores, with improvements of approximately 2.2 percentage points on Qwen2.5-Math-7B and 1.5 percentage points on DPSK-R1-Distill-1.5B over TBRM with replay. These results suggest that additional data reuse accounts for part of the improvement, while ReVal’s reward shaping provides a further benefit.

We also tested GRPO with replay-buffer updates. In this run, training collapsed and the model’s evaluated performance dropped to zero.

[1] Yuan et al. (2025). Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning, Section 3, Algorithm 1.

[2] TBRM official implementation: verl/tbrm/ray_trainer.py, lines 90–165.

Questions For Authors:

Q1:Gradient dynamics: Eq. (5) omits the gradient of V_theta(x), which appears in Eq. (2). Is V_theta(x) held fixed during each update, or should Eq. (5) also include its gradient? Please clarify which gradient is used in the experiments.

Thank you for pointing out the missing term. In our experiments, Vθ(x)V_\theta(x) is computed from the current model’s logits and fully differentiated; it is neither held fixed nor detached during an update. The correct gradient is ∇θLReVal=−2E(x,y)∼D[δ(x,y)(∇θVθ(x)+∇θlog⁡πθ(y∣x))].\nabla_\theta\mathcal L_{\mathrm{ReVal}} =-2\mathbb E_{(x,y)\sim\mathcal D} \left[\delta(x,y)\left( \nabla_\theta V_\theta(x) +\nabla_\theta\log\pi_\theta(y\mid x) \right)\right]. We have corrected Eq. (5) and the corresponding gradient in Eq. (7), and explicitly clarified in Appendix G that all experiments use the full gradient of Eq. (2). This correction does not change the experimental results.

Q2:Training efficiency: What performance threshold is used for each benchmark in Figure 4, and is it the same across methods? Please provide the timing breakdown for step=8, including all parameter updates, and clarify whether Figure 2's horizontal axis counts generation rounds or parameter updates.

Thank you for asking us to clarify the evaluation criterion. Figure 4 uses a single shared threshold based on the mean accuracy across the seven benchmarks, rather than separate benchmark-specific thresholds. Specifically, we use GRPO’s highest unsmoothed average score during training, achieved at generation round 580 (approximately 41.4%), as the common target for both methods. For each method, we identify the first evaluation checkpoint at which its average score meets or exceeds this target. ReVal reaches it at round 470, with an average score of 41.7%, requiring approximately 19% fewer generation rounds than GRPO. The two panels report the generation rounds and corresponding cumulative wall-clock training time to reach this same target. The reported times are approximately 6.2 h for ReVal and 7.5 h for GRPO. We clarify this criterion in Section 4.4 and the caption of Figure 4.

The horizontal axis in Figure 2 counts generation rounds, not parameter updates. Under step=8, ReVal performs eight parameter updates per generation round. The improvement shown in Figure 2 therefore concerns generation efficiency, rather than a reduction in the number of parameter updates.

For step=8, the timing breakdown is provided below. The total time includes all eight parameter updates.

ComponentTime per generation round (s)
Trajectory generation21.36
Reference-model computation2.8
All eight parameter updates and remaining overhead98.12
Total122.29

Q3:Evaluation: Appendix A caps evaluation benchmarks at 100 samples. Does this also apply to Table 1, and are the same subsets used across methods?

Thank you for requesting these clarifications. The evaluation cap also applies to Table 1. All methods use exactly the same fixed evaluation subset. For smaller benchmarks, such as AIME24, AIME25, and AMC, we report avg@16 to reduce evaluation variability. Other benchmarks, such as MATH, contain substantially more questions, making repeated evaluation with 16 responses very time-consuming. To accelerate evaluation, we randomly select a fixed subset of 100 questions from each larger benchmark following Knapsack RL [1]. Our preliminary comparisons using 100 versus full questions showed only small differences in the resulting scores, supporting this choice as a practical balance between evaluation cost and reliability.

Regarding independent training runs, LLM RL experiments can exhibit limited variation across training seeds, and single-seed training is used in practice [2,3]. To address the reviewer’s concern, we conducted an additional independent ReVal run on DPSK-R1-Distill-1.5B, bringing the total to two runs for this configuration.

MethodAIME24AIME25AMCMATHMINERVAOlympiadGPQAAvg
GRPO29.424.46582.627.646.328.843.44
ReVal32.123.868.684.630.346.633.345.61
ReVal (Additional Run)31.4522.0869.1285.3729.064734.3145.48

The additional results yield an average of 45.48%, compared with 45.61% for the original ReVal results and 43.44% for GRPO. The two ReVal results differ by only 0.13 percentage points, suggesting consistent average performance across the two runs.

[1] Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocation. arXiv:2509.25849, 2025.

[2] ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models. ICML, 2024.

[3] Generalizable End-to-End Tool-Use RL with Synthetic CodeGym. ICLR, 2026.

Dear Reviewer zJq6,

We sincerely appreciate your careful review and suggestions for strengthening the paper. Your feedback helped us clarify the initialization-calibration claim, correct the gradient expressions, and add TBRM comparisons with matched replay-buffer configurations and update schedules. In these comparisons, ReVal retains average gains of approximately 2.2 and 1.5 percentage points on the two models. We also clarified the evaluation and timing protocols and reported an additional independent training run.

Please let us know if you have any remaining questions. We would be glad to discuss them before the discussion closes. If these corrections and additional results address your main concerns, we sincerely hope you will consider a higher overall score.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.