- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ReVal, a value-based, off-policy RL method for LLM post-training. Building on TBRM, which treats LLM logits as soft Q-values, ReVal adds a reward-shaping term with the reference value V_ref as the potential, so that the loss vanishes at the reference policy when the reward is zero (“Calibrated Initialization”), and trains from a FIFO replay buffer. Trained on DeepScaleR, ReVal achieves higher average accuracy than GRPO and TBRM on DeepSeek-R1-Distill-1.5B (45.6 vs. 43.4 for GRPO) and Qwen2.5-Math-7B.
Critical strengths
-
A real initialization issue with a simple fix. With unnormalized logits, TBRM is not stationary at the reference model even under zero reward (Fig. 9). Subtracting V_ref(s₁) makes the reference policy a zero-loss point at no extra cost, which is useful for any logits-as-Q method.
-
Consistent empirical signal. ReVal stays above GRPO for most of training on DeepSeek-R1-Distill-1.5B (Fig. 10a) and clearly beats TBRM on both models (Table 1).
-
Informative ablations and failure cases. Reference resets avoid saturation (Fig. 5), β must be set according to response length (Sec. 4.5.2), and normalized rewards beat 0/1 and ±1 rewards (Fig. 7); collapse at β = 0.002 and the unstable regression variant are also reported (Figs. 6, 11).
Critical weaknesses
-
The theory conflates optimality with stationarity, and the contribution over TBRM is narrower than claimed. Def. 1 and Prop. 1 are stated in terms of the optimal policy, but the paper notes that the shaping term does not change the optimal solution (L205–208); since ReVal is TBRM plus this term, the two losses share the same optimum. What the argument shows is that the pretrained model is not a stationary point of TBRM, which is the property ReVal fixes. Off-policy training is also not new: TBRM already claims convergence from arbitrary off-policy data, and Eq. 2 is the trajectory balance loss with R = π_ref·e^(r/β) that TBA[1] trains from a replay buffer. What remains new is using V_θ(s₁) − V_ref(s₁) as log Z, and its benefit needs to be shown.
-
The experiments do not isolate the source of the gains. ReVal adds both calibration and replay (plus reward normalization and reference resets on Qwen), yet Table 1 compares it only with on-policy GRPO and TBRM, and the paper does not report the baselines' update counts. Without TBRM with the same buffer, ReVal without it, and GRPO with matched updates or the same buffer, the gains cannot be attributed to the value-based objective; on Qwen, ReVal without resets (Fig. 5) or with the 0/1 reward of Eq. 2 (Fig. 7) ends below GRPO's 38.4.
-
The evaluation cannot support gains of 2–4 points. No seeds or confidence intervals are reported, benchmarks are subsampled to at most 100 problems (App. A), and hyperparameters such as β are tuned with ablations on the same test benchmarks (Sec. 4.5). GPQA is four-way multiple choice, yet the DPSK base model (15.8) and all four Qwen results (12.8–24.8) fall below the 25% chance level, so the GPQA gains may partly reflect answer formatting.
[1] Bartoldson, B., Venkatraman, S., Diffenderfer, J., Jain, M., Ben-Nun, T., Lee, S., ... & Kailkhura, B. (2026). Trajectory balance with asynchrony: Decoupling exploration and learning for fast, scalable llm post-training. Advances in Neural Information Processing Systems, 38, 113901-113931.
Questions for authors
-
Regarding W1: do you agree that, with a fully expressive parameterization, TBRM's minimizer at r = 0 is π_ref, so that Prop. 1 is really about non-stationarity at initialization rather than about the optimum? If not, what property of the actual parameterization prevents it?
-
Regarding W2: Sec. 3.3, App. A, and Alg. 1 describe the update schedule differently. Which one was used, and how many gradient updates per rollout does each method perform? Do the DPSK runs in Table 1 also use reward normalization and reference resets? If GRPO gets at least as many updates and the gap persists, this concern would be largely resolved.
-
Regarding W3: why are the GPQA scores below chance? Please report the answer-extraction failure rate, and state which checkpoint Table 1 reports and how many seeds were run.
7 comments
Weakness 3: The evaluation cannot support gains of 2–4 points. No seeds or confidence intervals are reported, benchmarks are subsampled to at most 100 problems (App. A), and hyperparameters such as β are tuned with ablations on the same test benchmarks (Sec. 4.5).
Thank you for requesting these clarifications. The evaluation cap also applies to Table 1. All methods use exactly the same fixed evaluation subset. For smaller benchmarks, such as AIME24, AIME25, and AMC, we report avg@16 to reduce evaluation variability. Other benchmarks, such as MATH, contain substantially more questions, making repeated evaluation with 16 responses per question very time-consuming. To accelerate evaluation, we randomly select a fixed subset of 100 questions from each larger benchmark following Knapsack RL [1]. Our preliminary comparisons using 100-question subsets versus the full benchmarks showed only small differences in the resulting scores, supporting this choice as a practical balance between evaluation cost and reliability.
Regarding independent training runs, LLM RL experiments can exhibit limited variation across training seeds, and single-seed training is used in practice [2, 3]. Our original experiments used one training run per configuration. To address the reviewer’s concern, we conducted an additional independent ReVal run on DPSK-R1-Distill-1.5B, bringing the total to two runs for this configuration.
| Method | AIME24 | AIME25 | AMC | MATH | MINERVA | Olympiad | GPQA | Avg |
|---|---|---|---|---|---|---|---|---|
| GRPO | 29.40 | 24.40 | 65.00 | 82.60 | 27.60 | 46.30 | 28.80 | 43.44 |
| ReVal | 32.10 | 23.80 | 68.60 | 84.60 | 30.30 | 46.60 | 33.30 | 45.61 |
| ReVal (Additional Run) | 31.45 | 22.08 | 69.12 | 85.37 | 29.06 | 47.00 | 34.31 | 45.48 |
The additional results yield an average of 45.48%, compared with 45.61% for the original ReVal results and 43.44% for GRPO. The two ReVal results differ by only 0.13 percentage points, suggesting consistent average performance across the two runs.
[1] Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocation. arXiv:2509.25849, 2025.
[2] ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models. ICML, 2024.
[3] Generalizable End-to-End Tool-Use RL with Synthetic CodeGym. ICLR, 2026.
Question 2:Regarding W2: Sec. 3.3, App. A, and Alg. 1 describe the update schedule differently. Which one was used, and how many gradient updates per rollout does each method perform? Do the DPSK runs in Table 1 also use reward normalization and reference resets? If GRPO gets at least as many updates and the gap persists, this concern would be largely resolved.
Thank you for pointing out these inconsistencies. The experiments followed the schedule described in Appendix A. After each generation round, ReVal performs one update on the newly collected batch, followed by one update on a batch sampled from the FIFO replay buffer. GRPO and the original TBRM baseline each perform one update on the newly collected batch. Thus, ReVal performs two gradient updates per generation round, while GRPO and TBRM each perform one.
Section 3.3 incorrectly describes both ReVal updates as replay-buffer updates, while Algorithm 1 omits the update on newly collected data. We apologize for these reporting errors and will align both descriptions with the actual implementation.
The DPSK-R1-Distill-1.5B ReVal experiments in Table 1 also use reward centering and periodic reference resets, following the same procedure as Qwen2.5-Math-7B.
To assess the contribution beyond additional data reuse, we additionally evaluated TBRM with the same FIFO replay buffer and update schedule as ReVal. This comparison addresses the concern for TBRM; a GRPO comparison with a matched update budget remains necessary to fully address the corresponding concern.
Question 3:Regarding W3: why are the GPQA scores below chance? Please report the answer-extraction failure rate, and state which checkpoint Table 1 reports and how many seeds were run.
We found that the below-chance GPQA scores are largely explained by answer-extraction failures. For example, approximately 69% of responses from DeepSeek-R1-Distill-1.5B have no extractable answer, with almost all such failures arising from responses truncated at the generation limit. Among responses with an extractable answer, accuracy is 49.8%, well above chance. However, this subset differs across methods and checkpoints, making accuracy conditioned on successful extraction unsuitable for direct comparison. We therefore report the overall score, counting responses without an extractable answer as incorrect, so that every method is evaluated on the same full question set.
Thank you for the detailed response. Could you explain on why training collapsed when replay-buffer updates were added?
Thank you for pointing this out. Our original claim about TBRM’s optimal policy was incorrect, and we apologize for the confusion. When the task reward is zero, the reference policy is already optimal for KL-regularized RL, so further policy optimization is unnecessary. However, TBRM can have a nonzero loss and can still produce parameter updates; Our zero-reward experiment in Figure 9 illustrates the resulting policy drift in that run. ReVal instead ensures zero loss and zero loss gradient at this initialization, consistent with Calibrated Initialization. We have corrected Definition 1, Proposition 1, and the accompanying proof accordingly.
Although TBRM supports off-policy learning, effectively incorporating a replay buffer into LLM training is not straightforward. We systematically investigate replay frequency, reference-policy updates, reward scaling, and the treatment of incorrect responses. Directly adding replay to TBRM yields comparatively modest improvements: its average score increases by 0.47 percentage points on Qwen2.5-Math-7B and 1.82 percentage points on DPSK-R1-Distill-1.5B. Even with the same replay-buffer configuration and update schedule, TBRM remains 2.23 and 1.49 percentage points below ReVal, respectively, showing that adding replay alone does not reproduce ReVal’s gains.
[1] Yuan et al. (2025). Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning.
[2] Bartoldson et al. (2025). Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training.
Thank you for suggesting this controlled comparison. Although TBRM supports off-policy learning, effectively incorporating a replay buffer into LLM training is not straightforward. The original TBRM training procedure uses an on-policy schedule without replay-buffer updates [1, 2], which our original TBRM baseline followed.
Our work develops an initialization-calibrated objective and systematically investigates replay frequency, reference-policy updates, reward scaling, and the treatment of incorrect responses in Sections 4.2 and 4.5. These experiments guide a practical training recipe combining fresh-data and replay updates, periodic reference resets, model-appropriate reward scaling, and mean-centered rewards. To isolate the contribution of reward shaping within this recipe, we reran TBRM using the same FIFO replay-buffer configuration and update schedule as ReVal, with the other settings held constant. The results are summarized below.
Qwen2.5-Math-7B
DPSK-R1-Distill-1.5B
Directly adding replay to TBRM yields comparatively modest gains, improving its average score by approximately 0.5 percentage points on Qwen2.5-Math-7B and 1.8 percentage points on DPSK-R1-Distill-1.5B. Under matched replay settings, ReVal still outperforms TBRM with replay by approximately 2.2 and 1.5 percentage points, respectively. These controlled comparisons support the benefit of ReVal’s reward shaping beyond additional data reuse.
We also tested GRPO with replay-buffer updates. In this run, training collapsed and the model’s evaluated performance dropped to zero, further illustrating that effective off-policy training cannot be assumed to follow from simply adding replay updates.
[1] Yuan et al. (2025). Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning, Section 3, Algorithm 1.
[2] TBRM official implementation:
verl/tbrm/ray_trainer.py, lines 90–165.