acceptodds
Under review as a conference paper at ICLR 2027

Better Starts for Value-Based LLM Reasoning: Reference-Return Initialized TBRM

Abstract

Trajectory Bellman Residual Minimization (TBRM) enables value-based reinforcement learning for large language model reasoning by interpreting token logits as action values. However, initializing the model from a reference policy does not automatically calibrate its value estimate: action-independent shifts leave token probabilities unchanged but alter the log-partition used to represent value. This ambiguity can create a mismatch between the initial value and downstream task returns. We give a unified value-anchoring framework that organizes TBRM and the trajectory residual used by ReVal as different choices of a fixed prompt-wise initial value. Within this framework, we propose Reference-Return Initialized TBRM (RI-TBRM), which estimates the reference policy's mean return in an offline annotation pass and reuses the resulting value anchor throughout reinforcement learning. The method applies a fixed offset consistently to action and state values, preserving the initial policy and the trajectory Bellman residual functional without introducing a separate critic. Under reference-policy sampling, exact calibration minimizes the initial residual mean squared error among prompt-wise scalar value initializations and removes the value-mismatch term from the initial expected gradient. We further characterize the estimation error introduced by a finite number of independent calibration rollouts. RI-TBRM achieves the highest six-benchmark average on all three evaluated model backbones.

Primary area:
reinforcement learning
Keywords:
large language models, reinforcement learning, mathematical reasoning, value-based reinforcement learning, Bellman residual minimization, value initialization, reference policy, reward calibration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eAACOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the value initialization in TBRM for LLM reasoning. The authors propose a unified value-anchoring framework that studies the effect of prompt-wise scalar anchor on the initial Bellman-residual gradient. The proposed RI-TBRM uses anchor values at cached reference-policy returns and achieves competitive results on maths reasoning benchmarks across qwen models.

Critical strengths
  • The paper provides clean and useful conceptual unification with proper preliminary introduction that benefits broader readers.
  • The theoretical analysis on the initialization criterion is good.
  • The paper reports training dynamics with robustness analysis.
Critical weaknesses
  • One major concern lies in the experimentation results where the gains are quite small and limited in maths benchmarks. The lack of statistic support makes it difficult to believe whether the gains are robust and consistent.
  • The computational complexity is not discussed in the paper and the main comparison table might not be that intuitive in compute-matching.
  • The ablation is only limited in the anchor itself but not investigates more anchor value choices (e.g., v=\lambda b_K or task-tunable anchors).
Questions for authors
  • Can the authors provide more results such as statistical analysis (different random seeds) and evaluation on more benchmarks?
  • Please provide a comparison on the performance with compute control or discuss the complexity of these three methods.
  • The ablation on anchor value v=\lambda \hat b_K across different \lambda is encouraged.
4 comments
AuthorsOpenReview ↗

[1/2] Discussion Stage I

We sincerely appreciate the reviewer's positive assessment of our conceptual unification, the clarity of our theoretical analysis, and the usefulness of our framework. We thank you for the valuable suggestions and provide additional experiments and detailed explanations in response to the raised concerns.

W1 & Q1: Insufficient evidence of performance consistency

We evaluated the original seed 42 together with two additional seeds, 3407 and 1234, on Qwen2.5-Math-7B. The complete results across six mathematical reasoning benchmarks are reported below.

SeedMethodAIME24AIME25AMC23MATH500MinervaMathOlympiadBenchAvg
42TBRM19.797.2952.6659.0016.9128.5930.70
42ReVal19.588.2352.8157.6017.2827.5630.51
42RI-TBRM20.837.9253.5259.6018.3828.5931.47
3407TBRM20.217.1952.2758.0017.6525.9330.21
3407ReVal18.967.1952.5859.2018.3826.6730.50
3407RI-TBRM19.178.1254.5359.8016.9127.8531.06
1234TBRM19.277.6052.1958.4016.9127.5630.32
1234ReVal20.007.5051.2557.6016.1827.2629.96
1234RI-TBRM21.677.8151.9559.2018.3828.3031.22

RI-TBRM consistently achieves the highest six-benchmark average across all three seeds, outperforming both TBRM and ReVal in every run. Averaged across the three seeds, RI-TBRM achieves 31.25 ± 0.21, compared with 30.41 ± 0.26 for TBRM and 30.32 ± 0.31 for ReVal. These results provide additional evidence that the improvements of RI-TBRM are consistent across independent training runs rather than being specific to a single random seed.

W2 & Q2: Computational cost and training efficiency

We thank the reviewer for raising this important concern. To provide a more transparent comparison of computational costs, we measured the wall-clock training time of TBRM, ReVal, and RI-TBRM over 800 training steps on Qwen2.5-Math-7B using two independent seeds.

MethodSeed 3407Seed 1234Average
TBRM10:48:4310:58:5210:53:48
ReVal10:50:0210:57:2310:53:43
RI-TBRM10:46:4210:54:1110:50:27

All times are reported in hours:minutes:seconds and exclude the one-time reference-rollout preparation.

Online training efficiency. RI-TBRM exhibits comparable training costs to both baselines. Across the two seeds, its average training time is approximately 3.4 minutes shorter than TBRM and 3.3 minutes shorter than ReVal. These results indicate that reference-return anchoring introduces negligible additional computational overhead during policy optimization, as it only requires reading cached prompt-wise statistics without introducing an additional critic or reference rollout at each training step.

One-time reference rollout cost. RI-TBRM additionally requires approximately 27 minutes to estimate the reference-policy expected returns before training. Importantly, this is a one-time preprocessing cost rather than a recurring cost during optimization. Since the reference policy, training prompts, and reward function remain fixed, the resulting cached statistics can be reused across independent training runs and random seeds sharing the same reference configuration.

Consequently, when the cache is reused across NN training runs, the amortized preprocessing cost per run is

Tpreprocessamortized=27N minutes.T_{\mathrm{preprocess}}^{\mathrm{amortized}}=\frac{27}{N}\ \mathrm{minutes}.

For example, with three independent training seeds, the amortized preprocessing cost is only 9 minutes per run.

Overall, these measurements show that RI-TBRM achieves consistently higher six-benchmark average accuracy across three training seeds while maintaining comparable online training time. Although the initial preprocessing introduces an additional cost for the first run, this cost is reusable and becomes increasingly amortized across repeated experiments, making RI-TBRM computationally practical without requiring a separate critic.

AuthorsOpenReview ↗

[2/2] Discussion Stage II

W3 & Q3: Limited exploration of anchor values

Following your suggestion, we evaluate v=λb^Kv=\lambda\widehat b_K with λ∈{0.5,1,1.5}\lambda\in\{0.5,1,1.5\} on Qwen2.5-Math-7B:

Anchor scale λ\lambdaAIME24AIME25AMC23MATH500MinervaMathOlympiadBenchSix-benchmark AvgDifference from $$\lambda=1=1
0.518.657.2953.3659.2017.6527.5630.62−0.44 pp
1.019.178.1254.5359.8016.9127.8531.06—
1.517.197.9246.9559.4016.9128.8929.54−1.52 pp

The default unscaled reference-return anchor has the highest reported average among the three tested choices.

Dear Reviewer eAAC,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end in about 2 days, we gently request that you read the rebuttal and check whether your concerns are addressed.

Thanks again for your time and help in improving the quality of this paper.

Regards from the authors.

Thank you for the added experiments and explanations. My concerns on the limited gains and generalization beyond maths benchmarks still remain and therefore I have to maintain my score. The concerns on the compute and hyper-parameter ablation have been resolved.

Reviewer_ZvbjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Reference-Return Initialized TBRM (RI-TBRM) to improve value-based reinforcement learning for LLM reasoning. It identifies an initialization mismatch: token logits can define the same generation policy while implying different value estimates. RI-TBRM addresses this by anchoring each prompt’s initial value to the reference policy’s estimated expected return, using offline samples cached throughout training without an additional critic. The authors unify TBRM and ReVal’s trajectory residual as alternative anchor choices and prove that exact reference-return calibration minimizes initial residual mean squared error and removes a value-mismatch term from the initial expected gradient. Across six mathematical reasoning benchmarks and three Qwen backbones, RI-TBRM achieves the highest average scores among evaluated methods. However, the improvements are modest, and broader evaluation and multiple-seed experiments are needed to establish robustness.

Critical strengths
  1. The paper identifies the ambiguity between policy probabilities and logit-derived values, and unifies TBRM and ReVal’s trajectory residual through an explicit value-anchor framework.
  2. Reference-return anchoring minimizes initial residual MSE under stated assumptions and removes the initial gradient’s value-mismatch term, requiring only cached statistics and no additional critic.
  3. RI-TBRM achieves the highest six-benchmark average across all three evaluated backbones, supporting its usefulness across different model sizes and initial capabilities.
Critical weaknesses
  1. The analysis characterizes the exact reference-initialization point; it does not establish whether a frozen anchor remains beneficial as the policy changes during training.
  2. Improvements over ReVal range from 0.38 to 1.05 percentage points, but results use a single reported seed, making robustness and statistical significance difficult to assess.
Questions for authors
  1. The theoretical guarantees apply only at reference-policy initialization. As the policy evolves during training, does the frozen reference-return anchor remain beneficial, and what evidence supports this?
  2. Given the modest improvements over ReVal, are the gains consistent across multiple random seeds and statistically significant?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Zvbj

We are grateful to the reviewer for acknowledging our identification of the policy-value ambiguity, the theoretical soundness of reference-return anchoring, and the practical simplicity of RI-TBRM. We sincerely appreciate your insightful comments and address each concern with further analysis and experimental results below.

W1 & Q1: Whether the frozen anchor remains beneficial as the policy changes

The anchor and reference quantities remain fixed, while the model's log-partition correction remains trainable. The value estimate therefore continues to evolve with the policy. The new seed-3407 diagnostic on Qwen2.5-Math-7B shows the following raw temporal means:

Training intervalTBRMReValRI-TBRM
Steps 1–100−0.1629−0.1930−0.0424
Steps 101–400−0.1494−0.1725−0.0445
Steps 401–800−0.1321−0.1539−0.0436

RI-TBRM's signed diagnostic is closer to zero in all three intervals, including the later portion of training. This provides evidence of a persistent separation in the evaluated run.

W2 & Q2: Modest gains and single-seed evaluation

We evaluated the original seed 42 together with two additional seeds, 3407 and 1234, on Qwen2.5-Math-7B. The complete results across six mathematical reasoning benchmarks are reported below.

SeedMethodAIME24AIME25AMC23MATH500MinervaMathOlympiadBenchAvg
42TBRM19.797.2952.6659.0016.9128.5930.70
42ReVal19.588.2352.8157.6017.2827.5630.51
42RI-TBRM20.837.9253.5259.6018.3828.5931.47
3407TBRM20.217.1952.2758.0017.6525.9330.21
3407ReVal18.967.1952.5859.2018.3826.6730.50
3407RI-TBRM19.178.1254.5359.8016.9127.8531.06
1234TBRM19.277.6052.1958.4016.9127.5630.32
1234ReVal20.007.5051.2557.6016.1827.2629.96
1234RI-TBRM21.677.8151.9559.2018.3828.3031.22

RI-TBRM consistently achieves the highest six-benchmark average across all three seeds, outperforming both TBRM and ReVal in every run. Averaged across the three seeds, RI-TBRM achieves 31.25 ± 0.21, compared with 30.41 ± 0.26 for TBRM and 30.32 ± 0.31 for ReVal. These results provide additional evidence that the improvements of RI-TBRM are consistent across independent training runs rather than being specific to a single random seed.

Dear Reviewer Zvbj,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end in about 2 days, we gently request that you read the rebuttal and check whether your concerns are addressed.

Thanks again for your time and help in improving the quality of this paper.

Regards from the authors.

Reviewer_RtdVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies value initialization in TBRM for LLM reasoning. The key observation is that shifting all token logits by the same amount preserves the policy but changes the log-partition used as the value, so the same initial policy can lead to different Bellman residuals and learning updates. The authors introduce a unified value-anchoring framework that interprets TBRM and ReVal as different choices of the initial value anchor. The authors then propose RI-TBRM, which anchors each prompt using the estimated expected return of the reference policy. The paper shows that this choice minimizes the initial residual MSE and removes a value-mismatch term from the initial expected gradient. Experiments on three model backbones and multiple reasoning benchmarks show improvements over TBRM and ReVal, with RI-TBRM achieving the best average performance on all three backbones.

Critical strengths
  1. The paper identifies a meaningful initialization issue in TBRM and provides a clean framework that unifies TBRM, ReVal, and the proposed method through different value anchors. I find this perspective simple and useful.

  2. The proposed reference-return anchor is well motivated theoretically, rather than being just an empirical trick. The empirical results are also reasonably consistent, where RI-TBRM achieves the best average performance across all three evaluated backbones.

Critical weaknesses
  1. The empirical gains are consistent but relatively modest, and it is unclear how robust they are to training randomness. The paper does not appear to report results across multiple independent RL training seeds, which makes differences below one point on two of the three backbones somewhat difficult to assess.

  2. The theory mainly justifies the proposed anchor in terms of the initial residual and expected gradient. The connection between better initialization under these criteria and better final policy performance is supported empirically, but not theoretically characterized.

Questions for authors
  1. Have you evaluated TBRM, ReVal, and RI-TBRM across multiple independent training seeds? How large is the run-to-run variance relative to the reported improvements?

  2. Do you observe a relationship between the initial value mismatch and the eventual performance improvement from RI-TBRM? This could help clarify when the proposed initialization is most useful in practice.

4 comments
AuthorsOpenReview ↗

[1/2] Discussion Stage I

We sincerely thank the reviewer for recognizing the significance of the value initialization problem, the novelty of our unified framework, and the theoretical motivation of RI-TBRM. We greatly appreciate your constructive feedback and provide additional experimental evidence and theoretical analysis to address your concerns below.

W1 & Q1: Modest gains and robustness to training randomness

We evaluated the original seed 42 together with two additional seeds, 3407 and 1234, on Qwen2.5-Math-7B. The complete results across six mathematical reasoning benchmarks are reported below.

SeedMethodAIME24AIME25AMC23MATH500MinervaMathOlympiadBenchAvg
42TBRM19.797.2952.6659.0016.9128.5930.70
42ReVal19.588.2352.8157.6017.2827.5630.51
42RI-TBRM20.837.9253.5259.6018.3828.5931.47
3407TBRM20.217.1952.2758.0017.6525.9330.21
3407ReVal18.967.1952.5859.2018.3826.6730.50
3407RI-TBRM19.178.1254.5359.8016.9127.8531.06
1234TBRM19.277.6052.1958.4016.9127.5630.32
1234ReVal20.007.5051.2557.6016.1827.2629.96
1234RI-TBRM21.677.8151.9559.2018.3828.3031.22

RI-TBRM consistently achieves the highest six-benchmark average across all three seeds, outperforming both TBRM and ReVal in every run. Averaged across the three seeds, RI-TBRM achieves 31.25 ± 0.21, compared with 30.41 ± 0.26 for TBRM and 30.32 ± 0.31 for ReVal. These results provide additional evidence that the improvements of RI-TBRM are consistent across independent training runs rather than being specific to a single random seed.

AuthorsOpenReview ↗

[2/2] Discussion Stage II

W2 & Q2: Connection Between Value Initialization and Final Policy Performance

We thank the reviewer for this insightful comment. While our initial analysis focuses on reference-policy initialization, we can further characterize how value estimation error affects policy optimization throughout training.

Theoretical analysis. At any training step tt, define the value estimation error as

et(x)=V^θt,v(x)−Vβπθt(x),e_t(x)=\hat V_{\theta_t,v}(x)-V_\beta^{\pi_{\theta_t}}(x),

where V^θt,v(x)\hat V_{\theta_t,v}(x) is the model's estimated value and Vβπθt(x)V_\beta^{\pi_{\theta_t}}(x) is the expected KL-regularized return under the current policy.

Under fresh on-policy sampling, with the sampling distribution held fixed during differentiation, a fixed reference and anchor, and full residual differentiation, the expected residual gradient satisfies

∇θLt(θt)=−β−1∇θJβ(θt)+Ex[et(x)∇θVt(x)]\nabla_{\theta} L_t(\theta_t)=-\beta^{-1}\nabla_{\theta}J_{\beta}(\theta_t)+E_x[e_t(x)\nabla_{\theta}V_t(x)]

The second term characterizes the deviation from the KL-regularized policy gradient induced by value estimation error. By the Cauchy–Schwarz inequality,

∥Bt∥≤Gtϵt,\|B_t\|\leq G_t\epsilon_t,

where

e_t(x)\nabla_\theta\hat V_{\theta_t,v}(x) \right],$$ $$\epsilon_t^2=\mathbb E_x[e_t(x)^2], \qquad G_t^2=\mathbb E_x\left[ \|\nabla_\theta\hat V_{\theta_t,v}(x)\|^2 \right].$$ **Importantly, this relationship holds at arbitrary training steps, rather than only at initialization.** It establishes that reducing the current value estimation error tightens an upper bound on the gradient deviation from KL-regularized policy optimization, provided the value-gradient second moment is bounded. **Empirical evidence.** To examine whether the benefit persists throughout training, we analyze the value-error dynamics on Qwen2.5-Math-7B (seed 3407). We compute the mean squared error directly from the unsmoothed training records: | Training interval | TBRM | ReVal | RI-TBRM | |---|---:|---:|---:| | Steps 1–100 | 0.027587 | 0.038204 | **0.002629** | | Steps 101–400 | 0.023314 | 0.030748 | **0.002431** | | Steps 401–800 | 0.018528 | 0.024707 | **0.002383** | | **Overall (1–800)** | 0.021455 | 0.028659 | **0.002431** | RI-TBRM consistently achieves the lowest mean squared error across all training intervals, including the later stages. Over the complete training trajectory, its recorded squared error is **88.7% lower than TBRM** and **91.5% lower than ReVal**. In the same seed, RI-TBRM also achieves the highest six-benchmark average (**31.06**), compared with **30.21** for TBRM and **30.50** for ReVal. Moreover, RI-TBRM achieves the highest six-benchmark average across **all three independent training seeds**, further supporting the consistency of the observed performance improvements. These results provide complementary theoretical and empirical evidence: value estimation error can affect the policy optimization gradient throughout training, while RI-TBRM exhibits substantially smaller squared errors together with consistently better downstream performance. This supports the practical benefit of reference-return initialization beyond the initial optimization steps.

Dear Reviewer RtdV,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end in about 2 days, we gently request that you read the rebuttal and check whether your concerns are addressed.

Thanks again for your time and help in improving the quality of this paper.

Regards from the authors.

I thank the authors for their clarification. My questions are mostly answered, but my initial concern of weak improvements still stands. I therefore remain my assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.