- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the value initialization in TBRM for LLM reasoning. The authors propose a unified value-anchoring framework that studies the effect of prompt-wise scalar anchor on the initial Bellman-residual gradient. The proposed RI-TBRM uses anchor values at cached reference-policy returns and achieves competitive results on maths reasoning benchmarks across qwen models.
Critical strengths
- The paper provides clean and useful conceptual unification with proper preliminary introduction that benefits broader readers.
- The theoretical analysis on the initialization criterion is good.
- The paper reports training dynamics with robustness analysis.
Critical weaknesses
- One major concern lies in the experimentation results where the gains are quite small and limited in maths benchmarks. The lack of statistic support makes it difficult to believe whether the gains are robust and consistent.
- The computational complexity is not discussed in the paper and the main comparison table might not be that intuitive in compute-matching.
- The ablation is only limited in the anchor itself but not investigates more anchor value choices (e.g., v=\lambda b_K or task-tunable anchors).
Questions for authors
- Can the authors provide more results such as statistical analysis (different random seeds) and evaluation on more benchmarks?
- Please provide a comparison on the performance with compute control or discuss the complexity of these three methods.
- The ablation on anchor value v=\lambda \hat b_K across different \lambda is encouraged.
4 comments
[2/2] Discussion Stage II
W3 & Q3: Limited exploration of anchor values
Following your suggestion, we evaluate with on Qwen2.5-Math-7B:
| Anchor scale | AIME24 | AIME25 | AMC23 | MATH500 | MinervaMath | OlympiadBench | Six-benchmark Avg | Difference from $$\lambda |
|---|---|---|---|---|---|---|---|---|
| 0.5 | 18.65 | 7.29 | 53.36 | 59.20 | 17.65 | 27.56 | 30.62 | −0.44 pp |
| 1.0 | 19.17 | 8.12 | 54.53 | 59.80 | 16.91 | 27.85 | 31.06 | — |
| 1.5 | 17.19 | 7.92 | 46.95 | 59.40 | 16.91 | 28.89 | 29.54 | −1.52 pp |
The default unscaled reference-return anchor has the highest reported average among the three tested choices.
Dear Reviewer eAAC,
Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end in about 2 days, we gently request that you read the rebuttal and check whether your concerns are addressed.
Thanks again for your time and help in improving the quality of this paper.
Regards from the authors.
Thank you for the added experiments and explanations. My concerns on the limited gains and generalization beyond maths benchmarks still remain and therefore I have to maintain my score. The concerns on the compute and hyper-parameter ablation have been resolved.
[1/2] Discussion Stage I
We sincerely appreciate the reviewer's positive assessment of our conceptual unification, the clarity of our theoretical analysis, and the usefulness of our framework. We thank you for the valuable suggestions and provide additional experiments and detailed explanations in response to the raised concerns.
We evaluated the original seed 42 together with two additional seeds, 3407 and 1234, on Qwen2.5-Math-7B. The complete results across six mathematical reasoning benchmarks are reported below.
RI-TBRM consistently achieves the highest six-benchmark average across all three seeds, outperforming both TBRM and ReVal in every run. Averaged across the three seeds, RI-TBRM achieves 31.25 ± 0.21, compared with 30.41 ± 0.26 for TBRM and 30.32 ± 0.31 for ReVal. These results provide additional evidence that the improvements of RI-TBRM are consistent across independent training runs rather than being specific to a single random seed.
We thank the reviewer for raising this important concern. To provide a more transparent comparison of computational costs, we measured the wall-clock training time of TBRM, ReVal, and RI-TBRM over 800 training steps on Qwen2.5-Math-7B using two independent seeds.
All times are reported in hours:minutes:seconds and exclude the one-time reference-rollout preparation.
Online training efficiency. RI-TBRM exhibits comparable training costs to both baselines. Across the two seeds, its average training time is approximately 3.4 minutes shorter than TBRM and 3.3 minutes shorter than ReVal. These results indicate that reference-return anchoring introduces negligible additional computational overhead during policy optimization, as it only requires reading cached prompt-wise statistics without introducing an additional critic or reference rollout at each training step.
One-time reference rollout cost. RI-TBRM additionally requires approximately 27 minutes to estimate the reference-policy expected returns before training. Importantly, this is a one-time preprocessing cost rather than a recurring cost during optimization. Since the reference policy, training prompts, and reward function remain fixed, the resulting cached statistics can be reused across independent training runs and random seeds sharing the same reference configuration.
Consequently, when the cache is reused across N training runs, the amortized preprocessing cost per run is
Tpreprocessamortized=N27 minutes.
For example, with three independent training seeds, the amortized preprocessing cost is only 9 minutes per run.
Overall, these measurements show that RI-TBRM achieves consistently higher six-benchmark average accuracy across three training seeds while maintaining comparable online training time. Although the initial preprocessing introduces an additional cost for the first run, this cost is reusable and becomes increasingly amortized across repeated experiments, making RI-TBRM computationally practical without requiring a separate critic.