- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TEMPO, a multi-timescale value learning framework for language-model reinforcement learning that aims to improve token-level credit assignment. Inspired by heterogeneous temporal value signals in the brain, TEMPO employs multiple critic heads with different discount factors, computes horizon-specific advantages, and combines them through RMS normalization for PPO optimization. The authors further analyze the effective reward kernel induced by advantage fusion and distinguish multi-horizon value prediction from the advantage readout used for policy updates. Experiments on mathematical reasoning and CommonsenseQA report improvements over PPO in several settings. However, controlled ablations reveal that using only the longest-horizon advantage substantially outperforms averaging multiple horizons, while diverse critic heads show no clear advantage over duplicated long-horizon heads. These findings highlight the importance of actor readout design in multi-timescale reinforcement learning.
Critical strengths
Interesting perspective and insightful empirical analysis. The paper offers an interesting perspective on token-level credit assignment by distinguishing multi-timescale value prediction from the advantage signals used for policy optimization. The controlled experiments reveal that averaging advantages across different horizons can substantially underperform using the longest-horizon advantage alone. This finding provides useful insights into the design of multi-timescale critic architectures for language-model reinforcement learning.
Critical weaknesses
1. Limited methodological novelty and unclear benefits of the proposed approach. Multi-horizon value estimation and advantage construction have been extensively studied in prior reinforcement learning literature. TEMPO mainly combines multiple value heads with RMS-normalized advantage averaging, without introducing a fundamentally new optimization principle. More importantly, the controlled experiments show that averaging multi-timescale advantages substantially underperforms using only the longest-horizon advantage, while diverse value heads provide no clear improvement over duplicated long-horizon heads. These findings call into question the necessity and effectiveness of the proposed multi-timescale architecture.
2. Insufficient experimental validation. The most important controlled comparison, which reports a 7.4-point advantage for long-horizon readout over RMS averaging, is based on a single training seed. The reported bootstrap confidence intervals capture evaluation-question variability rather than training randomness. Furthermore, some broader experimental comparisons involve different training budgets or implementation configurations, limiting their ability to isolate the effect of the proposed method. Consequently, the empirical evidence is insufficient to establish consistent performance gains or generalizability across models and tasks.
Questions for authors
1. Methodological novelty. What are the key contributions of TEMPO beyond existing multi-horizon value estimation and advantage estimation methods?
2. Effectiveness of multi-timescale learning. Given that long-horizon readout outperforms RMS averaging, what is the actual benefit of using multiple timescales?
3. Experimental robustness. Do the main readout comparisons remain consistent across multiple training seeds and different model scales?
4. Baseline comparison. How does TEMPO compare with a standard single-head PPO critic under identical training conditions?