acceptodds
Under review as a conference paper at ICLR 2027

TEMPO: Multi-timescale Value Learning and Policy Credit for Lanuage Models

Abstract

Reinforcement learning for language models must turn response-level rewards into credit for individual tokens. Inspired by heterogeneous temporal value signals in the brain, TEMPO trains a multi-timescale critic, normalizes each horizon’s advantages by their root-mean-square amplitude, and combines them for policy optimization. We analyze the reward kernel induced by this construction and distinguish temporal prediction from actor readout. In the original Qwen3-14B comparison, TEMPO improves greedy accuracy on all five mathematical bench- marks, including MATH500 from 78.4% to 84.0% and MATH5000 from 79.40% to 85.18%. On CommonsenseQA, Qwen2.5-7B-Instruct reaches 83.95 ± 0.54% validation accuracy across three training seeds, compared with 82.69 ± 0.91% for PPO, while generating substantially shorter answers. Controlled Qwen3-1.7B ex- periments then isolate the readout choice: with the same four-horizon target design, long-horizon readout achieves 72.2% MATH500 accuracy versus 64.8% for RMS averaging; a duplicated long-horizon control achieves 72.8%. Shared-pool sampled evaluation and logged advantage geometry clarify this separation. The results show why amplitude balance and temporal diversity must be evaluated alongside the reward weighting supplied to the actor, providing a concrete account of when a multi-timescale construction changes language-model policy learning. Code is available at https://anonymous.4open.science/r/TEMPO-5D50.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, language-model post-training, temporal credit assignment, multi-timescale value learning, brain-inspired learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kRqUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TEMPO, a multi-timescale value learning framework for language-model reinforcement learning that aims to improve token-level credit assignment. Inspired by heterogeneous temporal value signals in the brain, TEMPO employs multiple critic heads with different discount factors, computes horizon-specific advantages, and combines them through RMS normalization for PPO optimization. The authors further analyze the effective reward kernel induced by advantage fusion and distinguish multi-horizon value prediction from the advantage readout used for policy updates. Experiments on mathematical reasoning and CommonsenseQA report improvements over PPO in several settings. However, controlled ablations reveal that using only the longest-horizon advantage substantially outperforms averaging multiple horizons, while diverse critic heads show no clear advantage over duplicated long-horizon heads. These findings highlight the importance of actor readout design in multi-timescale reinforcement learning.

Critical strengths

Interesting perspective and insightful empirical analysis. The paper offers an interesting perspective on token-level credit assignment by distinguishing multi-timescale value prediction from the advantage signals used for policy optimization. The controlled experiments reveal that averaging advantages across different horizons can substantially underperform using the longest-horizon advantage alone. This finding provides useful insights into the design of multi-timescale critic architectures for language-model reinforcement learning.

Critical weaknesses

1. Limited methodological novelty and unclear benefits of the proposed approach. Multi-horizon value estimation and advantage construction have been extensively studied in prior reinforcement learning literature. TEMPO mainly combines multiple value heads with RMS-normalized advantage averaging, without introducing a fundamentally new optimization principle. More importantly, the controlled experiments show that averaging multi-timescale advantages substantially underperforms using only the longest-horizon advantage, while diverse value heads provide no clear improvement over duplicated long-horizon heads. These findings call into question the necessity and effectiveness of the proposed multi-timescale architecture.

2. Insufficient experimental validation. The most important controlled comparison, which reports a 7.4-point advantage for long-horizon readout over RMS averaging, is based on a single training seed. The reported bootstrap confidence intervals capture evaluation-question variability rather than training randomness. Furthermore, some broader experimental comparisons involve different training budgets or implementation configurations, limiting their ability to isolate the effect of the proposed method. Consequently, the empirical evidence is insufficient to establish consistent performance gains or generalizability across models and tasks.

Questions for authors

1. Methodological novelty. What are the key contributions of TEMPO beyond existing multi-horizon value estimation and advantage estimation methods?

2. Effectiveness of multi-timescale learning. Given that long-horizon readout outperforms RMS averaging, what is the actual benefit of using multiple timescales?

3. Experimental robustness. Do the main readout comparisons remain consistent across multiple training seeds and different model scales?

4. Baseline comparison. How does TEMPO compare with a standard single-head PPO critic under identical training conditions?

Reviewer_Gv6KOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the token-level credit assignment problem of PPO in reinforcement learning for LLMs. In PPO, a 0/1 reward signal is typically obtained only at the end of the response and is then distributed to earlier tokens through the value function and GAE. The discount factor γ determines how quickly the reward decays with distance. The authors argue that the commonly used single discount factor (γ = 1) has limitations. Inspired by the neuroscience finding that “dopamine neurons encode reward predictions across multiple timescales,” the authors propose TEMPO. TEMPO lets the critic output 4 value heads, each trained with a different discount factor (γ = 0.97, 0.99, 0.997, and 1), and each head computes its own GAE advantage. The advantages of all heads are then normalized by their within-batch RMS and averaged to form the actor’s advantage for the PPO update. However, in the final controlled experiment, the authors find that the actor performs best when it reads only the advantage from the γ = 1 value head. They therefore change the paper’s conclusion to “the readout matters,” but this in fact refutes TEMPO itself.

Critical strengths

The approach is simple and adds almost no overhead. It only adds a few output heads to the critic and is easy to integrate into existing PPO code.

Critical weaknesses
  1. The method is refuted by its own experiments, and the experimental logic is far from rigorous. In the controlled experiment on Qwen3-1.7B, TEMPO’s averaged readout (64.8%) is clearly lower than reading only the γ = 1 head (72.2%). Moreover, using 4 auxiliary heads with different γ values (72.2%) shows no gain over using 4 heads that all have γ = 1 (72.8%). In other words, of the method’s two core design choices, one is ineffective and the other is harmful, and the source of the improvement on 14B cannot be explained.
  2. The method is unsound in principle. Long CoT reasoning can reach tens of thousands of tokens, whereas a discount of γ < 1 decays almost to 0 after a few hundred tokens (e.g., 0.99^500 ≈ 0.007), so it cannot propagate the reward back to earlier tokens at all. Averaging these heads only dilutes the credit assigned to early tokens.
  3. The commonsense QA experiment does not support the motivation. The responses in the commonsense QA experiment are only about 7 tokens long, so the problem of “credit assignment in long reasoning” does not exist there at all. This experiment therefore cannot support the paper’s motivation.
  4. The evaluation in Table 1 does not follow standard practice: on datasets such as AIME, each of the 30 problems is evaluated with only a single sample. The standard practice is to repeat each problem 128 or 256 times and report the average.
Questions for authors

See above

Reviewer_bozLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TEMPO, a multi-timescale value learning framework for reinforcement learning in large language models. The method employs a shared critic with multiple value heads corresponding to different discount factors, computes horizon-specific advantages, and normalizes them using root-mean-square (RMS) scaling before aggregation for PPO-based policy optimization. The authors also provide a theoretical analysis of the resulting reward kernel, clarifying how normalization and actor readout affect the weighting of delayed rewards and intermediate KL costs. Experiments with Qwen models of different sizes show improvements over PPO in several settings, particularly on mathematical reasoning benchmarks. Controlled experiments further investigate critic target diversity and actor readout strategies. The results show that long-horizon actor readout outperforms normalized advantage averaging, while diverse auxiliary horizons perform similarly to duplicated undiscounted heads. These findings demonstrate the importance of actor readout, while the additional benefit of multi-timescale value learning remains unclear.

Critical strengths
  1. The paper introduces a biologically inspired multi-timescale value learning framework for language model reinforcement learning, using multiple discount factors to construct token-level advantage signals. The separation of temporal prediction from actor readout provides a useful perspective on policy credit assignment.

  2. The paper provides a reward-kernel decomposition that clarifies how RMS-normalized advantage aggregation changes the weighting of delayed rewards and intermediate KL costs.

  3. The paper investigates critic target diversity and actor readout strategies through controlled experiments and advantage-similarity analyses. The experimental evaluation also includes sampled evaluation, negative results, and a three-seed CommonsenseQA study, providing useful evidence about the effects of different design choices.

Critical weaknesses
  1. Multi-horizon value prediction and long-horizon actor readout have been investigated in prior work, including Sun (2026). The RMS fusion and conditional reward-kernel decomposition appear to provide limited incremental novelty beyond existing approaches.

  2. Table 2 shows that A4-long (72.2%) slightly underperforms D4-long (72.8%), while A4-mean (64.8%) performs substantially worse. These results do not establish clear benefits from diverse critic targets or multi-timescale advantage aggregation. The controlled experiments also lack a matched single-head critic and a scalar critic trained on the mixed target.

  3. Although Figure 3 demonstrates distinct temporal advantage signals, it does not establish whether such diversity benefits policy optimization. The reward-kernel analysis also leaves the individual effects of critic learning, discount selection, and advantage aggregation insufficiently separated.

  4. TEMPO shows improvements on Qwen3-14B but does not consistently outperform baselines on Qwen2.5-7B. The original matched-budget 14B comparison and controlled 1.7B experiments use single training seeds, while the historical three-seed 14B results use unequal training budgets. Additionally, MATH500 overlaps with MATH5000, and AIME contains only 30 questions, making small improvements difficult to interpret.

  5. The main attribution experiments use a different model size and horizon spectrum from the original 14B experiments. The 1.7B controls include an undiscounted head, whereas the original 14B spectrum ends at γ = 0.999. These differences make it difficult to directly explain the larger-model improvements using the controlled results.

Questions for authors
  1. Can the current matched experimental setup include a single undiscounted critic and a scalar mixed-target critic, with comparable effective value-loss scaling, to better establish the benefits of multi-timescale value learning?

  2. What is the principal new finding beyond previously studied target decoupling, and which experiment most directly establishes it?

  3. Can the same actor-readout controls be evaluated on Qwen3-14B while keeping the horizon spectrum and training configuration fixed, to better connect the controlled findings with the original larger-model results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.