acceptodds
Under review as a conference paper at ICLR 2027

Information-Time Proximal Policy Optimization

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved the reasoning capabilities of Large Language Models (LLMs). However, existing methods typically measure temporal progression by token count, despite the highly non-uniform information flow along autoregressive trajectories. We propose Information-Time Proximal Policy Optimization (InfoPPO), which measures temporal distance by accumulated state-wise information rather than raw token count. This provides a common basis for temporal credit propagation and policy updates. InfoPPO enables non-trivial discounting in long-horizon reasoning, retaining effective-horizon contraction in information time while mitigating excessive attenuation of terminal supervision. Moreover, the information-time policy-improvement analysis naturally leads to a state-dependent update constraint, which we implement through adaptive clipping. It scales the update range with local information density, enabling targeted policy updates while preserving proximal control. Theoretically, we extend performance-difference and policy-improvement analyses to the information-time Markov Decision Process (MDP), deriving a policy-improvement lower bound. We further connect this analysis to practical LLM policy optimization by relating state-wise information density to local policy movement, while also providing theoretical grounding for the adaptive update mechanism. Experiments on Qwen3 models demonstrate consistent gains over competitive baselines across five challenging competition-style mathematical reasoning benchmarks. InfoPPO also maintains stable accuracy and response length across non-trivial discount settings under which token-time PPO deteriorates.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AgqHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper challenges a basic modeling convention in RLVR: treating every generated token as one equal unit of temporal progression.

The authors argue that autoregressive reasoning trajectories exhibit highly non-uniform information flow, so token distance does not necessarily correspond to information distance. They introduce a state-dependent information density and define information-time discounting

In practical LLM RLVR, the predictive entropy of the frozen old policy is used as a proxy for �0��0�. High-entropy states are interpreted as positions with a broader set of plausible next-token continuations and therefore higher local information density. 21990_Information_Time_Proxima

The same information density is used in two places:

  • temporal credit assignment via information-time discounting and trace decay;
  • state-dependent PPO clipping for policy improvement.

The paper extends the performance-difference lemma to the information-time MDP and derives a lower bound under state-dependent policy-divergence constraints.

Experiments on Qwen3-4B, 8B, and 14B across AMC23, AIME24, AIME25, AIME26, and BeyondAIME show higher average accuracy than PPO, DAPO, and DAPO-FT, while maintaining shorter response lengths than standard PPO.

Critical strengths
  1. The research question is original and fundamental.

    Asking whether “one token should equal one RL time step” is conceptually simple but important. Reparameterizing time by information accumulation is a genuinely interesting direction for long-horizon language-model RL.

  2. The method is conceptually unified.

    The same quantity ρ(s)\rho(s) controls both temporal discounting and policy-update scale. This gives the method a coherent design rather than a collection of unrelated tricks.

  3. The theoretical development is substantive.

    The paper does not merely define a new heuristic clock; it reformulates the performance-difference analysis and derives an information-dependent policy-improvement bound.

  4. The non-trivial discounting experiment directly tests the paper’s core motivation.

    Token-time PPO degrades sharply when γ<1\gamma<1, whereas information-time discounting preserves accuracy and produces more stable response lengths.

    This is one of the strongest pieces of evidence in the paper.

  5. The gains are consistent across model scales.

    InfoPPO performs well from Qwen3-4B to 14B rather than relying on a single model configuration.

Critical weaknesses
  1. Predictive entropy is not obviously equivalent to “information density.”

    Next-token uncertainty may reflect lexical diversity or stylistic ambiguity rather than reasoning importance. Conversely, some semantically crucial mathematical transitions may occur at low entropy.

  2. The two main modifications are not fully disentangled.

    The final method changes both credit propagation and the clipping geometry. It would be useful to separately evaluate:

    • information-time GAE only;
    • adaptive clipping only;
    • the combination.
  3. There remains a theory-to-practice gap in the information clock.

    The theory is stated using a common state-dependent clock, whereas the implementation recomputes entropy and uses batch-level normalization. This introduces an additional sample-dependent component that is not fully captured by the idealized analysis.

  4. A key control experiment is missing: a shuffled or randomized information clock.

    If the same ρt\rho_t values are randomly permuted within each trajectory, does performance degrade? Such an experiment would directly test whether the location of high-information states matters, rather than only the average discount scale.

Questions for authors
  1. Why is Shannon entropy the preferred proxy for reasoning information density? Did the authors compare alternatives such as surprisal, top-k mass, margin, or value uncertainty?
  2. Can the authors add a matched-effective-horizon baseline?
  3. What happens if ρt\rho_t values are randomly shuffled within a trajectory?
  4. Can information-time GAE and adaptive clipping be ablated separately?
  5. Why should high-information states permit larger policy movement? Could the opposite design also be justified?
  6. Can the relationship between predictive entropy and reasoning transitions be quantified beyond qualitative token examples?
Reviewer_9DcxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes InfoPPO, an information-time proximal policy optimization method designed to improve long-horizon reasoning in LLM reinforcement learning. The method uses normalized predictive entropy instead of token count to measure temporal progression and incorporates information time into reward discounting and generalized advantage estimation (GAE). Additionally, the authors introduce an information-dependent adaptive clipping mechanism to dynamically regulate policy updates. The paper derives a theoretical policy-improvement lower bound and demonstrates improvements over PPO, DAPO, and DAPO-FT on Qwen3 models across multiple mathematical reasoning benchmarks.

Critical strengths
  1. The paper addresses a practically meaningful problem related to temporal discounting and credit assignment in long-horizon LLM reinforcement learning.
  2. The information-time formulation provides a relatively simple and unified approach to advantage estimation and adaptive policy updates, supported by theoretical analysis.
  3. The experiments cover different model scales and challenging mathematical reasoning benchmarks, demonstrating consistent performance improvements and robustness across different discount factors and GAE trace-decay parameters.
Critical weaknesses
  1. The method introduces two major components, Information-Time GAE and Adaptive Clipping, but lacks sufficient component-level ablations. It remains unclear how much each component contributes to the overall performance improvement. Evaluating these components independently would help establish their effectiveness and necessity.
  2. The advantage of information-dependent discounting over simpler alternatives has not been sufficiently demonstrated. Since normalized predictive entropy is generally small, the effective per-token discount factor may be very close to 1. Although Figure 4 includes experiments with γ = 1, the compared methods also differ in other optimization settings. A baseline using a comparable fixed effective discount factor, while controlling other settings, would better establish the independent benefit of non-uniform information-time discounting.
  3. The use of predictive entropy as a measure of reasoning-relevant information is not sufficiently justified. High predictive entropy does not necessarily correspond to important reasoning decisions and may simply reflect uncertainty over inconsequential token choices. More direct experimental evidence connecting predictive entropy to reasoning importance, or comparisons with alternative information measures, would strengthen the motivation.
Questions for authors

Please refer to the weaknesses.

Reviewer_vXKHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

InfoPPO uses normalized next-token entropy to define how much time passes at each token. This changes discounting and trace decay in GAE, and also sets the PPO clipping range. The paper derives a performance-difference identity and an improvement bound under a state-wise divergence constraint, followed by analyses of local logit updates and changes to the entropy clock. Experiments on Qwen3-4B, 8B, and 14B report improvements over PPO, DAPO, and DAPO-FT on five math benchmarks. There is also a smaller Llama experiment and a study of the discount, normalization, and clipping settings.

Critical strengths

Using one quantity to regulate both credit propagation and update size is an interesting idea. The method is simple to add to PPO and uses information already available from the old policy. The theoretical treatment is more careful than just substituting entropy into a discount factor: it states the divergence condition and considers what happens when the clock is recomputed across updates.

The results are encouraging across the three Qwen3 scales. Ordinary PPO uses gamma=lambda=1 in the main comparison, which is an appropriate baseline for long responses. The discount and trace-decay sweeps show where the tested token-time settings break down, and the response-length plots are helpful for interpreting the accuracy gains. Appendix E's Llama results give some evidence beyond the main model family.

Critical weaknesses

I cannot tell which part of InfoPPO accounts for the gains. The final method changes temporal decay and clipping at the same time. Appendix C varies the clipping parameters, but does not isolate the information clock from adaptive clipping. In particular, a constant information density just rescales gamma and lambda. A baseline matched to that effective decay would help distinguish the value of token-dependent information from the benefit of discounting less aggressively. The same issue applies to the contribution of the clipping rule.

The link between the theory and the implementation is less settled than the main text suggests. Proposition 4.3 analyzes a local gradient step on the logits at one state. An LLM parameter update affects many states together, and clipping the PPO objective does not enforce a hard state-wise TV bound. There is a second difference: the clock-stability argument holds the normalization scale fixed, while the main experiments use batch normalization. Appendix A.2 notes this distinction, but stable average entropy does not show that the state-wise assumptions hold. These are limits on what the analysis establishes for the implemented algorithm, rather than an identified error in the fixed-clock bound.

Questions for authors
  1. How do the following compare under a matched budget: a constant clock with the same average effective decay, information-time GAE with fixed clipping, and adaptive clipping with ordinary GAE?

  2. Which parts of the theoretical guarantee apply to the actual shared-parameter, batch-normalized optimizer? Is there a check of state-wise policy movement or clock drift beyond the plots of batch averages?

  3. How many independent training seeds are behind Table 1? Please also explain checkpoint selection and which data were used to choose gamma/lambda and the clipping parameters. The 16 evaluation samples per problem do not capture variation between training runs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.