- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper challenges a basic modeling convention in RLVR: treating every generated token as one equal unit of temporal progression.
The authors argue that autoregressive reasoning trajectories exhibit highly non-uniform information flow, so token distance does not necessarily correspond to information distance. They introduce a state-dependent information density and define information-time discounting
In practical LLM RLVR, the predictive entropy of the frozen old policy is used as a proxy for . High-entropy states are interpreted as positions with a broader set of plausible next-token continuations and therefore higher local information density. 21990_Information_Time_Proxima
The same information density is used in two places:
- temporal credit assignment via information-time discounting and trace decay;
- state-dependent PPO clipping for policy improvement.
The paper extends the performance-difference lemma to the information-time MDP and derives a lower bound under state-dependent policy-divergence constraints.
Experiments on Qwen3-4B, 8B, and 14B across AMC23, AIME24, AIME25, AIME26, and BeyondAIME show higher average accuracy than PPO, DAPO, and DAPO-FT, while maintaining shorter response lengths than standard PPO.
Critical strengths
-
The research question is original and fundamental.
Asking whether “one token should equal one RL time step” is conceptually simple but important. Reparameterizing time by information accumulation is a genuinely interesting direction for long-horizon language-model RL.
-
The method is conceptually unified.
The same quantity controls both temporal discounting and policy-update scale. This gives the method a coherent design rather than a collection of unrelated tricks.
-
The theoretical development is substantive.
The paper does not merely define a new heuristic clock; it reformulates the performance-difference analysis and derives an information-dependent policy-improvement bound.
-
The non-trivial discounting experiment directly tests the paper’s core motivation.
Token-time PPO degrades sharply when , whereas information-time discounting preserves accuracy and produces more stable response lengths.
This is one of the strongest pieces of evidence in the paper.
-
The gains are consistent across model scales.
InfoPPO performs well from Qwen3-4B to 14B rather than relying on a single model configuration.
Critical weaknesses
-
Predictive entropy is not obviously equivalent to “information density.”
Next-token uncertainty may reflect lexical diversity or stylistic ambiguity rather than reasoning importance. Conversely, some semantically crucial mathematical transitions may occur at low entropy.
-
The two main modifications are not fully disentangled.
The final method changes both credit propagation and the clipping geometry. It would be useful to separately evaluate:
- information-time GAE only;
- adaptive clipping only;
- the combination.
-
There remains a theory-to-practice gap in the information clock.
The theory is stated using a common state-dependent clock, whereas the implementation recomputes entropy and uses batch-level normalization. This introduces an additional sample-dependent component that is not fully captured by the idealized analysis.
-
A key control experiment is missing: a shuffled or randomized information clock.
If the same values are randomly permuted within each trajectory, does performance degrade? Such an experiment would directly test whether the location of high-information states matters, rather than only the average discount scale.
Questions for authors
- Why is Shannon entropy the preferred proxy for reasoning information density? Did the authors compare alternatives such as surprisal, top-k mass, margin, or value uncertainty?
- Can the authors add a matched-effective-horizon baseline?
- What happens if values are randomly shuffled within a trajectory?
- Can information-time GAE and adaptive clipping be ablated separately?
- Why should high-information states permit larger policy movement? Could the opposite design also be justified?
- Can the relationship between predictive entropy and reasoning transitions be quantified beyond qualitative token examples?