- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
In LLM RL training, there are two common ways to aggregate token contributions: token-mean and sequence-mean. The authors claim that token-mean prefers longer trajectories and vice versa. They propose to use token-mean on reasoning parts for better exploration, and sequence-mean on action parts for efficient actions. They also propose inverse-step advantage penalty and globally normalized surrogate loss to facilitate training.
Critical strengths
- Good motivation. The Figure 2 effectively demonstrate the difference between token-mean and sequence-mean.
Critical weaknesses
- The authors say that "Token-mean favors richer CoT exploration at the cost of behavioral redundancy" and "applies a token-mean constraint on reasoning tokens to foster sufficient exploration". Longer reasoning parts in experiments does not only mean sufficient exploration, but also repetition, unnecessary deliberation. A direct comparison between token-mean and sequence-mean that shows better exploration would be helpful.
- Section 4.1 claims that "This decoupled optimization introduces additional backward passes and memory overhead". The method only changes the coefficient, nothing about gradient. It does not require additional gradient computation. The contribution is overstated.
- Inverse-step advantage penalty is actually a heuristic trick, it does not deserve a section. Furthermore, for negative advantage trajectories, inverse-step advantage penalty actually makes smaller negative updates. This trick needs further justification.
Questions for authors
- Equation 9-14 use trajectory level advantage, section 5.1 says "GRPO with the step-level advantage", and Algorithm 1 uses a value function to compute baseline, instead of GRPO style. Could you please clarify which one you use?
- Similar to the issue that longer reasoning does not directly imply better exploration, how could you say that shorter action tokens could mean efficient actions? Would smaller number of function calls be a better representation of efficient action?