- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper focuses on the issue of output distribution collapse in LLMs after RL post-training. The authors propose to address this problem by introducing gradient diversity directly into the training objective, claiming that traditional output-space diversity measures are too coarse. Their proposed method, GraDIC, redistributes a fixed positive update weight among individual positive rollout gradients within GRPO, favoring updates that point in distinct directions.
Critical strengths
- The targeted problem is of high practical importance, and novel methodological approaches are welcome.
- The method is novel and well-motivated. Operating directly at the gradient level is a sound idea, and the implementation choices offer valuable insights for the community.
- The authors provide helpful theoretical analysis regarding GRPO rollout gradient dynamics.
Critical weaknesses
- Main results show high variance across benchmarks and base models. This is likely driven by evaluating on small mathematical benchmark sets. Testing on larger datasets such as GSM8K and MATH-500, alongside additional model families (e.g., DeepSeek-Math or Mathstral), would help establish reliable trends.
- The claim that GraDIC preserves a broader, more useful distribution of reasoning paths is not fully supported by Table 1. Pass@k does not isolate diversity: improved performance on hard problems can occur without an increase in generation breadth.
- Figure 2 does not clearly demonstrate increased generation diversity. Gradient dissimilarity is the internal optimization objective of GraDIC rather than an established proxy for semantic output diversity. Furthermore, semantic embedding similarity appears on par with standard GRPO and worse than the base model.
- The source code is not provided, which limits the reproducibility of the findings.
Questions for authors
- Could you think of a proper way to test model's diversity on the mathematical tasks?
- Could you please release the code to improve reproducibility?
I would like to thank the authors for their further explanation. My concerns, especially W1, W2, and Q2 are addressed. I would maintain my positive evaluation.