acceptodds
Under review as a conference paper at ICLR 2027

Gradient-Space Diversity as Credit Assignment for Post-Training Large Language Models

Abstract

Reinforcement learning has become a standard approach for post-training large language models, but improvements in task performance often come with reduced output diversity and increasingly narrow solution patterns. Most existing diversity-aware training methods quantify variation in output, embedding, or hidden-state space and optimize the resulting metric as an auxiliary reward. In contrast, this work leverages training gradients as the primary signal to measure and promote diversity. First, under a local first-order approximation, we show that GRPO propagates group-relative advantages through the rollout-gradient Gram matrix, exposing gradient redundancy among verified solutions as a hidden dimension of credit assignment. Motivated by this observation, we introduce GraDiC, a novel training method that directly operates on this gradient space. GraDiC measures their coverage using the order-2 Vendi Score and differentiates this objective with respect to each response's update weight, yielding a closed-form influence score that quantifies the response's marginal contribution to gradient-space diversity. The resulting update mechanism promotes underrepresented gradient directions and mitigates the tendency of policy updates to concentrate along dominant directions. Experiments on challenging mathematical reasoning benchmarks show that GraDiC substantially improves pass@ for compared with standard baselines.

Primary area:
generative models
Keywords:
language model training, response diversity, reasoning, model robustness, exploration, efficient training, post-training methods

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cBpKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on the issue of output distribution collapse in LLMs after RL post-training. The authors propose to address this problem by introducing gradient diversity directly into the training objective, claiming that traditional output-space diversity measures are too coarse. Their proposed method, GraDIC, redistributes a fixed positive update weight among individual positive rollout gradients within GRPO, favoring updates that point in distinct directions.

Critical strengths
  1. The targeted problem is of high practical importance, and novel methodological approaches are welcome.
  2. The method is novel and well-motivated. Operating directly at the gradient level is a sound idea, and the implementation choices offer valuable insights for the community.
  3. The authors provide helpful theoretical analysis regarding GRPO rollout gradient dynamics.
Critical weaknesses
  1. Main results show high variance across benchmarks and base models. This is likely driven by evaluating on small mathematical benchmark sets. Testing on larger datasets such as GSM8K and MATH-500, alongside additional model families (e.g., DeepSeek-Math or Mathstral), would help establish reliable trends.
  2. The claim that GraDIC preserves a broader, more useful distribution of reasoning paths is not fully supported by Table 1. Pass@k does not isolate diversity: improved performance on hard problems can occur without an increase in generation breadth.
  3. Figure 2 does not clearly demonstrate increased generation diversity. Gradient dissimilarity is the internal optimization objective of GraDIC rather than an established proxy for semantic output diversity. Furthermore, semantic embedding similarity appears on par with standard GRPO and worse than the base model.
  4. The source code is not provided, which limits the reproducibility of the findings.
Questions for authors
  1. Could you think of a proper way to test model's diversity on the mathematical tasks?
  2. Could you please release the code to improve reproducibility?
Reviewer_sLwAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Reinforcement learning generally improves the performance of modern LLMs at the cost of response diversity. This paper proposes a simple remedy, GraDic, which measures the diversity of responses through the induced gradients of the model parameters and adjusts each response’s weight according to its similarity to other responses. In this way, GraDic promotes the underrepresented samples in the batch while downplaying redundant, overrepresented responses, preserving the diversity during reinforcement learning fine-tuning. Experiments on widely used mathematics and reasoning benchmarks show substantial improvements in Pass@128 without compromising Pass@1.

Critical strengths

The paper is clearly presented. The authors first introduce the Vendi score family and then focus on the order-2 score. Using the weighted second-moment matrix of normalized gradient directions, they derive a measure of gradient diversity and a corresponding per-response credit signal that captures each response’s relative novelty. The theoretical motivation is well developed, and the derivation provides a clear rationale for the proposed reweighting approach.

Critical weaknesses

The connection between the standard Vendi formulation and the K(w) in Equation 13 is unclear. The Vendi score requires a kernel matrix k(xi, xj) which measures the similarity between each sample pairs. However, it’s unclear why the K in Equation 13 suffices for this goal. The K(w) in Equation 13 defines the covariance across each component rather than the similarity among samples. A more plausible definition would be to define K as <gi, gj> after some normalization, which not only preserves the tr(K(w)^2) under order-2 cases, but also provides a more consistent realization.

Extracting the gradient and computing the Vendi score adds negligible computational overhead. At 1.5B scale, it increases the runtime by 11.6%. Since the length of the gradient vector scales linearly with the model size and the computational complexity of tr(K^2) scales quadratically with the model size, it's unclear whether this method is cost-effective at production scale.

Questions for authors

The connection between gradient diversity and semantic diversity remains unclear. To what extent does promoting diverse gradient directions encourage meaningfully different responses or reasoning strategies? At the end of Section 6, the authors note that GraDic produces responses with higher semantic similarity than the base model despite substantially greater full-layer gradient diversity. This observation suggests a potential mismatch between the optimization objective and the desired form of diversity. How might this affect GraDic’s applicability beyond mathematics and reasoning, particularly in domains where semantic diversity is a central objective?

Could the authors include qualitative case studies comparing responses from GraDic, standard RL fine-tuning, and the base model?

1 comment

I would like to thank the authors for their further explanation. My concerns, especially W1, W2, and Q2 are addressed. I would maintain my positive evaluation.

Reviewer_7VZMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GraDiC, a modification of GRPO that redistributes positive advantages according to gradient-space diversity. The method extracts projected gradients from an intermediate transformer layer, differentiates a weighted order-2 Vendi objective to obtain response-level credits, and applies exponential reweighting while preserving total positive update mass. Experiments on two Qwen2.5-Math model sizes show improved average pass@128 over several diversity-aware baselines, with broadly comparable pass@1.

Critical strengths
  1. The method provides a clear connection between a set-level diversity objective and response-level advantage reweighting. The fixed-feature ascent results give the proposed credit rule a useful mathematical interpretation.

  2. GraDiC improves average pass@128 at both evaluated model scales while largely maintaining pass@1. The evaluation includes several relevant diversity-aware baselines, along with ablations on Vendi order, gradient layer, and reweighting strength.

Critical weaknesses
  1. The conceptual novelty should be more narrowly positioned: EVOL-RL already studies selection-induced coverage collapse and uses novelty to refine credit within selection groups [1], while gradient-guided exploration and G-Vendi optimization have precedents in [2-4]. GraDiC differs in its verified-reward setting and Vendi-derived advantage reweighting, but the incremental benefit of this specific construction needs clearer isolation.

  2. Appendix E attributes an output-weight gradient to G²RL, whereas [2] defines its feature through gradients with respect to the final hidden states. This discrepancy undermines the interpretation of the layer comparison and leaves the implementation of the closest baseline in Table 1 unclear.

  3. The existing ablations do not isolate the benefit of Vendi-derived credit from the benefit of intermediate-layer gradient features. A matched-feature comparison against simpler pairwise novelty reweighting is needed to establish the contribution of the proposed credit mechanism.

  4. The guarantees concern reweighting fixed gradient features, whereas practical training uses projected single-layer gradients and updates the entire policy. Neither higher pass@k nor lower gradient cosine similarity directly establishes greater diversity of correct reasoning strategies, and the OOD interpretation remains insufficiently validated.

  5. The average pass@128 gains over G²RL are approximately 3.3–3.5 percentage points, but no training-seed variation or confidence intervals are reported. Baseline-specific tuning details and compute-matched comparisons are also insufficiently documented to assess the robustness of this advantage.

[1] Evolving language models without labels: Majority drives selection, novelty promotes variation

[2] Can LLMs Guide Their Own Exploration? Gradient-Guided Reinforcement Learning for LLM Reasoning

[3] Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning

[4] Spokes: Optimizing for Diverse Pretraining Data Selection

Questions for authors
  1. Which exact gradient feature and reward-shaping implementation were used for G²RL in Table 1? Does the output-weight formulation in Appendix E reflect only a description error, or was it also used in the baseline implementation?

  2. Can you provide a matched-feature comparison between GraDiC credit and pairwise novelty reweighting, with repeated training runs and uncertainty estimates?

Comments on the paper

3 comments

Thank you for your detailed response and sharing the code!

Main results While I have no issue with the specific model and benchmarks provided in the rebuttal, I originally requested different, specific options. Could you please elaborate on why you opted for these alternatives? Additionally, I noticed that most of the previously benchmarked methods are missing from the updated tables, even though they achieved the best performance on several models and benchmarks. What I originally asked for does not require retraining the already present models, but simply running inference with them on the new benchmarks.

Regarding pass@k as a diversity metric:

"Higher pass@k indicates that the model can uncover novel solutions lying in rare modes of the response distribution" - While plausible, this is not guaranteed. A model might simply have a higher overall probability of producing a valid solution, as you note yourselves when stating that "evaluating at a single k may only reflect raw capability". "Full pass@k curve as k increases" - If samples are independent and identically distributed, the entire pass@k curve is strictly determined by the single-sample success probability. Consequently, its shape carries no additional information about diversity or rare modes.

Diversity evaluation I agree that evaluating solution diversity on mathematical benchmarks is challenging, and to the best of my knowledge, no perfect metric exists. However, on open-ended textual prompts, the task is significantly simpler and well-supported by existing literature [1, 2].

Evaluating solution diversity with an LLM-as-a-Judge is a solid approach. Using an open-source model instead of GPT might reduce computational overhead and allow you to scale the evaluation beyond 11 prompts. The detailed examples of the two tasks and their solution types are very clear and informative.

The claim that gradient dissimilarity (Figure 2) directly reflects diversity in the model's optimization space is accurate. However, it is unclear from that figure alone what practical utility or meaningful diversity ("useful distribution of reasoning paths") this translates to. The only conclusion one can draw from it is that Gradic indeed optimizes the objective it was designed to improve.

[1] NoveltyBench: Evaluating Language Models for Humanlike Diversity - https://arxiv.org/abs/2504.05228

[2] Benchmarking Linguistic Diversity of Large Language Models - https://arxiv.org/abs/2412.10271

Thank you for clarifying the G²RL implementation and for running the matched-feature ablation across three training seeds. These additions address two of my original concerns: I am satisfied with the explanation of the discrepancy in Appendix E, and the ablation shows that Vendi reweighting improves pass@128 even when the gradient features are held fixed. My rating remains 2 for the following reasons.

  1. The ablation narrows the contribution. With layer-16 features fixed, replacing pairwise novelty with Vendi credit improves average pass@128 by 1.94 and 1.71 points on the two models. Keeping pairwise novelty and changing only the features yields larger gains of 2.73 and 2.48 points. I credit the paper for the improvement from Vendi reweighting, but the full gain over G²RL should not be attributed to that mechanism. Its practical advantage at matched training cost also remains unclear.

  2. The strategy analysis provides direct but limited evidence. Requiring every method to produce at least 16 correct responses leaves 11 of 30 AIME24 problems for Qwen and 16 of 30 for MiMo. On those subsets, GraDiC retains more distinct strategies than GRPO and G²RL, while the base model retains the most. This supports a narrower conclusion: GraDiC better preserves existing strategies on problems where all methods already produce enough correct solutions. Whether that benefit extends to harder problems remains open.

  3. G²RL is a reasonable primary comparator: it is the closest method and the strongest baseline on average pass@128 in Table 1. But the new OOD experiments use it as the only trained baseline, and the strategy analysis adds only GRPO. DQO is a particularly relevant omission from these follow-up experiments: at both model sizes, it ranks second among baselines on average pass@128 and exceeds GraDiC on average pass@1. The OOD gains of 0.99 and 0.73 points over G²RL are encouraging, but they do not establish a broader advantage over other competitive methods.

I also appreciate the clarification that the theoretical guarantee applies to fixed, projected single-layer features. Whether it explains the behavior of the full-policy update remains untested. Overall, the rebuttal makes the pass@128 result more convincing, but the evidence for the broader claims about reasoning diversity and generalization is still too limited for me to recommend acceptance. I am therefore maintaining my rating of 2.

Thank you for the detailed response!

Once again, pass@k is a widely used metric for quality, and therefore its usage to demonstrate the quality of the method raises no questions. However, pass@k cannot be used to rigorously demonstrate diversity. For that purpose, the appropriate approach is the one presented in the rebuttal, using an LLM Judge to group responses by solution style. I strongly insist that this distinction should be clearly presented in the revised manuscript.

Besides providing the only convincing evidence of Gradic's positive effect on diversity, this approach also addresses the issue of pass@k scores being very close across most models and benchmarks: maintaining comparable quality while achieving greater diversity is, in itself, a fully satisfactory outcome.

Under this condition, I am willing to raise my initial score for the revised version of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.