acceptodds
Under review as a conference paper at ICLR 2027

Nash Credit Routing for Optimizing Responses Beyond GRPO

Abstract

A GRPO update can improve its sampled objective while reducing the likelihood of an individual successful response. Repeated, directionally similar successes can also dominate the positive update through multiplicity alone. We introduce Nash Credit Routing (NCR), a constrained correction to GRPO that reallocates a fixed positive-credit budget across successful responses and selectively refunds penalties on failed-response segments. The correction remains near GRPO, preserves at least its first-order sampled-objective improvement, and causes no additional first-order likelihood harm to any sampled success. NCR assigns positive bargaining powers by minimizing directional collision under a multiplicity-calibrated ratio constraint, then maximizes weighted Nash welfare over normalized likelihood gains. Across the evaluated models and mathematics benchmarks, NCR achieves higher macro-average pass@1 and pass@128 than the compared baselines.

Primary area:
reinforcement learning
Keywords:
large language models, reinforcement learning with verifiable rewards

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_23QSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Nash Credit Routing (NCR) to address sample imbalance in GRPO, where repeated responses can dominate the update and weaken useful learning signals. NCR reallocates a fixed positive-credit budget among successful responses and selectively reduces penalties on failed-response segments. It selects bargaining powers by minimizing directional collision and maximizes weighted log welfare over normalized additional likelihood gains. The theoretical analysis provides local first-order protections relative to GRPO, characterizes robust optimality at a solution-induced KL radius, and derives conditional bounds on held-out accuracy changes. Experiments on three Qwen models show improvements over GRPO, NTHR, and Div-R1, with the most consistent gains observed in mathematical Pass@128.

Critical strengths
  1. NCR provides a concrete, principled framework for credit allocation. It jointly optimizes positive-credit redistribution and negative-segment refunds under explicit constraints on the correction magnitude and first-order progress relative to GRPO. This is a meaningful algorithmic contribution, and the idealized analysis carefully distinguishes relative protection from a guarantee that every successful response increases in likelihood.

  2. I find the theoretical explanation of the Nash objective informative. Theorem 2 connects weighted Nash welfare to worst-case alignment under relevance uncertainty at a solution-induced KL radius. By comparing Nash and linear allocation using the same bargaining powers and feasible correction set, the analysis helps clarify the role of the log-welfare objective. This provides an interpretable mathematical rationale for the design and a useful basis for comparing alternative credit-allocation rules within the stated local framework.

Critical weaknesses
  1. I find the connection between the theoretical guarantees and practical training insufficiently validated. The protections concern full-gradient, single-group, direct updates. Appendix D explicitly acknowledges that the selected-token proxy does not preserve full-gradient inner products, while Appendix J explains that minibatch aggregation and optimizer transformations do not automatically preserve the guarantees. Although these limitations are clearly stated, measurements showing whether the protections hold during actual training are unavailable. The robust optimality result also applies at an uncertainty radius induced by the Nash solution, and its interpretation in terms of held-out accuracy relies on a strong test-gradient representation assumption. The pooled proxy diagnostic does not establish the applicability of this assumption or bridge the gap to actual training updates; moreover, its within-prompt alignment remains negative. I therefore view the theory as a useful conditional justification, but the practical operation of the proposed mechanism still requires stronger validation.

  2. I find the empirical evaluation insufficient to establish the generalizability and practical benefits of NCR. Although the experiments cover three models across Qwen2.5 and Qwen3, effectiveness beyond the Qwen family remains unclear, as no results are reported on other widely used open-weight model families, such as Llama. Sensitivity to sampling temperature is also unexamined, despite the method's reliance on the distribution and repetition of sampled response directions. Furthermore, the strongest gains concern Pass@128, which requires a substantial sampling budget and measures whether the candidate set contains a correct answer, rather than whether a deployable selector can identify it. In comparison, the mathematical macro-average Pass@1 improvements over GRPO are modest: 1.01, 0.17, and 0.35 percentage points across the three models. Although Figure 2 reports Pass@k at several sampling budgets, the paper does not evaluate selected-answer accuracy or demonstrate an advantage under matched token or inference-time budgets. There are also reporting issues that need clarification: the text describes a positive-only ablation, whereas Table 2 reports response-level refunds, and Table 3 does not explain how NCR module time contributes to total wall-clock time. These limitations leave me uncertain about the method's broader applicability and whether its additional training computation delivers a meaningful practical benefit.

Questions for authors
  1. Could you clarify whether the training runs behind Tables 1–2 follow the update procedure specified in Appendix E, including coefficient refresh and solver fallback, and provide the segmentation rules and optimizer choice? I would also find it helpful to see measurements on a manageable subset of training steps showing whether the proxy protections hold in full-gradient geometry, after minibatch aggregation, and under the applied optimizer update. NCR and GRPO should be compared using the same cached batch and identical initial optimizer states.

  2. Although the experiments cover multiple model sizes and two Qwen generations, I would like to understand whether the benefits extend beyond this family. Do you have results on another widely used open-weight model family, such as Llama? Given that sampling temperature can change the frequency and similarity of sampled responses, could you also evaluate temperature sensitivity using the existing checkpoints while keeping the prompts, top-p, response-length limits, and other evaluation settings fixed?

  3. Could you provide a numerical comparison of Pass@2, Pass@4, Pass@8, and Pass@16 across the evaluated methods to clarify NCR’s gains at smaller sampling budgets? Figure 2 already presents Pass@k curves, but a table reporting these values for each model and benchmark, together with their macro-averages and differences relative to GRPO, would make the magnitude and consistency of the improvements easier to assess. I would find this particularly helpful for evaluating the method’s benefits when only a small number of responses can be generated.

  4. Could you clarify the discrepancy between the positive-only ablation described in the text and the response-level refund variant reported in Table 2? Is a result available with negative-credit refunds disabled? Please also explain whether the runtime components in Table 3 overlap or are nested, and provide an end-to-end GRPO runtime comparison under the same hardware and workload.

  5. What evidence supports the applicability of the test-gradient representation assumption at the Nash-induced radius? I would also appreciate clarification of the interpretation following Theorem 3 in light of Appendix I: a positive bound requires an alignment margin large enough to exceed the error terms, and the difference between two worst-case advantages over GRPO is not generally the worst-case direct advantage over linear routing.

1 comment

The authors’ response has addressed my main concerns, and I will raise my score to 3 (weak accept).

Reviewer_66DNOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Nash Credit Routing (NCR), a constrained modification of GRPO that jointly redistributes positive credit among successful responses and selectively reduces penalties on failed-response segments. NCR first determines bargaining powers by minimizing directional redundancy, then maximizes weighted Nash welfare over normalized likelihood improvements relative to GRPO. Its constraints limit the correction magnitude, preserve first-order progress on the sampled objective, and prevent additional first-order harm to sampled successful responses.

Critical strengths
  1. Improving the exploration ability is important for LLM RL.

  2. Jointly optimizing positive-response credit and negative-segment refunds provides a structured way to address gradient interference.

  3. NCR improved pass@128 ability on evaluated model–benchmark combinations.

Critical weaknesses
  1. Appendix L.2 describes splitting failed responses at blank lines and merging adjacent segments to retain at most 16 segments per response, but this description concerns the local diagnostic experiment, however the merging policy is not fully specified.

  2. Table 3 reports per-step runtimes, but token-feature computation depends on the number of processed tokens, thus controlled profiling across response lengths and segment counts would clarify scalability. The paper also does not report peak GPU memory usage or its increase relative to GRPO.

  3. The last layer approximation for gradients actually follows THR and NTHR, so it should be discussed and cited in properly in corresponding parts.

  4. Experiments on Llama or another model family would strengthen the evidence that both the performance gains and resource requirements generalize beyond Qwen.

[1] On the effect of negative gradient in group relative deep reinforcement optimization.

[2] Token hidden reward: Steering exploration-exploitation in group relative deep reinforcement learning.

Questions for authors

See weakness

Reviewer_KgjYOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes to reweighing of different samples’ gradients in GRPO groups. In standard GRPO, on-policy samples are weighted based on their reward relative to the group average, so the group update is a mix of each answer’s (log prob) gradient direction, weighted by the relative reward. The authors (based on previous work) say this can cause a few problems: repeated, similar correct answers can dominate the update, while other correct answers have less influence because they are less likely to be sampled by the base model, and wrong answers may contain pieces of correct reasoning that get penalized along with the rest of the answer.

The authors use ideas from economics about welfare to formulate a heuristic based on an optimization problem that distributes weight more evenly among correct responses and moderates negative penalties, subject to constraints that attempt to preserve the gradient’s direction and magnitude and improvements to the log-probabilities of correct responses. They also incorporate the similarities between gradients produced by different samples and use their similarities to aggregate answers into “soft” bins that count as similar solutions in the final weighting. Overall, the paper proposes this heuristic method to improve GRPO that aims to make the update less focused on a few common directions and give more weight to underrepresented correct directions, leading to a more diverse but still correct policy.

Critical strengths
  • The paper is trying to address an important problem of diversity collapse in models trained with GRPO.
Critical weaknesses
  • The paper is rather dense and difficult to read.

  • The computational overhead needs to be quantified. Using per-sample gradient information and gradient similarities to calculate the GRPO weights appears to introduce additional cost. The paper reports this overhead in runtime, it should also do memory, and clarify whether the comparisons account for it through matched compute budgets ( if not this is an important experiment) .

  • Experimental evidence as it stands is weak. Several results do not point to a consistent win for the full approach, and the tables do not include error bars or uncertainty estimates. This is particularly important for AIME benchmarks, which contain few questions: the uncertainty is most likely larger than the reported gaps.

  • The experiments on instruction-tuned models do not clearly establish the benefit of the algorithm. These models have already been heavily trained on tasks such as math, which makes it difficult to assess how much the proposed method adds. Experiments starting from a base model, such as Qwen3-1.7B-Base, are important to see whether the gains become clearer in this setting. As they stand, I do not find the results particularly compelling.

  • Limited experiments The paper should evaluate training on other tasks, such as science/medical MQA and instruction following. Experiments with models outside the Qwen family would also help establish whether the findings generalize.

Questions for authors

Same as above, plus I add a few extra points:

  • Did you account for compute when making your comparisons?
  • It may be beneficial to plot the diversity of the generated samples.
  • How much more memory does your method require? If we adjusted for that by changing the batch size and then accounted for compute, do you still see the same benefits?
  • Overall, I think if the experimental setting were better, perhaps this method may be helpful in getting a more diverse model that could do well in Pass@128. However, there is lots of research of training models to be more diverse that you are not considering and properly citing and comparing.

Comments on the paper

1 comment

Thank you for your detailed response and additional results, all my concerns are addressed and I recommand for acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.