- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Nash Credit Routing (NCR) to address sample imbalance in GRPO, where repeated responses can dominate the update and weaken useful learning signals. NCR reallocates a fixed positive-credit budget among successful responses and selectively reduces penalties on failed-response segments. It selects bargaining powers by minimizing directional collision and maximizes weighted log welfare over normalized additional likelihood gains. The theoretical analysis provides local first-order protections relative to GRPO, characterizes robust optimality at a solution-induced KL radius, and derives conditional bounds on held-out accuracy changes. Experiments on three Qwen models show improvements over GRPO, NTHR, and Div-R1, with the most consistent gains observed in mathematical Pass@128.
Critical strengths
-
NCR provides a concrete, principled framework for credit allocation. It jointly optimizes positive-credit redistribution and negative-segment refunds under explicit constraints on the correction magnitude and first-order progress relative to GRPO. This is a meaningful algorithmic contribution, and the idealized analysis carefully distinguishes relative protection from a guarantee that every successful response increases in likelihood.
-
I find the theoretical explanation of the Nash objective informative. Theorem 2 connects weighted Nash welfare to worst-case alignment under relevance uncertainty at a solution-induced KL radius. By comparing Nash and linear allocation using the same bargaining powers and feasible correction set, the analysis helps clarify the role of the log-welfare objective. This provides an interpretable mathematical rationale for the design and a useful basis for comparing alternative credit-allocation rules within the stated local framework.
Critical weaknesses
-
I find the connection between the theoretical guarantees and practical training insufficiently validated. The protections concern full-gradient, single-group, direct updates. Appendix D explicitly acknowledges that the selected-token proxy does not preserve full-gradient inner products, while Appendix J explains that minibatch aggregation and optimizer transformations do not automatically preserve the guarantees. Although these limitations are clearly stated, measurements showing whether the protections hold during actual training are unavailable. The robust optimality result also applies at an uncertainty radius induced by the Nash solution, and its interpretation in terms of held-out accuracy relies on a strong test-gradient representation assumption. The pooled proxy diagnostic does not establish the applicability of this assumption or bridge the gap to actual training updates; moreover, its within-prompt alignment remains negative. I therefore view the theory as a useful conditional justification, but the practical operation of the proposed mechanism still requires stronger validation.
-
I find the empirical evaluation insufficient to establish the generalizability and practical benefits of NCR. Although the experiments cover three models across Qwen2.5 and Qwen3, effectiveness beyond the Qwen family remains unclear, as no results are reported on other widely used open-weight model families, such as Llama. Sensitivity to sampling temperature is also unexamined, despite the method's reliance on the distribution and repetition of sampled response directions. Furthermore, the strongest gains concern Pass@128, which requires a substantial sampling budget and measures whether the candidate set contains a correct answer, rather than whether a deployable selector can identify it. In comparison, the mathematical macro-average Pass@1 improvements over GRPO are modest: 1.01, 0.17, and 0.35 percentage points across the three models. Although Figure 2 reports Pass@k at several sampling budgets, the paper does not evaluate selected-answer accuracy or demonstrate an advantage under matched token or inference-time budgets. There are also reporting issues that need clarification: the text describes a positive-only ablation, whereas Table 2 reports response-level refunds, and Table 3 does not explain how NCR module time contributes to total wall-clock time. These limitations leave me uncertain about the method's broader applicability and whether its additional training computation delivers a meaningful practical benefit.
Questions for authors
-
Could you clarify whether the training runs behind Tables 1–2 follow the update procedure specified in Appendix E, including coefficient refresh and solver fallback, and provide the segmentation rules and optimizer choice? I would also find it helpful to see measurements on a manageable subset of training steps showing whether the proxy protections hold in full-gradient geometry, after minibatch aggregation, and under the applied optimizer update. NCR and GRPO should be compared using the same cached batch and identical initial optimizer states.
-
Although the experiments cover multiple model sizes and two Qwen generations, I would like to understand whether the benefits extend beyond this family. Do you have results on another widely used open-weight model family, such as Llama? Given that sampling temperature can change the frequency and similarity of sampled responses, could you also evaluate temperature sensitivity using the existing checkpoints while keeping the prompts, top-p, response-length limits, and other evaluation settings fixed?
-
Could you provide a numerical comparison of Pass@2, Pass@4, Pass@8, and Pass@16 across the evaluated methods to clarify NCR’s gains at smaller sampling budgets? Figure 2 already presents Pass@k curves, but a table reporting these values for each model and benchmark, together with their macro-averages and differences relative to GRPO, would make the magnitude and consistency of the improvements easier to assess. I would find this particularly helpful for evaluating the method’s benefits when only a small number of responses can be generated.
-
Could you clarify the discrepancy between the positive-only ablation described in the text and the response-level refund variant reported in Table 2? Is a result available with negative-credit refunds disabled? Please also explain whether the runtime components in Table 3 overlap or are nested, and provide an end-to-end GRPO runtime comparison under the same hardware and workload.
-
What evidence supports the applicability of the test-gradient representation assumption at the Nash-induced radius? I would also appreciate clarification of the interpretation following Theorem 3 in light of Appendix I: a positive bound requires an alignment margin large enough to exceed the error terms, and the difference between two worst-case advantages over GRPO is not generally the worst-case direct advantage over linear routing.
The authors’ response has addressed my main concerns, and I will raise my score to 3 (weak accept).