- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PeerCredit, a token-level credit assignment method for RLVR that leverages reward informantion across different prompts. Unlike conventional methods such as GRPO, which assign the same advantage to all tokens, PeerCredit redistributes negative credit more selectively within unsuccessful responses. The method constructs gradient directions from other prompts and measures their alignment with individual token gradients to identify tokens that should receive less negative credit. Experiments on several mathematical reasoning benchmarks using Qwen3 1.7B, 4B, and 8B show consistent improvements in average performance over baselines, including GRPO and ResRL. Further experiments on tool-use benchmarks support the effectiveness of PeerCredit.
Critical strengths
-
PeerCredit can potentially complement existing RLVR optimizations.
- Rather than introducing a new policy optimization algorithm, this paper provides an additional token-level credit assignment mechanism that reweights the advantages used by the underlying algorithm.
- This design is appealing, as it could potentially be integrated with other group-based RLVR methods beyond GRPO, although such compatibility is not explictly evaluated in the paper.
-
This paper introduces a promising perspective on leveraging cross-prompt information for credit assignment.
- Exisitng RLVR methods primarily rely on reward information from multiple rollouts of the same prompt. But, PeerCredit introduces that, although different prompts induce different response distributions, their gradients lie in the same parameter space, allowing reward signals from one problem to inform token-level updates for another.
- I find this direction promising, as it explores how information from multiple trajectories across different problems can be used to improve credit assignment.
Critical weaknesses
1. The reliability and availability of PeerCredit.
- Although the peer-gradient estimater () is theoretically unbiased, its reliability with only four rollouts per prompt is unclear. I think sparse binary rewards may lead to high-variance gradient estimates, potentially affecting the quality of token-level credit assignment.
- When all rollouts for a prompt receive the same reward, its peer gradient becomes zero. Such groups have zero advantages and cannot benefit from credit redistribution. While this limitation is partly inherited from GRPO, it would be helpful to understand how frequently these cases occur and how they affect PeerCredit's effectiveness.
- The paper does not directly investigate the effect of the number of rollouts per prompt on the reliability of the estimated signals.
2. The robustness of the performance gains and their trade-off with computational overhead need further investigation.
- Although PeerCredit achieves the best average performance across all three model sizes, its improvements over ResRL are relatively modest (+2.3, +2.7, and +0.6 for 1.7B, 4B, and 8B, respectively). The gain is particularly small at 8B, and the reported results also show noticeable variation across training seeds. These make it difficult to draw a clear conclusion about how consistently the improvements hold across model scales.
- While PeerCredit can reuse existing rollouts, computing and storing token-level gradients and peer directions introduces additional computational and memory overhead. Given the relatively modest gains over strong baselines, it would be useful to evaluate its efficiency under matched wall-clock time or GPU-hour budgets.
Questions for authors
-
The authors provide a reasonable computational motivation for using fianl transformer block's MLP down-projection. But, I wonder whether gradient alignment in this restricted paramter space reflects compatibility across the full model. Have the authors considered using different layers or parameter subsets?
-
Since PeerCredit relies on reward differences among rollouts, I wonder whether its effectiveness varies with the number of successful responses within a group. With four rollouts, how does the method behave when a prompt produces 0, 1, 2, 3, or 4 successful responses?
-
Would selecting more relevant peer prompt improve credit assignment? Would selecting semantically or structurally related problems lead to more informative gradient signals?
-
PeerCredit measures compatibility between each token's gradient and gradient directions from other prompts that are estimated to increase rewards, rather than the semantic correctness of individual reasoning steps. Therefore, it remains unclear whether the assigned weights meaningfully distinguish useful reasoning from erroneous reasoning. If possible, an analysis using reasoning trajectories with annotated correct and incorrect steps could provide additional insight into what the credit assignment mechanism captures.
7 comments
W2. Robustness across model scales and computational overhead
Multi-seed mathematical reasoning results
We added two training seeds (800 and 900) to evaluate performance across model scales.
| Model | PeerCredit (mean ± std.) | Seeds | ResRL reference | Gain (pp) |
|---|---|---|---|---|
| 1.7B | 50.92 ± 1.20 | 2 | 48.6 | +2.32 |
| 4B | 60.27 ± 1.76 | 5 | 57.0 | +3.27 |
| 8B | 65.65 ± 0.99 | 5 | 64.7 | +0.95 |
Qwen3-1.7B
| Seed | AIME24 | AIME25 | AMC23 | MATH-500 | Minerva | Olympiad | Avg@16 |
|---|---|---|---|---|---|---|---|
| 1001 | 28.96 | 25.83 | 73.75 | 86.54 | 36.83 | 58.68 | 51.77 |
| 1002 | 25.42 | 25.63 | 69.53 | 85.30 | 37.16 | 57.37 | 50.07 |
| Mean ± std. | 27.19 ± 2.50 | 25.73 ± 0.14 | 71.64 ± 2.98 | 85.92 ± 0.88 | 37.00 ± 0.23 | 58.03 ± 0.93 | 50.92 ± 1.20 |
Qwen3-4B
| Seed | AIME24 | AIME25 | AMC23 | MATH-500 | Minerva | Olympiad | Avg@16 |
|---|---|---|---|---|---|---|---|
| 1001 | 45.42 | 40.00 | 90.47 | 91.06 | 42.83 | 64.16 | 62.32 |
| 1002 | 43.13 | 38.54 | 87.34 | 83.51 | 39.84 | 59.40 | 58.63 |
| 1003 | 42.50 | 37.71 | 86.41 | 87.80 | 37.87 | 57.24 | 58.25 |
| 800† | 44.58 | 39.58 | 89.53 | 89.75 | 41.70 | 62.84 | 61.33 |
| 900† | 44.17 | 39.38 | 88.91 | 89.38 | 41.25 | 61.76 | 60.80 |
| Mean ± std. | 43.96 ± 1.16 | 39.04 ± 0.91 | 88.53 ± 1.64 | 88.30 ± 2.92 | 40.70 ± 1.91 | 61.08 ± 2.77 | 60.27 ± 1.76 |
Qwen3-8B
| Seed | AIME24 | AIME25 | AMC23 | MATH-500 | Minerva | Olympiad | Avg@16 |
|---|---|---|---|---|---|---|---|
| 1001 | 56.04 | 45.42 | 88.91 | 93.14 | 45.59 | 70.35 | 66.57 |
| 1002 | 53.34 | 42.08 | 87.66 | 92.15 | 44.35 | 67.92 | 64.58 |
| 1003 | 50.42 | 37.92 | 90.47 | 93.88 | 45.13 | 69.79 | 64.60 |
| 800† | 55.21 | 44.17 | 89.84 | 93.65 | 45.47 | 70.19 | 66.42 |
| 900† | 54.79 | 43.75 | 89.06 | 93.50 | 45.38 | 70.10 | 66.10 |
| Mean ± std. | 53.96 ± 2.21 | 42.67 ± 2.91 | 89.19 ± 1.06 | 93.26 ± 0.68 | 45.18 ± 0.50 | 69.67 ± 1.00 | 65.65 ± 0.99 |
† denotes additional training seeds.
Multi-seed BFCL v3 results
We also added two BFCL v3 runs (seeds 1005 and 1006). Together with seeds 1001, 1003, and 1004, the five-run results are:
| Seed | MT Overall | Base | MF | MP | LC | ST Non-Live | ST Live | Overall |
|---|---|---|---|---|---|---|---|---|
| 1001 | 41.00 | 51.00 | 44.50 | 35.50 | 33.00 | 89.46 | 78.76 | 69.08 |
| 1003 | 40.38 | 52.00 | 43.00 | 34.00 | 32.50 | 88.58 | 79.83 | 69.19 |
| 1004 | 38.88 | 47.00 | 42.00 | 32.50 | 34.00 | 89.12 | 80.32 | 68.91 |
| 1005† | 40.75 | 51.50 | 43.50 | 34.50 | 33.50 | 89.25 | 80.05 | 69.10 |
| 1006† | 40.88 | 51.50 | 44.00 | 34.50 | 33.50 | 89.33 | 80.10 | 69.15 |
| Mean ± std. | 40.38 ± 0.87 | 50.60 ± 2.04 | 43.40 ± 0.96 | 34.20 ± 1.10 | 33.30 ± 0.57 | 89.15 ± 0.34 | 79.81 ± 0.61 | 69.09 ± 0.11 |
MT denotes Multi-Turn, MF Missing Functions, MP Missing Parameters, LC Long Context, and ST Single-Turn.
Across five seeds, PeerCredit achieves 69.09 ± 0.11% on BFCL v3 Overall.
W2. Computational overhead
We decomposed training time using identical six-batch inputs on eight H20 GPUs. All values are average seconds per batch group.
| Training stage | GRPO | PeerCredit |
|---|---|---|
| Response generation | 283.78 | 283.78 |
| Old-policy probabilities | 42.42 | 42.42 |
| Reference-policy probabilities | 41.43 | 41.43 |
| Standard actor update | 159.47 | 158.72 |
| Gradient-factor extraction | 0 | 46.76 |
| Peer computation and allocation | 0 | 43.97 |
| Total | 527.10 | 617.08 |
PeerCredit introduces 89.98 seconds of additional computation per six-batch unit, corresponding to a 17.07% increase over GRPO. The additional cost is concentrated in gradient-factor extraction and peer computation/allocation.
Question 1. Do different layers or parameter subsets provide informative gradient-alignment signals?
We selected the final-layer MLP down-projection based on its exact gradient factorization and computational efficiency. Since no cross-position attention follows this projection, its per-token gradient admits an exact outer-product factorization: . Only two vectors need to be stored per token, rather than a full parameter-gradient matrix.
To examine the sensitivity to parameter selection, we compared four matrices within the final transformer block on 704 independent test questions, using the Qwen3-4B seed-1001 step-200 checkpoint.
| Gradient parameter block | Accuracy (%) | Down minus alternative (pp) | 95% CI |
|---|---|---|---|
| Final MLP down-projection | 71.45 | -- | -- |
| Final MLP up-projection | 71.22 | +0.23 | [-0.71, 1.24] |
| Final MLP gate-projection | 71.31 | +0.14 | [-0.76, 1.15] |
| Final attention output matrix | 71.20 | +0.25 | [-1.01, 1.65] |
| Equal-total uniform control | 70.63 | +0.82 | [0.12, 1.63] |
The final down-projection achieves the highest accuracy among the four parameter subsets and improves over the equal-total uniform control by 0.82 percentage points.
These results demonstrate the effectiveness of the selected parameter subset, combining an exact per-token gradient representation with efficient computation and strong downstream performance.
Question 2. How does PeerCredit behave with different numbers of successful responses per group?
We analyzed 3,072 groups, each containing four rollouts, and categorized them by the number of successful responses.
| Successful responses | Number of groups |
|---|---|
| 0 | 823 |
| 1 | 228 |
| 2 | 213 |
| 3 | 328 |
| 4 | 1,480 |
We further measured the overall penalty reduction and additional redistribution for each category:
| Successful responses | PeerCredit penalty reduction | ResRL penalty reduction | PeerCredit additional redistribution |
|---|---|---|---|
| 0 | No eligible negative-token pool | Same | -- |
| 1 | 8.55% | 4.82% | 0.245% |
| 2 | 8.55% | 8.28% | 0.234% |
| 3 | 8.46% | 11.78% | 0.242% |
| 4 | No eligible negative-token pool | Same | -- |
For groups containing both successful and unsuccessful responses, PeerCredit maintains an overall penalty reduction of approximately 8.5%, with approximately 0.24% of the original negative-credit mass redistributed across tokens.
Among the 1,480 groups with four successful responses, 156 still exhibit different actual rewards, allowing them to provide nonzero peer signals to other prompts.
Question 3. Do semantically related peer prompts improve credit assignment?
We compared semantic peer selection with uniform sampling while fixing the peer-set size at 128. Semantic selection uses cosine similarity between question representations extracted by the frozen initial model.
For Qwen3-4B, seed 1003, at step 100:
| Peer selection strategy | AIME24 | AIME25 | AMC23 | MATH-500 | Minerva | Olympiad | Avg@16 |
|---|---|---|---|---|---|---|---|
| 128 semantically related peers | 39.58 | 35.00 | 82.50 | 79.00 | 32.05 | 52.36 | 53.42 |
| 128 uniformly sampled peers | 41.04 | 34.58 | 74.06 | 80.35 | 34.81 | 52.99 | 52.97 |
Semantic peer selection improves Avg@16 by 0.44 percentage points under the same peer budget, supporting the benefit of incorporating prompt relevance into peer selection.
We sincerely thank the reviewer for the thoughtful comments on estimator reliability, rollout efficiency, performance robustness, and the computational cost of PeerCredit. We appreciate the opportunity to further characterize the behavior of the peer-gradient estimator and the conditions under which PeerCredit provides useful cross-prompt learning signals. We conducted additional rollout-sensitivity experiments, analyzed reward-homogeneous groups, and expanded our multi-seed evaluations. Our responses follow.
W1a / W1c. Reliability of the peer-gradient estimator and the effect of rollout count
Sensitivity to independent sampling. We fixed the trained model, target question and response, and peer-question set. For each peer question, we independently generated two sets of responses and compared the resulting token weights using their root mean squared (RMS) difference.
The RMS differences remain within a similar range across all three rollout settings, characterizing the sampling sensitivity of the final token weights.
Online training with G=2. We additionally evaluated Qwen3-4B with the original down-projection representation and two rollouts per prompt.
The G=2 setting reaches 55.44% Avg@16 at step 200, demonstrating effective online learning with two rollouts per prompt. We will add G=8 results within 2 days.
W1b. How frequently do reward-homogeneous groups occur, and what is their impact?
We analyzed the complete 200-step training log of Qwen3-4B, seed 1001. Each group consists of four responses sampled for one question in one training iteration. With 256 groups per step, the total is 51,200 groups.
Reward-homogeneous groups include both groups in which all four responses fail with identical rewards and groups in which all four succeed with identical rewards.
For these groups, the group-relative advantages are zero (A=0), and reweighting preserves zero (wA=0). Consequently, they produce neither reward-driven policy updates nor nonzero peer-gradient directions.
The remaining 18,166 groups (35.48%) exhibit reward variation and can provide cross-prompt learning signals. Across the 200 training steps, 66–128 such groups occur per step, with an average of 90.83.
Thus, PeerCredit receives nonzero cross-prompt reward signals from a substantial portion of training groups, while applying token-level redistribution to eligible negative-advantage tokens.