acceptodds
Under review as a conference paper at ICLR 2027

PeerCredit: Cross-Prompt Token Credit Assignment for Reinforcement Learning with Verifiable Rewards

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves reasoning models using automatically checkable outcome rewards, but its response-level advantages do not specify how negative credit should be distributed across tokens in failed responses. Existing methods derive finer-grained signals mainly from rollouts of the same prompt, leaving cross-prompt reward information unused. We propose PeerCredit, which introduces cross-prompt reward information as an additional source of evidence for negative credit assignment. PeerCredit constructs reward-weighted peer gradients, measures token-level gradient alignment, removes variation explained by token probability, relative token position, and response length, and redistributes a fixed negative-credit budget so that tokens better aligned with reward-preferred peer gradients receive less penalty while the total budget is preserved. Across eight benchmarks spanning mathematical reasoning and tool use, PeerCredit achieves the best mathematical average at all three Qwen3 scales, with gains of up to 15.0 points over GRPO and 2.7 points over ResRL, while also leading on both tool-use benchmarks. Further analyses support cross-prompt reward evidence as a useful principle for improving fine-grained credit assignment in RLVR beyond within-prompt supervision.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning with Verifiable Rewards, Token-Level Credit Assignment, Large Language Model Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pd4eOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PeerCredit, a token-level credit assignment method for RLVR that leverages reward informantion across different prompts. Unlike conventional methods such as GRPO, which assign the same advantage to all tokens, PeerCredit redistributes negative credit more selectively within unsuccessful responses. The method constructs gradient directions from other prompts and measures their alignment with individual token gradients to identify tokens that should receive less negative credit. Experiments on several mathematical reasoning benchmarks using Qwen3 1.7B, 4B, and 8B show consistent improvements in average performance over baselines, including GRPO and ResRL. Further experiments on tool-use benchmarks support the effectiveness of PeerCredit.

Critical strengths
  1. PeerCredit can potentially complement existing RLVR optimizations.

    • Rather than introducing a new policy optimization algorithm, this paper provides an additional token-level credit assignment mechanism that reweights the advantages used by the underlying algorithm.
    • This design is appealing, as it could potentially be integrated with other group-based RLVR methods beyond GRPO, although such compatibility is not explictly evaluated in the paper.
  2. This paper introduces a promising perspective on leveraging cross-prompt information for credit assignment.

    • Exisitng RLVR methods primarily rely on reward information from multiple rollouts of the same prompt. But, PeerCredit introduces that, although different prompts induce different response distributions, their gradients lie in the same parameter space, allowing reward signals from one problem to inform token-level updates for another.
    • I find this direction promising, as it explores how information from multiple trajectories across different problems can be used to improve credit assignment.
Critical weaknesses

1. The reliability and availability of PeerCredit.

  • Although the peer-gradient estimater (HhH_h) is theoretically unbiased, its reliability with only four rollouts per prompt is unclear. I think sparse binary rewards may lead to high-variance gradient estimates, potentially affecting the quality of token-level credit assignment.
  • When all rollouts for a prompt receive the same reward, its peer gradient becomes zero. Such groups have zero advantages and cannot benefit from credit redistribution. While this limitation is partly inherited from GRPO, it would be helpful to understand how frequently these cases occur and how they affect PeerCredit's effectiveness.
  • The paper does not directly investigate the effect of the number of rollouts per prompt on the reliability of the estimated signals.

2. The robustness of the performance gains and their trade-off with computational overhead need further investigation.

  • Although PeerCredit achieves the best average performance across all three model sizes, its improvements over ResRL are relatively modest (+2.3, +2.7, and +0.6 for 1.7B, 4B, and 8B, respectively). The gain is particularly small at 8B, and the reported results also show noticeable variation across training seeds. These make it difficult to draw a clear conclusion about how consistently the improvements hold across model scales.
  • While PeerCredit can reuse existing rollouts, computing and storing token-level gradients and peer directions introduces additional computational and memory overhead. Given the relatively modest gains over strong baselines, it would be useful to evaluate its efficiency under matched wall-clock time or GPU-hour budgets.
Questions for authors
  1. The authors provide a reasonable computational motivation for using fianl transformer block's MLP down-projection. But, I wonder whether gradient alignment in this restricted paramter space reflects compatibility across the full model. Have the authors considered using different layers or parameter subsets?

  2. Since PeerCredit relies on reward differences among rollouts, I wonder whether its effectiveness varies with the number of successful responses within a group. With four rollouts, how does the method behave when a prompt produces 0, 1, 2, 3, or 4 successful responses?

  3. Would selecting more relevant peer prompt improve credit assignment? Would selecting semantically or structurally related problems lead to more informative gradient signals?

  4. PeerCredit measures compatibility between each token's gradient and gradient directions from other prompts that are estimated to increase rewards, rather than the semantic correctness of individual reasoning steps. Therefore, it remains unclear whether the assigned weights meaningfully distinguish useful reasoning from erroneous reasoning. If possible, an analysis using reasoning trajectories with annotated correct and incorrect steps could provide additional insight into what the credit assignment mechanism captures.

7 comments

We sincerely thank the reviewer for the thoughtful comments on estimator reliability, rollout efficiency, performance robustness, and the computational cost of PeerCredit. We appreciate the opportunity to further characterize the behavior of the peer-gradient estimator and the conditions under which PeerCredit provides useful cross-prompt learning signals. We conducted additional rollout-sensitivity experiments, analyzed reward-homogeneous groups, and expanded our multi-seed evaluations. Our responses follow.

W1a / W1c. Reliability of the peer-gradient estimator and the effect of rollout count

Sensitivity to independent sampling. We fixed the trained model, target question and response, and peer-question set. For each peer question, we independently generated two sets of responses and compared the resulting token weights using their root mean squared (RMS) difference.

Peer responses per prompt (GG)RMS difference
20.01899
40.02079
80.01924

The RMS differences remain within a similar range across all three rollout settings, characterizing the sampling sensitivity of the final token weights.

Online training with G=2G=2. We additionally evaluated Qwen3-4B with the original down-projection representation and two rollouts per prompt.

StepAIME24AIME25AMC23MATH-500MinervaOlympiadAvg@16
10033.9628.5473.4487.0041.3459.3053.93
20035.4232.0875.4787.6540.5361.5155.44

The G=2G=2 setting reaches 55.44% Avg@16 at step 200, demonstrating effective online learning with two rollouts per prompt. We will add G=8G=8 results within 2 days.

W1b. How frequently do reward-homogeneous groups occur, and what is their impact?

We analyzed the complete 200-step training log of Qwen3-4B, seed 1001. Each group consists of four responses sampled for one question in one training iteration. With 256 groups per step, the total is 51,200 groups.

StatisticCount
Total four-response groups51,200
Total responses204,800
Groups with four identical rewards33,034 (64.52%)
Responses in these groups132,136
Groups with different rewards18,166 (35.48%)
Responses in these groups72,664

Reward-homogeneous groups include both groups in which all four responses fail with identical rewards and groups in which all four succeed with identical rewards.

For these groups, the group-relative advantages are zero (A=0A=0), and reweighting preserves zero (wA=0wA=0). Consequently, they produce neither reward-driven policy updates nor nonzero peer-gradient directions.

The remaining 18,166 groups (35.48%) exhibit reward variation and can provide cross-prompt learning signals. Across the 200 training steps, 66–128 such groups occur per step, with an average of 90.83.

Thus, PeerCredit receives nonzero cross-prompt reward signals from a substantial portion of training groups, while applying token-level redistribution to eligible negative-advantage tokens.

W2. Robustness across model scales and computational overhead

Multi-seed mathematical reasoning results

We added two training seeds (800 and 900) to evaluate performance across model scales.

ModelPeerCredit (mean ± std.)SeedsResRL referenceGain (pp)
1.7B50.92 ± 1.20248.6+2.32
4B60.27 ± 1.76557.0+3.27
8B65.65 ± 0.99564.7+0.95

Qwen3-1.7B

SeedAIME24AIME25AMC23MATH-500MinervaOlympiadAvg@16
100128.9625.8373.7586.5436.8358.6851.77
100225.4225.6369.5385.3037.1657.3750.07
Mean ± std.27.19 ± 2.5025.73 ± 0.1471.64 ± 2.9885.92 ± 0.8837.00 ± 0.2358.03 ± 0.9350.92 ± 1.20

Qwen3-4B

SeedAIME24AIME25AMC23MATH-500MinervaOlympiadAvg@16
100145.4240.0090.4791.0642.8364.1662.32
100243.1338.5487.3483.5139.8459.4058.63
100342.5037.7186.4187.8037.8757.2458.25
800†44.5839.5889.5389.7541.7062.8461.33
900†44.1739.3888.9189.3841.2561.7660.80
Mean ± std.43.96 ± 1.1639.04 ± 0.9188.53 ± 1.6488.30 ± 2.9240.70 ± 1.9161.08 ± 2.7760.27 ± 1.76

Qwen3-8B

SeedAIME24AIME25AMC23MATH-500MinervaOlympiadAvg@16
100156.0445.4288.9193.1445.5970.3566.57
100253.3442.0887.6692.1544.3567.9264.58
100350.4237.9290.4793.8845.1369.7964.60
800†55.2144.1789.8493.6545.4770.1966.42
900†54.7943.7589.0693.5045.3870.1066.10
Mean ± std.53.96 ± 2.2142.67 ± 2.9189.19 ± 1.0693.26 ± 0.6845.18 ± 0.5069.67 ± 1.0065.65 ± 0.99

† denotes additional training seeds.

Multi-seed BFCL v3 results

We also added two BFCL v3 runs (seeds 1005 and 1006). Together with seeds 1001, 1003, and 1004, the five-run results are:

SeedMT OverallBaseMFMPLCST Non-LiveST LiveOverall
100141.0051.0044.5035.5033.0089.4678.7669.08
100340.3852.0043.0034.0032.5088.5879.8369.19
100438.8847.0042.0032.5034.0089.1280.3268.91
1005†40.7551.5043.5034.5033.5089.2580.0569.10
1006†40.8851.5044.0034.5033.5089.3380.1069.15
Mean ± std.40.38 ± 0.8750.60 ± 2.0443.40 ± 0.9634.20 ± 1.1033.30 ± 0.5789.15 ± 0.3479.81 ± 0.6169.09 ± 0.11

MT denotes Multi-Turn, MF Missing Functions, MP Missing Parameters, LC Long Context, and ST Single-Turn.

Across five seeds, PeerCredit achieves 69.09 ± 0.11% on BFCL v3 Overall.

W2. Computational overhead

We decomposed training time using identical six-batch inputs on eight H20 GPUs. All values are average seconds per batch group.

Training stageGRPOPeerCredit
Response generation283.78283.78
Old-policy probabilities42.4242.42
Reference-policy probabilities41.4341.43
Standard actor update159.47158.72
Gradient-factor extraction046.76
Peer computation and allocation043.97
Total527.10617.08

PeerCredit introduces 89.98 seconds of additional computation per six-batch unit, corresponding to a 17.07% increase over GRPO. The additional cost is concentrated in gradient-factor extraction and peer computation/allocation.

Question 1. Do different layers or parameter subsets provide informative gradient-alignment signals?

We selected the final-layer MLP down-projection based on its exact gradient factorization and computational efficiency. Since no cross-position attention follows this projection, its per-token gradient admits an exact outer-product factorization: ∇Ulog⁡π(yi∣si)=eiai⊤\nabla_U \log \pi(y_i\mid s_i)=e_i a_i^\top. Only two vectors need to be stored per token, rather than a full parameter-gradient matrix.

To examine the sensitivity to parameter selection, we compared four matrices within the final transformer block on 704 independent test questions, using the Qwen3-4B seed-1001 step-200 checkpoint.

Gradient parameter blockAccuracy (%)Down minus alternative (pp)95% CI
Final MLP down-projection71.45----
Final MLP up-projection71.22+0.23[-0.71, 1.24]
Final MLP gate-projection71.31+0.14[-0.76, 1.15]
Final attention output matrix71.20+0.25[-1.01, 1.65]
Equal-total uniform control70.63+0.82[0.12, 1.63]

The final down-projection achieves the highest accuracy among the four parameter subsets and improves over the equal-total uniform control by 0.82 percentage points.

These results demonstrate the effectiveness of the selected parameter subset, combining an exact per-token gradient representation with efficient computation and strong downstream performance.

Question 2. How does PeerCredit behave with different numbers of successful responses per group?

We analyzed 3,072 groups, each containing four rollouts, and categorized them by the number of successful responses.

Successful responsesNumber of groups
0823
1228
2213
3328
41,480

We further measured the overall penalty reduction and additional redistribution for each category:

Successful responsesPeerCredit penalty reductionResRL penalty reductionPeerCredit additional redistribution
0No eligible negative-token poolSame--
18.55%4.82%0.245%
28.55%8.28%0.234%
38.46%11.78%0.242%
4No eligible negative-token poolSame--

For groups containing both successful and unsuccessful responses, PeerCredit maintains an overall penalty reduction of approximately 8.5%, with approximately 0.24% of the original negative-credit mass redistributed across tokens.

Among the 1,480 groups with four successful responses, 156 still exhibit different actual rewards, allowing them to provide nonzero peer signals to other prompts.

Question 3. Do semantically related peer prompts improve credit assignment?

We compared semantic peer selection with uniform sampling while fixing the peer-set size at 128. Semantic selection uses cosine similarity between question representations extracted by the frozen initial model.

For Qwen3-4B, seed 1003, at step 100:

Peer selection strategyAIME24AIME25AMC23MATH-500MinervaOlympiadAvg@16
128 semantically related peers39.5835.0082.5079.0032.0552.3653.42
128 uniformly sampled peers41.0434.5874.0680.3534.8152.9952.97

Semantic peer selection improves Avg@16 by 0.44 percentage points under the same peer budget, supporting the benefit of incorporating prompt relevance into peer selection.

Reviewer_JkaQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a novel credit assignment method for RLVR that leverages reward information from responses across different prompt groups. The proposed method computes reward-weighted gradient directions and uses them to estimate token-level credit assignment under a constrained budget. Empirical results show that the proposed method achieves comparable performance to RL baselines.

Critical strengths
  1. The method is well motivated and addresses the critical issues of cross-prompt credit assignment and negative credit assignment, which are underexplored in the literature.

  2. The method is well designed, with proper regularization and well-motivated budget constraints.

Critical weaknesses
  1. Empirical results are relatively moderate compared to baselines. Math reasoning performance is pretty close to ResRL, and the same holds for the tool-use experiments on BFCL v3.

  2. I have a concern about whether the token level is the right granularity for reward-guided cross-prompt agreement. Since the peer support of each token is measured by its gradient alignment with reward-weighted directions from other prompts, it is possible that tokens that appear frequently across prompts, e.g., templates or filler sentences in the thinking process, have systematically higher alignment with peer directions regardless of the problem. The residualization removes variation explained by token probability, position, and response length, but not by token identity or frequency. If the reduced penalty mostly goes to such templated tokens, PeerCredit might decrease the diversity of responses across prompt groups. An analysis of which tokens receive the largest penalty reduction would help clarify this, and some form of semantic or span-level grouping of tokens may be needed to ensure that credits are not assigned to tokens with spurious patterns.

  3. Related to W1, it is unclear whether the improved performance comes from a learning signal that solely encourages longer responses. I recognize that PeerCredit residualizes the scores against log response length and relative position on the target side, but the peer gradients g_hj are summed over tokens without length normalization, so longer peer responses (including truncated ones) contribute more to the peer direction H_h. Figure 6 also shows that rollouts with PeerCredit are longer than those with GRPO and at the same level as ResRL, which achieves similar performance to PeerCredit across multiple benchmarks. Is it likely that the model mainly learns to produce longer responses rather than proper credit assignment? A length-controlled comparison or an ablation with length-normalized peer gradients would help address this concern.

  4. The token-level score gradient estimated from a single matrix of a single layer is not robust and seems arbitrary. There exist different low-rank estimation methods for gradient directions in the data attribution literature (e.g., [1] and [2]), and some of these works have shown that the choice of gradient estimation method matters for downstream performance. Therefore, additional ablations should be conducted to compare different gradient computation methods.

References [1] Wang et al., 2026. Detecting and Suppressing Reward Hacking with Gradient Fingerprints.

[2] Li et al., 2025. Where Did It Go Wrong? Attributing Undesirable LLM Behaviors via Representation Gradient Tracing

Questions for authors
  1. There is an asymmetric credit assignment in PeerCredit, but the paper lacks analysis of how much positive and negative credit is assigned to different tokens. How much compensation was made by the credit redistribution, and which tokens received increased credit?

  2. PeerCredit involves an extra forward pass compared with the GRPO baseline. A clearer decomposition of the training overhead should be included, in addition to the step time shown in Figure 6.

7 comments

We sincerely thank the reviewer for the thoughtful feedback on the empirical robustness and mechanism of PeerCredit. We particularly appreciate the concerns regarding seed variability, token-level credit allocation, response length, and gradient parameterization. These comments motivated additional multi-seed experiments and targeted analyses of the credit assignment mechanism. We provide our detailed responses below.

W1. Performance gains and robustness across seeds

We expanded the mathematical reasoning experiments with two additional seeds (800 and 900) and added two BFCL v3 seeds (1005 and 1006).

1.1 Multi-seed mathematical reasoning results

ModelPeerCredit (mean ± std.)SeedsResRL referenceGain (pp)
1.7B50.92 ± 1.20248.6+2.32
4B60.27 ± 1.76557.0+3.27
8B65.65 ± 0.99564.7+0.95

PeerCredit achieves higher multi-seed mean performance than the reported ResRL reference across all three model scales.

1.2 Complete per-seed results across six benchmarks

Qwen3-1.7B

Model / SeedAIME24AIME25AMC23MATH-500MinervaOlympiadAvg@16
1.7B / 100128.9625.8373.7586.5436.8358.6851.77
1.7B / 100225.4225.6369.5385.3037.1657.3750.07
Mean ± std.27.19 ± 2.5025.73 ± 0.1471.64 ± 2.9885.92 ± 0.8837.00 ± 0.2358.03 ± 0.9350.92 ± 1.20

Qwen3-4B

Model / SeedAIME24AIME25AMC23MATH-500MinervaOlympiadAvg@16
4B / 100145.4240.0090.4791.0642.8364.1662.32
4B / 100243.1338.5487.3483.5139.8459.4058.63
4B / 100342.5037.7186.4187.8037.8757.2458.25
4B / 800†44.5839.5889.5389.7541.7062.8461.33
4B / 900†44.1739.3888.9189.3841.2561.7660.80
Mean ± std.43.96 ± 1.1639.04 ± 0.9188.53 ± 1.6488.30 ± 2.9240.70 ± 1.9161.08 ± 2.7760.27 ± 1.76

Qwen3-8B

Model / SeedAIME24AIME25AMC23MATH-500MinervaOlympiadAvg@16
8B / 100156.0445.4288.9193.1445.5970.3566.57
8B / 100253.3442.0887.6692.1544.3567.9264.58
8B / 100350.4237.9290.4793.8845.1369.7964.60
8B / 800†55.2144.1789.8493.6545.4770.1966.42
8B / 900†54.7943.7589.0693.5045.3870.1066.10
Mean ± std.53.96 ± 2.2142.67 ± 2.9189.19 ± 1.0693.26 ± 0.6845.18 ± 0.5069.67 ± 1.0065.65 ± 0.99

† denotes additional training seeds.

1.3 Multi-seed BFCL v3 results

We additionally evaluated two BFCL v3 seeds (1005 and 1006). Together with seeds 1001, 1003, and 1004, the five-run results are:

SeedMT OverallBaseMFMPLCST Non-LiveST LiveOverall
100141.0051.0044.5035.5033.0089.4678.7669.08
100340.3852.0043.0034.0032.5088.5879.8369.19
100438.8847.0042.0032.5034.0089.1280.3268.91
1005†40.7551.5043.5034.5033.5089.2580.0569.10
1006†40.8851.5044.0034.5033.5089.3380.1069.15
Mean ± std.40.38 ± 0.8750.60 ± 2.0443.40 ± 0.9634.20 ± 1.1033.30 ± 0.5789.15 ± 0.3479.81 ± 0.6169.09 ± 0.11

MT denotes Multi-Turn, MF Missing Functions, MP Missing Parameters, LC Long Context, and ST Single-Turn. Across five seeds, BFCL v3 Overall reaches 69.09 ± 0.11%.

Together, these results demonstrate consistent performance across model scales and both mathematical reasoning and tool-use tasks.

W2. Do frequent tokens or templates receive disproportionate penalty reduction?

We analyze the additional penalty reduction from two perspectives: token frequency and semantic content categories.

2.1 Frequency-based analysis

We define the relative allocation ratio for a token category C as:

R(C)=additional-reduction shareoriginal negative-credit share×100%.R(C) = \frac{\text{additional-reduction share}}{\text{original negative-credit share}} \times 100\%.

A value of 100% indicates proportional allocation.

High-frequency token setRelative allocation ratio
Top 1%97.6%
Top 5%101.0%
Top 10%101.1%

All three values are close to 100%, indicating that frequent token types do not receive disproportionate additional penalty reduction.

2.2 Content-level allocation analysis

We further categorized failed responses generated by PeerCredit and ResRL. Both sets were analyzed using PeerCredit's credit allocation rule.

We report each category's original negative-credit share and its share of the additional penalty reduction.

Content categoryPC originalPC reductionResRL originalResRL reduction
Templates29.33%32.68%28.63%32.47%
Effective reasoning37.67%30.36%39.48%32.67%
Correct but non-progressing reasoning11.58%9.62%11.37%9.54%
Substantive errors8.38%7.51%8.49%8.12%
Error-dependent subsequent reasoning6.35%4.97%6.29%4.91%
Unverified content6.69%14.85%5.74%12.29%

Templates receive approximately one-third of the total additional penalty reduction, while the remaining approximately two-thirds is distributed across other content categories. Unverified content also receives a substantial share of the additional reduction.

These results characterize the actual allocation of PeerCredit: its additional penalty reduction is distributed across multiple semantic categories rather than being concentrated in frequent templates.

W3. Are PeerCredit's performance gains primarily due to longer responses?

We conducted complementary experiments during training and evaluation.

3.1 Length-normalized peer gradients during training

In the original PeerCredit, token gradients from each peer response are summed, potentially allowing longer responses to contribute more simply because they contain more tokens. We therefore replaced the sum with the average gradient per token:

Hhnorm=1Th∑j=1Thghj,H_h^{\mathrm{norm}} = \frac{1}{T_h}\sum_{j=1}^{T_h}g_{hj},

where ThT_h denotes the number of tokens in peer response hh.

For Qwen3-4B, seed 1003, at step 200:

ConfigurationAIME24AIME25AMC23MATH-500MinervaOlympiadAvg@16
Original PeerCredit40.4235.8385.7887.1040.7659.9358.30
Length-normalized peer gradients43.5433.7585.7885.5442.2660.1758.51

Length normalization increases Avg@16 from 58.30% to 58.51%, demonstrating that PeerCredit maintains its learning performance without unnormalized accumulation of gradients from longer peer responses.

3.2 Matched generation-length limits during evaluation

We further evaluated PeerCredit and ResRL under identical maximum generation lengths.

Maximum generation lengthMethodAvg@16 (%)Mean response lengthTruncation rate
4096PeerCredit57.811872.1015.54%
4096ResRL53.651640.4811.06%
8192PeerCredit62.322218.845.57%
8192ResRL57.011892.464.02%

At identical generation limits, PeerCredit improves Avg@16 by 4.16 and 5.31 percentage points at 4096 and 8192 tokens, respectively.

Together, the length-normalized training experiment and matched-budget evaluation demonstrate that PeerCredit maintains its learning advantage without relying on unnormalized peer-gradient accumulation.

W4. Comparison with alternative gradient estimation methods

We integrated two alternative gradient representations into PeerCredit:

  • GRIFT-style: inspired by Detecting and Suppressing Reward Hacking with Gradient Fingerprints (Wang et al., 2026).
  • RepT-style: inspired by Where Did It Go Wrong? Attributing Undesirable LLM Behaviors via Representation Gradient Tracing (Li et al., 2025).

We compared both with the original final-layer MLP down-projection on Qwen3-4B, seed 1001, with G=4.

Gradient representationStepAIME24AIME25AMC23MATH-500MinervaOlympiadAvg@16
Original down5035.6325.8373.9181.6637.4153.2151.27
GRIFT-style5027.7123.3366.7284.8939.4854.6749.47
RepT-style5029.5822.0865.1682.4538.1054.1548.59
Original down10045.0040.0084.3882.2537.5056.0157.52
GRIFT-style10038.5431.4681.5678.5137.9451.5753.26
RepT-style10045.4240.6384.5382.5437.1655.3257.60

At step 50, Original down outperforms GRIFT-style and RepT-style by 1.80 and 2.68 percentage points, respectively. At step 100, it outperforms GRIFT-style by 4.26 percentage points and achieves performance comparable to RepT-style.

Combined with its exact outer-product factorization and efficient storage of two vectors per token, the final-layer down-projection provides an effective and computationally efficient gradient representation for PeerCredit.

Reviewer_a3EaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PeerCredit, a token-level negative-credit assignment method for reinforcement learning with verifiable rewards. It constructs reward-weighted gradients from other prompts and measures their alignment with target-token gradients in the final MLP down-projection. After removing variation associated with token probability, position, and response length, the method redistributes a fixed prompt-level reference budget and uses the resulting weights to modify policy advantages. Experiments on mathematical reasoning and tool-use benchmarks report improvements over the compared baselines, with ablations examining the contributions of cross-prompt information, residualization, and token-level allocation.

Critical strengths

S1. An interesting use of cross-prompt reward information for credit assignment

The paper explores a meaningful source of supervision beyond trajectories from the same prompt. By relating negative token updates to reward-supported directions from other problems, PeerCredit provides an optimization-based perspective on which tokens in unsuccessful responses should receive less penalty. The budget-constrained formulation also explicitly distinguishes changing the allocation of negative credit from changing its reference total, without requiring additional token-level annotations or a process reward model.

S2. Broad evaluation with informative ablations

The evaluation covers six mathematical reasoning benchmarks at three Qwen3 model sizes and two tool-use benchmarks with Qwen3-8B. The reported mathematical averages improve across all three sizes. The ablations are also thoughtfully controlled: the no-peer variant retains the reference budget and positive-response scaling, while the response-level variant preserves each response’s allocated negative-credit amount. These comparisons provide useful evidence for the contributions of reward-conditioned peer information and within-response token allocation.

Critical weaknesses

W1. The selected gradient block would benefit from further validation

Section 4.1 and Appendix A.1 provide a reasonable computational rationale for choosing the final MLP down-projection, including exact outer-product factorization and a short differentiation path. However, exactness within this block does not by itself establish how well its alignment scores reflect gradient interactions during full-model optimization; the first-order interpretation in Appendix A.2 explicitly holds the remaining parameters fixed. The paper does not report comparisons across parameter blocks or layers. A small-scale sensitivity study, or an analysis of agreement with gradients over a broader parameter subset, would help clarify the robustness and representativeness of this central credit-assignment signal.

W2. The role of the residualization features could be better characterized

Residualization appears important, as its removal produces the largest drop among the ablations in Table 4. However, Appendix D.1 removes the probability, position, and length features together while retaining intercept-only centering. This establishes the value of the combined feature set but leaves their individual contributions unclear. A feature-group ablation or an analysis of how residualization changes token weights would help explain why these components should be removed and which effects account for the observed improvement.

Questions for authors

Please see Weaknesses.

4 comments

We sincerely thank the reviewer for the careful reading of our manuscript and the constructive suggestions on validating our gradient parameterization and characterizing the residualization features. These comments helped us identify opportunities to strengthen the empirical validation and clarify the methodological choices. In response, we conducted additional parameter-block sensitivity experiments and feature-group ablations. We address both concerns below.

W1. Representativeness of the selected gradient block

We agree that an exact gradient factorization within one parameter block does not, by itself, establish that its alignment scores fully represent gradient interactions across the entire model. We select the final-layer MLP down-projection based on its mathematical tractability and computational efficiency.

Our analysis uses the Qwen3-4B model, including the initial model and the seed-1001 step-200 checkpoint (six-benchmark Avg@16: 62.32%). Since no cross-position attention follows the final MLP down-projection, the per-token gradient with respect to the down-projection weight matrix UU admits an exact outer-product factorization, i.e., ∇Ulog⁡π(yi∣si)=eiai⊤\nabla_U \log \pi(y_i \mid s_i) = e_i a_i^\top, where eie_i and aia_i denote the output-side gradient vector and input activation, respectively.

Thus, only two vectors need to be stored per token, rather than its full parameter-gradient matrix. Computing this gradient also requires backpropagation only through the final RMSNorm and output layer. For earlier blocks, subsequent attention operations introduce cross-position dependencies, making exact per-token gradient computation more involved.

To assess sensitivity to the parameterization, we compared credit signals derived from different matrices in the final transformer block and evaluated them on 704 independent test questions:

Credit-signal parameter blockAccuracy (%)Down minus alternative (pp)95% CI
Final MLP down-projection71.45----
Final MLP up-projection71.22+0.23[-0.71, 1.24]
Final MLP gate-projection71.31+0.14[-0.76, 1.15]
Final attention output matrix71.20+0.25[-1.01, 1.65]

The down-projection achieves the highest accuracy among the four parameter blocks. The differences between down-projection and the other three matrices are relatively small, with confidence intervals containing zero.

These results support the effectiveness of the selected parameterization, we will incorporate this sensitivity analysis and clarify the scope of the gradient approximation in the revised manuscript.

W2. Individual contributions of residualization features

We agree that jointly removing the probability, position, and response-length features does not reveal their individual contributions. We therefore conducted one-at-a-time feature-group ablations using Qwen3-4B, seed 1001, at step 200. Each variant removes only the specified feature group while retaining the other residualization features. All experiments use the same training configuration.

ConfigurationAIME24AIME25AMC23MATH-500MinervaOlympiadAvg@16
Full method45.4240.0090.4791.0642.8364.1662.32
Remove probability and squared-probability features44.3840.2185.7875.4030.5649.0454.23
Remove relative-position feature43.7536.6785.7868.3428.1044.3451.16
Remove log-response-length feature48.9642.5088.2869.1426.6345.9753.58

Removing the probability-related, relative-position, and log-response-length features reduces Avg@16 by 8.09, 11.16, and 8.74 percentage points, respectively. The relative-position feature has the largest individual effect, while the probability and response-length features also make substantial contributions.

These results demonstrate that all three feature groups contribute to the effectiveness of residualization, with complementary effects rather than a single feature accounting for the observed improvement. We will include these individual ablations in the revised Appendix D.1.

Dear Reviewer, I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work. Thank you for your time and effort in reviewing our paper!

Thank you for your response, which has addressed my concerns. I will maintain my current positive score.

Thank you for your positive feedback. We appreciate your constructive comments, which have helped us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.