acceptodds
Under review as a conference paper at ICLR 2027

Contextual Credit Diffusion for LLM Post-training

Abstract

LLM post-training has long relied on a single objective at each position: optimizing a weighted log-likelihood of a single reference token in the vocabulary. This reference token is either the sampled action in RL methods such as PPO and GRPO, or the ground-truth label in SFT. The loss can be written in the unified form (L = -E_t [ w_t cdot log pi_ theta(a_t^ref mid s_t) ] ), where (w_t ) is an advantage in RL methods or a constant in SFT. However, solving a reasoning problem often admits multiple valid paths that all lead to the correct answer, yet the forced-reference objective locks the model onto a single path at each step and penalizes alternative tokens in the vocabulary that could also lead to a valid answer. We propose **Contextual Credit Diffusion (CCD)**. CCD diffuses the reference token’s credit across the vocabulary according to their contextual interchangeability, measured by the logit difference in the model’s next-token distribution under the current context, replacing the single-reference constraint with a dense reward. This offers a potential solution to both the credit assignment problem in RL and the one-hot label problem in SFT. CCD consistently outperforms RL and SFT baselines on six reasoning benchmarks, achieving significant gains. Furthermore, a token-replacement experiment shows that replacing the reference token with candidate tokens whose diffused credit is high yields the same final answer in most cases, confirming that the reward captures contextual interchangeability and identifies alternative valid paths.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Reinforcement Learning, Supervised Fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DuzBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new loss function applicable to both SFT and RL. The CCD coefficient is defined based on the logit difference between two tokens coupled with exponential decay. By incorporating CCD with probability weighting, the optimization shifts from being anchored to a single reference token to jointly optimizing a group of tokens with similar logit values. This approach only modifies the loss function, requires no teacher model, and incurs negligible computational overhead. Experiments use one training dataset for SFT and one for RL, and evaluate several Qwen3 model sizes on six mathematical reasoning benchmarks.. Results demonstrate that the proposed CCD algorithm yields performance gains across different configurations on both experimental datasets.

Critical strengths

The proposed CCD loss indeed provides a possible solution. Since the method does not substantially increase computational overhead, it is also practically feasible from an engineering perspective.

Critical weaknesses
  1. While the paper attempts to analyze the CCD loss and explicate its design rationale, the existing evidence does not sufficiently establish the mechanism. Critical issues remain largely undiscussed: how the CCD loss reshapes the probability distribution of non-reference tokens, and why such distributional changes can mathematically account for performance improvements.

  2. The discussion of the “single reference” limitation in RL requires more explanation. When expectation is taken over sampled actions (in policy gradient), the resulting gradient incorporates the advantage value of every possible action. In multi trajectory training regimes, different actions conditioned on the same context can be sampled independently, each receiving direct training signals derived from its respective subsequent return.

  3. The paper’s token replacement experiments lack detailed specifications. The exact procedure for selecting candidate tokens remains ambiguous, and tokens sampled from different positions may yield markedly different impacts on model outputs. A thorough account of the experimental implementation is needed. Additionally, experiments in the study show that the CCD coefficients of most tokens are small. However, if the CCD coefficients of the topk tokens selected by predicted probability are small, then it is likely that the set of tokens optimized by this loss is not semantically interchangeable.

Questions for authors
  1. Under what conditions can a non-reference candidate be promoted or suppressed? And a small controlled experiment containing valid alternatives and invalid tokens with similar logits would help.
  2. Since standard RL can already obtain direct feedback for different actions under the same prefix through repeated sampling, which remaining limitation is CCD intended to address?
  3. Could you compare the full CCD loss with reference-only probability weighting and with diffusion using the same proximity coefficients but without probability weighting?
  4. How are questions or trajectories, token positions, and candidate tokens selected in the replacement experiment?
4 comments
AuthorsOpenReview ↗

Part I: Response to Q1

We sincerely thank the reviewer for the positive assessment. It encourages us a lot. We hope our responses below address your concerns and questions.

Q1: Promotion/suppression conditions and controlled experiment.

(1) Under what conditions can a non-reference candidate be promoted or suppressed?

We thank the reviewer for this question. We provide a detailed derivation of how CCD promotes or suppresses candidate tokens. For a given context, let the probability of token aa be defined by the softmax function:

pa=exp⁡(za)∑jexp⁡(zj).p_a=\frac{\exp(z_a)}{\sum_j\exp(z_j)}.

The derivative of its log-probability with respect to the logit of token jj is:

∂log⁡pa∂zj=I[a=j]−pj.\frac{\partial\log p_a}{\partial z_j}=\mathbb{I}[a=j]-p_j.

For a single token position, the CCD surrogate loss is:

L=−At∑a∈Tsg⁡[paϕa]log⁡pa.\mathcal{L}=-A_t\sum_{a\in\mathcal{T}}\operatorname{sg}[p_a\phi_a]\log p_a.

Here, AtA_t denotes the advantage in RL and is set to 1 in SFT. Let wa=paϕaw_a=p_a\phi_a and W=∑a∈TwaW=\sum_{a\in\mathcal{T}}w_a. Since the weighting coefficients are detached during optimization, they are treated as constants when computing gradients. Applying the softmax derivative yields:

∂L∂zj=−At∑a∈Twa(I[a=j]−pj).\frac{\partial\mathcal{L}}{\partial z_j}=-A_t\sum_{a\in\mathcal{T}}w_a(\mathbb{I}[a=j]-p_j).

For a selected candidate token j∈Tj\in\mathcal{T}, this simplifies to:

∂L∂zj=−At(wj−pjW)=Atpj(W−ϕj).\frac{\partial\mathcal{L}}{\partial z_j}=-A_t(w_j-p_jW)=A_t p_j(W-\phi_j).

Therefore, under gradient descent with learning rate η>0\eta>0, the local logit update is:

Δzj=−η∂L∂zj=ηAtpj(ϕj−W).\Delta z_j=-\eta\frac{\partial\mathcal{L}}{\partial z_j}=\eta A_t p_j(\phi_j-W).

Since pj>0p_j>0, the update direction depends on the advantage sign and the candidate's proximity coefficient relative to the total CCD weight:

  • Positive advantage (including SFT, where the coefficient is 1): A candidate is promoted when ϕj>W\phi_j>W and suppressed when ϕj<W\phi_j<W.
  • Negative advantage (RL): The direction reverses: a candidate is suppressed when ϕj>W\phi_j>W and promoted when ϕj<W\phi_j<W.

Thus, CCD does not indiscriminately promote all non-reference candidates. Its update direction depends jointly on the signed advantage and the candidate's contextual predictive proximity relative to the aggregate CCD weight.

Why can CCD improve task performance?

The key distinction is that maximizing the probability of a single reference token is not necessarily equivalent to maximizing the probability of producing a correct final answer, since multiple token choices may lead to successful reasoning trajectories. For positive advantages, the CCD logit update can be expressed as a probability redistribution toward a proximity-weighted soft target. When predictive proximity is positively associated with the downstream success probabilities of candidate actions, this redistribution favors alternatives that can also lead to correct outcomes, rather than concentrating supervision exclusively on the sampled reference token. Thus, CCD can improve task performance by aligning token-level credit allocation more closely with the multiplicity of successful reasoning paths.

We acknowledge that providing a rigorous mathematical explanation for why this mechanism improves downstream performance is challenging, particularly given the complexity of LLM optimization. Our design is motivated by the intuition that multiple alternative tokens can lead to successful reasoning paths, while the token-replacement experiment and component ablations provide empirical support for its effectiveness.

(2) Controlled experiment with valid and invalid alternatives.

We thank the reviewer for this suggestion. However, formally defining valid and invalid alternative tokens is challenging, as it is difficult to precisely determine whether two tokens are semantically similar under a given context.

If we define a candidate as valid when its continuation produces the same final answer as the continuation from the reference token, and invalid otherwise, we believe our existing token-replacement experiment already provides large-scale empirical evidence addressing this concern.

Specifically, the reference token is itself a token from the same vocabulary, and the CCD coefficient directly measures a candidate's predictive proximity to this reference through their logit difference. Therefore, candidates with higher CCD coefficients have logits closer to that of the reference token. In Figure 3, we group candidates into coefficient bins of width 0.1 and compare the final answers obtained after independently continuing from the reference and candidate tokens. The results show that higher CCD coefficients are associated with higher final-answer agreement, providing empirical support for the relationship between predictive proximity and outcome preservation across 100,000 token replacements.

We have further clarified the experimental setup in Section 5.3 of the revised PDF and in our response to Q4.

AuthorsOpenReview ↗

Part II: Response to Q2&W2

Q2 & W2: Single-reference limitation in LLM RL.

We thank the reviewer for this observation. In practical LLM RL, policy-gradient methods use Monte Carlo estimates based on sampled trajectories. At each visited state, the gradient contribution is computed from the sampled action and its estimated advantage:

gt=A^t∇θlog⁡πθ(at∣st).g_t=\hat{A}_t\nabla_\theta\log\pi_\theta(a_t\mid s_t).

Under standard unbiased-estimation assumptions, the expectation of this sampled gradient is equivalent to the full-action policy gradient:

E[gt∣st]=∑aπθ(a∣st)A(st,a)∇θlog⁡πθ(a∣st).\mathbb{E}[g_t\mid s_t]=\sum_a\pi_\theta(a\mid s_t)A(s_t,a)\nabla_\theta\log\pi_\theta(a\mid s_t).

However, this equivalence holds in expectation. It does not imply that the advantages of all actions are available or explicitly evaluated during training. In autoregressive LLM reasoning, the state at step tt consists of the prompt and all previously sampled tokens:

st=(q,a1,…,at−1).s_t=(q,a_1,\ldots,a_{t-1}).

Crucially, unlike the shared initial prompt, intermediate prefixes are almost never revisited across independently sampled reasoning trajectories. This is a key distinction between LLM RL and conventional RL settings with frequently revisited states: in autoregressive generation, each state contains the entire previously generated token sequence, so two trajectories must generate exactly the same prefix to revisit the same state. For reasoning trajectories, the probability of such an exact match becomes vanishingly small. Consequently, most visited intermediate state effectively receives direct outcome-based supervision for only one sampled action, even when multiple trajectories are generated from the same prompt.

Moreover, repeatedly sampling alternative actions at every intermediate prefix is computationally prohibitive in LLM RL. Outcome-based rewards are typically obtained only after generating and verifying the complete response. Therefore, evaluating an alternative action at a given prefix requires generating its subsequent continuation and observing the final outcome, rather than evaluating the token in isolation.

For a response of length TT, sampling KK alternative actions at each prefix requires O(KT)O(KT) additional continuation rollouts, corresponding to up to O(KT2)O(KT^2) generated tokens. Such computational overhead is unacceptable.

Instead, CCD uses the model's existing next-token distribution to propagate the observed trajectory advantage to multiple candidate tokens:

gtCCD=A^t∑apaϕa∇θlog⁡pa.g_t^{\mathrm{CCD}}=\hat{A}_t\sum_a p_a\phi_a\nabla_\theta\log p_a.

This introduces negligible additional computational overhead without requiring any extra rollouts. Alternatively, CCD can be viewed as assigning each candidate token a proximity-weighted approximate advantage, A~(s,a)=ϕ(a,s)A^t\tilde{A}(s,a)=\phi(a,s)\hat{A}_t, where candidates closer to the sampled reference token receive an advantage closer to that of the reference token. This enables credit sharing across alternative actions using feedback from a single sampled trajectory.

AuthorsOpenReview ↗

Part III: Response to Q3

Q3: Ablation of Probability Weighting and Credit Diffusion.

We thank the reviewer for this important suggestion. We agree that CCD introduces two components: reference-token probability weighting and proximity-based credit diffusion. To disentangle their contributions, we conduct additional controlled ablations under the Qwen3-4B-Base + GRPO setting, including reference-only probability weighting (DFT-RL), diffusion without proximity weighting (Uniform), diffusion without probability weighting (w/o Prob), and reversed proximity weighting (Inverted).

Experimental Setup. All CCD variants share the same training configuration, candidate selection procedure, and group-relative advantage estimation. At each token position, let ata_t denote the sampled reference token, pap_a the student's full-vocabulary softmax probability, and StS_t the top-10 candidate set, which always includes the reference token. We define the CCD proximity coefficient as:

ϕa=exp⁡(−∣za−zat∣τ),τ=5.\phi_a=\exp\left(-\frac{|z_a-z_{a_t}|}{\tau}\right),\qquad\tau=5.

All variants follow the same surrogate loss:

L=−At∑a∈Stsg⁡[wa]log⁡pa.\mathcal{L}=-A_t\sum_{a\in S_t}\operatorname{sg}[w_a]\log p_a.

Here, AtA_t is the GRPO group-relative advantage, and sg⁡\operatorname{sg} denotes stop-gradient. The variants differ only in their definitions of the candidate weights waw_a:

1. Full CCD (probability weighting + proximity diffusion).

wa=paϕa.w_a=p_a\phi_a.

The weight incorporates both the candidate's predicted probability and its predictive proximity to the sampled reference token.

2. DFT-RL (reference-only probability weighting).

wa={pa,a=at,0,a≠at.w_a=\begin{cases}p_a,&a=a_t,\\\\0,&a\neq a_t.\end{cases}

This retains probability weighting for the reference token while removing all non-reference credit diffusion. It directly tests whether CCD's gains can be explained by reference-only probability weighting.

3. Uniform (diffusion without proximity weighting).

wa=pa.w_a=p_a.

This sets the proximity coefficient to 1 for all selected candidates while retaining probability weighting. Note that the candidate weights are still their original full-vocabulary probabilities, rather than equal weights.

4. w/o Prob (proximity diffusion without probability weighting).

wa=ϕa.w_a=\phi_a.

This preserves the same candidate set and proximity coefficients as CCD but removes the multiplicative probability weighting, directly addressing the reviewer's suggested ablation.

5. Inverted (reversed proximity weighting).

wa=paψa,ψa={1,a=at,1−ϕa,a≠at.w_a=p_a\psi_a,\qquad\psi_a=\begin{cases}1,&a=a_t,\\\\1-\phi_a,&a\neq a_t.\end{cases}

This preserves the reference-token weight while assigning larger proximity factors to non-reference candidates farther from the reference token.

Results. We evaluate all variants on six mathematical reasoning benchmarks under the same Qwen3-4B-Base training setting.

MethodAIME24AIME25AMCMATH500MinervaOlympiadBenchAvg.
Initial Model9.36.136.660.614.827.125.8
GRPO10.613.140.374.217.230.030.9
GRPO + DFT-RL13.010.040.274.413.233.330.7
GRPO + Uniform6.76.732.748.23.718.619.4
GRPO + w/o Prob7.10.036.957.96.325.722.3
GRPO + Inverted6.76.731.646.93.016.318.5
GRPO + CCD11.415.245.375.221.134.433.8

The results show that all ablation variants except DFT-RL suffer from training collapse, with Uniform, w/o Prob, and Inverted performing substantially worse than the initial model. Although DFT-RL avoids collapse, it still underperforms standard GRPO (30.7 vs. 30.9). We believe these results demonstrate that both probability weighting and the CCD proximity coefficient are essential for effective training.

We have included the complete ablation study in Appendix C of the revised PDF.

AuthorsOpenReview ↗

Part IV: Response to Q4&W3

Q4&W3:Clarification of the Replacement Experiment.

We thank the reviewer for pointing out the lack of experimental details. We conduct this experiment using Qwen3-4B-Base with tau = 5 on 6,000 problems, equally sampled from the RL and SFT training datasets. The reference responses are obtained from model rollouts for RL and ground-truth solutions for SFT.

We initially randomly sample 50,000 random token positions and collect non-reference candidates with CCD coefficients above 0.1. We then randomly select 100,000 candidates from this pool.

For each candidate, we fix the preceding prefix and perform two separate greedy continuations, starting respectively from the original reference token and its replacement. We compare their extracted final answers, rather than merely their correctness. In particular, two incorrect continuations count as agreeing only if they produce the same incorrect answer.

We additionally conduct a random-replacement control. At the token positions corresponding to these 100,000 candidates, we randomly sample 10 replacement tokens from the vocabulary at each position and evaluate their final-answer agreement using the same procedure. The random replacements achieve only 58% agreement, providing a baseline for interpreting the substantially higher agreement rates observed for candidates selected by CCD proximity.

We have added these experimental details, the random-replacement control, and the agreement criterion in Section 5.3 of the revised PDF.

We thank you again for your positive feedback. We hope that our additional experiments and clarifications have addressed your questions, and we would be very grateful if you could adjust your rating accordingly.

Reviewer_548NOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper observes that both RL-style and SFT post-training reduce to optimizing a weighted log-likelihood of a single forced reference token, and argues this penalizes alternative tokens that could lead to equally valid reasoning paths. It proposes Contextual Credit Diffusion (CCD): a dense token-level reward, motivated by a geometric argument that logit differences measure contextual interchangeability. strength.

Critical strengths
  1. It unifies different training styles into a clean framework, and the critique of the "uniform treatment of non-reference tokens" is well motivated.
  2. The improvements consistently hold across 3 model sizes and 3 different post-training paradigms.
  3. The overall writing and organization are clean and easy to follow.
Critical weaknesses
  1. In the replacement experiment, candidates are sampled only from the >0.1 coefficient range. There is no low-coefficient arm demonstrating that the coefficient actually discriminates interchangeable from non-interchangeable tokens.
  2. Evaluation breadth and reporting are thins: only math reasoning, and only Qwen3 models up to 8B.
  3. More fine-grained ablations on the coefficient itself would be helpful. For example, an inverted variant that gives higher credit to tokens farther from the reference, or a uniform variant that gives all top-k tokens the same credit.
  4. The OTR method discussed in the paper is not included in the experiments.
  5. As I understand it, OTR suppresses a non-reference token in proportion to its probability, so the tokens the model favors most are punished hardest. CCD instead appears to give the most credit to tokens that are close to the reference token. If this understanding is correct, then when the reference token itself has high probability, these are the same tokens, so the two methods would push the same candidates in opposite directions: what OTR punishes most is what CCD rewards most. Given this potential conflict, a further analysis with OTR would be great.
Questions for authors

See weaknesses

5 comments
AuthorsOpenReview ↗

Part I: Response to W1, W2

We sincerely thank the reviewer for the positive assessment. It encourages us a lot. We hope our responses below address your concerns and questions.

W1: Token-Replacement Experiment and Random-Replacement Control.

We thank the reviewer for this suggestion. We first clarify the experimental setup of our token-replacement experiment. We use Qwen3-4B-Base with tau = 5 on 6,000 problems, equally sampled from the RL and SFT training datasets. The reference responses are obtained from model rollouts for RL and ground-truth solutions for SFT.

We randomly sample 50,000 token positions and collect non-reference candidate tokens with CCD coefficients above 0.1, from which we randomly select 100,000 candidates. For each candidate, we fix the preceding prefix and perform two separate greedy continuations, starting from the reference token and the candidate token, respectively. We compare their extracted final answers rather than merely their correctness. Two incorrect continuations are counted as agreeing only if they produce exactly the same incorrect answer.

To address the reviewer's concern about the lack of a low-coefficient control, we additionally conduct a random-replacement experiment. At the positions corresponding to the 100,000 selected candidates, we randomly sample 10 replacement tokens from the vocabulary at each position and evaluate their final-answer agreement using the same procedure. Random replacement achieves only 58% final-answer agreement, substantially below the over 80% agreement observed for CCD candidates with coefficients above 0.1. Furthermore, as shown in Figure 3, agreement increases with the CCD coefficient, reaching 100% for coefficients above 0.9.

We believe these results support the effectiveness of the CCD proximity coefficient in identifying candidate tokens that are more likely to preserve the final answer.

We have added the experimental details and random-replacement control to Section 5.3 of the revised PDF.

W2: Evaluation Beyond Mathematical Reasoning.

We thank the reviewer for this suggestion. To evaluate whether CCD generalizes beyond mathematical reasoning, we conduct additional code-generation experiments using Qwen3-4B-Base. Specifically, we train on the Eurus dataset and evaluate on three widely used coding benchmarks: HumanEval, MBPP, and LiveCodeBench.

We compare the initial model, standard GRPO, and GRPO+CCD under the same training configuration to isolate the contribution of CCD. The results are presented below:

MethodHumanEvalMBPPLiveCodeBenchAvg.
Initial Model65.934.63.634.7
GRPO80.448.611.346.8
GRPO+CCD82.351.612.048.6

These results demonstrate that CCD also improves performance beyond mathematical tasks.

AuthorsOpenReview ↗

Part II: Response to W3

W3: Fine-grained Ablation of the CCD Coefficient.

We thank the reviewer for this valuable suggestion. Following the reviewer's recommendation, we conduct additional experiments with Uniform and Inverted proximity coefficients to investigate whether the effectiveness of CCD depends on assigning greater credit to candidates closer to the reference token. We further include DFT-RL and w/o Prob to examine the respective contributions of probability weighting and proximity-based credit diffusion.

Experimental Setup. All variants are evaluated under the same Qwen3-4B-Base + GRPO training setting, sharing the same training configuration, candidate selection procedure, and group-relative advantage estimation. At each token position, let ata_t denote the sampled reference token, pap_a the student's full-vocabulary softmax probability, and StS_t the top-10 candidate set, which always includes the reference token. The CCD proximity coefficient is defined as:

ϕa=exp⁡(−∣za−zat∣τ),τ=5.\phi_a=\exp\left(-\frac{|z_a-z_{a_t}|}{\tau}\right),\qquad\tau=5.

All variants follow the same surrogate loss:

L=−At∑a∈Stsg⁡[wa]log⁡pa.\mathcal{L}=-A_t\sum_{a\in S_t}\operatorname{sg}[w_a]\log p_a.

Here, AtA_t is the GRPO group-relative advantage, and sg⁡\operatorname{sg} denotes stop-gradient. The variants differ only in their candidate weights waw_a:

1. Full CCD (probability weighting + proximity diffusion).

wa=paϕa.w_a=p_a\phi_a.

This combines the candidate's predicted probability with its proximity to the sampled reference token.

2. DFT-RL (reference-only probability weighting).

wa={pa,a=at,0,a≠at.w_a=\begin{cases}p_a,&a=a_t,\\\\0,&a\neq a_t.\end{cases}

This retains probability weighting for the reference token while removing non-reference credit diffusion.

3. Uniform (without proximity weighting).

wa=pa.w_a=p_a.

This sets the proximity coefficient to 1 for all selected candidates while retaining probability weighting. Thus, candidates retain their original probabilities rather than receiving identical final weights.

4. w/o Prob (without probability weighting).

wa=ϕa.w_a=\phi_a.

This retains proximity-based credit diffusion but removes the multiplicative probability weighting.

5. Inverted (reversed proximity weighting).

wa=paψa,ψa={1,a=at,1−ϕa,a≠at.w_a=p_a\psi_a,\qquad\psi_a=\begin{cases}1,&a=a_t,\\\\1-\phi_a,&a\neq a_t.\end{cases}

This preserves the reference-token weight while assigning larger proximity factors to non-reference candidates farther from the reference token.

Results. We evaluate all variants on six mathematical reasoning benchmarks.

MethodAIME24AIME25AMCMATH500MinervaOlympiadBenchAvg.
Initial Model9.36.136.660.614.827.125.8
GRPO10.613.140.374.217.230.030.9
GRPO + DFT-RL13.010.040.274.413.233.330.7
GRPO + Uniform6.76.732.748.23.718.619.4
GRPO + w/o Prob7.10.036.957.96.325.722.3
GRPO + Inverted6.76.731.646.93.016.318.5
GRPO + CCD11.415.245.375.221.134.433.8

The results directly support the importance of the CCD proximity coefficient. Both variants suggested by the reviewer, Uniform and Inverted, suffer from training collapse, achieving average scores of only 19.4 and 18.5, respectively, compared with 33.8 for full CCD. Removing probability weighting (w/o Prob) also leads to training collapse. Although DFT-RL avoids collapse, it still underperforms standard GRPO (30.7 vs. 30.9). These results demonstrate that both probability weighting and the CCD proximity coefficient are essential for effective training, and that the direction of proximity-based credit diffusion matters.

We have included the complete ablation study in Appendix C of the revised PDF.

AuthorsOpenReview ↗

Part III: Response to W4

W4: Comparison with OTR.

We thank the reviewer for this suggestion. We have added OTR as an additional baseline to further compare its effectiveness with CCD under the Qwen3-4B-Base setting. We report results under both SFT and RL training settings across six mathematical reasoning benchmarks.

SFT Results.

MethodAIME24AIME25AMCMATH500MinervaOlympiadBenchAvg.
Initial Model9.36.136.660.614.827.125.8
SFT15.012.743.771.922.736.533.8
DFT15.613.144.576.729.434.135.6
EAFT12.716.045.670.022.633.533.4
Label Smoothing5.88.533.453.516.224.723.7
OTR15.812.544.875.024.035.034.5
CCD14.016.846.776.930.335.636.7

RL Results.

MethodAIME24AIME25AMCMATH500MinervaOlympiadBenchAvg.
Initial Model9.36.136.660.614.827.125.8
GRPO10.613.140.374.217.230.030.9
GRPO + OTR10.45.834.962.18.124.124.2
GRPO + DFT-RL13.010.040.274.413.233.330.7
GRPO + Uniform6.76.732.748.23.718.619.4
GRPO + w/o Prob7.10.036.957.96.325.722.3
GRPO + Inverted6.76.731.646.93.016.318.5
GRPO + CCD11.415.245.375.221.134.433.8
Reviewer_7Z4rOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper reformulate the RL and SFT objectives as a common one-step decision problem, in which existing methods differ only in how they assign reward across the vocabulary. It then presents a different way of obtaining a dense training objective: rather than using an additional teacher model, it relies on the model's own language modeling to reward tokens that the model scores similarly to the reference token in the current context. The method (CCD) applies to both SFT and RL with little overhead, and shows better pass@1 performance on mathematical reasoning benchmarks than the baselines.

Critical strengths

The method itself is simple, requires no additional teacher model, and the top-kk truncation keeps the overhead small (1--4% per step). It also works on both RL and SFT, whereas the closest prior work (DFT, OTR, EAFT) is SFT-only. The ablations are informative: τ\tau and top-kk sweeps include entropy, response length and training accuracy, and the finding that k=100k=100 underperforms k=10k=10 is reported even though it goes against the paper's own narrative. The approach shows better pass@1 performance in the tested regime.

Critical weaknesses
  • The paper omits the fact that the logits difference corresponds to the difference of log-probabilities which opens the gate to link the distribution sharpening effect considering that we get a factor πθ(a)1+1/τ/πθ(aref)1/τ\pi_\theta(a)^{1+1/\tau}/\pi_\theta(a^{\mathrm{ref}})^{1/\tau} for the tokens candidates below the reference in probability. The paper also doesn't address the fact that as training sharpens the policy, the logit gaps widen and ϕ\phi decays to zero off the reference and there is no more reward or advantages diffusion.
  • The RL results lack ablations on which effect actually matter. Relative to GRPO, GRPO+CCD does two things at once: it rescales the reference token's gradient by πθ(aref)\pi_\theta(a^{\mathrm{ref}}) (DFT applied to RL), it adds the diffused terms on the remaining top-kk tokens. The DFT-RL variant is missing so Table 1 cannot attribute the gains to credit diffusion. The SFT experiments include DFT as a baseline, and there the margin is only +1.1+1.1 and +1.5+1.5 points rather than the +2.9+2.9 to +3.2+3.2 over plain SFT mentioned in the abstract.
Questions for authors

Requests:

  • How does the method behave in the opposite regime, where the success rate is high enough that training is dominated by positive rather than negative trajectories? By the argument in Appendix B.1 the sign of the τ\tau-entropy relationship inverts there, so τ=5\tau = 5 may no longer be the right choice.
  • The metric pass@1 (avg@16) shows a slight improvement which can be attributed to a sharper distribution. It would be good to know whether this comes from more problems solved or simply the same problems solved more frequently. A pass@k curve (k=1,2,4,8,16,32,64,128), for the base model and the compared methods, would provide and answer.

Clarifications:

  • θ\theta is used in the reward definition , but when deriving the form of the gradient the reward is treated as independent of it. This point needs to be addressed and clarified in the paper, for example by simply stating that the reward is detached.
  • The choice of using τ=5.0\tau=5.0 for RL and τ=1.0\tau=1.0 for SFT would benefit from an explanation. For example a link between ablation B.1 on entropy and the fact that SFT fits the RL with positive rewards only regime.
  • The replacement experiment in section 5.3 is hard to follow as is. Which model was used ? What is the value of τ\tau used ? And does arriving at the exact same wrong answer counts or only correct answers do ? It also lacks the in-sequence position numbers: do these substitutions happen toward the end or early in the reasoning trajectory ?
4 comments
AuthorsOpenReview ↗

Part I: Response to Request 1,2 and Clarification 1,2

We sincerely thank the reviewer for the constructive feedback. We hope our responses below address your concerns and questions.

Request 1: Behavior in the High-Success-Rate Regime.

We thank the reviewer for this insightful question. To investigate the opposite regime, we conduct additional experiments using a stronger Qwen3-4B model, comparing GRPO with GRPO+CCD under different proximity temperatures (tau = 5, 10, 20).

Consistent with the reviewer's observation and our conclusion that the effect of tau on entropy depends on model capability, the relationship between tau and entropy indeed reverses in this stronger-model regime: larger values of tau lead to higher training entropy, unlike the trend observed with Qwen3-4B-Base. However, GRPO+CCD consistently achieves higher training accuracy than GRPO across all tested temperatures, demonstrating that CCD remains effective in this regime.

We acknowledge that tau is a hyperparameter that should be adjusted according to model capability, rather than using a universally fixed value. We have added the corresponding training curves and analysis in Appendix B.1 of the revised PDF.

Request 2: Pass@k Analysis.

To further investigate how CCD affects the distribution of successful reasoning trajectories, we conduct additional Pass@k experiments under the Qwen3-4B-Base + GRPO setting. We evaluate 128 sampled responses per problem using the unbiased Pass@k estimator, with k = 1, 2, 4, 8, 16, 32, 64, 128, and report the average across six mathematical reasoning benchmarks.

Due to practical computational constraints, these Pass@k evaluations were conducted on NVIDIA H100 GPUs, whereas both model training and the main evaluations were conducted on RTX 5090 GPUs. Differences in hardware and numerical execution may therefore account for minor discrepancies between the Pass@1 results reported here and those in our main experiments.

MethodPass@1Pass@2Pass@4Pass@8Pass@16Pass@32Pass@64Pass@128
Base27.6335.3942.2048.3253.9058.9763.7768.34
GRPO30.5437.6144.0449.8155.1360.3565.3869.63
GRPO+CCD32.3538.5944.5049.8354.7459.3663.8367.87

The results show that CCD works by more efficiently translating existing Pass@k potential into higher Pass@1 performance. We think this is consistent with the core motivation of CCD: by diffusing credit to alternative token choices that may lead to successful reasoning paths, CCD helps the model more effectively capture and reinforce these alternatives, increasing their probability of being sampled. Consequently, CCD achieves a significant improvement in Pass@1 over GRPO, while its advantage gradually diminishes and eventually reverses at larger k.

Detailed Pass@k results for each benchmark and the corresponding Pass@k curves are provided in Appendix E of the revised PDF.

Clarification 1: Stop-gradient treatment of the reward.

We thank the reviewer for pointing this out. Both the policy probability weights and the CCD proximity coefficients are detached during optimization, as explicitly specified in our CCD surrogate loss. We agree that this assumption should also be stated in the general gradient derivation. We have clarified in the revised PDF that the reward is treated as a detached weighting signal.

Clarification 2: Choice of Proximity Temperature.

SFT operates exclusively in the positive-advantage regime. Moreover, ground-truth tokens may differ substantially from the model's current predictive distribution. We therefore adopt a more conservative choice of tau = 1 for SFT to avoid excessive credit diffusion to less reliable alternatives.

For RL, the appropriate temperature depends strongly on the model's capability and training dynamics, particularly the balance between positive and negative trajectories and the resulting entropy behavior, as analyzed in Appendix B.1. We use tau = 5 in our RL experiments based on the observed behavior of the evaluated models, rather than suggesting it is universally optimal. We have clarified this rationale in the revised PDF. We acknowledge that tau is a hyperparameter that should be adjusted according to model capability.

AuthorsOpenReview ↗

Part II: Response to Clarification3, W1

Clarification 3: Token-Replacement Experiment Setup.

We thank the reviewer for pointing out the lack of experimental details. We conduct this experiment using Qwen3-4B-Base with tau = 5 on 6,000 problems, equally sampled from the RL and SFT training datasets. The reference responses are obtained from model rollouts for RL and ground-truth solutions for SFT.

We initially randomly sample 50,000 random token positions and collect non-reference candidates with CCD coefficients above 0.1. We then randomly select 100,000 candidates from this pool.

For each candidate, we fix the preceding prefix and perform two separate greedy continuations, starting respectively from the original reference token and its replacement. We compare their extracted final answers, rather than merely their correctness. In particular, two incorrect continuations count as agreeing only if they produce the same incorrect answer.

We additionally conduct a random-replacement control. At the token positions corresponding to these 100,000 candidates, we randomly sample 10 replacement tokens from the vocabulary at each position and evaluate their final-answer agreement using the same procedure. The random replacements achieve only 58% agreement, providing a baseline for interpreting the substantially higher agreement rates observed for candidates selected by CCD proximity.

We have added these experimental details, the random-replacement control, and the agreement criterion in Section 5.3 of the revised PDF.

W1: Does CCD necessarily lead to widening logit gaps and vanishing credit diffusion?

We thank the reviewer for this insightful observation. We agree that in SFT, when considering a single token position in isolation, CCD can encourage the logit gap between the reference token and lower-probability candidates to widen. However, actual training optimizes thousands of token positions simultaneously through shared model parameters. Gradients from different positions interact, so a local tendency toward logit-gap widening does not imply that logit gaps will continuously increase throughout training.

For RL, this conclusion is even less applicable, since both positive and negative advantages are present. Negative advantages can reverse the local update direction, while on-policy sampling continuously changes the visited contexts and reference tokens. Therefore, CCD does not necessarily cause logit gaps to widen or credit diffusion to vanish.

To directly investigate this concern, we conduct additional experiments on Qwen3-4B-Base under both RL and SFT settings, tracking the mean absolute logit gap, its 90th percentile, and the mean CCD diffusion coefficient over the first 500 training steps, using non-reference candidates from the top-10 token set.

The results reveal opposite trends that eventually stabilize:

  • RL: Both logit-gap statistics initially decrease and then stabilize, while the mean diffusion coefficient increases from approximately 0.28 to 0.49 before reaching a plateau.
  • SFT: Both logit-gap statistics initially increase and then stabilize, while the mean diffusion coefficient decreases from approximately 0.20 to 0.14 before reaching a nonzero plateau.

These results demonstrate that CCD does not inevitably lead to continuously widening logit gaps or vanishing credit diffusion. Even in SFT, where a local sharpening tendency exists, the diffusion coefficient stabilizes rather than continuously decaying toward zero.

We have included the complete analysis and training curves in Appendix D of the revised PDF.

AuthorsOpenReview ↗

Part III: Response to W2

W2: Ablation of Probability Weighting and Credit Diffusion.

We thank the reviewer for this important suggestion. To directly address the concern that CCD's improvements may arise solely from reference-token probability weighting, we introduce the requested DFT-RL baseline, which retains probability weighting for the sampled reference token but removes all non-reference credit diffusion. We further conduct three additional ablations, Uniform, w/o Prob, and Inverted, to investigate the respective contributions of probability weighting and proximity-based credit diffusion.

Experimental Setup. All variants are trained under the same Qwen3-4B-Base + GRPO setting, sharing the same training configuration, candidate selection procedure, and group-relative advantage estimation. At each token position, let ata_t denote the sampled reference token, pap_a the student's full-vocabulary softmax probability, and StS_t the top-10 candidate set, which always includes the reference token. We define the CCD proximity coefficient as:

ϕa=exp⁡(−∣za−zat∣τ),τ=5.\phi_a=\exp\left(-\frac{|z_a-z_{a_t}|}{\tau}\right),\qquad\tau=5.

All variants follow the same surrogate loss:

L=−At∑a∈Stsg⁡[wa]log⁡pa.\mathcal{L}=-A_t\sum_{a\in S_t}\operatorname{sg}[w_a]\log p_a.

Here, AtA_t is the GRPO group-relative advantage, and sg⁡\operatorname{sg} denotes stop-gradient. The variants differ only in their candidate weights waw_a:

1. Full CCD (probability weighting + proximity diffusion).

wa=paϕa.w_a=p_a\phi_a.

This combines probability weighting with proximity-based credit diffusion.

2. DFT-RL (reference-only probability weighting).

wa={pa,a=at,0,a≠at.w_a=\begin{cases}p_a,&a=a_t,\\\\0,&a\neq a_t.\end{cases}

This is the baseline specifically requested by the reviewer. It preserves the reference-token probability weight while removing all non-reference credit diffusion, allowing us to directly test whether probability weighting alone explains CCD's improvements.

3. Uniform (without proximity weighting).

wa=pa.w_a=p_a.

This sets all proximity coefficients to 1 while retaining probability weighting. The final candidate weights remain their full-vocabulary probabilities rather than equal weights.

4. w/o Prob (without probability weighting).

wa=ϕa.w_a=\phi_a.

This preserves proximity-based diffusion but removes the multiplicative probability weighting.

5. Inverted (reversed proximity weighting).

wa=paψa,ψa={1,a=at,1−ϕa,a≠at.w_a=p_a\psi_a,\qquad\psi_a=\begin{cases}1,&a=a_t,\\\\1-\phi_a,&a\neq a_t.\end{cases}

This preserves the reference-token weight while assigning larger proximity factors to non-reference candidates farther from the reference token.

Results. We evaluate all variants on six mathematical reasoning benchmarks under the same Qwen3-4B-Base training setting.

MethodAIME24AIME25AMCMATH500MinervaOlympiadBenchAvg.
Initial Model9.36.136.660.614.827.125.8
GRPO10.613.140.374.217.230.030.9
GRPO + DFT-RL13.010.040.274.413.233.330.7
GRPO + Uniform6.76.732.748.23.718.619.4
GRPO + w/o Prob7.10.036.957.96.325.722.3
GRPO + Inverted6.76.731.646.93.016.318.5
GRPO + CCD11.415.245.375.221.134.433.8

Full CCD substantially outperforms DFT-RL (33.8 vs. 30.7), while all other ablation variants suffer from training collapse. We believe these results clearly demonstrate that CCD's improvements cannot be attributed solely to reference-token probability weighting, and that both probability weighting and proximity-based credit diffusion are essential for effective training. We have included these additional ablations in the revised PDF.

We thank you again for your valuable feedback. We hope that our additional experiments and clarifications have addressed your concerns, and we would be very grateful if you could adjust your rating accordingly.

Reviewer_7Z4rOpenReview ↗

Thanks

I think the authors for the substantial work to address my comments and questions. I will raise my score in response and hope the authors will include all this in an updated version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.