- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a new loss function applicable to both SFT and RL. The CCD coefficient is defined based on the logit difference between two tokens coupled with exponential decay. By incorporating CCD with probability weighting, the optimization shifts from being anchored to a single reference token to jointly optimizing a group of tokens with similar logit values. This approach only modifies the loss function, requires no teacher model, and incurs negligible computational overhead. Experiments use one training dataset for SFT and one for RL, and evaluate several Qwen3 model sizes on six mathematical reasoning benchmarks.. Results demonstrate that the proposed CCD algorithm yields performance gains across different configurations on both experimental datasets.
Critical strengths
The proposed CCD loss indeed provides a possible solution. Since the method does not substantially increase computational overhead, it is also practically feasible from an engineering perspective.
Critical weaknesses
-
While the paper attempts to analyze the CCD loss and explicate its design rationale, the existing evidence does not sufficiently establish the mechanism. Critical issues remain largely undiscussed: how the CCD loss reshapes the probability distribution of non-reference tokens, and why such distributional changes can mathematically account for performance improvements.
-
The discussion of the “single reference” limitation in RL requires more explanation. When expectation is taken over sampled actions (in policy gradient), the resulting gradient incorporates the advantage value of every possible action. In multi trajectory training regimes, different actions conditioned on the same context can be sampled independently, each receiving direct training signals derived from its respective subsequent return.
-
The paper’s token replacement experiments lack detailed specifications. The exact procedure for selecting candidate tokens remains ambiguous, and tokens sampled from different positions may yield markedly different impacts on model outputs. A thorough account of the experimental implementation is needed. Additionally, experiments in the study show that the CCD coefficients of most tokens are small. However, if the CCD coefficients of the topk tokens selected by predicted probability are small, then it is likely that the set of tokens optimized by this loss is not semantically interchangeable.
Questions for authors
- Under what conditions can a non-reference candidate be promoted or suppressed? And a small controlled experiment containing valid alternatives and invalid tokens with similar logits would help.
- Since standard RL can already obtain direct feedback for different actions under the same prefix through repeated sampling, which remaining limitation is CCD intended to address?
- Could you compare the full CCD loss with reference-only probability weighting and with diffusion using the same proximity coefficients but without probability weighting?
- How are questions or trajectories, token positions, and candidate tokens selected in the replacement experiment?
4 comments
Part II: Response to Q2&W2
Q2 & W2: Single-reference limitation in LLM RL.
We thank the reviewer for this observation. In practical LLM RL, policy-gradient methods use Monte Carlo estimates based on sampled trajectories. At each visited state, the gradient contribution is computed from the sampled action and its estimated advantage:
Under standard unbiased-estimation assumptions, the expectation of this sampled gradient is equivalent to the full-action policy gradient:
However, this equivalence holds in expectation. It does not imply that the advantages of all actions are available or explicitly evaluated during training. In autoregressive LLM reasoning, the state at step consists of the prompt and all previously sampled tokens:
Crucially, unlike the shared initial prompt, intermediate prefixes are almost never revisited across independently sampled reasoning trajectories. This is a key distinction between LLM RL and conventional RL settings with frequently revisited states: in autoregressive generation, each state contains the entire previously generated token sequence, so two trajectories must generate exactly the same prefix to revisit the same state. For reasoning trajectories, the probability of such an exact match becomes vanishingly small. Consequently, most visited intermediate state effectively receives direct outcome-based supervision for only one sampled action, even when multiple trajectories are generated from the same prompt.
Moreover, repeatedly sampling alternative actions at every intermediate prefix is computationally prohibitive in LLM RL. Outcome-based rewards are typically obtained only after generating and verifying the complete response. Therefore, evaluating an alternative action at a given prefix requires generating its subsequent continuation and observing the final outcome, rather than evaluating the token in isolation.
For a response of length , sampling alternative actions at each prefix requires additional continuation rollouts, corresponding to up to generated tokens. Such computational overhead is unacceptable.
Instead, CCD uses the model's existing next-token distribution to propagate the observed trajectory advantage to multiple candidate tokens:
This introduces negligible additional computational overhead without requiring any extra rollouts. Alternatively, CCD can be viewed as assigning each candidate token a proximity-weighted approximate advantage, , where candidates closer to the sampled reference token receive an advantage closer to that of the reference token. This enables credit sharing across alternative actions using feedback from a single sampled trajectory.
Part III: Response to Q3
Q3: Ablation of Probability Weighting and Credit Diffusion.
We thank the reviewer for this important suggestion. We agree that CCD introduces two components: reference-token probability weighting and proximity-based credit diffusion. To disentangle their contributions, we conduct additional controlled ablations under the Qwen3-4B-Base + GRPO setting, including reference-only probability weighting (DFT-RL), diffusion without proximity weighting (Uniform), diffusion without probability weighting (w/o Prob), and reversed proximity weighting (Inverted).
Experimental Setup. All CCD variants share the same training configuration, candidate selection procedure, and group-relative advantage estimation. At each token position, let denote the sampled reference token, the student's full-vocabulary softmax probability, and the top-10 candidate set, which always includes the reference token. We define the CCD proximity coefficient as:
All variants follow the same surrogate loss:
Here, is the GRPO group-relative advantage, and denotes stop-gradient. The variants differ only in their definitions of the candidate weights :
1. Full CCD (probability weighting + proximity diffusion).
The weight incorporates both the candidate's predicted probability and its predictive proximity to the sampled reference token.
2. DFT-RL (reference-only probability weighting).
This retains probability weighting for the reference token while removing all non-reference credit diffusion. It directly tests whether CCD's gains can be explained by reference-only probability weighting.
3. Uniform (diffusion without proximity weighting).
This sets the proximity coefficient to 1 for all selected candidates while retaining probability weighting. Note that the candidate weights are still their original full-vocabulary probabilities, rather than equal weights.
4. w/o Prob (proximity diffusion without probability weighting).
This preserves the same candidate set and proximity coefficients as CCD but removes the multiplicative probability weighting, directly addressing the reviewer's suggested ablation.
5. Inverted (reversed proximity weighting).
This preserves the reference-token weight while assigning larger proximity factors to non-reference candidates farther from the reference token.
Results. We evaluate all variants on six mathematical reasoning benchmarks under the same Qwen3-4B-Base training setting.
| Method | AIME24 | AIME25 | AMC | MATH500 | Minerva | OlympiadBench | Avg. |
|---|---|---|---|---|---|---|---|
| Initial Model | 9.3 | 6.1 | 36.6 | 60.6 | 14.8 | 27.1 | 25.8 |
| GRPO | 10.6 | 13.1 | 40.3 | 74.2 | 17.2 | 30.0 | 30.9 |
| GRPO + DFT-RL | 13.0 | 10.0 | 40.2 | 74.4 | 13.2 | 33.3 | 30.7 |
| GRPO + Uniform | 6.7 | 6.7 | 32.7 | 48.2 | 3.7 | 18.6 | 19.4 |
| GRPO + w/o Prob | 7.1 | 0.0 | 36.9 | 57.9 | 6.3 | 25.7 | 22.3 |
| GRPO + Inverted | 6.7 | 6.7 | 31.6 | 46.9 | 3.0 | 16.3 | 18.5 |
| GRPO + CCD | 11.4 | 15.2 | 45.3 | 75.2 | 21.1 | 34.4 | 33.8 |
The results show that all ablation variants except DFT-RL suffer from training collapse, with Uniform, w/o Prob, and Inverted performing substantially worse than the initial model. Although DFT-RL avoids collapse, it still underperforms standard GRPO (30.7 vs. 30.9). We believe these results demonstrate that both probability weighting and the CCD proximity coefficient are essential for effective training.
We have included the complete ablation study in Appendix C of the revised PDF.
Part IV: Response to Q4&W3
Q4&W3:Clarification of the Replacement Experiment.
We thank the reviewer for pointing out the lack of experimental details. We conduct this experiment using Qwen3-4B-Base with tau = 5 on 6,000 problems, equally sampled from the RL and SFT training datasets. The reference responses are obtained from model rollouts for RL and ground-truth solutions for SFT.
We initially randomly sample 50,000 random token positions and collect non-reference candidates with CCD coefficients above 0.1. We then randomly select 100,000 candidates from this pool.
For each candidate, we fix the preceding prefix and perform two separate greedy continuations, starting respectively from the original reference token and its replacement. We compare their extracted final answers, rather than merely their correctness. In particular, two incorrect continuations count as agreeing only if they produce the same incorrect answer.
We additionally conduct a random-replacement control. At the token positions corresponding to these 100,000 candidates, we randomly sample 10 replacement tokens from the vocabulary at each position and evaluate their final-answer agreement using the same procedure. The random replacements achieve only 58% agreement, providing a baseline for interpreting the substantially higher agreement rates observed for candidates selected by CCD proximity.
We have added these experimental details, the random-replacement control, and the agreement criterion in Section 5.3 of the revised PDF.
We thank you again for your positive feedback. We hope that our additional experiments and clarifications have addressed your questions, and we would be very grateful if you could adjust your rating accordingly.
Part I: Response to Q1
We sincerely thank the reviewer for the positive assessment. It encourages us a lot. We hope our responses below address your concerns and questions.
Q1: Promotion/suppression conditions and controlled experiment.
(1) Under what conditions can a non-reference candidate be promoted or suppressed?
We thank the reviewer for this question. We provide a detailed derivation of how CCD promotes or suppresses candidate tokens. For a given context, let the probability of token a be defined by the softmax function:
pa=∑jexp(zj)exp(za).
The derivative of its log-probability with respect to the logit of token j is:
∂zj∂logpa=I[a=j]−pj.
For a single token position, the CCD surrogate loss is:
L=−At∑a∈Tsg[paϕa]logpa.
Here, At denotes the advantage in RL and is set to 1 in SFT. Let wa=paϕa and W=∑a∈Twa. Since the weighting coefficients are detached during optimization, they are treated as constants when computing gradients. Applying the softmax derivative yields:
∂zj∂L=−At∑a∈Twa(I[a=j]−pj).
For a selected candidate token j∈T, this simplifies to:
∂zj∂L=−At(wj−pjW)=Atpj(W−ϕj).
Therefore, under gradient descent with learning rate η>0, the local logit update is:
Δzj=−η∂zj∂L=ηAtpj(ϕj−W).
Since pj>0, the update direction depends on the advantage sign and the candidate's proximity coefficient relative to the total CCD weight:
Thus, CCD does not indiscriminately promote all non-reference candidates. Its update direction depends jointly on the signed advantage and the candidate's contextual predictive proximity relative to the aggregate CCD weight.
Why can CCD improve task performance?
The key distinction is that maximizing the probability of a single reference token is not necessarily equivalent to maximizing the probability of producing a correct final answer, since multiple token choices may lead to successful reasoning trajectories. For positive advantages, the CCD logit update can be expressed as a probability redistribution toward a proximity-weighted soft target. When predictive proximity is positively associated with the downstream success probabilities of candidate actions, this redistribution favors alternatives that can also lead to correct outcomes, rather than concentrating supervision exclusively on the sampled reference token. Thus, CCD can improve task performance by aligning token-level credit allocation more closely with the multiplicity of successful reasoning paths.
We acknowledge that providing a rigorous mathematical explanation for why this mechanism improves downstream performance is challenging, particularly given the complexity of LLM optimization. Our design is motivated by the intuition that multiple alternative tokens can lead to successful reasoning paths, while the token-replacement experiment and component ablations provide empirical support for its effectiveness.
(2) Controlled experiment with valid and invalid alternatives.
We thank the reviewer for this suggestion. However, formally defining valid and invalid alternative tokens is challenging, as it is difficult to precisely determine whether two tokens are semantically similar under a given context.
If we define a candidate as valid when its continuation produces the same final answer as the continuation from the reference token, and invalid otherwise, we believe our existing token-replacement experiment already provides large-scale empirical evidence addressing this concern.
Specifically, the reference token is itself a token from the same vocabulary, and the CCD coefficient directly measures a candidate's predictive proximity to this reference through their logit difference. Therefore, candidates with higher CCD coefficients have logits closer to that of the reference token. In Figure 3, we group candidates into coefficient bins of width 0.1 and compare the final answers obtained after independently continuing from the reference and candidate tokens. The results show that higher CCD coefficients are associated with higher final-answer agreement, providing empirical support for the relationship between predictive proximity and outcome preservation across 100,000 token replacements.
We have further clarified the experimental setup in Section 5.3 of the revised PDF and in our response to Q4.