acceptodds
Under review as a conference paper at ICLR 2027

What Should SFT Learn? Conditional Information Gain for Generalizable Post-Training

Abstract

Supervised fine-tuning (SFT) remains a widely used approach for post-training large language models, because fixed demonstrations can directly introduce desired reasoning strategies and behavioral patterns. Yet forcing a model to imitate externally provided trajectories may also interfere with behaviors already supported by pretraining, limiting generalization beyond the training distribution. Recent dynamic SFT methods mitigate this issue by reweighting token-level supervision according to model-side signals such as predictive probability, prior support, or distributional compatibility. However, model compatibility is not equivalent to learning value: a token that is weakly supported by the current model may either be an undesirable mismatch or precisely the task-specific decision that the demonstration is intended to teach. We propose CIG-SFT, an information-theoretic framework that estimates the task-conditioned information carried by each demonstrated token. Under teacher forcing, we compare the token likelihood from a frozen reference model with and without access to the task input while keeping the gold solution prefix fixed. The resulting conditional information gain quantifies how much the task itself reduces uncertainty about the demonstrated decision. We then convert this information into a KL-regularized projection of the current policy and derive an equivalent scalar-weighted SFT objective whose logit gradient exactly matches that of the projected target. Across mathematical reasoning and code-generation benchmarks, CIG-SFT consistently outperforms standard SFT and recent adaptive fine-tuning baselines, with particularly strong improvements in generalization to challenging evaluation distributions, while requiring neither additional training data nor online rollouts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, supervised fine-tuning, post-training, conditional information gain

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fCbfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CIG-SFT, a token-level weighting method that uses conditional information gain to guide supervised fine-tuning. It estimates the contribution of the task input to each demonstrated token using a frozen reference model and derives a KL-regularized, gradient-equivalent weighted SFT objective. Experiments on mathematical reasoning and code generation show improvements over standard SFT and several adaptive fine-tuning baselines.

Critical strengths

S1. The paper distinguishes task-conditioned information from token predictability and introduces a principled KL-regularized weighting objective.

S2. CIG-SFT consistently achieves competitive performance across mathematical reasoning and code generation, supported by ablation and sensitivity analyses.

Critical weaknesses

W1. The proposed method mainly combines a conditional log-likelihood ratio with an existing KL-regularized policy projection. However, the final token weights still depend on the current model's predictive probability, making the distinction from existing probability-based adaptive SFT methods less substantial.

W2. The mathematical reasoning experiments rely on a single training data source, without dedicated evaluation on high-quality reasoning trajectories distilled from stronger teacher models. Such demonstrations may contain valuable reasoning steps that are difficult for the student model to predict and consequently receive small training weights. It remains unclear whether CIG-SFT can effectively learn from these demonstrations.

W3. Tables 2/4 and Tables 7–9 report inconsistent Pass@16 results (58.6 vs. 56.7) for the default Qwen2.5-Math-1.5B configuration, which should be clarified.

Questions for authors

Q1. Can the authors provide stronger evidence that conditional information gain reflects token learning value rather than task-related predictability or distribution shifts?

Q2. Does CIG-SFT remain effective on high-quality teacher-distilled reasoning datasets? How does it perform against existing baselines in this setting?

4 comments

Weakness 1: Distinction from Probability-Based Adaptive SFT

We thank the reviewer for raising this concern. We agree that KL-regularized projection builds on established optimization techniques. Our key contribution is introducing task-conditioned information into supervision allocation, complementing the model's predictive compatibility rather than replacing it. We clarify the distinction in three aspects:

  1. CIG introduces information beyond predictive compatibility.

    Existing methods such as DFT assign supervision weights based on the model's predictive probability given the full context. In contrast, CIG additionally compares token predictions with and without the task prompt, while keeping the demonstrated response prefix fixed. This comparison captures the additional predictive information provided by the task, which cannot be determined from the full-context probability alone.

  2. Probability dependence does not imply probability-only weighting.

    Model probabilities remain part of our weighting mechanism to maintain predictive support, while CIG guides how supervision is adjusted. Consequently, tokens with similar predictive probabilities can receive different supervision weights depending on their task-conditioned information. For example, in our case study (Appendix A), token "substitute" receives a weight of 0.436 under CIG-SFT versus 0.098 under DFT, while token "is" receives 0.000 versus 0.983. These examples illustrate how CIG can strengthen supervision on selected task-related operations while suppressing some highly predictable generic expressions.

  3. Ablation results confirm the practical importance of this distinction.

    On Qwen2.5-Math-1.5B, CIG-SFT achieves 31.5% Avg@16, compared with 26.3% for DFT. Removing the task-information component reduces performance to 21.9%, while removing KL regularization yields 28.4%. These results support the contributions of both the task-conditioned information signal and the information-proximal optimization mechanism.

In summary, CIG-SFT differs from probability-based adaptive SFT not by eliminating predictive probability, but by incorporating task-conditioned information to guide supervision allocation. We will emphasize this distinction and the respective contributions of the two components in the revised manuscript.

Weakness 2 and Question 2: Learning from Strong-Teacher Reasoning Trajectories

We thank the reviewer for this valuable suggestion. To examine whether CIG-SFT can effectively learn from high-quality reasoning trajectories generated by stronger teachers, we conducted additional experiments using teacher-distilled mathematical reasoning data.

Experimental setup. We use Qwen2.5-Math-1.5B as the student model and selected 30K complete reasoning trajectories from the CoT subset of nvidia/Nemotron-SFT-Math-v4, generated by DeepSeek-V4-Pro (High mode). All trajectories fit within 4096 tokens without truncation. We compared SFT, DFT, iDFT, PriFT-prob, and CIG-SFT under identical training settings, evaluating on the same mathematical benchmarks as our main experiments.

MethodAverageMATH500MinervaOlympiadAIME24AIME25AMC23
SFT17.9/39.150.2/79.411.3/34.616.0/41.31.9/6.72.9/10.025.0/62.5
DFT29.1/52.466.5/88.621.2/47.429.6/57.510.4/16.76.9/26.740.2/77.5
iDFT27.1/51.962.7/87.016.5/41.226.4/53.85.6/13.36.3/33.345.0/82.5
PriFT-prob30.0/55.465.7/90.822.1/48.230.2/60.08.1/23.36.3/30.047.7/80.0
CIG-SFT31.6/55.668.3/90.623.8/46.029.4/58.511.5/26.75.4/26.751.1/85.0

The results show that CIG-SFT remains effective on strong-teacher demonstrations. It achieves the highest average Avg@16 and Pass@16 across the six benchmarks, improving over DFT by 2.5/3.2 points and over PriFT-prob by 1.6/0.2 points, respectively.

These findings provide additional evidence that CIG-SFT can effectively learn from reasoning trajectories distilled from stronger teachers, despite the potential predictive mismatch between teacher demonstrations and the student model. By combining task-conditioned information with model predictive support, CIG-SFT maintains its advantage over probability-based supervision methods in this setting.

We will include these additional experiments in the revised manuscript.

Weakness 3: Inconsistent Pass@16 Results

We thank the reviewer for identifying this inconsistency. After checking the original evaluation records, we confirmed this was a transcription error in the appendix tables, not a difference in experimental settings.

Specifically, the appendix mistakenly combined Avg@16 from seed 42 (31.5%) with Pass@16 from seed 43 (56.7%). The results in Tables 2 and 4 are correct and correspond to seed 42 (31.5% Avg@16 and 58.6% Pass@16).

For transparency, we provide the results across all five random seeds below:

Methodseed 42seed 43seed 44seed 45seed 46Average
SFT13.8/42.313.9/40.913.9/40.514.1/43.914.3/41.414.01/41.79
DFT26.3/49.026.7/49.826.1/49.426.0/49.526.7/50.626.37/49.66
IDFT25.2/48.926.4/49.326.0/45.225.6/43.927.0/49.626.04/47.37
PriFT-prob30.3/56.229.1/56.329.3/55.429.3/55.229.2/55.829.45/55.75
CIG-SFT31.5/58.630.5/56.730.6/54.731.0/56.431.4/57.431.00/56.76

We apologize for the oversight and will correct the affected entries in Appendix Tables 7–9 to ensure consistency with the original evaluation records.

Question 1: Relationship Between CIG and Learning Value

We thank the reviewer for this important question. We would like to clarify the motivation behind CIG and provide additional evidence linking task-conditioned information to downstream learning benefits.

  1. Learning value as our guiding objective.

    We introduce "learning value" to highlight a limitation of existing SFT methods: a token's predictive compatibility does not necessarily reflect how beneficial its supervision will be for downstream learning. Ideally, we aim to allocate supervision according to such learning benefits. However, the corresponding optimal weights are intractable to obtain and cannot be directly evaluated against reliable token-level ground truth.

    Rather than directly estimating these unknown optimal weights, CIG takes a practical step toward learning-value-aware supervision by incorporating the additional predictive information provided by the task prompt. We will accordingly refine the phrase "quantify learning value" in Lines 81–82 to clarify this intended contribution while preserving learning value as our guiding objective.

  2. Why task-conditioned information may indicate useful supervision.

    CIG measures how much conditioning on the task prompt reduces a token's surprisal. By the symmetry of pointwise mutual information, this also characterizes how much information the token carries about the task. Such tokens may reflect task-specific reasoning strategies or their concrete execution steps, motivating stronger supervision to learn task-dependent input–output relationships.

    Our case study in Appendix A illustrates this intuition. CIG-SFT strengthens supervision on the core reasoning operations "substitute" and "check", as well as the initial execution example "a=1", while reducing emphasis on some subsequent repetitive steps and generic expressions.

  3. Additional controlled evidence of downstream learning benefits.

    To examine whether CIG-guided adjustments provide learning benefits, we conduct additional controlled experiments on Qwen2.5-Math-1.5B, following the same settings as our main experiments. In addition to DFT and CIG-SFT, we considered four variants:

    • Up only: Keep only CIG's weight increases; use DFT weights elsewhere.
    • Down only: Keep only CIG's weight decreases; use DFT weights elsewhere.
    • Shuffled up: Shuffle the weight increases among similar upweighted tokens in Up only, preserving the upweighted positions and total increase.
    • Shuffled down: Shuffle the weight decreases among similar downweighted tokens in Down only, preserving the downweighted positions and total decrease.
    MethodAvg@16Pass@16
    DFT26.349.0
    Up only28.652.8
    Shuffled up28.149.8
    Down only31.055.9
    Shuffled down30.855.4
    CIG-SFT31.558.6

    The results reveal three findings:

    1. Both upweighting and downweighting are beneficial. Compared with DFT, "Up only" improves Avg@16/Pass@16 by 2.3/3.8 points, while "Down only" achieves larger gains of 4.7/6.9 points. This suggests that both strengthening and reducing supervision at CIG-selected positions contribute to downstream learning.

    2. Combining both adjustments performs best. Full CIG-SFT achieves improvements of 5.2/9.6 points over DFT, outperforming either individual adjustment. This supports the benefit of jointly allocating stronger and weaker supervision based on task-conditioned information.

    3. The assignment of weight magnitudes also matters. Shuffling the adjustment magnitudes among similar tokens reduces performance, particularly for upweighting (−0.5/−3.0 points). This suggests that CIG benefits not only from adjusting overall supervision strength but also from assigning different weights to individual tokens.

    These results suggest that CIG's effectiveness goes beyond simply measuring predictive changes: using this information to guide token-level supervision yields measurable downstream learning benefits. The shuffled controls further suggest that the assignment of adjustment magnitudes, not just their overall amount, contributes to these gains.

In summary, learning value defines our guiding objective, CIG provides a tractable step toward it, and downstream learning improvements offer indirect empirical evidence for the effectiveness of this step. We do not claim that CIG exactly captures oracle learning values. Instead, we show that incorporating task-conditioned information yields more effective supervision than the evaluated alternatives.

We provide further clarification, detailed case studies, and controlled experimental analyses in our response to Reviewer nESP. We will incorporate the corresponding clarifications and evidence into the revised manuscript.

AuthorsOpenReview ↗

Looking Forward to Your Feedback

Dear Reviewer fCbf,

Thank you again for your careful reading and thoughtful feedback. We have provided detailed responses to your comments, together with additional experiments and clarifications. As the discussion period is coming to a close, we would appreciate hearing your thoughts on our responses and whether you have any remaining questions or concerns. We would be happy to provide further clarification to support the discussion.

Thank you for your time and consideration.

Best regards,
The Authors

Reviewer_nESPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work presents that existing supervised fine-tuning (SFT) methods for large language models suffer from a key limitation: they assign supervision strength according to the model’s predictive compatibility with demonstration tokens, while such model compatibility is not equivalent to learning value. Specifically, the compatibility cannot tell whether a token represents valuable new task knowledge, trivial context continuation, or harmful distribution mismatch. To address this problem, the work proposes CIG-SFT, an information-theoretic framework for selective supervised fine-tuning (SFT) of LLMs. Firstly, it proposes pointwise task-conditioned information gain, as a novel supervision signal, to directly measure the actual learning value of each demonstrated token. Secondly, it derives an information-proximal projection, to adaptively control how far the model policy should shift toward each demonstration token based on task-conditioned information. Thirdly, it mathematically proves the gradient from this projected soft target can be fully reproduced by a simple scalar-weighted hard-target SFT loss, which allows straightforward implementation, without the extra data or online sampling needed by reinforcement learning approaches. Experimental results on mathematical reasoning and code generation datasets show that the proposed CIG-SFT framework ourperforms existing SFT methods and reinforcement learning methods. However, there are still two problems need further clarification. The paper does not fully explain why pointwise task-conditioned information gain captures each token’s actual learning value and only offers a rough analogy. Besides, adding case studies would help readers intuitively understand how this proposed information gain reflects the learning value of each token.

Critical strengths

1.This paper identifies an important limitation between model predictive compatibility and actual learning value in existing SFT methods, and proposes a framework CIG-SFT to tackle this issue. 2.The proposed CIG-SFT is simple to implement and requires no extra training data or online rollouts.

Critical weaknesses

1.This work proposes a complete solution to address the mismatch between model prediction and token learning value. However, the paper does not fully explain why pointwise task-conditioned information gain captures each token’s actual learning value, and only provides a rough analogy. 2.The experimental section presents extensive results to verify the efficacy of the proposed CIG-SFT framework. It conducts comparisons with existing methods on mathematical reasoning and code generation datasets and performs ablation studies. However, this section lacks case studies or similar analysis to intuitively illustrate why the proposed task-conditioned information gain represents the learning value of individual tokens.

Questions for authors

1.Is there any theoretical evidence supports that pointwise task-conditioned information gain can accurately capture the actual learning value of each token? At present, the paper only describes a rough analogy to back up this claim, instead of providing rigorous theoretical derivation. 2.Is there any case study can be added to intuitively demonstrate why the proposed task-conditioned information gain represents the learning value of individual tokens? These concrete examples would help readers better understand the core mechanism of the proposed CIG-SFT framework.

6 comments

Weakness 1: Why does CIG capture learning value

Weakness 2 and Question 2: Cases and analyses

We thank the reviewer for these constructive comments. We recognize that our original presentation may have caused confusion regarding the relationship between task-conditioned information gain and actual learning value. We would like to clarify this relationship and provide additional evidence through case studies and controlled experiments.

  1. Clarification of learning value and our intended contribution.

    • We introduce "learning value" as an intuitive concept to motivate a fundamental limitation of existing SFT methods: a token's predictive compatibility with the current model does not necessarily reflect how beneficial its supervision is for downstream task learning. Therefore, our motivation is to move toward supervision allocation that better reflects such learning benefits.

    • However, the ideal supervision weights are generally intractable to obtain, making it difficult to directly obtain the actual learning value of individual tokens. Rather than attempting to model these unknown optimal weights, we take a practical step toward learning-value-aware supervision: incorporating the additional predictive information provided by the task prompt as a complementary signal to the predictive compatibility. Specifically, when the task prompt reduces a token's surprisal, the symmetry of pointwise mutual information implies that the token also carries information about the task. Such tokens may encode task-specific reasoning strategies or concrete execution steps, motivating stronger supervision at these positions. We evaluate the effectiveness of this step through downstream generalization, without assuming that CIG is equivalent to actual learning value.

    • We agree that the phrase "quantify learning value" in Lines 81–82 may lead to confusion about our contribution. To clarify the relationship between our motivation and technical contribution, we will revise it as follows: "To move toward supervision allocation that better reflects token-level learning value, we propose task-conditioned information gain (CIG) as a tractable complementary signal to the predictive compatibility. By quantifying the additional predictive information provided by the task prompt, CIG incorporates task relevance into supervision weighting, with the goal of improving downstream learning effectiveness."

  2. Case study: How CIG redistributes token-level supervision.

    We have already provided a token-level case study in Appendix A. Here, we highlight three observations from a mathematical reasoning example:

    • Task-related core reasoning strategy: CIG-SFT increases the weights of tokens "substitute" (0.098 → 0.436) and "check" (0.523 → 0.891), which express the core solution strategy of the problem: selecting concrete values through substitution and checking the resulting conditions. Notably, these task-critical operations receive stronger supervision even when their predictive compatibility is relatively low.
    • Initial execution of the strategy: CIG-SFT also emphasizes the first concrete substitution example "a=1", increasing the weights of "a" (0.754 → 0.934) and "1" (0.487 → 0.729). In contrast, subsequent applications of the same strategy receive less consistent emphasis, as they can follow the pattern established by the initial example. This suggests that CIG differentiates the initial execution of a reasoning strategy from its repetitive application.
    • Generic expressions: CIG-SFT reduces supervision on tokens such as "method" (0.353 → 0.000) and parts of "the correct answer is", which contribute relatively little task-specific information despite their predictive compatibility.

    These observations illustrate that CIG-SFT can strengthen supervision on selected task-related operations and mathematical content while reducing emphasis on some generic expressions. This is consistent with our learning-value-oriented motivation: predictive compatibility alone may not provide sufficient information for deciding where supervision should be concentrated.

  1. Additional controlled experiments: Do these adjustments improve downstream learning?

    To further validate CIG-guided supervision allocation, we conducted controlled experiments on Qwen2.5-Math-1.5B following the same settings as our main experiments. In addition to DFT and CIG-SFT, we considered four variants:

    • Up only: Keep only CIG's weight increases; use DFT weights elsewhere.
    • Down only: Keep only CIG's weight decreases; use DFT weights elsewhere.
    • Shuffled up: Shuffle the weight increases among similar upweighted tokens in Up only, preserving the upweighted positions and total increase.
    • Shuffled down: Shuffle the weight decreases among similar downweighted tokens in Down only, preserving the downweighted positions and total decrease.
    MethodAvg@16Pass@16
    DFT26.349.0
    Up only28.652.8
    Shuffled up28.149.8
    Down only31.055.9
    Shuffled down30.855.4
    CIG-SFT31.558.6

    The results reveal three findings:

    1. Both upweighting and downweighting are beneficial. Compared with DFT, "Up only" improves Avg@16/Pass@16 by 2.3/3.8 points, while "Down only" achieves larger gains of 4.7/6.9 points. This suggests that both strengthening and reducing supervision at CIG-selected positions contribute to downstream learning.

    2. Combining both adjustments performs best. Full CIG-SFT achieves improvements of 5.2/9.6 points over DFT, outperforming either individual adjustment. This supports the benefit of jointly allocating stronger and weaker supervision based on task-conditioned information.

    3. The assignment of weight magnitudes also matters. Shuffling the adjustment magnitudes among similar tokens reduces performance, particularly for upweighting (−0.5/−3.0 points). This suggests that CIG benefits not only from adjusting overall supervision strength but also from assigning different weights to individual tokens.

    Connection to learning value. Although the true learning value of individual tokens and the corresponding optimal weights are inaccessible, these experiments provide an indirect, outcome-based evaluation of our learning-value-oriented objective. CIG-SFT achieves better downstream performance than DFT and the tested constrained alternatives, suggesting that task-conditioned information helps guide supervision toward more effective allocations. Importantly, this does not require claiming that CIG directly models oracle learning values or achieves locally optimal weights.

AuthorsOpenReview ↗

Details of the case

[Task Prompt]

Let \aa\\ and \bb\\ be real numbers satisfying \ab<0ab < 0\\. Then, which of the following is correct?

A: \∣a+b∣>∣a−b∣|a+b| > |a-b|\\

B: \∣a+b∣<∣a−b∣|a+b| < |a-b|\\

C: \∣a−b∣<∣∣a∣−∣b∣∣|a-b| < ||a|-|b||\\

D: \∣a−b∣<∣a∣+∣b∣|a-b| < |a|+|b|\\ Let's think step by step and output the final answer within \boxed{}.

[SFT Response]

We use the substitution method. Let \a=1a=1\\, \b=−1b=-1\\, and substitute these values to check;

Option A: \0>20 > 2\\ is not true,

Option C: \2<02 < 0\\ is not true,

Option D: \2<22 < 2\\ is not true,

Therefore, the correct answer is \boxedtextB\\boxed{\\text{B}}\\.

[Detailed token-level weights]

PositionTokenModel Predictive Probability ptp_t and DFT WeightOur CIG-SFT Weight
1We0.0810.043
2use0.0000.000
3the0.6600.595
4substitution0.0000.000
5method0.3530.000
6.0.0440.000
7Let0.3120.360
8$0.0710.018
9a0.7540.934
10=0.0150.000
1110.4870.729
12$,0.0900.000
13$0.7820.321
14b0.9980.959
15=-0.9850.963
1610.6780.858
17$,0.1240.000
18and0.1770.000
19substitute0.0980.436
20these0.3250.576
21values0.8590.679
22to0.0220.038
23check0.5230.891
24;\n\n0.0000.000
25Option0.0050.028
26A0.9370.956
27:0.8420.907
28$0.8420.924
2900.0000.000
30>0.5090.904
31[space]0.7490.675
3220.8570.945
33$0.4120.547
34is0.1910.000
35not0.2170.163
36true0.6870.000
37,\n\n0.0020.001
38Option0.9550.764
39C0.0160.014
40:0.9670.000
41$0.9640.861
4220.7610.926
43<0.4630.761
44[space]0.8510.359
4500.3800.000
46$0.9610.000
47is0.9830.000
48not0.9500.772
49true0.9960.782
50,\n\n0.8580.008
51[space]0.8950.000
52D0.9600.911
53:0.9810.000
54$0.9810.918
5520.9620.959
56<0.8390.940
57[space]0.9940.958
5820.9450.958
59$0.9580.902
60is0.9910.871
61not0.9670.926
62true0.9960.904
63,\n\n0.4700.000
64Therefore0.1780.118
65,0.8030.000
66the0.4280.000
67correct0.7690.527
68answer0.6520.000
69is0.9410.000
70$\0.0660.000
71boxed0.9720.000
72{\0.4940.550
73text0.9750.911
74{0.9890.960
75B0.8860.925
76}}0.9200.872
77$.0.3480.074
78<|endoftext|>0.6660.916
Reviewer_JSCsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CIG-SFT, a method for deciding how strongly to learn from each token in a fixed demonstration. A frozen model compares the token’s probability with and without the task prompt, keeping the preceding answer fixed. The resulting task-information score is combined with the current model’s prediction to produce a token-weighted SFT update. Experiments on math reasoning and code generation show improved aggregate performance over standard SFT and several adaptive SFT baselines, without requiring new demonstrations or online rollouts.

Critical strengths
  • Broad empirical gains with useful ablations. CIG-SFT improves performance over standard SFT and several adaptive SFT baselines across math and code tasks. The paper tests the contribution of its main components and examines sensitivity to learning rate, dataset size, and the proximity strength.

  • Clear method and derivation. The presentation makes it easy to follow the path from measuring a token’s task-conditioned information, to constructing a KL-regularized target, to obtaining a weighted SFT objective with the same gradient. This gives the proposed weight a clear interpretation and makes the method straightforward to implement.

Critical weaknesses
  • Task dependence does not necessarily imply learning value. A token copied from the task prompt may receive a high information score because the prompt makes it easy to predict, without representing a useful reasoning decision.

  • Weak supervision for unlikely tokens. Equation 19 makes the update weight approach zero when the current model assigns very low probability to a demonstrated token, even if that token receives a high task-information score. Early in training, this may weaken supervision precisely where the demonstration introduces an unfamiliar behavior. This is a broader concern with probability-weighted SFT methods, but it remains relevant to CIG-SFT’s motivation.

  • A minor point is that a prior study reports substantial evidence of MATH-500 contamination in Qwen2.5-Math-7B (Wu et al., 2026). Because this paper uses that model and benchmark, the MATH-500 gains alone are less convincing as evidence of generalization.

Wu, M., Zhang, Z., Dong, Q., et al. (2026). “Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination.” Proceedings of the AAAI Conference on Artificial Intelligence, 40(40), 33944–33952.

Questions for authors
  1. RL baseline configuration: Could the authors provide the training configuration for the GRPO and DAPO baselines in Table 2, including the training prompts, reward, group size, rollout budget, number of updates, hyperparameters, and checkpoint-selection rule? Section 4.1 specifies these details for supervised methods but not for RL.

  2. Normalization of the information score: What motivates dividing the log-likelihood gain it=log⁡p0−log⁡p0−i_t=\log p_0-\log p_0^{-} by the task-free surprisal −log⁡p0−-\log p_0^{-}, rather than using the positive part of (i_t) directly?

5 comments

Weakness 1: Copied tokens may receive a high score

We thank the reviewer for the careful reading and constructive feedback. We acknowledge that, in principle, a token copied from the task prompt could receive a high CIG score without providing meaningful reasoning supervision. We will explain why this is not common in practice and does not undermine the effectiveness of our approach.

  1. SFT demonstrations constrain the token sequences being scored. High-quality reasoning demonstrations:
    • generally avoid irrelevant repetition or copying from the task prompt;
    • may typically copy tokens like entities, values, or constraints, which serve a concrete purpose in the solution.
  2. CIG measures conditional probability gain, rather than the probability alone. A token appearing in the prompt may be easy to predict, but its CIG remains low if it is comparably predictable from the same context without the task prompt.
  3. The case study evidence in Appendix.
    • CIG-SFT can recognize important tokens not in the task prompt. In Case A, CIG-SFT emphasizes "substitute" and "check", although neither appears in the prompt, increasing their weights from 0.10/0.52 under DFT to 0.44/0.89.
    • CIG-SFT can recognize important tokens in the task prompt. In Case B, it emphasizes "sneakers", a task entity identifying the item whose cost is calculated.
    • CIG-SFT can weaken solution-irrelevant tokens in the prompt. Generic continuations and formatting tokens "\boxed" receive low weights in these examples.

Weakness 2: Weak supervision for unlikely tokens

We thank the reviewer for raising this important concern. We acknowledge that, as indicated by Eq. (19), the supervision weight approaches zero when the model assigns near-zero probability to a token, even with a high CIG score. This reflects a deliberate design for the trade-off between learning task-specific capability and preserving the model's existing capabilities. [1] provides supporting empirical evidence: masking high-perplexity tokens can better mitigate non-target task degradation than random masking or masking low-perplexity tokens.

  1. Conservative updates for extremely unlikely tokens. Under any fixed λ>0\lambda>0, the weight approaches zero as the model's predicted probability approaches zero. This follows from the KL-regularized projection, which penalizes large deviations from the current predictive distribution. While this does not imply that all unlikely tokens are unhelpful, it reflects a deliberate choice to limit potentially disruptive updates.
  2. Stronger supervision for task-informative tokens with low but non-negligible probability. Unlike near-zero-probability tokens, tokens with low but non-negligible probabilities still receive some predictive support from the current model. CIG-SFT allows stronger supervision for such tokens when they carry substantial task-related information. For example, a token has a prediction probability of 0.0598 and a CIG score of 0.7960. With λ=0.3\lambda = 0.3, its weight reaches 0.4414, 7.38×7.38\times that of DFT. This reflects our design principle: conservative updates for nearly unsupported tokens, while allowing task information to amplify supervision for tokens with existing predictive support. Currently, λ\lambda controls this trade-off empirically, without a theoretical guarantee of optimality.
  3. Empirical benefits of combining both predictive compatibility and task-conditional signals. On Qwen2.5-Math-1.5B, CIG-SFT achieves an Avg@16 of 31.5, compared with 26.3 for DFT. Removing CIG or the KL constraint reduces Avg@16 to 21.9 or 28.4, respectively. Furthermore, the average accuracy on the evaluated general tasks remains close to the base model (55.9% vs. 56.1%). These results support the benefit of combining task-information gain with model predictive support. We will further discuss this limitation and our empirical approach to balancing the trade-off in the appendix.

[1] Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learning

Weakness 3: Potential MATH-500 contamination

We thank the reviewer for raising this concern. We acknowledge the potential contamination issue in MATH-500. However, our original evaluation already includes five other benchmarks (Minerva Math, OlympiadBench, AIME24, AIME25, and AMC23), so our findings are not based solely on MATH-500. To further address this concern, we conduct an additional evaluation on LiveMathBench, a benchmark specifically designed to mitigate data contamination.

Evaluation on contamination-resistant LiveMathBench. LiveMathBench incorporates recent mathematical competition problems and periodically updates its test set to reduce data leakage. We use its v202505 version, containing 100 problems from multiple countries, including problems originating in non-English languages. Importantly, this version postdates the September 2024 release of Qwen2.5-Math-7B, substantially reducing the risk of pretraining contamination.

MethodLiveMathBench
Base7.6/38.0
SFT6.6/35.0
DFT13.8/37.0
ASFT13.1/38.0
iDFT12.4/35.0
PriFT-prob12.6/44.0
PriFT-mass13.4/41.0
CIG-SFT14.3/49.0

Using the existing Qwen2.5-Math-7B checkpoints without retraining, CIG-SFT achieves the best Avg@16 of 14.3 and Pass@16 of 49.0. These results provide additional evidence that CIG-SFT's improvements extend beyond MATH-500 to a newer benchmark designed to mitigate contamination.

Question 1: RL baseline configuration

We thank the reviewer for requesting these implementation details. The training configurations of GRPO and DAPO in Table 2 are summarized below.

ConfigurationSetting
Training promptsQwen2.5-Math chat template, using the same problem statements and CoT instructions as SFT
RewardFinal-answer correctness: +1 for correct, −1 for incorrect
Group size4 responses per prompt
Rollout budgetGRPO: 119,296; DAPO: 458,752 (excluding validation rollouts)
Training updates233 RL iterations, corresponding to 932 optimizer updates
HyperparametersAdamW, learning rate 1e-6, batch size 128, mini-batch size 32, PPO epochs 1, KL coefficient 0.01
Checkpoint selectionFinal checkpoint at iteration 233, without validation- or test-based selection

For a fair comparison, both methods use the same 28,666 problem prompts from the deduplicated NuminaMath-CoT training set, with a fixed random seed 42, consistent answer extraction, and identical evaluation protocols. We also adopt training hyperparameters following existing DFT/PriFT studies. We will include these implementation details in the revised paper.

Question 2: Normalization of the information score

We thank the reviewer for this insightful question. Our normalization is motivated by both theoretical considerations and empirical effectiveness.

  1. Design motivation. The raw log-likelihood gain measures the additional predictive information provided by the task prompt. However, different tokens have different task-free surprisals, leading to different upper bounds on their achievable gains. Importantly, the task-free surprisal is exactly the theoretical upper bound of the positive gain, since the log-probability conditioned on the task cannot exceed zero. Dividing by this bound therefore measures the relative reduction in surprisal attributable to the task, rather than its absolute reduction. This makes scores more comparable across tokens and naturally bounds them within [0,1][0, 1], facilitating a shared regularization strength in the subsequent KL projection.

  2. Empirical validation. We conduct a direct ablation on Qwen2.5-Math-1.5B, replacing the normalized score with the raw positive log-likelihood gain (unnormalized CIG-SFT) while keeping the training data, hyperparameters, evaluation protocol, and KL projection unchanged (λ = 0.3). | Method | Average | MATH500 | Minerva | Olympiad | AIME24 | AIME25 | AMC23 | | ------------ | ----------- | ----------- | ----------- | ------------ | ---------- | ---------- | --------- | | CIG-SFT | 31.5/58.6 | 68.4/89.6 | 24.1/48.5 | 32.1/59.4 | 9.4/40.0 | 5.2/26.7 | 50.0/87.5 | | unnormalized CIG-SFT | 26.6/55.8 | 60.3/90.4 | 19.8/48.5 | 28.1/60.3 | 7.9/33.3 | 3.3/20.0 | 40.0/82.5 |

    Across six benchmarks, normalization consistently improves Avg@16, achieving a macro-average of 31.5 vs. 26.6. It also improves the average Pass@16 from 55.8 to 58.6, although results vary across individual benchmarks.

These results support both the interpretability and practical effectiveness of our normalization design. We will clarify this motivation and include the ablation results in the revised paper.

AuthorsOpenReview ↗

Additional benchmarks for Weakness 3

Additional Evaluation on GSM-Infinite and UGMathBench. To further strengthen our evaluation, we additionally evaluate CIG-SFT on two benchmarks released in 2025, both postdating the September 2024 release of Qwen2.5-Math-7B:

  • GSM-Infinite (ICML 2025): A synthetic benchmark for evaluating multi-step mathematical reasoning with controllable complexity. We evaluate 290 problems from the Medium-0 subset, covering 29 levels of reasoning complexity (2–30 operations).
  • UGMathBench (ICLR 2025): An undergraduate-level mathematics benchmark covering 16 subjects, including algebra, calculus, and probability. We evaluate 500 problems sampled across all 16 subjects from its v1 dataset.

Contamination-Resistant Benchmark Design. Beyond their recent release dates, the two additional benchmarks incorporate mechanisms that help mitigate data contamination. GSM-Infinite constructs synthetic mathematical problems through programmatically generated computational graphs, without relying on LLM-generated questions. Its scalable generation mechanism reduces dependence on fixed, publicly available test problems. UGMathBench explicitly addresses test-set contamination through randomized problem variants, where different numerical instantiations preserve the underlying mathematical reasoning structure.

The results on Qwen2.5-Math-7B are shown below (Avg@16 / Pass@16, %):

MethodGSM-InfiniteUGMathBench
Base8.43 / 46.2119.85 / 57.60
SFT15.52 / 46.9023.24 / 53.40
DFT21.68 / 46.5539.56 / 57.80
ASFT22.65 / 54.1438.41 / 65.20
IDFT20.93 / 44.4835.04 / 50.20
PriFT-prob25.43 / 60.3441.54 / 67.20
PriFT-mass24.22 / 57.5941.39 / 63.20
CIG-SFT27.52 / 62.4146.06 / 69.00

CIG-SFT achieves the best performance on both benchmarks across both metrics. Compared with the strongest baseline, CIG-SFT improves Avg@16 by 2.09 and 4.52 percentage points on GSM-Infinite and UGMathBench, respectively, while also improving Pass@16 by 2.07 and 1.80 points.

Taken together, our additional evaluations on LiveMathBench, GSM-Infinite, and UGMathBench consistently outperform all compared methods across both metrics. These results provide stronger evidence that CIG-SFT's improvements extend beyond MATH-500 to newer benchmarks covering diverse mathematical reasoning tasks, while reducing concerns about potential pretraining data contamination.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary and Key Takeaways

Dear Area Chair,

As we have not received further reviewer feedback, we summarize the recognized contributions and our responses to the main concerns.

1. Contributions recognized in the reviews

  • Motivation: nESP recognized the importance of improving supervision allocation beyond model predictive compatibility, together with the method's simple implementation without extra training data or online rollouts.
  • Technical contribution: fCbf acknowledged the distinction between task-conditioned information and token predictability and the principled weighting formulation. JSCs positively assessed the derivation of a gradient-equivalent SFT objective from task-conditioned information.
  • Empirical support: JSCs highlighted improvements over standard and adaptive SFT across mathematics and code generation. fCbf also recognized the competitive performance and supporting ablation and sensitivity analyses.

2. Main concerns and our responses

  • Interpretation of CIG (nESP, JSCs, fCbf): We clarified that learning value serves as our guiding motivation, while the corresponding optimal token-level supervision weights are intractable to obtain. CIG takes a practical step toward more effective supervision allocation by introducing task-conditioned information as a complementary signal to predictive compatibility. Detailed token-level cases and additional controlled weighting experiments support the usefulness of this signal for downstream learning. This clarification preserves the method and its contribution.

  • Learning from strong-teacher demonstrations (fCbf): We additionally trained Qwen2.5-Math-1.5B on 30K strong-teacher reasoning trajectories. CIG-SFT achieves the highest average Avg@16/Pass@16 across six benchmarks (31.6/55.6), exceeding DFT (29.1/52.4) and PriFT-prob (30.0/55.4). These results support its ability to learn from stronger teachers.

  • Contamination between Qwen-Math and MATH500 (JSCs): CIG-SFT leads both metrics on the added contamination-resistant LiveMathBench, GSM-Infinite, and UGMathBench evaluations and achieves the best average performance on the decontaminated DeepSeekMath-7B-Base, supporting generalization beyond Qwen and MATH-500.

  • Reporting and reproducibility (fCbf, JSCs): We traced the Pass@16 discrepancy to an appendix transcription error mixing two seeds, and the main-table results remain correct. We also supplied RL configurations and a normalization ablation.

We believe these clarifications and targeted experiments address the principal concerns while preserving a coherent contribution across motivation, methodology, and empirical evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.