acceptodds
Under review as a conference paper at ICLR 2027

Tail-Calibrated Loss-Distribution Shaping for Reasoning Fine-Tuning

Abstract

Curricula based on sample ordering or selection regulate which examples are presented during training, but do not directly specify the relative weighting of token losses within each response. We propose Tail-Calibrated Loss-Distribution Shaping (TCLD), which adapts response-token loss aggregation using feedback from the current training-loss distribution. TCLD replaces arithmetic averaging with a power-mean objective whose parameter controls relative token gradient coefficients and recovers standard cross-entropy (CE) at . A Hill-tail controller calibrates at observation-window boundaries using no-gradient loss observations, reducing the relative coefficients of higher-loss tokens without a downstream-validation-accuracy search over . On a heterogeneous five-source reasoning pool, TCLD improves Macro- and Micro-averaged maj@3 accuracy over random-shuffle training (RS) by 2.74 and 2.02 percentage points, respectively, across three matched seeds with Qwen3-1.7B-Base, and outperforms the evaluated sample-level curriculum baselines in aggregate accuracy. At the 25% optimizer-step checkpoint, TCLD exceeds the full-budget RS baseline in mean Macro accuracy. Source-wise diagnostics show heterogeneous sensitivity to fixed powers and the competitive performance of automatic calibration. Gains over RS also extend to Qwen3-0.6B and SmolLM2-1.7B. Single-seed experiments on six ordered two-source streams show order-dependent gains and per-source trade-offs relative to RS. TCLD requires no changes to the model architecture or inference procedure.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Loss-Distribution Shaping, Reasoning Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EZwcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TCLD, which replaces arithmetic averaging of response-token cross-entropies with a power-mean objective. A detached controller estimates upper-tail statistics from a no-gradient observation pass, smooths them, and sets a power below one to reduce the relative influence of high-loss tokens. Experiments compare TCLD with shuffled SFT and sample-ordering curricula on a five-source reasoning mixture, with three matched seeds and two additional model configurations. Fixed-power, observation-window, and ordered-stream diagnostics examine calibration. TCLD improves aggregate accuracy over standard SFT, while its advantage over a competitive fixed power remains statistically unresolved.

Critical strengths
  1. The main empirical improvement is supported by controlled, repeated comparisons. Table 1 reports gains of 2.74 Macro and 2.02 Micro percentage points over shuffled SFT on Qwen3-1.7B, using three matched seeds and a common endpoint. Table 4 extends the improvement to Qwen3-0.6B and SmolLM2-1.7B. The resulting evidence supports the practical usefulness of the complete objective and shows that the gain is not confined to one initialization or model configuration.

  2. The paper is unusually precise about what its analysis establishes. Equation (3) characterizes relative gradient coefficients, while Sections 3.2–3.3 acknowledge that tail severity neither identifies semantic usefulness nor proves optimal weighting. The fixed-power confidence intervals and computational overhead are also disclosed. This transparency makes the contribution easier to assess and provides a useful starting point for further work on adaptive loss aggregation.

Critical weaknesses
  1. The distinctive contribution of dynamic tail calibration is not isolated convincingly. Table 2 improves over fixed p = 0.7 by only 0.79 Macro and 0.34 Micro points, with paired intervals including zero. Appendix L reports a mean controller power near 0.688, making the fixed comparator particularly relevant. Ordered-stream experiments compare against p = 1 rather than a competitive fixed or predetermined schedule. Thus, the evidence establishes useful loss shaping more clearly than a benefit from repeatedly observing and tracking the loss tail.

  2. The experiments leave the source of the objective's benefit unresolved. Equation (2) changes both relative token allocation and a batch-dependent common gradient scale, but Section 4.2 explicitly states that these effects are not separated experimentally. The principal baselines mostly change sample order rather than token weights. Consequently, the study does not establish whether the gains require the full controller–power-mean construction or arise from a simpler weighting/scaling intervention. This matters when assessing the method's additional complexity and observation cost.

Questions for authors
  1. Clarification: Are existing results available for a fixed power near the realized controller mean, a controller frozen after its first observation, or a replayed power schedule? If not, should the contribution be framed primarily as competitive automatic calibration rather than evidence that online feedback improves accuracy?

  2. Clarification: Were any controls separating Eq. (2)'s common scale from its relative token coefficients already run? Such results would materially affect my assessment of which part of the objective provides the improvement.

  3. Clarification: Were the tail fractions, smoothing coefficient, and index-to-power mapping fixed before downstream evaluations? Please describe their selection, since avoiding a search over p does not by itself establish independence from downstream tuning of the controller.

1 comment

Thank you for recognizing the empirical results and the scope of our analysis. We have added controls separating relative token weighting from the common gradient scale, a first-window-frozen comparison, and smoothing-sensitivity experiments. Revisions addressing your comments are highlighted in green in the revised manuscript.

1. Automatic calibration versus continued refresh.

The existing fixed p = 0.7 comparison is close to the realized controller mean of approximately 0.688. We have now added First-window Frozen TCLD, which retains each seed’s first calibrated power throughout training. Across three seeds, Macro/Micro maj@3 accuracy (mean ± sample standard deviation) is:

  • RS: 57.09 ± 0.20 / 60.99 ± 0.74
  • Fixed p = 0.7: 59.04 ± 0.62 / 62.67 ± 0.67
  • First-window Frozen: 59.68 ± 0.94 / 63.26 ± 0.56
  • Online TCLD: 59.83 ± 0.69 / 63.01 ± 0.26

Online TCLD’s mean gains over fixed p = 0.7 are 0.79 Macro / 0.34 Micro percentage points, with paired confidence intervals including zero. Relative to Frozen, the differences are +0.15 / −0.25 points, with mixed directions across seeds.

These results support competitive automatic calibration of the power parameter from training-loss statistics, without a downstream-validation-accuracy search over p. TCLD uses window-level feedback to recalibrate this parameter as training progresses. The Frozen comparison shows that initial calibration captures most of the observed mixed-pool improvement over RS, with Online and Frozen achieving similar aggregate performance. We have revised the contribution statement and discussion to emphasize automatic calibration and clarify the role of continued refresh. Section 4.3 and Appendix G report these comparisons.

2. Relative token weighting and common gradient scale.

The new three-seed controls give:

  • RS: 57.09 ± 0.20 / 60.99 ± 0.74
  • Floor-only CE: 57.13 ± 0.76 / 61.30 ± 0.22
  • Scale-only: 57.20 ± 0.05 / 61.23 ± 0.39
  • Relative-only: 58.71 ± 0.23 / 63.28 ± 0.43
  • TCLD: 59.83 ± 0.69 / 63.01 ± 0.26

Relative-only retains normalized token weights, while Scale-only applies the common gradient factor to uniformly weighted floored CE. Both replay the corresponding seed’s Online power schedule and recompute detached coefficients from their own current batch losses. Appendix B provides the objective definitions.

Relative weighting recovers a substantial part of the improvement, whereas Scale-only remains close to Floor-only CE. The complete objective achieves 1.12 points higher Macro, while Relative-only achieves 0.27 points higher Micro. These findings support an important role for relative token weighting and clarify the trade-off between the two aggregate metrics. Detailed results appear in Appendix G.

3. Controller choices and smoothing sensitivity.

The tail fractions, smoothing coefficient, and index-to-power mapping were used as shared design choices across datasets, without dataset-specific adjustment.

We use fixed tail fractions of 2.5%, 5%, and 7.5% and take the median of the usable Hill estimates to reduce dependence on a single threshold. Median aggregation across thresholds has a related precedent in the weighted log-excesses analysis of Gomes, de Haan, and Henriques; our implementation uses three ordinary Hill estimates. When all three are usable and two agree within an interval, their median remains within that interval despite an atypical third estimate. Exponential smoothing then moderates window-level fluctuations, following the established use of moving averages for noisy statistics, as exemplified by Adam (Kingma and Ba, 2015). Finally, the mapping pₖ = σ(zₖ) converts the smoothed log-tail state monotonically into (0, 1), so a smaller effective tail exponent produces stronger relative downweighting of high-loss tokens. Section 3.2 describes the mechanisms and their motivations.

We evaluated β = 0.5 and β = 0.8 alongside the shared default β = 0.4. The three-seed Macro/Micro summaries are:

  • β = 0.4: 59.83 ± 0.69 / 63.01 ± 0.26
  • β = 0.5: 59.01 ± 0.52 / 63.14 ± 0.50
  • β = 0.8: 59.43 ± 0.25 / 63.04 ± 0.66

All three summaries exceed RS in mean Macro and Micro accuracy. Across the tested settings, mean Macro varies by 0.82 percentage points and mean Micro by 0.13 points, suggesting limited aggregate sensitivity within this mixed pool. The β = 0.5 summary includes a seed42 aggregate combining source evaluations from separate same-configuration runs, as detailed in Appendix L. The sensitivity experiments retain the original default; Appendix L provides the results and evaluation conditions.

Reviewer_3TTwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes Tail-Calibrated Loss-Distribution Shaping (TCLD), which aggregates response-token cross-entropy losses using a power mean and adjusts its exponent p based on a Hill estimate of the training-loss upper tail. This reduces the relative gradient weight of high-loss tokens. On a mixed training pool of five math-related datasets, TCLD improves average accuracy over random-shuffle training and the sample-level curriculum baselines evaluated in the paper. The paper also reports single-source diagnostics, ordered training-stream experiments, and results on two additional small models.

Critical strengths
  1. Clear formulation and analysis. The method is precisely defined, and the gradient analysis explains how p changes the relative weighting of tokens.
  2. Promising empirical results. Under the matched settings of the main experiment, TCLD improves average performance over random-shuffle training. Results on two additional models provide preliminary evidence that the effect is not limited to the primary model.
Critical weaknesses
  1. Lack of novelty and discussion of related work. The idea of dynamically downweighting high-loss tokens is closely related to methods such as TALR and DFT, while adaptive power means have also been explored for reasoning optimization. The paper does not sufficiently discuss these closely related works, making it difficult to assess its incremental contribution.

  2. Lack of comparisons with appropriate token-level baselines. The main baselines focus on shuffled training and sample-level curriculum learning, without comparisons against token-level reweighting methods such as TALR and DFT

  3. Lack of reliable experimental results. Compared with a fixed p=0.7, the intervals for all three paired differences of TCLD include zero. Therefore, the current results do not reliably demonstrate the additional benefits of the dynamic controller.

  4. Limited generalizability due to narrow training tasks and model scales. The main mixed training pool consists of five mathematics-related datasets. The cross-model experiments cover Qwen3-0.6B, Qwen3-1.7B, and SmolLM2-1.7B, leaving the effectiveness of the method on larger models unclear.

  5. Insufficient evaluation of training-time overhead. TCLD requires an additional loss observation process to estimate the tail distribution and update p. It remains unclear whether the performance improvements justify the additional computational cost.

Questions for authors
  1. Could the authors include token-level reweighting baselines such as TALR and DFT under the same model, data, training budget, and evaluation settings?

  2. Does the Hill controller provide reproducible advantages over a fixed p=0.7 or simpler controllers based on loss statistics?

  3. Could the authors validate the main conclusions on mixed training pools containing more non-mathematical tasks, as well as on larger-scale models?

  4. How much additional wall-clock training time and computational cost does each loss observation and p update introduce?

2 comments

Thank you for your constructive comments on token-level comparisons, automatic calibration, generalization, and computational cost. We have expanded the experiments and related-work discussion to address these points. The corresponding revisions are highlighted in red in the revised manuscript.

1. Comparisons with token-level methods and clarification of novelty.

We added DFT and TALR under the same Qwen3-1.7B-Base five-source fine-tuning configuration. Across three seeds, Macro/Micro maj@3 accuracy (mean ± sample standard deviation) is:

  • RS: 57.09 ± 0.20 / 60.99 ± 0.74
  • DFT: 56.61 ± 0.56 / 59.26 ± 0.91
  • TALR: 58.85 ± 0.45 / 61.52 ± 0.71
  • TCLD: 59.83 ± 0.69 / 63.01 ± 0.26

TCLD’s mean improvements are 3.22 Macro / 3.75 Micro percentage points over DFT and 0.98 / 1.49 points over TALR. Table 1 reports these comparisons, with per-source and seed-level results in Appendix G.

The revised related-work discussion also distinguishes TCLD from DFT, TALR, and adaptive power-mean optimization. TCLD combines power-controlled response-token loss aggregation with automatic calibration from the current supervised loss distribution, using window-level tail estimation and state tracking.

2. Fixed-power comparisons and calibration sensitivity.

Section 4.3 compares TCLD with the following fixed-power references:

  • p=0.3p=0.3: 58.26 ± 0.69 / 61.81 ± 0.42
  • p=0.7p=0.7: 59.04 ± 0.62 / 62.67 ± 0.67
  • p=1.3p=1.3: 52.98 ± 0.97 / 56.82 ± 0.78

We selected p=0.7p=0.7 as a trajectory-informed reference because it is close to the observed controller mean of approximately 0.688. It provides a competitive fixed-power comparison: TCLD’s mean gains are 0.79 Macro / 0.34 Micro points, although the paired confidence intervals include zero. These results show that TCLD achieves competitive performance against a strong fixed-power reference while calibrating pp directly from training-loss statistics, without a downstream-validation-accuracy search over pp.

3. Non-mathematical tasks and larger models.

TCLD’s formulation is not specific to mathematics. Section 4.3 includes single-source experiments on EntailmentBank and QASC, where TCLD improves accuracy over RS from 64.71% to 72.73% and from 74.51% to 75.16%, respectively. These results provide evidence of applicability to non-mathematical reasoning. We emphasize mathematics in the primary mixed-pool evaluation because its answers support relatively objective correctness assessment.

To address model scale, we added a three-seed Qwen3-4B comparison:

  • RS: 69.80 ± 0.45 Macro / 73.85 ± 0.09 Micro
  • TCLD: 71.69 ± 0.20 Macro / 75.26 ± 0.26 Micro

TCLD improves mean Macro and Micro accuracy by 1.89 and 1.42 percentage points, respectively, with gains in both aggregate metrics for every seed. Section 4.5 and Appendix J report these results.

4. Training-time overhead.

As a concrete example, training Qwen3-1.7B-Base on a single A100 with seed340 and the same training budget took approximately 6.74 hours for RS and 7.71 hours for TCLD, an increase of approximately 58 minutes (14.2%). These training-loop measurements include TCLD’s loss observation and controller computation and exclude evaluation time.

Appendix N provides the timing details and cost breakdown. The learning curves compare accuracy at equal optimizer-step budgets, with the additional observation cost reported separately.

Thank for your additional experiments. I raise the soundness score from 1 to 2. Due to the limited novelty, I keep my overall assessment score.

Reviewer_osk7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Tail-Calibrated Loss-Distribution Shaping (TCLD), a method for improving reasoning SFT by replacing the arithmetic mean of token-level cross-entropy losses with a power-mean objective. The power parameter is dynamically calibrated using Hill-tail statistics estimated from the current training-loss distribution. Experiments on multiple reasoning datasets and model configurations show improvements over standard SFT and the evaluated curriculum-learning baselines.

Critical strengths
  1. Simple and interpretable method.

The proposed power-mean objective provides a straightforward way to adjust the relative gradient contributions of tokens with different loss values. The overall framework is easy to understand and does not require modifications to the model architecture or inference procedure.

Critical weaknesses
  1. Lack of comparisons with recent token-level SFT methods.

Although TCLD primarily modifies token-level loss aggregation during supervised fine-tuning, the main experiments compare it mainly against sample-level curriculum-learning methods, including static and dynamic sample ordering. These baselines address a different aspect of training and may not be the most appropriate competitors for evaluating the proposed loss function.

Recent SFT methods, such as Don't Force the Fit: Bounded Log-Likelihood Loss for Enhanced Reasoning in Large Language Models (BLL-Loss, ICML 2026), also reduce the influence of high-loss or low-probability tokens during reasoning fine-tuning. While the paper includes fixed-power ablations, it lacks direct comparisons with such existing token-level loss optimization methods. These comparisons would be important to establish whether TCLD offers advantages over closely related SFT approaches.

  1. Insufficient differentiation from prior power-mean optimization methods.

The proposed approach also appears closely related to Adapt to Thrive! Adaptive Power-Mean Policy Optimization for Improved LLM Reasoning (APMPO, ACL 2026 Findings). APMPO has already introduced a generalized power-mean objective with an adaptive power parameter for LLM reasoning optimization.

Although APMPO operates in an RL setting rather than SFT, both methods employ power-mean aggregation of token-level optimization terms and adapt the aggregation behavior based on training feedback. The paper does not discuss APMPO or clearly explain how its core methodological contribution differs from this prior work.

The authors should clarify what is fundamentally new beyond applying power-mean aggregation to SFT and replacing the adaptive control signal with Hill-tail statistics.

Questions for authors

Why are the main baselines primarily sample-level curriculum-learning methods rather than recent token-level SFT loss functions? Could the authors provide comparisons with methods such as BLL-Loss under matched training settings?

How does TCLD differ methodologically from APMPO (ACL 2026 Findings), which has already explored adaptive power-mean aggregation for LLM reasoning optimization? What is the specific novelty of the proposed approach beyond the different training paradigm and calibration signal?

1 comment

Thank you for highlighting the need for direct token-level comparisons and a more specific comparison with APMPO. We have addressed both points through additional experiments and an expanded methodological discussion. Revisions addressing your comments are highlighted in blue in the revised manuscript.

1. Comparison with BLL-Loss.

Table 1 compares BLL-Loss with TCLD under the common Qwen3-1.7B-Base five-source fine-tuning configuration. Across seeds 42, 100, and 340, Macro/Micro true maj@3 accuracy (mean ± sample standard deviation) is:

  • RS: 57.09 ± 0.20 / 60.99 ± 0.74
  • BLL-Loss: 57.31 ± 1.53 / 58.83 ± 0.36
  • TCLD: 59.83 ± 0.69 / 63.01 ± 0.26

TCLD’s mean gains over BLL are 2.52 Macro and 4.18 Micro percentage points. Section 4.2 reports the direct comparisons, and Appendix G provides the seed-level correct counts and per-source statistics. DFT and TALR are also included in the main table, extending the evaluation to both sample-level curricula and directly relevant token-level objectives.

2. Relationship to APMPO.

Section 2 and Appendix N now clarify the methodological differences beyond the RL/SFT setting:

  • Aggregation: APMPO aggregates magnitudes of clipped, advantage-weighted token policy-ratio terms within sampled responses. TCLD aggregates teacher-forced response-token cross-entropies within an ordinary training batch.
  • Feedback and update: APMPO sets its power through the reward-conditioned rule p = exp(−γR̄), where R̄ denotes the mean reward. TCLD estimates the current supervised loss tail over an observation window, combines Hill estimates, tracks the statistic using log-state EMA, and maps that state through a sigmoid.
  • Training workflow: APMPO uses sampled rollouts, verifiable rewards, old-policy likelihood ratios, and a reference-policy KL term. TCLD obtains its calibration signal from no-gradient observations of supervised examples.

TCLD builds on power-mean aggregation by introducing a supervised-loss-distribution-driven calibration mechanism. Window-level Hill-tail estimation and log-state tracking determine the aggregation power directly from the current model’s response-token losses, without requiring rollout rewards or a downstream-validation-accuracy search over p. This connects an interpretable control of relative token contributions to an automatic calibration procedure for supervised fine-tuning. The revised methodological comparison clarifies this distinction, while the added BLL, DFT, and TALR experiments demonstrate TCLD’s competitive performance against directly relevant token-level objectives.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.