acceptodds
Under review as a conference paper at ICLR 2027

DEL: Digit Entropy Loss for Numerical Learning of Large Language Models

Abstract

Number prediction stands as a fundamental capability of large language models (LLMs) in mathematical problem-solving and code generation. The widely adopted maximum likelihood estimation (MLE) for LLM training is not tailored to number prediction. Recently, penalty-driven approaches, e.g., Number Token Loss and Discretized Distance Loss, introduce an inductive bias of numerical distance but induce over-sharpened and over-flattened digit distributions, respectively. In this paper, we make an in-depth analysis on LLM numerical learning, and show that existing numerical learning methods conceptually follow a criterion-distance formulation, where the criterion term represents optimization pattern and the distance term instills geometric prior. Consequently, we present Digit Entropy Loss (DEL) for auto-regressive numerical learning, which reformulates the conventional unsupervised entropy optimization in three key designs: leveraging digit conditional probability and binary cross-entropy to guide the entropy optimization into a supervised manner; deprecating the distance term to bypass the issue of numerical distance; and generalizing the integer-based numerical learning to floating-point number optimization, enabling more accurate number prediction. Our DEL formulation can incorporate integers, decimals, and decimal points, expanding the learning objective from a single digit to the floating-point number domain. Experiments conducted on seven mathematical reasoning benchmarks with four representative LLMs, including CodeLlama, Mistral, DeepSeek, and Qwen-2.5, demonstrate that DEL consistently outperforms its counterparts in both overall prediction accuracy and numerical distance. Source codes and trained models will be publicly available.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Numerical Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tXSSOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper focuses on finetuning LLMs for better understanding on number tokens. It first shows that two recent numeric losses beyond cross-entropy, NTL and DIST2Loss, share a criterion–distance form CE+α∑C(p^)⋅D(Vd,gt)CE+\alpha\sum C(\hat p)\cdot D(V_d,g_t), where the criterion term C supports the gradients and the distance term D provides geometric proximity. The authors claim that D is the major source of their failure modes, showing over-sharpened or over-flattened digit distributions. Instead, the proposed method, DEL, removes D and replaces the criterion with a digit-probability-weighted BCE, computed from a sigmoid over one-hot digit set {0,1,...,9}. DEL also introduces a place-dependent weight u(t), linear over integer places and exponential over decimals, to support the floating-point numbers. Experiments across a wide range of LLMs show substantial improvements over previous methods.

Critical strengths
  1. The presentation is good. It is quite worth treasuring in this AI age.
  2. The formulation of NTL and DIST2 is informative, making it clear to understand the effectiveness of these losses.
  3. DEL keeps clean to serve as part of ordinary finetuning loop.
  4. The experiments seem solid.
Critical weaknesses
  1. I think the part of Deprecation of the Distance Term is not convincing. I understand what the authors want to show in figure 2, but maybe we can also consider expanding vocab to incorporate digit tokens as well as training a classifier to identify the "23" or the "5" case. Can this be a future direction? And it seems that there is not ablation on the distance term.
  2. No clear ablation of key components of DEL. DEL adds two things at once: extra gradient on digit positions, and a term that increases with the gold digit's logit. I suggest comparing: cross-entropy with a tuned multiplier on digit positions, and cross-entropy plus λ(1−p^(gt))\lambda(1-\hat p(g_t)). Doing this in a specific case is enough.
  3. Missing baselines [1-2].
  4. Typos:
  • line 957, "source codes" -> "source code"
  • line 157, "non-digital tokens" -> "non-digit tokens"
  • Figure 3, "non-digital entropy" -> "non-digit entropy"
  • line 721, "digital place" -> "digit place"
  • line 083/084 & 225/226, "high entropy tokens" -> "high-entropy tokens"
  • line 733, "analysis to" -> "analysis of"
  • line 367, "exempler" -> "example"

References

[1] Fei, X., Lu, J., Sun, Q., Feng, H., Wang, Y., Shi, W., Wang, A.-L., Tang, J., and Huang, C. Advancing Sequential Numerical Prediction in Autoregressive Models. ACL 2025 (short papers), pp. 562–574.

[2] Zuo, Z., Yue, L., Zheng, W., Wang, C., and Liu, X. Enhancing Numerical Prediction in LLMs via Smooth MMD Alignment. ICML 2026.

Questions for authors

Please see the weaknesses above.

3 comments
AuthorsOpenReview ↗

Response (1/1)

We sincerely thank the reviewer for the valuable and insightful comments. Below, we address each of the reviewer's concerns in detail.

CW 1: About deprecation of the distance term

We appreciate the reviewer pointing out a promising research direction: expanding vocabulary to include multi‑digit tokens (e.g., 10, 100) and building auxiliary classifiers to identify dual semantics of digits (symbol‑string case like serial‑number "23" versus magnitude‑numeric case like "5"). What we want to show in Fig. 2 and the corresponding analysis is the semantic complexity of numbers. Whether expanding the vocabulary or training additional classifiers, both are attempts to model the semantic complexity of numbers. On this point, we agree with the reviewers that these warrant further studying. Given that, the numerical distance penalty is a simplification of that complexity. This motivates our choice to remove the distance component and let DEL adaptively learn digit distributions purely from training data, without imposing pre‑defined geometric constraints.

We admit that the ablation on the distance term was placed in the appendix Table 9 in our original submission, which may reduce its visibility. In the revised manuscript, we will add a concise summary of this ablation within Section 4 “Deprecation of the Distance Term” in the main body, and explicitly point readers to Appendix Table 9 for full experimental details.

DEL in Eq. 7Distance term in NTLmACC
✓\checkmark55.1
✓\checkmark✓\checkmark54.3

$ $

CW 2: Ablation of key components of DEL

We study the diverged components in numerical learning in Appendix Table 9. As suggested by the reviewer, we supplement two more ablations. The first one is cross-entropy with a tuned multiplier on digit positions, which can be written as

L=H(p,g)+u(t)H(p^(⋅∥x<t),gt).\mathcal{L} = \mathcal{H}(\mathbf{p}, \mathbf{g}) + u(t)\mathcal{H}(\mathbf{\hat{p}}(\cdot\|x_{<t}), \mathbf{g}_t).

We call this loss function Digit-CE w/ floating-point number optimization.

The second is cross-entropy plus λ(1−p^(gt))\lambda(1-\mathbf{\hat{p}}(\mathbf{g}_t)), which can be written as

L=H(p,g)+λ∑gt∈Vd(1−p^(gt)).\mathcal{L} = \mathcal{H}(\mathbf{p}, \mathbf{g}) + \lambda\sum\limits_{\mathbf{g}_t\in\mathcal{V}_d}(1-\mathbf{\hat{p}}(\mathbf{g}_t)).

The experimental results are listed in the table below. We also report the results of digit-CE w/o FP number optimization for comparison as it was mentioned by reviewer 2SSx.

MethodmACCGSM8KMATHSVAMPSimulEqAQuASATMMLU
MLE52.854.042.371.644.245.356.455.5
Digit-CE53.453.940.572.044.747.660.954.2
Digit-CE w/ FP number optm.54.155.642.373.343.650.758.654.8
1−p^(gt)1-\mathbf{\hat{p}}(\mathbf{g}_t)53.254.341.672.745.346.157.755.0
DEL (Eq. 7)55.155.842.973.347.549.261.455.9

The results show that Digit-CE can boost the overall performance by +0.6 accuracy. And floating-point number optimization further improves the performance.

$ $

CW 3: Missing baselines [1-2]

We adopt NTIL's code [1] to train on a Qwen2.5-1.5B model. The training may face the problem of numerical instability, and get a 'nan' loss at the early training procedure. We adjust the NTIL training to use float64. Though the training is slow about 25%, it works. We also incorporate SMMD loss [2] into our training framework. The above two losses are trained on the same training set as our paper adopted. The results are shown in the table below.

MethodmACCGSM8KMATHSVAMPSimulEqAQuASATMMLU
MLE52.854.042.371.644.245.356.455.5
NTL-WAS53.855.143.573.346.143.159.156.3
NTIL54.255.042.174.444.646.761.854.8
SMMD53.955.342.273.944.749.258.653.5
DEL (Eq. 7)55.155.842.973.347.549.261.455.9
DEL (Eq. 7) w/ FP number optm.55.457.042.674.048.148.860.956.1

Thanks for the suggestion. We will add the comparison of the two methods into the revised manuscripts.

$ $

CW 4: Typos

Thanks for the comment. We will fix these typos accordingly.

Reviewer_tXSSOpenReview ↗

Good rebuttal

My apology to ignore the ablation of Deprecation of the Distance Term in Appendix. Overall, I think the rebuttal make adequate efforts on my comments. After reading other reviewers' comments and the authors' response, I am currently keeping my positive ratings to support this paper.

Personally, I would like to give the authors an additional advice that during the rebuttal period, it would be better if you can incorporate more results & discussions into the revised paper (as ICLR allows to resubmit the pdf).

Thank you for your engagement and for acknowledging that our responses have addressed your questions. We truly appreciate you taking the time to consider the ongoing discussions. During the time being, we will update the manuscript as long as we complete the revision.

Reviewer_2SSxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Digit Entropy Loss (DEL), an auxiliary objective for improving numerical generation in autoregressive language models. Experiments across multiple model families and mathematical reasoning benchmarks show consistent gains in average accuracy, supported by analyses of numerical errors, entropy, scaling, hyperparameters, and loss components. However, the mechanism behind DEL remains insufficiently established, particularly regarding the necessity of probability weighting and the claimed benefits of its floating-point extension.

Critical strengths
  1. Consistent gains under largely controlled comparisons. DEL improves mean accuracy across all five model configurations spanning four model families, with roughly 0.9–1.6 percentage-point gains over the strongest numerical-loss baseline. Reporting three-run averages and variability further strengthens the empirical evidence. The method is also attractive in that it only changes the training objective and requires no additional inference procedure.

  2. Analysis extends beyond aggregate benchmark accuracy. The paper includes numerical-error analysis, entropy measurements, and several ablations. In particular, comparisons with direct digit-entropy minimization and experiments adding positional weighting to NTL provide useful evidence toward understanding the source of the gains.

Critical weaknesses
  1. DEL’s specific mechanism is not sufficiently isolated. DEL simultaneously introduces digit-specific supervision, sigmoid BCE, and prediction-dependent softmax weighting. Missing controls—such as digit-conditional cross-entropy, digit-weighted standard cross-entropy, and unweighted digit BCE—make it difficult to determine whether the gains come from DEL’s formulation or simply from increased supervision on numerical tokens. The conceptual distinction from NTL is also not fully convincing. Although the paper argues that DEL avoids persistent sharpening induced by fixed distance penalties, Equation 15 shows that NTL gradients also vanish as non-target probabilities approach zero. Thus, this argument alone does not establish that DEL better avoids over-sharpening. Likewise, Figure 2’s non-monotonic representation distances do not by themselves show that numerical-distance priors are harmful.

  2. The floating-point contribution is under-supported and some implementation details are unclear. The proposed floating-point objective remains a sum of position-weighted token losses, while Appendix C acknowledges that it does not jointly model the semantics of an entire number. The current experiments do not directly evaluate fractional-digit accuracy, decimal placement, carries, or numerically equivalent representations. Moreover, since the auxiliary loss operates over digits only, the treatment of decimal points is unclear. Reproducibility would also benefit from clearer descriptions of number-span detection, tokenizer alignment, auxiliary-loss normalization, invalid numerical outputs, dataset-overlap checks, and hyperparameter-selection protocols.

Questions for authors
  1. How are numerical spans and place values identified and aligned with different tokenizers, especially for tokens containing multiple digits or mixed characters? How are decimal points, minus signs, and auxiliary-loss normalization handled?

  2. Can the authors add controlled baselines using digit-conditional cross-entropy and unweighted digit BCE with the same positional weighting and tuning budget?

  3. For Figure 3, what entropy is measured exactly, under what decoding/teacher-forcing setting, and how is it averaged? Please also distinguish the auxiliary penalty from the behavior of the full training objective.

  4. Can the authors report dedicated results on fractional-number examples, including fractional-digit accuracy or numerical error with and without positional weighting? Please also clarify the apparent column mismatch between Tables 4 and 7, the validation protocol, and overlap checks between MathInstruct and evaluation sets.

7 comments
AuthorsOpenReview ↗

Response (1/5)

We sincerely thank the reviewer for the valuable and insightful comments. Below, we address each of the reviewer's concerns in detail.

CW 1-A: DEL’s specific mechanism

Reponse: In our study, we have considered several forms of loss function, some of which overlapped with those proposed by the reviewer. We summarize each objective function as follows:

  • (A). Digit-conditional cross-entropy (also mentioned by Reviewer R6u9): Ldigit-CE=H(p,g)+αRdigit-CE=H(p,g)+αH(p^,gt)\mathcal{L}_{\text{digit-CE}} = \mathcal{H}(\mathbf{p},\mathbf{g}) + \alpha\mathcal{R}_{\text{digit-CE}} = \mathcal{H}(\mathbf{p},\mathbf{g}) + \alpha\mathcal{H}(\hat{\mathbf{p}},\mathbf{g}_t)

  • (B). Digit-weighted MLE: Ldigit-weighted-MLE=∑g∉VdH(p(⋅∣x<t),g)+α∑gt∈VdH(p(⋅∣x<t),gt)\mathcal{L}_{\text{digit-weighted-MLE}} = \sum\limits_{\mathbf{g}\notin V_d}\mathcal{H}(\mathbf{p}(\cdot|x_{<t}),\mathbf{g}) + \alpha\sum\limits_{\mathbf{g}_t\in V_d}\mathcal{H}(\mathbf{p}(\cdot|x_{<t}),\mathbf{g}_t)

  • (C). Unweighted digit BCE: Ldigit-BCE=H(p,g)+αRdigit-BCE=H(p,g)+α∑gt∈VdBCE(p˙(⋅∣x<t),yt)\mathcal{L}_{\text{digit-BCE}} = \mathcal{H}(\mathbf{p},\mathbf{g}) + \alpha\mathcal{R}_{\text{digit-BCE}} = \mathcal{H}(\mathbf{p},\mathbf{g}) + \alpha\sum\limits_{\mathbf{g}_t\in V_d}\mathbf{BCE}(\dot{\mathbf{p}}(\cdot|x_{<t}), y_t)

  • (D). Digital Focal Loss (Eq. 19): LFL=H(p,g)+α∑kAkγBk=H(p,g)+α∑k(yk(1−p˙)+(1−yk)p˙)γ(−yklog⁡p˙k−(1−yk)log⁡(1−p˙k))\mathcal{L}_{FL} = \mathcal{H}(\mathbf{p},\mathbf{g}) + \alpha\sum\limits_k A_k^\gamma B_k = \mathcal{H}(\mathbf{p},\mathbf{g}) + \alpha\sum\limits_k\Big(y_k(1-\dot{\mathbf{p}}) + (1-y_k)\dot{\mathbf{p}}\Big)^\gamma \Big(-y_k\log\dot{\mathbf{p}}_k - (1-y_k)\log\big(1-\dot{\mathbf{p}}_k\big)\Big)

  • (E). Shannon entropy Optimization (Eq. 6): LDE=H(p,g)+α∑gt∈VdH(p^(⋅∣x<t))=H(p,g)−α∑gt∈Vdu(t)p^(⋅∣x<t)log⁡p^(⋅∣x<t)\mathcal{L}_{\text{DE}}= \mathcal{H}(\mathbf{p},\mathbf{g}) + \alpha\sum\limits_{\mathbf{g}_t\in V_d}\mathcal{H}(\hat{\mathbf{p}}(\cdot|x_{<t})) = \mathcal{H}(\mathbf{p},\mathbf{g}) - \alpha\sum\limits_{\mathbf{g}_t\in V_d}u(t)\hat{\mathbf{p}}(\cdot|x_{<t}) \log \hat{\mathbf{p}}(\cdot|x_{<t})

  • (F). DEL (Eq. 7): LDEL=H(p,g)+α∑gt∈Vdp^(⋅∣x<t)BCE(p˙(⋅∣x<t),yt)\mathcal{L}_{DEL} = \mathcal{H}(\mathbf{p},\mathbf{g}) + \alpha\sum\limits_{\mathbf{g}_t\in V_d} \hat{\mathbf{p}}(\cdot|x_{<t})\mathbf{BCE}(\dot{\mathbf{p}}(\cdot|x_{<t}), y_t)

  • (G). DEL (Eq. 7) w/ numerical distance (an ablation experiment in Table 9): LDEL=H(p,g)+α∑gt∈Vd∣∣Vd−gt∣∣1p^(⋅∣x<t)BCE(p˙(⋅∣x<t),yt)\mathcal{L}_{DEL} = \mathcal{H}(\mathbf{p},\mathbf{g}) + \alpha\sum\limits_{\mathbf{g}_t\in V_d} ||V_d-\mathbf{g}_t||_1 \hat{\mathbf{p}}(\cdot|x_{<t})\mathbf{BCE}(\dot{\mathbf{p}}(\cdot|x_{<t}), y_t)

We list the benchmark results below.

MethodmACCGSM8KMATHSVAMPSimulEqAQuASATMMLU
MLE52.854.042.371.644.245.356.455.5
(A) Digit-CE53.453.940.572.044.747.660.954.2
(B) Digit-weighted MLE52.954.641.873.144.945.554.555.7
(C) Unweighted digit BCE53.054.342.771.542.646.555.957.3
(D) Digital Focal Loss (Eq. 19)53.055.943.072.844.745.354.554.8
(E) Shannon entropy Optimization (Eq. 6)54.254.642.472.747.747.260.554.2
(F) DEL (Eq. 7)55.155.842.973.347.549.261.455.9
(G) DEL (Eq. 7) w/ numerical distance54.354.441.971.645.950.460.056.0

Discussion:

It can be seen that the above digit supervision can improve the performance. It is worth noting that method B, which separates digit supervision from the original MLE, needs careful hyper-parameter tuning, i.e., the loss weight α\alpha. If we balance the two loss terms so as to make the model focus number more, in most cases, the accuracy drops. Therefore, using digit conditional probability is generally better and more stable than using the full space probability.

Together with the observation from NTL and DIST2Loss papers, adding supervision on numerical tokens can generally achieve better performance for auto-regressive LLM. We believe that our DEL offers a better objective in numerical learning, which creates an adaptive regularizer coupled with the digit-conditional probability and the digit binary classification rather than a static penalty.

AuthorsOpenReview ↗

Response (2/5)

CW 1-B: Conceptual distinction from NTL

Reponse: We acknowledge that the gradient of NTL vanishes when non-target digit probabilities approach zero, which is a property shared by DEL. In fact, this is a common property of smooth loss functions when the prediction approaches its optima. The critical distinction is, their gradient flows and optimization dynamics are qualitatively different.

For NTL, the digit-conditional probability p^\mathbf{\hat{p}} weights a fixed, hand-specified L1 numerical distance. As we analyzed in Section A.2 "Analysis to NTL", for some non-target digits with dk=9d_k=9, the gradient weight is larger than 9p^g9\hat{\mathbf{p}}_g, i.e., 9 times of the target-digit probability. This rigid relative weighting continuously biases the gradient flow toward the externally imposed numeric distance. In contrast, DEL couples p^\mathbf{\hat{p}} with BCE, whose gradient magnitude is determined by the model’s own current binary probability p˙\mathbf{\dot{p}}. The regularization of DEL is adaptive: when the model perform well on the binary classification task "Is kk a target digit?", the gradient can be reduced.

This structural difference is also empirically demonstrated by the digit entropy reported in Figure 3. Under identical training configurations, NTL generally yields lower digital entropy, showing its stronger tendency to sharpen the digit distribution.

$ $

CW 1-C: Are numerical-distance priors harmful?

Reponse: We clarify that we have never claimed numerical-distance priors are inherently harmful. Admittedly, we deprecate the fixed distance term in our design because it introduces a rigid numeric ordering constraint that conflicts with our target regularization behavior. This does not amount to a universal claim that such priors are harmful in general — they may still be beneficial under other settings or tasks.

Notably, NTL and DIST2Loss instantiate distance priors into distinct optimization objectives, which guide learned digit distributions toward qualitatively different behaviors, such as sharpening or flattening. This suggests that the numerical distance prior has a rich design space: with careful formulation, it can be tailored to induce the exact characteristic that one desires, thus endowing the model with new capabilities.

We will revise the manuscript to make it clear.

$ $

CW 2-A: The contribution of floating-point number optimization

1. The proposed floating-point number optimization remains a sum of position-weighted token losses. Can we jointly model the semantics of an entire number?

Reponse: Yes, we can. Establish a loss function between two holistic floating-point numbers is not complicate. A straightforward approach is to impose a MSE loss between the predicted number and the ground-truth number. To be specific, for a simple example, suppose the ground-truth sequence is "John gave me 10.5 dollar." There are 4 digit-relevant tokens: "1", "0", ".", "5". At the corresponding token positions, we take the 4 digit-conditional probability distributions on the digit vocabulary space Vd\mathcal{V}_d. Then, we obtain the predict digits by taking indexes which have the maximum probability. (Note that we can replace the argmax function with a generalized softmax function S(⋅,τ)\mathcal{S}(\cdot,\tau) that has a small temperature τ\tau and then calculate the expectation to avoid gradient vanishing.) We call this naive approach Argmax Holistic Number Loss. The results are shown in the table below. Unfortunately, this approach leads to a significant performance degradation, even though we tuned the loss weight α\alpha for it.

MethodmACC
MLE52.8
Argmax Holistic Number Loss36.6
Approximate Holistic Number Loss53.0

Another approach is to approximate the overall value of a floating-point number by calculating the expectation of each digit and multiplying it directly by the digit places. For example, suppose the model predict a digit probability distribution [0,0,0,0.2,0.8,0,0,0,0,0][0,0,0,0.2,0.8,0,0,0,0,0] in the hundreds place. We calculate the expectation by E=0×0+1×0+2×0+3×0.2+4×0.8+5×0+6×0+7×0+8×0+9×0=3.8.E = 0\times 0 + 1\times 0 + 2\times 0 + 3\times 0.2 + 4\times 0.8 + 5\times 0 + 6\times 0 + 7\times 0 + 8\times 0 + 9\times 0 = 3.8. We use this EE, i.e., 3.8, to approximate the original predicted digit, which should be 4. Then we multiply EE with its corresponding digit place, 100. For every digit place, we did the same and sum them up. As before, we use MSE loss for the criterion. We call this approach Approximate Holistic Number Loss. The results show that this method achieve a slight 0.2 accuracy gain over MLE. Despite improvement, the method is still inferior to the token-level numerical learning methods. We speculate that this may be due to the inevitable introduction of estimation errors caused by holistic numerical supervision, while the token-level numerical learning methods provide precise numerical supervision for each digit token.

AuthorsOpenReview ↗

Response (3/5)

2. Evaluations on fractional-digits

Reponse: We carefully consider the suggestion raised by the reviewer. In the following, we supplement evaluations on fractional-digits. We use Qwen2.5-1.5B as the base model, and evaluate on NumGLUE [r1] datasets since it output fractional numbers. We report the tenths and the hundredths results. Assessments of lower digit places were excluded due to few samples and lack of statistical significance. MAE and Accuracy are measured on the fractional-digit predictions. Decimal placement Accuracy, as suggested by the reviewer, measures the accuracy of the decimal point's position within the holistic number.

  • Numerical distance evaluation | Method | 10−110^{-1} | 10−210^{-2} | MAE | |:----|:----:|:----:|:----:| | MLE | 1.34 | 1.31 | 1.33 | | MixCE | 1.29 | 1.58 | 1.41 | | EMO | 1.45 | 1.39 | 1.42 | | NTL | 1.34 | 1.39 | 1.36 | | DIST2Loss | 1.31 | 1.47 | 1.38 | | DEL (Ours) | 1.13 | 1.14 | 1.14 | | DEL (Ours) w/o FP number optm. | 1.24 | 1.36 | 1.27 |

  • Numerical accuracy evaluation | Method | 10−110^{-1} | 10−210^{-2} | Acc. | Decimal placement Acc. | |:----|:----:|:----:|:----:|:----:| | MLE | 62.8 | 59.1 | 61.2 | 79.4 | | MixCE | 66.3 | 57.6 | 62.5 | 82.3 | | EMO | 62.3 | 56.0 | 59.6 | 79.4 | | NTL | 66.3 | 60.6 | 63.8 | 84.0 | | DIST2Loss | 64.6 | 54.5 | 60.3 | 84.6 | | DEL (Ours) | 70.9 | 62.1 | 67.1 | 86.3 | | DEL (Ours) | 68.8 | 61.3 | 64.5 | 84.4 |

The results show that the floating-point number optimization can produce positive gains on fractional digits, while DIST2Loss adopts the place weighting mechanism without the consideration of floating-point numbers and yields accuracy drops. A visual comparison of the place weighting between DIST2Loss and DEL can be found in Figure 4. Our method imposes a continuous penalty weight on the transition point between ones place and tenths place. We will add the above results to our paper.

[r1] Mishra S, Mitra A, Varshney N, et al. NumGLUE: A suite of fundamental yet challenging mathematical reasoning tasks. In ACL 2022: 3505-3523.

$ $

3. Evaluations on decimal placement

Reponse: Thanks for the suggestion. We supplement evaluation on decimal placement. The results are shown in the tables above. It shows that our method (Eq. 7) produces accuracy improvement on decimal placement, which is comparable to NTL and DIST2Loss. When we incorporate floating-point number optimization, decimal placement accuracy can be further improved.

$ $

4. About digit carrying

Reponse: Regarding carry accuracy, we agree that this is a valuable fine-grained metric for evaluating arithmetic behavior. However, it cannot be directly computed from final answer outputs alone. Identifying carry errors requires explicit ground-truth annotations of intermediate reasoning steps (e.g., per-column addition and carry propagation), which are not available in our benchmark datasets. Without step-level labels, we cannot reliably distinguish whether a wrong final answer is caused by a failed carry, incorrect digit prediction, or other reasoning errors.

$ $

5. About evaluation of numerically equivalent representations

Reponse: In our experiments, we already adopt numerical‑equivalence checking for final scalar answers: we compare predictions against ground‑truth values with floating‑point tolerance, following common practice in GSM8K and MATH. For GSM8K, SVAMP, SimulEq, they remove commas and handle trailing zeros before comparing two numbers. And MATH uses math verify to clean up the various LaTeX formats. Nevertheless, we highlight that our floating-point number optimization target a different research goal: Reduce the sensitivity of digit prediction to errors at different digit levels. It is not designed to evaluate general algebraic semantic equivalence across alternative mathematical forms (e.g., comparing 0.250.25 with 0.15+0.10.15+0.1, 1/41/4, one-quarter or zero point two five, etc.).

$ $

6. The treatment of decimal points

Reponse: In our method, we handle the decimal point by ignoring it, which means the decimal-token position produces a zero loss.

Reviewer_R6u9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new objective for autoregressive language models that improves the prediction of numerical tokens. In a first step, the work analyses methods proposed in previous literature and introduce a unified formulation described as criterion-distance numerical learning. The authors then propose a new loss function by removing the inductive bias of a distance function between tokens provided by previous works replacing it with a minimum entropy objective. The loss function is clearly motivated by the desire of determinacy of number prediction and the removal of geometric prior. The authors further propose a solution for the objective to handle floating point numbers. An extensive empirical analysis across code and mathematical tasks is performed to show the effectiveness of the method.

Critical strengths
  • The proposed loss leads to consistently improved empirical performance across different parameter sizes of four different models and a variety of different datasets covering different math and code generation tasks.
  • The paper presents a useful generalization and analysis of existing methods for improved numerical predictions.
  • The proposed method avoids reliance on a geometric prior or a soft target and thus enables to capture varying semantic information contained in numbers.
Critical weaknesses
  • The proposed loss function is not shown to capture the motivating property as an entropy minimizer. While eq.6 includes the Shannon entropy, it is not clear whether the final objective proposed in eq.7 still has similar properties to an entropy and can function as a proxy objective. Just sharing the same global optimum does not yet clearly establish that connection.
  • A simple baseline of using an additional CE loss just over the numerical tokens would be useful to judge if the more complicated construction in eq.7 is necessary, or if the method's performance is mainly a result of additional supervision. This ablation would further strengthen the usefulness of the proposed method.
Questions for authors
  • The entropy as written down in eq.6 appears well motivated together with the goal of particularly sharp distributions for numerical outputs. However, it is not clear if the objective in eq.7 still measures something similar to an entropy. Could you provide some intuition about what the loss actually measures, aside from stating that it is uniquely minimal at the one-hot target distribution?
  • A simple baseline of an additional cross-entropy term just over the number tokens (cross entropy between the target distribution and p_hat) would also capture the desired properties of a sharper distribution over numerical tokens and would also retain the unique minimum. Does the proposed method outperform this simple baseline?
  • How does the method handle alternative number representations? The same number in the text could be described as "3 million" or as "3.000.000". This case would lead to different weights, despite the equivalent semantic meaning of the leading "3" token. While the method does resolve some ambiguity regarding the semantic meaning of numbers, this issue seems to be not fully resolved.
  • The paper states that NTL distributions are overly sharp, could you clarify what the difference between the desired low-entropy distribution and a sharp distribution is?
  • The gradients in Figure 1 for the MLE objective should show that it also suppresses the probabilities of all non-target tokens as the softmax couples the prediction probabilities across the vocabulary.
5 comments
AuthorsOpenReview ↗

Response (1/2)

We sincerely thank the reviewer for the valuable and insightful comments. Below, we address each of the reviewer's concerns in detail.

CW 1: Relationship between DEL and Shannon Entropy Minimization

While DEL (Eq.7) draws design inspiration from this entropy‑minimizing intuition of Eq. 6 and aims to produce low‑entropy digit distributions at convergence, we do not formally claim that Eq.7 mathematically acts as a proxy for Shannon‑entropy minimization along optimization trajectories. There are two reasons.

  • First, pure entropy minimization (Eq. 6) suffers from many false global minima. Since it lacks ground‑truth supervision, it can collapse to one‑hot distributions over wrong digits, all of which yield zero entropy. Therefore, constructing a plain entropy‑minimizing proxy is not our objective.

  • Second, even when the optimization target is set to the correct digit distribution, we should not expect the Eq. 7 penalty term to strictly decrease entropy at every single training step. In fact, any supervised objective targeting the correct digit distribution can yield temporary entropy increases during optimization. As a simple illustration: suppose the target digit initially holds only 0.05 probability, while the remaining 0.95 probability mass is allocated across the nine wrong digits. When the objective boosts the probability of the target digit and suppresses wrong‑digit probabilities, probability mass can be redistributed such that the overall distribution becomes temporarily more uniform, leading to a temporary rise in Shannon entropy. Consistent with this view, temporary increases in digit entropy can be observed in our experiments (see Figure 3(b), comparing MLE and DEL).

We do not expect DEL to enforce trajectory‑level monotonic entropy reduction. Instead, we expect it to learn the correct digit distribution under supervision, and low‑entropy behavior emerges at convergence.

$ $

CW 2: A simple baseline: additional CE loss over the numerical tokens

Thanks for the suggestion. We write down the total loss function Ldigit-CE=H(p,g)+αH(p^,gt)\mathcal{L}_{\text{digit-CE}} = \mathcal{H}(\mathbf{p},\mathbf{g}) + \alpha\mathcal{H}(\hat{\mathbf{p}},\mathbf{g}_t). We note that this loss function is equivalent to DIST2Loss when setting τ=0\tau=0 in forming a target distribution qt\mathbf{q}_t (refer to Eq. 2 and Eq. 5). This is a special case of DIST2Loss when removing the numerical distance from the target distribution of qt\mathbf{q}_t. We report the benchmark results for this loss function, as shown below.

MethodmACCGSM8KMATHSVAMPSimulEqAQuASATMMLU
MLE52.854.042.371.644.245.356.455.5
Digit-CE53.453.940.572.044.747.660.954.2
DIST2Loss53.555.142.573.646.946.155.055.6
DEL (Ours)55.155.842.973.347.549.261.455.9

For a fair comparison, we remove the place weighting mechanism for these losses to study the effect of the pure penalty term. The results show that the digit CE loss leads to an improvement in mACC compared to MLE, but is slightly lower than DIST2Loss.

$ $

Q1: Intuition about what DEL measures

We thank the reviewer for this insightful question. Eq. 6 is standard Shannon entropy loss, whose intuition is straightforward: it directly penalizes probability mass spread across multiple digits and favors sharp, concentrated distributions, yet it lacks ground-truth supervision and can collapse to wrong digits.

Eq. 7 (DEL penalty) is not a direct measure of Shannon entropy. Its core intuition can be understood as a self-weighted binary supervision objective:

∑gt∈Vdp^(⋅∣x<t)BCE(p˙(⋅∣x<t),yt)\sum\limits_{\mathbf{g}_t\in V_d} \hat{\mathbf{p}}(\cdot|x_{<t})\mathbf{BCE}(\dot{\mathbf{p}}(\cdot|x_{<t}), y_t) For each digit candidate xtx_t, we form a binary prediction task: whether this digit is the ground-truth target gt\mathbf{g}_t. Unlike Shannon entropy, which penalizes uniform probability mass regardless of content, DEL penalizes incorrect classification on non-target digits. Unlike standard cross-entropy, where the weighting is fixed by the one-hot label, DEL’s weighting is dynamically determined by the model’s current predicted distribution. In one word, DEL's penalty term measures the model’s binary classification error for each digit candidate, weighted by the model’s own predicted confidence on that candidate. This is another semantic expression of digital low entropy: namely, the binary classification task has low error implies that low digital entropy. However, the inverse proposition does not hold true, e.g. under the optimization under Shannon entropy minimization.

$ $

Q2: Does the proposed method outperform this simple baseline?

Please see the response above CW 2: A simple baseline: additional CE loss over the numerical tokens.

AuthorsOpenReview ↗

Response (2/2)

Q3: The alternative number representations

DEL operates on all the explicit numbers. For a wide variety of numerical representations, such as "3 million", "3,000,000", "3000000", 3.0×1063.0\times 10^6, "3M", "3e6", etc., our implementation identifies digit tokens in the texts and applies DEL to these digit tokens. For example, given a text "John won a $3 million prize and plans to buy property", the numerical learning method (NTL, DIST2Loss, DEL) identifies the number "3" and applies the loss on this digit token. The other texts are supervised by the standard MLE. Currently, numerical learning methods do not perform special processing on numbers with the same semantics but different representations.

$ $

Q4: What is the difference between low-entropy distribution and a sharp distribution?

Mathematically, low entropy and sharpness are closely related: a low-entropy distribution is necessarily concentrated. Our distinction lies in how this concentrated behavior is induced, rather than the final shape of the distribution.

When we refer to NTL distributions as overly sharp, we mean the following observable property: NTL strongly penalizes probability mass on digits that are far away from the target in the hand‑designed metric space. As a result, the learned distribution tends to concentrate nearly all probability on the target digit while suppressing nearly all neighboring digit candidates to near‑zero probability. The sharpness is enforced by the metric structure embedded in the loss, regardless of whether the model has learned the underlying numerical reasoning.

In contrast, the low‑entropy behavior we aim for with DEL means that probability mass concentrates on the correct digit at convergence, but without an inductive bias that arbitrarily suppresses nearby non-target digits. DEL’s penalty is therefore adaptive: it only becomes severe when a non-target digit has both large softmax probability and large sigmoid probability.

$ $

Q5: The visualization of MLE in Figure 1

Thanks for the comment. We will revise Figure 1 to show the implicit effect of MLE on non-target digits.

Thank you for the clarifications and running further ablations. It is interesting and encouraging to see that the method empirically outperforms the ablation using additional supervision on the numerical tokens only.

However, I would still like to obtain a better understanding of the exact mechanism that leads to these gains. In your response to Q1, you compare to both a pure entropy loss and cross-entropy supervision. I understand the motivation of incorporating target labels, such that the loss guides towards the correct target distribution in contrast to a pure entropy-based formulation.

It remains less clear to me, where the advantage compared to an additional cross-entropy supervision comes from. I understand how the loss itself is constructed and how the formula differs from CE. However, both losses target the same low-entropy distribution and CE’s gradients also depend on the current predicted distribution, so the „dynamic weighting“ alone does not fully explain the difference.

Could you explain or give another intuition how DEL changes the optimization behaviour in comparison to CE, and which mechanisms lead to the improved performance?

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We sincerely appreciate all three reviewers for their constructive feedback and valuable suggestions on our submission. We are encouraged that the Reviewers valued the informative unified formulation of prior numerical‑learning losses and rich multi‑faceted analyses including entropy metrics and ablations (R6u9, 2SSx, tXSS). The reviewers highlighted our "consistent gains" and "solid" experimental results across diverse model families and scales (R6u9, 2SSx, tXSS).

In our response, we have addressed the Reviewers' concerns as follows:

1. Clarifications on Methodology.

  • Relationship between DEL and Shannon Entropy Minimization (R6u9).

  • Conceptual distinction from NTL (R6u9, 2SSx).

  • Numerical-distance prior is not harmful, but is incompatible in our DEL (2SSx).

2. Ablation studies.

  • DEL’s specific mechanism (2SSx, tXSS), including additional CE loss over the numerical tokens (R6u9).

  • More evaluation on floating-point number optimization (2SSx).

  • Ablation on the distance term (tXSS).

3. Implementation details.

  • The alternative number representations (R6u9).

  • Reproducibility supplements (2SSx).

We believe these additional experiments and clarifications significantly strengthen the paper and demonstrate DEL's soundness for numerical learning in auto-regressive LLMs. We thank the reviewers again for their careful reading and insightful comments, which have helped us substantially improve this work.

Hi reviewers,

Thank you for your reviewing service. The authors have provided fairly extensive rebuttals. Please confirm that you have read them, state which of your concerns are or are not addressed, and whether that changes your recommendations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.