- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper focuses on finetuning LLMs for better understanding on number tokens. It first shows that two recent numeric losses beyond cross-entropy, NTL and DIST2Loss, share a criterion–distance form , where the criterion term C supports the gradients and the distance term D provides geometric proximity. The authors claim that D is the major source of their failure modes, showing over-sharpened or over-flattened digit distributions. Instead, the proposed method, DEL, removes D and replaces the criterion with a digit-probability-weighted BCE, computed from a sigmoid over one-hot digit set {0,1,...,9}. DEL also introduces a place-dependent weight u(t), linear over integer places and exponential over decimals, to support the floating-point numbers. Experiments across a wide range of LLMs show substantial improvements over previous methods.
Critical strengths
- The presentation is good. It is quite worth treasuring in this AI age.
- The formulation of NTL and DIST2 is informative, making it clear to understand the effectiveness of these losses.
- DEL keeps clean to serve as part of ordinary finetuning loop.
- The experiments seem solid.
Critical weaknesses
- I think the part of Deprecation of the Distance Term is not convincing. I understand what the authors want to show in figure 2, but maybe we can also consider expanding vocab to incorporate digit tokens as well as training a classifier to identify the "23" or the "5" case. Can this be a future direction? And it seems that there is not ablation on the distance term.
- No clear ablation of key components of DEL. DEL adds two things at once: extra gradient on digit positions, and a term that increases with the gold digit's logit. I suggest comparing: cross-entropy with a tuned multiplier on digit positions, and cross-entropy plus . Doing this in a specific case is enough.
- Missing baselines [1-2].
- Typos:
- line 957, "source codes" -> "source code"
- line 157, "non-digital tokens" -> "non-digit tokens"
- Figure 3, "non-digital entropy" -> "non-digit entropy"
- line 721, "digital place" -> "digit place"
- line 083/084 & 225/226, "high entropy tokens" -> "high-entropy tokens"
- line 733, "analysis to" -> "analysis of"
- line 367, "exempler" -> "example"
References
[1] Fei, X., Lu, J., Sun, Q., Feng, H., Wang, Y., Shi, W., Wang, A.-L., Tang, J., and Huang, C. Advancing Sequential Numerical Prediction in Autoregressive Models. ACL 2025 (short papers), pp. 562–574.
[2] Zuo, Z., Yue, L., Zheng, W., Wang, C., and Liu, X. Enhancing Numerical Prediction in LLMs via Smooth MMD Alignment. ICML 2026.
Questions for authors
Please see the weaknesses above.
3 comments
Good rebuttal
My apology to ignore the ablation of Deprecation of the Distance Term in Appendix. Overall, I think the rebuttal make adequate efforts on my comments. After reading other reviewers' comments and the authors' response, I am currently keeping my positive ratings to support this paper.
Personally, I would like to give the authors an additional advice that during the rebuttal period, it would be better if you can incorporate more results & discussions into the revised paper (as ICLR allows to resubmit the pdf).
Thank you for your engagement and for acknowledging that our responses have addressed your questions. We truly appreciate you taking the time to consider the ongoing discussions. During the time being, we will update the manuscript as long as we complete the revision.
Response (1/1)
We sincerely thank the reviewer for the valuable and insightful comments. Below, we address each of the reviewer's concerns in detail.
CW 1: About deprecation of the distance term
We appreciate the reviewer pointing out a promising research direction: expanding vocabulary to include multi‑digit tokens (e.g.,
10,100) and building auxiliary classifiers to identify dual semantics of digits (symbol‑string case like serial‑number"23"versus magnitude‑numeric case like"5"). What we want to show in Fig. 2 and the corresponding analysis is the semantic complexity of numbers. Whether expanding the vocabulary or training additional classifiers, both are attempts to model the semantic complexity of numbers. On this point, we agree with the reviewers that these warrant further studying. Given that, the numerical distance penalty is a simplification of that complexity. This motivates our choice to remove the distance component and let DEL adaptively learn digit distributions purely from training data, without imposing pre‑defined geometric constraints.We admit that the ablation on the distance term was placed in the appendix Table 9 in our original submission, which may reduce its visibility. In the revised manuscript, we will add a concise summary of this ablation within Section 4 “Deprecation of the Distance Term” in the main body, and explicitly point readers to Appendix Table 9 for full experimental details.
$ $
CW 2: Ablation of key components of DEL
We study the diverged components in numerical learning in Appendix Table 9. As suggested by the reviewer, we supplement two more ablations. The first one is cross-entropy with a tuned multiplier on digit positions, which can be written as
L=H(p,g)+u(t)H(p^(⋅∥x<t),gt).
We call this loss function Digit-CE w/ floating-point number optimization.
The second is cross-entropy plus λ(1−p^(gt)), which can be written as
L=H(p,g)+λgt∈Vd∑(1−p^(gt)).
The experimental results are listed in the table below. We also report the results of digit-CE w/o FP number optimization for comparison as it was mentioned by reviewer 2SSx.
The results show that Digit-CE can boost the overall performance by +0.6 accuracy. And floating-point number optimization further improves the performance.
$ $
CW 3: Missing baselines [1-2]
We adopt NTIL's code [1] to train on a Qwen2.5-1.5B model. The training may face the problem of numerical instability, and get a 'nan' loss at the early training procedure. We adjust the NTIL training to use float64. Though the training is slow about 25%, it works. We also incorporate SMMD loss [2] into our training framework. The above two losses are trained on the same training set as our paper adopted. The results are shown in the table below.
Thanks for the suggestion. We will add the comparison of the two methods into the revised manuscripts.
$ $
CW 4: Typos
Thanks for the comment. We will fix these typos accordingly.