acceptodds
Under review as a conference paper at ICLR 2027

NuAlign: A Numerical Distance Perception Alignment for Large Language Models as Evaluators

Abstract

Large language models (LLMs) have shown huge potential as automatic evaluators, but the standard token-based likelihood objectives don't explicitly encode the numerical distance between scores. Previous research introduced numerical supervision through expected score regression or token-level numerical loss; however, the former can't distinguish distributions with the same average, and the latter doesn't differentiate the digit positions. To address this, we propose NuAlign, a numerical distance perception alignment for LLMs as evaluators that directly uses the full score deviations reconstructed during training to update the likelihood of the true score sequences. NuAlign consists of two stages. In Stage I, we learn to evaluate semantic and scoring boundaries through parameter-efficient supervised fine-tuning, thereby establishing a task-aware autoregressive scoring distribution. In Stage II, we utilize numerical scoring deviations to modulate token-level likelihood optimization, introducing numerical distance sensitivity while preserving the native autoregressive scoring distribution. We conducted experiments across multiple evaluation datasets and model families, achieving an MAE of 0.28 on the Mohler dataset (out of a maximum score of 5). Our code and data are available at: https://anonymous.4open.science/r/I-LOVE_ICLR-7561

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Automatic Evaluators, Numerical Distance Perception, Likelihood Optimization, Two-Stage Alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5VB3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes NuAlign, a two-stage method for aligning LLMs used as evaluators. Stage I performs task-aware supervised fine-tuning. Stage II introduces Numerical Distance Perception (NDP), where the full-score deviation is used to modulate the likelihood gradient of gold score tokens. The method is evaluated across three languages, five LLM backbones, and several scoring datasets.

Critical strengths
  1. The method is simple and practical, and the evaluation covers three datasets in different languages and five diverse LLM backbones.
  2. The adaptive cross-weighting rule avoids expensive grid searches for the mixing coefficient.
Critical weaknesses

The theoretical analysis does not sufficiently establish how the proposed reweighting addresses the limitations motivating the paper. NDP introduces numerical distance through a scalar weight on gold-token likelihood gradients, but does not directly impose distance-dependent corrections among alternative score tokens. Moreover, distinguishing equal-mean distributions may already follow from ordinary likelihood supervision. The paper therefore needs clearer evidence that its gains arise specifically from the proposed numerical-distance mechanism rather than generic error-based reweighting.

Questions for authors
  1. How does NuAlign compare with simple numerical-error-weighted CE under matched backbones and training budgets?

  2. Could you clarify its relationship to REAL (Zhang et al., ICML 2026), and provide a comparison or explain why the settings are not directly comparable?

    Zhang et al., "REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge," ICML 2026.

1 comment

Thank you very much for your valuable feedback. Next, I will first address the issues you raised in the “Critical Defects” section; we have broken down your feedback into two main issues and will respond to them one by one.

W1: NDP introduces numerical distance by applying a scalar weight to the likelihood gradient of the gold label, but does not directly apply distance-based corrections between candidate score labels.

Thank you for pointing out the distinction between NDP and distance-based corrections for candidate score labels. We agree that NDP does not apply numerical distance penalties separately to each candidate score label at the same prediction position. According to Equation (9), for valid supervision positions, the NDP gradient is $d_i \nabla_\theta \ell_{i,t}^{\text{CE}} $, so it preserves the local direction of the original likelihood gradient and modulates the gradient contribution for that sample using the bias between the reconstructed full score and the human-assigned score. Our design goal is to incorporate the severity of the full-score error into likelihood optimization, rather than establishing new distance constraints in the candidate score space. For multi-token scoring, the full-score deviation can distinguish between predictions with similar token-level losses but differing degrees of numerical error severity. Therefore, “numerical distance-aware” in this paper specifically refers to the dependence of the sample-level optimization signal on the full-score error; it does not imply that corrections for each error candidate are ranked according to their numerical distance. We will further revise the method’s motivation and related descriptions to clarify this scope of application.

W2: Distinguishing between distributions with equal means may already be achieved by ordinary likelihood supervision.

We are very grateful to the reviewers for this point. We agree that when two distributions with equal means assign different probabilities to the true score sequence, standard likelihood supervision may already be capable of distinguishing between them. In fact, §3.4 of the original manuscript (page 4, lines 194–198) explicitly states that TRACT’s additional likelihood supervision can distinguish between distributions that assign different probabilities to the true sequence. We have also further demonstrated in Appendix A.3.1 that NuAlign possesses a unique ability to distinguish between distributions with equal means. NuAlign’s contribution lies in utilizing the bias in reconstructing whole scores to modulate the likelihood updates for the true scoring sequence, rather than treating the discriminative capability already present in standard likelihood supervision as a new contribution. We will revise the relevant description to avoid ambiguity.

Reviewer_fUpGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NuAlign introduces a two-stage training method for autoregressive judges. Following supervised fine-tuning, it uses the difference between reconstructed predictions and human scores to adaptively weight the training of gold-score tokens. Experiments demonstrate reduced scoring errors and improved agreement with human scores.

Critical strengths
  1. The method has a clear motivation and a coherent design, incorporating numerical error into the training of autoregressive judges through sample weighting.

  2. The experiments provide reasonable evidence of effectiveness, with component ablations and additional-training controls supporting the contributions of the proposed design.

Critical weaknesses
  1. The necessity of the combined Stage II design is not fully established. NDP already weights gold-score CE gradients by numerical error. Retaining ordinary CE, introducing cross-weighting, and allowing gradients through λ add further adjustments. Existing ablations support the complete method, but do not isolate the additional contributions of these choices over using NDP alone.

  2. Training and evaluation focus on direct scoring under a fixed output template, leaving the preservation of language generation and explanation abilities untested. Since score-token training updates shared model parameters, it remains unclear whether the resulting judge can reliably produce coherent, reasonable explanations consistent with its scores.

Questions for authors
  1. With the same Stage I initialization and Stage II training budget, how does CE+NDP compare with NDP alone? The existing ONDP ablation removes Stage I and therefore does not isolate this comparison.

  2. What motivates the cross-weighting formula and the decision to backpropagate through λ? The fixed-weight comparisons support adaptive weighting, but have you compared against dynamically computing λ while stopping gradients through it, to isolate the benefit of differentiating through the weight?

  3. After two-stage fine-tuning, does the model retain its basic language generation ability, particularly the ability to produce coherent, reasonable explanations consistent with its scores? Have you assessed changes relative to the original model or the Stage I checkpoint?

1 comment

Thank you very much for your valuable feedback. Next, I will respond the issues you raised in the “Critical Weaknesses” section.

W1:The necessity of the combined Stage II design is not fully established. NDP already weights gold-score CE gradients by numerical error. Retaining ordinary CE, introducing cross-weighting, and allowing gradients through λ add further adjustments. Existing ablations support the complete method, but do not isolate the additional contributions of these choices over using NDP alone.

Thank you very much for pointing out that the contributions of the various components in Stage II have not yet been sufficiently disentangled. We have retained the original intent of standard CE to continue the direct likelihood supervision based on human-scored responses from Stage I, rather than allowing updates in Stage II to be determined solely by numerical error weights. NDP and standard CE are not two independent prediction objectives; NDP uses the full score deviation to modulate the likelihood gradient of the scoring tokens. Therefore, the Stage II objective aims to balance both the likelihood fit to the human-scored sequence and the sensitivity to numerical errors.

To further examine the roles of the various components in Stage II, we conducted two additional sets of experiments based on Qwen3-8B: (1) NDP_Only: Retaining only NDP in the Stage II objective while removing CE and cross-weighting; (2) λ_NoGradients: Removing the gradient of λ. The experimental results are shown in the table below.

MethodMohler MAEMohler RMSEMohler PearsonAES-ENEM MAEAES-ENEM RMSEAES-ENEM PearsonChinese-QA MAEChinese-QA RMSEChinese-QA Pearson
NDP_Only0.320.610.8158.5886.010.830.551.040.91
λ_NoGradients0.340.590.8359.31887.590.820.571.080.91
NuAlign0.320.570.8457.3385.240.840.540.980.93

As shown in the table, compared to NDP-only, NuAlign reduced the RMSE on Mohler, AES-ENEM, and Chinese-QA from 0.61, 86.01, and 1.04 to 0.57, 85.24, and 0.98, respectively; the Pearson correlation coefficients all improved; and the MAE decreased across all datasets except Mohler, where it remained unchanged. These results support the overall benefits of the combined design compared to using NDP alone. Furthermore, NuAlign achieves superior performance on all three datasets when computing gradients with dynamic λ, further supporting the incremental benefit of computing weight gradients with dynamic λ. These results complement existing ablation studies with fixed weights and independently validate the overall value of NuAlign.

W2:Training and evaluation focus on direct scoring under a fixed output template, leaving the preservation of language generation and explanation abilities untested.

Thank you very much for pointing this out. We agree that the scoring results under a fixed template are insufficient to demonstrate that language generation and interpretation capabilities have been preserved. To address this, we compared the Original backbone, Stage-I checkpoint, and the full NuAlign model without any additional training by performing direct inference on the test sets of three datasets using Qwen3-8B under the same prompt and decoding settings. We evaluated the models using three metrics: the non-empty explanation completion rate (NEE), the average number of output tokens (NOT), and MAE. The experimental results are shown in the table below.

MethodMohler NEEMohler NOTMohler MAEAES-ENEM NEEAES-ENEM NOTAES-ENEM MAEChinese-QA NEEChinese-QA NOTChinese-QA MAE
Original100%78.270.32100%108.16147.62100%51.632.612
Stage I92.14%68.280.3474.61%76.4564.1597.80%42.580.666
NuAlign94.11%71.690.3285.45%86.9959.3298.40%43.340.628

The experimental results show that NuAlign achieved non-empty explanation rates of 94.11%, 85.45%, and 98.40% on Mohler, AES-ENEM, and Chinese-QA, respectively all higher than those of Stage I, while the average output length also increased. This indicates that NuAlign is still capable of generating explanations for most samples, though the non-empty explanation rate has not yet reached the original model’s 100%. We limit this evidence to the model’s explanation output behavior in the scoring task and do not equate it with the full preservation of general language ability or explanation quality.

Reviewer_QzDROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper is about LLM's loss design when used as evaluators. The paper argues that CE loss only cares about gold label. The loss does not rely on 'how good the answer is', the scores. Existing numerical supervision methods all have limitations.

The paper proposed a two step method:

  1. standard SFT, but only supervise the final score. So model learns to score with certain format and score anchor.
  2. Teacher forcing the sequence but only get argmax for final score. Use the difference between the 'predicted' score and reference score as a weight to scale the CE loss. Larger gap means higher weight.

The experiment used three short answer/essay datasets with different languages

Critical strengths
  • method deriavation is complete.
Critical weaknesses
  • The baselines compared are not fair. Most mentioned baselines sft evaluators use different backbone models. When compared to only SFT baseline, the advantages are not strong enough.

  • There are many previous works that use deviation penalty to adapt loss, such as NTIL or DISTLoss, but they are not compared.

  • The 'evaluator' domain is not 'LLM-as-Judge', which only focus on very narrow essay rating. The compared dataset/benchmarks do not have good enough coverage.

  • To show the proposed method is a better training method for LLM evaluator. Broad range of tasks and baselines should be considered and compared.

Questions for authors
  • Did you run baseline methods with the same backbone LLM and report scores from there?
1 comment

Thank you very much for your valuable feedback. Next, I will respond the issues you raised in the “Critical Weaknesses” section.

W1: The baselines compared are not fair. Most mentioned baselines sft evaluators use different backbone models. When compared to only SFT baseline, the advantages are not strong enough.

Thank you very much for your valuable feedback. We understand your concerns; we did consider this issue when conducting our comparative experiments. In order to preserve the backbone used in the original paper for some of the methods, we employed a hybrid setup in our baseline configuration. We acknowledge that this setup does not fully isolate differences in the backbones; therefore, we have added a controlled comparison using a unified Qwen3-8B model. The experimental results are shown in the table below.

ModelsMohler MAE ↓Mohler RMSE ↓Mohler Pearson ↑AES-ENEM MAE ↓AES-ENEM RMSE ↓AES-ENEM Pearson ↑Chinese-QA MAE ↓Chinese-QA RMSE ↓Chinese-QA Pearson ↑
MTS---193.71215.940.582.252.790.73
Praetor1.842.160.41---1.081.930.74
CEM-FT0.40--62.4794.560.800.701.150.88
RAFT0.450.670.7661.1484.880.840.581.030.92
NTL0.400.690.7864.0391.090.820.571.090.92
TRACT0.450.800.7564.2190.770.820.661.310.88
Qwen3-8B (ours)0.32 ± 0.000.57 ± 0.010.84 ± 0.0057.33 ± 0.0685.24 ± 0.010.84 ± 0.000.54 ± 0.030.98 ± 0.040.93 ± 0.02
Qwen3-14B (ours)0.28 ± 0.010.50 ± 0.030.88 ± 0.0157.21 ± 0.0883.92 ± 0.150.84 ± 0.000.53 ± 0.050.97 ± 0.080.93 ± 0.01
Llama-3.1-8B-Instruct (ours)0.29 ± 0.000.53 ± 0.020.86 ± 0.0157.21 ± 0.0083.45 ± 0.150.85 ± 0.100.55 ± 0.021.00 ± 0.050.93 ± 0.01
Mistral-7b-instruct-v0.3 (ours)0.34 ± 0.010.61 ± 0.030.82 ± 0.0057.58 ± 0.1486.40 ± 0.250.85 ± 0.050.62 ± 0.031.06 ± 0.050.92 ± 0.03
Gemma2-2B (ours)0.42 ± 0.010.72 ± 0.050.77 ± 0.0158.20 ± 0.0585.15 ± 0.090.84 ± 0.020.56 ± 0.011.00 ± 0.030.93 ± 0.02

It is worth noting that in our original comparative experiments, only RAFT and TRACT were not based on Qwen3-8B; the new experimental results do not alter our original conclusions. Furthermore, compared to the SFT baseline, NuAlign’s results on the Qwen3-8B-based experiments showed that the MAE on the Mohler, AES-ENEM, and Chinese-QA datasets decreased from 0.34 to 0.32, from 60.92 to 57.33, and from 0.71 to 0.54, respectively representing reductions of 5.88%, 5.89%, and 23.94%, respectively.

W2: There are many previous works that use deviation penalty to adapt loss, such as NTIL or DISTLoss, but they are not compared.

Thank you very much for your valuable feedback. Based on Qwen3-8B, we conducted two additional comparative experiments using NTIL and DIST²Loss. The experimental results are shown in the table below.

MethodMohler MAEMohler RMSEMohler PearsonAES-ENEM MAEAES-ENEM RMSEAES-ENEM PearsonChinese-QA MAEChinese-QA RMSEChinese-QA Pearson
DIST²Loss0.60.930.6687.64115.230.70.651.150.9
NTIL0.40.70.7874.05103.960.770.591.110.91
NuAlign0.320.570.8457.3385.240.840.540.980.93

As shown in the table, NuAlign outperforms both NTIL and DIST²Loss in terms of MAE, RMSE, and Pearson correlation across all three datasets. Compared to the stronger-performing NTIL, NuAlign reduces the MAE by 20.00%,22.58%, and 8.47% on the Mohler, AES-ENEM, and Chinese-QA datasets, respectively. The simultaneous improvement in error metrics and correlation indicates that these gains are not achieved at the expense of the linear relationship between predictions and human scores, nor are they solely the result of correcting overall score offsets. From the perspective of its optimization mechanism, NuAlign treats the deviation from the reconstructed full score as a sample-level modulation signal; while retaining likelihood supervision from the true scoring sequence, it adjusts the training contribution based on the severity of numerical errors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.