acceptodds
Under review as a conference paper at ICLR 2027

Reinforcement Learning over Predictive Distributions for LLM Regression

Abstract

Large language models (LLMs) have emerged as flexible regressors capable of predicting real-valued quantities from heterogeneous inputs. Yet most LLM regression objectives optimize predictions independently, often yielding poor calibration. We introduce Distribution-Aware Reward (DAR), an on-policy reinforcement learning objective that instead jointly evaluates the empirical predictive distribution formed by multiple predictions for the same input. To translate this distribution-level objective into rollout-level rewards, we assign each prediction credit based on its leave-one-out contribution to the quality of the overall predictive distribution. This encourages predictions that are well-centered and appropriately dispersed around the target. We evaluate on three regression settings: a synthetic task probing interpolation and extrapolation, and two real-world scientific tasks involving code and molecular data. Across tasks, DAR produces better-calibrated uncertainty estimates while consistently reducing prediction error and improving ranking quality over supervised fine-tuning and pointwise reinforcement learning. Together, these results highlight the benefits of distribution-aware training for LLM regression.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Regression, Reinforcement Learning, Predictive Distributions

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FqsWOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Distribution-Aware Reward (DAR), a method based on reinforcement learning for fine-tuning LLMs as numerical regressors. DAT optimises the predictive distributions of LLMs using Continuous Ranked Probability Score (CRPS) to evaluate distribution quality. Leave-one-out credit assignment is used to reward individual predictions according to their contribution to the overall distribution quality. DAR is evaluated on synthetic regression, code performance prediction, and molecular property prediction tasks. Results show improvements in prediction accuracy, ranking performance, and uncertainty calibration compared with supervised fine-tuning and conventional pointwise RL objectives. The paper also investigates predictive diversity, calibration, and alternative reward formulations through additional analyses and ablations.

Critical strengths

The paper proposes an interesting distribution-aware RL objective based on CRPS and leave-one-out credit assignment that seems to perform well. It improves prediction accuracy and ranking performance over standard SFT and pointwise RL baselines across several tasks.

Critical weaknesses

W1. Fairness and completeness of the regression baselines.

The paper compares DAR against a value-head regression baseline and several alternative training objectives (Gaussian NLL, Gaussian CRPS, and quantile loss) in Table 7. However, the implementation of the latter baselines is unclear. Are these objectives applied to token-level predictions from an LLM fine-tuned end-to-end (like in DAR), or to a regression head operating on LLM representations?

Appendix D.2 indicates that the value-head baseline uses a frozen LLM backbone with a two-layer MLP regression head, whereas DAR fine-tunes the LLM end-to-end. This creates a potential confound. It is unclear if the improvements arise from a better training objective or increased compute budget and model capacity.

I would like to see a more controlled comparison against probabilistic regression heads that explicitly predict both the conditional mean and variance, trained with different objectives, such as Gaussian NLL, CRPS and the quantile loss. These baselines should be evaluated under both (i) a frozen LLM backbone with a trainable probabilistic head and (ii) end-to-end fine-tuning of the LLM and the probabilistic head. All comparisons should also report uncertainty calibration metrics (e.g. prediction-interval coverage), rather than only point prediction and ranking metrics.

W2. Motivation for autoregressive numerical prediction.

The paper proposes an interesting distribution-aware RL objective for improving the calibration and accuracy of numerical predictions generated by LLMs. However, I find the motivation for retaining numerical predictions in the autoregressive token space not sufficiently justified.

For the scalar regression tasks considered in this paper, it is unclear why one should prefer autoregressive numerical generation followed by relatively complex RL fine-tuning over a probabilistic regression head attached to the LLM's representations. Such a head could directly predict the parameters of a conditional distribution and be optimized end-to-end (as proposed in W1).

I do see a stronger motivation for the proposed approach in settings where the LLM must jointly generate natural language and numerical predictions, for example when providing explanations, reasoning, or longer textual responses containing numerical estimates.

However, all the main experiments consider simple scalar numerical outputs. Appendix H investigates generating intermediate reasoning before numerical predictions but it is limited to a very narrow setting and the additional reasoning steps do not lead to improved predictive performance (which seems insufficiently explored).

I would encourage the authors to clarify the advantages of autoregressive numerical prediction over end-to-end probabilistic regression heads. Ideally, the paper should either provide more experiments on tasks involving joint text and numerical generation, where the autoregressive formulation can be more clearly justified.

W3. Computational efficiency compared against an expensive reference point

The authors state that DAR has negligible computational overhead (Appendix D.5, Table 5). This is true relative to pointwise GRPO, because both methods already require 12 rollouts per training input. But DAR is not computationally more efficient compare to supervised dine-tuning of a probabilistic head. Furthermore, at test time, DAR uses 32 sampled predictions, while a probabilistic regression head could produce the distributional parameters in a single forward pass.

Questions for authors

See weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer FqsW [1/3]

We thank the reviewer for the encouraging assessment and thoughtful suggestions. We have carefully addressed the concerns and revised the manuscript accordingly. Below, we provide point-by-point responses to each comment.

[W1] Fairness and Completeness of Regression Baselines

We thank the reviewer for helping us clarify the fairness of our baseline comparisons. The original head-based baselines followed Tang et al.’s frozen-backbone setup [1]. The Value Head baseline uses L2 loss. For Gaussian NLL, Gaussian CRPS, and quantile loss, we replaced it with the corresponding prediction heads operating on the frozen LLM representations.

We have added end-to-end versions of all three probabilistic baselines. Both the Qwen3-4B-Instruct backbone and prediction head were jointly fine-tuned using the same setup as SFT. These comparisons address the difference in backbone trainability; training costs are discussed separately below.

MethodOutputKBSSAPPS
Gaussian NLLContinuous0.5830.621
Gaussian CRPSContinuous0.5750.679
QuantileContinuous0.5610.610
DAR (Ours)Autoregressive0.6600.960

Table 1. Code performance prediction (Qwen3-4B-Instruct): Spearman correlation (↑). Continuous baselines directly output a predictive distribution: Gaussian parameters trained with Gaussian NLL or Gaussian CRPS, or quantiles trained with the pinball loss (Quantile). DAR generates numeric predictions autoregressively.

MethodOutputESOLFreeSolvLIPO
Gaussian NLLContinuous0.8942.3970.780
Gaussian CRPSContinuous0.9182.2910.825
QuantileContinuous0.8752.4020.804
DAR (Ours)Autoregressive0.8712.3600.776

Table 2. Molecular property prediction (Qwen3-4B-Instruct): RMSE (↓). Same setting as Table 1.

DAR achieves the strongest ranking performance on both coding tasks and the lowest reported RMSE on LIPO and ESOL. Thus, DAR remains competitive when the probabilistic heads are also trained end-to-end.


We additionally report 90% prediction-interval coverage error using the same calibration procedure for all methods.

MethodOutputKBSSAPPSESOLFreeSolvLIPO
Gaussian NLLContinuous0.0540.0970.0120.0080.014
Gaussian CRPSContinuous0.0440.0950.0120.0080.010
QuantileContinuous0.0210.0950.0030.0230.002
SFTAutoregressive0.0540.1010.3250.2540.040
Pointwise MSE RLAutoregressive0.5490.4330.1120.4690.405
DAR (Ours)Autoregressive0.0150.0100.0420.0380.045

Table 3. Calibration of continuous and autoregressive models (Qwen3-4B-Instruct): 90% coverage error (↓). Coverage error is |empirical coverage of the central 90% prediction interval − 0.90|; lower is better.

DAR achieves the lowest coverage error on both coding tasks. It also outperforms pointwise MSE RL on all five datasets and SFT on four. However, calibration results are mixed across tasks. Continuous regression heads achieve lower coverage error on molecular tasks, highlighting their potential advantages in modeling continuous-valued outputs.

Nevertheless, DAR achieves the best calibration among autoregressive methods on four of five datasets, making it particularly useful for tasks that require generating both text and numerical predictions. We demonstrate this advantage in LLM-as-a-Judge tasks next. We have added the results and discussion to Appendix F.2.

AuthorsOpenReview ↗

Response to Reviewer FqsW [2/3]

[W2] Motivation for Autoregressive Numerical Prediction

The reviewer raises an interesting point about the value of probabilistic regression heads. Such heads may be simpler for scalar-only prediction tasks. Our work focuses on autoregressive numerical prediction, which provides a unified interface for generating reasoning, explanations, and numerical estimates. A broader comparison between these approaches is an interesting direction for future work.

Our goal is not to replace regression heads in all settings, but to improve the quality and calibration of numerical predictions within the autoregressive generation framework of existing LLMs [2,3,4]. This framework supports reasoning, explanations, and numerical estimates in a unified manner. DAR enables distribution-aware optimization without introducing a separate numerical prediction head.

To demonstrate the benefit of this formulation beyond scalar regression, we now evaluate DAR on ordinal LLM-as-a-Judge tasks. These tasks require models to generate both reasoning and numerical scores, providing a natural setting for autoregressive numerical prediction.

Following REAL [2], we initialize from Mistral-7B-TRACT, train on Feedback Collection, and evaluate on FLASK, MT-Bench, and Vicuna. We replace REAL's L2-based regression reward with DAR (CRPS on the mixture of score distributions plus a squared-error term on its mean); all other hyperparameters remain unchanged.

We re-evaluate the official TRACT [3] and REAL checkpoints using the same inference pipeline: one reasoning trace followed by the expected score from score-token probabilities. RAFT’s published correlations are included for reference because its checkpoint is unavailable [4].

MethodFLASKMT-BenchVicunaAvg.
RAFT41.839.952.844.8
TRACT50.452.356.353.0
REAL56.058.964.159.7
DAR (Ours)57.658.966.661.1

Table 4. LLM-as-a-Judge (Mistral-7B), out-of-domain benchmarks: Pearson correlation (↑). RAFT results are taken from [2]; all other models are evaluated by us using REAL's inference protocol. REAL and DAR are initialized from TRACT.

MethodFLASKMT-BenchVicunaAvg.
TRACT1.5401.2671.1001.303
REAL1.3531.1450.8291.109
DAR (Ours)1.1811.0430.7110.978

Table 5. LLM-as-a-Judge (Mistral-7B), out-of-domain benchmarks: RMSE (↓). Same setting as Table 4. RAFT is omitted because its RMSE is not reported in [2] and no checkpoint is available.

DAR improves average Pearson correlation by 1.4 points and reduces average RMSE by 0.13 relative to REAL. It improves Pearson correlation on FLASK and Vicuna, matches REAL on MT-Bench, and achieves lower RMSE on all three benchmarks.


We also evaluate the uncertainty of the approaches. We report RPS, the discrete counterpart of CRPS for ordered score distributions, and IS90, which penalizes interval width and targets outside the 90% prediction interval. Lower is better for both, and all methods use the same calibration procedure.

MethodRPS (↓)IS90 (↓)
TRACT0.8226.33
REAL0.6444.17
DAR (Ours)0.6073.66

Table 6. LLM-as-a-Judge (Mistral-7B), out-of-domain benchmarks: predictive distribution quality, RPS (↓) and IS90 (↓). Results are averaged over FLASK, MT-Bench, and Vicuna. RPS (ranked probability score) evaluates the full predicted score distribution. IS90 (90% interval score) evaluates the central 90% prediction interval by its width plus a penalty for missing the target.

DAR reduces RPS by 5.7% and IS90 by 12.2% relative to REAL. These results demonstrate improvements in both numerical prediction and distributional quality in a task that combines explanations and scores. They provide a concrete use case for distribution-aware training within the existing autoregressive generation framework.

We have clarified this motivation and added the judge experiments to the main text in Section 4.3 and Section 5.1.

AuthorsOpenReview ↗

Response to Reviewer FqsW [3/3]

[W3] Computational Cost

We agree that the reference point for our efficiency claim should be explicit. DAR’s negligible overhead is relative to pointwise GRPO under the same budget of 12 rollouts per input.

Training cost. In our Qwen3-4B experiments on four H100 GPUs, DAR requires approximately 3x the wall-clock training time of SFT and end-to-end regression-head training. The additional cost mainly comes from rollout generation and RL training.

Inference cost. For continuous regression, DAR averages 32 sampled numerical predictions. We measured the sampling cost using Qwen3-4B on a single L40S GPU, with 16 LIPO inputs and three repetitions.

For a single input, generating 32 predictions in parallel took 0.593 seconds, compared to 0.332 seconds for one prediction. This represents a 1.79x increase in median latency, demonstrating that parallel generation substantially reduces the latency overhead of multiple samples.

We have clarified the scope of our efficiency claim and added the training-cost comparison and task-specific inference measurements in Appendix D.7.

References

[1] Tang et al., Understanding llm embeddings for regression, TMLR 2025

[2] Zhang et al., REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge, ICML 2026

[3] Chiang et al., TRACT: Regression-Aware Fine-tuning Meets Chain-of-Thought Reasoning for LLM-as-a-Judge, ACL 2025

[4] Lukasik et al., Better Autoregressive Regression with LLMs via Regression-Aware Fine-Tuning, ICLR 2025

Reviewer_5qxPOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper tackles the question of LLM regression: predicting real-valued quantities from heterogeneous inputs.
  • It discusses the limitation of standard methods: SFT optimizing cross-entropy over numerical target’s string, or RL with negative MSE as the reward. Both treat individual predictions “pointwise”, meaning they are evaluated independently of the other samples, and thus the objective cannot capture distributional properties of the predictive distribution
  • They instead propose DAR: a score based on Continuous Ranked Probability Score (CRPS) which is classically used to evaluate a predictive distribution. Concretely, they use the negative empirical CRPS as a set-level reward, to be optimized using a set-based policy gradient estimator with leave-one-out marginal contribution.
  • They evaluate this on three tasks: a didactic synthetic distribution regression task, coding performance prediction, and molecular property prediction. Across these tasks they found better predictive accuracy measured by lower error, and higher rank correlation to the ground truth
  • After post-hoc recalibration, they found better correlation of variance with error, and also better calibration of the 90% prediction interval, indicating the predictive distribution are better calibrated
Critical strengths

I found this paper’s problem clearly stated, the limitation of existing methods clearly motivated, and the solution to be simple and principled. Overall, I think the direction of using foundational models to do prediction is an important and timely problem, and doing so in a calibrated fashion can open up many applications.

(With the caveat that I am not familiar with the LLM regression area, and cannot accurately judge the novelty of this paper in light of other works in the area)

Critical weaknesses

Training efficiency versus SFT

Based on my reading of the results, SFT and RL with MSE are roughly comparable in terms of accuracy (Tables 1 & 2), though SFT appear to have much better calibration (Figures 3 and 4). While the overall accuracy & calibration results are nevertheless in favour of DAR, I’m wondering if SFT enjoys more efficient training (both in terms of number of gradient steps to achieve a certain level of accuracy & calibration, and overall wallclock time since there is no sampling step)?

This is not to discredit DAR’s usefulness when the objective has the resources to be well-optimized, but to better understand cases where one may wish to trade off learning efficiency with prediction performance (e.g. if one wishes to predict a non-stationary target).

Misc

Some less critical comments about continuous value prediction and related works on the gradient estimator are discussed in the questions section below.

Questions for authors

Q1: CRPS with a continuous-value head

Can the author discuss optimizing the CRPS objective with a continuous-value head, and why is this perhaps less preferable vs. the proposed method (optimize a discrete token output distribution using policy gradient)?

Based on my current understanding, the “Value Head” approach in Table 1 and 2 is a continuous-value head optimized using MSE. Table 3 and 7 does contain a Gaussian CRPS baseline which I believe is close to what I am asking for above. Is the Gaussian CRPS trained with the backbone frozen, or full model unfrozen? Do you have some discussion on why using a continuous prediction head with a CRPS based objective might perform worse than doing discrete token prediction? Intuitively, the continuous head should be optimizable with much lower gradient variance as compared to a policy gradient based method?

Q2: Novelty of the policy gradient estimator

Regarding L472: “leave-one-out credit assignment differs from RLOO”. This is true. However, I think it’s worthwhile to note that the proposed set-based policy gradient with leave-one-out contribution does exist [1, 2, 3]; and more generally, optimizing over sets has been explored in works such as [4]. I think the two literatures are quite related from a “optimize for distributional properties” perspective and can benefit from each other (e.g. it may be that the calibration-based scores can benefit from further variance reduction using the control variates derived in the above works, and the set based RL folks can benefit from more objectives from the calibration literature, etc.). I’m also open to hearing if the authors believe their policy gradient estimator has novel components (beyond the set level objective being different).

Q3: Calibration before recalibration

Figure 4 reports the 90% interval coverage after post-hoc variance recalibration from Laves et al. Could the authors expand more on what is actually done to recalibrate (apologies if I missed this in the appendix)?

I am trying to understand if this implies the raw prediction calibration is less reliable, and for instance a separate held-out set is always needed for the calibration to be meaningful? To me this would not discredit the method but is a meaningful limitation to outline. Perhaps reporting the 90% interval coverage error before post-hoc recalibration would be useful?

Misc

Q4:: What is the dataset used to generate the result in Figure 3?

Q5:: I appreciate the significance tests in Tables 1 and 2. Could the authors nevertheless provide for e.g. confidence intervals or standard deviations for each reported result?


[1] Tang, Yunhao, et al. "Optimizing language models for inference time objectives using reinforcement learning." arXiv preprint arXiv:2503.19595 (2025).

[2] GX-Chen, Anthony, et al. "Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learning." arXiv preprint arXiv:2606.03962 (2026).

[3] Takashiro, Shota, et al. "On Advantage Estimates for Max@ K Policy Gradients." arXiv preprint arXiv:2606.06080 (2026).

[4] Orney, Ifdita Hasan, et al. "Poly-EPO: Training Exploratory Reasoning Models." arXiv preprint arXiv:2604.17654 (2026).

6 comments
AuthorsOpenReview ↗

Response to Reviewer 5qxP [1/2]

We appreciate the reviewer for the encouraging assessment and thoughtful suggestions. We have addressed the concerns and revised the manuscript accordingly. Below, we respond to each point.

[W1] Computational Cost: SFT vs. RL

We agree that computational efficiency is an important consideration and have clarified this trade-off in the revised paper. SFT is more efficient in wall-clock time. In our Qwen3-4B experiments on four H100 GPUs, SFT and RL-based methods show similar convergence patterns in both point prediction performance and calibration when measured by gradient steps. However, RL-based methods, including DAR, take approximately 3x longer to reach their best performance, mainly because of rollout generation and additional log-probability computations and policy updates.

The practical trade-off is between training cost and final predictive performance. SFT is an attractive option when compute is limited or frequent updates are needed, as with non-stationary targets. DAR requires more training time but achieves better final accuracy and calibration in our experiments. We have added the training-time comparison and clarified this trade-off in Appendix D.7.

[Q1] Continuous prediction heads and autoregressive regression

1. DAR remains competitive with continuous heads empirically.

We appreciate the reviewer’s insightful suggestion. A continuous prediction head can directly optimize CRPS with lower gradient variance, making it a promising alternative. DAR instead focuses on optimizing the pretrained LLM’s token distribution without introducing an additional regression head or changing the model architecture. This also makes DAR applicable to tasks like LLM-as-a-Judge, where textual reasoning and numerical scores are generated together.

The original Gaussian CRPS baseline in Tables 3 and 7 (Tables 5 and 18 in the revision) used a frozen backbone. We therefore added end-to-end experiments with Gaussian NLL and CRPS heads, jointly fine-tuning the Qwen3-4B-Instruct backbone and regression head.

MethodOutputKBSSAPPS
Gaussian NLLContinuous0.5830.621
Gaussian CRPSContinuous0.5750.679
DAR (Ours)Autoregressive0.6600.960

Table 1. Code performance prediction (Qwen3-4B-Instruct): Spearman correlation (↑). Continuous baselines output the parameters of a Gaussian predictive distribution, trained with Gaussian NLL or Gaussian CRPS; DAR generates numeric predictions autoregressively.

MethodOutputESOLFreeSolvLIPO
Gaussian NLLContinuous0.8942.3970.780
Gaussian CRPSContinuous0.9182.2190.825
DAR (Ours)Autoregressive0.8712.3600.776

Table 2. Molecular property prediction (Qwen3-4B-Instruct): RMSE (↓). Continuous baselines output the parameters of a Gaussian predictive distribution, trained with Gaussian NLL or Gaussian CRPS; DAR generates numeric predictions autoregressively.

DAR achieves stronger ranking performance on both coding tasks and the lowest RMSE on LIPO and ESOL. These results support continuous heads as a useful alternative while showing that DAR remains competitive without adding a regression head.


2. DAR supports joint generation of text and numerical predictions.

Another motivation for DAR is to retain autoregressive generation of both text and numerical predictions. This allows DAR to integrate naturally with reasoning LLMs that generate intermediate reasoning steps before numerical predictions. LLM-as-a-Judge is one such setting, where models generate reasoning traces followed by quality scores.

We therefore added judge experiments following REAL’s setup [1]. Starting from Mistral-7B-TRACT [2], we train on Feedback Collection and evaluate on FLASK, MT-Bench, and Vicuna. We replace REAL's L2-based regression reward with DAR (CRPS on the mixture of score distributions plus a squared-error term on its mean), keeping REAL's hyperparameters unchanged. We re-evaluate the official TRACT and REAL checkpoints using the same inference pipeline. We report Pearson correlation, the standard evaluation metric.

MethodFLASKMT-BenchVicunaAvg.
TRACT50.452.356.353.0
REAL56.058.964.159.7
DAR (Ours)57.658.966.661.1

Table 3. LLM-as-a-Judge (Mistral-7B), out-of-domain benchmarks: Pearson correlation (↑). All models are evaluated using REAL's inference protocol.

DAR matches or improves on REAL across all three benchmarks, increasing average Pearson correlation from 59.7 to 61.1. These experiments demonstrate its usefulness when numerical prediction is part of an autoregressive text output.

We have clarified the head-training configurations and added these comparisons and discussion in Section 4.3 and Section 5.1.

AuthorsOpenReview ↗

Response to Reviewer 5qxP [2/2]

[Q2] Relation to set-based policy gradients

We acknowledge that set-based policy gradients and leave-one-out credit assignment have been studied in prior work. We do not claim a new policy-gradient estimator. Our contribution is to use a CRPS-based distributional objective for autoregressive LLM regression and demonstrate its effectiveness for numerical prediction and uncertainty estimation.

Our comparison with RLOO concerns the reward construction: DAR measures each rollout’s contribution to the score of the full prediction set. We now present this distinction alongside its connection to existing set-based estimators.

We have revised the discussion in related work to clarify this scope and added the suggested references in L521. We also agree that the control variates developed in this literature are a promising direction for further reducing gradient variance.

[Q3] Calibration before recalibration

The original Figure 4 (Figure 3 in the revision) reports coverage after post-hoc recalibration. Although DAR optimizes a distribution-aware objective, autoregressive prediction methods generally do not guarantee calibrated prediction intervals from finite rollouts. Therefore, we apply the same post-hoc variance recalibration procedure to all methods for a fair comparison.

Following Laves et al. [3], we fit a single positive scale shared across inputs by minimizing Gaussian NLL. For each input, we rescale its numerical samples around their predictive mean:

p~ik=μi+s(pik−μi),\displaystyle \widetilde p_{ik}=\mu_i+s(p_{ik}-\mu_i),

where pikp_{ik} is the kk-th prediction for input ii, μi\mu_i is its predictive mean, and s>0s>0 is the fitted scale. We then construct 90% intervals from the adjusted distribution. This changes predictive dispersion without changing point predictions. The scale is fitted using a held-out validation set.

We now report coverage error both before and after recalibration. The same recalibration procedure is applied to all methods.

MethodKBSS (Code)APPS (Code)ESOL (Mol.)FreeSolv (Mol.)LIPO (Mol.)
SFT0.0570.1510.7230.7150.576
Pointwise MSE RL0.8140.7300.6430.8540.814
DAR (Ours)0.3660.3780.5370.6380.510

Table 4. Calibration before recalibration (Qwen3-4B-Instruct): raw 90% coverage error (↓). Coverage error is |empirical coverage of the central 90% prediction interval − 0.90|; lower is better.

MethodKBSS (Code)APPS (Code)ESOL (Mol.)FreeSolv (Mol.)LIPO (Mol.)
SFT0.0540.1010.3250.2540.040
Pointwise MSE RL0.5490.4330.1120.4690.405
DAR (Ours)0.0150.0100.0420.0380.045

Table 5. Calibration after post-hoc recalibration (Qwen3-4B-Instruct): 90% coverage error (↓). Same metric as Table 4, computed after applying the same post-hoc recalibration to every method.

DAR has lower raw coverage error on the three molecular tasks, while SFT performs better on the two coding tasks. After the same scale adjustment, DAR achieves the lowest coverage error on four of five datasets, with errors ranging from 0.010 to 0.045. These results distinguish the quality of raw uncertainty estimates from the coverage achievable after recalibration.

We have added the fitting procedure, raw coverage results, and discussion of calibration-data requirements in Appendix D.3.

[Q4, Q5] Figure 3 and standard deviations

The original Figure 3 (Figure 4 in the revision) reports results on KBSS, which we have now specified in the caption. We have also added standard deviations to Appendix D.5.

References

[1] Zhang et al., REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge, ICML 2026

[2] Chiang et al., TRACT: Regression-Aware Fine-tuning Meets Chain-of-Thought Reasoning for LLM-as-a-Judge, ACL 2025

[3] Laves et al., Well-calibrated regression uncertainty in medical imaging with deep learning, MIDL 2020

Dear Reviewer 5qxP,

We greatly appreciate your thorough and insightful review, which has been invaluable in improving our work. We have made every effort to carefully address your concerns. Specifically, we have clarified the comparisons with continuous-head baselines, added additional baselines in this category, conducted LLM-as-a-Judge experiments to further validate the motivation behind our method, provided a clearer explanation of the calibration procedure and corresponding results, and included additional statistical information to strengthen our analysis.

We hope that the additional results and clarifications have adequately addressed your concerns. We would be happy to provide further clarification or discuss any remaining questions. If you feel that our responses have sufficiently addressed your concerns, we would sincerely appreciate your reconsideration of the score in support of our submission. Thank you again for your thoughtful and constructive review!

Best Regards,

Paper18284 Authors

Reviewer_71AJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper is investigating the regression/ prediction task for llm. instead of use sft/rmse loss for the prediction number, the method samples out 12 rollouts and calculate a distribution aware distance DAR. and claim it is better than sft and rmse method.

Critical strengths

None

Critical weaknesses
  1. we need the theoritical results to show why use the dar rewards is better than other reward format, for example, compare with (Lukasik et al., 2025) and also [1].

  2. the benchmark should include more widely accpeted benchmark, like the llm as judge tasks. SYNTHETIC DISTRIBUTIONAL REGRESSION take too much space but i don't see how dar is better than other method in FIgure 2. the experiments are not solids, and also need to compare more baselines.

  3. the experiments are only qwen 3 4b. at lease we need to swap a bit across different model family and size

[1] REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge

Questions for authors
  1. I would be happy if the dar reward can be theoritcally proven to be better than other method (like lower variance, better calibration). this is the point i most concern about.

  2. can you add llm as judge experiment to show the performance. this is how the regression task for llm usually used for.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 71AJ [1/3]

We thank the reviewer for their thoughtful and constructive feedback. We have carefully addressed the concerns and revised the manuscript accordingly. Below, we provide detailed responses to each point.

[W1, Q1] Theoretical Justification

We thank the reviewer for highlighting the need for a clearer theoretical explanation. We have revised the paper to clarify why DAR can improve calibration by comparing what each reward constrains about the predictive distribution. Our explanation builds on established results for proper scoring rules [4], rather than introducing new theoretical claims.


1. RAFT: squared error on the mean constrains only the mean.

RAFT [1] introduced a principled regression-aware objective that applies squared error to the expected prediction. Fix an input, and let QQ and PP be the true and predicted distributions with means μQ\mu_Q and μP\mu_P. Writing Y−μP=(Y−μQ)+(μQ−μP)Y-\mu_P=(Y-\mu_Q)+(\mu_Q-\mu_P), the cross term vanishes since EQ[Y−μQ]=0\mathbb E_Q[Y-\mu_Q]=0, giving

EY∼Q[(Y−μP)2]=Var⁡Q(Y)+(μP−μQ)2\displaystyle \mathbb E_{Y\sim Q}[(Y-\mu_P)^2]=\operatorname{Var}_Q(Y)+(\mu_P-\mu_Q)^2

Any PP with the correct mean is optimal, regardless of its spread or shape. A model can therefore predict the right mean while producing intervals that are too narrow or too wide.


2. DAR: a CRPS-based reward constrains the whole distribution.

DAR scores predictions with CRPS, CRPS(P,y)=∫(FP(t)−1{y≤t})2dt\mathrm{CRPS}(P,y)=\int(F_P(t)-\mathbf{1}_{\lbrace y\le t\rbrace})^2dt. That is, it applies squared error to the event {Y≤t}\lbrace Y\le t\rbrace at every threshold tt. Applying the same identity at each tt gives

EY∼Q[CRPS(P,Y)]=∫FQ(t)(1−FQ(t))dt+∫(FP(t)−FQ(t))2dt\displaystyle \mathbb E_{Y\sim Q}[\mathrm{CRPS}(P,Y)]=\int F_{Q}(t)(1-F_{Q}(t))dt+\int(F_{P}(t)-F_{Q}(t))^2dt

Since the first term depends only on QQ, the expected CRPS is minimized only when FP(t)=FQ(t)F_P(t)=F_Q(t) almost everywhere, implying P=QP=Q. Thus, CRPS is strictly proper [4]. At this population optimum, the predictive distribution is calibrated [5]. For continuous, strictly increasing predictive CDFs, its quantiles satisfy Pr⁡(Y≤FP−1(α))=α\Pr(Y\le F_P^{-1}(\alpha))=\alpha. DAR builds on this property to optimize predictive distributions.


3. REAL: DAR scores the distribution formed across traces.

REAL [2] is a pioneering work that introduces regression-aware objectives into reinforcement learning. For each reasoning trace cc, REAL's reward is −(μc−y)2+λlog⁡πθ(y∣x,c)-(\mu_c-y)^2+\lambda\log\pi_\theta(y\mid x,c), where μc\mu_c is the expected score under that trace. Its log-likelihood term is strictly proper, so REAL can distinguish distributions with the same mean within a trace.

REAL's reward differs from DAR's in two ways.

(1) Scoring traces separately vs. jointly. REAL scores each trace separately, whereas DAR scores the traces jointly. Consider REAL's squared-error term across traces for the same input, and let m=EC[μC]m=\mathbb E_C[\mu_C]. Writing μC−y=(μC−m)+(m−y)\mu_C-y=(\mu_C-m)+(m-y), the cross term again vanishes, giving

EC[(μC−y)2]=(m−y)2+Var⁡C(μC)\displaystyle \mathbb E_C[(\mu_C-y)^2]=(m-y)^2+\operatorname{Var}_C(\mu_C)

REAL’s squared-error term therefore penalizes disagreement among traces, even when their average prediction is correct. In contrast, DAR scores the mixture of the score distributions of KK sampled traces, M=1K∑kπθ(⋅∣x,ck)M=\frac1K\sum_k\pi_\theta(\cdot\mid x,c_k), allowing uncertainty to be represented both within and across traces.

(2) Responding to spread at a fixed mean. REAL's reward depends on each trace's distribution only through its mean and the probability of the observed score. DAR's reward depends on the whole distribution, so it responds to spread even when the mean is fixed. It rewards spread that stays close to the target and penalizes spread that does not.

Example. Suppose the true score is 3, and consider traces that output the following score distributions:

  • P1P_1 assigns probabilities 0.25, 0.5, 0.25 to scores 2, 3, 4.
  • P2P_2 assigns the same probabilities (0.25, 0.5, 0.25) to scores 1, 3, 5.

Both have mean 3 and assign probability 0.5 to the true score, so REAL gives them identical rewards for any λ\lambda (squared error 0, log-likelihood log⁡0.5\log 0.5). DAR distinguishes them: P1P_1 has lower CRPS (0.125 vs. 0.25), and when both kinds of traces are sampled for the same input, DAR gives higher credit to the P1P_1 traces.

We have added this explanation to Section 2.2 (L184).

AuthorsOpenReview ↗

Response to Reviewer 71AJ [2/3]

[W2, Q2] Experiments on LLM-as-a-Judge benchmarks

We thank the reviewer for this valuable suggestion, which helped us strengthen our empirical evaluation. We have added LLM-as-a-Judge experiments following REAL’s setup [2]. In these tasks, the model assigns numerical quality scores to responses using scoring rubrics. Starting from Mistral-7B-TRACT [3], we train on Feedback Collection and evaluate on three out-of-distribution benchmarks: FLASK, MT-Bench, and Vicuna.

We replace REAL's L2-based regression reward with DAR, which applies CRPS to the mixture of the traces' score distributions and adds a squared-error term on the mixture mean. Unlike REAL's per-trace L2, this term does not penalize disagreement across traces. All other training components and hyperparameters remain unchanged.

We re-evaluate the official TRACT and REAL checkpoints alongside DAR using the same inference pipeline. Each example uses one generated reasoning trace, and the final prediction is the probability-weighted average of the possible score tokens following REAL’s protocol. RAFT [1]’s published correlation results are included for reference as its checkpoint is unavailable. We report two metrics, Pearson correlation and RMSE. Pearson correlation measures the strength of the linear relationship between predicted and reference scores, while RMSE measures the magnitude of numerical prediction errors.

MethodFLASKMT-BenchVicunaAvg.
RAFT41.839.952.844.8
TRACT50.452.356.353.0
REAL56.058.964.159.7
DAR (Ours)57.658.966.661.1

Table 1. LLM-as-a-Judge, out-of-domain benchmarks: Pearson correlation (↑). RAFT results are taken from [2]; all other models are evaluated by us using REAL's inference protocol. All RL methods are initialized from TRACT.

MethodFLASKMT-BenchVicunaAvg.
TRACT1.5401.2671.1001.303
REAL1.3531.1450.8291.109
DAR (Ours)1.1811.0430.7110.978

Table 2. LLM-as-a-Judge, out-of-domain benchmarks: RMSE (↓). RAFT is omitted because no checkpoint is available.

Compared with REAL, DAR improves average Pearson correlation by 1.4 points and reduces average RMSE by 0.13. Pearson correlation improves on FLASK and Vicuna and matches REAL on MT-Bench; RMSE improves on all three benchmarks.


We also evaluate the predicted score distributions and prediction intervals. RPS is the discrete counterpart of CRPS and evaluates the full distribution over ordered scores. IS90 evaluates 90% prediction intervals by penalizing both interval width and targets outside the interval. Lower values are better for both metrics.

MethodRPS (↓)IS90 (↓)
TRACT0.8226.33
REAL0.6444.17
DAR (Ours)0.6073.66

Table 3. LLM-as-a-Judge, out-of-domain benchmarks: predictive distribution quality, RPS (↓) and IS90 (↓). Results are averaged over FLASK, MT-Bench, and Vicuna. RPS evaluates the full predicted score distribution, and IS90 evaluates the central 90% prediction interval by its width plus a penalty for missing the target.

DAR reduces RPS by 5.7% and IS90 by 12.2% relative to REAL. Together, these results show improvements in point prediction accuracy, score distributions, and prediction interval quality after replacing REAL’s L2-based regression reward component with DAR. We have added the LLM-as-a-Judge experiments to the main text (Section 4.3, Section 5.1).

AuthorsOpenReview ↗

Response to Reviewer 71AJ [3/3]

[W2] Additional baseline comparisons

We have expanded our comparisons to include regression-aware methods on both LLM-as-a-Judge and continuous regression tasks. As discussed above, the judge experiments compare DAR with RAFT, TRACT, and REAL.

We additionally implemented RAFT [1] on all five continuous regression benchmarks using Qwen3-4B-Instruct. We do not include TRACT [3] and REAL [2], as both extend RAFT-inspired regression objectives with reasoning, while our continuous regression setting does not use reasoning traces.

RAFT computes a numerical prediction from model probabilities over candidate values. We evaluated several ways to represent continuous targets, including digit-based mappings and normalized numerical grids. We report the best-performing RAFT results among the evaluated adaptations below.

MethodESOLFreeSolvLIPO
SFT0.8812.7550.820
Pointwise MSE RL0.8782.5610.822
RAFT0.9892.5780.820
DAR (Ours)0.8712.3600.776

Table 4. Molecular property prediction (Qwen3-4B-Instruct): RMSE (↓).

DAR achieves the lowest RMSE among the compared methods on all three molecular property tasks.

MethodKBSS Spearman (↑)KBSS MAE (↓)APPS Spearman (↑)APPS MAE (↓)
SFT0.6100.0130.93029.50
Pointwise MSE RL0.6000.0120.92029.80
RAFT0.6110.0140.92029.60
DAR (Ours)0.6600.0110.96029.10

Table 5. Code performance prediction (Qwen3-4B-Instruct): Spearman correlation (↑) and MAE (↓). APPS MAE is reported in units of 10310^3.

DAR achieves the highest Spearman correlation and lowest MAE on both code benchmarks among other major autoregressive baselines. These results extend our comparisons beyond SFT and pointwise MSE rewards to regression-aware fine-tuning, supporting DAR’s effectiveness on continuous numerical prediction. We have added these baseline comparisons to Appendix F.1.

[W3] Generalization to Another Model Family

We have added Gemma3-4B-it experiments on KBSS and LIPO to evaluate generalization across model families. We follow the same training and calibration procedures as in the main paper.

MethodKBSS Spearman (↑)KBSS MAE (↓)LIPO RMSE (↓)
SFT0.610.0140.879
Pointwise MSE RL0.600.0150.866
DAR (Ours)0.640.0120.834

Table 6. Point prediction (Gemma3-4B-it): KBSS Spearman (↑) and MAE (↓), LIPO RMSE (↓).

MethodKBSSLIPO
SFT0.0350.047
Pointwise MSE RL0.1210.278
DAR (Ours)0.0150.040

Table 7. Calibration (Gemma3-4B-it): 90% coverage error (↓). Coverage error is |empirical coverage of the central 90% prediction interval − 0.90|; lower is better.

DAR achieves the best point prediction performance and lowest coverage error on both datasets, consistent with the improvements observed using Qwen3-4B.

Together with the LLM-as-a-Judge experiments, our evaluation now covers three model families—Qwen, Gemma, and Mistral—across 0.5B, 4B and 7B models. These results demonstrate DAR’s effectiveness across different model families and sizes, beyond Qwen3-4B. We have added these results to Appendix F.3.

[W2] Synthetic experiment

The synthetic experiment was designed to visualize extrapolation beyond the training range. In Figure 2 (Figure 7 in Appendix D.1 of the revision), DAR more closely follows the target trend in the unseen region x∈(6,10]x\in(6,10] than SFT and MSE. It also achieves the highest overall Spearman correlation. We have clarified this purpose and moved the synthetic experiments to the Appendix D.1 following your suggestion.

References

[1] Lukasik et al., Better Autoregressive Regression with LLMs via Regression-Aware Fine-Tuning, ICLR 2025

[2] Zhang et al., REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge, ICML 2026

[3] Chiang et al., TRACT: Regression-Aware Fine-tuning Meets Chain-of-Thought Reasoning for LLM-as-a-Judge, ACL 2025

[4] Gneiting & Raftery, Strictly Proper Scoring Rules, Prediction, and Estimation, JASA, 2007

[5] Gneiting, Balabdaoui & Raftery, Probabilistic Forecasts, Calibration and Sharpness, JRSS, 2007

Comments on the paper

2 comments

Dear Reviewers,

As the discussion phase draws to a close, please acknowledge the author's response and let them know whether it has addressed some or all of your concerns.

Thank you for the time and effort you and the other reviewers have put into this discussion.

Best,

Your AC

AuthorsOpenReview ↗

Summary of Reviewer Feedback and Authors' Responses

Dear Reviewers and Area Chairs,

We sincerely thank all reviewers for their constructive feedback and thoughtful evaluation. We appreciate their recognition of our work’s strengths. Their comments have helped us strengthen the theoretical motivation, experimental evaluation, and positioning of our method.


Strengths Highlighted by Reviewers

  • Clear motivation and promising approach: Reviewer 5qxP highlighted the clear motivation and principled design, while Reviewer FqsW described our method as an interesting approach.
  • Improved predictive performance: Reviewers 5qxP and FqsW recognized improvements in prediction accuracy, ranking, and uncertainty estimation.
  • Important research direction: Reviewer 5qxP emphasized the importance of calibrated regression with foundation models and its potential applications.
  • Clear presentation: All three reviewers (71AJ, 5qxP, FqsW) rated the paper’s presentation as clear and easy to follow.

Key Contributions and Changes in the Revision

Our work introduces Distribution-Aware Reward (DAR), an on-policy RL objective that optimizes predictive distributions formed by multiple LLM rollouts for regression. Unlike pointwise objectives, our method assigns rewards based on each prediction’s contribution to the overall distribution.

We have substantially strengthened the paper through the following additions:

  • Theoretical justification (Section 2.2, Appendices A–B): We added a theoretical comparison with RAFT [1] and REAL [2], distinguishing mean-level, trace-level, and distribution-level optimization. This clarifies the distinction between pointwise and distribution-level optimization (Reviewer 71AJ).
  • LLM-as-a-Judge evaluation (Section 4.3, Table 3): We added Mistral-7B experiments on FLASK, MT-Bench, and Vicuna. Our method improves average out-of-domain Pearson correlation from 59.7 to 61.1 over REAL (Reviewers 71AJ, 5qxP, FqsW).
  • Additional baselines (Appendices F.1–F.2): We added comparisons with RAFT and end-to-end probabilistic regression heads trained using Gaussian NLL, CRPS, and quantile loss. Our method achieves stronger ranking performance on both code benchmarks and competitive molecular prediction accuracy (Reviewers 71AJ, 5qxP, FqsW).
  • Generalization across models (Appendix F.3, Section 4.3): We extended the evaluation to Gemma3-4B and Mistral-7B, covering three model families alongside Qwen3-4B (Reviewer 71AJ).
  • Calibration and uncertainty (Section 5.1, Appendix D.3): We added raw coverage results and clarified post-hoc recalibration. Our method reduces average recalibrated 90% coverage error from 0.15 (SFT) and 0.39 (MSE) to 0.03 (Reviewer 5qxP).
  • Computational efficiency (Appendix D.7): We reported training and inference costs, clarifying the computational trade-offs between RL-based methods and SFT (Reviewers 5qxP, FqsW).
  • Novelty clarification (Section 6): We clarified that our contribution lies in the distribution-aware reward formulation for LLM regression, rather than a new policy-gradient estimator (Reviewer 5qxP).

All additional experiments, analyses, and clarifications have been incorporated into the revised manuscript.

We again thank the reviewers for their valuable feedback. Their suggestions have substantially strengthened our work, and we hope these revisions address the major concerns raised during the review process.

Best regards,

Paper18284 Authors


References

[1] Lukasik et al., Better Autoregressive Regression with LLMs via Regression-Aware Fine-Tuning, ICLR 2025.

[2] Zhang et al., REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge, ICML 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.