- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Distribution-Aware Reward (DAR), a method based on reinforcement learning for fine-tuning LLMs as numerical regressors. DAT optimises the predictive distributions of LLMs using Continuous Ranked Probability Score (CRPS) to evaluate distribution quality. Leave-one-out credit assignment is used to reward individual predictions according to their contribution to the overall distribution quality. DAR is evaluated on synthetic regression, code performance prediction, and molecular property prediction tasks. Results show improvements in prediction accuracy, ranking performance, and uncertainty calibration compared with supervised fine-tuning and conventional pointwise RL objectives. The paper also investigates predictive diversity, calibration, and alternative reward formulations through additional analyses and ablations.
Critical strengths
The paper proposes an interesting distribution-aware RL objective based on CRPS and leave-one-out credit assignment that seems to perform well. It improves prediction accuracy and ranking performance over standard SFT and pointwise RL baselines across several tasks.
Critical weaknesses
W1. Fairness and completeness of the regression baselines.
The paper compares DAR against a value-head regression baseline and several alternative training objectives (Gaussian NLL, Gaussian CRPS, and quantile loss) in Table 7. However, the implementation of the latter baselines is unclear. Are these objectives applied to token-level predictions from an LLM fine-tuned end-to-end (like in DAR), or to a regression head operating on LLM representations?
Appendix D.2 indicates that the value-head baseline uses a frozen LLM backbone with a two-layer MLP regression head, whereas DAR fine-tunes the LLM end-to-end. This creates a potential confound. It is unclear if the improvements arise from a better training objective or increased compute budget and model capacity.
I would like to see a more controlled comparison against probabilistic regression heads that explicitly predict both the conditional mean and variance, trained with different objectives, such as Gaussian NLL, CRPS and the quantile loss. These baselines should be evaluated under both (i) a frozen LLM backbone with a trainable probabilistic head and (ii) end-to-end fine-tuning of the LLM and the probabilistic head. All comparisons should also report uncertainty calibration metrics (e.g. prediction-interval coverage), rather than only point prediction and ranking metrics.
W2. Motivation for autoregressive numerical prediction.
The paper proposes an interesting distribution-aware RL objective for improving the calibration and accuracy of numerical predictions generated by LLMs. However, I find the motivation for retaining numerical predictions in the autoregressive token space not sufficiently justified.
For the scalar regression tasks considered in this paper, it is unclear why one should prefer autoregressive numerical generation followed by relatively complex RL fine-tuning over a probabilistic regression head attached to the LLM's representations. Such a head could directly predict the parameters of a conditional distribution and be optimized end-to-end (as proposed in W1).
I do see a stronger motivation for the proposed approach in settings where the LLM must jointly generate natural language and numerical predictions, for example when providing explanations, reasoning, or longer textual responses containing numerical estimates.
However, all the main experiments consider simple scalar numerical outputs. Appendix H investigates generating intermediate reasoning before numerical predictions but it is limited to a very narrow setting and the additional reasoning steps do not lead to improved predictive performance (which seems insufficiently explored).
I would encourage the authors to clarify the advantages of autoregressive numerical prediction over end-to-end probabilistic regression heads. Ideally, the paper should either provide more experiments on tasks involving joint text and numerical generation, where the autoregressive formulation can be more clearly justified.
W3. Computational efficiency compared against an expensive reference point
The authors state that DAR has negligible computational overhead (Appendix D.5, Table 5). This is true relative to pointwise GRPO, because both methods already require 12 rollouts per training input. But DAR is not computationally more efficient compare to supervised dine-tuning of a probabilistic head. Furthermore, at test time, DAR uses 32 sampled predictions, while a probabilistic regression head could produce the distributional parameters in a single forward pass.
Questions for authors
See weaknesses.
7 comments
Response to Reviewer FqsW [2/3]
[W2] Motivation for Autoregressive Numerical Prediction
The reviewer raises an interesting point about the value of probabilistic regression heads. Such heads may be simpler for scalar-only prediction tasks. Our work focuses on autoregressive numerical prediction, which provides a unified interface for generating reasoning, explanations, and numerical estimates. A broader comparison between these approaches is an interesting direction for future work.
Our goal is not to replace regression heads in all settings, but to improve the quality and calibration of numerical predictions within the autoregressive generation framework of existing LLMs [2,3,4]. This framework supports reasoning, explanations, and numerical estimates in a unified manner. DAR enables distribution-aware optimization without introducing a separate numerical prediction head.
To demonstrate the benefit of this formulation beyond scalar regression, we now evaluate DAR on ordinal LLM-as-a-Judge tasks. These tasks require models to generate both reasoning and numerical scores, providing a natural setting for autoregressive numerical prediction.
Following REAL [2], we initialize from Mistral-7B-TRACT, train on Feedback Collection, and evaluate on FLASK, MT-Bench, and Vicuna. We replace REAL's L2-based regression reward with DAR (CRPS on the mixture of score distributions plus a squared-error term on its mean); all other hyperparameters remain unchanged.
We re-evaluate the official TRACT [3] and REAL checkpoints using the same inference pipeline: one reasoning trace followed by the expected score from score-token probabilities. RAFT’s published correlations are included for reference because its checkpoint is unavailable [4].
| Method | FLASK | MT-Bench | Vicuna | Avg. |
|---|---|---|---|---|
| RAFT | 41.8 | 39.9 | 52.8 | 44.8 |
| TRACT | 50.4 | 52.3 | 56.3 | 53.0 |
| REAL | 56.0 | 58.9 | 64.1 | 59.7 |
| DAR (Ours) | 57.6 | 58.9 | 66.6 | 61.1 |
Table 4. LLM-as-a-Judge (Mistral-7B), out-of-domain benchmarks: Pearson correlation (↑). RAFT results are taken from [2]; all other models are evaluated by us using REAL's inference protocol. REAL and DAR are initialized from TRACT.
| Method | FLASK | MT-Bench | Vicuna | Avg. |
|---|---|---|---|---|
| TRACT | 1.540 | 1.267 | 1.100 | 1.303 |
| REAL | 1.353 | 1.145 | 0.829 | 1.109 |
| DAR (Ours) | 1.181 | 1.043 | 0.711 | 0.978 |
Table 5. LLM-as-a-Judge (Mistral-7B), out-of-domain benchmarks: RMSE (↓). Same setting as Table 4. RAFT is omitted because its RMSE is not reported in [2] and no checkpoint is available.
DAR improves average Pearson correlation by 1.4 points and reduces average RMSE by 0.13 relative to REAL. It improves Pearson correlation on FLASK and Vicuna, matches REAL on MT-Bench, and achieves lower RMSE on all three benchmarks.
We also evaluate the uncertainty of the approaches. We report RPS, the discrete counterpart of CRPS for ordered score distributions, and IS90, which penalizes interval width and targets outside the 90% prediction interval. Lower is better for both, and all methods use the same calibration procedure.
| Method | RPS (↓) | IS90 (↓) |
|---|---|---|
| TRACT | 0.822 | 6.33 |
| REAL | 0.644 | 4.17 |
| DAR (Ours) | 0.607 | 3.66 |
Table 6. LLM-as-a-Judge (Mistral-7B), out-of-domain benchmarks: predictive distribution quality, RPS (↓) and IS90 (↓). Results are averaged over FLASK, MT-Bench, and Vicuna. RPS (ranked probability score) evaluates the full predicted score distribution. IS90 (90% interval score) evaluates the central 90% prediction interval by its width plus a penalty for missing the target.
DAR reduces RPS by 5.7% and IS90 by 12.2% relative to REAL. These results demonstrate improvements in both numerical prediction and distributional quality in a task that combines explanations and scores. They provide a concrete use case for distribution-aware training within the existing autoregressive generation framework.
We have clarified this motivation and added the judge experiments to the main text in Section 4.3 and Section 5.1.
Response to Reviewer FqsW [3/3]
[W3] Computational Cost
We agree that the reference point for our efficiency claim should be explicit. DAR’s negligible overhead is relative to pointwise GRPO under the same budget of 12 rollouts per input.
Training cost. In our Qwen3-4B experiments on four H100 GPUs, DAR requires approximately 3x the wall-clock training time of SFT and end-to-end regression-head training. The additional cost mainly comes from rollout generation and RL training.
Inference cost. For continuous regression, DAR averages 32 sampled numerical predictions. We measured the sampling cost using Qwen3-4B on a single L40S GPU, with 16 LIPO inputs and three repetitions.
For a single input, generating 32 predictions in parallel took 0.593 seconds, compared to 0.332 seconds for one prediction. This represents a 1.79x increase in median latency, demonstrating that parallel generation substantially reduces the latency overhead of multiple samples.
We have clarified the scope of our efficiency claim and added the training-cost comparison and task-specific inference measurements in Appendix D.7.
References
[1] Tang et al., Understanding llm embeddings for regression, TMLR 2025
[2] Zhang et al., REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge, ICML 2026
[3] Chiang et al., TRACT: Regression-Aware Fine-tuning Meets Chain-of-Thought Reasoning for LLM-as-a-Judge, ACL 2025
[4] Lukasik et al., Better Autoregressive Regression with LLMs via Regression-Aware Fine-Tuning, ICLR 2025
Response to Reviewer FqsW [1/3]
We thank the reviewer for the encouraging assessment and thoughtful suggestions. We have carefully addressed the concerns and revised the manuscript accordingly. Below, we provide point-by-point responses to each comment.
[W1] Fairness and Completeness of Regression Baselines
We thank the reviewer for helping us clarify the fairness of our baseline comparisons. The original head-based baselines followed Tang et al.’s frozen-backbone setup [1]. The Value Head baseline uses L2 loss. For Gaussian NLL, Gaussian CRPS, and quantile loss, we replaced it with the corresponding prediction heads operating on the frozen LLM representations.
We have added end-to-end versions of all three probabilistic baselines. Both the Qwen3-4B-Instruct backbone and prediction head were jointly fine-tuned using the same setup as SFT. These comparisons address the difference in backbone trainability; training costs are discussed separately below.
DAR achieves the strongest ranking performance on both coding tasks and the lowest reported RMSE on LIPO and ESOL. Thus, DAR remains competitive when the probabilistic heads are also trained end-to-end.
We additionally report 90% prediction-interval coverage error using the same calibration procedure for all methods.
DAR achieves the lowest coverage error on both coding tasks. It also outperforms pointwise MSE RL on all five datasets and SFT on four. However, calibration results are mixed across tasks. Continuous regression heads achieve lower coverage error on molecular tasks, highlighting their potential advantages in modeling continuous-valued outputs.
Nevertheless, DAR achieves the best calibration among autoregressive methods on four of five datasets, making it particularly useful for tasks that require generating both text and numerical predictions. We demonstrate this advantage in LLM-as-a-Judge tasks next. We have added the results and discussion to Appendix F.2.