- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This submission introduces two finetuning techniques to improve how calibrated language model outputs are to specific distributions. The techniques involve training on either trie-derived prefixes of (soft) or samples from (hard) random distributions (e.g. normal, poisson, etc.). Using a range of evaluations on similar mathematical distribution settings, novelty bench, multiple-choice question-answer generation, and general model capability, the work shows that their finetuning method leads to more calibrated and diverse generations than two inference-time baselines, including transfer to the non-mathematical settings, with only some degradation in general task performance.
Critical strengths
Increasing the calibration or diversity of language model generations is an important research area and this work makes a contribution to this area. To me, the strongest finding is that training on samples from mathematical distributions transfers to more difficult-to-defined distributions over text like the ones given in novelty-bench. Additionally, the more efficient trie-based training method might be of interest to some in the community.
Critical weaknesses
As far as I know, the main problem with calibration in LLMs is that base models are somewhat calibrated but are difficult to control, while post-trained models can follow instructions but have collapsed distributions. A critical weakness of this work is that it doesn’t really interact with better, although not perfect, base models. Implicitly, the work tries to solve this problem by starting from a model that can follow instructions and then try to uncollapse its distributions. I think this is reasonable, but should be explicitly mentioned and should be compared to methods like SpectrumTuning, which starts from actual base models and tries to do instruction-tuning in a way that maintains/improves calibration (https://arxiv.org/abs/2510.06084). Comparisons to other works that try to improve calibration through training would also be helpful (e.g. https://arxiv.org/pdf/2505.01997v3).
Relatedly, this makes the use of Paloma perplexity for evaluation a bit confusing. As far as I can tell, Paloma is based on pretraining-style data, so the finetuned models achieving lower perplexity might not have any bearing on the ability to sample from difficult-to-describe distributions.
Questions for authors
Suggestion for future work: it would be interesting to try to mix in some instruction-tuning data (e.g. from tuluv3) to see if that allows maintaining some of the instruction-following ability.
4 comments
[2/2] Zhang et al., PALOMA and instruction data
"Comparisons to other works that try to improve calibration through training would also be helpful (e.g. https://arxiv.org/pdf/2505.01997v3)."
That work restores confidence calibration (ECE of multiple-choice answer probabilities) in aligned models, while we target distributional calibration of sampled outputs. It shares our premise that alignment hurts calibration, and our design of fine-tuning the aligned model to restore it. The revision cites and discusses it. The paper already compares, in both directions, with the closest training-time method for output distributions, Zhang et al. [1] (Appendix K of the revision).
"this makes the use of Paloma perplexity for evaluation a bit confusing."
We evaluated perplexity to rule out that this fine-tuning damages the model's general language modeling, since perplexity on a broad text corpus such as PALOMA is the standard way to measure it. We found that it does not. Perplexity is unchanged or lower for 11 of the 12 models with soft targets and 10 with hard targets.
"it would be interesting to try to mix in some instruction-tuning data (e.g. from tuluv3) [...]"
Thank you for the suggestion. Our GSM8K analysis points the same way. Where accuracy drops, the fine-tuned model has stopped reasoning and answers with the number directly (see our reply to Reviewer 1PbV), and mixing in instruction data such as Tülu 3 is a natural way to counter this. The revision adds it as a direction for future work.
References
[1] Y. Zhang, A. Schwarzschild, N. Carlini, J. Z. Kolter, D. Ippolito, Forcing Diffuse Distributions out of Language Models, 2024, COLM.
Dear reviewer, we are following up before the discussion period closes on Nov 18. We would be grateful to know whether our reply addresses your concerns. If it does, we would appreciate it if you would consider updating your score.
Sorry for the delay in my reply. Thank you for the response - I've updated my score accordingly.
[1/2] Pretrained models and Spectrum Tuning
We thank the reviewers for their careful and constructive reviews. We are glad that the reviewers found this an important research area [qVEd], with an interesting setting and an intuitive solution [1PbV], solid evidence of improved sampling [1PbV, v1hC], a wide range of experiments [v1hC], an efficient trie-based training method [qVEd], and a clear presentation [1PbV, qVEd, v1hC]. We especially thank them for highlighting the transfer from mathematical distributions to natural language as the most interesting finding [qVEd, v1hC]. All changes in the revision are marked in blue.
We agree, and we are grateful for the pointer to Spectrum Tuning. In the revision we clarify that we start from instruction-tuned models, and we ran the comparisons you suggest. Pretrained models mostly fail to produce a valid number, and our models match or outperform Spectrum Tuning in every setting we test, without its extreme failures on some families, at a fraction of its training cost, and starting from the instruction-tuned model instead of the base model.
Base models. We evaluated the pretrained checkpoints of 11 of our 12 models (GPT-OSS-20B has none) on our benchmark, using Spectrum Tuning's own prompt format for pretrained models. As you describe, they are hard to control. On the held-out families, a median of 20% of their samples is a valid number, against 87% for the instruction-tuned models and 93–96% after our fine-tuning. Most outputs are text about the task, and the numbers they do produce include extreme values, so their W1 is orders of magnitude worse (see table below).
Spectrum Tuning. Spectrum Tuning differs from our method because it is a full post-training run from the base model, taking 26–30 hours on 4 A100 GPUs per model (their App. G), while ours is a LoRA on the instruction-tuned model. On the same 4 A100s and for the same two models, it trains in about 3 minutes with the soft target and about 30 minutes with the hard target.
We evaluated the released Spectrum Tuning checkpoints for the two models we share. We used Spectrum Tuning's own prompt format, writing our task in the style of its training data ("Draws from a … distribution with parameters …"). We report our benchmark metric, the median normalized W1 (lower is better), with 95% bootstrap intervals in brackets.
*Only 35–38% (Qwen) and 14–18% (Gemma) of base-model samples are valid numbers, and W1 is computed on those. Spectrum Tuning starts from the base model, ours from the instruction-tuned one.
Spectrum Tuning almost always outputs a valid number (96–99%). On the held-out families its median is close to ours, and 4 of these 12 families (Bernoulli, Poisson, Categorical, Zipf) are in its training data. The medians also hide where Spectrum Tuning fails. On some families a few absurd samples (e.g. 4269532.76 for a Weibull(0.5, 0.5)) push its W1 to 25–3·10⁷ on held-out families and above 10⁴⁴ on unseen parameters. Our worst family across all four settings is 5.1. Per-family values are in Appendix L of the revision.
We also evaluated on NoveltyBench to test transfer to natural-language diversity. We generate 10 samples per prompt and report its main metric, utility, which credits each sample only if it differs from the earlier ones, weighted by its quality under a reward model. Both our checkpoints score better than Spectrum Tuning, which remains at the level of the instruction-tuned models. For Qwen3-14B and Gemma-3-12B respectively, utility is 5.40 and 4.32 with soft targets, 4.99 and 4.08 with hard targets, 3.74 and 3.70 for Spectrum Tuning, and 3.86 and 3.22 for the instruction-tuned models.
Its authors leave open whether distributional coverage can be combined with a chat model, or whether the two are "fundamentally in tension, requiring distinct models for diversity/coverage and for chat" (their App. A). Our results suggest that one model can have both, since our fine-tuning improves the calibration and NoveltyBench utility of the instruction-tuned model while largely keeping its benchmark accuracy.