acceptodds
Under review as a conference paper at ICLR 2027

Probabilistic Calibration is a Trainable Capability in Language Models

Abstract

Language models are increasingly used in settings where outputs should follow an intended distribution, yet their generation probabilities are often poorly calibrated to it. In many such requests the distribution is implicit and the valid outputs cannot be listed in advance, so the task cannot be passed to an external random number generator. We study whether this capability can be improved directly through fine-tuning. Concretely, we fine-tune language models on synthetic prompts that require sampling from mathematical distributions, and compare two Calibration Fine-Tuning variants: a soft-target method that converts the desired output distribution into trie-derived next-token targets, and a hard-target method that trains on sampled completions from the same target distribution. Across 12 models spanning four families, both methods substantially improve structured-sampling fidelity on held-out distribution families and unseen parameter settings, showing that training on cheap synthetic mathematical distributions improves sampling on unseen distributions, even in natural language. The gains also transfer to broader stochastic generation benchmarks, including open-ended random generation, where fine-tuned models broaden their first-token support by one to two orders of magnitude, and NoveltyBench. The gains sometimes reduce downstream capability, especially arithmetic reasoning, with costs varying by model. Overall, our results show that probabilistic calibration can be improved through fine-tuning with either objective, and that calibration learned on mathematical distributions transfers to natural-language stochastic generation. Code is available at https://anonymous.4open.science/r/calibration-finetuning-3C11.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, randomness, probabilistic calibration, distributional fine-tuning, output diversity, mode collapse, controllable generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qVEdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This submission introduces two finetuning techniques to improve how calibrated language model outputs are to specific distributions. The techniques involve training on either trie-derived prefixes of (soft) or samples from (hard) random distributions (e.g. normal, poisson, etc.). Using a range of evaluations on similar mathematical distribution settings, novelty bench, multiple-choice question-answer generation, and general model capability, the work shows that their finetuning method leads to more calibrated and diverse generations than two inference-time baselines, including transfer to the non-mathematical settings, with only some degradation in general task performance.

Critical strengths

Increasing the calibration or diversity of language model generations is an important research area and this work makes a contribution to this area. To me, the strongest finding is that training on samples from mathematical distributions transfers to more difficult-to-defined distributions over text like the ones given in novelty-bench. Additionally, the more efficient trie-based training method might be of interest to some in the community.

Critical weaknesses

As far as I know, the main problem with calibration in LLMs is that base models are somewhat calibrated but are difficult to control, while post-trained models can follow instructions but have collapsed distributions. A critical weakness of this work is that it doesn’t really interact with better, although not perfect, base models. Implicitly, the work tries to solve this problem by starting from a model that can follow instructions and then try to uncollapse its distributions. I think this is reasonable, but should be explicitly mentioned and should be compared to methods like SpectrumTuning, which starts from actual base models and tries to do instruction-tuning in a way that maintains/improves calibration (https://arxiv.org/abs/2510.06084). Comparisons to other works that try to improve calibration through training would also be helpful (e.g. https://arxiv.org/pdf/2505.01997v3).

Relatedly, this makes the use of Paloma perplexity for evaluation a bit confusing. As far as I can tell, Paloma is based on pretraining-style data, so the finetuned models achieving lower perplexity might not have any bearing on the ability to sample from difficult-to-describe distributions.

Questions for authors

Suggestion for future work: it would be interesting to try to mix in some instruction-tuning data (e.g. from tuluv3) to see if that allows maintaining some of the instruction-following ability.

4 comments
AuthorsOpenReview ↗

[1/2] Pretrained models and Spectrum Tuning

We thank the reviewers for their careful and constructive reviews. We are glad that the reviewers found this an important research area [qVEd], with an interesting setting and an intuitive solution [1PbV], solid evidence of improved sampling [1PbV, v1hC], a wide range of experiments [v1hC], an efficient trie-based training method [qVEd], and a clear presentation [1PbV, qVEd, v1hC]. We especially thank them for highlighting the transfer from mathematical distributions to natural language as the most interesting finding [qVEd, v1hC]. All changes in the revision are marked in blue.

"it doesn't really interact with better, although not perfect, base models. [...] should be compared to methods like SpectrumTuning [...]"

We agree, and we are grateful for the pointer to Spectrum Tuning. In the revision we clarify that we start from instruction-tuned models, and we ran the comparisons you suggest. Pretrained models mostly fail to produce a valid number, and our models match or outperform Spectrum Tuning in every setting we test, without its extreme failures on some families, at a fraction of its training cost, and starting from the instruction-tuned model instead of the base model.

Base models. We evaluated the pretrained checkpoints of 11 of our 12 models (GPT-OSS-20B has none) on our benchmark, using Spectrum Tuning's own prompt format for pretrained models. As you describe, they are hard to control. On the held-out families, a median of 20% of their samples is a valid number, against 87% for the instruction-tuned models and 93–96% after our fine-tuning. Most outputs are text about the task, and the numbers they do produce include extreme values, so their W1 is orders of magnitude worse (see table below).

Spectrum Tuning. Spectrum Tuning differs from our method because it is a full post-training run from the base model, taking 26–30 hours on 4 A100 GPUs per model (their App. G), while ours is a LoRA on the instruction-tuned model. On the same 4 A100s and for the same two models, it trains in about 3 minutes with the soft target and about 30 minutes with the hard target.

We evaluated the released Spectrum Tuning checkpoints for the two models we share. We used Spectrum Tuning's own prompt format, writing our task in the style of its training data ("Draws from a … distribution with parameters …"). We report our benchmark metric, the median normalized W1 (lower is better), with 95% bootstrap intervals in brackets.

Qwen3-14B held-out familiesQwen3-14B unseen parametersGemma-3-12B held-out familiesGemma-3-12B unseen parameters
Base model*1.3·10⁴ [0.20, 8.9·10¹¹]2.5·10³ [0.22, 5.9·10¹⁵]9.0·10¹³ [0.77, 1.0·10¹⁵]1.5·10¹⁵ [3.3·10⁸, 2.1·10¹⁵]
Instruction-tuned0.25 [0.23, 0.32]0.23 [0.19, 0.29]0.25 [0.21, 0.29]0.27 [0.24, 0.38]
Spectrum Tuning0.14 [0.07, 0.25]0.38 [0.08, 1.01]0.15 [0.10, 0.61]5.27 [0.22, 1.5·10⁵]
Ours (soft)0.10 [0.07, 0.14]0.07 [0.06, 0.10]0.14 [0.08, 0.22]0.15 [0.09, 0.21]
Ours (hard)0.12 [0.07, 0.14]0.05 [0.04, 0.09]0.16 [0.08, 0.24]0.09 [0.06, 0.13]

*Only 35–38% (Qwen) and 14–18% (Gemma) of base-model samples are valid numbers, and W1 is computed on those. Spectrum Tuning starts from the base model, ours from the instruction-tuned one.

Spectrum Tuning almost always outputs a valid number (96–99%). On the held-out families its median is close to ours, and 4 of these 12 families (Bernoulli, Poisson, Categorical, Zipf) are in its training data. The medians also hide where Spectrum Tuning fails. On some families a few absurd samples (e.g. 4269532.76 for a Weibull(0.5, 0.5)) push its W1 to 25–3·10⁷ on held-out families and above 10⁴⁴ on unseen parameters. Our worst family across all four settings is 5.1. Per-family values are in Appendix L of the revision.

We also evaluated on NoveltyBench to test transfer to natural-language diversity. We generate 10 samples per prompt and report its main metric, utility, which credits each sample only if it differs from the earlier ones, weighted by its quality under a reward model. Both our checkpoints score better than Spectrum Tuning, which remains at the level of the instruction-tuned models. For Qwen3-14B and Gemma-3-12B respectively, utility is 5.40 and 4.32 with soft targets, 4.99 and 4.08 with hard targets, 3.74 and 3.70 for Spectrum Tuning, and 3.86 and 3.22 for the instruction-tuned models.

Its authors leave open whether distributional coverage can be combined with a chat model, or whether the two are "fundamentally in tension, requiring distinct models for diversity/coverage and for chat" (their App. A). Our results suggest that one model can have both, since our fine-tuning improves the calibration and NoveltyBench utility of the instruction-tuned model while largely keeping its benchmark accuracy.

AuthorsOpenReview ↗

[2/2] Zhang et al., PALOMA and instruction data

"Comparisons to other works that try to improve calibration through training would also be helpful (e.g. https://arxiv.org/pdf/2505.01997v3)."

That work restores confidence calibration (ECE of multiple-choice answer probabilities) in aligned models, while we target distributional calibration of sampled outputs. It shares our premise that alignment hurts calibration, and our design of fine-tuning the aligned model to restore it. The revision cites and discusses it. The paper already compares, in both directions, with the closest training-time method for output distributions, Zhang et al. [1] (Appendix K of the revision).

"this makes the use of Paloma perplexity for evaluation a bit confusing."

We evaluated perplexity to rule out that this fine-tuning damages the model's general language modeling, since perplexity on a broad text corpus such as PALOMA is the standard way to measure it. We found that it does not. Perplexity is unchanged or lower for 11 of the 12 models with soft targets and 10 with hard targets.

"it would be interesting to try to mix in some instruction-tuning data (e.g. from tuluv3) [...]"

Thank you for the suggestion. Our GSM8K analysis points the same way. Where accuracy drops, the fine-tuned model has stopped reasoning and answers with the number directly (see our reply to Reviewer 1PbV), and mixing in instruction data such as Tülu 3 is a natural way to counter this. The revision adds it as a direction for future work.

References

[1] Y. Zhang, A. Schwarzschild, N. Carlini, J. Z. Kolter, D. Ippolito, Forcing Diffuse Distributions out of Language Models, 2024, COLM.

Dear reviewer, we are following up before the discussion period closes on Nov 18. We would be grateful to know whether our reply addresses your concerns. If it does, we would appreciate it if you would consider updating your score.

Sorry for the delay in my reply. Thank you for the response - I've updated my score accordingly.

Reviewer_v1hCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether LLMs can be finetuned to improve their ability to sample according to specified probability distributions. The authors propose and compare two such callibration finetuning, with a soft-target method that first discretize target distribution into numerical strings and then build next-token target distribution indirectly by constructing prefix trie; and a hard-target method that directly sample complete results from target distribution. Both method uses LoRA finetuning. Experiments cover 12 models from 4 families and 36 different mathematical distributions, with 24 used for training and 12 for held-out eval. Results show that both proposed approaches can improve the sampling ability and the performance on natural language generation tasks, while there's notable differences between transferability and capability retention.

Critical strengths

The story is interesting and well-motivated. The experiments cover a wide range of explorations, and the authors have considered multiple alternative perspectives when interpreting the results. Solid evidence is provided to justify the actual improvement in sampling abilities. I find it especially notable that performance gained from training on purely mathematical probabilistic distribution can transfer to natural language generation, which is really non-trivial and interesting finding that helps deepen our understanding of LLMs.

Critical weaknesses

About independence of held-out evaluation and hyperparameter selection

Appendix E suggest that the results run on 6 of the 12 held-out distribution families are used for hyperparameter configuration evaluation to help decide the final configurations. I find it slightly inappropriate to select the final configuration according to their performance on eval set even if none of these distributions contributes training examples. I am not 100% sure if I have understood this correctly, and please correct me if I were wrong.

About calibration on natural language

The authors use first-token support, unique-output rate, and NoveltyBench utility as metrics for open-ended transfer, which do not directly establish that probability distribution has been allocated more faithfully to the ground truth. I think the paper can benefit from distinguishing more clearly between improved natural-language diversity or utility, and improved calibration to natural-language target distributions.

Also the temperature control experiments has only been made on math sampling in Appendix I, but not on natural language. Since natural language often involves significantly larger candidate vocabulary, it is not clear how the claimed diversity benefit could be obtained through simpler decoding adjustments.

Questions for authors

Please address the major concerns listed above in Critical Weaknesses. I also have some minor points:

  1. In Eq. 4 the loss is normalized by token length. Under equal weighting of sampled examples with different lengths, wouldn't this design generally favor towards PY(y)/∣t∣P_Y(y)/|\mathbf{t}| instead of PY(y)P_Y(y)?

  2. Table 20 report TinyBenchmarks results "averaged across the six tasks", whereas in Section 4.2 line 303 only 5 is listed. From Table 21 I see that maybe it is the GSM8K that got split into Strict and Flex, but then the reported size of evaluated items (100) would be ambiguous and misleading. I think additional details should be provided here.

3 comments
AuthorsOpenReview ↗

[1/2] Hyperparameter selection, diversity vs. calibration

We thank the reviewers for their careful and constructive reviews. We are glad that the reviewers found this an important research area [qVEd], with an interesting setting and an intuitive solution [1PbV], solid evidence of improved sampling [1PbV, v1hC], a wide range of experiments [v1hC], an efficient trie-based training method [qVEd], and a clear presentation [1PbV, qVEd, v1hC]. We especially thank them for highlighting the transfer from mathematical distributions to natural language as the most interesting finding [qVEd, v1hC]. All changes in the revision are marked in blue.

"Appendix E suggest that the results run on 6 of the 12 held-out distribution families are used for hyperparameter configuration evaluation [...]"

We appreciate this observation, and your reading of Appendix E is correct. To clarify, the selection concerns only part of one of our seven evaluation axes, structured distribution sampling, and we ran a new experiment showing that the configuration has little effect even on this axis, so our conclusions do not depend on it.

The ablations used 2 of the 12 models and 6 of the 12 held-out families (Bernoulli, Poisson, Maxwell, TruncNorm, Chi, Weibull). Their purpose was to find an affordable setting and to check how sensitive the results are to it, since the ablated settings have no established default and finer output spaces quickly become expensive (building the output space takes up to 2.4 GB per prompt at five decimals and 24 GB at six). We therefore kept the sweep small, with at most 10 configurations per method. For hard targets, we picked the configuration that was most consistent across the two models rather than the best one for each. In this experiment, we evaluated all configurations on the other six held-out families, which no ablation used. There, the selected configuration is within 6% of the median configuration. The revision states which families the ablations used and reports the two groups separately in Appendix F of the revision.

"I think the paper can benefit from distinguishing more clearly between improved natural-language diversity or utility, and improved calibration [...]"

We found this suggestion very helpful. We agree that the two should be separated, and we distinguish them as follows. Diversity is how many different, yet valid, outputs a model produces, regardless of any target, and first-token support, unique rate and NoveltyBench measure it. Calibration is how close the output distribution is to a specified target, which requires a known target. Our original natural-language results measure diversity only.

To measure calibration, we developed a benchmark of 275 natural-language prompts with exact target distributions, which we called NL-Calib and have added to the anonymized repository for inspection. Targets are either set programmatically, as in "Sample one vegetable: cabbage with probability 0.25 and pea with probability 0.75", or taken from well-defined sets, as in "Sample one of the fifty states of the United States at random". The prompts range from direct requests like these to short scenarios where the model has to work out the distribution itself, such as "Picture a box containing 2 orange and 4 yellow balls, all identical apart from their color. You draw one blindly. Which color comes out?". A last group asks everyday questions, such as how a random American likes their steak, and uses the model's own stated percentages as the target. For each prompt we draw 500 samples and measure the total variation (TV) distance between the answers and the target, counting off-list answers as errors.

Averaged over the 12 models, fine-tuning lowers TV from 0.53 [0.52, 0.55] to 0.37 [0.35, 0.38] with soft targets and 0.36 [0.35, 0.37] with hard targets, with more valid answers (89% → 94%) and more options covered (73% → 92%). See the table below. In the revision, we reframed the natural-language results to better distinguish calibration from diversity (Section 4 and the new Table 2).

AuthorsOpenReview ↗

[2/2] Decoding baselines, Eq. 4 and TinyBenchmarks

"the temperature control experiments has only been made on math sampling in Appendix I, but not on natural language."

We thank the reviewer for pointing to this gap, and we ran temperature and min-p decoding baselines on two natural-language benchmarks, NL-Calib and NoveltyBench, for both the instruction-tuned and the fine-tuned models. We found that decoding alone improves diversity but has a small effect on calibration, and that it can be tuned orthogonally to fine-tuning, with the two applied together giving the best results on both benchmarks.

We use the curated split of NoveltyBench (100 prompts, 10 samples each), whose main metric, utility, credits each sample only if it differs from the earlier ones, weighted by its quality under a reward model. We decoded at temperature 1 and 2, with and without min-p sampling [1] (p = 0.1), a truncation method designed to keep high-temperature sampling coherent. With min-p at T = 2, the instruction-tuned models reach a NoveltyBench utility of 4.15, close to 4.23 for our models at T = 1, but their NL-Calib TV stays at 0.49 against 0.37. Applied together with fine-tuning, min-p at T = 2 gives TV 0.30 and utility 4.64. The table gives means over the 12 models, with 95% bootstrap intervals over prompts.

NL-Calib TV ↓NL-Calib valid ↑NL-Calib covered ↑NoveltyBench utility ↑
Instruction-tuned, T = 10.53 [0.52, 0.55]89%73%3.61 [3.39, 3.81]
Instruction-tuned, T = 1, min-p0.55 [0.53, 0.57]91%53%3.43 [3.22, 3.62]
Instruction-tuned, T = 20.61 [0.60, 0.62]65%86%1.89 [1.69, 2.10]
Instruction-tuned, T = 2, min-p0.49 [0.47, 0.50]87%67%4.15 [3.92, 4.38]
Ours (soft), T = 10.37 [0.35, 0.38]94%92%4.23 [3.97, 4.49]
Ours (soft), T = 1, min-p0.38 [0.37, 0.39]97%72%4.06 [3.80, 4.34]
Ours (soft), T = 20.74 [0.73, 0.75]28%95%1.16 [1.11, 1.23]
Ours (soft), T = 2, min-p0.30 [0.29, 0.31]92%87%4.64 [4.36, 4.91]
Ours (hard), T = 10.36 [0.35, 0.37]94%92%4.07 [3.83, 4.33]
Ours (hard), T = 1, min-p0.38 [0.36, 0.39]96%73%3.90 [3.65, 4.16]
Ours (hard), T = 20.65 [0.64, 0.66]39%96%1.39 [1.29, 1.49]
Ours (hard), T = 2, min-p0.30 [0.28, 0.31]92%87%4.56 [4.29, 4.85]

A higher temperature without min-p covers more options but gives more invalid answers, for the instruction-tuned models and for ours, and at T = 2 NoveltyBench utility collapses as outputs turn into random tokens. Per-model results are in Appendix J.1.

"In Eq. 4 the loss is normalized by token length. [...] wouldn't this design generally favor towards PY(y)/tP_Y(y)/t instead of PY(y)P_Y(y)?"

You are right that Eq. 4 as written, which divides each completion's loss by its own length, would shift the optimum towards PY(y)/∣t∣P_Y(y)/|t|. The equation misstates our implementation, which averages the cross-entropy over all completion tokens in a batch rather than per completion, so all completions in a batch get the same weight. The revision corrects Eq. 4 to the sequence-level loss and states that it is averaged over the completion tokens of a batch.

"Table 20 report TinyBenchmarks results 'averaged across the six tasks', whereas in Section 4.2 line 303 only 5 is listed."

You are right that GSM8K was counted twice, under strict and flexible matching on the same 100 items, which made the average confusing. The revision clarifies this, as the TinyBenchmarks average now covers the four multiple-choice tasks and GSM8K is reported separately on the full test set (see our reply to Reviewer 1PbV).

References

[1] M. N. Nguyen et al., Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs, 2025, ICLR.

Sorry for the late reply. I appreciate the additional experiments and clarifications you provided. My main concerns have been well addressed. I will keep my rating as 3 (weak accept), but I am slightly more positive about this submission. Good luck!

Reviewer_1PbVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors observed that instruction-tuned LLMs collapse onto a few outputs when asked to sample randomly. The authors proposed LoRA fine-tuning on 24 synthetic parametric distributions using either trie-based next-token KL (soft-target) or standard SFT on drawn samples (hard-target).

The results shows that their method cuts W₁ distance by 2–4× and logit KL by ~10× on 12 held-out distribution families, beating oracle temperature scaling and prior baselines. Soft and hard targets match at equal step budgets. But it only transfers partially to natural-language diversity, also reduces GSM8K accuracy (0.494 → 0.321) and breaks MCQ formatting on several models.

Critical strengths

interesting setting and intuitive solution LLMs do follow a non-uniform distribution, when asked to generate uniform distribution sampling numbers. The authors proposed an intuitive solution for that, with KL (soft) or SFT (hard) training objectives.

the design fulfilled its destiny as shown in the results, the model did become better random distribution generators.

clear paper flow the story line and experiments fit naturally into the theme.

Critical weaknesses

end-goal unclear or maybe it's an motivation issue, why in the first place do people want to use LLMs as random distribution samplers? I don't think the authors properly motivated the work.

performance drop is significant on tasks like GSM8K, the drop can be more than 10%. which is not a difficult dataset.

Questions for authors

Can this methods show better transfer onto things like creative writing? That would make the method much stronger.

Also see my weakness section. Overall an interesting idea.

3 comments
AuthorsOpenReview ↗

[1/2] Motivation and GSM8K

We thank the reviewers for their careful and constructive reviews. We are glad that the reviewers found this an important research area [qVEd], with an interesting setting and an intuitive solution [1PbV], solid evidence of improved sampling [1PbV, v1hC], a wide range of experiments [v1hC], an efficient trie-based training method [qVEd], and a clear presentation [1PbV, qVEd, v1hC]. We especially thank them for highlighting the transfer from mathematical distributions to natural language as the most interesting finding [qVEd, v1hC]. All changes in the revision are marked in blue.

"why in the first place do people want to use LLMs as random distribution samplers?"

We agree that the paper made this case too abstractly. We care about cases where the model itself has to do the sampling, because the possible answers are not known in advance and the target distribution is given in natural language. Prior work shows that when models generate data, such as synthetic training sets or evaluation questions, their outputs are repetitive and biased and miss the requested proportions [1, 2]. In open-ended requests such as creative writing, models keep returning the same few answers [3, 4].

Sampling from a stated distribution is also a capability one can reasonably expect from models that are used as general-purpose reasoners and problem solvers. We train on mathematical distributions because their targets are exact, cheap and checkable, and the skill transfers to language, as the creative-writing results below show. The revision makes these points clearer.

"performance drop is significant on tasks like GSM8K, the drop can be more than 10%."

We thank the reviewer for pushing on this front, which led us to run new evaluations on the full GSM8K test set. The drop turns out to affect only three of the twelve models, and the reason is that they answer directly with a number instead of reasoning. We reran the evaluation with reasoning enabled and found that most of the lost accuracy is restored for two of the three affected models (the two Qwen3 models), and even the third, GPT-OSS-20B, recovers most of the drop with a lower learning rate. The table gives accuracy on the 1,319 test problems (not TinyBenchmarks' 100) for all 36 checkpoints, with greedy decoding and flexible answer matching, with the reasoning mode off and, for models that have one, on.

ModelReasoning mode off (base / soft / hard)Reasoning mode on (base / soft / hard)
Qwen3-0.6B0.35 / 0.06 / 0.100.76 / 0.71 / 0.74
Qwen3-1.7B0.71 / 0.55 / 0.690.85 / 0.84 / 0.86
Qwen3-4B0.84 / 0.86 / 0.850.92 / 0.92 / 0.93
Qwen3-8B0.86 / 0.89 / 0.880.92 / 0.93 / 0.93
Qwen3-14B0.88 / 0.91 / 0.900.95 / 0.94 / 0.95
Gemma-3-1B0.31 / 0.27 / 0.25n/a
Gemma-3-4B0.75 / 0.75 / 0.74n/a
Gemma-3-12B0.87 / 0.87 / 0.87n/a
Gemma-3-27B0.90 / 0.90 / 0.90n/a
Llama-3.2-1B0.29 / 0.28 / 0.30n/a
Llama-3.2-3B0.68 / 0.66 / 0.65n/a
GPT-OSS-20B0.67 / 0.06 / 0.130.93 / 0.71 / 0.24

Our fine-tuning encourages these direct answers, since every training answer is a single number, given with the reasoning mode off. With the reasoning mode on, only hard-target GPT-OSS-20B still skips its reasoning, in 88% of answers. For GPT-OSS-20B, a learning rate of 5e-5 instead of the paper's 2e-4 keeps most of the calibration gain with soft targets (held-out W1 0.18 against 0.15, base 0.64) and brings GSM8K back to 0.56 with the reasoning mode off and 0.84 with it on (base 0.67 and 0.93). The revision reports these full-test results instead of tinyGSM8K, adds the learning-rate sweep to Appendix D.8 and updates the limitations.

References

[1] Y. Yu et al., Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias, 2023, NeurIPS D&B.

[2] M. Zhao, Y. Du, M. Wang, Large Language Models Are Bad Dice Players: LLMs Struggle to Generate Random Numbers from Statistical Distributions, 2026, ACL.

[3] Y. Zhang et al., NoveltyBench: Evaluating Language Models for Humanlike Diversity, 2025, COLM.

[4] L. Jiang et al., Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond), 2025, NeurIPS D&B.

AuthorsOpenReview ↗

[2/2] Transfer to creative writing

"Can this methods show better transfer onto things like creative writing?"

Yes. On NoveltyBench's creative-writing prompts, fine-tuning raises utility for 9 of the 12 models even though they are trained only on numbers, and we consider this the most interesting finding. NoveltyBench [3] has 20 creative-writing prompts. We draw 10 samples per prompt, and its utility credits each one only if it differs from the earlier ones, weighted by its quality under a reward model. Across the 12 models, utility rises by 18% on average (95% CI 12–24%), and by up to 58%. On its 1,000 real user requests (WildChat), 8 of 12 models gain 11–31%. Per model, before → after soft-target fine-tuning:

ModelCreative writing, 20 prompts
Qwen3-0.6B4.00 → 2.17 (−46%)
Qwen3-1.7B3.90 → 4.87 (+25%)
Qwen3-4B3.78 → 5.34 (+41%)
Qwen3-8B3.90 → 5.88 (+51%)
Qwen3-14B3.75 → 5.91 (+58%)
Gemma-3-1B3.30 → 4.19 (+27%)
Gemma-3-4B3.17 → 4.65 (+47%)
Gemma-3-12B3.50 → 4.32 (+23%)
Gemma-3-27B2.86 → 4.12 (+44%)
Llama-3.2-1B2.91 → 3.45 (+19%)
Llama-3.2-3B3.93 → 3.79 (−4%)
GPT-OSS-20B5.09 → 3.48 (−32%)

The main exceptions are Qwen3-0.6B and GPT-OSS-20B, the two with the largest GSM8K drop. At the lower learning rate of 5e-5, GPT-OSS-20B's creative-writing utility is close to the base model's (4.92 against 5.09).

References

[3] Y. Zhang et al., NoveltyBench: Evaluating Language Models for Humanlike Diversity, 2025, COLM.

The revision + new experimental results significantly improves the empiricals of the paper. Though not totally convinced on the motivation, the paper is in a much better standing.

Will raise my score accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.