acceptodds
Under review as a conference paper at ICLR 2027

Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs

Abstract

Recent work raises concerns that outputs from large language models (LLMs) tend to be under-diverse: they repeat or resemble one another more often than responses from the population they are meant to represent, a phenomenon known as mode collapse. In this work, we show that whether mode-collapse—or its opposite—occurs depends on the specific model and dataset used. Further, with sufficient supervised fine-tuning (SFT) data, LLM output diversity converges toward that of the target distribution from which fine-tuning data are sampled. To quantify this comparison, we measure the probability that two responses sampled independently conditional on the same fixed prompt coincide (collide). In one set of experiments, we use exact token sequences; in the other, we use the generalised version of that—the expected similarity between responses under a kernel. We define miscalibration as a nonzero model-minus-target collision gap. We derive a bias–variance decomposition of the expected gap between the model's and target's collision probabilities, showing that SFT is not inherently biased toward mode collapse or its opposite: finite-sample SFT can leave a model either under- or over-dispersed, depending on the model and dataset. Finally, we show that the absolute gap is bounded by the square root of the Kullback–Leibler (KL) divergence from the target distribution to the model. Consequently, a model sufficiently close to optimal under population cross-entropy cannot exhibit arbitrarily miscalibrated diversity. We test the decomposition and the bound in three experiments: (1) we fit 100 small transformers at each of 100 log-spaced sample sizes on each of two synthetic order-16 languages; (2) we fine-tune four LLMs using low-rank adaptation on responses from the General Social Survey, American National Election Studies, and World Values Survey; and (3) we repeat Experiment (2) on CodeNet, a dataset of human solutions to coding tasks, measuring program similarity with normalized Zhang–Shasha edit distance between canonical abstract syntax trees. We find substantial heterogeneity in over- and under-diversity across models and datasets. More target data moves model diversity toward the human (or synthetic target) level in all experiments, consistent with our theoretical predictions. These results show that diversity miscalibration can arise from finite-sample error and shrink as SFT better approximates the target distribution. Accordingly, as the sample size of target-distribution data increases, model diversity moves toward the target level.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, supervised fine-tuning, output diversity, mode collapse, over-dispersion, diversity calibration, collision probability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JLX1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies if supervised fine-tuning causes LLM mode collapse or if diversity miscalibration can be caused by finite-sample fitting. It measures diversity using collision probablity and provides a bias-variance decomposition showing that models can become either under-diverse or over-dispersed. Also, it proves that the collision gap is bounded by the sqr root of KL divergence between the model and target distributions.

the paper evaluates the claims using synthetic languages with known ground truth distributions, human survey responses from GSS, WVS, ANES and CodeNet programming solutions. Across these, increasing the amount of SFT data generally pushes the median collision ratio toward the target level, including from both under and over diverse starting points.

Critical strengths

Strengths

  • The theoretical framing is interpretable and useful. The bias-variance-alignment decomposition is one of the important contributions of the paper. it separates the collision gap into variance and bias terms that increase collision, and an alignment term that can act in both directions. This provides a clear way of understanding why finite-sample fitting can lead to under diversity or over dispersion depending on model and target distributions.

  • The paper challenges the common focus on under diversity by showing theoretically and empirically that miscalibration can happen in both directions, depending on the model and target

  • The evaluation is broad across several settings despite missing some analysis (discussed in weaknesses). The authors evaluate four LLMs, three large survey datasets, CodeNet, and a wide range of fine-tuning dataset sizes.

  • The synthetic experiment provides a strong setting for directly testing the theoretical decomposition. Because the target probabilities are known exactly, the authors can directly estimate variance, squared bias, and alignment across repeated fits. They train 100 independent models at each of 100 sample sizes for each synthetic language, allowing the components of the decomposition to be measured directly rather than inferred indirectly.

  • The intervention is practically simple. The reported calibration effect is achieved using standard LoRA-based SFT rather than a specialized diversity-preserving objective. This makes the result practically relevant as a simple baseline

Critical weaknesses

Weaknesses

  • Collision calibration is not the same as learning the target distribution. The main claim that fine-tuning "fixes mode collapse" seems stronger than what is actually shown by R→1R \to 1. Different distributions can have the same collision probability, so a model could still assign wrong probabilities to some answers and yet look perfectly calibrated under this metric. So what is really shown is that SFT calibrates the collision statistic, which is a narrower claim than recovering the actual target distribution or its modes.

  • The survey target is not independent from the fine-tuning data. For the survey experiments, the target collision is computed using all eligible respondents, including those used in the fine-tuning and validation splits. Because of this, it is hard to tell whether the model is really learning the underlying population distribution or just getting closer to the empirical sample it was trained on. I think using a completely held-out respondent set to compute the target would make this claim much stronger.

  • The theoretical mechanism is not directly verified in the real-data experiments. Theorem 1 says that if the KL divergence between the model and target goes to zero, then the collision gap should also go to zero. But the relevant KL is not actually measured in the survey or CodeNet experiments. The paper also mentions that the survey validation loss is not the same KL used in the bound, and that the CodeNet bound is not quantitatively informative for the reported results. So the experiments show that more data moves RR closer to 1, but they do not directly show the full mechanism of more data →\rightarrow lower KL →\rightarrow better diversity calibration.

  • The CodeNet experiment does not compare human and model outputs under the same validity condition. The human target only includes accepted Python solutions, while model outputs are included as long as they parse successfully. This means incorrect but structurally different programs can still increase the measured model diversity. Also, the decoding settings are not fully matched across models, and the paper notes that the two Qwen models use different default thinking behavior. These issues make the CodeNet results harder to interpret and weaken the claim that the same calibration effect clearly generalizes to open-ended code generation.

Questions for authors

Questions for the Authors

  1. The main result shows R→1R \to 1, but matching collision does not mean the model distribution approaches the target. Can the authors report a distribution-level metric (e.g. TV, JS, or KL) between model and human answer distributions for the survey experiments as NN increases? Does R→1R \to 1 correspond to actual distributional fit, or only to matching the collision statistic?

  2. The target collision is computed over all eligible respondents, including those already in the fine-tuning split. Can the authors recompute the target using only held-out respondents and report whether the R→1R \to 1 trends persist? This would help with distinguishing population level generalization from fitting a sample that possibly overlaps with the training data.

  3. The human target contains accepted solutions while model outputs only need to parse. Can the authors report pass rates and recompute diversity among only correct generated programs? Can they also clarify how much the unmatched Qwen decoding defaults affect the cross-model comparisons?

  4. Clarification question: The paper notes that the survey validation loss is not the KL governing the collision ratio, and that the CodeNet bound is not quantitatively informative. Do the authors have measurements of the option-conditioned KL for the survey experiment? If so, does it decrease with NN and track ∣R−1∣|R-1|? If not, how should the experiments be interpreted as validating Theorem 1 rather than simply being consistent with it?

Reviewer_V1WLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors overly concentrated and overly dispersed LLM outputs through collision-based measures. It develops a bias–variance analysis and evaluates fine-tuning on synthetic languages, three survey datasets, and CodeNet. With sufficient data, measured concentration generally approaches the reference.

Critical strengths
  • The empirical study challenges the assumption that LLMs are uniformly under-diverse by documenting both excessive concentration and over-dispersion.
  • Repeated independent synthetic fits and experiments across multiple models and domains strengthen the empirical contribution.
  • The appendices transparently explain the empirical targets and distinguish concentration calibration from full distribution recovery.
Critical weaknesses
  1. The connection between the theoretical analysis and the real-data findings remains primarily qualitative.

The theoretical analysis provides a useful framework for understanding why concentration errors can have either sign. The synthetic experiments also offer meaningful quantitative checks because the target distributions are known. However, the connection to the survey and CodeNet results is less direct.

The KL bound establishes that a model sufficiently close to the target distribution must also have a small absolute collision gap. It does not establish how training-data size, model capacity, or optimization determine that distributional closeness. Consequently, it does not quantitatively explain the data requirements or improvement trajectories observed in the real-data experiments.

Appendix E clearly acknowledges two additional obstacles. In the survey experiments, concentration is computed from probabilities normalized over answer options, whereas the reported training and validation losses cover the full-vocabulary answer encoding and termination tokens. These losses therefore cannot directly instantiate the KL bound for the evaluated option-conditioned distribution. For CodeNet, sequence-level KL is not estimated, and the appendix explicitly states that the bound is not quantitatively informative for the reported structural-similarity ratios.

These qualifications are appropriate, but they limit the theoretical contribution to explaining the real-data findings. The paper should more clearly distinguish quantitative verification in the synthetic setting from qualitative consistency in the real-data experiments. A stronger connection would require evaluating the relevant divergence and collision gap under the same distributional definition and showing that the resulting bound provides a meaningful constraint.

  1. Some main-text conclusions extend beyond the empirical targets directly evaluated.

The appendices carefully specify the scope of the measurements, but some main-text statements—particularly those concerning the restoration of “population-level heterogeneity”—are broader than these measurements directly support.

For the survey experiments, Appendix H.2 defines the reference using the observed, unweighted sample, including respondents contributing to training and validation. Matching this reference demonstrates calibration toward the empirical sample. It does not independently establish that the model reproduces a broader population distribution, particularly when sampling weights or differences between sampled respondents and the target population matter. This is a limitation on generalization, not evidence of undisclosed data leakage or an invalid empirical fitting experiment.

Similarly, CodeNet concentration is measured conditional on successful parsing. The same-problem comparison in Appendix H.3 usefully controls for changes in the evaluated problem set, but the metric still excludes unparseable generations. Improved structural calibration within this subset therefore does not, by itself, establish recovery of the diversity or quality of the complete output distribution.

The authors also correctly distinguish collision calibration from full distribution recovery in Appendix J. These distinctions should be reflected consistently in the abstract, introduction, and conclusions. Framing the findings as improved collision calibration toward the observed survey targets and improved structural calibration among parseable programs would make the claims more precise while preserving the paper’s central empirical contribution.

Questions for authors

see above

Reviewer_gToYOpenReview ↗
Rating
1
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether fine-tuning makes LLM output diversity closer to a target dataset. It measures how often two answers match. It gives a bias-variance decomposition and a KL divergence bound, then tests synthetic languages, three surveys, and CodeNet using LLMs.

Critical strengths
  1. The paper shows that models can be either too concentrated or too spread out relative to a target. Testing four LLMs on three surveys and CodeNet makes this a useful empirical observation.

  2. The synthetic study is thorough. Knowing the true probabilities allows the authors to measure all terms in the decomposition. Using distinct respondent pairs for the survey collision estimate is also a reasonable choice.

Critical weaknesses
  1. The theoretical contribution is relatively small. Eq. (6) is a standard, and Theorem 1 follows directly from Pinsker's inequality. The formulas appear correct, but it does not show that more fine-tuning data makes KL vanish. The title's claim seems stronger than the theorem.

  2. Matching the diversity score is a limited result. Two answer distributions can prefer opposite answers and still have the same collision probability.

  3. The target is not independently held out. The survey reference includes training and validation respondents and does not use survey weights (Appendix H.2). CodeNet also draws its human reference from both partitions. These results show fit to the observed samples, with limited evidence for wider population calibration.

  4. The code results do not establish useful diversity. Only programs that parse are scored, whereas the human reference contains accepted solutions. No execution-based correctness results are reported. Matching program structure could therefore include incorrect code. The same-problem check helps, but still conditions on parsing.

  5. Code decoding is not controlled across models. Appendix H.3 reports different Qwen thinking defaults and unlogged top-k settings. The claim that models start on opposite sides of the target needs a comparison with matched decoding settings.

  6. A simple calibration baseline is missing. For the survey answers, temperature scaling can change the collision score without new training data. Appendix J explains why matching this score differs from matching the whole distribution. An actual temperature baseline would help show what fine-tuning adds.

Questions for authors
  1. Do the CodeNet results hold with matched decoding and reported parsing and execution pass rates on the same problems?

  2. Do the findings hold with independently held-out human references, repeated fine-tuning seeds, and a fixed training-step budget?

  3. How does temperature calibration compare? Please also report answer-distribution errors and absolute collision gaps, rather than mainly median ratios.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.