acceptodds
Under review as a conference paper at ICLR 2027

Scaling Laws of Ensemble Uncertainty in Language Models

Abstract

Many use cases for large language models (LLMs), from AI safety to exploration and human interaction, require estimates of epistemic uncertainty. While ensembles are a practical option, multi-seed experiments show that their disagreement is small compared to individual token entropy and significantly underestimates the model's true epistemic uncertainty. We try to shed light on these observations and broader aspects of ensembling for uncertainty estimation in LLMs. The upshot of our findings is that ensembling becomes more profitable both for uncertainty estimation and as an alternative to scaling model size at larger scales. We relate epistemic uncertainty to reducible loss in calibrated models, separating the portion visible through ensemble disagreement from an invisible portion that includes model misspecification. We show that the visible portion decreases as a mild power law with model size and changes little with additional training data or context beyond an initial stabilization range. The invisible portion decreases faster, so ensembles reveal a growing share of average epistemic uncertainty as models scale. These scaling trends also predict that ensembling becomes more competitive in terms of loss as the size of a single model increases, with matched total parameters and fixed training data. We then identify a second power law for ensembles: relative to independently trained ensembles, that relates the ensemble diversity to the shared portion of the training across ensemble members. This implies that late branching is heavily penalized in diversity, and the issue is even more severe when ensembling with low-rank adapters.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
epistemic uncertainty, scaling laws, ensembles, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sJ6wOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how much of an LLM's epistemic uncertainty can be captured by deep ensembles as models scale. Treating epistemic uncertainty as the model's reducible loss, the paper splits it into a part that is visible through disagreement among ensemble members and an invisible part consisting of errors shared by all members, including model misspecification.

The main claim is that ensemble disagreement shrinks only slowly with model size and is largely insensitive to more training data or longer context, whereas the invisible part shrinks faster. As a result, ensembles reveal a growing share of epistemic uncertainty at larger scales, and an ensemble of smaller models can become competitive with a single larger model of the same total size. The paper also shows that ensembles whose members share part of their training history (late branching, LoRA) lose much of their diversity in a predictable way.

These claims are supported empirically by fitting scaling laws to ensembles built from public model families and to a newly trained family of byte-level models, with extrapolations to larger scales that depend on an assumed irreducible loss.

Critical strengths
  1. Scale and care of the empirical study. Multi-seed pretraining experiments are expensive and rarely available, and the paper assembles a broad set of ensemble comparisons across public model families together with a newly trained model family that allows controlled measurements (e.g., separating the effects of model size, training data, and context length) that public checkpoints do not support. The analyses are accompanied by extensive sensitivity checks and a transparent discussion of limitations. This matters to my assessment because, even where the interpretation is debatable, the measurements themselves are a useful reference for the community.

  2. The late-branching and LoRA results. The finding that ensembles sharing training history lose much of their diversity, in a predictable way, is measured in a controlled setting and does not depend on the unknown irreducible loss. It is also directly relevant to practice, since cheap ensembles such as LoRA ensembles and trajectory-based Bayesian approximations are commonly used for LLM uncertainty. I consider this the most robust and actionable contribution of the paper.

Critical weaknesses
  1. The central conclusion depends on an unknown quantity. The growing visible share is computed relative to the irreducible loss, which is unknown and approximated by the best available model's loss. The trend weakens as this value is lowered and reverses below a threshold that the paper cannot rule out, so the headline result rests on an unverifiable assumption.

  2. Large-scale evidence comes mostly from pairs of different models. True seed ensembles reach only about 1B parameters; beyond that, the "ensembles" pair different models (e.g., Qwen2.5 and Qwen3), whose disagreement reflects differences in data and training recipes rather than epistemic uncertainty. Yet these pairs drive key results, such as the highest projected visible share and the main example of an ensemble beating a larger model with fewer parameters, which are then extrapolated far beyond the measured range.

  3. Mismatch between motivation and results. The introduction motivates the work with applications that need uncertainty at individual inputs, such as active learning and deferral. However, the paper's notion of epistemic uncertainty is defined only on average over contexts, and the main results concern averaged quantities. Pointwise uncertainty is addressed only briefly, using a reference-model proxy with modest predictive accuracy, so it is unclear what the scaling results imply for the applications that motivate the paper.

  4. Limited novelty. The loss decomposition, the decrease of ensemble disagreement with scale, uncertainty scaling laws, and ensembles outperforming larger models have all been reported in prior work, as the authors acknowledge. The contribution is therefore mainly an empirical characterization at LLM scale. Relevant work suggesting that larger single models can match the benefits of ensembles (e.g., [1]) is not discussed.

[1] Abe, Taiga, et al. "Deep ensembles work, but are they necessary?." Advances in neural information processing systems 35 (2022): 33646-33660.

Questions for authors
  1. Key results use pairs of different models trained on different data and recipes (e.g., Qwen2.5 + Qwen3). Is there precedent for treating their disagreement as epistemic uncertainty, and how would the main results change with seed ensembles only?

  2. The growing visible share requires the irreducible loss to exceed a threshold. What evidence supports this beyond the best achieved model loss?

  3. Does a growing average visible share imply better per-input uncertainty estimates at scale?

  4. The fitted projections show no crossover for Pythia or Qwen3 up to 1T. How is this consistent with the claim that ensembling becomes increasingly competitive at scale?

4 comments

We thank the reviewer for their valuable feedback. Below, we believe we have addressed all of their questions and concerns expressed as weaknesses. We would be happy to address any remaining issues during the discussion period.

Weakness 1 and Question 2: The central conclusion depends on an unknown quantity... The trend weakens as this value is lowered and reverses below a threshold that the paper cannot rule out.

A: First, please note that our scaling law for the spread and the late branching law are independent of the irreducible loss. Only the results of Figures 5 and 7 use this estimate. Concretely, what depends on the noise floor is the exponent in the Chinchilla scaling law, which is well-studied and generally believed to be around .3.3 but may vary for specific datasets and for smaller studies. Our conclusions hold as long as this number is larger than the small exponents we observe for the spread ≈[.11−.16]\approx [.11-.16]. Therefore, our trend is already supported by estimates from the existing LLM scaling law literature.

Independently of existing studies, we get a lower bound for the irreducible loss using a nominal 95% F-based profile confidence interval, and furthermore we now report the same monotonically increasing trend of the visible fraction using a reference model; see Fig. 6 (top row).

Finally, we have visualized the irreducible loss floor and the model-size scaling of the loss. Qualitative inspection of Fig. 2(b) suggests a conservative estimate, and this same range is used in drawing the uncertainty bands for Figures 5 and 7.

Weakness 2 and Questions 2 and 4: Large-scale evidence comes mostly from pairs of different models... Yet these pairs drive key results, such as the highest projected visible share and the main example of an ensemble beating a larger model...

A: First, please note that most of the theory claims and many of the empirical results are true for heterogeneous ensembles. In particular, the theory of Section 2.2 applies to any ensemble; the only exception is the justification for normalization by S/(S−1)S/(S-1), which can be dropped without affecting our conclusions. What differs when going from a seed ensemble to an arbitrary ensemble is that the mean member loss Lˉ\bar{L} does not reflect individual members any longer, and therefore the gain due to ensemble spread may not improve over the best ensemble member in terms of (reducible) loss.

Second, we have now clearly separated the results for seed ensembles from other types in all reported experiments and, more importantly, all conclusions can be supported even if we confined the results to seed ensembles. Our seed ensembles range in size from 0.8 M to 7B, approximately four orders of magnitude, and the trends claimed by the paper persist across this range. We now provide examples of seed ensembles beating a larger model in two regimes (parameter-matched and compute-optimal) in Appendix G.3 (Table 23). We also clearly show the theoretical prediction of how the gap should close between ensemble and individual models vs empirical observations for seed ensembles in both of these scenarios; see the updated Figure 7.

Weakness 3 and Question 3: Mismatch between motivation and results... Pointwise uncertainty is addressed only briefly, using a reference-model proxy with modest predictive accuracy, so it is unclear what the scaling results imply for the applications that motivate the paper.

A: To make the direct connection between the average trend indicated by scaling laws and individual context-based uncertainties clearer, we added a set of plots (top row of Fig. 6) that show the visible fraction of uncertainty when using a reference model. This quantity is the exact analogue of what the scaling law predicts. In this experiment, the visible fraction is calculated by dividing the ensemble uncertainty/spread by the approximate reducible loss estimated by the KL to the reference model. We see a similar trend, but shifted upward compared to Fig. 5b because the reference models lower-bound the irreducible loss.

Q: Limited novelty. ... The contribution is therefore mainly an empirical characterization at LLM scale. Relevant work suggesting that larger single models can match the benefits of ensembles (e.g., [1]) is not discussed

A: Thank you for the pointer; we have added a brief discussion of Abe et al. (2022) in Section 3.4.

What is novel compared to the prior work, including the one pointed out by the reviewer, is that we characterize the benefits of ensembling both in achieving better accuracy and in uncertainty estimation, through power laws that relate them to model size and branching strategy. As a result, our conclusions can be the opposite of what is reported in such prior works depending on these parameters. For example, the paper above suggests that a single model of appropriate size can match the performance of an ensemble, while we draw the opposite conclusion with a more accurate statement: assuming our scaling laws, if we match the ensemble and the single model (either by total parameters and total compute, or in terms of compute-optimality and total compute), an ensemble always outperforms a single model beyond some model scale. This cross-over is now clearly characterized solely based on the two scaling laws in our claim of Section 3.4. The same is true for our arguments about uncertainty: prior work, such as Kirsch et al. (2024), reports that the ensemble diversity collapses, which in their terms questions the effectiveness of ensembling at scale. Using our scaling laws, we show the opposite: that despite the shrinkage of diversity, due to the nature of power laws, ensembling becomes more effective for uncertainty estimation at scale.

Question 1: Is there precedent for treating [the] disagreement [in heterogeneous ensembles] as epistemic uncertainty?

A: Yes, this is a common practice in various data modalities. We can provide a list of citations if required. Given the relationship between ensembling and Bayesian model averaging, ensembling different architectures is analogous to extending the model class in the Bayesian setting.

The remaining questions are answered in response to the weaknesses above.

Thank you for the response and the revised manuscript. The separation of seed and heterogeneous ensembles, App. E.1, and Fig. 6(f–i) and 16 address my Q3 and much of W1. I have a few remaining questions.

  1. Fig. 14 shows that the trend reverses at E=0E = 0. At what value of EE does it reverse, and how close is this to the lower end of the retained range? The trend also looks fairly flat near the lower end, so it would be good to mention this in the main text.

  2. The 6.9B Pythia pair is excluded from the primary fits (App. B.2), so the seed-only fits seem to go up to about 1B rather than 7B. Could you clarify this? References for my Q1 would also be helpful.

  3. The crossover sizes in App. G.2 are far beyond current model scales, and the measured seed ensembles in Fig. 7 still underperform the larger model. I think this should be stated in the main text, and the abstract and the claim of refuting Abe et al. [1] should be toned down accordingly.

  4. In Table 23, the matched-cost comparison is essentially a tie, as App. G.3 also notes, and the fixed-data reference is not annealed. How does this support that the pair beats a larger model in both scenarios?

References

[1] Abe, Taiga, et al. "Deep ensembles work, but are they necessary?" NeurIPS, 2022.

We appreciate your engagement!

We identified a recent release of the Delphi seed ensemble family of models with up to 9.6B parameters, and we added its results to the paper. It supports the general identified trends and further addresses your concern about larger seed ensembles.

Q: At what value [of the floor] does [the trend] reverse, and how close is this to the lower end of the retained range? The trend also looks fairly flat near the lower end, so it would be good to mention this in the main text.

A: The table below shows the reversal threshold for different families. The uncertainty range for the floor is [0.26019,0.33373], so the reversal thresholds do not overlap with our uncertainty for the floor. We have added this table to the same appendix section. Please note that Figure 5 shows the full range with the uncertainty band, so readers can see the lower end of the slopes; however, we have also added a sentence about this in the main text.

FamilyReversal threshold
Character-0.35597
Pythia0.18996
DataDecide C40.20088
DataDecide Dolma0.17154
Dense Qwen2.5/Qwen3 pairs0.14397
Delphi0.20384
Pythia, if we include 6.9B model0.14014

Q: The 6.9B Pythia pair is excluded from the primary fits (App. B.2), so the seed-only fits seem to go up to about 1B rather than 7B. Could you clarify this?

A: Pythia 6.9B is a true seed pair; however, it is from an older Pythia release, so we excluded it from the family fit. We still measure its loss and spread, and it does support our claims about visible fraction scaling.

Q: The crossover sizes in App. G.2 are far beyond current model scales, and the measured seed ensembles in Fig. 7 still underperform the larger model. I think this should be stated in the main text, and the abstract and the claim of refuting Abe et al. [1] should be toned down accordingly. In Table 23, the matched-cost comparison is essentially a tie, as App. G.3 also notes, and the fixed-data reference is not annealed. How does this support that the pair beats a larger model in both scenarios?

A: We adjusted the phrasings the reviewer referenced to make them more accurate and added the recommended statements, except for the abstract, which simply states the increased competitiveness of ensembles with scale. We note that the crossover range for the two families discussed starts around 20B to 3T model sizes, which are realistic for today's large open-weight models, but the upper end of the interval includes much larger models.

We also added specialized validation sets (i.e., more OOD datasets). Since the spread increases (ensemble members disagree more), the advantage of ensembling in lowering the loss also increases for these datasets, and here we see many cases of crossover among seed ensembles; see Figure 18.

Q: References for my Q1 would also be helpful.

A: Please find below some examples from different areas:

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep Residual Learning for Image Recognition. CVPR.

Zaidi, S., Zela, A., Elsken, T., Holmes, C., Hutter, F., & Teh, Y. W. (2021). Neural Ensemble Search for Uncertainty Estimation and Dataset Shift. NeurIPS.

Antorán, J., Allingham, J. U., & Hernández-Lobato, J. M. (2020). Depth Uncertainty in Neural Networks. NeurIPS.

Wang, X., Kondratyuk, D., Christiansen, E., Kitani, K. M., Alon, Y., & Eban, E. (2022). Wisdom of Committees: An Overlooked Approach to Faster and More Accurate Models. ICLR.

Gontijo-Lopes, R., Dauphin, Y., & Cubuk, E. D. (2022). No One Representation to Rule Them All: Overlapping Features of Training Methods. ICLR.

Simonyan, K., & Zisserman, A. (2015). Very Deep Convolutional Networks for Large-Scale Image Recognition. ICLR.

Shwartz-Ziv, R., & Armon, A. (2022). Tabular Data: Deep Learning Is Not All You Need. Information Fusion, 81, 84–90.

Caruana, R., Niculescu-Mizil, A., Crew, G., & Ksikes, A. (2004). Ensemble Selection from Libraries of Models. ICML.

Erickson, N., Mueller, J., Shirkov, A., Zhang, H., Larroy, P., Li, M., & Smola, A. (2020). AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data. arXiv:2003.06505.

Jiang, D., Ren, X., & Lin, B. Y. (2023). LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion. ACL.

Chen, L., Zaharia, M., & Zou, J. (2023). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv:2305.05176.

Wang, J., Wang, J., Athiwaratkun, B., Zhang, C., & Zou, J. (2025). Mixture-of-Agents Enhances Large Language Model Capabilities. ICLR.

Reviewer_9mE7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates LLMs and divides epistemic uncertainty into two portions: visible and invisible. The visible portion is defined to be the seed variance, that is, the difference between the mean member's loss (of an ensemble of models) and the pooled ensemble loss, while the invisible is the difference between the pooled ensemble loss and the irreducible loss. Through empirical validation, the work discovers that this invisible portion of epistemic uncertainty decreases faster than the visible as model size grows. Subsequently, as model size grows, a greater portion of the epistemic loss becomes contained by the visible portion. A secondary contribution resides in an analysis of diversity among ensembles as a result of shared training, finding deterioration even with relatively small shared training histories. LORA degradation is particularly egregious in this context.

Critical strengths
  1. Detailed paper with interesting fundamental claims of how epistemic uncertainty is influenced by the ensemble and the size of ensemble members.
  2. The claims are important for the field, and are likely of interest to anyone seeking to capture epistemic uncertainty with ensembling.
  3. The core of the claims are backed with extensive evidence.
Critical weaknesses
  1. The claim that "ensembling becomes more profitable...as an alternative to scaling model size at larger scales" is much more weakly supported than the rest of the contributions. This is primarily shown in the experiment for Figure 7, where no actual advantage in ensembling behavior is observed, merely projected.
  2. Expanding on the above, the experiment also only covers pair ensembles. It seems likely that greater ensembling divides up the budget even more and further deteriorates performance. This seems to be supported by the paper's own observed scaling law. Even with pair-ensembles, it seems that stronger models (such as Qwen3 vs Qwen2.5) become more difficult to be overcome by ensembling, and it is questionable if such models can even be overcome within any realistic or reasonable sizes — assuming the observed behaviours are indeed complete in their description of the phenomenon.
  3. Context seems important and understated. To borrow an analogy, suppose these experiments were performed with LLMs trained on 15th century knowledge and tested on modern developments, similar results may be attained, but ultimately what is "irreducible" and "aleatoric" in Eqn 2, is not truly so, and may be overcome with greater context. This EE, would therefore be higher, if there exists knowledge not available to any of the models used to estimate it. Consequently, the estimated EE is not the true EE, but would instead be Etrue+CE_{true} + C, where CC is constant and represents unknown but epistemic information. If CC is large, then the fraction in Figure 5b, when accounting for C, must shrink, because the ratio becomes visible/(visible+invisible+C)\text{visible} / (\text{visible} + \text{invisible} + C), and both visible and invisible shrink with model size. This may confound the precise effect the authors measure. However, the size of C also depends on what is being tested. This is likely not a huge problem for the paper since testing is not significantly OOD for training, but the paper could benefit from a precise statement on the matter. This may also serve to partly explain the difference Qwen2.5 and Qwen3 results.
Questions for authors

My questions mostly pertain to future work:

  1. How might ensembling with loss terms which compel diversity impact your results? Might it achieve the opposite effect of shared training?
  2. Do you expect these depiction of ensemble behavior be extendable to other modalities?
  3. How might one go about developing a deeper theory from these observations?
3 comments

We thank the reviewer for their valuable feedback. Below we believe we have addressed all of their questions and concerns expressed as weaknesses. We would be happy to address any remaining issues during the discussion period.

Weakness 1: The claim that "ensembling becomes more profitable...as an alternative to scaling model size at larger scales" is much more weakly supported...

A: We have expanded the results of that section and updated the corresponding Fig. 7. The figure now shows that the gap between the ensemble and the larger model—under both parameter and compute-matched or compute-matched and compute-optimal models—closes as predicted by the two scaling laws, and the cross-over claim is formalized in that section.

Additionally, using training checkpoints we were able to identify specific examples of cross-over for DataDecide seed ensembles, that we report in Appendix G.3, Table 23. The example shows that two independent DataDecide models can beat a larger model in both scenarios (i.e., when total parameters/compute are matched, and under compute-optimality and compute matched).

Weakness 2: ... the experiment also only covers pair ensembles. It seems likely that greater ensembling divides up the budget ...

A: Figure 7 now includes triples. Moreover, as seen in the formal claim of Section 3.4, something interesting happens as we increase the ensemble size: under the two scaling laws for loss and spread, and under matched budget (in either budget scenarios), the ensemble with more members will outperform the ensemble with fewer members "beyond some model size".

Weakness 3 (rephrased): Could missing context in the training set inflate the estimated irreducible loss and affect the visible-fraction trend?

A: There are two ways we can read the reviewer's question: (1) If they mean context in the training set, this can be seen already in the Chinchilla scaling law L(N,D)=E+aN−α+bD−βL(N,D)=E+aN^{-\alpha}+bD^{-\beta}: for a fixed data-set, model scaling leaves a residual due to insufficient data coverage on top of the irreducible loss. Not having an appropriate context in the training data is similar. However, as noted by the reviewer, this is not an issue for us since our validation sets are not OOD.

(2) If they mean context for individual prediction, then the irreducible loss (assuming calibration) is H(Y∣X)H(Y|X), which will indeed increase if X=[X1,X2]X=[X_1,X_2] is replaced by X=[X1]X=[X_1]. This is captured by the scaling law of the context length, but the reviewer is correct that there is no such term in the Chinchilla law. We did not understand the connection to Qwen 2 vs 3, but would be happy to investigate further if our clarification has not sufficiently addressed the issue.

Question 1: How might ensembling with loss terms which compel diversity impact your results?

A: Your question prompted new experiments that we have added to the paper (Appendix H.3). We see that a simple diversity penalty increases the spread, but mainly at the expense of high member losses, so that the overall ensemble loss stays the same. This is consistent across different model sizes. Intuitively, this means that while using a diversity penalty one captures more uncertainty, almost all of this added uncertainty can be attributed to the lower performance of individual member.

Question 2: Do you expect these depiction of ensemble behavior be extendable to other modalities?

A: The pooling identities in Section 2.2 in our theory section also apply to other categorical prediction tasks, with an analogous variance interpretation for Gaussian regression. If calibration remains negligible in other domains and spread follows a similar scaling law for those modalities (the wide ranging scaling results of Rosso et al. (2025) for BALD is encouraging for the latter point) then our conclusions on the benefit of ensembling at scale both for uncertainty estimation and loss reduction holds for the modality of interest. The third scaling law for late branching requires a separate validation.

Q: How might one go about developing a deeper theory from these observations?

A: We have been thinking about this but currently do not have a good answer. Figure 4 provides an empirical starting point, which shows disagreement resolves more slowly on difficult contexts. A next step could be to derive this contraction from a theoretical standpoint. On a separate front, the late branching law requires a better picture of the loss landscape geometry and provides an interesting empirical constraint for any such picture.

Thank you for the response and the additional experiments. The clarifications are helpful, and I agree with the Chinchilla framing. My point about Qwen2.5 and Qwen3 was only that they were trained on data from different periods, so their gap to the recent-text evaluation set (the C term in my review) may differ, and I wondered if it could partly explain their different fitted exponents.

Thank you for engaging in the discussion!

Yes, the two Qwen models have different training sets; the dataset used for Qwen 3 is almost twice as large and much more diverse (e.g., it covers four times as many languages). They also differ in subtle architectural choices.

If we try to estimate the irreducible floor, we get the following point estimates for each family.

Fitted modelsEstimated floor EE, nats/byte
Qwen2.5, 0.5–72B0.3598
Qwen3-Base, 0.6–14B0.3218
Qwen2.5, restricted to 0.5–14B0.3196

So including the larger/more models seems to have a more significant effect. Going back to your point about change in estimated floor due to missing context, we should have seen substantially different numbers for the two models if our validation set significantly favoured one of these models over the other (e.g., if it was in a language available in the Qwen 3 dataset but not in Qwen 2.5). Both of our validation sets (recent text and C4 validation) are safely considered in-distribution for all LLMs in our study, except our character-level model, which has a much smaller training set and therefore its own validation set.

Reviewer_1JVoOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how ensemble-based epistemic uncertainty scales with language model size. Under a calibration assumption, reducible loss is interpreted as epistemic uncertainty and split into a visible part (seed spread recovered by ensembling) and an invisible part (residual shared error). The authors fit power laws on public model families (Pythia, DataDecide, Qwen) plus self-trained character-level models, finding that visible spread decays slowly with size while the invisible part decays faster, so the recovered share grows with scale. They further derive a criterion for when an ensemble of smaller models beats a larger single model at matched total parameters, and identify a second power law for shared training: late branching loses spread roughly with the square root of the shared fraction, and LoRA ensembles retain only about 9–18% of independent-seed spread. Validation spans around 60 public model comparisons on recent web text and C4, controlled character-level training, and late-branching/LoRA sweeps.

Critical strengths
  1. The paper gives a clean and explanatory framework connecting reducible loss to epistemic uncertainty, and the visible/invisible split yields three testable scaling laws. This is consequential because it turns a vague question — does ensembling help more at scale? — into concrete, falsifiable predictions, and the predictions are supported across several model families.

  2. The empirical scope is unusually broad: public seed suites, recipe variants, cross-generation pairs, and controlled character-level models with matched data budgets and per-budget cooldowns. The late-branching and LoRA results are practically useful, since these are common cheap ways to build ensembles, and the paper shows quantitatively that they lose most of the diversity.

Critical weaknesses
  1. The visible/invisible split depends on two strong assumptions: that models are well calibrated, and on the value of the irreducible loss. Calibration evidence is mostly top-label ECE (about 0.4–1.6 points) on 96 recent-text documents, and full-distribution recalibration is only reported for character-level models and Pythia up to 1.4B, not for large Qwen. Since the plausible range of the irreducible loss is wide and the recovered-share numbers are sensitive to it, the quantitative claims (e.g., projected shares at one trillion parameters) should be read as conditional approximations rather than established values. This matters because the paper's headline conclusion — a growing visible share — is exactly the quantity most sensitive to these assumptions.

  2. Several comparisons used to support "ensembling beats a larger model" mix training recipes and model generations (e.g., Qwen2.5/Qwen3, DeepSeek-V3/V3.1), and the finite-member correction is only a descriptive rescaling for such non-iid members. The measured crossover examples (Qwen 14B+14B vs 32B) are therefore not clean seed comparisons, and the predicted crossovers rely on extrapolated loss and spread laws. This weakens the rigor of the practical recommendation.

Questions for authors
  1. Could you provide full-distribution recalibration results for the larger Qwen models, to support the visible/invisible split beyond top-label ECE?

  2. How much does the uncertainty in the irreducible loss shift the ensembling-vs-larger-model crossover point? Could you report an explicit interval rather than a single projected value?

1 comment

We thank the reviewer for their valuable feedback. Below we believe we have addressed all of their questions and concerns expressed as weaknesses. We would be happy to address any remaining issues during the discussion period.

Question and Weakness 1 Summary: Does full-distribution recalibration support the uncertainty interpretation, especially for large Qwen models?

A: Yes! Appendix A.3 and Table 4 now report recalibration gains for more than 30 checkpoints, including all Qwen models, up to 72B parameters, where we report gains below 0.34% of loss in all cases. Full recalibrations use three different methods of temperature, vector, and MLP which were originally applied only to smaller models.

Question 2 and Weakness 1 summary: How sensitive are recovered fraction and crossover sizes to the unknown irreducible loss, and can a crossover interval be reported?

A: We have now included uncertainty bands based on the estimated floor in ALL places where the plotted value depends on this estimate, including the plot of the visible fraction (Fig. 5b) and crossover in ensembling (Fig. 7). In Fig. 7, these bands are narrow and harder to see. Moreover, Fig. 2b now visualizes the estimated range of irreducible loss, which is quite conservative. Numerical intervals for the crossover location are in Appendix G.2.

Weakness 2 summary: Several comparisons used to support "ensembling beats a larger model" mix training recipes ...

A: The section now provides results for all families of models we study, and extrapolations are only made for seed ensemble families; see the new Fig. 7. In particular, we now consider two different scenarios (1) parameter and compute-matching of an ensemble with a single model, and (2) compute-matching under compute-optimality (previously we only had scenario 1). In both cases, the two scaling laws predict a cross-over at a finite scale (Eq. 10), which is visualized in Fig. 7.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global response [updated]

We thank all the reviewers for the time dedicated to the reading of our paper and for their feedback which has helped improve our paper. The reviews agree that the experiments are careful and wide in their scope. They also acknowledge that the findings can have significant practical implications. They generally raise several valid issues around the extent of support for the central claims, which we believe we have now addressed through better presentation of our results and new experiments that were not finalized at the time of submission. Below we list major changes to the paper in the current revision.

  1. We have revised the paper's writing to improve its clarity and to reduce repetition or presentation of the material that is clear from the figures. To this end, we have reduced the supporting material in the appendix to content that directly supports the claims and improved the appendix structure.

  2. We have ensured all claims that depend on uncertainty in the irreducible loss show this uncertainty using bands. To this end, we also show the estimated range of irreducible loss more explicitly in a new figure (2b).

  3. With the exception of late branching power-law, all central claims are now supported by four families of seed ensembles (our character-level model, Pythia, DataDecide, and Delphi) ranging more than four orders of magnitude in model size (.8M-9.7B), and at least one heterogeneous ensemble of Qwen models.

  4. Claims for context-dependent/reference-based estimate of uncertainty are now more directly supported by the plot of the visible fraction (Fig. 6) that parallels that of Fig. 5b and shows the same trend.

  5. Comparison of ensembles with a single larger model is now performed over two types of matched budget (matched total parameters/compute and matched total compute under compute-optimality), where the predictions using the scalings are matched against the observed gap in the new figure for both types. We also added examples of cross-over for the seed ensemble to Appendix G.3. We also made the claim in Section 3.4 more formal and showed the potential benefit of increasing ensemble size at larger and larger scales.

  6. Several more specific results, mainly reported in the appendix, are in response to reviewer questions, such as more extensive recalibration results and a study of the effect of diversity penalty.

  7. We added OOD validation sets (music, math proof, chemical smiles and chess; Figures 5c and 11), where the spread maintains a power-law form but increases in value. As a result, ensembling becomes more competitive with a larger model in terms of validation loss, and we see many examples of ensembles outperforming a larger model under both scenarios of matched budget (Figure 20).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.