acceptodds
Under review as a conference paper at ICLR 2027

Single-Model Ensembling via Test-Time Stochastic Ensembles

Abstract

Ensemble methods are a well-established paradigm of machine learning and have recently emerged as a successful technique for obtaining inference-time gains in deep neural networks. In contrast to prior work, we create a test-time stochastic ensemble (TTSE) from a single transformer by generating multiple stochastic forward passes from a frozen checkpoint and aggregating their outputs, without any additional training, checkpoints, or weight updates. We carefully investigate the precise design of the stochasticity and ensembling mechanism, finding that (1) adding isotropic Gaussian noise to Query and Key vectors is the most effective; (2) averaging probabilities is better than averaging logits; and (3) increasing the sample size enhances the differentiability of correct reasoning traces from incorrect ones. Across next-token prediction and reasoning tasks, our method consistently improves upon the deterministic-inference baseline and alternative stochastic methods, suggesting that successful ensemble methods do not actually require training multiple models, which are very expensive in practice.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Ensemble methods, Training-free inference, Stochastic forward pass, LLM reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fFNNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how inference-time stochastic perturbations and output aggregation can improve the predictive performance of a single frozen Transformer without updating its parameters. The authors present a test-time stochastic ensembling (TTSE) framework organized around four design choices: noise location, noise distribution, aggregation rule, and sample size. Their concrete instantiation, stochastic attention ensembling (SAE), injects independent isotropic Gaussian noise into the Query and Key vectors used in attention computation. For next-token prediction, SAE averages predictive probabilities across stochastic forward passes. For mathematical reasoning, it repeatedly scores previously generated candidate reasoning traces and aggregates these scores to support trace selection or answer voting. Experiments cover language models of different sizes and four mathematical competition datasets. The authors report improvements over deterministic inference and several alternative perturbation schemes under the evaluated settings. Their analyses further indicate that probability averaging outperforms logit averaging and that increasing the number of stochastic samples helps distinguish correct reasoning traces from incorrect ones.

Critical strengths

1、 The proposed method is simple and training-free, introducing Q/K perturbations into frozen Transformers without parameter updates. Its lightweight design facilitates integration into existing models.

2、The paper provides comprehensive ablations on perturbation locations, noise scales, sample sizes, and aggregation strategies, supporting its design choices.

3、The paper observes that stochastic ensembling increasingly separates correct and incorrect reasoning traces, improving trace selection and offering insights into their different responses to perturbations

Critical weaknesses

1、SAE requires 50 stochastic forward passes for next-token prediction and 16 passes per candidate reasoning trace, adding substantial inference cost. A comparison under matched compute budgets, such as using the extra computation for more reasoning traces, self consistency, or verification, would better demonstrate its practical value.

2、Next token prediction experiments use GPT models trained on FineWeb, while reasoning experiments focus on mathematical trace selection with Qwen-derived models. Generalization to other model families and the impact on end to end generation remain unclear. Further validation or narrower claims would address this concern.

Questions for authors

Q1. The proposed method improves performance through multiple stochastic forward passes, typically using N=16 for mathematical reasoning tasks and N=50 for next-token prediction, which introduces substantial additional inference overhead. Could the authors report the actual inference time and computational cost? Furthermore, under the same computational budget, does SAE still offer performance advantages over alternatives such as generating more reasoning traces, increasing the number of samples, or employing stronger verification methods?

Q2. I suggest discussing the relationship between TTSE and Calibrating Scientific Foundation Models with Inference Time Stochastic Attention, which constructs inference-time ensembles in frozen Transformers using post-softmax multinomial sampling. Could the authors clarify how this approach differs from SAE’s Q/K Gaussian perturbation? A comparison under a common experimental setting would be helpful if feasible. Given the publication timeline, I consider this a suggestion rather than a requirement for acceptance.

Reviewer_9WPmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes test-time stochastic ensembling (TTSE), which runs a frozen model N times with injected noise and aggregates the outputs. Its main instance, SAE, adds Gaussian noise to Query and Key vectors and averages the resulting probabilities. The authors report that SAE reduces validation loss more than other noise locations with gains growing with model size and that applying it to each scoring rules improves trace selection.

Critical strengths
  • The method is training-free with only a small change to attention, and the next-token ablations over noise location, noise scale and aggregator are systematic.
  • The improvement is consistent in direction across model sizes for next-token prediction and scoring rules for reasoning trace selection.
Critical weaknesses
  • SAE is chosen through an empirical search over noise locations but the paper does not clearly explain why Query and Key noise works better or why it separates correct from incorrect traces.
  • The paper suggests that ensembling does not require training multiple models but the gains of SAE yields modest gains and requires N forward passes per input without a compute-matched comparison.
Questions for authors
  • Please clarify why Query and Key noise is more effective than the other noise locations.
  • Please clarify what drives the wider log-probability gap between correct and incorrect traces under SAE.
  • Could the authors add a compute-matched comparison, such as spending the same additional compute on generating more traces?
Reviewer_DgU3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper improves ensembling by adding noise to the attention mechanism of a frozen model and aggregating multiple stochastic forward passes.

Critical strengths
  1. Overall, I think the proposed method can be effective (although its novelty is limited compared with existing Stochastic Attention methods such as Pei et al.).
  2. In the experiments, the authors analyze different design choices in reasonable detail, including noise location, noise scale, sample size, and aggregation method.
Critical weaknesses
  1. Novelty is limited and the core design is relatively direct.
    The method mainly adds noise to attention and aggregates multiple outputs. Although the authors systematically explore different designs, the methodological novelty remains limited relative to existing stochastic inference and ensembling methods. The closest prior work is Stochastic Attention (e.g., Pei et al.).

  2. Inference cost is high, and it is still unclear whether the gains justify the cost.
    When scoring reasoning traces, SAE typically runs N=16N=16 stochastic forward passes per trace. The paper does not sufficiently show that SAE remains better than existing methods under a similar total compute budget.

  3. The organization and writing of the related work section are not clear enough. In addition, the formatting of the appendix is not sufficiently professional.

Questions for authors

Please refer to the above comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.