- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how inference-time stochastic perturbations and output aggregation can improve the predictive performance of a single frozen Transformer without updating its parameters. The authors present a test-time stochastic ensembling (TTSE) framework organized around four design choices: noise location, noise distribution, aggregation rule, and sample size. Their concrete instantiation, stochastic attention ensembling (SAE), injects independent isotropic Gaussian noise into the Query and Key vectors used in attention computation. For next-token prediction, SAE averages predictive probabilities across stochastic forward passes. For mathematical reasoning, it repeatedly scores previously generated candidate reasoning traces and aggregates these scores to support trace selection or answer voting. Experiments cover language models of different sizes and four mathematical competition datasets. The authors report improvements over deterministic inference and several alternative perturbation schemes under the evaluated settings. Their analyses further indicate that probability averaging outperforms logit averaging and that increasing the number of stochastic samples helps distinguish correct reasoning traces from incorrect ones.
Critical strengths
1、 The proposed method is simple and training-free, introducing Q/K perturbations into frozen Transformers without parameter updates. Its lightweight design facilitates integration into existing models.
2、The paper provides comprehensive ablations on perturbation locations, noise scales, sample sizes, and aggregation strategies, supporting its design choices.
3、The paper observes that stochastic ensembling increasingly separates correct and incorrect reasoning traces, improving trace selection and offering insights into their different responses to perturbations
Critical weaknesses
1、SAE requires 50 stochastic forward passes for next-token prediction and 16 passes per candidate reasoning trace, adding substantial inference cost. A comparison under matched compute budgets, such as using the extra computation for more reasoning traces, self consistency, or verification, would better demonstrate its practical value.
2、Next token prediction experiments use GPT models trained on FineWeb, while reasoning experiments focus on mathematical trace selection with Qwen-derived models. Generalization to other model families and the impact on end to end generation remain unclear. Further validation or narrower claims would address this concern.
Questions for authors
Q1. The proposed method improves performance through multiple stochastic forward passes, typically using N=16 for mathematical reasoning tasks and N=50 for next-token prediction, which introduces substantial additional inference overhead. Could the authors report the actual inference time and computational cost? Furthermore, under the same computational budget, does SAE still offer performance advantages over alternatives such as generating more reasoning traces, increasing the number of samples, or employing stronger verification methods?
Q2. I suggest discussing the relationship between TTSE and Calibrating Scientific Foundation Models with Inference Time Stochastic Attention, which constructs inference-time ensembles in frozen Transformers using post-softmax multinomial sampling. Could the authors clarify how this approach differs from SAE’s Q/K Gaussian perturbation? A comparison under a common experimental setting would be helpful if feasible. Given the publication timeline, I consider this a suggestion rather than a requirement for acceptance.