- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies when dropout followed by normalization can be approximated by a deterministic scalar rescaling of the original output. The main idea is to decompose the dropout output into a shared normalization-induced scaling term and a residual numerator perturbation, leading to an analytic coefficient computable from the representation and dropout rate without Monte Carlo sampling. Theoretically, the paper derives concentration and MSE-optimality results showing that the approximation is accurate when the normalization radius concentrates and the output-visible numerator noise is small, and further proves a error rate under independent function-preserving replication. Empirically, it validates these predictions on MNIST networks, ten pretrained language models, and FaceNet, including comparisons to fitted scalars, independent/shared masks, and Monte Carlo mean targets.
Critical strengths
-
The mathematical framework is self-consistent, and the theoretical presentation is notably careful about its scope and assumptions.
-
The experimental design goes well beyond simply validating the main prediction, and the authors are transparent about the limits of applicability.
Critical weaknesses
-
The main limitation is the gap between the strength of the theoretical framework and its current practical scope. The cleanest guarantees apply to a local dropout–normalization setting, and the language-model experiments are deliberately constructed to match this setup by resampling only a selected late residual branch. While the paper evaluates several pretrained language models, it does not yet demonstrate that the analytic framework scales to contemporary large-scale LLMs or to realistic full-network dropout, where the local formula no longer directly applies. This is particularly unfortunate given that such large models are arguably where avoiding expensive Monte Carlo inference would be most practically valuable.
-
The practical gains are meaningful but somewhat specialized and uneven across settings. The largest improvements appear when approximating Monte Carlo mean predictions, whereas the scalar component often explains only a small fraction of the variability of an individual dropout realization. This suggests that the current method is most compelling for a relatively specific class of inference-time averaging problems rather than as a broadly applicable approximation of dropout behavior.
Questions for authors
-
Can the authors provide evidence or a principled argument for how the local analytic correction relates to the scalar effect observed under full-network dropout? For example, can local corrections be composed or propagated across layers, or do quantities predicted by the local theory correlate with the best scalar fitted at the final logits? A positive result here would substantially strengthen the practical scope of the framework.
-
Can the authors quantify how robust the reported one-pass gains are to noise in the finite Monte Carlo reference? Since two independent MC-1024 estimates are already available, reporting the gains against each half separately, or a variance-corrected estimate of error to the latent expectation, would help determine whether the practical improvements remain significant beyond MC estimation noise.
4 comments
Response to Reviewer RrAN (cont'd)
The three 95% intervals reported in Table 3 are entirely above zero. Moreover, the relative error reductions are very similar when using either of the two independent MC-1024 references (A or B), suggesting that the observed gains are not strongly dependent on the particular dropout masks used to estimate the reference mean.
For FaceNet, mean absolute error reductions are 90.41%, 90.26% and 91.91% against the averages from A, B and both sets, respectively (, 512 pairs). The corresponding 95% intervals for the absolute reduction also lie above zero under bootstrap resampling of queries.
Thank you for the detailed response and the additional analyses. The new results address my main concerns. Overall, I find the paper sound, interesting, and well supported by both the theoretical analysis and empirical evaluation. My assessment remains positive.
Dear Reviewer,
We thank you for your time and for your encouraging feedback. We are currently working on an updated version of the paper and we expect to make it available as fast as possible.
Best regards, The authors.
Response to Reviewer RrAN
Q1. How does the local correction relate to full-network dropout?
We investigate this relationship by propagating the effects of dropout from multiple layers to the final representation and comparing the resulting scalar with both the local prediction and the scalar fitted at the final logits. The propagated correction is closer to the fitted scalar than the local correction in 12 of the 13 tested language models. We predict the mean logits under dropout as c+a(ℓ0−c), where ℓ0 is the output without dropout and c is the fixed part of the logits.
Dropout in every feed-forward block. We apply independent dropout to the output of every feed-forward block, with keep probability q=0.9, including both encoder and decoder blocks in T5 and BART. We use a first-order approximation around the dropout-free activations to propagate 32 independent perturbations from all these blocks to the final representation. The local prediction of a accounts only for dropout in the last block. We compare both predictions with afit, fitted on 64 separate masks with dropout in all blocks by minimizing centered-logit MSE, and evaluate centered-logit MSE on another 64 masks. Each model uses 40 positions, four from each of ten passages. The propagation, fitting and evaluation masks are independent.
In nine of the ten GPT-2, OPT, T5 and BART models from the paper, the error in a relative to afit falls by \textbf{21.8--83.4%} when using propagation instead of the local correction (see Table 2 in common response). In these same nine models, centered-logit MSE on separate evaluation masks falls by 1.30--2.40% relative to deterministic inference.
In Qwen, Llama and Mistral, the error in a falls by 57.2%, 41.9% and 40.3% relative to the local correction. Their MSE reductions against deterministic inference are 7.24%, 4.32% and 7.09%.
The eight-block MNIST experiment in Appendix D.2 also tests dropout across layers. In three trained models, using all propagated noise sources reduces the mean absolute error of the predicted MSE reduction from 0.0612 to 0.0284.
Appendix D in the paper gives a way to propagate the perturbations. We represent their first-order effects on the final representation as η=Meffξ, where ξ collects the dropout perturbation, then set a=1/Teff, where
Teff2=1+r02ρTr(Meff⊤PMeff).
We recall that ρ=(1−q)/q, P is the normalization projector and r0 is the radius without dropout. In practice, we estimate the trace term using the 32 propagated perturbations, as detailed in Appendix A. The propagation accounts for how later layers transform earlier perturbations. Nonlinear effects and interactions between masks are not captured by this first-order approximation and may contribute to its error. This extension is useful when these extra terms are small.
Q2. Do the gains hold with different sets of dropout masks?
The gains remain consistent across two independent sets of dropout masks, each containing 1,024 realizations (sets A and B). We average the predicted probabilities within each set and over all 2,048 masks. The analytic correction is unchanged; only the Monte Carlo reference used to evaluate its error differs.
The local correction reduces probability L1 error against all three averages in all thirteen (thirteen models × three MC references) language models. For the original ten models, the reductions are 2.77--63.45% against A, 2.52--62.86% against B, and 2.62--66.46% against the combined average. All 30 intervals for the absolute error reduction lie above zero. To obtain the 95% intervals, we resample passages with replacement and recompute the mean error reduction, using the same fixed Monte Carlo reference for both methods. Thus, these intervals quantify variability across passages, not across dropout masks. Robustness to mask sampling is assessed separately by comparing the gains obtained with the two independent mask sets A and B.
Table 3 here reports the same comparisons for the three added models on 40 positions from ten passages.
Table 3. One-pass probability errors against dropout averages.