acceptodds
Under review as a conference paper at ICLR 2027

Normalization and the Scalar Structure of Dropout

Abstract

Dropout randomly changes individual features in a neural network. We ask when its effect after normalization can be approximated by one common scalar rescaling. For a trained network and a given input, normalization divides the features by a shared factor. When many small independent contributions determine this factor, its relative variation can become small. We replace it by a value computed directly from the features and dropout probability. At this fixed radius, the scalar minimizes mean squared error. It rescales the part of the output affected by normalization and leaves the fixed part unchanged. We then quantify how much of the change caused by dropout this scalar removes in mean square and how much random variation remains. Under replication that preserves predictions and uses independent copy masks, the output error is bounded in probability at the inverse square root of the number of copies. For RMS normalization, a nonzero limiting covariance rules out a faster rate. Experiments on MNIST networks, a pretrained FaceNet embedding, and ten language models test the theory. In the language models, scalar rescaling reduces error against Monte Carlo means of probability vectors. In FaceNet, it reduces error against Monte Carlo means of cosine scores. Individual dropout outputs can still remain highly variable.

Primary area:
learning theory
Keywords:
uncertainty quantification, high-dimensional probability, concentration of measure, normalized representations

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RrANOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies when dropout followed by normalization can be approximated by a deterministic scalar rescaling of the original output. The main idea is to decompose the dropout output into a shared normalization-induced scaling term and a residual numerator perturbation, leading to an analytic coefficient 1/T1/T computable from the representation and dropout rate without Monte Carlo sampling. Theoretically, the paper derives concentration and MSE-optimality results showing that the approximation is accurate when the normalization radius concentrates and the output-visible numerator noise is small, and further proves a k−1/2k^{-1/2} error rate under independent function-preserving replication. Empirically, it validates these predictions on MNIST networks, ten pretrained language models, and FaceNet, including comparisons to fitted scalars, independent/shared masks, and Monte Carlo mean targets.

Critical strengths
  1. The mathematical framework is self-consistent, and the theoretical presentation is notably careful about its scope and assumptions.

  2. The experimental design goes well beyond simply validating the main prediction, and the authors are transparent about the limits of applicability.

Critical weaknesses
  1. The main limitation is the gap between the strength of the theoretical framework and its current practical scope. The cleanest guarantees apply to a local dropout–normalization setting, and the language-model experiments are deliberately constructed to match this setup by resampling only a selected late residual branch. While the paper evaluates several pretrained language models, it does not yet demonstrate that the analytic framework scales to contemporary large-scale LLMs or to realistic full-network dropout, where the local formula no longer directly applies. This is particularly unfortunate given that such large models are arguably where avoiding expensive Monte Carlo inference would be most practically valuable.

  2. The practical gains are meaningful but somewhat specialized and uneven across settings. The largest improvements appear when approximating Monte Carlo mean predictions, whereas the scalar component often explains only a small fraction of the variability of an individual dropout realization. This suggests that the current method is most compelling for a relatively specific class of inference-time averaging problems rather than as a broadly applicable approximation of dropout behavior.

Questions for authors
  1. Can the authors provide evidence or a principled argument for how the local analytic correction relates to the scalar effect observed under full-network dropout? For example, can local corrections be composed or propagated across layers, or do quantities predicted by the local theory correlate with the best scalar fitted at the final logits? A positive result here would substantially strengthen the practical scope of the framework.

  2. Can the authors quantify how robust the reported one-pass gains are to noise in the finite Monte Carlo reference? Since two independent MC-1024 estimates are already available, reporting the gains against each half separately, or a variance-corrected estimate of error to the latent expectation, would help determine whether the practical improvements remain significant beyond MC estimation noise.

4 comments
AuthorsOpenReview ↗

Response to Reviewer RrAN

Q1. How does the local correction relate to full-network dropout?

We investigate this relationship by propagating the effects of dropout from multiple layers to the final representation and comparing the resulting scalar with both the local prediction and the scalar fitted at the final logits. The propagated correction is closer to the fitted scalar than the local correction in 12 of the 13 tested language models. We predict the mean logits under dropout as c+a(ℓ0−c)c+a(\ell_0-c), where ℓ0\ell_0 is the output without dropout and cc is the fixed part of the logits.

Dropout in every feed-forward block. We apply independent dropout to the output of every feed-forward block, with keep probability q=0.9q=0.9, including both encoder and decoder blocks in T5 and BART. We use a first-order approximation around the dropout-free activations to propagate 32 independent perturbations from all these blocks to the final representation. The local prediction of aa accounts only for dropout in the last block. We compare both predictions with afita_{\mathrm{fit}}, fitted on 64 separate masks with dropout in all blocks by minimizing centered-logit MSE, and evaluate centered-logit MSE on another 64 masks. Each model uses 40 positions, four from each of ten passages. The propagation, fitting and evaluation masks are independent.

In nine of the ten GPT-2, OPT, T5 and BART models from the paper, the error in aa relative to afita_{\mathrm{fit}} falls by \textbf{21.8--83.4%} when using propagation instead of the local correction (see Table 2 in common response). In these same nine models, centered-logit MSE on separate evaluation masks falls by 1.30--2.40% relative to deterministic inference.

In Qwen, Llama and Mistral, the error in aa falls by 57.2%, 41.9% and 40.3% relative to the local correction. Their MSE reductions against deterministic inference are 7.24%, 4.32% and 7.09%.

The eight-block MNIST experiment in Appendix D.2 also tests dropout across layers. In three trained models, using all propagated noise sources reduces the mean absolute error of the predicted MSE reduction from 0.0612 to 0.0284.

Appendix D in the paper gives a way to propagate the perturbations. We represent their first-order effects on the final representation as η=Meffξ\eta=M_{\mathrm{eff}}\xi, where ξ\xi collects the dropout perturbation, then set a=1/Teffa=1/T_{\mathrm{eff}}, where

Teff2=1+ρTr⁡(Meff⊤PMeff)r02.T_{\mathrm{eff}}^2=1+\frac{\rho \operatorname{Tr}(M_{\mathrm{eff}}^\top P M_{\mathrm{eff}})}{r_0^2}.

We recall that ρ=(1−q)/q\rho=(1-q)/q, PP is the normalization projector and r0r_0 is the radius without dropout. In practice, we estimate the trace term using the 32 propagated perturbations, as detailed in Appendix A. The propagation accounts for how later layers transform earlier perturbations. Nonlinear effects and interactions between masks are not captured by this first-order approximation and may contribute to its error. This extension is useful when these extra terms are small.

Q2. Do the gains hold with different sets of dropout masks?

The gains remain consistent across two independent sets of dropout masks, each containing 1,024 realizations (sets A and B). We average the predicted probabilities within each set and over all 2,048 masks. The analytic correction is unchanged; only the Monte Carlo reference used to evaluate its error differs.

The local correction reduces probability L1 error against all three averages in all thirteen (thirteen models ×\times three MC references) language models. For the original ten models, the reductions are 2.77--63.45% against A, 2.52--62.86% against B, and 2.62--66.46% against the combined average. All 30 intervals for the absolute error reduction lie above zero. To obtain the 95% intervals, we resample passages with replacement and recompute the mean error reduction, using the same fixed Monte Carlo reference for both methods. Thus, these intervals quantify variability across passages, not across dropout masks. Robustness to mask sampling is assessed separately by comparing the gains obtained with the two independent mask sets A and B.

Table 3 here reports the same comparisons for the three added models on 40 positions from ten passages.

Modelerror without correctionone-pass errorerror reduction (%)95% interval for error decreasedifference between A and B (%)gain against A (%)gain against B (%)
Qwen3-4B0.117630.0762935.1[0.03280, 0.04975]15.434.935.0
Llama-3.1-8B0.078540.0448742.9[0.02615, 0.04076]13.341.843.1
Mistral-7B-v0.30.027180.0103961.8[0.01438, 0.01937]20.059.960.7

Table 3. One-pass probability errors against dropout averages.

AuthorsOpenReview ↗

Response to Reviewer RrAN (cont'd)

The three 95% intervals reported in Table 3 are entirely above zero. Moreover, the relative error reductions are very similar when using either of the two independent MC-1024 references (A or B), suggesting that the observed gains are not strongly dependent on the particular dropout masks used to estimate the reference mean.

For FaceNet, mean absolute error reductions are 90.41%, 90.26% and 91.91% against the averages from A, B and both sets, respectively (q=0.4q=0.4, 512 pairs). The corresponding 95% intervals for the absolute reduction also lie above zero under bootstrap resampling of queries.

Thank you for the detailed response and the additional analyses. The new results address my main concerns. Overall, I find the paper sound, interesting, and well supported by both the theoretical analysis and empirical evaluation. My assessment remains positive.

Dear Reviewer,

We thank you for your time and for your encouraging feedback. We are currently working on an updated version of the paper and we expect to make it available as fast as possible.

Best regards, The authors.

Reviewer_RtNtOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The article studies the impact of dropout in a high-dimensional feature space when used after RMS and Layer normalization layers. A surrogate method is proposed to compute the output of dropout quickly based on concentration inequalities in probability theory. Numerical results in small and large scale problem show the validity of the theory.

Critical strengths

A rigorous analysis of the dropout output after normalized layers is given, together with a simple rescale idea to compute an approximate dropout without requiring costly Monte-Carlo simulations.

The impact of the theoretical analysis on large-scale problems through numerical experiments are given, which shows the validity of the analysis of the dropout output across each of such layers.

Critical weaknesses

The analysis is local in its nature for each normalization and dropout layer, but this does not tell the end-to-end behavior of the whole neural network, so it would be better to evaluate numerically such effects in some well-defined problems such as image classification.

Questions for authors

Clarify the sentence on lines 155-156, regarding the centered logits, cosine scores. Are you talking about the l_0, which is computed from a normalization layer, or are you talking about how the input h is computed?

It is quite important that the quantity V in eq. 15 to be small so that your approximation error in eq 21 is small. Do you know if V is indeed small in practice (numerical experiments)?

Clarify the notation of V_n, x_{t,n} etc in eq. 23.

Section 6 is not so well written. It is not clear what the definition of l_{k,zeta}, l_{T_k} is in Theorem 7. Also, is this theorem related to Corr 3? If so, why not use n instead of k?

Is the idea of the k copies evaluated later in Section 7?

Why the definition of V(Q) in eq. 37 has no use of P as in eq. 15?

Report the end-to-end effect of your approximate scheme on well-defined problems, such as image classification on MNIST.

6 comments
AuthorsOpenReview ↗

Response to Reviewer RtNt

Q1. What is centered in lines 155--156?

At this stage, h∈Rph \in \mathbb{R}^p is the pre-normalization feature representation, and ℓ0,ℓζ∈Rm\ell_0, \ell_\zeta \in \mathbb{R}^m in Equation (4) denote the final layer outputs after normalization and the linear head. For softmax classifiers, we center the output logits using

Cm=Im−m−11m1m⊤C_m = I_m - m^{-1}\mathbf{1}_m\mathbf{1}_m^\top

(which is the centering absorbed into AA and cc; see Appendix A) because softmax is shift-invariant. For FaceNet verification, l0l_0 and lζl_\zeta are uncentered cosine similarity scores (Cm=ImC_m = I_m).

We will state these two operations separately. We will also make the order explicit: the analysis concerns dropout followed by normalization and a linear output map.

Q2. Is the quantity V\mathcal{V} small in practice?

Equation (15) gives V=ρ∥APD∥F2/∥APh∥22\mathcal V=\rho\left\lVert{APD}\right\rVert_{\mathrm F}^2/\left\lVert{APh}\right\rVert_2^2 with ρ=(1−q)/q\rho=(1-q)/q. We can compute this ratio before drawing any masks. It is small when the dropout perturbation that reaches the logits is small relative to the signal without dropout. A high keep probability reduces the ratio for fixed activations and output map. The ratio is also small when the output map suppresses the directions affected by dropout.

As V\sqrt{\mathcal V} gives the relative RMSE left by rescaling at fixed radius, its values of 2.159% to 8.508% at q=0.9q=0.9 predict a modest residual in the ten original language models, in line with the measured 2.164% to 8.562% (Table 11 and Figure 7c in the paper). We also reported median V=0.0341\mathcal{V} = 0.0341 on FaceNet (Appendix F.6 and Figure 11c in the paper). For the three added models, the corresponding values range from 8.968% to 23.008% (Table 1 here, common response).

The larger residuals for Qwen and Llama show that individual masks can still produce substantial changes after scaling. For mean prediction, part of this variation averages out across masks. The one-pass correction reduces L1 error to the MC-2048 mean probabilities in all thirteen language models (Table 12 in the paper and Table 1 here).

The independent-copy experiment on MNIST also confirms the predicted reduction Vk=V1/k\mathcal V_k=\mathcal V_1/k (see Q5).

Q3. What do the subscripts in Equation (23) mean?

Equation (23) gives conditions for the relative error to become arbitrarily small along a sequence of models. In Theorem 7, for example, we start from one model and increase the number of feature copies with independent masks, while preserving the logits without dropout. We ask whether the radius becomes more stable across masks and the remaining noise in the logits shrinks relative to the signal. The subscript nn means that we evaluate these quantities on the nn-th model in the sequence.

Vn=ρn∥AnPnDn∥F2∥AnPnhn∥22\mathcal{V}_n=\frac{\rho_n\left\lVert{A_nP_nD_n}\right\rVert_F^2}{\left\lVert{A_nP_nh_n}\right\rVert_2^2}

χt,n=Rt,nrˉn2.\chi_{t,n}=\frac{R_{t,n}}{\bar r_n^2}.

Here χt,n\chi_{t,n} denotes the relative squared-radius fluctuation bound, Rt,nR_{t,n} is the squared-radius fluctuation bound from Equation (16), and rˉn2\bar r_n^2 is the expected squared radius. The quantity rvis,nr_{\mathrm{vis},n} is the stable rank of AnPnDnA_nP_nD_n in Equation (15), and KnK_n bounds the standardized mask tails. For a chosen failure probability δ\delta, t=log⁡(6/δ)t=\log(6/\delta) stays fixed while nn varies. We will add these definitions immediately before the corollary.

Q4. What are the outputs in Theorem 7, and how does the theorem relate to Corollary 3?

Theorem 7 provides a concrete sequence of models satisfying the conditions of Corollary 3. We replace each feature viv_i by kk copies of vi/kv_i/\sqrt{k} and combine the copies before applying the original output map. This preserves the logits and the normalization radius r0r_0 in the absence of dropout.

Let LkL_k denote this copying map. We set hk=Lkhh_k=L_kh, uk=Lkuu_k=L_ku, Ak=ALk⊤A_k=AL_k^\top, and Dk=diag⁡(uk)D_k=\operatorname{diag}(u_k). With PkP_k the RMSNorm or LayerNorm projector in dimension kpkp, the two outputs are

ℓk,ζ=c+AkPk(hk+Dkζ)rk,ζ\ell_{k,\zeta}=c+\frac{A_kP_k(h_k+D_k\zeta)}{r_{k,\zeta}}

ℓTk=c+ℓ0−cTk.\ell_{T_k}=c+\frac{\ell_0-c}{T_k}.

The first is the logit vector for a sampled dropout mask, with random radius rk,ζr_{k,\zeta}. The second is the deterministic prediction obtained by scaling the feature-dependent part of the logits, with Tk=E[rk,ζ2]/r0T_k=\sqrt{{\mathbb E}[r_{k,\zeta}^2]}/r_0.

With a nonzero clean signal APhAPh and independent masks at fixed qq, combining the copies averages kk independent perturbations per feature. This gives Vk=V1/k\mathcal V_k=\mathcal V_1/k, while the radius fluctuation bound satisfies χt,k=O(k−1/2)\chi_{t,k}=O(k^{-1/2}) at fixed tt. The stable rank rvis,kr_{\mathrm{vis},k} stays fixed, so both conditions in Equation (23) hold for n=kn=k. Theorem 7 also gives ∥ℓk,ζ−ℓTk∥2=OP(k−1/2)\|\ell_{k,\zeta}-\ell_{T_k}\|_2=O_P(k^{-1/2}).

We will define both outputs before the theorem.

AuthorsOpenReview ↗

Response to Reviewer RtNt (cont'd)

Q5. Are the kk copies tested in Section 7?

Section 7 and Appendix F.3 test the construction of Theorem 7 on the nine widest MNIST models at q=0.65q=0.65. We use the same replicated features in both tests. One gives each copy an independent mask, and the other shares a mask across the copies of each feature. With eight independent copies, the relative RMSE to the analytic prediction falls to 35.12--35.32% of its one-copy value, close to the predicted 35.36% (1/81/\sqrt{8}). This is consistent with the theoretical scaling Vk=V1/k\mathcal V_k=\mathcal V_1/k. With shared masks, the error stays at its one-copy value.

Q6. Why is PP absent from V(Q)\mathcal{V}(Q) in Equation (37)?

For LayerNorm, PP subtracts the feature mean from every coordinate. The output matrix AA already includes this centering step, so AP=AAP=A. The allowed rotations fix the direction that PP removes (Q1=1Q\mathbf{1}=\mathbf{1}). They commute with PP, so the rotated output matrix AQ=AQ⊤A_Q=AQ^\top also satisfies AQP=AQA_QP=A_Q. Both AhAh and AqiAq_i in Equation (37) already account for centering. For RMSNorm, P=IP=I. We will move the assumption ``Assume P=IpP = I_p (or, for LayerNorm, Q1p=1pQ\mathbf{1}_p = \mathbf{1}_p with AP=AAP = A, which ensures AQP=AQA_Q P = A_Q)'' just before Equation (37) rather than waiting until Theorem 8.

Q7. What is the end-to-end effect on MNIST classification?

We tested whether the one-pass correction reproduces the class probabilities and predicted digit obtained with MC-20. Dropout perturbs the features before normalization, so the digit with the highest probability after averaging can differ from the digit predicted without dropout.

On each of 512 MNIST test images, we averaged the class probabilities from 20 passes with dropout at q=0.65q=0.65. We compared this average with the one-pass probabilities and checked whether both methods ranked the same digit first. Across the three architectures, the correction matched the MC-20 prediction on 99.61--100% of the test images.

Architectureℓ1\ell_1 distance to MC-20top-class agreement
Residual MLP0.00138100.00%
Mini-ViT0.0039699.61%
PreAct ResNet0.00169100.00%

Table 4. One-pass probability approximation on MNIST.

In a separate experiment, we tested whether the correction affects accuracy against the true labels. For each of the nine MNIST models, we used the same 512 test images and applied the correction for q=0.65q=0.65 to the logits from a pass without dropout. Before and after correction, we chose the digit with the highest logit and checked it against the true label. Top-1 accuracy was unchanged in all nine models. For the residual MLP and ResNet, this is guaranteed since c=0c=0 and positive scalar rescaling preserves the predicted digit. Mini-ViT also retained its mean accuracy of 96.61%, even though its class-dependent offset cc allows the correction to change which digit ranks first.

Note that, although top-1 accuracy is preserved, the correction can change the predicted probabilities and hence the NLL. For the residual MLP and Mini-ViT, NLL decreases from 0.07480 to 0.07007 and from 0.10245 to 0.10028, respectively, whereas for ResNet it increases slightly from 0.05241 to 0.05700.

Reviewer_RtNtOpenReview ↗

Q1 and Q6

I thank the authors for their explanations and extra numerical results. Most of my questions are addressed. Regarding your answers in Q1 and Q6, I still wonder why AP=AAP=A in the cases that you have considered in the paper; could you explain in more detail? From your definition in Appendix A, I do not understand what the "Effective output matrix" means. I am also wondering when you compute cosine scores, the linear layer after the normalization layer N_{P,epsilon} does not seem to make sense for computing cosine scores, as one also needs to normalize the norm of the output vector (to be unit norm) after the linear layer. Please correct me if anything is misunderstood here.

Reviewer_rqApOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigated how to approximate dropout effect using simple scalar rescaling at specific environment(normalization after dropout). The authors replace the random normalization radius with its expected value, which yields the analytic scalar 1/T. MNIST 27 models, FaceNet, and 10 pretrained language models are used to validate the theoretical results, showing that the scalar approximates the MC mean well, while individual dropout outputs still remain highly variable.

Critical strengths
  1. The proposed scalar can be calculated from just one forward pass and the dropout probability. It does not need retraining, labels, or sampled masks to approximate the MC mean, which is convenient.
  2. Assuming a fixed normalization radius, the paper nicely proves that 1/T is the unique optimal scalar for minimizing MSE.
  3. The authors tested their theory on various setups, including MNIST, FaceNet, and 10 pretrained language models (GPT-2, OPT, T5, BART, etc.).
Critical weaknesses
  1. The language models used in the experiments (GPT-2, OPT, T5, BART) are quite old. Since there are no experiments on recent LLMs like Llama or Mistral, it is hard to see how useful this is for modern NLP.
  2. I am not sure about the practical benefit of this method. For single dropout outputs in language models, the scalar only reduces the predicted MSE by 0.015% to 0.362% (Table 11), which is very small. Also, approximating the MC mean does not really help downstream performance—FaceNet accuracy stays almost the same (Appendix F.6) and MNIST NLL actually gets slightly worse (Appendix F.7). Only FaceNet has latency data, and other datasets don’t show latency data.
Questions for authors
  1. Since the single-pass MSE reduction is so small in the older language models, do you think the scalar approximation will actually work better or be more useful in modern, larger LLMs?
  2. People usually use MC dropout to get uncertainty estimates, but 1/T only gives the mean. If we use this one-pass method, how can we get uncertainty?
4 comments
AuthorsOpenReview ↗

Response to Reviewer rqAp

Q1 / W1. Does the correction help more in newer language models?

Following the reviewer's suggestion, we evaluated Qwen3-4B, Llama-3.1-8B and Mistral-7B-v0.3. In agreement with our dimension-scaling analysis in Figure 7 and Table 11, increasing the hidden dimension up to p=4096p = 4096 can help stabilize the normalization radius and reduce the normalized residual dispersion while the new experiments also show reductions in the L1L_1 probability error against the MC-2048 mean. As detailed in our common response (Table 1), at the same keep probability q=0.9q=0.9, the predicted MSE reduction is 2.335--3.554% in the three added models, compared with 0.015--0.362% in the original ten (paper, Table 11). Observed MSE reductions are 1.936--2.939%, and probability L1 error to MC-2048 falls by 35.1--61.8%. These improvements concern the approximation of dropout-averaged outputs, rather than downstream task performance.

Importantly, our contribution is not limited to reducing MSE: the theory also predicts the expected reduction from quantities available without Monte Carlo sampling. Thus, the small gains observed in some of the original models are consistent with their small predicted gains, rather than an unexpected failure of the correction.

W2. What is the practical benefit, including task performance and time?

Our primary aim is to characterize when dropout followed by normalization admits a scalar approximation, derive the corresponding correction, and predict its expected MSE benefit. The experiments evaluate these predictions and explore their practical implications.

At fixed radius, the correction gives the mean logits. In the three added language models, one pass reduces probability L1 error to the Monte Carlo mean over 2,048 masks by 35.1--61.8% relative to inference without dropout (Table 1 here).

Approximating dropout-averaged predictions and improving task performance are distinct objectives. First, when the fixed logit component cc vanishes, as in the three added models and both T5 models, the analytic correction is a positive scalar rescaling that preserves the top-token prediction. In the three added models, this follows from the absence of an offset in the final RMSNorm and of a bias in the output head. Second, while this rescaling changes the predicted probabilities, it is not designed to minimize next-token NLL.

Modelcc varies across vocabulary tokensTest accuracy without dropoutTest accuracy with correction
GPT-2Yes31.67%31.67%
GPT-2 MediumYes32.50%32.50%
GPT-2 LargeYes35.00%35.00%
GPT-2 XLYes36.67%35.83%
OPT-125MYes34.17%34.17%
OPT-350MYes37.50%37.50%
OPT-1.3BYes37.50%37.50%
T5 SmallNo30.83%30.83%
T5 BaseNo44.17%44.17%
BART BaseYes20.83%20.83%
Qwen3-4BNo37.50%37.50%
Llama-3.1-8BNo46.67%46.67%
Mistral-7B-v0.3No50.83%50.83%

Table 5. Next-token top-1 accuracy on 120 WikiText-2 test positions (q=0.9q=0.9).

For the eight models with a token-dependent cc, accuracy remains unchanged in seven cases; only GPT-2 XL shows a decrease of one correct prediction out of 120.

On 120 test positions per model, the analytic correction gives NLL values of 3.013, 2.468 and 2.261 for Qwen, Llama and Mistral, respectively, compared with 3.026, 2.466 and 2.258 without dropout, and 3.027, 2.470 and 2.265 with local MC-32. Thus, NLL is slightly better than MC-32 in all three cases, but improves over deterministic inference only for Qwen.

In a separate timing test on ten passages per model, its runtime is within 1% of the time of a pass without dropout. On MNIST, the correction is 18.49--21.89 times as fast as MC-20, which uses twenty full passes.

When earlier activations are reused, local MC-32 requires only 0.980.98--1.58×1.58\times the runtime of a single pass, so the speed advantage of the analytic correction is limited in this setting. When MC samples require full forward passes, the cost increases to 8.338.33--22.86×22.86\times that of a single pass (cf. Table 6 here).

AuthorsOpenReview ↗

Response to Reviewer rqAp (cont'd)

For Tables 6 and 7 here, timings were measured on an RTX 4070 using three validation passages (12 positions each) and batches of four masks. We average the passage-wise medians of three runs. Timings include weight transfers for Llama and Mistral, and ratios are computed before rounding.

ModelOne pass (ms) reuse testMC-32 (ms) with reuseMC-32 / one pass with reuseOne pass (ms) full-pass testMC-32 (ms) with full passesMC-32 / one pass with full passes
GPT-28.411.81.40×6.456.98.86×
GPT-2 Medium12.717.91.41×13.1125.49.56×
GPT-2 Large15.218.21.20×16.0247.915.49×
GPT-2 XL20.325.71.26×24.2468.619.38×
OPT-125M6.28.11.31×6.352.48.33×
OPT-350M10.512.01.15×9.8117.511.94×
OPT-1.3B15.122.81.50×15.1335.722.19×
T5 Small10.211.91.18×8.473.78.73×
T5 Base19.423.21.20×16.2146.69.02×
BART Base7.912.41.58×7.667.38.80×
Qwen3-4B42.057.51.37×41.9957.622.86×
Llama-3.1-8B820.1807.20.98×775.37,021.69.06×
Mistral-7B-v0.3677.9681.61.01×731.56,620.89.05×

Table 6. Inference times with dropout in the last feed-forward block.

ModelOne pass (ms)MC-32 over all blocks (ms)MC-32 / one pass
GPT-25.555.610.12×
GPT-2 Medium11.9130.010.96×
GPT-2 Large19.1257.413.46×
GPT-2 XL22.1481.121.79×
OPT-125M5.147.89.30×
OPT-350M8.5123.614.50×
OPT-1.3B15.3344.722.57×
T5 Small9.581.08.52×
T5 Base17.9172.09.59×
BART Base8.975.98.56×
Qwen3-4B42.8984.423.01×
Llama-3.1-8B870.87,449.28.55×
Mistral-7B-v0.3738.26,154.18.34×

Table 7. Inference times with dropout in every feed-forward block.

Q2. How can the one-pass method provide uncertainty?

While we agree that this is not a complete measure of predictive uncertainty, our one-pass analysis does provide a measure of dropout-induced variability in logit space. At fixed radius, Equation (25) also gives the total logit variance:

$ {\mathbb E} \left\lVert{\widetilde\ell_\zeta-\ell_T}\right\rVert_2^2 =\frac{\rho\left\lVert{APD}\right\rVert_F^2}{\bar r^2}. $

This quantity is the trace of the logit covariance matrix and can be computed analytically, without sampling dropout masks.

We tested Equation (25) with local dropout on the ten original language models and the three added models.

Modelpredicted mean varianceobserved mean variancedifference (%)Pearson rrr2r^2prediction R2R^2
GPT-25,321.85,477.4+2.920.998210.996410.99552
GPT-2 Medium4,284.34,283.3-0.020.999070.998130.99808
GPT-2 Large605.8605.3-0.080.999830.999650.99959
GPT-2 XL478.5482.0+0.730.999900.999800.99964
OPT-125M1,054.71,052.4-0.220.999870.999740.99972
OPT-350M850.0850.3+0.030.999690.999380.99937
OPT-1.3B612.9612.7-0.030.999960.999910.99991
T5 Small1,921.51,912.2-0.480.996310.992640.99242
T5 Base1,079.51,079.9+0.040.997420.994850.99464
BART Base2,821.82,858.8+1.310.999290.998590.99772
Qwen3-4B85,101.089,898.5+5.640.986960.974090.92814
Llama-3.1-8B32,281.832,260.4-0.070.989210.978540.97839
Mistral-7B-v0.32,196.42,346.1+6.810.950630.903700.86839

Table 8. Local logit variance (q=0.9q=0.9).

Eleven of the thirteen models give prediction R2R^2 of 0.9784--0.9999, with measured mean variance within 3% of the prediction. Even for Qwen and Mistral, which show the largest discrepancies, prediction R2R^2 remains high (0.928 and 0.868, respectively).

Thus, our framework provides two complementary analytical quantities: a scalar correction approximating the mean dropout output and a measure of the residual logit dispersion. Both can be computed without Monte Carlo sampling, and the additional results in Table 8 empirically validate the accuracy of the latter.

Dear Reviewer rqAp,

Thank you again for your careful review and constructive comments.

We have now uploaded a revised version of the paper, incorporating the additional experiments and clarifications provided in our rebuttal, particularly regarding modern LLMs, practical performance, and uncertainty estimation.

We would be grateful to hear whether these new results and explanations have addressed your main concerns, or whether any important questions remain open. We would be happy to discuss them further.

Best regards, The authors

Thank you for your detailed response and for the considerable effort you put into conducting additional experiments. I appreciate the authors' efforts, and I found the new experimental results impressive.

While the main experimental results are encouraging, I believe some concerns remain unresolved regarding the additional evaluation metrics, particularly their implications for practical applicability.

I also appreciate the authors' attempt to address these concerns through mathematical analysis and theoretical justification.

Overall, although the additional experiments strengthen the paper, I do not think they fully resolve my original concerns. Therefore, I have decided to maintain my original rating of 2 (Weak Rejection).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Authors' common response to reviewers

To address the reviewers' concerns about the applicability of our local analysis to modern, larger language models, we extended our experiments to Qwen3-4B, Llama-3.1-8B and Mistral-7B-v0.3, with representation dimensions of 2,560--4,096 (versus 512--2,048 in the original ten models).

As in the original experiments, dropout is injected into the last feed-forward residual branch before final RMSNorm, with keep probability q=0.9q=0.9, while earlier activations remain fixed.

In the three new models, the one-pass correction reduces probability L1 error relative to the MC-2048 reference by 35.1--61.8% compared with inference without dropout.

Specifically, Table 1 here extends the diagnostics of Table 11 (Appendix F.2) to the three newer models and adds the relative reduction in probability L1 error to the MC-2048 mean, following Table 12. We use MC-2048 as a numerical reference because the exact mean of the predicted probabilities is not available in closed form. This metric highlights the benefit for approximating mean prediction, even when individual dropout outputs remain highly variable.

Modelrepresentation dimensionradius CVV\sqrt{\mathcal V}fitted RMSEpredicted MSE reductionobserved MSE reductionL1 error reduction
Qwen3-4B25605.00023.00823.7183.5542.47135.1
Llama-3.1-8B40961.90420.53720.3612.9312.93942.9
Mistral-7B-v0.340962.3858.9689.5212.3351.93661.8

Table 1. Local dropout in the three added language models (q=0.9q=0.9).

The predicted and observed MSE reductions remain reasonably close in these larger models. The gains remain similar with either set of 1,024 masks. See our response to RrAN, Q2, for the comparison of the two independent MC references.

To investigate the extension beyond the local setting, we also applied dropout at every feed-forward block output. Accounting for these propagated perturbations improves the prediction of the correction factor in 12 of the 13 tested language models compared with using only the last block.

Propagation reduces the error in the predicted coefficient in nine of the ten GPT-2, OPT, T5 and BART models from the paper, by 21.8--83.4% relative to the local correction. It also improves all three added models. These tests use 32 propagated perturbations and separate masks for fitting and evaluation. The results for all thirteen models are reported in Table 2 here, with the detailed protocol described in our response to RrAN, Q1. The trained eight-block MNIST experiment provides another test across layers (Appendix D.2).

We will incorporate both sets of additional experiments into the revised paper and clarify the scope and limitations of the proposed approximation.

Modelmasked blocksrepresentation dimensionerror in aa (last block)error in aa (all blocks)error reduction for aa (%)logit MSE reduction (%)
GPT-2127680.02780.021721.771.30
GPT-2 Medium2410240.02240.016824.921.39
GPT-2 Large3612800.01920.005969.071.91
GPT-2 XL4816000.01880.003780.421.93
OPT-125M127680.02110.006171.191.71
OPT-350M2410240.02620.004383.432.40
OPT-1.3B2420480.02500.010159.521.82
T5 Small6+65120.01650.0191-16.00-1.69
T5 Base12+127680.02080.011843.361.71
BART Base6+67680.02750.016739.361.50
Qwen3-4B3625600.06990.029957.187.24
Llama-3.1-8B3240960.06100.035441.944.32
Mistral-7B-v0.33240960.08360.049940.347.09

Table 2. Dropout at every feed-forward output in thirteen language models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.