acceptodds
Under review as a conference paper at ICLR 2027

Nonlinear Error Cancellation at Fixed ReLU Rounding Endpoints

Abstract

Scalar unbiasedness does not determine the mean rounded function of a nonlinear network. We prove a sharp separation between probability-law constraints for fixed signed Gaussian ReLU teachers, with unchanged width and original adjacent dyadic endpoints. For sufficiently small dyadic and widths with a power of two, the optimum over all laws preserving scalar stochastic-rounding marginals has order . Preserving each neuron's complete product-rounding law instead gives optimal order whenever is bounded below and is sufficiently small. The separation survives realization-specific affine correction in this width regime. A legal within-neuron correlation cancels the leading boundary moment. Ordered prefix coupling then preserves the modified complete local laws, while pointwise variation of the actual endpoint contrasts is , yielding variance . An arithmetic projection obstruction and a nonlinear quartic floor match the joint upper order for ordinary loss and best constant correction. Thus reciprocal-step width is necessary and sufficient for quartic loss on this arithmetic family; the necessity does not extend to general affine correction or convex means. For each fixed known density pair and local laws prescribed before sampling, almost surely the separation holds for every , all sufficiently fine dyadic , and all even ; this width condition is sufficient only. All teacher scalars are off-grid, directions are distinct, and nonaffine signal remains nonzero.

Primary area:
learning theory
Keywords:
stochastic rounding, ReLU networks, quantization, correlated rounding, approximation theory

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_1bpeOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I was not able to form a sufficiently clear understanding of the paper to summarize it, as several central objects are not defined in a clear and self-contained manner.

Critical strengths

None

Critical weaknesses
  1. Insufficient motivations. I didn't find any motivation for the problem; the abstract and the introduction read as a succession of technical statements where the key objects are not defined yet, and why the pursued technical goals matter is not discussed.

  2. Lack of mathematical rigor. In Section 2, several objects are introduced without any formal definition, such as f^C\hat f_C, L2(γ3)L^2(\gamma_3), "aligned scalar", "scalar SR", "product SR-law", "cancellation row", etc. Moreover, the distinction between the classes S,Π(P)S, \Pi(P) and Π(Q)\Pi(Q) that seem to be central to the theorem is not discussed. Given these critical issues in the definitions of key objects, I'm not able to assess the technical contribution of the paper.

  3. It is unclear to me what significant application or general conclusion should be drawn from the main theorem.

Questions for authors

The AI-use statement indicates that LLMs assisted with drafting and revision. Given the substantial clarity and coherence issues in the current presentation, could the authors clarify what human review and editing were performed on the final manuscript?

Reviewer_RGKNOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors present an analysis of the asymptotic functional error rate of finite-width shallow ReLU neural networks under stochastic rounding laws of the network parameters. The asymptotics are analyzed in terms of the rounding grid spacing hh and the width mm for a fixed input covariate distribution N(0,I3)\mathcal{N}(0, I_3) and in terms of the L2L^2 error with respect to this covariate under different choices of the rounding law correlation. Three "law domains" are investigated: (1) Π(P)\Pi(P) (product/stochastically independent rounding laws) (2) Π(Q)\Pi(Q) (neuron-internal correlation allowed) and (3) S\mathcal{S} (arbitrary correlations allowed, also "scalar" as subscript). The main result (theorem 1) establishes the exact joint asymptotic behavior Fm,hF_{m, h} in terms of hh and mm for all three regimes as well as for sufficiently large mm, rates h4h^4 for (2) and (3) as well as h4⋅log⁡(1/h)h^4 \cdot \log(1/h). The results seem to be based on a bias-variance decomposition and geometrically motivated error analysis. Numerical results for explicit parameters are presented in the end.

Critical strengths

The presented problem and its mathematical analysis are interesting and the setting seems to be on a sufficiently simplified level that the stochastic rounding problem in the presented setting becomes tractable. Therefore, knowing the asymptotic error law would be an interesting contribution.

Critical weaknesses

My main problem with the paper is its accessibility to a generic reader from the community. The paper is highly dense, hard to parse and has no reader-oriented structure. Ideas that are defined later and only make sense in context (e.g. ll. 38 "Two interlaced direction groups have opposite output signs and different hidden scales, ...") are stated as if the reader had already read the paper once. In addition, the paper frequently references the 44 page appendix for proofs and the 89 page supplementary material and gives no room to clarifying remarks or intuition despite being only marginally longer than 8 pages in total. Also, the numerical section includes what is typically agentic feedback (ll. 388 "All 189 tests pass. The 23 prefix tests comprise ...") and contributes little towards understanding the results.

Questions for authors

Can the authors give an alternative structure of the abstract and the introduction that would be easier to follow and conveys the central takeaways of their paper?

Reviewer_92SnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper considers the rounding of neural-network parameters changes and the impact of activation boundaries. Several bound are presented and proofs presented.

Critical strengths

The paper appears to be very sound.

Critical weaknesses

I found the writing far too compact in places. It is very difficult to follow in places and the impact of the paper difficult to quantify.

Questions for authors

How dow the results extend beyond a two layer bias free network?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.