Nonlinear Error Cancellation at Fixed ReLU Rounding Endpoints

Scalar unbiasedness does not determine the mean rounded function of a nonlinear network. We prove a sharp separation between probability-law constraints for fixed signed Gaussian ReLU teachers, with unchanged width and original adjacent dyadic endpoints. For sufficiently small dyadic $h$ and widths $m=2n$ with $n$ a power of two, the optimum over all laws preserving scalar stochastic-rounding marginals has order $\Theta (h^4+h^2/m^2)$. Preserving each neuron's complete product-rounding law instead gives optimal order $\Theta (h^4\log (1/h))$ whenever $mh$ is bounded below and $h$ is sufficiently small. The separation survives realization-specific affine correction in this width regime. A legal within-neuron correlation cancels the leading boundary moment. Ordered prefix coupling then preserves the modified complete local laws, while pointwise variation of the actual endpoint contrasts is $O(h)$, yielding variance $O(h^2/m^2)$. An arithmetic projection obstruction and a nonlinear quartic floor match the joint upper order for ordinary loss and best constant correction. Thus reciprocal-step width is necessary and sufficient for quartic loss on this arithmetic family; the necessity does not extend to general affine correction or convex means. For each fixed known density pair and local laws prescribed before sampling, almost surely the separation holds for every $c_0>0$, all sufficiently fine dyadic $h$, and all even $m\ge c_0/h$; this width condition is sufficient only. All teacher scalars are off-grid, directions are distinct, and nonaffine signal remains nonzero.