acceptodds
Under review as a conference paper at ICLR 2027

Brownian Heads for Deep ReLU Representations: Activation Mass and the Cost of Same-Sample Selection

Abstract

Deep representation learning often selects hidden features and fits the final predictor on the same sample, so fixed-feature analysis performed after selection can omit selection cost. We study the conditional empirical Rademacher complexity of deep ReLU representations followed by bounded-norm predictors in additive or L\'evy–Brownian RKHSs, termed Brownian heads. For a fixed representation, we derive an exact dual identity and sharp bounds in terms of activation mass, the average norm of the observed hidden vectors. Under same-sample selection, the representation supremum induces a quadratic Rademacher process. Brownian layer-cake and Gaussian-projection identities reduce it to coordinatewise or signed projected threshold traces, separating realized scale from selection complexity. For samples with pairwise-distinct inputs, explicit scalar ReLU families match the finite-trace and VC rates up to universal constants at the realized trace-and-envelope level. Induced-norm contraction also yields architecture-level bounds for rectangular, rank-deficient ReLU networks. Experiments verify the sharp bounds and rates, exhibit a selection gap at fixed activation mass, and assess the predictive feasibility of Brownian heads

Primary area:
learning theory
Keywords:
Deep representation learning, Same-sample representation selection, Brownian kernels

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pJNxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This works analyzes the empirical Rademacher complexities of predictor with a representation maps. They study the effects of learning both representation, and predictors using the same dataset. Their results cover family of representations, and apply to practical deep ReLU networks. Finally, they provide few experiments that corroborate with their theory.

Critical strengths

S1) Studying the effects of same sample representation selection is very relevant problem in deep learning, this work provide rigorous theory on empirical Rademacher complexity that is very essential for generalization theory.

S2) The paper is well organized, and I had no problem following through it.

Critical weaknesses

W1) The paper lacks population Rademacher bounds that are needed to provide final generalization bounds for considered learning problem.

Questions for authors

Q1) Can the authors provide insights whether computing population Rademacher bounds is feasible at-least under certain data distributions?

Q2) Could the authors clarify whether selection cost about picking features on the same data is a generally observed phenomenon or is it specific to Brownian heads? Please support your response with few references.

3 comments
AuthorsOpenReview ↗

Response to pJNx (1/2): population guarantees and their scope

Thank you for recognizing the relevance of same-sample representation selection and the organization of the paper. Your population question is important. The revision now separates the available fixed-class consequence from the additional dependence control needed for an arbitrary sample-generated family.

For a deterministic or independently learned representation zz, fix B≥0B\geq0 before the sample and let S∼PnS\sim P^n. With a measurable loss in [0,1][0,1] that is LℓL_\ell-Lipschitz in prediction, the fixed-class empirical-Rademacher deviation inequality gives, with probability at least 1−δ1-\delta, simultaneously for all f∈Fz,B(q)f\in\mathcal F_{z,B}^{(q)}, $ R(f)\leq\widehat R_S(f) +2L_\ell B\sqrt{\frac{\widehat M_{q,S}(z)}n} +3\sqrt{\frac{\log(2/\delta)}{2n}}. $ Here n≥1n\geq1, 0<δ<10 < \delta < 1, and RR and R^S\widehat R_S are population and empirical loss. The scoped population statement requires measurable predictors and a countable pointwise-dense loss subclass; an independently random representation also requires the conditional and joint measurability conditions. These are not new assumptions silently imposed on the arbitrary-family empirical theorem.

The fixed-representation result is submitted Corollary B.22, now revised Corollary B.25. It applies the standard fixed-class deviation theorem and scalar loss contraction (Mohri, Rostamizadeh and Talwalkar, 2018, Theorem 3.3 and Lemma 5.7). A separate expected/population-complexity calculation is not required for this high-probability step.

For a prespecified representation family, the same argument applies to its whole union loss class, with the corresponding measurability premises. Revised Corollary B.26 gives $ R(f)\leq\widehat R_S(f) +2L_\ell\widehat{\mathcal R}_S(\mathcal F_{\mathcal Z,B}^{(q)}) +3\sqrt{\frac{\log(2/\delta)}{2n}}, \qquad f\in\mathcal F_{\mathcal Z,B}^{(q)}, $ uniformly, so the empirical union bounds can be substituted when their displayed quantities are measurable and finite. This permits representation and head to be fitted on the same sample within that prespecified family; the selected representation need not be independent of the sample.

For an arbitrary family Z(S)\mathcal Z(S) generated from the sample, its realized conditional empirical complexity alone does not justify that population step. A containing prespecified class or an appropriate random-class dependence argument is needed. Remark B.27 gives a sample-generated singleton with zero training-state complexity but positive population loss, illustrating why conditioning on the realized family cannot discard that dependence. We retain this limitation explicitly; the arbitrary-family empirical results are not replaced by a fixed-class theorem. See pages 42--44 of the red revision.

AuthorsOpenReview ↗

Response to pJNx (2/2): distribution-level bounds and non-Brownian selection

Your Q1: population Rademacher complexity under distributions. For a measurable deterministic representation, define Mq=E∥z(X)∥qM_q=\mathbb E\\\|z(X)\\\|_q. If Mq<∞M_q < \infty and BB is fixed independently of the i.i.d. sample, the fixed-head bound and Jensen's inequality give $ \mathbb E_S\widehat{\mathcal R}_S(\mathcal F_{z,B}^{(q)}) \leq\frac B{\sqrt n}\mathbb E_S\sqrt{\widehat M_{q,S}(z)} \leq B\sqrt{\frac{M_q}{n}}. $ Thus no Gaussian-input assumption is needed; the stated finite activation moment suffices. For Mq=0M_q=0, the class vanishes almost surely on the input distribution and the complexity is zero. For a representation learned independently of the target sample, the same calculation holds conditionally when its conditional activation moment is finite. This is a separately scoped consequence. Averaging a bound for an arbitrary sample-generated family does not by itself supply the population-risk guarantee discussed in Part 1.

Your Q2: is selection cost specific to Brownian heads? No. It is a general union-class phenomenon: $ \mathbb E_\varepsilon\sup_z g_z(\varepsilon) \geq\sup_z\mathbb E_\varepsilon g_z(\varepsilon), $ where gzg_z is the Rademacher support function of the prediction class for representation zz. Strict inequality is possible without Brownian heads. On a two-point sample, consider bounded-norm linear-head evaluation classes $ \mathcal A=\{a(1,1):|a|\leq1\},\qquad \mathcal B=\{a(1,-1):|a|\leq1\}. $ Each has empirical Rademacher complexity 1/21/2, because E∥ε1±ε2∥/2=1/2\mathbb E\|\varepsilon_1\pm\varepsilon_2\|/2=1/2. Their union has complexity 1: the maximum of the two absolute sign sums is 2 for every sign pair. Its selection excess is therefore 1/21/2. This is an explicit non-Brownian example, not a replacement for the paper's model or a claim that every family has positive excess.

What is Brownian-specific is the quantitative link to activation mass, the feature-distance identity, and the layer-cake/Gaussian-projection structure enabling threshold bounds. The new centered-Gram and Gram-root results distinguish variation across representations from richness inside one representation. The finite-pool comparator is generic; the observed positive gaps do not establish a phenomenon exclusive to Brownian geometry or a systematic extra penalty from label dependence.

I thank the authors for clarifying all my questions. I maintain my positive recommendation for this submission.

Reviewer_CgQFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the empirical Rademacher complexity of bounded-norm Brownian RKHS heads on deep ReLU features. It gives sharp fixed-representation bounds in terms of activation mass, then uses threshold traces to bound the union over representations available for same-sample selection. Exact constructions, adapters, and frozen-feature experiments illustrate the analysis.

Critical strengths

The threshold-trace reduction is a useful way to analyze the quadratic Rademacher process created by choosing a representation. The fixed-head identity and its sharp constants are clearly stated. 2. The equal-mass adapter experiment shows that activation mass alone does not determine union complexity: the largest family has about 1.40 times the complexity of its strongest fixed member (Figure 2).

Critical weaknesses
  1. The main bound does not cleanly measure selection cost. For a single q=2 representation with z(x_i)=e_i, the projected traces in Eq. 12 contain every subset, so their complexity is about 1/2. Eq. 16 then gives no decay with n, although Theorem 2.2 gives an O(B/sqrt(n)) fixed-head rate. The trace term can therefore be large when no representation is selected.
  2. Corollary 3.4 is not shown to be informative for a deep network. Once its VC estimate exceeds n, the bound reverts to an n-independent norm envelope; layer-norm products can further enlarge it. No numerical value is given for any architecture in Section 4.
  3. Figure 2 reports the exact union complexity but never compares it with the proposed bounds in Eqs. 15-16. A matched label-independent adapter family is also missing. The observed union gap is real, but the experiment does not show whether the trace bound explains it or whether sample-dependent construction adds to ordinary finite-family selection.
Questions for authors
  1. Do the refinements in Propositions B.11-B.12 recover the fixed-head rate for a singleton family? Please show this on the z(x_i)=e_i example and make that distinction visible in the main result.
  2. For the Figure 2 families, what are the measured trace complexity, envelope, and numerical right-hand sides of Eqs. 15-16, compared with the exact union complexity? Is Corollary 3.4 nonvacuous for any reported architecture?
  3. How does the union gap change for a label-independent adapter pool matched in family size and activation mass? This would clarify what the label-dependent construction contributes.
6 comments
AuthorsOpenReview ↗

Response to CgQF (1/5): the singleton objection

Thank you for the singleton example and for separating an observed union gap from the tightness of its upper bounds. You are right: the submitted Propositions B.11--B.12 do not recover the fixed-head rate in that example. The original inequality is an upper bound on total union complexity, not an exact measure of the additional cost of selection.

Write RUR_U for the union complexity, R\*=sup⁡zRzR_\*=\sup_z R_z for the strongest fixed-head complexity, and Δ=RU−R\*≥0\Delta=R_U-R_\*\geq0, when these quantities are finite. A singleton has Δ=0\Delta=0 even if its projected trace family is rich.

For q=2q=2 and z(xi)=eiz(x_i)=e_i, every subset of [n][n] is a projected trace. With Sn=∑iεiS_n=\sum_i\varepsilon_i, $ \rho_{2,S}=\frac{n+\mathbb E|S_n|}{2n}\geq\frac12, \qquad \widehat M_{2,S}=1. $ The sample Gram matrix and the actual fixed-head complexity are $ K_z=2^{-1/2}I+(1-2^{-1/2})\mathbf1\mathbf1^\top, $ $ R_z=\frac Bn\mathbb E\sqrt{n/\sqrt2+(1-2^{-1/2})S_n^2}. $ Consequently, $ 2^{-1/4}\frac B{\sqrt n}\leq R_z\leq\frac B{\sqrt n}. $ The two root-mass caps in the submitted B.11 equal BB, and the width refinement in B.12 does not restore the missing decay. At n=500n=500 and B=1B=1, the finite binomial expectation gives Rz≃0.04401925R_z\simeq0.04401925, while B/n≃0.04472136B/\sqrt n\simeq0.04472136 and ρ≃0.51783232\rho\simeq0.51783232. This is an exact-example calculation, not an adapter benchmark.

What has changed. The new centered bound uses variation between off-diagonal Grams, which is zero here, and therefore returns B/nB/\sqrt n. The separate Gram-root bound returns exactly zero selection excess for this singleton. Both retain the original arbitrary, possibly uncountable family and allow singular and zero Grams; neither is a finite-pool substitute.

These additions are now visible in the main text as revised Theorems 3.3--3.4, with proofs in Sections A.9.1--A.9.2 and this example in A.9.3. The submitted matching theorem is now Theorem 3.5: its sharpness remains worst-case at the original trace/envelope level, not familywise sharpness of the new geometric terms. Part 2 states the two refinements; Parts 3--5 address the requested numerical comparison, control, and architecture-bound limitation.

AuthorsOpenReview ↗

Response to CgQF (2/5): two general-family refinements

Centered union bound. Condition on the sample and the original representation family before drawing auxiliary signs. Put M∗=sup⁡zTr⁡(Kz)/nM^*=\sup_z\operatorname{Tr}(K_z)/n. Let Hz=Off⁡(Kz)H_z=\operatorname{Off}(K_z), and use the original trace atoms HT=Off⁡(1T1T⊤)H_T=\operatorname{Off}(\mathbf1_T\mathbf1_T^\top) for ∥T∥≥2\|T\|\geq2. On their span VV, define $ \|H\|_{\mathcal A}=\inf_{H=\sum_T b_TH_T}\sum_T|b_T|, \qquad r_{\mathcal A}=\inf_{H_0\in V}\sup_z\|H_z-H_0\|_{\mathcal A}. $ Set rA=0r_{\mathcal A}=0 if V=0V=\\{0\\}. Revised Theorem 3.3 gives $ R_U\leq B\sqrt{M^*/n+2C_{\rm ch}r_{\mathcal A}\rho_{q,S}^2}. $ Section A.9.1 justifies Cch=2C_{\rm ch}=2. The radius is zero exactly when all off-diagonal sample Grams coincide. For a singleton, this recovers the fixed-head upper rate, rather than treating projection richness inside that representation as selection.

The proof centers the zero-diagonal quadratic process: each fixed H0H_0 has zero Rademacher expectation. The original threshold expansion then bounds the centered process in the same atomic norm, followed by Jensen's inequality. The auxiliary center is not an added feasible representation and does not change the prediction class. The result improves the original chaos branch with matching constants, not necessarily the separate direct-threshold branch.

A bound on selection excess itself. Let V\mathcal V consist of the canonical PSD square roots of matrices in the closure of the original sample-Gram image. With D=diam⁡(V,∥⋅∥F)D=\operatorname{diam}(\mathcal V,\\\|\cdot\\\|_F) and internal covering numbers N(u)N(u), revised Theorem 3.4 gives, for M∗<∞M^* < \infty, $ 0\leq\Delta\leq\frac{40B}{n}\int_0^D\sqrt{\log N(u)}\,du. $ Finite activation mass makes the root family compact in a finite-dimensional matrix space and the integral finite. The proof controls increments of the centered root-norm process and chains over internal covers. Continuity preserves both original suprema on taking the Gram closure; no attained original representation is assumed or added. Singular and zero matrices need no inverse. Identical full Grams give D=0D=0 and exactly zero excess, including every singleton.

Both results retain the original common finite output dimension and the stated nonnegative sample states for the q=1q=1 threshold identity. The zero-radius, zero-head and zero-Gram cases remain included. These geometric terms are upper bounds, not identities for every family's cost; no new sharpness or efficient exact radius/entropy computation is claimed.

The main statements are on page 7 of the red revision and the proofs and examples on pages 25--29. In particular, Example A.7 supplies a continuum of feasible ReLU representations with positive excess, so the strengthening is not limited to the singleton example.

AuthorsOpenReview ↗

Response to CgQF (3/5): numerical bounds versus measured complexity

Your Q2 and the numerical part of W3. Section C.8 now evaluates the requested quantities on CIFAR-10, CIFAR-100, and EuroSAT, with all five seeds, n=500n=500, width-12 rank-one ReLU adapters, and K∈1,2,4,8,16,32K\in\\{1,2,4,8,16,32\\}. Each candidate has activation mass one and B=1B=1. The new analysis uses 4096 shared sign draws; its first 128 draws retain the historical sign protocol. Kernel quadratic forms are evaluated directly in floating point, but sign expectations are Monte Carlo estimates, not exact expectations.

These adapters use additive geometry (q=1q=1). Thus we evaluate Eq. (15), not the distinct projected q=2q=2 expression in Eq. (16). Write A∗=max⁡c∑jmax⁡izc(xi)jA^*=\max_c\sum_j\max_i z_c(x_i)_j, M∗=1M^*=1, and ρ=ρ1,S\rho=\rho_{1,S}. With Cch=2C_{\rm ch}=2, the two branches are $ U_{15}^{\rm chaos}=\sqrt{1/n+4A^*\rho^2},\qquad U_{15}^{\rm direct}=\sqrt{A^*}\rho. $ The table gives five-seed means of per-context evaluations at K=32K=32 for the original label-dependent construction. U15U_{15} denotes the minimum of the two branches. ρ^\widehat\rho is a plug-in estimate; ρU\rho_U is its simultaneous upper confidence endpoint.

DatasetR^U\widehat R_Uρ^\widehat\rhoA∗A^*U15(ρ^)U_{15}(\widehat\rho)U15(ρU)U_{15}(\rho_U)
CIFAR-100.0605740.092157.2090.24700.2560
CIFAR-1000.0622210.093428.1930.26720.2768
EuroSAT0.0607890.089886.3570.22650.2349

The confidence-qualified centered union bounds are respectively 0.4917, 0.5302, and 0.4591. The evaluated Gram-root excess bounds are 2.8426, 2.9996, and 2.7430. These use feasible upper approximations of the radius and entropy integral, not optimized exact values. Section C.8, Tables 4--6 (pages 53--54), reports both arms and the uncertainty calculations.

The generic finite-pool comparator (revised Proposition B.13) is stronger here: RU≤(1+2log⁡32)/500≃0.16246236R_U\leq(1+\sqrt{2\log32})/\sqrt{500}\simeq0.16246236 and Δ≤2log⁡32/500≃0.11774100\Delta\leq\sqrt{2\log32/500}\simeq0.11774100, while R\*≤1/500≃0.04472136R_\*\leq1/\sqrt{500}\simeq0.04472136 and the root-mass union cap is 1. The new centered bound improves the chaos branch at the same ρU\rho_U, but the existing direct branch is smaller on these pools. The raw entropy evaluation is quantitatively uninformative here. We therefore do not claim numerical tightness or an empirical improvement over the strongest available bound.

Threshold calculations keep complete equal-value positive blocks, so ties cannot be separated. The confidence budget covers 2640 events across both arms and both draw counts. Feasible-center and internal-cover computations are float64 evaluations, not outward-rounded arithmetic certificates. The 32-candidate experiment is not an evaluation of the full architecture family; that separate limitation is addressed in Part 5.

Reviewer_4MFUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies deep ReLU representations followed by a Brownian kernel prediction head, focusing on the extra complexity introduced when representations are selected using the training data. For a fixed representation, it gives sharp empirical Rademacher complexity bounds in terms of average activation magnitude and the head’s norm. When representations can be selected, activation magnitude alone is insufficient; the authors bound the union’s complexity using threshold patterns induced on the sample and provide matching worst-case constructions.

Critical strengths
  • A sharp and interpretable characterization of fixed-representation complexity
  • Rigorous demonstration that representation selection can introduce additional empirical complexity that fixed-representation analyses fail to capture
  • Applicability to ReLU networks with relatively flexible architectural assumptions
Critical weaknesses
  • The paper is too technical and lacks sufficient interpretation
  • The examined model, RKHSs with Brownian head, is a continuous analog of standard neural networks. It is not clear how well this model captures the properties of standard neural networks.
Questions for authors
  1. How does your representation-selection analysis relate to the Brownian kernel and threshold-set analysis in [R1]?
  2. Can we generalize this analysis to transformer-based models? If not, why?

References: [R1] - Variation Brownian Kernel Ladders

4 comments
AuthorsOpenReview ↗

Response to 4MFU (1/2): interpretation and relation to VBKL

Thank you for recognizing the fixed-representation characterization and the relevance of accounting for representation selection. We agree that the model and the distinction from earlier Brownian work should be easier to identify before the technical development.

What model is being analyzed? The encoder is an ordinary finite deep ReLU network. The final predictor is a bounded-norm Brownian RKHS head, rather than an affine or softmax head. We do not claim that these heads are equivalent. The kernel identity $ K_q(z(x_i),z(x_i))=\|z(x_i)\|_q $ makes the fixed-head complexity depend on empirical activation mass, while different off-diagonal Gram entries can change the complexity of a representation union even when activation mass is identical. This is the question the revised introduction now explains explicitly. The frozen-feature prediction results assess feasibility, not universal superiority or the population cost of selecting an encoder.

Relation to Variation Brownian Kernel Ladders (your Q1). Brownian kernel identities and threshold decompositions are shared ingredients; we do not present those identities or the existence of selection effects as new. VBKL already analyzes unions of scalar Brownian pullback balls through a supremum of kernel quadratic forms, a diagonal/off-diagonal split, and signed-threshold control. Its model composes normalized scalar profiles into paths and then takes signed-measure mixtures of completed paths. We credit this shared machinery rather than presenting the reduction itself as new. Here the terminal class is a bounded-norm additive or Lévy–Brownian head on vector-valued deep ReLU representations, and the object is its conditional empirical complexity when the representation can be selected.

The specific results are the activation-mass specialization with attained sharp constants, Brownian trace-and-envelope upper bounds and matching worst-case scalar ReLU constructions, and architecture-level propagation under the stated induced-norm conditions. Matching at the trace/envelope level does not mean tightness for every representation family or simultaneous saturation of all architectural constraints.

The revision also adds two general-family refinements: a centered off-diagonal Gram bound that recovers the fixed-head upper rate for a singleton, and a separate bound on excess above the strongest fixed representation. These are new additions, not claims about what was already proved in the submitted version. Their numerical evaluations remain conservative on the tested pools.

The accompanying red-marked manuscript separates these issues in Sections 1--3 and the limitations. Red marks all changes relative to submission, including earlier post-submission revisions; it does not mean every marked passage was prompted by a reviewer. Revised Theorems 3.3--3.4 are the new refinements; the submitted matching Theorem 3.3 is now Theorem 3.5. Full statements and proofs remain in the manuscript rather than being repeated in each reply.

AuthorsOpenReview ↗

Response to 4MFU (2/2): transformer features and scope

Your Q2: extension to transformer-based models. There is a representation-level extension, but not a proved transformer-specific version of the full architecture analysis.

For a fixed representation, Theorem 2.2 uses the sample vectors and the Brownian kernel, not the internal mechanism that produced them. Its dual identity therefore also applies to transformer features followed by the same bounded-norm Brownian head: $ \widehat{\mathcal R}_S(\mathcal F_{z,B}^{(q)}) =\frac{B}{n}\mathbb E_\varepsilon \sqrt{\varepsilon^\top K_z\varepsilon}. $ For q=2q=2, the representation-family bound likewise applies when its displayed envelopes are finite and the realized family is fixed before drawing the independent auxiliary Rademacher and Gaussian variables. This is a statement about those features and that head, not about the usual transformer output head or its training dynamics.

The ReLU architecture propagation, VC specialization, and explicit lower-bound realizations cannot simply be transferred to attention, normalization, and softmax. Those operations require their own analysis. For q=1q=1, the one-sided threshold decomposition used in the paper requires nonnegative sample states; it cannot be applied unchanged to signed transformer features. Adding a terminal ReLU would change the model, so we do not make that substitution silently.

The same distinction applies to generalization. A family fixed independently of the estimation sample can enter a fixed-class population bound. For an arbitrary family generated from that sample, the empirical calculation remains conditional and needs additional control of class--sample dependence before yielding a population-risk guarantee. We retain this limitation rather than claiming a full transformer generalization theorem.

Thanks for your reply. I will go over it shortly.

For now, can you please explain in more detail the differences and similarities between the studied model and practical settings? You mentioned that "We do not claim that these heads are equivalent", but I wonder how close/far they are from practical ones.

AuthorsOpenReview ↗

Response to 4MFU: connection to practical classifiers

Thank you for following up. The model keeps the usual division between a learned representation and a prediction head. We use a finite ReLU encoder and allow its representation to be selected using the same training sample as the head. The main modeling choice is the last stage: its score functions and regularization.

For a feature vector uu, an affine classifier uses scores sc(u)=wc⊤u+bcs_c(u)=w_c^\top u+b_c, while our head uses sc(u)=⟨ac,Φq(u)⟩s_c(u)=\langle a_c,\Phi_q(u)\rangle with RKHS norm control. Softmax can follow either score vector; our earlier wording should have distinguished the scores from that final transformation. The basic Brownian head vanishes at zero, and Appendix A.1 treats a separately bounded intercept.

The connection to finite neural readouts is particularly concrete for the additive head. Although the model is formulated in an RKHS, the kernel-ridge solutions used in our experiments have finite expansions $ s_c(u)=\sum_{i=1}^{n}\alpha_{ic}K_q(u,z_i), $ where ziz_i are the training representations and kernel normalization is absorbed into the coefficients. On nonnegative features, $ K_1(u,v)=\sum_j\min(u_j,v_j),\qquad \min(t,a)=t-\operatorname{ReLU}(t-a)\quad(t,a\geq0). $

Thus a fitted additive Brownian head is exactly a finite, piecewise-linear ReLU readout. Its knots occur at training-feature values, the kernel expansion ties coefficients across coordinates, and the Brownian norm supplies its regularizer. These are the substantive differences from an MLP head with freely learned weights and knots. Additivity is in the final features; the encoder can still represent interactions among the original inputs. The kernel K1K_1 is also the histogram-intersection kernel used in vision classifiers (Maji et al., CVPR 2008, cited in Section 2). That work uses SVM fitting, while we use ridge fitting. For q=2q=2, the kernel instead combines Euclidean norms and distances and is invariant under a common rotation of the features.

Our closest empirical comparison is the transfer study in Appendix C.9. Brownian, linear-ridge, RBF and Laplace heads are fitted on the same frozen ImageNet-pretrained ResNet-18 features and labeled splits for CIFAR-10, CIFAR-100 and EuroSAT. With 1,000 labels, the five-seed mean accuracies for q=2q=2 and linear ridge are 80.67% versus 79.63%, 47.31% versus 45.41%, and 91.90% versus 90.58%, respectively. This directly tests replacing the head while keeping the learned features fixed.

The fitting procedure matters too. Here we use multiclass squared-loss ridge and argmax prediction, whereas a network trained end to end with cross-entropy jointly adjusts the encoder and its scores. Our separate adapter experiment isolates selection among controlled candidate representations. Dense kernel fitting requires quadratic Gram storage and cubic factorization time; the transfer experiments use small labeled target sets of 500 or 1,000 examples.

So the model is close to feature-based transfer pipelines, and its fitted additive heads have the explicit ReLU realization above. The Brownian choice gives this setting a tractable geometry: activation mass and variation among the candidate Gram matrices let us bound how representation selection increases conditional complexity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.