acceptodds
Under review as a conference paper at ICLR 2027

Low-Interaction-Rank Learning: Unifying Multiplicative Dual-Encoder Heads

Abstract

A multiplicative dual-encoder network computes a real-valued output for a pair of inputs as the inner product of their separate encodings. This architecture has been developed independently in operator learning, bipartite matching, contrastive vision-language models, retrieval, and other areas, yet no unified theory guides the basic design decisions: how many interaction modes to represent, how to normalize the encoders, and when the architecture should be avoided. We provide such a foundation by introducing the class of functions of low interaction rank, a class whose intrinsic complexity is measured by its interaction spectrum. Within this framework, approximation error decomposes into a spectral truncation term and an encoder-realization term; sample complexity is governed by the sum of the two encoder complexities rather than their product; and a usability criterion based on spectral decay determines when the architecture can succeed. The same framework exposes a central identifiability problem: the encoders are defined only up to a linear gauge symmetry that leaves the learned coordinates arbitrary. We show that normalization is gauge fixing and that whitening pins the interaction modes up to permutation and sign, thereby explaining the uninterpretability of contrastive dimensions and providing a constructive remedy. Experiments on synthetic kernels, operator learning, and CLIP models validate the theoretical predictions: spectral decay rates match the predicted scaling, whitening recovers the true modes, and independently trained CLIP models are related by a single rotation which, after removal by whitening, exposes interpretable concept axes. The code of this paper is provided at anonymous repository https://anonymous.4open.science/r/Mul-Net-FDF0 .

Primary area:
learning theory
Keywords:
Low-Rank, Multiplication Network

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_maE2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors consider learning two-input target functions with multiplicative dual-encoder heads. Based on the interaction spectrum obtained from the Schmidt decomposition of the two-input target function, they explore the approximation error, encoder identifiability, sample complexity, and fundamental architectural limitations.

Critical strengths

The strength of this paper is to show that the approximation error can be decomposed into the spectral error and encoder-realization error, which implies that rapid spectral decay indicates that the low-rank dual-encoder architectures are effective, while the flat spectrum implies that such architectures are unsuitable. The experiments provide good empirical support for the theoretical results.

Critical weaknesses

The weakness of this paper is, although this paper shows that the interaction spectrum is related to when low-rank dual encoders succeed, the interaction spectrum is unknown in practice. The spectrum might be estimated by computing it for learned DeepONet, but this depends on how successful training is. Also, I have not yet understood how the identifiability is practically meaningful, for instance, the necessity of the alignment of the learned trunk basis with the analytic operator eigenmodes.

Questions for authors
  1. Is there any method to estimate the unknown interaction spectrum of the target in order to know the current data are suitable for low-rank dual encoder ?

  2. How is the identifiablity proposed in this paper practically meaningful ? See also "Critical Weaknesses"

3 comments

We thank the reviewer for the questions about practical value and identifiability.

W1/Q1. Practical value and estimating the target spectrum

Response. Thank you for your insightful comment. Our paper focuses on analyzing the properties of multiplicative networks rather than proposing a new algorithm. The theory provides principles for future model and algorithm development. We illustrate its use through two simple procedures: choosing output dimension from the response spectrum before training, and truncating a fitted head after training. Both reduce dimension and clean-test error in the examples below.

Before training, the normalized singular values of the full input–query response matrix estimate the interaction spectrum under the sampled measures. Fast decay favors a low-width approximation; a flat spectrum indicates a large truncation cost. This estimate uses cross-pair responses directly, independently of encoder training; sparse paired labels require a model for missing responses (Section 4.3).

For heat with 32 training functions and label-noise standard deviation equal to 20% of clean-target RMS, retaining 99% of training-response energy selects the dimension before training. The table compares this rule with a fixed 32-dimensional head. NMSE is total squared error divided by total squared clean-target norm; entries are mean ± standard deviation over three seeds, scaled by 1000.

MethodOutput dimensionTest NMSE
Fixed width3210.134 ± 0.134
Spectral selection14/14/108.785 ± 0.689

The three selected dimensions correspond to the three seeds. Spectral selection lowers mean NMSE by 13.3% relative to fixed width, with improvement in every seed. Across other data conditions, the spectral rule gives no consistent accuracy advantage: wider models perform better with clean data, and more aggressive energy thresholds underfit. Finite-grid estimates are also less reliable for slow and flat spectra (Section 5.4; Appendices G.4–G.6).

After training, paired maps fitted on training embeddings order a POD-DeepONet's residual coordinates. A training-only, noise-adjusted rule retains 4 of 32 coordinates for heat and 6 for antiderivative, keeping the training mean without fine-tuning. The full coordinate change preserves predictions; truncation removes tail components. The table reports relative L2 error: each function's error norm is divided by its mean-removed target norm plus a numerical stabilizer, then the ratios are averaged. Entries are mean ± standard deviation over five optimization seeds on shared data, scaled by 1000.

MethodHeatAntiderivative
POD-DeepONet13.984 ± 0.37018.108 ± 0.332
Spectral truncation8.702 ± 0.40215.193 ± 0.331

Truncation lowers mean error by 37.8% on heat and 16.1% on antiderivative, improving every seed. POD's original variance ordering gives similar improvements; truncation raises error for unconstrained and soft-whitening heads. The evidence consists of archived summaries and a source snapshot; checkpoints, paired predictions, and source-version links are unavailable (Section 5.5; Appendix G.7).

Sections 5.4–5.5 present the examples and key results in blue.

W2/Q2. What identifiability is useful for

Response. Thank you for your insightful comment. The practical role of identifiability is to remove infinitely many equivalent encoder representations and constrain the redundant directions that prediction loss leaves flat. For example, (f,g)(f,g) and (kf,g/k)(kf,g/k) give identical predictions for any positive scale kk. Whitening and diagonalization fix this continuous coordinate freedom; at a canonical head with a simple spectrum, the corresponding moment penalty adds positive curvature along those directions.

The uniqueness concerns the interaction coordinates of a fixed fitted function under fixed reference measures. With nonsingular moments and a simple spectrum, the ordered coordinates are fixed up to paired signs; repeated values identify blocks. Matching physical operator eigenmodes is unnecessary. Agreement with target modes additionally requires prediction accuracy and spectral separation; exact recovery follows when the head attains the target's optimal truncation (Sections 3.1 and 4.2; Appendix F).

The local-curvature result supplies a mechanism for controlling redundant directions during training. It does not imply generally better full-Hessian conditioning or faster convergence (Section 3.2; Appendix D.4, new result in blue). Empirically, soft whitening lowers prediction error on all four archived DeepONet operators, while the synthetic study has higher fit error and mixed mode-alignment effects (Sections 5.2–5.3).

Thank you very much for the clarification. I would like to keep my current score as weak acceptance.

Thank you for your careful review and positive score. Your comments helped us strengthen our paper significantly.

Reviewer_axnNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper considers multiplicative dual-encoder architectures with two inputs encoded separately and representations combined through an inner product. It proposes an interaction-spectrum perspective to unify architectures used in retrieval, operator learning, and contrastive representation learning. The authors evaluate approximation error through spectral truncation and encoder-realization errors, which tries to relate generalization error to the complexities of the two encoder classes, and discuss when spectral decay makes the architecture suitable. They also analyze the linear gauge symmetry of dual encoders and propose whitening constraints to identify singular modes up to sign and ordering. An equality-function construction illustrates the limitation of low-interaction-rank models in terms of expressivity. Experiments on synthetic problems, operator-learning tasks, and CLIP representations are presented to support the approximation and identifiability arguments.

Critical strengths

The interaction-spectrum framework provides a coherent perspective on dual-encoder architectures across several application areas. The connections among spectral truncation, model capacity, and architecture selection are conceptually useful.

Critical weaknesses
  1. Theorem 1 appears to omit assumptions required by its proof. In Appendix B, the authors bound a product of coordinate wise encoder errors by assuming that the aggregate squared value-encoder error is at most one and that individual errors are sufficiently small. These restrictions are not stated in Assumption 1 or Theorem 1. Without them, the cross-error term can scale quadratically with the number of modes, so the claimed realization-error bound is not justified as written. Also, Theorem 3 appears to overstate the singular-value decay of general multidimensional analytic kernels. Exponential decay in the singular-value index is claimed without sufficient dimensional qualifications; for analytic kernels on higher-dimensional domains, stretched-exponential decay is generally possible. This issue affects the stated spectrum-based scaling conclusions and requires a corrected theorem or stronger assumptions.

  2. Even if the technical issues were resolved, Theorem 1 would remain a relatively straightforward consequence of the classical Schmidt/Eckart--Young decomposition and elementary tensor-product error bounds, offering limited mathematical novelty beyond a standard approximation-error decomposition.

Questions for authors
  1. Can the authors revise Theorem 1 to explicitly state the required aggregate realization-error assumptions, or derive a valid bound that retains the cross-error contribution without these assumptions?
  2. Can the authors justify the exponential spectral-decay claim in Theorem 3 for multidimensional analytic input domains, and clarify how the resulting interaction-rank scaling changes if only stretched-exponential decay is available?
3 comments

We thank the reviewer for the detailed examination. The theoretical corrections are marked in blue in the cited sections.

W1. Missing realization-error assumptions and multivariate analytic decay

Response. Thank you for your insightful comment. The corrected Theorem 1 holds for arbitrary finite realization errors achieved jointly by one admissible encoder pair, with their product retained. Under the stated holomorphic-extension assumption, analytic decay is dimension-dependent stretched exponential, reducing to ordinary exponential in one dimension.

The original small-error condition is confined to a simpler corollary. The product accounts for the potentially quadratic growth in width at fixed coordinatewise errors. Section 2.1 and Appendix C.2 give the corrected statement and proof; Q1 below explains the repair.

For an mm-dimensional input argument, a bounded Hilbert-valued holomorphic extension to a fixed polyellipse yields a squared spectral tail τd≤Cexp⁡(−cd1/m)\tau_d\leq C\exp(-cd^{1/m}) beyond width dd, with positive constants C,cC,c. Polynomial degree LL per coordinate uses up to (L+1)m(L+1)^m terms. Ordinary exponential decay is recovered in one dimension. Section 2.2 and Appendix C.3 state the corrected theorem, and Section 4.1 and Appendix E.2 update its width-scaling consequence.

W2. Contribution beyond classical low-rank approximation

Response. Thank you for your insightful comment. The contribution is to connect multiplicative-network design to approximation and recovery: output width controls spectral truncation error, encoder constraints contribute a two-sided realization cost, and normalization fixes coordinate ambiguity, with target-mode recovery bounded by prediction error under spectral gaps.

The Schmidt/Eckart–Young decomposition supplies the classical tail bound. Our realization bound requires one admissible encoder pair to approximate all coordinates jointly and retains the interaction of their errors. Sections 1–4 connect this bound to coordinate ambiguity, normalization, and independent calibration; Appendix B compares this analysis with prior approximation and spectral-learning theory.

Q1. How the general realization-error bound is repaired

Response. Thank you for your insightful comment. The corrected bound holds for arbitrary finite errors jointly realized by one admissible encoder pair. It retains the cross-error product, following your second suggestion, so the general theorem no longer needs the original proof's small-error restrictions.

Let σk\sigma_k be the target's interaction singular values, and let ηf,k,ηg,k\eta_{f,k},\eta_{g,k} bound the L2 approximation errors for its scaled left and unscaled right singular functions, jointly realized by one admissible encoder pair. For output dimension dd, define

X2=∑k=1dηf,k2,Y2=∑k=1dσk2ηg,k2,X^2=\sum_{k=1}^d\eta_{f,k}^2,\quad Y^2=\sum_{k=1}^d\sigma_k^2\eta_{g,k}^2,

Z2=∑k=1dηg,k2,τd=∑k>dσk2.Z^2=\sum_{k=1}^d\eta_{g,k}^2,\quad \tau_d=\sum_{k>d}\sigma_k^2.

The corrected bound is

inf⁡admissible (f,g)∥F−⟨f,g⟩∥L22≤τd+4X2+4Y2+3X2Z2,\inf_{\text{admissible }(f,g)}\Vert F-\langle f,g\rangle\Vert_{L^2}^2\leq\tau_d+4X^2+4Y^2+3X^2Z^2,

where FF is the target, ⟨f,g⟩\langle f,g\rangle is the encoder inner product, and L2 uses the fixed product input measure. The two-sided error interaction has norm at most XZXZ. Schmidt orthogonality and the residual operator norm control its cross terms while retaining coefficient one on the squared tail. Under the additional condition Z2≤1Z^2\leq1, the simpler corollary follows. Section 2.1 states the theorem, and Appendix C.2 gives the derivation.

Q2. Dimension dependence and sufficient interaction rank

Response. Thank you for your insightful comment. Under the stated holomorphic-extension assumption, the squared spectral tail obeys τd≤Cexp⁡(−cd1/m)\tau_d\leq C\exp(-cd^{1/m}) for output width dd and input dimension mm, with positive constants C,cC,c. The resulting sufficient width is of order (log⁡(1/ηn))m(\log(1/\eta_n))^m, where ηn\eta_n is estimation cost per retained dimension, subject to admissibility and encoder-realization cost.

Suppose an established oracle inequality bounds prediction risk at width dd by

R(d)≤Aexp⁡(−cdβ)+ηnd+Ereal(d),R(d)\leq A\exp(-cd^\beta)+\eta_n d+E_{\mathrm{real}}(d),

where A,c,βA,c,\beta are positive constants, ηn\eta_n is the estimation coefficient per retained dimension for sample size nn, and ErealE_{\mathrm{real}} is encoder-realization cost. For 0<ηn<A0<\eta_n<A, choose dn=max⁡(1,⌈(c−1log⁡(A/ηn))1/β⌉)d_n=\max(1,\lceil(c^{-1}\log(A/\eta_n))^{1/\beta}\rceil), provided this integer width is admissible. It makes the tail at most ηn\eta_n, giving

dn=O((log⁡(1/ηn))1/β),R(dn)≤ηn(1+dn)+Ereal(dn).d_n=O((\log(1/\eta_n))^{1/\beta}),\quad R(d_n)\leq\eta_n(1+d_n)+E_{\mathrm{real}}(d_n).

For the analytic theorem, β=1/m\beta=1/m. Sections 2.2 and 4.1 and Appendices C.3 and E.2 give the corrected decay and scaling statements.

I have read the authors' response and would like to keep my rating.

Thank you for your careful review and suggestions. Your comments helped us strengthen our paper significantly.

Reviewer_bgfdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a unified framework for understanding when dual-encoder models work well, how large their embeddings should be, and how to make their learned coordinates more identifiable. These models encode two inputs separately and combine their representations through an inner product. The paper connects this design across applications including CLIP, information retrieval, and DeepONet. The central idea is the interaction spectrum, which describes how the relationship between the two inputs is distributed across independent modes of variation. The paper uses this perspective to study approximation error, sample requirements, embedding-dimension selection, and situations in which combining the inputs earlier may be advantageous. The paper also examines representation ambiguity. It interprets normalization as a way to reduce this ambiguity and proposes whitening together with spectral diagonalization to recover canonical interaction modes under suitable assumptions.

Critical strengths
  1. The spectral formulation is useful, and its basic lower bound is correct.
  2. The core gauge identity and population canonicalization argument are sound.
  3. The paper distinguishes the finite-sample whitening estimator from a trained optimizer in its gap experiments.
  4. Some empirical findings are promising, and limitations are reported.
Critical weaknesses
  1. Several theoretical conclusions are broader than the stated assumptions support. The paper presents a comprehensive theory of approximation, identifiability, and sample efficiency, but some central guarantees require additional assumptions or correction. In particular, the approximation proof introduces restrictions that are absent from the theorem statement, and the claimed spectral decay for analytic targets is too general.
  2. The benefits of whitening for optimization are overstated.
  3. The CLIP experiments do not directly validate the transformation analyzed in the theory.
  4. The novelty needs better positioning against closely related spectral representation learning work. NeuralSVD [1] already studies neural operator decomposition through low-rank approximation and introduces a mechanism for recovering ordered singular-function components.

[1] Operator SVD with Neural Networks via Nested Low-Rank Approximation.

Questions for authors
  1. Could you clarify the conditions required for the approximation bound, the spectral-decay result for multivariate analytic targets, and the stated learning rates?
  2. Could you distinguish the guarantees about identifying canonical interaction modes from the claims about better conditioning and faster convergence?
  3. What is the principal advance over existing methods for learning ordered spectral representations?
4 comments
AuthorsOpenReview ↗

Response Part 1

We thank the reviewer for the assessment. Important corrections and additions are marked in blue in the cited sections.

W1. Assumptions and scope of the theoretical guarantees

Response. Thank you for your insightful comment. We have corrected the guarantees: the approximation bound holds for arbitrary finite errors jointly realized by one admissible encoder pair when their product is retained; a bounded holomorphic extension gives dimension-dependent stretched-exponential decay; and mode-norm bounds depend on the square root of squared prediction risk under spectral-gap conditions.

The approximation theorem requires one admissible pair to realize all coordinate approximations jointly. The original small-error condition is needed only for a simpler corollary (Section 2.1; Appendix C.2).

For an mm-dimensional input argument, the analytic result now gives τd≤Cexp⁡(−cd1/m)\tau_d\leq C\exp(-cd^{1/m}), where τd\tau_d is the squared spectral tail beyond output dimension dd and C,cC,c are positive constants. The theorem assumes a bounded Hilbert-valued holomorphic extension to a fixed polyellipse. Ordinary exponential decay follows in one dimension; the multivariate result retains the dimension dependence (Section 2.2; Appendix C.3).

For nn training observations, an O(n−1/2)O(n^{-1/2}) excess squared-risk contribution yields an O(n−1/4)O(n^{-1/4}) mode-norm contribution at fixed scales and spectral gaps. Realization and optimization errors enter through the same square root. Sections 4.1–4.2 and Appendices E–F state the learning and calibration assumptions separately.

W2. What whitening establishes for optimization

Response. Thank you for your insightful comment. We have replaced the general conditioning and convergence claim with a local result: at a canonical head with a simple spectrum, the moment penalty adds positive curvature along every nonzero direction that changes coordinates while preserving predictions.

Whitening fixes scale freedom, and diagonalizing the other encoder's second moment controls rotations; repeated eigenvalues retain block rotations. If an optimal admissible representation has zero penalty, adding the penalty preserves the minimum prediction loss. Section 3.2 and Appendix D.4 prove these statements and give a counterexample in which the full Hessian condition number increases.

We use prediction experiments to assess whether this normalization helps in practice. Soft whitening lowers prediction error on all four operators in the archived DeepONet study, whereas the synthetic Legendre study gives higher fit error and mixed mode-alignment effects. These comparisons evaluate the complete training objectives. Sections 5.2–5.3 and Appendices G.1–G.2 report the results.

W3. Whether the CLIP experiment tests the theoretical transformation

Response. Thank you for your insightful comment. The new direct test confirms that, for OpenAI ViT-B/32, the full coupled whitening and diagonalization preserves the original image–text scores to numerical precision and leaves retrieval hits unchanged. This tests the score-preserving transformation analyzed in the theory; the earlier cross-model alignment experiment measures coordinate agreement.

The test uses a fixed pretrained scorer, once-normalized features, uncentered nonsingular moments, zero regularization, and no subsequent row normalization. Section 3.1 and Appendix D.3 give the construction; Appendix G.3 marks the direct test and results in blue.

Rank-deficient cases are evaluated separately: projection onto LAION's numerical text support meets the reported relative score-error tolerance, while RN50 fails the same check. Truncation can reduce classification and retrieval accuracy, and a score-matrix SVD gives equivalent truncation results. The earlier cross-model correlation experiment is now described as in-sample coordinate alignment. Appendix G.3 explains these distinct comparisons and gives the full protocol.

W4. Relation to NeuralSVD

Response. Thank you for your insightful comment. NeuralSVD develops an objective for learning ordered operator singular functions. Our results explain properties of the multiplicative architecture across objectives, including ordinary prediction losses: how width limits approximation, how normalization fixes equivalent coordinates, and when prediction accuracy yields mode recovery.

NeuralSVD's nonnested objective is equivalent to squared Hilbert–Schmidt reconstruction up to a constant, and its subspace method uses Rayleigh–Ritz postprocessing. The shared low-rank foundation leads here to a two-sided neural realization bound, a local penalty-curvature result, and stability bounds for mode recovery and independent calibration. Appendix B explains the connection and difference in scope.

Appendix B also discusses SpIN, NeuralEF, Neural Eigenmap, and prior work on capacity, operator approximation, contrastive learning, and factorization geometry.

AuthorsOpenReview ↗

Response Part 2

Q1. Conditions for approximation, analytic decay, and learning rates

Response. Thank you for your insightful comment. The corrected approximation bound permits arbitrary finite, jointly realized encoder errors by retaining their product; a bounded holomorphic extension gives multivariate stretched-exponential decay. Under independent product-law sampling, bounded regression, and spectral-gap conditions, squared-risk bounds yield mode-norm bounds through a square root. The full assumptions are now explicit.

The approximation theorem uses fixed reference measures and one admissible encoder pair that realizes all coordinate approximations jointly; its bound retains the cross-error product (Section 2.1; Appendix C.2). The analytic theorem requires a bounded Hilbert-valued holomorphic extension to the specified polyellipse (Section 2.2; Appendix C.3).

The regression bound assumes independent observations with product input law, conditional-mean-zero noise, bounded labels and predictions, controlled encoder-class complexity, and an explicit optimization residual. The faster linear benchmark additionally requires restricted curvature and control of the realized noise operator (Section 4.1; Appendices E.1–E.2). Given an established oracle bound with estimation term ηnd\eta_n d, where ηn\eta_n is its coefficient per retained dimension, the analytic tail gives a sufficient width of order (log⁡(1/ηn))m(\log(1/\eta_n))^m. This choice must be admissible, and its encoder-realization cost remains in the risk bound.

Mode recovery additionally requires boundary and individual-mode spectral gaps and perturbations small relative to those gaps. Independent calibration assumes bounded canonical outputs, nonsingular moments, and squared-singular-value separation (Section 4.2; Appendix F).

Q2. Identification versus conditioning and convergence

Response. Thank you for your insightful comment. With nonsingular moments and a simple spectrum, canonicalization identifies the represented function's coordinates; these are target modes when the head attains the target's optimal truncation. The optimization result instead gives local curvature along prediction-preserving coordinate changes, without a general convergence-speed guarantee.

Section 3.1 states the identification result for output dimension dd. Section 3.2 proves the local-curvature result, and Appendix D.4 gives the full-Hessian counterexample. Section 4.2 and Appendix F bound target-mode error in terms of prediction error under spectral-gap conditions.

The sample-size experiment holds the represented function fixed and measures how accurately its modes can be estimated. It therefore tests calibration as sample size and spectral separation vary (Section 5.2; Appendix G.1.1).

Q3. Principal advance over ordered spectral-learning methods

Response. Thank you for your insightful comment. The principal advance is a joint account of approximation and identification for the multiplicative architecture: a two-sided neural realization bound at fixed width, a local curvature guarantee for normalization along prediction-preserving directions, and target-mode error bounds from prediction error under spectral gaps. These results also cover heads trained with ordinary prediction losses.

Classical low-rank approximation and ordered spectral-learning methods provide the foundations. Sections 1–4 connect these architectural properties to constrained encoder realization and independent calibration; Appendix B compares them with prior work. Sections 5.4–5.5 illustrate two uses: choosing output dimension from training responses and truncating a fitted head.

Thank you for the detailed rebuttal! However, the relationships between the revised theoretical results and the experiments are somewhat unclear. I would like to maintain my current rating.

Thank you for your careful review and the follow-up comments. In our paper, each experiment directly instantiates one theoretical result: synthetic studies test spectral truncation, canonical-coordinate recovery, and spectral-gap effects; DeepONet studies test normalization, pretraining dimension selection, and post-training truncation; and CLIP tests the score-preserving coupled transformation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank the reviewers, thAC, and the PC for their careful reading and constructive feedback. With one day remaining in the rebuttal period, we summarize our contributions and the revisions made in response to the reviews.

Contributions

  • A systematic study of a widely used architecture. We present the first systematic study that jointly analyzes approximation, identifiability, estimation, and architectural limits of multiplicative dual-encoder heads through a common interaction-spectrum framework. These heads are widely used in contrastive learning, retrieval, operator learning, and reinforcement learning. Understanding their shared properties provides a theoretical basis for developing models across these applications.

  • Properties that explain capacity, representation, and learning. We relate finite-width approximation error to the target’s spectral tail and both encoders’ realization errors, including their interaction. We characterize the coordinate ambiguity behind infinitely many equivalent representations and show how whitening and diagonalization select canonical modes under nonsingular moments and a simple spectrum. We also derive local penalty curvature along redundant directions, distinguish prediction from mode-recovery and calibration error, and establish a flat-spectrum limitation on low-dimensional heads.

  • Experimental evidence and useful design implications. Controlled experiments examine spectral bottlenecks and mode recovery. In DeepONet, soft whitening lowers prediction error on four operator-learning tasks. Two simple applications further illustrate the theory’s value: selecting output dimension before training reduces clean-test NMSE by 13.3% relative to a fixed 32-dimensional head in the small-sample noisy-heat experiment; truncating a fitted POD-DeepONet without fine-tuning reduces mean relative L2 test error by 37.8% on heat and 16.1% on antiderivative.

Responses and revisions

  • Further derivations and corrections to the theory. We corrected the approximation bound by retaining the two-encoder cross-error term, established dimension-dependent stretched-exponential decay for analytic targets, and updated the resulting width scaling. We distinguished squared-risk, mode-recovery, and independent-calibration rates, and derived the moment penalty’s local curvature along redundant prediction-preserving directions.

  • Concrete applications of the analysis. We added the pretraining dimension-selection and post-training truncation examples, with direct comparisons showing their effects on model dimension and test error.

  • Clearer positioning relative to prior work. We expanded Appendix B to discuss NeuralSVD and related spectral-learning studies, explaining how our joint analysis of architectural properties builds on these foundations and guides the study of multiplicative networks across training objectives.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.