- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the authors consider learning two-input target functions with multiplicative dual-encoder heads. Based on the interaction spectrum obtained from the Schmidt decomposition of the two-input target function, they explore the approximation error, encoder identifiability, sample complexity, and fundamental architectural limitations.
Critical strengths
The strength of this paper is to show that the approximation error can be decomposed into the spectral error and encoder-realization error, which implies that rapid spectral decay indicates that the low-rank dual-encoder architectures are effective, while the flat spectrum implies that such architectures are unsuitable. The experiments provide good empirical support for the theoretical results.
Critical weaknesses
The weakness of this paper is, although this paper shows that the interaction spectrum is related to when low-rank dual encoders succeed, the interaction spectrum is unknown in practice. The spectrum might be estimated by computing it for learned DeepONet, but this depends on how successful training is. Also, I have not yet understood how the identifiability is practically meaningful, for instance, the necessity of the alignment of the learned trunk basis with the analytic operator eigenmodes.
Questions for authors
-
Is there any method to estimate the unknown interaction spectrum of the target in order to know the current data are suitable for low-rank dual encoder ?
-
How is the identifiablity proposed in this paper practically meaningful ? See also "Critical Weaknesses"
3 comments
Thank you very much for the clarification. I would like to keep my current score as weak acceptance.
Thank you for your careful review and positive score. Your comments helped us strengthen our paper significantly.
We thank the reviewer for the questions about practical value and identifiability.
W1/Q1. Practical value and estimating the target spectrum
Response. Thank you for your insightful comment. Our paper focuses on analyzing the properties of multiplicative networks rather than proposing a new algorithm. The theory provides principles for future model and algorithm development. We illustrate its use through two simple procedures: choosing output dimension from the response spectrum before training, and truncating a fitted head after training. Both reduce dimension and clean-test error in the examples below.
Before training, the normalized singular values of the full input–query response matrix estimate the interaction spectrum under the sampled measures. Fast decay favors a low-width approximation; a flat spectrum indicates a large truncation cost. This estimate uses cross-pair responses directly, independently of encoder training; sparse paired labels require a model for missing responses (Section 4.3).
For heat with 32 training functions and label-noise standard deviation equal to 20% of clean-target RMS, retaining 99% of training-response energy selects the dimension before training. The table compares this rule with a fixed 32-dimensional head. NMSE is total squared error divided by total squared clean-target norm; entries are mean ± standard deviation over three seeds, scaled by 1000.
The three selected dimensions correspond to the three seeds. Spectral selection lowers mean NMSE by 13.3% relative to fixed width, with improvement in every seed. Across other data conditions, the spectral rule gives no consistent accuracy advantage: wider models perform better with clean data, and more aggressive energy thresholds underfit. Finite-grid estimates are also less reliable for slow and flat spectra (Section 5.4; Appendices G.4–G.6).
After training, paired maps fitted on training embeddings order a POD-DeepONet's residual coordinates. A training-only, noise-adjusted rule retains 4 of 32 coordinates for heat and 6 for antiderivative, keeping the training mean without fine-tuning. The full coordinate change preserves predictions; truncation removes tail components. The table reports relative L2 error: each function's error norm is divided by its mean-removed target norm plus a numerical stabilizer, then the ratios are averaged. Entries are mean ± standard deviation over five optimization seeds on shared data, scaled by 1000.
Truncation lowers mean error by 37.8% on heat and 16.1% on antiderivative, improving every seed. POD's original variance ordering gives similar improvements; truncation raises error for unconstrained and soft-whitening heads. The evidence consists of archived summaries and a source snapshot; checkpoints, paired predictions, and source-version links are unavailable (Section 5.5; Appendix G.7).
Sections 5.4–5.5 present the examples and key results in blue.
W2/Q2. What identifiability is useful for
Response. Thank you for your insightful comment. The practical role of identifiability is to remove infinitely many equivalent encoder representations and constrain the redundant directions that prediction loss leaves flat. For example, (f,g) and (kf,g/k) give identical predictions for any positive scale k. Whitening and diagonalization fix this continuous coordinate freedom; at a canonical head with a simple spectrum, the corresponding moment penalty adds positive curvature along those directions.
The uniqueness concerns the interaction coordinates of a fixed fitted function under fixed reference measures. With nonsingular moments and a simple spectrum, the ordered coordinates are fixed up to paired signs; repeated values identify blocks. Matching physical operator eigenmodes is unnecessary. Agreement with target modes additionally requires prediction accuracy and spectral separation; exact recovery follows when the head attains the target's optimal truncation (Sections 3.1 and 4.2; Appendix F).
The local-curvature result supplies a mechanism for controlling redundant directions during training. It does not imply generally better full-Hessian conditioning or faster convergence (Section 3.2; Appendix D.4, new result in blue). Empirically, soft whitening lowers prediction error on all four archived DeepONet operators, while the synthetic study has higher fit error and mixed mode-alignment effects (Sections 5.2–5.3).