- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the expresive power of neural networks with linear layers constrained to be diagonal-plus-low-rank (DLoR) as , via universal approximation (UAT). They motivate this by arguing that low-rank networks can interpolate finite data but remain blind to input directions outside the row space of first-layer weight (orthogonal blindness). In their DLoR constrcution, a dense layer (with nonlinearity applied) is recoved either additively (parallel low-rank branches stacked as a tall matrix) or multiplicatively (product of DoLR factors), with the activation linerized in between. Theoretically, plugging these into known dense results (Kidger–Lyons; Siegel) yields their UAT and parameter bounds. Empirically, they trained MLPs (sawtooth, MNIST) into DLoR networks with maching accuracy as validation of the constructions.
Critical strengths
1. The paper is technically correct and easy to follow. As far as my reading goes, the technical constructions and theory seem to hold. This includes their motivating their finite-data interpolation and orthogonal blindness arguments (Lem. 1-2 and Def. 1), the width and depth constructions (Thm. 1-2), and the UAT/param-bound results (Thm. 3-6). Specical cases are handled well (e.g., ReLU activation giving exact realization, retangular or singular weights). The paper is also clear and easy to follow in general. This matters because my concerns are about novelty and significance rather than correctness.
Critical weaknesses
1. Limited theoretical novelty. Overall, the main contributions follow from known results, and little seems specific to DLoR.
- a) Additive construction (Thms. 1 and 4). Its rank constraints live entirely within the parallel branches . This suggests to me that the low-rank constraint does not restrict the model class, because every dense layer can be split into low-rank branches via SVD in the first place, and that this costruction is about reparametrization. Importantly, the branches do not need the diagonal at all (), and the resulting block reduces to . This realizes at first order (when is linearized and cancelled out within the blocks), but neither nor is low-rank.
- b) Multiplicative construction (Thm. 2). Unlike the additive one, this uses the identity term in , which lets the rank- corrections accumulate. But, the factorization is a standard technique that replaces columns iteratively, and the parameter plays no role for expressivity, as we can simple factor and rescale.
- c) UAT results (Thms 3-6). From (a) and (b), once a dense layer is approximated via either width or depth, they substitue this block into the dense results of Kidger–Lyons and Siegel, which does not appear to involve new theoretical characterizations.
- d) Prior work exists for the same template. The overall approach (factor dense weights into structured factors, then neutralize the activation to transfer UAT) largely resembles prceding work for triangular and Toeplitz/Hankel [1], which should be cited. I talk a bit more about this in Questions.
2. Limited significance. I see little consequencesof the constructions, for example, for efficiency in pratice, and the framing seems to overstate the motivation.
- a) Parameter cost. Simulating one dense layer via either additive or multiplicative ways costs about parameters, so neither construction gives paramater savings over the original dense layer it replaces. This seems to contradict the PEFT/compression motivation in Sec. 1, which should be tempered. If some efficiency gains are realized via structures such as sparsity beyond raw parameter count (an analogy here would be Monarch or butterfly layers), this should be stated.
- b) Empirical results. The core experiments only verify the constructions. The sawtooth task is 1D so orthogonal blindness does not occur, and the MNIST includes no pure low-rank baselines or cost comparisons.
[1] Liu, Y., Jiao, S., & Lim, L. H. (2024). LU decomposition and Toeplitz decomposition of a neural network. Applied and Computational Harmonic Analysis, 68, 101601.
Questions for authors
Q1. Given the novelty concerns (1a-d) in Weaknesses, can the authors please clarify the paper's main technical novelty corresponding to each of (1a-b)? If any of the four weaknesses point is misattributed or incorrect, please correct me here. In addition, I'd like to ask specifically the following two points:
- As far as I understand, [1] seem to prove UAT for networks with triangular, Toeplitz or Hankel weights, where they factor dense weights into structured factors and linearize the activation in between. They also note (after their Thm. 2.4) that their proof extends to any factorization into structures preserved under scalar multiplication, which appears to include DLoR (at least for the uniformly continuous non-polynomial activations).
- For non-polynomial activations, Kidger & Lyons’s register model appears to already admit an implementation with hidden weights of the form of scaled identity plus a rank- correction at width .
If the authors could clarify on these extra points, it'd help me undersand the core novelty more clearly.
Q2. Given 2a, what does the DLoR strcuture offer in return in terms of compute, memeory, trainability, function class etc?
Thank you for the detailed rebuttal and the substantial revisions. I appreciate the authors' efforts to address my concerns, particularly the new theoretical characterization of representation costs, the matched-parameter experiments, and the additional analysis of numerical stability.
The new results based on κ(W) provide a clearer understanding of when DLoR can achieve parameter savings. I acknowledge that my original parameter-counting example represents the generic worst case, and that DLoR can be substantially more efficient for identity-centered, low-rank-perturbed matrices. This addresses an important part of my first concern.
However, my main concern regarding practical efficiency remains only partially resolved.
First, the demonstrated efficiency gains are concentrated in a specific structural regime. While the theoretical characterization is valuable, the paper does not establish how frequently this favorable regime occurs in real pretrained neural networks or whether it leads to meaningful end-to-end improvements.
Second, I appreciate the newly added matched-parameter experiments. Nevertheless, these comparisons primarily involve small synthetic tasks. In the MNIST experiments, DLoR still requires approximately 1.5 times as many parameter slots as truncated SVD to achieve comparable accuracy. Thus, the empirical evidence does not yet establish a consistent practical advantage over existing alternatives.
Third, the additional rank-1 training and conditioning analyses are useful. However, the reported sensitivity to activation choice and numerical instability at greater depths reinforces my original concerns about optimization and scalability. Moreover, the requested inference-latency evaluation remains missing. Theoretical parameter or FLOP reductions do not necessarily translate into lower inference latency, particularly when additional sequential layers are required.
Therefore, while I appreciate the improvements, I consider my original concerns about practical significance, computational efficiency, and scalability to be partially, rather than fully, addressed. Nevertheless, I still increase 1 score for your effort.