- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Rationally Linearized Multilayer Perceptrons to approximate MLPs and accelerate the computations. The authors provided detailed mathematical analyses of their methods. Experiments are conducted to show that RaliMPs can save the computation memory.
Critical strengths
The method is supported by detailed mathematical analyses.
Experiments can also support the claims.
Critical weaknesses
The description "efficient" across the paper is not clear. Readers may tend to understand it as "time efficiency" rather than "space efficiency". This increased the difficulty for reading.
Questions for authors
- Why can Eqn 1 be approximated by Eqn 2? I did not find the reason in the context.
- In Figure 2, I believe that the MSE is not small enough---is the task too difficult? I expect the MSE to be lower than
1e-3or even lower. Figure 2 seems to tell me that the networks are not trained sufficiently. Can the author find some easier tasks to compare them? I believe the MSE now is not low enough. - How does RaliMPs perform in CNNs? The authors may consider the parameter sharing in convolutions.
- In Figure 3, can the authors show the accuracy of baselines, namely the MLPs?
- In LLMs, how does the method run? Replace MLPs from the pre-trained models or train the whole models from the start?
- Is RaliMPs easy to train in large models? (including ViTs and LLMs)
3 comments
Dear Reviewer uNmD,
Thank you again for your positive assessment. In our response we clarify the meaning of "efficient", the step from Eq. 1 to Eq. 2 and the scale of the MSE in Fig. 2, add results for CNNs (ResNet-18 on CIFAR-100, 3 seeds). As the discussion period ends soon, we would be grateful for your feedback, and we are happy to clarify anything further. If our response addresses your questions, we would appreciate it if you would consider updating your score.
Sincerely,
The Authors.
Thanks for your detailed reply. All my concerns are addressed. I will maintain my score.
We thank the reviewer for their reviews.
Thank you very much for the comment. We mean here inference cost: deployed parameters, memory and latency (Fig. 3, Tables 3 and 11); the revision says so.
Eq. 2 is a random-feature approximation of Eq. 1 (Eq. 3); Sec. 2 constructs it, Lemma 1 shows it is unbiased and Theorem 1 bounds its error. We will clarify this in the camera ready version of the paper.
Thank you for an excellent question.
The values in Fig. 2 are on a relative scale, so they should not be compared with an absolute threshold such as 10⁻³. Each output coordinate of the target is standardized to mean 0 and variance 1 over the inputs. On this scale:
Without standardization the same fit could show an arbitrarily small MSE simply because the raw targets vary little, so we report the scale-free version. We have added this explanation to App. D.1. The values in Fig. 2 reflect the task, not under-training:
RaLiMP is the only method that goes below the uncompressed linear map: for GELU already at 16 features (0.34) and down to 0.23 at 128; for tanh and sigmoid at 128 features. For comparison, on SIREN image fitting (Fig. 5) RaLiMP reaches 41.0 dB PSNR, i.e. MSE ≈ 8×10⁻⁵.
In a CIFAR ResNet-18 trained on CIFAR-100 (3 seeds), RaLiMP in the last k blocks gives:
The dense ViT is the dashed "baseline" line (92.0 / 81.9 / 58.2, Table 13); we now label it. And we have also added the baseline ViT numbers for VTAB tasks in the caption.
We replace the last k FFNs of pretrained Qwen3-8B and train them (SFT, then distillation); GPT-2 is trained from scratch. We will clarify this in the camera ready version of the paper.
Training is stable for ViT and GPT-2 and generally follows the trajectories of EUGen and SNNK. For LLMs, (1) large model sizes and (2) the fact that no prior random feature method attempted this before need to be taken into account. However with unit normalization and exponent clipping, training is stable.
Finally we would like to point out that the presented recipe to incorporate these linear MLP methods is novel and has not been attempted in the past to the best of our knowledge.