acceptodds
Under review as a conference paper at ICLR 2027

Rationally Linearized MLPs

Abstract

We present *Rationally Linearized Multilayer Perceptrons* (**RaLiMPs**), a new approach to efficiently linearizing MLPs using low-degree rational approximations of standard activation functions. Inspired by polynomial-based MLP linearization, we show that rational approximations can provide accurate representations at substantially lower degree. RaLiMPs exploit these approximations to construct kernel representations with narrow bottlenecks, enabling efficient composition across MLP layers. We provide theoretical guarantees for the resulting approximation and evaluate RaLiMPs across applications ranging from implicit 3D models to large language models. Our experiments demonstrate favorable accuracy–efficiency tradeoffs, with RaLiMPs achieving competitive or improved accuracy while substantially reducing inference cost.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Linearized Multilayer Perceptron, Rational activations, Random features, Kernel method

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uNmDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Rationally Linearized Multilayer Perceptrons to approximate MLPs and accelerate the computations. The authors provided detailed mathematical analyses of their methods. Experiments are conducted to show that RaliMPs can save the computation memory.

Critical strengths

The method is supported by detailed mathematical analyses.

Experiments can also support the claims.

Critical weaknesses

The description "efficient" across the paper is not clear. Readers may tend to understand it as "time efficiency" rather than "space efficiency". This increased the difficulty for reading.

Questions for authors
  1. Why can Eqn 1 be approximated by Eqn 2? I did not find the reason in the context.
  2. In Figure 2, I believe that the MSE is not small enough---is the task too difficult? I expect the MSE to be lower than 1e-3 or even lower. Figure 2 seems to tell me that the networks are not trained sufficiently. Can the author find some easier tasks to compare them? I believe the MSE now is not low enough.
  3. How does RaliMPs perform in CNNs? The authors may consider the parameter sharing in convolutions.
  4. In Figure 3, can the authors show the accuracy of baselines, namely the MLPs?
  5. In LLMs, how does the method run? Replace MLPs from the pre-trained models or train the whole models from the start?
  6. Is RaliMPs easy to train in large models? (including ViTs and LLMs)
3 comments

We thank the reviewer for their reviews.

"Efficient".

Thank you very much for the comment. We mean here inference cost: deployed parameters, memory and latency (Fig. 3, Tables 3 and 11); the revision says so.

Eq. 1 → Eq. 2.

Eq. 2 is a random-feature approximation of Eq. 1 (Eq. 3); Sec. 2 constructs it, Lemma 1 shows it is unbiased and Theorem 1 bounds its error. We will clarify this in the camera ready version of the paper.

Fig. 2 MSE.

Thank you for an excellent question.

The values in Fig. 2 are on a relative scale, so they should not be compared with an absolute threshold such as 10⁻³. Each output coordinate of the target is standardized to mean 0 and variance 1 over the inputs. On this scale:

  • A model that ignores the input and always predicts the mean (0) has MSE = 1, the variance of the target.
  • A perfect model has MSE = 0.
  • In between, MSE = 1 − R². For example, MSE = 0.3 means the model accounts for 70% of the variation of the target across inputs (R² = 0.7), and the remaining 30% is left in the errors.

Without standardization the same fit could show an arbitrarily small MSE simply because the raw targets vary little, so we report the scale-free version. We have added this explanation to App. D.1. The values in Fig. 2 reflect the task, not under-training:

  • The models are trained to their optimum. For the low-rank baseline the best rank-r map has a closed form (reduced-rank regression), and the trained model matches it on the same data: held-out MSE 0.49 vs. 0.50 at rank 16 and 0.40 vs. 0.39 at rank 128 (GELU).
  • The task is hard by design. Each method compresses a full 256×256 layer into 16–128 features shared by all 256 outputs, and after centering a large part of the target is nonlinear. Even the uncompressed 256×256 linear map leaves 0.38 (GELU), 0.33 (tanh) and 0.21 (sigmoid). An MSE below 10⁻³ would require capturing this nonlinear part almost exactly with fewer features than the layer has inputs.

RaLiMP is the only method that goes below the uncompressed linear map: for GELU already at 16 features (0.34) and down to 0.23 at 128; for tanh and sigmoid at 128 features. For comparison, on SIREN image fitting (Fig. 5) RaLiMP reaches 41.0 dB PSNR, i.e. MSE ≈ 8×10⁻⁵.

CNNs

In a CIFAR ResNet-18 trained on CIFAR-100 (3 seeds), RaLiMP in the last k blocks gives:

Densek=2k=4
Top-178.7 ± 0.176.9 ± 0.372.1 ± 0.2
Inference params11.2M8.1M7.4M

Fig. 3 baseline.

The dense ViT is the dashed "baseline" line (92.0 / 81.9 / 58.2, Table 13); we now label it. And we have also added the baseline ViT numbers for VTAB tasks in the caption.

LLMs.

We replace the last k FFNs of pretrained Qwen3-8B and train them (SFT, then distillation); GPT-2 is trained from scratch. We will clarify this in the camera ready version of the paper.

Ease of training.

Training is stable for ViT and GPT-2 and generally follows the trajectories of EUGen and SNNK. For LLMs, (1) large model sizes and (2) the fact that no prior random feature method attempted this before need to be taken into account. However with unit normalization and exponent clipping, training is stable.

Finally we would like to point out that the presented recipe to incorporate these linear MLP methods is novel and has not been attempted in the past to the best of our knowledge.

Dear Reviewer uNmD,

Thank you again for your positive assessment. In our response we clarify the meaning of "efficient", the step from Eq. 1 to Eq. 2 and the scale of the MSE in Fig. 2, add results for CNNs (ResNet-18 on CIFAR-100, 3 seeds). As the discussion period ends soon, we would be grateful for your feedback, and we are happy to clarify anything further. If our response addresses your questions, we would appreciate it if you would consider updating your score.

Sincerely,

The Authors.

Thanks for your detailed reply. All my concerns are addressed. I will maintain my score.

Reviewer_yM1EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Rationally Linearized MLPs (RaLiMPs), a replacement for feedforward layers based on separable approximations of rational activation surrogates. The surrogate is decomposed into a polynomial part and irreducible quadratic partial fractions. Polynomial terms use EUGen-style features, while rational terms are represented through Laplace integrals, discretized using Gauss–Laguerre quadrature, and estimated using real and imaginary components of complex Gaussian features. The resulting weight factors can be fused with a subsequent linear layer; a sketched Hadamard-product construction extends the approach to gated feedforward blocks.

The paper derives per-frequency unbiasedness, second-moment formulas, and an approximation-error decomposition. It evaluates the method against SNNK, EUGen, and other baselines across synthetic approximation, vision transformers, GPT-2, Qwen3-8B, point-cloud segmentation, and coordinate networks. The reported results suggest useful compression and inference gains, with accuracy retention varying across tasks and compression levels.

Critical strengths
  1. The rational construction preserves the separability needed for efficient deployment. The partial-fraction and Laplace representation provides a concrete way to incorporate rational activation surrogates while retaining the separation between inputs and weights required for layer fusion (Sections 2.3–2.4 and Appendix C.4). This is a useful extension of existing kernelized MLPs: directly dividing separate approximations of a numerator and denominator would not generally preserve the same inner-product representation. The per-frequency unbiasedness and exact second-moment calculations also provide useful information about the estimator’s behavior and its dependence on input norms.

  2. The evaluation provides meaningful evidence across several model families. The experiments include both training from scratch and adapting pretrained models, as well as transformers and coordinate networks. Several comparisons with the closest kernelized-MLP baselines are favorable. For example, at k=22, PTv3 obtains 77.33 ScanNet mIoU versus 77.57 for the dense model, while the summed MLP latency decreases from 12.31 to 6.44 ms (Tables 2–3). For GPT-2 at k=30, perplexity is 15.69 versus 16.44 for SNNK and 18.42 for EUGen (Table 17). These results support the potential practical value of the proposed layer beyond a single benchmark.

Critical weaknesses
  1. Several important experimental results do not reconcile with the stated protocols. Appendix D.6.1 specifies 30 problems for each AIME benchmark, and D.6.5 defines pass@16 as the fraction of problems for which at least one of 16 generated answers is correct. Under this definition, one evaluation run must produce a percentage of the form 100s/30 for an integer s. However, Table 1 reports AIME24 pass@16 values of 86.0 and 79.7, and AIME25 values of 77.5 and 75.3, for the two RaLiMP configurations. These values are incompatible with that protocol even after rounding to one decimal place. Independently, MATH500 is described as single-sample evaluation over 500 problems, so accuracy changes in increments of 0.2 percentage points; the reported 94.5 for RaLiMP at k=8 and 89.9 for SNNK at k=4 cannot arise under that description. An unstated average over evaluation runs or a different estimator could explain these discrepancies, but the manuscript does not document either.

There is also an unresolved issue in the synthetic experiment. Appendix D.1 specifies independent Uniform[0,1] entries for 256-dimensional inputs and weights, implying an expected preactivation of 64. This places tanh and sigmoid in their saturated regime. Even a simple rank-one linear predictor of the resulting near-constant target has an expected MSE of approximately 0.0013, whereas the low-rank errors in Figure 2 are approximately 0.2–0.5. Missing preprocessing or baseline constraints therefore need clarification. Together, these issues prevent a reliable interpretation of key evidence for approximation quality and preserved LLM capability.

  1. The claimed separation of approximation accuracy and cost is stronger than the established theory. The discussion on p. 5 and Theorem 3 in Appendix A.4 suggest that increasing the number of poles improves approximation accuracy without increasing variance, and that the rational method has the asserted accuracy–cost scaling. However, Corollary 1(ii) explicitly depends on pole widths and coefficients through A²/q⁴+B²/q² and an exponential factor involving 1/q. Combining multiple atoms also requires controlling their aggregate variance, while the total feature width F=1+Km+4J n_freq m grows with both the number of atoms and frequencies.

For a fixed rational surrogate and quadrature scheme, increasing m reduces the random estimation error but generally leaves the rational-fit and quadrature errors unchanged. Achieving a smaller total error may require changing the poles, coefficients, cutoff, or frequency count; their dependence on the target accuracy is not tracked in Theorem 3. Consequently, the claimed total accuracy–cost advantage over polynomial linearization is not established by the current argument. This matters because it is a central theoretical justification for the method, although it does not invalidate the basic separable feature construction. Corollary 2 also needs correction: conditioning or truncating Gaussian projections to obtain bounded variables generally changes their mean, so the stated Hoeffding argument requires an appropriate bias and failure-event treatment.

Questions for authors
  1. Clarification of the LLM evaluation protocol. What exact problem counts, numbers of generated samples, numbers of independent evaluation runs, and aggregation formulas produced Table 1? Please explain how the reported AIME pass@16 and MATH500 single-sample percentages arise under the definitions in D.6.1 and D.6.5. Per-problem correctness records, or the corresponding integer counts together with the aggregation code, would directly resolve this concern.

  2. Clarification of the synthetic benchmark. What exact centering, scaling, normalization, bias treatment, and target construction were used for Figure 2? Under the distribution written in D.1, the expected preactivation is 64 and tanh/sigmoid are saturated. How does this setup produce the reported low-rank errors, given the much smaller error attainable by a simple rank-one predictor? Please clarify the baseline parameterization and provide the relevant data-generation and evaluation procedure.

  3. Clarification and correction of the theoretical guarantee. In Theorem 3, which quantities are held fixed as the target error decreases? Can the claimed result be stated for total approximation error and total feature width, explicitly accounting for rational-fit error, quadrature error, pole widths, coefficient magnitudes, and frequency count? Please also clarify how Corollary 2 handles the change in expectation and exceptional event introduced by bounding Gaussian projections.

  4. Clarification of the implemented input scale. Equation (33) prescribes τ² = c RMS(w·x), and Appendix C.1 explicitly explains why scaling τ itself this way is different. However, the experimental overview and Appendix D.2 state τ = c RMS(w·x), including τ = 4 RMS(w·x) for ViT. Which formula was actually used to generate the reported results? Since this changes both the approximated kernel and its variance, a precise implementation description is necessary.

4 comments

We thank the reviewer for their reviews.

Table 1.

Thank you for checking these. The revised Table 1 is recomputed; the counts behind each cell are below.:

  • problems per benchmark: AMC23 83, AIME24 30, AIME25 30, GPQA 198, GSM8K 1,319, MATH500 500, Minerva 272, OlympiadBench 674;
  • 16 samples per problem for AMC23, AIME and GPQA, one sample for the other benchmarks, and one evaluation run per model;
  • avg@16 = (correct samples) / (16 × problems), pass@16 = (problems with at least one correct sample) / (problems), single-sample accuracy = (correct answers) / (problems).

The counts behind every cell of the revised Table 1 (for AMC23, AIME and GPQA: correct samples / solved problems):

AMC23 (/1328, /83)AIME24 (/480, /30)AIME25 (/480, /30)GPQA (/3168, /198)GSM8K (/1319)MATH500 (/500)Minerva (/272)Olympiad (/674)
k = 4
SNNK1012 / 73258 / 21228 / 201527 / 1591221448110447
EUGen1050 / 75275 / 22237 / 211578 / 1601227451117438
RaLiMP1092 / 80293 / 26251 / 241762 / 1691260464124475
k = 8
SNNK880 / 70215 / 20190 / 181356 / 1541191437103421
EUGen851 / 68222 / 21197 / 191121 / 141116643181340
RaLiMP1043 / 78279 / 24233 / 231619 / 1661256472118464

The conclusions are unchanged: RaLiMP is the best method on every benchmark at both k.

Synthetic benchmark.

We have added the clarifications about the dataset: Pre-activations are centered, and inputs and targets are standardized per coordinate, so MSE = 1 is the mean predictor; the revised App. D.1 states the full protocol. The target is therefore not near-constant, and the rank-one estimate does not apply. The low-rank baseline is the rank-r map VUx with no bias, and its errors are those of the optimal rank-r map: the trained model matches the closed-form reduced-rank regression solution on the same data (held-out MSE 0.49 vs. 0.50 at r = 16, GELU). The remaining error comes from the nonlinearity: even the full 256×256 linear map leaves 0.38 for GELU. Fig. 2 now reports held-out MSE.

Theorem 3 and Corollary 2.

As the reviewer notes, neither point affects the construction. Lemma 1 (unbiasedness), Prop. 1 (exact second moments), Cor. 1 (variance per atom) and Theorem 1 (total error) are unchanged. Theorem 3 and Corollary 2 are supporting results, and the revision makes their scope explicit.

Theorem 3. We restate it for a fixed surrogate and a fixed quadrature, and let only the number of random features vary; this answers the question of what is held fixed. The notation is that of the paper:

  • the surrogate is R=P+∑j=1JgjR=P+\sum_{j=1}^{J}g_{j}, with P(t)=∑n=0KcntnP(t)=\sum_{n=0}^{K}c_{n}t^{n} and JJ atoms gj(t)=Aj+Bj(t−pj)(t−pj)2+qj2g_{j}(t) = \frac{A_{j}+B_{j}(t-p_{j})}{(t-p_{j})^{2}+q_{j}^{2}} (Eq. 14);
  • the quadrature uses nfreqn_{\rm freq} nodes sk≤smaxs_{k} \le s_{\text{max}};
  • each block uses m random projections, so the total width is F=1+Km+4JnfreqmF=1+Km+4Jn_{\rm freq}m (Sec. 2.4);
  • εrat=sup⁡∣t∣≤T∣f(t)−R(t)∣\varepsilon_{\rm rat}=\sup_{|t|\le T}|f(t)-R(t)| and εquad=sup⁡∣t∣≤T∣g(t)−g^(t)∣\varepsilon_{\rm quad}=\sup_{|t|\le T}|g(t)-\widehat g(t)| are the rational-fit and quadrature errors of Theorem 1, where g^\hat{g} is gg with its Laplace integral replaced by the retained nodes;
  • cA,j=∣Aj∣/qj2c_{A,j}=|A_{j}|/q_{j}^{2} and cB,j=∣Bj∣/qj2c_{B,j}=|B_{j}|/q_{j}^{2} are the coefficient scales of Cor. 1(ii).

Here τ\tau is the input scale of App. C.1 (Eq. 33). Consider inputs x and weight rows ww with t=w⋅x∈[−T,T]t=w\cdot x \in [-T,T] and ∥x+w∥2/τ2≤Λ\|x+w\|^2/\tau^2\le\Lambda for a constant Λ\Lambda, and let Φ\Phi, Ψ\Psi be the RaLiMP feature maps (Eq. 4). Define

Γrat=∑j=1J(cA,j2+cB,j2)1/2(esmax⁡Λ/qj−1)1/2,Γpoly=∑n=1K∣cn∣ sup⁡((∥x∥2∥w∥2+2t2)n−t2n)1/2,\Gamma_{\rm rat}=\sum_{j=1}^{J}\big(c_{A,j}^2+c_{B,j}^2\big)^{1/2}\big(e^{s_{\max}\Lambda/q_j}-1\big)^{1/2},\qquad \Gamma_{\rm poly}=\sum_{n=1}^{K}|c_n|\,\sup\big((\|x\|^2\|w\|^2+2t^2)^n-t^{2n}\big)^{1/2},

where the supremum is over these inputs. Γrat\Gamma_{\rm rat} comes from Cor. 1(ii) and Γpoly\Gamma_{\rm poly} from Prop. 2. The revised Theorem 3 states that, for every such input, the L2L^{2} error (E[(⟨Φ(x),Ψ(w)⟩−f(t))2])1/2\big(\mathbb E[(\langle\Phi(x),\Psi(w)\rangle-f(t))^2]\big)^{1/2}, with the expectation over the random projections, is at most

εrat+εquad+Γpoly+Γratm,\varepsilon_{\rm rat}+\varepsilon_{\rm quad}+\frac{\Gamma_{\rm poly}+\Gamma_{\rm rat}}{\sqrt m},

so any target error ε>εrat+εquad\varepsilon>\varepsilon_{\rm rat}+\varepsilon_{\rm quad} is reached with total width

F=1+(K+4Jnfreq)⌈(Γpoly+Γrat)2(ε−εrat−εquad)2⌉.F=1+(K+4Jn_{\rm freq})\Big\lceil\frac{(\Gamma_{\rm poly}+\Gamma_{\rm rat})^2}{(\varepsilon-\varepsilon_{\rm rat}-\varepsilon_{\rm quad})^2}\Big\rceil .

The bound depends on each quantity the reviewer asks about: the rational-fit error, the quadrature error, the pole widths qjq_{j}, the coefficients through cA,jc_{A,j} and cB,jc_{B,j}, and the number of frequencies nfreqn_{\rm freq}. The comparison with polynomial features is supported by Prop. 2, which is unchanged, and by the experiments in Fig. 2 and Tables 13–14.

Corollary 2. As in Prop. 1, the estimator of node sks_{k} is C^k=1m∑r=1mYr\widehat{C}_{k}=\frac{1}{m}\sum_{r=1}^{m}Y_{r}, where each summand YrY_{r} depends on the rr-th Gaussian projection ωr∼N(0,Id)\omega_r\sim\mathcal N(0,I_d) only through $\xi_{r} = \sqrt{s_{k}/2}\omega_{r}\cdot(x+w)/\tau. Fix δ∈(0,1)\delta \in (0,1) and let β=2log⁡(2m/δ)\beta=\sqrt{2\log(2m/\delta)}. The bounded-projection condition is now the event

E={∣ωr⋅(x+w)∣≤β∥x+w∥ for all r},P(Ec)≤δ/2.\mathcal E=\{|\omega_r\cdot(x+w)|\le\beta\|x+w\|\ \text{for all } r\},\qquad \mathbb P(\mathcal E^c)\le\delta/2 .

On E\mathcal E every summand is bounded by M=exp⁡(β∥x+w∥smax⁡/2/τ)M=\exp\big(\beta\|x+w\|\sqrt{s_{\max}/2}/\tau\big), the same constant as in the original statement. Truncating to E\mathcal E shifts the mean of each summand; assuming β≥smax⁡/2 ∥x+w∥/τ\beta\ge\sqrt{s_{\max}/2}\,\|x+w\|/\tau, we bound this shift in closed form by M/(2m), using the tail bound P(g>z)≤12e−z2/2\mathbb P(g>z)\le\frac12e^{-z^2/2} for a standard normal g and z0. The corrected statement is

P[∣C^k−cos⁡(skt/τ2)∣≥ε+δM2m]≤δ2+2exp⁡(−mε22M2)for every ε>0,\mathbb P\Big[\big|\widehat C_k-\cos(s_kt/\tau^2)\big|\ge\varepsilon+\frac{\delta M}{2m}\Big]\le\frac{\delta}{2}+2\exp\Big(-\frac{m\varepsilon^2}{2M^2}\Big)\quad\text{for every }\varepsilon>0,

i.e. the same Hoeffding rate with an added bias term of order 1/m1/m.

Input scale.

All experiments use τ=c⋅RMS(w⋅x)\tau=c\cdot \text{RMS}(w\cdot x), applied to both towers, so the atom is evaluated at t/τ2t/\tau^{2}; Eq. 33 is corrected. It is now Equation 46.

Dear Reviewer yM1E,

Thank you again for checking the protocol and the theory so carefully. In our response we recompute Table 1 and provide the integer counts behind every cell, restate Theorem 3 for a fixed surrogate and quadrature with each quantity you asked about made explicit, correct Corollary 2, and fix the input scale. As the discussion period ends soon, we would be grateful for your feedback, and we are happy to clarify anything further. If our response addresses your concerns, we would appreciate it if you would consider updating your score.

Sincerely,

The Authors.

Dear Reviewer,

With the rebuttal window closing soon, please let us know if our rebuttal has answered your questions. We would appreciate it if the Reviewer can adjust their score if their concerns are answered.

Best,

The Authors.

Reviewer_qFw3OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Rationally Linearized Multilayer Perceptrons (RaLiMP), a method for reducing the cost of MLP layers by approximating their activation functions with low-degree rational functions. The rational functions are decomposed into polynomial and quadratic-fraction terms, then represented with separable input and weight features using random projections and frequency quadrature. The authors analyze the estimator’s unbiasedness, variance, and approximation error over bounded pre-activation intervals. The method is motivated by limitations in prior random-feature MLP linearization approaches: Fourier-based constructions can be difficult to apply directly to some activations, while polynomial-feature approaches may require higher degrees and incur greater variance or feature costs.

The paper evaluates RaLiMP on synthetic layer approximation, ViT image classification, PTv3 point-cloud segmentation, GPT-2 and Qwen3 language models, NeRF, SIREN, and a PINN for the Burgers equation. The authors report that RaLiMP generally performs competitively with or better than the compared low-rank, SNNK, and EUGen methods at similar feature or inference budgets, while reducing parameters, memory, or latency in several settings. The Qwen3-8B experiments also apply supervised fine-tuning and on-policy distillation after replacing feedforward layers.

Critical strengths

RaLiMP addresses a specific limitation of prior random-feature MLP linearization methods. Its rational representation is designed to provide accurate activation approximations with relatively few features, while retaining the separable form needed to reduce MLP computation. The paper supports this design with analyses of approximation error and variance, making the connection between the motivation and the proposed method explicit.

The evaluation spans several model families and tasks, and reports both task performance and efficiency measures. This breadth provides evidence that RaLiMP can be applied beyond a single architecture or benchmark, and helps assess the trade-off between model quality and inference cost.

Critical weaknesses

1.The paper’s positioning relative to prior rational-function methods could be clearer. It compares RaLiMP mainly with MLP linearization methods such as SNNK and EUGen, but does not directly compare against other rational activation or rational approximation approaches. As a result, it is difficult to determine whether the gains come specifically from the proposed partial-fraction random-feature construction, from using a rational surrogate in general, or from their combination.

2.The theoretical error guarantee concerns a single-layer approximation for an individual input and weight vector, with pre-activations restricted to a bounded interval. The paper does not analyze how approximation errors propagate or accumulate across multiple layers. The theory therefore supports the single-layer approximation, but does not by itself establish an end-to-end error guarantee for deep networks with multiple replaced layers.

3.The Qwen implementation differs from the feature construction analyzed theoretically. The experiments normalize inputs to the unit sphere, clip the random-feature exponent, and train the random projections. These modifications may affect the estimator’s bias and variance, but the paper does not quantify their effects or explain which theoretical guarantees remain applicable to this implementation. The Qwen results are difficult to attribute to RaLiMP alone because the compressed models undergo supervised fine-tuning and on-policy distillation, which substantially improves some long-reasoning results. The paper does not clearly establish whether EUGen and SNNK use the same data, training pipeline, and budget, or provide a complete comparison before and after distillation. In addition, several major experiments lack results across multiple random seeds or uncertainty intervals, and the synthetic experiment primarily reports training MSE.

Questions for authors

1.How does RaLiMP differ from prior rational activation and rational approximation methods, and can the authors provide a comparison that isolates the contribution of the proposed partial-fraction random-feature construction from the use of a rational surrogate itself?

2.Have the authors analyzed how approximation errors propagate or accumulate across layers? If not, could they report end-to-end output error as the number of replaced layers increases?

3.Could the authors provide separate Qwen ablations for input normalization, exponent clipping, and trainable random projections, quantify their effects on approximation error or variance, and clarify which theoretical results apply to the resulting implementation?

4.Do the Qwen3-8B EUGen and SNNK baselines use the same fine-tuning and distillation data, training budget, and checkpoint-selection criteria as RaLiMP? Could the authors report results before and after distillation for each method?

5.Could the authors report synthetic approximation error on independent test inputs and provide multiple-seed results or uncertainty intervals for the main downstream experiments? This would clarify the generalization and stability of the reported gains.

6 comments

We thank the reviewer for their reviews.

Relation to prior rational methods; surrogate vs. construction.

This is the central contribution, so we answer in detail.

Prior rational methods do not reduce cost. Rational functions have been used in two ways:

  • As activations inside a dense MLP (Telgarsky, 2017; Molina et al., 2020; Boullé et al., 2020; Delfosse et al., 2024; Tang & Townsend, 2026). These improve expressivity but keep the full d×4dd \times 4d layer.
  • As approximants of scalar functions (Padé; Newman, 1964; AAA, Nakatsukasa et al., 2018). These fit f(t)f(t) but say nothing about evaluating f(w⋅x)f(w \cdot x) as an inner product.

Prior linearizations are not rational. The only separable constructions for MLPs are Fourier-type (SNNK) and polynomial (EUGen) features. A polynomial cannot represent bounded, saturating activations at low degree and diverges away from its expansion point (Sec. 1, Fig. 2).

The obstacle and our solution. A rational P/QP/Q is not linearizable directly: dividing two separable estimates is not separable (as Reviewer yM1E also notes). RaLiMP removes the division. It decomposes the surrogate into partial fractions, writes each atom as a Laplace integral over cos(st)\text{cos}(st), discretizes it by Gauss–Laguerre quadrature, and estimates each frequency with Gaussian features (Sec. 2.3–2.4). A sum of separable blocks is separable, so the layer collapses into the next linear map. The estimator is unbiased (Lemma 1), has exact second moments (Prop. 1) and a total-error bound (Thm. 1). App. B shows that polynomial and Fourier constructions are special cases of the same spectral representation. To our knowledge this is the first linearization of rational activations.

Isolating the surrogate from the construction. Two experiments separate the rational surrogate from the separable construction.

  • Surrogate without the construction. We replace GELU in the last k layers of a dense ViT by our rational surrogate, with no random features, and train with the same recipe. This model keeps the full 4.72M-parameter MLP in each layer, yet it stays below the dense GELU model in every setting, and RaLiMP matches or exceeds it in five of the six settings with 17× fewer parameters per replaced layer. The rational activation alone therefore does not explain RaLiMP's results (see comment below).
  • Construction without the rational term. EUGen is the same kind of separable construction with only the polynomial block, at the same width. RaLiMP outperforms it in every setting of Fig. 2 and Table 13, so the rational term is what improves the linearized layer.

References :

Neural Networks and Rational Functions. Telgarsky, ICML 2017

Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks. Molina et al. ICLR 2020

Rational neural networks. Boullé et al., Neurips 2020

Adaptive Rational Activations to Boost Deep Reinforcement Learning. Delfosse et al. ICLR 2024

Rational Neural Networks have Expressivity Advantages. Tang & Townsend ICML 2026

The AAA Algorithm for Rational Approximation. Nakatsukasa et al. SIAM Journal on Scientific Computing Vol 40 2018.

Rational approximation to ∣x∣|x|. Newman 1964

Rational surrogate vs. construction.

Rational activations (PAU; Boullé et al.; Tang & Townsend) keep the dense MLP; RaLiMP makes them collapsible. A dense ViT with GELU replaced by our surrogate (no random features) reaches comparable accuracy to RaLiMP, at 17× more parameters per layer:

top-1CIFAR-100 k=4k=11ImageNet k=4k=11Places365 k=4k=11MLP params / layer
Dense, rational surrogate87.076.780.773.957.255.44.72M
RaLiMP89.880.480.873.958.055.20.27M

Error across layers.

Theorem 1 bounds one replaced layer at one input. We add the following end-to-end bound, with full proof, to the appendix.

Setting. Fix an input xx. For ℓ=1,…,L\ell=1,\dots,L, let FℓF_\ell be the reference layer and F^ℓ(⋅,ξℓ)\widehat F_\ell(\cdot,\xi_\ell) its RaLiMP approximation, where ξℓ\xi_\ell are the random features of layer ℓ\ell and F^ℓ\widehat F_\ell is jointly measurable. Let h0=h0=x, hℓ=Fℓ(hℓ−1)h_\ell=F_\ell(h_{\ell-1}), h^ℓ=F^ℓ(h^ℓ−1,ξℓ)\widehat h_\ell=\widehat F_\ell(\widehat h_{\ell-1},\xi_\ell) and eℓ=h^ℓ−hℓe_\ell=\widehat h_\ell-h_\ell. Let Dℓ−1D_{\ell-1} be a set that contains hℓ−1h_{\ell-1} and, with probability one, h^ℓ−1\widehat h_{\ell-1}. For a fixed z, write μℓ(z)=E[F^ℓ(z,ξℓ)]\mu_\ell(z)=\mathbb E[\widehat F_\ell(z,\xi_\ell)] and vℓ(z)=E∥F^ℓ(z,ξℓ)−μℓ(z)∥2v_\ell(z)=\mathbb E\|\widehat F_\ell(z,\xi_\ell)-\mu_\ell(z)\|^2, with the expectation over ξℓ\xi_\ell only.

Theorem. Assume that for every ℓ\ell:

  • (A1) ∥Fℓ(z)−Fℓ(z′)∥≤Kℓ∥z−z′∥\|F_\ell(z)-F_\ell(z')\|\le K_\ell\|z-z'\| for all z,z′∈Dℓ−1z,z'\in D_{\ell-1};
  • (A2) ξ1,…,ξL\xi_1,\dots,\xi_L are independent;
  • (A3) for every z∈Dℓ−1z\in D_{\ell-1}: E∥F^ℓ(z,ξℓ)∥2<∞\mathbb E\|\widehat F_\ell(z,\xi_\ell)\|^2<\infty, ∥μℓ(z)−Fℓ(z)∥≤Bℓ\|\mu_\ell(z)-F_\ell(z)\|\le B_\ell and vℓ(z)≤Vℓv_\ell(z)\le V_\ell.

Let r0=0r_{0}=0 and rℓ=(Kℓrℓ−1+Bℓ)2+Vℓr_\ell=\sqrt{(K_\ell r_{\ell-1}+B_\ell)^2+V_\ell}. Then

(E∥eL∥2)1/2≤rL≤∑ℓ=1LBℓ2+Vℓ ∏j=ℓ+1LKj.\big(\mathbb E\|e_L\|^2\big)^{1/2} \le r_L \le \sum_{\ell=1}^{L}\sqrt{B_\ell^2+V_\ell}\,\prod_{j=\ell+1}^{L}K_j .

If Bℓ=0B_\ell=0 for all ℓ\ell, then E∥eL∥2≤∑ℓ=1LVℓ∏j=ℓ+1LKj2\mathbb E\|e_L\|^2\le\sum_{\ell=1}^{L}V_\ell\prod_{j=\ell+1}^{L}K_j^2, and this bound is attained (identity layers with independent centered noise of variance VℓV_\ell). For every δ∈(0,1)\delta \in (0,1), P(∥eL∥>rL/δ)≤δ\mathbb P(\|e_L\|>r_L/\sqrt\delta)\le\delta.

Proof. Let Gℓ=σ(ξ1,…,ξℓ)\mathcal G_\ell=\sigma(\xi_1,\dots,\xi_\ell). By induction h^ℓ\widehat h_\ell is Gℓ\mathcal G_\ell-measurable, and by (A2) ξℓ\xi_\ell is independent of Gℓ−1\mathcal G_{\ell-1}. Write sℓ=(E∥eℓ∥2)1/2s_\ell=(\mathbb E\|e_\ell\|^2)^{1/2}, so s0=0s_{0}=0.

Fix ℓ\ell and put aℓ=μℓ(h^ℓ−1)−Fℓ(hℓ−1)a_\ell=\mu_\ell(\widehat h_{\ell-1})-F_\ell(h_{\ell-1}), which is Gℓ−1\mathcal G_{\ell-1}-measurable. Then

eℓ=aℓ+Nℓ,Nℓ=F^ℓ(h^ℓ−1,ξℓ)−μℓ(h^ℓ−1).e_\ell=a_\ell+N_\ell,\qquad N_\ell=\widehat F_\ell(\widehat h_{\ell-1},\xi_\ell)-\mu_\ell(\widehat h_{\ell-1}).

We use a standard fact: if XX is G\mathcal G-measurable, ξ\xiis independent of G\mathcal G and Φ≥0\Phi \ge 0 is jointly measurable, then E[φ(X,ξ)∣G]=Φ(X)\mathbb E[\varphi(X,\xi)\mid\mathcal G]=\Phi(X) with Φ(s)=E[φ(s,ξ)]\Phi(s)=\mathbb E[\varphi(s,\xi)]. This follows from Tonelli's theorem, since (X,1A)(X,\mathbf 1_A) and ξ\xi are independent for every A∈GA\in\mathcal G. Apply it with X=(h^ℓ−1,aℓ)X=(\widehat h_{\ell-1},a_\ell) and φ((z,c),ξ)=∥c+F^ℓ(z,ξ)−μℓ(z)∥2\varphi((z,c),\xi)=\|c+\widehat F_\ell(z,\xi)-\mu_\ell(z)\|^2. For fixed z∈Dℓ−1z\in D_{\ell-1} and cc, the vector N=F^ℓ(z,ξℓ)−μℓ(z)N=\widehat F_\ell(z,\xi_\ell)-\mu_\ell(z) has mean zero and E∥N∥2=vℓ(z)<∞\mathbb E\|N\|^2=v_\ell(z)<\infty, so expanding the square gives Φ(z,c)=∥c∥2+vℓ(z)\Phi(z,c)=\|c\|^2+v_\ell(z); every term is integrable. Hence

E[∥eℓ∥2∣Gℓ−1]=∥aℓ∥2+vℓ(h^ℓ−1),sℓ2=E∥aℓ∥2+E vℓ(h^ℓ−1)≤E∥aℓ∥2+Vℓ.\mathbb E\big[\|e_\ell\|^2\mid\mathcal G_{\ell-1}\big]=\|a_\ell\|^2+v_\ell(\widehat h_{\ell-1}),\qquad s_\ell^2=\mathbb E\|a_\ell\|^2+\mathbb E\,v_\ell(\widehat h_{\ell-1})\le\mathbb E\|a_\ell\|^2+V_\ell .

The cross term vanishes because the noise of layer ℓ\ell has mean zero at every input, and the input h^ℓ−1\widehat h_{\ell-1} depends only on the earlier layers. Almost surely h^ℓ−1∈Dℓ−1\widehat h_{\ell-1}\in D_{\ell-1}, so

aℓ=(Fℓ(h^ℓ−1)−Fℓ(hℓ−1))+(μℓ(h^ℓ−1)−Fℓ(h^ℓ−1)),∥aℓ∥≤Kℓ∥eℓ−1∥+Bℓ,a_\ell=\big(F_\ell(\widehat h_{\ell-1})-F_\ell(h_{\ell-1})\big)+\big(\mu_\ell(\widehat h_{\ell-1})-F_\ell(\widehat h_{\ell-1})\big),\qquad \|a_\ell\|\le K_\ell\|e_{\ell-1}\|+B_\ell ,

by (A1) and (A3). Minkowski's inequality gives (E∥aℓ∥2)1/2≤Kℓsℓ−1+Bℓ(\mathbb E\|a_\ell\|^2)^{1/2}\le K_\ell s_{\ell-1}+B_\ell, hence sℓ≤gℓ(sℓ−1)s_\ell\le g_\ell(s_{\ell-1}) with gℓ(s)=(Kℓs+Bℓ)2+Vℓg_\ell(s)=\sqrt{(K_\ell s+B_\ell)^2+V_\ell}. Since gℓg_\ell is nondecreasing and s0=r0=0s_{0}=r_{0}=0, induction gives sℓ≤rℓs_\ell\le r_\ell, the first inequality.

For the second, (a+B2+v)2−(a+b)2≥0(a+\sqrt{B^{2}+v})^{2} - (a+b)^{2} \ge 0 for a,B,V≥0a, B, V \ge 0, so rℓ≤Kℓrℓ−1+Bℓ2+Vℓr_\ell\le K_\ell r_{\ell-1}+\sqrt{B_\ell^2+V_\ell}; unrolling from r0=0r_{0}=0 gives the sum. If all Bℓ=0B_\ell=0, then rℓ2=Kℓ2rℓ−12+Vℓr_\ell^2=K_\ell^2r_{\ell-1}^2+V_\ell exactly, and unrolling gives the stated bound. Finally, by Markov's inequality on ∥eL∥2\|e_L\|^2, P(∥eL∥>rL/δ)≤δ E∥eL∥2/rL2≤δ\mathbb P(\|e_L\|>r_L/\sqrt\delta)\le\delta\,\mathbb E\|e_L\|^2/r_L^2\le\delta (if rL=0r_{L}=0, then eL=0e_{L}=0 almost surely). ■\blacksquare

Application to RaLiMP. Consider a replaced layer Fℓ(z)=Wℓof(Wℓiz)F_{\ell}(z)=W^{o}_{\ell} f(W^{i}_\ell z) with rows w1,…,wlℓw_{1},\dots,w_{l_{\ell}} of WℓiW^{i}_{\ell}. Its RaLiMP version is F^ℓ(z)=WℓoR^(z)\widehat{F}_{\ell}(z)=W^{o}_{\ell}\widehat{R}(z), where R^i(z)\widehat{R}_{i}(z) is the RaLiMP estimate of the hidden unit f(wi⋅z)f(w_{i} \cdot z). Assume the inputs of the layer are normalized, ∥z∥=1\|z\|=1. Then ∣wi⋅z∣≤∥wi∥|w_i\cdot z|\le\|w_i\| and ∥z+wi∥≤1+∥wi∥\|z+w_i\|\le1+\|w_i\| for every input, so Theorem 1 and Cor. 1 hold uniformly: each unit has bias at most εrat+εquad\varepsilon_{\rm rat}+\varepsilon_{\rm quad} and variance at most some Vˉℓ\bar V_\ell. Hence (A3) holds with

Bℓ=∥Wℓo∥2lℓ (εrat+εquad),Vℓ=∥Wℓo∥22 lℓ Vˉℓ,B_\ell=\|W^{o}_{\ell}\|_{2}\sqrt{l_{\ell}}\,(\varepsilon_{\rm rat}+\varepsilon_{\rm quad}),\qquad V_{\ell}=\|W^{o}_{\ell}\|_{2}^{2}\,l_{\ell}\,\bar V_{\ell} ,

where ∥⋅∥2\|\cdot\|_2 is the spectral norm. The variance bound uses only E∥R^−ER^∥2=∑iVar(R^i)\mathbb{E}\|\widehat R-\mathbb E\widehat R\|^2=\sum_i\mathrm{Var}(\widehat R_i), so the units may share random features. If ff is λ\lambda-Lipschitz, (A1) holds with Kℓ=λ∥Wℓo∥2∥Wℓi∥2K_{\ell}=\lambda\|W^{o}_{\ell}\|_2\|W^{i}_{\ell}\|_2, and layers that are not replaced have Bℓ=Vℓ=0B_{\ell}=V_{\ell}=0. If the reference network uses the truncated surrogate P+g^P+\hat{g} in place of ff, the RaLiMP layers are unbiased and the sharper bound applies.

The bound is additive in the layers when Kj≈1K_{j} \approx 1, which is consistent with what we measured empirically. Replacing the last k FFNs of Qwen3-8B at initialization, the KL to the dense model is 0.15, 0.37, 0.78 and 1.47 for k=1,2,4,8k=1,2,4,8, i.e. it grows roughly linearly with no compounding blow-up (which is what the theory predicts).

k1248
KL / token0.150.370.781.47
Top-1 agreement89%86%77%63%

Fine-tuning then closes the gap (Table 1); Results in F.2.2 show the trained effect of kk.

Normalization, clipping, trainable projections.

  • Frozen projections.

  • We trained Qwen3-8B Stage 1 with every random projection frozen at its Gaussian initialization, with everything else as in the paper. Freezing changes accuracy by about 2 points and also slightly raises the training loss:

k=4 trainedk=4 frozenk=8 trainedk=8 frozen
AMC23 (avg@16)75.273.971.870.3
AIME24 (avg@16)51.050.842.339.6
AIME25 (avg@16)43.341.534.233.3
MMLU71.770.370.769.4
Final training loss0.8320.8410.8390.843
  • ViT-B/16. Fine-tuning with the projections frozen at their initialization.
ViT-B/16k=4 trainedk=4 frozenk=11 trainedk=11 frozen
CIFAR-10089.887.080.475.5
ImageNet-1k80.880.673.968.6
  • Normalization. On real Qwen3-8B activations (‖x‖ = 33–67), unnormalized features overflow on 39–99.9% of outputs; with unit-norm inputs, none do.
  • Clipping. It never activates on input features; with trained projections it activates on at most 8 of 1.77M weight-side features per layer, and without it training diverges.
  • Theory. Normalization is covered as stated: the results hold with xx, ww replaced by x/‖x‖x/‖x‖, w/‖w‖w/‖w‖, and the variance bound becomes uniform (‖x^+w^‖≤2‖x̂+ŵ‖ ≤ 2). The clip is a numerical safeguard, like gradient clipping, that never activates for this estimator. With trained projections the results describe the initialization.

Finally we would like to point out that these choices are standard practice for random-feature models. The spectral distribution of the random features is learned in kernel learning (Yang et al., 2015; Li et al., 2019) and in random-feature attention (Chowdhury et al., 2022), and learned feature maps replace random ones when pretrained Transformers are converted to linear attention (Kasai et al., 2021; Zhang et al., 2024; 2025), as in our Qwen3-8B setting. Performer (Choromanski et al., 2021) fine-tunes a pretrained Transformer after the swap, redraws its projections during training and uses a numerical stabilizer; RFA (Peng et al., 2021) ℓ2-normalizes queries and keys and notes that bounded features stabilize training; and Qin et al. (2022) stabilize kernel attention by normalization. Other types of truncations like Taylor polynomial approximations (Arora et al. 2024, Kacham et al. 2023) and locally sensitive sparse truncations are also common in the literature. (Chen et. al 2021).

References :

Rethinking attention with performers. Choromanski et al. ICLR 2021

Random Feature Attention. Peng et al. ICLR 2021

On Learning the Transformer Kernel. Chowdhury et al. TMLR 2022

The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry. Zhang et al. ICLR 2024

LoLCATs: On Low-Rank Linearizing of Large Language Models. Zhang et al. ICLR 2025

Finetuning Pretrained Transformers into RNNs. Kasai et al. EMNLP 2021

Implicit Kernel Learning. Li et al. AISTATS 2019

The Devil in Linear Transformer. Qin et al. EMNLP 2022

A la Carte – Learning Fast Kernels. Yang et al. AISTATS 2015

Simple linear attention language models balance the recall-throughput tradeoff. Arora et al. 2024

PolySketchFormer: Fast Transformers via Sketching Polynomial Kernels. Kacham et al. 2023

Scatterbrain: Unifying Sparse and Low-rank Attention Approximation. Chen et al. Neurips 2021

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.