acceptodds
Withdrawn from ICLR 2027

BEYOND 4D FFNS: HESSIAN POTENTIAL MLP FOR MEMORY AND LANGUAGE MODELING

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Position-wise feed-forward networks (FFNs) dominate Transformer cost, yet remain standard two-layer MLPs expanding to width before linear projection. We introduce the Hessian Feed-Forward Network (H-FFN), , expressing the sublayer as a sum of rank-one directional units derived from a scalar potential. For associative retrieval, a coaxial Memory-Track employs multi-scale contextual gating; for language modeling, Decoupled H-FFN separates content, routing, and emission. Under matched training budgets, Memory-Track reaches Needle-in-a-Haystack accuracy versus for a matched MLP, and Decoupled H-FFN achieves – perplexity gains across causal language modeling benchmarks ( on WikiText-103; on FineWeb-Edu). In transfer, analytic GELU matching improves GPT-2 fine-tuning () and accelerates ViT-B/16 convergence ( top-1 with fewer steps).

Primary area:
foundation or frontier models, including LLMs
Keywords:
Feed-forward networks, Hessian curvatures, Transformer architecture

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fZErOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Hessian Feed-Forward Network(H-FFN), which replaces 4d MLP with a sum of rank-one directional units derived from a scalar potential. The authors also develop H-FFN variants for memory tasks and language modeling tasks. Experiments show the superior performance of H-FFN over 4d MLP.

Critical strengths
  1. The paper presents a novel Hessian-based Transformer FFN, together with specific variants design for different tasks.
  2. The H-FFN shows strong performance on associative retrieval benchmarks, including NIAH and Gap Copy, demonstrating the potential of the proposed architecture for memory retrieval.
Critical weaknesses

While H-FFN performs strongly on associative retrieval tasks, the evidence for language modeling is largely insufficient to support the broader claim. In section 3.2, the size of the proposed model does not match the 4D baseline (5.25M vs. 4.92M). Therefore, where the gain comes from is unclear. More importantly, the language modeling experiments are conducted only at very small scale, which is not enough to demonstrate effectiveness in modern language modeling, where scalability is critical. A compute-matched scaling study, ideally following a Chinchilla-style compute-optimal setup, is needed to validate this claim. Section 3.3 is also under-justified: the authors replace only the final four layers without explaining this choice or providing ablations over the number or location of replaced layers.

Questions for authors
  1. See weakness.
  2. How does H-FFN compare against modern gated FFNs such as SwiGLU in LLaMA under matched parameter and compute budgets? The structural similarity between Channel-Split H-FFN and gated MLPs makes this comparison very important.
  3. How does H-FFN interact with the Muon optimizer?
3 comments
AuthorsOpenReview ↗

We add <=4.92M Morse Channel Split runs (399.0 PPL, 1.02x total FLOPs), three seed LLaMA SwiGLU baselines (582.4 PPL), a same stack SiLU gate ablation (415.9 vs. 450.3), GPT-2 layer depth ablations (L11 vs. L8-11 vs. all 12 layers), a 3000 step d=256 run (314.8 vs. 333.2), and Muon analysis, explaining how each control isolates the asked mechanism.

We thank the reviewer for noting the novelty of H FFN and raising constructive questions on LLaMA SwiGLU, compute matched scaling, GPT-2 layer selection, and Muon.

  1. Disentangling Params, Contextual Routing and LLaMA SwiGLU (Q2, W1): To separate whether gains stem from param count, 3 matrix SwiGLU layout, or ψ′′\psi'' gating, we designed a three step ablation under the exact Sec 3.2 flat cosine protocol (3000 steps):
  • Step A (SwiGLU baselines, n=3): Param matched LLaMA SwiGLU (dff=341d_{\text{ff}}=341, 4.92M, 16d216d^2 FFN FLOPs) gets 582.4 +/- 13.2 PPL and 4d SwiGLU (dff=512d_{\text{ff}}=512, 5.19M, 24d224d^2 FFN FLOPs) gets 498.8 +/- 11.8, both trailing Standard GELU (4.92M, 455.3 +/- 7.6). Thus a standard first order multiplicative gate hurts under this compact scratch regime.
  • Step B (Channel Split at <=4.92M via Morse): Pruning Legacy Channel Split from m=512 to 277 at step 1000 gives 4.89M params (34k fewer than Standard) and reaches 399.0 PPL (1.14x vs. Standard, 1.46x vs. LLaMA SwiGLU); Simplified 2 stage Morse (4.92M) reaches 411.4 PPL (native 5.25M in Table 2 is 388.0 +/- 2.3). Gains hold strictly at <=4.92M.
  • Step C (isolating ψ′′\psi'' inside Channel Split): Holding (U, A, V), Prefix MS ztz_t, Morse, and channel LRs identical at 4.92M, swapping only ψ′′(Az+b)\psi''(Az+b) for SiLU(Az+b)\text{SiLU}(Az+b) degrades PPL from 415.9 to 450.3. Together, Steps A to C prove both contextual (U, A, V) routing and bounded ψ′′\psi'' gating drive the gain.
  1. Compute Matched Advantage and Chinchilla Scope (W1):
  • Full model vs. trunk FLOPs (2 FLOPs/MAC): Standard 4D FFN uses 16d216d^2 FLOPs/token/layer vs. 26d226d^2 for native m=512 Channel Split (13d213d^2 MACs). Counting attention and the 32k tied LM head (64000d64000d FLOPs), Channel Split adds 31.2% trunk FLOPs and only 6.4% total FLOPs/step.
  • Winning under less compute: Native Channel Split hits 462.9 +/- 4.7 val PPL at step 2400 (85% total FLOPs) and 422.4 +/- 7.6 at step 2700 (96% total FLOPs), beating 3000 step Standard (468.0 +/- 16.0 val, 455.3 +/- 7.6 test). Pruning to m=277 at step 1000 (4.89M Morse) drops FFN cost below Standard (15.0d2<16d215.0d^2 < 16d^2/layer), hitting 452.2 val at step 2700 (92% total, 98.6% trunk FLOPs vs. Standard s0 final 486.1) and 399.0 test PPL at 3000 steps (1.02x avg total FLOPs).
  • Scaling scope (Chinchilla setup): We have not yet run a multi size Chinchilla iso FLOP scaling law sweep, which requires compute beyond this rebuttal; we will state this boundary in Sec 4.2. Within our compute, we add a 3000 step WT103 d=256 run (314.8 vs. 333.2 test) alongside the 100M to 150M token FineWeb Edu runs (1.06x to 1.15x) already in Sec 4.2.
  1. Ablating GPT-2 Replaced Layer Depth (W1): To test why replacing layers 8 to 11 works, we ran a depth sweep under the identical 200 step GPT-2 protocol (frozen attention):
  • Layer 11 only (4.7M trainable): 32.64 vs. 40.10 MLP PPL (1.23x).
  • Layers 8 to 11 (18.9M, Sec 3.3): 30.52 vs. 33.46 (1.10x; 500 steps: 27.60 vs. 29.65, 1.07x).
  • All 12 layers 0 to 11 (56.7M): 28.77 vs. 28.97 (1.01x). (Staged L11 to L8 swap gave 32.36 vs. 30.52 joint.) Late FFNs sit right below the LM head, where decoupling emission V gives maximal gain (1.23x at L11). Swapping earlier blocks compounds coordinate shifts across frozen downstream attention heads, narrowing relative margins (1.23x to 1.10x to 1.01x). Layers 8 to 11 (top third) balance capacity and frozen attention drift (to be added to Sec 3.3).
  1. Muon Optimizer Interaction (Q3): We have not yet tested H FFN with Muon; all reported experiments use AdamW for controlled baseline parity. Structurally, however, H FFN is directly compatible with Muon (Newton Schulz orthogonalization on 2D matrices):
  • Matrix eligibility and frame coverage: Core weights U,A,V∈Rm×dU, A, V \in \mathbb{R}^{m \times d} and readout (d×dd \times d) are 2D matrices eligible for Muon (keeping 1D biases b on AdamW). Because overcomplete rows (m>dm > d) lack orthogonality constraints, AdamW concentrates energy on active units (leaving lazy rows pruned by Morse), whereas Muon orthogonalizes updates to promote isotropic frame coverage.
  • Removing manual channel LRs: Under AdamW, bounded ψ′′(s)\psi''(s) compresses activation variance entering V, forcing asymmetric LRs (core/ctx at 4x, out at 16x). Since Muon normalizes 2D update singular values across matrices, testing whether Muon eliminates manual LR multipliers and shortens the (U, A, V) coordination delay at d=256 and d=384 is a priority we will highlight.

Thank you for the additional experiments. The response addresses some of my concerns, but my concerns remain:

  1. Parameter matching: Step B uses pruning, so it does not establish performance under a matched parameter budget from initialization.
  2. Scalability: The additional small-scale experiments do not address the need for a compute-matched evaluation across model scales. As noted in my original review, such evidence is essential to support the broader claim, although obtaining it may be beyond the scope of the rebuttal period.
  3. Clarity: The response references Section 4.2, which does not exist in the available manuscript.

I therefore maintain my original rating.

Thank you for the careful follow-up. We clarify all three points briefly below:

  1. Clarity (Section 4.2 Reference): In the submitted PDF, the FineWeb-Edu, d=256, and d=384 results appear in Section 5 (Discussion, page 9). The "Section 4.2" label came from an unsubmitted revision outline reorganized to address Reviewer 1's structural feedback. We apologize for the mismatch.

  2. Parameter Matching and Morse Pruning: You are right that Step B starts at m=512 (5.25M) before pruning to m=277 (4.89M) at step 1000. This actually reflects Morse persistence (pj=∥uj∥2∥vj∥2p_j=\|u_j\|_2\|v_j\|_2 on rank-one units λj(uj⊤x)vj\lambda_j(u_j^\top x)v_j), which drops degenerate directions and restricts the last 2000 steps to a 4.89M subspace, a property standard dense FFNs lack. Steps A and C show that initial width m or parameter count is not the sole driver: widening SwiGLU (Step A) fails to match Standard, and under identical 6d→512→2776d \to 512 \to 277 width and pruning trajectories (Step C), ψ′′\psi'' still outperforms SiLU.

  3. Scalability: In the revision, we will tighten our claims to the validated scales (compact scratch tasks and 86M/124M transfer) and explicitly note multi-scale Chinchilla iso-compute pre-training as a limitation and next step.

Reviewer_FyqPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes H-FFN, which uses the second derivative of a scalar potential as a gating nonlinearity. The architecture also incorporates contextual aggregation and decoupled content, routing, and output projections. Experiments cover associative retrieval, language modeling, and pretrained transfer.

Critical strengths
  • The gate has a simple closed-form implementation without explicitly constructing a Hessian.
  • The paper reports promising improvements on small-scale retrieval and language-modeling benchmarks and includes preliminary component ablations.
Critical weaknesses
  • The empirical evaluation is dominated by very small models and short training budgets. Evidence at larger widths is limited and includes worse test perplexity at d=384d=384, leaving scalability unclear.
  • The benefit of the proposed nonlinearity is not isolated from contextual aggregation, projection decoupling, and optimization changes. Matched SwiGLU/GEGLU baselines using the same surrounding architecture and training recipe are missing.
  • The claim of exact parameter parity conflicts with Table 2, which compares a 4.92M-parameter baseline against a 5.25M-parameter model. This weakens the fairness of the comparison.
  • The pretrained transfer experiments rely on a GELU-equivalent warm-start and do not establish the advantage of native curvature gating.
Questions for authors

Please see the weakness

2 comments
AuthorsOpenReview ↗

We explain why our 5M to 14M scratch regime and pretrained transfer tests are scientifically valid rather than toy probes, add a 3000 step d=256 WT103 run (314.8 vs. 333.2), three seed SwiGLU baselines (582.4), an isolated SiLU gate ablation (415.9 vs. 450.3), and <=4.92M Morse Channel Split runs (399.0 PPL).

We thank the reviewer for noting our closed form gate without explicit Hessians and our component ablations, and we address the concerns on scale, transfer design, nonlinearity isolation, and parameter parity in priority order.

  1. Why Our Scratch Regime Is Not a Toy Model and Scaling Evidence: While multi billion token pretraining is valuable, the 5M to 14M scratch regime is not a toy model: fundamental FFN optimization dynamics and capacity bottlenecks do not invert simply by shrinking width. Even though flat cosine LR makes Standard FFN much stronger (improving it from 565 to 455.3 PPL), Standard FFN still hits an early plateau that Channel Split consistently overtakes across widths and corpora:
  • WT103 d=256: Beyond the 500 step check in Sec 4.2 (2833 to 2048, 1.38x), our new 3000 step run reaches 314.8 vs. 333.2 test PPL (296.0 vs. 321.5 val).
  • FineWeb Edu (in Sec 4.2): 1.11x to 1.15x test PPL gains at d=128 (100M tokens, about 20 tokens/param) and 1.06x at d=256 (150M tokens: 298 to 281).
  • Why d=384 shows better val PPL (67.1 vs. 72.1, 1.07x) yet higher single seed test PPL (104.0 vs. 100.5): Training three separate (U, A, V) matrices from scratch incurs a coordination delay, crossing Standard at 20% steps for d=128, 87% for FineWeb d=256, and 90% (step 9500 of 10570) for d=384. On word level WT103, this late crossing leaves a short cosine decay tail and rare word holdout variance. We report d=384 candidly as a scaling boundary for optimizer refinement.
  1. Significance of Pretrained Transfer (Sec 3.3): Pretrained transfer is not meant to test native ψ′′\psi'' in disguise; rather, scratch training (Sec 3.1 to 3.2) validates native ψ′′\psi'', while Sec 3.3 tests whether rank one (U, V) decoupling improves frozen 100M scale GELU backbones without losing step 0 knowledge. Dropping native ψ′′\psi'' into GPT-2 breaks step 0 parity (L11 val PPL jumps from 48.8 to 128), whereas λ(s)=GELU(s)/s\lambda(s)=\text{GELU}(s)/s preserves step 0 exactly (MSE 1.3e-5). Fine tuning is not equivalent to MLP FT: decoupling UU and VV updates cuts GPT-2 PPL from 29.65 to 27.60 (layers 8 to 11; 40.10 to 32.64 on L11) and lifts ViT-B/16 to 79.0% in 1k steps vs. 76.6% in 3k. A 200 step hybrid probe adding native modulation matches pure GELU/s (32.67 vs. 32.64), confirming native ψ′′\psi'' drives scratch gains while (U, V) decoupling drives warm start transfer. We will state this clearly in Sec 3.3.

  2. Isolating ψ′′\psi'' Nonlinearity and Matched SwiGLU Baselines: To isolate ψ′′\psi'' from Prefix MS context ztz_t, (U, A, V) decoupling, and LR groups, we ran two new 3000 step WT103 controls under the exact Sec 3.2 flat cosine protocol:

  • Gate Only Ablation (4.92M): Holding (U, A, V), Prefix MS ztz_t, Morse pruning (6d to 512 to 277), and channel LRs (core/ctx at 4x, out at 16x, readout at 0.25x) identical, changing only λ=ψ′′(Az+b)\lambda=\psi''(Az+b) to first order λ=SiLU(Az+b)\lambda=\text{SiLU}(Az+b) degrades test PPL from 415.9 to 450.3 (val 415.6 to 472.5). This 34.4 test PPL (56.9 val) gain comes solely from ψ′′\psi'' (central excitation plus negative damping wells). Consistently, Table 1a already shows Base Symmetric H FFN (no ztz_t, no split, no group LRs) lifts NIAH from 80.9% to 88.1%.
  • Matched Dense SwiGLU (n=3): LLaMA SwiGLU (dff=341d_{\text{ff}}=341, 4.92M) gets 582.4 ± 13.2 PPL and 4d SwiGLU (dff=512d_{\text{ff}}=512, 5.19M) gets 498.8 ± 11.8, both trailing Standard GELU (455.3 ± 7.6). (Sec 4.1 also covers GeGLU/SwiGLU transfer.)
  1. Strict Parameter Parity (<=4.92M): In the revision, we will consistently frame Table 2 as tightly controlled budgets (6.7% total margin, since 4.10M params sit in shared tied embeddings) and add our new 3000 step Morse Channel Split runs at <=4.92M:
  • Legacy Channel Split + Morse (m=512 pruned to 277 at step 1000, 4.89M params, 34k fewer than Standard): 399.0 test PPL (1.14x vs. Standard 455.3; 1.46x vs. LLaMA SwiGLU 582.4).
  • Simplified Channel Split + two stage Morse (4.92M): 411.4 test PPL. Wider two matrix Decouple (5.28M: 488.4) does worse, ruling out parameter count.
Reviewer_FyqPOpenReview ↗

Reply By Reviewer FyqP

Thanks for the reply. I'm not convinced of the performance, given that the training tokens are even less than 1B and the PPLs are over 100. I suggest that the authors scale up the experiments a bit. I will keep my score.

Reviewer_XYoROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors tackle the computational cost of Transformer models, where feed-forward layers account for a large part of the cost of training and inference. They therefore propose replacing these layers with sums of directional projections. Using a scalar potential whose Hessian can be applied to the input, they avoid constructing a dense Hessian matrix. Each unit projects the input onto a learned direction and multiplies that projection by a gate derived from the second derivative of a scalar function. For retrieval, the proposed Memory-Track shares input and output directions and computes gates from multiscale context summaries, whereas for language modeling, the authors introduce separate matrices for extracting content, computing gates, and producing outputs. They report improved synthetic retrieval accuracy and lower WikiText-103 perplexity in small models, along with additional experiments on FineWeb-Edu, translation, GPT-2 fine-tuning, and vision transfer.

Critical strengths
  • The basic Hessian derivations seem to be sound and correct.
  • The authors avoid constructing a dense Hessian matrix, which saves the computation and memory needed to explicitly form and store it.
  • Separating content, routing, and emission seems reasonable because the features being extracted do not necessarily need to be the same features used to decide when to transmit them.
  • The NIAH improvement seems meaningful because the gain is large and consistent across the reported seeds.
  • Reporting validation perplexity over 3,000 training steps helps track the improvements and training behavior better than reporting only a final number.
  • The vision transformer result is promising because the proposed setup reaches higher accuracy in fewer training steps.
Critical weaknesses
  • The writing is hard to follow because the paper uses so much jargon, and at times the sentences seem almost disconnected, with poor flow between them.
  • The paper does not have an actual parameter-matched baseline, and the parameter-parity claim is false from the architecture itself, even without looking at the numbers in Table 2.
  • The experiments use very small models and older pretrained backbones, so it is hard to tell whether the improvements would hold for modern language models.
  • The channel split adds computation per step, so matching training steps does not mean matching FLOPs, and the claimed advantage under matched training budgets is not established.
  • The main scratch experiments only compare against GELU MLPs and not the most relevant baselines, such as parameter-matched SwiGLU and GeGLU, despite mentioning them.
  • The implementation does not compute the exact contraction defined in Equation 4 because it uses u_j^\top x+b_j rather than u_j^\top x, so the claimed identity needs to be corrected.
  • The learned directions are not constrained to be orthogonal, so interpreting them as eigenvectors and the gates as eigenvalues is not justified.
  • The best-performing language-modeling variant is generally not a Hessian operator because independent input and output projections can produce a nonsymmetric matrix, whereas a scalar potential's Hessian is symmetric.
Questions for authors

see weaknesses; id like the authors to address each of them.

2 comments
AuthorsOpenReview ↗

We state the paper structure and non orthogonal dictionary setup already in Sec 2.1, walk through the Eq. 4 calculus and full model FLOP math, add ≤4.92M Morse Channel Split runs (399.0 PPL), three seed SwiGLU baselines (582.4), and a same architecture SiLU gate ablation (415.9 vs. 450.3), and note planned revisions.

We thank the reviewer for noting the soundness of our Hessian derivation, the (U, A, V) intuition, and our NIAH and ViT gains, and for engaging closely with the equations.

  1. Writing and Structure: Terms like Hessian contraction and Morse persistence follow differential geometry usage. As organised in the paper, Sec 2 derives Hessian contraction (2.1), decouples (U, A, V) for causal rotation (2.2), adds Morse pruning to control params and test sparsity (2.3), and derives step 0 MLP equivalence for rapid large model transfer without full pre-training (2.4). Sec 3 tests each branch (3.1 recall, 3.2 LM, 3.3 transfer), and Sec 4 isolates limits (4.1) and scaling (4.2). We will pair geometric terms with brief intuition.

  2. Parameter Parity (<=4.92M): We will revise any exact parity wording to tightly controlled budgets (5.25M vs. 4.92M, a 6.7% margin, with 4.10M params in shared tied embeddings). New 3000 step Morse runs match or undercut 4.92M:

  • Legacy Channel Split + Morse (m=512 to 277 at step 1000, 4.89M params, 34k fewer than Standard): 399.0 test PPL vs. Standard 455.3 +/- 7.6 (4.92M).
  • Simplified Channel Split + two stage Morse (6d to 512 to 277, 4.92M): 411.4 to 415.9 PPL. Wider two matrix Decouple (5.28M: 488.4) is worse, confirming gains come from (U, A, V) routing rather than param count.
  1. Scale and Modern Backbones: Sec 4.2 already reports scratch scaling across widths (WT103 d=256 at 500 steps: 2833 to 2048; d=384: 72.1 to 67.1 val) and FineWeb Edu (100M to 150M tokens: 1.06x to 1.15x). We also add a 3000 step WT103 d=256 run (314.8 vs. 333.2 test). For transfer, λ(s)=GELU(s)/s\lambda(s)=\text{GELU}(s)/s maps two matrix GELU blocks at step 0, making GPT-2 (124M, 1.07x) and ViT-B/16 (86M, 79.0% in 1k steps vs. 76.6% in 3k) the exact testbeds.

  2. FLOPs and Compute Advantage: Separating FLOP math, crossover, and runtime:

  • Per step FLOPs (2/MAC): Standard 4D FFN uses 16d216d^2 FLOPs/token/layer vs. 26d226d^2 for m=512 Channel Split (13d213d^2 MACs). With attention and the 32k tied LM head (64000d64000d FLOPs), Channel Split adds 31.2% trunk FLOPs and just 6.4% total FLOPs/step. (Eager runtime 3368s vs. 1664s reflects unfused Python pooling, not FLOPs.)
  • Less compute crossover: Native Channel Split hits 462.9 +/- 4.7 val at step 2400 (85% total FLOPs) and 422.4 +/- 7.6 at step 2700 (96% FLOPs), beating 3000 step Standard (468.0 +/- 16.0 val, 455.3 +/- 7.6 test). Our 4.89M Morse arm prunes to m=277 at step 1000 (15.0d2<16d215.0d^2 < 16d^2/layer), hitting 452.2 val at step 2700 (92% total, 98.6% trunk FLOPs vs. Standard s0 final 486.1) and 399.0 test at 3000 steps (1.02x avg FLOPs).
  1. SwiGLU and GeGLU Baselines: Under the exact Sec 3.2 protocol (3000 steps, n=3), new parameter matched LLaMA SwiGLU (4.92M) gets 582.4 +/- 13.2 PPL and 4d SwiGLU (5.19M) gets 498.8 +/- 11.8. Holding (U, A, V), Prefix MS, Morse, and channel LRs identical at 4.92M, swapping ψ′′(Az+b)\psi''(Az+b) for SiLU(Az+b)\text{SiLU}(Az+b) worsens PPL from 415.9 to 450.3, isolating the curvature gate gain. (Sec 4.1 covers GeGLU/SwiGLU transfer.)

  2. Eq. 4 Contraction Calculus: Given Ψ(x)=∑jψ(sj)\Psi(x)=\sum_j \psi(s_j) with sj=uj⊤x+bjs_j=u_j^\top x+b_j, differentiating w.r.t. x drops bjb_j, giving ∇xsj=uj\nabla_x s_j=u_j and H(x)x=∑jψ′′(sj)(uj⊤x)ujH(x)x=\sum_j \psi''(s_j)(u_j^\top x)u_j. Thus bjb_j only shifts the gate query sjs_j, while the contracted factor is strictly uj⊤xu_j^\top x. The reviewer caught a typo in our submitted Sec 2.1 prose and Appendix B pseudocode (which wrote proj = s / + b); our actual training code (_PhiSpectralCore, _PhiPrefixMsMultwinLadderDecoupleCtxSplitCore) uses proj = x @ U.T without bjb_j, and we have fixed Sec 2.1 and Appendix B.

  3. Orthogonality and Eigenvectors: Unlike eigendecomposition QΛQ⊤Q \Lambda Q^\top (Q∈Rd×rQ \in \mathbb{R}^{d\times r}, Q⊤Q=IrQ^\top Q = I_r, r≤dr \le d), H(x)=U⊤diag⁡(λ)UH(x) = U^\top \operatorname{diag}(\lambda) U sums mm outer products with no orthogonality on U∈Rm×dU \in \mathbb{R}^{m\times d}, permitting m=4d>dm=4d > d. Thus uju_j form an overcomplete dictionary, not eigenvectors, and λj=ψ′′(sj)\lambda_j=\psi''(s_j) are scalar curvature gates, not eigenvalues (clarified in Sec 2.1).

  4. Symmetry of Decoupled H-FFN: Because ∇x2Ψ(x)\nabla_x^2 \Psi(x) is symmetric, V⊤ΛUV^\top \Lambda U (U≠VU \ne V) is nonsymmetric and thus not a single variable Hessian ∇x2Ψ(x)\nabla_x^2 \Psi(x) on xx alone. Yet from first principles, the curvature operator lives in coordinate space s=Ax~+bs = A\tilde{x}+b (including x~=x\tilde{x}=x): for Ψ(s)=∑jψ(sj)\Psi(s)=\sum_j \psi(s_j), its exact Hessian is Λ(s)=∇s2Ψ(s)=diag⁡(ψ′′(s))\Lambda(s)=\nabla_s^2 \Psi(s)=\operatorname{diag}(\psi''(s)). When inputs and outputs share one space (NIAH), Coaxial H-FFN pulls ∇s2Ψ(s)\nabla_s^2 \Psi(s) back symmetrically via U⊤ΛU=∇x2Ψ(Ux+b)U^\top \Lambda U = \nabla_x^2 \Psi(Ux+b). When tasks map across distinct subspaces (causal LM), Decoupled H-FFN keeps the exact coordinate Hessian ∇s2Ψ(s)\nabla_s^2 \Psi(s) while freeing U,A,VU, A, V to separate measurement, gating, and emission bases (clarified in Sec 2.2).

Thanks for the response and the additional experiments. Here's my assessment:

  • I think the clarifications on orthogonality and symmetry are helpful but they actually confirm my main concern: the best language model variant is basically a gated layer with separate content, gate and output matrices and not the Hessian of a potential, so the framing of the paper doesn't really hold anymore.
  • I also still have a few concerns about the experiments: 1. the SwiGLU numbers are surprising getting 582.4 for parameter-matched SwiGLU vs 455.3 for GELU, gated variants usually match or beat GELU so this makes me think the shared setup isn't tuned well for all the models, 2. 4.10M of the 4.92M params are in the tied embeddings, which means the 5.25M channel-split model has around 40% more non-embedding params than the standard one, 3. the FLOP overhead looks small only because the output layer dominates compute at d=128, and in the wider runs the gain shrinks and even reverses on test perplexity.

I think the direction is interesting but addressing these would require rewriting the framing and main claims of the paper, so I'm keeping my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.