- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Hessian Feed-Forward Network(H-FFN), which replaces 4d MLP with a sum of rank-one directional units derived from a scalar potential. The authors also develop H-FFN variants for memory tasks and language modeling tasks. Experiments show the superior performance of H-FFN over 4d MLP.
Critical strengths
- The paper presents a novel Hessian-based Transformer FFN, together with specific variants design for different tasks.
- The H-FFN shows strong performance on associative retrieval benchmarks, including NIAH and Gap Copy, demonstrating the potential of the proposed architecture for memory retrieval.
Critical weaknesses
While H-FFN performs strongly on associative retrieval tasks, the evidence for language modeling is largely insufficient to support the broader claim. In section 3.2, the size of the proposed model does not match the 4D baseline (5.25M vs. 4.92M). Therefore, where the gain comes from is unclear. More importantly, the language modeling experiments are conducted only at very small scale, which is not enough to demonstrate effectiveness in modern language modeling, where scalability is critical. A compute-matched scaling study, ideally following a Chinchilla-style compute-optimal setup, is needed to validate this claim. Section 3.3 is also under-justified: the authors replace only the final four layers without explaining this choice or providing ablations over the number or location of replaced layers.
Questions for authors
- See weakness.
- How does H-FFN compare against modern gated FFNs such as SwiGLU in LLaMA under matched parameter and compute budgets? The structural similarity between Channel-Split H-FFN and gated MLPs makes this comparison very important.
- How does H-FFN interact with the Muon optimizer?
3 comments
Thank you for the additional experiments. The response addresses some of my concerns, but my concerns remain:
- Parameter matching: Step B uses pruning, so it does not establish performance under a matched parameter budget from initialization.
- Scalability: The additional small-scale experiments do not address the need for a compute-matched evaluation across model scales. As noted in my original review, such evidence is essential to support the broader claim, although obtaining it may be beyond the scope of the rebuttal period.
- Clarity: The response references Section 4.2, which does not exist in the available manuscript.
I therefore maintain my original rating.
Thank you for the careful follow-up. We clarify all three points briefly below:
-
Clarity (Section 4.2 Reference): In the submitted PDF, the FineWeb-Edu, d=256, and d=384 results appear in Section 5 (Discussion, page 9). The "Section 4.2" label came from an unsubmitted revision outline reorganized to address Reviewer 1's structural feedback. We apologize for the mismatch.
-
Parameter Matching and Morse Pruning: You are right that Step B starts at m=512 (5.25M) before pruning to m=277 (4.89M) at step 1000. This actually reflects Morse persistence ( on rank-one units ), which drops degenerate directions and restricts the last 2000 steps to a 4.89M subspace, a property standard dense FFNs lack. Steps A and C show that initial width m or parameter count is not the sole driver: widening SwiGLU (Step A) fails to match Standard, and under identical width and pruning trajectories (Step C), still outperforms SiLU.
-
Scalability: In the revision, we will tighten our claims to the validated scales (compact scratch tasks and 86M/124M transfer) and explicitly note multi-scale Chinchilla iso-compute pre-training as a limitation and next step.
We add <=4.92M Morse Channel Split runs (399.0 PPL, 1.02x total FLOPs), three seed LLaMA SwiGLU baselines (582.4 PPL), a same stack SiLU gate ablation (415.9 vs. 450.3), GPT-2 layer depth ablations (L11 vs. L8-11 vs. all 12 layers), a 3000 step d=256 run (314.8 vs. 333.2), and Muon analysis, explaining how each control isolates the asked mechanism.
We thank the reviewer for noting the novelty of H FFN and raising constructive questions on LLaMA SwiGLU, compute matched scaling, GPT-2 layer selection, and Muon.