- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to recover the performance lost by converting dense GLU into MGLU. Specifically, they proposes PARE, a variant of MGLU, which tackle the limitations of the existing shared-weight with multi-mask methods and propose a new route expansion method. The key contribution of this study is that the authors explicitly split the conversion procedure into dense-to-share projection and function-preserving mask expansion, and provide separate solutions to them. The performance of PARE is mostly compared against a fixed single mask approach under several different experimental settings and benchmarks.
Critical strengths
-
Many recent studies are well surveyed and summarized in Section 2. This is quite a good strong point. Readers will take advantage of this section and easily get to know the recent solutions as well as their limitations.
-
The target problem is well focused to addressing the limitation of having a shared mask when dense weights are converted to a single shared weight.
-
The rich appendix provides much additional information, which significantly improves reproducibility.
Critical weaknesses
-
The reasoning in the introduction seems to have a logical gap. The referenced works establish that SwiGLU and GeGLU use separate gate and value projections, but they do not by themselves establish the broader claim that gated feed-forward blocks amplify the memory-bandwidth pressure in low-batch token generation. The authors should provide a clearer justification, e.g., by quantifying how the additional projections increase weight movement or memory traffic.
-
The contribution summary at the end of the Introduction section does not show the actual contributions. For example, the second bullet explains one advantage of the proposed method, rather than the academic contribution of this study. Likewise, the fourth bullet is another advantage of PARE. Many of recent ML conference submissions share the same issue. I would like to recommend clarifying the actual contributions rather than summarizing the benefits of the proposed method.
-
Proposition 1 and 3 do not deliver a meaningful or new result. First, the conclusion of proposition 1, that the gate and value matrices cannot be preserved exactly by a single shared matrix with a strict mask, is straightforward. Pretrained weight matrices are dense, so their supports most likely overlap, and it is self-evident that such matrices cannot be represented exactly under this sparse structure. I suggest presenting this statement as a remark rather than a proposition. Proposition 3 is also straightforward. Eq. (7) follows directly from summing identical route outputs and dividing by , and thus the condition is an immediate consequence. I suggest presenting Propositions 3 as remarks too.
-
In Section 4.2, the authors set all the initial masks the same. I wonder if the route will be effectively expanded. The exactly same initial copies of and their corresponding masks may not effectively diverge unless one of them have significantly different updates. The authors should substantiate this design choice either theoretically or empirically.
-
Route expansion is one of the key ideas of PARE, but the ablation in Table 1 does not strongly support its effectiveness. It reduces PPL by only 0.4 on a single benchmark. The performance gain in Table 3 is even smaller. These results suggest that the benefit of route expansion is marginal, especially considering that all routes are executed for every input. I recommend that the authors report results over additional benchmarks to clarify whether route expansion yields a consistent improvement.
Overall, PARE is a relatively simple procedure for converting a pretrained dense GLU into MGLU and recovering the performance lost in this conversion. I find the novelty weak. The theoretical results are straightforward and largely disconnected from the proposed method, which relies on simple coefficient copying (section 4.2). The empirical results are also not convincing, as the gain from route expansion is marginal.
Questions for authors
-
Could the authors clarify the connection between each proposition and the proposed method?
-
What is the evidence of the proposed multi-mask scheme improves the performance over the MGLU's original approach?
-
Is there any additional computational cost caused by the proposed method? System efficiency should be also discussed.
3 comments
Thank you for your rebuttal. However, my concerns almost have not been addressed.
Q1. The rebuttal did not directly address my concern. First, the remarks still do not clearly explain why the proposed independent margin sampling is helpful.
Q2. The performance gain of large models is marginal. E.g., Gemma's improvement of 0.045 PPL is nearly meaningless. This degree of improvement is not distinguished from noise. Thus, the proposed method is still not that convincing especially for large-scale experiments.
Q3. The PPL results in the rebuttal shows that PARE does not exhibit competitive performance when the total time is the same. I still do not see any significant strong points.
For the above reasons, I will keep my current score.
Thank you for the follow-up. Our previous explanation omitted the step connecting logit magnitudes to different mask updates. We clarify it below and address the two empirical concerns.
Q1. Why can independent margin sampling help? Each binary assignment is . Under our identity straight-through estimator, identical masks receive identical gradients in exact arithmetic. Copied logits with identical optimizer states therefore move together under the deterministic, symmetric updates. Separate storage alone leaves that symmetry intact.
Independent margins change the distances to zero while preserving every initial binary assignment. Even a common update can then produce different masks. For an illustrative coordinate, logits and both select the gate branch. Subtracting from both changes the first assignment to the value branch while the second stays with the gate. Once assignments differ, the routes can compute different gated responses and receive different subsequent gradients. This allows the routes to learn distinct uses of the shared weights while starting from exactly the same parent function.
The three-seed control tests the recovery consequence of this choice. Copied logits stayed identical across routes at all 2,000 updates, whereas independent margins produced distinct masks and lowered endpoint PPL by 0.4142 ± 0.0056 (mean paired difference ± sample SD). This supports independent margins under the tested protocol. Appendix B.2 gives the symmetry and threshold-crossing arguments, and Sec. 5.3 reports the matched control.
Q2. How much evidence does the Gemma gain provide? The 0.0454 PPL improvement over the fixed single-mask parent comes from one recovery seed. The stronger replicated evidence concerns the initialization comparison above on the 159M model. For the public checkpoints, the separate four-route comparison holds the architecture and recovery budget fixed. PARE reaches 9.6010 versus 9.8102 PPL on TinyLlama and 13.8145 versus 14.1474 on Gemma. Each uses one paired seed (Table 3).
Q3. What does the equal-time result imply? We agree that the fixed single-mask model gave the better endpoint at the tested time budget. Both arms start from the same parent, which we construct from the pretrained dense block by choosing a layerwise balanced mask and copying the corresponding gate and value coefficients (Sec. 4.2). The comparison asks whether to spend the available recovery time training that parent or expanding it first. On the 159M model, 5,136 fixed-mask steps reached 15.5917 PPL, versus 15.6536 for 2,000 four-mask steps at the same estimated training time (one seed).
The projection itself also addresses a choice required for single-mask deployment. Section 5.2 compares four assignment constraints. With either scoring rule, the layerwise balanced mask gave the lowest PPL both immediately after conversion and after 2,000 recovery steps. This result informs how to build the parent even when recovery ends with one mask. For a four-mask target, the matched comparison in Q1 addresses the subsequent choice of how to initialize the added routes.
What PARE contributes to pretrained conversion and when four routes help
Thank you for noting the value of the related-work survey and for showing us where the paper needs a quantitative motivation for weight sharing and a clearer link between the propositions and the conversion procedure.
Q1 (Weaknesses 2–3). What does PARE add, and how do the propositions inform it? MGLU supplies the architecture [R1], and replicating units with output rescaling is a known way to preserve a function [R2]. For a pretrained gated block, the projected parent fixes the signs of the mask logits but leaves their magnitudes free. PARE copies the signs and samples the magnitudes independently. The expanded block therefore starts with the parent's function, even though its routes start at different distances from their mask-switching thresholds.
We kept the proposition numbers to make the two conversion steps easy to reference. Proposition 1 gives the support and quota conditions under which both dense branch maps can be preserved. Proposition 2 explains why relaxing block quotas can lower the best attainable projection error. Proposition 3 gives the expansion rule used in the algorithm. Repeating the mask and dividing the shared down projection by the route count preserves the projected parent. The revised Introduction identifies the conversion procedure, the matched tests of initialization, and the fused FFN implementation as our contributions.
Revisions: Sec. 1 (pp. 2–3), Sec. 3.2 (pp. 4–5), Sec. 4.1 (p. 5).
Q2 (Weaknesses 4–5). Do the routes diverge, and does the choice of initialization matter? All four routes share one down projection. We compared two models with identical masks and outputs at entry, separately trainable route logits, and the same recovery schedule. With copied margins, the masks stayed identical at every update in all three seeds. With independent margins, their final mean pairwise Hamming distance was 0.06018–0.06022, and endpoint PPL was 0.4142 ± 0.0056 lower (mean paired difference ± sample SD). In a separate matched comparison, a 1% ownership exchange changed the entry function but finished only 0.0178 ± 0.0019 PPL above PARE. Together these controls show why the exact embedding alone leaves an important initialization choice open.
At fixed recovery tokens, Table 1 gives 15.65 PPL for PARE versus 16.06 for the fixed single-mask parent across three 159M seeds. Frozen-checkpoint evaluations on source-excluded FineWeb-Edu and C4 show the same ordering (Table 2). On TinyLlama and Gemma, the gains over fixed-mask recovery were smaller (0.1281 and 0.0454 PPL), and the fixed mask led on Gemma at steps 400–800. For a practitioner already converting to four routes, Table 3 gives the more direct comparison: at 1k steps PARE also finished below the 10% diverse entry on both TinyLlama (9.6010 vs. 9.8102) and Gemma (13.8145 vs. 14.1474, one seed per checkpoint). The six-task downstream mean improved by 0.13 percentage points in the 159M comparison.
Revisions: Sec. 5.3 (pp. 8–9), Table 3 (p. 9).
Q3 (Weakness 1 and the cost concern in Weakness 5). When does expansion pay off? If recovery time is the constraint, the fixed parent wins in our measured comparison. On the same RTX 4090, fixed-mask recovery took 11.63 s/step and PARE took 29.86 s/step. The estimated training time for 2,000 PARE steps allowed 5,136 fixed-mask steps. With cosine decay set to that budget, the fixed parent reached 15.5917 PPL versus 15.6536 for PARE (one seed).
At inference, all four routes execute, but they reuse the same input weights. In TinyLlama, the two dense FP16 input projections occupy 44 MiB. The fused PARE kernel uses 22 MiB for the shared weights and 11 MiB for the four mask bits stored in one byte per coordinate. Including the down projection gives 66 MiB per dense FFN versus 55 MiB per fused PARE FFN. Reading each array once would save 11 MiB per call. At one token per call, rotating through 22 layers, the fused PARE FFNs were 1.21–1.22× faster than the dense-source FFNs. The dense source had lower PPL (9.0277 vs. 9.6010). The FFN timing comparison therefore quantifies a workload-specific execution benefit at different model qualities.
Revisions: Sec. 1 (p. 1), Sec. 5.6 (pp. 9–10).
References
[R1] Yukito Tajima, Nakamasa Inoue, Yusuke Sekikawa, Ikuro Sato, and Rio Yokota. Masked Gated Linear Unit. NeurIPS, 2025. Paper
[R2] Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2Net: Accelerating Learning via Knowledge Transfer. ICLR, 2016. Paper