acceptodds
Under review as a conference paper at ICLR 2027

Lossy Projection, Exact Expansion: Retrofitting Pretrained Gated Feed-Forward Blocks

Abstract

Weight movement can limit autoregressive language-model inference, motivating gated feed-forward architectures that reuse weights across computational paths. Masked Gated Linear Units (MGLUs) construct gate and value projections using complementary masks over a shared matrix, with multiple masks providing additional routes without duplicating that matrix. Converting pretrained blocks into this representation can incur branch-reconstruction error, while initializing additional routes can further alter the projected function. We propose Projection-Aware Route Expansion (PARE), which separates dense-to-shared projection from function-preserving multi-mask expansion. Our analysis characterizes the reconstruction limits of complementary sharing and establishes an exact embedding of the projected single-mask parent through repeated masks and a rescaled down projection. Mask logits are initialized with matching signs but distinct magnitudes across routes, yielding identical binary masks at entry while permitting differentiation during recovery. In the controlled 159M experiments, four-mask PARE reaches 15.65 perplexity versus 16.06 for fixed single-mask recovery at the same recovery-token budget. Within the same four-mask architecture, PARE also achieves lower endpoint perplexity than quota-preserving and reconstruction-calibrated diverse initializations. These endpoint advantages persist when the frozen checkpoints are evaluated on source-excluded FineWeb-Edu and C4. Further experiments on TinyLlama-1.1B and Gemma-3-1B, PARE also achieves lower endpoint perplexity than fixed single-mask recovery. A matched four-mask comparison on TinyLlama additionally favors PARE over diverse initialization. These results support function-preserving route entry for improving conversion fidelity at fixed token budgets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Pretrained model conversion, function-preserving expansion, weight sharing, gated feed-forward networks, language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eVsaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to recover the performance lost by converting dense GLU into MGLU. Specifically, they proposes PARE, a variant of MGLU, which tackle the limitations of the existing shared-weight with multi-mask methods and propose a new route expansion method. The key contribution of this study is that the authors explicitly split the conversion procedure into dense-to-share projection and function-preserving mask expansion, and provide separate solutions to them. The performance of PARE is mostly compared against a fixed single mask approach under several different experimental settings and benchmarks.

Critical strengths
  1. Many recent studies are well surveyed and summarized in Section 2. This is quite a good strong point. Readers will take advantage of this section and easily get to know the recent solutions as well as their limitations.

  2. The target problem is well focused to addressing the limitation of having a shared mask when dense weights are converted to a single shared weight.

  3. The rich appendix provides much additional information, which significantly improves reproducibility.

Critical weaknesses
  1. The reasoning in the introduction seems to have a logical gap. The referenced works establish that SwiGLU and GeGLU use separate gate and value projections, but they do not by themselves establish the broader claim that gated feed-forward blocks amplify the memory-bandwidth pressure in low-batch token generation. The authors should provide a clearer justification, e.g., by quantifying how the additional projections increase weight movement or memory traffic.

  2. The contribution summary at the end of the Introduction section does not show the actual contributions. For example, the second bullet explains one advantage of the proposed method, rather than the academic contribution of this study. Likewise, the fourth bullet is another advantage of PARE. Many of recent ML conference submissions share the same issue. I would like to recommend clarifying the actual contributions rather than summarizing the benefits of the proposed method.

  3. Proposition 1 and 3 do not deliver a meaningful or new result. First, the conclusion of proposition 1, that the gate and value matrices cannot be preserved exactly by a single shared matrix with a strict mask, is straightforward. Pretrained weight matrices are dense, so their supports most likely overlap, and it is self-evident that such matrices cannot be represented exactly under this sparse structure. I suggest presenting this statement as a remark rather than a proposition. Proposition 3 is also straightforward. Eq. (7) follows directly from summing nmn_m identical route outputs and dividing by nmn_m, and thus the condition F1⊆FnmF_1 \subseteq F_{n_m} is an immediate consequence. I suggest presenting Propositions 3 as remarks too.

  4. In Section 4.2, the authors set all the nmn_m initial masks the same. I wonder if the route will be effectively expanded. The exactly same initial nmn_m copies of WoW_o and their corresponding masks may not effectively diverge unless one of them have significantly different updates. The authors should substantiate this design choice either theoretically or empirically.

  5. Route expansion is one of the key ideas of PARE, but the ablation in Table 1 does not strongly support its effectiveness. It reduces PPL by only 0.4 on a single benchmark. The performance gain in Table 3 is even smaller. These results suggest that the benefit of route expansion is marginal, especially considering that all routes are executed for every input. I recommend that the authors report results over additional benchmarks to clarify whether route expansion yields a consistent improvement.

Overall, PARE is a relatively simple procedure for converting a pretrained dense GLU into MGLU and recovering the performance lost in this conversion. I find the novelty weak. The theoretical results are straightforward and largely disconnected from the proposed method, which relies on simple coefficient copying (section 4.2). The empirical results are also not convincing, as the gain from route expansion is marginal.

Questions for authors
  1. Could the authors clarify the connection between each proposition and the proposed method?

  2. What is the evidence of the proposed multi-mask scheme improves the performance over the MGLU's original approach?

  3. Is there any additional computational cost caused by the proposed method? System efficiency should be also discussed.

3 comments
AuthorsOpenReview ↗

What PARE contributes to pretrained conversion and when four routes help

Thank you for noting the value of the related-work survey and for showing us where the paper needs a quantitative motivation for weight sharing and a clearer link between the propositions and the conversion procedure.

Q1 (Weaknesses 2–3). What does PARE add, and how do the propositions inform it? MGLU supplies the architecture [R1], and replicating units with output rescaling is a known way to preserve a function [R2]. For a pretrained gated block, the projected parent fixes the signs of the mask logits but leaves their magnitudes free. PARE copies the signs and samples the magnitudes independently. The expanded block therefore starts with the parent's function, even though its routes start at different distances from their mask-switching thresholds.

We kept the proposition numbers to make the two conversion steps easy to reference. Proposition 1 gives the support and quota conditions under which both dense branch maps can be preserved. Proposition 2 explains why relaxing block quotas can lower the best attainable projection error. Proposition 3 gives the expansion rule used in the algorithm. Repeating the mask and dividing the shared down projection by the route count preserves the projected parent. The revised Introduction identifies the conversion procedure, the matched tests of initialization, and the fused FFN implementation as our contributions.

Revisions: Sec. 1 (pp. 2–3), Sec. 3.2 (pp. 4–5), Sec. 4.1 (p. 5).

Q2 (Weaknesses 4–5). Do the routes diverge, and does the choice of initialization matter? All four routes share one down projection. We compared two models with identical masks and outputs at entry, separately trainable route logits, and the same recovery schedule. With copied margins, the masks stayed identical at every update in all three seeds. With independent margins, their final mean pairwise Hamming distance was 0.06018–0.06022, and endpoint PPL was 0.4142 ± 0.0056 lower (mean paired difference ± sample SD). In a separate matched comparison, a 1% ownership exchange changed the entry function but finished only 0.0178 ± 0.0019 PPL above PARE. Together these controls show why the exact embedding alone leaves an important initialization choice open.

At fixed recovery tokens, Table 1 gives 15.65 PPL for PARE versus 16.06 for the fixed single-mask parent across three 159M seeds. Frozen-checkpoint evaluations on source-excluded FineWeb-Edu and C4 show the same ordering (Table 2). On TinyLlama and Gemma, the gains over fixed-mask recovery were smaller (0.1281 and 0.0454 PPL), and the fixed mask led on Gemma at steps 400–800. For a practitioner already converting to four routes, Table 3 gives the more direct comparison: at 1k steps PARE also finished below the 10% diverse entry on both TinyLlama (9.6010 vs. 9.8102) and Gemma (13.8145 vs. 14.1474, one seed per checkpoint). The six-task downstream mean improved by 0.13 percentage points in the 159M comparison.

Revisions: Sec. 5.3 (pp. 8–9), Table 3 (p. 9).

Q3 (Weakness 1 and the cost concern in Weakness 5). When does expansion pay off? If recovery time is the constraint, the fixed parent wins in our measured comparison. On the same RTX 4090, fixed-mask recovery took 11.63 s/step and PARE took 29.86 s/step. The estimated training time for 2,000 PARE steps allowed 5,136 fixed-mask steps. With cosine decay set to that budget, the fixed parent reached 15.5917 PPL versus 15.6536 for PARE (one seed).

At inference, all four routes execute, but they reuse the same input weights. In TinyLlama, the two dense FP16 input projections occupy 44 MiB. The fused PARE kernel uses 22 MiB for the shared weights and 11 MiB for the four mask bits stored in one byte per coordinate. Including the down projection gives 66 MiB per dense FFN versus 55 MiB per fused PARE FFN. Reading each array once would save 11 MiB per call. At one token per call, rotating through 22 layers, the fused PARE FFNs were 1.21–1.22× faster than the dense-source FFNs. The dense source had lower PPL (9.0277 vs. 9.6010). The FFN timing comparison therefore quantifies a workload-specific execution benefit at different model qualities.

Revisions: Sec. 1 (p. 1), Sec. 5.6 (pp. 9–10).

References

[R1] Yukito Tajima, Nakamasa Inoue, Yusuke Sekikawa, Ikuro Sato, and Rio Yokota. Masked Gated Linear Unit. NeurIPS, 2025. Paper

[R2] Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2Net: Accelerating Learning via Knowledge Transfer. ICLR, 2016. Paper

Thank you for your rebuttal. However, my concerns almost have not been addressed.

Q1. The rebuttal did not directly address my concern. First, the remarks still do not clearly explain why the proposed independent margin sampling is helpful.

Q2. The performance gain of large models is marginal. E.g., Gemma's improvement of 0.045 PPL is nearly meaningless. This degree of improvement is not distinguished from noise. Thus, the proposed method is still not that convincing especially for large-scale experiments.

Q3. The PPL results in the rebuttal shows that PARE does not exhibit competitive performance when the total time is the same. I still do not see any significant strong points.

For the above reasons, I will keep my current score.

Thank you for the follow-up. Our previous explanation omitted the step connecting logit magnitudes to different mask updates. We clarify it below and address the two empirical concerns.

Q1. Why can independent margin sampling help? Each binary assignment is mi=1{πi>0}m_i=\mathbf{1}\{\pi_i>0\}. Under our identity straight-through estimator, identical masks receive identical gradients in exact arithmetic. Copied logits with identical optimizer states therefore move together under the deterministic, symmetric updates. Separate storage alone leaves that symmetry intact.

Independent margins change the distances to zero while preserving every initial binary assignment. Even a common update can then produce different masks. For an illustrative coordinate, logits 0.010.01 and 0.030.03 both select the gate branch. Subtracting 0.020.02 from both changes the first assignment to the value branch while the second stays with the gate. Once assignments differ, the routes can compute different gated responses and receive different subsequent gradients. This allows the routes to learn distinct uses of the shared weights while starting from exactly the same parent function.

The three-seed control tests the recovery consequence of this choice. Copied logits stayed identical across routes at all 2,000 updates, whereas independent margins produced distinct masks and lowered endpoint PPL by 0.4142 ± 0.0056 (mean paired difference ± sample SD). This supports independent margins under the tested protocol. Appendix B.2 gives the symmetry and threshold-crossing arguments, and Sec. 5.3 reports the matched control.

Q2. How much evidence does the Gemma gain provide? The 0.0454 PPL improvement over the fixed single-mask parent comes from one recovery seed. The stronger replicated evidence concerns the initialization comparison above on the 159M model. For the public checkpoints, the separate four-route comparison holds the architecture and recovery budget fixed. PARE reaches 9.6010 versus 9.8102 PPL on TinyLlama and 13.8145 versus 14.1474 on Gemma. Each uses one paired seed (Table 3).

Q3. What does the equal-time result imply? We agree that the fixed single-mask model gave the better endpoint at the tested time budget. Both arms start from the same parent, which we construct from the pretrained dense block by choosing a layerwise balanced mask and copying the corresponding gate and value coefficients (Sec. 4.2). The comparison asks whether to spend the available recovery time training that parent or expanding it first. On the 159M model, 5,136 fixed-mask steps reached 15.5917 PPL, versus 15.6536 for 2,000 four-mask steps at the same estimated training time (one seed).

The projection itself also addresses a choice required for single-mask deployment. Section 5.2 compares four assignment constraints. With either scoring rule, the layerwise balanced mask gave the lowest PPL both immediately after conversion and after 2,000 recovery steps. This result informs how to build the parent even when recovery ends with one mask. For a four-mask target, the matched comparison in Q1 addresses the subsequent choice of how to initialize the added routes.

Reviewer_yoskOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how to convert pretrained gated FFNs into shared-weight masked GLUs. It separates the lossy merging of gate/value projections from the subsequent expansion into multiple masks. PARE repeats the projected parent's mask, divides the down projection by the number of routes, and initializes independent mask logits with different magnitudes. This preserves the projected function initially while allowing masks to separate during recovery. Experiments compare recovery quality across initialization schemes and benchmark fused FFN implementations.

Critical strengths
  1. The comparison within the same four-mask architecture is the strongest part of the paper. PARE reaches 15.65 PPL versus 15.86 for the diverse initialization, and the reconstruction-calibrated control still ends at 15.89 despite receiving additional calibration compute. This supports a real initialization effect, rather than simply showing that four masks outperform one. The ordering also survives evaluation on separate FineWeb-Edu and C4 data.

  2. I appreciate the precision about what is preserved and what is measured. The exact construction preserves the projected parent, not the original dense model. Likewise, the kernel study investigates weight reuse and reports the more realistic 1.21x FFN speedup under layer rotation alongside the larger repeated-call result. These distinctions make the evidence easier to trust.

Critical weaknesses
  1. I find the additional contribution somewhat narrow. Repeating identical routes and dividing their summed output by the route count follows directly from the existing parameterization. The more interesting question is why this entry helps optimization, but the experiments do not fully resolve it. The diverse comparison uses only a 10% ownership-exchange rate, while reconstruction changes both function error and parameter location. The tied-logit control in Table 8 uses a different parent and scaling rule. Consequently, I can conclude that the proposed initializer works better than the tested alternatives, but less confidently attribute the benefit to subsequent mask differentiation or exact preservation itself.

  2. The demonstrated quality benefit is small relative to the additional recovery cost. PARE takes 29.86 seconds per step versus 11.63 for fixed-mask recovery, and Table 13 improves average downstream accuracy only from 45.66% to 45.79%. On TinyLlama, PPL improves from 9.7291 to 9.6010 but remains above the dense source's 9.0277. The FFN measurements provide useful evidence, although they do not establish the full-model deployment trade-off. For me, this leaves the practical significance insufficiently established for acceptance.

Questions for authors
  1. What happens with four repeated masks and the same W_o/4 initialization as PARE, but with masks kept identical throughout recovery? A comparison from the primary layerwise-balanced parent would help distinguish mask differentiation from changes in optimization caused by replication and down-projection scaling. Table 8 does not hold these factors fixed.

  2. Does the advantage over diverse initialization persist with substantially smaller ownership exchanges, such as 1% rather than 10%? This would help determine whether exact preservation matters or whether the current alternative is simply too disruptive.

  3. At the measured recovery-time budget of 2k PARE steps, what quality can fixed-mask recovery reach with its schedule adjusted to the same wall-clock budget? This comparison would materially affect my assessment of the practical benefit.

1 comment
AuthorsOpenReview ↗

Four-route initialization and the practical cost of recovery

Thank you for highlighting the comparison within the same four-route architecture and for asking what the fixed parent can achieve in the same training time, which helps us state more precisely when PARE offers an advantage.

The model with copied margins computed the same function as PARE at entry and finished 0.4142 PPL above it at 2,000 steps. The model with a 1% ownership exchange started from a perturbed function and finished only 0.0178 PPL above PARE. Both gaps are means over three seeds, each measured against the matching PARE run.

Q1 (Weakness 1). What if the four masks remain identical throughout recovery? In three paired seeds, we initialized each route with a copy of the same margins and trained all four logit tensors separately. In a separate seed-1 control, we kept all four masks frozen throughout recovery. All conditions start from the same layerwise-balanced parent with four routes, Wo/4W_o/4, and entry NLL 4.293958. Within each seed they use the same data order and 2,000-step schedule.

ControlPaired seedsControl PPL at 2kPARE PPL at 2kControl − PARE
Copied margins, separately trainable logits1–3 (mean)16.068815.6546+0.4142 ± 0.0056
Frozen repeated masks1 only15.820915.6554+0.1655

The three-seed difference is paired within seed (mean ± sample SD). With copied margins, all four masks stayed identical at every update, although 2.14–2.15% of their coordinates differed from entry at step 2,000. With independent margins, the final mean pairwise Hamming distance was 0.06018–0.06022 across the three seeds.

Revisions: Sec. 5.3 (p. 9).

Q2 (Weakness 1). Would a 1% exchange recover as well? Nearly. We kept the first route's parent mask and perturbed the other three using a subset of the coordinates selected by the 10% rule. The initial weights and absolute logit margins were identical within each pair. The 1% gaps were +0.0180, +0.0158, and +0.0195 PPL in 3 seeds (sample SD 0.0019). The corresponding gap in the separate three-seed 10% comparison was 0.2114 ± 0.0046 PPL. As you suggested, reducing the ownership exchange closes most of the observed gap.

Revisions: Sec. 5.3 (p. 8), Appendix E.1 (p. 25).

Q3 (Weakness 2). What can fixed-mask recovery achieve in the time used by PARE? In an existing single-seed control, fixed-mask recovery reaches 15.5917 PPL versus PARE's 15.6536. The measured step-time ratio on the same RTX 4090 gives a budget of 5,136 fixed-mask steps for 2,000 PARE steps. We set the fixed-mask cosine schedule to 5,136 steps from the outset. The time estimate excludes evaluation and checkpointing.

In the separate 24-hour comparison, each model ran on its own RTX 4090 on the same host, using the same recovery seed. The fixed parent led at all six evaluations, with its PPL lead narrowing from 0.2957 at 4 hours to 0.1055 at 24 hours.

Revisions: Sec. 5.6 (pp. 9–10), Appendix D.3 (p. 23).

Weakness 2: practical significance. At the 159M training-time budget in Q3, keeping the single mask gave the better endpoint. At matched recovery tokens, expanding to four routes raised mean accuracy across six tasks from 45.66% to 45.79%, with mixed changes across tasks. On TinyLlama, the fused PARE FFNs ran 1.21–1.22× faster than the dense-source FFNs under single-token rotation through 22 layers. The dense model retained lower perplexity (9.0277 vs. 9.6010).

For a conversion that already targets four routes, the choice of initializer improves recovery at the same training-token budget. After 1,000 steps, replacing the 10% diverse initialization with PARE lowered endpoint PPL from 9.8102 to 9.6010 on TinyLlama and from 14.1474 to 13.8145 on Gemma (one paired seed per checkpoint).

Revisions: Table 3 (p. 9), Sec. 5.6 (pp. 9–10).

Reviewer_FpLhOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies conversion of pretrained gated feed-forward blocks into Masked Gated Linear Units (MGLUs), which share weights between complementary gate and value projections. It separates dense-to-shared branch projection from expansion to multiple masks. The analysis gives support conditions for exact branch reconstruction and shows that repeating a parent mask while dividing the down projection by the route count preserves the projected parent function. PARE initializes independent mask logits with equal signs but unequal random magnitudes, then recovers weights and assignments using a straight-through estimator. Unlike the original MGLU architecture, the focus is pretrained conversion; relative to conventional function-preserving growth, latent margins enable diversification without changing initial binary masks. Experiments use a controlled “159M” configuration and public TinyLlama-1.1B and Gemma-3-1B checkpoints. At matched recovery tokens, four-mask PARE improves controlled-model perplexity over fixed-mask recovery and two diverse initializations, with consistent ordering on additional FineWeb-Edu and C4 evaluations. Public-checkpoint gains are smaller, and Gemma trajectories are mixed. Fused TinyLlama FFNs show workload-dependent speedups. The findings apply to the tested conversion protocols and budgets, without establishing lossless recovery of the original dense model.

Critical strengths
  1. Strengths (1)Clear problem decomposition and technically consistent analysis. The paper separates two distinct stages: reconstructing pretrained branch maps under complementary sharing is generally lossy, whereas repeating the parent mask and scaling the down projection by (1/n_m) exactly preserves the projected parent function (Sections 3.2 and 4.1; Propositions 1–3). Appendix A.2 additionally derives the optimal fixed-mask reconstruction using the full input second-moment matrix. This distinction helps identify which discrepancies arise from projection and which arise from expansion initialization. Under the stated assumptions, I did not identify a clear mathematical error in Propositions 1–3 or the Schur-complement reconstruction derivation. (2)A practical initialization that preserves the initial function while permitting route differentiation. The same-sign, different-magnitude mask-logit initialization is a reasonable way to reconcile function preservation with symmetry breaking (Equations (9)–(10) and Appendix B.2). Routes begin with identical binary masks but different latent margins, allowing their assignments to change at different times during recovery. The diagnostics in Section 5.3 confirm that the masks differentiate: the mean pairwise Hamming distance reaches 0.06017 at 2k steps, with no duplicate mask pair. These observations establish that recovery moves beyond identical route assignments, although they do not independently establish that differentiation causes the performance gains. (3)Controlled evidence extending beyond the single-mask comparison. Within the same four-route architecture, PARE outperforms the quota-preserving diverse initialization by a paired endpoint difference of (0.2114 \pm 0.0046) PPL. Reconstruction calibration also does not eliminate the gap (Tables 1 and 6–7). The primary comparisons use three seeds, and the observed differences substantially exceed the reported variation across seeds. Moreover, the ordering persists on additional FineWeb-Edu and C4 evaluations conducted after methods and checkpoints were fixed (Tables 2 and 10). Together, these results support the empirical conclusion that initialization paths affect finite-budget recovery and reduce concerns that the findings are specific to one development buffer or an individual seed. (4)Complete-FFN measurements with explicit consideration of cache effects. The efficiency evaluation includes the down projection and selects the faster dense baseline from multiple implementations (Section 5.6 and Appendices G.1–G.4). The paper also distinguishes repeated calls to the same layer from rotation through 22 independently evaluated FFNs, reporting a 1.21–1.22× speedup under the latter workload. The cache analysis in Tables 16–18 helps explain why larger continuous-call speedups depend on weight-access patterns. This provides useful measured evidence for the execution benefits of shared-weight FFNs while making the workload dependence of those benefits clear.
Critical weaknesses

(1) Limited incremental novelty. PARE combines the existing MGLU/MoEG architecture with the established replication-and-rescaling principle of function-preserving expansion. Its main distinction is the same-sign, different-magnitude logit initialization, which preserves initial binary masks while permitting subsequent differentiation. Proposition 3 follows directly from repeated-route summation, while Propositions 1–2 and Appendix A.2 provide relatively direct structural and reconstruction analyses. The contribution therefore appears primarily to be a targeted conversion and initialization recipe; stronger empirical isolation of the latent-margin mechanism would strengthen its novelty. (2) Mask differentiation is not cleanly isolated from reparameterization effects. The tied-logit control in Table 8 uses a strict 2:4 parent and rescales (W), whereas primary PARE uses a layerwise-balanced parent and rescales (W_o). Moreover, identical initial functions and AdamW learning rates do not guarantee equivalent training dynamics after rescaling. This is a potential confound, not an established explanation for the gains. A cleaner comparison would use the same balanced parent, (W_o/4), and independent logit tensors with matching optimizer-state representations, comparing copied versus independently sampled margins, alongside frozen repeated masks. (3) Limited attribution to function preservation. Section 5.3 tests only 10% ownership exchange. Reconstruction calibration removes most initial NLL excess, but residual discrepancies remain: entry NLL is approximately 4.3601 versus PARE’s 4.2940, and held-out FFN error remains 0.0402–0.0408. Calibration also changes shared weights, leaving function mismatch, mask geometry, and parameter displacement coupled, as acknowledged in Appendix H. Testing lighter perturbations, such as 0.1% or 1% exchange under otherwise matched conditions, would clarify the mechanism. Currently, the evidence establishes superiority over the tested entries rather than isolating function preservation as the cause.

Questions for authors
  1. Can the existing recovery logs establish how PARE compares with fixed-mask recovery at equal wall-clock budgets? Given the reported 2.57× difference in step time, does PARE’s token-matched advantage persist when recovery cost is matched?
  2. Under the primary configuration, does independently sampled margin initialization outperform copied margins when both conditions retain independently stored logit tensors, the same balanced parent, Wo/4W_o/4, and identical optimization? This would isolate margin diversity without the architectural and parameterization differences present in Table 8.
1 comment
AuthorsOpenReview ↗

Recovery at equal training time and the role of mask-logit margins

Thank you for recognizing the value of separating projection from expansion and for helping us clarify why the controls in Table 8 cannot isolate the effect of logit margins on recovery.

Both controls used the same four-route architecture. Copying the logit margins preserved PARE's entry function, yet raised perplexity by 0.4142 ± 0.0056 at 2,000 steps. Exchanging 1% of gate and value assignments changed the entry function and raised perplexity by only 0.0178 ± 0.0019. Each result is the mean difference from its paired PARE baseline across three seeds (± sample SD).

Q1. Does the advantage persist at equal recovery time? No, at the budget you ask about. In our single-seed control on one RTX 4090, PARE reached 15.6536 PPL in 2,000 steps, versus 16.0578 for the fixed parent at the same token budget. A fixed-mask step took 11.63 s versus 29.86 s for PARE. Matching the estimated training time of those 2,000 PARE steps (about 16.6 hours, excluding evaluation and checkpointing) allowed 5,136 fixed-mask steps. With cosine decay scheduled over that full horizon, the fixed parent reached 15.5917 PPL.

In a supplementary 24-hour comparison using RTX 4090, the fixed parent led at all six evaluations, but its lead narrowed at every four-hour interval, from 0.2957 PPL at 4 hours to 0.1055 at 24 hours.

In 10B-token pretraining from scratch, Tajima et al. [R1] reported lower validation perplexity with four masks than with one (23.9 vs. 25.0), at a higher training cost (33 vs. 20 GPU-hours, Tables 3 and 6). This motivates studying longer recovery budgets.

Revisions: Sec. 5.6 (pp. 9–10), Appendix D.3 (p. 23).

Q2 (Weakness 2). What do independent margins contribute? We ran the suggested control from the primary layerwise-balanced parent. Both four-route models start with the same binary masks, shared weights, and down projection initialized to Wo/4W_o/4. Both use four separately stored slices of mask logits with their own AdamW moments. In the control we copied the first route's sampled margins into the other three slices at initialization. We did not tie or synchronize them afterward. The models had the same entry NLL (4.293958) and saw the same recovery data in the same order.

After 2,000 steps, PARE reached 15.6546 PPL on average across three paired recovery seeds, versus 16.0688 with copied margins. The copied-minus-PARE gaps were +0.4148, +0.4083, and +0.4195 PPL. In every seed, the four copied logit tensors stayed identical at every update, even though 2.14–2.15% of mask coordinates differed from their initial assignments at step 2,000. Freezing the repeated masks gave 15.8209 PPL in seed 1, versus 15.6554 for PARE in that seed (gap +0.1655).

Revisions: Sec. 5.3 (p. 9), Appendix H (p. 36).

Weakness 1. What does PARE add to existing growth methods? MGLU [R1] supplies the architecture. Repeating routes and rescaling their output follows the replication principle used in Net2Net [R2]. The conversion problem leaves a further choice: two models can compute the same function at entry yet start with different distances to their mask thresholds. PARE copies the binary assignments but samples those distances independently. The copied-margin comparison above measures the consequence of that choice while keeping the function, parameterization, and optimizer fixed.

Revisions: Sec. 1 (pp. 2–3), Sec. 4.1 (p. 5).

Weakness 3. How much of the gain can be attributed to exact preservation? Reconstruction calibration reduced the diverse initializer's entry NLL from 5.1549 to 4.3601, still above the parent's 4.2940. It also changed the shared weights and down projection. To test a milder perturbation while holding the initial weights and absolute logit margins fixed, we exchanged 1% of gate coordinates for an equal number of value coordinates on three routes. At 2,000 steps, the 1% entry finished 0.0180, 0.0158, and 0.0195 PPL above its paired PARE run in seeds 1–3. At 10% exchange, the mean paired gap was 0.2114 ± 0.0046 PPL. The exchange changes both route assignments and the entry function, so this result measures sensitivity to perturbation size. It leaves their individual contributions unresolved.

Revisions: Sec. 5.3 (p. 8), Appendix E.1 (p. 25), Appendix H (p. 36).

References

[R1] Yukito Tajima, Nakamasa Inoue, Yusuke Sekikawa, Ikuro Sato, and Rio Yokota. Masked Gated Linear Unit. NeurIPS, 2025. Paper

[R2] Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2Net: Accelerating Learning via Knowledge Transfer. ICLR, 2016. Paper

Comments on the paper

1 comment
AuthorsOpenReview ↗

Authors’ Summary of Revisions and Contributions

Dear Area Chair and Reviewers,

Thank you for asking what makes initially identical routes diverge, how much the choice of initialization affects recovery, and what a single-mask model can achieve in the same training time.

Revisions prompted by the discussion

  • Route initialization. We compared Projection-Aware Route Expansion (PARE) with a four-route model that computes the same function at entry. When we copied the mask-logit magnitudes across routes, the four masks stayed identical throughout 2,000 updates, even though each route had separately trainable logits. PARE samples the magnitudes independently. The copied-margin model's final perplexity (PPL) was 0.4142 ± 0.0056 higher than PARE's. Exchanging 1% of gate and value assignments changed the entry function but finished only 0.0178 ± 0.0019 PPL above PARE. Both gaps are paired means ± sample SD across three recovery seeds (Sec. 5.3, Appendix E.1).
  • Recovery at equal training time. On the 159M model, the estimated optimizer time for 2,000 PARE steps allowed 5,136 fixed-mask steps on the same RTX 4090. With its learning-rate schedule adjusted to that budget, the fixed parent reached 15.5917 PPL versus 15.6536 for PARE in one paired seed (Sec. 5.6).
  • Storage in the measured implementation. The TinyLlama kernel stores four route bits in one byte per weight coordinate. Including the down projection, weights and masks occupy 55 MiB per fused feed-forward block versus 66 MiB for the dense block (Sec. 5.6).

Main contributions and significance

  • Lossy projection and exact expansion. Our analysis identifies when complementary sharing can preserve both pretrained branch maps. Once the single-mask parent is constructed, PARE repeats its mask and rescales the down projection to preserve its function exactly. At matched tokens, four-route PARE reached 15.65 PPL versus 16.06 for fixed single-mask recovery at 159M. The ordering held on separate FineWeb-Edu and C4 evaluations (Secs. 3–5).
  • Execution of recovered blocks. Our TinyLlama kernel fuses the shared projections, activation and route sum. At one token per call, rotating through 22 layers, the complete feed-forward blocks ran 1.21–1.22× faster than their dense counterparts (Sec. 5.6).

Table 3 illustrates what the initialization changes when the target architecture and recovery budget are fixed. After 1,000 steps, replacing the 10% diverse initialization with PARE lowered PPL from 9.8102 to 9.6010 on TinyLlama and, in the newly added Gemma comparison, from 14.1474 to 13.8145.

Thank you again for the constructive discussion.

Respectfully,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.