- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies whether endogenous signals can usefully allocate token-level feed-forward computation. It examines several gating failures, compares difficulty- and gradient-based targets with shuffled controls, and measures how gate-gradient structure changes during training. An online allocation experiment finds the tested gradient-following strategy worse than random allocation at two arithmetic scales. The paper also evaluates random token-level FFN dropout in Pythia-1.4B fine-tuning and benchmarks a gather-based implementation. Results reveal limited value from the tested allocation signals and document quality–compute trade-offs, although the broader claim that compute value is not learnable exceeds the demonstrated settings.
Critical strengths
-
The study uses meaningful negative controls and reports unfavorable outcomes. Section 4.2 compares measured targets with shuffled targets on shared checkpoint forks, while Sections 4.4–4.5 disclose failed predictions and weak transfer. These experiments distinguish successful fitting of allocation structure from actual task benefit. The distinction is consequential and could support a valuable, carefully scoped empirical paper.
-
The paper separates nominal savings from executable speedups. Section 4.7 and Appendix F show that simply masking FFN outputs provides no speedup, whereas gathering active rows reaches 1.35× throughput in the reported benchmark. The paper also acknowledges that shorter dense training wins its natural-language cost-matched comparisons. These practical observations help prevent misleading interpretations of gated-compute accounting.
Critical weaknesses
-
The evidence does not establish that the measured structure is not learnable. Table 1 tests one hard allocation strategy refreshed every 100 steps, with one seed per arm at each scale; supervised heads generally use a nearly frozen body. Failure of these procedures does not exclude better co-adaptation or allocation rules. Moreover, sorting gate gradients optimizes a local linear surrogate, not the nonlinear loss under finite gate changes or retraining, so it is not a general upper bound. These gaps directly undermine the title and central conclusion.
-
The positive efficiency case remains unsupported at a consistent total-cost comparison. Appendix D acknowledges that the 1.4B counter bills attention once rather than once per layer, so the approximately 50% figure concerns FFN-dominated accounting rather than total training FLOPs. The measured speedup comes from a separate short implementation benchmark, whereas the reported training driver uses masking. Section 4.5 additionally finds shorter dense training better at both billed-cost and estimated wall-clock matches. These disclosures support a trade-off study, but not a demonstrated advantage of the proposed recipe.
Questions for authors
-
Clarification: Is there a formal argument that the first-order water-filling procedure bounds nonlinear allocation or learning performance? If not, will the oracle language and non-learnability conclusion be restricted to the tested surrogate and update rule?
-
Clarification: Are the additional starved-150M seeds mentioned as queued in the limitations available? What existing evidence distinguishes failure of the specific 100-step hard allocator from a general inability to learn useful compute allocation?
-
Clarification: Can the existing runs be compared using cumulative total training cost with attention counted per layer and all stages included? Was equivalent training behavior verified for the gather implementation, and is there any reported regime where the recipe outperforms shorter dense training at matched realized cost?
3 comments
Response 2 of 3: the fixed allocation rule and separation of quality from total cost.
We also tested whether a simple rule based on current token activations already contains useful allocation structure. For each token, the selected rule takes the largest absolute activation value entering the FFN. Tokens with the smallest such values execute, up to an exact per-layer budget. The rule needs no learned router, future reward evaluation, gradient estimate, or additional dense FFN pass.
The rule was selected on development models and then left unchanged. Across four models and activity budgets of 37.5% and 50%, it wins all eight final comparisons against random masks executing the same number of rows. It also wins all 32 evaluations at steps 50, 100, 150, and 200. Averaging first within each model, the final loss difference is -0.0319 +/- 0.0062 and the answer-accuracy difference is +3.37 +/- 0.65 percentage points. Models 9 and 10 were not used to select the rule, although they had appeared in earlier unrelated experiments.
The shuffled control keeps the same scores and active-row count within each layer but assigns those scores to different tokens. It is worse than the fixed rule by 0.02530 loss on average. Rules based on previous-step gradients or on Adam-adjusted gradient direction also outperform random, but remain behind the fixed activation rule in all four development settings. These controls show that assigning computation to particular tokens matters, rather than merely executing a fixed fraction of them.
We now separate three claims.
First, token selection under a fixed execution budget. The comparisons above use equal training updates and equal numbers of executed FFN rows. They test whether the choice of tokens matters; they are not total-cost victories.
Second, quality of the full dense model. The learned router's benefit is specific to evaluation with FFN skipping. When the resulting models are evaluated densely, the learned-router model is worse than random by 0.0499 loss on average. Even its sparse loss remains 0.0240 above a model trained densely for the same number of updates. We therefore do not claim that the router learns a better dense model.
Third, practical total cost. The original 1.4B counter counted attention once rather than once per layer, so we withdrew that total-FLOP interpretation. We also found that PyTorch's counter on Apple MPS omitted ordinary backward-pass work. Old MPS-only totals are therefore not used for practical claims. Revised practical comparisons use operation counts checked against CPU execution, including forward and backward computation in every layer and additional work such as routing, reward evaluations, router updates, or weight averaging where applicable.
Response 3 of 3: row-skipping validation, dense-fewer-steps, and scale.
We agree that the original text connected two paths that should have been distinguished: the original training driver, which multiplied outputs by masks, and a separate timing benchmark, which actually skipped rows.
The revised validation checks that binary masking and row skipping produce equivalent forward outputs, that selected row counts are exact and exclude padding, and that the differentiable weight-update implementation matches ordinary AdamW including optimizer moments. We also checked gradients through the router-learning update against finite differences, and checked the outputs, input gradients, and parameter gradients of the sparse-update operations against autograd.
The fixed-rule training and ordinary router-training steps use actual row skipping. The periodic router-learning step uses a differentiable approximation around the binary mask; it is distinguished from ordinary execution and its additional work is counted.
These checks establish numerical properties of the operations used in the new experiments. We do not claim that they establish end-to-end equivalence of the original masked training driver and the separate timing benchmark. The 1.35x figure is now described only as a short kernel-level benchmark, not as end-to-end training acceleration.
We have not established a replicated setting in which any proposed recipe outperforms shorter dense training at matched actual cost. Under the corrected accounting, a variant with dense forward computation and partial backward computation gave a 0.000366 loss advantage on one development model. The unchanged protocol lost by 0.003133 on the second model. The two-model mean difference is +0.0014 loss relative to dense. We report both outcomes and make no practical-superiority claim.
For scale, the original starved-150M causal claim remains withdrawn because its experiment driver ignored --size. Separately, we completed a frozen 150M pilot for the fixed rule: two existing 4,000-step checkpoints, 50 continuation steps, 50% activity, paired random controls, and 128 newly generated held-out sequences per evaluation mode. Final sparse-loss differences are -0.04068 on model 0 and -0.04405 on model 1; sparse accuracy differences are +2.98 and +5.10 percentage points. Dense-loss differences are -0.01359 and -0.00126, although dense loss worsens from step 0 in all runs.
Both fixed-rule runs improve their own sparse loss, but random has the larger reduction during continuation on both models because the fixed rule already starts with lower sparse loss. The final advantage is smaller than the initial advantage. We therefore call this a 50-step final-quality signal at a second scale, not faster learning, a 200-step replication, or a practical-efficiency result.
Does the new evidence from actual mask-dependent weight updates address your concern about learnability, independently of practical efficiency? If a soundness concern remains, would it be more useful to discuss the difference between sparse and dense evaluation, or the total-cost comparison, first?
Response 1 of 3: the first-order limitation and direct test through actual model updates.
Thank you for identifying the exact gaps between the original experiments and its claims. We acknowledge all three questions directly. There is no formal nonlinear bound from the first-order procedure. The old starved-150M claim was unsupported and has been withdrawn. The original efficiency comparisons did not consistently combine total cost, actual execution, and training behavior. The next two comments cover the fixed-rule result, accounting, implementation checks, and practical boundary.
There is no formal argument that sorting gate gradients bounds nonlinear allocation or learning performance. The target -sum_l dL/dg_{t,l}, evaluated at g=1, measures how sensitive the loss is to a very small gate change with the model held fixed. Sorting this quantity is optimal only for that small-change linear approximation. It does not bound the loss after turning FFNs on or off, choosing a different allocation rule, training the model and router together, or retraining. We removed "oracle" and "upper bound" for this procedure and narrowed the conclusion to the tested approximation and update rule.
We agree that adding seeds to the same 100-step allocator would only measure its failure more precisely. We therefore changed both the allocation method and the replication structure.
The new router and model train together for 200 updates from a shared mid-training checkpoint. All allocation arms receive the same pregenerated training batches and execute the same number of FFN token rows. The router reads the current activations immediately before each FFN and selects exactly 50% of valid token rows.
Every ten steps, we compute the model weight update caused by that mask and evaluate the updated model on the next four ordinary training batches. The router learns by differentiating through this weight update and reducing the observed post-update loss. Those four batches are subsequently used for ordinary model training, so the method does not introduce extra training examples. Their additional evaluations and the backward computation needed to train the router through the weight update are counted as additional computation. The updated model weights and optimizer state become the actual next training state.
This addresses three specific gaps: the model now adapts together with the router; masks are refreshed during training rather than every 100 steps; and the learning signal comes from the actual finite weight update rather than a small gate change with the model held fixed.
Here, "sparse loss" means evaluation with the same 50% FFN execution used during training. On model 8, final sparse losses are 1.2600 for the learned router, 1.2967 for random masks, and 1.2982 for shuffled scores. Learned-router-minus-random is -0.0366, and learned-router-minus-shuffled is -0.0382. On model 9, the losses are 1.2343, 1.2979, and 1.3000, giving differences of -0.0636 and -0.0657. These confirmation models were not used to develop the router.
Across the two models, mean learned-router-minus-random is -0.0501 +/- 0.0135 standard error. Mean learned-router-minus-shuffled is -0.0519 +/- 0.0138 standard error. The learned router is ahead of both controls at steps 50, 100, 150, and 200 in each model. We interpret this as a limited positive counterexample to the original broad interpretation, not as evidence that all allocation methods work.