- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper presents a masking approach for the feed-forward passes in a transformer model attention block's fully connected layers, based on choosing high-probability configurations in the exponentially large state space of masking configurations. The paper presents the NORDER-II method, which constructs a probability distribution over the state space and then chooses the highest probability states, without enumerating the exponentially large state space. Authors experiment on three blocks of the Llama 3.2-1B and Gemma models, reaching comparable or improved predictive performance over several baseline masking methods, and comparable performance to the baseline un-masked model. The method is also shown to be empirically equivalent in computational costs w.r.t. the baseline masking methods.
Critical strengths
The idea presented is very interesting and could lead to efficiency gains in terms of computations in terms of throughput FLOPS in transformers. Such gains could be highly valuable, given that much of the computational costs over the lifetime of a transformer lie in the inference computations. The usage of an algorithm that can find the highest-probability states in a exponentially large state space and the usage of a joint state space, and then applying this to masking is also a very interesting and, to my knowledge, novel idea that is worth exploring further.
Critical weaknesses
- The paper is not sufficiently clear and rigorous in defining the method presented. Key aspects are left unclear on how the method works, justifications for choices made are not given, and where the novelty of the work lies. A key idea in the paper is this ability to sort an exponentially large state space by state probabilities, which is highly interesting but not well explained. The paper builds on the ORDER-II method but does not define it in the main text, nor the key operations used. At least the general idea should be presented, and the reader referred to the original publication. This is necessary for the reader in order to distinguish what is the novel contribution that NORDER-II makes on top of ORDER-II. Furthermore, many definitions and equations contain notation that is not defined, and key choices are not properly justified. For example, a key aspect of the method is having a sensible masking probability distribution, but the choices for the masking probability distribution used in the paper are not well justified or explained. Another example is how the masking groups are formed, and therefore what is being masked in the MLPs. These factors make the results difficult to reproduce without substantial guesswork, and difficult to evaluate the contribution.
- The overall motivation is left unclear in the main text and experimental results. The abstract states that the method opens a framework for probability-mass controlled computation, but it is left unclear what are the actual benefits of this are. My guess is that it is reduction in FLOPS, but I did not see any experiments comparing FLOPS of the masked approach compared to the original model. Furthermore, it appears to me that the construction of the mask itself requires evaluating the full network and then masking outputs (masked output requires construction of mask for layer , but construction of requires the un-masked output of ), eliminating any computational gains to be had as far as I can see.
Questions for authors
- What is the motivation for choosing this form for the masking probability distribution construction outlined in the paper (Eq. 5-6)?
- Is the NORDER-II (or ORDER-II) algorithm optimal in finding the correct ordering of the highest probability states by decreasing probability?
- Are efficiency gains in terms of FLOPS the main motivation for the method? If so, what are the efficiency gains in FLOPS that NORDER-II provides over the baseline model? How many groups are activated on average for the states chosen?
- How is the weight group partitioning done in practice for the Llama and Gemma model experiments? There are several approaches I can imagine. For example for G=12, 1) Each intermediate MLP layer (gate_proj, down_proj, up_proj) in every block independently partitioned into 12 groups 2) Combined weights of a single block's MLP are divided into 12 groups 3) Is the set of weights in all three blocks combined divided into 12 groups?
- Could you provide the mathematical definitions for , as well as an explanation of what refers to (Eq. 6)?
- Why are only 3 blocks experimented with, and not the whole model?
- Were experiments repeated for multiple random seeds, and if so, what are the means and standard deviations for the results?
- Did the authors empirically test how much wall-clock time does the method require compared to the baseline model?