- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a method named COMROUTE which aims to incentives aligning learning tasks via shared computational rules to encourage parameter sharing in mixture of experts. COMROUTE is explored in case of modular arithmetic tasks with , where the computational rules are provided. It is evaluated with models ranging from 2.3 Million parameters to an OLMoE-1B-7B. The paper shows that this method is able to reduce the effects of grokking (severely delayed generalisation).
Critical strengths
None
Critical weaknesses
-
I think this paper can be summed up as when the learning task has additional information about the problem, in this setting the labels of reusable rules, the model is able to generalise better. I do not find this surprising or unexpected in the slightest as additional information has been provided to assist the learning task which should in theory make it easier to learn. Furthermore, the paper provides no methodology of generating such rules for more complicated tasks. Which completely limits the impact of work and its broader implications. As it is unclear how such rules can be discovered. Additionally, there is no comparison to other methods, so even if such rules could be discovered it is unclear if this would out compete current methods for mixture of expert routing or other methods for mitigating grokking. Without such comparisons it is unclear what insights are exactly gained from this work.
-
Sensitively analysis of the hyperparameters and weighting on the loss is missing, which limits the strength and robustness of the findings. For example, for the loss, , , and is used for all experiments. Without a sensitivity analysis it can be argued that values used are over tuned for the problem, and in other scenarios the method would not be as effective. This makes it unclear how important each factor is in the resulting performance and how robust the method is. On the point of robust findings, the method is only explored for and no other prime numbers further limiting the insights. For example, for larger/smaller prime numbers, may affect the hyperparameters needed for the method to be effective. Without such analysis it is unclear how robust a method this is in practice, which limits the utility of the findings to the community.
Questions for authors
-
Could the authors provide a comparison of their method to commonly used mixture of expert methods and grokking mitigation strategies?
-
Can the authors provide a sensitively analysis of the hyperparameters and weighting on the loss?
-
Can the authors provide findings over a range of primes for example , , and ?
-
Can the author provide an explanation of how such computational rules might be discovered beyond current methods that assign experts based on domain/task?