acceptodds
Under review as a conference paper at ICLR 2027

Computation-Aligned Parameter Sharing Accelerates Grokking Beyond Weight Decay

Abstract

In grokking, a model memorizes its training data long before it generalizes, and closing the gap can take an order of magnitude more optimization steps. Accelerating grokking would therefore make generalization more compute-efficient. Weight decay is known to drive the transition, but it supplies only a global preference for smaller parameter norms and does not determine which training examples update the same parameters. We introduce ComRoute, a training incentive for computation-aligned parameter sharing on top of weight decay. Given labels for which examples share a reusable rule, ComRoute encourages those examples to route through overlapping experts in a sparse MoE model, so that their learning signals accumulate in shared parameters instead of redundant copies. Theoretically, in a stylized fixed-route model, we characterize generalization time in the small-initialization limit. Computation-aligned routing generalizes earlier than size-matched random routing whenever it strictly improves the threshold's critical learning rate; we also give two sufficient conditions under which a single ungrouped block is slower. Empirically, we show that on a factorized modular-arithmetic task with held-out factor combinations, a 2.3M-parameter MoE trained with ComRoute reaches 90% held-out accuracy after around 23K updates, 1.4 fewer than with size-matched random routing and 6 fewer than with weight decay alone. At 107M parameters and in OLMoE-1B-7B, ComRoute shows a similar overall trend. In short, weight decay helps a reusable solution win eventually, and ComRoute helps it win sooner.

Primary area:
foundation or frontier models, including LLMs
Keywords:
grokking, generalization, mixture-of-experts, computation-aligned routing, weight decay

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dKD7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a method named COMROUTE which aims to incentives aligning learning tasks via shared computational rules to encourage parameter sharing in mixture of experts. COMROUTE is explored in case of modular arithmetic tasks with mod  67\mod 67, where the computational rules are provided. It is evaluated with models ranging from 2.3 Million parameters to an OLMoE-1B-7B. The paper shows that this method is able to reduce the effects of grokking (severely delayed generalisation).

Critical strengths

None

Critical weaknesses
  1. I think this paper can be summed up as when the learning task has additional information about the problem, in this setting the labels of reusable rules, the model is able to generalise better. I do not find this surprising or unexpected in the slightest as additional information has been provided to assist the learning task which should in theory make it easier to learn. Furthermore, the paper provides no methodology of generating such rules for more complicated tasks. Which completely limits the impact of work and its broader implications. As it is unclear how such rules can be discovered. Additionally, there is no comparison to other methods, so even if such rules could be discovered it is unclear if this would out compete current methods for mixture of expert routing or other methods for mitigating grokking. Without such comparisons it is unclear what insights are exactly gained from this work.

  2. Sensitively analysis of the hyperparameters and weighting on the loss is missing, which limits the strength and robustness of the findings. For example, for the loss, L=Ltask+λrouteLroute+λbalLbal+λzLzL = L_{task} + \lambda_{route} L_{route} + \lambda_{bal} L_{bal} + \lambda_{z} L_z, λroute=0.3\lambda_{route}=0.3, λbal=0.01\lambda_{bal}=0.01 and λz=10−4\lambda_{z}=10^{-4} is used for all experiments. Without a sensitivity analysis it can be argued that values used are over tuned for the problem, and in other scenarios the method would not be as effective. This makes it unclear how important each factor is in the resulting performance and how robust the method is. On the point of robust findings, the method is only explored for mod  67\mod67 and no other prime numbers further limiting the insights. For example, for larger/smaller prime numbers, may affect the hyperparameters needed for the method to be effective. Without such analysis it is unclear how robust a method this is in practice, which limits the utility of the findings to the community.

Questions for authors
  1. Could the authors provide a comparison of their method to commonly used mixture of expert methods and grokking mitigation strategies?

  2. Can the authors provide a sensitively analysis of the hyperparameters and weighting on the loss?

  3. Can the authors provide findings over a range of primes for example mod  37\mod 37, mod  97\mod 97, mod  137\mod 137 and mod  503\mod 503?

  4. Can the author provide an explanation of how such computational rules might be discovered beyond current methods that assign experts based on domain/task?

Reviewer_WwyuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ComRoute, a computation-aligned routing objective that encourages examples sharing reusable computations to update overlapping experts, aiming to accelerate grokking beyond weight decay. The paper provides theoretical analysis under a simplified fixed-route model and evaluates the method on a synthetic compositional arithmetic benchmark across different MoE model scales. The theoretical mechanism is relatively clear, and the experiments on smaller models provide convincing evidence of accelerated generalization.

Critical strengths

The motivation is intuitive and well articulated. The idea that examples sharing reusable computations should update overlapping parameters provides a reasonable perspective on accelerating grokking. The paper also offers theoretical analysis and controlled experiments to support this intuition.

Critical weaknesses
  1. Dependence on gold computational group labels. The method assumes prior knowledge of which examples share reusable computations. However, such information is generally unavailable in realistic LLM pretraining, where identifying reusable computations and grouping examples accordingly is itself a challenging problem. This substantially limits the practical applicability of the proposed approach.
  2. Limited generality of the experimental evaluation. The experiments primarily rely on a synthetic arithmetic benchmark explicitly designed around reusable row and column computations, which closely matches the assumptions underlying the method. Although the authors also evaluate a pretrained OLMoE-1B-7B model, the evaluation remains focused on the same synthetic task. It is therefore unclear whether the findings generalize to more realistic language modeling or reasoning settings.
  3. Unclear contribution of computation alignment beyond routing regularization. The results suggest that a substantial portion of the improvement may come from routing regularization itself rather than computation-aware grouping. For example, in the 107M model, matched-random grouping already substantially outperforms weight decay alone, while aligned-column grouping reaches 90% held-out accuracy later than its matched-random counterpart. This raises concerns about the consistency of the additional benefits provided by computation alignment, particularly at larger scales.
Questions for authors
  1. How could the required computational group labels be obtained or approximated in realistic LLM training settings?
  2. Can the authors clarify whether the proposed mechanism is expected to generalize to tasks without an explicitly factorized computational structure? What evidence supports this expectation?
  3. How do the authors distinguish the benefits of computation alignment from those of general routing regularization, especially given the inconsistent improvements over matched-random grouping at larger scales?
Reviewer_QhC7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to shorten the memorization-to-generalization delay in grokking. Beyond weight decay, which only favors parameter-efficient solutions, the authors propose COMROUTE, an auxiliary routing loss for sparse MoE that encourages examples sharing a known reusable rule to route to overlapping experts, so their gradients accumulate in shared parameters. A stylized fixed-route theory shows generalization time scales as log(α⁻²)/(2r), where the growth rate r increases when same-rule examples are pooled into one parameter block. On a factorized modular-arithmetic benchmark with held-out factor combinations, COMROUTE reaches 90% held-out accuracy ~1.4× sooner than size-matched random grouping and ~6× sooner than weight decay alone at 2.3M parameters, with weaker but similar trends at 107M and in OLMoE-1B-7B. Expert analyses show the aligned routes implement factor-specific functions during the transition, which later merge.

Critical strengths
  • Novel angle on grokking. Prior work centers on weight decay (what the optimizer favors). This paper asks where the reusable computation is learned, i.e., which examples update the same parameters, and shows this is a separate, controllable factor.
  • Practically relevant architecture. The intervention is implemented on sparse MoE routing, so the question connects to models actually used at scale rather than only toy MLPs.
  • Coherent, well-scoped story. Theory, gradient-flow simulation, experiments at three scales, and mechanistic checkpoint analysis all point to the same mechanism, and the authors explicitly limit their claims to the timing of the transition rather than final accuracy.
Critical weaknesses
  • Dependence on gold group labels. COMROUTE requires knowing in advance which examples share a reusable rule. The paper offers no way to infer such groups, so as a method it is not applicable to realistic tasks; the contribution reads as an analysis paper rather than a method paper.
  • Unexplained negative result at scale M. Computation-aligned column grouping underperforms its matched-random control (−4.5 points), and most of the speedup at this scale appears to come from imposing any routing constraint. No analysis is given.
  • No comparison with existing grokking-acceleration methods [1,2,3, 4].

[1] Grokfast: Accelerated Grokking by Amplifying Slow Gradients. arXiv:2405.20233, 2024.

[2] Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Model. ICLR 2025.

[3] Egalitarian Gradient Descent: A Simple Approach to Accelerated Grokking. ICLR 2026.

[4] Bridging Lottery Ticket and Grokking: Understanding Grokking from Inner Structure of Networks. TMLR 2025.

Questions for authors
  • How many independent runs underlie each averaged curve? The number of seeds is not stated.
  • At XS, the four grouped settings branch from the weight-decay parent’s interpolation checkpoint (App. I.4). How does the “6× fewer than weight decay alone” claim change if grouped settings are trained from scratch?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.