acceptodds
Withdrawn from ICLR 2027

Boltzmann-MoE: Energy-based mixture-of-experts with native routing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In spite of the dominance of transformers in advanced AI models, our theoretical understanding of their inner workings from a statistical modeling perspective is still incomplete. In this work we show that transformers may be related to a very fundamental class of statistical models. Taking a first principles approach, we introduce the "Boltzmann-MoE", an energy-based model to learn both the overall data distribution, and which adapts itself to the context tokens. Boltzmann-MoE is a generalization of the Energy Transformer (ET) – a variant of looped transformers with some weight-tying – and naturally includes a mixture-of-experts with a native router. The Boltzmann-MoE energy is surprisingly simple and intuitive, consisting of two "free energies" (negative log-partition functions): the ET attention energy, which is a KDE on the context tokens, and the MoE energy is the log sum of Boltzmann weights measuring alignment of the token with memorized patterns. The forward pass of Boltzmann-MoE is a preconditioned gradient update, yielding softmax attention, softmax gating of the experts, and experts that combine a two-layer MLP with a gated linear unit. With a preconditioner split into dissipative and rotational parts, this update is a port-Hamiltonian flow whose learned rotational part adds deterministic exploration of the energy landscape while keeping the forward pass end-to-end differentiable. Our model is an alternative to standard transformers and MoE grounded in statistical physics. At M–M active parameters on Nemotron-CC, hybrids with a standard transformer encoder followed by Boltzmann-MoE blocks, which clearly outperform pure energy models, trail an iso-active Switch-MoE by – points of average zero-shot accuracy and need about twice its tokens to reach the same loss. In deep models, distinct energy blocks learn update fields – more aligned than those of standard transformer layers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture-of-experts, energy-based models, energy transformer, Boltzmann routing, free energy, associative memory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4PYNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an energy-based Transformer-like architecture from the unified perspective of free energy where MoE-like module and attention-like module naturally emerge from the gradients of free energy functions. The authors further provide theoretical explanations and analyses of the resulting routing mechanisms and representation updates.

Critical strengths
  1. The paper provides relatively clear and accessible explanations of the relevant background and theoretical principles, which help readers better understand the motivation behind the proposed architectural designs.

  2. The authors use free energy functions to provide a unified formulation for constructing and interpreting Attention and MoE modules. They also offer theoretical insights into the proposed architecture and explain the rationale behind its design choices.

  3. Beyond conventional language modeling metrics and downstream task accuracy, the authors provide insightful experimental analyses of the model's internal update behavior, offering an additional perspective on understanding the proposed architecture.

Critical weaknesses
  1. My main concern is that the motivation presented in the Introduction is not sufficiently clear, which caused some confusion for me. For example, the introduction gives me the impression that the authors aim to develop an energy-based model (EBM), and the authors also state that “we build an EBM …”. However, my understanding of the proposed method and experiments is that the energy-based formulation primarily serves to guide the design of the model architecture, while the model is still trained using next-token prediction or token-level cross-entropy loss, with generation following the standard autoregressive paradigm rather than sampling from the learned energy distribution. This distinction could benefit from further clarification.

  2. Related to the previous concern, since the energy functions are trained solely through the language modeling loss, I wonder how the learned energy functions relate to the language modeling objective in terms of capturing the distribution of hidden representations of the training data. A more detailed discussion of the relationship between these two aspects would help strengthen the theoretical motivation of the proposed approach.

  3. Some of the design choices could be discussed in greater detail to help readers better understand the proposed framework. For example, in Section 3.2, while I understand the connection between the maximum entropy principle and the Boltzmann distribution, the choice of constraint functions is equally important. The motivation behind the specific expectation constraints imposed on the joint distribution in Proposition 1 can be further discussed.

  4. I also have some concerns regarding the hyperparameter settings in the experiments, particularly how the learning rate and batch size were selected for different model sizes and architectures:

  • (a) The optimal learning rate may vary across model sizes and architectures. I noticed that the authors conducted a learning-rate sweep primarily on the order of 10^-3. However, I am concerned that these learning rates might be relatively large for some configurations. Reporting additional results with smaller learning rates would make the experimental comparisons more convincing.

  • (b) Furthermore, since Boltzmann-MoE introduces different update dynamics and batch-dependent Sinkhorn routing, it remains unclear how sensitive its performance is to batch size and whether its optimal hyperparameter settings are comparable to those of standard Transformer or Switch-MoE architectures. A more systematic sensitivity analysis would help establish the robustness and fairness of the experimental comparisons.

Questions for authors

see Weakness.

Reviewer_pN73OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

To be honest, I don’t understand what this paper is doing. This paper mixes free energy theories to attention and MoE without proper motivations and explanations. The writing is bad and the experimental results are bad. Reading this is a torture.

Critical strengths

I don’t see any. It tells people what we should not to do. Negative results are also a contribution I guess.

Critical weaknesses
  • I feel a sense of AI scientist in this paper, possibly as the first author for the whole research.
  • The results are bad. Basically, the experimental results and analyses just tell us this method does not work and could not scale.
  • The settings are bad and inconsistent, without clear ladder guidance for all model training.
  • The appendix contains lots of raw materials and some names borrowed from the code, which is weird to see these in an academic paper.
Questions for authors

Please read some decent papers and improve your academic agent.

Reviewer_NPxnOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes an energy formulation of transformers, most interestingly with a derivation of a Mixture-of-expert routing from a free energy.

I had no fun at all reading the paper. The authors have relied heavily on AI generation, and lots of the writing is very hard to penetrate. there are duplicated points made, bolded statements which contain undefined terms, Figure 1 is not even referenced from the text. I have spent 90 minutes trying to penetrate the gen-ai narrative to get to the core of the idea, and have run out of enthusiasm for the work.

Critical strengths

An energy-based formulation for transformers where a MoE component is derived from a free energy - this seems like an interesting contribution. The softmax-as-a-derivative-of-logsumexp idea is not new, though.

Critical weaknesses

the writing is really hard to read, and feels AI generated. Example from the abstract: "" At 130M–315MactiveparametersonNemotron-CC,hybridswithastandardtrans formerencoder followedbyBoltzmann-MoEblocks,whichclearlyoutperform pureenergymodels, trailaniso-activeSwitch-MoEby0.1–1.3pointsofaverage zero-shotaccuracyandneedabouttwiceitstokenstoreachthesameloss. Indeep models,distinctenergyblockslearnupdatefields3–4×morealignedthanthose ofstandardtransformerlayers."""

what are "distinct energy blocks"? iso-MoE i assume this means isotropically balanced MoE? what do you mean my more aligned? Overall, the abstract reads like an AI generated summary, and does not help the human reader understand the main discovery or contribution of the paper.

"""wheretheprobabilitydensityishigh,converselywheretheenergyislow.""" I cannot believe this was a human written sentence, it is tautological in the extreme.

there are so many undefined terms that are difficult for a human reader to grok. Example: you bold this sentence: "statesaretheKexpertsofamixture" but you have not really defined what states are yet.

When I reach section 3, the voice changes - it's as if section 3 is written by a different (non AI?) author. Suddenly, there is a logical order to your exposition. I conclude that this section was written by a human, and the intro/related section are AI generated.

You talk about the sign convention multiple times. Clearly you missed this part of revieing the AI draft.

The experiment section is a bullet-point list of comparisons - you have not taken the time to guide the reader through what you're trying to test and understand. These 1-2 line paragraphs are horrible style, straight out of an AI.

There may be some interesting ideas underneath all the ai generated content, but the paper gives me no joy to read.

Questions for authors

""" inBoltzmann-MoEwedefinetheinferencetobeaseriesofgradient-based updatesx←x−∇xE""" and: """Wedidnotchoosethesoftmax;itarosefromdifferentiatingalogsumexp""" This paper from May this year contains many similar ideas, you seem to have missed it: https://arxiv.org/abs/2605.07588

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.