- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an energy-based Transformer-like architecture from the unified perspective of free energy where MoE-like module and attention-like module naturally emerge from the gradients of free energy functions. The authors further provide theoretical explanations and analyses of the resulting routing mechanisms and representation updates.
Critical strengths
-
The paper provides relatively clear and accessible explanations of the relevant background and theoretical principles, which help readers better understand the motivation behind the proposed architectural designs.
-
The authors use free energy functions to provide a unified formulation for constructing and interpreting Attention and MoE modules. They also offer theoretical insights into the proposed architecture and explain the rationale behind its design choices.
-
Beyond conventional language modeling metrics and downstream task accuracy, the authors provide insightful experimental analyses of the model's internal update behavior, offering an additional perspective on understanding the proposed architecture.
Critical weaknesses
-
My main concern is that the motivation presented in the Introduction is not sufficiently clear, which caused some confusion for me. For example, the introduction gives me the impression that the authors aim to develop an energy-based model (EBM), and the authors also state that “we build an EBM …”. However, my understanding of the proposed method and experiments is that the energy-based formulation primarily serves to guide the design of the model architecture, while the model is still trained using next-token prediction or token-level cross-entropy loss, with generation following the standard autoregressive paradigm rather than sampling from the learned energy distribution. This distinction could benefit from further clarification.
-
Related to the previous concern, since the energy functions are trained solely through the language modeling loss, I wonder how the learned energy functions relate to the language modeling objective in terms of capturing the distribution of hidden representations of the training data. A more detailed discussion of the relationship between these two aspects would help strengthen the theoretical motivation of the proposed approach.
-
Some of the design choices could be discussed in greater detail to help readers better understand the proposed framework. For example, in Section 3.2, while I understand the connection between the maximum entropy principle and the Boltzmann distribution, the choice of constraint functions is equally important. The motivation behind the specific expectation constraints imposed on the joint distribution in Proposition 1 can be further discussed.
-
I also have some concerns regarding the hyperparameter settings in the experiments, particularly how the learning rate and batch size were selected for different model sizes and architectures:
-
(a) The optimal learning rate may vary across model sizes and architectures. I noticed that the authors conducted a learning-rate sweep primarily on the order of 10^-3. However, I am concerned that these learning rates might be relatively large for some configurations. Reporting additional results with smaller learning rates would make the experimental comparisons more convincing.
-
(b) Furthermore, since Boltzmann-MoE introduces different update dynamics and batch-dependent Sinkhorn routing, it remains unclear how sensitive its performance is to batch size and whether its optimal hyperparameter settings are comparable to those of standard Transformer or Switch-MoE architectures. A more systematic sensitivity analysis would help establish the robustness and fairness of the experimental comparisons.
Questions for authors
see Weakness.