- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors of the paper propose EPMORE (Explainable Process Mixture-of-Experts), a Transformer-based architecture that is designed to reduce feature entanglement through progressively expanding hidden dimensions, sparse activations, and intermediate supervision. Instead of maintaining a fixed hidden dimension throughout the network, the authors propose to use a hierarchy of representations;
Critical strengths
The authors pose a generally interesting question about feature entanglement in language models and whether architectural modifications can alleviate it.
Critical weaknesses
-
Poor presentation and organization. The paper is extremely difficult to read and follow. Its structure deviates substantially from conventional research-paper organization, with excessive use of bold text, inconsistent formatting, unclear explanations, and poorly connected sections. Overall, the presentation significantly undermines the paper's clarity.
-
Weak experimental methodology. The authors report training approximately 1B-parameter models from scratch using GSM8K, which contains only around 7.5K training examples, without clearly reporting any additional pretraining corpus. It is unclear what meaningful conclusions about language-model architectures can be drawn from such a limited training setup. Furthermore, the paper lacks convincing evaluations across diverse benchmarks and properly trained, competitive baselines.
-
Questionable definition and measurement of entanglement. The proposed operationalization of feature entanglement is insufficiently justified. In particular, higher intrinsic dimensionality and reduced neuron activation overlap do not necessarily imply semantic feature disentanglement. The proposed parameter independence metric also has mathematical problems (see Question 1), making the central empirical conclusions difficult to interpret.
-
Insufficient interpretability evidence. The interpretability claims are mainly supported by qualitative evidence, primarily a single simple factual example.
-
Poor figure quality. Many figures are extremely difficult to read, with very small text and insufficient explanations. Several provide limited useful information, making it difficult to understand the results or independently assess the authors' conclusions.
-
Unconvincing comparisons and theoretical justification. The baseline comparisons do not adequately control for architectural differences, activation functions, or training convergence.
The manuscript feels rushed and insufficiently developed--many strong claims concerning disentanglement, efficiency, and interpretability are not supported by the evidence; the presentation is below conference standards;
Questions for authors
See Weaknesses
Q1: The proposed parameter independence metric is mathematically questionable. By the inclusion-exclusion principle, Equation (2) -- PIL -- simply computes the normalized cardinality of the union of activated neurons, rather than their independence. As a result, the metric can assign higher scores to disjoint activation sets but also to fully overlapping dense activations, making its interpretation ambiguous. Could the authors elaborate on the choice of such a loss term?