acceptodds
Withdrawn from ICLR 2027

MoENet: Hidden Representations as Layer-Wise Experts for Adaptive Aggregation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Network depth remains a pivotal yet sensitive design variable in deep learning, which intermediate layers can yield representations superior to the final layer, while excessive depth invites overfitting and optimization pathologies. We present MoENet, which systematically harnesses hidden representations across depth by treating each layer as a distinct expert and adaptively aggregating them via input-aware routing weights. We further introduce a sample-dependent softmax temperature, so that the sharpness of expert selection is modulated on a per-input basis. Extensive experiments demonstrate consistent gains across diverse backbones (e.g., ResNet, ConvNeXt) at only negligible additional computational and parameter overhead. Beyond accuracy, MoENet markedly exhibits stronger resilience to overfitting at extreme depths. Finally, incorporating MoENet into the representative baselines of seven additional image recognition benchmarks consistently yields significant improvements, underscoring the generality and broad applicability of our approach.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Representation Learning, Mixture of Experts, computer vision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_czAqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MoENet, an adaptive feature aggregation framework that treats hidden representations from different network depths as layer-wise experts. The method employs input-dependent routing and a sample-dependent softmax temperature to dynamically adjust the contribution of each layer to the final prediction. The authors further analyze depth-coupled gradients and adopt soft training for joint optimization. Experiments across ResNet, ConvNeXt, and diverse computer vision tasks demonstrate improved predictive performance and stronger robustness in deep networks.

Critical strengths
  1. The research motivation is meaningful, and the method is simple and practical. MoENet exploits complementary information from hidden representations at different depths through a lightweight routing mechanism. The framework can be integrated into existing backbones with relatively low computational overhead.

  2. The paper evaluates MoENet across multiple architectures, network depths, and vision tasks. Ablation studies further demonstrate the effectiveness of dynamic routing, projection heads, and adaptive temperature.

Critical weaknesses
  1. Prior studies, such as Dynamic Layer Aggregation (AAAI 2019) [1] and LAYA (2025) [2], have explored dynamically aggregating hidden representations from different depths using input-dependent weights. In particular, LAYA also employs layer-wise projections and input-conditioned attention for feature aggregation, adopting a framework similar to MoENet. Although MoENet further introduces an input-adaptive temperature, the paper lacks sufficient discussion and comparison with these related approaches, leaving its independent methodological novelty insufficiently clarified.

  2. The paper presents test loss trajectories and performance comparisons across different network depths, showing that MoENet performs better in deeper networks. It also provides a depth-coupled gradient analysis as a possible explanation. However, the current experiments do not directly establish a connection between gradient coupling and overfitting mitigation. Further evidence is needed to support this mechanistic interpretation.

[1] Dou, Z.-Y., Tu, Z., Wang, X., Wang, L., Shi, S., & Zhang, T. (2019). Dynamic Layer Aggregation for Neural Machine Translation with Routing-by-Agreement. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 86–93. https://doi.org/10.1609/aaai.v33i01.330186

[2] Vessio, G. (2025). LAYA: Layer-wise Attention Aggregation for Interpretable Depth-Aware Neural Networks. arXiv preprint arXiv:2511.12723. https://arxiv.org/abs/2511.12723

Questions for authors
  1. What is the key methodological novelty of MoENet compared with existing input-dependent layer aggregation methods such as LAYA? What distinct advantages does the input-adaptive temperature provide?

  2. Can the authors provide more direct empirical evidence explaining how depth-coupled gradients affect generalization and contribute to mitigating overfitting in deep networks?

Reviewer_gwzwOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoENet, which treats intermediate network representations as depth-wise experts and adaptively aggregates them through input-dependent routing and learnable softmax temperatures. Experiments across multiple visual backbones and recognition tasks report consistent accuracy improvements, particularly for deeper networks, with relatively modest computational overhead.

Critical strengths
  • The idea of reusing existing hidden representations as experts offers a simple and potentially efficient alternative to introducing additional expert networks.

  • The experiments cover diverse architectures, network depths, and downstream tasks, suggesting broad applicability.

  • The ablation studies investigate key architectural and optimization choices, providing useful insights into the effects of dynamic routing, projection heads, and training objectives.

Critical weaknesses
  • Table 1 reports 82.1 mIoU for the Cityscapes baseline, but Appendix B.10 explicitly attributes this result to an Xception-71 model using coarse annotations and test-set evaluation. Appendix B.4 instead specifies a ResNet backbone, fine-only training, and validation-set evaluation.
  • Equations (2) and (3) aggregate intermediate features before a shared prediction head, whereas Equation (8) aggregates predictions from separate expert heads. These formulations are not generally equivalent, particularly under nonlinear transformations, leaving the actual training objective and the applicability of the gradient analysis in Equations (10)–(15) unclear.
  • The sparsity mechanism is mathematically questionable. Equation (16) penalizes router parameters rather than expert-selection weights, while Appendix B.2 introduces an L1 penalty on the routing probabilities that is identically one under the softmax constraint in Equation (5). Consequently, the proposed regularization does not establish the claimed sparse expert selection.
Questions for authors

see wekaness.

Reviewer_gEwvOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes to aggregate hidden layers to improve image recognition tasks. The idea is interesting but already well-discussed, eg. [1,2] - the authors should elaborate more how they differ. The fundamental question is very interesting ("is the final hidden representation necessarily the most suitable one for prediction?"), but neither treated theoretically nor empirically in a sufficient manner. The paper evaluates on 3 toy datasets. Why not use the actual ImageNet-1k or so? Also it only uses two architectures. How about transformers?

[1] Attentive Multi-Layer Fusion for Vision Transformers, ICML 2026 [2] LAYA: Layer-wise Attention Aggregation for Interpretable Depth-Aware Neural Networks, arxiv 2025

Critical strengths

see above

Critical weaknesses

see above

Questions for authors

see above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.