- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MoENet, an adaptive feature aggregation framework that treats hidden representations from different network depths as layer-wise experts. The method employs input-dependent routing and a sample-dependent softmax temperature to dynamically adjust the contribution of each layer to the final prediction. The authors further analyze depth-coupled gradients and adopt soft training for joint optimization. Experiments across ResNet, ConvNeXt, and diverse computer vision tasks demonstrate improved predictive performance and stronger robustness in deep networks.
Critical strengths
-
The research motivation is meaningful, and the method is simple and practical. MoENet exploits complementary information from hidden representations at different depths through a lightweight routing mechanism. The framework can be integrated into existing backbones with relatively low computational overhead.
-
The paper evaluates MoENet across multiple architectures, network depths, and vision tasks. Ablation studies further demonstrate the effectiveness of dynamic routing, projection heads, and adaptive temperature.
Critical weaknesses
-
Prior studies, such as Dynamic Layer Aggregation (AAAI 2019) [1] and LAYA (2025) [2], have explored dynamically aggregating hidden representations from different depths using input-dependent weights. In particular, LAYA also employs layer-wise projections and input-conditioned attention for feature aggregation, adopting a framework similar to MoENet. Although MoENet further introduces an input-adaptive temperature, the paper lacks sufficient discussion and comparison with these related approaches, leaving its independent methodological novelty insufficiently clarified.
-
The paper presents test loss trajectories and performance comparisons across different network depths, showing that MoENet performs better in deeper networks. It also provides a depth-coupled gradient analysis as a possible explanation. However, the current experiments do not directly establish a connection between gradient coupling and overfitting mitigation. Further evidence is needed to support this mechanistic interpretation.
[1] Dou, Z.-Y., Tu, Z., Wang, X., Wang, L., Shi, S., & Zhang, T. (2019). Dynamic Layer Aggregation for Neural Machine Translation with Routing-by-Agreement. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 86–93. https://doi.org/10.1609/aaai.v33i01.330186
[2] Vessio, G. (2025). LAYA: Layer-wise Attention Aggregation for Interpretable Depth-Aware Neural Networks. arXiv preprint arXiv:2511.12723. https://arxiv.org/abs/2511.12723
Questions for authors
-
What is the key methodological novelty of MoENet compared with existing input-dependent layer aggregation methods such as LAYA? What distinct advantages does the input-adaptive temperature provide?
-
Can the authors provide more direct empirical evidence explaining how depth-coupled gradients affect generalization and contribute to mitigating overfitting in deep networks?