- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TGRank, a task-guided framework for adaptive rank learning in low-rank tensor representations embedded within neural networks. Instead of fixing the Tucker or CP ranks before training, TGRank treats rank as a discrete structural variable and updates it during task optimization. The key observation is that rank decrease and rank increase are asymmetric: removing an existing rank unit can be evaluated using its task sensitivity, while a newly introduced unit has not yet been optimized and therefore requires separate candidate screening and short-term calibration. Accordingly, the method uses suppression sensitivity to guide rank reduction, and combines gradient-based candidate selection with calibrated task utility to guide rank increase. These local adjacent-rank updates allow different representation sites and tensor modes to learn nonuniform task-dependent ranks during training. The method is evaluated on super-resolution, image classification, and semantic segmentation using both Tucker and CP representations. Experiments show that TGRank consistently outperforms matched fixed-rank counterparts while often using smaller total ranks, and can either increase or decrease rank depending on the initial capacity and task demand. Additional analyses validate the proposed rank evidence, examine rank-learning dynamics, and show that both the learned rank structure and the adaptive optimization path contribute to the final performance.
Critical strengths
1.The paper presents a novel and well-motivated perspective on rank learning for low-rank tensor representations. Instead of treating tensor ranks as fixed hyperparameters determined before training, the proposed TGRank framework makes them task-dependent structural variables that can be adjusted during optimization. In particular, the asymmetric treatment of rank decrease and rank increase is conceptually meaningful, since existing components have already been task-optimized whereas newly introduced components have not. This design gives the method a clear methodological novelty beyond conventional fixed-rank tensor representations and generic rank-selection strategies.
2.The method addresses the heterogeneous representation requirements across different stages of a backbone network. Intermediate features at different stages may require different low-rank representation capacities, and imposing the same or manually predefined rank across stages can therefore be suboptimal. TGRank allows the rank of each stage and tensor mode to adapt according to downstream task evidence, leading to nonuniform task-dependent rank allocation. This is practically meaningful because it better matches the varying complexity of intermediate features across the network while maintaining or improving task performance.
Critical weaknesses
-
The comparison with prior adaptive rank/structure learning methods is still insufficient. Existing work has already explored automatic rank determination and tensor-network rank/structure search, but the current baselines are mainly limited to HOSVD-, MARS-, and FlexLoRA-based variants. More direct comparisons would help better establish the novelty and empirical advantage of TGRank.
-
The rank adaptation still relies on several manually specified design choices. The update procedure depends on predefined rank bounds, evaluation schedules, and the thresholds. This raises concerns about the robustness and generalizability of the method across different tasks, backbones, and model scales.
Questions for authors
- The main contribution of the paper is task-guided adaptive rank learning; however, adaptive rank selection, automatic rank determination, and tensor-network rank/structure search have already been extensively studied in the literature. For example, Zheng et al. proposed SVDinsTN for efficient tensor-network structure search through a regularized modeling framework. The current experimental comparisons mainly include HOSVD-based, MARS-based, and FlexLoRA-based variants. It would therefore be helpful for the authors to adapt representative existing adaptive tensor rank/structure learning methods, such as SVDinsTN, to the current setting and include them as more direct baselines. Such comparisons would provide a clearer assessment of the advantages of TGRank over prior adaptive rank/structure learning approaches and help better establish the novelty and empirical significance of the proposed method.
- Although TGRank learns the final ranks during training, the update process depends on predefined rank bounds, a prescribed evaluation schedule, the thresholds and , the persistence parameter , and the calibration horizon . The default setting uses , , together with fixed support thresholds. It would be useful to further investigate how sensitive the learned ranks are to these manually specified design choices. In particular, it remains unclear whether the same update rules and hyperparameter settings would remain effective when transferring to different tasks, backbone architectures, or larger-scale models. This issue is important for assessing the generality and practical robustness of the proposed rank adaptation scheme.
Thank you for considering the SVDinsTN-inspired approach. I have no further questions or concerns, and I recommend acceptance.