acceptodds
Under review as a conference paper at ICLR 2027

TGRank: Task-Guided Rank Learning for Low-Rank Tensor Representations

Abstract

Low-rank tensor representations model multidimensional feature tensors through factorized structures parameterized by continuous factorization variables and discrete ranks. While the factorization variables can be optimized end to end under task supervision, the ranks are typically prescribed before training and held fixed thereafter. Learning these ranks is nontrivial because rank decrease and rank increase operate on structures with different optimization histories: the former evaluates task-trained structure, whereas the latter must assess candidate structure whose task value has not yet been established. Motivated by this asymmetry, we propose TGRank, a task-guided framework that learns ranks during task optimization through adjacent rank changes. Specifically, TGRank employs direction-specific task evidence: rank decrease is guided by the task consequence of suppressing trained structure, whereas rank increase is guided by the task utility revealed after brief adaptation of candidate structure. Extensive experiments across super-resolution, image classification, and semantic segmentation demonstrate that TGRank learns nonuniform task-adaptive ranks and consistently outperforms matched fixed-rank counterparts.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Low-rank tensor representations, rank learning, tensor decomposition, representation learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CbgMOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TGRank, a task-guided framework for adaptive rank learning in low-rank tensor representations embedded within neural networks. Instead of fixing the Tucker or CP ranks before training, TGRank treats rank as a discrete structural variable and updates it during task optimization. The key observation is that rank decrease and rank increase are asymmetric: removing an existing rank unit can be evaluated using its task sensitivity, while a newly introduced unit has not yet been optimized and therefore requires separate candidate screening and short-term calibration. Accordingly, the method uses suppression sensitivity to guide rank reduction, and combines gradient-based candidate selection with calibrated task utility to guide rank increase. These local adjacent-rank updates allow different representation sites and tensor modes to learn nonuniform task-dependent ranks during training. The method is evaluated on super-resolution, image classification, and semantic segmentation using both Tucker and CP representations. Experiments show that TGRank consistently outperforms matched fixed-rank counterparts while often using smaller total ranks, and can either increase or decrease rank depending on the initial capacity and task demand. Additional analyses validate the proposed rank evidence, examine rank-learning dynamics, and show that both the learned rank structure and the adaptive optimization path contribute to the final performance.

Critical strengths

1.The paper presents a novel and well-motivated perspective on rank learning for low-rank tensor representations. Instead of treating tensor ranks as fixed hyperparameters determined before training, the proposed TGRank framework makes them task-dependent structural variables that can be adjusted during optimization. In particular, the asymmetric treatment of rank decrease and rank increase is conceptually meaningful, since existing components have already been task-optimized whereas newly introduced components have not. This design gives the method a clear methodological novelty beyond conventional fixed-rank tensor representations and generic rank-selection strategies.

2.The method addresses the heterogeneous representation requirements across different stages of a backbone network. Intermediate features at different stages may require different low-rank representation capacities, and imposing the same or manually predefined rank across stages can therefore be suboptimal. TGRank allows the rank of each stage and tensor mode to adapt according to downstream task evidence, leading to nonuniform task-dependent rank allocation. This is practically meaningful because it better matches the varying complexity of intermediate features across the network while maintaining or improving task performance.

Critical weaknesses
  1. The comparison with prior adaptive rank/structure learning methods is still insufficient. Existing work has already explored automatic rank determination and tensor-network rank/structure search, but the current baselines are mainly limited to HOSVD-, MARS-, and FlexLoRA-based variants. More direct comparisons would help better establish the novelty and empirical advantage of TGRank.

  2. The rank adaptation still relies on several manually specified design choices. The update procedure depends on predefined rank bounds, evaluation schedules, and the thresholds. This raises concerns about the robustness and generalizability of the method across different tasks, backbones, and model scales.

Questions for authors
  1. The main contribution of the paper is task-guided adaptive rank learning; however, adaptive rank selection, automatic rank determination, and tensor-network rank/structure search have already been extensively studied in the literature. For example, Zheng et al. proposed SVDinsTN for efficient tensor-network structure search through a regularized modeling framework. The current experimental comparisons mainly include HOSVD-based, MARS-based, and FlexLoRA-based variants. It would therefore be helpful for the authors to adapt representative existing adaptive tensor rank/structure learning methods, such as SVDinsTN, to the current setting and include them as more direct baselines. Such comparisons would provide a clearer assessment of the advantages of TGRank over prior adaptive rank/structure learning approaches and help better establish the novelty and empirical significance of the proposed method.
  2. Although TGRank learns the final ranks during training, the update process depends on predefined rank bounds, a prescribed evaluation schedule, the thresholds τ−\tau_- and τ+\tau_+, the persistence parameter KK, and the calibration horizon HH. The default setting uses H=2H=2, K=3K=3, together with fixed support thresholds. It would be useful to further investigate how sensitive the learned ranks are to these manually specified design choices. In particular, it remains unclear whether the same update rules and hyperparameter settings would remain effective when transferring to different tasks, backbone architectures, or larger-scale models. This issue is important for assessing the generality and practical robustness of the proposed rank adaptation scheme.
1 comment

Thank you for considering the SVDinsTN-inspired approach. I have no further questions or concerns, and I recommend acceptance.

Reviewer_qzCUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Traditional low rank tensor methods typically set the rank to a fixed value before training. To address this problem, this paper proposes TGRank, a task guided rank learning framework. During the training process, it dynamically adjusts its rank through increasing or decreasing one unit at a time. Experiments on the visual tasks of super-resolution, image classification, and semantic segmentation have shown that TGRank can learn non-uniform rank structures. Compared with fixed rank models aligned with parameter quantities, TGRank can achieve better task performance while reducing the total rank.

Critical strengths
  • The paper has a proper logical flow and motivation. It first demonstrates the phenomenon of pre-specified ranks through rank-performance responses across representation sites (Figure 1), then highlights the asymmetry in rank learning, and finally naturally deduces TGRank. 

  • The paper gives the novel Insight into asymmetry of rank decrease and rank increase. Driven by this, it gives specific approaches for both directions.

  • TGRank autonomously decides whether to expand or compress based on task feedback, improving task performance while simultaneously reducing the overall model scale.

Critical weaknesses
  • Rank increase first uses zero state derivatives to screen candidate directions, and decides whether to formally introduce them based on the actual task benefits they bring. Rank decrease uses first-order Taylor approximation to determine which rank unit has the least impact on the task after deletion. Does this method have any theoretical basis or advantages.
  • The rank-performance responses across representation sites demonstrate that, under the same retraining protocol, the rank required to reach performance saturation is not always the same as the pre-training rank and varies across representation sites. This motivates treating rank as a learnable structural variable during task optimization. However, there is a lack of direct evidence in the paper proving the claimed asymmetry in rank learning. -The paper uses the calibration steps of only H=2. However, the introduced structures might require a longer period of co-adaptation with the existing network to reveal their true value. Using the loss reduction from just 2 gradient steps to represent long-term task utility is likely to underestimate candidate directions that perform poorly initially but have the long-term potential, or overestimate directions that easily decrease loss early on but eventually perform poorly. This makes the method prone to local optima.
  • Although the paper claims that the training time only increases by 0.9%, Table 5 shows that the rank-evaluation cost increases significantly from 18.4s to 41.8s as H increases.
  • Does the search strategy restrict update efficiency. For instance, if the initial rank of a tensor is set to 4 but the task actually requires a capacity of 20, the model must undergo 16 consecutive evaluations to increase the rank to the target value. This may cause the network to remain in an underfitting state for a long period.
  • It seems that TGRank introduces a large number of hyperparameters, which may make the method highly dependent on expert experience, such as τ+\tau_+,τ−\tau_-, calibration steps H,(ρS,ρP,ρU\rho_S, \rho_P, \rho_U). Thus, the problem of empirically predefining the rank has been replaced by the problem of empirically setting these new hyperparameters.
  • According to the experimental setup, the backbones used are MambaIRv2-Light, ResNet-18, and SegFormer-B0. The experimental evaluation is limited to relatively lightweight models and vision tasks. The paper lacks validation of the proposed method on larger-scale models, which leaves its scalability and general applicability unverified.
  • The clarity and readability of the terminology could be improved to help readers from different domains. For example, the term "site" appears in the introduction but its detailed definition is only provided in Section 4.1 (Representation setting).
Questions for authors
  • Does this asymmetric mechanism have theoretical foundations or proven advantages?
  • Since the rank-performance responses only show varying saturation ranks, what direct evidence in the paper explicitly proves the claimed asymmetry in rank learning?
  • Given that new structures often require long-term co-adaptation, does using only H=2 calibration steps to estimate long-term task utility risk falling into local optima by underestimating or overestimating candidates?
  • Although the overall training time increases by only 0.9%, Table 5 shows the cost rises significantly (from 18.4s to 41.8s); is this overhead acceptable in practice?
  • Does the restriction of applying one rank change per evaluation limit update efficiency?
  • Since TGRank introduces numerous hyperparameters such as τ+\tau_+,τ−\tau_-, calibration steps H,(ρS,ρP,ρU\rho_S, \rho_P, \rho_U), has the problem of empirically predefining the rank simply been replaced by the problem of empirically tuning these new hyperparameters?
  • Could the scalability and general applicability of TGRank be further verified on larger models?
  • Could the clarity of the terminology be improved for cross-domain readers, given that the term "site" appears in the introduction but is only detailed in Section 4.1?
Reviewer_g35vOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem of dynamically learning the discrete ranks of low-rank tensor representations (specifically Tucker and CP factorizations) jointly with continuous network parameters during downstream task training,with highlighting an optimization asymmetry between rank decrease and rank increase. To address this, the paper proposes TGRank, an adjacent-rank learning framework driven by direction-specific task evidence. Specifically, rank decrease uses accumulated first-order Taylor suppression sensitivity while rank increase combines zero-state gradient magnitude screening with short-horizon calibration utility evaluated on a held-out batch after H adaptation steps. Experiments across three types of tasks demonstrate that TGRank discovers nonuniform rank allocations and consistently improves over matched fixed-rank and adaptive baselines with negligible training overhead.

Critical strengths
  • The motivation is clear, specially demonstrated empirically in Figure 1, where saturation ranks vary across representation sites after end-to-end training.

*Formulating rank adaptation around the asymmetry in optimization histories between existing and newly introduced rank units is intuitive and well-articulated.

  • Thorough empirical verification of task evidence. Section 5.1 (Table 4) and Section 5.3 (Table 5) go beyond reporting final downstream metrics by directly validating whether suppression sensitivty and calibration unitility correlate with their intended intervention targets (held-out PSNR drop, Recall@1 of best-gain candidates, and beneficial-increase precision).
Critical weaknesses
  • Lack of self-containment regarding candidate generation and parameterization in Sec. 3.2): While the scalar scaling factors ac,ia_{c,i} and gc,jg_{c,j} are clearly defined, the main text omits critical details on how the candidate rank units ΔZc,i\Delta Z_{c,i} are constructed and parameterized during rank increase. How many candidate units i are maintained per coordinate, how their underlying tensor factors are initialized (e.g., random Gaussian, orthogonal complement, or SVD-based?), and how their norms are constrained so that Pc,iP_{c,i} is comparable across candidates, are unclear.

*The details of tie-breaking and scalability of single-coordinate updates are not reported. At each scheduled evaluation, TGRank restricts updates to at most one coordinate changing by one unit across the entire network (Eq. 2 and Section 3.3). First, the main text does not specify the global selection rule when multiple coordinates simultaneously satisfy q^-<\tau_{-} and q^+ > \tau_{+}. Second, while changing at most one rank unit per evaluation works for 4 representation sites, how does this one-hot adjacent update scale to deeper architectures with dozens of tensorized layers or much larger initial ranks?

*The conceptual asymmetry between pruning trained parameters and growing zero-initialized parameters has been similarly discussed in dynamic sparse training (e.g., RigL, Firefly) and adaptive LoRA literature. A brief discussion connecting TGRank's asymmetric criteria to dynamic network growth/pruning would strengthen the positioning.

Questions for authors
  • Could you clarify in detail how the candidate pool {ΔZ\Delta Z} for rank increase is generated, how many candidates are evaluated per coordinate in Eq. (7), and how you prevent the scale/norm of ΔZc,i\Delta Z_{c,i} from biasing the zero-state derivative PciP_{ci}

  • When multiple candidates satisfy the persistence and the threshold condition at a scheduled evaluation, how does TGRank select which single coordinate and direction to update?

  • Regarding the computational complexity, could you provide the exact parameter counts and FLOPs (both for the low-rank tensor modules and the overall network) corresponding to the learned nonuniform ranks in Tables 1, 2, and 3, rather than only reporting the sum of ranks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.