- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GEAR, which turns the tangent-normal decomposition into rank evolution through tangent-space optimization, matrix-free normal-spectrum estimation for rank expansion, and task-induced geometric initialization. The method can effectively address the current issue, i.e., existing dynamic-rank methods mainly focus on capacity allocation, leaving unresolved whether the current adaptation space is sufficient for the task. The author conducted sufficient experiments and comparisons across language and vision benchmarks. The results denote that the proposed GEAR achieves the best average performance in all evaluated settings with substantially fewer trainable parameters, which verifies the effectiveness of the proposed GEAR.
Critical strengths
S1: This paper addresses a fundamental issue: existing dynamic-rank methods primarily focus on capacity allocation, leaving unresolved whether the current adaptation space is sufficient for the task. I think the solved problem is meaningful.
S2: The author proposes an effective method, i.e., GEAR. The method can effectively solve the above issues.
S3: This paper provides sufficient experiments and comparisons and finally verifies the effectiveness of the proposed method.
Critical weaknesses
W1: In Theorem 2, the author proposed the Local Optimality of Rank Expansion. However, I think since the loss functions of actual neural networks have complex nonlinear structures, the authors need to further explain the applicability of this theoretical conclusion in practical optimization processes.
W2: The proposed GEAR uses randomized probing to estimate the dominant normal-gradient directions. Although Theorem 3 provides an error bound, the authors have not yet fully demonstrated whether the actual estimation error is small enough to ensure the reliability of rank-extended decisions.
W3: I believe the authors should include some experiments on hyperparameter sensitivity analysis. I think the performance of the proposed GEAR method is influenced by several parameters, including the Probe Width and the prescribed acceptance threshold .
W4: Some of the efficiency analyses are insufficient. In Table 4 of the appendix, GEAR’s training time on DTD is longer than that of LoRA and FlexLoRA. I suggest that the authors provide additional results to support the effectiveness of their method. The authors could present training times and GPU memory usage on large models, e.g., LLaMA3-8B.
Questions for authors
In this paper, the authors proposed an effective method, i.e., GEAR. However, I still have some concerns. I think these concerns should be explained and clarified. Besides, I think the authors should add some extra experiments. Please refer to W1-W4.
Thank you for the rebuttal and the additional comparisons. The results show that GEAR is practical in the reported 8B setting and that its rank-allocation decisions matter. I am keeping my rating of 2 because normal-residual rank growth is established, and the added value of GEAR’s allocation rule over GeoLoRA remains unresolved. At comparable final adapter sizes, does GEAR retain its accuracy advantage when GeoLoRA uses its published global budget allocation?