- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Adaptive-Rank Kernels (ARK) to increase and adjust the numerical rank of adaptation updates under a fixed trainable-parameter budget. ARK retains the dimensions of the LoRA factors, applies an elementwise nonlinearity to their product, and learns a kernel scale. The authors define Rank Scaling Density, characterize scale–rank responses under Gaussian random factors, and select Tanh and Swish kernels. Evaluation includes GLUE with BERT and RoBERTa, and GSM8K with Qwen2.5-3B/7B fine-tuned on MetaMathQA, alongside analyses of kernel choice, scale learning, initialization, hybrid kernels, and efficiency.
Critical strengths
-
The problem is important, and the method is simple and practical. Improving update expressiveness with limited adaptation parameters is worthwhile. ARK adds nonlinearities and a small number of scale parameters to a compact factorization, largely preserving the trainable-parameter budget while retaining the ability to merge updates into the backbone.
-
The evaluation offers useful coverage and parameter-budget comparisons. The paper evaluates two encoder backbones and two generative-model sizes, compares multiple PEFT methods, and matches parameter budgets for the main LoRA-family comparisons. The stated local reproduction and separate tuning of baselines provide a reasonable basis for comparison.
-
The paper includes targeted ablations and analyses. Comparisons of fixed versus learned scales, initialization schemes, hybrid kernels, and training rank trajectories support the practical value of several design choices. In particular, the fixed-scale ablations indicate benefits from learning the scale, although the mechanism behind these benefits remains unresolved.
Critical weaknesses
-
Central numerical-rank arguments contain invalid implications, and the theoretical advance is limited. Theorem 6 in Appendix C.9 uses a fixed algebraic-rank ceiling for finite polynomials to exclude scale-dependent numerical rank, although a fixed ceiling does not imply constant numerical rank. For example, the numerical rank of can vary with at fixed factors and budget. Theorems 9 and 10 in C.10 also transfer algebraic-rank bounds for addition and Hadamard products to relative-threshold numerical rank, where these inequalities need not hold. The valid portions largely rely on standard linear algebra, while the screening and composition properties that would distinguish the theoretical contribution remain unestablished. The theoretical advance therefore does not adequately support the paper’s current positioning.
-
Some main results cannot be reconciled with the stated protocol, and the effectiveness of key baseline training needs verification. D.1 specifies binary exact match on all 1,319 GSM8K test problems and the median over three seeds. However, the scores 82.98, 83.21, and 82.82 in Table 2 cannot result from this aggregation after rounding. In Table 1, LoRA’s MRPC and RTE scores are at majority-class accuracy levels, and these tasks account for approximately 70% of the average gain over LoRA. This does not establish training degeneration, but per-seed results, prediction distributions, and training records are needed to rule out ineffective baseline training as an explanation for the main gains.
-
Relevant comparisons are incomplete, and the performance gains are not sufficiently connected to the claimed rank-adaptation mechanism. Kernelized low-dimensional factors producing high-rank updates already appear in KLoRA. Comparisons with relevant recent methods such as GoRA and the SNELLA kernel component are missing, and fidelity to KLoRA’s independently learned coefficients needs confirmation. For unconstrained Tanh factors, the scale can be absorbed into a factor, so the fixed-scale ablation supports an optimization benefit without establishing a rank-adaptation benefit. Moreover, standard Tanh remains near rank on MRPC and RTE despite substantial gains, limiting rank expansion as an explanation for the main improvements.
-
Important differences exist between the paper, implementation, and efficiency interpretation. The Swish implementation differs from the formula by a scale factor that changes during training; the supplied constructor does not directly freeze the fixed- Swish variant; and explicitly setting
--gammachanges the trainable parameter set. Complete main-experiment pipelines are also absent, preventing verification of the actual configurations. Table 3 normalizes efficiency to ARK-Linear, which does not establish overhead relative to standard PEFT LoRA. On Qwen-7B, ARK exceeds DyLoRA by only 0.03 percentage points while reporting higher runtime and memory, leaving a superior accuracy–resource trade-off unestablished.
Questions for authors
-
Theory: Can the authors revise Theorems 6, 9, and 10 and distinguish algebraic-rank guarantees from numerical-rank guarantees? Which provable screening and composition guarantees remain, and what is their theoretical advance over existing results?
-
Main results: Can the authors provide per-seed correct-answer counts, evaluation denominators, and the aggregation script for Table 2 to explain the discrepancies with the three-seed median protocol? Can they also provide LoRA prediction distributions and training records on MRPC and RTE to verify effective baseline training?
-
Related methods and mechanism: Does KLoRA retain the original independently learned slice coefficients? Can the authors provide matched-budget, matched-protocol comparisons with recent rank-adaptation methods such as GoRA and the SNELLA kernel component? How does the available evidence distinguish rank-adaptation benefits from optimization benefits? For symmetric initialization, can they report the spectrum or rank of the actual change ?
-
Implementation and costs: Which implementation and configurations generated the main tables and fixed- ablations, and how do they reconcile the Swish formula, freezing behavior, and
--gammabehavior? Can the authors report end-to-end runtime and peak memory against standard PEFT LoRA and DyLoRA under the actual training settings?
These requests concern clarifications and evidence that could materially change the assessment.
14 comments
R85o: Response to W3/Q3: Why Scale Optimization Enables Adaptive Rank
ARK makes the numerical rank of nonlinear updates controllable and learnable at a fixed factor budget. Our rank-control analysis and new training measurements jointly explain how this happens.
1. An absorbable scale can still be essential to optimization. A positive in can be absorbed into unconstrained factors. Algebraic reparameterizability, however, does not imply equivalent optimization dynamics. Standard LoRA's multiplier is also absorbable into a factor, yet its choice matters: rsLoRA replaces it with and demonstrates markedly different learning behavior as rank grows (Kalajdzievski, 2023). This analysis draws on Tensor Programs IV (Yang and Hu, 2021). Thus, treating a scalar as an explicit optimization coordinate is consequential even when it does not enlarge the representable function class.
ARK extends the familiar role of kernel scale, exemplified by the RBF bandwidth, to Tanh and Swish and introduces a new criterion for selecting that scale: its effect on induced numerical rank. For these non-homogeneous kernels, varying with fixed changes the relative singular-value spectrum, not merely the overall update magnitude. Our Rank Scaling Law (Sec. 4.2) characterizes this response and identifies scales with stable, predictable rank behavior. Learning module-wise gives optimization an explicit handle on that response. Factor absorbability does not imply identical training trajectories.
2. Better optimization is a mechanism for adaptive rank, not a competing explanation. Ordinary LoRA can theoretically yield rank below , but a nondegenerate factor product generically has algebraic rank (Appendix C.1.2, Propositions 1–2); it has no explicit mechanism to allocate rank by module. AdaLoRA addresses this with importance-guided singular-value pruning. ARK instead makes rank adjustment differentiable through the nonlinear kernel scale, and permits upward expansion beyond the factor-rank ceiling as well as downward adjustment, without increasing factor dimensions. The central empirical question is whether separately learning this coordinate actually improves the rank dynamics.
3. Original-protocol paired training directly answers that question. Using the original unnormalized Tanh/fixed-Tanh grid and per-method learning-rate selection under symmetric initialization, the three-task mean is 79.883 (learned ) versus 78.382 (fixed ). Across 9/9 matched task–learning-rate–seed pairs, learning causes larger total numerical-rank movement; 8/9 show greater final inter-module rank dispersion. The mean rank-change statistic rises from 15.67 to 56.67, and inter-module rank IQR from 19.67 to 55.75. RTE improves in all three seeds. Notably, learned has a lower average final median rank (292.33 versus 322.83): it supports more active, module-dependent rank allocation, not indiscriminate rank maximization. This evidence links learned scale, realized rank adaptation, and improved task performance in the original ARK implementation.
A supplementary same-LR, same-seed QQP probe yields rank 100/230.5 and accuracy 88.964/63.183 (learned/fixed); this is not a separately tuned comparison.
4. Direct scale intervention on trained factors. Holding the trained fixed, we varied only across 72 modules in an auxiliary normalized probe. All 72/72 modules changed numerical rank monotonically over scale multipliers ; the pooled median rank progressed 449/489/512/527.5/538.5/550, with a median within-module controllable range of 102. Adjacent rank thresholds preserve the pattern. These trained factors do not require the Gaussian screening assumption; the auxiliary parameterization is distinct from our original-protocol runs.
5. Actual residual updates under symmetric initialization. For symmetric residual initialization, the relevant change is , rather than alone. The paired trajectories measure kernel-rank adaptation, not the unmeasured singular spectrum of . Independently, fixed-factor interventions establish rank controllability, and original-protocol training establishes the benefit of learning that control.
The evidence therefore supports a specific mechanism: an optimization-accessible rank controller that reallocates—and can expand—numerical rank at fixed parameter cost.
Supplementary Results for R85o W3/Q3: Learned Scale and Measured Rank Adaptation
We provide the underlying measurements supporting the mechanism discussed in our preceding response. The key observation is not that learnable scale always increases the final rank, but that it produces more active, module-specific rank adjustment while improving task performance.
1. Paired training results with per-method learning-rate selection. We analyzed the Tanh and fixed-Tanh training records under symmetric residual initialization (BERT-Base-Cased, factor rank ). Learning rates were selected separately for each method and task by the median over three seeds, following the submitted protocol. Each cell below reports learnable / fixed .
| GLUE task | Selected LR (L/F) | Score (L/F) | Rank movement (L/F) | Final rank IQR (L/F) | Final median rank (L/F) |
|---|---|---|---|---|---|
| MRPC | 2e-3 / 2e-3 | 85.294 / 85.539 | 62.5 / 26.5 | 52.75 / 31.0 | 279 / 312 |
| STS-B | 2e-3 / 2e-3 | 88.650 / 88.596 | 65.5 / 19.0 | 78.25 / 20.75 | 294 / 330.5 |
| RTE | 2e-3 / 2e-4 | 65.704 / 61.011 | 42.0 / 1.5 | 36.25 / 7.25 | 304 / 326 |
| Mean | Per-method selection | 79.883 / 78.382 | 56.67 / 15.67 | 55.75 / 19.67 | 292.33 / 322.83 |
Here, rank movement summarizes changes in the numerical rank of the kernel matrix during training; final rank IQR measures the dispersion across adapted modules. Additionally, across nine explicitly matched task–LR–seed training pairs, learnable shows greater rank movement in 9/9, and greater final inter-module rank IQR in 8/9. These matched-pair counts are distinct from the per-method LR-selected summary above. For RTE, all three selected-LR seed scores favor learnable scale: 65.704/60.289, 64.260/61.011, 65.704/62.455 (L/F).
Crucially, the learned controller produces substantially more rank movement and differentiation even though its final median rank is lower. Rank adaptation therefore describes task-dependent adjustment, rather than maximizing rank indiscriminately.
2. Direct intervention on trained factors. To separate scale control from factor updates, we froze in 72 trained modules and multiplied each module's learned by the factors below. This intervention uses normalized kernel inputs.
| multiplier | 0.50x | 0.75x | 1.00x | 1.25x | 1.50x | 2.00x |
|---|---|---|---|---|---|---|
| Pooled median numerical rank | 449 | 489 | 512 | 527.5 | 538.5 | 550 |
All 72/72 modules exhibited a monotone, nonconstant rank response, with median within-module controllable range 102. The monotone response held under neighboring numerical-rank thresholds. Since remain fixed and were learned by training, this directly demonstrates that a scale-only intervention changes induced rank beyond the Gaussian factor distribution used for kernel screening.
Additional GLUE coverage. Across the available eight-task evaluations, learnable scale also achieves a higher aggregate task score. The five additional tasks were measured at a common learning rate and single seed. Across all eight GLUE tasks, the aggregate score is 80.954 with learnable versus 71.535 with fixed (+9.419 points).
Together, these experiments supply quantitative evidence for scale-controlled rank adaptation: the paired training results show more active rank trajectories and higher average task scores with learned ; the fixed-factor intervention independently confirms a direct scale-to-rank response on trained factors. As discussed above, these measurements concern the kernel-matrix rank; for symmetric residual initialization, the actual update is a distinct spectral object.
R85o W3/Q3 — SNELLA Mix-K: Independent Validation and a New Adaptive-Rank Perspective
We thank the reviewer for highlighting SNELLA. Its Mix-K provides an opportunity to demonstrate what ARK contributes beyond designing expressive nonlinear kernels. Our new experiments recognize SNELLA's strengths while revealing a property of Mix-K not explicitly investigated in its original design.
1. Complementary adaptation objectives.
SNELLA contributes to memory-efficient sparse fine-tuning. Its Mix-K combines Piecewise Linear and normalized exponential components for high-rank expressivity and optimization stability, while adaptive bi-level sparsity allocation determines which weights receive the update budget.
ARK studies a different variable: the numerical rank attained by each module under a fixed trainable-parameter budget.
Our original submission established this distinction using SNELL/KLoRA's Piecewise Linear kernel (Sec. 3, Table 4, Apps. C.4–C.9): it attains high numerical rank, yet positive homogeneity makes that rank invariant under common positive scaling.
ARK therefore introduces Rank Scaling Density (Def. 2) to measure induced numerical rank and evaluates stability, monotonicity, and predictability (Sec. 4.2) to identify useful scale responses. We then learn a scale per module and measure attained ranks during fine-tuning.
SNELLA has learnable kernel coefficients, but its reported kernel design and evaluation center on expressivity and optimization stability, not a characterized scale–rank law or a validated module-level rank controller. Its sparsity adaptivity and ARK's numerical-rank adaptivity are complementary.
2. NEW: Mix-K satisfies ARK's selection criteria.
We evaluated Mix-K with our existing rank-scaling pipeline and seven independent factor draws:
Mix-K was designed independently for different objectives, yet ARK identifies it as a promising rank-controllable kernel. This is positive external validation of our selection methodology: a kernel need not be originally designed for numerical-rank learning to be recognized by ARK's measurable criteria. Notably, this result credits Mix-K's strong response rather than treating it as an inferior baseline.
These are fixed-coefficient kernel diagnostics (α=1, β=0, with specified Piecewise Linear coefficients), not a reproduction of SNELLA's learned parameters or downstream training.
3. A new explanation of Mix-K's useful properties.
Removing Mix-K's normalized exponential term recovers Piecewise Linear elementwise (maximum absolute difference 0.0). Their rank responses nevertheless differ:
Thus, the added component introduces rank controllability rather than merely a higher maximum rank. This complements SNELLA's original expressivity–stability explanation, and illustrates why ARK studies the response of the numerical spectrum, not only attainable rank. At D=1024, both constructions approach similar high-rank ceilings; what distinguishes them is whether the attained rank can move across scales.
4. A new method enabled by the ARK framework.
The result suggests ARK-Mix-K: expose a learnable, module-specific pre-kernel scale for Mix-K and optimize it jointly with the low-rank factors. SNELLA supplies an effective kernel; ARK supplies its rank-response characterization, selection criteria, and an explicit route to rank control.
Our screening establishes candidate suitability, not yet the behavior of trained ARK-Mix-K or downstream improvements. This separates a demonstrated kernel property from the follow-up question of how task optimization would exploit it. Nevertheless, it shows how ARK can both reinterpret an existing construction and guide new adaptive-rank designs.
Conclusion. We recognize SNELLA's contributions to high-rank expressivity, optimization stability, and adaptive sparsification. Our independent tests show that ARK provides a complementary, reusable numerical-rank perspective and can discover promising controllable kernels beyond Tanh/Swish.
The shared comment provides the complete measurements. Matched-budget downstream comparisons, KLoRA implementation fidelity, and performance attribution are addressed separately.