acceptodds
Under review as a conference paper at ICLR 2027

LoRA's ARK: Towards an Iso-Budget Rank Scaling Law with Adaptive-Rank Kernels

Abstract

Low-Rank Adaptation and its variants dominate parameter-efficient fine-tuning, yet practitioners still have to guess one hyperparameter: the rank r. A rank too low structurally limits the update; a rank too high buys headroom only by paying proportionally more parameters and memory. Behind both horns lies one coupling: the same r sets how expressive the update can be and how many parameters it takes, so no setting of r improves one without hurting the other. Existing solutions either expand rank through fixed parameterizations that lack task-specific adaptivity, or achieve adaptivity only downward, by pruning a rank budget that must be provisioned up front. We argue that the numerical rank should instead be a property that each module learns under a fixed parameter budget. To this end, we propose Adaptive-Rank Kernels (ARK). Instead of selecting kernels by traditional Mercer validity, which only bounds the attainable rank from above, we select them by the rank they measurably induce in an update synthesized from low-rank factors. We then train the kernel's own scale as a learnable rank controller, so that different modules settle at different numerical ranks under a fixed trainable-parameter count: an iso-budget rank scaling law. On language understanding (eight GLUE tasks with BERT) and mathematical reasoning (GSM8k with Qwen2.5), ARK matches or exceeds the PEFT baselines at a matched trainable-parameter budget. It averages 81.47 on GLUE, 5.3 points above LoRA and 3.1 above the strongest baseline at the same budget, statistically indistinguishable from full fine-tuning (81.72) while training about 1% of the parameters, and it leads the matched-cost methods on GSM8k at both model scales. Measuring where the updates actually settle shows the gain comes with the mechanism we asked for: under one fixed budget the attained numerical ranks spread widely across modules and tasks instead of standing at r, so the rank of the update and the parameter count are decoupled in practice. The code will be publicly available.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Parameter-Efficient Fine-Tuning, Kernel Method

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_R85oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Adaptive-Rank Kernels (ARK) to increase and adjust the numerical rank of adaptation updates under a fixed trainable-parameter budget. ARK retains the dimensions of the LoRA factors, applies an elementwise nonlinearity to their product, and learns a kernel scale. The authors define Rank Scaling Density, characterize scale–rank responses under Gaussian random factors, and select Tanh and Swish kernels. Evaluation includes GLUE with BERT and RoBERTa, and GSM8K with Qwen2.5-3B/7B fine-tuned on MetaMathQA, alongside analyses of kernel choice, scale learning, initialization, hybrid kernels, and efficiency.

Critical strengths
  1. The problem is important, and the method is simple and practical. Improving update expressiveness with limited adaptation parameters is worthwhile. ARK adds nonlinearities and a small number of scale parameters to a compact factorization, largely preserving the trainable-parameter budget while retaining the ability to merge updates into the backbone.

  2. The evaluation offers useful coverage and parameter-budget comparisons. The paper evaluates two encoder backbones and two generative-model sizes, compares multiple PEFT methods, and matches parameter budgets for the main LoRA-family comparisons. The stated local reproduction and separate tuning of baselines provide a reasonable basis for comparison.

  3. The paper includes targeted ablations and analyses. Comparisons of fixed versus learned scales, initialization schemes, hybrid kernels, and training rank trajectories support the practical value of several design choices. In particular, the fixed-scale ablations indicate benefits from learning the scale, although the mechanism behind these benefits remains unresolved.

Critical weaknesses
  1. Central numerical-rank arguments contain invalid implications, and the theoretical advance is limited. Theorem 6 in Appendix C.9 uses a fixed algebraic-rank ceiling for finite polynomials to exclude scale-dependent numerical rank, although a fixed ceiling does not imply constant numerical rank. For example, the numerical rank of γS+(γS)⊙3\gamma S+(\gamma S)^{\odot3} can vary with γ\gamma at fixed factors and budget. Theorems 9 and 10 in C.10 also transfer algebraic-rank bounds for addition and Hadamard products to relative-threshold numerical rank, where these inequalities need not hold. The valid portions largely rely on standard linear algebra, while the screening and composition properties that would distinguish the theoretical contribution remain unestablished. The theoretical advance therefore does not adequately support the paper’s current positioning.

  2. Some main results cannot be reconciled with the stated protocol, and the effectiveness of key baseline training needs verification. D.1 specifies binary exact match on all 1,319 GSM8K test problems and the median over three seeds. However, the scores 82.98, 83.21, and 82.82 in Table 2 cannot result from this aggregation after rounding. In Table 1, LoRA’s MRPC and RTE scores are at majority-class accuracy levels, and these tasks account for approximately 70% of the average gain over LoRA. This does not establish training degeneration, but per-seed results, prediction distributions, and training records are needed to rule out ineffective baseline training as an explanation for the main gains.

  3. Relevant comparisons are incomplete, and the performance gains are not sufficiently connected to the claimed rank-adaptation mechanism. Kernelized low-dimensional factors producing high-rank updates already appear in KLoRA. Comparisons with relevant recent methods such as GoRA and the SNELLA kernel component are missing, and fidelity to KLoRA’s independently learned coefficients needs confirmation. For unconstrained Tanh factors, the scale can be absorbed into a factor, so the fixed-scale ablation supports an optimization benefit without establishing a rank-adaptation benefit. Moreover, standard Tanh remains near rank rr on MRPC and RTE despite substantial gains, limiting rank expansion as an explanation for the main improvements.

  4. Important differences exist between the paper, implementation, and efficiency interpretation. The Swish implementation differs from the formula by a scale factor that changes during training; the supplied constructor does not directly freeze the fixed-γ\gamma Swish variant; and explicitly setting --gamma changes the trainable parameter set. Complete main-experiment pipelines are also absent, preventing verification of the actual configurations. Table 3 normalizes efficiency to ARK-Linear, which does not establish overhead relative to standard PEFT LoRA. On Qwen-7B, ARK exceeds DyLoRA by only 0.03 percentage points while reporting higher runtime and memory, leaving a superior accuracy–resource trade-off unestablished.

Questions for authors
  1. Theory: Can the authors revise Theorems 6, 9, and 10 and distinguish algebraic-rank guarantees from numerical-rank guarantees? Which provable screening and composition guarantees remain, and what is their theoretical advance over existing results?

  2. Main results: Can the authors provide per-seed correct-answer counts, evaluation denominators, and the aggregation script for Table 2 to explain the discrepancies with the three-seed median protocol? Can they also provide LoRA prediction distributions and training records on MRPC and RTE to verify effective baseline training?

  3. Related methods and mechanism: Does KLoRA retain the original independently learned slice coefficients? Can the authors provide matched-budget, matched-protocol comparisons with recent rank-adaptation methods such as GoRA and the SNELLA kernel component? How does the available evidence distinguish rank-adaptation benefits from optimization benefits? For symmetric initialization, can they report the spectrum or rank of the actual change Kt−K0K_t-K_0?

  4. Implementation and costs: Which implementation and configurations generated the main tables and fixed-γ\gamma ablations, and how do they reconcile the Swish formula, freezing behavior, and --gamma behavior? Can the authors report end-to-end runtime and peak memory against standard PEFT LoRA and DyLoRA under the actual training settings?

These requests concern clarifications and evidence that could materially change the assessment.

14 comments
AuthorsOpenReview ↗

R85o W3/Q3 — SNELLA Mix-K: Independent Validation and a New Adaptive-Rank Perspective

We thank the reviewer for highlighting SNELLA. Its Mix-K provides an opportunity to demonstrate what ARK contributes beyond designing expressive nonlinear kernels. Our new experiments recognize SNELLA's strengths while revealing a property of Mix-K not explicitly investigated in its original design.

1. Complementary adaptation objectives.

SNELLA contributes to memory-efficient sparse fine-tuning. Its Mix-K combines Piecewise Linear and normalized exponential components for high-rank expressivity and optimization stability, while adaptive bi-level sparsity allocation determines which weights receive the update budget.

ARK studies a different variable: the numerical rank attained by each module under a fixed trainable-parameter budget.

Our original submission established this distinction using SNELL/KLoRA's Piecewise Linear kernel (Sec. 3, Table 4, Apps. C.4–C.9): it attains high numerical rank, yet positive homogeneity makes that rank invariant under common positive scaling.

ARK therefore introduces Rank Scaling Density (Def. 2) to measure induced numerical rank and evaluates stability, monotonicity, and predictability (Sec. 4.2) to identify useful scale responses. We then learn a scale per module and measure attained ranks during fine-tuning.

SNELLA has learnable kernel coefficients, but its reported kernel design and evaluation center on expressivity and optimization stability, not a characterized scale–rank law or a validated module-level rank controller. Its sparsity adaptivity and ARK's numerical-rank adaptivity are complementary.

2. NEW: Mix-K satisfies ARK's selection criteria.

We evaluated Mix-K with our existing rank-scaling pipeline and seven independent factor draws:

  • D=32, r=4: Rank Scaling Density spans 0.250–8.000, Spearman 0.756, with a predictable double-4PL response.
  • D=1024, r=32: Density spans 0.031–28.848, Spearman 0.974, and 4PL fit R²=0.996.
  • Mix-K passes the tested stability and monotonicity criteria at both widths. At D=1024, its Spearman exceeds ARK-Tanh's 0.918.

Mix-K was designed independently for different objectives, yet ARK identifies it as a promising rank-controllable kernel. This is positive external validation of our selection methodology: a kernel need not be originally designed for numerical-rank learning to be recognized by ARK's measurable criteria. Notably, this result credits Mix-K's strong response rather than treating it as an inferior baseline.

These are fixed-coefficient kernel diagnostics (α=1, β=0, with specified Piecewise Linear coefficients), not a reproduction of SNELLA's learned parameters or downstream training.

3. A new explanation of Mix-K's useful properties.

Removing Mix-K's normalized exponential term recovers Piecewise Linear elementwise (maximum absolute difference 0.0). Their rank responses nevertheless differ:

  • Piecewise Linear: High numerical rank, essentially invariant under positive factor scaling.
  • Mix-K: Comparable high-rank capacity, but a stable, predictable scale-dependent transition.

Thus, the added component introduces rank controllability rather than merely a higher maximum rank. This complements SNELLA's original expressivity–stability explanation, and illustrates why ARK studies the response of the numerical spectrum, not only attainable rank. At D=1024, both constructions approach similar high-rank ceilings; what distinguishes them is whether the attained rank can move across scales.

4. A new method enabled by the ARK framework.

The result suggests ARK-Mix-K: expose a learnable, module-specific pre-kernel scale for Mix-K and optimize it jointly with the low-rank factors. SNELLA supplies an effective kernel; ARK supplies its rank-response characterization, selection criteria, and an explicit route to rank control.

Our screening establishes candidate suitability, not yet the behavior of trained ARK-Mix-K or downstream improvements. This separates a demonstrated kernel property from the follow-up question of how task optimization would exploit it. Nevertheless, it shows how ARK can both reinterpret an existing construction and guide new adaptive-rank designs.

Conclusion. We recognize SNELLA's contributions to high-rank expressivity, optimization stability, and adaptive sparsification. Our independent tests show that ARK provides a complementary, reusable numerical-rank perspective and can discover promising controllable kernels beyond Tanh/Swish.

The shared comment provides the complete measurements. Matched-budget downstream comparisons, KLoRA implementation fidelity, and performance attribution are addressed separately.

AuthorsOpenReview ↗

R85o: Response to W3/Q3: Why Scale Optimization Enables Adaptive Rank

ARK makes the numerical rank of nonlinear updates controllable and learnable at a fixed factor budget. Our rank-control analysis and new training measurements jointly explain how this happens.

1. An absorbable scale can still be essential to optimization. A positive γ\gamma in Kγ(A,B)=tanh⁡(γAB⊤)K_\gamma(A,B)=\tanh(\gamma AB^\top) can be absorbed into unconstrained factors. Algebraic reparameterizability, however, does not imply equivalent optimization dynamics. Standard LoRA's α/r\alpha/r multiplier is also absorbable into a factor, yet its choice matters: rsLoRA replaces it with α/r\alpha/\sqrt r and demonstrates markedly different learning behavior as rank grows (Kalajdzievski, 2023). This analysis draws on Tensor Programs IV (Yang and Hu, 2021). Thus, treating a scalar as an explicit optimization coordinate is consequential even when it does not enlarge the representable function class.

ARK extends the familiar role of kernel scale, exemplified by the RBF bandwidth, to Tanh and Swish and introduces a new criterion for selecting that scale: its effect on induced numerical rank. For these non-homogeneous kernels, varying γ\gamma with A,BA,B fixed changes the relative singular-value spectrum, not merely the overall update magnitude. Our Rank Scaling Law (Sec. 4.2) characterizes this response and identifies scales with stable, predictable rank behavior. Learning γ\gamma module-wise gives optimization an explicit handle on that response. Factor absorbability does not imply identical training trajectories.

2. Better optimization is a mechanism for adaptive rank, not a competing explanation. Ordinary LoRA can theoretically yield rank below rr, but a nondegenerate factor product generically has algebraic rank rr (Appendix C.1.2, Propositions 1–2); it has no explicit mechanism to allocate rank by module. AdaLoRA addresses this with importance-guided singular-value pruning. ARK instead makes rank adjustment differentiable through the nonlinear kernel scale, and permits upward expansion beyond the factor-rank ceiling as well as downward adjustment, without increasing factor dimensions. The central empirical question is whether separately learning this coordinate actually improves the rank dynamics.

3. Original-protocol paired training directly answers that question. Using the original unnormalized Tanh/fixed-Tanh grid and per-method learning-rate selection under symmetric initialization, the three-task mean is 79.883 (learned γ\gamma) versus 78.382 (fixed γ\gamma). Across 9/9 matched task–learning-rate–seed pairs, learning γ\gamma causes larger total numerical-rank movement; 8/9 show greater final inter-module rank dispersion. The mean rank-change statistic rises from 15.67 to 56.67, and inter-module rank IQR from 19.67 to 55.75. RTE improves in all three seeds. Notably, learned γ\gamma has a lower average final median rank (292.33 versus 322.83): it supports more active, module-dependent rank allocation, not indiscriminate rank maximization. This evidence links learned scale, realized rank adaptation, and improved task performance in the original ARK implementation.

A supplementary same-LR, same-seed QQP probe yields rank 100/230.5 and accuracy 88.964/63.183 (learned/fixed); this is not a separately tuned comparison.

4. Direct scale intervention on trained factors. Holding the trained A,BA,B fixed, we varied only γ\gamma across 72 modules in an auxiliary normalized probe. All 72/72 modules changed numerical rank monotonically over scale multipliers {0.5,0.75,1,1.25,1.5,2}\{0.5,0.75,1,1.25,1.5,2\}; the pooled median rank progressed 449/489/512/527.5/538.5/550, with a median within-module controllable range of 102. Adjacent rank thresholds preserve the pattern. These trained factors do not require the Gaussian screening assumption; the auxiliary parameterization is distinct from our original-protocol runs.

5. Actual residual updates under symmetric initialization. For symmetric residual initialization, the relevant change is ΔWt=Kt−K0\Delta W_t=K_t-K_0, rather than KtK_t alone. The paired trajectories measure kernel-rank adaptation, not the unmeasured singular spectrum of Kt−K0K_t-K_0. Independently, fixed-factor interventions establish rank controllability, and original-protocol training establishes the benefit of learning that control.

The evidence therefore supports a specific mechanism: an optimization-accessible rank controller that reallocates—and can expand—numerical rank at fixed parameter cost.

AuthorsOpenReview ↗

Supplementary Results for R85o W3/Q3: Learned Scale and Measured Rank Adaptation

We provide the underlying measurements supporting the mechanism discussed in our preceding response. The key observation is not that learnable scale always increases the final rank, but that it produces more active, module-specific rank adjustment while improving task performance.

1. Paired training results with per-method learning-rate selection. We analyzed the Tanh and fixed-Tanh training records under symmetric residual initialization (BERT-Base-Cased, factor rank r=32r=32). Learning rates were selected separately for each method and task by the median over three seeds, following the submitted protocol. Each cell below reports learnable / fixed γ\gamma.

GLUE taskSelected LR (L/F)Score (L/F)Rank movement (L/F)Final rank IQR (L/F)Final median rank (L/F)
MRPC2e-3 / 2e-385.294 / 85.53962.5 / 26.552.75 / 31.0279 / 312
STS-B2e-3 / 2e-388.650 / 88.59665.5 / 19.078.25 / 20.75294 / 330.5
RTE2e-3 / 2e-465.704 / 61.01142.0 / 1.536.25 / 7.25304 / 326
MeanPer-method selection79.883 / 78.38256.67 / 15.6755.75 / 19.67292.33 / 322.83

Here, rank movement summarizes changes in the numerical rank of the kernel matrix during training; final rank IQR measures the dispersion across adapted modules. Additionally, across nine explicitly matched task–LR–seed training pairs, learnable γ\gamma shows greater rank movement in 9/9, and greater final inter-module rank IQR in 8/9. These matched-pair counts are distinct from the per-method LR-selected summary above. For RTE, all three selected-LR seed scores favor learnable scale: 65.704/60.289, 64.260/61.011, 65.704/62.455 (L/F).

Crucially, the learned controller produces substantially more rank movement and differentiation even though its final median rank is lower. Rank adaptation therefore describes task-dependent adjustment, rather than maximizing rank indiscriminately.

2. Direct intervention on trained factors. To separate scale control from factor updates, we froze A,BA,B in 72 trained modules and multiplied each module's learned γ\gamma by the factors below. This intervention uses normalized kernel inputs.

γ\gamma multiplier0.50x0.75x1.00x1.25x1.50x2.00x
Pooled median numerical rank449489512527.5538.5550

All 72/72 modules exhibited a monotone, nonconstant rank response, with median within-module controllable range 102. The monotone response held under neighboring numerical-rank thresholds. Since A,BA,B remain fixed and were learned by training, this directly demonstrates that a scale-only intervention changes induced rank beyond the Gaussian factor distribution used for kernel screening.

Additional GLUE coverage. Across the available eight-task evaluations, learnable scale also achieves a higher aggregate task score. The five additional tasks were measured at a common learning rate and single seed. Across all eight GLUE tasks, the aggregate score is 80.954 with learnable γ\gamma versus 71.535 with fixed γ\gamma (+9.419 points).

Together, these experiments supply quantitative evidence for scale-controlled rank adaptation: the paired training results show more active rank trajectories and higher average task scores with learned γ\gamma; the fixed-factor intervention independently confirms a direct scale-to-rank response on trained factors. As discussed above, these measurements concern the kernel-matrix rank; for symmetric residual initialization, the actual update Kt−K0K_t-K_0 is a distinct spectral object.

Reviewer_91i4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Adaptive-Rank Kernels (ARK), a PEFT method that aims to break the coupling between parameter budget and attainable adaptation rank. The main idea is to apply a nonlinear kernel transformation to a low-rank update and use a learnable scaling parameter to control the resulting rank. The paper introduces the Iso-Budget Rank Scaling Law, Rank Scaling Density, and Adaptive-Rank Kernels, and empirically evaluates the approach on GLUE and GSM8K. The central claim is that adaptive rank behavior can be achieved without increasing the number of trainable parameters.

Critical strengths
  • The Iso-Budget Rank Scaling Law is an interesting perspective and provides a fresh way to think about rank adaptation under a fixed parameter budget.
  • Using kernel-based transformations for rank learning is a simple yet appealing idea.
  • The paper is clearly written and easy to read. The motivation, method, and experimental setup are well presented.
  • Experimental results are competitive and promising across both language understanding and reasoning benchmarks.
Critical weaknesses

Major

  • Def. 1 only holds in initialization and does not extend to fine-tuning; the connection between Def. 1 and the ARK fine-tuning remains unclear. More generally, there is a gap between the proposed definitions and the optimization process used in ARK. It is unclear to what extent the definition explains the rank behavior observed during and after fine-tuning.
  • Although several definitions are introduced, the practical implications and guarantees of these definitions could be better articulated. In particular, it is unclear what can be concluded theoretically about adaptive rank learning during optimization.
  • The interpretation of the kernelized update is somewhat subtle since kernel methods can correspond to infinite-dimensional feature spaces. The paper could discuss this aspect explicitly.
  • The experimental comparison is missing several representative methods that explicitly target high-rank adaptations, such as HiRA [1], LoHA/FedPara [2], MoKA [3], and RoSA [4].
  • The efficiency tradeoff is not entirely convincing. Unlike LoRA, ARK requires materializing AB⊤AB^\top to apply nonlinearity, preventing the efficient factorized computation A(B⊤v)A(B^\top v). While the reported overhead is small, it is unclear whether this remains true across different architectures, ranks, and training settings.
  • The methodology in Eq. (3) is closely related to prior nonlinear/activation-based LoRA variants such as [5]. The paper should position itself more clearly with respect to that literature and discuss the specific novelty of ARK.

Minor

  • The kernel discussion would benefit from a brief introduction to Mercer/PSD kernels and a clearer comparison with the Mercer-Tanh construction.

[1] HiRA: Parameter-efficient hadamard high-rank adaptation for large language models
[2] Fedpara: Low-rank hadamard product for communication-efficient federated learning
[3] MoKA: mixture of Kronecker adapters
[4] Rosa: Accurate parameter-efficient fine-tuning via robust adaptation
[5] Loran: Improved low-rank adaptation by a non-linear transformation

Questions for authors
  • How does Def. 1 justify the rank behavior observed during and after fine-tuning, given that it is defined as expectation at initialization?
  • What theoretical guarantees do Defs. 1–3 provide for fine-tuning beyond characterizing rank behavior at initialization?
  • How should the proposed kernels be interpreted when the associated feature space may be infinite-dimensional?
  • Can the authors compare against HiRA, FedPara, MoKA, and RoSA?
  • How does ARK differ from prior nonlinearity-/activation-based LoRA method?
  • Can the authors provide additional evidence that the efficiency overhead remains small across different settings?
11 comments
AuthorsOpenReview ↗

91i4: Response to W6/Q5 — Beyond Nonlinear Expressivity: Why ARK Introduces a Distinct Rank-Selection and Learning Framework

We thank the reviewer for highlighting LoRAN. Its Sinter transformation contributes to nonlinear low-rank adaptation by improving weight-update fitting. ARK investigates a different question: which kernels enable predictable, learnable numerical-rank control under a fixed parameter budget? We tested Sinter using ARK's existing framework to examine this distinction directly.

1. From high-rank expressivity to controllable numerical rank.

Our original submission established this distinction through SNELL/KLoRA's Piecewise Linear kernel (Sec. 3; Table 4; Apps. C.4–C.9). Despite producing high-rank updates, its positive homogeneity makes numerical rank invariant under common positive scaling—a property we proved (Thm. 7) and examined experimentally. High-rank capacity does not imply scale-controllable rank.

This is precisely why downstream accuracy or the existence of a high-dimensional feature map cannot, by itself, determine whether a kernel is suitable for rank learning. ARK develops a methodology around this insight. Rank Scaling Density (Def. 2) measures the numerical rank induced by a kernel rather than relying on feature-space dimensionality. Sec. 4.2 screens its scale response for stability, monotonicity, and predictability (Fig. 2; App. C.6). ARK then learns a scale per adapted module and examines attained ranks after fine-tuning (Sec. 5.3). The kernel is evaluated before task training, and its scale becomes a trainable module-specific variable rather than a fixed normalization constant. This connects rank characterization, kernel selection, module-level control, and empirical validation under the same trainable-parameter budget.

2. NEW: Sinter does not meet ARK's selection criteria in its published configuration.

We evaluated LoRAN's Sinter, f(x)=x[1+asin⁡(ωx)]f(x)=x[1+a\sin(\omega x)], with its reported a=5×10−5a=5\times10^{-5} and ω=104\omega=10^4, using seven independent factor draws and our existing numerical-rank protocol.

SettingSinter: RSD range / SpearmanARK-Tanh: RSD range / Spearman
D=32,r=4D=32,r=40–5.321 / 0.4500–8.000 / 0.974
D=1024,r=32D=1024,r=320–1.031 / −0.0430–31.978 / 0.918

At D=32D=32, Sinter's response is non-monotonic and reaches only 5.321 of the attainable D/r=8D/r=8 ceiling, despite a high fitted-curve R2R^2. This demonstrates why goodness of fit alone is not the selection criterion: the rank must also move consistently in a useful direction. At D=1024D=1024, its numerical rank stays near the factor rank rr across positive scales; ARK-Tanh instead exhibits substantial, predictable rank expansion. A nonlinearity selected for fitting quality need not provide the controllable rank response required by ARK.

3. ARK explains the difference and reveals alternative operating regions.

Sinter's nonlinear term has relative amplitude a=5×10−5a=5\times10^{-5}. Our singular-spectrum analysis shows that its contribution is sensitive to the numerical-rank threshold: at D=1024D=1024, the float32 relative threshold exceeds this amplitude, consistent with the near-flat response.

The diagnostic also pinpoints the role of the amplitude rather than treating Sinter as intrinsically incapable of rank expansion. A controlled parameter sweep provides further insight. At D=32,r=4D=32,r=4, increasing aa to 10−310^{-3} raises the maximum RSD from 5.321 to 7.89, close to the ceiling of 8. ARK thus offers more than a binary judgment of a kernel: its rank-response diagnostics identify parameter regimes in which a nonlinear construction can become rank-responsive.

LoRAN's strength is its nonlinear design for improved adaptation quality. We recognize that contribution, while our experiments reveal a property its original design did not set out to characterize: the scale-dependent numerical-rank behavior of the resulting update. ARK provides the measurements and selection principles to make that property a deliberate design target, not an incidental consequence of choosing an activation.

4. Implication for novelty.

Our original SNELL/KLoRA analysis established why rank expansion alone is insufficient. The new LoRAN experiment independently confirms the value of that distinction and shows how ARK's framework can explain previously proposed nonlinearities. ARK's contribution is a reusable methodology for characterizing, selecting, and learning numerical-rank responses under a fixed budget—not simply another nonlinear LoRA variant.

We will add LoRAN to the related-work comparison. Our shared cross-method results also identify SNELLA Mix-K as a promising external candidate, showing that the same framework can both diagnose limited responses and discover useful ones.

AuthorsOpenReview ↗

91i4: Response to W3/W7/Q3: Why Induced Rank Matters Beyond Infinite-Dimensional Feature Spaces

We thank the reviewer for raising this fundamental question. The distinction between the potentially infinite dimension of an implicit feature space and the numerical rank actually realized by a kernel is precisely a central motivation of ARK. Rather than overlooking this issue, our paper explicitly investigates it in Sec. 3 and Appendix C.2, motivating our departure from traditional Mercer-based kernel selection toward Rank Scaling Density (RSD).

1. Motivation and theoretical foundation.

Classical kernel theory characterizes the existence and dimensionality of an implicit feature space. However, potentially infinite-dimensional expressivity does not guarantee a high numerical rank in the finite matrix actually used for adaptation.

We formalize this distinction in Appendix C.2 (Eqs. 21–23). For Mercer kernels, the numerical rank of a finite self-kernel matrix is upper-bounded by the feature-space dimension, but need not approach that bound. In particular, an RBF self-kernel can remain algebraically full-rank even when most of its singular values fall below a specified numerical tolerance. We further distinguish this self-kernel interpretation (Def. 4) from the cross-kernel update K(A,B)K(A,B) used in PEFT, whose induced numerical rank is directly characterized by RSD (Def. 2).

Appendix C.2 also goes beyond Mercer kernels. For a symmetric self-kernel matrix S=K(X,X)S=K(X,X), Eq. (22) gives Sij=⟨ψi,Jψj⟩S_{ij}=\langle\psi_i,J\psi_j\rangle, where JJ records positive and negative eigenvalue directions. This provides a pseudo-Euclidean interpretation even for indefinite kernels; the PSD/Mercer case recovers the usual Hilbert-space representation. Eq. (23) then establishes E[rankτK(X,X)]≤dim⁡H\mathbb{E}[\mathrm{rank}_{\tau}K(X,X)]\leq\dim\mathcal H: feature-space dimension upper-bounds, but does not determine, the realized rank. Crucially, this bound concerns self-kernel RSD ρself\rho_{\mathrm{self}} (Def. 4), not the cross-kernel RSD ρ\rho (Def. 2) measured on the actual PEFT update.

This distinction is fundamental to our design: the dimension of an implicit space describes potential capacity, whereas the induced numerical rank characterizes the realized update at the dimensions relevant to adaptation.

2. Concrete evidence motivating our criterion.

Our paper examines this gap from two complementary directions:

  • RBF: Despite its infinite-dimensional feature space, its induced numerical rank exhibits a non-monotonic response to scale and eventually collapses under the numerical-rank criterion (Fig. 2b). Thus, infinite-dimensional capacity alone does not ensure controllable rank expansion.
  • Mercer-Tanh: Applying Tanh separately to the low-rank factors preserves an inner dimension of rr, leaving the resulting matrix rank bounded by rr (Appendix C.3.3).

Where Tanh is applied is decisive. Mercer-Tanh uses KMT(A,B)=tanh⁡(γA+δ)tanh⁡(γB+δ)⊤K_{\mathrm{MT}}(A,B)=\tanh(\gamma A+\delta)\tanh(\gamma B+\delta)^\top (omitting outer scale and bias), which still factors through rr columns and has a PSD self-kernel. ARK-Tanh instead uses KARK(A,B)=tanh⁡(γAB⊤)K_{\mathrm{ARK}}(A,B)=\tanh(\gamma AB^\top): applying Tanh after multiplication allows the update to exceed that rank ceiling. Thus the induced rank, not Mercer membership, is the relevant design criterion.

These examples explain why Mercer validity or implicit feature-space dimensionality alone cannot serve as our kernel-selection criterion. Instead, Sec. 4.1 introduces RSD to measure the rank actually induced, while Sec. 4.2 further examines the stability and predictability of its response to scale. This becomes an operational selection rule: Fig. 2 rejects Chi-Square for instability and RBF for non-monotonicity, while Tanh and Swish satisfy the stability, monotonicity, and predictability criteria. These are induced-rank measurements, independent of downstream accuracy.

3. Clarity improvement.

We appreciate that this motivation, although developed in Sec. 3 and Appendix C.2, could be more accessible without requiring readers to connect several sections. We will add a concise clarification in Sec. 4.1 distinguishing implicit feature-space dimension, self-kernel geometry, and the induced rank of the cross-kernel update, with explicit references to Appendix C.2, the RBF example, and the contrasting Mercer-Tanh/ARK-Tanh constructions.

We thank the reviewer for helping us make this central motivation more explicit.

AuthorsOpenReview ↗

91i4: Response to W5/Q6: Materialization, Computational Crossover, and Rank-Dependent Cost

We thank the reviewer for highlighting this important efficiency question. The trade-off between materializing AB⊤AB^\top and applying the low-rank factors separately is already investigated theoretically and empirically in Appendix D.7 (Tables 13–15 and Fig. 12). Materialization changes the computation order; whether it is advantageous depends on the workload, rather than admitting a blanket faster/slower conclusion.

1. The original submission establishes the computation-order crossover. Standard linear LoRA admits two equivalent paths, X(AB⊤)X(AB^\top) and (XA)B⊤(XA)B^\top. Under the forward operation-count convention in Appendix D.7, the factorized path costs 2Nr(din+dout)2Nr(d_{\mathrm{in}}+d_{\mathrm{out}}), while constructing and applying the materialized linear update costs dindout(2r+1)d_{\mathrm{in}}d_{\mathrm{out}}(2r+1). Their crossover is N∗=2r+12rdindoutdin+dout.N^*=\frac{2r+1}{2r}\frac{d_{\mathrm{in}}d_{\mathrm{out}}}{d_{\mathrm{in}}+d_{\mathrm{out}}}.

At r=32r=32 for Qwen2.5-7B, N∗N^* is approximately 1,820 tokens for the query projection and 455 tokens for the value projection. Thus, at sufficiently large token counts, materializing the linear update can be cheaper in the forward pass. This is verified in the original Table 14: at width 3,584 and N=16,384N=16{,}384, adapter-only forward times are 366.2 μs for factorized LoRA, 36.6 μs for the materialized linear control, and 49.2 μs for ARK-Tanh.

2. The original paper also measures the cost that forward FLOPs alone cannot capture. Appendix D.7 examines the backward path and activation storage, so we do not extrapolate the forward crossover into a universal end-to-end speedup. In the corresponding forward-plus-backward operator experiment (Table 15), the extra cost of applying Tanh over the same materialized linear path is 0.075 ms, approximately 0.6% of the frozen projection time. At the training-system level, Table 3 reports ARK-Tanh at 1.01× the training time and 1.08× the peak reserved CUDA memory of the LoRA implementation used in our controlled experimental framework, for the Qwen2.5-7B setting. The update can be merged for inference. These measurements serve different purposes: the materialized-linear control isolates the kernel operation, while the within-framework LoRA comparison describes total training overhead.

3. Additional rank-dependent measurements. We further measured the incremental Tanh cost relative to the materialized-linear path at a fixed operator configuration while sweeping rr:

Factor rank rrAdditional Tanh cost over materialized linear path
80.079 ms
160.061 ms
320.074 ms
640.062 ms

Across this eightfold rank range, the measured increment stays within 0.061–0.079 ms, or approximately 0.53%–0.69% of the frozen projection time, without a pronounced upward trend. This isolates the nonlinearity's incremental cost; the total difference from factorized LoRA also includes dense-update construction and backward computation.

4. Clarity improvement. We will make the theoretical crossover, measured forward/backward trade-offs, and distinction between algorithmic computation order and implementation-level optimization easier to locate in the main text, with explicit references to Appendix D.7. This also clarifies that the end-to-end ratios in Table 3 refer to the common experimental framework, not to a claim that an unoptimized ARK implementation outperforms every independently optimized LoRA implementation.

Reviewer_8WWrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the limitations of existing rank expansion and adaptive-rank methods for LoRA, this paper proposes Adaptive-Rank Kernels (ARK), which apply Tanh or Swish nonlinear transformations to the low-rank matrix product and introduce one learnable scale parameter per adapted module. This enables different modules’ update matrices to attain different numerical ranks and exceed the rank ceiling of r in conventional LoRA, thereby expanding the expressive capacity of weight updates. Experiments show that ARK is competitive under comparable parameter budgets, but its gains are mainly observed on BERT, with limited improvements over strong baselines on other datasets and models. Nevertheless, the paper presents extensive experiments and provides reasonably thorough justification for its conceptual definitions.

Critical strengths
  1. The idea is interesting. ARK addresses limitations of existing methods by introducing adaptivity at the module level.
  2. Although the paper introduces many concepts and definitions, it provides substantial experimental evidence to justify them, which strengthens the overall argument.
  3. The paper presents an extensive experimental evaluation, demonstrating ARK’s competitive performance, with particularly notable gains on BERT.
Critical weaknesses
  1. Table 1 is a key results table, yet it lacks comparisons with recent methods: it includes no methods from 2025 and only one from 2024. Moreover, ARK’s advantages do not consistently carry over to other settings. The improvement over LoRA on RoBERTa is only 0.18 points. On GSM8k, the gains over strong baselines under matched parameter budgets are only 0.15 and 0.03 points for the 3B and 7B models, respectively.
  2. The method introduces numerical-rank adaptivity alongside Tanh/Swish nonlinearities and learnable scale parameters. However, the paper does not provide sufficient evidence to attribute the performance gains specifically to numerical-rank adaptivity.
Questions for authors
  1. Why are different models used for different datasets, rather than evaluating each model on all the datasets?
  2. Why does Table 1 omit comparisons with more recent work, such as methods from 2025 or early 2026? Could the authors include more recent and representative LoRA variants?
  3. How can the authors establish that the performance gains are attributable to adaptive adjustment of the numerical rank rather than other components introduced by the method?
5 comments
AuthorsOpenReview ↗

8WWr: Response to W1/Q1: Model–Dataset Selection and Cross-Method Evaluation Coverage

We thank the reviewer for asking why different models are used for different datasets rather than evaluating every model on every dataset. Our evaluation examines complementary model–task settings, with multiple backbones within each task family, and we have now substantially expanded the overlap of evaluated methods between Tables 1 and 2.

1. Model–dataset pairing follows task-specific evaluation protocols. BERT-Base-Cased and RoBERTa-Base are evaluated on the same eight GLUE tasks using encoder-based prediction protocols (Table 1; Appendix D.4, Table 12). Qwen2.5-3B and Qwen2.5-7B are fine-tuned on MetaMathQA and evaluated on GSM8K using autoregressive generation and answer extraction (Table 2). This covers two architecture families and both discriminative and generative objectives. Evaluating each architecture on every task would involve changing task formulations and evaluation protocols, rather than simply adding directly comparable cells.

2. We have substantially expanded the overlap of methods between Tables 1 and 2. Computational resources at submission time led us to prioritize representative methods in each task family, so the original comparison tables did not contain identical method sets. We have now completed eight-task GLUE evaluations for six additional methods already included in the generative-model comparison: DoRA, DeLoRA, RandLoRA, GraLoRA, OSF-736, and WaveFT. The results are presented directly below, alongside the submitted Table 1 LoRA and ARK-Tanh results.

MethodMRPCSTS-BRTECoLAMNLIQNLIQQPSST-2Avg.
LoRA (Tab. 1)68.3887.0452.7156.7675.9289.6988.1390.8376.18
DoRA84.5688.6166.4353.7282.5890.1988.8189.9180.60
DeLoRA83.3388.7567.1554.8081.5890.7688.2890.9480.70
RandLoRA83.8288.0667.8755.2180.8790.3087.2791.4080.60
GraLoRA81.3787.8667.5153.4082.2690.2188.7690.4880.23
OSF-73676.7286.0057.7645.8372.7785.5883.4787.2774.43
WaveFT78.4387.2063.1850.0874.7786.9385.1189.6876.92
ARK-Tanh (Tab. 1)83.8288.5866.7958.8282.8390.9289.0590.9481.47

GLUE validation metrics follow Table 1 (CoLA: Matthews; STS-B: Spearman; others: accuracy). Avg. is the equally weighted mean across eight tasks. New method rows report the additional evaluations; LoRA and ARK-Tanh are reproduced from the submitted Table 1.

These results directly address the cross-method coverage gap: the additional methods are now evaluated on all eight encoder tasks, and ARK-Tanh retains the highest eight-task average among the rows shown.

AuthorsOpenReview ↗

8WWr: Response to W2/Q3: Evidence for Adaptive Numerical Rank

We thank the reviewer for this important question. ARK's contribution is not simply adding Tanh/Swish to LoRA: it enables the numerical rank of a weight update to vary at a fixed factor rank and introduces a learnable, module-wise scale to control that response. The submitted paper already evaluates these components through the Rank Scaling Law (Sec. 4.2; Fig. 2), a fixed-versus-learned scale ablation (Table 4), and the rank trajectories of trained modules (Fig. 3; Appendix D.6).

1. The ablation isolates the benefit of learning the scale. Table 4 compares the same nonlinear kernel with fixed versus learnable γ\gamma, retaining the factor dimensions and adaptation budget:

KernelFixed γ\gammaLearned γ\gammaGain
Tanh78.5679.73+1.17
Swish77.8479.27+1.43

Thus, even after introducing the nonlinearity, learning its rank-controlling scale yields a further and consistent improvement across both kernel families.

2. Training exhibits active, module-specific rank adaptation. Fig. 3 shows that with factor rank r=32r=32, ARK modules attain different numerical ranks across tasks and layers, including ranks substantially above 32. We further analyzed the original Tanh training records under symmetric residual initialization, using the submitted per-method learning-rate selection protocol. The learned-scale variant reaches a three-task mean of 79.883, compared with 78.382 for fixed scale. Crucially, in 9/9 matched task–learning-rate–seed pairs, learned γ\gamma produces larger rank changes during training; in 8/9 pairs, its final ranks are more differentiated across modules. The average rank-change statistic increases from 15.67 to 56.67, and the final inter-module rank IQR from 19.67 to 55.75.

3. Adaptation selects ranks rather than merely increasing them. The learned-scale variant ends with a lower average median rank (292.33 vs. 322.83), despite stronger rank adjustment and higher mean accuracy. This is the behavior motivating ARK: different modules can move toward different numerical ranks under the same parameter budget, rather than requiring every module to use a larger rank.

These results provide complementary evidence: the kernel makes rank expansion accessible, the scale ablation establishes its practical benefit, and the training trajectories show that the learned controller accompanies substantive, task-dependent rank adjustments.

AuthorsOpenReview ↗

8WWr: Response to W1/Q2: Expanded Comparisons with Recent PEFT Methods

We thank the reviewer for emphasizing the importance of recent and representative baselines. Our earlier response to W1/Q1 reported eight-task GLUE results for six additional methods—DoRA, DeLoRA, RandLoRA, GraLoRA, OSF-736, and WaveFT. Building on that comparison, we have now completed evaluations of HiRA '25, GoRA '25, RoSA '24, and MoKA '25, including combinations of GoRA and RoSA with ARK.

Eight-task GLUE averages.

MethodAverageMethodAverage
DoRA ’2480.60DeLoRA ’2580.70
RandLoRA ’2480.60GraLoRA ’2580.23
OSF-736 ’2574.43WaveFT ’2576.92
RoSA ’2479.92RoSA ’24 + ARK80.67
GoRA ’2578.36GoRA ’25 + ARK80.97
MoKA ’25 (r=16r=16)80.32LoRA ’2176.18
ARK-Tanh81.47

In the separate eight-task HiRA comparison, ARK-Tanh's mean exceeds HiRA by 2.08 points. The recent baseline evaluations use the same eight GLUE tasks; the GoRA, RoSA, and MoKA campaign employs BERT-Base-Cased, the submitted four-rate learning-rate grid, and the median of three seeds at the learning rate selected using seed 0. Results from separate completed campaigns are reported as descriptive comparisons, rather than as a single identical-seed run.

ARK-Tanh achieves 81.47 across eight GLUE tasks, exceeding all the newly reported methods in this comparison. Moreover, GoRA+ARK improves GoRA's eight-task mean by 2.61 points, and RoSA+ARK improves RoSA's by 0.75 points. These combinations show that ARK can enhance existing PEFT constructions in addition to serving as a standalone alternative. The full evaluation settings and aggregate composition results are given in our Expanded Recent PEFT Comparisons and ARK Composability (8WWr W1/Q2; 91i4 W4/Q4; R85o W3/Q3).

Comments on the paper

7 comments
AuthorsOpenReview ↗

Nonlinear/Kernelized LoRA: Original Motivation and New Cross-Method Validation (91i4 W6/Q5; R85o W3/Q3)

We thank the reviewers for pointing out LoRAN and SNELLA. We used their suggestions to test whether ARK can explain independently designed kernels and identify new rank-controllable candidates. The results build on our original analysis.

1. The core distinction was already established, theoretically and empirically.

The Rank-Dilemma (Sec. 3) distinguishes high-rank updates from controllable numerical rank at fixed parameter budget. ARK formalizes the Iso-Budget Rank Scaling Law (Def. 1): a continuous controller adjusts rank without enlarging the factors.

This is not post-hoc. Sec. 3 discusses SNELL/KLoRA; Table 4 evaluates Piecewise Linear (a.k.a. KLoRA) downstream; Apps. C.4–C.5 analyze its rank scaling; App. C.8 establishes non-PSD behavior; and Thm. 7 (App. C.9) explains its rank invariance under positive scaling. Its high rank but flat response was our original motivating example, not a rebuttal-stage experiment.

2. ARK contributes a selection-and-control methodology, not another activation.

Instead of selecting by Mercer validity, feature-space size or downstream accuracy alone, we define Rank Scaling Density, ρ = E[numerical rank]/r (Def. 2), on the synthesized update. Its scale response is screened for stability, monotonicity and predictability before task training (Sec. 4.2; Fig. 2; App. C.6). ARK then learns a scale per adapted module and measures attained ranks after fine-tuning (Sec. 5.3; Fig. 3): a characterization → selection → control → validation framework. For positive scale, varying the pre-activation multiplier is equivalent to changing the variance of the random factors, while keeping their dimensions and parameter budget fixed (Sec. 4.2). We measure the numerical spectrum of the actual finite update, rather than infer it from feature-space dimension.

3. NEW: Applying the original protocol to LoRAN Sinter and SNELLA Mix-K.

Using our existing pipeline (seven factor draws, original relative-rank metric), we tested two new candidates; Piecewise Linear remains the original-paper reference. Each cell gives ρ range / Spearman:

KernelD=32, r=4: ρ / SpearmanD=1024, r=32: ρ / Spearman
ARK-Tanh0–8 / 0.9740–31.978 / 0.918
ARK-Swish0–8 / 0.9580–31.799 / 0.635
Piecewise Linear (KLoRA; previously analyzed)0–8 / 0.2150–28.821 / 0.073
LoRAN Sinter (NEW)0–5.321 / 0.4500–1.031 / −0.043
SNELLA Mix-K (NEW)0.250–8 / 0.7560.031–28.848 / 0.974

Sinter uses reported a=5×10⁻⁵, ω=10⁴. Mix-K uses a controlled fixed-coefficient configuration (α=1, β=0 and specified Piecewise Linear coefficients). D=1024 uses 256 scale points.

4. Three distinct outcomes test the framework's explanatory and discovery power.

Piecewise Linear: high rank without control. At D=1024 it reaches about 922/1024 numerical rank, yet is flat across positive scales. Our existing positive-homogeneity analysis explains why: scaling changes magnitude, not normalized singular-value ratios.

LoRAN Sinter: effective fitting need not entail rank controllability. Using the published coefficients, its response is non-monotonic at D=32 and nearly rank-neutral at D=1024 under the tested float32 rank tolerance. This does not dispute LoRAN's downstream gains. With a controlled amplitude change to a=10⁻³, the D=32 maximum ρ rises to 7.89: the conclusion concerns the tested configuration.

SNELLA Mix-K: ARK identifies a promising external candidate. The tested configuration meets the stability, monotonicity, and predictability criteria at both sizes. At D=1024, Spearman is 0.974 (4PL R²=0.996), versus 0.918 for ARK-Tanh. Removing the normalized exponential component recovers Piecewise Linear elementwise (maximum difference 0.0), isolating the component that changes the scale response from a high-rank plateau into a controllable transition. This isolates controllability rather than maximum rank, and identifies an independently proposed kernel.

Implication. Our original SNELL/KLoRA analysis already established why high rank is insufficient; the new LoRAN and SNELLA tests independently demonstrate the diagnostic and discovery value of ARK's framework. ARK's contribution is a reusable, numerical-rank-centered method for characterizing, selecting, and learning low-budget kernel updates, not merely applying nonlinearity. Reviewer-specific replies discuss each result; downstream matched-budget comparisons and implementation-fidelity questions are separate issues.

AuthorsOpenReview ↗

General Response — Author Responses and Ongoing Experimental Updates

We sincerely thank all three reviewers for their thoughtful and constructive feedback. We particularly appreciate their recognition of the scientific importance of adaptive rank learning under a fixed parameter budget, the novel perspective of the Iso-Budget Rank Scaling Law, and the simplicity and practical relevance of ARK. We are also grateful for their careful consideration of our extensive experimental evaluation and the theoretical and empirical analyses motivating our approach.

We apologize for the delayed response. Additional experiments have required coordination of our laboratory's shared GPU resources, and several evaluations are still underway. We appreciate the reviewers' patience and understanding.

We are grateful for the opportunity to further clarify the motivation and contributions of Adaptive-Rank Kernels (ARK). Our initial responses elaborate on several important aspects of the original work: the distinction between implicit feature-space dimensionality and induced numerical rank, which motivates our kernel-selection criterion; the theoretical and empirical analysis of computational trade-offs associated with materializing low-rank updates; and the positioning of ARK relative to existing nonlinear LoRA methods. We also clarify the rationale behind our model–dataset evaluation design and present additional cross-method comparisons.

We are continuing to extend our experimental comparisons and analyses in light of the reviewers' suggestions. Further results, methodological details, and responses to the remaining questions will be shared during the discussion period.

We sincerely appreciate the reviewers' valuable feedback and look forward to further discussion.

Area_Chair_W6VNOpenReview ↗

Please Join the Discussion

Dear Reviewers,

The authors have submitted their responses. Please read them and provide your feedback. Thank you!

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.