acceptodds
Withdrawn from ICLR 2027

Lie-Tuning: Exact Low-Rank Matrix Flows for Rank-Preserving Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Parameter-efficient fine-tuning must balance adaptation capacity against preservation of a pretrained linear map. LoRA constrains the rank of an additive update, but Wθ = W₀ + BA does not guarantee rank(Wθ) = rank(W₀). Orthogonal multiplicative methods provide that guarantee, but they also preserve the entire singular spectrum and therefore exclude anisotropic scaling. We propose Lie-Tuning, a framework that decouples rank preservation from spectral rigidity. It adapts a frozen map through invertible general-linear actions, Wθ = exp(ξ_L)W₀ or Wθ = W₀ exp(ξ_R). Invertibility preserves the rank of W₀ exactly, while a general, non-skew-symmetric generator can change singular values through scaling and shearing. The two action sides provide complementary inductive biases: left multiplication post-composes the pretrained map in its output space, whereas right multiplication pre-composes it in its input space. We instantiate this framework with MExRA, whose generator ξ = BA is low rank, and MExBA, whose generator is block diagonal. For MExRA, we derive the exact identity exp(BA) = I + Bφ₁(AB)A. It retains all orders of the exponential while reducing an ambient d × d exponential to an r × r kernel, requiring O(dr) trainable parameters and O(dr² + r³) kernel construction. The same φ₁ representation has a second computational consequence: it forms exp(ξ) − I without explicitly subtracting the identity, avoiding near-identity cancellation and directly representing the first variation about exact identity initialization. This matrix analogue of expm1 makes the residual numerically resolvable during training. The learned action remains mergeable for inference. On LLaMA-2-7B, MExRA reaches 25.00% HumanEval Pass@1 in the regularized setting versus LoRA's 21.95%. On GSM8K, its three-seed mean accuracy is 35.28%, compared with 33.46% for LoRA and 29.11% for OFT. Right-acting MExBA achieves 46.11% German Belebele accuracy versus 43.78% for LoRA. MExRA also achieves the highest GSM8K accuracy among the compared methods throughout sequential adaptation, while numerical tests verify accurate exponential residual evaluation near identity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
parameter-efficient fine-tuning, low-rank adaptation, Lie groups, matrix exponential, rank-preserving adaptation, numerical stability, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7pdnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Lie-Tuning, which adapts frozen linear layers through invertible matrix-exponential transformations. It preserves matrix rank while allowing singular values to change, unlike purely orthogonal adaptation. Its MExRA variant uses a low-rank generator and an exact reduction to a small matrix function; MExBA uses block-diagonal generators. A subtraction-free residual construction addresses numerical cancellation near identity initialization. Experiments on LLaMA-2-7B cover code generation, mathematical reasoning, cross-lingual transfer, sequential adaptation, and numerical accuracy. The strongest repeated-run evidence is the three-seed GSM8K comparison.

Critical strengths
  1. A coherent method with a concrete computational realization. Sections 3.1–3.5 distinguish rank preservation from spectral rigidity and provide an exact small-kernel implementation rather than relying on an ambient matrix exponential. The residual construction also targets a specific numerical issue near identity initialization. This combination makes the framework more useful than an abstract statement about invertible transformations alone.
  2. Positive evidence and a well-documented subset of the experiments. Table 3 reports three-seed GSM8K results under closely matched trainable-parameter budgets: MExRA reaches 35.28 ± 0.98%, versus 33.46 ± 1.23% for LoRA and 29.11 ± 1.18% for OFT. Section 4.6 tests numerical accuracy across matrix families, sizes, scales, and storage precisions. The supplement provides detailed GSM8K and numerical configurations, fixed revisions, commands, and tests. Its planning and numerical dry-run commands were checked successfully; these checks do not reproduce the reported scores.
Critical weaknesses
  1. The practical accuracy–cost advantage is not yet convincing across the headline tasks. Section 4.1 explicitly identifies the code, cross-lingual, and sequential-adaptation results as single runs. In Table 2, the regularized HumanEval gain over LoRA is 3.05 percentage points, corresponding to five of 164 problems. In Table 5, the final GSM8K lead is only 0.53 points, while MExRA already starts from a higher first-stage score. These absolute scores alone do not establish better retention. Meanwhile, Table 8 reports 5.37 samples/s for MExRA versus 16.62 for LoRA, approximately 32% of LoRA's throughput. The authors acknowledge these limits, and the three-seed GSM8K gain is encouraging, but the current evidence does not show whether the other small gains are stable or justify the additional training cost. This materially affects the practical case for the method.
  2. Several key task results lack the detailed protocol supplied for GSM8K. Section 4.4 names C-Eval and German Belebele as evaluation benchmarks and gives learning rate and weight decay, but does not clearly identify the corresponding fine-tuning corpora, sample counts, and training/evaluation splits. Section 4.5 lists the math, code, and general-instruction stages without fully specifying the data and configuration at each stage. The released package explicitly covers GSM8K and numerical accuracy; it does not include dedicated runners/configurations for Tables 2, 4, and 5. Consequently, the cross-lingual and sequential results supporting the directional and composition claims are difficult to independently reconstruct. This is a specific coverage gap, not a claim that the entire submission lacks useful code.
  3. Rank preservation is not yet shown to protect useful pretrained knowledge. The introduction motivates the method partly by LoRA's lack of a rank guarantee, but the experiments do not report whether LoRA actually changes the ranks of the adapted projections, or whether such changes harm retained abilities. If a projection is already full rank, an additive update can also leave that rank unchanged. More fundamentally, transformations with the same rank can have very different outputs and conditioning. The stated invariant is valid, but the paper needs evidence linking it to the practical preservation benefit, rather than treating rank alone as a sufficient proxy for preserving the pretrained map.
Questions for authors
  1. Please identify the fine-tuning corpora, sample counts, train/evaluation splits, adapted projection modules, and evaluation prompts for Table 4 and each stage of Table 5. Can you point to existing configurations or scripts that reproduce Tables 2, 4, and 5?
  2. Beyond the reported single runs, are existing seed results or per-example predictions available for HumanEval and the cross-lingual tasks? For Table 5, how should the retention claim be interpreted after accounting for the different first-stage scores? Clarifying existing evidence would affect my assessment; I am not requesting a full new experiment suite during rebuttal.
  3. What GPU setup, sequence length, and batch settings produced Table 8? Are any existing comparisons available under a matched training-time budget, or other evidence showing when the accuracy gain outweighs the measured throughput reduction?
  4. Do the existing runs show that LoRA loses rank, or that rank/spectrum changes correlate with retained ability? What evidence separates the benefit of rank preservation from other optimization differences?
1 comment

Thank you for requesting these details. We clarify the fine-tuning corpora, data selection, adapted modules, and evaluation protocols below.

ExperimentFine-tuning corpusTraining subsetEvaluation split
Table 2sahil2801/CodeAlpaca-20kFull training split, approximately 20k examplesHumanEval test, 164 problems
Table 4: ChineseBelleGroup/train_1M_CN20,000 training examples, selected after shuffling with seed 42C-Eval validation splits; macro-average across subjects
Table 4: GermanGerman Alpaca instruction data*Up to 20,000 training examples, selected after shuffling with seed 42Belebele deu_Latn test, 900 questions
Table 5: Mathmeta-math/MetaMathQA10,000 training examples, selected after shuffling with seed 42GSM8K test after this stage
Table 5: Codesahil2801/CodeAlpaca-20k10,000 training examples, selected after shuffling with seed 42GSM8K test after this stage
Table 5: Generalyahma/alpaca-cleaned10,000 training examples, selected after shuffling with seed 42GSM8K test after this stage

For German, philschmid/alpaca-de is the provisionally identified source, matching the loader’s first choice; fallback German Alpaca sources were also supported.

No additional validation partition is carved out of these fine-tuning subsets. C-Eval’s validation split is used exclusively for evaluation. The seed of 42 above refers to data selection.

For Tables 2, 4, and 5, adapters target q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj throughout LLaMA-2-7B. The pretrained weights remain frozen during each adaptation stage. We use learning rate 3×10−43\times10^{-4}, global batch size 16, rank 16, and block size 32 for block-structured methods. Table 2 compares weight decay 0 and 10−310^{-3}; Tables 4 and 5 use 10−310^{-3}. MExRA and its orthogonal low-rank variants, on either action side, apply decay to BB while leaving AA undecayed. LoRA, OFT, and the block variants use standard AdamW decay.

Tables 2 and 4 use three training epochs. Table 5 uses one epoch per stage, an exception to the general three-epoch setting in Section 4.1. The stage order is math → code → general instruction tuning. After each stage, the adapter is merged into the model, and the next stage starts from the merged checkpoint with a newly initialized adapter and optimizer. Earlier-stage training examples are not replayed.

For C-Eval, we use a Chinese-language zero-shot multiple-choice prompt matching the language of the benchmark:

以下是关于{subject_name}的单项选择题,请直接给出正确答案的选项。

题目:{question}
A. {option_A}
B. {option_B}
C. {option_C}
D. {option_D}
答案:

The same template is used for all compared methods, without in-context demonstrations. The evaluator selects among the A/B/C/D answer-token logits and reports the macro-average of subject accuracies.

For German Belebele, the zero-shot prompt is:

Lies den folgenden Text und beantworte die Frage.

Text: {passage}

Frage: {question}
A. {option_A}
B. {option_B}
C. {option_C}
D. {option_D}
Antwort:

All methods use this same template. The evaluator selects among the four answer-token logits and reports question-level accuracy.

For Table 5, all three stage checkpoints use the same zero-shot GSM8K prompt:

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{question}

### Response:

Evaluation covers all 1,319 GSM8K test questions using greedy decoding with at most 512 new tokens. The scorer extracts the numerical answer following “The answer is,” or otherwise the last number, and compares it with the reference using an absolute tolerance of 10−510^{-5}.

For Table 2, adapted models use the Alpaca-style instruction “Please complete the following Python code snippet.” The HumanEval code prompt appears in the input field and as the response prefix. The base model receives the original code prompt directly. Each problem receives one greedy completion, capped at 512 new tokens, followed by completion postprocessing and the official HumanEval functional-correctness evaluator.

The submitted supplement includes the adapter implementations in model.py, together with dedicated pipelines for the standalone GSM8K and numerical experiments. Task-specific runners for Tables 2, 4, and 5 were not included in that archive; the clarification above supplies their data selection, adaptation targets, and evaluation protocols.

Reviewer_6mZMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Lie-Tuning, a PEFT framework that adapts a frozen weight through an invertible general-linear action, Wθ = exp(ξ)W₀ or W₀exp(ξ). This preserves rank exactly while, unlike orthogonal methods such as OFT, allowing the singular spectrum to change.

Critical strengths

Paper is clean and well written. this work separates rank preservation from spectral rigidity, treats action side, group, and generator structure as independent design axes, and includes orthogonal special cases. The exact ϕ₁ reduction, its inversion-free block-exponential extraction, and the extension to bias terms through the affine group are correct

Critical weaknesses
  1. Emperically it should be scaled, and experiment uses LLaMA-2-7B. The paper should test newer and larger models (LLaMA-3-8B/70B, Qwen2.5-7B/14B, Mistral-7B). Standard PEFT suites are also missing: commonsense reasoning (BoolQ, PIQA, HellaSwag, etc.), MT-Bench/AlpacaEval, and GLUE.

  2. OFT and BOFT originate in vision and diffusion, ViT/VTAB and diffusion fine-tuning results would show whether the framework generalizes.

  3. Should compare with other baseline too, The closest baselines are missing: LoRMA (multiplicative low-rank, nearly identical in form), LieRA, DoRA, PiSSA.

  4. All methods share one learning rate (3e-4) with no per-method tuning, Can report with other learning rate as well, this will give a fair case against OFT.

  5. The paper never measures rank, null space, or the singular spectrum of adapted weights to show that preservation or spectral flexibility actually matters or differs from LoRA. Also memory overhead should be reported.

Questions for authors

See Weakness.

1 comment

Dear Reviewer 6mZM,

Thank you for recognizing the mathematical framework, exact small-kernel construction, and affine extension. We clarify the experimental scope, related methods, optimization settings, and structural guarantees below.

1. Model, task, and domain coverage

We evaluate code generation, mathematical reasoning, cross-lingual transfer, and sequential adaptation on LLaMA-2-7B. A common backbone allows comparisons across action sides and generator structures. Table 3 provides three-seed evidence.

The rank-preservation theorem and exact exponential reduction apply to rectangular linear maps independently of the surrounding architecture. Their applicability extends beyond LLaMA, while empirical benefits on newer/larger models, vision, and diffusion require further evaluation. Our current empirical claims concern the reported backbone and tasks.

2. LoRMA, LieRA, and additional baselines

Both LoRMA and LieRA are discussed in Section 2, with substantive differences in parameterization and invariants.

LoRMA's additive-inflation variant, LoRMA+, uses Wθ=(I+cBA)W0W_\theta=(I+cBA)W_0, whereas MExRA uses Wθ=exp⁡(cBA)W0W_\theta=\exp(cBA)W_0. The former can become singular when cBAcBA has eigenvalue −1-1; the exponential is invertible for every finite generator in exact arithmetic. MExRA evaluates this nonlinear transformation exactly through a reduced ϕ1\phi_1 kernel.

LieRA uses element-wise multiplication, with the efficient first-order update W+W⊙ΔWW+W\odot\Delta W. Lie-Tuning uses ordinary left/right matrix multiplication by a matrix exponential. Element-wise multiplication does not generally preserve matrix rank; invertible matrix multiplication does.

These distinctions do not replace empirical comparisons. The submitted experiments compare against LoRA and OFT; they do not establish superiority over LoRMA, LieRA, DoRA, or PiSSA.

3. Learning rates and OFT

The experiments do not all use learning rate 3×10−43\times10^{-4}. Section 4.3 uses 2×10−42\times10^{-4}, zero weight decay, and no additional regularization for all three methods in the three-seed GSM8K experiment.

The backbone, data, query/output projection targets, global batch size, sequence length, and epoch budget are shared. LoRA and MExRA each use 8,388,608 trainable parameters; OFT uses block size 64 with 8,257,536 parameters.

MethodGSM8K accuracy, mean ± SD
LoRA33.46±1.23%33.46\pm1.23\%
OFT29.11±1.18%29.11\pm1.18\%
MExRA35.28±0.98%35.28\pm0.98\%

MExRA exceeds OFT's mean accuracy at every reported epoch. This provides repeated-run evidence under a common protocol without weight-decay differences. A shared learning rate controls one variable but does not establish each method's individually optimized performance; we interpret the results within the stated protocol.

4. Rank preservation and retained knowledge

LoRA constrains the update rank without enforcing the pretrained map's rank. Orthogonal multiplication preserves rank and the entire singular spectrum. General-linear exponential actions preserve rank while permitting spectral changes. An invertible left action also preserves the input nullspace; an invertible right action preserves the output image.

These are structural guarantees, not evidence that LoRA necessarily loses rank. A LoRA update can leave a full-rank projection full rank, and unchanged rank alone does not guarantee unchanged outputs, conditioning, or retained knowledge. The experiments evaluate complete adapters; they do not isolate rank preservation as the causal explanation for retention. The contribution is a computationally realizable family that preserves rank while allowing spectral adaptation.

5. Memory and computational cost

MExRA uses O(dr)O(dr) trainable parameters and avoids an ambient d×dd\times d exponential. Peak training memory also depends on activations and differentiation through the matrix-function computation; parameter counts alone do not determine it.

Table 8 reports throughput on one A100: 5.37 samples/s for MExRA versus 16.62 for LoRA. Appendix H.4 discusses this training-cost tradeoff. Peak GPU-memory measurements are not reported. After merging, inference retains the original linear-layer architecture without a separate adapter branch.

Reviewer_yj64OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Lie-Tuning adapts frozen weight matrices through left or right multiplication by matrix exponentials. It considers low-rank and block-diagonal generators, with general-linear and orthogonal variants. The low-rank exponential is evaluated through a small phi_1 kernel, and an augmented matrix exponential provides a subtraction-free residual near identity. Experiments cover code generation, mathematical reasoning, cross-lingual transfer, sequential adaptation, and numerical accuracy.

Critical strengths
  1. The distinction between preserving matrix rank and preserving the entire singular spectrum is useful, and the numerical study addresses a concrete implementation problem. At kernel norm 10^−8, the stable FP32 residual has relative error about 2.53 × 10^−8, compared with 0.176 for explicit exponential-minus-identity evaluation (Table 6).
Critical weaknesses
  1. The central MExRA formulation and its exact computational reduction overlap with earlier work that is not cited. ELRA already defines W_new = exp(ηAB)W_0 and derives exp(ηAB) = I + ηA phi_1(ηBA)B in Proposition 1. Renaming the factors and absorbing η gives the same construction as Section 3.2 here. The block adapters and numerical analysis may provide additional contributions, but the present novelty claim needs to be assessed after accounting for this overlap. Adding a citation alone would not establish the significance of the remaining advance. Exponential Low-Rank Adapters, Section 3.2

  2. The downstream value of the numerical refinement is not established by the arithmetic tests alone. Table 6 compares stable, naive, and first-order residuals, but these alternatives are not compared in end-to-end training with initialization, scaling, and regularization held fixed. Several of those choices differ across the reported adapters. This matters because the improvements must justify a substantial training overhead: MExRA runs at 5.37 samples/s versus 16.62 for LoRA (Table 8).

Questions for authors
  1. After explicitly accounting for ELRA’s formulation and small-kernel identity, which contributions remain new, and what matched comparison demonstrates their benefit over ELRA?

  2. With initialization, scaling, optimizer groups, and training data fixed, do stable exact residuals improve final accuracy or training stability over naive exact evaluation and a first-order residual? What is the corresponding wall-clock cost?

1 comment

Dear Reviewer yj64,

Thank you for your comments. We would like to clarify the status of the ELRA reference and distinguish the evidence for numerical accuracy from the evidence for downstream performance.

The PDF URL supplied in the review currently returns a 404 error. The OpenReview record we located for ELRA is marked “ICLR 2026 Conference Desk Rejected Submission.” We have not verified a peer-reviewed publication corresponding to the cited version. Could you provide an accessible, identifiable version and clarify its publication status?

This distinction matters under the ICLR 2027 reviewing guidelines. The guidelines allow that authors may be unaware of work not published in peer-reviewed proceedings or journals, and explicitly state that “the lack of such comparisons cannot be a basis for rejection.” We therefore ask that the missing-citation concern and the request for a matched ELRA baseline be assessed under this policy. This policy question is separate from a technical discussion of any substantiated overlap.

Our contribution should also be assessed beyond the low-rank exponential identity alone. The paper studies rank-preserving adaptation with spectral flexibility, examines low-rank and block-structured generators, and develops a finite-precision analysis and implementation of near-identity residual evaluation. As your review acknowledges, the block adapters and numerical analysis are additional components. Their technical value should be evaluated explicitly, while any comparison with ELRA should distinguish the shared formulation from specific implementation choices and analyses.

Regarding numerical accuracy, Table 6 provides direct evidence for the residual-evaluation procedure: at kernel norm 10−810^{-8}, the stable FP32 residual has relative error approximately 2.53×10−82.53\times10^{-8}, compared with 0.1760.176 for explicit exponential-minus-identity evaluation. Stable and naive exact evaluation represent the same mathematical flow; their difference here is numerical fidelity. The first-order residual instead changes the mathematical model by discarding higher-order terms. Importantly, the “Naive” baseline in this experiment is not presented as ELRA’s implementation.

Regarding end-to-end benefit, the submitted downstream experiments evaluate complete adapters rather than isolating these three residual implementations under identical training settings. The three-seed GSM8K result—35.28±0.98%35.28\pm0.98\% for MExRA versus 33.46±1.23%33.46\pm1.23\% for LoRA—supports the performance of the complete adapter under the reported protocol. It does not, by itself, attribute that improvement specifically to cancellation avoidance. We agree that numerical fidelity and its effect on final task accuracy are distinct empirical questions.

Finally, the reported throughput of 5.37 versus 16.62 samples/s represents a substantial training-cost tradeoff: approximately 3.1 times the time for the same number of samples in the measured setting. However, this compares complete MExRA and LoRA implementations; it does not isolate the incremental cost of stable residual evaluation relative to another exact exponential implementation. Table 8 therefore cannot determine that incremental cost. We do not claim a universal training-speed advantage over LoRA.

We would appreciate clarification of the ELRA source and reassessment of the missing-citation/comparison concern under the conference policy, alongside a separate evaluation of the numerical and empirical contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.