- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Lie-Tuning, which adapts frozen linear layers through invertible matrix-exponential transformations. It preserves matrix rank while allowing singular values to change, unlike purely orthogonal adaptation. Its MExRA variant uses a low-rank generator and an exact reduction to a small matrix function; MExBA uses block-diagonal generators. A subtraction-free residual construction addresses numerical cancellation near identity initialization. Experiments on LLaMA-2-7B cover code generation, mathematical reasoning, cross-lingual transfer, sequential adaptation, and numerical accuracy. The strongest repeated-run evidence is the three-seed GSM8K comparison.
Critical strengths
- A coherent method with a concrete computational realization. Sections 3.1–3.5 distinguish rank preservation from spectral rigidity and provide an exact small-kernel implementation rather than relying on an ambient matrix exponential. The residual construction also targets a specific numerical issue near identity initialization. This combination makes the framework more useful than an abstract statement about invertible transformations alone.
- Positive evidence and a well-documented subset of the experiments. Table 3 reports three-seed GSM8K results under closely matched trainable-parameter budgets: MExRA reaches 35.28 ± 0.98%, versus 33.46 ± 1.23% for LoRA and 29.11 ± 1.18% for OFT. Section 4.6 tests numerical accuracy across matrix families, sizes, scales, and storage precisions. The supplement provides detailed GSM8K and numerical configurations, fixed revisions, commands, and tests. Its planning and numerical dry-run commands were checked successfully; these checks do not reproduce the reported scores.
Critical weaknesses
- The practical accuracy–cost advantage is not yet convincing across the headline tasks. Section 4.1 explicitly identifies the code, cross-lingual, and sequential-adaptation results as single runs. In Table 2, the regularized HumanEval gain over LoRA is 3.05 percentage points, corresponding to five of 164 problems. In Table 5, the final GSM8K lead is only 0.53 points, while MExRA already starts from a higher first-stage score. These absolute scores alone do not establish better retention. Meanwhile, Table 8 reports 5.37 samples/s for MExRA versus 16.62 for LoRA, approximately 32% of LoRA's throughput. The authors acknowledge these limits, and the three-seed GSM8K gain is encouraging, but the current evidence does not show whether the other small gains are stable or justify the additional training cost. This materially affects the practical case for the method.
- Several key task results lack the detailed protocol supplied for GSM8K. Section 4.4 names C-Eval and German Belebele as evaluation benchmarks and gives learning rate and weight decay, but does not clearly identify the corresponding fine-tuning corpora, sample counts, and training/evaluation splits. Section 4.5 lists the math, code, and general-instruction stages without fully specifying the data and configuration at each stage. The released package explicitly covers GSM8K and numerical accuracy; it does not include dedicated runners/configurations for Tables 2, 4, and 5. Consequently, the cross-lingual and sequential results supporting the directional and composition claims are difficult to independently reconstruct. This is a specific coverage gap, not a claim that the entire submission lacks useful code.
- Rank preservation is not yet shown to protect useful pretrained knowledge. The introduction motivates the method partly by LoRA's lack of a rank guarantee, but the experiments do not report whether LoRA actually changes the ranks of the adapted projections, or whether such changes harm retained abilities. If a projection is already full rank, an additive update can also leave that rank unchanged. More fundamentally, transformations with the same rank can have very different outputs and conditioning. The stated invariant is valid, but the paper needs evidence linking it to the practical preservation benefit, rather than treating rank alone as a sufficient proxy for preserving the pretrained map.
Questions for authors
- Please identify the fine-tuning corpora, sample counts, train/evaluation splits, adapted projection modules, and evaluation prompts for Table 4 and each stage of Table 5. Can you point to existing configurations or scripts that reproduce Tables 2, 4, and 5?
- Beyond the reported single runs, are existing seed results or per-example predictions available for HumanEval and the cross-lingual tasks? For Table 5, how should the retention claim be interpreted after accounting for the different first-stage scores? Clarifying existing evidence would affect my assessment; I am not requesting a full new experiment suite during rebuttal.
- What GPU setup, sequence length, and batch settings produced Table 8? Are any existing comparisons available under a matched training-time budget, or other evidence showing when the accuracy gain outweighs the measured throughput reduction?
- Do the existing runs show that LoRA loses rank, or that rank/spectrum changes correlate with retained ability? What evidence separates the benefit of rank preservation from other optimization differences?
Thank you for requesting these details. We clarify the fine-tuning corpora, data selection, adapted modules, and evaluation protocols below.
sahil2801/CodeAlpaca-20kBelleGroup/train_1M_CNdeu_Latntest, 900 questionsmeta-math/MetaMathQAsahil2801/CodeAlpaca-20kyahma/alpaca-cleanedFor German,
philschmid/alpaca-deis the provisionally identified source, matching the loader’s first choice; fallback German Alpaca sources were also supported.No additional validation partition is carved out of these fine-tuning subsets. C-Eval’s validation split is used exclusively for evaluation. The seed of 42 above refers to data selection.
For Tables 2, 4, and 5, adapters target
q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj, anddown_projthroughout LLaMA-2-7B. The pretrained weights remain frozen during each adaptation stage. We use learning rate 3×10−4, global batch size 16, rank 16, and block size 32 for block-structured methods. Table 2 compares weight decay 0 and 10−3; Tables 4 and 5 use 10−3. MExRA and its orthogonal low-rank variants, on either action side, apply decay to B while leaving A undecayed. LoRA, OFT, and the block variants use standard AdamW decay.Tables 2 and 4 use three training epochs. Table 5 uses one epoch per stage, an exception to the general three-epoch setting in Section 4.1. The stage order is math → code → general instruction tuning. After each stage, the adapter is merged into the model, and the next stage starts from the merged checkpoint with a newly initialized adapter and optimizer. Earlier-stage training examples are not replayed.
For C-Eval, we use a Chinese-language zero-shot multiple-choice prompt matching the language of the benchmark:
The same template is used for all compared methods, without in-context demonstrations. The evaluator selects among the A/B/C/D answer-token logits and reports the macro-average of subject accuracies.
For German Belebele, the zero-shot prompt is:
All methods use this same template. The evaluator selects among the four answer-token logits and reports question-level accuracy.
For Table 5, all three stage checkpoints use the same zero-shot GSM8K prompt:
Evaluation covers all 1,319 GSM8K test questions using greedy decoding with at most 512 new tokens. The scorer extracts the numerical answer following “The answer is,” or otherwise the last number, and compares it with the reference using an absolute tolerance of 10−5.
For Table 2, adapted models use the Alpaca-style instruction “Please complete the following Python code snippet.” The HumanEval code prompt appears in the input field and as the response prefix. The base model receives the original code prompt directly. Each problem receives one greedy completion, capped at 512 new tokens, followed by completion postprocessing and the official HumanEval functional-correctness evaluator.
The submitted supplement includes the adapter implementations in
model.py, together with dedicated pipelines for the standalone GSM8K and numerical experiments. Task-specific runners for Tables 2, 4, and 5 were not included in that archive; the clarification above supplies their data selection, adaptation targets, and evaluation protocols.