acceptodds
Under review as a conference paper at ICLR 2027

QSpan: Training-Aware Low-Rank Quantization Error Compensation for Fully Quantized LoRA Fine-tuning

Abstract

Parameter-efficient fine-tuning (PEFT) reduces adaptation costs, and quantized PEFT further improves efficiency by primarily quantizing the frozen backbone weights. However, most methods retain activations, gradients, and the LoRA branch in high precision. Fully quantizing both the backbone and LoRA branch reduces these remaining costs but substantially degrades adaptation, as the frozen backbone cannot correct quantization errors and rank-constrained adapters have limited compensation capacity. Existing methods prioritize weight-quantization errors without considering their effects on the low-precision forward computation and training objective. We introduce **QSpan**, a training-aware framework that determines which quantization-error directions each adapter should span and how rank should be distributed across layers. QSpan prioritizes directions of the weight-quantization error according to their exposure through quantized activations and sensitivity to quantized gradients. It then initializes adapters along the most consequential directions through a closed-form solution and allocates a fixed total rank budget toward layers with large fractions of uncompensated training-weighted quantization error. Across language reasoning, personalized language modeling, and text-to-image personalization, QSpan consistently outperforms fully quantized LoRA and representative quantization-aware baselines under matched NVFP4 precision and adaptation budgets. These results demonstrate that aligning limited adapter capacity with training-relevant quantization errors enables effective fully quantized PEFT.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LoRA, Quantization, Fully Quantized Fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_e71hOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

QSpan addresses quantization error in fully quantized LoRA fine-tuning. It initializes adapters through an SVD of the quantization residual, weighted by quantized activation and gradient statistics, and assigns layer ranks before training from the resulting spectra. Experiments cover language reasoning, language personalization, and diffusion personalization under NVFP4. Diffusion results include a fidelity–text-alignment trade-off (Table 4).

Critical strengths

1.The problem is relevant, and the method is coherent. Using activation and gradient statistics to prioritize compensation directions is reasonable. Equations 4–6 correctly solve the stated weighted low-rank approximation problem.

2.The improvements are reasonably consistent across models and tasks (Tables 1–4), and Table 5 shows gains from both initialization and rank allocation. Appendix A describes native NVFP4 execution, while Tables 7–11 report substantial training-memory savings over BF16 LoRA. These results give the implementation practical value.

Critical weaknesses

1.I am not convinced that the paper establishes enough beyond combining existing techniques. PiSSA initializes adapters from the principal components of pretrained weights; LoftQ uses low-rank factors to compensate quantization error; and RA-LoRA already allocates ranks before fine-tuning. QSpan's distinctive choice is the weighting by quantized activation and gradient covariances (Sections 2.3–3.2). EoRA already uses input statistics, and Fisher-weighted LQ-LoRA incorporates gradient-derived importance into quantization-error compensation, but the latter is not discussed.

Table 5 shows that QSpan's components help relative to vanilla LoRA, but does not compare against an existing weighted initialization combined with rank allocation. Nor does it isolate quantized versus BF16 calibration gradients at fixed ranks. Initialization-only gains over EoRA are 0.04–1.32 points, without reported run variability (Tables 1 and 5). QSpan-RandG is undefined and evaluated only through residual metrics (Table 12). The experiments establish that the recipe works, but leave its specific advance over existing ingredients unclear.

2.The analysis and budget matching leave further uncertainty. Appendix B drops the first-order term between Eqs. 8 and 9 without showing that it is negligible. Frozen pretrained weights need not be stationary for the downstream loss. The weighted SVD is correct for its surrogate, but the claimed connection to loss discrepancy is not established.

Section 3.2 fixes total rank, which does not fix adapter parameter count, Padapter=∑lrl(din,l+dout,l)P_{\mathrm{adapter}}=\sum_l r_l(d_{\mathrm{in},l}+d_{\mathrm{out},l}). Figure 5(b) shifts rank toward MLP projections, so actual counts are needed to determine whether changed capacity contributes to the gains.

Table 11 reports modest extra training memory over fully quantized LoRA, but calibration and decomposition time and peak memory are unreported. The calibration-gradient model state, damping values, rank bounds, and treatment of runtime α/rl\alpha/r_l scaling are also insufficiently specified (Eq. 6; Algorithm 1; Appendices A and D). These gaps limit reproducibility and assessment of the practical benefit.

Questions for authors

1.Does QSpan outperform an existing weighted initialization combined with RA-LoRA under the same NVFP4 execution and actual parameter budget? At fixed ranks, how much does using quantized rather than BF16 calibration gradients help? Please include run variability.

2.What justifies omitting the first-order term in Eq. 8? Can its contribution be measured, or should Eq. 3 be presented as a sensitivity surrogate?

3.What are the adapter parameter counts in Table 5 and the baseline comparisons? If they differ materially, do the allocation gains persist when counts are matched?

4.Which model supplies calibration gradients, and what damping, rank bounds, and α/rl\alpha/r_l rescaling are used? What is QSpan-RandG, and how does it perform on downstream tasks?

5.What are the initialization time and peak memory, including calibration and decomposition, separately from training?

10 comments

We thank the reviewer for recognizing QSpan’s relevance, sound weighted-SVD formulation, consistent gains, and practical value. We address the remaining concerns with additional baselines and parameter-budget controls, a revised derivation retaining the first-order term, and initialization-cost measurements and implementation details.

Q1. Distinction from existing methods and additional comparisons

(a) Distinction from Fisher-weighted LQ-LoRA

Our contribution goes beyond gradient-derived importance. While Fisher-weighted LQ-LoRA uses Fisher-weighted reconstruction, QSpan jointly leverages quantized activations and output gradients to initialize adapters and uses the resulting weighted residual spectrum for rank allocation. Crucially, it assigns capacity according to how much training-relevant error remains uncompensated, rather than treating initialization and rank allocation as separate design choices.

We implemented the Fisher-weighted LQ-LoRA baseline with reference to its official code and adapted it to our fully quantized NVFP4 setting. In Table 1, LQ-LoRA denotes the Fisher-weighted LQ-LoRA. Its commonsense averages are 73.94/80.73/71.73/68.13 across the four models, below vanilla NVFP4 LoRA in every case under identical adapter parameter budgets. LQ-LoRA uses a diagonal Fisher weighting designed for weight-only quantization with high-precision forward and backward computation, whereas QSpan uses activation and gradient statistics collected under W4A4G4 execution, capturing the activation and gradient quantization absent from LQ-LoRA's setting.This consistently poor performance suggests that Fisher-weighted LQ-LoRA, as adapted here, is not well suited to fully quantized NVFP4 fine-tuning.

(b) Comparison with existing initialization-plus-allocation combinations

We agree that EoRA already incorporates input statistics. QSpan additionally incorporates quantized output-gradient statistics, prioritizing residual directions according to both their effect on forward activations and their sensitivity to backward signals.

Following the reviewer’s suggestion, we evaluated RA-EoRA, in which both the initialization and the rank allocation follow EoRA's activation-only criterion. RA-LoRA already provides the corresponding LoftQ-based initialization-plus-allocation baseline. Although allocation improves EoRA in several settings, its benefit is not uniform. More importantly, QSpan outperforms both RA-EoRA and RA-LoRA on all four models in both reasoning settings. The following section further shows that this advantage persists when QSpan’s adapter parameter count is matched to uniform LoRA.

(c) Controlling actual adapter parameter count

We agree that fixing total rank does not fix adapter parameter count when projection dimensions differ. We therefore introduce QSpan†, which retains QSpan’s compensation objective and allocation criterion but replaces the total-rank constraint with an adapter-parameter budget matched to uniform LoRA.

Tables 1–2 report all individual benchmark scores and actual adapter parameter counts. QSpan† matches the uniform-LoRA budgets to the reported precision, apart from a 0.01M difference for LLaMA commonsense reasoning.

Table 1. Commonsense reasoning

Accuracy (%). Params denotes adapter parameter count in millions. Bold rows highlight QSpan†, not per-column best results.

LLaMA-3.2-3B-Instruct

MethodParamsBoolQPIQASIQAHSWGARC-eARC-cOBQAAvg.
LoRA (BF16)48.63M69.6381.1274.3686.7969.3888.4777.4782.2078.68
LoRA (NVFP4)48.63M67.8380.1473.6483.3566.0686.3674.7480.0076.52
EoRA48.63M67.2880.7974.1684.1766.9387.0875.1780.4077.00
LQ-LoRA48.63M65.4079.2070.8078.4064.7084.3071.2077.6073.94
RA-LoRA49.14M68.9980.9074.0583.2366.3086.3674.6683.2077.21
RA-EoRA50.86M67.8080.7973.5984.2165.4387.7575.4381.8077.10
QSpan57.28M68.2981.7774.5185.2668.1187.3776.8883.4078.20
QSpan†48.62M68.6581.9974.3185.3567.8887.1676.0983.2078.08

Qwen3-4B

MethodParamsBoolQPIQASIQAHSWGARC-eARC-cOBQAAvg.
LoRA (BF16)66.06M69.2484.7776.4190.1672.3095.8388.9989.0083.34
LoRA (NVFP4)66.06M69.4584.1775.9088.4170.6495.0888.1488.8082.57
EoRA66.06M69.3385.4776.1589.4272.5395.4189.1689.2083.34
LQ-LoRA66.06M66.8083.9075.7085.0071.4093.4084.6085.0080.73
RA-LoRA66.94M68.5384.3975.7487.7571.2795.5088.8289.4082.68
RA-EoRA69.05M70.1285.5376.2389.2971.6495.5489.2589.6083.40
QSpan77.30M69.4285.2078.5689.6174.5195.1688.1490.6083.90
QSpan†66.06M69.4884.9378.1889.7673.7795.3788.1490.5083.77

Qwen3.5-2B

MethodParamsBoolQPIQASIQAHSWGARC-eARC-cOBQAAvg.
LoRA (BF16)33.64M64.0181.2370.3282.3664.4091.7982.6880.0077.10
LoRA (NVFP4)33.64M62.1477.8668.7877.9761.5689.6578.1676.0074.01
EoRA33.64M62.3978.5667.2578.6262.9090.2479.0179.4074.80
LQ-LoRA33.64M60.9076.8068.6068.5063.4087.2072.6075.8071.73
RA-LoRA38.50M64.3178.8467.7677.7564.0990.3677.8278.8074.97
RA-EoRA39.86M58.9077.4868.7879.8663.3090.4078.5079.8074.63
QSpan40.50M64.6579.5470.2779.5666.3890.7079.3581.6076.51
QSpan†33.64M63.7678.7371.0879.8065.5191.0878.9282.2076.39

Gemma4-E2B

MethodParamsBoolQPIQASIQAHSWGARC-eARC-cOBQAAvg.
LoRA (BF16)52.67M65.6978.4073.0878.3265.4388.8977.8278.8075.80
LoRA (NVFP4)52.67M62.7875.9567.4064.8362.4384.9374.4973.4070.78
EoRA52.67M63.5274.9268.9969.2462.1984.5177.8274.0071.90
LQ-LoRA52.67M62.7072.4068.0057.8061.0082.9068.4071.8068.13
RA-LoRA56.60M64.5075.1467.3565.5162.9884.2674.1574.0070.99
RA-EoRA63.93M64.0776.0169.3770.0260.6284.7976.7176.0072.20
QSpan71.31M64.7476.1270.4775.0164.8886.0376.5478.8074.07
QSpan†52.67M64.7478.3570.6274.8063.8885.9476.6278.4074.17

Table 2. Mathematical reasoning

Accuracy (%). Params denotes adapter parameter count in millions. Bold rows highlight QSpan†.

LLaMA-3.2-3B

MethodParamsGSM8KMATHAvg.
LoRA (BF16)48.63M73.8436.6655.25
LoRA (NVFP4)48.63M69.3731.7450.56
EoRA48.63M71.1931.4651.33
RA-LoRA49.14M71.4930.3650.93
RA-EoRA50.86M71.7831.0851.43
QSpan56.11M72.6332.1852.41
QSpan†48.63M72.7832.6452.71

Qwen3-4B

MethodParamsGSM8KMATHAvg.
LoRA (BF16)66.06M84.0046.1465.07
LoRA (NVFP4)66.06M83.0243.2463.13
EoRA66.06M84.0843.0263.55
RA-LoRA66.94M82.1143.1462.62
RA-EoRA69.05M83.6243.7063.66
QSpan75.50M84.5343.7064.12
QSpan†66.06M84.2344.1664.20

Qwen3.5-2B

MethodParamsGSM8KMATHAvg.
LoRA (BF16)33.64M77.4138.9258.16
LoRA (NVFP4)33.64M72.8633.4653.16
EoRA33.64M72.3333.2052.76
RA-LoRA38.50M72.7132.6852.69
RA-EoRA39.86M73.0133.0853.04
QSpan39.96M74.2234.1654.19
QSpan†33.64M74.1133.9854.05

Gemma4-E2B

MethodParamsGSM8KMATHAvg.
LoRA (BF16)52.67M80.8941.0260.96
LoRA (NVFP4)52.67M74.7536.5255.64
EoRA52.67M76.1935.1655.68
RA-LoRA56.60M74.5335.5855.05
RA-EoRA63.93M75.0635.2455.15
QSpan73.38M77.1837.0657.12
QSpan†52.67M76.7237.0256.87
Reviewer_HkBMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores quantizing LoRA to W4A4G4 under a training-aware setting, where the frozen model backbone, the LoRA matrices, the gradients, and the activations are all quantized. The paper proposes QSpan, which incorporates the information of quantized activations and gradients into LoRA initialization, so that the adapters are aware of the quantization errors most relevant to training. The experiments on several 2B-4B language models and two diffusion models show that QSpan improves over many existing quantization-aware LoRA baselines, while reducing training memory compared with BF16 LoRA.

Critical strengths

Additional strength after rebuttal: The authors developed a fully quantized (W4A4G4) NVFP4 LoRA pipeline and evaluated it on a broad range of settings, suggesting that fully quantized LoRA fine-tuning with NVFP4 can achieve good accuracy.


The original review:

  1. The experiments cover a wide range of models (Llama3.2 3B, Qwen3 4B, Qwen3.5 2B, Gemma4 E2B) and tasks (8 commonsense reasoning, 2 math reasoning, personalized language modeling, and text-to-image personalization).

  2. The main idea starts from a reasonable math motivation and proposes a good solution using some interesting techniques.

  3. The research question (W4A4G4) is important and the paper also provides engineering optimization.

Critical weaknesses

Updated after rebuttal: The first concern has been addressed, and I have raised my score from 2 to 3. The second concern is not fully addressed.


The original review:

  1. My main concern is the latency–memory trade-off. This is the key reason for my rating of 2, and I am open to raising it to 3 if this concern is addressed.

The main memory savings come from staged activations. However, in practice, the standard first choice for reducing activation memory is activation checkpointing [1] combined with LoRA or QLoRA. QLoRA itself also recommends turning on gradient checkpointing [2], while the trade-off will be extra latency from recomputing activations[3]. The BF16 LoRA and QLoRA baselines in the paper appear to be measured without checkpointing, while the NVFP4 pipeline itself uses selective recomputation and additional quantization/packing operations (Appendix A). Thus, comparing memory alone against LoRA/QLoRA without checkpointing does not seem to be apples-to-apples.

Please report the memory and training latency together, comparing QLoRA with checkpointing against QSpan. It would be nice to have memory-latency curves of QLoRA (4-bit weights with BF16 computation) with checkpointing on 4/8/12/... layers. If time is limited, please at least compare QSpan against QLoRA with checkpointing on all layers and report the peak memory and training latency under a fair setting. Since QSpan relies on optimized transformer engine and custom CUDA kernels, please use a well-optimized QLoRA implementation for this comparison. If using huggingface transformers + peft + bitsandbytes, please clarify the versions and explain the fairness of comparison.

If QSpan achieves comparable (or only slightly higher) memory usage compared with QLoRA with full checkpointing, while showing a clear latency advantage, I would raise my rating to 3.

Also, since QSpan itself is an initialization and rank-allocation method, if the results show a latency advantage, does it come from hardware-accelerated FP4 GEMMs and the associated implementation optimizations, rather than from QSpan itself? Please clarify whether the memory and latency benefits come from the hardware and implementation, or from the QSpan design.

  1. My second concern is the experimental setting. This is the main reason I would only raise my rating to 3 after the first concern is addressed, and I am open to raising it from 3 to 4 if this concern is also addressed.

The language-model experiments use a maximum sequence length of 512 (Appendix D), which is short for current LLM fine-tuning. Since activation memory grows as sequence length increases [3], the memory savings of fully quantized PEFT are more important for longer sequences[4], e.g., 2048, 4096, or beyond. This is also where the accuracy of fully quantized training may be most affected. Could the authors provide accuracy, memory, and latency results in a long-context setting? For instance, LongBench v1 [5] or another suitable dataset.

[1] Chen, Tianqi, et al. "Training deep nets with sublinear memory cost." arXiv preprint arXiv:1604.06174 (2016).

[2] Dettmers, Tim, et al. "Qlora: Efficient finetuning of quantized llms." Advances in neural information processing systems 36 (2023): 10088-10115.

[3] Korthikanti, Vijay Anand, et al. "Reducing activation recomputation in large transformer models." Proceedings of Machine Learning and Systems 5 (2023): 341-353.

[4] Chen, Yukang, et al. "Longlora: Efficient fine-tuning of long-context large language models." International Conference on Learning Representations. Vol. 2024. 2024.

[5] Bai, Yushi, et al. "Longbench: A bilingual, multitask benchmark for long context understanding." Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers). 2024.

Questions for authors
  1. QSpan relies on the NVFP4 on Blackwell, does the benefit depend on the hardware? Can QSpan also work well on Hopper, Ampere, etc?

  2. Appendix A describes several memory optimizations for the NVFP4 pipeline. What implementations are used for the BF16 LoRA and weight-only quantized baselines in the memory comparisons? Do they use comparable memory optimizations, particularly selective recomputation and buffer reuse?

  3. In appendix A, the authors implement a custom attention operator for Llama and Qwen3, while Gemma 4 and Qwen3.5 use checkpointing token mixer. How much engineering work is required to transfer the method to other different models?

  4. QSpan seems to compute RX−12R_X^{-\frac{1}{2}}, RG−12R_G^{-\frac{1}{2}} and the SVD of weighted residual for each layer. How long do they take? How does the cost of those computation increase as the model becomes larger?

  5. Does the custom implementation also work in parallel settings? For example, sharding the model into several devices, especially when sharding the model with Mixture-of-Experts (MoE) architectures [1]? Does the engineering complexity increase in such settings?

[1] Shazeer, Noam, et al. "Outrageously large neural networks: The sparsely-gated mixture-of-experts layer." ICLR 2017.

7 comments

We thank the reviewer for recognizing the importance of fully quantized PEFT, our mathematical motivation, and the breadth of our experiments. We address the two main concerns about memory–latency comparisons with activation checkpointing and additional 2K/4K-context evaluations, while distinguishing pipeline-level efficiency gains from QSpan’s accuracy improvements.

Q1. Memory–latency Trade-off and Activation Checkpointing + Baselines

We thank the reviewer for this important suggestion. Table 1 compares peak training memory and latency for BF16 LoRA, QLoRA with NF4 backbone and BF16 adapters, and QLoRA with NVFP4 backbone and BF16 adapters, each with and without activation checkpointing (also known as gradient checkpointing) across all layers. We also include fully quantized NVFP4 LoRA fine-tuning without checkpointing to illustrate the memory–latency trade-off.

The BF16 LoRA and QLoRA baselines are implemented with transformers 5.5.4, peft 0.18.1, and bitsandbytes 0.50.2. All configurations are measured on the same B200 GPU with the same model, LoRA rank and sequence length.

Table 1. Peak training memory and latency with and without activation checkpointing. Slowdown is relative to BF16 / BF16 without activation checkpointing for each model.

ModelBase / LoRAActivation CheckpointingPeak memory (GB) ↓Latency (s) ↓Slowdown
LLaMA-3.2-3BBF16 / BF16X43.600.851.00×
BF16 / BF16O22.531.191.40×
NF4 / BF16X32.131.101.29×
NF4 / BF16O20.421.401.65×
NVFP4 / BF16X29.400.851.00×
NVFP4 / BF16O21.561.011.19×
NVFP4 / NVFP4X22.820.620.73×
Qwen3-4BBF16 / BF16X63.961.361.00×
BF16 / BF16O26.811.871.38×
NF4 / BF16X42.611.751.29×
NF4 / BF16O24.122.231.64×
NVFP4 / BF16X35.401.230.90×
NVFP4 / BF16O25.611.511.11×
NVFP4 / NVFP4X26.750.860.63×
Qwen3.5-2BBF16 / BF16X67.000.671.00×
BF16 / BF16O31.630.991.48×
NF4 / BF16X39.901.081.61×
NF4 / BF16O30.891.261.88×
NVFP4 / BF16X38.391.151.72×
NVFP4 / BF16O30.961.412.10×
NVFP4 / NVFP4X33.780.590.88×
Gemma4-E2BBF16 / BF16X74.671.311.00×
BF16 / BF16O43.641.801.37×
NF4 / BF16X62.851.711.31×
NF4 / BF16O43.092.011.53×
NVFP4 / BF16X58.941.491.14×
NVFP4 / BF16O42.881.711.31×
NVFP4 / NVFP4X48.250.880.67×

Activation checkpointing reduces peak memory but increases latency for all three baseline configurations across all four models. Fully quantized NVFP4 execution is faster than every checkpointed baseline, while reducing both memory and latency relative to standard BF16 LoRA and NF4/BF16 QLoRA without checkpointing. Checkpointing can achieve lower memory usage, but at the cost of additional recomputation. Fully quantized execution therefore provides a complementary memory–latency operating point.

These experiments demonstrate the favorable memory–latency trade-off of our optimized fully quantized PEFT pipeline compared with QLoRA’s weight-only quantization and activation checkpointing’s recomputation-based memory savings. These gains reflect our pipeline optimizations leveraging native NVFP4 computation.

Q2. Evaluation with Longer Contexts

To evaluate beyond the original sequence length of 512, we additionally fine-tuned and evaluated on HotpotQA at sequence lengths of 2048 and 4096, using inputs containing ten paragraphs and a question. Following the LLoCO [1] setup, we trained on 20K randomly sampled training examples and evaluated answer F1/EM on the full validation set of 7,405 questions, with answer-only supervision and greedy decoding.

All methods use rank 32 (α = 64) on all linear layers; EoRA and QSpan use 128 calibration samples. Training uses AdamW for three epochs with a learning rate of 2e-4, cosine scheduling, and 4% warm-up. Peak allocated memory and CUDA Graph latency are measured on a single B200 with micro-batch size 4 and gradient accumulation of 8, resulting in a similar peak memory footprint to the batch-16, length-512 setting in BF16 LoRA in Table 1.

[1] "LLoCO: Learning Long Contexts Offline," Sijun et al., EMNLP 2024

Table 2. Accuracy, peak training memory, and training latency on HotpotQA. Bold indicates the best F1/EM among NVFP4 methods.

Qwen3-4B

Seq. lengthMethodF1 ↑EM ↑Training memory (GB) ↓Latency (ms) ↓
2048Zero-shot (BF16)42.4831.72——
2048LoRA (BF16)49.5838.4963.962865.1
2048LoRA (NVFP4)48.5937.6527.641824.2
2048EoRA48.2737.6427.641805.3
2048QSpan48.8537.8128.231826.6
4096Zero-shot (BF16)42.7431.91——
4096LoRA (BF16)49.1738.50118.416465.4
4096LoRA (NVFP4)47.9737.6151.694600.6
4096EoRA48.0437.2351.694639.7
4096QSpan48.8237.9552.594597.3

LLaMA-3.2-3B

Seq. lengthMethodF1 ↑EM ↑Training memory (GB) ↓Latency (ms) ↓
2048Zero-shot (BF16)38.4928.56——
2048LoRA (BF16)52.1840.8043.601816.4
2048LoRA (NVFP4)50.0139.0121.981261.5
2048EoRA50.0939.1821.981260.7
2048QSpan50.3539.5422.421263.1
4096Zero-shot (BF16)38.7228.75——
4096LoRA (BF16)50.5939.5880.854037.0
4096LoRA (NVFP4)48.6038.0540.913074.4
4096EoRA48.9938.2740.913100.9
4096QSpan49.8238.9841.563077.3

Longer contexts increase the practical value of low-precision activation storage. Peak training memory rises substantially with sequence length for all methods, but QSpan’s fully quantized execution keeps it well below BF16 LoRA. Importantly, the absolute memory savings increase at the longer sequence length on both models, while training latency remains lower than BF16 LoRA.

QSpan’s accuracy benefits also persist at longer contexts. It achieves the highest F1 and EM among the NVFP4 methods on both models at both sequence lengths, with larger improvements over standard NVFP4 LoRA at 4096 tokens. Although a gap to BF16 LoRA remains, these results show that QSpan improves the accuracy–efficiency trade-off beyond the original short-sequence setting, with only modest memory overhead and comparable latency relative to standard NVFP4 LoRA.

Reviewer_FEv9OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

QSpan addresses fully quantized PEFT, defined here as NVFP4 W4A4G4 computation in the linear projections of both the frozen backbone and the trainable LoRA branch. The method first estimates quantized input-activation and output-gradient statistics from calibration samples. Under a local quadratic and K-FAC-style approximation, these statistics induce a two-sided weighted norm on the frozen backbone's weight-quantization error. A truncated SVD of the whitened error gives a closed-form LoRA initialization. QSpan then allocates a fixed total rank budget across layers according to the fraction of this training-weighted residual left beyond a uniform baseline rank. The same decomposition thus determines both the initial adapter directions and where additional rank goes.

Critical strengths

The method is well matched to the failure mode it targets. When the backbone cannot move and LoRA rank is limited, a plain truncated SVD of weight error may spend capacity on directions that scarcely affect the current training objective. QSpan's weighted residual reduces to activation-aware compensation when the gradient-side factor is identity and to ordinary weight-error SVD when both factors are identity. This makes the relation to LoftQ and EoRA concrete. The paper also uses actual NVFP4 GEMMs for the relevant forward and backward projections, with packed activations in both branches, so the precision setting is more than a storage-only approximation.

The breadth of validation is a second strength. Gains appear on four language backbones and extend beyond multiple-choice reasoning to math and user personalization. Table 5 shows that initialization improves all four commonsense averages and that reallocating rank improves them again without increasing the logical rank budget. The residual analysis is directionally consistent: on LLaMA-3.2-3B, training-weighted residual falls from 6.54 for uniform LoRA to 1.76 after QSpan initialization and 1.66 after rank allocation, in the table's units. Memory is also reported rather than simply inferred from bitwidth: the appendix gives language-model peak memory reductions from 43.600–74.674 GB in BF16 to 22.816–48.254 GB in uniform NVFP4, with QSpan's extra peak memory generally around 1–2% on those models.

Critical weaknesses
  1. The main accuracy tables give single values without seed variation or confidence intervals. Some improvements over strong NVFP4 baselines are modest: Qwen3-4B's commonsense average is 83.90 versus 83.34 for EoRA, and its math average is 64.12 versus 63.55. On FLUX.2-Klein, QSpan leads on DINO and CLIP-I, but its CLIP-T score of 0.3012 is below the BF16 and several quantized baselines, indicating a subject-fidelity/text-alignment tradeoff. Repeated fine-tuning runs and evaluation uncertainty are needed before treating every small margin as robust.

  2. The efficiency story is incomplete at initialization. The paper documents peak training memory and says the rank allocation adds no extra decomposition beyond the initialization SVD, but it does not give a comparable end-to-end time breakdown for collecting 128 calibration samples, forming two covariance factors for each layer, taking matrix square roots and performing the decompositions. The local Fisher/Gauss-Newton and K-FAC approximations are plausible design choices, yet the calibration gradients may be sensitive to the chosen tasks and examples. As written, it is hard to tell whether the gains require task-specific labeled calibration data and whether a cheaper activation-only or random-gradient variant would be almost as good at the same total setup cost.

Questions for authors
  1. Could the authors report mean and variance over independent fine-tuning seeds, ideally with paired comparisons for the strongest NVFP4 baselines on the four language models? This is particularly important for the sub-point differences on Qwen3-4B and the diffusion metrics.

  2. How long does QSpan initialization take, and what is its peak temporary memory, relative to the training run and to EoRA/LoftQ initialization? Please include an ablation varying calibration-set size and source, and a comparison with an activation-only initialization plus the same rank-allocation rule. That would clarify whether the backward-sensitivity factor is the material contribution.

5 comments

We thank the reviewer for recognizing QSpan’s training-aware design, the breadth of our evaluation, and our actual NVFP4 implementation. We also appreciate the valuable feedback on experimental robustness and initialization efficiency.

Q1: Robustness across random seeds

Following the reviewer’s suggestion, we conducted five-seed commonsense reasoning experiments on all four language models and additional five-seed experiments on FLUX.2-Klein. The tables below report mean ± standard deviation, with the highest mean for each model and metric in bold.

Table 1. Commonsense reasoning accuracy over five fine-tuning seeds.

ModelMethodBoolQPIQASIQAHellaSwagWinoGrandeARC-eARC-cOBQAAvg
LLaMA-3.2-3BLoRA (NVFP4)67.91±0.2480.45±0.3973.90±0.2182.98±0.4566.76±0.6086.36±0.2174.62±0.3380.16±0.5076.64±0.14
EoRA67.84±0.3680.87±0.2473.84±0.5284.06±0.7666.50±0.6787.04±0.4274.44±0.8080.72±0.8176.91±0.21
QSpan68.41±0.3881.85±0.4474.48±0.2785.08±0.2467.70±0.2787.05±0.3476.86±0.8582.94±1.0078.05±0.19
Qwen3-4BLoRA (NVFP4)69.05±0.6284.35±0.2475.84±0.2688.37±0.5071.18±0.7095.21±0.2488.07±0.7589.20±0.7682.66±0.20
EoRA69.51±0.5185.26±0.1776.48±0.5789.24±0.1271.89±0.7295.54±0.2788.60±0.5190.00±0.8683.32±0.19
QSpan69.50±0.5985.21±0.0877.99±0.8289.25±0.2874.50±0.7495.42±0.2788.04±0.5390.96±1.4983.86±0.26
Qwen3.5-2BLoRA (NVFP4)62.39±1.1677.35±0.8968.94±0.3378.09±0.6662.11±0.7690.33±0.2978.95±0.8278.05±2.0974.53±0.36
EoRA62.28±1.3577.88±0.7368.43±1.1078.06±0.3063.20±1.0290.35±0.4679.03±0.8578.54±0.3074.72±0.30
QSpan64.69±0.9979.14±0.5969.98±0.5579.62±0.2166.16±0.8590.41±0.2879.78±0.6180.08±1.8076.23±0.31
Gemma4-E2BLoRA (NVFP4)62.20±0.6374.97±2.5968.15±1.0872.17±1.6761.77±0.5784.51±0.7875.37±0.9775.40±0.8071.82±0.46
EoRA63.84±0.2575.06±1.4668.73±0.7971.16±1.7962.81±0.4484.51±1.6775.81±0.7075.20±1.3772.14±0.42
QSpan64.59±0.4177.04±0.7870.86±0.5275.00±0.3464.28±0.5386.09±0.2077.05±0.8278.56±1.1374.18±0.23

QSpan retains the highest average accuracy on all four language models, preserving the overall ordering observed in the original fixed-seed experiments. This also holds for Qwen3-4B, where the reviewer highlighted the small margin: QSpan achieves 83.86±0.26, compared with EoRA’s 83.32±0.19.

Table 2. Multi-seed personalization results on FLUX.2-Klein (mean ± standard deviation). Higher is better for all metrics.

MethodDINO ↑CLIP-I ↑CLIP-T ↑
LoRA (NVFP4)0.7106±0.00200.8056±0.00150.3075±0.0009
EoRA0.7218±0.00330.8135±0.00290.3043±0.0007
QSpan0.7307±0.00150.8195±0.00170.3014±0.0010

The multi-seed results reproduce the original trends: QSpan achieves the highest mean DINO and CLIP-I scores, with a slightly lower CLIP-T score. In our experiments, CLIP-T tended to decrease as personalization training progressed. At the same number of training iterations, QSpan’s higher subject-fidelity scores and modestly lower CLIP-T are consistent with more effective subject-specific learning, suggesting that QSpan reaches stronger personalization within the same training iterations. We therefore interpret this pattern as stronger personalization with an accompanying text-alignment trade-off, rather than a general degradation in generation quality.

Q2. Initialization cost

We additionally measured the initialization time and memory of LoftQ, EoRA, and QSpan, alongside QSpan’s training time.

Table 3. Initialization entries report time / memory. Training times are estimated from per-update latency multiplied by the number of updates and are approximate due to rounding. Actual end-to-end times are longer due to additional overhead from various reasons, including weight loading, checkpoint saving, and server CPU contention.

ModelLoftQ init. (time / memory)EoRA init. (time / memory)QSpan init. (time / memory)Commonsense training (time)MetaMath training (time)
LLaMA-3.2-3B2.4s / 7.7GB4.6s / 20.8GB7.5s / 25.1GB438s738s
Qwen3-4B3.1s / 9.4GB5.8s / 30.8GB10.1s / 38.2GB606s1,020s
Qwen3.5-2B2.3s / 4.5GB2.9s / 12.4GB5.5s / 14.3GB414s702s
Gemma4-E2B2.3s / 10.0GB5.6s / 32.0GB10.0s / 36.1GB618s1,044s

QSpan’s initialization takes 5.5–10.1 seconds and is performed only once, not at every training update. This is small relative to the subsequent fine-tuning time. Its additional initialization-memory requirement supports quantization-error compensation informed by both activation statistics and backward sensitivity.

For larger models, initialization cost depends on the number and dimensions of the adapted matrices, as well as calibration computation; it should not be extrapolated solely from parameter count. The table reports end-to-end initialization rather than isolated SVD timings. We have not benchmarked larger models and therefore limit our empirical overhead claims to the evaluated architectures.

Q3. Activation-only allocation and parameter-budget controls

To test whether activation-only initialization with rank allocation is sufficient, we evaluated RA-EoRA, in which both the initialization and the rank allocation follow EoRA's activation-only criterion. We also evaluated QSpan†, which retains QSpan’s objective and allocation criterion while matching uniform LoRA’s adapter-parameter budget rather than its total rank.

Table 4. Commonsense reasoning accuracy (%) / adapter parameters (millions).

ModelEoRARA-EoRAQSpanQSpan†
LLaMA-3.2-3B77.00 / 48.63M77.10 / 50.86M78.20 / 57.28M78.08 / 48.62M
Qwen3-4B83.34 / 66.06M83.40 / 69.05M83.90 / 77.30M83.77 / 66.06M
Qwen3.5-2B74.80 / 33.64M74.63 / 39.86M76.51 / 40.50M76.39 / 33.64M
Gemma4-E2B71.90 / 52.67M72.20 / 63.93M74.07 / 71.31M74.17 / 52.67M

Table 5. Mathematical reasoning accuracy (%) / adapter parameters (millions).

ModelEoRARA-EoRAQSpanQSpan†
LLaMA-3.2-3B51.33 / 48.63M51.43 / 50.86M52.41 / 56.11M52.71 / 48.63M
Qwen3-4B63.55 / 66.06M63.66 / 69.05M64.12 / 75.50M64.20 / 66.06M
Qwen3.5-2B52.76 / 33.64M53.04 / 39.86M54.19 / 39.96M54.05 / 33.64M
Gemma4-E2B55.68 / 52.67M55.15 / 63.93M57.12 / 73.38M56.87 / 52.67M

QSpan† outperforms RA-EoRA across all four models in both settings, despite using fewer adapter parameters in every comparison. It matches uniform LoRA’s parameter budgets to the reported precision, except for a 0.01M difference on LLaMA commonsense, while remaining within 0.13/0.30 points of QSpan on commonsense/math. These results support training-aware initialization and allocation beyond an activation-only alternative, without relying on increased adapter capacity.

Q4. Calibration Data

QSpan estimates quantized activation and output-gradient statistics from calibration samples, which should reasonably represent the target distribution but need not come from the exact fine-tuning dataset. In our text-to-image personalization experiments, we use CustomConcept101 for calibration and DreamBooth for fine-tuning, without including target subjects in calibration. For LaMP, QSpan is calibrated once per task and reused across all 100 users, demonstrating that user-specific calibration is not required in this setting. To assess sensitivity to calibration-set size, we additionally evaluated QSpan and EoRA using 32, 64, and 128 calibration samples on three language models.

Table 6. Commonsense reasoning average accuracy (%) across calibration-set sizes.

ModelMethod3264128
LLaMA-3.2-3BQSpan77.9377.9678.20
EoRA77.1377.0377.00
Gemma4-E2BQSpan74.1274.1574.07
EoRA72.4072.0371.90
Qwen3.5-2BQSpan76.1776.7776.51
EoRA74.4774.9774.80

QSpan consistently outperforms EoRA across all tested calibration sizes and models. Even with only 32 samples, QSpan maintains its advantage, while increasing the sample count does not yield consistent improvements. These results suggest that QSpan is relatively insensitive to calibration-set size within the evaluated range and does not require all 128 samples to achieve competitive performance.

Together, these findings demonstrate that calibration statistics can be reused across users and that QSpan remains effective with a small calibration set. Sensitivity to calibration-data source remains to be systematically evaluated.

Comments on the paper

1 comment

As only a few days remain in the author-reviewer discussion period, I encourage reviewers to review the authors' rebuttal and actively participate in the discussion. In particular, please clarify which concerns have been addressed and identify any remaining concerns or questions that may require further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.