- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
QSpan addresses quantization error in fully quantized LoRA fine-tuning. It initializes adapters through an SVD of the quantization residual, weighted by quantized activation and gradient statistics, and assigns layer ranks before training from the resulting spectra. Experiments cover language reasoning, language personalization, and diffusion personalization under NVFP4. Diffusion results include a fidelity–text-alignment trade-off (Table 4).
Critical strengths
1.The problem is relevant, and the method is coherent. Using activation and gradient statistics to prioritize compensation directions is reasonable. Equations 4–6 correctly solve the stated weighted low-rank approximation problem.
2.The improvements are reasonably consistent across models and tasks (Tables 1–4), and Table 5 shows gains from both initialization and rank allocation. Appendix A describes native NVFP4 execution, while Tables 7–11 report substantial training-memory savings over BF16 LoRA. These results give the implementation practical value.
Critical weaknesses
1.I am not convinced that the paper establishes enough beyond combining existing techniques. PiSSA initializes adapters from the principal components of pretrained weights; LoftQ uses low-rank factors to compensate quantization error; and RA-LoRA already allocates ranks before fine-tuning. QSpan's distinctive choice is the weighting by quantized activation and gradient covariances (Sections 2.3–3.2). EoRA already uses input statistics, and Fisher-weighted LQ-LoRA incorporates gradient-derived importance into quantization-error compensation, but the latter is not discussed.
Table 5 shows that QSpan's components help relative to vanilla LoRA, but does not compare against an existing weighted initialization combined with rank allocation. Nor does it isolate quantized versus BF16 calibration gradients at fixed ranks. Initialization-only gains over EoRA are 0.04–1.32 points, without reported run variability (Tables 1 and 5). QSpan-RandG is undefined and evaluated only through residual metrics (Table 12). The experiments establish that the recipe works, but leave its specific advance over existing ingredients unclear.
2.The analysis and budget matching leave further uncertainty. Appendix B drops the first-order term between Eqs. 8 and 9 without showing that it is negligible. Frozen pretrained weights need not be stationary for the downstream loss. The weighted SVD is correct for its surrogate, but the claimed connection to loss discrepancy is not established.
Section 3.2 fixes total rank, which does not fix adapter parameter count, . Figure 5(b) shifts rank toward MLP projections, so actual counts are needed to determine whether changed capacity contributes to the gains.
Table 11 reports modest extra training memory over fully quantized LoRA, but calibration and decomposition time and peak memory are unreported. The calibration-gradient model state, damping values, rank bounds, and treatment of runtime scaling are also insufficiently specified (Eq. 6; Algorithm 1; Appendices A and D). These gaps limit reproducibility and assessment of the practical benefit.
Questions for authors
1.Does QSpan outperform an existing weighted initialization combined with RA-LoRA under the same NVFP4 execution and actual parameter budget? At fixed ranks, how much does using quantized rather than BF16 calibration gradients help? Please include run variability.
2.What justifies omitting the first-order term in Eq. 8? Can its contribution be measured, or should Eq. 3 be presented as a sensitivity surrogate?
3.What are the adapter parameter counts in Table 5 and the baseline comparisons? If they differ materially, do the allocation gains persist when counts are matched?
4.Which model supplies calibration gradients, and what damping, rank bounds, and rescaling are used? What is QSpan-RandG, and how does it perform on downstream tasks?
5.What are the initialization time and peak memory, including calibration and decomposition, separately from training?
10 comments
Table 1. Commonsense reasoning
Accuracy (%). Params denotes adapter parameter count in millions. Bold rows highlight QSpan†, not per-column best results.
LLaMA-3.2-3B-Instruct
| Method | Params | BoolQ | PIQA | SIQA | HS | WG | ARC-e | ARC-c | OBQA | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| LoRA (BF16) | 48.63M | 69.63 | 81.12 | 74.36 | 86.79 | 69.38 | 88.47 | 77.47 | 82.20 | 78.68 |
| LoRA (NVFP4) | 48.63M | 67.83 | 80.14 | 73.64 | 83.35 | 66.06 | 86.36 | 74.74 | 80.00 | 76.52 |
| EoRA | 48.63M | 67.28 | 80.79 | 74.16 | 84.17 | 66.93 | 87.08 | 75.17 | 80.40 | 77.00 |
| LQ-LoRA | 48.63M | 65.40 | 79.20 | 70.80 | 78.40 | 64.70 | 84.30 | 71.20 | 77.60 | 73.94 |
| RA-LoRA | 49.14M | 68.99 | 80.90 | 74.05 | 83.23 | 66.30 | 86.36 | 74.66 | 83.20 | 77.21 |
| RA-EoRA | 50.86M | 67.80 | 80.79 | 73.59 | 84.21 | 65.43 | 87.75 | 75.43 | 81.80 | 77.10 |
| QSpan | 57.28M | 68.29 | 81.77 | 74.51 | 85.26 | 68.11 | 87.37 | 76.88 | 83.40 | 78.20 |
| QSpan† | 48.62M | 68.65 | 81.99 | 74.31 | 85.35 | 67.88 | 87.16 | 76.09 | 83.20 | 78.08 |
Qwen3-4B
| Method | Params | BoolQ | PIQA | SIQA | HS | WG | ARC-e | ARC-c | OBQA | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| LoRA (BF16) | 66.06M | 69.24 | 84.77 | 76.41 | 90.16 | 72.30 | 95.83 | 88.99 | 89.00 | 83.34 |
| LoRA (NVFP4) | 66.06M | 69.45 | 84.17 | 75.90 | 88.41 | 70.64 | 95.08 | 88.14 | 88.80 | 82.57 |
| EoRA | 66.06M | 69.33 | 85.47 | 76.15 | 89.42 | 72.53 | 95.41 | 89.16 | 89.20 | 83.34 |
| LQ-LoRA | 66.06M | 66.80 | 83.90 | 75.70 | 85.00 | 71.40 | 93.40 | 84.60 | 85.00 | 80.73 |
| RA-LoRA | 66.94M | 68.53 | 84.39 | 75.74 | 87.75 | 71.27 | 95.50 | 88.82 | 89.40 | 82.68 |
| RA-EoRA | 69.05M | 70.12 | 85.53 | 76.23 | 89.29 | 71.64 | 95.54 | 89.25 | 89.60 | 83.40 |
| QSpan | 77.30M | 69.42 | 85.20 | 78.56 | 89.61 | 74.51 | 95.16 | 88.14 | 90.60 | 83.90 |
| QSpan† | 66.06M | 69.48 | 84.93 | 78.18 | 89.76 | 73.77 | 95.37 | 88.14 | 90.50 | 83.77 |
Qwen3.5-2B
| Method | Params | BoolQ | PIQA | SIQA | HS | WG | ARC-e | ARC-c | OBQA | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| LoRA (BF16) | 33.64M | 64.01 | 81.23 | 70.32 | 82.36 | 64.40 | 91.79 | 82.68 | 80.00 | 77.10 |
| LoRA (NVFP4) | 33.64M | 62.14 | 77.86 | 68.78 | 77.97 | 61.56 | 89.65 | 78.16 | 76.00 | 74.01 |
| EoRA | 33.64M | 62.39 | 78.56 | 67.25 | 78.62 | 62.90 | 90.24 | 79.01 | 79.40 | 74.80 |
| LQ-LoRA | 33.64M | 60.90 | 76.80 | 68.60 | 68.50 | 63.40 | 87.20 | 72.60 | 75.80 | 71.73 |
| RA-LoRA | 38.50M | 64.31 | 78.84 | 67.76 | 77.75 | 64.09 | 90.36 | 77.82 | 78.80 | 74.97 |
| RA-EoRA | 39.86M | 58.90 | 77.48 | 68.78 | 79.86 | 63.30 | 90.40 | 78.50 | 79.80 | 74.63 |
| QSpan | 40.50M | 64.65 | 79.54 | 70.27 | 79.56 | 66.38 | 90.70 | 79.35 | 81.60 | 76.51 |
| QSpan† | 33.64M | 63.76 | 78.73 | 71.08 | 79.80 | 65.51 | 91.08 | 78.92 | 82.20 | 76.39 |
Gemma4-E2B
| Method | Params | BoolQ | PIQA | SIQA | HS | WG | ARC-e | ARC-c | OBQA | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| LoRA (BF16) | 52.67M | 65.69 | 78.40 | 73.08 | 78.32 | 65.43 | 88.89 | 77.82 | 78.80 | 75.80 |
| LoRA (NVFP4) | 52.67M | 62.78 | 75.95 | 67.40 | 64.83 | 62.43 | 84.93 | 74.49 | 73.40 | 70.78 |
| EoRA | 52.67M | 63.52 | 74.92 | 68.99 | 69.24 | 62.19 | 84.51 | 77.82 | 74.00 | 71.90 |
| LQ-LoRA | 52.67M | 62.70 | 72.40 | 68.00 | 57.80 | 61.00 | 82.90 | 68.40 | 71.80 | 68.13 |
| RA-LoRA | 56.60M | 64.50 | 75.14 | 67.35 | 65.51 | 62.98 | 84.26 | 74.15 | 74.00 | 70.99 |
| RA-EoRA | 63.93M | 64.07 | 76.01 | 69.37 | 70.02 | 60.62 | 84.79 | 76.71 | 76.00 | 72.20 |
| QSpan | 71.31M | 64.74 | 76.12 | 70.47 | 75.01 | 64.88 | 86.03 | 76.54 | 78.80 | 74.07 |
| QSpan† | 52.67M | 64.74 | 78.35 | 70.62 | 74.80 | 63.88 | 85.94 | 76.62 | 78.40 | 74.17 |
Table 2. Mathematical reasoning
Accuracy (%). Params denotes adapter parameter count in millions. Bold rows highlight QSpan†.
LLaMA-3.2-3B
| Method | Params | GSM8K | MATH | Avg. |
|---|---|---|---|---|
| LoRA (BF16) | 48.63M | 73.84 | 36.66 | 55.25 |
| LoRA (NVFP4) | 48.63M | 69.37 | 31.74 | 50.56 |
| EoRA | 48.63M | 71.19 | 31.46 | 51.33 |
| RA-LoRA | 49.14M | 71.49 | 30.36 | 50.93 |
| RA-EoRA | 50.86M | 71.78 | 31.08 | 51.43 |
| QSpan | 56.11M | 72.63 | 32.18 | 52.41 |
| QSpan† | 48.63M | 72.78 | 32.64 | 52.71 |
Qwen3-4B
| Method | Params | GSM8K | MATH | Avg. |
|---|---|---|---|---|
| LoRA (BF16) | 66.06M | 84.00 | 46.14 | 65.07 |
| LoRA (NVFP4) | 66.06M | 83.02 | 43.24 | 63.13 |
| EoRA | 66.06M | 84.08 | 43.02 | 63.55 |
| RA-LoRA | 66.94M | 82.11 | 43.14 | 62.62 |
| RA-EoRA | 69.05M | 83.62 | 43.70 | 63.66 |
| QSpan | 75.50M | 84.53 | 43.70 | 64.12 |
| QSpan† | 66.06M | 84.23 | 44.16 | 64.20 |
Qwen3.5-2B
| Method | Params | GSM8K | MATH | Avg. |
|---|---|---|---|---|
| LoRA (BF16) | 33.64M | 77.41 | 38.92 | 58.16 |
| LoRA (NVFP4) | 33.64M | 72.86 | 33.46 | 53.16 |
| EoRA | 33.64M | 72.33 | 33.20 | 52.76 |
| RA-LoRA | 38.50M | 72.71 | 32.68 | 52.69 |
| RA-EoRA | 39.86M | 73.01 | 33.08 | 53.04 |
| QSpan | 39.96M | 74.22 | 34.16 | 54.19 |
| QSpan† | 33.64M | 74.11 | 33.98 | 54.05 |
Gemma4-E2B
| Method | Params | GSM8K | MATH | Avg. |
|---|---|---|---|---|
| LoRA (BF16) | 52.67M | 80.89 | 41.02 | 60.96 |
| LoRA (NVFP4) | 52.67M | 74.75 | 36.52 | 55.64 |
| EoRA | 52.67M | 76.19 | 35.16 | 55.68 |
| RA-LoRA | 56.60M | 74.53 | 35.58 | 55.05 |
| RA-EoRA | 63.93M | 75.06 | 35.24 | 55.15 |
| QSpan | 73.38M | 77.18 | 37.06 | 57.12 |
| QSpan† | 52.67M | 76.72 | 37.02 | 56.87 |
We thank the reviewer for recognizing QSpan’s relevance, sound weighted-SVD formulation, consistent gains, and practical value. We address the remaining concerns with additional baselines and parameter-budget controls, a revised derivation retaining the first-order term, and initialization-cost measurements and implementation details.
Q1. Distinction from existing methods and additional comparisons
(a) Distinction from Fisher-weighted LQ-LoRA
Our contribution goes beyond gradient-derived importance. While Fisher-weighted LQ-LoRA uses Fisher-weighted reconstruction, QSpan jointly leverages quantized activations and output gradients to initialize adapters and uses the resulting weighted residual spectrum for rank allocation. Crucially, it assigns capacity according to how much training-relevant error remains uncompensated, rather than treating initialization and rank allocation as separate design choices.
We implemented the Fisher-weighted LQ-LoRA baseline with reference to its official code and adapted it to our fully quantized NVFP4 setting. In Table 1, LQ-LoRA denotes the Fisher-weighted LQ-LoRA. Its commonsense averages are 73.94/80.73/71.73/68.13 across the four models, below vanilla NVFP4 LoRA in every case under identical adapter parameter budgets. LQ-LoRA uses a diagonal Fisher weighting designed for weight-only quantization with high-precision forward and backward computation, whereas QSpan uses activation and gradient statistics collected under W4A4G4 execution, capturing the activation and gradient quantization absent from LQ-LoRA's setting.This consistently poor performance suggests that Fisher-weighted LQ-LoRA, as adapted here, is not well suited to fully quantized NVFP4 fine-tuning.
(b) Comparison with existing initialization-plus-allocation combinations
We agree that EoRA already incorporates input statistics. QSpan additionally incorporates quantized output-gradient statistics, prioritizing residual directions according to both their effect on forward activations and their sensitivity to backward signals.
Following the reviewer’s suggestion, we evaluated RA-EoRA, in which both the initialization and the rank allocation follow EoRA's activation-only criterion. RA-LoRA already provides the corresponding LoftQ-based initialization-plus-allocation baseline. Although allocation improves EoRA in several settings, its benefit is not uniform. More importantly, QSpan outperforms both RA-EoRA and RA-LoRA on all four models in both reasoning settings. The following section further shows that this advantage persists when QSpan’s adapter parameter count is matched to uniform LoRA.
(c) Controlling actual adapter parameter count
We agree that fixing total rank does not fix adapter parameter count when projection dimensions differ. We therefore introduce QSpan†, which retains QSpan’s compensation objective and allocation criterion but replaces the total-rank constraint with an adapter-parameter budget matched to uniform LoRA.
Tables 1–2 report all individual benchmark scores and actual adapter parameter counts. QSpan† matches the uniform-LoRA budgets to the reported precision, apart from a 0.01M difference for LLaMA commonsense reasoning.