- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ADE for delta compression, which stores only the weight difference between each fine-tuned model and the shared base model. ADE compresses each delta with SVD and 4-bit quantization, and adapts it to new tasks by training two scaling vectors per layer. Experiments on six fine-tuned models show higher accuracy than prior methods and lower latency for short inputs.
Critical strengths
- The paper considers accuracy, inference speed, and adaptability to new tasks together for serving many fine-tuned models.
- Even without extra training, ADE−L still matches or outperforms previous delta compression methods on all six models (Table 1).
Critical weaknesses
- The latency comparison may be unfair. ADE runs on custom CUDA kernels optimized by the authors, while the baselines run on their released code or standard PyTorch (Appendix K), so part of the speedup may come from engineering rather than the method.
- ADE uses real data to decide how much to compress each layer, by measuring how much the layer’s output changes (Eq. 4). However, it decides which parts to keep only by their size in the weight matrix via SVD. So, the kept parts may not be the ones that matter most on real data.
Questions for authors
Please refer to the weakness part.
3 comments
[W2] Data-agnostic selection of retained components
We agree with the reviewer. ADE uses calibration activations to allocate ranks across modules (Eq. 4), whereas the standard SVD selects the components within a module by their magnitude in weight space. We tested activation-aware selection in ADE. In our setting, however, it did not improve accuracy and even lowered it slightly.
Activation-aware SVD widens the value range of the factors. We replaced the standard SVD with SVD-LLM [1]. SVD-LLM multiplies the delta by the Cholesky factor of the activation covariance before the SVD, and maps the right factor back by . This transformation widens the value range of the factors: the 99th percentile of the group range/RMS of grows from 6.8 to 9.1 (Table R1, without rotation). The range/RMS of a quantization group is the difference between its maximum and minimum values divided by the RMS of its values, and a larger ratio means coarser quantization; its 99th percentile is the value above which the 1% of groups with the largest ratio lie. Under the same bit-width and group size, the wider range increases the quantization error. Because each quantization group mixes large and small values, the relative quantization error of the factor rises from 0.104 to 0.124. Quantization therefore cancels the advantage that SVD-LLM had in output space.
Table R1. Quantization of the factors (WizardMath-13B, all 280 modules at the ranks of ADE$_{-\mathcal{L}}$, 4-bit, ).
| Statistic | SVD, no rotation | SVD-LLM, no rotation | SVD, rotation (ADE) | SVD-LLM, rotation (ADE) |
|---|---|---|---|---|
| Group range/RMS of (99th percentile) | 6.8 | 9.1 | 6.5 | 6.5 |
| Relative quantization error of | 0.104 | 0.124 | 0.100 | 0.099 |
| 0.084 | 0.084 | 0.064 | 0.051 |
ADE's rotation reduces this error, but accuracy does not improve. ADE quantizes the factors after applying the rotation . This largely removes the penalty above: the group range/RMS and the relative quantization error of the two decompositions become equal (Table R1).
An ablation confirmed that activation-aware SVD does not help. We replaced only the SVD of ADE$_{-\mathcal{L}}$ in Table 2 of the paper with SVD-LLM. The GSM8K accuracy of WizardMath-13B dropped slightly, from 58.4 to 57.6.
In summary, with the rotation , activation-aware SVD lowers the output error on the calibration activations (Table R1, 0.051 vs. 0.064). In delta compression, however, this does not translate into higher accuracy. ADE therefore keeps the standard SVD, which does not require a Cholesky factorization of the activation covariance of each module.
Revisions. We have added a statement to Section 3.2 of the revised paper that activation-aware decompositions are compatible with ADE.
[1] SVD-LLM: Truncation-aware singular value decomposition for large language model compression.
We sincerely hope that our responses have satisfactorily clarified the points you raised. If you have any further questions or concerns, please let us know. We would be grateful for the opportunity to provide any additional clarification before the rebuttal period concludes.
We sincerely thank the reviewer for the careful review and valuable comments on our latency evaluation and rank allocation, which helped improve our work. We have addressed all comments and provided detailed responses below. All changes in the revised paper are highlighted in blue.
[W1] Fairness of latency comparison
We agree that kernel implementations can affect latency. However, the speed advantage of ADE does not come from engineering effort on our side. The baselines are measured with their fastest available implementations, and the remaining gap reflects which computations each delta representation allows.
For each baseline, we measured every available application path, so no baseline is measured with a disadvantageous implementation. BitDelta’s official binary kernel takes 0.185 ms per layer at T=1. It is slower than unpacking the signs, adding them to the base weight, and running a cuBLAS GEMM (0.122 ms), so we report the latter. For DARE, we report cuSPARSE SpMM or scatter-add followed by a cuBLAS GEMM, whichever is faster at each T (0.100 vs 0.126 ms at T=1).
Two further results support this. First, in multi-model serving, BitDelta runs its released multi-tenant kernel, an optimized implementation by its authors, and ADE still has 32.2–49.3% (T=256) and 13.3–36.1% (T=4096) lower decoding latency (Section 4.5). Second, under CUDA Graphs, which remove the launch overhead of every method, ADE still has 10.9–33.2% lower latency than the fastest baseline for T≤128 (Table 8).
The possible implementations differ across methods because of their delta representations, not because of engineering. ADE stores the delta as two low-rank factors with quantization groups aligned with the multiplication direction. This allows two orders of computation. For short inputs, it multiplies the input with the factors one after the other, so the cost scales with the rank r rather than with the full matrix. For long inputs, it multiplies the two INT4 factors directly to reconstruct the delta. BitDelta must expand its sign matrix into a full FP16 matrix, and DARE’s sparse matrix requires irregular memory access. They also cannot be decomposed into smaller contractions. For reference, applying an uncompressed FP16 delta with two cuBLAS GEMMs, which involves no custom kernel, takes 0.084 ms at T=1, whereas ADE takes 0.067 ms.
Revisions. Following the reviewer’s suggestion, we have stated in Appendix K.1 that the comparison uses the released or standard implementations and listed every measured path of each baseline. The FLOP analysis of each application path in Appendix E shows which orders of computation each representation allows and what they cost. We hope this addresses the reviewer’s concern.