acceptodds
Under review as a conference paper at ICLR 2027

ADE: Accurate, Inference-efficient, and Tunable Delta Compression for Task-specific Fine-tuned Foundation Models

Abstract

Given multiple task-specific fine-tuned models with the same base, how can we deploy them compactly while preserving each model's performance? Delta compression stores only the difference between a fine-tuned model and its base, enabling compact deployment of many models from a shared base. However, existing methods largely treat the delta as an isolated artifact: they ignore the base model, disregard inference computation, and produce a static result that cannot be further adapted. We propose ADE (Accurate Delta compression with Efficient inference), an accurate, inference-efficient, and tunable delta compression framework with (1) base-awareness, allocating parameters by the sensitivity of each module's output, including the base weights, for accuracy, (2) computation-awareness, aligning quantization with the direction of matrix multiplication for fast low-bit inference, and (3) task-awareness, supporting near-zero-overhead tuning on the compressed delta for low-cost adaptation. Under the same memory budget, ADE is up to 3.9% relatively more accurate than the best prior method, and has up to 18.2% lower inference latency than the fastest prior method.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Delta Compression, Model Compression, LLM Efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nFYBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ADE for delta compression, which stores only the weight difference between each fine-tuned model and the shared base model. ADE compresses each delta with SVD and 4-bit quantization, and adapts it to new tasks by training two scaling vectors per layer. Experiments on six fine-tuned models show higher accuracy than prior methods and lower latency for short inputs.

Critical strengths
  1. The paper considers accuracy, inference speed, and adaptability to new tasks together for serving many fine-tuned models.
  2. Even without extra training, ADE−L still matches or outperforms previous delta compression methods on all six models (Table 1).
Critical weaknesses
  1. The latency comparison may be unfair. ADE runs on custom CUDA kernels optimized by the authors, while the baselines run on their released code or standard PyTorch (Appendix K), so part of the speedup may come from engineering rather than the method.
  2. ADE uses real data to decide how much to compress each layer, by measuring how much the layer’s output changes (Eq. 4). However, it decides which parts to keep only by their size in the weight matrix via SVD. So, the kept parts may not be the ones that matter most on real data.
Questions for authors

Please refer to the weakness part.

3 comments

We sincerely thank the reviewer for the careful review and valuable comments on our latency evaluation and rank allocation, which helped improve our work. We have addressed all comments and provided detailed responses below. All changes in the revised paper are highlighted in blue.


[W1] Fairness of latency comparison

We agree that kernel implementations can affect latency. However, the speed advantage of ADE does not come from engineering effort on our side. The baselines are measured with their fastest available implementations, and the remaining gap reflects which computations each delta representation allows.

For each baseline, we measured every available application path, so no baseline is measured with a disadvantageous implementation. BitDelta’s official binary kernel takes 0.185 ms per layer at T=1T = 1. It is slower than unpacking the signs, adding them to the base weight, and running a cuBLAS GEMM (0.122 ms), so we report the latter. For DARE, we report cuSPARSE SpMM or scatter-add followed by a cuBLAS GEMM, whichever is faster at each TT (0.100 vs 0.126 ms at T=1T = 1).
Two further results support this. First, in multi-model serving, BitDelta runs its released multi-tenant kernel, an optimized implementation by its authors, and ADE still has 32.2–49.3% (T=256T = 256) and 13.3–36.1% (T=4096T = 4096) lower decoding latency (Section 4.5). Second, under CUDA Graphs, which remove the launch overhead of every method, ADE still has 10.9–33.2% lower latency than the fastest baseline for T≤128T \le 128 (Table 8).

The possible implementations differ across methods because of their delta representations, not because of engineering. ADE stores the delta as two low-rank factors with quantization groups aligned with the multiplication direction. This allows two orders of computation. For short inputs, it multiplies the input with the factors one after the other, so the cost scales with the rank rr rather than with the full matrix. For long inputs, it multiplies the two INT4 factors directly to reconstruct the delta. BitDelta must expand its sign matrix into a full FP16 matrix, and DARE’s sparse matrix requires irregular memory access. They also cannot be decomposed into smaller contractions. For reference, applying an uncompressed FP16 delta with two cuBLAS GEMMs, which involves no custom kernel, takes 0.084 ms at T=1T = 1, whereas ADE takes 0.067 ms.

Revisions. Following the reviewer’s suggestion, we have stated in Appendix K.1 that the comparison uses the released or standard implementations and listed every measured path of each baseline. The FLOP analysis of each application path in Appendix E shows which orders of computation each representation allows and what they cost. We hope this addresses the reviewer’s concern.

[W2] Data-agnostic selection of retained components

We agree with the reviewer. ADE uses calibration activations to allocate ranks across modules (Eq. 4), whereas the standard SVD selects the components within a module by their magnitude in weight space. We tested activation-aware selection in ADE. In our setting, however, it did not improve accuracy and even lowered it slightly.

Activation-aware SVD widens the value range of the factors. We replaced the standard SVD with SVD-LLM [1]. SVD-LLM multiplies the delta by the Cholesky factor S\mathbf{S} of the activation covariance before the SVD, and maps the right factor back by S−⊤\mathbf{S}^{-\top}. This transformation widens the value range of the factors: the 99th percentile of the group range/RMS of UΣ1/2\mathbf{U}\boldsymbol{\Sigma}^{1/2} grows from 6.8 to 9.1 (Table R1, without rotation). The range/RMS of a quantization group is the difference between its maximum and minimum values divided by the RMS of its values, and a larger ratio means coarser quantization; its 99th percentile is the value above which the 1% of groups with the largest ratio lie. Under the same bit-width and group size, the wider range increases the quantization error. Because each quantization group mixes large and small values, the relative quantization error of the factor UΣ1/2\mathbf{U}\boldsymbol{\Sigma}^{1/2} rises from 0.104 to 0.124. Quantization therefore cancels the advantage that SVD-LLM had in output space.

Table R1. Quantization of the factors (WizardMath-13B, all 280 modules at the ranks of ADE$_{-\mathcal{L}}$, 4-bit, G=128G = 128).

StatisticSVD, no rotationSVD-LLM, no rotationSVD, rotation H\mathbf{H} (ADE)SVD-LLM, rotation H\mathbf{H} (ADE)
Group range/RMS of UΣ1/2\mathbf{U}\boldsymbol{\Sigma}^{1/2} (99th percentile)6.89.16.56.5
Relative quantization error of UΣ1/2\mathbf{U}\boldsymbol{\Sigma}^{1/2}0.1040.1240.1000.099
∥(ΔWr−Q(ΔWr))X∥/∥ΔWX∥\lVert(\Delta \mathbf{W}_r - Q(\Delta \mathbf{W}_r))\mathbf{X}\rVert / \lVert \Delta \mathbf{W} \mathbf{X} \rVert0.0840.0840.0640.051

ADE's rotation reduces this error, but accuracy does not improve. ADE quantizes the factors after applying the rotation H\mathbf{H}. This largely removes the penalty above: the group range/RMS and the relative quantization error of the two decompositions become equal (Table R1).

An ablation confirmed that activation-aware SVD does not help. We replaced only the SVD of ADE$_{-\mathcal{L}}$ in Table 2 of the paper with SVD-LLM. The GSM8K accuracy of WizardMath-13B dropped slightly, from 58.4 to 57.6.

In summary, with the rotation H\mathbf{H}, activation-aware SVD lowers the output error on the calibration activations (Table R1, 0.051 vs. 0.064). In delta compression, however, this does not translate into higher accuracy. ADE therefore keeps the standard SVD, which does not require a Cholesky factorization of the activation covariance of each module.

Revisions. We have added a statement to Section 3.2 of the revised paper that activation-aware decompositions are compatible with ADE.

[1] SVD-LLM: Truncation-aware singular value decomposition for large language model compression.

We sincerely hope that our responses have satisfactorily clarified the points you raised. If you have any further questions or concerns, please let us know. We would be grateful for the opportunity to provide any additional clarification before the rebuttal period concludes.

Reviewer_w7WqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the compression of weight deltas between task-specific fine-tuned models and a shared backbone. ADE combines three designs: (1) allocating SVD ranks using normalized module-output reconstruction error, (2) quantizing both factors along their shared rank dimension with paired rotations, (3) adapting the compressed delta through row/column-scaling vectors that are folded into existing quantization scales. Experiments cover mathematical reasoning, code generation, and multimodal tasks at two compression budgets.

Critical strengths
  1. I appreciate that the three designs work together. The quantization layout supports low-bit reconstruction while also allowing the learned scaling vectors to be absorbed into existing scales. The fold-in is exact, because gamma and rho line up with the row and column groups, and it leaves the inference path untouched.

  2. Both application paths are actually implemented with kernels and timed. Also, the problem matters in practice and the paper reads well.

Critical weaknesses
  1. The main ingredients have clear precedents. Delta-CoMe[1] already combines and quantization, rotations are an established approach to reducing quantization error, and PEQA[2] studies task adaptation by updating scales while freezing quantized weights. I see ADE’s contribution primarily in how these ideas are made compatible within a particular delta representation. This is potentially useful, but the distinction should be stated more carefully.

[1] Delta-CoMe: Training-free delta-compression with mixed-precision for large language models.

[2] Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization.

  1. The main tuned accuracy comparison mixes methodological and training differences. Section 4.1 states that BitDelta uses a general calibration corpus, whereas ADE uses domain-specific instruction data. Could you please pro

  2. The abstract and introduction currently characterize prior work too broadly. BitDelta[3] already calibrates scales using full-model outputs and studies multi-tenant decoding, while DeltaZip[4] explicitly co-designs delta compression and serving. It is therefore inaccurate to describe previous methods generally as ignoring the base model or inference computation. ADE could instead emphasize its particular normalization, rank-allocation rule, and factor layout.

[3] BitDelta: Your Fine-Tune May Only Be Worth One Bit

[4] DeltaZip: Efficient Serving of Multiple Full-Model-Tuned LLMs

Questions for authors
  1. Does the advantage over BitDelta persist with matched data and comparable training budgets?

  2. As for kernel and time test, which baseline kernels were used, and does the speedup hold for cached decoding and shared-backbone serving with multiple deltas?

7 comments

We sincerely thank the reviewer for the thoughtful review and for the insightful comments on prior work and experimental comparisons, which are very helpful for clarifying our contributions. We have carefully addressed all of the comments and provide detailed responses to each point below. All changes in the revised paper are highlighted in blue.


[W1] Novelty over prior techniques

We thank the reviewer for recognizing the value of combining these ideas within a single delta representation. Beyond making them compatible, ADE addresses the limitations that each idea has when applied directly to delta compression.

Quantization: Prior delta quantization does not consider the multiplication direction and wastes FLOPs; ADE aligns its groups with the multiplication axis to multiply directly in INT4. Delta-CoMe [1] groups both factors row-wise for storage, so it dequantizes them to FP16 before multiplication, and it allocates bits by singular-value magnitude. ADE instead allocates capacity by output sensitivity, including the base model. Appendix E.2 details the FLOPs difference between the two layouts.

Rotation: Existing rotation methods incur online rotations at inference; ADE’s rotation adds no overhead. Rotation methods such as QuaRot [2] apply rotations online during inference. ADE multiplies the two SVD factors by an orthogonal rotation matrix H\mathbf{H} and its transpose H⊤\mathbf{H}^\top along the rank axis, so the two rotations cancel within the product (Eq. 8).

Tuning: Existing methods give limited consideration to tuning efficiency on compressed weights; ADE controls many scales with few parameters and adds no deployment memory. PEQA [3] trains every quantization scale directly, and the LoRA [4] adds memory at deployment. In contrast, each entry of ADE’s tuning vectors adjusts all group scales in its row or column, and folds into them after training (Eq. 10). ADE also outperforms rank-1 LoRA with the same number of trainable parameters (Table 3).

Revisions. We have clarified these distinctions in the Related Work (Section 2.1.2) of the revised paper. We have also rewritten the second paragraph of the Introduction to state that no prior method addresses these three aspects jointly.

[1] Delta-CoMe: Training-free delta-compression with mixed-precision for large language models.
[2] QuaRot: Outlier-free 4-bit inference in rotated LLMs.
[3] Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization.
[4] LoRA: Low-rank adaptation of large language models.


[W2] Confounded training data in accuracy comparison

Please refer to our response to Q1, where we compare ADE with BitDelta trained on the same in-domain data.


[W3] Overly broad characterization of prior work

We agree that categorical terms such as "ignore" and "disregard" do not fully reflect prior work, since BitDelta and DeltaZip do consider the base model and serving efficiency, respectively. Our intended claim is that no prior method jointly addresses base-aware capacity allocation, a layout aligned with the computation, and tunability of the compressed delta.

We have revised the paper as follows (changes highlighted in blue):

  • Abstract and Introduction. We have softened the categorical statements by qualifying them with "most existing methods," so that they no longer assert that all prior methods ignore the base model or inference computation.
  • Related Works (Section 2). We have added that BitDelta calibrates its binarization scales through output distillation and serves multiple deltas with a dedicated multi-tenant kernel, and that DeltaZip compresses deltas into a format designed for multi-model serving, acknowledging the aspects in which these methods do account for the base model and serving. Finally, we have added a paragraph stating how ADE differs from prior methods: its normalization of the output error by the combined base–delta output, its sensitivity-proportional rank allocation, and its factor layout aligned with the multiplication direction.
  • Experiments (Section 4.5 and Appendix K.3). Section 4.5 now compares ADE against BitDelta's released multi-tenant kernel: ADE has 13–49% lower decoding latency when serving up to 16 models.

We hope these clarifications characterize prior work more fairly while better highlighting ADE's distinct contributions.

[Q1] BitDelta under matched data and training budget

ADE outperforms BitDelta under matched data, and even the untrained ADE$_{-\mathcal{L}}$ outperforms the trained BitDelta.

Matched data: BitDelta and ADE already share the same teacher, the uncompressed aligned model, and differ only in training data. We retrained BitDelta on the same in-domain data as ADE:

Compression ratio 1/16

ModelBitDelta (C4)BitDelta (in-domain)ADE$_{-\mathcal{L}}$ADE
WizardMath-7B51.151.151.553.4
WizardMath-13B58.457.958.458.8
Magicoder-7B68.568.869.370.4
WizardCoder-13B66.968.070.671.2
LLaVA-7B56.959.759.759.9
LLaVA-13B60.362.762.963.0

Compression ratio 1/32

ModelBitDelta (C4)BitDelta (in-domain)ADE$_{-\mathcal{L}}$ADE
WizardMath-7B35.041.648.650.6
WizardMath-13B50.651.957.258.0
Magicoder-7B68.367.769.069.3
WizardCoder-13B64.865.370.170.4
LLaVA-7B56.456.759.059.2
LLaVA-13B60.061.862.862.9

In-domain data does not consistently improve BitDelta, and ADE outperforms it. This indicates that ADE’s gain comes from its structure rather than from the training data.

Training budget: BitDelta trains one scale per weight matrix, whereas ADE trains (di+do)(d_i + d_o) parameters, and this gap cannot be closed structurally. However, ADE$_{-\mathcal{L}}$, which uses no training, outperforms BitDelta trained on matched data in all settings. Moreover, the deployment cost is identical: ADE folds its tuning vectors into the existing quantization scales, so neither method adds parameters at deployment.

Revisions. We have replaced the BitDelta results in Tables 1 and 6 with those distilled on the same in-domain data as ADE, and stated this in Section 4.1.

[Q2] Baseline kernels and multi-delta serving scenarios

We thank the reviewer for the practical question. We answer each points below.

Each baseline is measured with the fastest of its available implementations. For each baseline, we measured every available application path, so no baseline is measured with a disadvantageous implementation. For BitDelta, we measured its official binary kernel (0.185 ms per layer at T=1T = 1). It is slower than unpacking the signs, adding them to the base weight, and running a cuBLAS GEMM (0.122 ms), so we report the latter. For DARE, we measured cuSPARSE SpMM and scatter-add followed by a cuBLAS GEMM, and report the faster one at each TT (0.100 vs 0.126 ms at T=1T = 1). For Delta-CoMe and ImPart, we measured both factored application and reconstruction. Delta-CoMe's factored application requires keeping its factors dequantized to FP16, which takes much more memory than the compressed delta, so we report its released forward, the fastest path that keeps the delta compressed. ImPart applies its sparse factors directly or reconstructs the delta, whichever is faster. Appendix K.1 lists every path.

A KV cache leaves the cost of applying the delta unchanged. With a KV cache, each decoding step processes one new token per request. Our measurement at T=1T = 1 is therefore the cost of the linear layers in one decoding step. The KV cache affects only the attention computation, which is identical for all methods because they all share the base model’s attention and modify only the linear layers. The cost of applying the delta is thus unchanged, and so is the absolute latency gap between methods.

ADE serves multiple deltas on a shared base by choosing the application path for each model rather than for the whole batch. The tokens of a batch are grouped by model, and each group applies its own delta. The tokens of each model use whichever application is faster, selected by measurement, and all groups share WbX\mathbf{W}_b \mathbf{X} computed once for the batch. In particular, a long prefill is a single prompt and thus forms a single-model group by itself, so it can be reconstructed even within a mixed batch.

Multi-model serving with a KV cache (Section 4.5, Appendix K.3). We serve N∈{1,2,4,8,16}N \in \lbrace 1, 2, 4, 8, 16 \rbrace fine-tuned models that share Llama-2-7B on one A100 GPU, split a batch of 32 requests uniformly over them, and keep every delta resident in compressed form. We measure one decoding step in which each request attends to a KV cache of T∈{256,4096}T \in \lbrace 256, 4096 \rbrace tokens, and the prefill of 32 prompts of TT tokens. ADE has the lowest time per output token (TPOT) among the delta compression methods at every NN and TT: 32.2–49.3% (T=256T = 256) and 13.3–36.1% (T=4096T = 4096) lower than BitDelta with its released multi-tenant kernel. For 4096-token prompts, the time to first token (TTFT) of ADE is within 4.4% of the base model, whereas for 256-token prompts BitDelta prefills faster, with ADE taking 5.3–37.6% longer. A request that generates MM tokens is therefore faster with ADE than with BitDelta once M≥9M \ge 9 for every NN and TT (115.8 vs 155.7 s for M=1000M = 1000 at N=8N = 8 and T=4096T = 4096).

Revisions. Following the reviewer’s suggestion, we have listed the measured paths of each baseline in Appendix K.1, stated that short inputs include every decoding step, and reported the multi-model experiment with a KV cache in Section 4.5 and Appendix K.3. We hope this addresses the reviewer’s questions.

Reviewer_zeNXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ADE, a delta compression method for serving multiple fine-tuned models from one base. Each delta is truncated by SVD and quantized to 4 bits. The method has three parts. (1) The retained rank of each linear module is allocated in proportion to a sensitivity score, which is the truncation error of the delta measured on calibration activations and normalized by the output energy of the full fine-tuned module. (2) The singular values are split evenly into the two SVD factors, both factors are rotated by a shared orthogonal matrix along the rank axis, and the factors are group-quantized along the rank axis. The rotations cancel in the product, so the delta can be reconstructed with an INT4 Tensor Core contraction and r/G rescalings per entry. (3) Two per-module vectors that rescale the rows and columns of the dequantized delta are trained and then folded into the group scales, so adaptation adds no parameters at inference. Experiments on six fine-tuned models (math, code, multimodal) at alpha = 1/16 and 1/32 compare against BitDelta, Delta-CoMe, DARE, and ImPart. The untuned variant ADE-L is 0.0 to 1.9% relatively above the best baseline, and the tuned variant is up to 3.9% above. Latency on an A100 at batch size 1 is lower than all baselines for T <= 128 and higher than BitDelta and DARE for T >= 2048. An adaptation experiment compares the fold-in vectors with rank-1 LoRA and DoRA on new tasks.

Critical strengths

S1. The quantization layout is the part I find useful. The paper groups both SVD factors along the rank axis, row-wise on the U side and column-wise on the V side, with the singular values split evenly between them. Because the group boundaries line up on the contraction axis, the two INT4 codes can be multiplied directly and the scale is applied once per group after accumulation (Eq. 7). This cuts the floating-point rescalings per output entry from r to r/G. Prior SVD-based delta methods quantize for storage and dequantize to FP16 before the product, so they save memory but not compute. Appendix E.2 reports 36.1% lower latency per layer at T = 2048 against the direction-unaware layout. This is a concrete kernel-level gain, not a storage gain.

S2. The rotation and the tuning vectors both come for free at inference. H is applied to the U side and H^T to the V side, so they cancel in the product and nothing is computed online, unlike QuaRot or SpinQuant where some rotations stay in the forward pass. The row and column tuning vectors fold exactly into the existing group scales (Eq. 10), so the adapted delta has the same size and the same low-bit path as before training. For a serving system this matters more than the accuracy numbers.

Critical weaknesses

W1. I may be missing something, but I could not find a gain in the setting the paper is motivated by. The introduction describes serving many fine-tuned models from one base. In that case a batch would usually contain requests for different models, so the delta cannot be reconstructed into a single FP16 weight and only the factored path seems applicable. As far as I can tell, Table 4 does not cover this case. It reports batch size 1, with the factored path for T <= 128 and the reconstructed path for larger T. The reconstructed path appears to materialize a full FP16 Wb + delta per layer, which is roughly the memory the method aims to save, and it would only help when all tokens in the batch belong to the same model. For T >= 2048 ADE is also slower than BitDelta and DARE (239.5 vs 207.4 ms end-to-end). Table 4 has no base-only or FP16 fine-tuned row either, so I was not able to estimate the absolute overhead of applying the delta. It is possible the authors have these numbers and left them out for space. If so, I would appreciate seeing them, because at the moment I cannot tell whether ADE would be faster in a deployment I would actually run.

W2. The latency comparison seems to be between implementations rather than methods, though I may be reading Appendix K too strictly. ADE uses custom CUDA kernels with INT4 Tensor Core contraction, while the baselines use released code, PyTorch, or cuSPARSE. A 6.25% dense matrix on cuSPARSE is probably not a strong implementation of DARE. All runs are in eager mode without CUDA graphs, so at T = 1 the per-layer gap (67 us vs 122 us) may reflect kernel launch count more than arithmetic. The 18.2% figure in the abstract corresponds to the end-to-end number at T = 128 only. I would feel more comfortable with the speed claims if the baselines were measured with comparable kernels, or if the paper stated more plainly that it compares released implementations.

W3. The accuracy results do not seem large enough to compensate. Without training, ADE-L is 0.0 to 1.9% above the best training-free baseline and ties BitDelta on WizardMath-13B. The 3.9% headline appears to come from training the fold-in vectors on in-domain instruction data with the uncompressed model as the teacher, which the training-free baselines do not receive. All results are single seed, and many margins in Tables 2 and 3 are below one point, so I am not sure how much of the difference is noise. My overall impression is that the accuracy gain depends on extra training and the speed gain depends on the measurement setup. I would be glad to be corrected on either point.

Questions for authors

Q1. Please add base-only and FP16 fine-tuned latency to Table 4 under the same protocol. Q2. In Table 2, why do gamma-only and rho-only fall below the untuned ADE-L on both models (e.g., 69.8 and 69.6 vs 70.6 on WizardCoder-13B)? Q3. For LLaVA-7B at alpha = 1/32, ADE-L is 59.0 in Table 7 and 56.1 in the CHAT column of Table 3. What is CHAT, and why do these differ?

6 comments

We sincerely thank the reviewer for carefully reading our paper and for constructive comments, which are very helpful in improving our work. We have addressed all comments and provided detailed responses to each point below. All changes in the revised paper are highlighted in blue.


[W1] Practicality in multi-model serving

We thank the reviewer for raising the multi-model serving scenario, which is the setting that motivates our proposed ADE. We agree that a batch mixing requests for different models cannot be served by reconstructing a single weight. However, the reconstructed path neither stores a full FP16 model nor requires the whole batch to come from one model. We clarify both points below and report the requested latency of the base and fine-tuned models.

The reconstructed path keeps at most one layer-sized weight in memory, not a full FP16 model. The memory that ADE saves is the persistent storage of NN fine-tuned models, which drops from NFNF to (1+Nα)F(1 + N\alpha)F for a model of size FF and compression ratio α\alpha (Section 2). The reconstructed path does not keep Wb+ΔW^\mathbf{W}_b + \Delta\hat{\mathbf{W}} in memory (Wb\mathbf{W}_b: base weight, ΔW^\Delta\hat{\mathbf{W}}: compressed delta). For each linear layer, it writes the reconstructed weight into a temporary buffer of that layer’s size and multiplies it with the input. It then frees the buffer, and the next layer reuses the same memory. At most one layer-sized buffer exists at a time, 86 MiB for the largest projection of a 7B model. Accordingly, the peak memory of ADE when decoding for 16 models is 32.3 GiB in the multi-model experiment of the revised paper (Appendix K.3, Table 12), on par with BitDelta (32.4 GiB), whereas eight uncompressed FP16 deltas already exceed the 80 GB GPU.

A mixed-model batch can use both application paths. ADE chooses between reconstructed and factored application for each model, not for the whole batch. The tokens of a batch are grouped by model, and each group applies its own delta. The tokens of each model use whichever application is faster, selected by measurement, and all groups share WbX\mathbf{W}_b \mathbf{X} computed once for the batch. In particular, a long prefill is a single prompt and thus forms a single-model group by itself, so it can be reconstructed even within a mixed batch.

ADE adds the least latency for short inputs and a small, amortized cost for long inputs. The base-only and FP16 fine-tuned models have identical shapes, so both run at the dense latency. As for every method in Table 7, we report the faster of the factored and reconstructed paths at each TT for the uncompressed FP16 delta. Under the protocol of Table 7 in Appendix K of the revised paper (formerly Table 4; 7B model, A100, batch size 1), the end-to-end latency in milliseconds is:

MethodT=1T=1T=32T=32T=128T=128T=2048T=2048T=8192T=8192
Base-only (Wb\mathbf{W}_b) = FP16 fine-tuned (Wa\mathbf{W}_a)29.129.630.0188.61039.8
Uncompressed FP16 delta (Wb+ΔW\mathbf{W}_b + \Delta \mathbf{W})36.237.041.2214.11064.1
Fastest baseline41.141.846.2207.41059.3
ADE35.836.237.8239.51091.5

For long single-model inputs, ADE is 15.5% slower than BitDelta at T=2048T = 2048 and 3.0% at T=8192T = 8192, since multiplying the two factors is a fixed cost per layer that is amortized as TT grows (Appendix K.2). This cost applies only to prefill. In multi-model serving, the TTFT of ADE for 4096-token prompts is within 4.4% of the base model, and its lower TPOT outweighs the prefill gap after nine generated tokens.

ADE has the lowest decoding latency when serving multiple fine-tuned models. We serve N∈{1,2,4,8,16}N \in \lbrace 1, 2, 4, 8, 16 \rbrace fine-tuned models sharing Llama-2-7B on one A100 GPU, split a batch of 32 requests uniformly over them, and keep every delta resident in compressed form (Section 4.5, Figure 4, Appendix K.3). ADE has the lowest time per output token (TPOT) among the delta compression methods at every NN and context length T∈{256,4096}T \in \lbrace 256, 4096 \rbrace: 32.2–49.3% (T=256T = 256) and 13.3–36.1% (T=4096T = 4096) lower than BitDelta with its released multi-tenant kernel. For 4096-token prompts, the time to first token (TTFT) of ADE is within 4.4% of the base model, whereas for 256-token prompts BitDelta prefills faster, with ADE taking 5.3–37.6% longer. A request that generates MM tokens is therefore faster with ADE than with BitDelta once M≥9M \ge 9 for every NN and TT (115.8 vs 155.7 s for M=1000M = 1000 at N=8N = 8 and T=4096T = 4096).

Revisions. Following the reviewer’s suggestion, we have added the rows of the base model alone, whose latency equals that of an FP16 fine-tuned model of the same shape, and of the uncompressed FP16 delta to Table 7 in Appendix K.2 (formerly Table 4) and reported the multi-model serving experiment in Section 4.5 and Appendix K.3. We hope this addresses the reviewer’s concern.

[W2] Fairness of latency comparison

We agree that Table 4 (Table 7 in the revised paper) compares implementations, and that kernel quality and launch overhead can affect latency in eager mode. We clarify below how we chose each baseline’s implementation and that ADE’s advantage does not come from launch count. We have also replaced the single 18.2% value with the reduction at each length and with the decoding latency in multi-model serving.

We reported the best implementation for each method. For each baseline, we measured different available application paths and reported the faster one, so no baseline is measured with a disadvantageous implementation. BitDelta’s official binary kernel takes 0.185 ms per layer at T=1T = 1, which is slower than unpacking the signs, adding them to the base, and running a cuBLAS GEMM (0.122 ms), so we report the latter. For DARE, we report cuSPARSE SpMM or scatter-add followed by a cuBLAS GEMM, whichever is faster at each TT (0.100 vs 0.126 ms at T=1T = 1). For ImPart, we likewise report the faster of factored application and reconstruction at each TT. The reconstruction path of DARE scatters its retained entries into a dense buffer and runs a cuBLAS GEMM, so it does not depend on the quality of a sparse kernel; we report it whenever it is faster than cuSPARSE SpMM. Under CUDA Graphs, DARE is the closest baseline at T=1T = 1, and ADE still has 10.9% lower latency (Table 8). The kernels also differ because the representations differ: the sign matrix of BitDelta and the sparse matrix of DARE must be applied as a full do×did_o \times d_i matrix, whereas the two low-rank factors of ADE can be applied to the input one after the other.

ADE’s advantage does not come from fewer launch overhead. We counted the kernel launches per linear layer at T=1T = 1 with torch.profiler, using the same implementations as in Table 7 (formerly Table 4). BitDelta launches 2 kernels, fewer than ADE’s 3 (a base GEMM and two delta kernels), yet it is slower (0.122 vs 0.067 ms). DARE launches 6 kernels and ImPart 4. ADE’s advantage therefore does not come from fewer launches.

ADE remains the fastest under CUDA Graphs. We captured the end-to-end forward of every method as one CUDA Graph, which removes the launch overhead from all methods alike (Table 8, Appendix K). ADE remains the fastest delta compression method at T=1T = 1, 32, and 128, with 10.9%, 33.2%, and 14.4% lower latency than the fastest baseline (DARE, ImPart, and BitDelta, respectively), and it is also the fastest for each of the query, gate, and down projections.

The speedup value of 18.2% is not a selected best case. ADE is the fastest method at every short sequence length. To state this clearly, we have reported the reduction at each length in Appendix K.2, and replaced the single value in the abstract and conclusion with the time per output token in multi-model serving, which is 13–49% lower than that of the fastest prior method. Relative to the fastest baseline, ADE reduces the end-to-end latency by 12.9%, 13.4%, and 18.2%, and the per-layer latency by 33.0%, 36.7%, and 23.4%, at T=1T = 1, 32, and 128, respectively.

Revisions. Following the reviewer’s suggestion, we have stated in Appendix K that the comparison uses the released or standard implementations, listed every measured path of each baseline, and added the launch counts and the CUDA Graph results (Table 8). We have also replaced the 18.2% value in the abstract, introduction, and conclusion with the 13–49% lower decoding latency in multi-model serving. We hope this addresses the reviewer’s concern.

[W3] Source of accuracy gains

ADE’s accuracy gain does not depend on training and it is consistent across seeds: even without any training, ADE$_{-\mathcal{L}}$ outperforms all baselines, and the fold-in adaptation provides an additional gain on top of it.

Without training, ADE$_{-\mathcal{L}}$ already outperforms all baselines, including the trained BitDelta. Across the 12 settings in Tables 1 and 6 (6 models ×\times 2 compression ratios), ADE$_ {-\mathcal{L}}$ outperforms the best training-free baseline in all 12 settings. It also outperforms BitDelta, which distills from the uncompressed teacher. Moreover, the gap grows with compression: ADE$_ {-\mathcal{L}}$ improves over the best competitor by up to 1.9% at α=1/16\alpha = 1/16, and by up to 3.9% at α=1/32\alpha = 1/32. Thus, the 3.9% gain in our abstract is also achieved without any training.

The ordering in Table 1 holds over three seeds. We repeated the comparison between ADE$_ {-\mathcal{L}}$ and the best baseline on the six models at α=1/16\alpha = 1/16 (Section 4.2, Table 1) with three random seeds, which change the calibration samples. ADE$_ {-\mathcal{L}}$ keeps the ordering of the single-run results on every model. We are running the same repetition for the ablation (Section 4.3, Table 2) and the adaptation to new tasks (Section 4.4, Table 3), and will post the results during the discussion period.

ADE$_ {-\mathcal{L}}$ and the best baseline on the six models in Table 1 (comparison, α=1/16\alpha = 1/16).

MethodWizard-7BWizard-13BMagic-7BWizardCoder-13BLLaVA-7BLLaVA-13B
Best baseline51.3 ±\pm 0.08 (Delta-CoMe)58.2 ±\pm 0.08 (DARE)68.6 ±\pm 0.16 (BitDelta)69.0 ±\pm 0.40 (Delta-CoMe)59.7 ±\pm 0.13 (BitDelta)62.7 ±\pm 0.00 (BitDelta)
ADE$_{-\mathcal{L}}$51.5 ±\pm 0.0858.5 ±\pm 0.2369.1 ±\pm 0.4670.3 ±\pm 0.6159.7 ±\pm 0.0162.8 ±\pm 0.10

Revisions. Running every method and setting with multiple seeds is beyond the rebuttal period, so we will run all experiments with three seeds for the camera-ready version and report the mean and standard deviation in the main tables. We hope these results address the reviewer’s concern.


[Q1] Latency of base-only and FP16 fine-tuned models

Please refer to our response to W1, where we report the latency of the base-only and FP16 fine-tuned models.


[Q2] Single-vector ablation in Table 2

This stems from how the two vectors act on the compressed delta. γ\boldsymbol{\gamma} rescales each output channel (row), and ρ\boldsymbol{\rho} rescales each input channel (column). When both are trained, each vector corrects the error along its own direction. When one is removed, the error along the other direction is either left uncorrected or must be absorbed by the remaining vector, which can do so only coarsely. For example, with γ\boldsymbol{\gamma} alone, reducing the error of a single input channel requires rescaling the entire row, which also changes the other input channels that were already accurate. Such coarse updates can reduce the training loss while increasing the error on other channels, so a single vector can end slightly below the untuned ADE$_{-\mathcal{L}}$, whereas both vectors together improve over it on both models in Table 2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Revisions

We thank all three reviewers for their constructive feedback. We have addressed every point raised in the three reviews with additional experiments and manuscript revisions, including the two follow-up points from Reviewer w7Wq. Below, we summarize the main changes and findings.


1. Practical multi-model serving efficiency

To evaluate ADE in its intended deployment setting, we added experiments serving up to 16 fine-tuned models sharing a single base model, with KV caches and a batch of 32 requests. ADE achieves 13–49% lower time per output token (TPOT) than BitDelta's released multi-tenant kernel. We also reported TTFT, peak memory usage, and end-to-end generation latency. Although ADE incurs higher prefill latency in some settings, its decoding advantage compensates for this overhead after nine generated tokens. These results are included in Section 4.5 and Appendix K.3.

2. Fairness of latency comparisons

We clarified that each baseline was evaluated using its fastest available implementation. To examine the effect of kernel launch overhead, we measured kernel launch counts and repeated the latency experiments with CUDA Graphs. ADE remains 10.9–33.2% faster than the strongest baseline for short inputs under CUDA Graphs. We also added base-only and uncompressed FP16 delta latency measurements to make the overhead of delta application explicit.

3. Accuracy and robustness

To address concerns about differences in training data, we retrained BitDelta using the same in-domain data as ADE. Even without additional training, ADE-L matches or outperforms the trained BitDelta across all 12 settings (six models and two compression ratios). We also repeated the main comparison, ablation, and task-adaptation experiments with three random seeds. ADE-L matches or exceeds the best baseline on all six models, with improvements larger than the standard deviations of both methods on four models. The ablation and adaptation results also maintain consistent ordering across seeds.

4. Benefit of the joint design over combining existing techniques

Directly applying existing techniques to compressed deltas, as originally proposed, increases TPOT by 104% over ADE in the eight-model serving setting, whereas ADE obtains their benefits without this cost. We also compared against PEQA-style tuning, which achieves comparable accuracy but requires 1.8× more trainable parameters in the TinyLlama setting. These results help clarify the importance of ADE's joint design rather than simply combining existing techniques. In addition, we revised the Introduction and Related Work to state more precisely how ADE differs from prior work in capacity allocation, quantization layout, and adaptation.

5. Additional analyses and clarifications

In response to the concern that retained components are selected in weight space (nFYB W2), we replaced the SVD in ADE-L with SVD-LLM. With ADE's rotation, it lowers the output error on calibration activations (0.051 vs 0.064) but does not improve downstream accuracy (57.6 vs 58.4 on WizardMath-13B). We therefore keep the standard SVD. We explained why training only one of the two vectors can fall below the untuned ADE-L (zeNX Q2); over three seeds, the full ADE remains above every ablated variant.


Changes in the revised manuscript are highlighted in blue. Results posted after the revision (the three-seed tables and the comparison with directly applied existing techniques) will be added to the final version.

We appreciate the reviewers' careful consideration and the opportunity to address their concerns. We hope the revised manuscript and additional evidence help clarify both the practical benefits and the technical contributions of ADE.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.