- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TESSA replaces LoRA for multi-adapter serving with shard-aligned updates. Each adapter fine-tunes a contiguous suffix of the base weight: the last r columns for column-parallel layers and the last r rows for row-parallel layers. Only the difference from the frozen weights is stored. Equivalently, it is LoRA with one factor fixed to a coordinate-selection matrix. Because the adapter needs only one matrix product, base and adapter computation can be merged into a single wider GEMM. Placing the suffix on the unsharded axis also removes adapter-specific tensor-parallel (TP) communication. Experiments report quality comparable to LoRA at matched parameter budgets, and up to 3.46× the throughput of vLLM’s S-LoRA-style implementation.
Critical strengths
-
The paper tackles a real systems problem. Its measurement of where LoRA overhead goes under TP (Fig. 2, App. A.3) is informative.
-
Evaluation is broad: dense and MoE models, TP1–TP8 and EP, fixed-length and trace-driven workloads. The authors discuss the bias in the learning-rate selection in App. C.5.
Critical weaknesses
The novelty of selection-based PEFT is limited. The paper acknowledges that TESSA is LoRA with one factor fixed to a coordinate selector. This makes it a special case of PaCA (Woo et al., 2025), which already computes the adapter as a single product. It is also closely related to frozen-factor LoRA [R2, R3]. S²FT [R1] already fine-tunes contiguous dense sub-blocks of the weights and claims efficient multi-adapter switching and “parallelism through reduced matrix multiplications”
The TP argument does not require contiguity either. Randomly selected columns (as in PaCA) are balanced across shards in expectation and need no collective. The requirement to use the unsharded axis arises only because the suffix is contiguous.
PaCA should be included as a serving baseline, or the paper should explain what contiguity adds beyond avoiding a gather.
[R1] X. Yang et al. S²FT: Efficient, scalable and generalizable LLM fine-tuning by structured sparsity. In Proc. NeurIPS, 2024.
[R2] L. Zhang et al. LoRA-FA: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv:2308.03303, 2023.
[R3] J. Zhu et al. Asymmetry in low-rank adapters of foundation models. In Proc. ICML, 2024.
Questions for authors
-
How does TESSA compare with vLLM’s default (not fully sharded) LoRA mode, with the better of the two modes chosen per configuration?
-
How much of the speedup remains with separate-only execution, and against a LoRA baseline whose shrink is fused into the base GEMM?
-
How do PaCA (Woo et al., 2025) and S²FT [R1] compare in serving throughput and quality? What does contiguity add?
-
The abstract can be shortened better to outline the content and contribution of this work.
9 comments
W2 + W3 + Q3. Contiguity vs. PaCA: TP argument, serving baseline, what contiguity adds
Reviewer comment:
W2, W3
The TP argument does not require contiguity either. Randomly selected columns (as in PaCA) are balanced across shards in expectation and need no collective. The requirement to use the unsharded axis arises only because the suffix is contiguous.
PaCA should be included as a serving baseline, or the paper should explain what contiguity adds beyond avoiding a gather.
Q3.
How do PaCA (Woo et al., 2025) and S²FT [4] compare in serving throughput and quality? What does contiguity add?
Response:
We thank the reviewer for these helpful comments and suggestions. We clarify why balance in expectation does not guarantee balanced adapter placement and evaluate the effects of channel selection, supporting contiguous suffixes as a practical choice rather than a requirement for communication-free execution. TESSA achieves comparable adaptation quality to PaCA and S²FT in various tasks. In our serving experiments, TESSA delivers up to 1.69× the throughput of PaCA and 2.15× that of S²FT at approximately matched parameters. We present the analyses and results below and have incorporated them into the revised manuscript (Sections 3.1, 3.3, and 4.1; Appendices C.7 and D.7).
1. Shard-aligned placement and channel selection
Random column selection must either add collectives or accept uneven adapter placement across GPUs, whereas TESSA balances adapter placement by construction without additional collectives. Expected balance does not ensure equal counts for each adapter: rebalancing can require remote activations, while keeping updates local can increase memory requirements on overloaded GPUs and limit throughput. Larger adapter footprints in MoE models [8] can amplify this memory imbalance.
The noncontiguous selection suggested by the reviewer may also avoid this imbalance if the selection is constrained by the shard boundaries so that every TP shard holds an equal number of selected coordinates. However, this constraint ties the selected channels to the TP degree used for serving. TESSA instead places the suffix on the unsharded axis, so the same adapter remains balanced at any TP degree supported by the base model without re-selecting channels. Beyond balanced placement and the removal of gathers, the contiguous suffix also benefits adaptation quality: contiguous selection achieves higher accuracy than random and evenly spaced selection at narrow widths, as shown in the next paragraph.
Effect of channel selection on adaptation quality. To examine how channel selection affects downstream quality, we compared two contiguous selections (prefix and suffix) against evenly spaced and random selections and LoRA at matched parameter budgets. In these math experiments, contiguous suffix selection methods generally achieve higher accuracy than others at narrow widths and perform comparably across the tested widths, suggesting that selecting an aligned, contiguous block may matter more than its trailing position in these experiments. This trend is consistent with the findings of RAPA [7] on the benefits of cross-layer index alignment.
Figure R4.A. Channel-selection ablation on Qwen3-8B-Base for GSM8K and MATH-500.
2. Serving throughput and adaptation quality
We evaluated PaCA and S²FT baselines in the same vLLM fork as TESSA on the Figure 5 workloads at approximately matched adapter parameter budgets.
Serving comparison with PaCA.
Experimental setup. For PaCA [1], which does not specify a multi-adapter serving strategy, we implemented it using TESSA’s kernels. We additionally evaluated PaCA + unified GEMM + dispatch as a control to measure the benefit of applying these TESSA execution optimizations to PaCA; they are components of TESSA, not of the original PaCA method.
Results. Across five matched adapter budgets (Figure R4.B), TESSA achieves 1.48–1.69× the throughput of PaCA on decode-heavy workloads and 1.01–1.25× on prefill-heavy workloads. Against this control variant, TESSA achieves 1.03–1.09× decode-heavy throughput at TP1 and 1.19–1.32× at TP4/TP8, with comparable prefill-heavy throughput.
W2 + W3 + Q3. Contiguity vs. PaCA: TP argument, serving baseline, what contiguity adds (continued)
Response (continued):
Serving comparison with S²FT.
Experimental setup. For S²FT, we evaluated S²FT-official, which ports the adapter operations from the publicly released implementation [9], and S²FT-tuned, which uses tuned vLLM Punica kernels. Following S²FT [4], both baselines and TESSA (down-only) adapt only down_proj at approximately matched parameter budgets.
Results. TESSA (down-only) achieves 1.21–2.15× the throughput of S²FT-tuned on decode-heavy workloads and 1.02–1.18× on prefill-heavy workloads; the corresponding gains over S²FT-official are 1.41–2.17× and 1.22–1.47× (Figure R4.C). Figure R4.C also includes full TESSA, with approximately the same total budget distributed across the attention and MLP projections used in Figure 5.
Where these gains come from. TESSA’s shard-aligned placement lets each GPU compute adapter updates locally without additional adapter-specific communication, explaining TESSA’s advantage under tensor parallelism. Unified GEMM execution reduces kernel-launch overhead and improves GPU utilization, so its benefit is most pronounced on decode-heavy workloads with small token batches. On prefill-heavy workloads, larger token batches amortize fixed communication and launch overheads, and redundant computation can diminish unified GEMM's advantage, so the dispatcher prefers separate execution when it is faster.
Adaptation quality comparison with PaCA and S²FT. Table R4.B compares LoRA, TESSA, PaCA, and S²FT on Qwen3-8B-Base and Llama-3.1-8B at approximately matched adapter budgets, with S²FT restricted to down projections following its GQA configuration [4]. TESSA achieves broadly comparable quality to two baselines, with relative performance varying by model and task, while providing the serving benefits above. Detailed settings are provided in Appendix C.1 of the revised manuscript.
Table R4.B. Downstream task accuracy (%) of LoRA, TESSA, PaCA, and S²FT on Qwen3-8B-Base and Llama-3.1-8B.
| Method | Commonsense | GSM8K | MATH-500 | HumanEval | MBPP | Average |
|---|---|---|---|---|---|---|
| Qwen3-8B-Base | ||||||
| LoRA | 88.60 | 90.40 | 83.87 | 86.18 | 82.36 | 86.28 |
| TESSA | 88.84 | 94.44 | 86.53 | 86.59 | 82.98 | 87.88 |
| S²FT (down-only) | 89.53 | 94.41 | 88.47 | 86.18 | 82.63 | 88.24 |
| PaCA | 89.52 | 94.11 | 87.67 | 86.38 | 83.25 | 88.19 |
| Llama-3.1-8B | ||||||
| LoRA | 83.53 | 77.53 | 36.40 | 51.63 | 67.11 | 63.24 |
| TESSA | 87.00 | 79.96 | 37.93 | 53.25 | 67.02 | 65.03 |
| S²FT (down-only) | 87.59 | 80.74 | 39.20 | 48.78 | 67.55 | 64.77 |
| PaCA | 87.42 | 80.01 | 38.93 | 51.83 | 67.46 | 65.13 |
W1. Limited novelty of selection-based PEFT (PaCA / LoRA-FA / Asymmetry / S²FT)
Reviewer comment:
Response:
We appreciate the reviewer's constructive feedback. Below, we clarify what TESSA builds upon and what it adds relative to existing partial-weight adaptation methods, and we have revised the abstract, introduction (Section 1), and Section 3 accordingly.
The main contribution of TESSA is to redesign how partial weight updates are laid out and executed for efficient multi-adapter serving under tensor parallelism. Prior selection-based and frozen-factor PEFT methods, including PaCA [1], LoRA-FA [2], and Asymmetry [3], primarily focus on training efficiency or adaptation quality rather than multi-adapter serving. Although S²FT [4] also explores adapter switching and parallel execution, efficient multi-adapter serving under tensor parallelism remains less explored. TESSA builds on these ideas to address this gap, with two key distinctions:
Shard-aligned placement avoids additional adapter-specific communication. TESSA selects weights along the unsharded axis, allowing each GPU to compute its adapter updates locally with balanced placement. Unlike PaCA’s input-column selection, this requires selecting output rows in row-parallel layers; TESSA is therefore not simply a contiguous-column special case of PaCA.
Unified GEMM and workload-aware dispatch improve concurrent multi-adapter execution. S²FT’s merge–unmerge switching can limit batching efficiency across different adapters [5, 6], while its separate base and adapter computation can underutilize the GPU during decoding. TESSA combines unified GEMM with workload-aware dispatch to improve utilization and select the faster execution path for each workload. In our experiments, TESSA achieves up to 1.69× the throughput of PaCA and 2.15× that of optimized S²FT (Figures R4.B and R4.C, respectively).