acceptodds
Under review as a conference paper at ICLR 2027

TESSA: RETHINKING LORA FOR EFFICIENT MULTI- ADAPTER SERVING

Abstract

Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning by learning small, task-specific updates. Serving requests with different LoRA adapters on a shared base LLM requires executing adapter-specific low-rank computations during inference while reusing the base model, enabling inexpensive adapter swapping and heterogeneous-adapter batching. While these computations are theoretically lightweight, multi-LoRA serving can incur substantial practical overhead due to two sequential, GPU-unfriendly matrix multiplications and additional inter-GPU communication under tensor parallelism. Specifically, despite adding less than 1% to both computation and parameter memory, LoRA adapters can reduce serving throughput by 15.5% with the reduction growing to 42.9% as tensor parallelism scales. Existing multi-LoRA systems, such as S-LoRA and Punica, optimize adapter execution but retain the two-stage shrink–expand structure, leaving GPU-inefficient small sequential matrix products and adapter-specific inter-GPU communication as persistent bottlenecks. In this paper, we propose Tensor-parallel Shard-aligned Slice Adaptation (TESSA), a new adapter structure tailored to efficient multi-adapter serving, rather than optimizing execution around the existing LoRA structure. TESSA trains contiguous, shard-aligned subsets of the base weight matrix and represents each adapter as a weight delta over the selected subset. Contiguous partial updates enable base and adapter computations to be executed as a single wider, GPU-friendly matrix multiplication. Moreover, shard-aligned placement eliminates additional adapter-specific communication under tensor parallelism. We demonstrate that TESSA substantially improves serving efficiency across diverse workloads while maintaining adaptation quality comparable to LoRA under approximately matched adapter parameter budgets. Our vLLM implementation achieves up to 3.46 the throughput of S-LoRA at approximately matched adapter budgets.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Parameter-efficient fine-tuning, Low-Rank Adaptation, Multi-adapter serving, Efficient LLM inference, Tensor parallelism, Large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_4aviOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TESSA replaces LoRA for multi-adapter serving with shard-aligned updates. Each adapter fine-tunes a contiguous suffix of the base weight: the last r columns for column-parallel layers and the last r rows for row-parallel layers. Only the difference from the frozen weights is stored. Equivalently, it is LoRA with one factor fixed to a coordinate-selection matrix. Because the adapter needs only one matrix product, base and adapter computation can be merged into a single wider GEMM. Placing the suffix on the unsharded axis also removes adapter-specific tensor-parallel (TP) communication. Experiments report quality comparable to LoRA at matched parameter budgets, and up to 3.46× the throughput of vLLM’s S-LoRA-style implementation.

Critical strengths
  1. The paper tackles a real systems problem. Its measurement of where LoRA overhead goes under TP (Fig. 2, App. A.3) is informative.

  2. Evaluation is broad: dense and MoE models, TP1–TP8 and EP, fixed-length and trace-driven workloads. The authors discuss the bias in the learning-rate selection in App. C.5.

Critical weaknesses

The novelty of selection-based PEFT is limited. The paper acknowledges that TESSA is LoRA with one factor fixed to a coordinate selector. This makes it a special case of PaCA (Woo et al., 2025), which already computes the adapter as a single product. It is also closely related to frozen-factor LoRA [R2, R3]. S²FT [R1] already fine-tunes contiguous dense sub-blocks of the weights and claims efficient multi-adapter switching and “parallelism through reduced matrix multiplications”

The TP argument does not require contiguity either. Randomly selected columns (as in PaCA) are balanced across shards in expectation and need no collective. The requirement to use the unsharded axis arises only because the suffix is contiguous.

PaCA should be included as a serving baseline, or the paper should explain what contiguity adds beyond avoiding a gather.

[R1] X. Yang et al. S²FT: Efficient, scalable and generalizable LLM fine-tuning by structured sparsity. In Proc. NeurIPS, 2024.

[R2] L. Zhang et al. LoRA-FA: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv:2308.03303, 2023.

[R3] J. Zhu et al. Asymmetry in low-rank adapters of foundation models. In Proc. ICML, 2024.

Questions for authors
  1. How does TESSA compare with vLLM’s default (not fully sharded) LoRA mode, with the better of the two modes chosen per configuration?

  2. How much of the speedup remains with separate-only execution, and against a LoRA baseline whose shrink is fused into the base GEMM?

  3. How do PaCA (Woo et al., 2025) and S²FT [R1] compare in serving throughput and quality? What does contiguity add?

  4. The abstract can be shortened better to outline the content and contribution of this work.

9 comments

W1. Limited novelty of selection-based PEFT (PaCA / LoRA-FA / Asymmetry / S²FT)

Reviewer comment:

The novelty of selection-based PEFT is limited. The paper acknowledges that TESSA is LoRA with one factor fixed to a coordinate selector. This makes it a special case of PaCA (Woo et al., 2025), which already computes the adapter as a single product. It is also closely related to frozen-factor LoRA [2, 3]. S²FT [4] already fine-tunes contiguous dense sub-blocks of the weights and claims efficient multi-adapter switching and "parallelism through reduced matrix multiplications"

Response:

We appreciate the reviewer's constructive feedback. Below, we clarify what TESSA builds upon and what it adds relative to existing partial-weight adaptation methods, and we have revised the abstract, introduction (Section 1), and Section 3 accordingly.

The main contribution of TESSA is to redesign how partial weight updates are laid out and executed for efficient multi-adapter serving under tensor parallelism. Prior selection-based and frozen-factor PEFT methods, including PaCA [1], LoRA-FA [2], and Asymmetry [3], primarily focus on training efficiency or adaptation quality rather than multi-adapter serving. Although S²FT [4] also explores adapter switching and parallel execution, efficient multi-adapter serving under tensor parallelism remains less explored. TESSA builds on these ideas to address this gap, with two key distinctions:

  • Shard-aligned placement avoids additional adapter-specific communication. TESSA selects weights along the unsharded axis, allowing each GPU to compute its adapter updates locally with balanced placement. Unlike PaCA’s input-column selection, this requires selecting output rows in row-parallel layers; TESSA is therefore not simply a contiguous-column special case of PaCA.

  • Unified GEMM and workload-aware dispatch improve concurrent multi-adapter execution. S²FT’s merge–unmerge switching can limit batching efficiency across different adapters [5, 6], while its separate base and adapter computation can underutilize the GPU during decoding. TESSA combines unified GEMM with workload-aware dispatch to improve utilization and select the faster execution path for each workload. In our experiments, TESSA achieves up to 1.69× the throughput of PaCA and 2.15× that of optimized S²FT (Figures R4.B and R4.C, respectively).

W2 + W3 + Q3. Contiguity vs. PaCA: TP argument, serving baseline, what contiguity adds

Reviewer comment:

W2, W3

The TP argument does not require contiguity either. Randomly selected columns (as in PaCA) are balanced across shards in expectation and need no collective. The requirement to use the unsharded axis arises only because the suffix is contiguous.

PaCA should be included as a serving baseline, or the paper should explain what contiguity adds beyond avoiding a gather.

Q3.

How do PaCA (Woo et al., 2025) and S²FT [4] compare in serving throughput and quality? What does contiguity add?

Response:

We thank the reviewer for these helpful comments and suggestions. We clarify why balance in expectation does not guarantee balanced adapter placement and evaluate the effects of channel selection, supporting contiguous suffixes as a practical choice rather than a requirement for communication-free execution. TESSA achieves comparable adaptation quality to PaCA and S²FT in various tasks. In our serving experiments, TESSA delivers up to 1.69× the throughput of PaCA and 2.15× that of S²FT at approximately matched parameters. We present the analyses and results below and have incorporated them into the revised manuscript (Sections 3.1, 3.3, and 4.1; Appendices C.7 and D.7).

1. Shard-aligned placement and channel selection

Random column selection must either add collectives or accept uneven adapter placement across GPUs, whereas TESSA balances adapter placement by construction without additional collectives. Expected balance does not ensure equal counts for each adapter: rebalancing can require remote activations, while keeping updates local can increase memory requirements on overloaded GPUs and limit throughput. Larger adapter footprints in MoE models [8] can amplify this memory imbalance.

The noncontiguous selection suggested by the reviewer may also avoid this imbalance if the selection is constrained by the shard boundaries so that every TP shard holds an equal number of selected coordinates. However, this constraint ties the selected channels to the TP degree used for serving. TESSA instead places the suffix on the unsharded axis, so the same adapter remains balanced at any TP degree supported by the base model without re-selecting channels. Beyond balanced placement and the removal of gathers, the contiguous suffix also benefits adaptation quality: contiguous selection achieves higher accuracy than random and evenly spaced selection at narrow widths, as shown in the next paragraph.

Effect of channel selection on adaptation quality. To examine how channel selection affects downstream quality, we compared two contiguous selections (prefix and suffix) against evenly spaced and random selections and LoRA at matched parameter budgets. In these math experiments, contiguous suffix selection methods generally achieve higher accuracy than others at narrow widths and perform comparably across the tested widths, suggesting that selecting an aligned, contiguous block may matter more than its trailing position in these experiments. This trend is consistent with the findings of RAPA [7] on the benefits of cross-layer index alignment.

Figure R4.A. Channel-selection ablation on Qwen3-8B-Base for GSM8K and MATH-500.

2. Serving throughput and adaptation quality

We evaluated PaCA and S²FT baselines in the same vLLM fork as TESSA on the Figure 5 workloads at approximately matched adapter parameter budgets.

Serving comparison with PaCA.

Experimental setup. For PaCA [1], which does not specify a multi-adapter serving strategy, we implemented it using TESSA’s kernels. We additionally evaluated PaCA + unified GEMM + dispatch as a control to measure the benefit of applying these TESSA execution optimizations to PaCA; they are components of TESSA, not of the original PaCA method.

Results. Across five matched adapter budgets (Figure R4.B), TESSA achieves 1.48–1.69× the throughput of PaCA on decode-heavy workloads and 1.01–1.25× on prefill-heavy workloads. Against this control variant, TESSA achieves 1.03–1.09× decode-heavy throughput at TP1 and 1.19–1.32× at TP4/TP8, with comparable prefill-heavy throughput.

Figure R4.B. Serving throughput of TESSA and PaCA variants across adapter parameter budgets. Top: decode-heavy; bottom: prefill-heavy workloads.

W2 + W3 + Q3. Contiguity vs. PaCA: TP argument, serving baseline, what contiguity adds (continued)

Response (continued):

Serving comparison with S²FT.

Experimental setup. For S²FT, we evaluated S²FT-official, which ports the adapter operations from the publicly released implementation [9], and S²FT-tuned, which uses tuned vLLM Punica kernels. Following S²FT [4], both baselines and TESSA (down-only) adapt only down_proj at approximately matched parameter budgets.

Results. TESSA (down-only) achieves 1.21–2.15× the throughput of S²FT-tuned on decode-heavy workloads and 1.02–1.18× on prefill-heavy workloads; the corresponding gains over S²FT-official are 1.41–2.17× and 1.22–1.47× (Figure R4.C). Figure R4.C also includes full TESSA, with approximately the same total budget distributed across the attention and MLP projections used in Figure 5.

Figure R4.C. Serving throughput of TESSA and S²FT variants across adapter parameter budgets. Top: decode-heavy; bottom: prefill-heavy workloads.

Where these gains come from. TESSA’s shard-aligned placement lets each GPU compute adapter updates locally without additional adapter-specific communication, explaining TESSA’s advantage under tensor parallelism. Unified GEMM execution reduces kernel-launch overhead and improves GPU utilization, so its benefit is most pronounced on decode-heavy workloads with small token batches. On prefill-heavy workloads, larger token batches amortize fixed communication and launch overheads, and redundant computation can diminish unified GEMM's advantage, so the dispatcher prefers separate execution when it is faster.

Adaptation quality comparison with PaCA and S²FT. Table R4.B compares LoRA, TESSA, PaCA, and S²FT on Qwen3-8B-Base and Llama-3.1-8B at approximately matched adapter budgets, with S²FT restricted to down projections following its GQA configuration [4]. TESSA achieves broadly comparable quality to two baselines, with relative performance varying by model and task, while providing the serving benefits above. Detailed settings are provided in Appendix C.1 of the revised manuscript.

Table R4.B. Downstream task accuracy (%) of LoRA, TESSA, PaCA, and S²FT on Qwen3-8B-Base and Llama-3.1-8B.

MethodCommonsenseGSM8KMATH-500HumanEvalMBPPAverage
Qwen3-8B-Base
LoRA88.6090.4083.8786.1882.3686.28
TESSA88.8494.4486.5386.5982.9887.88
S²FT (down-only)89.5394.4188.4786.1882.6388.24
PaCA89.5294.1187.6786.3883.2588.19
Llama-3.1-8B
LoRA83.5377.5336.4051.6367.1163.24
TESSA87.0079.9637.9353.2567.0265.03
S²FT (down-only)87.5980.7439.2048.7867.5564.77
PaCA87.4280.0138.9351.8367.4665.13
Reviewer_yD6oOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TESSA, an adapter design for efficient multi-adapter serving. Instead of learning both factors of a standard LoRA update, TESSA trains a fixed contiguous suffix of selected weight matrices. This structure reduces adapter computation and eliminates additional adapter-specific communication while retaining the base model’s tensor-parallel communication. The paper combines an analysis of LoRA overhead with adaptation-quality and serving-performance experiments.

Critical strengths

The paper carefully examines where the overhead of multi-adapter serving comes from. The breakdown of computation and communication helps explain why optimizing the implementation of the standard LoRA structure alone may leave substantial overhead. This provides a clear systems motivation for revisiting the adapter design.

TESSA is an appealing example of co-design between adapter parameterization and the computation and communication required for serving. The design is simple to understand, and the connection between the restricted parameter structure and the systems benefits is convincing.

Critical weaknesses

My main concern is the strength of the evidence for adaptation quality. The paper explains TESSA as a restricted LoRA form with one factor fixed to a coordinate-selection matrix, but this algebraic relationship does not establish that it preserves the adaptation capability of standard LoRA across workloads. The existing quality comparisons, including the additional models and tasks in the appendix, are useful. However, I would like stronger evidence on more recent models and more demanding tasks before concluding that the systems gains can generally be obtained without sacrificing quality.

In particular, results on challenging reasoning or agent tasks would help assess when restricting updates to a fixed contiguous suffix is sufficient and when it becomes limiting. I like the design, but the remaining uncertainty about quality generalization currently leads me to lean toward weak rejection. I would be open to revisiting this recommendation if stronger evidence addresses this concern.

Questions for authors

Could the authors provide an additional comparison against LoRA using a more recent model, such as Qwen3.6/3.8, on a more demanding workload, such as AIME or an agent task? These are examples rather than a request to evaluate all of them. It would be particularly useful to compare serving efficiency at matched adaptation quality and to discuss when the fixed-suffix restriction may limit performance.

4 comments

W1 + Q1. Quality generalization on recent models and demanding tasks

Reviewer comment:

W1.

My main concern is the strength of the evidence for adaptation quality. The paper explains TESSA as a restricted LoRA form with one factor fixed to a coordinate-selection matrix, but this algebraic relationship does not establish that it preserves the adaptation capability of standard LoRA across workloads. The existing quality comparisons, including the additional models and tasks in the appendix, are useful. However, I would like stronger evidence on more recent models and more demanding tasks before concluding that the systems gains can generally be obtained without sacrificing quality.In particular, results on challenging reasoning or agent tasks would help assess when restricting updates to a fixed contiguous suffix is sufficient and when it becomes limiting. I like the design, but the remaining uncertainty about quality generalization currently leads me to lean toward weak rejection. I would be open to revisiting this recommendation if stronger evidence addresses this concern.

Q1.

Could the authors provide an additional comparison against LoRA using a more recent model, such as Qwen3.6/3.8, on a more demanding workload, such as AIME or an agent task? These are examples rather than a request to evaluate all of them. It would be particularly useful to compare serving efficiency at matched adaptation quality and to discuss when the fixed-suffix restriction may limit performance.

Response:

We understand the reviewer’s concern and thank the reviewer for proposing valuable experiments. We evaluated Qwen3.6-27B on formal theorem proving and Qwen3.5-9B on medical-agent tasks, finding comparable adaptation quality to LoRA across the tested budgets. At comparable quality, TESSA achieves 1.77–2.10× the decode-heavy throughput and 1.24–1.36× the prefill-heavy throughput of S-LoRA across the two evaluations. The theorem-proving results are also included in Appendices C.6 and D.4 of the revised manuscript.

Experimental setup

Formal theorem proving. Following Pythagoras-Prover’s use of LoRA for supervised fine-tuning on Lean proofs [5], we fine-tuned Qwen3.6-27B [1] on 10,000 Lean 4 proof examples curated from the Goedel-Prover-V2 SFT dataset [3] across five LoRA ranks and TESSA suffix widths. We evaluated all configurations on miniF2F-test [2] under an 8K-token budget using the Goedel-Prover-V2 Lean verifier [4].

Medical agent evaluation. Using MedAgentGym [6] as a reference for the experimental setup, we generated trajectories for EHR-SeqSQL [7] and N-PowerAI [8] with DeepSeek-V4-Flash-0731 [10] and fine-tuned Qwen3.5-9B [9] on 863 trajectories and report five-seed mean success rates on 643 held-out episodes.

Adaptation quality

Formal theorem proving. TESSA achieves proof accuracy comparable to LoRA at similar adapter budgets. In Figure R3.A (left), TESSA w32 (117M trainable parameters) and LoRA r32 (159M) reach similar pass@8 scores of 68.85% and 68.65%, respectively. TESSA scores higher on pass@1 at several tested budgets, with TESSA w128 (470M) reaching 51.05% versus 49.28% for LoRA r128 (638M). At the smallest budget, TESSA w4 (14.68M) reaches 45.75% pass@1 and 66.59% pass@8, versus 45.62% and 68.03% for LoRA r4 (19.92M), with paired confidence intervals including zero for both differences.

Medical agent evaluation. In Figure R3.A (right), TESSA achieves comparable overall success rates to LoRA. For example, LoRA r16 (29.1M parameters) achieves 54.4% versus 52.9% for TESSA w16 (20.7M), while LoRA r32 (58.2M) achieves 56.2% versus 53.8% for TESSA w32 (41.4M). At the largest tested size for each method, TESSA w128 (165.7M) achieves a slightly higher mean success rate of 57.5% than LoRA r128 (232.8M, 56.5%), and also slightly exceeds the best tested LoRA configuration, r64 (116.4M, 57.0%).

Figure R3.A. Adaptation quality across adapter parameter budgets: miniF2F (left) and MedAgentGym (right).

W1 + Q1. Quality generalization on recent models and demanding tasks (continued)

Response (continued):

Serving efficiency

Formal theorem proving. Figure R3.B compares Qwen3.6-27B throughput on H100 GPUs at TP2 and TP4 using the Figure 5 decode-heavy (128 input / 512 output tokens) and prefill-heavy (8,192 input / 128 output tokens) workloads. Each point pairs one serving run with the two-seed mean miniF2F pass@8 at an 8K generation budget. At the same 70.28% pass@8, TESSA w128 achieves 1.77–1.85× the decode-heavy throughput and 1.24–1.32× the prefill-heavy throughput of LoRA r64. At 68.85% versus 68.65% pass@8, TESSA w32 achieves 1.84–1.89× and 1.30–1.32× the throughput of LoRA r32, respectively.

Figure R3.B. Serving throughput versus miniF2F pass@8

Medical agent evaluation. Figure R3.C compares Qwen3.5-9B throughput at TP1 and TP2 using the same Figure 5 workloads and untuned LoRA kernels. At comparable success rates (57.5% versus 57.0%), TESSA w128 achieves 1.85–2.10× the decode-heavy throughput and 1.31–1.36× the prefill-heavy throughput of LoRA r64.

Figure R3.C. Serving throughput versus MedAgentGym success rate.

References

[1] Qwen Team. Qwen3.6-27B model card. https://huggingface.co/Qwen/Qwen3.6-27B

[2] K. Zheng, J. M. Han, and S. Polu. MiniF2F: a cross-system benchmark for formal Olympiad-level mathematics. ICLR, 2022.

[3] Goedel-LM. Goedel-Prover-V2 SFT Dataset (Goedel-LM/SFT_dataset_v2). https://huggingface.co/datasets/Goedel-LM/SFT_dataset_v2

[4] Y. Lin et al. Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correction. ICLR, 2026.

[5] J. Ong Jun Leang et al. Pythagoras-Prover: Advancing Efficient Formal Proving via Augmented Lean Formalisation. arXiv:2606.12594, 2026. https://arxiv.org/abs/2606.12594

[6] R. Xu et al. MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Science. arXiv:2506.04405, 2025. https://arxiv.org/abs/2506.04405

[7] J. Ryu et al. EHR-SeqSQL: A Sequential Text-to-SQL Dataset for Interactively Exploring Electronic Health Records. Findings of ACL, 2024.

[8] P. Ruan et al. N-Power AI: A Specialized Agent Framework for Automated Sample Size and Power Analysis in Clinical Trial Design. bioRxiv, 2025. https://doi.org/10.1101/2025.02.06.636776

[9] Qwen Team. Qwen3.5-9B model card. https://huggingface.co/Qwen/Qwen3.5-9B

[10] DeepSeek-AI. DeepSeek-V4-Flash-0731 model card. https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731

Dear Reviewer yD6o,

We would like to sincerely thank you again for the time and effort you have devoted to reviewing our paper. Your insightful feedback and suggestions have helped us improve our paper.

We fully understand how demanding it is to review multiple submissions in such a short period. As the discussion deadline approaches, however, we would be very grateful if you could take a moment to share your thoughts on our responses. We would be happy to address any remaining concerns.

Best regards, The Authors

Thank you for the kind response, and adding more proof and agents results. I raised my score to support the empirical evidence.

Reviewer_8fUnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TESSA, an adaptation method designed for efficient multi-adapter serving. It finetunes contiguous suffixes of pretrained weight matrices, which reduces adapter computation to a single matrix product, and places the suffix along the unsharded axis under tensor parallelism to eliminate adapter-specific communication. It also includes workload-aware dispatch between separate and unified execution. Experiment results show substantial serving speedups with comparable adaptation performance compared with LoRA.

Critical strengths
  1. This paper focuses on an important and practical problem: the inference overhead of serving multiple LoRA adapters can be substantial despite their small parameter counts.
  2. The motivation of the proposed method is clear: try to avoid LoRA's shrink-expand path and adapter-specific inter-GPU communication.
  3. The evaluation experiments are comprehensive: dense and MoE models, fixed-length sweeps and open-loop production-trace replay, and a vLLM integration.
Critical weaknesses
  1. The technical contribution is limited. PaCA already finetunes a subset of input columns of the pretrained weight. The paper describes TESSA as 'a new adapter structure', which overclaims its novelty.
  2. The paper compares with PaCA only on adaptation quality (Appendix C.3 and C.5), not on serving. The current results therefore do not show the benefit of TESSA over a PaCA-style implementation served with comparable optimizations in the same system.
  3. The ablation studies are not complete. TESSA combines contiguous parameter selection, TP-aware placement, unified GEMM, optimized kernels, and workload-aware dispatch. However, the evaluation does not adequately separate their contributions.
Questions for authors
  1. Please state explicitly what distinguishes TESSA from PaCA at the method level.
  2. Please report throughput of PaCA-style implementations served in the same vLLM fork as a direct baseline.
  3. Please provide an end-to-end ablation isolating contiguous selection, shard-aligned placement, unified GEMM, optimized kernels, and workload-aware dispatch.
6 comments

W1 + Q1. Limited contribution vs. PaCA / method-level distinction

Reviewer comment:

W1.

The technical contribution is limited. PaCA already finetunes a subset of input columns of the pretrained weight. The paper describes TESSA as 'a new adapter structure', which overclaims its novelty.

Q1.

Please state explicitly what distinguishes TESSA from PaCA at the method level.

Response:

We thank the reviewer for this insightful comment. Compared with PaCA [1] and other partial-weight adaptation methods, our contribution is to redesign how partial weight updates are laid out and executed so that multi-adapter serving operates efficiently under tensor parallelism, while maintaining adaptation quality broadly comparable to LoRA. Together, these design and execution choices yield up to 1.69× the throughput of PaCA, as shown in Figure R2.A below. Much as recent works such as BD-LoRA [2] and zFLoRA [3] have demonstrated the value of adapting LoRA’s structure to eliminate TP communication or runtime overhead while preserving adaptation quality, we believe that TESSA is an important and practical contribution. We also acknowledge that some wording in the original manuscript may have overstated our contribution. So we have revised the abstract, introduction (Section 1), and Section 3 to clarify what TESSA builds upon and what it adds.

Figure R2.A. Serving throughput of TESSA and PaCA.

Table R2.A. Method-level comparison between TESSA and PaCA

AspectPaCA [1]TESSA
Weight selectionRandom input columns in every projection.Contiguous suffixes along the unsharded axis: input columns in column-parallel layers and output rows in row-parallel layers.
Tensor-parallel placementRandom selection can produce uneven adapter sizes across shards; balancing placement requires additional communication.Equal-width local adapter slices reuse the base model’s TP collectives without additional adapter communication.
Multi-adapter executionDoes not specify a concurrent multi-adapter serving strategy.Unified GEMM combines base and adapter computation; workload-aware dispatch selects unified or separate execution based on predicted latency.

W2 + Q2. PaCA serving baseline in the same vLLM fork

Reviewer comment:

W2.

The paper compares with PaCA only on adaptation quality (Appendix C.3 and C.5), not on serving. The current results therefore do not show the benefit of TESSA over a PaCA-style implementation served with comparable optimizations in the same system.

Q2.

Please report throughput of PaCA-style implementations served in the same vLLM fork as a direct baseline.

Response:

We thank the reviewer for suggesting this informative comparison. In the same vLLM fork and at approximately matched adapter budgets, TESSA achieves throughput speedups of 1.48–1.69× on decode-heavy workloads and 1.01–1.25× on prefill-heavy workloads over our PaCA implementation. We present these results below and in Appendix D.7 of the revised manuscript. As an additional comparison, TESSA achieves up to 2.15× the throughput of optimized S²FT at approximately matched adapter budgets (Appendix D.7 of the revised manuscript).

Experimental setup. Because PaCA does not provide an execution strategy or implementation for multi-adapter serving, we implemented it in the same vLLM fork as TESSA. To minimize differences due to kernel optimization, our PaCA implementation reuses most of the kernels used by TESSA. We evaluated five approximately matched adapter budgets on H100 GPUs, using the same fixed-length workloads as in Figure 5. To compare against PaCA with comparable execution optimizations, we additionally evaluated PaCA + unified GEMM + dispatch, applying these TESSA components to PaCA as a control.

Results. TESSA outperforms PaCA (Figure R2.A), achieving 1.48–1.69× throughput on decode-heavy workloads and 1.01–1.25× on prefill-heavy workloads. Against PaCA + unified GEMM + dispatch, TESSA still achieves 1.03–1.09× decode-heavy throughput at TP1 and 1.19–1.32× at TP4/TP8, with comparable prefill-heavy throughput (Appendix D.7 of the revised manuscript). These gains stem from combining shard-aligned adaptation with unified GEMM and workload-aware dispatch. Under tensor parallelism, shard-aligned adapter slices let each GPU compute its adapter updates locally without additional inter-GPU communication, which our PaCA implementation uses to handle its unevenly distributed columns. During decoding, unified GEMM reduces kernel-launch overhead and improves GPU utilization, while workload-aware dispatch chooses separate execution when the cost of redundant computation outweighs these benefits, typically with larger prefill batches.

Figure R2.A. Serving throughput of TESSA and PaCA.

W3 + Q3. Incomplete ablation / end-to-end ablation

Reviewer comment:

W3.

The ablation studies are not complete. TESSA combines contiguous parameter selection, TP-aware placement, unified GEMM, optimized kernels, and workload-aware dispatch. However, the evaluation does not adequately separate their contributions.

Q3.

Please provide an end-to-end ablation isolating contiguous selection, shard-aligned placement, unified GEMM, optimized kernels, and workload-aware dispatch.

Response:

We thank the reviewer for suggesting this informative ablation study. We conduct an end-to-end ablation and present the results and analysis below, which we have also included in Appendix D.5 of the revised manuscript. The results show that each component of TESSA addresses a different bottleneck: shard alignment balances adapter work across GPUs, the optimized kernels accelerate prefill-heavy workloads, unified GEMM accelerates decode-heavy workloads, and the dispatcher retains the faster of the two execution paths.

Experimental setup. We conducted an end-to-end ablation by cumulatively adding optimizations to a baseline with random column selection and naive kernels, following the stages in Table R2.B. We measured total throughput on three models with multiple TP degrees, using the fixed-length prefill and decode-heavy workloads from Figure 5 at two adapter parameter budgets.

Table R2.B. Ablation stages and execution control.

StageChange from the preceding stage
S0Baseline with random column selection and naive kernels (separate execution)
S1Switch to contiguous suffix selection
S2Align adapter placement with TP shards
S3Optimize kernels for separate execution
S4Add unified GEMM and workload-aware dispatch
Forced unifiedStart from S3, force unified GEMM, and disable dispatch

Contiguous selection, shard alignment, and kernel optimization. Figure R2.B shows the throughput as each optimization is added from S0 to S4, so the contribution of each optimization corresponds to the difference between adjacent stages.

  • Contiguous selection (S0 → S1). Replacing random columns with a contiguous suffix removes the gather of the selected inputs but has little effect on throughput at TP1. With tensor parallelism, contiguous selection alone can reduce throughput by 10–23% at the larger adapter budget and TP ≥ 4, because an input-axis suffix in row-parallel layers falls within the shard of a single rank and concentrates the adapter work on that rank. S1 is therefore an intermediate stage that TESSA does not use on its own.

  • Shard alignment (S1 → S2). S2 removes this imbalance by placing the suffix along the unsharded axis, as in TESSA, so that every rank performs an equal share of the adapter work. With tensor parallelism, S2 achieves 1.11–1.88× the throughput of S1 on decode-heavy workloads and 1.10–2.19× on prefill-heavy workloads. These results show why TESSA designs contiguous selection and shard alignment as a pair.

  • Kernel optimization (S2 → S3). Optimizing the kernels for separate execution yields 4.1–9.0× the throughput of S2 on prefill-heavy workloads, where large token batches make the adapter computation costly, and up to 1.84× on decode-heavy workloads.

Unified GEMM and workload-aware dispatch.

  • Unified GEMM (S3 → forced unified). Combining the base projection and the updates of multiple adapters in a single GEMM reduces kernel launches and increases parallel work for small token batches. As a result, on decode-heavy workloads, forced unified GEMM achieves 1.23–2.07× the throughput of S3. On prefill-heavy workloads, however, large token batches already amortize the launch overhead and increase the redundant computation for adapters not used by each token, so separate execution is faster in six of the ten tested configurations.

  • Workload-aware dispatch (S3 → S4). The dispatcher selects unified GEMM or separate execution for each projection and achieves at least 97.8% of the throughput of the faster forced path in every tested configuration. Roughly speaking, TESSA benefits from unified GEMM on decode-heavy workloads and from separate execution on prefill-heavy workloads.

Figure R2.B. End-to-end throughput across cumulative optimization stages S0–S4 on decode-heavy and prefill-heavy workloads.

Reviewer_uvuEOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3(+2)
Read review
Paper summary

This paper shows that multi-adapter LoRA serving suffers from execution and communication bottlenecks caused by sequential shrink-expand matrix multiplications and extra inter-GPU collectives under tensor parallelism (TP). To address this, the authors propose TESSA, an adapter structure that trains only a contiguous suffix of the base weight matrix and aligns the suffix axis with the TP sharding layout. This design reduces adapter computation to a single matrix product that can be merged with the base projection via a unified GEMM while eliminating adapter-specific TP communication. To demonstrate the effectiveness of the proposed method, the authors integrate TESSA into vLLM and perform experiments across dense and MoE models, evaluating fine-tuning accuracy against LoRA and serving throughput against S-LoRA and BD-LoRA under fixed-length and production-trace workloads.

Critical strengths

(S1) Co-design of adapter structure and tensor-parallel serving: By replacing the two-stage shrink-expand structure of LoRA with a single shard-aligned suffix update, TESSA removes additional adapter-specific all-gather and all-reduce collectives under tensor parallelism. Combining this structure with a workload-aware dispatcher in vLLM delivers clear throughput and latency improvements over S-LoRA and BD-LoRA across TP1 to TP8 configurations while maintaining comparable fine-tuning accuracy on several benchmarks.

Critical weaknesses

(W1) Missing evaluation on small on-device models: In practice, LoRA is most widely used in on-device or single-GPU settings with small dense models (e.g., 1B to 3B parameters) where base model capacity is limited, rather than on frontier-scale TP8 clusters. However, the paper only evaluates dense models of 7B/8B and above (and a 14B-total MoE model), and in single-device (TP1) deployments TESSA offers no TP communication savings at all.

(W2) Unaddressed peak memory overhead and redundant computation in Unified GEMM: Unified GEMM (Algorithm 1) introduces 2M(A−1)rd2M(A-1)rd redundant FLOPs by zero-masking unselected adapters (Appendix B.1, running up to 2.8x slower than separate execution at M=4096M=4096 in Figure 10) and requires materializing expanded tensors of width hin+Arh_{\text{in}} + Ar or hout+Arh_{\text{out}} + Ar (X~\widetilde{X}, W~\widetilde{W}, and [Ybase∣ΔY1∣⋯∣ΔYA][Y_{\text{base}} \mid \Delta Y_1 \mid \dots \mid \Delta Y_A]) in memory. The paper completely omits any discussion or measurement of this peak activation and workspace memory overhead, and Figure 10 even excludes input packing and unified-output assembly times from its latency measurements.

(W3) High variance and suspected cherry-picking in quality evaluations: While Table 1 reports only mean scores to claim quality gains over LoRA, Table 5 (Appendix C.2) reveals massive 95% confidence intervals for Qwen3-8B LoRA on GSM8K (89.71±23.7789.71 \pm 23.77) and MATH-500 (81.73±32.4781.73 \pm 32.47), and ±15.18\pm 15.18 for LLaMA-3.1-8B TESSA on MATH-500, indicating collapsed seeds rather than a reliable advantage. Furthermore, the hyperparameter selection rules in Appendix C.1 and C.5 are asymmetric in favor of TESSA, such as first fixing LoRA and then picking TESSA's learning rate to maximize its gain over the chosen LoRA configuration.

(W4) Lack of analysis on why updating only a fixed channel subset works: Unlike LoRA, which adapts representations across all dimensions, TESSA updates only an arbitrary contiguous suffix of width rr while leaving the vast majority of feature channels untouched. Aside from a brief observation on residual alignment in Appendix C.5, the paper lacks a fundamental analysis of why modifying an uncurated subset of trailing channels is sufficient to match full-dimension adaptation, or when this restriction fails at the smaller widths (RT=12R_T=12 to 2828 in Table 10) used in the serving sweeps.

Questions for authors

Please see the critical weaknesses.

6 comments

W1. Missing evaluation on small on-device models

Reviewer comment:

In practice, LoRA is most widely used in on-device or single-GPU settings with small dense models (e.g., 1B to 3B parameters) where base model capacity is limited, rather than on frontier-scale TP8 clusters. However, the paper only evaluates dense models of 7B/8B and above (and a 14B-total MoE model), and in single-device (TP1) deployments TESSA offers no TP communication savings at all.

Response:

We thank the reviewer for this helpful suggestion. On 1–3B models, TESSA achieves adaptation quality comparable to LoRA across all three models, while reaching 3.1–3.5× the single-GPU throughput of S-LoRA on decode-heavy workloads and 1.2–1.6× on prefill-heavy workloads, even without TP communication savings. We present the results below and have also included them in Appendices C.5 and D.3 of the revised manuscript.

TESSA maintains adaptation quality on 1–3B models. We evaluated Qwen3-1.7B-Base and Llama-3.2-1B/3B on the benchmark families in Table 1 at approximately matched adapter budgets, with detailed experimental settings and scoring procedures provided in Appendix C.5 of the revised manuscript.

As shown in Table R1.A, TESSA achieves broadly comparable accuracy to LoRA across the three models. For example, on Qwen3-1.7B-Base, TESSA achieves 82.6% on GSM8K and 68.5% on HumanEval, comparable to 81.8% and 69.5% for LoRA. This trend also holds for Llama-3.2-1B and 3B, where TESSA performs on par with LoRA across all tasks. Overall, TESSA maintains accuracy comparable to LoRA on small dense models, consistently across diverse tasks and model families.

Table R1.A. Downstream task performance on 1–3B models at approximately matched adapter parameter budgets. Base* denotes the model without fine-tuning.

MathCodeCommonsense
ModelMethodGSM8KMATH-500HumanEvalMBPPAvg.
Qwen3-1.7B-BaseBase*72.750.421.339.746.0
LoRA81.861.869.570.882.2
TESSA82.662.968.569.582.2
Llama-3.2-1BBase*2.33.13.724.31.3
LoRA6.63.518.537.273.2
TESSA5.63.719.337.774.6
Llama-3.2-3BBase*4.65.924.447.44.8
LoRA38.011.633.951.681.6
TESSA36.29.934.851.983.7

TESSA also improves serving throughput on small models on a single GPU. Even for small models on a single GPU, TESSA can outperform S-LoRA by fusing the base projection and adapter update into a single GEMM. We use the fixed-length workloads from Figure 5: (input=128, output=512) for decode-heavy workload, and (input=8192, output=128) for prefill-heavy workload. We extend that setup to three smaller model configurations on a single H100 or A100.

Under these conditions, TESSA achieved 3.1–3.5× throughput (decode-heavy) and 1.2–1.6× throughput (prefill-heavy) relative to S-LoRA across the three model configurations. These gains can be explained by the combination of partial-weight adaptation, which reduces each adapter computation to a single matrix multiplication, and unified GEMM execution, which combines it with the base projection for more efficient execution even without TP communication savings.

Figure R1.A. Single-GPU serving throughput of TESSA and S-LoRA on 1–3B models (H100). Top: decode-heavy; bottom: prefill-heavy workloads.

Figure R1.B. Single-GPU serving throughput of TESSA and S-LoRA on 1–3B models (A100). Top: decode-heavy; bottom: prefill-heavy workloads.

W2. Unaddressed peak memory overhead and redundant computation in Unified GEMM

Reviewer comment:

Unified GEMM (Algorithm 1) introduces 2M(A−1)rd2M(A-1)rd redundant FLOPs by zero-masking unselected adapters (Appendix B.1, running up to 2.8x slower than separate execution at M=4096M=4096 in Figure 10) and requires materializing expanded tensors of width hin+Arh_{\mathrm{in}}+Ar or hout+Arh_{\mathrm{out}}+Ar (X~\tilde{X}, W~\tilde{W}, and [Ybase∣ΔY1∣⋯∣ΔYA][Y_{\mathrm{base}}\mid\Delta Y_1\mid\cdots\mid\Delta Y_A]) in memory. The paper completely omits any discussion or measurement of this peak activation and workspace memory overhead, and Figure 10 even excludes input packing and unified-output assembly times from its latency measurements.

Response:

We thank the reviewer for this careful observation. We agree that unified GEMM is not beneficial in every case, as it computes redundant FLOPs on unselected adapters and requires a temporary expanded buffer. For this reason, TESSA does not apply unified GEMM to every projection. As described in Section 3.2, a workload-aware dispatcher uses unified GEMM only where it is faster, and otherwise switches to separate execution, which computes the base projection and adapter updates in separate kernels. As a result, including input packing and output assembly does not change the trade-off in Figure 10, and the additional per-GPU peak memory over S-LoRA is at most 0.447% of the KV cache pool across all tested configurations.

Specifically, the redundant FLOPs and the expanded buffer arise from the same source: unified GEMM computes the update of every active adapter for every token. Consequently, both grow with the same factors. The redundant FLOPs scale with M(A−1)rM(A-1)r, and the buffer requires bMArbMAr bytes, where MM is the number of token rows, AA is the number of active adapters, rr is the adapter width, and bb is the number of bytes per element. Therefore, the redundant FLOPs slow down unified GEMM only when MM, AA, and rr are large enough, and this is precisely where the dispatcher selects separate execution, which incurs no redundant FLOPs and almost no peak memory overhead. The 2.8× slowdown in Figure 10 corresponds to this case. Unified GEMM is used only where it is faster, such as for the small token batches in decoding.

Table R1.B. Peak-memory overhead over S-LoRA across execution modes (MiB/GPU). KV tokens/pool equiv.: the dispatch-enabled overhead expressed as KV cache tokens and as a fraction of the KV cache pool.

ModelTPDispatchForced unifiedForced separateKV tokens equiv.KV pool equiv.
Llama-3.1-8B11.098.00.080.003%
Llama-3.1-8B218.8210.00.73000.040%
Llama-3.1-8B470.0246.00.82,2410.140%
Qwen3-32B20.00.00.90≈0%
Qwen3-32B464.6228.00.91,0340.176%
Qwen3-32B892.5224.50.72,9600.285%
Llama-3.3-70B425.5120.60.13260.145%
Llama-3.3-70B866.4255.20.81,7000.447%

Finally, regarding Figure 10, the original plot measured GEMM time only to illustrate this trade-off, as noted in its caption, whereas the dispatcher used in our serving experiments was calibrated with the actual serving kernels and full-layer timings that include input packing and output assembly. Following the reviewer’s suggestion, we have updated Figure 10 in Appendix B.2 to include input packing for unified GEMM and output assembly for separate execution (Figure R1.C below). Even with these operations included, the trade-off remains the same: for small token batches (M=64M=64), unified GEMM is faster than or on par with separate execution across ranks and active adapters, whereas for large token batches (M=4096M=4096), unified GEMM becomes up to 2.8× slower as the rank and the number of active adapters grow. Since the dispatcher selects separate execution in the latter case, these overheads do not affect TESSA’s serving performance. We also clarify the calibration procedure in Appendix B.2.

Figure R1.C. Unified-to-separate latency ratios including input packing and output assembly.

W3. High variance and suspected cherry-picking in quality evaluations

Reviewer comment:

While Table 1 reports only mean scores to claim quality gains over LoRA, Table 5 (Appendix C.2) reveals massive 95% confidence intervals for Qwen3-8B LoRA on GSM8K (89.71±23.7789.71\pm23.77) and MATH-500 (81.73±32.4781.73\pm32.47), and ±15.18\pm15.18 for LLaMA-3.1-8B TESSA on MATH-500, indicating collapsed seeds rather than a reliable advantage. Furthermore, the hyperparameter selection rules in Appendix C.1 and C.5 are asymmetric in favor of TESSA, such as first fixing LoRA and then picking TESSA's learning rate to maximize its gain over the chosen LoRA configuration.

Response:

We thank the reviewer for their careful review and apologize for not checking these aspects more thoroughly. We corrected an error in the confidence-interval calculation and reran the quality and suffix-alignment experiments using symmetric learning-rate selection. The reruns under symmetric learning-rate selection show that TESSA maintains quality comparable to LoRA and preserve the main conclusions of the suffix-alignment analysis: faster early validation-loss reduction and stronger early intervention effects on the probed example (original Appendix C.5; revised Appendix C.7). We present the corrections and results below and have updated the corresponding tables and appendix sections in the revised manuscript (Section 4.1 and Appendices C.1, C.2, and C.7).

The confidence intervals for the math results in Table 5 were miscalculated and therefore overestimated. When preparing the table, we mistook the already computed 95% confidence interval half-widths for standard deviations and applied the Student’s tt scaling factor again. This inflated all 12 math CI half-widths by 2.48x, while the mean scores remain unchanged. We rechecked the uncertainty calculations for the other quality results with available seed-level records and found no other such error. Table R1.C lists the corrected intervals.

Table R1.C. Original math scores with corrected confidence intervals (mean ± CI, %).

Model / methodGSM8K (reported → corrected)MATH-500 (reported → corrected)
Llama-3.1-8B LoRA79.33 ± 7.95 → ± 3.2038.47 ± 7.13 → ± 2.87
Llama-3.1-8B TESSA79.63 ± 2.36 → ± 0.9538.40 ± 15.18 → ± 6.11
Qwen3-8B-Base LoRA89.71 ± 23.77 → ± 9.5781.73 ± 32.47 → ± 13.07
Qwen3-8B-Base TESSA94.44 ± 3.11 → ± 1.2587.53 ± 3.78 → ± 1.52
Qwen3-14B-Base LoRA95.15 ± 1.86 → ± 0.7590.80 ± 5.66 → ± 2.28
Qwen3-14B-Base TESSA95.10 ± 0.72 → ± 0.2991.20 ± 7.50 → ± 3.02

The reruns with symmetric learning-rate selection yielded narrower confidence intervals, with TESSA achieving comparable or higher scores than LoRA. Even after the above correction, the confidence intervals for the results highlighted by the reviewer remained wide. To address this variability and put LoRA and TESSA on equal footing, we reran the GSM8K and MATH-500 comparisons on Qwen3-8B-Base and Llama-3.1-8B.

Each method selected its learning rate independently from the same grid {2×10−5,10−4,3×10−4}\{2\times10^{-5},10^{-4},3\times10^{-4}\}. Among the learning rates, we selected the one with the highest mean validation gain across the two benchmarks and three seeds, breaking ties with the smaller rate. For Qwen3-14B-Base, this procedure selects the learning rates already used for both methods, so we retain the existing results in the revised manuscript (Section 4.1 and Appendix C.2).

As shown in Table R1.D, the 95% CIs for all three flagged results narrowed to within ± 3%, indicating stable results across seeds. TESSA achieves absolute accuracy gains of about 5% on both Qwen3-8B-Base tasks and remains comparable to LoRA on Llama-3.1-8B (paired gains of +1.09 ± 2.97 and +0.60 ± 2.59 % on GSM8K and MATH-500). Table R1.D reports five-seed means to examine seed-to-seed variability, while revised Table 6 in Appendix C.2 reports three-seed means for consistency with the other evaluations.

Table R1.D. Math performance under symmetric learning-rate selection (five-seed mean ± 95% CI).

ModelMethodSelected LRGSM8KMATH-500
Llama-3.1-8BLoRA3 × 10⁻⁴78.20 ± 1.5237.20 ± 2.14
Llama-3.1-8BTESSA3 × 10⁻⁴79.29 ± 1.5837.80 ± 1.75
Qwen3-8B-BaseLoRA10⁻⁴89.60 ± 2.9582.28 ± 2.94
Qwen3-8B-BaseTESSA3 × 10⁻⁴94.59 ± 0.6587.76 ± 2.38

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.