acceptodds
Under review as a conference paper at ICLR 2027

Balanced-KnOTS: Task-Balanced Subspace Selection for Fixed-Rank LoRA Merging

Abstract

Post-hoc LoRA merging can replace a growing adapter library with one static model, but under a fixed deployment rank, every expert must compete for the same limited shared space. Existing shared-SVD methods optimize aggregate reconstruction, which can preserve substantial total energy while severely underrepresenting individual tasks. We identify this as an aggregate-diagnostic blind spot and show that discarded task components create an irrecoverable projection bottleneck: no subsequent merger restricted to the selected space can reconstruct them. We introduce task-wise projection coverage to quantify per-expert representational availability and propose Balanced-KnOTS, which removes norm-induced task weighting and optimizes a mean–soft-min objective to protect the coverage lower tail before applying the same TIES procedure in the retained coordinates. On 20 independently trained Qwen3-8B security-task LoRA experts consolidated into a single rank-16 adapter, Balanced-KnOTS raises worst-task coverage from 32.84% to 56.98% while leaving mean coverage essentially unchanged. Relative to equal-rank Joint KnOTS-TIES, it improves five of six held-out task metrics, increasing mean expert recovery from 82.80% to 87.55% and worst-task recovery from 54.12% to 63.66%. These results show that fixed-rank consolidation requires controlling how shared capacity is allocated across experts, not only how retained directions are merged.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LoRA merging, shared subspace selection, task-wise projection coverage, fixed-rank consolidation, multi-task learning.

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_Vd1NOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Balanced-KnOTS, a task-balanced subspace selection method for merging multiple LoRA adapters into a single adapter with a fixed rank. The authors argue that conventional joint SVD maximizes aggregate reconstruction energy but may fail to preserve task-specific information, especially when the number of tasks increases. To address this issue, the paper introduces task-wise projection coverage to measure the information retained for each task and proposes a mean-soft-min objective to balance average and worst-task coverage. The selected subspace is then combined with TIES to construct the merged adapter. Experiments on 20 LoRA experts trained on security-related tasks show improvements in worst-task projection coverage and downstream task performance compared with Joint KnOTS-TIES.

Critical strengths
  • The paper is overall well-written and easy to follow
  • Their motivation is clear and makes sense. Their method addresses their proposed challenge, supported by empirical results.
  • Their experiments are comprehensive with sufficient ablation study.
Critical weaknesses
  • The main concern is the diversity of their experiments. They only tested their method on one model with one set of datasets. I recommend adding more model families and benchmarks.
  • Another concern is their computational cost. I think their method would be costly, and I suggest that the authors report the runtime instead of a theoretical cost analysis.
Questions for authors
  • In Sec.4.3, I wonder if the searched subspace by joint SVD still favors the tasks with a higher coverage?
  • Sec.5 is not well motivated by Sec.4 (lines 251-252): why do we need a dynamic router rather than a static LoRA? How do a router solve the problems in Sec.4?
  • There is a trade-off when we balance the subspace: when the main task is suppressed, other tasks may benefit, but the main task's performance may drop; when the main task is well preserved, other tasks suffer. For me, it is hard to tell which option is better.
  • I am interested in the comparison between the task coverage and the performance drop after merging. Not sure whether the main task will suffer a higher performance drop.
Reviewer_5PfPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Balanced-KnOTS addresses the uneven allocation of shared low-rank capacity in LoRA merging, where optimizing aggregate reconstruction can underrepresent individual tasks. It introduces task-wise projection coverage and a mean–soft-min objective to balance subspace allocation across experts before applying TIES merging.

Critical strengths
  1. The paper is clearly written and well organized. The discussion progresses logically from identifying the limitations of existing methods to theoretical analysis and method design, making the overall approach easy to follow.

  2. The experimental evaluation is relatively comprehensive. In addition to comparisons with existing methods, the paper includes ablation studies and experiments with different numbers of experts and rank budgets, providing evidence for the effectiveness of the proposed components across different settings.

Critical weaknesses
  1. Balanced-KnOTS mainly introduces task normalization and a Mean–Soft-Min objective into the shared subspace selection stage of KnOTS, while retaining the existing TIES merging procedure. These modifications are relatively straightforward, and the methodological novelty could be further strengthened.

  2. All experiments are conducted using Qwen3-8B and a single collection of security-related LoRA experts trained under a unified configuration. Evaluations across different model architectures, model scales, and task domains would help demonstrate the generalizability of the proposed method.

  3. Although the paper merges 20 LoRA experts, the main experiments evaluate downstream performance on only six tasks. Therefore, the reported mean and worst Expert Recovery scores do not fully reflect how well the method preserves the capabilities of all 20 experts.

  4. The paper provides a controlled comparison with Joint KnOTS-TIES under the same rank budget, but other baselines, such as TSV and Iso-C, use different output representation sizes. More comparisons under matched deployment parameter budgets would help establish the advantages of the proposed method.

  5. The relationship between projection coverage and downstream task performance remains unclear. The paper reports a weak correlation between coverage improvements and Expert Recovery improvements (Spearman ρ = 0.222), and some cases show increased coverage but decreased task performance. This raises questions about how reliably the proposed optimization objective reflects actual multi-task performance.

  6. The advantages of Balanced-KnOTS become less pronounced as the shared rank budget increases. With 20 experts and rank 64, its worst Expert Recovery is even 1.49 percentage points lower than Joint KnOTS-TIES. Additionally, its offline merging process takes approximately 5.7 times longer, which may limit its practical benefits when rank constraints are less restrictive.

Questions for authors

See Weaknesses.

Reviewer_4F9tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies post-hoc LoRA merging under a fixed deployment-rank budget. It introduces task-wise projection coverage to measure how much of each expert update remains representable in a shared subspace. Balanced-KnOTS selects this subspace using task normalization and a mean–soft-min coverage objective, then applies TIES to the projected updates. Experiments consolidate 20 Qwen3-8B security-task experts into one rank-16 adapter and evaluate six downstream tasks. Compared with equal-rank Joint KnOTS-TIES, the method improves minimum coverage from 32.84% to 56.98%, mean expert recovery from 82.80% to 87.55%, and minimum recovery from 54.12% to 63.66%.

Critical strengths
  1. A practically relevant setting with a useful controlled comparison. Consolidating independently trained experts into a single static adapter under a fixed rank budget is a meaningful deployment problem. The comparison with Joint KnOTS-TIES holds the experts, output rank, and downstream merging configuration fixed, providing useful evidence that shared-subspace selection affects the final result. Improvements on five of six task metrics make the proposed approach worth investigating.

  2. Informative analysis of the limitations of aggregate reconstruction. The paper shows that aggregate coverage can conceal substantial differences across experts. It also explicitly reports counterexamples to a simple coverage–performance relationship, including the degradation in minimum expert recovery after task normalization. This transparency helps distinguish parameter representability from downstream effectiveness and makes the empirical analysis more informative.

Critical weaknesses

1. The connection between balanced coverage and final task performance is insufficiently validated.
Coverage measures whether individual expert updates remain representable, whereas deployment requires one merged update to serve all tasks. Components preserved during subspace selection may still be removed by TIES through trimming or sign selection. Moreover, retained Frobenius energy need not correspond to useful task behavior: normalization increases minimum coverage from 32.84% to 48.55% but decreases minimum expert recovery from 54.12% to 45.88%, and the reported coverage–performance association is weak. These findings do not invalidate the final gains, but leave their connection to the optimized objective unclear. Evaluating projected experts before TIES and examining subsequent aggregation would help distinguish functional preservation from merging effects.

2. Novelty and empirical generality remain insufficiently established.
The normalized mean–soft-min formulation is closely related to fair subspace selection and multi-criteria dimensionality reduction. The paper should clarify its distinctive contribution in LoRA merging and justify the chosen objective relative to other reasonable balancing objectives. Meanwhile, the main downstream results concern one base model and one security-domain expert pool; the mapping between 20 experts and six evaluations, and the uncertainty of the reported gains, also require clarification. The referenced appendices may address some robustness concerns, but were unavailable for this assessment, so their adequacy remains unverified. The available evidence therefore supports a promising result in the reported setting more strongly than a broadly validated merging approach.

Questions for authors
  1. Intermediate-stage evidence: Have you evaluated each expert after projection into the selected shared space but before TIES? Does Balanced-KnOTS improve these projected experts’ task performance, and how does this compare with the final merged-model improvements?

  2. Interaction with TIES: Is there evidence that the additional components retained by Balanced-KnOTS survive trimming and sign selection and contribute to the final gains? Any comparison should account for the different coordinate bases selected by the methods.

  3. Novelty and objective choice: How does the formulation relate to Fair PCA or multi-criteria dimensionality reduction? What evidence supports the mean–soft-min objective over another reasonable balancing objective?

  4. Evaluation clarification: How do the 20 experts map to the six downstream evaluations, and how is the reference expert selected for each recovery score? Do the appendices provide uncertainty estimates and evidence that the main gains persist across expert subsets or independently constructed settings? Please identify the relevant results and clarify how the merging hyperparameters were selected.

Reviewer_dDSUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper examined the problem of imbalanced task coverage with joint SVD-based LoRA merging methods, and proposed to balance that coverage across tasks. The proposed method -- Balanced-KnOTS, consistently improves task coverage when merging 20 LoRA adapters of Qwen models. Although the performance is not uniformly good, and the method brings significant compute overhead compare to baselines.

Critical strengths
  1. Clear problem setup. I can see that balancing subspace selection to avoid bullying some under-represented tasks is important in some use cases.
  2. Clean method design. The proposed method is simple and elegant, making it easy to be used in existing systems.
Critical weaknesses
  1. Lack of sensitivity study on the choice of hyperparameters. I wonder how sensitive your method's performance is to hyperparameter selection, especially the lambda and tau.
  2. High computational overhead. Table 10 shows that Balanced-KnOTS incurrs ~6x compute overhead compare to joint SVD. That sounds like a significant limitation and thus requires more thorough mentioning/discussion in the main text. Also would like to hear your thoughts on how it can be improved.
  3. Limited experiment scope and analysis. The experiments focus on a single set of LLMs with mixed results on the coding task. I think more empirical study on different base models and tasks would provide more useful evidence to see whether the proposed method is broadly applicable or not. More in-depth analysis on the failure case of the coding task would be useful too.
Questions for authors
  1. Do you think your approach can be used to prioritize any user-specified subset of tasks? That might be an interesting application area to look into.
  2. How much does having balanced performance matters in practice?
  3. How good is your method compare to serving LoRA adapters directly, in terms of task performance and space/time consumption?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.