acceptodds
Under review as a conference paper at ICLR 2027

H-OM: Hadamard-Orthogonal Model Merging with Recoverable Task Components

Abstract

Model merging combines task-specific updates into a single model without joint retraining. However, existing independent and sequential methods for model merging cannot adapt to real-world deployments due to dependence on original adapters or preconditioned tasks. In this paper, to address the problem, we propose Hadamard Orthogonal Merging (H-OM) that constructs mutually orthogonal output subspaces for Low-Rank Adaptation (LoRA). H-OM learns to adapt to each task learns within its assigned subspace to simultaneously allow parallel adaptation and maintain orthogonal updates. Furthermore, we develop Coordinate-Exposure Uniformity (CEU) to guide subspace construction for effective adaptation, and induce subspaces with uniform coordinate exposure from disjoint sets of randomized Hadamard basis vectors. H-OM is flexible to enable component removal and reweighting without retaining the original adapters, since it can exactly recover the weight updates for tasks from their projection in the assigned subspaces under linear merging. Experiments on VTAB-1K and NLP-15 demonstrate that H-OM yields state-of-the-art accuracy in model merging with evidently fewer trainable parameters than standard LoRA. Remarkably, H-OM recovers all 19 VTAB-1K task updates from the merged model using the pretrained base model and reproducible basis metadata, without compromising single-task accuracy.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Model Merging, Low-Rank Adaptation, Task Vectors, Orthogonal Subspace Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JaXfOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Hadamard Orthogonal Merging (H-OM), which assigns each task a mutually orthogonal output subspace before task-specific fine-tuning. Different tasks can be trained independently and then linearly merged. Because the task subspaces are mutually orthogonal, each weighted task component in the merged model can be recovered by projection, enabling component deletion, replacement, and reweighting without retaining the original task adapters. The authors further propose Coordinate-Exposure Uniformity (CEU) to measure the uniformity of coordinate exposure induced by a subspace, and use a randomized Hadamard basis to achieve maximum CEU. Experiments on VTAB-1k and NLP-15 demonstrate improved merged accuracy with fewer trainable parameters than standard LoRA, as well as component recovery after serialization and reloading.

Critical strengths
  1. The method is simple, and the recovery property is clearly stated and correct.

  2. The merging performance gains are reasonably consistent under the evaluated protocols.

  3. The appendices provide a relatively thorough account of key limitations and the practical recovery workflow.

Critical weaknesses
  1. Limited applicability and a substantial adaptation cost. H-OM requires coordinated subspace allocation before fine-tuning and therefore cannot directly merge arbitrary existing adapters. This restricts its applicability. Moreover, Table 16 shows that its single-task accuracy drops from 68.36% with standard LoRA to 62.98%, a substantial loss of 5.38 percentage points.

  2. Limited theoretical scope and insufficient attribution to CEU. Maximizing CEU equalizes coordinate leverage scores but does not guarantee sufficient retention of actual task gradients or effective adaptation. The CEU ablation also changes subspace orientation and thus gradient alignment, establishing correlation within the tested construction family rather than isolating CEU’s contribution. Furthermore, Hadamard improves over Gaussian QR by only 0.44 percentage points.

  3. Parameter separability provides limited evidence of selective task editing. Proposition 1 guarantees component recovery and removal, but not prediction-level independence. Removing a component changes retained-task accuracy by up to 6.50 percentage points, while the removed task loses only 4.80 points on average. These results limit the evidence for selective behavioral editing, although parameter-level editing is correctly established.

  4. Insufficient validation of continued task addition. The default allocations nearly exhaust the available capacity: 760/768 directions on VTAB and 4095/4096 on LLaMA, leaving insufficient room for another task of the same subspace width. Table 17 evaluates reserving capacity but not performance after subsequent task additions. Moreover, the suggested subspace sharing would invalidate the orthogonality assumption underlying Proposition 1, requiring clarification of its implications for exact recovery.

  5. Limited baseline coverage and differences in training assumptions. Despite using multiple merge operators, Table 1 includes only standard LoRA, KnOTS, and MergOPT. KnOTS operates on independently trained adapters, whereas H-OM controls their training subspaces in advance; the comparison therefore requires qualification regarding these different assumptions. Comparisons with additional representative methods, such as DC-Merge [1] and ESM [2], would strengthen the claimed performance advantage.

[1] DC-Merge: Improving Model Merging with Directional Consistency. CVPR, 2026.

[2] Model Merging in the Essential Subspace. CVPR, 2026.

Questions for authors
  1. What practical settings justify preallocated subspaces despite the 5.38-point single-task accuracy loss, and how does H-OM compare with joint multi-task training?

  2. Can you isolate CEU’s contribution from subspace–gradient alignment and verify the advantage over Gaussian QR across multiple basis seeds?

  3. Can you report the full cross-task impact of component removal and reweighting to assess editing selectivity?

  4. How does performance evolve when new tasks are added to reserved capacity, and how would subspace sharing affect exact recovery?

  5. Can you include DC-Merge and ESM and clarify the different training assumptions in comparisons with post-hoc merging methods?

7 comments
AuthorsOpenReview ↗

Responses to Reviewer JaXf

We are grateful to the reviewer for a careful reading and thoughtful suggestions, which have tangibly improved the clarity of our exposition and the breadth of our empirical evidence. We respond to each point in turn below.

Comment 1: Given the 5.38-point drop in single-task accuracy, what practical settings still justify preallocating subspaces? How does H-OM compare with joint multitask training?

Comment 5: Could you include DC-Merge and ESM in the comparison and clarify how H-OM's training assumptions differ from those of post-hoc merging methods?

Response:

We thank the reviewer for asking these significant questions. To clarify H-OM's training assumptions and practical significance, We will address Comments 1 and 5 together.

1. When subspace preallocation is worthwhile.

H-OM's primary goal is to support task addition, removal, and version replacement in a single merged model after release. Exact recovery is the mechanism that enables this maintenance, not a separate predictive route: predictions are always served by the merged model, and a recovered update is used only to edit the merged weights. Preallocation is worthwhile when teams can agree on a shared pretrained base and task subspaces before fine-tuning, and downstream maintainers expect to edit the released model after the original adapters are no longer available. Given the shared base, merged checkpoint, and subspace metadata, maintainers can recover an existing task's weighted update for removal or replacement, and they can add independently trained tasks when unused subspaces are available. The price of this design is that orthogonality constraints during training limit single-task adaptation, a trade-off inherent to orthogonal methods in general. Since the goal of model merging is to obtain a better merged model, this cost should be weighed against the quality of the merge rather than judged in isolation.

2. Accuracy comparison with joint training, DC-Merge, and ESM

We added joint multitask training, DC-Merge, and ESM to the ViT-B/16 evaluation on all 19 VTAB-1k tasks.

Table 1. Merged accuracy on all 19 VTAB-1k tasks using CLIP ViT-B/16. Entries report mean accuracy (%) ± standard deviation across three random seeds.

MethodNatural (7)Specialized (4)Structured (8)Overall (19)
Joint multitask training80.63 ± 0.2883.10 ± 0.7536.78 ± 0.9562.69 ± 0.61
Task Arithmetic78.09 ± 0.1878.46 ± 0.3523.12 ± 0.5555.02 ± 0.22
DC-Merge78.86 ± 0.1980.45 ± 2.4925.62 ± 0.8856.78 ± 0.37
ESM78.94 ± 0.6180.10 ± 2.9324.80 ± 0.9056.39 ± 0.47
H-OM78.15 ± 0.2180.72 ± 1.1328.65 ± 1.2257.85 ± 0.64

Joint multitask training attains the highest accuracy in this comparison, 62.69%, 4.84 pp above H-OM. It is an upper bound rather than a competing merge, since it requires all task data to be optimized together. Among the methods that merge independently trained task updates, H-OM exceeds DC-Merge [1] and ESM [2] by 1.07 and 1.46 pp.

The difference in training assumptions is one of timing and coordination. DC-Merge and ESM operate on task models after fine-tuning and require no agreement beforehand, so they can be applied to checkpoints that already exist. H-OM fixes the task subspaces before fine-tuning; after that point each team trains independently on its own data, and the resulting weighted linear merge carries a known task decomposition that maintainers can use to edit the released checkpoint.

References.

[1] Zhang et al. DC-Merge: Improving Model Merging with Directional Consistency. CVPR 2026.

[2] Li et al. Model Merging in the Essential Subspace. CVPR 2026.

AuthorsOpenReview ↗

Responses to Reviewer JaXf

Comment 2: Can you isolate CEU’s contribution from subspace–gradient alignment and verify the advantage over Gaussian QR across multiple basis seeds?

Response:

1. CEU’s contribution

To examine how coordinate exposure relates to accuracy, and to separate it from the alignment between task gradients and the assigned subspace, we built a controlled family of bases. Starting from the Hadamard dictionary, we linearly interpolate it with the identity matrix at different mixing ratios and orthonormalize the result by QR, which yields orthonormal bases spanning CEU from 0.20 to 0.80. Every basis in this comparison is fixed before training and independent of task data, so none of them is selected or tuned for alignment with task gradients; within the controlled family all members share the same origin and only the mixing ratio varies, which changes how energy is spread across coordinates while the dictionary family is held fixed. We compare these bases with a Gaussian QR basis, a real Fourier basis, and Hadamard on both merged accuracy and single-task accuracy, each averaged over 10 independent runs.

Table 2. Effect of CEU on single-task and merged accuracy on VTAB-1k using ViT-B/16.

BasisCEU ↑Single-task Acc. (%) ↑Merged Acc. (%) ↑
Controlled0.2061.89 ± 1.2353.86 ± 1.15
Controlled0.4062.31 ± 1.0854.46 ± 0.96
Controlled0.6063.08 ± 0.8555.14 ± 0.79
Controlled0.8064.02 ± 0.7656.47 ± 0.72
Gaussian QR0.9564.87 ± 0.6957.21 ± 0.70
Real Fourier1.0065.21 ± 0.6157.61 ± 0.66
Hadamard1.0065.33 ± 0.5857.85 ± 0.64

Raising CEU from 0.20 to 0.80 along the controlled bases raises single-task accuracy from 61.89% to 64.02% and merged accuracy from 53.86% to 56.47%, so both improve as coordinate exposure becomes more uniform. At CEU at or near 1, the single-task accuracies converge to within 0.46 points of one another (64.87, 65.21, and 65.33), while the merged accuracies still span 0.64 points (57.21, 57.61, and 57.85). CEU therefore accounts for the single-task trend, whereas merged accuracy at CEU = 1 still depends on the construction. The real Fourier and Hadamard bases both attain CEU = 1 and yield similar merged accuracies, 57.61% and 57.85%, which shows that uniform coordinate exposure and comparable merging performance can be achieved with distinct orthogonal constructions.

2. Why choose Hadamard? Although Gaussian QR reaches CEU 0.950.95 and achieves accuracy close to Hadamard , we choose Hadamard for three reasons. First, it guarantees CEU=1\mathrm{CEU}=1 by construction. Second, it avoids dense O(d3)O(d^3) QR factorization: a Hadamard dictionary is generated in O(d2)O(d^2) and supports fast Walsh--Hadamard projection without explicitly materializing QQ. Table 3 reports measured speedups and peak-memory reductions. Third, because the assigned basis is a Hadamard matrix, projection onto it can be computed by the fast Walsh-Hadamard transform in O(dlogd)O(dlogd) without materializing QQ, which accelerates both the projection steps during training and the component extraction and removal at deployment.

Table 3. Basis construction time and peak GPU memory, measured for constructing one basis per adapted module across all 64 modules. Speedup and memory reduction are relative to Gaussian QR.

DimensionMethodConstruction time (ms)Peak GPU memory (MiB)
4096Gaussian QR6714.05208.6
Hadamard131.44152.1
Speedup / reduction≈51.08×27.1% lower
8192Gaussian QR28084.30792.7
Hadamard472.05584.1
Speedup / reduction≈59.49×26.3% lower
AuthorsOpenReview ↗

Responses to Reviewer JaXf

Comment 3: Can you report the full cross-task impact of component removal and reweighting to assess editing selectivity?

Comment 4: How does performance evolve when new tasks are added to reserved capacity, and how would subspace sharing affect exact recovery?

Response:

1. Impact of component addition, removal, and reweighting.

We agree that removing a task update can affect predictions on retained tasks. KnOTS [3] gives a simple example in which merging two orthogonal task vectors changes the predictions of a nonlinear network, and OSRM [4] explains how this arises within a single layer: an update learned for one task acts on another task's hidden features, shifts the layer output, and thereby affects the final prediction. Since such coupling persists even when the parameter-space edit is exact, we measure exactness and prediction change separately.

Starting from a 19-task model, we added five independently trained updates in reserved orthogonal subspaces and then removed them in reverse order, each time projecting the added component out of the current merged update.

Table 4. Sequential addition of five tasks using reserved orthogonal capacity. Mean accuracy is computed on the original 19 tasks only.

Tasks in MergeTask AddedMean Acc. of the 19 Original Tasks (%)Added Task Acc. (%) Before → After
19—57.649—
20CIFAR-1057.61592.4 → 94.8 (+2.4)
21MNIST57.57683.0 → 91.9 (+8.9)
22GTSRB57.50670.9 → 78.8 (+7.9)
23Food-10157.42083.7 → 85.9 (+2.2)
24Stanford Cars57.35367.1 → 71.2 (+4.1)

Table 5. Removal of the five added tasks in reverse order. Mean accuracy is computed on the original 19 tasks only.

Tasks in MergeTask RemovedMean Acc. of the 19 Original Tasks (%)Removed Task Acc. (%) Before → After
24—57.353—
23Stanford Cars57.42071.2 → 67.1 (-4.1)
22Food-10157.50685.9 → 83.7 (-2.2)
21GTSRB57.57678.8 → 70.9 (-7.9)
20MNIST57.61591.9 → 83.0 (-8.9)
19CIFAR-1057.64994.8 → 92.4 (-2.4)

For the five added tasks, removing a component lowers that task's own accuracy by 5.1 percentage points on average, while every other task changes by less than 0.1 percentage points, so a removal is localized to the removed task. Adding the five tasks costs the original 19 tasks 0.296 percentage points in mean accuracy, and reverse-order removal restores it exactly: at each step from 24 back to 19 tasks, predictions agree on 100% of the evaluated samples with the checkpoint holding the same task set during forward addition.

Table 6. Mean accuracy on the original 19 tasks and on the five added tasks for a 24-task merge under two coefficient settings. Coefficients are set per task as λ = T^(−0.3) .

Merge coefficientMean Acc. of the 19 Original Tasks (%)Mean Acc. of the 5 Added Tasks (%)
24^(-0.3)57.3584.52
19^(-0.3)56.8383.74

The two rows differ only in the coefficient applied to every task component, so the comparison isolates the effect of reweighting itself. At a 24-task merge, λ = T^(−0.3) weights each component at 0.4134 instead of 0.3854, scaling the merged update up by 7.3%, and both groups degrade. A coefficient sized for 19 tasks therefore over-scales a merge of 24. The coefficient cannot be held fixed as the task set changes, and the T^(-0.3) schedule supplies the required scaling.

References.

[3] Stoica et al. Model Merging with SVD to Tie the Knots. ICLR 2025.

[4] Zhang and Zhou. Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging. ACL 2025.

Reviewer_miiBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes H-OM, a LoRA-based model merging method that aims to support independent task adaptation and the recovery of individual task updates after merging. The method preassigns mutually orthogonal subspaces to different tasks and uses randomized Hadamard bases to construct these subspaces. The authors also introduce a Coordinate-Exposure Uniformity metric to guide subspace selection. Based on the orthogonal structure, the method allows individual task updates to be recovered and modified through projection under linear merging, without retaining the original adapters. The authors evaluate the method on vision and language benchmarks, comparing it with existing merging approaches and conducting additional experiments on subspace construction, parameter efficiency, and task-update recovery.

Critical strengths
  1. The authors integrate preallocated orthogonal task subspaces, LoRA-based adaptation, and task-update recovery into a relatively unified framework.
  2. The paper compares the proposed method with various model merging approaches on both vision and language tasks, although its generalizability and practical value in real-world deployment still require further validation.
Critical weaknesses
  1. Insufficient distinction between the proposed method and existing work. The main ideas of H-OM, including orthogonal subspace constraints for LoRA, mitigating task interference, and editing merged models, have been explored in prior studies. O-LoRA[1] and OSRM[2] have investigated orthogonal subspace learning, with OSRM specifically constraining LoRA subspaces before fine-tuning for subsequent merging. SLAO[3] and SOS-LoRA[4] further explore orthogonal structures in LoRA adaptation, while EMR-Merging[5] investigates task-specific information recovery and editing. Although these methods differ from H-OM in their implementations, the paper does not sufficiently clarify its substantive novelty beyond combining existing ideas. In particular, the exact recovery guarantee follows directly from the orthogonality of the predefined subspaces, making the theoretical contribution less convincing. [1] Orthogonal Subspace Learning for Language Model Continual Learning (EMNLP 2023) [2] Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging (ACL 2025) [3] Merge before Forget: A Single LoRA Continual Learning via Continual Merging (ICLR 2026) [4] SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling (ACL 2026) [5] EMR-Merging: Tuning-Free High-Performance Model Merging (NeurIPS 2024)

  2. Insufficient justification for CEU and the Hadamard-based subspace construction. CEU is essentially an application of Jain's fairness index to existing subspace leverage scores, while the maximum CEU achieved by Hadamard bases follows from their established mathematical properties. Moreover, uniform coordinate exposure does not necessarily imply better gradient retention or task adaptation, as also acknowledged in Appendix A.1. The experiments mainly demonstrate a correlation between CEU and merging accuracy rather than establishing its independent contribution. In addition, Table 14 shows only a 0.44-point improvement over Gaussian QR, raising questions about the necessity of the specific Hadamard construction.

  3. Limited validation of practical motivation and general applicability. The paper emphasizes recovering and editing task updates without retaining the original LoRA adapters, but does not demonstrate why this is preferable to simply storing these lightweight adapters. The FP32 serialization-and-reloading experiment verifies numerical recoverability without evaluating practical storage or computational costs. Moreover, exact recovery is guaranteed only for linear merging, whereas several other merging operators are also evaluated. The fixed orthogonal subspaces additionally limit task adaptation capacity, as reflected in the lower single-task accuracy reported in Table 16. These limitations leave the practical benefits and broader applicability of the method insufficiently established.

Questions for authors
  1. Novelty: What are the key technical contributions of H-OM beyond existing orthogonal subspace methods such as O-LoRA and OSRM, particularly given that the exact recovery property follows directly from orthogonal projection?
  2. Necessity of CEU: What evidence supports CEU as the key factor driving improved merging performance, rather than other properties of the subspace construction? Given the small performance gap between Hadamard and Gaussian QR, why is the Hadamard construction necessary?
  3. Practical significance: In what deployment scenarios is recovering task updates from a merged model preferable to retaining the original lightweight LoRA adapters? Can the authors clarify the practical benefits and costs, especially considering the reduced single-task accuracy and the restriction of exact recovery to linear merging? Our current assessment is primarily limited by concerns about the method's novelty, the justification for CEU, and its practical significance. We are open to reconsidering our assessment if the authors can provide convincing clarifications or additional evidence addressing these concerns.
7 comments
AuthorsOpenReview ↗

Responses to Reviewer miiB

We are grateful to the reviewer for a careful reading and thoughtful suggestions, which have tangibly improved the clarity of our exposition and the breadth of our empirical evidence. We respond to each point in turn below.

Comment 1: Novelty: What are the key technical contributions of H-OM beyond existing orthogonal subspace methods such as O-LoRA and OSRM, particularly given that the exact recovery property follows directly from orthogonal projection?

Response:

We apologize for the ambiguity in our original presentation, which may have conveyed that exact recovery follows from orthogonal projection alone. It does not: orthogonal projection only reads out a structure that already exists, and in H-OM that structure comes from the parameterization rather than from the projection itself. Every task update is trained inside a subspace assigned before training and held fixed, so each task owns a preassigned orthogonal basis whose projection isolates its contribution. What makes recovery exact is therefore not the orthogonality of the projection but the fact that this orthogonality is imposed during training, which is why orthogonality alone is necessary but not sufficient.

The related methods provide no such preassigned basis. OSRM[1] constrains input directions derived from other tasks' features, SOS-LoRA[2] imposes orthogonality among experts inside a single adapter, SLAO[3] continually folds incoming tasks into one accumulating adapter, and O-LoRA[4] learns its subspaces sequentially. None of them leaves a per-task orthogonal basis to project onto, and EMR-Merging[5] reconstructs approximately through a mask. H-OM is the only method in our comparison that guarantees exact recovery without retaining the original adapters, and its concrete application scenarios of this maintenance are given in our response to Comment 3.

References.

[1] Zhang and Zhou. Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging. ACL 2025.

[2] Chang et al. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling. ACL 2026.

[3] Qiao and Mahdavi. Merge before Forget: A Single LoRA Continual Learning via Continual Merging. ICLR 2026.

[4] Wang et al. Orthogonal Subspace Learning for Language Model Continual Learning. Findings of EMNLP 2023.

[5] Huang et al. EMR-Merging: Tuning-Free High-Performance Model Merging. NeurIPS 2024.

AuthorsOpenReview ↗

Responses to Reviewer miiB

Comment 2: Necessity of CEU: What evidence supports CEU as the key factor driving improved merging performance, rather than other properties of the subspace construction? Given the small performance gap between Hadamard and Gaussian QR, why is the Hadamard construction necessary?

Response:

1. Necessity of CEU? To examine how coordinate exposure relates to accuracy, we built a controlled family of bases. Starting from the Hadamard dictionary, we linearly interpolate it with the identity matrix at different mixing ratios and orthonormalize the result by QR, which yields orthonormal bases spanning CEU from 0.20 to 0.80. We compare these bases with a Gaussian QR basis, a real Fourier basis, and Hadamard on both merged accuracy and single-task accuracy.

Table 1. Effect of CEU on single-task and merged accuracy on VTAB-1k using ViT-B/16.

BasisCEU ↑Single-task Acc. (%) ↑Merged Acc. (%) ↑
Controlled0.2061.89 ± 1.2353.86 ± 1.15
Controlled0.4062.31 ± 1.0854.46 ± 0.96
Controlled0.6063.08 ± 0.8555.14 ± 0.79
Controlled0.8064.02 ± 0.7656.47 ± 0.72
Gaussian QR0.9564.87 ± 0.6957.21 ± 0.70
Real Fourier1.0065.21 ± 0.6157.61 ± 0.66
Hadamard1.0065.33± 0.5857.85 ± 0.64

Raising CEU from 0.20 to 0.80 along the controlled bases raises single-task accuracy from 61.89% to 64.02% and merged accuracy from 53.86% to 56.47%, so both improve as coordinate exposure becomes more uniform. At CEU at or near 1, the single-task accuracies converge to within 0.46 points of one another (64.87, 65.21, and 65.33), while the merged accuracies still span 0.64 points (57.21, 57.61, and 57.85). CEU therefore accounts for the single-task trend, whereas merged accuracy at CEU = 1 still depends on the construction. The real Fourier and Hadamard bases both attain CEU = 1 and yield similar merged accuracies, 57.61% and 57.85%, which shows that uniform coordinate exposure and comparable merging performance can be achieved with distinct orthogonal constructions.

2. Why choose Hadamard? Although Gaussian QR reaches CEU 0.950.95 and achieves accuracy close to Hadamard , we choose Hadamard for two reasons. First, it guarantees CEU=1\mathrm{CEU}=1 by construction. Second, it avoids dense O(d3)O(d^3) QR factorization: a Hadamard dictionary is generated in O(d2)O(d^2) and supports fast Walsh--Hadamard projection without explicitly materializing QQ. Table 2 reports measured speedups and peak-memory reductions.

Table 2. Basis construction time and peak GPU memory, measured for constructing one basis per adapted module across all 64 modules. Speedup and memory reduction are relative to Gaussian QR.

DimensionMethodConstruction time (ms)Peak GPU memory (MiB)
4096Gaussian QR6714.05208.6
Hadamard131.44152.1
Speedup / reduction≈51.08×27.1% lower
8192Gaussian QR28084.30792.7
Hadamard472.05584.1
Speedup / reduction≈59.49×26.3% lower

We choose Hadamard because it guarantees CEU=1\mathrm{CEU}=1 for every valid column assignment and avoids dense QR factorization. Its structure also supports fast projection for task-component extraction and editing, as measured in our response to Comment 3.

AuthorsOpenReview ↗

Responses to Reviewer miiB

Comment 3: Practical significance: In what deployment scenarios is recovering task updates from a merged model preferable to retaining the original lightweight LoRA adapters? Can the authors clarify the practical benefits and costs, especially considering the reduced single-task accuracy and the restriction of exact recovery to linear merging?

Response:

1. Deployment scenarios.

Retaining the adapters is the simpler route as long as they exist and may be kept: the merge can be recomputed from them whenever needed, and no recovery mechanism is required. Recovery becomes preferable when that premise fails, which happens in three situations. A consortium of hospitals illustrates all three: each hospital trains on its own data and independently manages its checkpoints and adapters, while one shared model serves administrative tasks such as routing referral forms and classifying discharge documents.

  1. Separation between the task owner and the deployer. An adapter is produced by the hospital that holds the corresponding data, while the model is deployed and maintained centrally or by another party. After the task has been handed over, the adapter may no longer be accessible, so a maintenance request arriving later cannot be served by retrieving it. Because the checkpoint carries the decomposition together with the subspace metadata, the corresponding component can still be recovered and edited.

  2. A mismatch between the rate at which tasks change and the serving life of the model. A hospital may revise its referral workflow while the model continues to answer unchanged queries. Retaining adapters would mean keeping every earlier version and re-merging on each revision; with H-OM the replacement is trained in the same subspace and the old component is replaced in place, so no earlier version has to be kept.

  3. Growth in the number of tasks. Retaining adapters scales with the task count and the model size (see Response: 2. Storage cost of retaining LoRA adapters), so the stored set grows with every task that is added, whereas the checkpoint does not: its size is independent of how many tasks it serves.

We also note that the linear merge is a defining property of H-OM. The merged model is a coefficient-weighted sum of task updates confined to fixed subspaces, so every maintenance operation is exactly the addition or subtraction of a known component, or a change of its coefficient.

Reviewer_NV3VOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

H-OM constructs mutually orthogonal LoRA output subspaces using randomized Hadamard basis vectors, guided by Coordinate-Exposure Uniformity (CEU) for effective task adaptation. This enables parallel training, interference-free merging, and flexible task removal or reweighting without retaining the original adapters.

Critical strengths

The paper proposes an intriguing approach to model merging: assigning orthogonal subspaces to different tasks prior to training. This enables independent, parallel training for each task while minimizing parameter interference during the merging process.

The recoverability of task-specific updates is a particularly valuable design feature. Even after multiple LoRA adapters have been merged into a single model, individual task updates can be recovered, removed, or reweighted via orthogonal projection—without the need to retain the original adapter modules. This facilitates dynamic editing of the model post-merging.

Critical weaknesses

1.The visual experiments are limited to a single backbone, CLIP ViT-B/32. Experiments on visual models of different scales are missing.

2.The method requires fixed, mutually orthogonal subspaces to be allocated to different tasks before training, which is constrained by the output dimension of the model. As the number of tasks increases, the available subspace for each task becomes smaller, potentially limiting the scalability of the method.

3.Although the method guarantees orthogonality between task updates in the parameter space, this does not necessarily imply functional independence between tasks. The experiments in the appendix show that removing one task's update can still lead to noticeable performance changes in other tasks.

4.Exact task-update recovery relies on linear merging. Although the paper also evaluates TIES, DARE, and other merging methods, it does not investigate whether task updates can still be recovered under these methods.

Questions for authors

Why are the visual experiments conducted only on CLIP ViT-B/32? Other model merging studies also commonly use ViT-B/16 and ViT-L/14. I suggest including experiments on these backbones to further validate the generalizability of the method.

5 comments
AuthorsOpenReview ↗

Responses to Reviewer NV3V

We are grateful to the reviewer for a careful reading and thoughtful suggestions, which have tangibly improved the clarity of our exposition and the breadth of our empirical evidence. We respond to each point in turn below.

Comment 1: The visual experiments are limited to a single backbone, CLIP ViT-B/32. Experiments on visual models of different scales are missing.

Response:

Following the reviewer's suggestion, we have conducted additional experiments on all 19 VTAB-1k tasks using CLIP ViT-B/16 and ViT-L/14.

Table 1. Merged accuracy on all 19 VTAB-1k tasks using CLIP ViT-B/16. Entries report mean accuracy (%) ± standard deviation across three random seeds.

MethodNatural (7)Specialized (4)Structured (8)Overall (19)
Task Arithmetic78.09 ± 0.1878.46 ± 0.3523.12 ± 0.5555.02 ± 0.22
DC-Merge78.86 ± 0.1980.45 ± 2.4925.62 ± 0.8856.78 ± 0.37
ESM78.94 ± 0.6180.10 ± 2.9324.80 ± 0.9056.39 ± 0.47
Gaussian QR77.51 ± 0.2580.07 ± 1.2028.01 ± 1.3057.21 ± 0.70
H-OM78.15 ± 0.2180.72 ± 1.1328.65 ± 1.2257.85 ± 0.64
H-OM (shared)79.21 ± 0.1882.65 ± 1.0529.50 ± 1.1559.01 ± 0.58

H-OM achieves 57.85% overall merged accuracy on ViT-B/16, exceeding DC-Merge [1], ESM [2], and Task Arithmetic [3] by 1.07, 1.46, and 2.83 percentage points, respectively.

Table 2. Overall merged accuracy (%) across CLIP backbones on all 19 VTAB-1k tasks. The advantage is H-OM over Task Arithmetic in percentage points.

BackboneH-OMTask ArithmeticAdvantage
ViT-B/1657.85 ± 0.6455.02 ± 0.22+2.83 pp
ViT-L/1461.75 ± 0.6257.28 ± 0.31+4.47 pp

The same conclusion holds on the larger ViT-L/14 backbone. As summarized in Table 2, H-OM reaches 61.75% overall accuracy against 57.28% for Task Arithmetic, an advantage of 4.47 percentage points. The gain is larger than the 2.83 points observed on ViT-B/16 and grows with backbone capacity.

References.

[1] Zhang et al. DC-Merge: Improving Model Merging with Directional Consistency. CVPR 2026.

[2] Li et al. Model Merging in the Essential Subspace. CVPR 2026.

[3] Ilharco et al. Editing Models with Task Arithmetic. ICLR 2023.

AuthorsOpenReview ↗

Responses to Reviewer NV3V

Comment 2. The method requires fixed, mutually orthogonal subspaces to be allocated to different tasks before training, which is constrained by the output dimension of the model. As the number of tasks increases, the available subspace for each task becomes smaller, potentially limiting the scalability of the method.

Response.

We agree that the output dimension limits the number of orthogonal directions available in each adapted layer. We evaluate two ways of managing this constraint: adjusting the subspace width per task, and sharing one subspace among tasks.

1.Select directions independently for each task.

Each task selects directions using only its own trained checkpoint. Writing the update as ΔWi=QiθiAi\Delta W_i=Q_i\theta_i A_i and setting Zi=θiAiZ_i=\theta_i A_i, the kk-th row of ZiZ_i holds the coefficients along the kk-th assigned Hadamard direction, and because the columns of QiQ_i are orthonormal the squared norm of that row is the energy the direction carries. We rank directions by this energy, retain the smallest set whose cumulative energy reaches a prescribed fraction of ∣ΔWi∣F2\\|\Delta W_i\\|_F^2, drop the unused columns of QiQ_i and the matching rows of θi\theta_i, and merge the pruned updates with the original coefficients. The retained directions stay disjoint across tasks, so projection still recovers each pruned weighted update.

Table 3. Task-local subspace pruning on ViT-B/16 across 19 VTAB-1k tasks. Reduction is measured relative to the unpruned merge.

Energy thresholdRetained directionsReductionMerged accuracy (%)
100%7600.00%57.846
95%61119.61%57.850
90%52730.66%57.598
85%46838.42%57.893
80%41745.13%57.813

Pruning 38% to 45% of the directions keeps merged accuracy within 0.25 percentage points of the unpruned merge, and at the 85% threshold accuracy is marginally higher (57.893% against 57.846%). The discarded columns remain unused by the task and stay orthogonal to every retained direction, so they can be assigned to tasks added later.

2.Shared subspace.

On ViT-B/16, we divided the 19 tasks into nine pairs and one singleton. Each group used one width-40, rank-8 adapter trained jointly on its tasks, with a separate classifier head for each task. We linearly merged the ten group adapters into a single backbone and evaluated it on all 19 tasks using their group-trained heads.

As shown in the H-OM (shared) row of Table 1, the merged model achieved 59.01±0.5859.01\pm0.58% overall accuracy, 1.16 percentage points above the task-wise c=40c=40 configuration. Sharing reduced the number of allocated directions per adapted layer from 760 to 400. Under linear merging, each group update remains independently recoverable and editable, with the tasks within that group edited together.

AuthorsOpenReview ↗

Responses to Reviewer NV3V

Comment 3: Although the method guarantees orthogonality between task updates in the parameter space, this does not necessarily imply functional independence between tasks. The experiments in the appendix show that removing one task's update can still lead to noticeable performance changes in other tasks.

Response.

We agree that removing a task update can affect predictions on retained tasks. Knots[4] give a simple example in which merging two orthogonal task vectors changes the predictions of a nonlinear network. OSRM[5] illustrate how this can arise within a single layer. An update learned for one task can act on another task's hidden features and shift the layer output, which can in turn affect the final prediction.

Reversibility under repeated addition and removal.

We designed this experiment to measure two things: whether adding and removing task components is exact, and what modifying the task set costs the existing 19 tasks. Starting from a 19-task model, we added five independently trained updates in reserved orthogonal subspaces and then removed them in reverse order, each time by projecting the added component out of the current merged update.

Table 4. Mean accuracy on the original 19 tasks before and after adding five tasks and after removing them in reverse order. Mean accuracy is computed on the original 19 tasks only.

StageMean accuracy on the original 19 tasks (%)
Initial 19-task model57.649
After adding five tasks57.353
After removing the five added tasks57.649

As shown in Table 4,expanding to 24 tasks lowers the mean accuracy of the original 19 tasks from 57.649% to 57.353%, 0.296 percentage points in total and about 0.06 per added task, and removal is exact: at each rollback stage from 24 to 19 tasks, predictions on all 17,711 test samples agree with those of the forward checkpoint holding the same task set and mean accuracy returns to 57.649%.

References.

[4] Stoica et al. Model Merging with SVD to Tie the Knots. ICLR 2025.

[5] Zhang and Zhou. Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging. ACL 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.