- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Hadamard Orthogonal Merging (H-OM), which assigns each task a mutually orthogonal output subspace before task-specific fine-tuning. Different tasks can be trained independently and then linearly merged. Because the task subspaces are mutually orthogonal, each weighted task component in the merged model can be recovered by projection, enabling component deletion, replacement, and reweighting without retaining the original task adapters. The authors further propose Coordinate-Exposure Uniformity (CEU) to measure the uniformity of coordinate exposure induced by a subspace, and use a randomized Hadamard basis to achieve maximum CEU. Experiments on VTAB-1k and NLP-15 demonstrate improved merged accuracy with fewer trainable parameters than standard LoRA, as well as component recovery after serialization and reloading.
Critical strengths
-
The method is simple, and the recovery property is clearly stated and correct.
-
The merging performance gains are reasonably consistent under the evaluated protocols.
-
The appendices provide a relatively thorough account of key limitations and the practical recovery workflow.
Critical weaknesses
-
Limited applicability and a substantial adaptation cost. H-OM requires coordinated subspace allocation before fine-tuning and therefore cannot directly merge arbitrary existing adapters. This restricts its applicability. Moreover, Table 16 shows that its single-task accuracy drops from 68.36% with standard LoRA to 62.98%, a substantial loss of 5.38 percentage points.
-
Limited theoretical scope and insufficient attribution to CEU. Maximizing CEU equalizes coordinate leverage scores but does not guarantee sufficient retention of actual task gradients or effective adaptation. The CEU ablation also changes subspace orientation and thus gradient alignment, establishing correlation within the tested construction family rather than isolating CEU’s contribution. Furthermore, Hadamard improves over Gaussian QR by only 0.44 percentage points.
-
Parameter separability provides limited evidence of selective task editing. Proposition 1 guarantees component recovery and removal, but not prediction-level independence. Removing a component changes retained-task accuracy by up to 6.50 percentage points, while the removed task loses only 4.80 points on average. These results limit the evidence for selective behavioral editing, although parameter-level editing is correctly established.
-
Insufficient validation of continued task addition. The default allocations nearly exhaust the available capacity: 760/768 directions on VTAB and 4095/4096 on LLaMA, leaving insufficient room for another task of the same subspace width. Table 17 evaluates reserving capacity but not performance after subsequent task additions. Moreover, the suggested subspace sharing would invalidate the orthogonality assumption underlying Proposition 1, requiring clarification of its implications for exact recovery.
-
Limited baseline coverage and differences in training assumptions. Despite using multiple merge operators, Table 1 includes only standard LoRA, KnOTS, and MergOPT. KnOTS operates on independently trained adapters, whereas H-OM controls their training subspaces in advance; the comparison therefore requires qualification regarding these different assumptions. Comparisons with additional representative methods, such as DC-Merge [1] and ESM [2], would strengthen the claimed performance advantage.
[1] DC-Merge: Improving Model Merging with Directional Consistency. CVPR, 2026.
[2] Model Merging in the Essential Subspace. CVPR, 2026.
Questions for authors
-
What practical settings justify preallocated subspaces despite the 5.38-point single-task accuracy loss, and how does H-OM compare with joint multi-task training?
-
Can you isolate CEU’s contribution from subspace–gradient alignment and verify the advantage over Gaussian QR across multiple basis seeds?
-
Can you report the full cross-task impact of component removal and reweighting to assess editing selectivity?
-
How does performance evolve when new tasks are added to reserved capacity, and how would subspace sharing affect exact recovery?
-
Can you include DC-Merge and ESM and clarify the different training assumptions in comparisons with post-hoc merging methods?
7 comments
Responses to Reviewer JaXf
Comment 2: Can you isolate CEU’s contribution from subspace–gradient alignment and verify the advantage over Gaussian QR across multiple basis seeds?
Response:
1. CEU’s contribution
To examine how coordinate exposure relates to accuracy, and to separate it from the alignment between task gradients and the assigned subspace, we built a controlled family of bases. Starting from the Hadamard dictionary, we linearly interpolate it with the identity matrix at different mixing ratios and orthonormalize the result by QR, which yields orthonormal bases spanning CEU from 0.20 to 0.80. Every basis in this comparison is fixed before training and independent of task data, so none of them is selected or tuned for alignment with task gradients; within the controlled family all members share the same origin and only the mixing ratio varies, which changes how energy is spread across coordinates while the dictionary family is held fixed. We compare these bases with a Gaussian QR basis, a real Fourier basis, and Hadamard on both merged accuracy and single-task accuracy, each averaged over 10 independent runs.
Table 2. Effect of CEU on single-task and merged accuracy on VTAB-1k using ViT-B/16.
| Basis | CEU ↑ | Single-task Acc. (%) ↑ | Merged Acc. (%) ↑ |
|---|---|---|---|
| Controlled | 0.20 | 61.89 ± 1.23 | 53.86 ± 1.15 |
| Controlled | 0.40 | 62.31 ± 1.08 | 54.46 ± 0.96 |
| Controlled | 0.60 | 63.08 ± 0.85 | 55.14 ± 0.79 |
| Controlled | 0.80 | 64.02 ± 0.76 | 56.47 ± 0.72 |
| Gaussian QR | 0.95 | 64.87 ± 0.69 | 57.21 ± 0.70 |
| Real Fourier | 1.00 | 65.21 ± 0.61 | 57.61 ± 0.66 |
| Hadamard | 1.00 | 65.33 ± 0.58 | 57.85 ± 0.64 |
Raising CEU from 0.20 to 0.80 along the controlled bases raises single-task accuracy from 61.89% to 64.02% and merged accuracy from 53.86% to 56.47%, so both improve as coordinate exposure becomes more uniform. At CEU at or near 1, the single-task accuracies converge to within 0.46 points of one another (64.87, 65.21, and 65.33), while the merged accuracies still span 0.64 points (57.21, 57.61, and 57.85). CEU therefore accounts for the single-task trend, whereas merged accuracy at CEU = 1 still depends on the construction. The real Fourier and Hadamard bases both attain CEU = 1 and yield similar merged accuracies, 57.61% and 57.85%, which shows that uniform coordinate exposure and comparable merging performance can be achieved with distinct orthogonal constructions.
2. Why choose Hadamard? Although Gaussian QR reaches CEU and achieves accuracy close to Hadamard , we choose Hadamard for three reasons. First, it guarantees by construction. Second, it avoids dense QR factorization: a Hadamard dictionary is generated in and supports fast Walsh--Hadamard projection without explicitly materializing . Table 3 reports measured speedups and peak-memory reductions. Third, because the assigned basis is a Hadamard matrix, projection onto it can be computed by the fast Walsh-Hadamard transform in without materializing , which accelerates both the projection steps during training and the component extraction and removal at deployment.
Table 3. Basis construction time and peak GPU memory, measured for constructing one basis per adapted module across all 64 modules. Speedup and memory reduction are relative to Gaussian QR.
| Dimension | Method | Construction time (ms) | Peak GPU memory (MiB) |
|---|---|---|---|
| 4096 | Gaussian QR | 6714.05 | 208.6 |
| Hadamard | 131.44 | 152.1 | |
| Speedup / reduction | ≈51.08× | 27.1% lower | |
| 8192 | Gaussian QR | 28084.30 | 792.7 |
| Hadamard | 472.05 | 584.1 | |
| Speedup / reduction | ≈59.49× | 26.3% lower |
Responses to Reviewer JaXf
Comment 3: Can you report the full cross-task impact of component removal and reweighting to assess editing selectivity?
Comment 4: How does performance evolve when new tasks are added to reserved capacity, and how would subspace sharing affect exact recovery?
Response:
1. Impact of component addition, removal, and reweighting.
We agree that removing a task update can affect predictions on retained tasks. KnOTS [3] gives a simple example in which merging two orthogonal task vectors changes the predictions of a nonlinear network, and OSRM [4] explains how this arises within a single layer: an update learned for one task acts on another task's hidden features, shifts the layer output, and thereby affects the final prediction. Since such coupling persists even when the parameter-space edit is exact, we measure exactness and prediction change separately.
Starting from a 19-task model, we added five independently trained updates in reserved orthogonal subspaces and then removed them in reverse order, each time projecting the added component out of the current merged update.
Table 4. Sequential addition of five tasks using reserved orthogonal capacity. Mean accuracy is computed on the original 19 tasks only.
| Tasks in Merge | Task Added | Mean Acc. of the 19 Original Tasks (%) | Added Task Acc. (%) Before → After |
|---|---|---|---|
| 19 | — | 57.649 | — |
| 20 | CIFAR-10 | 57.615 | 92.4 → 94.8 (+2.4) |
| 21 | MNIST | 57.576 | 83.0 → 91.9 (+8.9) |
| 22 | GTSRB | 57.506 | 70.9 → 78.8 (+7.9) |
| 23 | Food-101 | 57.420 | 83.7 → 85.9 (+2.2) |
| 24 | Stanford Cars | 57.353 | 67.1 → 71.2 (+4.1) |
Table 5. Removal of the five added tasks in reverse order. Mean accuracy is computed on the original 19 tasks only.
| Tasks in Merge | Task Removed | Mean Acc. of the 19 Original Tasks (%) | Removed Task Acc. (%) Before → After |
|---|---|---|---|
| 24 | — | 57.353 | — |
| 23 | Stanford Cars | 57.420 | 71.2 → 67.1 (-4.1) |
| 22 | Food-101 | 57.506 | 85.9 → 83.7 (-2.2) |
| 21 | GTSRB | 57.576 | 78.8 → 70.9 (-7.9) |
| 20 | MNIST | 57.615 | 91.9 → 83.0 (-8.9) |
| 19 | CIFAR-10 | 57.649 | 94.8 → 92.4 (-2.4) |
For the five added tasks, removing a component lowers that task's own accuracy by 5.1 percentage points on average, while every other task changes by less than 0.1 percentage points, so a removal is localized to the removed task. Adding the five tasks costs the original 19 tasks 0.296 percentage points in mean accuracy, and reverse-order removal restores it exactly: at each step from 24 back to 19 tasks, predictions agree on 100% of the evaluated samples with the checkpoint holding the same task set during forward addition.
Table 6. Mean accuracy on the original 19 tasks and on the five added tasks for a 24-task merge under two coefficient settings. Coefficients are set per task as λ = T^(−0.3) .
| Merge coefficient | Mean Acc. of the 19 Original Tasks (%) | Mean Acc. of the 5 Added Tasks (%) |
|---|---|---|
| 24^(-0.3) | 57.35 | 84.52 |
| 19^(-0.3) | 56.83 | 83.74 |
The two rows differ only in the coefficient applied to every task component, so the comparison isolates the effect of reweighting itself. At a 24-task merge, λ = T^(−0.3) weights each component at 0.4134 instead of 0.3854, scaling the merged update up by 7.3%, and both groups degrade. A coefficient sized for 19 tasks therefore over-scales a merge of 24. The coefficient cannot be held fixed as the task set changes, and the T^(-0.3) schedule supplies the required scaling.
References.
[3] Stoica et al. Model Merging with SVD to Tie the Knots. ICLR 2025.
[4] Zhang and Zhou. Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging. ACL 2025.
Responses to Reviewer JaXf
We are grateful to the reviewer for a careful reading and thoughtful suggestions, which have tangibly improved the clarity of our exposition and the breadth of our empirical evidence. We respond to each point in turn below.
Comment 1: Given the 5.38-point drop in single-task accuracy, what practical settings still justify preallocating subspaces? How does H-OM compare with joint multitask training?
Comment 5: Could you include DC-Merge and ESM in the comparison and clarify how H-OM's training assumptions differ from those of post-hoc merging methods?
Response:
We thank the reviewer for asking these significant questions. To clarify H-OM's training assumptions and practical significance, We will address Comments 1 and 5 together.
1. When subspace preallocation is worthwhile.
H-OM's primary goal is to support task addition, removal, and version replacement in a single merged model after release. Exact recovery is the mechanism that enables this maintenance, not a separate predictive route: predictions are always served by the merged model, and a recovered update is used only to edit the merged weights. Preallocation is worthwhile when teams can agree on a shared pretrained base and task subspaces before fine-tuning, and downstream maintainers expect to edit the released model after the original adapters are no longer available. Given the shared base, merged checkpoint, and subspace metadata, maintainers can recover an existing task's weighted update for removal or replacement, and they can add independently trained tasks when unused subspaces are available. The price of this design is that orthogonality constraints during training limit single-task adaptation, a trade-off inherent to orthogonal methods in general. Since the goal of model merging is to obtain a better merged model, this cost should be weighed against the quality of the merge rather than judged in isolation.
2. Accuracy comparison with joint training, DC-Merge, and ESM
We added joint multitask training, DC-Merge, and ESM to the ViT-B/16 evaluation on all 19 VTAB-1k tasks.
Table 1. Merged accuracy on all 19 VTAB-1k tasks using CLIP ViT-B/16. Entries report mean accuracy (%) ± standard deviation across three random seeds.
Joint multitask training attains the highest accuracy in this comparison, 62.69%, 4.84 pp above H-OM. It is an upper bound rather than a competing merge, since it requires all task data to be optimized together. Among the methods that merge independently trained task updates, H-OM exceeds DC-Merge [1] and ESM [2] by 1.07 and 1.46 pp.
The difference in training assumptions is one of timing and coordination. DC-Merge and ESM operate on task models after fine-tuning and require no agreement beforehand, so they can be applied to checkpoints that already exist. H-OM fixes the task subspaces before fine-tuning; after that point each team trains independently on its own data, and the resulting weighted linear merge carries a known task decomposition that maintainers can use to edit the released checkpoint.
References.
[1] Zhang et al. DC-Merge: Improving Model Merging with Directional Consistency. CVPR 2026.
[2] Li et al. Model Merging in the Essential Subspace. CVPR 2026.