- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MergeCodec, a model merging framework based on a learnable encoder–decoder architecture. Unlike traditional methods that directly perform task-vector weighting, pruning, or conflict resolution in parameter space, MergeCodec first compresses the task vectors of individual experts into a low-dimensional latent space, aggregates them through summation, and then uses a shared decoder to generate the merged model update.
The authors employ end-to-end supervised training over expert subsets of varying sizes and compositions, enabling the trained merging module to generalize to new expert combinations without requiring additional task data, gradient-based optimization, or combination-specific coefficient search.
Critical strengths
Existing model merging methods typically rely on handcrafted merging rules, coefficient tuning, or separate optimization for different expert combinations. This paper aims to learn a reusable merging function that can be applied across different expert combinations, thereby reducing the computational overhead of subsequent merges. This research direction is potentially valuable, particularly in scenarios where different combinations of experts need to be merged frequently.
On ViT-B/32, MergeCodec achieves average accuracies of 89.13%, 88.46%, and 88.73% when merging 8, 14, and 20 experts, respectively, with relatively small gaps compared to the corresponding MTL models. Notably, although the model is never trained on 14-expert subsets, it still achieves strong performance at this subset size, suggesting that the learned merging function exhibits a certain degree of generalization across different numbers of experts.
Critical weaknesses
MergeCodec uses labeled data from all training tasks to optimize its encoder and decoder through an end-to-end classification loss. This training process itself allows the model to learn how to adjust model parameters to improve multi-task performance. Therefore, substantial performance gains may be achievable even without the proposed compressed representations or two-sided nonlinear mappings. Furthermore, the linear variant in Table 3 already achieves strong performance. On ViT-B/32, the linear Codec achieves accuracies of 88.60%, 87.68%, and 88.02% when merging 8, 14, and 20 experts, respectively, while the full model provides only modest additional improvements of 0.53, 0.78, and 0.71 percentage points. Moreover, the Encoder-only variant even outperforms the full Two-sided variant in the single-expert and 8-expert settings.
The paper compares MergeCodec primarily against TA, TSV-M, WUDI, WUDI+ProbSurgery, and MTL. However, these methods differ substantially in their access to supervision and their optimization objectives. MergeCodec has access to labeled data from the training tasks and performs 4,000 steps of end-to-end optimization during training. In contrast, WUDI does not require additional task data, TA and TSV-M primarily perform merging directly in parameter space, and ProbSurgery uses unlabeled images for adaptation.
Questions for authors
Compared with existing methods that also learn reusable merging mappings, such as HyperFix, does MergeCodec still demonstrate significant advantages when training data, model parameter counts, and computational budgets are matched?
The authors should also include Closed-Form Spectral Regularization for Multi-Task Model Merging as an additional baseline in the experimental comparisons and report its performance in the main results tables to provide a more comprehensive evaluation of MergeCodec against existing model merging methods.
4 comments
part2
Q1. Direct comparison with HyperFix and training-resource accounting
We added an independent HyperFix reimplementation extended to the same twenty-expert B/32 bank. Both methods share the base model, expert checkpoints, frozen heads, preprocessing, and the eleven main-table evaluation combinations.
| Evaluated K | HyperFix | MergeCodec | Gain (pp) |
|---|---|---|---|
| 8 | 84.920% | 89.135% | +4.215 |
| 14 | 83.760% | 88.458% | +4.698 |
| 20 | 83.786% | 88.730% | +4.944 |
MergeCodec leads on all eleven combinations and 122 of 130 task–combination entries. HyperFix improves over mean merging by 19.220–22.683 points, providing an effective learned reference.
To make the comparison assessable, we report the actual training configurations:
| Training property | HyperFix | MergeCodec |
|---|---|---|
| Supervision | Expert-logit distillation | Labeled task loss |
| Trainable parameters | 349.97M | 114.75M |
| Updates | 13,500 | 4,000 |
| Training cardinalities | 1–6 | 4, 8, 12, 16, 20 |
| Task batches | 47,250 | 48,000 |
| Image presentations, including repetition | 5,857,945 | 1,534,331 |
Both use BF16 training and FP32 evaluation. These are method-specific training recipes, with the differences in data access, capacity, and training exposure disclosed explicitly. The HyperFix results establish a substantial performance advantage in this shared expert-bank setting; the Linear/Encoder-only/Two-sided experiment provides the controlled evidence that nonlinearity adds value under matched supervision, update counts, and closely matched parameter capacity.
Q2. Closed-Form Spectral Regularization for Multi-Task Model Merging
We have completed the requested comparison using both SWUDI and SWUDI-A, and will add them as columns in the revised ViT-B/32 main results table.
We use the official implementation with the same expert checkpoints, frozen heads, test splits, and eleven evaluation subsets. SWUDI uses the authors’ published B/32 TA8 settings, and , transferred unchanged to K=14/20. SWUDI-A uses the adaptive-rank variant. Both use scale=1, without additional parameter search.
| K | WUDI | WUDI + ProbSurgery | SWUDI | SWUDI-A | MergeCodec |
|---|---|---|---|---|---|
| 8 | 84.073 | 88.521 | 84.561 | 84.543 | 89.135 |
| 14 | 77.079 | 86.645 | 79.650 | 79.674 | 88.458 |
| 20 | 66.573 | 84.407 | 75.598 | 75.611 | 88.730 |
Values are macro accuracy percentages; K=8/14 average five combinations each.
The spectral methods substantially improve over WUDI, including an approximately 9-point improvement at K=20. MergeCodec further exceeds the stronger spectral result by 4.574/8.784/13.119 points at K=8/14/20. It leads over both variants on all eleven combination averages, and on all twenty individual tasks in Full20.
These additions broaden the comparison to effective closed-form spectral methods and a reusable learned merger. Their respective training and adaptation requirements will accompany the results, while the controlled architectural experiments establish the incremental contribution of MergeCodec’s nonlinear composition.
Thank you for the detailed response and additional experiments. The controlled architectural ablations, unseen-expert evaluations, and compression analysis adequately address my main concerns. The authors have clarified these differences. Overall, my concerns have been largely addressed, and I am satisfied with the response.
Thank you for your thoughtful follow-up. We are glad that the controlled architectural ablations, unseen-expert evaluations, and compression analysis have addressed your main concerns. We will incorporate these results and clarifications into the revised manuscript.
Given your updated assessment, we would be grateful if you could also reconsider your overall rating in light of the additional evidence. Thank you again for your constructive feedback, which has helped us strengthen the paper.
part1
Thank you for recognizing the value of learning a reusable merging function and its generalization to untrained subset sizes. Your main concern is whether the proposed representation and nonlinear mappings contribute beyond supervised training itself. We address this with a controlled, multi-seed architectural comparison, a direct HyperFix comparison, and both variants of the requested spectral-regularization baseline.
Contribution of nonlinearity under matched supervision
Supervised training is an intentional component of MergeCodec: it trains a shared operator to compose supplied expert updates across different subsets. To evaluate the additional contribution of nonlinearity, we compare Linear, Encoder-only, and Two-sided codecs under the same supervised training protocol.
The strongest additional evidence comes from strict unseen-expert transfer. We train on sixteen B/32 experts while excluding SVHN, Flowers102, PCAM, and Rendered-SST2 from SVD initialization, rank allocation, and codec optimization, including their weights, images, and labels. The four experts are supplied only after the codec is frozen.
The three architectures share the same SVD initialization, layer ranks, code dimensions, optimizer, 4,000-update budget, and paired subset/image sampling within each seed. Their shared parameter counts are 115.086M, 115.143M, and 115.200M, respectively—a difference below 0.1%.
Across training seeds 42, 43, and 44:
Values are mean accuracy ± sample SD. Two-sided > Encoder-only > Linear holds at every evaluated K in all three seeds. The complete codec improves over Linear by 2.324/2.285/2.298 percentage points at K=1/2/4.
All six paired, pointwise 95% t intervals comparing Two-sided against either alternative are positive. At K=4, the gain over Linear is 2.298 points [2.169, 2.427], and the additional gain over Encoder-only is 0.683 points [0.557, 0.809]. Because these variants receive the same supervision and training schedule, these results provide direct evidence for an architectural benefit in transferring composition to unseen experts.
The nonlinear decoder also has a specific functional role. A linear decoder satisfies
D(t∑zt)=t∑D(zt),whereas a nonlinear decoder can respond to the aggregate code, allowing the effect of an expert to depend on the other experts included in the composition. The controlled transfer results demonstrate a practical benefit from this additional expressiveness.
We also expanded the seen-expert evaluation to nine untrained cardinalities, K={3,5,7,9,11,13,15,17,19}, with five combinations each. Two-sided exceeds Linear at every cardinality mean, by 0.140–0.783 points. Encoder-only remains strong for smaller subsets, while Two-sided leads on all twenty combinations at K=13,15,17,19, supporting the value of nonlinear decoding for these larger compositions.
Role of compression and interpretation of supervision
Compression provides a controllable representation and storage budget. Our expanded B/32 evaluation shows that effective composition persists with substantially smaller cached expert libraries:
These totals include the base, all task heads, decoder, and twenty expert codes. The corresponding base-plus-raw-task-vector library occupies 6.845 GiB. Thus, compression offers a measured storage–accuracy tradeoff while preserving the ability to construct different expert combinations.
The learned behavior also follows the supplied expert vectors. In the original eight-expert input experiment, the codec achieves 89.31% on the included tasks. On the twelve omitted tasks—which were nevertheless available during codec training—it achieves 59.91%, close to the base model’s 60.70%. Together with strict unseen-expert transfer, this supports learning to compose the supplied capabilities.
We will make supervision and reuse requirements explicit alongside the comparisons. In our protocol, TA and TSV-M use test-selected coefficients; WUDI operates on task vectors; ProbSurgery additionally uses unlabeled test images; and MergeCodec uses labeled training data to learn its reusable operator. After training, MergeCodec uses λ=1 without task examples, coefficient search, or gradient updates for each new merge. Architectural attribution is grounded in the matched-supervision experiments above.