acceptodds
Under review as a conference paper at ICLR 2027

MergeCodec: Learning to Compress and Compose Task Vectors

Abstract

Model merging aims to combine experts derived from a shared pretrained model, and yet preserving their capabilities across varying task compositions and numbers of experts remains challenging. Existing methods typically rely on hand-crafted merging rules or combination-specific coefficient optimization. We introduce MergeCodec, which learns a reusable merging map through task supervision. MergeCodec encodes task vectors into compact latent representations, composes selected experts by summing their codes, and decodes the result into a merged model update. Training end-to-end over subsets with varying task identities and cardinalities encourages the latent space to support robust composition. Once trained, the encoder can be reused for arbitrary expert combinations without additional task data, gradient updates, or per-combination calibration. Across 20 tasks and 3 CLIP backbones, a single MergeCodec per backbone successfully merges any number of experts, achieving performance comparable to multi-task models trained separately for each subset. Despite never being trained on single subsets, MergeCodec also largely preserves the accuracy of individual experts. Moreover, MergeCodec generalizes to out-of-distribution settings, successfully merging experts unseen during training, and its generalization improves as the training expert bank expands. These results demonstrate that model merging can be learned as a reusable compositional mapping, yielding task-vector representations that generalize across both novel expert combinations and previously unseen experts.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
model merging, task vectors, representation learning, model compression, multi-task learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_dXYgOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MergeCodec, a model merging framework based on a learnable encoder–decoder architecture. Unlike traditional methods that directly perform task-vector weighting, pruning, or conflict resolution in parameter space, MergeCodec first compresses the task vectors of individual experts into a low-dimensional latent space, aggregates them through summation, and then uses a shared decoder to generate the merged model update.

The authors employ end-to-end supervised training over expert subsets of varying sizes and compositions, enabling the trained merging module to generalize to new expert combinations without requiring additional task data, gradient-based optimization, or combination-specific coefficient search.

Critical strengths

Existing model merging methods typically rely on handcrafted merging rules, coefficient tuning, or separate optimization for different expert combinations. This paper aims to learn a reusable merging function that can be applied across different expert combinations, thereby reducing the computational overhead of subsequent merges. This research direction is potentially valuable, particularly in scenarios where different combinations of experts need to be merged frequently.

On ViT-B/32, MergeCodec achieves average accuracies of 89.13%, 88.46%, and 88.73% when merging 8, 14, and 20 experts, respectively, with relatively small gaps compared to the corresponding MTL models. Notably, although the model is never trained on 14-expert subsets, it still achieves strong performance at this subset size, suggesting that the learned merging function exhibits a certain degree of generalization across different numbers of experts.

Critical weaknesses

MergeCodec uses labeled data from all training tasks to optimize its encoder and decoder through an end-to-end classification loss. This training process itself allows the model to learn how to adjust model parameters to improve multi-task performance. Therefore, substantial performance gains may be achievable even without the proposed compressed representations or two-sided nonlinear mappings. Furthermore, the linear variant in Table 3 already achieves strong performance. On ViT-B/32, the linear Codec achieves accuracies of 88.60%, 87.68%, and 88.02% when merging 8, 14, and 20 experts, respectively, while the full model provides only modest additional improvements of 0.53, 0.78, and 0.71 percentage points. Moreover, the Encoder-only variant even outperforms the full Two-sided variant in the single-expert and 8-expert settings.

The paper compares MergeCodec primarily against TA, TSV-M, WUDI, WUDI+ProbSurgery, and MTL. However, these methods differ substantially in their access to supervision and their optimization objectives. MergeCodec has access to labeled data from the training tasks and performs 4,000 steps of end-to-end optimization during training. In contrast, WUDI does not require additional task data, TA and TSV-M primarily perform merging directly in parameter space, and ProbSurgery uses unlabeled images for adaptation.

Questions for authors

Compared with existing methods that also learn reusable merging mappings, such as HyperFix, does MergeCodec still demonstrate significant advantages when training data, model parameter counts, and computational budgets are matched?

The authors should also include Closed-Form Spectral Regularization for Multi-Task Model Merging as an additional baseline in the experimental comparisons and report its performance in the main results tables to provide a more comprehensive evaluation of MergeCodec against existing model merging methods.

4 comments

Thank you for recognizing the value of learning a reusable merging function and its generalization to untrained subset sizes. Your main concern is whether the proposed representation and nonlinear mappings contribute beyond supervised training itself. We address this with a controlled, multi-seed architectural comparison, a direct HyperFix comparison, and both variants of the requested spectral-regularization baseline.

Contribution of nonlinearity under matched supervision

Supervised training is an intentional component of MergeCodec: it trains a shared operator to compose supplied expert updates across different subsets. To evaluate the additional contribution of nonlinearity, we compare Linear, Encoder-only, and Two-sided codecs under the same supervised training protocol.

The strongest additional evidence comes from strict unseen-expert transfer. We train on sixteen B/32 experts while excluding SVHN, Flowers102, PCAM, and Rendered-SST2 from SVD initialization, rank allocation, and codec optimization, including their weights, images, and labels. The four experts are supplied only after the codec is frozen.

The three architectures share the same SVD initialization, layer ranks, code dimensions, optimizer, 4,000-update budget, and paired subset/image sampling within each seed. Their shared parameter counts are 115.086M, 115.143M, and 115.200M, respectively—a difference below 0.1%.

Across training seeds 42, 43, and 44:

Held-out expert count KLinearEncoder-onlyTwo-sided
183.445 ± 0.12383.854 ± 0.03585.769 ± 0.207
282.493 ± 0.06383.379 ± 0.04684.778 ± 0.156
480.967 ± 0.07382.582 ± 0.11883.265 ± 0.108

Values are mean accuracy ± sample SD. Two-sided > Encoder-only > Linear holds at every evaluated K in all three seeds. The complete codec improves over Linear by 2.324/2.285/2.298 percentage points at K=1/2/4.

All six paired, pointwise 95% t intervals comparing Two-sided against either alternative are positive. At K=4, the gain over Linear is 2.298 points [2.169, 2.427], and the additional gain over Encoder-only is 0.683 points [0.557, 0.809]. Because these variants receive the same supervision and training schedule, these results provide direct evidence for an architectural benefit in transferring composition to unseen experts.

The nonlinear decoder also has a specific functional role. A linear decoder satisfies

D ⁣(∑tzt)=∑tD(zt),D\!\left(\sum_t z_t\right)=\sum_t D(z_t),

whereas a nonlinear decoder can respond to the aggregate code, allowing the effect of an expert to depend on the other experts included in the composition. The controlled transfer results demonstrate a practical benefit from this additional expressiveness.

We also expanded the seen-expert evaluation to nine untrained cardinalities, K={3,5,7,9,11,13,15,17,19}, with five combinations each. Two-sided exceeds Linear at every cardinality mean, by 0.140–0.783 points. Encoder-only remains strong for smaller subsets, while Two-sided leads on all twenty combinations at K=13,15,17,19, supporting the value of nonlinear decoding for these larger compositions.

Role of compression and interpretation of supervision

Compression provides a controllable representation and storage budget. Our expanded B/32 evaluation shows that effective composition persists with substantially smaller cached expert libraries:

ConfigurationComplete cached composition serviceFull20 accuracy
C42.172 GiB88.73%
C160.794 GiB87.98%
C1280.387 GiB85.19%

These totals include the base, all task heads, decoder, and twenty expert codes. The corresponding base-plus-raw-task-vector library occupies 6.845 GiB. Thus, compression offers a measured storage–accuracy tradeoff while preserving the ability to construct different expert combinations.

The learned behavior also follows the supplied expert vectors. In the original eight-expert input experiment, the codec achieves 89.31% on the included tasks. On the twelve omitted tasks—which were nevertheless available during codec training—it achieves 59.91%, close to the base model’s 60.70%. Together with strict unseen-expert transfer, this supports learning to compose the supplied capabilities.

We will make supervision and reuse requirements explicit alongside the comparisons. In our protocol, TA and TSV-M use test-selected coefficients; WUDI operates on task vectors; ProbSurgery additionally uses unlabeled test images; and MergeCodec uses labeled training data to learn its reusable operator. After training, MergeCodec uses λ=1 without task examples, coefficient search, or gradient updates for each new merge. Architectural attribution is grounded in the matched-supervision experiments above.

Q1. Direct comparison with HyperFix and training-resource accounting

We added an independent HyperFix reimplementation extended to the same twenty-expert B/32 bank. Both methods share the base model, expert checkpoints, frozen heads, preprocessing, and the eleven main-table evaluation combinations.

Evaluated KHyperFixMergeCodecGain (pp)
884.920%89.135%+4.215
1483.760%88.458%+4.698
2083.786%88.730%+4.944

MergeCodec leads on all eleven combinations and 122 of 130 task–combination entries. HyperFix improves over mean merging by 19.220–22.683 points, providing an effective learned reference.

To make the comparison assessable, we report the actual training configurations:

Training propertyHyperFixMergeCodec
SupervisionExpert-logit distillationLabeled task loss
Trainable parameters349.97M114.75M
Updates13,5004,000
Training cardinalities1–64, 8, 12, 16, 20
Task batches47,25048,000
Image presentations, including repetition5,857,9451,534,331

Both use BF16 training and FP32 evaluation. These are method-specific training recipes, with the differences in data access, capacity, and training exposure disclosed explicitly. The HyperFix results establish a substantial performance advantage in this shared expert-bank setting; the Linear/Encoder-only/Two-sided experiment provides the controlled evidence that nonlinearity adds value under matched supervision, update counts, and closely matched parameter capacity.

Q2. Closed-Form Spectral Regularization for Multi-Task Model Merging

We have completed the requested comparison using both SWUDI and SWUDI-A, and will add them as columns in the revised ViT-B/32 main results table.

We use the official implementation with the same expert checkpoints, frozen heads, test splits, and eleven evaluation subsets. SWUDI uses the authors’ published B/32 TA8 settings, t=1300t=1300 and r=0.65r=0.65, transferred unchanged to K=14/20. SWUDI-A uses the adaptive-rank variant. Both use scale=1, without additional parameter search.

KWUDIWUDI + ProbSurgerySWUDISWUDI-AMergeCodec
884.07388.52184.56184.54389.135
1477.07986.64579.65079.67488.458
2066.57384.40775.59875.61188.730

Values are macro accuracy percentages; K=8/14 average five combinations each.

The spectral methods substantially improve over WUDI, including an approximately 9-point improvement at K=20. MergeCodec further exceeds the stronger spectral result by 4.574/8.784/13.119 points at K=8/14/20. It leads over both variants on all eleven combination averages, and on all twenty individual tasks in Full20.

These additions broaden the comparison to effective closed-form spectral methods and a reusable learned merger. Their respective training and adaptation requirements will accompany the results, while the controlled architectural experiments establish the incremental contribution of MergeCodec’s nonlinear composition.

Thank you for the detailed response and additional experiments. The controlled architectural ablations, unseen-expert evaluations, and compression analysis adequately address my main concerns. The authors have clarified these differences. Overall, my concerns have been largely addressed, and I am satisfied with the response.

Thank you for your thoughtful follow-up. We are glad that the controlled architectural ablations, unseen-expert evaluations, and compression analysis have addressed your main concerns. We will incorporate these results and clarifications into the revised manuscript.

Given your updated assessment, we would be grateful if you could also reconsider your overall rating in light of the additional evidence. Thank you again for your constructive feedback, which has helped us strengthen the paper.

Reviewer_E14XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MergeCodec encodes task vectors into a low-rank latent space, sums the latent representations of selected experts, and decodes the result to generate merged updates. A shared encoder and decoder are trained using a supervised loss across various task combinations. Once trained, the codec can be reused for new expert combinations without additional data or optimization. The method is evaluated on 20 classification tasks across three CLIP backbones.

Critical strengths
  • The methodology is straightforward, easy to understand, and effective.
  • A single trained codec can be reused across expert combinations without additional combination-specific optimization. The method maintains high accuracy across three CLIP backbones. On ViT-B/32, it achieves performance comparable to MTL, including on combinations of 14 experts, a size not seen during training.
Critical weaknesses

W1. The novelty of the proposed method and the underlying reasons for its performance improvements over prior work remain unclear. HyperFix [1] has already proposed a hypernetwork-based nonlinear merging approach that can be reused across multiple expert combinations. LS-Merge [2] and Task Singular Vectors [3] also explore latent-space merging and low-rank task-vector representations. A direct comparison is necessary to clarify the additional benefits of MergeCodec over existing merging methods. In particular, the learned linear codec also achieves performance close to that of the proposed method (Table 3). Furthermore, while WUDI [4] explicitly models task interference, this paper does not investigate whether the performance gains arise from reduced interference, implicit update rescaling, or the effects of supervised training. Demonstrating strong performance alone does not establish a distinct merging principle.

[1] HyperFix: Combinatorial Nonlinear Correction for Task Vector Merging (arxiv 2026) [2] LS-Merge: Merging Language Models in Latent Space (ICLR 2026) [3] Task Singular Vectors: Reducing Task Interference in Model Merging (CVPR 2025) [4] Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025)

W2. The paper does not show whether MergeCodec’s practical benefits justify its training cost. MergeCodec requires supervised training using labels from the original tasks and incurs substantial costs. Its training time per update is also nearly twice that of MTL20 (Table 6). However, the paper does not include a baseline in which a single trained MTL20 model is reused across all evaluation combinations. Thus, comparisons solely against combination-specific MTL(S) make it difficult to assess the practical advantages of a reusable learnable merger.

W3. Reusability matters especially when experts are added or updated. By design, task vectors derived from the same base model can be re-encoded using the frozen codec. However, the paper does not verify whether performance remains stable when existing expert checkpoints are updated or new expert configurations are introduced. This limits the practical claim that the codec can be used without retraining in a continuously evolving expert repository.

W4. Scalability to LLMs remains uncertain. The current codec employs tensor-wise trainable encoders and decoders. An analysis of the initial training costs and memory requirements for LLMs with 3B parameters or more would help establish the feasibility of scaling this approach. Furthermore, expert checkpoints are increasingly distributed as LoRA adapters rather than fully fine-tuned models. It remains unclear whether applying MergeCodec to updates that are already represented in low-rank form offers any advantage over existing LoRA merging approaches.

Questions for authors

Q1. Can the authors provide a direct comparison with HyperFix under identical or comparable supervision and training budgets?

Q2. What specific advantages does the nonlinear structure offer over a learned linear codec, particularly given the results in Table 3?

Q3. Why does MergeCodec maintain high accuracy as the number of experts increases? Comparing task-interference metrics used in WUDI alongside layer-wise representation discrepancies could help clarify the underlying mechanism.

Q4. How would performance and total computational cost change if a single trained MTL20 model were reused across all task combinations? Furthermore, does the codec maintain its performance without retraining when existing expert checkpoints are updated or new expert splits are introduced?

9 comments

Thank you for recognizing the effectiveness and reusability of MergeCodec. Your questions motivate a clearer explanation of our contribution and additional experiments on architectural controls, parameter transformations, expert updates, and language-model scalability.

The central contribution is learning compact expert representations whose sums can be decoded into effective models across different expert sets. This couples compression with composition: the same frozen operator supports cached expert codes, new combinations, and newly supplied expert checkpoints, while producing a standard model without additional per-example inference computation.

Q1. Contribution relative to prior work and direct comparison with HyperFix

HyperFix learns subset-conditioned corrections to a linear merge. MergeCodec jointly learns an encoder for individual task vectors and a decoder for their aggregate:

\theta_S=\theta_0+D_\psi\left(\sum_{t\in S}z_t\right).$$ The encoder and decoder are optimized through the task losses of composed models. Each expert code is independently cacheable, while decoding responds to the selected combination. Low-rank and latent-space representations, as explored by TSV and LS-Merge, provide relevant foundations; our contribution is to **train these representations explicitly for reusable composition**, including transfer to experts excluded from codec training. We have added an independent HyperFix reimplementation on the same twenty-expert ViT-B/32 bank, sharing the base model, expert checkpoints, classification heads, preprocessing, and all 11 evaluation combinations. | Evaluated K | HyperFix | MergeCodec | Gain (pp) | |---|---:|---:|---:| | 8 | 84.920% | **89.135%** | **+4.215** | | 14 | 83.760% | **88.458%** | **+4.698** | | 20 | 83.786% | **88.730%** | **+4.944** | MergeCodec leads on **all 11 combinations** and 122 of 130 task–combination entries. HyperFix improves over mean merging by 19.220–22.683 points, providing an effective learned baseline. For transparency, HyperFix uses expert-logit distillation on a stratified training subset, with 13,500 updates balanced across K=1–6; MergeCodec uses its original label-supervised, 4,000-update recipe. The runs process 47,250 and 48,000 task batches, respectively, with different batch sizes and supervision. Their trainable parameter counts are 349.97M and 114.75M. This comparison establishes performance under the respective training recipes; the following architectural experiment directly controls supervision, sampling, and optimization budget. **Q2. What does nonlinearity contribute beyond a learned linear codec?** The architectural distinction is that a nonlinear decoder can respond to the aggregate expert representation. With a linear decoder, decoding a sum is equivalent to summing independently decoded codes. With nonlinear decoding, the contribution of an expert can depend on the other experts included in the composition. We tested this distinction in the strict unseen-expert setting, where transfer is particularly informative. SVHN, Flowers102, PCAM, and Rendered-SST2 are excluded from SVD initialization, rank allocation, and codec training. Linear, Encoder-only, and Two-sided codecs share the same initialization, ranks, code dimensions, optimizer, 4,000-update budget, and paired subset/image sampling within each seed. Their shared parameter counts differ by less than 0.1%. Across three training seeds: | Held-out expert count K | Linear | Encoder-only | Two-sided | |---|---:|---:|---:| | 1 | 83.445 ± 0.123 | 83.854 ± 0.035 | **85.769 ± 0.207** | | 2 | 82.493 ± 0.063 | 83.379 ± 0.046 | **84.778 ± 0.156** | | 4 | 80.967 ± 0.073 | 82.582 ± 0.118 | **83.265 ± 0.108** | Values are mean accuracy ± sample SD. **Two-sided > Encoder-only > Linear holds at every evaluated K in all three seeds.** The complete codec improves over Linear by approximately **2.3 percentage points** throughout this panel. At K=4, the paired gain over Linear is **2.298 points**, with a pointwise 95% t interval of **[2.169, 2.427]**. Its additional gain over Encoder-only is **0.683 points**, with interval **[0.557, 0.809]**. These results show that nonlinear decoding contributes to unseen-expert transfer beyond nonlinear encoding alone, under matched supervision and training schedules.

Q3. Why does performance remain strong across expert counts? What do interference and representation diagnostics reveal?

Training across diverse expert sets is central to the design: the shared transformation is optimized to maintain task performance under different combinations and cardinalities. The original sampling ablation supports this choice. Relative to training exclusively on Full20, Mixed training improves K=8 and K=14 accuracy by 4.07 and 1.92 points, respectively, while retaining 88.73% Full20 accuracy.

Following your suggestion, we additionally analyzed parameter transformations and expert-relative representation discrepancies across all 11 B/32 evaluation combinations.

The learned transformation changes both update direction and magnitude. The cosine similarities between Two-sided and learned Linear updates are 0.573/0.533/0.501 at K=8/14/20. Even allowing the best independent scalar multiplier for every parameter tensor leaves relative residual norms of 0.681/0.726/0.761. Thus, the learned update contains substantial structure beyond global or per-tensor rescaling of the Linear update.

We also tested the functional consequences by exchanging directions and norms between the two trained models. Norm matching is performed separately for each parameter tensor, without selecting coefficients using labels:

Update directionPer-tensor normK=8K=14K=20
LinearLinear88.59687.68488.022
LinearTwo-sided85.58284.63384.564
Two-sidedLinear87.49886.79087.198
Two-sidedTwo-sided89.13588.45888.730

The complete learned configuration performs best in every group. These exchanges are consistent with joint adaptation of update direction and magnitude: transferring either component alone does not reproduce the complete codec’s performance.

For representation analysis, we measured the expert-relative error used in WUDI’s diagnostics,

Ex∥fmerged(x)−fexpert(x)∥2∥fexpert(x)∥2,\mathbb{E}_x \frac{\|f_{\mathrm{merged}}(x)-f_{\mathrm{expert}}(x)\|_2} {\|f_{\mathrm{expert}}(x)\|_2},

using the same 128 images per task for every method and examining all encoder depths. MergeCodec is closer to the original expert than Linear at many earlier and intermediate depths, while the final representations follow a different pattern:

KLinear: final normalized feature errorMergeCodecWUDI
80.6300.6570.581
140.7140.7210.772
200.7530.7651.008

These measurements clarify the learned behavior: higher joint-task accuracy can coexist with representations that differ from those of individual experts. MergeCodec directly optimizes the composed model’s task performance, allowing layer-dependent representational changes. Together with the controlled nonlinear ablation, the diagnostics support a learned composition mechanism involving coordinated parameter transformations, rather than attributing the gains solely to a uniform reduction in expert-relative feature error.

Q4. Fixed MTL20, total cost, and reuse when experts change

We evaluated the requested fixed-MTL20 baseline across all three CLIP backbones and K=8/14/20. MergeCodec remains within 0.174–0.519 percentage points of this fixed model across the nine group means. This establishes fixed MTL20 as a strong reference for serving the original task bank.

We also completed a fresh B/32 cost–accuracy comparison on the same H100 80GB, with BF16 training and FP32 evaluation:

MetricMergeCodecFixed MTL20
Preparation + training + export72.26 min39.98 min
Peak allocated training memory19.437 GiB2.281 GiB
Full20 accuracy88.730%89.009%
Exported-model evaluation memory, batch 1280.7294 GiB0.7294 GiB

The timing starts from available base/expert checkpoints and includes fresh SVD preparation for MergeCodec. Its additional upfront investment provides a composition operator that can subsequently accept different expert sets and checkpoint versions.

We directly tested checkpoint replacement with the codec frozen. Starting from the original B/32 codec trained on twenty experts, we continued fine-tuning DTD, EuroSAT, Flowers102, and CIFAR100 experts for 500 steps each. We then replaced their task vectors while keeping the codec, SVD initialization, ranks, base model, heads, and λ=1 unchanged.

Expert-library changeFull20 accuracy
Original library88.730%
Individual checkpoint replacements88.703–88.736%
All four checkpoints replaced88.670%

The checkpoint changes have norms equal to 4.8–21.5% of their original task-vector norms. Replacing all four changes overall accuracy by only −0.060 points; the remaining sixteen tasks change by −0.074 points on average. This directly demonstrates reuse under the tested checkpoint updates without retraining the codec.

Expert selection provides another practical operation. Across five leave-one-expert-out configurations, omitting an expert reduces its associated task accuracy by 22.788 points on average, while the remaining nineteen tasks change by only −0.030 points. This supports selective control through the supplied expert set.

Most importantly, reuse extends beyond checkpoints associated with codec-training tasks. In the original strict OOD experiment, the same four experts are excluded from initialization and training. Their joint accuracy improves overall from 79.01% to 83.15% as the training bank expands from four to sixteen experts. The three-seed result above further confirms transfer at sixteen training experts.

These capabilities motivate an evolving expert repository: a frozen codec can compose selected experts, accept replacement checkpoints, and merge previously unseen experts. The observed transfer also motivates our longer-term goal of a general-purpose codec for compatible experts derived from a common pretrained base, trained on a broader collection of experts and task data.

W4. Language-model scalability and LoRA

To provide measured evidence of scalability beyond CLIP, we completed experiments with FLAN-T5-Base and Qwen2.5-1.5B. On FLAN’s eight-task GLUE evaluation, MergeCodec C4 achieves 87.07, compared with 79.59 for development-calibrated TA.

For Qwen2.5-1.5B, we trained codecs over eight full-parameter experts on a single H100 80GB:

ConfigurationShared codec parameters4,000-update training timePeak allocated memoryFull8 score
C4180.63M4.075 h44.138 GiB80.49
C6413.25M2.017 h42.158 GiB81.66
C10240.66M1.947 h42.018 GiB77.41

Scores are task-balanced mixed-metric means. Training times include optimization and checkpoint writing; initialization and model loading are outside this timer. Peak memory includes the frozen expert library. These completed runs provide concrete feasibility evidence at the 1.5B scale, with substantial composition capability retained even with a small shared codec.

For LoRA experts, the relevant opportunity is learning a reusable composition operator for adapter sets. Their factorized updates can also be exploited computationally: if an update is UV⊤UV^\top, the codec’s initial projection can be evaluated as

P⊤(UV⊤)=(P⊤U)V⊤,P^\top(UV^\top)=(P^\top U)V^\top,

without first materializing the dense input update. This provides a natural route to adapting the interface to LoRA. The prospective benefit is learned composition across selected adapters; the language-model results reported above establish this capability for full-parameter expert checkpoints.

Reviewer_4z7cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MergeCodec, a learned encoder-sum-decoder mechanism for composing task vectors from experts trained from a common pretrained CLIP backbone. Each task vector is encoded into a compact latent representation, the selected expert codes are summed, and a decoder maps the aggregate code back into a model update. The codec is trained using supervised task losses on mixed subsets of image-classification experts with varying identities and cardinalities. After training, the codec is frozen and reused with a fixed coefficient, without per-combination gradient updates or coefficient calibration.

Experiments use ViT-B/32, ViT-B/16, and ViT-L/14 backbones and evaluate multi-task subsets, singleton reconstruction, held-out expert combinations, out-of-distribution experts, and codec architecture/compression/sampling ablations. The codec approaches separately trained multi-task models on several reported combinations and performs competitively with task-arithmetic and other merging baselines.

Critical strengths

A simple reusable composition interface. The permutation-invariant encode-sum-decode structure is operationally attractive and allows a trained codec to materialize a merged model without test-time inference overhead.

A relatively broad set of diagnostics. The paper evaluates multiple subset sizes, singleton recovery, unseen experts, architecture variants, compression factors, sampling strategies, storage, and timing. This is more thorough than a standard model-merging comparison.

Critical weaknesses

The “compact” and “data-free” claims are overstated. The default codec contains approximately 114.75M shared parameters, while the stated C=4C=4 compression factor explicitly excludes these shared parameters. The shared codec plus cached codes still has substantial storage, and training costs 1.86–2.41 times the reported MTL20 update. No break-even analysis compares the method against storing expert checkpoints or using ordinary task arithmetic. Moreover, the codec is supervised using labeled data from the training task bank and must be trained separately for each backbone, so it is not a generally data-free or architecture-independent merger.

Generalization and statistical evidence are insufficient. The full 20-task combination is used during training and evaluation, so the k=20k=20 result is not a held-out composition. The OOD experiments hold out only four datasets within the same CLIP family, base model, parameterization, preprocessing, and image-classification task family. All codec runs use a fixed seed, with no confidence intervals or significance tests. The closest reusable-composition baseline, HyperFix, is discussed but not evaluated, and the MTL and codec training budgets are not matched.

Questions for authors

Please report the complete storage, peak-memory, and training costs, including the base model, shared codec, residual maps, cached codes, task heads, optimizer states, and expert checkpoints. Since the stated C=4C=4 compression factor excludes the 114.75M shared codec parameters, please clarify the actual compression ratio and give the break-even number of merged subsets compared with storing checkpoints or using task arithmetic.

Please clarify which expert subsets and cardinalities are strictly held out from codec training. In particular, was the full 20-task combination used to optimize the codec and then evaluated on the same test split? Please add strictly held-out compositions, cardinalities outside the training range, and tests with different backbones or pretrained checkpoints to establish whether the claimed OOD generalization extends beyond the same CLIP/task family.

Please rerun the main ViT-B/16 and ViT-L/14 experiments over multiple random seeds with confidence intervals or paired significance tests, and include HyperFix plus a parameter-matched linear/low-rank composition baseline. Are the reported 0.27–0.73 percentage-point gaps to separately trained MTL statistically meaningful after these controls?

5 comments

Thank you for recognizing the reusable composition interface and the breadth of our diagnostics. Your questions help us make the contribution more concrete: MergeCodec learns a compressed expert representation and a reusable composition operator that supports new combinations, untrained cardinalities, and previously unseen experts. We address your concerns with complete resource accounting, explicit evaluation protocols, additional language-model experiments, and new multi-seed and baseline comparisons.

Q1. Complete storage, memory, training costs, and storage break-even

We will make the distinction between training and reuse explicit: the codec is trained with supervision; subsequent merging is data-free and optimization-free, using the frozen codec and a fixed coefficient λ=1.

Similarly, C=4 denotes the per-expert encoding dimension ratio. To quantify the actual system-level benefit, we have completed an FP32 storage audit including the base model, all task heads, shared encoder/decoder bases, nonlinear residual maps, and cached codes, including their uncompressed coordinates.

Storage component or deployment configurationViT-B/32ViT-B/16ViT-L/14
Base model + all task heads0.3300.3231.136
Shared encoder + decoder, including residual maps0.4270.4201.493
Codes for all 20 experts1.6291.5985.647
Complete cached composition service2.1722.1327.529
Complete service retaining the encoder2.3862.3428.276
Base + 20 raw task vectors + heads6.8456.71623.724

All values are in GiB. The cached service contains the base, heads, decoder, and 20 codes; the encoder and original expert checkpoints can be released after encoding. Retaining the encoder additionally enables newly supplied checkpoints to be encoded.

For ViT-B/32, this gives an actual 3.15× storage reduction relative to the raw task-vector library, or 2.87× when retaining the encoder. Compared with storing 20 full expert checkpoints and shared heads, which occupy 6.520 GiB, the cached service provides a 3.00× reduction. The serialized cached deployment package occupies 2.173 GiB, including file-format overhead.

We also account separately for training states: Adam state tensors occupy 0.855/0.841/2.986 GiB for B/32, B/16, and L/14, respectively. The corresponding resumable codec checkpoints occupy 1.283/1.262/4.481 GiB. Optimizer states and gradients are released for deployment.

The requested storage break-even can be stated directly. Let B denote backbone storage, H shared heads, D the decoder, and z one expert code. A cached library uses B+H+D+20z, while storing Q complete subset-specific models uses QB+H. For all three C4 backbones, the cached library becomes smaller at Q≥7. This comparison generates requested models on demand rather than also retaining every generated checkpoint.

Compression is also configurable: on B/32, C16 reduces the complete cached service to 0.794 GiB while retaining 87.98% Full20 accuracy, compared with 88.73% at C4.

For computation, we completed a fresh B/32 comparison on the same H100 80GB, with BF16 training and FP32 evaluation:

MetricMergeCodecFixed MTL20
Complete 4,000-update training process68.74 min39.98 min
Preparation + training + export72.26 min39.98 min
Peak allocated training memory19.437 GiB2.281 GiB
Full20 accuracy88.730%89.009%
Exported-model evaluation memory, batch 1280.7294 GiB0.7294 GiB

These measurements start from available base/expert checkpoints and include fresh SVD preparation for MergeCodec. The respective training recipes use 48,000 and 80,000 task batches.

The additional training investment produces a reusable composition service. With the complete expert library resident on GPU, cached Full20 merging takes 30.93 ms, versus 23.41 ms for TA, while its peak allocated memory is 2.570 GiB, versus 7.189 GiB for TA. The advantage here is the smaller expert library and stronger merged-model performance. After composition, the codec and code library can be released: the deployed model has the standard backbone computation and the same measured evaluation memory as MTL20.

Q2. Strictly held-out compositions, cardinalities, experts, and applicability beyond CLIP

We will make the evaluation categories explicit in the main text. The original submission already distinguishes the following settings in §4.2, §4.3, and Appendices B and E:

EvaluationTraining configurationWhat is held out?
Main K=8 subsetsTwenty experts; training K={4,8,12,16,20}All five evaluated eight-expert combinations
Main K=14 subsetsSame configurationAll five evaluated combinations and cardinality 14
Full20Same configurationHeld-out test images; the composition is a seen-composition reference
Singleton reconstructionSame configurationK=1, below the minimum training cardinality
Strict unseen-expert evaluationSixteen experts; training K={3,6,10,13,16}Four experts and their associated training data; evaluation K={1,2,4}

Thus, the full20 combination is indeed used during training, using training images, and evaluated on the separate official test images. The ten non-full main-table combinations are excluded from codec training. In particular, the K=8 results test composition generalization at a trained cardinality, while K=14 additionally tests an untrained cardinality.

The strict expert-OOD setting provides a separate test of transfer. SVHN, Flowers102, PCAM, and Rendered-SST2 are excluded from SVD initialization, rank allocation, and codec optimization, including their expert weights, images, and labels. Their checkpoints are supplied only after the codec is frozen. Importantly, K=2 evaluates multi-expert composition below the minimum training cardinality of three.

We have now repeated this setting over three training seeds. Merging the four held-out experts achieves 83.265 ± 0.108%, compared with 81.624% for TSV-M and 81.515% for WUDI. This demonstrates that the learned operator can compose capabilities absent from its initialization and supervised training.

We additionally evaluated the frozen twenty-expert B/32 codec at K={3,5,7,9,11,13,15,17,19}, using five combinations per K. All nine cardinalities are absent from training; K=3 is below the training range. Across these 45 additional combinations, the complete codec exceeds the learned linear variant at every cardinality mean, by 0.140–0.783 percentage points.

To address applicability beyond the CLIP/image-classification family, we also completed experiments with FLAN-T5-Base on eight GLUE tasks and Qwen2.5-1.5B on eight language tasks:

Backbone and full-bank evaluationMergeCodec C4Calibrated TA
FLAN-T5-Base, eight tasks87.0779.59
Qwen2.5-1.5B, eight tasks80.4966.94

These are task-balanced mixed-metric scores on a 0–100 scale. TA uses development-set calibration for FLAN and test-set calibration for Qwen; MergeCodec uses λ=1 throughout. Qwen also evaluates untrained cardinalities K=3,5,7.

Each pretrained base has its own codec, which is then frozen and reused within its compatible expert parameterization. These results establish the method’s applicability across vision encoders, encoder–decoder language models, and decoder-only language models, alongside the strict unseen-expert transfer demonstrated on B/32.

Q3. Multiple seeds, confidence intervals, HyperFix, and a parameter-matched linear baseline

We have repeated the complete ViT-B/16 and ViT-L/14 main evaluations using training seeds 42, 43, and 44, covering all 11 merged subsets per backbone.

BackboneKMergeCodec mean ± SD (%)MTL(S) (%)Codec − MTL, 95% CI (pp)
B/16891.209 ± 0.08492.025[−1.025, −0.607]
B/161490.750 ± 0.00491.300[−0.560, −0.541]
B/162091.083 ± 0.01791.348[−0.307, −0.224]
L/14893.575 ± 0.02694.007[−0.496, −0.368]
L/141492.981 ± 0.07193.442[−0.637, −0.284]
L/142093.085 ± 0.09593.604[−0.755, −0.282]

The intervals are pointwise 95% t intervals across codec training seeds, using the fixed subset-specific MTL references. The small accuracy gap is reproducible under codec training randomness: mean gaps are 0.265–0.816 points, while macro-accuracy SDs are only 0.004–0.095 points. This supports our central accuracy–reuse result: one trained codec consistently remains close to separately trained subset-specific MTL models while serving all evaluated combinations without further optimization.

For the architectural control, we evaluated a learned linear low-rank codec with the same SVD initialization, layer ranks, code dimensions, optimizer, 4,000-update budget, and paired subset/image sampling. Shared parameter counts are 115.086M for Linear and 115.200M for Two-sided, a difference below 0.1%.

On the strict held-out-expert evaluation:

KLinear, mean ± SD (%)Two-sided, mean ± SD (%)Two-sided − Linear, paired 95% CI (pp)
183.445 ± 0.12385.769 ± 0.207[2.074, 2.575]
282.493 ± 0.06384.778 ± 0.156[2.052, 2.519]
480.967 ± 0.07383.265 ± 0.108[2.169, 2.427]

The approximately 2.3-point gain persists across all three seeds and all evaluated cardinalities. This provides direct evidence that nonlinear composition improves transfer beyond a learned low-rank linear mapping with essentially the same parameter capacity.

Finally, we added an independent HyperFix reimplementation extended to the same twenty-expert B/32 bank, sharing the base, expert checkpoints, heads, preprocessing, and evaluation subsets. HyperFix uses expert-logit distillation on a stratified training subset, with 13,500 updates balanced across K=1–6; MergeCodec retains its original supervised training recipe.

Evaluated KHyperFixMergeCodecGain (pp)
884.920%89.135%+4.215
1483.760%88.458%+4.698
2083.786%88.730%+4.944

MergeCodec leads on all 11 evaluated combinations. HyperFix itself improves over mean merging by 19.220–22.683 points, providing an effective learned reference. The added comparison demonstrates a substantial advantage for MergeCodec in this shared expert-bank setting, complementing the closely parameter-matched linear control and the multi-seed evidence above.

Reviewer_FtvNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MergeCodec, a supervised encoder–decoder for composing task vectors from experts fine-tuned from a common pretrained model. Each task vector is encoded separately, selected codes are summed, and the aggregate is decoded into a merged model update. Training across subsets of varying membership and size enables reuse without per-combination optimization. Experiments on three CLIP backbones demonstrate performance close to separately trained multi-task models, functional recovery of individual experts, and transfer to four experts excluded from codec initialization and training.

Critical strengths
  • Reusable composition with strong accuracy. A single frozen codec per backbone achieves accuracy close to separately trained multitask models across multiple expert combinations, without per-combination optimization or calibration. Although it slightly underperforms these multitask references, the ability to reuse one codec makes the result meaningful when different expert combinations are required.

  • Evidence that composition generalizes beyond training combinations. Evaluation on held-out combinations, an unseen subset size, and four experts excluded from both initialization and training supports the claim that the codec learns a reusable composition procedure. The sampling ablation further shows that training across diverse subsets contributes to this capability. Together, these results strengthen the central claim beyond performance on a single fixed merge.

Critical weaknesses
  • The contribution of nonlinear composition is not fully isolated. The supervised linear codec already achieves 88.60/87.68/88.02% accuracy with 8/14/20 experts, compared with 89.13/88.46/88.73% for the complete method. While these improvements support using nonlinear transformations, much of the in-domain performance is retained by the linear variant. Corresponding unseen-expert ablations are missing, leaving unclear whether nonlinearity materially contributes to the strongest generalization result. This limits our understanding of which components enable transferable composition.

  • The practical payoff of the additional supervised training remains unclear. The method requires existing fine-tuned experts and substantial supervised codec training. Repeated reuse could justify this investment, but the paper does not establish an end-to-end cost–accuracy comparison or identify the reuse regime in which it becomes preferable to separately training multitask models. Fast post-training merging alone does not establish this advantage, particularly because the training profiles and main MTL accuracy results use different precision settings.

Questions for authors
  1. Clarification — contribution to unseen-expert transfer. Are results available for matched Linear and Encoder-only variants on the existing held-out-expert evaluation? How do they compare with the complete codec, and what evidence supports the contribution of nonlinear decoding to transfer?

  2. Additional validation — robustness. Have the key architectural ablations or held-out-expert comparisons been repeated across training seeds? If so, what is the variability, and are the reported gains over the linear codec and the strongest merging baselines consistent?

  3. Clarification — practical reuse scenario. What intended application benefits from repeatedly constructing subset-specific models rather than reusing a single MTL model trained on the full task bank? Given the reported upfront training cost, what advantages or constraints would favor the codec in that setting?

4 comments

Thank you for recognizing the value of reusable composition and the evidence of generalization beyond training combinations. Your questions identify three important issues: the contribution of nonlinearity to unseen-expert transfer, its robustness across training seeds, and the practical value relative to a fixed multitask model. We address them with the following additional experiments.

Q1. Contribution of nonlinear encoding and decoding to unseen-expert transfer

We agree that the learned linear codec is already strong, and that the in-domain results alone do not establish the contribution of nonlinear decoding to transfer.

We therefore evaluated Linear, Encoder-only, and Two-sided codecs on the original strict held-out-expert setting: ViT-B/32 trained on 16 experts, with SVHN, Flowers102, PCAM, and Rendered-SST2 excluded from SVD initialization, rank allocation, and codec training. The variants share the SVD initialization, ranks, code dimensions, optimizer settings, and 4,000-update budget. Within each training seed, their sampled subsets and image batches are matched. Shared parameter counts differ by less than 0.1%. Evaluation uses the frozen codecs with λ=1.

The results below report mean accuracy ± sample SD over three training seeds (42, 43, and 44):

Held-out expert count KLinearEncoder-onlyTwo-sided
183.445 ± 0.12383.854 ± 0.03585.769 ± 0.207
282.493 ± 0.06383.379 ± 0.04684.778 ± 0.156
480.967 ± 0.07382.582 ± 0.11883.265 ± 0.108

Two-sided improves over Linear by 2.324/2.285/2.298 percentage points at K=1/2/4. Its additional gains over Encoder-only are 1.916/1.399/0.683 points. Thus, nonlinear decoding provides an incremental benefit on this held-out-expert setting, beyond nonlinear encoding alone.

Q2. Robustness across training seeds and comparison with strong baselines

All three seeds preserve the aggregate ranking Two-sided > Encoder-only > Linear at every evaluated K. All six paired, pointwise 95% t intervals for Two-sided minus either architectural variant are positive. For example, at K=4, the intervals are [2.169, 2.427] points versus Linear and [0.557, 0.809] versus Encoder-only. These intervals use three independent codec training runs with the expert split and initialization fixed.

The comparison with existing merging baselines is more nuanced. At K=4, all three Two-sided runs exceed the original fixed TSV-M and WUDI references, whose accuracies are 81.624% and 81.515%, respectively. At K=2, Two-sided averages 84.778%, versus WUDI’s 84.734%, and the difference changes sign across codec seeds. We therefore claim consistent architectural gains on this split, but not consistent superiority over the strongest baseline in every setting. The baseline results are held fixed, including the original test-calibrated TSV-M reference.

We also repeated the ViT-B/16 and ViT-L/14 main experiments over three codec seeds. Their K=8/14/20 macro-accuracy SDs range from 0.004 to 0.095 points, while mean gaps to the fixed subset-specific MTL references remain 0.265–0.816 points.

Q3. Practical reuse scenario and the cost of supervised codec training

The central motivation for MergeCodec is to learn a reusable merging operator for an evolving collection of experts. This includes selecting experts for a particular deployment, incorporating updated checkpoints, and—most importantly—composing experts that were never available during codec training.

Generalization to unseen experts is a key distinction from a fixed MTL model. Our original strict OOD experiment provides direct evidence: four target experts—SVHN, Flowers102, PCAM, and Rendered-SST2—are excluded from SVD initialization, rank allocation, and codec training, and are introduced only after the codec is frozen. As the training expert bank expands, performance on the same four held-out experts improves overall:

Number of training experts481216
Accuracy when merging all four held-out experts79.01%82.42%82.16%83.15%

Thus, the learned operator remains useful beyond its original training bank: it can construct a model from newly supplied experts without accessing their training data or retraining the codec. Our additional three-seed experiment confirms this capability at 16 training experts, achieving 83.265 ± 0.108% on the four held-out experts, compared with 81.624% for TSV-M and 81.515% for WUDI.

These findings motivate a broader goal: a general-purpose codec for compatible experts derived from a common pretrained base. A sufficiently broad collection of experts and task data could allow the merging operator to be trained once and reused across many subsequently supplied experts. The current OOD results provide initial support for this direction, making larger-scale expert collections a particularly promising avenue for future work.

An evolving expert repository provides a concrete reuse scenario. For example, an organization may maintain experts contributed by different teams, receive improved versions over time, and deploy models assembled from explicitly approved expert sets. A fixed MTL20 is a useful solution for the original task bank, while MergeCodec exposes expert selection and replacement directly through its composition interface. We evaluated both operations:

  • Expert selection: In five leave-one-expert-out experiments, removing a selected expert changed average accuracy on the remaining 19 tasks by only −0.030 percentage points, while accuracy on the omitted expert’s associated task decreased by 22.788 points on average. Removing other experts at the same composition size left those target tasks essentially unchanged. This demonstrates that expert selection can selectively influence task behavior while preserving performance on the retained tasks.
  • Expert replacement: After updating four expert checkpoints, we replaced them while keeping the codec and SVD initialization frozen. Full20 accuracy remained 88.670%, compared with 88.730% before replacement. The same learned operator therefore accommodated the updated checkpoints with only a 0.060-point change in overall accuracy.

Given available expert checkpoints, these composition operations require no codec retraining or task training data at merge time.

We also quantified the upfront investment under a consistent precision setting. We completed a ViT-B/32 comparison on the same H100 80GB, using BF16 training and FP32 evaluation for both methods. Starting from existing checkpoints, the measurements are:

MetricMergeCodecFixed MTL20
Preparation + training + export72.26 min39.98 min
Peak allocated training memory19.437 GiB2.281 GiB
Full20 accuracy88.730%89.009%
Exported-model evaluation memory, batch 1280.7294 GiB0.7294 GiB

The MergeCodec preparation time includes fresh SVD initialization. Its additional upfront investment produces a reusable composition operator while retaining accuracy close to MTL20. Once a model is composed, deployment uses a standard backbone with the same measured evaluation memory as MTL20.

The practical benefit is therefore the ability to reuse one learned operator across selected, updated, and previously unseen experts. Our experiments demonstrate these capabilities directly, and the improvement with a broader training expert bank supports the longer-term potential of MergeCodec as a general-purpose merging interface.

Thank you for the additional experiments and for clarifying the intended use case. I see the main contribution as showing that a learned merging rule can transfer to unseen experts from the same pretrained base. I find this a useful but incremental extension of prior latent-space merging work and maintain my weak accept rating.

Thank you for the thoughtful follow-up and your positive assessment. We appreciate your recognition of transfer to unseen experts as the central contribution. Our goal is to learn composable expert representations that remain useful beyond the original training bank: the frozen codec can merge experts excluded from both initialization and training, without target-task data or combination-specific optimization.

We will foreground this contribution in the revision. We view the demonstrated transfer as an initial step toward a broadly reusable codec for compatible experts derived from a common pretrained base, with larger and more diverse expert collections offering a promising direction for further development.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Request for reviewer feedback on our rebuttal

Dear Area Chair,

We posted our point-by-point responses on November 12, but have not yet received follow-up comments from the reviewers. As the discussion period ends on November 18, could you kindly encourage the reviewers to read our responses and share any remaining concerns, so that we can address them before the deadline?

Thank you for your help.

AuthorsOpenReview ↗

Rebuttal Summary for the Area Chair

Dear Area Chair,

Thank you for overseeing the review process. We would like to summarize the main evidence added during rebuttal and the current discussion.

The central contribution of MergeCodec is learning a reusable composition operator that transfers to previously unseen experts derived from a common pretrained base. Once trained, the frozen codec composes supplied expert checkpoints without target-task data, combination-specific optimization, or coefficient calibration. This transfer motivates its potential as a broadly reusable interface for evolving expert repositories.

We strengthened the evidence in four areas:

  1. Controlled architectural comparisons and robustness. We evaluated Linear, Encoder-only, and Two-sided codecs with matched supervision, initialization, ranks, sampling, optimizer, and 4,000-update schedules; shared parameter counts differ by less than 0.1%. Four target experts are excluded from initialization and training. Across three seeds, Two-sided consistently leads at every evaluated cardinality. Four-expert OOD accuracy reaches 83.265 ± 0.108%, exceeding Linear by 2.298 percentage points, with a paired 95% confidence interval of [2.169, 2.427]. We also repeated the B/16 and L/14 main experiments across three seeds.
  2. Additional relevant baselines. On the same twenty-expert B/32 bank, MergeCodec exceeds our independent HyperFix reimplementation by 4.215–4.944 points across the K=8/14/20 group means, leading on all eleven combinations. We also added both SWUDI and SWUDI-A from the requested spectral-regularization work; MergeCodec leads over both on all eleven combinations. These comparisons disclose method-specific supervision and resources, while the architectural controls above directly test the incremental benefit of nonlinearity.
  3. Practical reuse and complete cost accounting. Replacing four updated expert checkpoints while freezing the codec changes Full20 accuracy from 88.730% to 88.670%. Across five expert-exclusion configurations, retained-task accuracy changes by only −0.030 points on average, alongside selective changes on the omitted experts’ tasks. We also evaluated fixed MTL20 across all three backbones and measured complete B/32 preparation, training, and export: 72.26 minutes for MergeCodec versus 39.98 minutes for MTL20 on the same H100 and precision settings. The additional investment produces a reusable operator; exported models have the same measured evaluation memory as MTL20.
  4. Applicability beyond CLIP. Completed FLAN-T5-Base and Qwen2.5-1.5B experiments extend the approach to encoder–decoder and decoder-only language models. We distinguish this cross-family applicability from the strict unseen-expert transfer demonstrated within a common pretrained base. The discussion reflects substantial progress. FtvN identifies unseen-expert transfer as the central contribution and maintains Weak Accept. dXYg explicitly states that the controlled ablations, unseen-expert evaluations, and compression analysis adequately address the main concerns and expresses satisfaction with the response. 4z7c retains concerns about practical significance, training-budget differences, and generalization; our follow-up clarifies the intended use case and the scope of the corresponding evidence. We would also welcome E14X’s assessment of the added experiments.

We respectfully ask that the final assessment consider the demonstrated transfer and reuse capabilities alongside the measured training cost. We believe these results establish learning composable expert representations as a useful direction toward more general model merging.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.