acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Model Merging with Multi-Layer Aggregated Routing Evidence

Abstract

Dynamic model merging aims to retain the efficiency of a unified model while better preserving task-specific expertise through input-dependent routing at both the task and head levels. However, even the routing components of the state-of-the-art approach remain limited: the task routing is highly sensitive to the choice of routing layer, whereas the head routing relies on simple max-logit competition. To address these limitations, we propose multi-layer aggregated routing evidence (MATE) as a unified mechanism for improving both task and head routing. Specifically, we first introduce a metric to quantify task routing separability across different layers. Empirical analysis of this metric reveals that different layers provide complementary evidence for identifying the tasks relevant to an input. Motivated by this observation, MATE selects multiple informative layers according to their routing separability and aggregates their layer-wise routing scores to produce a more robust signal that can be directly used for task selection. Moreover, we observe that this signal also encodes useful information about the suitability of the corresponding classification heads. This motivates us to further combine the MATE signal with the max-logit score to guide classification head selection. Extensive experiments on vision benchmarks demonstrate that MATE consistently achieves state-of-the-art performance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Model Merging, Multi-Layer Routing, Task Subspaces, Multi-Task Learning, Computer Vision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_s5DkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MATE extends MASS for dynamic model merging when the task identity is unknown. It uses a small proxy set to select routing layers according to a relative projection residual margin, averages their task-relevance scores, and reuses the aggregated scores to guide classification-head selection. The task-subspace representation, adaptive merging procedure, and two-pass inference structure follow MASS. The paper evaluates these changes across three CLIP ViT backbones and three task counts.

Critical strengths

The changes are simple and improve on MASS in all nine evaluated settings. Some improvements are meaningful: ViT-B/32 with 14 tasks increases from 82.90 to 85.09, and ViT-B/16 with 20 tasks increases from 81.11 to 82.95. The routing analysis and component ablation provide useful evidence about how the changes affect task identification and final classification. The supplementary material improves the transparency of the work. It includes the implementation, configurations, layer-wise proxy statistics, and per-task evaluation records. Averaging the supplied per-task records recovers all 18 MASS and full-MATE entries in the main comparison to the precision reported in the paper. This is useful support for the reported results, although it does not constitute an independent reproduction.

Critical weaknesses

My main reservation is whether the contribution beyond MASS is sufficiently substantial. MASS already observed that routing-layer preferences vary across tasks. MATE adds margin-based layer selection, score averaging, and routing-guided head selection, but the current comparison does not clearly isolate the benefit of aggregating several layers from that of selecting a better layer using proxy data. The supplementary configurations specify M9 for the two ViT-B backbones and M20 for ViT-L/14 as the single-layer baseline. In eight of the nine settings, these differ from the highest-margin layer computed from the supplied proxy statistics. A higher proxy margin does not guarantee higher classification accuracy, but this makes a single layer selected under the same conditions an important comparison for assessing the central contribution. There is also a discrepancy between the routing-score definition and the implementation. Equation (4) applies softmax directly to negative projection residuals, whereas the supplied code standardizes the scores across tasks before softmax. This changes the score distribution used for thresholding, cross-layer averaging, and head selection. The paper should specify the actual computation used for the reported results.

Questions for authors

How does multi-layer aggregation compare with a single layer selected using the same proxy samples and relative-margin criterion? If this comparison has already been performed, please identify the corresponding results and keep the head-selection rule fixed so that the effect of aggregation is clear.

Were the reported experiments run with the score standardization implemented in router.py? Please reconcile Equation (4) with the implementation and clarify the routing-score computation used in the comparisons.

4 comments
AuthorsOpenReview ↗

To Reviewer s5Dk (1/2)

W1 & Q1. Multi-layer aggregation versus proxy-based single-layer selection

Thank you for highlighting the need to distinguish the contribution of multi-layer aggregation from that of proxy-based layer selection. We agree that MASS already observed task-dependent routing-layer preferences. Building on this observation, MATE introduces the relative projection residual margin to quantify layer-wise task separability, selects multiple informative candidate layers, and aggregates their routing scores. The aggregated scores also guide the final classification head decision.

Controlled comparison with a proxy-selected single layer

To isolate the contribution of multi-layer routing aggregation, we conducted an additional comparison of five routing strategies. All variants use the same task models, TSV-M merging of the selected task subspaces, and routing-guided head decision, abbreviated as HG below. Only routing-layer selection and aggregation vary. Proxy-Best Single and MATE Top-5 use the same proxy samples and task-averaged margin Aˉℓ\bar A_\ell.

Table R1. Average classification accuracy (%) under different candidate layer selection and aggregation strategies. Bold indicates the highest reported accuracy in each configuration.

BackboneTasksSingle-Layer (MASS) + HGProxy-Best Single + HGRandom-5 + HGAll-MLP + HGMATE Top-5 + HG
ViT-B/32888.5288.2086.86 ± 2.4587.2288.91
ViT-B/321483.7783.6583.40 ± 1.7084.5585.09
ViT-B/322081.4377.8778.59 ± 1.0480.8982.25
ViT-B/16891.3191.3190.78 ± 0.5091.0591.44
ViT-B/161488.1783.9786.14 ± 3.1688.5488.55
ViT-B/162080.9580.5179.99 ± 3.0581.3682.95
ViT-L/14893.5593.7293.77 ± 0.0593.8093.66
ViT-L/141491.3991.6692.21 ± 0.4092.4591.58
ViT-L/142090.5590.0091.29 ± 0.9890.5391.00

Specifically:

  • Single-Layer (MASS) uses the originally specified routing layer: M9 for ViT-B/32 and ViT-B/16, and M20 for ViT-L/14.
  • Proxy-Best Single selects the single layer with the highest task-averaged margin Aˉℓ\bar A_\ell, computed from the same proxy data used by MATE.
  • Random-5 randomly selects five MLP layers. We report the mean and standard deviation over three random seeds.
  • All-MLP aggregates all MLP layers.
  • MATE Top-5 selects the five layers with the highest task-averaged margins, following Eq. (8) with KL=5K_L=5.

As shown in Table R1, with routing-guided head decision fixed across all variants, MATE Top-5 outperforms Single-Layer (MASS) in all nine configurations and Proxy-Best Single in seven. These results indicate that MATE’s gains generally extend beyond using proxy data to select a stronger individual routing layer.

Furthermore, Random-5 and All-MLP both outperform Proxy-Best Single in all three ViT-L/14 settings. This provides additional evidence for the value of combining task-discriminative routing information across layers. Meanwhile, MATE Top-5 outperforms both Random-5 and All-MLP in all six ViT-B/32 and ViT-B/16 configurations, supporting the usefulness of margin-based candidate layer selection over these simpler multi-layer alternatives.

The results also show that MATE Top-5 is not the best layer combination in every configuration, particularly on ViT-L/14. The task-averaged margin is therefore a useful separability criterion, rather than a guarantee of the layer combination with the highest final classification accuracy.

Together, these controlled comparisons support the contributions of both candidate layer selection and multi-layer routing aggregation. Multi-layer aggregation uses complementary routing evidence beyond a single proxy-selected layer, while the relative projection residual margin provides a quantitative basis for identifying informative candidate layers.

AuthorsOpenReview ↗

To Reviewer s5Dk (2/2)

W2 & Q2. Routing-score standardization and consistency with Eq. (4)

Thank you for pointing out this discrepancy. Yes. After rechecking the implementation, we confirm that all experiments reported in the original manuscript used the Z-score standardization in router.py, rather than applying softmax directly to raw negative projection residuals.

MATE was developed using the public MASS codebase and inherits the standardization in its topk routing mode. Eq. (4) followed the formulation in the MASS paper but omitted this operation from the actual implementation. We acknowledge that the equation therefore did not fully describe the computation used for the reported results.

Actual routing-score computation

For input xx, let ri,ℓ(x)r_{i,\ell}(x) denote the projection residual for task ii at layer ℓ\ell. The residuals are first standardized across tasks within the current layer:

r~i,ℓ(x)=ri,ℓ(x)−μℓ(x)σℓ(x)+ϵ,\widetilde r_{i,\ell}(x) = \frac{r_{i,\ell}(x)-\mu_\ell(x)} {\sigma_\ell(x)+\epsilon},

where μℓ(x)\mu_\ell(x) and σℓ(x)\sigma_\ell(x) are the mean and standard deviation of the residuals across all tasks for the current input at layer ℓ\ell, and ϵ\epsilon is a numerical stability constant.

Each layer then computes its routing scores as

wℓ,i(x)=exp⁡!(−r~i,ℓ(x))∑j=1Texp⁡!(−r~j,ℓ(x)).w_{\ell,i}(x) = \frac{\exp\\!\left(-\widetilde r_{i,\ell}(x)\right)} {\sum_{j=1}^{T}\exp\\!\left(-\widetilde r_{j,\ell}(x)\right)}.

These scores form a soft task distribution at each layer. MATE uniformly averages the distributions over the candidate layer set, as specified in Eq. (9):

si(x)=1∣Lcand∣∑ℓ∈Lcandwℓ,i(x).s_i(x) = \frac{1}{|\mathcal L_{\mathrm{cand}}|} \sum_{\ell\in\mathcal L_{\mathrm{cand}}} w_{\ell,i}(x).

Thus, cross-layer aggregation operates on task probabilities computed from standardized residuals, rather than on raw projection residuals from different layers. The aggregated scores si(x)s_i(x) are then used for candidate task selection and routing-guided head decision.

Distinction from the relative projection residual margin

Z-score standardization is inherited from MASS and is not a new contribution of MATE. It serves a different purpose from the relative projection residual margin. The latter uses the representation-norm normalization in Eq. (5) to quantify layer-wise task separability for offline candidate layer selection, whereas Z-score standardization is applied when computing online routing scores.

We will correct Eq. (4) and explicitly describe the standardization and routing-score computation in the revised manuscript, ensuring that the mathematical description matches the implementation used in the experiments. The uniform aggregation in Eq. (9) remains unchanged.

Thank you for the detailed response and additional experiments. My main concerns have been addressed. However, the method retains the task-subspace representation and dynamic merging framework used in MASS, with the main changes focusing on layer selection, multi-layer routing aggregation, and classification-head selection. These changes are relatively simple, so my assessment remains borderline between 2 and 3. Taking the additional experiments and revisions into account, I will raise my rating from 2 to 3.

AuthorsOpenReview ↗

To Reviewer s5Dk

Thank you for your thoughtful reassessment and for raising your rating. We sincerely appreciate your recognition of our additional experiments and revisions. Your feedback has been very helpful in improving our paper.

Reviewer_gMX9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MATE, a dynamic model merging method that extends MASS through multi-layer routing aggregation and routing-guided head decision. MATE uses the relative projection residual margin to select informative routing layers and aggregates their routing scores to guide both candidate task selection and classification head selection. Experiments across three CLIP ViT backbones and three task scales show consistent improvements over MASS.

Critical strengths

s1. The proposed method addresses concrete limitations of an existing strong baseline. The paper identifies the sensitivity of single-layer task routing to the choice of routing layer and the limitations of max-logit competition for classification head selection. Both issues are relevant to the performance of dynamic model merging, and the proposed components are directly motivated by these observations.

s2. The paper provides useful routing analyses and ablation studies. In addition to final classification accuracy, the paper evaluates top-1 task accuracy and task survival, which help explain how multi-layer routing aggregation affects candidate task selection. The ablation study also examines the contributions of multi-layer routing aggregation and routing-guided head decision separately.

Critical weaknesses

w1. Limited methodological novelty.

The main methodological contribution of MATE is to extend the single-layer routing mechanism in MASS to a weighted aggregation of routing evidence from multiple layers. While this modification is reasonable and leads to some performance improvements, it is conceptually straightforward and represents a relatively incremental extension of the existing framework. Moreover, the experimental analysis focuses primarily on routing-layer selection and aggregation, without establishing a substantially new mechanism or broader methodological insight. Consequently, I find the level of novelty insufficient to justify the paper's contribution.

w2. Limited and insufficiently validated performance gains, particularly on larger backbones.

Although MATE consistently outperforms MASS in the reported results, the improvements become relatively small on ViT-L/14, reaching only 0.69, 0.63, and 0.16 percentage points in the 8-task, 14-task, and 20-task settings, respectively. In particular, the 0.16-point improvement raises concerns about whether the observed advantage is practically meaningful and robust to training variability. Without sufficient evidence from repeated runs, variance estimates, or statistical analysis, it is difficult to determine whether these marginal gains reflect a reliable improvement over MASS. Given the incremental nature of the proposed modification, the limited gains on larger models substantially weaken the paper's overall contribution.

Questions for authors

Q1. Beyond replacing single-layer routing with weighted multi-layer aggregation, what is the key methodological insight that distinguishes MATE from a straightforward extension of MASS? In particular, do the authors have evidence that multi-layer aggregation provides benefits beyond selecting a stronger individual routing layer?

Q2. Are the reported accuracy improvements averaged over multiple random seeds? If so, could the authors provide the standard deviations or confidence intervals, particularly for ViT-L/14? Given that the improvement over MASS is only 0.16 percentage points in the 20-task setting, understanding the variability of these results is important for assessing whether the gains are robust.

Q3. Could the authors clarify why the improvements over MASS become smaller on the larger ViT-L/14 backbone? Is this trend consistent across runs, and what does it imply about the scalability of the proposed routing mechanism?

3 comments
AuthorsOpenReview ↗

To Reviewer gMX9 (1/3)

W1 & Q1. Methodological contribution and the value of multi-layer routing aggregation

Thank you for raising this question. MATE builds on MASS’s dynamic model merging framework. Its contribution lies in identifying and using complementary routing evidence for both task and head selection, rather than in averaging multiple layers alone. We also clarify that Eq. (9) uniformly averages the selected layers’ routing scores and does not introduce learned layer weights.

First, MATE addresses the sensitivity of single-layer task routing to layer selection. We introduce the relative projection residual margin, which measures the gap between the relative projection residuals of the true task and its most competitive incorrect task. This metric quantifies layer-wise task separability. Its task-averaged form, Aˉℓ\bar A_\ell, guides the selection of informative candidate routing layers, whose routing scores are subsequently aggregated.

Second, MATE uses the aggregated routing evidence for classification head selection. Retaining the true task in the candidate set does not ensure that its classification head wins under simple max-logit competition. Our routing-guided head decision uses the aggregated task relevance scores to weight candidate head logits, allowing the same routing evidence to guide the final head decision.

Comparison with selecting a stronger individual layer

To examine whether multi-layer aggregation provides benefits beyond selecting a stronger single layer, we conducted a controlled comparison of five routing strategies. All variants use the same task models, TSV-M merging of the selected task subspaces, and routing-guided head decision, abbreviated as HG below. Only candidate layer selection and routing aggregation vary.

Table R1. Average classification accuracy (%) under different candidate layer selection strategies.

BackboneTasksSingle-Layer (MASS) + HGProxy-Best Single + HGRandom-5 + HGAll-MLP + HGMATE Top-5 + HG
ViT-B/32888.5288.2086.86 ± 2.4587.2288.91
ViT-B/321483.7783.6583.40 ± 1.7084.5585.09
ViT-B/322081.4377.8778.59 ± 1.0480.8982.25
ViT-B/16891.3191.3190.78 ± 0.5091.0591.44
ViT-B/161488.1783.9786.14 ± 3.1688.5488.55
ViT-B/162080.9580.5179.99 ± 3.0581.3682.95
ViT-L/14893.5593.7293.77 ± 0.0593.8093.66
ViT-L/141491.3991.6692.21 ± 0.4092.4591.58
ViT-L/142090.5590.0091.29 ± 0.9890.5391.00

Specifically:

  • Single-Layer (MASS) uses the original routing layer from MASS.
  • Proxy-Best Single selects the layer with the highest task-averaged margin Aˉℓ\bar A_\ell.
  • Random-5 randomly selects five MLP layers. We report the mean and standard deviation over three random seeds.
  • All-MLP aggregates all MLP layers.
  • MATE Top-5 selects the five layers with the highest task-averaged margins, following Eq. (8) with KL=5K_L=5.

With routing-guided head decision enabled in all variants, MATE Top-5 outperforms Single-Layer (MASS) in all nine configurations and Proxy-Best Single in seven. Thus, its gains generally extend beyond using the same proxy data to select a single layer with higher task separability.

On ViT-L/14 with 8 and 14 tasks, MATE Top-5 is slightly below Proxy-Best Single. However, both Random-5 and All-MLP exceed this single-layer baseline in these settings, providing additional evidence that combining routing information across layers can be beneficial.

Regarding layer selection, MATE Top-5 outperforms Random-5 and All-MLP in all six ViT-B/32 and ViT-B/16 configurations. Simpler multi-layer combinations perform better in some ViT-L/14 configurations. These results support the task-averaged margin as a useful layer-selection criterion, while showing that individual-layer separability does not guarantee the layer combination with the highest final accuracy.

Complementary evidence from the component ablation

The original ablation study provides a separate comparison: enabling multi-layer routing aggregation alone improves accuracy in eight of the nine configurations. Adding routing-guided head decision to this component further improves accuracy in seven configurations.

Together, these controlled comparisons and component ablations support the value of combining routing evidence beyond selecting a stronger individual layer. MATE uses the relative projection residual margin to guide the selection and aggregation of task-discriminative evidence, then carries the aggregated routing scores into classification head selection. These two mechanisms address complementary limitations in task routing and the final classification decision.

AuthorsOpenReview ↗

To Reviewer gMX9 (2/3)

W2 & Q2. Random seeds and the reliability of the reported gains

Thank you for highlighting the importance of variability. We first clarify that the classification accuracies in the original manuscript were obtained using fixed task-model weights, rather than averaged over independently trained task models.

MATE introduces no additional model training or parametric router optimization. Given fixed task models, selected routing layers, and inference conditions, online routing, task subspace merging, and classification head selection involve no random sampling. The main additional source of randomness introduced by MATE is therefore the selection of proxy examples for offline candidate layer selection.

Stability under proxy resampling

To examine this source of randomness, we repeated candidate layer selection using 8, 16, and 32 proxy examples per task across all three backbones and nine configurations, with five random seeds for each proxy size and configuration.

Table R2. Stability of Top-5 candidate layer selection across proxy sizes and random draws.

Proxy examples per taskConfigurationsSeeds per configurationRuns matching the main experimentMatch rate
89544/4597.8%
169545/45100%
32 (default)9545/45100%

A match means that the selected Top-5 layer set is identical to that used in the corresponding main experiment, regardless of the ordering within the set.

As shown in Table R2, candidate layer selection is stable under the tested proxy resampling conditions. Among the 135 runs, 134 recover the original candidate layer set. In particular, at the default size of 32 examples per task, all five seeds recover the same set in every configuration.

Since proxy data are used only for offline layer selection, identical candidate layer sets leave subsequent routing, task subspace merging, and head decisions unchanged when the task models and other inference conditions are fixed. These results indicate that the selected routing layers do not depend on a particular proxy draw in the tested default-setting runs.

We would like to distinguish this evidence from variation across independently trained task models. The additional experiment evaluates proxy-sampling stability, not classification accuracy variance across task-model training seeds. It therefore does not establish statistical significance for the smaller accuracy gains under training variability. Below, we further contextualize the ViT-L/14 results using the fine-tuned reference and the routing metrics reported in the paper.

AuthorsOpenReview ↗

To Reviewer gMX9 (3/3)

W2 & Q3. Interpreting the gains on ViT-L/14

Thank you for this question. One relevant factor is that MASS already achieves high accuracy on ViT-L/14. Absolute accuracy gains alone do not capture how much of the remaining gap to the independently fine-tuned models is recovered.

A central goal of model merging is to preserve the task-specific performance of independently fine-tuned models within a unified model. Accordingly, we use fine-tuned accuracy as a reference and quantify MATE's improvement by measuring the fraction of MASS's remaining performance gap that MATE closes:

GapReduction=AccMATE−AccMASSAccFT−AccMASS×100\mathrm{Gap\\ Reduction} = \frac{ \mathrm{Acc}_{\mathrm{MATE}} - \mathrm{Acc}_{\mathrm{MASS}} }{ \mathrm{Acc}_{\mathrm{FT}} - \mathrm{Acc}_{\mathrm{MASS}} } \times 100\\%.

Here, AccFT\mathrm{Acc}_{\mathrm{FT}} denotes the average accuracy of the independently fine-tuned models.

Table R3. Accuracy gaps to the fine-tuned reference on ViT-L/14.

TasksFine-tuned (%)MASS (%)MATE (%)MASS gap (pp)MATE gap (pp)Gap reduction
894.2392.9793.661.260.5754.8%
1493.4390.9591.582.481.8525.4%
2094.0090.8491.003.163.005.1%

In the 8-task and 14-task settings, MATE reduces the gap between MASS and the fine-tuned reference by 54.8% and 25.4%, respectively. Although the absolute accuracy gains are 0.69 and 0.63 percentage points, they recover substantial portions of the remaining performance gap. While we have not conducted statistical significance tests, we believe this gap-based comparison provides a more informative perspective on MATE's improvements, which may be underestimated when considering absolute accuracy gains alone.

Routing improvements in the 20-task setting

The final accuracy gain in the 20-task setting is smaller, at 0.16 percentage points. Nevertheless, Table 2 of the original paper shows that MATE improves both top-1 task accuracy and task survival by 0.95 percentage points on ViT-L/14. These results indicate improved task identification and true-task retention even when the gain in final classification accuracy is modest.

The component ablation in Table 3 further shows that multi-layer routing aggregation alone raises accuracy from 90.84% to 91.11%. Adding routing-guided head decision reduces this to 91.00%, partially offsetting the gain from multi-layer routing aggregation. Thus, the smaller improvement of the full method does not imply that multi-layer routing aggregation ceases to be useful on the larger backbone.

Overall, MATE narrows the gap to the fine-tuned reference in the 8-task and 14-task settings and improves task identification and true-task retention in the 20-task setting. These findings support the usefulness of multi-layer routing aggregation on the evaluated larger backbone.

Reviewer_x2ddOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes MATE to address the routing-sensitivity challenge in the primary method, MASS. Based on MASS, it adds Multi-Layer Routing Aggregation and Routing-Guided Head Decision. Specifically, it computes task-separation margins on a small proxy set (e.g., 32 examples per task), selects the most informative MLP layers, and averages their routing distributions. The final prediction maximizes candidate head logits multiplied by this aggregated routing score. Results show that MATE outperforms MASS on image classification tasks.

Critical strengths
  • The paper identifies that different tasks favor different routing layers, and that retaining the correct task subspace does not guarantee its head wins the max-logit competition.
  • It proposes a valid method to improve MASS performance.
Critical weaknesses
  1. The work heavily relies on MASS, offering incremental enhancements that are hard to transfer beyond image classification. The writing is poor and incomprehensible without the MASS context. Furthermore, the public MASS repo already standardizes residual logits by per-input mean/std (similar to Eq. 5). It would be better to explicitly clarify the methodological differences from MASS.

  2. Using 32 proxy examples violates the data-free premise of baselines like TA, TSV-M, and MASS, making direct comparisons unfair. Authors should compare against small-data methods (e.g., Ada-Merging, Twin Merging) and add simple baselines like last-five/all-layer averaging or averaging residuals before softmax to isolate the aggregator's effect.

  3. The sensitivity analysis ignores proxy size, random draws, and layer count. Since random 32 examples cannot cover all classes in datasets like DTD, it would be better to report performance variance to prove the gains.

  4. The paper omits end-to-end computational and memory costs. The algorithm requires two forward passes and massive SVD computations. Furthermore, heterogeneous per-input routing incurs high storage costs for merged models and SVD matrices.

  5. While Eq. 5 uses normalized residuals, Eqs. 9 revert to unnormalized absolute residuals. Averaging wl,iw_{l,i} is thus highly sensitive to the absolute numerical scale of ri,lr_{i,l}, meaning layers with larger magnitudes will dominate.

  6. Equations (11)-(12) multiply task scores by raw logits, causing severe instability. Raw logits are highly sensitive to head-specific absolute value offsets. In the paper's own ablation study (Table 3), adding Head Guidance actually degrades performance (e.g., from 83.36% to 82.95% on ViT-B/16, and 91.11% to 91.00% on ViT-L/14).

Questions for authors

See Weakness

8 comments
AuthorsOpenReview ↗

To Reviewer x2dd (1/5)

W1. Methodological differences from MASS and clarity of presentation

Thank you for this constructive feedback. We agree that MATE builds on MASS’s task-subspace representations and two-pass dynamic merging framework. Our work addresses two routing limitations within this framework, as detailed below.

First, MATE leverages complementary task-routing evidence from representations at multiple network layers. We introduce the relative projection residual margin, defined as

mi,ℓ(x)=min⁡j≠irˉj,ℓ(x)−rˉi,ℓ(x),m_{i,\ell}(x) = \min_{j\neq i}\bar r_{j,\ell}(x)-\bar r_{i,\ell}(x),

where the relative projection residual is given by

rˉi,ℓ(x)=ri,ℓ(x)∥zℓ(x)∥2+ϵ.\bar r_{i,\ell}(x) = \frac{r_{i,\ell}(x)}{\lVert z_\ell(x)\rVert_2+\epsilon}.

This margin quantifies how well each layer distinguishes the target task from its closest competing task. Based on this criterion, MATE selects informative candidate layers and aggregates their routing scores (Eqs. (5)–(9)). In contrast, MASS performs task routing using projection residuals from a single preselected layer, making its routing performance sensitive to layer selection.

Second, MATE improves classification-head selection by incorporating task-relevance evidence into the final prediction. MASS selects the task head and class associated with the largest raw logit:

(i∗,c∗)=arg⁡max⁡i∈Ω(x),,c∈Ci;oi(x)[c].(i^{\ast},c^{\ast}) = \underset{i\in\Omega(x),\\,c\in\mathcal{C}_i}{\arg\max}\\;o_i(x)[c].

However, this rule may select an incorrect task head even when the true task is retained in the candidate set. In contrast, MATE uses the aggregated routing scores to guide competition among candidate heads. Specifically, it reweights each head's logits as

o~i(x)=si(x),oi(x),\widetilde{o}_i(x)=s_i(x)\\,o_i(x),

and makes the final prediction according to

(i∗,c∗)=arg⁡max⁡i∈Ω(x),,c∈Ci;o~i(x)[c].(i^{\ast},c^{\ast}) = \underset{i\in\Omega(x),\\,c\in\mathcal{C}_i}{\arg\max}\\;\widetilde{o}_i(x)[c].

Here, the weighting factor is the aggregated task-relevance score defined in Eq. (9), and the head-selection procedure corresponds to Eqs. (11)–(12). In this way, routing evidence contributes not only to candidate-task selection but also to the final classification-head decision.

Importantly, we would like to emphasize that although residual standardization is also used in MASS's implementation, the key contribution of our Relative Projection Residual Margin lies in the margin criterion, rather than normalization itself. As defined above, the margin measures the separation between the true task and its closest competing task, providing a discriminative criterion for evaluating layer-wise task separability and selecting informative routing layers.

We appreciate your careful observation regarding this implementation detail. MATE was developed based on the public MASS codebase. After rechecking the code, we confirmed that its topk routing mode applies per-input Z-score standardization across tasks before Softmax. Our Eq. (4) followed the formulation in the MASS paper, which did not explicitly include this step. We will correct Eq. (4) accordingly and clarify the actual routing-score computation in the revised manuscript.

AuthorsOpenReview ↗

To Reviewer x2dd (2/5)

W2. Proxy-data requirements and additional aggregation baselines

We agree that MATE is not strictly data-free. Below, we clarify the proxy-data usage and present more experiments as you suggest.

Proxy-data usage

MATE randomly samples 32 examples from each task’s training set to estimate layer-wise task separability and select candidate routing layers offline. It requires task identities, but no class labels or parameter updates. After layer selection, inference no longer accesses proxy data.

This usage differs from AdaMerging, which optimizes merging coefficients using unlabeled test data, and Twin-Merging, which uses validation data to train a learnable router. MATE instead uses a small training subset for one-time layer selection, without additional router training or merging-coefficient optimization. Therefore, these methods operate under different data-access and optimization settings, making direct performance comparisons difficult to interpret as fair, controlled comparisons.

Candidate layer selection under matched conditions

To assess multi-layer routing aggregation, we compare five strategies: MASS Single (the original routing layer), Proxy-Best Single (the layer with the highest task-averaged margin), Random-5 (five randomly selected MLP layers), All-MLP (all MLP layers), and MATE Top-5 (the five layers with the highest task-averaged margins). All variants share the same task models, dynamic TSV-M merging, and routing-guided head decision (HG), differing only in routing-layer selection and aggregation. Random-5 reports the mean and standard deviation over three layer-selection seeds.

Table R1. Average classification accuracy (%) under different candidate layer selection strategies.

BackboneTasksMASS Single + HGProxy-Best Single + HGRandom-5 + HGAll-MLP + HGMATE Top-5 + HG
ViT-B/32888.5288.2086.86 ± 2.4587.2288.91
ViT-B/321483.7783.6583.40 ± 1.7084.5585.09
ViT-B/322081.4377.8778.59 ± 1.0480.8982.25
ViT-B/16891.3191.3190.78 ± 0.5091.0591.44
ViT-B/161488.1783.9786.14 ± 3.1688.5488.55
ViT-B/162080.9580.5179.99 ± 3.0581.3682.95
ViT-L/14893.5593.7293.77 ± 0.0593.8093.66
ViT-L/141491.3991.6692.21 ± 0.4092.4591.58
ViT-L/142090.5590.0091.29 ± 0.9890.5391.00

As shown in Table R1, MATE Top-5 outperforms MASS Single in all nine configurations and Proxy-Best Single in seven. The gains therefore generally extend beyond proxy-based selection of a better single layer. Combining multi-layer routing evidence also improves accuracy. MATE Top-5 outperforms Random-5 and All-MLP in all six ViT-B configurations, but not consistently on ViT-L/14. The relative projection residual margin thus supports layer selection, although individual-layer separability does not guarantee the most accurate layer combination.

Aggregation before versus after softmax

Following your suggestion, we compare three aggregation strategies:

  • A. Post-Softmax (original MATE): apply Z-score standardization and softmax independently at each layer, then average the routing probabilities.
  • B. Pre-Softmax Standardized: apply Z-score standardization at each layer, average the standardized scores, then apply softmax.
  • C. Pre-Softmax Raw: average the raw negative projection residuals, then apply softmax.

All three strategies share the Top-5 candidate layers, dynamic merging, and head selection.

Table R2. Classification accuracy (%) under different routing aggregation strategies.

BackboneTasksA. Post-SoftmaxB. Pre-Softmax StandardizedC. Pre-Softmax Raw
ViT-B/32888.9188.9188.96
ViT-B/321485.0985.1684.83
ViT-B/322082.2582.2482.11
ViT-B/16891.4491.4291.30
ViT-B/161488.5588.7488.47
ViT-B/162082.9583.1283.20
ViT-L/14893.6693.6693.72
ViT-L/141491.5891.6191.63
ViT-L/142091.0091.0190.71
Average—88.3888.4388.33

The three strategies achieve similar average accuracies of 88.38%, 88.43%, and 88.33%. Pre-Softmax Standardized slightly exceeds MATE on average. These results suggest that multi-layer routing benefits do not depend on softmax aggregation order in the tested configurations. Together with Table R1, they support MATE’s contribution of selecting and combining task-discriminative evidence from multiple layers, rather than establishing a universally superior aggregation operator. We will clarify proxy-data requirements and add these layer-selection and aggregation comparisons to the revision.

AuthorsOpenReview ↗

To Reviewer x2dd (3/5)

W3. Sensitivity to proxy size, random sampling, and layer count

We appreciate this suggestion. We conducted additional experiments across three ViT backbones and all nine task configurations to examine proxy-size sensitivity, sampling stability, and candidate layer count.

Proxy size and random sampling

We recomputed the relative projection residual margins using 8, 16, and 32 proxy examples per task and compared the resulting Top-5 layer sets. For each proxy size, we used five random seeds across all nine configurations.

Table R3. Stability of Top-5 layer selection across proxy sizes and random draws.

Proxy examples per taskConfigurationsSeeds per configurationRuns matching the main experimentMatch rate
89544/4597.8%
169545/45100%
32 (default)9545/45100%

A match means that the selected Top-5 layer set is identical to that used in the corresponding main experiment, regardless of the ordering within the set.

As shown in Table R3, layer selection remains stable across the tested proxy sizes and random draws. Among the 135 runs, 134 recover the original Top-5 set, with only one run using eight examples changing a single selected layer. Notably, both 16 and 32 examples per task yield identical candidate layer sets across all nine configurations and five random seeds.

We used 32 examples per task as a conservative default rather than a performance-tuned optimum. Reducing the proxy size to 16 preserves the selected layers in all tested runs, indicating that MATE does not depend on the specific choice of 32 examples. Since proxy data are used only for offline layer selection, identical candidate layer sets lead to the same subsequent inference configuration when the models and other inference conditions are fixed.

Regarding DTD, we acknowledge that 32 examples cannot cover all 47 classes. However, the relative projection residual margin measures separability between task subspaces rather than within-task classification performance. Its computation requires task identities but not complete class coverage. The results above support stable layer selection with small proxy sets, without implying that these samples fully represent the within-task class distribution.

Candidate layer count

Next, we fixed the layer rankings obtained from the default proxy sets and evaluated different candidate layer counts. Dynamic TSV-M merging, candidate task selection, and routing-guided head decision remained unchanged. Only the number of layers included in routing aggregation varied.

Table R4. Classification accuracy (%) for different candidate layer counts (K_L).

BackboneTasks123458All
ViT-B/32888.2088.5088.2788.7588.9188.8787.22
ViT-B/321483.6583.3684.0684.5585.0985.4684.55
ViT-B/322077.8780.7981.8681.9882.2580.9980.89
ViT-B/16891.3191.2591.0091.3191.4491.2991.05
ViT-B/161483.9787.5387.5188.1388.5589.0188.54
ViT-B/162080.5183.5083.2183.0882.9581.7781.36
ViT-L/14893.7293.6593.7193.6893.6693.8293.80
ViT-L/141491.6691.5691.4291.3591.5892.0292.45
ViT-L/142090.0090.4190.7490.7791.0091.4190.53
Average—86.7787.8487.9888.1888.3888.2987.82

Average denotes the equally weighted mean across the nine configurations, and All denotes aggregation over all MLP layers.

Increasing the candidate layer count from one to five raises average accuracy from 86.77% to 88.38%, a gain of approximately 1.61 percentage points. Further increasing the count to eight or all MLP layers yields 88.29% and 87.82%, respectively, without additional average improvement.

Although the optimal count varies by configuration, KL=5K_L=5 achieves the highest cross-configuration average among the tested choices while avoiding the routing overhead of aggregating more layers. We therefore use Top-5 as a uniform default balancing overall accuracy and computational cost, rather than claiming that it is optimal in every configuration.

We will add these sensitivity results and clarify the effects of proxy size and random sampling, together with the rationale for the default layer count.

Comments on the paper

1 comment
AuthorsOpenReview ↗

To all Reviewers and Area Chair

Dear Reviewers and Area Chair,

We sincerely thank you for your time and thoughtful comments. We have responded to your questions point by point and revised the manuscript accordingly. We would be grateful if you could let us know whether our responses and revision have addressed your concerns. We welcome any further questions and would be happy to provide additional clarification.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.