acceptodds
Under review as a conference paper at ICLR 2027

Merge Behavior, Not Just Weights: Data-Free Model Merging via Subspace Reconstruction

Abstract

Model merging combines fine-tuned models without retraining, but success ultimately depends on whether the merged model behaves similarly to each expert on its own task. Existing methods largely operate on task-vector weights or their singular subspaces, but without task inputs, they lack effecttive constraints on the merged model's output behavior. We therefore ask how to preserve expert responses without using task inputs during merging? Starting from the observation that matching responses within an expert's output subspace reconstructs its projected output behavior, we construct data-free, layer-local response targets from task vectors and propose Subspace Behavior Reconstruction (SBR). SBR reconstructs each expert's self-response within its task-specific input and output spaces while suppressing unintended cross-task responses. To account for capabilities that may be shared across tasks, we further introduce SBR-R, which preserves cross-task responses on which the two experts agree. Extensive experiments demonstrate that SBR-R more closely preserves expert predictive distributions in CLIP diagnostics and consistently outperforms prior state-of-the-art data-free methods in average performance on CLIP, Qwen-14B LoRA, and InternVL2.5.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Model Merging, Multitask Learning, Transfer Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dXgWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Subspace Behavior Reconstruction (SBR), a method for merging task-specific models without using task data. The authors argue that preserving expert responses is more useful than directly matching weight updates. They use the leading singular vectors of each task update to define input and output subspaces. SBR reconstructs each expert's response within its own subspaces and suppresses responses across different experts. The extended method, SBR-R, retains cross-task responses when the corresponding experts show positive agreement. A separate component handles input directions outside the selected subspaces. The authors evaluate their methods on CLIP, Qwen-14B LoRA, InternVL2.5, and BERT.

Critical strengths

The paper is well motivated, and the proposed method follows naturally from the analysis of expert responses. The distinction between self-responses and cross-responses is useful for understanding what the merging objective controls. Both methods construct their targets from model weights, so task data are not required during merging. The experiments cover several model architectures and include comparisons with recent methods. SBR-R consistently improves average accuracy over the strongest reported external baseline on all three CLIP backbones. The component ablations help explain the contributions of the different loss terms. The authors also examine internal responses and final predictions using real inputs after merging, which provides useful evidence beyond benchmark accuracy.

Critical weaknesses

The theoretical analysis mainly concerns layer-wise projected responses. The resulting bound can be loose and includes a target-mismatch term, so it offers limited guidance on whether the chosen cross-response targets improve downstream predictions. The positive-agreement rule is reasonable, but agreement between weight-derived responses does not necessarily indicate useful shared task behavior. The experiments show encouraging improvements, although the margins over strong baselines are relatively small on some architectures. The reported results also depend on the numerical solver. In particular, the dense-model accuracy can decrease substantially as the solver approaches convergence, suggesting that early stopping contributes significantly to performance. Finally, the use of fixed expert checkpoints limits the assessment of statistical variability, and some comparisons involve different output adapter capacities.

Questions for authors

I have some questions regarding the experimental setup and symbols that I would like to discuss with the author(s).

  1. In Eq. (21), since UkU_k and VkV_k are singular vectors of WkW_k and span(Vk)⊆span(Z)\mathrm{span}(V_k)\subseteq\mathrm{span}(Z), we have Uk⊤WkPZ⊥=0U_k^\top W_kP_{Z^\perp}=0. Thus, the first term appears to penalize responses along the retained output directions rather than reconstruct nonzero expert tail responses. Is this interpretation correct? Could the authors clarify the role of this term in Generalist calibration?

  2. The solver-depth analysis shows that better convergence does not necessarily lead to better downstream accuracy. How much of the reported improvement comes from the proposed response targets, and how much comes from the regularization introduced by finite-step optimization? It would help to clarify how sensitive the relative advantage of SBR-R is to the stopping budget.

  3. In the Qwen experiments, several baselines use a smaller output adapter rank than SBR-R. Could the authors clarify how much this capacity difference affects the comparison? A capacity-matched analysis would help isolate the benefit of the proposed reconstruction objective.

  4. The InternVL experiments use a global merge scale selected through preliminary comparisons on the evaluation benchmarks. Could the authors clarify the scope of the data-free claim regarding hyperparameter selection? It would also be useful to know whether the relative performance trends remain similar under an independently chosen scale.

6 comments
AuthorsOpenReview ↗

Q1: The Role of the Projected-Response Term in Generalist Calibration

Thank you for your thoughtful feedback and for the effort you devoted to the review. Below, we respond to each of your questions, describe the additional analyses we carried out, and explain how we revised the manuscript in response.

In general, generalist calibration used to align responses to inputs outside the joint expert subspace, and Uk⊤WkPZ⊥U_k^\top W_kP_{Z^\perp} is written in this form to parallel the Specialists objective; SBR-R outperforms SBR at matched CG budgets and SWUDI at the same Qwen LoRA output rank; and for InternVL, we use the same scaling coefficient ss for all methods without tuning it separately for ours, ensuring a fair comparison. We provide the details below.

Q1: The Role of the Projected-Response Term in Generalist Calibration

Thank you for pointing this out. Your interpretation is correct. Under the paper's construction, Uk⊤WkPZ⊥=0U_k^\top W_kP_{Z^\perp}=0, so the first term in Eq. (21) is indeed a zero-target constraint.

We wrote Eq. (21) in this form to make it easier to compare with the corresponding Specialists objective. We have clarified this point in the revised manuscript. Specifically, Generalist calibration aligns responses to inputs projected onto less important subspace Z⊥Z^\perp by PZ⊥P_{Z^\perp}, while Specialists calibration aligns responses to inputs projected onto more important subspace ZZ by PZP_Z.

For inputs in Z⊥Z^\perp, the two terms of the Generalist objective address different aspects of the output. The first term, which you asked about, concerns the output directions UkU_k of each expert. Since Uk⊤WkPZ⊥=0U_k^\top W_kP_{Z^\perp}=0, this term suppresses the merged response along UkU_k to prevent inputs in Z⊥Z^\perp from interfering with these directions.

The second term helps preserve responses outside the retained output directions Uk⊤U_k^\top. Since these directions carry less information about individual experts' task-specific behavior, we align the full merged response with the sum of the expert responses to retain behavior that can be useful across tasks.

We hope this explanation clarifies the role of the two terms and our Generalist calibration. We have revised the manuscript to make this distinction clearer and would be happy to discuss any remaining questions.

AuthorsOpenReview ↗

Q2: Contributions of Response Targets and Early Stopping

Q2: Contributions of Response Targets and Early Stopping

Thank you for raising this question. Below, we further analyze why better convergence does not necessarily lead to better downstream accuracy. Our results indicate that the gains come primarily from the proposed response targets, while finite-step optimization helps avoid shortcuts during optimization. We also propose an explicit regularization method to achieve the same goal.

To investigate the influence of solver-depth, we measured the Specialists update norms on eight-task ViT-B/32. For the original objective, we varied only the CG budget. Here the update magnitude below is the square root of the sum of squared Frobenius norms across the 50 visual matrices. The table below summarizes the supplementary results.

Solver settingSBR update magnitudeSBR accuracy (%)SBR-R update magnitudeSBR-R accuracy (%)
Original objective, 10 steps6.210686.49525.733187.1617
Original objective, 25 steps10.394287.09199.296887.7171
Original objective, 50 steps13.860286.487511.761087.3297
Original objective, 100 steps17.535084.195513.854886.5754
Original objective, residual-certified convergence28.090362.988717.987982.0149
Full-expert-energy constraint7.170986.67687.170987.5763

As CG runs for more steps, the Specialists updates in the merged model steadily grow in magnitude. Accuracy peaks at 25 steps and then declines, with the largest updates and lowest accuracies observed at convergence. We suspect that increasing the update magnitude offers a shortcut for reducing the proxy loss, which may explain why solving the original objective to convergence hurts downstream accuracy. To curb this growth, we introduce a full-expert-energy constraint on the Specialists update.

Specifically, we keep the original Specialists loss from Eq. (20). Suppressing layer indices for readability, for KK experts and $v\in\lbrace \mathrm{SBR},\mathrm{SBR\text{-}R}\rbrace $, the loss is

\mathcal L_s^v(W_s) &=\frac{1}{K}\sum_{i=1}^{K} \big||U_i^\top W_s V_i-U_i^\top W_i V_i\big||_F^2\\\\ &\quad+\frac{1}{\sqrt K}\sum_{i=1}^{K}\sum_{j\ne i} \big||U_j^\top W_s V_i-T_{ji}^v\big||_F^2 . \end{aligned}$$ Here, $U_i$ and $V_i$ are expert $i$'s retained output and input bases. The first term matches each expert's self-response, and the second matches the chosen cross-response target $T_{ji}^v$. At each layer, we minimize Eq. (20) over the Specialists update $W_s$ in the joint input and output spaces, subject to $$||W_s||_F \le \sqrt{\sum_{i=1}^{K}||W_i||_F^2}.$$ The right-hand side is the combined norm of the experts' full weight updates from the shared base at that layer, before truncation. The solver stops once the residual and norm-budget checks pass. With this constraint, the mean accuracies of SBR and SBR-R in the eight-task ViT-B/32 experiment recover from 62.99% and 82.01% at unconstrained convergence to 86.68% and 87.58%, respectively, compared with **86.49% and 87.33% at 50 CG steps. This removes the need to choose a fixed CG iteration count, as the norm budget comes from the expert weights, and numerical convergence determines when to stop without consulting task data. In General, the matched comparisons indicate that the **response target is the primary driver of our improvement, while finite-step optimization acts as implicit regularization by limiting the solver’s tendency to reduce the reconstruction loss through excessively large weight updates**.
AuthorsOpenReview ↗

Q3: Output Capacity in Qwen LoRA Comparisons

Q3: Output Capacity in Qwen LoRA Comparisons

Thank you for raising this point. We use a rank-128 output adapter because it reflects the core idea of SBR-R: a larger optimization space allows us to better approximate each expert’s core subspace. Several baselines in the table, including TSV-M and Iso-CTS, also output rank-128 adapters.

However, SBR-R can also produce a rank-32 adapter in two ways: by truncating the rank-128 adapter to rank 32 after merging, or by optimizing directly under a rank-32 constraint. The table below shows the results:

MethodOutput rankMMLUTruthfulQABBQCNN/DMFour-task mean
Original SBR-R12867.1063.6689.7220.2260.18
SBR-R truncated to rank 323267.1564.1688.7220.6960.18
SBR-R directly solved at rank 323266.3064.1689.2220.0959.94
TSV-M12865.9461.4090.2318.6759.06
Iso-CTS12867.4064.6686.4720.1159.66
SWUDI3266.6163.6689.4719.4459.80

At the same rank of 32, our SBR-R still exceeds other SOTA methods, which can full demostrate our robust improvement and show this capacity difference does not affect the comparison results, ours still perform consist advantage over other SOTA methods. Whereas, truncating the rank-128 adapter to rank 32 changes the mean score by less than 0.01 points. This small change is consistent with how much of the update the truncation retains: across 200 layers, the first 32 singular directions preserve a median of 97.47% of the squared Frobenius norm of the update.

To further assess how much capability affects our SBR-R, we also ran two experiments that constrained the rank during merging to 64 and 80. The resulting mean scores were 60.13 and 60.19, respectively, both close to the rank-128 result (60.18). This also suggests that the full rank-128 capacity is not required to reach a similar score in this setting.

Reviewer_SLzjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Subspace Behavior Reconstruction (SBR) and its extension SBR-R for data-free model merging. The key idea is to preserve expert behavior by matching responses within expert-specific output subspaces, rather than only operating on task-vector weights. Experiments on CLIP, Qwen-14B LoRA, and InternVL2.5 show improvements over prior data-free methods.

Critical strengths

1.The paper addresses the important challenge of preserving task-specific capabilities when merging models without access to task data. Its focus on behavioral consistency provides a meaningful perspective beyond conventional weight-space aggregation.

2.Experiments cover CLIP, Qwen-14B LoRA, and InternVL2.5, with comparisons against multiple data-free merging baselines and ablations of the main components.

Critical weaknesses
  1. Potentially unfair comparison in Qwen LoRA experiments. SBR/SBR-R use output rank 128, whereas several baselines use rank 32. Although the paper acknowledges this difference, the lack of a matched-capacity comparison makes it difficult to isolate the gains attributable to the proposed merging strategy.

  2. SBR-R achieves 87.72% accuracy at 25 CG steps, compared with 87.33% at the reported 50 steps and 82.01% at convergence. This substantial variation suggests that finite-step optimization acts as an important part of the method. More systematic comparisons across CG budgets, including matched-budget comparisons between SBR and SBR-R, would clarify the robustness and source of the reported gains.

  3. Theorem 1 bounds joint-space response error but does not directly guarantee task-level performance preservation. Although the paper provides empirical behavioral validation on held-out inputs, the evidence does not establish that minimizing the proposed proxy reliably predicts downstream accuracy across tasks and architectures.

  4. Insufficient validation of the cross-response target rule. SBR-R constructs cross-response targets using Frobenius inner-product-based agreement. The theoretical analysis justifies this rule within a restricted class of radial targets, but does not establish that it is preferable to other plausible target constructions or that positive response agreement reliably indicates beneficial shared behavior. Direct comparisons with alternative target rules would strengthen this design choice.

  5. Limited evidence of statistical robustness. The experiments rely on fixed checkpoints without repeated runs or uncertainty estimates. Given the modest improvements in some settings, it remains unclear whether these gains are robust to checkpoint or training variation.

Questions for authors

See weaknesses.

8 comments
AuthorsOpenReview ↗

W1: Capacity-Matched Qwen LoRA Comparisons

We appreciate the time and care you put into reviewing our work. Your comments helped us see where the paper needed more explanation or evidence. We respond to each point below and have incorporated the corresponding changes into the revised manuscript.

In general, our additional comparisons show that our SBR-R outperforms all other SOTA methods at the same Qwen LoRA output rank. Besides, SBR-R has stable improvement over SBR over various situations, highlighting the benefit of its cross-response design. Large-scale experiments also demostrate that response error is a good proxy for performance prediction across various architectures and tasks.

W1: Capacity-Matched Qwen LoRA Comparisons

Thank you for raising this point. We use a rank-128 output adapter because it reflects the core idea of SBR-R: a larger optimization space allows us to better approximate each expert’s core subspace. Several baselines in the table, including TSV-M and Iso-CTS, also output rank-128 adapters.

However, SBR-R can also produce a rank-32 adapter in two ways: by truncating the rank-128 adapter to rank 32 after merging, or by optimizing directly under a rank-32 constraint. The table below shows the results:

MethodOutput rankMMLUTruthfulQABBQCNN/DMFour-task mean
Original SBR-R12867.1063.6689.7220.2260.18
SBR-R truncated to rank 323267.1564.1688.7220.6960.18
SBR-R directly solved at rank 323266.3064.1689.2220.0959.94
TSV-M12865.9461.4090.2318.6759.06
Iso-CTS12867.4064.6686.4720.1159.66
SWUDI3266.6163.6689.4719.4459.80

At the same rank of 32, our SBR-R still exceeds other SOTA methods, which can full demostrate our robust improvement and show this capacity difference does not affect the comparison results, ours still perform consist advantage over other SOTA methods. For the truncated variant, truncating the rank-128 adapter to rank 32 changes the mean score by less than 0.01 points. This small change is consistent with how much of the update the truncation retains: across 200 layers, the first 32 singular directions preserve a median of 97.47% of the squared Frobenius norm of the update.

To further assess how much capability affects our SBR-R, we also ran two experiments that constrained the rank during merging to 64 and 80. The resulting mean scores were 60.13 and 60.19, respectively, both close to the rank-128 result (60.18). This also suggests that the full rank-128 capacity is not required to reach a similar score in this setting.

AuthorsOpenReview ↗

W2: Sensitivity to CG Budgets and Matched Solver Comparisons

W2: Sensitivity to CG Budgets and Matched Solver Comparisons

Thank you for highlighting the role of early stopping. To assess its effect, we supplement comparsion between SBR with SBR-R at several matched CG budgets, which shows the consistent improvement of SBR-R compared with SBR.

In the original submission, we did not treat the number of CG iterations as a tunable hyperparameter. Actually, we used fix CG steps throughout all main experiments, though tuning this can get much stronger results than that we reported in the paper. In response to the reviewer's comment, we further analyzed the drop in downstream accuracy observed at higher iteration counts.

To investigate this, we measured the Specialists update norms on eight-task ViT-B/32. For the original objective, we varied only the CG budget. Here the update magnitude below is the square root of the sum of squared Frobenius norms across the 50 visual matrices. The table below summarizes the supplementary results.

Solver settingSBR update magnitudeSBR accuracy (%)SBR-R update magnitudeSBR-R accuracy (%)
Original objective, 10 steps6.210686.49525.733187.1617
Original objective, 25 steps10.394287.09199.296887.7171
Original objective, 50 steps13.860286.487511.761087.3297
Original objective, 100 steps17.535084.195513.854886.5754
Original objective, residual-certified convergence28.090362.988717.987982.0149
Full-expert-energy constraint7.170986.67687.170987.5763

As CG runs for more steps, the Specialists updates in the merged model steadily grow in magnitude. Accuracy peaks at 25 steps and then declines, with the largest updates and lowest accuracies observed at convergence. We suspect that increasing the update magnitude offers a shortcut for reducing the proxy loss, which may explain why solving the original objective to convergence hurts downstream accuracy. To curb this growth, we introduce a full-expert-energy constraint on the Specialists update.

Specifically, we keep the original Specialists loss from Eq. (20). Suppressing layer indices for readability, for KK experts and $v\in\lbrace \mathrm{SBR},\mathrm{SBR\text{-}R}\rbrace $, the loss is

\mathcal L_s^v(W_s) &=\frac{1}{K}\sum_{i=1}^{K} \big||U_i^\top W_s V_i-U_i^\top W_i V_i\big||_F^2\\\\ &\quad+\frac{1}{\sqrt K}\sum_{i=1}^{K}\sum_{j\ne i} \big||U_j^\top W_s V_i-T_{ji}^v\big||_F^2 . \end{aligned}$$ Here, $U_i$ and $V_i$ are expert $i$'s retained output and input bases. The first term matches each expert's self-response, and the second matches the chosen cross-response target $T_{ji}^v$. At each layer, we minimize Eq. (20) over the Specialists update $W_s$ in the joint input and output spaces, subject to $$||W_s||_F \le \sqrt{\sum_{i=1}^{K}||W_i||_F^2}.$$ The right-hand side is the combined norm of the experts' full weight updates from the shared base at that layer, before truncation. The solver stops once the residual and norm-budget checks pass. With this constraint, the mean accuracies of SBR and SBR-R in the eight-task ViT-B/32 experiment recover from **62.99% and 82.01%** at unconstrained convergence to **86.68% and 87.58%**, respectively, compared with **86.49% and 87.33%** at 50 CG steps. These demostrate the consistent improvement of SBR-R compared with SBR. Besides, this also removes the need to choose a fixed CG iteration count, as the norm budget comes from the expert weights, and numerical convergence determines when to stop without consulting task data.
AuthorsOpenReview ↗

W3: Projected-Response Bounds and Downstream Performance

W3: Projected-Response Bounds and Downstream Performance

Thank you for pointing this out. Overall, we view preserving layer responses as a useful proxy for preserving downstream task performance in model merging. Matching intermediate responses is actually an established design choice: RegMean (ICLR ’23) matches linear-layer outputs, while ProDistill (ICML ’25) distills expert behavior layer by layer. This motivates our response reconstruction objective, which aims to preserve expert performance by approximating expert responses layer by layer.

Following the approach in Fig. 1, we measured core response error on real inputs and compared it with downstream task performance. A negative within-task rank correlation means lower error tends to accompany better downstream task performance.

ModelTasksMethod–task pairsRank correlationLower error and higher score
CLIP ViT-B/32848−0.6296/120
Qwen-14B LoRA424−0.5745/60
InternVL2.5-1B520−0.4421/30

Across the three settings, lower core response error was generally associated with better downstream task performance within the same task, supporting its use as an empirical proxy.

Reviewer_cyQbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SBR and SBR-R for data-free merging of task vectors. Using the leading right and left singular vectors to define proxy input and output subspaces, the method splits merging into two parts: a specialist objective that reconstructs each expert's self-response while controlling pairwise cross-responses, and a generalist solve that reconstructs the complementary input subspace. SBR suppresses cross-responses toward zero, whereas SBR-R retains a source response in proportion to its positive agreement with the peer expert. A frame-bound analysis relates these proxy losses to joint output-subspace error, including an explicit target-mismatch term. In the experiments, dense models are merged using a fixed 50-step reduced conjugate gradient (CG) solver, while Qwen LoRA uses a residual-certified PCG solve. Evaluations cover three CLIP backbones, a 14-task CLIP benchmark, Qwen-14B LoRA, InternVL2.5, and BERT. The empirical gains are most consistent on CLIP, whereas improvements over the strongest baselines on Qwen and InternVL are relatively modest.

Critical strengths
  1. The paired-response formulation cleanly separates self-response reconstruction from cross-response target selection, and the specialist/generalist split makes the underlying subspace constraints explicit. Moreover, Theorem 1 explicitly accounts for the mismatch between the chosen targets and the original expert responses rather than assuming the targets are ideal. This offers a meaningful modeling step beyond simply changing the solver for the WUDI objective [1, 2].

  2. The empirical results on CLIP are consistent across backbones and numbers of tasks. Relative to SWUDI, SBR-R improves average accuracy by 1.31, 1.24, and 0.96 percentage points across the three backbones, and reaches 79.59% vs. 77.92% on the 14-task benchmark. Holding the truncation rank and 50-step solver fixed, replacing SBR's zero target with SBR-R's agreement target improves ViT-B/32 accuracy from 86.49% to 87.33%, isolating the benefit of the target design. The real-input prediction diagnostics in the appendix are also a welcome addition beyond standard weight-space metrics.

Critical weaknesses
  1. The method is sensitive to the truncation rank and solver settings, and the experiments show that solving the proposed optimization problem more accurately can actually hurt final task performance. In the ViT-B/32 solver-depth study (Appendix B.4, Table S2), as the number of CG iterations increases and the relative residual steadily drops, accuracy peaks at 87.72% at 25 steps, slips to 87.33% at the default 50 steps, and drops sharply to 82.01% when run to the residual-certified solution. A similar drop appears on BERT, where increasing the budget from 50 to 100 steps lowers average accuracy from 77.38% to 59.57% (Table S3). This indicates a noticeable gap between the proposed proxy objective and true behavior preservation. While treating early-stopped CG as an iterative regularizer is reasonable, it makes the stopping budget and truncation rank (e.g., 14-task accuracy drops from 79.59% at rank 24 to 76.98% at rank 48 in Table S7) critical hyperparameters. In a strict data-free merging scenario, it is unclear how to choose the rank and stopping point for a new set of experts without peeking at downstream validation data, or whether the gains of SBR-R over SBR hold robustly across different solver depths.

  2. Agreement between two experts does not reliably indicate whether a response is useful for their respective tasks, and the agreement rule can unnecessarily suppress compatible, task-relevant responses. Specifically, SBR-R compares A_ji = U_j^T W_i V_i with B_ji = U_j^T W_j V_i and scales down the source response whenever positive agreement is low. As a result, if a peer expert simply has near-zero response on a shared subspace direction (absence of response rather than genuine conflict), SBR-R still suppresses the source expert's response even when it is useful for the source task and harmless to the peer. Appendix A.2 (Eq. 40) explicitly illustrates this with a simple two-expert example (W_1 = [1, 0], W_2 = [0, 1]): although the merged weight [1, 1] perfectly preserves both experts on their respective inputs, both cross-targets evaluate to zero, causing the objective to shrink the weights to [1/(1 + sqrt(2)), 1/(1 + sqrt(2))] ≈ [0.414, 0.414], which the generalist step cannot undo. While Theorem 1 and Proposition A.4 remain mathematically valid under their stated definitions, they do not justify equating lack of peer agreement with harmful interference. The paper lacks a quantitative analysis of how often this over-suppression occurs in practice and how a rule that distinguishes absent peer response from true directional conflict would perform.

Questions for authors
  1. In a strictly data-free setting, how should one select the truncation rank and CG stopping budget for an unseen collection of expert models prior to task evaluation? Furthermore, does the improvement of SBR-R over SBR hold consistently across different rank and iteration settings (including near-converged solves with explicit regularization)?

  2. Building on the counterexample in Appendix A.2, can you quantify how often SBR-R suppresses non-conflicting source responses simply because the peer expert is inactive along those directions? Under matched rank and solver settings, how does SBR-R compare against a target rule that only suppresses responses when the two experts actively conflict, while preserving source responses when the peer response is negligible?

  3. Could the over-suppression of compatible responses under exact minimization help explain why running CG to convergence hurts downstream accuracy in Tables S2 and S3? Evaluating expert-response fidelity and task accuracy as a function of solver depth on separated conflict vs. non-conflict directions would help clarify why early stopping is so critical.

6 comments
AuthorsOpenReview ↗

Q1 (1/2): Data-Free Rank Selection

Thank you for taking the time to read our paper carefully and share your feedback. We address each point below with clarifications and additional evidence where relevant, and we have revised the manuscript accordingly.

We noted your particular focus on CG stopping and the potential over-suppression of expert responses. In practice, our analysis suggests that CG can reduce the original loss by increasing the update magnitude, so early stopping acts as implicit regularization and our proposed full-expert-energy constraint below controls that magnitude directly, allowing CG to converge without early stopping; our measurements also show that over-suppression due to negligible peer responses is rare and has little effect on merging in practice. We provide the details below.

Q1: Data-Free Rank Selection and Sensitivity to CG Stopping

Thank you for raising this point. On rereading the manuscript, we realized that our description left it unclear how to choose the truncation rank and when to stop CG. Actually, both choices can be made adaptively without task data. Below, we explain how we select the rank from expert weights and use a residual criterion to stop CG.

1. The truncation rank

Weight updates often have long-tailed singular value spectra, with much of the update energy concentrated in the leading directions. This motivates retaining those directions and using stable rank, a standard measure of energy concentration, to estimate how many to keep. Appendix E.5 of the original submission describes how we use this estimate to choose the truncation rank without task data.

Specifically, let WiℓW_{i\ell} be expert ii's weight update at layer ℓ\ell relative to the shared pretrained model. We first compute its stable rank:

=\frac{\sum_k\sigma_{i\ell,k}^{2}}{\sigma_{i\ell,1}^{2}}.$$ Here, $\sigma_{i\ell,k}$ is the $k$-th largest singular value of $W_{i\ell}$, and $\sigma_{i\ell,1}$ is the largest. For each layer, we round up each expert's stable rank and use the largest value as the reference rank, so the reference accounts for the expert with the largest rank estimate: $$c_\ell=\max_i\left\lceil\operatorname{sr}(W_{i\ell})\right\rceil.$$ To avoid assigning an unusually small or large rank to a layer based on its own estimate, we pool the reference ranks across all $L$ matrices and choose one shared rank $r$. When all matrices can accommodate it, this amounts to rounding the average reference rank to the nearest integer. For a smaller matrix at layer $\ell$, we cap the number of directions at $d_\ell=\min(m_\ell,n_\ell)$ and choose the shared rank that best fits all layer estimates after applying these caps: $$r^*=\underset{r\in\{1,\ldots,\max_\ell d_\ell\}}{\arg\min} \sum_{\ell=1}^{L}\left(\min(r,d_\ell)-c_\ell\right)^2.$$ The table compares stable-rank selection with the original SBR-R settings and the strongest external baseline SWUDI. | Setting | Stable-rank selection | SBR-R with selected rank (%) | Original rank setting | SBR-R with original rank (%) | Strongest baseline SWUDI (%) | |---|---:|---:|---|---:|---:| | CLIP ViT-B/32, 8 tasks | Rank 32 | 87.33 | Rank 32 | 87.33 | 86.02 | | BERT, 3 tasks | Rank 11 | 78.30 | Rank 179 (rank ratio 0.35) | 77.38 | 76.6 | | CLIP ViT-B/32, 14 tasks | Rank 36 | 79.08 | Rank 24 | 79.59 | 77.92 | | InternVL2.5-1B, 10 tasks | Rank 57 | 59.78 | Rank 56 | 59.80 | 59.20 | We recommend this rule as the default for choosing a shared truncation rank without task data. It outperforms Strongest SOTA methods in various architecture, where it even improves on our original rank setting in BERT. We adjusted this stable-rank selection from the appendix to main text of the revised manuscript.
AuthorsOpenReview ↗

Q1 (2/2): CG Stopping and Matched Solver Comparisons

2. Why more CG steps hurt and how to solve

In the original submission, we did not treat the number of CG iterations as a tunable hyperparameter. Actually, we used fix CG steps throughout all main experiments, though tuning this can get much stronger results than that we reported in the paper. In response to the reviewer's comment, we further analyzed the drop in downstream accuracy observed at higher iteration counts.

To investigate this, we measured the Specialists update norms on eight-task ViT-B/32. For the original objective, we varied only the CG budget. Here the update magnitude below is the square root of the sum of squared Frobenius norms across the 50 visual matrices. The table below summarizes the supplementary results.

Solver settingSBR update magnitudeSBR accuracy (%)SBR-R update magnitudeSBR-R accuracy (%)
Original objective, 10 steps6.210686.49525.733187.1617
Original objective, 25 steps10.394287.09199.296887.7171
Original objective, 50 steps13.860286.487511.761087.3297
Original objective, 100 steps17.535084.195513.854886.5754
Original objective, residual-certified convergence28.090362.988717.987982.0149
Full-expert-energy constraint7.170986.67687.170987.5763

As CG runs for more steps, the Specialists updates in the merged model steadily grow in magnitude. Accuracy peaks at 25 steps and then declines, with the largest updates and lowest accuracies observed at convergence. We suspect that increasing the update magnitude offers a shortcut for reducing our proxy loss, which may explain why solving the original objective to convergence hurts downstream accuracy. To curb this growth, we introduce a full-expert-energy constraint on the Specialists update.

Specifically, we keep the original Specialists loss from Eq. (20). Suppressing layer indices for readability, for KK experts and $v\in\lbrace \mathrm{SBR},\mathrm{SBR\text{-}R}\rbrace $, the loss is

\mathcal L_s^v(W_s) &=\frac{1}{K}\sum_{i=1}^{K} \big||U_i^\top W_s V_i-U_i^\top W_i V_i\big||_F^2\\\\ &\quad+\frac{1}{\sqrt K}\sum_{i=1}^{K}\sum_{j\ne i} \big||U_j^\top W_s V_i-T_{ji}^v\big||_F^2 . \end{aligned}$$ Here, $U_i$ and $V_i$ are expert $i$'s retained output and input bases. The first term matches each expert's self-response, and the second matches the chosen cross-response target $T_{ji}^v$. At each layer, we minimize $\mathcal L_s^v(W_s)$ over the Specialists update $W_s$ in the joint input and output spaces, subject to $$||W_s||_F \le \sqrt{\sum_{i=1}^{K}||W_i||_F^2}.$$ The right-hand side is the combined norm of the experts' full weight updates from the shared base at that layer, before truncation. The solver stops once the residual and norm-budget checks pass. With this constraint, the mean accuracies of SBR and SBR-R in the eight-task ViT-B/32 experiment recover from **62.99% and 82.01%** at unconstrained convergence to **86.68% and 87.58%**, respectively, compared with **86.49% and 87.33%** at 50 CG steps. This removes the need to choose a fixed CG iteration count, as the norm budget comes from the expert weights, and numerical convergence determines when to stop without consulting task data. **3. SBR and SBR-R Across Ranks and Solver Settings** We compare SBR and SBR-R at matched ranks and solver settings on the eight-task ViT-B/32 benchmark. Each cell gives the mean accuracy (%) of SBR / SBR-R. | Rank | 25 CG steps | 50 CG steps | Unconstrained convergence | Convergence with the full-expert-energy constraint | |---:|---:|---:|---:|---:| | 24 | 87.31 / 87.67 | 86.56 / 87.17 | 72.88 / 84.73 | 87.39 / 87.81 | | 28 | 87.19 / 87.70 | 86.57 / 87.15 | 67.18 / 83.23 | 87.07 / 87.74 | | 32 | 87.09 / 87.72 | 86.49 / 87.33 | 62.99 / 82.01 | 86.68 / 87.58 | | 48 | 86.35 / 87.47 | 85.59 / 86.78 | 33.86 / 68.31 | 84.62 / 86.41 | SBR-R consistently achieves higher mean accuracy than SBR in the matched comparisons completed so far, demonstrating the value of selectively preserving cross-expert responses based on agreement.
AuthorsOpenReview ↗

Q2: Cross-Response Targets for Non-Conflicting Directions

Q2: Cross-Response Targets for Non-Conflicting Directions

Thank you for raising this point. Eq. (19) below is written in piecewise form, and Appendix A.2 discusses the limiting case where either Aji=Uj⊤WiViA_{ji}=U_j^\top W_iV_i or Bji=Uj⊤WjViB_{ji}=U_j^\top W_jV_i is zero.

\qquad \widehat\beta_{ji}= \begin{cases} 0, & A_{ji}=0\ \text{or}\ B_{ji}=0,\\\\[2pt] \dfrac{[\langle A_{ji},B_{ji}\rangle_F]_+} {||A_{ji}||_F\max\lbrace ||A_{ji}||_F,||B_{ji}||_F\rbrace }, & \text{otherwise}. \end{cases}$$ However, **in practice, such case is actually rare and will not impact our merging method**. We checked 2,800 cross-response block pairs across the 50 visual weight matrices in eight-task ViT-B/32. We also compared the 89,600 corresponding column pairs, each of which represents the two responses to one input direction. | Cross-response condition | Cases / total | Fraction | | --- | ---: | ---: | | Source response block: $A_{ji}=0$ | 0 / 2,800 | 0% | | Peer response block: $B_{ji}=0$ | 0 / 2,800 | 0% | | Weak peer response at the block level: $\|\|B_{ji}\|\|_F<0.25\|\|A_{ji}\|\|_F$ | 0 / 2,800 | 0% | | Weak peer column ($\delta=0.10$) | 6 / 89,600 | 0.0067% | | Weak peer column ($\delta=0.25$) | 549 / 89,600 | 0.6127% | Neither exact-zero case occurred, so the zero-response branch of Eq. (19) did not affect merging in this setting. Futhermore, for input direction $k$, we call the peer response weak if $$||B_{ji}[:,k]||_2 < \delta\,||A_{ji}[:,k]||_2, \qquad \delta\in\lbrace 0.10,0.25\rbrace ,$$ where $[:,k]$ denotes column $k$. The last two rows report how often this condition holds. Even at the 25% threshold, fewer than 1% of the column pairs met this condition. Thus, peer inactivity was rare at both the block and individual input direction levels and had little practical effect on merging in this setting. This brings us to the second question: *How does SBR-R compare with a rule that suppresses responses only when the experts actively conflict and preserves the source response when the peer response is negligible?* **1. Preserving the full source response block.** First, we ask whether the peer response is weak across the whole block. We would keep the full source response if the peer block's norm were less than 25% of the source block's norm. As shown in the table above, no block pair met this condition, so this exception leaves the original SBR-R targets and results unchanged. **2. Preserving the source response for individual input directions.** Even when the peer response is strong across the block, it can be weak in individual columns, as shown in the last two rows of the table. At each threshold, we restored the full source response as the target only for the flagged columns. Mean accuracy was **87.3242%** at the 10% threshold and **87.3516%** at the 25% threshold, compared with the similar result **87.3297%** for the original SBR-R. **3. Suppressing responses when the two experts actively conflict.** For each input direction, we set the target to zero only if the source and peer responses have a negative inner product and the peer response norm is at least 10% of the source response norm. Otherwise, we preserve the full source response. With all other settings unchanged, mean accuracy was **84.7623%**, compared with **87.3297%** for the original SBR-R. Overall, **negligible peer responses were rare in practice, and SBR-R already accounts for agreement and opposition between expert responses when setting cross-response targets**. We acknowledge that alternative rules may distinguish weak peer responses from genuine conflicts more precisely, and we leave their exploration to future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Contributions and Rebuttal Discussion

We sincerely thank the reviewers and the Area Chair for the time and care you have devoted to reading our paper and discussing our work. We greatly appreciate your thoughtful questions and constructive feedback. Reviewers acknowledged our unique contribution and highlighted:

  • cyQb: “a meaningful modeling step beyond simply changing the solver for the WUDI objective”.
  • SLzj: “a meaningful perspective beyond conventional weight-space aggregation”.
  • dXgW: “SBR-R consistently improves average accuracy over the strongest reported external baseline”.

Our contribution is a novel weight-only reconstruction objective for preserving expert behavior, with explicit self/cross targets and complementary-space calibration. Our rebuttal provides:

  1. Data-free applicability. Our original paper provides an adaptive, weight-only truncation-rank selection strategy. Thus, when applying SBR-R to new, unseen merging settings, our method can be used directly without searching for any hyperparameters, consistent with the data-free setting. Our experiments also show that, under strict data-free merging, SBR-R consistently achieves the best merging performance across traditional models (ViT), LLMs (Qwen), and VLMs (InternVL).

  2. The theoretical connection. Our response reconstruction objective has theoretical guarantees. We show that the response reconstruction error optimized in weight space provides an upper bound on the real response difference between the merged and expert models. This explains why our method is effective, and our rebuttal further clarifies the previously unclear steps in this derivation.

  3. Component Ablations. The cross-response term is one part of our response reconstruction objective, and removing it reduces ViT-B/32 accuracy by 24.59%, showing that the gains from the cross-response mechanism are substantial. Besides, following WUDI (ICML ’25), we retain the default fixed CG budget. Our additional analysis in the rebuttal shows that this implicit regularization is related to growth in update magnitude. Reviewer cyQb also agree the benefit of the norm constraint: “The norm constraint also mitigates the observed convergence-related degradation”.

The reviewers have acknowledged that our additional explanations and experiments clarified key concerns, and Reviewer SLzj noted that “the revision has improved the paper”. For the remaining questions that our earlier responses did not explain clearly enough, we have provided detailed clarifications and additional experimental analyses in our follow-up responses.

Thank you all once again for your valuable time, thoughtful feedback, and careful consideration of our work. We sincerely appreciate your efforts and welcome any further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.