- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Subspace Behavior Reconstruction (SBR), a method for merging task-specific models without using task data. The authors argue that preserving expert responses is more useful than directly matching weight updates. They use the leading singular vectors of each task update to define input and output subspaces. SBR reconstructs each expert's response within its own subspaces and suppresses responses across different experts. The extended method, SBR-R, retains cross-task responses when the corresponding experts show positive agreement. A separate component handles input directions outside the selected subspaces. The authors evaluate their methods on CLIP, Qwen-14B LoRA, InternVL2.5, and BERT.
Critical strengths
The paper is well motivated, and the proposed method follows naturally from the analysis of expert responses. The distinction between self-responses and cross-responses is useful for understanding what the merging objective controls. Both methods construct their targets from model weights, so task data are not required during merging. The experiments cover several model architectures and include comparisons with recent methods. SBR-R consistently improves average accuracy over the strongest reported external baseline on all three CLIP backbones. The component ablations help explain the contributions of the different loss terms. The authors also examine internal responses and final predictions using real inputs after merging, which provides useful evidence beyond benchmark accuracy.
Critical weaknesses
The theoretical analysis mainly concerns layer-wise projected responses. The resulting bound can be loose and includes a target-mismatch term, so it offers limited guidance on whether the chosen cross-response targets improve downstream predictions. The positive-agreement rule is reasonable, but agreement between weight-derived responses does not necessarily indicate useful shared task behavior. The experiments show encouraging improvements, although the margins over strong baselines are relatively small on some architectures. The reported results also depend on the numerical solver. In particular, the dense-model accuracy can decrease substantially as the solver approaches convergence, suggesting that early stopping contributes significantly to performance. Finally, the use of fixed expert checkpoints limits the assessment of statistical variability, and some comparisons involve different output adapter capacities.
Questions for authors
I have some questions regarding the experimental setup and symbols that I would like to discuss with the author(s).
-
In Eq. (21), since and are singular vectors of and , we have . Thus, the first term appears to penalize responses along the retained output directions rather than reconstruct nonzero expert tail responses. Is this interpretation correct? Could the authors clarify the role of this term in Generalist calibration?
-
The solver-depth analysis shows that better convergence does not necessarily lead to better downstream accuracy. How much of the reported improvement comes from the proposed response targets, and how much comes from the regularization introduced by finite-step optimization? It would help to clarify how sensitive the relative advantage of SBR-R is to the stopping budget.
-
In the Qwen experiments, several baselines use a smaller output adapter rank than SBR-R. Could the authors clarify how much this capacity difference affects the comparison? A capacity-matched analysis would help isolate the benefit of the proposed reconstruction objective.
-
The InternVL experiments use a global merge scale selected through preliminary comparisons on the evaluation benchmarks. Could the authors clarify the scope of the data-free claim regarding hyperparameter selection? It would also be useful to know whether the relative performance trends remain similar under an independently chosen scale.
6 comments
Q2: Contributions of Response Targets and Early Stopping
Q2: Contributions of Response Targets and Early Stopping
Thank you for raising this question. Below, we further analyze why better convergence does not necessarily lead to better downstream accuracy. Our results indicate that the gains come primarily from the proposed response targets, while finite-step optimization helps avoid shortcuts during optimization. We also propose an explicit regularization method to achieve the same goal.
To investigate the influence of solver-depth, we measured the Specialists update norms on eight-task ViT-B/32. For the original objective, we varied only the CG budget. Here the update magnitude below is the square root of the sum of squared Frobenius norms across the 50 visual matrices. The table below summarizes the supplementary results.
| Solver setting | SBR update magnitude | SBR accuracy (%) | SBR-R update magnitude | SBR-R accuracy (%) |
|---|---|---|---|---|
| Original objective, 10 steps | 6.2106 | 86.4952 | 5.7331 | 87.1617 |
| Original objective, 25 steps | 10.3942 | 87.0919 | 9.2968 | 87.7171 |
| Original objective, 50 steps | 13.8602 | 86.4875 | 11.7610 | 87.3297 |
| Original objective, 100 steps | 17.5350 | 84.1955 | 13.8548 | 86.5754 |
| Original objective, residual-certified convergence | 28.0903 | 62.9887 | 17.9879 | 82.0149 |
| Full-expert-energy constraint | 7.1709 | 86.6768 | 7.1709 | 87.5763 |
As CG runs for more steps, the Specialists updates in the merged model steadily grow in magnitude. Accuracy peaks at 25 steps and then declines, with the largest updates and lowest accuracies observed at convergence. We suspect that increasing the update magnitude offers a shortcut for reducing the proxy loss, which may explain why solving the original objective to convergence hurts downstream accuracy. To curb this growth, we introduce a full-expert-energy constraint on the Specialists update.
Specifically, we keep the original Specialists loss from Eq. (20). Suppressing layer indices for readability, for experts and $v\in\lbrace \mathrm{SBR},\mathrm{SBR\text{-}R}\rbrace $, the loss is
\mathcal L_s^v(W_s) &=\frac{1}{K}\sum_{i=1}^{K} \big||U_i^\top W_s V_i-U_i^\top W_i V_i\big||_F^2\\\\ &\quad+\frac{1}{\sqrt K}\sum_{i=1}^{K}\sum_{j\ne i} \big||U_j^\top W_s V_i-T_{ji}^v\big||_F^2 . \end{aligned}$$ Here, $U_i$ and $V_i$ are expert $i$'s retained output and input bases. The first term matches each expert's self-response, and the second matches the chosen cross-response target $T_{ji}^v$. At each layer, we minimize Eq. (20) over the Specialists update $W_s$ in the joint input and output spaces, subject to $$||W_s||_F \le \sqrt{\sum_{i=1}^{K}||W_i||_F^2}.$$ The right-hand side is the combined norm of the experts' full weight updates from the shared base at that layer, before truncation. The solver stops once the residual and norm-budget checks pass. With this constraint, the mean accuracies of SBR and SBR-R in the eight-task ViT-B/32 experiment recover from 62.99% and 82.01% at unconstrained convergence to 86.68% and 87.58%, respectively, compared with **86.49% and 87.33% at 50 CG steps. This removes the need to choose a fixed CG iteration count, as the norm budget comes from the expert weights, and numerical convergence determines when to stop without consulting task data. In General, the matched comparisons indicate that the **response target is the primary driver of our improvement, while finite-step optimization acts as implicit regularization by limiting the solver’s tendency to reduce the reconstruction loss through excessively large weight updates**.Q3: Output Capacity in Qwen LoRA Comparisons
Q3: Output Capacity in Qwen LoRA Comparisons
Thank you for raising this point. We use a rank-128 output adapter because it reflects the core idea of SBR-R: a larger optimization space allows us to better approximate each expert’s core subspace. Several baselines in the table, including TSV-M and Iso-CTS, also output rank-128 adapters.
However, SBR-R can also produce a rank-32 adapter in two ways: by truncating the rank-128 adapter to rank 32 after merging, or by optimizing directly under a rank-32 constraint. The table below shows the results:
| Method | Output rank | MMLU | TruthfulQA | BBQ | CNN/DM | Four-task mean |
|---|---|---|---|---|---|---|
| Original SBR-R | 128 | 67.10 | 63.66 | 89.72 | 20.22 | 60.18 |
| SBR-R truncated to rank 32 | 32 | 67.15 | 64.16 | 88.72 | 20.69 | 60.18 |
| SBR-R directly solved at rank 32 | 32 | 66.30 | 64.16 | 89.22 | 20.09 | 59.94 |
| TSV-M | 128 | 65.94 | 61.40 | 90.23 | 18.67 | 59.06 |
| Iso-CTS | 128 | 67.40 | 64.66 | 86.47 | 20.11 | 59.66 |
| SWUDI | 32 | 66.61 | 63.66 | 89.47 | 19.44 | 59.80 |
At the same rank of 32, our SBR-R still exceeds other SOTA methods, which can full demostrate our robust improvement and show this capacity difference does not affect the comparison results, ours still perform consist advantage over other SOTA methods. Whereas, truncating the rank-128 adapter to rank 32 changes the mean score by less than 0.01 points. This small change is consistent with how much of the update the truncation retains: across 200 layers, the first 32 singular directions preserve a median of 97.47% of the squared Frobenius norm of the update.
To further assess how much capability affects our SBR-R, we also ran two experiments that constrained the rank during merging to 64 and 80. The resulting mean scores were 60.13 and 60.19, respectively, both close to the rank-128 result (60.18). This also suggests that the full rank-128 capacity is not required to reach a similar score in this setting.
Q1: The Role of the Projected-Response Term in Generalist Calibration
Thank you for your thoughtful feedback and for the effort you devoted to the review. Below, we respond to each of your questions, describe the additional analyses we carried out, and explain how we revised the manuscript in response.
In general, generalist calibration used to align responses to inputs outside the joint expert subspace, and Uk⊤WkPZ⊥ is written in this form to parallel the Specialists objective; SBR-R outperforms SBR at matched CG budgets and SWUDI at the same Qwen LoRA output rank; and for InternVL, we use the same scaling coefficient s for all methods without tuning it separately for ours, ensuring a fair comparison. We provide the details below.
Q1: The Role of the Projected-Response Term in Generalist Calibration
Thank you for pointing this out. Your interpretation is correct. Under the paper's construction, Uk⊤WkPZ⊥=0, so the first term in Eq. (21) is indeed a zero-target constraint.
We wrote Eq. (21) in this form to make it easier to compare with the corresponding Specialists objective. We have clarified this point in the revised manuscript. Specifically, Generalist calibration aligns responses to inputs projected onto less important subspace Z⊥ by PZ⊥, while Specialists calibration aligns responses to inputs projected onto more important subspace Z by PZ.
For inputs in Z⊥, the two terms of the Generalist objective address different aspects of the output. The first term, which you asked about, concerns the output directions Uk of each expert. Since Uk⊤WkPZ⊥=0, this term suppresses the merged response along Uk to prevent inputs in Z⊥ from interfering with these directions.
The second term helps preserve responses outside the retained output directions Uk⊤. Since these directions carry less information about individual experts' task-specific behavior, we align the full merged response with the sum of the expert responses to retain behavior that can be useful across tasks.
We hope this explanation clarifies the role of the two terms and our Generalist calibration. We have revised the manuscript to make this distinction clearer and would be happy to discuss any remaining questions.