- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Error-Subspace Calibration (ESCal), a post-merging calibration method that constructs low-dimensional subspaces from expert–merged weight differences. It uses fixed read/write bases and learns only small mapping matrices to align merged representations with task-specific experts. Experiments across vision and language benchmarks demonstrate improvements over existing calibration methods.
Critical strengths
-
ESCal uses merging-error subspaces to guide feature correction, reducing trainable parameters while achieving competitive performance.
-
The method is evaluated across multiple architectures and merging algorithms. Ablation studies further support the effectiveness of the proposed subspace construction.
Critical weaknesses
-
The analysis focuses on local weight-induced errors but does not fully explain why the selected subspaces capture accumulated feature discrepancies across nonlinear layers.
-
How is the terminal BRIDGE read basis constructed when the output head has fewer than (r) nonzero singular values?
Questions for authors
-
Could the authors evaluate ESCal on top of SWUDI from Closed-Form Spectral Regularization for Multi-Task Model Merging, comparing uncalibrated models with SurgeryV2 and ESCal under identical calibration budgets?
-
How much of the actual feature discrepancy is captured by the selected error subspaces?
2 comments
I appreciate the authors' thorough response and the additional experimental results. The clarification regarding the terminal read basis satisfactorily resolves my concern, while the new feature-error analysis and comparisons provide further support for the proposed approach. The authors have adequately addressed my main questions and strengthened the empirical evidence.
Response to Reviewer J2j5
We thank the reviewer for their positive assessment and suggestions for further validation.
W1/Q2. Error-subspace capture of accumulated feature discrepancies.
As discussed in our response to Reviewer jCz2 (W1/Q2), the newly added Figure 3 in Appendix D.1 of the revised manuscript compares captured feature-error energy, actual error reduction, and residual feature MSE for weight-derived, feature-derived, and random bases. Weight-derived bases leave less residual feature error in later blocks than feature-derived bases despite capturing less of the feature-error energy before calibration.
W2. Terminal read-basis construction and available rank.
The terminal read basis is constructed from a feature projection, rather than the final class-logit matrix, so its available rank is not determined by the number of classes. In vision, we use the expert–merged difference of
visual.proj, a 768 × 512 matrix whose numerical rank was 512 for all eight tasks examined. In NLU, we use the expert’s 768 × 768classifier.dense.weight.T, rather thanclassifier.out_proj. Its numerical rank in FP32 was 767–768, exceeding the evaluated ranks of 32, 64, and 128.Our implementation does not explicitly discard near-zero singular values and could therefore include numerically null directions if the requested rank exceeded the available numerical rank. However, this issue did not arise in the settings examined. We have clarified the terminal projection used in each architecture in Section 3.2 of the revised manuscript.
Q1. Calibration on top of SWUDI.
Thank you for the suggestion. We have additionally evaluated ESCal on top of SWUDI, comparing it with existing post-merging calibration methods under identical calibration budgets. The results are summarized below.
ESCal achieves the highest average accuracy of 89.3369%, improving the uncalibrated SWUDI baseline by 4.05 percentage points. These results further demonstrate that ESCal can effectively complement SWUDI and improve merged-model performance across different base merging methods.