- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates a specific issue in multimodal expert model merging: whether a low-dimensional space capable of accurately representing all expert updates can also preserve the optimal shared response reconstruction solution. The authors demonstrate that these two requirements are not equivalent, as task input correlations can cause the shared optimal solution to deviate from the joint input space of the expert updates.
Based on this observation, the proposed method, FerVA, retains full input coordinates on the language side while compressing the reconstruction problem via an output basis shared among experts; on the vision side, it learns mixture weights for a frozen vision projector, with the calibrator trained via an auxiliary decoder before being transferred to the final model. The paper provides a theoretical characterization of lossless output compression, reconstruction loss resulting from input constraints, and the minimal sufficient input space. Experiments include a controlled analysis of recovery directions and evaluations across two backbone models and two continual learning benchmarks. The full method achieves the highest final average scores across four settings; controlled experiments further demonstrate that selecting recovery directions based on the reconstruction objective is more effective than random recovery at the same rank.
Critical strengths
The theoretical issues are clearly defined, and the conclusions offer practical guidance. The paper distinguishes between the representability of expert updates and the representability of optimal shared-response solutions, while quantifying the exact objective gap and the minimal sufficient space resulting from input constraints. These findings inform the selection of low-dimensional merged spaces, preventing the erroneous assumption that a merge process is lossless simply because "all experts can be represented exactly."
Controlled recovery experiments effectively bridge theory and task performance. With fixed experts, input statistics, visual interfaces, and deployment gains, full input recovery improved FAA from 69.54 to 76.04; at a recovery rank of 16, target-guided recovery achieved 75.54, whereas three random bases yielded only 71.18–71.33. This demonstrates that the choice of recovery direction plays a substantive role, rather than merely increasing parameter capacity. Results across two backbones and two benchmarks further substantiate the method's practical utility.
Critical weaknesses
The incremental gains attributable specifically to the coordinate-based approach—distinct from existing full-input reconstruction methods—have not been fully isolated. Appendix C.1 clarifies that, given consistent statistics, task weights, and regularization, the language-side solution recovers the dense optimal reconstruction corresponding to RegMean; furthermore, MAny already employs a hybrid of weighted full-input activation reconstruction and prototype-driven visual projection. Consequently, while the experiment demonstrating a recovery of 6.50 points validates the paper's "coordinate insight," it does not directly explain the source of the performance improvement over MAny. Further differentiation is required to isolate the contribution of the coordinate-based implementation from other configuration changes—specifically by standardizing statistics, regularization, numerical precision, deployment gains, and visual interfaces. This distinction is crucial for establishing the substantive novelty of the proposed method.
The independent contribution of the final visual calibrator and the design of the auxiliary decoder lack direct empirical validation. Table 5 replaces the full visual module with a uniform mixture and removes both prototype routing and learned calibration; thus, it cannot isolate the benefits of the added calibrator. Although Table 6 provides a comparison using a fixed language model, it relies on an early configuration, meaning the ~0.32-point gain does not directly reflect the performance of the final FerVA. Nor does the paper directly compare "training the calibrator on the final decoder" versus "training via an auxiliary decoder followed by transfer." The conditional transfer bounds presented in the appendix cannot substitute for these experiments; consequently, it remains difficult to determine whether the additional visual training and its complex design are truly necessary.
Questions for authors
Alignment Comparison with Existing Methods: When using the same experts, source samples, statistic normalization, regularization, numerical precision, deployment gains, and visual interfaces, what are the FAA scores for FerVA and MAny, respectively? Which specific modifications contribute primarily to the performance gains shown in Table 1?
Independent Gains from Final Visual Calibration: If the language weights of the final FerVA model are fixed and only the prototype routing and the final learned calibrator are compared, what are their respective FAA scores and per-task performance?
Necessity of the Auxiliary Decoder: Under identical data and training budgets, how do the performance and cost of training the visual calibrator directly via the final decoder compare to training it via an auxiliary decoder followed by transfer? Is there direct evidence supporting the latter approach?
8 comments
W1. What do the faithful space and its coordinate implementation each contribute?
Thank you for pointing out that the theoretical contribution of the faithful space should be distinguished from the effects of the numerical solution procedure. FerVA's central question is: which coordinates can be compressed without changing the reconstruction optimum, and which input directions must be retained? We adopt RegMean's activation-weighted reconstruction objective and characterize the minimal input space that is faithful for this family of expert updates, given the source statistics and the experts' admissible input directions. We also quantify the additional reconstruction error induced by restricting the input space. Source statistics can change the orientation of this minimal space, potentially taking it beyond the joint span of the experts' original input directions. FerVA therefore retains the full input space and compresses only the output side. Under the fixed-statistics and regularization conditions stated in the paper, this parameterization preserves the corresponding dense reconstruction optimum when the normal matrix is positive definite and the shared output basis is orthonormal and spans the output spaces of all expert updates.
Thank you also for asking about the task relevance of the faithful-space characterization. With the deployment coefficient fixed at 1 and prototype routing alone used in both conditions, full input outperforms restricted input on all six tasks and improves FAA by approximately 7.02 points. This comparison directly tests whether directions excluded by the input restriction have downstream value. The largest improvements are on CLEVR-Math and VizWiz-caption, at 16.00 and 12.02 points, respectively, showing substantial task benefits from retaining these directions in this experiment. The table reports the task-level results, with FAA and differences calculated from the listed scores before rounding.
| Task | Full input | Restricted input | Full input minus restricted input (points) |
|---|---|---|---|
| ImageNet-R | 89.27 | 81.77 | +7.50 |
| ArxivQA | 91.47 | 91.37 | +0.10 |
| VizWiz-caption | 59.80 | 47.78 | +12.02 |
| IconQA | 73.93 | 68.53 | +5.40 |
| CLEVR-Math | 77.73 | 61.73 | +16.00 |
| Flickr30k | 57.81 | 56.74 | +1.07 |
| FAA | 75.00 | 67.99 | Approx. +7.02 |
Thank you for further asking us to distinguish the space characterization from existing full-input methods. MAny already uses the full input space, so the comparison above addresses the cost of restricting that space. The output-side contribution is to preserve the corresponding dense reconstruction result in compact coordinates. With the same direct solver and prototype routing, shared coordinates reduce the right-hand side of the equations from thousands of columns to 96, while matching the dense reference's scores on all six tasks at the reported precision. The input-side loss characterization and output-side fidelity result jointly inform FerVA's design: retaining input directions with task value while compressing output coordinates that admit a lossless representation.
W2. Contribution of final visual calibration and value of the auxiliary decoder
Thank you for pointing out that learned calibration should be examined separately under the final language configuration. The new matched ablation shows that adding calibration to prototype routing increases FAA from 75.61 to 76.20, an improvement of approximately 0.60 points. Both conditions share the same reconstructed language weights, visual experts, prototype routing, and deployment coefficient, and use identical test samples and scoring protocols. Only whether calibration is enabled changes. All six tasks improve. Unlike removing the entire visual module, this comparison changes only the calibrator, directly measuring its independent contribution beyond the existing routing mechanism. The revised manuscript will replace the earlier-configuration calibration comparison with the matched ablation below. FAA and differences are calculated from the listed scores before rounding.
| Task | Prototype routing only | Prototype routing with calibration | Difference after adding calibration (points) |
|---|---|---|---|
| ImageNet-R | 88.30 | 88.90 | +0.60 |
| ArxivQA | 89.03 | 89.37 | +0.34 |
| VizWiz-caption | 66.07 | 66.96 | +0.89 |
| IconQA | 74.90 | 75.73 | +0.83 |
| CLEVR-Math | 78.40 | 78.97 | +0.57 |
| Flickr30k | 56.95 | 57.29 | +0.34 |
| FAA | 75.61 | 76.20 | Approx. +0.60 |
Beyond the calibrator's contribution, thank you for suggesting that we directly test the auxiliary decoder's value during training. Auxiliary supervision performs better in all five paired training orders, with an average FAA improvement of approximately 0.49 points. Using five training-sample order seeds, we compare calibrators trained through the auxiliary decoder and then transferred with calibrators trained directly through the final decoder. Both are evaluated using the final decoder. In all five paired performance comparisons, the two supervision strategies use identical calibration data and training budgets. Improvements range from 0.30 to 0.59 points, showing that the advantage of auxiliary supervision recurs across the examined training orders and providing empirical support for this training strategy. Differences and summary statistics in the table are calculated from the listed FAA values.
| Training-sample order seed | Auxiliary-supervision FAA | Direct-supervision FAA | Auxiliary minus direct supervision (points) |
|---|---|---|---|
| 20260804 | 76.20 | 75.61 | +0.59 |
| 20260805 | 76.18 | 75.88 | +0.30 |
| 20260806 | 76.20 | 75.65 | +0.55 |
| 20260807 | 76.16 | 75.74 | +0.42 |
| 20260808 | 76.18 | 75.60 | +0.58 |
| Mean ± sample standard deviation | 76.18 ± 0.02 | 75.70 ± 0.12 | +0.49 ± 0.13 |
Thank You for Your Time and Constructive Feedback
Thank you for recognizing the guidance provided by the faithful-space characterization and the support from the equal-rank recovery experiment for choosing recovery directions. To save you time, we have highlighted the key conclusions and important points in bold. The distinction you raise clarifies the paper's contributions: the theory characterizes which input directions should be retained and which output coordinates can be compressed losslessly, while the additional comparisons separately examine compact representation and implemented construction procedures. With the same direct solver, shared output coordinates reduce the right-hand side of the equations to 96 columns while matching the dense reference's scores on all six tasks at the reported precision. With source statistics, ridge regularization, the deployment coefficient, and the visual interface matched, FerVA retains a 3.41-point FAA advantage over the MAny implementation evaluated here. The former result supports fidelity under a compact representation, while the latter reports the empirical difference between implemented language-construction procedures. Below, we also use separate comparisons to assess learned calibration and auxiliary supervision and describe offline costs and deployment. All experiments are conducted on UCIT–LLaVA-1.5-7B. FAA denotes the final average score across the six tasks.