- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AttnCal, a merge of task-specialized multimodal models into one static checkpoint. Its central proposal rebuilds the rotary query-key operator rather than averaging weights. Each frequency pair is an outer product of rank at most one, so the average of expert operators can exceed the head's rank, while the product of averaged factors changes the match. AttnCal constructs, per head, a new rank-at-most-one operator that minimizes the worst expert's relative error summed over frequencies, using weights alone, and restores factors at the original width. It then calibrates the query and key on task answers under one visual map and fits a shared visual residual with the language weights frozen.
The paper evaluates UCIT and DCL on LLaVA-1.5-7B and InternVL-Chat-7B against merged baselines. AttnCal obtains the highest final average score on all four protocols. On six UCIT tasks with LLaVA-1.5-7B, it reaches 76.07 FAA, 82.72 CAA, and 0.33 BWT, compared with 72.15, 81.60, and for the strongest FAA baseline. Separate task experts remain stronger than the single merged model.
Critical strengths
- This paper addresses a practically important inconsistency in multimodal model merging: averaging query and key weights need not preserve the rotary interaction that the original head can store.
- The experimental evaluation is relatively comprehensive, covering four reproduced benchmark-backbone protocols with 3 different metrics.
Critical weaknesses
-
The final-average lead is not tied to the head-coupled query–key objective. Figure 3 compares three native-width constructions before task calibration and says neither panel is task accuracy. Tables 1–4 then score only the full pipeline: on UCIT with LLaVA-1.5-7B, 76.07 FAA against 72.15 for the matched MAny run. Table 5, with the language model held fixed, gives only +0.40 FAA to the final visual residual. Equation 2 still adds a rank-8 query–key correction trained with answer cross-entropy, and Appendix C builds the other 160 non-query–key projections by a separate reconstruction. Appendix F bounds the initialization, not the calibrated checkpoint. No table runs the other two Figure 3 targets through that same calibration.
-
That comparison also trains AttnCal and leaves the strongest baseline untrained. Appendix C fits the auxiliary visual map, the rank-8 query–key correction, and the visual continuation for 372 updates each, on 62 source examples per task, with answer-token cross-entropy. MAny, the strongest FAA baseline in Tables 1–4, is training-free. A gain of a fitted pipeline over an unfitted merge does not show that the operator construction is what won.
Questions for authors
- Could the authors provide anonymous code repository for review?
- If the normalized mean rank-one fit and the frequency-wise minimax from Figure 3 are passed through Equation 2 with the same non-Q/K weights, the same 62-example schedule, and the same visual basis, what are their FAA, CAA, and BWT on UCIT-LLaVA?
- For the expert-selection mechanism in Table 5 and Appendix A.2, does inference use the task label, a task-free score, or no choice among expert projectors? Is that procedure active for the Table 1 AttnCal row?
- After the rank-8 update in Equation 2, what is the worst-expert relative operator error , compared with the initialization plotted in Figure 3? and if answer calibration stay near the minimax solution.
11 comments
W1. How can the construction's contribution and theoretical guarantees be isolated?
Changing only the query and key initialization under matched information access, other components, and calibration improves FAA, CAA, BWT, and operator error, supporting the independent contribution of head-coupled minimax. Head-coupled minimax improves FAA by 0.4356 and 0.5034 percentage points over normalized mean rank-one and frequency minimax, respectively. Relative to normalized mean rank-one, CAA and BWT also improve by 0.3324 and 0.1908 points. Its post-calibration operator error is 28.40% lower than normalized mean rank-one and 5.32% lower than frequency minimax. On UCIT–LLaVA, we fix the other 160 projections, the 62 source examples per task and their order, the rank-8 visual basis, optimizer, training seed, and calibration schedule. All three constructions receive rank-8 query–key calibration at a learning rate of 10⁻⁴, with 62 updates per seen task in each fitting phase and 372 updates per phase at the final stage. We use a common configuration without separately tuning the initializations. This control directly connects operator fidelity with downstream performance after identical calibration. Task metrics are calculated from all 21 stage–task combinations. Each stage restricts experts, statistics, visual bases, and fitting samples to seen tasks, separately from the fixed six-expert visual pool in the original Table 1. FAA averages the final-stage task scores, CAA averages the stage-wise task averages, and BWT measures the average change on earlier tasks from their first evaluation to the final stage. The final-stage operator metric takes each head's largest squared relative error over the six experts and averages over 1,024 heads. We will label these evaluation protocols explicitly in the revision.
| Construction | FAA ↑ | CAA ↑ | BWT ↑ | Mean worst-expert operator error after calibration ↓ |
|---|---|---|---|---|
| Normalized mean rank-one | 76.2167 | 83.7193 | −0.6440 | 0.003467003 |
| Frequency minimax | 76.1489 | 83.5634 | −0.6553 | 0.002622141 |
| Head-coupled minimax | 76.6523 | 84.0517 | −0.4532 | 0.002482520 |
To test repeatability, we run three paired seeds under another configuration selected before confirmation. All constructions share the remaining weights, sample order, and visual basis within each seed. The query–key learning rate is 2.5 × 10⁻⁵, the visual learning rate is 10⁻⁴, auxiliary visual fitting and query–key calibration each use 372 updates, and final visual fitting uses 186 updates. Head-coupled minimax exceeds both alternatives on every seed, improving mean FAA by 0.4225 and 0.4135 percentage points over normalized mean rank-one and frequency minimax. Its sample standard deviation is 0.0312. The paired differences from frequency minimax are 0.4239, 0.3773, and 0.4392 points, consistent in direction and similar in magnitude. The leading ordering therefore holds in both tested calibration configurations. The common configuration is selected in two steps. We first use 120 source tuning examples to choose the query–key learning rate from 2.5 × 10⁻⁵, 5 × 10⁻⁵, and 10⁻⁴, averaging answer cross-entropy equally across tasks and across the three constructions and MAny. A separate 600-example source set then selects the budget from 0, 46, 93, and 186 updates, averaging over these four methods and both Retain and Reset. Both choices are frozen before confirmation, and every construction and seed uses the selected configuration. The table reports final-stage FAA over the six tasks, with SD denoting the sample standard deviation across training seeds.
| Training seed | Head-coupled minimax FAA ↑ | Normalized mean rank-one FAA ↑ | Frequency minimax FAA ↑ |
|---|---|---|---|
| 2028 | 76.7822 | 76.3506 | 76.3583 |
| 2029 | 76.7584 | 76.3517 | 76.3811 |
| 2030 | 76.8203 | 76.3911 | 76.3811 |
| Mean ± SD | 76.7870 ± 0.0312 | 76.3645 ± 0.0231 | 76.3735 ± 0.0132 |
The initialization certificate and the calibrated model are evaluated separately. Appendix F applies to the constructed initialization, whereas answer calibration changes the query and key weights. We therefore recompute the same minimax objective for the calibrated weights, keeping the experts, normalization, and target fixed. The measured worst-expert error supplies a feasible upper bound, and a numerical lower bound is recomputed for the same objective. For the default calibrated state, the upper–lower bound gap averages 3.56 × 10⁻⁵ and reaches a maximum of 2.85 × 10⁻⁴ across heads. This provides a numerically estimated upper bound on suboptimality after calibration. Q4 reports the full error and bound comparison. The revision will distinguish this numerical assessment from the analytical initialization certificate and retain the fixed-input scope of the attention-error bound.
W2. Does AttnCal still outperform MAny when calibration opportunities are matched?
With visual experts, routing, and calibration budgets matched, AttnCal exceeds MAny on all three paired seeds by a mean of 2.1091 FAA percentage points. We reuse the MAny merged weights from Table 1 with its published parameters and give both methods the same calibration schedule. They use the same six frozen visual projectors and expert prototypes, the same CLIP image features and cosine-similarity routing, and softmax temperature 0.1. They also share the LLaVA-1.5-7B backbone, the 62 source examples per task and their order, rank-8 corrections, and visual basis. Each method independently fits its corrections using AdamW at 10⁻⁴, with 372 updates each for auxiliary visual fitting, query–key calibration, and final visual fitting. Evaluation uses the same 3,000 examples for each of the six tasks. Calibration improves MAny from 72.1594 FAA to a mean of 74.0805, confirming that additional training helps, but does not remove the difference from AttnCal. This comparison evaluates each complete language-weight construction under matched visual conditions and training opportunity. W1 isolates the query–key contribution by fixing the other 160 projections. The original Table 1 reports 76.07 and 72.15 for the original complete-method comparison, while the table below reports the additional matched-calibration runs.
| Training seed | MAny FAA with matched calibration | AttnCal FAA | AttnCal − MAny (pp) |
|---|---|---|---|
| 2028 | 74.1100 | 76.6523 | +2.5423 |
| 2029 | 74.0522 | 75.9578 | +1.9056 |
| 2030 | 74.0794 | 75.9589 | +1.8795 |
| Mean ± SD | 74.0805 ± 0.0289 | 76.1897 ± 0.4007 | +2.1091 ± 0.3754 |
Thank You for Your Constructive Feedback
Thank you for recognizing the practical importance of preserving native rotary interactions and the evaluation across four benchmark–backbone settings. To save you time, we have highlighted the key conclusions and information in bold. The new controls directly support the independent task contribution of the head-coupled construction and the complete method's advantage under equal calibration, addressing the two central attribution concerns. With the other 160 projections and calibration conditions fixed, head-coupled minimax improves all three task metrics and operator error. Three paired seeds under another common configuration also favor it over both alternatives. When visual experts, routing, and calibration budgets are matched with MAny, AttnCal retains a mean FAA advantage of 2.1091 percentage points. We also clarify expert availability at each stage and distinguish the initialization certificate from numerical evaluation after calibration.