acceptodds
Under review as a conference paper at ICLR 2027

AttnCal: Native Attention Construction and Visual Calibration for Multimodal Model Merging

Abstract

Attention logits depend jointly on query and key (Q/K) projections, so preserving each projection separately need not preserve their interaction. In multimodal merging, this interaction must also process visual features from a consolidated interface. AttnCal addresses both requirements through native attention construction and visual-conditioned calibration. It reconstructs joint rotary-position operators under the original head-width constraint, targeting the worst expert-relative reconstruction error within each head. The construction uses expert weights alone, provides computable objective bounds, and controls attention-logit error for fixed hidden inputs uniformly over relative position. AttnCal calibrates the constructed Q/K factors under a fitted visual map, then fixes the language weights while adapting a shared visual residual, retaining one static model and the original visual interface. Across four reproduced benchmark–backbone protocols, AttnCal achieves the highest final average score (FAA) among merged baselines. On six UCIT tasks with LLaVA-1.5-7B and 3,000 examples per task, it reaches 76.07 FAA, 82.72 continual average score (CAA), and 0.33 backward transfer (BWT), compared with 72.15, 81.60, and for the strongest FAA baseline. These results connect a native-feasible attention target to the visual state under which the merged model is calibrated and deployed.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Model Merging

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Ppm9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AttnCal, a merge of task-specialized multimodal models into one static checkpoint. Its central proposal rebuilds the rotary query-key operator rather than averaging weights. Each frequency pair is an outer product of rank at most one, so the average of expert operators can exceed the head's rank, while the product of averaged factors changes the match. AttnCal constructs, per head, a new rank-at-most-one operator that minimizes the worst expert's relative error summed over frequencies, using weights alone, and restores factors at the original width. It then calibrates the query and key on task answers under one visual map and fits a shared visual residual with the language weights frozen.

The paper evaluates UCIT and DCL on LLaVA-1.5-7B and InternVL-Chat-7B against merged baselines. AttnCal obtains the highest final average score on all four protocols. On six UCIT tasks with LLaVA-1.5-7B, it reaches 76.07 FAA, 82.72 CAA, and 0.33 BWT, compared with 72.15, 81.60, and −1.52-1.52 for the strongest FAA baseline. Separate task experts remain stronger than the single merged model.

Critical strengths
  1. This paper addresses a practically important inconsistency in multimodal model merging: averaging query and key weights need not preserve the rotary interaction that the original head can store.
  2. The experimental evaluation is relatively comprehensive, covering four reproduced benchmark-backbone protocols with 3 different metrics.
Critical weaknesses
  1. The final-average lead is not tied to the head-coupled query–key objective. Figure 3 compares three native-width constructions before task calibration and says neither panel is task accuracy. Tables 1–4 then score only the full pipeline: on UCIT with LLaVA-1.5-7B, 76.07 FAA against 72.15 for the matched MAny run. Table 5, with the language model held fixed, gives only +0.40 FAA to the final visual residual. Equation 2 still adds a rank-8 query–key correction trained with answer cross-entropy, and Appendix C builds the other 160 non-query–key projections by a separate reconstruction. Appendix F bounds the initialization, not the calibrated checkpoint. No table runs the other two Figure 3 targets through that same calibration.

  2. That comparison also trains AttnCal and leaves the strongest baseline untrained. Appendix C fits the auxiliary visual map, the rank-8 query–key correction, and the visual continuation for 372 updates each, on 62 source examples per task, with answer-token cross-entropy. MAny, the strongest FAA baseline in Tables 1–4, is training-free. A gain of a fitted pipeline over an unfitted merge does not show that the operator construction is what won.

Questions for authors
  1. Could the authors provide anonymous code repository for review?
  2. If the normalized mean rank-one fit and the frequency-wise minimax from Figure 3 are passed through Equation 2 with the same non-Q/K weights, the same 62-example schedule, and the same visual basis, what are their FAA, CAA, and BWT on UCIT-LLaVA?
  3. For the expert-selection mechanism in Table 5 and Appendix A.2, does inference use the task label, a task-free score, or no choice among expert projectors? Is that procedure active for the Table 1 AttnCal row?
  4. After the rank-8 update in Equation 2, what is the worst-expert relative operator error LiL_i, compared with the initialization plotted in Figure 3? and if answer calibration stay near the minimax solution.
11 comments
AuthorsOpenReview ↗

Thank You for Your Constructive Feedback

Thank you for recognizing the practical importance of preserving native rotary interactions and the evaluation across four benchmark–backbone settings. To save you time, we have highlighted the key conclusions and information in bold. The new controls directly support the independent task contribution of the head-coupled construction and the complete method's advantage under equal calibration, addressing the two central attribution concerns. With the other 160 projections and calibration conditions fixed, head-coupled minimax improves all three task metrics and operator error. Three paired seeds under another common configuration also favor it over both alternatives. When visual experts, routing, and calibration budgets are matched with MAny, AttnCal retains a mean FAA advantage of 2.1091 percentage points. We also clarify expert availability at each stage and distinguish the initialization certificate from numerical evaluation after calibration.

AuthorsOpenReview ↗

W1. How can the construction's contribution and theoretical guarantees be isolated?

Changing only the query and key initialization under matched information access, other components, and calibration improves FAA, CAA, BWT, and operator error, supporting the independent contribution of head-coupled minimax. Head-coupled minimax improves FAA by 0.4356 and 0.5034 percentage points over normalized mean rank-one and frequency minimax, respectively. Relative to normalized mean rank-one, CAA and BWT also improve by 0.3324 and 0.1908 points. Its post-calibration operator error is 28.40% lower than normalized mean rank-one and 5.32% lower than frequency minimax. On UCIT–LLaVA, we fix the other 160 projections, the 62 source examples per task and their order, the rank-8 visual basis, optimizer, training seed, and calibration schedule. All three constructions receive rank-8 query–key calibration at a learning rate of 10⁻⁴, with 62 updates per seen task in each fitting phase and 372 updates per phase at the final stage. We use a common configuration without separately tuning the initializations. This control directly connects operator fidelity with downstream performance after identical calibration. Task metrics are calculated from all 21 stage–task combinations. Each stage restricts experts, statistics, visual bases, and fitting samples to seen tasks, separately from the fixed six-expert visual pool in the original Table 1. FAA averages the final-stage task scores, CAA averages the stage-wise task averages, and BWT measures the average change on earlier tasks from their first evaluation to the final stage. The final-stage operator metric takes each head's largest squared relative error over the six experts and averages over 1,024 heads. We will label these evaluation protocols explicitly in the revision.

ConstructionFAA ↑CAA ↑BWT ↑Mean worst-expert operator error after calibration ↓
Normalized mean rank-one76.216783.7193−0.64400.003467003
Frequency minimax76.148983.5634−0.65530.002622141
Head-coupled minimax76.652384.0517−0.45320.002482520

To test repeatability, we run three paired seeds under another configuration selected before confirmation. All constructions share the remaining weights, sample order, and visual basis within each seed. The query–key learning rate is 2.5 × 10⁻⁵, the visual learning rate is 10⁻⁴, auxiliary visual fitting and query–key calibration each use 372 updates, and final visual fitting uses 186 updates. Head-coupled minimax exceeds both alternatives on every seed, improving mean FAA by 0.4225 and 0.4135 percentage points over normalized mean rank-one and frequency minimax. Its sample standard deviation is 0.0312. The paired differences from frequency minimax are 0.4239, 0.3773, and 0.4392 points, consistent in direction and similar in magnitude. The leading ordering therefore holds in both tested calibration configurations. The common configuration is selected in two steps. We first use 120 source tuning examples to choose the query–key learning rate from 2.5 × 10⁻⁵, 5 × 10⁻⁵, and 10⁻⁴, averaging answer cross-entropy equally across tasks and across the three constructions and MAny. A separate 600-example source set then selects the budget from 0, 46, 93, and 186 updates, averaging over these four methods and both Retain and Reset. Both choices are frozen before confirmation, and every construction and seed uses the selected configuration. The table reports final-stage FAA over the six tasks, with SD denoting the sample standard deviation across training seeds.

Training seedHead-coupled minimax FAA ↑Normalized mean rank-one FAA ↑Frequency minimax FAA ↑
202876.782276.350676.3583
202976.758476.351776.3811
203076.820376.391176.3811
Mean ± SD76.7870 ± 0.031276.3645 ± 0.023176.3735 ± 0.0132

The initialization certificate and the calibrated model are evaluated separately. Appendix F applies to the constructed initialization, whereas answer calibration changes the query and key weights. We therefore recompute the same minimax objective for the calibrated weights, keeping the experts, normalization, and target fixed. The measured worst-expert error supplies a feasible upper bound, and a numerical lower bound is recomputed for the same objective. For the default calibrated state, the upper–lower bound gap averages 3.56 × 10⁻⁵ and reaches a maximum of 2.85 × 10⁻⁴ across heads. This provides a numerically estimated upper bound on suboptimality after calibration. Q4 reports the full error and bound comparison. The revision will distinguish this numerical assessment from the analytical initialization certificate and retain the fixed-input scope of the attention-error bound.

AuthorsOpenReview ↗

W2. Does AttnCal still outperform MAny when calibration opportunities are matched?

With visual experts, routing, and calibration budgets matched, AttnCal exceeds MAny on all three paired seeds by a mean of 2.1091 FAA percentage points. We reuse the MAny merged weights from Table 1 with its published parameters and give both methods the same calibration schedule. They use the same six frozen visual projectors and expert prototypes, the same CLIP image features and cosine-similarity routing, and softmax temperature 0.1. They also share the LLaVA-1.5-7B backbone, the 62 source examples per task and their order, rank-8 corrections, and visual basis. Each method independently fits its corrections using AdamW at 10⁻⁴, with 372 updates each for auxiliary visual fitting, query–key calibration, and final visual fitting. Evaluation uses the same 3,000 examples for each of the six tasks. Calibration improves MAny from 72.1594 FAA to a mean of 74.0805, confirming that additional training helps, but does not remove the difference from AttnCal. This comparison evaluates each complete language-weight construction under matched visual conditions and training opportunity. W1 isolates the query–key contribution by fixing the other 160 projections. The original Table 1 reports 76.07 and 72.15 for the original complete-method comparison, while the table below reports the additional matched-calibration runs.

Training seedMAny FAA with matched calibrationAttnCal FAAAttnCal − MAny (pp)
202874.110076.6523+2.5423
202974.052275.9578+1.9056
203074.079475.9589+1.8795
Mean ± SD74.0805 ± 0.028976.1897 ± 0.4007+2.1091 ± 0.3754
Reviewer_NBCCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AttnCal, a multimodal model-merging framework that enables high-performance expert integration by constraining the minimax reconstruction error of the Q/K operators and performing visual-residual calibration with the language model frozen. The main contributions are summarized as following:

  1. Native-width joint construction of Q/K operators, together with theoretical bounds.
  2. A two-stage, visual-state-guided calibration procedure.
  3. State-of-the-art merging performance across multiple multimodal benchmarks and backbone architectures.
Critical strengths
  1. Theoretical rigor: The method is supported by a substantial theoretical analysis, including native-feasibility results, primal-dual lower bounds, and a fixed-input upper bound on attention-logit reconstruction error. Formulating multimodal model merging at the operator level provides a novel and well-motivated perspective.

  2. Systematic evaluation: The paper conducts a broad evaluation across multiple multimodal benchmarks and backbone architectures, reporting both final average accuracy (FAA) and continual-learning metrics such as CAA and BWT. Fine-grained ablations are also included, containing visual-residual scale sweeps and visual-handoff control experiments, to examine the contributions of the individual components.

Critical weaknesses
  1. The practical gain from Stage 2 visual calibration appears marginal. According to Table 5, adding the visual residual improves FAA from 75.68 to 76.07, corresponding to a relatively modest gain of 0.40 points. Moreover, in the handoff study in Table 7, the “Keep” setting, which does not perform further visual-residual training, achieves a higher mean FAA than the “Retain” setting, which continues fine-tuning from the retained visual state. These results weaken the case for the necessity of the more complex Stage 2 procedure and suggest that the claimed benefit of sharing the visual state may not be robust across different runs, given the observed variation.

  2. The computational overhead and efficiency of AttnCal are not sufficiently discussed. AttnCal involves a complex two-stage fine-tuning procedure, including minimax optimization and Q/K calibration in Stage 1, followed by additional visual-residual training in Stage 2. Compared with the other baseline methods evaluated in the paper, it remains unclear whether this two-stage fine-tuning introduces substantially higher computational overhead. The paper does not provide an efficiency comparison between AttnCal and commonly used model-merging methods.

Questions for authors
  1. Clarification on the necessity of Stage 2. Could the authors clarify the results in Table 7 by explaining in greater detail the differences among the Keep, Retain, and Reset settings, and how these settings relate to the Stage 2 visual-residual fine-tuning described in the main text? In particular, why does Keep outperform the other two settings on average? Is this result expected, and how can it be interpreted from the perspective of visual-language interaction?

  2. Additional computational cost. Could the authors report the additional time and GPU memory required by AttnCal, particularly for the minimax optimization in Stage 1 and the visual-residual fine-tuning in Stage 2, compared with conventional model-merging methods?

7 comments
AuthorsOpenReview ↗

Thank You for Your Constructive Feedback

Thank you for recognizing the operator-level merging formulation, native feasibility, and fixed-input error bounds, and for considering visual adaptation together with its computational cost. To save you time, we have highlighted the key conclusions and information in bold. The additional evidence supports repeatable improvements from short visual adaptation and a total offline time advantage for the default calibrated pipeline in the measured implementation. After freezing the 93-step configuration, all five new seeds favor Retain over Keep, with a mean FAA improvement of 0.3661 percentage points. Separately, cost measurements use the default configuration of 372 updates per fitting phase. Across five measurements per method on the same eight-A800 machine, with cache warm-up and alternating execution order, AttnCal reduces mean total offline time by 62.17% relative to MAny.

AuthorsOpenReview ↗

W1. Are the benefits of Stage 2 repeatable across runs?

The paired comparisons support validation-selected short adaptation, while the benefit of the default longer continuation varies across runs. Table 5 tests deployment of the fitted residual: FAA rises from 75.68 to 76.07, a 0.40-point difference before rounding. Table 7 tests further training of a map Keep already retains. The auxiliary map is fitted first, then query and key weights are calibrated under it. Stage 2 fixes these language weights and updates visual coefficients. Our comparisons fix sample order and share query and key weights and the visual basis within each seed. Retain continues from fitted coefficients, while Reset zeros them before same-budget fitting. At 372 updates, Retain exceeds both controls on seed 2028 but falls below them on the other two. Its mean is nearly equal to Keep, with paired-difference sample SD 0.3493. Means, differences, and sample SDs are computed before rounding to four decimals, including +0.0000. Table differences are in percentage points.

Training seedKeep FAARetain FAA (372 steps)Reset FAA (372 steps)Retain − Keep (pp)Retain − Reset (pp)
202876.250076.652376.1556+0.4023+0.4967
202976.183975.957876.1094−0.2261−0.1516
203076.135075.958976.0700−0.1761−0.1111
Mean76.189676.189776.1117+0.0000+0.0780
Sample SD0.05770.40070.04280.34930.3632

After freezing the source-selected 93-step configuration, Retain exceeds Keep on all five new seeds by 0.3205–0.4076 points, averaging 0.3661, and Reset by 0.6853 on average. We use 20 source examples per task, totaling 120, for tuning and a separate 120 for confirmation, disjoint from fitting and evaluation. Candidate visual learning rates are 2.5 × 10⁻⁵, 5 × 10⁻⁵, and 10⁻⁴, budgets are 93, 186, and 372, and update multipliers are 0.25, 0.5, and 1. Answer cross-entropy is averaged equally over tasks, the initial three seeds, and Retain and Reset, selecting 10⁻⁴, 93 updates, and the original magnitude. Differences within 10⁻¹² favor fewer updates, then lower learning rate, then a multiplier closer to the original magnitude. Confirmation does not trigger reselection. Query–key learning rate stays at 10⁻⁴. Seed 2028 reuses an existing state, while 2029 and 2030 are fitted independently. Seeds 2031–2035 are run after freezing the configuration. Across all eight, mean improvements over Keep and Reset are 0.2676 and 0.5747 points, with paired-difference sample SDs 0.1403 and 0.1693. Initial and confirmation runs are separated below.

Run groupTraining seedKeep FAARetain FAA (93 steps)Reset FAA (93 steps)Retain − Keep (pp)Retain − Reset (pp)
Historical202876.250076.352275.9194+0.1022+0.4328
Historical202976.183976.243975.8722+0.0600+0.3717
Historical203076.135076.283375.9167+0.1483+0.3666
Additional203176.188376.536875.9261+0.3485+0.6107
Additional203276.205076.525575.9150+0.3205+0.6105
Additional203376.190676.598275.7600+0.4076+0.8382
Additional203476.198976.565075.8950+0.3661+0.6700
Additional203576.186776.574475.8772+0.3877+0.6972
Additional mean—76.193976.560075.8747+0.3661+0.6853
All eight mean—76.192376.459975.8852+0.2676+0.5747

Another source-selected configuration uses query–key learning rate 2.5 × 10⁻⁵ and visual learning rate 10⁻⁴, with 372 updates each for auxiliary visual fitting and query–key calibration. Retain and Reset then receive 186 updates with each seed's language weights and visual basis fixed. Retain exceeds Keep and Reset on all three seeds by means of 0.5781 and 0.9718 points. Selection first chooses the query–key learning rate from 2.5 × 10⁻⁵, 5 × 10⁻⁵, and 10⁻⁴ on 120 source examples, averaging answer cross-entropy equally over tasks and head-coupled minimax, normalized mean rank-one, frequency minimax, and MAny. A separate 600-example set selects the budget from 0, 46, 93, and 186, averaging over these methods and Retain and Reset, with ties favoring fewer updates. Both choices are frozen before confirmation. Use short Retain when it improves independent validation performance over Keep, and otherwise preserve the map.

Training seedKeep FAARetain FAA (186 steps)Reset FAA (186 steps)Retain − Keep (pp)Retain − Reset (pp)
202876.207876.782275.7561+0.5744+1.0261
202976.233376.758475.8489+0.5251+0.9095
203076.185676.820375.8406+0.6347+0.9797
Mean76.208976.787075.8152+0.5781+0.9718
AuthorsOpenReview ↗

W2. How much computational overhead does the two-stage procedure introduce?

Including default calibration, AttnCal uses 62.17% less mean total offline time than MAny in five repeated measurements on the same machine. Both methods start from the same trained experts and covariance statistics. We use one eight-A800 machine, uniformly warm the cache, alternate method order, and measure each method five times. Timing includes weight construction and saving, model assembly, and applicable calibration. Both methods construct weights on eight GPUs and assemble the model on one GPU. AttnCal additionally performs all three fitting phases on one GPU, with 372 updates per phase. Its mean total time is 1007.40 seconds, compared with 2663.00 seconds for MAny, saving 27.59 minutes. Assembly and calibration add 426.00 seconds, while weight construction and saving reduce time by 2081.60 seconds. The standard deviations of the individual components are much smaller than the corresponding differences between methods. Thus, construction savings cover calibration cost in this implementation, hardware, and fixed configuration. The table reports component means and standard deviations and mean total time, excluding supplementary tuning and confirmation work.

MethodWeight construction and save (s, mean ± SD)Model assembly and applicable calibration (s, mean ± SD)Mean total offline time (s)
MAny2542.00 ± 16.81121.00 ± 1.312663.00
AttnCal460.40 ± 7.77547.00 ± 3.471007.40

Independent memory measurements on A100 and A800 hardware show that AttnCal's maximum per-GPU allocation over the complete offline pipeline is 2.5782 GiB below MAny's. Each same-machine comparison starts from the same experts and covariance statistics and measures peak memory allocated by PyTorch. AttnCal peaks at 14.2072 GiB during query–key calibration, while MAny peaks at 16.7854 GiB during weight construction.

HardwareMAny peak (GiB)AttnCal peak (GiB)AttnCal − MAny (GiB)
A100 80 GB16.785414.2072−2.5782
A800 80 GB16.785414.2072−2.5782

The single-GPU breakdown below isolates the additional fitting costs on UCIT–LLaVA. Final visual fitting takes 149.66 seconds, and all three fitting phases together take 433.93 seconds, each using 372 updates. The total is calculated from unrounded component times. Head-coupled minimax construction takes 1524.98 seconds, close to frequency minimax at 1508.41 seconds and above normalized mean rank-one at 48.47 seconds. Common input initialization and reading take another 329.31 seconds. This breakdown uses layer-wise construction on one GPU and is distinct from the eight-GPU construction used in the total-time comparison. Final visual fitting adds update time but does not exceed the memory peak of query–key calibration. Memory is PyTorch allocated memory, including the frozen model during fitting. The 1.1954 GiB construction peak is measured independently.

AttnCal stepTime (s)Peak GPU memory (GiB)
Stage 1: head-coupled minimax1524.981.1954
Stage 1: FP16 conversion and save20.06—
Stage 1: auxiliary visual fitting131.8914.1095
Stage 1: Q/K calibration152.3914.2072
Stage 2: final visual fitting149.6614.1386
Total fitting433.9314.2072
Reviewer_b6hFOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AttnCal for multimodal model merging. It jointly reconstructs Q/K interactions under native attention-width constraints, calibrates Q/K under a fitted visual map, and subsequently adapts a shared visual residual. Experiments cover two backbones and two benchmarks.

Critical strengths

The paper identifies a meaningful limitation of independently merging Q/K projections and explicitly accounts for architectural representability.

The construction includes computable objective bounds and clearly states the fixed-input scope of its attention-logit guarantees.

Results show consistent final-average improvements across four benchmark–backbone settings, with additional continual-learning metrics and visual-state controls

Critical weaknesses

The contribution of the proposed Q/K construction is not sufficiently isolated. Figure 3 compares reconstruction and attention errors, but matched downstream comparisons using alternative initializations followed by identical calibration are needed.

The necessity of Stage 2 is unclear. In Table 7, keeping the existing visual map without further training outperforms continuation in all three repeats, with mean FAA of 76.14 versus 75.68.

The DCL–InternVL visual-strength coefficient is selected using validation data from all five tasks, including future tasks. Prefix-local selection is needed to establish performance under a strictly online continual-learning protocol.

Questions for authors

Refer to weakness.

6 comments
AuthorsOpenReview ↗

Thank You for Your Constructive Feedback

Thank you for recognizing the native-width formulation and fixed-input error bounds, and for suggesting direct tests of the individual design choices. To save you time, we have highlighted the key conclusions and information in bold. The new evidence supports the independent task contribution of the head-coupled construction, validation-selected short visual adaptation, and operation with only seen-task information. The matched construction comparisons improve all three task metrics and operator error. Five additional seeds confirm the benefit of 93-step Retain, while the complete DCL–InternVL evaluation restricts experts, statistics, fitting data, and validation data to seen tasks.

AuthorsOpenReview ↗

W1. What does the query–key construction contribute under matched calibration?

Under the matched protocol, head-coupled minimax achieves the highest FAA, CAA, and BWT and the lowest post-calibration worst-expert operator error among the three constructions. Head-coupled minimax improves FAA by 0.4356 and 0.5034 percentage points over normalized mean rank-one and frequency minimax, respectively. Relative to normalized mean rank-one, CAA and BWT also improve by 0.3324 and 0.1908 points, while operator error decreases by 28.40%. Its operator error is 5.32% below frequency minimax. On UCIT–LLaVA, we fix the other 160 projections, the 62 source examples per task and their order, the rank-8 visual basis, optimizer, training seed, and calibration schedule. Only the query and key initialization changes. All constructions receive rank-8 query–key calibration at a learning rate of 10⁻⁴, with 62 updates per seen task in each fitting phase and 372 updates per phase at the final stage. The configuration is shared rather than tuned separately for each initialization. This control isolates the effect of query–key initialization on calibrated performance. Task metrics cover all 21 stage–task combinations, with experts, statistics, visual bases, and fitting samples restricted to seen tasks at each stage. These results are reported separately from the fixed six-expert visual pool in the original Table 1. FAA is the final-stage task average, CAA is the average of the stage-wise task averages, and BWT is the average change on earlier tasks from their first evaluation to the final stage. The operator metric takes the largest squared relative error over six experts for each head and averages over 1,024 heads at the final stage.

ConstructionFAA ↑CAA ↑BWT ↑Mean worst-expert operator error after calibration ↓
Normalized mean rank-one76.216783.7193−0.64400.003467003
Frequency minimax76.148983.5634−0.65530.002622141
Head-coupled minimax76.652384.0517−0.45320.002482520

To examine repeatability, we compare three paired seeds under another configuration selected before confirmation. Within each seed, all three constructions share the remaining weights, sample order, and visual basis. The query–key learning rate is 2.5 × 10⁻⁵ and the visual learning rate is 10⁻⁴. Auxiliary visual fitting and query–key calibration each use 372 updates, followed by 186 updates for final visual fitting. Head-coupled minimax exceeds both alternatives on every seed, with mean FAA improvements of 0.4225 and 0.4135 percentage points over normalized mean rank-one and frequency minimax, respectively. Its sample standard deviation is 0.0312. The differences from frequency minimax are 0.4239, 0.3773, and 0.4392 points, showing consistent direction and similar magnitude across runs. The leading ordering is consistent across both tested configurations. SD denotes the sample standard deviation across training seeds.

Training seedHead-coupled minimax FAA ↑Normalized mean rank-one FAA ↑Frequency minimax FAA ↑
202876.782276.350676.3583
202976.758476.351776.3811
203076.820376.391176.3811
Mean ± SD76.7870 ± 0.031276.3645 ± 0.023176.3735 ± 0.0132
AuthorsOpenReview ↗

W2. If Keep performs better under the default budget, is Stage 2 necessary?

The evidence supports Stage 2 as validation-selected short adaptation of an existing visual state to the calibrated language model. In Table 7, Keep averages 76.14 FAA versus Retain's 75.68. The auxiliary visual map is fitted first, then query and key weights are calibrated with it fixed. Keep retains this jointly adapted state. Stage 2 freezes the final language weights and updates visual coefficients. Table 7 varies sample order, whereas our new comparisons fix order and vary training seeds. At 372 updates, Retain exceeds both controls on seed 2028 but falls below them on the other two, averaging nearly the same FAA as Keep.

Within each seed, query and key weights and the visual basis are fixed. Keep preserves the map, Retain continues fitting its coefficients, and Reset zeros them before same-budget fitting. The query–key learning rate is 10⁻⁴. We hold out 20 source examples per task for tuning and another 20 for confirmation, separate from fitting, covariance estimation, and evaluation. Candidate visual learning rates are 2.5 × 10⁻⁵, 5 × 10⁻⁵, and 10⁻⁴, budgets are 93, 186, and 372, and update multipliers are 0.25, 0.5, and 1. Answer cross-entropy is averaged equally over tasks, the initial three seeds, and Retain and Reset. Selection yields 10⁻⁴, 93 updates, and the original magnitude. Differences within 10⁻¹² favor fewer updates, then lower learning rate, then a multiplier closer to the original magnitude. Confirmation does not trigger reselection. Seed 2028 reuses an existing state, while the other two are independently fitted. Retain improves on Keep by 0.1035 percentage points and Reset by 0.3904 on average. These comparisons isolate continuation and starting-state benefits. Here, pp denotes percentage points.

Training seedKeep FAARetain FAA (372 steps)Reset FAA (372 steps)Retain FAA (93 steps)Reset FAA (93 steps)Retain (93 steps) − Keep (pp)Retain − Reset (93 steps, pp)
202876.250076.652376.155676.352275.9194+0.1022+0.4328
202976.183975.957876.109476.243975.8722+0.0600+0.3717
203076.135075.958976.070076.283375.9167+0.1483+0.3666
Mean76.189676.189776.111776.293175.9028+0.1035+0.3904

With configuration and sample order frozen, all five new seeds 2031–2035, unused in selection, favor Retain over Keep by 0.3205–0.4076 points, averaging 0.3661. The mean advantage over same-budget Reset is 0.6853 points, supporting both further adaptation and retention of the learned starting state.

Training seedKeep FAARetain FAA (93 steps)Reset FAA (93 steps)Retain − Keep (pp)Retain − Reset (pp)
203176.188376.536875.9261+0.3485+0.6107
203276.205076.525575.9150+0.3205+0.6105
203376.190676.598275.7600+0.4076+0.8382
203476.198976.565075.8950+0.3661+0.6700
203576.186776.574475.8772+0.3877+0.6972
Mean76.193976.560075.8747+0.3661+0.6853

A second source-selected configuration uses query–key learning rate 2.5 × 10⁻⁵, visual learning rate 10⁻⁴, and 372 updates each for auxiliary visual fitting and query–key calibration. With language weights and visual basis fixed within each seed, Retain and Reset receive 186 updates. Retain exceeds Keep and Reset on all three seeds by means of 0.5781 and 0.9718 points. Selection first chooses the query–key learning rate from 2.5 × 10⁻⁵, 5 × 10⁻⁵, and 10⁻⁴ on 120 source examples, averaging answer cross-entropy equally over tasks and head-coupled minimax, normalized mean rank-one, frequency minimax, and MAny. A separate 600-example set selects the budget from 0, 46, 93, and 186, averaging over these methods and Retain and Reset, with ties favoring fewer updates. Both choices are frozen before confirmation.

Training seedKeep FAARetain FAA (186 steps)Reset FAA (186 steps)Retain − Keep (pp)Retain − Reset (pp)
202876.207876.782275.7561+0.5744+1.0261
202976.233376.758475.8489+0.5251+0.9095
203076.185676.820375.8406+0.6347+0.9797
Mean76.208976.787075.8152+0.5781+0.9718

Table 5 instead tests deployment of the fitted residual: with language weights and routing fixed, FAA rises from 75.68 to 76.07. For new task combinations, use short Retain when it improves independent validation performance over Keep, and otherwise preserve the map.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Thank You for Your Time, Encouragement, and Thoughtful Feedback

Thank you all for reading our paper so carefully and for sharing your feedback. We appreciate Reviewer b6hF’s recognition of the importance of respecting the native attention width and clearly stating the scope of the error guarantees. We thank Reviewer NBCC for highlighting the operator-level formulation and its theoretical analysis, and Reviewer Ppm9 for noting the practical importance of preserving rotary query–key interactions and the breadth of the evaluation.

Your feedback means a great deal to us. It helped us see where our explanations needed to be clearer and prompted more direct comparisons that deepened our understanding of when the method works well. We hope the individual responses and additional results address your concerns.

We understand that you may have many other commitments, and we are grateful for the time you have already given to our paper. If you have a chance to share any further thoughts on our responses, we would be sincerely grateful. We would be glad to clarify anything that remains unclear.

AuthorsOpenReview ↗

Thank You and Summary of the Rebuttal Discussion

Dear Area Chair and Reviewers,

As the discussion draws to a close, we thank the Area Chair for guiding the review and all 3 reviewers for their careful reading and positive overall ratings. Reviewers b6hF and Ppm9 each indicated that our responses had addressed most of their concerns. We also appreciate Reviewer NBCC’s follow-up on the practical significance of the Stage 2 improvement, which helped us state more precisely when this step is useful.

The reviewers highlighted complementary strengths of the paper. Reviewer b6hF recognized the formulation that preserves native attention-head width and the error bounds for fixed inputs. Reviewer NBCC recognized the operator-level approach to merging and its theoretical analysis, including feasibility under the native head-width constraint. Reviewer Ppm9 highlighted the practical importance of preserving native rotary interactions and the breadth of the evaluation across 4 benchmark–backbone combinations. Their questions prompted the comparisons summarized below.

On UCIT–LLaVA, we isolated the contribution of the query and key initialization by holding the other projections and calibration conditions fixed. Head-coupled minimax achieved lower worst-expert operator error and higher final average accuracy than both normalized mean rank-one and frequency minimax. Its accuracy advantage held across all 3 paired seeds under a second shared configuration [b6hF W1, Ppm9 W1]. Ppm9 also asked how the complete method compares with MAny when both receive the same calibration opportunity. With visual experts, routing, and calibration budgets matched, AttnCal exceeded MAny on all 3 paired seeds, with a mean difference of 2.1091 percentage points in final average accuracy [Ppm9 W2].

The questions from b6hF and NBCC prompted us to test whether further training improves an already fitted visual mapping. We selected a 93-step configuration using source validation data, fixed it, and evaluated it on 5 new seeds. Continuing to fit the mapping improved final average accuracy over keeping it unchanged on all 5 seeds, by 0.3661 percentage points on average [b6hF W2, NBCC W1]. This result supports short adaptation when source validation favors it. For NBCC’s cost question, we measured the default configuration of 372 updates per fitting phase. Final visual fitting took 149.66 seconds on 1 GPU, with peak memory below that of query–key calibration [NBCC W2, Q2].

In response to b6hF’s question about future-task information, we evaluated AttnCal on DCL–InternVL with experts, statistics, fitting data, and validation data restricted to tasks seen at each stage. Under this strictly online protocol, AttnCal achieved a final average accuracy of 64.41 [b6hF W3].

The additional results are documented in our rebuttal. Our original submission had a 9-page main-text limit. If a future version allows up to 10 pages, we would welcome the extra space to present the matched comparisons alongside the method and explain more clearly when Stage 2 is useful and what information is available at each stage of evaluation. We thank the reviewers for helping us identify where these details would be most useful to readers, and the Area Chair for guiding the discussion.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.