acceptodds
Under review as a conference paper at ICLR 2027

Learning from Teachers without Training: Functional Geometry for Capability Transfer

Abstract

Post-training endows pretrained models with advanced capabilities but typically relies on iterative rollout generation and gradient-based optimization, making methods such as On-Policy Distillation (OPD) computationally expensive. We investigate whether a Base model can directly acquire a capability already learned by a post-trained Teacher, without reproducing the training process through which that capability was obtained. We find that parameter-space displacement poorly reflects the actual computational consequence of a Teacher update. In contrast, evaluating the same update on Base activations induces a functional geometry that better captures its nonlinear consequence and remains consistent across different activation domains. Motivated by these observations, we propose Functional Geometry Calibration (FGC), a training-free framework that calibrates channel-wise Teacher updates under the Base-induced functional geometry and restores the response magnitude of the corresponding Base channels. FGC requires neither rollout generation nor gradient-based optimization, and independently calibrated capability updates can be directly composed to integrate complementary capabilities from multiple Teachers. Experiments on multimodal reasoning and visual perception show that FGC enables a Base model to acquire capabilities from individual post-trained Teachers and to compose complementary capabilities from multiple Teachers using only checkpoints and a small set of calibration inputs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Training-free Model Adaptation, Multimodal Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PfiFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to transfer the capabilities of post-trained teacher models to a base model without requiring additional training. The authors observe that functional responses, derived from the base model’s activations, better reflect the computational impact of teacher updates than the magnitude of parameter changes; furthermore, post-training tends to approximately preserve the response scales across channels. Based on these insights, the authors propose Functional Geometric Calibration (FGC), a method that facilitates capability transfer through channel-wise update scaling and the restoration of response scales. FGC also allows for the direct combination of independently calibrated updates from multiple teachers. Experiments in multimodal mathematical reasoning and visual perception demonstrate that this approach can efficiently integrate complementary capabilities using only a small number of calibration samples.

Critical strengths
  1. The motivation is clear. For a post-trained teacher model that has already been obtained, the paper explores whether the capabilities embedded in its weights can be directly reused, thereby avoiding the costs associated with regenerating training data and performing iterative optimization.

  2. The FGC method is simple, computationally inexpensive, and easy to implement. It primarily consists of channel-wise scaling and response magnitude restoration, requiring no gradient-based training, matrix decomposition, or matrix inversion.

Critical weaknesses
  1. The experiments utilize only Qwen2.5-VL-7B-Instruct, along with two teacher models derived from the same base model, focusing primarily on mathematical reasoning and visual perception. The authors are encouraged to extend this work to other model architectures, parameter scales, post-training methodologies, and capability domains.

  2. The paper proposes "Post-training preserves the Base functional radius.". It remains unclear whether this is specific to individual models or a universal principle.

  3. The response magnitudes of the teacher model and the base model are very similar, a fact that may be linked to the relatively small scale of the post-training updates themselves, and which does not necessarily imply the existence of specific functional-geometric constraints imposed by the post-training process.

Questions for authors

See Weakness

3 comments

We sincerely thank the reviewer for the thoughtful questions and suggestions. We have extended our evaluation to a different model generation and scale and conducted further analyses of functional-radius preservation, as detailed below.

  • For Question 1

To further evaluate the generalizability of FGC, we conduct additional experiments on Qwen3-VL-4B-Instruct with two new post-trained Teachers, MMFineReason-4B and P2R-4B. This extends our evaluation across model generations (Qwen2.5-VL and Qwen3-VL) and parameter scales (7B and 4B).

Performance comparison on Qwen3-VL-4B across mathematical reasoning and visual perception benchmarks.

MethodVistaMVMathVerse avg.TDVDTLVIVOV*HR-4KHR-8K
Qwen3-VL-4B-Instruct77.045.142.353.042.044.741.929.979.677.972.3
MMFineReason-4B80.053.673.181.369.872.871.270.681.280.176.5
P2R-4B74.345.742.253.642.943.041.630.181.779.675.8
OrthoMerge78.748.051.960.752.954.252.738.883.379.374.8
ISO-Merger77.948.753.061.955.154.354.639.282.278.474.3
FGC (Ours)81.853.071.279.770.272.769.763.785.380.176.6

FGC outperforms OrthoMerge and ISO-Merger across all six aggregate benchmark metrics. In particular, FGC achieves 71.2 on MathVerse, compared with 53.0 and 51.9 for the two baselines, approaching the reasoning Teacher's 73.1. This advantage extends across all five MathVerse subsets, supporting the effectiveness of FGC in preserving specialized capabilities across different model generations and scales.

We have included these results in Section 5.5 (Cross-Model Generalization) of the revised manuscript.

  • For Question 2

We clarify that functional-radius preservation is an empirical observation rather than a universal principle or theoretical guarantee. To examine its generality, we extend the analysis from Qwen2.5-VL-7B to Qwen3-VL-4B, covering two model generations and parameter scales with four post-trained Teachers. For each Base--Teacher pair, we measure the functional-radius ratio ρj\rho_j across all language FFN layers and Gate, Up, and Down projections using the corresponding Base activations.

Median absolute log-deviation of functional-radius ratios across Base--Teacher pairs.

BaseTeacherGateUpDown
Qwen2.5-VL-7BZwZ-7B0.0004380.0005120.000505
VL-Rethinker-7B0.0005900.0006690.000639
Qwen3-VL-4BP2R-4B0.0001200.0001640.000158
MMFineReason-4B0.0085150.0059530.005195

Approximate functional-radius preservation is observed across both model settings, with median deviations below 10−310^{-3} for three Teachers. MMFineReason-4B exhibits larger deviations of approximately 0.0050.005--0.0090.009, indicating that the degree of preservation varies across Teachers. These findings extend the observation beyond the original Qwen2.5-VL-7B setting without establishing a universal property of post-training.

We have clarified its empirical nature in Section 3.2 and included the cross-model analysis in Appendix B.2 of the revised manuscript.

  • For Question 3

We agree that small post-training updates can contribute to functional-radius preservation, which we interpret as an empirical observation rather than a geometric constraint imposed by post-training. To examine this explanation, we measure the relative functional update magnitude Ej=∥ΔWj∥A/∥WB,j∥AE_j=\lVert\Delta W_j\rVert_A/\lVert W_{B,j}\rVert_A and radius deviation Dj=∣log⁡ρj∣D_j=\lvert\log\rho_j\rvert across all 28 language FFN layers of ZwZ-7B and VL-Rethinker-7B. By the reverse triangle inequality, ∣ρj−1∣≤Ej\lvert\rho_j-1\rvert\leq E_j, showing that sufficiently small functional updates naturally lead to similar Base and Teacher response magnitudes.

Relative functional update magnitude and radius deviation.

TeacherProjectionMedian EjE_jMedian DjD_j
ZwZ-7BGate0.0014940.000438
Up0.0030360.000512
Down0.0035340.000505
VL-Rethinker-7BGate0.0021060.000590
Up0.0038970.000669
Down0.0045120.000639

The median EjE_j ranges from 0.00150.0015 to 0.00450.0045, confirming that the typical functional updates are small relative to the Base responses. Together with the reverse triangle inequality, these results support the reviewer's explanation. However, Functional Rebase does not assume a constrained post-training trajectory; it restores the Base response scale after Functional Calibration. Its effectiveness is independently supported by the ablation study, where MathVision improves from 33.233.2 without rebase to 37.537.5 with Functional Rebase.

We have clarified the empirical nature of functional-radius preservation in Section 3.2 of the revised manuscript.

Thanks for the author's response. I noticed that the MathVerse average and five subset scores for Qwen3-VL-4B-Instruct in the newly added Table 6 (42.3 / 53.0 / 42.0 / 44.7 / 41.9 / 29.9) are exactly the same as the corresponding results for FGC–ZwZTeacher in Table 3. Could the authors please explain why these results are the same?

We sincerely thank the reviewer for identifying this inconsistency. Upon rechecking the results, we found that the MathVerse scores for FGC–ZwZ Teacher in Table 3 were mistakenly copied from the Qwen3-VL-4B-Instruct results during table preparation. We apologize for this reporting error.

The correct MathVerse scores are 46.8 (avg.), 56.1 (TD), 44.3 (VD), 47.6 (TL), 44.5 (VI), and 41.2 (VO). The results in Table 6 and the visual perception results in Table 3 are unaffected. We have corrected Table 3 in the updated manuscript.

Reviewer_tBzsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether a Base model can acquire a post-trained Teacher’s capability without training, using only checkpoints and a few calibration inputs. It characterizes each channel’s update ΔW_j by its functional response on Base activations, ‖AΔW_jᵀ‖. This is a norm induced by the activation covariance G_A = AᵀA/N. The paper reports two findings: 1) Functional magnitude predicts a channel’s intervention-based nonlinear FFN consequence better than parameter magnitude (Table 1), and 2) Post-training leaves each channel’s functional radius ‖W_j‖_A almost unchanged (median ratio 1.0000). The proposed Method (FGC) is composed of three terms: 1) Functional Calibration rescales each channel’s update by (E_j/median E)^α, where E_j is the update’s response relative to the Base channel’s response, 2) Functional Rebase then radially rescales the resulting weight row so its functional radius matches the Base row’s, and 3) For multiple Teachers, independently calibrated updates are added with unit weight. A vision-encoder variant, which equalizes rather than amplifies channel responses, is used only for single-Teacher transfer.

Critical strengths
  1. The method is simple, closed-form and cheap. FGC needs only checkpoints, 8 calibration examples, and activation–weight products, with no rollouts, gradients, or matrix factorization. It is several times faster than OrthoMerge and ISO-Merger (Table 4). If its gains held up, it would be easy to adopt for merging same-Base specialists.

  2. The functional-space diagnostics are informative. Appendix B.2 compares Teacher and OPD-trained updates from the same Base. Their alignment rises from 0.155 in parameter space to 0.420 in activation space, and the covariance controls (diagonal-only or mismatched features) remove most of this effect.

Critical weaknesses
  1. The novelty claim overlooks closely related prior work. The paper claims to be “the first” to characterize transfer through Base-activation-induced geometry. However, RegMean (Jin et al., ICLR 2023) already merges linear layers in closed form using the same per-layer activation Gram matrix AᵀA. Fisher merging (Matena & Raffel, 2022) and activation-aware weight importance (Wanda, AWQ, and GPTQ’s XᵀX objective) are also standard. None of these are cited, and RegMean and Fisher merging are natural missing baselines.

  2. The motivating findings are close to tautological. In Table 1, F_j directly measures the change a channel update makes to the FFN’s intermediate activations, while S_j measures the downstream effect of removing that same change. F_j predicting S_j better than the parameter norm P_j is therefore expected, and P_j is a weak comparison. Similarly, ρ_j ≈ 1 mostly reflects the fact that post-training deltas are tiny relative to the weights (‖W+ΔW‖_A ≈ ‖W‖_A to first order), not a structural property of post-training.

  3. Functional Rebase is not the small correction the paper describes. Because ΔW is tiny, rescaling the whole row W_temp also rescales W_B. The resulting change, (r−1)·W_B, can be comparable in size to the update itself. To first order, the rebase removes the component of the calibrated update that lies along W_B in the A-metric. The paper never analyzes this, and composition breaks the level-set property anyway, since the sum of two rebased updates is no longer on the Base radius.

  4. The design choices look like modality-specific tuning rather than a principle. The language side amplifies high-response channels (α = +0.25), while the vision side equalizes them (exponent β−1 = −0.5, clipped to [0.5, 2]). The vision variant is used for single-Teacher transfer but dropped for multi-Teacher composition. No result shows that the central thesis, calibrating with functional magnitude E instead of parameter magnitude P, improves end-to-end transfer.

  5. The headline result appears to be selected on the test set. α = 0.25 is chosen from Fig. 2, which reports test benchmarks, and it is exactly where MathVision peaks: 33.8 / 37.5 / 34.8 / 32.8 / 31.2 across α. The paper’s largest gain (MathVision 37.5 vs 34.2) is therefore the maximum of a non-monotonic test-set sweep. The rebase ablation’s main gain (33.2 to 37.5) is also concentrated on MathVision.

  6. The evaluation is too noisy to support the reported margins. Every number comes from a single sampled run at temperature 1.0, and Fig. 2 indicates Mini subsets (MathVision-mini has about 300 problems; V* has 191). With GPT-4o answer extraction and margins of 1–3 points, most differences between FGC and ISO-Merger are within plausible run-to-run variance. FGC also exceeds the reasoning Teacher itself on MathVision by about 6 points, which calls for variance estimates.

7.The baselines and ablations are incomplete. TIES, DARE, DELLA, WUDI and Param∆ are discussed but not compared.

  1. The Linear Merge coefficients are not reported. FGC adds both updates at unit weight (total scale 2), so part of its advantage may come from merge scale.

  2. It is better to show the additional ablations in terms of, plain unit-weight addition without calibration or rebase, calibration with P or F instead of E, α = 0 (rebase only), random channel rescaling; and the number of calibration examples.

  3. The paper presents FGC as an alternative to post-training and OPD, but it includes no OPD comparison and requires Teachers fine-tuned from the same Base. It is effectively a model-merging method.

Questions for authors
  1. How was α selected? If it was chosen on the reported test benchmarks, can you select it on a held-out validation split and report the resulting test numbers?

  2. Can you report mean ± std over multiple sampling seeds (or greedy decoding) for Tables 2–4? Do the MathVision and V* margins over ISO-Merger hold?

  3. Are MathVista and MathVision evaluated on the full sets or the testmini subsets in Table 2? How many problems does each contain?

  4. What coefficients does Linear Merge use? How does plain unit-weight addition of both Teacher updates (no calibration, no rebase) perform?

  5. How does FGC compare with RegMean, Fisher merging, TIES and DARE under the same calibration data?

  6. Does using E (functional) rather than P (parameter magnitude) in Functional Calibration actually improve downstream results? This is the paper’s central thesis.

  7. Can you characterize what Functional Rebase does to the update? In particular, how large is (r−1)·W_B relative to ΔW̃, and does rebase effectively remove the A-radial component of the update?

  8. Why does the language side amplify high-response channels while the vision side equalizes them? What happens with a single shared rule?

5 comments

We sincerely thank the reviewer for the constructive feedback. We have conducted additional analyses, baseline comparisons, and revisions to provide further clarification, as detailed below.

  • For Weakness 1 & Weakness 7 & Question 5

We would like to clarify that the reviewer's characterization of our novelty claim does not fully reflect the statement made in our Introduction. Specifically, our paper states: "To our knowledge, we are the first to characterize post-training capability transfer through the functional geometry induced by Base representations, revealing a fundamental mismatch between parameter displacement and the computational significance of Teacher updates." We do not claim to be the first to use activation Gram matrices or activation-aware geometry in general.

Regarding baseline selection, RegMean and Fisher Merging share certain mathematical connections with our approach, but primarily address conventional NLP model fusion rather than capability transfer in generative vision-language models. Our comparisons prioritize methods evaluated in more closely related settings, such as OrthoMerge and ISO-Merger. We have also added the relevant references in the Related Work section of the revised manuscript.

Following the reviewer's suggestion, we additionally evaluate TIES using the same Base model, two post-trained Teachers, and language-side merging scope.

Comparison with the additional TIES baseline.

MethodMathVistaMathVisionMathVerseV*HR-4KHR-8K
TIES74.132.249.972.861.954.4
FGC (Ours)74.437.551.374.963.355.9

FGC outperforms TIES across all six benchmark metrics, including improvements of 5.35.3 points on MathVision and 2.12.1 points on V*. We have included the additional TIES results in Section 5.2 (Main Results) of the revised manuscript.

  • For Weakness 2

We clarify that FjF_j measures the immediate response of a linear projection, whereas SjS_j measures its intervention-based consequence on the complete nonlinear SwiGLU FFN, including Gate, Up, and Down interactions. Moreover, they are evaluated on disjoint input examples, so their strong correlation is an empirical result rather than a mathematical identity. The comparison with PjP_j is intentional, as our objective is to assess whether parameter displacement adequately captures the computational significance of Teacher updates. Beyond response magnitude, Table 1 also evaluates the channel-wise allocation of nonlinear consequence, where activation-conditioned measures consistently outperform PjP_j across Spearman, TV, and JS metrics.

Regarding functional-radius preservation, we agree that small post-training updates can contribute to the similarity between Base and Teacher response magnitudes. Our additional analysis of ZwZ-7B and VL-Rethinker-7B confirms that the median Base-relative functional update magnitudes range from 0.00150.0015 to 0.00450.0045 across Gate, Up, and Down projections. We therefore interpret radius preservation as an empirical observation rather than evidence of a specific geometric constraint imposed by post-training.

  • For Weakness 3 & Question 7

Our paper does not describe Functional Rebase as a "small correction." As defined in Section 4.2, it restores the Base functional radius through radial rescaling, which can modify the calibrated update. Let bjb_j denote the Base channel weights and uju_j denote the calibrated update. Writing the rebased endpoint as WFGC,j=rj(bj+uj)W_{\mathrm{FGC},j}=r_j(b_j+u_j), the resulting update is

ΔWFGC,j=rjuj+(rj−1)bj.\Delta W_{\mathrm{FGC},j}=r_j u_j+(r_j-1)b_j.

For small relative updates, its first-order approximation is

\approx u_j-\frac{\langle b_j,u_j\rangle_A}{\lVert b_j\rVert_A^2}b_j,$$ showing that Functional Rebase approximately removes the $A$-radial component of the calibrated update. We quantify this effect across all 28 language FFN layers of both Teachers using the correction ratio $R_j=\lVert(r_j-1)b_j\rVert_A/\lVert u_j\rVert_A$ and the relative first-order approximation error. **Geometric effect of Functional Rebase.** | Teacher | Projection | Median $R_j$ | Approx. Error | |:---|:---|---:|---:| | VL-Rethinker-7B | Gate | 0.375 | 0.00138 | | | Up | 0.189 | 0.00249 | | | Down | 0.146 | 0.00272 | | ZwZ-7B | Gate | 0.380 | 0.00084 | | | Up | 0.206 | 0.00165 | | | Down | 0.179 | 0.00202 | The rebase correction is substantial relative to the calibrated update, mainly for Gate projections, while small approximation errors support the first-order interpretation. We clarify this in Section 4.2 of the revised manuscript. For multi-Teacher composition, independently rebased updates do not generally preserve the Base functional radius after addition. The radius constraint applies to each Teacher endpoint individually, while composition relies on the additivity of functional responses under fixed Base activations rather than radius preservation after summation.
  • For Weakness 4 & Weakness 9 & Question 6

To directly examine whether EE provides an advantage over PP and FF, we conduct additional end-to-end experiments by replacing EE with PP or FF in Functional Calibration. All three variants use the same Teachers, calibration inputs, calibration exponent α=0.25\alpha=0.25, Functional Rebase, and multi-Teacher composition setting, differing only in the calibration measure.

Ablation of channel-wise calibration variants.

MethodMathVistaMathVisionMathVerseV*HR-4KHR-8K
P-Calibration74.036.250.274.963.956.1
F-Calibration73.532.650.174.963.656.1
FGC (E-Calibration)74.437.551.374.963.355.9

As shown in the table, EE-based calibration achieves the best performance across all three mathematical reasoning benchmarks, outperforming PP-based calibration by 0.40.4, 1.31.3, and 1.11.1 points on MathVista, MathVision, and MathVerse, respectively. It also outperforms FF-based calibration, particularly on MathVision (37.537.5 vs. 32.632.6). Meanwhile, visual perception performance remains comparable across the three variants.

These results demonstrate that PP and FF can also serve as alternative calibration measures within the same framework, while the proposed EE provides better mathematical reasoning performance by accounting for both the functional response of Teacher updates and the response magnitude of the corresponding Base channels. We have included this additional ablation in Section 5.3 (Ablation Study) of the revised manuscript.

  • For Weakness 4 & Question 8

The opposite calibration strategies reflect differences in the cross-domain stability of functional responses, rather than solely benchmark-specific tuning. To examine this, we analyze the channel-wise responses of the original Teacher updates on Base activations from ZwZ-RL-VQA and MMFineReason, without applying FGC.

Cross-domain consistency of functional responses.

TeacherModalitySpearmanTop-5% Overlap
VL-RethinkerLanguage0.78661.15%
Vision0.77348.63%
ZwZLanguage0.81659.45%
Vision0.60638.87%

The top-5% overlap of high-response channels is higher for language modules (59.45%–61.15%) than for vision modules (38.87%–48.63%). This suggests that high-response language channels are more stable across calibration domains, motivating their amplification, whereas more domain-sensitive visual responses motivate conservative equalization. We have included this analysis in Appendix C.1 (Vision-Side Instantiation) of the revised manuscript.

We further evaluate this design by comparing FGC with a Unified Calibration variant that applies the same positive exponent α=0.25\alpha=0.25 to both modalities in single-Teacher transfer, while retaining Functional Rebase.

Comparison of modality-specific and unified calibration strategies.

MethodMathVistaMathVisionMathVerseV*HR-4KHR-8K
FGC (Ours)74.034.549.875.964.156.1
Unified Calibration73.033.249.974.964.055.3

Unified Calibration does not improve overall performance. FGC achieves higher scores on MathVista (74.0 vs. 73.0), MathVision (34.5 vs. 33.2), and V* (75.9 vs. 74.9), while MathVerse and HR-Bench remain comparable. Together with the cross-domain analysis, these results provide empirical support for our modality-specific calibration strategies. We have included this ablation in Section 5.3 (Ablation Study) of the revised manuscript.

  • For Weakness 5 & Question 1

We clarify that α=0.25\alpha=0.25 was fixed as a moderate calibration strength before the sensitivity analysis, rather than selected based on test-set performance. The subsequent analysis in Appendix C.2 shows relatively stable performance across different α\alpha values on MathVista, V*, and HR-Bench, while MathVision is more sensitive. We also acknowledge that Functional Rebase provides its largest improvement on MathVision, with more modest effects on other benchmarks. We have clarified the parameter selection procedure in Appendix C.2 (Sensitivity to Calibration Strength) of the revised manuscript.

  • For Weakness 6 & Question 2

Due to time and computational resource constraints, we focus our repeated evaluations on FGC and ISO-Merger, the baseline specifically highlighted by the reviewer. We conduct three independent evaluation runs using the same model checkpoints and sampling configuration (temperature =1.0=1.0), generating new predictions for each run. We report the mean ±\pm standard deviation and run-wise performance differences.

Performance across three independent evaluation runs (mean ±\pm std).

MethodMathVistaMathVisionMathVerseV*HR-4KHR-8K
ISO-Merger73.57 ± 0.0633.55 ± 0.0050.34 ± 0.0973.82 ± 0.0064.12 ± 0.0056.25 ± 0.00
FGC (Ours)74.27 ± 0.1237.61 ± 0.1951.16 ± 0.1175.22 ± 0.3064.00 ± 0.2155.79 ± 0.07
FGC - ISO+0.70 ± 0.17+4.06 ± 0.19+0.82 ± 0.03+1.40 ± 0.30-0.12 ± 0.21-0.46 ± 0.07

FGC maintains its advantages over ISO-Merger on MathVision and V* across all three runs, with mean improvements of 4.06 ± 0.19 and 1.40 ± 0.30 points, respectively. The differences remain positive in every run, indicating that these improvements persist across repeated evaluations. We have included the results and variance analysis in Appendix C.3 (Evaluation Stability) of the revised manuscript.

  • For Weakness 8 & Question 4

Linear Merge uses equal coefficients of 0.50.5 for the two Teachers. To examine whether FGC's improvements arise from larger merge coefficients, we additionally evaluate Raw Addition, WB+ΔW1+ΔW2W_B+\Delta W_1+\Delta W_2, without Functional Calibration or Functional Rebase, under the same experimental setting.

Comparison of merging strategies with different update coefficients.

MethodMathVistaMathVisionMathVerseV*HR-4KHR-8K
Linear Merge74.033.649.572.861.954.5
Raw Addition73.933.950.473.363.355.6
FGC (Ours)74.437.551.374.963.355.9

Increasing the merge coefficients alone improves performance on several benchmarks. However, FGC outperforms Raw Addition on MathVista, MathVision, MathVerse, and V*, with the largest improvement of 3.63.6 points on MathVision. These results indicate that FGC's improvements cannot be attributed solely to larger merge coefficients. We have included Raw Addition in Section 5.2 (Main Results) and the merge-scale comparison in Appendix C.4 (Effect of Merge Scale) of the revised manuscript.

  • For Weakness 10

FGC is not intended to replace the post-training process that produces specialized Teachers. Instead, it transfers capabilities already acquired by post-trained Teachers without repeating iterative training procedures such as OPD. We acknowledge that FGC currently requires Teachers derived from a shared Base model.

For comparison, we evaluate a basic policy-gradient-based OPD baseline implemented in verl, using on-policy Student rollouts and Teacher supervision. We report its performance at training step 150 under the same Qwen2.5-VL-7B Base setting.

Comparison with OPD at training step 150.

MethodMathVistaMathVisionMathVerseV*HR-4KHR-8K
OPD70.232.646.273.861.653.3
FGC (Ours)74.437.551.374.963.355.9

FGC achieves higher scores across all six benchmarks without requiring additional Student training during capability transfer. However, this comparison uses a limited OPD training budget and does not establish superiority over fully converged OPD. We have included this comparison in Appendix C.5 (Comparison with On-Policy Distillation) of the revised manuscript.

  • For Question 3

MathVista and MathVision are evaluated on their testmini subsets, containing 1,000 and 304 problems, respectively. We have clarified the evaluation splits and dataset sizes in Section 5.1 (Experimental Setup) of the revised manuscript.

Reviewer_ahuVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates training-free capability transfer from post-trained teacher models to a shared base model. The authors first provide two empirical observations. First, the functional magnitude of a teacher update, measured through its response on base-model activations, correlates more strongly with its nonlinear FFN intervention effect than its Euclidean parameter magnitude. Second, post-trained teacher channels approximately preserve the functional radius of their corresponding base-model channels.

Motivated by these observations, the authors propose Functional Geometry Calibration (FGC), which consists of two steps: Functional Calibration, which rescales channel-wise teacher updates according to their base-relative functional magnitudes, and Functional Rebase, which restores the base-induced functional radius of the resulting weights.

Experiments on Qwen2.5-VL-7B demonstrate promising results on multimodal reasoning and visual perception benchmarks, together with relatively low merging overhead.

Critical strengths
  1. The paper provides empirical evidence that functional magnitudes better characterize nonlinear FFN intervention effects than parameter magnitudes. The strong correlations reported in Table 1 are encouraging.
  2. FGC requires only a small number of calibration examples and channel-wise operations, without gradient-based optimization or expensive matrix decomposition. The reported merging time is substantially lower than those of the selected baselines.
Critical weaknesses
  1. Section 3.2 reports an interesting observation that post-trained Teacher channels approximately preserve the functional radius of their corresponding Base channels. Specifically, the authors report a median functional-radius ratio of 1.0000 across Gate, Up, and Down projections, with median absolute log-deviations below 10−310^{-3} . However, the experimental setup underlying this observation is not sufficiently described. Moreover, the reported median statistics alone are insufficient to characterize how consistently this property holds across different channels, layers, and Teachers. Since functional-radius preservation serves as a central motivation for the proposed Functional Rebase, a more detailed empirical analysis is necessary.
  2. The experiments are conducted using only one 7B Base model and two specialized Teachers, which limits the generality of the conclusions. Several improvements over the strongest baselines are relatively small, making their reliability difficult to assess. The sensitivity to calibration data selection is also unclear, given that only eight calibration examples are used per Teacher.
  3. The language-side FGC amplifies updates with larger relative functional responses, whereas the vision-side implementation applies inverse-response scaling to suppress highly responsive channels. The paper does not sufficiently explain why these opposite strategies are appropriate for different modalities. Furthermore, the multi-Teacher experiments only compose language-side updates, while the visual backbone remains unchanged.
Questions for authors
  1. Could the authors clarify the experimental setup in Section 3.2 and provide more detailed statistics to support the reported observation?
  2. Can the authors provide evidence that FGC generalizes beyond Qwen2.5-VL-7B, such as results on other model architectures or model scales?
  3. What motivates the opposite calibration strategies for language and vision modules, and why are vision-side updates excluded from multi-Teacher composition?
3 comments

We sincerely thank the reviewer for the careful evaluation of our work and the constructive comments and suggestions. We appreciate the opportunity to clarify the raised concerns and provide additional experimental evidence. Below, we address each weakness and question in detail.

  • For Weakness 1 & Question 1

To further examine functional-radius preservation across channels, layers, and Teachers, we analyze ZwZ-7B and VL-Rethinker-7B, both derived from Qwen2.5-VL-7B-Instruct. Using Base activations from ZwZ-RL-VQA and MMFineReason, we compute the functional-radius ratio ρj\rho_j across all 28 language FFN layers and Gate, Up, and Down projections, without applying FGC. Beyond the original median statistics, we report the 95th and 99th percentiles of ∣log⁡ρj∣|\log\rho_j| and the proportion of measurements satisfying ∣log⁡ρj∣<0.01|\log\rho_j|<0.01.

Functional-radius preservation across Teachers and FFN projections.

TeacherProjectionMedian ρ\rhoMedian ∣log⁡ρ∣\lvert\log\rho\rvertP95P99<0.01\lt 0.01 (%)
ZwZ-7BGate1.0000010.0004380.0023980.00420399.97
Up0.9999920.0005120.0049210.01301398.31
Down0.9999990.0005050.0029590.00619699.69
VL-Rethinker-7BGate0.9999990.0005900.0034560.00651299.72
Up1.0000080.0006690.0057620.01273798.17
Down0.9999940.0006390.0045730.01240298.57

Across both Teachers and all three projections, the median absolute log-deviations remain below 7×10−47\times10^{-4}, and more than 98%98\% of channel-level measurements satisfy ∣log⁡ρj∣<0.01|\log\rho_j|<0.01. The maximum layer-wise median deviation is approximately 0.00570.0057, indicating that functional-radius preservation is observed across most channels and layers rather than only in the pooled medians.

We have clarified the empirical nature of this observation in Section 3.2 and included the detailed experimental setup and additional statistics in Appendix B.2 of the revised manuscript.

  • For Weakness 2 & Question 2

We thank the reviewer for raising the concern about generalizability. To further evaluate FGC beyond the original Qwen2.5-VL-7B setting, we conduct additional experiments on Qwen3-VL-4B-Instruct with two new post-trained Teachers, MMFineReason-4B and P2R-4B, covering a different model scale and architecture generation.

Performance comparison on Qwen3-VL-4B across mathematical reasoning and visual perception benchmarks.

MethodVistaMVMathVerse avg.TDVDTLVIVOV*HR-4KHR-8K
Qwen3-VL-4B-Instruct77.045.142.353.042.044.741.929.979.677.972.3
MMFineReason-4B80.053.673.181.369.872.871.270.681.280.176.5
P2R-4B74.345.742.253.642.943.041.630.181.779.675.8
OrthoMerge78.748.051.960.752.954.252.738.883.379.374.8
ISO-Merger77.948.753.061.955.154.354.639.282.278.474.3
FGC (Ours)81.853.071.279.770.272.769.763.785.380.176.6

FGC outperforms OrthoMerge and ISO-Merger across all six aggregate benchmark metrics, with particularly substantial improvements on MathVerse (71.2 vs. 53.0 and 51.9). Notably, FGC retains much of the mathematical reasoning Teacher's performance (73.1), whereas the other merging methods exhibit considerable degradation. One possible explanation is the larger Teacher--Base capability gap in this setting: MMFineReason improves MathVerse from 42.3 to 73.1, compared with 43.6 to 50.9 for Qwen2.5-VL-7B. This suggests that FGC may be particularly effective at preserving strongly specialized capabilities during multi-Teacher composition.

We have included these additional experiments and results in Section 5.5 (Cross-Model Generalization) of the revised manuscript.

Regarding evaluation reliability, we conduct three independent evaluation runs for FGC and ISO-Merger. FGC maintains its advantages on MathVision and V* across all three runs, with mean improvements of 4.06±0.194.06\pm0.19 and 1.40±0.301.40\pm0.30 points, respectively (Appendix C.3).

  • For Weakness 3 & Question 3

The different calibration strategies are motivated by the distinct functional response characteristics of language and vision modules. To investigate this distinction, we analyze the cross-domain stability of the original Teacher updates using calibration inputs from ZwZ-RL-VQA and MMFineReason, without applying FGC.

Cross-domain consistency of functional responses.

TeacherModalitySpearmanTop-5% Overlap
VL-RethinkerLanguage0.78661.15%
Vision0.77348.63%
ZwZLanguage0.81659.45%
Vision0.60638.87%

The results show that high-response language channels are more consistent across calibration domains, motivating their amplification, whereas high-response vision channels are more domain-sensitive, motivating conservative response equalization. We have included this cross-domain stability analysis in Appendix C.1 (Vision-Side Instantiation) of the revised manuscript.

To further evaluate this design, we compare the original modality-specific FGC with a unified calibration variant that applies the same positive exponent α=0.25\alpha=0.25 to both language and vision modules in single-Teacher transfer.

Calibration strategy comparison for VL-Rethinker.

MethodMathVistaMathVisionMathVerse
FGC (Ours)74.034.549.8
Unified Calibration73.033.249.9

Calibration strategy comparison for ZwZ.

MethodV*HR-Bench 4KHR-Bench 8K
FGC (Ours)75.964.156.1
Unified Calibration74.964.055.3

FGC outperforms Unified Calibration on MathVista (74.0 vs. 73.0), MathVision (34.5 vs. 33.2), and V* (75.9 vs. 74.9), while MathVerse and HR-Bench remain comparable. Together with the cross-domain stability analysis, these results provide empirical support for the modality-specific calibration strategies. We have included this ablation in Section 5.3 (Ablation Study) of the revised manuscript.

Regarding multi-Teacher composition, the compared baselines operate only on language-side parameters and leave the visual backbone unchanged. For a fair comparison, we adopt the same language-only merging setting.

Thank you for the detailed response and additional experiments, which have addressed most of my concerns. I am considering raising my rating accordingly.

Comments on the paper

1 comment

We sincerely thank all reviewers for their careful reading and constructive feedback. As the discussion period will end in less than a day, we would greatly appreciate it if you could let us know whether our rebuttal responses and rebuttal revisions (with major changes and additional experimental results highlighted in blue) have addressed your concerns. If any questions or concerns remain, we would be happy to provide further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.