acceptodds
Under review as a conference paper at ICLR 2027

Residual Geometry Merge: State-Wise Merging of Expert Language Models

Abstract

Model merging combines specialized checkpoints into a single model with multiple capabilities. For large language models, this offers a practical way to integrate expert checkpoints that share a common base without additional training or routing. Existing methods typically combine expert updates directly or transform them before recombination. However, a transformer block is not merely a collection of independent matrices: its projections jointly read from and write to shared residual-stream states. We propose Residual Geometry Merge (RGM), an architecture-aware model merging framework that preserves residual-stream geometry induced by expert checkpoints. Starting from the Transformer Circuits view, RGM uses weight Gram shifts to summarize how each expert changes directions at residual-stream states, aggregates incident shifts at each state, and solves state-wise self-retention equations to derive merge coefficients. The method is coefficient-adaptive and distinct from uniform task-vector addition, routing, and subspace-based merging. Experiments across three shared-base LLM merging settings—Qwen2.5-7B-Instruct with coding, tool-use, and memory experts; DeepSeek-R1-Distill-Qwen-1.5B with coding and mathematics experts; and OpenMath-Nemotron-1.5B with two mathematics experts—show that RGM achieves the strongest overall performance in all three settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model Merging, Knowledge Transfer, Multi-task Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TSMGOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Residual Geometry Merge, a data-free method for merging expert LLMs that share a base model. It summarizes each expert's effect at every residual-stream state by weight Gram shifts and solves a small linear system per state to obtain merge coefficients, switching to a norm-normalized variant when experts overlap strongly. Across three LLM merging settings, RGM obtains the best average score.

Critical strengths

Simple and data-free.

Critical weaknesses
  1. The improvements are within noise. In Table 1, RGM's 63.95±0.10 is not really better than RAM's 63.90±0.48 or ISO-Merger's 63.90±0.18.

  2. The role of the core design is not demonstrated. In the three-expert 7B setting, RGM is 1.86 points above TA-sum, but PM-Delta, which drops both the residual-state unit and the Gram representation, is still 1.63 points above TA-sum and only 0.23 below RGM. So most of the gain can be obtained without the core construction.

Questions for authors
  1. Is there a significance test (e.g. a paired bootstrap over problems) showing that the advantage of RGM over RAM and ISO-Merger goes beyond evaluation noise?
  2. What does the distribution of the solved coefficients look like in the 7B setting, and how are parameters outside the three residual nodes (embeddings, LM head, norms, biases) merged? Given that PM-Delta already recovers most of the gain, what does the residual-state Gram construction add on top of it?
5 comments

We thank the reviewer for raising the questions about evaluation noise, coefficient behavior, and the incremental contribution of the residual-state Gram construction. We agree that the absolute gain over a strong merge baseline should not be overstated. We therefore report matched-run statistics below and have revised the paper to distinguish an observed mean ranking from a statistically separable improvement.

There is also a structural reason to expect the gains between strong merge methods to be modest. The merge methods in the comparison use the same fixed expert checkpoints and shared base model. A merge rule can reweight and combine the information already present in those experts, but it does not create a new expert, add model capacity, or introduce an additional task-specific training signal. Once a strong baseline has captured most of the experts' complementary capabilities, a large additional gain should not be expected. The non-significant differences from RAM and ISO-Merger are consistent with this interpretation.

1. Matched-run significance analysis. For each method, we paired the overall-average score from each of the three runs with the corresponding RGM score and performed a paired t-test. The differences below are RGM minus the comparison method:

ComparisonPer-run differencesMean differencePaired test
TA-sum+1.9270, +1.8493, +1.8074+1.8612t(2)=53.11, p=0.00035, 95% CI [+1.7104, +2.0120]
PM-Delta+0.1981, +0.2466, +0.2518+0.2322t(2)=13.58, p=0.0054, 95% CI [+0.1586, +0.3057]
ISO-Merger+0.0075, +0.3707, -0.2420+0.0454t(2)=0.255, p=0.822, 95% CI [-0.720, +0.811]
RAM+0.5954, +0.0432, -0.4948+0.0479t(2)=0.152, p=0.893, 95% CI [-1.307, +1.403]
Domain Expert reference+0.2726, +0.4238, +0.2600+0.3188t(2)=6.06, p=0.026, 95% CI [+0.092, +0.545]

Thus, RGM is not statistically separable from RAM or ISO-Merger in these three matched runs. We have clarified the wording in the paper to describe RGM as having the highest observed overall average, without implying a statistically significant advantage over those methods. The Domain Expert reference is formed by aligning the CURE code results, ToolRL BFCL results, and MemAgent memory results by run.

2.1. Coefficient distribution. We have added Appendix E.2, which visualizes the solved coefficients in each setting. For the two-expert NormBalanced-RGM branch, the normalized self-retention system is symmetric under exchanging the experts, yielding a shared coefficient at each state. In the Qwen2.5-7B-Instruct setting, which uses the Raw-RGM branch, CURE and ToolRL coefficients are below one at most residual states, while MemAgent coefficients remain nearly constant at one; overall, the coefficients remain close to one with localized state-wise deviations. The complete coefficient ranges in the 7B setting are:

ExpertRange of solved coefficients
CURE0.932--1.453
ToolRL0.772--1.018
MemAgent0.995--1.008

As discussed in Section 5.1, RGM recovers task-vector summation in the orthogonal limit and introduces state-wise coefficient corrections when residual-state Gram sections overlap. In the 7B setting, the expert-induced residual-state sections are nearly orthogonal on average, so most coefficients remain near one. Thus RGM remains close to TA-sum while responding to state-specific overlap, yielding 1.81--1.93-point improvements across the three matched runs.

2.2. Parameters outside the residual nodes. The parameters outside the three residual nodes are merged by uniform arithmetic averaging across the expert checkpoints:

=\theta_0+\frac{1}{K}\sum_i \Delta_i.$$ This includes token embeddings, the LM head, normalization parameters, and biases. Each non-node parameter tensor is averaged uniformly. We have made this rule explicit at the end of the Setup subsection in Section 3, after defining the three residual states and their incident projections.

2.3. What the residual-state Gram construction adds beyond PM-Delta. The direct answer is the matched-run comparison reported above: the complete RGM construction improves over PM-Delta by 0.1981, 0.2466, and 0.2518 points across the three runs, for a mean gain of 0.2322 points. Thus, although PM-Delta accounts for most of the improvement over TA-sum, the full residual-state Gram construction provides a small but consistent additional benefit. The existing ablation is consistent with this interpretation, with RGM reaching 63.95 compared with 63.72 for PM-Delta.

We would also like to give a toy example illustrating why Gram shifts can reveal overlap that raw task-vector inner products miss: Let uu be a unit residual direction, let e1,e2e_1,e_2 be orthonormal output directions, and consider:

\Delta_1=\alpha e_1u^\top,\qquad \Delta_2=\alpha e_2u^\top,\quad \alpha>0.$$ The updates are orthogonal in parameter space, $\langle\Delta_1,\Delta_2\rangle_F=0$, but their input-side Gram shifts are identical: $$S_i=(W_0+\Delta_i)^\top(W_0+\Delta_i)-W_0^\top W_0 =(2\alpha+\alpha^2)uu^\top,\qquad i\in\{1,2\}.$$ Thus, both updates produce the same change in squared response along the residual direction $u$, despite having zero raw parameter overlap. The raw-delta system therefore assigns $c_1=c_2=1$, whereas the identical Gram shifts give $c_1=c_2=\tfrac{1}{2}$ under the normalized self-retention system. This example isolates the change from raw deltas to Gram shifts; the additional effect of state-wise aggregation is reflected in the ablation below. We have also evaluated these three ablations with NormBalanced coefficient estimation on DeepSeek-R1-Distill-Qwen-1.5B. Their overall averages (mean $\pm$ std) are: | Method | Overall Avg. | RGM gain | |---|---:|---:| | PM-Delta | 41.51 $\pm$ 0.28 | +3.97 | | PM-Gram | 45.01 $\pm$ 0.14 | +0.47 | | Node-Delta | 41.89 $\pm$ 0.24 | +3.59 | | NormBalanced-RGM | **45.48 $\pm$ 0.07** | -- | In this setting, NormBalanced-RGM has the highest observed mean on all nine benchmarks. Its 3.97-point gain over PM-Delta is substantially larger than the 0.2322-point gain in the Qwen 7B Raw-RGM setting, and it is also 0.47 points above PM-Gram, the strongest ablation. The coefficient diagnostics explain why this setting is a high-gain regime. PM-Delta and Node-Delta measure overlap between raw task-vector deltas; their mean overlaps are only 0.096 and 0.091, so their solved coefficients stay near 0.91. PM-Gram instead compares Gram shifts, raising the mean overlap to 0.348 and lowering the mean coefficient to 0.749; this representation change accounts for most of the 3.50-point gain over PM-Delta. Full RGM additionally sums the Gram shifts of all projections incident to a residual node. This introduces cross-projection terms in the node-level Gram matrix that PM-Gram cannot see. Its mean overlap reaches 0.455 and its mean coefficient falls to 0.693. In the symmetric two-expert system, $c_v\approx 1/(1+\rho_v)$, so these lower coefficients are the direct result of the stronger shared residual-state effect. The remaining 0.47-point gain over PM-Gram is therefore the contribution of state-wise aggregation on top of the Gram representation. The numerical effect of moving from PM-Gram to the full RGM actually follows from RGM's architecture-aware formulation: it jointly considers projections that read from or write to a shared residual state. Its coefficient-estimation unit therefore follows the Transformer's residual-stream computation graph rather than individual parameter entries or isolated tensors. Coefficients are shared according to how projections interact through the residual stream, whereas coordinate-wise and per-matrix rules do not encode these architectural relationships. The additional 0.47-point gain over PM-Gram is thus the empirical effect of this architecture-aware solving unit on top of the Gram-shift representation. We have revised the claims and discussion accordingly and report the matched comparisons above. We have also added the complete coefficient visualization in Appendix E.2 and clarified the merge rule for parameters outside the residual nodes in Section 3.

We would appreciate your feedback on our response, particularly the additional NormBalanced ablations reported in part 2.3. On the DeepSeek-R1-Distill-Qwen-1.5B setting, RGM improves over PM-Delta by 3.97 points and PM-Gram by 0.47 points in overall average, providing additional evidence for the contribution of the residual-state Gram construction. Please let us know if there are remaining concerns we could clarify.

Reviewer_AeopOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Residual Geometry Merge (RGM), a checkpoint-only method for merging specialised language models that share a common base. It represents expert-induced changes using read-side and write-side weight Gram matrices, aggregates these changes at three residual-stream states within each transformer layer, and solves self-retention equations to obtain merge coefficients. A cross-pressure diagnostic selects between raw and norm-balanced coefficient estimation before downstream evaluation. The coefficients are then applied to the original task-vector updates.

Experiments cover three LLM merging settings and an additional eight-task vision setting. RGM achieves the highest reported aggregate score among the compared merging methods in each LLM setting. The main proposed contribution is using residual-state Gram geometry to determine merge coefficients.

Critical strengths
  1. The paper addresses a relevant practical problem. Consolidating independently trained experts into one checkpoint without additional training data or inference-time routing is useful for deploying specialised capabilities. RGM preserves the original model architecture and inference procedure.

  2. The method has a clear architectural motivation. Projections that read from or write to a shared residual state are considered jointly, and the read/write Gram constructions place their effects in compatible dimensions. The coefficient equations and the checkpoint-based branch-selection rule are explicitly specified.

  3. The experiments examine complementary expert capabilities and overlapping capabilities within the same domain. The OpenMath result is encouraging, improving the aggregate score from 66.34 for ISO-Merger to 67.07 and exceeding both individual experts. Table 2 provides a useful ablation separating the unit of coefficient estimation from the representation of expert effects. Appendix D also checks agreement between the solver’s surrogate quantities and the realised weight-space merge.

Critical weaknesses
  1. The experiments do not sufficiently establish the benefit of the proposed geometry over simpler coefficient choices.

Table 2 is a useful ablation, but it is restricted to the low-overlap 7B setting. The full method achieves 63.95, compared with 63.72 for both PM-Delta and Node-Delta. Corresponding ablations are not reported for the norm-balanced branch used in the two 1.5B settings.

A particularly relevant comparison is missing in those two-expert settings. With unit-normalised expert sections, ignoring the small normalisation epsilon, Equation 23 gives the same coefficient to both experts at a given state. The resulting merge therefore applies a state-dependent scale to the sum of their task vectors. Tables 4 and 5 do not include uniform averaging or globally rescaled task-vector summation. These comparisons would help establish how much the proposed state-specific Gram signal contributes beyond controlling the overall update magnitude.

  1. The benefit of the branch-selection rule is not adequately demonstrated.

The cross-pressure threshold has an explicit motivation from the raw self-retention equation, and the authors state that it is applied before evaluation. However, only the DeepSeek setting directly compares Raw RGM with NormBalanced-RGM. The Qwen setting does not report the normalised branch, and the OpenMath setting does not report the raw branch.

Consequently, the experiments show that normalisation helps in one setting, but do not establish that the proposed selection rule improves over simply using the same branch throughout. This matters because selecting when to preserve or remove magnitude information is presented as a central contribution.

  1. The demonstrated advantage over established merging methods remains limited.

The gains over the strongest competing methods are 0.05 points in the Qwen setting and 0.03 points in the DeepSeek setting. These results support competitiveness, but provide limited evidence of a practically substantial improvement.

The larger OpenMath gain is more encouraging, although Table 5 compares against only TA-sum and ISO-Merger. RAM, TIES and TSVM, which appear in the other LLM experiments, are absent from this comparison. The additional vision experiment improves over Task Arithmetic, but includes no other merging baseline. Taken together, the results leave the broader significance of the proposed construction insufficiently established.

Questions for authors
  1. In the two 1.5B settings, how does NormBalanced-RGM compare with uniform averaging and globally rescaled task-vector summation? For two unit-normalised sections, Equation 23 appears to give equal coefficients to both experts at each state, making a comparison between the proposed state-dependent scales and a single global scale particularly informative. Please clarify how any comparison coefficients are selected.

  2. Can the representation and grouping ablations from Table 2 be evaluated in at least one of the 1.5B settings using norm-balanced coefficient estimation? This would help determine whether residual-state Gram aggregation remains beneficial in the regime where the normalised branch is used.

  3. Can you report both Raw RGM and NormBalanced-RGM for all three LLM settings? In particular, does Raw RGM outperform NormBalanced-RGM on Qwen, and does NormBalanced-RGM outperform Raw RGM on OpenMath? These comparisons would directly test the usefulness of the proposed branch-selection rule.

  4. How does RGM compare with RAM, TIES and TSVM in the OpenMath setting? Since this setting provides the clearest improvement over the reported competitors, completing this comparison would materially affect my assessment of the empirical contribution.

8 comments

We thank the reviewer for the detailed and constructive feedback, which helped us identify important ways to strengthen the empirical evaluation and clarify the scope of our claims.

1. NormBalanced-RGM versus global task-vector scaling. We evaluated this comparison in the DeepSeek-R1-Distill-Qwen-1.5B setting. For two experts, uniform averaging is the special case α=0.5\alpha=0.5 of a globally scaled task-vector sum, θ0+α(Δ1+Δ2)\theta_0+\alpha(\Delta_1+\Delta_2). We tested the fixed global scales α∈{0.25,0.5,0.75,1.0,1.25}\alpha\in\{0.25,0.5,0.75,1.0,1.25\}, applying each coefficient uniformly to the summed update. Since no separate validation set was available, we report the full sweep rather than selecting a coefficient based on evaluation performance.

Merge ruleLB ACCLB UTLCB v5 ACCLCB v5 UTOverall Avg.
Domain Expert22.4 ±\pm 0.1225.0 ±\pm 0.1325.2 ±\pm 0.0840.9 ±\pm 0.1643.59 ±\pm 0.16
Global scale (α=0.25\alpha=0.25)21.2 ±\pm 0.2423.8 ±\pm 0.3122.5 ±\pm 0.1437.8 ±\pm 0.2241.09 ±\pm 0.24
TA-avg (α=0.5\alpha=0.5)26.2 ±\pm 0.1929.4 ±\pm 0.2525.4 ±\pm 0.1143.8 ±\pm 0.1844.87 ±\pm 0.15
Global scale (α=0.75\alpha=0.75)27.1 ±\pm 0.2830.4 ±\pm 0.3424.6 ±\pm 0.1841.9 ±\pm 0.2544.96 ±\pm 0.19
TA-sum (α=1.0\alpha=1.0)20.9 ±\pm 0.1623.5 ±\pm 0.2817.3 ±\pm 0.2630.3 ±\pm 0.3240.20 ±\pm 0.35
Global scale (α=1.25\alpha=1.25)18.2 ±\pm 0.3520.6 ±\pm 0.4213.8 ±\pm 0.2924.5 ±\pm 0.3837.50 ±\pm 0.38
NormBalanced-RGM26.6 ±\pm 0.0729.7 ±\pm 0.1426.2 ±\pm 0.0944.7 ±\pm 0.0645.48 ±\pm 0.07
Merge ruleAIME 2024 (Avg@32)AIME 2025 (Avg@32)AMC 2023 (Avg@8)Minerva (Avg@4)OlympiadBench (Avg@4)Overall Avg.
Domain Expert51.2 ±\pm 1.1937.1 ±\pm 0.5290.7 ±\pm 0.7841.9 ±\pm 0.6158.0 ±\pm 0.0343.59 ±\pm 0.16
Global scale (α=0.25\alpha=0.25)46.8 ±\pm 0.6832.5 ±\pm 0.5588.2 ±\pm 0.7240.5 ±\pm 0.4256.5 ±\pm 0.3541.09 ±\pm 0.24
TA-avg (α=0.5\alpha=0.5)52.3 ±\pm 0.5235.8 ±\pm 0.4890.2 ±\pm 0.5442.6 ±\pm 0.3158.1 ±\pm 0.2644.87 ±\pm 0.15
Global scale (α=0.75\alpha=0.75)53.1 ±\pm 0.6135.9 ±\pm 0.5890.4 ±\pm 0.6342.9 ±\pm 0.3858.2 ±\pm 0.3244.96 ±\pm 0.19
TA-sum (α=1.0\alpha=1.0)48.4 ±\pm 0.5434.0 ±\pm 0.6288.3 ±\pm 0.8241.2 ±\pm 0.4957.9 ±\pm 0.8140.20 ±\pm 0.35
Global scale (α=1.25\alpha=1.25)45.2 ±\pm 0.8532.1 ±\pm 0.7486.4 ±\pm 0.9539.8 ±\pm 0.5656.9 ±\pm 0.6437.50 ±\pm 0.38
NormBalanced-RGM53.1 ±\pm 0.5336.3 ±\pm 0.7690.9 ±\pm 0.2843.4 ±\pm 0.1358.3 ±\pm 0.1645.48 ±\pm 0.07

NormBalanced-RGM achieves the highest overall average among these comparisons. The best globally scaled sum is α=0.75\alpha=0.75 (44.96), 0.52 points below NormBalanced-RGM (45.48). This comparison supports the value of state-dependent scaling beyond adjusting the overall magnitude of the summed task vector.

For a direct view of RGM's state-dependent scales, we have added Appendix E.2, which plots the solved coefficients across all 84 residual nodes in each of the three settings. In the two NormBalanced settings, both experts share the same coefficient at each residual node. These coefficients vary across nodes, whereas a globally scaled task-vector sum uses a single scale throughout.

2. NormBalanced-branch ablations. We evaluated the same three representation/grouping ablations with NormBalanced coefficient estimation on DeepSeek-R1-Distill-Qwen-1.5B. We report mean ±\pm std:

MethodLB ACCLB UTLCB v5 ACCLCB v5 UTOverall Avg.
PM-Delta22.65 ±\pm 0.3125.40 ±\pm 0.3619.35 ±\pm 0.2433.40 ±\pm 0.3041.51 ±\pm 0.28
PM-Gram26.35 ±\pm 0.1829.45 ±\pm 0.2425.45 ±\pm 0.1443.65 ±\pm 0.1945.01 ±\pm 0.14
Node-Delta23.10 ±\pm 0.2525.95 ±\pm 0.3019.85 ±\pm 0.2034.25 ±\pm 0.2641.89 ±\pm 0.24
NormBalanced-RGM26.60 ±\pm 0.0729.74 ±\pm 0.1426.20 ±\pm 0.0944.74 ±\pm 0.0645.48 ±\pm 0.07
MethodAIME 2024 (Avg@32)AIME 2025 (Avg@32)AMC 2023 (Avg@8)Minerva (Avg@4)OlympiadBench (Avg@4)Overall Avg.
PM-Delta49.69 ±\pm 0.6834.58 ±\pm 0.6488.96 ±\pm 0.7841.64 ±\pm 0.4657.93 ±\pm 0.5841.51 ±\pm 0.28
PM-Gram52.75 ±\pm 0.5835.95 ±\pm 0.5490.35 ±\pm 0.5242.95 ±\pm 0.3158.15 ±\pm 0.2845.01 ±\pm 0.14
Node-Delta50.10 ±\pm 0.6234.79 ±\pm 0.5989.17 ±\pm 0.7341.82 ±\pm 0.4258.00 ±\pm 0.5141.89 ±\pm 0.24
NormBalanced-RGM53.12 ±\pm 0.5336.32 ±\pm 0.7690.94 ±\pm 0.2843.38 ±\pm 0.1358.25 ±\pm 0.1645.48 ±\pm 0.07

NormBalanced-RGM has the highest observed mean on all nine benchmarks. It exceeds PM-Gram, the strongest ablation by overall average, by 0.47 points; the gains over PM-Delta and Node-Delta are 3.97 and 3.59 points, respectively. These margins are larger than those in our Qwen2.5-7B-Instruct ablation, where RGM's observed overall average exceeded the three variants by 0.23–0.34 points. This is a cross-setting comparison between different coefficient-estimation branches, but it shows that the observed advantage over the ablations is more pronounced in this setting.

The coefficient diagnostics give a direct explanation for this pattern. With raw task-vector deltas, the mean pairwise overlap is only 0.096 for PM-Delta and 0.091 for Node-Delta, and their solved coefficients average 0.912 and 0.916 (ranges 0.861--0.952 and 0.898--0.937). The raw-delta representation therefore sees little shared effect and keeps coefficients close to one; changing the solving unit alone yields only a 0.38-point gain (Node-Delta over PM-Delta). Replacing raw deltas by Gram shifts changes the measured overlap: PM-Gram has mean overlap 0.348 and mean coefficient 0.749, consistent with its 3.50-point gain over PM-Delta.

The full RGM further aggregates all projection matrices incident to each residual node. If Sv,e,iS_{v,e,i} denotes the Gram shift of expert ii on edge ee at node vv, the node representation mv,i=∑e∈E(v)Sv,e,im_{v,i}=\sum_{e\in E(v)}S_{v,e,i} gives Γv[i,j]=⟨mv,i,mv,j⟩F\Gamma_v[i,j]=\langle m_{v,i},m_{v,j}\rangle_F. Expanding this inner product adds the cross-projection terms ∑e≠e′⟨Sv,e,i,Sv,e′,j⟩F\sum_{e\ne e'}\langle S_{v,e,i},S_{v,e',j}\rangle_F, which are absent from PM-Gram. In this setting, their contribution is positive: the mean overlap rises from 0.348 for PM-Gram to 0.455 for full RGM, while the mean coefficient falls from 0.749 to 0.693. For the symmetric two-expert system, cv≈1/(1+ρv)c_v\approx 1/(1+\rho_v), so this increase in overlap directly produces the lower coefficients. Thus PM-Gram captures the main gain from changing the representation, while residual-state aggregation contributes the additional 0.47 points.

3. Raw versus NormBalanced RGM. We have now evaluated both coefficient-estimation branches in all three LLM settings. The Qwen2.5-7B-Instruct NormBalanced-RGM result and the OpenMath-Nemotron-1.5B Raw RGM result are newly evaluated; we have added both rows to the corresponding main-result tables (Tables 1 and 5). Both branches were already reported for the DeepSeek setting in Table 4. The resulting overall averages are:

SettingRaw RGMNormBalanced-RGMBranch selected by the pre-merge rule
Qwen2.5-7B-Instruct63.95 ±\pm 0.1062.44 ±\pm 0.79Raw
DeepSeek-R1-Distill-Qwen-1.5B43.93 ±\pm 0.2845.48 ±\pm 0.07NormBalanced
OpenMath-Nemotron-1.5B65.98 ±\pm 0.1767.07 ±\pm 0.06NormBalanced

Raw RGM is stronger in the low-overlap Qwen setting, whereas NormBalanced-RGM is stronger in both high-overlap 1.5B settings. Thus, the branch-selection rule selects the higher-scoring branch in all three settings: Raw RGM for Qwen and NormBalanced-RGM for DeepSeek and OpenMath. These results support selecting between the two branches from the pre-merge cross-pressure diagnostic rather than using either branch unconditionally.

We also note an implementation detail relevant to reproducing the two 1.5B Raw-RGM rows, now stated explicitly in Appendix A.1. These are forced Raw-branch comparisons, rather than the branch selected by the pre-merge rule. The unconstrained raw branch solve produced negative coefficients; without post-solve clipping, the resulting forced Raw merges degenerated and no longer produced normal outputs. We therefore clipped negative solved coefficients to zero before applying them to the corresponding task-vector updates. This safeguard is used only for these auxiliary branch comparisons.

4. OpenMath comparison with RAM and TIES. We additionally evaluated RAM and TIES on OpenMath-Nemotron-1.5B. NormBalanced-RGM achieves an overall average of 67.07, compared with 66.40 for RAM and 66.22 for TIES. The full results are:

MethodAIME 2024AIME 2025AMC 2023MinervaOlympiadBenchOverall Avg.
RAM71.35 ±\pm 0.5261.20 ±\pm 0.4795.47 ±\pm 0.6232.26 ±\pm 0.6471.74 ±\pm 0.2266.40 ±\pm 0.16
TIES70.83 ±\pm 0.6861.15 ±\pm 0.4295.31 ±\pm 0.7832.26 ±\pm 0.8571.56 ±\pm 0.3166.22 ±\pm 0.25
NormBalanced-RGM72.50 ±\pm 0.4261.61 ±\pm 0.3695.78 ±\pm 0.7832.81 ±\pm 0.0972.65 ±\pm 0.3967.07 ±\pm 0.06

NormBalanced-RGM has the highest observed mean on all five benchmarks and exceeds RAM and TIES by 0.67 and 0.85 points in overall average, respectively.

5. Additional ViT-B/32 baselines. The reviewer also noted that the ViT-B/32 experiment compared RGM only with Task Arithmetic. We additionally evaluated DARE and TIES under the same eight-task protocol. The benchmark columns below follow ascending fine-tuned accuracy, from SUN397 (75.29) to EuroSAT (99.89):

MethodSUN397CarsDTDRESISC45SVHNGTSRBMNISTEuroSATOverall Avg.
Finetuned expert75.2977.6879.4196.1197.4698.7399.6999.8990.53
DARE65.2863.4550.5371.7863.7952.3587.4773.4466.01
TIES59.1158.5654.7970.9886.1570.8498.3681.6172.55
NormBalanced-RGM64.1963.0652.9872.3575.2863.5294.4082.5771.04

RGM improves over DARE by 5.03 points in the eight-task average, while TIES has a 1.51-point higher average than RGM in this setting. On SUN397 and Cars, the two benchmarks with the lowest fine-tuned expert accuracies, RGM is higher than TIES by 5.08 and 4.50 points, respectively. TIES is higher on DTD and gains larger margins on SVHN and GTSRB, resulting in its higher overall average.

Reviewer_kbHvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses the problem of merging models and propose Residual Geometry Merge, that preserves residual stream geometry induced by expert checkpoint. Experiments are conducted across three base LLMs merging settings, and results reported in the paper suggest improvements over existing methods in several scenarios.

Critical strengths
  • The approach proposed by the paper seems to be new and performant
  • The paper evaluate the approach across three different LLM merging settings
  • The ablation study is conducted the the results are reported
  • The related work section is well presented, easy to read and to understand
  • The proposed approach appears to surpass existing methods in several scenarios
Critical weaknesses
  • Insufficient supporting examples in the paper, which sometimes the paper difficult to understand. Among examples, the claim that "Existing shared-base merging methods primarily decide how to combine expert task vectors in parameter space." needs illustrative examples
  • Many terminologies that are important to understand the paper are either not defined, or not defined before the first use. For instance, "RL", "heterogeneous merge of coding".
Questions for authors
  • What precisely does "heterogeneous merge of coding" mean, and how does it differ from the other merging regimes evaluated?
1 comment

We thank the reviewer for the encouraging assessment and for pointing out that our discussion of related work was too abstract. We have added Appendix A.2, which gives concrete descriptions of every baseline used in our comparisons. For example, TA-avg and TA-sum average or sum corresponding task-vector coordinates; RAM treats inactive, expert-specific, and shared coordinates differently; and TIES trims small coordinates before resolving sign conflicts. These examples make concrete what we mean by combining expert updates in parameter space, while also distinguishing coordinate-wise rules from subspace-based methods. RGM also constructs the merged checkpoint by combining expert task vectors, but its coefficients are not determined independently at individual parameter coordinates. Its coefficients are estimated from expert-induced Gram shifts aggregated at residual-stream states.

We have also clarified the terminology at first use. We now write “reinforcement learning (RL)” and use RL experts to mean checkpoints post-trained with reinforcement learning. In the Qwen2.5-7B setting, “heterogeneous” refers to differences in expert capabilities, not model architectures: the three descendants of the same base model specialize in coding, tool use, and long-context memory. By contrast, the OpenMath-Nemotron-1.5B setting merges two mathematics experts in a homogeneous, same-domain regime. We now write “experts with heterogeneous capabilities” when first introducing the 7B setting.

These additions address the presentation and terminology issues raised by the reviewer and do not require additional experiments.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision uploaded

We thank all reviewers for their constructive feedback. In response, we have updated the manuscript to clarify the method and its comparisons. The main changes are summarized below:

  1. Appendix A.2 (Baseline Details): added concrete descriptions and mathematical definitions of all merge baselines used in our comparisons, including TA-avg, TA-sum, RAM, TIES, TSVM, and ISO-Merger, together with their implementation settings.
  2. Additional branch comparisons in Tables 1 and 5: added NormBalanced-RGM results for the Qwen2.5-7B-Instruct setting and Raw-RGM results for the OpenMath-Nemotron-1.5B setting, providing direct comparisons with the branch selected by the pre-merge diagnostic.
  3. Additional OpenMath baselines in Table 5: added newly evaluated RAM and TIES results for the OpenMath-Nemotron-1.5B setting, strengthening the comparison against established parameter-space merge baselines.
  4. Appendix E.2 (Solved Coefficients Across Layers): added coefficient curves for all 84 residual nodes in each of the three merging settings, making the state-wise coefficient patterns directly inspectable.
  5. Merge rule outside residual nodes: clarified that parameter tensors not assigned to residual-node projections, including token embeddings, the LM head, normalization parameters, and biases, are merged by uniform arithmetic averaging across experts.
  6. Terminology and result wording: clarified that “heterogeneous” refers to differences in expert capabilities, and revised claims such as “best” to “highest observed overall average” where appropriate.
  7. Forced Raw-RGM branch-comparison detail: added an implementation note in Appendix A.1 clarifying that, in the two 1.5B settings, the unconstrained raw branch solve produced negative coefficients. Without post-solve clipping, the resulting forced Raw merges degenerated and no longer produced normal outputs. We therefore clipped negative solved coefficients to zero before applying them to the corresponding task-vector updates. This safeguard is used only for these auxiliary branch comparisons.

Please let us know if there are any additional comments or suggestions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.