- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Residual Geometry Merge, a data-free method for merging expert LLMs that share a base model. It summarizes each expert's effect at every residual-stream state by weight Gram shifts and solves a small linear system per state to obtain merge coefficients, switching to a norm-normalized variant when experts overlap strongly. Across three LLM merging settings, RGM obtains the best average score.
Critical strengths
Simple and data-free.
Critical weaknesses
-
The improvements are within noise. In Table 1, RGM's 63.95±0.10 is not really better than RAM's 63.90±0.48 or ISO-Merger's 63.90±0.18.
-
The role of the core design is not demonstrated. In the three-expert 7B setting, RGM is 1.86 points above TA-sum, but PM-Delta, which drops both the residual-state unit and the Gram representation, is still 1.63 points above TA-sum and only 0.23 below RGM. So most of the gain can be obtained without the core construction.
Questions for authors
- Is there a significance test (e.g. a paired bootstrap over problems) showing that the advantage of RGM over RAM and ISO-Merger goes beyond evaluation noise?
- What does the distribution of the solved coefficients look like in the 7B setting, and how are parameters outside the three residual nodes (embeddings, LM head, norms, biases) merged? Given that PM-Delta already recovers most of the gain, what does the residual-state Gram construction add on top of it?
5 comments
2.3. What the residual-state Gram construction adds beyond PM-Delta. The direct answer is the matched-run comparison reported above: the complete RGM construction improves over PM-Delta by 0.1981, 0.2466, and 0.2518 points across the three runs, for a mean gain of 0.2322 points. Thus, although PM-Delta accounts for most of the improvement over TA-sum, the full residual-state Gram construction provides a small but consistent additional benefit. The existing ablation is consistent with this interpretation, with RGM reaching 63.95 compared with 63.72 for PM-Delta.
We would also like to give a toy example illustrating why Gram shifts can reveal overlap that raw task-vector inner products miss: Let be a unit residual direction, let be orthonormal output directions, and consider:
\Delta_1=\alpha e_1u^\top,\qquad \Delta_2=\alpha e_2u^\top,\quad \alpha>0.$$ The updates are orthogonal in parameter space, $\langle\Delta_1,\Delta_2\rangle_F=0$, but their input-side Gram shifts are identical: $$S_i=(W_0+\Delta_i)^\top(W_0+\Delta_i)-W_0^\top W_0 =(2\alpha+\alpha^2)uu^\top,\qquad i\in\{1,2\}.$$ Thus, both updates produce the same change in squared response along the residual direction $u$, despite having zero raw parameter overlap. The raw-delta system therefore assigns $c_1=c_2=1$, whereas the identical Gram shifts give $c_1=c_2=\tfrac{1}{2}$ under the normalized self-retention system. This example isolates the change from raw deltas to Gram shifts; the additional effect of state-wise aggregation is reflected in the ablation below. We have also evaluated these three ablations with NormBalanced coefficient estimation on DeepSeek-R1-Distill-Qwen-1.5B. Their overall averages (mean $\pm$ std) are: | Method | Overall Avg. | RGM gain | |---|---:|---:| | PM-Delta | 41.51 $\pm$ 0.28 | +3.97 | | PM-Gram | 45.01 $\pm$ 0.14 | +0.47 | | Node-Delta | 41.89 $\pm$ 0.24 | +3.59 | | NormBalanced-RGM | **45.48 $\pm$ 0.07** | -- | In this setting, NormBalanced-RGM has the highest observed mean on all nine benchmarks. Its 3.97-point gain over PM-Delta is substantially larger than the 0.2322-point gain in the Qwen 7B Raw-RGM setting, and it is also 0.47 points above PM-Gram, the strongest ablation. The coefficient diagnostics explain why this setting is a high-gain regime. PM-Delta and Node-Delta measure overlap between raw task-vector deltas; their mean overlaps are only 0.096 and 0.091, so their solved coefficients stay near 0.91. PM-Gram instead compares Gram shifts, raising the mean overlap to 0.348 and lowering the mean coefficient to 0.749; this representation change accounts for most of the 3.50-point gain over PM-Delta. Full RGM additionally sums the Gram shifts of all projections incident to a residual node. This introduces cross-projection terms in the node-level Gram matrix that PM-Gram cannot see. Its mean overlap reaches 0.455 and its mean coefficient falls to 0.693. In the symmetric two-expert system, $c_v\approx 1/(1+\rho_v)$, so these lower coefficients are the direct result of the stronger shared residual-state effect. The remaining 0.47-point gain over PM-Gram is therefore the contribution of state-wise aggregation on top of the Gram representation. The numerical effect of moving from PM-Gram to the full RGM actually follows from RGM's architecture-aware formulation: it jointly considers projections that read from or write to a shared residual state. Its coefficient-estimation unit therefore follows the Transformer's residual-stream computation graph rather than individual parameter entries or isolated tensors. Coefficients are shared according to how projections interact through the residual stream, whereas coordinate-wise and per-matrix rules do not encode these architectural relationships. The additional 0.47-point gain over PM-Gram is thus the empirical effect of this architecture-aware solving unit on top of the Gram-shift representation. We have revised the claims and discussion accordingly and report the matched comparisons above. We have also added the complete coefficient visualization in Appendix E.2 and clarified the merge rule for parameters outside the residual nodes in Section 3.We would appreciate your feedback on our response, particularly the additional NormBalanced ablations reported in part 2.3. On the DeepSeek-R1-Distill-Qwen-1.5B setting, RGM improves over PM-Delta by 3.97 points and PM-Gram by 0.47 points in overall average, providing additional evidence for the contribution of the residual-state Gram construction. Please let us know if there are remaining concerns we could clarify.
We thank the reviewer for raising the questions about evaluation noise, coefficient behavior, and the incremental contribution of the residual-state Gram construction. We agree that the absolute gain over a strong merge baseline should not be overstated. We therefore report matched-run statistics below and have revised the paper to distinguish an observed mean ranking from a statistically separable improvement.
There is also a structural reason to expect the gains between strong merge methods to be modest. The merge methods in the comparison use the same fixed expert checkpoints and shared base model. A merge rule can reweight and combine the information already present in those experts, but it does not create a new expert, add model capacity, or introduce an additional task-specific training signal. Once a strong baseline has captured most of the experts' complementary capabilities, a large additional gain should not be expected. The non-significant differences from RAM and ISO-Merger are consistent with this interpretation.
1. Matched-run significance analysis. For each method, we paired the overall-average score from each of the three runs with the corresponding RGM score and performed a paired t-test. The differences below are RGM minus the comparison method:
Thus, RGM is not statistically separable from RAM or ISO-Merger in these three matched runs. We have clarified the wording in the paper to describe RGM as having the highest observed overall average, without implying a statistically significant advantage over those methods. The Domain Expert reference is formed by aligning the CURE code results, ToolRL BFCL results, and MemAgent memory results by run.
2.1. Coefficient distribution. We have added Appendix E.2, which visualizes the solved coefficients in each setting. For the two-expert NormBalanced-RGM branch, the normalized self-retention system is symmetric under exchanging the experts, yielding a shared coefficient at each state. In the Qwen2.5-7B-Instruct setting, which uses the Raw-RGM branch, CURE and ToolRL coefficients are below one at most residual states, while MemAgent coefficients remain nearly constant at one; overall, the coefficients remain close to one with localized state-wise deviations. The complete coefficient ranges in the 7B setting are:
As discussed in Section 5.1, RGM recovers task-vector summation in the orthogonal limit and introduces state-wise coefficient corrections when residual-state Gram sections overlap. In the 7B setting, the expert-induced residual-state sections are nearly orthogonal on average, so most coefficients remain near one. Thus RGM remains close to TA-sum while responding to state-specific overlap, yielding 1.81--1.93-point improvements across the three matched runs.
2.2. Parameters outside the residual nodes. The parameters outside the three residual nodes are merged by uniform arithmetic averaging across the expert checkpoints:
=\theta_0+\frac{1}{K}\sum_i \Delta_i.$$ This includes token embeddings, the LM head, normalization parameters, and biases. Each non-node parameter tensor is averaged uniformly. We have made this rule explicit at the end of the Setup subsection in Section 3, after defining the three residual states and their incident projections.