Residual Geometry Merge: State-Wise Merging of Expert Language Models

Model merging combines specialized checkpoints into a single model with multiple capabilities. For large language models, this offers a practical way to integrate expert checkpoints that share a common base without additional training or routing. Existing methods typically combine expert updates directly or transform them before recombination. However, a transformer block is not merely a collection of independent matrices: its projections jointly read from and write to shared residual-stream states. We propose Residual Geometry Merge (RGM), an architecture-aware model merging framework that preserves residual-stream geometry induced by expert checkpoints. Starting from the Transformer Circuits view, RGM uses weight Gram shifts to summarize how each expert changes directions at residual-stream states, aggregates incident shifts at each state, and solves state-wise self-retention equations to derive merge coefficients. The method is coefficient-adaptive and distinct from uniform task-vector addition, routing, and subspace-based merging. Experiments across three shared-base LLM merging settings—Qwen2.5-7B-Instruct with coding, tool-use, and memory experts; DeepSeek-R1-Distill-Qwen-1.5B with coding and mathematics experts; and OpenMath-Nemotron-1.5B with two mathematics experts—show that RGM achieves the strongest overall performance in all three settings.