acceptodds
Under review as a conference paper at ICLR 2027

Repairing Nonlinear Model Merges: A Rank–Width Separation for GELU Networks

Abstract

Parameter addition need not add the functions implemented by task updates. We characterize the capacity of every local repair for nondegenerate two-layer exact-GELU networks with fixed hidden biases. The optimal population error has exactly four regimes: first order when final adapter rank cannot contain the summed first-order update; second order when an explicit neuronwise compatibility condition fails; third order when compatibility holds but exact additivity fails; or zero. A unique quadratic repair rescales rows and preserves the actual summed-update ranks, so additional rank cannot improve these error orders. A quantitative extension characterizes near-compatibility: uniformly over bounded direction families, third-order local accuracy is possible exactly when a coefficient obstruction is of the order of the update scale, and a correction within the concatenated adapter budget achieves it. Globally, we determine the minimum exact hidden width. The exact-additivity condition is also necessary and sufficient for any same-width representation, even at distant parameters and with an affine output branch. Rank-one tasks can instead require width 3m. A calibration interpolation threshold and reproducible synthetic experiments distinguish population expressivity from finite-sample fitting and task preservation.

Primary area:
learning theory
Keywords:
model merging, GELU networks, low-rank adaptation, expressivity, nonlinear repair

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_72gpOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates how two fixed task vectors change a 2-layer GELU base model, when applied together. The independent application of the two task vectors is compared to applying both vectors to the network at the same time. The difference between these is what the repair aims to remove. The paper considers four cases, which determine how much of the difference can be removed. Experiments on synthetic datasets validate the predicted errors.

Critical strengths

The paper considers a similar task-arithmetic problem to Ortiz-Jiménez et al., 2023, but moves beyond linearization by studying GELU networks. The derived error cases could be used to better apply multiple task vectors to a base model.

Critical weaknesses

The paper is very hard to read and understand. Starting from (but not exclusive to) the introduction concepts are introduced, without defining them and no explanation of the meaning of the theorems is given in the paper. The notation is also not introduced and often ambiguous. A non-exhaustive list of undefined (or only later defined) terms and ambiguous notation includes:

  • All terms in equation 1
  • The terms "population fidelity", "repair", "nondegenerate", "fixed hidden biases", "coefficient obstruction", "concatenated adapter budget", "affine ridge classes", "opposite-ridge symmetries", "effective weight matrices", "SVD-tail distance"
  • ambiguous notation includes: μ\mu, β\beta, α\alpha, η\eta, δ\delta

The motivation of the problem and its relation to related work like Ortiz-Jiménez et al., 2023 is not properly discussed.

The experiments are limited to a verification of the error terms and there is no connection to actual applications, like merging or low-rank adaptation. This limits the practical usefulness and there is a lack of motivation for the theory. An explicit repair is given (Theorem 3), but it only applies under condition S2S_2, so its practical applicability is unclear.

Questions for authors
  • Could you explain the different cases more intuitively?
  • What consequences does your theory imply for practical use-cases?
  • Why do the task directions depend on ϵ\epsilon in Theorem 5, when they are fixed in Equation 1 and Theorem 1?
  • Could you measure how far typical taks updates (like LoRA weights) deviate from your assumptions, like S2S_2 in realistic settings?
Reviewer_pt1vOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Task arithmetic does arithmetic on model weights to obtain a new model whose functioning is approximately the arithmetic on model outputs. Since models are intrinsically non-linear, arithmetic on linear model weights then leads to errors in the desired outputs. The paper studies in detail the error in a two-layer GELU model, identifying several high-order terms and the hidden width needed to correct the error given the rank of weight modification.

Critical strengths
  1. The paper's mathematical statements are rigorous and consistent.
Critical weaknesses
  1. The paper is poorly written. The scientific question is not well motivated. A lot of terminology, variables, and names are not explained or defined when they first show up. The abstract is not readable. The main text piles up mathematical assumptions and statements with little interpretation. I have difficulty summarizing takeaways or conclusions.
  2. The whole paper focuses on a specific two-layer GELU model. It is fine for rigorous theory to be developed on a clear and simple setup. But we need to understand what the solution from this simple model can possibly imply for more realistic models. Which mathematical statements are strictly true for GELU, which can be generalized, and which can be conceptually generalized, etc.?

Due to the weaknesses in clarity and relevance, I cannot suggest acceptance.

Questions for authors
  1. Why use the GELU model? What do you expect to change/stay if other nonlinearities are used?
  2. What practical tips do the results suggest for merging LLMs?
  3. Is your paper focusing on two things: expanding the error and how to further perturb the merged weight via LoRA to reduce the error?
Reviewer_BXwuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies Task Arithmetic(TA) in a 2-layer GELU network. It identifies the problem of TA: the addition of a parameter will not add a function, and thus requires "repairing". The paper attempts to answer how close any parameter can get to the functional sum gϵ=fa+fb−fbaseg_{\epsilon} = f_{a}+f_b - f_{base}. For the proposed setting, it proves that the repair can improve the error from order ε² to ε³ when there is no neuron interaction between the first task and the second task update.

Critical strengths
  1. The paper identifies the problem with Task Arithmetic's parameter space update and how it is not equivalent to the addition of functions.
  2. The paper provides code, a dataset, and seeds for reproducibility.
Critical weaknesses
  1. The paper never defines model merging, task vectors, or whether it is fine-tuned weights minus pretrained weights. Motivation for why anyone would want to merge. The introduction and abstract need to be rewritten to better introduce the topic.
  2. The paper defines ϵ\epsilon, and the results on how the error can be reduced as it tends to zero. But in practice, ϵ\epsilon is never close to zero; its value close to zero actually means we move to the base model and learn nothing about the task. The practical relevance of the small task update scaling factor is not shown.
  3. There is no related work section in the paper; there is only a paragraph about the relationship to existing work.
  4. Why are biases fixed, and why is there no residual connection? The paper states these assumptions, but doesn't justify them.
  5. There is no comparison in terms of experiments for real-world merge datasets or scenarios.
Questions for authors

See weaknesses.

Reviewer_y7yfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper concerns model merging, specifically the relationship between weight addition and functional addition -- i.e. the relationship between 1) the result of adding two parameter updates to the same model; and 2) the result of adding the two functions that result from applying both weight updates separately. Using a one-hidden-layer GELU network, the main contribution of the paper is a taxonomy of four possible scenarios of model merging, split by achievable error and conditions on the

Critical strengths

The taxonomy of model merge scenarios seems useful. Use of GELU activation in the proofs is interesting and can probably be leveraged in other theoretical work.

Critical weaknesses
  1. The paper is really difficult to follow. Symbols are almost never defined at first use, context is missing, and key foundations of the paper are very poorly defined (e.g. the paper says "The tasks are a = (P, Q) and b = (R, S)" but this isn't explained anywhere; and the four entries in Table 1 are similarly incomprehensible at first read).
  2. Discussion of related work is thin. Reference list is much shorter than one would expect of an ICLR paper, and discussion of related work is limited to just a single 12-line paragraph.
  3. Experimental evidence is thin. Even for a theory paper, one would expect a more thorough validation, numerical exploration of other activation functions, larger models, other tasks, etc.

Overall this paper has an interesting idea that is worth publishing. But it requires a far more careful rewrite to make it human-friendly, as well as stronger numerical and mathematical evidence.

Questions for authors
  1. Do I understand correctly this is largely a negative result? If I understand correctly, even rank-one updates violate the conditions for the repairable regime. If so, I'd probably suggest reframing the paper about the impossibility of accurate merges.
  2. Can you provide an example of an actual MLP fine-tuned on two non-trivial tasks merged with this procedure? The paper's comments on S2S_2 suggest this set is effectively never occupied, which would suggest this kind of setup effectively forbids merging.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.