- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper investigates how two fixed task vectors change a 2-layer GELU base model, when applied together. The independent application of the two task vectors is compared to applying both vectors to the network at the same time. The difference between these is what the repair aims to remove. The paper considers four cases, which determine how much of the difference can be removed. Experiments on synthetic datasets validate the predicted errors.
Critical strengths
The paper considers a similar task-arithmetic problem to Ortiz-Jiménez et al., 2023, but moves beyond linearization by studying GELU networks. The derived error cases could be used to better apply multiple task vectors to a base model.
Critical weaknesses
The paper is very hard to read and understand. Starting from (but not exclusive to) the introduction concepts are introduced, without defining them and no explanation of the meaning of the theorems is given in the paper. The notation is also not introduced and often ambiguous. A non-exhaustive list of undefined (or only later defined) terms and ambiguous notation includes:
- All terms in equation 1
- The terms "population fidelity", "repair", "nondegenerate", "fixed hidden biases", "coefficient obstruction", "concatenated adapter budget", "affine ridge classes", "opposite-ridge symmetries", "effective weight matrices", "SVD-tail distance"
- ambiguous notation includes: , , , ,
The motivation of the problem and its relation to related work like Ortiz-Jiménez et al., 2023 is not properly discussed.
The experiments are limited to a verification of the error terms and there is no connection to actual applications, like merging or low-rank adaptation. This limits the practical usefulness and there is a lack of motivation for the theory. An explicit repair is given (Theorem 3), but it only applies under condition , so its practical applicability is unclear.
Questions for authors
- Could you explain the different cases more intuitively?
- What consequences does your theory imply for practical use-cases?
- Why do the task directions depend on in Theorem 5, when they are fixed in Equation 1 and Theorem 1?
- Could you measure how far typical taks updates (like LoRA weights) deviate from your assumptions, like in realistic settings?