- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes to regularize LoRA adapter using relations from zigzag algebra. Specifically, they split the LoRA rank dimension into groups as vertices of a chain graph, where arrows between neighbouring groups are the uncentered cross-covariance blocks of their LoRA down projection outputs over the batch. They motivate this as giving the parameter space a graded, composiitonal relation through training. The zigzag relations are softly enforced in loss as (1) non-returning length-2 paths vanish, (2) the two return cycles at middle agree, and (3) length-3 paths vanish (plus locality term that prevents collapse). Empirically, they finetune Qwen2.5-7B-Instruct on MMLU-auxiliary data and compare against LoRA/LoRA-MoE methods on multiple tasks, plus ablations.
Critical strengths
1. The paper is generally clear and easy to follow. The loss terms for are given explcitly, and the construction is descibed clearly.
Critical weaknesses
1. The zigzag framing is neither fully motivated or realized.
- a) Why these relations? The zigzag relations come from representation theory, where they were chosen for algebraic properties relevant there. But, the paper does not explain why these particular algberaic relations suit LoRA rank grouping in particular: why the middle group's two round trips should be equal, why non-returning two-step relations should vanish, or why the graph should be a chain at all. The only motivation I could spot was the analogy to graded human knowledge, which is not algebraically connected to cross-covariance of LoRA down-projection outputs.
- b) The relations are only realizable at zero. With and , forces equal rnages, while forces orthogonal ranges. So as Sec 3.3 and 4.3 mention, this implies . Thus, the zigzag terms vanish exactly when the middle group has zero cross-moments with its neighbors, which seems to be in tension with the paper's claim that zigzag sturcture keeps related groups closely connected (e.g., in abstract).
- c) Other claims do not seem fully supported. The abstract states that relations are inferred from training signals and that graded structure is learned from training dynamics, but the graph and the relations imposed on it are pre-determined (only the arrow values depend on data), and the groups are arbitrarily fixed slices of the rank dimension. So, it remains unclear to me what relational structure is learned beyond the cross-moment values on this fixed graph.
- d) There seem to be directly relevant prior works such as [1] that regularizes cross-covariance of activations, or [2] that applies local decorrelation. This also limits the conceptual novely of the paper.
[1] Cogswell, M., Ahmed, F., Girshick, R., Zitnick, L., & Batra, D. (2015). Reducing overfitting in deep networks by decorrelating representations. ICLR 2016.
[2] Rodríguez, P., Gonzalez, J., Cucurull, G., Gonfaus, J. M., & Roca, X. (2016). Regularizing cnns with locally constrained decorrelations. ICLR 2017.
2. The experiments do not isolate the claims.
The orthogonality baseline (Table 6) does not seem controlled, as it penalizes all group pairs rather than graph edges, and its penalty form and whether locality terms are retained are not specified. In addition, the gain seems to come from the down-projection side (Table 5). Together with A-only and A+B performing similarly, while B-only gives no improvement (Table 5), this seems to leave open an alternative plausible explanation: the penalty may shrink the down-projection outputs, which changes the effective balance between the two LoRA factors [3]. Several baselines also fall below plain LoRA, and all methods seem to share an optimization configuration. Because of these, the gains cannot be attributed to relational structure. The experimental scope is also limited to one backbone, fine-tuned on one dataset, with MCQA evaluations alone. This leaves the generality of the proposed framework across models and adaptation settings untested.
[3] Hayou, S., Ghosh, N., & Yu, B. (2024). Lora+: Efficient low rank adaptation of large models. ICML 2024.
Questions for authors
Q1. Can the authors please clarify (1a) in Weaknesses?
Q2. Can the authors clarify (1b-c) in Weaknesses? In particular, what structure is the regularizer intended to induce algebraically beyond driving the middle group towards zero cross-covariance with the two neighbors? What makes zigzag relations a suitable inductive bias for LoRA rank groups?
Q3. For the K=6 construction and the orthogonal baselines, I cannot seem to find the exact loss and penalty weights in the text. Can the authors specify these? Also, the default config (rank 24, 3 groups) is reported multiple times across Table 1, Table 5, and Sec 4.3 (as well as plain LoRA) with different numbers. Which runs do these correspond to?