acceptodds
Under review as a conference paper at ICLR 2027

Beyond Continuity Or Discontinuity: Zigzag Algebra for Compositional Relations in Low-Rank Adaptation

Abstract

Modern neural networks continuously update parameters in a shared space, while existing methods provide only limited forms of relational organization. We introduce a framework that partitions the trainable parameter space into groups, infers their relations from the model’s own training signals, and organizes them through zigzag algebra. Zigzag algebra is a path-based algebra in which vertices are connected by directional arrows and relations constrain how these arrows can compose. We map parameter groups to vertices and their interactions to arrows, allowing strongly related groups to remain closely connected, weakly related groups to interact less, and unrelated groups to become effectively separated. Unlike orthogonality, expert routing, or heuristically assigned group labels, our approach learns this graded relational structure directly from training dynamics while using algebraic paths to constrain higher-order interactions. We instantiate the framework within LoRA by partitioning the low-rank space and imposing these algebraic relations during optimization, without additional inference-time parameters, routing modules, or architectural branches.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, LoRA, Abstract algebra

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JA79OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes to regularize LoRA adapter using relations from zigzag algebra. Specifically, they split the LoRA rank dimension into groups as vertices of a chain graph, where arrows between neighbouring groups are the uncentered cross-covariance blocks of their LoRA down projection outputs over the batch. They motivate this as giving the parameter space a graded, composiitonal relation through training. The zigzag relations are softly enforced in loss as (1) non-returning length-2 paths vanish, (2) the two return cycles at middle agree, and (3) length-3 paths vanish (plus locality term that prevents collapse). Empirically, they finetune Qwen2.5-7B-Instruct on MMLU-auxiliary data and compare against LoRA/LoRA-MoE methods on multiple tasks, plus ablations.

Critical strengths

1. The paper is generally clear and easy to follow. The loss terms for K=3K=3 are given explcitly, and the construction is descibed clearly.

Critical weaknesses

1. The zigzag framing is neither fully motivated or realized.

  • a) Why these relations? The zigzag relations come from representation theory, where they were chosen for algebraic properties relevant there. But, the paper does not explain why these particular algberaic relations suit LoRA rank grouping in particular: why the middle group's two round trips should be equal, why non-returning two-step relations should vanish, or why the graph should be a chain at all. The only motivation I could spot was the analogy to graded human knowledge, which is not algebraically connected to cross-covariance of LoRA down-projection outputs.
  • b) The relations are only realizable at zero. With C=A12C = A_{12} and D=A32D=A_{32}, CC⊤=DD⊤CC^\top = DD^\top forces equal rnages, while D⊤C=0D^\top C = 0 forces orthogonal ranges. So as Sec 3.3 and 4.3 mention, this implies C=D=0C=D=0. Thus, the zigzag terms vanish exactly when the middle group has zero cross-moments with its neighbors, which seems to be in tension with the paper's claim that zigzag sturcture keeps related groups closely connected (e.g., in abstract).
  • c) Other claims do not seem fully supported. The abstract states that relations are inferred from training signals and that graded structure is learned from training dynamics, but the graph and the relations imposed on it are pre-determined (only the arrow values depend on data), and the groups are arbitrarily fixed slices of the rank dimension. So, it remains unclear to me what relational structure is learned beyond the cross-moment values on this fixed graph.
  • d) There seem to be directly relevant prior works such as [1] that regularizes cross-covariance of activations, or [2] that applies local decorrelation. This also limits the conceptual novely of the paper.

[1] Cogswell, M., Ahmed, F., Girshick, R., Zitnick, L., & Batra, D. (2015). Reducing overfitting in deep networks by decorrelating representations. ICLR 2016.

[2] Rodríguez, P., Gonzalez, J., Cucurull, G., Gonfaus, J. M., & Roca, X. (2016). Regularizing cnns with locally constrained decorrelations. ICLR 2017.

2. The experiments do not isolate the claims.

The orthogonality baseline (Table 6) does not seem controlled, as it penalizes all group pairs rather than graph edges, and its penalty form and whether locality terms are retained are not specified. In addition, the gain seems to come from the down-projection side (Table 5). Together with A-only and A+B performing similarly, while B-only gives no improvement (Table 5), this seems to leave open an alternative plausible explanation: the penalty may shrink the down-projection outputs, which changes the effective balance between the two LoRA factors [3]. Several baselines also fall below plain LoRA, and all methods seem to share an optimization configuration. Because of these, the gains cannot be attributed to relational structure. The experimental scope is also limited to one backbone, fine-tuned on one dataset, with MCQA evaluations alone. This leaves the generality of the proposed framework across models and adaptation settings untested.

[3] Hayou, S., Ghosh, N., & Yu, B. (2024). Lora+: Efficient low rank adaptation of large models. ICML 2024.

Questions for authors

Q1. Can the authors please clarify (1a) in Weaknesses?

Q2. Can the authors clarify (1b-c) in Weaknesses? In particular, what structure is the regularizer intended to induce algebraically beyond driving the middle group towards zero cross-covariance with the two neighbors? What makes zigzag relations a suitable inductive bias for LoRA rank groups?

Q3. For the K=6 construction and the orthogonal baselines, I cannot seem to find the exact loss and penalty weights in the text. Can the authors specify these? Also, the default config (rank 24, 3 groups) is reported multiple times across Table 1, Table 5, and Sec 4.3 (as well as plain LoRA) with different numbers. Which runs do these correspond to?

Reviewer_Ya5gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new regularization framework for low-rank adaptation (LoRA) that uses zigzag algebra to organize interactions among groups of trainable rank components, with the stated goal of moving beyond uniform orthogonality constraints and discrete expert routing. The method partitions the LoRA rank dimension into groups, represents neighboring groups through cross-moment matrices computed from their activations or weights, and treats compositions of these matrices as algebraic paths. For a three-group chain, the proposed objective penalizes non-returning paths and higher-order compositions while encouraging the two return cycles at the central vertex to agree; additional locality terms are intended to prevent the primitive interactions from collapsing. These losses are combined with the task cross-entropy objective and optimized through standard backpropagation, without introducing additional trainable parameters or inference-time routing modules. The experiments use Qwen2.5-7B-Instruct, train on 99,842 examples from the MMLU auxiliary training split, and evaluate primarily on MMLU-Pro, MMLU, and SciQ, with comparisons against standard LoRA, DoRA, OLoRA, SOS-LoRA, MoLA, MixLoRA, and orthogonality-based alternatives. The strongest reported result is 45.79% on MMLU-Pro at rank 126 with three groups, compared with 43.65% for plain LoRA, while the best reported average across the three main benchmarks is 71.29% at rank 126 with six groups, compared with 70.68% for plain LoRA. The ablations suggest that the proposed path-based penalty can improve MMLU-Pro performance, particularly at larger ranks, but the gains are not uniform across benchmarks or configurations, and the evidence that the algebraic formulation itself is responsible for the improvement remains incomplete.

Critical strengths

The paper uses zigzag algebra to control how different LoRA rank groups interact, rather than simply encouraging them to be separate. This provides a different way to design regularization for low-rank adaptation. The method adds no trainable parameters or inference-time modules. It uses existing LoRA activations and standard backpropagation, making it relatively easy to add to existing training pipelines.

Promising results at higher ranks: On MMLU-Pro, the method reaches 45.79% at rank 126 with three groups, compared with 43.65% for plain LoRA. It also outperforms the tested orthogonality baselines, suggesting that controlling group interactions may be more useful than simply separating them.

Critical weaknesses

(W1)The main contribution needs stronger evidence: The paper does not clearly show that the zigzag-algebra relations work better than simpler regularizers that penalize similar matrix products. Moreover, the exact relations have only a trivial solution under the proposed transpose-tied construction, raising questions about how closely the trained model follows the intended algebraic structure.

(W2) Limited and inconsistent performance gains: The experiments use only Qwen2.5-7B-Instruct and one training dataset. The method improves MMLU-Pro, especially at higher ranks, but does not consistently outperform other methods. For example, at rank 72 with three groups, its average accuracy is 70.30%, below SOS-LoRA's 70.74%.

(W3) The role of the learned group relationships is unclear: The paper reports lower activation similarities and smaller path values, but does not show whether the groups learn distinct and useful functions. In addition, removing both locality terms changes accuracy only slightly, from 42.96% to 42.99%, leaving their practical contribution unclear.

(W4) Some experimental details are missing: The paper does not clearly explain whether cross-moment statistics are accumulated across training batches, which matters given the microbatch size of six. It also lacks a comparison with CPS-LoRA and reports no actual training-time or memory overhead, making it difficult to judge the method's robustness and practical benefits.

Questions for authors

(Q1) What evidence shows that the zigzag relations are doing more than acting as generic path-product regularization? The paper acknowledges that the exact relations have only the trivial solution under the transpose-tied cross-moment construction. Could the authors compare the proposed loss against a simpler control that penalizes the same path products, but does not impose the full zigzag-algebra relation set? A useful experiment would retain the same groups, matrix normalization, loss coefficient search, and training budget, while changing only the relation terms. The authors could also compare against a generic penalty on adjacent-group correlations and path products.

(Q2) Can the authors demonstrate that the learned relational structure has a functional interpretation? The paper reports reduced cosine similarities, smaller path norms, and improved MMLU-Pro accuracy. Could the authors provide one analysis that directly connects these structural measurements to model behavior? For example, they could compare the learned cross-moment statistics across a small set of layers or rank groups, or test whether the components show different functional contributions under a controlled ablation. A small-scale analysis would be enough; a broad interpretability study is not necessary.

(Q3) How robust is the method relative to the strongest structured-subspace baseline? Could the authors clarify the SOS-LoRA group/experts configuration and, if practical, compare Zigzag LoRA against CPS-LoRA under a matched training setup?

(Q4) How are the cross-moment matrices estimated during training? The appendix reports a microbatch size of six, gradient accumulation of five, and use of the LoRA-A activations at the supervised answer position. Are the cross-moment matrices computed from each microbatch separately, or are activations or moments accumulated across microbatches?

(Q5) The paper argues that the method is GPU-friendly and introduces no inference-time overhead, but it does not report measured training throughput or peak memory. Could the authors provide a small comparison of steps per second and memory use for plain LoRA and Zigzag LoRA?

Reviewer_uYn5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Zigzag LoRA, a parameter-efficient fine-tuning method that introduces algebra-inspired constraints on interactions and compositions among LoRA rank groups. The method uses cross-moment matrices to represent relations and paths, without adding trainable parameters or inference-time modules. Experiments on three multiple-choice benchmarks show promising results in some settings, but whether the learned representations exhibit the claimed algebraic structure remains unclear.

Critical strengths

The paper explores an algebra-inspired approach to structuring LoRA rank groups. It also includes comparisons with several existing LoRA methods and orthogonality-based baselines.

Critical weaknesses
  1. The exact zigzag relations admit only a trivial solution, while the proposed method relies on soft regularization. It remains unclear whether the improvements come from zigzag-specific properties or general higher-order regularization.

  2. Experiments use only one backbone and three multiple-choice benchmarks. The ablation studies do not sufficiently isolate the contributions of individual components, particularly the zigzag and locality terms.

  3. The motivation does not clearly explain why zigzag algebra is necessary. Some contribution claims are overstated, and the placement of tables disrupts the logical flow and readability.

Questions for authors

Please refer to the concerns raised in the Critical Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.