acceptodds
Under review as a conference paper at ICLR 2027

Null-Space Orthogonal Fine-Tuning for Multimodal Continual Instruction Tuning

Abstract

Multimodal Continual Instruction Tuning (MCIT) requires Multimodal Large Language Models (MLLMs) to adapt to new tasks while retaining previously learned knowledge, making the balance between stability and plasticity a central challenge. Orthogonal Fine-Tuning (OFT), traditionally applied to parameter-efficient downstream adaptation, offers a potential starting point for MCIT through orthogonal transformations. In MCIT, orthogonality alone does not explicitly control interference with previous tasks. A null-space constraint can reduce such interference, but simply imposing it on OFT is insufficient to guarantee orthogonality. Therefore, we propose Null-space Orthogonal Fine-Tuning (NOFT), which parameterizes task-specific transformations to jointly satisfy the null-space and orthogonality constraints, thereby improving stability on continual adaptation. To enhance plasticity under OFT's fixed block-diagonal structure, NOFT further introduces plasticity-aware permutation, which rearranges the trainable block positions to capture more current-task gradient information without increasing the parameter budget. Experiments with LLaVA-v1.5-7B and Qwen3-VL-8B on two MCIT benchmarks demonstrate that NOFT effectively balances stability and plasticity and achieves state-of-the-art performance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Instruction Tuning, Parameter-Efficient Fine-Tuning, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XRcSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces NOFT, a parameter-efficient framework for multimodal continual instruction tuning (MCIT). The proposed method combines null-space constraints with orthogonal transformations to reduce forgetting of previously learned tasks. It also introduces a gradient-guided permutation strategy to improve adaptation to new tasks. Experimental results show improved overall performance and reduced forgetting.

Critical strengths
  1. The paper identifies two limitations of OFT in continual learning and proposes solutions. A null-space constrained orthogonal parameterization is designed to reduce interference with previously learned tasks, and a gradient-guided permutation rearranges the trainable blocks according to current-task gradients.
  2. Experiments with LLaVA-v1.5-7B and Qwen3-VL-8B on UCIT and TriGap benchmarks show that NOFT achieves the best average final performance and the smallest forgetting gap among the methods included in the forgetting analysis.
Critical weaknesses
  1. Using the null space of previous-task features to reduce forgetting is not new. Adam-NSCL [1] and NESS [2] have both done this, and NESS directly restricts weight updates to such a null space. The proposed parameterization is also very close to PSOFT [3], which applies an orthogonal transformation within a subspace of the weights. NOFT mainly replaces that subspace with the null space. For the permutation, BOFT [4] and GSOFT [5] have already explored ways to relax the block-diagonal structure, and GSOFT also uses permutations. I think the paper should discuss more clearly what is new compared with these works.
  2. The permutation is motivated as a way to improve plasticity, but the ablation only reports final performance, so it is hard to tell whether it actually helps the model learn new tasks. Its gain over random permutation is small, not much larger than the variation across random seeds, and NOFT’s own variance is not reported. NOFT’s immediate performance on LLaVA is even slightly lower than DISCO’s on both benchmarks. As far as I understand, the improvement seems to come mostly from less forgetting rather than better plasticity.

[1] Training Networks in Null Space of Feature Covariance for Continual Learning, CVPR 2021

[2] Learning in the Null Space: Small Singular Values for Continual Learning, CPAL 2026

[3] Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation, ICLR 2026

[4] Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization, ICLR 2024

[5] Group and Shuffle: Efficient Structured Orthogonal Parametrization, NeurIPS 2024

Questions for authors
  1. Apart from the choice of subspace, how is Eq. 3 different from PSOFT? How does NOFT compare with NESS?
  2. The null-space dimension m varies across layers and tasks. How is the block size set when m is not divisible by b? Does the number of trainable parameters change from task to task?
  3. Although Table 4 reports a peak memory of 42.2 GB, how much memory do the stored covariance matrices take, and is it included in Table 4?
  4. How does NOFT perform across models of different sizes? The current experiments only cover two models of similar sizes (7B/8B), making it difficult to assess its generalizability across model scales.
8 comments

Thank you for your careful review and constructive feedback. We address your questions and concerns as follows.

Q1: Novelty of NOFT and comparisons with PSOFT, NESS, Adam-NSCL, BOFT, and GSOFT.

We clarify the design motivations and underlying principles of NOFT in multimodal continual instruction tuning (MCIT) and compare it with the related methods above, followed by experimental evaluations on LLaVA-v1.5-7B and Qwen3-VL-8B using Average Immediate Performance (AIP), Forgetting Gap (FG), and Average Final Performance (AFP).

Q1-a: Comparison of NOFT with PSOFT regarding novelty and parameterization.

Derivation of Eq. (3) from stability requirements in MCIT. Orthogonal Fine-Tuning (OFT) preserves the norms and pairwise angles of weight vectors, but orthogonality alone does not constrain interference from weight updates on previous-task features. Meanwhile, directly projecting OFT-induced weight updates into the null space does not generally preserve the orthogonality of the resulting transformation. From the joint requirements of preserving weight geometry and constraining interference with previous tasks, we derive a transformation form in Eq. (3) that simultaneously satisfies the orthogonality and null-space constraints to maintain stability.

Methodological differences from PSOFT. Although PSOFT can be rewritten in a form similar to Eq. (3), it aims to balance parameter efficiency and task adaptability rather than the stability–plasticity trade-off in MCIT. Under the different subspaces induced by these objectives, PSOFT and NOFT employ distinct designs in their parameter-efficiency and task-adaptation mechanisms.

For parameter efficiency, PSOFT confines updates to a low-rank principal subspace of tens to hundreds of dimensions and learns a dense low-dimensional orthogonal matrix within it. In NOFT, however, the null space only identifies directions with low interference on previous-task features, without indicating their usefulness for the current task. Compressing this null space to a low rank would therefore restrict new-task learning. Instead, we retain the uncompressed null space of thousands of dimensions and achieve parameter efficiency through sparse block-diagonal orthogonal transformations.

For task adaptation, PSOFT introduces learnable scaling vectors into the transformation matrix, which improves downstream adaptability by relaxing strict orthogonality. However, this relaxation directly conflicts with our stability requirement in MCIT. Therefore, tailored to the sparse block-diagonal design, NOFT preserves strict orthogonality and proposes a novel plasticity-aware permutation that adaptively reorganizes trainable blocks based on current-task gradients, facilitating new-task learning without increasing trainable parameters or relaxing the stability constraints (further compared in Q1-c).

Experimental evaluation. Tables R3-1 and R3-2 (in the next comment) compare PSOFT and NOFT across four benchmark settings. Specifically, while PSOFT attains high AIP, it suffers from severe forgetting, with FG ranging from -13.90 to -26.32 across the four settings. In contrast, NOFT maintains comparable AIP on LLaVA-v1.5-7B and higher AIP on Qwen3-VL-8B, while keeping FG between -0.28 and -1.53 across all settings. Consequently, on LLaVA-v1.5-7B and Qwen3-VL-8B, NOFT surpasses PSOFT in AFP by 11.31 and 17.75 points on UCIT and by 12.46 and 25.94 points on TriGap, respectively. Moreover, removing the orthogonality of the transformation in NOFT (NOFT (w/o Orth.) in Table R3-1) worsens FG on UCIT from -1.53 to -4.18 on LLaVA-v1.5-7B and from -0.28 to -3.44 on Qwen3-VL-8B. Overall, these results demonstrate that PSOFT achieves strong plasticity on new tasks but suffers from severe forgetting. In contrast, NOFT better balances stability and plasticity, with the orthogonality constraint further helping to mitigate forgetting.

Table R3-1. Comparison with orthogonal and null-space baselines on the UCIT benchmark.

LLaVA-v1.5-7BQwen3-VL-8B
MethodAIP ↑\uparrow (plasticity)FG ↑\uparrow (stability)AFP ↑\uparrow (overall)AIP ↑\uparrow (plasticity)FG ↑\uparrow (stability)AFP ↑\uparrow (overall)
OFT77.05-10.9166.1484.39-17.6366.76
PSOFT77.84-13.9063.9485.26-17.2268.04
Adam-NSCL74.25-4.6669.5980.94-2.5578.39
NESS74.21-3.6170.6083.81-3.0980.72
NOFT (w/o Orth.)76.94-4.1872.7685.96-3.4482.52
NOFT76.78-1.5375.2586.07-0.2885.79

Table R3-2. Comparison with orthogonal and null-space baselines on the TriGap benchmark.

LLaVA-v1.5-7BQwen3-VL-8B
MethodAIP ↑\uparrow (plasticity)FG ↑\uparrow (stability)AFP ↑\uparrow (overall)AIP ↑\uparrow (plasticity)FG ↑\uparrow (stability)AFP ↑\uparrow (overall)
OFT49.53-15.4934.0470.48-25.9644.52
PSOFT50.52-14.9135.6173.12-26.3246.80
Adam-NSCL45.53-4.4841.0568.79-2.6466.15
NESS46.13-3.6042.5370.60-2.7367.87
NOFT49.35-1.2848.0773.78-1.0472.74

Q1-b: Comparison of NOFT with Adam-NSCL and NESS beyond null-space constraint.

Although Adam-NSCL, NESS, and NOFT all restrict updates to the approximate null space of previous-task features to reduce interference, they differ in the geometric constraints imposed on the weight updates. Both Adam-NSCL and NESS formulate their updates additively (Wt=W_t = Wt−1+W_{t-1} + ΔWt\Delta W_t). Adam-NSCL projects candidate parameter updates generated by Adam onto the null space at each step, while NESS adopts a LoRA-style parameterization using a fixed null-space basis and a single trainable coefficient matrix. Crucially, their null-space constraints alone do not additionally guarantee the preservation of norms and pairwise angles of weight vectors across sequential tasks. In contrast, NOFT constrains the update via a strict orthogonal transformation (Wt=W_t = RtR_t Wt−1W_{t-1} with Rt⊤R_t^\top Rt=IR_t = I). By jointly enforcing the null-space and orthogonality constraints, NOFT not only leaves layer outputs on previous-task features nearly unchanged, but also guarantees Wt⊤W_t^\top Wt=W_t = Wt−1⊤W_{t-1}^\top Wt−1W_{t-1} to preserve the norms and pairwise angles of weight vectors for stability.

Empirically, NOFT achieves performance gains in both plasticity and stability over Adam-NSCL and NESS. First, in Table R3-1, removing orthogonality (NOFT (w/o Orth.)) under the null-space constraint degrades FG by 2.65 and 3.16 points and reduces AFP by 2.49 and 3.27 points on LLaVA-v1.5-7B and Qwen3-VL-8B, respectively. Second, in Tables R3-1 and R3-2, NOFT surpasses both Adam-NSCL and NESS in AIP and FG across all four settings, yielding AFP gains of 5.66–7.40 points over Adam-NSCL and 4.65–5.54 points over NESS.

Q1-c: Comparison of NOFT with BOFT and GSOFT in block-diagonal relaxation and permutation design.

BOFT, GSOFT, and NOFT address distinct objectives via orthogonal fine-tuning. BOFT and GSOFT focus on balancing parameter efficiency and transformation expressiveness. Their core idea is to multiply several sparse block-diagonal matrices to construct a dense orthogonal matrix, using butterfly factorizations in BOFT or group-and-shuffle structures in GSOFT. Notably, the fixed permutation in GSOFT is designed as a topological shuffle to construct dense matrices, rather than directly targeting task plasticity. In comparison, NOFT focuses on the stability–plasticity trade-off in multimodal continual instruction tuning (MCIT). Starting from a single sparse block-diagonal matrix, NOFT employs a plasticity-aware permutation to reorganize trainable parameters to positions with larger current-task gradient magnitudes. This provides greater potential for loss reduction and enhances adaptation to new tasks under stability constraints.

To evaluate the performance gains of the plasticity-aware permutation Pt{P}_t over BOFT and GSOFT, we conduct an additional ablation study by incorporating their orthogonal parameterization strategies into NOFT with matched parameters and training setups. As shown in Table R3-3, under the null-space constraint, BOFT and GSOFT achieve lower plasticity (AIP) and overall performance (AFP) than NOFT across both LLaVA-v1.5-7B and Qwen3-VL-8B. This validates the effectiveness of our permutation design in MCIT.

we will include these results in the revised manuscript.

Table R3-3. Ablation study of plasticity-aware permutation (PtP_t) on the UCIT benchmark. Notation X→YX \rightarrow Y denotes replacing component XX with strategy YY. Values in parentheses indicate differences relative to NOFT.

LLaVA-v1.5-7B    Qwen3-VL-8B
VariantAIP ↑\uparrow (plasticity)FG ↑\uparrow (stability)AFP ↑\uparrow (overall)    AIP ↑\uparrow (plasticity)FG ↑\uparrow (stability)AFP ↑\uparrow (overall)
NOFT76.78-1.5375.2586.07-0.2885.79
w/o PtP_t76.02 (-0.76)-1.49 (+0.04)74.53 (-0.72)85.14 (-0.93)-0.47 (-0.19)84.67 (-1.12)
Pt→P_t \rightarrow Random75.94 (-0.84)-1.20 (+0.33)74.74 (-0.51)85.30 (-0.77)-0.25 (+0.03)85.05 (-0.74)
Pt→P_t \rightarrow BOFT75.83 (-0.95)-1.25 (+0.28)74.58 (-0.67)85.25 (-0.82)-0.51 (-0.23)84.74 (-1.05)
Pt→P_t \rightarrow GSOFT75.99 (-0.79)-1.67 (-0.14)74.32 (-0.93)85.39 (-0.68)-0.77 (-0.49)84.62 (-1.17)
Reviewer_kPeKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, Null-space Orthogonal Fine-Tuning (NOFT) is proposed to parameterize task-specific transformations to jointly satisfy the null-space and orthogonality constraints to improve stability on continual adaptation. Under NOFT, the plasticity-aware permutation is designed to rearrange the trainable block positions to capture more current-task gradient information without increasing the parameter budget. Experiments with LLaVA-v1.5-7B and Qwen3-VL-8B on two MCIT benchmarks demonstrate that NOFT effectively balances stability and plasticity and achieves state-of-the-art performance.

Critical strengths
  1. The formulations for null-space-constrained orthogonal transformation and plasticity-aware permutation are clearly presented.
  2. An excessive number of SOTA methods, such as SICO, KeepLoRA, SAME and CL-MoE, are included for comparison.
  3. Efficiency comparisons on UCIT using LLaVA-v1.5-7B are provided.
  4. The ablation experiments are comprehensive.
Critical weaknesses
  1. The introduction requires more sufficient and in-depth discussion of recent studies on orthogonal fine-tuning.
  2. In the introduction, the authors state that the task-specific component is designed to interfere with previously acquired knowledge while preserving the geometric properties of orthogonal fine-tuning. However, they do not justify why these geometric properties are important.
  3. In the introduction, the authors claim that NOFT effectively balances stability and plasticity, yet they do not explain the origins of this stability and plasticity.
  4. Figure 1 only illustrates the architecture of NOFT and fails to show its differences from other OFT architectures.
  5. Tables 1 and 2 report the average final performance after sequential learning over all tasks on TriGap and UCIT. Nevertheless, NOFT does not yield the best results on InfovQA, aRXIvQA, Rodeside and FloodNetVQA.
  6. The authors do not analyze why the two metrics, Average Final Performance and Average Immediate Performance, are suitable for measuring the stability plasticity trade-off.
  7. The authors do not explain the differences between NOFT and the compared baselines.
Questions for authors

N/A

6 comments

Thank you for your thorough comments and suggestions. The following are the detailed answers to your concerns.

Q1: Discussion of recent orthogonal fine-tuning studies in the Introduction.

Recent developments in orthogonal fine-tuning (OFT) are reviewed in the Related Work section of the original manuscript. Overall, existing OFT studies primarily focus on single-task adaptation or LLM pre-training, and their designs do not prevent interference across sequential tasks in Multimodal Continual Instruction Tuning (MCIT). In the following, we summarize these developments along three main directions:

(1) Improving parameter and computational efficiency of OFT. Vanilla OFT [1] uses block-diagonal parameterizations to reduce trainable parameters. BOFT [2], GOFT [3], and GSOFT [4] refine this parameterization for better parameter efficiency by multiplying sparse matrices via butterfly factorization, Givens rotations, and group-and-shuffle structures, respectively, while OFTv2 [5] accelerates orthogonal matrix construction by approximating matrix inversion with a truncated Neumann series. Compared with multiplying multiple sparse matrices with fixed topological patterns, our NOFT introduces a plasticity-aware permutation to adaptively reorganize the trainable blocks of a single block-diagonal matrix based on current-task gradients. This design enhances new-task adaptation without increasing trainable parameters.

(2) Enhancing downstream adaptation capability of OFT. Strict orthogonal transformation fixes the norms and pairwise angles of weight vectors, which may limit downstream flexibility. To improve adaptability, qGOFT (an extension of GOFT [3]) relaxes orthogonality via soft regularization, and PSOFT [6] confines updates to the low-rank principal subspace of pre-trained weights while introducing learnable scaling vectors to relax orthogonality. Unlike these single-task methods that trade strict orthogonality for adaptability, NOFT enforces orthogonality within the null space of previous tasks to mitigate the forgetting of previously learned knowledge.

(3) Extending OFT to LLM pre-training. Orthogonal parameterization has also been extended to training LLMs from scratch. POET [7] applies learnable orthogonal transformations to both sides of a fixed random weight matrix to preserve its singular value spectrum for stable training, and POET-X [8] further improves the computational and memory efficiency of these updates for billion-parameter pre-training. These methods only require general orthogonal designs to achieve stable pre-training from scratch, without considering catastrophic forgetting. In contrast, NOFT targets continual task adaptation via null-space constraints and plasticity-aware permutations.

Experimental comparison. Excluding pre-training methods [7,8] and selecting the most representative fine-tuning designs from each category above, we directly evaluate NOFT on the UCIT benchmark against vanilla OFT, BOFT [2], GSOFT [4], and PSOFT [6] under the sequential instruction tuning setting. As shown in Table R2-1, all compared baselines achieve comparable plasticity (AIP) to NOFT on both backbones. However, since these methods are designed for single-task adaptation without constraining interference with previously learned tasks during new-task adaptation, they all exhibit a notable decline in stability (FG) and much lower final performance (AFP). In contrast, by enforcing orthogonal updates within the null space of previous tasks alongside the plasticity-aware permutation, NOFT effectively mitigates forgetting while preserving high plasticity, achieving the best stability (FG) and final performance (AFP) on both backbones.

We will include a concise overview of these recent OFT developments in the revised Introduction to better contextualize our work.

Table R2-1. Comparison of NOFT with representative OFT methods and parameterization variants on the UCIT benchmark. Values in parentheses indicate differences relative to NOFT.

LLaVA-v1.5-7B    Qwen3-VL-8B
MethodAIP ↑\uparrow (plasticity)FG ↑\uparrow (stability)AFP ↑\uparrow (overall)    AIP ↑\uparrow (plasticity)FG ↑\uparrow (stability)AFP ↑\uparrow (overall)
NOFT76.78-1.5375.2586.07-0.2885.79
PSOFT77.84 (+1.06)-13.90 (-12.37)63.94 (-11.31)85.26 (-0.81)-17.22 (-16.94)68.04 (-17.75)
BOFT76.56 (-0.22)-11.35 (-9.82)65.21 (-10.04)84.75 (-1.32)-15.52 (-15.24)69.23 (-16.56)
GSOFT76.90 (+0.12)-12.42 (-10.89)64.48 (-10.77)85.13 (-0.94)-18.24 (-17.96)66.89 (-18.90)
OFT77.05 (+0.27)-10.91 (-9.38)66.14 (-9.11)84.39 (-1.68)-17.63 (-17.35)66.76 (-19.03)

Q2: Discussion on the importance of geometric properties in the Introduction.

Following prior orthogonal fine-tuning (OFT) literature [1,2,6,7,8], the geometric properties refer to the norms and pairwise angles of weight vectors within each linear layer, which are strictly preserved by the orthogonal transformation RR with R⊤R=IR^\top R = I (Eqs. 1,2 in Section 2). The benefits of preserving these properties are shown below:

(1) Preserving pairwise angles helps retain pre-trained knowledge. Orthogonal transformations rotate weight vectors wi,wjw_i, w_j within a layer simultaneously while keeping pairwise cosine similarities invariant for any pair of weight vectors wi,wjw_i, w_j, i.e., cos⁡(Rwi,Rwj)=cos⁡(wi,wj)\cos(Rw_i, Rw_j) = \cos(w_i, w_j). Prior orthogonal fine-tuning studies including OFT [1], BOFT [2], and PSOFT [6] suggest that these pairwise angular relationships among weight vectors encode the semantic and relational structure acquired during pre-training. Keeping this angular structure invariant during task adaptation helps preserve pre-trained knowledge and mitigate forgetting.

(2) Preserving the spectral norm supports stable fine-tuning. Orthogonal transformations adapt the directions of weight vectors while keeping each vector’s norm invariant as ∣Rwi∣2=∣wi∣2\\|Rw_i\\|_2 = \\|w_i\\|_2. Combined with the invariant pairwise angles, this keeps the entries of the Gram matrix (W⊤W)ij=∣wi∣∣wj∣cos⁡(wi,wj)(W^\top W)_{ij} = \\|w_i\\| \\|w_j\\| \cos(w_i, w_j) invariant, which directly preserves the matrix spectral norm ∣RW∣2=∣W∣2=λmax⁡(W⊤W)\\|RW\\|_2 = \\|W\\|_2 = \sqrt{\lambda_{\max}(W^\top W)}, where λmax⁡\lambda_{\max} denotes the largest eigenvalue. Consistent with the findings of POET [7] and POET-X [8], preserving the spectral norm keeps each linear layer’s maximum amplification factor unchanged during adaptation, supporting stable optimization.

In NOFT, these geometric properties are maintained by the orthogonality constraint, and the results in Table R2-2 further indicate that preserving them helps enhance stability. Specifically, when we remove the orthogonality constraint (w/o Orth. RtR_t) under the same training settings as NOFT, Forgetting Gap (FG) drops by 2.65 and 3.16 points on LLaVA-v1.5-7B and Qwen3-VL-8B, resulting in Average Final Performance (AFP) reductions of 2.49 and 3.27 points.

We will clarify the role of these geometric properties in the revised Introduction.

Table R2-2. Ablation study of orthogonality constraint on the UCIT benchmark. Values in parentheses denote differences relative to NOFT.

LLaVA-v1.5-7B    Qwen3-VL-8B
VariantAIP ↑\uparrow (plasticity)FG ↑\uparrow (stability)AFP ↑\uparrow (overall)    AIP ↑\uparrow (plasticity)FG ↑\uparrow (stability)AFP ↑\uparrow (overall)
NOFT76.78-1.5375.2586.07-0.2885.79
w/o Orth. RtR_t76.94 (+0.16)-4.18 (-2.65)72.76 (-2.49)85.96 (-0.11)-3.44 (-3.16)82.52 (-3.27)

Reference

[1] Controlling Text-to-Image Diffusion by Orthogonal Finetuning, NeurIPS 2023.

[2] Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization, ICLR 2024.

[3] Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation, ICML 2024.

[4] Group and Shuffle: Efficient Structured Orthogonal Parametrization, NeurIPS 2024.

[5] Orthogonal Finetuning Made Scalable, EMNLP 2025.

[6] Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation, ICLR 2026.

[7] Reparameterized LLM Training via Orthogonal Equivalence Transformation, NeurIPS 2025.

[8] POET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformation, ICML 2026.

Q3: Discussion on the origins of NOFT's stability and plasticity in the Introduction.

Paragraph 4 of the Introduction outlines how our designs contribute to stability and plasticity. In the following, we explain their specific mechanisms in detail.

Stability in NOFT. NOFT derives a null-space-constrained orthogonal transformation that restricts weight updates to the previous-task null space while preserving orthogonality. Together, these two constraints improve stability during continual adaptation:

(1) Null-space constraint. We compute an approximate null space from the feature covariance matrices of previous tasks. By confining weight updates to this null space, the updates leave the layer outputs on previous-task features nearly unchanged, thereby preserving previously learned knowledge.

(2) Orthogonality constraint. While the null-space constraint limits output changes on previous-task features, keeping the transformation orthogonal preserves the norms and pairwise angles of weight vectors throughout continual adaptation. This structural constraint maintains the geometric relations of the pre-trained weights, helping to further mitigate forgetting.

Plasticity in NOFT. Plasticity originates from optimizing trainable parameters to adapt the model to incoming tasks. For parameter efficiency, vanilla OFT uses a block-diagonal structure that confines trainable parameters to fixed positions, which may limit adaptation when those fixed positions do not align with the current task. Since gradient components with larger magnitudes offer greater local potential for loss reduction, allocating the limited trainable parameters to these positions enables more effective task adaptation. Therefore, NOFT introduces plasticity-aware permutation to reorganize the trainable block positions using the gradient of the current task, capturing gradient components with larger magnitudes to enhance plasticity without increasing the parameter budget.

Experimental verification. Beyond the mechanism discussion above, we provide a component ablation study on the UCIT benchmark in Table R2-3, reporting Average Immediate Performance (AIP) for plasticity, Forgetting Gap (FG) for stability, and Average Final Performance (AFP) for overall performance. The results align with our design motivation, indicating that the null-space and orthogonality constraints primarily maintain stability, whereas the plasticity-aware permutation mainly enhances plasticity. Across both LLaVA-v1.5-7B and Qwen3-VL-8B, removing the null-space constraint (w/o UtU_t) or the orthogonality constraint (w/o Orth. RtR_t) causes a notable decline in FG with minor changes in AIP, while removing the permutation (w/o PtP_t) keeps FG similar to NOFT but lowers AIP.

We will further clarify these mechanisms in Paragraph 4 of the revised Introduction.

Table R2-3. Ablation study of NOFT on the UCIT benchmark. Values in parentheses indicate differences relative to NOFT.

LLaVA-v1.5-7B    Qwen3-VL-8B
VariantAIP ↑\uparrow (plasticity)FG ↑\uparrow (stability)AFP ↑\uparrow (overall)    AIP ↑\uparrow (plasticity)FG ↑\uparrow (stability)AFP ↑\uparrow (overall)
NOFT76.78-1.5375.2586.07-0.2885.79
w/o Orth. RtR_t76.94 (+0.16)-4.18 (-2.65)72.76 (-2.49)85.96 (-0.11)-3.44 (-3.16)82.52 (-3.27)
w/o UtU_t76.88 (+0.10)-11.10 (-9.57)65.78 (-9.47)85.79 (-0.28)-14.36 (-14.08)71.43 (-14.36)
w/o Pt{P}_t76.02 (-0.76)-1.49 (+0.04)74.53 (-0.72)85.14 (-0.93)-0.47 (-0.19)84.67 (-1.12)

Q4: Figure 1 only illustrates the architecture of NOFT and fails to show its differences from other OFT architectures.

In addition to Figure 1, we will add an architectural comparison figure in the revised manuscript to contrast NOFT with other OFT architectures. Compared with prior OFT methods, the two core modules illustrated in Figure 1 are newly introduced in NOFT:

(1) Null-Space-Constrained Transformation. Traditional OFT methods optimize across the full orthogonal manifold, whereas NOFT introduces a null-space constraint from previous-task features. This restricts the orthogonal transformation to the intersection of the orthogonal manifold and the null-space-induced affine space, mitigating interference with prior tasks.

(2) Plasticity-Aware Permutation. Instead of keeping trainable blocks at fixed diagonal positions as in standard OFT, NOFT introduces a permutation module to reorganize trainable block positions. This design captures larger current-task gradient components within a limited parameter budget, thereby enhancing task adaptation.

Reviewer_5YvBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the stability–plasticity trade-off in Multimodal Continual Instruction Tuning (MCIT) and proposes Null-Space Orthogonal Fine-Tuning (NOFT). For stability, the authors derive a parameterization that jointly satisfies null-space and orthogonality constraints, using historical visual and textual features to construct an approximate null space that mitigates forgetting. For plasticity, they introduce a gradient-guided permutation mechanism that reorganizes trainable orthogonal blocks to improve adaptation without increasing the parameter budget. Experiments on two MLLMs and two MCIT benchmarks demonstrate consistent improvements in final performance and reduced forgetting.

Critical strengths
  1. NOFT provides a theoretically grounded parameterization that jointly satisfies the null-space and orthogonality constraints.

  2. Extensive experiments across two backbones and two benchmarks demonstrate consistent performance improvements, supported by ablation studies and forgetting analysis.

Critical weaknesses

The performance gains from plasticity-aware permutation are relatively modest, raising concerns about whether its additional computational complexity is justified.

Questions for authors
  1. Could the authors report Average Immediate Performance and Forgetting Gap for the ablations in Table 3 to clarify whether the proposed components contribute to stability by reducing forgetting and to plasticity by improving immediate task performance?

  2. Could the authors report the additional computational overhead introduced specifically by plasticity-aware permutation, given its relatively modest performance gains over random permutation?

3 comments

We appreciate your insightful comments and thorough review. The following are the detailed answers to your questions and concerns.

Q1: Supplementing plasticity and stability metrics in the ablation studies, and analyzing the contribution of each component.

Thank you for this suggestion. In Table R1-1 (core component ablations) and Table R1-2 (comparison of permutation variants), we report Average Immediate Performance (AIP) for plasticity, Forgetting Gap (FG) for stability, and Average Final Performance (AFP) across all tasks after completing the sequential adaptation process.

In Table R1-1, across both LLaVA-v1.5-7B and Qwen3-VL-8B, variants removing the null-space basis (w/o UtU_t) or the orthogonality constraint (w/o Orth. RtR_t), as well as the variant replacing UtU_t with a randomly initialized basis (Ut→U_t \rightarrow Random), all exhibit a notable decline in FG. These results indicate that the null-space basis UtU_t and the orthogonality constraint on transformation RtR_t primarily contribute to stability by mitigating forgetting.

In Table R1-2, across both backbones, the variant without the plasticity-aware permutation (w/o PtP_t), as well as variants replacing PtP_t with a random permutation (Pt→P_t \rightarrow Random) or with alternative orthogonal parameterizations based on butterfly factorization (Pt→P_t \rightarrow BOFT) and group-and-shuffle structure (Pt→P_t \rightarrow GSOFT), all maintain a comparable FG to NOFT but exhibit a lower AIP. These results indicate that PtP_t primarily contributes to plasticity by improving immediate task performance, and show the advantage of our permutation module in enhancing plasticity over random permutation and alternative orthogonal parameterizations.

Overall, the null-space and orthogonality constraints primarily maintain stability while the plasticity-aware permutation mainly enhances plasticity, thereby enabling NOFT to achieve the highest final performance (AFP) across both backbones. We will update Table 3 in Section 5.1 with the results in Table R1-1 and include Table R1-2 in the revised manuscript.

Table R1-1. Ablation study of core components on the UCIT benchmark. Notation X→YX \rightarrow Y denotes replacing component XX with YY. Values in parentheses indicate differences relative to NOFT.

LLaVA-v1.5-7B    Qwen3-VL-8B
VariantAIP ↑\uparrow (plasticity)FG ↑\uparrow (stability)AFP ↑\uparrow (overall)    AIP ↑\uparrow (plasticity)FG ↑\uparrow (stability)AFP ↑\uparrow (overall)
NOFT76.78-1.5375.2586.07-0.2885.79
w/o Orth. RtR_t76.94 (+0.16)-4.18 (-2.65)72.76 (-2.49)85.96 (-0.11)-3.44 (-3.16)82.52 (-3.27)
w/o UtU_t76.88 (+0.10)-11.10 (-9.57)65.78 (-9.47)85.79 (-0.28)-14.36 (-14.08)71.43 (-14.36)
w/o PtP_t76.02 (-0.76)-1.49 (+0.04)74.53 (-0.72)85.14 (-0.93)-0.47 (-0.19)84.67 (-1.12)
Ut→U_t \rightarrow Random76.57 (-0.21)-8.72 (-7.19)67.85 (-7.40)85.72 (-0.35)-9.69 (-9.41)76.03 (-9.76)
OFT77.05 (+0.27)-10.91 (-9.38)66.14 (-9.11)84.39 (-1.68)-17.63 (-17.35)66.76 (-19.03)

Table R1-2. Comparison of plasticity-aware permutation (PtP_t) and its variants on the UCIT benchmark. Notations follow Table R1-1.

LLaVA-v1.5-7B    Qwen3-VL-8B
VariantAIP ↑\uparrow (plasticity)FG ↑\uparrow (stability)AFP ↑\uparrow (overall)    AIP ↑\uparrow (plasticity)FG ↑\uparrow (stability)AFP ↑\uparrow (overall)
NOFT76.78-1.5375.2586.07-0.2885.79
w/o PtP_t76.02 (-0.76)-1.49 (+0.04)74.53 (-0.72)85.14 (-0.93)-0.47 (-0.19)84.67 (-1.12)
Pt→P_t \rightarrow Random75.94 (-0.84)-1.20 (+0.33)74.74 (-0.51)85.30 (-0.77)-0.25 (+0.03)85.05 (-0.74)
Pt→P_t \rightarrow BOFT75.83 (-0.95)-1.25 (+0.28)74.58 (-0.67)85.25 (-0.82)-0.51 (-0.23)84.74 (-1.05)
Pt→P_t \rightarrow GSOFT75.99 (-0.79)-1.67 (-0.14)74.32 (-0.93)85.39 (-0.68)-0.77 (-0.49)84.62 (-1.17)

Q2: Computational cost and performance gains of the plasticity-aware permutation.

To evaluate both the computational cost and performance gains of the permutation module, we have conducted two additional analyses on the UCIT benchmark.

(1) Computational cost of the plasticity-aware permutation. As shown in Table R1-3, the plasticity-aware permutation introduces minimal computational overhead, accounting for only 5.2% and 7.4% of the total training time on LLaVA-v1.5-7B and Qwen3-VL-8B, respectively.

Specifically, this permutation is constructed once at the beginning of each task in two steps: (i) running a single forward–backward pass (without weight updates) over 5% of current-task samples to collect gradients, and (ii) constructing the permutation matrix via a greedy strategy to rearrange the trainable block positions to the coordinates with larger gradient magnitudes. Among these two steps, gradient collection takes 4.2% and 5.9% of total training time on LLaVA-v1.5-7B and Qwen3-VL-8B, while greedy permutation construction takes 1.0% and 1.5%, respectively.

Table R1-3. Detailed time cost of NOFT on the UCIT benchmark.

BackboneTotal TimePlasticity-aware PermutationNull-space MaintenanceModel Optimization
LLaVA-v1.5-7B15.4 h0.8 h (5.2%)2.8 h11.8 h
Qwen3-VL-8B14.8 h1.1 h (7.4%)3.1 h10.6 h

(2) Statistical significance analysis of performance gains. To verify the performance gains of plasticity-aware permutation, we conduct statistical significance tests across three random seeds. For each seed, we train NOFT, the random-permutation variant (Rand. $ {P}_t$), and the no-permutation variant (w/o Pt{P}_t) under identical experimental settings, and then perform two-sided paired tt-tests across the three seeds to compare NOFT against each variant. As shown in Table R1-4 below, NOFT improves the overall performance (Mean AFP) over the random-permutation and no-permutation variants by 0.51 and 0.72 points on LLaVA-v1.5-7B, and by 0.74 and 1.12 points on Qwen3-VL-8B, respectively. For both backbones, the paired tt-tests against the two baseline variants yield p<0.05p < 0.05, indicating that the gains from the plasticity-aware permutation PtP_t are statistically significant and consistent across the three evaluated seeds.

These findings provide a more comprehensive assessment of the computational cost and performance benefits of the plasticity-aware permutation module.

We will include the detailed results of both analyses in the Appendix of the revised manuscript.

Table R1-4. Statistical significance analysis using two-sided paired tt-tests across three random seeds.

LLaVA-v1.5-7B  Qwen3-VL-8B
Method3-Seed AFPMean ± StdMean Diffpp-value  3-Seed AFPMean ± StdMean Diffpp-value
NOFT[74.85, 75.35, 75.55]75.25 ± 0.36+0.00—[85.25, 85.95, 86.17]85.79 ± 0.48+0.00—
Rand. $ {P}_t$[74.18, 74.94, 75.10]74.74 ± 0.49-0.510.024[84.43, 85.51, 85.21]85.05 ± 0.56-0.740.041
w/o Pt{P}_t[74.37, 74.38, 74.84]74.53 ± 0.27-0.720.037[84.51, 84.45, 85.05]84.67 ± 0.33-1.120.036

Dear Reviewer 5YvB,

Thank you for your dedicated time and valuable feedback. With less than two days left in the discussion phase, we wanted to follow up to see if our responses have addressed your initial concerns. If there are any additional points or feedback you'd like us to consider, please let us know. We greatly value your insights and are eager to resolve any remaining concerns you might have before the deadline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.