- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces NOFT, a parameter-efficient framework for multimodal continual instruction tuning (MCIT). The proposed method combines null-space constraints with orthogonal transformations to reduce forgetting of previously learned tasks. It also introduces a gradient-guided permutation strategy to improve adaptation to new tasks. Experimental results show improved overall performance and reduced forgetting.
Critical strengths
- The paper identifies two limitations of OFT in continual learning and proposes solutions. A null-space constrained orthogonal parameterization is designed to reduce interference with previously learned tasks, and a gradient-guided permutation rearranges the trainable blocks according to current-task gradients.
- Experiments with LLaVA-v1.5-7B and Qwen3-VL-8B on UCIT and TriGap benchmarks show that NOFT achieves the best average final performance and the smallest forgetting gap among the methods included in the forgetting analysis.
Critical weaknesses
- Using the null space of previous-task features to reduce forgetting is not new. Adam-NSCL [1] and NESS [2] have both done this, and NESS directly restricts weight updates to such a null space. The proposed parameterization is also very close to PSOFT [3], which applies an orthogonal transformation within a subspace of the weights. NOFT mainly replaces that subspace with the null space. For the permutation, BOFT [4] and GSOFT [5] have already explored ways to relax the block-diagonal structure, and GSOFT also uses permutations. I think the paper should discuss more clearly what is new compared with these works.
- The permutation is motivated as a way to improve plasticity, but the ablation only reports final performance, so it is hard to tell whether it actually helps the model learn new tasks. Its gain over random permutation is small, not much larger than the variation across random seeds, and NOFT’s own variance is not reported. NOFT’s immediate performance on LLaVA is even slightly lower than DISCO’s on both benchmarks. As far as I understand, the improvement seems to come mostly from less forgetting rather than better plasticity.
[1] Training Networks in Null Space of Feature Covariance for Continual Learning, CVPR 2021
[2] Learning in the Null Space: Small Singular Values for Continual Learning, CPAL 2026
[3] Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation, ICLR 2026
[4] Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization, ICLR 2024
[5] Group and Shuffle: Efficient Structured Orthogonal Parametrization, NeurIPS 2024
Questions for authors
- Apart from the choice of subspace, how is Eq. 3 different from PSOFT? How does NOFT compare with NESS?
- The null-space dimension m varies across layers and tasks. How is the block size set when m is not divisible by b? Does the number of trainable parameters change from task to task?
- Although Table 4 reports a peak memory of 42.2 GB, how much memory do the stored covariance matrices take, and is it included in Table 4?
- How does NOFT perform across models of different sizes? The current experiments only cover two models of similar sizes (7B/8B), making it difficult to assess its generalizability across model scales.
8 comments
Table R3-1. Comparison with orthogonal and null-space baselines on the UCIT benchmark.
| LLaVA-v1.5-7B | Qwen3-VL-8B | ||||||
|---|---|---|---|---|---|---|---|
| Method | AIP (plasticity) | FG (stability) | AFP (overall) | AIP (plasticity) | FG (stability) | AFP (overall) | |
| OFT | 77.05 | -10.91 | 66.14 | 84.39 | -17.63 | 66.76 | |
| PSOFT | 77.84 | -13.90 | 63.94 | 85.26 | -17.22 | 68.04 | |
| Adam-NSCL | 74.25 | -4.66 | 69.59 | 80.94 | -2.55 | 78.39 | |
| NESS | 74.21 | -3.61 | 70.60 | 83.81 | -3.09 | 80.72 | |
| NOFT (w/o Orth.) | 76.94 | -4.18 | 72.76 | 85.96 | -3.44 | 82.52 | |
| NOFT | 76.78 | -1.53 | 75.25 | 86.07 | -0.28 | 85.79 |
Table R3-2. Comparison with orthogonal and null-space baselines on the TriGap benchmark.
| LLaVA-v1.5-7B | Qwen3-VL-8B | ||||||
|---|---|---|---|---|---|---|---|
| Method | AIP (plasticity) | FG (stability) | AFP (overall) | AIP (plasticity) | FG (stability) | AFP (overall) | |
| OFT | 49.53 | -15.49 | 34.04 | 70.48 | -25.96 | 44.52 | |
| PSOFT | 50.52 | -14.91 | 35.61 | 73.12 | -26.32 | 46.80 | |
| Adam-NSCL | 45.53 | -4.48 | 41.05 | 68.79 | -2.64 | 66.15 | |
| NESS | 46.13 | -3.60 | 42.53 | 70.60 | -2.73 | 67.87 | |
| NOFT | 49.35 | -1.28 | 48.07 | 73.78 | -1.04 | 72.74 |
Q1-b: Comparison of NOFT with Adam-NSCL and NESS beyond null-space constraint.
Although Adam-NSCL, NESS, and NOFT all restrict updates to the approximate null space of previous-task features to reduce interference, they differ in the geometric constraints imposed on the weight updates. Both Adam-NSCL and NESS formulate their updates additively ( ). Adam-NSCL projects candidate parameter updates generated by Adam onto the null space at each step, while NESS adopts a LoRA-style parameterization using a fixed null-space basis and a single trainable coefficient matrix. Crucially, their null-space constraints alone do not additionally guarantee the preservation of norms and pairwise angles of weight vectors across sequential tasks. In contrast, NOFT constrains the update via a strict orthogonal transformation ( with ). By jointly enforcing the null-space and orthogonality constraints, NOFT not only leaves layer outputs on previous-task features nearly unchanged, but also guarantees to preserve the norms and pairwise angles of weight vectors for stability.
Empirically, NOFT achieves performance gains in both plasticity and stability over Adam-NSCL and NESS. First, in Table R3-1, removing orthogonality (NOFT (w/o Orth.)) under the null-space constraint degrades FG by 2.65 and 3.16 points and reduces AFP by 2.49 and 3.27 points on LLaVA-v1.5-7B and Qwen3-VL-8B, respectively. Second, in Tables R3-1 and R3-2, NOFT surpasses both Adam-NSCL and NESS in AIP and FG across all four settings, yielding AFP gains of 5.66–7.40 points over Adam-NSCL and 4.65–5.54 points over NESS.
Q1-c: Comparison of NOFT with BOFT and GSOFT in block-diagonal relaxation and permutation design.
BOFT, GSOFT, and NOFT address distinct objectives via orthogonal fine-tuning. BOFT and GSOFT focus on balancing parameter efficiency and transformation expressiveness. Their core idea is to multiply several sparse block-diagonal matrices to construct a dense orthogonal matrix, using butterfly factorizations in BOFT or group-and-shuffle structures in GSOFT. Notably, the fixed permutation in GSOFT is designed as a topological shuffle to construct dense matrices, rather than directly targeting task plasticity. In comparison, NOFT focuses on the stability–plasticity trade-off in multimodal continual instruction tuning (MCIT). Starting from a single sparse block-diagonal matrix, NOFT employs a plasticity-aware permutation to reorganize trainable parameters to positions with larger current-task gradient magnitudes. This provides greater potential for loss reduction and enhances adaptation to new tasks under stability constraints.
To evaluate the performance gains of the plasticity-aware permutation over BOFT and GSOFT, we conduct an additional ablation study by incorporating their orthogonal parameterization strategies into NOFT with matched parameters and training setups. As shown in Table R3-3, under the null-space constraint, BOFT and GSOFT achieve lower plasticity (AIP) and overall performance (AFP) than NOFT across both LLaVA-v1.5-7B and Qwen3-VL-8B. This validates the effectiveness of our permutation design in MCIT.
we will include these results in the revised manuscript.
Table R3-3. Ablation study of plasticity-aware permutation () on the UCIT benchmark. Notation denotes replacing component with strategy . Values in parentheses indicate differences relative to NOFT.
| LLaVA-v1.5-7B | Qwen3-VL-8B | ||||||
|---|---|---|---|---|---|---|---|
| Variant | AIP (plasticity) | FG (stability) | AFP (overall) | AIP (plasticity) | FG (stability) | AFP (overall) | |
| NOFT | 76.78 | -1.53 | 75.25 | 86.07 | -0.28 | 85.79 | |
| w/o | 76.02 (-0.76) | -1.49 (+0.04) | 74.53 (-0.72) | 85.14 (-0.93) | -0.47 (-0.19) | 84.67 (-1.12) | |
| Random | 75.94 (-0.84) | -1.20 (+0.33) | 74.74 (-0.51) | 85.30 (-0.77) | -0.25 (+0.03) | 85.05 (-0.74) | |
| BOFT | 75.83 (-0.95) | -1.25 (+0.28) | 74.58 (-0.67) | 85.25 (-0.82) | -0.51 (-0.23) | 84.74 (-1.05) | |
| GSOFT | 75.99 (-0.79) | -1.67 (-0.14) | 74.32 (-0.93) | 85.39 (-0.68) | -0.77 (-0.49) | 84.62 (-1.17) |
Thank you for your careful review and constructive feedback. We address your questions and concerns as follows.
We clarify the design motivations and underlying principles of NOFT in multimodal continual instruction tuning (MCIT) and compare it with the related methods above, followed by experimental evaluations on LLaVA-v1.5-7B and Qwen3-VL-8B using Average Immediate Performance (AIP), Forgetting Gap (FG), and Average Final Performance (AFP).
Q1-a: Comparison of NOFT with PSOFT regarding novelty and parameterization.
Derivation of Eq. (3) from stability requirements in MCIT. Orthogonal Fine-Tuning (OFT) preserves the norms and pairwise angles of weight vectors, but orthogonality alone does not constrain interference from weight updates on previous-task features. Meanwhile, directly projecting OFT-induced weight updates into the null space does not generally preserve the orthogonality of the resulting transformation. From the joint requirements of preserving weight geometry and constraining interference with previous tasks, we derive a transformation form in Eq. (3) that simultaneously satisfies the orthogonality and null-space constraints to maintain stability.
Methodological differences from PSOFT. Although PSOFT can be rewritten in a form similar to Eq. (3), it aims to balance parameter efficiency and task adaptability rather than the stability–plasticity trade-off in MCIT. Under the different subspaces induced by these objectives, PSOFT and NOFT employ distinct designs in their parameter-efficiency and task-adaptation mechanisms.
For parameter efficiency, PSOFT confines updates to a low-rank principal subspace of tens to hundreds of dimensions and learns a dense low-dimensional orthogonal matrix within it. In NOFT, however, the null space only identifies directions with low interference on previous-task features, without indicating their usefulness for the current task. Compressing this null space to a low rank would therefore restrict new-task learning. Instead, we retain the uncompressed null space of thousands of dimensions and achieve parameter efficiency through sparse block-diagonal orthogonal transformations.
For task adaptation, PSOFT introduces learnable scaling vectors into the transformation matrix, which improves downstream adaptability by relaxing strict orthogonality. However, this relaxation directly conflicts with our stability requirement in MCIT. Therefore, tailored to the sparse block-diagonal design, NOFT preserves strict orthogonality and proposes a novel plasticity-aware permutation that adaptively reorganizes trainable blocks based on current-task gradients, facilitating new-task learning without increasing trainable parameters or relaxing the stability constraints (further compared in Q1-c).
Experimental evaluation. Tables R3-1 and R3-2 (in the next comment) compare PSOFT and NOFT across four benchmark settings. Specifically, while PSOFT attains high AIP, it suffers from severe forgetting, with FG ranging from -13.90 to -26.32 across the four settings. In contrast, NOFT maintains comparable AIP on LLaVA-v1.5-7B and higher AIP on Qwen3-VL-8B, while keeping FG between -0.28 and -1.53 across all settings. Consequently, on LLaVA-v1.5-7B and Qwen3-VL-8B, NOFT surpasses PSOFT in AFP by 11.31 and 17.75 points on UCIT and by 12.46 and 25.94 points on TriGap, respectively. Moreover, removing the orthogonality of the transformation in NOFT (NOFT (w/o Orth.) in Table R3-1) worsens FG on UCIT from -1.53 to -4.18 on LLaVA-v1.5-7B and from -0.28 to -3.44 on Qwen3-VL-8B. Overall, these results demonstrate that PSOFT achieves strong plasticity on new tasks but suffers from severe forgetting. In contrast, NOFT better balances stability and plasticity, with the orthogonality constraint further helping to mitigate forgetting.