acceptodds
Withdrawn from ICLR 2027

SHARD: Subspace-centric Structured Harmonization And Relation Distillation-guided Activation for Continual Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Continual Instruction Tuning (MCIT) allows models to incrementally learn from task streams without retraining. However, existing approaches are hindered by cumulative knowledge conflicts between incremental knowledge from LoRA subspace and pretrained knowledge, and ineffective task-specific subspace activation.To address these challenges, we propose a novel Subspace-centric Structured Harmonization And Relation Distillation-guided Activation (SHARD) framework, which consists of Structure-aware Knowledge Harmonization (SKH) and Relation Distillation-guided Subspace Activation (RDSA). Specifically, SKH first identifies critical LLM substructures by learning task-specific structure importance through a continuous relaxation of L0 regularization using hard-concrete distributions. SKH subsequently employs a projection-aware mapping, driven by a semantic outer-product formulation that strictly adheres to the Transformer topology, systematically translating these 1D structural priors into precise 2D parameter-level masks. SKH ultimately utilizes these mapping masks to dynamically modulate the magnitude mask, explicitly guiding new knowledge injection toward less critical pretrained regions within the LoRA subspace for progressive harmonization with pretrained knowledge. Meanwhile, RDSA employs relation distillation to align the distance relations between old prototypes and both new task samples and their own evolving representations. This prevents representation drift, allowing the prototypes to act as stable semantic anchors that accurately route task-specific LoRA subspaces. Extensive experiments reveal the effectiveness and generality of our SHARD.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
MLLM, Multimodal Continual Instruction Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_g7pPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Although the proposed framework is technically sophisticated, several important concerns remain. First, the fundamental premise of SKH, which assumes that task-specific structural importance can indicate regions of pretrained knowledge requiring protection, lacks sufficient theoretical motivation and empirical evidence. Second, the approach does not fundamentally address the scalability challenge of continual learning, as maintaining independent LoRA experts for each task may lead to increasing memory and inference costs. Finally, the experimental validation is conducted on relatively limited task sequences and does not adequately investigate long-term scalability, computational efficiency, memory consumption, or robustness under more diverse and challenging continual learning scenarios.

Critical strengths

The writting is generally good.

Critical weaknesses
  1. SKH increases training complexity but incurs no analysis cost.

  2. The issue of the unlimited growth in the number of LoRA experts has not been resolved.

  3. RDSA's prototype routing may rely excessively on the visual domain.

  4. There is a lack of theoretical explanation for relation distillation. Why does maintaining the relationship preserve task identity?

  5. RDSA maintains a task prototype for each task and employs cosine similarity for routing. This approach relies on the implicit assumption that a task can be effectively represented by a single direction within the projected image feature space. For real-world, complex tasks, this assumption is highly questionable. A single task may encompass multiple visual-semantic clusters. For instance, medical datasets often contain a mix of organs, imaging modalities, and disease types. Relying on a single prototype risks "prototype averaging" or even causing the centers of distinct tasks to converge too closely.

  6. The authors employ the projector from the previous stage as a "teacher," guiding the current projector to align its relation distribution regarding old prototypes with that of the previous stage. This effectively constitutes a "teacher chain," where the teacher from stage t−1t-1 guides the projector at stage tt, while the teacher at t−1t-1 is itself the product of prior distillation. Consequently, over long task sequences, there is a risk of cumulative error. However, as the paper’s benchmarks involve relatively short task sequences, it remains unproven whether stability can be maintained across sequences comprising dozens or even hundreds of tasks.

Questions for authors

Please see the weakness section.

Reviewer_XEZvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes SHARD, a subspace-centric framework for Multimodal Continual Instruction Tuning that addresses cumulative conflict between incremental LoRA knowledge and pretrained knowledge, and the mis-activation of task subspaces when different tasks share identical instructions. SHARD combines structure-aware regularization that steers LoRA updates away from critical pretrained regions with relation-distilled task prototypes for stable task-agnostic routing, and reports consistent MFN/BWT gains over DISCO and other MCIT baselines on UCIT and MLLM-DCL.

Critical strengths
  1. The routing-failure diagnosis is concrete and verifiable (0% accuracy on Flickr30k, Table 4), and RDSA's fix is validated by the same routing-accuracy metric rather than aggregate scores alone.

  2. The projection-aware mapping is topology-motivated, and ablations (Table 10) confirm a naive broadcast-multiply alternative performs substantially worse, proving the design is necessary.

Critical weaknesses
  1. While SKH and RDSA are each empirically effective (see Strengths), both are local modifications to DISCO's existing architecture. SKH adds a regularizer on where LoRA writes, and RDSA swaps the router's input modality and adds a distillation loss, rather than introducing a new conceptual framework. This limits the paper's contribution to incremental engineering improvements on a known architecture.

  2. The headline comparisons in Tables 1-2 report single runs without seeds or confidence intervals. Several gains over the strongest baselines are only 1-2 points and inconsistent across individual tasks, making it hard to rule out training noise as the source of the claimed SOTA margin.

Questions for authors
  1. Are all experiments based on a single run? Can you report std over multiple seeds to assess whether the gains exceed training noise?

  2. Can you add a 2×2 ablation (text/image prototype × with/without relation distillation)? Based on existing numbers, the bare image prototype (67.06 / -10.01) actually underperforms DISCO's text prototype (67.31 / -8.87), suggesting the gain comes entirely from distillation, not the image modality itself. This would require revising the "image features suit routing better" claim.

  3. What are the training time, memory, and storage increases over DISCO, given the added per-task L0 search plus T sets of experts/prototypes/structure importance?

  4. Constants like ε_mag, ε_str, etc. lack sensitivity analysis, How were they chosen, and are they fixed across datasets/backbones?

Reviewer_iR1jOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SHARD for multimodal continual instruction tuning, combining structure-aware regularization to protect pretrained knowledge from disruptive LoRA updates with relation distillation to stabilize task prototypes for routing task-specific LoRA adapters.

Critical strengths
  1. Empirical improvements across benchmarks and backbones. SHARD improves final performance over DISCO on both UCIT and MLLM-DCL. On MLLM-DCL with LLaVA-1.5, it improves both initial task learning and final performance, suggesting that the retention gains do not simply come at the expense of plasticity. Additional results on InternVL-Chat-7B support its applicability beyond a single backbone.

  2. Targeted and informative ablation studies. The paper separately examines the contributions of SKH and RDSA, with the full method outperforming both ablated variants. Further experiments on projected image features and relation distillation provide useful empirical support for these design choices within the evaluated setting.

Critical weaknesses
  1. The novelty over closely related work is insufficiently established. SKH’s basic magnitude mask and weighted LoRA regularizer closely resemble LoRASculpt [1], while RDSA shares PRD’s [2] core idea of preserving prototype–sample relations using new-task data without replay. Although SHARD introduces structure-aware calibration and applies relation distillation to task-specific adapter routing, the additional value of these modifications remains unclear. The authors should explicitly discuss these connections and compare against adapted versions of LoRASculpt’s regularizer and PRD, including their direct combination within the same framework, to establish the incremental contribution.

References

[1] Liang et al. LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models. arXiv:2503.16843, 2025.

[2] Asadi et al. Prototype-Sample Relation Distillation: Towards Replay-Free Continual Learning. ICML, 2023.

  1. The scope of the routing solution is insufficiently tested. Equation (21) computes routing weights from image features alone. Consequently, different instructions for the same image receive identical routing weights, even when they may require different task adapters. The reported experiments address similar instructions across different image distributions, but do not establish reliability when tasks share visual content or have overlapping visual domains. Controlled evaluations of these cases and comparison with an image-and-text router would clarify this limitation.

  2. Important methodological and evaluation details need clarification. Equation (22) does not explicitly define how the prototype and relation losses form L_sub, and reuses L_SKH for both the total objective and its regularization term. There is also a metric inconsistency: Equations (23), (24), and (26) imply BWT = MFN − MFT, which gives −6.01 for SHARD on UCIT, rather than the reported −7.22. The reported value appears consistent with averaging over the previous T−1 tasks instead of all T tasks. Please clarify the implemented training objective and metric normalization, and make the equations consistent with the reported results.

Questions for authors

Please address the questions and concerns raised in the Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.