acceptodds
Under review as a conference paper at ICLR 2027

Granularity Matters: Task-Level Preservation and Direction-Level Transfer for Continual Multimodal Instruction Tuning

Abstract

Continual multimodal instruction tuning (CMIT) requires multimodal large language models to acquire new knowledge without forgetting previous tasks. Task-level preservation addresses this challenge by protecting the expert learned for each task, while leaving historical knowledge confined within individual experts. Historical expert transfer can overcome this confinement, but assigning a single strength to the complete expert suppresses useful directions when the value is low and amplifies irrelevant or conflicting directions when it is high. Effective transfer therefore requires direction-level transfer without weakening task-level preservation. We propose PHAROS, a novel framework that decouples task-level preservation from direction-level transfer. PHAROS preserves each task with a private LoRA expert and its associated multimodal projector. To enable adaptive and selective transfer, PHAROS derives functional directions from the layer-wise updates of each historical private expert. For each new task, PHAROS learns an independent signed gate for every historical direction, allowing the sign and strength of the same direction to adapt across target tasks. The gated directions form a task-dependent historical transfer branch alongside the new private expert, enabling selective historical transfer while preserving its dedicated trainable capacity. Extensive experiments on CoIN and UCIT show that PHAROS achieves average accuracies of 70.11% and 77.60%, outperforming the best prior results by 4.77% and 3.58%, respectively. Further experiments reveal that task-level preservation and direction-level transfer provide complementary gains, while the learned gates assign different signs and strengths to functional directions from the same historical expert across target tasks. These results demonstrate the value of preserving task knowledge as complete experts while transferring historical knowledge through finer functional directions.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Continual multimodal instruction tuning, multimodal large language models, LoRA expert

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_nAQsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PHAROS, a framework that separates task-level preservation from direction-level transfer. Each task retains a frozen private LoRA expert and its associated multimodal projector. To support later tasks, PHAROS decomposes historical LoRA update operators into normalized rank-one directions using compact SVD. A new task learns an independent signed gate for each historical direction, alongside its own private expert. At inference time, image and text prototypes select the appropriate task-specific configuration. Experiments cover CoIN, UCIT, alternative task orders, and a larger language backbone.

Critical strengths
  1. The construction of transfer directions is well motivated. The paper explicitly recognizes that equivalent LoRA factorizations can have different factor coordinates. Deriving directions from the SVD of the update operator avoids tying the interpretation to an arbitrary factorization. Normalization then separates the scale inherited from the source expert from the strength learned for a target task. This gives the direction-level transfer mechanism a clear mathematical interpretation.

  2. The reported results combine strong acquisition with effective retention. PHAROS achieves 70.11 Avg.ACC, 70.39 New.ACC, and 0.32 forgetting on CoIN. On UCIT, it reaches a final Last score of 77.60, compared with 72.80 for EBLoRA. The alternative task orders and nearly flat post-acquisition trajectories provide further evidence that the preservation mechanism works consistently across the evaluated sequences.

Critical weaknesses
  1. The experiments do not directly test the paper’s main claim about transfer granularity. Table 5 adds task preservation, direction transfer, and projector preservation sequentially. This demonstrates that the components are useful, but it does not compare expert-level and direction-level transfer while keeping preservation, projectors, and routing unchanged. It also does not establish whether allowing negative gates is beneficial relative to nonnegative direction weights. Since projector preservation is added last, the table does not show the effect of removing direction-level transfer from the otherwise complete framework.

The gate visualizations show that different directions receive different weights, but this behavior follows from the flexibility of the parameterization. The more consequential question is whether this flexibility improves transfer relative to a complete-expert coefficient. That comparison is central to the paper’s argument.

  1. The strongest gains require a clearer account of experimental comparability. Appendix A.4 states that the CoIN baselines are taken from a published comparison rather than rerun in the authors’ codebase. The provenance is clearly acknowledged, but matching the backbone and task order does not fully establish that the training and evaluation settings are comparable. Appendix A.3 also omits several important settings, including the learning rate, training budget, and the gate bound and initialization.

This matters particularly for Grounding. Relative to MoBLoRA, the score increases from 32.95 to 65.00. That task contributes approximately 4.01 points to the eight-task mean improvement of 4.77 points—about 84% of the gain. A closer comparison of the Grounding setup would therefore have a substantial effect on how I interpret the main result.

Questions for authors
  1. With task preservation, projector preservation, and routing held fixed, how do complete-expert weighting, nonnegative direction weighting, and signed direction weighting compare?
  2. Could the authors provide the full training configuration, particularly the LoRA settings, learning rate, training budget, gate bound β\beta, and gate initialization? What exactly is the “Baseline” in Table 5?
  3. Are the Grounding data splits, training settings, coordinate formats, and evaluation procedures aligned with those used by the strongest baselines? A matched comparison would help explain the unusually large improvement on this task.
4 comments
AuthorsOpenReview ↗

We sincerely thank reviewer nAQs for the careful and constructive assessment

Response to Reviewer nAQs

We sincerely thank the reviewer for the careful and constructive assessment. The questions were especially valuable for strengthening our controlled comparisons, training transparency, and Grounding protocol reporting.

1. Complete-expert, nonnegative, and signed weighting

The controls share private rank 8, projector preservation, routing, data, task order, seed, training budget, and first-task checkpoint before separate continual-training chains. Complete-expert weighting learns one signed coefficient per source expert and module, while nonnegative and signed weighting independently gate normalized directions.

WeightingImageNet-RArXivQAViz-capIconQACLEVR-MathFlickr30kMean
Complete expert90.8094.3961.3382.5075.1357.5076.94
Nonnegative directions90.8093.4960.5183.3375.9057.5676.93
Signed directions (PHAROS)90.8094.7061.0284.2777.3357.5677.61

Mean is the arithmetic mean of the displayed scores, not trajectory-level UCIT Avg. Signed direction weighting improves the mean by 0.67 points over complete-expert weighting and 0.68 over nonnegative directions. It improves four tasks and ties two relative to nonnegative directions. Relative to complete-expert weighting, it improves four tasks, ties ImageNet-R, and is 0.31 points lower on Viz-cap. These task-wise differences show the value of independently controlling the sign and strength of historical directions.

2. Training configuration and Table 5 baseline

We appreciate the request for a clearer baseline definition. Table 5 uses our HiDe-LLaVA run [1], rather than PHAROS without historical transfer. Its score matrix gives 54.05 Avg.ACC and 63.08 New.ACC, and the revised table labels it HiDe-LLaVA (Baseline).

The controlled comparisons use the following configuration:

SettingConfiguration
Backbone / anchorsFrozen LLaVA-v1.5-7B / CLIP ViT-L/14-336
LoRA settingsRank 8; scale 2; q/k/v/o/up/down/gate projections
Learning ratesPrivate 2×10−42\times10^{-4}; gates 10−310^{-3}; projector 2×10−52\times10^{-5}
Gate bound / initializationg=tanh⁡(θ)∈[−1,1]g=\tanh(\theta)\in[-1,1]; θ=0\theta=0
Optimizer / scheduleAdamW; cosine; warmup 0.03; weight decay 0
TrainingOne epoch; no replay; seed 42
Effective batchUCIT 96; CoIN 48; CoIN VizWiz 24
Precision / lengthBF16, TF32 / 2048 tokens
MemoryGradient checkpointing; DeepSpeed ZeRO-2
PreservationHistorical experts, directions, gates, projectors frozen
Routing / constraintHard top-1 anchor; private orthogonality disabled

All transfer controls reuse the same first-task checkpoint and subsequently follow their own causal checkpoint chains.

These settings are now reported in the Training Configuration subsection of the appendix.

3. Grounding protocol and the large improvement

We agree that the large Grounding gain merits careful protocol reporting. The main table reports 65.00 for PHAROS, 36.59 for EProj [3], and 32.95 for MoBLoRA [4], corresponding to differences of 28.41 and 32.05 points. EProj is the strongest Grounding baseline, while MoBLoRA is the strongest baseline in overall Avg.ACC. These values compare complete methods.

PHAROS follows the official CoIN Grounding split with 55,885 training and 30,969 test records [2], normalized xyxy coordinates in the square-padded image frame, and the IoU>0.5\mathrm{IoU}>0.5 success criterion. EProj and MoBLoRA are compared through their published results on the same CoIN split and task definition. PHAROS uses one epoch, private rank 8, effective batch 48, greedy decoding, and the vicuna_v1 template.

This establishes alignment in the benchmark split, coordinate target, and evaluation criterion. The published baselines retain their original optimization settings and prediction pipelines, so the comparison is benchmark-matched rather than a strictly training- and executable-scorer-matched rerun. We have made this scope explicit and interpret the 65.00 score as a complete-method result rather than attributing the full gap to direction-level transfer alone.

The corresponding manuscript changes are highlighted in blue in the Table 5 baseline label, the Historical Transfer Controls subsection, and the appendix Training Configuration.

References

[1] Guo et al. HiDe-LLaVA. ACL, 2025.
[2] Chen et al. CoIN. NeurIPS Datasets and Benchmarks, 2024.
[3] He et al. Continual Instruction Tuning for Large Multimodal Models. 2023.
[4] Chen et al. From Experts to Bases (MoBLoRA). ACL, 2026.

Thank you for adding the comparison in Table 6. This is the control I was looking for: with preservation and routing held fixed, does it help to weight individual directions rather than the complete historical expert? The results favor signed direction-level transfer: its six-task mean is 77.61, compared with 76.94 for complete-expert weighting and 76.93 for nonnegative direction weighting. I also appreciate the clearer identification of the Table 5 baseline and the additional training details.

My remaining reservation is about the size and scope of this contribution. Preserving an expert as a whole while reusing selected directions from its update is a sensible distinction. Table 6 now gives direct evidence for it, but the gain over the simpler alternatives is modest in this single run; even the variant without historical transfer reaches 76.76. I see the larger Grounding gain as encouraging evidence for PHAROS as a complete system. As the response clarifies, however, the published baselines share the benchmark split and evaluation criterion but retain their own training and prediction pipelines. That comparison therefore does not tell us how much of the gain comes from direction-level transfer itself.

The revision has made me more confident that the proposed transfer mechanism helps. I remain less convinced that its advantage over simpler forms of reuse is substantial, so I am keeping my rating at 2 (weak rejection).

AuthorsOpenReview ↗

Thank you for your response

Thank you for your response. In Table 6, we isolate the effect of the historical transfer strategy by keeping the preservation and routing mechanisms, private rank, data, task order, seed, training budget, and first-task checkpoint identical across all variants. Under this fair setting, signed direction-level transfer raises the six-task mean from 76.76% to 77.61% on UCIT, delivering an average improvement of nearly 1% and a gain of up to 3.83% on CLEVR-Math. The component analysis on CoIN shows a consistent benefit, where direction-level transfer increases the average performance from 67.96% to 69.82%, an improvement of nearly 2%. These results demonstrate that our direction-level transfer provides a clear and substantial advantage over simpler forms of reuse across both benchmarks.

To further verify that this benefit generalizes beyond PHAROS, we will add experiments that incorporate signed direction-level transfer into representative continual-learning baselines under matched settings.

We sincerely thank you again for the constructive feedback. If any concerns remain, we would be happy to discuss them further.

AuthorsOpenReview ↗

We sincerely thank you again

If our response and the additional experiments adequately address your concerns, we would sincerely appreciate your reconsideration of the current rating. We would also be happy to provide further clarification if any concerns remain.

Reviewer_bEZPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PHAROS, a continual multimodal instruction tuning framework that separates task-level preservation from direction-level knowledge transfer. Each task maintains a private LoRA expert and multimodal projector. Historical LoRA updates are decomposed into rank-one directions using SVD, and independently learned signed gates control their contributions to subsequent tasks. A CLIP-based routing mechanism selects the appropriate task configuration at inference time. Experiments on CoIN and UCIT report average accuracies of 70.11% and 77.60%, respectively. The authors also evaluate task-order robustness, forgetting, backbone size, and the contributions of individual components.

Critical strengths
  1. A meaningful approach to selective historical knowledge transfer. I find the distinction between task-level preservation and direction-level transfer to be the most interesting contribution. Decomposing the effective LoRA updates via SVD and learning signed gates for each direction provides a more flexible approach than assigning a single transfer weight to an entire historical expert. The operator-based decomposition is also technically well motivated, since it avoids dependence on arbitrary LoRA factor coordinates.

  2. Strong empirical results with useful component analysis. The evaluation covers two established continual multimodal benchmarks, different task orders, and a larger backbone. Table 5 shows that direction-level transfer improves CoIN accuracy from 67.96% to 69.82% and reduces forgetting from 2.99 to 1.35. These results provide evidence that historical transfer contributes to more than task-level preservation.

Critical weaknesses
  1. The specific advantage of direction-level transfer is not sufficiently established. My main concern is whether the proposed SVD-based signed gating is substantially better than simpler historical transfer mechanisms. Table 5 shows that task-level preservation accounts for most of the performance improvement, increasing CoIN accuracy from 54.05% to 67.96%, while direction-level transfer contributes an additional 1.86 percentage points.

The current experiments do not clearly isolate whether this additional gain comes from SVD decomposition, signed gating, or simply increased flexibility in combining historical parameters. Direct comparisons with expert-level scalar gating, nonnegative direction gates, and parameter-matched low-rank transfer would strengthen the methodological contribution.

  1. Scalability and the consistency of benchmark improvements remain concerns. PHAROS stores separate LoRA experts and projectors for each task, together with an expanding historical direction bank. While this design protects previous task knowledge, its storage and inference costs increase as tasks accumulate. The reported low forgetting may also partly reflect task-specific parameter isolation and successful task routing.

In addition, the CoIN improvement is heavily concentrated in Grounding, where accuracy increases from 36.59% to 65.00%. This single task accounts for approximately 3.55 of the overall 4.77-point improvement over the strongest prior aggregate result. It remains unclear whether direction-level transfer provides similarly meaningful benefits across other task types.

Questions for authors
  1. Table 5 shows that direction-level transfer adds 1.86 percentage points beyond task-level preservation. Can the authors compare the proposed approach with expert-level scalar gating and simpler low-rank transfer methods under matched parameter budgets?

  2. What is the specific contribution of signed gating and SVD normalization? Would nonnegative gates or singular-value-weighted directions achieve similar results?

  3. How does performance differ between Oracle task selection and the proposed CLIP-based routing? Can the authors quantify the effect of routing errors on forgetting?

  4. How do memory and inference costs scale with the number of tasks? Can the authors clarify whether the method remains practical for longer task sequences?

  5. Can the authors provide a per-task analysis showing where direction-level transfer helps, particularly given the large contribution of Grounding to the overall CoIN improvement?

1 comment
AuthorsOpenReview ↗

We sincerely thank reviewer bEZP for the thoughtful and constructive assessment

Response to Reviewer bEZP

We sincerely thank the reviewer for the thoughtful and constructive assessment. These questions helped us sharpen the evidence for transfer granularity and clarify routing, scalability, and the distribution of task-wise gains.

1. Transfer granularity

We added UCIT controls sharing private rank 8, projector preservation, routing, data, task order, seed, training budget, and first-task checkpoint before separate continual-training chains. Complete-expert transfer uses one signed coefficient per source expert and module; raw-factor transfer matches PHAROS's direction-wise gate count.

MethodImageNet-RArXivQAViz-capIconQACLEVR-MathFlickr30kMean
Complete-expert transfer90.8094.3961.3382.5075.1357.5076.94
Nonnegative directions90.8093.4960.5183.3375.9057.5676.93
Raw-factor directions90.8094.4661.4775.2665.8357.0174.14
PHAROS90.8094.7061.0284.2777.3357.5677.61

Mean is the arithmetic mean of the displayed scores, not trajectory-level UCIT Avg. PHAROS exceeds complete-expert transfer by 0.67 points and raw factors by 3.48.

2. Signed gates, SVD, and normalization

Signed gates improve the mean by 0.68 points over nonnegative gates, improving ArXivQA, Viz-cap, IconQA, and CLEVR-Math and tying the other tasks. This supports signed contributions in this setting, not a universal advantage.

Raw LoRA terms depend on factor coordinates: for invertible QQ, (BQ)(Q−1A)=BA(BQ)(Q^{-1}A)=BA although its rank-one terms change. PHAROS derives unit-norm, mutually orthogonal directions from the complete operator; repeated singular values still permit subspace rotations. Relative to raw factors, PHAROS gains 9.01 points on IconQA and 11.50 on CLEVR-Math but is 0.45 lower on Viz-cap. This supports operator-derived directions.

3. Task-free routing

We sincerely agree that routing quality should be evaluated separately from knowledge preservation. During our experiments, we observed that routing errors can introduce additional forgetting. PHAROS is not specifically designed to address this source of forgetting; future work can quantify its effect and develop targeted mitigation strategies.

4. Feasibility on longer task sequences

We agree that longer task sequences raise an important scalability question. PHAROS remains applicable as tasks accumulate, but preserving each task's expert, functional directions, and projector introduces increasing storage: these dominant states grow linearly with task capacity, while the current allocated gate layout grows quadratically. The measured totals are 393.64M adaptation-state elements for six-task UCIT and 517.89M for eight-task CoIN. This growth makes storage the main bottleneck for substantially longer streams. Future work will address it through on-demand direction reconstruction, projector sharing or compression, redundant-direction pruning, and sparse historical activation, with the goal of retaining PHAROS's preservation and transfer benefits under a bounded storage budget.

5. Distribution of task-wise gains

The task-wise analysis clarifies where historical transfer is most useful. Relative to PHAROS without historical transfer, the UCIT changes are 0.00, +0.77+0.77, +0.42+0.42, +0.14+0.14, +3.83+3.83, and −0.02-0.02 points in task order, yielding a mean gain of 0.86. CLEVR-Math receives the largest benefit, indicating that the value of historical transfer varies with the target task.

On CoIN, Grounding contributes about 4.01 points to PHAROS's 4.77-point mean advantage over MoBLoRA [1]. Importantly, the remaining seven tasks still improve by +0.88+0.88 points on average, with positive differences on six tasks. Thus, Grounding is the largest contributor, while the broader result also reflects gains across most of the task sequence.

The revisions are highlighted in blue in Historical Transfer Controls and Storage Growth in the main Experiments section, the appendix derivation, and the conclusion.

Reference

[1] Chen et al. From Experts to Bases (MoBLoRA). ACL, 2026.

Reviewer_dv5XOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PHAROS for continual multimodal instruction tuning, combining task-level preservation with direction-level knowledge transfer. It preserves task-specific LoRA experts and projectors while decomposing historical LoRA updates into SVD-based directions. Learnable signed gates enable selective knowledge transfer, and CLIP-based retrieval supports task-free inference. Experiments on CoIN and UCIT demonstrate consistent improvements over existing methods.

Critical strengths
  1. Well-motivated framework. PHAROS separates task-level knowledge preservation from fine-grained historical knowledge transfer.

  2. SVD provides functional directions independent of arbitrary LoRA factorization, it’s more principled.

Critical weaknesses

1. The paper lacks direct comparisons with expert-level transfer and alternative fine-grained representations, weakening the empirical support for its central claim.

2. The accumulation of task-specific experts, projectors, gates, and historical directions may introduce substantial memory overhead, which is not quantitatively evaluated.

Questions for authors
  1. Could the authors provide a controlled comparison between the proposed direction-level transfer and task-/expert-level transfer under the same experimental settings to better validate the central claim regarding transfer granularity?

  2. Could the authors compare the proposed SVD-based decomposition with alternative fine-grained representations, such as raw LoRA rank-1 components, to clarify the specific benefits of SVD for historical knowledge transfer?

  3. Could the authors provide a quantitative analysis of the cumulative memory overhead as the number of tasks increases, including the storage costs of private experts, projectors, gates, and historical directions? How does the performance–memory trade-off compare with existing methods such as PCLR and MoBLoRA?

2 comments
AuthorsOpenReview ↗

We sincerely thank reviewer dv5X for the thoughtful and constructive assessment

Response to Reviewer dv5X

We sincerely thank the reviewer for the thoughtful and constructive assessment. The questions helped us strengthen the transfer-granularity evidence and report the memory–performance trade-off more transparently.

1. Controlled comparison of transfer granularity

We agree that a no-transfer ablation alone cannot validate direction-level control. We therefore added UCIT controls using the same private rank (8), projector preservation, routing, data, task order, seed, training budget, and first-task checkpoint before separate continual-training chains. Complete-expert transfer learns one signed coefficient for each historical expert and adapted module, whereas PHAROS gates its normalized directions independently.

MethodImageNet-RArXivQAViz-capIconQACLEVR-MathFlickr30kMean
No historical transfer90.8093.9360.6084.1373.5057.5876.76
Complete-expert transfer90.8094.3961.3382.5075.1357.5076.94
Raw-factor directions90.8094.4661.4775.2665.8357.0174.14
PHAROS90.8094.7061.0284.2777.3357.5677.61

Mean is the arithmetic mean of the displayed scores, not the trajectory-level UCIT Avg. PHAROS exceeds complete-expert transfer by 0.67 points. It improves ArXivQA, IconQA, CLEVR-Math, and Flickr30k, ties ImageNet-R, and is 0.31 points lower on Viz-cap.

2. Operator-derived versus raw-factor directions

The raw-factor control retains PHAROS's gate count, signed parameterization, normalization, private rank, and preservation policy. Raw LoRA terms depend on factor coordinates: for invertible QQ, (BQ)(Q−1A)=BA(BQ)(Q^{-1}A)=BA although the individual rank-one terms change. PHAROS instead derives unit-Frobenius-norm, mutually orthogonal directions from the complete update operator through SVD; repeated singular values still permit rotations within their shared subspace.

At the same gate count, PHAROS improves the six-task mean over raw factors by 3.48 points, including +9.01+9.01 on IconQA and +11.50+11.50 on CLEVR-Math. It is also higher on ArXivQA and Flickr30k, ties ImageNet-R, and is 0.45 points lower on Viz-cap. This supports the operator-derived representation.

3. Cumulative storage and performance–memory trade-off

The retained adaptation states are:

StateCoIN (8 tasks)UCIT (6 tasks)
Private experts159.91119.93
Historical directions159.91119.93
Projector bank167.84125.88
Extra projector copy20.9820.98
Gates0.1150.065
Other adaptation states9.146.86
Total517.89393.64

The count excludes the backbone, optimizer, and intermediate checkpoints but includes frozen tensors, allocated slots, duplicate copies, and cached reconstructible directions.

For task capacity TT, expert size EE, projector size PP, MM adapted modules, private rank rr, and remaining states OTO_T, the implementation stores

Ntotal(T)=2TE+(T+1)P+MrT2+OT,N_{\mathrm{total}}(T)=2TE+(T+1)P+MrT^2+O_T,

while MrT(T−1)/2MrT(T-1)/2 gate coefficients are causally used. The dominant matrices therefore grow linearly, while allocated gates grow quadratically.

MethodAvg.ACCCumulative total (M)
PGP [1]55.77340.80
EProj [1]60.791300.24
PCLR-small [1]60.58343.16
PCLR [1]62.19766.70
PHAROS70.11517.89

The baseline totals are published adapter counts, whereas PHAROS is measured from final checkpoint states; we therefore use this table to indicate scale.

The revisions are highlighted in blue in the new Historical Transfer Controls and Storage Growth analyses in the main Experiments section, the detailed appendix derivation, and the conclusion.

References

[1] Meng et al. PCLR. ICLR, 2026, Appendix Q.
[2] Chen et al. From Experts to Bases (MoBLoRA). ACL, 2026.

AuthorsOpenReview ↗

We sincerely thank you

If our response and the additional experiments adequately address your concerns, we would sincerely appreciate your reconsideration of the current rating. We would also be happy to provide further clarification if any concerns remain.

Reviewer_6mMQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the PHAROS method to improve knowledge transfer from a previous task to a new task in Continual Multimodal Instruction Tuning. The method preserves LoRA experts and a multimodal projector for each task to contain task-specific knowledge. When learning new tasks, PHAROS uses SVD to extract rank-1 direction vectors from previous LoRA experts, while learning a signed gate for each direction. In inference, the method uses the CLIP prototype to identify the task-id for using previously learned knowledge. Experiments on CoIN and UCIT show PHAROS outperforming previous methods.

Critical strengths
  1. The paper is well-organized, and motivation has a clear connection with method design. With LoRA, SVD, and learnable gating mechanisms, PHAROS separates task-level knowledge preservation and direction-level knowledge transfer.
  2. Considering multimodal large language model architectural characteristics, the paper preserves both the LoRA expert and the multimodal projector, preventing changes in visual representations from interfering with old tasks.
Critical weaknesses
  1. For the proposed Direction-level Transfer, the paper does not sufficiently validate its effectiveness. As shown in Table 5, Task-level Preservation contributes the majority of the performance gain, increasing Avg.ACC by 13.91 percentage points. Similar task-specific LoRA expert and task routing design is commonly used in several existing CMIT methods, such as [1][2]. In contrast, the Direction-level Transfer improves performance by 1.86 percentage points, without comparing against an expert-level transfer baseline that assigns a single coefficient to each historical expert or initializing a new LoRA expert from the historical expert. Thus, it remains unclear whether the improvement stems from finer-grained transfer.
  2. As PHAROS maintains a separate LoRA expert and multimodal projector for each task, the number of parameters grows at least linearly with the task sequence length. Moreover, the performance gains may be partially attributed to the increased parameter capacity rather than the proposed transfer mechanism.

[1] Hide-llava: Hierarchical decoupling for continual instruction tuning of multimodal large language model. ACL 2025

[2] Enhancing Multimodal Continual Instruction Tuning with BranchLoRA. ACL 2025

Questions for authors
  1. What is the baseline method in Table 5?
  2. Can you add a comparison of experiments over an Expert-level Transfer and the initializing of new task LoRA experts from historical experts to verify whether the proposed Direction-level Transfer performance gain stems from a finer-grained knowledge transfer mechanism?
  3. Can you report the storage overhead and number of parameters for PHAROS compared to existing methods, such as MoBLoRA and KeepLoRA, and further analyze how these overheads and parameters increased with the number of tasks?
3 comments
AuthorsOpenReview ↗

We sincerely thank reviewer 6mMQ for the careful and constructive assessment

Response to Reviewer 6mMQ

We sincerely thank the reviewer for the careful and constructive assessment, which helped us clarify the baseline, strengthen the controls, and report storage transparently.

1. Baseline in Table 5

The original “Baseline” row is our HiDe-LLaVA run [1] on CoIN, rather than PHAROS without historical transfer. Its score matrix gives 54.05 Avg.ACC and 63.08 New.ACC. We have renamed it “HiDe-LLaVA (Baseline)” in Table 5 and clarified its role in the accompanying analysis.

2. Expert-level transfer

We agree that comparison with no historical transfer alone cannot establish the benefit of direction-level signed control. We therefore added UCIT controls using the same private rank (8), projector preservation, routing policy, data, task order, seed, training budget, and first-task checkpoint. Complete-expert transfer learns one signed coefficient for each historical expert in each adapted module. Nonnegative directions test the role of signed contributions, while raw-factor directions match PHAROS's direction-wise gate count but replace operator-derived directions with normalized raw LoRA factors.

MethodImageNet-RArXivQAViz-capIconQACLEVR-MathFlickr30kMean
PHAROS w/o historical transfer90.8093.9360.6084.1373.5057.5876.76
Complete-expert transfer90.8094.3961.3382.5075.1357.5076.94
Nonnegative directions90.8093.4960.5183.3375.9057.5676.93
Raw-factor directions90.8094.4661.4775.2665.8357.0174.14
PHAROS90.8094.7061.0284.2777.3357.5677.61

PHAROS achieves the highest six-task mean, exceeding no historical transfer, complete-expert transfer, nonnegative directions, and raw-factor directions by 0.86, 0.67, 0.68, and 3.48 points, respectively. Compared with complete-expert transfer, it improves four tasks, ties ImageNet-R, and is 0.31 points lower on Viz-cap. Compared with nonnegative directions, signed transfer improves four tasks and ties two. The largest gains over raw factors occur on IconQA and CLEVR-Math.

3. Adaptation parameters and storage growth

We agree that PHAROS's cumulative storage cost must be explicit. We therefore compare its performance and cumulative adaptation storage after eight CoIN tasks with methods whose published totals are available:

MethodAvg.ACCCumulative total (M)
PGP [2]55.77340.80
EProj [2]60.791300.24
PCLR-small [2]60.58343.16
PCLR [2]62.19766.70
PHAROS70.11517.89

PHAROS achieves the highest Avg.ACC while requiring less cumulative storage than EProj and PCLR, although more than PGP and PCLR-small. The baseline values are published adapter totals, whereas PHAROS is measured from its final checkpoint; we therefore use the comparison to characterize the performance–storage trade-off rather than claim a strictly matched storage ranking. PHAROS is not constant-memory. Its count excludes the shared backbone, optimizer states, and intermediate checkpoints but includes frozen tensors, allocated slots, and duplicate copies. The component-wise breakdown is provided in the revised manuscript.

Let TT be task capacity, EE one private expert, PP one projector, MM the number of adapted modules, rr the private rank, and OTO_T the remaining retained states. The current implementation stores

Ntotal(T)=2TE+(T+1)P+MrT2+OT,N_{\mathrm{total}}(T)=2TE+(T+1)P+MrT^2+O_T,

while only MrT(T−1)/2MrT(T-1)/2 gate coefficients are causally used. Thus, experts, cached directions, and projectors grow linearly with task capacity, whereas the allocated gate layout grows quadratically.

Matched saved-state totals were unavailable for MoBLoRA and KeepLoRA. MoBLoRA's 124.83M value is an inference-loaded count [3], while no compatible KeepLoRA total was available [4]; neither is mixed into the cumulative table. However, we selected some methods within the relevant fields for comparison.

The corresponding revisions are highlighted in blue in the CoIN component analysis (Table 5), the new Historical Transfer Controls and Storage Growth analyses in the main Experiments section, the detailed storage-growth derivation in the appendix, and the conclusion.

References

[1] Guo et al. HiDe-LLaVA. ACL, 2025.
[2] Meng et al. PCLR. ICLR, 2026, Appendix Q.
[3] Chen et al. From Experts to Bases (MoBLoRA). ACL, 2026.
[4] Luo et al. KeepLoRA. ICLR, 2026.

Thanks for the authors' additional Expert-level Transfer controlled experiment, and explanation of the baseline definition in Table 5. The results show that PHAROS's Direction-level Transfer performs better than Complete-expert transfer and Nonnegative directions in the supplementary experiments, providing more support for the proposed transfer method.

However, I have some reservations about the performance improvement and its practical significance. Compared with Complete-expert transfer, PHAROS improves by 0.67 percentage points. Additionally, although the authors provide supplementary analyses of storage overhead and the number of parameters, the comparisons with MoBLoRA and KeepLoRA remain incomplete, and the impact of increased parameters is not clear.

Overall, the authors' additional experiments have enhanced my confidence in the effectiveness of the proposed method, but its advantage over existing knowledge-reuse methods is not sufficient. I will maintain my original rating.

AuthorsOpenReview ↗

Thank you for your response

Thank you for your response. Table 6 isolates the historical transfer strategy by keeping all other settings identical. Under this fair comparison, PHAROS outperforms complete-expert and nonnegative direction transfer by 0.67% and 0.68% on average, with maximum task-level gains of 2.20% and 1.43%. Compared with no historical transfer, it provides nearly one point of average improvement and up to 3.83% on CLEVR-Math. On CoIN, direction-level transfer improves Avg.ACC from 67.96% to 69.82%. These results demonstrate that PHAROS provides a clear and substantial advantage across both benchmarks.

Regarding adaptation scale, we additionally compare PHAROS with PGP, EProj, PCLR-small, and PCLR. PHAROS achieves 70.11% Avg.ACC with 517.89M cumulative adaptation-state elements, using fewer adaptation parameters than EProj and PCLR while outperforming all four methods. We will further add matched adaptation-parameter comparisons with MoBLoRA and KeepLoRA.

We sincerely thank you again. If these clarifications and additional evaluations address your concerns, we would be grateful if you could reconsider your initial rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

We sincerely thank the Area Chair and Reviewers 6mMQ, dv5X, bEZP, and nAQs for their constructive suggestions

Response to the Area Chair and All Reviewers

We sincerely thank the Area Chair and Reviewers 6mMQ, dv5X, bEZP, and nAQs for their careful evaluation, insightful comments, and constructive suggestions. Their feedback has been invaluable in improving the clarity, rigor, and completeness of our work.

The main revisions and findings are summarized as follows:

  • Direction-level historical transfer. We added controlled comparisons among no historical transfer, complete-expert transfer, nonnegative direction transfer, raw-factor directions, and PHAROS. PHAROS achieves the highest six-task mean of 77.61%, outperforming these variants by 0.86, 0.67, 0.68, and 3.48 percentage points, respectively. These results demonstrate that independently controlling the sign and strength of operator-derived functional directions enables more effective historical transfer than treating an expert as a single unit or transferring raw factor directions.

  • Continual performance and task-wise benefits. PHAROS achieves 70.11% Avg.ACC on CoIN and 77.60% Avg. on UCIT, exceeding the strongest previous results by 4.77% and 3.58%, respectively. The expanded task-wise analysis further shows that PHAROS improves performance across diverse multimodal tasks while maintaining near-zero forgetting, demonstrating the complementary advantages of task-level preservation and direction-level transfer.

  • Experimental configuration and reproducibility. We clarified the baseline definition and reported the complete LoRA settings, learning rates, training budget, gate bound, gate initialization, and Grounding protocol. These additions provide a transparent and reproducible account of how PHAROS preserves task knowledge in private experts while adaptively transferring relevant historical knowledge to new tasks.

  • Storage, scalability, and robustness. We added a component-wise breakdown of retained adaptation states, analyzed their growth with the number of tasks, and compared the performance–memory trade-off with existing methods.

All corresponding revisions are highlighted in blue. We again sincerely thank the Area Chair and Reviewers 6mMQ, dv5X, bEZP, and nAQs for their valuable time and thoughtful feedback, which have helped us substantially strengthen the manuscript.

Dear Reviewers,

Thank you for your efforts so far. As we are halfway through the discussion period, I would like to kindly encourage you to review the authors' rebuttals and engage in further discussions, particularly if the authors have addressed your questions or concerns.

Thank you for your time and contributions!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.