- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes PHAROS, a framework that separates task-level preservation from direction-level transfer. Each task retains a frozen private LoRA expert and its associated multimodal projector. To support later tasks, PHAROS decomposes historical LoRA update operators into normalized rank-one directions using compact SVD. A new task learns an independent signed gate for each historical direction, alongside its own private expert. At inference time, image and text prototypes select the appropriate task-specific configuration. Experiments cover CoIN, UCIT, alternative task orders, and a larger language backbone.
Critical strengths
-
The construction of transfer directions is well motivated. The paper explicitly recognizes that equivalent LoRA factorizations can have different factor coordinates. Deriving directions from the SVD of the update operator avoids tying the interpretation to an arbitrary factorization. Normalization then separates the scale inherited from the source expert from the strength learned for a target task. This gives the direction-level transfer mechanism a clear mathematical interpretation.
-
The reported results combine strong acquisition with effective retention. PHAROS achieves 70.11 Avg.ACC, 70.39 New.ACC, and 0.32 forgetting on CoIN. On UCIT, it reaches a final Last score of 77.60, compared with 72.80 for EBLoRA. The alternative task orders and nearly flat post-acquisition trajectories provide further evidence that the preservation mechanism works consistently across the evaluated sequences.
Critical weaknesses
- The experiments do not directly test the paper’s main claim about transfer granularity. Table 5 adds task preservation, direction transfer, and projector preservation sequentially. This demonstrates that the components are useful, but it does not compare expert-level and direction-level transfer while keeping preservation, projectors, and routing unchanged. It also does not establish whether allowing negative gates is beneficial relative to nonnegative direction weights. Since projector preservation is added last, the table does not show the effect of removing direction-level transfer from the otherwise complete framework.
The gate visualizations show that different directions receive different weights, but this behavior follows from the flexibility of the parameterization. The more consequential question is whether this flexibility improves transfer relative to a complete-expert coefficient. That comparison is central to the paper’s argument.
- The strongest gains require a clearer account of experimental comparability. Appendix A.4 states that the CoIN baselines are taken from a published comparison rather than rerun in the authors’ codebase. The provenance is clearly acknowledged, but matching the backbone and task order does not fully establish that the training and evaluation settings are comparable. Appendix A.3 also omits several important settings, including the learning rate, training budget, and the gate bound and initialization.
This matters particularly for Grounding. Relative to MoBLoRA, the score increases from 32.95 to 65.00. That task contributes approximately 4.01 points to the eight-task mean improvement of 4.77 points—about 84% of the gain. A closer comparison of the Grounding setup would therefore have a substantial effect on how I interpret the main result.
Questions for authors
- With task preservation, projector preservation, and routing held fixed, how do complete-expert weighting, nonnegative direction weighting, and signed direction weighting compare?
- Could the authors provide the full training configuration, particularly the LoRA settings, learning rate, training budget, gate bound , and gate initialization? What exactly is the “Baseline” in Table 5?
- Are the Grounding data splits, training settings, coordinate formats, and evaluation procedures aligned with those used by the strongest baselines? A matched comparison would help explain the unusually large improvement on this task.
4 comments
Thank you for adding the comparison in Table 6. This is the control I was looking for: with preservation and routing held fixed, does it help to weight individual directions rather than the complete historical expert? The results favor signed direction-level transfer: its six-task mean is 77.61, compared with 76.94 for complete-expert weighting and 76.93 for nonnegative direction weighting. I also appreciate the clearer identification of the Table 5 baseline and the additional training details.
My remaining reservation is about the size and scope of this contribution. Preserving an expert as a whole while reusing selected directions from its update is a sensible distinction. Table 6 now gives direct evidence for it, but the gain over the simpler alternatives is modest in this single run; even the variant without historical transfer reaches 76.76. I see the larger Grounding gain as encouraging evidence for PHAROS as a complete system. As the response clarifies, however, the published baselines share the benchmark split and evaluation criterion but retain their own training and prediction pipelines. That comparison therefore does not tell us how much of the gain comes from direction-level transfer itself.
The revision has made me more confident that the proposed transfer mechanism helps. I remain less convinced that its advantage over simpler forms of reuse is substantial, so I am keeping my rating at 2 (weak rejection).
Thank you for your response
Thank you for your response. In Table 6, we isolate the effect of the historical transfer strategy by keeping the preservation and routing mechanisms, private rank, data, task order, seed, training budget, and first-task checkpoint identical across all variants. Under this fair setting, signed direction-level transfer raises the six-task mean from 76.76% to 77.61% on UCIT, delivering an average improvement of nearly 1% and a gain of up to 3.83% on CLEVR-Math. The component analysis on CoIN shows a consistent benefit, where direction-level transfer increases the average performance from 67.96% to 69.82%, an improvement of nearly 2%. These results demonstrate that our direction-level transfer provides a clear and substantial advantage over simpler forms of reuse across both benchmarks.
To further verify that this benefit generalizes beyond PHAROS, we will add experiments that incorporate signed direction-level transfer into representative continual-learning baselines under matched settings.
We sincerely thank you again for the constructive feedback. If any concerns remain, we would be happy to discuss them further.
We sincerely thank you again
If our response and the additional experiments adequately address your concerns, we would sincerely appreciate your reconsideration of the current rating. We would also be happy to provide further clarification if any concerns remain.
We sincerely thank reviewer nAQs for the careful and constructive assessment
Response to Reviewer nAQs
We sincerely thank the reviewer for the careful and constructive assessment. The questions were especially valuable for strengthening our controlled comparisons, training transparency, and Grounding protocol reporting.
1. Complete-expert, nonnegative, and signed weighting
The controls share private rank 8, projector preservation, routing, data, task order, seed, training budget, and first-task checkpoint before separate continual-training chains. Complete-expert weighting learns one signed coefficient per source expert and module, while nonnegative and signed weighting independently gate normalized directions.
Mean is the arithmetic mean of the displayed scores, not trajectory-level UCIT Avg. Signed direction weighting improves the mean by 0.67 points over complete-expert weighting and 0.68 over nonnegative directions. It improves four tasks and ties two relative to nonnegative directions. Relative to complete-expert weighting, it improves four tasks, ties ImageNet-R, and is 0.31 points lower on Viz-cap. These task-wise differences show the value of independently controlling the sign and strength of historical directions.
2. Training configuration and Table 5 baseline
We appreciate the request for a clearer baseline definition. Table 5 uses our HiDe-LLaVA run [1], rather than PHAROS without historical transfer. Its score matrix gives 54.05 Avg.ACC and 63.08 New.ACC, and the revised table labels it HiDe-LLaVA (Baseline).
The controlled comparisons use the following configuration:
All transfer controls reuse the same first-task checkpoint and subsequently follow their own causal checkpoint chains.
These settings are now reported in the Training Configuration subsection of the appendix.
3. Grounding protocol and the large improvement
We agree that the large Grounding gain merits careful protocol reporting. The main table reports 65.00 for PHAROS, 36.59 for EProj [3], and 32.95 for MoBLoRA [4], corresponding to differences of 28.41 and 32.05 points. EProj is the strongest Grounding baseline, while MoBLoRA is the strongest baseline in overall Avg.ACC. These values compare complete methods.
PHAROS follows the official CoIN Grounding split with 55,885 training and 30,969 test records [2], normalized xyxy coordinates in the square-padded image frame, and the IoU>0.5 success criterion. EProj and MoBLoRA are compared through their published results on the same CoIN split and task definition. PHAROS uses one epoch, private rank 8, effective batch 48, greedy decoding, and the
vicuna_v1template.This establishes alignment in the benchmark split, coordinate target, and evaluation criterion. The published baselines retain their original optimization settings and prediction pipelines, so the comparison is benchmark-matched rather than a strictly training- and executable-scorer-matched rerun. We have made this scope explicit and interpret the 65.00 score as a complete-method result rather than attributing the full gap to direction-level transfer alone.
The corresponding manuscript changes are highlighted in blue in the Table 5 baseline label, the Historical Transfer Controls subsection, and the appendix Training Configuration.
References
[1] Guo et al. HiDe-LLaVA. ACL, 2025.
[2] Chen et al. CoIN. NeurIPS Datasets and Benchmarks, 2024.
[3] He et al. Continual Instruction Tuning for Large Multimodal Models. 2023.
[4] Chen et al. From Experts to Bases (MoBLoRA). ACL, 2026.