- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes GeoSpec, a progressive MoE-LoRA framework for multimodal continual instruction tuning. It introduces Task-Conditioned Geometric Adaptation (TCGA) to guide expert optimization using task-specific activation geometry, and Compatibility-Guided Expert Routing (CGER) to improve expert selection by combining prototype similarity with discriminative task scores. Experiments on CoIN, UCIT, and TriGap demonstrate consistent improvements over existing methods, supported by ablation studies and computational cost comparisons.
Critical strengths
- Well-motivated methodology. The paper addresses both expert specialization and routing ambiguity. The geometry-guided adaptation strategy is theoretically motivated, and the metric ablations provide evidence for its effectiveness.
- Competitive experimental performance. GeoSpec achieves the highest average performance among the compared methods across three benchmarks. The component ablations and efficiency analysis further support its practical value.
Critical weaknesses
- Insufficient isolation of methodological contributions. TCGA combines geometric preconditioning, gradient projection, FAM optimization, and rank-constrained realization. Although the metric is ablated, the individual contributions of the remaining components are not sufficiently investigated.
- Limited robustness evaluation. The reported improvements over the strongest baselines are relatively modest on UCIT and TriGap. Results across multiple random seeds, different task orders, and explicit forgetting metrics are missing, making it difficult to assess the robustness of the improvements.
Questions for authors
- Could the authors provide additional ablations isolating the effects of gradient projection, FAM optimization, and metric-consistent rank realization?
- How sensitive is CGER to the number of historical support examples and the expert selection strategy (e.g., top-1 versus top-2 routing)?
- Could the authors report results across multiple random seeds and different task orders, together with forgetting metrics such as backward transfer (BWT)?
4 comments
Response to Reviewer r65k
For task orders, following the task-order protocol of SAME (Xie et al., 2026, App. G), we evaluate GeoSpec, SAME and CRAM on CoIN under its three task orders:
- Order 1, as in Table 3: ScienceQA → TextVQA → ImageNet → GQA → VizWiz → REC → VQAv2 → OCR-VQA
- Order 2: ScienceQA → ImageNet → GQA → OCR-VQA → REC → VQAv2 → VizWiz → TextVQA
- Order 3: OCR-VQA → VQAv2 → REC → VizWiz → GQA → ImageNet → TextVQA → ScienceQA
Table R2(c) keeps the columns in the task order of Table 3.
Table R2(c). CoIN under three task orders.
| Order | Method | ScienceQA | TextVQA | ImageNet | GQA | VizWiz | REC | VQAv2 | OCR-VQA | AVG |
|---|---|---|---|---|---|---|---|---|---|---|
| Order 1 | SAME | 77.39 | 63.15 | 97.13 | 62.13 | 49.25 | 72.68 | 66.82 | 64.54 | 69.14 |
| CRAM | 75.06 | 63.48 | 77.21 | 59.73 | 43.32 | 72.74 | 62.45 | 59.21 | 64.15 | |
| GeoSpec | 80.11 | 63.87 | 97.29 | 61.47 | 62.45 | 73.82 | 67.42 | 64.31 | 71.34 | |
| Order 2 | SAME | 77.23 | 63.40 | 97.05 | 61.09 | 49.32 | 72.78 | 66.91 | 64.38 | 69.02 |
| CRAM | 75.31 | 63.52 | 78.16 | 60.02 | 44.15 | 72.50 | 62.44 | 59.35 | 64.43 | |
| GeoSpec | 80.09 | 63.79 | 97.33 | 61.44 | 62.39 | 73.88 | 67.48 | 64.40 | 71.35 | |
| Order 3 | SAME | 77.52 | 63.31 | 97.24 | 61.76 | 49.33 | 72.37 | 66.50 | 64.33 | 69.05 |
| CRAM | 74.93 | 64.02 | 78.14 | 60.12 | 43.85 | 73.45 | 62.18 | 59.42 | 64.51 | |
| GeoSpec | 80.13 | 63.85 | 97.24 | 61.20 | 62.08 | 73.44 | 67.52 | 64.28 | 71.22 |
As shown in Table R2(c), GeoSpec ranks first under all three orders, leading SAME by 2.17–2.33 points and CRAM by 6.71–7.19 points. Its advantage therefore holds regardless of the task order.
(2) Forgetting metrics such as BWT. We report forgetting from the stage-wise accuracy matrices of GeoSpec and the reproduced HiDe-LLaVA, SAME and CRAM (App. C.2). Following MCITlib [1], MFN is the final average, i.e., the paper's AVG, MFT the accuracy on each task right after it is learned, and MAA the mean of the stage-wise averages. BWT is the mean change from each task's accuracy right after it is learned to its final accuracy, and Average Forgetting is the mean drop from each task's best accuracy before the final stage to its final accuracy [2]; both are truncated at 0, so BWT ≤ 0, Average Forgetting ≥ 0, and a method without net forgetting receives 0.
Table R3. Plasticity and forgetting.
| Benchmark | Method | MFN | MFT | MAA | BWT | Avg. Forgetting |
|---|---|---|---|---|---|---|
| UCIT | HiDe-LLaVA | 63.44 | 66.74 | 77.30 | −3.96 | 3.96 |
| UCIT | SAME | 73.51 | 75.89 | 82.93 | −2.86 | 3.08 |
| UCIT | CRAM | 74.82 | 74.80 | 82.47 | 0.00 | 0.00 |
| UCIT | GeoSpec | 75.97 | 76.02 | 83.57 | −0.06 | 0.06 |
| TriGap | HiDe-LLaVA | 32.84 | 37.45 | 36.34 | −5.12 | 5.19 |
| TriGap | SAME | 48.22 | 48.80 | 44.77 | −0.65 | 0.79 |
| TriGap | CRAM | 47.38 | 47.80 | 43.78 | −0.46 | 0.48 |
| TriGap | GeoSpec | 49.70 | 49.74 | 45.96 | −0.04 | 0.04 |
As shown in Table R3, GeoSpec achieves the highest MFT and MAA on both benchmarks with a near-zero BWT of −0.06 / −0.04 on UCIT / TriGap. The highest MFT reflects the task-specific specialization of TCGA, and the near-zero BWT reflects reliable access to the accumulated experts through CGER without test-time task identities, matching the two challenges that GeoSpec addresses.
[1] Guo et al. MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark. arXiv:2508.07307, 2025.
[2] Chaudhry et al. Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence. ECCV 2018.
Response to Reviewer r65k
Q2: Sensitivity of CGER to the number of historical support examples and to top-1 vs. top-2 routing.
A3: Thank you for the question. We answer the two parts in turn.
(1) Number of historical support examples. For each task, CGER keeps the answer-free 512-d CLIP-B image and text features of 64 training inputs, 2 KiB per sample, rather than images, instructions or answers, and uses them only to refit the router. Experts are trained on the current task only (Sec. 3.1, l. 156–157). The 64 support inputs are drawn at random with the run's seed, and we have added the support size and its memory to App. B.2. Table R4(a) lists the cumulative support samples and memory after each task.
Table R4(a). Cumulative number of retained support samples and their cumulative memory after each task.
| Benchmark | Quantity | Task 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| CoIN | Cumulative support samples | 64 | 128 | 192 | 256 | 320 | 384 | 448 | 512 | – | – |
| Cumulative memory (MiB) | 0.13 | 0.25 | 0.38 | 0.50 | 0.63 | 0.75 | 0.88 | 1.00 | – | – | |
| UCIT | Cumulative support samples | 64 | 128 | 192 | 256 | 320 | 384 | – | – | – | – |
| Cumulative memory (MiB) | 0.13 | 0.25 | 0.38 | 0.50 | 0.63 | 0.75 | – | – | – | – | |
| TriGap | Cumulative support samples | 64 | 128 | 192 | 256 | 320 | 384 | 448 | 512 | 576 | 640 |
| Cumulative memory (MiB) | 0.13 | 0.25 | 0.38 | 0.50 | 0.63 | 0.75 | 0.88 | 1.00 | 1.13 | 1.25 |
As shown in Table R4(a), the full stream retains at most 640 support samples, only 1.25 MiB on TriGap and below 0.6% of its Continual state, a negligible cost that preserves GeoSpec's efficiency advantage in Table 7. To measure the sensitivity to this budget, we vary the number of support samples per task from 16 to 256 (Table R4(b)).
Table R4(b). Sensitivity to the support budget on CoIN.
| Support per task | CoIN cache (MiB) | CoIN routing accuracy (%) | CoIN AVG |
|---|---|---|---|
| 16 | 0.25 | 72.54 | 60.20 |
| 32 | 0.50 | 85.32 | 69.44 |
| 64 (ours) | 1.00 | 92.81 | 71.34 |
| 128 | 2.00 | 92.79 | 71.30 |
| 256 | 4.00 | 92.81 | 71.32 |
As shown in Table R4(b), routing accuracy and AVG on CoIN saturate at 64 support samples per task and change by at most 0.26 points up to 256, so CGER needs only a small support set, 1.00 MiB on CoIN, and is insensitive to larger budgets.
(2) Top-1 vs. top-2 routing. On TriGap, we replace the top-2 mixing with top-1 routing, keeping the fusion weights and α unchanged (Table R5).
Table R5. TriGap with top-2 and top-1 routing.
| Routing | PMC-VQA | DocVQA | ChartQA | IconQA | InfoVQA | ArxivQA | Road | ChemVQA | FloodVQA | CLEVR | AVG |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Top-2, 0.7/0.3 (ours) | 44.37 | 45.78 | 19.76 | 72.53 | 40.55 | 92.97 | 12.93 | 27.83 | 82.21 | 58.10 | 49.70 |
| Top-1 | 44.34 | 45.62 | 19.80 | 72.31 | 40.19 | 92.44 | 12.94 | 27.80 | 82.15 | 58.11 | 49.57 |
As shown in Table R5, top-1 routing reaches 49.57 AVG versus 49.70 for the top-2 mixing, and both remain above SAME and CRAM, so GeoSpec's lead comes from task-specific experts and complementary routing evidence rather than from the mixing rule.
Dear Reviewer r65k,
As the discussion period is drawing to a close, we would like to kindly follow up on our responses. We hope that the additional results and clarifications have addressed your concerns. If any questions remain, we would be glad to clarify them while there is still time. We sincerely appreciate your time and the effort you have devoted to reviewing our paper.
Best regards,
Authors
Response to Reviewer r65k
A1: Thank you for this constructive suggestion. First, the metric itself is isolated in Table 5: with all other components fixed, replacing Λ by Identity or a permuted metric reduces AVG by 2.82–3.89 points.
Second, we remove or replace each remaining component on UCIT with the CGER router fixed, in a single run; the seed std of UCIT AVG in Table R2(b) is at most 0.27. Normalized SGD-m rescales each step to FAM's step norm, so it uses the same learning rate.
Table R1. Component ablation on UCIT.
As shown in Table R1, removing or replacing any component lowers AVG by 1.60–2.82 points, well above the seed std of 0.27, so each component contributes, and the full TCGA, which uses the task-conditioned geometry consistently to shape both the expert update and its fixed-rank realization, gives the best result. We have added this ablation to Sec. 5.5 and App. D.3.
A2: Thank you for this suggestion. We answer the two parts in turn.
(1) Multiple random seeds and task orders. We ran GeoSpec, SAME and CRAM with three random seeds on TriGap and UCIT, the two benchmarks with the smallest single-run margins, 1.48 and 1.15. A seed fixes initialization, data order and support sampling, and is shared across methods. Tables R2(a) and R2(b) report each seed and their mean.
Table R2(a). TriGap: results of each random seed and their mean; seed 42 is the run reported in Table 1.
Table R2(b). UCIT: results of each random seed and their mean; seed 42 is the run reported in Table 2.
As shown in Tables R2(a) and R2(b), GeoSpec outperforms the strongest baseline under every seed, by 1.42 on TriGap and 1.57 on UCIT on average, about five times the largest seed std of 0.28, so its state-of-the-art results are robust rather than a single-run effect.