acceptodds
Under review as a conference paper at ICLR 2027

GeoSpec: Geometry-Guided Expert Specialization for Multimodal Continual Instruction Tuning

Abstract

Multimodal continual instruction tuning requires multimodal large language models to acquire new tasks sequentially while preserving previously learned capabilities. Progressive expert-based methods can isolate past experts from subsequent updates, but two challenges remain: how to adapt each new expert to the functional demands of the current task, and how to reliably access the accumulated experts without test-time task identities. We introduce GeoSpec, a progressive MoE-LoRA framework that addresses these challenges through geometry-guided expert adaptation and compatibility-guided routing. For each arriving task, Task-Conditioned Geometric Adaptation (TCGA) derives a task-conditioned local-output geometry from current-task activations and uses this geometry consistently to shape both expert updates and their fixed-rank realization. At inference, Compatibility-Guided Expert Routing combines task-prototype affinity with discriminative task evidence learned from answer-free, task-labeled support inputs, providing complementary signals for expert selection under task ambiguity. GeoSpec achieves the highest final AVG among the compared methods on all three evaluated MCIT benchmarks, reaching 71.34 on CoIN, 75.97 on UCIT, and 49.70 on TriGap.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual learning, Multimodal Continual Instruction Tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_r65kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GeoSpec, a progressive MoE-LoRA framework for multimodal continual instruction tuning. It introduces Task-Conditioned Geometric Adaptation (TCGA) to guide expert optimization using task-specific activation geometry, and Compatibility-Guided Expert Routing (CGER) to improve expert selection by combining prototype similarity with discriminative task scores. Experiments on CoIN, UCIT, and TriGap demonstrate consistent improvements over existing methods, supported by ablation studies and computational cost comparisons.

Critical strengths
  1. Well-motivated methodology. The paper addresses both expert specialization and routing ambiguity. The geometry-guided adaptation strategy is theoretically motivated, and the metric ablations provide evidence for its effectiveness.
  2. Competitive experimental performance. GeoSpec achieves the highest average performance among the compared methods across three benchmarks. The component ablations and efficiency analysis further support its practical value.
Critical weaknesses
  1. Insufficient isolation of methodological contributions. TCGA combines geometric preconditioning, gradient projection, FAM optimization, and rank-constrained realization. Although the metric is ablated, the individual contributions of the remaining components are not sufficiently investigated.
  2. Limited robustness evaluation. The reported improvements over the strongest baselines are relatively modest on UCIT and TriGap. Results across multiple random seeds, different task orders, and explicit forgetting metrics are missing, making it difficult to assess the robustness of the improvements.
Questions for authors
  1. Could the authors provide additional ablations isolating the effects of gradient projection, FAM optimization, and metric-consistent rank realization?
  2. How sensitive is CGER to the number of historical support examples and the expert selection strategy (e.g., top-1 versus top-2 routing)?
  3. Could the authors report results across multiple random seeds and different task orders, together with forgetting metrics such as backward transfer (BWT)?
4 comments
AuthorsOpenReview ↗

Response to Reviewer r65k

W1 & Q1: Isolating the effects of gradient projection, FAM optimization and metric-consistent rank realization.

A1: Thank you for this constructive suggestion. First, the metric itself is isolated in Table 5: with all other components fixed, replacing Λ by Identity or a permuted metric reduces AVG by 2.82–3.89 points.

Second, we remove or replace each remaining component on UCIT with the CGER router fixed, in a single run; the seed std of UCIT AVG in Table R2(b) is at most 0.27. Normalized SGD-m rescales each step to FAM's step norm, so it uses the same learning rate.

Table R1. Component ablation on UCIT.

VariantProjectionDirectionRank-r realizationMetricUCITΔ vs. TCGA
TCGA✓FAMΛ-weightedΛ75.97–
w/o projection✗FAMΛ-weightedΛ73.30−2.67
Euclidean realization✓FAMEuclideanΛ74.37−1.60
SGD-m direction✓SGD-mΛ-weightedΛ74.33−1.64
Identity, Table 5✓FAMEuclideanI73.15−2.82

As shown in Table R1, removing or replacing any component lowers AVG by 1.60–2.82 points, well above the seed std of 0.27, so each component contributes, and the full TCGA, which uses the task-conditioned geometry consistently to shape both the expert update and its fixed-rank realization, gives the best result. We have added this ablation to Sec. 5.5 and App. D.3.

W2 & Q3: Results across random seeds and task orders, with forgetting metrics such as BWT.

A2: Thank you for this suggestion. We answer the two parts in turn.

(1) Multiple random seeds and task orders. We ran GeoSpec, SAME and CRAM with three random seeds on TriGap and UCIT, the two benchmarks with the smallest single-run margins, 1.48 and 1.15. A seed fixes initialization, data order and support sampling, and is shared across methods. Tables R2(a) and R2(b) report each seed and their mean.

Table R2(a). TriGap: results of each random seed and their mean; seed 42 is the run reported in Table 1.

SeedMethodPMC-VQADocVQAChartQAIconQAInfoVQAArxivQARoadChemVQAFloodVQACLEVRAVG
42SAME44.0044.2018.3666.3039.6692.9011.5724.0081.2359.9348.22
CRAM42.9043.7716.4865.1739.0992.0013.0322.7380.1458.5347.38
GeoSpec44.3745.7819.7672.5340.5592.9712.9327.8382.2158.1049.70
43SAME43.1744.0118.1265.8740.0493.3011.5721.1781.5159.1747.79
CRAM43.1343.3216.8464.7738.9491.7713.0022.5080.4756.8747.16
GeoSpec42.8345.7320.4870.5340.4691.0012.5728.6381.1858.3349.17
44SAME44.5343.3218.4865.0739.6093.6010.5322.4781.9459.1347.87
CRAM42.7043.5617.0063.2338.9393.4012.6723.6380.2056.4747.18
GeoSpec43.7745.5421.2069.8740.2193.2013.0329.1781.2955.3349.26
MeanSAME43.9043.8418.3265.7539.7793.2711.2222.5581.5659.4147.96
CRAM42.9143.5516.7764.3938.9992.3912.9022.9580.2757.2947.24
GeoSpec43.6645.6820.4870.9840.4192.3912.8428.5481.5657.2549.38

Table R2(b). UCIT: results of each random seed and their mean; seed 42 is the run reported in Table 2.

SeedMethodImageNet-RArxivQAVizWiz-CapIconQACLEVR-MathFlickr30kAVG
42SAME90.3392.7347.8479.6773.1057.3973.51
CRAM89.1092.9060.7980.1368.8057.2174.82
GeoSpec90.9791.8360.7184.3770.1357.8075.97
43SAME90.2394.0347.6379.2772.7357.5373.57
CRAM89.3092.6360.9778.9069.3357.1074.71
GeoSpec90.7792.2761.8784.3071.4057.9276.42
44SAME90.5091.8047.8978.9772.1057.1673.07
CRAM89.0391.8761.2777.0770.0057.3374.43
GeoSpec91.0790.9361.4984.9771.1058.0376.27
MeanSAME90.3592.8547.7979.3072.6457.3673.38
CRAM89.1492.4761.0178.7069.3857.2174.65
GeoSpec90.9491.6861.3684.5570.8857.9276.22

As shown in Tables R2(a) and R2(b), GeoSpec outperforms the strongest baseline under every seed, by 1.42 on TriGap and 1.57 on UCIT on average, about five times the largest seed std of 0.28, so its state-of-the-art results are robust rather than a single-run effect.

AuthorsOpenReview ↗

Response to Reviewer r65k

For task orders, following the task-order protocol of SAME (Xie et al., 2026, App. G), we evaluate GeoSpec, SAME and CRAM on CoIN under its three task orders:

  • Order 1, as in Table 3: ScienceQA → TextVQA → ImageNet → GQA → VizWiz → REC → VQAv2 → OCR-VQA
  • Order 2: ScienceQA → ImageNet → GQA → OCR-VQA → REC → VQAv2 → VizWiz → TextVQA
  • Order 3: OCR-VQA → VQAv2 → REC → VizWiz → GQA → ImageNet → TextVQA → ScienceQA

Table R2(c) keeps the columns in the task order of Table 3.

Table R2(c). CoIN under three task orders.

OrderMethodScienceQATextVQAImageNetGQAVizWizRECVQAv2OCR-VQAAVG
Order 1SAME77.3963.1597.1362.1349.2572.6866.8264.5469.14
CRAM75.0663.4877.2159.7343.3272.7462.4559.2164.15
GeoSpec80.1163.8797.2961.4762.4573.8267.4264.3171.34
Order 2SAME77.2363.4097.0561.0949.3272.7866.9164.3869.02
CRAM75.3163.5278.1660.0244.1572.5062.4459.3564.43
GeoSpec80.0963.7997.3361.4462.3973.8867.4864.4071.35
Order 3SAME77.5263.3197.2461.7649.3372.3766.5064.3369.05
CRAM74.9364.0278.1460.1243.8573.4562.1859.4264.51
GeoSpec80.1363.8597.2461.2062.0873.4467.5264.2871.22

As shown in Table R2(c), GeoSpec ranks first under all three orders, leading SAME by 2.17–2.33 points and CRAM by 6.71–7.19 points. Its advantage therefore holds regardless of the task order.

(2) Forgetting metrics such as BWT. We report forgetting from the stage-wise accuracy matrices of GeoSpec and the reproduced HiDe-LLaVA, SAME and CRAM (App. C.2). Following MCITlib [1], MFN is the final average, i.e., the paper's AVG, MFT the accuracy on each task right after it is learned, and MAA the mean of the stage-wise averages. BWT is the mean change from each task's accuracy right after it is learned to its final accuracy, and Average Forgetting is the mean drop from each task's best accuracy before the final stage to its final accuracy [2]; both are truncated at 0, so BWT ≤ 0, Average Forgetting ≥ 0, and a method without net forgetting receives 0.

Table R3. Plasticity and forgetting.

BenchmarkMethodMFNMFTMAABWTAvg. Forgetting
UCITHiDe-LLaVA63.4466.7477.30−3.963.96
UCITSAME73.5175.8982.93−2.863.08
UCITCRAM74.8274.8082.470.000.00
UCITGeoSpec75.9776.0283.57−0.060.06
TriGapHiDe-LLaVA32.8437.4536.34−5.125.19
TriGapSAME48.2248.8044.77−0.650.79
TriGapCRAM47.3847.8043.78−0.460.48
TriGapGeoSpec49.7049.7445.96−0.040.04

As shown in Table R3, GeoSpec achieves the highest MFT and MAA on both benchmarks with a near-zero BWT of −0.06 / −0.04 on UCIT / TriGap. The highest MFT reflects the task-specific specialization of TCGA, and the near-zero BWT reflects reliable access to the accumulated experts through CGER without test-time task identities, matching the two challenges that GeoSpec addresses.

[1] Guo et al. MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark. arXiv:2508.07307, 2025.

[2] Chaudhry et al. Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence. ECCV 2018.

AuthorsOpenReview ↗

Response to Reviewer r65k

Q2: Sensitivity of CGER to the number of historical support examples and to top-1 vs. top-2 routing.

A3: Thank you for the question. We answer the two parts in turn.

(1) Number of historical support examples. For each task, CGER keeps the answer-free 512-d CLIP-B image and text features of 64 training inputs, 2 KiB per sample, rather than images, instructions or answers, and uses them only to refit the router. Experts are trained on the current task only (Sec. 3.1, l. 156–157). The 64 support inputs are drawn at random with the run's seed, and we have added the support size and its memory to App. B.2. Table R4(a) lists the cumulative support samples and memory after each task.

Table R4(a). Cumulative number of retained support samples and their cumulative memory after each task.

BenchmarkQuantityTask 12345678910
CoINCumulative support samples64128192256320384448512––
Cumulative memory (MiB)0.130.250.380.500.630.750.881.00––
UCITCumulative support samples64128192256320384––––
Cumulative memory (MiB)0.130.250.380.500.630.75––––
TriGapCumulative support samples64128192256320384448512576640
Cumulative memory (MiB)0.130.250.380.500.630.750.881.001.131.25

As shown in Table R4(a), the full stream retains at most 640 support samples, only 1.25 MiB on TriGap and below 0.6% of its Continual state, a negligible cost that preserves GeoSpec's efficiency advantage in Table 7. To measure the sensitivity to this budget, we vary the number of support samples per task from 16 to 256 (Table R4(b)).

Table R4(b). Sensitivity to the support budget on CoIN.

Support per taskCoIN cache (MiB)CoIN routing accuracy (%)CoIN AVG
160.2572.5460.20
320.5085.3269.44
64 (ours)1.0092.8171.34
1282.0092.7971.30
2564.0092.8171.32

As shown in Table R4(b), routing accuracy and AVG on CoIN saturate at 64 support samples per task and change by at most 0.26 points up to 256, so CGER needs only a small support set, 1.00 MiB on CoIN, and is insensitive to larger budgets.

(2) Top-1 vs. top-2 routing. On TriGap, we replace the top-2 mixing with top-1 routing, keeping the fusion weights and α unchanged (Table R5).

Table R5. TriGap with top-2 and top-1 routing.

RoutingPMC-VQADocVQAChartQAIconQAInfoVQAArxivQARoadChemVQAFloodVQACLEVRAVG
Top-2, 0.7/0.3 (ours)44.3745.7819.7672.5340.5592.9712.9327.8382.2158.1049.70
Top-144.3445.6219.8072.3140.1992.4412.9427.8082.1558.1149.57

As shown in Table R5, top-1 routing reaches 49.57 AVG versus 49.70 for the top-2 mixing, and both remain above SAME and CRAM, so GeoSpec's lead comes from task-specific experts and complementary routing evidence rather than from the mixing rule.

Dear Reviewer r65k,

As the discussion period is drawing to a close, we would like to kindly follow up on our responses. We hope that the additional results and clarifications have addressed your concerns. If any questions remain, we would be glad to clarify them while there is still time. We sincerely appreciate your time and the effort you have devoted to reviewing our paper.

Best regards,

Authors

Reviewer_rqjgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

a vision language assistant has to learn a stream of new skills like reading charts answering science questions captioning photos or locating objects without forgetting the old ones. geospec gives every new skill its own small add on module like a lora expert and locks it away once its trained so later skills cant overwrite it.

the paper adds two main ideas. first is geometry guided training so when training a new module they judge each candidate change by how much it actually changes the layers outputs on this skills own inputs instead of just the raw size of the weight change. second is compatability guided routing where at test time a seperate small classifier built on clip image and text features guesses which skill a question belongs to and switches on the two most likely modules like 70% and 30%.

Critical strengths
  • the ablations are really well designed. table 4 and 8 and the identity vs permuted metric control in table 5 and router variants in table 6 are super clean. the permuted control is honestly the most convincing evidence that the gain is tied to activation aligned diagonal stastistics rather than just arbitrary non uniform scaling. this matters alot cause its the only evidence that isolates tcgas mechanism.

  • also it has a strong practical profile. geospec has the best final avg on three mcit benchmarks with lower peak gpu memory like 25.1 vs 32.5 gib and fewer per task trainable parameters than same and cram in table 7. if the results actually hold under seeds and fair baselines the method is pretty simple to deploy.

Critical weaknesses

the evidence doesnt really support the sota claim stastistically though. there is only one run with no variance and no task orders and the margins are just 1.15 to 2.20 avg points. the baselines mix copied numbers with in house reproductions that visibly failed like sigma scores 0.00 on scienceqa vizwiz and vqav2 in coin. protoadas trigap avg of 47.23 doesnt even match its per task mean of 46.93. plus the claimed mechanisms arent really demonstrated. with fixed math and no tangent projection fams column profile gives math which cancels the math in math exactly. after projection and the rank retraction the realised updates under math and under math still have a cosine of about 0.65 in a one step simulation and about 0.89 or 0.41 without fams row and column scaling when the step is small relative to the current adapter so a big part of the task conditioned direction never even reaches the expert. continual learning is basicly reduced to task id prediction from cached features and labels of all past tasks. on benchmarks where instruction templates leak task identity routing is near saturated from yi et al 2026 the flex paper they only cite for ridge routing so routing ambiguity just isnt convincingly addressed.

-Bogachev et al. 2026, cited in Sec. 4.1 i think this is the Eckart–Young (1936) / Mirsky theorem. Eckart & Young is never cited.

Questions for authors
  • check table resuls for ProtoAda on TriGap (Table 1): reported 47.23, mean of the ten printed cells 46.93

  • please report mean and std over 3 or more seeds and 2 or more task orders for geospec same and cram on all three benchmarks. with fixed math and no projection math implies math which cancels math in math so which part of the task conditioned geometry actually survives in the realised update of eq 6. please report tcga with fam replaced by adam or sgd and the identity variant with a learning rate sweep. what is math per task and the memory of the cached support features anyway. what is avg under task oracle routing and how do the macro match in table 6 and avg change when instruction templates are paraphrased or normalised like on flex from yi et al 2026. how were the fusion weights like 0.20 0.30 0.25 0.25 the 0.7 0.3 mixing and alpha even chosen. please add sensitivity and top 1 results. why does sigma obtain 0.00 on scienceqa vizwiz and vqav2 and why is same re run on coin instead of just using its published numbers. please also correct protoadas trigap avg. finally maybe compare tcga with lora pro riemannion and lora rite under the same cger router and report forgetting and bwt.

  • cross check these citatation they are arxiv but accpeted in conferences (outdated venues) : SAME (Xie et al.), CRAM (Tang et al. 2026b), SiGMA (Park & Kim), Hyper-LLaVA (Xu et al.), CoRe-MoE (Liu et al. 2026a), Zhang et al. 2026a

  • Kingma & Ba "2017"; Hu et al. LoRA "2021"Outdated / incomplete No venues given (Adam: ICLR 2015; LoRA: ICLR 2022). The tables say "FT-LoRA (ICLR'22)" while the bibliography says 2021.

5 comments
AuthorsOpenReview ↗

Response to Reviewer rqjg

W1 & Q1: Statistical support for the comparison with multiple seeds and task orders for GeoSpec, SAME and CRAM.

A1: Thank you for raising this important point. We agree that multiple runs give a more reliable comparison, and answer the two parts in turn.

(1) Multiple random seeds. We ran GeoSpec, SAME and CRAM with three random seeds on TriGap and UCIT, the two benchmarks with the smallest single-run margins, 1.48 and 1.15. A seed fixes initialization, data order and support sampling, and is shared across methods.

Table R1(a). TriGap: results of each random seed.

SeedMethodPMC-VQADocVQAChartQAIconQAInfoVQAArxivQARoadChemVQAFloodVQACLEVRAVG
42SAME44.0044.2018.3666.3039.6692.9011.5724.0081.2359.9348.22
CRAM42.9043.7716.4865.1739.0992.0013.0322.7380.1458.5347.38
GeoSpec44.3745.7819.7672.5340.5592.9712.9327.8382.2158.1049.70
43SAME43.1744.0118.1265.8740.0493.3011.5721.1781.5159.1747.79
CRAM43.1343.3216.8464.7738.9491.7713.0022.5080.4756.8747.16
GeoSpec42.8345.7320.4870.5340.4691.0012.5728.6381.1858.3349.17
44SAME44.5343.3218.4865.0739.6093.6010.5322.4781.9459.1347.87
CRAM42.7043.5617.0063.2338.9393.4012.6723.6380.2056.4747.18
GeoSpec43.7745.5421.2069.8740.2193.2013.0329.1781.2955.3349.26
MeanSAME43.9043.8418.3265.7539.7793.2711.2222.5581.5659.4147.96
CRAM42.9143.5516.7764.3938.9992.3912.9022.9580.2757.2947.24
GeoSpec43.6645.6820.4870.9840.4192.3912.8428.5481.5657.2549.38

Table R1(b). UCIT: results of each random seed.

SeedMethodImageNet-RArxivQAVizWiz-CapIconQACLEVR-MathFlickr30kAVG
42SAME90.3392.7347.8479.6773.1057.3973.51
CRAM89.1092.9060.7980.1368.8057.2174.82
GeoSpec90.9791.8360.7184.3770.1357.8075.97
43SAME90.2394.0347.6379.2772.7357.5373.57
CRAM89.3092.6360.9778.9069.3357.1074.71
GeoSpec90.7792.2761.8784.3071.4057.9276.42
44SAME90.5091.8047.8978.9772.1057.1673.07
CRAM89.0391.8761.2777.0770.0057.3374.43
GeoSpec91.0790.9361.4984.9771.1058.0376.27
MeanSAME90.3592.8547.7979.3072.6457.3673.38
CRAM89.1492.4761.0178.7069.3857.2174.65
GeoSpec90.9491.6861.3684.5570.8857.9276.22

As shown in Tables R1(a) and R1(b), GeoSpec outperforms the strongest baseline under every seed, by 1.42 on TriGap and 1.57 on UCIT on average, about five times the largest seed std of 0.28, so its state-of-the-art results are robust rather than a single-run effect.

(2) Multiple task orders. Following the task-order protocol of SAME, we evaluate GeoSpec, SAME and CRAM on CoIN under its three task orders:

  • Order 1, as in Table 3: ScienceQA → TextVQA → ImageNet → GQA → VizWiz → REC → VQAv2 → OCR-VQA
  • Order 2: ScienceQA → ImageNet → GQA → OCR-VQA → REC → VQAv2 → VizWiz → TextVQA
  • Order 3: OCR-VQA → VQAv2 → REC → VizWiz → GQA → ImageNet → TextVQA → ScienceQA

Table R1(c). CoIN under three task orders.

OrderMethodScienceQATextVQAImageNetGQAVizWizRECVQAv2OCR-VQAAVG
Order 1SAME77.3963.1597.1362.1349.2572.6866.8264.5469.14
CRAM75.0663.4877.2159.7343.3272.7462.4559.2164.15
GeoSpec80.1163.8797.2961.4762.4573.8267.4264.3171.34
Order 2SAME77.2363.4097.0561.0949.3272.7866.9164.3869.02
CRAM75.3163.5278.1660.0244.1572.5062.4459.3564.43
GeoSpec80.0963.7997.3361.4462.3973.8867.4864.4071.35
Order 3SAME77.5263.3197.2461.7649.3372.3766.5064.3369.05
CRAM74.9364.0278.1460.1243.8573.4562.1859.4264.51
GeoSpec80.1363.8597.2461.2062.0873.4467.5264.2871.22

As shown in Table R1(c), GeoSpec ranks first under all three orders, leading SAME by 2.17–2.33 points and CRAM by 6.71–7.19 points. Its advantage therefore holds regardless of the task order.

AuthorsOpenReview ↗

Response to Reviewer rqjg

W2 & Q2: SiGMA's 0.00 scores on ScienceQA, VizWiz and VQAv2 in CoIN; ProtoAda's TriGap AVG of 47.23 does not match its per-task mean of 46.93; and why SAME is re-run on CoIN instead of using its published numbers.

A2: Thank you for checking the tables so carefully. We answer the three points in turn.

(1) SiGMA's 0.00 scores on CoIN. SiGMA does not report CoIN results, and its released configuration was not designed for this benchmark, so these scores do not reflect its capability and we have removed the row from Table 3.

(2) ProtoAda's TriGap AVG. The ProtoAda row is taken from Table 1 of the ProtoAda paper, whose reported AVG of 47.23 was miscalculated; the mean of its ten per-task scores is 46.93. We have corrected the AVG to 46.93 in Table 1, and the ranking is unchanged. All other 30 AVG entries in Tables 1–3 match their per-task means.

(3) Re-running SAME instead of using its published numbers. For a fairer comparison, we reproduce SAME and CRAM, the two strongest and most closely related progressive MoE-LoRA methods, with the same training settings as GeoSpec, including batch size, GPU hardware and random seeds; the backbone, data and evaluators follow the original papers, and other baselines use their published numbers as described in App. B.3. Table R2 lists the published and reproduced results.

Table R2. Published and reproduced AVG of SAME and CRAM; published values follow App. B.3, and – = not reported.

MethodSourceCoINUCITTriGap
SAMEPublished66.8273.0746.53
SAMEReproduced (ours)69.1473.5148.22
CRAMPublished–74.7347.79
CRAMReproduced (ours)64.1574.8247.38
GeoSpecOurs71.3475.9749.70

As shown in Table R2, GeoSpec outperforms both the published and the reproduced results of SAME and CRAM wherever they are available, so its lead does not depend on our reproduction.

W3 & Q3: With fixed Λ and no tangent projection, FAM's column profile cancels Λ. Which part of the task-conditioned geometry survives in Eq. (6)? FAM replaced by Adam or SGD, and Identity with a learning-rate sweep.

A3: Thank you for this careful analysis. We agree with the observation for the unprojected case and answer the three parts in turn.

(1) Which part of the task-conditioned geometry survives. Table 5 of the paper shows that Λ is not cancelled end to end. All three variants share FAM, the projection, the rank-r realization, the learning rate and the fitted CGER router; only the metric differs.

Table 5 of the paper. TCGA metric ablation with the fitted CGER router fixed.

MetricUCIT AVGTriGap AVG
Identity73.1546.85
Permuted72.0846.64
TCGA75.9749.70

Since Permuted keeps the same diagonal values on other coordinates, its gap shows that the gain depends on the alignment of Λ with the activations. The cancellation requires an unprojected gradient and a Λ fixed over FAM's moment windows, as stated in Remark 1 of App. A.5. The key difference in TCGA is the projection: the gradient is projected onto the first-order LoRA-factor space (Eq. 15) before FAM, so FAM receives the Λ-Riemannian gradient

gW=(I−Πout) ∇L A⊤(AΛA⊤)†A+Πout ∇L Λ−1g_W=(I-\Pi_{\rm out})\,\nabla\mathcal{L}\,A^\top(A\Lambda A^\top)^{\dagger}A+\Pi_{\rm out}\,\nabla\mathcal{L}\,\Lambda^{-1}

The second term is the unprojected case of Remark 1; the first keeps Λ through the Λ-orthogonal projection onto the expert's input subspace row(A), which a diagonal rescaling cannot remove. This holds even when Λ is fixed, so the surviving term does not rely on Λ changing during training. We have revised Sec. 4.1 and App. A.5 accordingly.

(2) FAM replaced by Adam or SGD. TCGA forms its update in the full weight space (Eq. 6). FAM keeps a rank-4r first moment and row–column second-moment statistics, so its optimizer state is O(r(m+d)) per layer and no dense m × d matrix is formed (App. A.2, Table 28), whereas Adam on the same direction needs dense m × d moments for every adapted layer. This memory efficiency is why TCGA uses FAM; as stated in Sec. 4.1, our contribution is the task-conditioned geometry rather than the optimizer. We replace FAM with Adam and with SGD with momentum, i.e., FAM's rank-4r momentum without row–column or second-moment scaling, rescaled to FAM's step norm, keeping Λ and all other settings unchanged (Table R3).

Table R3. TCGA on UCIT with FAM replaced by Adam or SGD-m.

Direction ruleUCIT AVG
FAM (ours)75.97
Adam75.42
SGD-m, normalized74.33

As shown in Table R3, FAM matches or exceeds both replacements without dense m × d moments, and both replacements still exceed the 73.15 of the Identity metric in Table 5. The gain of TCGA therefore comes from the task-conditioned geometry, our core contribution, while FAM adds memory efficiency on top of it.

AuthorsOpenReview ↗

Response to Reviewer rqjg

(3) Identity with a learning-rate sweep. We train the Identity variant with four learning rates, including the 2e-4 used in the paper (Table R4(a)), and then compare its best setting with the other variants of Table 5 (Table R4(b)).

Table R4(a). Learning-rate sweep of the Identity variant.

Benchmark1e-42e-4 (paper)4e-48e-4
UCIT71.2873.1573.6573.20
TriGap46.0146.8547.5247.23

Table R4(b). Table 5 of the paper with the Identity variant at its best learning rate.

MetricUCIT AVGTriGap AVG
Identity, 4e-473.6547.52
Permuted72.0846.64
TCGA75.9749.70

As shown in Tables R4(a) and R4(b), even at its best learning rate the Identity variant stays 2.32 / 2.18 points below TCGA on UCIT / TriGap, so the gain of TCGA comes from the task-conditioned metric rather than from a larger effective step size.

W4 & Q4: Routing as task-ID prediction from cached features and labels, and near-saturated routing when templates leak task identity. What is |S_e| per task and the memory of the cached support features? What is AVG under task-oracle routing? How do the macro match and AVG change when instruction templates are normalized as in FLEX (Yi et al., 2026)?

A4: Thank you for this question. We answer the three parts in turn.

(1) |S_e| per task and the memory of the cached support features. Like HiDe-LLaVA and CRAM, which store one centroid per task, CGER selects frozen task experts from stored task-level features. For each task, it keeps the answer-free CLIP-B image and text features of 64 training inputs, 2 KiB per sample, and uses them only to refit the router, while experts are trained on the current task only (Sec. 3.1, l. 156–157). This cache is counted in the Continual state of Table 7: the 223.4 MiB on TriGap already includes its 1.25 MiB, and we have revised the caption to state this. Table R5(a) lists the cumulative support samples and memory after each task, which we have added to App. B.2.

Table R5(a). Cumulative number of retained support samples and their cumulative memory after each task.

BenchmarkQuantityTask 12345678910
CoINCumulative support samples64128192256320384448512––
Cumulative memory (MiB)0.130.250.380.500.630.750.881.00––
UCITCumulative support samples64128192256320384––––
Cumulative memory (MiB)0.130.250.380.500.630.75––––
TriGapCumulative support samples64128192256320384448512576640
Cumulative memory (MiB)0.130.250.380.500.630.750.881.001.131.25

As shown in Table R5(a), the full stream retains at most 640 support samples, only 1.25 MiB on TriGap and below 0.6% of its Continual state, a negligible cost that preserves GeoSpec's efficiency advantage in Table 7.

(2) AVG under task-oracle routing. Under task-oracle routing, each test sample uses the expert of its own task (Table R5(b)).

Table R5(b). GeoSpec under CGER and under task-oracle routing.

CoINScienceQATextVQAImageNetGQAVizWizRECVQAv2OCR-VQAAVG
CGER (ours)80.1163.8797.2961.4762.4573.8267.4264.3171.34
Task-oracle routing80.4764.0197.3262.7262.6073.9067.4564.2871.59
UCITImageNet-RArxivQAVizWiz-CapIconQACLEVR-MathFlickr30kAVG
CGER (ours)90.9791.8360.7184.3770.1357.8075.97
Task-oracle routing91.0791.9760.7484.3870.2757.9976.07
TriGapPMC-VQADocVQAChartQAIconQAInfoVQAArxivQARoadChemVQAFloodVQACLEVRAVG
CGER (ours)44.3745.7819.7672.5340.5592.9712.9327.8382.2158.1049.70
Task-oracle routing44.4345.9719.8472.5040.4393.2612.9827.9382.1658.1149.76

As shown in Table R5(b), CGER stays within 0.25 / 0.10 / 0.06 points of task-oracle routing on CoIN / UCIT / TriGap, so CGER accesses the accumulated experts almost as well as known task identities, which addresses routing ambiguity.

Reviewer_Rg5SOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GeoSpec, a progressive MoE-LoRA framework for multimodal continual instruction tuning. GeoSpec aims to improve task-specific expert adaptation and expert selection without requiring task identities at inference time.

The proposed framework consists of two main components. Task-Conditioned Geometric Adaptation (TCGA) uses current-task activation statistics to guide expert updates and fixed-rank realization. Compatibility-Guided Expert Routing (CGER) combines task-prototype similarity with discriminative task scores to select appropriate experts.

Experiments on three multimodal continual instruction tuning benchmarks, CoIN, UCIT, and TriGap, show that GeoSpec achieves the highest final average performance among the compared methods. The authors also provide component-level ablations and computational cost analysis to evaluate the proposed framework.

Critical strengths

1. Well-motivated problem formulation.

The paper addresses two important challenges in multimodal continual instruction tuning: task-specific expert specialization and routing ambiguity. The proposed framework provides a meaningful approach to improving both expert adaptation and task-ID-free expert selection.

2. Comprehensive experimental evaluation and ablation studies.

The experiments cover three established benchmarks and compare GeoSpec with several recent methods. The authors also provide component-level ablations that separately evaluate TCGA and CGER, as well as additional experiments on the task-conditioned metric and routing signals. These results provide useful evidence supporting the proposed design.

Critical weaknesses

1. Insufficient analysis of catastrophic forgetting.

My main concern is that the experimental evaluation primarily reports the final average performance (AVG) after completing all tasks. Although this metric is useful for evaluating overall continual learning performance, it does not directly characterize how previously learned tasks are affected as new tasks arrive.

Since mitigating catastrophic forgetting is an important motivation of the paper, additional analysis of task-wise performance throughout the learning sequence would strengthen the evaluation. In particular, reporting Average Forgetting or Backward Transfer (BWT) would help distinguish improvements in new-task learning from the preservation of previously learned capabilities.

2. Clarification of historical support usage.

CGER refits the router using task-labeled support features from previously observed tasks. However, the number of retained support samples and the associated memory overhead are not sufficiently quantified. Clarifying these details would help assess the practical requirements of the proposed routing mechanism.

Questions for authors

Q1. Continual learning and forgetting analysis.

Could the authors provide task-wise performance results at different stages of the continual learning process, together with Average Forgetting or BWT metrics? This would help clarify whether GeoSpec primarily improves new-task learning, reduces forgetting, or benefits from more effective expert routing.

Q2. Historical support budget.

Could the authors clarify how many historical support samples or features are retained per task for CGER, and whether the associated memory overhead is included in the resource comparison?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Rg5S

W1 & Q1: Task-wise performance across stages, together with Average Forgetting or BWT.

A1: Thank you for this constructive suggestion. We answer the two parts in turn.

(1) Task-wise performance across stages. We report the accuracy on every learned task after each training stage for GeoSpec and the reproduced HiDe-LLaVA, SAME and CRAM on UCIT (Table R1) and TriGap (Table R2), using the per-stage models of the runs in Tables 1–2. As shown in Tables R1 and R2, every task learned by GeoSpec stays within 0.18 points of its just-learned accuracy across all later stages, whereas SAME's VizWiz-Cap on UCIT ends 13.02 points lower. GeoSpec thus acquires new tasks while preserving previously learned capabilities, the central requirement of MCIT.

Table R1. UCIT: task-wise accuracy after each training stage.

MethodStageImageNet-RArxivQAVizWiz-CapIconQACLEVR-MathFlickr30kAvg
HiDe-LLaVA190.83–––––90.83
290.7393.13––––91.93
390.4392.5348.35–––77.10
490.2790.4045.2168.83––73.68
589.9387.4743.6565.1348.03–66.84
689.1386.4042.8763.0747.9051.2763.44
SAME190.53–––––90.53
290.5392.33––––91.43
390.4092.4760.86–––81.24
490.3393.1061.1881.27––81.47
590.3092.8761.2279.6372.97–79.40
690.3392.7347.8479.6773.1057.3973.51
CRAM189.10–––––89.10
289.1092.90––––91.00
389.1092.9060.69–––80.90
489.1092.9060.6980.10––80.70
589.1092.9060.6980.1368.80–78.32
689.1092.9060.7980.1368.8057.2174.82
GeoSpec191.07–––––91.07
291.0791.93––––91.50
391.0791.9360.79–––81.26
491.0791.8760.7984.37––82.03
591.0791.8360.7184.3770.13–79.62
690.9791.8360.7184.3770.1357.8075.97
AuthorsOpenReview ↗

Response to Reviewer Rg5S

Table R2. TriGap: task-wise accuracy after each training stage.

MethodStagePMC-VQADocVQAChartQAIconQAInfoVQAArxivQARoadChemVQAFloodVQACLEVRAvg (seen)
HiDe-LLaVA144.50–––––––––44.50
243.1344.70––––––––43.92
341.0741.8117.72–––––––33.53
440.9739.1014.9253.77––––––37.19
541.1736.9711.3247.0027.56–––––32.80
636.6334.0510.9243.3325.9283.33––––39.03
737.3334.2810.8843.5025.6182.307.00–––34.41
835.9734.2611.0043.8326.1281.737.4011.30––31.45
936.8333.1011.6841.7025.3579.977.579.7058.05–33.77
1036.4032.9310.5240.3025.0479.107.6011.8358.1026.5332.84
SAME144.10–––––––––44.10
244.1345.15––––––––44.64
344.2045.0919.96–––––––36.42
443.9045.0018.5668.60––––––44.02
544.3344.3018.7268.9039.56–––––43.16
643.9344.1818.5268.8039.5692.87––––51.31
743.6044.2118.1668.7339.7393.0013.10–––45.79
844.1744.2318.2068.3039.6992.8312.7023.27––42.92
944.1744.3517.8468.3339.6693.1011.3023.6081.45–47.09
1044.0044.2018.3666.3039.6692.9011.5724.0081.2359.9348.22
CRAM143.13–––––––––43.13
243.1344.01––––––––43.57
343.1343.9018.20–––––––35.08
443.2043.7817.2067.00––––––42.80
543.2043.8917.1267.0339.28–––––42.10
642.9043.8317.1267.0339.2592.00––––50.36
742.9043.8116.5666.8739.2592.0013.00–––44.91
842.9043.7416.3666.8039.1092.0013.0022.73––42.08
942.9043.7616.5666.9039.0992.0013.0322.7380.14–46.35
1042.9043.7716.4865.1739.0992.0013.0322.7380.1458.5347.38
GeoSpec144.43–––––––––44.43
244.4345.96––––––––45.20
344.4345.9719.88–––––––36.76
444.4345.9719.8872.53––––––45.70
544.4345.9119.8872.5340.55–––––44.66
644.3745.8919.8672.5340.5593.00––––52.70
744.3745.9019.8472.5340.5592.9912.93–––47.02
844.3745.8919.7972.5340.5592.9712.9327.90––44.62
944.3745.8819.7672.5340.5592.9712.9327.9082.11–48.78
1044.3745.7819.7672.5340.5592.9712.9327.8382.2158.1049.70
AuthorsOpenReview ↗

Response to Reviewer Rg5S

(2) Average Forgetting and BWT. We summarize the matrices in Tables R1–R2 following MCITlib [1]: MFN is the final average, i.e., the paper's AVG, MFT the accuracy on each task right after it is learned, and MAA the mean of the stage-wise averages. BWT is the mean change from each task's accuracy right after it is learned to its final accuracy, and Average Forgetting is the mean drop from each task's best accuracy before the final stage to its final accuracy [2]; both are truncated at 0, so BWT ≤ 0, Average Forgetting ≥ 0, and a method without net forgetting receives 0.

Table R3. Plasticity and forgetting. Bold and italics mark the best and second-best results within each benchmark.

BenchmarkMethodMFNMFTMAABWTAvg. Forgetting
UCITHiDe-LLaVA63.4466.7477.30−3.963.96
UCITSAME73.5175.8982.93−2.863.08
UCITCRAM74.8274.8082.470.000.00
UCITGeoSpec75.9776.0283.57−0.060.06
TriGapHiDe-LLaVA32.8437.4536.34−5.125.19
TriGapSAME48.2248.8044.77−0.650.79
TriGapCRAM47.3847.8043.78−0.460.48
TriGapGeoSpec49.7049.7445.96−0.040.04

As shown in Table R3, GeoSpec achieves the highest MFT and MAA on both benchmarks with a near-zero BWT of −0.06 / −0.04 on UCIT / TriGap. The highest MFT reflects the task-specific specialization of TCGA, and the near-zero BWT reflects reliable access to the accumulated experts through CGER without test-time task identities, matching the two challenges that GeoSpec addresses.

[1] Guo et al. MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark. arXiv:2508.07307, 2025.

[2] Chaudhry et al. Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence. ECCV 2018.

W2 & Q2: Number of retained support samples and whether their memory is included in the resource comparison.

A2: Thank you for this question. We answer the two parts in turn.

(1) Number of retained support samples. For each task, CGER keeps the answer-free 512-d CLIP-B image and text features of 64 training inputs, 2 KiB per sample, rather than images, instructions or answers, and uses them only to refit the router. Experts are trained on the current task only (Sec. 3.1, l. 156–157). The 64 support inputs are drawn at random with the run's seed, and we have added the support size and its memory to App. B.2. Table R4 lists the cumulative support samples and memory after each task.

(2) Memory in the resource comparison. Yes. The Continual state in Table 7 counts the expert bank, the routing parameters and the cached support features, so the 223.4 MiB on TriGap already includes the 1.25 MiB support cache. The cache is used only to refit the router at task boundaries and is not needed for generation, as stated in App. B.2. We have revised the caption of Table 7 to list the cache explicitly.

Table R4. Cumulative number of retained support samples and their cumulative memory after each task.

BenchmarkQuantityTask 12345678910
CoINCumulative support samples64128192256320384448512––
Cumulative memory (MiB)0.130.250.380.500.630.750.881.00––
UCITCumulative support samples64128192256320384––––
Cumulative memory (MiB)0.130.250.380.500.630.75––––
TriGapCumulative support samples64128192256320384448512576640
Cumulative memory (MiB)0.130.250.380.500.630.750.881.001.131.25

As shown in Table R4, the full stream retains at most 640 support samples, only 1.25 MiB on TriGap and below 0.6% of its Continual state, a negligible cost that preserves GeoSpec's efficiency advantage in Table 7. We have added Table R4 to App. B.2 as Table 10.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

Dear ACs and Reviewers,

We sincerely appreciate the time and effort you have devoted to evaluating our paper. We are encouraged to see that our contributions are consistently recognized across several aspects:

  • Motivation. Reviewers find the two challenges we address, task-specific expert specialization and routing ambiguity, important and well motivated, and our framework a meaningful approach to both [Reviewers Rg5S and r65k]. The geometry-guided adaptation is considered theoretically motivated [Reviewer r65k].
  • Ablation design. The ablations are described as comprehensive [Reviewer Rg5S] and "really well designed" [Reviewer rqjg]. In particular, the identity-vs-permuted metric control (Table 5) is regarded as the most convincing evidence that TCGA's gain is tied to activation-aligned statistics rather than to arbitrary non-uniform scaling [Reviewer rqjg]; with the router fixed, TCGA improves AVG by 3.77 / 3.51 points on UCIT / TriGap (Table 4).
  • Performance and practicality. GeoSpec achieves the highest final AVG among the compared methods on all three benchmarks [Reviewers Rg5S and r65k], and the efficiency analysis supports its practical value [Reviewers rqjg and r65k]: 11.64M trainable parameters per task versus 122.14M for SAME and 57.65M for CRAM, and 25.1 GiB peak GPU memory versus 32.5 and 29.5 GiB.
  • Presentation. The paper is found clear and easy to follow [Reviewers Rg5S and r65k].

We also thank all reviewers for their insightful and constructive suggestions, which have greatly helped us improve the paper. All new results use the backbone, benchmarks, task orders and evaluation protocols of the submission unless noted otherwise. In addition to the pointwise responses below, we summarize the revisions according to each reviewer's comments:

To Reviewer Rg5S:

  1. We report stage-wise accuracy matrices of GeoSpec, HiDe-LLaVA, SAME and CRAM, together with MFT, MAA, BWT and Average Forgetting, in Sec. 5.5 and App. C.2.
  2. We report the number of retained support samples and their memory after each task, and clarify that this memory is included in the Continual state of Table 7, in Sec. 5.4 and App. B.2.

To Reviewer rqjg:

  1. We report three random seeds for GeoSpec, SAME and CRAM on UCIT and TriGap and three task orders on CoIN following SAME in Sec. 5.5 and App. C.1, remove the SiGMA row from Table 3, correct ProtoAda's AVG, and list the published and reproduced results of SAME and CRAM in App. B.3.
  2. We show which part of the task-conditioned metric survives in the update in Remark 1 of App. A.5, and report TCGA with FAM replaced by Adam or SGD-m and the Identity variant under a learning-rate sweep in Sec. 5.5 and App. D.3.
  3. We report the support budget and its memory, AVG under task-oracle routing, results of SAME, CRAM and GeoSpec under FLEX-normalized templates on TriGap, and the sensitivity of the router settings together with top-1 routing, in App. B.2 and App. D.5.
  4. We compare TCGA with LoRA-Pro, Riemannion and LoRA-RITE under the same CGER router in Sec. 2 and App. D.4, and update the citations.

To Reviewer r65k:

  1. We isolate gradient projection, FAM and rank realization in Sec. 5.5 and App. D.3.
  2. We report multi-run, task-order and forgetting results in Sec. 5.5 and App. C.
  3. We report the sensitivity of CGER to the support budget and to top-1 vs. top-2 routing in App. D.5.

We have highlighted all modifications in the revised paper in blue. We hope these additions address the reviewers' concerns and further improve our work. If any further clarification would help, we would be happy to address it and incorporate the changes into the final version. Thank you again for your time and efforts!

Best,

Authors

Dear Reviewers,

Thank you for your efforts so far. As the discussion period is coming to an end, I would like to kindly encourage you to review the authors' rebuttals and engage in further discussions, particularly if the authors have addressed your questions or concerns.

Thank you for your time and contributions!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.