acceptodds
Under review as a conference paper at ICLR 2027

ELGAR: Execution-Linked Gating for Adapter Routing in Continual Learning

Abstract

Selecting a visual expert changes the representation that language experts must process. Execution-Linked Gating for Adapter Routing (ELGAR) turns this dependency into an allocation rule: an instruction-conditioned gate selects a visual projector, and a language gate observes that same projector's output to combine frozen low-rank adaptation (LoRA) updates, with one projector and one autoregressive path at deployment. Our finite-candidate analysis characterizes when an observation supports an optimal allocation and shows how a pooled bank mean can hide distinctions preserved by the selected output. Three paired source-order repeats on six UCIT tasks with 3,000 examples each give 75.53 mean final average score (FAA) for selected-output routing, versus 74.85 for bank-mean routing and 73.75 for static allocation. The additional 0.68-point gain is positive in every repeat with the visual route and task-origin supervision fixed. Separate matched pairs favor output content over projector identity in every repeat, with a mean gain of 0.62 points. Complete ELGAR reaches 75.63 FAA on UCIT–LLaVA and leads reproduced continual baselines in FAA and backward transfer across all four UCIT/DCL–backbone settings. Thus observing the executed representation adds measurable allocation value beyond knowing which branch ran.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_1zYJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

ELGAR couples visual projector selection with language-adapter routing for continual multimodal instruction tuning. The language gate uses the projector’s normalized pooled output together with image and instruction summaries to combine frozen LoRA experts through learned, potentially signed module coefficients. Training uses a soft projector mixture, answer supervision, and task-origin supervision. At inference, the model executes one projector and one generation trajectory.

The paper provides an observation-sufficiency analysis and evaluates the method on UCIT and DCL with two backbones.

Critical strengths
  1. The paper identifies a concrete dependency: selecting a visual projector changes the representation processed by the language experts. Giving the downstream allocator access to that representation is a sensible design, and I find the contribution sufficiently novel.

  2. The experimental study is solid, particularly the matched controls. The experiments distinguish dynamic allocation from the additional value of selected-output features while fixing the visual route and expert bank. Selected-output routing improves FAA over bank-mean routing by 0.68 points on average across three paired source-order repeats. A separate comparison against projector identity gives a 0.62-point average improvement. Both studies report positive aggregate differences in every pair (Figure 3 and Appendix H).

  3. The complete system achieves higher FAA than the strongest reproduced continual-learning baseline in each of the four benchmark–backbone settings (Tables 1–4). The UCIT improvements of 3.48 and 3.43 points are substantial. The theoretical analysis also helps distinguish whether an observation contains useful allocation information from whether a fitted gate can exploit it.

Critical weaknesses
  1. Some protocol details need clarification. Appendix N specifies 128 source examples per task and the calibration budget, but the handling of these examples across stages is less clear. The paper should explain whether calibration restarts or continues at each stage, which previous-task examples remain accessible, and how these conditions compare with the baseline protocols. Expert-bank construction and missing optimization settings, such as the learning rate, also need fuller documentation. These are reporting gaps; the manuscript does not establish that the comparisons are unfair.

  2. The central observation improvement is modest, and its validation is concentrated in one setting. The paired studies support the benefit on UCIT–LLaVA, but varying source order does not establish robustness to different expert banks or calibration samples. An observation control on another backbone or benchmark would strengthen the evidence. Complete-system gains also vary: on DCL–LLaVA, FAA improves by 0.44 points over the best FAA baseline, while CAA is 1.98 points below the best CAA baseline.

  3. The controls show that selected-output features outperform the tested alternatives. A comparison with similarly informative features from a non-executed projector would help determine how much of the benefit depends specifically on matching the executed branch. I view this as a useful extension rather than a necessary condition for the reported empirical finding.

Questions for authors
  1. Does each stage-prefix calibration restart from the specified initialization or continue from the previous stage? How are previous-task calibration examples retained and accessed?

  2. Could you provide the complete stage-wise score matrices and the expert-training and calibration configurations needed for reproduction?

  3. Are observation-control results available for InternVL or DCL? These would help assess the scope of the central finding.

9 comments
AuthorsOpenReview ↗

Thank You for Your Time and Thoughtful Review

Thank you for recognizing the execution-linked research question, the matched comparisons, and the performance of the complete system. To save you time, we have highlighted key conclusions and important points in bold throughout our responses. The additional results extend the stability analysis beyond calibration order to resampling and independently trained expert weights: executed-output FAA on UCIT–LLaVA differs by only 0.1356 points between two calibration draws and by 0.6711 points between two new sets of expert weights; the matched comparison using new weights on the same hardware also retains a positive observation-related improvement. We also clarify historical-example access, expert training, and stage-wise calibration, and distinguish final performance from performance throughout learning. Below, we address stability separately from the relative advantage of an observation, linking each conclusion to its experimental conditions.

AuthorsOpenReview ↗

W1: How are stage-wise calibration, historical-data access, and expert training configured?

Thank you for requesting clarification of the transitions between stages. At every stage, ELGAR reinitializes the routers and AdamW and trains for six epochs using a fixed set of 128 calibration examples from each seen task. Later stages reuse earlier examples and experts, accessing only seen tasks. The table specifies the resulting UCIT budgets.

UCIT Tasks Seen123456
Available Calibration Examples128256384512640768
Optimizer Updates76815362304307238404608

Thank you for suggesting that data-access conditions be compared with the baseline protocols. All six baselines use their corresponding DCL–LLaVA implementations and retain their respective LoRA ranks and scaling parameters. Reproductions use the same backbone, task order, and evaluation protocol. When using results reported by the original authors under the same evaluation protocol, we retain their values and distinguish them from reproduced results. At each stage, the baseline training configurations supply the current task's data without historical-example replay and, from the second stage onward, load the preceding stage's checkpoint to inherit its weights. In contrast, ELGAR explicitly retains and reuses 128 examples from each seen task during calibration. All six baselines apply LoRA to the q, k, v, and o projections in attention and the gate, up, and down projections in the feed-forward layers. The per-GPU microbatch size is 8, the effective global batch size is 16, and the learning rate is 2e−5. DISCO, HiDE, and O-LoRA use one GPU with two gradient-accumulation steps; the other three use two GPUs with one accumulation step, preserving the same effective global batch size. The remote-sensing, medical, autonomous-driving, science, and finance tasks are trained for 1, 3, 1, 2, and 1 epochs, respectively. The table below lists the LoRA rank, scaling parameter α, and completed optimizer updates at each stage for each method. One step denotes one parameter update after gradient accumulation.

DCL–LLaVA BaselineConfigured LoRA RankLoRA Scaling αRS Optimizer UpdatesMed Optimizer UpdatesAD Optimizer UpdatesSci Optimizer UpdatesFin Optimizer Updates
DISCO16032037504284375038443747
CL-MoE16032037504284375038443748
HiDE16032037504284375038443747
O-LoRA16032037504284375038423747
MoELoRA357037504284375038443748
LoRA-FT326437504284375038443748

Expert training and router calibration serve distinct roles: task experts are first trained independently from a shared base and retained, then frozen while the routers learn to combine them. The expert learning rate therefore governs task adaptation, whereas the gate learning rates below govern how these existing capabilities are used. The table lists the number of experts, LoRA parameters, and training learning rates for all four settings; for DCL–InternVL, it reports the actual peak learning rate during training.

SettingIndependent ExpertsLoRA RankLoRA Scaling αEffective Global Batch SizeExpert Training Learning Rate
UCIT–InternVL61632641e−4
UCIT–LLaVA6163216LoRA: 2e−4; visual projector: 2e−5
DCL–InternVL51632321e−4 (peak)
DCL–LLaVA53264162e−5

After freezing the experts, calibration follows the same rule in all four settings: 128 source examples per seen task are processed one at a time for six epochs, with consistent learning rates for the gates and module coefficients across settings. The difference in total updates between UCIT and DCL arises from their final stages containing six and five tasks, respectively, rather than from separately chosen calibration epoch counts. The table therefore presents the common optimization rule and the calibration budget as the number of seen tasks increases.

Final-stage SettingCalibration ExamplesOptimizer Updates (6 Epochs)Visual and Language Gate Learning RateModule Coefficient Learning RateWeight Decay
UCIT–InternVL7684,6081e−42e−30
UCIT–LLaVA7684,6081e−42e−30
DCL–InternVL6403,8401e−42e−30
DCL–LLaVA6403,8401e−42e−30
AuthorsOpenReview ↗

W2: How robust is performance across settings, and how should FAA and CAA be interpreted?

Thank you for suggesting robustness checks across backbones and benchmarks. Actual executed output consistently improves performance in the matched UCIT–LLaVA comparisons and achieves final performance close to the references in the listed InternVL comparisons; these results distinguish performance stability from relative advantage. Matched LLaVA comparisons favor output over bank mean and identity by 0.6789 and 0.6180 points. InternVL differences average −0.1050/−0.0933 over three independent UCIT calibration draws; DCL differences are −0.022/−0.050. Information helps when it distinguishes useful expert combinations.

Thank you for suggesting a comparable test on another benchmark. Retrained DCL–LLaVA experts share frozen projectors. Within-bank output–mean differences are +0.0099 and −0.0817. The differences have opposite signs but are both below 0.1 points in magnitude.

DCL–LLaVA Expert InitializationObservationRSMedADSciFinFAA
IndependentSelected output70.370063.554653.120054.429390.932766.4813
IndependentBank mean70.580063.166553.180054.557990.872766.4714
Previous-task LoRASelected output72.670063.309552.720054.328491.443266.8942
Previous-task LoRABank mean72.800063.268652.940054.317791.553266.9759

Thank you for suggesting that changes in calibration order be distinguished from changes in calibration examples. After resampling the calibration examples, executed-output FAA on the two UCIT–LLaVA draws is 75.0894 and 75.2250, a difference of only 0.1356 percentage points, indicating stability under the tested sampling changes. Both draws fix visual routing and experts, use 128 examples per task and 4,608 language updates, and share formal evaluation examples.

UCIT–LLaVA Calibration DrawObservationImageNet-RArxivQAVizWiz-captionIconQACLEVR-MathFlickr30kFAA
Draw 1Selected output90.966790.333362.700072.866777.800055.870075.0894
Draw 1Bank mean91.266790.266761.120073.600078.166755.940075.0600
Draw 2Selected output91.033391.166761.740073.100077.500056.810075.2250
Draw 2Bank mean90.900091.233360.620072.666778.133356.990075.0906

Executed-output FAA is 75.1144 and 75.7856 for two new expert sets trained with the same recipe, a difference of 0.6711 points; the same-hardware comparison for the second set retains a 0.6600-point advantage over bank mean. Six experts per set train independently for one epoch. Calibration fixes visual routing, 768 examples, seed 20261112 and 4,608 updates; evaluation uses 3,000 examples per task. The first seed's +0.6283 difference also reflects hardware changes. Flickr30k differs by −0.2700 and +0.0700 points; the other five tasks improve.

UCIT–LLaVA Expert-training SeedObservationImageNet-RArxivQAVizWiz-captionIconQACLEVR-MathFlickr30kFAA
20261113Selected output91.066789.833364.220070.233379.733355.600075.1144
20261113Bank mean90.933389.200062.780068.500079.633355.870074.4861
20261114Selected output91.900092.900064.160069.500079.533356.720075.7856
20261114Bank mean91.733391.900062.570068.766779.133356.650075.1256

Thank you for also asking whether these observations hold on another backbone. Across the three tested calibration draws and three InternVL weight sets, the ranges of final executed-output scores are 0.1050 and 0.3000 points, respectively, while the direction relative to bank mean varies by setting. InternVL weight-set differences from bank mean are −0.30/−0.11/−0.03 points; these are not same-recipe seed repeats.

Thank you for pointing out the different rankings under FAA and CAA. The two metrics measure performance after learning is complete and performance throughout the learning sequence, respectively; the ranking difference on DCL–LLaVA primarily reflects changes between early and final stages. Each DCL stage contributes one-fifth to CAA. CL-MoE leads the first four stages, by 7.45 and about 4.23 points in stages 1 and 2; ELGAR leads stage 5 by 5.49 but averages 1.98 lower in CAA. The final stage enables composition of five experts rather than one. Original final FAA is 63.5360, 0.4320 points above Many. With a unified finance prompt, reevaluated final FAA is 64.5412 versus 63.1045, a 1.4367-point difference; the original matrix and CAA remain separate. For deployment after all tasks have been introduced, FAA directly measures the relevant performance; for continuous deployment during learning, CAA additionally captures early-stage performance.

Reviewer_ZT7rOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

I have contacted the AC that I am not qualified to evaluate this manuscript. I didn't want to the system to flag me for not submitting something, and the system requires me to click an option for each question; apologies to the authors and other reviewers if this ends up visible to them.

Critical strengths

I have contacted the AC that I am not qualified to evaluate this manuscript.

Critical weaknesses

I have contacted the AC that I am not qualified to evaluate this manuscript.

Questions for authors

I have contacted the AC that I am not qualified to evaluate this manuscript.

1 comment
AuthorsOpenReview ↗

Thank You for Your Candid Clarification

Thank you for candidly explaining that this submission falls outside your area of expertise and for bringing this to the Area Chair's attention. We value your feedback and would like to understand accurately what the score in the review record represents. We have not yet received explicit clarification from the Area Chair on how this score will be treated in the final assessment. If it was entered to satisfy a required system field rather than to express your technical assessment of the submission, we would appreciate your clarifying this in the review discussion with the Area Chair. If you have any specific questions about the manuscript or supplementary results, please feel free to raise them; we would be happy to provide an explanation.

Reviewer_bSVYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ELGAR, an execution-linked adapter routing method for multimodal continual learning. The main idea is to first select a visual projector based on the image and instruction, and then use the output of the selected projector to guide the dynamic combination of language-side LoRA experts. In this way, the method coordinates the expert selection between visual and language modules. The authors also provide theoretical analysis and evaluate the method on two benchmarks (UCIT and DCL) with two vision-language models.

Critical strengths

The motivation is intuitive, as the language expert selection should consider the actual output from the selected visual projector. The paper also provides a finite-candidate analysis to study the sufficiency of different observations for expert allocation, which gives some theoretical support to the proposed design.

The authors evaluate their method on multiple benchmarks (UCIT/DCL) and backbones (LLaVA/InternVL). They also conduct several controlled experiments to isolate and verify the benefits of using the selected visual projector output for language routing.

Critical weaknesses

The overall framework is mainly built upon existing visual projector selection and LoRA expert routing methods. The main contribution is more like an improvement in the routing features, rather than introducing a fundamentally new routing mechanism.

As shown in Figure 4, the visual router almost behaves like a task-level classifier, where each task is nearly always assigned to its corresponding projector. Given this observation, it is not very clear how necessary the additional dynamic combination in the language router is, and whether there is some redundancy between the two routing modules. More analysis on how these two routers cooperate would be helpful.

The training process relies heavily on task-origin labels to supervise the routers. As shown in the experiments, removing this supervision leads to a clear performance drop in FAA. This dependence may limit the general applicability of the method to more realistic continual learning settings where task labels are not always available.

Questions for authors

NA

7 comments
AuthorsOpenReview ↗

Thank You for Your Time and Thoughtful Review

Thank you for recognizing the motivation for execution-linked routing, the finite-candidate analysis, and the cross-benchmark experiments. To save you time, we have highlighted key conclusions and important points in bold throughout our responses. Our central contribution is to identify what information language routing should observe: the same branch identity can correspond to different actual outputs, and these differences can affect the appropriate expert combination. The theoretical construction and matched comparisons jointly support this distinction. Further experiments show that dynamic calibration learns more effective combinations and that, in the tested settings, fixing the learned weights by branch after training retains nearly the same performance. Below, we explain this information design, the distinct roles of calibration and deployment, and how answer supervision supports routing when task-origin labels are limited.

AuthorsOpenReview ↗

W1: Is the novelty in the routing mechanism or in the information observed by the router?

Thank you for distinguishing the routing mechanism from the information it uses. ELGAR adopts existing forms of visual projector selection and LoRA weighting. Its novelty is to let the language allocator observe the projected features actually received by the decoder, preserving decision-relevant information omitted by branch identity. The finite-candidate analysis characterizes when this information is needed: when multiple requests have the same observation, they must share at least one optimal allocation for an allocator to make an optimal choice for all of them. The construction with fixed nonlinear projectors further shows that two requests can select the same projector and have identical image–text summaries and bank means, yet produce different projected outputs and require different optimal language-expert allocations. Including the actual output therefore makes these requests distinguishable to the router.

To test this analysis, we fixed visual execution, the expert bank, image–text summaries, and the calibration budget in UCIT–LLaVA, replacing only branch identity with the actual executed output. FAA improved under all three calibration orders, by an average of 0.6180 points. The theory establishes when branch identity loses decision-relevant information, while the paired experiments show that the model can use the additional information. Together, they support the contribution of changing what the router observes.

Source-order SeedSelected output FAAProjector identity FAASelected output − Projector identity (pp)
2026091175.586175.0861+0.5000
2026091275.757275.0861+0.6711
2026091375.256774.5739+0.6828
Mean of three repeats75.533374.9154+0.6180
AuthorsOpenReview ↗

W2: When visual selection is accurate, is dynamic language-expert allocation still needed?

Thank you for asking, based on Figure 4, whether the two routing components are redundant. Language-expert allocation remains useful even when visual branches are selected almost entirely by task; whether inference also requires per-example weight adjustment is a separate question. Visual routing selects the decoder representation; language allocation combines LoRA experts. All four UCIT–LLaVA conditions share frozen backbone and experts, matched calibration budgets and 3,000 evaluation examples per task. Disabling language allocation fixes query weights and module coefficients uniformly; disabling visual routing supplies a fixed mean of projector outputs. Adding language allocation improves FAA by 13.45 points with visual routing and 18.45 without it; adding visual routing with language allocation improves FAA by 5.66 points. Their contributions therefore depend on one another.

ConfigurationLanguageVisualFAA ↑
ELGAROnOn75.63
Remove language moduleOffOn62.18
Remove visual moduleOnOff69.97
Remove both modulesOffOff51.52

Thank you for asking whether the language-side improvement comes from dynamic allocation. We compare learnable global static weights with executed-output allocation, sharing visual routing, experts and 4,608 updates across three calibration orders, with 3,000 evaluation examples per task. Executed-output allocation performed better in all three repeats, with an average FAA advantage of 1.78 percentage points. Input-dependent weighting is thus more effective than one shared combination under these conditions.

Source-order SeedStatic FAASelected output FAASelected output − Static (pp)
2026091173.855675.5861+1.73
2026091273.672275.7572+2.09
2026091373.721775.2567+1.54
Mean of three repeats73.749875.5333+1.78

At the finer visual-branch level, fixing the weights after training changed FAA by less than 0.1 points on both backbones, showing that the learned combinations can be deployed using a simpler branch-level rule in these settings. We estimate mean language weights per branch from 768 calibration examples and select that branch's combination at inference. The decreases are 0.0361 points for LLaVA and 0.0883 for InternVL, distinguishing learning useful combinations from adjusting weights within a branch. LLaVA task scores use 3,000 examples each.

UCIT Setting and TaskDynamic AllocationBranch-fixed WeightsDynamic − Branch-fixed (pp)
ImageNet-R90.933391.0000−0.0667
ArxivQA92.600092.5667+0.0333
VizWiz-caption63.610063.3500+0.2600
IconQA71.233371.1000+0.1333
CLEVR-Math78.633378.7667−0.1333
Flickr30k56.370056.3800−0.0100
LLaVA: FAA (six-task mean)75.563375.5272+0.0361
InternVL: FAA (six-task mean)76.786776.6983+0.0883

For the branch-fixed control trained from scratch, the value of dynamic calibration lies in the combinations it learns: it exceeds directly calibrated branch-fixed weights by 0.6589 points, and fixing its weights by branch after training retains 0.6228 points of that advantage. It learns one combination per visual branch with the same frozen backbone, visual router, experts, ordered 768 examples and 4,608 updates as dynamic calibration, then uses the same six-task evaluation. Directly fitted fixed weights improve ArxivQA and Flickr30k but reduce the other four scores, most notably IconQA. Dynamic calibration therefore learns better combinations that can subsequently be deployed with simpler branch-level weights, retaining language allocation without requiring per-example weighting in these settings.

UCIT–LLaVA TaskDynamic AllocationPosthoc Branch-fixed AllocationIndependently Fitted Branch-fixed Allocation
ImageNet-R90.933391.000090.9000
ArxivQA92.600092.566793.1000
VizWiz-caption63.610063.350062.3900
IconQA71.233371.100067.8000
CLEVR-Math78.633378.766777.7667
Flickr30k56.370056.380057.4700
FAA (six-task mean)75.563375.527274.9044
Reviewer_hLsJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ELGAR, a framework for routing frozen task-specific visual projectors and language LoRA experts in continual multimodal instruction tuning. An image-and-instruction-conditioned router selects a visual projector, and a language allocator uses the selected projector’s output, together with image and instruction summaries, to determine the mixture of language experts. The central idea is to condition language allocation on the visual representation actually delivered to the decoder. The method is evaluated on UCIT and DCL using LLaVA and InternVL, with controlled comparisons against static allocation, bank-mean routing, and projector-identity conditioning.

Critical strengths
  1. The motivation is intuitive: selecting a visual projector changes the representation processed by the language model, so exposing this representation to the language allocator is a reasonable design.
  2. The complete system achieves strong final performance and retention relative to the reported baselines. Deployment requires only one selected projector, and the additional conditioning feature is obtained from a representation already computed for the decoder.
Critical weaknesses
  1. It is unclear to me how much the performance gap with other baselines is coming from the other design choices other than the main contributions provided by ELGAR. Could the authors clarify how the baselines differ in their use of projectors and LoRA experts—for example, whether projectors are shared or task-specific, where LoRA is applied, and whether experts are continually updated, merged, or retained separately? These choices may affect final performance independently of the proposed selected-output conditioning. A table documenting the configurations actually used, including deviations from the original methods, would be helpful. The matched internal ablations provide useful evidence for the proposed connection, but additional controlled comparisons would clarify how much of the advantage over external baselines comes from this connection versus differences in projector and expert organization.
  2. Regarding the frozen model components, could the authors specify how ELGAR’s task-specific projectors and LoRA experts were obtained? While Appendix N describes router calibration, the preceding expert-training procedure remains unclear, including checkpoint sources, training data, initialization, LoRA configurations, and optimization budgets, since these frozen experts determine the representations and capabilities available to the routers. I'm curious whether router training is sensitive to how these components are trained.
  3. I also have question regarding when task boundaries are less distinguishable. Figure 4 shows that the visual router selects the task-associated projector for approximately 97.7–100% of examples across UCIT tasks. This suggests that the evaluated distributions provide strong cues for identifying the source task. How does the method behave when task distributions overlap or projector selection is unreliable? Since deployment commits to one projector without revisiting the decision, an unsuitable selection could affect both the decoder’s visual input and the language allocator.
Questions for authors

Please see weakness which covers all the questions I have.

9 comments
AuthorsOpenReview ↗

Thank You for Your Time and Thoughtful Review

Thank you for recognizing ELGAR's motivation and the complete system's final performance and knowledge retention. To save you time, we have highlighted key conclusions and important points in bold throughout our responses. Your questions prompted us to distinguish more clearly the roles of information, expert organization, and training configuration. The additional evidence shows that the actual projector output can provide useful information for language allocation even when visual execution is fixed, and that input-adaptive composition outperforms the tested globally fixed composition when the same experts are retained. We also report the training and data-access conditions of the external baselines and examine new expert weights and inputs with shared semantics, so that each conclusion refers to a clearly defined comparison.

AuthorsOpenReview ↗

W1: How can baseline configuration, expert organization, and observation be distinguished?

Thank you for pointing out that complete-system differences may reflect both expert organization and the information used for language allocation. We examine these factors separately through controls that retain the same expert weights, and report the complete-system comparison with external baselines separately.

The six baselines configure a single projector shared across tasks and continue updating it from the preceding stage, whereas ELGAR independently trains and retains a projector and a LoRA expert for each task, freezes both during calibration, and combines LoRA updates at inference. All six baselines apply LoRA to attention q, k, v, o and feed-forward gate, up, down projections. Baselines use current-task data without replay; ELGAR reuses 128 calibration examples per seen task. Baseline batch size is 16 and learning rate 2e−5. RS, Med, AD, Sci and Fin denote remote sensing, medicine, autonomous driving, science and finance; epochs are 1,3,1,2,1. The five task columns count optimizer updates.

MethodRankαRSMedADSciFin
DISCO16032037504284375038443747
CL-MoE16032037504284375038443748
HiDE16032037504284375038443747
O-LoRA16032037504284375038423747
MoELoRA357037504284375038443748
LoRA-FT326437504284375038443748
ELGAR32/expert6437504284375038443748

DISCO retains selected subspaces; O-LoRA freezes old orthogonal subspaces; CL-MoE/MoELoRA route over expert pools; HiDE uses top-layer anchor weighting and lower-layer fusion; LoRA-FT continues one adapter. Many mixes projectors by image–prototype similarity and uses LoRA rank 16, α=32. ELGAR retains separate LoRAs and combines them according to the input at inference, whereas Many recursively merges language updates from previously seen tasks into a stage-specific state and uses a scaling coefficient of 2.0 for the merged update at deployment.

The FAA scores of ELGAR with selected output, ELGAR with bank mean, and Many are 64.5412, 64.6764, and 63.1045, respectively. ELGAR's advantage over Many is approximately 1.4367 points and remains when the observation is replaced by the bank mean. Under matched prompts, decoding and scoring, the gap decomposes as −0.1352+1.5718 points, so the residual includes system differences beyond observation. Selected-output scores outside finance retain same-protocol historical aggregates; Sci rounding preserves both difference signs.

Input-adaptive allocation achieves 64.6764 FAA, compared with 50.5180 for globally fixed composition, with improvements on all five tasks. For this control, the bank-mean model retains its experts, visual execution and module coefficients; only input-dependent weights become their mean over 640 calibration examples. Evaluation shares all 47,469 examples. This tests global fixed composition, not Many's merging rule.

DCL–LLaVASelected outputBank mean/adaptiveManyFixed composition
RS69.110068.730059.440057.6200
Med62.175762.042960.909143.5240
AD50.490051.720053.810028.2400
Sci53.0352.998651.791849.1466
Fin87.900387.890389.571774.0592
FAA≈64.541264.676463.104550.5180

Selected-output routing achieves higher FAA than bank-mean routing in all three paired calibration-order repeats on UCIT–LLaVA, with an average difference of 0.6789 percentage points. Both controls fix experts, visual execution and budget; repeats change calibration order. The identity control uses a separate visual router: Actual output achieves higher FAA in all three pairs, averaging 0.6180 percentage points higher, with the main differences arising on VizWiz-caption and IconQA.

Order seedOutput FAABank mean FAAIdentity FAA
2026091175.586175.087875.0861
2026091275.757275.041775.0861
2026091375.256774.433974.5739
Mean75.533374.854474.9154

With the observation held fixed, inheriting the preceding task's LoRA improves FAA by 0.4129 points for selected output and 0.5045 points for bank mean. Within the respective expert banks, selected output differs from bank mean by +0.0099 and −0.0817 points. Retrained DCL–LLaVA experts share frozen visual projectors and reset their optimizers per task. All four exceed the original setting, also reflecting retraining; the main setting remains independently initialized.

InitializationObservationFAA
IndependentSelected output66.4813
IndependentBank mean66.4714
Previous-task LoRASelected output66.8942
Previous-task LoRABank mean66.9759
AuthorsOpenReview ↗

W2: How are the frozen experts trained, and how sensitive is routing to replacing their weights?

Thank you for suggesting that we distinguish frozen-expert training from router calibration. All six UCIT–InternVL experts start from the same InternVL-Chat-7B checkpoint and initial visual projector and are trained independently for one epoch on their respective UCIT task training sets. Each task's projector and LoRA are retained and frozen during calibration. LoRA covers attention q/k/v/o and feed-forward gate/up/down projections, with rank 16, α=32 and dropout 0.05. AdamW uses learning rate 1e−4, cosine scheduling, 0.03 warmup and zero weight decay. Training uses BF16, sequence length 2,048 and square-padded images with fourth-to-last-layer visual features; UCIT evaluation uses RGB images.

TaskTraining ExamplesEpochsOptimizer UpdatesEffective Global Batch Size
ImageNet-R23998137564
ArxivQA40000162564
VizWiz-caption40000162564
IconQA29859146764
CLEVR-Math40000162564
Flickr30k40000162564

Thank you for asking about expert initialization and training budgets on DCL–LLaVA. Five experts independently start from LLaVA-v1.5-7B, with learning rate 2e−5 for LoRA and projectors, cosine scheduling, 0.03 warmup and zero weight decay. Training uses two GPUs, microbatch 8 per GPU, one accumulation step, BF16, TF32, gradient checkpointing, ZeRO-2 and sequence length 2,048. Square-padded images use penultimate CLIP-L/14-336 features and a two-layer GELU projector of dimensions 1024→4096→4096. Both experts and projectors are then frozen.

TaskLoRA RankLoRA Scaling αEffective Global Batch SizeEpochsOptimizer Updates
RS32641613750
Med32641634284
AD32641613750
Sci32641623844
Fin32641613748

Thank you for asking whether the other settings follow the same expert-construction principle. UCIT–LLaVA and DCL–InternVL independently train six and five experts from LLaVA-v1.5-7B and InternVL-Chat-7B, respectively, retaining their projectors. Both use LoRA rank 16, α=32 and dropout 0.05. Budgets follow benchmark task order.

SettingEffective Global Batch SizeExpert Training Learning RateEpochs per TaskOptimizer Updates per Task
UCIT–LLaVA16LoRA: 2e−4; visual projector: 2e−5One epoch per task1500, 2500, 2500, 1866, 2500, 2500
DCL–InternVL321e−4 (peak for all five tasks)1, 3, 1, 2, 11875, 2142, 1875, 1922, 1874

Thank you for raising the question of sensitivity to expert training. Across three sets of expert weights, UCIT–InternVL FAA ranges from 76.3267 to 76.6267, a maximum variation of 0.30 points, and both new sets score higher than the original weights. Selected-output observations and calibration settings are retained. These three weight sets are not same-recipe seed repeats, so their variation reflects the tested weights rather than the seed alone.

Expert WeightsELGAR FAA
Original expert weights76.3267
Newly trained expert weights, set 176.6267
Newly trained expert weights, set 276.6183
Mean ± sample SD across the three sets of expert weights76.5239 ± 0.1709

Thank you for further considering the effect of replacing expert weights under the same training recipe. The two new UCIT–LLaVA weight sets achieve FAA scores of 75.1144 and 75.7856, a difference of 0.6711 points, with final performance remaining between 75.1 and 75.8. Six experts per set independently train for one epoch with seeds 20261113 and 20261114. We fix the visual gate, 768 calibration examples, calibration seed 20261112 and 4,608 updates, then evaluate 3,000 examples per task. Task-level variation accompanies close overall means. The sets use different GPU hardware, so between-set differences reflect both training and evaluation conditions.

UCIT–LLaVA Expert-training SeedImageNet-RArxivQAVizWiz-captionIconQACLEVR-MathFlickr30kELGAR FAA
2026111391.066789.833364.220070.233379.733355.600075.1144
2026111491.900092.900064.160069.500079.533356.720075.7856

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank You for Your Encouragement and Thoughtful Comments

Thank you again for the time and care you put into reviewing our work. We appreciate hLsJ’s recognition of ELGAR’s motivation and its ability to retain performance on earlier tasks, bSVY’s comments on the finite-candidate analysis and the evaluation across benchmarks and backbones, and 1zYJ’s recognition of the matched controls that separate the effect of dynamic allocation from the value of observing the executed projector output. We also thank ZT7r for candidly explaining that our submission falls outside their area of expertise and for bringing this to the Area Chair’s attention. We appreciate the care they took to help ensure that our work receives a fair and informed assessment.

Your suggestions helped shape the follow-up experiments in our responses. Your feedback matters a great deal to us. It has helped us think more carefully about what our experiments show and how to explain it clearly. We understand that you may have many demands on your time, and we already appreciate the time you’ve given us. If you have a chance to read our responses and share any further thoughts, we would be very grateful for the opportunity to learn from them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.