- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
ELGAR couples visual projector selection with language-adapter routing for continual multimodal instruction tuning. The language gate uses the projector’s normalized pooled output together with image and instruction summaries to combine frozen LoRA experts through learned, potentially signed module coefficients. Training uses a soft projector mixture, answer supervision, and task-origin supervision. At inference, the model executes one projector and one generation trajectory.
The paper provides an observation-sufficiency analysis and evaluates the method on UCIT and DCL with two backbones.
Critical strengths
-
The paper identifies a concrete dependency: selecting a visual projector changes the representation processed by the language experts. Giving the downstream allocator access to that representation is a sensible design, and I find the contribution sufficiently novel.
-
The experimental study is solid, particularly the matched controls. The experiments distinguish dynamic allocation from the additional value of selected-output features while fixing the visual route and expert bank. Selected-output routing improves FAA over bank-mean routing by 0.68 points on average across three paired source-order repeats. A separate comparison against projector identity gives a 0.62-point average improvement. Both studies report positive aggregate differences in every pair (Figure 3 and Appendix H).
-
The complete system achieves higher FAA than the strongest reproduced continual-learning baseline in each of the four benchmark–backbone settings (Tables 1–4). The UCIT improvements of 3.48 and 3.43 points are substantial. The theoretical analysis also helps distinguish whether an observation contains useful allocation information from whether a fitted gate can exploit it.
Critical weaknesses
-
Some protocol details need clarification. Appendix N specifies 128 source examples per task and the calibration budget, but the handling of these examples across stages is less clear. The paper should explain whether calibration restarts or continues at each stage, which previous-task examples remain accessible, and how these conditions compare with the baseline protocols. Expert-bank construction and missing optimization settings, such as the learning rate, also need fuller documentation. These are reporting gaps; the manuscript does not establish that the comparisons are unfair.
-
The central observation improvement is modest, and its validation is concentrated in one setting. The paired studies support the benefit on UCIT–LLaVA, but varying source order does not establish robustness to different expert banks or calibration samples. An observation control on another backbone or benchmark would strengthen the evidence. Complete-system gains also vary: on DCL–LLaVA, FAA improves by 0.44 points over the best FAA baseline, while CAA is 1.98 points below the best CAA baseline.
-
The controls show that selected-output features outperform the tested alternatives. A comparison with similarly informative features from a non-executed projector would help determine how much of the benefit depends specifically on matching the executed branch. I view this as a useful extension rather than a necessary condition for the reported empirical finding.
Questions for authors
-
Does each stage-prefix calibration restart from the specified initialization or continue from the previous stage? How are previous-task calibration examples retained and accessed?
-
Could you provide the complete stage-wise score matrices and the expert-training and calibration configurations needed for reproduction?
-
Are observation-control results available for InternVL or DCL? These would help assess the scope of the central finding.
9 comments
W1: How are stage-wise calibration, historical-data access, and expert training configured?
Thank you for requesting clarification of the transitions between stages. At every stage, ELGAR reinitializes the routers and AdamW and trains for six epochs using a fixed set of 128 calibration examples from each seen task. Later stages reuse earlier examples and experts, accessing only seen tasks. The table specifies the resulting UCIT budgets.
| UCIT Tasks Seen | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|
| Available Calibration Examples | 128 | 256 | 384 | 512 | 640 | 768 |
| Optimizer Updates | 768 | 1536 | 2304 | 3072 | 3840 | 4608 |
Thank you for suggesting that data-access conditions be compared with the baseline protocols. All six baselines use their corresponding DCL–LLaVA implementations and retain their respective LoRA ranks and scaling parameters. Reproductions use the same backbone, task order, and evaluation protocol. When using results reported by the original authors under the same evaluation protocol, we retain their values and distinguish them from reproduced results. At each stage, the baseline training configurations supply the current task's data without historical-example replay and, from the second stage onward, load the preceding stage's checkpoint to inherit its weights. In contrast, ELGAR explicitly retains and reuses 128 examples from each seen task during calibration. All six baselines apply LoRA to the q, k, v, and o projections in attention and the gate, up, and down projections in the feed-forward layers. The per-GPU microbatch size is 8, the effective global batch size is 16, and the learning rate is 2e−5. DISCO, HiDE, and O-LoRA use one GPU with two gradient-accumulation steps; the other three use two GPUs with one accumulation step, preserving the same effective global batch size. The remote-sensing, medical, autonomous-driving, science, and finance tasks are trained for 1, 3, 1, 2, and 1 epochs, respectively. The table below lists the LoRA rank, scaling parameter α, and completed optimizer updates at each stage for each method. One step denotes one parameter update after gradient accumulation.
| DCL–LLaVA Baseline | Configured LoRA Rank | LoRA Scaling α | RS Optimizer Updates | Med Optimizer Updates | AD Optimizer Updates | Sci Optimizer Updates | Fin Optimizer Updates |
|---|---|---|---|---|---|---|---|
| DISCO | 160 | 320 | 3750 | 4284 | 3750 | 3844 | 3747 |
| CL-MoE | 160 | 320 | 3750 | 4284 | 3750 | 3844 | 3748 |
| HiDE | 160 | 320 | 3750 | 4284 | 3750 | 3844 | 3747 |
| O-LoRA | 160 | 320 | 3750 | 4284 | 3750 | 3842 | 3747 |
| MoELoRA | 35 | 70 | 3750 | 4284 | 3750 | 3844 | 3748 |
| LoRA-FT | 32 | 64 | 3750 | 4284 | 3750 | 3844 | 3748 |
Expert training and router calibration serve distinct roles: task experts are first trained independently from a shared base and retained, then frozen while the routers learn to combine them. The expert learning rate therefore governs task adaptation, whereas the gate learning rates below govern how these existing capabilities are used. The table lists the number of experts, LoRA parameters, and training learning rates for all four settings; for DCL–InternVL, it reports the actual peak learning rate during training.
| Setting | Independent Experts | LoRA Rank | LoRA Scaling α | Effective Global Batch Size | Expert Training Learning Rate |
|---|---|---|---|---|---|
| UCIT–InternVL | 6 | 16 | 32 | 64 | 1e−4 |
| UCIT–LLaVA | 6 | 16 | 32 | 16 | LoRA: 2e−4; visual projector: 2e−5 |
| DCL–InternVL | 5 | 16 | 32 | 32 | 1e−4 (peak) |
| DCL–LLaVA | 5 | 32 | 64 | 16 | 2e−5 |
After freezing the experts, calibration follows the same rule in all four settings: 128 source examples per seen task are processed one at a time for six epochs, with consistent learning rates for the gates and module coefficients across settings. The difference in total updates between UCIT and DCL arises from their final stages containing six and five tasks, respectively, rather than from separately chosen calibration epoch counts. The table therefore presents the common optimization rule and the calibration budget as the number of seen tasks increases.
| Final-stage Setting | Calibration Examples | Optimizer Updates (6 Epochs) | Visual and Language Gate Learning Rate | Module Coefficient Learning Rate | Weight Decay |
|---|---|---|---|---|---|
| UCIT–InternVL | 768 | 4,608 | 1e−4 | 2e−3 | 0 |
| UCIT–LLaVA | 768 | 4,608 | 1e−4 | 2e−3 | 0 |
| DCL–InternVL | 640 | 3,840 | 1e−4 | 2e−3 | 0 |
| DCL–LLaVA | 640 | 3,840 | 1e−4 | 2e−3 | 0 |
W2: How robust is performance across settings, and how should FAA and CAA be interpreted?
Thank you for suggesting robustness checks across backbones and benchmarks. Actual executed output consistently improves performance in the matched UCIT–LLaVA comparisons and achieves final performance close to the references in the listed InternVL comparisons; these results distinguish performance stability from relative advantage. Matched LLaVA comparisons favor output over bank mean and identity by 0.6789 and 0.6180 points. InternVL differences average −0.1050/−0.0933 over three independent UCIT calibration draws; DCL differences are −0.022/−0.050. Information helps when it distinguishes useful expert combinations.
Thank you for suggesting a comparable test on another benchmark. Retrained DCL–LLaVA experts share frozen projectors. Within-bank output–mean differences are +0.0099 and −0.0817. The differences have opposite signs but are both below 0.1 points in magnitude.
| DCL–LLaVA Expert Initialization | Observation | RS | Med | AD | Sci | Fin | FAA |
|---|---|---|---|---|---|---|---|
| Independent | Selected output | 70.3700 | 63.5546 | 53.1200 | 54.4293 | 90.9327 | 66.4813 |
| Independent | Bank mean | 70.5800 | 63.1665 | 53.1800 | 54.5579 | 90.8727 | 66.4714 |
| Previous-task LoRA | Selected output | 72.6700 | 63.3095 | 52.7200 | 54.3284 | 91.4432 | 66.8942 |
| Previous-task LoRA | Bank mean | 72.8000 | 63.2686 | 52.9400 | 54.3177 | 91.5532 | 66.9759 |
Thank you for suggesting that changes in calibration order be distinguished from changes in calibration examples. After resampling the calibration examples, executed-output FAA on the two UCIT–LLaVA draws is 75.0894 and 75.2250, a difference of only 0.1356 percentage points, indicating stability under the tested sampling changes. Both draws fix visual routing and experts, use 128 examples per task and 4,608 language updates, and share formal evaluation examples.
| UCIT–LLaVA Calibration Draw | Observation | ImageNet-R | ArxivQA | VizWiz-caption | IconQA | CLEVR-Math | Flickr30k | FAA |
|---|---|---|---|---|---|---|---|---|
| Draw 1 | Selected output | 90.9667 | 90.3333 | 62.7000 | 72.8667 | 77.8000 | 55.8700 | 75.0894 |
| Draw 1 | Bank mean | 91.2667 | 90.2667 | 61.1200 | 73.6000 | 78.1667 | 55.9400 | 75.0600 |
| Draw 2 | Selected output | 91.0333 | 91.1667 | 61.7400 | 73.1000 | 77.5000 | 56.8100 | 75.2250 |
| Draw 2 | Bank mean | 90.9000 | 91.2333 | 60.6200 | 72.6667 | 78.1333 | 56.9900 | 75.0906 |
Executed-output FAA is 75.1144 and 75.7856 for two new expert sets trained with the same recipe, a difference of 0.6711 points; the same-hardware comparison for the second set retains a 0.6600-point advantage over bank mean. Six experts per set train independently for one epoch. Calibration fixes visual routing, 768 examples, seed 20261112 and 4,608 updates; evaluation uses 3,000 examples per task. The first seed's +0.6283 difference also reflects hardware changes. Flickr30k differs by −0.2700 and +0.0700 points; the other five tasks improve.
| UCIT–LLaVA Expert-training Seed | Observation | ImageNet-R | ArxivQA | VizWiz-caption | IconQA | CLEVR-Math | Flickr30k | FAA |
|---|---|---|---|---|---|---|---|---|
| 20261113 | Selected output | 91.0667 | 89.8333 | 64.2200 | 70.2333 | 79.7333 | 55.6000 | 75.1144 |
| 20261113 | Bank mean | 90.9333 | 89.2000 | 62.7800 | 68.5000 | 79.6333 | 55.8700 | 74.4861 |
| 20261114 | Selected output | 91.9000 | 92.9000 | 64.1600 | 69.5000 | 79.5333 | 56.7200 | 75.7856 |
| 20261114 | Bank mean | 91.7333 | 91.9000 | 62.5700 | 68.7667 | 79.1333 | 56.6500 | 75.1256 |
Thank you for also asking whether these observations hold on another backbone. Across the three tested calibration draws and three InternVL weight sets, the ranges of final executed-output scores are 0.1050 and 0.3000 points, respectively, while the direction relative to bank mean varies by setting. InternVL weight-set differences from bank mean are −0.30/−0.11/−0.03 points; these are not same-recipe seed repeats.
Thank you for pointing out the different rankings under FAA and CAA. The two metrics measure performance after learning is complete and performance throughout the learning sequence, respectively; the ranking difference on DCL–LLaVA primarily reflects changes between early and final stages. Each DCL stage contributes one-fifth to CAA. CL-MoE leads the first four stages, by 7.45 and about 4.23 points in stages 1 and 2; ELGAR leads stage 5 by 5.49 but averages 1.98 lower in CAA. The final stage enables composition of five experts rather than one. Original final FAA is 63.5360, 0.4320 points above Many. With a unified finance prompt, reevaluated final FAA is 64.5412 versus 63.1045, a 1.4367-point difference; the original matrix and CAA remain separate. For deployment after all tasks have been introduced, FAA directly measures the relevant performance; for continuous deployment during learning, CAA additionally captures early-stage performance.
Thank You for Your Time and Thoughtful Review
Thank you for recognizing the execution-linked research question, the matched comparisons, and the performance of the complete system. To save you time, we have highlighted key conclusions and important points in bold throughout our responses. The additional results extend the stability analysis beyond calibration order to resampling and independently trained expert weights: executed-output FAA on UCIT–LLaVA differs by only 0.1356 points between two calibration draws and by 0.6711 points between two new sets of expert weights; the matched comparison using new weights on the same hardware also retains a positive observation-related improvement. We also clarify historical-example access, expert training, and stage-wise calibration, and distinguish final performance from performance throughout learning. Below, we address stability separately from the relative advantage of an observation, linking each conclusion to its experimental conditions.