acceptodds
Withdrawn from ICLR 2027

From Discrete Specialists to Continuous Expertise: Continual Adaptation of Medical MLLMs via Generative LoRA

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Medical multimodal large language models (MLLMs) should continually acquire capabilities across heterogeneous clinical tasks while retaining prior expertise. Existing parameter-efficient approaches face a fundamental trade-off: a single shared adapter risks interference, whereas expanding pools of task-level specialists increase storage and require expert routing. Both represent expertise at a fixed granularity, limiting scalable adaptation to individual clinical cases. We introduce C-MAGE, a framework that moves from discrete specialists toward continuous expertise by learning a shared generative rule for input-specific adaptation. Conditioned on image-instruction features, clinical department embeddings, and layer and projection identifiers, a shared generator predicts compact coefficients that combine shared basis matrices into instance-specific LoRA factors for the query and value projections of a frozen MLLM. This formulation integrates clinical priors, case-specific variation, and parameter reuse within a unified adaptation space. The framework is trained end-to-end through response prediction on current data and a small task-balanced replay memory, requiring neither pre-optimized expert checkpoints nor expert routing at inference. For evaluation, we construct MedCLAS, a clinically organized benchmark comprising 151,187 samples from 13 datasets across six departments and unifying classification, visual question answering, and report generation through a common instruction-following interface. Experiments demonstrate strong performance across heterogeneous medical tasks, showing that continual adaptation can proceed by refining shared generative expertise rather than accumulating discrete specialists. Code will be made publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Continual Learning, Medical MLLM, Parameter-efficient adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HTGwOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes C-MAGE, a framework for continual adaptation of medical multimodal language models. A shared hypernetwork generates instance-specific LoRA factors through shared basis matrices, conditioned on image-instruction features and department metadata. The authors also introduce MedCLAS, which organizes 13 datasets across six departments into a continual-learning benchmark. Experiments compare C-MAGE with continual-learning baselines and examine its components, task orders, replay budgets, and backbones.

Critical strengths
  1. Useful benchmark organization. MedCLAS brings classification, visual question answering, and report generation into a common continual-learning setting across medical specialties, enabling evaluation of heterogeneous capability acquisition and retention.

  2. Concrete evidence for the proposed parameterization. The shared-basis formulation substantially outperforms direct LoRA-factor generation in the reported ablation. Additional evaluations across task orders, replay budgets, and backbones help assess its applicability beyond a single configuration.

Critical weaknesses

The methodological novelty is not sufficiently established. HyLoVQA[1] already applies sample-conditioned LoRA generation to continual VQA, Text-to-LoRA[2] supports training the generator directly through task supervision, and HyperLoRA[3] explores low-dimensional generation using basis matrices and predicted coefficients. Although the proposed factorization with bases shared across layers has concrete differences, the paper lacks sufficient discussion and comparison with these closely related approaches to establish its distinct methodological contribution.

[1] Wang, Y., Xiong, C., Qin, Z., Zhang, M., Xiao, K., & Li, Z. (2026). HyLoVQA: Dynamic Hypernetwork-Generated Low-Rank Adaptation for Continual Visual Question Answering. arXiv preprint arXiv:2605.22035.

[2] Charakorn, R., Cetin, E., Tang, Y., & Lange, R. T. (2025). Text-to-LoRA: Instant transformer adaption. In Proceedings of the 42nd International Conference on Machine Learning (Vol. 267, pp. 7485–7514). Proceedings of Machine Learning Research.

[3] Li, M., Chen, J., Feng, W., Li, B., Dai, F., Zhao, S., & He, Q. (2025, June). Hyperlora: Parameter-efficient adaptive generation for portrait synthesis. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 13114-13123). IEEE.

Questions for authors
  1. Could authors clarify the methodological differences from HyLoVQA, Text-to-LoRA, and HyperLoRA, and explain which limitations of these approaches are addressed by sharing factorization bases across layers?

  2. Why do shared basis matrices yield such a substantial improvement? Is there further evidence distinguishing easier parameter generation from improved knowledge sharing?

  3. Do all baselines receive the same department information and use comparable training budgets? Were the main experiments repeated independently, particularly to establish whether the 1.06-point advantage over Replay without department conditioning is stable?

7 comments
AuthorsOpenReview ↗

Response#1-HTGw

We sincerely thank Reviewer HTGw for the constructive assessment. We appreciate your recognition of MedCLAS, the empirical support for shared-basis parameterization, and the additional sensitivity analyses. Regarding your concerns, we respond point by point below.

Concern 1: The distinctions from HyLoVQA, Text-to-LoRA, and HyperLoRA are insufficiently established. Which limitations are addressed by sharing bases across layers?

Response: Thank you for highlighting these approaches. C-MAGE’s methodological contribution is a clinically conditioned adaptation space that explicitly shares factor subspaces across layers while preserving case-specific specialization. This structure supports continual refinement across heterogeneous hospital services without accumulating dataset-specific adapters. The distinction lies in how adaptations are represented, conditioned, and continually shared—not simply in using a hypernetwork.

Regarding the closest relevant one, HyLoVQA generates LoRA adapters from retrieved visual-object and textual-task anchors, with semantic-functional alignment for continual VQA. C-MAGE instead combines multimodal case features and department priors to generate adaptations within cross-layer shared factor subspaces. We additionally evaluate HyLoVQA on MedCLAS under the same experimental protocol:

MethodFinal Avg. Score ↑BWT ↑
HyLoVQA27.74−8.81
C-MAGE68.16−6.34

C-MAGE improves the final Avg. Score by 40.42 points and BWT by 2.47 points. Even without department conditioning, C-MAGE achieves 65.87, showing that its advantage persists without the additional clinical prior. These results provide direct evidence beyond a qualitative distinction between architectures.

Regarding Text-to-LoRA, it supports adapter reconstruction, direct task supervision, and layer/module conditioning; we do not claim these mechanisms as new. Its original formulation learns task-description-conditioned adapters. C-MAGE learns a continually updated mapping from individual medical images, instructions, and department context to adaptations within shared factor subspaces. Consequently, cases within the same service can receive different adaptations according to their visual evidence and response requirements.

For HyperLoRA, it predicts coefficients over trainable LoRA matrix bases for portrait synthesis. Beyond the totally different task, C-MAGE instead introduces projection-specific factor bases shared across transformer layers, with coefficient matrices generated for each clinical case and adaptation site. Both the generator and bases are refined through sequential medical task supervision.

What cross-layer sharing addresses. Maintaining separate bases for every layer duplicates basis parameters and provides no explicit coupling through those bases. C-MAGE replaces these separate parameter sets with common trainable factor subspaces. Learning signals from different layers and successive tasks therefore update the same bases, while layer-conditioned coefficients preserve distinct adaptations at each site. For our 32-layer backbone, this reduces basis parameters 32-fold relative to a layer-specific implementation of the same factorization. The empirical evidence supports the value of structured generation: with image–instruction conditioning retained and department conditioning absent, shared-basis generation improves the final Avg. Score from 34.75 to 65.87. Adding the clinical prior further increases it to 68.16. This ablation evaluates the shared-basis formulation as a whole; the cross-layer storage reduction follows directly from its architecture. Together, these distinctions establish C-MAGE as a structured approach to continual medical adaptation, with explicit mechanisms for cross-layer parameter reuse, case-level specialization, and service-level clinical conditioning.

AuthorsOpenReview ↗

Response#2-HTGw

Concern 2: Why do shared basis matrices yield such a substantial improvement? Is there further evidence distinguishing easier parameter generation from improved knowledge sharing?

Response: Shared bases jointly simplify parameter generation and introduce explicit sharing among generated adaptations. The generator predicts low-dimensional coordinates within trainable factor subspaces, rather than every LoRA factor entry. Under our configuration, this requires 4,096 coefficients per adaptation site instead of 131,072 query-factor entries or 81,920 value-factor entries, revealing a 32-fold and 20-fold reduction, respectively. The bases also provide adaptation-subspace sharing beyond sharing the hypernetwork itself. The same projection-specific bases receive learning signals from different inputs, transformer layers, and successive tasks. Case- and layer-conditioned coefficients preserve distinct adaptations within these common subspaces. This makes parameter reuse explicit while retaining specialization for individual clinical cases.

The following comparisons provide complementary evidence:

ConfigurationFinal Avg. Score ↑
Direct factor generation34.75
Shared-basis generation without department conditioning65.87
Department-only conditioning62.80
Full C-MAGE68.16

With image-instruction conditioning retained and department conditioning absent, shared-basis generation improves performance by 31.12 points over direct generation. The department-only comparison further holds the generator, shared bases, coefficient dimensionality, and training protocol fixed. Introducing case-level conditioning improves the score by 5.36 points and BWT from −8.65 to −6.34. Thus, compact generation alone does not reproduce the full model’s performance; case-adaptive use of the shared space contributes additional gains. These results establish the effectiveness of the structured parameterization and the value of case-specific conditioning.

Concern 3: Do baselines have the same department information and comparable training budgets? Is the improvement without department conditioning stable across independent runs?

Response: The reported results are averages over three independent runs, and our comparisons use a common backbone, dataset splits, task sequence, evaluation protocol, and dataset-level training budget. Department conditioning is an explicit component of our hospital-oriented method, and its contribution is separately evaluated. Hospital-wide continual adaptation of medical MLLMs across heterogeneous specialties, modalities, and task types remains comparatively underexplored. This motivates exploiting the clinical structure available in hospital workflows: department context supplies a coarse service-level prior, while image-instruction features specify case-level adaptation requirements.

Thank you for the detailed responses and additional experiments, which address most of my concerns.

I now lean toward weak accept based on the clarified contribution, the additional ablation, and the benchmark contribution. However, I remain uncertain about the very large performance gap in the newly added HyLoVQA comparison, as the adaptation and implementation details are insufficient to assess its reliability.

I am therefore raising my rating while lowering my confidence to reflect this remaining uncertainty.

Reviewer_qUpmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces C-MAGE, a continual adaptation framework for medical multimodal large language models (MLLMs). It uses a shared hypernetwork conditioned on image-instruction features, clinical department embeddings, and layer/projection identifiers to generate instance-specific LoRA factors within shared basis spaces. A replay buffer is used to mitigate catastrophic forgetting while keeping the pretrained MLLM frozen. The authors also introduce MedCLAS, a benchmark comprising 151,187 samples from 13 datasets across six clinical departments, covering classification, visual question answering, and report generation. Experiments report an average score of 68.16 and backward transfer of −6.34, outperforming several continual-learning baselines. However, the methodological novelty, experimental fairness, and validity of the benchmark-level conclusions remain insufficiently established.

Critical strengths
  1. Continual adaptation across heterogeneous medical tasks is important, and the proposed benchmark covers multiple departments, modalities, and response formats.

  2. Shared basis matrices and input-conditioned LoRA coefficients offer a reasonable way to generate specialized adaptations without storing a separate adapter for every task.

Critical weaknesses
  1. C-MAGE combines a conventional hypernetwork, factorized parameter generation, department embeddings, and replay-based continual learning. The paper does not sufficiently establish a fundamental methodological advance over existing hypernetwork-based parameter-efficient adaptation. The reported ablation mainly validates the proposed design choices, rather than demonstrating that continuous expertise generation is intrinsically superior to simpler alternatives.

  2. The primary result relies on an equally weighted average across heterogeneous metrics, potentially obscuring substantial differences between tasks. Moreover, the replay buffer retains up to 200 samples per dataset, so its total size grows with the task stream; this weakens the scalability argument. The evaluation also uses only a limited set of task orders, while the dataset-level results show substantial variation. These limitations undermine the claims of robust continual learning and scalable adaptation.

Questions for authors
  1. What fundamental capability does C-MAGE provide that cannot be achieved by a standard input-conditioned hypernetwork generating LoRA parameters? Can the authors demonstrate a clear advantage under parameter-matched and compute-matched comparisons?

  2. Since the replay memory grows linearly with the number of observed datasets, how does the method perform under a fixed total memory budget as the task stream expands?

  3. Do the main conclusions hold under additional task permutations and when performance is reported separately for each task, rather than primarily through an equally weighted aggregate of heterogeneous metrics?

6 comments
AuthorsOpenReview ↗

Response#1-qUpm

We sincerely thank Reviewer qUpm for the detailed feedback and for recognizing the importance of heterogeneous medical continual learning, the breadth of MedCLAS, and the rationale for generating specialized adaptations without storing separate task-specific adapters. We address your concerns point by point.

Concern 1: Methodological contribution beyond standard input-conditioned hypernetworks, evidence against simpler alternatives, and fairness of resource comparisons.

Response: We respectfully disagree with the characterization of C-MAGE as merely a combination of established components. Its contribution is a structured adaptation space that makes parameter reuse explicit across clinical cases and transformer layers during continual learning. The evidence addresses both direct parameter generation and simpler department-specific adaptation.

C-MAGE generates compact, case-conditioned coefficients over projection-specific factor bases shared across layers. This couples adaptations through common trainable subspaces while preserving input-dependent variation. Under our configuration, coefficient generation requires 20-32 times fewer output values per adaptation site than direct factor generation. The practical value of this structure is directly tested:

ConfigurationFinal Avg. Score ↑
Direct factor generation with image-instruction conditioning34.75
Shared-basis generation without department conditioning65.87
Department-only conditioning62.80
Full C-MAGE68.16

With image-instruction conditioning retained and the department prior absent, shared-basis generation improves performance by 31.12 points over direct factor generation. Full C-MAGE also exceeds department-only conditioning by 5.36 points. Thus, neither direct parameter generation nor department-level specialization reproduces the performance of the proposed formulation. These comparisons establish a useful inductive bias for the evaluated continual-learning setting, rather than a claim of greater theoretical expressivity than a universal hypernetwork.

To make the positioning concrete, we identify Text-to-LoRA, HyperLoRA, and HyLoVQA as relevant antecedents. Text-to-LoRA supports task-description-conditioned adaptation and direct task supervision; HyperLoRA uses predicted coefficients and LoRA bases for portrait synthesis. We acknowledge these precedents. C-MAGE specifically organizes adaptation through cross-layer shared factor subspaces conditioned on multimodal clinical cases and department context. HyLoVQA provides the closest quantitative comparison because it studies anchor-conditioned LoRA generation for continual VQA. We additionally evaluate it on MedCLAS:

MethodFinal Avg. Score ↑BWT ↑
HyLoVQA27.74−8.81
C-MAGE68.16−6.34

C-MAGE improves the final Avg. Score by 40.42 points and BWT by 2.47 points. Its advantage persists without department conditioning, where it achieves 65.87. The gain therefore cannot be attributed solely to the additional clinical prior.

Note: Comparisons use the same backbone, dataset splits, task sequence, evaluation protocol, and dataset-level training budget. Historical-sample allowances are matched for methods using replay, and the main results average three independent runs. Together, the controlled ablations and closest-method comparison demonstrate substantial empirical benefits from C-MAGE’s structured generative adaptation, beyond using an input-conditioned hypernetwork alone.

Concern 2: The replay buffer grows with the number of datasets. Does this undermine scalability?

Response: C-MAGE’s scalability advantage is architectural: it supports additional datasets without expanding the adaptation model. For a fixed backbone, generator architecture, and department vocabulary, its persistent adaptation parameter count remains unchanged. Replay is a separately budgeted training resource. Our protocol retains up to 200 samples per dataset. Across the complete 13-dataset stream, the buffer therefore contains at most 2,600 samples, approximately 2.2% of the 118,525 training samples. This policy permits historical-sample retention while limiting the contribution from each dataset. The same per-dataset allowance applies to Replay, MR-LoRA’s router-training memory, and ProgLoRA’s task-recall memory; C-MAGE receives no additional replay capacity. The replay buffer grows linearly under this policy, and we explicitly distinguish that growth from adaptation checkpoint storage. C-MAGE eliminates the accumulation of dataset-specific adapters while operating within the same replay allowance as competing methods. Our scalability claim concerns this fixed-size adaptation parameterization, with replay storage accounted for separately. A fixed total buffer would impose an additional memory constraint and requires a separate evaluation.

AuthorsOpenReview ↗

Response#2-qUpm

Concern 3: Averaging heterogeneous metrics may obscure task-level variation, and the limited task-order study may not support robust conclusions.

Response: We respectfully disagree with the assessment that these concerns leave our main empirical conclusion unsupported. Our evaluation includes complete task-level results, alternative aggregation schemes, and multiple task orders, including a newly added pathology-first sequence. Together, these provide evidence beyond a single aggregate score or learning sequence.

Metric aggregation. Table 1 already reports the individual task metrics, allowing task-specific strengths and limitations to be assessed directly. The Avg. Score provides a compact summary of this heterogeneous stream; it does not replace the task-level evidence. To assess sensitivity to metric weighting, we recomputed C-MAGE’s aggregate performance using three schemes:

AggregationC-MAGE
Equal weight per metric slot68.16
Equal weight per dataset70.31
Equal weight per department69.74

These calculations use the rounded Table 1 entries. Dataset weighting first averages metrics within each dataset; department weighting first averages metric slots within each department. C-MAGE maintains comparable aggregate performance across all three schemes, showing that its performance summary remains stable when datasets or departments receive equal weight.

Task-order robustness. We expanded the original three-order study with Shuffled Order III, using the task sequence [4, 5, 6, 7, 0, 1, 2, 3, 8, 9, 11, 12, 10], which begins with pathology instead of radiology.

Task order1234567891011121314151617Avg. Score ↑
Standard77.0660.4671.8037.9865.1678.2183.9116.6278.7467.7233.6384.60100.0081.7660.4785.5375.0068.16
Shuffled I73.6654.9769.4835.0160.0975.7886.8223.9085.4360.7027.1478.49100.0087.6457.0976.7672.3266.19
Shuffled II71.2456.6765.1233.4961.0274.5482.3516.9671.1970.4845.1388.55100.0081.1053.0470.5772.3265.52
Shuffled III (pathology-first)78.7261.1573.2633.0066.4284.4682.9414.4379.1265.8229.7483.8399.8986.8858.7882.2468.7567.61

Across these four orders, the final Avg. Score spans 65.52-68.16. The pathology-first sequence achieves 67.61, only 0.55 percentage points below the standard order. Comparable final aggregate performance therefore does not depend on radiology-first training. The detailed results also expose order-dependent variation: relative to the standard order, Shuffled III improves seven metric entries and decreases ten. We explicitly distinguish this variation in individual capabilities from the stability of final aggregate performance. Our robustness conclusion concerns the latter, while the complete tables make the former transparent. Together, the task-level evidence, aggregation sensitivity checks, and expanded order study support effective continual adaptation across heterogeneous clinical tasks under the evaluated protocols.

Dear Reviewer qUpm,

We would also like to draw your attention to our follow-up discussion with Reviewer HTGw, who acknowledged our clarified contribution. Their detailed references prompted a closer examination of C-MAGE’s distinctions from related methods. That exchange provides additional context directly relevant to your novelty concern. Together with our controlled ablations, it highlights the methodological rationale and empirical benefits of the proposed design.

We would greatly appreciate your perspective on these clarifications and any remaining concerns, and would be happy to respond promptly.

Reviewer_t11iOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes C-MAGE for continual learning in medical multimodal large language models. It uses a shared generator and shared bases to produce LoRA adapters for individual inputs, with department information as an additional condition, and is trained with replay. The authors also build MedCLAS from public medical data. The benchmark includes 13 tasks across six medical departments and covers classification, visual question answering, and report generation. In the main experiment, C-MAGE achieves the highest final average score among the non-oracle baselines.

Critical strengths

1.MedCLAS combines different medical departments, imaging modalities, and task types through a common image–instruction–response interface. This provides a useful benchmark for continual learning across heterogeneous tasks.

2.The ablation supports the shared-basis design. With image–instruction conditioning retained, introducing shared bases improves the final average score with directly generating full LoRA factors.

3.In the main experimental setting, C-MAGE improves the final average score over Replay. Among the 17 non-BLEU metrics included in the average, 15 improve, one remains unchanged, and one declines, showing gains across multiple tasks.

Critical weaknesses

1.The ablations in Figure 4(a) do not isolate the contribution of instance conditioning, since all three variants retain image–instruction features in the generator. A control that removes these features while retaining department information, with the remaining architecture and training setup unchanged, would help clarify this contribution.

2.The task-order study does not include Replay, the strongest non-oracle baseline in Table 1, so it is unclear whether the gains over Replay hold across orders. Please include Replay under the same tested orders. All three orders also start with radiology and keep the dataset order within each department fixed. An order starting with another department would help assess sensitivity to the starting point.

3.Kvasir-VQA is listed as open-ended in Table 2, but its metric is labeled as accuracy in Table 1, while Appendix C.3 specifies recall for open-ended VQA. Please confirm which metric was used, clarify whether this is a labeling issue, and specify how VQA recall and report-generation F1 are calculated.

4.The claimed storage benefit is not quantified against methods that retain task-specific experts. Please report the total adaptation parameter count and storage size, including the generator, shared bases, and conditioning embeddings, and compare how these costs grow over the task sequence.

Questions for authors

see weekness

1 comment
AuthorsOpenReview ↗

Response#1-t11i

We sincerely thank Reviewer t11i for the constructive and specific feedback. We appreciate your recognition of MedCLAS, the effectiveness of the shared-basis design, and the improvements across multiple tasks. Below, we address your concerns point by point.

Concern 1: The current ablations do not isolate the contribution of instance conditioning.

Response: Thank you for this suggestion. The image-instruction representation enables case-adaptive LoRA generation, while the department embedding provides an auxiliary clinical prior. Figure 4(a) evaluates the shared-basis parameterization and the additional benefit of this prior, retaining instance conditioning throughout. To directly isolate the contribution of instance conditioning, we added a department-only variant that removes only the image-instruction features while preserving the generator, shared bases, department and layer/projection embeddings, and training protocol. Consequently, inputs from the same department receive identical LoRA factors at each layer and projection, enabling a controlled comparison between instance-specific and department-specific adaptation. As shown in the table below, full C-MAGE improves the final Avg. Score from 62.80 to 68.16 and BWT from −8.65 to −6.34. These results demonstrate that instance-level conditioning improves both final performance and knowledge retention beyond department-level specialization, establishing the value of case-adaptive LoRA generation.

OrderFinal Avg. Score ↑BWT ↑
Full C-MAGE68.16-6.34
C-MAGE (Department-prior only)62.80-8.65

Concern 2: Robustness to task order and the choice of starting department.

Response: Thank you for raising this point. To assess whether C-MAGE’s performance depends on radiology-first training, we added Shuffled Order III with the task sequence [4, 5, 6, 7, 0, 1, 2, 3, 8, 9, 11, 12, 10], which begins with pathology. C-MAGE maintains comparable final aggregate performance across the four evaluated orders, achieving Avg. Scores of 68.16, 66.19, 65.52, and 67.61. In particular, the pathology-first sequence finishes only 0.55 percentage points below the standard order, showing that strong final performance does not depend on starting with radiology. C-MAGE also consistently achieves higher final Avg. Scores than Replay in all completed matched-order comparisons. We will revise Fig. 4 to include these expanded comparisons and add a plot of the additional Shuffled Order III results.

OrderFinal Avg. Score ↑BWT ↑
Standard Order68.16-6.34
Shuffled Order I66.19-7.73
Shuffled Order II65.52-8.09
Shuffled Order III (Pathology department-first)67.61-9.21

These results support the robustness of C-MAGE’s final aggregate performance across the tested task sequences and starting departments. BWT and dataset-specific scores nevertheless vary across orders, indicating that the learning sequence influences how individual capabilities are retained.

Concern 3: Kvasir-VQA has inconsistent metric labels.

Response: Thank you for identifying this inconsistency. Kvasir-VQA is evaluated as an open-ended VQA task using recall, as specified in Appendix C.3. The ACC label in Table 1 was a typo, which we have corrected. We have also checked the corresponding metric labels throughout the manuscript to ensure consistency.

Concern 4: The storage advantage is not quantified against task-specific expert methods.

Response: We quantify storage by counting all persistent adaptation components: the generator, coefficient heads, conditioning embeddings, shared bases, and normalization parameters. The shared bases contribute ∑ρ∈{Q,V}k(din+dout,ρ)\sum_{\rho\in\{Q,V\}} k(d_{\mathrm{in}}+d_{\mathrm{out},\rho}) parameters. Generated coefficients and LoRA factors are temporary and do not contribute to persistent storage. Excluding the shared pretrained MLLM backbone, this complete accounting gives C-MAGE 6.24M adaptation parameters (11.90 MiB in BF16), slightly fewer than a single rank-16 Q/V LoRA adapter spanning all transformer layers (6.82M parameters, 13.00 MiB). Under this matched configuration, storing 13 task-specific experts requires 169.00 MiB—approximately 14.2 times C-MAGE’s adaptation weight storage, even before including routing modules. Adapting all seven attention and MLP projections increases each rank-16 adapter to 41.94M parameters (80.00 MiB), bringing a 13-expert bank to approximately 1.02 GiB, again excluding routing parameters. C-MAGE therefore supports continual adaptation with a persistent parameter footprint comparable to a single LoRA adapter, without accumulating task-specific experts or expert-selection modules.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary and Invitation for Further Discussion

We thank all three reviewers for their careful reading and constructive feedback. Across the rebuttal, we have directly addressed the central concerns about methodological contribution, component effectiveness, evaluation robustness, and storage scalability:

1. Methodological contribution and positioning (qUpm, HTGw). We clarified C-MAGE’s distinctions from HyLoVQA, Text-to-LoRA, and HyperLoRA, while explicitly acknowledging their relevant precedents. Our contribution is a structured and clinically conditioned adaptation space: projection-specific factor bases are shared across transformer layers, while case- and layer-conditioned coefficients preserve specialization. This enables continual refinement of shared expertise without accumulating task-specific adapters. The contribution lies in this formulation and its demonstrated benefits for heterogeneous medical continual learning.

2. Direct evidence for shared-basis generation and instance-level specialization (all reviewers). Our ablations address whether simpler alternatives reproduce C-MAGE’s performance:

  • Shared-basis parameterization: with instance conditioning retained and department conditioning absent, replacing direct factor generation with shared-basis generation improves the final Avg. Score from 34.75 to 65.87.
  • Instance-level adaptation: the newly added department-only control preserves the generator, shared bases, and training protocol. Restoring case-level conditioning improves the score from 62.80 to 68.16 and BWT from −8.65 to −6.34.
  • Clinical prior: department conditioning further improves shared-basis generation from 65.87 to 68.16.

Together, these comparisons establish complementary benefits from structured parameter generation, case-specific specialization, and clinical context. We also added a HyLoVQA comparison and subsequently provided dataset-level results for closer examination.

3. Evaluation robustness and transparency (t11i, qUpm, HTGw). We expanded the evaluation to four task orders, including a new pathology-first sequence. Final Avg. Scores range from 65.52 to 68.16, with the pathology-first result only 0.55 points below the standard order. We provided complete task-level results and alternative dataset- and department-weighted summaries, while explicitly acknowledging order-dependent variation in individual capabilities. We also clarified that the main results average three independent runs, described the common experimental protocol, and corrected the Kvasir-VQA metric label.

4. Quantified storage scalability and clarified replay accounting (t11i, qUpm). Counting all persistent adaptation components, C-MAGE requires 6.24M parameters / 11.90 MiB in BF16, compared with 169.00 MiB for 13 matched task-specific Q/V LoRA experts—approximately 14.2× less adaptation-weight storage. We explicitly separated this architectural advantage from replay memory: adaptation parameters remain fixed under the specified architecture and department vocabulary, whereas the evaluated per-dataset replay allowance grows with the stream.

We also respectfully highlight the reviewers’ positive assessments:

  • t11i recognized MedCLAS as a useful heterogeneous continual-learning benchmark, with evidence supporting our method design.
  • qUpm recognized the importance and breadth of the medical continual-learning setting and the rationale for generating specialized adaptations without separate task-specific adapters.
  • HTGw stated that our responses and additional experiments “address most of my concerns” and raised their rating based on the clarified contribution, additional ablation, and benchmark contribution. We subsequently supplied task-level results in response to their remaining uncertainty about the HyLoVQA comparison.

We believe this evidence strengthens both contributions: MedCLAS as a reusable testbed for heterogeneous medical continual learning, and C-MAGE as a compact framework for continually refining shared, case-adaptive expertise.

We hope this summary assists the reviewers and AC in assessing the clarified contribution and strengthened evidence. We would particularly welcome engagement from Reviewers t11i and qUpm in the discussion. We would greatly appreciate their perspectives on whether these clarifications and additional results address their concerns, and would be happy to respond promptly to any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.