acceptodds
Under review as a conference paper at ICLR 2027

DART: Distribution-Guided Adaptive Routing for Federated Continual Learning of Generative Vision-Language Models

Abstract

Federated continual learning (FCIL) enables vision-language models (VLMs) to continually learn from decentralized, evolving data while retaining previously learned knowledge. Although mixture-of-experts (MoE) adaptation supports task specialization, client heterogeneity and sequential updates can bias routing toward dominant or recent tasks, while updates to the shared backbone can degrade historical experts even when they are correctly selected. We propose DART, a Distribution-guided Adaptive RouTing framework that jointly addresses routing bias and backbone-induced forgetting. Clients summarize expert-specific routing inputs in a fixed reference space, while the server maintains per-expert historical distributions and integrates them with current client statistics. Balanced synthetic features sampled from these distributions calibrate a global router without replaying historical raw image and text examples. Meanwhile, historical experts are frozen and backbone updates are constrained to a bootstrapped residual subspace, reducing interference with previously learned representations. Extensive experiments show that DART consistently outperforms state-of-the-art baselines, with performance improvements of up to 7.18%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Federated Continual Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vb2oOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies federated continual instruction tuning of generative vision-language models, addressing both unreliable selection of historical projector experts and backbone updates that undermine their effectiveness. It proposes DART, which maintains expert-specific Gaussian summaries of routing inputs in a fixed feature space and calibrates a global router using balanced synthetic features. Historical projector experts remain frozen, while shared-backbone adaptation uses a fixed LoRA input factor constructed from aggregated gradients outside a growing protection subspace. Experiments cover task-related and capability-related Hom-FCIT benchmarks, with additional component ablations, routing diagnostics, historical-computation analyses, and communication and storage measurements.

Critical strengths
  1. The paper identifies a relevant coupled problem: retaining historical projector parameters requires both reliable expert selection and compatibility with the continually updated language backbone.

  2. The framework combines distributional routing memory with constrained shared-backbone adaptation, providing a concrete approach to historical expert reuse without retaining raw historical examples.

  3. The component ablations, routing diagnostics, and historical-output-drift measurements provide useful evidence about the complementary roles of router calibration and backbone protection.

Critical weaknesses

W1. The rationale for the protection thresholds could be clearer. Appendix A.4 specifies an initial weight-energy threshold of 0.6 and a gradient-energy threshold of 0.99, but does not explain how these values were chosen. Since these choices determine the size of the protected subspace and the directions available for subsequent adaptation, a brief discussion of their selection and the expected trade-off between historical-task retention and new-task adaptation would help clarify the method’s practical behavior.

W2. Insufficient detail to verify the reported communication total (DART): Section 5.3 and Appendix A.4–A.5 describe client uploads of mean parameter gradients at task boundaries. Appendix B.9 states that this traffic is included in the reported 168.95 GiB total, but the manuscript does not sufficiently specify the transmitted gradient representation, including tensor shapes, dtype, and any compression or sketching procedure. Without these details, the reported total and the associated communication advantage cannot be independently verified. Please specify the transmitted objects and provide a breakdown of the task-boundary communication included in the total. It should also be clear that the accuracy experiments use the same gradient representation and communication implementation assumed in this accounting.

W3. The practical benefit of synthetic-data router calibration over direct Gaussian routing requires clearer characterization: Table 11 provides a useful matched control that keeps the final backbone, expert bank, and distribution statistics fixed. The learned DART router achieves 63.40, compared with 63.51 for direct Gaussian-likelihood routing and 63.02 for soft mean-distance routing. The 0.11-point difference between DART and Gaussian routing is too small to establish a reliable ranking without uncertainty estimates. These results support the usefulness of distributional memory, but do not establish an additional accuracy advantage from synthetic sampling followed by 400 optimization steps. A compact learned router may nevertheless offer meaningful deployment benefits. The paper should clarify this contribution and directly quantify the accuracy–cost trade-off against Gaussian routing, including inference latency, deployed memory requirements, and the additional cost of synthetic sampling and router optimization.

Questions for authors
  1. How should the expert-balanced calibration in Equation (5) be interpreted or adjusted when deployment workloads contain substantially different task frequencies?

  2. Were the two protection thresholds inherited from prior work or selected empirically, and what considerations guided their choice?

5 comments
AuthorsOpenReview ↗

Response to Reviewer vb2o

Dear Reviewer vb2o, thank you for your time and effort in reviewing our submission. We would like to respectfully point out that the Paper Summary, W1, W4, and both Questions for Authors appear to refer to a different paper, FedAGP, whereas W2 and W3 correctly address our work. We believe some comments may have been inadvertently mixed up.

We would sincerely appreciate it if you could revisit our manuscript and reconsider your overall assessment and rating in light of our paper's actual contributions. Thank you again for your understanding and consideration. Below, we address the two relevant points, W2 and W3, in turn.

W2 (Communication accounting): Thank you for highlighting the need for more transparent communication accounting. We provide a detailed breakdown in Table 1, specifying the transmitted variables, tensor shapes, data types, transfer counts, and corresponding upload/download costs. At task boundaries, 224 gradient matrices are compressed into rank-32 factors Gℓ≈LℓRℓ⊤G_\ell\approx L_\ell R_\ell^\top, where Lℓ=Uℓ,32Sℓ,32L_\ell=U_{\ell,32}S_{\ell,32} and Rℓ=Vℓ,32R_\ell=V_{\ell,32}. Model parameters use BF16, whereas gradient factors and distribution statistics use FP32. Each gradient upload requires 305 MiB, contributing 23.83 GiB across 40 initial and 40 final uploads over eight tasks. The breakdown also accounts for LoRA factors, projectors and routing statistics. The total communication comprises 72.14 GiB uplink and 96.81 GiB downlink, yielding the reported 168.95 GiB.

To directly verify accuracy under the same compressed communication implementation, we reran DART using rank-32 gradient factors under Non-IID β=1\beta=1. The four-task setting achieves Last/Avg accuracies of 60.01%/64.33%, while the eight-task setting achieves 63.27%/74.85%. Compared with the original dense-gradient eight-task Last accuracy of 63.40%, compression reduces accuracy by only 0.13 percentage points. We will explicitly distinguish the dense-gradient and compressed results in the revised manuscript and associate the reported 168.95 GiB communication cost with the corresponding rank-32 implementation. These results demonstrate that the reported communication efficiency is achievable with minimal accuracy degradation under the evaluated setting.

Table 1. Measured communication breakdown over eight tasks with rank-32 gradient compression.

Transmitted variableTensor specificationDtypeTransfersTotal (GiB)
Client-to-server uploads
Initial gradient Lℓ,RℓL_\ell,R_\ell[do,ℓ,32],[di,ℓ,32][d_{o,\ell},32], [d_{i,\ell},32] (224 pairs)FP324011.9141
Final gradient Lℓ,RℓL_\ell,R_\ell[do,ℓ,32],[di,ℓ,32][d_{o,\ell},32], [d_{i,\ell},32] (224 pairs)FP324011.9141
LoRA BℓB_\ell160×[4096,32], 64×[11008,32]160\times[4096,32],\ 64\times[11008,32]BF1640032.4219
Projector PtP_t[4096,1024], [4096,4096], 2×[4096][4096,1024],\ [4096,4096],\ 2\times[4096]BF1640015.6311
Distribution μ,v,Q,λ\mu,v,Q,\lambda[5120], [5120], [5120,32], [32][5120],\ [5120],\ [5120,32],\ [32]FP324000.2594
Upload subtotal72.1406
Server-to-client downloads
Shared LoRA AℓA_\ell192×[32,4096], 32×[32,11008]192\times[32,4096],\ 32\times[32,11008]BF1636524.7729
Global LoRA BℓB_\ell160×[4096,32], 64×[11008,32]160\times[4096,32],\ 64\times[11008,32]BF1668155.1982
Global projector PtP_t[4096,1024], [4096,4096], 2×[4096][4096,1024],\ [4096,4096],\ 2\times[4096]BF1643116.8425
Download subtotal96.8137
Total communication168.9543

W3 (Router accuracy–cost trade-off): Thank you for this insightful comment. We agree that Table 11 does not establish an accuracy advantage of the learned router over direct Gaussian routing (63.40 vs. 63.51), and we do not claim superiority based on this small difference. DART's main contribution lies in distribution-guided historical-expert routing with bootstrapping-based orthogonal backbone updates to mitigate forgetting. The learned router is a model component in the implementation of this framework.

Regarding the accuracy–cost trade-off, Table 2 shows that both routing strategies achieve comparable accuracy, suggesting that both can effectively leverage distributional memory. However, our learned router offers substantial deployment advantages over direct Gaussian routing: approximately 81$\times$ lower inference latency (0.263 vs. 21.242 ms/sample) and 346$\times$ lower client-side storage requirements (0.156 vs. 53.93 MiB), at the expense of longer calibration/preprocessing time (4.07 vs. 1.62 s).

Table 2. Accuracy and CPU deployment costs of routing strategies.

MetricLearned RouterGaussian Routing
Last (%)63.401663.5118
Calibration/Preprocessing (s)4.07±0.254.07\pm0.251.62
Routing latency (ms/sample)0.26321.242
Client storage (MiB)0.15653.93

Thank you for pointing this out. I sincerely apologize for the copy-and-paste error that introduced comments from another paper into my review, and for the confusion this caused. I will correct the affected sections.

Your detailed communication breakdown and compressed-gradient rerun address my concern regarding W2. The additional latency and storage comparisons also address W3, and I accept these clarifications.

After re-reading the manuscript, I would appreciate clarification on the following point:

W1. Rationale for the protection thresholds: Appendix A.4 specifies an initial weight-energy threshold of 0.6 and a gradient-energy threshold of 0.99, but does not explain how these values were chosen. Since these choices determine the protected subspace’s size and the directions available for subsequent adaptation, could you clarify whether they were inherited from prior work or selected empirically? A brief discussion of the selection rationale and the trade-off between historical-task retention and new-task adaptation would be helpful.

If this concern is satisfactorily addressed, I will consider increasing my score.

AuthorsOpenReview ↗

Response to Reviewer

We thank the reviewer for the prompt response and additional comment.

Our threshold choices are motivated by established projection-based continual learning methods. In particular, KeepLoRA [ICLR 2026] adopts an initial energy threshold of 0.6 for pretrained weights and a historical-feature protection threshold of 0.99 in its LLaVA experiments. Related methods, including GPM [ICLR 2021], SGP [AAAI 2023], and InfLoRA [CVPR 2024], also employ high energy-retention thresholds (0.95–0.999) to preserve historical knowledge. Following these practices, we set the initial weight threshold to 0.6 to protect the dominant pretrained directions while preserving sufficient residual capacity for adaptation, and the historical gradient protection threshold to 0.99 to balance knowledge retention and new-task plasticity.

To further validate the latter choice, we conduct a small sensitivity analysis using 200 samples and an existing model checkpoint, fixing the initial weight threshold at 0.6 while varying the historical gradient protection threshold among 0.95, 0.99, and 0.999. We evaluate four complementary metrics:

  • New task loss: Loss on the new task after adaptation, measuring new-task fitting (lower is better).
  • Old Δ\Deltaloss: Increase in historical-task loss after adaptation, measuring interference with previously learned knowledge (lower is better).
  • Storage: Memory required for the accumulated protection subspace (lower is better).
  • Min. free dims: Minimum remaining dimensions in the orthogonal complement of the protection subspace across all adapted layers, measuring available adaptation capacity (higher is better).

Table 1. Sensitivity to historical protection thresholds.

ThresholdNew task loss ↓Old Δ\Deltaloss ↓Storage (GiB) ↓Min. free dims ↑
0.950.5717+0.0172813.355563
0.990.5757+0.0051813.502527
0.9990.5805+0.0029613.991403

As shown in Table 1, stronger protection consistently reduces historical-task interference, but at the cost of increased storage and reduced adaptation capacity. Importantly, the threshold of 0.99 reduces historical-task loss degradation by approximately 70% compared with 0.95 (0.00518 vs. 0.01728), with only a marginal increase in new-task loss (0.5757 vs. 0.5717). Compared with 0.999, it achieves lower new-task loss (0.5757 vs. 0.5805), reduces protection storage by 0.489 GiB, and preserves 124 additional free dimensions, while maintaining a small historical-task loss increase of 0.00518. These results provide additional empirical support for selecting 0.99 as a reasonable balance among historical knowledge retention, new-task plasticity, and resource efficiency in the evaluated setting. They also suggest that jointly considering these four metrics provides a practical way to assess threshold choices for different continual learning scenarios. We have added this sensitivity analysis and discussion to Appendix A.4 of the revised manuscript.

We hope these additional results and clarifications address the reviewer's concerns and support a more favorable assessment of our paper. Thank you again for your time and consideration.

Reviewer_iTj3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DART for federated continual instruction tuning of generative vision-language models. It addresses two sources of forgetting: unreliable selection of historical projector experts and changes to the shared language backbone that affect those experts' downstream computations. Clients summarize routing descriptors in a frozen feature space, and the server retains task-specific distribution statistics to generate balanced synthetic features for router calibration. Historical projector experts are frozen, while backbone adaptation uses a shared, fixed LoRA input factor constructed in the orthogonal complement of an expanding protection subspace. The paper provides a conditional bound on historical computation drift and evaluates the framework on task-related and capability-related federated instruction-tuning benchmarks, with additional routing and representation diagnostics.

Critical strengths
  1. A coherent treatment of two complementary sources of forgetting. DART recognizes that retaining historical projector parameters and selecting the appropriate expert are insufficient if the shared decoder changes substantially. Combining distribution-based routing memory with constrained backbone updates addresses both issues within a consistent framework. The fixed routing coordinates also make historical statistical summaries reusable without retaining raw historical examples.

  2. Useful empirical support for the proposed mechanisms. The component ablations support the benefits of historical distribution memory and orthogonal adaptation: on the eight-task benchmark, the complete method achieves a Last score of 63.40, compared with 56.49 for uniform mixing and 58.20 for calibrated routing with unconstrained backbone updates. The historical-computation diagnostic further reports a reduction in mean relative hidden-state drift from 0.1149 to 0.0141. The appendix acknowledges that gradient protection does not guarantee complete activation coverage and supplements the analysis with comparisons against random subspaces of identical rank. This combination of evidence and explicit qualification strengthens the empirical case for the framework.

Critical weaknesses
  1. The plasticity and storage consequences of an expanding protection subspace are not sufficiently evaluated. DART progressively adds directions to the historical protection subspace and restricts subsequent updates to its orthogonal complement. While this promotes retention, it also reduces the directions available for learning new tasks. Appendix A.4 explicitly states that the current construction fails when fewer than the required LoRA rank of free directions remain. Appendix B.9 further reports that protection bases already occupy approximately 14.02 GiB of server-side storage after eight tasks, accounting for most of the additional saved state. The current experiments establish effectiveness on the tested sequences, but do not sufficiently characterize layer-wise rank growth, remaining adaptation capacity, or the impact of stronger protection on subsequent task acquisition. Consequently, the tradeoff among retention, plasticity, and resource requirements over longer task sequences remains unclear.

  2. The value of distribution memory is supported more clearly than the additional value of the trained router-calibration procedure. In Appendix Table 11, direct Gaussian-likelihood routing using the same saved statistics achieves a Last score of 63.51, compared with 63.40 for DART. This small difference does not establish a significant advantage for either approach, but it leaves unclear how much additional benefit comes from synthetic feature sampling and router optimization beyond retaining and using historical distribution information. A compact trained linear router may offer deployment advantages over evaluating Gaussian likelihoods directly, yet these advantages are not sufficiently quantified. Since router calibration is a central contribution, a clearer comparison of its additional computation against its accuracy, inference, or deployment benefits would strengthen the case for this particular design.

Questions for authors
  1. Protection-space growth and long-term plasticity. Can the authors report how the protection rank and remaining free dimensions evolve across layers and tasks, together with their relationship to new-task performance? How would the method continue learning once fewer than the specified LoRA rank of free directions remain?

  2. Protection-threshold selection and sensitivity. How were the pretrained-weight protection threshold of 0.6 and the historical-gradient energy threshold of 0.99 selected? How sensitive are historical-task retention, new-task acquisition, and protection-space size to these choices? This would help establish whether the reported benefits depend on a large protection-space budget.

  3. Additional benefits of trained router calibration. Given the comparable performance of direct Gaussian-likelihood routing in Table 11, can the authors compare the two approaches in calibration cost, inference latency, and client-side deployment storage? What is the principal measurable advantage of the trained calibration procedure, and under which conditions is it preferable?

  4. Task-boundary gradient communication. Equations (6) and (7) in Section 5.3 require the server to aggregate layer-parameter gradients at task boundaries to construct the update directions and protection space. Do clients transmit full gradient matrices or compressed representations? Please clarify how these uploads are included in the total communication reported in Figure 6, including their precision and transmission frequency. If compression is assumed, what is the scheme, and were the main accuracy results obtained under the same communication protocol?

5 comments
AuthorsOpenReview ↗

Response to Reviewer — Part 1: Protection Space Growth

Thank you for your constructive comments. We address all weaknesses (Ws) and questions (Qs) below. If our responses adequately address your concerns, we would be grateful if you could consider raising your overall rating to reflect stronger support for our paper.

W1, Q1. (Protection Space Growth): Thank you for highlighting the important trade-off between protection-space growth, plasticity, and storage. To quantify these effects, we analyzed the protection rank and remaining free dimensions across all 224 LoRA modules over eight tasks, yielding 1,792 layer-wise records. As shown in Table 1, the average protected ratio increases from 22.35% to 57.42%, while the minimum remaining free dimensions decrease from 2,546 to 466. Server-side protection storage correspondingly increases from 4.88 to 14.02 GiB. Importantly, all modules retain at least 466 free dimensions after eight tasks, substantially exceeding the required LoRA rank of 32.

To further examine the effects on new-task learning and historical-task retention, we conducted a controlled 16-step training experiment at Task 8, comparing orthogonal and unconstrained backbone updates while fixing the correct expert selection. As shown in Table 2, on the new task (super-CLEVR), orthogonal updates reduce the new-task loss from 5.2513 to 1.3726, while unconstrained updates reduce it to 0.5973. Conversely, across the seven historical tasks, orthogonal updates increase the average old-task loss only from 0.5561 to 0.5575, whereas unconstrained updates increase it to 0.7202. These results quantify the short-term stability–plasticity trade-off: orthogonal protection limits the speed of new-task learning but substantially mitigates performance degradation on historical tasks. Regarding long-term plasticity, we clarify that the current fixed-rank implementation becomes infeasible when fewer than the specified LoRA rank of free directions remain. One possible extension is to adaptively reduce the LoRA rank according to the remaining free dimensions.

Table 1. Protection subspace growth across 224 LoRA modules.

TaskProtected ratio (%)Min. free dimStorage (GiB)
122.3525464.88
328.3517056.35
548.1871011.10
857.4246614.02

Table 2. New-task learning and historical-task retention after 16 training steps at Task 8. Lower loss is better.

Update strategyNew-task lossAvg. old-task loss
Before training5.25130.5561
Orthogonal1.37260.5575
Unconstrained0.59730.7202
AuthorsOpenReview ↗

Response to Reviewer — Part 2: Threshold Sensitivity

Q2. (Threshold Sensitivity): We thank the reviewer for the constructive feedback. Our threshold choices are motivated by established projection-based continual learning methods. In particular, KeepLoRA [ICLR 2026] adopts an initial energy threshold of 0.6 for pretrained weights and a historical-feature protection threshold of 0.99 in its LLaVA experiments. Related methods, including GPM [ICLR 2021], SGP [AAAI 2023], and InfLoRA [CVPR 2024], also employ high energy-retention thresholds (0.95–0.999) to preserve historical knowledge. Following these practices, we set the initial weight threshold to 0.6 to protect the dominant pretrained directions while preserving sufficient residual capacity for adaptation, and the historical gradient protection threshold to 0.99 to balance knowledge retention and new-task plasticity.

To further validate the latter choice, we conduct a small sensitivity analysis using 200 samples and an existing model checkpoint, fixing the initial weight threshold at 0.6 while varying the historical gradient protection threshold among 0.95, 0.99, and 0.999. We evaluate four complementary metrics:

  • New task loss: Loss on the new task after adaptation, measuring new-task fitting (lower is better).
  • Old Δ\Deltaloss: Increase in historical-task loss after adaptation, measuring interference with previously learned knowledge (lower is better).
  • Storage: Memory required for the accumulated protection subspace (lower is better).
  • Min. free dims: Minimum remaining dimensions in the orthogonal complement of the protection subspace across all adapted layers, measuring available adaptation capacity (higher is better).

Table 1. Sensitivity to historical protection thresholds.

ThresholdNew task loss ↓Old Δ\Deltaloss ↓Storage (GiB) ↓Min. free dims ↑
0.950.5717+0.0172813.355563
0.990.5757+0.0051813.502527
0.9990.5805+0.0029613.991403

As shown in Table 1, stronger protection consistently reduces historical-task interference, but at the cost of increased storage and reduced adaptation capacity. Importantly, the threshold of 0.99 reduces historical-task loss degradation by approximately 70% compared with 0.95 (0.00518 vs. 0.01728), with only a marginal increase in new-task loss (0.5757 vs. 0.5717). Compared with 0.999, it achieves lower new-task loss (0.5757 vs. 0.5805), reduces protection storage by 0.489 GiB, and preserves 124 additional free dimensions, while maintaining a small historical-task loss increase of 0.00518. These results provide additional empirical support for selecting 0.99 as a reasonable balance among historical knowledge retention, new-task plasticity, and resource efficiency in the evaluated setting. They also suggest that jointly considering these four metrics provides a practical way to assess threshold choices for different continual learning scenarios. We have added this sensitivity analysis and discussion to Appendix A.4 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer — Part 3: Router and Communication

W2, Q3. (Router Efficiency): Thank you for the comment. Our Calibrated Router is a deployment-efficient implementation of DART's distribution-guided routing framework, rather than a mechanism intended to outperform all alternative routing strategies in accuracy. Nevertheless, as shown in Table 1, the Calibrated Router and direct Gaussian routing achieve comparable Last scores of 63.40% and 63.51%, respectively. We further measure their computational and storage costs, as shown in Table 1. The Calibrated Router reduces single-sample routing latency from 21.242 to 0.263 ms (approximately 80.8×80.8\times faster) and client-side routing storage from 53.93 to 0.156 MiB (approximately 345.7×345.7\times smaller). The additional server-side calibration requires only 4.074.07 seconds. These results demonstrate that the Calibrated Router substantially reduces client-side inference latency and storage requirements while maintaining comparable accuracy.

Table 1. Accuracy and deployment costs of routing strategies.

MetricCalibrated RouterGaussian Routing
Last (%)63.401663.5118
Calibration/Preprocessing (s)4.074.071.62
Routing latency (ms/sample)0.26321.242
Client storage (MiB)0.15653.93

Q4. (Gradient Communication): Thank you for the reviewer's comment. We clarify that the original main experiments use dense task-boundary gradients, whereas the reported 168.95 GiB communication cost corresponds to a rank-32 compressed-gradient implementation.

Specifically, each client compresses 224 gradient matrices into rank-32 factors Gℓ≈LℓRℓ⊤G_\ell\approx L_\ell R_\ell^\top, where GℓG_\ell denotes the parameter gradient matrix of LoRA-adapted module ℓ\ell, Lℓ=Uℓ,32Sℓ,32L_\ell=U_{\ell,32}S_{\ell,32}, and Rℓ=Vℓ,32R_\ell=V_{\ell,32}. Gradient factors use FP32, while model parameters use BF16. Each client uploads 305 MiB per gradient probe at both task start and task end. Across eight tasks with five participating clients, these 80 uploads total 23.83 GiB. As shown in Table 2, the overall communication includes 72.14 GiB of uploads and 96.81 GiB of downloads, totaling 168.95 GiB.

To verify accuracy under the same compressed communication protocol, we reran DART using rank-32 gradient factors under Non-IID β=1\beta=1. The four-task and eight-task Last/Avg scores are 60.01%/64.33% and 63.27%/74.85%, respectively. Compared with the original dense-gradient eight-task Last of 63.40%, compression reduces performance by only 0.13 percentage points. We have clarified the distinction between the two implementations in the revised manuscript and provided the detailed communication breakdown in Appendix B.9.

Table 2. Task-boundary gradient communication and total communication over eight tasks.

Transmitted objectDtypeUploadsTotal (GiB)
Task-start gradient factorsFP324011.9141
Task-end gradient factorsFP324011.9141
Other uploadsBF16/FP32--48.3124
Upload subtotal72.1406
DownloadsBF16--96.8137
Total communication168.9543
Reviewer_mYZdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes dart: distribution-guided adaptive routing for federated continual learning of generative vision-language models. The proposed method trains experts of projections and the B matrix of LoRA during local training/fine-tuning, calibrating the router in the aggregation phase by the global server, and bootstrapping the shared B matrix during the training by a local client. This paper discusses an interesting topic, i.e., federated continual instruction tuning. The reported results show a significant margin in comparison to the existing works. However, this paper has several substantial issues that need clarification. Please see weaknesses and questions.

Critical strengths
  1. This paper is well motivated, i.e., a routing error in experts-based methods for the federated continual learning problem.

  2. Looking at the reported results, the proposed method seems to have a promising contribution for the problem.

  3. The paper presents several visualization figures to help the reader understand the paper's technical substance.

Critical weaknesses
  1. In general, this writing is hard to follow; it has many unclear symbols such as Z, Q, etc.

  2. From my perspective, I don’t see a strong novelty in this paper that is enough for a top venue such as ICLR. The proposed experts routing, calibration, and bootstrapping are somewhat common.

  3. Continual learning is a study of handling catastrophic forgetting. This paper does not include an analysis of how the proposed method can handle/minimize forgetting, or at least an analysis of stability and plasticity.

  4. The ablation study is not intuitive, and it needs to be revised in a proper way, i.e., how each component contributes to the performance.

  5. The reported results are mainly taken from DISCO, which may run on different settings and environments. Thus, the validity of the comparison is questionable.

Questions for authors
  1. Method Clarity 1: The proposed method tries to address routing error to select the correct expert. However, Eq. 1 shows that the method weighted sum all the experts. Thus, it utilizes all the experts, rather than selecting a correct expert at a time. So, do you want to select an expert or a weighted sum of the experts? Please clarify.

  2. Method Clarity 2: The proposed methods train a LoRA matrix (B) along with the projector experts. It is not clear why only the B matrix is trained. Why is only the B matrix being trained? How do you initialize A and B matrices in each client? If matrix A is initiated with random numbers, then does it work if it is kept frozen? In addition, why projectors for the vision encoder and LoRA for the language model? Why not LoRA or projectors for both the vision encoder and the language model?

  3. As shown in Figure 2, each client shares P, B, G, mu, and sigma with the server, then receives averaged P, B, A, and a calibrated router. Why bother calibrating the router in the server when the client does not share its router parameters (R, b) with the server? Isn’t it a waste to calibrate the router from zero with augmented samples from mu and sigma? While we can get a better aggregated router from the locally trained routers?

  4. What does "initial protection basis Q" actually mean? How does Eq. 6 work? How do you get Q_{t-1}? The "orth" function looks like a condition rather than a computation. What does residual space Q_s mean?

  5. Equation 7 is confusing. How can we get an A_t matrix from a SVD decomposed gradient projection?

  6. What is \omega_{c,t,m} in equation 8?

6 comments
AuthorsOpenReview ↗

Response to Reviewer — Part 1: Notation and Novelty

Apologies for the delayed response, as we have only just completed the additional experiments and manuscript revisions to address your concerns. Below, we address all weaknesses (Ws) and questions (Qs). If our responses adequately resolve your concerns, we would be grateful if you could reconsider your assessment and raise your overall rating. Thank you.

W1. (Notation and Clarity): We thank the reviewer for highlighting the lack of clarity in our notation and apologize for the insufficient explanations. We have revised the manuscript to explicitly define the key symbols and improve notational consistency throughout. In particular, ZtZ_t denotes the router-weighted visual representation generated by the projector experts and subsequently fed into the shared language model. QtQ_t represents the accumulated protection basis after task tt, initialized using the dominant singular directions of the pretrained weights and progressively expanded with principal directions extracted from task-end gradients via SVD. We have also clarified the task indices and routing notation (lines 144–158), as well as the construction and updating of the protection subspace (lines 275–306). All revisions are highlighted in blue in the revised manuscript.

W2. (Novelty): Thank you for raising this concern. While MoE routing and orthogonal updates are established techniques, DART's novelty lies in a targeted hybrid adaptation framework that addresses two coupled sources of forgetting in federated continual learning: (1) historical expert misselection caused by client heterogeneity and sequential task learning, and (2) shared backbone drift that disrupts historical computations even when the correct experts are selected.

DART addresses these challenges through module-specific adaptation. It employs task-specialized MoE experts in the VLM projector, where parameter drift is particularly substantial, and distribution-guided server-side calibration to improve historical expert selection. Meanwhile, bootstrapping-based orthogonal LoRA updates constrain interference in the shared LLM while preserving new-task plasticity. This design avoids maintaining task-specific experts throughout the entire LLM and balances the complementary stability–plasticity properties of MoE and orthogonal adaptation. Thus, DART's contribution is not the individual techniques but their principled integration to jointly address expert misselection and historical computation drift. We formalize the latter in Eq. (2) of the manuscript and substantiate the framework's effectiveness through component ablations and representation drift analyses in the experiments.

AuthorsOpenReview ↗

Response to Reviewer — Part 2: Forgetting, Plasticity, and Component Ablations

W3. (Forgetting and Plasticity): Thank you for highlighting the importance of analyzing catastrophic forgetting and the stability–plasticity trade-off. DART mitigates forgetting through two complementary mechanisms: server-side router calibration reduces historical expert misselection, while orthogonal LoRA updates suppress interference with historical task computations caused by shared backbone drift.

We have analyzed both components in the original manuscript, Fig. 4 analyzes the effectiveness of router calibration in improving expert selection accuracy (89.37% to 98.46%), while Fig. 5 examines the role of orthogonal updates in reducing mean relative representation drift across historical tasks (0.1149 to 0.0141).

Following the continual learning evaluation framework of GEM [NeurIPS 2017], we further added a stability–plasticity analysis using mean task-end performance (PP) and backward transfer (BWT). With Ri,jR_{i,j} denoting task-jj performance after learning task ii, we define P=18∑t=18Rt,tP=\frac{1}{8}\sum_{t=1}^{8}R_{t,t} and BWT=17∑t=17(R8,t−Rt,t)\mathrm{BWT}=\frac{1}{7}\sum_{t=1}^{7}(R_{8,t}-R_{t,t}).

As shown in Table 1, in the baseline comparison, DART maintains comparable new-task learning performance to DISCO (PP: 63.49 vs. 64.78) and outperforms O-LoRA (63.49 vs. 56.33), while substantially reducing historical-task forgetting (BWT: −0.10-0.10 vs. −9.78-9.78/−12.36-12.36). In the component ablation, current-only calibration achieves comparable new-task learning performance (PP: 64.31 vs. 63.49), but BWT decreases from −0.10-0.10 to −9.95-9.95. Similarly, removing orthogonal updates reduces PP from 63.49 to 60.11 and BWT from −0.10-0.10 to −5.13-5.13. These results demonstrate that both historical distribution-aware calibration and orthogonal LoRA updates effectively mitigate forgetting while maintaining new-task learning capability.

More detailed task-wise stability–plasticity analyses have been added to the revised manuscript (lines 480–485 and 498–510).

Table 1. Stability–plasticity analysis on eight-task Hom-FCIT with β=1\beta=1.

GroupMethodPPBWT
Baseline comparisonDISCO64.78−9.78-9.78
O-LoRA56.33−12.36-12.36
DART63.49-0.10
Component ablationCurrent-only calibration64.31−9.95-9.95
w/o Orthogonal updates60.11−5.13-5.13
DART63.49-0.10

W4. (Component Ablations): We have revised Table 2 in the original manuscript (with the corresponding results below) and the accompanying descriptions to explicitly distinguish the two key mechanisms of DART: distribution-guided global router calibration and orthogonal backbone bootstrapping.

Table 2. Component ablations on eight-task Hom-FCIT. Historical calibration uses both current and historical distributions.

StrategyHist. calib.Orth. bootstrapLast
Average Mixing✓56.49
Current-only Calibration✓55.13
Calibration w/o Orth.✓58.20
Orth. w/o Calibration✓46.57
DART✓✓63.40

For expert routing, Average Mixing assigns equal weights to all historical projector experts, while Current-only Calibration trains the router using only current-task distributions, excluding historical distributional memory. For backbone adaptation, Calibration w/o Orthogonality retains global router calibration but replaces orthogonal updates with unconstrained LoRA, whereas Orthogonality w/o Calibration retains orthogonal bootstrapping but removes router calibration.

As shown in Table 3, full DART achieves 63.40 Last, outperforming Average Mixing (56.49) and Current-only Calibration (55.13) by 6.91 and 8.27 points, respectively, demonstrating the benefits of calibrated expert routing and historical distributional memory. With router calibration retained, orthogonal bootstrapping improves Last from 58.20 to 63.40 (+5.20 points). Conversely, removing router calibration reduces Last to 46.57 (-16.83 points), despite retaining orthogonal bootstrapping. These ablations clarify the individual contributions and complementary roles of the two mechanisms.

AuthorsOpenReview ↗

Response to Reviewer — Part 3: Comparison Fairness

W5. (Comparison Fairness): Thank you for raising this important concern regarding comparison fairness. The original manuscript already aligns the main federated settings with DISCO, including 50 clients, 5 participating clients per round, 10 communication rounds per task, 1 local epoch, FedAvg aggregation, and random seed 42. However, DART employs projector-based MoE with orthogonal LoRA updates to the shared LLM backbone, whereas DISCO maintains task-specific LoRA subspaces within the LLM and selectively activates them during inference. Therefore, matching LoRA ranks and insertion positions does not necessarily imply identical trainable capacity or parameter budgets.

To further evaluate DART under more stringent configuration-aligned comparisons, we reran DISCO and O-LoRA on the same hardware (two NVIDIA A800 80GB GPUs) and conducted two configuration-alignment experiments on Hom-FCIT (β=1\beta=1). We matched LoRA rank, target modules, insertion positions, and effective batch size while retaining each method's underlying adaptation mechanism. Detailed configurations are provided in Table 3.

As shown in Table 4, under DISCO's original configuration (rank 8, FFN-only, 96 insertion positions), DART achieves Last improvements of 2.18/2.50 percentage points over the rerun DISCO on the four-stage/eight-task benchmarks (57.19/57.82 vs. 55.01/55.32). Conversely, when DISCO adopts DART's rank and module coverage (rank 32, Attention+FFN, 224 insertion positions), DART still achieves Last improvements of 1.51/0.89 points (59.55/63.40 vs. 58.04/62.51). DART also consistently outperforms the rerun O-LoRA baseline.

These results demonstrate that DART's improvements persist under both configuration-alignment conditions and are not solely attributable to a larger LoRA rank or broader module coverage. Although the methods retain their respective adaptation mechanisms and method-specific parameters, the additional experiments strengthen the fairness and reliability of our comparisons. We will incorporate the completed configuration-alignment experiments into the revised manuscript and clarify the corresponding experimental settings and comparison protocols.

Table 3. Configuration comparison for the additional experiments on Hom-FCIT (β=1\beta=1).

SettingDISCO (Orig.)DART (DISCO)DISCO (DART)DART (Orig.)
LoRA rank883232
Target modulesFFNFFNAttn+FFNAttn+FFN
LoRA positions9696224224
Effective batch1281286464
LoRA scaling2220.05–0.25
LoRA LR2×10−42\times10^{-4}2×10−42\times10^{-4}2×10−42\times10^{-4}2×10−42\times10^{-4}
Projector LR2×10−52\times10^{-5}2×10−52\times10^{-5}2×10−52\times10^{-5}2×10−52\times10^{-5}
LoRA trainingA+BB onlyA+BB only

Table 4. Baseline comparisons on Hom-FCIT (β=1\beta=1). Results are Last/Avg. †\dagger denotes published DISCO results.

MethodFour-stageEight-task
O-LoRA (rerun)47.93/56.6745.52/65.97
DISCO (published)†^\dagger55.47/62.0756.22/73.03
DISCO (original, rerun)55.01/61.7855.32/71.95
DART (DISCO settings)57.19/62.8657.82/73.93
DISCO (DART settings)58.04/62.9962.51/74.34
DART (original)59.55/64.8763.40/74.97

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank the AC and all reviewers for their constructive feedback on our proposed Distribution-guided Adaptive RouTing (DART) framework. We have carefully addressed the major concerns through additional experiments, detailed clarifications, and manuscript revisions (highlighted in blue). The key points are summarized below:

  • Novelty and methodological clarity: Clarified DART's complementary mechanisms for mitigating historical expert misselection and shared-backbone interference.

  • Catastrophic forgetting and ablations: DART achieves BWT of −0.10-0.10 versus −9.78-9.78 for DISCO. Component ablations further validate the contributions of router calibration and orthogonal adaptation.

  • Baseline fairness: Additional controlled experiments under aligned configurations confirm DART's improvements over DISCO by 0.890.89–2.502.50 percentage points.

  • Router calibration: Server-side calibration achieves 98.7698.76% routing accuracy. The learned router provides 81×81\times faster inference than Gaussian routing while maintaining comparable accuracy.

We believe these additional results strengthen the evidence for DART's contributions and address the main concerns raised during the review. We hope our rebuttal and revised manuscript support a favorable final assessment. Thank you again for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.