- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies federated continual instruction tuning of generative vision-language models, addressing both unreliable selection of historical projector experts and backbone updates that undermine their effectiveness. It proposes DART, which maintains expert-specific Gaussian summaries of routing inputs in a fixed feature space and calibrates a global router using balanced synthetic features. Historical projector experts remain frozen, while shared-backbone adaptation uses a fixed LoRA input factor constructed from aggregated gradients outside a growing protection subspace. Experiments cover task-related and capability-related Hom-FCIT benchmarks, with additional component ablations, routing diagnostics, historical-computation analyses, and communication and storage measurements.
Critical strengths
-
The paper identifies a relevant coupled problem: retaining historical projector parameters requires both reliable expert selection and compatibility with the continually updated language backbone.
-
The framework combines distributional routing memory with constrained shared-backbone adaptation, providing a concrete approach to historical expert reuse without retaining raw historical examples.
-
The component ablations, routing diagnostics, and historical-output-drift measurements provide useful evidence about the complementary roles of router calibration and backbone protection.
Critical weaknesses
W1. The rationale for the protection thresholds could be clearer. Appendix A.4 specifies an initial weight-energy threshold of 0.6 and a gradient-energy threshold of 0.99, but does not explain how these values were chosen. Since these choices determine the size of the protected subspace and the directions available for subsequent adaptation, a brief discussion of their selection and the expected trade-off between historical-task retention and new-task adaptation would help clarify the method’s practical behavior.
W2. Insufficient detail to verify the reported communication total (DART): Section 5.3 and Appendix A.4–A.5 describe client uploads of mean parameter gradients at task boundaries. Appendix B.9 states that this traffic is included in the reported 168.95 GiB total, but the manuscript does not sufficiently specify the transmitted gradient representation, including tensor shapes, dtype, and any compression or sketching procedure. Without these details, the reported total and the associated communication advantage cannot be independently verified. Please specify the transmitted objects and provide a breakdown of the task-boundary communication included in the total. It should also be clear that the accuracy experiments use the same gradient representation and communication implementation assumed in this accounting.
W3. The practical benefit of synthetic-data router calibration over direct Gaussian routing requires clearer characterization: Table 11 provides a useful matched control that keeps the final backbone, expert bank, and distribution statistics fixed. The learned DART router achieves 63.40, compared with 63.51 for direct Gaussian-likelihood routing and 63.02 for soft mean-distance routing. The 0.11-point difference between DART and Gaussian routing is too small to establish a reliable ranking without uncertainty estimates. These results support the usefulness of distributional memory, but do not establish an additional accuracy advantage from synthetic sampling followed by 400 optimization steps. A compact learned router may nevertheless offer meaningful deployment benefits. The paper should clarify this contribution and directly quantify the accuracy–cost trade-off against Gaussian routing, including inference latency, deployed memory requirements, and the additional cost of synthetic sampling and router optimization.
Questions for authors
-
How should the expert-balanced calibration in Equation (5) be interpreted or adjusted when deployment workloads contain substantially different task frequencies?
-
Were the two protection thresholds inherited from prior work or selected empirically, and what considerations guided their choice?
5 comments
Thank you for pointing this out. I sincerely apologize for the copy-and-paste error that introduced comments from another paper into my review, and for the confusion this caused. I will correct the affected sections.
Your detailed communication breakdown and compressed-gradient rerun address my concern regarding W2. The additional latency and storage comparisons also address W3, and I accept these clarifications.
After re-reading the manuscript, I would appreciate clarification on the following point:
W1. Rationale for the protection thresholds: Appendix A.4 specifies an initial weight-energy threshold of 0.6 and a gradient-energy threshold of 0.99, but does not explain how these values were chosen. Since these choices determine the protected subspace’s size and the directions available for subsequent adaptation, could you clarify whether they were inherited from prior work or selected empirically? A brief discussion of the selection rationale and the trade-off between historical-task retention and new-task adaptation would be helpful.
If this concern is satisfactorily addressed, I will consider increasing my score.
Response to Reviewer
We thank the reviewer for the prompt response and additional comment.
Our threshold choices are motivated by established projection-based continual learning methods. In particular, KeepLoRA [ICLR 2026] adopts an initial energy threshold of 0.6 for pretrained weights and a historical-feature protection threshold of 0.99 in its LLaVA experiments. Related methods, including GPM [ICLR 2021], SGP [AAAI 2023], and InfLoRA [CVPR 2024], also employ high energy-retention thresholds (0.95–0.999) to preserve historical knowledge. Following these practices, we set the initial weight threshold to 0.6 to protect the dominant pretrained directions while preserving sufficient residual capacity for adaptation, and the historical gradient protection threshold to 0.99 to balance knowledge retention and new-task plasticity.
To further validate the latter choice, we conduct a small sensitivity analysis using 200 samples and an existing model checkpoint, fixing the initial weight threshold at 0.6 while varying the historical gradient protection threshold among 0.95, 0.99, and 0.999. We evaluate four complementary metrics:
- New task loss: Loss on the new task after adaptation, measuring new-task fitting (lower is better).
- Old loss: Increase in historical-task loss after adaptation, measuring interference with previously learned knowledge (lower is better).
- Storage: Memory required for the accumulated protection subspace (lower is better).
- Min. free dims: Minimum remaining dimensions in the orthogonal complement of the protection subspace across all adapted layers, measuring available adaptation capacity (higher is better).
Table 1. Sensitivity to historical protection thresholds.
| Threshold | New task loss ↓ | Old loss ↓ | Storage (GiB) ↓ | Min. free dims ↑ |
|---|---|---|---|---|
| 0.95 | 0.5717 | +0.01728 | 13.355 | 563 |
| 0.99 | 0.5757 | +0.00518 | 13.502 | 527 |
| 0.999 | 0.5805 | +0.00296 | 13.991 | 403 |
As shown in Table 1, stronger protection consistently reduces historical-task interference, but at the cost of increased storage and reduced adaptation capacity. Importantly, the threshold of 0.99 reduces historical-task loss degradation by approximately 70% compared with 0.95 (0.00518 vs. 0.01728), with only a marginal increase in new-task loss (0.5757 vs. 0.5717). Compared with 0.999, it achieves lower new-task loss (0.5757 vs. 0.5805), reduces protection storage by 0.489 GiB, and preserves 124 additional free dimensions, while maintaining a small historical-task loss increase of 0.00518. These results provide additional empirical support for selecting 0.99 as a reasonable balance among historical knowledge retention, new-task plasticity, and resource efficiency in the evaluated setting. They also suggest that jointly considering these four metrics provides a practical way to assess threshold choices for different continual learning scenarios. We have added this sensitivity analysis and discussion to Appendix A.4 of the revised manuscript.
We hope these additional results and clarifications address the reviewer's concerns and support a more favorable assessment of our paper. Thank you again for your time and consideration.
Response to Reviewer vb2o
Dear Reviewer vb2o, thank you for your time and effort in reviewing our submission. We would like to respectfully point out that the Paper Summary, W1, W4, and both Questions for Authors appear to refer to a different paper, FedAGP, whereas W2 and W3 correctly address our work. We believe some comments may have been inadvertently mixed up.
We would sincerely appreciate it if you could revisit our manuscript and reconsider your overall assessment and rating in light of our paper's actual contributions. Thank you again for your understanding and consideration. Below, we address the two relevant points, W2 and W3, in turn.
W2 (Communication accounting): Thank you for highlighting the need for more transparent communication accounting. We provide a detailed breakdown in Table 1, specifying the transmitted variables, tensor shapes, data types, transfer counts, and corresponding upload/download costs. At task boundaries, 224 gradient matrices are compressed into rank-32 factors Gℓ≈LℓRℓ⊤, where Lℓ=Uℓ,32Sℓ,32 and Rℓ=Vℓ,32. Model parameters use BF16, whereas gradient factors and distribution statistics use FP32. Each gradient upload requires 305 MiB, contributing 23.83 GiB across 40 initial and 40 final uploads over eight tasks. The breakdown also accounts for LoRA factors, projectors and routing statistics. The total communication comprises 72.14 GiB uplink and 96.81 GiB downlink, yielding the reported 168.95 GiB.
To directly verify accuracy under the same compressed communication implementation, we reran DART using rank-32 gradient factors under Non-IID β=1. The four-task setting achieves Last/Avg accuracies of 60.01%/64.33%, while the eight-task setting achieves 63.27%/74.85%. Compared with the original dense-gradient eight-task Last accuracy of 63.40%, compression reduces accuracy by only 0.13 percentage points. We will explicitly distinguish the dense-gradient and compressed results in the revised manuscript and associate the reported 168.95 GiB communication cost with the corresponding rank-32 implementation. These results demonstrate that the reported communication efficiency is achievable with minimal accuracy degradation under the evaluated setting.
Table 1. Measured communication breakdown over eight tasks with rank-32 gradient compression.
W3 (Router accuracy–cost trade-off): Thank you for this insightful comment. We agree that Table 11 does not establish an accuracy advantage of the learned router over direct Gaussian routing (63.40 vs. 63.51), and we do not claim superiority based on this small difference. DART's main contribution lies in distribution-guided historical-expert routing with bootstrapping-based orthogonal backbone updates to mitigate forgetting. The learned router is a model component in the implementation of this framework.
Regarding the accuracy–cost trade-off, Table 2 shows that both routing strategies achieve comparable accuracy, suggesting that both can effectively leverage distributional memory. However, our learned router offers substantial deployment advantages over direct Gaussian routing: approximately 81$\times$ lower inference latency (0.263 vs. 21.242 ms/sample) and 346$\times$ lower client-side storage requirements (0.156 vs. 53.93 MiB), at the expense of longer calibration/preprocessing time (4.07 vs. 1.62 s).
Table 2. Accuracy and CPU deployment costs of routing strategies.