- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
FLAME is a continual meta-learning framework designed to resolve two major challenges in existing Continual Meta-Learning approaches: (1) the adaptation bottleneck where existing methods, relying on MAML, constrain the adaptation to gradient descent; (2) the task-awareness bottleneck as existing methods assume known task boundaries and identities both during training and inference. The experiments across vision and text datasets demonstrate that FLAME achieves competitive results with existing baselines in both domain- and class-incremental settings.
Critical strengths
- Task-agnosticity is a significant contribution to the continual meta-learning literature as in real-world this scenario is the most common. Using latent task representations to drive boundary detection and clustering makes FLAME's performance comparable to an oracle version where task identities are given.
- The paper demonstrates the scalability and flexibility of FLAME with different model architectures and different modalities, e.g., text and images.
- The paper is clearly structured and well written. The comparison with baselines is insightful and the ablation experiments are extensive, highlighting the importance of each component of FLAME.
Critical weaknesses
- Clustering is performed separately from the meta-model optimization loop adding complexity to the proposed method. While this limitation is recognized by the author in the paper discussion, no method is proposed to address it.
- While FLAME outperforms other baselines in RA, it is usually only comparable or even worse in LA. A deeper analysis into why FLAME lags behind MAML-based method would strengthen the discussion.
- More recent references could be added to the text (e.g., meta-learning surveys, methods for continual meta-learning, etc)
- The baselines considered are quite old and FLAME is only applied to “small” networks. While a scalability test is added to the appendix, this is still quite limited to a ViT with 16 layers and only on a single dataset.
Questions for authors
- FLAME is evaluated under task-based streaming protocols, but most real-time settings are characterized by a task-free continuous stream of data. Can FLAME be extended to this scenario?
- While Appendix E.4 demonstrates the compute cost and memory footprint of FLAME vs. La-MAML, a more general analysis of the memory and cost required for training and inference with FLAME is missing. For instance, what is the time required for the GMM update and buffer replacing? What is the memory/cost required for the hypernetwork to generate LoRA parameters? What is the complexity of the bi-level optimization and how does it scale with larger networks?
- Does the order of the tasks in the stream impact performance? The paper reports results on two different orders but authors could add a statistical analysis by trying also different orders.
- How has the threshold 1.4826 MAD (r ) been selected?
- “when the underlying data distribution drifts, we expect not only a distribution shift at the task embedding space, but also a decline in the predictive accuracy of the adapted DNN due to context-query mismatch.” Did the authors verify this hypothesis? If two subsequent tasks are very similar, it might also happen that the accuracy remains the same or even improves with more training. Was forward transfer considered by the authors?
- What happens to FLAME when very similar tasks are presented to the model? An analysis of FLAME in datasets such as Digits-5, PACS or DomainNet could help answering the question.
- How are the hyperparameters for the baselines selected?
- More recent meta-learning-based baselines should be used in the method comparison, together with some strong continual learning baselines that not necessarily use meta-learning.
- Minor: adding a pseudocode will help the method understanding.
Thanks to the authors' response and their thoughts on my concern. I feel that the authors have slightly shifted the main emphasis toward the technical novelty, including challenging scenarios (non-stationary tasks) and methodologies, rather than pointing out why/how a baseline MAML is limited in tackling a non-stationary sequence of tasks. I don't think the authors' rebuttal sufficiently explains why MAML or a meta-learning baseline fails on non-stationary cases. Some algorithms show technical performance gains, but I still think they should address why MAML fundamentally fails on time-varying tasks (which somewhat conflicts with the episodic-varying nature of meta-training). Thus, I will maintain my original rating: weak rejection.