acceptodds
Under review as a conference paper at ICLR 2027

FLAME: Feedforward Latent Task Inference for Continual Meta-Learning

Abstract

Continual‑meta learning promises rapid adaptation to new tasks without forgetting old ones. Despite significant progress, two major challenges remain. First, existing works mainly considered MAML as the meta-learner where adaptation is limited to gradient descent and meta-knowledge is encoded in initial model weights, creating bottlenecks as the task landscape becomes complex. Second, existing works largely operate in task-aware settings that rely on known task boundaries and identities to pair context/query samples, while the few task-agnostic variants forgo the bi-level meta-objective. We address these two challenges with FLAME, a with for continual adaptation. FLAME uses a feedforward meta-learner to rapidly adapt either the classifier layer or, in a nested formulation, the feature extractor of a DNN. It then leverages the feedforward task embedding to detect task boundaries and estimate task identities, approximating bi-level meta-optimization in a fashion. The continually-consolidated task relations in the replay reservoir further enables recognition of known $vs.$ novel tasks, allow continual improvements on recurring tasks over various lengths of local stationary windows. We demonstrated the improved performance of FLAME in comparison to both task-agnostic and task-aware baselines, in domain- and class-incremental learning across image and text datasets.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Meta Learning, Meta Learning, Lifelong Learning, Hypernetworks, Continual Learning, Task Agnostic Continual Learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_Uz6yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FLAME is a continual meta-learning framework designed to resolve two major challenges in existing Continual Meta-Learning approaches: (1) the adaptation bottleneck where existing methods, relying on MAML, constrain the adaptation to gradient descent; (2) the task-awareness bottleneck as existing methods assume known task boundaries and identities both during training and inference. The experiments across vision and text datasets demonstrate that FLAME achieves competitive results with existing baselines in both domain- and class-incremental settings.

Critical strengths
  • Task-agnosticity is a significant contribution to the continual meta-learning literature as in real-world this scenario is the most common. Using latent task representations to drive boundary detection and clustering makes FLAME's performance comparable to an oracle version where task identities are given.
  • The paper demonstrates the scalability and flexibility of FLAME with different model architectures and different modalities, e.g., text and images.
  • The paper is clearly structured and well written. The comparison with baselines is insightful and the ablation experiments are extensive, highlighting the importance of each component of FLAME.
Critical weaknesses
  • Clustering is performed separately from the meta-model optimization loop adding complexity to the proposed method. While this limitation is recognized by the author in the paper discussion, no method is proposed to address it.
  • While FLAME outperforms other baselines in RA, it is usually only comparable or even worse in LA. A deeper analysis into why FLAME lags behind MAML-based method would strengthen the discussion.
  • More recent references could be added to the text (e.g., meta-learning surveys, methods for continual meta-learning, etc)
  • The baselines considered are quite old and FLAME is only applied to “small” networks. While a scalability test is added to the appendix, this is still quite limited to a ViT with 16 layers and only on a single dataset.
Questions for authors
  • FLAME is evaluated under task-based streaming protocols, but most real-time settings are characterized by a task-free continuous stream of data. Can FLAME be extended to this scenario?
  • While Appendix E.4 demonstrates the compute cost and memory footprint of FLAME vs. La-MAML, a more general analysis of the memory and cost required for training and inference with FLAME is missing. For instance, what is the time required for the GMM update and buffer replacing? What is the memory/cost required for the hypernetwork to generate LoRA parameters? What is the complexity of the bi-level optimization and how does it scale with larger networks?
  • Does the order of the tasks in the stream impact performance? The paper reports results on two different orders but authors could add a statistical analysis by trying also different orders.
  • How has the threshold 1.4826 MAD (r ) been selected?
  • “when the underlying data distribution drifts, we expect not only a distribution shift at the task embedding space, but also a decline in the predictive accuracy of the adapted DNN due to context-query mismatch.” Did the authors verify this hypothesis? If two subsequent tasks are very similar, it might also happen that the accuracy remains the same or even improves with more training. Was forward transfer considered by the authors?
  • What happens to FLAME when very similar tasks are presented to the model? An analysis of FLAME in datasets such as Digits-5, PACS or DomainNet could help answering the question.
  • How are the hyperparameters for the baselines selected?
  • More recent meta-learning-based baselines should be used in the method comparison, together with some strong continual learning baselines that not necessarily use meta-learning.
  • Minor: adding a pseudocode will help the method understanding.
Reviewer_GCzvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FLAME (Feedforward Latent Task Inference for Continual Meta-Learning), a framework for continual meta-learning (CML) operating in a task-agnostic setting where neither task boundaries nor task identities are explicitly provided. The authors aim to address two limitations in prior online/continual meta-learning (O/CML) methods: (1) reliance on MAML-style inner-loop gradient descent from weight initializations, which struggles to scale across complex task distributions, and (2) inability of task-agnostic methods to maintain the bi-level meta-learning objective (context-query pairing) without ground-truth task labels. FLAME addresses these via three components:

  1. Feedforward What-How Meta-Models: In domain-incremental settings, a shared feature extractor computes class-wise prototype vectors rtr^t from the context set, and a hypernetwork hh maps each prototype to linear classifier weights wtw^t. In class-incremental (multi-dataset) settings, a nested hypernetwork maps a learnable dataset embedding sds^d to low-rank adaptation matrices (LoRA for Transformers, LLoRA for CNNs) to modulate the feature extractor prior to classifier generation.
  2. Two-Channel Task Boundary Detection: Using the most recent N×KN \times K samples as context, FLAME monitors both context-conditioned prediction loss residuals and query feature mean drift, standardized via trailing Median Absolute Deviation (MAD) over R=100R=100 batches, flagging a task switch whenever the combined positive residual score S>κS > \kappa.
  3. Task-Relational Reservoir via Bayesian GMM: At each detected boundary, FLAME updates a Bayesian Gaussian Mixture Model (GMM) over task embeddings in the replay buffer (capacity M=30,000M=30,000), pruning low-weight components, rebalancing M/QM / Q samples per cluster, and using cluster assignments as pseudo-task IDs to pair context and query samples during replay. Experiments on domain-incremental benchmarks (MNIST-Rotations, MNIST-Permutations, both non-recurring and recurring) and multi-dataset streams (4 vision datasets and 5 text classification datasets), along with stress tests on α\alpha-stationarity and gradual boundaries, show strong Retained Accuracy (RA) and test-time adaptation speed compared to MAML-based and feedforward baselines.
Critical strengths
  1. Clear Conceptual Separation of "What" (Latent Task Inference) and "How" (Hypernetwork / LoRA Adaptation): Replacing iterative inner-loop MAML updates with prototype-conditioned hypernetworks and hypernetwork-generated (L)LoRA adapters provides a clean, fast-adapting alternative for O/CML. Table 4 convincingly demonstrates that feedforward adaptation reduces test-time adaptation latency (TTA-10) by over 15×15\times compared to 5-step MAML while achieving higher accuracy on miniImageNet and CIFAR-FS.
  2. Consolidating Recurring Tasks via Latent Clustering Instead of Boundary Counters: A key failure mode of naive boundary-detection CML is buffer fragmentation when previously seen tasks recur (assigning a new task ID at every boundary and diluting buffer capacity per task). Using a Bayesian GMM to merge recurring tasks into existing clusters is well-motivated and empirically validated by the rare-task stress test in Appendix C.6.3 (Figure 11, FLAME vs. FLAME-B).
  3. Extensive Stress-Testing of Stationarity and Boundary Assumptions: Section 4.4 and Appendices D–E go beyond standard benchmark tables to thoroughly test α\alpha-stationarity (varying mean visit lengths from 5 to 234 batches), gradual local-mixing boundaries (W∈{0,…,5}W \in \{0, \dots, 5\}), hyperparameter sensitivity (η2\eta^2 variance analysis across 60 boundary and 36 clustering configs), and buffer sampling wall-clock time (Figure 18).
Critical weaknesses
  1. Unspecified Inference-Time Mechanism for Selecting Dataset Embeddings sds^d in Nested Adaptation: In the class-incremental (multi-dataset) setting (Section 3.1, Figure 1B, and Appendix B.6), the feature extractor is adapted via LLoRA/LoRA matrices generated from a learnable dataset-specific embedding vector sds^d (of dimension 128 or 32). However, the paper never explains:

    • How a new vector sds^d is initialized, allocated, or tied to the GMM clusters during training, and
    • Crucially, how sds^d is selected at test time when evaluating on unlabeled episodes from all previously seen datasets in a task-agnostic manner. Since sds^d is a learned parameter vector rather than a feedforward function of the test episode's support images, how does FLAME infer which sds^d to feed into the hypernetwork at test time without oracle dataset IDs?
  2. Permutation Sensitivity of Task Representations RtR^t in Episodic Class-Incremental Clustering: In Section 3.1, the task representation is defined by stacking class prototypes into a matrix Rt=[r1t,…,rNt]R^t = [r_1^t, \dots, r_N^t] of shape F×NF \times N. In domain-incremental MNIST, the 10 classes are fixed in a canonical order (0…9)(0 \dots 9), so RtR^t has consistent coordinates across batches. However, in episodic NN-way KK-shot class-incremental learning (Section 4.2), each episode randomly samples N=5N=5 classes from a dataset and assigns them arbitrary episode-local labels {1,…,5}\{1, \dots, 5\}. Consequently, column nn of RtR^t represents a different semantic class in every episode and is not permutation-invariant. It is unclear what exact vector is fed into the Bayesian GMM and the embedding drift channel μ\mu in the NN-way KK-shot setting (e.g., is RtR^t averaged across all NN classes into an FF-dimensional vector?).

  3. Missing Related Work on the Neural Processes (NP / CNP) Family and Latent Task Representation Meta-Learning: Since FLAME's core "What-How" mechanism relies on feedforward aggregation of context samples into a latent task representation and updating/structuring this latent space without inner-loop gradient updates, the related work in Section 2 (General DNN adaptation and Feedforward O/CML) is missing a discussion of the Neural Processes (NP) / Conditional Neural Processes (CNP) family of meta-learners. In particular, prior work has systematically investigated how feedforward latent task encoders compare against gradient-based MAML and how structuring the latent task representation space improves generalization—most notably What Matters For Meta-Learning Vision Regression Tasks? (Gao et al., CVPR 2022), which demonstrated that CNP-based feedforward task representations outperform MAML without test-time fine-tuning and introduced functional contrastive learning in the latent task space. Discussing this line of work would better contextualize FLAME's feedforward latent task inference and task-space clustering.

Questions for authors
  1. Clarification on the Boundary Detection Refractory Window: In Appendix C.6.1 (line 1374) and D.5 (line 1680), you state that after a boundary is detected, the detection rule is only consulted again "after the fixed local stationarity window". Was this refractory window set equal to the true task length (234 batches on MNIST, 2,000 steps on vision/text streams) in the main experiments (Tables 1–3)? What are the boundary detection Precision/Recall and final RA/LA in Tables 1–3 if the rule S>κS > \kappa is evaluated at every batch (with only the m=5m=5 batch warm-up) without silencing it across the stationary window?

  2. Test-Time Selection of Dataset Embeddings sds^d: In the multi-dataset experiments (Section 4.2, Tables 2–3), how does FLAME infer which learnable dataset embedding sds^d to use to generate the (L)LoRA weights (Ad,Bd,Md)(A^d, B^d, M^d) during meta-testing when task/dataset identity is unknown?

  3. Task Embedding Representation for GMM Clustering in NN-way KK-shot Episodes: Since each 5-way 5-shot episode in Section 4.2 samples a different subset of 5 classes in random order, how is the task embedding constructed from RtR^t (of shape F×NF \times N) so that it is invariant to class permutation and class identity within a dataset before fitting the Bayesian GMM?

  4. Baselines on DINOv3 ViT-S/16 without Oracle Boundaries: In Appendix C.4 (Table 9), how does FLAME perform on the DINOv3 ViT-S/16 backbone when using online boundary detection and GMM clustering (rather than oracle boundaries), and how does it compare against La-MAML and FTML equipped with the same frozen DINOv3 backbone and LoRA?

  5. Connection to Neural Processes (CNP) and Latent Task Representation Learning: Could the authors expand Section 2 to discuss the Neural Processes / Conditional Neural Processes (CNP) family of feedforward meta-learners, including What Matters For Meta-Learning Vision Regression Tasks? (Gao et al., CVPR 2022), and comment on how FLAME's prototype-based latent task inference and GMM task-space clustering relate to CNP-style latent task aggregation and functional contrastive task-space regularization?

Note on Rating: I am willing to adjust/increase my rating if the authors can formally address the concerns and questions above—including rigorous mathematical/algorithmic clarifications, the missing related work discussion, and fixing the manuscript formatting oversights—during the rebuttal phase.

Reviewer_uBWHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes FLAME for continual meta-learning with inferred task boundaries and identities. It generates classifiers from labeled context embeddings and optionally generates LoRA parameters to adapt feature extractors. A boundary detector and Bayesian Gaussian mixture model organize replay samples into inferred tasks, enabling context–query pairing during continual optimization. Experiments on image and text datasets report improved retention and efficient adaptation.

Critical strengths
  1. The framework connects feedforward classifier generation, feature adaptation, and inferred replay grouping within a coherent architecture. Using the same context representation for adaptation and task organization is a useful design choice.

  2. The evaluation covers domain shifts, recurring tasks, sequential image datasets, and text classification. FLAME reports strong retained accuracy, including 94.94% on MNIST-Rotations and 89.72% on MNIST-Permutations, while remaining competitive across the more heterogeneous dataset sequences.

Critical weaknesses
  1. Appendix D.5 states that the detector re-arms only after a full task presentation, which contributes to its perfect precision. This introduces a timing assumption that is not explicit in the main detection rule. Appendix E.1 evaluates detection without this window, but does not resolve how the complete method performs without presentation-length information.

  2. The nested model requires a dataset embedding sds_d to generate the feature extractor, but the inference procedure for selecting this embedding is insufficiently specified. It is unclear which encoder produces the features used for assignment before sds_d is selected, and how GMM components remain associated with embeddings after clustering changes.

  3. The rare-task comparison changes both task consolidation and memory allocation. Merging recurring tasks changes the number of groups and their M/QtM/Q_t storage quotas, potentially preserving more rare-task examples.

Questions for authors
  1. What is the detector’s exact refractory period in each experiment? Is it set using the true presentation length?

  2. How is sds_d selected for a test episode without dataset identity? Please specify the encoder used before assignment and how cluster merging or pruning updates the association with dataset embeddings.

  3. In the rare-task experiment, how do buffer occupancy and replay frequency for the rare task differ between FLAME and FLAME-B? Do existing results separate these effects from improved context matching?

  4. Which operations are included in the reported replay-draw time? Why does sampling require several seconds for La-MAML, and were both methods implemented with comparable indexing and data-transfer procedures?

Reviewer_MUMqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FLAME, a continual meta-learning framework that combines feedforward adaptation with latent task inference. FLAME uses hypernetworks to generate task-specific parameters without gradient-based inner-loop adaptation, while estimating task boundaries and identities through feature representations and GMM-based clustering. Empirical results show improvement in retained accuracy across domain and class incremental learning settings.

Critical strengths

S1: Convincing problem settings and a reasonable approach

  • The paper addresses an important problem in continual meta-learning by combining fast task adaptation with task-agnostic learning. The overall framework is reasonably motivated and provides a practical approach to handling sequentially encountered tasks without explicit task identities.

S2: A broad range of experimental validations

  • The experimental evaluation is relatively comprehensive, covering multiple datasets, task sequences, and model architectures. The results demonstrate consistent improvements in retained accuracy over several baselines, supporting the practical effectiveness of the proposed framework.
Critical weaknesses

W1: A doubtful motivation of the key setting

  • From my perspective, the motivation behind the proposed approach is not sufficiently justified. This work identifies gradient-based adaptation and the reliance on model initialization as major bottlenecks of MAML, especially in complex task landscapes. However, the paper does not adequately explain what fundamental limitations arise from these characteristics or why they constrain adaptation to more complex tasks.
  • To be specific, I acknowledge that MAML can be biased toward a new task distribution, i.e., Pt+1(T)P_{t+1}(\mathcal{T}), and unable to cover the past distribution, i.e., Pt(T)P_{t}(\mathcal{T}). However, IF we can provide a combined distribution of Pt+1∪PtP_{t+1} \cup P_{t}, not continuously, to MAML, it may be able to find a proper initialization even with the complicated task landscape. Then, what is the main bottleneck? Is it a complicated landscape or time-varying task distributions?

W2: Insufficient justification of hypernetworks

  • The justification for using a hypernetwork seems to be insufficiently provided. Even though the paper demonstrates the computational efficiency and performance advantages of feedforward adaptation over MAML, it is unclear whether these improvements arise specifically from hypernetwork-based parameter generation.
  • Especially, the current experiments do not sufficiently distinguish the proposed approach from simpler alternatives, such as prototype-based classification or conventional task-specific adaptation. Also, this paper does not provide any analysis of the capacity of the hypernetwork, and showing the connection between the capacity of the hypernetwork and task complexity is required to justify this design.

W3: Unclear true benefit of FLAME

  • Again, from my perspective, the paper does not clearly demonstrate how FLAME improves the underlying meta-learning capability beyond retaining knowledge of previously encountered tasks. Although this framework demonstrates strong retained accuracy, it is unclear whether the proposed approach improves the transfer of meta-knowledge across tasks or primarily benefits from task-specific adaptation and replay. This distinction is important for understanding the core contribution to continual meta-learning.

W4: Limited comparisons to other meta-learners with task-representations

  • The authors should clarify its novelty relative to prior task-conditioned meta-learning methods. TADAM (Oreshkin et al., 2018) uses task representations to condition feature extraction, while MMAML (Vuorio et al., 2019) combines task embeddings with MAML-based adaptation (these two are examples, and the use of task representations is not limited to them).
  • An explicit comparison would clarify whether FLAME introduces new representation or adaptation mechanisms, or primarily extends existing ideas to task inference and replay-based continual meta-learning.
Questions for authors

Q1 (please see W1 for details):

  • This paper doesn't provide a sufficient explanation of the relationship between task complexity and these limitations. A more detailed justification is needed to explain how increasing task complexity leads to these bottlenecks and how the proposed FLAME framework specifically addresses them.

Q2 (please see W3 for details):

  • Although the paper provides experimental comparisons between FLAME and MAML, the justification for adopting a hypernetwork is not sufficiently provided.
  • Could the authors provide further experimental evidence demonstrating the advantages of hypernetwork-based parameter generation over simpler alternatives, such as prototype-based classification or conventional task-specific adaptation?
  • Moreover, analyses of the relationship between hypernetwork capacity and task complexity would help justify the proposed design.

Q3: About similar task representations

  • Additionally, the connection between the clusters and the task-specific adaptation requirements is not sufficiently demonstrated. Could the authors provide additional empirical evidence showing whether the inferred clusters appropriately represent the task relationships required for effective meta-learning, especially when different tasks have similar latent representations?

Q4: Overlapping task distribution changes

  • This question is quite related to Q3. If we handle overlapping task distribution sequences. For example, for a previous task distribution, we contain classes 0, 1, and 2. Then the next distributions contain classes 1, 2, and 3. Then, such overlapping distributions make the task discriminations vague. How does FLAME work in these cases?
1 comment

Thanks to the authors' response and their thoughts on my concern. I feel that the authors have slightly shifted the main emphasis toward the technical novelty, including challenging scenarios (non-stationary tasks) and methodologies, rather than pointing out why/how a baseline MAML is limited in tackling a non-stationary sequence of tasks. I don't think the authors' rebuttal sufficiently explains why MAML or a meta-learning baseline fails on non-stationary cases. Some algorithms show technical performance gains, but I still think they should address why MAML fundamentally fails on time-varying tasks (which somewhat conflicts with the episodic-varying nature of meta-training). Thus, I will maintain my original rating: weak rejection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.