acceptodds
Under review as a conference paper at ICLR 2027

Continual Learning Without Continual Training

Abstract

Continual learning requires models to adapt to new domains and new classes while retaining prior knowledge. Many existing methods rely on continued optimization, using regularization, replay, or parameter expansion to prevent new updates from overwriting previously learned knowledge. Instead, we propose replacing continual training with continual inference: a PFN-based model that is meta-trained, and then frozen, adapting to new classes only by extending an in-context evidence set. Our model, Latent Concept PFN, performs in-context Bayesian inference over a latent concept space that captures semantic structure shared across domains and classes. As each new domain or class arrives, exemplars are added to the memory; adaptation reflects updated posterior beliefs over latent concepts rather than gradient updates. No parameters are changed, reducing forgetting. The same method handles both domain and class incremental continual learning without task identity. Concept annotations are only used during meta-training, acting as a soft anchor on the latent space rather than a fixed bottleneck. Unlike fixed-vocabulary concept methods, the model also handles noisy, ambiguous, or incomplete annotations by combining concept labels with raw input evidence to discover distinctions beyond the predefined concept set. Experiments on class and domain incremental learning datasets demonstrate competitive continual learning performance while learning interpretable latent concepts.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Domain-Incremental, Class-Incremental, Concepts, Interpretability, Meta-Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Pt4HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Latent Concept PFN, which combines a concept-supervised, meta-trained latent encoder with a frozen TabPFN classifier. During continual learning, all parameters remain fixed, and adaptation occurs through a growing exemplar context.

Critical strengths

The approach explores a useful alternative to continual parameter optimization. The EMNIST-to-MNIST setting separates source and target classes, and the reported Split MNIST result is competitive with parameter-updating baselines.

Critical weaknesses
  1. The proposed model and TabPFN baseline differ in front-end architecture and meta-training, so their performance gap does not isolate concept anchoring.
  2. Table 1 contradicts the least-forgetting claim: TabPFN has lower FM on Rotated MNIST, and InfLoRA has lower FM on Split MNIST and CXR.
  3. Stable seven-dimensional concept read-outs cannot establish stability of the full 16-dimensional context-dependent latent or explain error causation.
  4. Evaluation uses growing memories and 128 latent samples but covers only four or five stages and at most ten classes.
  5. Please standardize abbreviations and cross-reference formatting, including "Sec." and "Section," "Eq." and "Eq," and "App." and "Appendix".
Questions for authors
  1. Does concept supervision improve performance under a matched meta-training protocol?
  2. Can the authors report FM computed per seed before averaging, alongside direct latent-drift measurements as the context grows?
  3. What is the accuracy–latency trade-off when reducing the 128 inference-time latent samples?
  4. Could the authors clarify which figures are intended by “Figs. 3, 3, 1”?
7 comments

We thank the reviewer for taking the time to provide detailed feedback. We will standardise the cross-referencing throughout, and we have run a matched meta-training ablation, per-seed forgetting metrics, latent-drift measurements and an accuracy–latency sweep, which we address below.

W1/Q1 - Matched meta-training protocol

The TabPFN baseline differs from our front-end in architecture and training: it uses a frozen ImageNet ResNet-18 with PCA, whereas our embedder is a CNN meta-trained on EMNIST. We chose this baseline because passing pretrained image features to TabPFN is the standard way to apply TabPFN to images. However, we agree that this comparison does not isolate the contribution of concept anchoring, so we ran the following ablation.

The concept-free arm uses identity episodes only and no concept loss (λc = 0), with the same training budget as our headline models. Everything else (encoder, frozen TabPFN, meta-training data) is kept the same, so this measures the full setup without any use of concepts. The only-ID arm also uses identity episodes only, but keeps the concept loss (λc = 1): concept information is present, but there are no rule or shift episodes.

AblationRotated MNIST AA% changeSplit MNIST AA% changeCXR TB AA (bal. acc.)% change
Full (ours)0.789 ± 0.031—0.794 ± 0.021—0.749 ± 0.007—
Concept-free0.704 ± 0.026−10.6% ± 3.90.700 ± 0.012−11.8% ± 3.50.649 ± 0.015−13.3% ± 2.3
Only ID episodes0.690 ± 0.024−12.4% ± 2.80.705 ± 0.049−11.3% ± 4.90.652 ± 0.029−13.0% ± 3.5

Removing concepts altogether costs 10.6% on Rotated MNIST, 11.8% on Split MNIST and 13.3% on CXR (5 seeds, paired). Re-introducing concept supervision alone does not recover this. The only-ID arm is within noise of the concept-free arm on all three datasets (Rotated MNIST 0.690 vs 0.704, Split MNIST 0.705 vs 0.700, CXR 0.652 vs 0.649). Supervising the latent with a fixed, static vocabulary is therefore not helpful on its own. The gain comes from the rule and shift episodes, which together allow the model to handle new classes. On Rotated MNIST, the concept-free arm (0.704) also falls below the TabPFN baseline on ImageNet features (0.785, Table 1), so our architecture and in-domain meta-training do not account for the gain on their own.

To answer Q1 directly: under a fully matched protocol (the only-ID and concept-free arms differ only in λc), concept supervision alone does not improve performance. The improvement comes from using concepts to define the meta-training episodes (rule and shift episodes).

W2 - Least-forgetting claim

We thank the reviewer for pointing this out; the opening sentence of Section 5 was imprecise. Our method has low forgetting in every setting, but not the lowest. With FM computed on each seed's accuracy matrix and then averaged (Q2), TabPFN forgets slightly less on Rotated MNIST (−0.012 vs −0.007), and InfLoRA on Split MNIST (0.045 vs 0.073) and CXR (0.000 vs 0.007). In the revised paper, we will replace the sentence with "The Latent Concept PFN has the second-lowest forgetting measure in every setting and stays competitive without a single parameter update," and state explicitly where TabPFN and InfLoRA forget less.

W3 / Q2 - Direct latent drift as context grows

Table A. Latent drift over the whole stream (Latent Concept PFN, full 16-d latent, frozen Table 1 models, 5 seeds, mean ± sd)

Rotated MNISTSplit MNIST
Linear-probe forgetting [1]−0.001 ± 0.0040.001 ± 0.002
Prototype drift [2]0.087 ± 0.0110.102 ± 0.024
Representation discrepancy, aligned / unaligned [3]0.15 / 0.450.12 / 0.46
Per-stage drift, stored / arriving0.057 / 0.0760.076 / 0.430
AUROC drift → forgetting (flip rate)0.70 ± 0.04 (5%)0.77 ± 0.10 (10%)

Each domain or task is compared between the stage it arrived and the end of the stream. Linear-probe forgetting: accuracy of a linear probe refit on the latent at arrival minus at the end. Prototype drift: shift of each class mean, divided by the mean distance between class means. Representation discrepancy: worst-case shift of the latent after the best linear map between the two stages (aligned) or with no map (unaligned), divided by the largest latent norm. Per-stage drift: per-query shift of the latent between consecutive stages, divided by the class separation, for domains or tasks already stored vs. the one arriving. AUROC: how well a query's latent drift from arrival to the end predicts that it goes from correct to wrong; flip rate is the fraction of queries correct at arrival that end wrong.

Table B. The Split MNIST event in Section 5 (seed 42, digits 2/3 when 6/7 arrive; 108 of 320 correctly classified images flipped)

FlippedStayed correct
Read-out change (7-d)0.0250.037
Latent drift (16-d)0.0990.100
AUROC drift → flip0.43

Change in each image's representation when 6/7 arrive. Read-out change: mean absolute change per concept of the 7-d concept read-out, as in Section 5. Latent drift: shift of the full 16-d latent, divided by the class separation. AUROC: how well latent drift predicts which images flip (0.5 = no information).

On the frozen Table 1 models, with the same streams, queries and protocol (no retraining), we computed the posterior mean of the full 16-d latent for every test query and memory exemplar, plus 300 held-out MNIST images per class from the training split that were never used as memory exemplars (the model was meta-trained only on EMNIST letters). We used these held-out images for the probe and the prototypes. We compared the representation of each domain or task between the stage it arrived and the end of the stream, over 5 seeds. We computed three standard metrics: linear-probe forgetting [1], class-prototype drift [2], and representation discrepancy with an exact linear alignment [3]. We also measured per-query drift of the latent, relative to the class separation, on stored and arriving data, and how well it predicts forgetting (AUROC).

On the Split MNIST event we discuss in Section 5 (Table B), flipped and unflipped images moved equally in the full latent: per-query drift is 0.099 vs. 0.100 of the class separation. Drift does not identify which images flipped (AUROC 0.43). This matches the 7-d read-out (0.025 vs. 0.037 per concept), so the claim now holds on the full 16-d latent. The information to separate 2s from 3s is also kept. A linear probe on the latent classifies them with 0.993 accuracy before 6/7 arrive and 0.985 after. The errors therefore come from the new 7 exemplars occupying the same region as the 2s that read like the letter ‘z’, not from the 2s’ own representation changing. In the revised paper, we will limit the stability claim in Section 5 to these images.

Across the whole stream (Table A), the information in the latent is kept: linear-probe forgetting is −0.001 on Rotated MNIST and 0.001 on Split MNIST. The latent itself does move moderately. Class prototypes shift by 0.087 and 0.102 of the distance between them. Most of this movement is shared, since a single linear map reduces the representation discrepancy from 0.45 to 0.15 (Rotated MNIST) and from 0.46 to 0.12 (Split MNIST). The movement is concentrated on new data: on Split MNIST, the per-query drift of the arriving task’s queries is 0.430 of the class separation, compared with 0.076 for tasks already stored. This is the adaptation through context the method is designed for. Across the stream, per-query drift is moderately predictive of which queries are forgotten (AUROC 0.70 and 0.77), but forgetting is rare (5% and 10% of queries correct at arrival). Some errors therefore involve a query’s own latent moving, while others, like the event discussed above, come from new neighbours in the memory.

[1] M. Davari, N. Asadi, S. Mudur, R. Aljundi, and E. Belilovsky, ‘Probing representation forgetting in supervised and unsupervised continual learning’, in 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), Los Alamitos, CA, USA: IEEE Computer Society, Jun. 2022, pp. 16691–16700. doi: 10.1109/CVPR52688.2022.01621. [2] L. Yu et al., ‘Semantic drift compensation for class-incremental learning’, in 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), Los Alamitos, CA, USA: IEEE Computer Society, Jun. 2020, pp. 6980–6989. doi: 10.1109/CVPR42600.2020.00701. [3] J. Kim, Y. Kim, and J.-Y. Sohn, ‘Measuring representational shifts in continual learning: a linear transformation perspective’, in Proceedings of the 42nd international conference on machine learning, in Proceedings of machine learning research, vol. 267. PMLR, Jul. 2025, pp. 30267–30289. [Online]. Available: https://proceedings.mlr.press/v267/kim25p.html

W5 / Q4 - Formatting and cross-references

We thank the reviewer for catching this. The reference should read (Fig. 3, Table 1). The per-task curves are in Fig. 3 and the forgetting metrics in Table 1. We will correct this and standardise abbreviations and cross-reference formatting (Sec., Eq., App.) throughout the revised PDF.

W4 - Stream length, memory size and number of classes

To test how the method scales with memory size, we vary k, the number of exemplars stored per class (per domain for Rotated MNIST, per hospital for CXR), and compare against the TabPFN baseline (ImageNet ResNet-18 → PCA-16) on the same memories and queries over 5 seeds. We evaluate two protocols. In the whole memory protocol, the full exemplar memory is passed as context, so context size grows with k and with the number of domains seen. In the bounded protocol (App. D), the context is capped: at each prediction we draw at most 50 class-balanced rows from the memory, repeat this 8 times and average the predictions. The bounded protocol keeps the cost per prediction fixed however large the memory grows, which is the relevant setting for long streams.

Bounded context (≤ 50 rows × 8 draws):

DatasetMethodk=1k=2k=5k=10k=20k=50
Rotated MNISTOurs0.6830.7210.7390.7390.7470.747
TabPFN0.5890.6900.7240.7320.7370.730
Split MNISTOurs0.5970.7210.7940.8240.8380.846
TabPFN0.3400.4630.6360.7140.7450.765
DatasetMethodk=1k=2k=4k=8k=16k=32
CXR TB (bal. acc.)Ours0.6650.6990.7510.7500.7460.753
TabPFN0.5490.5950.5980.6360.6740.677

Whole memory:

DatasetMethodk=1k=2k=5k=10k=20k=50
Rotated MNISTOurs0.6830.7400.7890.8200.8440.863
TabPFN0.5890.7060.7850.8170.8400.863
Split MNISTOurs0.5970.7210.7940.8390.8680.890
TabPFN0.3400.4630.6360.7350.8080.842
DatasetMethodk=1k=2k=4k=8k=16k=32
CXR TB (bal. acc.)Ours0.6650.6990.7510.7480.7500.758
TabPFN0.5490.5950.5980.6390.6680.697

From this, we can see that our method requires far fewer exemplars than a frozen TabPFN on ImageNet features to reach the same accuracy, and this holds even with the bounded subsampled context. With the bounded context, four films per class per hospital on CXR (0.751) exceed TabPFN at every memory size tested, up to 32 (0.677), and five exemplars per class on Split MNIST (0.794) exceed TabPFN with fifty (0.765). With the whole memory, the CXR result still holds (TabPFN reaches 0.697 at k = 32), and on Split MNIST TabPFN needs k = 20 (0.808) to overtake our k = 5. On Rotated MNIST the advantage is largest at small k and narrows as memory grows. Note that the front-end was meta-trained on contexts of at most 56 rows, yet whole-memory accuracy keeps rising well beyond this (up to 2,500 rows on Rotated MNIST), so the model can use larger memories without retraining. With the bounded context (App. D), the inference cost is fixed and does not grow with the length of the stream or the size of the memory. So, subsampling helps to make it scalable.

The method is less suited where many exemplars per task are available under pure covariate shift: on Rotated MNIST, TabPFN catches up once k ≥ 5 with the whole memory. The bounded context also trades accuracy for cost when many domains share a small budget (Rotated MNIST: ≈0.75 bounded vs 0.86 whole memory at k = 50). We will add both points to the limitations.

We are currently running an additional, larger experiment and will post results in the coming days. Regarding the number of classes, the limit comes from the frozen backbone, TabPFN v2 (Hollmann et al., 2025), which supports up to 10 classes natively. We use v2 because it is openly licensed (Apache 2.0 with attribution). The method is not tied to this backbone: TabPFN-3 supports up to 160 classes natively, and Prior Labs' many-class extension extends v2 beyond 10 classes.

Reviewer_CjDbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Latent Concept PFN for continual image classification without parameter updates on the data stream. An image encoder and context-attending transformer are meta-trained with concept supervision and synthetic rule/shift episodes, then frozen and combined with a frozen TabPFN classifier. Adaptation occurs by adding labeled exemplars to the context memory. Experiments on two MNIST streams and a chest-radiograph stream show promising class-incremental accuracy and low domain-incremental forgetting, together with analyses of concept read-outs and information beyond the annotated vocabulary.

Critical strengths
  • The method provides a concrete mechanism for adapting to new classes and domains through context, using the same EMNIST-trained model for both MNIST streams. With five exemplars per class, it achieves 0.794 average accuracy on Split MNIST, compared with 0.749 for the strongest listed baseline (Table 1).

  • The soft concept interface provides informative semantic read-outs while retaining predictive information outside the annotated vocabulary. The latent/read-out comparison and matched-concept digit examples support this flexibility (Fig. 2), and the finding-prediction results on VinDr provide an additional check of semantic validity (Fig. 5).

Critical weaknesses
  • The contribution of concept anchoring is not isolated from supervised episodic representation learning. The TabPFN baseline uses frozen ImageNet ResNet-18 features and unsupervised PCA, whereas the proposed method learns its front end and a context-attending transformer with a specialized episodic prior; even the image architectures differ on MNIST (Appendices E and G.1). Fig. 2 replaces features after training and does not establish the benefit of concept supervision during training. Since adaptation through frozen in-context models already appears in CURE and the in-context streaming framework of Lourenço et al., a matched training ablation is important for establishing the incremental contribution of the proposed concept-anchored representation.

  • The evidence does not fully support the claimed stability advantage. The claim of least forgetting in every setting conflicts with Table 1: TabPFN has better BWT/FM on Rotated MNIST, and InfLoRA has better BWT/FM on Split MNIST and CXR. Furthermore, the small change in the seven-dimensional concept read-out in Section 5 does not establish stability of the full 16-dimensional, context-dependent representation, which explicitly retains information outside that read-out. This leaves the proposed explanation of stability incomplete and limits how strongly the retention results support the central motivation.

Questions for authors

Please refer to the weaknesses.

4 comments

We thank the reviewer for their constructive feedback. We have run a matched meta-training ablation and direct latent-drift measurements, and address each point below.

W1 - Concept Contribution

The TabPFN baseline differs from our front-end in architecture and training: it uses a frozen ImageNet ResNet-18 with PCA, whereas our embedder is a CNN meta-trained on EMNIST. We chose this baseline because passing pretrained image features to TabPFN is the standard way to apply TabPFN to images. However, we agree that this comparison does not isolate the contribution of concept anchoring, so we ran the following ablation.

The concept-free arm uses identity episodes only and no concept loss (λc = 0), with the same training budget as our headline models. Everything else (encoder, frozen TabPFN, meta-training data) is kept the same, so this measures the full setup without any use of concepts. The only-ID arm also uses identity episodes only, but keeps the concept loss (λc = 1): concept information is present, but there are no rule or shift episodes.

AblationRotated MNIST AA% changeSplit MNIST AA% changeCXR TB AA (bal. acc.)% change
Full (ours)0.789 ± 0.031—0.794 ± 0.021—0.749 ± 0.007—
Concept-free0.704 ± 0.026−10.6% ± 3.90.700 ± 0.012−11.8% ± 3.50.649 ± 0.015−13.3% ± 2.3
Only ID episodes0.690 ± 0.024−12.4% ± 2.80.705 ± 0.049−11.3% ± 4.90.652 ± 0.029−13.0% ± 3.5

Removing concepts altogether costs 10.6% on Rotated MNIST, 11.8% on Split MNIST and 13.3% on CXR (5 seeds, paired). Re-introducing concept supervision alone does not recover this. The only-ID arm is within noise of the concept-free arm on all three datasets (Rotated MNIST 0.690 vs 0.704, Split MNIST 0.705 vs 0.700, CXR 0.652 vs 0.649). Supervising the latent with a fixed, static vocabulary is therefore not helpful on its own. The gain comes from the rule and shift episodes, which together allow the model to handle new classes. On Rotated MNIST, the concept-free arm (0.704) also falls below the TabPFN baseline on ImageNet features (0.785, Table 1), so our architecture and in-domain meta-training do not account for the gain on their own.

CURE (Lee et al., 2026) and Lourenço et al. (2026) also treat learning from a stream as context management for frozen in-context models, a framing our method shares (Section 2). Both work on tabular data and give the in-context model the raw features, or summaries of them, with no learned representation. Our memory instead lives in a latent concept space that is meta-trained and shared across domains and classes. The concept-free arm above keeps our encoder and the frozen in-context classifier but removes concept anchoring, and loses 10.6–13.3% AA. A frozen in-context classifier with a memory is therefore not enough on its own. The gain depends on how the latent space is meta-trained.

The latent concept space also allows us to use a smaller memory. To show this, we vary k, the number of exemplars stored per class (per domain for Rotated MNIST, per hospital for CXR), and compare against the TabPFN baseline (ImageNet ResNet-18 → PCA-16) on the same memories and queries over 5 seeds. At each prediction we draw at most 50 class-balanced rows from the memory, repeat this 8 times and average the predictions (the bounded protocol, App. D), so the cost per prediction stays fixed however large the memory grows. Results with the whole memory as context are given in our response to Reviewer d2UX.

DatasetMethodk=1k=2k=5k=10k=20k=50
Rotated MNISTOurs0.6830.7210.7390.7390.7470.747
TabPFN0.5890.6900.7240.7320.7370.730
Split MNISTOurs0.5970.7210.7940.8240.8380.846
TabPFN0.3400.4630.6360.7140.7450.765
DatasetMethodk=1k=2k=4k=8k=16k=32
CXR TB (bal. acc.)Ours0.6650.6990.7510.7500.7460.753
TabPFN0.5490.5950.5980.6360.6740.677

Our method requires far fewer exemplars than a frozen TabPFN on ImageNet features to reach the same accuracy. On CXR, our method with four films per class per hospital (0.751) exceeds TabPFN at every memory size tested, up to 32 (0.677), and on Split MNIST, five exemplars per class (0.794) exceed TabPFN with fifty (0.765). On Rotated MNIST the advantage is largest at small k and narrows as memory grows.

W2 - Stability of latent space

Table A. Latent drift over the whole stream (Latent Concept PFN, full 16-d latent, frozen Table 1 models, 5 seeds, mean ± sd)

Rotated MNISTSplit MNIST
Linear-probe forgetting [1]−0.001 ± 0.0040.001 ± 0.002
Prototype drift [2]0.087 ± 0.0110.102 ± 0.024
Representation discrepancy, aligned / unaligned [3]0.15 / 0.450.12 / 0.46
Per-stage drift, stored / arriving0.057 / 0.0760.076 / 0.430
AUROC drift → forgetting (flip rate)0.70 ± 0.04 (5%)0.77 ± 0.10 (10%)

Each domain or task is compared between the stage it arrived and the end of the stream. Linear-probe forgetting: accuracy of a linear probe refit on the latent at arrival minus at the end. Prototype drift: shift of each class mean, divided by the mean distance between class means. Representation discrepancy: worst-case shift of the latent after the best linear map between the two stages (aligned) or with no map (unaligned), divided by the largest latent norm. Per-stage drift: per-query shift of the latent between consecutive stages, divided by the class separation, for domains or tasks already stored vs. the one arriving. AUROC: how well a query's latent drift from arrival to the end predicts that it goes from correct to wrong; flip rate is the fraction of queries correct at arrival that end wrong.

Table B. The Split MNIST event in Section 5 (seed 42, digits 2/3 when 6/7 arrive; 108 of 320 correctly classified images flipped)

FlippedStayed correct
Read-out change (7-d)0.0250.037
Latent drift (16-d)0.0990.100
AUROC drift → flip0.43

Change in each image's representation when 6/7 arrive. Read-out change: mean absolute change per concept of the 7-d concept read-out, as in Section 5. Latent drift: shift of the full 16-d latent, divided by the class separation. AUROC: how well latent drift predicts which images flip (0.5 = no information).

On the frozen Table 1 models, with the same streams, queries and protocol (no retraining), we computed the posterior mean of the full 16-d latent for every test query and memory exemplar, plus 300 held-out MNIST images per class from the training split that were never used as memory exemplars (the model was meta-trained only on EMNIST letters). We used these held-out images for the probe and the prototypes. We compared the representation of each domain or task between the stage it arrived and the end of the stream, over 5 seeds. We computed three standard metrics: linear-probe forgetting [1], class-prototype drift [2], and representation discrepancy with an exact linear alignment [3]. We also measured per-query drift of the latent, relative to the class separation, on stored and arriving data, and how well it predicts forgetting (AUROC).

On the Split MNIST event we discuss in Section 5 (Table B), flipped and unflipped images moved equally in the full latent: per-query drift is 0.099 vs. 0.100 of the class separation. Drift does not identify which images flipped (AUROC 0.43). This matches the 7-d read-out (0.025 vs. 0.037 per concept), so the claim now holds on the full 16-d latent. The information to separate 2s from 3s is also kept. A linear probe on the latent classifies them with 0.993 accuracy before 6/7 arrive and 0.985 after. The errors therefore come from the new 7 exemplars occupying the same region as the 2s that read like the letter ‘z’, not from the 2s’ own representation changing. In the revised paper, we will limit the stability claim in Section 5 to these images.

Across the whole stream (Table A), the information in the latent is kept: linear-probe forgetting is −0.001 on Rotated MNIST and 0.001 on Split MNIST. The latent itself does move moderately. Class prototypes shift by 0.087 and 0.102 of the distance between them. Most of this movement is shared, since a single linear map reduces the representation discrepancy from 0.45 to 0.15 (Rotated MNIST) and from 0.46 to 0.12 (Split MNIST). The movement is concentrated on new data: on Split MNIST, the per-query drift of the arriving task’s queries is 0.430 of the class separation, compared with 0.076 for tasks already stored. This is the adaptation through context the method is designed for. Across the stream, per-query drift is moderately predictive of which queries are forgotten (AUROC 0.70 and 0.77), but forgetting is rare (5% and 10% of queries correct at arrival). Some errors therefore involve a query’s own latent moving, while others, like the event discussed above, come from new neighbours in the memory.

[1] M. Davari, N. Asadi, S. Mudur, R. Aljundi, and E. Belilovsky, ‘Probing representation forgetting in supervised and unsupervised continual learning’, in 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), Los Alamitos, CA, USA: IEEE Computer Society, Jun. 2022, pp. 16691–16700. doi: 10.1109/CVPR52688.2022.01621. [2] L. Yu et al., ‘Semantic drift compensation for class-incremental learning’, in 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), Los Alamitos, CA, USA: IEEE Computer Society, Jun. 2020, pp. 6980–6989. doi: 10.1109/CVPR42600.2020.00701. [3] J. Kim, Y. Kim, and J.-Y. Sohn, ‘Measuring representational shifts in continual learning: a linear transformation perspective’, in Proceedings of the 42nd international conference on machine learning, in Proceedings of machine learning research, vol. 267. PMLR, Jul. 2025, pp. 30267–30289. [Online]. Available: https://proceedings.mlr.press/v267/kim25p.html

W3 - Least-forgetting claim

We thank the reviewer for pointing this out; the opening sentence of Section 5 was imprecise. Our method has low forgetting in every setting, but not the lowest. With FM computed on each seed's accuracy matrix and then averaged, TabPFN forgets slightly less on Rotated MNIST (−0.012 vs −0.007), and InfLoRA on Split MNIST (0.045 vs 0.073) and CXR (0.000 vs 0.007). In the revised paper, we will replace the sentence with "The Latent Concept PFN has the second-lowest forgetting measure in every setting and stays competitive without a single parameter update," and state explicitly where TabPFN and InfLoRA forget less.

Thank you for the detailed response. The controlled ablation and the analysis of all 16 dimensions have addressed my concerns. I have raised the score accordingly.

Reviewer_d2UXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes to use latent concept Prior-data fitted networks (PFNs) as a meta learning strategy for in-context continual learning. The proposed approach uses transformers to carry out posterior inference via PFNs. During meta learning, class labels are derived from a function of the concept and image features, and data perturbation has been applied to improve robustness. The model is trained to take a context dataset as input and predict the labels on the queries. The proposed approach also use an auxiliar loss based on the underlying concepts/attributes of each example. Continual inference is accomplished by updating the in-context examples. Experiments on rotated mnist, split mnist, and cxr demonstrate the proposed approach forgets the least in every setting, and outperforms the baseline in the class incremental setting. The paper also argue concept inference offers interpretability.

Critical strengths
  1. I liked the probabilistic formulation of the problem that casts continual learning as a meta-learning problem with the use of prior fitted networks. The problem formulation is clean and well-defined.
  2. Strong empirical results demonstrate the proposed approach outperforms the baselines with the least forgetting and outperforms baselines in the class-incremental setting.
  3. The paper analyzes the contributions between concept c and latent z to demonstrate they are complementary.
Critical weaknesses
  1. Evaluation fairness. My main concern of this paper is regarding the use of the concepts and fairness in evaluation. The paper makes the assumption that the concepts are shared across the continual learning stream. My understanding is that the concepts are provided as additional labeled data to facilitate learning, whereas the baseline methods, DER++, L2P, InfLoRA, do not make use of the concept information, which makes the evaluation unfair. Even thought the paper demonstrates concepts and latents are complementary, I did not find a clear comparison without the use of concepts.
  2. Scalability. It is unclear how scalable is this memory based approach, and when it should not be preferred. The experiments use small memories: 50, 250, or 128 examples in total. This shows that the method works on short continual streams, but do not establish scalability to large number of classes or long-running streams.
Questions for authors

Please address comments in the weaknesses section.

Additional question: Could you elaborate how to understand the rule episodes intuitively? It is not clear to me why this is cc to yy as xx and xˉ\bar{x} are also involved with the use of a random function and binning? Do you have ablation removing this component during training?

7 comments

We thank the reviewer for their detailed assessment. We have run a concept-free ablation, an ablation containing concepts but no rule or shift episodes, as well as a memory-size sweep, and address each point below.

W1 - Evaluation fairness (use of concepts)

We acknowledge that the method uses concept supervision at training time. However, concept annotations are only used during meta-training and never during the continual learning stream, the phase on which all methods are compared. For the MNIST streams, meta-training is done on EMNIST letters, which are class-disjoint from the stream. The exemplar memory stores only input–label pairs (Eq. 10), and predictions condition on these pairs alone, exactly as for the TabPFN baseline. At test time, the concept head is used only to produce read-outs for interpretation. This is analogous to how ImageNet pretraining shapes the feature space for L2P and InfLoRA: both are inductive biases acquired before the stream begins.

The supervision budget in fact favours the baselines. Our concept annotations for the MNIST experiments are a fixed lookup table of 26 letter classes × 7 binary attributes (182 bits). In contrast, L2P and InfLoRA build on a ViT-B/16 pretrained on ImageNet-21k (~14M images, ~21k classes) and are further trained with gradient updates on the full data at every step (five epochs per task). DER++ likewise performs gradient updates on the stream data at every step, whereas our model receives no parameter updates on the stream. Every method brings inductive bias from its pretraining; ours takes the form of a small, interpretable concept vocabulary rather than large-scale visual pretraining. We believe this makes the comparison fair. To isolate the contribution of the concepts directly, we ran the following ablation.

The concept-free arm uses identity episodes only and no concept loss (λc = 0), with the same training budget as our headline models. Everything else (encoder, frozen TabPFN, meta-training data) is kept the same, so this measures the full setup without any use of concepts. The only-ID arm also uses identity episodes only, but keeps the concept loss (λc = 1): concept information is present, but there are no rule or shift episodes.

AblationRotated MNIST AA% changeSplit MNIST AA% changeCXR TB AA (bal. acc.)% change from headline AA
Full (ours)0.789 ± 0.031—0.794 ± 0.021—0.749 ± 0.007—
Concept-free0.704 ± 0.026−10.6% ± 3.90.700 ± 0.012−11.8% ± 3.50.649 ± 0.015−13.3% ± 2.3
Only ID episodes0.690 ± 0.024−12.4% ± 2.80.705 ± 0.049−11.3% ± 4.90.652 ± 0.029−13.0% ± 3.5

Removing concepts altogether costs 10.6% on Rotated MNIST, 11.8% on Split MNIST and 13.3% on CXR (5 seeds, paired). Re-introducing concept supervision alone does not recover this. The only-ID arm is within noise of the concept-free arm on all three datasets (Rotated MNIST 0.690 vs 0.704, Split MNIST 0.705 vs 0.700, CXR 0.652 vs 0.649). Supervising the latent with a fixed, static vocabulary is therefore not helpful on its own. The gain comes from the rule and shift episodes, which together allow the model to handle new classes and new domains, we break this down in Q1.

W2 - Scalability

To test how the method scales with memory size, we vary k, the number of exemplars stored per class (per domain for Rotated MNIST, per hospital for CXR), and compare against the TabPFN baseline (ImageNet ResNet-18 → PCA-16) on the same memories and queries over 5 seeds. We evaluate two protocols. In the whole memory protocol, the full exemplar memory is passed as context, so context size grows with k and with the number of domains seen. In the bounded protocol (App. D), the context is capped: at each prediction we draw at most 50 class-balanced rows from the memory, repeat this 8 times and average the predictions. The bounded protocol keeps the cost per prediction fixed however large the memory grows, which is the relevant setting for long streams.

Bounded context (≤ 50 rows × 8 draws):

DatasetMethodk=1k=2k=5k=10k=20k=50
Rotated MNISTOurs0.6830.7210.7390.7390.7470.747
TabPFN0.5890.6900.7240.7320.7370.730
Split MNISTOurs0.5970.7210.7940.8240.8380.846
TabPFN0.3400.4630.6360.7140.7450.765
DatasetMethodk=1k=2k=4k=8k=16k=32
CXR TB (bal. acc.)Ours0.6650.6990.7510.7500.7460.753
TabPFN0.5490.5950.5980.6360.6740.677

Whole memory:

DatasetMethodk=1k=2k=5k=10k=20k=50
Rotated MNISTOurs0.6830.7400.7890.8200.8440.863
TabPFN0.5890.7060.7850.8170.8400.863
Split MNISTOurs0.5970.7210.7940.8390.8680.890
TabPFN0.3400.4630.6360.7350.8080.842
DatasetMethodk=1k=2k=4k=8k=16k=32
CXR TB (bal. acc.)Ours0.6650.6990.7510.7480.7500.758
TabPFN0.5490.5950.5980.6390.6680.697

From this, we can see that our method requires far fewer exemplars than a frozen TabPFN on ImageNet features to reach the same accuracy, and this holds even with the bounded subsampled context. With the bounded context, four films per class per hospital on CXR (0.751) exceed TabPFN at every memory size tested, up to 32 (0.677), and five exemplars per class on Split MNIST (0.794) exceed TabPFN with fifty (0.765). With the whole memory, the CXR result still holds (TabPFN reaches 0.697 at k = 32), and on Split MNIST TabPFN needs k = 20 (0.808) to overtake our k = 5. On Rotated MNIST the advantage is largest at small k and narrows as memory grows. Note that the front-end was meta-trained on contexts of at most 56 rows, yet whole-memory accuracy keeps rising well beyond this (up to 2,500 rows on Rotated MNIST), so the model can use larger memories without retraining. With the bounded context (App. D), the inference cost is fixed and does not grow with the length of the stream or the size of the memory. So, subsampling helps to make it scalable.

The method is less suited where many exemplars per task are available under pure covariate shift: on Rotated MNIST, TabPFN catches up once k ≥ 5 with the whole memory. The bounded context also trades accuracy for cost when many domains share a small budget (Rotated MNIST: ≈0.75 bounded vs 0.86 whole memory at k = 50). We will add both points to the limitations.

To the larger dataset point, we are currently running an additional, larger experiment and will post results in the coming days. Regarding the number of classes, the limit comes from the frozen backbone, TabPFN v2 (Hollmann et al., 2025), which supports up to 10 classes natively. We use v2 because it is openly licensed (Apache 2.0 with attribution). The method is not tied to this backbone: TabPFN-3 supports up to 160 classes natively, and Prior Labs' many-class extension extends v2 beyond 10 classes.

Q1 - Rule episodes

In identity episodes, the label is the source class, so the model could learn one fixed mapping from concepts to letters and memorise it. On the class-incremental stream, however, the classes are new and have their own mapping from concepts to classes, and the decision boundaries in concept space change as new classes arrive. For example, after the first Split MNIST task, a curved stroke is enough to separate 0 from 1. When 2 and 3 arrive, they are also curved, so the same concept no longer suffices: the model must now use the loop to separate 0 from 3, and the horizontal stroke to separate 2 from 3. Rule episodes train exactly this. The same concept code is mapped to different labels in different episodes, so the model learns to read from the context which concepts matter for the classes currently present, rather than memorising a fixed concept-to-class mapping.

Input xConcepts cIdentity episode: yRule episode 1: y = f₁( c )Rule episode 2: y = f₂( c )
a1000101010
o1000101110
e1010101202
k0101011321

Table: Each letter is its own class in an identity episode; in a rule episode the label is a function of c that is redrawn every episode, so the same concept code receives different labels.

We sample the rule from TabPFN's own BNN prior, as this is the family of functions TabPFN was pretrained to infer from context. The BNN outputs a real number per example, so we bin these outputs at the episode's own quantiles to obtain discrete, roughly balanced class labels. When κ = 0, the label is a function of the concepts alone. However, a predefined concept vocabulary is often incomplete, so for some episodes the rule also depends on the pixels (κ > 0), which forces z to carry information beyond the vocabulary. For example, a and o in the table above share a concept code, so when κ = 0 they always share a label and cannot be distinguished without pixel information. Similarly, in Section 5, open-topped 4s and nearly vertical 7s have almost identical concept codes, yet z separates them: one latent dimension captures open versus closed tops, a feature not in the vocabulary, and another separates those open-topped 4s from 7s (Fig. 2).

AblationRotated MNIST AA% changeSplit MNIST AA% changeCXR TB AA (bal. acc.)% change
Full (ours)0.789 ± 0.031—0.794 ± 0.021—0.749 ± 0.007—
No rule0.792 ± 0.023+0.5% ± 2.90.782 ± 0.047−1.5% ± 5.50.726 ± 0.019−3.2% ± 3.0
No shift0.713 ± 0.022−9.6% ± 1.80.776 ± 0.045−2.3% ± 4.90.697 ± 0.007−6.9% ± 0.9
No rule, no shift0.690 ± 0.024−12.4% ± 2.80.705 ± 0.049−11.3% ± 4.90.652 ± 0.029−13.0% ± 3.5
AblationTask 2/3, after 4/5Task 2/3, after 6/7Task 2/3, finalTask 6/7, on arrival
Full (ours)0.804 ± 0.0800.733 ± 0.1200.683 ± 0.1120.855 ± 0.013
No rule0.806 ± 0.1230.735 ± 0.1160.702 ± 0.1460.808 ± 0.022
No shift0.751 ± 0.0840.717 ± 0.1060.686 ± 0.1150.788 ± 0.015
No rule, no shift0.694 ± 0.0310.512 ± 0.1220.495 ± 0.1350.723 ± 0.073

Removing shift episodes costs most on the domain-incremental streams (−9.6% on Rotated MNIST, −6.9% on CXR), consistent with their role of preparing the model for changes in input appearance. Removing rule episodes as well costs a further 2.8% on Rotated MNIST and 6.1% on CXR. Averaged over tasks, removing rule episodes alone has a smaller effect, but a per-task breakdown on Split MNIST shows where they matter. First, rule episodes improve accuracy on newly arriving classes: Task 6/7 reaches 0.855 on arrival with rule episodes and 0.808 without. Second, they protect old classes when new classes with overlapping concept codes arrive. Without shift episodes, Task 2/3 retains 0.717 after 6 and 7 arrive; removing rule episodes as well drops it to 0.512. This collapse occurs exactly when 7s enter the memory, the same 2-versus-7 confusion we analyse in Section 5. With shift episodes present, the full model retains Task 2/3 at a similar level to the no-rule arm, so the two episode types partly compensate for each other on old classes, and rule episodes additionally improve accuracy on new ones. We will add the intuition above and this ablation to the revised paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised PDF

Summary of revisions

We thank the reviewers for engaging with our responses. We have uploaded the revised PDF, with all changes marked in blue. The main text remains within 10 pages and new results are summarised in Sections 4–6, with full details in the appendix.

  1. Matched concept-free ablation [d2UX, CjDb, Pt4H]. Table 2 and a new paragraph in Section 5, with details of how the arms are built and a per-task breakdown in App. N.2 (Table 14). Concept supervision alone does not improve accuracy; the 10.6–13.3% gain over the concept-free arm comes from the rule and shift episodes used in meta-training.
  2. Role of concept supervision [Pt4H]. We removed the statements that concept anchoring improves accuracy (Sections 2, 3.2, 5 and 6), attribute the gain to the meta-training episodes, and state the role of concept supervision as making the latent inspectable.
  3. Use of concepts [d2UX]. New "Use of concepts" paragraph in Section 4: annotations are used only to meta-train on the source dataset, and the stream and memory contain only images and labels, for all methods.
  4. Rule episodes [d2UX]. The intuition and a worked example of the three episode types are in App. E (Fig. 5), with a pointer from Section 3.3. The per-task effect of rule episodes is in Section 5 and App. N.2.
  5. Memory size [d2UX, CjDb, Pt4H]. A sweep over exemplars per class k for our method and TabPFN, with the whole memory and a bounded context, is in App. L (Table 9, Fig. 9) and summarised in Section 5. Where the advantage narrows (Rotated MNIST at large k) is now an explicit limitation in Section 6.
  6. Latent drift [CjDb, Pt4H]. A new paragraph in Section 5 and the measures in Section 4, with full results in App. M (Tables 10–11). The stability claim for the Split MNIST event in Section 5 is now limited to those images and confirmed on the full 16-d latent.
  7. Forgetting [CjDb, Pt4H]. Section 5 now states "second-lowest forgetting in every setting" and says where TabPFN and InfLoRA forget less. All metrics in Table 1 are now computed on each seed's accuracy matrix and then averaged.
  8. Number of latent samples [Pt4H]. S = 16 is within 0.006 AA of S = 128 at about 8× lower latency (Section 3.4 with full sweep in App. K, Table 8).
  9. Longer stream [d2UX, Pt4H]. A 25-step stream mixing Split MNIST pairs and Rotated MNIST angles is summarised in Section 5, with details in App. J (Tables 6–7). We have not yet validated a many-class extension, so the method is currently limited to 10 classes, as now stated in Section 6.
  10. Presentation [Pt4H]. We standardised abbreviations and cross-references (Sec., Eq., App., Fig., Table) and corrected minor typos.

Differences from our responses

  • Standard deviations. We said per-seed FM with standard deviations would be in Table 1. To stay within the page limit, Table 1 reports means. Standard deviations for all metrics are instead in App. N.1 Table 12 and per-seed FM in Table 13.
  • Rerun baselines. Computing metrics per seed requires each seed's accuracy matrix, which had not been saved for three baseline cells. We reran them with the same settings and report the reruns in Table 1: InfLoRA on Rotated MNIST (AA 0.958 → 0.950) and Split MNIST (0.714 → 0.757), and L2P on Rotated MNIST (0.813 → 0.796). Our method still has the highest AA on Split MNIST, and no conclusion changes.
  • Bounded context. The context protocol previously called "matched" (App. D.3) is now called "bounded", to avoid confusion with the matched ablation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.