- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Latent Concept PFN, which combines a concept-supervised, meta-trained latent encoder with a frozen TabPFN classifier. During continual learning, all parameters remain fixed, and adaptation occurs through a growing exemplar context.
Critical strengths
The approach explores a useful alternative to continual parameter optimization. The EMNIST-to-MNIST setting separates source and target classes, and the reported Split MNIST result is competitive with parameter-updating baselines.
Critical weaknesses
- The proposed model and TabPFN baseline differ in front-end architecture and meta-training, so their performance gap does not isolate concept anchoring.
- Table 1 contradicts the least-forgetting claim: TabPFN has lower FM on Rotated MNIST, and InfLoRA has lower FM on Split MNIST and CXR.
- Stable seven-dimensional concept read-outs cannot establish stability of the full 16-dimensional context-dependent latent or explain error causation.
- Evaluation uses growing memories and 128 latent samples but covers only four or five stages and at most ten classes.
- Please standardize abbreviations and cross-reference formatting, including "Sec." and "Section," "Eq." and "Eq," and "App." and "Appendix".
Questions for authors
- Does concept supervision improve performance under a matched meta-training protocol?
- Can the authors report FM computed per seed before averaging, alongside direct latent-drift measurements as the context grows?
- What is the accuracy–latency trade-off when reducing the 128 inference-time latent samples?
- Could the authors clarify which figures are intended by “Figs. 3, 3, 1”?
7 comments
Table B. The Split MNIST event in Section 5 (seed 42, digits 2/3 when 6/7 arrive; 108 of 320 correctly classified images flipped)
| Flipped | Stayed correct | |
|---|---|---|
| Read-out change (7-d) | 0.025 | 0.037 |
| Latent drift (16-d) | 0.099 | 0.100 |
| AUROC drift → flip | 0.43 |
Change in each image's representation when 6/7 arrive. Read-out change: mean absolute change per concept of the 7-d concept read-out, as in Section 5. Latent drift: shift of the full 16-d latent, divided by the class separation. AUROC: how well latent drift predicts which images flip (0.5 = no information).
On the frozen Table 1 models, with the same streams, queries and protocol (no retraining), we computed the posterior mean of the full 16-d latent for every test query and memory exemplar, plus 300 held-out MNIST images per class from the training split that were never used as memory exemplars (the model was meta-trained only on EMNIST letters). We used these held-out images for the probe and the prototypes. We compared the representation of each domain or task between the stage it arrived and the end of the stream, over 5 seeds. We computed three standard metrics: linear-probe forgetting [1], class-prototype drift [2], and representation discrepancy with an exact linear alignment [3]. We also measured per-query drift of the latent, relative to the class separation, on stored and arriving data, and how well it predicts forgetting (AUROC).
On the Split MNIST event we discuss in Section 5 (Table B), flipped and unflipped images moved equally in the full latent: per-query drift is 0.099 vs. 0.100 of the class separation. Drift does not identify which images flipped (AUROC 0.43). This matches the 7-d read-out (0.025 vs. 0.037 per concept), so the claim now holds on the full 16-d latent. The information to separate 2s from 3s is also kept. A linear probe on the latent classifies them with 0.993 accuracy before 6/7 arrive and 0.985 after. The errors therefore come from the new 7 exemplars occupying the same region as the 2s that read like the letter ‘z’, not from the 2s’ own representation changing. In the revised paper, we will limit the stability claim in Section 5 to these images.
Across the whole stream (Table A), the information in the latent is kept: linear-probe forgetting is −0.001 on Rotated MNIST and 0.001 on Split MNIST. The latent itself does move moderately. Class prototypes shift by 0.087 and 0.102 of the distance between them. Most of this movement is shared, since a single linear map reduces the representation discrepancy from 0.45 to 0.15 (Rotated MNIST) and from 0.46 to 0.12 (Split MNIST). The movement is concentrated on new data: on Split MNIST, the per-query drift of the arriving task’s queries is 0.430 of the class separation, compared with 0.076 for tasks already stored. This is the adaptation through context the method is designed for. Across the stream, per-query drift is moderately predictive of which queries are forgotten (AUROC 0.70 and 0.77), but forgetting is rare (5% and 10% of queries correct at arrival). Some errors therefore involve a query’s own latent moving, while others, like the event discussed above, come from new neighbours in the memory.
[1] M. Davari, N. Asadi, S. Mudur, R. Aljundi, and E. Belilovsky, ‘Probing representation forgetting in supervised and unsupervised continual learning’, in 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), Los Alamitos, CA, USA: IEEE Computer Society, Jun. 2022, pp. 16691–16700. doi: 10.1109/CVPR52688.2022.01621. [2] L. Yu et al., ‘Semantic drift compensation for class-incremental learning’, in 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), Los Alamitos, CA, USA: IEEE Computer Society, Jun. 2020, pp. 6980–6989. doi: 10.1109/CVPR42600.2020.00701. [3] J. Kim, Y. Kim, and J.-Y. Sohn, ‘Measuring representational shifts in continual learning: a linear transformation perspective’, in Proceedings of the 42nd international conference on machine learning, in Proceedings of machine learning research, vol. 267. PMLR, Jul. 2025, pp. 30267–30289. [Online]. Available: https://proceedings.mlr.press/v267/kim25p.html
W5 / Q4 - Formatting and cross-references
We thank the reviewer for catching this. The reference should read (Fig. 3, Table 1). The per-task curves are in Fig. 3 and the forgetting metrics in Table 1. We will correct this and standardise abbreviations and cross-reference formatting (Sec., Eq., App.) throughout the revised PDF.
W4 - Stream length, memory size and number of classes
To test how the method scales with memory size, we vary k, the number of exemplars stored per class (per domain for Rotated MNIST, per hospital for CXR), and compare against the TabPFN baseline (ImageNet ResNet-18 → PCA-16) on the same memories and queries over 5 seeds. We evaluate two protocols. In the whole memory protocol, the full exemplar memory is passed as context, so context size grows with k and with the number of domains seen. In the bounded protocol (App. D), the context is capped: at each prediction we draw at most 50 class-balanced rows from the memory, repeat this 8 times and average the predictions. The bounded protocol keeps the cost per prediction fixed however large the memory grows, which is the relevant setting for long streams.
Bounded context (≤ 50 rows × 8 draws):
| Dataset | Method | k=1 | k=2 | k=5 | k=10 | k=20 | k=50 |
|---|---|---|---|---|---|---|---|
| Rotated MNIST | Ours | 0.683 | 0.721 | 0.739 | 0.739 | 0.747 | 0.747 |
| TabPFN | 0.589 | 0.690 | 0.724 | 0.732 | 0.737 | 0.730 | |
| Split MNIST | Ours | 0.597 | 0.721 | 0.794 | 0.824 | 0.838 | 0.846 |
| TabPFN | 0.340 | 0.463 | 0.636 | 0.714 | 0.745 | 0.765 |
| Dataset | Method | k=1 | k=2 | k=4 | k=8 | k=16 | k=32 |
|---|---|---|---|---|---|---|---|
| CXR TB (bal. acc.) | Ours | 0.665 | 0.699 | 0.751 | 0.750 | 0.746 | 0.753 |
| TabPFN | 0.549 | 0.595 | 0.598 | 0.636 | 0.674 | 0.677 |
Whole memory:
| Dataset | Method | k=1 | k=2 | k=5 | k=10 | k=20 | k=50 |
|---|---|---|---|---|---|---|---|
| Rotated MNIST | Ours | 0.683 | 0.740 | 0.789 | 0.820 | 0.844 | 0.863 |
| TabPFN | 0.589 | 0.706 | 0.785 | 0.817 | 0.840 | 0.863 | |
| Split MNIST | Ours | 0.597 | 0.721 | 0.794 | 0.839 | 0.868 | 0.890 |
| TabPFN | 0.340 | 0.463 | 0.636 | 0.735 | 0.808 | 0.842 |
| Dataset | Method | k=1 | k=2 | k=4 | k=8 | k=16 | k=32 |
|---|---|---|---|---|---|---|---|
| CXR TB (bal. acc.) | Ours | 0.665 | 0.699 | 0.751 | 0.748 | 0.750 | 0.758 |
| TabPFN | 0.549 | 0.595 | 0.598 | 0.639 | 0.668 | 0.697 |
From this, we can see that our method requires far fewer exemplars than a frozen TabPFN on ImageNet features to reach the same accuracy, and this holds even with the bounded subsampled context. With the bounded context, four films per class per hospital on CXR (0.751) exceed TabPFN at every memory size tested, up to 32 (0.677), and five exemplars per class on Split MNIST (0.794) exceed TabPFN with fifty (0.765). With the whole memory, the CXR result still holds (TabPFN reaches 0.697 at k = 32), and on Split MNIST TabPFN needs k = 20 (0.808) to overtake our k = 5. On Rotated MNIST the advantage is largest at small k and narrows as memory grows. Note that the front-end was meta-trained on contexts of at most 56 rows, yet whole-memory accuracy keeps rising well beyond this (up to 2,500 rows on Rotated MNIST), so the model can use larger memories without retraining. With the bounded context (App. D), the inference cost is fixed and does not grow with the length of the stream or the size of the memory. So, subsampling helps to make it scalable.
The method is less suited where many exemplars per task are available under pure covariate shift: on Rotated MNIST, TabPFN catches up once k ≥ 5 with the whole memory. The bounded context also trades accuracy for cost when many domains share a small budget (Rotated MNIST: ≈0.75 bounded vs 0.86 whole memory at k = 50). We will add both points to the limitations.
We are currently running an additional, larger experiment and will post results in the coming days. Regarding the number of classes, the limit comes from the frozen backbone, TabPFN v2 (Hollmann et al., 2025), which supports up to 10 classes natively. We use v2 because it is openly licensed (Apache 2.0 with attribution). The method is not tied to this backbone: TabPFN-3 supports up to 160 classes natively, and Prior Labs' many-class extension extends v2 beyond 10 classes.
We thank the reviewer for taking the time to provide detailed feedback. We will standardise the cross-referencing throughout, and we have run a matched meta-training ablation, per-seed forgetting metrics, latent-drift measurements and an accuracy–latency sweep, which we address below.
W1/Q1 - Matched meta-training protocol
The TabPFN baseline differs from our front-end in architecture and training: it uses a frozen ImageNet ResNet-18 with PCA, whereas our embedder is a CNN meta-trained on EMNIST. We chose this baseline because passing pretrained image features to TabPFN is the standard way to apply TabPFN to images. However, we agree that this comparison does not isolate the contribution of concept anchoring, so we ran the following ablation.
The concept-free arm uses identity episodes only and no concept loss (λc = 0), with the same training budget as our headline models. Everything else (encoder, frozen TabPFN, meta-training data) is kept the same, so this measures the full setup without any use of concepts. The only-ID arm also uses identity episodes only, but keeps the concept loss (λc = 1): concept information is present, but there are no rule or shift episodes.
Removing concepts altogether costs 10.6% on Rotated MNIST, 11.8% on Split MNIST and 13.3% on CXR (5 seeds, paired). Re-introducing concept supervision alone does not recover this. The only-ID arm is within noise of the concept-free arm on all three datasets (Rotated MNIST 0.690 vs 0.704, Split MNIST 0.705 vs 0.700, CXR 0.652 vs 0.649). Supervising the latent with a fixed, static vocabulary is therefore not helpful on its own. The gain comes from the rule and shift episodes, which together allow the model to handle new classes. On Rotated MNIST, the concept-free arm (0.704) also falls below the TabPFN baseline on ImageNet features (0.785, Table 1), so our architecture and in-domain meta-training do not account for the gain on their own.
To answer Q1 directly: under a fully matched protocol (the only-ID and concept-free arms differ only in λc), concept supervision alone does not improve performance. The improvement comes from using concepts to define the meta-training episodes (rule and shift episodes).
W2 - Least-forgetting claim
We thank the reviewer for pointing this out; the opening sentence of Section 5 was imprecise. Our method has low forgetting in every setting, but not the lowest. With FM computed on each seed's accuracy matrix and then averaged (Q2), TabPFN forgets slightly less on Rotated MNIST (−0.012 vs −0.007), and InfLoRA on Split MNIST (0.045 vs 0.073) and CXR (0.000 vs 0.007). In the revised paper, we will replace the sentence with "The Latent Concept PFN has the second-lowest forgetting measure in every setting and stays competitive without a single parameter update," and state explicitly where TabPFN and InfLoRA forget less.
W3 / Q2 - Direct latent drift as context grows
Table A. Latent drift over the whole stream (Latent Concept PFN, full 16-d latent, frozen Table 1 models, 5 seeds, mean ± sd)
Each domain or task is compared between the stage it arrived and the end of the stream. Linear-probe forgetting: accuracy of a linear probe refit on the latent at arrival minus at the end. Prototype drift: shift of each class mean, divided by the mean distance between class means. Representation discrepancy: worst-case shift of the latent after the best linear map between the two stages (aligned) or with no map (unaligned), divided by the largest latent norm. Per-stage drift: per-query shift of the latent between consecutive stages, divided by the class separation, for domains or tasks already stored vs. the one arriving. AUROC: how well a query's latent drift from arrival to the end predicts that it goes from correct to wrong; flip rate is the fraction of queries correct at arrival that end wrong.