acceptodds
Under review as a conference paper at ICLR 2027

RepICL: Reusable In-Context Prediction Across Heterogeneous Representation Spaces

16 views

Abstract

Frozen representations are widely reused for downstream classification, yet each new task typically requires fitting a new predictor. We ask whether the few-shot prediction procedure itself can instead be learned once and reused across datasets and representation spaces. To study this question, we introduce RepShiftBench, comprising 1,218 encoder–dataset tasks across text, image, and audio, with separate evaluation of generalization to unseen datasets, unseen encoders, jointly unseen datasets and encoders, and unseen modalities. The benchmark exposes a substantial gap: Logistic Regression fitted independently on each episode outperforms every evaluated in-context learner across all settings. We introduce RepICL, a meta-trained in-context learner that canonicalizes each episode through episodic whitening before prediction. Its inductive variant, RepICL-I, surpasses Logistic Regression in all 12 benchmark settings, while RepICL-T substantially outperforms existing transductive methods. Ablations identify episodic whitening as the primary source of these gains, while showing that it is not a universally beneficial preprocessing step. Across both variants, the gains concentrate on queries for which simple support prototypes favor the wrong class or provide little separation between the true class and competing classes. Transduction provides its largest additional gains when limited support coverage gives a misleading view of class separation. Together, these results demonstrate that a shared few-shot prediction procedure can generalize beyond the representation spaces observed during training.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Few-Shot Classification, In-Context Learning, Meta-Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_giHHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether a single few-shot predictor can be learned once and reused across heterogeneous frozen representation spaces—across datasets, encoders, and even modalities—without test-time updates. RepICL canonicalizes each episode via episodic whitening (episode-wise PCA + coordinate-wise standardization), then applies a shared Transformer-based in-context predictor with prototype classification, in inductive (RepICL-I) and transductive (RepICL-T) variants. On the authors’ RepShiftBench, RepICL-I surpasses per-episode logistic regression in all 12 settings, and RepICL-T gains more under transduction.

Critical strengths

Well written and easy to follow. The motivation—in-context learners break when the representation space changes—is clearly posed.

Critical weaknesses

1.The paper cites cross-domain few-shot learning only in passing (Tseng et al., 2020; Li et al., 2022) and compares against none of it. The central claim—a shared predictor that generalizes across representation spaces without test-time updates—overlaps substantially with what cross-domain few-shot methods on Meta-Dataset (Triantafillou et al., 2020) are designed to test, yet none appears in Table 1. LoRA in Appendix E.2 is a reasonable test-time-adaptation baseline but is not a cross-domain few-shot method. Without one or two recent cross-domain few-shot baselines on the same fixed episodes, the incremental value of RepICL cannot be assessed.

2.Applying an episode-level transformation so a shared predictor can span representation spaces is not new in kind. TAIL (Woerner et al., 2026) already projects encoder outputs into a common latent space and transfers to unseen modalities; the paper distinguishes itself mainly by arguing TAIL’s evaluation is not systematic—a critique of protocol, not mechanism. Representation normalization and alignment have also been studied via episode-statistic normalization (Bronskill et al., 2020), whitening with reference statistics (Su et al., 2021), and relative representations (Moschella et al., 2023). Episodic whitening is a variant of this family, and the paper does not clearly state what it adds beyond per-episode PCA + standardization. The authors should say plainly whether the contribution is the whitening choice, the meta-training on whitened episodes, or the benchmark—right now all three are claimed, but the ablation suggests only the pairing matters.

3.RepICL is PCA, standardization, a linear projection with label embeddings, a Transformer, and a prototype classifier. PCA and standardization are routine preprocessing; the label-embedding-plus-Transformer architecture is inherited from CAML (Fifty et al., 2024) and initialized from its checkpoint; the prototype classifier follows Prototypical Networks (Snell et al., 2017) and FEAT (Ye et al., 2020). Table 3 supports this reading: the same whitening does not help TabPFN or TabICL, so the benefit is the pairing, not a separable new mechanism. The paper needs either a mechanism-level explanation of why the pairing is necessary or a more modest positioning.

4.The method section states two different maximum PCA ranks for the inductive and transductive variants but describes both as matching the maximum ranks under the same five-way five-shot configuration; the numbers fit different estimation subsets (support-only vs. support-plus-query), yet the sentence conflates them, which is confusing in the core method description. Separately, TAIL scores 5.70% on Text-held-out Dataset adaptation in Table 1 while others exceed 56%; the appendix blames feature scale and applies norm calibration, but a near-random collapse still raises the question of how much the cross-encoder evaluation depends on baseline fragility to scale rather than genuine generalization.

Tian, H., Liu, F., Cheung, K. C., Fang, Z., See, S., Liu, T., & Han, B. Cross-domain Few-shot Classification via Invariant-content Feature Reconstruction. Zhuo, W., & Luo, S. (2026). Modality-free Graph In-context Alignment. Zhu, L., Yin, W., Wu, F., Gu, Q., Wang, X., Zhou, C., & Ye, N. Bayes-CAL: Robust Cross-Modal Alignment by Bayesian Approach for Few-Shot OoD Generalization. FAD: Frequency Adaptation and Diversion for Cross-domain Few-shot Learning. (2025).

Questions for authors

See Weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer giHH (1/2): W1

W1. Cross-domain few-shot learning (CDFSL) baselines.

Following the reviewer's suggestion, we evaluated the feature-adaptation component of MOKD [1], a recent cross-domain few-shot learning method (ICML 2024), on RepShiftBench and added the results to Table 1. The method is introduced in Section 6.1, with implementation details in Appendix C. RepICL-I achieves higher accuracy than MOKD in all 12 settings without test-time parameter updates. MOKD exceeds Logistic Regression only in modality adaptation under the image-held-out fold, and falls below it in the other 11 settings. These results suggest that methods developed for cross-domain few-shot image classification do not necessarily provide consistent gains over Logistic Regression across heterogeneous frozen representation spaces, consistent with the observations in Appendix E.

Several of the suggested methods require inputs or model access beyond frozen representations. IFR [2] uses augmented images and spatial features, FAD [3] adapts intermediate backbone features, and Bayes-CAL [4] uses class and domain names for image-text alignment. These methods therefore cannot be applied directly through RepShiftBench's common interface of pooled representations and support labels. MF-GIA [5] studies in-context alignment over pre-vectorized graph features, but its formulation uses graph structure and differs from few-shot prediction across general frozen encoder representations.

The LoRA comparison in Appendix E.2 serves as a reference for per-episode adaptation of the feature encoder at test time, rather than as a CDFSL baseline. Prior work has evaluated LoRA among parameter-efficient methods for few-shot image classification [6] and applied LoRA to few-shot adaptation of vision-language models [7]. LoRA uses the same labeled support examples but additionally accesses the raw inputs and encoder parameters.

References

[1] Hongduan Tian, Feng Liu, Tongliang Liu, Bo Du, Yiu-Ming Cheung, and Bo Han. MOKD: Cross-domain finetuning for few-shot classification via maximizing optimized kernel dependence. Proceedings of the 41st International Conference on Machine Learning, PMLR 235, pp. 48154–48185, 2024.

[2] Hongduan Tian, Feng Liu, Ka Chun Cheung, Zhen Fang, Simon See, Tongliang Liu, and Bo Han. Cross-domain few-shot classification via invariant-content feature reconstruction. International Journal of Computer Vision, 134, article 54, 2026.

[3] Ruixiao Shi, Fu Feng, Yucheng Xie, Jing Wang, and Xin Geng. FAD: Frequency adaptation and diversion for cross-domain few-shot learning. arXiv preprint arXiv:2505.08349, 2025.

[4] Lin Zhu, Weihan Yin, Fan Wu, Qinying Gu, Xinbing Wang, Chenghu Zhou, and Nanyang Ye. Bayes-CAL: Robust cross-modal alignment by Bayesian approach for few-shot OoD generalization. International Journal of Computer Vision, 133(10), pp. 7076–7109, 2025.

[5] Wei Zhuo and Siqiang Luo. Modality-free graph in-context alignment. International Conference on Learning Representations, 2026.

[6] Samyadeep Basu, Shell Xu Hu, Daniela Massiceti, and Soheil Feizi. Strong baselines for parameter-efficient few-shot fine-tuning. Proceedings of the AAAI Conference on Artificial Intelligence, 38(10), pp. 11024–11031, 2024.

[7] Maxime Zanella and Ismail Ben Ayed. Low-rank few-shot adaptation of vision-language models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 1593–1603, 2024.

AuthorsOpenReview ↗

Response to Reviewer giHH (2/2): W2-W4

W2. Contribution and distinction from prior work.

Our primary methodological contribution is RepICL, which couples episodic whitening as an episode-dependent statistical normalization interface with meta-training a shared predictor on whitened episodes to enable reusable few-shot prediction across heterogeneous frozen representation spaces. RepShiftBench provides a complementary contribution by systematically evaluating generalization across unseen datasets, encoders, jointly unseen pairs, and modalities. To clarify this positioning, we revised the contribution framing (Abstract and Introduction, Sec. 1), the comparison with TAIL and prior normalization/alignment methods (Related Work, Sec. 2), and the rationale for episodic whitening as a statistical normalization interface (Sec. 5.1).

The distinction is not only in the evaluation protocol, but also in how the predictor handles representation-space heterogeneity. TAIL uses randomized projections to map heterogeneous features into a common latent space. However, these data-independent projections do not explicitly normalize the statistical structure of each episode. In our setting, a shared predictor must operate across previously unseen encoders whose feature coordinates, scales, and covariance structures may differ substantially. RepICL therefore estimates PCA directions and coordinate scales independently from each episode, reducing representation-specific statistical variation and providing a more consistent interface for learning relationships among examples across representation spaces. This does not require semantic alignment between encoder coordinates.

Beyond the methodological difference, RepShiftBench systematically separates dataset and encoder novelty, while TAIL's evaluation does not examine these axes separately. Our improvements also extend beyond RepShiftBench: using TAIL's released evaluation code, RepICL-T achieves 92.31% mean accuracy despite being meta-trained without image-modality datasets or encoders, compared with 90.47% for our reproduction of TAIL (Appendix D.1).

We have also expanded the Related Work to acknowledge prior uses of normalization, whitening, and representation alignment, while clarifying how their objectives and assumptions differ from reusable few-shot prediction across previously unseen frozen encoder spaces.

W3. Evidence for the whitening–meta-training pairing.

Our ablations provide empirical support for the role of episodic whitening within RepICL's meta-training. Table 2 shows substantial performance degradation when removing coordinate-wise standardization or replacing episodic PCA with fixed or trainable projections. Importantly, these variants still meta-train the shared predictor on their respective transformed representations, yet do not match the performance achieved by the full model meta-trained on whitened episodes.

Table 3 further shows that applying episodic whitening as a drop-in preprocessing step to Logistic Regression, TabICL, or TabPFN does not improve their performance. Together, these findings are consistent with RepICL benefiting from the combination of episode-dependent statistical normalization and a predictor trained on the resulting representations, rather than whitening being a universally beneficial preprocessing step.

The main experiments and component ablations supporting this interpretation were already present in the original submission. We have revised the Ablation Study (Sec. 6.3) to clarify their interpretation of these findings and Appendix D.2 to provide further implementation details for the fixed and trainable projection ablations.

W4. PCA rank and TAIL's sensitivity to feature scale.

(a) PCA rank. Thank you for pointing out this ambiguity. We have clarified the relationship between the PCA estimation sets and their maximum ranks in Episodic Whitening (Sec. 5.1): “We set rA=min⁡(dE,∣A∣−1,rmax⁡)r_A = \min(d_E, |A| - 1, r_{\max}). RepICL-I estimates PCA from the 25 support examples alone, whereas RepICL-T also includes the 50 unlabeled queries, giving maximum ranks after centering of rmax⁡=24r_{\max} = 24 and 7474, respectively.”

(b) TAIL and feature scale. Table 1 reports 52.79% for TAIL on Text-held-out dataset adaptation; we could not find 5.70%. The reviewer is right that TAIL is sensitive to scale. Without calibration it drops to 24.3–38.2% (Appendix C.4), so every TAIL result in Table 1 uses scale calibration. With calibration it is still below LR in all 12 settings. We therefore do not attribute the gap to feature scale alone. Generalizing to new representation spaces remains difficult after scale is corrected, and this is the research problem RepICL addresses.

Thank you for the revisions. I'm happy to see W4(a) is now fixed, and I appreciate the addition of MOKD for W1—though I'd gently note that a single CDFSL baseline still leaves the cross-domain comparison somewhat thin.

On W3, I hope this comes across as constructive rather than repetitive: the response confirms that the pairing works (whitening alone doesn't), which I fully agree with, but I was hoping for an explanation of why the pairing matters rather than the observation itself. Similarly, W2 describes what the method does, but it would be helpful to say why it must be PCA specifically, and which of the three contributions is the primary one.

I'll keep my score for now, but I'd genuinely encourage the authors to address these "why" questions directly in a future version—I think doing so would strengthen the paper considerably.

Reviewer_ef4jOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to learn a reusable few-shot. in-context prediction model that can used across datasets and embedding models that does not require any training for new tasks. To achieve this they 1) introduced a multi-modal benchmark called RepShiftBench that contains many tasks, modalities and encoders and 2) proposed RepICL a meta-trained in-content learner that utilizes whitenining to improve performance and compete with downstream classification. Experiment results show that RepICL can generalize beyond representation spaces (encoders) observed ruing training.

Critical strengths

This paper is presented very clearly, and the main idea and insight are clearly communicated.

This paper proposed a novel benchmark RepShiftBench that considers encoder heterogeneity allowing the authors to demonstrate generalization beyond the representation spaces.

Authors showed and provide an important insight whitening suggesting it as a mandatory step for generalization to unseen tasks.

Critical weaknesses
  1. In the whitening for RepICL-T, do we have to assume that we have access to all the queries? In practice, don't we get the test samples one by one?

  2. In the experiment table, can you please also report the computation and memory cost of both training and test? I think it would be very valuable for readers to understand and compare.

  3. (minor) The definition of episode is not provided in the introduction which can be hard to follow for first-time readers.

Questions for authors

Please read weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer ef4j

W1. Query access.

RepICL provides inductive and transductive variants for different prediction requirements. When each query requires an immediate prediction, RepICL-I estimates the episode transformation from the support set alone and predicts each query independently. When a group of unlabeled queries is available before prediction, RepICL-T can additionally use their joint information. We compare RepICL-T with transductive methods having the same query access. Requiring the queries to be available together is part of the transductive inference setting, not a restriction specific to RepICL.

W2. Computation and memory.

We have added RepICL's peak GPU memory usage during training and inference to Appendix B, alongside the existing training-time results. Per-episode inference timings are already reported in Appendix E.3.

We report resource costs separately from Table 1 because most baselines require no meta-training in our evaluation setup, making training-cost columns difficult to interpret uniformly across methods. The tables below summarize training and inference resource costs. We also include TabPFN and TabICL as GPU-based inference references, since both run on GPU in our evaluation. We use their respective released checkpoints without retraining, so their training time and training memory were not measured in our study. Logistic Regression is included as a CPU inference-time reference.

Training resources

MethodParamsTrain timePeak VRAM
RepICL-I303M31.3 h14.4 GiB
RepICL-T303M44.7 h19.5 GiB
TAIL302M24.3 h6.4 GiB

Inference resources

MethodTime / episodePeak VRAM
RepICL-I5.3 ms1.1 GiB
RepICL-T5.9 ms1.1 GiB
TAIL4.3 ms1.7 GiB
TabPFN0.74 s0.8 GiB
TabICL1.5 s5.5 GiB
Logistic Regression6.7 ms (CPU)–

W3. Definition of an episode.

We have added a brief definition at the term's first occurrence in the Introduction: an episode consists of a labeled support set and a query set to be classified. We keep this explanation concise to preserve the flow of the Introduction, while the formal definition and detailed setup are provided in Section 3.

I thank the authors for taking the time to respond to my comments. After reading all the reviews, I maintain my positive score.

Reviewer_YSdoOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on a few-shot learning procedure that doesn't require individual fitting of task-specific classifiers. To unify few-shot tasks and enable a learner to adapt to all tasks simutaneously, the authors proposed to perform episode-specific whitening with PCA and coordinate standarization. Then, the standarized training and query samples are sent to a transformer-based learner for few-shot learning. The paper also proposes a comprehensive benchmark, referred to as RepShiftBench.

Critical strengths
  1. The benchmark is a clear strength providing value. Consisting of more than 1000 dataset tasks, the proposed benchmark enables thorough testing, focuses not only the dataset, but also the encoders, and also supports an empirical finding by the authors that seem a bit surprising: the simple logistic regression method outperforming most in-context learners.
  2. The obtained experimental results from both proposed variants are strong and consistent across all the evaluation settings and various data modalities.
Critical weaknesses
  1. The reviewer is highly concerned regarding the technical novelty of the paper. The major method components, such as applying transformer for in-context learning, prototype-based few-shot classification, the concept of inductive/transductive variants, are well explored in the meta-learning literature. The novelty mainly resides in episodic-specific whitening with PCA and coordinate standarization, which is also not entirely novel, as embedding unifications have also been explored in the settings such as transfer learning.
  2. The experimental results suggest that improvements under RepICL-I over logistic regression are mild, especially considering the capacity of the underlying transformer. While the other variant shows promising results, it relies on the extra information from unlabeled query.
Questions for authors

For whitening and standarization across task embeddings, have the authors considered other techniques beside PCA? Perhaps it is beneficial by having representation-alignment mappings that are more complex than linear mappings.

1 comment
AuthorsOpenReview ↗

Response to Reviewer YSdo

W1. Technical novelty.

We agree that the individual components, including whitening, Transformer-based in-context learning, and prototype classification, are established techniques. Our methodological contribution is not a new whitening algorithm, but using episodic whitening as an episode-dependent statistical normalization interface for a shared predictor meta-trained on the resulting representations, enabling reusable few-shot prediction across heterogeneous frozen representation spaces. RepShiftBench provides a complementary contribution by systematically evaluating generalization across dataset, encoder, joint, and modality shifts. It also reveals a substantial empirical gap: per-episode Logistic Regression outperforms all evaluated prior in-context learners across all 12 benchmark settings.

Regarding embedding unification, prior approaches often aim to align representations across different spaces or improve representation quality. Our setting differs: we seek to reuse a single predictor on representations from previously unseen encoders, without requiring explicit alignment to a reference space. Episodic whitening is estimated independently for each episode to reduce representation-specific statistical variation, rather than to establish shared semantic coordinates across encoders.

We have clarified the contribution framing (Abstract, Introduction [Sec. 1], and Conclusion [Sec. 7]), distinguished prior normalization and alignment work from our setting (Related Work, Sec. 2), and explained the role of episodic whitening in shared prediction (Sec. 5.1). We also refined the interpretation of existing ablations (Sec. 6.3). The core method and supporting evidence were already present in the original submission; these revisions primarily clarify our contribution positioning and make the underlying rationale more explicit.

W2. Inductive and transductive gains.

Our focus is on learning a shared predictor that can be reused across unseen representation spaces without test-time parameter updates. While RepICL-I's gains over Logistic Regression are modest (0.30–1.26 points), the key finding is that a single meta-trained predictor can outperform a strong classifier fitted separately on each episode, even when applied to previously unseen representation spaces. RepICL-I achieves this in all 12 settings, significantly in 10 (p < 0.01; Table 1), whereas none of the other evaluated inductive in-context learners surpasses Logistic Regression in any setting.

RepICL-T additionally uses unlabeled query information, so we compare it with transductive baselines that have the same query access. Despite this access, the best existing transductive baseline remains below inductive Logistic Regression in 11 of 12 settings. This includes TAIL, which has a comparable parameter count (302M vs. 303M) but remains below Logistic Regression in all 12 settings. RepICL-T exceeds Logistic Regression in all 12 settings by 1.16–5.02 points and the best transductive baseline by 2.65–6.66 points, significantly in all 12 settings.

Q1. Alternatives to PCA.

To investigate whether a nonlinear transformation offers additional benefits, we trained both RepICL variants using RBF kernel PCA instead of linear PCA. We use the RBF kernel k(x,y)=exp⁡(−γ∥x−y∥2)k(x,y)=\exp(-\gamma\lVert x-y\rVert^2), with γ\gamma set to the inverse median pairwise squared distance within each episode's fitting pool. The results below compare the two transformations under the four adaptation settings in Table 1: dataset (D), encoder (E), joint (J), and modality (M) adaptation. All values are accuracy (%).

I/T denote RepICL-I/RepICL-T; RBF denotes RBF kernel PCA. Linear PCA results are from Table 1.

Held-out foldSettingI: PCAI: RBFT: PCAT: RBF
TextD61.4160.8463.5562.65
TextE75.8575.4577.9977.45
TextJ66.2265.8668.2967.38
TextM70.1069.3073.2672.02
AudioD73.5272.7975.2774.31
AudioE76.7976.1178.1377.26
AudioJ75.8675.4577.4776.85
AudioM55.1454.2955.6254.37
ImageD59.1958.5361.7960.68
ImageE72.8272.1976.5875.51
ImageJ62.5662.1965.0264.33
ImageM83.2683.0685.6985.29

Linear PCA consistently outperforms RBF kernel PCA across all 12 settings for both variants. These results support our choice of linear PCA for episodic whitening. More expressive learned alignment mappings remain an interesting direction, although they would also need to generalize to encoders unseen during meta-training.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback. We have revised the manuscript to clarify the contributions of RepICL and RepShiftBench and more explicitly explain the role of episodic whitening as an episode-dependent statistical normalization interface for a shared meta-trained predictor.

The revisions clarify the research motivation and contributions (Abstract and Introduction, Sec. 1), distinguish prior normalization and alignment methods (Related Work, Sec. 2), explain the rationale for episodic whitening (Sec. 5.1), refine the interpretation of existing ablations (Sec. 6.3 and Appendix D.2), and align the Conclusion (Sec. 7) with this framing. We have also added the MOKD feature-adaptation comparison (Table 1), clarified the episode definition (Sec. 1), and expanded training resource reporting (Appendix B).

The core method and primary empirical findings remain unchanged; the revisions primarily sharpen their presentation and interpretation, alongside the additional evaluations and details requested by reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.