- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a single in-context predictor for classification over foundation-model embeddings: it adapts the tabular foundation model TabICL with shared LoRA adapters trained across many task–encoder pairs. During training, each step applies a fresh random orthogonal rotation to the embeddings, which keeps their geometry but removes any fixed coordinate system, which helps generalization across unseen tasks. At inference, the model classifies new tasks and encoders with no gradient updates by averaging predictions over several random rotations. On a new benchmark of 7 domains, 75 tasks, 29 encoders and 308 task–encoder pairs, it outperforms the evaluated baselines.
Critical strengths
The problem is clearly stated, and the underlying scientific question—Can we build a task predictor that is agnostic to the input modality?—is both relevant and well motivated. The use of tabular foundation models seems to be a promising approach to addressing this question.
The use of random projections is interesting, providing a clear benefit in terms of improving the model's ability to generalize across different input modalities.
Compared to the evaluated baselines, the proposed method appears to achieve noticeable performance improvements across multiple modalities, although these results should be interpreted with some caution (see weaknesses below).
Critical weaknesses
Lack of comparison with meta-learning approaches. The proposed approach is closely related to meta-learning, where a model is trained episodically on support and query sets, with the objective of predicting the query set given only the support set at inference time. However, none of the baselines considered in the experimental evaluation appears to follow a similar episodic training procedure, potentially making the comparison unfair.
Lack of ablation on cross-modal training. To properly assess the benefits of learning from multiple modalities, I believe it would be important to compare the proposed model against versions trained on individual modalities. Such an ablation would help showing whether cross-modal training actually improves performance over modality-specific training, and to what extent the different modalities contribute to the model's generalization capabilities.
Additional comments and suggestions:
- Although potentially computationally expensive, it would be interesting to investigate how much each modality or domain contributes to performance on the others. For instance, one might expect training on protein-related tasks to benefit downstream other biological tasks more than training on sentiment analysis tasks.
- I believe the paper The Platonic Representation Hypothesis (2024) could be interesting to discuss, as it investigates the extent to which representations learned from different modalities converge toward a shared representation of reality. This perspective could help motivate why learning task predictors across modalities might be beneficial.
- Minor presentation issue: The organization of Section 5 seems uneven. Most subsections consist of approximately one paragraph, whereas Section 5.4 spans an entire page. (this is more to help increase the paper's readability, but it is a very minor weakness)
Questions for authors
-
Did the authors consider including meta-learning approaches as baselines? If so, could they clarify why these methods were excluded from the experimental evaluation?
-
Do the authors have any experimental results or analyses that directly assess the benefits of training on multiple modalities within their proposed framework? In particular, how does the performance of the proposed model compare to that of a counterpart trained on a single modality?
2 comments
As the discussion period closes on November 18, we would welcome your feedback on our response. We would be happy to clarify any points or address any remaining questions or concerns.
Thank you for your time and consideration.
Thank you for identifying these two gaps. We conducted additional experiments addressing both: episodically trained meta-learning baselines and a comparison between joint and single-domain training. We also evaluated transfer to entirely held-out domains.
W1 / Q1. Comparison with meta-learning approaches
We agree that comparison with other episodically trained predictors strengthens the evaluation. We therefore added adapted implementations of GPICL (Kirsch et al., 2022) and CAML (Fifty et al., 2024), two relevant transformer-based meta-learning approaches that learn from labeled context examples without parameter updates at inference. Both are trained on the same meta-training task–encoder pairs, use the same data splits, and follow the same checkpoint-selection criterion as our method.
To accommodate their fixed input width, we map embeddings to 512 dimensions using random linear projections. Random projection has precedent as dimensionality-reduction preprocessing for tabular in-context predictors, including TabFlex (Zeng et al., 2025). We configure both output heads to support up to 10 classes, covering all tasks in our benchmark without changing their labels. These adaptations enable evaluation across the same heterogeneous task–encoder pairs; our method directly accepts each encoder’s original dimensionality.
A same-backbone control further addresses episodic training. In Table 4, TabICL with LoRA trained on the same episodes using raw embeddings achieves 0.8822 overall and 0.8690 on unseen tasks and/or encoders, versus 0.8886 and 0.8879 for ours. Both average 16 inference views—feature permutations for the raw-trained model and orthogonal transformations for ours—so this comparison supports the combined training-and-inference procedure.
W2 / Q2. How does training across domains affect performance and transfer?
We compare three training settings: only the evaluation domain, all other domains, and all seven domains. The first tests domain specialization; the second tests transfer to a domain excluded from meta-training; and the third evaluates joint training. These experiments use a single matched task–encoder partition seed, the same training recipe, maximum optimization budget, and early-stopping procedure. (Table 3 averages three runs, so the all-domain reference scores is a bit differ between the two evaluations).
Joint training provides one shared predictor with competitive performance across domains. Compared with single-domain training, its largest observed gains are in molecules and audio. The graph-domain average also improves, while differences in the remaining four domains are at most 0.0015. Benefits are not uniform: on the held-out graph-task subset, the graph specialist performs better.
Most of this performance transfers even when the evaluation domain is excluded from meta-training. Across all 308 pairs, the leave-one-domain-out models achieve 0.8854, compared with 0.8883 for the matched all-domain model. They outperform frozen TabICL and the tuned MLP by point estimate in every domain.
To investigate which source domains transfer well, we also evaluated each specialist across all seven domains. Transfer is not confined to related domains: on molecules, protein-only, text-only, and time-series-only training achieve 0.8570, 0.8557, and 0.8577, respectively, compared with 0.8450 for molecule-only training. We will include the complete 7×7 transfer matrix in the revision. This matrix characterizes transfer from individual source domains; it does not isolate their marginal contributions within joint training.
Related work and presentation. Thank you for suggesting The Platonic Representation Hypothesis. We will discuss its connection to our motivation while distinguishing representation alignment from our finding that a shared predictor can transfer across embedding spaces. We will also reorganize Section 5.4 into shorter thematic subsections to improve readability.