acceptodds
Under review as a conference paper at ICLR 2027

MT-Tab: Leveraging Inter-Target Dependencies in Tabular Foundation Models for Multi-Target Prediction

Abstract

Tabular data are widely used across industrial domains such as manufacturing, finance, and healthcare. Tabular foundation models (TFMs), including TabPFN and TabICL, showcase strong generalization capabilities, outperforming Gradient Boosted Decision Trees (GBDTs) through in-context learning. However, existing TFMs are designed for single-target prediction. When applied to multi-target prediction problems which are common in real-world applications, they suffer from three limitations: (i) inference costs and resource requirements scale linearly with the number of targets, (ii) dependencies among targets are not fully leveraged, and (iii) directly modeling a joint distribution during pretraining is computationally expensive and increases architectural complexity. In particular, (ii) is the central motivation of this paper: single-target TFMs fundamentally exclude from the prediction path the information among multiple targets that is usable for prediction, which we call target dependency. Therefore, we propose MT-Tab, a multi-target Tabular foundation model framework that reformulates pretrained TFMs to tackle multi-target prediction problems. MT-Tab unfolds multi-target labels along the meta/batch dimension via target unfolding, passes them through the pretrained target encoder, and integrates them into the TFM's transformer input through prompt expansion. To preserve pretrained priors, we keep the TFM parameters frozen and employ Low-Rank Adaptation (LoRA) for efficient task adaptation. On the newly constructed MT-TabBench with 10 tasks, MT-Tab achieves the highest Elo rating among 17 methods including 16 baselines. Specifically, MT-Tab yields up to a 7.1% performance gain over the strongest single-target TFM under target imbalance by capturing target dependencies, while achieving up to 11.2 faster inference in many targets.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Tabular Foundation model, Multi-target prediction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mVsAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes MT-Tab, which adapts pretrained single-target tabular models to multi-target tasks. They use target unfolding and extra prompts to share info between targets, and train with LoRA. The authors claim good performance and big speedups.

Critical strengths

Simple and practical idea. No need to pretrain a multi-target model from scratch, just adapt an existing one with LoRA.

Clear memory and speed wins during inference, especially when there are many targets to predict.

Tested on a good amount of tabular datasets.

Critical weaknesses

Not convinced the model actually learns target dependencies. The correlation plot does not prove causation. Also, the permutation test messes up both the target order and the input-to-target mapping at the same time, so it proves very little. It is totally possible the gain just comes from shared regularized features in LoRA, not target interaction.

The 11.2x speedup claim is exaggerated. That number comes from comparing with running each target one by one in serial. But in Table A.8, when compared to a normal batched single-target setup, the speedup drops to around 1.5x for 32 targets. The peak memory saving is real, but the speed claim is oversold.

Missing obvious baselines. Why not compare with a simple multi-head setup where a frozen backbone just outputs multiple heads with LoRA? Need to know if target unfolding is actually better than that.

Questions for authors

Can you run an experiment where cross-target attention is masked out, so targets cannot see each other? Keep everything else the same. This would show if target interactions really help.

What is the real wall-clock speedup against a properly batched single-target baseline on the main benchmarks, not just serial inference?

How does this compare to just adding multiple output heads to the pretrained backbone with LoRA?

7 comments
AuthorsOpenReview ↗

Author Response [1/4]

We thank the reviewer for the constructive feedback on our paper.

[Weakness 1] Demonstrating target dependency

[W1] Not convinced the model actually learns target dependencies. The correlation plot does not prove causation. Also, the permutation test messes up both the target order and the input-to-target mapping at the same time, so it proves very little.

Target attention map

First, Appendix Figure A.1 provides evidence that the trained MT-Tab exhibits different patterns of information use depending on target correlation. A comparison of Student and OkCupid, both of which have 3 targets, shows that attention to other targets in the highly correlated Student dataset exceeds the uniform baseline and is comparable to attention to the target itself. In contrast, in the weakly correlated OkCupid dataset, attention to other targets is below the uniform baseline, and self-target attention is highest in each row. This observation is consistent with the tendency for performance gains to increase with target correlation. However, visualization alone cannot isolate the contribution of target dependency to predictive performance. We therefore conducted masking experiments that restrict cross-target information pathways while retaining shared LoRA weights, complementing the existing correlation and permutation analyses.

Target attention masking

Information can flow between targets through two pathways. In addition to the direct pathway in which target B attends to target A, there is an indirect pathway in which the shared features of the data XX (fxf_x) attend to target A, and target B subsequently attends to the updated features. Thus, blocking only target-to-target attention leaves indirect information sharing across layers intact. We designed the following two controls to address this issue.

  • strict-mask: This control blocks both direct target-to-target connections and connections through which features attend to targets, restricting both direct and indirect information flow. It retains shared features and a single forward pass but also removes the pathway through which features attend to their own target. Let TT denote the number of rows, fxf_x feature tokens, and mm the number of targets. Its computational cost is O(T(fx+m)2)\mathcal{O}(T(f_x+m)^2), the same as MT-Tab.
  • X-copy-mask: To preserve the bidirectional interaction between features and their own target that is removed by strict-mask, we copy the features for each target. Each pathway processes only its own feature copy and target, without sharing feature states across targets. LoRA weights are shared across all pathways. Its computational cost is O(mT(fx+1)2)\mathcal{O}(mT(f_x+1)^2), the same as TabPFN-LoRA.

The increased computation in X-copy-mask arises not from the mask itself, but from processing the same input features as mm independent states to prevent information from different targets from mixing. Batching these pathways does not eliminate the repeated computation.

Query → KeyTabPFN-LoRA (single-target)MT-Tabstrict-maskX-copy-mask
Feature → Feature✓✓✓✓
Feature → Target✓ (1 target)✓✗✓ (own target only)
Target → Feature✓✓✓✓
Target → Own target✓✓✓✓
Target → Other target— (1 target)✓✗✗
  • ✓: available attention path; ✗: unavailable attention path; —: not applicable.

∣ρS∣‾\overline{\lvert\rho_S\rvert} denotes the mean absolute Spearman correlation across target pairs. Values in parentheses indicate relative gains over single-target LoRA. Positive values indicate improvement.

  • RMSE: (sst−sv)/∣sst∣(s_{\mathrm{st}}-s_{\mathrm{v}})/\lvert s_{\mathrm{st}}\rvert
  • AUC: (sv−sst)/∣sst∣(s_{\mathrm{v}}-s_{\mathrm{st}})/\lvert s_{\mathrm{st}}\rvert
dataset∣ρS∣‾\overline{\lvert\rho_S\rvert} (group)TabPFN-LoRA (single-target)MT-Tabstrict-maskX-copy-mask
Energy RMSE ↓\downarrow0.974 (high)0.721 ± 0.380.305 ± 0.02 (+0.577)1.102 ± 0.17 (-0.528)0.670 ± 0.05 (+0.071)
Student RMSE ↓\downarrow0.897 (high)2.732 ± 0.072.613 ± 0.01 (+0.044)2.688 ± 0.07 (+0.016)2.649 ± 0.01 (+0.030)
Insurance AUC ↑\uparrow0.070 (low)71.94 ± 0.0673.08 ± 0.08 (+0.016)71.92 ± 0.27 (-0.000)72.81 ± 0.10 (+0.012)
OkCupid AUC ↑\uparrow0.031 (low)71.96 ± 3.0574.21 ± 0.15 (+0.031)74.20 ± 0.41 (+0.031)72.69 ± 1.70 (+0.010)

On Energy, the relative gain is +0.577 for MT-Tab and +0.071 for X-copy-mask. MT-Tab also achieves a lower RMSE than both controls on Student. Across all four datasets, MT-Tab achieves larger relative gains over single-target TabPFN-LoRA than X-copy-mask. These results support the view that allowing attention to other targets can improve predictive performance by exploiting inter-target dependencies.

AuthorsOpenReview ↗

Author Response [2/4]

[Weakness 2] Inference cost on real-world benchmarks

[W2] The 11.2x speedup claim is exaggerated. That number comes from comparing with running each target one by one in serial. But in Table A.8, when compared to a normal batched single-target setup, the speedup drops to around 1.5x for 32 targets. The peak memory saving is real, but the speed claim is oversold.

Comparison baselines and measurement scope

The 11.2×11.2\times speedup reported in the paper is relative to Independent sequential execution in a controlled setting with 32 targets. We will state this controlled setting explicitly and distinguish it from the target counts in real datasets and comparisons against the batched baseline. On the benchmark datasets, we compared the latency and memory consumption of Independent sequential, Independent batched, and MT-Tab. Here, nfxn_{f_x}, mm, and nsupportn_{\mathrm{support}} denote the numbers of input features, targets, and support examples, respectively.

Datasetnfxn_{f_x}mmnsupportn_{\mathrm{support}}MethodLatency mean ± std (ms)Peak Memory (MiB)
Energy82614Independent sequential98.82 ± 5.12103.98
Energy82614Independent batched63.79 ± 0.84133.12
Energy82614MT-Tab65.10 ± 0.66132.60
Student303519Independent sequential170.34 ± 1.25141.28
Student303519Independent batched77.40 ± 1.63287.55
Student303519MT-Tab77.28 ± 3.24161.19
Insurance7410000Independent sequential286.05 ± 0.63272.26
Insurance7410000Independent batched243.78 ± 2.221088.36
Insurance7410000MT-Tab103.13 ± 0.22426.04
OkCupid11310000Independent sequential238.04 ± 0.73319.58
OkCupid11310000Independent batched219.69 ± 0.761022.75
OkCupid11310000MT-Tab103.56 ± 0.22422.11

Compared with the batched baseline, MT-Tab is approximately 2.36×2.36\times and 2.12×2.12\times faster on Insurance and OkCupid, with peak memory reductions of 60.9%60.9\% and 58.7%58.7\%, respectively. On Student, latency is nearly identical (77.28 vs. 77.40 ms), while MT-Tab reduces peak memory by 43.9%43.9\%. For larger nsupportn_{\mathrm{support}} (Insurance and OkCupid), Independent batched repeats target-wise processing of support features, resulting in substantially higher forward cost and latency, whereas MT-Tab processes the shared support features once in a joint forward pass, avoiding this duplication and keeping the cost comparatively low. On Energy, where nfxn_{f_x}, mm, and nsupportn_{\mathrm{support}} are all small, MT-Tab has comparable peak memory and only slightly higher latency than the batched baseline (65.10 vs. 63.79 ms). This small absolute difference of 1.31 ms is unlikely to be a practical concern in GPU-based industrial serving environments. We will clarify that batched inference can accelerate single-target inference while emphasizing that the magnitude of MT-Tab's benefits depends on the workload. We will also add a visualization to reduce ambiguity and improve clarity.

AuthorsOpenReview ↗

Author Response [3/4]

[Weakness 3] Multi-target head ablation

[W3] Why not compare with a simple multi-head setup where a frozen backbone just outputs multiple heads with LoRA? Need to know if target unfolding is actually better than that.

Prompt expansion and output heads

To assess the effect of Target Unfold & Prompt Expansion, we need an alternative that accepts the observed labels of multi-targets and predicts all targets without expanding target-specific tokens. Simply removing target tokens changes the input information. Moreover, processing every target separately requires mm forward passes, increasing computational cost. We therefore designed an ablation that, similarly to the TabICL family, aggregates multi-target encodings into a single representation and predicts multi-targets using target-specific output heads. Target embeddings ek=ϕy(yk)e_k=\phi_y(y_k) are aggregated using a softmax-weighted sum, eˉ=∑kαkek\bar e=\sum_k\alpha_k e_k, where αk=softmax⁡k(q⊤ek)\alpha_k=\operatorname{softmax}_k(q^\top e_k). Concatenating the features with this pooled target token yields fx+1f_x+1 Transformer tokens, and target-specific output heads are applied to the shared output representation. All other components are kept identical to MT-Tab, except for target encoding and the multi-target head.

Results

Values in parentheses indicate relative gains over single-target LoRA in the same row.

Datasetmm∣ρS∣‾\overline{\lvert\rho_S\rvert} (Group)TabPFN-LoRA (single-target)MT-TabMT-Tab (w/o Prompt exp.)
Energy RMSE20.974 (high)0.721 ± 0.380.305 ± 0.02 (+0.577)0.599 ± 0.01 (+0.169)
Student RMSE30.897 (high)2.732 ± 0.072.613 ± 0.01 (+0.044)2.656 ± 0.04 (+0.028)
Insurance AUC %40.070 (low)71.94 ± 0.0673.08 ± 0.08 (+0.016)72.44 ± 0.11 (+0.007)
OkCupid AUC %30.031 (low)71.96 ± 3.0574.21 ± 0.15 (+0.031)73.70 ± 0.48 (+0.024)

In this table, MT-Tab (w/o Prompt exp.) outperforms single-target LoRA in mean performance, while MT-Tab achieves better mean performance on all four datasets. In particular, Energy RMSE decreases from 0.599 to 0.305. These results support the performance benefit of prompt expansion, which preserves distinct target tokens and enables attention between targets. Alongside these results, we will clarify how MT-Tab's preservation of target-specific representations differs from simply adding output heads.

Reviewer_YiLXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes MT-Tab, which turns a pretrained single-target tabular foundation model (TabPFN-2.5) into a multi-target predictor. Each target is encoded with the frozen target encoder and the m target tokens are concatenated with the feature tokens along the feature axis.

The authors train LoRA adapters, and all targets are predicted in one forward pass. The authors build MT-TabBench (9 datasets / 10 metrics) and report the highest Elo among 17 methods.

Critical strengths

Importance of research The problem matters in practice. Multi-target tabular prediction is common, and current TFMs don't support it natively.

Simplicity and Expansiveness The method is simple, and can work with various kinds of LoRA variants.

Strong inference efficiency gain As shown in the results section, one-shot multi-target is more efficient that multiple inferences of single-target.

Critical weaknesses

Key question on performance The gains are narrow and concentrated on a few dayasets. In Tab 9, the performance of MT-Tab is only leading 3/9 datasets. One of the main claim of the paper is there are strong Inter-Target Dependencies, which seems did not help much with the downstream performance.

Questions for authors

Pls see the weakness section. The main claim needs to hold for this paper, or alternatively, can the authors demonstrate what the MT-Tab has learnt Inter-Target Dependencies?

5 comments
AuthorsOpenReview ↗

Author Response [1/3]

We thank the reviewer for the careful review and constructive comments.

[Weakness 1] Downstream performance gain and target dependency

[W1] The gains are narrow and concentrated on a few dayasets. In Tab 9, the performance of MT-Tab is only leading 3/9 datasets. One of the main claim of the paper is there are strong Inter-Target Dependencies, which seems did not help much with the downstream performance.

Benchmark design and performance gain of MT-Tab

MT-TabBench was designed to cover diverse domains and a broad range of target correlations, from low to high correlation. Selecting only highly correlated tasks could bias the evaluation toward conditions favorable to the method. In contrast, including a wide range of correlations reveals both settings with substantial gains and those with limited gains, which we believe strengthens the robustness of the benchmark design.

Using the mean absolute Spearman correlation ∣ρS∣‾\overline{\lvert\rho_S\rvert} reported in Table A.4, the following table summarizes relative gains over single-target TabPFN-LoRA from Table 2.

Correlation groupDatasets∣ρS∣‾\overline{\lvert\rho_S\rvert} range, higher → lowerMean relative gain
highEnergy, Student, SARCOS0.974 → 0.349+0.196
midACS, Customer, Higgs0.274 → 0.198+0.100
lowInsurance, Online, OkCupid0.070 → 0.031+0.019

The mean relative gains are +0.196 for the high-correlation group, +0.100 for the mid-correlation group, and +0.019 for the low-correlation group, with larger gains in the more highly correlated groups.

Performance across the full benchmark

The correlation-based analysis identifies conditions under which MT-Tab offers larger gains over the same-backbone baseline. At the same time, MT-Tab achieves competitive overall performance across the full benchmark, including low-correlation datasets with smaller improvements. In Table A.14, MT-Tab ranks first not only in Elo but also in Competitiveness, Average rank, W/T/L, and Normalized regret.

MethodElo ↑\uparrowCompetitiveness (%) ↑\uparrowAverage rank ↓\downarrowW/T/LNormalized regret ↓\downarrow
MT-Tab1687.094.794.30127/0/330.090
TabFM (Prompt)1670.987.574.55124/1/350.151
TabFM (LoRA)1631.486.405.20117/2/410.188
TabICLv2 (LoRA)1584.884.246.00110/0/500.223
TabICLv2 (Prompt)1536.781.616.90101/0/590.232

Across benchmark construction types, MT-Tab also maintains strong rankings: tied at rank 2 on Native, rank 3 on Cache rebuild, and rank 2 on x2y conversion (Table A.11). This does not mean that it ranks 1 on every construction type, but it demonstrates competitiveness beyond any single construction strategy.

Practical use scenarios

We consider the following two scenarios in practical use.

  • When target correlation is high, MT-Tab can offer substantial performance gains, making its additional adaptation cost relative to Prompt a potentially worthwhile trade-off. If the adapted model is used repeatedly, this cost can be amortized over many inference calls.
  • Even when target correlation is low and performance gains are modest, MT-Tab can still offer efficient inference by jointly predicting all targets in a single forward pass. MT-Tab therefore remains worth considering for its efficient inference relative to single-target execution, while generally offering comparable or improved performance relative to existing TFM-Prompt or LoRA approaches. However, latency and memory benefits relative to batched single-target inference depend on the dataset.

For practical deployment of MT-Tab, it is helpful to consider its performance gains, adaptation cost, the expected number of inference calls, and memory constraints.

AuthorsOpenReview ↗

Author Response [2/3]

[Question 1] Demonstrating Inter-Target Dependency

[Q1] Can the authors demonstrate what the MT-Tab has learnt Inter-Target Dependencies?

Target attention map

First, Appendix Figure A.1 provides evidence that the trained MT-Tab exhibits different patterns of information use depending on target correlation. A comparison of Student and OkCupid, both of which have 3 targets, shows that attention to other targets in the highly correlated Student dataset exceeds the uniform baseline and is comparable to attention to the target itself. In contrast, in the weakly correlated OkCupid dataset, attention to other targets is below the uniform baseline, and self-target attention is highest in each row (diagonal). This observation is consistent with the tendency for performance gains to increase with target correlation. However, visualization alone cannot isolate the contribution of target dependency to predictive performance. We therefore conducted masking experiments that restrict cross-target information pathways while retaining shared LoRA weights, complementing the existing correlation and permutation analyses.

Target attention masking

Information can flow between targets through two pathways. In addition to the direct pathway in which target B attends to target A, there is an indirect pathway in which the shared features of the data XX (fxf_x) attend to target A, and target B subsequently attends to the updated features. Thus, blocking only target-to-target attention leaves indirect information sharing across layers intact. We designed the following two controls to address this issue.

  • strict-mask: This control blocks both direct target-to-target connections and connections through which features attend to targets, restricting both direct and indirect information flow. It retains shared features and a single forward pass but also removes the pathway through which features attend to their own target. Let NN denote the number of rows, fxf_x feature tokens, and mm the number of targets. Its computational cost is O(N(fx+m)2)\mathcal{O}(N(f_x+m)^2), the same as MT-Tab.
  • X-copy-mask: To preserve the bidirectional interaction between features and their own target that is removed by strict-mask, we copy the features for each target. Each pathway processes only its own feature copy and target, without sharing feature states across targets. LoRA weights are shared across all pathways. Its computational cost is O(mN(fx+1)2)\mathcal{O}(mN(f_x+1)^2), the same as TabPFN-LoRA.

In short, strict-mask restricts information pathways while retaining shared computation, whereas X-copy-mask is a complementary control intended to exclude cross-target sharing while preserving the use of each target's own information. The increased computation in X-copy-mask arises not from the mask itself, but from processing the same input features as mm independent states to prevent information from different targets from mixing. Batching these pathways does not eliminate the repeated computation.

Query → KeyTabPFN-LoRA (single-target)MT-Tabstrict-maskX-copy-mask
Feature → Feature✓✓✓✓
Feature → Target✓ (1 target)✓✗✓ (own target only)
Target → Feature✓✓✓✓
Target → Own target✓✓✓✓
Target → Other target— (1 target)✓✗✗
  • ✓: available attention path; ✗: unavailable attention path; —: not applicable.

∣ρS∣‾\overline{\lvert\rho_S\rvert} denotes the mean absolute Spearman correlation across target pairs. Values in parentheses indicate relative gains over single-target LoRA. Positive values indicate improvement.

  • RMSE: (sst−sv)/∣sst∣(s_{\mathrm{st}}-s_{\mathrm{v}})/\lvert s_{\mathrm{st}}\rvert
  • AUC: (sv−sst)/∣sst∣(s_{\mathrm{v}}-s_{\mathrm{st}})/\lvert s_{\mathrm{st}}\rvert
dataset∣ρS∣‾\overline{\lvert\rho_S\rvert} (group)TabPFN-LoRA (single-target)MT-Tabstrict-maskX-copy-mask
Energy RMSE ↓\downarrow0.974 (high)0.721 ± 0.380.305 ± 0.02 (+0.577)1.102 ± 0.17 (-0.528)0.670 ± 0.05 (+0.071)
Student RMSE ↓\downarrow0.897 (high)2.732 ± 0.072.613 ± 0.01 (+0.044)2.688 ± 0.07 (+0.016)2.649 ± 0.01 (+0.030)
Insurance AUC ↑\uparrow0.070 (low)71.94 ± 0.0673.08 ± 0.08 (+0.016)71.92 ± 0.27 (-0.000)72.81 ± 0.10 (+0.012)
OkCupid AUC ↑\uparrow0.031 (low)71.96 ± 3.0574.21 ± 0.15 (+0.031)74.20 ± 0.41 (+0.031)72.69 ± 1.70 (+0.010)

On Energy, the relative gain is +0.577 for MT-Tab and +0.071 for X-copy-mask. MT-Tab also achieves a lower RMSE than both controls on Student. Across all four datasets, MT-Tab achieves larger relative gains over single-target TabPFN-LoRA than X-copy-mask. These results support the view that allowing attention to other targets can improve predictive performance by exploiting inter-target dependencies.

AuthorsOpenReview ↗

Author Response [3/3]

Questions

[Q1] The main claim needs to hold for this paper, or alternatively, can the authors demonstrate what the MT-Tab has learnt Inter-Target Dependencies?

  • As summarized in [Weakness 1], gains are larger in higher-correlation groups, while MT-Tab remains competitive overall. Figure A.1 shows stronger cross-target attention on Student than on weakly correlated OkCupid. Beyond visualization, we added strict-mask and X-copy-mask controlled ablations that block direct and indirect cross-target information flow ([Question 1]). MT-Tab achieves lower mean RMSE than both controls on Energy and Student. Together, these results provide evidence that MT-Tab learns to exploit inter-target dependency, with benefits that depend on the dataset.

We will incorporate the clarifications and additional results discussed above into the revised manuscript as soon as possible.

Reviewer_qowZOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MT-Tab, a framework that extends pretrained single-target tabular foundation models to multi-target prediction. It employs target unfolding and prompt expansion to enable interactions among target representations, together with dataset-specific lora fine-tuning. Experiments on MT-TabBench demonstrate competitive predictive performance and improved inference efficiency.

Critical strengths

MT-Tab provides an effective way to extend existing single-target TFMs to multi-target prediction without additional pretraining. The proposed architecture is simple and potentially useful for applications requiring simultaneous predictions of multiple outputs.

The paper compares against various TFM and conventional baselines, and includes analyses of target correlations, target imbalance, few-shot performance, and inference efficiency.

Critical weaknesses

The central claim is that MT-Tab improves performance by leveraging target dependencies. However, the current experiments do not sufficiently isolate this contribution. Although the authors provide correlation analysis and target permutation experiments, the latter also disrupts the relationship between inputs and targets. A controlled ablation that disables cross-target interactions while preserving the same architecture, lora configuration, and training budget would provide stronger evidence.

MT-Tab requires lora fine-tuning on each downstream dataset, and Table 4 shows that prompt expansion without adaptation underperforms the single-target baseline. Therefore, its multi-target capability largely depends on task-specific optimization rather than directly transferring from the pretrained model. Furthermore, while the reported inference speedups are substantial compared with sequential single-target inference, the advantage over batched inference is smaller (approximately 1.48× at 32 targets in Appendix Table A.8). These limitations somewhat weaken the claims regarding generality and scalability.

Questions for authors

Can the authors provide a controlled ablation that removes cross-target information sharing while keeping the other components unchanged? This would help establish whether the improvements genuinely arise from modeling target dependencies.

Can the learned LoRA adapters generalize across different multi-target datasets, or must they always be retrained? How does the additional adaptation cost affect the overall computational efficiency compared with independent single-target models?

5 comments
AuthorsOpenReview ↗

Author Response [1/5]

We thank the reviewer for carefully reviewing our paper.

[Weakness 1] Contribution of inter-target dependency

[W1] the current experiments do not sufficiently isolate this contribution. Although the authors provide correlation analysis and target permutation experiments, the latter also disrupts the relationship between inputs and targets.

Target attention map

First, Appendix Figure A.1 provides evidence that MT-Tab exhibits different patterns of information use depending on target correlation. A comparison of Student and OkCupid, both of which have 3 targets, shows that attention to other targets in the highly correlated Student dataset exceeds the uniform baseline and is comparable to attention to the target itself. In contrast, in the weakly correlated OkCupid dataset, attention to other targets is below the uniform baseline, and self-target attention is highest in each diagonal row. This observation is consistent with the tendency for performance gains to increase with target correlation. However, visualization alone cannot isolate the contribution of target dependency to predictive performance. We therefore conducted masking experiments that restrict cross-target information pathways while keeping the other components unchanged, complementing the existing correlation and permutation analyses.

Target attention masking (controlled ablation)

Information can flow between targets through two pathways. In addition to the direct pathway in which target B attends to target A, there is an indirect pathway in which the shared features of the data XX (fxf_x) attend to target A, and target B subsequently attends to the updated features. Thus, blocking only target-to-target attention leaves indirect information sharing across layers intact. We designed the following two controls to address this issue.

  • strict-mask: This control blocks both direct target-to-target connections and connections through which features attend to targets, restricting both direct and indirect information flow. It retains shared features and a single forward pass but also removes the pathway through which features attend to their own target. Let TT denote the number of rows, fxf_x feature tokens, and mm the number of targets. Its computational cost is O(T(fx+m)2)\mathcal{O}(T(f_x+m)^2), the same as MT-Tab.
  • X-copy-mask: To preserve the bidirectional interaction between features and their own target that is removed by strict-mask, we copy the features for each target. Each pathway processes only its own feature copy and target, without sharing feature states across targets. LoRA weights are shared across all pathways. Its computational cost is O(mT(fx+1)2)\mathcal{O}(mT(f_x+1)^2), the same as TabPFN-LoRA.

The increased computation in X-copy-mask arises not from the mask itself, but from processing the same input features as mm independent states to prevent information from different targets from mixing. Batching these pathways does not eliminate the repeated computation.

Query → KeyTabPFN-LoRA (single-target)MT-Tabstrict-maskX-copy-mask
Feature → Feature✓✓✓✓
Feature → Target✓ (1 target)✓✗✓ (own target only)
Target → Feature✓✓✓✓
Target → Own target✓✓✓✓
Target → Other target— (1 target)✓✗✗
  • ✓: available attention path; ✗: unavailable attention path; —: not applicable.

∣ρS∣‾\overline{\lvert\rho_S\rvert} denotes the mean absolute Spearman correlation across target pairs. Values in parentheses indicate relative gains over single-target LoRA. Positive values indicate improvement.

  • RMSE: (sst−sv)/∣sst∣(s_{\mathrm{st}}-s_{\mathrm{v}})/\lvert s_{\mathrm{st}}\rvert
  • AUC: (sv−sst)/∣sst∣(s_{\mathrm{v}}-s_{\mathrm{st}})/\lvert s_{\mathrm{st}}\rvert
dataset∣ρS∣‾\overline{\lvert\rho_S\rvert} (group)TabPFN-LoRA (single-target)MT-Tabstrict-maskX-copy-mask
Energy RMSE ↓\downarrow0.974 (high)0.721 ± 0.380.305 ± 0.02 (+0.577)1.102 ± 0.17 (-0.528)0.670 ± 0.05 (+0.071)
Student RMSE ↓\downarrow0.897 (high)2.732 ± 0.072.613 ± 0.01 (+0.044)2.688 ± 0.07 (+0.016)2.649 ± 0.01 (+0.030)
Insurance AUC ↑\uparrow0.070 (low)71.94 ± 0.0673.08 ± 0.08 (+0.016)71.92 ± 0.27 (-0.000)72.81 ± 0.10 (+0.012)
OkCupid AUC ↑\uparrow0.031 (low)71.96 ± 3.0574.21 ± 0.15 (+0.031)74.20 ± 0.41 (+0.031)72.69 ± 1.70 (+0.010)

Across all four datasets, MT-Tab achieves larger relative gains over single-target TabPFN-LoRA than X-copy-mask. These results support the view that allowing attention to other targets can improve predictive performance by exploiting inter-target dependencies. The particularly large gain (+0.577) on highly correlated Energy further suggests that these benefits could be greater when target correlations are strong.

AuthorsOpenReview ↗

Author Response [2/5]

[Weakness 2-1] Generalization of learned adapters

[W2-1] Table 4 shows that prompt expansion without adaptation underperforms the single-target baseline. Therefore, its multi-target capability largely depends on task-specific optimization rather than directly transferring from the pretrained model.

Role of LoRA adaptation

MT-Tab reuses an existing single-target model without additional multi-target pretraining, while performing downstream LoRA adaptation. Table 4 of the main paper includes V1, which applies only Target Unfold & Prompt Expansion.

MethodTarget Unfold & Prompt ExpansionLoRAEnergy RMSE ↓\downarrowInsurance AUC (%) ↑\uparrow
TabPFN-Prompt (single-target)✗✗0.835 ± 0.1272.759 ± 0.12
TabPFN-LoRA (single-target)✗✓0.721 ± 0.3871.937 ± 0.06
MT-Tab V1✓✗1.074 ± 0.0070.858 ± 0.00
MT-Tab✓✓0.305 ± 0.0273.080 ± 0.08

MT-Tab with LoRA outperforms V1. The performance drop without LoRA may reflect not only a lack of downstream task adaptation but also a mismatch between the expanded prompt format and the single-target format used during pretraining. The single-target TFM receives a Transformer input of shape T×B×(fx+1)×ET \times B \times (f_x+1) \times E, whereas MT-Tab expands it to T×B×(fx+m)×ET \times B \times (f_x+m) \times E. Here, BB is the table batch size, and EE is the embedding dimension. LoRA provides a mechanism for adapting to this format change as well as to the downstream task.

Transfer between different datasets (Cross-domain transfer)

We evaluated models adapted on a source dataset on a target dataset with the same number of targets, without additional weight updates. Dataset characteristics and correlation statistics are taken from Table A.9 and Table A.4, respectively.

DatasetTaskTrain rowsTest rowsnfxn_{fx}nmn_m∣ρS∣‾\overline{\lvert\rho_S\rvert}
CustomerClassification103,90425,9761840.266
InsuranceClassification18,8384,710740.070
OnlineClassification9,8642,4661530.052
OkCupidClassification21,3415,3361130.031
target (source)nmn_mTabPFN-PromptTabPFN-LoRAMT-TabMT-Tab transfer
Customer (Insurance) · AUC % ↑494.12 ± 0.0494.03 ± 0.0295.07 ± 0.0590.67 ± 0.07
Insurance (Customer) · AUC % ↑472.76 ± 0.1271.94 ± 0.0673.08 ± 0.0871.02 ± 0.15
Online (OkCupid) · AUC % ↑389.12 ± 0.3388.11 ± 1.2988.95 ± 0.4386.22 ± 0.66
OkCupid (Online) · AUC % ↑372.22 ± 0.8371.96 ± 3.0574.21 ± 0.1571.17 ± 2.69

MT-Tab transfer has a lower mean AUC than Prompt in all four directions. Even with the same number of targets, changes in domain and target semantics make generalization challenging, requiring dataset-specific adaptation. Real-TabPFN [1] also shows that performance on unseen target datasets can decline depending on the source datasets used to train the TFM. In addition, format change in MT-Tab (the expanded prompt format) may further limit cross-dataset adapter reuse. Future work will explore joint adaptation across diverse datasets, input-format alignment, and pretrained knowledge preservation to improve adapter transfer.

Transfer between targets within the same dataset (In-domain transfer)

To examine a setting with reduced domain shift, we conducted experiments separately on Customer and Insurance. Within each dataset, we partitioned the targets into two disjoint groups of 2 targets each. We froze the weights learned on one group and transferred them to the other.

DatasetGroupTarget columnsnfxn_{fx}nmn_m∣ρS∣‾\overline{\lvert\rho_S\rvert} (all targets)
CustomerAy_satisfaction, y_Class1820.266
CustomerBy_CustomerType, y_TypeofTravel1820.266
InsuranceAy_Response, y_Accomodation_Type720.070
InsuranceBy_Holding_Policy_Type, y_Reco_Insurance_Type720.070
target (source)TabPFN-2.5 PromptMT-Tab transfer
Customer B (Customer A) · AUC % ↑88.97 ± 0.0492.32 ± 0.16
Customer A (Customer B) · AUC % ↑90.70 ± 0.0891.22 ± 0.21
Insurance B (Insurance A) · AUC % ↑82.07 ± 0.0081.96 ± 0.11
Insurance A (Insurance B) · AUC % ↑61.25 ± 0.0061.33 ± 0.07

Performance improves in both directions on Customer, whereas the differences on Insurance are small. Consistent with our earlier findings, these results suggest that transfer between target groups within the same dataset may yield larger gains when target dependencies are stronger.

References

[1] GARG, Anurag, et al. Real-tabpfn: Improving tabular foundation models via continued pre-training with real-world data. arXiv preprint arXiv:2507.03971, 2025.

AuthorsOpenReview ↗

Author Response [3/5]

[Weakness 2-2] Total computational cost including adaptation

[W2-2] MT-Tab requires lora fine-tuning on each downstream dataset.

Full training data

Prompt requires no additional weight training, whereas MT-Tab incurs a downstream adaptation cost. We measured training and inference costs to make this distinction explicit.

ModelDatasetmmTrain (s)Train Peak MBInfer (=1k) sInfer Peak MBscore
TabPFN-PromptEnergy20.00—2.68644.5RMSE 0.835±0.12
TabPFN-LoRAEnergy2452.36188.30.23354.3RMSE 0.721±0.38
MT-TabEnergy292.32134.20.12172.1RMSE 0.305±0.02
TabPFN-PromptInsurance40.00—6.50426.2AUC% 72.76±0.12
TabPFN-LoRAInsurance41690.03560.11.81425.9AUC% 71.94±0.06
MT-TabInsurance4318.28835.61.51712.9AUC% 73.08±0.08

MT-Tab requires 92.32 seconds of training on Energy and 318.28 seconds on Insurance, longer than Prompt but shorter than the 452.36 seconds and 1690.03 seconds required by independent TabPFN-LoRA. Its inference time is also shorter than that of both single-target methods in this comparison.

Combined cost in the 5-shot setting

We also measured costs in the 5-shot setting to examine the trade-off when less training data is available.

ModelDatasetmmTrain (s)Train Peak MBInfer (=1k) sInfer Peak MBTotal wall (s)Peak MB (max)score
TabPFN-PromptACS (5-shot)20.00—2.7799.62.7799.6AUC% 64.22±0.27
TabPFN-LoRAACS (5-shot)27.68654.10.28190.87.96654.1AUC% 63.33±0.50
MT-TabACS (5-shot)24.50275.00.11143.04.61275.0AUC% 69.96±0.30
TabPFN-PromptHiggs (R) (5-shot)70.00—4.40640.84.40640.8RMSE 0.393±0.00
TabPFN-LoRAHiggs (R) (5-shot)723.08912.20.79412.423.87912.2RMSE 0.383±0.00
MT-TabHiggs (R) (5-shot)74.24352.60.14448.14.38448.1RMSE 0.376±0.00

The total wall time of TabPFN-Prompt is 2.77s. on ACS and 4.40s. on Higgs, compared with 4.61s. and 4.38s. for MT-Tab, respectively. In the 5-shot setting, the smaller amount of data required for adaptation can narrow the cost gap relative to Prompt, while MT-Tab achieves comparable or improved performance relative to Prompt and LoRA in this setting.

multi-target pretraining and downstream adaptation costs

Unlike Prompt, MT-Tab requires downstream adaptation, but it uses an existing single-target model rather than pretraining a new model. As summarized in Table A.2, the budgets for TabPFN-MT multi-target pretraining and MT-Tab LoRA adaptation are as follows.

MethodTraining stageHardwareElapsed timeGPU-hours
TabPFN-MTMulti-target pretraining8 × NVIDIA A5000∼4\sim4 h∼32\sim32
MT-TabDownstream LoRA adaptation1 × H100 96GB≤1\leq 1 h (max.)≤1\leq 1

MT-Tab can be applied to downstream tasks through relatively lightweight adaptation without separate multi-target pretraining.

Practical use scenarios

We consider the following two scenarios in practical use.

  • When target correlation is high, MT-Tab can offer substantial performance gains, making its additional adaptation cost relative to Prompt a potentially worthwhile trade-off. If the adapted model is used repeatedly, this cost can be amortized over many inference calls.
  • Even when target correlation is low and performance gains are modest, MT-Tab can remain attractive for repeated inference by jointly predicting all targets in a single forward pass. Independent single-target TabPFN instead requires mm target-specific forward evaluations per query to predict mm targets, repeatedly processing the same input features. These evaluations can be batched, so this does not imply an mm-fold latency reduction; latency and memory benefits relative to batched single-target inference depend on the dataset.

In industrial deployment scenarios where a model is adapted once and then serves thousands to hundreds of thousands of queries, MT-Tab's LoRA adaptation is a one-time cost, whereas the cost of separate target-wise evaluations recurs for every query. When MT-Tab has a lower per-query inference cost, the cumulative savings can outweigh its initial adaptation overhead, allowing the total adaptation-plus-inference cost to become substantially lower than that of TabPFN-Prompt at sufficiently large query volumes, even though Prompt requires no downstream weight updates. For practical deployment of MT-Tab, it is therefore helpful to consider its performance gains, the total adaptation-plus-inference cost over the expected number of inference calls, and memory constraints.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.