- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes MT-Tab, which adapts pretrained single-target tabular models to multi-target tasks. They use target unfolding and extra prompts to share info between targets, and train with LoRA. The authors claim good performance and big speedups.
Critical strengths
Simple and practical idea. No need to pretrain a multi-target model from scratch, just adapt an existing one with LoRA.
Clear memory and speed wins during inference, especially when there are many targets to predict.
Tested on a good amount of tabular datasets.
Critical weaknesses
Not convinced the model actually learns target dependencies. The correlation plot does not prove causation. Also, the permutation test messes up both the target order and the input-to-target mapping at the same time, so it proves very little. It is totally possible the gain just comes from shared regularized features in LoRA, not target interaction.
The 11.2x speedup claim is exaggerated. That number comes from comparing with running each target one by one in serial. But in Table A.8, when compared to a normal batched single-target setup, the speedup drops to around 1.5x for 32 targets. The peak memory saving is real, but the speed claim is oversold.
Missing obvious baselines. Why not compare with a simple multi-head setup where a frozen backbone just outputs multiple heads with LoRA? Need to know if target unfolding is actually better than that.
Questions for authors
Can you run an experiment where cross-target attention is masked out, so targets cannot see each other? Keep everything else the same. This would show if target interactions really help.
What is the real wall-clock speedup against a properly batched single-target baseline on the main benchmarks, not just serial inference?
How does this compare to just adding multiple output heads to the pretrained backbone with LoRA?
7 comments
Author Response [2/4]
[Weakness 2] Inference cost on real-world benchmarks
[W2] The 11.2x speedup claim is exaggerated. That number comes from comparing with running each target one by one in serial. But in Table A.8, when compared to a normal batched single-target setup, the speedup drops to around 1.5x for 32 targets. The peak memory saving is real, but the speed claim is oversold.
Comparison baselines and measurement scope
The speedup reported in the paper is relative to Independent sequential execution in a controlled setting with 32 targets. We will state this controlled setting explicitly and distinguish it from the target counts in real datasets and comparisons against the batched baseline. On the benchmark datasets, we compared the latency and memory consumption of Independent sequential, Independent batched, and MT-Tab. Here, , , and denote the numbers of input features, targets, and support examples, respectively.
| Dataset | Method | Latency mean ± std (ms) | Peak Memory (MiB) | |||
|---|---|---|---|---|---|---|
| Energy | 8 | 2 | 614 | Independent sequential | 98.82 ± 5.12 | 103.98 |
| Energy | 8 | 2 | 614 | Independent batched | 63.79 ± 0.84 | 133.12 |
| Energy | 8 | 2 | 614 | MT-Tab | 65.10 ± 0.66 | 132.60 |
| Student | 30 | 3 | 519 | Independent sequential | 170.34 ± 1.25 | 141.28 |
| Student | 30 | 3 | 519 | Independent batched | 77.40 ± 1.63 | 287.55 |
| Student | 30 | 3 | 519 | MT-Tab | 77.28 ± 3.24 | 161.19 |
| Insurance | 7 | 4 | 10000 | Independent sequential | 286.05 ± 0.63 | 272.26 |
| Insurance | 7 | 4 | 10000 | Independent batched | 243.78 ± 2.22 | 1088.36 |
| Insurance | 7 | 4 | 10000 | MT-Tab | 103.13 ± 0.22 | 426.04 |
| OkCupid | 11 | 3 | 10000 | Independent sequential | 238.04 ± 0.73 | 319.58 |
| OkCupid | 11 | 3 | 10000 | Independent batched | 219.69 ± 0.76 | 1022.75 |
| OkCupid | 11 | 3 | 10000 | MT-Tab | 103.56 ± 0.22 | 422.11 |
Compared with the batched baseline, MT-Tab is approximately and faster on Insurance and OkCupid, with peak memory reductions of and , respectively. On Student, latency is nearly identical (77.28 vs. 77.40 ms), while MT-Tab reduces peak memory by . For larger (Insurance and OkCupid), Independent batched repeats target-wise processing of support features, resulting in substantially higher forward cost and latency, whereas MT-Tab processes the shared support features once in a joint forward pass, avoiding this duplication and keeping the cost comparatively low. On Energy, where , , and are all small, MT-Tab has comparable peak memory and only slightly higher latency than the batched baseline (65.10 vs. 63.79 ms). This small absolute difference of 1.31 ms is unlikely to be a practical concern in GPU-based industrial serving environments. We will clarify that batched inference can accelerate single-target inference while emphasizing that the magnitude of MT-Tab's benefits depends on the workload. We will also add a visualization to reduce ambiguity and improve clarity.
Author Response [3/4]
[Weakness 3] Multi-target head ablation
[W3] Why not compare with a simple multi-head setup where a frozen backbone just outputs multiple heads with LoRA? Need to know if target unfolding is actually better than that.
Prompt expansion and output heads
To assess the effect of Target Unfold & Prompt Expansion, we need an alternative that accepts the observed labels of multi-targets and predicts all targets without expanding target-specific tokens. Simply removing target tokens changes the input information. Moreover, processing every target separately requires forward passes, increasing computational cost. We therefore designed an ablation that, similarly to the TabICL family, aggregates multi-target encodings into a single representation and predicts multi-targets using target-specific output heads. Target embeddings are aggregated using a softmax-weighted sum, , where . Concatenating the features with this pooled target token yields Transformer tokens, and target-specific output heads are applied to the shared output representation. All other components are kept identical to MT-Tab, except for target encoding and the multi-target head.
Results
Values in parentheses indicate relative gains over single-target LoRA in the same row.
| Dataset | (Group) | TabPFN-LoRA (single-target) | MT-Tab | MT-Tab (w/o Prompt exp.) | |
|---|---|---|---|---|---|
| Energy RMSE | 2 | 0.974 (high) | 0.721 ± 0.38 | 0.305 ± 0.02 (+0.577) | 0.599 ± 0.01 (+0.169) |
| Student RMSE | 3 | 0.897 (high) | 2.732 ± 0.07 | 2.613 ± 0.01 (+0.044) | 2.656 ± 0.04 (+0.028) |
| Insurance AUC % | 4 | 0.070 (low) | 71.94 ± 0.06 | 73.08 ± 0.08 (+0.016) | 72.44 ± 0.11 (+0.007) |
| OkCupid AUC % | 3 | 0.031 (low) | 71.96 ± 3.05 | 74.21 ± 0.15 (+0.031) | 73.70 ± 0.48 (+0.024) |
In this table, MT-Tab (w/o Prompt exp.) outperforms single-target LoRA in mean performance, while MT-Tab achieves better mean performance on all four datasets. In particular, Energy RMSE decreases from 0.599 to 0.305. These results support the performance benefit of prompt expansion, which preserves distinct target tokens and enables attention between targets. Alongside these results, we will clarify how MT-Tab's preservation of target-specific representations differs from simply adding output heads.
Author Response [1/4]
We thank the reviewer for the constructive feedback on our paper.
[Weakness 1] Demonstrating target dependency
Target attention map
First, Appendix Figure A.1 provides evidence that the trained MT-Tab exhibits different patterns of information use depending on target correlation. A comparison of Student and OkCupid, both of which have 3 targets, shows that attention to other targets in the highly correlated Student dataset exceeds the uniform baseline and is comparable to attention to the target itself. In contrast, in the weakly correlated OkCupid dataset, attention to other targets is below the uniform baseline, and self-target attention is highest in each row. This observation is consistent with the tendency for performance gains to increase with target correlation. However, visualization alone cannot isolate the contribution of target dependency to predictive performance. We therefore conducted masking experiments that restrict cross-target information pathways while retaining shared LoRA weights, complementing the existing correlation and permutation analyses.
Target attention masking
Information can flow between targets through two pathways. In addition to the direct pathway in which target B attends to target A, there is an indirect pathway in which the shared features of the data X (fx) attend to target A, and target B subsequently attends to the updated features. Thus, blocking only target-to-target attention leaves indirect information sharing across layers intact. We designed the following two controls to address this issue.
The increased computation in X-copy-mask arises not from the mask itself, but from processing the same input features as m independent states to prevent information from different targets from mixing. Batching these pathways does not eliminate the repeated computation.
∣ρS∣ denotes the mean absolute Spearman correlation across target pairs. Values in parentheses indicate relative gains over single-target LoRA. Positive values indicate improvement.
On Energy, the relative gain is +0.577 for MT-Tab and +0.071 for X-copy-mask. MT-Tab also achieves a lower RMSE than both controls on Student. Across all four datasets, MT-Tab achieves larger relative gains over single-target TabPFN-LoRA than X-copy-mask. These results support the view that allowing attention to other targets can improve predictive performance by exploiting inter-target dependencies.