acceptodds
Under review as a conference paper at ICLR 2027

Predict by Pointing: A Task-Agnostic Output Head for Tabular In-Context Learning

Abstract

Tabular foundation models learn in-context: given a labeled support set, they predict unlabeled queries in a single forward pass. Simple point losses extend naturally to vector-valued targets but are often less competitive than distributional regression objectives, whose heads are typically scalar-specific. Consequently, there is no common prediction head spanning support-adaptive classification, strong scalar regression, and native multi-output regression. In this work, we introduce Prediction Over In-context Targets (POINT), a task-agnostic output head that instead treats the labeled support set itself as the prediction space. POINT assigns probability directly to the support set; classification probabilities are obtained by summing mass over examples of the same class, while regression predictions are obtained by taking the expectation of continuous support labels. The output space therefore adapts to the context rather than being fixed by the task. To handle this new space, POINT is trained with POINT-CE, a support-distribution cross-entropy objective to handle both classification and regression simultaneously. We further show that, under support coverage, minimizing POINT-CE in expectation recovers conditional class probabilities for classification and the conditional mean for regression. This connects the same support-level objective to the quantities targeted by categorical likelihoods and distributional regression. Across tabular in-context-learning experiments, POINT achieves competitive performance on both classification and regression while retaining the same support-based prediction principle. Experiment results suggest that predicting by pointing to labels already present in context can provide a common prediction head for tabular models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tabular Foundation Model, In Context Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XtkcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents POINT, a shared output head for tabular in-context learning. It predicts a probability distribution over support examples. Classification adds up the weights of examples from the same class. Regression takes a weighted average of their labels. The same readout also works with vector labels.

The paper introduces POINT-CE to train this head. It constructs a target distribution over support examples whose weighted labels match the query target. The theory connects this objective to class probabilities, regression means, and distributional prediction under stated conditions.

The experiments cover 57 classification datasets, three regression development datasets, ten further regression datasets, and small multi-output and synthetic studies. The results support the basic feasibility of the shared interface. However, several comparisons need stronger controls, and some proposed extensions need clearer implementation details.

Critical strengths
  1. shared prediction interface is simple and meaningful. The same learned head handles classification and regression through support-example probabilities. POINT-CE gives these tasks a common training interface. This offers a useful way to share the prediction layer, beyond sharing only the encoder.

  2. The controlled experiments provide positive evidence. In Table 3, adding the full POINT-CE objective improves classification accuracy from 80.65% to 82.04% and reduces regression GM RMSE from 2.500 to 2.232. Table 1 also compares shared and separate metric heads under the same episode budgets. These results support the value of the approach. The theory further helps explain what the common loss can recover and where support coverage becomes important.

Critical weaknesses
  1. The evidence does not yet cleanly establish the source and stability of the broader gains. Table 2 uses query-local retrieval for POINT, but equivalent retrieval for the other neural heads is unclear. Appendix C.1 also states that these evaluations informed design and serving choice. The results therefore need independent confirmation. Small differences at the larger training budget are difficult to assess without repeated training runs and paired uncertainty estimates. The multi output comparison has a further control problem: it changes both the output structure and the training loss.

  2. Some parts of the proposed method are not fully connected to the practical implementation and evidence. The main theory studies POINT-CE, while the main regression model also uses Smooth L1. The paper correctly acknowledges that this can change the population optimum. The vector experiment does not clearly explain targets outside the support-label convex hull. The tree-teacher extension also needs a rule for zero reference weights. In addition, the contribution statement promises several component ablations that I could not find in the reported experiments.

Questions for authors
  1. Match retrieval conditions across output heads. Section 5.3 gives POINT query-local contexts with 512 retrieved support rows. Please state whether the quantile, histogram, and bar heads receive exactly the same rows. Compare the heads using identical preprocessing, support pools, retrieval results, and inference views. A POINT comparison with and without retrieval would also help separate retrieval gains from head gains.

  2. Add independent confirmation and report uncertainty. Please freeze the method and evaluate it on datasets that did not inform its design. Clearly identify which datasets were used for development. Report multiple pretraining seeds and paired differences across datasets, with confidence intervals for the main summaries. The split SD in Table 1 does not measure training randomness. At the larger budget, accuracy is 83.01% versus 82.88% for the quantile baseline, and both have regression GM RMSE of 1.841. These results support competitive performance, but do not establish a clear advantage.

  3. Make the multi-output comparison fair and reproducible. Vector POINT uses a uniform-reference POINT-CE teacher, while the three scalar models use pinball supervision. Please compare joint and independent scalar models under matched objectives, training budgets, and preprocessing. Give the Linnerud split protocol, repeat counts, and uncertainty. Several additional multi-output datasets would help test whether the result extends beyond this one example.

  4. Explain vector targets outside the support-label convex hull. A weighted average of support labels must remain inside their convex hull. Appendix B.2 acknowledges this limit, but the Linnerud implementation is unclear. Please report how often query targets fall outside this region and explain the projection or fallback rule. Give target scaling, solver tolerances, and reconstruction errors. If the teacher uses a projected target, state that it preserves the projected target rather than the original one. Table 7 only tests known targets generated inside the hull, so it does not resolve this issue.

  5. Connect the practical regression loss to the theoretical claims. Table 5 uses Smooth L1 plus 0.3 POINT-CE. Appendix A.10 correctly shows that this combination need not recover the conditional mean. Please bring this qualification into the main text. Add matched CE-only results on the broader regression suite and vary the auxiliary-loss weight. A squared-error auxiliary would also be a useful control. Reporting pinball loss from POINT’s induced predictive distribution would help test whether the theoretical distributional benefits appear on real datasets.

  6. Define the tree-teacher projection when reference weights contain zeros. Equation (3) needs more than membership in the full support-label convex hull. For finite KL, the target must lie in the hull of labels with positive reference weight. For example, support labels (0, 1), reference weights (1, 0), and target 0.5 give no feasible teacher with finite KL. Please explain how this is handled for Extra Trees. Possible fixes include adding positive reference mass, mixing with a positive Gaussian reference before projection, or using a stated fallback. Report how often this case occurs and check the final teacher’s mean error.

  7. Align the claimed ablations with the experiments shown. The contribution list says the experiments isolate retrieval geometry, robust point losses, and teacher smoothing. Table 3 mainly tests classification and regression POINT-CE switches. Table 8 tests the role of support labels in learning the representation. Please add matched comparisons for Euclidean versus dot-product scoring, the point-loss choice, and the teacher reference or bandwidth. Otherwise, narrow the contribution statement to the ablations actually reported.

  8. Treat the distillation result as preliminary. The Extra-Trees teacher blend changes macro nRMSE from 0.4631 to 0.4628, with mixed changes across datasets. This shows that the interface can be used, but does not yet establish a reliable transfer benefit. Please report paired uncertainty and repeated runs. If improved performance is a main claim, test the blend weight and the fraction of replaced episodes.

  9. Keep distributional and representation claims within their tested scope. Table 4 shows useful finite-sample improvements as support size grows. However, its fixed Gaussian width does not establish the stated asymptotic consistency conditions. Please keep this distinction clear. A width sweep across support sizes would help explain the smoothing effect. Similarly, Theorem 4.3 shows that label-dependent representations can beat weights that ignore current support labels. It does not establish a general advantage over label-adaptive kernel methods or show that practical pretraining learns the constructed solution. Table 8’s label-based selector is therefore an important control. Broader noise or finite-capacity studies would strengthen this part, but are extensions rather than requirements for the theorem’s correctness.

  10. Clarify training comparability, benchmark scope, and reproducibility. “TABICLV2-style encoder” leaves the initialization unclear. Please state whether this means the architecture or pretrained weights. Give the initialization, seeds, episode sampling, and loss weights for each control. Comparisons that change both the head and its loss should be described as comparisons of the complete designs. Keep the benchmark limits near the relevant tables: the evaluated subsets are not a complete TabArena run, and KRR uses at most 750 support rows. Provide dataset-level classification results and complete baseline search settings. Release the split indices, retrieval settings, checkpoints, and production teacher-solver tolerances and iteration limits.

  11. Support any efficiency interpretation with end-to-end measurements. Predicting several targets in one pass is useful, but it does not by itself establish a speed or memory advantage. Table 7 measures teacher-solver time, not full training or inference cost. If efficiency is part of the intended contribution, report latency, peak memory, and training cost under stated hardware and batch sizes. Include retrieval and encoder costs, especially when comparing one vector model with several scalar models. Otherwise, keep the claim limited to single-pass prediction.

The highest-priority revisions are the matched retrieval comparison, independent confirmation with uncertainty estimates, and a fair multi-output evaluation. The remaining points are needed to define the extensions clearly and keep the claims aligned with the evidence.

12 comments

Weaknesses

W1. Source and stability of the gains

Table 2 gives POINT query-local retrieval, while the retrieval for the other heads is unclear. The evaluations informed design choices, so the results need independent confirmation. Small differences at the larger budget need repeated runs and paired uncertainty. The multi-output comparison changes both the output structure and the loss.

(a) Matched retrieval. In the submitted Table 2, POINT received 512 retrieved rows per query, whereas the quantile, histogram and bar heads received the full support. We had used retrieval only for POINT because its readout over a very long context was slow, but we implemented a fused kernel to resolve this. We re-evaluated all four heads in both settings, on the same 132 splits and with the same retrieved rows. The table reports macro nRMSE. Below retrieval results are the mean over three new runs.

HeadFull support512 retrieved rows
POINT0.49260.4523
Logit + Quantile0.50420.4557
Logit + Histogram0.50220.4600
Logit + Bar0.56660.4738

Retrieval helps every head, by 0.04 to 0.09 macro nRMSE. Without retrieval, POINT has the lowest error. With retrieval, POINT and the quantile head are level, with 0.4523 against 0.4557, and both are ahead of the histogram and bar heads.

(b) Independent confirmation. We can clarify and qualify the sentence in App. C.1. We selected architecture and training configuration based on held-out synthetic episodes and on one split of each of Airfoil, Concrete and QSAR. Here, we evaluated on another held-out suite of 32 TabArena classification datasets. The table reports the mean and the seed standard deviation over three seeds.

ModelAccuracyLog loss
POINT87.14 ± 0.17%0.307 ± 0.002
Logit + Quantile87.10 ± 0.09%0.308 ± 0.001

(c) Repeated runs at the larger budget. We trained two more seeds of POINT and of the quantile head at the larger budget. We reports the mean and standard deviation over these.

Head (10,000 updates)AccuracyLog lossAirfoil RMSEConcrete RMSEQSAR RMSE
POINT83.08 ± 0.140.427 ± 0.0091.419 ± 0.0284.835 ± 0.0660.878 ± 0.002
Logit + Quantile83.04 ± 0.160.432 ± 0.0101.431 ± 0.0214.825 ± 0.1000.880 ± 0.003

The two heads are level on all five metrics, within one standard deviation.

(d) Multi-output control. In the new experiment, the scalar models use the same objective as the vector model, with the vector model better on six of eight datasets. Further, the pinball objective alone improves the scalar models on six of eight datasets.

W2. Connection between method, implementation and evidence

The theory studies POINT-CE, while the regression model also uses Smooth L1. The vector experiment does not explain targets outside the support-label convex hull. The tree teacher needs a rule for zero reference weights. The contribution statement promises component ablations that the experiments do not report.

(a) Theory and the practical loss. The theory covers POINT-CE alone, as App. A.10 states, and the practical loss adds Smooth L1. We therefore trained the objective that the theory analyses, POINT-CE without Smooth L1, and compared it with the full objective at the small budget of Table 3 (one seed).

Regression objectiveAirfoil RMSEConcrete RMSEQSAR RMSEBroad macro nRMSE
Smooth L1 + POINT-CE2.5766.1950.8950.4989
POINT-CE only2.3666.3060.9140.4864

POINT-CE alone stays within 0.21 RMSE of the full objective, and it is better on the ten broad datasets (paired difference -0.0125, 95% CI [-0.0189, -0.0063]). The analyzed objective is therefore a complete regression objective in practice. In contrast, Smooth L1 without any POINT-CE raises Concrete RMSE to 6.899 (Q5).

(b) Vector targets outside the hull. The vector teacher needs no projection, because every training target lies inside the hull by construction. We train the vector models on synthetic episodes whose targets are convex combinations of support labels, and we verified this on 16,384 training targets. At test time, the hull limits the prediction instead. We found that the submitted solver did not converge for the 14 targets of water quality. A corrected solver improves this dataset from 0.962 to 0.952 normalized RMSE and leaves the other datasets unchanged (Q4).

(c) Zero reference weights. We add a floor of 10−610^{-6} to every Extra-Trees weight and renormalize before the tilt, which is the first fix that you list. The floor is needed for 1% to 5% of the queries on five datasets, and for 58% on QSAR-TID-11, where the trees select about 6 of 3,828 support rows (Q6).

(d) Missing ablations. We report these report these in our response to Q7, together with matched comparisons under the final objective.

Questions

Q1. Matching retrieval across the heads

(a) Do the quantile, histogram and bar heads receive the same 512 retrieved rows as POINT? (b) Compare the heads under identical preprocessing, support pools, retrieval results and inference views. (c) Compare POINT with and without retrieval, to separate retrieval gains from head gains.

(a) Rows in the submitted Table 2. No. In the submitted Table 2, the three task-specific heads received the full support, whereas POINT received 512 retrieved rows per query.

(b) Matched comparison. All heads are under identical conditions. They share the same 132 splits, the same preprocessing fitted on the support, and one inference view per head, as App. C.1 specifies. The retrieval uses the features only. The table in our response to W1(a) reports the result. Under matched conditions, POINT and the quantile head are level, and both are ahead of the histogram and bar heads.

(c) POINT with and without retrieval. The table for W1(a) separates the two effects. Retrieval lowers the error of every head; the quantile head from 0.5042 to 0.4557 and the histogram head from 0.5022 to 0.4600. Retrieval therefore accounts for the gap in Table 2. In contrast, the heads are at parity at a fixed setting. Without retrieval, POINT is 0.012 ahead of the quantile head, and the 95% CI of this difference includes zero. With retrieval, the two heads are level, with 0.4523 against 0.4557. Retrieval lowers the error of every head. It lowers POINT from 0.4926 to 0.4523.

Q2. Independent confirmation and uncertainty

(a) Freeze the method and evaluate it on datasets that did not inform its design. (b) Identify the datasets used for development. (c) Report multiple pretraining seeds and paired differences with confidence intervals. (d) The split SD in Table 1 does not measure training randomness. (e) The larger-budget results show competitive performance, without a clear advantage.

(a) Held-out evaluation. We evaluated frozen checkpoints once on 32 TabArena classification datasets not otherwise used. We fixed the selection rules a priori: at most 10 classes, at most 100 features, and the split protocol of App. C.1. The table for W1(b) reports the result for three seeds of each head, and the two heads are at parity there.

(b) Development datasets. We selected the architecture and the training configuration on held-out synthetic episodes and on one split of each of Airfoil, Concrete and QSAR. All other real datasets served for evaluation only. The 32 held-out datasets of (a) are disjoint from all of them. The sentence in App. C.1 is broader than this, and we correct it in W1(b).

(c) Seeds and paired uncertainty. Three analyses now use three pretraining seeds. These are the larger-budget comparison of POINT and the quantile head (W1(c)), the four objectives of Table 3 (our reply to Reviewer BhE1, W2), and the held-out suite (W1(b)). For the paired comparisons on the broad regression suite, we compute the gaps on identical dataset and split pairs, average the splits within each dataset, and bootstrap over the ten datasets with 5,000 resamples.

(d) Split SD. The ± values in Table 1 are standard deviations over the 30 evaluation splits of one checkpoint; we have added the seed standard deviation.

(e) Larger budget. We agree that the larger-budget results show parity. Paired over the 57 datasets, POINT wins on 24, ties on 10 and loses on 23 against the quantile head. The mean paired difference is +0.13 accuracy points (95% CI [-0.81, +1.07]) and -0.003 log loss (CI [-0.027, +0.021]). The three-seed comparison in W1(c) gives the same picture.

Q3. A fair and reproducible multi-output comparison

(a) Compare the vector model and independent scalar models under matched objectives, budgets and preprocessing. (b) Give the Linnerud split protocol, repeat counts and uncertainty. (c) Add further multi-output datasets.

(a) Matched comparison. We added scalar POINT models with the same objective, architecture, preprocessing and 1,000 training steps as the vector model, and we kept the pinball-supervised scalar models of Fig. 2a as a second control. The table reports the RMSE divided by the target standard deviation, as the mean over 30 splits and five training seeds. Bold marks the better of the two matched models.

Dataset (targets)mm scalar models, pinballmm scalar models, same objectiveVector POINT
Linnerud (3)1.1881.0961.109
Energy efficiency (2)0.3050.4940.437
Concrete slump (3)0.7200.8680.815
Jura (3)0.6970.8200.757
EDM (2)0.7500.8470.828
Solar flare (3)1.0480.9951.011
Water quality (14)0.9530.9780.962
atp1d (6)0.4870.6410.501

Under the matched objective, the vector model is better on six of eight datasets. In contrast, the pinball objective improves the scalar models on six datasets, so the submitted comparison mixed the objective and the output structure, as you pointed out.

(b) Linnerud protocol. The submitted experiment uses 30 ShuffleSplit splits with a test size of 0.25 and random_state 43. We standardize inputs and targets with the statistics of the training split, and we pretrain the models for 1,000 steps on synthetic episodes only. The submitted numbers come from one training seed. Over five seeds, the vector model reaches an RMSE of 11.258 ± 0.012, against 11.999 ± 0.056 for the pinball scalar models, and it is better in every seed.

(c) Additional datasets. The table above adds seven public datasets with 2 to 14 targets. These small models train on synthetic data only, so they trail Extra Trees on six datasets. In contrast, the pretrained checkpoints of Table 1, applied once per target, beat Extra Trees and Random Forest on five of the eight.

Reviewer_BhE1OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes POInT, a unified support-indexed prediction head for tabular in-context learning. It predicts a probability distribution over support examples and aggregates their labels for classification and regression. The proposed POInT-CE objective constructs task-specific teacher distributions, using uniform same-class weights for classification and a mean-preserving Gaussian projection for regression. The paper provides theoretical consistency and distributional approximation results, along with experiments comparing shared and task-specific prediction heads.

Critical strengths

The unified prediction interface is conceptually appealing and naturally supports classification, scalar regression, and vector-valued outputs. The moment-preserving teacher construction provides a principled way to supervise continuous targets through support-level cross-entropy.

The theoretical analysis is relatively thorough and explicitly addresses support coverage and approximation limitations. The experiments demonstrate competitive performance against task-specific heads, while the shared-head and loss ablations provide useful evidence supporting the proposed framework.

Critical weaknesses

The empirical contribution of POInT-CE is not sufficiently isolated. The categorical objective combines class-level NLL with within-class regularization, making it unclear whether the improvements arise from the proposed support-level supervision or changes in effective loss weighting. Similarly, the benefit of the Gaussian moment-preserving teacher over simpler alternatives remains unclear.

The interaction between classification and regression supervision is insufficiently understood. Table 3 shows that adding regression POInT-CE alone worsens the overall regression RMSE, whereas combining it with classification POInT-CE improves performance. The paper acknowledges this mixed behavior but does not adequately investigate its cause or stability.

The broader advantages of the unified head remain insufficiently established. Under the larger training budget, POInT matches rather than substantially improves upon the strongest task-specific regression baseline. Moreover, multi-output evaluation is limited, and the main real-world regression experiments do not directly assess distributional prediction quality.

Questions for authors
  1. Would a matched NLL-reweighting baseline help distinguish the contribution of categorical support-level supervision? For regression, how does the Gaussian teacher compare with simpler moment-preserving interpolation targets?

  2. Is the interaction between classification and regression POInT-CE observed in Table 3 consistent across pretraining seeds or training budgets?

  3. How well does the shared head generalize to more diverse multi-output regression tasks and real-world distributional prediction metrics?

5 comments

W1 and Q1. Isolating the contribution of POINT-CE

Do the gains come from support-level supervision or from a change in the effective loss weighting, and would a matched NLL-reweighting baseline separate the two? How does the Gaussian moment-preserving teacher compare with simpler alternatives, such as moment-preserving interpolation targets?

(a) Support-level supervision or loss reweighting. Table 3 shows that adding classification POINT-CE improves accuracy from 80.65% to 81.33%, reduces log loss from 0.507 to 0.476, and lowers Airfoil RMSE from 2.515 to 2.225. However, as Eq. (11) shows, adding 0.3LPOINT-CE0.3L_{\mathrm{POINT\text{-}CE}} also increases the effective NLL weight by 0.3.

To separate these effects, we trained a matched control that increases the NLL weight by 0.3 without adding classification POINT-CE. Both variants follow the Full configuration of Table 3, including the classification loss scale of 1.50. All experiments use the same code version and three seeds, with results reported as mean ± seed standard deviation.

ObjectiveAccuracy (%) ↑Log loss ↓Airfoil RMSE ↓Concrete RMSE ↓QSAR RMSE ↓
Full objective81.80 ± 0.400.455 ± 0.0012.226 ± 0.0805.866 ± 0.1180.891 ± 0.003
NLL reweighting, without classification POINT-CE81.39 ± 0.410.467 ± 0.0032.615 ± 0.2086.232 ± 0.3880.896 ± 0.007
Untilted Gaussian teacher81.43 ± 0.710.465 ± 0.0122.248 ± 0.1715.966 ± 0.0970.892 ± 0.004
Nearest-label teacher81.69 ± 0.480.462 ± 0.0022.330 ± 0.1885.959 ± 0.1520.892 ± 0.004

The full objective improves accuracy, log loss, Airfoil RMSE, and Concrete RMSE over the NLL-reweighted control in all three seeds, while QSAR results remain close. Therefore, the improvement cannot be explained by increased NLL weighting alone. Support-level supervision contributes beyond this change in loss scale.

(b) Gaussian teacher versus simpler alternatives. Two additional teachers help examine the role of the moment constraint. The untilted Gaussian uses the same reference distribution but removes the mean-preserving constraint. The nearest-label teacher instead assigns all probability mass to the support label closest to the target.

Compared with both alternatives, the moment-preserving Gaussian achieves better average accuracy, log loss, and RMSE across all three regression datasets. The differences are modest, however. The nearest-label teacher has higher log loss in every seed, while the regression differences change direction across seeds.

Taken together, these results suggest that preserving the target mean is beneficial on average, particularly for classification log loss, although its effect on regression varies across runs. The complementary matched teacher comparison appears in Q7(c).

W2 and Q2. Interaction between classification and regression POINT-CE

What causes the mixed effect of regression POINT-CE in Table 3, which hurts regression alone and helps when combined with classification POINT-CE? Is this behaviour stable across pretraining seeds? Is it stable across training budgets?

(a) Cause. The effect of regression POINT-CE depends on the weight of the classification loss, and Table 3 contains two such weights. Its three ablation rows use a classification loss scale of 0.25, and its Full row uses 1.50. At 0.25, regression POINT-CE trades Airfoil against Concrete. In a three-seed rerun at that scale, adding it alone raises Airfoil RMSE by 0.25 and lowers Concrete RMSE by 0.95. At 1.50, it helps both datasets. We therefore retrained all four objectives at 1.50 with three seeds and one code version. The table reports the mean and the seed standard deviation.

ObjectiveAccuracyLog lossAirfoil RMSEConcrete RMSEQSAR RMSE
1. Both POINT-CE terms81.80 ± 0.400.455 ± 0.0012.226 ± 0.0805.866 ± 0.1180.891 ± 0.003
2. Classification POINT-CE only81.68 ± 0.560.460 ± 0.0112.319 ± 0.0675.904 ± 0.1680.886 ± 0.005
3. Regression POINT-CE only81.65 ± 0.770.463 ± 0.0052.437 ± 0.0456.248 ± 0.2370.898 ± 0.005
4. Neither81.39 ± 0.630.477 ± 0.0152.511 ± 0.1976.271 ± 0.1840.887 ± 0.007

At this scale, both terms help, and their effects add up. Regression POINT-CE lowers Airfoil and Concrete RMSE on average, both alone (row 4 to row 3) and on top of classification POINT-CE (row 2 to row 1). Classification POINT-CE lowers log loss, Airfoil RMSE and Concrete RMSE (row 3 to row 1, and row 4 to row 2). The full objective is therefore the best row on accuracy, log loss, Airfoil and Concrete, and QSAR stays within 0.012 across the rows.

The mechanism is the sharpness of the kernels. With regression POINT-CE, a regression query spreads its mass over about 275 support rows, against 30 without any POINT-CE and 66 with classification POINT-CE alone (effective number of rows, 1/∑ipi21/\sum_i p_i^2). Classification POINT-CE broadens the classification kernel in the same way, from about 50 rows to about 300.

(b) Stability across seeds. The three-seed matched comparison supports the mean trends in (a). The full objective has the best mean accuracy, log loss, Airfoil RMSE, and Concrete RMSE; classification POINT-CE alone has the lowest mean QSAR RMSE. Relative to the full objective, removing both POINT-CE terms worsens all four of the former metrics on average. The additional gain from including regression POINT-CE on top of classification POINT-CE is modest (Airfoil: 2.319 to 2.226; Concrete: 5.904 to 5.866) relative to the observed seed variability. We therefore do not claim that the combined objective dominates in every seed; the evidence supports a mean improvement at this training budget.

(c) Stability across budgets. We have the four-objective comparison at the small budget only. At the larger budget of 10,000 updates, we have one related check with one seed, which varies the weight of regression POINT-CE while classification POINT-CE stays on. Lowering this weight from 0.05 to 0.01 improves Airfoil RMSE from 1.568 to 1.461, Concrete RMSE from 5.054 to 4.789 and log loss from 0.429 to 0.417, whereas accuracy stays between 82.7% and 82.8%. A smaller regression weight is therefore better on both regression datasets at this budget, so the Concrete benefit of regression POINT-CE at the small budget does not carry over in this check.

W3 and Q3. Advantages of the unified head

POINT matches the strongest task-specific regression baseline. What does the unified head add? The multi-output evaluation is limited. How well does the shared head generalize to more diverse multi-output tasks? The real-world regression experiments do not assess distributional quality. How does the head perform on real-world distributional metrics?

(a) What the unified head adds. We agree that POINT matches the best task-specific head in accuracy, and it does so with one head for classification, regression and vector targets. Both heads reach a geometric-mean RMSE of 1.841 at the larger budget (Table 1), and POINT is within 0.01 macro nRMSE of the best head under matched retrieval. Parts (b) and (c) therefore test what one head covers beyond accuracy.

(b) Multi-output evaluation. We extended the study from Linnerud to eight multi-target datasets, with 30 splits and five training seeds. The vector model predicts all targets in one pass, whereas the scalar baseline trains one POINT model per target with the same objective. The table reports the RMSE divided by the target standard deviation, and bold marks the better model.

Dataset (targets)mm scalar POINT modelsVector POINT
Linnerud (3)1.0961.109
Energy efficiency (2)0.4940.437
Concrete slump (3)0.8680.815
Jura (3)0.8200.757
EDM (2)0.8470.828
Solar flare (3)0.9951.011
Water quality (14)0.9780.962
atp1d (6)0.6410.501

The vector model wins on six of the eight datasets, and it needs one pass where the baseline needs mm (0.75 ms against 4.41 ms on water quality). These small models train on synthetic data only, so they trail Extra Trees on six datasets, and the pinball-supervised scalar models of Fig. 2a are also stronger on six. In contrast, the pretrained checkpoints of Table 1, applied once per target, beat Extra Trees and Random Forest on five of the eight.

(c) Distributional quality on real data. POINT's predictive distribution Fp=∑ipiδyiF_p=\sum_i p_i\delta_{y_i} matches the quantile head on real data. On the 13 regression datasets (222 splits), macro CRPS is 0.2226 for POINT against 0.2239, and macro pinball loss over 199 levels is 0.1113 against 0.1120, in units of the support-label standard deviation. The 95% confidence intervals of both differences include zero. The quantile head trains on the pinball loss directly, whereas POINT reaches the same score through POINT-CE alone, which is consistent with Thm. 4.2.

Reviewer_UPSQOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes POINT, a paradigm for tabular foundation models that unifies classification and regression by learning a probability distribution over the labeled support rows. Instead of using task-specific output heads (e.g., class logits for classification and quantiles/bins for regression), POINT uses a shared metric head to score query-support pairs and outputs a softmax distribution over the support examples. To supervise this shared distribution, the authors introduce POINT-CE, a cross-entropy loss against a moment-preserving teacher distribution. The authors provide theoretical analysis showing that excess POINT-CE risk controls classification log-loss and regression mean error, and empirically demonstrate that the shared head is competitive with task-specific heads on various tabular benchmarks.

Critical strengths
  1. Elegant Unified Interface: The introduction of a single probability simplex over support rows and the POINT-CE loss is a conceptually significant contribution. It elegantly resolves the mismatch between categorical targets and continuous targets by using a shared optimization core (cross-entropy against a moment-preserving teacher).

  2. Theoretical Rigor: The paper establishes a strong theoretical link between the support-level cross-entropy excess risk and downstream tasks (classification log-loss and regression mean error). The connection to Wasserstein distance and integrated pinball risk (Theorem 4.2) provides a solid bridge to the distributional regression objectives widely used in modern TFMs.

Critical weaknesses
  1. Lack of Adaptive Bandwidth to Representation Scale: The softmax temperature is governed by a global learned bandwidth hh and a fixed scale TT. Since the metric MLP output uu can vary significantly across different datasets, a global hh acts as a fixed statistical prior. For datasets where the uu variance is vastly different from the pretraining episodes, the softmax will either become too sharp (degenerating into a 1-NN retriever) or too smooth (losing locality). The paper lacks a per-episode adaptive bandwidth mechanism (e.g., scaling hh by the median pairwise distance of the support set).

  2. Regression Teacher is Sensitive to Support Label Density and Imbalance: The Gaussian reference for the regression teacher assigns mass based on the absolute distances between support labels, fundamentally treating the support set as an IID observation of the conditional distribution. If the support labels are highly imbalanced (e.g., dense in one region and sparse in another), the Gaussian reference will be heavily skewed. While the mean-tilt λλ forces the first moment to match, it does not correct the variance or shape. For out-of-hull targets or support gaps, this leads to large approximation errors, which is explicitly reflected in the paper's own error bound (ΔSΔS). The paper fails to analyze or ablate the impact of support label imbalance on the teacher distribution.

Questions for authors
  1. Adaptive Bandwidth: Since hh is globally learned and TT is fixed, how does POINT handle datasets with drastically different uu variance? Have the authors considered normalizing the embeddings uu or using a per-episode context-dependent bandwidth (e.g., based on support distances) to decouple the softmax sharpness from the representation scale?

  2. Support Label Imbalance: For regression, the Gaussian reference relies heavily on the empirical density of support labels. How does the teacher distribution behave when the support labels are highly imbalanced (e.g., a mix of dense and sparse regions)? Does the mean-preserving tilt λλ suffer from numerical instability or severe shape distortion in these cases?

  3. Retrieval vs. Distribution Head: The paper claims a distinction from standard retrieval methods (like TabR) by using a support-indexed prediction head. However, for long tables, the paper uses FAISS to retrieve a subset of 512 support rows. Does this hard retrieval step break the theoretical guarantee of the support-level simplex? How does the model handle cases where the retrieved subset lacks coverage for the target?

  4. Cross-Task Transfer: The paper claims a benefit of joint training is potential cross-task transfer (classification teaching discrimination, regression teaching graded allocation). Can the authors provide an ablation study isolating this transfer benefit (e.g., comparing joint training vs. training separate models with the same total budget) to substantiate this claim quantitatively?

4 comments

Weaknesses

W1 and Q1. Adaptive bandwidth and representation scale

Does a global hh with a fixed TT make the softmax too sharp or too smooth on datasets whose embedding scale differs from pretraining? Did you consider normalizing uu, or a per-episode bandwidth such as one scaled by the median pairwise support distance?

(a) Kernel sharpness on real data. We do not observe this (collapse toward 1-NN). We measure sharpness as the effective number of neighbours, 1/∑ipi21/\sum_i p_i^2, which is 1 for a 1-NN kernel and SS for uniform weights over SS support rows. The table gives the range across datasets for the paper's checkpoint.

DatasetsMedian distance between embeddings uuEffective neighbours (share of support)Queries with over half the mass on one row
13 regression69 to 14013% to 67%at most 0.1%
57 classification9 to 15113% to 99.6%0%

The embedding scale varies widely across datasets, however the encoder reads the whole support set, and we train hh jointly with it. The encoder therefore chooses the scale of uu for each dataset, and this choice sets the sharpness.

At the smooth end, seven binary classification datasets have a nearly uniform kernel. On three of them (GAMETES, hill-valley, eye movements) the model finds almost no signal, and accuracy is 51% to 58%. On the other four one class dominates, and the nearly uniform weights still beat the class prior. For example, log loss is 0.031 against 0.044 on mc1.

(b) Normalization and per-episode bandwidth. Yes, we tested three rules at inference time, without retraining, and none give a consistent gain. The first rule is your suggestion: hh is rescaled by the median pairwise distance between the support embeddings of each episode, normalized by its average over the development splits. The second rule normalizes each embedding to unit norm and the third selects the temperature by 5-fold cross-validation on the support set. We report the change relative to the learned bandwidth for the paper's checkpoint, on the 57 classification datasets and the 10 broad regression datasets.

RuleAccuracy (points)Log lossMacro nRMSE
Median-distance bandwidth-3.25+0.807+0.010
Unit-norm embeddings-1.58+0.018+0.002
Cross-validated temperature-0.68-0.012+0.000 (7 datasets)

Median-distance scaling is clearly worse because it divides out the scale of uu that the encoder uses to set the sharpness. Unit-norm embeddings lower accuracy and leave regression unchanged. We attribute this loss to the norm term of Sec. 3.1, which normalization removes. Further, the cross-validated temperature trades 0.68 accuracy points for a 0.012 lower log loss, which is not statistically significant, and it needs five extra forward passes per split. We therefore keep the learned bandwidth.

W2 and Q2. Regression teacher under label density and imbalance

Does an imbalanced support-label distribution skew the Gaussian reference, and how does the teacher behave? The tilt matches the mean. Does it leave the variance and shape uncorrected, and can the shape distort severely? Is the tilt λ\lambda numerically stable under imbalance? Do out-of-hull targets and support gaps cause large approximation errors? Where is the analysis of label imbalance on the teacher?

(a) Behaviour under imbalance. Imbalance barely moves the teacher. We standardize the labels by the mean and standard deviation (SD) of the support labels, and all label distances below are in these SD units. The Gaussian reference is a kernel that we centre at the query target yy, with a width of 0.05 SD, so it gives mass only to labels close to yy. We tested this on the three development datasets by duplicating the support rows on one side of the label median up to 8 times, with the targets fixed. The teacher mass on that side moves from 0.499 to 0.510.

(b) Variance and shape. Yes, the tilt fixes only the mean, because one observed target identifies only the mean. The Gaussian reference therefore sets the width. The tilt shifts the kernel centre from yy to y+λσ2y+\lambda\sigma^2 and keeps the width σ\sigma. With a median ∣λ∣\lvert\lambda\rvert of about 2 on the development datasets, the centre moves by 0.005 SD, which is one tenth of the kernel width. Accordingly, the teacher variance stays between 0.0028 and 0.0033 in the experiment of (a), which is about one tenth of the Eq. (12) bound. The shape distorts only where no support label lies near the target, as the table in (e) shows.

(c) Numerical stability. The tilt stays numerically stable. The mean of the tilted teacher increases strictly with λ\lambda (Prop. A.2), so we find λ\lambda by bisection, and targets at the ends of the support range use the explicit rule of App. B.1. We measure stability by the teacher mean error ∣∑iqiyi−y∣|\sum_i q_i y_i - y|. During training, this error is 1.11.1 to 1.2×10−71.2\times10^{-7} on average and at most 3.0×10−63.0\times10^{-6}, over 3,000 regression updates from three seeds. Under imbalance, the tilt grows and the solution stays accurate. The table groups the real queries of (e) by how the support labels within 0.25 SD of the target split between its left and right side.

Left and right split of nearby labelsMedian ∣λ∣\lvert\lambda\rvert
Up to 60/401.1
60/40 to 75/252.8
75/25 to 90/109.4
Beyond 90/1048
All on one side (0.9% of the queries)9,060

The teacher mean error stays at most 3.7×10−53.7\times10^{-5} in all five groups, and the bisection reaches the end of its search interval for only 0.004% of the interior targets. A large tilt therefore reflects one-sided labels, and the solver still satisfies the mean constraint.

(d) Out-of-range targets and support gaps. Both effects exist, and both are rare on real data. On the ten TabArena datasets, 89 of 667,877 targets (0.0133%) lie outside the support-label range, and they contribute at most 3.7% of the squared error of any dataset. Support gaps are more frequent: 0.3% of the queries have at most two support labels within 0.1 SD of the target, as the table in (e) shows.

(e) Analysis of imbalance. We added the controlled experiment of (a) and a grouping of the real queries from all 13 regression datasets. The table reports the teacher variance as a fraction of the Eq. (12) bound, and POINT's RMSE in SD units.

Support labels within 0.1 SD of the targetQueriesTeacher variance / boundPOINT RMSE
0480.581.68
1 to 22880.151.25
3 to 56270.090.89
6 to 204,6150.080.64
21 to 10032,3840.070.38
More than 10083,4340.040.52

The teacher variance stays inside the bound in every group, and it exceeds 15% of the bound only for the 48 queries with no nearby label. In contrast, POINT's error grows where labels are sparse, which is the support-gap effect of (d).

Questions

Q3. Retrieval and the support-indexed head

Does the hard retrieval of 512 support rows break the guarantees of the support-level simplex, and does the distinction from TabR remain? How does the model handle a retrieved subset that lacks coverage for the target?

(a) Retrieval and the simplex guarantees. The guarantees hold for the retrieved context. We state Theorems 4.1 and 4.2 for the support set that the model receives, and under retrieval this set is the 512 retrieved rows. We select them from the features alone, so the selection is independent of the query label. The distinction from TabR also remains, because retrieval only selects the rows, and POINT then reads their observed labels through one distribution over them. Further, retrieval improves accuracy. On the 132 splits of Table 2, POINT's macro nRMSE falls from 0.4926 with the full support to 0.4631.

(b) Missing coverage. By default, POINT's prediction stays inside the range of the retrieved labels, and Eq. (10) gives the resulting bias. This case is rare, as the table shows for the same queries with and without retrieval.

DatasetsTargets outside the full-support rangeTargets outside the retrieved range
10 broad (667,877 queries)0.013% (89)0.16% (1,078)
3 development0.30%0.41%

Moreover, the affected targets lie close to the range, with a median distance of at most 0.4 standard deviations of the support labels on 11 of the 13 datasets.

Further, we tested a local-linear readout for this response, which can predict outside the range without retraining. It keeps POINT's weights and adds a weighted linear fit around the query, y^=yˉ+B^(x−xˉ)\hat y = \bar y + \hat B(x - \bar x), where xˉ\bar x and yˉ\bar y are the weighted means of the context features and labels and B^\hat B is the weighted regression slope. It lowers the normalized RMSE on 7 of 8 multi-output datasets with small supports, whereas it changes the RMSE by at most 0.03 on the three development datasets with the paper's checkpoints. We have not adopted it so far, because out-of-range targets are rare and the correction is not as elegant as the plain weighted mean. The revised paper describes this readout and both results in App. B.4.

Q4. Cross-task transfer

Can you isolate the transfer benefit of joint training, for example against separate models with the same total budget?

Yes, joint training transfers from classification to regression. We trained POINT specialists on one task only and compare each with the joint model at the small budget (2,000 updates, one seed). The first specialist sees the same episodes of its task as the joint model, which is half the updates. The second specialist follows your proposal and receives the full 2,000 updates. This favours the specialist, because it sees twice as many episodes as the joint model does. The table reports the specialist minus the joint model. A positive RMSE entry means that the specialist is worse, and a positive accuracy entry means that it is better.

Specialist minus joint POINTAirfoil RMSEConcrete RMSEQSAR RMSEAccuracy (points)Log loss
Same episodes of its task+1.068+2.247+0.043-0.33-0.007
Same total updates+0.100+0.3440.000+1.01-0.064

For regression, the joint model wins in both rows, beating the specialist with twice as many regression episodes, which is better on only 7 of 30 Airfoil splits and 4 of 30 Concrete splits. Classification episodes therefore help regression more than additional regression episodes do. In contrast, the classification specialist matches the joint model with the same classification episodes. Its gain of 1.01 accuracy in the second row therefore comes from seeing twice as many classification episodes, and regression episodes add little to classification.

Part of this transfer goes through the shared encoder, because a joint logit and quantile model also gains over its regression specialist (Airfoil RMSE 3.483 to 3.025). The shared head adds a smaller effect on top. Over three seeds, it lowers the three RMSEs by 0.03, 0.15 and 0.01 against separate heads, which is within one seed standard deviation.

Dear Reviewer UPSQ,

As we have not yet received any feedback on our rebuttal, we would like to kindly follow up. With just over two days remaining in the rebuttal and discussion period, we would greatly appreciate it if you could let us know whether our responses have sufficiently addressed your concerns. If any points remain unclear or unresolved, we would be happy to provide further clarification before the period closes.

Best regards

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.