- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents POINT, a shared output head for tabular in-context learning. It predicts a probability distribution over support examples. Classification adds up the weights of examples from the same class. Regression takes a weighted average of their labels. The same readout also works with vector labels.
The paper introduces POINT-CE to train this head. It constructs a target distribution over support examples whose weighted labels match the query target. The theory connects this objective to class probabilities, regression means, and distributional prediction under stated conditions.
The experiments cover 57 classification datasets, three regression development datasets, ten further regression datasets, and small multi-output and synthetic studies. The results support the basic feasibility of the shared interface. However, several comparisons need stronger controls, and some proposed extensions need clearer implementation details.
Critical strengths
-
shared prediction interface is simple and meaningful. The same learned head handles classification and regression through support-example probabilities. POINT-CE gives these tasks a common training interface. This offers a useful way to share the prediction layer, beyond sharing only the encoder.
-
The controlled experiments provide positive evidence. In Table 3, adding the full POINT-CE objective improves classification accuracy from 80.65% to 82.04% and reduces regression GM RMSE from 2.500 to 2.232. Table 1 also compares shared and separate metric heads under the same episode budgets. These results support the value of the approach. The theory further helps explain what the common loss can recover and where support coverage becomes important.
Critical weaknesses
-
The evidence does not yet cleanly establish the source and stability of the broader gains. Table 2 uses query-local retrieval for POINT, but equivalent retrieval for the other neural heads is unclear. Appendix C.1 also states that these evaluations informed design and serving choice. The results therefore need independent confirmation. Small differences at the larger training budget are difficult to assess without repeated training runs and paired uncertainty estimates. The multi output comparison has a further control problem: it changes both the output structure and the training loss.
-
Some parts of the proposed method are not fully connected to the practical implementation and evidence. The main theory studies POINT-CE, while the main regression model also uses Smooth L1. The paper correctly acknowledges that this can change the population optimum. The vector experiment does not clearly explain targets outside the support-label convex hull. The tree-teacher extension also needs a rule for zero reference weights. In addition, the contribution statement promises several component ablations that I could not find in the reported experiments.
Questions for authors
-
Match retrieval conditions across output heads. Section 5.3 gives POINT query-local contexts with 512 retrieved support rows. Please state whether the quantile, histogram, and bar heads receive exactly the same rows. Compare the heads using identical preprocessing, support pools, retrieval results, and inference views. A POINT comparison with and without retrieval would also help separate retrieval gains from head gains.
-
Add independent confirmation and report uncertainty. Please freeze the method and evaluate it on datasets that did not inform its design. Clearly identify which datasets were used for development. Report multiple pretraining seeds and paired differences across datasets, with confidence intervals for the main summaries. The split SD in Table 1 does not measure training randomness. At the larger budget, accuracy is 83.01% versus 82.88% for the quantile baseline, and both have regression GM RMSE of 1.841. These results support competitive performance, but do not establish a clear advantage.
-
Make the multi-output comparison fair and reproducible. Vector POINT uses a uniform-reference POINT-CE teacher, while the three scalar models use pinball supervision. Please compare joint and independent scalar models under matched objectives, training budgets, and preprocessing. Give the Linnerud split protocol, repeat counts, and uncertainty. Several additional multi-output datasets would help test whether the result extends beyond this one example.
-
Explain vector targets outside the support-label convex hull. A weighted average of support labels must remain inside their convex hull. Appendix B.2 acknowledges this limit, but the Linnerud implementation is unclear. Please report how often query targets fall outside this region and explain the projection or fallback rule. Give target scaling, solver tolerances, and reconstruction errors. If the teacher uses a projected target, state that it preserves the projected target rather than the original one. Table 7 only tests known targets generated inside the hull, so it does not resolve this issue.
-
Connect the practical regression loss to the theoretical claims. Table 5 uses Smooth L1 plus 0.3 POINT-CE. Appendix A.10 correctly shows that this combination need not recover the conditional mean. Please bring this qualification into the main text. Add matched CE-only results on the broader regression suite and vary the auxiliary-loss weight. A squared-error auxiliary would also be a useful control. Reporting pinball loss from POINT’s induced predictive distribution would help test whether the theoretical distributional benefits appear on real datasets.
-
Define the tree-teacher projection when reference weights contain zeros. Equation (3) needs more than membership in the full support-label convex hull. For finite KL, the target must lie in the hull of labels with positive reference weight. For example, support labels (0, 1), reference weights (1, 0), and target 0.5 give no feasible teacher with finite KL. Please explain how this is handled for Extra Trees. Possible fixes include adding positive reference mass, mixing with a positive Gaussian reference before projection, or using a stated fallback. Report how often this case occurs and check the final teacher’s mean error.
-
Align the claimed ablations with the experiments shown. The contribution list says the experiments isolate retrieval geometry, robust point losses, and teacher smoothing. Table 3 mainly tests classification and regression POINT-CE switches. Table 8 tests the role of support labels in learning the representation. Please add matched comparisons for Euclidean versus dot-product scoring, the point-loss choice, and the teacher reference or bandwidth. Otherwise, narrow the contribution statement to the ablations actually reported.
-
Treat the distillation result as preliminary. The Extra-Trees teacher blend changes macro nRMSE from 0.4631 to 0.4628, with mixed changes across datasets. This shows that the interface can be used, but does not yet establish a reliable transfer benefit. Please report paired uncertainty and repeated runs. If improved performance is a main claim, test the blend weight and the fraction of replaced episodes.
-
Keep distributional and representation claims within their tested scope. Table 4 shows useful finite-sample improvements as support size grows. However, its fixed Gaussian width does not establish the stated asymptotic consistency conditions. Please keep this distinction clear. A width sweep across support sizes would help explain the smoothing effect. Similarly, Theorem 4.3 shows that label-dependent representations can beat weights that ignore current support labels. It does not establish a general advantage over label-adaptive kernel methods or show that practical pretraining learns the constructed solution. Table 8’s label-based selector is therefore an important control. Broader noise or finite-capacity studies would strengthen this part, but are extensions rather than requirements for the theorem’s correctness.
-
Clarify training comparability, benchmark scope, and reproducibility. “TABICLV2-style encoder” leaves the initialization unclear. Please state whether this means the architecture or pretrained weights. Give the initialization, seeds, episode sampling, and loss weights for each control. Comparisons that change both the head and its loss should be described as comparisons of the complete designs. Keep the benchmark limits near the relevant tables: the evaluated subsets are not a complete TabArena run, and KRR uses at most 750 support rows. Provide dataset-level classification results and complete baseline search settings. Release the split indices, retrieval settings, checkpoints, and production teacher-solver tolerances and iteration limits.
-
Support any efficiency interpretation with end-to-end measurements. Predicting several targets in one pass is useful, but it does not by itself establish a speed or memory advantage. Table 7 measures teacher-solver time, not full training or inference cost. If efficiency is part of the intended contribution, report latency, peak memory, and training cost under stated hardware and batch sizes. Include retrieval and encoder costs, especially when comparing one vector model with several scalar models. Otherwise, keep the claim limited to single-pass prediction.
The highest-priority revisions are the matched retrieval comparison, independent confirmation with uncertainty estimates, and a fair multi-output evaluation. The remaining points are needed to define the extensions clearly and keep the claims aligned with the evidence.
12 comments
Questions
Q1. Matching retrieval across the heads
(a) Do the quantile, histogram and bar heads receive the same 512 retrieved rows as POINT? (b) Compare the heads under identical preprocessing, support pools, retrieval results and inference views. (c) Compare POINT with and without retrieval, to separate retrieval gains from head gains.
(a) Rows in the submitted Table 2. No. In the submitted Table 2, the three task-specific heads received the full support, whereas POINT received 512 retrieved rows per query.
(b) Matched comparison. All heads are under identical conditions. They share the same 132 splits, the same preprocessing fitted on the support, and one inference view per head, as App. C.1 specifies. The retrieval uses the features only. The table in our response to W1(a) reports the result. Under matched conditions, POINT and the quantile head are level, and both are ahead of the histogram and bar heads.
(c) POINT with and without retrieval. The table for W1(a) separates the two effects. Retrieval lowers the error of every head; the quantile head from 0.5042 to 0.4557 and the histogram head from 0.5022 to 0.4600. Retrieval therefore accounts for the gap in Table 2. In contrast, the heads are at parity at a fixed setting. Without retrieval, POINT is 0.012 ahead of the quantile head, and the 95% CI of this difference includes zero. With retrieval, the two heads are level, with 0.4523 against 0.4557. Retrieval lowers the error of every head. It lowers POINT from 0.4926 to 0.4523.
Q2. Independent confirmation and uncertainty
(a) Freeze the method and evaluate it on datasets that did not inform its design. (b) Identify the datasets used for development. (c) Report multiple pretraining seeds and paired differences with confidence intervals. (d) The split SD in Table 1 does not measure training randomness. (e) The larger-budget results show competitive performance, without a clear advantage.
(a) Held-out evaluation. We evaluated frozen checkpoints once on 32 TabArena classification datasets not otherwise used. We fixed the selection rules a priori: at most 10 classes, at most 100 features, and the split protocol of App. C.1. The table for W1(b) reports the result for three seeds of each head, and the two heads are at parity there.
(b) Development datasets. We selected the architecture and the training configuration on held-out synthetic episodes and on one split of each of Airfoil, Concrete and QSAR. All other real datasets served for evaluation only. The 32 held-out datasets of (a) are disjoint from all of them. The sentence in App. C.1 is broader than this, and we correct it in W1(b).
(c) Seeds and paired uncertainty. Three analyses now use three pretraining seeds. These are the larger-budget comparison of POINT and the quantile head (W1(c)), the four objectives of Table 3 (our reply to Reviewer BhE1, W2), and the held-out suite (W1(b)). For the paired comparisons on the broad regression suite, we compute the gaps on identical dataset and split pairs, average the splits within each dataset, and bootstrap over the ten datasets with 5,000 resamples.
(d) Split SD. The ± values in Table 1 are standard deviations over the 30 evaluation splits of one checkpoint; we have added the seed standard deviation.
(e) Larger budget. We agree that the larger-budget results show parity. Paired over the 57 datasets, POINT wins on 24, ties on 10 and loses on 23 against the quantile head. The mean paired difference is +0.13 accuracy points (95% CI [-0.81, +1.07]) and -0.003 log loss (CI [-0.027, +0.021]). The three-seed comparison in W1(c) gives the same picture.
Q3. A fair and reproducible multi-output comparison
(a) Compare the vector model and independent scalar models under matched objectives, budgets and preprocessing. (b) Give the Linnerud split protocol, repeat counts and uncertainty. (c) Add further multi-output datasets.
(a) Matched comparison. We added scalar POINT models with the same objective, architecture, preprocessing and 1,000 training steps as the vector model, and we kept the pinball-supervised scalar models of Fig. 2a as a second control. The table reports the RMSE divided by the target standard deviation, as the mean over 30 splits and five training seeds. Bold marks the better of the two matched models.
| Dataset (targets) | scalar models, pinball | scalar models, same objective | Vector POINT |
|---|---|---|---|
| Linnerud (3) | 1.188 | 1.096 | 1.109 |
| Energy efficiency (2) | 0.305 | 0.494 | 0.437 |
| Concrete slump (3) | 0.720 | 0.868 | 0.815 |
| Jura (3) | 0.697 | 0.820 | 0.757 |
| EDM (2) | 0.750 | 0.847 | 0.828 |
| Solar flare (3) | 1.048 | 0.995 | 1.011 |
| Water quality (14) | 0.953 | 0.978 | 0.962 |
| atp1d (6) | 0.487 | 0.641 | 0.501 |
Under the matched objective, the vector model is better on six of eight datasets. In contrast, the pinball objective improves the scalar models on six datasets, so the submitted comparison mixed the objective and the output structure, as you pointed out.
(b) Linnerud protocol. The submitted experiment uses 30 ShuffleSplit splits with a test size of 0.25 and random_state 43. We standardize inputs and targets with the statistics of the training split, and we pretrain the models for 1,000 steps on synthetic episodes only. The submitted numbers come from one training seed. Over five seeds, the vector model reaches an RMSE of 11.258 ± 0.012, against 11.999 ± 0.056 for the pinball scalar models, and it is better in every seed.
(c) Additional datasets. The table above adds seven public datasets with 2 to 14 targets. These small models train on synthetic data only, so they trail Extra Trees on six datasets. In contrast, the pretrained checkpoints of Table 1, applied once per target, beat Extra Trees and Random Forest on five of the eight.
Weaknesses
W1. Source and stability of the gains
(a) Matched retrieval. In the submitted Table 2, POINT received 512 retrieved rows per query, whereas the quantile, histogram and bar heads received the full support. We had used retrieval only for POINT because its readout over a very long context was slow, but we implemented a fused kernel to resolve this. We re-evaluated all four heads in both settings, on the same 132 splits and with the same retrieved rows. The table reports macro nRMSE. Below retrieval results are the mean over three new runs.
Retrieval helps every head, by 0.04 to 0.09 macro nRMSE. Without retrieval, POINT has the lowest error. With retrieval, POINT and the quantile head are level, with 0.4523 against 0.4557, and both are ahead of the histogram and bar heads.
(b) Independent confirmation. We can clarify and qualify the sentence in App. C.1. We selected architecture and training configuration based on held-out synthetic episodes and on one split of each of Airfoil, Concrete and QSAR. Here, we evaluated on another held-out suite of 32 TabArena classification datasets. The table reports the mean and the seed standard deviation over three seeds.
(c) Repeated runs at the larger budget. We trained two more seeds of POINT and of the quantile head at the larger budget. We reports the mean and standard deviation over these.
The two heads are level on all five metrics, within one standard deviation.
(d) Multi-output control. In the new experiment, the scalar models use the same objective as the vector model, with the vector model better on six of eight datasets. Further, the pinball objective alone improves the scalar models on six of eight datasets.
W2. Connection between method, implementation and evidence
(a) Theory and the practical loss. The theory covers POINT-CE alone, as App. A.10 states, and the practical loss adds Smooth L1. We therefore trained the objective that the theory analyses, POINT-CE without Smooth L1, and compared it with the full objective at the small budget of Table 3 (one seed).
POINT-CE alone stays within 0.21 RMSE of the full objective, and it is better on the ten broad datasets (paired difference -0.0125, 95% CI [-0.0189, -0.0063]). The analyzed objective is therefore a complete regression objective in practice. In contrast, Smooth L1 without any POINT-CE raises Concrete RMSE to 6.899 (Q5).
(b) Vector targets outside the hull. The vector teacher needs no projection, because every training target lies inside the hull by construction. We train the vector models on synthetic episodes whose targets are convex combinations of support labels, and we verified this on 16,384 training targets. At test time, the hull limits the prediction instead. We found that the submitted solver did not converge for the 14 targets of water quality. A corrected solver improves this dataset from 0.962 to 0.952 normalized RMSE and leaves the other datasets unchanged (Q4).
(c) Zero reference weights. We add a floor of 10−6 to every Extra-Trees weight and renormalize before the tilt, which is the first fix that you list. The floor is needed for 1% to 5% of the queries on five datasets, and for 58% on QSAR-TID-11, where the trees select about 6 of 3,828 support rows (Q6).
(d) Missing ablations. We report these report these in our response to Q7, together with matched comparisons under the final objective.