Predict by Pointing: A Task-Agnostic Output Head for Tabular In-Context Learning

Tabular foundation models learn in-context: given a labeled support set, they predict unlabeled queries in a single forward pass. Simple point losses extend naturally to vector-valued targets but are often less competitive than distributional regression objectives, whose heads are typically scalar-specific. Consequently, there is no common prediction head spanning support-adaptive classification, strong scalar regression, and native multi-output regression. In this work, we introduce Prediction Over In-context Targets (POINT), a task-agnostic output head that instead treats the labeled support set itself as the prediction space. POINT assigns probability directly to the support set; classification probabilities are obtained by summing mass over examples of the same class, while regression predictions are obtained by taking the expectation of continuous support labels. The output space therefore adapts to the context rather than being fixed by the task. To handle this new space, POINT is trained with POINT-CE, a support-distribution cross-entropy objective to handle both classification and regression simultaneously. We further show that, under support coverage, minimizing POINT-CE in expectation recovers conditional class probabilities for classification and the conditional mean for regression. This connects the same support-level objective to the quantities targeted by categorical likelihoods and distributional regression. Across tabular in-context-learning experiments, POINT achieves competitive performance on both classification and regression while retaining the same support-based prediction principle. Experiment results suggest that predicting by pointing to labels already present in context can provide a common prediction head for tabular models.