acceptodds
Under review as a conference paper at ICLR 2027

Conditional Completion Ranking for Tabular Self-Supervised Learning

Abstract

Self-supervised learning (SSL) for tabular data aims to capture dependencies among heterogeneous features. However, existing reconstruction and view-alignment objectives often emphasize target prediction or instance-level invariance, which may not fully capture the complex, multi-modal relationships among tabular features. We instead formulate tabular SSL as learning the *conditional-to-marginal shift*, which measures how the probability of a multi-column candidate completion changes after conditioning on the observed features relative to its marginal probability. Based on this formulation, we propose Conditional Completion Ranking (CCR), which distinguishes samples from the joint distribution from samples drawn from the product of marginals. We show that the optimal CCR score recovers the log conditional-to-marginal density ratio, without explicitly modeling the full conditional distribution. Experiments on 50 datasets from two benchmarks show that CCR achieves the best average rank in the few-shot settings under both frozen and fine-tuned evaluation. The code is available in the supplementary materials.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, tabular learning, density-ratio estimation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_qRmNOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes a new self-supervised pretraining method for tabular data. Using an approach that mimics in spirit mask reconstruction and contrastive learning, the authors reframe SSL for tabular data as trying to learn the conditional-to-marginal shift. In short, their method consists in splitting a samples with dd features into two non-overlapping subsets uu and vv, and measures how the context uu changes the probability of existence of vv, i.e. how different p(v∣u)p(v\mid u) differs from p(v)p(v). Since computing both is intractable, the authors use a proxy task that estimates it via a binary classifier: real samples are sampled from the true data distribution p(u,v) and « false » samples are constructed by sampling non-existing pairs from the data, i.e. sampling u from p(u) and v from p(v) independently. A shared encoder between u and v as well as a linear classifier are trained on the task of identifying whether a pair (u,v) is a real pair or a false pair. For the downstream task, only the encoder is kept. The authors evaluate their model on a benchmark of 50 datasets and obtain strong performance in comparison to other tested models.

Critical strengths
  • The approach proposed is novel and mitigates identified caveats of existing methods.
  • The paper is well structured and easy to follow.
  • Experiments are convincing.
  • The approach is both empirically and theoretically founded.
Critical weaknesses
  • Ablation investigations could be improved (see the question section).
  • Some elements require reading the appendix to be fully understood.
Questions for authors

Q1: In the introduction, line 79-80, MM and Mˉ\bar{M} are not properly defined until later in the paper.

Q2: Section 5.2 is unclear and could be refined. The authors mention supported pairs and suppressed pairs on line 401 and 402 but never define them beforehand.

Q3: Why did the authors include hu⋅hvh_u \cdot h_v in eq. (7)? This would be interesting to ablate.

Q4: Have the authors investigated scaling their approach to cross dataset ? One could consider distributions conditioned on a dataset also.

Q5: While the performance look very strong as such, have the authors investigated other setups like training a different encoder between u and v?

Q6: Including the dot product in the representation used for classification might be optimizing similarity between u and v when originating from the true data distribution, and pushing u and v apart when drawn independently. If could interesting to have an analysis of the representation of u and v and their relations.

Reviewer_ykVZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes CCR, a self-supervised learning method for tabular data that formulates tabular SSL as a conditional-to-marginal shift learning problem.

CCR learns the dependency between feature subsets by performing binary classification on context-completion pairs from the same row versus completions sampled from other rows. The authors show that the optimal scoring function corresponds to the conditional-to-marginal log-density ratio. The learned representations are then used for downstream classification tasks.

Experiments on 50 tabular datasets show that CCR achieves the best average rank among the evaluated tabular SSL methods in few-shot settings. Additional synthetic experiments validate the learned scores, while ablation studies demonstrate the effectiveness of random masking and the proposed training objective.

Critical strengths

1. Simple and effective method. CCR is a simple yet effective method, and the formulation of tabular SSL as a conditional-to-marginal shift problem is also well-defined.

2. Comprehensive and Strong results. The experiment is comprehensive, with 50 evaluation datasets showing CCR achieves the best average rank in few shot setting compare to previous tabular SSL and TabFM methods in few-shot setting. The ablations also cover most of the important design choices. The result convinces me that CCR is a strong methods for tabular SSL.

Critical weaknesses

1. Moderate Novelty. The method itself appears closely related to a single-table variation of Self-VPCL [1], as both use a similar feature-partition contrastive construction. While CCR introduces a different masking strategy and loss formulation, I am not fully convinced that these changes provide enough methodological novelty.

2. Practical Advantage over TFM. With the recent success of tabular foundation models (TFM), the practical value and the real performance of per-table tabular SSL is still unclear to me. Although CCR outperforms TabPFN and TabICL in the few-shot setting, CCR additionally uses the full unlabeled target-table training set and requires table pre-training. Therefore, the current comparison does not fully convince me that CCR would outperform modern TFMs under a comparable target-data and adaptation budget. Recent work such as LoopTabFM [2] shows that TabPFN can also exploit unlabeled target examples through a semi-supervised procedure. A comparison with such methods would make the practical advantage of CCR more convincing.


[1] Z. Wang and J. Sun, "TransTab: Learning Transferable Tabular Transformers Across Tables," Advances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022.

[2] Y. Li, X. Chang, M. Kara, X. Liu, A. K. Roy-Chowdhury, and S. Oymak, "When and How Unlabeled Data Provably Improve In-Context Learning," Advances in Neural Information Processing Systems (NeurIPS), vol. 38, 2025.

Questions for authors

Major

  1. Could the authors clarify the main differences between CCR and TransTab's Self-VPCL [1], especially regarding the masking strategy and loss function? Although TransTab focuses on cross-table learning, its Self-VPCL objective seems applicable to single-table SSL. The fixed-mask ablation in Sec. 5.3 does not seem to cover Self-VPCL-style vertical partitioning. A comparison using the samearchitecture and loss, but with Self-VPCL-style partitioning, would help clarify the benefit of CCR's masking strategy.

  2. It would be nice to provide an experiment like LoopTabFM or any semi-supervised TFM methods according to weakness 2.

Minor

  1. I think D.6 as an important ablation study to prove the training work under the same architecture with different learning strategy, putting in the main paper will make the claim better, or at least mention it in the main paper.
Reviewer_LkqBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Conditional Completion Ranking (CCR), a self-supervised framework for learning the conditional-to-marginal shift in tabular data. Given an observed context, CCR compares its observed completion with alternatives sampled jointly from the marginal completion distribution. It distinguishes samples from the joint distribution from samples drawn from the product of marginals, and the optimal score recovers the conditional-to-marginal log-density ratio. Experiments on OpenML-CC18 and TabArena show that CCR achieves the best average rank in few-shot settings under both frozen-representation evaluation and supervised fine-tuning.

Critical strengths
  1. The objective has a clear statistical interpretation. Sampling completion features jointly preserves dependencies within the target subset. The conditional-to-marginal formulation then provides a principled way to learn context-dependent compatibility while allowing multiple completions to remain plausible.
  2. The empirical evaluation is thorough. CCR is evaluated under frozen-representation evaluation and supervised fine-tuning, with matched runs and dataset-level significance tests. The Random Encoder and Scratch controls, together with fixed-backbone objective comparisons, provide useful evidence that pretraining contributes to the reported gain
Critical weaknesses
  1. The contribution specific to the CCR objective is less clear than the overall benchmark improvement. The density-ratio characterization follows the standard binary discrimination argument discussed in Related Work. The closest matched alternative, block InfoNCE, performs similarly, but its comparison reports only aggregate means. This leaves limited evidence for the importance of binary cross-entropy beyond the context–completion construction itself.

  2. The probability-shift interpretation is better validated on synthetic data than on real tables. The theoretical result concerns a population-optimal score, while the recovery experiments use simple synthetic distributions. The Adult examples do not compare learned scores with estimated conditional and marginal probabilities. Consequently, interpreting their sign and magnitude as feature-value attribution remains insufficiently validated.

  3. The benchmark selection is not fully specified. Datasets with an “excessively low sample-to-feature ratio” are excluded without an explicit cutoff. This criterion could affect coverage of high-dimensional settings, where the paper acknowledges difficulties in density-ratio estimation. The exact selection rule and excluded datasets are needed to assess the scope and reproducibility of the benchmark.

Questions for authors
  1. Can the authors provide dataset-level paired differences and uncertainty estimates for CCR versus block InfoNCE? Are the improvements consistent across datasets?
  2. How are the feature-value attribution claims validated beyond the synthetic experiments and selected Adult profiles? Is there evidence that learned scores remain calibrated as conditional-to-marginal probability shifts on real tables?
  3. What exact sample-to-feature ratio cutoff was used, and which datasets were excluded because of this criterion?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.