- Rating
- 4
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work proposes a new self-supervised pretraining method for tabular data. Using an approach that mimics in spirit mask reconstruction and contrastive learning, the authors reframe SSL for tabular data as trying to learn the conditional-to-marginal shift. In short, their method consists in splitting a samples with features into two non-overlapping subsets and , and measures how the context changes the probability of existence of , i.e. how different differs from . Since computing both is intractable, the authors use a proxy task that estimates it via a binary classifier: real samples are sampled from the true data distribution p(u,v) and « false » samples are constructed by sampling non-existing pairs from the data, i.e. sampling u from p(u) and v from p(v) independently. A shared encoder between u and v as well as a linear classifier are trained on the task of identifying whether a pair (u,v) is a real pair or a false pair. For the downstream task, only the encoder is kept. The authors evaluate their model on a benchmark of 50 datasets and obtain strong performance in comparison to other tested models.
Critical strengths
- The approach proposed is novel and mitigates identified caveats of existing methods.
- The paper is well structured and easy to follow.
- Experiments are convincing.
- The approach is both empirically and theoretically founded.
Critical weaknesses
- Ablation investigations could be improved (see the question section).
- Some elements require reading the appendix to be fully understood.
Questions for authors
Q1: In the introduction, line 79-80, and are not properly defined until later in the paper.
Q2: Section 5.2 is unclear and could be refined. The authors mention supported pairs and suppressed pairs on line 401 and 402 but never define them beforehand.
Q3: Why did the authors include in eq. (7)? This would be interesting to ablate.
Q4: Have the authors investigated scaling their approach to cross dataset ? One could consider distributions conditioned on a dataset also.
Q5: While the performance look very strong as such, have the authors investigated other setups like training a different encoder between u and v?
Q6: Including the dot product in the representation used for classification might be optimizing similarity between u and v when originating from the true data distribution, and pushing u and v apart when drawn independently. If could interesting to have an analysis of the representation of u and v and their relations.