acceptodds
Under review as a conference paper at ICLR 2027

MVSTab: Bridging Statistical and Semantic Subspaces for Tabular Representation Learning

Abstract

Self-supervised representation learning on tabular data predominantly relies on heuristic feature perturbations or ad-hoc column partitioning. However, these paradigms frequently suffer from a fundamental semantic-statistical gap: they optimize solely for empirical correlations while remaining blind to domain-level semantic structures. To bridge this divide, we propose MVSTab, a multi-view tabular representation learning framework that synergizes empirical statistical patterns with high-level semantic priors. Departing from heuristic masking, MVSTab introduces a dual-driven partitioning strategy that combines language model representations with correlation clustering to disentangle features into complementary semantic and statistical subspaces. To prevent representations from exploiting superficial numerical shortcuts, we design a semantic-aware hard negative generation mechanism. By selectively perturbing intra-group dependencies while preserving single-feature marginal plausibility, this mechanism compels the model to resolve nuanced logical contradictions rather than trivially detectable statistical outliers. Optimized via a tri-objective contrastive loss and aggregated through latent cross-attention, MVSTab produces dense, fixed-dimensional, and logically consistent embeddings. Extensive evaluations across diverse benchmarks show that MVSTab consistently outperforms competitive tree-based and deep learning baselines, especially in label-scarce regimes. Crucially, downstream analyses reveal that MVSTab uniquely excels at uncovering stealthy, logic-driven anomalies that evade conventional density- and reconstruction-based approaches.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Tabular Representation Learning, Self-Supervised Learning, Contrastive Learning, Multi-View Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_FaxsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes MTSTab, a tabular representation learning framework that constructs systematic representations from two complementary views: statistical and semantic. The statistical view performs data-driven column partitioning, while the semantic view leverages semantic information to partition columns rather than relying on heuristic grouping strategies. The paper empirically demonstrates that MTSTab achieves strong performance across several downstream tasks, including anomaly detection, classification, and regression.

Critical strengths
  1. The paper introduces a hard-negative sample synthesis strategy that preserves the marginal distribution of each column. This makes the synthesized samples more difficult to distinguish using shallow linear models.
Critical weaknesses
  1. In the Introduction, the paper claims that the proposed method can provide "human-level" semantic priors. However, this claim requires stronger justification. In particular, the paper should provide a more explicit discussion of why LLM-generated embedding vectors derived from column metadata can be regarded as providing human-level semantic information. Without further evidence or analysis, I believe that this claim is somewhat overstated.

  2. (Major) Sections 3.2, 3.3, and 3.4 introduce a large number of notations without sufficiently explicit definitions in the main text. Examples include TiT_i, wlw_l, LLM(⋅)LLM(\cdot), hvh_v, and even the VICReg loss. This significantly reduces the readability of the methodology section and makes the proposed framework difficult to follow. The paper would benefit from a more systematic definition of the notation and a clearer description of the role of each component.

  3. Section 3.4 provides a relatively detailed description of how hard-negative samples are constructed, whereas the procedure for generating positive augmented representations is described only briefly in a single sentence. The augmentation procedure should be described in greater detail.

  4. In the experimental section, the main reported results focus primarily on anomaly detection among the three downstream tasks considered in the paper. However, since the primary contribution of the paper is a general-purpose tabular representation learning framework, I believe that classification and regression results should receive greater emphasis in the main paper. At present, the experimental focus on anomaly detection baselines makes it difficult to fully evaluate the main representation learning contribution.

Questions for authors
  1. In line 253, K-Means clustering is applied to {E1,…,EME_1,\ldots,E_M}, where the value of MM appears to range from 4 to 52 in the considered datasets. Please clarify how K-Means clustering is applied when the number of available vectors is quite small, and specify the value or selection procedure for the hyperparameter KK.

  2. The loss function contains three hyperparameters, α\alpha, β\beta, and γ\gamma. Please clarify the search spaces considered for these hyperparameters and report the values selected in the experiments. In addition, a sensitivity analysis would be useful for understanding how strongly the performance depends on these hyperparameters.

  3. As I understand it, the final global representation hfusedh_{fused} does not directly appear in the main loss function in Equation (3). Please clarify how hfusedh_{fused} is trained and which objective explicitly extracts the final global representation.

Reviewer_A46KOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MVSTab, a Self-Supervised Learning (SSL)-based tabular representation learning framework integrating statistical feature dependencies with LLM-derived semantic information. The method partitions features into correlation-based statistical and metadata driven semantic subspaces, applies within-view and cross-view contrastive objectives alongside semantic-aware hard negative sampling, and aggregates representations through latent cross-attention. The framework is evaluated across 12 tabular datasets involving anomaly detection, classification, and regression. The authors report competitive performance and claim that the combination of statistical and semantic representations improves feature dependency modeling, robustness, and downstream generalization.

Critical strengths
  • Combining empirical feature correlations with LLM-derived semantic metadata is an interesting direction for tabular representation learning, particularly when statistical dependencies alone are insufficient.
  • The framework incorporates statistical and semantic feature grouping, contrastive objectives, and latent cross-attention within a unified architecture.
  • The experiments cover anomaly detection, classification, and regression across 12 datasets, demonstrating the applicability of the proposed framework to multiple predictive settings.
  • The ablation studies, sensitivity analysis, and synthetic anomaly experiments provide useful insights into several architectural components. The authors also report their GPU model and implementation details.
Critical weaknesses
  • Insufficient positioning against prior feature structuring literature. The first paragraph in Introduction talks about the lack of structural locality in tabular data without any reference. This is a very bold statement, may be the closest reference to this point is GOTabPFN [1]. The proposed statistical and semantic partitioning is closely related to the broader literature on feature grouping, reorganization, and structured representation learning. GOTabPFN [1] is particularly relevant for discussing feature structure-aware representations, although its methodology differs from the proposed approach. The manuscript should better distinguish its technical novelty from these existing directions.

[1]. Habib et al. "GOTabPFN: From Feature Ordering to Compact Tokenization for Tabular Foundation Models on High-Dimensional Data." ICML 2026.

  • Incomplete discussion of related Contrastive Learning-based approaches. In the second paragraph of Introduction and Related Work, the paper talks about creating multi-view in tabular data and considering from contrastive learning perspective. A similar approach has been adopted in ZAYAN [2]. The paper motivates multi-view contrastive tabular representation learning but does not sufficiently distinguish its contributions from recent feature-centric Contrastive Learning-based methods. For instance, ZAYAN [2] investigates Self-Supervised Learning-based feature level Contrastive Learning with dynamic perturbations and masking. A clearer comparison is needed to establish which aspects of MVSTab are genuinely new and which are combinations of existing techniques.

[2]. Habib et al. "ZAYAN: Disentangled Contrastive Transformer for Tabular Remote Sensing Data." ICPR 2026.

  • Unclear research motivation and contribution boundaries. I had to struggle to understand the main goal of the paper. Sometimes it felt like a tabular anomaly detection, sometimes tabular clustering, sometimes classification. The paper alternates between general tabular representation learning, feature partitioning, logical anomaly detection, and downstream supervised prediction without consistently establishing the primary scientific problem. Consequently, the narrative feels fragmented, and it is difficult to identify whether the main contribution lies in semantic partitioning, contrastive learning, or anomaly detection.

  • Outdated framing of modern tabular learning. Related works and baselines are mostly old. Tabular Foundation Models (TFMs) are not discussed there. I quote from the paper (L121-L124), “Without the spatial or sequential structure of images and text, tabular data remains challenging for representation learning, and gradient-boosted decision trees often outperform generic deep networks Speiser et al. (2019); Pasaribu et al. (2024); Shwartz-Ziv & Armon (2022)”. This is relatively older concept now with the rise of TFMs. So the related-work discussion does not adequately reflect recent developments in TFMs. Although TabPFN and TabDPT appear in the manuscript, TFMs are not systematically discussed. Please refer to the TabArena [3] and BeyondArena [4] leaderboards. TabArena [3] and BeyondArena [4] provide a more nuanced perspective on the relative strengths of foundation models, deep learning, and gradient-boosted trees. Furthermore, TabDPT is presented within the Self-Supervised Learning discussion (L145) without clearly identifying its role as a TFM that incorporates Self-Supervised Learning.

[3]. Erickson et al. "TabArena: A Living Benchmark for Machine Learning on Tabular Data." NeurIPS 2025

[4]. Purucker et al. "Beyond IID: How General Are Tabular Foundation Models, Really?." arXiv preprint arXiv:2606.30410 (2026). (tabarena.ai)

  • Theoretical justification does not establish the claimed independence. In Section 3.3 (L255-L256), the paper mentioned, “Theoretical proofs regarding the schema-data independence of this dual partitioning are deferred to Appendix A.” While looking into the Appendix A, I did not find what was actually mentioned in the main paper. However, Appendix A primarily derives independence by assuming that feature metadata is fixed independently of the sampled observations. Under this assumption, the result is largely immediate and does not establish a meaningful orthogonality property between statistical and semantic partitions. The low Jaccard and ARI scores in Table 4 indicate limited partition agreement but do not constitute proof of statistical independence. The theoretical claims therefore appear substantially stronger than their actual justification. The paper makes stronger theoretical claims than its mathematical analysis can adequately support. I don’t know if the authors did this intentionally to trick the reviewers (as often many reviewers do not look into appendices) or it was an AI slop.

  • Insufficient validation of the computational scalability claims. The paper claims to alleviate quadratic attention complexity through feature partitioning. However, statistical partitioning itself requires constructing an M×MM \times M correlation matrix, while the online complexity still contains a sum of squared subspace sizes. Without explicit bounds on individual subspace sizes, the worst-case complexity can remain quadratic. Moreover, the evaluated datasets contain at most 63 features, providing little empirical support for the claimed suitability for high-dimensional tabular learning. In case of high-dimensional datasets e.g. features more than 2K/10K/20K, the computational cost will explode. Please see GOTabPFN [1] for some high-dimensional datasets.

  • Limited experimental evidence for the central semantic contribution. The ablation experiments remove several architectural components, but they do not adequately isolate the contribution of LLM-derived semantic knowledge. Comparisons against randomly assigned semantic groups, shuffled feature metadata, statistical-only partitioning, and alternative feature grouping strategies would be particularly informative. The current results cannot clearly distinguish benefits from semantic knowledge from those arising simply through additional partitioning and representation capacity.

  • Overstated conclusions regarding anomaly detection and reasoning. Although MVSTab achieves the highest anomaly-detection accuracy across 5 reported datasets, it does not consistently achieve the best AUROC. For example, its AUROC on Sleep (0.797), Bank (0.693), and Donors (0.852) is below the strongest competing result. Moreover, the synthetic feature swapping experiments primarily examine violations of observed feature dependencies. These results do not sufficiently establish broader claims regarding logical reasoning, semantic consistency, or detection of logically contradictory samples.

  • Limited statistical and experimental rigor. Results are averaged over only 4 random seeds, and the main performance tables do not provide standard deviations, confidence intervals, or significance tests. Additionally, random 80/20 splits on small datasets, including datasets with only a few hundred samples, raise concerns about the stability and generalizability of the reported improvements.

Questions for authors

Please see the points raised in Critical Weaknesses and refer to the following questions:

  1. What is the main technical novelty of the proposed feature partitioning relative to existing structure-aware feature ordering methods such as GO-LR in GOTabPFN [1]?
  2. How does MVSTab fundamentally differ from existing feature centric Self-Supervised Learning-based contrastive approaches such as ZAYAN [2]?
  3. What is the primary scientific objective of the paper: general-purpose representation learning or semantic-aware anomaly detection? Can the authors clarify how the different contributions support this objective?
  4. Why are recent TFMs and benchmarking studies, including TabArena [3] and BeyondArena [4], not adequately discussed? Can the authors clarify the categorization of TabDPT?
  5. How does Appendix A establish a nontrivial independence property between statistical and semantic partitions beyond the assumption of fixed metadata?
  6. How does the proposed framework avoid quadratic complexity in the worst case, particularly when global feature views or individual statistical subspaces contain many features?
  7. Can the authors provide ablations using shuffled metadata, random semantic partitions, and statistical-only views to isolate the actual contribution of semantic knowledge?
  8. Why are the scalability claims not validated on genuinely high-dimensional datasets containing thousands of features?
  9. Can the authors provide uncertainty estimates and clarify why the observed robustness to synthetic feature corruption constitutes evidence of semantic or logical reasoning?
Reviewer_DCWBOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a representation learning architecture for self-supervised learning of tabular embeddings by mixing semantic priors from LLMs with statistical properties of the table. The authors evaluate the proposed method on a small set of tabular datasets and show that the proposed method is competitive with existing methods for anomaly detection, classification, and regression on those datasets.

Critical strengths

The authors propose a novel architecture to handle features in different groups based on their semantic and statistical properties. The idea of using LLMs to come up with such groups seems to be a new application.

Critical weaknesses

The current suite of evaluations is quite small compared to contemporary works in tabular learning. The authors should consider widely accepted tabular benchmarks for learning such as OpenML-CC18 or TabArena/BeyondArena for classification/regression, or TEmBed for dataset representation. 12 tables is not enough to draw any meaningful conclusions about the proposed approach. The experiment section is also quite difficult to follow, with critical details about the evaluation suite referred to in the appendix. The authors also make some claims that are fully supported by the results I can see in the main manuscript. The presentation of the results can use some improvement as well. Not enough context is given on how the results should be interpreted or what they even mean (figure 3).

Questions for authors
  • can you expand a bit more on this claim in 4.2:

    while our multi-view latent representation provides a highly separable global space for general anomaly thresholding, the continuous distribution ranking evaluated by AUROC might occasionally favor algorithms explicitly optimized for local density estimation or feature reconstruction. This depends on the intrinsic data manifold.

    • how exactly does it depend on the intinsic data manifold? you can't just say this without providing further intuition or example.
  • Is the proposed method specifically designed for anomaly detection only? That's not how the initial framing of the paper lead me to believe, but the majority of the evaluation in the main paper is on anomaly detection. Where are the classification/regression results referred to in 4.1 ?
  • what competitive tree-based and deep learning baselines is the method outperforming? I see that there are evaluations with 4 datasets for regression and classification each in the appendix. I don't believe these results were discussed at all in the main paper, so that seems like a superfluous claim to make in the abstract.
  • can you run this method on one of the more widely accepted tabular benchmarks?
  • Why does a larger IF-shift indicate harder negative samples rather than samples that are more easily detected by Isolation Forest?
  • What is an anomaly score multiplier? What is Figure 3 showing? What are the groups? What am I supposed to take away from this?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.