- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TabNSM, a neural architecture for high-dimensional tabular regression, particularly targeting datasets with redundant features derived from text embeddings. The core component, the Adaptive Sparse Interaction Module (ASIM), combines instance-adaptive sparse attention with feature mixing to capture relevant feature interactions while reducing computational overhead. The framework further incorporates a multi-stage regression head, an ordinal binning loss (GridLoss), and error-aware sampling (RISE). Experiments on seven wide tabular datasets demonstrate competitive performance against baseline methods, achieving the lowest prediction error on five datasets.
Critical strengths
- The paper addresses computational and memory challenges in high-dimensional tabular regression, particularly for large datasets containing redundant embedding-derived features.
- The authors compare TabNSM against extensive methods, including traditional machine learning models, deep tabular architectures, and pretrained tabular foundation models. The evaluation covers multiple application domains and datasets containing up to one million samples.
- The paper includes component ablations, computational efficiency measurements, feature attribution validation, and robustness experiments.
Critical weaknesses
- The paper’s motivation relies on an insufficiently validated assumption of instance-specific feature sparsity. Five of the seven benchmarks derive most of their features from the same sentence-embedding pipeline, where individual dimensions lack inherent semantic meaning. It remains unclear whether the proposed sparsity reflects an intrinsic property of tabular data or an artifact of the embedding representation, limiting the evidence for generalizability.
Notably, Appendix M.7 shows that TabNSM’s performance advantage diminishes substantially when the embeddings are compressed using PCA and augmented with independent noise. The authors also acknowledge that the model’s performance is highly dependent on the structure of the original embedding block. This raises concerns that the proposed architecture may be tailored to a specific embedding representation rather than addressing a general challenge in tabular regression.
- The paper motivates TabNSM by highlighting limitations of existing methods, but these limitations are not universal. The quadratic complexity of dense attention applies primarily to attention-based architectures, while methods such as MLPs and GBDTs avoid pairwise feature attention. Moreover, the evaluated datasets contain at most approximately 400 features, making the practical significance of quadratic complexity unclear. Although the authors criticize SSMs for their sensitivity to feature ordering, TabNSM’s sliding-window attention suffers from a similar limitation. Finally, the context-length and memory constraints of tabular foundation models primarily affect in-context learning approaches rather than conventional supervised models. Together, these concerns suggest that the paper’s motivation combines limitations of different model families without sufficiently establishing a unified problem that TabNSM uniquely addresses.
Questions for authors
see above