acceptodds
Withdrawn from ICLR 2027

TabNSM: Neural Sparse Mixer for Tabular Regression

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern prediction tables increasingly contain hundreds of columns derived from text or large-language-model embeddings, yet only a few matter for any given record. Existing neural models often compare every pair of columns, which becomes costly as tables widen, and models that condition on the training set can exceed practical memory limits. We introduce **TabNSM**, a neural model for regression on these wide tables. Its core *Adaptive Sparse Interaction Module* selects a small, record-specific set of relevant column groups, models their interactions, and combines them with broader table information, so that computation and memory grow linearly rather than quadratically with the number of columns. Across seven highly redundant real-world datasets, TabNSM attains the lowest prediction error on five and a significantly better average rank than each of 41 tuned and pretrained baselines (**rank 1.43 of 42, p ≤ 0.016** against every baseline), with a 21% error reduction on the most interaction-rich table. On the largest tables it trains within a modest GPU-memory budget that does not grow with dataset size, where models that condition on the training set run out of memory or fall back to a small fixed context. Because TabNSM is trained rather than conditioned, its feature attributions can also be validated by removing and retraining on the top-ranked features, which sharply degrades accuracy, unlike attributions from in-context models. An ordinal binning loss (GridLoss) and error-aware sampling (RISE) are secondary training additions. TabNSM is built for the wide, redundant tables that embedding pipelines increasingly produce; tuned trees and in-context models remain ahead on low-dimensional tables and on TabArena.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Tabular regression, deep learninsparse attention, feature interaction, high-dimensional data, adaptive feature selection, ordinal regression, difficulty-aware sampling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RuqYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TabNSM, a neural architecture for high-dimensional tabular regression, particularly targeting datasets with redundant features derived from text embeddings. The core component, the Adaptive Sparse Interaction Module (ASIM), combines instance-adaptive sparse attention with feature mixing to capture relevant feature interactions while reducing computational overhead. The framework further incorporates a multi-stage regression head, an ordinal binning loss (GridLoss), and error-aware sampling (RISE). Experiments on seven wide tabular datasets demonstrate competitive performance against baseline methods, achieving the lowest prediction error on five datasets.

Critical strengths
  1. The paper addresses computational and memory challenges in high-dimensional tabular regression, particularly for large datasets containing redundant embedding-derived features.
  2. The authors compare TabNSM against extensive methods, including traditional machine learning models, deep tabular architectures, and pretrained tabular foundation models. The evaluation covers multiple application domains and datasets containing up to one million samples.
  3. The paper includes component ablations, computational efficiency measurements, feature attribution validation, and robustness experiments.
Critical weaknesses
  1. The paper’s motivation relies on an insufficiently validated assumption of instance-specific feature sparsity. Five of the seven benchmarks derive most of their features from the same sentence-embedding pipeline, where individual dimensions lack inherent semantic meaning. It remains unclear whether the proposed sparsity reflects an intrinsic property of tabular data or an artifact of the embedding representation, limiting the evidence for generalizability.

Notably, Appendix M.7 shows that TabNSM’s performance advantage diminishes substantially when the embeddings are compressed using PCA and augmented with independent noise. The authors also acknowledge that the model’s performance is highly dependent on the structure of the original embedding block. This raises concerns that the proposed architecture may be tailored to a specific embedding representation rather than addressing a general challenge in tabular regression.

  1. The paper motivates TabNSM by highlighting limitations of existing methods, but these limitations are not universal. The quadratic complexity of dense attention applies primarily to attention-based architectures, while methods such as MLPs and GBDTs avoid pairwise feature attention. Moreover, the evaluated datasets contain at most approximately 400 features, making the practical significance of quadratic complexity unclear. Although the authors criticize SSMs for their sensitivity to feature ordering, TabNSM’s sliding-window attention suffers from a similar limitation. Finally, the context-length and memory constraints of tabular foundation models primarily affect in-context learning approaches rather than conventional supervised models. Together, these concerns suggest that the paper’s motivation combines limitations of different model families without sufficiently establishing a unified problem that TabNSM uniquely addresses.
Questions for authors

see above

Reviewer_C3CjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes TabNSM, a neural architecture for tabular regression on high-dimensional datasets with substantial feature redundancy, often including text-derived embeddings. Its core component, the ASIM, tokenizes individual input features and combines attention over local windows, compressed global summaries, and feature blocks adaptively selected for each instance and query. The resulting representations are further processed through Feature-Token Mixing (FTM), residual connections, and a multi-stage regression head to produce the final prediction.

The authors aim to improve predictive performance on highly redundant tabular datasets while reducing computational complexity by modeling interactions among adaptively selected feature blocks rather than relying on dense attention over all feature pairs. The main idea is to focus computation on selected feature interactions, and the authors claim that the computational complexity scales linearly with the number of input features.

Critical strengths
  1. The proposed method is evaluated against a wide range of baseline models and achieves notable improvements on selected datasets. For example, on EVP, TabNSM achieves an RMSE, approximately 20% lower than that of the strongest baseline, FT-Transformer. Furthermore, its GPU memory usage ranges from 477 to 1,621 MB across the three large-scale datasets for which memory consumption was measured. These results demonstrate its practical potential in the evaluated settings.

  2. The authors evaluate gradient-times-input feature attributions by removing the top-ranked features and retraining the model from scratch, using the removal of bottom-ranked and randomly selected features as controls. They also conduct model-weight and label-randomization tests. The results on AQ and EVP suggest that the resulting feature rankings identify information important for prediction, providing additional evidence for the reliability of the model's feature attributions.

Critical weaknesses
  1. The claim of linear computational complexity appears inconsistent with the operations.

-For the O(DdtokL)O(Dd_{tok}L) complexity analysis in Proposition 3.1 and Appendix B.1 to be linear in D,D, the context length LL per query must remain independent of D.D. However, Appendix K specifies that the number of compressed keys is C(D)=⌊D−br⌋,C(D)=\lfloor \frac{D-b}{r} \rfloor, and each query scores all compressed keys before selecting the top-ranked blocks. Explicitly computing these scores therefore requires O(DC(D)dtok)=O(D2dtok/r)O(DC(D)d_{tok})=O(D^2d_{tok}/r) operations, resulting in quadratic complexity when r is fixed. Although aggregation over the selected blocks may scale linearly, this does not eliminate the cost of identifying those blocks.

Furthermore, Eq. (16) in Appendix L defines W3,W4∈RD×D.W_3, W_4 \in \mathbb{R}^{D\times D}. These dense matrix operations require O(D2dtok)O(D^2d_{tok}) computation, and the two matrices alone contain 2D22D^2 parameters. This appears inconsistent with the O(Ddtok)O(Dd_{tok}) complexity attributed to FTM and the approximately linear parameter growth reported in Figure 3.

-Since linear scalability is a central contribution of the paper, these inconsistencies directly affect the overall assessment. If the actual implementation uses fixed-width bottlenecks or a more efficient key-retrieval mechanism, the authors should clarify the implementation details and provide a corresponding complexity analysis.

  1. The novelty over TabNSA and the evidence supporting foreground-based interaction learning are not sufficiently established.

-TabNSA already combines feature tokenization, sparse attention based on compressed representations, block selection, sliding windows, input-dependent gating, and nonlinear feature mixing through TabMixer. Therefore, it is unclear which components of ASIM are inherited from existing methods and which represent new contributions.

-Although TabNSM introduces modifications to the attention--mixer connections, residual blending, and multi-stage regression head, the experiments do not sufficiently isolate the effects of these modifications. Moreover, TabNSA is not included in Table 3. Comparisons with a large number of baselines do not substitute for a direct comparison with the most closely related architecture.

-The authors also argue that instance-specific foreground selection facilitates the learning of meaningful feature interactions. However, the current evidence does not sufficiently support this claim. Proposition 3.2 provides a conditional analysis of uniform aggregation under the assumption that the selected feature set T contains all informative features in S. That is, S⊆TS\subseteq T. This result does not establish that ASIM can identify such a feature set or learn genuinely predictive interactions among the selected features.

Questions for authors
  1. Clarification of the consistency between the implementation details and the complexity analysis.

-How does the overall computational complexity remain linear in DD under fixed sparse hyperparameters, when accounting for the per-query scoring of all compressed keys in Appendix K and the D×DD\times D matrix multiplications in Appendix L?

-Furthermore, it needs to clearly distinguish whether the claimed linear scalability refers to computational time, activation memory, or the number of model parameters.

  1. Clarification of the novelty and additional contributions compared with TabNSA.

-What specific operations or architectural connections distinguish ASIM from the sparse attention-TabMixer combination in TabNSA? Can you demonstrate the effectiveness of these modifications through direct comparisons using identical preprocessing, data splits, and comparable hyperparameter-tuning budgets?

  1. Validation of the feature selection and interaction learning mechanisms.

-ASIM is described as adaptively selecting informative feature blocks for each instance and query and learning interactions among them. However, I believe that the benefit of selecting predictive feature blocks should be distinguished from the benefit of learning nonlinear interactions among the selected features.

-The current comparison between sparse and dense attention may demonstrate the effectiveness of the sparse architecture. Still, it does not provide sufficient evidence that instance-adaptive block selection identifies genuinely predictive feature interactions. In particular, because FTM and the subsequent regression head can also learn nonlinear feature combinations, it is difficult to attribute the observed performance improvements specifically to ASIM's block selection mechanism.

-Are there experimental results comparing learned instance-adaptive block selection with fixed or random block selection under the same sparsity budget? Furthermore, is there direct evidence that the selected blocks capture feature relationships that are important for prediction?

Reviewer_5fxZOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TabNSM, a neural architecture for high-dimensional tabular regression combining adaptive sparse attention, feature-token mixing, a multi-stage regression head, GridLoss, and difficulty-aware sampling. The Authors report the best RMSE on 7 out of 9 datasets. However, results on established and well-maintained benchmarks such as TabArena-Lite are far less competitive.

Critical strengths
  • Experiments are thorough, including 30+ baselines, component ablations, sensitivity analyses, and evaluations across different dataset sizes and feature dimensions.
Critical weaknesses
  • 54 pages.
  • Novelty. Clear similarity to the TabNSA method and TabMixer approach.
  • Narrow scope: The proposed model focuses exclusively on regression tasks, with the Authors emphasizing its advantages on wide, high-dimensional tables with redundant features. This limits the general applicability of the approach.
  • Weak performance on established benchmarks: On TabArena-Lite, TabNSM achieves an average rank of only 55.2 out of 80 methods, even after tuning and ensembling. This is substantially worse than established methods such as LightGBM and CatBoost, raising concerns about the generalizability of the reported improvements.
Questions for authors
  • How do the authors explain the substantial discrepancy between the strong results on their nine selected datasets and the much weaker performance on TabArena-Lite?

  • What additional information does GridLoss capture compared with a suitably transformed MAE or Huber loss?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.