acceptodds
Under review as a conference paper at ICLR 2027

Hyperbolic Manifold Constrained Tabular Neural Network

Abstract

Tabular prediction is central to a wide range of real-world applications. Tabular data typically contain heterogeneous features as well as rich and complex relational information that can imply a latent structural manifold. Hyperbolic geometry can help capture complex structural relations in data. However, most existing tabular prediction models are constructed and optimized in Euclidean space. How to incorporate hyperbolic geometry into supervised tabular learning remains underexplored. We propose HTNN, a supervised hyperbolic manifold constrained tabular neural network for tabular prediction. HTNN consists of a hyperbolic feature-value representation layer for heterogeneous categorical and numerical features, followed by a conventional MLP predictor. HTNN employs a geometry-aware training and geometry-free inference optimization framework. The geometry-aware training allows hyperbolic geometry to shape the latent representation learning of heterogeneous feature values. After training, the latent hyperbolic representations can be converted into ordinary Euclidean space for efficient geometry-free inference. We conducted extensive experiments on the TALENT benchmark. HTNN ranks first among 36 methods on 200 classification datasets and third among 34 methods on 100 regression datasets. Experimental results show that the proposed hyperbolic manifold constrained tabular neural network is effective.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Tabular Prediction, Manifold Geometry

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_R8hdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

HTNN is a tabular network. Each categorical value gets a learnable point on a Lorentz hyperboloid (κ = 1, fixed). Each numerical feature j gets m_j hyperbolic points, and their tangent-space readouts at the origin are combined with coefficients that are affine in the input value, α_j(x) = a_j x + b_j. m_j is set by a label-free entropy/sample-size rule (Eq. 7). The tangent readouts are concatenated and fed to an MLP. Training uses RiemannianAdam for the manifold parameters and AdamW for the rest, in float64. After training the model converts exactly to Euclidean form: cached categorical embedding vectors, a linear numerical embedding u_j x + c_j per feature, and the MLP. The paper argues that the hyperbolic metric acts on training by preserving radial updates and damping transverse ones (Eq. 14, App. A). On the full TALENT benchmark it reports mean rank 1/36 on 200 classification datasets (4.49, against 9.60 for TabPFN-v3) and 3/34 on 100 regression datasets. TALENT-tiny ablations cover geometry, the two pathways, the parameterisation, the coefficient maps and the capacity rule, all significant after Holm correction.

Critical strengths
  1. Well-conceived, technically advanced construction with exact geometry-free inference. The idea is not fully original (it combines existing hyperbolic embeddings with FT-Transformer-style numerical tokenizers), but the construction is careful and advanced. Sec. 3.4 / Eq. 10 show that the trained model collapses exactly to Euclidean lookups, linear numerical embeddings and an MLP (max |Δ| = 5.5e-17, Sec. 4.5). The deployed model is therefore a standard, cheap architecture, and any gain is attributable purely to the training parameterisation. That makes for a clean scientific question.
  2. Scale and reporting of the evaluation. Full TALENT (300 datasets), 15 seeds, per-dataset results (Table 6), complete W/T/L and Holm-adjusted Wilcoxon tests (Table 4), and a stated provenance for every baseline (App. C.2). If the numbers hold up, a result this broad would be significant.
Critical weaknesses
  1. The "hyperbolic geometry" is a training-time preconditioner, and the paper's framing and ablations do not separate it from simpler explanations.

    • log_o is a global diffeomorphism H^d → R^d. So the categorical pathway is a Euclidean embedding table whose gradient is preconditioned by P_H(v) (Eq. 14).
    • The numerical pathway is a linear embedding u_j x + c_j (the FT-Transformer tokenizer) trained through an overparameterised factorisation B_j = Σ_k v_{j,k} β_{j,k}^T (Eq. 16), plus the same preconditioner.
    • Nothing about hierarchy or latent manifold structure in the data is used or tested: no δ-hyperbolicity estimate, no analysis of the learned embeddings, no curvature sweep.
    • The ablation that is meant to isolate geometry (HTNN-Euc, Table 1) changes the optimizer too: RiemannianAdam vs AdamW, with different weight-decay semantics. HTNN also appears to get a separately tuned "Riemannian learning rate" for its embeddings (Table 3) that the Euclidean variants do not. App. F admits that the trajectory comparison "evaluates the realized end-to-end updates of the two training procedures rather than geometry under an identical optimizer."
    • Missing controls that would make the claim testable:
      • Euclidean embeddings with the same P_H(v) preconditioner;
      • a Euclidean variant with its own embedding learning rate;
      • a κ sweep (κ → ∞ recovers Euclidean).
    • More fundamentally, the paper gives no argument, theoretical or even narrative, for why an a-priori fixed, constant-curvature hyperbolic embedding should be superior. The embeddings feed a nonlinear MLP, which absorbs any smooth invertible reparameterisation of the embedding space; indeed the trained model is exactly Euclidean (Eq. 10). Any benefit must therefore be an optimisation effect, which needs comparison against Euclidean alternatives (separate embedding learning rate, norm constraints, embedding weight decay). Table 1 itself suggests the geometry is not the main driver: HTNN-Softmax/Sigmoid keep the Lorentz geometry yet rank below HTNN-Euc (6.65/6.97 vs 5.68 classification; 7.93/8.20 vs 5.53 regression).
    • Without these, the conclusion "negative curvature [is] the effective geometric choice" (Sec. 4.3) is not supported.
  2. I am not convinced by the results. Their size and uniformity are implausible given the controlled evidence and the baselines, and suggest selective reporting or an evaluation asymmetry. Reporting only favourable results is common in this area, but here it appears to be the case to an unusual degree:

    • (a) The inference model is a plain MLP with linear numerical embeddings, yet it beats TabPFN-v3, TabICL-v2, RealMLP, TabR and MLP-PLR with large mean-rank margins (Table 5). On classification it is significant against every baseline.
    • (b) TabM, which the authors ran themselves (App. C.2), comes 25th in classification (21.55) and 22nd in regression, behind plain MLP in both (Table 5). This contradicts the TabM paper and other independent benchmarks, and suggests the author-run pipelines are under-tuned or misconfigured. xRFM, also author-run, deserves the same check.
    • (c) In Tables 7–9, adaptive HTNN never loses to any of 8 architectural variants or 24 fixed-capacity variants (e.g. 28/2/0 against nearly every variant; regression mean rank 1.00 ± 0.00 against all 24 fixed-m variants). App. A shows that every m ≥ 2 has the same inference function class, so m only affects optimisation. Zero losses across this many near-equivalent variants is far outside what seed and HPO noise usually produce.
    • (d) HTNN runs in float64 with 100 Optuna trials per dataset. The precision of the TALENT-record baselines, and whether their HPO budget matches, are not stated.
    • The authors should explain (c), re-run TabM/xRFM with their official configurations, and release code and per-dataset scores for all variants.
Questions for authors
  1. [Clarification] In HTNN-Euc / HTNN-CENL, do the Euclidean embeddings share the MLP's AdamW learning rate and weight decay, or get their own tuned learning rate as HTNN's manifold parameters do? Is weight decay applied to the Lorentz parameters, and how?
  2. [Clarification] Please report a Euclidean-embedding model trained with the preconditioner P_H(v) applied explicitly (same optimizer family), and a curvature sweep κ ∈ {0.1, 1, 10, 100}. If the gain persists without the manifold, or vanishes smoothly as κ grows, that decides what the paper is about.
  3. [Clarification] TabM ranks below plain MLP on both tasks. Which TabM variant, embeddings, HPO space and number of trials were used? Are the results consistent with the TabM authors' reported TALENT/other-benchmark numbers?
  4. [Clarification] How can adaptive m_j beat fixed m (for every m ∈ {2..12}) on every non-tied dataset, given that the inference class is identical for m ≥ 2? On datasets where Eq. 7 assigns almost all features the same m, does adaptive HTNN tie with that fixed-m variant?
  5. [Clarification] Were the baselines taken from TALENT records trained in float32? How do HTNN's results change in float32 (this also bears on the fitting-time cost in Sec. 4.5)?
  6. [Clarification] Is model selection (HPO, early stopping, choice among the 15 runs) done purely on validation, identically for HTNN, the variants and the author-run baselines? Will code be released?
  7. [Clarification] Does the network learn the geometry of the data, and if so, how? The motivation (Sec. 1) is that tabular data "imply latent manifold structure" that hyperbolic space can capture. But the curvature is fixed (−1), log_o is a global diffeomorphism, and the trained model collapses exactly to a Euclidean MLP, so the geometry adds no representational capacity and acts only through the training dynamics (Eq. 14). Please provide evidence that the learned representations reflect structure in the data: e.g. δ-hyperbolicity of the datasets versus HTNN's gain over HTNN-Euc, an analysis or visualisation of the learned HFV embeddings (radial ordering vs. category frequency or hierarchy), or a learned or swept curvature.
2 comments

Part 1/2

Thank you for your detailed technical review and for recognizing the strengths of our contribution. We conducted additional experiments and analyses to address your seven clarification questions and related concerns below.

Q1. “In HTNN-Euc / HTNN-CENL, do the Euclidean embeddings share the MLP’s AdamW learning rate and weight decay, or get their own tuned learning rate as HTNN’s manifold parameters do? Is weight decay applied to the Lorentz parameters, and how?”

  • Yes. The Euclidean representations in HTNN-Euc and HTNN-CENL share the MLP’s AdamW learning rate and weight decay. (§3.3, Table 3, and Appendix E)
  • Yes. Weight decay is applied to the Lorentz parameters through Geoopt’s RiemannianAdam, and these parameters use the separately tuned Riemannian learning rate. (§3.3 and Table 3)

Q2. “Please report a Euclidean-embedding model trained with the preconditioner PH(v)P_{\mathrm H}(\mathbf v) applied explicitly [...] and a curvature sweep κ∈0.1,1,10,100\kappa\in{0.1,1,10,100}.”

  • We conducted an additional experiment by explicitly applying PH(v)P_{\mathrm H}(\mathbf v) in Eq. (14) to HTNN-Euc under its AdamW training procedure.

    ConfigurationClassification mean rankRegression mean rank
    HTNN1.46671.4000
    HTNN-Euc + PH(v)P_{\mathrm H}(\mathbf v)1.73331.6667
    HTNN-Sphere3.35003.2667
    HTNN-Euc3.45003.6667

    We notice that explicitly applying PH(v)P_{\mathrm H}(\mathbf v) recovers most of HTNN’s improvement over HTNN-Euc. However, full HTNN remains the best, reflecting the additional contribution of geometry-consistent mechanisms such as Riemannian momentum accumulation.

  • We added a new curvature-sweep experiment over κ∈{0.1,1,10,100}\kappa\in\{0.1,1,10,100\}, corresponding to curvatures −10-10, −1-1, −0.1-0.1, and −0.01-0.01.

    ConfigurationCurvatureClassification mean rankRegression mean rank
    κ=0.1\kappa=0.1−10-103.18333.5333
    κ=1\kappa=1 (HTNN)−1-11.35001.1333
    κ=10\kappa=10−0.1-0.12.80002.9333
    κ=100\kappa=100−0.01-0.013.11672.5333
    HTNN-Euc004.55004.8667

    We notice that curvature −1-1 ranks first on both tasks and outperforms every alternative. At the same radius r=1r=1, Eq. (14) predicts that the four negative curvatures retain approximately 7.2%, 72.4%, 96.7%, and 99.7% of the transverse gradient, respectively. Performance declines when transverse attenuation is either stronger or weaker than the moderate attenuation produced by curvature −1-1.

Q3. “TabM ranks below plain MLP on both tasks. Which TabM variant, embeddings, HPO space and number of trials were used? [...]”

  • We evaluated TabM and xRFM using TALENT’s standard commands and complete pipeline at commit 08301d670a7c854bcf3a73298763484ba58eecdb, including its model variants, embeddings, HPO spaces, and budget of 100 trials per dataset (Appendix C.2). We did so because the TabM paper does not report TALENT results, while the xRFM paper reports only aggregate TALENT results without the per-dataset scores required for our ranking and statistical analyses.

Q4. “How can adaptive mjm_j beat fixed mm [...] given that the inference class is identical for m≥2m\ge2? On datasets where Eq. 7 assigns almost all features the same mm, does adaptive HTNN tie with that fixed-mm variant?”

  • Sharing the same inference function class does not imply matched training procedures for adaptive and fixed capacity. Although all configurations with m≥2m\ge2 fold into the same class of affine numerical mappings, mm changes the HFV factorization and its optimization trajectory.
  • Eq. (7) assigns nearly the same mm to all numerical features on only a small number of datasets. Even in these cases, the adaptive and fixed-capacity models undergo independent hyperparameter searches and select different hyperparameters. They should produce similar results only when both their capacity assignments and hyperparameters are identical. Therefore, ties between adaptive HTNN and the fixed-mm variants are not expected in our evaluation.

Part 2/2

Q5. “Were the baselines taken from TALENT records trained in float32? How do HTNN’s results change in float32 [...]?”

  • We use the baseline scores directly from the TALENT benchmark leaderboard (Appendix C.2). The actual training precision of the historical TALENT records is unknown to us.
  • Training HTNN in FP32 is numerically unstable. Geoopt’s official Lorentz documentation explicitly recommends working with this manifold in double precision (§4.1, Geoopt Lorentz documentation).
  • We confirmed that HTNN and TALENT’s trainable baselines use the same budget of 100 HPO trials per dataset. HTNN’s search space is reported in Table 3.

Q6. “Is model selection [...] done purely on validation, identically for HTNN, the variants and the author-run baselines? Will code be released?”

  • Yes. Model selection uses only validation data. We use the TALENT benchmark splits, select configurations and stopping points using validation data, and average the test scores from all 15 runs rather than selecting among them. (§4.1 and Appendices C.1–C.3)
  • Yes. The code is already available in the anonymous repository linked in the supplementary material. The official GitHub repository link will be added to the main text in the camera-ready version.

Q7. “Does the network learn the geometry of the data, and if so, how? [...] Please provide evidence that the learned representations reflect structure in the data: e.g. δ\delta-hyperbolicity of the datasets versus HTNN’s gain over HTNN-Euc [...] or a learned or swept curvature.”

  • We performed an additional analysis and observed that HTNN generally gains more over HTNN-Euc on datasets with more tree-like inputs. On TALENT-tiny, the Spearman correlations between training-input Gromov δ\delta and performance gain are −0.28 for classification and −0.45 for regression. Since a lower Gromov δ\delta indicates stronger tree-likeness, these negative correlations mean that more tree-like inputs correspond to larger gains from negative-curvature training. We add the corresponding classification and regression scatter plots to the revised PDF.
  • Two other possible approaches are: (1) visualizing the learned representations and (2) comparing them against a ground-truth task decision system. Both approaches have limitations. For (1), visualization lacks a quantitative criterion. Representations learned by successfully trained models commonly exhibit some degree of clustering or apparent hierarchy, so visual inspection alone cannot determine which structure is more faithful. For (2), direct comparison requires a reliable ground-truth task decision system, which is unknown for these tasks. Even if a tree model is used as a reference, aligning continuous DNN behavior with discrete tree behavior remains difficult. We therefore do not claim that HTNN recovers a unique ground-truth hierarchy. Our mechanistic claim concerns how negative curvature shapes optimization.

Clarification: The meaning of HTNN-Softmax and HTNN-Sigmoid ablations.

  • These variants replace unconstrained affine coefficients with nonnegative, bounded coefficients. Softmax additionally requires the coefficients to sum to one. This changes how numerical representations are composed. The direct geometric comparisons are HTNN-Euc, HTNN-Sphere, and the curvature sweep. The explicit PH(v)P_{\mathrm H}(\mathbf v) control separately tests the induced optimization mechanism. Retaining Lorentz representations does not imply that arbitrary coefficient-composition rules preserve performance. (§4.3 and Appendix E)

We hope these updates resolve the main weaknesses raised in your review. We are happy to discuss any further details during the discussion period.

Reviewer_asBLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HTNN, a tabular neural network with categorical and numerical feature-value representations learned on a Lorentz manifold and read out in tangent coordinates by an MLP. Numerical features combine multiple representations using learned affine coefficients, with feature-wise capacity selected from training-data entropy. The analysis characterizes the resulting optimization metric and shows that numerical representations remain affine functions of their inputs. After training, manifold operations can be replaced exactly by categorical lookup vectors and numerical affine maps. HTNN ranks first for classification and third for regression on the 300-dataset TALENT benchmark, with further studies of geometry, capacity, optimization dynamics, and inference efficiency.

Critical strengths
  1. The breadth and consistency of the predictive results are compelling. The evaluation covers 200 classification and 100 regression datasets, with 15 runs per model–dataset pair. Table 4 reports 192/0/8 wins/ties/losses against MLP for classification and 98/2/0 for regression. For an MLP-style inference model, this is a striking result that warrants closer investigation of training geometry.
  2. The paper connects its design to both mechanism and deployment. Geometry and capacity controls are supplemented by local-response and trajectory measurements. The exact inference conversion preserves predictions to a reported maximum absolute difference of 5.47 × 10^-17 and gives median speedups of 1.87× and 1.84× over the original HTNN for classification and regression. This separates the cost of learning the representation from the cost of using it.
Critical weaknesses
  1. The mechanism is better supported at the level of optimization dynamics than at the level of learned structure or generalization. The paper does not directly demonstrate recovery of a latent hierarchy. Appendix F verifies the metric response and reduced transverse motion, but does not establish that this reduction causes the predictive gains. The different optimizers in the HTNN/HTNN-Euc trajectory comparison leave room for alternative explanations. This limits the explanatory claim, rather than invalidating the method.
  2. The large gains over conventional MLPs need a more tightly matched reference and clearer cost accounting. HTNN combines a different parameterization, two optimizers, double precision, and 100 tuning trials per dataset, while many baseline scores come from released TALENT records. Figure 4 also places HTNN at the highest median downstream fitting cost among the displayed methods. The current evidence does not fully separate the geometric benefit from training-recipe differences or establish its value under a fixed total compute budget.
Questions for authors

Clarifications relevant to the assessment:

  1. Do you have direct evidence that HTNN recovers meaningful hierarchical or relational structure in the feature representations? If not, would an optimization-metric interpretation be more precise than the latent-structure motivation? A visualization alone would be less informative than a quantitative structural test.
  2. Appendix F establishes the local response and reduced transverse updates. Is there evidence that this reduction explains the accuracy gains, such as a controlled tangent-space implementation of the induced metric or a transverse-damping intervention with matched optimizer state and regularization? How much of the gain survives such a control?
  3. The MLP comparison: Given the 192/0/8 and 98/2/0 results, can you document the differences in preprocessing, predictor capacity, HPO budget, precision, and optimization between HTNN and the released MLP baseline? Beyond the existing HTNN-Euc ablation, what do the closest matched comparisons show about the size of the geometric benefit?
  4. Training cost: Including HPO, how does the accuracy–compute trade-off change with fewer search trials or lower precision? Given the slower fitting and faster folded inference, in what retraining and prediction-volume regime is the additional training cost worthwhile?

Broader directions, not additional requirements for acceptance:

  1. Which parts of the HFV parameterization depend on using an MLP? Could the same layer and exact folding be used with a Transformer or another tabular backbone, or combined with stronger Euclidean training recipes? It would be useful to distinguish properties that follow algebraically from benefits that still require experiments.
  2. Could the method be used during cross-dataset pretraining, or as a supervised feature adapter to an existing pretrained model? For a frozen in-context predictor, what would be required to preserve the expected input representation and feature/category symmetries? Which setting is the most natural extension of the current method?
4 comments

Part 1/2

Thank you for recognizing HTNN’s strong empirical results and geometry-free deployment. We conducted additional experiments and analyses to address your four questions below, followed by a discussion of the broader directions you raised.

Q1. “Do you have direct evidence that HTNN recovers meaningful hierarchical or relational structure in the feature representations? If not, would an optimization-metric interpretation be more precise than the latent-structure motivation?”

  • We performed an additional analysis and observed that HTNN generally gains more over HTNN-Euc on datasets with more tree-like inputs. On TALENT-tiny, the Spearman correlations between training-input Gromov δ\delta and performance gain are −0.28 for classification and −0.45 for regression. Since a lower Gromov δ\delta indicates stronger tree-likeness, these negative correlations mean that more tree-like inputs correspond to larger gains from negative-curvature training. We add the corresponding classification and regression scatter plots to the revised PDF.
  • Two other possible approaches are: (1) visualizing the learned representations and (2) comparing them against a ground-truth task decision system. Both approaches have limitations. For (1), visualization lacks a quantitative criterion. Representations learned by successfully trained models commonly exhibit some degree of clustering or apparent hierarchy, so visual inspection alone cannot determine which structure is more faithful. For (2), direct comparison requires a reliable ground-truth task decision system, which is unknown for these tasks. Even if a tree model is used as a reference, aligning continuous DNN behavior with discrete tree behavior remains difficult. We therefore do not claim that HTNN recovers a unique ground-truth hierarchy. Our mechanistic claim concerns how negative curvature shapes optimization.
  • The optimization-metric interpretation demonstrates how hyperbolic geometry affects training, while the Gromov δ\delta analysis above empirically supports our latent-structure motivation.

Q2. “Is there evidence that [transverse reduction] explains the accuracy gains, such as a controlled tangent-space implementation of the induced metric [...]? How much of the gain survives such a control?”

  • We conducted an additional experiment by explicitly applying the inverse-metric operator PH(v)P_{\mathrm H}(\mathbf v) in Eq. (14) to HTNN-Euc under its AdamW training procedure. This control isolates the effect of the induced metric from full manifold optimization and tests how much of HTNN’s gain can be explained by transverse attenuation.

    ConfigurationClassification mean rankRegression mean rank
    HTNN1.46671.4000
    HTNN-Euc + PH(v)P_{\mathrm H}(\mathbf v)1.73331.6667
    HTNN-Sphere3.35003.2667
    HTNN-Euc3.45003.6667

    We observe that explicitly applying PH(v)P_{\mathrm H}(\mathbf v) recovers most of the performance gap between HTNN-Euc and HTNN. This provides direct evidence that the transverse attenuation induced by the hyperbolic metric explains most of HTNN’s predictive gain. However, full HTNN remains the best-performing configuration on both tasks, reflecting the contribution of additional mechanisms in complete Riemannian optimization, such as manifold-aware momentum accumulation.

  • We conducted an additional curvature-sweep experiment to examine how different strengths of transverse attenuation affect predictive performance. We found that curvature −1-1 is the most suitable among the tested configurations. At the same radius r=1r=1, Eq. (14) predicts that curvatures of −10-10, −1-1, −0.1-0.1, and −0.01-0.01 retain approximately 7.2%, 72.4%, 96.7%, and 99.7% of the transverse gradient, respectively. Performance declines when transverse attenuation is either too strong (curvature −10-10) or too weak (curvatures −0.1-0.1 and −0.01-0.01). This supports the predictive benefit of moderate transverse attenuation in HTNN.

    CurvatureClassification mean rankRegression mean rank
    −10-103.18333.5333
    −1-1 (HTNN)1.35001.1333
    −0.1-0.12.80002.9333
    −0.01-0.013.11672.5333
    00 (HTNN-Euc)4.55004.8667

Part 2/2

Q3. “Given the 192/0/8 and 98/2/0 results, can you document the differences in preprocessing, predictor capacity, HPO budget, precision, and optimization between HTNN and the released MLP baseline? [...] What do the closest matched comparisons show about the size of the geometric benefit?”

  • HTNN’s MLP receives feature-value representations rather than preprocessed tabular features directly. This changes the input dimension and model capacity. §3.2 and Eq. (10) describe this architectural difference. Appendix C.1 specifies HTNN’s missing-value handling, numerical standardization, and categorical encoding.
  • We confirm that both models use 100 HPO trials per dataset and select their own network width and depth. Table 3 specifies HTNN’s search space. The MLP scores come directly from the official TALENT records rather than from our own rerun (Appendix C.2).
  • We confirm that HTNN uses RiemannianAdam for HFV representations, AdamW for all other parameters, and FP64. The actual training precision of the MLP results reported by TALENT is unknown to us (§3.3 and §4.1).
  • HTNN-Sphere provides a geometric comparison using the same optimizer family. Both use RiemannianAdam. In Table 1, HTNN and HTNN-Sphere have classification mean ranks of 1.38 and 5.12, and regression mean ranks of 1.20 and 4.67, respectively. (§4.3 and Appendix E)
  • RealMLP provides an additional reference using feature representations followed by an MLP. Both HTNN and RealMLP use the representation + MLP approach. In Table 4, HTNN outperforms RealMLP on 158 of 200 classification datasets and 62 of 100 regression datasets.

Q4. “Including HPO, how does the accuracy–compute trade-off change with fewer search trials or lower precision? Given the slower fitting and faster folded inference, in what retraining and prediction-volume regime is the additional training cost worthwhile?”

  • We performed an additional analysis of the recorded Optuna trials and can now report how often smaller HPO budgets recover the final selected configuration. Across the 300 TALENT datasets, the final selected configuration was identified within the first 25, 50, and 100 trials on 46.7% (140/300), 73.0% (219/300), and 100% (300/300) of the datasets, respectively.
  • Training HTNN in FP32 is numerically unstable. Geoopt’s official Lorentz documentation explicitly recommends working with this manifold in double precision. (§4.1; Geoopt Lorentz documentation)
  • HTNN is currently best suited to settings with infrequent retraining and high-volume prediction, where its additional training cost can be amortized over deployment. We are also actively investigating further improvements to its training and inference efficiency.

Broader directions

“Could the same layer and exact folding be used with a Transformer or another tabular backbone [...]?”

  • Yes. The downstream backbone following HFV is not restricted to an MLP and can use other architectures.

“Could the method be used during cross-dataset pretraining, or as a supervised feature adapter to an existing pretrained model?”

  • Yes. We are investigating how to combine HFV with LoRA-style adaptation for broader use.

Thank you again for these constructive questions. We hope the additional experiments and clarifications address them, and we are happy to clarify any remaining points.

I appreciate the authors’ responses and clarifications. I attempted to reproduce the main results using the released HTNN implementation. My run summary covers all 300 TALENT datasets, with 100 HPO trials per dataset and 15 final runs. The MLP and MLP-PLR baselines are taken from the released TALENT results. Using the four-decimal win/tie/loss rule in Appendix C.3, I obtain:

Task and baselinePaper, W/T/LReproduction, W/T/L
Classification vs MLP192/0/893/2/105
Classification vs MLP-PLR179/1/2071/3/126
Regression vs MLP98/2/072/3/25
Regression vs MLP-PLR82/3/1552/3/45

As a cross-check, replacing my HTNN scores with the scores in Table 6, while keeping these same baseline values, reconstructs all four W/T/L counts in Table 4. This suggests that the discrepancy is not simply due to different baseline records or a reversed metric direction.

The reproduced classification accuracy is on average 2.87 percentage points below Table 6. These are means and standard deviations across the final runs. Regression still improves over MLP, but the broad advantage over both baselines is not reproduced.

Could you help identify what I may have missed? Please provide the exact code snapshot, execution commands, dependency versions, and TALENT data/split version used for Tables 4 and 6. Which options are required to reproduce the documented preprocessing, float64 computation, and validation-based checkpoint selection?

The reported empirical gains were central to my initial rating of 3. I would appreciate a reproducible explanation of these differences before deciding whether that assessment can be retained.

Thank you very much for your interest in our work and for making this substantial effort to reproduce our results.

We identified two training details that were missing from this cleaned implementation while we try to anonymize the code.

  • Classification tasks require cross-entropy with label_smoothing=0.1.

  • A CosLog4 learning-rate scheduler is applied to both the Euclidean and Riemannian optimizers.

We have updated the anonymous repository accordingly.

Nevertheless, your reproduced results remain substantially lower than expected. To investigate this discrepancy, we separately reran HTNN on the 45 TALENT-tiny datasets using the provided per-dataset hyperparameter configurations, without rerunning the 100-trial HPO. Due to the limited remaining computation time, we focused this diagnostic on TALENT-tiny. Each result is averaged over 15 runs. Using the four-decimal win/tie/loss rule in Appendix C.3, we obtain:

TaskDatasetsvs MLPvs MLP-PLR
Classification3023/0/719/1/10
Regression1514/0/112/0/3
Overall4537/0/831/1/13

These results show that HTNN remains consistently stronger than both MLP and MLP-PLR in aggregate W/T/L, even without rerunning the full HPO. We will re-run the whole full TELNET datasets experiments and revised the results Tables in the camera ready version.

Please let us know if you have any further questions or if additional details from our side would help identify the remaining discrepancy.

Reviewer_4ep3OpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes HTNN, a supervised tabular prediction model that learns hyperbolic representations of categorical and numerical features and feeds their tangent-space representations into a standard MLP. The method combines geometry-aware training with an exact post-training conversion to ordinary Euclidean operations, which removes manifold computations at inference time.

Critical strengths

S1: The experiments span 300 classification and regression datasets and diverse baselines, and the reported aggregate performance is strong, particularly on classification.

S2:The trained representation layer can be converted into lookup tables and affine transformations without retraining, which preserves predictions while reducing inference cost.

S3: Ablations examine the geometric choices, feature pathways, representation capacity, and direct affine parameterization. The gradient-flow analysis further offers a concrete explanation of how negative curvature modifies representation updates.

Critical weaknesses

W1: The paper integrates hyperbolic geometry into tabular representations, but other non-Euclidean geometries are also available, such as spherical manifolds and mixed-curvature product spaces. Despite the inclusion of a spherical baseline, the authors should further justify the choice of a fixed negative curvature and clarify the conditions under which it is appropriate.

W2: Existing methods, such as numerical feature embeddings [1] and ModernNCA [2], already learn nonlinear representations and task-relevant geometric relationships. The authors should clarify the distinct benefits of explicit negative-curvature constraints over existing representation learning, reparameterization, and gradient preconditioning methods.

[1] Gorishniy, Y., Rubachev, I., & Babenko, A. (2022). On embeddings for numerical features in tabular deep learning. Advances in Neural Information Processing Systems, 35, 24991-25004. [2] Ye, H. J., Yin, H. H., Zhan, D. C., & Chao, W. L. (2025, May). Revisiting nearest neighbor for tabular data: A deep tabular baseline two decades later. In International Conference on Learning Representations (Vol. 2025, pp. 94917-94936).

W3: Given the industrial applications of tabular prediction, the authors should evaluate performance as the numbers of rows and columns increase, as well as under varying missing-data rates. If additional scalability experiments are infeasible, the time and space complexity should at least be analyzed.

W4: The contributions of the hyperbolic representations and the optimizer choice have not been sufficiently disentangled, because the hyperbolic and Euclidean variants are trained with RiemannianAdam and AdamW, respectively. The authors should clarify whether the performance gains stem from the geometric constraints, the optimization mechanisms, or their interaction.

Questions for authors

See the weaknesses above.

4 comments

Part 1/3

Thank you for recognizing HTNN’s strengths in its broad empirical evaluation, exact geometry-free conversion, and optimization analysis. We conducted further experiments and analyses to address W1–W4, as detailed below.

W1. “The authors should further justify the choice of a fixed negative curvature and clarify the conditions under which it is appropriate.”

  • We conducted an additional curvature-sweep experiment and found that −1-1 is the most suitable curvature among the tested configurations. We compared curvatures −10-10, −1-1, −0.1-0.1, and −0.01-0.01, together with HTNN-Euc. Curvature −1-1 ranks first on both tasks, and the statistical tests confirm that it significantly outperforms every alternative (all Holm-adjusted p≤0.0021p\le 0.0021). At the same radius r=1r=1, Eq. (14) predicts that these curvatures retain approximately 7.2%, 72.4%, 96.7%, and 99.7% of the transverse gradient, respectively. The results support moderate transverse attenuation: stronger or weaker attenuation produces lower predictive performance.

    ConfigurationCurvatureClassification mean rankRegression mean rank
    κ=0.1\kappa=0.1−10-103.18333.5333
    κ=1\kappa=1 (HTNN)−1-11.35001.1333
    κ=10\kappa=10−0.1-0.12.80002.9333
    κ=100\kappa=100−0.01-0.013.11672.5333
    HTNN-Euc004.55004.8667
  • We performed an additional analysis and observed that HTNN generally gains more over HTNN-Euc on datasets with more tree-like inputs. On TALENT-tiny, the Spearman correlations between training-input Gromov δ\delta and performance gain are −0.28 for classification and −0.45 for regression. Since a lower Gromov δ\delta indicates stronger tree-likeness, these negative correlations mean that more tree-like inputs correspond to larger gains from negative-curvature training. We add the corresponding classification and regression scatter plots to the revised PDF.

  • Our experiments show that directly learning the curvature can cause numerical instability. When the positive Lorentz scale kk is optimized directly, its Riemannian-gradient conversion and exponential map involve

    grad⁡Rf(x)=J∇f(x)+⟨x,J∇f(x)⟩Lkx\operatorname{grad}^{R}f(x)=J\nabla f(x)+\frac{\langle x,J\nabla f(x)\rangle_{\mathcal L}}{k}x

    Exp⁡x(u)=cosh⁡(∥u∥Lk)x+ksinh⁡(∥u∥Lk)u∥u∥L\operatorname{Exp}_{x}(u)=\cosh\left(\frac{\Vert u\Vert_{\mathcal L}}{\sqrt{k}}\right)x+\sqrt{k}\sinh\left(\frac{\Vert u\Vert_{\mathcal L}}{\sqrt{k}}\right)\frac{u}{\Vert u\Vert_{\mathcal L}}

    where J=diag⁡(−1,1,…,1)J=\operatorname{diag}(-1,1,\ldots,1). As kk approaches zero, the 1/k1/k and 1/k1/\sqrt{k} terms become unbounded and can cause NaNs.

  • Mixed-curvature product spaces introduce an additional choice about how multiple geometries are decomposed and combined. HTNN instead uses a single fixed-curvature geometry, allowing the effect of curvature to be isolated through matched Euclidean, spherical, and curvature-sweep controls. Evaluating mixed-curvature product spaces therefore falls outside the scope of this controlled single-geometry comparison.

Part 2/3

W2. “The authors should clarify the distinct benefits of explicit negative-curvature constraints over existing representation learning, reparameterization, and gradient preconditioning methods.”

  • Gorishniy et al. focus on piecewise-linear and periodic embeddings of numerical features. ModernNCA learns sample-level distances and predicts from neighbors. HTNN provides a new way of constructing representations by using hyperbolic geometry to shape the optimization process (§3.2, §3.4, and Appendix A). Therefore, HTNN differs from these methods in both motivation and methodology.

  • We confirmed that the complete comparison pool covers all three categories raised in the review.

    Category raised in the reviewBaselines included in our comparison
    Representation learningNumerical and feature embeddings: MLP-PLR, RealMLP. Feature-interaction representations: FT-Transformer, AutoInt, TabTransformer, ExcelFormer, TabNet, SwitchTab, DANets, and DCNv2. Structured, prototype, and capsule representations: NODE, PTaRL, TANGOS, and TabCaps. Sample- and kernel-level representations: TabR, ModernNCA, and xRFM.
    Reparameterization and factorizationMLP-PLR, RealMLP, DCNv2, NODE, and TabM. These methods respectively modify the input parameterization, network parameterization, cross-layer factorization, differentiable-tree parameterization, or parameter-efficient ensemble factorization.
    Adaptive gradient preconditioningMLP, ResNet, NODE, SwitchTab, TabNet, TabCaps, TANGOS, DANets, FT-Transformer, AutoInt, DCNv2, SNN, TabTransformer, PTaRL, MLP-PLR, ExcelFormer, TabM, TabR, and ModernNCA are trained with AdamW’s adaptive diagonal preconditioning in the TALENT implementation. RealMLP uses its own adaptive Adam-family optimization procedure.

    HTNN is evaluated against the complete task-specific comparison pools containing these methods, with the results reported in Tables 4 and 5.

W3. “The authors should evaluate performance as the numbers of rows and columns increase, as well as under varying missing-data rates.”

  • We performed further subgroup analyses of our experimental results. HTNN remains highly competitive as the numbers of columns and training rows increase. It ranks first in all classification terciles, within the top three in every column tercile for regression, and third in the largest-training-set regression tercile. Across all missingness groups, HTNN remains first for classification and third for regression. Datasets are ordered by scale and divided into equal-sized terciles. Ties at tercile boundaries are broken by dataset name.

    Scalability with the number of columns

    TaskTercileColumnsHTNN position (mean rank)Median HTNN fitting time
    ClassificationBottom 33%3–131st (4.750)51 s
    ClassificationMiddle 33%13–211st (4.600)58 s
    ClassificationTop 33%29–1441st (3.900)126 s
    RegressionBottom 33%5–82nd (7.600)137 s
    RegressionMiddle 33%8–93rd (5.600)120 s
    RegressionTop 33%10–272nd (4.000)283 s

    Scalability with the number of training rows

    TaskTercilerowsHTNN position (mean rank)Median HTNN fitting time
    ClassificationBottom 33%471–1,6221st (3.600)63 s
    ClassificationMiddle 33%1,910–4,6081st (3.700)107 s
    ClassificationTop 33%7,039–41,9401st (5.950)165 s
    RegressionBottom 33%348–2,5923rd (4.600)110 s
    RegressionMiddle 33%2,672–11,1965th (6.800)140 s
    RegressionTop 33%11,560–50,3883rd (5.800)378 s

    Parentheses report within-group mean ranks. Fitting times exclude HPO.

    Robustness across missing-data rates

    Training-feature missingnessClassification datasetsHTNN classification positionRegression datasetsHTNN regression position
    001791st (4.63)873rd (6.02)
    0<rate≤1%0<\mathrm{rate}\le 1\%121st (2.75)73rd (6.29)
    rate>1%\mathrm{rate}>1\%91st (3.89)63rd (6.33)

Part 3/3

W4. “The authors should clarify whether the performance gains stem from the geometric constraints, the optimization mechanisms, or their interaction.”

  • We further scrutinised the mathematical formulation and implementation. We confirm that, within HTNN, the Riemannian optimizer must be used together with the hyperbolic representation. A Lorentz representation must satisfy ⟨z,z⟩L=−κ\langle z,z\rangle_{\mathcal L}=-\kappa. For a nonzero tangent update g∈TzHκdg\in T_z\mathbb H_\kappa^d, the Euclidean additive rule z+=z−ηgz^{+}=z-\eta g gives

    ⟨z+,z+⟩L=−κ+η2⟨g,g⟩L≠−κ.\langle z^{+},z^{+}\rangle_{\mathcal{L}}=-\kappa+\eta^{2}\langle g,g\rangle_{\mathcal{L}}\neq-\kappa.

    so the updated point no longer lies on the Lorentz manifold. HTNN therefore uses RiemannianAdam to perform geometry-consistent manifold updates. The performance gain should consequently be attributed to the interaction between the HFV representation and its Riemannian optimization rather than to either component in isolation. (§3.3)

We hope these additional experiments and analyses thoroughly address your concerns. Please let us know if you have any remaining questions or if further clarification would be helpful.

Reviewer_4ep3OpenReview ↗

Final Recommendation

Thank you for the detailed rebuttal. Sorry for the late response, and all my concerns have been addressed. I have raised my assessment.

Reviewer_a22AOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces HTNN, a neural network that uses hyperbolic geometry for tabular prediction. It embeds categorical and numerical feature values in hyperbolic space, then maps them to a shared tangent space for prediction with an MLP. After training, these mappings can be replaced by lookup and affine operations, preserving the predictions while avoiding hyperbolic computations at inference. On the TALENT benchmark, HTNN ranks first among 36 methods on 200 classification datasets and third among 34 methods on 100 regression datasets.

Critical strengths
  1. Introducing hyperbolic geometry seems novel, as it is relatively uncommon in tabular learning.
  2. The experiments are thorough and include a wide range of baselines.
  3. The method uses hyperbolic space to support training while retaining standard Euclidean operations at inference, reducing inference overhead and improving practicality.
Critical weaknesses
  1. The method is time-consuming, particularly during training.
  2. The font sizes in the figures are too small, especially the axis labels in Figure 4.
Questions for authors
  1. Do the authors plan to release the code?
  2. Why are HyperDT and HyperRF not included in the comparisons?
1 comment

Thank you for recognizing HTNN’s novelty, thorough experiments, and efficient inference. We address each concern and question below.

“The method is time-consuming, particularly during training.”

We recognize that HTNN is less competitive in training time and that its inference time is competitive mainly against substantially larger models. Both fitting and inference times are strongly affected by the model configurations selected through hyperparameter search. Figure 4 therefore reports the realized runtimes of independently tuned systems rather than an architecture-normalized comparison. HTNN trades this additional training cost for the best classification rank and the third-best regression rank, while exact folding removes all manifold operations from deployment. We are also actively investigating ways to further improve HTNN’s training efficiency.

“The font sizes in the figures are too small, especially the axis labels in Figure 4.”

We revised Figure 4 to ensure that all text, including the axis labels, uses a font size of at least 8 pt.

“Do the authors plan to release the code?”

Yes. The code is already available in the anonymous repository linked in the supplementary material. The camera-ready version will include the official GitHub repository link in the main text.

“Why are HyperDT and HyperRF not included in the comparisons?”

We confirmed that HyperDT and HyperRF cannot be applied directly to raw TALENT tables. Both methods require each sample to be represented as a hyperbolic point, whereas TALENT provides heterogeneous numerical and categorical columns without a predefined hyperbolic encoding. Applying these methods therefore requires an additional mechanism for encoding each complete table row into hyperbolic space, which is not part of either method. (§2)

We hope these clarifications address your concerns. We remain happy to discuss any further details during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.