- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
HTNN is a tabular network. Each categorical value gets a learnable point on a Lorentz hyperboloid (κ = 1, fixed). Each numerical feature j gets m_j hyperbolic points, and their tangent-space readouts at the origin are combined with coefficients that are affine in the input value, α_j(x) = a_j x + b_j. m_j is set by a label-free entropy/sample-size rule (Eq. 7). The tangent readouts are concatenated and fed to an MLP. Training uses RiemannianAdam for the manifold parameters and AdamW for the rest, in float64. After training the model converts exactly to Euclidean form: cached categorical embedding vectors, a linear numerical embedding u_j x + c_j per feature, and the MLP. The paper argues that the hyperbolic metric acts on training by preserving radial updates and damping transverse ones (Eq. 14, App. A). On the full TALENT benchmark it reports mean rank 1/36 on 200 classification datasets (4.49, against 9.60 for TabPFN-v3) and 3/34 on 100 regression datasets. TALENT-tiny ablations cover geometry, the two pathways, the parameterisation, the coefficient maps and the capacity rule, all significant after Holm correction.
Critical strengths
- Well-conceived, technically advanced construction with exact geometry-free inference. The idea is not fully original (it combines existing hyperbolic embeddings with FT-Transformer-style numerical tokenizers), but the construction is careful and advanced. Sec. 3.4 / Eq. 10 show that the trained model collapses exactly to Euclidean lookups, linear numerical embeddings and an MLP (max |Δ| = 5.5e-17, Sec. 4.5). The deployed model is therefore a standard, cheap architecture, and any gain is attributable purely to the training parameterisation. That makes for a clean scientific question.
- Scale and reporting of the evaluation. Full TALENT (300 datasets), 15 seeds, per-dataset results (Table 6), complete W/T/L and Holm-adjusted Wilcoxon tests (Table 4), and a stated provenance for every baseline (App. C.2). If the numbers hold up, a result this broad would be significant.
Critical weaknesses
-
The "hyperbolic geometry" is a training-time preconditioner, and the paper's framing and ablations do not separate it from simpler explanations.
- log_o is a global diffeomorphism H^d → R^d. So the categorical pathway is a Euclidean embedding table whose gradient is preconditioned by P_H(v) (Eq. 14).
- The numerical pathway is a linear embedding u_j x + c_j (the FT-Transformer tokenizer) trained through an overparameterised factorisation B_j = Σ_k v_{j,k} β_{j,k}^T (Eq. 16), plus the same preconditioner.
- Nothing about hierarchy or latent manifold structure in the data is used or tested: no δ-hyperbolicity estimate, no analysis of the learned embeddings, no curvature sweep.
- The ablation that is meant to isolate geometry (HTNN-Euc, Table 1) changes the optimizer too: RiemannianAdam vs AdamW, with different weight-decay semantics. HTNN also appears to get a separately tuned "Riemannian learning rate" for its embeddings (Table 3) that the Euclidean variants do not. App. F admits that the trajectory comparison "evaluates the realized end-to-end updates of the two training procedures rather than geometry under an identical optimizer."
- Missing controls that would make the claim testable:
- Euclidean embeddings with the same P_H(v) preconditioner;
- a Euclidean variant with its own embedding learning rate;
- a κ sweep (κ → ∞ recovers Euclidean).
- More fundamentally, the paper gives no argument, theoretical or even narrative, for why an a-priori fixed, constant-curvature hyperbolic embedding should be superior. The embeddings feed a nonlinear MLP, which absorbs any smooth invertible reparameterisation of the embedding space; indeed the trained model is exactly Euclidean (Eq. 10). Any benefit must therefore be an optimisation effect, which needs comparison against Euclidean alternatives (separate embedding learning rate, norm constraints, embedding weight decay). Table 1 itself suggests the geometry is not the main driver: HTNN-Softmax/Sigmoid keep the Lorentz geometry yet rank below HTNN-Euc (6.65/6.97 vs 5.68 classification; 7.93/8.20 vs 5.53 regression).
- Without these, the conclusion "negative curvature [is] the effective geometric choice" (Sec. 4.3) is not supported.
-
I am not convinced by the results. Their size and uniformity are implausible given the controlled evidence and the baselines, and suggest selective reporting or an evaluation asymmetry. Reporting only favourable results is common in this area, but here it appears to be the case to an unusual degree:
- (a) The inference model is a plain MLP with linear numerical embeddings, yet it beats TabPFN-v3, TabICL-v2, RealMLP, TabR and MLP-PLR with large mean-rank margins (Table 5). On classification it is significant against every baseline.
- (b) TabM, which the authors ran themselves (App. C.2), comes 25th in classification (21.55) and 22nd in regression, behind plain MLP in both (Table 5). This contradicts the TabM paper and other independent benchmarks, and suggests the author-run pipelines are under-tuned or misconfigured. xRFM, also author-run, deserves the same check.
- (c) In Tables 7–9, adaptive HTNN never loses to any of 8 architectural variants or 24 fixed-capacity variants (e.g. 28/2/0 against nearly every variant; regression mean rank 1.00 ± 0.00 against all 24 fixed-m variants). App. A shows that every m ≥ 2 has the same inference function class, so m only affects optimisation. Zero losses across this many near-equivalent variants is far outside what seed and HPO noise usually produce.
- (d) HTNN runs in float64 with 100 Optuna trials per dataset. The precision of the TALENT-record baselines, and whether their HPO budget matches, are not stated.
- The authors should explain (c), re-run TabM/xRFM with their official configurations, and release code and per-dataset scores for all variants.
Questions for authors
- [Clarification] In HTNN-Euc / HTNN-CENL, do the Euclidean embeddings share the MLP's AdamW learning rate and weight decay, or get their own tuned learning rate as HTNN's manifold parameters do? Is weight decay applied to the Lorentz parameters, and how?
- [Clarification] Please report a Euclidean-embedding model trained with the preconditioner P_H(v) applied explicitly (same optimizer family), and a curvature sweep κ ∈ {0.1, 1, 10, 100}. If the gain persists without the manifold, or vanishes smoothly as κ grows, that decides what the paper is about.
- [Clarification] TabM ranks below plain MLP on both tasks. Which TabM variant, embeddings, HPO space and number of trials were used? Are the results consistent with the TabM authors' reported TALENT/other-benchmark numbers?
- [Clarification] How can adaptive m_j beat fixed m (for every m ∈ {2..12}) on every non-tied dataset, given that the inference class is identical for m ≥ 2? On datasets where Eq. 7 assigns almost all features the same m, does adaptive HTNN tie with that fixed-m variant?
- [Clarification] Were the baselines taken from TALENT records trained in float32? How do HTNN's results change in float32 (this also bears on the fitting-time cost in Sec. 4.5)?
- [Clarification] Is model selection (HPO, early stopping, choice among the 15 runs) done purely on validation, identically for HTNN, the variants and the author-run baselines? Will code be released?
- [Clarification] Does the network learn the geometry of the data, and if so, how? The motivation (Sec. 1) is that tabular data "imply latent manifold structure" that hyperbolic space can capture. But the curvature is fixed (−1), log_o is a global diffeomorphism, and the trained model collapses exactly to a Euclidean MLP, so the geometry adds no representational capacity and acts only through the training dynamics (Eq. 14). Please provide evidence that the learned representations reflect structure in the data: e.g. δ-hyperbolicity of the datasets versus HTNN's gain over HTNN-Euc, an analysis or visualisation of the learned HFV embeddings (radial ordering vs. category frequency or hierarchy), or a learned or swept curvature.
2 comments
Part 2/2
Q5. “Were the baselines taken from TALENT records trained in float32? How do HTNN’s results change in float32 [...]?”
- We use the baseline scores directly from the TALENT benchmark leaderboard (Appendix C.2). The actual training precision of the historical TALENT records is unknown to us.
- Training HTNN in FP32 is numerically unstable. Geoopt’s official Lorentz documentation explicitly recommends working with this manifold in double precision (§4.1, Geoopt Lorentz documentation).
- We confirmed that HTNN and TALENT’s trainable baselines use the same budget of 100 HPO trials per dataset. HTNN’s search space is reported in Table 3.
Q6. “Is model selection [...] done purely on validation, identically for HTNN, the variants and the author-run baselines? Will code be released?”
- Yes. Model selection uses only validation data. We use the TALENT benchmark splits, select configurations and stopping points using validation data, and average the test scores from all 15 runs rather than selecting among them. (§4.1 and Appendices C.1–C.3)
- Yes. The code is already available in the anonymous repository linked in the supplementary material. The official GitHub repository link will be added to the main text in the camera-ready version.
Q7. “Does the network learn the geometry of the data, and if so, how? [...] Please provide evidence that the learned representations reflect structure in the data: e.g. -hyperbolicity of the datasets versus HTNN’s gain over HTNN-Euc [...] or a learned or swept curvature.”
- We performed an additional analysis and observed that HTNN generally gains more over HTNN-Euc on datasets with more tree-like inputs. On TALENT-tiny, the Spearman correlations between training-input Gromov and performance gain are −0.28 for classification and −0.45 for regression. Since a lower Gromov indicates stronger tree-likeness, these negative correlations mean that more tree-like inputs correspond to larger gains from negative-curvature training. We add the corresponding classification and regression scatter plots to the revised PDF.
- Two other possible approaches are: (1) visualizing the learned representations and (2) comparing them against a ground-truth task decision system. Both approaches have limitations. For (1), visualization lacks a quantitative criterion. Representations learned by successfully trained models commonly exhibit some degree of clustering or apparent hierarchy, so visual inspection alone cannot determine which structure is more faithful. For (2), direct comparison requires a reliable ground-truth task decision system, which is unknown for these tasks. Even if a tree model is used as a reference, aligning continuous DNN behavior with discrete tree behavior remains difficult. We therefore do not claim that HTNN recovers a unique ground-truth hierarchy. Our mechanistic claim concerns how negative curvature shapes optimization.
Clarification: The meaning of HTNN-Softmax and HTNN-Sigmoid ablations.
- These variants replace unconstrained affine coefficients with nonnegative, bounded coefficients. Softmax additionally requires the coefficients to sum to one. This changes how numerical representations are composed. The direct geometric comparisons are HTNN-Euc, HTNN-Sphere, and the curvature sweep. The explicit control separately tests the induced optimization mechanism. Retaining Lorentz representations does not imply that arbitrary coefficient-composition rules preserve performance. (§4.3 and Appendix E)
We hope these updates resolve the main weaknesses raised in your review. We are happy to discuss any further details during the discussion period.
Part 1/2
Thank you for your detailed technical review and for recognizing the strengths of our contribution. We conducted additional experiments and analyses to address your seven clarification questions and related concerns below.
Q1. “In HTNN-Euc / HTNN-CENL, do the Euclidean embeddings share the MLP’s AdamW learning rate and weight decay, or get their own tuned learning rate as HTNN’s manifold parameters do? Is weight decay applied to the Lorentz parameters, and how?”
Q2. “Please report a Euclidean-embedding model trained with the preconditioner PH(v) applied explicitly [...] and a curvature sweep κ∈0.1,1,10,100.”
We conducted an additional experiment by explicitly applying PH(v) in Eq. (14) to HTNN-Euc under its AdamW training procedure.
We notice that explicitly applying PH(v) recovers most of HTNN’s improvement over HTNN-Euc. However, full HTNN remains the best, reflecting the additional contribution of geometry-consistent mechanisms such as Riemannian momentum accumulation.
We added a new curvature-sweep experiment over κ∈{0.1,1,10,100}, corresponding to curvatures −10, −1, −0.1, and −0.01.
We notice that curvature −1 ranks first on both tasks and outperforms every alternative. At the same radius r=1, Eq. (14) predicts that the four negative curvatures retain approximately 7.2%, 72.4%, 96.7%, and 99.7% of the transverse gradient, respectively. Performance declines when transverse attenuation is either stronger or weaker than the moderate attenuation produced by curvature −1.
Q3. “TabM ranks below plain MLP on both tasks. Which TabM variant, embeddings, HPO space and number of trials were used? [...]”
08301d670a7c854bcf3a73298763484ba58eecdb, including its model variants, embeddings, HPO spaces, and budget of 100 trials per dataset (Appendix C.2). We did so because the TabM paper does not report TALENT results, while the xRFM paper reports only aggregate TALENT results without the per-dataset scores required for our ranking and statistical analyses.Q4. “How can adaptive mj beat fixed m [...] given that the inference class is identical for m≥2? On datasets where Eq. 7 assigns almost all features the same m, does adaptive HTNN tie with that fixed-m variant?”