- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how decoder-only language models handle records that consist almost entirely of numerical measurements. It argues that two structures are only weakly supervised by conventional target-only fine-tuning: (i) within-value sensitivity, i.e. the fact that a digit's position determines its numerical magnitude, which token cross-entropy does not weight; and (ii) across-field structure, i.e. the statistical dependence among measurements in the same record, which target-only tuning never directly supervises. To address these, the authors propose MoMaTune, a modular two-stage post-training recipe. Stage 1 (MODEL) applies ordinary next-token prediction densely over label-free serialized records, optionally augmented with MATCH, a same-record consistency objective that pulls together representations of two partial views of the same example, and with field-order permutation as data augmentation. Stage 2 (TUNE) performs answer-masked supervised fine-tuning in a fixed field order for the downstream target. Validation combines frozen-embedding linear probing (ROC-AUC / R2 with logistic and ridge probes) on eight public datasets totalling 169,285 records (339 of 340 input fields numerical), plus direct-decoder masked-value imputation and digit-perturbation diagnostics, across Qwen2.5-1.5B and larger Qwen3-family checkpoints (4B–27B). The main claim is that MODEL improves embedding quality and numerical-magnitude sensitivity, MATCH adds robustness to partial views, and MODEL→TUNE beats TUNE alone.
Critical strengths
none
Critical weaknesses
- Evaluation is not on an established tabular benchmark. The empirical case rests on only eight public datasets (seven from OpenML plus USDA FoodData), scored with a frozen-embedding linear probe and masked-value imputation. This is far from the standard protocol used in the tabular literature, which routinely reports on recognized suites such as TabArena, OpenML-CC18, or TabBench, and which reports model-level predictive metrics rather than a mean of R2/AUC over eight tasks. A single unweighted 'Emb-8' average over eight datasets — two classification and six regression — is a weak basis for claims that the recipe 'improves linear-probe target prediction' generally, and the same eight datasets appear to be reused for every claim (embedding, imputation, and diagnostics), so there is no held-out battery to test generalization. The reported gains (e.g. EMB-8 from 0.688 to 0.753) are also within a range where dataset selection and probe choice can plausibly dominate.
- Baselines are incomplete for the stated problem. The comparison omits the most directly relevant numerical-representation and tabular-embedding methods. Specifically: (a) FoNE (Fourier number embeddings; arXiv:2502.09741) targets precisely the within-value / magnitude-sensitivity gap that the paper identifies, and is a natural competitor that should be included or at least discussed; and (b) TabEmbed (a generalist tabular embedding model; arXiv:2605.04962) is a directly comparable tabular-embedding baseline for the embedding-based evaluation. The current baseline set is essentially frozen decoder LMs plus a few general-purpose text embedding models, which does not establish that MoMaTune's numerical-structure objectives are the right tool relative to methods explicitly designed for numbers or for tables.
- The paper never acknowledges that its data are tabular. Every 'numerical record' is a row of named numerical features, i.e. a tabular dataset; the field-conditional baseline it reports is a tabular imputation baseline. Yet the words 'table', 'tabular', 'column', and 'feature' are essentially absent from the framing, related work, and claims. This is not merely cosmetic: it (i) prevents the work from being positioned against the large tabular representation-learning literature, (ii) makes the choice of eight datasets and the embedding-probe protocol look arbitrary rather than aligned with an established evaluation culture, and (iii) makes the missing FoNE / TabEmbed comparisons (Weakness 2) and the absent tabular benchmark (Weakness 1) easy to overlook. The work should be reframed as numerical-record / tabular modeling and evaluated as such.
Questions for authors
Q1. Can the authors report results on a recognized tabular benchmark suite (e.g. TabArena, OpenML-CC18, or TALENT) with the same MODEL→TUNE recipe, at least at 1.5B? Concretely: what is the delta over TUNE-only and over a strong tabular baseline (e.g. gradient-boosted trees or a tabular foundation model) measured by standard predictive metrics, not by a frozen-embedding probe average?
Q2. Why are FoNE and TabEmbed not compared or discussed? If they were omitted deliberately, what is the argument that MoMaTune's dense next-token objective dominates Fourier-based number embeddings and tabular-embedding pretraining for the tasks shown? A direct comparison, or a clear statement of scope, is needed before the 'comprehensive evaluation' claim can stand.
Thank you for the detailed responses. Overall, the rebuttal has addressed most of my concerns. However, the clarification on prompt truncation raises a remaining concern. Table RT14 shows that all Tune training examples in four datasets were truncated, removing the question and a substantial portion of the input fields. Although answer supervision was preserved, could the authors clarify whether the evaluation prompts were similarly truncated and whether the main conclusions hold when complete inputs are retained?